potok/opencv
1
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2026-07-30 15:53:03 +04:00
Code Packages Releases Wiki Activity

Merge pull request #28837 from varun-jaiswal17:gemma3-tokenizer

Browse Source 
Add Gemma3 tokenizer support for dnn #28837

- Adds Gemma3 tokenizer support
- Implements character-level BPE
- Adds 6 tests covering English, phrase, mixed case, numbers, special tokens, and encode/decode
- add gemma3_inference.py

Merge with: 
- **Companion PR**  : https://github.com/opencv/opencv_extra/pull/1346
- forward pass bug in gemma3_inference.py : https://github.com/opencv/opencv/pull/28836

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
This commit is contained in:
Varun Jaiswal
2026-04-29 12:30:27 +05:30
committed by GitHub
parent c83b86eb57
commit faf34f95af
10 changed files with 565 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/core/src/persistence_json.cpp
+27 -1
View File
@@ -527,7 +527,33 @@ public:
case 't' : { buf[i++] = '\t'; break; }
case 'b' : { buf[i++] = '\b'; break; }
case 'f' : { buf[i++] = '\f'; break; }
case 'u' : { CV_PARSE_ERROR_CPP( "'\\uXXXX' currently not supported" ); break; }
case 'u' : {
if (i + 4 >= CV_FS_MAX_LEN)
CV_PARSE_ERROR_CPP("string is too long");
ptr++;
uint32_t codepoint = 0;
for (int k = 0; k < 4; k++, ptr++) {
char hex = *ptr;
uint32_t digit = 0;
if (hex >= '0' && hex <= '9') digit = (uint32_t)(hex - '0');
else if (hex >= 'a' && hex <= 'f') digit = (uint32_t)(hex - 'a') + 10u;
else if (hex >= 'A' && hex <= 'F') digit = (uint32_t)(hex - 'A') + 10u;
else CV_PARSE_ERROR_CPP("invalid \\uXXXX escape sequence");
codepoint = (codepoint << 4) | digit;
}
if (codepoint < 0x80) {
buf[i++] = (char)codepoint;
} else if (codepoint < 0x800) {
buf[i++] = (char)(0xC0 | (codepoint >> 6));
buf[i++] = (char)(0x80 | (codepoint & 0x3F));
} else {
buf[i++] = (char)(0xE0 | (codepoint >> 12));
buf[i++] = (char)(0x80 | ((codepoint >> 6) & 0x3F));
buf[i++] = (char)(0x80 | (codepoint & 0x3F));
}
beg = ptr;
continue;
}
default : { CV_PARSE_ERROR_CPP( "Invalid escape character" ); }
break;
}