1
0
mirror of https://github.com/opencv/opencv.git synced 2026-07-30 15:53:03 +04:00

Merge pull request #27534 from JorgeV92:gsoc2025-tokenizer

GSoC 2025: Add Tokenizer Support to DNN Module #27534

merge with https://github.com/opencv/opencv_extra/pull/1276

### Summary
This pull request introduces initial support for a tokenizer module under `modules/dnn/src/tokenizer` as part of Google Summer of Code 2025 (Project: Tokenization for OpenCV DNN).

### Status
- [x] Project structure in place
- [x] Initial BPE tokenizer loading
- [x] Regex splitting (in progress)
- [x] Encoding logic for GPT-2 tokenizer (in progress)
- [ ] Documentation (to be improved)

### Goals
The goal is to support Hugging Face-compatible tokenization (e.g., GPT-2) natively in C++ to be integrated with DNN inference pipelines. 

The core pipeline lives in `dnn/src/tokenizer/core_bpe.hpp` and `dnn/src/tokenizer/encoding.hpp`. For Unicode handling I’m using `dnn/src/tokenizer/unicode.hpp`, which is adapted from llama.cpp.


### Feedback
Please share early feedback on:
- General design structure
- Integration strategy with `dnn`
- Code organization or naming conventions

### Reference
Project: https://summerofcode.withgoogle.com/programs/2025/projects/79SW6eNK
This commit is contained in:
Jorge Velez
2026-04-06 02:46:13 -05:00
committed by GitHub
parent d0313879da
commit a49a293d3c
14 changed files with 8867 additions and 58 deletions
+5 -6
View File
@@ -425,13 +425,12 @@ public:
++ptr;
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
key_name += *ptr;
++ptr;
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
} else {
++ptr;
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
if (*ptr != '\\' && *ptr != '"') key_name += *ptr;
} else if (*ptr != '"') {
// normal byte: append current, do NOT skip ahead first
key_name += *ptr;
}
++ptr;
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
} while( cv_isprint(*ptr) && *ptr != '"' );
if( *ptr != '"' )