mirror of
https://github.com/opencv/opencv.git
synced 2026-07-30 15:53:03 +04:00
Merge pull request #27534 from JorgeV92:gsoc2025-tokenizer
GSoC 2025: Add Tokenizer Support to DNN Module #27534 merge with https://github.com/opencv/opencv_extra/pull/1276 ### Summary This pull request introduces initial support for a tokenizer module under `modules/dnn/src/tokenizer` as part of Google Summer of Code 2025 (Project: Tokenization for OpenCV DNN). ### Status - [x] Project structure in place - [x] Initial BPE tokenizer loading - [x] Regex splitting (in progress) - [x] Encoding logic for GPT-2 tokenizer (in progress) - [ ] Documentation (to be improved) ### Goals The goal is to support Hugging Face-compatible tokenization (e.g., GPT-2) natively in C++ to be integrated with DNN inference pipelines. The core pipeline lives in `dnn/src/tokenizer/core_bpe.hpp` and `dnn/src/tokenizer/encoding.hpp`. For Unicode handling I’m using `dnn/src/tokenizer/unicode.hpp`, which is adapted from llama.cpp. ### Feedback Please share early feedback on: - General design structure - Integration strategy with `dnn` - Code organization or naming conventions ### Reference Project: https://summerofcode.withgoogle.com/programs/2025/projects/79SW6eNK
This commit is contained in:
@@ -425,13 +425,12 @@ public:
|
||||
++ptr;
|
||||
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
|
||||
key_name += *ptr;
|
||||
++ptr;
|
||||
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
|
||||
} else {
|
||||
++ptr;
|
||||
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
|
||||
if (*ptr != '\\' && *ptr != '"') key_name += *ptr;
|
||||
} else if (*ptr != '"') {
|
||||
// normal byte: append current, do NOT skip ahead first
|
||||
key_name += *ptr;
|
||||
}
|
||||
++ptr;
|
||||
CV_PERSISTENCE_CHECK_END_OF_BUFFER_BUG_CPP();
|
||||
} while( cv_isprint(*ptr) && *ptr != '"' );
|
||||
|
||||
if( *ptr != '"' )
|
||||
|
||||
Reference in New Issue
Block a user