Merge pull request #27534 from JorgeV92:gsoc2025-tokenizer

GSoC 2025: Add Tokenizer Support to DNN Module #27534

merge with https://github.com/opencv/opencv_extra/pull/1276

### Summary
This pull request introduces initial support for a tokenizer module under `modules/dnn/src/tokenizer` as part of Google Summer of Code 2025 (Project: Tokenization for OpenCV DNN).

### Status
- [x] Project structure in place
- [x] Initial BPE tokenizer loading
- [x] Regex splitting (in progress)
- [x] Encoding logic for GPT-2 tokenizer (in progress)
- [ ] Documentation (to be improved)

### Goals
The goal is to support Hugging Face-compatible tokenization (e.g., GPT-2) natively in C++ to be integrated with DNN inference pipelines. 

The core pipeline lives in `dnn/src/tokenizer/core_bpe.hpp` and `dnn/src/tokenizer/encoding.hpp`. For Unicode handling I’m using `dnn/src/tokenizer/unicode.hpp`, which is adapted from llama.cpp.


### Feedback
Please share early feedback on:
- General design structure
- Integration strategy with `dnn`
- Code organization or naming conventions

### Reference
Project: https://summerofcode.withgoogle.com/programs/2025/projects/79SW6eNK

modules/dnn/include/opencv2/dnn/dnn.hpp

@@ -2099,6 +2099,60 @@ public:
     CV_WRAP int getMaxCandidates() const;
 };
 
+
+/**
+ * @brief High-level tokenizer wrapper for DNN usage.
+ *
+ * Provides a simple API to encode and decode tokens for LLMs.
+ * Models are loaded via Tokenizer::load().
+ *
+ * @code
+ * using namespace cv::dnn;
+ * Tokenizer tok = Tokenizer::load("/path/to/model/");
+ * std::vector<int> ids = tok.encode("hello world");
+ * std::string text = tok.decode(ids);
+ * @endcode
+ */
+class CV_EXPORTS_W_SIMPLE Tokenizer {
+public:
+    /**
+     * @brief Construct a tokenizer with a given method default BPE.
+     * For BPE method you normally call Tokenizer::load() to initialize model data.
+     */
+    Tokenizer();
+
+    /**
+     * @brief Load a tokenizer from a model directory.
+     *
+     * Expects the directory to contain:
+     *  - `config.json` with field `model_type` with value "gpt2" or "gpt4".
+     *  - `tokenizer.json` produced by the corresponding model family.
+     *
+     * The argument is a path prefix; this function concatenates file
+     * names directly (e.g. `model_dir` + "config.json"), so `model_dir` must
+     * end with an appropriate path separator.
+     *
+     * @param model_config  Path to config.json for model.
+     * @return A Tokenizer ready for use. Throws cv::Exception if files are missing or `model_type` is unsupported.
+     */
+    CV_WRAP static Tokenizer load(CV_WRAP_FILE_PATH const std::string& model_config);
+
+    /**
+     * @brief Encode UTF-8 text to token ids (special tokens currently disabled).
+     *
+     * Calls the underlying `CoreBPE::encode` with an empty allowed-special set.
+     *
+     * @param text  UTF-8 input string.
+     * @return Vector of token ids (32-bit ids narrowed to int for convenience).
+     */
+    CV_WRAP std::vector<int> encode(const std::string& text);
+
+    CV_WRAP std::string decode(const std::vector<int>& tokens);
+    struct Impl;
+private:
+    Ptr<Impl> impl_;
+};
+
 //! @}
 CV__DNN_INLINE_NS_END
 }