GSoC 2025: Add Tokenizer Support to DNN Module #27534
merge with https://github.com/opencv/opencv_extra/pull/1276
### Summary
This pull request introduces initial support for a tokenizer module under `modules/dnn/src/tokenizer` as part of Google Summer of Code 2025 (Project: Tokenization for OpenCV DNN).
### Status
- [x] Project structure in place
- [x] Initial BPE tokenizer loading
- [x] Regex splitting (in progress)
- [x] Encoding logic for GPT-2 tokenizer (in progress)
- [ ] Documentation (to be improved)
### Goals
The goal is to support Hugging Face-compatible tokenization (e.g., GPT-2) natively in C++ to be integrated with DNN inference pipelines.
The core pipeline lives in `dnn/src/tokenizer/core_bpe.hpp` and `dnn/src/tokenizer/encoding.hpp`. For Unicode handling I’m using `dnn/src/tokenizer/unicode.hpp`, which is adapted from llama.cpp.
### Feedback
Please share early feedback on:
- General design structure
- Integration strategy with `dnn`
- Code organization or naming conventions
### Reference
Project: https://summerofcode.withgoogle.com/programs/2025/projects/79SW6eNK
core(opencl): fix inplace transpose race by enforcing LLSS ordering via local barrier #28686
The former inplace transpose implementation allowed a reordering of global-memory operations across work-items. Specifically, the intended LLSS (Load–Load–Store–Store) access pattern could be reordered by the GPU into LSLS (Load–Store–Load–Store), causing partially written tiles to be observed by other work-items and producing incorrect output.
This patch introduces a tiled LDS-based algorithm and adds an explicit:
barrier(CLK_LOCAL_MEM_FENCE);
between the load and store phases.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Nms empty detection fix#28749
Requires opencv_extra: https://github.com/opencv/opencv_extra/pull/1330
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
core: add NEON implementation for rotate function #28609
- This PR adds a NEON intrinsics-based implementation for the rotate function in matrix_transform.cpp for Windows-ARM64.
- The optimized implementation uses ARM NEON intrinsics to accelerate the internal transpose step used by the rotate function.
- In the x64 architecture, the rotate operation benefits from IPP-based optimized implementations. However, on ARM64, the execution falls back to the scalar implementation, which results in lower performance.
- To achieve performance parity with x64, a NEON-based SIMD implementation has been added for ARM64.
- After introducing these changes, the rotate function showed noticeable performance improvements on ARM64 platforms.
<img width="1009" height="817" alt="image" src="https://github.com/user-attachments/assets/8bec0041-b19c-4fc8-9103-532746224515" />
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
Fixes#28396 : out-of-bounds read in SIMD type conversion #28397Fixes#28396Fixes#27080
The vx_load_expand function in WASM intrinsics was using
wasm_v128_load which always loads a full 128-bit register
(16 bytes), even when the function only needed 8 elements.
For example, when converting uint8 to float32:
- vx_load_expand needs 8 uint8 elements
- But wasm_v128_load reads 16 bytes from memory
- This causes an 8-byte out-of-bounds read
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
dnn: fix dst_dp assertion in broadcast for size-1 dims causing crash in lightglue.onnx model #28692
The original assertion CV_Assert(dst_dp == 1) does not handle valid cases where the innermost dimension size is 1 like [10, 5, 1], resulting in dst_dp == 0.
This occurs during broadcasting in LightGlue ONNX model and leads to assertion failure.
Allow dst_dp == 0 for size-1 dimensions to handle this edge case correctly.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Added SIMD support for 64 bit float and fallback for 64 bit int#28663
Merged with : https://github.com/opencv/ci-gha-workflow/pull/299
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Block layout-based convolution in DNN #28585
merge together with https://github.com/opencv/opencv_extra/pull/1321
Some core parts of the new engine in DNN module have been revised substantially:
1. all tests seem to pass, except for `Test_Graph_Simplifier.ResizeSubgraph`, which has been disabled because it does not take the newly added `TransformLayoutLayer` into account. The test should be reworked perhaps.
1. convolution and related operations (maxpool/avgpool) now use so-called block layout (`DATA_LAYOUT_BLOCK`), where `NxCxHxW` tensors are represented as `NxC1xHxWxC0`, where `C1=(C + C0-1)/C0` and `C0` is a power-of-two (usually 4, 8, 16 or 32).
1. graph is now pre-processed and `TransformLayoutLayer` is inserted to convert data from NCHW or NHWC layout to the block layout or vice versa. The transformations are done in a lazy way only when they are really needed. For example, in the whole Resnet only 2 transformations are performed.
1. transformer-based models and other models that do not use convolutions will run as usual, without going to block layout.
1. there is yet another graph preprocessing stage added that embeds constant weights/scale and bias into convolution and batch norm layers.
1. 'batchnorm', 'activation' and 'adding a residual' are now fused with convolution, just like in the old engine. That brings some noticeable acceleration.
1. optimized convolution kernels have been added.
* depthwise convolution, as well as maxpool and avgpool support C0=4, 8, 16 etc. _as long as_ C0 is divisible by the number of fp32 lanes in a SIMD register of the target platform (e.g. on ARM with NEON there must be `C0 % 4 == 0`, on x64 with AVX2 `C0 % 8 == 0`).
* non-depthwise convolution only supports C0=8 for now. C0=8 seems to be a sweetspot for ARM with NEON, x64 with AVX2 or RISC-V with RVV (with 128- or 256-bit registers). For some platforms with dedicated matrix accelerators C0=16 or even C0=32 might be more efficient, but we could add the respective kernels later.
* only fp32 kernels have been added. fp16/bf16 kernels might be added a little later.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
- [x] Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Optimized flip Horizontal #28614
- Refactor flipHoriz implementation.
- Optimizations for horizontal image flipping added.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
core: fix heap-buffer-overflow in YAML parseKey for empty keys #28620
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
### Description
Fixes https://github.com/opencv/opencv/issues/28619
Moves the "empty key" check before the backward do-while scan in `YAMLParser::parseKey()`.
**Problem:** When parsing a YAML mapping with an empty key (e.g. `: 10` at column 0), `endptr == ptr` after the forward scan finds `:`. The do-while loop `do c = *--endptr; while(c == ' ')` always executes at least once, so it decrements `endptr` to `ptr-1` and reads one byte before the heap allocation (ASan: heap-buffer-overflow READ of size 1).
**Fix:** Check `endptr == ptr` before entering the backward loop. If the key is empty, raise `CV_PARSE_ERROR_CPP("An empty key")` immediately without the OOB read.
This contribution was developed with AI assistance (Claude Code).
* added empty set support
* accept unnamed dynamic dims
* Fix for LSTM test failure
* removed converToND
* openvino failing test fix
* ARM CI issue fix
* ARM issue fix
Fix absdiff with int arguments #28267
I believe the fix to the undefined behavior described in #27080 is simply casting to unsigned before subtraction, because
1. casting int to unsigned is well-defined; negative values get represented modulo $2^{32}$
2. overflow in unsigned subtraction is well-defined and the results are modulo $2^{32}$
Since we are computing everything modulo $2^{32}$ and the result must always be a non-negative number below $2^{32}$, this computation should be well-defined and correct.
I have verified this on ARM Apple Clang and on x64 Linux gcc with `-O3` and both produce the correct values in the reproducer of #27080. The test I have added fails with the integer overflow on both platforms I have tested. Perhaps @fengyuentau could verify if this also fixes the issue on the platforms he has tested?
I have added a fix and removed the workarounds. I recommended this approach in #28229, but he decided to revert it.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
core: add NEON support for cvFloor in fast_math.hpp #28243
- This PR adds NEON intrinsics-based implementation for the cvFloor function in fast_math.hpp for Windows-ARM64.
- Both float and double overloads now use NEON intrinsics for cvFloor Function.
- calchist and calchist1d function uses cvFloor function for its computations.
- After adding these changes both functions showed improvement in performance.
**Performance Benchmarks:**
<img width="956" height="273" alt="image" src="https://github.com/user-attachments/assets/a00c98cd-d245-4d11-a9fd-361a3bd89f59" />
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
Fixes#28436.
Replaces a nullptr dereference (unguaranteed to cause a SEGFAULT) by `std::terminate` which will call the current `std::terminate_handler` and then SIGARBT.
PR #27972 added _dst.create(size(), type()) in copyTo's empty() block.
In Debug builds, Mat::release() was resetting flags to MAGIC_VAL,
clearing the type information and causing assertion failures when
destination has fixedType().
Preserve type flags in Mat::release() debug mode by using:
flags = (flags & CV_MAT_TYPE_MASK) | MAGIC_VAL
Thanks to @akretz for suggesting this better approach.
Vincent Rabaud
Abhishek Gola
Alexander Smorkalov
Jorge Velez
Lurie97
Prasad Ayush Kumar
pratham-mcw
Alexander Smorkalov
Anshu
Adrian Kretz
Yang Guanyuhan
Varun Jaiswal
usernotfound-101
Vadim Pisarevsky
Madan mohan Manokar
Matt Van Horn
JoyBoy900908
Murat Raimbekov
ekharkov
Sergey Fedorov
Denis Barucic
Erellu
WalkingDevFlag
Mark Harfouche
Kumataro