mirror of
https://github.com/opencv/opencv.git
synced 2026-07-21 19:33:03 +04:00
Vadim Pisarevsky
1b483ffea6
Block layout-based convolution in DNN #28585 merge together with https://github.com/opencv/opencv_extra/pull/1321 Some core parts of the new engine in DNN module have been revised substantially: 1. all tests seem to pass, except for `Test_Graph_Simplifier.ResizeSubgraph`, which has been disabled because it does not take the newly added `TransformLayoutLayer` into account. The test should be reworked perhaps. 1. convolution and related operations (maxpool/avgpool) now use so-called block layout (`DATA_LAYOUT_BLOCK`), where `NxCxHxW` tensors are represented as `NxC1xHxWxC0`, where `C1=(C + C0-1)/C0` and `C0` is a power-of-two (usually 4, 8, 16 or 32). 1. graph is now pre-processed and `TransformLayoutLayer` is inserted to convert data from NCHW or NHWC layout to the block layout or vice versa. The transformations are done in a lazy way only when they are really needed. For example, in the whole Resnet only 2 transformations are performed. 1. transformer-based models and other models that do not use convolutions will run as usual, without going to block layout. 1. there is yet another graph preprocessing stage added that embeds constant weights/scale and bias into convolution and batch norm layers. 1. 'batchnorm', 'activation' and 'adding a residual' are now fused with convolution, just like in the old engine. That brings some noticeable acceleration. 1. optimized convolution kernels have been added. * depthwise convolution, as well as maxpool and avgpool support C0=4, 8, 16 etc. _as long as_ C0 is divisible by the number of fp32 lanes in a SIMD register of the target platform (e.g. on ARM with NEON there must be `C0 % 4 == 0`, on x64 with AVX2 `C0 % 8 == 0`). * non-depthwise convolution only supports C0=8 for now. C0=8 seems to be a sweetspot for ARM with NEON, x64 with AVX2 or RISC-V with RVV (with 128- or 256-bit registers). For some platforms with dedicated matrix accelerators C0=16 or even C0=32 might be more efficient, but we could add the respective kernels later. * only fp32 kernels have been added. fp16/bf16 kernels might be added a little later. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable - [x] Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake
OpenCV: Open Source Computer Vision Library
Resources
- Homepage: https://opencv.org
- Courses: https://opencv.org/courses
- Docs: https://docs.opencv.org/5.x/
- Q&A forum: https://forum.opencv.org
- previous forum (read only): http://answers.opencv.org
- Issue tracking: https://github.com/opencv/opencv/issues
- Additional OpenCV functionality: https://github.com/opencv/opencv_contrib
- Donate to OpenCV: https://opencv.org/support/
Contributing
Please read the contribution guidelines before starting work on a pull request.
Summary of the guidelines:
- One pull request per issue;
- Choose the right base branch;
- Include tests and documentation;
- Clean up "oops" commits before submitting;
- Follow the coding style guide.
Additional Resources
- Submit your OpenCV-based project for inclusion in Community Friday on opencv.org
- Subscribe to the OpenCV YouTube Channel featuring OpenCV Live, an hour-long streaming show
- Follow OpenCV on LinkedIn for daily posts showing the state-of-the-art in computer vision & AI
- Apply to be an OpenCV Volunteer to help organize events and online campaigns as well as amplify them
- Follow OpenCV on Mastodon in the Fediverse
- Follow OpenCV on Twitter
- OpenCV.ai: Computer Vision and AI development services from the OpenCV team.