Extended primitive core operations to support new types #28964
The support was already there, this PR tests them on edge cases and patch the fix.
closes: https://github.com/opencv/opencv/issues/24580
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Fixed issues identified by PVS Studio #28185
Partially fixes https://github.com/opencv/opencv/issues/28167
Paper: https://pvs-studio.com/en/blog/posts/cpp/1321/
Closed items: N2, N4, N5, N6, N7, N8, N10, N11, N13, N14.
To be continued...
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Use MatShape instead of MatSize inside cv::Mat/cv::UMat #27757
**Merge together with https://github.com/opencv/opencv_contrib/pull/3996**
---
This PR continues cv::Mat/cv::UMat refactoring. See #26056, where `MatShape` was introduced. Now it's put inside cv::Mat/cv::UMat instead of a weird `MatSize`. MatSize is now an alias for MatShape:
**before:**
```
struct MatShape { ... };
struct MatSize { ... };
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { ... /* constructs MatShape out of MatSize and returns it;
layout is always 'unknown', because we don't store it */ }
MatSize size; // size is not valid without the parent cv::Mat,
// because size.p may point to Mat::rows or to Mat::cols,
// depending on the dimensionality, and dims() returns Mat::dims.
MatStep step; // may allocate memory, depending on the dimensionality.
...
};
```
**after:**
```
struct MatShape { ... };
typedef MatShape MatSize; // they are now synonyms
struct Mat {
...
int dims;
int rows;
int cols;
...
MatShape shape() const { return size; } // just return the embedded shape (including the proper layout information)
MatSize size; // size is self-contained data structure that can be used without the parent cv::Mat.
// size.dims is now a copy of dims; size.p[*] contains copies of Mat::rows and Mat::cols when dims <= 2.
MatStep step; // does not allocate extra memory buffers.
...
};
```
There are several reasons to do that:
1. the main reason is to be able to store data layout (MatShape::layout) inside each cv::Mat/cv::UMat. This is necessary for the proper shape inference in DNN module. In particular, it's necessary for the next step of DNN inference optimization where we introduce block-layout-optimized convolution and other operations. Later on, we can use layout information to support non-interleaved images (e.g. RRR...GGG...BBB...) or even batches of such images in core/imgproc modules.
2. the other reason is to represent 3D/4D/5D etc. tensors as cv::Mat/cv::UMat instances more conveniently, without extra dynamic memory allocation. Before this patch we allocated some memory buffers dynamically to store shape & steps for more than 2D arrays. Now the whole cv::Mat/cv::UMat header can be stored completely on stack/in a container. Creating another copy of Mat/UMat header is now done more efficiently.
3. the third reason is to introduce the new coding pattern: `dst.create(src.size, <dst_type>);`. The pattern is suitable for most of element-wise (including cloning) and filtering operations. This pattern does not only look crisp and self-documenting, it will also automatically copy shape (including layout) from the source tensor into the destination matrix/tensor.
4. in the future we might add `colorspace` member to MatShape that will allow to distinguish RGB from BGR or NV12. `dst.create(src.size, <dst_type>);` will then copy the colorspace information as well.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Fixed FP16 mat comparison in tests #26400
make sure that if both compared FP16/BF16 values are bitwise-equal, assume their difference to be 0 (zero), just like in the case of FP32 and FP64, don't try to compare them as floating-point numbers, because they can be NaN's.
**fixes** #24894
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
New dnn engine #26056
This is the 1st PR with the new engine; CI is green and PR is ready to be merged, I think.
Merge together with https://github.com/opencv/opencv_contrib/pull/3794
---
**Known limitations:**
* [solved] OpenVINO is temporarily disabled, but is probably easy to restore (it's not a deal breaker to merge this PR, I guess)
* The new engine does not support any backends nor any targets except for the default CPU implementation. But it's possible to choose the old engine when loading a model, then all the functionality is available.
* [Caffe patch is here: #26208] The new engine only supports ONNX. When a model is constructed manually or is loaded from a file of different format (.tf, .tflite, .caffe, .darknet), the old engine is used.
* Even in the case of ONNX some layers are not supported by the new engine, such as all quantized layers (including DequantizeLinear, QuantizeLinear, QLinearConv etc.), LSTM, GRU, .... It's planned, of course, to have full support for ONNX by OpenCV 5.0 gold release. When a loaded model contains unsupported layers, we switch to the old engine automatically (at ONNX parsing time, not at `forward()` time).
* Some layers , e.g. Expat, are only partially supported by the new engine. In the case of unsupported flavours it switches to the old engine automatically (at ONNX parsing time, not at `forward()` time).
* 'Concat' graph optimization is disabled. The optimization eliminates Concat layer and instead makes the layers that generate tensors to be concatenated to write the outputs to the final destination. Of course, it's only possible when `axis=0` or `axis=N=1`. The optimization is not compatible with dynamic shapes since we need to know in advance where to store the tensors. Because some of the layer implementations have been modified to become more compatible with the new engine, the feature appears to be broken even when the old engine is used.
* Some `dnn::Net` API is not available with the new engine. Also, shape inference may return false if some of the output or intermediate tensors' shapes cannot be inferred without running the model. Probably this can be fixed by a dummy run of the model with zero inputs.
* Some overloads of `dnn::Net::getFLOPs()` and `dnn::Net::getMemoryConsumption()` are not exposed any longer in wrapper generators; but the most useful overloads are exposed (and checked by Java tests).
* [in progress] A few Einsum tests related to empty shapes have been disabled due to crashes in the tests and in Einsum implementations. The code and the tests need to be repaired.
* OpenCL implementation of Deconvolution is disabled. It's very bad and very slow anyway; need to be completely revised.
* Deconvolution3D test is now skipped, because it was only supported by CUDA and OpenVINO backends, both of which are not supported by the new engine.
* Some tests, such as FastNeuralStyle, checked that the in the case of CUDA backend there is no fallback to CPU. Currently all layers in the new engine are processed on CPU, so there are many fallbacks. The checks, therefore, have been temporarily disabled.
---
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
extended Norm tests to prove that cv::norm() already supports all the types.
cv::norm() already provides enough functionality; just extended tests to prove it. See #24887
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
imgproc: add optimized warpAffine kernels for 8U/16U/32F + C1/C3/C4 inputs #25984
Merge wtih https://github.com/opencv/opencv_extra/pull/1198.
Merge with https://github.com/opencv/opencv_contrib/pull/3787.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
C-API cleanup: moved cvErrorStr to new interface, minor ts changes #26101
Merge with opencv/opencv_contrib#3786
**Note:** `toString` might be too generic name (even though it is in `cv::Error::` namespace), another variant is `codeToString` (we have `typeToString` and `depthToString` in check.hpp).
**Note:** _ts_ module seem to have no other C API usage except for `ArrayTest` class which requires refactoring.
Add support for QNX #25832
Build and test instruction for QNX:
https://github.com/chachoi-world/qnx-ports/blob/main/opencv/README.md
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Mask support with CV_Bool in ts and core #25902
Partially cover https://github.com/opencv/opencv/issues/25895
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Added flag to GaussianBlur for faster but not bit-exact implementation #25792
Rationale:
Current implementation of GaussianBlur is almost always bit-exact. It helps to get predictable results according platforms, but prohibits most of approximations and optimization tricks.
The patch converts `borderType` parameter to more generic `flags` and introduces `GAUSS_ALLOW_APPROXIMATIONS` flag to allow not bit-exact implementation. With the flag IPP and generic HAL implementation are called first. The flag naming and location is a subject for discussion.
Replaces https://github.com/opencv/opencv/pull/22073
Possibly related issue: https://github.com/opencv/opencv/issues/24135
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
Added bool support to dnn #25605
Added bool support to dnn pipeline (CPU, OpenVINO and CUDA pipelines).
Added bool support to these layers(CPU and OpenVINO):
- Equal, Greater, GreaterOrEqual, Less, LessOrEqual
- Not
- And, Or, Xor
- Where
Enabled all the conformance tests for these layers.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Tests added for mixed type arithmetic operations #25671
### Changes
* added accuracy tests for mixed type arithmetic operations
_Note: div-by-zero values are removed from checking since the result is implementation-defined in common case_
* added perf tests for the same cases
* fixed a typo in `getMulExtTab()` function that lead to dead code
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
C-API cleanup: apps, imgproc_c and some constants #25075
Merge with https://github.com/opencv/opencv_contrib/pull/3642
* Removed obsolete apps - traincascade and createsamples (please use older OpenCV versions if you need them). These apps relied heavily on C-API
* removed all mentions of imgproc C-API headers (imgproc_c.h, types_c.h) - they were empty, included core C-API headers
* replaced usage of several C constants with C++ ones (error codes, norm modes, RNG modes, PCA modes, ...) - most part of this PR (split into two parts - all modules and calib+3d - for easier backporting)
* removed imgproc C-API headers (as separate commit, so that other changes could be backported to 4.x)
Most of these changes can be backported to 4.x.
* started adding support for new types (16f, 16bf, 32u, 64u, 64s) to arithmetic functions
* fixed several tests; refactored and extended sum(), extended inRange().
* extended countNonZero(), mean(), meanStdDev(), minMaxIdx(), norm() and sum() to support new types (F16, BF16, U32, U64, S64)
* put missing CV_DEPTH_MAX to some function dispatcher tables
* extended findnonzero, hasnonzero with the new types support
* extended mixChannels() to support new types
* minor fix
* fixed a few compile errors on Linux and a few failures in core tests
* fixed a few more warnings and test failures
* trying to fix the remaining warnings and test failures. The test `MulTestGPU.MathOpTest` was disabled - not clear whether to set tolerance - it's not bit-exact operation, as possibly assumed by the test, due to the use of scale and possibly limited accuracy of the intermediate floating-point calculations.
* found that in the current snapshot G-API produces incorrect results in Mul, Div and AddWeighted (at least when using OpenCL on Windows x64 or MacOS x64). Disabled the respective tests.
* attempt to add 0d/1d mat support to OpenCV
* revised the patch; now 1D mat is treated as 1xN 2D mat rather than Nx1.
* a step towards 'green' tests
* another little step towards 'green' tests
* calib test failures seem to be fixed now
* more fixes _core & _dnn
* another step towards green ci; even 0D mat's (a.k.a. scalars) are now partly supported!
* * fixed strange bug in aruco/charuco detector, not sure why it did not work
* also fixed a few remaining failures (hopefully) in dnn & core
* disabled failing GAPI tests - too complex to dig into this compiler pipeline
* hopefully fixed java tests
* trying to fix some more tests
* quick followup fix
* continue to fix test failures and warnings
* quick followup fix
* trying to fix some more tests
* partly fixed support for 0D/scalar UMat's
* use updated parseReduce() from upstream
* trying to fix the remaining test failures
* fixed [ch]aruco tests in Python
* still trying to fix tests
* revert "fix" in dnn's CUDA tensor
* trying to fix dnn+CUDA test failures
* fixed 1D umat creation
* hopefully fixed remaining cuda test failures
* removed training whitespaces
Skip test on SkipTestException at fixture's constructor
* Skip test on SkipTestException at fixture's constructor
* Add warning supression
* Skip Python tests if no test file found
* Skip instances of test fixture with exception at SetUpTestCase
* Skip test with exception at SetUp method
* Try remove warning disable
* Add CV_NORETURN
* Remove FAIL assertion
* Use findDataFile to throw Skip exception
* Throw exception conditionally
* started working on adding 32u, 64u, 64s, bool and 16bf types to OpenCV
* core & imgproc tests seem to pass
* fixed a few compile errors and test failures on macOS x86
* hopefully fixed some compile problems and test failures
* fixed some more warnings and test failures
* trying to fix small deviations in perf_core & perf_imgproc by revering randf_64f to exact version used before
* trying to fix behavior of the new OpenCV with old plugins; there is (quite strong) assumption that video capture would give us frames with depth == CV_8U (0) or CV_16U (2). If depth is > 7 then it means that the plugin is built with the old OpenCV. It needs to be recompiled, of course and then this hack can be removed.
* try to repair the case when target arch does not have FP64 SIMD
* 1. fixed bug in itoa() found by alalek
2. restored ==, !=, > and < univ. intrinsics on ARM32/ARM64.
Replaced sprintf with safer snprintf
* Straightforward replacement of sprintf with safer snprintf
* Trickier replacement of sprintf with safer snprintf
Some functions were changed to take another parameter: the size of the buffer, so that they can pass that size on to snprintf.
Abhishek Gola
Alexander Smorkalov
Alexander Smorkalov
Vadim Pisarevsky
Maksim Shabunin
Yuantao Feng
Alexander Smorkalov
James Choi
alexlyulkov
Rostislav Vasilikhin
Yuantao Feng
Kumataro
Dmitry Kurtaev
Alexander Alekhin
Tomoaki Teshima
Alexander Alekhin
Sean McBride
cudawarped