1
0
mirror of https://github.com/opencv/opencv.git synced 2026-07-29 15:23:05 +04:00

Merge pull request #26056 from vpisarev:new_dnn_engine

New dnn engine #26056

This is the 1st PR with the new engine; CI is green and PR is ready to be merged, I think.
Merge together with https://github.com/opencv/opencv_contrib/pull/3794

---

**Known limitations:**
* [solved] OpenVINO is temporarily disabled, but is probably easy to restore (it's not a deal breaker to merge this PR, I guess)
* The new engine does not support any backends nor any targets except for the default CPU implementation. But it's possible to choose the old engine when loading a model, then all the functionality is available.
* [Caffe patch is here: #26208] The new engine only supports ONNX. When a model is constructed manually or is loaded from a file of different format (.tf, .tflite, .caffe, .darknet), the old engine is used.
* Even in the case of ONNX some layers are not supported by the new engine, such as all quantized layers (including DequantizeLinear, QuantizeLinear, QLinearConv etc.), LSTM, GRU, .... It's planned, of course, to have full support for ONNX by OpenCV 5.0 gold release. When a loaded model contains unsupported layers, we switch to the old engine automatically  (at ONNX parsing time, not at `forward()` time).
* Some layers , e.g. Expat, are only partially supported by the new engine. In the case of unsupported flavours it switches to the old engine automatically (at ONNX parsing time, not at `forward()` time).
* 'Concat' graph optimization is disabled. The optimization eliminates Concat layer and instead makes the layers that generate tensors to be concatenated to write the outputs to the final destination. Of course, it's only possible when `axis=0` or `axis=N=1`. The optimization is not compatible with dynamic shapes since we need to know in advance where to store the tensors. Because some of the layer implementations have been modified to become more compatible with the new engine, the feature appears to be broken even when the old engine is used.
* Some `dnn::Net` API is not available with the new engine. Also, shape inference may return false if some of the output or intermediate tensors' shapes cannot be inferred without running the model. Probably this can be fixed by a dummy run of the model with zero inputs.
* Some overloads of `dnn::Net::getFLOPs()` and `dnn::Net::getMemoryConsumption()` are not exposed any longer in wrapper generators; but the most useful overloads are exposed (and checked by Java tests).
* [in progress] A few Einsum tests related to empty shapes have been disabled due to crashes in the tests and in Einsum implementations. The code and the tests need to be repaired.
* OpenCL implementation of Deconvolution is disabled. It's very bad and very slow anyway; need to be completely revised.
* Deconvolution3D test is now skipped, because it was only supported by CUDA and OpenVINO backends, both of which are not supported by the new engine.
* Some tests, such as FastNeuralStyle, checked that the in the case of CUDA backend there is no fallback to CPU. Currently all layers in the new engine are processed on CPU, so there are many fallbacks. The checks, therefore, have been temporarily disabled.

---

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
This commit is contained in:
Vadim Pisarevsky
2024-10-16 15:28:19 +03:00
committed by GitHub
parent 12738deaef
commit 3cd57ea09e
112 changed files with 11201 additions and 558 deletions
+7
View File
@@ -1124,6 +1124,13 @@ CV_EXPORTS_W void flipND(InputArray src, OutputArray dst, int axis);
*/
CV_EXPORTS_W void broadcast(InputArray src, InputArray shape, OutputArray dst);
/** @brief Broadcast the given Mat to the given shape.
* @param src input array
* @param shape target shape. Note that negative values are not supported.
* @param dst output array that has the given shape
*/
CV_EXPORTS void broadcast(InputArray src, const MatShape& shape, OutputArray dst);
enum RotateFlags {
ROTATE_90_CLOCKWISE = 0, //!<Rotate 90 degrees clockwise
ROTATE_180 = 1, //!<Rotate 180 degrees clockwise
+266
View File
@@ -67,6 +67,113 @@ enum AccessFlag { ACCESS_READ=1<<24, ACCESS_WRITE=1<<25,
CV_ENUM_FLAGS(AccessFlag)
__CV_ENUM_FLAGS_BITWISE_AND(AccessFlag, int, AccessFlag)
/**
* @brief Enum of data layout for model inference.
* @see Image2BlobParams
*/
enum DataLayout
{
DATA_LAYOUT_UNKNOWN = 0,
DATA_LAYOUT_ND = 1, //!< OpenCV data layout for 2D data.
DATA_LAYOUT_NCHW = 2, //!< OpenCV data layout for 4D data.
DATA_LAYOUT_NCDHW = 3, //!< OpenCV data layout for 5D data.
DATA_LAYOUT_NHWC = 4, //!< Tensorflow-like data layout for 4D data.
DATA_LAYOUT_NDHWC = 5, //!< Tensorflow-like data layout for 5D data.
DATA_LAYOUT_PLANAR = 6, //!< Tensorflow-like data layout, it should only be used at tf or tflite model parsing.
DATA_LAYOUT_BLOCK = 7, //!< Block layout (also referred to as 'NC1HWC0'), which some accelerators need and even on CPU a better performance may be achieved.
// for compatibility with the old code in DNN
DNN_LAYOUT_UNKNOWN = 0,
DNN_LAYOUT_ND = 1, //!< OpenCV data layout for 2D data.
DNN_LAYOUT_NCHW = 2, //!< OpenCV data layout for 4D data.
DNN_LAYOUT_NCDHW = 3, //!< OpenCV data layout for 5D data.
DNN_LAYOUT_NHWC = 4, //!< Tensorflow-like data layout for 4D data.
DNN_LAYOUT_NDHWC = 5, //!< Tensorflow-like data layout for 5D data.
DNN_LAYOUT_PLANAR = 6, //!< Tensorflow-like data layout, it should only be used at tf or tflite model parsing.
DNN_LAYOUT_BLOCK = 7, //!< Block layout (also referred to as 'NC1HWC0'), which some accelerators need and even on CPU a better performance may be achieved.
};
CV_EXPORTS std::string layoutToString(DataLayout layout);
/**
* @brief Represents shape of a matrix/tensor.
* Previously, MatShape was defined as an alias of std::vector<int>,
* but now we use a special structure that provides a few extra benefits:
* 1. avoids any heap operations, since the shape is stored in a plain array. This reduces overhead of shape inference etc.
* 2. includes information about the layout, including the actual number of channels ('C') in the case of block layout.
* 3. distinguishes between empty shape (total() == 0) and 0-dimensional shape (dims == 0, but total() == 1).
*/
struct CV_EXPORTS_W_SIMPLE MatShape
{
enum {MAX_DIMS=10};
MatShape();
explicit MatShape(size_t dims, const int* sizes=nullptr, DataLayout layout=DATA_LAYOUT_UNKNOWN, int C=0);
explicit MatShape(size_t dims, int value, DataLayout layout=DATA_LAYOUT_UNKNOWN);
explicit MatShape(int dims, int value, DataLayout layout=DATA_LAYOUT_UNKNOWN);
explicit MatShape(const std::vector<int>& shape, DataLayout layout=DATA_LAYOUT_UNKNOWN, int C=0);
explicit MatShape(const int* begin, const int* end, DataLayout layout=DATA_LAYOUT_UNKNOWN, int C=0);
explicit MatShape(std::initializer_list<int> shape);
MatShape(const MatShape& shape);
MatShape& operator = (const MatShape& shape);
static MatShape scalar();
template<class _It> MatShape(_It begin, _It end);
// try to mimic basic std::vector<int> functionality
size_t size() const; // returns 0 in the case of scalar tensor. So, please don't use 'size()==0' to check for an empty shape. Use empty() instead.
CV_WRAP bool empty() const; // equivalent to total()==0, but may be slightly faster.
CV_WRAP bool isScalar() const; // dims==0
CV_WRAP void clear();
void resize(size_t newSize, int value=0);
void reserve(size_t maxSize);
void assign(size_t newSize, int value);
void assign(int newSize, int value);
void assign(const int* begin, const int* end);
void assign_(const int* begin, const int* end);
template<class _It> void assign(_It begin, _It end);
void insert(int* where, int value);
void insert(int* where, const int* begin, const int* end);
void insert_(int* where, const int* begin, const int* end);
void insert(int* where, size_t count, int value);
void insert(int* where, int count, int value);
template<class _It> void insert(int* where, _It begin, _It end);
CV_WRAP void erase(int* where);
int* data();
const int* data() const;
int* begin();
const int* begin() const;
int* end();
const int* end() const;
int& back();
const int& back() const;
void push_back(int value);
void emplace_back(int value);
int& operator [](size_t idx);
const int& operator [](size_t idx) const;
CV_WRAP bool hasSymbols() const; // negative elements in the shape may denote 'symbols' instead of actual values.
// compute shape of the result with possible broadcasting
CV_WRAP MatShape expand(const MatShape& another) const;
// convert shape to/from block layout
CV_WRAP MatShape toBlock(int C0) const;
CV_WRAP MatShape fromBlock(DataLayout newLayout) const;
size_t total() const; // returns the total number of elements in the tensor (including padding elements, i.e. the method ignores 'C' in the case of block layout). Returns 1 for scalar tensors. Returns 0 for empty shapes.
operator std::vector<int>() const;
std::string str() const;
int dims;
DataLayout layout;
int C;
int p[MAX_DIMS];
};
CV_EXPORTS bool operator == (const MatShape& shape1, const MatShape& shape2);
CV_EXPORTS bool operator != (const MatShape& shape1, const MatShape& shape2);
CV__DEBUG_NS_BEGIN
class CV_EXPORTS _OutputArray;
@@ -227,6 +334,7 @@ public:
int cols(int i=-1) const;
int rows(int i=-1) const;
Size size(int i=-1) const;
MatShape shape(int i=-1) const;
int sizend(int* sz, int i=-1) const;
bool sameSize(const _InputArray& arr) const;
size_t total(int i=-1) const;
@@ -236,6 +344,7 @@ public:
bool isContinuous(int i=-1) const;
bool isSubmatrix(int i=-1) const;
bool empty() const;
bool empty(int i) const;
void copyTo(const _OutputArray& arr) const;
void copyTo(const _OutputArray& arr, const _InputArray & mask) const;
size_t offset(int i=-1) const;
@@ -367,10 +476,19 @@ public:
template<typename _Tp> std::vector<std::vector<_Tp> >& getVecVecRef() const;
ogl::Buffer& getOGlBufferRef() const;
cuda::HostMem& getHostMemRef() const;
void create(Size sz, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void create(int rows, int cols, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void create(int dims, const int* size, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void create(const MatShape& shape, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void createSameSize(const _InputArray& arr, int mtype) const;
void fit(Size sz, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void fit(int rows, int cols, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void fit(int dims, const int* size, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void fit(const MatShape& shape, int type, int i=-1, bool allowTransposed=false, _OutputArray::DepthMask fixedDepthMask=static_cast<_OutputArray::DepthMask>(0)) const;
void fitSameSize(const _InputArray& arr, int mtype) const;
void release() const;
void clear() const;
void setTo(const _InputArray& value, const _InputArray & mask = _InputArray()) const;
@@ -876,6 +994,20 @@ public:
*/
Mat(const std::vector<int>& sizes, int type);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
*/
Mat(const MatShape& shape, int type);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
*/
Mat(std::initializer_list<int> shape, int type);
/** @overload
@param ndims Array dimensionality.
@param sizes Array of integers specifying an n-dimensional array shape.
@@ -897,6 +1029,25 @@ public:
*/
Mat(const std::vector<int>& sizes, int type, const Scalar& s);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
@param s An optional value to initialize each matrix element with. To set all the matrix elements to
the particular value after the construction, use the assignment operator
Mat::operator=(const Scalar& value) .
*/
Mat(const MatShape& shape, int type, const Scalar& s);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
@param s An optional value to initialize each matrix element with. To set all the matrix elements to
the particular value after the construction, use the assignment operator
Mat::operator=(const Scalar& value) .
*/
Mat(std::initializer_list<int> shape, int type, const Scalar& s);
/** @overload
@param m Array that (as a whole or partly) is assigned to the constructed matrix. No data is copied
@@ -968,6 +1119,34 @@ public:
*/
Mat(const std::vector<int>& sizes, int type, void* data, const size_t* steps=0);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
@param data Pointer to the user data. Matrix constructors that take data and step parameters do not
allocate matrix data. Instead, they just initialize the matrix header that points to the specified
data, which means that no data is copied. This operation is very efficient and can be used to
process external data using OpenCV functions. The external data is not automatically deallocated, so
you should take care of it.
@param steps Array of ndims-1 steps in case of a multi-dimensional array (the last step is always
set to the element size). If not specified, the matrix is assumed to be continuous.
*/
Mat(const MatShape& shape, int type, void* data, const size_t* steps=0);
/** @overload
@param shape Array shape.
@param type Array type. Use CV_8UC1, ..., CV_64FC4 to create 1-4 channel matrices, or
CV_8UC(n), ..., CV_64FC(n) to create multi-channel (up to CV_CN_MAX channels) matrices.
@param data Pointer to the user data. Matrix constructors that take data and step parameters do not
allocate matrix data. Instead, they just initialize the matrix header that points to the specified
data, which means that no data is copied. This operation is very efficient and can be used to
process external data using OpenCV functions. The external data is not automatically deallocated, so
you should take care of it.
@param steps Array of ndims-1 steps in case of a multi-dimensional array (the last step is always
set to the element size). If not specified, the matrix is assumed to be continuous.
*/
Mat(std::initializer_list<int> shape, int type, void* data, const size_t* steps=0);
/** @overload
@param m Array that (as a whole or partly) is assigned to the constructed matrix. No data is copied
by these constructors. Instead, the header pointing to m data or its sub-array is constructed and
@@ -1332,6 +1511,18 @@ public:
*/
Mat reshape(int cn, const std::vector<int>& newshape) const;
/** @overload
* @param cn New number of channels. If the parameter is 0, the number of channels remains the same.
* @param newshape New shape in the form of MatShape.
*/
Mat reshape(int cn, const MatShape& newshape) const;
/** @overload
* @param cn New number of channels. If the parameter is 0, the number of channels remains the same.
* @param newshape New shape in the form of initializer list.
*/
Mat reshape(int cn, std::initializer_list<int> newshape) const;
/** @brief Transposes a matrix.
The method performs matrix transposition by means of matrix expressions. It does not perform the
@@ -1522,6 +1713,18 @@ public:
*/
void create(const std::vector<int>& sizes, int type);
/** @overload
@param shape The new shape.
@param type New matrix type.
*/
void create(const MatShape& shape, int type);
/** @overload
@param shape The new shape.
@param type New matrix type.
*/
void create(std::initializer_list<int> shape, int type);
/** @brief Creates the matrix of the same size as another array.
The method is similar to _OutputArray::createSameSize(arr, type),
@@ -1531,6 +1734,50 @@ public:
*/
void createSameSize(InputArray arr, int type);
/** @brief Similar to create(rows, cols, type), but only reallocates memory if the existing buffer size is not enough.
@param rows New number of rows.
@param cols New number of columns.
@param type New matrix type.
*/
void fit(int rows, int cols, int type);
/** @overload
@param size Alternative new matrix size specification: Size(cols, rows)
@param type New matrix type.
*/
void fit(Size size, int type);
/** @overload
@param ndims New array dimensionality.
@param sizes Array of integers specifying a new array shape.
@param type New matrix type.
*/
void fit(int ndims, const int* sizes, int type);
/** @overload
@param sizes Array of integers specifying a new array shape.
@param type New matrix type.
*/
void fit(const std::vector<int>& sizes, int type);
/** @overload
@param shape The new shape.
@param type New matrix type.
*/
void fit(const MatShape& shape, int type);
/** @overload
@param shape The new shape.
@param type New matrix type.
*/
void fit(std::initializer_list<int> shape, int type);
/** @brief Similar to createSameSize(arr, type), but only reallocates memory if the existing buffer is not enough.
@param arr The other array.
@param type New matrix type.
*/
void fitSameSize(InputArray arr, int type);
/** @brief Increments the reference counter.
The method increments the reference counter associated with the matrix data. If the matrix header
@@ -1833,6 +2080,10 @@ public:
*/
size_t step1(int i=0) const;
/** @brief Returns the shape.
*/
MatShape shape() const;
/** @brief Returns true if the array has no elements.
The method returns true if Mat::total() is 0 or if Mat::data is NULL. Because of pop_back() and
@@ -2522,6 +2773,7 @@ public:
// number of channels and/or different number of rows. see cvReshape.
UMat reshape(int cn, int rows=0) const;
UMat reshape(int cn, int newndims, const int* newsz) const;
UMat reshape(int cn, const MatShape& shape) const;
//! matrix transposition by means of matrix expressions
UMat t() const;
@@ -2549,11 +2801,21 @@ public:
void create(Size size, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void create(int ndims, const int* sizes, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void create(const std::vector<int>& sizes, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void create(const MatShape& shape, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
//! fits the new shape into existing data buffer if possible, otherwise reallocates data.
void fit(int rows, int cols, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void fit(Size size, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void fit(int ndims, const int* sizes, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void fit(const std::vector<int>& sizes, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void fit(const MatShape& shape, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
//! allocates new matrix data unless the matrix already has specified size and type.
// the size is taken from the specified array.
void createSameSize(InputArray arr, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
void fitSameSize(InputArray arr, int type, UMatUsageFlags usageFlags = USAGE_DEFAULT);
//! increases the reference counter; use with care to avoid memleaks
void addref();
//! decreases reference counter;
@@ -2602,6 +2864,10 @@ public:
//! returns the total number of matrix elements
size_t total() const;
/** @brief Returns the shape.
*/
MatShape shape() const;
//! returns N if the matrix is 1-channel (N x ptdim) or ptdim-channel (1 x N) or (N x 1); negative number otherwise
int checkVector(int elemChannels, int depth=-1, bool requireContinuous=true) const;
+58 -3
View File
@@ -71,9 +71,6 @@
namespace cv
{
CV__DEBUG_NS_BEGIN
//! @cond IGNORED
////////////////////////// Custom (raw) type wrapper //////////////////////////
@@ -86,6 +83,64 @@ int rawType()
return (int)CV_MAKETYPE(CV_8U, elemSize);
}
/////////////////////////// MatShape ////////////////////////////////
inline size_t MatShape::size() const { return dims >= 0 ? dims : 0; }
inline bool MatShape::empty() const { return total() == 0; }
inline bool MatShape::isScalar() const { return dims == 0; }
inline int* MatShape::data() { return p; }
inline const int* MatShape::data() const { return p; }
inline int& MatShape::operator [](size_t idx)
{
CV_Assert(idx < (size_t)(dims >= 0 ? dims : 1));
return p[idx];
}
inline const int& MatShape::operator [](size_t idx) const
{
CV_Assert(idx < (size_t)(dims >= 0 ? dims : 1));
return p[idx];
}
template<typename _It> inline MatShape::MatShape(_It begin, _It end)
{
int buf[MAX_DIMS];
int count = 0;
for (_It it = begin; it != end; ++it, ++count) {
CV_Assert(count < MAX_DIMS);
buf[count] = (int)*it;
}
clear();
assign_(buf, buf + count);
}
template<typename _It> inline void MatShape::assign(_It begin, _It end)
{
int buf[MAX_DIMS];
int count = 0;
for (_It it = begin; it != end; ++it, ++count) {
CV_Assert(count < MAX_DIMS);
buf[count] = (int)*it;
}
assign_(buf, buf + count);
}
template<class _It> inline void MatShape::insert(int* where, _It begin, _It end)
{
int buf[MAX_DIMS];
int count = 0;
for (_It it = begin; it != end; ++it, ++count) {
CV_Assert(count < MAX_DIMS);
buf[count] = (int)*it;
}
insert_(where, buf, buf + count);
}
CV__DEBUG_NS_BEGIN
//////////////////////// Input/Output Arrays ////////////////////////
inline void _InputArray::init(int _flags, const void* _obj)
@@ -50,6 +50,7 @@
#endif
#include <cstdio>
#include <ostream>
#if defined(__GNUC__) || defined(__clang__) // at least GCC 3.1+, clang 3.5+
# if defined(__MINGW_PRINTF_FORMAT) // https://sourceforge.net/p/mingw-w64/wiki2/gnu%20printf/.
@@ -484,6 +485,11 @@ int print(const Matx<_Tp, m, n>& matx, FILE* stream = stdout)
return print(Formatter::get()->format(cv::Mat(matx)), stream);
}
// numpy/ONNXRuntime-style matrix pretty-printer
CV_EXPORTS std::ostream& pprint(std::ostream& strm, InputArray tensor, int indent=0,
int edge=3, int wholeTensorThreshold=100,
char parens='\0');
//! @endcond
///////////////////////////////// Formatted string generation /////////////////////////////////