Merge pull request #23597 from dmatveev:dm/gapi_onnx_py_integration

G-API: Integration branch for ONNX & Python-related changes #23597 # Changes overview ## 1. Expose ONNX backend's Normalization and Mean-value parameters in Python * Since Python G-API bindings rely on `Generic` infer to express Inference, the `Generic` specialization of `onnx::Params` was extended with new methods to control normalization (`/255`) and mean-value; these methods were exposed in the Python bindings * Found some questionable parts in the existing API which I'd like to review/discuss (see comments) UPD: 1. Thanks to @TolyaTalamanov normalization inconsistencies have been identified with `squeezenet1.0-9` ONNX model itself; tests using these model were updated to DISABLE normalization and NOT using mean/value. 2. Questionable parts were removed and tests still pass. ### Details (taken from @TolyaTalamanov's comment): `squeezenet1.0.*onnx` - doesn't require scaling to [0,1] and mean/std because the weights of the first convolution already scaled. ONNX documentation is broken. So the correct approach to use this models is: 1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44 but without normalization step: ``` # DON'T DO IT: # mean_vec = np.array([0.485, 0.456, 0.406]) # stddev_vec = np.array([0.229, 0.224, 0.225]) # norm_img_data = np.zeros(img_data.shape).astype('float32') # for i in range(img_data.shape[0]): # norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i] # # add batch channel # norm_img_data = norm_img_data.reshape(1, 3, 224, 224).astype('float32') # return norm_img_data # INSTEAD return img_data.reshape(1, 3, 224, 224) ``` 2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters: ``` net = cv.gapi.onnx.params('squeezenet', model_filename) net.cfgNormalize('data_0', False) ``` **Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution. --- `squeezenet1.1.*onnx` - requires scaling to [0,1] and mean/std - onnx documentation is correct. 1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44 2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters: ``` net = cv.gapi.onnx.params('squeezenet', model_filename) net.cfgNormalize('data_0', True) // default net.cfgMeanStd('data_0', [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) ``` **Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution. ## 2. Expose Fluid & kernel package-related functionality in Python * `cv::gapi::combine()` * `cv::GKernelPackage::size()` (mainly for testing purposes) * `cv::gapi::imgproc::fluid::kernels()` Added a test for the above. ## 3. Fixed issues with Python stateful kernel handling Fixed error message when `outMeta()` of custom python operation fails. ## 4. Fixed various issues in Python tests 1. `test_gapi_streaming.py` - fixed behavior of Desync test to avoid sporadic issues 2. `test_gapi_infer_onnx.py` - fixed model lookup (it was still using the ONNX Zoo layout but was NOT using the proper env var we use to point to one). ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake
2026-07-29 15:23:05 +04:00 · 2023-05-30 15:52:17 +01:00
parent 93d490213f
commit fc5d412ba7
13 changed files with 261 additions and 79 deletions
@@ -785,15 +785,14 @@ static void unpackMetasToTuple(const cv::GMetaArgs&        meta,
    }
 }

-static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
-                         const cv::GMetaArgs&       meta,
-                         const cv::GArgs&           gargs)
+static cv::GArg run_py_setup(cv::detail::PyObjectHolder setup,
+                             const cv::GMetaArgs        &meta,
+                             const cv::GArgs            &gargs)
 {
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

-    cv::GArg out;
-
+    cv::GArg state;
    try
    {
        // NB: Doesn't increase reference counter (false),
@@ -801,23 +800,20 @@ static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
        // In case exception decrement reference counter.
        cv::detail::PyObjectHolder args(PyTuple_New(meta.size()), false);
        unpackMetasToTuple(meta, gargs, args);
-        // NB: Take an onwership because this state is "Python" type so it will be wrapped as-is
-        // into cv::GArg and stored in GPythonBackend. Object without ownership can't
-        // be dealocated outside this function.
-        cv::detail::PyObjectHolder result(PyObject_CallObject(setup.get(), args.get()), true);

+        PyObject *py_kernel_state = PyObject_CallObject(setup.get(), args.get());
        if (PyErr_Occurred())
        {
            PyErr_PrintEx(0);
            PyErr_Clear();
-            throw std::logic_error("Python kernel failed with error!");
+            throw std::logic_error("Python kernel setup failed with error!");
        }
        // NB: In fact it's impossible situation, because errors were handled above.
-        GAPI_Assert(result.get() && "Python kernel returned NULL!");
+        GAPI_Assert(py_kernel_state && "Python kernel setup returned NULL!");

-        if (!pyopencv_to(result.get(), out, ArgInfo("arg", false)))
+        if (!pyopencv_to(py_kernel_state, state, ArgInfo("arg", false)))
        {
-            util::throw_error(std::logic_error("Unsupported output meta type"));
+            util::throw_error(std::logic_error("Failed to convert python state"));
        }
    }
    catch (...)
@@ -826,7 +822,7 @@ static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
        throw;
    }
    PyGILState_Release(gstate);
-    return out;
+    return state;
 }

 static GMetaArg get_meta_arg(PyObject* obj)
@@ -947,7 +943,7 @@ static PyObject* pyopencv_cv_gapi_kernels(PyObject* , PyObject* py_args, PyObjec
            gapi::python::GPythonFunctor f(
                id.c_str(), std::bind(run_py_meta, cv::detail::PyObjectHolder{out_meta}, _1, _2),
                std::bind(run_py_kernel, cv::detail::PyObjectHolder{run}, _1),
-                std::bind(setup_py, cv::detail::PyObjectHolder{setup}, _1, _2));
+                std::bind(run_py_setup, cv::detail::PyObjectHolder{setup}, _1, _2));
            pkg.include(f);
        }
        else
@@ -14,21 +14,14 @@ try:
    if sys.version_info[:2] < (3, 0):
        raise unittest.SkipTest('Python 2.x is not supported')

-    CLASSIFICATION_MODEL_PATH = "onnx_models/vision/classification/squeezenet/model/squeezenet1.0-9.onnx"
-
-    testdata_required = bool(os.environ.get('OPENCV_DNN_TEST_REQUIRE_TESTDATA', False))
+    CLASSIFICATION_MODEL_PATH = "vision/classification/squeezenet/model/squeezenet1.0-9.onnx"

    class test_gapi_infer(NewOpenCVTests):
-        def find_dnn_file(self, filename, required=None):
-            if not required:
-                required = testdata_required
-            return self.find_file(filename, [os.environ.get('OPENCV_DNN_TEST_DATA_PATH', os.getcwd()),
-                                            os.environ['OPENCV_TEST_DATA_PATH']],
-                                required=required)
+        def find_dnn_file(self, filename):
+            return self.find_file(filename, [os.environ.get('OPENCV_GAPI_ONNX_MODEL_PATH')], False)

        def test_onnx_classification(self):
            model_path = self.find_dnn_file(CLASSIFICATION_MODEL_PATH)
-
            if model_path is None:
                raise unittest.SkipTest("Missing DNN test file")

@@ -45,6 +38,7 @@ try:
            comp = cv.GComputation(cv.GIn(g_in), cv.GOut(g_out))

            net = cv.gapi.onnx.params("squeeze-net", model_path)
+            net.cfgNormalize("data_0", False)
            try:
                out_gapi = comp.apply(cv.gin(in_mat), cv.gapi.compile_args(cv.gapi.networks(net)))
            except cv.error as err:
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+
+import numpy as np
+import cv2 as cv
+import os
+import sys
+import unittest
+
+from tests_common import NewOpenCVTests
+
+
+try:
+
+    if sys.version_info[:2] < (3, 0):
+        raise unittest.SkipTest('Python 2.x is not supported')
+
+    class gapi_kernels_test(NewOpenCVTests):
+
+        def test_fluid_core_package(self):
+            fluid_core = cv.gapi.core.fluid.kernels()
+            self.assertLess(0, fluid_core.size())
+
+        def test_fluid_imgproc_package(self):
+            fluid_imgproc = cv.gapi.imgproc.fluid.kernels()
+            self.assertLess(0, fluid_imgproc.size())
+
+        def test_combine(self):
+            fluid_core = cv.gapi.core.fluid.kernels()
+            fluid_imgproc = cv.gapi.imgproc.fluid.kernels()
+            fluid = cv.gapi.combine(fluid_core, fluid_imgproc)
+            self.assertEqual(fluid_core.size() + fluid_imgproc.size(), fluid.size())
+
+except unittest.SkipTest as e:
+
+    message = str(e)
+
+    class TestSkip(unittest.TestCase):
+        def setUp(self):
+            self.skipTest('Skip tests: ' + message)
+
+        def test_skip():
+            pass
+
+    pass
+
+
+if __name__ == '__main__':
+    NewOpenCVTests.bootstrap()
@@ -24,7 +24,7 @@ try:
                in_types=[cv.GOpaque.Int],
                out_types=[cv.GOpaque.Int])
    class GStatefulCounter:
-        """Accumulate state counter on every call"""
+        """Accumulates state counter on every call"""

        @staticmethod
        def outMeta(desc):
@@ -45,6 +45,22 @@ try:
            return state.counter


+    class SumState:
+        def __init__(self):
+            self.sum = 0
+
+
+    @cv.gapi.op('stateful_sum',
+                in_types=[cv.GOpaque.Int, cv.GOpaque.Int],
+                out_types=[cv.GOpaque.Int])
+    class GStatefulSum:
+        """Accumulates sum on every call"""
+
+        @staticmethod
+        def outMeta(lhs_desc, rhs_desc):
+            return cv.empty_gopaque_desc()
+
+
    class gapi_sample_pipelines(NewOpenCVTests):
        def test_stateful_kernel_single_instance(self):
            g_in  = cv.GOpaque.Int()
@@ -124,6 +140,64 @@ try:
            cc.stop()


+        def test_stateful_multiple_inputs(self):
+            @cv.gapi.kernel(GStatefulSum)
+            class GStatefulSumImpl:
+                """Implementation for GStatefulCounter operation."""
+
+                @staticmethod
+                def setup(lhs_desc, rhs_desc):
+                    return SumState()
+
+                @staticmethod
+                def run(lhs, rhs, state):
+                    state.sum+= lhs + rhs
+                    return state.sum
+
+
+            g_in1 = cv.GOpaque.Int()
+            g_in2 = cv.GOpaque.Int()
+            g_out = GStatefulSum.on(g_in1, g_in2)
+            comp = cv.GComputation(cv.GIn(g_in1, g_in2), cv.GOut(g_out))
+            pkg  = cv.gapi.kernels(GStatefulSumImpl)
+
+            lhs_list = [1, 10, 15]
+            rhs_list = [2, 14, 32]
+
+            ref_out = 0
+            for lhs, rhs in zip(lhs_list, rhs_list):
+                ref_out += lhs + rhs
+                gapi_out = comp.apply(cv.gin(lhs, rhs), cv.gapi.compile_args(pkg))
+                self.assertEqual(ref_out, gapi_out)
+
+
+        def test_stateful_multiple_inputs_throw(self):
+            @cv.gapi.kernel(GStatefulSum)
+            class GStatefulSumImplIncorrect:
+                """Incorrect implementation for GStatefulCounter operation."""
+
+                # NB: setup methods is intentionally
+                # incorrect - accepts one meta arg instead of two
+                @staticmethod
+                def setup(desc):
+                    return SumState()
+
+                @staticmethod
+                def run(lhs, rhs, state):
+                    state.sum+= lhs + rhs
+                    return state.sum
+
+
+            g_in1 = cv.GOpaque.Int()
+            g_in2 = cv.GOpaque.Int()
+            g_out = GStatefulSum.on(g_in1, g_in2)
+            comp = cv.GComputation(cv.GIn(g_in1, g_in2), cv.GOut(g_out))
+            pkg  = cv.gapi.kernels(GStatefulSumImplIncorrect)
+
+            with self.assertRaises(Exception): comp.apply(cv.gin(42, 42),
+                                                          args=cv.gapi.compile_args(pkg))
+
+
 except unittest.SkipTest as e:

    message = str(e)
@@ -17,7 +17,7 @@ try:

    @cv.gapi.op('custom.delay', in_types=[cv.GMat], out_types=[cv.GMat])
    class GDelay:
-        """Delay for 10 ms."""
+        """Delay for 50 ms."""

        @staticmethod
        def outMeta(desc):
@@ -30,7 +30,7 @@ try:

        @staticmethod
        def run(img):
-            time.sleep(0.01)
+            time.sleep(0.05)
            return img


@@ -289,8 +289,7 @@ try:
            ccomp.start()

            # Assert
-            max_num_frames  = 10
-            proc_num_frames = 0
+            max_num_frames  = 50

            out_counter = 0
            desync_out_counter = 0
@@ -307,12 +306,11 @@ try:
                else:
                    none_counter += 1

-                proc_num_frames += 1
-                if proc_num_frames == max_num_frames:
+                if out_counter == max_num_frames:
                    ccomp.stop()
                    break

-            self.assertLess(0, proc_num_frames)
+            self.assertLess(0, out_counter)
            self.assertLess(desync_out_counter, out_counter)
            self.assertLess(0, none_counter)