1
0
mirror of https://github.com/opencv/opencv.git synced 2026-07-30 15:53:03 +04:00

Merge pull request #23597 from dmatveev:dm/gapi_onnx_py_integration

G-API: Integration branch for ONNX & Python-related changes #23597

# Changes overview

## 1. Expose ONNX backend's Normalization and Mean-value parameters in Python

* Since Python G-API bindings rely on `Generic` infer to express Inference, the `Generic` specialization of `onnx::Params` was extended with new methods to control normalization (`/255`) and mean-value; these methods were exposed in the Python bindings
* Found some questionable parts in the existing API which I'd like to review/discuss (see comments)

UPD:
1. Thanks to @TolyaTalamanov normalization inconsistencies have been identified with `squeezenet1.0-9` ONNX model itself; tests using these model were updated to DISABLE normalization and NOT using mean/value.
2. Questionable parts were removed and tests still pass.

### Details (taken from @TolyaTalamanov's comment):

`squeezenet1.0.*onnx` - doesn't require scaling to [0,1] and mean/std because the weights of the first convolution already scaled. ONNX documentation is broken. So the correct approach to use this models is:

1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44 but without normalization step:
```
# DON'T DO IT:
# mean_vec = np.array([0.485, 0.456, 0.406])
# stddev_vec = np.array([0.229, 0.224, 0.225])
# norm_img_data = np.zeros(img_data.shape).astype('float32')
# for i in range(img_data.shape[0]):
#     norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]
#     # add batch channel
#     norm_img_data = norm_img_data.reshape(1, 3, 224, 224).astype('float32')
#     return norm_img_data

# INSTEAD
return img_data.reshape(1, 3, 224, 224)
```

2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', False)
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.

---

`squeezenet1.1.*onnx` - requires scaling to [0,1] and mean/std - onnx documentation is correct.
1. ONNX: apply preprocessing from the documentation: https://github.com/onnx/models/blob/main/vision/classification/imagenet_preprocess.py#L8-L44
2. G-API: Convert image from BGR to RGB and then pass to `apply` as-is with configuring parameters:
```
net = cv.gapi.onnx.params('squeezenet', model_filename)
net.cfgNormalize('data_0', True) // default
net.cfgMeanStd('data_0', [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
```
**Note**: Results might be difference because `G-API` doesn't apply central crop but just do resize to model resolution.

## 2. Expose Fluid & kernel package-related functionality in Python

* `cv::gapi::combine()`
* `cv::GKernelPackage::size()` (mainly for testing purposes)
* `cv::gapi::imgproc::fluid::kernels()`

Added a test for the above.

## 3. Fixed issues with Python stateful kernel handling

Fixed error message when `outMeta()` of custom python operation fails.

## 4. Fixed various issues in Python tests

1. `test_gapi_streaming.py` - fixed behavior of Desync test to avoid sporadic issues
2. `test_gapi_infer_onnx.py` - fixed model lookup (it was still using the ONNX Zoo layout but was NOT using the proper env var we use to point to one).

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
This commit is contained in:
Dmitry Matveev
2023-05-30 15:52:17 +01:00
committed by GitHub
parent 93d490213f
commit fc5d412ba7
13 changed files with 261 additions and 79 deletions
+11 -15
View File
@@ -785,15 +785,14 @@ static void unpackMetasToTuple(const cv::GMetaArgs& meta,
}
}
static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
const cv::GMetaArgs& meta,
const cv::GArgs& gargs)
static cv::GArg run_py_setup(cv::detail::PyObjectHolder setup,
const cv::GMetaArgs &meta,
const cv::GArgs &gargs)
{
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
cv::GArg out;
cv::GArg state;
try
{
// NB: Doesn't increase reference counter (false),
@@ -801,23 +800,20 @@ static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
// In case exception decrement reference counter.
cv::detail::PyObjectHolder args(PyTuple_New(meta.size()), false);
unpackMetasToTuple(meta, gargs, args);
// NB: Take an onwership because this state is "Python" type so it will be wrapped as-is
// into cv::GArg and stored in GPythonBackend. Object without ownership can't
// be dealocated outside this function.
cv::detail::PyObjectHolder result(PyObject_CallObject(setup.get(), args.get()), true);
PyObject *py_kernel_state = PyObject_CallObject(setup.get(), args.get());
if (PyErr_Occurred())
{
PyErr_PrintEx(0);
PyErr_Clear();
throw std::logic_error("Python kernel failed with error!");
throw std::logic_error("Python kernel setup failed with error!");
}
// NB: In fact it's impossible situation, because errors were handled above.
GAPI_Assert(result.get() && "Python kernel returned NULL!");
GAPI_Assert(py_kernel_state && "Python kernel setup returned NULL!");
if (!pyopencv_to(result.get(), out, ArgInfo("arg", false)))
if (!pyopencv_to(py_kernel_state, state, ArgInfo("arg", false)))
{
util::throw_error(std::logic_error("Unsupported output meta type"));
util::throw_error(std::logic_error("Failed to convert python state"));
}
}
catch (...)
@@ -826,7 +822,7 @@ static cv::GArg setup_py(cv::detail::PyObjectHolder setup,
throw;
}
PyGILState_Release(gstate);
return out;
return state;
}
static GMetaArg get_meta_arg(PyObject* obj)
@@ -947,7 +943,7 @@ static PyObject* pyopencv_cv_gapi_kernels(PyObject* , PyObject* py_args, PyObjec
gapi::python::GPythonFunctor f(
id.c_str(), std::bind(run_py_meta, cv::detail::PyObjectHolder{out_meta}, _1, _2),
std::bind(run_py_kernel, cv::detail::PyObjectHolder{run}, _1),
std::bind(setup_py, cv::detail::PyObjectHolder{setup}, _1, _2));
std::bind(run_py_setup, cv::detail::PyObjectHolder{setup}, _1, _2));
pkg.include(f);
}
else