mirror of
https://github.com/opencv/opencv.git
synced 2026-07-29 23:33:05 +04:00
Merge branch 4.x
This commit is contained in:
Alexander Alekhin
@@ -0,0 +1,4 @@
|
||||
High Level GUI and Media (highgui module) {#tutorial_table_of_content_highgui}
|
||||
=========================================
|
||||
|
||||
Content has been moved to this page: @ref tutorial_table_of_content_app
|
||||
@@ -0,0 +1,4 @@
|
||||
Image Input and Output (imgcodecs module) {#tutorial_table_of_content_imgcodecs}
|
||||
=========================================
|
||||
|
||||
Content has been moved to this page: @ref tutorial_table_of_content_app
|
||||
@@ -0,0 +1,4 @@
|
||||
Video Input and Output (videoio module) {#tutorial_table_of_content_videoio}
|
||||
=========================================
|
||||
|
||||
Content has been moved to this page: @ref tutorial_table_of_content_app
|
||||
|
Before Width: | Height: | Size: 9.2 KiB After Width: | Height: | Size: 9.2 KiB |
|
Before Width: | Height: | Size: 29 KiB After Width: | Height: | Size: 29 KiB |
|
Before Width: | Height: | Size: 2.0 KiB After Width: | Height: | Size: 2.0 KiB |
|
Before Width: | Height: | Size: 135 KiB After Width: | Height: | Size: 135 KiB |
|
Before Width: | Height: | Size: 29 KiB After Width: | Height: | Size: 29 KiB |
|
Before Width: | Height: | Size: 111 KiB After Width: | Height: | Size: 111 KiB |
|
Before Width: | Height: | Size: 53 KiB After Width: | Height: | Size: 53 KiB |
|
Before Width: | Height: | Size: 120 KiB After Width: | Height: | Size: 120 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 39 KiB After Width: | Height: | Size: 39 KiB |
|
Before Width: | Height: | Size: 7.2 KiB After Width: | Height: | Size: 7.2 KiB |
|
Before Width: | Height: | Size: 12 KiB After Width: | Height: | Size: 12 KiB |
@@ -1,6 +1,8 @@
|
||||
Using Creative Senz3D and other Intel RealSense SDK compatible depth sensors {#tutorial_intelperc}
|
||||
=======================================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_orbbec_astra}
|
||||
|
||||
**Note**: This tutorial is partially obsolete since PerC SDK has been replaced with RealSense SDK
|
||||
+2
@@ -1,6 +1,8 @@
|
||||
Using Kinect and other OpenNI compatible depth sensors {#tutorial_kinect_openni}
|
||||
======================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_video_write}
|
||||
@next_tutorial{tutorial_orbbec_astra}
|
||||
|
||||
+40
-21
@@ -1,6 +1,8 @@
|
||||
Using Orbbec Astra 3D cameras {#tutorial_orbbec_astra}
|
||||
======================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_kinect_openni}
|
||||
@next_tutorial{tutorial_intelperc}
|
||||
|
||||
@@ -9,12 +11,12 @@ Using Orbbec Astra 3D cameras {#tutorial_orbbec_astra}
|
||||
|
||||
This tutorial is devoted to the Astra Series of Orbbec 3D cameras (https://orbbec3d.com/product-astra-pro/).
|
||||
That cameras have a depth sensor in addition to a common color sensor. The depth sensors can be read using
|
||||
the OpenNI interface with @ref cv::VideoCapture class. The video stream is provided through the regular camera
|
||||
interface.
|
||||
the open source OpenNI API with @ref cv::VideoCapture class. The video stream is provided through the regular
|
||||
camera interface.
|
||||
|
||||
### Installation Instructions
|
||||
|
||||
In order to use a depth sensor with OpenCV you should do the following steps:
|
||||
In order to use the Astra camera's depth sensor with OpenCV you should do the following steps:
|
||||
|
||||
-# Download the latest version of Orbbec OpenNI SDK (from here <https://orbbec3d.com/develop/>).
|
||||
Unzip the archive, choose the build according to your operating system and follow installation
|
||||
@@ -70,24 +72,32 @@ In order to use a depth sensor with OpenCV you should do the following steps:
|
||||
|
||||
### Code
|
||||
|
||||
To get both depth and color frames, two @ref cv::VideoCapture objects should be created:
|
||||
The Astra Pro camera has two sensors -- a depth sensor and a color sensor. The depth sensor
|
||||
can be read using the OpenNI interface with @ref cv::VideoCapture class. The video stream is
|
||||
not available through OpenNI API and is only provided via the regular camera interface.
|
||||
So, to get both depth and color frames, two @ref cv::VideoCapture objects should be created:
|
||||
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Open streams
|
||||
|
||||
The first object will use the regular Video4Linux2 interface to access the color sensor. The second one
|
||||
is using OpenNI2 API to retrieve depth data.
|
||||
The first object will use the OpenNI2 API to retrieve depth data. The second one uses the
|
||||
Video4Linux2 interface to access the color sensor. Note that the example above assumes that
|
||||
the Astra camera is the first camera in the system. If you have more than one camera connected,
|
||||
you may need to explicitly set the proper camera number.
|
||||
|
||||
Before using the created VideoCapture objects you may want to setup stream parameters by setting
|
||||
objects' properties. The most important parameters are frame width, frame height and fps:
|
||||
Before using the created VideoCapture objects you may want to set up stream parameters by setting
|
||||
objects' properties. The most important parameters are frame width, frame height and fps.
|
||||
For this example, we’ll configure width and height of both streams to VGA resolution, which is
|
||||
the maximum resolution available for both sensors, and we’d like both stream parameters to be the
|
||||
same for easier color-to-depth data registration:
|
||||
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Setup streams
|
||||
|
||||
For setting and getting some property of sensor data generators use @ref cv::VideoCapture::set and
|
||||
For setting and retrieving some property of sensor data generators use @ref cv::VideoCapture::set and
|
||||
@ref cv::VideoCapture::get methods respectively, e.g. :
|
||||
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Get properties
|
||||
|
||||
The following properties of cameras available through OpenNI interfaces are supported for the depth
|
||||
The following properties of cameras available through OpenNI interface are supported for the depth
|
||||
generator:
|
||||
|
||||
- @ref cv::CAP_PROP_FRAME_WIDTH -- Frame width in pixels.
|
||||
@@ -106,15 +116,16 @@ generator:
|
||||
- @ref cv::CAP_PROP_OPENNI_FRAME_MAX_DEPTH -- A maximum supported depth of the camera in mm.
|
||||
- @ref cv::CAP_PROP_OPENNI_BASELINE -- Baseline value in mm.
|
||||
|
||||
After the VideoCapture objects are set up you can start reading frames from them.
|
||||
After the VideoCapture objects have been set up, you can start reading frames from them.
|
||||
|
||||
@note
|
||||
OpenCV's VideoCapture provides synchronous API, so you have to grab frames in a new thread
|
||||
to avoid one stream blocking while another stream is being read. VideoCapture is not a
|
||||
thread-safe class, so you need to be careful to avoid any possible deadlocks or data races.
|
||||
|
||||
Example implementation that gets frames from each sensor in a new thread and stores them
|
||||
in a list along with their timestamps:
|
||||
As there are two video sources that should be read simultaneously, it’s necessary to create two
|
||||
threads to avoid blocking. Example implementation that gets frames from each sensor in a new thread
|
||||
and stores them in a list along with their timestamps:
|
||||
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Read streams
|
||||
|
||||
@@ -130,17 +141,25 @@ VideoCapture can retrieve the following data:
|
||||
|
||||
-# data given from the color sensor is a regular BGR image (CV_8UC3).
|
||||
|
||||
When new data is available a reading thread notifies the main thread. A frame is stored in the
|
||||
ordered list -- the first frame is the latest one:
|
||||
When new data are available, each reading thread notifies the main thread using a condition variable.
|
||||
A frame is stored in the ordered list -- the first frame in the list is the earliest captured,
|
||||
the last frame is the latest captured. As depth and color frames are read from independent sources
|
||||
two video streams may become out of sync even when both streams are set up for the same frame rate.
|
||||
A post-synchronization procedure can be applied to the streams to combine depth and color frames into
|
||||
pairs. The sample code below demonstrates this procedure:
|
||||
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Show color frame
|
||||
@snippetlineno samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp Pair frames
|
||||
|
||||
Depth frames can be picked the same way from the `depthFrames` list.
|
||||
In the code snippet above the execution is blocked until there are some frames in both frame lists.
|
||||
When there are new frames, their timestamps are being checked -- if they differ more than a half of
|
||||
the frame period then one of the frames is dropped. If timestamps are close enough, then two frames
|
||||
are paired. Now, we have two frames: one containing color information and another one -- depth information.
|
||||
In the example above retrieved frames are simply shown with cv::imshow function, but you can insert
|
||||
any other processing code here.
|
||||
|
||||
After that, you'll have two frames: one containing color information and another one -- depth
|
||||
information. In the sample images below you can see the color frame and the depth frame showing
|
||||
the same scene. Looking at the color frame it's hard to distinguish plant leaves from leaves painted
|
||||
on a wall, but the depth data makes it easy.
|
||||
In the sample images below you can see the color frame and the depth frame representing the same scene.
|
||||
Looking at the color frame it's hard to distinguish plant leaves from leaves painted on a wall,
|
||||
but the depth data makes it easy.
|
||||
|
||||
|
||||
|
||||
+10
@@ -1,6 +1,16 @@
|
||||
Reading Geospatial Raster files with GDAL {#tutorial_raster_io_gdal}
|
||||
=========================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_trackbar}
|
||||
@next_tutorial{tutorial_video_input_psnr_ssim}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Marvin Smith |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Geospatial raster data is a heavily used product in Geographic Information Systems and
|
||||
Photogrammetry. Raster data typically can represent imagery and Digital Elevation Models (DEM). The
|
||||
standard library for loading GIS imagery is the Geographic Data Abstraction Library [(GDAL)](http://www.gdal.org). In this
|
||||
@@ -0,0 +1,10 @@
|
||||
Application utils (highgui, imgcodecs, videoio modules) {#tutorial_table_of_content_app}
|
||||
=======================================================
|
||||
|
||||
- @subpage tutorial_trackbar
|
||||
- @subpage tutorial_raster_io_gdal
|
||||
- @subpage tutorial_video_input_psnr_ssim
|
||||
- @subpage tutorial_video_write
|
||||
- @subpage tutorial_kinect_openni
|
||||
- @subpage tutorial_orbbec_astra
|
||||
- @subpage tutorial_intelperc
|
||||
+10
@@ -1,6 +1,16 @@
|
||||
Adding a Trackbar to our applications! {#tutorial_trackbar}
|
||||
======================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@next_tutorial{tutorial_raster_io_gdal}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
|
||||
- In the previous tutorials (about @ref tutorial_adding_images and the @ref tutorial_basic_linear_transform)
|
||||
you might have noted that we needed to give some **input** to our programs, such
|
||||
as \f$\alpha\f$ and \f$beta\f$. We accomplished that by entering this data using the Terminal.
|
||||
+8
@@ -1,8 +1,16 @@
|
||||
Video Input with OpenCV and similarity measurement {#tutorial_video_input_psnr_ssim}
|
||||
==================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_raster_io_gdal}
|
||||
@next_tutorial{tutorial_video_write}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
+9
-2
@@ -1,9 +1,16 @@
|
||||
Creating a video with OpenCV {#tutorial_video_write}
|
||||
============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_video_input_psnr_ssim}
|
||||
@next_tutorial{tutorial_kinect_openni}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
@@ -59,7 +66,7 @@ extension, its first version. A direct limitation of this is that you cannot sav
|
||||
larger than 2 GB. Furthermore you can only create and expand a single video track inside the
|
||||
container. No audio or other track editing support here. Nevertheless, any video codec present on
|
||||
your system might work. If you encounter some of these limitations you will need to look into more
|
||||
specialized video writing libraries such as *FFMpeg* or codecs as *HuffYUV*, *CorePNG* and *LCL*. As
|
||||
specialized video writing libraries such as *FFmpeg* or codecs as *HuffYUV*, *CorePNG* and *LCL*. As
|
||||
an alternative, create the video track with OpenCV and expand it with sound tracks or convert it to
|
||||
other formats by using video manipulation programs such as *VirtualDub* or *AviSynth*.
|
||||
|
||||
@@ -109,7 +116,7 @@ const string NAME = source.substr(0, pAt) + argv[2][0] + ".avi"; // Form the n
|
||||
@code{.cpp}
|
||||
CV_FOURCC('P','I','M,'1') // this is an MPEG1 codec from the characters to integer
|
||||
@endcode
|
||||
If you pass for this argument minus one than a window will pop up at runtime that contains all
|
||||
If you pass for this argument minus one then a window will pop up at runtime that contains all
|
||||
the codec installed on your system and ask you to select the one to use:
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Camera calibration With OpenCV {#tutorial_camera_calibration}
|
||||
==============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_camera_calibration_square_chess}
|
||||
@next_tutorial{tutorial_real_time_pose}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 4.0 |
|
||||
|
||||
|
||||
Cameras have been around for a long-long time. However, with the introduction of the cheap *pinhole*
|
||||
cameras in the late 20th century, they became a common occurrence in our everyday life.
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
Create calibration pattern {#tutorial_camera_calibration_pattern}
|
||||
=========================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@next_tutorial{tutorial_camera_calibration_square_chess}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Laurent Berger |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
|
||||
The goal of this tutorial is to learn how to create calibration pattern.
|
||||
|
||||
|
||||
+7
@@ -1,9 +1,16 @@
|
||||
Camera calibration with square chessboard {#tutorial_camera_calibration_square_chess}
|
||||
=========================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_camera_calibration_pattern}
|
||||
@next_tutorial{tutorial_camera_calibration}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Victor Eruhimov |
|
||||
| Compatibility | OpenCV >= 4.0 |
|
||||
|
||||
|
||||
The goal of this tutorial is to learn how to calibrate a camera given a set of chessboard images.
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 8.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 83 KiB |
@@ -1,8 +1,15 @@
|
||||
Interactive camera calibration application {#tutorial_interactive_calibration}
|
||||
==============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_real_time_pose}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Vladislav Sovrasov |
|
||||
| Compatibility | OpenCV >= 3.1 |
|
||||
|
||||
|
||||
According to classical calibration technique user must collect all data first and when run @ref cv::calibrateCamera function
|
||||
to obtain camera parameters. If average re-projection error is huge or if estimated parameters seems to be wrong, process of
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Real Time pose estimation of a textured object {#tutorial_real_time_pose}
|
||||
==============================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_camera_calibration}
|
||||
@next_tutorial{tutorial_interactive_calibration}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Edgar Riba |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
|
||||
Nowadays, augmented reality is one of the top research topic in computer vision and robotics fields.
|
||||
The most elemental problem in augmented reality is the estimation of the camera pose respect of an
|
||||
|
||||
@@ -1,58 +1,8 @@
|
||||
Camera calibration and 3D reconstruction (calib3d module) {#tutorial_table_of_content_calib3d}
|
||||
==========================================================
|
||||
|
||||
Although we get most of our images in a 2D format they do come from a 3D world. Here you will learn how to find out 3D world information from 2D images.
|
||||
|
||||
- @subpage tutorial_camera_calibration_pattern
|
||||
|
||||
*Languages:* Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Laurent Berger
|
||||
|
||||
You will learn how to create some calibration pattern.
|
||||
|
||||
- @subpage tutorial_camera_calibration_square_chess
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Victor Eruhimov
|
||||
|
||||
You will use some chessboard images to calibrate your camera.
|
||||
|
||||
- @subpage tutorial_camera_calibration
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 4.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
Camera calibration by using either the chessboard, circle or the asymmetrical circle
|
||||
pattern. Get the images either from a camera attached, a video file or from an image
|
||||
collection.
|
||||
|
||||
- @subpage tutorial_real_time_pose
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Edgar Riba
|
||||
|
||||
Real time pose estimation of a textured object using ORB features, FlannBased matcher, PnP
|
||||
approach plus Ransac and Linear Kalman Filter to reject possible bad poses.
|
||||
|
||||
- @subpage tutorial_interactive_calibration
|
||||
|
||||
*Compatibility:* \> OpenCV 3.1
|
||||
|
||||
*Author:* Vladislav Sovrasov
|
||||
|
||||
Camera calibration by using either the chessboard, chAruco, asymmetrical circle or dual asymmetrical circle
|
||||
pattern. Calibration process is continuous, so you can see results after each new pattern shot.
|
||||
As an output you get average reprojection error, intrinsic camera parameters, distortion coefficients and
|
||||
confidence intervals for all of evaluated variables.
|
||||
|
||||
@@ -1,9 +1,17 @@
|
||||
Adding (blending) two images using OpenCV {#tutorial_adding_images}
|
||||
=========================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_operations}
|
||||
@next_tutorial{tutorial_basic_linear_transform}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
We will learn how to blend two images!
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Changing the contrast and brightness of an image! {#tutorial_basic_linear_transform}
|
||||
=================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_adding_images}
|
||||
@next_tutorial{tutorial_discrete_fourier_transform}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Discrete Fourier Transform {#tutorial_discrete_fourier_transform}
|
||||
==========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_basic_linear_transform}
|
||||
@next_tutorial{tutorial_file_input_output_with_xml_yml}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
+7
@@ -1,9 +1,16 @@
|
||||
File Input and Output using XML and YAML files {#tutorial_file_input_output_with_xml_yml}
|
||||
==============================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_discrete_fourier_transform}
|
||||
@next_tutorial{tutorial_how_to_use_OpenCV_parallel_for_}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
How to scan images, lookup tables and time measurement with OpenCV {#tutorial_how_to_scan_images}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_the_basic_image_container}
|
||||
@next_tutorial{tutorial_mat_mask_operations}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
+6
@@ -1,8 +1,14 @@
|
||||
How to use the OpenCV parallel_for_ to parallelize your code {#tutorial_how_to_use_OpenCV_parallel_for_}
|
||||
==================================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_file_input_output_with_xml_yml}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 6.9 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.1 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.7 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.3 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.8 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.2 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.4 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.8 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 8.9 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 6.1 KiB |
@@ -1,9 +1,16 @@
|
||||
Mask operations on matrices {#tutorial_mat_mask_operations}
|
||||
===========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_how_to_scan_images}
|
||||
@next_tutorial{tutorial_mat_operations}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Mask operations on matrices are quite simple. The idea is that we recalculate each pixel's value in
|
||||
an image according to a mask matrix (also known as kernel). This mask holds values that will adjust
|
||||
how much influence neighboring pixels (and the current pixel) have on the new pixel value. From a
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
Operations with images {#tutorial_mat_operations}
|
||||
======================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_mat_mask_operations}
|
||||
@next_tutorial{tutorial_adding_images}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Input/Output
|
||||
------------
|
||||
|
||||
|
||||
+7
@@ -1,8 +1,15 @@
|
||||
Mat - The Basic Image Container {#tutorial_mat_the_basic_image_container}
|
||||
===============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@next_tutorial{tutorial_how_to_scan_images}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Bernát Gábor |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,97 +1,12 @@
|
||||
The Core Functionality (core module) {#tutorial_table_of_content_core}
|
||||
=====================================
|
||||
|
||||
Here you will learn the about the basic building blocks of the library. A must read and know for
|
||||
understanding how to manipulate the images on a pixel level.
|
||||
|
||||
- @subpage tutorial_mat_the_basic_image_container
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
You will learn how to store images in the memory and how to print out their content to the
|
||||
console.
|
||||
|
||||
- @subpage tutorial_how_to_scan_images
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
You'll find out how to scan images (go through each of the image pixels) with OpenCV.
|
||||
Bonus: time measurement with OpenCV.
|
||||
|
||||
|
||||
- @subpage tutorial_mat_mask_operations
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
You'll find out how to scan images with neighbor access and use the @ref cv::filter2D
|
||||
function to apply kernel filters on images.
|
||||
|
||||
- @subpage tutorial_mat_operations
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
Reading/writing images from file, accessing pixels, primitive operations, visualizing images.
|
||||
|
||||
- @subpage tutorial_adding_images
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
We will learn how to blend two images!
|
||||
|
||||
- @subpage tutorial_basic_linear_transform
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
We will learn how to change our image appearance!
|
||||
|
||||
- @subpage tutorial_discrete_fourier_transform
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
You will see how and why use the Discrete Fourier transformation with OpenCV.
|
||||
|
||||
|
||||
- @subpage tutorial_file_input_output_with_xml_yml
|
||||
|
||||
*Languages:* C++, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Bernát Gábor
|
||||
|
||||
You will see how to use the @ref cv::FileStorage data structure of OpenCV to write and read
|
||||
data to XML or YAML file format.
|
||||
|
||||
- @subpage tutorial_how_to_use_OpenCV_parallel_for_
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \>= OpenCV 2.4.3
|
||||
|
||||
You will see how to use the OpenCV parallel_for_ to easily parallelize your code.
|
||||
|
||||
@@ -1,6 +1,14 @@
|
||||
# How to run custom OCR model {#tutorial_dnn_OCR}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_custom_layers}
|
||||
@next_tutorial{tutorial_dnn_text_spotting}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Zihao Mu |
|
||||
| Compatibility | OpenCV >= 4.3 |
|
||||
|
||||
## Introduction
|
||||
|
||||
@@ -43,4 +51,4 @@ The input of text recognition model is the output of the text detection model, w
|
||||
|
||||
DenseNet_CTC has the smallest parameters and best FPS, and it is suitable for edge devices, which are very sensitive to the cost of calculation. If you have limited computing resources and want to achieve better accuracy, VGG_CTC is a good choice.
|
||||
|
||||
CRNN_VGG_BiLSTM_CTC is suitable for scenarios that require high recognition accuracy.
|
||||
CRNN_VGG_BiLSTM_CTC is suitable for scenarios that require high recognition accuracy.
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
# How to run deep networks on Android device {#tutorial_dnn_android}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_halide_scheduling}
|
||||
@next_tutorial{tutorial_dnn_yolo}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.3 |
|
||||
|
||||
## Introduction
|
||||
In this tutorial you'll know how to run deep learning networks on Android device
|
||||
using OpenCV deep learning module.
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
# Custom deep learning layers support {#tutorial_dnn_custom_layers}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_javascript}
|
||||
@next_tutorial{tutorial_dnn_OCR}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.4.1 |
|
||||
|
||||
## Introduction
|
||||
Deep learning is a fast growing area. The new approaches to build neural networks
|
||||
usually introduce new types of layers. They could be modifications of existing
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
Load Caffe framework models {#tutorial_dnn_googlenet}
|
||||
===========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@next_tutorial{tutorial_dnn_halide}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Vitaliy Lyudvichenko |
|
||||
| Compatibility | OpenCV >= 3.3 |
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
# How to enable Halide backend for improve efficiency {#tutorial_dnn_halide}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_googlenet}
|
||||
@next_tutorial{tutorial_dnn_halide_scheduling}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.3 |
|
||||
|
||||
## Introduction
|
||||
This tutorial guidelines how to run your models in OpenCV deep learning module
|
||||
using Halide language backend. Halide is an open-source project that let us
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
# How to schedule your network for Halide backend {#tutorial_dnn_halide_scheduling}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_halide}
|
||||
@next_tutorial{tutorial_dnn_android}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.3 |
|
||||
|
||||
## Introduction
|
||||
Halide code is the same for every device we use. But for achieving the satisfied
|
||||
efficiency we should schedule computations properly. In this tutorial we describe
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
# How to run deep networks in browser {#tutorial_dnn_javascript}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_yolo}
|
||||
@next_tutorial{tutorial_dnn_custom_layers}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.3.1 |
|
||||
|
||||
## Introduction
|
||||
This tutorial will show us how to run deep learning models using OpenCV.js right
|
||||
in a browser. Tutorial refers a sample of face detection and face recognition
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 61 KiB |
+220
@@ -0,0 +1,220 @@
|
||||
# Conversion of PyTorch Classification Models and Launch with OpenCV C++ {#pytorch_cls_c_tutorial_dnn_conversion}
|
||||
|
||||
@prev_tutorial{pytorch_cls_tutorial_dnn_conversion}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Anastasia Murzova |
|
||||
| Compatibility | OpenCV >= 4.5 |
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* convert PyTorch classification models into ONNX format
|
||||
* run converted PyTorch model with OpenCV C/C++ API
|
||||
* provide model inference
|
||||
|
||||
We will explore the above-listed points by the example of ResNet-50 architecture.
|
||||
|
||||
## Introduction
|
||||
Let's briefly view the key concepts involved in the pipeline of PyTorch models transition with OpenCV API. The initial step in conversion of PyTorch models into cv::dnn::Net
|
||||
is model transferring into [ONNX](https://onnx.ai/about.html) format. ONNX aims at the interchangeability of the neural networks between various frameworks. There is a built-in function in PyTorch for ONNX conversion: [``torch.onnx.export``](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export).
|
||||
Further the obtained ``.onnx`` model is passed into cv::dnn::readNetFromONNX or cv::dnn::readNet.
|
||||
|
||||
## Requirements
|
||||
To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:
|
||||
|
||||
```console
|
||||
virtualenv -p /usr/bin/python3.7 <env_dir_path>
|
||||
source <env_dir_path>/bin/activate
|
||||
```
|
||||
|
||||
For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup.
|
||||
|
||||
Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies.
|
||||
The below line initiates requirements installation into the previously activated virtual environment:
|
||||
|
||||
```console
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a classification model conversion pipeline
|
||||
2. provide the inference, process prediction results
|
||||
|
||||
### Model Conversion Pipeline
|
||||
The code in this subchapter is located in the ``samples/dnn/dnn_model_runner`` module and can be executed with the line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_resnet50_onnx
|
||||
```
|
||||
|
||||
The following code contains the description of the below-listed steps:
|
||||
1. instantiate PyTorch model
|
||||
2. convert PyTorch model into ``.onnx``
|
||||
|
||||
```python
|
||||
# initialize PyTorch ResNet-50 model
|
||||
original_model = models.resnet50(pretrained=True)
|
||||
|
||||
# get the path to the converted into ONNX PyTorch model
|
||||
full_model_path = get_pytorch_onnx_model(original_model)
|
||||
print("PyTorch ResNet-50 model was successfully converted: ", full_model_path)
|
||||
```
|
||||
|
||||
``get_pytorch_onnx_model(original_model)`` function is based on ``torch.onnx.export(...)`` call:
|
||||
|
||||
```python
|
||||
# define the directory for further converted model save
|
||||
onnx_model_path = "models"
|
||||
# define the name of further converted model
|
||||
onnx_model_name = "resnet50.onnx"
|
||||
|
||||
# create directory for further converted model
|
||||
os.makedirs(onnx_model_path, exist_ok=True)
|
||||
|
||||
# get full path to the converted model
|
||||
full_model_path = os.path.join(onnx_model_path, onnx_model_name)
|
||||
|
||||
# generate model input
|
||||
generated_input = Variable(
|
||||
torch.randn(1, 3, 224, 224)
|
||||
)
|
||||
|
||||
# model export into ONNX format
|
||||
torch.onnx.export(
|
||||
original_model,
|
||||
generated_input,
|
||||
full_model_path,
|
||||
verbose=True,
|
||||
input_names=["input"],
|
||||
output_names=["output"],
|
||||
opset_version=11
|
||||
)
|
||||
```
|
||||
|
||||
After the successful execution of the above code we will get the following output:
|
||||
|
||||
```console
|
||||
PyTorch ResNet-50 model was successfully converted: models/resnet50.onnx
|
||||
```
|
||||
|
||||
The proposed in ``dnn/samples`` module ``dnn_model_runner`` allows us to reproduce the above conversion steps for the following PyTorch classification models:
|
||||
* alexnet
|
||||
* vgg11
|
||||
* vgg13
|
||||
* vgg16
|
||||
* vgg19
|
||||
* resnet18
|
||||
* resnet34
|
||||
* resnet50
|
||||
* resnet101
|
||||
* resnet152
|
||||
* squeezenet1_0
|
||||
* squeezenet1_1
|
||||
* resnext50_32x4d
|
||||
* resnext101_32x8d
|
||||
* wide_resnet50_2
|
||||
* wide_resnet101_2
|
||||
|
||||
To obtain the converted model, the following line should be executed:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name <pytorch_cls_model_name> --evaluate False
|
||||
```
|
||||
|
||||
For the ResNet-50 case the below line should be run:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50 --evaluate False
|
||||
```
|
||||
|
||||
The default root directory for the converted model storage is defined in module ``CommonConfig``:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class CommonConfig:
|
||||
output_data_root_dir: str = "dnn_model_runner/dnn_conversion"
|
||||
```
|
||||
|
||||
Thus, the converted ResNet-50 will be saved in ``dnn_model_runner/dnn_conversion/models``.
|
||||
|
||||
### Inference Pipeline
|
||||
Now we can use ```models/resnet50.onnx``` for the inference pipeline using OpenCV C/C++ API. The implemented pipeline can be found in [samples/dnn/classification.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/classification.cpp).
|
||||
After the build of samples (``BUILD_EXAMPLES`` flag value should be ``ON``), the appropriate ``example_dnn_classification`` executable file will be provided.
|
||||
|
||||
To provide model inference we will use the below [squirrel photo](https://www.pexels.com/photo/brown-squirrel-eating-1564292) (under [CC0](https://www.pexels.com/terms-of-service/) license) corresponding to ImageNet class ID 335:
|
||||
```console
|
||||
fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
```
|
||||
|
||||
|
||||
|
||||
For the label decoding of the obtained prediction, we also need ``imagenet_classes.txt`` file, which contains the full list of the ImageNet classes.
|
||||
|
||||
In this tutorial we will run the inference process for the converted PyTorch ResNet-50 model from the build (``samples/build``) directory:
|
||||
|
||||
```
|
||||
./dnn/example_dnn_classification --model=../dnn/models/resnet50.onnx --input=../data/squirrel_cls.jpg --width=224 --height=224 --rgb=true --scale="0.003921569" --mean="123.675 116.28 103.53" --std="0.229 0.224 0.225" --crop=true --initial_width=256 --initial_height=256 --classes=../data/dnn/classification_classes_ILSVRC2012.txt
|
||||
```
|
||||
|
||||
Let's explore ``classification.cpp`` key points step by step:
|
||||
|
||||
1. read the model with cv::dnn::readNet, initialize the network:
|
||||
|
||||
```cpp
|
||||
Net net = readNet(model, config, framework);
|
||||
```
|
||||
|
||||
The ``model`` parameter value is taken from ``--model`` key. In our case, it is ``resnet50.onnx``.
|
||||
|
||||
* preprocess input image:
|
||||
|
||||
```cpp
|
||||
if (rszWidth != 0 && rszHeight != 0)
|
||||
{
|
||||
resize(frame, frame, Size(rszWidth, rszHeight));
|
||||
}
|
||||
|
||||
// Create a 4D blob from a frame
|
||||
blobFromImage(frame, blob, scale, Size(inpWidth, inpHeight), mean, swapRB, crop);
|
||||
|
||||
// Check std values.
|
||||
if (std.val[0] != 0.0 && std.val[1] != 0.0 && std.val[2] != 0.0)
|
||||
{
|
||||
// Divide blob by std.
|
||||
divide(blob, std, blob);
|
||||
}
|
||||
```
|
||||
|
||||
In this step we use cv::dnn::blobFromImage function to prepare model input.
|
||||
We set ``Size(rszWidth, rszHeight)`` with ``--initial_width=256 --initial_height=256`` for the initial image resize as it's described in [PyTorch ResNet inference pipeline](https://pytorch.org/hub/pytorch_vision_resnet/).
|
||||
|
||||
It should be noted that firstly in cv::dnn::blobFromImage mean value is subtracted and only then pixel values are multiplied by scale.
|
||||
Thus, we use ``--mean="123.675 116.28 103.53"``, which is equivalent to ``[0.485, 0.456, 0.406]`` multiplied by ``255.0`` to reproduce the original image preprocessing order for PyTorch classification models:
|
||||
|
||||
```python
|
||||
img /= 255.0
|
||||
img -= [0.485, 0.456, 0.406]
|
||||
img /= [0.229, 0.224, 0.225]
|
||||
```
|
||||
|
||||
* make forward pass:
|
||||
|
||||
```cpp
|
||||
net.setInput(blob);
|
||||
Mat prob = net.forward();
|
||||
```
|
||||
|
||||
* process the prediction:
|
||||
|
||||
```cpp
|
||||
Point classIdPoint;
|
||||
double confidence;
|
||||
minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
|
||||
int classId = classIdPoint.x;
|
||||
```
|
||||
|
||||
Here we choose the most likely object class. The ``classId`` result for our case is 335 - fox squirrel, eastern fox squirrel, Sciurus niger:
|
||||
|
||||
|
||||
+362
@@ -0,0 +1,362 @@
|
||||
# Conversion of PyTorch Classification Models and Launch with OpenCV Python {#pytorch_cls_tutorial_dnn_conversion}
|
||||
|
||||
@prev_tutorial{tutorial_dnn_OCR}
|
||||
@next_tutorial{pytorch_cls_c_tutorial_dnn_conversion}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Anastasia Murzova |
|
||||
| Compatibility | OpenCV >= 4.5 |
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* convert PyTorch classification models into ONNX format
|
||||
* run converted PyTorch model with OpenCV Python API
|
||||
* obtain an evaluation of the PyTorch and OpenCV DNN models.
|
||||
|
||||
We will explore the above-listed points by the example of the ResNet-50 architecture.
|
||||
|
||||
## Introduction
|
||||
Let's briefly view the key concepts involved in the pipeline of PyTorch models transition with OpenCV API. The initial step in conversion of PyTorch models into cv.dnn.Net
|
||||
is model transferring into [ONNX](https://onnx.ai/about.html) format. ONNX aims at the interchangeability of the neural networks between various frameworks. There is a built-in function in PyTorch for ONNX conversion: [``torch.onnx.export``](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export).
|
||||
Further the obtained ``.onnx`` model is passed into cv.dnn.readNetFromONNX.
|
||||
|
||||
## Requirements
|
||||
To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:
|
||||
|
||||
```console
|
||||
virtualenv -p /usr/bin/python3.7 <env_dir_path>
|
||||
source <env_dir_path>/bin/activate
|
||||
```
|
||||
|
||||
For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup.
|
||||
|
||||
Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies.
|
||||
The below line initiates requirements installation into the previously activated virtual environment:
|
||||
|
||||
```console
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a classification model conversion pipeline and provide the inference
|
||||
2. evaluate and test classification models
|
||||
|
||||
If you'd like merely to run evaluation or test model pipelines, the "Model Conversion Pipeline" part can be skipped.
|
||||
|
||||
### Model Conversion Pipeline
|
||||
The code in this subchapter is located in the ``dnn_model_runner`` module and can be executed with the line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_resnet50
|
||||
```
|
||||
|
||||
The following code contains the description of the below-listed steps:
|
||||
1. instantiate PyTorch model
|
||||
2. convert PyTorch model into ``.onnx``
|
||||
3. read the transferred network with OpenCV API
|
||||
4. prepare input data
|
||||
5. provide inference
|
||||
|
||||
```python
|
||||
# initialize PyTorch ResNet-50 model
|
||||
original_model = models.resnet50(pretrained=True)
|
||||
|
||||
# get the path to the converted into ONNX PyTorch model
|
||||
full_model_path = get_pytorch_onnx_model(original_model)
|
||||
|
||||
# read converted .onnx model with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromONNX(full_model_path)
|
||||
print("OpenCV model was successfully read. Layer IDs: \n", opencv_net.getLayerNames())
|
||||
|
||||
# get preprocessed image
|
||||
input_img = get_preprocessed_img("../data/squirrel_cls.jpg")
|
||||
|
||||
# get ImageNet labels
|
||||
imagenet_labels = get_imagenet_labels("../data/dnn/classification_classes_ILSVRC2012.txt")
|
||||
|
||||
# obtain OpenCV DNN predictions
|
||||
get_opencv_dnn_prediction(opencv_net, input_img, imagenet_labels)
|
||||
|
||||
# obtain original PyTorch ResNet50 predictions
|
||||
get_pytorch_dnn_prediction(original_model, input_img, imagenet_labels)
|
||||
```
|
||||
|
||||
To provide model inference we will use the below [squirrel photo](https://www.pexels.com/photo/brown-squirrel-eating-1564292) (under [CC0](https://www.pexels.com/terms-of-service/) license) corresponding to ImageNet class ID 335:
|
||||
```console
|
||||
fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
```
|
||||
|
||||
|
||||
|
||||
For the label decoding of the obtained prediction, we also need ``imagenet_classes.txt`` file, which contains the full list of the ImageNet classes.
|
||||
|
||||
Let's go deeper into each step by the example of pretrained PyTorch ResNet-50:
|
||||
* instantiate PyTorch ResNet-50 model:
|
||||
|
||||
```python
|
||||
# initialize PyTorch ResNet-50 model
|
||||
original_model = models.resnet50(pretrained=True)
|
||||
```
|
||||
|
||||
* convert PyTorch model into ONNX:
|
||||
|
||||
```python
|
||||
# define the directory for further converted model save
|
||||
onnx_model_path = "models"
|
||||
# define the name of further converted model
|
||||
onnx_model_name = "resnet50.onnx"
|
||||
|
||||
# create directory for further converted model
|
||||
os.makedirs(onnx_model_path, exist_ok=True)
|
||||
|
||||
# get full path to the converted model
|
||||
full_model_path = os.path.join(onnx_model_path, onnx_model_name)
|
||||
|
||||
# generate model input
|
||||
generated_input = Variable(
|
||||
torch.randn(1, 3, 224, 224)
|
||||
)
|
||||
|
||||
# model export into ONNX format
|
||||
torch.onnx.export(
|
||||
original_model,
|
||||
generated_input,
|
||||
full_model_path,
|
||||
verbose=True,
|
||||
input_names=["input"],
|
||||
output_names=["output"],
|
||||
opset_version=11
|
||||
)
|
||||
```
|
||||
|
||||
After the successful execution of the above code, we will get ``models/resnet50.onnx``.
|
||||
|
||||
* read the transferred network with cv.dnn.readNetFromONNX passing the obtained in the previous step ONNX model into it:
|
||||
|
||||
```python
|
||||
# read converted .onnx model with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromONNX(full_model_path)
|
||||
```
|
||||
|
||||
* prepare input data:
|
||||
|
||||
```python
|
||||
# read the image
|
||||
input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)
|
||||
input_img = input_img.astype(np.float32)
|
||||
|
||||
input_img = cv2.resize(input_img, (256, 256))
|
||||
|
||||
# define preprocess parameters
|
||||
mean = np.array([0.485, 0.456, 0.406]) * 255.0
|
||||
scale = 1 / 255.0
|
||||
std = [0.229, 0.224, 0.225]
|
||||
|
||||
# prepare input blob to fit the model input:
|
||||
# 1. subtract mean
|
||||
# 2. scale to set pixel values from 0 to 1
|
||||
input_blob = cv2.dnn.blobFromImage(
|
||||
image=input_img,
|
||||
scalefactor=scale,
|
||||
size=(224, 224), # img target size
|
||||
mean=mean,
|
||||
swapRB=True, # BGR -> RGB
|
||||
crop=True # center crop
|
||||
)
|
||||
# 3. divide by std
|
||||
input_blob[0] /= np.asarray(std, dtype=np.float32).reshape(3, 1, 1)
|
||||
```
|
||||
|
||||
In this step we read the image and prepare model input with cv.dnn.blobFromImage function, which returns 4-dimensional blob.
|
||||
It should be noted that firstly in cv.dnn.blobFromImage mean value is subtracted and only then pixel values are multiplied by scale. Thus, ``mean`` is multiplied by ``255.0`` to reproduce the original image preprocessing order:
|
||||
|
||||
```python
|
||||
img /= 255.0
|
||||
img -= [0.485, 0.456, 0.406]
|
||||
img /= [0.229, 0.224, 0.225]
|
||||
```
|
||||
|
||||
* OpenCV cv.dnn.Net inference:
|
||||
|
||||
```python
|
||||
# set OpenCV DNN input
|
||||
opencv_net.setInput(preproc_img)
|
||||
|
||||
# OpenCV DNN inference
|
||||
out = opencv_net.forward()
|
||||
print("OpenCV DNN prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get the predicted class ID
|
||||
imagenet_class_id = np.argmax(out)
|
||||
|
||||
# get confidence
|
||||
confidence = out[0][imagenet_class_id]
|
||||
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
|
||||
print("* confidence: {:.4f}".format(confidence))
|
||||
```
|
||||
|
||||
After the above code execution we will get the following output:
|
||||
|
||||
```console
|
||||
OpenCV DNN prediction:
|
||||
* shape: (1, 1000)
|
||||
* class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
* confidence: 14.8308
|
||||
```
|
||||
|
||||
* PyTorch ResNet-50 model inference:
|
||||
|
||||
```python
|
||||
original_net.eval()
|
||||
preproc_img = torch.FloatTensor(preproc_img)
|
||||
|
||||
# inference
|
||||
out = original_net(preproc_img)
|
||||
print("\nPyTorch model prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get the predicted class ID
|
||||
imagenet_class_id = torch.argmax(out, axis=1).item()
|
||||
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
|
||||
|
||||
# get confidence
|
||||
confidence = out[0][imagenet_class_id]
|
||||
print("* confidence: {:.4f}".format(confidence.item()))
|
||||
```
|
||||
|
||||
After the above code launching we will get the following output:
|
||||
|
||||
```console
|
||||
PyTorch model prediction:
|
||||
* shape: torch.Size([1, 1000])
|
||||
* class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
* confidence: 14.8308
|
||||
```
|
||||
|
||||
The inference results of the original ResNet-50 model and cv.dnn.Net are equal. For the extended evaluation of the models we can use ``py_to_py_cls`` of the ``dnn_model_runner`` module. This module part will be described in the next subchapter.
|
||||
|
||||
### Evaluation of the Models
|
||||
|
||||
The proposed in ``samples/dnn`` ``dnn_model_runner`` module allows to run the full evaluation pipeline on the ImageNet dataset and test execution for the following PyTorch classification models:
|
||||
* alexnet
|
||||
* vgg11
|
||||
* vgg13
|
||||
* vgg16
|
||||
* vgg19
|
||||
* resnet18
|
||||
* resnet34
|
||||
* resnet50
|
||||
* resnet101
|
||||
* resnet152
|
||||
* squeezenet1_0
|
||||
* squeezenet1_1
|
||||
* resnext50_32x4d
|
||||
* resnext101_32x8d
|
||||
* wide_resnet50_2
|
||||
* wide_resnet101_2
|
||||
|
||||
This list can be also extended with further appropriate evaluation pipeline configuration.
|
||||
|
||||
#### Evaluation Mode
|
||||
|
||||
The below line represents running of the module in the evaluation mode:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name <pytorch_cls_model_name>
|
||||
```
|
||||
|
||||
Chosen from the list classification model will be read into OpenCV cv.dnn.Net object. Evaluation results of PyTorch and OpenCV models (accuracy, inference time, L1) will be written into the log file. Inference time values will be also depicted in a chart to generalize the obtained model information.
|
||||
|
||||
Necessary evaluation configurations are defined in the [test_config.py](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) and can be modified in accordance with actual paths of data location:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestClsConfig:
|
||||
batch_size: int = 50
|
||||
frame_size: int = 224
|
||||
img_root_dir: str = "./ILSVRC2012_img_val"
|
||||
# location of image-class matching
|
||||
img_cls_file: str = "./val.txt"
|
||||
bgr_to_rgb: bool = True
|
||||
```
|
||||
|
||||
To initiate the evaluation of the PyTorch ResNet-50, run the following line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50
|
||||
```
|
||||
|
||||
After script launch, the log file with evaluation data will be generated in ``dnn_model_runner/dnn_conversion/logs``:
|
||||
|
||||
```console
|
||||
The model PyTorch resnet50 was successfully obtained and converted to OpenCV DNN resnet50
|
||||
===== Running evaluation of the model with the following params:
|
||||
* val data location: ./ILSVRC2012_img_val
|
||||
* log file location: dnn_model_runner/dnn_conversion/logs/PyTorch_resnet50_log.txt
|
||||
```
|
||||
|
||||
#### Test Mode
|
||||
|
||||
The below line represents running of the module in the test mode, namely it provides the steps for the model inference:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name <pytorch_cls_model_name> --test True --default_img_preprocess <True/False> --evaluate False
|
||||
```
|
||||
|
||||
Here ``default_img_preprocess`` key defines whether you'd like to parametrize the model test process with some particular values or use the default values, for example, ``scale``, ``mean`` or ``std``.
|
||||
|
||||
Test configuration is represented in [test_config.py](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) ``TestClsModuleConfig`` class:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestClsModuleConfig:
|
||||
cls_test_data_dir: str = "../data"
|
||||
test_module_name: str = "classification"
|
||||
test_module_path: str = "classification.py"
|
||||
input_img: str = os.path.join(cls_test_data_dir, "squirrel_cls.jpg")
|
||||
model: str = ""
|
||||
|
||||
frame_height: str = str(TestClsConfig.frame_size)
|
||||
frame_width: str = str(TestClsConfig.frame_size)
|
||||
scale: str = "1.0"
|
||||
mean: List[str] = field(default_factory=lambda: ["0.0", "0.0", "0.0"])
|
||||
std: List[str] = field(default_factory=list)
|
||||
crop: str = "False"
|
||||
rgb: str = "True"
|
||||
rsz_height: str = ""
|
||||
rsz_width: str = ""
|
||||
classes: str = os.path.join(cls_test_data_dir, "dnn", "classification_classes_ILSVRC2012.txt")
|
||||
```
|
||||
|
||||
The default image preprocessing options are defined in [default_preprocess_config.py](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/default_preprocess_config.py). For instance:
|
||||
|
||||
```python
|
||||
BASE_IMG_SCALE_FACTOR = 1 / 255.0
|
||||
PYTORCH_RSZ_HEIGHT = 256
|
||||
PYTORCH_RSZ_WIDTH = 256
|
||||
|
||||
pytorch_resize_input_blob = {
|
||||
"mean": ["123.675", "116.28", "103.53"],
|
||||
"scale": str(BASE_IMG_SCALE_FACTOR),
|
||||
"std": ["0.229", "0.224", "0.225"],
|
||||
"crop": "True",
|
||||
"rgb": "True",
|
||||
"rsz_height": str(PYTORCH_RSZ_HEIGHT),
|
||||
"rsz_width": str(PYTORCH_RSZ_WIDTH)
|
||||
}
|
||||
```
|
||||
|
||||
The basis of the model testing is represented in [samples/dnn/classification.py](https://github.com/opencv/opencv/blob/master/samples/dnn/classification.py). ``classification.py`` can be executed autonomously with provided converted model in ``--input`` and populated parameters for cv.dnn.blobFromImage.
|
||||
|
||||
To reproduce from scratch the described in "Model Conversion Pipeline" OpenCV steps with ``dnn_model_runner`` execute the below line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50 --test True --default_img_preprocess True --evaluate False
|
||||
```
|
||||
|
||||
The network prediction is depicted in the top left corner of the output window:
|
||||
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
# Conversion of TensorFlow Classification Models and Launch with OpenCV Python {#tf_cls_tutorial_dnn_conversion}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Anastasia Murzova |
|
||||
| Compatibility | OpenCV >= 4.5 |
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* obtain frozen graphs of TensorFlow (TF) classification models
|
||||
* run converted TensorFlow model with OpenCV Python API
|
||||
* obtain an evaluation of the TensorFlow and OpenCV DNN models
|
||||
|
||||
We will explore the above-listed points by the example of MobileNet architecture.
|
||||
|
||||
## Introduction
|
||||
Let's briefly view the key concepts involved in the pipeline of TensorFlow models transition with OpenCV API. The initial step in conversion of TensorFlow models into cv.dnn.Net
|
||||
is obtaining the frozen TF model graph. Frozen graph defines the combination of the model graph structure with kept values of the required variables, for example, weights. Usually the frozen graph is saved in [protobuf](https://en.wikipedia.org/wiki/Protocol_Buffers) (```.pb```) files.
|
||||
After the model ``.pb`` file was generated it can be read with cv.dnn.readNetFromTensorflow function.
|
||||
|
||||
## Requirements
|
||||
To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:
|
||||
|
||||
```console
|
||||
virtualenv -p /usr/bin/python3.7 <env_dir_path>
|
||||
source <env_dir_path>/bin/activate
|
||||
```
|
||||
|
||||
For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup.
|
||||
|
||||
Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies.
|
||||
The below line initiates requirements installation into the previously activated virtual environment:
|
||||
|
||||
```console
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a TF classification model conversion pipeline and provide the inference
|
||||
2. evaluate and test TF classification models
|
||||
|
||||
If you'd like merely to run evaluation or test model pipelines, the "Model Conversion Pipeline" tutorial part can be skipped.
|
||||
|
||||
### Model Conversion Pipeline
|
||||
The code in this subchapter is located in the ``dnn_model_runner`` module and can be executed with the line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_mobilenet
|
||||
```
|
||||
|
||||
The following code contains the description of the below-listed steps:
|
||||
1. instantiate TF model
|
||||
2. create TF frozen graph
|
||||
3. read TF frozen graph with OpenCV API
|
||||
4. prepare input data
|
||||
5. provide inference
|
||||
|
||||
```python
|
||||
# initialize TF MobileNet model
|
||||
original_tf_model = MobileNet(
|
||||
include_top=True,
|
||||
weights="imagenet"
|
||||
)
|
||||
|
||||
# get TF frozen graph path
|
||||
full_pb_path = get_tf_model_proto(original_tf_model)
|
||||
|
||||
# read frozen graph with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromTensorflow(full_pb_path)
|
||||
print("OpenCV model was successfully read. Model layers: \n", opencv_net.getLayerNames())
|
||||
|
||||
# get preprocessed image
|
||||
input_img = get_preprocessed_img("../data/squirrel_cls.jpg")
|
||||
|
||||
# get ImageNet labels
|
||||
imagenet_labels = get_imagenet_labels("../data/dnn/classification_classes_ILSVRC2012.txt")
|
||||
|
||||
# obtain OpenCV DNN predictions
|
||||
get_opencv_dnn_prediction(opencv_net, input_img, imagenet_labels)
|
||||
|
||||
# obtain TF model predictions
|
||||
get_tf_dnn_prediction(original_tf_model, input_img, imagenet_labels)
|
||||
```
|
||||
|
||||
To provide model inference we will use the below [squirrel photo](https://www.pexels.com/photo/brown-squirrel-eating-1564292) (under [CC0](https://www.pexels.com/terms-of-service/) license) corresponding to ImageNet class ID 335:
|
||||
```console
|
||||
fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
```
|
||||
|
||||
|
||||
|
||||
For the label decoding of the obtained prediction, we also need ``imagenet_classes.txt`` file, which contains the full list of the ImageNet classes.
|
||||
|
||||
Let's go deeper into each step by the example of pretrained TF MobileNet:
|
||||
* instantiate TF model:
|
||||
|
||||
```python
|
||||
# initialize TF MobileNet model
|
||||
original_tf_model = MobileNet(
|
||||
include_top=True,
|
||||
weights="imagenet"
|
||||
)
|
||||
```
|
||||
|
||||
* create TF frozen graph
|
||||
|
||||
```python
|
||||
# define the directory for .pb model
|
||||
pb_model_path = "models"
|
||||
|
||||
# define the name of .pb model
|
||||
pb_model_name = "mobilenet.pb"
|
||||
|
||||
# create directory for further converted model
|
||||
os.makedirs(pb_model_path, exist_ok=True)
|
||||
|
||||
# get model TF graph
|
||||
tf_model_graph = tf.function(lambda x: tf_model(x))
|
||||
|
||||
# get concrete function
|
||||
tf_model_graph = tf_model_graph.get_concrete_function(
|
||||
tf.TensorSpec(tf_model.inputs[0].shape, tf_model.inputs[0].dtype))
|
||||
|
||||
# obtain frozen concrete function
|
||||
frozen_tf_func = convert_variables_to_constants_v2(tf_model_graph)
|
||||
# get frozen graph
|
||||
frozen_tf_func.graph.as_graph_def()
|
||||
|
||||
# save full tf model
|
||||
tf.io.write_graph(graph_or_graph_def=frozen_tf_func.graph,
|
||||
logdir=pb_model_path,
|
||||
name=pb_model_name,
|
||||
as_text=False)
|
||||
```
|
||||
|
||||
After the successful execution of the above code, we will get a frozen graph in ``models/mobilenet.pb``.
|
||||
|
||||
* read TF frozen graph with with cv.dnn.readNetFromTensorflow passing the obtained in the previous step ``mobilenet.pb`` into it:
|
||||
|
||||
```python
|
||||
# get TF frozen graph path
|
||||
full_pb_path = get_tf_model_proto(original_tf_model)
|
||||
```
|
||||
|
||||
* prepare input data with cv2.dnn.blobFromImage function:
|
||||
|
||||
```python
|
||||
# read the image
|
||||
input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)
|
||||
input_img = input_img.astype(np.float32)
|
||||
|
||||
# define preprocess parameters
|
||||
mean = np.array([1.0, 1.0, 1.0]) * 127.5
|
||||
scale = 1 / 127.5
|
||||
|
||||
# prepare input blob to fit the model input:
|
||||
# 1. subtract mean
|
||||
# 2. scale to set pixel values from 0 to 1
|
||||
input_blob = cv2.dnn.blobFromImage(
|
||||
image=input_img,
|
||||
scalefactor=scale,
|
||||
size=(224, 224), # img target size
|
||||
mean=mean,
|
||||
swapRB=True, # BGR -> RGB
|
||||
crop=True # center crop
|
||||
)
|
||||
print("Input blob shape: {}\n".format(input_blob.shape))
|
||||
```
|
||||
|
||||
Please, pay attention at the preprocessing order in the cv2.dnn.blobFromImage function. Firstly, the mean value is subtracted and only then pixel values are multiplied by the defined scale.
|
||||
Therefore, to reproduce the image preprocessing pipeline from the TF [``mobilenet.preprocess_input``](https://github.com/tensorflow/tensorflow/blob/02032fb477e9417197132648ec81e75beee9063a/tensorflow/python/keras/applications/mobilenet.py#L443-L445) function, we multiply ``mean`` by ``127.5``.
|
||||
|
||||
As a result, 4-dimensional ``input_blob`` was obtained:
|
||||
|
||||
``Input blob shape: (1, 3, 224, 224)``
|
||||
|
||||
* provide OpenCV cv.dnn.Net inference:
|
||||
|
||||
```python
|
||||
# set OpenCV DNN input
|
||||
opencv_net.setInput(preproc_img)
|
||||
|
||||
# OpenCV DNN inference
|
||||
out = opencv_net.forward()
|
||||
print("OpenCV DNN prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get the predicted class ID
|
||||
imagenet_class_id = np.argmax(out)
|
||||
|
||||
# get confidence
|
||||
confidence = out[0][imagenet_class_id]
|
||||
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
|
||||
print("* confidence: {:.4f}\n".format(confidence))
|
||||
```
|
||||
|
||||
After the above code execution we will get the following output:
|
||||
|
||||
```console
|
||||
OpenCV DNN prediction:
|
||||
* shape: (1, 1000)
|
||||
* class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
* confidence: 0.9525
|
||||
```
|
||||
|
||||
* provide TF MobileNet inference:
|
||||
|
||||
```python
|
||||
# inference
|
||||
preproc_img = preproc_img.transpose(0, 2, 3, 1)
|
||||
print("TF input blob shape: {}\n".format(preproc_img.shape))
|
||||
|
||||
out = original_net(preproc_img)
|
||||
|
||||
print("\nTensorFlow model prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get the predicted class ID
|
||||
imagenet_class_id = np.argmax(out)
|
||||
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
|
||||
|
||||
# get confidence
|
||||
confidence = out[0][imagenet_class_id]
|
||||
print("* confidence: {:.4f}".format(confidence))
|
||||
```
|
||||
|
||||
To fit TF model input, ``input_blob`` was transposed:
|
||||
|
||||
```console
|
||||
TF input blob shape: (1, 224, 224, 3)
|
||||
```
|
||||
|
||||
TF inference results are the following:
|
||||
|
||||
```console
|
||||
TensorFlow model prediction:
|
||||
* shape: (1, 1000)
|
||||
* class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
|
||||
* confidence: 0.9525
|
||||
```
|
||||
|
||||
As it can be seen from the experiments OpenCV and TF inference results are equal.
|
||||
|
||||
### Evaluation of the Models
|
||||
|
||||
The proposed in ``dnn/samples`` ``dnn_model_runner`` module allows to run the full evaluation pipeline on the ImageNet dataset and test execution for the following TensorFlow classification models:
|
||||
* vgg16
|
||||
* vgg19
|
||||
* resnet50
|
||||
* resnet101
|
||||
* resnet152
|
||||
* densenet121
|
||||
* densenet169
|
||||
* densenet201
|
||||
* inceptionresnetv2
|
||||
* inceptionv3
|
||||
* mobilenet
|
||||
* mobilenetv2
|
||||
* nasnetlarge
|
||||
* nasnetmobile
|
||||
* xception
|
||||
|
||||
This list can be also extended with further appropriate evaluation pipeline configuration.
|
||||
|
||||
#### Evaluation Mode
|
||||
|
||||
To below line represents running of the module in the evaluation mode:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name <tf_cls_model_name>
|
||||
```
|
||||
|
||||
Chosen from the list classification model will be read into OpenCV ``cv.dnn_Net`` object. Evaluation results of TF and OpenCV models (accuracy, inference time, L1) will be written into the log file. Inference time values will be also depicted in a chart to generalize the obtained model information.
|
||||
|
||||
Necessary evaluation configurations are defined in the [test_config.py](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) and can be modified in accordance with actual paths of data location::
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestClsConfig:
|
||||
batch_size: int = 50
|
||||
frame_size: int = 224
|
||||
img_root_dir: str = "./ILSVRC2012_img_val"
|
||||
# location of image-class matching
|
||||
img_cls_file: str = "./val.txt"
|
||||
bgr_to_rgb: bool = True
|
||||
```
|
||||
|
||||
The values from ``TestClsConfig`` can be customized in accordance with chosen model.
|
||||
|
||||
To initiate the evaluation of the TensorFlow MobileNet, run the following line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name mobilenet
|
||||
```
|
||||
|
||||
After script launch, the log file with evaluation data will be generated in ``dnn_model_runner/dnn_conversion/logs``:
|
||||
|
||||
```console
|
||||
===== Running evaluation of the model with the following params:
|
||||
* val data location: ./ILSVRC2012_img_val
|
||||
* log file location: dnn_model_runner/dnn_conversion/logs/TF_mobilenet_log.txt
|
||||
```
|
||||
|
||||
#### Test Mode
|
||||
|
||||
The below line represents running of the module in the test mode, namely it provides the steps for the model inference:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name <tf_cls_model_name> --test True --default_img_preprocess <True/False> --evaluate False
|
||||
```
|
||||
|
||||
Here ``default_img_preprocess`` key defines whether you'd like to parametrize the model test process with some particular values or use the default values, for example, ``scale``, ``mean`` or ``std``.
|
||||
|
||||
Test configuration is represented in [test_config.py](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) ``TestClsModuleConfig`` class:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestClsModuleConfig:
|
||||
cls_test_data_dir: str = "../data"
|
||||
test_module_name: str = "classification"
|
||||
test_module_path: str = "classification.py"
|
||||
input_img: str = os.path.join(cls_test_data_dir, "squirrel_cls.jpg")
|
||||
model: str = ""
|
||||
|
||||
frame_height: str = str(TestClsConfig.frame_size)
|
||||
frame_width: str = str(TestClsConfig.frame_size)
|
||||
scale: str = "1.0"
|
||||
mean: List[str] = field(default_factory=lambda: ["0.0", "0.0", "0.0"])
|
||||
std: List[str] = field(default_factory=list)
|
||||
crop: str = "False"
|
||||
rgb: str = "True"
|
||||
rsz_height: str = ""
|
||||
rsz_width: str = ""
|
||||
classes: str = os.path.join(cls_test_data_dir, "dnn", "classification_classes_ILSVRC2012.txt")
|
||||
```
|
||||
|
||||
The default image preprocessing options are defined in ``default_preprocess_config.py``. For instance, for MobileNet:
|
||||
|
||||
```python
|
||||
tf_input_blob = {
|
||||
"mean": ["127.5", "127.5", "127.5"],
|
||||
"scale": str(1 / 127.5),
|
||||
"std": [],
|
||||
"crop": "True",
|
||||
"rgb": "True"
|
||||
}
|
||||
```
|
||||
|
||||
The basis of the model testing is represented in [samples/dnn/classification.py](https://github.com/opencv/opencv/blob/master/samples/dnn/classification.py). ``classification.py`` can be executed autonomously with provided converted model in ``--input`` and populated parameters for cv.dnn.blobFromImage.
|
||||
|
||||
To reproduce from scratch the described in "Model Conversion Pipeline" OpenCV steps with ``dnn_model_runner`` execute the below line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name mobilenet --test True --default_img_preprocess True --evaluate False
|
||||
```
|
||||
|
||||
The network prediction is depicted in the top left corner of the output window:
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 81 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
@@ -0,0 +1,140 @@
|
||||
# Conversion of TensorFlow Detection Models and Launch with OpenCV Python {#tf_det_tutorial_dnn_conversion}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Anastasia Murzova |
|
||||
| Compatibility | OpenCV >= 4.5 |
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* obtain frozen graphs of TensorFlow (TF) detection models
|
||||
* run converted TensorFlow model with OpenCV Python API
|
||||
|
||||
We will explore the above-listed points by the example of SSD MobileNetV1.
|
||||
|
||||
## Introduction
|
||||
Let's briefly view the key concepts involved in the pipeline of TensorFlow models transition with OpenCV API. The initial step in the conversion of TensorFlow models into cv.dnn.Net
|
||||
is obtaining the frozen TF model graph. A frozen graph defines the combination of the model graph structure with kept values of the required variables, for example, weights. The frozen graph is saved in [protobuf](https://en.wikipedia.org/wiki/Protocol_Buffers) (```.pb```) files.
|
||||
There are special functions for reading ``.pb`` graphs in OpenCV: cv.dnn.readNetFromTensorflow and cv.dnn.readNet.
|
||||
|
||||
## Requirements
|
||||
To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:
|
||||
|
||||
```console
|
||||
virtualenv -p /usr/bin/python3.7 <env_dir_path>
|
||||
source <env_dir_path>/bin/activate
|
||||
```
|
||||
|
||||
For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup.
|
||||
|
||||
Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies.
|
||||
The below line initiates requirements installation into the previously activated virtual environment:
|
||||
|
||||
```console
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a TF classification model conversion pipeline and provide the inference
|
||||
2. provide the inference, process prediction results
|
||||
|
||||
### Model Preparation
|
||||
The code in this subchapter is located in the ``samples/dnn/dnn_model_runner`` module and can be executed with the below line:
|
||||
|
||||
```console
|
||||
python -m dnn_model_runner.dnn_conversion.tf.detection.py_to_py_ssd_mobilenet
|
||||
```
|
||||
|
||||
The following code contains the steps of the TF SSD MobileNetV1 model retrieval:
|
||||
|
||||
```python
|
||||
tf_model_name = 'ssd_mobilenet_v1_coco_2017_11_17'
|
||||
graph_extraction_dir = "./"
|
||||
frozen_graph_path = extract_tf_frozen_graph(tf_model_name, graph_extraction_dir)
|
||||
print("Frozen graph path for {}: {}".format(tf_model_name, frozen_graph_path))
|
||||
```
|
||||
|
||||
In ``extract_tf_frozen_graph`` function we extract the provided in model archive ``frozen_inference_graph.pb`` for its further processing:
|
||||
|
||||
```python
|
||||
# define model archive name
|
||||
tf_model_tar = model_name + '.tar.gz'
|
||||
# define link to retrieve model archive
|
||||
model_link = DETECTION_MODELS_URL + tf_model_tar
|
||||
|
||||
tf_frozen_graph_name = 'frozen_inference_graph'
|
||||
|
||||
try:
|
||||
urllib.request.urlretrieve(model_link, tf_model_tar)
|
||||
except Exception:
|
||||
print("TF {} was not retrieved: {}".format(model_name, model_link))
|
||||
return
|
||||
|
||||
print("TF {} was retrieved.".format(model_name))
|
||||
|
||||
tf_model_tar = tarfile.open(tf_model_tar)
|
||||
frozen_graph_path = ""
|
||||
|
||||
for model_tar_elem in tf_model_tar.getmembers():
|
||||
if tf_frozen_graph_name in os.path.basename(model_tar_elem.name):
|
||||
tf_model_tar.extract(model_tar_elem, extracted_model_path)
|
||||
frozen_graph_path = os.path.join(extracted_model_path, model_tar_elem.name)
|
||||
break
|
||||
tf_model_tar.close()
|
||||
```
|
||||
|
||||
After the successful execution of the above code we will get the following output:
|
||||
|
||||
```console
|
||||
TF ssd_mobilenet_v1_coco_2017_11_17 was retrieved.
|
||||
Frozen graph path for ssd_mobilenet_v1_coco_2017_11_17: ./ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb
|
||||
```
|
||||
|
||||
To provide model inference we will use the below [double-decker bus photo](https://www.pexels.com/photo/bus-and-car-on-one-way-street-3626589/) (under [Pexels](https://www.pexels.com/license/) license):
|
||||
|
||||
|
||||
|
||||
To initiate the test process we need to provide an appropriate model configuration. We will use [``ssd_mobilenet_v1_coco.config``](https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssd_mobilenet_v1_coco.config) from [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection#tensorflow-object-detection-api).
|
||||
TensorFlow Object Detection API framework contains helpful mechanisms for object detection model manipulations.
|
||||
|
||||
We will use this configuration to provide a text graph representation. To generate ``.pbtxt`` we will use the corresponding [``samples/dnn/tf_text_graph_ssd.py``](https://github.com/opencv/opencv/blob/master/samples/dnn/tf_text_graph_ssd.py) script:
|
||||
|
||||
```console
|
||||
python tf_text_graph_ssd.py --input ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb --config ssd_mobilenet_v1_coco_2017_11_17/ssd_mobilenet_v1_coco.config --output ssd_mobilenet_v1_coco_2017_11_17.pbtxt
|
||||
```
|
||||
|
||||
After successful execution ``ssd_mobilenet_v1_coco_2017_11_17.pbtxt`` will be created.
|
||||
|
||||
Before we run ``object_detection.py``, let's have a look at the default values for the SSD MobileNetV1 test process configuration. They are located in [``models.yml``](https://github.com/opencv/opencv/blob/master/samples/dnn/models.yml):
|
||||
|
||||
```yml
|
||||
ssd_tf:
|
||||
model: "ssd_mobilenet_v1_coco_2017_11_17.pb"
|
||||
config: "ssd_mobilenet_v1_coco_2017_11_17.pbtxt"
|
||||
mean: [0, 0, 0]
|
||||
scale: 1.0
|
||||
width: 300
|
||||
height: 300
|
||||
rgb: true
|
||||
classes: "object_detection_classes_coco.txt"
|
||||
sample: "object_detection"
|
||||
```
|
||||
|
||||
To fetch these values we need to provide frozen graph ``ssd_mobilenet_v1_coco_2017_11_17.pb`` model and text graph ``ssd_mobilenet_v1_coco_2017_11_17.pbtxt``:
|
||||
|
||||
```console
|
||||
python object_detection.py ssd_tf --input ../data/pexels_double_decker_bus.jpg
|
||||
```
|
||||
|
||||
This line is equivalent to:
|
||||
|
||||
```console
|
||||
python object_detection.py --model ssd_mobilenet_v1_coco_2017_11_17.pb --config ssd_mobilenet_v1_coco_2017_11_17.pbtxt --input ../data/pexels_double_decker_bus.jpg --width 300 --height 300 --classes ../data/dnn/object_detection_classes_coco.txt
|
||||
```
|
||||
|
||||
The result is:
|
||||
|
||||
|
||||
|
||||
There are several helpful parameters, which can be also customized for result corrections: threshold (``--thr``) and non-maximum suppression (``--nms``) values.
|
||||
+332
@@ -0,0 +1,332 @@
|
||||
# Conversion of PyTorch Segmentation Models and Launch with OpenCV {#pytorch_segm_tutorial_dnn_conversion}
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* convert PyTorch segmentation models
|
||||
* run converted PyTorch model with OpenCV
|
||||
* obtain an evaluation of the PyTorch and OpenCV DNN models
|
||||
|
||||
We will explore the above-listed points by the example of the FCN ResNet-50 architecture.
|
||||
|
||||
## Introduction
|
||||
The key points involved in the transition pipeline of the [PyTorch classification](https://link_to_cls_tutorial) and segmentation models with OpenCV API are equal. The first step is model transferring into [ONNX](https://onnx.ai/about.html) format with PyTorch [``torch.onnx.export``](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export) built-in function.
|
||||
Further the obtained ``.onnx`` model is passed into cv.dnn.readNetFromONNX, which returns cv.dnn.Net object ready for DNN manipulations.
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a segmentation model conversion pipeline and provide the inference
|
||||
2. evaluate and test segmentation models
|
||||
|
||||
If you'd like merely to run evaluation or test model pipelines, the "Model Conversion Pipeline" part can be skipped.
|
||||
|
||||
### Model Conversion Pipeline
|
||||
The code in this subchapter is located in the ``dnn_model_runner`` module and can be executed with the line:
|
||||
|
||||
``
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.segmentation.py_to_py_fcnresnet50
|
||||
``
|
||||
|
||||
The following code contains the description of the below-listed steps:
|
||||
1. instantiate PyTorch model
|
||||
2. convert PyTorch model into ``.onnx``
|
||||
3. read the transferred network with OpenCV API
|
||||
4. prepare input data
|
||||
5. provide inference
|
||||
6. get colored masks from predictions
|
||||
7. visualize results
|
||||
|
||||
```python
|
||||
# initialize PyTorch FCN ResNet-50 model
|
||||
original_model = models.segmentation.fcn_resnet50(pretrained=True)
|
||||
|
||||
# get the path to the converted into ONNX PyTorch model
|
||||
full_model_path = get_pytorch_onnx_model(original_model)
|
||||
|
||||
# read converted .onnx model with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromONNX(full_model_path)
|
||||
print("OpenCV model was successfully read. Layer IDs: \n", opencv_net.getLayerNames())
|
||||
|
||||
# get preprocessed image
|
||||
img, input_img = get_processed_imgs("test_data/sem_segm/2007_000033.jpg")
|
||||
|
||||
# obtain OpenCV DNN predictions
|
||||
opencv_prediction = get_opencv_dnn_prediction(opencv_net, input_img)
|
||||
|
||||
# obtain original PyTorch ResNet50 predictions
|
||||
pytorch_prediction = get_pytorch_dnn_prediction(original_model, input_img)
|
||||
|
||||
pascal_voc_classes, pascal_voc_colors = read_colors_info("test_data/sem_segm/pascal-classes.txt")
|
||||
|
||||
# obtain colored segmentation masks
|
||||
opencv_colored_mask = get_colored_mask(img.shape, opencv_prediction, pascal_voc_colors)
|
||||
pytorch_colored_mask = get_colored_mask(img.shape, pytorch_prediction, pascal_voc_colors)
|
||||
|
||||
# obtain palette of PASCAL VOC colors
|
||||
color_legend = get_legend(pascal_voc_classes, pascal_voc_colors)
|
||||
|
||||
cv2.imshow('PyTorch Colored Mask', pytorch_colored_mask)
|
||||
cv2.imshow('OpenCV DNN Colored Mask', opencv_colored_mask)
|
||||
cv2.imshow('Color Legend', color_legend)
|
||||
|
||||
cv2.waitKey(0)
|
||||
```
|
||||
|
||||
To provide the model inference we will use the below picture from the [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/) validation dataset:
|
||||
|
||||
|
||||
|
||||
The target segmented result is:
|
||||
|
||||
|
||||
|
||||
For the PASCAL VOC colors decoding and its mapping with the predicted masks, we also need ``pascal-classes.txt`` file, which contains the full list of the PASCAL VOC classes and corresponding colors.
|
||||
|
||||
Let's go deeper into each code step by the example of pretrained PyTorch FCN ResNet-50:
|
||||
* instantiate PyTorch FCN ResNet-50 model:
|
||||
|
||||
```python
|
||||
# initialize PyTorch FCN ResNet-50 model
|
||||
original_model = models.segmentation.fcn_resnet50(pretrained=True)
|
||||
```
|
||||
|
||||
* convert PyTorch model into ONNX format:
|
||||
|
||||
```python
|
||||
# define the directory for further converted model save
|
||||
onnx_model_path = "models"
|
||||
# define the name of further converted model
|
||||
onnx_model_name = "fcnresnet50.onnx"
|
||||
|
||||
# create directory for further converted model
|
||||
os.makedirs(onnx_model_path, exist_ok=True)
|
||||
|
||||
# get full path to the converted model
|
||||
full_model_path = os.path.join(onnx_model_path, onnx_model_name)
|
||||
|
||||
# generate model input to build the graph
|
||||
generated_input = Variable(
|
||||
torch.randn(1, 3, 500, 500)
|
||||
)
|
||||
|
||||
# model export into ONNX format
|
||||
torch.onnx.export(
|
||||
original_model,
|
||||
generated_input,
|
||||
full_model_path,
|
||||
verbose=True,
|
||||
input_names=["input"],
|
||||
output_names=["output"],
|
||||
opset_version=11
|
||||
)
|
||||
```
|
||||
|
||||
The code from this step does not differ from the classification conversion case. Thus, after the successful execution of the above code, we will get ``models/fcnresnet50.onnx``.
|
||||
|
||||
* read the transferred network with cv.dnn.readNetFromONNX passing the obtained in the previous step ONNX model into it:
|
||||
|
||||
```python
|
||||
# read converted .onnx model with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromONNX(full_model_path)
|
||||
```
|
||||
|
||||
* prepare input data:
|
||||
|
||||
```python
|
||||
# read the image
|
||||
input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)
|
||||
input_img = input_img.astype(np.float32)
|
||||
|
||||
# target image sizes
|
||||
img_height = input_img.shape[0]
|
||||
img_width = input_img.shape[1]
|
||||
|
||||
# define preprocess parameters
|
||||
mean = np.array([0.485, 0.456, 0.406]) * 255.0
|
||||
scale = 1 / 255.0
|
||||
std = [0.229, 0.224, 0.225]
|
||||
|
||||
# prepare input blob to fit the model input:
|
||||
# 1. subtract mean
|
||||
# 2. scale to set pixel values from 0 to 1
|
||||
input_blob = cv2.dnn.blobFromImage(
|
||||
image=input_img,
|
||||
scalefactor=scale,
|
||||
size=(img_width, img_height), # img target size
|
||||
mean=mean,
|
||||
swapRB=True, # BGR -> RGB
|
||||
crop=False # center crop
|
||||
)
|
||||
# 3. divide by std
|
||||
input_blob[0] /= np.asarray(std, dtype=np.float32).reshape(3, 1, 1)
|
||||
```
|
||||
|
||||
In this step we read the image and prepare model input with cv2.dnn.blobFromImage function, which returns 4-dimensional blob.
|
||||
It should be noted that firstly in ``cv2.dnn.blobFromImage`` mean value is subtracted and only then pixel values are scaled. Thus, ``mean`` is multiplied by ``255.0`` to reproduce the original image preprocessing order:
|
||||
|
||||
```python
|
||||
img /= 255.0
|
||||
img -= [0.485, 0.456, 0.406]
|
||||
img /= [0.229, 0.224, 0.225]
|
||||
```
|
||||
|
||||
* OpenCV ``cv.dnn_Net`` inference:
|
||||
|
||||
```python
|
||||
# set OpenCV DNN input
|
||||
opencv_net.setInput(preproc_img)
|
||||
|
||||
# OpenCV DNN inference
|
||||
out = opencv_net.forward()
|
||||
print("OpenCV DNN segmentation prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get IDs of predicted classes
|
||||
out_predictions = np.argmax(out[0], axis=0)
|
||||
```
|
||||
|
||||
After the above code execution we will get the following output:
|
||||
|
||||
```
|
||||
OpenCV DNN segmentation prediction:
|
||||
* shape: (1, 21, 500, 500)
|
||||
```
|
||||
|
||||
Each prediction channel out of 21, where 21 represents the number of PASCAL VOC classes, contains probabilities, which indicate how likely the pixel corresponds to the PASCAL VOC class.
|
||||
|
||||
* PyTorch FCN ResNet-50 model inference:
|
||||
|
||||
```python
|
||||
original_net.eval()
|
||||
preproc_img = torch.FloatTensor(preproc_img)
|
||||
|
||||
with torch.no_grad():
|
||||
# obtaining unnormalized probabilities for each class
|
||||
out = original_net(preproc_img)['out']
|
||||
|
||||
print("\nPyTorch segmentation model prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get IDs of predicted classes
|
||||
out_predictions = out[0].argmax(dim=0)
|
||||
```
|
||||
|
||||
After the above code launching we will get the following output:
|
||||
|
||||
```
|
||||
PyTorch segmentation model prediction:
|
||||
* shape: torch.Size([1, 21, 366, 500])
|
||||
```
|
||||
|
||||
PyTorch prediction also contains probabilities corresponding to each class prediction.
|
||||
|
||||
* get colored masks from predictions:
|
||||
|
||||
```python
|
||||
# convert mask values into PASCAL VOC colors
|
||||
processed_mask = np.stack([colors[color_id] for color_id in segm_mask.flatten()])
|
||||
|
||||
# reshape mask into 3-channel image
|
||||
processed_mask = processed_mask.reshape(mask_height, mask_width, 3)
|
||||
processed_mask = cv2.resize(processed_mask, (img_width, img_height), interpolation=cv2.INTER_NEAREST).astype(
|
||||
np.uint8)
|
||||
|
||||
# convert colored mask from BGR to RGB for compatibility with PASCAL VOC colors
|
||||
processed_mask = cv2.cvtColor(processed_mask, cv2.COLOR_BGR2RGB)
|
||||
```
|
||||
|
||||
In this step we map the probabilities from segmentation masks with appropriate colors of the predicted classes. Let's have a look at the results:
|
||||
|
||||
|
||||
|
||||
For the extended evaluation of the models, we can use ``py_to_py_segm`` script of the ``dnn_model_runner`` module. This module part will be described in the next subchapter.
|
||||
|
||||
### Evaluation of the Models
|
||||
|
||||
The proposed in ``dnn/samples`` ``dnn_model_runner`` module allows to run the full evaluation pipeline on the PASCAL VOC dataset and test execution for the following PyTorch segmentation models:
|
||||
* FCN ResNet-50
|
||||
* FCN ResNet-101
|
||||
|
||||
This list can be also extended with further appropriate evaluation pipeline configuration.
|
||||
|
||||
#### Evaluation Mode
|
||||
|
||||
The below line represents running of the module in the evaluation mode:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.segmentation.py_to_py_segm --model_name <pytorch_segm_model_name>
|
||||
```
|
||||
|
||||
Chosen from the list segmentation model will be read into OpenCV ``cv.dnn_Net`` object. Evaluation results of PyTorch and OpenCV models (pixel accuracy, mean IoU, inference time) will be written into the log file. Inference time values will be also depicted in a chart to generalize the obtained model information.
|
||||
|
||||
Necessary evaluation configurations are defined in the [``test_config.py``](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py):
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestSegmConfig:
|
||||
frame_size: int = 500
|
||||
img_root_dir: str = "./VOC2012"
|
||||
img_dir: str = os.path.join(img_root_dir, "JPEGImages/")
|
||||
img_segm_gt_dir: str = os.path.join(img_root_dir, "SegmentationClass/")
|
||||
# reduced val: https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/data/pascal/seg11valid.txt
|
||||
segm_val_file: str = os.path.join(img_root_dir, "ImageSets/Segmentation/seg11valid.txt")
|
||||
colour_file_cls: str = os.path.join(img_root_dir, "ImageSets/Segmentation/pascal-classes.txt")
|
||||
```
|
||||
|
||||
These values can be modified in accordance with chosen model pipeline.
|
||||
|
||||
To initiate the evaluation of the PyTorch FCN ResNet-50, run the following line:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.segmentation.py_to_py_segm --model_name fcnresnet50
|
||||
```
|
||||
|
||||
#### Test Mode
|
||||
|
||||
The below line represents running of the module in the test mode, which provides the steps for the model inference:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.segmentation.py_to_py_segm --model_name <pytorch_segm_model_name> --test True --default_img_preprocess <True/False> --evaluate False
|
||||
```
|
||||
|
||||
Here ``default_img_preprocess`` key defines whether you'd like to parametrize the model test process with some particular values or use the default values, for example, ``scale``, ``mean`` or ``std``.
|
||||
|
||||
Test configuration is represented in [``test_config.py``](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) ``TestSegmModuleConfig`` class:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestSegmModuleConfig:
|
||||
segm_test_data_dir: str = "test_data/sem_segm"
|
||||
test_module_name: str = "segmentation"
|
||||
test_module_path: str = "segmentation.py"
|
||||
input_img: str = os.path.join(segm_test_data_dir, "2007_000033.jpg")
|
||||
model: str = ""
|
||||
|
||||
frame_height: str = str(TestSegmConfig.frame_size)
|
||||
frame_width: str = str(TestSegmConfig.frame_size)
|
||||
scale: float = 1.0
|
||||
mean: List[float] = field(default_factory=lambda: [0.0, 0.0, 0.0])
|
||||
std: List[float] = field(default_factory=list)
|
||||
crop: bool = False
|
||||
rgb: bool = True
|
||||
classes: str = os.path.join(segm_test_data_dir, "pascal-classes.txt")
|
||||
```
|
||||
|
||||
The default image preprocessing options are defined in ``default_preprocess_config.py``:
|
||||
|
||||
```python
|
||||
pytorch_segm_input_blob = {
|
||||
"mean": ["123.675", "116.28", "103.53"],
|
||||
"scale": str(1 / 255.0),
|
||||
"std": ["0.229", "0.224", "0.225"],
|
||||
"crop": "False",
|
||||
"rgb": "True"
|
||||
}
|
||||
```
|
||||
|
||||
The basis of the model testing is represented in ``samples/dnn/segmentation.py``. ``segmentation.py`` can be executed autonomously with provided converted model in ``--input`` and populated parameters for ``cv2.dnn.blobFromImage``.
|
||||
|
||||
To reproduce from scratch the described in "Model Conversion Pipeline" OpenCV steps with ``dnn_model_runner`` execute the below line:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.pytorch.segmentation.py_to_py_segm --model_name fcnresnet50 --test True --default_img_preprocess True --evaluate False
|
||||
```
|
||||
+406
@@ -0,0 +1,406 @@
|
||||
# Conversion of TensorFlow Segmentation Models and Launch with OpenCV {#tf_segm_tutorial_dnn_conversion}
|
||||
|
||||
## Goals
|
||||
In this tutorial you will learn how to:
|
||||
* convert TensorFlow (TF) segmentation models
|
||||
* run converted TensorFlow model with OpenCV
|
||||
* obtain an evaluation of the TensorFlow and OpenCV DNN models
|
||||
|
||||
We will explore the above-listed points by the example of the DeepLab architecture.
|
||||
|
||||
## Introduction
|
||||
The key concepts involved in the transition pipeline of the [TensorFlow classification](https://link_to_cls_tutorial) and segmentation models with OpenCV API are almost equal excepting the phase of graph optimization. The initial step in conversion of TensorFlow models into cv.dnn.Net
|
||||
is obtaining the frozen TF model graph. Frozen graph defines the combination of the model graph structure with kept values of the required variables, for example, weights. Usually the frozen graph is saved in [protobuf](https://en.wikipedia.org/wiki/Protocol_Buffers) (```.pb```) files.
|
||||
To read the generated segmentation model ``.pb`` file with cv.dnn.readNetFromTensorflow, it is needed to modify the graph with TF [graph transform tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms).
|
||||
|
||||
## Practice
|
||||
In this part we are going to cover the following points:
|
||||
1. create a TF classification model conversion pipeline and provide the inference
|
||||
2. evaluate and test TF classification models
|
||||
|
||||
If you'd like merely to run evaluation or test model pipelines, the "Model Conversion Pipeline" tutorial part can be skipped.
|
||||
|
||||
### Model Conversion Pipeline
|
||||
The code in this subchapter is located in the ``dnn_model_runner`` module and can be executed with the line:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.py_to_py_deeplab
|
||||
```
|
||||
|
||||
TensorFlow segmentation models can be found in [TensorFlow Research Models](https://github.com/tensorflow/models/tree/master/research/#tensorflow-research-models) section, which contains the implementations of models on the basis of published research papers.
|
||||
We will retrieve the archive with the pre-trained TF DeepLabV3 from the below link:
|
||||
|
||||
```
|
||||
http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
|
||||
```
|
||||
|
||||
The full frozen graph obtaining pipeline is described in ``deeplab_retrievement.py``:
|
||||
|
||||
```python
|
||||
def get_deeplab_frozen_graph():
|
||||
# define model path to download
|
||||
models_url = 'http://download.tensorflow.org/models/'
|
||||
mobilenetv2_voctrainval = 'deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz'
|
||||
|
||||
# construct model link to download
|
||||
model_link = models_url + mobilenetv2_voctrainval
|
||||
|
||||
try:
|
||||
urllib.request.urlretrieve(model_link, mobilenetv2_voctrainval)
|
||||
except Exception:
|
||||
print("TF DeepLabV3 was not retrieved: {}".format(model_link))
|
||||
return
|
||||
|
||||
tf_model_tar = tarfile.open(mobilenetv2_voctrainval)
|
||||
|
||||
# iterate the obtained model archive
|
||||
for model_tar_elem in tf_model_tar.getmembers():
|
||||
# check whether the model archive contains frozen graph
|
||||
if TF_FROZEN_GRAPH_NAME in os.path.basename(model_tar_elem.name):
|
||||
# extract frozen graph
|
||||
tf_model_tar.extract(model_tar_elem, FROZEN_GRAPH_PATH)
|
||||
|
||||
tf_model_tar.close()
|
||||
```
|
||||
|
||||
After running this script:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.deeplab_retrievement
|
||||
```
|
||||
|
||||
we will get ``frozen_inference_graph.pb`` in ``deeplab/deeplabv3_mnv2_pascal_trainval``.
|
||||
|
||||
Before going to the network loading with OpenCV it is needed to optimize the extracted ``frozen_inference_graph.pb``.
|
||||
To optimize the graph we use TF ``TransformGraph`` with default parameters:
|
||||
|
||||
```python
|
||||
DEFAULT_OPT_GRAPH_NAME = "optimized_frozen_inference_graph.pb"
|
||||
DEFAULT_INPUTS = "sub_7"
|
||||
DEFAULT_OUTPUTS = "ResizeBilinear_3"
|
||||
DEFAULT_TRANSFORMS = "remove_nodes(op=Identity)" \
|
||||
" merge_duplicate_nodes" \
|
||||
" strip_unused_nodes" \
|
||||
" fold_constants(ignore_errors=true)" \
|
||||
" fold_batch_norms" \
|
||||
" fold_old_batch_norms"
|
||||
|
||||
|
||||
def optimize_tf_graph(
|
||||
in_graph,
|
||||
out_graph=DEFAULT_OPT_GRAPH_NAME,
|
||||
inputs=DEFAULT_INPUTS,
|
||||
outputs=DEFAULT_OUTPUTS,
|
||||
transforms=DEFAULT_TRANSFORMS,
|
||||
is_manual=True,
|
||||
was_optimized=True
|
||||
):
|
||||
# ...
|
||||
|
||||
tf_opt_graph = TransformGraph(
|
||||
tf_graph,
|
||||
inputs,
|
||||
outputs,
|
||||
transforms
|
||||
)
|
||||
```
|
||||
|
||||
To run graph optimization process, execute the line:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.tf_graph_optimizer --in_graph deeplab/deeplabv3_mnv2_pascal_trainval/frozen_inference_graph.pb
|
||||
```
|
||||
|
||||
As a result ``deeplab/deeplabv3_mnv2_pascal_trainval`` directory will contain ``optimized_frozen_inference_graph.pb``.
|
||||
|
||||
After we have obtained the model graphs, let's examine the below-listed steps:
|
||||
1. read TF ``frozen_inference_graph.pb`` graph
|
||||
2. read optimized TF frozen graph with OpenCV API
|
||||
3. prepare input data
|
||||
4. provide inference
|
||||
5. get colored masks from predictions
|
||||
6. visualize results
|
||||
|
||||
```python
|
||||
# get TF model graph from the obtained frozen graph
|
||||
deeplab_graph = read_deeplab_frozen_graph(deeplab_frozen_graph_path)
|
||||
|
||||
# read DeepLab frozen graph with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromTensorflow(opt_deeplab_frozen_graph_path)
|
||||
print("OpenCV model was successfully read. Model layers: \n", opencv_net.getLayerNames())
|
||||
|
||||
# get processed image
|
||||
original_img_shape, tf_input_blob, opencv_input_img = get_processed_imgs("test_data/sem_segm/2007_000033.jpg")
|
||||
|
||||
# obtain OpenCV DNN predictions
|
||||
opencv_prediction = get_opencv_dnn_prediction(opencv_net, opencv_input_img)
|
||||
|
||||
# obtain TF model predictions
|
||||
tf_prediction = get_tf_dnn_prediction(deeplab_graph, tf_input_blob)
|
||||
|
||||
# get PASCAL VOC classes and colors
|
||||
pascal_voc_classes, pascal_voc_colors = read_colors_info("test_data/sem_segm/pascal-classes.txt")
|
||||
|
||||
# obtain colored segmentation masks
|
||||
opencv_colored_mask = get_colored_mask(original_img_shape, opencv_prediction, pascal_voc_colors)
|
||||
tf_colored_mask = get_tf_colored_mask(original_img_shape, tf_prediction, pascal_voc_colors)
|
||||
|
||||
# obtain palette of PASCAL VOC colors
|
||||
color_legend = get_legend(pascal_voc_classes, pascal_voc_colors)
|
||||
|
||||
cv2.imshow('TensorFlow Colored Mask', tf_colored_mask)
|
||||
cv2.imshow('OpenCV DNN Colored Mask', opencv_colored_mask)
|
||||
|
||||
cv2.imshow('Color Legend', color_legend)
|
||||
```
|
||||
|
||||
To provide the model inference we will use the below picture from the [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/) validation dataset:
|
||||
|
||||
|
||||
|
||||
The target segmented result is:
|
||||
|
||||
|
||||
|
||||
For the PASCAL VOC colors decoding and its mapping with the predicted masks, we also need ``pascal-classes.txt`` file, which contains the full list of the PASCAL VOC classes and corresponding colors.
|
||||
|
||||
Let's go deeper into each step by the example of pretrained TF DeepLabV3 MobileNetV2:
|
||||
|
||||
* read TF ``frozen_inference_graph.pb`` graph :
|
||||
|
||||
```python
|
||||
# init deeplab model graph
|
||||
model_graph = tf.Graph()
|
||||
|
||||
# obtain
|
||||
with tf.io.gfile.GFile(frozen_graph_path, 'rb') as graph_file:
|
||||
tf_model_graph = GraphDef()
|
||||
tf_model_graph.ParseFromString(graph_file.read())
|
||||
|
||||
with model_graph.as_default():
|
||||
tf.import_graph_def(tf_model_graph, name='')
|
||||
```
|
||||
|
||||
* read optimized TF frozen graph with OpenCV API:
|
||||
|
||||
```python
|
||||
# read DeepLab frozen graph with OpenCV API
|
||||
opencv_net = cv2.dnn.readNetFromTensorflow(opt_deeplab_frozen_graph_path)
|
||||
```
|
||||
|
||||
* prepare input data with cv2.dnn.blobFromImage function:
|
||||
|
||||
```python
|
||||
# read the image
|
||||
input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)
|
||||
input_img = input_img.astype(np.float32)
|
||||
|
||||
# preprocess image for TF model input
|
||||
tf_preproc_img = cv2.resize(input_img, (513, 513))
|
||||
tf_preproc_img = cv2.cvtColor(tf_preproc_img, cv2.COLOR_BGR2RGB)
|
||||
|
||||
# define preprocess parameters for OpenCV DNN
|
||||
mean = np.array([1.0, 1.0, 1.0]) * 127.5
|
||||
scale = 1 / 127.5
|
||||
|
||||
# prepare input blob to fit the model input:
|
||||
# 1. subtract mean
|
||||
# 2. scale to set pixel values from 0 to 1
|
||||
input_blob = cv2.dnn.blobFromImage(
|
||||
image=input_img,
|
||||
scalefactor=scale,
|
||||
size=(513, 513), # img target size
|
||||
mean=mean,
|
||||
swapRB=True, # BGR -> RGB
|
||||
crop=False # center crop
|
||||
)
|
||||
```
|
||||
|
||||
Please, pay attention at the preprocessing order in the ``cv2.dnn.blobFromImage`` function. Firstly, the mean value is subtracted and only then pixel values are multiplied by the defined scale.
|
||||
Therefore, to reproduce TF image preprocessing pipeline, we multiply ``mean`` by ``127.5``.
|
||||
Another important point is image preprocessing for TF DeepLab. To pass the image into TF model we need only to construct an appropriate shape, the rest image preprocessing is described in [feature_extractor.py](https://github.com/tensorflow/models/blob/master/research/deeplab/core/feature_extractor.py) and will be invoked automatically.
|
||||
|
||||
* provide OpenCV ``cv.dnn_Net`` inference:
|
||||
|
||||
```python
|
||||
# set OpenCV DNN input
|
||||
opencv_net.setInput(preproc_img)
|
||||
|
||||
# OpenCV DNN inference
|
||||
out = opencv_net.forward()
|
||||
print("OpenCV DNN segmentation prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
|
||||
# get IDs of predicted classes
|
||||
out_predictions = np.argmax(out[0], axis=0)
|
||||
```
|
||||
|
||||
After the above code execution we will get the following output:
|
||||
|
||||
```
|
||||
OpenCV DNN segmentation prediction:
|
||||
* shape: (1, 21, 513, 513)
|
||||
|
||||
```
|
||||
|
||||
Each prediction channel out of 21, where 21 represents the number of PASCAL VOC classes, contains probabilities, which indicate how likely the pixel corresponds to the PASCAL VOC class.
|
||||
|
||||
* provide TF model inference:
|
||||
|
||||
```python
|
||||
preproc_img = np.expand_dims(preproc_img, 0)
|
||||
|
||||
# init TF session
|
||||
tf_session = Session(graph=model_graph)
|
||||
|
||||
input_tensor_name = "ImageTensor:0",
|
||||
output_tensor_name = "SemanticPredictions:0"
|
||||
|
||||
# run inference
|
||||
out = tf_session.run(
|
||||
output_tensor_name,
|
||||
feed_dict={input_tensor_name: [preproc_img]}
|
||||
)
|
||||
|
||||
print("TF segmentation model prediction: \n")
|
||||
print("* shape: ", out.shape)
|
||||
```
|
||||
|
||||
TF inference results are the following:
|
||||
|
||||
```
|
||||
TF segmentation model prediction:
|
||||
* shape: (1, 513, 513)
|
||||
```
|
||||
|
||||
TensorFlow prediction contains the indexes of corresponding PASCAL VOC classes.
|
||||
|
||||
* transform OpenCV prediction into colored mask:
|
||||
|
||||
```python
|
||||
mask_height = segm_mask.shape[0]
|
||||
mask_width = segm_mask.shape[1]
|
||||
|
||||
img_height = original_img_shape[0]
|
||||
img_width = original_img_shape[1]
|
||||
|
||||
# convert mask values into PASCAL VOC colors
|
||||
processed_mask = np.stack([colors[color_id] for color_id in segm_mask.flatten()])
|
||||
|
||||
# reshape mask into 3-channel image
|
||||
processed_mask = processed_mask.reshape(mask_height, mask_width, 3)
|
||||
processed_mask = cv2.resize(processed_mask, (img_width, img_height), interpolation=cv2.INTER_NEAREST).astype(
|
||||
np.uint8)
|
||||
|
||||
# convert colored mask from BGR to RGB
|
||||
processed_mask = cv2.cvtColor(processed_mask, cv2.COLOR_BGR2RGB)
|
||||
```
|
||||
|
||||
In this step we map the probabilities from segmentation masks with appropriate colors of the predicted classes. Let's have a look at the results:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
* transform TF prediction into colored mask:
|
||||
|
||||
```python
|
||||
colors = np.array(colors)
|
||||
processed_mask = colors[segm_mask[0]]
|
||||
|
||||
img_height = original_img_shape[0]
|
||||
img_width = original_img_shape[1]
|
||||
|
||||
processed_mask = cv2.resize(processed_mask, (img_width, img_height), interpolation=cv2.INTER_NEAREST).astype(
|
||||
np.uint8)
|
||||
|
||||
# convert colored mask from BGR to RGB for compatibility with PASCAL VOC colors
|
||||
processed_mask = cv2.cvtColor(processed_mask, cv2.COLOR_BGR2RGB)
|
||||
```
|
||||
|
||||
The result is:
|
||||
|
||||
|
||||
|
||||
As a result, we get two equal segmentation masks.
|
||||
|
||||
### Evaluation of the Models
|
||||
|
||||
The proposed in ``dnn/samples`` ``dnn_model_runner`` module allows to run the full evaluation pipeline on the PASCAL VOC dataset and test execution for the DeepLab MobileNet model.
|
||||
|
||||
#### Evaluation Mode
|
||||
|
||||
To below line represents running of the module in the evaluation mode:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.py_to_py_segm
|
||||
```
|
||||
|
||||
The model will be read into OpenCV ``cv.dnn_Net`` object. Evaluation results of TF and OpenCV models (pixel accuracy, mean IoU, inference time) will be written into the log file. Inference time values will be also depicted in a chart to generalize the obtained model information.
|
||||
|
||||
Necessary evaluation configurations are defined in the [``test_config.py``](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py):
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestSegmConfig:
|
||||
frame_size: int = 500
|
||||
img_root_dir: str = "./VOC2012"
|
||||
img_dir: str = os.path.join(img_root_dir, "JPEGImages/")
|
||||
img_segm_gt_dir: str = os.path.join(img_root_dir, "SegmentationClass/")
|
||||
# reduced val: https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/data/pascal/seg11valid.txt
|
||||
segm_val_file: str = os.path.join(img_root_dir, "ImageSets/Segmentation/seg11valid.txt")
|
||||
colour_file_cls: str = os.path.join(img_root_dir, "ImageSets/Segmentation/pascal-classes.txt")
|
||||
```
|
||||
|
||||
These values can be modified in accordance with chosen model pipeline.
|
||||
|
||||
#### Test Mode
|
||||
|
||||
The below line represents running of the module in the test mode, which provides the steps for the model inference:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.py_to_py_segm --test True --default_img_preprocess <True/False> --evaluate False
|
||||
```
|
||||
|
||||
Here ``default_img_preprocess`` key defines whether you'd like to parametrize the model test process with some particular values or use the default values, for example, ``scale``, ``mean`` or ``std``.
|
||||
|
||||
Test configuration is represented in [``test_config.py``](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/common/test/configs/test_config.py) ``TestSegmModuleConfig`` class:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class TestSegmModuleConfig:
|
||||
segm_test_data_dir: str = "test_data/sem_segm"
|
||||
test_module_name: str = "segmentation"
|
||||
test_module_path: str = "segmentation.py"
|
||||
input_img: str = os.path.join(segm_test_data_dir, "2007_000033.jpg")
|
||||
model: str = ""
|
||||
|
||||
frame_height: str = str(TestSegmConfig.frame_size)
|
||||
frame_width: str = str(TestSegmConfig.frame_size)
|
||||
scale: float = 1.0
|
||||
mean: List[float] = field(default_factory=lambda: [0.0, 0.0, 0.0])
|
||||
std: List[float] = field(default_factory=list)
|
||||
crop: bool = False
|
||||
rgb: bool = True
|
||||
classes: str = os.path.join(segm_test_data_dir, "pascal-classes.txt")
|
||||
```
|
||||
|
||||
The default image preprocessing options are defined in ``default_preprocess_config.py``:
|
||||
|
||||
```python
|
||||
tf_segm_input_blob = {
|
||||
"scale": str(1 / 127.5),
|
||||
"mean": ["127.5", "127.5", "127.5"],
|
||||
"std": [],
|
||||
"crop": "False",
|
||||
"rgb": "True"
|
||||
}
|
||||
```
|
||||
|
||||
The basis of the model testing is represented in ``samples/dnn/segmentation.py``. ``segmentation.py`` can be executed autonomously with provided converted model in ``--input`` and populated parameters for ``cv2.dnn.blobFromImage``.
|
||||
|
||||
To reproduce from scratch the described in "Model Conversion Pipeline" OpenCV steps with ``dnn_model_runner`` execute the below line:
|
||||
|
||||
```
|
||||
python -m dnn_model_runner.dnn_conversion.tf.segmentation.py_to_py_segm --test True --default_img_preprocess True --evaluate False
|
||||
```
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
@@ -0,0 +1,324 @@
|
||||
# High Level API: TextDetectionModel and TextRecognitionModel {#tutorial_dnn_text_spotting}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_OCR}
|
||||
@next_tutorial{pytorch_cls_tutorial_dnn_conversion}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Wenqing Zhang |
|
||||
| Compatibility | OpenCV >= 4.5 |
|
||||
|
||||
## Introduction
|
||||
In this tutorial, we will introduce the APIs for TextRecognitionModel and TextDetectionModel in detail.
|
||||
|
||||
---
|
||||
#### TextRecognitionModel:
|
||||
|
||||
In the current version, @ref cv::dnn::TextRecognitionModel only supports CNN+RNN+CTC based algorithms,
|
||||
and the greedy decoding method for CTC is provided.
|
||||
For more information, please refer to the [original paper](https://arxiv.org/abs/1507.05717)
|
||||
|
||||
Before recognition, you should `setVocabulary` and `setDecodeType`.
|
||||
- "CTC-greedy", the output of the text recognition model should be a probability matrix.
|
||||
The shape should be `(T, B, Dim)`, where
|
||||
- `T` is the sequence length
|
||||
- `B` is the batch size (only support `B=1` in inference)
|
||||
- and `Dim` is the length of vocabulary +1('Blank' of CTC is at the index=0 of Dim).
|
||||
|
||||
@ref cv::dnn::TextRecognitionModel::recognize() is the main function for text recognition.
|
||||
- The input image should be a cropped text image or an image with `roiRects`
|
||||
- Other decoding methods may supported in the future
|
||||
|
||||
---
|
||||
|
||||
#### TextDetectionModel:
|
||||
|
||||
@ref cv::dnn::TextDetectionModel API provides these methods for text detection:
|
||||
- cv::dnn::TextDetectionModel::detect() returns the results in std::vector<std::vector<Point>> (4-points quadrangles)
|
||||
- cv::dnn::TextDetectionModel::detectTextRectangles() returns the results in std::vector<cv::RotatedRect> (RBOX-like)
|
||||
|
||||
In the current version, @ref cv::dnn::TextDetectionModel supports these algorithms:
|
||||
- use @ref cv::dnn::TextDetectionModel_DB with "DB" models
|
||||
- and use @ref cv::dnn::TextDetectionModel_EAST with "EAST" models
|
||||
|
||||
The following provided pretrained models are variants of DB (w/o deformable convolution),
|
||||
and the performance can be referred to the Table.1 in the [paper]((https://arxiv.org/abs/1911.08947)).
|
||||
For more information, please refer to the [official code](https://github.com/MhLiao/DB)
|
||||
|
||||
---
|
||||
|
||||
You can train your own model with more data, and convert it into ONNX format.
|
||||
We encourage you to add new algorithms to these APIs.
|
||||
|
||||
|
||||
## Pretrained Models
|
||||
|
||||
#### TextRecognitionModel:
|
||||
|
||||
```
|
||||
crnn.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=1ooaLR-rkTl8jdpGy1DoQs0-X0lQsB6Fj
|
||||
sha: 270d92c9ccb670ada2459a25977e8deeaf8380d3,
|
||||
alphabet_36.txt: https://drive.google.com/uc?export=dowload&id=1oPOYx5rQRp8L6XQciUwmwhMCfX0KyO4b
|
||||
parameter setting: -rgb=0;
|
||||
description: The classification number of this model is 36 (0~9 + a~z).
|
||||
The training dataset is MJSynth.
|
||||
|
||||
crnn_cs.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=12diBsVJrS9ZEl6BNUiRp9s0xPALBS7kt
|
||||
sha: a641e9c57a5147546f7a2dbea4fd322b47197cd5
|
||||
alphabet_94.txt: https://drive.google.com/uc?export=dowload&id=1oKXxXKusquimp7XY1mFvj9nwLzldVgBR
|
||||
parameter setting: -rgb=1;
|
||||
description: The classification number of this model is 94 (0~9 + a~z + A~Z + punctuations).
|
||||
The training datasets are MJsynth and SynthText.
|
||||
|
||||
crnn_cs_CN.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=1is4eYEUKH7HR7Gl37Sw4WPXx6Ir8oQEG
|
||||
sha: 3940942b85761c7f240494cf662dcbf05dc00d14
|
||||
alphabet_3944.txt: https://drive.google.com/uc?export=dowload&id=18IZUUdNzJ44heWTndDO6NNfIpJMmN-ul
|
||||
parameter setting: -rgb=1;
|
||||
description: The classification number of this model is 3944 (0~9 + a~z + A~Z + Chinese characters + special characters).
|
||||
The training dataset is ReCTS (https://rrc.cvc.uab.es/?ch=12).
|
||||
```
|
||||
|
||||
More models can be found in [here](https://drive.google.com/drive/folders/1cTbQ3nuZG-EKWak6emD_s8_hHXWz7lAr?usp=sharing),
|
||||
which are taken from [clovaai](https://github.com/clovaai/deep-text-recognition-benchmark).
|
||||
You can train more models by [CRNN](https://github.com/meijieru/crnn.pytorch), and convert models by `torch.onnx.export`.
|
||||
|
||||
#### TextDetectionModel:
|
||||
|
||||
```
|
||||
- DB_IC15_resnet50.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=17_ABp79PlFt9yPCxSaarVc_DKTmrSGGf
|
||||
sha: bef233c28947ef6ec8c663d20a2b326302421fa3
|
||||
recommended parameter setting: -inputHeight=736, -inputWidth=1280;
|
||||
description: This model is trained on ICDAR2015, so it can only detect English text instances.
|
||||
|
||||
- DB_IC15_resnet18.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=1sZszH3pEt8hliyBlTmB-iulxHP1dCQWV
|
||||
sha: 19543ce09b2efd35f49705c235cc46d0e22df30b
|
||||
recommended parameter setting: -inputHeight=736, -inputWidth=1280;
|
||||
description: This model is trained on ICDAR2015, so it can only detect English text instances.
|
||||
|
||||
- DB_TD500_resnet50.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=19YWhArrNccaoSza0CfkXlA8im4-lAGsR
|
||||
sha: 1b4dd21a6baa5e3523156776970895bd3db6960a
|
||||
recommended parameter setting: -inputHeight=736, -inputWidth=736;
|
||||
description: This model is trained on MSRA-TD500, so it can detect both English and Chinese text instances.
|
||||
|
||||
- DB_TD500_resnet18.onnx:
|
||||
url: https://drive.google.com/uc?export=dowload&id=1vY_KsDZZZb_svd5RT6pjyI8BS1nPbBSX
|
||||
sha: 8a3700bdc13e00336a815fc7afff5dcc1ce08546
|
||||
recommended parameter setting: -inputHeight=736, -inputWidth=736;
|
||||
description: This model is trained on MSRA-TD500, so it can detect both English and Chinese text instances.
|
||||
|
||||
```
|
||||
|
||||
We will release more models of DB [here](https://drive.google.com/drive/folders/1qzNCHfUJOS0NEUOIKn69eCtxdlNPpWbq?usp=sharing) in the future.
|
||||
|
||||
```
|
||||
- EAST:
|
||||
Download link: https://www.dropbox.com/s/r2ingd0l3zt8hxs/frozen_east_text_detection.tar.gz?dl=1
|
||||
This model is based on https://github.com/argman/EAST
|
||||
```
|
||||
|
||||
## Images for Testing
|
||||
|
||||
```
|
||||
Text Recognition:
|
||||
url: https://drive.google.com/uc?export=dowload&id=1nMcEy68zDNpIlqAn6xCk_kYcUTIeSOtN
|
||||
sha: 89205612ce8dd2251effa16609342b69bff67ca3
|
||||
|
||||
Text Detection:
|
||||
url: https://drive.google.com/uc?export=dowload&id=149tAhIcvfCYeyufRoZ9tmc2mZDKE_XrF
|
||||
sha: ced3c03fb7f8d9608169a913acf7e7b93e07109b
|
||||
```
|
||||
|
||||
## Example for Text Recognition
|
||||
|
||||
Step1. Loading images and models with a vocabulary
|
||||
|
||||
```cpp
|
||||
// Load a cropped text line image
|
||||
// you can find cropped images for testing in "Images for Testing"
|
||||
int rgb = IMREAD_COLOR; // This should be changed according to the model input requirement.
|
||||
Mat image = imread("path/to/text_rec_test.png", rgb);
|
||||
|
||||
// Load models weights
|
||||
TextRecognitionModel model("path/to/crnn_cs.onnx");
|
||||
|
||||
// The decoding method
|
||||
// more methods will be supported in future
|
||||
model.setDecodeType("CTC-greedy");
|
||||
|
||||
// Load vocabulary
|
||||
// vocabulary should be changed according to the text recognition model
|
||||
std::ifstream vocFile;
|
||||
vocFile.open("path/to/alphabet_94.txt");
|
||||
CV_Assert(vocFile.is_open());
|
||||
String vocLine;
|
||||
std::vector<String> vocabulary;
|
||||
while (std::getline(vocFile, vocLine)) {
|
||||
vocabulary.push_back(vocLine);
|
||||
}
|
||||
model.setVocabulary(vocabulary);
|
||||
```
|
||||
|
||||
Step2. Setting Parameters
|
||||
|
||||
```cpp
|
||||
// Normalization parameters
|
||||
double scale = 1.0 / 127.5;
|
||||
Scalar mean = Scalar(127.5, 127.5, 127.5);
|
||||
|
||||
// The input shape
|
||||
Size inputSize = Size(100, 32);
|
||||
|
||||
model.setInputParams(scale, inputSize, mean);
|
||||
```
|
||||
Step3. Inference
|
||||
```cpp
|
||||
std::string recognitionResult = recognizer.recognize(image);
|
||||
std::cout << "'" << recognitionResult << "'" << std::endl;
|
||||
```
|
||||
|
||||
Input image:
|
||||
|
||||
|
||||
|
||||
Output:
|
||||
```
|
||||
'welcome'
|
||||
```
|
||||
|
||||
|
||||
## Example for Text Detection
|
||||
|
||||
Step1. Loading images and models
|
||||
```cpp
|
||||
// Load an image
|
||||
// you can find some images for testing in "Images for Testing"
|
||||
Mat frame = imread("/path/to/text_det_test.png");
|
||||
```
|
||||
|
||||
Step2.a Setting Parameters (DB)
|
||||
```cpp
|
||||
// Load model weights
|
||||
TextDetectionModel_DB model("/path/to/DB_TD500_resnet50.onnx");
|
||||
|
||||
// Post-processing parameters
|
||||
float binThresh = 0.3;
|
||||
float polyThresh = 0.5;
|
||||
uint maxCandidates = 200;
|
||||
double unclipRatio = 2.0;
|
||||
model.setBinaryThreshold(binThresh)
|
||||
.setPolygonThreshold(polyThresh)
|
||||
.setMaxCandidates(maxCandidates)
|
||||
.setUnclipRatio(unclipRatio)
|
||||
;
|
||||
|
||||
// Normalization parameters
|
||||
double scale = 1.0 / 255.0;
|
||||
Scalar mean = Scalar(122.67891434, 116.66876762, 104.00698793);
|
||||
|
||||
// The input shape
|
||||
Size inputSize = Size(736, 736);
|
||||
|
||||
model.setInputParams(scale, inputSize, mean);
|
||||
```
|
||||
|
||||
Step2.b Setting Parameters (EAST)
|
||||
```cpp
|
||||
TextDetectionModel_EAST model("EAST.pb");
|
||||
|
||||
float confThreshold = 0.5;
|
||||
float nmsThreshold = 0.4;
|
||||
model.setConfidenceThreshold(confThresh)
|
||||
.setNMSThreshold(nmsThresh)
|
||||
;
|
||||
|
||||
double detScale = 1.0;
|
||||
Size detInputSize = Size(320, 320);
|
||||
Scalar detMean = Scalar(123.68, 116.78, 103.94);
|
||||
bool swapRB = true;
|
||||
model.setInputParams(detScale, detInputSize, detMean, swapRB);
|
||||
```
|
||||
|
||||
|
||||
Step3. Inference
|
||||
```cpp
|
||||
std::vector<std::vector<Point>> detResults;
|
||||
model.detect(detResults);
|
||||
|
||||
// Visualization
|
||||
polylines(frame, results, true, Scalar(0, 255, 0), 2);
|
||||
imshow("Text Detection", image);
|
||||
waitKey();
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
|
||||
|
||||
## Example for Text Spotting
|
||||
|
||||
After following the steps above, it is easy to get the detection results of an input image.
|
||||
Then, you can do transformation and crop text images for recognition.
|
||||
For more information, please refer to **Detailed Sample**
|
||||
```cpp
|
||||
// Transform and Crop
|
||||
Mat cropped;
|
||||
fourPointsTransform(recInput, vertices, cropped);
|
||||
|
||||
String recResult = recognizer.recognize(cropped);
|
||||
```
|
||||
|
||||
Output Examples:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Source Code
|
||||
The [source code](https://github.com/opencv/opencv/blob/master/modules/dnn/src/model.cpp)
|
||||
of these APIs can be found in the DNN module.
|
||||
|
||||
## Detailed Sample
|
||||
For more information, please refer to:
|
||||
- [samples/dnn/scene_text_recognition.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/scene_text_recognition.cpp)
|
||||
- [samples/dnn/scene_text_detection.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/scene_text_detection.cpp)
|
||||
- [samples/dnn/text_detection.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/text_detection.cpp)
|
||||
- [samples/dnn/scene_text_spotting.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/scene_text_spotting.cpp)
|
||||
|
||||
#### Test with an image
|
||||
Examples:
|
||||
```bash
|
||||
example_dnn_scene_text_recognition -mp=path/to/crnn_cs.onnx -i=path/to/an/image -rgb=1 -vp=/path/to/alphabet_94.txt
|
||||
example_dnn_scene_text_detection -mp=path/to/DB_TD500_resnet50.onnx -i=path/to/an/image -ih=736 -iw=736
|
||||
example_dnn_scene_text_spotting -dmp=path/to/DB_IC15_resnet50.onnx -rmp=path/to/crnn_cs.onnx -i=path/to/an/image -iw=1280 -ih=736 -rgb=1 -vp=/path/to/alphabet_94.txt
|
||||
example_dnn_text_detection -dmp=path/to/EAST.pb -rmp=path/to/crnn_cs.onnx -i=path/to/an/image -rgb=1 -vp=path/to/alphabet_94.txt
|
||||
```
|
||||
|
||||
#### Test on public datasets
|
||||
Text Recognition:
|
||||
|
||||
The download link for testing images can be found in the **Images for Testing**
|
||||
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
example_dnn_scene_text_recognition -mp=path/to/crnn.onnx -e=true -edp=path/to/evaluation_data_rec -vp=/path/to/alphabet_36.txt -rgb=0
|
||||
example_dnn_scene_text_recognition -mp=path/to/crnn_cs.onnx -e=true -edp=path/to/evaluation_data_rec -vp=/path/to/alphabet_94.txt -rgb=1
|
||||
```
|
||||
|
||||
Text Detection:
|
||||
|
||||
The download links for testing images can be found in the **Images for Testing**
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
example_dnn_scene_text_detection -mp=path/to/DB_TD500_resnet50.onnx -e=true -edp=path/to/evaluation_data_det/TD500 -ih=736 -iw=736
|
||||
example_dnn_scene_text_detection -mp=path/to/DB_IC15_resnet50.onnx -e=true -edp=path/to/evaluation_data_det/IC15 -ih=736 -iw=1280
|
||||
```
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 48 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.8 KiB |
@@ -1,9 +1,16 @@
|
||||
YOLO DNNs {#tutorial_dnn_yolo}
|
||||
===============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_android}
|
||||
@next_tutorial{tutorial_dnn_javascript}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Alessandro de Oliveira Faria |
|
||||
| Compatibility | OpenCV >= 3.3.1 |
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
|
||||
@@ -2,81 +2,23 @@ Deep Neural Networks (dnn module) {#tutorial_table_of_content_dnn}
|
||||
=====================================
|
||||
|
||||
- @subpage tutorial_dnn_googlenet
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3
|
||||
|
||||
*Author:* Vitaliy Lyudvichenko
|
||||
|
||||
In this tutorial you will learn how to use opencv_dnn module for image classification by using GoogLeNet trained network from Caffe model zoo.
|
||||
|
||||
- @subpage tutorial_dnn_halide
|
||||
|
||||
*Languages:* Halide
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3
|
||||
|
||||
*Author:* Dmitry Kurtaev
|
||||
|
||||
This tutorial guidelines how to run your models in OpenCV deep learning module using Halide language backend.
|
||||
|
||||
- @subpage tutorial_dnn_halide_scheduling
|
||||
|
||||
*Languages:* Halide
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3
|
||||
|
||||
*Author:* Dmitry Kurtaev
|
||||
|
||||
In this tutorial we describe the ways to schedule your networks using Halide backend in OpenCV deep learning module.
|
||||
|
||||
- @subpage tutorial_dnn_android
|
||||
|
||||
*Languages:* Java
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3
|
||||
|
||||
*Author:* Dmitry Kurtaev
|
||||
|
||||
This tutorial will show you how to run deep learning model using OpenCV on Android device.
|
||||
|
||||
- @subpage tutorial_dnn_yolo
|
||||
|
||||
*Languages:* C++, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3.1
|
||||
|
||||
*Author:* Alessandro de Oliveira Faria
|
||||
|
||||
In this tutorial you will learn how to use opencv_dnn module using yolo_object_detection with device capture, video file or image.
|
||||
|
||||
- @subpage tutorial_dnn_javascript
|
||||
|
||||
*Languages:* JavaScript
|
||||
|
||||
*Compatibility:* \> OpenCV 3.3.1
|
||||
|
||||
*Author:* Dmitry Kurtaev
|
||||
|
||||
In this tutorial we'll run deep learning models in browser using OpenCV.js.
|
||||
|
||||
- @subpage tutorial_dnn_custom_layers
|
||||
|
||||
*Languages:* C++, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 3.4.1
|
||||
|
||||
*Author:* Dmitry Kurtaev
|
||||
|
||||
How to define custom layers to import networks.
|
||||
|
||||
- @subpage tutorial_dnn_OCR
|
||||
- @subpage tutorial_dnn_text_spotting
|
||||
|
||||
*Languages:* C++
|
||||
#### PyTorch models with OpenCV
|
||||
In this section you will find the guides, which describe how to run classification, segmentation and detection PyTorch DNN models with OpenCV.
|
||||
- @subpage pytorch_cls_tutorial_dnn_conversion
|
||||
- @subpage pytorch_cls_c_tutorial_dnn_conversion
|
||||
- @subpage pytorch_segm_tutorial_dnn_conversion
|
||||
|
||||
*Compatibility:* \> OpenCV 4.3
|
||||
|
||||
*Author:* Zihao Mu
|
||||
|
||||
In this tutorial you will learn how to use opencv_dnn module using custom OCR models.
|
||||
#### TensorFlow models with OpenCV
|
||||
In this section you will find the guides, which describe how to run classification, segmentation and detection TensorFlow DNN models with OpenCV.
|
||||
- @subpage tf_cls_tutorial_dnn_conversion
|
||||
- @subpage tf_det_tutorial_dnn_conversion
|
||||
- @subpage tf_segm_tutorial_dnn_conversion
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
AKAZE local features matching {#tutorial_akaze_matching}
|
||||
=============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_detection_of_planar_objects}
|
||||
@next_tutorial{tutorial_akaze_tracking}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Fedor Morozov |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
AKAZE and ORB planar tracking {#tutorial_akaze_tracking}
|
||||
=============================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_akaze_matching}
|
||||
@next_tutorial{tutorial_homography}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Fedor Morozov |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
|
||||
+6
@@ -1,9 +1,15 @@
|
||||
Detection of planar objects {#tutorial_detection_of_planar_objects}
|
||||
===========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_feature_homography}
|
||||
@next_tutorial{tutorial_akaze_matching}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Victor Eruhimov |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
The goal of this tutorial is to learn how to use *features2d* and *calib3d* modules for detecting
|
||||
known planar objects in scenes.
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Feature Description {#tutorial_feature_description}
|
||||
===================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_feature_detection}
|
||||
@next_tutorial{tutorial_feature_flann_matcher}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Feature Detection {#tutorial_feature_detection}
|
||||
=================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_corner_subpixels}
|
||||
@next_tutorial{tutorial_feature_description}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Feature Matching with FLANN {#tutorial_feature_flann_matcher}
|
||||
===========================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_feature_description}
|
||||
@next_tutorial{tutorial_feature_homography}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,9 +1,16 @@
|
||||
Features2D + Homography to find a known object {#tutorial_feature_homography}
|
||||
==============================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_feature_flann_matcher}
|
||||
@next_tutorial{tutorial_detection_of_planar_objects}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Ana Huamán |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
Basic concepts of the homography explained with code {#tutorial_homography}
|
||||
====================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_akaze_tracking}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Compatibility | OpenCV >= 3.0 |
|
||||
|
||||
@tableofcontents
|
||||
|
||||
Introduction {#tutorial_homography_Introduction}
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 117 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 79 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 51 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 7.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.1 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.7 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.9 KiB |
@@ -1,128 +1,15 @@
|
||||
2D Features framework (feature2d module) {#tutorial_table_of_content_features2d}
|
||||
=========================================
|
||||
|
||||
Learn about how to use the feature points detectors, descriptors and matching framework found inside
|
||||
OpenCV.
|
||||
|
||||
- @subpage tutorial_harris_detector
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
Why is it a good idea to track corners? We learn how to use the Harris method to detect
|
||||
corners.
|
||||
|
||||
- @subpage tutorial_good_features_to_track
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
Where we use an improved method to detect corners more accurately.
|
||||
|
||||
- @subpage tutorial_generic_corner_detector
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
Here you will learn how to use OpenCV functions to make your personalized corner detector!
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
- @subpage tutorial_corner_subpixels
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
Is pixel resolution enough? Here we learn a simple method to improve our corner location accuracy.
|
||||
|
||||
- @subpage tutorial_feature_detection
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
In this tutorial, you will use *features2d* to detect interest points.
|
||||
|
||||
- @subpage tutorial_feature_description
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
In this tutorial, you will use *features2d* to calculate feature vectors.
|
||||
|
||||
- @subpage tutorial_feature_flann_matcher
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
In this tutorial, you will use the FLANN library to make a fast matching.
|
||||
|
||||
- @subpage tutorial_feature_homography
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Ana Huamán
|
||||
|
||||
In this tutorial, you will use *features2d* and *calib3d* to detect an object in a scene.
|
||||
|
||||
- @subpage tutorial_detection_of_planar_objects
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 2.0
|
||||
|
||||
*Author:* Victor Eruhimov
|
||||
|
||||
You will use *features2d* and *calib3d* modules for detecting known planar objects in
|
||||
scenes.
|
||||
|
||||
- @subpage tutorial_akaze_matching
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 3.0
|
||||
|
||||
*Author:* Fedor Morozov
|
||||
|
||||
Using *AKAZE* local features to find correspondence between two images.
|
||||
|
||||
- @subpage tutorial_akaze_tracking
|
||||
|
||||
*Languages:* C++
|
||||
|
||||
*Compatibility:* \> OpenCV 3.0
|
||||
|
||||
*Author:* Fedor Morozov
|
||||
|
||||
Using *AKAZE* and *ORB* for planar object tracking.
|
||||
|
||||
- @subpage tutorial_homography
|
||||
|
||||
*Languages:* C++, Java, Python
|
||||
|
||||
*Compatibility:* \> OpenCV 3.0
|
||||
|
||||
This tutorial will explain the basic concepts of the homography with some
|
||||
demonstration codes.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user