Merge remote-tracking branch 'upstream/3.4' into merge-3.4

doc/py_tutorials/py_video/py_lucas_kanade/py_lucas_kanade.markdown

@@ -1,225 +1,4 @@
 Optical Flow {#tutorial_py_lucas_kanade}
 ============
 
-Goal
-----
-
-In this chapter,
-    -   We will understand the concepts of optical flow and its estimation using Lucas-Kanade
-        method.
-    -   We will use functions like **cv.calcOpticalFlowPyrLK()** to track feature points in a
-        video.
-
-Optical Flow
-------------
-
-Optical flow is the pattern of apparent motion of image objects between two consecutive frames
-caused by the movemement of object or camera. It is 2D vector field where each vector is a
-displacement vector showing the movement of points from first frame to second. Consider the image
-below (Image Courtesy: [Wikipedia article on Optical
-Flow](http://en.wikipedia.org/wiki/Optical_flow)).
-
-![image](images/optical_flow_basic1.jpg)
-
-It shows a ball moving in 5 consecutive frames. The arrow shows its displacement vector. Optical
-flow has many applications in areas like :
-
--   Structure from Motion
--   Video Compression
--   Video Stabilization ...
-
-Optical flow works on several assumptions:
-
--#  The pixel intensities of an object do not change between consecutive frames.
-2.  Neighbouring pixels have similar motion.
-
-Consider a pixel \f$I(x,y,t)\f$ in first frame (Check a new dimension, time, is added here. Earlier we
-were working with images only, so no need of time). It moves by distance \f$(dx,dy)\f$ in next frame
-taken after \f$dt\f$ time. So since those pixels are the same and intensity does not change, we can say,
-
-\f[I(x,y,t) = I(x+dx, y+dy, t+dt)\f]
-
-Then take taylor series approximation of right-hand side, remove common terms and divide by \f$dt\f$ to
-get the following equation:
-
-\f[f_x u + f_y v + f_t = 0 \;\f]
-
-where:
-
-\f[f_x = \frac{\partial f}{\partial x} \; ; \; f_y = \frac{\partial f}{\partial y}\f]\f[u = \frac{dx}{dt} \; ; \; v = \frac{dy}{dt}\f]
-
-Above equation is called Optical Flow equation. In it, we can find \f$f_x\f$ and \f$f_y\f$, they are image
-gradients. Similarly \f$f_t\f$ is the gradient along time. But \f$(u,v)\f$ is unknown. We cannot solve this
-one equation with two unknown variables. So several methods are provided to solve this problem and
-one of them is Lucas-Kanade.
-
-### Lucas-Kanade method
-
-We have seen an assumption before, that all the neighbouring pixels will have similar motion.
-Lucas-Kanade method takes a 3x3 patch around the point. So all the 9 points have the same motion. We
-can find \f$(f_x, f_y, f_t)\f$ for these 9 points. So now our problem becomes solving 9 equations with
-two unknown variables which is over-determined. A better solution is obtained with least square fit
-method. Below is the final solution which is two equation-two unknown problem and solve to get the
-solution.
-
-\f[\begin{bmatrix} u \\ v \end{bmatrix} =
-\begin{bmatrix}
-    \sum_{i}{f_{x_i}}^2  &  \sum_{i}{f_{x_i} f_{y_i} } \\
-    \sum_{i}{f_{x_i} f_{y_i}} & \sum_{i}{f_{y_i}}^2
-\end{bmatrix}^{-1}
-\begin{bmatrix}
-    - \sum_{i}{f_{x_i} f_{t_i}} \\
-    - \sum_{i}{f_{y_i} f_{t_i}}
-\end{bmatrix}\f]
-
-( Check similarity of inverse matrix with Harris corner detector. It denotes that corners are better
-points to be tracked.)
-
-So from the user point of view, the idea is simple, we give some points to track, we receive the optical
-flow vectors of those points. But again there are some problems. Until now, we were dealing with
-small motions, so it fails when there is a large motion. To deal with this we use pyramids. When we go up in
-the pyramid, small motions are removed and large motions become small motions. So by applying
-Lucas-Kanade there, we get optical flow along with the scale.
-
-Lucas-Kanade Optical Flow in OpenCV
------------------------------------
-
-OpenCV provides all these in a single function, **cv.calcOpticalFlowPyrLK()**. Here, we create a
-simple application which tracks some points in a video. To decide the points, we use
-**cv.goodFeaturesToTrack()**. We take the first frame, detect some Shi-Tomasi corner points in it,
-then we iteratively track those points using Lucas-Kanade optical flow. For the function
-**cv.calcOpticalFlowPyrLK()** we pass the previous frame, previous points and next frame. It
-returns next points along with some status numbers which has a value of 1 if next point is found,
-else zero. We iteratively pass these next points as previous points in next step. See the code
-below:
-@code{.py}
-import numpy as np
-import cv2 as cv
-
-cap = cv.VideoCapture('slow.flv')
-
-# params for ShiTomasi corner detection
-feature_params = dict( maxCorners = 100,
-                       qualityLevel = 0.3,
-                       minDistance = 7,
-                       blockSize = 7 )
-
-# Parameters for lucas kanade optical flow
-lk_params = dict( winSize  = (15,15),
-                  maxLevel = 2,
-                  criteria = (cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 0.03))
-
-# Create some random colors
-color = np.random.randint(0,255,(100,3))
-
-# Take first frame and find corners in it
-ret, old_frame = cap.read()
-old_gray = cv.cvtColor(old_frame, cv.COLOR_BGR2GRAY)
-p0 = cv.goodFeaturesToTrack(old_gray, mask = None, **feature_params)
-
-# Create a mask image for drawing purposes
-mask = np.zeros_like(old_frame)
-
-while(1):
-    ret,frame = cap.read()
-    frame_gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
-
-    # calculate optical flow
-    p1, st, err = cv.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
-
-    # Select good points
-    good_new = p1[st==1]
-    good_old = p0[st==1]
-
-    # draw the tracks
-    for i,(new,old) in enumerate(zip(good_new,good_old)):
-        a,b = new.ravel()
-        c,d = old.ravel()
-        mask = cv.line(mask, (a,b),(c,d), color[i].tolist(), 2)
-        frame = cv.circle(frame,(a,b),5,color[i].tolist(),-1)
-    img = cv.add(frame,mask)
-
-    cv.imshow('frame',img)
-    k = cv.waitKey(30) & 0xff
-    if k == 27:
-        break
-
-    # Now update the previous frame and previous points
-    old_gray = frame_gray.copy()
-    p0 = good_new.reshape(-1,1,2)
-
-cv.destroyAllWindows()
-cap.release()
-@endcode
-(This code doesn't check how correct are the next keypoints. So even if any feature point disappears
-in image, there is a chance that optical flow finds the next point which may look close to it. So
-actually for a robust tracking, corner points should be detected in particular intervals. OpenCV
-samples comes up with such a sample which finds the feature points at every 5 frames. It also run a
-backward-check of the optical flow points got to select only good ones. Check
-samples/python/lk_track.py).
-
-See the results we got:
-
-![image](images/opticalflow_lk.jpg)
-
-Dense Optical Flow in OpenCV
-----------------------------
-
-Lucas-Kanade method computes optical flow for a sparse feature set (in our example, corners detected
-using Shi-Tomasi algorithm). OpenCV provides another algorithm to find the dense optical flow. It
-computes the optical flow for all the points in the frame. It is based on Gunner Farneback's
-algorithm which is explained in "Two-Frame Motion Estimation Based on Polynomial Expansion" by
-Gunner Farneback in 2003.
-
-Below sample shows how to find the dense optical flow using above algorithm. We get a 2-channel
-array with optical flow vectors, \f$(u,v)\f$. We find their magnitude and direction. We color code the
-result for better visualization. Direction corresponds to Hue value of the image. Magnitude
-corresponds to Value plane. See the code below:
-@code{.py}
-import cv2 as cv
-import numpy as np
-cap = cv.VideoCapture("vtest.avi")
-
-ret, frame1 = cap.read()
-prvs = cv.cvtColor(frame1,cv.COLOR_BGR2GRAY)
-hsv = np.zeros_like(frame1)
-hsv[...,1] = 255
-
-while(1):
-    ret, frame2 = cap.read()
-    next = cv.cvtColor(frame2,cv.COLOR_BGR2GRAY)
-
-    flow = cv.calcOpticalFlowFarneback(prvs,next, None, 0.5, 3, 15, 3, 5, 1.2, 0)
-
-    mag, ang = cv.cartToPolar(flow[...,0], flow[...,1])
-    hsv[...,0] = ang*180/np.pi/2
-    hsv[...,2] = cv.normalize(mag,None,0,255,cv.NORM_MINMAX)
-    bgr = cv.cvtColor(hsv,cv.COLOR_HSV2BGR)
-
-    cv.imshow('frame2',bgr)
-    k = cv.waitKey(30) & 0xff
-    if k == 27:
-        break
-    elif k == ord('s'):
-        cv.imwrite('opticalfb.png',frame2)
-        cv.imwrite('opticalhsv.png',bgr)
-    prvs = next
-
-cap.release()
-cv.destroyAllWindows()
-@endcode
-See the result below:
-
-![image](images/opticalfb.jpg)
-
-OpenCV comes with a more advanced sample on dense optical flow, please see
-samples/python/opt_flow.py.
-
-Additional Resources
---------------------
-
-Exercises
----------
-
--#  Check the code in samples/python/lk_track.py. Try to understand the code.
-2.  Check the code in samples/python/opt_flow.py. Try to understand the code.
+Tutorial content has been moved: @ref tutorial_optical_flow

doc/py_tutorials/py_video/py_meanshift/py_meanshift.markdown

@@ -1,185 +1,4 @@
 Meanshift and Camshift {#tutorial_py_meanshift}
 ======================
 
-Goal
-----
-
-In this chapter,
-
--   We will learn about Meanshift and Camshift algorithms to find and track objects in videos.
-
-Meanshift
----------
-
-The intuition behind the meanshift is simple. Consider you have a set of points. (It can be a pixel
-distribution like histogram backprojection). You are given a small window ( may be a circle) and you
-have to move that window to the area of maximum pixel density (or maximum number of points). It is
-illustrated in the simple image given below:
-
-![image](images/meanshift_basics.jpg)
-
-The initial window is shown in blue circle with the name "C1". Its original center is marked in blue
-rectangle, named "C1_o". But if you find the centroid of the points inside that window, you will
-get the point "C1_r" (marked in small blue circle) which is the real centroid of window. Surely
-they don't match. So move your window such that circle of the new window matches with previous
-centroid. Again find the new centroid. Most probably, it won't match. So move it again, and continue
-the iterations such that center of window and its centroid falls on the same location (or with a
-small desired error). So finally what you obtain is a window with maximum pixel distribution. It is
-marked with green circle, named "C2". As you can see in image, it has maximum number of points. The
-whole process is demonstrated on a static image below:
-
-![image](images/meanshift_face.gif)
-
-So we normally pass the histogram backprojected image and initial target location. When the object
-moves, obviously the movement is reflected in histogram backprojected image. As a result, meanshift
-algorithm moves our window to the new location with maximum density.
-
-### Meanshift in OpenCV
-
-To use meanshift in OpenCV, first we need to setup the target, find its histogram so that we can
-backproject the target on each frame for calculation of meanshift. We also need to provide initial
-location of window. For histogram, only Hue is considered here. Also, to avoid false values due to
-low light, low light values are discarded using **cv.inRange()** function.
-@code{.py}
-import numpy as np
-import cv2 as cv
-
-cap = cv.VideoCapture('slow.flv')
-
-# take first frame of the video
-ret,frame = cap.read()
-
-# setup initial location of window
-r,h,c,w = 250,90,400,125  # simply hardcoded the values
-track_window = (c,r,w,h)
-
-# set up the ROI for tracking
-roi = frame[r:r+h, c:c+w]
-hsv_roi =  cv.cvtColor(roi, cv.COLOR_BGR2HSV)
-mask = cv.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
-roi_hist = cv.calcHist([hsv_roi],[0],mask,[180],[0,180])
-cv.normalize(roi_hist,roi_hist,0,255,cv.NORM_MINMAX)
-
-# Setup the termination criteria, either 10 iteration or move by atleast 1 pt
-term_crit = ( cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 1 )
-
-while(1):
-    ret ,frame = cap.read()
-
-    if ret == True:
-        hsv = cv.cvtColor(frame, cv.COLOR_BGR2HSV)
-        dst = cv.calcBackProject([hsv],[0],roi_hist,[0,180],1)
-
-        # apply meanshift to get the new location
-        ret, track_window = cv.meanShift(dst, track_window, term_crit)
-
-        # Draw it on image
-        x,y,w,h = track_window
-        img2 = cv.rectangle(frame, (x,y), (x+w,y+h), 255,2)
-        cv.imshow('img2',img2)
-
-        k = cv.waitKey(60) & 0xff
-        if k == 27:
-            break
-        else:
-            cv.imwrite(chr(k)+".jpg",img2)
-
-    else:
-        break
-
-cv.destroyAllWindows()
-cap.release()
-@endcode
-Three frames in a video I used is given below:
-
-![image](images/meanshift_result.jpg)
-
-Camshift
---------
-
-Did you closely watch the last result? There is a problem. Our window always has the same size when
-car is farther away and it is very close to camera. That is not good. We need to adapt the window
-size with size and rotation of the target. Once again, the solution came from "OpenCV Labs" and it
-is called CAMshift (Continuously Adaptive Meanshift) published by Gary Bradsky in his paper
-"Computer Vision Face Tracking for Use in a Perceptual User Interface" in 1998.
-
-It applies meanshift first. Once meanshift converges, it updates the size of the window as,
-\f$s = 2 \times \sqrt{\frac{M_{00}}{256}}\f$. It also calculates the orientation of best fitting ellipse
-to it. Again it applies the meanshift with new scaled search window and previous window location.
-The process is continued until required accuracy is met.
-
-![image](images/camshift_face.gif)
-
-### Camshift in OpenCV
-
-It is almost same as meanshift, but it returns a rotated rectangle (that is our result) and box
-parameters (used to be passed as search window in next iteration). See the code below:
-@code{.py}
-import numpy as np
-import cv2 as cv
-
-cap = cv.VideoCapture('slow.flv')
-
-# take first frame of the video
-ret,frame = cap.read()
-
-# setup initial location of window
-r,h,c,w = 250,90,400,125  # simply hardcoded the values
-track_window = (c,r,w,h)
-
-# set up the ROI for tracking
-roi = frame[r:r+h, c:c+w]
-hsv_roi =  cv.cvtColor(roi, cv.COLOR_BGR2HSV)
-mask = cv.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))
-roi_hist = cv.calcHist([hsv_roi],[0],mask,[180],[0,180])
-cv.normalize(roi_hist,roi_hist,0,255,cv.NORM_MINMAX)
-
-# Setup the termination criteria, either 10 iteration or move by atleast 1 pt
-term_crit = ( cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 1 )
-
-while(1):
-    ret ,frame = cap.read()
-
-    if ret == True:
-        hsv = cv.cvtColor(frame, cv.COLOR_BGR2HSV)
-        dst = cv.calcBackProject([hsv],[0],roi_hist,[0,180],1)
-
-        # apply meanshift to get the new location
-        ret, track_window = cv.CamShift(dst, track_window, term_crit)
-
-        # Draw it on image
-        pts = cv.boxPoints(ret)
-        pts = np.int0(pts)
-        img2 = cv.polylines(frame,[pts],True, 255,2)
-        cv.imshow('img2',img2)
-
-        k = cv.waitKey(60) & 0xff
-        if k == 27:
-            break
-        else:
-            cv.imwrite(chr(k)+".jpg",img2)
-
-    else:
-        break
-
-cv.destroyAllWindows()
-cap.release()
-@endcode
-Three frames of the result is shown below:
-
-![image](images/camshift_result.jpg)
-
-Additional Resources
---------------------
-
--#  French Wikipedia page on [Camshift](http://fr.wikipedia.org/wiki/Camshift). (The two animations
-    are taken from here)
-2.  Bradski, G.R., "Real time face and object tracking as a component of a perceptual user
-    interface," Applications of Computer Vision, 1998. WACV '98. Proceedings., Fourth IEEE Workshop
-    on , vol., no., pp.214,219, 19-21 Oct 1998
-
-Exercises
----------
-
--#  OpenCV comes with a Python sample on interactive demo of camshift. Use it, hack it, understand
-    it.
+Tutorial content has been moved: @ref tutorial_meanshift

doc/py_tutorials/py_video/py_table_of_contents_video.markdown

@@ -1,13 +1,13 @@
 Video Analysis {#tutorial_py_table_of_contents_video}
 ==============
 
--   @subpage tutorial_py_meanshift
+-   @ref tutorial_meanshift
 
     We have already seen
     an example of color-based tracking. It is simpler. This time, we see significantly better
     algorithms like "Meanshift", and its upgraded version, "Camshift" to find and track them.
 
--   @subpage tutorial_py_lucas_kanade
+-   @ref tutorial_optical_flow
 
     Now let's discuss an important concept, "Optical Flow", which is related to videos and has many applications.