Lections on Image analysis, OpenCV

7. Working with camera.Background and motion analysis

USU / IMM Fall 2010www.uralvision.blogspot.com perevalovds@gmail.com

http://people.rit.edu/andpph/photofile-misc/strobe-motion-ta-08.jpg

Working with the camera

Getting frames from the cameraVideoCapture capture; // A class to work with cameras and videoscapture.open (0); // Open the first chamber (numbered from 0)// But if you specify a string VideoCapture capture.open ("myfile.avi"); - will read the avi-file

if (! capture.isOpened ()) { // If the camera does not complete the workcout <<"no camera" <<endl;return -1;}

Mat image, smoothed, edges;for (;;) {capture>> image; // get the frame

cvtColor (image, smoothed, CV_BGR2GRAY);GaussianBlur (smoothed, smoothed, Size (7,7), 1.5, 1.5);Canny (smoothed, edges, 0, 30, 3);imshow ("image", image);imshow ("edges", edges);if (waitKey (30)> = 0) break; // wait for keypress 30 ms, if pressed - exit}

// Destructor capture himself off the camerareturn 0;

Getting frames from the camera

Camera calibrationConsider a simple calibration when a camera looks at the side of the plane. (There are more sophisticated calibration methods, which allow to eliminate the distortion at the edges of images from the camera connected with the inaccuracy of optics).

Original image Image with the selected calibration points

(Angles of the red 4-gon)

Calibration results:image is aligned

Camera Calibration1. To calibrate the need to specify the coordinates of several points on the image and indicate to what point they should go.

Must be at least 4 points to search for perspective projection. In this case any point should not lie on one line.

Note: the coordinates are real numbers, which improves the accuracy of the result. You can use the sub-pixel methods that calculate the position of objects with an accuracy higher than pixel.

2. Then, using the getPerspectiveTransform calculate the optimal matrix of perspective transformation (a method of least squares type, which helps to use a lot of points taken from the error).

3. Finally, this matrix is passed to the functionwarpPerspective for calibrated images.

Camera Calibrationconst int K = 4, // we use 4 points.cv:: Point2f src [K]; // coordinates of points on the imagecv:: Point2f dst [K]; // coordinates of the resulting points

src [0] = cv:: Point2f (..., ...); // coordinates of the corners of the imagesrc [1] = cv:: Point2f (..., ...); // should go clockwise from top left cornersrc [2] = cv:: Point2f (..., ...); // - in accordance with dst, see belowsrc [3] = cv:: Point2f (..., ...);

dst [0] = cv:: Point2f (0, 0) // the resulting point; w and h- The size of the resultdst [1] = cv:: Point2f (w, 0);dst [2] = cv:: Point2f (w, h);dst [3] = cv:: Point2f (0, h);

Mat transform = getPerspectiveTransform(Src, dst); // compute the matrix transforms.

warpPerspective(Image, imageResult, transform, cv:: Size (w, h), INTER_LINEAR);// Get from the input image image result - imageResult

Methods for analysis of background

Methods for analysis of background

Problem - to calculate the pixels are the background and the background.

1. Learning the background

Idea: store the image background and then consider the difference between shots with the camera from the stored images.

A more advanced method - "code book", he remembers the brightness of the background in each pixel for a while, that allows to deal with objects such as opening doors.

There are also more complex algorithms.

Problems:- Rapid changes in lighting - should be specificallyconsider- The shadows of objects are often perceived asno background, which is usually undesirable, soalso be specially taken into account (through the color analysis).

2. Adaptive learning

Simple method, where the background is averaging of the last N frames

const float k = 0.01;back = (1-k) * back + k * image;a low pass filter

A few seconds later the object came into the picture becomes the background. For some tasks it is good.

Plus - no need to specifically construct a situation where there is no one in the frame for storing the background.

There are more sophisticated algorithms automatically calculate the background with the help of several accounting staff.

3.Using data on the depth

1. Stereo camera2. Camera with the calculation of flight time (based on a laser rangefinder, time of filght cameras)3. Kinect

Provide information on the distance to each pixel, allowing long-discarded objects by implementing a threshold depth of processing.

Motion Analysis

The difference between two frames -the algorithm "Motion Detector"1. Let image1, image2 - two consecutive frames from the camera, single-channel.2. Building a module of their differenceMat diff;absdiff (image1, image2, diff);3. Perform threshold processingMat bin;// Here we will find the pixels that correspond to areas of motionthreshold (diff, bin, 100 / * threshold * /, 255, CV_THRESH_BINARY);

The difference between two frames -the algorithm"Motion Detector"This method allows you to find the area to which there was a movement.

For the analysis directions movement of such objects - the notion of Optical Flow.

What is the optical flowOptical Flow(Optical flow, optic flow) - is a vector field of apparent motion of objects, surfaces and edges in a visual scene, caused by relative motion between the observer (eye camera) and the stage.

http://www.ultimategraphics.co.jp/jp/images/stories/ultimate/BCC/optical.jpg

Note: it seems that the point of the camera moves down, but maloteksturirovannoy of flow is not - as the local pixels in the neighborhood do not change

The main application of optical flow

1. To determine the direction in which the moving objects in the frame.

2. The production of films - for a smooth morphing between successive frames, or between the shots taken adjacent cells (the most characteristic is used in the movie "The Matrix").

3. In principle can be applied to stereo vision - to determine the distance to an object by analyzing the optical flow of personnel coming from the two chambers.

Methods of calculating the optical flow

Input: two frames. Output - vector field (fx (x, y), fy (x, y)) - Vector of geometric shift pixels from the first frame on the second.

(I) Block ("Naive" methods) For each point searched shift that minimizes the difference in the local window.

(II) Differential (Most used)Estimation of the derivatives of x, y, t. 1. Lucas-Kanade - Very fast.2.Farneback- Good enough, but slow3.CLG - Very high quality but not yet implemented in OpenCV4. Pyramidal Lucas-Kanade, calculated only on "points of interest"5. Horn-Schunk - not very resistant to noise

(III) based on discrete optimization (Intensive)The solution is constructed using the methods of min-cut, max-flow, linear programming and belief propagation.

Methods of calculating the optical flow

1. Lucas-KanadeLocal method, which uses the Taylor expansion in time t in the neighborhood of each pixel independently.Dignity: Quickly calculated.Shortcomings: "Aperture problem" - in areas with a uniform texture, works poorly due to the fact that it is local.Realized only in the C-version of OpenCV,cvCalcOpticalFlowLK.

Flow superimposed on the image.Note that the flow inside the palm is not found.

Flow drawn in increments of 10 pixels, the function line. (Note: The values of the flux smoothed over the neighborhood for a more stable result.)

Methods of calculating the optical flow

2. FarnebackLocal method used to calculate an approximation of the image by a polynomial function.

Dignity: Significantly more resistant to the problems of the aperture.

Shortcomings: Runs noticeably slower Lucas-Kanade.

Methods of calculating the optical flow

void calcOpticalFlowFarneback(const Mat & prevImg, // first frame, an 8-bit single channel imageconst Mat & nextImg, // second frame, the type and size as a prevImgMat & flow, // resulting flow will be of the type CV_32FC2double pyrScale, // <1, the scale of the construction of the pyramid.0.5int levels, // Number of levels of the pyramid5int winsize, // window averaging. The more - the result// More diffuse, but also more resistant to noise5int iterations, // Number of iterations at each level of the pyramid 3int polyN, // window size to calculate the approximation// Polynomial7double polySigma, // parameter is Gaussian for smoothing derivatives// In the construction of approximation1.5int flags // flags//OPTFLOW_USE_INITIAL_FLOW- Do not use!

//OPTFLOW_FARNEBACK_GAUSSIAN- Use a Gaussian for the averaging// Gives a more accurate result at the expense of quality.

)

Methods of calculating the optical flow

3. CLG

"Combining Local and Global"The method of combining Lucas-Kanade and Horn-Schunk

Now (2010), he and his modification is considered one of the best quality / speed.

Not yet implemented in OpenCV.

Difficult task (for exam):find a realization of CLG in C / C + +, compile, and run on a pair of test images.