introduction to machine vision
TRANSCRIPT
Zeeshan Zia | NED University of Engineering and Technology (2nd August, 2010)
Introduction to Machine Vision
Zeeshan Zia, M.Sc. (EE)[email protected], [email protected]
Zeeshan Zia | Introduction to Machine Vision
Objective
To give you lots of ideas… Give you a feel for extracting information from images…
Zeeshan Zia | Introduction to Machine Vision
Syllabus
Day 1:- Introduction, Demos (30 minutes)
- MATLAB Basics (30 minutes) - Image Processing Basics (30 minutes) - Project 1 : Content Based Image Retrieval (30 minutes)
Day 2: - Project 2: Depth from Stereo and Optical Flow (40 minutes) - Project 3: Segmentation (30 minutes) - Project 4: Object Recognition (40 minutes) - Conclusion and Research Paper Writing ( 20 minutes)
Zeeshan Zia | Introduction to Machine Vision
Machine Vision / Computer Vision / Image Understanding / Photogrammetry
Trying to make Computers extract some useful information from images…
Recognize faces, finger prints, other biometrics…
Detect anamolies in X-Rays, Ultrasounds, …
Point out suspicious human activity like fighting (surveillance)
Use ‚vision‘ as a sensor in Control Systems…
Make truly ‚autonomous‘ and ‚mobile‘ robots possible…
Sophisticated special effects in movies…
New and friendlier interfaces to Computers and other machines…
Zeeshan Zia | Introduction to Machine Vision
Applications
Medical Imaging People tracking Biometrics Vehicle automation (cars, airplanes) Plant automation 3d reconstruction Computer Graphics Biological and genetics research Showbiz and Augmented Reality City Modelling Lots of engineering applications…
Zeeshan Zia | Introduction to Machine Vision
Demo (Obama-Kanade)
Zeeshan Zia | Introduction to Machine Vision
Demo (Mount Everest – ETHZ)
Zeeshan Zia | Introduction to Machine Vision
Demo (Car and Pedestrian detection)
Zeeshan Zia | Introduction to Machine Vision
Demo (First-person action recognition)
My Master thesis – report available on my webpage…
Zeeshan Zia | Introduction to Machine Vision
Demo (Sport events analysis – TUM)
Zeeshan Zia | Introduction to Machine Vision
Demo (Gaze Tracking from outside)
Zeeshan Zia | Introduction to Machine Vision
Demo (Robotic Arm)
Zeeshan Zia | Introduction to Machine Vision
Demo (India Driving Challenge)
Zeeshan Zia | Introduction to Machine Vision
Demo (Cell Tracking)
Zeeshan Zia | Introduction to Machine Vision
MATLAB and Basic Image Manipulation
MATLAB is a programming language for technical computing.
MATLAB stands for MATrix LABoratory
Very easy to learn
Very complicated mathematical operations provided built-in
Zeeshan Zia | Introduction to Machine Vision
LTI/LSI Systems: Convolution
Continuous System : h(t) is the system‘s IMPULSE RESPONSE (1D case), x(t) is the input, and y(t) is the output.
Zeeshan Zia | Introduction to Machine Vision
Convolution : Discrete Case
Zeeshan Zia | Introduction to Machine Vision
Convolution : 2D discrete
Zeeshan Zia | Introduction to Machine Vision
Averaging Filter
1/9 1/9 1/9h = 1/9 1/9 1/9
1/9 1/9 1/9
MATLAB: new_image = imfilter(old_image,h,’conv’);
Zeeshan Zia | Introduction to Machine Vision
Horizontal Edge Detector
1 1 1h = 0 0 0
-1 -1 -1
Zeeshan Zia | Introduction to Machine Vision
Vertical Edge Detector
1 0 -1h = 1 0 -1
1 0 -1
Zeeshan Zia | Introduction to Machine Vision
Day 2
0. Brief Review1. Project 1: Content Based Image Retrieval (30 minutes)
2. Project 2: Depth from Stereo and Optical Flow (40 minutes) 3. Project 3: Segmentation (20 minutes) 4. Project 4: Object Recognition (20 minutes) 5. Conclusion and Research Paper Writing ( 20 minutes)
Projects, actually sub-fields / important problems of Computer Vision…people have spent their lives working on every one of these problems!
Zeeshan Zia | Introduction to Machine Vision
Review
Computer Vision : Trying to make Computers extract some useful information from images...similar terms are Machine Vision, Image Understanding, Photogrammetry
Inverse problem of Computer Graphics!
Zeeshan Zia | Introduction to Machine Vision
Review
Computer Vision : Trying to make Computers extract some useful information from images...similar terms are Machine Vision, Image Understanding, Photogrammetry
Inverse problem of Computer Graphics!Image Processing : Restoring, enhancing, compressing images
(no extraction of information from the image)
Computer Vision = Image Processing + Machine Learning + MathematicsComputer Vision = Image Processing + Machine Learning + Mathematics
Zeeshan Zia | Introduction to Machine Vision
Review
Computer Vision : Trying to make Computers extract some useful information from images...similar terms are Machine Vision, Image Understanding, Photogrammetry
Inverse problem of Computer Graphics!Image Processing : Restoring, enhancing, compressing images
(no extraction of information from the image)
Computer Vision = Image Processing + Machine Learning + MathematicsComputer Vision = Image Processing + Machine Learning + Mathematics
Applications areas:Although somewhat immature in the wild…Machine Vision can be very useful and accurate in controlled
environments like industrial plants or laboratories
Zeeshan Zia | Introduction to Machine Vision
References
MATLAB (easy to follow tutorial, takes 4-5 hours in total)
Digital Signal Processing Workbook of Electronics Department, NEDUET
Best (very easy to follow) textbook for Image Processing:
Digital Image Processing using MATLAB (Gonzalez and Woods)
For Computer Vision:
www.videolectures.net
Think of a problem, consult me (Google my name for my email address) on how much time it will take and which techniques are good candidates for trying out first, directly start working on the project!
Zeeshan Zia | Introduction to Machine Vision
Review
MATLAB is a programming language for technical computing.
Mathematical operations: sin(50 * pi/180), sqrt(9) etc..
Matrix operations, accessing matrix elements
Images in MATLAB represented as matrices
Color Channels Red, Green, and Blue represented as separate channels (accessed by third matrix element)
Basic Image Manipulation Commands
a = imread(‘myimage.jpg’); imshow(a);
Zeeshan Zia | Introduction to Machine Vision
Project 1: Content Based Image Retrieval
We implement this using only Intensity Histograms (for color images).
Many other techniques available which give better results.
A complete area of research…extends to finding movie snippets, or short description of clips by a few key-frames, or automatic story understanding.
Zeeshan Zia | Introduction to Machine Vision
Intensity Histograms
0
1000
2000
3000
4000
5000
6000
0 50 100 150 200 250
There can be other histograms also…(histogram of edges!)
Zeeshan Zia | Introduction to Machine Vision
Euclidean Distance
Remember the distance between two vectors in Cartesian coordinates?
A = [2,7,9]B = [1,6,4]
|AB| = ?
|AB| = square root of (2-1)2 + (7-6)2 + (9-4)2
Zeeshan Zia | Introduction to Machine Vision
Project 2: Depth from Stereo
Image 2
Image 1
W (X,Y,Z)
BaseLine distance
y
x
x
y
(x2,y2)
(x1,y1)
Optical Axis
Zeeshan Zia | Introduction to Machine Vision
Relating Depth to Image Coordinates
Zeeshan Zia | Introduction to Machine Vision
Relating Depth to Image Coordinates
Zeeshan Zia | Introduction to Machine Vision
By Similar Triangles
Zeeshan Zia | Introduction to Machine Vision
How does the computer know which point is which?
Zeeshan Zia | Introduction to Machine Vision
Steps
1. Find matching points on the two images.
2. Calculate disparity and visualize it.
Note: Unfortunately, depth can only be reconstructed upto a multiplying factor unless focal length of the camera is known (camera calibration, not considered here).
Zeeshan Zia | Introduction to Machine Vision
Step 1: Finding Correspondences
A common approach to finding correspondences is to search for local regions that appear similar
We try to match a window of pixels on the left image with a corresponding sized window on the right image, using CORRELATION.
Zeeshan Zia | Introduction to Machine Vision
Cross-Correlation as a distance measure
Remember Example 3.6-2 of Bruce Carlson???
Zeeshan Zia | Introduction to Machine Vision
Step 2: Disparity Map
Based on the calculated disparities a disparity map is obtained.
The disparity map is a gray scale map where the intensity represents depth.
The lighter shades (greater disparities) represent regions with less depth as opposed to the darker regions which are further away from us.
Zeeshan Zia | Introduction to Machine Vision
Examples
Matlab code…again much better code is possible (faster and accurate)…this is just for understanding the basic idea.
Zeeshan Zia | Introduction to Machine Vision
Optic Flow
Optic flow is the apparent visual motion that we experience as we move through the world.
Zeeshan Zia | Introduction to Machine Vision
Optic Flow
Optic flow is the apparent visual motion that we experience as we move through the world.
Very fast MATLAB Implementationof Depth from Stereo andOptic Flowavailable online.
Zeeshan Zia | Introduction to Machine Vision
Optic Flow – uses/assignments?
Write a program that uses your webcam to detect the number of people passing by the camera.
Write a program that performs visual surveillance of a road.
Place two cameras and a laptop on a toy cars, and use depth from stereo to automatically navigate the car to a final destination.
Bees use Optic Flow…they have eyes that have very poor resolution…because of this ‚poor resolution‘ (=less number of pixels / data)…they can process the data very fast, even though they have compound eyes. A lot of work is being done in this area….special ultra-small cameras have been developed that give only the optic flow. Ever killed a fly with a slow and steady hand???
Zeeshan Zia | Introduction to Machine Vision
Project 3: Segmentation
What do we mean by segmentation and why do we need it?
• Segmentation can roughly be described as grouping of similar information in an image.
• Instead of having to work with all the pixels, a segmentation allows us to work with a much more compact representation.
• Sometimes, we are interested in the segmentation itself.
- Especially in medical image analysis (e.g., segmenting out a tumor)
Zeeshan Zia | Introduction to Machine Vision
Segmentation : some examples
Zeeshan Zia | Introduction to Machine Vision
Figure-Ground Separation
Zeeshan Zia | Introduction to Machine Vision
Super-Pixels
Zeeshan Zia | Introduction to Machine Vision
Project 3: Segmentation by Clustering
One simple way of performing segmentation is to use clustering algorithms:
Zeeshan Zia | Introduction to Machine Vision
Project 3: Segmentation by Clustering
One simple way of performing segmentation is to use clustering algorithms:
• Clustering (a problem from machine learning) tries to group data points together. The points are usually vectors in some vector space.
Zeeshan Zia | Introduction to Machine Vision
Project 3: Segmentation by Clustering
One simple way of performing segmentation is to use clustering algorithms:
• Clustering (a problem from machine learning) tries to group data points together. The points are usually vectors in some vector space.
• We can apply this to the problem of segmentation by identifying each pixel with a feature vector and clustering these.
• This feature vector may include:
- The pixel’s position
- Pixel intensity or color
- etc…..a description of local texture
Zeeshan Zia | Introduction to Machine Vision
Clustering
Temperature
Hum
idity
Zeeshan Zia | Introduction to Machine Vision
Clustering
Temperature
Hum
idity
Zeeshan Zia | Introduction to Machine Vision
Clustering
Temperature
Hum
idity
m1
m2
m3
Zeeshan Zia | Introduction to Machine Vision
Clustering of pixels in image
Make one vector per pixel of the image,
Xi = [x, y, R, G, B]
Apply clustering on these vectors…call all pixels that end up in the same cluster as one segment!
WHY ?
Zeeshan Zia | Introduction to Machine Vision
K-Means Clustering Algorithm
Step 1: Determine number of clusters K
Step 2: Randomly choose K different mean vectors: m1, m2, …, mK
Step 3: Choose a data vector Xi, and calculate Euclidean distance between this vector and all the mean vectors one-by-one.
Step 4: Assign this data vector to the cluster with the minimum distance
Step 5: Set i to i + 1, and go back to step 3
Step 6: Calculate new mean vectors based on assignments
Step 7: Set i = 0, and go back to step 3
Zeeshan Zia | Introduction to Machine Vision
K-Means Clustering: Example
Zeeshan Zia | Introduction to Machine Vision
Discussion
Apply segmentation to optical flow, to find segments of high motion!!!
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
• Most active area of Computer Vision research
• Involves many many challenges – next slides…
• ‘Object’ here is taken in a very general sense, and includes everything from cars and chairs, to trees and animals or birds, or faces, or pedestrians…
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Challenges 4: Scale variation
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Challenges 6: Background Clutter
Find the dustbin…
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Single Object Recognition
A solved problem!
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Challenges 7: Intra-class variation
Zeeshan Zia | Introduction to Machine Vision
Project 4: Object Recognition
Challenges 7: Intra-class variation
My PhD focus is intra-class and viewpoint invariance –alongwith all the other problems..
Zeeshan Zia | Introduction to Machine Vision
Project 4 : SIFT Based Object Recognition
Zeeshan Zia | Introduction to Machine Vision
Scale Invariance
Zeeshan Zia | Introduction to Machine Vision
Rotational Invariance
Zeeshan Zia | Introduction to Machine Vision
Steps
TRAINING
Step 1: Take TRAINING IMAGE, identify strong scale and rotation invariance features.
Step 2: Form histograms for each feature (128 numbers), and store these histograms
DETECTION
Step 1: Take TEST IMAGE (unseen), obtain features from it, and form histograms.
Step 2: Match (euclidean distance) each feature of TEST IMAGE with that of TRAINING IMAGE. If a lot of features match, the object is present at the position of the matches!
Zeeshan Zia | Introduction to Machine Vision
Results
Optimized code available online, very easy to use.
Advantages:1. Scale and Rotation invariant2. Very robust for well textured objects.3. Real-time implementations
Disadvantages:1. Not viewpoint invariant.2. Not category invariant.
Zeeshan Zia | Introduction to Machine Vision
Results
Optimized code available online, very easy to use.
Advantages:1. Scale and Rotation invariant2. Very robust for well textured objects.3. Real-time implementations
Disadvantages:1. Not viewpoint invariant.2. Not category invariant.
Zeeshan Zia | Introduction to Machine Vision
Conclusions
MATLAB discussed For faster (and possibly real-time implementations) use C or C+
+. OpenCV library…
Delve right into projects instead of spending too much time on theory (advice for undergraduate students).
Feel free to contact me for project ideas, pointers…
Zeeshan Zia | Introduction to Machine Vision
Research Papers
How does science work? Scientists do experiments, publish their results in papers, others read those papers…
Who are scientists? Usually academic Professors doing research…
How is the performance of a scientist measured? By quantity and quality of her research papers…
How can you convince a foreign university professor to give you scholarships?
By showing potential for publishing research papers! IEEE Conferences, paper format, ieeexplore.ieee.org, scholar.google.com, Go home and Go home and search for the topics we covered today!search for the topics we covered today!