March 28, 2014 Sam Siewert

Computer and Machine Vision

Lecture Week 11 Part-2

Segmentation, Camera Calibration and Feature Alignment

Outline of Week 11 Exam #1 Results Overview and Solutions Wrap up of SIFT and David Lowe’s Paper – Primitive Operations to Accelerate SIFT – Uses of SIFT and Comparison to Hough and

RANSAC Introduction to Segmentation, Camera Calibration and Feature Alignment

Sam Siewert 2

Recall Image Pyramid from Chapter 3 in CV

Page 133 – A fundamental for scale invariance (SIFT search for keypoint candidates in different scales) Resolution Decimation – Reduction in Resolution, or downscaling Resolution Interpolation – Increased Resolution, or upscaling (Can’t create data where there was none in original, but we can interpolate) OpenCV Image Pyramids – Low Pass Filtering (Gaussian kernel convolution) followed by pixel decimation (removal of odd or even numbered rows and columns)

Sam Siewert 3

L=0, e.g. 9x9 L=1, 5x5 L=2, 3x3 Rows 0…8, Col 0…8 Drop rows 1,3,5,7 to go to L=1 5x5 Drop rows 1,3 to go to L=2 3x3

Feature Vector Concept General Concept - Summarize Data with Fewer Data points E.g. an Intensity Histogram, Feature Vector for a Color Image

– If Histogram(Image_A) = Histogram(Image_B), Image_A = Image_B more likely – Much like a Fingerprint for Human Identification (not definitive, but narrows search) – The Intensity Histogram is Not a Robust Feature Vector (Lighting Sensitivity) – Increasing Dimensions, Normalization, Scale, Orientation and Quality Improves

Sam Siewert 4

R

G

B

R

G

B

Image Gradient Concept

Sam Siewert 5

Example Feature Vectors Any N dimensional vector that represents and object A form of compression for the reference database and faster compare Vector and Distance Metric Definitions are Key, But Once Defined, Simple Distance Measures Can Be Applied

– Corners of Objects – Radial Distance Profile

(Centroidal) – Various Histograms Derived

from Scans with Binning (Blobs)

Sam Siewert 6

Feature Vectors Unique Patterns That Define Objects of Interest How to Generalize Keypoints of Feature Vectors is the Challenge Bio-inspired (Human Observation) is A Promising Approach, but Search is Costly! Operation in Real-Time? PCA – Principal Component Analysis

Sam Siewert 7

http://en.wikipedia.org/wiki/File:GaussianScatterPCA.png

http://en.wikipedia.org/wiki/File:Sift_keypoints_filtering.jpg

N Dimensional Distance Harder to Visualize than 3D Vector Differences, but Same Concept as Euclidian Quick Review of Euclidian – One Dimension: abs(p1 – p2), or Sqrt((p1 – p2)2)

– Two Dimension: Sqrt((p1-q1)2 + (p2 – q2)2)

– Three Dimension: Sqrt((p1-q1)2 + (p2 – q2)2 + (p3-

q3)2) – N Dimension: Sqrt((p1-q1)2 + (p2 – q2)2 + … + (pn-

qn)2)

Sam Siewert 8

Other Distance Measures … Defines the Difference Between {A} and {B} in General with a Metric (Number) Euclidian Distance is One Such Metric Manhattan Distance is Another (Sum of Orthogonal Line Segments on a Grid Between 2 Points) Hamming Distance is Another - # of Positions at Which 2 Equal Length Strings Have Different Symbols – E.g. 1111_0000 and 0000_1111 have maximum distance or “toned” and “roses” has distance of 3 in ASCII

Sam Siewert 9

Final Words on SIFT Re-Read David Lowe’s Paper 1. Scale Space Extrema Detection - Find Best Candidate Image

Locations to Form Keypoint feature vectors (scale invariant through search in Gaussian pyramid)

2. Keypoint Localization - Filter Out Keypoints (to remove minimum contrast and noisy edge curvature feature vectors)

3. Keypoint Orientation – So Descriptors are relative to orientation of the keypoint (to provide orientation invariance) and for magnitude to select proper level in Gaussian image

4. Keypoint Descriptor – Representation in database is a feature vector with magnitude, orientation for nxn sub-regions with vectors of m dimensions (e.g. Figure 7 2x2 with 8x8 feature vector)

Notes from the Paper on the overall Algorithm follow … Sam Siewert 10

SIFT Interesting Aspects Feature Selection with Mathematical Significance Calculations Scale, Orientation and Lighting Invariance Goals Keypoints are Complex Feature Vectors to Use for Match Bio-Inspired (see page 14, local image descriptor) – Edelman, Intrator and Poggio on Neurons for 3D recognition (gradient at orientation with spatial frequency – key trigger in visual cortex)

Sam Siewert 11

Image Correspondence and Registration

Historically Features as Simple as Intensity and Color Vectors in an n x n ROI Have Been Used to Register Images Simpler Feature Vectors Work for Left-Eye, Right-Eye Images (Simple Baseline Separation of Known Distance) Still Complicated by Camera Calibration Sam Siewert 12

Hough and RANSAC Recall Hough – Idea is to Fit Threshold Points to Lines,

Circles/Elipses, Shape (RANSAC fits to Lines) – Feature Matching as Well

SIFT Results in Many More Keypoint feature vectors compared to Hough Advantage for Correspondence and Recognition (Like more Detailed Fingerprint) Likewise, Sometimes a Disadvantage – Large Search for Match Detection

Sam Siewert 13

Alternatives in Machine Vision

Shape and Pattern Recognition Alternatives to

Hough and SIFT Sam Siewert

14

Centroidal Profile Find Centroid Based on Threshold and Pixel COM (As Discussed Before) Define Shape In Terms of Radius as a Function of Angle Around Centroid (Shape Signature)

Sam Siewert 15

From Computer and Machine Vision, E.R. Davies, page 271.

Centroidal Profile Problems Notched or Partially Occluded Objects Computationally Complex Must Have Accurate Centroid First Suggested for Inspection Applications (Within Shape Tolerances?)

Sam Siewert 16

From Computer and Machine Vision, E.R. Davies, page 272.

Segmentation, Camera Calibration and Feature

Alignment

Sam Siewert

17

Segmentation (CV Chapter 5) Foreground and Background – E.g. Background Elimination (Simplify to 2 regions over time) – Simple Threshold (Simplify to 2 regions by intensity) – Edge Detection to Define Contour Boundaries (Gradient separation)

Clustering Methods – Use Distance Measures to Assign Pixels to a Center Value (could be

random color or intensity) – E.g. OpenCV K-means Clustering – Divide Image of N pixels into K clusters with Each Belonging to Cluster

with Nearest Mean

Watershed – Gradient Lines (Like Contour Map) Define Regions, Fill Between them Sam Siewert 18

http://en.wikipedia.org/wiki/K-means_clustering

Segmentation Use

Divides Image Logically Rather than Arbitrary Selection or ROI (Region of Interest) as a Grid Does Localized Feature Vectors in Each ROI Classification and Object Recognition Step Defines Features Rather than Looking for Feature Matches (E.g. Lines in Image like Hough) Sam Siewert 19

Cameras and Calibration

Basics for 2D and Considerations for 3D Mapping and Ranging

Sam Siewert

20

Camera Calibration Intrinsics – Features of the Camera Itself – Optical Characteristics (Fish

Eye – wider in center, Hourglass – thinner in center)

– Detector Characteristics – Pixels with Value at Zero Illumination (Christmas Light) or Pixels with Low Value, Pixel Type (Bayer Filter, or Other)

Extrinsics – Camera Mount, Thermal Issues, Vibration, Orientation and Viewpoint Sam Siewert 21

http://en.wikipedia.org/wiki/Bayer_filter

CV Ch. 6 & Learning OpenCV Ch. 11 Read to Understand Basic Concepts – Coordinate Transformation Between Pixel and Real

World Coordinates E.g. Tilt, Pan, Zoom of Camera and Location in X, Y, Z of target of Interest in Scene Why is this an issue and why is calibration required?

– Camera optical distortions – Lighting conditions – Thermal and Mechanical Perturbations (change in camera mount

over time) – Detector Manufacturing Imperfection (Same Camera Model,

Different Image from Same Scene)

– Radial Distortions

Practical Methods Provided in Learning OpenCV and Here Sam Siewert 22

Sam Siewert 23

Camera Mount Basic Concepts Single Camera Tilt/Pan Object Tracking (Common Configuraiton) – 2 Axes of Rotation (Others are Fixed) – +/- 45 deg Tilt Rotation Servo (typical, could be up to 90 deg) – +/- 45 deg Pan Rotation Servo (typical, could be up to 180 deg) – Side or Rear Mounting of Tilt Servo

Ideally No Vibration or Rotation, Just Commanded Tilt/Pan

Sam Siewert 24

Dual Camera Tilt/Pan Tracking Baseline with 2 Fixed Cameras (for Stereo Ranging) – Pan Servo Pans the Entire Baseline – Tilt Servo Tilts Pan Servo – Side Mounting Plate

Ideally Baseline Separation is Constant (no bending, twisting of the beam)

Sam Siewert 25

Pixel Coordinates Define Image Coordinates to Track Object Centroid Tilt/Pan With Servos to Keep Target Centroid in FOV Center E.g. ½ Resolution 320x240 AR=4:3 Image from Analog Camera

Reference Target Pixel Address

160 , 120

NTSC 320 Pixel Column Format

N T S

C 2

4 0 P

i x e l

R o w

F o

r m a t

Frame Origin Pixel Address 0 , 0

X

Y

Sam Siewert 26

Stereo Ranging with Common Tilt/Pan Fixed Camera Baseline

Sam Siewert 27

Pixel to Servo Calibration at Distance

Pan 1 Servo Increment and Find Centroid X Pixel Change Tilt 1 Servo Increment and Find Centroid Y Pixel Change

Reference Target Pixel Address

160 , 120

Tilt One Servo Increment

Pan One Servo Increment

NTSC 320 Pixel Column Format

N T S

C 2

4 0 P

i x e l

R o w

F o

r m a t

Tilt Shifted Frame

Pan Shifted Frame

Pan Shifted Target Pixel

Address 158 , 120

Tilt Shifted Target Pixel Address

160 , 122

Sam Siewert 28

Characterize Camera FOV at Distances FOV Width/Height Linear as a Function of Distance Pixels/Inch Not Necessarily Linear as Function of Distance Use to Calibrate Servo Step Size (Gain) for Target Centering

– For Servo Step Sizes of 1 Increment in Tilt/Pan, Camera Will Track Slowly – For Larger Servo Step Sizes May Over-shoot – Determine Dead-bands (Servo Limits for Pixel Change Accuracy)

Calibrate with Reference Image at Known Distance

Generalize and Expand Auto-Calibration Sequence as Described for N x M known Targets Most Common is the Chess/Checker Board OpenCV Support for Calibration Using Chess Board

Sam Siewert 29

Sam Siewert 30

Finding Centroid In Image Target Known by Color, Shape, or Brightness Raster to Find Target Edges and Max Width and Height Scanline for Symmetric targets Mark Centroid in Image for Easy Debug Noise in Image Will Cause Centroid Error – Use Averaging Pan Right to Move Target Left in Image Tilt Up to Move Target Down in Image

Sam Siewert 31

Tips for Practical Real-time Calibration Consider Controlled Lighting or For Mobile Robots, On-Board Lighting with LEDs (Control your Own Lighting) Use Frame Grabber ADC Sensitivity Settings to Control Brightness Consider Automatic Calibration Sequences – Place Reference Targets to Set Pixel Step Size as a Function of Servo

Step Size – Use Stereo Range Estimation to Determine Distance to Target and Set

Tracking Gains for Current Distance

When Target is Lost, Go Into Search Mode – Start from Max Tilt/Pan and Raster to Min Tilt/Pan to Find Target – For Search Modes Use Coarser Step Size

Consider Small Servo Step Size, But High Frame Rate and Servo Command Rate Be Careful of Processor Overload with Image Processing at 30 FPS

Feature Alignment

Mechanical vs. Digital Image Processing

Sam Siewert

32

Mosaics by Scan Mirror and Gimbal

Tilt, Pan, Rotate Gimbal to Stabilize Field of View Electro-mechanically (Works, but Expensive) – Steadicam (Automatic Gimbal to Hold FoV) – CCTV Security Cameras

Scan Mirror in Optical Path to Stabilize Images (Spitzer Space Telescope) – Scan Mirror Moves in Counter-veiling

Direction of Telescope Rotation (Ideally Single Axis)

– Works Because Perturbations in Deep Space are Minimal

Sam Siewert 33

http://en.wikipedia.org/wiki/Steadicam

Digital Stabilization Let Camera Vibrate, Tilt/Pan/Rotate on Non-Rigid Mounts – Digitally Stabilize – Find Successive Corresponding Feature

Keypoints in Images Over time – Re-align Center of Each Image – Crop Outermost Pixels

Common Use of SIFT and Other Feature Registration Methods (Hough, RANSAC, etc.)

Sam Siewert 34

Mosaics iOS and Android Panorama Feature – Successive Images are Taken – Correspondence Found at Keypoints – Stitch Images Together and Crop Overlaps

Alternative is Tilt/Pan with Gimbal and Scan Mirror – Tilt or Pan at Continuous Known Rate – Rotate Scan Mirror in Optical Path in Opposite Direction to

Stare at Target FoV – At Known Radial Distance Snap Back to Stabilize Next

Image – Same Concept as Staring at a Fixed Object from a Moving

Vehicle

Sam Siewert 35

Stereo Correspondence Use Similar Triangles To Estimate Distance – Right-Eye, Left-Eye Images with Known Intrinsics and Extrinsics – Keypoints Found in Both are Common Target Centroid – Planar? Out of Plane Causes Error – Vanishing Points

Need Numerous Common Targets to Form Passive Depth Map

Sam Siewert 36

Human eye concept from http://en.wikipedia.org/wiki/Binocular_disparity

The dl, dr, f must be known from intrinsic calibration The b is known extrinsic, and solve for d