by: navid einackchi

1

By: Navid EinackchiBy: Navid Einackchi

Isfahan University of Isfahan University of TechnologyTechnology

Electrical and Computer Department

Spring 2007Spring 2007

Image alignment and stitching using Object Recognition MethodsImage alignment and stitching using Object Recognition Methods

Supervisors: Dr Rasoul Amirfattahi

Dr javad Askari

Advisor: Dr M. Saraee

Master Thesis of Computer Engineering- Artificial Intelligence and Robotic

به نام خدا

By: Navid EinackchiBy: Navid Einackchi Spring 2007Spring 2007

Image alignment and stitching using object recognition methodsImage alignment and stitching using object recognition methods

Presentation ProcessPresentation Process

• IntroductionIntroduction– Definition– Aligning concepts– Stitching Concepts

• Object Recognition ProblemObject Recognition Problem– Salient Feature– What is Descriptor– Matching

• Proposed Method for AligningProposed Method for Aligning– Using Harris Detector– Using SIFT Detector– Stitching

• Concluding and SuggestionsConcluding and Suggestions

Some slides are fro

m other

Some slides are fro

m other

sources

sources




• IntroductionIntroduction– Definition– Aligning Concepts– Stitching Concepts






Image Alignment and StitchingImage Alignment and Stitching

• DefinitionDefinition • ApplicationsApplications

– Mosaic Image

– Panorama

– Virtual Environment

– Robotic

تصوير حاصل




• DefinitionDefinition • ApplicationsApplications

– Mosaic Image

– Panorama


– Robotic




• DefinitionDefinition• ApplicationsApplications

– Mosaic Image

– Panorama


– Robotic



Aligning ConceptsAligning Concepts

• Image TransformationsImage Transformations– Applicable Transformations

• Find transformationsFind transformations– How many parameters?

– Calculating Parameters



Image TransformationsImage Transformations

• TransitionTransition– Number of Parameters: 2

– Number of Points: 1

11

y

x

ihg

fed

cba

y

x




• EuclideanEuclidean– Number of Parameters: 3


11

y

x

ihg

fed

cba

y

x

cossin

sincosR




• SimilaritySimilarity– Number of Parameters: 4


11

y

x

ihg

fed

cba

y

x

cos.sin.

sin.cos.

ss

sssR




• AffineAffine– Number of Parameters: 6– Number of Points: 3

11

y

x

ihg

fed

cba

y

x




• PerspectivePerspective– Number of Parameters: 8– Number of Points: 4

11

y

x

ihg

fed

cba

y

x

1



Image Transformation ComputationImage Transformation Computation

• DirectDirect– Mapping one image into another using different parameters– Define an Error Function based on pixel intensity difference

• Using Corresponding pointsUsing Corresponding points– Obtaining Corresponding points– Computing Transformation parameters using corresponding points



Direct MethodDirect Method

Error Function




Error Function

؟؟



Using Corresponding pointsUsing Corresponding points



• How many points?How many points?

• How to select?How to select?– Using human

– Automatically

Using Corresponding pointsUsing Corresponding points




•IntroductionIntroduction–Definition

–Aligning Concepts–Stitching Concepts

•Object Recognition ProblemObject Recognition Problem–Salient Feature

–What is Descriptor–Matching

•Proposed Method for AligningProposed Method for Aligning–Using Harris Detector

–Using SIFT Detector–Stitching

•Concluding and SuggestionsConcluding and Suggestions



StitchingStitching

• StitchingStitching• ChoicesChoices

– Final Plane

– Pixel weighting



Object Recognition MethodsObject Recognition Methods

• Local Feature DetectionLocal Feature Detection– Edge– Corner– Hole

• Interest Regions MatchingInterest Regions Matching– Appearance Matching– Geometric Matching

• Finding Object Relations in ImageFinding Object Relations in Image



Salient FeaturesSalient Features

• Features of Feature!Features of Feature!– Invariants against transformations– Ability to explain the image– Ability to being Matched (Repeatability)

• Selected FeaturesSelected Features– Corners– Holes



Harris DetectorHarris Detector

PlainNo changes in every direction

EdgeHigh changes in Edge

Direction

CornerHigh changes in every

direction



Harris DetectorHarris Detector(Continued)(Continued)

2

2,

( , ) x x y

x y x y y

I I IM w x y

I I I

Slow change

direction

Fast change Direction

(max)-1/2

(min)-1/2




1

2

“Corner” 1, 2 both bigboth big

1 ~ 2

Edge1 >> 2

Edge2 >> 1

Plain




•Harris measurementHarris measurement

2det traceR M k M

1 2

1 2

det

trace

M

M




•Harris measurement responseHarris measurement response

1

2

“Corner”

“Edge”

“Edge”

“Plain”

R > 0

R < 0

R < 0|R| small

Local Maxim

um bigger than Threshold

Local Maxim

um bigger than Threshold




• FeaturesFeatures– Invariant against rotation

– Invariant against intensity

R

x

Threshold

R

x




•ExampleExample



• Scale ProblemScale Problem

Scale InvariantScale Invariant



Scale InvariantScale Invariant

Finding Appropriate Window Size

Finding Appropriate Window Size



Scale SelectionScale Selection

• Using Differential of Gaussian FiltersUsing Differential of Gaussian Filters– Laplacian of Gaussian (LoG)– Difference of Gaussian (DoG)

- =Window thst m

aximize fil

ter

Window thst m

aximize fil

ter



Automatic Scale SelectionAutomatic Scale Selection



SIFT Detector SIFT Detector

• Bulb DetectorBulb Detector• Invariant against scaleInvariant against scale• Using DoGUsing DoG

– Bulb detection

– Scale detection

مقیاس

x

y

DoG

DoG



SIFT DetectorSIFT Detector

• Detecting Bulb and Scale SimultaneouslyDetecting Bulb and Scale Simultaneously– Bulb position = maximum in plain

– Scale = maximum in third dimension

Bulb position

Scale position



DescriptorDescriptor

• Vector which describes region of interest pointVector which describes region of interest point– Pixels– Histogram– Differential

• Feature of DescriptorFeature of Descriptor– Invariant against image distortion (view, angle, intensity)– Able to express similarities and differences



SIFT DescriptorSIFT Descriptor

• Based on value and direction of gradientsBased on value and direction of gradients• 128 dimensional Vector128 dimensional Vector• ConstructionConstruction

1. Calculation of value and direction of gradient of each pixel

2. Making direction histogram of gradients

3. Rotating interest region based on dominant direction

4. Dividing the region into 16 region

5. Constructing direction histogram for each region

0



Construction ExampleConstruction Example



DescriptorDescriptor FeaturesFeatures

• Invariant againstInvariant against– Rotation– Position– Affine changes in intensity (I Ia + b)



MatchingMatching

• Appearance MatchingAppearance Matching– Comparing distance of each two descriptors– Euclidean Distance– K nearest neighbor

• Geometrical MatchingGeometrical Matching– Based on geometrical relations between points– Estimation of image transformation– Rejecting false corresponding points



Example of Geometrical MatchingExample of Geometrical Matching



Automatically image alignmentAutomatically image alignment

• Direct MethodDirect Method

• Based on interest pointsBased on interest points



CompareCompare

Interest pointsInterest points Direct MethodDirect Method

LowLow (Order of number of interest (Order of number of interest

points)points)

HighHigh((order of number of pixelsorder of number of pixels))

Time and Memory Time and Memory orderorder

HighHighLowLow

((Based on pixel intensityBased on pixel intensity))Resisting against Resisting against intensity changesintensity changes

YesYes NoNoAbility to recognize Ability to recognize

corresponding corresponding imageimage



Brown and Lowe AlgorithmBrown and Lowe Algorithm

O(nO(n44 m m44))



Brown and Lowe AlgorithmBrown and Lowe Algorithm



Similarity Transform rather than perspectiveSimilarity Transform rather than perspective

• WhyWhy– Detector are unable to handle perspective – Small portion of each images is overlapped– Increasing the tolerance of true corresponding

• So We use Similarity TransformationSo We use Similarity Transformation



First Proposed AlgorithmFirst Proposed Algorithm

1.1. Detecting Interest points using Harris detectorDetecting Interest points using Harris detector

2.2. Defining descriptors for each pointDefining descriptors for each point

3.3. Finding corresponding points using appearanceFinding corresponding points using appearance• Comparing interest points with each other

• K Nearest Neighbor

4.4. Finding relation between 2 imagesFinding relation between 2 images• Using Similarity

• Based on voting

5.5. Rejecting wrong corresponding points based on geometrical relationsRejecting wrong corresponding points based on geometrical relations



Voting MechanismVoting Mechanism



100

0cossin

0sincos

ss

ss

similarityT

Voting MechanismVoting Mechanism

Sc

ale

Sc

ale

Angle

O(nO(n22 m m22)) از مرتبهاز مرتبه

3469

2255



Applied Rotation

Estimated Rotation

Applied Scale

Estimated Scale

EvaluationEvaluation



ConcludingConcluding

• Very good for rotationVery good for rotation• Good for little scalingGood for little scaling• Weak for big scalingWeak for big scaling

– Due to Harris detector weakness








•Proposed Method for AligningProposed Method for Aligning–Using Harris Detector–Using SIFT Detector

–Stitching




ImprovementImprovement

• Using less time order algorithmUsing less time order algorithm

• Using Scale Invariant Feature TransformUsing Scale Invariant Feature Transform



Second proposed AlgorithmSecond proposed Algorithm

1.1. Detecting Interest pointsDetecting Interest points• SIFT Detector

2.2. Finding Corresponding pointsFinding Corresponding points• Based on appearance• Using k-d tree• Distance ratio threshold

3.3. Finding relation between imagesFinding relation between images• Voting on step 2 based on Appearance corresponding

4.4. Rejecting wrong corresponding pointsRejecting wrong corresponding points• Based on geometric relations between points



Distance Ration ThresholdDistance Ration Threshold

• Find 1NN and its distance = D1Find 1NN and its distance = D1• Find 2NN and its distance = D2Find 2NN and its distance = D2• Find ratio of these distances (D1/D2 = ratio)Find ratio of these distances (D1/D2 = ratio)

– If Lower than threshold -> reject

– Else accept



Distance Ration ThresholdDistance Ration ThresholdMy ImprovementMy Improvement

• Find 1NN and its distance = D1Find 1NN and its distance = D1• Multiply D1 by ratio = rangeMultiply D1 by ratio = range• Look for range in k-d treeLook for range in k-d tree



Results of threshold on ratioResults of threshold on ratio

• Threshold Ratio = 0.5Threshold Ratio = 0.5



Results of threshold on ratioResults of threshold on ratio

• Threshold Ratio = 0.7Threshold Ratio = 0.7



Second proposed AlgorithmSecond proposed Algorithm

1.1. Detecting Interest pointsDetecting Interest points• SIFT Detector

2.2. Finding Corresponding pointsFinding Corresponding points• Based on appearance• Using k-d tree• Distance ratio threshold

3.3. Finding relation between imagesFinding relation between images• Voting on step 2 based on Appearance corresponding

4.4. Rejecting wrong corresponding pointsRejecting wrong corresponding points• Based on geometric relations between points



• Each interest point hasEach interest point has– Dominant Direction

– Scale

• Use this information for votingUse this information for voting

Estimating Geometrical Estimating Geometrical RelationRelation



Estimation HistogramEstimation Histogram

Using threshold methodUsing 3NN method

Sca

leS

cale

Direction

Sca

leS

cale

Direction

O(nm)O(nm)

O(n)O(n)

Maximum Maximum



Percent of True Corresponding

True Corresponding Points

Potential Corresponding points

Method

487% 19 39 Distance Ratio t=0.5




0.06% 49 710 Distance Ratio t=1

0.07% 52 710*2→710 2 Nearest Neighbor



Comparing Knn and Ratio Threshold settingComparing Knn and Ratio Threshold settingBased on Number of true corresponding pointsBased on Number of true corresponding points



Comparing Knn and Ratio Threshold settingComparing Knn and Ratio Threshold settingBased on Number of true corresponding pointsBased on Number of true corresponding points

Percent of True Corresponding

True Corresponding Points

Potential Corresponding points

Method





0.06% 49 710 Distance Ratio t=1






Ratio Difference 2nd Maximum 1st Maximum Method

8.57 26.5 3.5 30 Distance Ratio t=0.5

7.75 27 4 31 Distance Ratio t=0.6

6.85 30.75 5.25 36 Distance Ratio t=0.7

1.95 46 48 94 2 Nearest Neighbor



Comparing Knn and Ratio Threshold settingComparing Knn and Ratio Threshold setting(In Tolerating False Corresponding)(In Tolerating False Corresponding)



Comparing Knn and Ratio Threshold settingComparing Knn and Ratio Threshold setting(In Tolerating False Corresponding)(In Tolerating False Corresponding)

Ratio Difference 2nd Maximum 1st Maximum Method

8.57 26.5 3.5 30 Distance Ratio t=0.5

7.75 27 4 31 Distance Ratio t=0.6

6.85 30.75 5.25 36 Distance Ratio t=0.7






Applied Rotation

Estimated Rotation

Applied Scale

Estimated Scale

EvaluationEvaluation



StitchingStitching

• Pixel Weights based on their distancePixel Weights based on their distance



StitchingStitching



Other examplesOther examples

• Adobe Photoshop could not stitch these imagesAdobe Photoshop could not stitch these images



Other ExamplesOther Examples



ConcludingConcluding

• IdeasIdeas– Using Harris and SIFT as a detector– Using Similarity transform as a general image transform– Using voting based on direction and scale of each interest points– Introducing Evaluation Method

• ResultsResults– Harris is more reliable but SIFT is much better in presence of scale transform– Similarity can handle prospective transform for human eyes– Voting can obtain similarity transform with much lower time order– Threshold ratio has a better behavior than Knn



ThanksThanksQuestions?Questions?

by: navid einackchi

Documents