Featuredescriptionandmatching

MatchingfeaturepointsWeknowhowtodetectgoodpointsNextquestion:Howtomatchthem?

Twointerrelatedquestions:1. Howdowedescribeeachfeaturepoint?2. Howdowematchdescriptions?

?

Featuredescriptor

𝑥" 𝑥# 𝑦" 𝑦#

Featurematching

• Measurethedistancebetween(orsimilaritybetween)everypairofdescriptors

𝒚𝟏 𝒚𝟐

𝒙𝟏 𝑑(𝑥", 𝑦") 𝑑(𝑥", 𝑦#)

𝒙𝟐 𝑑(𝑥#, 𝑦") 𝑑(𝑥#, 𝑦#)

Invariancevs.discriminability

• Invariance:– Distancebetweendescriptorsshouldbesmallevenifimageistransformed

• Discriminability:– Descriptorshouldbehighlyuniqueforeachpoint(farawayfromotherpointsintheimage)

Imagetransformations• Geometric

Rotation

Scale

• PhotometricIntensitychange

Invariance

• Mostfeaturedescriptorsaredesignedtobeinvariantto– Translation,2Drotation,scale

• Theycanusuallyalsohandle– Limited3Drotations(SIFTworksuptoabout60degrees)– Limitedaffinetransformations(somearefullyaffineinvariant)– Limitedillumination/contrastchanges

Howtoachieveinvariance

Designaninvariantfeaturedescriptor– Simplestdescriptor:asingle0

• What’sthisinvariantto?• Isthisdiscriminative?

– Nextsimplestdescriptor:asinglepixel• What’sthisinvariantto?• Isthisdiscriminative?

Theapertureproblem

Theapertureproblem

• Useawholepatchinsteadofapixel?

SSD

• Useasdescriptorthewholepatch• Matchdescriptorsusingeuclidean distance• 𝑑 𝑥, 𝑦 = ||𝑥 − 𝑦||#

SSD

SSD

NCC- NormalizedCrossCorrelation

• Lightingandcolorchangepixelintensities• Example:increasebrightness/contrast• 𝐼2 = 𝛼𝐼 + 𝛽• Subtractpatchmean:invarianceto𝛽• Dividebynormofvector:invarianceto𝛼• 𝑥′ = 𝑥−< 𝑥 >

• 𝑥′′ = 92||92||

• similarity=𝑥22 ⋅ 𝑦′′

NCC- Normalizedcrosscorrelation

Basiccorrespondence

• Imagepatchasdescriptor,NCCassimilarity• Invariantto?

– Photometrictransformations?– Translation?– Rotation?

• Finddominantorientationoftheimagepatch– Thisisgivenbyxmax,theeigenvectorofM correspondingtolmax (the

larger eigenvalue)– Rotatethepatchaccordingtothisangle

Rotationinvarianceforfeaturedescriptors

FigurebyMatthewBrown

Take40x40squarewindowarounddetectedfeature– Scaleto1/5size(using

prefiltering)– Rotatetohorizontal– Sample8x8squarewindow

centeredatfeature– Intensitynormalizethe

windowbysubtractingthemean,dividingbythestandarddeviationinthewindow

CSE576:ComputerVision

MultiscaleOrientedPatcheS descriptor

8 pixels

AdaptedfromslidebyMatthewBrown

Detectionsatmultiplescales

• Finddominantorientationoftheimagepatch– Thisisgivenbyxmax,theeigenvectorofM correspondingtolmax (the

larger eigenvalue)– Rotatethepatchaccordingtothisangle

Rotationinvarianceforfeaturedescriptors

FigurebyMatthewBrown

Detour:Imagetransformations

• Whatdoesitmeantorotateapatch?• Eachpixelhascoordinates(x,y)• RotationrepresentedbyamatrixR• Pixel’snewcoordinates:

• I’(x’,y’)=I(x,y)

x0

y0

�= R

xy

�

Detour:Imagetransformations

• Whatifdestinationpixelisfractional?• Flipcomputation:foreverydestinationpixelfigureoutsourcepixel– Useinterpolationifsourcelocationisfractional

• I’(x’,y’)=I(x,y)

xy

�= R�1

x0

y0

�

Take40x40squarewindowarounddetectedfeature– Scaleto1/5size(using

prefiltering)– Rotatetohorizontal– Sample8x8squarewindow

centeredatfeature– Intensitynormalizethe

windowbysubtractingthemean,dividingbythestandarddeviationinthewindow

CSE576:ComputerVision

MultiscaleOrientedPatcheS descriptor

8 pixels

AdaptedfromslidebyMatthewBrown

MOPSdescriptor

x

y

•Youcancombinetransformationstogethertogetthefinaltransformation

T=?

x

y

Translate

T=MT1

x

y

Rotate

T=MRMT1

x

y

Scale

T=MSMRMT1

x

y

Translate

T=MT2MSMRMT1

x

y

Crop

Detectionsatmultiplescales

InvarianceofMOPS

• Intensity

• Scale

• Rotation

ColorandLighting

Out-of-planerotation

Out-of-planerotation

DepthDiscontinuity

Normaldiscontinuit

y

AlbedoEdge

Shadow

Betterrepresentationthancolor:Edges

Towardsabetterfeaturedescriptor

• Matchpatternofedges– Edgeorientation– cluetoshape

• Beresilienttosmalldeformations– Deformationsmightmovepixelsaround,butslightly

– Deformationsmightchangeedgeorientations,butslightly

Invariancetodeformationbyquantization

37 42

Between30and45

Invariancetodeformationbyquantization

g(✓) =

8>>>><

>>>>:

0 if 0 < ✓ < 2⇡/N1 if 2⇡/N < ✓ < 4⇡/N2 if 4⇡/N < ✓ < 6⇡/N

. . .N � 1 if 2(N � 1)⇡/N

Spatialinvariancebyhistograms

2blueballs,oneredbox

balls boxes

2

1

T.Tuytelaars,B.Leibe

RotationInvariancebyOrientationNormalization

• Computeorientationhistogram• Selectdominantorientation• Normalize:rotatetofixedorientation

0 2p

[Lowe,SIFT,1999]

TheSIFTdescriptor

• Computeedgemagnitudes+orientations• Quantizeorientations(invariancetodef)• Divideintospatialcells• Computeorientationhistogramineachcell(spatialinvariance)

DistinctiveImageFeaturesfromScale-InvariantKeypoints.Lowe.InIJCV2004

TheSIFTdescriptor

SIFT– LoweIJCV2004

Basicidea:• DoG forscale-spacefeaturedetection• Take16x16squarewindowarounddetectedfeature

• Computegradientorientationforeachpixel• Throwoutweakedges(thresholdgradientmagnitude)• Createhistogramofsurvivingedgeorientations

ScaleInvariantFeatureTransform

AdaptedfromslidebyDavidLowe

0 2p

angle histogram

SIFTdescriptorCreatehistogram

• Dividethe16x16windowintoa4x4gridofcells(2x2caseshownbelow)• Computeanorientationhistogramforeachcell• 16cells*8orientations=128dimensionaldescriptor

AdaptedfromslidebyDavidLowe

SIFTvectorformation• Computedonrotatedandscaledversionofwindowaccordingtocomputedorientation&scale– resamplethewindow

• BasedongradientsweightedbyaGaussian

Ensuresmoothness• Trilinear interpolation

– agivengradientcontributesto8bins:4inspacetimes2inorientation

Reduceeffectofillumination• 128-dimvectornormalizedto1• Thresholdgradientmagnitudestoavoidexcessiveinfluenceofhighgradients– afternormalization,clampgradients>0.2– renormalize

PropertiesofSIFTExtraordinarilyrobustmatchingtechnique

– Canhandlechangesinviewpoint• Uptoabout60degreeoutofplanerotation

– Canhandlesignificantchangesinillumination• Sometimesevendayvs.night(below)

– Fastandefficient—canruninrealtime– Lotsofcodeavailable:

http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

Summary• Keypoint detection:repeatableanddistinctive– Corners,blobs,stableregions– Harris,DoG

• Descriptors:robustandselective– spatialhistogramsoforientation– SIFTandvariantsaretypicallygoodforstitchingandrecognition

– But,neednotsticktoone

Whichfeaturesmatch?

Featurematching

GivenafeatureinI1,howtofindthebestmatchinI2?1. Definedistancefunctionthatcomparestwo

descriptors2. TestallthefeaturesinI2,findtheonewithmin

distance

FeaturedistanceHowtodefinethedifferencebetweentwofeaturesf1,f2?

– Simpleapproach:L2 distance,||f1 - f2 ||– cangivegoodscorestoambiguous(incorrect)matches

I1 I2

f1 f2

f1 f2f2'

FeaturedistanceHowtodefinethedifferencebetweentwofeaturesf1,f2?

• Betterapproach:ratiodistance=||f1 - f2 ||/||f1 - f2’||• f2 isbestSSDmatchtof1 inI2• f2’is2nd bestSSDmatchtof1 inI2• giveslargevaluesforambiguousmatches

I1 I2