lecture 4: 3d reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · from image plane...
TRANSCRIPT
![Page 1: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/1.jpg)
Lecture4:3DReconstruction
CSE152:ComputerVisionManmohanChandraker
![Page 2: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/2.jpg)
Quiz
![Page 3: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/3.jpg)
Gradients1(a) ForimageI,areIx andIy smallorlargeatpointsA,B,CandD?
x
y
A B
C
D
![Page 4: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/4.jpg)
Corners2(a) Isthissecondmomentmatrixlikelytorepresentacorner?Why?
2(b) Isthissecondmomentmatrixlikelytorepresentacorner?Why?
2(c) Isthissecondmomentmatrixlikelytorepresentacorner?Why?
![Page 5: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/5.jpg)
SIFT3(a)WhatareinvariancesencodedbytheSIFTfeature?
3(b) Youobtained10truepositives,5falsepositivesand5falsenegativesataparticularSIFTmatchingthreshold.
Howwouldyouchangethethresholdto:(i) Increasetruepositives?(ii) Decreasefalsepositives?(iii) Decreasefalsenegatives?
![Page 6: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/6.jpg)
Recap
![Page 7: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/7.jpg)
Basicidea:• Take16x16squarewindowarounddetectedfeature• Computeedgeorientation (angleofthegradient- 90°)foreachpixel• Throwoutweakedges(threshold gradientmagnitude)• Createhistogramofsurviving edgeorientations
ScaleInvariantFeatureTransform
Adaptedfromslide byDavidLowe
0 2p
angle histogram
![Page 8: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/8.jpg)
Modelingprojection
• Thecoordinatesystem– Wewillusethepinholemodelasanapproximation– Puttheopticalcenter(CenterOfProjection)attheorigin– Puttheimageplane(ProjectionPlane)infrontoftheCOP– Thecameralooksdownthenegativezaxis.
![Page 9: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/9.jpg)
Modelingprojection
• Projectionequations– ComputeintersectionwithPPofrayfrom(x,y,z)toCOP– Derivedusingsimilartriangles
– Wegettheprojectionbythrowingoutthelastcoordinate:
![Page 10: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/10.jpg)
Homogeneous coordinatesIs this a linear transformation?
Trick: add one more coordinate:
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
• No, since division by z is nonlinear
![Page 11: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/11.jpg)
Modelingprojection
• Amatrixmultiplicationusinghomogeneouscoordinates
![Page 12: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/12.jpg)
Perspectiveprojection
(intrinsics) (convertsfrom3Draystopointsonimageplane)
Projection
dd
![Page 13: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/13.jpg)
Fromimageplanetopixelcoordinates
(intrinsics)Projection
(0,0)
(-cx,-cy)
dd
d
(convertsfrom3Draystopointsonimageplane)
![Page 14: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/14.jpg)
Fromimageplanetopixelcoordinates
(intrinsics)
Ingeneral,
:aspectratio(1unlesspixelsarenotsquare)
:skew (0unlesspixelsareshapedlikerhombi/parallelograms)
:principalpoint ((0,0) unlessopticalaxisdoesn’t intersectprojectionplaneatorigin)
(upper triangularmatrix)
(convertsfrom3Draysincameracoordinatesystemtopixelcoordinates)
Projection
dd
dd
![Page 15: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/15.jpg)
Cameraparameters
• Howcanwemathematicallydescribeacamera?
• Weneedtodescribeitsinternalparameters• Weneedtodescribeitsposeintheworld
![Page 16: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/16.jpg)
ATaleofTwoCoordinateSystems
“TheWorld”
Camera
x
y
z
v
w
u
o
COP
Twoimportantcoordinatesystems:1.Worldcoordinatesystem2.Cameracoordinatesystem
![Page 17: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/17.jpg)
Cameraparameters• Toprojectapoint(x,y,z)inworld coordinatesintoacamera
• Firsttransform(x,y,z)intocamera coordinates• Needtoknow
– Cameraposition(inworldcoordinates)– Cameraorientation(inworldcoordinates)
• Thenprojectintotheimageplane– Needtoknowcameraintrinsics
• Thesecanallbedescribedwithmatrices.
![Page 18: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/18.jpg)
Extrinsics• Howdowegetthecamerato“canonicalform”?
– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)
0
Step1:Translateby-c
![Page 19: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/19.jpg)
Extrinsics• Howdowegetthecamerato“canonicalform”?
– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)
0
Step1:Translateby-c
Howdowerepresenttranslationasamatrixmultiplication?
![Page 20: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/20.jpg)
Extrinsics• Howdowegetthecamerato“canonicalform”?
– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)
0
Step1:Translateby-cStep2:RotatebyR
3x3rotationmatrix
![Page 21: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/21.jpg)
Extrinsics• Howdowegetthecamerato“canonicalform”?
– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)
0
Step1:Translateby-cStep2:RotatebyR
![Page 22: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/22.jpg)
Projectionmatrix
translationrotationprojectionintrinsics
Denotethisbyt.
![Page 23: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/23.jpg)
Projectionmatrix
0
=
(inhomogeneous imagecoordinates)
![Page 24: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/24.jpg)
StructurefromMotion(SFM)VisualSLAM
![Page 25: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/25.jpg)
FeaturedetectionSeveralimagesobserveascenefromdifferentviewpoints
![Page 26: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/26.jpg)
FeaturedetectionDetectfeaturesusing,forexample,SIFT[Lowe,IJCV2004]
![Page 27: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/27.jpg)
FeaturematchingMatchfeaturesbetweeneachpairofimages
![Page 28: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/28.jpg)
Structurefrommotion
Camera1
Camera2
Camera3R1,t1
R2,t2R3,t3
X1
X4X3
X2
X5X6
X7
minimizeg(R,T,X)
p1,1p1,2
p1,3
non-linearleastsquares
![Page 29: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/29.jpg)
Bundleadjustment• Minimizesumofsquaredreprojection errors:
– Optimizedwithnon-linearleastsquares– Levenberg-Marquardtisapopularchoice
• Practicalchallenges?– Initialization– Outliers
predictedimagelocation
observedimagelocationindicatorvariable:
whetherpoint i visibleinimagej
![Page 30: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/30.jpg)
Featurematching
![Page 31: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/31.jpg)
Robustness• Let’sconsiderasimplerexample…linefitting
• Howcanwefixthis?Problem:Fitalinetothesedatapoints Leastsquaresfit
![Page 32: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/32.jpg)
Idea
• Givenahypothesizedline• Countthenumberofpointsthat“agree” withtheline– “Agree” =withinasmalldistanceoftheline– Thesearetheinlierstothatline
• Forallpossiblelines,selecttheonewiththelargestnumberofinliers
![Page 33: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/33.jpg)
Countinginliers
![Page 34: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/34.jpg)
Countinginliers
Inliers:3
![Page 35: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/35.jpg)
Countinginliers
Inliers:20
![Page 36: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/36.jpg)
Howdowefindthebestline?
• Unlikeleast-squares,nosimpleclosed-formsolution
• Hypothesize-and-test– Tryoutmanylines,keepthebestone– RANSAC:RandomSampleConsensus
• Numberofsamplesdependson– Outlierratio– Probabilityofcorrectanswer– Modelsize
![Page 37: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/37.jpg)
RANSAC
• Generalversion:1. Randomlychooses samples
• Typicallys =minimumsamplesizetofitamodel
2. Fitamodel(say,line)tothosesamples3. Countthenumberofinliersthatapproximatelyfit
themodel4. RepeatN times5. Choosethemodelwiththelargestsetofinliers
![Page 38: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/38.jpg)
Two-ViewReconstruction
![Page 39: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/39.jpg)
Image1
Image2
R1,t1
R2,t2
X
FundamentalMatrix
x1x2
xT1Fx2 = 0
x1↔ x2
![Page 40: Lecture 4: 3D Reconstructioncseweb.ucsd.edu/.../lec04_twoviewreconstruction.pdf · From image plane to pixel coordinates (intrinsics) In general, : aspect ratio (1 unless pixels are](https://reader035.vdocuments.mx/reader035/viewer/2022080720/5f7a2394e43ba32a24565fd2/html5/thumbnails/40.jpg)
RANSACtoEstimateFundamentalMatrix
• Formanytimes– Pick8points– Computeasolutionforusingthese8points– Countnumberofinlierswithcloseto0
• Picktheonewiththelargestnumberofinliers
FxT1Fx2