putting objects in perspective
DESCRIPTION
Putting Objects in Perspective. Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Robotics Institute. Understanding an Image. Today: Local and Independent. What the Detector Sees. Local Object Detection. True Detection. False Detections. Missed. Missed. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/1.jpg)
Putting Objects in PerspectivePutting Objects in Perspective
Derek Hoiem
Alexei A. Efros
Martial Hebert
Carnegie Mellon University
Robotics Institute
![Page 2: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/2.jpg)
Understanding an ImageUnderstanding an Image
![Page 3: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/3.jpg)
Today: Local and IndependentToday: Local and Independent
![Page 4: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/4.jpg)
What the Detector SeesWhat the Detector Sees
![Page 5: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/5.jpg)
Local Object DetectionLocal Object Detection
True Detection
True Detections
MissedMissed
False Detections
Local Detector: [Dalal-Triggs 2005]
![Page 6: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/6.jpg)
Work in Context Work in Context
• Image understanding in the 70’sGuzman (SEE) 1968
Hansen & Riseman (VISIONS) 1978
Barrow & Tenenbaum 1978
Yakimovsky & Feldman 1973
Brooks (ACRONYM) 1979
Marr 1982
Ohta & Kanade 1973
• Recent work in 2D context
Kumar & Hebert 2005
Torralba, Murphy, Freeman 2004
Fink & Perona 2003
He, Zemel, Cerreira-Perpiñán 2004
Carbonetto, Freitas, Banard 2004
Winn & Shotton 2006
![Page 7: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/7.jpg)
Real Relationships are 3DReal Relationships are 3D
Close
Not Close
![Page 8: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/8.jpg)
Recent Work in 3DRecent Work in 3D
[Torralba, Murphy & Freeman 2003]
[Han & Zu 2003]
[Oliva & Torralba 2001]
[Han & Zu 2005]
![Page 9: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/9.jpg)
• Biederman’s Relations among Objects in a Well-Formed Scene (1981):
– Support
– Size
– Position
– Interposition
– Likelihood of Appearance
Objects and ScenesObjects and Scenes
Hock, Romanski, Galie, & Williams 1978
![Page 10: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/10.jpg)
• Biederman’s Relations among Objects in a Well-Formed Scene (1981):
Hock, Romanski, Galie, & Williams 1978
– Support
– Size
– Position
– Interposition
– Likelihood of Appearance
Contribution of this PaperContribution of this Paper
![Page 11: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/11.jpg)
Object SupportObject Support
![Page 12: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/12.jpg)
Surface EstimationSurface Estimation
Image Support Vertical Sky
V-Left V-Center V-Right V-Porous V-Solid
[Hoiem, Efros, Hebert ICCV 2005]
Software available online
ObjectSurface?
Support?
![Page 13: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/13.jpg)
Object Size in the ImageObject Size in the Image
Image World
![Page 14: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/14.jpg)
Input Image
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
Loose Viewpoint Prior
![Page 15: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/15.jpg)
Input Image
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
Loose Viewpoint Prior
![Page 16: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/16.jpg)
Object Position/Sizes Viewpoint
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
![Page 17: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/17.jpg)
Object Position/Sizes Viewpoint
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
![Page 18: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/18.jpg)
Object Position/Sizes Viewpoint
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
![Page 19: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/19.jpg)
Object Size ↔ Camera Viewpoint Object Size ↔ Camera Viewpoint
Object Position/Sizes Viewpoint
![Page 20: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/20.jpg)
What does surface and viewpoint say about objects?What does surface and viewpoint say about objects?
Image
P(object) P(object | surfaces)
P(surfaces) P(viewpoint)
P(object | viewpoint)
![Page 21: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/21.jpg)
Image
P(object | surfaces, viewpoint)
What does surface and viewpoint say about objects?What does surface and viewpoint say about objects?
P(object)
P(surfaces) P(viewpoint)
![Page 22: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/22.jpg)
Scene Parts Are All InterconnectedScene Parts Are All Interconnected
Objects
3D SurfacesCamera Viewpoint
![Page 23: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/23.jpg)
Input to Our AlgorithmInput to Our Algorithm
Surface Estimates Viewpoint Prior
Surfaces: [Hoiem-Efros-Hebert 2005]
Local Car Detector
Local Ped Detector
Object Detection
Local Detector: [Dalal-Triggs 2005]
![Page 24: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/24.jpg)
Scene Parts Are All InterconnectedScene Parts Are All Interconnected
Objects
3D SurfacesViewpoint
![Page 25: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/25.jpg)
Our Approximate ModelOur Approximate Model
Objects
3D SurfacesViewpoint
![Page 26: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/26.jpg)
s1
o1
θ
on...
sn…
Local Object Evidence
Local Surface Evidence
Local Object Evidence
Local Surface Evidence
Viewpoint
Objects
Local Surfaces
Inference over Tree Easy with BP Inference over Tree Easy with BP
![Page 27: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/27.jpg)
Viewpoint estimationViewpoint estimation
Viewpoint Prior
HorizonHeight Height Horizon
Like
liho
od
Like
liho
od
Viewpoint Final
![Page 28: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/28.jpg)
Object detectionObject detection
4 TP / 2 FP
3 TP / 2 FP
4 TP / 1 FP
Ped Detection
Car Detection
Local Detector: [Dalal-Triggs 2005]
4 TP / 0 FP
Car: TP / FP
Ped: TP / FP
Initial (Local) Final (Global)
![Page 29: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/29.jpg)
Experiments on LabelMe DatasetExperiments on LabelMe Dataset
• Testing with LabelMe dataset: 422 images
– 923 Cars at least 14 pixels tall
– 720 Peds at least 36 pixels tall
![Page 30: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/30.jpg)
Each piece of evidence improves performanceEach piece of evidence improves performance
Local Detector from [Murphy-Torralba-Freeman 2003]
Car Detection Pedestrian Detection
![Page 31: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/31.jpg)
Can be used with any detector that outputs confidencesCan be used with any detector that outputs confidences
Local Detector: [Dalal-Triggs 2005] (SVM-based)
Car Detection Pedestrian Detection
![Page 32: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/32.jpg)
Accurate Horizon EstimationAccurate Horizon Estimation
Median Error:
8.5% 4.5% 3.0%
90% Bound:
[Murphy-Torralba-Freeman 2003]
[Dalal- Triggs 2005]
Horizon Prior
![Page 33: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/33.jpg)
Qualitative ResultsQualitative Results
Initial: 2 TP / 3 FP Final: 7 TP / 4 FP
Local Detector from [Murphy-Torralba-Freeman 2003]
Car: TP / FP Ped: TP / FP
![Page 34: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/34.jpg)
Qualitative ResultsQualitative Results
Local Detector from [Murphy-Torralba-Freeman 2003]
Car: TP / FP Ped: TP / FP
Initial: 1 TP / 14 FP Final: 3 TP / 5 FP
![Page 35: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/35.jpg)
Qualitative ResultsQualitative Results
Car: TP / FP Ped: TP / FP
Local Detector from [Murphy-Torralba-Freeman 2003]
Initial: 1 TP / 23 FP Final: 0 TP / 10 FP
![Page 36: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/36.jpg)
Qualitative ResultsQualitative Results
Local Detector from [Murphy-Torralba-Freeman 2003]
Car: TP / FP Ped: TP / FP
Initial: 0 TP / 6 FP Final: 4 TP / 3 FP
![Page 37: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/37.jpg)
Summary & Future WorkSummary & Future Work
meters
met
ers Ped
Ped
Car
Reasoning in 3D:
• Object to object
• Scene label
• Object segmentation
![Page 38: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/38.jpg)
ConclusionConclusion
• Image understanding is a 3D problem
– Must be solved jointly
• This paper is a small step
– Much remains to be done
![Page 39: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/39.jpg)
Thank youThank you
![Page 40: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/40.jpg)
![Page 41: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/41.jpg)
A Return to Scene UnderstandingA Return to Scene Understanding
• Guzman (SEE), 1968
• Hansen & Riseman (VISIONS), 1978
• Barrow & Tenenbaum 1978
• Brooks (ACRONYM), 1979
• Marr, 1982
• Ohta & Kanade, 1978
• Yakimovsky & Feldman, 1973
[Ohta & Kanade 1978]
![Page 42: Putting Objects in Perspective](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815a86550346895dc7f43d/html5/thumbnails/42.jpg)
ImagesImages