internet vision - lecture 3
DESCRIPTION
Internet Vision - Lecture 3. Tamara Berg Sept 10. New Lecture Time. Mondays 10:00am-12:30pm in 2311 Monday (9/15) we will have a general Computer Vision & Machine Learning review Please look at papers and decide which one you want to present by Monday - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/1.jpg)
Internet Vision - Lecture 3
Tamara BergSept 10
![Page 2: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/2.jpg)
New Lecture Time
Mondays 10:00am-12:30pm in 2311
Monday (9/15) we will have a general Computer Vision & Machine Learning review
Please look at papers and decide which one you want to present by Monday – read topic/titles/abstracts to get an idea of which
you are interested in
![Page 3: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/3.jpg)
Thanks to Lalonde et al for providing slides!
![Page 4: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/4.jpg)
![Page 5: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/5.jpg)
Algorithm Outline
![Page 6: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/6.jpg)
Inserting objects into images
Have an image and want to add realistic looking objects to that image
![Page 7: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/7.jpg)
Inserting objects into images
User picks a location where they want to insert an object
![Page 8: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/8.jpg)
Inserting objects into images
Based some properties calculated about the image, possible objects are presented.
![Page 9: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/9.jpg)
Inserting objects into images
User selects which object to insert and the object is placed in the scene at the correct scale for the location
![Page 10: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/10.jpg)
Inserting objects into images – Possible approaches
Insert a clip art object Insert a clip art object with some idea of the environment
Insert a rendered object with full model of the environment
![Page 11: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/11.jpg)
![Page 12: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/12.jpg)
![Page 13: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/13.jpg)
![Page 14: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/14.jpg)
![Page 15: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/15.jpg)
Some objects will be easy to insert because they already “fit” into the scene
![Page 16: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/16.jpg)
Collect a large database of objects.Let the computer decide which examples are easy to insert.Allow the user to select only among those.
![Page 17: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/17.jpg)
When will an object “fit”?
1.) When the lighting conditions of the scene and object are similar2.) When the camera pose of the scene & object match
![Page 18: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/18.jpg)
2D vs 3D
Use 3d information for:
1.) Annotating objects in the clip-art library with camera pose2.) Estimating the camera pose in the query image3.) Computing illumination context in both library & query images
![Page 19: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/19.jpg)
![Page 20: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/20.jpg)
![Page 21: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/21.jpg)
Phase 1 - Database Annotation
For each object we want:– Estimate of its true size and the camera pose it
was captured under– Estimate of the lighting conditions it was captured
under
![Page 22: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/22.jpg)
Phase 1 - Database AnnotationEstimate object size
Objects closer to the camera appear larger than objects further from the camera
![Page 23: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/23.jpg)
Phase 1 - Database AnnotationEstimate object size
*If* you know the camera pose then you can estimate the real height of an object from:location in the image,pixel height
![Page 24: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/24.jpg)
Phase 1 - Database AnnotationEstimate object size
Annotate objects with their true heights and resize examples to a common reference size
![Page 25: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/25.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
Don’t know camera pose or object heights! Trick - Infer camera pose & object heights across all object classes in the database given only the height distribution for one class
![Page 26: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/26.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
Start with known heights for people
![Page 27: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/27.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
Estimate camera pose for images with multiple people
![Page 28: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/28.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
Use these images to estimate a prior over the distribution of poses
How do people usually take pictures? Standing on the ground at eye level.
![Page 29: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/29.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
Use the learned pose distribution to estimate heights of other object categories that appear with people.
Iteratively use these categories to learn more categories.
Annotate all objects in the database with their true size and originating camera pose.
![Page 30: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/30.jpg)
Phase 1 - Database AnnotationEstimate object size & camera pose
![Page 31: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/31.jpg)
Phase 1 - Database Annotation
For each object we want:– Estimate of its true size and the camera pose it
was captured under– Estimate of the lighting conditions it was captured
under
![Page 32: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/32.jpg)
Phase 1 - Database AnnotationEstimate lighting conditions
Estimate which pixels are ground, sky, vertical
Black box for now (we’ll cover this paper later in the course)
Ground
Vertical
Sky
![Page 33: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/33.jpg)
Phase 1 - Database AnnotationEstimate lighting conditions
Distribution of pixel colors
![Page 34: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/34.jpg)
![Page 35: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/35.jpg)
Phase 2 – Object Insertion
Query Image
![Page 36: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/36.jpg)
Phase 2 – Object Insertion
User specifies horizon line – use to calculate camera pose with respect to ground plane (lower -> tilted down, higher -> tilted up).
Illumination context is calculated in the same way as for the database images.
![Page 37: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/37.jpg)
Phase 2 – Object Insertion
Insert an object into the scene that has matching lighting, and camera pose to the query image
![Page 38: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/38.jpg)
Phase 2 – Object Insertion
But wait it still looks funny!
![Page 39: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/39.jpg)
Phase 2 – Object Insertion
Shadows are important!
![Page 40: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/40.jpg)
Phase 2 – Object Insertion
![Page 41: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/41.jpg)
Phase 2 – Object Insertion
![Page 42: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/42.jpg)
Phase 2 – Object Insertion
![Page 43: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/43.jpg)
Phase 2 – Object Insertion
Shadow Transfer
![Page 44: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/44.jpg)
Categorize images for easy selection in user interface
![Page 45: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/45.jpg)
![Page 46: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/46.jpg)
![Page 47: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/47.jpg)
![Page 48: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/48.jpg)
![Page 49: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/49.jpg)
![Page 50: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/50.jpg)
![Page 51: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/51.jpg)
![Page 52: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/52.jpg)
![Page 53: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/53.jpg)
![Page 54: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/54.jpg)
![Page 55: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/55.jpg)
![Page 56: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/56.jpg)
![Page 57: Internet Vision - Lecture 3](https://reader035.vdocuments.mx/reader035/viewer/2022062422/5681334e550346895d9a562e/html5/thumbnails/57.jpg)
Big Picture
• It’s all about the data!
• Use lots of data to turn a hard problem into an easier one!– Place “my car” in a scene is much harder than
place “some car” in a scene. Allow the computer to choose from among many examples of a class to find the easy ones.