3d computer vision introduction
TRANSCRIPT
![Page 2: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/2.jpg)
Administrivia
• Classes: M & W, 1.25-2:45 Room WEB L126
• Instructor: Guido Gerig [email protected] (801) 585 0327
• Prerequisites: CS 6640 ImProc (or equiv.) • Textbook: Computer Vision: Algorithms and Applications“
by Richard Szeliski • Organization web-site (slides, documents and assignments)
![Page 3: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/3.jpg)
Web-Site
• Linked to UofU class list • Linked to my home page • http://www.sci.utah.edu/~gerig/CS6320-
S2012/CS6320_3D_Computer_Vision.html
![Page 4: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/4.jpg)
Teaching Assistant
• TA: Kathlea Quebbeman, SoC [email protected]
• HW/SW: Matlab+ev. Imaging Toolbox CADE lab WEB 130 http://www.cade.utah.edu/ TA office Hours: M,W 3-5, send email for other appointments
![Page 5: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/5.jpg)
Prerequisites
• General Prerequisites: – Data structures – A good working knowledge of MATLAB programming (or
willingness and time to pick it up quickly!) – Linear algebra – Vector calculus
• Assignments include theoretical paper questions and programming tasks (ideally Matlab or C++).
• Image Processing CS 6640 (or equivalent). • Students who do not have background in signal
processing / image processing: Eventually possible to follow class, but requires significant special effort to learn some basic procedures necessary to solve practical computer problems.
![Page 6: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/6.jpg)
Textbook
![Page 7: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/7.jpg)
Grading
• Assignments (4-6 theory/prog.): 50% • Quizzes & written assignments: 10% • Final project (incl. design, proposal,
demo, presentation, report): 30% • Class participation (active participation
in summaries and discussions): 10% • Final project replaces final exam • Successful final project required for
passing grade
![Page 8: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/8.jpg)
Other Resources
• Cvonline: http://homepages.inf.ed.ac.uk/rbf/CVonline/
• A first point of contact for explanations of different image related concepts and techniques. CVonline currently has about 2000 topics, 1600 of which have content.
• See list of other relevant books in syllabus.
![Page 9: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/9.jpg)
Some Basics
• Instructor and TA do not use email list to communicate.
• It will be your responsibility to regularly read the News&Announcements on the web-page.
• We don’t need a laptop for the class, please keep them closed !!!!!
• Please interact, ask questions, clarifications, input to instructor and TA.
• Cell phones …., you surely know.
![Page 10: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/10.jpg)
Syllabus
• See separate syllabus (linked to web-site).
• Document
![Page 11: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/11.jpg)
Goal and Objectives
From Snapshots, a 3-D View NYT, August 21, 2008, Personal Tech http://www.nytimes.com/2008/08/21/technology/personaltech/21pogue.html
Stuart Goldenberg
![Page 12: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/12.jpg)
Modeling 3D Structure from Pictures or 3D Sensors
![Page 13: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/13.jpg)
Modeling ctd.
![Page 14: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/14.jpg)
Goal and objectives
• To introduce the fundamental problems of computer vision.
• To introduce the main concepts and techniques used to solve those.
• To enable participants to implement solutions for reasonably complex problems.
• To enable the student to make sense of the literature of computer vision.
![Page 15: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/15.jpg)
Why study Computer Vision?
• Images and movies are everywhere • Fast-growing collection of useful applications
– building representations of the 3D world from pictures
– automated surveillance (who’s doing what) – movie post-processing – CAM (computer-aided manufacturing – Robot navigation – face finding
• Various deep and attractive scientific mysteries – how does object recognition work?
• Greater understanding of human vision
![Page 16: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/16.jpg)
CV: What is the problem?
![Page 17: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/17.jpg)
CV: A Hard Problem
![Page 18: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/18.jpg)
Main topics
• Shape (and motion) recovery “What is the 3D shape of what I see?”
• Segmentation “What belongs together?”
• Tracking “Where does something go?”
• Recognition “What is it that I see?”
![Page 19: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/19.jpg)
Main topics
• Camera & Light – Geometry, Radiometry, Color
• Digital images – Filters, edges, texture, optical flow
• Shape (and motion) recovery – Multi-view geometry – Stereo, motion, photometric stereo, …
• Segmentation – Clustering, model fitting, probalistic
• Tracking – Linear dynamics, non-linear dynamics
• Recognition – templates, relations between templates
![Page 20: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/20.jpg)
Motivation
• (some slides modified from Marc Pollefeys, UNC Chapel Hill & ETH Zurich)
![Page 21: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/21.jpg)
Clothing
• Scan a person, custom-fit clothing
![Page 22: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/22.jpg)
Forensics
![Page 23: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/23.jpg)
3D urban modeling
drive by modeling in Baltimore
![Page 24: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/24.jpg)
Structure from Motion
![Page 25: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/25.jpg)
Industrial inspection
• Verify specifications • Compare measured model with CAD
![Page 26: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/26.jpg)
Scanning industrial sites
as-build 3D model of off-shore oil platform
![Page 27: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/27.jpg)
Robot navigation
small tethered rover
pan/tilt stereo head
ESA projectour task: Calibration + Terrain modelling + Visualization
![Page 28: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/28.jpg)
Robot navigation
small tethered rover
pan/tilt stereo head
ESA projectour task: Calibration + Terrain modelling + Visualization
![Page 29: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/29.jpg)
Architecture
Survey Stability analysis Plan renovations
![Page 30: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/30.jpg)
Architecture
Survey Stability analysis Plan renovations
![Page 31: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/31.jpg)
Cultural heritage
Virtual Monticello
Allow virtual visits
![Page 32: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/32.jpg)
Cultural heritage
Stanford’s Digital Michelangelo
Digital archiveArt historic studies
![Page 33: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/33.jpg)
IBM’s pieta project Photometric stereo + structured light
more info: http://researchweb.watson.ibm.com/pieta/pieta_details.htm
![Page 34: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/34.jpg)
Archaeology
accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720)
![Page 35: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/35.jpg)
Sony’s Eye Toy: Computer Vision for the masses
Background segmentation/ motion detection Color segmentation …
![Page 36: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/36.jpg)
Shape from …
many different approaches/cues
![Page 37: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/37.jpg)
Optical flow
Where do pixels move?Where do pixels move?
![Page 38: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/38.jpg)
Optical flow
Where do pixels move?
![Page 39: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/39.jpg)
Active Vision: Structured Light
![Page 40: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/40.jpg)
Static Light Pattern Projection
Segmentation of stripes from set of images Geometry between projector,
sensor and objects
![Page 41: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/41.jpg)
Active Vision: Structured Light
Segmentation: Binarization and coding of stripes
3D model extracted from stripe pattern
![Page 42: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/42.jpg)
Spatiotemporal Volumes
![Page 43: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/43.jpg)
Motion Tails
![Page 44: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/44.jpg)
Object Tracking: Using Deformable Models in Vision
![Page 45: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/45.jpg)
Object Tracking: Using Deformable Models in Vision: II
![Page 46: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/46.jpg)
Object Tracking III
![Page 47: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/47.jpg)
Face detection
http://www.umiacs.umd.edu/~aswch/test.html http://vasc.ri.cmu.edu/cgi-bin/demos/findface.cgi
![Page 48: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/48.jpg)
Examples of Student Projects
![Page 49: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/49.jpg)
Student Project: Playing Chess, Recognition and Simulation
• Track individual chess pieces
• Maintain state of board
• Graphically represent state changes and state
• D. Allen, D. McLaurin UNC
• Major ideas: – 3D from stereo – detect and describe
changes – Use world knowledge
(chess)
![Page 50: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/50.jpg)
Calibration, Rendering & Replay
Movie
![Page 51: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/51.jpg)
Enhanced Correlation for Stereo Vision
• Andrew Nashel • Correlation map produced by
precomputation method:
![Page 52: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/52.jpg)
Results – Lord Buddha Images – Pre-Processed Images
Guozhen Fan and Aman Shah
Original Image
Obtained Surfaces from different angles
Surface Normals Albedo Map
![Page 53: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/53.jpg)
Photometric StereoChristopher Bireley
Bandage Dog
Imaging Setup
![Page 54: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/54.jpg)
Photometric StereoChristopher Bireley
Albedo image Surface Normals
3D mesh
![Page 55: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/55.jpg)
Structured Light James Clark
Positioned camera with
object in view Result
![Page 56: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/56.jpg)
Structured Light ctd. James Clark
![Page 57: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/57.jpg)
Webcam Based Virtual Whiteboard
Jon Bronson James Fishbaugh
![Page 58: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/58.jpg)
Webcam Based Virtual Whiteboard
Jon Bronson James Fishbaugh
![Page 59: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/59.jpg)
Structured Light Anuja Sharma, Abishek Kumar
![Page 60: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/60.jpg)
Structured Light Anuja Sharma, Abishek Kumar
![Page 61: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/61.jpg)
Realtime Glowstick DetectionAndrei Ostanin
movie
![Page 62: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/62.jpg)
Scientific Computing and Imaging Institute, University of Utah
3D shape from silhouettes: Two Mirrors and uncalibrated camera
Forbes et al., ICCV2005
Christine Xu, Computer Vision Student Project
![Page 63: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/63.jpg)
Scientific Computing and Imaging Institute, University of Utah
3D shape from silhouettes
Segmentation of contours
Think about the geometry -> calculate relationship between silhouettes
Result: 3D Object
![Page 64: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/64.jpg)
Next class: Cameras Chapter 2: Image Formation
![Page 65: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/65.jpg)
Next class: Image Formation Chapter 2: Textbook
• Please find pdf copies of Chapters 1&2 on the website.
Assignment:
• Read Chapter 1 for additional materials • Read Chapter 2 for preparation of 2nd lecture
![Page 66: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/66.jpg)
Cultural heritage
Virtual Monticello
Allow virtual visits
![Page 67: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/67.jpg)
Cultural heritage
Stanford’s Digital Michelangelo
Digital archiveArt historic studies
![Page 68: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/68.jpg)
Shape from …
many different approaches/cues
![Page 69: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/69.jpg)
Optical flow
Where do pixels move?Where do pixels move?
![Page 70: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/70.jpg)
Active Vision: Structured Light
Segmentation: Binarization and coding of stripes
3D model extracted from stripe pattern
![Page 71: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/71.jpg)
Spatiotemporal Volumes
![Page 72: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/72.jpg)
Motion Tails
![Page 73: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/73.jpg)
Photometric StereoChristopher Bireley
Bandage Dog
Imaging Setup
![Page 74: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/74.jpg)
Photometric StereoChristopher Bireley
Albedo image Surface Normals
3D mesh
![Page 75: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/75.jpg)
Realtime Glowstick DetectionAndrei Ostanin
movie
![Page 76: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/76.jpg)
Scientific Computing and Imaging Institute, University of Utah
3D shape from silhouettes: Two Mirrors and uncalibrated camera
Forbes et al., ICCV2005
Christine Xu, Computer Vision Student Project
![Page 77: 3D Computer Vision Introduction](https://reader031.vdocuments.mx/reader031/viewer/2022012514/618dcf27dd27066ae828a9bc/html5/thumbnails/77.jpg)
Scientific Computing and Imaging Institute, University of Utah
3D shape from silhouettes
Segmentation of contours
Think about the geometry -> calculate relationship between silhouettes
Result: 3D Object