cs 143: introduction to computer vision · expression detection (smile), snapchat filters (face...
TRANSCRIPT
![Page 1: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/1.jpg)
![Page 2: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/2.jpg)
![Page 3: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/3.jpg)
![Page 4: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/4.jpg)
![Page 5: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/5.jpg)
![Page 6: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/6.jpg)
![Page 7: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/7.jpg)
![Page 8: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/8.jpg)
Jitendra Malik, UC Berkeley
Three ‘R’s of Computer Vision
![Page 9: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/9.jpg)
Jitendra Malik, UC Berkeley
Three ‘R’s of Computer Vision
“The classic problems of computational vision:
reconstruction
recognition
(re)organization.”
![Page 10: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/10.jpg)
Have you ever used computer vision?
How? Where?
Think-Pair-Share
![Page 11: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/11.jpg)
Have you ever used computer vision?
How? Where?
Reconstruction? Recognition? (Re)organization?
Think-Pair-Share
![Page 12: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/12.jpg)
Laptop: Biometrics auto-login (face recognition, 3D), OCR
Smartphones: QR codes, computational photography (Android Lens Blur, iPhone Portrait Mode), panorama construction (Google Photo Spheres), face detection, expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel)
Web: Image search, Google photos (face recognition, object recognition, scene recognition, geolocalization from vision), Facebook (image captioning), Google maps aerial imaging (image stitching), YouTube (content categorization)
VR/AR: Outside-in tracking (HTC VIVE), inside out tracking (simultaneous localization and mapping, HoloLens), object occlusion (dense depth estimation)
Motion: Kinect, full body tracking of skeleton, gesture recognition, virtual try-on
Medical imaging: CAT / MRI reconstruction, assisted diagnosis, automatic pathology, connectomics, endoscopic surgery
Industry: Vision-based robotics (marker-based), machine-assisted router (jig), automated post, ANPR (number plates), surveillance, drones, shopping
Transportation: Assisted driving (everything), face tracking/iris dilation for drunkeness, drowsiness, automated distribution (all modes)
Media: Visual effects for film, TV (reconstruction), virtual sports replay (reconstruction), semantics-based auto edits (reconstruction, recognition)
![Page 13: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/13.jpg)
Optical character recognition (OCR)
Mail digit recognition, AT&T labs
http://www.research.att.com/~yann/
Technology to convert images of text into text
If you have a scanner, it probably came with OCR software
License plate readershttp://en.wikipedia.org/wiki/Automatic_number_plate_recognition
JH
LiveCamera
Translation
![Page 14: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/14.jpg)
Face detection
• Almost all digital cameras detect faces
• Snapchat face filters
![Page 15: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/15.jpg)
![Page 16: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/16.jpg)
![Page 17: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/17.jpg)
Smile detection
Sony Cyber-shot® T70 Digital Still Camera JH
![Page 18: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/18.jpg)
Object recognition (in supermarkets)
![Page 19: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/19.jpg)
How does it work? Think-Pair-Share
![Page 20: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/20.jpg)
How does it work?
![Page 21: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/21.jpg)
Thanks to Vivek Ramanujan
![Page 22: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/22.jpg)
Vision-based biometrics
“How the Afghan Girl was Identified by Her Iris Patterns”
Read the story (Wikipedia)
JH
![Page 23: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/23.jpg)
Facial login without a password…
![Page 24: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/24.jpg)
Facial login without a password…
![Page 25: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/25.jpg)
Facial login without a password…
Liang et al. 2014
![Page 26: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/26.jpg)
Video call eye gaze correction
Kuster et al., SIGGRAPH Asia 2012– https://cgl.ethz.ch/publications/papers/paperKus12.php
Apple FaceTime Attention Correction
![Page 27: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/27.jpg)
Object recognition (in mobile phones)
e.g., Google Lens
![Page 28: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/28.jpg)
3D from images
Building Rome in a Day: Agarwal et al. 2009
![Page 29: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/29.jpg)
Human shape capture
![Page 30: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/30.jpg)
Human shape capture
![Page 31: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/31.jpg)
Human shape capture
![Page 32: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/32.jpg)
Human shape capture
![Page 33: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/33.jpg)
Star Wars: Rogue One – Peter Cushing / Admiral Tarkin
Special effects: shape capture
![Page 34: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/34.jpg)
Special effects: shape capture
![Page 35: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/35.jpg)
Special effects: motion capture
![Page 36: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/36.jpg)
Interactive Games
Object Recognition: http://www.youtube.com/watch?feature=iv&v=fQ59dXOo63o
Mario: http://www.youtube.com/watch?v=8CTJL5lUjHg
3D: http://www.youtube.com/watch?v=7QrnwoO1-8A
Robot: http://www.youtube.com/watch?v=w8BmgtMKFbY
JH
![Page 37: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/37.jpg)
Sports
Sportvision first down line
Nice explanation on www.howstuffworks.com
JH
Virtual pitch markings Free viewpoint video
[Canon 2017]
![Page 38: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/38.jpg)
Medical imaging
Image guided surgery
Grimson et al., MIT3D imaging
MRI, CT
JH
![Page 39: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/39.jpg)
AutoCars - Uber bought CMU’s lab
![Page 40: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/40.jpg)
![Page 41: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/41.jpg)
![Page 42: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/42.jpg)
Industrial robots
Vision-guided robots position nut runners on wheels
JH
![Page 43: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/43.jpg)
Vision in spaaaaace
Vision systems (JPL) used for several tasks• Panorama stitching
• 3D terrain modeling
• Obstacle detection, position tracking
• For more, read “Computer Vision on Mars” by Matthies et al.
NASA'S Mars Exploration Rover Spirit captured this westward view from atop
a low plateau where Spirit spent the closing months of 2007.
JH
![Page 44: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/44.jpg)
2019 Chang-e 4 Moon Lander
![Page 45: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/45.jpg)
Mobile robotshttp://www.robocup.org/
Saxena et al. 2008
STAIR at Stanford
Skydio 2 drone6x fisheye cameras for obstacle avoidanceOnboard NVIDIA GPU
![Page 46: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/46.jpg)
Augmented Reality and Virtual Reality
MS HoloLens, Oculus, Magic Leap,ARCore / ARKit
![Page 47: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/47.jpg)
Augmented Reality and Virtual Reality
Niantic
Real-time monocular depth estimation and camera tracking
Oculus (Quest)
Real-time 3D hand pose estimation
![Page 48: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/48.jpg)
Jitendra Malik, UC Berkeley
Three ‘R’s of Computer Vision
“[Further progress in] the classic problems of computational vision:
reconstruction
recognition
(re)organization
[requires us to study the interaction among these processes].”
![Page 49: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/49.jpg)
Computer Vision and Nearby Fields
JH
Derogatory summary of computer vision:
“Machine learning applied to visual data.”
![Page 50: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/50.jpg)
Computer Vision and Nearby Fields
Derogatory summary of computer vision:
“Machine learning applied to visual data.”
JH
Model of the visual
world
Images, videos,sensor data…
Images, videos,interaction
Digital worldReal world
Computer Graphics Computer Vision
Information
![Page 51: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/51.jpg)
Superhuman state of the art?
Deep learning is an enormous disruption to the field.
Since 2012, rapid expansion and commercialization.
Why?
“With enough data, computer vision matches or even
outperforms human vision at most recognition tasks.”
JH
WHAT.
![Page 52: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/52.jpg)
Vision and Society
Lots of data = lots of potential bias in the data.
Needs understanding of possible failures.
+
Responsible approach.
+
Techniques to overcome bias.
![Page 53: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/53.jpg)
https://www.bbc.com/news/technology-51148501
“Regulators want time to work out how toprevent the technology being abused.
The technology allows faces captured on CCTVto be checked in real time against watch lists,often compiled by police.
The Commission … suggests that new rules willbe introduced to bolster existing regulationsurrounding privacy and data rights …
… and urged EU countries to create anauthority to monitor the new rules.
During the ban, which would last betweenthree and five years, "a sound methodology forassessing the impacts of this technology andpossible risk management measures could beidentified and developed".
![Page 54: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/54.jpg)
![Page 55: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/55.jpg)
![Page 56: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/56.jpg)
![Page 57: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/57.jpg)
![Page 58: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/58.jpg)
![Page 59: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/59.jpg)
![Page 60: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/60.jpg)
![Page 61: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/61.jpg)
COURSE ETHOS AND SCOPE
![Page 62: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/62.jpg)
“To create the ship is not
to weave the webs,
to forge the nails,
to read the stars,
but to give the taste of the sea.”
Saint Exupéry
![Page 63: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/63.jpg)
Scope of Computer Vision
Robotics
Neuroscience
Graphics
Computational Photography
Machine Learning
Medical Imaging
Human Computer Interaction
Optics
CSCI 1430: Pull from all
of them!
JH
DeepLearning
Image Processing
Artificial Intelligence
![Page 64: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/64.jpg)
Prerequisites
• Linear algebra, basic calculus and probability.
• Programming, data structures.
This is an upper-level course.
We move fast.
If you’re rusty, we point you to refreshers.
![Page 65: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/65.jpg)
HERE TO HELP!
![Page 66: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/66.jpg)
Jason Senthil
Mary Dong
Joy Zheng
Andrew ParkRaymond Cao Kyle Cui
![Page 67: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/67.jpg)
Cong Huang
Yang Zhang
Top Piriyakulkij Yuting Liu
TrevorHouchens
EliotLaidlaw
![Page 68: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/68.jpg)
Eleanor Tursman
Neev Parikh
Nine Prasertsup
Josh Roy
Claire Chen
Katie Scholl
QiaoJiang
![Page 69: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/69.jpg)
ETAs!
Isabella Ting
Katie Friis
![Page 70: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/70.jpg)
Isa Milefchik (HTA) George Lee (HTA)
![Page 71: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/71.jpg)
TA HoursCourse webpage
![Page 72: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/72.jpg)
My Office Hours
jamestompkin.com
![Page 73: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/73.jpg)
COURSE SETUP
![Page 74: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/74.jpg)
CSCI 1430 – James Hays
• Continuing his course – many materials & projects from him + previous staff – serious thanks!
• If you see a little ‘JH’ in the slide corner, then it’s his.
![Page 75: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/75.jpg)
Contact
• Course runs quiet hours – 9pm to 9am.
– We will ignore you (temporarily).
• Piazza first
– TAs have set Piazza hours.
• [email protected] second
![Page 76: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/76.jpg)
Waitlist / Override codes
• Yes, we are oversubscribed (50+)
• Yes, we have priority
– seniors + grads, juniors, sophomores, fresh
• We will authorize overrides later in shopping week once it calms down
• Request an override on cab.brown.edu
![Page 77: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/77.jpg)
1. Search for course 2. Make sure added to cart
3. Request override
![Page 79: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/79.jpg)
Textbooks
KletteConcise Computer Vision
Digital copy at our library
More introductory than Szeliski.
https://ccv.wordpress.fos.auckland.ac.nz/
![Page 80: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/80.jpg)
Textbooks
![Page 81: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/81.jpg)
Projects / Grading
• 100% projects (7 total)
• Project 0: Setup / intro
• Projects 1-5: Structured conceptual / code
• Project 6: Final group project
Project Percent
0 2%
1–5 ~14.6%
6 ~25%
![Page 82: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/82.jpg)
Submission system: Gradescope
![Page 83: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/83.jpg)
• Anonymous submissions please.– Don’t put your name into Gradescope; only your Brown email.– Use fake name if you need to.
• Written questions:– It’s a template to help us grade efficiently.– Use _only_ the space provided – Please don’t make more/fewer pages– You can put extra pages at the end
![Page 84: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/84.jpg)
Gear-up Session (video captured):
TONIGHT 6pm Barus and Holley 168
What is Git?
What is Github?
What is Gradescope?
What is Visual Studio Code?
What is Python / Virtual Environment?
How they work together.
I promise it’s worth it.
![Page 85: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/85.jpg)
How it works – project example
You:
Us:
Repo copy from our Github
Work / debug in Python in VSCode
Push to your own repo
Submit repo to Gradescope
Push template repo to Github
Grade your repo on Gradescope
Office hours+ Piazza
Autograderfeedback
Release date
Due date
![Page 86: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/86.jpg)
Proj 1: Image Filtering and Hybrid Images
• Implement image filtering to separate high and low frequencies.
• Combine high frequencies and low frequencies from different images to create a scale-dependent image.
JH
![Page 87: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/87.jpg)
Proj 2: Local Feature Matching
• Implement interest point detector, SIFT-like local feature descriptor, and simple matching algorithm.
JH
![Page 88: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/88.jpg)
Proj 3: Scene Recognition with Bag of Words
• Quantize local features into a “vocabulary”, describe images as histograms of “visual words”, train classifiers to recognize scenes based on these histograms.
JH
![Page 89: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/89.jpg)
Proj 4: Convolutional Neural Nets
• Proj 3 again, but state of the art.
JH
![Page 90: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/90.jpg)
Proj 5: Multi-view Geometry
• Recover camera calibration from feature point matches.
• Foundation for almost all measurement in computer vision.
JH
![Page 91: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/91.jpg)
Proj 6: Group Final Project
• Free choice with a set of suggested projects
• Up to four people
• Go wild
Project examples
• Real-time eye tracking
• Multi-view geometry reconstruction
• Computational photography
• Style Transfer
• Adversarial CNN hacking
![Page 92: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/92.jpg)
Friendly neighborhood style transfer
Nathaniel Nguyen, Kyle Cui, Jules Becker, Leonardo Ko
CSCI1430 Spring 2019 Final Project
![Page 93: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/93.jpg)
Naive frame-by-frame rendering
CSCI1430 Spring 2019 Final Project
Nathaniel Nguyen, Kyle Cui, Jules Becker, Leonardo Ko
![Page 94: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/94.jpg)
Temporal Consistency Constraints
Nathaniel Nguyen, Kyle Cui, Jules Becker, Leonardo Ko
CSCI1430 Spring 2019 Final Project
![Page 95: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/95.jpg)
Result Video
Baseline Video
Hyperlapse Stabilization - Michael Mao, Jiaju Ma, James Li
CSCI1430 Spring 2019 Final Project
![Page 96: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/96.jpg)
100
Darius Atmar, Yueyi Sun, Zejiang Shen
CSCI1430 Spring 2019 Final Project
![Page 97: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/97.jpg)
Any questions at this stage?!
• Waitlist on cab.brown.edu – request override
– If you’re on the waitlist, still submit project 0+1
• Gear Up Session TONIGHT 6pm B&H 168
– Lecture captured if you can’t make it
• TA hours from today
• Project 0 due FRIDAY 9pm
• Project 1 due NEXT FRIDAY 9pm
![Page 98: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/98.jpg)
JAMES
![Page 99: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/99.jpg)
I work in here.
Graphics
InteractionVision
![Page 100: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/100.jpg)
![Page 101: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/101.jpg)
![Page 102: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/102.jpg)
![Page 103: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/103.jpg)
![Page 104: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/104.jpg)
Instructor: James Tompkin
Max Planck InstituteGermany
University College LondonUK
![Page 105: CS 143: Introduction to Computer Vision · expression detection (smile), Snapchat filters (face tracking), Google Tango (3D reconstruction), Night Sight (Pixel) Web: Image search,](https://reader030.vdocuments.mx/reader030/viewer/2022040611/5ed8f4b76714ca7f4768e232/html5/thumbnails/105.jpg)
[email protected]: CIT 547
Render pixels?Capture pixels?Interact with pixels?
Watch my research overview video!
I am probably interested.