Lecture 1 -

Fei-Fei Li

Lecture 1: Introduction to “Computer Vision”

Professor Fei-Fei Li Stanford Vision Lab

23-Sep-11 1

Lecture 1 -

Fei-Fei Li

Welcome to CS231a: Computer Vision

Slid

e ad

apte

d fr

om S

vetl

ana

Laze

bnik

23-Sep-11 2

Lecture 1 -

Fei-Fei Li

Today’s agenda

• Introduction to computer vision • Course overview

23-Sep-11 3

Lecture 1 -

Fei-Fei Li

Quiz?

23-Sep-11 4

Lecture 1 -

Fei-Fei Li

What about this?

23-Sep-11 5

Lecture 1 -

Fei-Fei Li

Image (or video) Sensing device Interpreting device Interpretations

garden, spring, bridge, water, trees, flower, green, etc.

What is (computer) vision?

23-Sep-11 6

Lecture 1 -

Fei-Fei Li

What is it related to?

Computer Vision

Neuroscience

Machine learning

Speech

Information retrieval

Maths

Computer Science

Biology

Engineering

Physics

Robotics Cognitive sciences

Psychology

graphics,algorithms, system,theory,…

Image processing

23-Sep-11 7

Lecture 1 -

Fei-Fei Li

The goal of computer vision • To bridge the gap between pixels and “meaning”

What we see What a computer sees Sou

rce:

S. N

aras

imha

n

23-Sep-11 8

Lecture 1 -

Fei-Fei Li

Image (or video) Sensing device Interpreting device Interpretations

garden, spring, bridge, water, trees, flower, green, etc.

What is (computer) vision?

23-Sep-11 9

Lecture 1 -

Fei-Fei Li

1981: Nobel Prize in medicine

Hubel & Wiesel

23-Sep-11 10

Lecture 1 -

Fei-Fei Li

Potter, Biederman, etc. 1970s

Human vision is superbly efficient

23-Sep-11 11

Lecture 1 -

Fei-Fei Li

Thorpe, et al. Nature, 1996

23-Sep-11 12

Lecture 1 -

Fei-Fei Li

Thorpe, et al. Nature, 1996

150 ms !!

23-Sep-11 13

Lecture 1 -

Fei-Fei Li

Change blindess

Rensink, O’regan, Simon, etc.

23-Sep-11 14

Lecture 1 -

Fei-Fei Li

Rensink, O’regan, Simon, etc.

Change blindess

23-Sep-11 15

Lecture 1 -

Fei-Fei Li

segmentation

23-Sep-11 16

Lecture 1 -

Fei-Fei Li

Perception

23-Sep-11 17

Lecture 1 -

Fei-Fei Li 23-Sep-11 18

Lecture 1 -

Fei-Fei Li 23-Sep-11 19

Lecture 1 -

Fei-Fei Li

Image (or video) Sensing device Interpreting device Interpretations

garden, spring, bridge, water, trees, flower, green, etc.

What is (computer) vision?

23-Sep-11 20

Lecture 1 -

Fei-Fei Li

The goal of computer vision • To bridge the gap between pixels and “meaning”

What we see What a computer sees Sou

rce:

S. N

aras

imha

n

23-Sep-11 21

Lecture 1 -

Fei-Fei Li

Origins of computer vision: an MIT undergraduate summer project

L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.

23-Sep-11 22

Lecture 1 -

Fei-Fei Li

What kind of information can we extract from an image?

• Metric 3D information • Semantic information

23-Sep-11 23

Lecture 1 -

Fei-Fei Li

Vision as measurement device Real-time stereo Structure from motion

NASA Mars Rover

Pollefeys et al.

Reconstruction from Internet photo collections

Goesele et al.

23-Sep-11 24

Lecture 1 -

Fei-Fei Li

Vision as a source of semantic information sky

water

Ferris wheel

amusement park

Cedar Point

12 E

tree

tree

tree

carousel deck

people waiting in line

ride

ride ride

umbrellas

pedestrians

maxair

bench

tree

Lake Erie

people sitting on ride

Objects Activities Scenes Locations Text / writing Faces Gestures Motions Emotions…

The Wicked Twister

Slid

e cr

edit

: Kr

iste

n G

raum

an

23-Sep-11 25

Lecture 1 -

Fei-Fei Li

Why study computer vision?

Personal photo albums

Surveillance and security

Movies, news, sports

Medical and scientific images

• Vision is useful: Images and video are everywhere!

23-Sep-11 26

Lecture 1 -

Fei-Fei Li

Why study computer vision? • Vision is useful • Vision is interesting • Vision is difficult

– Half of primate cerebral cortex is devoted to visual processing

– Achieving human-level visual perception is probably “AI-complete”

23-Sep-11 27

Lecture 1 -

Fei-Fei Li

Why is computer vision difficult?

23-Sep-11 28

Lecture 1 -

Fei-Fei Li

Challenges: viewpoint variation

Michelangelo 1475-1564

slide credit: Fei-Fei, Fergus & Torralba

23-Sep-11 29

Lecture 1 -

Fei-Fei Li

Challenges: illumination

image credit: J. Koenderink

23-Sep-11 30

Lecture 1 -

Fei-Fei Li

Challenges: scale

slid

e cr

edit:

Fei

-Fei

, Fer

gus

& T

orra

lba

23-Sep-11 31

Lecture 1 -

Fei-Fei Li

Challenges: deformation

Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba

23-Sep-11 32

Lecture 1 -

Fei-Fei Li

Challenges: occlusion

Magritte, 1957

slide credit: Fei-Fei, Fergus & Torralba 23-Sep-11 33

Lecture 1 -

Fei-Fei Li

Challenges: background clutter

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 34

Lecture 1 -

Fei-Fei Li

Challenges: Motion

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 35

Lecture 1 -

Fei-Fei Li

Challenges: object intra-class variation

slid

e cr

edit:

Fei

-Fei

, Fer

gus

& T

orra

lba

23-Sep-11 36

Lecture 1 -

Fei-Fei Li

Challenges: local ambiguity

slid

e cr

edit:

Fei

-Fei

, Fer

gus

& T

orra

lba

23-Sep-11 37

Lecture 1 -

Fei-Fei Li

Challenges or opportunities? • Images are confusing, but they also reveal the

structure of the world through numerous cues • Our job is to interpret the cues!

Imag

e so

urce

: J. K

oend

erin

k

23-Sep-11 38

Lecture 1 -

Fei-Fei Li

Depth cues: Linear perspective

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 39

Lecture 1 -

Fei-Fei Li

Depth cues: Aerial perspective

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 40

Lecture 1 -

Fei-Fei Li

Depth ordering cues: Occlusion

Sou

rce:

J. K

oend

erin

k

23-Sep-11 41

Lecture 1 -

Fei-Fei Li

Shape cues: Texture gradient

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 42

Lecture 1 -

Fei-Fei Li

Shape and lighting cues: Shading

Sou

rce:

J. K

oend

erin

k

23-Sep-11 43

Lecture 1 -

Fei-Fei Li

Position and lighting cues: Cast shadows

Sou

rce:

J. K

oend

erin

k

23-Sep-11 44

Lecture 1 -

Fei-Fei Li

Grouping cues: Similarity (color, texture, proximity)

slid

e cr

edit:

Sve

tlana

Laz

ebni

k

23-Sep-11 45

Lecture 1 -

Fei-Fei Li

Grouping cues: “Common fate”

Imag

e cr

edit:

Arth

us-B

ertra

nd (v

ia F

. Dur

and)

23-Sep-11 46

Lecture 1 -

Fei-Fei Li

Bottom line • Perception is an inherently ambiguous problem

– Many different 3D scenes could have given rise to a particular 2D picture

23-Sep-11 47

Lecture 1 -

Fei-Fei Li

Bottom line • Perception is an inherently ambiguous problem

– Many different 3D scenes could have given rise to a particular 2D picture

• Possible solutions – Bring in more constraints (more images) – Use prior knowledge about the structure of the world

• Need a combination of different methods 23-Sep-11 48

Lecture 1 -

Fei-Fei Li

Computer Vision in the Real World

23-Sep-11 49

Lecture 1 -

Fei-Fei Li

Special effects: shape and motion capture

Sour

ce: S

. Sei

tz

23-Sep-11 50

Lecture 1 -

Fei-Fei Li

3D urban modeling

Bing maps, Google Streetview Source: S. Seitz

23-Sep-11 51

Lecture 1 -

Fei-Fei Li

3D urban modeling: Microsoft Photosynth

http://labs.live.com/photosynth/ Source: S. Seitz

23-Sep-11 52

Lecture 1 -

Fei-Fei Li

Face detection

• Many new digital cameras now detect faces – Canon, Sony, Fuji, …

Source: S. Seitz

23-Sep-11 53

Lecture 1 -

Fei-Fei Li

Smile detection

Sony Cyber-shot® T70 Digital Still Camera Source: S. Seitz

23-Sep-11 54

Lecture 1 -

Fei-Fei Li

Face recognition: Apple iPhoto software

http://www.apple.com/ilife/iphoto/

23-Sep-11 55

Lecture 1 -

Fei-Fei Li

Biometrics

How the Afghan Girl was Identified by Her Iris Patterns

Source: S. Seitz

23-Sep-11 56

Lecture 1 -

Fei-Fei Li

Biometrics

Fingerprint scanners on many new laptops, other devices

Face recognition systems now beginning to appear more widely http://www.sensiblevision.com/ Source: S. Seitz

23-Sep-11 57

Lecture 1 -

Fei-Fei Li

Optical character recognition (OCR)

Digit recognition, AT&T labs

Technology to convert scanned docs to text • If you have a scanner, it probably came with OCR software

License plate readers http://en.wikipedia.org/wiki/Automatic_number_plate_recognition

Source: S. Seitz

23-Sep-11 58

Lecture 1 -

Fei-Fei Li

Toys and Robots

Lecture 1 -

Fei-Fei Li

Mobile visual search: Google Goggles

23-Sep-11 60

Lecture 1 -

Fei-Fei Li

Mobile visual search: iPhone Apps

23-Sep-11 61

Lecture 1 -

Fei-Fei Li

Automotive safety

• Mobileye: Vision systems in high-end BMW, GM, Volvo models – “In mid 2010 Mobileye will launch a world's first application of full

emergency braking for collision mitigation for pedestrians where vision is the key technology for detecting pedestrians.”

Source: A. Shashua, S. Seitz

23-Sep-11 62

Lecture 1 -

Fei-Fei Li

Vision in supermarkets

LaneHawk by EvolutionRobotics “A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk, you are assured to get paid for it… “ Source: S. Seitz

23-Sep-11 63

Lecture 1 -

Fei-Fei Li

Vision-based interaction (and games)

Microsoft’s Kinect

Source: S. Seitz Assistive technologies

Sony EyeToy

23-Sep-11 64

Lecture 1 -

Fei-Fei Li

Vision for robotics, space exploration

Vision systems (JPL) used for several tasks • Panorama stitching • 3D terrain modeling • Obstacle detection, position tracking • For more, read “Computer Vision on Mars” by Matthies et al.

NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.

Sour

ce: S

. Sei

tz

23-Sep-11 65

Lecture 1 -

Fei-Fei Li

The computer vision industry

• A list of companies here: http://www.cs.ubc.ca/spider/lowe/vision.html

23-Sep-11 66

Lecture 1 -

Fei-Fei Li

Today’s agenda

• Introduction to computer vision • Course overview

23-Sep-11 67

Lecture 1 -

Fei-Fei Li

TA’s of the class • Kevin Tang

– cs231a-aut1112-staff@lists. – Office hour: Thur 4-5pm

• Jiahui Shi – cs231a-aut1112-staff@lists. – Office hour: Mon 4-5pm

• Yongwhan Lim – cs231a-aut1112-staff@lists. – Office hour: Fri 3:30-4:30pm

• Hao Su – cs231a-aut1112-staff@lists. – Office hour: Tues 5-6pm

23-Sep-11 68

Lecture 1 -

Fei-Fei Li

Syllabus

• Go to website…

23-Sep-11 69

Lecture 1 -

Fei-Fei Li

Course Project: overview

• 40% of your grade • Form your team:

– either 2 people or 1 person – but the quality is judged regardless of the number

of people on the team – be nice to your partner: do you plan to drop the

course?

23-Sep-11 70

Lecture 1 -

Fei-Fei Li

Course Project: overview (continued) • Start immediately • Some important dates:

– Mon, Oct 17 • Finalize team • Project proposal due

– Mon, Nov 7 • Milestone due (2-3 pages)

– Tues, Dec 13 • Final program code and writeup submission

– Thurs, Dec 15 • Presentation

23-Sep-11 71

Lecture 1 -

Fei-Fei Li

• Original research ideas encouraged • Useful datasets:

– ImageNet (www.image-net.org) – PASCAL

• Need Fei-Fei’s approval – Email is the best way – Do it BEFORE Jan 27

23-Sep-11 72

Course Project: overview (continued)

Lecture 1 -

Fei-Fei Li

Grading policy

• Problem Sets: 40% – We have 5 problem sets – Homework 0: very important! (more details…) – Late policy

• 5 free late days – use them in your ways • Afterwards, 25% off per day late • Not accepted after 3 late days per PS

– Collaboration policy • Read the student code book, understand what is ‘collaboration’

and what is ‘academic infraction’

• Midterm Exam: 20% – In class: Mon, Oct 31

23-Sep-11 73

Lecture 1 -

Fei-Fei Li

Grading policy

• Course project: 40% – presentation: 5% – write-up: 10%

• clarity, structure, language, references: 3% • background literature survey, good understanding of the problem: 3% • good insights and discussions of methodology, analysis, results, etc.: 4%

– technical: 15% • correctness: 5% • depth: 5% • innovation: 5%

– evaluation and results: 10% • sound evaluation metric: 3% • thoroughness in analysis and experimentation: 3%

• A word about ‘the curve’

23-Sep-11 74