16-899A Pixels to Percepts

Instructors:• Alexei (Alyosha) Efros, 225 Smith Hall, CMU• .Lavanya Sharan, Disney Research Pittsburgh

Web Page:• http://graphics.cs.cmu.edu/courses/P2P/

Today

Instructors

Why This Course?

Why Perception?

Administrative stuff

Overview of the course

Old Lady or Young Girl?

Lavanya SharanPostdoctoral Researcher

Visual perception, both behavioral and computational aspects.

Computer Vision Human Visual Perception Computer Graphics

Light or dark?

Real or fake Oreo?

Material category?

Emotion?

Where does he land?GRADUATE WORK

RECENT WORK

Alexei “Alyosha” Efros

Associate Professor, Robotics & CSD

Ph.D 2003, from UC Berkeley (signed by Arnie!)

Research Fellow, University of Oxford, ’03-’04

Research Interests

Computer Vision, Computer Graphics, Data-driven Approaches

PhD Thesis on Texture and Action Synthesis

Smart Erase button in Microsoft Digital Image Pro:

Why This Course?A perfect storm:• I wanted to do this for a long time

• but felt I wasn’t qualified

• Lavanya happened to be in Pittsburgh• and she is more than qualified

• The first-ever textbook on Perception for CS types (Thompson et al) is coming out• We will be giving out printouts soon

• There are great perception courses in Psych dept (e.g. by Klatzky), but our aims are different:• The aim to be much more applied• There are real problems in vision/graphics that we can solve• Exposure for these who might never take perception otherwise

Why study Perception?

• It’s fascinating!• Ok, but so is astronomy

• It’s still in the “early days”• Relatively easy to get up to speed• Still lots of low-hanging fruit

OK, OK, but why should I care? • I just want to create a seeing robot • and/or create the next “Avatars”

Two answers: 1) a classic one and 2) a real one

Classic Answer

Understanding Human Visual Perception should help us design better vision algorithms• After all, human is the only vision system known to work

Just like understanding how birds fly should help us design airplanes• …WAIT A MINUTE• Ok, so there is a fine line between “helping” and “mindless

copying” • Beware of papers claiming to be “biologically inspired”

The Real Answer

Human Perception is an integral part of most vision and graphics endeavors

i.e. you can’t get away from it!

“What does it mean, to see? The plain man's answer (and Aristotle's, too). would be, to know what is where by looking.”

-- David Marr, Vision (1982)

Vision: a split personality“What does it mean, to see? The plain man's answer (and

Aristotle's, too). would be, to know what is where by looking. In other words, vision is the process of discovering from images what is present in the world, and where it is.”

Answer #1: pixel of brightness 243 at position (124,54)

…and depth .7 meters

Answer #2: looks like bottom edge of whiteboard showing at the top of the image

Which answer is better?

Is the difference just a matter of scale?

depth map

Measurement vs. Perception

Measurement vs. Perception

Proof!

Measurement vs. Perception

Pablo PicassoThe Guitar Player

Vision as Measurement

Real-time stereo on Mars

Structure from Motion

Physics-based Vision

Virtualized Reality

Vision as Understanding

• Object / Scene Recognition• Action / Activity Recognition• even scene geometry

Perception in Graphics

• One of central goals of graphics is to simulate visual experience as realistically as possible

• So, if we could simulate the physics of light perfectly, why bother with perception?

1) Practical Reasons

2) Artistic Reasons

Complexity

•Amazingly real•But so sterile, lifeless… Why?

Device Limitations

• Low Dynamic Range• Limited modalities (visual + audio)

• No smell, taste, temperature

• Limited field of view

Artistic Reasons

• Reality is boring• Visual content

creators (film directors, game designers) want hyper-reality

How to study Perception?

Ok, you convinced me: Perception is useful.

But why not just “open up the brain” and figure out how it works?

Two most popular methods in physiology:• Cell recordings

• Recordings from small number of cells (1-100, out of millions)

• fMRI• Overall activity in the brain as function of time• Very low spatial resolution

Tale of Martians with an old PC

Cell recordings fMRI

Instead, in this course…

Course Goals

Read some interesting papers together• Learn something new: both you and us!

Get up to speed on big chunk of perception research

Use perception / human studies in your own research

Try your hand at a perception study

Learn how to speak

Learn how think critically about papers

Course Motto: 50/50

50% Lectures / 50% student presentations

50% textbook / 50% research papers

50% from perception side / 50% from vision/graphics side

50% Alyosha / 50% Lavanya

Course Organization

Requirements:1. Class Participation (25%)

• Keep annotated bibliography• Post on the Class Blog before each class • Ask questions / debate / flight / be involved!

2. Three Projects (75%)• Two Analysis Projects (25% + 25%)

• Implement / Evaluate a paper and present it in class• 1 CS paper, 1 perception paper

• Synthesis (Final) Project (25%)• Produce a publishable result (study or implementation)• Can be continuation of analysis project or something

new• Can be done solo or in groups of 2

Class ParticipationKeep annotated bibliography of readings (always a good

idea!). The format is up to you. At least, it needs to have:• Summary of key points• A few Interesting insights, “aha moments”, keen observations,

etc.• Weaknesses. Unanswered questions. Areas of further

investigation, improvement.

Before each class:• Submit your summary for current readings in

hard copy (printout/xerox)• Submit a comment on the Class Blog

• ask a question, answer a question, post your thoughts, praise, criticism, start a discussion, etc.

Analysis Project1. Pick a paper / set of papers from the list2. Understand it as if you were the author

• e.g. Re-implement it; try to replicate the study• If there is code, understand the code completely• Run it on data the same data (you can contact authors for data and

even code sometimes)

3. Understand it better than the author• Run it on LOTS of new data (e.g. LabelMe dataset, Flickr dataset,

etc, etc), or try to replicate the study with new data• Figure out how it succeeds, how it fails, where it fails, and, most

importantly WHY it fails• Look at which parts of the code do the real work, and which parts

are just window-dressing• Maybe suggest directions for improvement.

4. Prepare an amazing 45 min presentation• Discuss with us twice – once when you start the project, and 3 days

before the presentation

Synthesis Project

Can grow out of analysis project, or your own research

1-2 people per project.

Project proposals mid-semester

Project presentations at the end of semester.

Results presented as a CVPR-format paper.

Hopefully, a few papers may be submitted to CVPR / VSS etc

End of Semester Awards

We will vote for:• Best Analysis Project• Best Synthesis Project• Best Blog Comment

Prize: dinner in a French restaurant in Paris (transportation not included!) or some other worthy prizes

Course Outline (cont.)

• Faces (week 11)• Visual Attention (week 12)• Motion Perception (week 13)• Visual Realism (week 14)• Perception and Art (week 15)• Final Project Presentations

Subject to Change

Perception & Art

Livingstone