computer graphics & computer vision

7/28/2019 Computer Graphics & Computer Vision

1/49

Page 1

Computer Graphics and Computer Vision

Professor Demetri Terzopoulos

Graphics and Vision: A Unified ViewGraphics and Vision: A Unified ViewThis major field is about Computers and ImagesThis major field is about Computers and Images

Computer Graphics (CG)Computer Graphics (CG)

Computational models images and videosComputational models images and videos

SynthesisSynthesis

Forward problemForward problem

Computer Vision (CV)Computer Vision (CV)

Images and videos computational modelsImages and videos computational models

AnalysisAnalysis

Inverse problemInverse problem


2/49

Page 2

History of Computer Graphics and Computer VisionHistory of Computer Graphics and Computer Vision

Two PhD projects at MIT in the early 1960sTwo PhD projects at MIT in the early 1960s Ivan E. Sutherland, 1963Ivan E. Sutherland, 1963

Sketchpad, a manSketchpad, a man--machine graphical communication systemmachine graphical communication system

Lawrence G. Roberts,Lawrence G. Roberts, 19631963

Machine perception of threeMachine perception of three--dimensional solidsdimensional solids

CG and CV have developed as independent fieldsCG and CV have developed as independent fields

,,

Exciting synergies and crossExciting synergies and cross--fertilization with other fieldsfertilization with other fields Physics, biology, cognitive science, artificial intelligence, art, .Physics, biology, cognitive science, artificial intelligence, art, .

What is an Image / Video?What is an Image / Video?Array of pixels (Array of pixels (oneone orormmoorree numbers)numbers)

A video is a time sequence of imagesA video is a time sequence of images

How they are formed:How they are formed:

Objects in the world (static or dynamic)Objects in the world (static or dynamic)

Illumination (light sources)Illumination (light sources)

Imaging device (eye, camera)Imaging device (eye, camera)

Illumination

Camera

We want to synthesize and analyze themWe want to synthesize and analyze them


3/49

Page 3

Why? Images and movies are everywhere!Why? Images and movies are everywhere! Entertainment: Motion pictures & GamesEntertainment: Motion pictures & Games

Virtual worlds (especially for movies and games)Virtual worlds (especially for movies and games)

Industrial designIndustrial design

Scientific and medical visualizationScientific and medical visualization

HumanHuman--computer interactioncomputer interaction

Fine artsFine arts

Etc.Etc.

RoboticsRobotics

AutomotiveAutomotive

Visual Surveillance / BiometricsVisual Surveillance / Biometrics

Industrial inspectionIndustrial inspection

Medical imagingMedical imaging

Remote sensingRemote sensing

Image and video retreivalImage and video retreival Etc.Etc.

For a deeper understanding of the physical world and living systems, including the human brainFor a deeper understanding of the physical world and living systems, including the human brain

Computer GraphicsComputer GraphicsThe art and science of creating imagery by computerThe art and science of creating imagery by computer

Three main research themesThree main research themes

ModelingModeling

How do we model (mathematically represent) objects?How do we model (mathematically represent) objects?

How do we construct models of specific objects?How do we construct models of specific objects?

AnimationAnimation

How do we represent the motions of objects?How do we represent the motions of objects?

How do we ive animators control of this motion?How do we ive animators control of this motion?

RenderingRendering

How do we simulate the realHow do we simulate the real--world behavior of light?world behavior of light?

How do we simulate the formation of images?How do we simulate the formation of images?


4/49

Page 4

Standard Display DevicesStandard Display Devices

PlasmaLCD

(Liquid Crystal Display)

CRT

(Cathode Ray Tube)

Exotic Display DevicesExotic Display Devices

ImmersiveImmersiveImmersiveImmersive HeadHead--MountedMountedHeadHead--MountedMounted

AutostereoscopicAutostereoscopicAutostereoscopicAutostereoscopic VolumetricVolumetricVolumetricVolumetric


5/49

Page 5

MoviesMovies

To reality and beyond !To reality and beyond !

MoviesMoviesSpecial effectsSpecial effects


6/49

Page 6

MoviesMovies

Digital CompositingDigital Compositing


7/49

Page 7

Digital compositingDigital compositing

Digital CompositingDigital Compositing


8/49

Page 8

CartoonsCartoons

GamesGamesFocus on interactivityFocus on interactivity


9/49

Page 9

GamesGames

Metal GearMetal Gear

Solid 4Solid 4


10/49

Page 10

ComputerComputer--Aided DesignAided Design

Precision modelingPrecision modeling

Engineering visualizationEngineering visualization

ComputerComputer--Aided DesignAided DesignIts not just about visualizationIts not just about visualization

Simulation is also usefulSimulation is also useful


11/49

Page 11

Visualization: ScientificVisualization: Scientific

Molecular Biology

Visualization: MedicalVisualization: Medical


12/49

Page 12

Visualization: ArchitecturalVisualization: Architectural

http://www.diamondschmitt.com/

Visualization: InfoVisualization: Info

Smith and Fiore


13/49

Page 13

Graphical User InterfacesGraphical User Interfaces

3D User Interfaces3D User Interfaces

Steven Schkolne

Sculpture Art


14/49

Page 14

Digital ArtDigital Art

Genetically evolvedGenetically evolved

Carl Sims

Digital ArtDigital Art

The Highest-Grossing Film of All Time, Jason Salavon^2nd


15/49

Page 15

Elements of CGElements of CG

The graphics pipelineThe graphics pipeline

Modeling Animation Rendering

ModelingModelingRepresenting objects geometrically on a computerRepresenting objects geometrically on a computer

Polygon meshes

Texture maps

ur ace pa c es

Parametric curves


16/49

Page 16

ModelingModeling

Alternative representationsAlternative representations

Voxels

Blobs

Subdivision surfaces

ModelingModelingAltering geometric modelsAltering geometric models

Affine transform

Ying, Kristjansson, Biermann, Zorin


17/49

Page 17

Scanning ShapesScanning Shapes

Digital Michaelangelo Project

Plant ModelingPlant ModelingP. Prusinkiewicz, Univ. of CalgaryP. Prusinkiewicz, Univ. of Calgary


18/49

Page 18

Plant ModelingPlant Modeling

RenderingRenderingKey elementsKey elements

Illumination

Geometric Shape& Reflectance

Camera

Pixels


19/49

Page 19

Camera ModelCamera Model

RenderingRenderingDraw visible surfaces onto displayDraw visible surfaces onto display

Camera(view direction)

ImagePlane

Clipping ScanConversion

Camera(eye position)


20/49

Page 20

Reflectance ModelingReflectance Modeling

Reflectance ModelingReflectance Modeling

Diana the Huntress, Henrik Wann Jensen


21/49

Page 21

Subsurface ScatteringSubsurface Scattering

Translucency and varied levels of light penetration can be created usingTranslucency and varied levels of light penetration can be created usingsubsurface scattering effects (nVIDIA)subsurface scattering effects (nVIDIA)

RenderingRenderingNonNon--photorealistic renderingphotorealistic rendering

Aaron Hertzmann


22/49

Page 22

Rendering: InformationRendering: Information

Agrawala and Stolte Mapblast.com

AnimationAnimationKeyframe animationKeyframe animation

Keyframing


23/49

Page 23

AnimationAnimation

Motion captureMotion capture

Animating Golem in LOTRAnimating Golem in LOTR


24/49

Page 24

Markerless Motion CaptureMarkerless Motion Capture

Max Plank Institute of Informatics, GermanyMax Plank Institute of Informatics, Germany

AnimationAnimationPhysical simulationPhysical simulation

Dynamics R. Fedkiw et al., Stanford U.


25/49

Page 25

Atlanta in MotionAtlanta in Motion

J. Hodgins, et al.,J. Hodgins, et al.,Georgia TechGeorgia Tech

NBCNBC --Los AngelesLos AngelesCoverage of Prof. Petros Faloutsos digital stuntmanCoverage of Prof. Petros Faloutsos digital stuntman


26/49

Page 26

Falling SidewaysFalling Sideways

Falling Backward, Rolling Over, Rising, andFalling Backward, Rolling Over, Rising, and

Balancing in GravityBalancing in Gravity

and Iand I cancan get up!get up! Help, Ive fallen! Help, Ive fallen!


27/49

Page 27

Rising From a Supine PositionRising From a Supine Position

Virtual CelebrityVirtual CelebrityVirtual Celebrity Productions, LLCVirtual Celebrity Productions, LLC


28/49

Page 28

FacialFacialAnatomyAnatomy

Skin Model

Muscle Model

Epidermis

Dermis

Muscle Layer

RealReal--Time Facial SimulationTime Facial Simulation


29/49

Page 29

NeckNeck--HeadHead--Face AnimationFace Animation

AnimationAnimationBehavioral animationBehavioral animation


30/49

Page 30

Evolved Virtual CreaturesEvolved Virtual Creatures

Karl Sims, 1994Karl Sims, 1994

Genetic algorithmsGenetic algorithms

Mimic biological evolutionMimic biological evolution

AnimationAnimation

Pixar: Luxo Jr. (1986)Pixar: Luxo Jr. (1986)


31/49

Page 31

AnimationAnimation

Example: Geris GameExample: Geris Game -- PixarPixar

AnimationAnimationThe AnimatrixThe Animatrix

Final Flight of the OsirisFinal Flight of the Osiris


32/49

Page 32

Reality is *Very* ComplexReality is *Very* Complex

Virtual RealityVirtual RealityArtificial life and other natural phenomenaArtificial life and other natural phenomena

Discuss laterDiscuss later


33/49

Page 33

Many Open Research ProblemsMany Open Research Problems

How to model/render/animate a complex scene?How to model/render/animate a complex scene?

Computer VisionComputer VisionRelated fieldsRelated fields

Image processingImage processing

Pattern recognitionPattern recognition

Visual perceptionVisual perception

Image understandingImage understanding

True IU seems to involve a great deal of human intelligenceTrue IU seems to involve a great deal of human intelligence

Automated s stems are still far from human erformanceAutomated s stems are still far from human erformance

Some good solutions in constrained special casesSome good solutions in constrained special cases

(e.g., inspection: IC manufacturing, circuit boards)(e.g., inspection: IC manufacturing, circuit boards)

Inverse problems are generally tougher to solveInverse problems are generally tougher to solve


34/49

Page 34

WHAT DO YOU SEE?WHAT DO YOU SEE?

AND NOW?AND NOW?


35/49

Page 35

Vision Research ThemesVision Research Themes

Edge and region extractionEdge and region extraction

Image segmentationImage segmentation

Shape from XShape from X

ContoursContours

ShadingShading

StereoStereo

MotionMotion

Object trackingObject tracking

Object recognition (including faces)Object recognition (including faces) Event and activity recognitionEvent and activity recognition

Interactive Image Analysis Using SnakesInteractive Image Analysis Using Snakes


36/49

Page 36

Retinal Angiogram SegmentationRetinal Angiogram Segmentation

Cartographic Modeling: RiversCartographic Modeling: Rivers

P. Fua. Fast, Accurate and Consistent Modeling of Drainage andSurrounding Terrain. International Journal of Computer Vision, 1998.


37/49

Page 37

Cartographic Modeling: Buildings and RoadsCartographic Modeling: Buildings and Roads

Automated Image RetreivalAutomated Image Retreival

Find pictures of theFind pictures of the

Sydney operaSydney opera

househouse

in a large databasein a large database

on the webon the web


38/49

Page 38

Space Robotics: Mars RoverSpace Robotics: Mars Rover

SelfSelf--Driving Cars: Intelligent TransportationDriving Cars: Intelligent TransportationProfessor Stefano SoattoProfessor Stefano Soatto


39/49

Page 39

Visual ServoingVisual Servoing

Tom Drummond, Univ. of Cambridge

ModelModel--Based Heart Motion TrackingBased Heart Motion Tracking


40/49

Page 40

3D Reconstruction3D Reconstruction

From image(s) to model(s)From image(s) to model(s) Inverting 3D projectionInverting 3D projection

ImageImage--based modelingbased modeling

Viewpoint

Image

3D model

Cooking with KurtCooking with KurtModelModel--based scenebased scene

reconstruction/animationreconstruction/animation


41/49

Page 41

Human Modeling from VideoHuman Modeling from Video

M. Dimitri jevic, S. Ilic, and P. Fua CVPR04, S. I lic, and P. Fua PAMI06M. Dimitri jevic, S. Ilic, and P. Fua CVPR04, S. I lic, and P. Fua PAMI06

PerformancePerformance--Based AnimationBased Animation[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]


42/49

Page 42

PerformancePerformance--Based AnimationBased Animation

[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]

PerformancePerformance--Based AnimationBased Animation[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]


43/49

Page 43

ALife Modeling PyramidALife Modeling Pyramid

Computational model of animals and humans

BehaviorBehavior

LearningLearning

CognitionCognition

PhysicsBiomechanics / LocomotionBiomechanics / LocomotionPerceptionPerception

Body

An Artificial Fish ModelAn Artificial Fish Model

Innate Characteristics

!Motor Controllers

Behavior Routines

Action SelectionLearning

OptimizationSensors

Perception

Motor

Display Model

Behavior

Biomechanical Model


44/49

Page 44

ALife Characters in Interactive GamesALife Characters in Interactive GamesThese autonomous characters are aliveThese autonomous characters are alive

CyberLifes Creatures (1996)

Nintendogs (2005)


45/49

Page 45

ALife for VisionALife for Vision

Deformable organisms for automated medical imageDeformable organisms for automated medical imagesegmentationsegmentation

Memory andprior knowledge

Plan or schedule

Skeleto

n

uscles

Interactions withother cr eatures

Musc le actuation causesshapedeformation

BrainPerception

Perceptual attentionmechanism

Memory andprior knowledge

Plan or schedule

Skeleto

n

uscles

Interactions withother cr eatures

Musc le actuation causesshapedeformation

BrainPerception

Perceptual attentionmechanism

Sensors

and limbsUnderlying

medial based Shaperepresentation

Sensors

and limbsUnderlying

medial based Shaperepresentation

fornix

2

3N-2

1

genu

spleniumrostrum

body

NN-1

upper/right

lower/leftfornix

2

3N-2

1

genu

spleniumrostrum

body

NN-1

upper/right

lower/left

Active Visual TrackingActive Visual Tracking

StereoRetinalImages

Retinal Backprojection ofColor Model

ObliqueView

TopView


46/49

Page 46

Active Vision SystemActive Vision System

Color Models

Stabilization

Module

Foveation

Module

Retinal ImageStream

Further VisualAnalysis

Optical flow minimization Color histogramintersection & backprojection

aze on ro

LocomotionControl

SelfSelf--Animating Virtual PedestriansAnimating Virtual Pedestrians

In a Virtual Train StationIn a Virtual Train Station (Wei Shao)(Wei Shao)

Autonomous pedestrian behaviorAutonomous pedestrian behavior

Old Penn Station in NYC

Concourses

Main Waiting Room


47/49

Page 47

Simulation of Pedestrian Activity in the Train StationSimulation of Pedestrian Activity in the Train Station

Virtual VisionVirtual Vision

Simulated visual sensor networksSimulated visual sensor networks


48/49

Page 48

Persistently Observing a PedestrianPersistently Observing a Pedestrian

CS Faculty Conducting CG and CV ResearchCS Faculty Conducting CG and CV Research

StanleyStanley OsherOsher (CV(CV -- Mathematics)Mathematics)

StefanoStefano SoattoSoatto (CV(CV -- Computer Science)Computer Science)

DemetriDemetri TerzopoulosTerzopoulos (CG & CV(CG & CV -- Computer Science)Computer Science)

ZhuowenZhuowen TuTu (CV(CV -- Neurology)Neurology)

AlanAlan YuilleYuille (CV(CV -- Statistics)Statistics)

SongSong--Chun Zhu (CVChun Zhu (CV -- Statistics)Statistics)


49/49

Courses OfferedCourses Offered

CS 174ACS 174A Introduction to Computer GraphicsIntroduction to Computer Graphics

CS 174BCS 174B ImageImage--Based Modeling and RenderingBased Modeling and Rendering

CS 174CCS 174C Computer AnimationComputer Animation

Plus several graduate courses in graphics and vision:Plus several graduate courses in graphics and vision:

Artificial Life for Computer Graphics and VisionArtificial Life for Computer Graphics and Vision

Machine VisionMachine Vision

Humanoid Character SimulationHumanoid Character Simulation

Building an Autonomous RobotBuilding an Autonomous Robot

Etc.Etc.

Thank you!Thank you!

computer graphics & computer vision

Documents