mitsubishi electric research laboratoriesspecial effects in the real worldraskar 2006 ramesh raskar...

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006

Ramesh RaskarMitsubishi Electric Research Labs (MERL)

Cambridge, MA, USA

Special Effects in the Real WorldSpecial Effects in the Real World


Parent Organization: Parent Organization: MELCOMELCO

• Mitsubishi Electric (MELCO), including subsidiaries– $32B in annual sales– 125,000 people– 10 business sectors:

• Space Development and Satellite Communications• Communications and Information Processing• Visual Equipment and Systems• Electronic Devices• Home Electronics• Energy• Industrial Equipment• Public-Use Systems• Transportation


MERLMERL

• Mitsubishi Electric Research Laboratories– Dick Waters, President & CEO

• Two labs, co-located in Cambridge, MA:– MERL Technology Lab

• Advanced systems development• ~35 researchers

– MERL Research Lab• Joe Marks, Director• Basic research in applied computing• 23 researchers, interns, consultants• CV, HCI, Multimedia, CG, AI


Special EffectsSpecial Effects

– Photorealism and Animation •Stays on screens•Yet to affect daily life in real time

– Post-processed content– Unlike internet,databases, UI


Changing AppearanceChanging Appearance

Virtual light source


Changing Virtual IlluminationChanging Virtual Illumination


Special Effects in the Real WorldSpecial Effects in the Real World

– Photorealism and Animation •Limited to display screens•Yet to affect daily life in real time

– Unlike other fields of info-tech

– Fusion: real world with graphics•Next big challenge in CG ?•New opportunities in advt, gaming, animation•Believable co-existence


Changing AppearanceChanging Appearance

Projector Projector

Virtual light source


Changing Virtual IlluminationChanging Virtual Illumination

Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)


Virtual MotionVirtual Motion


Virtual MotionVirtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)


Dynamic AugmentationDynamic Augmentation

Projecting on Tracked Objects


Virtual Reflectance Virtual Illumination

InteractionVirtual Motion

ShaderLampsShaderLamps

www.ShaderLamps.com


Projector-based AugmentationProjector-based Augmentation

www.ShaderLamps.com

Virtual Reflectance Virtual Illumination

InteractionVirtual Motion


Poor Man’s PalacePoor Man’s Palace

Surround CG for Daily Environment


Maya: World is an IllusionMaya: World is an Illusion

• Fake World– We all live in one

• Real-life Fakes– Not just photos and videos but surroundings

• Privacy


Augmented Reality (AR)Augmented Reality (AR)

With See-through Head-mounted Display

© Andrei State

Superman X-ray vision


hand-helddisplay

spatial see-throughdisplay

head-mounted display

retinal display

Head-attached Body-attached spatial

projector

projector

projector

Spatial Augmented Reality

Classification of Augmented RealityClassification of Augmented Reality


Being ThereBeing There

• Recreate Large Environments– ‘BeingThere’, walk-around– Museums, Exhibitions, Training

Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.


Desired Virtual Model

© Andrei State

Projected Guidance for Placement


Head-mounted Display Projectors

© Andrei State

Augmented RealityAugmented Reality


OpportunitiesOpportunities

• Entertainment- Animating objects, Museums, Special Effects, Laser Shows

• Advertising- Car showrooms, Furniture store

• Design and Ergonomics- Prototyping products, visualize material changes

• Information Augmentation- Office, factory or assembly Training


Spatially Augmented Reality Spatially Augmented Reality (SAR)(SAR)

HMD-VR Spatially Immersive-VR

AR using HMDSpatially Augmented

Reality

e.g. CAVE

Video or Optical see-through

SAR, Shaderlamps


StepsSteps

– Preprocessing• Scan 3D object• Roughly align projectors• Compute correction

– Run time• Render images of 3D model• Correct


StepsSteps

– Preprocessing• Scan 3D object and create virtual model

Faro arm


StepsSteps

– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose


StepsSteps

– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose

– Run-time• Inverse Rendering of 3D Models


Shape cue allows perceptual unwrappingShape cue allows perceptual unwrapping


‘Wallpaper the surface’‘Wallpaper the surface’

Shape Adaptive ProjectionShape Adaptive ProjectionMinimum Stretch DisplayMinimum Stretch Display


Shape Adaptive ProjectionShape Adaptive ProjectionMinimum Stretch DisplayMinimum Stretch Display

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,

SIGGRAPH 2003


Spatially Augmented RealitySpatially Augmented Reality

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,

SIGGRAPH 2003


Mitsubishi Pocket ProjectorMitsubishi Pocket Projector


A new way to play tic-tac-toeA new way to play tic-tac-toe

• Handheld projector.

• Grid projects to a fixed position on the wall.

• Cursor is guided by pointing the projector.


Pocket Projector GamesPocket Projector Games


WarehousingWarehousing

RoutingRouting

Library Library

Baggage

handling

Baggage

handling

CurrencyCurrency

Livestock trackingLivestock tracking

RFID


Micro Controller

Memory Computer

READER

Micro Controller

Memory

Conventional Passive RFID


Tagged Books in a LibraryTagged Books in a Library Id : List of books in RF range

No Precise Location Data Are books in sorted order ?

Which book is upside down ?


Prototype TagPrototype Tag

RF tag + photosensor

RF tag + photosensor


Find tag location using

handheld Projector

Find tag location using

handheld Projector

Photosensing

Wireless Tags

Photosensing

Wireless Tags

Many geometric ops Many geometric ops

R F I R F I

R F I DR F I D

Interactive stabilized projection

Interactive stabilized projection

(Radio Frequency Id & Geometry)(Radio Frequency Id & Geometry)

GG

Siggraph 2004Siggraph 2004


AR with Photosensing RFID AR with Photosensing RFID and Handheld Projectorand Handheld Projector


Visual feedback of 2D positionVisual feedback of 2D position

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions


RFID

(Radio Frequency Identification)

RFIG

(Radio Frequency Id and Geometry)


CurvedPlanar Non-planar

SingleProjector

MultipleProjectors

Projector

jUser : T

?

Pocket-ProjObjects

Projectors and SurfacesProjectors and Surfaces

2000

2000 2002

2001

1999

2002

20031999

1998


Augmentation of “five senses”Augmentation of “five senses”

• Visual Displays• Auditory Displays• Haptic Displays• Locomotion Interfaces• Olfactory Displays• Taste Displays ?• Sixth Sense ..

Extendingsensorymodalities


Sense of BalanceSense of Balance

Galvanic Vestibular Stimulation

Masahiko Inami (UEC)


Scent DisplayScent Display

Tracking nose direction

Aiming atthe nose

Air cannon

Deliveringscented air byvortex rings Camera

Platform

Yasuyuki Yanagida, ATR

Spatio-Temporal Controlled Delivery of Scent


Air CannonAir Cannon

Generating a doughnut-shaped vortex ring

Aperture

Push


DisplaysDisplays ContendersContenders

OrganicLED

LightEmittingPolymers

E-Ink


HolodustHolodustPerlin et alPerlin et al


Fake WorldFake World


Poor Man’s PalacePoor Man’s Palace

Surround CG for Daily Environment


RecapRecap

• Special Effects in Real World– Photorealism yet to impact daily life– Poor Man’s Palace

• Spatial Augmented Reality– Solution for fusion: Projectors, RFID, Sensors– Games for pocket projector– Display anywhere for advertising

• Future Problems– Natural phenomenon, complex reflectance, other displays– All senses: haptic, olfactory, preprioception

• Next Challenge in Real World Animation– Movies and games in our surroundings– 3D Advertising


Spatial Augmented RealitySpatial Augmented Reality

• Graphics– Photorealism yet to impact daily life– Special effects in real world

• Spatial Augmentation– Solution for fusion– Needs hardware components– Projectors, RFID, Sensors

• Next Challenge in Animation

raskar.com

mitsubishi electric research laboratoriesspecial effects in the real worldraskar 2006 ramesh raskar...

Documents