COSC 426 A d R liCOSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 2011

Lecture 1Lecture 1

Mark BillinghurstPhD Electrical EngineeringPhD Electrical Engineering

University of WashingtonInteraction DesignInteraction Design

Museum experiencesTools for designersTools for designers

Augmented RealityMobile AR Evaluation Mobile AR, Evaluation, Multimodal Interfaces, Collaborative

CollaborationCollaborationEnhanced FtF and remote collaborationSocial networkingSocial networking

OverviewOne tow hour lectures weekOne tow hour lectures week

Friday 9 – 11amYou will learnYou will learn

Introduction to Augmented RealityAugmented Reality technologyAugmented Reality technologyAR Interaction techniquesInteraction DesignInteraction DesignAR authoring toolsResearch directions in AR

Complete a simple project

C O lCourse OutlineWk 1 (July 15th): Introduction to AR(J y )Wk 2 (July 22nd): AR TechnologyWk 3 (July 29th): AR Developer Tools(J y ) pWk 4 (Aug 5th): AR Interaction TechniquesWk 5 (Aug 12th): AR ApplicationsWk 5 (Aug 12 ): AR ApplicationsWk 6 (Aug 19th): Outdoor and Mobile ARWk 7,8 (Aug 26th, Sept 2nd): Holidays Wk 7,8 (Aug 26 , Sept 2 ): Holidays Wk 9 (Sept 9th): Collaborative AR Wk 10 (Sept 16th): Usability Testing Wk 10 (Sept 16 ): Usability Testing Wk 11 (Sept 23rd): AR research DirectionsWk 12 (Sept 30th): Final Project Presentations Wk 12 (Sept 30 ): Final Project Presentations

Assessment - Update

Research project – 40%Group work (2-4 people)Group work (2-4 people)Due Sept 30th

Two Class Assignments – 20 %Programming assignments, individual workg g g ,

Final Exam – 40%E k O 3 d 14thExam week Oct 3rd – 14th

IntroductionIntroduction

A Brief History of Time

T dTrendsmaller, cheaper, more functions, more intimate

T h l b blTechnology becomes invisibleIntuitive to use I f i lInterface over internalsForm more important than functionH t d d iHuman centered design

A B f H f CA Brief History of Computing

Trendsmaller, cheaper, faster, more intimate, intelligent objects

Computers need to become invisiblehide the computer in the real world

- Ubiquitous / Tangible Computing

put the user inside the computer Vi l R li- Virtual Reality

Invisible Interfaces

Jun Rekimoto, Sony CSL

Virtual Reality

19891989…

Cheap HMDs

Virtual Reality

Immersive VR Head mounted display, glovesSeparation from the real worldSeparation from the real world

VR Today

$3-5 Billion VR business (+ > $150 Billion Graphics Industry)Visualization, simulation, gaming, CAD/CAE, multimedia, graphics arts

1977 – Star Wars

2008 - CNN

Augmented Reality DefinitionD f Ch [A 97]Defining Characteristics [Azuma 97]

Combines Real and Virtual Images- Both can be seen at the same time

Interactive in real-time- The virtual content can be interacted with

Registered in 3Dg- Virtual objects appear fixed in space

Augmented Reality Examples

Put AR pictures here

AR vs VR

Virtual Reality: Replaces RealityScene Generation: requires realistic imagesq gDisplay Device: fully immersive, wide FOVTracking and Sensing: low accuracy is okayTracking and Sensing: low accuracy is okay

Augmented Reality: Enhances RealityScene Generation: minimal rendering okayDisplay Device: non-immersive small FOVDisplay Device: non immersive, small FOVTracking and Sensing: high accuracy needed

Milgram’s Reality-Virtuality continuum

Mixed Reality

Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment

Reality - Virtuality (RV) Continuum

Metaverse

Neal Stephenson’s “SnowCrash”The Metaverse is the convergence of: The Metaverse is the convergence of:

1) virtually enhanced physical reality 2) h ll l 2) physically persistent virtual space

Metaverse Roadmapphttp://metaverseroadmap.org/

Metaverse Dimensions• Augmentation technologies that layer information onto

our perception of the physical environment.• Simulation refers to technologies that model reality • Intimate technologies are focused inwardly, on the Intimate technologies are focused inwardly, on the

identity and actions of the individual or object;• External technologies are focused outwardly towards • External technologies are focused outwardly, towards

the world at large;

Metaverse Components

Four Key ComponentsVirtual WorldsAugmented RealityMirror WorldsMirror WorldsLifelogging

Mirror Worlds

Mirror worlds are informationally-enhanced virtual models of the physical world.p y

Google Earth, MS Street View, Google Maps

LifeLogging

Technologies record and report the intimate states and life histories of objects and usersj

Nokia LifeBlog, Nike+, FitBits

Steve Mann - LifeLogginggg g

Gordon Bell: LifeLogging

1 TB to store 65 years of data1 TB to store 65 years of data

Summary

Augmented Reality has three key featuresCombines Real and Virtual ImagesgInteractive in real-timeRegistered in 3DRegistered in 3D

AR can be classified alongside other technologiesMilgrams Mixed Reality continuumMetaVerse MetaVerse

AR HistoryAR History

A Brief History of AR (1)1960’ S h l d / S ll’ 1960’s: Sutherland / Sproull’s first HMD system was see-throughthrough

A Brief History of AR (2)

1960 - 70’s: US Air Force helmet mounted displays (T Furness)displays (T. Furness)

A Brief History of AR (3)

1970 - 80’s: US Air Force Super Cockpit (T. Furness)p p ( )

A Brief History of AR (4)

990’ “A ” Early 1990’s: Boeing coined the term “AR.” Wire harness assembly application begun (T. Caudell, D. Mizell). E l t id 1990’ UNC lt d i li ti j tEarly to mid 1990’s: UNC ultrasound visualization project

A Brief History of AR (5)

1994: Motion stabilized display [Azuma]1994: Motion stabilized display [Azuma]1995: Fiducial tracking in video see-through [Bajura / Neumann]1996: UNC hybrid magnetic-vision trackery g

A Brief History of AR (6)

1996: MIT Wearable Computing efforts1998: Dedicated conferences beginL 90’ C ll b i d i iLate 90’s: Collaboration, outdoor, interactionLate 90’s: Augmented sports broadcasts1998 - 2001: Mixed Reality Systems Lab1998 - 2001: Mixed Reality Systems Lab

History Summary

1960’s – 80’s: Early Experimentation1980’s – 90’s: Basic Research1980 s 90 s: Basic Research

Tracking, displays

1995 – 2005: Tools/ApplicationsInteraction, usability, theoryy y

2005 - : Commercial ApplicationsG M di l I dGames, Medical, Industry

Gartner Hype Cycle

2007 - AR Reaches Mainstream

MIT Technology ReviewMarch 2007list of the 10 most exciting technologiesexciting technologies

EconomistDec 6th 2007Dec 6th 2007Reality, only better

Gartner’s top 10 disruptive technologies 2008-2012:Multicore and hybrid processors Virtualisation and fabric computing S i l k d i l f Social networks and social software Cloud computing and cloud/Web platforms Web mash s Web mashups User Interface Ubiquitous computing Ubiquitous computing Contextual computing Augmented reality Augmented reality Semantics

2009 - AR in Magazines

Esquire MagazineDec 2009 issue12 pages AR content

M OthMany OthersWiredColorsRed BullRed BullEtc

Google Searches for AR

2008 - Browser Based ARFlash + camera + 3D graphicsHigh impactg p

High marketing value

Large potential install baseLarge potential install base1.6 Billion web users

Ease of developmentEase of developmentLots of developers, mature tools

L f Low cost of entryBrowser, web camera

Impact of Web-based AR

Boffswana Living Sasquatchhttp://www.boffswana.com/news/?p=605p p

In first month100K 100K unique visits 500K page views6 minutes on page

Mobile Phone ARMobile Phones

cameracameraprocessord ldisplay

AR on Mobile PhonesSimple graphicsOptimized computer visionOptimized computer visionCollaborative Interaction

AR Advertising (HIT Lab NZ 2007)

T d l d AR li i (200K)Txt message to download AR application (200K)See virtual content popping out of real paper advertTested May 2007 by Saatchi and SaatchiTested May 2007 by Saatchi and Saatchi

2008: Location Aware Phones

Nokia NavigatorMotorola Droid

2009 - Outdoor Information OverlayM bil h b dMobile phone basedTag real world locations g

GPS + Compass inputOverlay graphics data on live videoOverlay graphics data on live video

ApplicationsTravel guide, Advertising, etc

Wikitude, Layar, Junaio, etc.., y , J ,Android based, Public API released

Layar (www.layar.com)

Location based dataGPS + compass locationpMap + camera view

AR L l ldAR Layers on real worldCustomized dataAudio, 3D, 2D content

Easy authoringEasy authoringAndroid, iPhone

$784 million USD in 2014

Some Commercial AR Companies

ARToolworks (http://www artoolworks com/)ARToolworks (http://www.artoolworks.com/)ARToolKit, FLARToolKit, SDKs

Metaio (http://www.metaio.com/)Marketing, Industry, SDKsMarketing, Industry, SDKs

Total Immersion (http://www.t-immersion.com/)Marketing, Theme Parks, AR Experiences

Qualcomm Qua co (http://developer.qualcomm.com/dev/augmented-reality)

Mobile AR QCAR SDKMobile AR, QCAR SDK

FLARToolKit (ARToolworks)

Flash version of ARToolKitFLARToolKit + Papervison3D = AR on the webFLARToolKit + Papervison3D = AR on the webExample - GE Ecomagination Website

1.2 million views on YouTubehttp://ge.ecomagination.com/smartgrid/#/augmented_realityp g g g g y

Software Platform (Metaio)

Unifeye software platformModular platform for building AR applicationsp g pp

Markerless Tracking (Total Immersion)

D’Fusion markerless tracking platform Turn key solutiony

Example - Topps baseball cards AR h f l b b ll dAR characters on top of real baseball cards

Summary

Augmented Reality has a long history going back to the 1960’sInterest in AR has exploded over the last two years and is being commercialized quicklyyears and is being commercialized quicklyAR is growing in a number of areas

Mobile ARWeb based ARWeb based ARAdvertising experiences

Sample AR ApplicationsSample AR Applications

Applications

MedicineManufacturingInformation overlayInformation overlayArchitectureMuseumMarketingMarketingGaming

Applications: medical“X-ray vision” for surgeonsAid visualization, minimally-invasive operations. Aid visualization, minimally invasive operations. Training. MRI, CT data.

Ul d j UNC Ch l HillUltrasound project, UNC Chapel Hill.

CourtesyUNCChapelHill

Medical AR TrialsSauer et al. 2000 at Siemens Corporate Research, NJStereo video see throughStereo video see through

F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,Single Camera Tracker: System Design and Needle Biopsy Phantom Trial,MICCAI 2002

Assembly and maintenance © 1996 S. Feiner, B. MacIntyre, & A. Webster, Columbia University

© 1993 S F i B M I t &© 1993 S. Feiner, B. MacIntyre, & D. Seligmann, Columbia University

PS3 - Eye of Judgment (2007)C t Vi i T kiComputer Vision TrackingCard based battle gameC ll b ti ARCollaborative AROctober 24th 2007

AR Books – Markerless Tracking

AR Jam story bookAR Jam story book

AR A t tiAR Annotations

Columbia University

© 1993 S. Feiner, B. MacIntyre, M. Haupt, & E. Solomon,C l bi U i i

© 1997 S. Feiner, B. MacIntyre, T. Höllerer, & A. Webster,

Columbia University, ,

Columbia University

HRL

Applications: Annotating Environment

P bli d i iPublic and private annotationsAid recognition, “extended memory”g , y

The World = Your User Interface (Feiner)

E E h Al

© 2002, Scientific American

Everyone, Everywhere, Always

Application: Broadcast Augmentation

Adding virtual content to live sports broadcasts“First down” line in American footballFirst down line in American footballHockey puck trails, virtual advertisementsNational flags in swimming lanes in 2000 OlympicsNational flags in swimming lanes in 2000 Olympics

Commercial applicationPrinceton Video Image is one company

- http://www.pvi.tv/

Broadcast Examples

Interactive Museum ExperiencesBlackMagic

Virtual America’s Cup410,000 people in six months

MagicPlanetTeManawa science museumVirtual AstronomyCollaborative AR experience

AR VolcanoInteractive AR kioskScienceworks museum, Melbourne

Museum Archeology

LifePlus (2002-2004) Natural feature trackinggVirtual charactersMobile AR systemMobile AR system

Archeoguide (2000-2002)Cultural heritage on-site guideHybrid trackingHybrid trackingVirtual overlay

SummaryAR h l b d d l d AR technology can be used to develop a wide range of applicationsPromising application areas include

GamesGamesEducationEngineeringEngineeringMedicineMMuseumsEtc..

AR Experience DesignAR Experience Design

“The product is no longer The product is no longer the basis of value The the basis of value. The

experience is ”experience is.

Venkat RamaswamyThe Future of Competition.

Gilmore + Pine: Experience EconomyGilmore Pine: Experience Economy

E tiexperiences Emotion

services

productsproducts

components Function

Sony CSL © 2004

The Value of Good User Experience

5050c

$3.50

20c

Good Experience DesignReactrixReactrix

Top down projectionCamera based inputCamera based inputReactive GraphicsNo instructionsNo instructionsNo training

Using the N-gage

SideTalkinghttp://www.sidetalkin.com

Interaction Design

“Designing interactive products to support people in their everyday and working lives”everyday and working lives

Preece, J., (2002). Interaction Design

Design of User Experience with TechnologyHigher in the value chain than product designHigher in the value chain than product design

Interaction Design involves answering three questions:What do you do? - How do you affect the world?What do you feel? – What do you sense of the world?What do you know? – What do you learn?

Interaction Design Process

Who are your Users?

Everyone!y

Designing for Everyone

Designing for Everyone pleases No one

Understanding Specific Needs

Interaction Design is All About You

Users should be involved throughoutinvolved throughout the Design ProcessConsider all the needs Co s de a t e eedsof the user

B ildi C lli AR E iBuilding Compelling AR Experiences

experiencesUsability

applications Interaction

tools Authoringtools Authoring

components Tracking, Display

Summary

In order to build AR applications you need to focus on the user experiencepGreat user experience is based on

L l l AR h lLow level AR component technologyAuthoring toolsApplication/Interaction designUser experience textingUser experience texting