cosc 426 lect 2. - ar technology

138
L 2 AR T h l Lecture 2: AR Technology Mark Billinghurst [email protected] July 2011 COSC 426: Augmented Reality COSC 426: Augmented Reality

Upload: mark-billinghurst

Post on 18-May-2015

1.810 views

Category:

Technology


4 download

DESCRIPTION

Lecture 2 in the COSC 426 class on Augmented Reality. Taught by Mark Billinghurst

TRANSCRIPT

Page 1: COSC 426 Lect 2. - AR Technology

L 2 AR T h lLecture 2: AR Technology

Mark Billinghurst

[email protected]

July 2011

COSC 426: Augmented RealityCOSC 426: Augmented Reality

Page 2: COSC 426 Lect 2. - AR Technology

Key Points from Lecture 1

Page 3: COSC 426 Lect 2. - AR Technology

Augmented Reality DefinitionDefining Characteristics [Azuma 97]

Combines Real and Virtual Imagesg- Both can be seen at the same time

Interactive in real-time- Virtual content can be interacted with

Registered in 3Dg- Virtual objects appear fixed in space

Page 4: COSC 426 Lect 2. - AR Technology

What is not Augmented Reality?

Location-based servicesB d d i (QR d )Barcode detection (QR-codes)Augmenting still imagesg g gSpecial effects in movies…… but they can be combined with AR!… but they can be combined with AR!

Page 5: COSC 426 Lect 2. - AR Technology

Milgram’s Reality-Virtuality Continuum

Mixed Reality

Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment

Reality - Virtuality (RV) Continuum

Page 6: COSC 426 Lect 2. - AR Technology

Metaverse

Page 7: COSC 426 Lect 2. - AR Technology

AR History Summary

1960’s – 80’s: Early Experimentation1980’s – 90’s: Basic Research1980 s 90 s: Basic Research

Tracking, displays

1995 – 2005: Tools/ApplicationsInteraction, usability, theoryy y

2005 - : Commercial ApplicationsG M di l I dGames, Medical, Industry

Page 8: COSC 426 Lect 2. - AR Technology

Applications

MedicineManufacturingInformation overlayInformation overlayArchitectureMuseumMarketingMarketingGaming

Page 9: COSC 426 Lect 2. - AR Technology

Interaction Design Process

Page 10: COSC 426 Lect 2. - AR Technology

Interaction Design is All About You

Users should be involved throughoutinvolved throughout the Design ProcessConsider all the needs Co s de a t e eedsof the user

Page 11: COSC 426 Lect 2. - AR Technology

B ildi C lli AR E iBuilding Compelling AR Experiences

experiencesUsability

applications Interaction

tools Authoringtools Authoring

components Tracking, Display

Page 12: COSC 426 Lect 2. - AR Technology

AR Technology

Page 13: COSC 426 Lect 2. - AR Technology

Building Compelling AR ExperiencesBuilding Compelling AR Experiences

experiences

applications

toolstools

components Display, Tracking

Sony CSL © 2004

Page 14: COSC 426 Lect 2. - AR Technology

AR TechnologyKey Technologies

Displayp yTrackingInput

DisplayTracking

InputProcessing Processing

Input

Page 15: COSC 426 Lect 2. - AR Technology

AR Displays

Page 16: COSC 426 Lect 2. - AR Technology

AR Displays

AR

Primarily IndoorEnvironments

Primarily Outdoor(Daylight) Environments

Visual Displays

Virtual Images Projection CRT Display

Not Head-Mounted

Liquid Crystal

Head-MountedDisplay (HMD)

Cathode Ray Tube (CRT)or Virtual Retinal Display (VRD)

Head-MountedDisplay (HMD)

Projection DisplayNavigational Aids in Cars

Not Head Mounted(e.g. vehicle mounted)

e.g. windowreflections

seen off windows

e.g. Reach-In

using beamsplitter

e.g. Shared SpaceMagic Book

Displays LCDs

e.g. WLVAand IVRD

p y ( )Many Military Applications& Assistive Technologies

e.g. Head-UpDisplay (HUD)

gMilitary Airborne Applications

Page 17: COSC 426 Lect 2. - AR Technology

Head Mounted Displays

Page 18: COSC 426 Lect 2. - AR Technology

Head Mounted Displays (HMD)- Display and Optics mounted on Head- May or may not fully occlude real world- Provide full-color images- Considerations

• Cumbersome to wear• Brightness• Low power consumption• Low power consumption• Resolution limited• Cost is high?g

Page 19: COSC 426 Lect 2. - AR Technology

Types of Head Mounted Displays

OccludedSee-thru

Multiplexed

Page 20: COSC 426 Lect 2. - AR Technology

Immersive VR Architecture

head position/orientation

VirtualWorld

HeadTracker

head position/orientationNon see-thruImage source

& optics

Host P

Data BaseM d l

RenderingE i FrameProcessor Model Engine Buffer

to networkvirtualobject

DisplayDriver

Page 21: COSC 426 Lect 2. - AR Technology

See-thru AR Architecture

head position/orientationHead

Tracker

head position/orientationsee-thru

combinerreal world

Host P

Data BaseM d l

RenderingE i FrameProcessor Model Engine Buffer

to network Virtual Image DisplayDriver

superimposedover real world object

Image sourceg

Page 22: COSC 426 Lect 2. - AR Technology

Optical see-through head-mounted display

Virtual imagesfrom monitorsfrom monitors

RealWorld

OpticalCombiners

Page 23: COSC 426 Lect 2. - AR Technology

Optical See-Through HMD

Page 24: COSC 426 Lect 2. - AR Technology

Optical see-through HMDsVirtual Vision VCAP

Sony GlasstronSony Glasstron

Page 25: COSC 426 Lect 2. - AR Technology

DigiLensC HOECompact HOE

Solid state opticsSwitchable Bragg GratingStacked SBGFast switchingUltra compact

www.digilens.com

Page 26: COSC 426 Lect 2. - AR Technology

The Virtual Retinal Display

Image scanned onto retinaage sca e o to et aCommercialized through Microvision

Nomad System - www.mvis.comNomad System www.mvis.com

Page 27: COSC 426 Lect 2. - AR Technology

Strengths of optical ARSimpler (cheaper)Di i f l ldDirect view of real world

Full resolution, no time delay (for real world)SafetyLower distortion Lower distortion

No eye displacementbut COASTAR video see-through avoids this

Page 28: COSC 426 Lect 2. - AR Technology

Video AR Architecture

head position/orientation

Head-mounted camera aligned to

display opticsHead

Tracker

head position/orientation display optics

Video

Video image of real world

Host P

Graphicsd

DigitalMi Frame

Processor

Processor renderer Mixer Buffer

to networkDisplayDriver

Non see-thru

Virtual image inset into video of realImage source

& optics

video of real world

Page 29: COSC 426 Lect 2. - AR Technology

Video see-through HMDVideocameras Video

Graphics

Monitors CombinerCombiner

Page 30: COSC 426 Lect 2. - AR Technology

Video See-Through HMD

Page 31: COSC 426 Lect 2. - AR Technology

Video see-through HMD

’ COAS A MR Laboratory’s COASTAR HMD(Co-Optical Axis See-Through Augmented Reality)Parallax free video see through HMDParallax-free video see-through HMD

Page 32: COSC 426 Lect 2. - AR Technology

TriVisiowww.trivisio.comStereo video inputp

PAL resolution cameras

2 x SVGA displays2 x SVGA displays30 degree FOVUser adjustable convergenceUser adjustable convergence

$6,000 USD

Page 33: COSC 426 Lect 2. - AR Technology

Vuzix Display

www.vuzix.comWrap 920$350 USD$350 USDTwin 640 x 480 LCD displays 31 degree diagonal field of view Weighs less than three ounces Weighs less than three ounces

Page 34: COSC 426 Lect 2. - AR Technology

Strengths of Video ARTrue occlusion

Kiyokawa optical display that supports occlusiony p p y pp

Digitized image of real worldFl b l Flexibility in compositionMatchable time delaysMore registration, calibration strategies

Wide FOV is easier to supportWide FOV is easier to support

Page 35: COSC 426 Lect 2. - AR Technology

Optical vs. Video AR SummaryBoth have proponentsVideo is more popular today?

Likely because lack of available optical productsy p p

Depends on application?M f i i l i hManufacturing: optical is cheaperMedical: video for calibration strategies

Page 36: COSC 426 Lect 2. - AR Technology

Eye multiplexed AR Architecture

head position/orientationHead

Tracker

head position/orientation

real world

Host P

Data BaseM d l

RenderingE i FrameProcessor Model Engine Buffer

to networkDisplayDriver

Virtual Image inset intoreal world scene

OpaqueImage source

Page 37: COSC 426 Lect 2. - AR Technology

Virtual Image ‘inset’ into real

Page 38: COSC 426 Lect 2. - AR Technology

Virtual Vision Personal Eyewear

Page 39: COSC 426 Lect 2. - AR Technology

Virtual image inset into real world

Page 40: COSC 426 Lect 2. - AR Technology

Spatial/Projected AR

Page 41: COSC 426 Lect 2. - AR Technology

Spatial Augmented Reality

Project onto irregular surfacesGeometric RegistrationProjector blending, High dynamic range

Book: Bimber, Rasker “Spatial Augmented Reality”p g y

Page 42: COSC 426 Lect 2. - AR Technology

Projector-based ARUser (possiblyhead-tracked)

Projector

Examples:Raskar MIT Media LabReal objects Raskar, MIT Media LabInami, Tachi Lab, U. Tokyowith retroreflective

covering

Page 43: COSC 426 Lect 2. - AR Technology

Example of projector-based AR

Ramesh Raskar, UNC, MERL, ,

Page 44: COSC 426 Lect 2. - AR Technology

Example of projector-based AR

Ramesh Raskar, UNC Chapel Hill

Page 45: COSC 426 Lect 2. - AR Technology

The I/O Bulb

P j + CProjector + CameraJohn Underkoffler, Hiroshi Ishii MIT Media Lab

Page 46: COSC 426 Lect 2. - AR Technology

Head Mounted Projector

Head Mounted ProjectorJannick Rolland (UCF)J ( )

Retro-reflective MaterialPotentially portablePotentially portable

Page 47: COSC 426 Lect 2. - AR Technology

Head Mounted Projector

NVIS P 50 HMPDNVIS P-50 HMPD1280x1024/eyeStere sc icStereoscopic50 degree FOV

iwww.nvis.com

Page 48: COSC 426 Lect 2. - AR Technology

HMD vs. HMPD

Head Mounted Display Head Mounted Projected Display

Page 49: COSC 426 Lect 2. - AR Technology

Pico Projectors

Microvision - www.mvis.com3M S Phili t3M, Samsung, Philips, etc

Page 50: COSC 426 Lect 2. - AR Technology

MIT Sixth Sense

Body worn camera and projectorhttp://www.pranavmistry.com/projects/sixthsense/p p y p j

Page 51: COSC 426 Lect 2. - AR Technology

Other AR Displays

Page 52: COSC 426 Lect 2. - AR Technology

Video Monitor AR

Videocameras Monitor

Stereoglassescameras g

Video

Graphics Combiner

Page 53: COSC 426 Lect 2. - AR Technology

Virtual Showcase

Mirrors on a projection tableH d k d Head tracked stereoUp to 4 usersM hi d l bjMerges graphic and real objectsExhibit/museum applications

Fraunhofer Institute (2001)Bimber, Frohlich

Page 54: COSC 426 Lect 2. - AR Technology

Augmented Paleontology

Bimber et. al. IEEE Computer Sept. 2002

Page 55: COSC 426 Lect 2. - AR Technology

Alternate Displays

LCD Panel Laptop PDA

Page 56: COSC 426 Lect 2. - AR Technology

Handheld Displays

Mobile PhonesCameraDisplay InputInput

Page 57: COSC 426 Lect 2. - AR Technology

Other Types of AR DisplayAudio

spatial soundpambient audio

T til Tactile physical sensation

Hapticvirtual touchvirtual touch

Page 58: COSC 426 Lect 2. - AR Technology

Haptic Input

AR Haptic WorkbenchCSIRO 2003 Adcock et al CSIRO 2003 – Adcock et. al.

Page 59: COSC 426 Lect 2. - AR Technology

Phantom

Sensable Technologies (www.sensable.com)6 DOF Force Feedback Device

Page 60: COSC 426 Lect 2. - AR Technology

AR Haptic Interface

Phantom, ARToolKit, Magellang

Page 61: COSC 426 Lect 2. - AR Technology

AR Tracking and Registration

Page 62: COSC 426 Lect 2. - AR Technology

TrackingLocating the users viewpointg pPosition (x,y,z)Orientation (r p y)Orientation (r,p,y)

RegistrationPositioning virtual object wrt real world

Page 63: COSC 426 Lect 2. - AR Technology

Tracking Requirements

Head Stabilized Body Stabilized World Stabilized

Augmented Reality Information Display World StabilizedBody StabilizedHead Stabilized

Increasing Tracking Requirements

Page 64: COSC 426 Lect 2. - AR Technology

Tracking Technologies

• Mechanical• Electromagneticg• Optical• Acoustic

I ti l d d d k i• Inertial and dead reckoning• GPS• Hybrid• Hybrid

Page 65: COSC 426 Lect 2. - AR Technology

AR Tracking Taxonomy

ARTRACKING

IndoorEnvironment

OutdoorEnvironment

TRACKING

Limited Range Extended Range Low Accuracy &Not Robust

High Accuracy& Robust

Low Accuracyat 15-60 Hz

High Accuracy& High Speed

H b id

Many Fiducialsin space/time

b t

Not HybridizedGPS or

C

Hybrid TrackingGPS and

C d

e g AR Toolkit e g IVRD

HybridTracking

e g HiBall

butno GPS

e g WLVA

Camera orCompass

e g BARS

Camera andCompass

e.g. AR Toolkit e.g. IVRD e.g. HiBall e.g. WLVA e.g. BARS

Page 66: COSC 426 Lect 2. - AR Technology

Tracking Types

Magnetic T k

Inertial T k

Ultrasonic T k

Optical T k

Mechanical TrackerTracker Tracker Tracker Tracker

Marker Based Markerless Specialized

Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Trackingg g g

Page 67: COSC 426 Lect 2. - AR Technology

Tracking Systems Mechanical Tracker

Magnetic TrackerMagnetic TrackerUltrasonic TrackerInertial TrackerVision (Optical Tracking)Vision (Optical Tracking)

Specialized (Infrared, Retro-Reflective)M l (DVC W b )Monocular (DVCam, Webcam)

Page 68: COSC 426 Lect 2. - AR Technology

Mechanical TrackerId h l h Idea: mechanical arms with joint sensors

++: high accuracy haptic feedback Microscribe

++: high accuracy, haptic feedback -- : cumbersome, expensive

Page 69: COSC 426 Lect 2. - AR Technology

Magnetic TrackerIdea: difference between a magnetic transmitter Idea: difference between a magnetic transmitter and a receiver

6DOF b

Flock of Birds (Ascension)

++: 6DOF, robust -- : wired, sensible to metal, noisy, expensivey p

Page 70: COSC 426 Lect 2. - AR Technology

I l T kInertial TrackerIdea: measuring linear and angular orientation rates (accelerometer/gyroscope)

++: no transmitter cheap small high frequency wireless

IS300 (Intersense) Wii Remote

++: no transmitter, cheap, small, high frequency, wireless -- : drift, hysteris only 3DOF

Page 71: COSC 426 Lect 2. - AR Technology

Ultrasonics TrackerIdea: Time of Flight or Phase Coherence Sound WavesIdea: Time of Flight or Phase-Coherence Sound Waves

Ultrasonic Logitech

++: Small, Cheap

Logitech IS600

-- : 3DOF, Line of Sight, Low resolution, Affected Environment Conditon (pressure, temperature)

Page 72: COSC 426 Lect 2. - AR Technology

Global Positioning System (GPS)

Created by US in 1978Currently 29 satellites

Satellites send position + time GPS Receiver positioningp g

4 satellites need to be visibleDifferential time of arrivalTriangulation

Accuracyy5-30m+, blocked by weather, buildings etc

Page 73: COSC 426 Lect 2. - AR Technology

Problems with GPSTakes time to get satellite fixTakes time to get satellite fix

Satellites moving around

Earths atmosphere affects signalp gAssumes consistent speed (the speed of light). Delay depends where you are on EarthWeather effects

Signal reflectionMulti-path reflection off buildings

Signal blockingT b ildi t iTrees, buildings, mountains

Satellites send out bad dataMisreport their own positionMisreport their own position

Page 74: COSC 426 Lect 2. - AR Technology

Accurate to < 5cm close to base station (22m/100 km)Accurate to 5cm close to base station (22m/100 km)Expensive - $20-40,000 USD

Page 75: COSC 426 Lect 2. - AR Technology

Optical Tracking

Page 76: COSC 426 Lect 2. - AR Technology

Optical TrackerIdea: Image Processing and Computer Visiong g pSpecialized

f f SInfrared, Retro-Reflective, Stereoscopic

ART Hi-Ball

Monocular Based Vision Trackingg

Page 77: COSC 426 Lect 2. - AR Technology

Outside-In vs. Inside-Out Tracking

Page 78: COSC 426 Lect 2. - AR Technology

Optical Tracking Technologies

Scalable active trackersScalable active trackersInterSense IS-900, 3rd Tech HiBall

3 d T h IPassive optical computer visionLine of sight may require landmarks

3rd Tech, Inc.

Line of sight, may require landmarks Can be brittle.C i i i i ll i iComputer vision is computationally-intensive

Page 79: COSC 426 Lect 2. - AR Technology

HiBall Tracking System (3rd Tech)O Inside-Out Tracker

$50K USD

Scalable over large areaF t d t (2000H ) Fast update (2000Hz) Latency Less than 1 ms.

AccuratePosition 0 4mm RMSPosition 0.4mm RMSOrientation 0.02° RMS

Page 80: COSC 426 Lect 2. - AR Technology
Page 81: COSC 426 Lect 2. - AR Technology

Starting simple: Marker trackingHas been done for more than 10 yearsS l l i iSeveral open source solutions existFairly simple to implementy p p

Standard computer vision methods

A l k id 4 iA rectangular marker provides 4 corner pointsEnough for pose estimation!

Page 82: COSC 426 Lect 2. - AR Technology

Marker Based Tracking: ARToolKit

http://artoolkit.sourceforge.net/

Page 83: COSC 426 Lect 2. - AR Technology

C d SCoordinate Systems

Page 84: COSC 426 Lect 2. - AR Technology

M k T k OMarker Tracking – Overview

Page 85: COSC 426 Lect 2. - AR Technology

Marker Tracking – Fiducial DetectionThreshold the whole image to black and whiteThreshold the whole image to black and whiteSearch scanline by scanline for edges (white to black)Follow edge until either

Back to starting pixelImage border

Check for sizeReject fiducials early that are too small (or too large)

Page 86: COSC 426 Lect 2. - AR Technology

Marker Tracking – Rectangle FittingS i h bi i “ ” h Start with an arbitrary point “x” on the contourThe point with maximum distance must be a corner c0C d l h h h Create a diagonal through the centerFind points c1 & c2 with maximum distance left and right of diag.New diagonal from c1 to c2Find point c3 right of diagonal with maximum distance

Page 87: COSC 426 Lect 2. - AR Technology

Marker Tracking – Pattern checkingCalculate homography using the 4 corner pointsCalculate homography using the 4 corner points

“Direct Linear Transform” algorithmMaps normalized coordinates to marker coordinatesp(simple perspective projection, no camera model)

Extract pattern by samplingCheck pattern

Id (implicit encoding)T l ( li d l i )Template (normalized cross correlation)

Page 88: COSC 426 Lect 2. - AR Technology

Marker Tracking – Corner refinementRefine corner coordinatesRefine corner coordinates

Critical for high quality trackingRemember: 4 points is the bare minimum!So these 4 points should better be accurate…

Detect sub-pixel coordinatesE.g. Harris corner detectorg- Specialized methods can be faster and more accurateStrongly reduces jitter!g y j

Undistort corner coordinatesR di l di t ti f lRemove radial distortion from lens

Page 89: COSC 426 Lect 2. - AR Technology

M k k P Marker tracking – Pose estimationC Calculates marker position and rotation relative to the cameraInitial estimation directly from homography

Very fast but coarseVery fast, but coarseJitters a lot…

R fi i G N i iRefinement via Gauss-Newton iteration6 parameters (3 for position, 3 for rotation) to refineAt each iteration we optimize on the reprojection error

Page 90: COSC 426 Lect 2. - AR Technology

Coordinates for Marker Tracking

Page 91: COSC 426 Lect 2. - AR Technology

Coordinates for Marker Tracking

Marker Camera•GoalR t ti & T l ti

•Camera Ideal Screen•Perspective model•Obtained from Camera Calibration

•Ideal Screen Observed Screen•Nonlinear function (barrel shape)•Marker Observed Screen•Correspondence of 4 vertices

•Rotation & Translation•Obtained from Camera Calibration•Obtained from Camera Calibration•Real time image processing

Page 92: COSC 426 Lect 2. - AR Technology

F M k T CFrom Marker To CameraRotation & Translation

TCM : 4x4 transformation matrixfrom marker coord. to camera coord.

Page 93: COSC 426 Lect 2. - AR Technology

Tracking challenges in ARToolKit

Unfocused camera, motion blur

Dark/unevenly lit scene, vignetting

Jittering (Photoshop illustration)

Occlusion(image by M. Fiala)

False positives and inter-marker confusion(image by M. Fiala)

Image noise(e.g. poor lens, block coding /

compression, neon tube)

Page 94: COSC 426 Lect 2. - AR Technology

Tracking, Tracking, Tracking

Page 95: COSC 426 Lect 2. - AR Technology

Other Marker Tracking LibrariesTarTaghttp://www.artag.net/

ARToolKitPlus [Discontinued]http://studierstube.icg.tu-graz ac at/handheld ar/artoolkitplus phpgraz.ac.at/handheld_ar/artoolkitplus.php

stbTrackerhtt // t di t b i thttp://studierstube.icg.tu-graz.ac.at/handheld_ar/stbtracker.php

MXRToolKitMXRToolKithttp://sourceforge.net/projects/mxrtoolkit/

Page 96: COSC 426 Lect 2. - AR Technology

Markerless Tracking

Page 97: COSC 426 Lect 2. - AR Technology
Page 98: COSC 426 Lect 2. - AR Technology

Markerless Tracking

M i T k I i l Ul i O i l

No more Markers! Markerless Tracking

Magnetic Tracker Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Template-Based Interest Point Tracking Tracking Tracking

Page 99: COSC 426 Lect 2. - AR Technology

Natural feature trackingTracking from features of the surrounding environmentenvironment

Corners, edges, blobs, ...

G ll diffi l h k kiGenerally more difficult than marker trackingMarkers are designed for their purposeg p pThe natural environment is not…

Less well established methodsLess well-established methodsUsually much slower than marker tracking

Page 100: COSC 426 Lect 2. - AR Technology

Natural Feature TrackingFeatures Points

Use Natural Cues of Real ElementsEdges

Contours

Features Points

Surface Texture Interest Points

Model or Model-Free++: no visual pollution

SurfacesSurfaces

Page 101: COSC 426 Lect 2. - AR Technology

Texture Tracking

Page 102: COSC 426 Lect 2. - AR Technology

T k b dTracking by detectionThis is what most trackers do Camera ImageThis is what most trackers do…Targets are detected every frame

Keypoint detection

g

Popular becausetracking and detection

yp

Descriptor creationd t hiare solved simultaneously and matching

Outlier RemovalOutlier Removal

Pose estimationand refinementand refinement

Pose

Page 103: COSC 426 Lect 2. - AR Technology

Natural feature tracking – What is a keypoint?

It depends on the detector you use!For high performance use the FAST corner For high performance use the FAST corner detector

A l FAST t ll i l f iApply FAST to all pixels of your imageObtain a set of keypoints for your image

R d h f - Reduce the amount of corners using non-maximum suppression

Describe the keypoints

E. Rosten and T. Drummond (May 2006). "Machine learning for high‐speed corner detection". 

Page 104: COSC 426 Lect 2. - AR Technology

Corner keypoint

Page 105: COSC 426 Lect 2. - AR Technology

Natural feature tracking – DescriptorsAgain depends on your choice of a descriptor!Again depends on your choice of a descriptor!Can use SIFT

E i h d i k iEstimate the dominant keypointorientation using gradientsC m ensate f r detectedCompensate for detectedorientationDescribe the keypoints in termsDescribe the keypoints in termsof the gradients surrounding it

Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones.IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 

Page 106: COSC 426 Lect 2. - AR Technology

NFT D b NFT – Database creationOffline steppSearching for corners in a static imageFor robustness look at corners on multiple scalesFor robustness look at corners on multiple scales

Some corners are more descriptive at larger or smaller scalesW d ’t k h f ill b f iWe don’t know how far users will be from our image

Build a database file with all descriptors and their i i h i i l iposition on the original image

Page 107: COSC 426 Lect 2. - AR Technology

NFT R l kNFT – Real-time trackingSearch for keypoints Camera ImageSearch for keypointsin the video imageC t th d i t Keypoint detection

Camera Image

Create the descriptorsMatch the descriptors from the

Keypoint detection

Descriptor creation

live video against thosein the database

pand matching

O tli R lBrute force is not an optionNeed the speed-up of special

Outlier Removal

Pose estimationand refinementdata structures

- E.g., we use multiple spill trees

and refinement

PosePose

Page 108: COSC 426 Lect 2. - AR Technology

NFT O l lNFT – Outlier removalCascade of removal techniquesCascade of removal techniquesStart with cheapest, finish with most expensive…

First simple geometric testsFirst simple geometric tests- E.g., line tests

• Select 2 points to form a line• Check all other points being on correct side of line

Then, homography-based tests

Page 109: COSC 426 Lect 2. - AR Technology

NFT P fNFT – Pose refinementPose from homography makes good starting pointBased on Gauss-Newton iteration

Try to minimize the re-projection error of the keypoints

Part of tracking pipeline that mostly benefits Part of tracking pipeline that mostly benefits from floating point usageCan still be implemented effectively in fixed pointCan still be implemented effectively in fixed pointTypically 2-4 iterations are enough…

Page 110: COSC 426 Lect 2. - AR Technology

NFT R l kNFT – Real-time trackingSearch for keypoints

Camera Image

in the video imageCreate the descriptors Keypoint detection

Camera Image

pMatch the descriptors from thelive video against those

Keypoint detection

Descriptor creationlive video against thosein the databaseRemove the keypoints that

and matching

Outlier RemovalRemove the keypoints thatare outliersU h i i k i

Outlier Removal

Pose estimationand refinementUse the remaining keypoints

to calculate the pose f h

and refinement

Poseof the camera

Page 111: COSC 426 Lect 2. - AR Technology

NFT R lNFT – Results

Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones.IEEE Transactions on Visualization and Computer Graphics, May/June, 2010 

Page 112: COSC 426 Lect 2. - AR Technology

Edge Based TrackingRAPiD [Drummond et al. 02]

Initialization, Control Points, Pose Prediction (Global Method)

Page 113: COSC 426 Lect 2. - AR Technology

Line Based TrackingVisual Servoing [Comport et al. 2004]

Page 114: COSC 426 Lect 2. - AR Technology

Model Based TrackingOpenTL - www.opentl.org

General purpose library for model based visual tracking

Page 115: COSC 426 Lect 2. - AR Technology

OpenTL Features

Page 116: COSC 426 Lect 2. - AR Technology
Page 117: COSC 426 Lect 2. - AR Technology

Visual Modalities Used For Tracking

Page 118: COSC 426 Lect 2. - AR Technology

The Tracking Pipeline

Page 119: COSC 426 Lect 2. - AR Technology

Marker vs. natural feature trackingMarker tracking

Usually requires no database to be storedMarkers can be an eye-catcherTracking is less demandingg gThe environment must be instrumented with markersMarkers usually work only when fully in viewy y y

Natural feature trackingA database of keypoints must be stored/downloadedA database of keypoints must be stored/downloadedNatural feature targets might catch the attention lessN t l f t t t t ti ll hNatural feature targets are potentially everywhereNatural feature targets work also if partially in view

Page 120: COSC 426 Lect 2. - AR Technology

Hybrid Tracking

Page 121: COSC 426 Lect 2. - AR Technology

Example: Outdoor Hybrid Tracking

Combinescomputer visioncomputer vision

- natural feature trackinginertial gyroscope sensorsinertial gyroscope sensors

Both correct for each otherInertial gyro - provides frame to frameInertial gyro provides frame to frame prediction of camera orientationComputer vision - correct for gyro driftComputer vision correct for gyro drift

Page 122: COSC 426 Lect 2. - AR Technology

Outdoor AR Tracking System

You, Neumann, Azuma outdoor AR system (1999)

Page 123: COSC 426 Lect 2. - AR Technology

Robust Outdoor Tracking

H b id T kiHybrid TrackingComputer Vision, GPS, inertial

Going OutReitmayer & Drummond (Univ. Cambridge)Reitmayer & Drummond (Univ. Cambridge)

Page 124: COSC 426 Lect 2. - AR Technology

Handheld Display

Page 125: COSC 426 Lect 2. - AR Technology

Registration

Page 126: COSC 426 Lect 2. - AR Technology

The Registration ProblemVirtual and Real must stay properly alignedIf tIf not:

Breaks the illusion that the two coexistPrevents acceptance of many serious applications

Page 127: COSC 426 Lect 2. - AR Technology

Sources of registration errorsS Static errors

Optical distortionsMechanical misalignmentsTracker errorsIncorrect viewing parameters

Dynamic errorsDynamic errorsSystem delays (largest source of error)

1 d l 1/3 i t ti - 1 ms delay = 1/3 mm registration error

Page 128: COSC 426 Lect 2. - AR Technology

Reducing static errorsDistortion compensationManual adjustmentsManual adjustmentsView-based or direct measurements

[Azuma94] [Caudell92] [Janin93] etc.

Camera calibration (video)Camera calibration (video)[ARGOS94] [Bajura93] [Tuceryan95] etc.

Page 129: COSC 426 Lect 2. - AR Technology

View Based Calibration (Azuma 94)

Page 130: COSC 426 Lect 2. - AR Technology

Dynamic errorsApplication Loop

Tracking Calculate ViewpointSimulation

Render Scene

Draw to Display

x,y,zr,p,y

Simulation

20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms

Total Delay = 50 + 2 + 33 + 17 = 102 ms1 ms delay = 1/3 mm = 33mm error1 ms delay 1/3 mm 33mm error

Page 131: COSC 426 Lect 2. - AR Technology

Reducing dynamic errors (1)

Reduce system lag[Olano95] [Wloka95a] [Regan SIGGRAPH99]

Reduce apparent lagReduce apparent lagImage deflection [Burbidge89] [Regan94] [So92] [Kiji ISMR 2001][Kijima ISMR 2001]Image warping [Mark 3DI 97]

Page 132: COSC 426 Lect 2. - AR Technology

Reducing System Lag

Application Loop

Tracking Calculate Viewpoint

Render Scene

Draw to Display

x,y,zr,p,y

ViewpointSimulation

Scene Display

Faster Tracker Faster CPU Faster GPU Faster DisplayFaster Tracker Faster CPU Faster GPU Faster Display

Page 133: COSC 426 Lect 2. - AR Technology

Reducing Apparent Lag

x,y,zVirtual Display

Ph i l

Virtual Display

Physical

Tracking

yr,p,yPhysical

Display(640x480)

Physical Display

(640x480)TrackingUpdate1280 x 960

Last known position

1280 x 960

Latest positionLast known position Latest position

Application Loop

Tracking Calculate Viewpoint

Render Scene

Draw to Display

x,y,zr,p,y

pSimulation

p y

Page 134: COSC 426 Lect 2. - AR Technology

Reducing dynamic errors (2)Match input streams (video)

Delay video of real world to match system lagDelay video of real world to match system lag

Predictive Tracking[Azuma94] [Emura94] Inertial sensors helpful

Azuma / Bishop 1994u a / s op 99

Page 135: COSC 426 Lect 2. - AR Technology

Predictive Tracking

Pos Nowition

TimePast Future

Time

Can predict up to 80 ms in future (Holloway)

Page 136: COSC 426 Lect 2. - AR Technology

Predictive Tracking (Azuma 94)

Page 137: COSC 426 Lect 2. - AR Technology

Wrap-upTracking and Registration are key problemsRegistration errorRegistration error

Measures against static errorM i d i Measures against dynamic error

AR typically requires multiple tracking technologiesyp y q p g gResearch Areas: Hybrid Markerless Techniques, Deformable Surface Mobile Outdoors Deformable Surface, Mobile, Outdoors

Page 138: COSC 426 Lect 2. - AR Technology

More Information

M k Billi h t• Mark Billinghurst– [email protected]

• Websiteshi l b– www.hitlabnz.org