fondamentals of virtual and augmented reality click to edit … · 2017-10-04 · 3 “virtual and...

Click to edit Master title style

“Virtual and Augmented Reality”

Interaction speciality – Computer Science Master - University Paris-Saclay

Fondamentals of Virtual and Augmented Reality

AR: an introduction

Jean-Marc Vezien

[email protected]

2

“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay

Plan of the lecture

1. Augmented reality:

what and why

2. User tracking

3. Real world reconstruction

4. 3D Graphics

5. Augmentation

3


Wikipedia says…

Augmented reality (AR) is a term for a live

direct or indirect view of a physical, real-world

environment whose elements are augmented by

computer-generated sound, video, graphics,

haptic or GPS data.

Augmentation is conventionally in real-time and

in semantic context with environmental

elements, such as sports scores on TV during a

match.

With the help of advanced AR technology

(computer vision and object recognition) the

information about the surrounding world

becomes interactive, e.g. artificial information

about the environment can be overlaid.

Augmented reality was coined by Thomas

Caudell, working at Boeing, in 1990. ARToolkit (Kato & Billinghurst, 2001)

4


Real/virtual Continuum (Milgram 1994)

5



6



7



8



9



10



11


AR: examples

Not a new idea….

• User position and gaze provides context ,

sometimes with help (markers)

• Audio is nice (non-obstrusive), can be

stopped anytime.

• No computer involved

• No sensing involved : limited interaction

Now associated with head-tracking…

Started in 1957 (Roosevelt home)

Principio system

(2007)

12


AR: examples

See-through augmented vision: « classic AR »

Tourism: Archeoguide (2002)

13


AR: examples

Assemply / Maintenance / Repair

Maintenance: help technician with

contextualized content

BMW, 2010

Matris project, 2007

Fiducial Text

ActionGraphics Icons

14


AR: examples

Way-findingIphone app: NY nearest subway (and many cities, airports have their own app)

https://www.youtube.com/watch?v=ps49T0iJwVg

Augmented Car Finder

Note:

• Data is collected off-line

• Access to database is native on mobile phones

(http protocol)

15


AR: examples

Safe landing

Hunter system: Landing guiding system by AR

tracking techniques (computer vision based)

www.novemberkiloecho.com

16


War "games" (1/2)

Track target w.r.t. weapon

Coupled with (off-line) Geographical Information

system

… and strategic realtime info (GPS, detectors)

gun-mounted display

17


War "games" (2/2)

Tanagram.com concept (closed)

18


Heads Up Displays

TopOwl helmet (helicopter)

In operation since 2008

Thalesgroup.com

Eye tracking

Symbology

Gun pointing

Night VisionHeavy and $$$ !

19


Heads Up Displays (2)

Skully web helmet (2015)

(+ rear cameras)

Intuitive Aerial (Sweden, 2013)

20


MR: examples

Games and promotional contents

EyePet for PSP (2009)

Topps 3D baseball cards (2009)

PSVita (2012)

Father.io (2016): Massive Multiplayer Laser Tag

21


Pokémon GO !

The Pokemon Company (Nintendo) + Niantic (Ingress)

Since July 2016

game brought in more than $1.2 Billion !

6% of android US users still use it.

Success = the right mix of game + social

Possible only with AR (or connected VR ?)

22


Other examples

Medical: Surgery planning, nurse and student training…

University of North Carolina

Augmented virtuality (presented on screen)

Track hand-held device or body

Coupled with (off-line) anatomic data

23


DaVinci Robot

Intuitive Surgical

3 000 ex.

1 M.$ per unit30 problems over 17000

interventions, 1 death (bad

manip.)

Less stress

More precision

Comfortable

Complete asepsis

No tremor

abdomen

surgery

24


• Militaires: « future soldier », BARS

• Médical : assistance chirurgicale

• Tourisme

• Customization (mode)

Virtual try-ons and Customize

Augmented virtuality or

AR

Track users anatomy

and motion (better still)

Coupled with external

data on-line

Augment.comIkea (2014)

ForeverMark (2011)

25


MR: the SACARI example

Mixed virtuality: user not moving with camera,

indirect view of real world

Tele-immersion : provide sense of

presence of remote environment

Internship available !

26


Blurred boundaries: augmented movies

Is this real ? Augmented ? Virtual ?

On-line ? Off-line ?

27


Blurred boundaries

Is this still AR ?

AR will be everywhere the

moment efficient, cheap

see-through displays will

become available

Google moto: The World's

Information in Context

Street View is close to

Augmented Virtuality

Serguei Brin (2012)

28


The Ingredients for AR

Capture real world

29



Capture real world

+

Capture virtual world

(Computer Graphics)

30



Capture real world

+

Capture virtual world

(Computer Graphics)

Present to user

(Augmentation)

31


Real 3D world

Two ingredients necessary for

successful AR:

Real-time 3D tracking of

user viewpoint w.r.t. world

3D scene analysis

realistic augmentation

WorldViz

Shadow Zone

32


Real 3D world


successful AR:

Real-time 3D tracking of

user viewpoint w.r.t. world

3D scene analysis


WorldViz

33


Real-Time tracking

Many means to compute positioning information:

Geolocalization

Electro-magnetic

Acoustic

Inertial

Vision-based (active or passive)

Covered in « stereo and tracking » course

34


Image-based 3D tracking

It is a special case of adaptive

rendering ( for HMDs)

Environment control ++

Markers are still needed

(often)

Special case of

structure/motion estimation

AR meets movie industry:

the director is the user !

markersKnown as motion estimation / match moving

35


From [3]

= user

From [3]

Inside-out vs. Outside-in

= user

Inside-out: user wears camera Outside-in: camera observes user

Inside-out better at estimating relative rotation

Outside-in much more convenient : don’t have to wear a camera

Special case: console gaming

Inside-out easier for wearable AR: mobility, ego-centered

reference frame

Needed for future nomadic apps

Cameras are getting small and commodity items

36


M

Z

Y

X

P

Tt

TtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1

Projection equation:

image 3D world

(i,j,k,Tx,Ty,Tz) can be computed if m and M are known.

Not linear in the motion parameters !!

m = P . M

motion structure

The Maths

37


M

Z

Y

X

P

Tt

TtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1


image 3D world



m = P . M

motion structure

The Maths

World camera =

Camera position

38


M

Z

Y

X

P

Tt

TtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1


image 3D world



m = P . M

motion structure

The Maths

3D 2D projection

39


M

Z

Y

X

P

Tt

TtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1

image 3D world



m = P . M

motion structure

The Maths

Sensor image (pixels)


40


Dementhon-Davis (1992)

Several methods to recover calibration exist (Tsai, Lowe…)

" Model-Based object pose in 25 lines of code" is a simple

yet elegant method to obtain pose rapidly if a rigid 3D model

is provided.

i

i

i

i

i

i

i

ii

xPI

PTz

kTz

TxP

Tz

i

TzPk

TxPi

Z

Xx

1

.

1.

.

.

. 0

i

ii

yPJy

1

. 0

),( 00 yx Tz

Pk ii

.

Projection of T(Tx,Ty,Tz) Relative depth around Z0=Tz

Idea consists in linearizing xi and yi to compute iJI ,,

ziz TTZ

41


i

o

i

i

o

i

i PJyy

PIxx

.

.

0

011

1

Geometrical interpretation:

P

O

Image plane

P0

Reference plane

Z0=Tz

Orthographic

projection

Z

Y

i Z0

Perspective

projection

Linear approximation of perspective

iio

y

iii

o

i

o

x

y

xxx

42


POSIT algorithm

Summarize:

Tz

jPJyyyy

Tz

iPIxxxx

i

o

iii

i

o

iii

=Jwith.)1(

=Iwith.)1(

00

00

If εi is known then I,J,Tz,X0=x0Tz,Y0=y0Tz can be computed

If I,J,Tz k is known

Iterate starting with

Converges rapidly:

Tz

Pk ii

.

0i

43


3D tracking: off-the-shelf solution

• Kato & Billinghurst (HIT lab, univ. of Washington) introduce

in 1999 a tool for teleconferencing:

"Marker Tracking and HMD Calibration for a video-

based Augmented Reality Conferencing System.“

Soon to become ARToolkit

44


ARToolkit: main characteristics

• Fast and cheap 6D marker tracking.

• Distributed with complete source code.

OpenSource with GPL license for

noncommercial usage.

• Multiplatform (Linux, MacOS and Windows) .

• Multiple input sources (USB, Firewire) ,

multiple format (RGB, YUV) supported.

• Multiple camera tracking supported.

• GUI initializing interface.

• Easy calibration routine.

• Fast rendering based on OpenGL.

• 3D VRML support.

• Simple and modular API (in C and C++).

• Complete set of samples and utilities.

• Supports both video and optical see-through

AR.

45


ARToolkit: main processing loop

Since v4, Pose estimation is performed using the

Iterative Closest Point (ICP) algorithm (Besl, 1992)

46


Important details

Virtual objects appear only when complete markers are visible:

Size

Movement

Orientation

Lighting conditions

Pattern Size (cm.) Usable Range (cm.)

6.98 40.64

8.89 63.5

10.79 86.36

In practice: close range only !

47


3D tracking: off-the-shelf solution

www.layar.com

web-based, per page service

www.wikitude.com: AR SDK

Vuforia (qualcomm) multiplatform SDK

Including Unity

Inglobe AR: AR solutions, ARPlayer+plugins (40.000 registered users)

2490€

$499

focus on printed text

48


Real 3D world


successful AR:

Real-time 3d tracking of user

viewpoint w.r.t. world

3D scene analysis


49


2 components: Light and Geometry

Lighting coherency Motion coherency

Geometric coherency

Light probe

3D real world analysis

50


Aim: Capture light coming from light sources in real world

10,000:1 is the (static) human eye ratio between brightest and

darkest shade: cannot be represented on 8 bits

Stored in High Dynamic Range images (.hdr image format)

From all directions at a given point

Light Probe

u

v

51


Half-life 2: normal and HDR

Note: advanced graphics techniques can

use hdr images (Unreal engine)

u

v

1,1),( vu

u

v1tan 22 vu

A direction vector in the world (Dx, Dy, Dz)

u = r. Dx and v = r.Dy with

unit vector pointing in the direction (u,v) is obtained by rotating

(0,0,-1) by:

1) degrees around the y (up) axis

2) degrees around the -z (forward) axis.

22

1 )(cos1

DyDx

Dzr

Light Probe (2)

52


World content

Geometry: 3D reconstruction : what is

where ?

Semantic : augmentation of context

object recognition

interpretation of content

Comprendre l’environnement 3D en termes : http://www.truevisionsys.com

53


Markers cannot always be present !

If structure is known: a priori model tracking (see before):

an object is a (collection of) marker

Remember : projection combines structure and motion (m

= P.M)

Idea: recover both simultaneouly !

+ gives tracking and reconstruction at the same time

- highly non-linear : requires iterations + data filtering (remove

outliers )

Geometry: 3D reconstruction

Factorization method introduced by Tomasi and Kanade (1992)

54


mij = Pj. Mi

Factorization method

2x1 = 2x3 3x1

FPF

FPF

P

P

yy

xx

yy

xx

m

1

1

111

111

jth frame

ith point

P

P

P

ZZ

YY

XX

M

1

1

1

FP

P

P 1

• Compact representation:

• Consider P points projecting on F frames:

• There is an of possible (P,M) pairs!

• Solved via SVD decomposition (but not uniquely)

Carlo Tomasi and Takeo Kanade. (November 1992)

"Shape and motion from image streams under orthography: a factorization method."

International Journal of Computer Vision, 9 (2): 137–154.

Images !

m is big, but rank 3 ! m= P.M

55


S. Christy et R. Horaud : "Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations"

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 18, Number 11, Pages 1098--1104 - November 1996

Factorization: results

Hanno Ackermann, University of Hanovre (2008)

56


Computer Vision is good at locating features:

points

regions

textures

contours

… at different scales !

Image-based 3D reconstruction

Harris (1988)

SURF detector (OpenCV), Bay (2006)

2D region detection (region-growing)

57


Problems :

Robustness of detection

Matching (space/time)

Will become prevalent eventually.

For now: limited use , constraints

(model-based)

Moving fast with new AR devices

Markerless tracking

Wang & Popovic, MIT, 2010

58


Boujou: http://vicon.com/boujou/

Commercial solutions

Pricey (10 000$)

59


New real-time sensors: depth cameras (Zcam)

Infer depth for ALL pixels

Early : laser, stereoPerceptron

LIDAR , 1995

Devernay, 1994


60


Now: structured lighting : Zcam for 100 € !

Kinect, 2010

Augmented Reality Magic

Mirror using the Kinect

Tobias Blum


Solves the "where" but not the "what":

objects must still be identified (segmented)

61



Hololens (2016, 3000$)

Augmented reality « see-through »

glasses

Markerless 3D scanner with :

depth camera (Kinect-like) Spatial

Mapping

4 environment cameras

Inertial Measurement UnitHead tracking

SLAM = Simultaneous Localization and Mapping

= markerless tracking !

62


Stereo for AR: augmentation

Object location (table…)

Object occlusion

Shadows and light

interactions

In real-time !!

Will soon happen in

movie industry ($$$)

3D reconstruction is necessary for realistic CG blending:

Source: X3d consortium

63


3D analysis of

real images

(camera +

reconstruction)

Augmentation: CG blending

64


CG graphics+3D analysis of

real images

(camera +

reconstruction)


65


CG graphics

Compositing masks

(shadows, occlusions…)

+3D analysis of

real images

(camera +

reconstruction)


match move

66


CG graphics

Compositing masks

(shadows, occlusions…)

Augmented image

+3D analysis of

real images

(camera +

reconstruction)


match move

67


Matching constraints:

• geometry (epipolar)

• photometry

H. Jin, P. Favaro, and S. Soatto.

A Semi-direct Approach to Structure From Motion. The Visual Computer, 19(6): 377-394, October 2003.

Example of augmentation pipeline (1)

image analysis

Region matching (1992)

68



3D reconstruction

3D reconstruction of regions based on planar equations:

Hypothesis: world is piece-wise planar

69


Prior information:

explicit 3D model

motion constraints

3D registration by ICP = Iterative

Closest Point (Besl 92):

always converges

model points do not coincide with

reconstruction points


3D reconstruction

70



3D virtual content

Two steps:

Geometric modeling

(sometimes based on 3D

scans): Maya, Unity,

Blender, Sketchup, etc.

Photo-realistic rendering:

textures, lights, reflectance,

shadows, etc.

Note: Light probes

environment mapping

71


Light interaction: a must

Geometric coherency is good

but… light is essential !

Shadows in real-time =

geometry + light

Standard technique for CG renderers:

Ground (real)

(virtual)PSP

.'

dnEc

.nLd

.

E

S

72


Shadow projection techniques

Many different techniques !

Plane Projected

Shadows

Projected

Shadows

Depth Shadow

MappingVertex Projection

Shadow

Volumes

Quick, not much

calculations

Quick, almost no

calculations

Very quick, not

much calculations

Slow with high-res

meshes

Slow, lots of

calculations

High detail Detail depends

on texture

Detail depends

on textureHigh Detail High detail

No self-

shadowingNo self-

shadowingSelf-shadowing

No self-shadowing Self-shadowing

No shadow

receivers

Shadow

receivers

Shadow

receivers

No shadow

receivers

Shadow

receivers

73


Shadow mapping

M. Haller, S. Drab, and W. Hartmann, 2003.

"A real-time shadow approach for an augmented reality application using shadow

volumes," in VRST 03: Proceedings of the ACM symposium on Virtual reality software and

technology, New York, NY, USA, 2003, pp. 56-65

Shadow volume with stencil drawing

74



animations

Two steps:

Virtual motions must be

coherent with real ones :

match-moving

Motion is camera-centric

Virtual camera must be

identical to real one (off-line

calibration is necessary)

Accommodate zoom !

75


Occlusion mask

Virtual object rendering

(alone)


Step by step rendering

76


Real objects: reference white

(albedo)

"black" virtual objects:

Shadow computation



77


Final composition =

real image *(1-mask) * attenuation

+

Virtual objects * mask

+ reflexions * (1-mask)



78


Final compositing

79


Other example (1)

Objets déformables + occultations réel/virtuel

80


Other example (2)

Reference white

Shadows of

virtual objects on

the real scene

81


Other example (3)

Final result

82


Conclusion

• Augmented reality is a reality

• Is a complicated process, requires

a lot of expertise and hard/software

• In-depth coherency of geometry

and photometry

• Real-time challenges

Elaborate rendering (GPU)

Match moving (markerless)

3D reconstruction (occlusions)

User interaction: hand and

body tracking

83


THE END !

Denno Coil, 2007

fondamentals of virtual and augmented reality click to edit … · 2017-10-04 · 3 “virtual and...

Documents