fondamentals of virtual and augmented reality click to edit … · 2019-10-02 · 3 “virtual and...

Click to edit Master title style

“Virtual and Augmented Reality”

Interaction speciality – Computer Science Master - University Paris-Saclay

Fondamentals of Virtual and Augmented Reality

AR: an introduction

Jeanne Vezien

[email protected]

2

“Virtual and Augmented Reality” course Interaction speciality – Computer Science Master - University Paris-Saclay

Plan of the lecture

1. Augmented reality:

what and why

2. User tracking

3. Real world reconstruction

4. 3D Graphics

5. Augmentation

3


Wikipedia says…

Augmented reality (AR) is a term for a live

direct or indirect view of a physical, real-world

environment whose elements are augmented by

computer-generated sound, video, graphics,

haptic or GPS data.

Augmentation is conventionally in real-time and

in semantic context with environmental

elements, such as sports scores on TV during a

match.

With the help of advanced AR technology

(computer vision and object recognition) the

information about the surrounding world

becomes interactive, e.g. artificial information

about the environment can be overlaid.

Augmented reality was coined by Thomas

Caudell, working at Boeing, in 1990. ARToolkit (Kato & Billinghurst, 2001)

4


AR

InteractiveReal-time

Contextualized

5


The dinosaur

is rendered:

- Interactively

- In real-time

- In context

AR example

6


2018: the competition began

• Hololens (Microsoft, 2016)

• Commercial in 2017

• 50 000 units sold.

• V2 in 2019

• Magic Leap (Magic Leap Inc, 2018)

• Released august 2018

• Maximum Hype

• No business (yet)

7


2019: the competition is on

Hololens 2

3500$

Real time hand tracking

(grab, pinch, push,

slide…)

Gaze tracking

Magic Leap 1

2.295 $

8


Real/virtual Continuum (Milgram 1994)

9



10



11



12



13



14



15


AR: examples

Not a new idea….

• User position and gaze provides context ,

sometimes with help (markers)

• Audio is nice (non-obstrusive), can be

stopped anytime.

• No computer involved

• No sensing involved : limited interaction

Now associated with head-tracking,

localization…

Started in 1957 (Roosevelt home)

Principio system

(2007)

16


AR: examples

See-through augmented vision: « classic AR »

Tourism: Archeoguide (2002)

17


AR: examples

Assemply / Maintenance / Repair

Maintenance: help technician with

contextualized content

BMW, 2010

Matris project, 2007

Fiducial Text

ActionGraphics

Icons

18


AR: examples

Way-findingIphone app: NY nearest subway (and many cities, airports have their own app)

https://www.youtube.com/watch?v=ps49T0iJwVg

Augmented Car Finder

Note:

• Data is collected off-line

• Access to database is native on mobile phones

(http protocol)

19


AR: examples

Safe landing

Hunter system: Landing guiding system by AR

tracking techniques (computer vision based)

www.novemberkiloecho.com

20


War "games" (1/2)

Track target w.r.t. weapon

Coupled with (off-line) Geographical Information

system

… and strategic realtime info (GPS, detectors)

gun-mounted display

21


War "games" (2/2)

Tanagram.com concept (closed)

22


Heads Up Displays

TopOwl helmet (helicopter)

In operation since 2008

Thalesgroup.com

Eye tracking

Symbology

Gun pointing

Night VisionHeavy and $$$ !

23


Heads Up Displays (2)

Skully web helmet (2015)

(+ rear cameras)

Intuitive Aerial (Sweden, 2013)

24


MR: examples

Games and promotional contents

EyePet for PSP (2009)

Topps 3D baseball cards (2009)

PSVita (2012)

Father.io (2016): Massive Multiplayer Laser Tag

25


Pokémon GO !

The Pokemon Company (Nintendo) + Niantic (Ingress)

Since July 2016

Game brought in more than $1.2 Billion !

2018: 104M$ / month !

Success = the right mix of game + social

Possible only with AR (or connected VR ?)

26


Other examples

Medical: Surgery planning, nurse and student training…

University of North Carolina

Augmented virtuality (presented on screen)

Track hand-held device or body

Coupled with (off-line) anatomic data

27


DaVinci Robot

Intuitive Surgical

3 000 ex.

1 M.$ per unit30 problems over 17000

interventions, 1 death (bad

manip.)

Less stress

More precision

Comfortable

Complete asepsis

No tremor

abdomen

surgery

28


• Militaires: « future soldier », BARS

• Médical : assistance chirurgicale

• Tourisme

• Customization (mode)

Virtual try-ons and Customize

Augmented virtuality or

AR

Track users anatomy

and motion (better still)

Coupled with external

data on-line

Augment.com (2017)Ikea (2014)

ForeverMark (2011)

29


MR: the SACARI example

Mixed virtuality: user not moving with camera,

indirect view of real world

Tele-immersion : provide sense of presence of remote environment

Internship possibilities in VENISE team !

N. Khenak et al. 2019

30


Blurred boundaries: augmented movies

Is this real ? Augmented ? Virtual ?

On-line ? Off-line ?

Avatar (2009)Zoe Zaldana

31


Blurred boundaries

Is this still AR ?

AR will be everywhere

when efficient, cheap see-

through displays will

become available

Google moto: The World's

Information in Context

Street View is close to

Augmented Virtuality

Google glass 2 (2019)

32


The Ingredients for AR

Capture real world

33



Capture real world

+

Capture virtual world

(Computer Graphics)

34



Capture real world

+

Capture virtual world

(Computer Graphics)

Present to user

(Augmentation)

35


Real 3D world capture

Two ingredients necessary for

successful AR:

Real-time 3D tracking of user

viewpoint w.r.t. world

3D scene analysis realistic

augmentation WorldViz tracking

Structure sensor

(Occipital, 2019) https://structure.io/

36


Real 3D world


successful AR:

Real-time 3D tracking of

user viewpoint w.r.t. world

3D scene analysis

realistic augmentation

37


Real-Time tracking

Many means to compute positioning information:

Geolocalization

Electro-magnetic

Acoustic

Inertial

Vision-based (active or passive)

Covered in « stereo and tracking » course

38


Image-based 3D tracking

It is a special case of adaptive

rendering ( for HMDs)

Environment control ++

Markers are still needed

(often)

Special case of

structure/motion estimation

AR meets movie industry:

the user is the director !

markersKnown as motion estimation / match moving

39


From [3]

= user

From [3]

Inside-out vs. Outside-in

= user

Inside-out: user wears camera Outside-in: camera observes user

• Inside-out better at estimating relative rotation

• Outside-in much more convenient : don’t have to wear a camera

• Special case: console gaming

• Inside-out easier for wearable AR: mobility, ego-centered

reference frame

• Needed for nomadic apps

• Cameras are getting small and commodity items

40


Projection is necessary to map 3D points on 2D pixels:

P(X,Y,Z)

o

p(x,y)

f

z

x

Focal point

Focal plane

11

Z

Y

X

Z

Y

X

y

x

s

Z

Yy

Z

Xx

0100

0010

0001

3D tracking: equations

41


Objective: find camera parameters and position

110001

333231

232221

131211

Z

Y

X

TzRRR

TyRRR

TxRRR

y

x

s

0100

0010

0001

1100

0

0

1

y

x

v

u

v

u

cv

cu

Tracker

Camera

(R,T)

Perspective equation (extrinsic parameters):

Cam. Position,

(R,T)

Intrinsic parameters (scale, offset):

Pixels

M(X,Y,Z)

3D tracking: equations

42


M

Z

Y

X

P

TtTtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1

Projection equation:

image 3D worldm = P . M

motion structure

The Maths

43


M

Z

Y

X

P

TtTtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1



motion structure

The Maths

World camera =

Camera position

44


M

Z

Y

X

P

TtTtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1



motion structure

The Maths

3D 2D projection

45


M

Z

Y

X

P

TtTtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1


motion structure

The Maths

Sensor image (pixels)


46


M

Z

Y

X

P

TtTtTt

v

u

v

u

sz

y

x

cv

cu

110

k

j

i

0100

0010

0001

100

0

0

1



motion structure

The Maths

(i,j,k,Tx,Ty,Tz) can be computed if m and M are known.

Not linear in the motion parameters !!

47


Dementhon-Davis (1992)

Several methods to recover calibration exist (Tsai, Lowe…)

" Model-Based object pose in 25 lines of code" is a simple

yet elegant method to obtain pose rapidly if a rigid 3D model

is provided.

i

i

i

i

i

i

i

ii

xPI

PTz

kTz

TxP

Tz

i

TzPk

TxPi

Z

Xx

1

.

1.

.

.

. 0

i

ii

yPJy

1

. 0

),( 00 yx Tz

Pk ii

.

Projection of T(Tx,Ty,Tz) Relative depth around Z0=Tz

Idea consists in linearizing xi and yi to compute iJI ,,

ziz TTZ

48


i

o

i

i

o

i

i PJyy

PIxx

.

.

0

011

1

Geometrical interpretation:

P

O

Image plane

P0

Reference plane

Z0=Tz

Orthographic

projection

Z

Y

i Z0

Perspective

projection

Linear approximation of perspective

iio

y

iii

o

i

o

x

y

xxx

49


POSIT algorithm

Summarize:

Tz

jPJyyyy

Tz

iPIxxxx

i

o

iii

i

o

iii

=Jwith.)1(

=Iwith.)1(

00

00

If εi is known then I,J,Tz,X0=x0Tz,Y0=y0Tz can be computed

If I,J,Tz k is known

Iterate starting with

Converges rapidly:

Tz

Pk ii

.

0i

50


3D tracking: off-the-shelf solution

• Kato & Billinghurst (HIT lab, univ. of Washington) introduce

in 1999 a tool for teleconferencing:

"Marker Tracking and HMD Calibration for a video-

based Augmented Reality Conferencing System.“

Soon to become ARToolkit

51


ARToolkit: main characteristics

• Fast and cheap 6D marker tracking.

• Distributed with complete source code.

OpenSource with GPL license for

noncommercial usage.

• Multiplatform (Linux, MacOS and Windows) .

• Multiple input sources (USB, Firewire) ,

multiple format (RGB, YUV) supported.

• Multiple camera tracking supported.

• GUI initializing interface.

• Easy calibration routine.

• Fast rendering based on OpenGL.

• 3D VRML support.

• Simple and modular API (in C and C++).

• Complete set of samples and utilities.

• Supports both video and optical see-through

AR.

52


ARToolkit: main processing loop

Since v4, Pose estimation is performed using the

Iterative Closest Point (ICP) algorithm (Besl, 1992)

53


Important details

Virtual objects appear only when complete markers are visible:

Size

Movement

Orientation

Lighting conditions

Pattern Size (cm.) Usable Range (cm.)

6.98 40.64

8.89 63.5

10.79 86.36

In practice: close range only !

54


3D tracking: off-the-shelf solution

www.layar.com

web-based, per page service

www.wikitude.com: AR SDK

Vuforia (qualcomm) multiplatform SDK

Including Unity

Inglobe AR: AR solutions, ARPlayer+plugins (40.000 registered users)

2490€

$499

focus on printed text

55


Markerless 3D tracking and AR dev: battle of

the SDKs

Spatial anchors

Cloud anchors

1.2ARKit 2

ARWorldMap

56


Real 3D world


successful AR:

Real-time 3d tracking of user

viewpoint w.r.t. world

3D scene analysis

realistic augmentation

57


2 components: Light and Geometry

Lighting coherency Motion coherency

Geometric coherency

Light probe

3D real world analysis

58


Aim: Capture light coming from light sources in real world

10,000:1 is the (static) human eye ratio between brightest and

darkest shade: cannot be represented on 8 bits

Stored in High Dynamic Range images (.hdr image format)

From all directions at a given point

Light Probe

u

v

59


Half-life 2: normal and HDR

Note: advanced graphics techniques can

use hdr images (Unreal engine)

u

v

1,1),( vu

u

v1tan 22 vu

A direction vector in the world (Dx, Dy, Dz)

u = r. Dx and v = r.Dy with

unit vector pointing in the direction (u,v) is obtained by rotating

(0,0,-1) by:

1) degrees around the y (up) axis

2) degrees around the -z (forward) axis.

22

1 )(cos1

DyDx

Dzr

Light Probe (2)

60


World content

Geometry: 3D reconstruction : what is

where ?

Semantic : augmentation of context

object recognition

interpretation of content

Comprendre l’environnement 3D en termes : http://www.truevisionsys.com

61


Markers cannot always be present !

If structure is known: a priori model tracking (see before):

an object is a (collection of) marker

Remember : projection combines structure and motion (m

= P.M)

Idea: recover both simultaneouly !

+ gives tracking and reconstruction at the same time

- highly non-linear : requires iterations + data filtering (remove

outliers )

Geometry: 3D reconstruction

Factorization method introduced by Tomasi and Kanade (1992)

62


mij = Pj. Mi

Factorization method

2x1 = 2x3 3x1

FPF

FPF

P

P

yy

xx

yy

xx

m

1

1

111

111

jth frame

ith point

P

P

P

ZZ

YY

XX

M

1

1

1

FP

P

P 1

• Compact representation:

• Consider P points projecting on F frames:

• There is an of possible (P,M) pairs!

• Solved via SVD decomposition (but not uniquely)

Carlo Tomasi and Takeo Kanade. (November 1992)

"Shape and motion from image streams under orthography: a factorization method."

International Journal of Computer Vision, 9 (2): 137–154.

Images !

m is big, but rank 3 ! m= P.M

63


Modern Approach: SfM

• Structure from Motion (or SLAM)

Inertial-Aided Sequential 3D Metric Surface

Reconstruction from Monocular Image Streams

Aufderheide et al. (2013).

• Initial pose estimation with factorization with a few keyframes

• Non-linear (costly) optimization (bundle adjustment)

• Real-time solutions are starting to emerge

64


S. Christy et R. Horaud : "Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations"

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 18, Number 11, Pages 1098--1104 - November 1996

Factorization: results

Hanno Ackermann, University of Hanovre (2008)

65


Computer Vision is good at locating features:

points

regions

textures

contours

… at different scales !

Image-based 3D reconstruction

Harris (1988)

SURF detector (OpenCV), Bay (2006)

2D region detection (region-growing)

66


Problems :

Robustness of detection

Matching (space/time)

Is becoming robust enough

For now: limited use , constraints

(model-based)

Moving fast with new AR devices

Markerless tracking

Wang & Popovic, MIT, 2010

67


Boujou: http://vicon.com/boujou/

Commercial solutions

Pricey (3500€)

68


New real-time sensors: depth cameras (Zcam)

Infer depth for ALL pixels

Early : laser, stereoPerceptron

LIDAR , 1995

Devernay, 1994


69


Now: structured lighting : Zcam for 100 € !

Kinect, 2010

Augmented Reality Magic

Mirror using the Kinect

Tobias Blum


Solves the "where" but not the "what":

objects must still be identified (segmented)

70



Hololens (2016, 3000$)

Augmented reality « see-through »

glasses

Markerless 3D scanner with :

depth camera (Kinect-like) Spatial

Mapping

4 environment cameras

Inertial Measurement UnitHead tracking

SLAM = Simultaneous Localization and Mapping

= markerless tracking !

}

71



Magic Leap (2018, 2295$)

Augmented reality « see-through »

glasses

2 focus: near/far !

Markerless 3D scanner with :

depth camera (Kinect-like) Spatial

Mapping

4 environment cameras

Inertial Measurement Unit

Magnetic tracker

Head tracking

SLAM based on Deep Neural Network

}DeTone & Malisiewicz

Towards Geometriv Deep SLAM

https://arxiv.org/pdf/1707.07410.pdf

72


Stereo for AR: augmentation

Object location (table…)

Object occlusion

Shadows and light

interactions

In real-time !!

Will soon happen in

movie industry ($$$)

3D reconstruction is necessary for realistic CG blending:

Source: X3d consortium

73


3D analysis of

real images

(camera +

reconstruction)

Augmentation: CG blending

74


CG graphics+3D analysis of

real images

(camera +

reconstruction)


75


CG graphics

Compositing masks

(shadows, occlusions…)

+3D analysis of

real images

(camera +

reconstruction)


match move

76


CG graphics

Compositing masks

(shadows, occlusions…)

Augmented image

+3D analysis of

real images

(camera +

reconstruction)


match move

77


Matching constraints:

• geometry (epipolar)

• photometry

H. Jin, P. Favaro, and S. Soatto.

A Semi-direct Approach to Structure From Motion. The Visual Computer, 19(6): 377-394, October 2003.

Example of augmentation pipeline (1)

image analysis

Region matching (1992)

78



3D reconstruction

3D reconstruction of regions based on planar equations:

Hypothesis: world is piece-wise planar

79


Prior information:

explicit 3D model

motion constraints

3D registration by ICP = Iterative

Closest Point (Besl 92):

always converges

model points do not coincide with

reconstruction points


3D reconstruction

80



3D virtual content

Two steps:

Geometric modeling

(sometimes based on 3D

scans): Maya, Unity,

Blender, Sketchup, etc.

Photo-realistic rendering:

textures, lights, reflectance,

shadows, etc.

Note: Light probes

environment mapping

81


Light interaction: a must

Geometric coherency is good

but… light is essential !

Shadows in real-time =

geometry + light

Standard technique for CG renderers:

Ground (real)

(virtual)PSP

.'

dnEc

.nLd

.

E

S

82


Shadow projection techniques

Many different techniques !

Plane Projected

Shadows

Projected

Shadows

Depth Shadow

MappingVertex Projection

Shadow

Volumes

Quick, not much

calculations

Quick, almost no

calculations

Very quick, not

much calculations

Slow with high-res

meshes

Slow, lots of

calculations

High detail Detail depends

on texture

Detail depends

on textureHigh Detail High detail

No self-

shadowingNo self-

shadowingSelf-shadowing

No self-shadowing Self-shadowing

No shadow

receivers

Shadow

receivers

Shadow

receivers

No shadow

receivers

Shadow

receivers

83


Shadow mapping

M. Haller, S. Drab, and W. Hartmann, 2003.

"A real-time shadow approach for an augmented reality application using shadow

volumes," in VRST 03: Proceedings of the ACM symposium on Virtual reality software and

technology, New York, NY, USA, 2003, pp. 56-65

Shadow volume with stencil drawing

84



animations

Two steps:

Virtual motions must be

coherent with real ones :

match-moving

Motion is camera-centric

Virtual camera must be

identical to real one (off-line

calibration is necessary)

Accommodate zoom !

85


Occlusion mask

Virtual object rendering

(alone)


Step by step rendering

86


Real objects: reference white

(albedo)

"black" virtual objects:

Shadow computation



87


Final composition =

real image *(1-mask) * attenuation

+

Virtual objects * mask

+ reflexions * (1-mask)



88


Final compositing

89


Other example (1)

Objets déformables + occultations réel/virtuel

90


Other example (2)

Reference white

Shadows of

virtual objects on

the real scene

91


Other example (3)

Final result

92


Conclusion

• Augmented reality is a reality

• Is a complicated process, requires

a lot of expertise and hard/software

• In-depth coherency of geometry

and photometry

• Real-time challenges

Elaborate rendering (GPU)

Match moving (markerless)

3D reconstruction (occlusions)

User interaction: hand and

body tracking

93


THE END !

Denno Coil, 2007

fondamentals of virtual and augmented reality click to edit … · 2019-10-02 · 3 “virtual and...

Documents