Online Tracking of Outdoor Lighting Variations for Augmented Reality

with Moving Cameras

Yanli Liu1,2 and Xavier Granier2,3,4

1: College of Computer Science, Sichuan University, P.R.China

2: INRIA Bordeaux Sud-Ouest, France

3:LP2N (CNRS, Univ. Bordeaux, Institut d'Optique)

4:LaBRI (CNRS, University of Bordeaux)

Augmented reality mobility

Motivation

MotivationTwo consistency

Geometric consistency Devices Camera position

GPS, UWB, Omnisense WiFi, cell information

Camera pose Linear accelerometers

Tracking via computer vision [Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image

and Vision Computing 2008]

Illumination consistency outdoor lighting is largely dependent on

weather and time

MotivationTwo problems

Online process first step toward real-time solutions

Moving viewpoints Handhold

camera jitter

Previous WorkMarkers or lighting probes [Debevec Siggraph’ 98,

Agusanto ISMAR’03, Kanbara ICPR’04, Hensley I3D’07]

too dense sampling our method does not require any

supplemental devices

Debevec Siggraph’ 98

Previous WorkThree components of shading

BRDF geometry lighting

Fix other one or two components [Wang PG’02, Li ICCV’03, Hara PAMI’05, Andersen ICPR’06, Sun ICCV’09]

3D reconstruction controlled environment (indoor or lab)

[Wang PG’02]original image rendered image

Previous WorkTime-lapse outdoor video analysis [Sunkavalli

Siggraph’07, Sunkavalli CVPR 08]

take whole video sequence as input Post-processing

[Sunkavalli Siggraph’07]

Previous WorkLearning based outdoor illumination

estimation [Liu TVC’09, Liu CAVW’10, Xing C&G’11]

offline stage learning fixed viewpoint

Liu CAVW’10

moving viewpoints

Our MethodKey ideas

Tracking illumination variation by tracking feature points

Feature points tracking is error prone.

Select reliable feature points using global illumination constraint and spatial-temporal coherency.

Outdoor lighting [Sunkavalli SIG’07, Sunkavalli CVPR 08, Madsen InTech 2010]

the sunlight directional light colored intensity sun direction

the skylight ambient light colored intensity

( )sun tL

( )sky tL

( )l t

Illumination and BRDF model

Illumination and BRDF modelNeutral reflection model [Lee PAMI’90, Montoliu

LNCS’05, Eibenberger ICIP 2010, ICCV 2011]

the color of the specular reflection is the same as the color of the incident lighting.

Phong-like model

, ( )( ) [ , ( ) ] ( )

( )

pm sunp pp pp pp

skyp

n h tt k n l t tsI L

tL

System Initialization Tracking illumination variation by tracking

feature points 3D geometry vs normals planar feature points

KLT feature-points

clustered feature-points

first frame

plane segmentation[Hoiem IJCV’07]

mean-shift color segmentation

threshold-based Shadow detection

System InitializationBRDF initialization

pixels difference at in sun lit regions depend on specular parameters and :

Assuming piecewise constant , and

Spatially varying diffuse

p

pk pm

,

2

( , )i j

diffdiffpi i p

j p

E k m I I

p

2, 1,

(2) (1)

, ,[ ]p p

p p

diffp pp

diff m msunp p pp

I I I

n h n hI k L

pk pm 1sunL

2 2

, 1 ( ) ( )( 1) ( 1)sun skysmooth sun sky

t t t L t Lt tE L L

Energy function

Outdoor lighting is nearly constant during time intervals less than 1/5 second.

control the smooth degree of skylight

Alignment-based weight

, 1( ) (1 ) ( )data smootht tE t tE E

,

2

, ,,

( )i j

datappi j p

i j p

t I IE

2( )/

, ,,

1= ijpI I

i j pi j

ez

Tracking Lighting Variation with Reliable Feature Points

Tracking Reliable Features and Their AttributesFeature points labeling

Three attributes: Normal (plane, homography matrix) BRDF parameters Shadow situation

Spatial & temporal coherency

Tracking Reliable Features and Their AttributesFeature points labeling

t -1 t

paired point is labeled in 1t

current point is not in shadow

22

, ,( ) (1 )i j p i p i jpd q H p I I

, ,i ik m

ppq

compute lighting

,i ik m

tL

p

Results and DiscussionQuantitative results

PC: Intel i7 2.67GHz and 6GB RAMMATLABVideo resolution 640 480

Average fps and average number of feature points estimated on 1,000 frames

Results and discussionQuantitative results

Average percentage of different steps in total computational cost

Results and DiscussionVisual results

Building scene Wall scene

ConclusionFully image-based pipeline

online tracking of lighting variations of outdoor videos.

Manages lighting changes and misalignment of feature points

Ensure a stable estimation on a sparse set feature points.

Limitations and Future WorkRough shadow detection

3D reconstruction vs shadow detection Sun-lit features

Initialization automatic initialization: easy but may fail in

some cases manual initialization: may be tedious for a

non-expert user. Semi-assisted initialization

Limitations and Future WorkTracking independently on R, G, and B

channels priori model of outdoor illumination color or

spectra difficult to optimization

The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints.

Thanks for your attention!

Questions?