Synchronization and Calibration of Camera

Networks from Silhouettes

Sudipta N. Sinha Marc Pollefeys

University of North Carolina at Chapel Hill, USA.

2

Goal

To recover the Calibration & Synchronization of a Camera Network from only Live Video or Archived Video Sequences.

3

Motivation

• Easy Deployment and Calibration of Cameras.– No Offline Calibration ( Patterns, LED etc)– No physical access to environment

• Possibility of using unsynchronized video streams (camcorders, web-cams etc.)

• Applications in wide-area surveillance camera networks (3D tracking etc).

• Digitizing 3D events

4

Why use Silhouettes ? Visual Hull (Shape-from-Silhouette) System

• Many silhouettes from dynamic objects

• Background segmentation

Feature-based ?• Features Matching hard for

wide baselines• Little overlap of backgrounds• Few features on foreground

5

Prior Work : Calibration from Silhouettes

Epipolar Geometry from Silhouettes • Porrill and Pollard, ’91• Astrom, Cipolla and Giblin, ’96

Structure-and-motion from Silhouettes• Vijayakumar, Kriegman and Ponce’96 (orthographic)• Furukawa and Ponce’04 (orthographic)• Wong and Cipolla’01 (circular motion, at least to start)• Yezzi and Soatto’03 (needs initialization)

Sequence to Sequence Alignment • Caspi, Irani,’02 (feature based)

6

Our Approach

• Compute Epipolar Geometry from Silhouettes in synchronized sequences (CVPR’04).

• Here, we extend this to unsynchronized sequences.

• Synchronization and Calibration of camera network.

7

Multiple View Geometry of Silhouettes

Frontier PointsEpipolar Tangents

• Always at least 2 extreme frontier points per silhouette

• Only 2-view correspondence in general.

x1 x2

x’1x’2

0Fxx1

T

2

0xFx1

T

2

8

Camera Network Calibration from Silhouettes

• 7 or more corresponding frontier points needed to compute epipolar geometry

• Hard to find on single silhouette and possibly occluded

• However, video sequences contain many silhouettes.

9

Camera Network Calibration from Silhouettes

• If we know the epipoles, draw 3 outer epipolar tangents (need at least two silhouettes in each view)

• Compute an epipolar line homography H-T

• Epipolar Geometry F=[e]xH

10

RANSAC-based algorithm

Repeat {• Generate a Hypothesis for the Epipolar Geometry• Verify the Model

}

Refine the best hypothesis.

• Note : RANSAC is used to explore 4D space of epipoles apart from dealing with noisy silhouettes

11

Compact Representation for SilhouettesTangent Envelopes• Store the Convex Hull

of the Silhouette.

• Tangency Points for a discrete set of angles.

• Approx. 500 bytes/frame. Hence a whole video sequences easily fits in memory.

• Tangency Computations are efficient.

12

RANSAC-based algorithm

Generate Hypothesis for Epipolar Geometry

• Pick 2 corresponding frames, pick random tangents for each of the silhouettes.

• Compute epipoles.

• Pick 1 more tangent from additional frames

• Compute homography

• Generate Fundamental Matrix.

13

RANSAC-based algorithm

Verify the ModelFor all tangents

Compute Symmetric Epipolar Transfer Error Update Inlier Count

(Abort Early if Hypothesis doesn’t look Promising)

14

What if videos are unsychronized ?

For fixed fps video, same constraints are valid up to an extra unknown temporal offset.

• Add a random temporal offset to RANSAC hypothesis.

• Use multi-resolution approach:– Keyframes with slow motion, rough

synchronization– ones with fast motion provide fine

synchronization

15

Computed Fundamental Matrices

16

Synchronization experiment

• Total temporal offset search range [-500,+500] (i.e. ±15 secs.)• Unique peaks for correct offsets• Possibility for sub-frame synchronization

# Promising

Candidates

Sequence Offset (# frames)

# Iterations(In millions)

17

Camera Network Synchronization

• Consider directed graph with offsets as branch value

• For consistency loops should add up to zero

• MLE by minimizing

+3

-5+8

+6

+2

0

22 tt

ground truth

in frames (=1/30s)

18

From epipolar geometry to full calibration

• Solve for camera triplet (Levi and Werman, CVPR’03;

Sinha et al. CVPR’04)

• Assemble complete camera network.

19

Metric Cameras and Visual-Hull Reconstruction from 4 views

Final calibration quality comparable to explicit calibration procedure

20

Validation experiment:Reprojection of silhouettes

21

Taking Sub-frame Synchronization into account

Reprojection error reduced from 10.5% to 3.4% of the pixels in the silhouette

Temporal Interpolation of Silhouettes.

to appear (Sinha, Pollefeys, 3DPVT’04)

22

Conclusion and Future Work

• Camera network calibration & synchronization just from dynamic silhouettes.

• Great for visual-hull systems.• Applications for surveillance systems.

• Extend to active PTZ camera network and asynchronous video streams.

Acknowledgments• NSF Career, DARPA.• Peter Sand, (MIT) for Visual Hull dataset.