SocialSync: Synchronization for a

Smartphone Camera Network

Richard Latimer

Rice University

September 7, 2014

2

2

Intro

3

Why Smartphones?

• 1 billion smartphones sold in 2014

• Growth in computer vision applications

• Advanced development platform

Best camera is the one with you!

4

4

What if Smartphones Could Collaborate?

Refocusing Free Viewpoint Television

3D Depth

Lytro

PelicanPhilips

Kilner et al.

5

Prior Work: Camera Arrays

Traditional camera arrays are assumed to be embedded and tightly synchronized.

High-Speed Videography Using a Dense Camera Array

[Wilburn et al.]

High Performance Imaging Using Large Camera Arrays

[Wilburn et al.]

6

6

A Smartphone Camera ArrayInvolves the OS Stack

Application

Application Framework

Library Runtime

Linux Kernel/Drivers

Hardware

7

7

Social Capture: Problem Statement

If you use multiple smartphone cameras to capture video simultaneously, there is an inherit synchronization challenge that manifests (due to OS stack) when the scene is moving.

8

8

Scene Setup: How Bad is Misalignment?

Scene Setup

100

100

100

100

Resolution TargetSmartphone

Camera Array

9

9

Outcome: How Bad is Misalignment?

Even when photos are captured simultaneously, misalignment of frames can

cause problems when target is moving.

Moving Resolution Target Captured by 4 Phones Simultaneously

10

10

Source of Problem

Frame 0 Frame 1 Frame 2

Frame 0 Frame 1

Frame 1 Frame 2 Frame 3

time

Camera 1:

Camera 2:

Camera 3: Frame 2

Sub-FrameMisalignment

Frame 0

CaptureRequest

11

11

Goal of SocialSync

Objective: Align frame sequences from smartphone cameras within tolerance ranges less

than t milliseconds, where t < frame duration.

Frame 0 Frame 1 Frame 2

Frame 0 Frame 1

Frame 1 Frame 2 Frame 3

time

Camera 1:

Camera 2:

Camera 3: Frame 2

Sub-FrameMisalignment

Frame 0

CaptureRequest

12

12

1. Characterization

2. SocialSync Protocol

3. Evaluation

13

13

1. Characterization

• Measurement Test Apparatus

• Results

2. SocialSync Protocol

3. Evaluation

14

14

What We Use: Android 4.4

15

15

Network Setup

100

Hub and Spoke Network

• Server provides global reference clock

• Server broadcasts image capture request

100

100

100

100

16

System for Measuring Capture Timestamps

16

17

17

1. Characterization

• Measurement Test Apparatus

• Results

2. SocialSync Protocol

3. Evaluation

18

18

Random Variables Associated with Capturing a Frame

Setup Time: Delay to setup camera before capture of first frame

Delivery Time: After capture, there is delay to when frame reaches application and a timestamp can be recorded.

19

19

Camera Setup

Setup Frame 0 Frame 1 Frame 2

time

20

20

Camera 1: Is Setup Time Consistent?

Setup

Setup

Setup

time

21

21

Camera 1: Is Setup Time Consistent?

Setup

Setup

Setup

time

Frame 0

Frame 0

Frame 0

22

22

Variability in Camera Setup Makes Aligning Frames Challenging

Each time camera started, the start of the image sequence will be delayed by a variable amount of time.

• Setup can vary at least 35 ms

• StDev = 9.4 ms

• Not deterministic

23

23

Frame Delay

Setup Frame 0

time

TD(0)

Frame 1

TD(1)

Frame N

TD(n)

. . . . .

24

Measurement of Frame Delay

25

25

Delivery Time of Frame to Application

If we keep measuring capture times of frames, they are spaced evenly

Delivery time of frame to app is a proxy for estimating capture time with delay:• μ = 67 ms• σ = 4.5 ms

26

26

1. Characterization

2. SocialSync Protocol

• Estimation

• Alignment

3. Evaluation

27

27

Setup Frame 0 Frame 1 Frame 2

Setup Frame 0 Frame 1

Setup Frame 0 Frame 1 Frame 2

time

Camera 1:

Camera 2:

Camera 3: Frame 2

Goal of Social Sync

Sub-FrameMisalignment

SyncTolerance

28

Social Sync Protocol

Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames

Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment

28

29

29

Setup Frame 0 Frame 1 Frame 2

Setup Frame 0 Frame 1

Setup Frame 0 Frame 1 Frame 2

time

Camera 1:

Camera 2:

Camera 3: Frame 2

If Misalignment is Due to Camera Setup

Sub-FrameMisalignment

Frame 2

30

30

Setup Frame 0 Frame 1 Frame 2

Setup Frame 0 Frame 1

Setup Frame 0 Frame 1 Frame 2

time

Camera 1:

Camera 2:

Camera 3:

Could We Estimate Capture Time By Calibrating for Setup Time?

Sub-FrameMisalignment

31

31

Recall Wide Variability in Setup

If setup time had little variance, could align capture timestamps by offsetting

• Setup can vary at least 35 ms, when worst case frame misalignment was 33 ms (inverse of FPS)

• Can’t calibrate for setup time to reduce misalignment!

32

32

Does Frame Delay Provide a Better Estimate of the Capture Timestamp?

Setup Frame 0

time

TD(0)

Frame 1

TD(1)

Frame N

TD(n)

. . . . .

33

Measurement of Frame Delay

34

How does estimate of mean improve with number of samples?

Stats 101: Standard error of a sample mean is

Based on Central Limit Theorem, 95% confidence sample mean is within range:

35

35

Mean Frame Delay Recap

Setup Frame 0

time

TD(0)

36

36

Mean Frame Delay Recap

Setup Frame 0

time

TD(0)

TC(0)

Mean Delay Est.:

37

37

Mean Frame Delay Recap

Setup Frame 0

time

TD(0)

TC(0)

Frame 1

TD(1)

TC(1)

Mean Delay Est.:

38

38

Mean Frame Delay Recap

Setup Frame 0

time

TD(0)

TC(0)

Frame 1

TD(1)

Frame N

TD(n)

. . . . .

TC(1) TC(n)

Mean Delay Est.:

39

Samples Required for Estimate

Since mean delay is , we can determine the capture image sequence position for milliseconds using:

• 22 samples for 2 ms accuracy

• 85 samples for 1 ms accuracy

39

40

Social Sync Protocol

Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames

Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment

40

41

41

Synchronization Tolerance

time

Request Sync ToleranceGoal: Want

synchronization to a reference

image sequence to be aligned within some

tolerance

Capture Request

42

42

Synchronization Attempt 1

Setup Frame 0

time

Sync Tolerance

43

43

Synchronization Attempt 1

time

. .

Sync Tolerance

Frame NSetup Frame 0

44

44

Synchronization Attempt k

Setup Frame 0

time

Sync Tolerance

45

45

Synchronization Attempt k

time

. .

Sync Tolerance

Frame NSetup Frame 0

46

Obtaining Synchronization

• Let T be the worst case misalignment between frames

• Let t be the range specified by the sync tolerance, where t ≤T

Let p = t/T, be the probability of synchronization error being reduced for a single phone

46

47

Single Camera Sync Probability After k Attempts

Pk = 1-(1-p)k

Expected Number of Sync Cameras

n*Pk

47

48

Expected Attempts to Sync N Cameras

48

49

49

•Characterization

•SocialSync Protocol

•Evaluation

50

Evaluating Synchronization

NaïveSync SocialSync

4 Cameras

8 Cameras

23 ms

35 ms

5 ms

6 ms

50

Smartphones in array to reduceartifacts not related to timesynchronization error

51

8 Cameras

Naïve Sync

SocialSync

Reference Views Depth Map 51

52

4 Cameras

Reference NaïveSync SocialSync

52

53

4 Cameras

NaïveSync SocialSyncReference

53

54

Conclusion

1. Characterize Android Camera2. Developed SocialSync Protocol3. Demonstrated SocialSync

Future Work:- Greater control of Android camera system- In wild implementation

54

55

Acknowledgements

1. Ashok, Lin, Ashu and their students2. Co-Author Jason Holloway3. LF4CV Reviewers, Paper Accepted4. National Science Foundation

55