Complementary Sets of Shutter Sequences for Motion Deblurring

Hae-Gon Jeon1 Joon-Young Lee1 Yudeog Han2 Seon Joo Kim3 In So Kweon1

1 Robotics and Computer Vision Lab., KAIST 2 Agency for Defense Development 3 Yonsei Univerisity

hgjeon@rcv.kaist.ac.kr jylee@rcv.kaist.ac.kr ydhan@add.re.kr seonjookim@yonsei.ac.kr iskweon@kaist.ac.kr

Abstract

In this paper, we present a novel multi-image motion

deblurring method utilizing the coded exposure technique.

The key idea of our work is to capture video frames with

a set of complementary fluttering patterns to preserve spa-

tial frequency details. We introduce an algorithm for gen-

erating a complementary set of binary sequences based on

the modern communication theory and implement the coded

exposure video system with an off-the-shelf machine vision

camera. The effectiveness of our method is demonstrated on

various challenging examples with quantitative and quali-

tative comparisons to other computational image capturing

methods used for image deblurring.

1. IntroductionImage deblurring is a challenging task that is inherently

an ill-posed problem due to the loss of high frequency infor-

mation during the imaging process. In the past decade, there

have been significant developments in the image deblurring

research that improve the performance over the traditional

deblurring solutions such as Richardson-Lucy [29, 23] and

Wiener filter [42].

One research direction that has gained interest is to use

multiple blurred images for deblurring, which shows bet-

ter performance over the single image deblurring methods

in general due to the complementary information provided.

Yuan et al. used a blurred/noise image pairs to estimate the

blur kernel [43], Cai et al. proposed to use multiple sev-

erly motion blurred images [4], and Chen et al. performed

an iterative blur kernel estimation and a dual image deblur-

ring [6]. Cho et al. [7] presented a video deblurring ap-

proach that uses sharp regions in a frame to restore blurry

regions of the same content in nearby frames. In [32, 33],

motion blur in a video is reduced by increasing the frame-

rate for temporal super-resolution.

Another development in the image deblurring research

is to modify the way images are captured to make the de-

blurring problem more feasible. In [21, 20, 28], videos are

captured by modulating each pixel independently by binary

patterns, which enables the recovery of high temporal reso-

lution. On the other hand, this paper is particularly related

to the works that employ imaging systems that control the

exposures for the whole image, not on the pixel level, during

image captures. In [27], Raskar et al. presented the coded

exposure photography that flutters the camera’s shutter open

and closed in a special manner within the exposure time

in order to preserve the spatial frequency details, thereby

enabling the deconvolution problem to become well-posed

(Fig. 1(a)). Jeon et al. [14] improved deconvolution perfor-

mance by computing optimized fluttering patterns.

Instead of the fluttering shutter within a single expo-

sure, Agrawal et al. [2] proposed a varying exposure video

framework which varies the exposure time of successive

frames (Fig. 1(b)). The main idea is to image the same ob-

ject with varying PSFs (Point Spread Functions) by varying

the exposures so that the nulls in the frequency component

of one frame can be filled by other frames, making the de-

blurring problem well-posed. Holloway et al. [12] applied

the concept of coded exposure into a video camera. This

approach captures a series of coded exposure images with

different fluttering patterns in successive frames and per-

form temporal resolution upsampling via compressed sens-

ing. As mentioned in [12], this approach cannot handle

scenes without the spatio-temporal continuity and requires

many observations for estimating an object motion.

In this paper, we propose a coded exposure video scheme

which combines the benefits of the coded exposure imag-

ing [27, 14] and the varying exposure video [2]. Rather

than varying the exposures between the frames, we capture

a video under a fixed exposure time and apply the coded

exposure on each frame (Fig. 1(c)). To minimize the fre-

quency information loss during the image capture, we in-

troduce a method for generating a complementary set of

fluttering patterns that compensate for the losses in succes-

sive frames. By using the complementary sets of fluttering

patterns, we can generate various exposure time sequences

for a flexible frame rate capture and achieve higher quality

deblurring results with improved SNR compared to the pre-

vious methods. In addition, we show our framework can be

applied to other application such as the privacy protection

for video surveillance by adding intentional blur [19].

13541

PSNR : 18.68, SSIM : 0.79

Pattern

Frequency

PSNR : 19.64, SSIM : 0.74

Frequency

Pattern

PSNR : 20.88, SSIM : 0.89

Pattern

Frequency

(a) (b) (c)

Figure 1. Comparisons of different computational imaging techniques for image deblurring. (a) Coded exposure imaging [27]. (b) Varying

exposure video [2]. (c) Proposed coded exposure video.

2. Complementary Set of Sequences and

Coded Exposure

The key idea of this paper is to capture video frames with

a set of fluttering patterns that compensate frequency losses

in each frame, so that the captured images preserve spatial

frequencies. To generate such fluttering patterns, we intro-

duce a complementary set of binary sequences [38], which

is widely used in many engineering applications such as the

multiple-input multiple-output (MIMO) radar and the code

division multiple access (CDMA) technique [35]. In this

section, we theoretically show the advantage of the coded

exposure video with the complementary set of fluttering

patterns over the coded exposure imaging [27, 14] and the

varying exposure video [2].

2.1. Coded Exposure Imaging vs. Coded ExposureVideo

In [27], it has been shown that a fluttering pattern with a

flat spectrum improves the quality of the deblurring in the

coded exposure imaging. To measure the flatness, they use

the sum of an autocovariance function of a fluttering pattern.

It is also shown in [14] that an autocorrelation function of

a binary sequence can be approximated by an autocovari-

ance function. With a binary sequence U = [u1, · · · , un] of

length n, the relationship between the autocorrelation and

the modulated transfer function (MTF : a magnitude of fre-

quency response of binary sequence) via the Fourier trans-

form of the sequence is derived as ([13])

n−1∑

k=1

Ψ2k =

1

2

∫ 1

0

[|F(U)|2 − n

]2dθ, (1)

where F(U) represents the Fourier transform of the se-

quence U . Ψk denotes kth element of the autocorrelation

function Ψ of the sequence, which is defined as

Ψk =

n−k∑

j=1

ujuj+k. (2)

In [14], it is shown that a smaller value of Eq. (1) reflects

higher merit factor, which in turn results in better deblurring

performance. Ukil proves in [39] that the minimum value

of Eq. (1) is bounded by n/2.

In our coded exposure video framework, a complemen-

tary set is defined as a set of binary sequences where the

sum of autocorrelation functions of the sequences in the set

is zero. If we have a complementary set ∆ consisting of

p(≥ 2) sequences {U1, · · · , Up} of length n, the relation-

ship is denoted as

p∑

i=1

Ψik = 0 s.t. k 6= 0, (3)

where Ψik denotes kth element of the autocorrelation func-

tion Ψ of the ith sequence in ∆.

It is shown in [34] that a complementary set ∆ is com-

puted by minimizing

n−1∑

k=1

|p∑

i=1

Ψik|2 =

1

2

∫ 1

0

[p∑

i=1

|F(Ui)|2 − pn

]2

dθ. (4)

In the optimal case, the minimum value of Eq. (4) becomes

zero from Eq. (3). This means that the joint spectrum of a

complementary set has flatter spectrum than that of a single

binary sequence (coded exposure imaging) since Eq. (1) is

bounded to n/2 for a single binary sequence as mentioned

above. We estimate a spectral gain of one example comple-

mentary set of sequences using a numerical measure, pro-

posed by Tendero et al. [37], and the gain of the sequence

set is 0.85 while the gain of an optimal snapshot of a single

coded pattern is 0.56.

2.2. Performance Invariance to Object Velocity

When the object moves over a range of n pixels dur-

ing a shot, the optimal length of a fluttering pattern U =[u1, · · · , un] is n. As demonstrated in [24], if the ob-

ject moves twice as fast, the effective PSF is stretched

3542

12n [u1, u1, · · · , un, un] and the invertivility of the PSF can-

not be guaranteed.

We show that complementary set of sequences mini-

mizes the loss of spectral information even when the effec-

tive PSFs are super-sampled or stretched due to the velocity

of an object. As an example, we derive the change of MTF

due to a stretched sequence by a factor 2:

bi =

2n−i−1∑

j=0

U2nU2n(j + i)

=

n−q−1∑

p=0

[U2n(2p)U2n(2p+2q)+U2n(2p+1)U2n(2p+2q+1)]

=

n−q−1∑

p=0

[Un(p)Un(p+ q) + Un(p)Un(p+ q)] = 2Ψk,

where j = 2p and i = 2q. (5)

The variance of MTF of the stretched PSF is constant times

bigger than the PSF as follows:

n−1∑

i=1

bi = 4

n−1∑

k=1

Ψk2 = 2

∫ 1

0

[|F(Un)|2 − n]2dθ. (6)

As mentioned in Sec. 2.1, the optimal bound of12

∫ 1

0[F(Un)− n]dθ in the complementary set of sequences

is theoretically zero. Thus, the optimal bound of the deci-

mated PSF also become zero. In practice, because our set

of sequences is close to the optimal bound, the proposed

complementary set can handle the velocity dependency is-

sue and it can be shown for the case of any factor. Our sys-

tem’s robustness to the object velocity is also demonstrated

in the Experiments Section.

2.3. Varying Exposure Video vs. Coded ExposureVideo

To compare our coded exposure video framework with

the varying exposure video [2], we analyze the upper bound

of MTFs for these two methods. The main criteria for the

comparisons are the variance, the minimum, and the mean

of MTF. As shown in [27, 25], the MTF of a binary se-

quence in coded exposure photography has a direct impact

on the performance of image deblurring. The variance and

the mean of MTF are related to deconvolution noise, and

the peakiness of the spectrum has an ill effect on deblur-

ring since it destroys the spatial frequencies in the blurred

image.

The MTF of the varying exposure method [2] is the joint

spectrum of different durations of rectangle functions in the

time domain (Fig. 2(b)). The upper bound of the joint MTF

Xvary of p varying exposures is derived as

|Xvary(ω)| =p∑

i=1

|F(Π(li))| =p∑

i=1

| sinli2 ω

sin ω2

| ≤ | p

sin ω2

|, (7)

0 0.5 1 1.5 2 2.5 3-15

-10

-5

0

5

Log

L1 = 30

L2 = 35

L3 = 42

|Xvary()|

Optimal Bound

0 0.5 1 1.5 2 2.5 3-15

-10

-5

0

5

Log

Sequence1Sequence2Sequence3|Xrand()|

Optimal Bound

0 0.5 1 1.5 2 2.5 3-15

-10

-5

0

5

Log

Sequence1Sequence2Sequence3|Xcomp()|

Optimal Bound

(a) Joint MTF of [2] (b) Joint MTF of [27] (c) Proposed

Figure 2. Joint MTF of each method and its theoretical upper

bound. (a) Varying exposure video [2]. (b) Different fluttering

patterns by random sample search [27]. (c) The proposed method

by complementary set of fluttering patterns.

where Π(l) denotes a rectangle function of length l, and ωis a circular frequency at [−π, π].

To compute the upper bound for the coded exposure

video, let Φ denote a binary sequence of +1’s and −1’s.

Parker et al. [26] showed that the sum of all the Fourier

transform components of a complementary set of p se-

quences Φi, i = [1, · · · , p] of length n is at most√pn.

Since a fluttering pattern in the coded exposure imaging is

made of +1’s and 0’s due to its physical nature, the upper

bound of the joint MTF Xcomp of a complementary set is

computed as

|Xcomp(ω)| =1

2

p∑

i=1

|F(Φi +Π(n))| (8)

≤ 1

2

p∑

i=1

|F(Φi)|+1

2

p∑

i=1

|F(Π(n))| ≤ 1

2

√pn+ | p

2 sin ω2

|.

Fig. 2 compares the joint MTF of the varying exposure

video and the coded exposure video with the theoretical up-

per bound in Eq. (7) and Eq. (8). For the varying exposure

video in (a), the exposure lengths of [30, 35, 42] are used as

in [2]. For the coded exposure video, we use three random

sequences from [27] in (b) and our complementary set of

binary sequences in (c) which will be explained in the next

section. As can be seen in the figure, no null frequency is

observed at the MTFs of each coded pattern in the comple-

mentary set. This means that each single frame becomes

invertible as with the conventional coded exposure imaging

[27, 14]. In (b), the peaky spectrums of the random binary

sequences are moderated but the variance of the joint MTF

is still large. The peaky spectrums of each sequence in the

complementary set are well compensated by the joint MTF

(c), and the joint MTF (c) has much flatter and higher MTF

than the joint MTFs of both the varying exposure method

(a) and the set of the random sample sequences (b).

Theoretical upper bounds as well as the actual perfor-

mance measurements of the MTF properties for the varying

exposure video and the coded exposure video are plotted in

Fig. 3. To verify the effectiveness of the complementary

set of fluttering patterns, the random sample search method

in [27] is used to generate the binary sequences for single

3543

Varying Coded

0

2

4

6

8

10

12

14

16

Variance

10lo

g 10

Varying Coded

-12

-10

-8

-6

-4

-2

0Mean

10lo

g 10

Varying Coded-25

-20

-15

-10

-5

0Minimum

10lo

g 10

× # of pattern : 1× # of patterns : 3+ # of patterns : 2+ # of patterns : 3+ # of patterns : 4+ # of patterns : 5− Lower Bound− Lower Bound

× # of pattern : 1× # of patterns : 3+ # of patterns : 2+ # of patterns : 3+ # of patterns : 4+ # of patterns : 5− Upper Bound− Upper Bound

× # of pattern : 1× # of patterns : 3+ # of patterns : 2+ # of patterns : 3+ # of patterns : 4+ # of patterns : 5− Upper Bound− Upper Bound

×: randomly generated fluttering patterns, +: complementary set of fluttering patterns

Figure 3. The theoretical upper bound and the actual performance

measurement of the varying exposure video and the coded expo-

sure video in terms of MTF properties.

image and three images cases. Although the MTFs of both

the coded exposure and the varying exposure do not reach

the lower (variance) and the upper bound (mean and mini-

mum), the coded exposure patterns shows better MTF prop-

erties than both the varying exposure method and the set of

random sequences. Specifically, the complementary set has

a jointly flat spectrum with higher mean and minimum MTF

value which are even better than the theoretical bounds of

the varying exposure method. This shows that the comple-

mentary set preserves spatial frequencies well by compen-

sating frequency losses in each frame. It is worth noting that

while utilizing all the sequences in a complementary set is

ideal in theory, utilizing a partial set of the complementary

set is also effective as shown in Fig. 2 and Fig. 3.

3. Coded Exposure Video

3.1. Hardware Setup

Constructing a hardware system for the coded exposure

video is not trivial. We implemented the coded exposure

video system using a Point Grey Flea3 GigE camera which

supports the multiple exposure pulse-width mode (Trigger

Mode 5) and a ATmega128 micro-controller to generate ex-

ternal trigger pulses (Fig. 4). The micro-controller sends

the binary sequence signal to the camera as external trig-

gers through the serial communication (step 1 in Fig. 4).

The camera finishes taking a photo after c number of peaks

in the sequence (gray area in the figure), which indicates

the end of a sequence (step 2). The camera then trans-

mits the image and the signal to the computer (step 3) and

the computer passes the parameter c of the next sequence

to the camera (step 4). This process is repeated again to

take a new set of images under a new sequence (step 1).

We used this system to capture both the varying exposure

video and the coded exposure video. Each shutter chop

is 1ms long and the frame rate is fixed at 5 frames per

second regardless of the sequence length due to the hard-

ware limitation. The implementation manual and the source

MicrocontrollerMachine Vision

CameraComputer

External Trigger

Ethernet

(4)

(3)(1)

(2)

Serial Communication

Microcontroller

Camera

Computer

Object

Pattern 1: 111010011100 (c=3) Pattern 2: 110001000101 (c=4)

110001000101

Frame 2Pattern 2

200ms

(1) (2)(3)(4) (1) (2)(3)(4) (1)

Frame 1Pattern 1

111010011100

1ms

1110100111

Frame 3Pattern 1

00

Figure 4. Hardware Setup for the Coded Exposure Video.

code of our framework are released in our website https:

//sites.google.com/site/hgjeoncv/complementary sets.

3.2. Sequence Generation

In the coded exposure imaging, a fluttering pattern of a

camera shutter generally consists of a sequence longer than

20 bits, or even longer than 100 bits. Since the length of the

fluttering pattern may vary due to the illumination condition

or the object motion, it is beneficial to have a flexibility in

the length of the pattern. In this subsection, we introduce

a method for generating a complementary set of fluttering

patterns of flexible length.

Our strategy for obtaining the flexibility in the sequence

length is to generate the complementary set by expanding a

small-sized initial set that is known to be a complementary

set. Since the research in the complementary set construc-

tion has a long history, many known complementary sets

exist such as

∆ =

[0 10 0

],

[1110110111100010

],

000010100100001001111101101000100011001110010111

,

where ∆ denotes a complementary set in a matrix form. In

∆, each row vector represents one sequence and the set of

all row vectors is a complementary set.

From an initial complementary set ∆(p,n) which con-

sists of p sequences of length n, we can iteratively gener-

ate larger complementary sets [38]. With a complementary

set ∆, a new complementary set ∆1 with larger length se-

quence is obtained by

∆1 =

[∆ ∆ ∆ ∆

∆ ∆ ∆ ∆

], (9)

where ∆ denotes the matrix with all the elements δs in ∆

flipped. After applying the expansion t times, we obtain a

complementary matrix ∆t ∈ R2tp×4tn, which contains 2tp

sequences of length 4tn.

Another option for generating variable length sequences

is to divide ∆ into two matrices with the same length as

∆ =[∆L ∆R

]. (10)

3544

In this case, both matrices ∆L and ∆R become comple-

mentary sets [10].

With the two matrix operations in Eq. (9) and Eq. (10),

we can generate a complementary set, whose size is 2tp ×22tn or 2tp × 22t−1n. Since there are many well-known

initial complementary sets with various sizes, we can gen-

erate complementary sets with huge flexibility of sequence

length using the two methods.

In the video deblurring scenario, the required number of

sequences (or images) are usually limited to 2∼5 because it

is enough to compensate for the frequency losses and tak-

ing many pictures may create additional problems such as

the alignment and the field-of-view issue. Therefore, we

first generate a complementary set that fits with the required

sequence length, and then select the required number of se-

quences among many candidate sequences in the set.

As for the criteria for selecting sequences from the avail-

able set of sequences, we consider the number of open

chops. In general, the generated sequences have similar

number of open chops, e.g. n/2, however it could be

slightly different especially for short length sequences. In

this case, selecting sequences with equal number of open

chops can be an important criterion to avoid flickering be-

tween frames.

3.3. A Blurred Object Extraction and Deblurring

One practical issue with the coded exposure imaging

is to extract an accurate matte image for the moving ob-

ject deblurring. It is challenging because a blur profile be-

comes locally non-smooth due to the exposure fluttering.

Agrawal and Xu [1] proposed the fluttering pattern design

rules that minimize the transitions and maximize continu-

ous open chops, and showed that both criteria of PSF es-

timation [8] and invertibility can be achieved. In [36], a

blurred object is extracted from a static background with

user strokes in order to estimate motion paths and magni-

tudes. McCloskey et al. [25] presented a PSF estimation

algorithm for coded exposure assuming that the image only

contains a motion blurred object. In this paper, we deal with

this matting issue by jointly estimating the PSF, object mat-

ting, and the multi-image deblurring. We assume that the

images are captured from a static camera and a moving ob-

ject is blurred by a constant velocity 1-D motion.

Initialization To accurately extract the blurred object

(Fig. 5(a)), we first capture background images and model

each pixel of the background using a Gaussian mixture

model (GMM). When an object passes over the background,

we estimate an initial foreground layer by computing the

Mahalanobis distance between pixels of each image and the

background model. Since the estimated layers are some-

what noisy, we refine the layers by applying morphological

operations and make trimaps. Using the trimaps (Fig. 5(b)),

we extract the blurred object at each image via the closed

45 50 55 60 65 70 75 80-20

0

20

40

60

80PSF3

Blur Size

Mat

chin

g S

core

45 50 55 60 65 70 75 80-20

0

20

40

60

80

100

Blur Size

Mat

chin

g S

core

PSF1

45 50 55 60 65 70 75 80-20

-10

0

10

20

30

40

50

60

70

Blur Size

Mat

chin

g S

core

PSF2

(a) Inputs (b) Trimaps (c) Deblurring and PSFs

(d) Aligned image (e) Initial deblurring (f) Refined deblurring

Figure 5. Multi-image Deblurring Procedure

form matting [18].

After the object matting, we estimate the PSF of each

image based on the method in [25], which can handle the

cases of constant velocity, constant acceleration, and har-

monic motion. Specifically, we first perform the radon

transform and choose the direction with the maximal vari-

ance as a blur direction. This is because high spatial fre-

quencies of a blurred image is collapsed according to a blur

direction. Then, we compute matching scores between the

power spectral density of the blurred image and the MTF

of the fluttering pattern for various blur size. We deter-

mine the size of blur by choosing the highest matching

score. With this method, we estimate the blur kernel of

each coded blurred image independently (Fig. 5(c)). This

is useful because we do not suffer from the violation of the

constant motion assumption between frames that often oc-

curs in practice.

With the estimated PSFs, the captured images are de-

blurred independently. Then, we align all images by affine

matrix of the deblurred images using SIFT feature match-

ing [22], and merge all the captured images along with the

alpha maps (Fig. 5(d)).

Iterative Refinement After the initialization, we itera-

tively optimize between a latent image and the segmenta-

tion masks. Based on the merged image with the PSFs, we

perform a non-blind multi-image deblurring by minimizing

the following energy term:

argminY

m∑

j=1

‖Bj −KjY ‖2 + λd‖∇Y ‖ρ, (11)

where Y is a latent deblurred image, ∇Y is the gradient

of the latent image, Bj is a set of linearly blurred images

captured by a set of PSF matrices Kj . λd is the smoothness

3545

0 5 10 15 20 25 3015

20

25

30

35

Image number in the synthetic dataset

(dB

)

PSNR

# of Images : 2# of Images : 3# of Images : 4# of Images : 5

0 5 10 15 20 25 300.8

0.85

0.9

0.95

1

Image number in the synthetic dataset

SSIM

# of Images : 2# of Images : 3# of Images : 4# of Images : 5

Figure 6. The performance variations of the proposed method ac-

cording to the number of fluttering patterns used.

weight and m is the number of images. We set ρ = 0.8for image deblurring [17] and ρ = 0.5 for the merged alpha

map deblurring α according to [36]. The deblurred alpha

map α is re-blurred to obtain a guidance alpha map α which

is incorporated as a soft constraint in the close form matting

to refine the alpha map α for the moving object [16]:

argminα

αT Lα+ λm(α− α)T D(α− α), (12)

where L is the Laplacian matrix of the closed form matting,

D is a diagonal matrix and λm is the weight for the soft

constraint.

With the refined alpha maps, we optimize the set of affine

matrices H that minimizes the energy function similar to the

stereo matching as follows:

argminH

m−1∑

j=1

{λa min(|Xref −HjXj |, τcolor)

+ (1− λa)min(|∇Xref −∇(HjXj)|, τgrad)},

where X is independently deblurred image and Xref is

the reference view. λa balances the color and the gradi-

ent terms, and τcolor, τgrad are truncation values to account

for outliers correspondences.

As shown in Fig. 5(f), our algorithm shows a promising

result of moving object deblurring in complex background.

The refinement is iterated 2 or 3 times for the final result

and takes 5 minutes for an image with 800×600 resolu-

tion in our MATLAB implementation. We empirically set

{λd, λm, λa, τcolor, τgrad} = {0.01, 0.1, 0.5, 0.3, 0.5}.

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

20

25

30

PSNR

Image number in the synthetic dataset

(dB

)

Single pattern3 same patterns3 different patternsVarying exposureProposed

1 3 5 7 9 11 13 15 17 19 21 23 25 27 290.75

0.8

0.85

0.9

0.95

1SSIM

Image number in the synthetic dataset

Figure 7. The quantitative comparisons of different methods on

image deblurring.

4. Experiments

To verify the effectiveness of the proposed method, we

perform both quantitative and qualitative comparisons with

other computational imaging methods; the coded exposure

imaging [27] and the varying exposure video [2]. For the

coded exposure method [27], we use a fluttering pattern of

length 48 generated by the author’s code1. The exposure se-

quences [30, 35, 42ms] stated in [2] is used for the varying

exposure method. The fluttering patterns of length 48 of the

proposed method is generated by applying Eq. (9) once to

the initial set 2.

4.1. Synthetic Experiments

For quantitative evaluations, we perform synthetic ex-

periments. As the synthetic data, we use 29 images down-

loaded from Kodak Lossless True Color Image Suite [30].

Image blur is simulated by the 1D filtering with different ex-

posure sequences generated by each method. To simulate a

real photography, we add the intensity dependent Gaussian

noise with the standard deviation σ = 0.01√i where i is the

noise-free intensity of the blurred images in [0, 1] [31]. The

peak signal-to-noise ratio (PSNR) and the gray-scale struc-

tural similarity (SSIM) [41] are used as the quality metrics.

For fair comparisons, we conduct parameter sweeps for im-

age deblurring and the highest PSNR and SSIM values of

each image/method are reported.

1www.umiacs.umd.edu/˜aagrawal/MotionBlur/SearchBestSeq.zip2The initial set we used in this work is [000010100100; 001001111101;

101000100011; 001110010111]

3546

(a) Coded Exposure [27] (b) Varying Exposure [2] (c) Proposed

Figure 9. Multiple objects deblurring with different velocities and directions. Blur size of the car (close) and the panel (far) (unit: pixel):

(a) 50 and 45. (b) [36 44 39] and [25 31 27]. (c) [46 50 52] and [35 38 44].

(a) A random sequences set (b) Proposed

Figure 8. Comparison of results using a random set and the pro-

posed complementary set when PSFs are stretched (Blur size : 96

pixels).

Fig. 6 reports the averaged PSNR and SSIM of the pro-

posed method according to the number of images used. We

can observe that better performance is achieved with more

images, however the performance gain ratio is reduced as

the number of images increases. The experiment shows that

utilizing three fluttering patterns is a good trade-off between

the performance gain and the burden of multi-image deblur-

ring for the proposed method. Therefore we use three flut-

tering patterns for the remaining experiments.

Quantitative comparisons of different methods are

shown in Fig. 73. For a complete verification, we addition-

ally consider two sets of coded exposure sequences gener-

ated by the random sample search [27]. Each set of three

sequences consists of the same fluttering pattern and three

different patterns, respectively. We include the two set of

sequences as baseline extensions of a single coded expo-

sure to coded exposure video. The proposed method out-

performs the previous methods for all the dataset, with large

margins especially in SSIM. This is because the proposed

method yields high-quality deblurring results while the pre-

vious methods fail to recover textured regions due to the

loss of high spatial frequencies. Fig. 1 shows examples of

the synthetic result. As shown in the figures, our method

outperforms the other methods both qualitatively and quan-

titatively.

3Here, we only report odd numbered dataset results due to limited page

space. All PSNR and SSIM values are reported in supplementary material.

(a) Coded Exposure (Blur size : 80 pixels) [27]

(b) Varying Exposure (Blur size : [60 67 75] pixels) [2]

(c) Proposed (Blur size : [80 80 82] pixels)

Figure 10. Outdoor Experiment

4.2. Real­world Experiments

In Fig. 8, we show an empirical validation of the per-

formance issue discussed in Sec. 2.2. We captured a reso-

lution chart in a carefully controlled environment, and the

chart moved two pixels during one exposure chop. We com-

pare our complementary set with a set of randomly gener-

3547

Worst (SSIM=0.34) Worst (SSIM=0.82) Worst (SSIM=0.98)

(a) Random sets (b) MURA (c) Proposed

Figure 11. Privacy-protecting video surveillance

ated fluttering patterns. As shown in Fig. 8, when PSF is

stretched, the deblurred image captured by the random se-

quence set has noticeable artifacts around edges. On the

other hand, the proposed complementary set preserves de-

tails on the resolution chart well.

Fig. 9 shows the results of deblurring multiple ob-

jects with different velocities. To segment each object

trimap separately, we performed multi-label optimization

via graph-cuts [3]. Then, each object was deblurred and

pasted onto the background independently. In Fig. 9, one

object is highly textured and moving fast, while the other

one is lowly textured and moving slow. The proposed

method shows the best results compared to other deblurring

methods.

We then performed another real-world experiment in

outdoor by capturing a fast moving object as shown

in Fig. 10. The motion direction and the blur kernel is es-

timated automatically in the complex background that has

similar color as the moving car. Once again, our coded ex-

posure video scheme outperforms the other two methods.

4.3. Application to a privacy protection method forvideo surveillance

The privacy protection for video surveillance has be-

come an important issue recently as the video surveillance

has become a commonplace. Various attempts have been

made to address the issue in computer vision [40, 5] and an

interesting study is the use of a coprime blur scheme, which

strategically blurs surveillance videos for privacy protec-

tion [19].

The coprime blur scheme encrypts video streams by ap-

plying two different blur kernels which satisfy coprimality,

and forms a public stream and a private stream. An un-

blurred stream can be recovered by a coprime deblurring

algorithm when both the private and public streams can be

accessed. Since it is very difficult to apply blind deconvolu-

tion with only a public stream, the privacy in video streams

is protected and higher level of security can be achieved by

choosing different blur kernels for each frame. In [19], Li et

al. synthesized the coprime blur kernels from two binary

sequences and presented an efficient deblurring algorithm.

They also highlighted the importance of constructing a bank

of blur kernels with flat spectrum because it directly affects

the security-level and the quality of recovered videos.

Our complementary sets of fluttering patterns can be di-

rectly applied to design the coprime blur kernels in the same

manner4. Because we can generate diverse sets of flut-

tering patterns with various lengths and flat spectrum, our

method is suitable to achieve both high-level security and

high-quality recovery.

We performed an experiment to show the effectiveness

of our framework applied to the coprime blur scheme. We

first generate a pair of coprime blur kernel by using the

modified uniformly redundant array (MURA) [11], and six

pairs of coprime blur kernels by using the random sample

search [27] and our complementary sequences. Then, we

encrypt the video5 by synthetically blurring each frame and

decrypt it according to the coprime method in [19]6.

Both the encryption and the decryption results by the co-

prime method are shown in Fig. 11. The first rows represent

encrypted frames with coprime blur kernels and the second

rows show the deblurred results. The coprime method con-

sistently produces high quality reconstruction results with

our complementary sequences while it suffers from severe

artifacts in some cases when other sequences are used7.

This is because the random sample search fails to gener-

ate good sequences with long length due to the large search

space as discussed in [14] and the MURA includes deep

dips that result in spectral leakage as shown in [27]. On

the other hand, our complementary sequences are able to

produce good sequence pairs with various length and sets.

5. Conclusion

In this paper, we introduced a novel coded exposure

video framework for multi-image deblurring. The proposed

method essentially combines the coded exposure imaging

and the varying exposure video, taking advantages of the

two methods to yield superior deblurring results. The limi-

tation of the current work is that we only solve for 1D linear

blur of a constant velocity object. Note that many real world

object motions such as a walking person or a moving car

result in 1D motion blur as mentioned in [27]. In addition,

affine blur from slanted blurred scene can be interpreted as

a 1D motion blur problem [27]. In the future, we would like

to extend our method to overcome the current limitations

such as known background and linear motion assumptions

as well as to apply our framework for multi-image super-

resolution and visual odometry for robotics.

4According to [15], two different sequences are generally coprime. To

verify the coprimality of complementary pairs of sequences, we generated

120 complementary pairs of sequences of length 256 and confirmed that

all the pairs satisfy the coprimality by the Euclid’s algorithm [9].5dataset: http://homepages.inf.ed.ac.uk/rbf/CAVIAR/6We used the deblurring code by the author to decrypt the video -

http://fengl.org/publications/7The corresponding video is available on supplementary material.

3548

Acknowledgements. This work was supported by the National

Research Foundation of Korea(NRF) grant funded by the Korea

government(MSIP) (No.2010- 0028680). Hae-Gon Jeon was par-

tially supported by Global PH.D Fellowship Program through the

National Research Foundation of Korea(NRF) funded by the Min-

istry of Education (NRF-2015H1A2A1034617).

References

[1] A. Agrawal and Y. Xu. Coded exposure deblurring: Optimized codes

for PSF estimation and invertibility. In Proceedings of IEEE Confer-

ence on Computer Vision and Pattern Recognition (CVPR), 2009.

[2] A. Agrawal, Y. Xu, and R. Raskar. Invertible motion blur in video.

In Proceedings of ACM SIGGRAPH, 2009.

[3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy mini-

mization via graph cuts. IEEE Transactions on Pattern Analysis and

Machine Intelligence (PAMI), 23(11):1222–1239, 2001.

[4] J.-F. Cai, H. Ji, C. Liu, and Z. Shen. Blind motion deblurring using

multiple images. Journal of Computational Physics, 228(14):5057–

5071, 2009.

[5] A. Chattopadhyay and T. E. Boult. Privacycam: a privacy preserving

camera using uclinux on the blackfin dsp. In Proceedings of IEEE

Conference on Computer Vision and Pattern Recognition (CVPR),

2007.

[6] J. Chen, L. Yuan, C. keung Tang, and L. Quan. Robust dual motion

deblurring. In Proceedings of IEEE Conference on Computer Vision

and Pattern Recognition (CVPR), 2008.

[7] S. Cho, J. Wang, and S. Lee. Video deblurring for hand-held cam-

eras using patch-based synthesis. ACM Transactions on Graphics,

31(4):64:1–64:9, 2012.

[8] S. Dai and Y. Wu. Motion from blur. In Proceedings of IEEE Con-

ference on Computer Vision and Pattern Recognition (CVPR), 2008.

[9] E. K. Donald. The art of computer programming. Sorting and search-

ing, 3:426–458, 1999.

[10] P. Fan, N. Suehiro, N. Kuroyanagi, and X. Deng. Class of binary

sequences with zero correlation zone. In Electronics Letters, vol-

ume 35, pages 777–779, 1999.

[11] S. R. Gottesman and E. Fenimore. New family of binary arrays for

coded aperture imaging. Applied optics, 28(20):4344–4352, 1989.

[12] J. Holloway, A. C. Sankaranarayanan, A. Veeraraghavan, and

S. Tambe. Flutter shutter video camera for compressive sensing of

videos. In Proceedings of IEEE International Conference on Com-

putational Photography (ICCP), 2012.

[13] J. M. Jensen, H. E. Jensen, and T. Høholdt. The merit factor of binary

sequences related to difference sets. IEEE Transactions on Informa-

tion Theory, 37(3):617–626, 1991.

[14] H.-G. Jeon, J.-Y. Lee, Y. Han, S. J. Kim, and I. S. Kweon. Flutter-

ing pattern generation using modified Legendre sequence for coded

exposure imaging. In Proceedings of IEEE International Conference

on Computer Vision (ICCV), 2013.

[15] E. Kaltofen, Z. Yang, and L. Zhi. On probabilistic analysis of ran-

domization in hybrid symbolic-numeric algorithms. In Proceedings

of the international workshop on Symbolic-numeric computation,

2007.

[16] S. Kim, Y.-W. Tai, Y. Bok, H. Kim, and I. S. Kweon. Two-phase

approach for multi-view object extraction. In Proceedings of Inter-

national Conference on Image Processing (ICIP), 2011.

[17] A. Levin, R. Fergus, F. Durand, and W. T. Freeman. Image and depth

from a conventional camera with a coded aperture. ACM Transac-

tions on Graphics, 26(3), 2007.

[18] A. Levin, D. Lischinski, and Y. Weiss. A closed-form solution to

natural image matting. IEEE Transactions on Pattern Analysis and

Machine Intelligence (PAMI), 30(2):228–242, 2008.

[19] F. Li, Z. Li, D. Saunders, and J. Yu. A theory of coprime blurred

pairs. In Proceedings of IEEE International Conference on Computer

Vision (ICCV), 2011.

[20] D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. Nayar. Ef-

ficient space-time sampling with pixel-wise coded exposure for high

speed imaging. IEEE Transactions on Pattern Analysis and Machine

Intelligence (PAMI), 99:1, 2013.

[21] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and

D. J. Brady. Coded aperture compressive temporal imaging. Optics

express, 21(9):10526–10545, 2013.

[22] D. G. Lowe. Distinctive image features from scale-invariant key-

points. International Journal on Computer Vision (IJCV), 60(2):91–

110, 2004.

[23] L. B. Lucy. An iterative technique for the rectification of observed

distributions. Astronomical Journal, 79:745–754, 1974.

[24] S. McCloskey. Velocity-dependent shutter sequences for motion de-

blurring. In Proceedings of European Conference on Computer Vi-

sion (ECCV), pages 309–322, 2010.

[25] S. McCloskey, Y. Ding, and J. Yu. Design and estimation of coded

exposure point spread functions. IEEE Transactions on Pattern Anal-

ysis and Machine Intelligence (PAMI), 34(10):2071–2077, 2012.

[26] M. G. Parker, K. G. Paterson, and C. Tellambura. Golay Comple-

mentary Sequences. In Wiley Encyclopedia of Telecommunications,

2003.

[27] R. Raskar, A. Agrawal, and J. Tumblin. Coded exposure photogra-

phy: motion deblurring using fluttered shutter. ACM Transactions on

Graphics, 25(3):795–804, 2006.

[28] D. Reddy, A. Veeraraghavan, and R. Chellappa. P2C2: Pro-

grammable pixel compressive camera for high speed imaging. In

Proceedings of IEEE Conference on Computer Vision and Pattern

Recognition (CVPR), pages 329–336.

[29] W. H. Richardson. Bayesian-based iterative method of image restora-

tion. Journal of the Optical Society of America, 62:55–59, 1972.

[30] R.W.Franzen. Kodak lossless true color image suite, June 1999.

[31] Y. Y. Schechner, S. K. Nayar, and P. N. Belhumeur. Multiplexing

for optimal lighting. IEEE Transactions on Pattern Analysis and

Machine Intelligence (PAMI), 29(8):1339–1354, Aug 2007.

[32] O. Shahar, A. Faktor, and M. Irani. Space-time super-resolution from

a single video. In Proceedings of IEEE Conference on Computer

Vision and Pattern Recognition (CVPR), 2011.

[33] E. Shechtman, Y. Caspi, and M. Irani. Space-time super-resolution.

IEEE Transactions on Pattern Analysis and Machine Intelligence

(PAMI), 27(4):531–545, 2005.

[34] M. Soltanalian, M. M. Naghsh, and P. Stoica. A fast algorithm

for designing complementary sets of sequences. Signal Processing,

93(7):2096–2102, 2013.

[35] P. Spasojevic and C. N. Georghiades. Complementary sequences for

ISI channel estimation. IEEE Transactions on Information Theory,

47(3):1145–1152, 2001.

[36] Y.-W. Tai, N. Kong, S. Lin, and S. Y. Shin. Coded exposure imaging

for projective motion deblurring. In Proceedings of IEEE Conference

on Computer Vision and Pattern Recognition (CVPR), 2010.

[37] Y. Tendero, J.-M. Morel, and B. Rouge. The flutter shutter paradox.

SIAM Journal on Imaging Sciences, 6(2):813–847, 2013.

[38] C.-C. Tseng and C. L. Liu. Complementary sets of sequences. IEEE

Transactions on Information Theory, 18(5):644–652, 1972.

[39] A. Ukil. Low autocorrelation binary sequences: Number theory-

based analysis for minimum energy level, barker codes. Digital Sig-

nal Processing, 20(2):483–495, 2010.

[40] M. Upmanyu, A. M. Namboodiri, K. Srinathan, and C. Jawahar. Effi-

cient privacy preserving video surveillance. In Proceedings of IEEE

International Conference on Computer Vision (ICCV), 2009.

[41] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image

quality assessment: from error visibility to structural similarity. IEEE

Transactions on Image Processing (TIP), 13(4):600–612, 2004.

[42] N. Wiener. Extrapolation, Interpolation, and Smoothing of Station-

ary Time Series. The MIT Press, 1964.

[43] L. Yuan, J. Sun, L. Quan, and H.-Y. Shum. Image deblurring with

blurred/noisy image pairs. ACM Transactions on Graphics, 26(3),

2007.

3549