Probabilistic video stabiliProbabilistic video stabilization using Kalman filterization using Kalman filteri

ng and mosaickingng and mosaicking

ABSTRACTABSTRACT

The removal of unwanted, parasitic The removal of unwanted, parasitic vibrations in a video sequence vibrations in a video sequence induced by camera motion is an induced by camera motion is an essential part of video acquisition.essential part of video acquisition.

We present a new image processing We present a new image processing method to remove such vibrations method to remove such vibrations and reconstruct a video sequence and reconstruct a video sequence void of sudden camera movement.void of sudden camera movement.

INTRODUCTION-1INTRODUCTION-1

An approach (An approach (optical stabilizationoptical stabilization) ) consists of implementing an optical consists of implementing an optical system that compensates for system that compensates for unwanted camera motion using unwanted camera motion using motion sensormotion sensor and and active optical active optical systemsystem..

The most powerful, but makes video The most powerful, but makes video cameras significantly cameras significantly expensiveexpensive..

INTRODUCTION-2INTRODUCTION-2

This paper is focus on another This paper is focus on another approach consists in performing approach consists in performing post-processingpost-processing of the video of the video sequence to eliminate unwanted sequence to eliminate unwanted motion in the video (swings and motion in the video (swings and twists) caused by a person holding twists) caused by a person holding the camera or mechanical vibration.the camera or mechanical vibration.

VIDEO STABILIZATION AND VIDEO STABILIZATION AND RECONSTRUCT FRAMEWORK-1RECONSTRUCT FRAMEWORK-1

The overall algorithm consists of the following The overall algorithm consists of the following steps:steps:1.Video sequence stabilization1.Video sequence stabilization

1)1)Estimation of the pair-wise Estimation of the pair-wise transformations between adjacent transformations between adjacent frafra

mes.mes.2)2)Estimation of the intentional motion Estimation of the intentional motion parameters (parameters (Kalman filtering in timeKalman filtering in time).).3)3)Compensation of each frame for Compensation of each frame for unun

wanted motion (wanted motion (frame warpingframe warping).).

VIDEO STABILIZATION AND VIDEO STABILIZATION AND RECONSTRUCT FRAMEWORK-2RECONSTRUCT FRAMEWORK-2

2.Reconstruction of undefined regions u2.Reconstruction of undefined regions using mosaicking:sing mosaicking:

1)1)Estimation of the transformation Estimation of the transformation between distant frame.between distant frame.2)2)Warping distant frames and Warping distant frames and cc

onstructing mosaic for undefined onstructing mosaic for undefined reregions in each frame.gions in each frame.

The block diagram of overall The block diagram of overall algorithmalgorithm

Estimation of the pair-wise Estimation of the pair-wise transformations between adjacent transformations between adjacent

frames-1frames-1 Under an affine transformation, pixel locationsUnder an affine transformation, pixel locations

in frames and in frames and are related by a transformation are related by a transformation given by given by

where andwhere and are pixel coordinates before and after tr are pixel coordinates before and after transformation respectively.ansformation respectively.elements of matrix A describe elements of matrix A describe zoom, rotation zoom, rotation andand doll dolly motiony motion of the camera, and vector of the camera, and vector b b describes describes panninpanningg and and tracking motiontracking motion..

Estimation of the pair-wise Estimation of the pair-wise transformations between adjacent transformations between adjacent

frames-2frames-2 Transform ofTransform of ,aligning frames ,aligning frames and and ,is ,is

estimated by minimizing the following cost funestimated by minimizing the following cost function with respect toction with respect to

where where mm=1=1 and and is the set of all locations in th is the set of all locations in the image plane for which transformed coordinae image plane for which transformed coordinates lie in the limits of the valid image coordinattes lie in the limits of the valid image coordinate.e.

The choice of function The choice of function φφ(x) is crucial for robust(x) is crucial for robustness of the transformation.ness of the transformation.

Estimation of the pair-wise Estimation of the pair-wise transformations between adjacent transformations between adjacent

frames-3frames-3 Here we use an approximation to the Here we use an approximation to the

-norm given by-norm given by

with with ββ=0.01=0.01,insures differentiability of the cost functio,insures differentiability of the cost function near zero; n near zero; p=1p=1 chosen empirically from several test schosen empirically from several test sequences…equences…

In order to avoid local minima of the cost function (2) In order to avoid local minima of the cost function (2) and accelerate the convergence we use a and accelerate the convergence we use a multi-scale imulti-scale implementmplement..

Estimating intentional motion Estimating intentional motion parameters-1parameters-1

The The cumulative transformcumulative transform for frame for frame nn, denoted , denoted byby ,can be obtained as follows: ,can be obtained as follows:

Elements of matrix A describe Elements of matrix A describe zoom, rotation zoom, rotation andand dolly motion dolly motion of the camera, and vector of the camera, and vector b b describes describes panningpanning and and tracking motiontracking motion..

Similarly, we describe the image transform Similarly, we describe the image transform parameters representing parameters representing intentional motionintentional motion in in terms of terms of intentional cumulative transformintentional cumulative transform

Estimating intentional motion Estimating intentional motion parameters-2parameters-2

Optimal estimation ofOptimal estimation of is carry out using a is carry out using a recursivrecursive Kalman filtering algorithme Kalman filtering algorithm..

We treatWe treat as noisy observations of intentional cu as noisy observations of intentional cumulative transform parameters obeying physics-based mulative transform parameters obeying physics-based dynamic model.dynamic model.

The state model for each of the parametersThe state model for each of the parameters depends on real-life expected behavior of these depends on real-life expected behavior of these

parameters.parameters. Two distinct behavior patterns can be identified leadinTwo distinct behavior patterns can be identified leadin

g to different dynamic models for different parameters.g to different dynamic models for different parameters.

Two distinct behavior patterns Two distinct behavior patterns leading to different dynamic leading to different dynamic

models for different parametersmodels for different parameters We introduce velocity variables We introduce velocity variables

for eachfor each ,respectively. ,respectively. It is reasonable to assume the It is reasonable to assume the

independence of dynamic models for each independence of dynamic models for each of the 4 parameters. For example, andof the 4 parameters. For example, and

follow the dynamic model given by follow the dynamic model given by

wherewhere is white Gaussian noise with is white Gaussian noise with variance . variance .

Two distinct behavior patterns Two distinct behavior patterns leading to different dynamic leading to different dynamic

models for different parametersmodels for different parameters The remaining parametersThe remaining parameters and and are are

assumed to be constant in the absence of assumed to be constant in the absence of noise.noise.

The simple dynamic model for The simple dynamic model for andand we have (for )we have (for )

The overall state-space model for the The overall state-space model for the intentional cumulative transform intentional cumulative transform parameters is given byparameters is given by

variance of the noise term is different for variance of the noise term is different for each kind of variable. each kind of variable.

The The observed cumulative transform parametersobserved cumulative transform parameters are treated as noisy observations of the are treated as noisy observations of the

intentional cumulative transform parameters.intentional cumulative transform parameters. The The observation modelobservation model for each parameters is for each parameters is

independent, leading to observation modelindependent, leading to observation model

Observation noise variances describe the Observation noise variances describe the variability of unwanted transformation between variability of unwanted transformation between frames.frames.

Compensation of each frame for Compensation of each frame for unwanted transformation (frame unwanted transformation (frame

warping)warping) Resulting transformResulting transform is given by is given by

wherewhere and and are initial and are initial and transformed coordinates in frame transformed coordinates in frame n.n.

Using (9), a warped frame Using (9), a warped frame is is computed as follows computed as follows

p.s. computing image values at non-p.s. computing image values at non-integer locations in (10) is carried out by integer locations in (10) is carried out by cubic interpolationcubic interpolation..

Reconstruction of undefined regiReconstruction of undefined regions using mosaickingons using mosaicking

After compensating transformation is applied to eaAfter compensating transformation is applied to each frame, undefined regions appear near the edge och frame, undefined regions appear near the edge of each frame.f each frame.

The extent of these regions varies from frame to fraThe extent of these regions varies from frame to frame and presents unacceptable visual artifacts.me and presents unacceptable visual artifacts.

Use frame trimming and magnification or filling by Use frame trimming and magnification or filling by a constant value, lead to severe quality degradatioa constant value, lead to severe quality degradation of the resulting video and limit the range of possin of the resulting video and limit the range of possible correcting transformations.ble correcting transformations.

Here we propose to use mosaicking for each frame iHere we propose to use mosaicking for each frame in order to exploit temporal correlations between frn order to exploit temporal correlations between frames.ames.

Mosaicking illustrationMosaicking illustration

Estimation of the transformation Estimation of the transformation between distant frames-1between distant frames-1

In order to properly align up to In order to properly align up to MM future and past future and past frames with respect to the current warped frame frames with respect to the current warped frame nn, we need to find registration parameters of , we need to find registration parameters of these frames with respect to the current frame.these frames with respect to the current frame.

For a given frame For a given frame nn,, as initial conditions, as initial conditions, we sequentially estimate the global transform we sequentially estimate the global transform parameters between frame parameters between frame nn and and nn ± ± mm, where , where 2≤ 2≤ mm ≤ ≤MM..

For each For each mm, cascaded transforms, cascaded transforms and and are used to initialize the solution for are used to initialize the solution for . .

Estimation of the transformation Estimation of the transformation between distant frames-2between distant frames-2

The coordinate transformation obtained The coordinate transformation obtained using cascaded transforms is given byusing cascaded transforms is given by

For instance, transformation for past frame For instance, transformation for past frame (-(-mm) with respect to frame ) with respect to frame nn can be found can be found by inverting the registration of frame by inverting the registration of frame nn as as a future frame (a future frame (n n - - mm))

Warping distant frames and Warping distant frames and composing mosaic for regions in composing mosaic for regions in

each frame-1each frame-1 Each frame out of 2M neighboring frames is Each frame out of 2M neighboring frames is

aligned with respect to the warped current aligned with respect to the warped current frame given by (10).frame given by (10).

Aligning transform for frame Aligning transform for frame is formed by is formed by cascading inverted registration transform cascading inverted registration transform

with the correcting transform with the correcting transform defined in (9).defined in (9).

The resulting warping transform is given byThe resulting warping transform is given by

Warping distant frames and Warping distant frames and composing mosaic for regions in composing mosaic for regions in

each frame-2each frame-2 And the warped frame is computed as followsAnd the warped frame is computed as follows

For each undefined pixel For each undefined pixel xx in the target in the target frame ,the reconstructed image value is found frame ,the reconstructed image value is found as followsas follows

where the weightswhere the weights are set to the inverse are set to the inverse of the errors of registrationof the errors of registration obtained obtained by minimizing (2).by minimizing (2).

RESULTSRESULTS We test our technique on 3 real-life video We test our technique on 3 real-life video

sequences (which we call sequences (which we call AA, , BB and and CC).). To simplify the task we modify the motion model, To simplify the task we modify the motion model,

first we assume only translational motion between first we assume only translational motion between described by vector described by vector bb. .

Using this model, the cumulative transform Using this model, the cumulative transform parameters are given byparameters are given by

the components of the components of for sequence A are shown for sequence A are shown in Figure3.in Figure3.

Cumulative motion parameters Cumulative motion parameters for sequence Afor sequence A

Assume static camera (performing “total Assume static camera (performing “total motion compensation”), the correcting motion compensation”), the correcting transform becomestransform becomes

the result of applying such compensating the result of applying such compensating transform is illustrated in Figure4.transform is illustrated in Figure4.

In the figure, it can be seen that landmark objects In the figure, it can be seen that landmark objects in the corrected sequence do not move with in the corrected sequence do not move with respect to the frame coordinates, while rotational respect to the frame coordinates, while rotational vibrations remain uncorrected. Shift vibrations remain uncorrected. Shift AA

Full 6-parameter inter-frame Full 6-parameter inter-frame affine motion modelaffine motion model

Full result of Full result of AA, , BB, and , and CC..

CONCLUSIONSCONCLUSIONS Using our technique we obtained Using our technique we obtained

promising preliminary results on random promising preliminary results on random test sequences with complex motion and test sequences with complex motion and severe vibrations.severe vibrations.

We compared our results with one of We compared our results with one of commercial products and showed a commercial products and showed a significant improvement of performance significant improvement of performance for our technique. Compared for our technique. Compared AA and and BB..

Our method of stabilization can be easily Our method of stabilization can be easily adapted to perform additional processing, adapted to perform additional processing, such as sampling rate conversion, static such as sampling rate conversion, static mosaic construction, ego-motion mosaic construction, ego-motion estimation.estimation.