b.tech thesis

59
Video Stabilization A BACHELOR’S THESIS submitted in partial fulfillment of the requirements for the award of the degree of BACHELOR OF TECHNOLOGY in ELECTRONICS & COMMUNICATION (B.Tech. in ECE) Submitted by: ARSHDEEP SINGH (Enrollment No. - IEC2006034) Under the Guidance of: Dr. Pavan Chakraborty Assistant Professor IIIT-Allahabad

Upload: shivang-kaul

Post on 04-Jul-2015

80 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: B.tech Thesis

Video Stabilization

A BACHELOR’S THESISsubmitted in partial fulfillment

of the requirements for the award of the degree

of

BACHELOR OF TECHNOLOGY

in

ELECTRONICS & COMMUNICATION(B.Tech. in ECE)

Submitted by:

ARSHDEEP SINGH

(Enrollment No. - IEC2006034)

Under the Guidance of:

Dr. Pavan Chakraborty Assistant Professor

IIIT-Allahabad

INDIAN INSTITUTE OF INFORMATION TECHNOLOGY ALLAHABAD – 211 012 (INDIA)

May, 2010

Page 2: B.tech Thesis

CANDIDATE’S DECLARATION

I hereby declare that the work presented in this thesis entitled “Video Stabilization”, submitted

in the partial fulfillment of the degree of Bachelor of Technology (B.Tech), in Electronics &

Communication at Indian Institute of Information Technology, Allahabad, is an authentic record

of my original work carried out under the guidance of Dr. Pavan Chakraborty due

acknowledgements have been made in the text of the thesis to all other material used. This thesis

work was done in full compliance with the requirements and constraints of the prescribed

curriculum.

Place: Allahabad Arshdeep SinghDate: May 19, 2010 IEC2006034

CERTIFICATE

This is to certify that the above statement made by the candidate is correct to the best of my knowledge.

Date: May 19, 2010 Dr. Pavan ChakrabortyPlace: Allahabad Assistant Professor, IIIT-A

Committee on Final Examination for Evaluation of the Thesis

_______________________ _______________________

_______________________ _______________________

Page 3: B.tech Thesis

ACKNOWLEDGEMENTS

I would like to thank my guide for sparing his valuable time and effectively guiding me in the project. He provided me with the general overview of the work I was required to do and was always available for help when I encountered roadblocks in my project. Without Mentor, this project would never have been a success. In the end I would like to thank Larry Page and Sergey Brin for their prestigious product Google without whose extensive search, I could not have even moved an inch. Their constant support and guidance has been inevitable for the timely completion of the project.

Thanks a ton!

Arshdeep Singh

B.Tech. Final Year, IIIT-A

Place: Allahabad Date: May 19, 2010

Page 4: B.tech Thesis

ABSTRACT

The aim of the project was to develop an efficient and robust algorithm for

compensation of camera motion which introduces shakiness and jitters in videos.

Shakiness may be introduced due to shooting video with handheld camera or shooting

from moving vehicle. I implemented the video stabilization process using block

matching algorithm. Given algorithm removes shakiness from YUV raw video and

produces stabilized YUV video. The algorithm is based on extracting motion between

consecutive frames of raw video. Intentional camera motion, such as panning,

translation motion with respect to the scene, is usually smooth with slow variations

from frame to frame (in the time domain). On the other hand, unwanted, parasitic

camera motion involves rapid motion variations over frame to frame. That is to say,

high frequency components in the motion vector variations over time are considered

to be effects of unwanted camera motion. Recovery of the intentional motion

parameter can be achieved by using a low-pass filter. I have used moving average filter

as low pass filter. After extracting noisy component of motion vector, frames can be

stabilized using corrective shifting according to noisy motion component. Algorithm

Developed here have been implemented in C++ and successfully run on shaky video

giving accurate results without incurring significant quality degradation.

Page 5: B.tech Thesis

Table of Contents

1. Introduction..……………………….……………………………………………………1

1.1 Currently existing technologies…………………………………………………..2

1.2 Analysis of previous research in this area………………………………………...4

1.3 Problem definition and scope.…………………………………………………….6

1.4 Formulation of the present problem………………………………………………7

1.5 Organization of the thesis………………………..………………………………..8

2. Description of Hardware and Software Used………….………………………………9

2.1 Hardware…………………………………………………………….……………9

2.2 Software……………………………………………………………….………….9

3. Theoretical Tools – Analysis and Development………………………………………10

3.1 YUV format……………………………………………………………………..10

3.2 Digital video stabilization algorithms...……….....................................................11

3.3 Block matching algorithm…..……………………………….………....………..12

3.3.1. Motion Estimation……………..……………………………………..12

3.3.2. Motion Correction……………..……………………………………..13

3.3.3. Motion smoothing……………..……………………………………..15

3.3.4. Motion Correction……………..……………………………………..16

4. Development of Software ………………………………………..................................18

4.1 Requirement Specification………………………………………………………18

4.2 Design……………………...……………………………………………………18

4.3 Flowchart of the algorithm……………………..………….……………………20

4.4 Implementation …………..…………………......................................................21

5. Testing and Analysis……………………………………………………….…………..23

5.1 Demo1………….……………………………………………………………….23

6. Conclusions……………………………………………………………………………..27

7. Recommendations and Future Work………………………………………………....28

Appendix - Explanation of the Source Code………………………………………....29

Bibliograph……………………………………………………………………………..38

Page 6: B.tech Thesis

P a g e | 1

1 INTRODUCTION

Taking videos with a hand-held camera or from moving vehicles introduces shaking, which

incontrovertibly reduces video quality. Video stabilization is a process to compensate for camera

motion using image processing techniques. Starting from the root I had to understand what video

stabilization is and what are the applications of stabilizing videos in real world. After

understanding the basics I started with YUV videos which are the videos in raw format. It was

kind of preprocessing the data which is captured by camera for stabilization process before

saving it in standard formats like .mpeg, .avi etc.

In short, Aim of the project is to learn and execute the project in following sequential steps:-

Understanding fundamentals of video stabilization

Learning YUV video Format

Studying the different algorithms for digital video stabilization.

Implement the above algorithm in C++, so working on some compiler for

code development and debugging.

Compare the results with the online available products.

Page 7: B.tech Thesis

P a g e | 2

1.1 Currently existing technologies

Mechanical Compensation:

The sensor which captures the image can be moved in a way as to counteract the motion of the

camera, a technology referred to as mechanical image stabilization. When the camera rotates,

causing angular error, gyroscopes encode information to the actuator that moves the sensor. The

sensor is adjusted to maintain the projection of the image onto the image plane, which is a

function of the focal length of the lens being used; modern cameras acquire information about

focal length from the lens mounted.

Digital Stabilization by means of Image Processing:

There are several different approaches available for digital video stabilization. Some of them are

Differential approaches, some are matching based, some Energy based approaches and some are

Phase based approaches.

Comparison of Mechanical and Digital video stabilization:

2. Digital Stabilization by means of Image Processing

Currently there exist two ways that can estimate the relative motion between frames of aShaky video.

1. Mechanical Compensation

Page 8: B.tech Thesis

P a g e | 3

Most of the cameras are mounted inside gimbals for mechanical stabilization, Yet there still is a

need for electronic stabilization. There are some reasons, why mechanical stabilization does not

means exclusion of the electronic stabilization.

First reason is that image vibration can not be removed completely by means of mechanical

stabilization. The performance of gimbals to compensate for vibration is typically specified by

the minimum angular displacement that can be mechanically stabilized.

A second reason that favors electronic image stabilization is its low cost. High performance

gimbals can provide stabilization as good as 12 micro-radians; however, this kind of optical

system is very expensive.

Third reason, the advanced mechanical stabilization systems introduce certain weight and

volume impact.

Flexibility in control is a further advantage of electronic stabilization. As opposed to mechanical

stabilization, an operator at the observation console, or smart software, can dynamically adjust or

even enable /disable the stabilization process and its parameters.

Page 9: B.tech Thesis

P a g e | 4

1.2 Analysis of previous research in this area

1 A robust and efficient video stabilization algorithm

The research paper [1] represents the video stabilization process using affine transformation.

The motion estimation between two sequential frames, f (x, y, t) and f (x, y, t − 1) is done using

a 6 parameter affine transform.

With relatively few parameters, the affine model captures a sufficiently rich range of motions

such as translation, scale, and rotation. The basic framework described here can easily be

extended to accommodate higher-order polynomial models that capture a richer set of motions

(or even a lower-order model that models the inter-frame motion with only a translation). In

order to estimate the affine parameters, a quadratic error function should be minimized.

Pros

Model can capture the translation, rotation and scaling motion between two frames.

Cons

Have to estimate relatively large number of parameters .So computationally extensive.

Page 10: B.tech Thesis

P a g e | 5

2 Fuzzy-based Motion Estimation for Video Stabilization using SIFT Interest points

In this paper [2] they presented a technique which infers inter-frame motion by tracking SIFT

features through consecutive frames. This algorithm does not depend on the point detector

adopted prior to SIFT descriptor creation: therefore performance have been evaluated against a

wide set of point detection algorithms, in order to investigate how to increase stabilization

quality with an appropriate detector. Feature points are detected and their stability is evaluated

through a combination of geometric error measures and fuzzy logic modeling.

Two different error measures have been adopted to evaluate the quality of a matching:

(a) Angle between the two local motion vectors: this measure performs well with rotational

components.

(b) Euclidean distance between expected and real point: this measure does perform well

since rejects matching that do not agree with the found translational components but may

results inaccurate for border points when a rotation occurs.

Compensation is done using fuzzy model. Fuzzy logic model takes as input the two error

measures mention before and gives as output a single quality index, a real value in the range [0,

1] which represents extent of matching between a pair of points.

Page 11: B.tech Thesis

P a g e | 6

1.3 Problem definition and scope:

Taking videos with a hand-held camera introduces shaking, which incontrovertibly reduces video

quality. Video stabilization is a process, which compensate for camera motion. In this project I

am implementing Video stabilization program in C++ .User is supposed to give as input a shaky

raw video in YUV format and software developed will output processed and stable video

sequence in YUV format.

Scope of the project:

This project can be used in several applications. In this era of technology use of personal cameras

and mobile phones are increasing day by day. The videos shot by these devices contain

significant hand shaking effects. It will be possible to remove these annoying effects from the

videos by means of digital stabilization methods. Video conferencing with the pocket PC will be

one of the most attractive aspects in the upcoming future. But to remove hand shacking effect of

the hand held PC, which cause video image annoyingly obscured; stabilization can play an

important role in video coding and dynamic image processing. Software developed can be

integrated with digital camera to convert raw video to a stabilized video and further processing

on that stabilized video.

Page 12: B.tech Thesis

P a g e | 7

1.4 Formulation of the present problem

I formulated the problem in a sequential manner .I planned my work according to the following

steps.

Fig – 1.1(Flowchart of the processes)

1 Reading of frame from a video sequence

2 Calculating Motion w.r.t. previous frame

3 Forming Globalmotion vector for all frames

4 Extracting Desired motion from over all motion

5 Extracting noisy motion from overall motion

6 Writing stabilized frames by compensating for noisy motion

Page 13: B.tech Thesis

P a g e | 8

1.4 Organization of the thesis:

Organization of thesis proceeds as follow-

Chapter 2 describes about hardware used for grabbing videos and software (converter, video

Player, dev c ++) used for development of the project.

Chapter 3 describes the theory and algorithm on which whole project is based. It discuss about

YUV frame format, Block matching algorithm steps as - motion estimation, motion

smoothing, motion correction.

Chapter 4 describes the complete software development process as- specification, design,

flowchart of algorithm. It also explains the data structures used to handle the input

frames.

Chapter 5 describes the testing and analysis of the software. Frames of input video and

Corresponding stabilized frames are shown as well and analysis is made on results

obtained.

Chapter 6 concludes the project.

Chapter 7 describes the modification possible and future prospective of the project.

Page 14: B.tech Thesis

P a g e | 9

2) Description of Hardware and Software Used

2.1 Hardware

I have used canon digital camera for making the video so that running simulation can

be done on it.

2.2 Software

Different soft wares were used during development of project.

1. Dev -C++

C++ implementation of the software was done using Dev –c++.Software was used for

coding and debugging purpose.

2. YUVTools_3.0 (converter)

This software was used to convert avi, mp4 etc file formats to YUV format. Resolution

of the YUV video can be varied using mentioned software.

3. Elecard YUV Viewer

This software was used as YUV video player.

Page 15: B.tech Thesis

P a g e | 10

3) Theoretical Tools – Analysis and Development

Very first step to start with Video stabilization using Block matching algorithm is to understand

raw video YUV 4:2:0 format.

3.1 YUV Format

YUV Format contains three components to encode a color image or video sequence. The three

components are as follows:

Y’ – Luma component (Black and White Component)

U and V – Chrominance Component ( Color Components)

Y'UV was first invented when engineers wanted color television in a black and white

infrastructure. They needed a signal transmission method that was compatible with Black and

white (B&W) TV while being able to add color. The luma component already existed as the

B&W signal; they added the UV signal to this as a solution.

A single YUV420 frame looks like as shown in figure below.

Fig-3.1

Page 16: B.tech Thesis

P a g e | 11

For each pixel we have different Y values. U and V values are shared amongst 2x2 blocks of

pixels as shown in the figure. In the byte stream first all the Y values are stored in contiguous

locations and then U and in the last V values are stored. In my application since I needed to

estimate the motion between continuous frames I used only Y values and while writing the

compensated frame U and V values were also modified according to estimated motion.

3.2 Digital Video Stabilization Algorithms:

There are several different approaches available for Digital Video Stabilization. Some of them

are Differential Approaches, some are Matching based, some Energy based Approaches and

some are Phase Based Approaches. Mainly used approaches for video stabilization are-

Block Matching Algorithm

Affine Transformation Based Video Stabilization

NR Iterative Approach for Video Stabilization

Page 17: B.tech Thesis

P a g e | 12

3.3 Block Matching Algorithm:

In Block Matching Algorithm we estimate the motion between consecutive frames by matching

the blocks of current frame with the previous one.

Fig-3.1- figure shows the block diagram of our motion stabilization architecture.

After reading in video frame sequences, the motion estimator will compare the difference

between every two consecutive frames, expect the possible motions, and generate motion

vectors. Then, these motion vectors will be sent to the motion smoother, which helps to remove

those unwanted motions. Finally, the motion corrector does adjustments to current frames based

on those smoothed motion vector sequences, and output stabilized frames. The basic assumption

in the algorithm is that I have assumed the motion is translational only i.e. there is no rotation, no

zoom-in, no zoom-out etc.

3.3.1 Motion Estimation:

This is the first step in a three step process to stabilize any video. For motion estimation I divided

a frame into blocks of 16x16 pixels. Each block of current frame is matched with the

corresponding blocks in the previous frame in its proximity of 8 pixels in each direction. To

decide the motion vector of each block, we calculate the Sum of Squared Differences (SSD). The

deviation which gives minimum SSD for a block is taken as the local motion of that block.

Page 18: B.tech Thesis

P a g e | 13

Fig-3.2- Block matching for best match

The local distances for each block are collected in a Local Motion Vector and the mode of that

vector is taken to be the Global Motion between the two consecutive frames.

Fig-3.3

The above figure shows the quiver plot of Local Motion Vectors for a frame. It is apparent from

the above graph that most of the Local Motion Vectors are in the same direction. These Local

Page 19: B.tech Thesis

P a g e | 14

Vectors represents the Global Motion of video sequence. The variation from the desired motion

is depicted as some random values in the graph. These Local Vectors represents the Local

Motions of the objects in that frame. For example in a video the motion of the background can be

considered as Global Motion and the motion of objects can be considered as Local Motion.

The right figure shows the histogram of the Local Motion Vectors in the two

directions-horizontal and vertical. As explained above the horizontal component Global Motion

will be the mode of the all the horizontal components of Local Motion Vectors. Similarly we can

find the vertical component of the Global Motion. These values is stored in Global Motion

Vector. Till now we have encountered two vectors:

Local Motion Vector:

Two components : X and Y

X: For horizontal direction motion

Y: For vertical direction motion

No. of instances of Local Motion Vector is one less than the number of frames

in the video sequence. One for each frame starting from second frame.

Size of one vector is equal to the no. of 16x16 blocks in a frame.

Contains the motion of blocks in current frame w.r.t. previous frame.

Global Motion Vector:

It also contains two components x and y as described above.

Only one vector for the entire video sequence.

Size of Global Motion Vector is one less than the no. of frames in the video

As if there are n frames in the video; we have to store n-1 relative motions.

Contains the overall motion of current frame w.r.t. previous frame.

Page 20: B.tech Thesis

P a g e | 15

3.3.2 Motion Smoother:

The Global Motion Vector contains all the global motions for consecutive frames. The Global

Motion Vector is then accumulated in a new vector New Global Motion Vector.

Newglobalvector [ i ] = globalvector [ i ] + newglobalvector [ i-1 ]

This New Global Motion Vector is then smoothed to remove the shakes and jitters which results into the

Smooth Motion Vector. In our algorithm we took MA (Moving Average) as the smoothening filter with

window size N = 5.

By accumulating the Global Motion Vector we can now find how much is the displacement between any

two frames. We cannot use this vector for motion compensation as all the desired motion is also

accumulated in this process. We don’t have to remove the desired motion; we only have to compensate

the unwanted motion. So we smooth out the New Global Motion Vector to find out the Smooth Motion

Vector which only contains the desired motion as by smoothing out we have removed all the unwanted

random motion.

The above figure shows the original New Global Motion Vector and the Smooth Motion Vector.

It is apparent form the figures that the unwanted random shakes are removed by this

smoothening process leaving only the desired motion in the Smooth Motion Vector.

The new vectors introduced in the smoothening process-

Page 21: B.tech Thesis

P a g e | 16

New Global Motion Vector:

The accumulated version of Global Motion Vector

Size and structure is same as Global Motion Vector

Smooth Motion Vector:

Moving Average version of New Global Motion Vector

Size and structure same as Global Motion Vector

3.3.3 Motion Correction:

Motion corrector is used to perform motion compensation and construct smoothed frames. After

finding out all the desired motion in form of Smooth Motion Vector now we have to find out the

unwanted motion. To find this what I did is I subtracted the Smooth Motion Vector from New

Global Motion Vector. By doing this I removed all the desired motion from the New Global

Motion Vector, leaving behind the unwanted motion. This motion was stored in Difference

Vector.

diffvector[i] = newglobalvecor[i] – smoothvector[i]

After finding out the unwanted motion we compensated this motion while writing the new

stabilized video sequence.

Page 22: B.tech Thesis

P a g e | 17

Fig-3.4

In the above figure I1, I2, I3 are the input frames; GMV1, GMV2 are components of Global

Motion Vector; SMV1, SMV2 are components of Smooth Motion Vector; DV1, DV2 are

components of Difference Vector and O1, O2 are final smoothed stabilized frames. Motion

corrector generates the smoothed frame 2 by applying DV1 on the frame 1, and smoothed frame

3 by applying DV2 on input frame 2 (I2) and so on. The thick lines in the above figure show the

motion correction.

Page 23: B.tech Thesis

P a g e | 18

4) Development of Software:

Software development process went through following sequential steps –

4.1 Requirements specification:

As mentioned, Software is supposed to process already captured raw shaky video in YUV format

and produce a stabilized video in YUV raw format. Software should have user friendly interface.

Video has to be stabilized such that quality degradation is insignificant. Software has to be

flexible in the sense; we can vary different parameters to get better results. Resolution and size of

output video is same as that of input.

4.2 Design:

Software is modularized according to following module-

1. GrabeFrame-

This module grabs the frames from input video sequence and calculates total number of

frames. Y, U, V pixel values are stored separately in some data structures.

2. Get motion-

This module is responsible for comparing two consecutive frames and calculates the local

motion vector as well as global motion vector.

3. Calmotion-

This module is actually implementing block matching algorithm and compares the

current frame with previous one. Entire frame is divided into 16 x 16 blocks. Each block

is compared with its corresponding block in previous frame in proximity of 16 pixels.

Page 24: B.tech Thesis

P a g e | 19

SSD (some of squares of difference) is calculated. A block in current frame matches to

other in previous frame if SSD is Minimum. So for each block minimum SSD is

calculated and a local motion vector of the size equal to no. of 16 x 16 blocks is created.

4. Mode-

This module implements the mode function. Mode of the localmotion vector is calculated

and stored as a globalvector element.

5. MovingAvg-

This module works as a low-pass filter. Input to this module is accumulated version of

the globalvector, newglobalvector. This vector contains shaky motion due to camera

movement as well as desired motion. Shaky motion is high pass component in

newglobalvector. This is removed due to low pass filtering. Hence we have output of this

module as smoothvector. Subtracting the smoothvector from newglobalvector gives the

vector whose elements give information about noisy component only. This vector we call

diffvector.

6. Write-

Write module is responsible for writing back the stabilized frames to the output file. Very

First frame of input is written as same.ith element of diffvector corresponds to the shifting

of (i+1)th frame with respect to ith frame due to camera vibration only. Hence ,when

Writing back i+1th frame, it is shifted according to diffvector [i]. Edges are filled with

black color.

Page 25: B.tech Thesis

P a g e | 20

4.3 Flowchart:

yes

no

no

Yes

No

yes

Grab a frame from input

Fist frame?

Grab another frame from input

Calculate the local motion for each block of current frame w.r.t previous frame

Create a localmotion vector for each frame

Take the mode of local motion vector and store as an element in global motion vector

All frames written?

Globalvector

Accumulated version NewGlobalvector

Moving average filter window size=5

START

smoothvector

Diffvector = NewGlobalvector - smoothvector

Grab first frame from input and write it output

Grab next frame and write it compensating with corresponding Diffvector element

All frames grabbed?

STOP

Page 26: B.tech Thesis

P a g e | 21

4.4 Implementation

Algorithm was implemented with C++.First task was to read pixel values from input YUV video and store them in appropriate data structure. To store a complete frame following class was defined-

class Myframe{

public:

unsigned char *Ybuff,*Ubuff,*Vbuff;

Myframe(bool Cap_UV);

};

Myframe::Myframe(bool Cap_UV){

Ybuff=(unsigned char*)malloc(sizeof(char)*Ybuff_size);

if (Cap_UV){

Ubuff=(unsigned char*)malloc(sizeof(char)*Ubuff_size);

Vbuff=(unsigned char*)malloc(sizeof(char)*Ubuff_size);

}

}

Ybuff is a pointer to a contiguous memory block which is used to keep Y pixel values.

Ubuff is a pointer to contiguous memory block which is used to keep U pixel values.

Vbuff is a pointer to a contiguous memory block which is used to keep V pixel values.

When calculating the local motion vectors and global motion vectors, we use only Y values and

V values are required only when writing back the stabilized frames. When bool variable

Cap_UV is 0, declaring object of class Myframe only allocate space for Y values. When 1,

allocates memory for all three Y,U. and V values.

Elements of localmotionvector , globalmotionvector or any other vector are pair (x , y),where x

denotes the horizontal component and y is vertical component, we use a simple data structure,

having two integer variables-

Page 27: B.tech Thesis

P a g e | 22

struct vector{

int h;

int w;

}

Any variable of struct vector type can hold indices of shifting of a frame with respect to another.

Page 28: B.tech Thesis

P a g e | 23

5) Testing and Analysis

The performance of my approach gave good results for the quality. First of all the code was

tested on simulated videos. The results observed were of super fine quality as even a small jitter

was removed with very high accuracy. After that some videos were shot using a canon digital

camera inside the office environment. Very heavy shakes and vibrations were introduced in the

video while shooting. When the code was tested on these videos, the results observed in the

beginning were not satisfactory but after debugging and few modifications, results were

satisfactory even for real videos.

Demo 1-

This demo shows video stabilization results for a yuv video taken in office environment.Video

was taken with canon digital camera. Video obatained from camera was in mp4 format.this video

was converted to yuv format with resolution 176 x 144 using a converter “yuv_tools”. The

Command line console is shown on next page after running the program on given video.

Page 29: B.tech Thesis

P a g e | 24

Fig-5.1 console window after running code on input video Z.yuv

As it is clear from console, inputs to program are video file name(input),output video name,

width and height of video.

After running program, it shows the total number of frames in the video file and number of

frames for which motion calculation has been skipped. Motion estimation is not done for all the

frames. If a frame has high correlation with its previous frame, than motion calculation can be

skipped for that frame as there is no significant effect of shaking on that frame w.r.t. previous

frame.

Consecutive frames of input video and corresponding stbilized frames are shown in Fig-5.2 and

Fig-5.3 respectively.Frames were captured by using YUV player.

Page 30: B.tech Thesis

P a g e | 25

Fig-5.2- consecutive frames of input video.

Here are corresponding stabilized frames (same frame number)from output video.

Page 31: B.tech Thesis

P a g e | 26

Fig - 5.3- corresponding stabilized frames of output video.

As shown in above frames,we see that output frames are less shaky than those of corresponding

input frames.Frames have been compensated for shaky motion.

There is some black colour at the edges of frames.This is because, When we shift

the frame corresponding to diffvector,to keep the size of frame same ,we have to fill some pixels

there.we fill edges with black pixel values.

Page 32: B.tech Thesis

P a g e | 27

6) Conclusion

I have verified global motion estimation algorithm based on a 3-step estimation model that can

remove hand shacking effect with good performance for translational video motions. I also have

implemented the algorithm with C++.

Video stabilizer developed here can stabilize videos accurately and removes the

shakiness without incurring any significant quality degradation. Results were satisfactory for

simulated as well as real videos taken by handheld camera.

Page 33: B.tech Thesis

P a g e | 28

7) Future work and Recommendations

With the development of video stabilization algorithm for the Pocket PC, we can integrate it with

wireless network to conference people whenever you are. However, limited by the transmission

bandwidth and processing speed of Pocket PC, to achieve the goal of video conferencing, we

want to get much lower bit rate of video sequences. One way we can do is to compress the image

more and adopt the MPEG-7 technology to extract the image we want, i.e. human’s face out of

the background. Since background compared to human’s face is not that desired, so we don’t

have to send its information in every frame.

Otherwise, video stabilization can also be used to electronics, such as camcorders. It’s hard to

eliminate hand-shaking effect; however, we can use this algorithm to remove it. It will be useful

to make the video quality better and let people enjoy multimedia world more.

Video stabilization process is implemented using block matching algorithm, which

requires extensive computation for SSD computation. The above algorithm can be modified by

introducing parallel processing (like CUDA) for computation of motion vectors. It will help in

reducing running time of program.

Page 34: B.tech Thesis

P a g e | 29

Appendix 1

Source code and its explanation

/*****************************************************************************************************************************

8Pixel_MA

In this code, following parameters can be changed

1. WSize-> I have taken window size = 16 and movement of WSize/2

2. N-> window of Moving Avg Filter. We worked with N=5.

3. threshold in CalMotion Function. its value willbe a optimization between eficiency and accuracy.

******************************************************************************************************************************/

#include<iostream>

#include<fstream>

#include<cstring>

#include<cstdio>

#include<ctype.h>

using namespace std;

#define WSize 16 //size of window to be matched with the next frame

#define N 5 //Window size for Moving avarage

int Height=0, Width=0, Framesize=0, Ybuff_size=0, Ubuff_size=0, Step=0;

struct vector *GlobalVector;

Page 35: B.tech Thesis

P a g e | 30

int FrameCount = 1; //Frame which is to be captured next

int TotalFrames = 0; //Total no. of frames in the input video

bool Cap_UV; /*Decides whether to initialize UV Buffer or not. While calculating global motion

we need not to capture U and V buffers*/

int SkipCount =0; //To know how many frames were skipped during Motion Estimation

struct vector{

int h;

int w;

};

class Myframe{

public:

unsigned char *Ybuff,*Ubuff,*Vbuff;

Myframe(bool Cap_UV);

};

Myframe::Myframe(bool Cap_UV){

Ybuff=(unsigned char*)malloc(sizeof(char)*Ybuff_size);

if (Cap_UV){

Ubuff=(unsigned char*)malloc(sizeof(char)*Ubuff_size);

Vbuff=(unsigned char*)malloc(sizeof(char)*Ubuff_size);

}

Page 36: B.tech Thesis

P a g e | 31

}

void GrabeFrame(FILE* in,Myframe Frame);

void GetMotion(unsigned char* CurrF,unsigned char* PrevF);

struct vector* CalMotion(unsigned char *Curr, int h, int w, unsigned char *Prev, struct vector* LVector);

void MovingAvg(struct vector* Vector, struct vector* SmoothVector);

void Write(FILE* in,FILE* out, struct vector *Vector);

struct vector Mode(struct vector *arr, int Counter);

/************************************************************************************************

MAIN FUNCTION

*************************************************************************************************/

int main(){

char str[20],str1[20];

Cap_UV = false; //True means capture U and V buffers otherwise not

//GETTING INPUT DATA FROM USER

cout << "Enter the input file name:";

cin >> str;

cout<<"Enter the output file name: ";

Page 37: B.tech Thesis

P a g e | 32

cin>>str1;

cout<<"Enter the Width of the video: ";

cin>>Width;

cout<<"Enter the height of the video: ";

cin>>Height;

//INITIALIZATION OF GLOBAL VARIABLES

Step = Width;

Framesize = Height*Width*(1.5);

Ybuff_size = Height*Width;

Ubuff_size = Ybuff_size/4;

//OPENING THE INPUT FILE IN READABLE MODE

FILE *in;

if((in = fopen(str,"rb")) == NULL){

cout << "Can't open file";

exit(0);

}

//OPENING THE OUTPUT FILE IN WRITABLE MODE

FILE *out;

if((out = fopen(str1,"wb")) == NULL){

Page 38: B.tech Thesis

P a g e | 33

cout << "Can't open file";

exit(0);

}

rewind(out);

//FINDING TOTAL NO. OF FRAMES

char ch;

do{

long offset = (FrameCount)*Framesize;

fseek(in,offset,SEEK_SET);

ch=fgetc(in);

FrameCount++;

}while( ch!=EOF );

TotalFrames = FrameCount-1;

cout << "Total no. of frames in the file are "<<TotalFrames <<"\n";

//ASSIGN MEMORY TO GLOBAL VECTOR

GlobalVector = (vector*) malloc ( sizeof(vector)* (TotalFrames-1) );

//REINTIALIZE FRAMEcOUNT TO GRAB FRAMES

FrameCount = 1;

//INITAILIZATION OF TWO FRAMES

Page 39: B.tech Thesis

P a g e | 34

Myframe Frame1 ( Cap_UV );

Myframe Frame2 ( Cap_UV );

GrabeFrame ( in,Frame1);

GrabeFrame ( in,Frame2);

cout<<"\n MOTION ESTIMATION IS IN PROCESS";

do{

/*CHECK FOR LOCAL MOTION BETWEEN TWO FRAMES

IF ODD FRAME WAS CAPTURED THEN FRAMECOUNT WILL BE EVEN AND CAPTURED FRAME WILL BE IN FRAME1

SO FRAME1 WILL BE LATEST FRAME AND FRAME2 WILL BE THE OLDER FRAME

WE WILL DETECT THE MOTION OF FRAME1 W.R.T FRAME2 AND VICE VERSA*/

if (FrameCount%2==0)

GetMotion(Frame1.Ybuff ,Frame2.Ybuff);

else

GetMotion(Frame2.Ybuff ,Frame1.Ybuff);

//cout<<"\n"<<FrameCount-1<<" AND "<<FrameCount-2;

//GRAB NEXT FRAME

if (FrameCount%2==0)

GrabeFrame(in,Frame2);

else

GrabeFrame(in,Frame1);

Page 40: B.tech Thesis

P a g e | 35

}while( (FrameCount-1)!= TotalFrames);

//TO CALCULATE THE RELATIVE MOTION OF LAST FRAME GRABED

if (FrameCount%2==0)

GetMotion(Frame1.Ybuff ,Frame2.Ybuff);

else

GetMotion(Frame2.Ybuff ,Frame1.Ybuff);

cout<<"\nMOTION ESTIMATION HAS BEEN DONE";

cout<<"\n Total no. of Frames for which motion calculation is skipped are "<<SkipCount;

/*TO GET THE GLOBALVECTOR POINTER AT THE INITIAL POSITION

EACH TIME IN GET MOTION FUNCTION WE INCREMENTED GLOBAL MOTION POINTER TO STORE THE NEXT FRAME MOTION*/

for( int i=0; i<TotalFrames-1; i++){ //TOTALFRAMES-1->BECAUSE GLOABAL VECTOE HAS TOTALFRAMES-1 ELEMENTS

GlobalVector--;

}

/*ASSIGNING MEMORY TO DIFFERENT ARRAYS

NewVector contains the accumulated Global Vector

SmoothVector contains the smoothed version of NewVector

DiffVector contains the difference of NewVector and SmoothVector which is then passed to Write Function*/

struct vector* NewVector = (vector*) malloc ( sizeof(vector)* (TotalFrames-1) );

Page 41: B.tech Thesis

P a g e | 36

struct vector* SmoothVector = (vector*) malloc ( sizeof(vector)* (TotalFrames-1) );

struct vector* DiffVector = (vector*) malloc ( sizeof(vector)* (TotalFrames-1) );

for( int i=0; i<TotalFrames-1; i++){ //TOTALFRAMES-1->BECAUSE GLOABAL VECTOE HAS TOTALFRAMES-1 ELEMENTS

if(i!=0){

NewVector[i].h = GlobalVector[i].h + NewVector[i-1].h;

NewVector[i].w = GlobalVector[i].w + NewVector[i-1].w;

}

else{

NewVector[i].h = GlobalVector[i].h ;

NewVector[i].w = GlobalVector[i].w;

}

}

//MOVING AVARAGE OF NewVector TO FIND SmoothVector

MovingAvg(NewVector, SmoothVector);

//Finding the DiffVector

for(int i=0; i<TotalFrames-1; i++){

if(i<N/2){

DiffVector[i].h = NewVector[i].h ;

DiffVector[i].w = NewVector[i].w ;

Page 42: B.tech Thesis

P a g e | 37

}

else{

DiffVector[i].h = (NewVector[i].h - SmoothVector[i].h);

DiffVector[i].w = (NewVector[i].w - SmoothVector[i].w);

}

}

//Before Writing into the file setting Cap_UV true,

Cap_UV = true;

//Call to Write

Write(in, out,DiffVector);

//CLOSING THE FILES

fclose(in);

fclose(out);

return(0);

}

Page 43: B.tech Thesis

P a g e | 38

8) Bibliography

[1] Hung-Chang Chang ,Shang-Hong Lai , Kuang-Rong Lu, “A robust and efficient video Stabilization algorithm”, Multimedia and Expo, 2004. ICME '04. 2004 IEEE International Conference on 27-30 June 2004

[2] Battiato S. , Gallo G. , Puglisi G. ,“Fuzzy based Motion Estimation for Video stabilization Using SIFT interest points ”,conference on “Digital Photography” V , SanJose , CA , USA 19 January 2009

[3] P. Anandan , “A Computational Framework and an Algorithm for the Measurement of Visual Motion” , International Journal of Computer Vision ,Volume 2, Number 3 / january, 1989 , pages 283-310

[4] Ondrej, M. ; Frantisek, Z. ; Martin, D. ,“Software video stabilization in a fixed point Arith- metic ”, Applications of Digital Information and Web Technologies, 2008. ICADIWT 2008 First International conference on 4-6 Aug. 2008, Pages 389 – 393

[5] Chao-Ho Chen ;Yi-Li Kuo ;Tsong-Yi Chen; Jie-Ru Chen ; “Real-Time Video Stabilization Based on Motion Compensation” , Innovative Computing, Information and Control (ICICIC), 2009 Fourth International Conference on 7-9 Dec. 2009 , pages 1495 – 1498

[6] www.wikipedia.com last accessed on May 18 , 2010

[7] www.cplusplus.com last accessed on May 18, 2010

[8] www.sunrayimage.com last accessed on May 18, 2010