a project report on gesture recognition using matlab

8/10/2019 A Project Report on GESTURE RECOGNITION USING MATLAB

1/33

GYAN GANGA COLLEGE OFTECHNOLOGY

A

Project Report on

GESTURE RECOGNITION USING MATLAB

Submitted in partial ful f i l lment for the award of the degreeOf

BACHELOR OF ENGINEERING

I n

ELECTRONICS & COMMUNICATION ENGINEERING

Submitted By:AMEE VISHWAKARMA, APOORVA SRIVASTAVA, DEBOLINA SUR,MOMITA SAHA & MONALISA HAZRA

Enrollment No.:

0208EC101017, 0208EC101028, 0208EC101036, 0208EC101055, 0208EC101056

Guided by:Mr. Rajender Yadav

Designation: Asst. Professor

Deptt.of Electronics & Communication Engineering

Gyan Ganga College of Technology, Jabalpur


2/33

CERTIFICATE

This is to certify that the Minor Project entitled GESTURE

RECOGNITION USING MATLAB submitted by AMEE

VISHWAKARMA, APOORVA SRIVASTAVA, DEBOLINA SUR,

MOM ITA SAHA & MONALI SA HAZRA has been carried out under my

guidance & supervision. The project is approved for submission towards

partial fulfillment as required for the award of degree of BACHELOR OF

ENGINEERING in ELECTRONICS & COMMUNICATION from Gyan

Ganga College of Technology, Jabalpur under RAJIV GANDHI

PROUDYOGIKI VISHWAVIDYALAYA, BHOPAL (M .P.)

SIGNATURE SIGNATURE

Mrs. Papiya Dutta Mr. Rajender Yadav

HEAD OF THE DEPARTMENT GUIDE


3/33

GYAN GANGA COLLEGE OF TECHNOLOGY

CERTIFICATE

This is to certify that the Minor Project report entitledGESTURE

RECGNITION USING MATLAB has been submitted by AMEE

VISHWAKARMA, APOORVA SRIVASTAVA, DEBOLINA SUR,

MOMI TA SAHA & MONALI SA HAZRA for partial fulfillment of the

required for the award of degree of BACHELOR OF ENGINEERING

in ELECTRONI CS & COMMUNI CATION at Gyan Ganga College of

Technology under RAJIV GANDHI PROUDYOGIKI

VISHWAVIDYALAYA, BHOPAL (M.P.)

Internal Examiner External Examiner

Date: Date:


4/33

ACKNOWLEDGEMENT

We would like to express my sincere gratitude to Dr. R.K Ranjan, Principal and Mrs. Papiya Dutta, H.O.D of

Electronics and Communication Department of Gyan Ganga College Of Technology for providing me with an

opportunity to do our minor project on GESTURE RECOGNI TION USING MATLAB.

This project bears on imprint of many people. We sincerely thank our project guide Mr. RAJENDER YADAV

Assistant Professor, Department of Electronics & Communication, Gyan Ganga College of Technology,

Jabalpur whose help, stimulating suggestions and encouragement, helped to coordinate the project especially in

writing this report. We would also like to acknowledge with much appreciation the crucial role of the officials

and other staff members of the institute who rendered their help during the period of project work. Last but not

the least we wish to avail this opportunity to appreciate and give a special thanks to every team member for

their supportive contribution, project enhancing comments and tips that improved the presentation skills, report

writing and brought about clarity in the software work.

Place: Jabalpur

Date:


5/33

TABLE OF CONTENTS

Index Page No.

Chapter 1:- Introduction to Hand Gesture Recognition 1

1.1

Introduction... 1

1.2 Motivation. 1

1.3 Gesture Analysis. 1

Chapter 2:- Objectives & Tools 3

2.1 Introduction 3

2.2 Objectives... 3

2.3 Tools.. 3

Chapter 3:- Literature Review & Algorithm 4

3.1 MATLAB Overview 4

3.2 Literature Review on Gesture Recognition.. 8

3.3 Neural Networks... 9

3.4 Neuron Model... 10

3.5 Perceptron.. 11

3.6 Image Database.. 12

3.7 Orientation Histogram 13

3.8 Operation. 15

3.9 Algorithm 17

Chapter 4:- Results & Discussion 19

APPENDIX I: - Commands 21

APPENDIX II: - Coding 23

References 28


6/33

CHAPTER 1

INTRODUCTION TO HAND GESTURE RECOGNITION

1.1 INTRODUCTION:

This project is to create a method to recognize hand gestures, based on a pattern recognition technique

developed byMcConnell; employing histograms of local orientation. The orientation histogram will be used as

a feature vector for gesture classification and interpolation.

Computer recognition of hand gestures may provide a more natural-computer interface.

Hand gesture recognition is an important area of computer vision and pattern recognition field. Gestures are the

way by which one can communicate non-verbally.

Gesture recognition is a field, in which there is large number of innovations. Gestures can be defined as aphysical action, which is used to convey the information. There are various input output devices for

interacting with the computer, but now days emphasis is given ,how to make human computer interaction

more easy going, and for that purpose hand gesture recognition comes in light. Hand can be used as an input

device, by making its gesture understandable to computer, and for this purpose, this project aims at recognizing

the various hand gestures.

1.2 MOTIVATION:

Hand gesture recognition is done in this project by aiming basic shapes made by hand. Communication in our

daily life is generally vocal, but body language has its own significance, like hand gestures, facial expressions

and sometimes they play an important role in conveying the information. Hand gesture would be an ideal

option for expressing the feelings, or in order to convey something, like representing a number. It has many

areas of application like sign languages are used for various purposes and in case of people who are deaf and

dumb, sign language plays an important role. Gestures are the very first form of communication. So this area

influenced us very much to carry on the further work related to hand gesture recognition.

1.3 GESTURE ANALYSIS:

Gestures are defined as a physical activity that conveys some message, whether it can be facial expressions

body language, hand movements etc. Gesture can be defined as the motion of the body in order to

communicate with others.

1


7/33

Hand gestures can be classified in two categories: static and dynamic. A static gesture is a particular hand

configuration and pose, represented by a single image. A dynamicgesture is a moving gesture, represented by

a sequence of images. We will focus on the recognition of static images.

In gesture analysis, there are mainly three steps to follow

Hand localization

Hand feature extraction

Hand model parameter computation of features

There are two types of classification

Rule Based Classification

Learning Based Classification

We have our project based on LEARNING BASED CLASSIFI CATION. Under this classification, H idden

Markov Models (HMM)approach has been used. An HMM consists of a number of hidden states, each with a

probability of transitioning from itself to another state. The transitioning probabilities are modeled as nth order

Markov processes. An important feature of the topology is that states are allowed to transition themselves.

Alternatively, models based on finite state machines have been used to capture the sequential nature of gestures

by requiring a series of states estimated from visual data to match in order, to a learned model of ordered

states.

The advantage of this representation over HMMs is that it does not require a large set of data in order to train

the models. Finally, temporal extensions to neural networks, (time-delay neural networks), have been used to

learn mappings between training exemplars (2D or 3D features) and gestures. Also, much care must be takenduring the training stage; otherwise the network may over fit on the training gesture set and not generalizes

well to variations of gestures outside the training set.

2


8/33

CHAPTER 2

OBJECTIVES & TOOLS

2.1 INTRODUCTION:

This project on Gesture Recognition Using MATLAB emphasizes on easy & swift communication using

minimum tools which can be easily accessible. The main objectives & tools used for this project are discussed

in the nest sections.

2.2 OBJECTIVES:

The main objectives of our project are:

Response time should be very fast.

Computer vision algorithms should be reliable and work for different people.

The vision-based interfaces will be replacing existing ones, which are often of low cost.

User-friendlier man-machine interaction.

Reduce time consumption.

Reduce error scope.

Easy operation for operator of the system.

2.3 TOOLS:

This project is developed using the following tools:

HARDWARE:

Processor : Intel Core i3

Memory : 3 GB

Processor Speed : 2.30GHz

RAM memory : 2.00 GB

SOFTWARE:

Windows XP/7

MATLAB 7.01

3


9/33

CHAPTER 3

LITERATURE REVIEW & ALGORITHM

3.1 MATLAB Overview

The name MATLAB stands for matrix laboratory. MATLAB is a high-performance language for technica

computing. It integrates computation, visualization, and programming in an easy-to-use environment where

problems and solutions are expressed in familiar mathematical notation. Typical uses include:

Math and computation

Algorithm development

Modeling, simulation, and prototyping

Data analysis, exploration, and visualization

Scientific and engineering graphics

Application development, including Graphical User Interface building

MATLAB is an interactive system whose basic data element is an array that does not require dimensioning

This allows solving many technical computing problems, especially those with matrix and vector formulations,

in a fraction of the time it would take to write a program in a scalar non-interactive language such as C orFORTRAN.

The reason we have decided to use MATLAB for the development of this project is its toolboxes. Toolboxes

allow learning and applying specialized technology. Toolboxes are comprehensive collections of MATLAB

functions (M-files) that extend the MATLAB environment to solve particular classes of problems. It includes

among others image processing and neural networks toolboxes.

In 2004, MATLAB had around one million users across industry and academia. MATLAB users come from

various backgrounds of engineering,science, andeconomics. MATLAB is widely used in academic and

research institutions as well as industrial enterprises.

4
http://en.wikipedia.org/wiki/Sciencehttp://en.wikipedia.org/wiki/Economicshttp://en.wikipedia.org/wiki/Economicshttp://en.wikipedia.org/wiki/Science


10/33

GRAPHICS AND GRAPHICAL USER INTERFACE PROGRAMMING

MATLAB supports developing applications withgraphical user interface features. MATLAB includes

GUIDE (GUI development environment) for graphically designing GUIs. It also has tightly integrated graph-

plotting features. For example the functionplotcan be used to produce a graph from two vectorsxandy. The

code:

x = 0:pi/100:2*pi;

y = sin(x);

plot(x,y)

produces the following figure of thesine function:

Figure 1: sine function

A MATLAB program can produce three-dimensional graphics using the functionssurf,plot3or mesh.

[X,Y] = meshgrid(-10:0.25:10,-10:0.25:10);

f = sinc(sqrt((X/pi).^2+(Y/pi).^2));

mesh(X,Y,f);

axis([-10 10 -10 10 -0.3 1])xlabel('{\bfx}')

ylabel('{\bfy}')

zlabel('{\bfsinc} ({\bfR})')

hidden off

5
http://en.wikipedia.org/wiki/Graphical_user_interfacehttp://en.wikipedia.org/wiki/Sine_wavehttp://en.wikipedia.org/wiki/Sine_wavehttp://en.wikipedia.org/wiki/Sine_wavehttp://en.wikipedia.org/wiki/Sine_wavehttp://en.wikipedia.org/wiki/Graphical_user_interface


11/33

This code produces awireframe 3D plot of the two-dimensional unnormalize

sinc function:

Figure 2: Wireframe3D plot of 2D sinc function

[X,Y] = meshgrid(-10:0.25:10,-10:0.25:10);

f = sinc(sqrt((X/pi).^2+(Y/pi).^2));

surf(X,Y,f);

axis([-10 10 -10 10 -0.3 1])

xlabel('{\bfx}')

ylabel('{\bfy}')

zlabel('{\bfsinc} ({\bfR})')

This code produces a surface3D plot of the two-dimensional unnormalizedsinc function:

6
http://en.wikipedia.org/wiki/Wire_frame_modelhttp://en.wikipedia.org/wiki/Sinc_functionhttp://en.wikipedia.org/wiki/Sinc_functionhttp://en.wikipedia.org/wiki/Sinc_functionhttp://en.wikipedia.org/wiki/Sinc_functionhttp://en.wikipedia.org/wiki/Wire_frame_model


12/33

Figure 3: Surface 3D plot of 2D sinc function

In MATLAB, graphical user interfaces can be programmed with the GUI design environment (GUIDE) tool.

INTERFACING WITH OTHER LANGUAGES:

MATLAB can call functions and subroutines written in theC programming language orFORTRAN. A

wrapper function is created allowing MATLAB data types to be passed and returned. The dynamically

loadable object files created by compiling such functions are termed "MEX-files"(for MATLAB executable)

Libraries written in Perl,Java,ActiveX or.NET can be directly called from MATLAB, and many MATLAB

libraries (for exampleXML orSQL support) are implemented as wrappers around Java or ActiveX libraries

Calling MATLAB from Java is more complicated, but can be done with a MATLAB toolbox which is sold

separately byMathWorks, or using an undocumented mechanism called JMI (Java-to-MATLAB Interface)(which should not be confused with the unrelatedJava Metadata Interface that is also called JMI).

As alternatives to theMuPADbased Symbolic Math Toolbox available from MathWorks, MATLAB can be

connected toMaple orMathematica.

Libraries also exist to import and exportMathML.

7
http://en.wikipedia.org/wiki/C_(programming_language)http://en.wikipedia.org/wiki/Fortranhttp://en.wikipedia.org/wiki/MEX_filehttp://en.wikipedia.org/wiki/Java_(programming_language)http://en.wikipedia.org/wiki/ActiveXhttp://en.wikipedia.org/wiki/.NET_Frameworkhttp://en.wikipedia.org/wiki/XMLhttp://en.wikipedia.org/wiki/SQLhttp://en.wikipedia.org/wiki/MathWorkshttp://en.wikipedia.org/wiki/Java_Metadata_Interfacehttp://en.wikipedia.org/wiki/MuPADhttp://en.wikipedia.org/wiki/Maple_(software)http://en.wikipedia.org/wiki/Mathematicahttp://en.wikipedia.org/wiki/MathMLhttp://en.wikipedia.org/wiki/MathMLhttp://en.wikipedia.org/wiki/Mathematicahttp://en.wikipedia.org/wiki/Maple_(software)http://en.wikipedia.org/wiki/MuPADhttp://en.wikipedia.org/wiki/Java_Metadata_Interfacehttp://en.wikipedia.org/wiki/MathWorkshttp://en.wikipedia.org/wiki/SQLhttp://en.wikipedia.org/wiki/XMLhttp://en.wikipedia.org/wiki/.NET_Frameworkhttp://en.wikipedia.org/wiki/ActiveXhttp://en.wikipedia.org/wiki/Java_(programming_language)http://en.wikipedia.org/wiki/MEX_filehttp://en.wikipedia.org/wiki/Fortranhttp://en.wikipedia.org/wiki/C_(programming_language)


13/33

3.2 Literature Review on Gesture Recognition:

Research on hand gestures can be classified into three categories.

The first category, glove based analysis, employs sensors (mechanical or optical) attached to a glove

that transduce finger flexions into electrical signals for determining the hand posture.

The second category, vision based analysis, is based on the way human beings perceive information

about their surroundings.

The third category, analysis of drawing gestures, usually involves the use of a stylus as an input device

Analysis of drawing gestures can also lead to recognition of written text.

Our project is based on analysis of drawing gestures.

OBJECT RECOGNITION: It is classified into two categories

Figure 4

LARGE OBJECT TRACKING - The large-object-tracking method makes use of a low-cost

detector/processor to quickly calculate moments. This is called the artificial retina chip. This chip combines

image detection with some low-level image processing. The chip can compute various functions useful in the

fast algorithms for interactive graphics applications.

8

OBJECT

RECOGNITION

LARGE OBJECT

TRACKING

SHAPE

RECOGNITION


14/33

SHAPE RECOGNITION - If the hand signals fell in a predetermined set, and the camera views a close-up of

the hand, example-based approach may be used, combined with a simple method top analyze hand signals

called orientation histograms. These example-based applications involve two phases; training and running. In

the training phase, the user shows the system one or more examples of a specific hand shape. The computer

forms and stores the corresponding orientation histograms. In the run phase, the computer compares the

orientation histogram of the current image with each of the stored templates and selects the category of the

closest match, or interpolates between templates, as appropriate. This method should be robust against small

differences in the size of the hand but probably would be sensitive to changes in hand orientation.

3.3 NEURAL NETWORKS:

Neural networks are composed of simple elements operating in parallel. Neural networks are models that are

capable of machine learning and pattern recognition. They are usually presented as systems of interconnected

neurons that can compute values from inputs by feeding information through the network. Commonly neural

networks are adjusted, or trained, so that a particular input leads to a specific target output. There, the network

is adjusted, based on a comparison of the output and the target, until the network output matches the target.

Typically many such input/target pairs are used, in thissupervised learning, to train a network.

Neural networks have been trained to perform complex functions in various fields of application including

pattern recognition, identification, classification, speech and vision and control systems.

Figure 5:Neural Net block diagram

9


15/33

There are two modes of learning: Supervised and unsupervised.

SUPERVISED LEARNING: Supervised learning is based on the system trying to predict outcomes for

known examples and is a commonly used for training method. It compares its predictions to the target answer

and "learns" from its mistakes. The data start as inputs to the input layer neurons. The neurons pass the inputs

along to the next nodes. As inputs are passed along, the weighting, or connection, is applied and when the

inputs reach the next node, the weightings are summed and either intensified or weakened. This continues until

the data reach the output layer where the model predicts an outcome. In a supervised learning system, the

predicted output is compared to the actual output for that case. If the predicted output is equal to the actua

output, no change is made to the weights in the system. But, if the predicted output is higher or lower than the

actual outcome in the data, the error is propagated back through the system and the weights are adjusted

accordingly.

This feeding error backwards through the network is called "back-propagation."

UNSUPERVISED LEARNING: Neural networks which use unsupervised learning are most effective for

describing data rather than predicting it. The neural network is not shown any outputs or answers as part of the

training process--in fact, there is no concept of output fields in this type of system. The advantage of the neural

network for this type of analysis is that it requires no initial assumptions about what constitutes a group or how

many groups are there. The system starts with a clean slate and is not biased about which factors should be

most important.

3.4 NEURON MODEL:

A class of statistical models will be called neuron if they:

Consist of sets of adaptive weights (numerical parameters that are tuned by a learning algorithm

Are capable of approximating non-linear functions of their inputs

10


16/33

Figure 6:Neuron

The scalar input p is transmitted through a connection that multiplies its strength by the scalar weight w, to

form the product wp, again a scalar. Here the weighted input wp is the only argument of the transfer function f,

which produces the scalar output a. The neuron on the right has a scalar bias, b. The bias is much like a weight

except that it has a constant input of 1. The transfer function net input n, again a scalar, is the sum of the

weighted input wp and the bias b. This sum is the argument of the transfer function f. Here f is a transfer

function, typically a step function or a sigmoid function, that takes the argument n and produces the output a. w

and b are both adjustable scalar parameters of the neuron. The central idea of neural networks is that such

parameters can be adjusted so that the network exhibits some desired or interesting behavior.

Thus, the network can be trained to do a particular job by adjusting the weight or bias parameters, or perhaps

the network itself will adjust these parameters to achieve some desired end. All of the neurons in the program

written in MATLAB have a bias.. A

3.5 PERCEPTRON:

The perceptron is a program that learns concepts, i.e. it can learn to respond with True (1) or False (0) for

inputs presented to it, by repeatedly "studying" examples presented to it.

The structure of a single perceptron is very simple. There are two inputs, a bias, and an output. Both the inputsand outputs of a perceptron are binary - that is they can only be 0 or 1. Each of the inputs and the bias is

connected to the main perceptron by a weight. A weight is generally a real number between 0 and 1. When the

input number is fed into the perceptron, it is multiplied by the corresponding weight. After this, the weights are

all summed up and fed through a hard-limiter. Basically, a hard-limiter is a function that defines the threshold

values for 'firing' the perceptron. For example, the limiter could be:

11


17/33

For example- If the sum of the input multiplied by the weights is -2, the limiting function would return 0. Or

if the sum was 3, the function would return 1.

Figure 7:Perceptron block diagram

3.6 IMAGE DATABASE:

The starting point of the project was the creation of a database with all the images that would be used for

training and testing. The image database can have different formats. Images can be either hand drawn, digitized

photographs or a 3D dimensional hand. Photographs were used, as they are the most realistic approach.

Images came from two main sources. Various ASL databases on the Internet and Photographs took with a

digital camera. This meant that they have different sizes, different resolutions and sometimes almost

completely different angles of shooting. Two operations were carried out in all of the images. They were

converted to grayscale and the background was made uniform. The internet databases already had uniform

backgrounds.

Camera had to be processed in Adobe Photoshop. Drawn images can still simulate translational variances with

the help of an editingprogram (e.g.Adobe Photoshop).

The database itself was constantly changing throughout the completion of the project as that would decide the

robustness of the algorithm. Therefore, it had to be done in such way that different situations could be tested

and thresholds above which the algorithm didnt classify correct would be decided.

12


18/33

An example is shown below. In the first row are the training images. In the second, the testing images.

Train image 1 Train image 2 Train image 3

Test Image 1 Test Image 2 Test Image 3

Figure 8

3.7 ORIENTATION HISTOGRAMS:

To have the gestures same regardless of where they occur with the images boarders, position is ignored

altogether, and a histogram is tabulated of how often each orientation element occurred in the image. Clearly,

this throws out information and some distinct images will be confused by their orientation histograms. In

practice, however, one can choose a set of training gestures with substantially different orientation histograms

from each other.

One can calculate the local orientation using image gradients. In this project two 3 tap x and y derivative

filters have been used. The outputs of the x and y derivative operators will be dx and dy. Then the gradient

direction is atan (dx, dy). The edge orientation is used as the only feature that will be presented to the neural

network. The reason for this is that if the edge detector was good enough it would have allowed testing the

network with images from different databases.

13


19/33

Another feature that could have been extracted from the image would be the gradient magnitude using the

formula below

Where,

a= dx

b= dy

This would lead though to testing the algorithm with only similar images. Apart from this the images before

resized should be of approximately the same size. This is the size of the hand itself in the canvas and not the

size of the canvas. Once the image has been processed the output will be a single vector containing a number

of elements equal to the number of bins of the orientation histogram.

Figure shows the orientation histogram calculation for a simple image. Blurring can be used to allow

neighboring orientations to sense each other.

Figure 9: Orientation histogram

14


20/33

3.8 OPERATION:

The program can be divided in 6 steps.

Step1

The first thing for the program to do is to read the image database. A for loop is used to read an entire folder of

images and store them in MATLABs memory. The folder is selected by the user from menus. A menu will

firstly pop-up asking whether to run the algorithm on test or train sets. Then, a second menu will pop-up for the

user to choose which ASL sign he wants to use.

Step2

Resize all the images that were read in Step1 to 150x140 pixels. This size seems the optimal for offering

enough detail while keeping the processing time low.

Step3.

Next thing to do is to find the edges. As mentioned before 2 filters were used.

For the x direction x = [0 -1 1]

For the y direction y=

which is the same as x but transposed and multiplied by1.

Step 4

Divide the two resulting matrices (images) dx and dy element by element and then take the atan ( tan1 ). This

will give the gradient orientation.

Step 5

Then the MATLAB function im2col is called to rearrange the image blocks into columns. This is not a

necessary step but it has to be done to display the orientation histogram. Rose creates an angle histogram

which is a polar plot showing the distribution of values grouped according to their numeric range. Each group

is shown as one bin. Below are some examples. While developing the algorithm those histograms are the

fastest way of getting a good idea how good the detection is done.

15


21/33

Orientation histogram of a_1 Orientation histogram of a_2

Figure 10

Here we can see the original images that generated the histograms above in the same order

Original image a_1 Original image a_2

Figure 11

16


22/33

3.9Algorithm

17


23/33

1. Read Text Files from Disk:

This is the first step where we input image to the system.

2. Determine Number of Neurons & Targets:

For multidimensional arrays RGB image is being considered. To access a sub-image, the syntax is,

subimage=RGB(20:40,50:80,:85);

Foroptimal number determination of neurons, always the highest among the input and output is taken.

So number of neurons determined are 85.

Since we have taken 5 test images in our project, number of targets to be matched are 5.

3. Initialize Pre-Processing Layer:

Pre-processing of image is done for enhancement of image and also for getting results with minimum

error. In the proposed algorithm the image is pre-processed using RGB color model. Three primary

colors red(R), green (G), and blue (B) are used. The main advantage of this color space is its

simplicity..

4. Initialize Learning Layer:

This is a method for initialization of weights of neural networks to reduce the training time.

5. Train Perceptron:

A perceptron learns to distinguish patterns through modifying its weights. In the perceptron, the mostcommon form of learning is by adjusting the weights by the difference between the desired outputand

the actual output.

6. Plot Error:

This step plots the graph of vector with error bars. The error is calculated by subtracting the output A

from target T.

7. Select Test Set:

Test images are selected so that they can be matched with the trained images to obtain the desired

output.

8. Display Output:

Finally, the output is displayed showing the similarity or difference between trained & test images

about their orientation histogram.

18


24/33

CHAPTER 4

RESULTS & DISCUSSION

4.1 RESULT:

4.2 DISCUSSION:

CONCLUSION:

We proposed a simple hand gesture recognition algorithm, followed by various steps like pre-processing

image converted into RGB , so that varying lightening conditions will not cause any problem. Then smudge

elimination is done in order to get the finest image. These pre-processing steps are as important as any other

step. After performing the pre-processing on the image, the second step is to determine the orientation of the

image, only horizontal and vertical orientation is considered here and images with uniform background is

taken.

The strength of this approach includes its simplicity, ease of implementation, and it does not required any

significant amount of training or post processing as rule based learning is used. It provides the higher

recognition rate with minimum computational time.

The weakness of this method is that certain parameters and threshold values are taken experimentally that is it

does not follow any systematic approach for gesture recognition, and many parameters taken in this algorithm

are based on assumption made after testing number of images.

In this system we have only considered the static gesture, but in real time we need to extract the gesture form

the video or moving scene.

To realize the ultimate goal of humans interfacing with machines on their own natural terms gestures are

expressive, meaningful body motions involving physical movements of the fingers, hands, arms, head, face, or

body with the intent of:

1) Conveying meaningful information interacting with the environment2) Gesture recognition is an extensively developed technology available designed to identify human position

action, and manipulation. Gestures are used to facilitate communication with digital applications.

3)Among the various ways of gesture recognition like Hand, Face and Body Gesture Recognition, Hand

Gesture Recognition is efficient technique to o recognize human gestures due to its simple and greater

accuracy features.

19


25/33

FUTURE SCOPE:

The future scope lies in making this algorithm applicable for various orientations of hand gestures, also

different classification scheme can be applied. Gesture recognition could be used in many settings in the future

The algorithm can be improved so that images with non uniform background can also be used, this will

enhance the human computer interaction.

Visually impaired people can make use of hand gestures for human computer interaction like

controlling television, in games and also in gesture to speech conversion.

Georgia Institute of Technology researchers have created the Gesture Panel System to replace

traditional vehicle dashboard controls. Drivers would change, for example, the temperature or sound-

system volume by maneuvering their hand in various ways over a designated area. This could increase

safety by eliminating drivers current need to take their eyes off the road to search for controls.

During the next few years, according to Gartner's Fenn, gesture recognition will probably be used

primarily in niche applications because making mainstream applications work with the technology wil

take more effort than it's worth.

Hand recognition system can be useful in many fields like robotics, computer human interaction and so

make hand gesture recognition offline system for real time will be future work to do.

Support Vector Machine can be modified for reduction of complexity. Reduced complexity provides us

less computation time so we can make system to work real time.

Facial Gesture Recognition Method could be used in vehicles to alert drivers who are about to fall

asleep.

20


26/33

APPENDIX I

COMMANDS:

1. echo on: The commands in a script M-file will not automatically be displayed in the Command Window. To

display the commands along with results we use echo.

2. clc: Clear command window clears all input and output from the command window display, giving a clear

screen.

3. pause: Each time MATLAB reaches a pause statement, it stops executing the M-file until the user presses a

key. Pauses should be placed after important comments, after each graph, and after critical points where your

script generates numerical outputs. Pauses allow viewer to read and understand results.

4. fid: file ID=> An integer file identifier obtained from fopen.

5. fopen: Opens file.

6.fid=fopen(train.txt,rt) : Opens the file with the type of access specified by permission. rt means Read

Text (t-text mode).

7.fscanf (file scan format): Read data from device, and format as text.

8. P1=fscanf(fid,%f,[19,inf]); :

A=fscanf(str,format,sizeA)

Read data from string and converts it according to format. Format is a C language conversion specification.

Conversion specification involves the % character and the conversion characters d, i, o, u, x, X, f, e, E, g, G, c

ands.

Size A can be integer or can have the form [m,n].

[m,n]: mbyn matrix filled in column order, n can beinf,but mcannot.

n: Column vector withnelements.

inf: Column vector with the number of elements in the file (default).

str: Character string.

21


27/33

%f: It denotes floating point numbers. Floating point fields can obtain any of the following (inf, -inf, NaN or -

NaN).

[19]: Neural network interface. (Face segmentation using a Gaussian mixture model of skin or deformable

models of body / face parts)

9. TS1: Test vector.

10.T: Target vector.

11. T=fscanf(fid,%f,[8,inf]); : [8] means 8 bytes (since set A is always double).

12. Determine optimal number of neurons: For multidimensional arrays RGB image is being considered.

subimage=RGB(20:40,50:80,:85);

For optimal number determination of neurons, always the highest among the input and output is taken. So we

have,

S1=85

13. initp: This command initializes weights and biases.

14.Pre-processing: Image pre-processing is the very first step, pre-processing of image is done for

enhancement of image and also for getting results with minimum error. Reduction in the dimensionality of the

input raw data is probably the most important reason for pre-processing.

15. A1=simup(P,W1,b1); : The network takes P as the input data processes and transforms it to A1. A1 is

85x5 matrix.A1 will then be the input to the hidden layer that will train the network. A data sheet of 5 images-

after the first layer pre-processing contains 85x5=425 elements.

16. simup: It simulates perception/preprocessing layer.

17. TP=[1 500]; : Creates vector of 500 equally spaced frequencies.

18. clf reset: Clears current figure and resets the window.

19. figure(gcf): Sets the units of current figure.

gcf: get current figure.

20. epochs: Presentation of the set of training (input/target) vectors to a network and the calculation of new

weights and biases.

21. ploterr (errors): Plots the graph of vector with error bars. The error is calculated by subtracting the output

A from target T.

22. close(W): Closes the waiting bar.

22


28/33

APPENDIX II

CODING:

echo on

clc

pause

clc

%Store the training information in a test file

fid=fopen(train.txt,rt);

P1=fscanf(fid,%f,[19,inf]);

P=P1;

%%Open some text file using code to write and fetch the required information about image

fid=fopen(testA.txt,rt);

TS1=fscanf(fid,%f,[19,inf]);

%(As here we are only testing alphabet A)

Fid=fopen(target8.txt,rt);

T=fscanf(fid,%f,[8,inf]);

%%It has been found that the optimal number of neurons for the hidden layer is 85

S1=85;

S2=5;

%%Now we have to initialize pre-processing layer

[W1,b1]=initp(P,S1);

%%We also have to initialize learning layer

[W2,b2]=initp(S1,T);

Pause

%%Now train the network

23


29/33

A1=simup(P,W1,b1); %First layer is used to preprocess the input vectors

TP=[1:500];

pause

clf reset

figure(gcf)

%%Resize the frame size

Setfsize(600,300);

[W2,b2,epochs,errors]=trainp(W2,b2,A1,T,TP);

pause

clc

ploterr(errors);

pause

M=MENU(Choose a file resolution,Test A);

If M==1

TS=TS1;

else

disp(Wrong Input);

a1=simup(TS,W1,b1);

a2=simup(a1,W2,b2);

echo off

%%Create a Menu

clc

F=MENU(Choose a database set,Test Set,Train Set);

If F==1

K=MENU(Choose a file,Test A);

%%For testing a Datasheet

24


30/33

If K==1

loop=5

for i=1:loop

string=[test\A\num2str(i).tif];

Rimages{i}=imread(string);

end

end

end

end;

%%For training

If F==2

loop=3 %%Set loop to 3 considering all train sets have 3 images

L=MENU(Choose a file,Train A);

If L==1

for i=1:loop

string=[train\A\num2str(i).tif];

Rimages{i}=imread(string);

end

end

end

T{i}=imresize(Timages{i},[150,140]);

x=[0 -1 1]; %x-derivative filter

y=[0 1 -1]; %y-derivative filter

dx{i}=convn(T{i},x,same);

dy{i}=convn(T{i},y,same);

gradient{i}=dy{i}./dx{i};

25


31/33

theta{i}=atan(gradient{i});

cl{i}=im2col(theta{i},[1,1],distinct);

N{i}=(cl{i}*180)/3.14159265359;

C1{i}=(N{i}>0)&(N{i}10.0001)&(N{i}20.0001)&(N{i}30.0001)&(N{i}40.0001)&(N{i}50.0001)&(N{i}60.0001)&(N{i}70.0001)&(N{i}80.0001)&(N{i}90.0001)&(N{i}-89.9)&(N{i}-80.0001)&(N{i}


32/33

S12{i}=sum(C12{i});

C13{i }=(N{i}>-70.0001)&(N{i}-60.0001)&(N{i}-50.0001)&(N{i}-40.0001)&(N{i}-30.0001)&(N{i}-20.0001)&(N{i}-10.0001)&(N{i}


33/33

REFERENCES

[1] Klimis Symeonidis, Final Report-Hand Gesture Recognition using Neural Networks

[2] A Study on Hand Gesture Recognition Technique, Delhi University

[3] Kishan Mehrotra, Elements of Artificial Neural Networks

[4] Christopher M.Bishop, Neural Networks for Pattern Recognition, Clarendon Press.Oxford 1995

[5] M.Hajek, Neural Networks

[6] Thomas Holleczek, Daniel Roggen, MATLAB based GUI for Gesture Recognition with HiddenMarkov Models

a project report on gesture recognition using matlab

Documents