project1 report (1)

241-401 Computer Engineering Project 1

Term 1/2554

Gesture-Based Control of Software

Mr. Boonyasak Natetawesak, 5110110316

Supervisor

Dr. Andrew Davison

Department of Computer Engineering

Faculty of Engineering

Prince of Songkhla University

19th

July, 2554

i


ABSTRACT

Using body to control computer is new technology so it still lacks accuracy and speed. But

many scientists and engineers are doing hard work to develop techniques to read human

movements correctly and fast. It makes gesture-based system grow so fast. That why many

industrials look at it, and develop it for industry.

Testing of gesture system in this project will use a little game. That game is not complicate

but can use hand to control instead of mouse.

I will divided project in 2 parts, first is game implementation that can adapt gesture-based

control to play game. The other is image processing field, which use JMF to capture pictures

and JavaCV to process the pictures.

CONTENT

Abstract i

1. Introduction 1

1.1 Introduction 1

1.2 Objectives 1

1.3 Scope 2

2. Literation review 3

2.1 J2SE 3

2.2 JMF 3

2.3 JavaCV 5

2.4 Related Chapter about Gesture 7

2.4.1 Chapter 3 Motion Detection 7

2.4.2 Chapter 5 Blobs Drumming 8

3. Material and methods 9

3.1 Webcam 9

3.2 Netbeans IDE 9

3.3 Notepad++ 10

4. Task 11

4.1 Schedule 11

4.2 Task Complete 12

Reference 14

LIST OF FIGURES

Figure 1 Minority Report screenshot 1

Figure 2 Mgestyk’s application 1

Figure 3 JMF webcam device tested 3

Figure 4 YUV Snapshot test 3

Figure 5 Using JMF libraries to connect with webcam 4

Figure 6 Light test output 6

Figure 7 Koala test output 6

Figure 8 Motion Detector test-rig result 7

Figure 9 Face detect 7

Figure 10 Center of moving 7

Figure 11 HSV Selector 8

Figure 12 Blobs Drumming 8

Figure 13 Ordinary webcam 9

Figure 14 Humanoid shape webcam 9

Figure 15 Netbeans screenshot 10

Figure 16 Notepad++ screenshot 10

Figure 17 GUI concept design 12

Figure 18 Cursor moving 13

Figure 19 User control cursor 13

Figure 20 Mouse clicking 13

Figure 21 User click cursor 13

1


1. INTRODUCTION

1.1 Introduction

As you know body language is important way to communication with another person. And

gesture-based system is takes those advantages of body language to be controller. But it still

lacks of flexibility or accuracy. Many scientists and engineers are researching and

developing about techniques of read face, body and hands movements correctly and fast.

Even if today gesture-based systems are incomplete system. But many industrials are

interested in gesture-based like MS’s Kinect, Sony’s Candy Eye or Mgestyk are developed

gesture-based control and apply it to their applications.

For example, think about you’re in Sci-fi Holly Woods movies e.g. Minority Report that you

can use your hand to move, select, zoom or close application windows, that is gesture-based

control system but in real, technology still doesn’t reach technology in the movies.

Figure 1 Minority Report screenshot Figure 2 Mgestyk’s application

1.2 Objectives

In this project have 2 main parts, one is about gesture which is heart of project. That I must

implement. And the other is the game to test my gesture working. The game is matching

card game that player can select card by move cursor with their hand and pick it.

2


1. Design aims

This project divide to two parts; the game and the gesture. The game will use ordinary Java

and I’ll design the GUI first in this project. And the gesture I’ll plan about how to use gesture

in the game.

2. Implementation aims

I’ll design the GUI and coding it without gesture first. And when I’d implemented gesture,

I’ll adapt it to the game. Gesture will use JMF to snap pictures and JavaCV to process them.

3. Learning aims

Doing this project, I think I’ll learn about programing in field of image processing. I’ve to

learn how to use JMF to connect with camera, JavaCV to process image data, and combined

it together to control game.

1.3 Scope

Program will capture using JMF libraries and process by JavaCV libraries. This things will

make users control computer by waving a hand like “Minority Report” movie.

This term I’ve designed the GUI of the game. And study about how image processing work

with example code I’ve got from my advisor.

For the game I’ll implement without gesture first. And adapt gesture in it.

3


2. LITERATURE REVIEW

2.1 J2SE

Java 2 Standard Edition is an object-oriented language. [3], [4] Developed at Sun

Microsystems. Java platform derived much of its syntax from C/C++ but simpler and fewer

low-level facilities. Compile to bytecode and run on JVM that not depend to any computer

architectures that make java can run anywhere without recoding.

Because this is Java project so I’m using java for coding in this project, the version I’ve used

is JDK 1.6.0_23. Because it’s pure object-oriented programming and less complicate than

C/C++ so it’s easy to code in object style applications, even if their lack of image processing

part in Java but I can use JavaCV for wrapping Java code with OpenCV instead.

2.2 JMF

Java Media Framework (JMF) [5], [6], [12] is an API that enables audio, video and other

time-based media to be add to applications or applets which built on Java. JMF also supports

development of cross-platform multimedia applications. The version will be used is JMF

2.1.1e which is stable version on window.

1. JMF Studio

JMF Studio is application to capture a picture from webcam.

2. JMF Registry

JMF Registry is preference for JMF Studio. It can setting a programs, devices, plugins etc.

Figure 3 JMF webcam device tested Figure 4 YUV Snapshot test

4


Example code

The full code is in Appendix part.

First ShowJPics create frame and set container to contains picture from webcam, then create

JPicsPanel object and add to container, and check for event come from close button, if it can

receive an event then exit. Fix size and set for visibility. After that create a ShowJPics object

in main method.

pp = new JPicsPanel(); // the sequence of snaps appear here

c.add( pp, "Center");

addWindowListener( new WindowAdapter() {

public void windowClosing(WindowEvent e)

{ pp.closeDown(); // stop snapping pics

System.exit(0);

}

Figure 5 Using JMF libraries to connect with webcam

5


2.3 JavaCV

JavaCV is wrappers to used libraries in field of computer vision. [7] Public licensed under the

GNU General Public License version 2 (GPLv2). JavaCV work with JNA or JNI to map

Java code with native code but in this project I’ve use JNA instead of JNI. But the bad thing

is it still lacks of documentation so it make JavaCV is a hard part to study.

Features of JavaCV: JavaCV has many options for users to use.

JavaCV can use many libraries of computer vision: OpenCV, ARToolkitPlus,

PGRFlyCapture and FFmpeg.

Come with hardware accelerated full-screen image display.

Easy-to-use methods to execute code in parallel on multi-cores.

Geometric and color calibration of cameras and projectors.

Detection and matching of feature points.

In this project I’ll use it in part of image processing, because its features very support image

processing, it’ll makes thing easier.

6


Example code

package testjavacv;

import com.googlecode.javacv.*;

import static com.googlecode.javacv.jna.cxcore.*;

import static com.googlecode.javacv.jna.highgui.*;

public class TestJavaCV {

public static void main(String[] args) {

CanvasFrame frame = new CanvasFrame("Cute_Koala");

// CanvasFrame frame = new CanvasFrame("Lighthouse");

frame.setCanvasSize(320, 320);

IplImage img = cvLoadImage("Koala.jpg");

// IplImage img = cvLoadImage("Koala.jpg");

if(img == null){

System.err.println("Could not load image file.");

}

frame.showImage(img);

}

}

Explain code

Start with declare a variable points to CanvasFrame object to create picture frame then set

frame size. After that use IplImage class keep image which load with cvLoadImage. And

show out via showImage method.

Output

Figure 6 Light test output Figure 7 Koala test output

7


2.4 Related Chapter About Gesture

2.4.1 Chapter 3 Motion Detection

This chapter explains how to detect movement via images that snapped from webcam and

drawn rapidly on panel. And on COG (Center of Gravity) will draw crosshair there. [15] By

using JMF and JavaCV in this chapter.

I got these from Dr. Andrew’s work and test it on my device.

Example

Figure 8 Motion Detector test-rig result

Figure 9 Face detect Figure 10 Center of moving

Motion Detector use new image snapped by webcam to compare with previous image and the

difference will be white while the same view is black via using Threshold.

White point in figure 17 and target in figure 18 using calculation COG (Center of gravity) to

know where is center of movement.

8


2.4.2 Chapter 5 Blobs Drumming

This chapter use simple technique for detect blobs in an image for looking same color. This

application looks for light blue and red blobs. [15] I’ve got from Dr. Andrew too. And I got

it work on my device too.

Example

Figure 11 HSV Selector

Figure 12 Blobs Drumming

This Blob detectors use HSV instead of RGB because of advantage in brightness ignorance.

HSV Selector is for configure the suitable HSV ranges for each card and save to text file.

Drumming is example on how to use blobs in game. By drawing drums and stick them with

detectors. By using configured HSV values from HSV selector make program knows the

card.

9


3. MATERIAL AND METHODS

3.1 Webcam

Webcam is a camera that can be accessed by and displayed on website.[1], [2] Have USB,

Ethernet or Wi-Fi connection. Often use for videoconference, security surveillance and

computer vision.

For this project, I didn’t decide for which one will exactly be used, but for now I’m using my

notebook built-in webcam for test the codes.

Figure 13 Ordinary webcam Figure 14 Humanoid shape webcam

3.2 Netbeans IDE

Netbeans is IDE (Integrated Development Environment) [8] was developed by Sun, available

for Windows, MacOS, Linux and Solaris. Netbeans consists of an open-source IDE, many

plugins, and application platform that make developers easier to create web, desktop and

mobile applications.

Platform which Netbeans supports:

Maven

All of Java platform include JavaScripts

HTML

PHP

Ruby on Rails

Groovy on grails

C/C++

I’ll use Netbeans as main editor because it has many tools, plugins and it easy to create

applications.

10


3.3 Notepad++

Notepad++ is open-source code editor, [9] run on MS Windows environment under GPL

License. It supports several languages.

Features:

Tabbed document interface.

Multiple Clipboards (plugin).

Split screen.

Find and replace.

Supports text encoding formats.

Auto-completion

Syntax highlighting/folding

Macro (recording and execution)

Many plugins

I’ll use Notepad++ as sub editor co with Netbeans, Notepad++ will be used in small part of

code because it less complicate than Netbeans.

Figure 15 Netbeans screenshot Figure 16 Notepad++ screenshot

11


4. TASK

4.1 Schedule

4.1.2 Project 1 1/2011

No. Topic June July August September

1. Study on example code

about motion detection.

2. Study about blob using.

3. Design GUI of game.

4. Writing draft proposal.

5. Implement game

without gesture.

6. Start implement

gesture.

7. Writing final proposal.

4.1.3 Project 2 2/2011

No. Topic November December January February

1. Continue gesture

implementation.

2. Adapt gesture in to

game

3. Improve program.

4. Writing draft proposal.

5. Integrate all part and

get its result.

6. For maintenances.

7. Writing final proposal.

Finished

12


4.2 Task complete

4.2.1 Read papers to get ideas.

Gesture Recognition for a Webcam-Controlled First Person Shooter.

Vision-Based Virtual Control Mechanism via Hand Gesture Recognition.

Recognition-Based Gesture Spotting in Video Games.

Visual Touchpad: A Two-Handed Gestural Input Device.

4.2.3 Design the GUI of the game.

Concept of this project is the game that can play without use mouse or keyboard but gestures.

The game is Matching Cards. There will be even number of cards and player needs to choose

the same pair of cards to win the game. If player lose all life, game over.

1. GUI Design

Figure 17 GUI concept design

2. Details

The blocks is layout in Java which use BoxLayout, BorderLayout and GridLayout.

Name: Show player name.

Life: The chances which player still left.

Time: Time counting.

Card: First time cards will show the front.

Play / Pause Button: Play is starts the game card flip back, time counter start. Pause will

pause the time and can’t select any cards.

13


Game play: When chose a card, the card will flip. If a pair of cards is the same, they will stay

open. If wrong cards flip back and a life disappear. Either last pair of correct cards or lose

all life game will stop.

Adaptation of gesture to the control:

Moving cursor:

Figure 18 Cursor moving Figure 19 User control cursor

Moving cursor by waving hand in front of camera. [11] [13] [14] [15]

Clicking:

Figure 20 Mouse clicking Figure 21 User click cursor

Clicking by clenching a hand. [12] [14]

4.2.2 Test example codes.

Snaps code is about snaps a picture via webcam using JMF.

Motion Detector has 2 codes, one is Motion Detector Test-rig that show movement

that webcam can detect. The other one is Motion Detector, that target user face even

user move it’ll change position by combination of JMF and JavaCV.

Blob Drumming is using blob in drumming game using cards instead of drum sticks.

Using JMF and JavaCV + Blob in this code.

4.2.3 Writing draft proposal.

14


REFERENCE

[1] Wikipedia, http://en.wikipedia.org/wiki/Webcam, access date: 01/28/11.

[2] Wise geek, http://www.wisegeek.com/what-is-a-webcam.htm, access date: 01/28/11.

[3] Oracle, http://www.oracle.com/us/technologies/java/index.html, access date: 01/28/11.

[4] Wise geek, http://www.wisegeek.com/what-is-j2se.htm, access date: 01/28/11.

[5] Wikipedia, http://en.wikipedia.org/wiki/Java_Media_Framework, access date: 01/28/11.

[6] Oracle, http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140239.html,

access date: 01/28/11.

[7] JavaCV, http://code.google.com/p/javacv/, access date: 01/30/11.

[8] Oracle, http://netbeans.org/community/releases/69/, access date: 01/30/11.

[9] Notepadd++ team, http://notepad-plus-plus.org/, access date: 01/30/11.

[10] Fivedots, http://fivedots.coe.psu.ac.th/~ad/jg2/ch09/, access date: 01/30/11.

[11] R. W. Wilson, A Salgian. 2008. “Gesture Recognition for a Webcam-Controlled First

Person Shooter”, Lecture Notes in Computer Science. Vol. 5359 : 889-896.

[12] C. H. Yeh, S. J. F. Jian, J. C. Bai, J. S. Liou, R. N. Yeh, S. C. Wang, P. Y. Sung. 2010.

“Vision-Based Virtual Control Mechanism via Hand Gesture Recognition”, Journal of

Computers. Vol. 21, No. 2.

[13] H. Kang, C. W. Lee, D. Jung. 2004. “Recognition-Based Gesture Spotting in Video

Games”, Pattern Recognition Letters. Vol. 25, No. 15.

[14] S. Malik, J. Laszlo. 2004. “Visual Touchpad: A Two-Handed Gestural Input

Device”,ICMI’04.

[15] Davison, A. 2007. Pro Java 6 3D Game Development, Apress.

http://www.springerlink.com/content/0302-9743/

project1 report (1)

Documents

writing draft

writing final

based gesture

hand gesture

image processing

motion detector

based control

wise geek