Traffic Police Gesture Recognition using Accelerometers

Ben Wang Department of Automation

Tsinghua University Beijing 100084, China

Email: wben06@mails.tsinghua.edu.cn

Tao Yuan Department of Automation

Tsinghua University Beijing 100084, China

Email: yt@mail.tsinghua.edu.cn

Abstract—When an automatic traffic light system is not used due to too heavy traffic, the traffic would be controlled by traffic police gesture. This paper is about the design of a system so that the traffic lights can follow the traffic police gestures. To simplify the system, a unique mapping between the traffic police gestures and the orientation and movement of hands is defined. The hand motion characters are extracted by fixing a 3-axis accelerometer on the back of each hand. A 2-level hierarchical classifier is used to recognize the gestures. First the gestures are categorized into three groups, according to the movement of each hand. Then a gesture is recognized by comparing it with the predefined templates. This real-time recognition algorithm is implemented by a micro-controller. It is envisaged that this will help drivers.

I. INTRODUCTION When traffic is very heavy, an automatic traffic light

system is not efficient to control traffic, causing traffic jam. In this case, it is necessary to switch off the traffic light and let traffic police guide traffic by gestures. In the case of bad weather or obstruction by other vehicles, however, sometimes it is difficult for all drivers to recognize the gestures. It would be useful if the traffic light can follow the traffic police gestures.

Two methods are considered suitable for gesture recognition. The first one is to use vision sensors like cameras to acquire images, which are analyzed to recognize the gestures [1] [2]. The second one is to place inertial sensor on the traffic police hand and extract the motion characters.

The most advantage of the vision method is that it can recognize gestures without adding any extra hindrance to the police. However, it suffers from poor illumination, e.g. at night or in fog weather.

Due to the advantages of low cost and small size, MEMS accelerometers have been used widely in gesture recognition. As different signals correspond to different gestures, hand gestures can be used to control home appliances, like TV and DVD [3]. An accelerator-glove has been designed by fixing

a 2-axis accelerometer on the back of each finger. By comparing the output signals of the accelerometers and the magnitude of gravity, the bending angles of the fingers can be recognized. It is used further to recognize 26 letters of American Sign Language (ASL) [4].

This paper presents a Chinese traffic police gesture recognition system. By fixing two 3-axis accelerometers on the back of both hands, so the arm movement and hand position, when the arm is steady, can be extracted. By recognizing traffic police gestures and synchronizing the traffic lights with them, it is envisage that this application will give help to vehicle drivers.

II. CHINESE TRAFFIC POLICE GESTURE SYSTEM The Chinese traffic police gesture system is defined and

regulated by Chinese Ministry of Public Security [5]. Essentially, it is an arm gesture system, which means the gestures are mainly expressed by two arms, which are either steady or oscillation. Currently, it consists of 8 gestures:

(1) stop,

(2) move straight,

(3) left turn,

(4) left turn waiting,

(5) right turn,

(6) lane changing,

(7) slow down,

(8) pull over.

In the recognition system described in this paper, a “stand in attention” gesture is added. It is police stand in attention. The reason for this is that traffic police will keep stand in attention when they do not change the current traffic. If a “stand in attention” gesture is recognized, it is unnecessary to change the traffic lights. Fig.1. shows the nine gestures.

1-4244-2581-5/08/$20.00 ©2008 IEEE 1080 IEEE SENSORS 2008 Conference

Fig. 1 Chinese traffic police gestures

In these gestures, “stop”, “move straight”, “left turn”, “left turn waiting” and “right turn” can be followed by standard traffic lights. If these gestures are detected, the traffic lights can be synchronized with them. Other gestures, like “lane changing”, “slow down” and “pull over”, cannot be shown in standard traffic lights. However, it is necessary to recognize all the gestures. First, the gestures, like “lane changing” and “slow down”, can be shown in some emergency traffic lights. Second, in long term, with the coming of intelligent automobile, unmanned or auto-driving vehicles, it is necessary to have them under some special conditions to follow the commands of traffic police. There are two possible solutions to this: the active way or the passive way. The first way is to use cameras on unmanned vehicles to recognize the traffic gestures. The latter way is to use on body sensors to recognize the traffic police gestures. Due to the illumination problems mentioned before, the latter way is a better choice than the first one.

III. THE SYSTEM

A. System description The interconnection of all the components is shown in

Fig. 2. The key components of this system are two 3-axis accelerometers. They can provide signals of arm movement and hand position with respect to the gravitational vector. The signals from the accelerometers are collected by a micro-controller and then analyzed. An NEC78K0547 micro-controller is used in this application [6]. This micro-controller has 128 Kbyte flash, 7 Kbyte RAM, 8 channels of 10-bits ADC, 2 UART (Universal Asynchronous serial receiver and transmitter) ports and 3 SPI (Serial Peripheral

Interface) ports. The micro-controller is connected to a PC via a USB port. In our laboratory, a PC is used to simulate traffic lights.

PC

Microcontroller

Left hand accelerometer

Right hand accelerometer

SPI

USB

Fig. 2 System block diagram

B. Selection and placement of sensors The LIS3LV02DQ is a successful product form ST, Inc.

[7]. It is a low cost, low power, digital output 3-axis accelerometer on a single IC chip, which can provide both dynamic and static acceleration signals. The measurement range is selectable to be ±2g or ±6g. Each axis output is represented by two bytes, in which 12 bits are significant. When the measurement range is set to ±2g, the sensitivity can be 1 mg. The accelerometer communicates with the micro-controller via a SPI port, and no additional A/D converter is needed. In the system, 12 bytes data are read sequentially by the micro-controller, beginning with the right hand accelerometer, X axis followed by the Y axis and then the Z axis, then the left hand accelerometer with the same sequence.

It is crucial to place the sensors in appropriate positions for obtaining unique gesture measurements with the least sensors. In this paper, after a close scrutiny of Chinese traffic police gestures, it is decided to fix two accelerometers on the back of hand, which is shown in Fig. 3. The X and Y axis are parallel to the hand, and the Z axis is perpendicular to the hand and points to the palm.

Fig. 3 Sensors on back of both hands

1081

C. Data collection and visualization In the system, the data processing and gesture recognition

are performed in the micro-controller. During the initial stage, to reach a better effect, it is necessary to modify the parameters for gesture recognition. It would be useful if the signals can be displayed directly on the PC screen. The micro-controller sends the data to the PC by sets, and the signals are displayed on the PC screen using the Matlab software.

IV. SIGNAL PROCESSING

A. Signal preprocessing Because the output of accelerometer may suffer form the

interference of high frequency white noise, it is necessary to process the signal by a low-pass filter [8]. Considering the capacity of the micro-controller, a 3rd order Butterworth filter is used.

B. Oscillation and steady detection If an arm is steady, the hand acceleration is almost

constant. If the arm is oscillating, the hand acceleration verifies much along with the time. In this paper, the resultant

acceleration 2 2 2r x y za a a a= + + is used to represent

the arm movement, where xa , ya and za represent the acceleration outputs of the X axis, Y axis and Z axis. As the standard deviation can be used to describe the variety of a data stream, the following criteria are used for steady and oscillation classification.

Criterion 1: For a sampled time instance M, if ( )S

N thMσ σ> for M = 1A ， 1 1A + ，… 1A N+ , then

1A is the beginning of an oscillation.

Criterion 2: For a sampled time instance M, if ( )S

N thMσ σ< for M = 2A ， 2 1A + ，… 2A N+ ，

then 2A is the end of an oscillation.

Where N is the number of data samples, ( )SN Mσ is a

standard deviation of ra over N samples, and thσ is the threshold of the standard deviation.

In Fig. 4, a typical profile of the magnitude of acceleration ra and its standard deviation are given. Fig. 4 also shows how to find the beginning and end of an oscillation.

0 50 100 150 200 250 300-1000

-500

0

500

1000

Samples

acce

lera

tion(

mg)

0 50 100 150 200 250 3000

200

400

600

Samples

Sta

ndar

d de

viat

ion(

mg

)

Threshold

2

M

A1 A2

Figure 4. A typical process containing oscillation and steady state is

separated

V. GESTURE RECOGNITION A two stage classification is used for gesture recognition.

First, according to whether the arm is oscillating or steady, the dispatcher divides the gestures into three categories. After that, the recognizer I and II are used to recognize the gesture. Fig. 5 shows how the gestures are classified and recognized.

Fig. 5 Classifier’s Hierarchy, each number representing a numbered

gesture in Fig 1 The recognizer I and II are introduced to demonstrate

how they work in the left hand steady and right hand oscillating category.

First, in recognizer I, the output of left hand accelerometer is used to make judgment. Several “if-then” statements are implemented.

1082

If((X axis is vertical-up)&&(Y axis is horizontal)&& (Z axis is horizontal))

Then gesture is “move straight”

If((X axis is horizontal)&&(Y axis is vertical-up)&& (Z axis is horizontal))

Then gesture is “right turn”

If((X axis is horizontal)&&(Y axis is vertical-down)&& (Z axis is horizontal))

Then gesture is “lane changing” or “slow down”

Else gesture is “unknown”.

Because the human gesture is not very precise, a fuzzy definition is used. In our experiment, an angle of -20° to 20° is defined as the horizontal region, 60° to 90° as vertical up and -115° to -75° as vertical down.

Because the gesture “lane changing” and “slow down” have the same steady left hand position, they have to be recognized in recognizer II. As can be seen in Fig. 1, in both gestures the right hand is in oscillation movement. However, the hand oscillates horizontally in the gesture “lane changing” and vertically in “slow down”. The mid value of Z axis is used to distinguish the two gestures. Fig. 6 shows how the two gestures are separated.

0 1 2 3 4 5-2000

-1500

-1000

-500

0

500

1000

1500

2000

time(s)

acce

lera

tion(

mg)

red, Z axis of "lane changing"blue, Z axis of "slow down"

0 1 2 3 4 5-2000

-1500

-1000

-500

0

500

1000

1500

2000

time(s)

mid

val

ue o

f ac

cele

ratio

n(m

g)

red, mid value of "lane changing"blue, mid value of "slow down"

VI. CONCLUSION A novel traffic police gesture recognition system is

presented in this paper. Two 3-axis accelerometers fixed on the back of hand are used as the sensing devices to detect the arm movement and hand position. An algorithm run in an MCU is used to realize the gestures in real time. The software in the PC is used to simulate the traffic lights.

The system has achieved a high recognition rate even when the gestures are not very standard. The main advantage of the on-body system compared to a vision based system is that it will not suffer form occlusion or illumination problems. Future work is to use a wireless module to substitute the USB port. Two nRF2401 wireless modules from NORDIA Company are under test. In addition, energy consumption is always an important concern in a battery-operated system. A “movement wakeup” strategy is being considered in the system.

The gesture recognition approach introduced in this paper can be easily transferred to traffic police gestures in other countries, such as India traffic police gestures.

ACKNOLEDGEMENT The authors would like to thank Engineer Shengli Yang

at NEC-Tsinghua Embedded Research and Education Center for his help in hardware establishment, also, Professor Wuqiang Yang at The University of Manchester is thanked for his help in English.

VII. REFERENCES [1] Yin X M and Xie, M, “Finger identification and hand posture

recognition for human-robot interaction,” Image and Vision Computing, 25 (8), pp. 1291-1300, 2007.

[2] Prieto Abraham, Bellas F, Duro R.J, Lopez-pena F, “An adaptive visual gesture based interface for human machine interaction in intelligent workspaces,” Proc. of IEEE International Conference on Virtual Environments, 2006, pp 43-48

[3] Kallio S, Kela J, Korpipaa P and Mantyjarvi J, “User independend gesture interaction for small handheld devices,” International Journal of Pattern Recognition and Artifical Intelligence, 20 (4), pp. 505-524, 2006.

[4] Hernandez-Rebollar J L, Lindeman R W, and Kyriakopoulos N, “A multi-class pattern recognition system for practical finger spelling translation,” Proc. of 4th IEEE International Conference on Multimodal Interfaces, Oct. 2002, pp 185-190.

[5] Chinese Ministry of Public Security website: www.maps.gov.cn [6] NEC Company website: www.necel.com [7] ST Company website: www.st.com [8] I.J. Jang, W.B. Park, “ Signal processing of the accelerometer for

gesture awareness on handheld devices,” Proc. of IEEE International Workshop on Robot and Human Interactive Communication. Oct. 2003, pp 139-144.

Fig. 6 “lane changing” and “slow down” separated by mid value of Z axis

1083