Abstract—It is difficult for the astronauts in a space capsule to control the outside robot under the real space condition. Multi-mode based human-machine interface is proposed to complete the task by the primary information of human being such as Electroencephalogram (EEG) and electromyography (EMG). Digital signal processor is adopted to process the information. EEG and EMG are sampled before their feature is extracted. The detail order is formed to control the machine such as robot. The speech recognition based on some fixed Chinese words is included in the device. Many tests under the simulated space condition in the lab proved that the developed system is capable to control the robot for key operation or stick control on a panel with reasonable reliability.

I. INTRODUCTION pace environment is complex which is full of all sorts of electromagnetic radiation with low air pressure. It is necessary for the astronauts to communicate with robots

outside the capsule and to realize accurately control of robots. The control signal source to space service robot is always the focus in this field. The “biological sensors system” is the result after millions of years of evolution. The biological sensor has advantage over the other artificial sensor in signal interpretation and treatment method, energy consumption, volume and sensitivity. Controlling robot by biological signal is a good taste and the most ideal human-machine technology. Brain-computer interface (BCI) [1]-[3] technology is an important human-machine interaction based on electroence -phalogram. BCI has attracted more researchers’ attention and achieved rapid development in recent years, because of the improvement of people’s understanding in nervous system, the development of computer science and the rapid progress of signal processing technology. The development of BCI related theory and technology, such as cerebral neuroscience, behavioral science, psychology, information

Manuscript received October 29, 2009. This work is supported by fund of Chinese National Programs for High

Technology Research and Development Project (2007AA04Z254), Tianjin, China Science & Technology Research and Development Project (08ZCKFSF03400), Chinese Academy of Sciences (CAS) Supporting Tianjin’s Science & Technology Research and Development Project (TJZX2-YW-06), Tianjin Higher School Science & Technology Development Fund (20090704) , China Postdoctoral Science Fund (20090460501) and Science & Technology fund of Tianjin University of Technology and Education (YJS10-05, KJY11-08). Genghuang Yang is with the Tianjin Key Laboratory of Information Sensing & Intelligent Control in Tianjin University of Technology and Education, Tianjin, 300222, China (phone and fax: 8602288181115; e-mail: ygenghuang@yahoo.com.cn). Feifei Wang, Shigang Cui, Li Zhao, Qingguo Meng, and Hongda Chen are with Tianjin University of Technology and Education, Tianjin, 300222, China.

fusion processing, pattern recognition, neural network, promotes the boom of brain-machine interface. The most representative of research in brain-machine interface is G. Pfurtscheller etcs’ series of study based on events related potential of BCI system, which realized two representative brain-machine interface system named Graz I and Graz II. The electromyography sampled from skin surface is one type of biological signal, which can directly reflect the movement intentions of human body, is very suitable to serve as the control source for space robot. It is the effective interface of human-computer interaction system. The studies on EMG become very meaningful in this field. Human being transfers one's intentions to control robot conveniently and flexibly by EMG. The recognition of EMG has obtained high positive rate as the technology of acquisition and processing of signal tend to reach perfection. For example, the recognition rate of a single action reaches above 95%. A lot of research is in the field of control based on EMG. Shanghai Jiaotong Univ. and Fudan. Univ. carry out the subject “the control of nerve movement and the research of control source information” cooperatively. The research purpose is to extract neural information by using the neural information to control the electronic driven prosthesis. Multiple patterns based human-machine interface in our research combines both EEG and EMG. There is difference in control objective between EEG and EMG. The steady state visual evoked potential (SSVEP) is the stimulus paradigm in the use of EEG and simple coding based on the recognition of EMG is applied to control. Two sets of signal regulation circuit are designed to sample weak EEG and EMG signal from human being’s skin surface [2-3]. Digital signal processor is used to process the sampled data by embedded fast Fourier transform (FFT) algorithm and energy distribution (ED) algorithm. FFT describes the spectrum of EEG and extracts the characteristics based on SSVEP. ED extracts the characteristics of muscular EMG by the energy of signal. Differential control instructions are formed to control the movement and operation of a robot [4-6]. Multiple patterns based human-machine interface achieved the following object. The robot moves to the front of the panel of button and joystick. With specific instruction, the robot pushes the button upward, downward, to right or to left. To mark the processing of control in the above action of robot, that is, to mark the start or the end of each instruction and to form the whole control instruction series, speech recognition function based on fixed Chinese vocabulary is added to the device.

Multi-mode Human-Machine Interface Oriented to Space Service Robot

Genghuang Yang, Feifei Wang, Shigang Cui, Li Zhao, Qingguo Meng, and Hongda Chen

S

714

Fourth International Workshop on Advanced Computational Intelligence Wuhan, Hubei, China; October 19-21, 2011

978-1-61284-375-9/11/$26.00 ©2011 IEEE

II. SYSTEM STRUCTURE DESIGN Multi-mode human-computer interface oriented from space service robot mainly includes the developed device for information collection and signal processing, the steady-state visual stimulator, wireless data transmission module, robot and its control system, cameras and video transmission equipment, as Fig. 1. shows.

LCD Displayer

Subjecty

Steady-state Visual Stimulus

Panel

Camera

Device of Data

Acquisition

PC

Manipulator

RS-232

Robot Control System

Wireless Receiver

RS-232

Wireless Transmitter

Sensor Lines

Room1

Room 2

Wireless Communication

WIFI

WIFI

Fig. 1. System structure

A. Device for Information Collection and Signal Processing The developed device completes the data acquisition of EEG and EMG. The algorithm solidified in the device realizes the characteristic extraction of EEG and EMG. The output and

Fig. 2. Hardware structure

instructions are used to describe the subject's control idea. The device structure is as Fig. 2. shows. Device adopt double processor (DSP-F2812+ MCU -SPCE061B), and DSP just accomplished all data collection and analysis. Speech training and identification algorithms realized directly through the sunplus microcontroller; DSP reserve 2 groups of RS-232 communication interface, one realizes its of the robot control command output, secondly realized data interaction with other devices or PC; Double processor via SPI interface realized data sharing. The hardware and software system based on Digital signal Processor (DSP) Digital Singnal Processor, based on the characteristics of EEG signals and the needs of signal processing, choose special of ADC (Analog to produce stats - to Digital Converter -, ADC) transition chip American ADI 16 AD transform AD976 in the design. By the reason of processing multi-channel EEG, if every road use an AD976, it will increase overall volume, improving power consumption and circuit raise cost, so we carries on the corresponding EEG choice with multi-channel switch selector CD74HCT4067.

Based on characteristics of EEG and EMG signal, EEG and EMG amplifier circuit with high-precision, strong anti-jamming ability is designed. The amplitude of normal EEG in human being is 5-200μV, but the potential of cerebral cortex may steep to 1mV. The EEG is weak signal. The frequency of EEG varies from 1Hz to 100Hz, which is marked as low frequency. EEG amplifier circuit with floating ground tracking technology is shielded to reduce electromagnetic interference from power supply. Common-code Rejection Ratio (CMRR) of the amplifier is about 110dB and the input impedance is about 1000MΩ per channel. The qualities of the amplifier can overcome poor signal disturbance caused by resistance when electrons touch skin. Notch Filter for 50Hz signal, reduces power line’s interference. The integrated amplifier with low noising for biological signal amplifier and high qualified high and low pass filter, safeguard the whole system of outstanding performance. There is difference in EMG of normal human being caused by physical condition and different body parts. In our work, the EMG is sampled from the forearm. Different EMG is generated through pressing forearm muscles when different action of wrist takes. The amplitude ranges from 20mV to 200mV and the frequency ranges from 30Hz to 400Hz. The design of EMG amplifier is similar with EEG amplifier circuit. Only the pass band should be adjusted. Fig. 3 shows the paste position of EEG and EMG electrode.

Fig. 3. Electrodes

715

B. The Steady-state Panel of Visual Stimulation The panel is consisted of seven different white flash square of 1.5cm×1.5 cm with black background. As Fig. 4 shows, these flashing blocks can be set with different flashing frequency which has access to evoked potentials in relevant frequency. Flash modules are composed by LED highlighted lamp, which is covered by flat acrylic board. LED brightness is reduced by the semitransparent board and the light gets blur. The white LED light scattering from square panel, can effectively stimulate subjects and induce steady-state vision related characteristic of EEG.

Fig. 4. The panel of steady-state visual stimulation

C. Wireless Data Transmission Modules Wireless data transmission modules are formed by wireless transmitters and receivers. Two desktop computers are connected to wireless router separately to establish the data connection between the device and control system for robot. The device is connected to wireless data transmitter through RS-232 interface and the wireless data receiver is also connected to control system for robot by RS-232. The control instructions are transmitted to control system for robot through wireless network.

D. Robot and Its Control Mobile Robots’ Pioneer3–DX robot is selected to be the robot for testing, as Fig. 5 shows. Hitachi H8S with more efficient signal processing speed and more powerful function expansion ability is the controller. The CPU of on-board computer is also upgraded to P-III. ARIA and AROS software

Fig. 5. Robot

platform provides the customers with more perfect experiment and simulation.

Cyton Alpha 7D 1G is used as the manipulator. As Fig. 6 shows, the manipulator has seven degrees of freedom and a fixture of manipulator.

Fig. 6. Manipulator structure

E. Cameras and Video Transmission Equipment The camera, video card and desktop computer realize the real-time video display of manipulator’s operating in front of the panel. Subject can observe whether his issued instructions is right or not by watching video which forms the simple feedback.

III. SOFTWARE DESIGN The design of software for the multi-mode

human-computer interaction design includes the acquisition and analysis of EEG and EMG, speech recognition and control, steady-state visual stimulator flashing control in frequency and control of robot.

A. The EEG Acquisition and Analysis EEG acquisition and analysis is realized mainly in DSP. Steady-state visual evoked potential is related to visual stimulation frequency. This is the EEG pattern in our work. Fast Fourier Transform (FFT) algorithm is used to extract the characteristics of EEG. FFT is sensitive to the sampling frequency. The tiny error in sampling frequency results in great error of spectrum. It is important to keep the sampling frequency high accuracy. The highest frequency in our work is 25Hz, so the EEG sampling frequency determined for 128Hz which is higher than 50Hz. The samples’ interval is generated by the timer in DSP. As there are two channels in data acquisition, switch selector CD74HCT4067 is operated real-timely to finish data acquisition in proper order. Data acquisition flowchart is as Fig.7. shows. The 4/8 based FFT algorithm is embedded in DSP to obtain the spectral characteristic of EEG. Data window length for FFT analysis is 2 seconds, which is 256 sampling points. The outstanding spectrum diagram of SSVEP focuses on flickering frequency or its multiples of the visual stimulator.

716

Fig. 7. Timer interrupt service flowchart

B. Collection and Analysis of EMG EMG acquisition and analysis is realized in DSP. EMG

characteristic is extracted by the way of measuring energy. EMG energy is mainly concentrated from 100Hz to 200Hz based on the action of our work. The sampling frequency is 1000Hz. The samples’ interval is generated by the timer in DSP. Data acquisition flowchart is the same with Fig. 7.

In a test, subjects squeeze arm muscle by different wrist movements; EMG with different characteristics is generated. There is difference of EMG’s energy between action and no action. EMG’s characteristic is obtained directly by measuring energy. The Root of Mean Square (RMS) of samples in every 2 seconds sample by sample is calculated. The different threshold is also preset to compare with RMS series so as to distinguish different action of wrist. The result of the comparison is control information for robot and so on. To guarantee the reliability of the robot’s controlling, a single threshold is used in the first stage of our work, that is, EMG’s effective information is a binary number. Three consecutive bits form the whole control information.

C. Speech Recognition and Control The development is carried out based on commercial

platform provided by Sunplus Co., Ltd. According to the detail application, the voice interaction is built to be the important part of the human-machine interaction which is integrated in the multi-mode control system.

The voice interaction is mainly based on simple fixed vocabulary in Chinese which packages and orient the control command extracted in EEG and EMG in the whole process. For example, the recognition of “start” and “break”, “over” and the other words make sure of the starting different signal acquisition procedure, and guarantee the control command work in the proper order for the whole control task.

D. Steady-state Visual Stimulation Control in Frequency Flashing block frequency which is set by key board is

realized by microcontroller timer. The key board is designed by polling in input port. The users can set the frequency of each flash block at expect frequency ranging from 5-25Hz.

E. Robot control The software design of robot control [7], [8] is accom- plished based on ARIA. ARIA is developed for Mobile Robots, in object-oriented way for robot control application with C++ computer language. ARIA is simple client software which provide convenient interface for movement control and sensor operation of Pioneer3–DX robot. There are powerful functionality and adaptability in ARIA which is an ideal choice for robot high-level programming. Specific program- ming of modules realizes buttons or joystick operation with the manipulator.

IV. DESIGN OF EXPERIMENTS AND DATA ANALYSIS One subject and one assistant instructor attend in the

experiments, experimental equipment as mentioned above. Subjects and robots in different laboratories were shown as

shown in Fig. 1. As Fig. 8 shows, the subject lies on the test bed with angle of inclination of - 6 ° which is used in space simulation testing. The aim of the experiment is that subject controls a robotic arm to push or release a key on panel through EEG and EMG as Fig. 9 shows.

Fig. 8. Subject

Fig. 9. Robot Subject is wearing EEG electrodes. As Fig. 8 shows, the

visual stimulation panel is fixed on a bracket which is located above the eyes of subject by a distance of 40cm. The specific

717

Fig. 10. Robot control experimental process

location should be adjusted to make the subject feel comfortable.

Auxiliary instructors set each flash block frequency of steady-state visual stimulator, and place flash module position of steady-state operation visual evoked related to concrete panel buttons. After having a rest for 30 seconds, subjects were issued voice commands “start”. The experimental process is executed as Fig. 10 shows.

In EEG experiment, the assistant instructor wears earphones, send order to the subject. One flash square is selected according to the related control task such as the designated button. The subject closes the eyes and has a rest for 30 seconds. After assistant instructor says voice commands “start”, the subject must carefully gaze at the designated flash square for 10 seconds. The assistant instructor observes the robotic arm by real-time video which camera is installed in the room of robot to see if the operation on the panel is correct or not.

The steady-state visual evoked potential which bandwidth is 12Hz-25Hz. 12Hz, 15Hz and 18Hz are selected to be the final frequency to stimulus the subject. EEG is analyzed by FFT. The frequency (12Hz, 15Hz or 18Hz) related to

maximum in the frequency spectrum is result of EEG feature extraction. There are 10 subjects attending the experiment, in which 5 are boys aged from 20 to 28 (subject No. 1, 3, 5, 7, 9) and 5 are girls aged from 20 to 26 (subject No. 2, 4, 6, 8, 10). Each subject has 10 tests at every frequency. There are 285 groups of data which is the command for button operation. The accuracy is 95%. The result of the experiment is divided into two groups, in which one is the accuracy EEG feature extraction and the other is the accuracy of robot operation. The test result is as table I shows. “Sn” stands for “Subject number”, “Cn” stands for “Correct number”, “Ac” stands for “Accurcy rate”.

Table I EEG ANALYSIS RESULTS

Sn 12Hz 15Hz 18Hz Cn Ac % Cn Ac % Cn Ac %

1 10 100 10 100 8 80 2 10 100 10 100 9 90 3 9 90 10 100 10 100 4 10 100 9 90 9 90 5 10 100 10 100 10 100 6 10 100 9 90 8 80 7 10 100 10 100 10 100 8 8 80 9 90 10 100 9 10 100 10 100 9 90

10 10 100 10 100 8 80

In EMG experiment, the subject’s arm wears EMG

electrode and lies on test bed. Assistant instructor wears earphones and sends order to the subject. The order is the action of the wrist which is related to the button operation on the panel. The subject closes the eyes and has a rest for 10 seconds. After assistant instructor says voice commands “start”, the subject must pull the wrist once per second to produce the command. The assistant instructor observes the robotic arm by real-time video which camera is installed in the room of robot to see if the operation on the panel is correct or not.

As the above describes, the way of EMG bits is used to be the instructions. By calculating the energy of EMG, if the value is more than the threshold, it is regarded as digital signal “1”, otherwise, as “0”. To ensure the reliability of command output, different buttons is selected respectively. The series of “I”, “I” and “III” are the final operating instructions. There are 4 subjects attending the experiment, in which 2 are boys aged from 20 to 26 (subject No. 1, 3, 5, 7, 9) and 2 are girls aged from 20 to 27 (subject No. 2, 4, 6, 8, 10). Each subject has 30 tests at every EMG series (“I”, “II” and “III”). Before the subjects participate in EMG test, they need simple training. The data of the training help to do threshold calibration. The EMG signal of a subject is shown as Fig. 11.

There are 360 groups of data which is the command for button operation. The test result is as table II shows. There are 1102 groups of data which is the command for button operation. The accuracy is 91.8%. Male subjects have obvious advantage over female subjects in EMG characteristic extraction. The manipulator operates key accurately 997

718

Fig. 11. EMG after the signal amplification

times. The accuracy is 90.5%. “Sn” stands for “Subject number”, “Cn” stands for “Correct number”, “Ac” stands for “Accurcy rate”.

Table II EMG ANALYSIS RESULTS

Sn I II III IV

Cn Ac % Cn Ac % Cn Ac % Cn Ac % 1 30 100 29 97 28 93 28 93 2 28 93 28 93 29 97 29 97 3 29 97 27 90 26 87 25 83 4 28 93 26 87 25 83 25 83 5 30 100 28 93 27 90 27 90 6 29 97 28 93 26 87 26 87 7 30 100 29 97 29 97 28 93 8 29 97 27 90 26 87 24 80 9 29 97 30 100 28 93 27 90

10 27 90 27 90 26 87 25 83

V. CONCLUSION The human-machine interface based on DSP and MCU in

our research can accurately sample EEG and EMG with low noise. The designed algorithm can simply and efficiently extract steady-state visual evoked potentials and EMG energy which are the characteristics for control. Assisted by speech recognition to package the command, the instructions of EEG or EMG form the order of the robot to operate the button on the panel with high accuracy. The designed experiments and analysis of the obtained data show that the developed device runs very well to be a multi-mode human-machine interface. It is a good taste in lab for the aerospace applications in the future.

REFERENCES [1] E. Dochin, K.M. Spencer, R. Wijensinghe, “The mental prosthesis:

Assessing the speed of a P300-based brain-computer interface,” IEEE Trans. Rehabil. Eng., vol. 8, pp.174-179, Aug. 2000.

[2] G.H. Yang, L.T. Xiao, L. Zhao, and S.G. Cui, “Design and implementation of a brain-computer interface based on virtual instrumentation,” Int. J. Adv. Mechatronic Syst., vol. 2, pp. 36-45, Feb. 2010.

[3] G.H. Yang, L. Zhao, S.G. Cui, “Brain-Computer Interface Based Camera Carrier in Aerospace,” in Proceedings of IEEE International Conference on Automation and Logistics, Shenyang, China, 2009, pp.1877-1882.

[4] M. Fatourechi, A. Bashashati, R.K. Ward, and G.E. Birch, “EMG and EOG artifacts in brain computer interface systems: a survey,” Clin. Neurophysiol., vol. 118, pp. 480-494, Mar. 2007.

[5] A. Schlǒgl, C. Keinrath, D. Zimmermann, R. Scherer, R. Leeb, and G. Pfurtscheller, “A fully automated correction method of EOG artifacts in EEG recordings,” Clin. Neurophysiol., vol. 118, pp. 98-104, Jan. 2007.

[6] S.L. Sun, C.S. Zhang, “Assessing features for electroencephalographic signal categorization,” in Proceeding of International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, USA, 2005, vol. 5, pp.417-420.

[7] Z. Ji, P. Song, “Design of a reconfigurable platform manipulator,” J. Robot. Syst., vol. 15, pp. 341-346, Jun. 1998.

[8] I.M. Chen, S.H. Yeo, G. Chen, and G.L. Yang, “Kernel for modular robot applications: automatic modeling techniques,” Int. J. Robot. Res., vol. 18, pp. 225-242, Feb. 1999.

719