blue eyes technologies

BlueEyesHumanOperatorMonitoringSystem

Blue EyesTechnologies

•What isBlue Eyes ?

•Motivation

•Physiological foundations

•Systemdesign

•System implementation

•Futureimprovements

•Conclusion

BlueEyesHumanOperatorMonitoringSystem What is BlueEyes ?

• The complex solution for human-operator monitoring:– conscious brain involvement monitoring– triggering user-defined actions– data recording

What is BlueEyes not?• Doesn’t predict nor interfere with

operator’s thoughts• Cannot force directly the operator to work


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Who is BlueEyes for ?

• Generic control rooms– Power station– Flight control centers– Operating theatres – anesthesiologists

• Common application– A simpler system version for drivers


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Motivation

• Human error - a frequent reason of catastrophes and ecological disasters

• Lack of complex solution exploiting wireless link technology, database systems and AI algorithms

– temporal indisposition– weariness – oversight

• Monitoring of consciousbrain involvement

• Capable sensors available


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Wires, wires, wires...

A serious limitation of:

– Mobility– Comfort– Safety


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Personal Area Network !

A wireless Personal AreaNetwork

Solution:


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem BlueEyes - benefits

• Prevention from dangerous incidents• Minimization of

– ecological consequences– financial loss– a threat to a human life

• The reconstructionof the course ofoperator’s work


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Physiological foundations

Preconscious brain

• manages the knowledge

• logical reasoning• makes decisions

• illogical• “automatic”

Conscious brain

Human brain


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Conscious brain - operation modes

Exploratory mode Planning mode


•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Eye movements

Saccade - a rapid eye jump to a new location assigned by the conscious attention process.

Saccades presence ⇔ Exploratory mode

• Central vision andthe conscious brain

• Retina - central vision (macula)•What isBlue Eyes ?

•Motivation


•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem The eye movement sensor

• Eye position measuring - direct infrared oculography

• Two axialaccelerometer

• Ambient lightsensor

Jazz Multisensor


•Motivation


•Systemdesign



•Conclusion


• A reliable real-time buffering, processing and recording

• A clear visualization interface• Mechanisms for creating custom data

analyzers• To ensure a possibility of distributing the

processing among a number of computers

Design objectives

• A mobile data acquisition device

•Objectives

•System overview

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem System overview

Central System UnitData Acquisition Unit

Atmel 89C52microcontroller

JazzMultisensor

ConnectionManager

Data Analysis

VisualizationModuleBluetooth Bluetooth

Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system.

DataLogger

•Objectives

•System overview

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Implementation - DAU

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Implementation - DAU

• Bluetooth module inside• Jazz sensor

interface• Microphone and

earphone sockets• 5-key keyboard• Two lines, 8

characters LCD• Powered from

6 AA batteries

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion


BlueEyesData Acquisition Unit

ID card

DAU - components

MC 145483PCM codec

ID cardinterface

Simplekeyboard

Jazz Multisensor

Beeper

LCD display

LED indicators


UART MAX232

UAR

T

Bluetoothmodule

microphone

earphone

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem DAU - features

• Lightweight• Runs on batteries - low

power consumption• Easy to use - does not

disturb the operator working

• ID cards for operator authorization

• Voice transmission using hardware PCM codec

Data Acquisition Unit


JazzMultisensor

Bluetooth

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Implementation - CSU

• Custom-builtBluetoothcommunicationroutines

• Intel USB camera• ID card programmer• Developed using Visual C++ 6.0 • Deployment environment:

Windows 2000,MS SQL Server

•DAU

•CSU

• Standard MDI userinterface

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem CSU - saccade detector

• Saccade detector and conscious brain involvement level calculation

Eye movement velocity

-250-200-150-100

-500

50100150200250

0 8000 16000 24000 32000Time [ms]

Ve

locit

y

saccadeRaw eyemovement data

Differentiation

Conscious brain involvement level

Visual attention level

0

10

20

30

40

50

60

0 8000 16000 24000 32000

Time [ms]

Att

en

tio

n le

vel

(L

va

)

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem CSU - Visualization Module

•DAU

•CSU

• Raw and processed data visualization using:

• audio and videodata playback

– VU-meters– pie-charts– time series

•Motivation



•Systemdesign



•Conclusion


Bluetooth

Bluetooth

..

.

..

.

CSU - Connection Manager

BluetoothConnection

Manager

OperatorData

Manager

Alerts

Separateddata streams

...

...

OperatorManager

Alert buffer

Raw DataBuffer

Alerts

OperatorManager

Alert buffer

Raw DataBuffer

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem CSU - features

Central System Unit

ConnectionManager

Data Analysis

VisualizationModuleBluetooth

• Connection management

• Data processing• Visualization• Data recording

• Accessverification

• System maintenance

ConnectionManager

•DAU

•CSU

DataLogger

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem CSU - Data Analysis

• Custom data analyzers - decision tree based classifiers

Example: Operator gesture recognition(simplified)

Vertical Acceleration variance value high

Horizontal Acceleration variance high

Horizontal Acceleration variance high

Y N

Y N YN

No answer “NO”“YES”

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem CSU - Data Analysis

• Custom data analyzers - decision tree based classifiers

• Automatic decision tree coefficients generation - C4.5 based algorithm

Data samplesAssignedcategory

“Yes”

No answer

“Yes”

“No”

No answer

... ... Decision tree coefficients

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Data security

• Only registered mobile devicescan connect to the system

• Bluetooth connection authentication• Bluetooth connection encryption• Access rights restrictions• Personal and physiological data

encryption

•DAU

•CSU

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Future improvements

• DAU– small CMOS camera to monitor the

operator’s point of gaze– single PCB (SMD technlogy)– low voltage ICs - LiIO batteries power

• CSU– data mining algorithms – advanced database encryption using e.g.

AES algorithm

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Conclusion

• In the near future ,ordinary household devices- such as television , refrigerators ,ovens may be able to do their jobs when we look at them and speak to them.

• Future applications of blue eye technology is limitless

•Motivation



•Systemdesign



•Conclusion

BlueEyesHumanOperatorMonitoringSystem Thank you for your attention

Any Queries….???

Blue EyesTechnology

•Motivation



•Systemdesign



•Conclusion

blue eyes technologies

Documents

blueeyes human operatorwires

motivation human error

motivation prevention

eye movement sensor

course ofoperators work

wirelesslink technology

consciousattention process

simpler system version