blue eye technology

BlueEyes

HumanOperatorMonitoringSystem

BlueEyesHuman-Operator Monitoring System

Poznań University of Technology, Polandhttp://www.put.poznan.pl

Team mentor

Team members Paweł KowalikPiotr KubiaczykKrystian NowakStanisław OsińskiTomasz Pużak

Jan Kniat, Ph. D.

•Motivation

•What isBlueEyes ?

•Physiological foundations

•Systemdesign

•System implementation

•Futureimprovements

•Project worksummary

•Systemdemonstration

BlueEyes

HumanOperatorMonitoringSystem Agenda

• Motivation• What is BlueEyes (and what is not) ?• Physiological foundations• System design• System implementation• Future improvements• Project work summary• System demonstration

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Motivation

• Human error - a frequent reason of catastrophes and ecological disasters

•Motivation



•Systemdesign





• Lack of complex solution exploiting wireless link technology, database systems and AI algorithms

– temporal indisposition– weariness – oversight

• Monitoring of consciousbrain involvement

• Capable sensors available

BlueEyes

HumanOperatorMonitoringSystem What is BlueEyes ?

• The complex solution for human-operator monitoring:– conscious brain involvement monitoring– triggering user-defined actions– data recording

•Motivation



•Systemdesign





What is BlueEyes not?

• Doesn’t predict nor interfere with operator’s thoughts

• Cannot force directly the operator to work

BlueEyes

HumanOperatorMonitoringSystem Who is BlueEyes for ?

• Generic control rooms– Power station– Captain bridge– Flight control centers– Operating theatres – anesthesiologists

• Common application– A simpler system version for drivers

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem BlueEyes - benefits

• Prevention from dangerous incidents• Minimization of

– ecological consequences– financial loss– a threat to a human life

• The reconstructionof the course ofoperator’s work

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Physiological foundations

Preconscious brain

•Motivation



•Systemdesign





• manages the knowledge

• logical reasoning• makes decisions

• alogical• „automatic”

Conscious brain

Human brain

BlueEyes

HumanOperatorMonitoringSystem Conscious brain - operation modes

Exploratory mode Planning mode

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Eye movements

Saccade - a rapid eye jump to a new location assigned by the conscious attention process.

Saccades presence Exploratory mode

• Central vision andthe conscious brain

• Retina - central vision (macula)•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem The eye movement sensor

• Eye position measuring - direct infrared oculography

• Oxy- and deoxyhemoglobin measurment• Two axial

accelerometer• Ambient light

sensor

Jazz Multisensor

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Wires, wires, wires...

A serious limitation of:– mobility– comfort– safety

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Personal Area Network !

A wireless Personal AreaNetwork

•Motivation



•Systemdesign





Solution:

BlueEyes


• A reliable real-time buffering, processing and recording

• A clear visualization interface• Mechanisms for creating custom data

analyzers• To ensure a possibility of distributing the

processing among a number of computers

Design objectives

• A mobile data acquisition device•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem System overview

Central System UnitData Acquisition Unit

Atmel 89C52microcontroller

JazzMultisensor

ConnectionManager

Data Analysis

VisualizationModuleBluetooth Bluetooth

Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system.

DataLogger

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem DAU - features

• Lightweight• Runs on batteries - low

power consumption• Easy to use - does not

disturb the operator working

• ID cards for operator authorization

• Voice transmission using hardware PCM codec

Data Acquisition Unit


JazzMultisensor

Bluetooth

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem CSU - features

Central System Unit

ConnectionManager

Data Analysis

VisualizationModuleBluetooth

• Connection management

• Data processing• Visualization• Data recording

• Accessverification

• System maintenance

ConnectionManager

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

DataLogger

BlueEyes

HumanOperatorMonitoringSystem CSU - saccade detector

• Saccade detector and conscious brain involvement level calculation

Eye movement velocity

-250-200-150-100

-500

50100150200250

0 8000 16000 24000 32000Time [ms]

Ve

locit

y

saccadeRaw eyemovement data

Differentiation

Conscious brain involvement level

Visual attention level

0

10

20

30

40

50

60

0 8000 16000 24000 32000

Time [ms]

Att

en

tio

n le

ve

l (L

va

)

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem CSU - Visualization Module

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

• Raw and processed data visualization using:

• audio and videodata playback

– VU-meters– pie-charts– time series

BlueEyes

HumanOperatorMonitoringSystem Data security

• Only registered mobile devicescan connect to the system

• Bluetooth connection authentication• Bluetooth connection encryption• Access rights restrictions• Personal and physiological data encryption

•Motivation



•Systemdesign





•Objectives

•System overview

•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem Implementation - DAU

• Bluetooth module inside• Jazz sensor

interface• Microphone and

earphone sockets• 5-key keyboard• Two lines, 8

characters LCD• Powered from

6 AA batteries

•Motivation



•Systemdesign





•DAU

•CSU

• Made from scratch by the team

BlueEyes

HumanOperatorMonitoringSystem Implementation - DAU

Bluetooth module

PCM codec board

Batteries andvoltage level monitor

Microcontroller board

•Motivation



•Systemdesign





•DAU

•CSU

BlueEyes

HumanOperatorMonitoringSystem Implementation - CSU

• Custom-builtBluetoothcommunicationroutines

• Intel USB camera• ID card programmer• Developed using Visual C++ 6.0 • Deployment environment:

Windows 2000,MS SQL Server

•Motivation



•Systemdesign





•DAU

•CSU

• Standard MDI userinterface

BlueEyes

HumanOperatorMonitoringSystem Future improvements

• DAU– small CMOS camera to monitor the

operator’s point of gaze– single PCB (SMD technlogy)– low voltage ICs - LiIO batteries power

• CSU– data mining algorithms – advanced database encryption using

e.g. AES algorithm

•Motivation



•Systemdesign





BlueEyes

HumanOperatorMonitoringSystem Project work summary

• Bluetooth and PAN technology acquired• New skills on designing software and

hardware systems gained• A mobile device implemented• 31 KLOC of C++ code and

4 KLOC of 8051 assembler code written• All parts of the Project Kit efficiently used• New approach to system safety proposed

and implemented

•Motivation



•Systemdesign





BlueEyes



System demonstration

•Motivation



•Systemdesign





BlueEyes


Thank you for your attention


Poznań University of Technology, Polandhttp://www.put.poznan.pl

•Motivation



•Systemdesign





BlueEyes


BlueEyesData Acquisition Unit

ID card

DAU - components

MC 145483PCM codec

ID cardinterface

Simplekeyboard

Jazz Multisensor

Beeper

LCD display

LED indicators


UART MAX232

UA

RT

Bluetoothmodule

microphone

earphone

BlueEyes


• Initializes Bluetooth modules

• Inquires for new devices in range

• Establishes connections

• Supports operator roaming

• Feeds Operator Manager with physiological dataBluetooth

Bluetooth

...

...

CSU - Connection Manager

BluetoothConnection

Manager

OperatorData

Manager

Alerts

Separated

data streams

...

...

OperatorManager

Alert buffer

Raw DataBuffer

Alerts

OperatorManager

Alert buffer

Raw DataBuffer

•Data demultiplexer•Data producer queues

BlueEyes

HumanOperatorMonitoringSystem CSU - Data Analysis

• Custom data analyzers - decision tree based classifiers

Example: Operator gesture recognition(simplified)

Vertical Acceleration variance value high

Horizontal Acceleration variance high

Horizontal Acceleration variance high

Y N

Y N YN

No answer “NO”“YES”

BlueEyes

HumanOperatorMonitoringSystem CSU - Data Analysis

• Custom data analyzers - decision tree based classifiers

• Automatic decision tree coefficients generation - C4.5 based algorithm

Data samplesAssignedcategory

“Yes”

No answer

“Yes”

“No”

No answer

... ... Decision tree coefficients

BlueEyes

HumanOperatorMonitoringSystem CSU - intermodule communication

• Uses buffered thread-safe queues and producer-consumer scheme

Dataproducer

Dataqueue

Dataconsumer

Dataconsumer

creates

Dataconsumer

Processing

Processing

Dataconsumer &

producer

Dataconsumer &

producer

Dataqueue

Dataqueue

register at

Dataconsumer

BlueEyes

HumanOperatorMonitoringSystem CSU - intermodule communication

Data AnalysisVisualization

ModuleData

Logger

Operator manager (OM) OM OM

ConnectionManager

...

Separated physiological data streams

Processed data

Alerts

Recorded (off-line) data

blue eye technology

Technology

simpler system version

design objectives

drivers motivation

agenda motivation

motivation human error

number of computers

blueeyes benefits prevention

data logger