INTRODUCTION

People express their mental states, including emotions, thoughts, and desires, all the

time through facial expressions, vocal nuances and gestures. This is true even when they are

interacting with machines. Our mental states shape the decisions that we make, govern how

we communicate with others, and affect our performance. The ability to attribute mental

states to others from their behavior and to use that knowledge to guide our own actions and

predict those of others is known as theory of mind or mind-reading.

Existing human-computer interfaces are mind-blind — oblivious to the user’s mental

states and intentions. A computer may wait indefinitely for input from a user who is no

longer there, or decide to do irrelevant tasks while a user is frantically working towards an

imminent deadline. As a result, existing computer technologies often frustrate the user, have

little persuasive power and cannot initiate interactions with the user. Even if they do take the

initiative, like the now retired Microsoft Paperclip, they are often misguided and irrelevant,

and simply frustrate the user. With the increasing complexity of computer technologies and

the ubiquity of mobile and wearable devices, there is a need for machines that are aware of

the user’s mental state and that adaptively respond to these mental states.

1

WHAT IS MIND READING?

A computational model of mind-reading

Drawing inspiration from psychology, computer vision and machine learning, the

team in the Computer Laboratory at the University of Cambridge has developed mind-

reading machines — computers that implement a computational model of mind-reading to

infer mental states of people from their facial signals. The goal is to enhance human-

computer interaction through empathic responses, to improve the productivity of the user and

to enable applications to initiate interactions with and on behalf of the user, without waiting

for explicit input from that user. There are difficult challenges:

Fig: Processing stages in the mind-reading system

Using a digital video camera, the mind-reading computer system analyzes a person’s

facial expressions in real time and infers that person’s underlying mental state, such as

whether he or she is agreeing or disagreeing, interested or bored, thinking or confused.

2

Prior knowledge of how particular mental states are expressed in the face is combined

with analysis of facial expressions and head gestures occurring in real time. The model

represents these at different granularities, starting with face and head movements and

building those in time and in space to form a clearer model of what mental state is being

represented. Software from Nevenvision identifies 24 feature points on the face and tracks

them in real time.

Movement, shape and colour are then analyzed to identify gestures like a smile or

eyebrows being raised. Combinations of these occurring over time indicate mental states. For

example, a combination of a head nod, with a smile and eyebrows raised might mean interest.

The relationship between observable head and facial displays and the corresponding hidden

mental states over time is modeled using Dynamic Bayesian Networks.

3

WHY MIND READING?

Current projects in Cambridge are considering further inputs such as body posture and

gestures to improve the inference. We can then use the same models to control the animation

of cartoon avatars. We are also looking at the use of mind-reading to support on-line

shopping and learning systems

4

Fig:Monitoring a car driver

The mind-reading computer system presents information about your mental state as

easily as a keyboard and mouse present text and commands. Imagine a future where we are

surrounded with mobile phones, cars and online services that can read our minds and react to

our moods. How would that change our use of technology and our lives? We are working

with a major car manufacturer to implement this system in cars to detect driver mental states

such as drowsiness, distraction and anger.

Current projects in Cambridge are considering further inputs such as body posture and

gestures to improve the inference. We can then use the same models to control the animation

of cartoon avatars. We are also looking at the use of mind-reading to support on-line

shopping and learning systems.

The mind-reading computer system may also be used to monitor and suggest

improvements in human-human interaction. The Affective Computing Group at the MIT

Media Laboratory is developing an emotional-social intelligence prosthesis that explores new

technologies to augment and improve people’s social interactions and communication skills.

5

HOW DOES IT WORK?

Fig: Futuristic headband

The mind reading actually involves measuring the volume and oxygen level of the

blood around the subject's brain, using technology called functional near-infrared

spectroscopy (fNIRS).

The user wears a sort of futuristic headband that sends light in that spectrum into the

tissues of the head where it is absorbed by active, blood-filled tissues. The headband then

measures how much light was not absorbed, letting the computer gauge the metabolic

demands that the brain is making. The results are often compared to an MRI, but can be

gathered with lightweight, non-invasive equipment.

6

Wearing the fNIRS sensor, experimental subjects were asked to count the number of

squares on a rotating onscreen cube and to perform other tasks. The subjects were then asked

to rate the difficulty of the tasks, and their ratings agreed with the work intensity detected by

the fNIRS system up to 83 percent of the time.

"We don't know how specific we can be about identifying users' different emotional

states," cautioned Sergio Fantini, a biomedical engineering professor at Tufts. "However, the

particular area of the brain where the blood-flow change occurs should provide indications of

the brain's metabolic changes and by extension workload, which could be a proxy for

emotions like frustration."

"Measuring mental workload, frustration and distraction is typically limited to

qualitatively observing computer users or to administering surveys after completion of a task,

potentially missing valuable insight into the users' changing experiences.

A computer program which can read silently spoken words by analyzing nerve signals

in our mouths and throats has been developed by NASA.

Preliminary results show that using button-sized sensors, which attach under the chin

and on the side of the Adam's apple, it is possible to pick up and recognize nerve signals and

patterns from the tongue and vocal cords that correspond to specific words.

"Biological signals arise when reading or speaking to oneself with or without actual

lip or facial movement," says Chuck Jorgensen.

7

HEAD AND FACIAL ACTION UNIT ANALYSIS

Twenty four facial landmarks are detected using a face template in the initial frame,

and their positions tracked across the video. The system builds on Facestation [1], a

feature point tracker that supports both real time and offline tracking of facial features on a

live or recorded video stream. The tracker represents faces as face bunch graphs [23] or

stack-like structures which efficiently com- bine graphs of individual faces that vary in

factors such as pose, glasses, or physiognomy. The tracker outputs the position of twenty

four feature points, which we then use for head pose estimation and facial feature extraction.

EXTRACTING HEAD ACTION UNITS

Natural human head motion typically ranges between 70-90o of downward pitch,

55o of upward pitch, 70o of yaw (turn), and 55o of roll (tilt), and usually occurs

as a combination of all three rotations [16]. The output positions of the localized

feature points are sufficiently accurate to permit the use of efficient, image-based head pose

estimation. Expression invariant points such as the nose tip, root, nostrils, inner and outer

eye corners are used to estimate the pose. Head yaw is given by the ratio of left to right eye

widths. A head roll is given by the orientation angle of the two inner eye corners. The

computation of both head yaw and roll is invariant to scale variations that arise from moving

toward or away from the camera. Head pitch is determined from the vertical displacement

of the nose tip normalized against the distance between the two eye corners to account for

scale variations. The system supports up to 50o , 30o and 50o of yaw, roll and pitch

respectively. Pose estimates across consecutive frames are then used to identify head action

units. For example, a pitch of 20o degrees at time t followed by 15o at time t + 1

indicates a downward head action, which is AU54 in the FACS coding.

EXTRACTING FACIAL ACTION UNITS

Facial actions are identified from component-based facial features (e.g.mouth)

comprised of motion, shape and color descriptors. Motion and shape-based analysis are

particularly suitable for a real time video system, in which motion is inherent and places a

strict upper bound on the computational complexity of methods used in order to meet time

8

constraints. Color-based analysis is computationally efficient, and is invariant to the scale or

viewpoint of the face, especially when combined with feature localization (i.e. limited to

regions already defined by feature point tracking). The shape descriptors are first stabilized

against rigid head motion. For that, we imagine that the initial frame in the sequence is a

reference frame attached to the head of the user. On that frame, let (Xp , Yp ) be an

“anchor” point.

2D projection of the approximated real point around which the head rotates in 3D

space. The anchor point is initially defined as the midpoint between the two mouth corners

when the mouth is at rest, and is at a distance d from the line joining the two inner eye

corners l. In subsequent frames the point is measured at distance d from l, after accounting

for head turns.

Fig 2: Polar distance in determining a lip corner pull and lip pucker

On each frame, the polar distance between each of the two mouth corners and the

anchor point is computed. The average percentage change in polar distance calculated with

respect to an initial frame is used to discern mouth displays. An increase or decrease of 10%

or more, determined empirically, depicts a lip pull or lip pucker respectively (Figure 2). In

addition, depending on the sign of the change we can tell whether the display is in its onset,

apex, offset. The advantages of using polar distances over geometric mouth width and

height (which is what is used in Tian et al [20]) are support for head motion and

resilience to inaccurate feature point tracking, especially with respect to lower lip points.

Fig 3 : Plot of aperture (red) and teeth (green) in luminance-saturation space

9

The mouth has two color regions that are of interest: aperture and teeth. The extent of

aperture present inside the mouth depicts whether the mouth is closed, lips parted, or jaw

dropped, while the presence of teeth indicates a mouth stretch. Figure 3 shows a plot of teeth

and aperture samples in luminance-saturation space .Luminance given by the relative

lightness or darkness of the color, acts as a good discriminator for the two types of mouth

regions. A sample of n=125000 pixels was used to learn the probability distribution

functions of aperture and teeth. A lookup table defining the probability of a pixel being

aperture given its luminance is computed for the range of possible luminance values (0% for

black to 100% for white). A similar lookup table is computed for teeth. Online

classification into mouth actions proceeds as follows: For every frame in the sequence, we

compute the luminance value of each pixel in the mouth polygon. The luminance value is

then looked up to determine the probability of the pixel being aperture or teeth. Depending

on empirically determined thresholds the pixel is classified as aperture or teeth or neither.

Finally, the total number of teeth and aperture pixels are used to classify the mouth region

into closed (or lips part), jaw drop, or mouth stretch. Figure 4 shows classification results of

1312 frames into closed, jaw drop and mouth stretch.

Fig 4: Classifying 1312 mouth regions into closed, jaw drop or stretch

10

COGNITIVE MENTAL STATE INFERENCE

The HMM level outputs likelihood for each of the facial expressions and head

displays .However, on their own, each display is a weak classifier that does not entirely

capture an underlying cognitive mental state. Bayesian networks have successfully been

used as an ensemble of classifiers, where the combined classifier performs much better

than any individual one in the set [15].

In such probabilistic graphical models, hidden states (the cognitive mental states in

our case) influence a number of observation nodes, which describe the observed facial and

head displays. In dynamic Bayesian networks (DBN), temporal dependency across

previous states is also encoded. Training the DBN model entails determining the param- eters

and structure of a DBN model from data. Maximum likelihood estimates is used to learn the

parameters, while sequential backward elimination picks the (locally) optimal network

structure for each mental state model. More details on how the parameters and structure are

learnt can be found in [13].

EXPERIMENTAL E V ALUATION

For our experimental evaluation we use the Mind reading dataset (MR) [3]. MR is a

computer-based guide to emotions primarily collected to help individuals diagnosed with

Autism recognize facial expressions of emotion. A total of 117 videos, recorded at 30

fps with durations varying between 5 to 8 seconds, were picked for testing. The videos

conveyed the following cognitive mental states: agreement, concentrating, disagreement,

thinking and un- sure and interested. There are no restrictions on the head or body

movement of actors in the video. The process of labeling involved a panel of 10 judges who

were asked could this be the emotion name. ? When 8 out of 10 agree, a statistically

significant majority, the video is included in MR. To our knowledge MR is the only

available, labeled

11

Fig 5: ROC curves for head and facial displays

resource with such a rich collection of mental states and emotions, even if they are posed.

We first evaluate the classification rate of the display recognition layer and then the overall

classification ability of the system.

DISPLAY R ECOGNITION

We evaluate the classification rate of the display recognition component of the system

on the following 6 displays: 4 head displays (head nod, head shake, tilt display, turn

display) and 2 facial displays (lip pull, lip pucker). The classification results for each of the

displays are shown using the Receiver Operator Characteristic (ROC) curves (Figure 5).

ROC curves depict the relationship between the rate of correct classifications and number of

false positives (FP). The classification rate of display d is computed as the ratio of correct

detections to that of all occurrences of d in the sampled videos. The FP rate for d is given

by the ratio of samples falsely classified as d to that of all non-d occurrences. Table 2

shows the classification rate that the system uses, and the respective FP rate for each display.

A non-neutral initial frame is the main reason behind undetected and falsely detected

displays. To illustrate this, consider a sequence that starts as a lip pucker. If the lip pucker

12

persists (i.e. no change in polar distance) the pucker display will pass undetected. If on the

other hand, the pucker returns to neutral (i.e. increase in polar distance). It will be falsely

classified as a lip pull display. This problem could be solved by using the polar angle and

color analysis to approximate the initial mouth state. The other reason accounting for

misclassified mouth display is that of inconsistent illumination. Possible solutions to dealing

with illumination changes include extending the color-based analysis to account for overall

brightness changes or having different models for each possible lighting condition.

MENTAL STATE R ECOGNITION

We then evaluate the overall system by testing the inference of cognitive mental

states, using leave-5-out cross validation. Figure 6 shows the results of the various stages

of the mind reading system for a video portraying the mental state choosing, which belongs

to the mental state group thinking. The mental state with the maximum likelihood over the

entire video (in this case thinking) is taken as the classification of the system.

87.4% of the videos were correctly classified. The recognition rate of a mental

class m is given by the total number of videos of that class whose most likely class (summed

over the entire video) matched the label of the class m. The false positive rate for class

m (given by the percentage of files misclassified as m) was highest for agreement (5.4%)

and lowest for thinking (0%). Table 2 summarizes the results of recognition and false

positive rates for 6 mental states.

A closer look at the results reveals a number of interesting points. First, onset

frames of a video occasionally portray a different mental state than that of the peak. For

example, the onset of disapproving videos were misclassified as unsure .Although this

incorrectly biased the overall classification to unsure, one could argue that this result is not

entirely incorrect and that the videos do indeed start off with the person being unsure.

Second, subclasses that do not clearly exhibit the class signature are easily

misclassified. For example, the assertive and decided videos in the agreement group were

misclassified as concentrating, as they exhibit no smiles, and only very weak head nods.

Finally, we found that some mental states were “closer” to each other and could co-occur.

For example, a majority of the unsure files scored high for thinking too.

13

WEB SEARCH

For the first test of the sensors, scientists trained the software program to recognize

six words - including "go", "left" and "right" - and 10 numbers. Participants hooked up to the

sensors silently said the words to themselves and the software correctly picked up the signals

92 per cent of the time.

Then researchers put the letters of the alphabet into a matrix with each column and

row labeled with a single-digit number. In that way, each letter was represented by a unique

pair of number co-ordinates. These were used to silently spell "NASA" into a web search

engine using the program.

"This proved we could browse the web without touching a keyboard”.

14

MIND-READING COMPUTERS TURN HEADS AT HIGH-TECH FAIR

Devices allowing people to write letters or play pinball using just the power of their

brains have become a major draw at the world's biggest high-tech fair.

Huge crowds at the CeBIT fair gathered round a man sitting at a pinball table,

wearing a cap covered in electrodes attached to his head, who controlled the flippers with

great proficiency without using hands."He thinks: left-hand or right-hand and the electrodes

monitor the brain waves associated with that thought, send the information to a computer,

which then moves the flippers," said Michael Tangermann, from the Berlin Brain Computer

Interface. But the technology is much more than a fun gadget, it could one day save your life

Scientists are researching ways to monitor motorists' brain waves to improve reaction

times in a crash. In an emergency stop situation, the brain activity kicks in on average around

200 milliseconds before even an alert driver can hit the brake. There is no question of braking

automatically for a driver -- "we would never take away that kind of control,"

15

"However, there are various things the car can do in that crucial time, tighten the seat

belt, for example," he added. Using this brain-wave monitoring technology, a car can also tell

whether the driver is drowsy or not, potentially warning him or her to take a break. At the

g.tec stall, visitors watched a man with a similar "electrode cap" sat in front of a screen with

a large keyboard, with the letters flashing in an ordered sequence.

The user concentrates hard when the chosen letter flashes and the brain waves

stimulated at this exact moment are registered by the computer and the letter appears on the

screen. The technology takes a long time at present -- it took the man around four minutes to

write a five-lettered word -- but researchers hope to speed it up in the near future. Another

device allows users to control robots by brain power. The small box has lights flashing at

different

16

ADVANTAGES AND USES

Mind Controlled Wheelchair

1. This prototype mind-controlled wheelchair developed from the University of Electro

Communications in Japan lets you feel like half Professor X and half Stephen

Hawking—except with the theoretical physics skills of the former and the telekinetic

skills of the latter.

2. A little different from the Brain-Computer Typing machine, this thing works by

mapping brain waves when you think about moving left, right, forward or back, and

then assigns that to a wheelchair command of actually moving left, right, forward or

back.

3. The result of this is that you can move the wheelchair solely with the power of your

mind. This device doesn't give you MIND BULLETS (apologies to Tenacious D) but

it does allow people who can't use other wheelchairs get around easier.

4. The sensors have already been used to do simple web searches and may one day help

space-walking astronauts and people who cannot talk. The system could send

commands to rovers on other planets, help injured astronauts control machines, or aid

disabled people.

5. In everyday life, they could even be used to communicate on the sly - people could

use them on crowded buses without being overheard

6. The finding raises issues about the application of such tools for screening suspected

terrorists -- as well as for predicting future dangerousness more generally. We are

closer than ever to the crime-prediction technology of Minority Report.

7. The day when computers will be able to recognize the smallest units in the English

language—the 40-odd basic sounds (or phonemes) out of which all words or

verbalized thoughts can be constructed. Such skills could be put to many practical

17

uses. The pilot of a high-speed plane or spacecraft, for instance, could simply order by

thought alone some vital flight information for an all-purpose cockpit display.

DISADVANTAGES AND PROBLEMS

Tapping Brains for Future Crimes

1. Researchers from the Max Planck Institute for Human Cognitive and Brain Sciences,

along with scientists from London and Tokyo, asked subjects to secretly decide in

advance whether to add or subtract two numbers they would later are shown. Using

computer algorithms and functional magnetic resonance imaging, or fMRI, the

scientists were able to determine with 70 percent accuracy what the participants'

intentions were, even before they were shown the numbers. The popular press tends to

over-dramatize scientific advances in mind reading. FMRI results have to account for

heart rate, respiration, motion and a number of other factors that might all cause

variance in the signal. Also, individual brains differ, so scientists need to study a

subject's patterns before they can train a computer to identify those patterns or make

predictions.

2. While the details of this particular study are not yet published, the subjects' limited

options of either adding or subtracting the numbers means the computer already had a

50/50 chance of guessing correctly even without fMRI readings. The researchers

indisputably made physiological findings that are significant for future experiments,

but we're still a long way from mind reading.

3. Still, the more we learn about how the brain operates, the more predictable human

beings seem to become. In the Dec. 19, 2006, issue of The Economist, an article

questioned the scientific validity of the notion of free will: Individuals with particular

congenital genetic characteristics are predisposed, if not predestined, to violence.

4. Studies have shown that genes and organic factors like frontal lobe impairments, low

serotonin levels and dopamine receptors are highly correlated with criminal behavior.

Studies of twins show that heredity is a major factor in criminal conduct. While no

18

one gene may make you a criminal, a mixture of biological factors, exacerbated by

environmental conditions, may well do so.

5. Looking at scientific advances like these, legal scholars are beginning to question the

foundational principles of our criminal justice system.

6. For example, University of Florida law professor Christopher Slobogin, who is

visiting at Stanford this year, has set forth a compelling case for putting prevention

before retribution in criminal justice.

7. It's a tempting thought. If there is no such thing as free will, then a system that

punishes transgressive behavior as a matter of moral condemnation does not make a

lot of sense. It's compelling to contemplate a system that manages and reduces the risk

of criminal behavior in the first place.

8. Max Planck Institute, neuroscience and bioscience are not at a point where we can

reliably predict human behavior. To me, that's the most powerful objection to a

preventative justice system -- if we aren't particularly good at predicting future

behavior, we risk criminalizing the innocent.

9. We aren't particularly good at rehabilitation, either, so even if we were sufficiently

accurate in identifying future offenders, we wouldn't really know what to do with

them.

10. Nor is society ready to deal with the ethical and practical problems posed by a system

that classifies and categorizes people based on oxygen flow, genetics and

environmental factors that are correlated as much with poverty as with future

criminality.

11. In time, neuroscience may produce reliable behavior predictions. But until then, we

should take the lessons of science fiction to heart when deciding how to use new

predictive techniques.

19

12. The preliminary tests may have been successful because of the short lengths of the

words and suggests the test be repeated on many different people to test the sensors

work on everyone.

13.The initial success "doesn't mean it will scale up", he told New Scientist. "Small-

vocabulary, isolated word recognition is a quite different problem than conversational

speech, not just in scale but in kind."

14.that genes and organic factors like frontal lobe impairments, low serotonin levels and

dopamine receptors are highly correlated with criminal behavior. Studies of twins

show that heredity is a major factor in criminal conduct. While no one gene may

make you a criminal, a mixture of biological factors, exacerbated by environmental

conditions, may well do so.

15.Using computer algorithms and functional magnetic resonance imaging, or fMRI, the

scientists were able to determine with 70 percent accuracy what the participants'

intentions were, even before they were shown the numbers.

20

CONCLUSION

Tufts University researchers have begun a three-year research project which, if

successful, will allow computers to respond to the brain activity of the computer's user. Users

wear futuristic-looking headbands to shine light on their foreheads, and then perform a series

of increasingly difficult tasks while the device reads what parts of the brain are absorbing the

light. That info is then transferred to the computer, and from there the computer can adjust it's

interface and functions to each individual. 

One professor used the following example of a real world use: "If it knew which air

traffic controllers were overloaded, the next incoming plane could be assigned to another

controller." 

Hence if we get 100% accuracy these computers may find various applications in

many fields of electronics where we have very less time to react.

21

BIBILOGRAPHY

www.eurescom.de/message/default_Dec2004.asp

blog.marcelotoledo.org/2007/10

www.newscientist.com/article/dn4795-nasa-develops-mindreading-system

http://blogs.vnunet.com/app/trackback/95409

22