Biofeedback inVirtual Reality applications and Gaming

Bonie Rosario, Jr. Sebastian Osorio

Tom Iancovici

University of Massachusetts Lowell

Intro to Biosensors16.441.201

Prof. Xingwei Wang

OutlineWhat is Biofeedback?ElectroencephalogramThe Galvanic Skin Response and

Heart Rate VariabilityBiosensor ResearchResearch findingsOur Conclusions

What is Biofeedback?Biofeedback = the technique of

using monitoring devices to furnish information regarding an automatic bodily function, such as heart rate or blood pressure, in an attempt to gain some voluntary control over these functions[1].

Electroencephalogram The electroencephalography is defined as a graphic

representation of the potential difference between two different cerebral locations plotted over time[3].

EEG activity can be subdivided into various types of frequency rhythm bands. Research has indicated that different EEG frequency bands are associated with different mental states. The major five are alpha, theta, beta delta and gamma.

EEG signals tend to have amplitude in the range of 2V to 100 V

EEG Commercial Products

NeuroSky MindWave Emotiv EPOC

A recent study compared the quality score of an EEG System based on price, functionality and usability. In terms of usability, Emotiv’s EPOC scored the highest.

There are several affordable commercial products EEG Systems. Two of the most popular and user-friendly are produced by NeuroSky and Emotiv.

An example of a Two-channel EEG amplifier

An example of an EEG device prototype

How EEG Signals are Measured: Hardware

Emotive EPOC Hardware and Software

The Emotive EPOC EEG Headset samples the EEG signal at a rate of 120 Samples per Second (SPS) using an Analog-to-Digital Converter (ADC). It samples signals from 14 different electrodes and filters out artifacts, or noise, outside the bandwidth range of 0.2 – 45 Hz. It then is Emotiv EPOC headset Specification

How EEG Signals are Measured: Software The Emotiv EPOC Headset

I. Expressive Suite: User facial recognitionII. Affective Suite: Monitors player emotional states in real-timeII. Cognitive Suite: Reads and interprets a player’s conscious thoughts and intent

Cognitive Suite Expressive Suite

How EEG Signals are Measured: Software

Affective Suite

SDK of the Research Edition

The heart rate and the galvanic skin response are very accurate manifestations of human reaction such as boredom, stress, and excitement. They are also very easy to measure using non-intrusive techniques.

GSR and HRV Biosensor

http://www.hsphysicsteacher.com/electriccurres/13%20Electric%20Current%20&%20Resistance/13.06%20galvanic%20skin%20response.jpg

http://www.healthy-heart-meditation.com/images/IOM-Technology.jpg

Stress

• Sweat glands are activated

HRV• Increas

ed Heart Rate

Low resistan

ce pathwa

y

• Apply small voltage to human skin to induce a current

How GSR and HRV are Measured

HRV= Heart Rate Variable

GSR=Voltage/CurrentAlso known as Instantaneous Skin Resistance

What About Current Input Devices?

Mouse and Keyboard are relatively easy to use and are recognizable by the public

“Typing” and “Clicking” seem to be the most efficient ways to interact with a computer.

Multi-Biosensor Gaming System

Research by the University of Saskatchewan, Canada

Combines◦ EMG (Electrical

Activation of Muscle Tissue)

◦ GSR (Galvanic Skin Response)

◦ HRS (Heart Rate Variability)

◦ Respiration◦ Temperature◦ Eye Movement / Gaze

Source: Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L. 2011. “Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction,” CHI 2011, Vancouver, BC, Canada.

Purpose of ResearchEvaluate how feasible it is to

incorporate biofeedback peripherals with current computer gaming hardware and software

Determine which kinds of biosensors feel natural to what types of activities

Study10 participants who are not

affluent with video gamesPlay a Two-Dimensional Shooting

Game◦Using only a traditional video game

controller◦Using a controller enhanced with

biofeedback peripherals◦Using a controller enhanced with

biofeedback peripherals mapped to a different set of functions

Gameplay

Increasing enemy sprite size

Movement and flamethrower length

Gameplay

Snowfall and final boss battle

Medusa’s Gaze

Testing Different BiosensorsMechanic Cond. 1 Cond. 2 Target size RESP GSR

Speed/jump EKG EMG Weather/boss TEMP EKG

Flamethrower GSR RESP

Avatar control Gamepad Gamepad

Medusa’s Gaze Gaze Gaze

General Results9 out of the 10 like the biosensors“made for a very immersive game, out

of what is basically just a very simple platform shooter.”

“the sensors added a new dimension to the game, and gave a greater sense of involvement.”

“variation, and more enjoyment while playing because there are always new skills to improve on.”

“sensors [made the] game complicated.”

More General Results

“I like the idea of using multiple physiological inputs. Distributing the functions around the body is intuitive in some cases.”

“[It] basically boils down to an "extended controller" where the buttons are not buttons but other actions, similar to waving a Wii remote.”

[Preferred biosensor for each activity]

GSR Feedback“I liked that it was always a

challenge to control just with my thoughts […] and forced me to use a part of my brain I wouldn't normally use in a video game.”

“I disliked the fact that one of the only ways that I found I was able to use the GSR was by biting my lip which isn't actually all that fun after it starts hurting.”

-overreaction

EKG FeedbackI couldn't control as instantly as

the others - the effect from it tended to last over longer periods of time.”

“better suited to changing the game context than what the character is doing.”

EMG Feedback“It was fairly easy to use. It was

effective and worked.” “[…] having this sensor tied to

jumping/speed felt natural” - muscle strain during repeated

use

Respiratory Feedback“It was neat to see the

immediate reaction from my body to the game.”

“[It] felt very natural, particularly when it was tied to target size in the game […]. It's one I felt I could control to a fine degree.”

Immediate Reaction

Temperature Feedback“It was easy to use for short

periods of time but hard to remember to breathe deep into the sensor […].”

“I like […] when it was tied to weather […], because it felt like a natural thing to do. […]”

“Breathing rapidly to increase temperature also brought my heart rate up.”

-intrusiveness

Gaze Feedback“Now that was just cool […] I

liked being able to roast one frozen combatant while immobilizing another.”

“I found it frustrating when you would look at a platform intending to jump on it and end up freezing it.”-Problems of overfocusing

Additional Feedback“[The] muscle and breathing sensors

were simple enough that they were practically like a new button on the controller for me, but very awesome ones since rather than tapping a button, it was an instinctive action.”

“The breathing sensor and GSR sensor felt like controllers though, because I was very aware they were attached to me. The EKG and EMG were completely unnoticeable and fun to use…”

Researchers’ ConclusionsBiofeedback devices made

games more interactive and more fun

Preferred sensors were one that were easy to control and produces noticeable results in real-time

Sensors were appreciated when mapped to activities that felt natural

Our FindingsEEG allows direct control without

physical contact◦Assistive Technology◦Accuracy Issues + set up◦Cost

GSR and HRV reflect users’ state of mind◦More intelligent gaming systems◦Guides gamers to feel particular emotions◦Personalizes each user’s gaming

experience

ConclusionsInnovativeNot enough researchNot cost effective yetUnlimited potentialFun and “out of the box”

applications of biomedical theory

DemoDemoCampus Catalyst Competition

◦Entrepreneurship Showcase◦May 5, 2011 at Merrimack College

References Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L. 2011.

“Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction,” CHI 2011, Vancouver, BC, Canada.

[1]"Biofeedback - Definition of Biofeedback by the Free Online Dictionary, Thesaurus and Encyclopedia." Dictionary, Encyclopedia and Thesaurus - The Free Dictionary. Web. 26 Feb. 2011. <http://www.thefreedictionary.com/biofeedback>.

[2]Tarvainen, M.P.; Karjalainen, P.A.; Koistinen, A.S.; Valkonen-Korhonen, M.V.; , "Principal component analysis of galvanic skin responses," Engineering in Medicine and Biology Society, 2000. Proceedings of the 22nd Annual International Conference of the IEEE , vol.4, no., pp.3011-3014 vol.4, 2000doi: 10.1109/IEMBS.2000.901513URL: http://xplorebcpaz.ieee.org/stamp/stamp.jsp?tp=&arnumber=901513&isnumber=19495