neurophone: brain- mobile phone interface using a wireless eeg headset andrew t. campbell, tanzeem...
TRANSCRIPT
![Page 1: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/1.jpg)
NEUROPHONE: BRAIN-MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET
Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu,
Matthew K. Mukerjee!, Mashfiqui Rabbi, and Rajeev D. S. Raizada
Dartmouth College, Hanover, NH, USA
![Page 2: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/2.jpg)
Motivation• Mobile phones and neural signals are present are
accessible to many people. • Recent advances in technology has led to the
development in low-cost EEG headsets. • Smart phones are now powerful enough to run
sophisticated machine learning algorithms.• It is thus easy to interface neural signals with mobile
computing paradigms.
![Page 3: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/3.jpg)
Introduction• This group proposed to used neural signals to control a
mobile phone. • They developed the NeuroPhone system that translates
and decodes neural signals to drive a mobile app using off-the-shelf wireless EEG headsets.
• This paper demonstrates their brain-controlled address app:• An application that uses the brain signals to select address
contacts to call.
![Page 4: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/4.jpg)
Introduction• They implement their mobile app using two different
paradigms: P300 dialing and “Wink”-triggered dialing. • P300 signals are positive transient deflections in EEG that are
elicited in response to a rare or novel stimulus• The eye “Wink” is a type of EMG signal that is generated in
response to the contraction of skeletal muscle contraction.
![Page 5: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/5.jpg)
Challenges• Research Grade EEG headsets
• Expensive (Often costing tens of thousands of dollars)• Offer very robust and reliable EEG signals
• Off-the-shelf EEG headsets• More affordable ($100-$500)• Electrode design and amplification are not as robust
• Results in noisy, low-quality signals.• Require more sophisticated processing techniques to classify neural
events.
• Most Off-the-shelf headsets are wireless and thus encrypt the EEG signals. • They are designed for synchronization with a computer (using wireless
dongle). • They complicate the process of developing a clean brain-mobile
interface.
![Page 6: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/6.jpg)
Challenges• There is an energy cost for brain-mobile interfacing:
• Continuously streaming raw brain-signals wirelessly• Running classifiers on the phone introduces heavy processor
loads.
• Brain-mobile phones could likely be used in applications such as: walking, riding in a car or bicycle, shopping, etc. • Many of these cases present significant noise artifacts in the EEG
signals. • These signals will need to be filtered out to improve the brain-mobile
interface
![Page 7: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/7.jpg)
NeuroPhone• The NeuroPhone system uses the
iPhone to display pictures of contacts in the phone’s address book.
• The pictures are displayed and flashed in random order.
• For the EEG mode, the user concentrates on a picture of the person they wish to call.
• For the wink mode, the person winks with the left or right eye to make the intended phone call
![Page 8: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/8.jpg)
P300
• Whenever the user concentrates on a target stimulus among a pool of non-target stimulus, the target stimulus (flash) will elicit a positive peak in the EEG at around 300ms after stimulus onset (P-300).
• The P300 signal can be found on most EEG channels• Common on central and
parietal channels
![Page 9: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/9.jpg)
NeuroPhone - P300 Paradigm
• In This case, there are 6 total stimuli on the screen (5 non-target and 1 target). The user visually attends to one of the photos while each photo is flashed in a random order. Whenever the target photo flashes, a P300 should be generated.
![Page 10: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/10.jpg)
Wireless EEG Headset• Emotiv EPOC headset
• 14 data electrodes (2 reference electrodes)• Transmits encrypted data wirelessly to a
windows-based machine. (802.11) 2.4GHz• Low SNR• Contains build in gyroscope• ~$300
![Page 11: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/11.jpg)
Pre-Processing• Signals were band-passed filtered to keep only the
relevant information within the P300 range. • Signal averaging was performed to increase the SNR
• This improves the quality of the signal while simultaneously adding lag to the system
![Page 12: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/12.jpg)
Classification• To reduce complexity, only a subset of relevant channels
are used for classification. • Wink Mode
• Multivariate, naive Bayesian classifier.
• P300 Mode• Decision stump classifer
![Page 13: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/13.jpg)
Implementation• Laptop relay is used for decoding of the encrypted Emotiv
signals• Encrypted EEG signals are sent from the phone to a laptop for
decryption (via WiFi). • Decrypted EEG signals are sent back to the phone.• Signals are sampled at 128 samples per second and transferred to
the phone at 4kbps per channel.
![Page 14: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/14.jpg)
Wink Mode Classification• Emotiv head-set was put on
backwards to place two electrodes directly above the eyes.
• Data was collected by having the subject wink multiple times. – Data were labeled as “wink” or “non-
wink”
• A Bayesian classifier was trained by calculating the mean and variance of each wink and non-wink and building respective Gaussian models. – As can be seen, the two models do not
overlap leading to good classification
![Page 15: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/15.jpg)
P300 Classification• The Gaussian distributions overlap too much and
therefore cannot be classified with a Bayesian classifier. • Signals from each of the six stimuli were band-passed
filtered between 0-9Hz. • The highest signal segment at around 300ms after
stimulus onset is extracted. • For classification, a decision stump is used where the
threshold is set to the maximum value of the extracted segment.
![Page 16: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/16.jpg)
• Multiple sessions were collected on three subjects. • Subjects performed the test while sitting and while walking• The classifier was trained on five sessions from a single
subject and then tested on the remaining subjects. (I think). • Results are shown in table 1
– Precision: % of classified winks that are actual winks– Recall: % of actual winks that are classified as winks. – Accuracy: % of total events that are classified correctly
Results (Wink-Mode)
![Page 17: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/17.jpg)
Results (P300 mode)• Data was collected with same set of subjects while sitting,
with loud background music and while standing up.
![Page 18: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/18.jpg)
Discussion• Although data was classified using the P300 mode, large
amounts of averaging is needed to get decent classification accuracies. • This “unresponsiveness” of the system proves to be very
frustrating for the end user. • i.e. it can take 100 seconds to initiate a phone call with only 89% chance
of dialing the right person (with six to choose from).
• This System is currently not in any form to be used by subjects on a regular basis. • Looking into single trial classification techniques to speed up the
system.
![Page 19: NEUROPHONE: BRAIN- MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K. Mukerjee!,](https://reader036.vdocuments.mx/reader036/viewer/2022062304/56649d0c5503460f949e12df/html5/thumbnails/19.jpg)
Phone Loading Statistics• The CPU usage when running the application:
• 3.3% for the iPhone (iphone 3g?).
• Total memory usage:• 9.40MB memory used
• (9.14MB are for GUI elements).
• Continuous streaming raw EEG channels to the phone, and processing signals lead to battery drain (no quantitative measure given)• Looking into duty cycling to solve this phone.