Android and the Brain-Computer Interface

JIM MCKEETH EMBARCADERO TECHNOLOGIES

JIM.MCKEETH@EMBARCADERO.COM @JIMMCKEETH

‹#›

About Jim McKeeth• Lead World Wide Developer Evangelist • Host of Podcast at Delphi.org • Longtime developer ◦ Object Pascal, Java, C#, JavaScript, Objective-C, etc.

• Invented and patented swipe to unlock in 2000 ◦ US Patent # 8352745 & 6766456, etc.

• Improvisational performer • I am not a neuroscientist, but I find this all fascinating!

2

‹#›3

• Founded 1993 •Provides Architect, Database and Developer Tools • 3.2 Million Customers including 97% of Fortune 2000 • 35+ Product Awards – Consistent Innovation • 500+ Employees in 29 Countries •www.embarcadero.com

Embarcadero Technologies

‹#›

Agenda•The Science of Reading the Brain

•The Science of Direct Brain Input

•Consumer Brain-Computer Interfaces

•Demonstration - Volunteer flying the Quadricopter

• Summary & MindTunes

4Images  from  Wikimedia  Commons  or  Product  websites.

‹#›

“In five years, science fiction will be real!”

5

-Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt)

What will the world be like in five years?

‹#›

Brain-Computer Interface

6ST:TNG  Episode  93  (4x19):  The  Nth  Degree  (1991)  

‹#›

Brain Reading and Writing

XSQUIDS  from  Strange  Days  (1995)

Reading the BrainThe Science of

‹#›

Vocabulary and Roots• Encephalon = Latin. The brain. • -graphy (sufix) = Greek. One who draws or writes. • Encephalography = Drawing (or picture) of the brain • Poly- (prefix) = Greek. Many • Tomos = Greek. Cut into parts or sections. • Neuro- (prefix) = Greek. Relating to nerves or nervous system • Hemo- (prefix) = Greek. Pertaining to blood. • Dynamics = Greek. Active; in motion

X

‹#›

Understanding the Brain

X

Hemodynamics  (Blood  flow)De

lta  

4  Hz

Theta  

4-­‐7  Hz

Alph

a  7-­‐14  Hz

Mu  

8-­‐13  Hz

Beta

 15-­‐30  Hz

Gam

ma  

30-­‐100  Hz

Neu

ral  O

scillation

‹#›

Polygraph (aka Lie Detector)•Measures physical manifestations

of stress that come with lying •Blood pressure, pulse, respiration,

skin conductivity, etc.

• Looks for changes.

•Requires specialized training.

• Invented in 1921

8

‹#›

Heart Rate Monitors on Android•Bluetooth GATT Profiles supports

all Bluetooth Smart Heart Rate Monitors.

•Your app doesn’t need to support individual heart rate monitors.

•Assigned Number: 0x180D

9

https://developer.bluetooth.org/TechnologyOverview/Pages/HRP.aspx

https://developer.bluetooth.org/gatt/profiles/Pages/ProfilesHome.aspx

‹#›

Magnetic Resonance Imaging (MRI)• Uses oscillating magnetic fields at

different resonant frequencies. • Functional MRI (fMRI) uses

software to analyze images. • Shows activity (blood flow) in

different parts of the brain. • Combined with brain mapping can

reveal thought information.

10

‹#›

CT / CAT Scan (Computed Tomography X-ray)• Faster & cheaper than an MRI

•Uses ionizing X-ray radiation

• Lower detail images

• Sometimes combined with Positron  Emission  Tomography  (PET)  scans which use gamma  radiation.

X

‹#›

Near-Infrared Spectroscopy (NIRS)• Spectroscopic use of near-

infrared (700-900nm) light through the skull and detecting degree remerging light is attenuated.

• Detects blood oxygenation levels in different areas of the brain.

• More portable than fMRI but only scans cortical tissue.

X

‹#›

Electroencephalography (EEG)•Measures electrical activity along the scalp.

•Detects voltage fluctuations resulting from ionic current flows within neurons of brain.

•Millisecond-range resolution (faster than CT or MRI)

•Typical clinical usage has 19+ input electrodes as well as ground and reference.

• Invented between 1875 and 1924

11

‹#›

Electroencephalography (continued)• Series of electrodes placed next

to scalp.

•Difference in voltage is measured between each electrode and the reference.

X

‹#›

Magnetoencephalography (MEG)

12

Uses Arrays of SQUIDs (Superconducting QUantum Interference Devices)

Direct Brain InputThe Science of

‹#›

Change Perception of Reality

14The  “headjack”  from  The  Matrix  movies  -­‐  1999

‹#›

Transcranial Direct-Current Stimulation (tDCS)•Constant, low current delivered via electrodes. •Results: ◦ Increase cognitive performance ◦ Enhance language & math ability ◦ Improve attention span ◦ Aid problem solving, memory & coordination •Recent DIY “Brain Hacking” Craze ◦ http://www.reddit.com/r/tDCS

15

Accelerated Learning tdcsplacements.com

http://thebrainstimulator.net/

‹#›

tDCS with Mobile• foc.us just launched new tDCS

headsets with Bluetooth API

•Developer devices are available for preorder

•Targeting athletes and gamers

• http://www.foc.us/index.php/v2-tdcs-developer-edition

• http://www.foc.us/bluetooth-api16

‹#›

Transcranial Magnetic Stimulation (TMS)• Similar strength to MRI •Magnetic induction of electrical

currents •Used to treat neurological conditions •Produces: ◦ Flashes of lights in vision (phosphene) ◦Muscular activity

17

‹#›

Ultrasonic Transcranial Stimulation Device• 2000 patent from Sony (US6584357 B1) ◦ Finer targeting than TMS ◦ Theoretically invoking sensory experiences ◦Continued research and patents (US7350522 B2)

18

http://www.newscientist.com/article/mg18624944.600    

Speculative work,

“There were not any

experiments done.”

‹#›

Ultrasonic Neuromodulation for Spatial Resolution• Virginia Tech Carilion

Research Institute study published in Jan 2014 Nature Neuroscience

• Used “targeted ultrasound” to boost “sensory perception”

• Targeted an area of the brain as small as the size of an “M&M”

19http://www.eurekalert.org/pub_releases/2014-­‐01/vt-­‐udt011014.php  

‹#›X

"In neuroscience, it's easy to disrupt things. We can distract you, make you feel numb, trick you with optical illusions. It's easy to make things worse, but it's hard to make them better. These findings make us believe we're on the right path."

-- William "Jamie " Tyler, Assistant Professor Virginia Tech Carilion Research Institute

‹#›20

Using More than 10% of Our Brain?

LUCY2014

We all use 100% of our

Brain Capacity

Interesting MilestonesScience Fiction Becoming Reality

‹#›

Brain-to-Brain Input

22The  “Hat”  from  Brainstorm  -­‐  1983

‹#›

Brain-to-Brain Input at University of Washington in 2013• Input from Computer

• EEG from brain

•Transmitted over campus internet

•Replayed through TMS to another brain

•Output to computer

23More information http://bit.ly/EEG2TMS & http://bit.ly/bci2cbi

‹#›

Cybernetic Limbs

XCaptain  America:  The  Winter  Soldier  -­‐  2014

‹#›

DARPA’s Brain Controlled Prosthetic Arm•Part of DARPA’s Innovation

Pathway.

•Uses microchip implanted in brain.

• 27 different fine grained controls to arm, hand and fingers.

• Expected on market by 2016

Xhttp://www.fastcompany.com/1725799/darpas-­‐mind-­‐controlled-­‐

prosthetic-­‐arm-­‐could-­‐be-­‐market-­‐four-­‐years  

‹#›

See What You See - Hear What You Hear

24The  “Monitor”  from  Ender’s  Game  –  2013

‹#›

See What You See• In 2010 another team lead by Shinji

Nishimoto partially reconstructed YouTube videos based on brain recordings.

• Used fMRI • Overlaid similar videos to create composite video.

• http://youtu.be/nsjDnYxJ0bo & http://youtu.be/EdGjB6XCXOA

25http://bit.ly/ReconstructingVisual

‹#›

See What You See

X

• In 2008 IBM used a feedback loop based on the fusiform face area of the observer’s brain for facial recognition.

•Patent # US20100049076 A1

‹#›

Hear What You Hear• 2012 - UC Berkeley & Brian Pasley decoded

internal neural processing of auditory information.

•Reconstructed sound on computer from electrical signals in observer’s brains.

•Resulting sound was recognizable when compared to original.

26http://bit.ly/ReconstructingSpeech  

‹#›

Identifying Thoughts• 2008 & 2009 Using fMRI recognized thoughts

within the human brain. • Discovered similar thoughts appear similar is

different subject’s brains.

• Was able to discern which of 10 objects subject was thinking about.

• Also predicted subject intentions up to 10 seconds before subject expressed them.

Xhttp://bit.ly/IdentifyThoughts  

‹#›

Data Entry via BCI

27ST:TNG  Episode  93  (4x19):  The  Nth  Degree  (1991)  

‹#›

Typing via BCI•Brain researcher Jonathan Wolpaw

at Wadsworth Center in Albany, NY

•Wear EEG while looking at a flashing matrix of characters

• Each character flashes with a different pattern

•~15 seconds to type a character

•The Verge http://embt.co/typebci

28

. . . . . . . .

. A B C D E F .

. G H I J K L .

. M N O P Q R .

. S T U V W X .

. Y Z 1 2 3 4 .

. 5 6 7 8 9 0 .

. . . . . . . .

‹#›

Uploading Your Consciousness

29Transcendence  -­‐  2014  

‹#›

Human Connectome Project•Map of neural connection the brain

•The neurological version of the genome

•Unique to each person, changes over time

•Representation of memories, thoughts, etc.

•www.ted.com/talks/sebastian_seung

• humanconnectomeproject.org

• humanconnectome.org 30Image  by  Xavier  Gigandet  et.  al.

‹#›

OpenWorm Project•Mapped the 300+ neurons and 7000+

connections of a worm (C. elegans)

•Created a software model of connectome

•Connected to LEGO robotic representation

•Behaved like worm (without training or programming)

• http://bit.ly/openwormlego & http://www.openworm.org/

31

‹#›

As of July 2014

Wearable User Interfaces

Brain-Computer Interface

Internet of Things

Autonomous Vehicles

expectations

Speech-to-Speech Translation

32http://www.gartner.com/newsroom/id/2819918

time

Plateau ofProductivitySlope of EnlightenmentTrough of

DisillusionmentPeak ofInflated

ExpectationsInnovation

Trigger

Plateau will be reached in:more than 10 years5 to 10 years2 to 5 yearsless than 2 years obsolete

before plateau

Gartner's 2014 Hype Cycle for Emerging Technologies August 11, 2014

BCI

Hype Cycle

Brain-Computer InterfacesRead-Only Consumer

‹#›

Consumer BCI Devices• Emotiv EPOC, Insight & EPOC+ www.emotiv.com

•NeuroSky’s MindWave neurosky.com

•OpenBCI www.openbci.com

•Neuroelectrics Enobio www.neuroelectrics.com

• InteraXon’s Muse choosemuse.com

X

‹#›34

• EEG with 1 dry electrode • Measures “attention,” “meditation” and eye-

blinks • Bluetooth communication • SDK for iOS, Android, PC & Mac • $129.99 to $79.95 retail neurosky.com • Hacking: ◦ http://makezine.com/2011/05/10/behind-the-brain-blinker/ ◦ http://hackaday.com/2012/12/20/modifying-an-eeg-headset-for-

lucid-dreaming/

NeuroSky’s MindWave (also in XWave headsets)

‹#›

Emotiv EPOC• 16 wet electrodes ◦ 14 EEG electrodes – read brain waves

+ 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial expressions –

4 Processing suites • Wireless (encrypted) connection to Windows, Linux or Mac • $399 desktop or $499 for Bluetooth Smart (coming soon) • www.emotiv.com

35

‹#›

InteraXon’s Muse• 7 dry EEG sensors via Bluetooth • Focuses on “brain fitness” • 4 hour battery • Basic Mac & Windows SDK now,

full SDK coming soon (incl. iOS & Android)

• Works with Windows, OS X, iOS, Android, etc. • $299 available now • choosemuse.com

36

‹#›

OpenBCI• Open source hardware & software • Bluetooth, Arduino, EEG • 8 wet Electrodes per board (can daisy

chain boards) • Full open EEG & BCI platform • $449 - $799 (now shipping) • www.openbci.com • No iOS or Android yet

37

‹#›

Emotiv Insight• Expected 2015 • Starting at $299 • 5+2 dry sensors • Bluetooth 4.0 LE (Smart) • Battery Life: 4+ hours • SDK: Android, iOS, Mac, Linux

and Windows Platforms

38www.emotiv.com  -­‐  Comparison  http://bit.ly/1cfliTn  

‹#›

No More Woof (“work in progress”)• EEG for Dogs

•Recognize brain patterns and match with mental states:

•Hungry, Playful, Curious, etc.

•Translates to human

•By Nordic Society for Invention and Discovery (NSID)

•Raised $22K on IndiGoGo39http://www.nordicinvention.com/nmw.html

Emotiv EPOCMore Details on the

‹#›

Further Specs• Channels: ◦ AF3, AF4, F3, F4, F7, F8, FC5, FC6, P3  (CMS),  P4  (DRL), P7, P8, T7, T8, O1, O2

• Battery: ◦ 12 hours operation

• Sampling rate: ◦ ~128Hz (2048Hz internal)

• Resolution: ◦ 16 bits (14 bits effective) 1 LSB = 0.51μV

• Bandwidth: ◦ 0.2 - 45Hz, digital notch filters at 50Hz and 60Hz

• More information: ◦ http://emotiv.wikia.com/wiki/Emotiv_EPOC

X

Int.  10-­‐20  locations

‹#›

Head Rotation Suite•Built in solid state gyroscope

•Tracks angular velocity of head movement ◦ Yaw & Pitch ◦No roll detection

• Separate from EEG

X

‹#›

Expressiv Suite• Detect facial expressions ◦ Eyelid & eyebrow positions – Wink, blink, furrow

◦ Horizontal eye movement ◦ Smile, laugh, clenching & smirking • EEG sensors picking up signals to

muscles (not brain waves) • Very fast (10ms)

41

‹#›

Uses for Facial Detection•Realistic avatar expressions

•Anything requiring fast input

• Facial gesture detection

•Basic eye tracking

• Fast input & responses

42

‹#›

Affectiv Suite• Detect emotions / mental states ◦ Excitement ◦ Engagement or Boredom ◦ Meditation ◦ Frustration • Passive detection • Variable levels of each • Short & long term tracking

43

‹#›

Uses for Emotion Tracking•Realtime media evaluation

• Stress analysis and management

•Mood monitoring

• For more accuracy combine with other data sources

• Eye tracking, heart rate, etc.

44

‹#›

Cognitiv Suite• Detect conscious thoughts • Requires training • Detect 13 thought patterns with

intensity level • Left, right, up, down, forward, pull • CW, CCW, left, right, sway-back,

sway-forward, Disappear • Not as fast as Expresiv

45

‹#›

Uses for Cognitive Thought Identification•Whole new input method

•Variable intensity for fine control

•Observe subconscious responses

•More training improves accuracy

• Improves the model

•Trains your brain to focus

46

‹#›

The Pieces

47

Emotiv EPOC Brain-Computer Interface

Google Glass Wearable Computer

Parrot.AR Drone 2 Simi-Autonomous

‹#›

About Appmethod• Integrated Development Environment • Multi-platform visual designer • Native compiler for Win32, Win64, OS X, iOS & Android • Framework for rapid prototyping and accelerated

development with C++ or Object Pascal • Architecture allowing shared code & UI across platforms • Advanced language features like ARC, Generics, etc.

48

www.appmethod.com

Starts at $25/mo

per platform or

C++ Android Phone for

Free

‹#›

Laptop

EDKMy  App

How it Works

49

Brain

Electrical  

Signals

Wireless  Signal

AT*REF=1,2

90718208

Provides  WiFi    

Hotspot

Feedback

Emotiv EPOC SDK + Parrot AR.Drone +

Google Glass

Demonstration

Tweet @Appmethod and #IoTor G+, LinkedIn, Facebook, etc.

‹#›

Emotiv EPOC SDK•Platforms

◦Windows

◦Mac OS X

◦Ubuntu

◦ Fedora

◦ iOS & Android coming soon

51

• Languages

◦C / C++ / Objective-C

◦C# (.NET)

◦ Python

◦Object Pascal / Delphi

◦ Java coming soon

‹#›

MindTunes• Three disabled musicians • Collaborated with DJ Fresh and

Julien Castet (tech) • Created dance track • http://youtu.be/HyF4ZxGhPHw

• Full documentary • http://youtu.be/PgfxKZiSCDQ

52

‹#›

More Information• Email: jim.mckeeth@embarcadero.com • Emotiv EPOC: www.emotiv.com

• Parrot AR.Drone: projects.ardrone.org • Appmethod: www.appmethod.com • My blog: delphi.org • Full Slide Deck: http://www.slideshare.net/jimmckeeth/jim-mc-

keeth-wearable-thought-input • Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC

53