Toward Brain-Computer InterfacingMinhye Chang

Contents

5. The Berilin Brain-Computer Interface: Machine Learning-Based Detection of User Specific Brain States

7. Brain Interface Design for Asynchronous Control

Chapter 5

• Berlin Brain-Computer Interface procject

• Premovement Potentials in Executed and Phan-

tom Movements

• BCI Control-Based on Imagined Movements

• Lines of Further Improvement

Brelin Brain-Computer Interface

Brelin Brain-Computer Interface

• EEG-based BCI system with machine learning techniques– High quality feedback w/o subject training– Copes with the huge intersubject variability

• Spatial resolution of the somatotopy• Discriminability of premovement potentials in voluntary mo-

vements• Sensorimotor rhythms caused by motor imagery

– 128 channel EEG– Subjects w/ no or very little experience w/ BCI control– Information transfer rate above 35bits per minute.

Premovement Potentials

• Bereitschaftspotenzial, RP (readiness potential)– Activity in the motor cortex leading up to voluntary muscle movement– To build a classifier

• Letting healthy subjects actually perform the movements– Movement imagination poses a dual task: motor command prepara-

tion plus vetoing the actual movement

• Predictions about imminent movements– Exclude a possible confound with feedback from muscle and joint re-

ceptors– Assistance of action control in time-critical behavioral contexts

Left vs. Right Hand Finger Movements

• Readiness potential– Pronounced cortical negativation

• Left-hand vs. Right-hand finger tapping experiment– Electrodes : CCP3 and CCP4– Predominantly contralateral negativation before movement

• Self-paced finger-monements on a computer keyboard– Tap-rates of 30, 45, 60, and 120 taps per minute– 128 Ag/AgCl scalp electrodes– EMG, EOG

• Discriminate premovement potentials as fast as two taps per second

• Highly subject-specific

Left vs. Right Hand Finger Movements

Preprocessing

• Extracts the low-frequency content w/ an emphasis on the late part of the signal

Starting points: 128 samples

Emphasize the late signal content

FT coefficients : Pass-band

Four feature components per channel

Classification

• Regularized linear discriminant analysis (RLDA)– RP features normally distributed with equal covariance matrices.– The data processing preserve gaussianity

RP-Based Feedback in Asynchronous Mode

• Calibration session– The data is used to train a classifier.

• A useful continuous feedback in an asynchronous mode– Classifier must work for a broader interval of time– The system needs to detect the buildup of movement intentions

• Quite simple strategy : jittering– Extracts several with some time jitter b/w training samples– Two samples per key press: at 150 and at 50 ms before key press– Invariant to time shifts

• EMG activity at about 120ms

RP-Based Feedback in Asynchronous Mode

• Movement intention detector– Distinguishes b/w motor preparation intervals and “rest” intervals

• In a BCI feedback experiment (-160 to -80ms)

Detection of ‘Phantom Limb Commands’

• ERD(ERS) • Attenuation (amplification) of pericentral μ and β rhythms in the cor-

responding motor areas

• Lack of a time marker signal– Listened to an electronic metronome– Deep sound: rest, higher sound: perform either a finger tap or a phan-

tom movement

• 8 patients with amputations showed significant ″phantom-re-lated″ ERD/ERS of μ- and/or β -frequencies – Signed r2 values of the differences in ERD curves

• Gross somatotopic arrangement– Hand versus foot

• Finely graded representation of individual fingers• Examine the discriminability of BCI signals from close-by brain

regions– 128-channel EEGs– During self-paced movements of various limbs– Significantly reflect specific activations in sensorimotor cortices

Exploring the Limits

Exploring the Limits

• Averaged premovement potential patterns of one subject in different self-paced limb moving tasks

• Significantly reflect specific activations in sensorimotor cor-tices

• In repetitive movements, the discrimination decays already af-ter about 1s

• Modulations of sensorimotor rhythms evoked by imagined movements

• 6 subjects who had no or very little experience w/ BCI feed-back– 118-channel EEGs– Recorded EOG and EMG

• Calibration measurement (machine training)– Estimate parameters of a brain-signal to control-signal translation al-

gorithm

BCI Control-Based on Imagined Movements

• In the training sessions– Visual stimuli for 3.5s: (L) left hand, (R) right hand, or (F) right foot– Two types were selected for feedback; binary classifier

• Bias and scaling of the linear classifier– Different experimental condition of the (exciting) feedback situation

Experimental Setup

• 1st feedback application: position-controlled cursor– Classifier output translated to the horizontal position of a cursor.

Experimental Setup

Holding for 500msActivatedSuccess

• 2nd feedback application: rate-controlled cursor– At each update step a fraction of the classifier output was added to

the actual cursor position.

• 3rd feedback application: basket game– Operated in a synchronous mode– Horizontal position was controlled by

the classifier output

Experimental Setup

• Spatial filters– Optimize the discriminability of brain signals based on ERD/ERS ef-

fects of the motor rhythms– Common spatial pattern (CSP) analysis

• Feature calculation– Log of the variance in those surrogate channels– Calculated every 40 ms from sliding windows of 250 to 1000ms (sub-

ject-specific) for online operation

• Details about the processing methods and the selection of pa-rameters : Blankertz et al. (2005)

Processing and Classification

• Information transfer rate (ITR) in bits per minute (bpm)

– Compared to ROC curves, ITR considers different duration of trials and different number of classes

• Highest ITRs: rate-controlled cursor, asynchronous protocol

Results

• It can be operated at a high decision speed– Average trial length for 1st, 2nd, and 3rd was 3s, 2.5s, and 2.1~3s resp.

• The fastest subject : average speed of one decision every 1.7s• Subject who showed the most reliable performance

– only 2% of the total 200trials were misclassified at an average speed of one decision per 2.1s

– Sentences 135 characters in 30 minutes (4.5 letters per minute) in a free-spelling mode

Results

• It is possible to voluntarily modulate motorsensory rhythms w/o concurrent EMG activity (Vaughan et al., 1998)

• Squared biserial correlation coefficient, γ2

– For the classifier output and for the bandpass filtered and rectified EMG signals of the feedback sessions

– Occurrence of minimal EMG activity in some trials does not correlate with the EEG-based classifier

Investigating the Dependency

• CSP with Simultaneous Spectral Optimization– CSP strongly depends on the choice of the bandpass filter– Broadband filter for general choice– Subject-specific choices– Optimized spatial filters (usual CSP technique) + temporal finite im-

pulse response(FIR) filter : enhance the discriminability

• Significant superiority of the proposed CSSSP • The spatial and/or the spectral filter can be used for source

localization of the respective brain rhythms.

CSSSP

• Nonstationarities in EEG signals– How much of this nonstatioarity is reflected in the EEG features– How strongly is the classifier output affected– How can this be remedied

• The most serious shift occurred b/w the initial calibration measurement and online operation

• Shifts during online operation were largely compensated for by the CSP filters or the final classifier

• Simple adaption of classification bias successfully cured the problem.

Need for Adaptivity

Chapter 7

• Introduction

• Asynchronous Control

• EEG-Based Asynchronous Brain-Switches

• Asynchronous Control Design Issues

• Neil Squire Society– The only national not-for-profit organization in Canada– Neil Squire Brain Interface lab focused on BI system design specifically

for asynchronous control environments.

• Synchronized control environments where the system dictates the control of the user

• Robust multistate, asynchronous brain-controlled switch in the most natural manner in the real-world environments

Introduction

Asynchronous Control

• Output signal levels are changed or commands are issued only when control is intended– Intentional control (IC) / no control (NC) state– Remains neutral or unchanged during the NC state

• Asynchronous control– Characteristic of most real-world control applications– Most people expect from interface technology

Asynchronous Control

NC state; stable and unchanged IC; available for control

• The neutral or unchanging system output response during NC states

• Poor idling is indicated by false switch activations• How well BI transducers idle

– Rate of false activations or false positive error

• True and false activation rates as performance metrics

System Idling

With no gas (the NC state), the engine idles. Poor at idling

• Synchronized BI system– System-driven control strategy which cause significant user frustration

and fatigue

• Asynchronous TV controller– Simply change channels at any time they wish– The channel selection is stable when users are watching TV (NC state)

• Synchronous TV controller– Regularly poll the user to ask– Renders the changing of channels to specific periods

Synchronized BI System

• Two serious drawbacks– control decision regardless of whether the person is actually intending

control– User would need to wait for the system polling period to occur

Synchronized BI System

• Essence of the temporal control paradigms– Idle support and system availability

• Continuously available control– Always ready for the user to control

• Periodically available control– For Initial trial-based technology development– For restricting the signal processing complexity– Blocks a user’s attempt to control for the periods b/w control periods– “no control is possible at this time”

System availability

• Unintended action during NC states– Control paradigms that do not support idling– “Midas touch problem” by the eye-tracking community

• Four primary control paradigms– Constantly engaged mode : impractical– Asynchronous mode : most natural assistive device operation

Control Paradigms

• Specific signal processing algorithms to handle the NC state– Optimized TP rate and minimized FP rates– Trade-off b/w TP accuracy and FP rates → specific characteristics of

an application

• Brain-switch based on the outlier processing method(OPM)– Extract single-trial voluntary movement-related potentials (VMRPs)– TP rates greater than 90%– FP rates b/w 10 and 30% → limits OPM as an asynchronous switch– Multiposition asynchronous brain-switch

Asynchronous Brain-Switches

• Attempted VMRPs– Voluntary movement : existing and natural internal control system– “attempted” : subject with SCI attempt to move their fingers– By a very different neural mechanism

• The low-frequency asynchronous switch design (LF-ASD)– Relative power in the 1-4Hz band from ensemble VMRP increased– Focus on low frequency band– Wavelet analysis of the EEG signal over SMA and MI– Lower end of FP activations– TP rates of 30 to 78% during IC states with very low FP rates of 0.5 to

2% during NC

Asynchronous Brain-Switches

• Additional signal processing– EEG signal normalization– Switch output debounce– Feature set dimensionality reduction blocks– Increased the TP rate by an average of 33%

• Feature vectors navigate the feature space– System classification accuracy of more than 97%

• Levine and Huggins, Millán et al. (2004a), Yom-Tov and Inbar (2003), and Townsend et al. (2004)

Additional signal processing

• Very low FP rates during the NC state– High FP error rates cause undue user frustration– False activation appear uncontrollable to the user– Subjects would rather experience more trouble performing accurate

hits ( low TP rate)

• Evaluating asynchronous control– Receiver operating characteristic (ROC) curves– BI transducers with low FP error rates

Asynchronous Control Design Issue1

• “Tuning” the performance– ROC curve shows possible operating setups– by tuning various parameters

• Two-state LF-ASD brain-switch (on/off)– Scaling the relative magnitudes of NC state feature vectors vs. IC state

ones– FP rates under 1% during the time b/w FPs (30s or more)

Asynchronous BI Performance

• FPs typically clump together– Large periods of system idle time with free of FPs

• Switch-output jitter reduction methods– Switch debounce block– Improve error rates by reducing the FP jitter in the switch output– Trade-off : transducer availability– Debounce time ↑ → time the transducer is available ↓,

control time ↓

Switch-output jitter reduction

Borisoff et al. (2004)

• Intra-False Positive Rates– Dependent on the output classification rate of a transducer– FP rate of 1% with classification output every 1/16th sec →

FP every 1/16 * 100 = 6.3s

• Average time b/w errors greater than 30s– Reasonable design goal : FP rate of under 0.25%

• False activation rates : time rates w/ raw percentages

Asynchronous Control Design Issue2

• Temporal characteristics of NC and IC states– Environmental controllers : periods b/w the IC commands– Neural control of a robotic device : intercontrol times during periods

of intense usage

• Ubiquitous ON/OFF problem– Users with amputation have to turn the system on/off by themselves– Confirms user intent to turn the system to the awake mode– Eliminate FPs and require a simple commands

Asynchronous Control Design Issue3

• Costs associated with an FP– Multiple operating levels of the sleep mode– Full awake mode : sequence through higher modes, higher FP rates

and intentional command sequences

• How to structure tasks during different phases– Customization, training, and testing phases

Asynchronous Control Design Issue4

Sleep mode Awake mode

Low costs of FPs Easily corrected

High FP ratesLess intricate

• Initial customization– Accurate time-stamping for calibration and classifier– System-guided and system-paced tasks– Train and test in very similar control environments

• Apparatus necessary– Self guided and self-paced tasks → Self-report errors– Contamination of data – Accurate assessment of asynchronous system performance

Asynchronous Control Design Issue4

Thank you for your attention