recent advances in an ssvep-based bcidesa/ssvep-masaki.pdf · 2020. 2. 27. · outline. ucsd, cogs...

Recent Advances in an SSVEP- based BCI

Masaki Nakanishi, PhDSwartz Center for Computational Neuroscience,

Institute for Neural Computation, University of California San Diego

COGS 189; February 28, 2020

UCSD, COGS 189, 02-28-2020

2

Masaki Nakanishi masaki@sccn.ucsd.edu

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

3

Masaki Nakanishi masaki@sccn.ucsd.edu

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

4

Masaki Nakanishi masaki@sccn.ucsd.edu

§ The brain’s electrical responses to repetitive visual stimulation

§ Sinusoidal-like waveforms at stimulus frequency and its harmonics

Steady-state VEP (SSVEP)

Vialatte et al., Prog. Neurobiol., 90(4): 418-438, 2010

UCSD, COGS 189, 02-28-2020

5

Masaki Nakanishi masaki@sccn.ucsd.edu

An SSVEP-based BCI

Wang et al., IEEE Eng. Med. Biol. Mag, 27(5): 64-71, 2008

UCSD, COGS 189, 02-28-2020

6

Masaki Nakanishi masaki@sccn.ucsd.edu

Comparison of EEG features for BCIs

Nicolas-Alonso et al., Sensors, 12: 1211-1279, 2012

UCSD, COGS 189, 02-28-2020

7

Masaki Nakanishi masaki@sccn.ucsd.edu

Comparison of EEG features for BCIs

Nicolas-Alonso et al., Sensors, 12: 1211-1279, 2012

→ 300~ bits/min→ Even higher

The performance of an SSVEP-based BCI has been dramaticallyimproved in the past decade.

UCSD, COGS 189, 02-28-2020

8

Masaki Nakanishi masaki@sccn.ucsd.edu

SSVEP-based BCI speller ten years ago

UCSD, COGS 189, 02-28-2020

9

Masaki Nakanishi masaki@sccn.ucsd.edu

- 20 healthy adults- 40 commands- 800 ms / input- 89.83 ± 6.07 %- 325.33 ± 38.17 bits/min

Chen et al., Proc. Nat. Acad. Sci. USA, 2015; Nakanishi et al., IEEE Trans. Biomed. Eng., 2018

High-speed BCI speller today

UCSD, COGS 189, 02-28-2020

10

Masaki Nakanishi masaki@sccn.ucsd.edu

Information transfer rate (ITR) [bits/min]

Accuracy of target identification

The number of targets Average time for a selection [s]

Cheng et al., IEEE Biomed. Eng., 49(10): 1181-1186, 2002

UCSD, COGS 189, 02-28-2020

11

Masaki Nakanishi masaki@sccn.ucsd.edu

BCI Performance improvement

Nakanishi et al., 2014

Chen et al., 2015

Nakanishi et al., 2018

UCSD, COGS 189, 02-28-2020

12

Masaki Nakanishi masaki@sccn.ucsd.edu

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

13

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Designing effective stimulus presentation- Display-based stimulation method- Challenges in designing visual stimulation

§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based method, Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

14

Masaki Nakanishi masaki@sccn.ucsd.edu

Display-based stimulus presentation§ Display-based stimulation is better than LED-based one because parameters (e.g.,

color, size, frequency) can be flexibly configured.§ Stimulus frequency can be produced by reversing the stimulus pattern between

white and black (e.g., ‘000111000111’).

UCSD, COGS 189, 02-28-2020

15

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Stability of the stimulation MUST be tested before experiments to make sure if the stimulation is precise.

§ Our laboratory uses a phototransistor to measure luminance changes.

Stability test of stimulation

UCSD, COGS 189, 02-28-2020

16

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Designing effective stimulus presentation- Display-based stimulation method- Challenges in designing visual stimulation

§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based method, Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

17

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.- e.g., 11 Hz cannot be presented under 60 Hz refresh rate

=> We solved it by proposing a frequency approximation method (Nakanishi et al., 2014a)

§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)- Higher frequency resolution required longer SSVEP data epochs to reliably classify- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.

=> We solved it by proposed hybrid frequency and phase tagging techniques(Nakanishi et al., 2014b; Chen et al., 2015)

Challenges in designing visual stimulation

UCSD, COGS 189, 02-28-2020

18

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.

- e.g., 11 Hz cannot be presented under 60 Hz refresh rate

=> We solved it by proposing a frequency approximation method (Nakanishi et al., 2014a)

§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)

- Higher frequency resolution required longer SSVEP data epochs to reliably classify

- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.

=> We solved it by proposed hybrid frequency and phase tagging techniques(Nakanishi et al., 2014b; Chen et al., 2015)

Challenges in designing visual stimulation

UCSD, COGS 189, 02-28-2020

19

Masaki Nakanishi masaki@sccn.ucsd.edu

Frequency approximation approach

Nakanishi et al., PLoS One, 9(6): e99235, 2014

f：Stimulus frequencyi：Frame index

UCSD, COGS 189, 02-28-2020

20

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.

- e.g., 11 Hz cannot be presented under 60 Hz refresh rate

=> We solved it by proposing a frequency approximation method (Nakanishi et al., 2014a)

§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)

- Higher frequency resolution required longer SSVEP data epochs to reliably classify

- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.

=> We solved it by proposed hybrid frequency and phase tagging techniques(Nakanishi et al., 2014b; Chen et al., 2015)

Challenges in designing visual stimulation

UCSD, COGS 189, 02-28-2020

21

Masaki Nakanishi masaki@sccn.ucsd.edu

Frequencies from differentfrequency ranges shouldnot co-exist in a system.

Amplitude responses in SSVEPs

Wang et al., IEEE Trans. Neural. Syst. Rehabil. Eng., 14(2): 234-239, 2006

UCSD, COGS 189, 02-28-2020

22

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Frequency which can be presented on a computer monitor was limited byits refresh rate- Impossible to present the frequencies by which the refresh rate cannot be divided.

- e.g., 11 Hz cannot be presented under 60 Hz refresh rate

=> We solved it by proposing a frequency approximation method (Nakanishi et al., 2014a)

§ Stimulus frequencies still need to be selected from a narrow range- SSVEPs have different amplitudes in three different frequency ranges (Wang et al., 2006)

- Higher frequency resolution required longer SSVEP data epochs to reliably classify

- e.g., 0.5 Hz resolution requires 2-s data; 0.2 Hz resolution requires 5-s data.

=> We solved it by proposed hybrid frequency and phase tagging techniques(Nakanishi et al., 2014b; Chen et al., 2015)

Challenges in designing visual stimulation

UCSD, COGS 189, 02-28-2020

23

Masaki Nakanishi masaki@sccn.ucsd.edu

Mixed frequency-phase modulation

Nakanishi et al., Int. J. Neural Syst., 24(6): 1450019, 2014

Φ：Initial phase

UCSD, COGS 189, 02-28-2020

24

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Stimulus design for a BCI speller

- 26 English alphabets, 10 digits, 4 symbols

- Frequency range : 8 – 15.8 Hz with an interval of 0.2 Hz

- Phase range : 0 – 2 ! with an interval of 0.35 !

Stimulus design for our speller application

Freq. (Hz)

Phase( )

>> HIGH SPEED BCI 8.0

0.00

8.2

0.35

8.4

0.70

8.6

1.05

8.8

1.40

9.0

1.75

9.2

0.10

9.4

0.45

9.6

0.80

9.8

1.15

10.0

1.50

10.2

1.85

10.4

0.20

10.6

0.55

10.8

0.90

11.0

1.25

11.2

1.60

11.4

1.95

11.6

0.30

11.8

0.65

12.0

1.00

12.2

1.35

12.4

1.70

12.6

0.05

12.8

0.40

13.0

0.75

13.2

1.10

13.4

1.45

13.6

1.80

13.8

0.15

14.0

0.50

14.2

0.85

14.4

1.20

14.6

1.55

14.8

1.90

15.0

0.25

15.2

0.60

15.4

0.95

15.6

1.30

15.8

1.65

Freq. (Hz)

Phase( )

>> HIGH SPEED BCI 8.0

0.00

8.2

0.35

8.4

0.70

8.6

1.05

8.8

1.40

9.0

1.75

9.2

0.10

9.4

0.45

9.6

0.80

9.8

1.15

10.0

1.50

10.2

1.85

10.4

0.20

10.6

0.55

10.8

0.90

11.0

1.25

11.2

1.60

11.4

1.95

11.6

0.30

11.8

0.65

12.0

1.00

12.2

1.35

12.4

1.70

12.6

0.05

12.8

0.40

13.0

0.75

13.2

1.10

13.4

1.45

13.6

1.80

13.8

0.15

14.0

0.50

14.2

0.85

14.4

1.20

14.6

1.55

14.8

1.90

15.0

0.25

15.2

0.60

15.4

0.95

15.6

1.30

15.8

1.65

>> HIGH SPEED BCI>> HIGH SPEED BCI

AA B C D E F G H

I J K L M N O P

Q R S T U V W X

Y Z 0 1 2 3 4 5

6 7 8 9 , . <

Nakanishi et al., IEEE Trans. Biomed. Eng., 65(1): 104-112, 2018

UCSD, COGS 189, 02-28-2020

25

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Designing effective stimulus presentation- Display-based stimulation method- Challenges in designing visual stimulation

§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based / Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

26

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Scalp EEG recordings can be modeled as an instantaneous linear combination ofcortical source signals.

EEG mixture model and spatial filtering

! = #$ = #%& = & ((ℎ*+ #% = 1)§ Source signals can be estimated as:

Spatial filter

UCSD, COGS 189, 02-28-2020

27

Masaki Nakanishi masaki@sccn.ucsd.edu

Comparison of spatial filtering techniquesApproach Hypothesis

Average combination (AVG) The SSVEP manifests globally over the scalp without phase variations.

Principal component analysis (PCA) The SSVEP signal is uncorrelated from the background EEG.

Independent component analysis (ICA) The SSVEP signal is statistically independent from the background EEG.

Minimum energy combination (MEC) The optimal spatial filter results from minimizing an estimate of the noises.

Canonical correlation analysis (CCA) The optimal spatial filter can maximize correlation between SSVEPs and computer-generated SSVEP models.

Task-related component analysis (TRCA) The optimal spatial filter can maximize inter-trial correlation

Garcia-Molina et al., IEEE EMBS Conf. Neural Eng., 2011

UCSD, COGS 189, 02-28-2020

28

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Task-related component analysis (TRCA) finds a linear coefficient which maximizes the reproducibility across trials

TRCA-based spatial filtering

UCSD, COGS 189, 02-28-2020

29

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Assume there are two source signals: 1) task-related signal ! " ∈ ℝ; 2) task-unrelated signal %(") ∈ ℝ.

§ A linear generative model of observed multi-channel signal ((") ∈ ℝ)* is assumed as:

Problem setting for TRCA

+, " = ./,,! " + .2,,% " , 3 = 1,2, … , 78

§ The problem is to recover the task-related signal ! " from a linear sum of observed signals ((") as:

9 " =:,;/

)*<,+, " =:

,;/

)*<,./,,! " + <,.2,,%(")

§ Ideally, the problem has a solution of ∑,;/)* <,./,, = 1 and ∑,;/)* <,.2,, = 0, leading to

the final solution 9 " = !(")

UCSD, COGS 189, 02-28-2020

30

Masaki Nakanishi masaki@sccn.ucsd.edu

§ The problem can be solved by inter-trial covariance maximization§ The h-th trial of EEG signal and the estimated task-related component are

described as ! " and # " , ℎ = 1,2, … , *+.§ The covariance between ℎ,-th and ℎ--th trials of # is described as:

Problem solution using TRCA (1/2)

."/,"0 = Cov # "/ , # "0 = 45/,506,

7895/950Cov :5/

"/ , :50"0

§ All possible combination of trials are summed as:

4"/,"06,"/;"0

7<."/,"0 = 4

"/,"06,"/;"0

7<4

5/,506,

7895/950Cov :5/

"/ , :50"0 = =>?=

UCSD, COGS 189, 02-28-2020

31

Masaki Nakanishi masaki@sccn.ucsd.edu

§ To obtain a finite solution, the variance of ! " is constrained as:

Problem solution using TRCA (2/2)

Var !(") = )*+,*-./

012*+2*-Cov 6*+, 6*- = 7897 = 1

§ The constrained optimization problem can be solved using the method of Lagrange multiplier as:

; 7, < = 78=7 − < 7897− 1?@ 7,A?7 = =7 − < 97 = 0

§ The optimal coefficient vector is obtained as the eigenvector of the matrix 9C/=

UCSD, COGS 189, 02-28-2020

32

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Designing effective stimulus presentation- Display-based stimulation method- Challenges in designing visual stimulation

§ Proposing advanced signal processing- Preprocessing – Spatial filtering- Target identification algorithm – Model-based method, Template-based method

Our achievements

UCSD, COGS 189, 02-28-2020

33

Masaki Nakanishi masaki@sccn.ucsd.edu

Model-based methods§ Canonical correlation analysis (CCA)-based method- CCA takes two sets of multi-dimensional variables as an input

- CCA finds a pair of linear coefficients that maximize the correlation between two variablesprojected onto the coefficients.

- Target stimulus frequency can be identified by finding the model maximizing correlation.

Multi-channel EEG

Computer-generated SSVEP models

Lin et al., IEEE. Trans. Biomed. Eng., 54(6): 1172-1176, 2007

UCSD, COGS 189, 02-28-2020

34

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Correlation between scalp EEG and individual templates after spatial filtering.§ Individual template can be obtained by averaging training data across trials

Template-based method

Nakanishi et al., Int. J. Neural Syst., 24(6): 1450019, 2014

UCSD, COGS 189, 02-28-2020

35

Masaki Nakanishi masaki@sccn.ucsd.edu

Performance comparison (Template-based)§ 40 visual stimuli were presented on a

23.6-inch LCD monitor.§ EEG data were recorded from 35

subjects with 9 electrodes placed over parietal and occipital areas.

§ The experiment consisted of 6 trials,in which the subjects gazed at one of the stimuli for 5 s.

UCSD, COGS 189, 02-28-2020

36

Masaki Nakanishi masaki@sccn.ucsd.edu

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

37

Masaki Nakanishi masaki@sccn.ucsd.edu

UCSD, COGS 189, 02-28-2020

38

Masaki Nakanishi masaki@sccn.ucsd.edu

Glaucoma -緑内障 / 青光眼

Normal optic nerve

Glaucomatous optic nerve

Retinal ganglion cells

Weinreb, et al., JAMA, 2014

UCSD, COGS 189, 02-28-2020

39

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Early detection- Glaucomatous visual field losses progress without

noticeable initial symptoms, resulting frequently in latediagnosis or late detection of progressive damage.

Challenges in glaucoma assessment

§ Lack of objectivity and portability- Conventional assessment methods have significant

drawbacks such as large test-retest variability,cumbersome clinic-based setting.

UCSD, COGS 189, 02-28-2020

40

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Previous studies showed a good correspondence between the results of conventional visualfield assessment and the amplitude of SSVEPs

§ Current data recordings are time consuming and uncomfortable for patients due to skinpreparation and gel application

Glaucoma assessment using VEPs

Hood et al., Vis. Neurosci., 2000

UCSD, COGS 189, 02-28-2020

41

Masaki Nakanishi masaki@sccn.ucsd.edu

Glaucoma assessment using VEPs

Hood et al., Vis. Neurosci., 2000

§ Previous studies showed a good correspondence between the results of conventional visualfield assessment and the amplitude of SSVEPs

§ Current data recordings are time consuming and uncomfortable for patients due to skinpreparation and gel application

UCSD, COGS 189, 02-28-2020

42

Masaki Nakanishi masaki@sccn.ucsd.edu

Glaucoma diagnosis with a portable BCINeuromonitoring unit

- High-precision data acquisition unit- Bluetooth + Wi-Fi modules

Electrodes- 6 EEG + 2 EOG sensors- Sampling rate at 500 Hz

VR Display- Visual stimuli were programed by Unity

Pz

PO4PO3

OzO2

O1

Dry electrodes

Visual stimulus

UCSD, COGS 189, 02-28-2020

43

Masaki Nakanishi masaki@sccn.ucsd.edu

§ Visual stimuli eliciting multi-focal SSVEPs in 20 sectors over the 35-degree field of vision were presented on the nGoggle’s display

§ Stimulus frequencies: 8 - 11.8 Hz with an interval of 0.2 Hz

Stimulus design

UCSD, COGS 189, 02-28-2020

44

Masaki Nakanishi masaki@sccn.ucsd.edu

Demographical characteristicsGlaucoma

(n = 62 eyes of33 subjects)

Control (n = 30 eyes of

17 subjects)

P-Value

Age, years 68.2 ± 11.0 66.1 ± 9.9 0.57

Gender, female, n (%) 8 (47) 16 (48) 0.92

Race, n (%) 0.50

White 19 (58) 9 (53)

Black 12 (36) 8 (47)

Asian 2 (6) 0 (0)

SAP 24-2 MD, dB -4.0 (-12.7 to -1.8) -0.6 (-2.4 to 1.0) < 0.001

SAP 24-2 PSD, dB 4.7 (2.2 to 9.9) 1.9 (1.4 to 3.0) < 0.001

SSVEP CCA ! 0.289 ± 0.020 0.334 ± 0.024 < 0.001

* SAP: Standard automated perimetry; MD: Mean deviation; PSD: Pattern standard deviation;CCA: Canonical correlation analysis

Nakanishi et al., JAMA Ophthalmol. 2017

UCSD, COGS 189, 02-28-2020

45

Masaki Nakanishi masaki@sccn.ucsd.edu

Demographical characteristicsGlaucoma

(n = 62 eyes of33 subjects)

Control (n = 30 eyes of

17 subjects)

P-Value

Age, years 68.2 ± 11.0 66.1 ± 9.9 0.57

Gender, female, n (%) 8 (47) 16 (48) 0.92

Race, n (%) 0.50

White 19 (58) 9 (53)

Black 12 (36) 8 (47)

Asian 2 (6) 0 (0)

SAP 24-2 MD, dB -4.0 (-12.7 to -1.8) -0.6 (-2.4 to 1.0) < 0.001

SAP 24-2 PSD, dB 4.7 (2.2 to 9.9) 1.9 (1.4 to 3.0) < 0.001

SSVEP CCA ! 0.289 ± 0.020 0.334 ± 0.024 < 0.001

* SAP: Standard automated perimetry; MD: Mean deviation; PSD: Pattern standard deviation;CCA: Canonical correlation analysis

Nakanishi et al., JAMA Ophthalmol. 2017

UCSD, COGS 189, 02-28-2020

46

Masaki Nakanishi masaki@sccn.ucsd.edu

Assessment of visual field deficits

Nakanishi et al., JAMA Ophthalmol. 2017

UCSD, COGS 189, 02-28-2020

47

Masaki Nakanishi masaki@sccn.ucsd.edu

Diagnostic ability

AUC: 0.92 (95%CI: 0.88 - 0.96)

AUC: 0.81 (95%CI: 0.72 - 0.90)

Nakanishi et al., JAMA Ophthalmol. 2017

UCSD, COGS 189, 02-28-2020

48

Masaki Nakanishi masaki@sccn.ucsd.edu

1. Introduction- Steady-state visual evoked potentials (SSVEPs)- An SSVEP-based BCI- Current status

2. Our contributions to the field- Effective visual stimulus presentation- Advanced signal processing

3. Clinical applications- Communication for ALS patients- Glaucomatous visual field assessment

4. Summary

Outline

UCSD, COGS 189, 02-28-2020

49

Masaki Nakanishi masaki@sccn.ucsd.edu

§ The performance of an SSVEP-based BCI has been significantly improved in the past decade.

§ SCCN has contributed to the improvement.- Designing effective visual stimulation (frequency approximation, Mixed freq/phase tagging)- Proposing advanced signal processing (TRCA, Template-based target identification)- Clinical applications (Communication support for ALS patients, Glaucoma detection)

Summary

We are looking for volunteers that would like to participate in our experiments. Each participant will be paid at a rate of $15/hour. Please contact Nicole Wells (scheduling@sccn.ucsd.edu) for more info.

UCSD, COGS 189, 02-28-2020

50

Masaki Nakanishi masaki@sccn.ucsd.edu

Thank you for your kind attention