computer vision techniques applied to eeg signal analysis ......waveform-based algorithms •...

Histogram of Gradient Orientation

Dr. Rodrigo Ramele

Computer Vision Techniques applied to EEG Signal Analysis

Brain Computer-Interfaces

Motivations

• Clinical and Physician Involvement

Yuste2017, Perdikis2014

• Practical, relevant, and invariant features that convey good-enough information.

“Can we mimic the physician approach to analyze and discriminate Electroencephalographic signals by automatic processing the shape

of the waveforms ?”

Waveform-Based Algorithms

• Nowadays clinical EEG still entails a visually interpreted test

• Permutation Entropy• Matching Pursuit• Slope Horizontal Chain Code

• Peak Picking/aEEG/Period Amplitude Analysis/…

Band&Pompe 2002,Mallat 1993,Alvarado-Gonzalez 2016

Histogram of Gradient orientation

Histogram of Gradient Orientation

Signal Preprocessing

Signal Segmentation

Signal Plotting

Keypoint Localization

Calculation of Histogram of Gradient Orientation

Signal Segment Localization

Signal Plotting: Digitalization

(0,0)

(Wx, Hy)

+

+

Signal Plotting: Interpolation

Bresenham 1965

Histogram of Gradient Orientation

Histogram of Gradient Orientation

Histogram of Gradient Orientation

Histogram of Gradient Orientation

h(θ, i, j) = ∑p

ωang(∠J(p) − θ) ωij (p − kp) J(p) ωij(v) = ω( 5 vx

Δs St− xi)ω( 5 vy

Δs Sv− yi)

ωang(α) =1

∑r=−1

ω( 8α2π

+ 8r)

St =λ Fs γt

Δs

Sv =ΔμV γ

Δs

Sx × Sy

SvSt

γ

γt

Lowe 2004, Vedaldi 2010

{TrilinearInterpolation ω(z) = max(0,1 − |z | )

Δs = 2 3 (4 + 1)

Sx = ⌊Δs St⌋ + 1Sy = ⌊Δs Sv⌋ + 1

Histogram of Gradient Orientation

h(θ, i, j) = ∑p

ωang(∠J(p) − θ) ωij (p − kp) J(p) ωij(v) = ω( 5 vx

Δs St− xi)ω( 5 vy

Δs Sv− yi)

ωang(α) =1

∑r=−1

ω( 8α2π

+ 8r)

St =λ Fs γt

Δs

Sv =ΔμV γ

Δs

Lowe 2004, Vedaldi 2010

{TrilinearInterpolation ω(z) = max(0,1 − |z | )

Histogram of Gradient Orientation

0 45 90 135 180 225 270 315Bin Angles

Num

ber o

f wei

ghte

d gr

adie

nts

Histogram of Gradient Orientation

1

234

56 7 8

9

101112

13

14 15 1617

181920

21

22 23 24

25

26272829

3031 32

33

343536

37

38 39 40

41

424344

45

46 47 48

49

505152

53

54 55 56

5758

59

128127126

125

124 123 122

121

120119118

117

116 115 114

113

112111110

109

108107

106

105

104103102

101

100

60

61

62 63 64

65

666768

69

70 71 72

73

747576

77

78 79 80

81

828384

85

86 87 88

89

909192

93

94 95 96

97

9899

The Naive Bayes Near Neighbour

Naive Bayes Near Neighbour

Boiman2008

Naive Bayes Near Neighbour

Q

∑i= 1

k

∑h= 1

Class Label 1 Class Label 2

[q(bpc)1 , q(bpc)

2 , ⋯, q(bpc)Q ]

L1 L2

Q

∑i= 1

k

∑h= 1

Template Dictionaries

Query ImageDescriptors

<L̂ = L1

(1) (1)

(1)

(2)

(2)

(3)

(3)

(4)

The Berger Rhythm

Alpha Waves Wiggles

Alpha Waves Wiggles

1 10 20 30 40 50 64

0.5

0.80.9

Channel

Acc

ura

cy

Af3 O1O2 Af40

0.7

Channel

Acc

ura

cy

EPOC Emotiv Consumer Grade EEG Device, 14 subj EEG MI NHS Physionet, Run1 vs Run2, 25 subj

Mu Rhythm

Mu Rhythm

Dataset 002-2014 BNCI-Horizon 2020 (Styrl, Scharer et al 2015)

Mu Rhythm

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P140

0.5

0.7

Participant

Acc

ura

cy

C3

Cz

C4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P140

0.5

0.7

Participant

Acc

ura

cy

C3

Cz

C4

Baseline vs. Hand Movement Baseline vs. Feet Movement

Mu Rhythm

Niedermayer’s book 2010

P300 Waveform

P300 Waveform: Single Trial Detection

P300 Wave

•Dataset I - P300 ALS Public Dataset• 8 subjects, 8 channels, Fs=256 Hz, 7 words, 5 letters, ISI-0.125 s, 10 epochs

•Dataset II - P300 for Healthy Subjects•Dataset III - P300 Pseudo-Real Dataset

• 8 subjects, 4 active, 4 passive•Dataset IV - P300 Dataset IIb BCI Competition II (2003)

• 1 subject, 64 channels, Fs=240 Hz, 73 trials, 42/31, ISI=0.25 s, 15 epochs

P300 Wave: Dataset I

κ

2 4 6 8 10Intensification Sequences

0

30

70

90

Perc

ent C

orr

ect

12345678

P300 Wave: Waveform Dictionary

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

0 0.5 1.0-505

P300 Wave: Dataset I & II

Riccio 2013, Mak 2012, Nijboer 2009

P300 Wave: Dataset I & II

P300 Wave: Dataset III

0 50 100 150 200 250-10

-8

-6

-4

-2

0

2

4

6

8

10

Mic

rovolts

Multichannel ERP Template

Ouyang 2017, Verleger 2000

https://codeocean.com/capsule/5299343/tree/v2

P300 Wave: Dataset III

Experiment 1: Letter identification, 2: jitter noise, 3: P3 peak amplitude noise

P300 Wave: Dataset III

Dataset IV - P300 Dataset IIb BCI Competition II (2003)

Signal Segment Localization

Signal Segment Localization

Devuyst 2010

Signal Segment Localization

• K-Complexes are Sleep Research Graphoelements

• Biomarkers to determine transition from REM to Stage 2

• PSR: Polysomnography Recordings• Very long recordings.

• - Agreement Score between experts: 67%

• Asynchronous/ Self-pace BCIs

κ

Signal Plot Keypoint Detector

• SIFT: Scale Invariant Feature Transform

• FAST: Features from Accelerated Segment Test

• SURF: Speeded-Up Robust Features

Scale Space Detection

Iijima 62Otsu 81Witkin 83

Koenderink 84

Lindeberg 94

Lowe 04

Holmstron 13

P300

Viergever 97

Viergever 97, Lindeberg 2010

Scale Space Detection

Viergever 97, Lindeberg 2010

Iijima 62Otsu 81Witkin 83

Koenderink 84

Lindeberg 94

Lowe 04

Holmstron 13

P300

Viergever 97

Multichannel Representation

HIST Multichannel Representation

HIST Multichannel Representation

Van de sande 2010

HIST

HIST

EnsembleClassifier

Fusion HIST Classifier

ConcatenateDescriptors

MultichannelHIST

HIST Multichannel Representation❖ Motor Imagery

❖ Laplacian Electrode Configuration

Wolpaw 2012

HIST Multichannel Representation

h(θ, i, j) = ∑p

ωang(∠J(p) − θ) ωij (p − kp) J(p)

Steyrl 2016

Cz

C3

C4

Pz

HIST Multichannel Representation

h(θ, i, j) = ∑p

ωang(∠J(p) − θ) ωij (p − kp) J(p)

Steyrl 2016

Cz

C3

C4

Pz

Conclussions❖ Good performance when waveforms can be visually detected.

❖ Objective metric for waveform shapes.

❖ Stability of the shape of the P300 waveform.

❖ The intelligible property of this feature extraction method could be valuable in clinical settings.

❖ Scale Space Theory could have a revival in terms of finding 1-D signal features.

Some follow-up

Thanks !