Slide 1

Human Cognition: Decoding Perceived, Attended, Imagined Acoustic Events and Human-Robot Interfaces

Slide 2

The Team Adriano Claro Monteiro Alain de Cheveign Anahita Mehta Byron Galbraith Dimitra Emmanouilidou Edmund Lalor Deniz Erdogmus Jim OSullivan Mehmet Ozdas Lakshmi Krishnan Malcolm Slaney Mike Crosse Nima Mesgarani Jose L Pepe Contreras- Vidal Shihab Shamma Thusitha Chandrapala

Slide 3

The Goal To determine a reliable measure of imagined audition using electroencephalography (EEG). To use this measure to communicate.

Slide 4

What types of imagined audition? Speech: Short (~3-4s) sentences The whole maritime population of Europe and America. Twinkle-twinkle little star. London bridge is falling down, falling down, falling down. Music Short (~3-4s) phrases Imperial March from Star Wars. Simple sequence of tones. Steady-State Auditory Stimulation 20 s trials Broadband signal amplitude modulated at 4 or 6 Hz

Slide 5

The Experiment 64 channel EEG system (Brain Vision LLC thanks!) 500 samples/s Each trial consisted of the presentation of the actual auditory stimulus (perceived condition) followed (2 s later) by the subject imagining hearing that stimulus again (imagined condition).

Slide 6

The Experiment Careful control of experimental timing. Perceived...2s... Imagined...2 s x 5... Break... next stimulus 4, 3, 2, 1, +

Slide 7

Data Analysis - Preprocessing Filtering Independent Component Analysis (ICA) Time-Shift Denoising Source Separation (DSS) Looks for reproducibility over stimulus repetitions

Slide 8

The hypothesis: EEG recorded while people listen to (actual) speech varies in a way that relates to the amplitude envelope of the presented (actual) speech. EEG recorded while people IMAGINE speech will vary in a way that relates to the amplitude envelope of the IMAGINED speech. Data Analysis: Hypothesis-driven.

Slide 9

Phase consistency over trials... EEG from same sentence imagined over several trials should show consistent phase variations. EEG from different imagined sentences should not show consistent phase variations. Data Analysis: Hypothesis-driven.

Slide 10

Actual speechImagined speech Consistency in theta (4-8Hz) band Consistency in alpha (8-14Hz) band

Slide 11

Data Analysis: Hypothesis-driven.

Slide 12

Red line perceived music Green line imagined music

Slide 13

Data Analysis - Decoding

Slide 14

r = 0.30, p = 3e-5r = 0.19, p = 0.01 Londons BridgeTwinkle Original Reconstruction

Slide 15

Data Analysis - SSAEP

Slide 16

4Hz 6Hz Perceived Imagined Data Analysis - SSAEP

Slide 17

Data Analysis Data Mining/Machine Learning Approaches:

Slide 18

Data Analysis Data Mining/Machine Learning Approaches:

Slide 19

SVM Classifier Input EEG data (channels time) : Concatenate channels: Group N trials: Input covariance matrix: Class Labels Predicted Labels

Slide 20

SVM Classifier Results Mean DA = 90% Decoding imagined speech and music: Mean DA = 90%Mean DA = 87%

Slide 21

Raw EEG Signal (500Hz data) DSS Output (Look for repeatability) DCT Output (Reduce dimensionality) DCT Processing Chain

Slide 22

DCT Classification Performance Percentage accuracy

Slide 23

Data Analysis Data Mining/Machine Learning Approaches: Linear Discriminant Analysis on Different Frequency Bands Music vs Speech Speech 1 vs Speech 2 Music 1 vs Music 2 Speech vs Rest Music vs Rest - results ~ 50 66%

Slide 24

Summary Both hypothesis drive and machine-learning approaches indicate that it is possible to decode/classify imagined audition Many very encouraging results that align with our original hypothesis More data needed!! In a controlled environment!! To be continued...