a similarity retrieval system for multimodal functional brain ......codebook and encode each subject...
TRANSCRIPT
![Page 1: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/1.jpg)
A Similarity Retrieval System for Multimodal Functional Brain
Images
Rosalia F. TungarazaAdvisor: Prof. Linda G. Shapiro
Ph.D. Defense
Computer Science & EngineeringUniversity of Washington
![Page 2: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/2.jpg)
Functional Brain Imaging
Study how the brain works
Imaging while subject performs a task
Image represents some aspect of the brain e.g.
fMRI: brain blood oxygen level
ERP: scalp electric activity
![Page 3: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/3.jpg)
Motivation
3
Given a database of functional brain images from various subjects, cognitive tasks, and image modality.
Database users need to retrieve similar images
A system that can automatically perform this retrieval will reduce amount of time and effort users spend during this task
![Page 4: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/4.jpg)
Contributions
1. Created a similarity retrieval system for multimodal brain images
I. fMRI, ERP, and combined fMRI-ERP
II. User interface
2. Developed feature extraction methods for fMRIand ERP data
3. Developed pair-wise similarity metrics
4. Simulated human expert similarity scores
![Page 5: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/5.jpg)
Outline
Background fMRI ERP Existing Similarity Retrieval Systems for
these modalities
Feature Extraction Process Similarity Metric User Interface Retrieval Performance Simulate Human Expert
5
![Page 6: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/6.jpg)
Functional Magnetic Resonance Imaging (fMRI)
6
A non-invasive brain imaging technique
Records blood oxygen level in brain
While imaging, subject performs a task
![Page 7: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/7.jpg)
fMRI Statistical Images
Statistical Analysis
Voxel Thresholding
![Page 8: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/8.jpg)
Event-Related Potentials (ERP)
8
A non-invasive brain imaging technique
Records electric activity along scalp
While imaging, subject performs a task
@ 2004 by Nucleus Communications, Inc.
![Page 9: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/9.jpg)
ERP Source Localization
9
Researchers want to identify the electric activity and its source for each electrode
But, multiple sources for each electrode
LORETA approximates anatomic locations of sources
![Page 10: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/10.jpg)
Comparison of fMRI and ERP Data
10
fMRI ERP
Spatial
resolution Good (in mm) undefined/poor
Temporal
resolution Poor (in sec) Excellent (in msec)
![Page 11: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/11.jpg)
Similarity Retrieval Systems for fMRI Images
11
![Page 12: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/12.jpg)
Similarity Retrieval Systems for ERP Images
12
No relevant literature found
![Page 13: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/13.jpg)
Similarity Retrieval Systems for Combined fMRI-ERP Images
13
No relevant literature found
![Page 14: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/14.jpg)
Outline
Background
Feature Extraction Process fMRI features ERP features
Similarity Metric User Interface Retrieval Performance Simulate Human Expert
14
![Page 15: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/15.jpg)
Threshold
Perform connected component analysis
Perform clustering
Approximate cluster centroids
Compute region vectors
Original database
fMRI Feature Extraction
k=3
k=2
k=1
1. Centroid2. Avg Activation Value 3. Var Activation Value4. Volume5. Avg Distance to
Centroid6. Var of those Distances
![Page 16: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/16.jpg)
Select time-segment
Compute voxel-wise statistically significant difference between means
Threshold
ERP Feature Extraction
Compute feature(X,Y,Z)
positions of retained voxels
![Page 17: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/17.jpg)
Outline
Background Feature Extraction Process
Similarity Metric Summed Minimum Distance Similarity Score for Combined fMRI-ERP Images
User Interface Retrieval Performance Simulate Human Expert
17
![Page 18: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/18.jpg)
Summed Minimum Distance (SMD) for fMRI and ERP Images
18
Subject Q Subject T
Q2T =
SMD = (Q2T+T2Q) / 2
![Page 19: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/19.jpg)
Sample SMD Scores
19
SMD
![Page 20: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/20.jpg)
Similarity Score for Combined fMRI-ERP Images
SIM(i,j) = αSMDfMRI(i,j) + (1-α)SMDERP(i,j)
![Page 21: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/21.jpg)
Outline
Background Feature Extraction Process Similarity Metric
User Interface
Retrieval Performance Simulate Human Expert
21
![Page 22: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/22.jpg)
GUI: Front Page
![Page 23: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/23.jpg)
GUI: Retrievals with SMD Scores
23
SMD
![Page 24: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/24.jpg)
GUI: Query-Target Activations (fMRI)
![Page 25: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/25.jpg)
GUI: Query-Target Activations (ERP)
![Page 26: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/26.jpg)
Outline Background Feature Extraction Process Similarity Metric User Interface
Retrieval Performance Data Sets fMRI Retrieval Performance ERP Retrieval Performance Combined fMRI-ERP Retrieval Performance
Simulate Human Expert26
![Page 27: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/27.jpg)
Data Sets for fMRI Retrievals
27
Central-Cross -- 24 subjects (Face Recognition)
AOD -- 15 subjects (Sound Recognition)
SB -- 15 subjects (Memorization)
Checkerboard -- 12 subjects (Face Recognition)
d g f p g n
![Page 28: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/28.jpg)
Data Set for ERP Retrievals
28
View Human Faces (Face Up) -- 15 subjects
View Houses (House Up)-- 15 subjects
… …
![Page 29: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/29.jpg)
Data Set for Combined fMRI-ERP Retrievals
29
ERP: same data set as used in ERP retrieval
fMRI: • Task: Face recognition using a house up background
• Same subjects and images as data set for ERP retrieval
![Page 30: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/30.jpg)
fMRI Retrieval Performance
30
1. RFX Retrievals
2. Individual Brain Retrieval
3. Testing Group Homogeneity
4. Feature Selection
![Page 31: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/31.jpg)
fMRI Retrieval Score
31
Perfect score : Retrieval Score = 0
Random score: Retrieval Score ~ 0.5
Worst score: Retrieval Score = 1
![Page 32: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/32.jpg)
fMRI Individual Brain Retrievals
32
Use individual brain as query
Mean Retrieval Scores
(Top 6% activated voxels)
Checkerboard 0.09
SB 0.16
Central-Cross 0.21
AOD 0.26
![Page 33: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/33.jpg)
Testing Group Homogeneity for fMRI
33
CB
AOD
Central-Cross
SB
![Page 34: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/34.jpg)
ERP Retrieval Performance
34
![Page 35: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/35.jpg)
Subject #8 RetrievalsTo
p
Ret
riev
als
Bo
tto
m
Ret
riev
als
![Page 36: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/36.jpg)
Combined fMRI-ERP Retrieval
SIM(i,j) = αSMDfMRI(i,j) + (1-α)SMDERP(i,j)
α = 0.0, ERP only
α = 1.0, fMRI onlyα = 0.6
α = 0.3
![Page 37: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/37.jpg)
Outline Background Feature Extraction Process Similarity Metric User Interface Retrieval Performance
Simulate Human Expert Simulation Method Data Set Testing Function Performance
37
![Page 38: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/38.jpg)
Simulate Human Expert
Current retrieval system requires some expert knowledge
Estimate a function to generate similarity scores with high correlation to expert scores
![Page 39: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/39.jpg)
Simulation Method
1. Uniform feature representation: create codebook and encode each subject
2. Concatenate the codebook features for each pair of subjects
3. Create eigenfeatures
4. Estimate a function
5. Test function performance
![Page 40: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/40.jpg)
S1 S2 S3
S1 S2 S3
Create the reference brain (codebook)
Encode Images
1. Uniform Feature Representation
![Page 41: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/41.jpg)
2. Concatenate Codebook Features
XYZ CentroidA Avg Activation Value VA Var Activation ValueS Size (Volume)D Avg Distance to CentroidVD Var of those Distances
![Page 42: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/42.jpg)
3. Create Eigenfeatures
![Page 43: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/43.jpg)
4. Estimate a Function
Linear function using linear regression
Non-linear function using generalized regression neural networks (GRNN)
S1,S1
S1,S2
S1,S3
S3,S3
… … … … … … … …
0.00
0.30
0.90
0.00
…
![Page 44: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/44.jpg)
5. Test Function Performance
The Pearson Correlation Coefficient (CC)
The Average Absolute Error (A-ABSE)
The Root Mean Square Error (RMSE)
![Page 45: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/45.jpg)
Data Set
fMRI data (Central-Cross)
-- 23 subjects
-- Face Recognition task
+Human Expert Generated Pair-wise Similarity Matrix
![Page 46: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/46.jpg)
Overall Function Performance
![Page 47: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/47.jpg)
Feature SelectionBad
Good
Good
Bad
Good
Bad
XYZ CentroidA Avg Activation Value VA Var Activation ValueS Size (Volume)D Avg Distance to CentroidVD Var of those Distances
![Page 48: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/48.jpg)
Contributions
1. Created a similarity retrieval system for multimodal brain images
I. fMRI, ERP, and combined fMRI-ERP
II. User interface
2. Developed feature extraction methods for fMRIand ERP data
3. Developed pair-wise similarity metrics
4. Simulated human expert similarity scores
![Page 49: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/49.jpg)
Future Direction
Add more modalities to the system
Obtain more expert scores for function
estimation
Obtain more data
Develop similarity metrics other than
SMD
![Page 50: A Similarity Retrieval System for Multimodal Functional Brain ......codebook and encode each subject 2. Concatenate the codebook features for each pair of subjects 3. Create eigenfeatures](https://reader036.vdocuments.mx/reader036/viewer/2022071501/611fe25efceb4111cb547434/html5/thumbnails/50.jpg)
Acknowledgements
50
Prof. Linda Shapiro
Prof. James Brinkley
Prof. Jeffrey Ojemann
Prof. John Kramlich
NSF grant number DBI-0543631
Computer Vision Lab
Collaborators in Radiology department
Practice-talk participants