sonification of fmri data nik sawe music 220c. overview phd studies assess decision-making on...
TRANSCRIPT
![Page 1: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/1.jpg)
Sonification of fMRI Data
Nik SaweMusic 220C
![Page 2: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/2.jpg)
Overview
• PhD studies assess decision-making on environmental issues through neuroimaging
• Neural activation suggests underlying physiological bases for framing effects, heuristics, affect (emotion) and their impact on decision-making
![Page 3: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/3.jpg)
How We Image the Brain
• Functional MRI allows us to take realtime pictures of the brain’s response to stimuli.
• Using headsets and hand input devices, can present subjects with a wide range of tasks.
![Page 4: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/4.jpg)
The BOLD Signal
• BOLD: Blood Oxygenation Level-Dependent
• fMRI evaluates brain activity indirectly, by measuring changes in the local amount of oxygenated blood
• Complex regressions account for fluctuations due to heart rate, breathing, etc.
• Validity confirmed through optogenetics
![Page 5: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/5.jpg)
Motivations for Sonification
• Can hear patterns of activation that would be less obvious through visualization of time courses
![Page 6: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/6.jpg)
Motivations for Sonification
• May be able to hear “conversations” between different brain regions that would be less obvious through traditional neuroimaging analyses
• Intuitive level of interpretation that may provide clues for further analytic techniques
![Page 7: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/7.jpg)
Limitations of fMRI
• Poor temporal resolution– One pass through each brain region
every 1-2 seconds (most often 2)
![Page 8: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/8.jpg)
Limitations of fMRI
• Poor temporal resolution– One pass through each brain region
every 1-2 seconds (most often 2)
• For most study designs, need many repeated trials in one person to get an accurate read
![Page 9: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/9.jpg)
Translatable fMRI Outputs
![Page 10: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/10.jpg)
Sonification Methodology
Built in R from a simple initial formula– Pitch: 128 * [(Xi – Xl)/(Xh-Xl)]– Velocity: 128*Pi
Xi : signal at timepoint iXh: maximum signalXl: minimum signalPi: a given network's proportional
contribution to the total signal strength of all sampled networks at timepoint i
![Page 11: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/11.jpg)
Sonification Methodology
Use these new values as downstream MIDI values, convert to MIDI file via Java
First trial: utilized data from one subject in my first study (environmental philanthropy to save parks threatened with potentially destructive land use development)
Used 3 networks: attentional, visual, default mode network
![Page 12: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/12.jpg)
Visual Cortex Quartet
Final project: Sampled from the visual cortex as subject undergoes retinotopy
![Page 13: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/13.jpg)
Sonification Methodology
Program had several stages: • Scale converter: created array of
MIDI values based on desired scale • Instrument filter: selected valid (in
scale range) notes for a given instrument
• Signal to MIDI converter: Gated signals below a threshold value (5%) and did not play them
![Page 14: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/14.jpg)
Sonification Methodology
• Velocity based upon relative prominence of the voxel signal given other voxels’ activity
• Duration based on arbitrary equation of:– ((128-note value)+velocity)/20
![Page 15: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/15.jpg)
The Next Step
• Scan whole brain while watching a silent film
• May obtain complementary EEG data
• Will have PCA networks to work with, as well as a wealth of regions
• Signals do not all have to be pitch modulation
![Page 16: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/16.jpg)
Mapping Ideas
• Activity in the attendant PCA network helps define duration and velocity for each region, based on its relative contribution
• Talairach (spatial) coordinates define surround sound mapping
![Page 17: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/17.jpg)
Mapping: Anterior Insula
• Handles “negative arousal” / response to physiologically as well as morally aversive stimuli
• Control how discordant the note selection is in other regions
![Page 18: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/18.jpg)
![Page 19: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/19.jpg)
Nucleus Accumbens
• Handles “positive arousal”/reward/approach behavior
• Control weighting towards major scales
• May be able to make a balancing equation of AI vs Nacc
![Page 20: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/20.jpg)
Mapping: Amygdala
• Fear/apprehension/anxiety region• Control tempo, accelerating at tense
moments• Control percussive
elements• Trigger clusters
![Page 21: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/21.jpg)
Mapping: Fusiform Gyrus
• Recognizes faces: triggering of voice samples?
![Page 22: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/22.jpg)
Parahippocampal Gyrus
• Spatial/landscape encoding• Spatial manipulation of
samples/Doppler?
![Page 23: Sonification of fMRI Data Nik Sawe Music 220C. Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation](https://reader030.vdocuments.mx/reader030/viewer/2022032607/56649edb5503460f94beb307/html5/thumbnails/23.jpg)
Incorporation of EEG
• Since temporal resolution is only 2 sec passes, would be good to have variation that decides interleaving of notes
• This interpolation can be decided by activity in relevant EEG signals