Decoding Seen and Attended Edge Orientation and Motion Direction

from the Human Brain Activity Measured by functional Magnetic

Resonance Imaging (fMRI)

Presented by: Arash Ashari

1

Kamitani, Y., and Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685.Kamitani, Y., Tong, F. (2006). Decoding seen and attended motion directions from activity in the human visual cortex. Current Biology, 16: 1096-1102.

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

2

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

3

MRI – Magnetic Resonance Imaging

Using Nuclear Magnetic Resonance (NMR) technology, magnetic field influence the nucleus of hydrogen.

The MRI transmit Radio Frequency (RF) wave to the nucleus, and measures changes in magnetic field.

The RF changes according to the chemical structure of the tissue. The computerized images give a detailed anatomical view of the

organ.

High Spatial ResolutionHigh Spatial Resolution Used in brain structure research Used in brain structure research

and localization of brain-tumor and localization of brain-tumor

http://www.bm.technion.ac.il/courses/335014/projects04/mid_term_presentations_05/Mapping_visual_cortex.ppt

fMRI –Functional MRI The new neuroimaging method for probing the intact human brain. The technique is based on:

• In neural activity an additional supply of oxygenated blood is delivered. • Oxygenated Hemoglobin (Hb) is magnetically transparent (diamagnetic).• Deoxygenated Hb is not transparent (paramagnetic).

The change in the ratio of oxygenated to deoxygenated can be detected in the MR signal.

This signaling mechanism is the Blood Oxygen Level-Dependent (BOLD)

fMRI-Characteristics It measures changes in the subject in real time, without external Indicator. The exam can be repeated many times, without any harm. High Spatial Resolution [mm] Relatively High Temporal Resolution [Sec] High SNR allows to measure the size of differences, not just their presence or

absence.

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

7

Visual Cortex

• The term visual cortex refers to the primary visual cortex/V1 and extra-striate visual cortical areas such as V2, V3, V4, and V5/MT.

• The primary visual cortex, V1, receives information directly from the lateral geniculate nucleus. V1 transmits information to two primary pathways, called the dorsal stream and the ventral stream.

Primary Visual Pathways

– The dorsal stream begins with V1, goes through Visual area V2, then to the dorsomedial area and Visual area MT and to the posterior parietal cortex. The dorsal stream, sometimes called the "Where Pathway", is associated with motion, representation of object locations, and control of the eyes and arms.

– The ventral stream begins with V1, goes through visual area V2, then through visual area V4, and to the inferior temporal cortex. The ventral stream, sometimes called the "What Pathway", is associated with form recognition and object representation. It is also associated with storage of long-term memory.

http://en.wikipedia.org/wiki/Visual_cortex

Visual Perception

• It is commonly assumed that human visual perception is based on the neural coding of fundamental features, such as Orientation, Color, Motion and so forth.

• So it can be hypothesized that functional neuroimaging (fMRI) identifies brain areas that show robust responses to visual orientation and motion.

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

11

Orientation Decoder

Subjects views one of eight possible stimulus orientations while activity is monitored in early visual areas (V1−V4 and MT+) using standard fMRI procedures (3T MRI scanner, spatial resolution 3*3*3 mm). For each 16-s 'trial' or stimulus block, a square-wave annular grating is presented at the specified orientation (0, 22.5, ..., 157.5°), and flashes on and off every 250 ms with a randomized spatial phase to ensure that there is no mutual information between orientation and local pixel intensity.

Orientation decoding accuracy

fMRI activity patterns in the human visual cortex are sufficiently reliable to predict what stimulus orientation the subject is viewing on individual trials.

Ensemble fMRI activity in areas V1/V2 led to precise decoding of which of the eight orientations the subject saw on individual stimulus trials.

Root mean squared error (RMSE) between the true and the predicted orientations: 17.9°, 21.0°, 22.2° and 31.2°, respectively for subjects S1−S4

The ability to extract robust orientation information from ensemble fMRI activity allows us to compare orientation selectivity across different human visual areas. Orientation selectivity is most pronounced in early areas V1 and V2, and declines in progressively higher visual areas. Unlike areas V1 through V4, human area MT+ showed no evidence of orientation selectivity; consistent with the idea that this region is more sensitive to motion than to stimulus form.

Orientation decoding accuracy across visual areas

Source of orientation Information

The orientation preference of individual voxels on the flattened surface of left ventral V1 and V2 for subjects S2 and S3.

Voxel colors depict the orientation detector for which each voxel provides the largest weight.

Mind-Reading of Attended Orientation

The robust effects found in V1 and V2 suggest that top-down voluntary attention acts very early in the processing stream to bias orientation-selective signals when two competing stimuli are entirely overlapping.

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

17

Direction decoder

1. The decoder receives fMRI voxel intensities, averaged for each 16-s stimulus block, as inputs.

2. The next layer consisting of “linear ensemble direction detectors” calculates the weighed sum of voxel inputs.

3. Voxel weights are optimized using a statistical learning algorithm applied to independent training data, so that each detector’s output become larger for its direction than for the others.

4. The direction of the most active detector is used as the prediction of the decoder.

Comparison of Orientation and Direction Selectivity

Mind-Reading of Attended Direction

Feature-based attention can alter the strength of direction-selective responses throughout the visual pathway, with top-down bias effects emerging at very early stages of visual processing. ( Attention should bias the pattern of neural activity to more closely resemble the activity pattern that would be induced by the attended feature alone.)

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

21

Data Analysis

1. Three dimensional (3D) motion correction2. Linear trend removal (using automated image registration software)3. Sorting the voxels according to the responses to the visual areas.4. Shifting the data 4 s to account for the hemodynamic delay5. Averaging the fMRI signal intensity of each voxel for each 16 s

stimulus trail.6. Normalization relative to average of the entire time course within

each run7. Labeling the activity patterns according to corresponding stimulus

Orientation or Direction.8. Classification (using cross-validation for train and test)

Classification

The classification is done by a linear ensemble Orientation/Direction detector:θk : preferred Orientation/direction

x=(x1, x2,..., xd) : voxel inputs

Linear detector function: where wi is the “weight” of voxel i, and w0 is the “bias”

Classification (con…)

Linear Support Vector Machine (SVM) is used to calculate a linear discriminant function for the pairs of all orientation/direction;

Then the pairwise discriminant functions comparing θk and the other directions are simply added to yield the linear detector function:

Outline

• functional Magnetic Resonance Imaging (fMRI)

• Visual Cortex• Orientation Decoder• Direction Decoder• Data Analysis• fMRI- Accomplishments and Future works

25

fMRI- Accomplishments and Future works Identification of the position of several

retinotopically organized visual areas. Measurement of the retinotopic

organization within these areas. Identification of the location of

orientation/motion-sensitive region. Measurement of the responses

associated with contrast, color and motion.

Measurement of localized deficits in activity in subjects with cortical damage.

Measurement of the effects of attentional modulation on visually evoked responses.

High inter-subject variability of functional activity and sheer dimensionality of fMRI data are the two major problems in fMRI data classification.

Another Active Project on Mind-Reading

• Classifying cognitive processes based on fMRI images: How can we use statistical machine learning methods to classify the hidden cognitive process of a human subject, based on their observed fMRI data?

• In this fMRI study Mitchll et al trained their algorithms to decode whether the words being read by a human subject are about tools, buildings, food, or several other semantic categories. The trained classifier is 90% accurate, for example, discriminating whether the subject is reading words about tools or buildings.

http://www.cs.cmu.edu/afs/cs/project/theo-73/www/index.html

Thank you

Much remains to be learned about how the human brain represents the basic attributes of visual experiences.

• Any Questions?