What functional brain imaging reveals about the neuroarchitecture of object

knowledge Kai-min Kevin Chang, Vicente Malave, Svetlana Shinkareva, and Marcel Adam Just

Center for Cognitive Brain Imaging, Carnegie Mellon University

Categories Exemplars

body parts leg, arm, eye, foot, hand

furniture chair, table, bed, desk, dresser

vehicles car, airplane, train, truck, bicycle

animals horse, dog, bear, cow, cat

kitchen utensils glass, knife, bottle, cup, spoon

tools chisel, hammer, screwdriver, pliers, saw

buildings apartment, barn, house, church, igloo

part of building window, door, chimney, closet, arch

clothing coat, dress, shirt, skirt, pants

insects fly, ant, bee, butterfly, beetle

vegetables lettuce, tomato, carrot, corn, celery

man made objects refrigerator, key, telephone, watch, bell

refrigerator

celery

antX

X

3s

7s

time

OverviewWe used functional Magnetic Resonance Imaging to study the cortical systems that underpin semantic representation of object knowledge.

MethodParticipant was presented with black and white line drawings of 60 objects from a range of categories (see below) and were instructed to think of the same properties consistently during each presentation.

fMRI procedure:•Cerebral activation was measured using BOLD contrast (Kwong et al., 1992; Ogawa et al., 1990).•Functional images were acquired on a Siemens Allegra 3.0T scanner at the Brain Imaging Research Center.•EPI acquisition sequence, 17 oblique axial slices, TR=1000ms, TE=30ms, 64x64 acquisition matrix, 5 mm thickness, 1 mm gap, flip angle 60°.•Image were corrected for slice acquisition timing, and motion-corrected with Statistical Parametric Mapping software (SPM99, Wellcome Department of Cognitive Neurology, London, UK).•Data were normalized to the Montreal Neurological Institute (MNI) template, and resampled to 3x3x6 mm voxels for Machine Learning analysis.•Analyses of a single brain region at a time used region definitions derived from the Anatomical Automatic Labeling (AAL) system (Tzourio-Mazoyer et al., 2002).

Machine Learning

EvaluationClassification results were evaluated using k-fold cross validation, where one example per class was left out for each fold. Both raw and rank accuracies are reported.

4CAPS ModelingWe implemented a set of neural processing regions in the 4CAPS neural architecture (Just & Varma, in press), along with their corresponding specialization to process each feature. The 4CAPS model was used to simulate human performing the picture-naming task.

TheoreticalEach object is associated with a set of distinguishing sensory/functional features (e.g. visual-motion, visual-parts, function, etc.) according to Cree & McRae (2003)'s semantic feature norming studies. The sensory/functional features are thought to be processed by a set of neural processing regions that are mostly agreed by the current state of the literature. Correlation = 0.5265

Data-drivenCan we instead learn the set of activated neural processing centers from our data? Yes, but we need to check if it is generailzable, otherwise we may overfit to the data.

EvaluationThe activation pattern of the neural processing regions in 4CAPS is correlated to human's cortical activation pattern reading the same word.

Mean PSC matrixwords x voxels

Preprocessed fMRI datatime x voxels

Construct mean Percent Signal Change (PSC) matrix per stimulus event averaged over 4 successive images

Full training setwords training x voxels Full test set

words test x voxels

Training setwords training x voxels select

Test setwords test x voxels select

Partition data into training and test sets

Select Voxels (features)

Evaluate classification

Subset of voxels selected from Training set

Train classifier to identify categories based on

selected features

Get classification accuracy

Alternative Feature (voxel)

Selection Methods

Select individually most discriminating voxels:

Threshold Number of Misclassifications

select voxels with the minimum number of misclassifications made by the best threshold chosen for each voxel (Ben-Dor et al., 2000)

Wilcoxon select voxels with highest statistic value

Select conjointly most discriminating voxels:

Logistic regression select voxels with highest absolute value of the regression weights

Select most stable voxels :

Stability of differential response for wordsselect voxels with the largest average

pairwise correlations across the word presentation

2 Alternative Classifiers:

Gaussian Naïve Bayes

Uses Bayes rule to estimate the probability distribution from the training set. Makes conditional independence assumption of the features.

Logistic Regression

Uses parametric form to directly fit the distribution P(Y|X).

Classification performance is evaluated using k-fold cross validation

ConclusionWe have shown that high classification accuracies can be obtained for distinguishing the 12 categories of objects and even the 60 exemplars. Furthermore, the neural processing regions associated with the sensory/functional features are modeled in 4CAPS. Simulation of 4CAPS shows comparable activation pattern to human.

Experiment Correlations

Across trials 0.7838

Across words 0.6658

Across studies0.0225 on all categories

0.3039 on tools and dwellings

Across subjects N/A

Experiment Chance Raw Rank

Category(12-way) 8.3% 52% 88%

Exemplar(60-way) 1.6% 37% 91%

Hammer

A_tool

Has_a_hand

Has_a_handle

Has_a_metal_head

Has_a_clefted_head/claw

Found_in_tool_boxes

Has_a_wooden_handle

Is_loudIs_heavyMade_of_metal

Made_of_word

Used_for_carpentry

Used_for_construction

Used_for_pounding

Used_for_pounding_

nails

Used_for_pulling_nails

encyclopedic function smell soundtactile taste taxonomicVisual-color

Visual_form_and_

surface_properties

Visual-motion

tool dwelling

Bilateral fusiform

gyrus

Left sensory motor cortex

Piriform cortex

Right lateral orbital

frontal area

Left ventralpre-motor cortex

Superior temporal sulcus

Left middle

temporal gyrus

Ventro occipito temporal

cortex

Left parahippocampal

gyrus

Right parahippocampal

gyrus

Left inferior parietal lobule

Left postcentral

gyrus

Left precentral

gyrus

Left cuneus

PRECENT POSTCENTLSTANT LSTMID LSTPOS LIES LPARAHIP

RPARAHIP

LFUSIFORM LIPL

RFUSIFORM

LSES

Word

Feature

Knowledge Type

Neural Processing

Regions

ROI

Machine Learning Flowchart Correlation Between Neural and 4CAPS Activation Vector

4CAPS Modeling Hierarchy