INTRODUCTION TO ARTIFICIAL INTELLIGENCE

Massimo Poesio

LECTURE 8Concepts in the brain

USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONS

• Studies of brain-damaged patients have been shown to provide useful insights in the organization of conceptual knowledge in the brain

• Some patients are unable to identify or name man made objects and others may not be able to identify or name natural kinds (like animals)– Warrington and Shallice 1984, Caramazza & Shilton 1998

• fMRI has been used to identify these distinctions in healthy patients as well– E.g., Haxby et al 2000, Martin & Chao 2003

• See, e.g., Mahon & Caramazza 2011, Martin 2007 for review

Warrington & Shallice 1984

• Warrington and Shallice (1984) reported a patient called JBR who following an acute lesion to the left temporal lobe (as a result of herpes encephalitis) had a selective deficit when asked to name pictures from just one semantic category – living things.

• By contrast JBR was able to name non-living objects very well including those with low frequency names such as ‘accordion’ that were matched for the number of letters in the name and the visual complexity of the object.

• Other patients have shown opposite pattern

Evidence from semantic category deficits

• Modality-specific deficits – Patients are unable to name visually presented objects, but can name

them from other modalities and can access other semantic information about visually presented stimuli (Beauvois, 1982)

– Other visual processing is fine.

• Category-specific deficits (e.g., Warrington & McCarthy, 1983, 1987; Warrington & Shallice, 1984; Gainotti & Silveri, 1996)

– Patients show impairments in processing living things vs. man-made objects and vice versa.

– Interesting exceptions: fruits, vegetables & other foods; musical instruments

A PET Study on categories (Nature 1996)

Study

• 16 adults (8M, 8F) participated in a PET (positron emission tomography) study.– Involves injecting subject with a positron emitting radioactive

substance (dye) – Regions with more metabolic activity will absorb more of the

substance and thus emit more positrons– Positron-electron collisions yield gamma rays, which are detected

• Increased rCBF (regional changes in cerebral blood flow) was measured– When subjects viewed line drawings of animals and tools.

The experiment

• Subjects looked at pictures of animals and tools and named them silently.

• They also looked at noise patterns (baseline 1)• And novel nonsense objects (baseline 2)• Each stimulus was presented for 180ms followed by a

fixation cross of 1820 ms.• Drawings were controlled for name frequency and

category typicality

PremotorACC

Left middle temporal gyrus

Calcarine Sulcus

Conclusions• Both animal and tool naming activate the ventral

temporal lobe region.• Tools differentially activate the ACC, pre-motor and

left middle temporal region (known to be related to processing action words).

• Naming animals differentially activated left medial occipital lobe (early visual processing)

• The object categories appear to be in a distributed circuit that involves activating different salient aspects of the category.

REPRESENTATION OF CONCEPTS IN THE BRAIN: COMPETING HYPOTHESES

• Unitary Content Hypothesis– Semantic information is stored in an abstract, amodal

format organized by category.

• Multiple Semantics Hypothesis– Semantic information is stored in many modality-specific

semantic subsystems. Information in each subsystem is stored in a modality specific format.

– Our intuitive sense of information being organized by categories is based on strong connections between related parts of these modality-specific semantic systems.

Unitary Content Hypotheses (UCH)

(Caramazza et al., 1990; Caramazza & Shelton, 1998; Riddoch et al., 1988; Pylyshyn, 1973)

Multiple Semantics Hypotheses (MSH)

(Paivio, 1971; Beauvois, 1982; Shallice, 1987, 1988; McCarthy & Warrington, 1988)

Representation of words in semantic memory: the Functional Web hypothesis

• A word is represented in the cortex as a functional web

• Spread over a wide area of cortex– Includes perceptual information– As well as specifically conceptual information

• For nominal concepts, mainly in– Angular gyrus– (?) For some, middle temporal gyrus– (?) For some, supramarginal gyrus

– Plus phonological information

Example: The concept DOG

• We know what a dog looks like– Visual information, in occipital lobe

• We know what its bark sounds like– Auditory information, in temporal lobe

• We know what its fur feels like– Somatosensory information, in parietal lobe

• All of the above..– constitute perceptual information– are subwebs with many nodes each– have to be interconnected into a larger web– along with further web structure for conceptual

information

Building a model of a functional web:First steps

V

C

Each node in this diagramrepresents the cardinal node* of a subweb of properties

For exampleM

T

*to be defined in a moment!

Add phonological recognition

V

M

C

For example, FORK

Labels for Properties:C – ConceptualM – Motor P – Phonological imageT – TactileV – Visual

T

P

The phonological image of the spoken form [fork] (in Wernicke’s area)

These are allcardinal nodes –each is supportedby a subweb

Add node in primary auditory area

V

M

CT

P

PA

Primary Auditory: the cortical structures in the primary auditory cortex that are activated when the ears receive the vibrations of the spoken form [fork]

For example, FORK

Labels for Properties:C – ConceptualM – Motor P – Phonological imagePA – Primary AuditoryT – TactileV – Visual

Add node for phonological production

V

M

CT

P

PA

PP

For example, FORK

Labels for Properties: C – Conceptual M – Motor P – Phonological image PA – Primary Auditory PP – Phonological Production T – Tactile V – Visual

Arcuate fasciculus

Part of the functional web for DOG(showing cardinal nodes only)

V

MC

T

P

PA

PP

Each node shown here is the cardinal node of a subweb

For example, the cardinal node of the visual subweb

An activated functional web(with two subwebs partly shown)

V

PRPA

M

C

PP

T

Visual features

C – Cardinal concept nodeM – MemoriesPA – Primary auditoryPP – Phonological productionPR – Phonological recognitionT – TactileV – Visual

FROM WORDNET TO BRAINNET

• Neural evidence, unlike the evidence used to compile dictionaries and WordNet, and like the evidence one gathers from corpora and certain behavioral experiments, is entirely objective (although it can be subjective in the sense of differing from subject to subject)

• The objective of our research is to combine evidence from brain data, from corpora, and from behavioral experiments (all of which is rather noisy) to develop a new architecture for conceptual knowledge: BrainNet

A CASE STUDY: ABSTRACT CONCEPTS

• Until recently, most work on concepts in CL / neuroscience / psychology focused on concrete concepts

• But the type of conceptual knowledge that really challenges traditional assumptions about its organization are `abstract concepts’ – or to be more precise, the set of categories of non-concrete concepts– Events / actions– States– ‘Urabstract’ concepts: LAW, JUSTICE, ART

• We are carrying out explorations of abstract knowledge using fMRI

Anderson et al 2012a, 2012b, 2013, submitted

THEORIES OF ABSTRACT CONCEPTS IN AI AND COGNITIVE (NEURO)SCIENCE

• In CL/AI: TAXONOMIC organization for both abstract and concrete concepts– ‘UPPER ONTOLOGIES’, e.g., DOLCE

• In psychology: ‘concreteness’ scale• Best known Cognitive Neuroscience: Paivio’s DUAL CODE

theory (Paivio, 1986)– CONCRETE: verbal system & visual system– ABSTRACT: verbal system only

• Schwanenflugel & Akin 1994: CONTEXT AVAILABILITY• Barsalou’s SCENARIO-BASED MODEL (Barsalou, 1999):

– Abstract knowledge organized around SCENARIOS

The DOLCE UPPER ONTOLOGY

QQualit

y

PQPhysicalQuality

AQAbstractQuality

TQTemporalQuality

PDPerdurant

EVEvent

STVStative

ACHAchievement

ACCAccomplishment

STState

PROProcess

PTParticular

RRegion

PRPhysicalRegion

ARAbstractRegion

TRTemporalRegion

TTime

Interval

SSpaceRegion

ABAbstrac

t

SetFact…

… … …

TLTemporalLocation

SLSpatial

Location

… … …

ASOAgentive

Social Object

NASONon-agentive Social Object

SCSociety

MOBMental Object

SOBSocial Object

FFeature

POBPhysicalObject

NPOBNon-physical

Object

PEDPhysicalEndurant

NPEDNon-physical

Endurant

EDEndurant

SAGSocial Agent

APOAgentive Physical

Object

NAPONon-agentive

Physical Object

…

ASArbitrary

Sum

MAmount of

Matter

… … … …

THE OBJECTIVES OF OUR EXPERIMENT

• Identify the representation in the brain of a variety of WordNet categories exemplifying both concrete and abstract concepts (abstract words chosen by inspecting the words rated as most abstract in the De Rosa et al norms 2005)– Really abstract: ATTRIBUTE, COMMUNICATION, EVENT, LOCATION,

‘URABSTRACT’ – A category of concrete objects: TOOLS– A complex category: SOCIAL-ROLE

• Comparing two types of classification:– TAXONOMIC (as in WordNet)– DOMAIN (cfr. Barsalou’s hypothesis about abstract concepts being ‘situated’)

• Two domains: LAW and MUSIC– Using WordNet Domain

STIMULI

CATEGORY LAW (English) MUSIC (English)

attributegiurisdizione jurisdiction sonorita' sonority

cittadinanza citizenship ritmo rhythm

impunita' impunity melodia melody

legalita' legality tonalita' tonality

illegalita' illegality intonazione pitchcommunication divieto prohibition canzone song

verdetto verdict pentagramma stave

ordinanza decree ballata ballad

addebito accusation ritornello refrain

ingiunzione injunction sinfonia symphony

STIMULI, 2: URABSTRACTS

CATEGORYurabstracts giustizia justice musica music

liberta' liberty blues blues

legge law jazz jazz

corruzione corruption canto singing

refurtiva loot punk punk

STIMULI, 3: SOCIAL ROLES

Social-role giudice judge musicista musician

ladro thief cantante singer

imputato defendantcompositore composer

testimone witness chitarrista guitarist

avvocato lawyer tenore tenor

THE OBJECTIVES OF OUR EXPERIMENT

• Identify the representation in the brain of a variety of WordNet categories exemplifying both concrete and abstract concepts (abstract words chosen by inspecting the words rated as most abstract in the De Rosa et al norms 2005)– Really abstract: ATTRIBUTE, COMMUNICATION, EVENT, LOCATION,

‘URABSTRACT’ – A category of concrete objects: TOOLS– A complex category: SOCIAL-ROLE

• Comparing two types of classification:– TAXONOMIC (as in WordNet)– DOMAIN (cfr. Barsalou’s hypothesis about abstract concepts being ‘situated’)

• Two domains: LAW and MUSIC– Using WordNet Domain

ABSTRACT CONCEPTS: DATA COLLECTION AND ANALYSIS

• 7 right-handed native speakers of Italian• Task:

– Words presented in white on grey screen for 10 sec– Cross in between, 7 sec– Subjects had to think of a situation in which the word applied

• Scanner: 4T Bruker MedSpec MRI scanner, EPI pulse sequence

– TR=1000ms, TE=33ms, 26° flip angle. – Voxel dimensions 3mm*3mm*5mm

• Preprocessing: using UCL’s Statistical Parameter Mapping Software– Data corrected for head motion

• Classification: using a single layer NN

MAIN QUESTIONS

• Can the taxonomic and domain classes be distinguished from the fMRI data?

• Is there a difference in classification accuracy between taxonomy and domain?

• Can the taxonomic and domain classes be predicted across participants?

RESULTS WITHIN PARTICIPANTS (CATEGORY DISTINCTIONS)

ALL CATEGORICAL DISTINCTIONS CAN BE PREDICTED ABOVE CHANCE

THERE ARE SIGNIFICANT DIFFERENCES BETWEEN CATEGORIES

RESULTS WITHIN PARTICIPANTS(DOMAIN)

WITHIN PARTICIPANTS RESULTS SUMMARY

• Can discriminate with accuracy well above chance both taxonomic and domain distinctions

• Easiest categories to recognize: TOOL, ATTRIBUTE, LOCATION, – Then SOCIAL ROLE, COMMUNICATION– Main confusions: communication / event

Red: AttributeBlue: ToolGreen: Location

R+G=YellowG+B=CyanR+B=PinkR+G+B=White

CATEGORY LOCALIZATION IN THE BRAIN

Red: Social-roleGreen: AttributeBlue: Urabstract

Red: Social-roleGreen: CommunicationBlue: Event

R+G=YellowG+B=CyanR+B=PinkR+G+B=White

• Concrete categories TOOL and LOCATION can be predicted across participant; ATTRIBUTE can also be significantly classified; but less concrete classes become conflated with ATTRIBUTE.

• In general DOMAIN can be predicted across participants, however domain membership is much better classified in the most abstract taxonomic classes (attribute, communication and urabstract)

CROSS PARTICIPANTS RESULTS SUMMARY

LAW MUSICAttribute giurisdizione jurisdiction sonorita' sonority

cittadinanza citizenship ritmo rhythmimpunita' impunity melodia melodylegalita' legality tonalita’ tonalityillegalita’ illegality intonazione pitch

communication divieto prohibition canzone songverdetto verdict pentagramma staveordinanza decree ballata balladaddebitoaccusation ritornello refrainingiunzione injunction sinfonia symphony

event arresto arrest concertoconcert

processotrial recital recitalreato crime assolo solofurto theft festival festivalassoluzione acquittal spettacolo show

social-role giudice judge musicista musicianladro thief cantantesingerimputato defendant compositore composertestimone witness chitarrista guitaristavvocatolawyer tenore tenor

tool manette handcuffs violino violin

toga robe tamburo drummanganello truncheon tromba trumpetcappio noose metronomo metronomegrimaldello skeleton key radio radio

Location tribunale court/tribunal palco stagecarcere prison auditorium auditoriumquesturapolice station discoteca discopenitenziario penitentiary conservatorio conservatorypatibolo gallows teatro theatre

urabstracts giustizia justice musica musicliberta' liberty blues blueslegge law jazz jazzcorruzione corruption canto singingrefurtiva loot punk punk

TAXONOMIC / DOMAIN ORGANIZATION

WHAT THE DATA SUGGESTS

READINGS

• Binder & Desai 2011, The Neurobiology of semantic memory, Cell (on the website)