NTL – Converging Constraints

• Basic concepts and words derive their meaning from embodied experience.

• Abstract and theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts.

• Structured Connectionist Models can capture both of these processes nicely.

• Grammar extends this by Constructions: pairings of form with embodied meaning.

Simulation-based language understanding“Harry walked to the cafe.”

Schema Trajector Goalwalk Harry cafe

Analysis Process

Simulation Specification

Utterance

SimulationCafe

Constructions

General Knowledge

Belief State

19Konvens, 09.10.2000

The ICSI/BerkeleyNeural Theory of Language Project

Background: Primate Motor Control• Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod,

Rizzolatti)– Should model coordinated, distributed, parameterized control programs

required for motor action and perception.– Should be an active structure.– Should be able to model concurrent actions and interrupts.

• Model– The NTL project has developed a computational model based on that

satisfies these requirements (x- schemas).– Details, papers, etc. can be obtained on the web at

http://www.icsi.berkeley.edu/NTL

Active representations• Representation based on stochastic Petri nets captures dynamic,

parameterized nature of actions• Many inferences about actions derive from what we know about

executing them• Generative model: action, planning, recognition, language.

Walking:bound to a specific walker with a

direction or goalconsumes resources (e.g., energy)may have termination condition

(e.g., walker at goal) ongoing, iterative action

walker=Harry

goal=home

energy

walker at goal

Active Motion Model

Evolving Responses of Competing Models over Time.

Nigel Goddard

1989

Language Development in Children

• 0-3 mo: prefers sounds in native language• 3-6 mo: imitation of vowel sounds only• 6-8 mo: babbling in consonant-vowel segments• 8-10 mo: word comprehension, starts to lose sensitivity to

consonants outside native language• 12-13 mo: word production (naming)• 16-20 mo: word combinations, relational words (verbs, adj.)• 24-36 mo: grammaticization, inflectional morphology• 3 years – adulthood: vocab. growth, sentence-level grammar

for discourse purposes

Learning Spatial Relation Words Terry Regier

A model of children learning spatial relations.Assumes child hears one word label of scene.Program learns well enough to label novel scenes

correctly.Extended to simple motion scenarios, like INTO.System works across languages.Mechanisms are neurally plausible.

Learning System

We’ll look at the details next lecture

dynamic relations(e.g. into)

structured connectionistnetwork (based on visual system)

Limitations

• Scale• Uniqueness/Plausibility• Grammar• Abstract Concepts• Inference• Representation• Biological Realism

physics lowest energy state

chemistry molecularminima

biology fitness, MEU neuroeconomics

vision threats, friends

language errors, NTL

Constrained Best Fit in Nature

inanimate animate

Learning Verb MeaningsDavid Bailey

A model of children learning their first verbs.Assumes parent labels child’s actions.Child knows parameters of action, associates with wordProgram learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation)System works across languagesMechanisms are neurally plausible.

Motor Control (X-schema) for SLIDE

Parameters for the SLIDE X-schema

Feature Structures for PUSH

System Overview

Learning Two Senses of PUSH

Model merging based on Bayesian MDL

Training ResultsDavid Bailey

English• 165 Training Examples, 18 verbs• Learns optimal number of word senses (21)• 32 Test examples : 78% recognition, 81% action• All mistakes were close lift ~ yank, etc.• Learned some particle CXN,e.g., pull up

Farsi • With identical settings, learned senses not in English

physics lowest energy state

chemistry molecularminima

biology fitness, MEU neuroeconomics

vision threats, friends

language errors, NTL

Constrained Best Fit in Nature

inanimate animate

Compositionality

• Traditional Context-free composition of logical forms

• Contemporary Constructional composition of conceptual frames

Formal Cognitive Linguistics

Embodied Construction Grammar

(Bergen, Chang & Paskin 2000)• Assumptions from Construction Grammar

– Constructions are form-meaning pairs(Lakoff 1987, Goldberg 1995)

– Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal)

• Constructions evoke and bind semantic schemas• Additional influences

– Cognitive Grammar (Langacker 1987)– Frame Semantics (Fillmore 1982)– Structured Connectionism (Feldman 1987)

Phonetics

Semantics

Pragmatics

Morphology

Syntax

Traditional Levels of Analysis

Phonetics

Semantics

Pragmatics

Morphology

Syntax

“Harry walked into the cafe.”

Utterance

Language understanding: analysis & simulation

“Harry walked into the cafe.”

Analysis Process

Simulation Specification

Utterance

Constructions

General Knowledge

Belief State

construction WALKED in context cconstituents:

form f of type [wakt]meaning walking construed as Walk-Action

semantic constraints:walking.time before c.speech-timewalking.aspect = encapsulated

designates walking

CAFE Simulation

Simulation specification

A simulation specification consists of:•semantic schemas evoked by constructions•bindings between schemas (labeled by the constructions that enforce them)

Conclusion• Language acquisition and use is a hallmark of being human

– Language seems to rely on fine-grained aspects of embodied (sensory-motor and social cognition) primitives and brain-like computation (massively parallel, distributed, spreading activation, temporal binding).

– Understanding requires imaginative simulation!– We have built a pilot system that demonstrates the use of motor control

representations in grounding the language of abstract actions and policies.• Sensory-Motor imagination and simulation is crucial in interpretation!

• Ongoing Work.– Formalize and use a compositional set of embodied conceptual primitives and

grammatical constructions.– Perform both behavioral and fMRI imaging experiments to test the predictions

of the simulation hypothesis. – Further refine and ground the model in details of neural anatomy and functional

architecture (basal-thalamic-cortical loops).