1

(Construction) Grammar does not Suffice for NLU

Jerome Feldman, ICSI & UC Berkeley

Natural Language Understanding (NLU) is a manifest goal for applied linguistics, but its theoretical importance is not as obvious. Integrated form-meaning pairs comprise the crux of Construction Grammar, but what is meaning? Decades of work at ICSI/UCB has established that embodied semantics (ECG) is necessary, but we also know that it is not sufficient. Language is inherently contextual and underspecified. An isolated grammar theory or program can only provide schematic analyses (SemSpecs in ECG) that are inadequate for full NLU.

https://github.com/icsi-berkeley/ecg_homepage/wiki

Natural Language Understanding

• Natural Language Processing (NLP) is the overall category– Search, Machine Translation, Sentiment Analysis, etc.

• . Natural Language Understanding (NLU) ~ action without human intervention– Google Search vs. Google Car

• Why NLU might be of interest to ICCG– Greater outside interest, support– CxG seems necessary for compositional approach– Full path understanding redefines many fundamental issues

• Current Mainstream Approaches – Templates: Siri, Cortana, Google, Alexa (next slide)– Machine Learning

• Natural Language Generation adds more complications– Habitability Problem– FCG – Luc Steels

Slide 2

3

Amazon Alexa Skills ~ Templates

GetHoroscope what is the horoscope for { Sign}

GetHoroscope what will be the horoscope for { Sign}be on {Date}

GetHoroscope get me my horoscope…

MatchSign do {FirstSign} and {Second Sign } get along

MatchSign what is the relationship between {FirstSign} and {Second Sign }

developer.amazon.com/alexa-skills-kit

The Winograd Challengehttp://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

All Pronoun Referent Resolution1. The city councilmen refused the demonstrators a permit because they

advocated/feared violence.

3. The trophy doesn't fit into the brown suitcase because it is too large/small.

5. Joan made sure to thank Susan for all the help she had received/given

7. Paul tried to call George on the phone, but he wasn't successful/ available.

9. The lawyer asked the witness a question, but he was reluctant to repeat/ answer it.

11. The delivery truck zoomed by the school bus because it was going so fast/slow.

15. The man couldn't lift his son because he was so weak/heavy 4

ECG2 - NLU Beyond the 1980s

1. Much more computation2. NLP technology3. Construction Grammar: form-meaning pairs

Conceptual compositionality + Idioms, etc.4. Cognitive Linguistics: Conceptual primitives, Metaphor, etc.

ECG = Embodied Construction Grammar; 6 uses of formalism5. Constrained Best Fit : Analysis, Simulation, Learning

Analysis uses Bayesian (form, meaning and context) best fit6. Under-specification: Meaning involves context, goals, etc.

SemSpec = Semantic/Simulation Specification7. Simulation Semantics: Meaning as action/simulation8. CPRM= Coordinated Probabilistic Relational Models; Petri Nets ++

Action formalism works as a generative model 9. Domain Semantics; Need rich semantics on the Action side10. General NLU front end: Modest effort to link to a new Action side

Slide 5

Language as Logic

Yet every sentence is not a proposition; only such are propositions that have in them truth or falsity. Thus a prayer is a sentence, but it is neither true nor false. Let us therefore dismissall other types of sentences but the proposition, for this last concerns our present inquiry, whereas the investigation of others belongs rather to the study of rhetoric or poetry.

Aristotle (De Interpretatione 17a1-8).

Functionalism

In fact, the belief that neurophysiology is even relevant to the functioning of the mind is just a hypothesis. Who knows if we’re looking at the right aspects of the brain at all. Maybe there are other aspects of the brain that nobody has even dreamt of looking at yet. That’s often happened in the history of science. When people say that the mental is just the neurophysiological at a higher level, they’re being radically unscientific. We know a lot about the mental from a scientific point of view. We have explanatory theories that account for a lot of things. The belief that neurophysiology is implicated in these things could be true, but we have very little evidence for it. So, it’s just a kind of hope; look around and you see neurons: maybe they’re implicated.

Noam Chomsky 1993, p.85

Embodiment

Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible.

Alan Turing (Intelligent Machines,1948)

< Continuity Principle of Darwin, American Pragmatists >

Actionability in Integrated Cognitive Science

1. All living things act; acting is what living things do.2. Natural selection constrains the fitness (utility) of these actions.3. Actionability is an agent's assessment of the expected utility of an external or

internal action.4. Volition is the key concept; agents perform volitional as well as automatic

actions5. This defines, but does claim to solve, actionability as a integrating issue for

Cognitive Science.6. No answers are suggested for hard mind-brain problems like subjective

agency.7. Actionability calculation often involves simulation of action and its

consequences.

Feldman JA(2016)Actionability and Simulation: No Representation without Communication. Front.Psychol.7:1204. doi:10.3389/fpsyg.2016.01204

Slide 9

Active representations• Many inferences about actions derive from what we know

about executing them• X-net representation based on stochastic Petri nets captures

dynamic, parameterized nature of actions• Used for acting, recognition, planning, and language

Walking: bound to a specific walker with a direction or goal

consumes resources (e.g., energy)may have termination condition

(e.g., walker at goal) ongoing, iterative actionwalker=Harry

goal=home

energy

walker at goal

“Indian government stumbling in implementing Liberalization Policy.”

How do we specify an event? Formalized event schema

• Key elements– preconditions, resources, effects, sub-events– evoked by frames (alternatively: predicates, words)

• Contrast with Event Recognition/Extraction, other NLP work

12

ISA

hasFrame hasParameterEVENT

COMPOSITE EVENT

FRAMEActorThemeInstrumentPatient

CONSTRUALPhase (enable, start,

finish, ongoing, cancel)Manner (scales, rate, path) Zoom (expand, collapse)

RELATION(E1,E2)SubeventEnable/DisableSuspend/ResumeAbort/Terminate Cancel/StopMutually ExclusiveCoordinate/Synch

CONSTRUCTSequenceConcurrent/Conc. SyncChoose/AlternativeIterate/RepeatUntil(while)If-then-Else/Conditional

PARAMETERPreconditionsEffectsResources - In, OutInputsOutputsDurationGrounding

Time, Location

Embodied Construction Grammar• Embodied representations

– active perceptual and motor schemas(image schemas, x-schemas, frames, etc.)

– situational and discourse context

• Construction Grammar– Linguistic units pair form and meaning/function– Meaning pole based on (conceptual) Schemas.– Both constituency and (lexical) dependencies used.

• Constraint-based– Based on feature unification (as in LFG, HPSG)– Best fit: Diverse factors flexibly interact.

Ideas from Cognitive Linguistics• Embodied Semantics (Lakoff, Langacker, Sweetser, Talmy)

• Radial categories (Rosch 1973, 1978; Lakoff 1985)– mother: birth / adoptive / surrogate / genetic, …

• Radical Construction Grammar (Croft 2001)– Reference, Modification, (Predication -> Event Description)

• Metaphor and metonymy (Lakoff & Johnson 1980, …)– ARGUMENT IS WAR, MORE IS UP– The ham sandwich wants his check.

• Mental spaces (Fauconnier 1994)– The girl with blue eyes in the painting really has green eyes.

• Conceptual blending (Fauconnier & Turner 2002, inter alia)– workaholic, information highway, fake guns– “Does the name Pavlov ring a bell?”

Conceptual Compositionality

Ontology Fragment

(type motion sub process)(type create sub process cause)(type move sub motion)(type walk sub motion)(type enter sub motion)(type exit sub motion)(type run sub motion)(type drive sub motion)

Semantic Specification

“the man built the house”

ECG WorkbenchECG Workbench:

● Based on Eclipse● Takes advantage of and fully integrates with

Eclipse RCP (Rich Client Platform)● Makes it easy to enter, edit and check consistency

of ECG grammars● Can analyze text licensed by the grammar,

producing a SemSpec (Semantic Specification)● Available at: https://github.com/icsi-berkeley/ecg_homepage

18

19

20

Embodied Construction Grammar: ECG

• ECG serves:1. as a technical tool for linguistic analysis2. to specify shared grammar, conceptual conventions of a

linguistic community3. as a computer specification for implementing linguistic

theories4. as a representation for models and theories of language

acquisition5. as a front-end module for applied language-understanding

tasks6. as a high-level functional description for biological and

behavioral experiments

physics lowest energy state

chemistry molecular fit

biology fitness, MEUNeuroeconomics

vision threats, friends

language errors, NTL, OT

Constrained Best Fit in Natureinanimate animate

society, politics framing, compromise

• Mother (I) give you this (a toy).

CHILDES Beijing Corpus (Tardiff, 1993; Tardiff, 1996)

ma1+ma gei3 ni3 zhei4+gemother give 2PS this+CLS

• You give auntie [the peach].

• Oh (go on)! You give [auntie] [that].

Productive Argument Omission (Mandarin)Johno Bryant & Eva Mok

1

2

3

ni3 gei3 yi22PS give auntie

ao ni3 gei3 yaEMP 2PS give EMP

4 gei3 give

• [I] give [you] [some peach].

Competition-based analyzer finds the best analysis

• An analysis is made up of:– A constructional tree– A set of contextual resolutions– A semantic specification

The best fit has the highest combined score

Analyzing ni3 gei3 yi2(You give auntie)

• Syntactic Fit: – P(Theme omitted | ditransitive cxn) = 0.65– P(Recipient omitted | ditransitive cxn) = 0.42

Two of the competing analyses:

ni3 gei3 yi2 omitted↓ ↓ ↓ ↓

Giver Transfer Recipient Theme

ni3 gei3 omitted yi2↓ ↓ ↓ ↓

Giver Transfer Recipient Theme

(1-0.78)*(1-0.42)*0.65 = 0.08 (1-0.78)*(1-0.65)*0.42 = 0.03

26

Metaphors in ECG "he slid into poverty"schema MetaphorSchema

rolesname: @metaphors source: @entitytarget: @entity

schema stateAsLocationMetaphorsubcase of MetaphorSchema

roles name: @stateAsLocationsource: @regiontarget: @abstractState

// reconstrues an RD's ontological-category to the source domain ~ best fitschema MetaphoricalRD

subcase of RD roles

met: MetaphorSchema

general construction MetaphoricalNominalsubcase of Nominal

constructionalconstituents

core: Nominalmeaning: MetaphoricalRDevokes MetaphorSchema as metconstraints

self.m.number <--> core.m.numberself.m.bounding <--> core.m.boundingself.m.scale <--> core.m.scaleself.m.amount <--> core.m.amountself.m.givenness <--> core.m.givennessself.m.met <--> met

Shared Ontology, Templates

Task API

SemSpec

USER Application(World)

Problem Solver(s)

Speech/Text

ECG Analyzer Specializer

Observations

GeneralNLUFrameworkAutonomousSystems

N-tuples(ActSpec):JSONagentcommunication

Commands and data

App-interactions

Situation model

N-tuples

Inferences

UI-Agent

Robot1, move North then return!

Robot1, move North then return!

Analyzer

Robot1, move North then return!

Parse serial processes

Initialize executable template

Assign values in SemSpec to template

Specializer

Parameters of message

First part of N-tuple received by Solver; contains information for first part of serial process (“move North”)

Second part of N-tuple received by Solver; contains information for second part of serial process (“return”)

Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading='north', goal=None)

N-Tuple ~ ActSpec

[Struct(protagonist='robot1_instance', heading='north', goal=None, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing'),

Struct(protagonist='robot1_instance', heading=None, goal={'location': 'home'}, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing')]

Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading=None, goal={'location': 'home'})

Solver

Unpacks N-tuple passed from Specializer.

Determines heading of movement vector (“move North”).

Returns to “home” position, stored in prior state variable.

Morse (Robot Simulator)

Moves robot

Gets the new state of theworld

Updates internalmodel of the world

Gets the updated state of the world

Moves the robot in simulation(and waits for it to arrive)

35

User: Robot1, move to the box!

Solver: Which box?

User: The red one

Solver: Which red one ?

User: The one near the green box.

Solver: Begin move to destination.

Clarification Dialog

36

37

ECG2 NLU Framework ~ again

Modularity and Retargeting

• Modularity of the system permits easy retargeting towards new applications

• Deltas: modifications that enable application-specific functionality.• Action:

• Subclass core modules (e.g. Core Problem Solver)• Implement n-tuple communication for new modules

• Language:• Add application-specific vocabulary (Token Tool)• Define application-specific N-tuple templates• Import or create application-specific schemas and

constructions (package system).

• Product: what the end user sees/uses• Developer uses ECG 2.0 environment to generate a product• e.g., Robot Control, StarCraft Interface

ECG 2.0 NLU Environment

• Tools• ECG Workbench• Token Tool• Template Editor (JSON)• Package System, Git• Kaldi speech analysis• Celex morphology

• Core modules• Core language

• ECG Analyzer, Core Specializer, Core UI-Agent, Core ECG Grammar

• Core action• Core Problem Solver

• Core Communication: • Transport, N-tuple templates, Core Agent

• Core modules function as an integrated starter application.

Software Tools: Token Tool

Deltas for a New Application (Pt. 1)

Identify application domainand relevant vocabulary, and design API.

Domain: Economic Policy MetaphorsVocabulary: • Source: MOTION (stumble, collapse…),

HEALTH (cure, prescribe, sick…) • Target: ECONOMIC POLICY (economy,

depression…)App: after-KARMA (Narayanan 1997)

Design n-tuple templates toconvey semantics to ProblemSolver.

Action_is_Motion: {mover: @Indiaactionary: @stumbleaspect: @progressiveframe: SelfMotionaction: {

actionary: @implementcreatedThing: @policycreator: @Indiaframe: Creation

}}

“Indian government stumbling in implementing Liberalization Policy.”

Deltas for a New Application (Pt. 2)

• Add relevant tokens (andschemas/cxns, if necessary) togrammar.

• Extend existing “Core” Specializer and Problem Solver as needed for application

• Build and test the new product. Examples:• India is stumbling in

implementing the policy.

India :: Country :: self.m.referent <-- @IndiaStumble :: SelfMotion :: self.m.actionary <-- @stumblePolicy :: AbstractNoun :: self.m.ontological-category <-- @policyImplement :: CreationType :: self.m.actionary <-- @implement

TBD

Problem ~ Indirect Speech Acts

• Utterances (“it’s warm in here”) can have three “levels” of meaning (Austin 1962):

� Locutionary: literal interpretation (e.g. assertion about room temperature)

� Illocutionary: the intention (e.g. request to open a window)

� Perlocutionary: the effect (e.g. the listener actually opening the window)

� Indirect speech act: literal interpretation is distinct from the intention of the utterance.

Solution in Construction Grammar

� Illocutionary Force Indicating Devices (IFIDs): indirect speech acts common in certain grammatical patterns (Searle & Vanderveken, 1985)

� Define constructions to exploit best-fit analysis of ECG Analyzer

� Could you push the blue box 5 inches south?

mood

mood

S = {“is the blue box big?”,“did Robot1 push the green

box?”“could you push the blue

box?”…}

S = {“could you push the blue box?”…}

• Resulting n-tuple then has indirect in the SpeechAct slot

Solution In Construction Grammar

� System could request clarification if uncertain of indirect speech act interpretation

SpeechAct: indirectPredicate_type: commandRetury_type: error_descriptorContent: {….}

mood

mood

Embodied Construction Grammar: ECG

• ECG serves:1. as a technical tool for linguistic analysis2. to specify shared grammar, conceptual conventions of a

linguistic community3. as a computer specification for implementing linguistic

theories4. as a representation for models and theories of language

acquisition5. as a front-end system for applied language-understanding

tasks6. as a high-level functional description for biological and

behavioral experiments

51

Bryant,J. Best-Fit Constructional Analysis. PhD thesis, University of California at Berkeley, 2008.

Eppe,M., S.Trott, V. Raghuram, J. Feldman and A. Janin “Application-Independent and Integration-Friendly Natural Language Understanding” Global Conference on Artificial Intelligence (GCAI 2016), EPiC Series in Computing, 2016

Feldman J. From Molecule to Metaphor: A Neural Theory of Language. MIT Pres, 2006

Feldman, J. , J. Bryant, and E. Dodge. A Neural Theory of Language and Embodied Construction Grammar. In The Oxford Handbook of Computational Linguistics, pages 38 – 111. Oxford University Press, 2009.

Feldman, J. Embodied Language, Best-fit Analysis, and Formal Compositionality, Physics of Life Reviews, Vol. 7, #4, pp. 385-410, 2010

Feldman, J. (2016) Actionability and Simulation:NoRepresentation withoutCommunication. Front.Psychol.7:1204. doi:10.3389/fpsyg.2016.01204

Trott,S. M. Eppe and J. Feldman “Communicating Intentions in Human-Robot Interaction” International Symposium on Robot and Human Interactive Communication (RO-MAN 2016)

References

52

Computing with Natural Language

• An integrated system combining – Deep semantic analysis of language in context with– A scalable simulation model

• Best-fit Language Analyzer– Embodied Construction Grammar (ECG)

• Construction Parser – John Bryant PhD Thesis 2008– Eva Mok PhD Thesis 2009– Ellen Dodge PhD Thesis 2010

• Scalable Domain Representation– Event Models

– Steve Sinha PhD Thesis 2008– Joe Makin PhD Thesis 2008

– Coordinated Probabilistic Relational Models – Leon Barrett PhD Thesis 2010– Steve Doubleday Thesis 2015

Cognitive Science and Neuroscience

Science is a reductionist enterprise - we look for explanations of phenomena at more basic levels. This does not entail "eliminative reduction" where only the lowest level has explanatory power. Theory, modelling, and experiment at multiple levels is important and these should be consistent. For Cognitive Science, the ancient formulation of knowledge as truth may be a serious barrier to understanding the mapping of thought to neurobiology and beyond.

Actionability, Simulation and Unified Cognitive Science1) Action is evolutionarily much older than symbolic thought, belief, etc.; also developmentally earlier2) Only living things act (in our sense); natural forces, mechanisms act by metaphorical extension.3) Fitness is nature’s assessment of actions; we define actionability as an organism’s internal

assessment of its available actions in context. 4) Actionability, not non-tautological truth, is what an agent/animal can actually compute.5) Communication is action and is needed for cooperation – from pheromones to language6) Actions include persistent change of internal state: self-concept, memory, world models, learning, etc.

The external world (e.g., other agents) is not static - internal models need simulation7) The brain is not a set of areas that represent things, rather a network of circuits that do things.8) In animals, perception is best-fit, active, and utility/affordance based.9) Mysteries remain; subjective experience, binding, self, free will, robots, etc.

10) One crucial divide/cline is volitional action and communication – boundary not clear, but birds areabove the line; protozoans, plants below. Assume, in nature, neurons are necessary for volition.

11) Volitional actions have automatic components and influence, e.g., speech12) Cognitive Science is bounded by [neurons, individuals]; unify with related sciences.13) Overall goal of the effort is consistency with all experimental findings.14) Theory remains central; multiple formalisms are needed – theories should cohere

Control, probability, computation, logic, dynamics, utility, process, system, learning,15) Formulation is multi-level in three ways:

Standard divisions by scale, complexity - synapse, neuron, circuit, etc.System formulation – whole and parts inseparable, body-environment coupling essential Higher level sciences describe the phenomena, e.g., linguistics, psychology.

16) Action models are multi-modal: describe execution, recognition, planning, language. 17) Volitional simulation proposed as the mechanism of planning, mind-reading, etc. With an

appropriate formalism, simulation can yield both causal and predictive inferences.18) Biological, social, and cultural co-evolution, including language.19) Linguistics based on embodied simulation semantics as the foundation of language and thought. 20) Additional mechanisms include construction grammar, mental spaces, mappings, etc.21) Rationalization and other mental illusions

Unified Cognitive Science

NeurobiologyPsychologyComputer ScienceLinguisticsPhilosophySocial SciencesExperience

Take all the Findings and Constraints Seriously

ECG WorkbenchECG Workbench:

● Based on Eclipse● Takes advantage of and fully integrates with

Eclipse RCP (Rich Client Platform)● Makes it easy to enter, edit and check consistency

of ECG grammars● Can analyze text licensed by the grammar,

producing a SemSpec (Semantic Specification)● Download: http://www1.icsi.berkeley.edu/~lucag/

Same Examples in SpanishSimple Commands:Robot1, muévete al norte!Robot1, muévete detrás de una caja roja!Robot1, empuje la caja azul al norte!

Serial Processes:Robot1, muévete a la caja azul y regresa!Robot1, muévete a la caja roja y grande y muévete a la caja roja y pequeña!

Conditionals + Referent Resolution:Robot1, si la caja verde está cerca de la caja roja y pequeña, la empuje al norte!Robot1, si la caja pequeña es roja, muévete a la caja verde y la empuje!

Questions:es la caja cerca de la caja azul verde?está la caja verde cerca de la caja roja y pequeña?cuáles cajas son rojas? cuáles cajas están cerca de la caja roja y pequeña?dónde está la caja verde?

59

OBJECTIVES

Perform and integrate basic research and converging evidence from cognitive linguistics, computation, and neural information processing to build and exploit cognitive models of language understanding (NLU) and use.

Provide and distribute an operational computational framework of action/simulation semantics to investigate the interaction between language, action, and cognition.

Action/Simulation semantics offers the possibility to build systems for computing with natural language that come close to human performance levels. This is necessary for joint action in complex naval scenarios with a mix of human and artificial agents.

60

OUTLINE also HANDOUT

1. Intro. NLU is not NLP, good for CXG, not solved, approachable2. Grammar does not suffice, CxG as {form, meaning} but what is meaning?3. NTL/ECG started from meaning, brain, neuro-computation4. Quote slides, then active meaning, Actionability, Simulation5. Now ECG intro, workbench basics , examples6. Best fit7. Example analyses still just SemSpec. Maybe Spanish, 8. Metaphor9. NLU, beyond the SemSpec, system diagram on hand-out10. Morse robot example11. Github release 12. Compositionality, Intention example

61

Action Language Understanding System

• Demonstrate utility through a series of scalable prototypes – that show the ability of the system to handle increasingly

complex language– in a general way across multiple tasks and environments – to support communication in communities comprised of both

human and artificial agents

• Current Goal: Implement a prototype system that can follow instructions and synthesize actions and procedures expressed in natural language. – This requires the system to analyze natural language and

translate this language in context into a coordinated network of actions and complex commands.

Slide 62

Software Tools: ECG Workbench

Robot1, move North then return!

65

Shared Ontology, Templates

Task API

SemSpec

USER Application(World)

Problem Solver(s)

Speech/Text

ECG Analyzer Specializer

Observations

GeneralNLUFrameworkAutonomousSystems

ECG:EmbodiedConstructionGrammarN-tuples:JSONagentcommunication

Commands and data

App-interactions

Situation model

N-tuples

Inferences

UI-Agent

� Discourse “moods”: assertion, query, command

Indirect Speech Acts

� Sometimes the literal “mood” is distinct from the intended “mood”

� Could you push the blue box 5 inches south?

� Solution: exploit ECG compositionality and best-fit constraints to identify intended mood