subtle: overview (situation understanding bot through language and environment) umass amherst, umass...

SUBTLE: Overview(Situation Understanding Bot Through Language and Environment)

UMass Amherst, UMass Lowell, Penn, Stanford, Cornell, George Mason

Mitch Marcus

University of Pennsylvania

10/16/2009 SUBTLE Year 2 Overview 2

Our Guiding Vision

Standing orders: clear the building: search for any bombs or weapons, find any injured persons and report them and their location so the medical team can find them, and tell any other persons to leave the building.

0:00 Commander: Jr, I want you to search the room you are in first and when you go to leave head for the North end of the building, continue around East and we will meet up at the end of a long hallway.

0:32 Jr: 10-43:13 Jr: I have cleared the first room and am proceeding North down a hallway.3:20 Commander: Mark that room as clear and make certain to check all the rooms in the

hall.3:45 Jr: Room is marked. Proceeding to the next room.3:56 Commander: Wait. Shouldn’t you be going into the hallway?4:02 Jr: No, I went directly into the next room.4:05 Commander: Can you show me the map and mark your position.4:15 Jr: I am displaying the map, I am the yellow dot.

5:23 Commander: Alright I see now, carry on.

Scientific Objective: Central Hypothesis

Effective communications with autonomous bots in real-time situations requires that bots understand not only what is literally said, but also what is intended, the implicit meaning

Example: Exchange between firefighters (Worcester Cold Storage Fire 1999):

· P2 (a) answers question (b) appears to give command· P1’s implicit question: How can we find you?

• Linguists call this the Question Under Discussion (QUD) · P2 answers the QUD

10/16/2009 3SUBTLE Year 2 Overview

Technical Approach: Key Ideas· We must develop techniques to analyze not only what

sentence a speaker used, but also what that speaker’s implicit meaning.

· The communication system must exploit the broad context of the environment.

· The linguistic specification should incorporate formal models of language• To guarantee computationally efficient analysis methods • To facilitate study of the habitability and effectiveness of the result

· The adequacy of specifications should be determined by empirical, corpus-based methods• Our research must proceed by collecting example corpora of

interactions in increasingly complex (simulated) environments


Potential Breakthroughs· New frameworks and theories for

• Robust automated understanding of linguistic intention• Linking linguistic intention to appropriate robot control

· New machine learning algorithms for structured problems such as Natural Language

· A formal computational specification of a significant subset of English

· A testbed system for investigating HRI using natural language• Should ultimately enable military designers to develop powerful

communication methods between bots and humans.


The SUBTLE team: senior participants· Natural Language Processing/Computer Science

• Aravind Joshi, Penn• Mitch Marcus, Penn• David Smith, UMass Amherst (Research Scientist To join Y3)• Fernando Peirera, Penn (On leave: Director of Research, Google)

· Robotics/Electrical Engineering, Computer Science• George Pappas, Penn• Holly Yanco, UMass Lowell• Hadas Kress-Gazit, Cornell (Penn grad student Cornell)

· Human Simulation & Graphics• Norm Badler, Penn• Jan Allbeck, George Mason (Penn grad student George Mason)

· Machine Learning: Andrew McCallum, UMass Amherst

· Linguistics: Chris Potts, Stanford

· Current Grad Students: 8


SUBTLE Year 2 Overview10/16/2009 7

People Actively Involved Year 2

· Faculty• Norm Badler• Aravind Joshi• Mitch Marcus• Andrew McCallum• George Pappas• Chris Potts • Holly Yanco

· New PhDs• Jan Allbeck• Hadas Kress-Gazit

· Limited involvement:• Fernando Pereira

· Newly Involved• David Smith• Florian Schwarz

· Grad Students• Chris Czyzewicz • Munjal Desai• Kuzman Ganchev• Dan Hestand• Karl Schultz• Qiuye Zhao• Pengfei Huang• Dan Brooks

· Undergraduates• Victoria Schwanda HCI, Cornell• Jessica Ouyang• Dan Keller• Phil Kovac

Joint Inference

Framework – Year 1 Review

Parsing

World ModelSemantics

PragmaticsLinear

Temporal Logic

ParameterizedAction

Representations

Parse tree, indices, semantic tags

Underspecified predicate logic

Graphics/Human

Simulation

Robotics

Linguistics

NaturalLanguageProcessing

MachineLearning



Year 1: Primary Accomplishments

· SUBTLE architecture intensively reworked· Key components built and tested

• LTL →FSA transducer• PAR (Procedural Action Representation) for PragBot

· “PragBot 1” corpus collection tool built, initial corpus collected, NLP analyzer built

· New results: Probabilistic pragmatics meets probabilistic inference

Joint Inference

Evolution of Framework

Parsing


PragmaticsLinear

Temporal Logic

ParameterizedAction

Representations



Graphics/Human

Simulation

Robotics

Linguistics


MachineLearning


PAR + LTL

Joi

nt

Infe

renc

e

PAR + LTL


Parsing


Pragmatics



Graphics/Human

Simulation

Robotics

Linguistics


MachineLearning


PAR + LTL

Joi

nt

Infe

renc

e

PAR + LTL


Parsing


Pragmatics



Graphics/Human

Simulation

Robotics

Linguistics


MachineLearning


Accomplishments to date I

· PhD dissertations completed• Hadas Kress-Gazit, Transforming high level tasks to low level

controllers, Dec 2008 (Asst Prof, Cornell)— Uses Linear Temporal Logic Model Checking to compile complex

constraints and requirements into real-time hybrid robot controller.• Jan Allbeck, Creating 3D Animated Human Behaviors for Virtual

Worlds, June 2009 (Asst Prof, George Mason)— Extended Parameterized Action Representation (PAR) for robust,

multi-agent setting. Developed extensive authoring tools.

· LTL controller is now outputting PAR representations to drive virtual animation for corpus collection, system development

· PAR simulation handles multi-agent setting needed for team exercises of Years 4-5.


· Task spec in English• “Nemo can only be in Regions 1, 3, 5 and 8. Look for Nemo and if you

find him, turn your video camera on and stay where you are. If he disappears again, turn the camera off and resume the search.” (There are12 regions which define the robot propositions {r1, …, r12}).

· Fragment encodes possible moves that enforce:• if R sees Nemo, stay in that region at next step with camera on.

Otherwise, camera off.

LTL Example: “Find Nemo”


- Next - Always◊ - Eventually

The resulting controller can control virtual robot & actual robot


Talk 5

CAROSA: Authoring PARs for NL Predicates, for multiple agents, …


Talk 6

Accomplishments II· A new formal account of Grice’s maxims, a foundation of

linguistic pragmatics · Embedded and tested in a computation implementation of a

rich account of formal linguistic pragmatics• In communicating

— Be truthful— Be relevant— Be brief

• Encoding (for the moment in Markov Logic): Assert(p) => True(p). True(p) =>Assert(p). //Truthful

Assert(p) => Qud(q) ^ About(p,q)). //Relevance

10 (Qud(q) ^ About(p,q)) => Assert(p)

Assert(p) => !CommanderBelieve(p). //Brief

10 !Assert(p) => CommanderBelieve(p)


Short term application: Response Relevance

· Pragmatics of Relevance, Brevity for Bot’s responses formalized within Probabilistic Markov Logic

// 1. General description of what it means to have completed the task.

Report(TaskComplete) <=> (FORALL x (Relevant(x) => Found(x))).

// 2. If you find something, report it.

5 Found(x) => Report(x)

// 3. Report only relevant things.

10 Report(x) => Relevant(x)

10 !Relevant(x) => !Report(x)

// 4. If you can report that the task is complete, report nothing else.

5 Report(TaskComplete) => ((y != TaskComplete) => !Report(y))


Talk 3

Pragmatics & Action → Pragmatics in Action


Talk 4

Integration and State Summarization


Other Accomplishments· Last Year: Initial PragBot web-based interface constructed for

corpus collection• 50 Human-Human corpus of collaborative interactions in simple

isomorphic environment captured & analyzed· New framework & toolkit for joint inference about to be released

• Allows efficient machine inference of probabilistic models across very large, complex knowledge bases• Joint Segmentation & Coreference of research paper citations: 1295

mentions, 134 entities, 36487 tokens• Compare with Markov Logic Networks (Alchemy)

—~25% reduction in error (segmentation & coref) —3-20x faster

· …


First Year Review Report – 10/2008 Develop a common simulation and experimentation scenario…which is

demanding and sophisticated…. You must get busy on building the corpus from a major search and rescue

simulation/experiment/game…. Build a robot language community…. Start to visit DoD labs and establish the groundwork for technology transfer. Continue to assume that future…bots will have better…capabilities than current

bots, yet whenever possible use both current bots and simulated future bots to test algorithms and tools.

Focus on developing how the world view (situation awareness) will influence the language and understanding of the bot.

While the first…2-3 years is focusing…mostly on the bot receiving and understanding information, the overall goal…is to provide a framework for dialog among a team of several bots and several humans … Don’t lose sight of this important goal while performing the preliminary work.

Develop more robust (human-based) performance metrics and base your models and goals on these.


First Year Review Report – 10/2008

1. Develop a common simulation and experimentation scenario…which is demanding and sophisticated….

2. You must get busy on building the corpus from a major search and rescue simulation/experiment/game….

3. Build a robot language community….

4. Start to visit DoD labs and establish the groundwork for technology transfer.



The PragBot I Data Collection Environment

During Year 1:· PragBot data collection interface implemented

• Two players in world with cards (each can hold 3). • Goal: pick suit, find 6-in-a-row

· PragBot symmetric corpus of 50 dialogues collected· NL analyzer built to map corpus to PAR· Guidance: “Develop a common simulation and experimentation

scenario … which is demanding and sophisticated….”, “build the corpus from a major search and rescue simulation/…/game….

10/16/2009 25

Example data: Cooperation

· % pragbot_chat_log_2007.11.01 AD at 19.48.20 EDT.txt · Player 2: i have 4H · Player 1: I want it! · Player 1: where is it? · Player 2: should i leave it for you somewhere? · Player 1: sure · Player 1: where are you? · Player 2: okay, where are you? · Player 1: I'm near the top · Player 2: i'm left side. · Player 1: next to the gap near the middle · Player 2: i'll leave the card in the upper left corner. · Player 1: awesome

SUBTLE Year 2 Overview

Last Year’s Annual Review

· First Annual Review: 3 Oct 2008 – UPenn• “Develop a common simulation and experimentation

scenario … which is demanding and sophisticated….”• Response: Pragbot 2.0


Talk 2

PragBot 2 scenarios are our targets

· The PragBot 2 world provides a simple yet relevant world for scenarios for testing human-robot communication using language

· Limits in world align with limits of current robot perception

· One extension: We will allow Jr to pick up objects in our test scenarios.• Will simulate in Robot demo


Workshop on Situated Understanding

· First Year Review: “Build a robot language community ….”

· When: two full days July 23-24, 2009 · Where: Institute for Research in Cognitive Science, Penn· Who: 26 participants, including many members of SUBTLE

advisory committee· Workshop Focus:

• Human-Robot Communication through NL and other contexts requiring recognition of goals and intentions

— Interactive fiction—Embodied agents—Social avatars—Avatars in games involving language communication —…


Workshop Talks· Four talks by SUBTLE participants · Dialog as Planning with Knowledge and Sensing

• Ron Petrick & Mark Steedman (Edinburgh)

· Towards a Robotic Architecture for Natural Spoken Human-Robot Dialogues • Matthias Scheutz (Indiana) & Kathleen Eberhard (Notre Dame)

· Intention Situated in Collaborative Dialogue• James Allen (Rochester)

· Models and Skills for Understanding Communicative Intentions • Matthew Stone (Rutgers)

· A Simple Computational Model of Interactive Language Comprehension• William Schuler (OSU)

· Engagement and Deixis for a Humanoid Robot • Candy Sidner (BAE)

· From Events to Natural Language in the Interactive Fiction System Curveship

• Nick Montfort (MIT)


Connections to DoD labs – Year 2

· “Start to visit DoD labs and establish the groundwork for technology transfer.”

· 2007—Visit NRL Talk: The generality of pragmatic inference: Message enrichment in multi- agent interactions (Potts)

· 5/09 HRI Workshop & Group Meeting sponsored by ARL’s Advanced Decision Architectures (ADA) Collaborative Technology (Yanco)

· 5/09—Visit ARL Talk: SUBTLE (Marcus) · 7/09—Situated Understanding Workshop (All)· 7/09—Kick-off meeting for new ONR MURI & Talk (Potts)


Behind plan: Two areas

• Corpus Collection• Implementation of Pragbot 2 more difficult than expected

• 3D implementation ambitious, given that game must completely download as webapp

• Initial game completed and limited data now collected, but scenario requires tuning.

• Connection of Pragmatics to Language• Alchemy system for Markov Logic good for small examples but• Alchemy can’t handle rich highly interconnected logical models• Expected: that’s why we were betting on McCallum’s Factorie• But we didn’t expect the limitations to hit so soon• Factorie beta due within a week or two!


Talk 7

Summary of progress

· Crucial Steps Forward• Architecture evolving as components connect

—LTLPAR—Pragmatics Robot’s expectations

• Pragbot 2 functioning• Two grad students to strong faculty positions, with continued

involvement (1 undergrad to HCI PhD program)• Situated Understanding Workshop building community

· Where behind• Just beginning to gather data• Factorie is crucial for progress

· Pieces about to come together


Schedule for the Day…8:35 Introduction Joe Myers8:45 SUBTLE: Overview Mitch Marcus9:30 Pragbot 2.0: Moving 'Pragbot' Language Interactions Toward

More Realistic SituationsChris Czyzewicz/Norm Badler

10:10 The interplay of linguistic & contextual inferences Chris Potts10:50 Break11:15 Integration and State Summarization Dan Hestand / Munjal Desai /Holly Yanco

11:55 Lunch12:30 p.m.

Meaning to motion: Transforming specifications to provably-correct control

Hadas Kress-Gazit/George Pappas

1:10 Integrating Linear Temporal Logic and a Parameterized Action Representation & Creating 3D Animated Human Behaviors for Virtual Worlds

Jan Allbeck/Norm Badler

1:50 Break2:05 Joint Inference for the NLP Pipeline:

Probabilistic Programming and the FACTORIE SystemDavid Smith/Andrew McCallum/Karl Schultz

2:45 SUBTLE Year 3 plans Mitch Marcus3:00 Advisory Committee Meeting4:00 Advisory Committee Initial Report4:30 Review Adjourns


BACKUP



Components built and tested

· PAR implemented and tested for SUBTLE world· High level autonomous actions added to robot

• Search, Find, Follow, Snapshot

· World Model Ontology DB and API implemented· LTL stress-tested in Urban Challenge enviroment

• Connection from Pseudo-English to LTL to Robot tested

subtle: overview (situation understanding bot through language and environment) umass amherst, umass...

Documents

overview slide

subtle year

subtle team

overview situation

penn mitch marcus

formal models of language

linguistic specification

environment umass amherst