ravenclaw
Post on 03-Jan-2016
51 Views
Preview:
DESCRIPTION
TRANSCRIPT
RavenClaw
Yet another (please read “An improved”) dialog management architecture for task-oriented spoken dialog systems
Presented by: Dan Bohus (dbohus@cs.cmu.edu)
Work by: Dan Bohus, Alex Rudnicky
Carnegie Mellon University, 2002
11-04-01 Modeling the cost of misunderstanding … 2
New DM Architecture: Goals Able to handle complex, goal-directed dialogs
Go beyond (information access systems and) the slot-filling paradigm
Easy to develop and maintain systems Developer focuses only on dialog task Automatically ensure a minimum set of task-
independent, conversational skills
Open to learning (hopefully both at task and discourse levels)
Open to dynamic SDS generation More careful, more structured code, logs, etc:
provide a robust basis for future research.
11-04-01 Modeling the cost of misunderstanding … 3
A View from far, far away
What did you just say ?
What’s your name ?
SELECT * WHERE …
Since that failed, I need you to push
button B
Can you repeat that, please ? Suspend… Resume …
Conversational Skills
Dialog Task Specification
Backend
Core
Let the developer focus only on the dialog task spec.: Don’t worry about misunderstandings, repeats, focus shift,
etc… merely describe (program) the task, assuming perfect knowledge of the world
Automatically generate the conversational mechanisms Examples
11-04-01 Modeling the cost of misunderstanding … 4
Outline
Goals A view from far away Main ideas
Dialog Task Specification / Execution Conversational skills
In more detail Dialog Task Specification / Execution Conversational skills
Conversational
DTS
Backend
Core
11-04-01 Modeling the cost of misunderstanding … 5
Dialog Task Spec & ExecutionCommunicator
Welcome Login Travel Locals Bye
AskRegistered AskName GreetUser GetProfile Leg1
DepartLocation ArriveLocation
Agencies and Microagents (for input, request, execute …) Handle Concepts
Execution with interleaved Input Passes. Execute the agents by top-down “planning” Do input passes when information is required
REMEMBER: This is just the dialog task
11-04-01 Modeling the cost of misunderstanding … 6
Handling inputsCommunicator
Welcome Login Travel Locals Bye
AskRegistered AskName GreetUser GetProfile Leg1
DepartLocation ArriveLocation
Input Pass Assemble an agenda of expectations (open concepts) Bind values from the input to the concepts Process non-understanding (if), analyze need for focus shifts Continue execution
11-04-01 Modeling the cost of misunderstanding … 7
Conversational Skills /Mechanisms
A lot of problems in SDS generated by lack of conversational skills. “It’s all in the little details!” Dealing with misunderstandings Generic channel/dialog mechanisms : repeats, focus
shift, context establishment, help, start over, etc, etc. Timing
Even when these mechanisms are in, they lack uniformity & consistency.
Development and maintenance are time consuming.
11-04-01 Modeling the cost of misunderstanding … 8
Conversational Skills / Mechanisms
More or less task independent mechanisms: Implicit/Explicit Confirmations, Clarifications,
Disambiguation = the whole Misunderstandings problem Context reestablishment Timeout and Barge-in control Back-channel absorption Generic dialog mechanisms:
Repeat, Suspend… Resume, Help, Start over, Summarize, Undo, Querying the system’s belief
The core takes care of these by dynamically inserting in the task tree agencies which handle these mechanisms.
11-04-01 Modeling the cost of misunderstanding … 9
Outline
Goals A view from far away Main ideas
Dialog Task Specification / Execution Conversational skills
In more detail Dialog Task Specification / Execution Conversational skills
Conversational
DTS
Backend
Core
11-04-01 Modeling the cost of misunderstanding … 10
Dialog Task Specification
Goal: able to handle complex domains, beyond information access, frame-based, slot-filling systems i.e. : Symphony, Intelligent checklists, Navigation, Route
planning
We need a powerful enough formalism to describe all these tasks: C++ code ? Declarative would be nice … but is it powerful enough ? Templatized C++ code … ?
11-04-01 Modeling the cost of misunderstanding … 11
Dialog Task Specification
A possible more formalized approach Tree of agents with:
Preconditions Success Criteria Focus Criteria (triggers)
Expressed mostly in terms of concepts Data, Type (basic, struct, array) Confidence, Availability, Ambiguousness,
Groundedness, System/User, TurnAcquired, TurnConveyed, etc…
11-04-01 Modeling the cost of misunderstanding … 12
An example DTS
UserLogin: AGENCYconcepts: registered(BOOL), name(STRING), id(STRING),
profile(PROFILE), profile_found(BOOL)achieves_when: profile || InformProfileNotFound
AskRegistered: REQUEST(registered) grammar: {[yes]->true,[no]->false,[guest]->false}AskName: REQUEST(name) precond: registered==no grammar: [user_name] max_attemps: 2InformGreetUser: INFORM precond: nameAskID: REQUEST(id) precond: registered==yes mapping: [user_id]DoProfileRetrieval: EXECUTE precond: name || id call: ABEProfile.Call >name, >id, <profile, <profile_foundInformProfileNotFound: INFORM precond: !profile_found
Given that the baseline is 259 lines of C++ code, this is pretty good.
11-04-01 Modeling the cost of misunderstanding … 13
Can a formalism cut it ?
People have repeatedly tried formalizing dialog … and failed We’re focusing only on the task (like in
robotics/execution) Actually, these agents are all C++ classes, so
we can backoff to code; the hope is that most of the behaviors can be easily expressed as above.
11-04-01 Modeling the cost of misunderstanding … 14
Other Ideas for DTS
4 Microagents: Inform, Request, Expect, Execute
Provide a library of “common task” and “common discourse” agencies Frame agency List browse agency Choose agency Disambiguate agency, Ground Agency, … Etc
11-04-01 Modeling the cost of misunderstanding … 15
DTS execution
Agency.Execute() decides what is executed next Various simple policies can be implemented
Left-to-right (open/closed), choice, etc
But free to do more sophisticated things (MDPs, etc) ~ learning at the task level
11-04-01 Modeling the cost of misunderstanding … 16
Input Pass1. Construct an agenda of expectations
(Partially?) ordered list of concepts expected by the system
2. Bind values/confidences to concepts The SI <> MI spectrum can be expressed in terms of the
way the agenda is constructed and binding policies, independent of task
3. Process non-understandings (iff) - try and detect source and inform user: Channel (SNR, clipping) Decoding (confidence score, prosody) Parsing ([garble]) Dialog level (POK, but no expectation)
11-04-01 Modeling the cost of misunderstanding … 17
Input Pass
4. Focus shifts Focus shifts seem to be task dependent.
Decision to shift focus is taken by the task (DTS)
But they also have a TI-side (sub-dialog size, context reestablishment). Context reestablishment is handled automatically, in the Core (see later)
11-04-01 Modeling the cost of misunderstanding … 18
Outline
Goals A view from far away Main ideas
Dialog Task Specification / Execution Conversational skills
In more detail Dialog Task Specification / Execution Conversational skills
Conversational
DTS
Backend
Core
11-04-01 Modeling the cost of misunderstanding … 19
Task-Independent, Conversational Mechanisms
Should be transparently handled by the core; little or no effort from the developer However, the developer should be able to write his own
customized mechanisms if needed
Handled by inserting extra “discourse” agents on the fly in the dialog task specification
11-04-01 Modeling the cost of misunderstanding … 20
Conversational Skills Universal dialog mechanisms:
Repeat, Suspend… Resume, Help, Start over, Summarize, Undo, Querying the system’s belief
The grounding / misunderstanding problems Timing and Barge-in control Focus Shifts, Context Establishment Back-channel absorption
Q: To which extent can we abstract these away from the Dialog Task ?
11-04-01 Modeling the cost of misunderstanding … 21
Repeat
Repeat (simple) The DTT is adorned with a “Repeat” Agency
automatically at start-up Which calls upon the OutputManager Not all outputs are “repeatable” (i.e. implicit
confirms, gui, )… which ones exactly… ?
Repeat (with referents) only 3%, they are mostly [summarize]
User-defined custom repeat agency
11-04-01 Modeling the cost of misunderstanding … 22
Help DTT adorned at start-up with a help agency Can capture and issue:
Local help (obtained from focused agent) ExplainMore help (obtained from focused)
What can I say ?
Contextual help (obtained from main topic) Generic help (give_me_tips)
Obtains Help prompts from the focused agent and the main topic (defaults provided)
Default help agency can be overwritten by user
11-04-01 Modeling the cost of misunderstanding … 23
Suspend … Resume
DTT adorned with a SuspendResume agency.
Forces a context reestablishment on the current main topic upon resume.
Context reestablishment also happens when focusing back after a sub-dialog Can maybe construct a model for that (given
size of sub-dialog, time issues, etc)
11-04-01 Modeling the cost of misunderstanding … 24
Start over, Summarize, Querying Start over:
DTT adorned with a Start-Over agency
Summarize: DTT adorned with a Summarize agency;
prompt generated automatically, problem shifted to NLG: can we do something corpus-based … work on automated summarization ?
Querying the system’s beliefs: Still thinking… problem with the grammars…
can meaningful Phoenix grammars for “what is [slot]” be automatically generated ?
11-04-01 Modeling the cost of misunderstanding … 25
Timing & barge-in control
Knowledge of barge-in location Information on what got conveyed is fed
back to the DM, through the concepts to the task level Special agencies can take special action
based on that (I.e. List Browsing) Can we determine what are non-barge-in-able
utterances in a TI manner ?
11-04-01 Modeling the cost of misunderstanding … 26
Confirmation, Clarif., Disamb., Misunderstandings, Grounding…
Largely unsolved in my head: this is next ! 2 components:
Confidence scores on concepts Obtaining them Updating them
Taking the “right” decision based on those scores:
Insert appropriate agencies on the fly in the dialog task tree: opportunity for learning
What’s the set of decisions / agencies ? How does one decide ?
11-04-01 Modeling the cost of misunderstanding … 27
Confidence scores
Obtaining conf. Scores : from annotator Updating them, from different sources:
(Un)Attacked implicit/explicit confirms Correction detection Elapsed time ? Domain knowledge Priors ?
But how do you integrate all these in a principled way ?
11-04-01 Modeling the cost of misunderstanding … 28
Mechanisms DepartureCity = <Seattle,0.71><SF,0.29> Implicit / Explicit confirmations
When do you leave from Seattle ? So you’re leaving from Seattle… When ?
Clarifications Did you say you were leaving from Seattle ?
Disambiguation I’m sorry was that Seattle or San Francisco?
How do you decide which ? Learning ?
11-04-01 Modeling the cost of misunderstanding … 29
Software Engineering
Provide a robust basis for future research. Modularity
Separability between task and discourse Separability of concepts and confidence
computations
Portability Mutiple servers Aggressive, structured, timed logging
top related