speechcycle confidential confidential 1 optimizing natural language interfaces: no data like more...
TRANSCRIPT
Confidential 1
SpeechCycle Confidential
Optimizing Natural Language Optimizing Natural Language Interfaces: Interfaces:
No Data Like More DataNo Data Like More Data
SpeechTEK New York, 2007SpeechTEK New York, 2007
Jonathan Bloom & Roberto PieracciniJonathan Bloom & Roberto Pieraccini
Confidential 2
SpeechCycle ConfidentialSpeechCycle Confidential
Just as speech scientists crunch data to optimize a speech recognizer,
speech companies need to crunch data to optimize a call at the dialog level
as well.
Voice user interface (VUI) design needs to be based on quantitative data as
much as possible.
We will provide an example of quantitative research in the area of VUI
design.
We will ask, “What is the future of this research-based VUI approach?”
Executive Summary
Confidential 4
SpeechCycle ConfidentialSpeechCycle Confidential
Where there is data, we use it. At those times, it is a science.
All else is faith-based.
Religion is a fine thing. But our customers do not pay us for theology
lessons. They pay us to save them $$$ and keep their customers happy.
VUI = RELIGION + SCIENCE
Confidential 5
SpeechCycle Confidential
As much as possible, speech companies need to collect data in order to optimize their voice user interfaces at
every level – and in every facet - of the interaction, from recognition accuracy to prompt wording to dialog structure. In doing so, VUI will become less anecdotal
and more scientific.
The Point
Confidential 6
SpeechCycle Confidential
One needs the ability to compare two or more versions of an application…
…running at the same time.
…taking calls from the same population of callers.
…taking calls from that population at random.
…gathering data at a fast enough pace to meet customer deadlines.
Requirements for Exploration
Confidential 8
SpeechCycle Confidential
Example
“All of our agents are currently helping other customers. Let’s get started with our automated internet troubleshooter, so you don’t have
to wait. [chime] To begin, briefly describe the problem, saying something like “I can’t connect to the internet”, or you can say “What
are my choices”.
“All of our agents are currently helping other customers. Let’s get started with our automated internet troubleshooter, so you don’t have
to wait. [chime] Are you calling because you’ve lost your internet connection? [pause] Please say yes or no.”
Confidential 9
SpeechCycle Confidential
Example
RESULTS
INTRO followed by SLM = 20.4% automation rate
INTRO followed by YES/NO question = 22.6% automation rate
(CHI square) Statistically significant at .05 level
Confidential 10
SpeechCycle Confidential
Summary (Up To Now)
Strong opinions and qualitative usability tests should not be the only sources of VUI knowledge.
With the right tools, continual access to data, and with enough data, speech companies can make dialog experimentation a regular part of the product lifecycle.
At this point, other than the randomization script in the call flow, a lot of this process is manual, so now we need to ask, “How can we make this process more automatic?”
What’s next?
Confidential 11
SpeechCycle Confidential
Reinforcement Learning theory
A finite choice of actions
What the system does
A finite or infinite set of interaction states
Identify the factors that can influence the choice of the best action
Policy
Choose the action based on current state of the interaction
Maximization of a return function
Reinforce the choice of the actions that provide a positive return at any particular state
The process converges to a locally optimal policy
Confidential 12
SpeechCycle Confidential
Reinforcement Learning in dialog research
Research has proven that reinforcement learning can be used for automating the design of dialog systems
(Markov Decision Processes) E. Levin and R. Pieraccini (AT&T) , 1997
(Partially Observable Markov Decision Processes) S. Young (Univ. Cambridge, 2004)
Unfortunately, full dialog learning and optimization requires a lot of interactions
Academic research uses simulated users
Restricting the optimization to a small number of reasonable “competing designs”
Alternative competing designs at different points in the application
Exploration and Exploitation principle