patterns for natural language applications

38
Patterns for Natural Language Applications - beyond declaring Intents and Entities Simple patterns, like adaptive greeting, randomness, maintaining context, or predictive follow-up, can make an already good Voice User Interface spectacular Wolf Paulus

Upload: wolf-paulus

Post on 11-Apr-2017

295 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Patterns for Natural Language Applications

Patterns for Natural Language Applications - beyond declaring Intents and Entities

Simple patterns, like adaptive greeting, randomness, maintaining context, or predictive follow-up, can make an already good Voice User Interface spectacular

Wolf Paulus

Page 2: Patterns for Natural Language Applications

Wolf Paulus, 2016

Page 3: Patterns for Natural Language Applications

Wolf Paulus, 2016

Seinfeld Episode “The Pool Guy” Season 7, Episode 8 - Broadcast date: November 16, 1995

"Why don't you just tell me the name of the movie you have selected?" works — because it's so hilariously implausible

Page 4: Patterns for Natural Language Applications

Wolf Paulus, 2016

Number of Options in Top Menu

Gender of IVR Voices

IVR Input Methods

Page 5: Patterns for Natural Language Applications

Wolf Paulus, 2016

“Hey Siri, what’s the best sushi place

in town?”

“Talk to Siri as you would to a friend …”

Page 6: Patterns for Natural Language Applications

Wolf Paulus, 2016Majel Barrett-RoddenberrySusan Bennett Jennifer Lee Taylor

Siri Google Now Cortana Alexa

Page 7: Patterns for Natural Language Applications

Wolf Paulus, 2016Ben Galbraith - Senior Director at Google

“When I switched from Alexa to Amazon, to avoid a namespace collision with my daughter Alexis, the emotional feel of the device shifted from talking to a witty female computer, to interfacing with the Amazon corporate machine.”

Page 8: Patterns for Natural Language Applications

Wolf Paulus, 2016

Talking to an IVR Talking to a friendSpeak in Commands Having a Conversation / Shared Control

Expectation of Consistency Adaptivity / Randomness / Surprise

Follow a Strict Hierarchy Maintaining Context / Tolerate Interruption

Expectation of Unconcerned Response Expectation of Empathetic Response

Page 9: Patterns for Natural Language Applications

Wolf Paulus, 2016

Adaptive Having a Conversation

Context

personal and personable

Page 10: Patterns for Natural Language Applications

Wolf Paulus, 2016

March 9, 2016

The speed makes a crucial difference. Compared with the trudge of chatting with Siri, speaking to Alexa feels natural, closer to speaking to a human than a machine.

Page 11: Patterns for Natural Language Applications

Wolf Paulus, 2016

Alexa, ask Mindy about the

balance of my checking account

Mindy

soun

d fil

e

{ “intent” : ”getAccountBalance”,

“slots” : [ {“accountType” : ”checking”},

{“bankName” : ”null”}

] }so

und

file

“Text”: “Wolf, the balance of your …”

Wolf, the balance of

your …

Page 12: Patterns for Natural Language Applications

Wolf Paulus, 2016

about my {AccountTypeSlot} account balance about my {AccountTypeSlot} balance about my {AccountTypeSlot} {BankNameSlot} about my {AccountTypeSlot} {BankNameSlot} about the balance of my {AccountTypeSlot} about the balance of my {AccountTypeSlot} a {BankNameSlot} about the balance of my {AccountTypeSlot} account with {BankNameSlot} about the balance of my {BankNameSlot} account about the balance of my {BankNameSlot} {AccountTypeSlot} account about {AccountTypeSlot} account about {AccountTypeSlot} account at {BankNameSlot} about {AccountTypeSlot} account with {BankNameSlot} about {BankNameSlot} account about {BankNameSlot} {AccountTypeSlot} account what is my {AccountTypeSlot} account balance what is my {AccountTypeSlot} balance what is my {AccountTypeSlot} {BankNameSlot} account balance what is my {AccountTypeSlot} {BankNameSlot} balance what is the balance of my {AccountTypeSlot} account what is the balance of my {AccountTypeSlot} account at {BankNameSlot}

Sample Utterances

Interaction Model

Interaction Model .. Invocation Name: Url: https://…….

Intent Schema

{ "intent": "GetAccountBalance" "slots": [{ "name": "AccountTypeSlot", "type": "ACCOUNTTYPESLOT" },{ "name": "BankNameSlot", "type": "BANKNAMESLOT" } ], "intent": .. many more intents

},

Account Types

Custom Slot Types

Bank Names

.. many more dictionaries

Page 13: Patterns for Natural Language Applications

Wolf Paulus, 2016

{ “intent” : ”getAccountBalance”,

“slots” : [ {“accountType” : ”checking”},

{“bankName” : ”null”}

] }

Page 14: Patterns for Natural Language Applications

Wolf Paulus, 2016

“Alexa, talk to Mindy ” Session Started

onIntent (..)

Session Ended

“What bills are due today”

OnLaunch (optional)

“Good news, there aren’t .. “

“Hello Wolf, how can ..”

“Thanks” ::“Talk to you soon, Wolf”

idea: pre-fetch data

Page 15: Patterns for Natural Language Applications

Wolf Paulus, 2016

Mindy

public interface IHandler { /*** @return the name of the IntentHandler, e.g. to identify it in the JSON Interaction model.*/ String getName(); /** @return Set of the utterances that will trigger this handler, to be inserted into the Interaction Model. */ Set<String> getUtterances(); /** @return Set of the slots that this intent handler evaluates, to be inserted into the json file. */ Set<MSlot> getSlots(); … }

idea: Make each IntentHandler self-describing

Interaction Model IntentHandler

Called to create the Interaction Model

Implementation and Interaction Model

stay in sync

Wolf Paulus, 2016

Page 16: Patterns for Natural Language Applications

Wolf Paulus, 2016Wolf Paulus, 2016

Page 17: Patterns for Natural Language Applications

Wolf Paulus, 2016

public class Utils { /** * Randomly returns one of the strings from the provided String array. * * @param sa {@link String[]} * @return {@link String} */public static String random( final String[] sa ) { return sa[ new Random().nextInt( sa.length ) ]; }

} Random().nextInt(k): Returns a pseudorandom, uniformly distributed int value between 0 (inclusive) and the specified value (exclusive), drawn from this random number generator's sequence

Page 18: Patterns for Natural Language Applications

Wolf Paulus, 2016

Adaptive

Page 19: Patterns for Natural Language Applications

Wolf Paulus, 2016

if (! isValid( user )) // Unknown User return random( Greeting.init_greeting );

if (last_login == null) // Know User / 1st time here return random( Greeting.intro_greeting );

int hours = new Period(new DateTime(), last_login).getHours();

if (hours >= 24) // 1st visit today return random( Greeting.greeting );

else // returning user return random( Greeting.short_greeting );

Session Started

onIntent (..)

Session Ended

OnLaunch

:

History

idea: JodaTime or java.time(JSR-310) Java SE 8

Page 20: Patterns for Natural Language Applications

Wolf Paulus, 2016

1.Messages for a user that has not successfully provided credentials yet 1.<p><s>Hi, I am Mindy, your virtual financial assistant, helping you, to be good with your … 2.… 3.…

2.Messages for an introduction, user successfully provided credentials, 1st meeting 1."<p><s>Hello %s, nice to finally meet you!</s><s>Now that I have access to the information you maintain in Mint, I can help … 2.…

3.Messages for a standard greeting 1."Hello %s, what's your question?” 2."Hello %s, what do you want to know?", 3."Hi %s, welcome back, how can I help you today?"

4.Messages for a very short greeting 1."Welcome back, %s” 2."Hey, %s”; 3."What's up?"; 4.…

Page 21: Patterns for Natural Language Applications

Wolf Paulus, 2016

Hi there, welcome to Capital One.

You can ask me for things like your account balance, or make a credit card payment.

How can I help?

Wolf Paulus, 2016

Page 22: Patterns for Natural Language Applications

Wolf Paulus, 2016

Hi there, welcome to Capital One.

You can ask me for things like your account balance, or make a credit card payment.

How can I help?

Wolf Paulus, 2016

personal and personable

Page 23: Patterns for Natural Language Applications

Wolf Paulus, 2016

public class RS {

/** Messages for a very short greeting */

static final String[] short_greeting = new String[] { "Welcome back %s", "Hey %s", "What's up %s?" };

}

Resource

Usage /** Short personal greeting */

String result= String.format( Utils.random(RS.short_greeting), session.getUsername() );

idea: Keep logic out of Resource classes

Page 24: Patterns for Natural Language Applications

Wolf Paulus, 2016

Context

Wolf Paulus, 2016

Page 25: Patterns for Natural Language Applications

Wolf Paulus, 2016

“What were the last three transactions?”

“What is the balance of my checking account at Bank of America?”

Mindy: “The balance of your checking account at Bank of America is twenty-seven dollars.”

Page 26: Patterns for Natural Language Applications

Wolf Paulus, 2016

Just 470 calories

Page 27: Patterns for Natural Language Applications

Wolf Paulus, 2016

Intent: GetAccountBalance (accnt_type) (bank_name)

“What were the last three transactions” “What is the balance of my checking account at BofA”

Intent: GetTransactions (number) (accnt_type) (bank_name) (payee)

Page 28: Patterns for Natural Language Applications

Wolf Paulus, 2016

Intent: GetTransactions (number) Intent: GetAccountBalance (accnt_type) (bank_name)

Intent: GetDeposit (accnt_type) (bank_name) (payer)

Have I been paid? What were the last three transactions What is the balance of my checking account at BofA

Page 29: Patterns for Natural Language Applications

Wolf Paulus, 2016

Intent: GetDeposit () Intent: GetTransactions (number) Intent: GetAccountBalance (accnt_type) (bank_name)

Intent: NoIntent ..

intent = stack.pop() stack.clear()

“No” Mindy:”Are we still talking about your checking account at BofA?”

“Have I been paid?” “What were the last three transactions” “What is the balance of my checking account at BofA"

Page 30: Patterns for Natural Language Applications

Wolf Paulus, 2016

“No” Mindy:”Are we still talking about your checking account at BofA?”

“Have I been paid?” “What were the last three transactions” “What is the balance of my checking account at BofA"

Intent: GetDeposit ()

Page 31: Patterns for Natural Language Applications

Wolf Paulus, 2016

Shared Control / Having a Conversation

Page 32: Patterns for Natural Language Applications

Wolf Paulus, 2016

Every response object may also include a re-prompt, which is only played, if the user remains silent after the response text was played.

User: How am I doing in my coffee budget. Mindy: “So far you spent $spent in the $budget category and have a total amount of $budgeted”

User: <silence>

Mindy: “$user, would you like to know about your $parent_budget as well?”

User: Alexa, talk to Mindy. Mindy: “Welcome back Wolf, your portfolio went up 2.7% today, good job!”

User: What’s the balance of my IRA

Volunteering information

Page 33: Patterns for Natural Language Applications

Wolf Paulus, 2016Wolf Paulus, 2016

Page 34: Patterns for Natural Language Applications

Wolf Paulus, 2016

Page 35: Patterns for Natural Language Applications

Wolf Paulus, 2016

Page 36: Patterns for Natural Language Applications

Wolf Paulus, 2016

User: Alexa, talk to Mindy. Mindy: “Hi Wolf, welcome back, how can I help you today?”

User: What bills are due today? Mindy: “Good news, there aren’t any bills due today or tomorrow.”

User: What’s my news brief. Mindy: “Over the last seven days, your investment in ALPHABET grew 4.5%. Your second best performer was SOUTHWEST, growing 4.46%. On the other-side, your investment in APPLE lost 0.38%”

User: Thank you. Mindy: “See you later Wolf.”

idea: Whissell's Dictionary of Affect in Language

Page 37: Patterns for Natural Language Applications

Wolf Paulus, 2016

Page 38: Patterns for Natural Language Applications

Wolf Paulus, 2016

Thanks for Coming

Patterns for Natural Language Applications