master thesis - simonis 3206963

Creating Realistic Agent-User Dialogs

P.P.G. Simonis

July, 2009

Utrecht University Supervised by:

Drs. N.L. Vergunst Dr. F.P.M. Dignum

Inf/Scr-08-66

Creating Realistic Agent-User Dialogs

Thesis number

Inf/Scr-08-66

Student information

Student P.P.G. Simonis Student card number 3206963

Master Program Agent Technology

Starting year 2007 Master’s Project information

Subject Social Interaction with Agents Title Creating Realistic Agent-User Dialogs

Supervising institute or organization

Utrecht University, Institute of Information & Computing Sciences, Faculty of Science

Daily supervisor Drs. N.L. Vergunst Second supervisor Dr. F.P.M. Dignum

Master Thesis – Creating Realistic Agent-User Dialogs – PPG Simonis 3206963

page 1 of 68

Acknowledgments I would like to take this opportunity to thank everyone that helped or supported me during the

creation of this thesis. First of all, I would like to thank Frank Dignum, one of my supervisors, who

introduced me to available research at Utrecht University which lead me to chose this research

subject. I would also like to thank Nieske Vergunst, my other supervisor, who provided me with

related work in the form of interesting articles. Her own work also proved to be very helpful along

setting up my own research. I would like to thank both for their help and support throughout my

research. Their guide and availability, in spite of busy schedules, proved very useful whenever I had

questions. In restructuring my thesis, they both also provided great help.

Then of course my family and friends who helped me think outside the box. Explaining what I was

doing helped me look at it from an outsider’s perspective. This can be very useful to regain overview.

Of course all their proof-readings of my thesis have provided me with helpful remarks. Last but not

least, I want to thank them for their patience since my graduation has been my main focus and

priority these last weeks.


page 2 of 68

Abstract When an agent and a user collaborate in performing tasks, communication comes into play. When

communicating with an agent, a user ideally wants it to be as communicatively skilled as a normal

person. When communicating with artificial intelligence like agents, a lot of aspects come into play.

Besides having to take care of any procedural steps, the agent will also have to be aware of the

domain and context its currently working in and use this in its reasoning and dialog.

To keep the research manageable, an example scenario on agent-user communication is introduced.

In this scenario, the agent assists the user in cooking. Not only will the agent guide the user through

the steps of a recipe, but it will also try to solve occurring problems. Both support and guidance

require communication.

Ideally the agent will be running on a companion robot. The agent will then represent the brain of

this robot. The recipe scenario introduced reduces a very large domain of companion robot to the

more simple task of supporting the user in preparing a recipe. Extending the scenario will ultimately

create a companion-agent to be used together with the a robot to create a companionship robot.

The object of this research is how to put the BDI concepts of agents to use in deliberating on dialog.

With this deliberation, the ultimate goal is to create realistic agent-user dialog. Goals and beliefs can

help structure not only the recipe procedure but also the flow of the dialog. Status information can

for example help guide the deliberation to a suitable answer to the user’s questions.


page 3 of 68

Table of contents 1 Introduction ................................................................................................................................ 5

1.1 The iCat ............................................................................................................................... 5

1.2 Agents ................................................................................................................................. 6

1.3 Recipe scenario ................................................................................................................... 7

2 Problem definition ...................................................................................................................... 9

2.1 Research objective .............................................................................................................. 9

2.2 Associated questions ......................................................................................................... 10

3 Related work ............................................................................................................................. 11

3.1 Agents ............................................................................................................................... 11

3.2 Human-robot interaction ................................................................................................... 14

3.3 Research............................................................................................................................ 15

4 Approach to the scenario .......................................................................................................... 17

4.1 Elements to support the dialog .......................................................................................... 17

4.2 Use of BDI features ............................................................................................................ 20

4.3 Recipe phases .................................................................................................................... 22

4.4 Recipe statuses .................................................................................................................. 24

5 Approach to the dialog .............................................................................................................. 26

5.1 Recipe selection phase ...................................................................................................... 26

5.2 Recipe planning phase ....................................................................................................... 27

5.3 Instructing phase ............................................................................................................... 28

5.4 Recipe finalization phase ................................................................................................... 31

5.5 Overall communication...................................................................................................... 31

6 Agent architecture .................................................................................................................... 33

7 Agent implementation .............................................................................................................. 35

7.1 Recipe selection phase ...................................................................................................... 35

7.2 Recipe planning phase ....................................................................................................... 38

7.3 Instructing phase ............................................................................................................... 41

7.4 Recipe finalization phase ................................................................................................... 43

8 Results ...................................................................................................................................... 44

8.1 Implementation result ....................................................................................................... 44

8.2 Resulting message based conversation .............................................................................. 45

8.3 Resulting ‘dialog’ ............................................................................................................... 46


page 4 of 68

9 Conclusion ................................................................................................................................ 48

10 Future work .......................................................................................................................... 49

10.1 Extended use of BDI elements ........................................................................................... 49

10.2 Extended dialogs ............................................................................................................... 50

10.3 User control....................................................................................................................... 51

10.4 Ontology for ingredients and utensils ................................................................................ 52

10.5 Timing and planning .......................................................................................................... 52

10.6 Integration with the iCat .................................................................................................... 53

Bibliography...................................................................................................................................... 54

Appendix 1: Diagrams and flowcharts ............................................................................................... 55

Recipe phases and statuses ........................................................................................................... 55

Dialog flowcharts .......................................................................................................................... 56

Appendix 2: 2APL illustrations ........................................................................................................... 59

Recipe selection phase .................................................................................................................. 59

Recipe planning phase .................................................................................................................. 60

Instructing phase .......................................................................................................................... 61

Recipe finalization phase............................................................................................................... 62

Implementation fragments ........................................................................................................... 63

Appendix 3: Example dialog .............................................................................................................. 65

The recipe ..................................................................................................................................... 65

The dialog ..................................................................................................................................... 66


page 5 of 68

1 Introduction Communication nowadays has many forms. Conversation alone can be performed in various

different ways. Mobile phones for example allow both use of text messages and phone conversations

whereas the internet allows chat conversations which can be extended with video and voice when

using a webcam and microphone. The conversational form dialog is the focus of my research. This, in

definition, refers to a conversation between two or more people. In my research however, we

assume a one-on-one dialog.

In computer science, software agents are introduced as pieces of software that act for a user or other

program. The idea is that agents are not strictly invoked for a task, but activate themselves. As with

humans, agents also communicate amongst each other. Agents however often have a good reason

for communicating whereas humans sometimes just “chat meaningless”. The agent’s reasons for

communicating could be retrieving information on the current world situation or trying to find out

what goals their partners have. The information they retrieve will then be put to use in their own

deliberation. Communication between agents is generally text-based because text messages are

relatively easy to generate and recognize when compared to speech for example.

The research I have done will form a contribution to Social Interaction with Agents’ (SIA) research, an

existing internal research program at Utrecht University. The goal of the SIA research is to realize an

agent with which a user can have a social interaction. Besides user-to-user or agent-to-agent

communication, users can also interact with agents and vice versa. This research is on exactly this

scenario; a dialog between an artificial agent and a user (human). An important keyword of my

research however is still missing in this description: realistic. My research will investigate how agent-

user dialogs can be made realistic. What exactly makes a dialog realistic will be explained later on.

1.1 The iCat Not only linguistic skills but also facial expressions have to be

taken into account. In order to give the artificial agent used for this

research a face, an iCat will be used. The pilot for the iCat was

developed and released by Philips in August 2005 (1). In the years

up to now, the iCat has been developed into a robotic agent

suitable for user interaction. The iCat is modeled off humans and

made for studying.

Philips describes the iCat as “a user interaction robot for studying

intelligent behavior and social interaction styles in the home. iCat

is aimed at developing software technologies for decision making

and reasoning. iCat 's interaction style is modeled of the

interaction between humans using speech, facial expressions and

intelligent behavior” (2).

Besides being a friendly companion as a game buddy, iCat can help to control the ambient intelligent

home switching on lights, heaters and other appliances. iCat can also help out in other duties such as


page 6 of 68

keeping track of everyone’s calendars and messages. iCat can even give advice when selecting music

and movies. According to Philips, “iCat has already proven to be a valuable research tool for studying

intelligent interaction styles”.

The iCat can be programmed by using Philips’ Open Platform for Personal Robotics (OPPR) software,

which is another advantage. The incorporated Philips’ Dynamic Module Library (DML) can be used to

make your own components, such as vision and speech recognition components (2). This technology

can be used to allow incorporating agents onto the iCat. This will allow the agents to control the iCat

by using them as ‘brain’ of this robot.

As illustrated on the left, OPPR is represented in 4 parts;

architecture, animation, connectivity and intelligence. To skip a

technical description of all inner workings of the iCat, I will only

briefly describe the OPPR system. For a more detailed

description of all the components, the iCat research community

(2) is a good source of information.

Inside the Architecture part of the OPPR system is the DML

Middleware solution. The DML Middleware defines a software

component model for developing modular applications. Every

module is an executable software component, which is

designed and implemented based on the DML component

model. This would in theory allow agents to run on the iCat. However, research on whether this

actually can be done and if so how, still needs to be done at this time.

Next to this opportunity to develop modular applications, the OPPR system also comes with an

Animation Editor and Animation Module. These can be used for creating animated motions for the

iCat robot by providing precise control over every actuator of the iCat. These can be used to generate

facial expressions for example, giving the agent an even more realistic face. Inside the Intelligence

part of the OPPR system is room for scripting to develop dialogues. This will however be taken care

of by the agent.

1.2 Agents As already mentioned above, an agent will be used to function as a brain for the iCat robot. The

realistic requirement is actually a reason why agents have been chosen to represent the artificial side

of the dialog. Agents are used to model human behavior which adds to the realism of the dialog.

Modeling a human onto an agent should be more easy than programming it in a ‘normal’

programming language. The agent will eventually use the available sensors and actuators of the iCat

in order to be able to communicate with the user.

For programming the agent, the choice has been made for “A Practical Agent Programming

Language” or in short, 2APL (pronounced: double-a-p-l). This language has been created at Utrecht

University. 2APL is a BDI-based agent-oriented programming language with a formal syntax that

supports an effective integration of declarative programming constructs such as belief and goals and

imperative style programming constructs such as events and plans. On top of that, 2APL allows an


page 7 of 68

effective and efficient implementation of an Multi Agent System (MAS) architecture through

combination of individual cognitive agents, a shared environment and multi-agent organization.

Primary reason to use 2APL is the ability to create cognitive agents based on the Beliefs, Desires and

Intentions (BDI) model. The use of this model allows using constructs as goals and beliefs to support

structuring both the procedure as the flow of the dialog. Another advantage of using 2APL over

another agent programming language, is the flexibility of plans. First of all, multiple plans concerning

the same goal can be separated by using a belief query (to check what the agent currently beliefs to

be true) to allow executing them in the correct order or at the correct time.

If the execution of an agent’s action fails, this failure can be repaired in 2APL with so-called plan

repair rules (PR-rules). A PR-rule indicates that if the execution of an agent’s plan (i.e., any plan that

can be instantiated with the abstract plan) fails and the agent has a certain belief, then the failed

plan should be replaced by another plan. When a matching PR-rule is found, this replacement can be

done during runtime.

Last but not least, 2APL has “programming constructs to improve the practical application of existing

BDI-based agent-oriented programming languages that have formal semantics” (3). These

programming constructs include operations such as testing, adopting and dropping declarative goals,

different execution modes for plans as well as event and exception handling mechanisms.

1.3 Recipe scenario Having a conversation with a artificial agent can take lots of different forms and can be on a limitless

number of subjects. To keep such a large area manageable, a recipe scenario is introduced. In this

scenario, the user is to prepare a recipe together with an iCat. The iCat will provide help to the user.

Helping in this case goes beyond just providing cooking information in the form of reading

instructions, keeping a timer or sounding an alarm. The iCat should also be able to offer performing a

task for the user for example. Other help could be explaining an action in the recipe in a more

detailed way. Providing support and solutions when problems occur of course is another vailuble way

to help the user.

In this scenario the iCat agent can be seen as a kitchen help with whom the user can have a dialog.

This dialog of course is set in the cooking context. Because of the kitchen setting and the cooking

context, the conversational boundaries are limited. This will make the research, and projects

belonging to it, more manageable. Other reasons for using a cooking scenario as Grosz and Kraus (4)

have recognized are:

(1) unlike tasks such as constructing space stations, making a meal is an undertaking for which

almost all readers have first-hand knowledge and good intuitions.

(2) This task contains the essential elements of the typical collaborative task in which computer

systems and robots in particular may be expected to participate:

- limited resources (including time constraints),

- multiple levels of action decomposition,

- a combination of group and individual activities,


page 8 of 68

- partial knowledge on the part of each agent, and

- the need for coordination and negotiation.

As Grosz and Kraus also mention, the previous work using this domain (such as the work on plan

recognition) provides a baseline for comparison. So using this scenario not only makes it more

manageable but also allows using ideas of other research. This has also been a reason for the

researchers at Utrecht University to also use this recipe scenario.


page 9 of 68

2 Problem definition This chapter will define the primary object of my research. There is a main question behind my

research for which, in doing this research, I hope to find an answer. Alongside this main question,

sub-questions form. These might form along the line of the main question or submerge as several

aspects of the research are looked into.

2.1 Research objective To start this chapter, I will introduce the main object of my research and the related question. The

title of this document already gives away the research objective: creating realistic agent-user dialogs.

Of course these agent-user dialogs will ideally be just as realistic as when the user is having a dialog

with any other person. The question related to this can then only be: “How do you create realistic

agent-user dialogs?”. Of course performing the research in the domain as mentioned earlier, this will

also reduce the size of the question. The pursuit of creating a realistic agent-user dialog will however

not change.

This main question brings along several different other questions, the first and most intriguing

stating: “When can we call an agent-user dialog realistic?” This for me is mostly determined by what

the agent asks the user and how the agent reacts to the user. Of course during the recipe procedure,

the agent will expect input from the user. The agent will communicate with the user in order to get

the information it needs. Besides the expected informational needs of the agent, the user might also

require information from the agent. These questions, for example a question on what to do next,

might be asked at unexpected timing. Even problems with kitchen utensils might occur during

execution of actions. A water boiler malfunctioning or the water supply temporarily being cut off

could have drastic consequences for a recipe. When these problems occur, the user might consult

the agent for a suitable solution.

The agent should be able to react to remarks and questions of the user. This requires the agent to

relate the question of the user to the current situation. Understanding and answering the user is

needed in order to reach their goal. Before the agent can react to the user, it needs to process the

provided information and deliberate on its answer.

Besides finding out what the user is asking, the agent also has to ask itself why the user asked this

question. To be able to do this deliberation, the agent should be aware of the conversational context

as well as the domain context. An interesting thing to look at along these lines is; What

conversational information provided by the user (or retrieved from the world) the agent can use in

its deliberation. Of course even as interesting is the accompanying question on; In what way

conversational and environmental information should be stored inside the agent to allow him to

work with it efficiently when communicating with a user.

Another feature which adds to making dialogs more realistic is the fact that the dialog is goal-

directed. This ensures utterances continue on previous remarks, adding structure to the dialog.

Having a goal also allows for questions in lines of achieving it. The agent could for example request


page 10 of 68

the user to do something in line of the goal they are trying to reach. Of course this goal can also be

used by the agent to deliberate on the question ‘why did the user ask or tell me this?’.

As an illustration of the sort of dialog we want to achieve, an example dialog is added in appendix 3.

This dialog feels realistic, to us at least, in the questions which are asked and remarks that are made.

Cohen and Levesque (5) used an dialog which did not feel realistic. Their example of a phone

conversation between an expert and an apprentice, to my opinion, has too many confirmation

points. It seems each instruction followed by a request for confirmation. The conversation itself

however is over a different medium (by phone). The lack of eye-contact might cause this need for

more confirmation checks. Being able to see each other is an advantage in the kitchen scenario when

needing confirmation. The iCat could ideally only use its camera to confirm the user has started

working on executing an action.

2.2 Associated questions A considerable amount of the research will be on examining how to implement the kitchen and

cooking domain into an agent. For example, how will recipes be represented? Of course there’s a lot

more to this then just copying the text from a cookbook. The agent needs to be able to work with it.

A second interesting section of this research is the information the agent keeps on the user. When

for example the agent believes the user can perform a certain action, it doesn’t need to explain to

the user how it is performed.

Also interesting in this research are the different phases of cooking a dish combined with

communication. The planning of a recipe requires communication just as the execution of the recipe

actions themselves. Communication in different phases might have totally different meaning. This

means the agent needs knowledge of the current situation in order to be able to communicate

correctly but also realistically.

My research will therefore examine how beliefs (on the current situation), goals and communication

can be combined to create more realistic dialogs. To illustrate the ideas produced alongside this

research, a 2APL agent will be created in which these techniques will be implemented. It is perhaps

the most interesting to see how the agent combines status information and received messages in its

deliberation in order to come to a suitable reply or course of action.


page 11 of 68

3 Related work As the title reveals, this chapter will focus on work related to my research on creating realistic agent-

user dialogs. Related research is discussed and examined on whether it is useful to use along my

research or not. Some research might link closely to my research objective where other concepts

form a general idea which might help implementation alone. Several general concepts like the

definition of an agent as well as an introduction to the BDI model and 2APL will pass the review first.

3.1 Agents Because the notion of ‘an agent’ differs across literature, I have chosen a definition of an agent.

When I started my Agent Technology master at Utrecht University, this definition was given to

illustrate the concept ‘agents’. This definition originates from M. Wooldridge’s book “Reasoning

about rational agents” (5).

Agent definition Agents are software / hardware entities that display a certain degree of autonomy / initiative and are

proactive / goal-directed. Agents are mostly described in terms of having ‘mental states’ (‘strong’

notion of agency). They show informational and motivational attitudes.

Furthermore, agents have the following properties:

1. Situated: acting in an environment.

2. Reactive: able to react adequately to (unexpected) situations arising in the environment.

3. Pro-active: setting / pursuing its own goals.

4. Social: able to co-operate with other agents

This definition may look extensive for such an abstract notion. If you read carefully however, you can

see this definition only hints on some agent features and displayed behavior. It leaves out how these

should be implemented. This is left for the creator of the agent to decide. I have used this definition

throughout several different courses at Utrecht University and found it suiting in every situation.

The BDI-model In the above given definition, terms like ‘mental states’ and ‘goals’ already provide hints towards the

BDI-model, a software model developed for programming intelligent agents. The letters BDI stand for

Beliefs, Desires and Intentions. Besides these three concepts, a BDI-agent also has Plans at its

disposal. These four concepts will be briefly discussed below.

Beliefs represent the informational state of the agent. In other words, the agent’s beliefs about the

world including itself and other agents. Beliefs can also include inference rules, allowing forward

chaining to lead to new beliefs. Often, this information is stored in a database (a belief base). By

using the term belief instead of knowledge, the fact that the agent’s beliefs might be false is

incorporated.

Desires (or goals) represent the motivational state of the agent. They represent objectives or

situations that the agent would like to accomplish or bring about. Examples of desires might be: ‘go

to the party’ or ‘become rich and famous’. Because the above given agent definition uses the term

goals, I will also use this term rather than using desires.


page 12 of 68

Intentions represent the deliberative state of the agent: that what the agent has chosen to do.

Intentions are goals to which the agent has to some extent committed. In implemented systems, this

means the agent has begun executing a plan.

Plans are sequences of actions that an agent can perform to achieve one or more of its goals. Plans

may include other plans: a plan to make spaghetti and meatballs for example includes a plan to boil

water to prepare the spaghetti in.

JADE and 2APL JADE is an abbreviation for Java Agent DEvelopment Framework (6). JADE is a software framework

for multi-agent systems which is created in Java. The JADE platform allows coordination of multiple

agents. Besides coordination, JADE also offers the ability to use an agent communication language

for communication between agents. Every JADE platform contains a main container containing an

agent management system (AMS) and a Directory Facilitator (DF) agent. The AMS is the only agent

that can create and kill other agents, kill additional containers, and shut down the platform. The DF

agent implements a “yellow pages”-service which advertises the services of agents in the platform so

other agents requiring those services can find them.

As mentioned earlier 2APL is an agent programming language created at Utrecht University (7). The

link between 2APL and JADE is that the 2APL platform is built on JADE (6) and uses related tools.

These tools for example allow monitoring mental attitudes of individual agents, monitoring their

reasoning and monitoring communications. 2APL is provided with its own Integrated Development

Environment (IDE).

All 2APL agents consist of the following parts: BeliefUpdates, Beliefs, Goals, Plans, PG-rules, PC-rules

and PR-rules. Next to the beliefs and goals already introduced with the BDI model, belief updates and

rules are introduced. The rules are used for different purposes. PG-rules specify the plans an agent

can generate to complete a certain goal. The PC-rules are procedural rules. These generate plans as a

reaction to either incoming messages, external events or the execution of abstract actions. The PR-

rules are rules for plan repair. These will get triggered when a plan failed and needs to be replaced

(repaired) by another plan. The belief updates are rules with pre- and post-conditions for updating

the agent’s beliefs.

The basic BDI-elements thus return in 2APL in the form of Beliefs, Goals and Plans. The Beliefs are

represented in logical form in the belief base. This belief base consists of facts (about the world or

the agent itself) which the agent holds to be true. These facts can also be formed as formulas

containing variables which the agent can then substitute to reason with its beliefs and deduce new

information. The goal base represents the goals the agent currently has. The goals are in formula

form as well. Last but not least, there is a plan base. These plans are made up out of actions the

agent can perform. The plans can be accompanied by a head. This head then forms a condition

needed to hold before the plan will be executed.

The last element of 2APL I will discuss here is the deliberation cycle. This cycle contains steps which

will cause the agent to adopt plans and execute the corresponding actions in order to reach the goals

the agent has. The discussion of this cycle will illustrate the deliberation process within a 2APL agent.


page 13 of 68

start Apply all PG-rules

Execute first action

of all plans

Process external events

Process internal events

Process messages Remove reached goals

Sleep until external

events or messages

arive

Rules applied,

Plans executed,

Events or Messages

Processed?

no

yes

The 2APL deliberation cycle illustrated above consists of 5 steps. The first is to apply all PG-rules. This

will generate plans for the agent according to its goals. Executing these plans will enable the agent to

(come closer towards) completing the goal. Second is to execute the first actions of the plans the

agent has generated. Next is to process external events such as received messages. These can also

lead to adoption of a plan through PC-rules. Step 5 is to process internal events. These contain for

example errors occurring on execution of plans.

After these 5 steps, the agent will check if it has done something in the last cycle. If so, it will perform

the cycle again. If not, the agent will enter sleep mode until external events or messages arrive.

When they do, the agent enters the cycle again. This way, the agent will generate plans (intentions)

to achieve active goals. This deliberation cycle is also adaptable so the programmer can actually alter

the order in which the agent handles events and plan generation.

More on 2APL such as the latest news can be read on the official website with Utrecht University (8).

How to program agents in 2APL is described in the 2APL user guide available through that same

website.


page 14 of 68

3.2 Human-robot interaction Communication nowadays has various forms that all are very different from each other. Just like

normal communication, communicating with artificial intelligence (AI) has gone through several

phases. At first the communication with AI was limited to text input. Nowadays, communication with

AI is growing towards Natural Language Processing (NLP) and Speech Processing. Although NLP is also

needed with text-based input, speech processing clearly refers to verbal communication with AI.

Human-robot interaction (HRI) studies interaction between users (people) and robots. The basic goal

of HRI is to develop principles and algorithms to allow more natural and effective communication

and interaction between humans and robots. This requires knowledge on multiple disciplines like

natural language understanding and robotics but also artificial intelligence and social science. Having

these principles and algorithms, the end-goal is creation of robots which can coordinate their

behaviors with the requirements and expectations of humans.

Although the HRI research has a very broad range, several aspects belonging to HRI are interesting

along the lines of my research. “Human-robot collaboration” for example is a field to which my

research could be accounted a member of. Although the iCat is not as skilled in performing actions as

the user (caused particularly due to a lack of limbs), the user and iCat are collaborating in performing

a recipe. Of course the iCat compensates its lack of actuators by supporting the user on other fields

such as keeping information on how to perform a recipe.

Second, the field “Social robot” also belongs to HRI research. This is of course the ‘ultimate’ goal of

the Social Interaction with Agents research at Utrecht University; to create an agent which can be

integrated with a robot to create a social robot. However, as the term ‘social robot’ describes, we

need a robot as physical representation of the agent. This could be taken care of by enabling the

agent to run on the iCat.

When looking at talking to artificial intelligence, we can also discover robots described as

“chatterbots”. A chatterbot is a type of conversational agent, a computer program designed to

simulate an intelligent conversation with one or more human users via auditory or textual methods.

Most chatterbots “converse" by recognizing cue words or phrases from the human user. This allows

them to use pre-prepared or pre-calculated responses which can move the conversation on in an

apparently meaningful way without requiring them to know what they are talking about.

Chatterbots however usually do not take part in goal-directed conversations. They “just talk”. Most

chatterbots do not attempt to gain a good understanding of a conversation to allow them to carry on

a meaningful dialog. This is a large difference considering our target agent, which is clearly directing

its conversation based on preparing a recipe and needs understanding of the conversation at hand.


page 15 of 68

3.3 Research Several fields of research provide a background to my research. Some work will relate closely

whereas other work can be used in specific areas. Cohen and Levesque in their paper “Confirmations

and Joint Action” analyzed features of communication that arise during joint or team activities (9).

When communicating on joint action, a lot of elements come into play. For example, are both parties

required to confirm everything that’s said? And how will things be confirmed? Can one assume

messages will reach the receiver? Cohen en Levesque specify joint intention to bind teams together.

Furthermore, they claim an account of joint intention is needed to characterize the nature of many

situation-specific expectations properly.

Cohen and Levesque assume “that the agents jointly intend to perform together some partially

specified sequence of actions.” In the recipe scenario, both user and agent intend to prepare a

recipe. What actions are used to reach the final result is not specified, just the result they would like

to reach. They make an additional statement that “commitment to an action sequence gives rise to a

commitment to elements of that sequence”. This can be illustrated as follows in the recipe scenario;

if you want to make spaghetti Bolognese, you have to make the required pasta sauce.

On confirmation in conversation, Cohen and Levesque write: “both parties should be committed to

have the goal to attain mutual belief of successful action”. They argue because both parties jointly

intend to engage in the task, the requirement arises for confirmation of actions. One party is

supposed to confirm whereas the other party may hold him to it. This can be used in the recipe

scenario when the agent has instructed the user. The user is suppose to react to this instruction

whereas the agent may request confirmation from the user.

Other closely related research is the work of N. Vergunst et al. on “Agent-Based Speech Act

Generation in a Mixed-Initiative Dialogue Setting” (10). Their paper presents a system that acts both

proactively and reactively, accounting for a mixed-initiative interaction. Furthermore, they use the

same scenario of a cooking assistant alongside treating some general principles for mixed-initiative

systems.

Their system is proactive in that it is goal-directed and that it takes initiative to check whether a

certain recipe is possible given the circumstances, instead of blindly following the user's commands.

It also keeps track of the user's capabilities and tailors its instructions to them. The system is also

reactive, waiting with its next instruction until it believes that the current task is done. The system

uses the concept of joint goals, which translate into joint plans that consist of tasks that will be

performed by one of the participants. It also uses communication actions to make sure all

participants are aware of the current status of the joint plans and goals. As will follow in the

illustration of my own research, I’ve used a large amount of the presented concepts in my own

research and agent implementation.

An interesting field belonging to Human-Robot Interaction research is “Dialog management”.

Research done in this field concerns use of a dialog system in robots (or agents) to converse with a

human with a coherent structure. Dialog systems have employed text, speech, graphics, gestures and

other modes for communication on both the input and output channel. Some systems include all

forms of input and output channels where others only incorporate a subset.


page 16 of 68

An example of a dialog system is OVIS (11). OVIS stands for: Openbaar Vervoer Informatie Systeem,

which is Dutch for ‘Public Transport Information System’. OVIS is developed in order to make it

possible to answer more calls on information about various forms of public transport. They decided

to use a spoken dialog system to try and automate part of this service. This dialog system is

specialized on dialog input and output. Since it concerns communication through phone, gestures,

text and graphics are not needed.

There are lots of dialog systems already out there. These can be categorized alongside different

dimensions such as modality which concerns text-based systems, spoken dialog systems and multi-

modal systems. Another way to categorize dialog systems is by separating them according to

initiative. The user might take initiative in the system or the system might take the initiative. Of

course a mixture is also possible. A last possible way of categorization I will introduce is on

application. Dialogue systems can of course be categorized based on what they were designed to be

used for. This could be ‘information service’ (such as OVIS), entertainment, companion systems or

even healthcare.

What is principal to any dialog system, no matter what it is designed for, is the dialog manager. The

dialog manager is a component that manages the state of the dialog and the dialog strategy. In the

created agent, we will see this is taken care of by the dialog module. The dialog strategy can be any

of the following three strategies; “System-initiative”, “User-initiative” or “Mixed-initiative”. In

“System-initiative” strategy, the system is in control to guide the dialog at each step while with the

“User-initiative” strategy, the user takes the lead and the system responds to whatever the user

directs. Of course in the “Mixed-initiative” strategy users can change the dialog direction. The system

follows the user request, but tries to direct the user back the original course. This is the most

commonly used dialog strategy in today's dialog systems. As will be illustrated further on also, the

current implemented agent applies the “system-initiative” strategy.


page 17 of 68

4 Approach to the scenario This chapter will elaborate on some principles that are of interest for this Master’s Project. Based on

the principles described in this chapter, I’ve taken on this Master’s Project. Several aspects which the

agent can use in order to come to a realistic answer to the question or information provided by the

user will be discussed. Furthermore, the way BDI features are used throughout the recipe scenario

will be presented, as will be the approach I took towards splitting the recipe scenario into

manageable phases. Last but not least, the expected dialog during these phases will be presented

alongside flowcharts.

4.1 Elements to support the dialog As mentioned in the problem definition, the agent needs to keep the dialog realistic along several

lines like being able to answer to unexpected remarks of the user. To be able to do this, the agent

can use conversational features, shared concepts between user and agent and also internal data like

beliefs and or goals. This section will elaborate on the elements the agent can use to support keeping

the dialog realistic.

4.1.1 Context

In order to come to an realistic answer to the user’s questions or remarks, the agent can use two

contexts available. The agent can make use of the conversational context as well as the domain

context. It can use this information to guide its deliberation on the provided information. This

deliberation should then lead him to take action like answering the user’s questions, performing an

action or adopting a goal. The following two examples illustrate this through a question the user asks

and a remark the user makes.

Imagine the user asking the question: “Can you turn on the stove?”. The agent should realize using

the kitchen setting in context of cooking a dish, the user wants to use the stove in the cooking

process. Using the recipe at hand, the agent can determine the stove is needed to for example boil

water in a pan to cook the pasta in. The agent can of course also first ask: “Why do you want me to

turn the stove on?” When the user replies: “To boil water on”, the agent could react with a simple

conformation and turn the oven on. The agent could also provide the user with an alternative such

as: “You could also use a water boiler to do this. Do you want to do this instead?”.

Another situation is where the user informs the agent: “The water boiler can’t be turned on”. Using

its internal recipe, the agent can deduce the user needs the water boiler to boil water to put the

pasta in. Of course it could also ask “Why do you need the water boiler?”. Now knowing water needs

to be boiled and the water boiler is broken, the deliberation could lead the agent to an alternative

approach. The agent can reply: “I suggest to use a pan to boil water”.

In the above situation, the user took initiative and asked or informed the agent. The agent will act

differently with other levels of user control. When the user has a high level of control, the agent can

wait until the user asks questions on steps to take (as shown above). When there’s a low level of user

control, the agent should ideally take initiative and walk the user through the process. The agent will

help the user performing the recipe by asking the user to perform the actions the recipe consist of.


page 18 of 68

This way of instructing the user indicates low level user control. Since the user can always ask

questions or make remarks, the level of user control can also switch when the user takes initiative.

4.1.2 Shared concepts

Another aspect to help structure the conversation is the fact that the agent and the user share

certain things. First of all, they’re joined by the task of preparing a recipe. They both have the same

goal, a joint goal (4). Because the goal is shared, the agent knows the goal of the user. The agent

therefore knows that anything the user attempts will most likely be to achieve its goal of cooking the

selected dish. This goal can be reached by executing a plan. Since both participants (user and agent)

are collaborating to reach the goal, this plan can be described as a collaborative plan. In the sections

below, both the joint goal and the collaborative plan will be discussed as well as their use in the

scenario.

Joint goal The joint goal will trigger plans inside the agent which leads the agent to handle the needed steps in

creating a dish together with the user. Until the recipe has been prepared, the goal will keep

triggering plans, it will keep the agent committed to executing plans.

Throughout the cooking process, the agent will have different sub-goals which bring it closer to

reaching its final goal of completing the recipe. Whenever the agent is interacting with the user,

these sub-goals will provide information of the current situation. This can be used when deliberating

on an answer to a question of the user. However, knowing the user wants to cook a dish is not very

detailed information.

To allow for more detailed status information, a recipe status can be created. This status illustrates

the agent’s location in the recipe procedure like “searching for a recipe” or “instructing”. The recipe

status will change along the progress the agent and user make in executing the recipe. The status can

be used to put questions of the user into context and of course to keep the agent updated on what

steps to take next.

Collaborative plan Of course because of the cooperation between the user and agent, a collaborative plan (4) should

form. The most important feature of the collaborative plan in this scenario is the question of who

(agent or user) will perform which action. To once again keep it simple and manageable, the choice

has been made to use a single person (user or agent) per action. The agent will either tell the user it

will perform the action or ask the user to perform it.

Grosz and Kraus in their work on collaborative plans also talk about partial and complete plans. An

agent has a complete plan when it has completely defined the way in which it will perform an action.

Furthermore, they make a distinction between individual (formed by a single agent) and shared plans

(formed by a group of agents). These two separations lead to a number of possible plan-type

combinations. As will be illustrated later on, the agent will create the plan for executing the recipe.

The user is introduced to this plan because he1 initialized the procedure (he decided to prepare a

recipe). Since the iCat is a simple robot without a lot of practical cooking skills due to a lack of limbs,

1 Of course the user could also be female. In the continuing of this document however, we will use ‘he’ to refer to the user. To avoid confusing the user with the agent, the agent will continue to be referred to as ‘it’.


page 19 of 68

the user will have to perform most actions himself. The agent will thus come up with a plan to

prepare the recipe and request the user to perform most actions.

4.1.3 Internal recipe

Another guide in structuring the conversation is the internal recipe the agent has to fall back on. We

assume the agent has a database of recipes available to work with. Each recipe consisting of sub-

steps following each other to prepare the recipe. Responsibility for creating and managing this

database lies with the programmer of the agent. Alongside implementing the agent, an example

scenario is implemented according to the dialog illustrated in appendix 3. The recipe database used

in this example implementation only contains 2 recipes, enough for a simple example. When the

agent will actually be used in the kitchen, this database of course needs much more recipes. The

agent will work with this recipe database when searching a recipe and creating an instruction list. The

programmer will be responsible for adding, removing or altering any recipes in this database.

The internal recipe is especially useful in the recipe scenario when the agent is instructing the user.

The agent can divide the tasks to be done between itself and the user. Assigning a task to the user of

course means the agent will need feedback. First of all on whether the user accepts, second on if the

task has been done.

Another interesting thing to look at here is when the user encounters a problem. Of course people

sometime ask a question when they’re stuck. This can also be seen as initiating a sub-dialog.

However, if the user fixes the problem without help of the agent, the agent probably will not have

noticed this problem ever occurred. When the user is less experienced, he could of course ask the

agent what to do in the current (problematic) situation. When the agent has just instructed the user,

it can use this information in trying to find what went wrong and possibly fix the problem. Another

approach the agent might take is analyzing the action and finding an alternative which also fits within

the recipe.

Take a pan Boil water

Boil water in

water boiler

Fill pan

with water

Turn on stove

Put pan

on stove

Wait for water

to boil in pan

Put boiled

water in pan

Put pasta

in pan

Wait for pasta

to be readyDrain water

Optional:

Put a liitle olive

oil in pan

Use lid of pan

to drain water

Put pasta in

colander

Put the pasta

back in to the pan

Cook pasta

Put pan

on stove

The agent’s internal recipe can be represented in tree form as illustrated above. Nodes represent

actions and children of a node represent their sub-actions. As you move down one level in the tree,


page 20 of 68

the actions get more and more atomic. We assume all composite actions (including the recipes

themselves) can be decomposed into these atomic actions.

Some actions might have an overlap in sub-actions (such as boil water as illustrated in the recipe

tree), others might have two completely different approaches (like draining the water). Because

some actions can be done in different ways, this means the agent can make a choice. These

alternative approaches to a single action can also be used when for example one of them failed.

This internal representation will be a guide to what action the agent should ask the user to do next.

The internal recipe will help structure the dialog since the agent can use it to keep the instructions in

the correct order. As already mentioned above, it also helps the agent to determine what was the

last instruction given to the user when a problem occurs.

4.2 Use of BDI features This section will illustrate how the BDI features are used inside the agent to manage for example

storage of recipes, creation of the instruction list and generating intentions. The BDI features will, as

illustrated below, not only be used for procedural steps but they will also be used in conversational

steps and efficient storage of facts.

4.2.1 Plans

The plans the agent has available are used to help structure the scenario in defining the actions to be performed after each other. This sequence of actions can of course also contain nested plans. This allows for a layered approach where high level plans can be worked out into sub-plans. This is exactly how I started working out the scenario: first on a high level and then down to lower and lower levels. The highest level of steps I created into recipe phases, which I will describe after this section on the use of BDI features. The level below define the agent’s responsibilities per phase. When for example selecting a recipe, the agent is responsible for searching its database for a recipe, checking the recipe and confirming the chosen recipe with the user. On an even lower level, we arrive at the plans. These implement the agent’s actions in sub-steps. Having this sequence of steps available helps structure the scenario for the agent.

4.2.2 Goals

Goals in 2APL can be used to trigger PG-rules. These rules will create plans to reach these goals. As is

defined by the BDI-model, goals of the agent are used to represent objectives. Goals can be used to

keep the agent focused on finishing the recipe procedure. During the execution of the recipe

procedure, other goals come into play to keep the agent committed to achieving a certain sub-goal

such as creating the instruction list.

4.2.3 Beliefs

When going through a recipe scenario, the agent needs to keep track of several kinds of beliefs. Not

only does the agent need to keep track of the current state of the kitchen containing available

ingredients and utensils. Its beliefs should also incorporate recipes and their corresponding

(sub)actions as well as skills of both the user and the agent itself. To allow working with these beliefs,

they need to be stored efficiently in the agent’s belief base. Updating these beliefs is another

concern which will also be discussed in this section.


page 21 of 68

Storage of facts For storing several aspects of the recipe procedure such as skills or the instruction list, choices have

been made. Storing beliefs in a way the agent can easily work with them is something not to be taken

lightly. Deliberation needs easy access of required facts.

For storing recipes in the agent’s belief base, lists are used. A recipe is stored alongside a code for

identification, a description for searching purposes and the first level of sub-actions inside a list. This

list can consist of atomic actions, composite actions or a combination of these two. The sub-actions

of a composite action are also stored in a list-form. Composite actions can therefore themselves be

seen as a sub-recipe.

Storing decompositions of composite actions alongside the recipes allows for reuse of these

composite actions in other recipes. It also allows for different decompositions of one action. Boiling

water can for example be done in a water boiler but also in a pan. Atomic actions are also stored

alongside a list but this list is empty since they have no sub-actions. As already mentioned in the

previous section when discussing the internal recipe, any defined composite action (a recipe can also

be seen as a composite action) should be able to be decomposed down to the level of atomic

actions.

The atomic action level has been introduced to prevent having to decompose an action like ‘take a

pan’ any further. This level is the lowest in action decomposition we will use in recipes. When

creating test recipes, this level of actions was determined by taking into account ‘basic kitchen skills’.

If the iCat agent is to help someone in preparing a recipe, we assume for example this user knows for

example what a pan is.

Ingredients are stored alongside their available amount (stock(eggs,12) for example) whereas

utensils are stored alongside their current status (like ‘available’, ‘filled with water’ or ‘dirty’). For

storing skills, a very basic approach has been chosen. The skills are saved as a belief statement

coupling the agent or a user to an action when the required skills are available. The actions

referenced in these statements could even refer to a (sub)recipe. The example below illustrates user

‘peter’ having the required skills for the action ‘boil egg’.

Beliefs:

skill( peter, boil_egg ).

As is done with recipe actions, a nested approach has also been used for the instruction list. The sub-

lists in the instruction list indicate sub-recipes. Sub-recipes could be separate parts of a recipe, like

making the Bolognese sauce to go with the pasta. They could also be actions with a lot of sub-actions

grouped together in a sub-list.

Updating beliefs 2APL incorporates BeliefUpdates in an agent. These actions update the belief base of an agent when

executed. A belief update can be used to store information received from the user (through

messages) or the environment (when the agent gains access to the iCat’s sensors). They can also be

used to store temporarily data or results of computations. Besides storing new beliefs, the belief

updates can also be used to alter (update) existing beliefs.


page 22 of 68

Each belief update is accompanied by a pre- and post-condition. The pre-condition indicates the

beliefs required for the belief update to be able to execute. The post-condition illustrates the agent’s

beliefs after the belief update has been executed. Below is a simple example illustrating slicing a

tomato. Pre-condition is that the agent beliefs it has a tomato, post-conditions are that the agent no

longer beliefs it has a tomato but instead now beliefs it has sliced tomato.

BeliefUpdates:

{ tomato } Slice(tomato) { not tomato, sliced_tomato }

4.3 Recipe phases To best handle the different steps of an agent performing a recipe together with a user, I have split

the recipe scenario into a number of phases. This section will discuss the contents of these recipe

phases. Figure 1 in appendix 1 displays the recipe phases following each other.

To illustrate the different phases, one recipe scenario will be traversed to illustrate start and end of

the phases. As the user enters the kitchen, he will start talking to the iCat. At one point, cooking

comes into play. Since having small talk with the user is not part of the recipe scenario created, the

recipe scenario starts with the user requesting a recipe. This triggers the agent to enter the first

phase: the recipe selection phase.

4.3.1 Recipe selection phase

The user describes what kind of recipe he would like to perform. The first thing the agent does in this

phase is saving the request. It will use this request alongside its goal ‘find recipe’ to search for a

recipe inside its recipe belief base. When a match is found, the agent continues to check the recipe.

The first check is on whether or not all recipe actions can be broken down into the lowest level of

instructions: atomic actions. This check ensures that the agent will always have a way to explain an

action (in the form of listing the sub-actions) to the user. The next check is on opportunities. This

check is done to ensure the user will have the opportunity to perform the recipe steps one after

another. This ensures the recipe steps are complete to be able to get to the result. The last check

concerns checking the kitchen on whether all the needed utensils and ingredients are present. This

check searches through the recipe for the required ingredients and utensils. These found items will

then be checked against the kitchen content. Any missing items will be listed in the agents belief

base.

When the first checks succeed and the kitchen is checked, the agent will ask the user for

confirmation on the choice of recipe. If there were any missing ingredients or utensils, the agent will

include this information. When the user does not confirm, the agent will search and check another

recipe. When the user confirms the choice made by the agent, the agent will adopt the joint goal to

perform the chosen recipe with the user. It will also store a status for the recipe selected. This status

will be set to ‘confirmed’ which triggers the agent to enter the recipe planning phase.


page 23 of 68

4.3.2 Recipe planning phase

As the name reveals, the execution of the recipe will be planned in this phase. The biggest step the

agent will take here is creating an instruction list. This list will contain actions the agent will ask the

user to perform. This list is created based on the top level actions of a recipe, the sub-actions for the

composite actions and the skills of the user.

Before this list will be been created, the agent will ask the user to gather the required utensils and

ingredients. Of course if any items were found missing during the recipe selection phase whilst

checking the recipe, the agent will ask the user to go shopping first. These steps can be seen as the

user placing the needed ingredients on the worktop, ready for use in the recipe. This will limit the

user’s movement away from the work area when cooking. When the assembly is done, the agent sets

the recipe status to ‘active’ which triggers the agent to enter the instructing phase.

4.3.3 Instructing phase

With the recipe set to ‘active’ and the instruction list created in the planning phase, the agent will

now continue to instruct the user. It will go through the contents of the instruction list one by one

and either 1) offer the user to perform it for him, 2) tell the user that it will perform it since only the

agent has the required skills or 3) requests the user to perform the action. After a message is sent,

the agent will pause the recipe to await the user’s reaction. After an action has been performed, the

agents beliefs will be updated according to the executed action. Afterwards, the action will be

removed from the instruction list.

When the instruction list is empty and the recipe status is active, this means the user and the agent

have worked through all actions to perform the recipe. This is a signal for the agent to inform the

user the recipe is completed. The agent will then remove the empty instruction list and set the recipe

status to ‘finished’. This triggers the agent to enter the final phase: the recipe finalization phase.

4.3.4 Recipe finalization phase

The recipe finalization phase takes care of the last steps in the recipe procedure. This phase can be

triggered by either a completed recipe or a cancelled recipe. When the recipe is ‘finished’, the agent

will of course inform the user of this fact. Afterwards, it will drop the joint goal and clean the recipe

status.

As was noted above, another way to reach the recipe finalization phase is through cancellation of the

recipe by the user. The user can at any moment choose to abort the recipe procedure. After

confirmation of this request, the agent will change the current recipe status to ‘cancelled’, triggering

the recipe finalization phase. Because this cancellation can be at any time, the ‘cleaning up’ has to be

more extensive. The agent will check for any remaining goals and or beliefs which will be dropped or

removed.


page 24 of 68

4.4 Recipe statuses In order to guide the agent through all of the different phases, recipe statuses are used. The main

trigger will be taken care of by goals whereas these statuses will guide on a more specific level. As

the term status reveals, the recipe statuses are used to save the progress the agent has made in

executing the recipe. These are belief the agent has and are therefore stored in its belief base.

Because the status changes multiple times during the execution of a recipe procedure, the agent’s

beliefs on the recipe status will alter during execution of the recipe scenario.

Because the beliefs concern procedural information the agent holds to be currently true, and not

something he strives to achieve, the choice has been made to save these statuses as beliefs. These

beliefs in combination with the agent’s goal trigger plans. When a recipe status is achieved, the

corresponding follow-up plan will be triggered.

An alternate approach could be creating sub-goals which describe the next recipe status to achieve.

This would however produce a large number of goals. Using beliefs for status information also allows

to better see where the agent currently is in the scenario. Having a goal only shows what you strive

to achieve, not what you have done so far. An additional advantage the use beliefs has over the use

of goals concerns implementation. Through the use of belief updates, the belief status information is

easily updated. Furthermore, these beliefs can be ideally combined with the agent’s goal in the 2APL

reasoning rules. When using the sub-goals, the agent would have to check for these goals inside the

plans triggered by the main goal.

As described above, these statuses all will be stored in the agent’s belief base. Appendix 1 figure 2 is

a state diagram illustrating the recipe statuses discussed above and the possible transactions

between different statuses. Below, I will describe the transitions between the different statuses

based on a recipe scenario.

Recipe selection phase At first, the agent will receive a message from the user concerning a recipe he would like to perform.

When this message is received, the agent adopts a goal to find a recipe for the user. The search

request is stored alongside the initial recipe status:

currentRecipe(User,searching,SearchRequest).

Combining the goal findRecipe(User) with this status, the agent searches a recipe. When a recipe

is found, it will create a status for it in order for it to be checked:

currentRecipe(Code,User,checking). This second status will ensure the search request is

saved. Combined with the findRecipe(User) goal, the checking process will start. When the

checking somehow fails (action decompositions might be missing) the recipe status will be removed

and the recipe will be rejected by creating a belief: recipeRejected(Code, Reason). Because the

second status is removed, the searching will continue. Because of the created belief, the agent will

not check the previously checked recipes again. When the checking succeeds, the second status will

be altered from checking to checked, a signal for the dialog agent to ask the user confirmation on

the recipe.


page 25 of 68

When the user denies the agent’s choice, it will again remove the second status and adopt a

rejectedRecipe(Code, Reason) belief before returning to its search for another recipe. When

however the user confirms the agent’s choice, the search request and checked recipe are merged

into one new recipe status: ‘confirmed’. This triggers the recipe planning phase.

Recipe planning phase The recipe planning phase starts with changing the recipe status to

’planning(gather_things)’. Since this phase consists of several steps, the recipe status will

also resemble different statuses. First off is gathering the required items. When this is done, the

status will be set to ‘planning(instruction_list)’. For the creation of the instruction list, a

goal will also be adopted which will be dropped again after successful creation of the instruction list.

After this list has been created, the recipe status is set to ‘planned’ indicating the planning has

finished. This will allow a final belief update to execute, changing the recipe status to ‘active’ and

with that starting the instructing phase.

Instructing phase The agent uses the instruction list to decide which action has to be done first. It will then send a

signal to the dialog agent to communicate this with the user. The dialog agent informs the user of the

next action and moves the recipe state to: ‘awaiting_reply(instruction)’. When the dialog

agent then receives a message in return, it decides upon which action to take. When the user

answers the agent the action has been performed, the agent will return the recipe status to

‘active’. This will trigger the main agent to continue with the next action.

When a sub-list is encountered, the agent will handle this as a separate recipe. It will first pause the

main recipe by changing the status to ‘subRecipe’. Afterwards, it will create an instruction list

containing the sub-list. It will also create a recipe status set to ‘active’ and adopt a joint goal to

execute the sub-recipe. Because of the instruction list, the recipe status and a joint goal, this sub-

recipe will be handled by the agent as any other recipe. After finishing the sub-recipe, the main

recipe status is restored to status ‘active’ continuing with its own instruction list.

Recipe finalization phase When an instruction list is empty, the agent will finish the recipe by moving the status to ‘finished’.

This triggers a plan to finalize the recipe. First of all, this plan will remove the empty instruction list.

This plan will also remove the corresponding recipe status and joint goal. As mentioned above, if the

recipe concerns a sub-recipe, this plan will resume the main recipe.

Another way to end the recipe is when the user aborts the execution. Whenever the user sends an

abort message to the agent, the agent will first confirm the user’s choice and afterwards, move the

recipe status to ‘cancelled’. This will trigger the same finalization plan but with a slightly different

plan. Since we do not know exactly at what point the user will send a abort message, a clean-up plan

is needed to ensure no data of the current recipe is left behind. Of course here also the recipe status

and joint goal are removed.


page 26 of 68

5 Approach to the dialog This chapter on the approach used to communication is split into sections according to the different

recipe phases. In each phase, the different kinds of messages expected to go back and forth will be

shortly discussed. We assume all message sent will reach their target. This removes the need to

check with the receiver if the message is received when an answer is overdue.

The way I approached the communication is by looking at both sides (user and agent) and analyzing

which messages the agent and user could send. From those initial messages, the possible reactions

could be analyzed. I found it useful to make the separation between “Agent -> User” and “User ->

Agent” communication to be able to see who initiates contact.

The basic approach to handle messages is to pause the activity and handle the message. When the

dialog is handled, the procedure can be resumed. Whenever the agent needs more information, it

can pause the procedure, ask the user for the needed information and await his reply. However, the

user can at any time interrupt the agent by asking a question. These messages might not always

come at a convenient time for the agent. It might for example be busy executing an update on its

beliefs.

When the user initiates a (section of a) dialog by, for example asking the agent “why?”, it depends on

the current state of the ‘cooking procedure’ as to what the agent will answer. During the several

recipe phases, the agent will also require different kinds of information from the user. Since the

agent will know what it needs, it will know what it can expect in return from the user.

The conversational context helps the agent with finding what information should be retrieved from

the users questions and or remarks. It also helps in finding why the user wants something. Using the

parsed conversation input from the user and the beliefs the agent currently has, the agent should be

able to derive a suitable reply. This can be in the form of adopting goals, answering questions by

providing information or executing actions.

In the ‘Dialog flowcharts’ section of appendix 1 are several flowcharts illustrating the dialogs

discussed below. These will be referenced within the different sections. This chapter will also make

use of the example dialog in appendix 3. This dialog is an illustration of what kind of dialog ideally

should be possible in the finished system. In the upcoming sections, the example dialog will be

referenced by adding a reference to line numbers as follows: [line 12-15].

5.1 Recipe selection phase The dialog flow related to the recipe selection phase is illustrated in figure 3 in appendix 1. As the

user is eventually going to eat the recipe prepared, we assume he is the one initiating creating it in

the first place. The initial message of the recipe selection phase (and thus the recipe scenario) is the

user requesting the agent to find a recipe. After the agent has received the message, it will start

searching for recipes matching the description [lines 4, 5]. Depending on what the agent finds, there

are different replies possible.


page 27 of 68

When no recipe is found, the agent will first inform the user of this fact. The fact that there are no

results can be for two reasons. The first is that the search term did not match any recipe. If this is the

case, the will request the user to provide a new search and wait for another description. This will end

the ‘current’ dialog. Receiving the new description will start a ‘new’ dialog.

The second case is when all found recipes were rejected (either by the user or the agent). These

recipes can be rejected by the agent itself since they did not pass the checks described below.

Rechecking agent rejected recipes will just result in another rejection since both the recipes and the

checks will not change. What are interesting rejected recipes to look at are those the user rejected.

These have passed the checks but the user did not confirm the agent’s choice. The agent could

gather these recipes and ask the user if he wants to reconsider previous made choices. If so, the

agent will offer the user the choice between earlier suggestions which got rejected by the user. If

not, the agent will request a new search.

Before asking the user to confirm a found recipe, the agent will check it. Several checks will be

performed. A failure in a test is a reason for the agent to reject the recipe and continue its search.

Whenever the checking of a recipe is complete, the agent will let the user know it found a recipe. It

will ask the user for confirmation on execution of the recipe [line 5]. When there are kitchen items

missing (ingredients or utensils), the agent will include this information when informing the user.

When the user decides the missing items to be crucial for successful execution of the recipe, the user

will reject the choice. Of course the user could also reject a recipe because he just doesn’t feel like

making a certain dish [line 6]. This will lead the agent to continue its search after saving the rejection

of the suggested recipe to the belief base [line 7].

When for example a pan is missing, the agent will request confirmation of the user in the following

way: “I found the following recipe: ___. For this, we need a pan which is not available. Do you want

to perform this recipe or should I search for another recipe?” Otherwise, the agent will just ask the

user confirmation by first telling the user what recipe it found: “I found the following recipe ___.”

Followed by a question for confirmation like: “Do you want to perform this recipe?”.

When the user rejects the recipe suggested by the agent, the agent will take note of this in its belief

base and continue searching through the recipes. When the user confirms the checked recipe, the

agent makes stores the recipe code and can start the next phase [line 8]. In the example dialog, the

recipe selection phase takes up line 4 up to and including line 8.

5.2 Recipe planning phase The dialog flow related to the recipe planning phase is illustrated in figure 4 in appendix 1. During the

recipe planning phase, there is some communication with the user. In the example dialog, the recipe

planning phase starts from line 9. In the example, the agent requests the user information on for

how many people the user would like to prepare the recipe. This information can then be used for

calculating the needed ingredients. The option of creating recipes for various amounts of people is

not incorporated into the created agent. This is however mentioned in more detail in future work.


page 28 of 68

Next, if there were any items missing from the kitchen though the user confirmed the recipe, the

agent will have to send the user shopping. After the user informs this has been done, the agent will

request him to gather the remaining needed utensils and afterwards the remaining needed

ingredients [lines 11 – 25]. The agent will wait for the user to notify he is done with gathering the

remaining things.

After all needed items are gathered, the agent will construct the instruction list. During the creation

of the instruction list, the agent will check the skill information. If the user has the required skill(s) for

the action but it is recently learned, the agent will ask the user if he remembers how to perform the

action. If so, the agent removes the ‘recently learned’ parameter and can add the action to the

instruction list knowing this action will not cause any problems. If the user does not remember, the

agent can also discard the recently learned parameter as well as the skill.

Afterwards, as is the same when the skill was not recently learned or not present, the agent checks

the action on availability for teaching the user. If an action is available for teaching, the agent will ask

the user if he wants to learn how to perform that action. This will expand the user’s skills. Depending

on the answer from the user, the agent will either store a learning parameter or decompose the

action into sub-actions to be put into the instruction list.

The creation of the instruction list does not require any dialog other than requesting the user to

teach him an action or if he remembers how an action is performed. If these requests are not

needed, the instruction list creation will not show in the dialog, as is the case in the example dialog in

appendix 3. After the instruction list has been created, the planning phase is completed. In the

example dialog, the planning phase takes up lines 9 up to and including 25.

5.3 Instructing phase The dialog flow related to the recipe planning phase is illustrated in figure 5 in appendix 1. During

instructing the user, there are lots of different messages the agent can send to the user. The different

messages ask different information from the user in return. The messages can be a request or simply

an informational message. Of course this phase is also the most sensitive for incoming user messages

such as occurring problems or questions. This phase will therefore form the largest part of the dialog.

As the agent starts going through the instruction list, it will send a message to the user depending on

the skill information. If for example both the agent and the user have the required skill(s), the agent

will offer the user to perform the action for him. If the user agrees, the agent will perform the action.

Afterwards the agent updates its beliefs and continues to the next action in the instruction list.

If the user rejects the agent’s offer, the agent will send another message now requesting the user to

perform the action. The agent will also send the user this request of execution of the current

instruction when the agent does not have the required skill(s). When the user confirms, the agent

notes the user will perform the action. The agent will then wait until the user informs the agent of

the fact that the action is done. Again, afterwards it will update its beliefs on the executed action

before continuing to the next action.


page 29 of 68

If the user however denies the request to execute the action, the agent checks (again) whether it

itself is skilled enough to perform the action. If so, it can perform the action itself (of course letting

the user know first). If not, the action needs to be done by the user which just refused to perform it.

The agent will now stress this fact by informing (instead of requesting) the user that he needs to

perform the action.

If the user again denies the task assigned to him, the agent will let him know this will abort the recipe

and ask him if he’s sure. If so, the agent will abort the recipe. If the user does not want to abort the

procedure, the agent will resume the dialog where they left off. An optional implementation instead

of aborting the recipe is trying to find an alternative action which results in the same outcome. This

however requires the ability to reason with action pre- and post-conditions which is not currently

available in 2APL. Since replacing an action with an alternative could be quite tricky, this is left open

for future work since there was not enough time to incorporate it into my research.

Another possible addition to the implementation is allowing the user to give a reason for his

rejection of the instruction. When for example the user rejects because the skill information is

incorrect (he does not have the required skills), the agent can fix this rejection by decomposing the

action into sub-actions. The user can currently not give a reason as to why he rejected the agent’s

request. This is also discussed in the chapter on future work.

When only the agent has the required skill(s) for the current action, it will inform the user of the fact

that it will perform the action. After the action has been performed, the agent will update its beliefs

on the performed action. Afterwards, the agent will continue to handle the next instruction.

A slightly different process occurs when the agent is teaching the user how to perform an action. Of

course the ideal way to do this is let the user perform all the actions himself. The agent will therefore

when teaching only request the user to perform the actions and not offer or perform them himself.

As mentioned above, when the user rejects a request, the agent will then inform him once again he

needs to perform the action. In this way mentioning this is the only way to complete the recipe. An

illustration of the teaching process can be found in the following sub-section.

Because of the limited abilities of the iCat, the agent will not be able to perform a lot of actions other

than monitoring (by using the camera) or keeping time. As shows in the example dialog, the agent

will request the user to perform all physical actions. The instruction phase takes up lines 26 up to and

including 73 in the example dialog.

Lines 26, 27, 28 and 29 illustrate the agent instructing the user. The user confirms the request. The

instruction can be seen as two steps; washing the parsley and chopping the parsley. As extra

information, the user also informs the agent of when he starts executing the ‘second’ step. The agent

will just answer with ‘okay’ leaving the user to finish the instruction. In line 29, the user informs the

agent he is done.


page 30 of 68

5.3.1 Teaching the user

To shorten future instructions, the agent is equipped with a teaching ability. Upon creating the

instruction list, actions will be checked against the user’s skills. Consider the user does not have the

skill for a certain action A, but does have the required skills for all its sub-actions. If the agent would

let the user know what order of sub-actions action A is built up of, the user will be able to perform

action A by himself in the future.

Starting the teaching process

The first step of course is asking the user whether or not he would like to learn it. This step takes

place when the agent is constructing the instruction list as the agent recognizes a possible action for

teaching. The agent offers the user to teach him this action. Since it might be confusing to the user

when the agent starts teaching an action while teaching another action, the agent will only offer to

teach the user an action which consists of sub-actions for which the user has the needed skills to

perform them.

If the user answers yes, the agent will add a belief to its belief base so it remembers it has to teach

the action to the user and to be able to update the user’s skills in the end. When the user answers

no, the agent will continue constructing the instruction list as normal. The agent will decompose the

action into sub-actions (the user has skills for these sub-actions because of the requirement set for

actions available for teaching) and finish constructing the rest of the instruction list.

The teaching process

When the agent encounters an action marked for teaching in its belief base, it will inform the user it

will teach him a new action. The agent then takes the user through the sub-actions of the composite

action. When all sub-actions are done, the agent will finish the teaching process by letting the user

know he has performed all necessary sub-actions. The agent can now discard the belief of the user

wanting to learn this action and update the user’s skills. Since this skill is freshly acquired, the agent

will set a parameter in its belief base indicating this composite action is recently learned.

Finishing the teaching process

Next time the agent encounters the recently learned action whilst creating the instruction list and

notices the ‘recently learned’-belief, it will ask the user if he remembers how it is done. If the user

answers yes, the agent can remove the ‘recently learned’-belief indicating the user has successfully

learned how to perform the action. This ensures the system will no longer ask the user on whether or

not he knows how to perform the action. The action can be added to the instruction list and the

agent can continue creating this list.

If the user answers he does not remember, the agent will remove both the skill information and the

‘recently learned’-belief. If available for teaching, the agent will then of course offer to teach the user

the action.


page 31 of 68

5.4 Recipe finalization phase This last phase does not contain a lot of communication. This phase actually only contains the

confirmation by the agent on whether the user is sure of cancelling the recipe. This of course is only

done when the user sent a cancellation message. Otherwise, the agent will reach this phase after the

recipe has been completed. This will trigger the agent to send an informational message to the user

about this fact. When the correct message has been sent, the agent will clean up any remaining goals

and beliefs which are no longer necessary to keep. This cleaning up of the agent’s beliefs of course

will not show up in the example dialog. Lines 74 up to and including 77 in the example dialog

illustrate the recipe finalization phase.

5.5 Overall communication This section will elaborate on messages which might be sent throughout the recipe procedure so in

any of the above described phases. Since their timing is unknown, this separate section is created.

These will be messages the user sends the agent since any agent initiated dialog can be related to

any of the recipe phases.

As we might have a curious user, willing to learn new things, he might ask the agents questions like

“how much salt should I add?” [line 43]. Replying to these kinds of questions requires the agent to

deliberate on the action at hand. For example, adding salt to the water in which pasta will be

prepared is done to improve the flavor. Another example question is illustrated in line 33 where the

user asks the agent “how small” he should make the meatballs.

This knowledge of the action will not be represented in the post-conditions. It will for example not

say: ‘water with salt to boost the flavor’ as a result of the action ‘add salt to water’. To be able to

answer these kinds of questions, the agent needs more information on the recipe actions besides

their direct results. These informative questions therefore open up a whole domain of additional

deliberation needed to keep the dialog ongoing and realistic.

Other messages the user might send the agent is informing of occurred problems. Of course the

instructing phase is where the most user messages concerning problems arise since this is where the

actions will be executed. Actions might for example fail leading the user to question which further

steps to take. Having the status information and the failed action, the agent could search for an

alternative action (or plan) which brings the user to the desired result. If for example there is a

problem with the oven, the user will inform the agent of this problem [line 51]. The agent derives the

oven is used for baking the meatballs. It will then check its recipe base in order to see whether or not

there’s an alternative recipe for ‘bake meatballs’. The alternative approach (using a pan instead of

the oven) is found and selected for instruction.

A suitable alternative might not always be at hand. In the example dialog, the agent is lucky an

alternative approach is available in its recipe base. Of course for a lot of actions per recipe, searching

a suitable alternative needs to be a flexible method. This needs to take into account the needed

utensils for the rest of the recipe, so the rest of the recipe can remain unchanged. Some actions


page 32 of 68

might partially replace another action, but the post-conditions have to match closely in order to

meet the pre-conditions of the following action.

As a final type of message the user might send the agent, there is the cancellation message. The user

could send this at any time during the recipe procedure. The agent will not ask the user for its

reasons but will request him if he is sure on cancelling the procedure. If not, the agent will disregard

the message and continue where they left off. Otherwise, the agent will resolve to the recipe

finalization phase.

Not all of these kinds of messages are as easy to implement as others. For handling some of these,

like questions on why recipe actions are done, the agent needs additional information. During the

discussion of the agent implementation which will follow, the messages I was able to implement will

be discussed. As is to be expected, the other kinds of messages will return in the chapter future work

alongside more detailed information.


page 33 of 68

6 Agent architecture Alongside my research I created a 2APL-agent. This chapter will present the created agent and it’s

architecture. To handle different features of the recipe scenario, several files have been created.

Splitting the agent over multiple files results in a modular design which improves ‘readability’ of the

code. It also allows for quick expansion or adaption of the code. In this section, all modules will be

introduced and their relation will be illustrated in a diagram. The resulting agent will ideally run on

the 2APL module which on its part runs on the internal structure of the iCat.

Below, I will list the different modules of the agent and their purpose. Some modules will only be

used for separating belief base data, others may also have plans or algorithms designed for managing

specific aspects of the recipe scenario. The names of the modules are defined to illustrate the focus

of the module in the recipe scenario.

Mainagent.2apl This file concerns the agent itself. All other modules have been

separated from this file to create the modular design. The main control

cycle the agent follows during execution of a recipe scenario is defined

in here. All other modules are called along this main cycle.

Dialog.2apl This module will take care of all communication towards and from the

user. Depending on the user’s messages, it will take appropriate action.

Recipes.2apl The ‘recipe module’ stores the recipes and the accompanying needed

ingredients and utensils. It also contains the methods for searching and

checking a recipe.

Actiondecompositions.2apl This module shows decomposition of actions into sub-actions. This will

only concern belief base statements.

Skills.2apl In the ‘skills module’, all the user’s and agent’s action related skills are

stored. This module will also only contain belief base statements.

Instructionlist.2apl The ‘instruction list module’ uses the ‘action decompositions module’

together with the ‘skills module’ in order to create a user specific

instruction list for the selected recipe.

Kitchen.2apl The ‘kitchen module’ represents the kitchen including all ingredients

and utensils. It stores the stock and available utensils. It also houses

methods for checking the kitchen on available ingredients and utensils

as well as a method for gathering these before the user and agent start

preparing a recipe.


page 34 of 68

Atomicactions.2apl This module contains the belief updates to be executed when an action

has been performed in the kitchen. When for example the user has

turned on the gas, the agent needs to update its beliefs to resemble

this fact.

Listoperations.2apl This last module stores belief updates on working with lists such as the

instruction list. This module separates this functionality from the rest

so adaption of these belief updates can be made when for example the

Prolog version incorporating 2APL changes.

As the module for running this 2APL agent on the iCat was not available at the time of my research,

these modules were loaded into the 2APL user interface and run on a computer. Below is a diagram

of which module includes what other module. The instruction list module for example needs to know

what skills the user has in order to create the right instructions. Therefore, the instruction list module

includes the skills module. Here one can also clearly see the ‘main agent module’ is on top of it all.

Main agent(mainagent.2apl)

Recipe module(recipes.2apl)

Kitchen module(kitchen.2apl)

Instruction module(instructions.2apl)

Action decompositions(actiondecompositions.2apl)

Atomic action implementations(atomicactions.2apl)

Skills module(skills.2apl)

Dialog module(dialog.2apl)

List operations(listoperations.2apl)

To allow testing the created agent, an user agent has been introduced. This agent houses procedural

rules containing plans to respond to the agent’s messages. The ‘user agent’ can be instructed to start

a recipe scenario by sending it a specific message. The user implemented along the agent however

has an initial plan to request a recipe. When the created agent runs on the iCat, a real user will talk to

the iCat. This speech will then have to be parsed to messages with which the agent can work.


page 35 of 68

7 Agent implementation This section will illustrate the agent implemented alongside the recipe phases. During each phase,

one or more interesting features of the agent will be discussed. These will also be illustrated with

fragments of code. These fragments concern interesting algorithms or belief base reasoning rules to

be found in appendix 2.

7.1 Recipe selection phase The recipe selection phase is the initial phase of a recipe scenario. In this phase, the agent is to

search a recipe for the user, check it on several features and suggest it for confirmation.

7.1.1 Searching

The user (which like mentioned earlier is currently simulated in the form of an agent itself) will send a

message to the agent concerning a search request. This message is, like all other messages, handled

by the dialog module. Below is the 2APL PC-rule regarding this message.

PC-rules:

//agent receives a message from User to perform a recipe together

message( User, inform, _, _, recipe(Text) ) <- true |

{

adopta(findRecipe(User)); //adopt the goal to find a recipe

SearchRecipe(User,Text) //initiate the search

}

The result to receiving the initial message is for the agent to adopt a goal. This goal is to ‘find a

recipe’ for the user. Of course the agent will also store the provided info in its belief base for

reference during the search. This information is stored alongside the recipe status: searching.

The agent currently just matches the users search text to the description stored alongside the

recipes. Since this match has to be exact, one can see this sincerely limits the finding of a suitable

recipe. However, expanding this search method can be done in various ways all depending on

additional information. If for example the user would request a ‘vegetarian’ or ‘healthy’ recipe, this

would require an ontology for ingredients used in the recipes. More on this will follow in future work.

7.1.2 Checking

When a match is found, a recipe status will be added for this recipe stating it’s being checked.

Recipes will be checked on three aspects.

Decomposition check First of all, the decomposition will be checked. Any recipe will be checked on availability to be

decomposed into atomic actions. This can be done quite quick by using the belief base rules

illustrated in appendix 2, figure 1.

If the predicate checkAtomic(recipe) fails, the agent knows the recipe can’t be decomposed into

atomic actions only. An algorithm will find the actions which could not be decomposed and will list


page 36 of 68

them as missing. This algorithm is illustrated in appendix 2, figure 2. Having missing actions will cause

the recipe checking to fail after which a new recipe needs to be searched and checked. We have kept

this check as a precaution to new recipes which get added to the agent without knowing about this

requirement.

Opportunity check The second check performed concerns the pre- and post-conditions of the actions. This check is done

to ensure the user will have the opportunity to perform the recipe steps one after another. It is used

to check pre- and post-conditions of succeeding actions in the recipe.

Consider the agent checking the recipe for ‘spaghetti and meatballs’. Frying the meatballs requires

use of the oven (let’s consider the alternative of using a pan unavailable). If the kitchen does not

contain an oven, this pre-condition is not met. This action can’t be performed because of the missing

oven. Therefore, the recipe can’t be fully executed and will not be selected. Because of the missing

oven, the pre-condition to the action is not met. Therefore, it is impossible to perform the recipe in

that situation. Of course this situation requires information on importance of utensils as well.

Rejecting the recipe because a fork which is not available would be too much pre-caution. The user

could maybe even use a spoon instead. For this an ontology for the utensils would also come in

handy.

Rejecting the recipe straight away is however not the most ideal approach. What would be more

ideal is to figure out how the recipe can be ‘altered’ in order to keep it executable. Imagine the user

just used the blender to create a mix of nuts and afterwards needs it to blend some vegetables into a

sauce. The blender would be dirty after mixing the nuts. Since a clean blender is needed to blend the

sauce, this action can’t be done after mixing the nuts. This can be fixed by requiring a second blender

but even more effectively, by incorporating a ‘clean blender’ action somewhere before the blending

of the sauce. The incorporation of extra actions alters the recipe definition. This of course has to be

done with care. Allowing the agent to alter the recipe is not implemented in the current version

because it has too few ways to reason about what action to insert or replace.

The opportunity check will check pre-conditions for the first action on the kitchen belief base. The

other pre-conditions can of course also follow from the kitchen belief base but might also follow

from the post-conditions of previous actions. For example, slicing the paprika fulfills the pre-

condition of ‘sliced paprika’ for the next action of ‘add sliced paprika to the sauce’. Because

reasoning with pre- and post-conditions of belief updates in 2APL is not available in (yet), this

opportunity check is written to always succeed.

When finishing the recipe check, we know if all the needed utensils and ingredients described

alongside the recipe and actions are present, the recipe actions in theory are possible to be executed

one after another. This check could ideally be done in a virtual environment created by the agent. In

this environment, the agent can simulate executing all actions after each other and checking if the

required beliefs are met. Any errors can be reported back to the agent to have detailed information

on where the recipe is ‘incorrect’ or ‘incomplete’.

The opportunity checking can for the above reason be seen as a check of the correct specification of

a recipe. Consider the action ‘bake meatballs’ which uses the oven. While ‘bake meatballs’ is active,

the condition ‘oven taken’ is in effect. After the action ‘bake meatballs’ is done, the post-condition


page 37 of 68

‘oven free’ or ‘oven dirty’ becomes active. Now take another action which just like ‘bake meatballs’

needs to use a (clean) oven.

If the second action is planned during execution of ‘bake meatballs’ (‘oven taken’ holds), we

encounter a problem. At first glance one would say the user needs two ovens. This pre-condition

can’t be stored with the actions because the two actions don’t need to be planned in this particular

order. They might not even be in the same recipe otherwise. This means that if the user were to use

two ovens, this would have to be stored with the pre-conditions of the recipe as a whole.

Alternatively, we could conclude the second action can only be executed after the action ‘bake

meatballs’ has been completed. This would mean the recipe should accompany a ‘wait for bake

meatballs to complete’ action. In the situation where ‘bake meatballs’ would lead to ‘oven dirty’, the

recipe should even accompany an extra action like ‘clean oven’ before continuing to the second

action.

Ingredients and utensils Third and last, a final check will be performed. This last check is actually more a procedure than a

check since it has no positive or negative result. In this procedure, the agent will search the needed

utensils and ingredients for the selected recipe. Currently, these are retrieved from the recipe itself.

Alongside the recipe code and description, a list of needed utensils and ingredients is stored. These

lists could also be created per action. Ideally however, the needed ingredients and utensils for an

atomic action would be retrieved from its corresponding belief update. As mentioned above, this

feature is missing from 2APL.

Retrieving ingredients from belief updates however does not replace the need for a list of ingredients

(and possibly utensils) at recipe level. Consider for example slicing a tomato. The action itself is

directed towards 1 tomato. The recipe however might require 3 tomatoes to be sliced. Therefore,

the recipe could state how many tomatoes are needed for the whole recipe.

The needed ingredients and needed utensils found are then checked against the contents of the

kitchen. If there is anything missing, this will be listed in a belief statement. The agent will use this

information later on.

7.1.3 Confirming

Now that we have successfully found and checked a recipe, we can ask the user confirmation of our

choice. If there were any utensils or ingredients missing, the agent will include this information. If the

user confirms, we save our choice in the belief base. If however the user rejects the agent’s choice,

the agent will continue searching for another recipe. Of course a note is also made of this rejection.

Because of the recipe checking process, we are sure at this point the recipe selected by the agent is

executable in theory. This means the user and agent together have the required skills to perform the

needed actions. We finalize the selection procedure by saving the confirmed recipe. This is done by

setting the recipe status to ‘confirmed’. The goal to find a recipe is dropped. Next, a joint goal is

adopted which will continue the recipe procedure. Both user and agent are now joint in the task of

preparing the selected recipe.

Currently, the agent will remove all rejected recipes before requesting a new search. This will

illustrate the different recipes being checked each time a search is started. This is done because it’s


page 38 of 68

also interesting on what grounds the agent rejects the recipe. Since the recipe base will not change,

the recipes will be rejected on the same ground each time. Also, since there are only a few recipes

implemented, keeping the ‘rejected’ belief base data would lead to almost no search request

generating results. The recipes which get rejected by either the user or the agent will be stored in the

belief base alongside the reason for their rejection.

7.2 Recipe planning phase In the second phase of the recipe procedure, the agent will plan performing the recipe steps. Most

important of this phase is creating the instruction list.

7.2.1 Gathering ingredients and utensils

The first step however is to gather the needed ingredients and utensils. The dialog module will

handle the dialog and request the user to gather all needed items. If there were any items missing

when the agent checked the kitchen, the agent will first instruct the user to go shopping for these

items.

Gathering the ingredients will require some parsing of beliefs. Since the ingredients noted alongside

a recipe are grouped inside a list, the list elements need to be parsed to ingredients and their

corresponding amounts. In 2APL, beliefs on ingredients are stored as prolog facts. What possibilities

there are for parsing list items depends on the prolog version incorporated with 2APL. 2APL is

currently being developed in Java (the version I used along my research) as well as in C++ (for

integration with for example the iCat) and these two use a different prolog version. Therefore the

parsing is left open.

The basic idea on this step of the recipe procedure is to place all needed ingredients and utensils on

the kitchen worktop. Usually, the utensils needed for cooking already are in the kitchen so putting

them all on the worktop might only limit the space to work. Some utensils like a blender might

however be stored in a kitchen cabinet, which illustrates the use of this action. Ingredients usually

have to be gathered from different places like the refrigerator, the freezer or the cellar.

As was mentioned above, the ingredients and utensils are gathered before the planning continues.

This step can however also be postponed to the moment the ingredient or utensil is actually needed

when performing an action. This would however ‘pollute’ the cooking instructions with gathering

instructions. To keep the cooking separate from the preparations, the gathering of needed items has

been moved to the planning phase. Placing the items on the kitchen worktop before continuing will

also limit the user to moving away from the worktop while cooking.

7.2.2 Creating the instruction list

To guide the user through the recipe, an instruction list is created. This list contains the actions to

take in order to complete the recipe. The creation of the instruction list is done by an algorithm

inside the instruction module illustrated in appendix 2, figure 3. As with the recipe phases, the

instruction list creation is also guided by a goal and the recipe status.

First of all the top level of the recipe will be retrieved. This first level of actions is stored alongside

action decompositions because it essentially is the first decomposition of the recipe. Below are the


page 39 of 68

‘top level actions’ of an example recipe. I will use this example to illustrate the creation of the

instruction list.

Beliefs:

recipe( example_recipe, [action1, action2, action3, action4, action5] ).

As you can see, this recipe consists of 5 top level actions. These can be atomic or composite actions.

Composite actions consist of sub-actions which in their case can also be composite or atomic. The

decompositions of the composite actions belonging to this recipe are displayed below. The

composite actions have a list of sub-actions stored with them. Actions which have an empty list are

atomic actions.

Beliefs:

/* first level of actions */

action( action1, [] ).


action( action3, [subaction31, subaction32] ).


action( action5, [subaction51, subaction52, subaction53] ).

/* second level of actions */

action( subaction31, [subaction311, subaction312, subaction313] ).

action( subaction32, [] ).


action( subaction52, [subaction521, subaction522] ).


/* third level of actions */






These actions and their decompositions can be ideally visualized by representing the recipe in tree

form. Each leaf of a tree represents an atomic action. The children of a node represent the sub-

actions of which the represented action consists. Visualizing the tree in this way allows a much

clearer view of which sub-action belongs to which action. A recipe tree for the example recipe is

displayed on the next page. Action 3 for example is built up out of sub-action 3.1 and sub-action 3.2.

These children can themselves also be built up out of sub-actions, as is the case for sub-action 3.1.


page 40 of 68

Example recipe

Action 1 Action 2 Action 3 Action 4 Action 5

Subaction 5.3Subaction 5.2Subaction 5.1

Subaction

5.2.1

Subaction 3.2Subaction 3.1

Subaction

3.1.3

Subaction

3.1.2

Subaction

3.1.1

Subaction

5.2.2

When creating the instruction list, the recipe actions will be checked on the skills of both user and

agent. The stored ‘skill beliefs’ for this example are displayed below. As you can see, the statements

match an action to a user (or even the agent itself) when he is skilled of performing it. Since not

everyone will have the same skills on performing a recipe, the instruction list for one recipe might

differ per user. Depending on the available skill information, some composite actions might need to

be decomposed to sub-actions inside the instruction list.

Beliefs:

skill( user1, subaction31 ).

skill( user1, subaction32 ).

skill( user1, action5 ).

Now we know how the actions of the recipe are built up and what the user’s skills are, we can begin

constructing the instruction list. This is done by the instruction module which will combine the top

level actions and the user’s skills with the known decompositions. The actions will be decomposed

into the instruction level the user is skilled of performing.

The agent will start with the top level recipe actions, so the first action the instruction agent

encounters is action1. The agent checks whether this is an atomic action by looking up the action in

the belief base. From the recipe actions agent belief base follows that action1 is indeed an atomic

action. Since the user (user1) is skilled of performing atomic actions, this action can be added to the

instruction list. The agent then continues to action2 for which the same action is taken.

Next in the top level actions is action3. This time, the action concerns a composite action. The

instruction agent now checks the skills for user1. There is no reference for action3 in this belief base.

The agent will then check whether action 3 is available for teaching. To keep the instruction list

algorithm clear, we ignore this check and consider the check failed.

The agent now continues creating an empty instruction list for action3. This list will contain the

instructions for action3. It continues with looking up the list of actions action3 is made up of. It then

checks each of these actions. The first action it encounters is subaction31. It checks if the action is


page 41 of 68

atomic, it is not the case. Next it checks if the user has skill information concerning subaction31.

From the skills agent’s belief base follows that user1 is skilled of performing subaction31. The agent

therefore adds subaction31 to the action3 instruction list and continues to subaction32. It again

checks whether it is atomic and if not, if the user is skilled of performing the composite action. It

finds the skill information and therefore can also add subaction32 to the action3 instruction list.

Now the list of sub-actions for action3 is empty. The system now knows the instruction list for

action3 has been completed. The system now adds a sub-list of instructions for action3 into the

instruction list for the recipe. The instruction list so far is displayed below.

Beliefs:

instructionList( test_recipe, [action1, action2, [action3, subaction31,

subaction32]] ).

The next step for the instruction agent is to continue with the next top level action, action4. It checks

and notices this is an atomic action and therefore can immediately add it to the instruction list. The

next action it checks is action5. Although this is not an atomic action, user1 does have skill

information on this action. This allows the agent to add action5 to the instruction list. Since there are

no more top level actions, the instruction list is now ready. The final instruction list is displayed

below.

Beliefs:

instructionList( test_recipe, [action1, action2, [action3, subaction31,

subaction32], action4, action5] ).

All the actions in the instruction list, user1 is skilled of performing. The sub-list indicates a recipe

action the user is not skilled of performing yet along with the sub-actions it is built up of. A sub-list

without nested sub-lists indicates an action which is available to teach the user. In the situation

above, action3 for example is available for teaching.

Now that the instruction list is created, the agent will update the recipe status to

currentRecipe(Code, User, planned) to signal the planning is done.

7.3 Instructing phase When the recipe is planned, the main agent picks up the process and switches the recipe status to

currentRecipe(Code, User, active). This in the start of the instructing phase. The instructing

phase concerns exactly what it describes, it is meant to instruct the user to perform the actions in the

instruction list. In performing these actions / steps, the recipe will be performed. The way the actions

are executed is pretty simple. Because of the way the instruction list is created, it will only contain

actions for which the user or agent itself has the required skills. We therefore can simply go through

these actions one by one.


page 42 of 68

As the agent works through the instruction list, it will go through a number of checks. First of all, the

agent will check for a sub-recipe. A sub-recipe is indicated by a nested list inside the instruction list.

When this is encountered, the agent will start a specific plan to continue with this sub-recipe. More

on this will follow in the next section. If the next instruction doesn’t concern a sub-recipe, it can still

be an atomic or a composite action.

With every action in the instruction list, the agent will first check whether or not the agent is in a

teaching process. The choice has been made to only let the user perform actions while being taught

something. Therefore, while teaching, the agent will not offer to perform actions for the user.

Next, the agent will check available skills of the agent and the user. This is used to determine what

the agent should communicate to the user. If for example both have the required skills, the agent can

offer the user to perform the action for him. If only the agent has the required skill, it can just inform

the user of the fact that it will perform the current action. If only the user possesses the required

skill, the agent will request the user to perform the action. Figure 4 in appendix 2 illustrates these

possible scenarios and their related follow-up steps.

After an action has been performed, the agent will update its beliefs on the world which has changed

because the action was executed. The executed instruction will, after the beliefs have been updated,

be removed from the instruction list and the agent will continue with the handling of the next

instruction.

7.3.1 Sub-recipes

Instead of a single (composite or atomic) action, the agent can also encounter a sub-list inside the

instruction list. This list indicates a sub-recipe. When the agent encounters a sub-recipe, this means

the user and agent do not have the required skills to perform the action the sub-list was created for.

Therefore, the action was decomposed into sub-actions when the instruction list was created.

The first thing the agent will do now is put the recipe in hold and handle the sub-recipe as any other

recipe by creating a corresponding joint goal, recipe status and instruction list. As illustrated in the

example dialog in appendix 3, the agent can also briefly introduce the sub-recipe to the user as is

done in line 36. Because the ‘main’ recipe is in hold, it will first finish the sub-recipe before

continuing with the ‘main’ recipe. When the actions for the sub-recipe have all been performed, the

empty instruction list will be removed. The joint goal for the sub-recipe will be dropped and the

agent will continue with the main recipe.

An example of a sub-recipe is the recipe for making spaghetti as part of the ‘spaghetti with meatballs’

recipe. The 2APL code illustrating the agent’s actions when a sub-recipe is encountered is illustrated

in appendix 2 figure 5.

7.3.2 Pause and continue

The dialog agent instructs or informs the user on the current action to be performed. While waiting

for the user’s reply, the recipe scenario is paused. Of course when the agent informs the user it will

perform the action, this pause is not needed since the agent can just continue with executing the

action. When requesting the user to perform an action, the agent needs to await the user’s reply.

Based on the user’s reply, follow-up steps can be taken. When the user for example accepts the task,

the agent need only wait for the user to inform the action is done. Since the current agent does not


page 43 of 68

incorporate working with the iCat’s camera, we assume the user will inform the agent when he is

done with executing an action.

When, for example, the user is the only one capable of performing the action, he really needs to

perform the action. When however the agent also possesses the needed skill, it can perform the

action when the user rejects the task. The dialog related to this is illustrated in figure 5 appendix 1.

7.4 Recipe finalization phase When all actions have been done, the recipe has been prepared. This will update the recipe status to

‘finished’ and start the finalization phase. This phase will also be triggered at the end of a sub-recipe.

The difference between finalizing a sub-recipe and the ‘main’ recipe is that when a sub-recipe is

finalized, the ‘main’ recipe will continue. These steps can be found in appendix 2, figure 7. Of course

when the ‘main’ recipe is finalized, the agent will first inform the user of the fact that the recipe is

completed. Afterwards (in both sub-recipe finalization as ‘main’ recipe finalization) the agent will

drop the joint goal and remove the recipe status.

The agent will also finalize the recipe when its status is set to ‘cancelled’. Because this cancellation

could happen at any time, the cleaning of the belief base and goal base needs to be done more

carefully. The agent will have to check for any remaining beliefs or goals and remove them. Of course

the agent will first inform the user the recipe is cancelled before cleaning up any remaining goals and

beliefs.

When talking about cleaning up, the dirty pans and used appliances come to mind. Although this step

is related to cooking since this is how the utensils got dirty, this step will usually follow after the user

enjoys his creation. For example, doing the dishes is mostly done after having enjoyed the recipe.

The cleaning is also not a process where the iCat can assist since it movement capabilities are limited

to facial expressions and moving its head. Since this step could also consist of simple loading all dirty

utensils into a dishwasher, this procedure is not included into the created agent.


page 44 of 68

8 Results This chapter will summarize the used approach and the results achieved by it. One of the visible

results next to this document is the 2APL agent created. This agent structures handling procedural

steps alongside conversational steps. The agent uses BDI principles in order to create dialog as

realistic as possible with the user. During creation of the agent, choices were made concerning

creating realistic dialog. Other choices concerned storage of beliefs and working with these beliefs in

order to come to these kinds of dialogs. The implementation results, illustrated with code and the

final dialog in message form, will be discussed in the following sections.

As already mentioned in the agent architecture, the agent has been split into modules. The resulting

modular design improves readability of the code and allows for quick expansion or adaption of the

code. Each module is designed for its own responsibilities during a recipe scenario. The created 2APL

agent contains a number of procedures to handle a recipe scenario from searching a recipe to

informing the user the recipe is ready.

As for the scenario itself, this also has been divided into sections or rather recipe phases. During

these phases, communication with the user will take place. As an approach to switching between

processing data and communication with the user, an dialog module has been introduced. This

module takes care of sending messages to the user and handling the received messages. The dialog

module uses the active goals, stored beliefs and current recipe status in order to derive a suitable

reply to the user’s questions or remarks.

Other modules hold responsibilities for creating the instruction list, managing searching, checking

and storage of recipes or storage of skill information. The chosen way of implementation leaves

room for expanding or adapting the code as was mentioned before. Because the module for running

2APL on the iCat is not available yet, the agents have been created on a PC. Communication with the

user now happens through messages sent back and forward through the 2APL user interface. The

resulting dialog thus is text based.

8.1 Implementation result As already mentioned before, copying a recipe from a cookbook is not enough to allow the agent to

work with the provided information. Therefore, the recipe for “spaghetti and meatballs” as is

translated to the below described beliefs to be stored in the agent’s belief base. The example dialog

is implemented in the agent and the resulting ‘dialog’ (messages sent back and forth) is illustrated at

the end of this section.

In the ‘Implementation fragments’ section of appendix 2 several figures are included which illustrate

2APL code of the implementation. I have chosen to keep these segments of code separate from the

rest to maintain readability. Figure 8 concerns the two belief base statements representing the

recipe database. Since the matching is done from top to bottom, when the user requests a spaghetti

recipe, the agent will first find the recipe for ‘spaghetti with mozzarella meatballs’. The lists

representing the needed utensils and ingredients for this recipe have been left empty for this recipe.


page 45 of 68

This is done because the user has been implemented to reject this recipe so no further information

except the availability of the recipe is needed.

Following in figure 9 are the decompositions of the recipes and their sub-actions. As you can see, the

‘spaghetti with mozzarella meatballs’ recipe does not contain any sub actions. This is done for the

same reason as leaving out the needed ingredients and utensils. The ‘spaghetti and meatballs’ recipe

however does contain sub-actions. These are all composite actions decomposed into sub-actions as

described below. All atomic actions have been left out since there are a lot of them. If an action is not

defined into sub-actions in figure 9, it concerns an atomic action.

To allow the agent to offer the user to perform some actions for him, I implemented some skill

information for the agent. These skills in the agent’s belief base are illustrated in figure 10. The user

is not provided with any skills. I have chosen not to do this in order to see a full decomposition of a

recipe.

A last part of the belief base illustrated in figure 11 concerns the kitchen. The available utensils and

ingredients are listed as belief statements. These concern of course most of the needed ingredients,

since we would also like to see the taken action when ingredients are missing.

8.2 Resulting message based conversation An example of the resulting dialog between the created agent and the simulated user is displayed in

the following section. The messages going back and forth illustrate the dialog flow. For example, the

agent will offer the user to perform an action for him when it has the skills [lines 26 and 47].

Depending on the user’s reply, an appropriate reply is taken. When the user rejects the offer, the

agent will request the user to perform the action [lines 27-29]. If the user accepts the offer however,

the agent will proceed and execute the action itself [line 48 and 49]. Another nice example of dialog

flow are lines 58 up to and including 62. In line 58, the agent requests the user to perform an action.

The user denies this assignment. Because the agent does not have skills for this action, it will inform

the user he needs to perform this action [line 60]. The user accepts the task in line 61 and informs

the agent it is done in line 62.

If both agent and user have the required skills, the agent will first offer the user to perform this

action for him. When the user rejects this offer, the agent will ask the user to perform the action

himself. It is counter intuitive if the user then denies to perform the action himself after first having

denied the agent to perform the action for him. Why would the user deny the offer of the agent if he

does not want to perform the action himself? Because the agent will first offer the user to perform

the action for him, the agent will never just start executing an action for the user. Requesting

confirmation like this ensures the agent knows the user agrees with it executing the action.


page 46 of 68

8.3 Resulting ‘dialog’ As already mentioned before, the ‘dialog’ displayed below concerns messages passed back and forth

between user and agent. It concerns 1 recipe scenario in which the user wants to make spaghetti.

Sender Performative Content

1 user request recipe(spaghetti)

2 agent inform suggestRecipe(spaghetti_mozarella_meatballs)

3 user inform answer(no)

4 agent inform suggestRecipe(spaghetti_meatballs)

5 user inform answer(ok)

6 agent request shopping([onion, garlic])

7 user inform answer(done)

8 agent request gather(utensils, [water_boiler, pan, pan, oven_dish, bowl])


10 agent request gather(ingredients, [water, spaghetti, meat, egg, tomato, salt, pepper, oil, onion, garlic])


12 agent request teach(make_meatballs)


14 agent inform willTeach(make_meatballs)

15 agent request pleaseDo(wash_and_chop_parsley)


17 agent request pleaseDo(combine_meat_egg_parsley)


19 agent request pleaseDo(season_with_salt_and_pepper)


21 agent request pleaseDo(roll_mixture_into_balls)


23 agent inform teachingDone(make_meatballs)

24 agent request pleaseDo(fill_water_boiler)


26 agent request shallDo(turn_water_boiler_on)


28 agent request pleaseDo(turn_water_boiler_on)


30 agent request pleaseDo(wait_until_water_boils)


32 agent request pleaseDo(put_pan_on_gas)


34 agent request pleaseDo(put_water_in_pan)


36 agent inform pleaseDo(turn_gas_on)




page 47 of 68



41 agent request pleaseDo(add_salt_to_water)




45 agent request pleaseDo(add_spaghetti)


47 agent request shallDo(cook_until_al_dente)

48 user inform answer(yes)

49 agent inform willDo(cook_until_al_dente)

50 agent request pleaseDo(drain_spaghetti)


52 agent request pleaseDo(heat_pan_with_oil)


54 agent request pleaseDo(fry_meatballs)


56 agent request pleaseDo(put_meatballs_aside)


58 agent request pleaseDo(clean_pan)


60 agent inform needToDo(clean_pan)



63 agent request pleaseDo(chop_onion)


65 agent request pleaseDo(optional_chop_garlic)


67 agent request pleaseDo(heat_pan_with_oil)


69 agent request pleaseDo(saute_onion)


71 agent request pleaseDo(optional_saute_garlic)


73 agent request pleaseDo(slice_tomatoes)


75 agent request pleaseDo(add_sliced_tomatoes)


77 agent request pleaseDo(add_meatballs)


79 agent request pleaseDo(let_simmer_5_minutes)


81 agent inform say("recipe completed!")


page 48 of 68

9 Conclusion My research set out with the question whether or not it is possible to create realistic agent-user

dialogs. A short and simple answer would be yes. Several techniques and approaches have been

researched to answer this question. Several of them have been selected for implementation of an

agent to illustrate they work. Storage of recipes and accompanying actions in list-form for example

has proven to be flexible in storage as well as in implementing alternative approaches to the same

action.

BDI-features like goals and beliefs can be used alongside the procedural steps of the recipe scenario.

These then can be used to provide the additional background information needed to deliberate on a

realistic dialog. Also, in using a combination of a goal and a belief statement concerning an accurate

status, flexible decisions can be taken on what intention to adopt next.

Storage of facts in the agent’s belief base requires efficiency as well as availability. Storing actions

alongside a list of their associated sub-actions allows reuse and alternative implementations to the

same action. Storing skills in simple predicates makes them quickly available for use in creating the

instruction list.

What I learned while working out my research, is that the requirement ‘realistic’ requires the agent

to have generous amounts of information available of different aspects of the recipe just to be able

to instruct the user. Storage of this information requires equal as much attention since this data

needs to be easily available. The demand for information to use in deliberation rises when having to

answer the user’s questions. The amount of available background information of course becomes

even more important when the user takes over control and the agent will perform a purely

supportive role.

In conclusion, a BDI-agent (for example written in 2APL) is suitable for the process of communicating

with a user. 2APL agents are available for handling both procedural and communicative actions next

to each other. Furthermore, these concepts can be used to make the dialog between agent and user

more realistic.

Since the work has been somewhat limited to the recipe scenario, of course using 2APL agents on a

larger scale remains open for research. Deliberating on dialog in different domains beside the recipe

scenario will allow the iCat to be used outside the kitchen also. Maybe someday the iCat will be

available as companion robot with intelligent agents representing its brain.


page 49 of 68

10 Future work Some features of the created agent are currently either not implemented or implemented in a very

simple way. These features might not be implemented to their full extent because of they would

require an individual research project on itself or they might not be possible to be implemented yet.

In this final chapter, I intend to illustrate on what areas this research can be expanded which are

interesting enough for further research.

10.1 Extended use of BDI elements Using the BDI elements such as goals and belief updates in a broader way can improve the way the

agent works. Below there are three examples of where an advantage can be reached over the

current situation. Although some of these examples can be seen as ‘upgrading 2APL’, the results of

these upgrades can of course be more widely used than just in this research.

10.1.1 Belief updates

After an action has been performed in the kitchen, the agent needs to update its beliefs. When for

example parley has been chopped by the user, the agent’s beliefs should no longer contain that

parsley but now should contain ‘chopped parsley’ for example. For atomic actions, this is done by

executing a belief update specified inside the atomic actions module. These belief updates contain

pre- and post-conditions. For the chopping parsley example, the pre-conditions could be: parsley, a

knife and a cutting board. These pre-conditions illustrate what ingredients and utensils are needed

for this recipe action. Combining the pre-conditions of all recipe actions, all needed ingredients and

utensils can be found.

The post-conditions of the ‘chop parsley’ action also have its use. These can be used in reasoning on

opportunity of executing actions one after another. Post-conditions of earlier actions might fulfill

pre-conditions of follow-up actions. Chopping parsley results in the post-condition ‘chopped parsley’

which in its turn becomes a pre-condition for the action ‘add chopped parsley to meatball mixture’.

Having the ability to call belief updates like internal actions will eliminate the matching of

instructions in the instruction list to a belief update. Being able to reason with pre- and post-

conditions of belief updates will also allow checking if the beliefs can be updated before executing

the update. All this will simplify the procedure inside the agent.

Another advantage of reasoning with pre- and post-conditions is allowing flexible runtime re-

planning of a recipe. When a recipe action fails, a replacing action (or plan) can be looked up

according to the needed post-conditions of the failed action. Baking meatballs for example can be

made in an oven or in a pan on the gas. This re-planning example is illustrated by lines 50 up to and

including 52 of the example dialog in appendix 3.


page 50 of 68

10.1.2 Goals

Working with beliefs in 2APL is made easy by using belief updates. Similarly, working with goals could

be made more easy by introducing goal updates. Currently, the agent uses a joint goal and a recipe

status belief. If goals could as easily be updated as the recipe status, this status information could be

included in the goal. Because goals currently can’t be updated as easy as beliefs, integrating this

status into the goal would require dropping the goal and adopting a new goal every time the status

changes.

Also, currently 2APL agents generate plans for any active goal. If a priority could be assigned to goals,

this would eliminate the need for putting goals in hold. A goal added to the top of the goal base

would then for example take over priority. When this goal is dropped (achieved), the remaining

goal(s) will continue.

10.1.3 Beliefs

An interesting procedural feature would be adding the availability to cook for a variable amount of

people. This requires storage of amounts of ingredients needed in the belief base in ‘needed quantity

per person’. Calculating the needed amounts of ingredients by multiplying these numbers could

prove an difficult task. The needed amount of water to boil 1 egg is the same as for 2 eggs. Some

parameters might depend on the number of persons, where others might not. Requesting the user

on for how many people he intends to cook could then be the initial step of the planning phase, to

determine the amount of needed ingredients. This is illustrated in lines 9 and 10 of the example

dialog.

Another way to improve the use of the beliefs is enabling parsing of ingredients stored in the list

accompanying a recipe. The way parsing can be implemented depends on the Prolog version

accompanying 2APL. Since this is different for the Java and the C++ version of Prolog, first a definitive

choice of Prolog has to be made as well as a selection between the Java or C++ version of 2APL.

10.2 Extended dialogs To allow a more realistic dialog, the dialog handling needs to be expanded to be able to react to

more situations. When you for example allow the user to give reason for his rejection, this reason

can be helpful in deliberation. It can be used in finding an alternative recipe.

Take for example the user’s remark in line 6 of the example dialog in appendix 3: “I don’t know, I’m

not a big fan of mozzarella, but I like spaghetti.” This rejection houses much more information than

just a ‘no’ to the suggestion the agent made. This answer to the agent’s suggestion can be used to

specify the search for another recipe. The user informed he is ‘not a big fan of mozzarella’ which

removes all recipes including mozzarella from the search. Furthermore, the remark ‘I like spaghetti’

allows the agent to focus on spaghetti recipes.

As was mentioned earlier, having a reason for rejection can also be used when the user needs to let

the agent know the skill information it has is incorrect. This can then of course be easily fixed by

decomposing the current instruction into sub-actions.


page 51 of 68

Of course the instructions from the agent towards the user also allow for extensions. Take lines 30

and 42 for example. Here, the agent grouped several small actions together in one instruction. The

instruction in line 30 for example could be split into two main parts: combining ingredients into a

bowl (could even be split into sub-parts) and afterwards seasoning them. Splitting this instruction in

parts could annoy the user if he spends more time listening to instruction than actually preparing the

recipe.

The agent can also use additional information the user provides when having problems in executing

actions. When for example the user informs there is a problem with the oven when asked to turn it

on, the agent can use this information to find an alternative. He will try and find a way to get the

desired results without having to use the oven. This is illustrated in the example dialogs in lines 51 an

d 52.

Another possible expansion to the dialog handling is allowing the possibility of interruption of the

agent by the user. In lines 66, 67 and 68 of the example dialog, the user interrupts the agent. He asks

a question while the agent in instructing him on the next action to perform. As illustrated in line 68,

the system will stop its current instruction and first handle the user’s remark. After this is done, he

will continue the instruction.

As mentioned before, the way the agent answers questions is important for a more realistic dialog.

When you enable the user to ask questions on whatever subject, the agent needs to be able to

deliberate on an answer. Consider the example in lines 33 and 43. In these lines, the user asks the

agent a question on the execution of an action. In the first example, the user asks the agent “how

small” he should make the meatballs. In the second example, he asks why he should add the salt.

These questions ask of the agent to further specify the instruction it just gave the user. To be able to

do this, it needs more information on the action at hand. For example, if the agent knows adding salt

to the water in which pasta is boiled will boost the flavor of the pasta, it can explain this action to the

user (as is done in line 44).

10.3 User control The agent created takes initiative in communicating and instructs the user with every step. A nice

additional feature would be a variable amount of user control defining who takes the lead. When the

user takes initiative (control), the agent doesn’t have to and can just support him when he gets lost

or has questions. When the user awaits the agent’s instruction (low user control), the agent will have

to react to this and guide them both through the process.

Some users might for example know how to perform certain steps and might just continue without

needing instructions on what to do next. In line 67 of the example dialog for instance, the user takes

over the initiative by asking the agent if the tomatoes are next. He even continues in line 69 with

more certainty. Here he notifies the agent he knows the meatballs need to be added to the sauce

next. When the user informs he is done with that, the agent is able to regain the initiative by

providing the user with the next instruction [line 72].


page 52 of 68

This switching of user control requires the agent to carefully monitor the user’s actions but also be

more flexible in instructing. The agent should in high user control situations ideally only confirm the

user’s steps and support him on any arising questions.

10.4 Ontology for ingredients and utensils When having retrieved all ingredients and utensils for a certain recipe, these might be used alongside

an ontology to extend recipe searching. The ontology could be used to determine a ‘healthy’ or

‘vegetarian’ recipe for example.

In this ontology, ingredients could for example be grouped into categories like vegetable, fruit, pasta,

dairy products, etcetera. Utensils in their turn could be categorized into measuring utensils, mixing

bowls, pans, pots, appliances, etcetera. This ontology could then also be used to simplify recipe

actions. These can then be represented in a more general way with parameters for the agent to fill

in. ‘Boil cauliflower’ could for example be reduced to ‘Boil vegetable’ where the variable could be

anything from cauliflower to broccoli.

The ontology for ingredients also helps when ingredients are missing. The agent might suggest the

user to make macaroni using a recipe for pasta because the spaghetti is all out. It also allows for

more accurate recipe searching. A recipe could be rejected because of use of dairy products or meat.

The ontology comes in handy for utensils when deciding if a recipe is executable. If for example an

appliance like a microwave is missing, the agent can’t just ask the user to buy one.

This ontology might also be used to divide ingredients into categories of necessity. Some ingredients

(like the garlic in the example dialog [line 52]) might be optional whereas others are a must have

(such as the spaghetti when making spaghetti and meatballs).

10.5 Timing and planning The agent might use timing to check back on the user when his answer is overdue. It could also be

the user started his assignment without confirming to the user he accepts the assignment. Of course

when not having a camera available to check what the user is doing, this could certainly prove a

helpful feature. This ‘checking back’ also becomes very useful in situations with high user control

where the agent only needs to support him in answering questions.

Timing can of course also ideally be used alongside recipe actions which require the user to keep

time such as ‘boil spaghetti´. This is illustrated in the example recipe where the agent keeps track of

the time while the user can continue creating the spaghetti sauce.

When the user is for example asked to gather required utensils and ingredients, the agent could

check once in a while to see if the user is done. Or, when there´s no camera available, to ask if

everything is going ok so far.

Another feature which requires timing is planning. As is illustrated in lines 15 and 16, the user might

ask the agent when to perform an action. It might come in handy to already have chopped the


page 53 of 68

parsley for example. To allow more flexible planning of recipe actions, a deadline could be

incorporated.

The planning is also useful when performing parallel actions. An example of parallel execution of

actions is also incorporated in the example dialog. As the user lets the spaghetti boil until it’s ‘al

dente’, he continues with making the spaghetti sauce.

Planning becomes extremely useful when allowing more users to participate in the recipe scenario.

More users means more actions available to be performed in parallel. The agent will have to divide

the actions over the users and itself. To support this parallel execution of actions, the agent needs to

keep in mind which actions are able to be performed, which need to be performed and which can’t

be performed yet. This could be derived from pre- and post-conditions of the actions but constraints

could also be added to actions. Draining the spaghetti for example could be restrained by the

requirement of the spaghetti being ‘al dente’.

Another advantage to be gained with planning is inserting possible ‘pause moments’ in the recipe.

Some features of creating a recipe require actions to be performed shortly after one other whereas

other sections don’t ask this accurate timing. Waiting too long with performing actions after each

other might have a negative effect on the recipe. For example, when waiting too long to drain the

spaghetti (by for example first creating the sauce) does not add to the taste of the spaghetti.

Between the spaghetti being ‘al dente’ and ‘draining the spaghetti’ therefore is not a good moment

to pause or do other things first. Often there will be a possible pause moment between sub-recipes.

When one part of the recipe is done, the user often is able to briefly recuperate before continuing

with the rest of the recipe.

10.6 Integration with the iCat

Of course this agent works fine on a pc, but the goals is to have it work on and with the iCat. For this

a module has to be designed. This module will allow a 2APL agent to run on the iCat. When the

agents are available to ‘run’ on the iCat, there are a lot of iCat features which can be used inside

2APL.

The multiple sensors (microphone and touch sensors, webcam) can provide useful data to be used in

2APL. The number of actuators available (loudspeaker and motors for head movement and facial

expressions) would ideally be activated through 2APL. To allow the microphone to be fully

operational, the iCat needs some form of speech recognition. Related to this is the speech output

which depends on speech generation. These last two requirements lead to a whole other research

area. On other fields such as vision, research also needs to be done in order to be able to incorporate

the iCat monitoring the recipe procedure and the user’s actions into the agent.

Last but not least, the different interaction style opens up a whole new field of research. Though the

iCat is made to be addressed with speech, the 2APL software used to develop the agent works with

messages in text form. Integrating the agent into the iCat therefore requires additional research on

parsing the speech input from the user to (text) messages the agent can use.


page 54 of 68

Bibliography 1. Philips Research. Technologies - Robotics. [Online]

http://www.research.philips.com/technologies/projects/robotics.html.

2. Philips. iCat Research Community. [Online] http://www.hitech-projects.com/icat/.

3. Practical Extensions in Agent Programming Languages. Dastani, M. M., Hobo, D. and Meyer, J-J.

Ch. 2007, Proceedings AAMAS.

4. Collaborative Plans for Complex Group Action. Grosz, B.J., and Kraus, S. 1996, Artificial Intelligence

86 (2), pp. 269-357.

5. Wooldridge, Michael J. Reasoning about rational agents. Cambridge Massachussetts / London,

England : The MIT Press, 2000.

6. Jade. Java Agent DEvelopment Framework. [Online] http://jade.tilab.com/.

7. 2APL: a practical agent programming language. Dastani, Mehdi. [ed.] Francesca Toni and Jamal

Bentahar. Special Issue on Computational Logic-based Agents, 2008, International Journal of

Autonomous Agents and Multi-Agent Systems (JAAMAS), Vol. 16(3), pp. 214-248.

8. Universiteit Utrecht. 2APL Homepage - A Practical Agent Programming Language. [Online]

http://www.cs.uu.nl/2apl.

9. Confirmations and Joint Action. Cohen, P. R. and Levesque, H. J. 1991, Proceedings IIJCAI.

10. Agent-Based Speech Act Generation in a Mixed-Initiative Dialogue Setting. Nieske L. Vergunst,

Robbert-Jan Beun, Frank P.M. Dignum, Rogier M. van Eijk, John-Jules Ch. Meyer. 2009 : [to be

published].

11. H. Strik, A. Russel, H. van den Heuvel, C. Cucchiarini, L. Boves. A spoken dialog system for the

Dutch public transport information service. Int. Journal of Speech Technology. 1997, Vol. 2, 2, pp.

121-131.


page 55 of 68

Appendix 1: Diagrams and flowcharts

Recipe phases and statuses

Agent

resting

Recipe

Selection

Recipe

Planning

Instructing

the user

Recipe

Finalization

Figure 1: recipe phases from start to end.

Searching

Message from user

Checking Checked Confirmed

Active Awaiting_reply( )

Ready Cancelled

Message from user

Planning

(gather_items)Planned

Planning

(instruction_list)

subRecipe

Request user

confirmation

Message

from user

Checks succesful? User confirms?

nono

yesyes

Figure 2: recipe statuses from start to end.


page 56 of 68

Dialog flowcharts

User request

Search recipe

Agent

informs user

Ask user

confirmation

No recipe foundNo executable

recipe found

Incomplete

recipe foundRecipe found

Agent requests

new search

User reply

yes no

Confirm recipe

Figure 3: dialog around selecting recipe


page 57 of 68

Recipe

missing items?

done

yes

no

Request user to

get items

(shopping)

User reply

Generate

instruction list

Find needed

utensils and

ingredients

Request user to

gather needed

utensils

User reply

done

Request user to

gather needed

ingredients

User reply

done

Start recipe

Start planning

If available for teaching

Ask user if wants to learn

Ask if user

remembers

yes

no User reply

If recently

learnedyes

no

(Save wants to learn parameter)

Continue to next instruction

Figure 4: dialog around planning a recipe


page 58 of 68

Check skill

Offer to

perform action

Inform agent will

perform action

request user to

perform action

deny

deny

ok

okAgent performs

action

Update beliefs on

executed action

yes

no

next action

Go through

instruction list

Does agent have

skills?

Wait until the

user is done

yes

no

Inform user needs

to do actionok

deny

Request user to

abort recipe

no, restart skill checkyesAbort recipe

Both skill &

not teaching

if user wants to

learn action

Introduce teaching

processyes

Figure 5: dialog around handling an instruction


page 59 of 68

Appendix 2: 2APL illustrations

Recipe selection phase

Beliefs:

atomicAction(A) :- action(A, []).

checkAtomic(X) :- atomic(X,_).

atomic(Action, Action) :- atomicAction(Action).

atomic(Action, Result) :- action(Action, List), atomic(List, Result).

atomic([H|T], Result) :- atomic(H, X), atomic(T,Y), Result = [X|Y].

atomic([H|T], Result) :- atomic(H, X), T = [], Result = [X].

atomic(Recipe, Result) :- recipe(Recipe, List), atomic(List, Result).

Figure 1: belief base rules to check decomposition of a recipe or action.

PC-Rules:

//checks if a recipe can be decomposed into atomic actions

//reports any missing actions

checkRecipeDecomposition(R) <- currentRecipe(Code,User,checking) |

{

if B(not checkAtomic(R)) then

{

//if recipe/action is defined in actionDecompositions

if B(action(R,_) or recipe(R,_)) then

{

B(firstDecomposition(R,FirstDec));

CreateList(R,User,workList,FirstDec);

while B( not listEmpty(R,workList)

and list(R,User,workList,[H|T]) ) do

{

checkRecipeDecomposition(H); //recursive check

RemoveFirstElement(R,User,workList)

};

RemoveList(R,User,workList)

}

else

{

Missing(Code,decomposition,R) //report missing action / recipe

}

}

}

Figure 2: 2APL algorithm for listing missing decompositions of actions or recipes.


page 60 of 68

Recipe planning phase

Generate

instruction list

Check actionUser

skill?

yes

noAvailable for

learning?

Add to

instruction list

no

yes

Ask user if

wants to learn

User

replyyes no

Save learning

parameter

next action

Agent

skill?

yes

noGet first

decomposition

recursive call

Figure 3: algorithm flow for checking actions on skills for creation of the instruction list.


page 61 of 68

Instructing phase

handleInstruction(Instruction) <- currentRecipe(Code,User,active) and

skillCode(Instruction,User,SCode) |

{

PauseForInstruction(Code);

//if both user and agent are skilled...

if B( SCode = 1 and not teaching(User,_) ) then

{

//...offer to perform the action

send(User, request, shallDo(Instruction))

}

//if it's an action for which only agent has skill...

else if B( SCode = 2 and not teaching(User,_) ) then

{

//...inform agent will perform the action

send(User, inform, willDo(Instruction));

//perform action in kitchen domain

//update beliefs on executed action

updateBeliefs(Instruction);

//remove instruction from the instruction list

RemoveFirstElement(Code,User,instructionList);

ContinueRecipe(Code, active)

}

//if it's an action for which only user has skill...

else if B( SCode = 3 ) then

{

//...request the user to perform the action

send(User, request, pleaseDo(Instruction))

}

}

Figure 4: different possible communication based on who has required skills

if B( Instruction = [SubRecipe | SubList] ) then

{

PauseForSubRecipe(Code);

if B( wantsToLearn(User,SubRecipe) ) then //check learning parameter,

//if set: start learning process

{

StartTeaching(User,SubRecipe); //add belief agent is teaching

send(User,inform,willTeach(SubRecipe)) //inform the teaching started

};

adopta(jointgoal(SubRecipe));

StartSubRecipe(Code,SubRecipe,User,SubList)

}

Figure 5: action taken when the agent encounters a sub-recipe.


page 62 of 68

PC-rules:

//message concerns information on finishing an instruction

message( User, inform, _, _, answer(Answer) ) <-

currentRecipe(Code, User, awaiting_reply(instruction, SCode))

and list(Code, User, instructionList, [Instruction|T]) |

{

...

if B( Answer = ok ) then //user accepts task

{

skip //simply wait until he is done

}

else if B( Answer = no ) then

{

PauseForMessage(Code, abort);

send(User, request, abort(Code)) //request abort...

}

else if B( Answer = done ) then

{

//update beliefs on executed action

updateBeliefs(Instruction);

//remove instruction from list after it's done

RemoveFirstElement(Code, User, instructionList);

//continue the instruction phase

ContinueRecipe(Code, instruction)

}

...

}

Figure 6: possible steps the agent can take after receiving an answer of the user.

Recipe finalization phase

...

//if there’s an unfinished recipe in hold for a sub-recipe...

if B(currentRecipe(Code2,User,subRecipe) and

list(Code2,User,instructionList,[[Code]|T]) ) then

{

CleanBelief(currentRecipe,Code,User,finished);

dropgoal(jointgoal(Code));

//remove the sub-recipe code from the instruction list

RemoveFirstElement(Code2,User,instructionList);

//...and resume the recipe

ResumeAfterSubRecipe(Code2)

}

...

Figure 7: steps taken on finalization of a sub-recipe.


page 63 of 68

Implementation fragments

Beliefs:

recipe( 'spaghetti', spaghetti_mozarella_meatballs, utensils([]),

ingredients([]) ).

recipe( 'spaghetti', spaghetti_meatballs, utensils([water_boiler, pan,

pan, oven_dish,bowl]), ingredients([water, spaghetti, meat, egg, tomato,

salt, pepper, oil, onion, garlic]) ).

Figure 8: recipe representation in 2APL.

Below: Figure 9: recipe actions and their decompositions.

Beliefs:

/* actiondecompositions.2apl */

recipe( spaghetti_mozarella_meatballs, [] ).

/* spaghetti_meatballs */

recipe( spaghetti_meatballs, [make_meatballs, cook_spaghetti,

bake_meatballs, make_spaghetti_sauce] ).

/* make_meatballs */

action( make_meatballs, [combine_meat_egg_parsley,

season_with_salt_and_pepper,

roll_mixture_into_balls] ).

/* cook_spaghetti */

action( cook_spaghetti, [boil_water, add_salt_to_water,

wait_until_water_boils, add_spaghetti,

cook_until_al_dente, drain_spaghetti] ).

/* boil_water (2 ways) */

action( boil_water, [fill_water_boiler, turn_water_boiler_on,

wait_until_water_boils, put_pan_on_gas,

put_water_in_pan, turn_gas_on, wait_until_water_boils]).

action( boil_water, [put_pan_on_gas, put_water_in_pan, turn_gas_on,

wait_until_water_boils] ).

/* bake_meatballs (2 ways) */

action( bake_meatballs, [heat_pan_with_oil, fry_meatballs,

put_meatballs_aside, clean_pan] ).

action( bake_meatballs, [line_baking_dish_with_foil,

place_meatballs_in_dish, turn_oven_on_at_175C,

put_dish_in_oven, wait_8_12_minutes,

turn_oven_off, remove_dish_from_oven] ).

/* make_spaghetti_sauce */


page 64 of 68

action( make_spaghetti_sauce, [heat_pan_with_oil, saute_onion,

optional_saute_garlic, slice_tomatoes,

add_sliced_tomatoes, add_meatballs,

let_simmer_5_minutes] ).

Figure 9 [continued]: recipe actions and their decompositions.

Beliefs:

skill( agent, cook_until_al_dente).

skill( agent, turn_water_boiler_on).

skill( agent, turn_gas_on).

Figure 10: skill information for the agent.

Beliefs:

/* Utensils */

available(pan).

available(bowl).

available(oven_dish).

available(water_boiler).

/* Ingredients */

available(gas).

property(gas, off).

available(water).

available(parsley).

available(salt).

available(pepper).

available(oil).

stock(spaghetti, 500).

stock(meat, 750).

stock(egg, 12).

stock(tomato, 10).

Figure 11: kitchen contents for the implemented example.


page 65 of 68

Appendix 3: Example dialog This appendix will introduce an agent-user dialog. The presented dialog is also used to implement an

example scenario in the created agent. Below is the recipe as presented in a cookbook. An example

dialog is written out as illustration of how the agent and user would communicate when preparing

this recipe. This is the kind of dialog the system is eventually required to be capable of handling and

generating.

The recipe Spaghetti with meatballs

Directions 1) Combine meat, egg and chopped parsley in a bowl

and season with salt and pepper.

2) Roll the mixture into small balls.

3) Cook the spaghetti: bring a large pot of salted

water to a boil. Add the spaghetti. Cook until al

dente (about 8 minutes). Drain the spaghetti.

4) Bake meatballs, there are two alternatives:

a. Heat pan and 2 tablespoons oil and fry

meatballs for approximately 3 minutes until

nearly cooked through and set aside, cook

in batches if needed, when completed clean the pan.

b. Place meatballs into an aluminum foil lined baking dish. Bake at 175C (fan forced) for

about 8 to 12 minutes.

5) Add remaining 2 tablespoons of oil and sauté onion and garlic in oil for 3 minutes, add sliced

tomatoes and meatballs, let them simmer for 5 minutes.

6) Serve meatball and sauce over the spaghetti.

I found this recipe online (source: http://www.recipezaar.com/373538) and made a few adaptations

myself to make it more suitable as an example recipe to be used in an example scenario and dialog.

Ingredients: - 750 g minced beef

- 400 g (± 3 medium) sliced tomatoes

- 500 g spaghetti

- 1 egg

- 1 tablespoon parsley (chopped)

- 1 onion (finely chopped)

- salt (to taste)

- pepper (freshly ground to taste)

- 4 tablespoons oil (for frying)

http://www.recipezaar.com/373538


page 66 of 68

The dialog Below is the dialog between user and system representing a recipe scenario. It describes a recipe

scenario from beginning to end.

1 System Hello! 2 User Hello! I’m hungry.

3 System What would you like to cook?

4 User Eh, I’m in the mood for pasta. Do you have any recipes I could make? 5 System Okay. I’ll look for a recipe.

[pause] I found a recipe for spaghetti with mozzarella stuffed meatballs. How’s that?

6 User I don’t know, I’m not a big fan of mozzarella, but I like spaghetti. 7 System Ok, I will look for another recipe.

[pause] I found a recipe for spaghetti with meatballs, is that ok?

8 User Oh yes, that sounds really nice. 9 System For how many people do you want to prepare the recipe?

10 User Five people.

11 System Okay. For this recipe you need: 750 grams of minced beef.

12 User I’ll get that, hold on. [pause] Ok, got it.

13 System Three medium tomatoes, an onion and a tablespoon of chopped parsley.

14 User Okay let me get that. I have to step out to the garden now. I’ll tell you when I’m done with the parsley. [pause] Yep, done.

15 User Okay, I’ve got the parsley here, should I cut it immediately or can I leave that until later?

16 System It’s ok to wait until later.

17 System 500 grams of spaghetti.

18 User Ok. That was easy, it comes in 500 grams packages. 19 System Great. Next, you need an egg.

20 User Okay, what next?

21 System Some salt and pepper. 22 User I have that, in the pepper mill and salt dispenser.

23 System And the last thing we need is 4 tablespoons oil.

24 User I’ve got some diet vegetable-oil. Is that ok?

25 System Yes, that’s fine. 26 Ok, you can start by washing and chopping the parsley.

27 User Okay. [pause] Okay, it’s dry now, I’ll cut it.

28 System Okay.

29 User [cutting sound] Okay, done.

30 System Now combine the meat, egg and chopped parsley in a bowl and season it with salt and pepper.


page 67 of 68

31 User [pause] Also done! What’s next?

32 System Next you need to roll the mixture into small balls.

33 User How small?

34 System That’s up to you. The smaller you make them, the more balls you’ll have in the sauce. Making them a bit smaller makes them more easy to eat.

35 User Okay. [pause] All done!

36 System Great! We’ll continue with making the spaghetti.

37 User Good.

38 System First boil some water. 39 User I’ll use the water boiler, that’s a bit faster I believe.

40 System Okay

41 User Okay, the water boiled, what’s next? 42 System Now put the water in pan on the gas, add some salt and wait for it to boil again, this

won’t take long. 43 User How much salt should I add?

44 System The salt is added to boost the flavor of the spaghetti, it’s up to you how much you want to add. You can also leave it out if you want.

45 User Okay, I’ll just add a little salt. [pause] Okay, the water’s boiling in the pan now.

46 System Okay, then you can now add the spaghetti.

47 User [pause] Done.

48 System Since this needs to cook for 8 minutes, we can continue with the sauce.

49 User Can you keep the time for me? 50 System Sure.

Ok, if you use the oven I will suggest that you turn it on. 51 User Oh sorry, there’s a problem with the oven. Is there another way?

52 System Sure, we’ll just use a pan. First however, you need to chop the onion. If you like, you can also add some garlic.

53 User [chopping sound] Okay, I chopped the onion and also some garlic.

54 System Great. Now, get a pan and heat it with 2 tablespoons of oil.

55 User [pause] Ok, that’s ready. What now?

56 System Now you can fry the meatballs…

57 User Right. 58 System …for about 3 minutes until nearly cooked through and then set them aside.

59 User How do you mean aside?

60 System Well, when they’re nearly cooked through, just put them in a bowl or oven dish. 61 User I think they’re done, I’ll get an oven dish.

[pause] Okay.

62 System The spaghetti is done, you can turn off the gas. 63 User Should I drain the spaghetti?


page 68 of 68

64 System Yes, we need to do that first before we continue with the sauce.

65 User Ok, that’s done.

66 System Add two tablespoons of oil to the pan… 67 User Again?

68 System …and.. [pause] Yes, it’s needed for preparation of the rest of the sauce. Afterwards, sauté the onion and garlic for 3 minutes.

69 User [pause] Okay, that’s done, now add the tomato?

70 System Yes, but first slice them of course. 71 User Of course

[pause] Okay. And the meatballs are next.

72 System Correct, go ahead and add them now.

73 User Okay, done.

74 System Now let that simmer for about five minutes. In the meantime, you can drain the spaghetti.

75 User Good idea. [pause] Ready.

76 System Well, that’s all. The recipe is finished.

77 User Ok, thanks. I’ll gather everyone.

78 System Bon appétit. [smile]

79 User Merci [wink]

master thesis - simonis 3206963

Documents