Building Artificial Memory to Autonomous Agents using Dynamic andHierarchical Finite State Machine

Tatiana Figueiredo Evers and Soraia Raupp Musse{tatiana, soraiarm}exatas.unisinos.br

PIPCA - Masters in Applied ComputingUniversity of Vale do Rio dos Sinos – UNISINOS

Av. Unisinos 950, 93022-000 Sao Leopoldo, RS, Brazilphone: +55 51 5908161 and fax: +55 51 5908162

Abstract

This paper presents a model for building agents’ memoriesin virtual environments. This approach allows the gener-ation of agents’ memory during the simulation, withoutuser intervention. Moreover, the memory can be used togenerate behavioural reactive rules. We use dynamic andhierarchical finite state machine (DHFSM) in order torepresent the agents past experiences.

Keywords:Behavioral animation, Artificial memory, FiniteState Machine

1. Introduction

Autonomous agents need artificial memory in order to learnthe best way to accomplish their tasks. Although many tasksare solved by purely reactive systems, more complex tasksrequire the utilization of mental models, like memory [5].This paper presents an approach to build artificial memoryof autonomous agents using dynamic and hierarchical Fi-nite State Machine (DHFSM). At the beginning of the sim-ulation, the agents do not have any data in their memory.During the execution of the tasks lists and triggered events,the memory of each agent is built according to the appliedbehaviours. The memory of task action and behavior ofagents is represented through a DHFSMAfter the simulation the user can visualize and modify thegraphical representation of DHFSM, as well as generate aset of time-based behavioral rules. Afterwards, the agents’rules and the DHFSM can be used as input to another sim-ulation or even simulator, as discussed in this paper.The next section presents the related works and the DHFSM

functionalities. Section 3 describes the way the memory isbuilt in our work. In section 4 we present a case study andsome results. In the last section we discuss some conclu-sions and future work.

2. Related Works

There are several studies about behavioural animation.Reynolds [14] developed Boids: a model to simulate aschool of fish and a flock of birds using particle systems.This model managed competing goals (flock centering,collision avoidance and velocity matching).Tu and Terzopoulos [18] have worked on behaviouralanimation for creating artificial life, where virtual agentsare endowed with synthetic vision and perception ofenvironment. The repertory of fish behaviours relies ontheir perception of the dynamic environment, and the fishreactions are not entirely predictable because they are notscripted.Badler et al [2] [3] developed several behavior modellingmechanisms, including PaTNets (parallel Transition Nets).Animals (real or artificial) can only learn from experienceif they remember past events in their memory. The sizeand importance of the memory relies on the way in whichthe environment changes. Some authors have relatedthe problem of artificial memory in agent’s simulations[10] [15].Musse [12] used memory to represent the group knowl-edge from past experiences. The group memory deals withinformation about the past tasks (motion or action) and timeneeded to accomplish the tasks (frames). Noser et al [13]used dynamic octrees to represent the visual memory ofeach agent. This approach allows an actor to memorizetheir environment and to adapt according to the changes inthe dynamic environment.

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE

Koza [9] describes a basic form of memory, involvingfunctions which read and write values to small fixedvariables (figure 1 a) ). Teller [16] extends the Koza’smodel by providing evolved programs with functions towrite and access linear array indexed memory (figure 1 b)). Andre [1] extended Teller’s approach to a memory thatwas isomorphic to the world. The method is applied to aproblem of ‘gold’ collection in a five by five toroidal grid(figure 1 c) ).Brave [5] proposed an approach which allows the cre-ation of dynamic mental models. The memory is notpre-designed by the animator of the system and is a set ofisolated nodes that the agent connects freely (figure 1 d) ).The model uses genetic programming to produce agentscapable of building and using stored representations. Thismethod is applied to the motion planning problem.

Figure 1. Five types of memory (figure in-spired in Brave [5]) (a) settable variables, (b)linear indexed memory, (c) 5x5 memory iso-morphic to a 5x5 world, (d) pallet of isolatednodes with dynamically labeled connections.(e) our memory model using DHFSM

In figure 1 e) we can observe a DHFSM generated using ourapproach: M1, M2, M3 and M4 represent the transitionsand a, b, c, d are the states.

2.1. Finite State Machine

A Finite State Machine (FSM) consists of a set of states, aset of inputs, a set of outputs, and a state transition function[7]. Formally a FSM is a quintuple:

(Q,∑

, δ, q0, F ) (1)

whereQ is a finite set of states,∑

is a finite alphabet ofinputs,q0 ∈ Q is the initial state,F ⊂ Q is a set of finitestates,δ is transition functionQ×∑ → Q, which isδ(q, a),it specifies one stateq to each inputa.The transition function takes the inputs and the current state,and returns a single new state and a set of outputs. FSMscan be used to represent deterministic behaviors. Usually, aFSM is represented using state transition diagrams.

2.2. Hierarchical FSM

A hierarchical FSM (HFSM) is an extension to the tradi-tional FSM, where each state of HFSM can be either aHFSM or FSM. Like a FSM, a HFSM consists of statesand transitions. However it allows to create states that con-tain other states and transitions. HFSM like procedural pro-gramming languages uses functions to abstract complex al-gorithms, HFMSs provide abstraction through compositestates [8]. Some developer game engines, like Motivate [8]use HFSM: the behavior for the actor is created through theBehavior Editor, the process consists of building the HFSMthrough a graphical editor.Several authors have discussed and created models based onFSM to build the behaviors of the agents.Kearney et al [6] proposed a Hierarchical Concurrent StateMachine (HSCM), this approach uses multiple, concurrentstate machines.More focused on urban context, Donikian et al. [11] cre-ated a model to simulate autonomous vehicle behavior andgenerate scenes in driving simulation. He proposed an ex-tension of HSCM [6], one hierarchy of parallel transitionssystems (HPTS). The decisional module consists of a reac-tive system, which is composed of HFSM representing eachone of possible behaviours.Ulincny [19] proposed a model for Crowd simulation basedon the combination of rules and FSM for controlling agents’behaviors. This approach is based on layers. At higherlevel, the rules select complex behaviors based on agent’sstate and environment state. At lower level, complex be-haviors are implemented by HFSMs. Each behavior is con-trolled by one FSM.Our approach does not use pre-defined HFSMs to define ac-tions of the actors like others models ( [6] [8] [11] [19]),our DHFSMs are created in real time to model the memoryof the agents and describe their past experiences.

3. A Model for Building Artificial Memory

According to Thalmann [17], memory is generally definedas the process of reproducing or recalling what has beenlearned and retained especially through associative mecha-nisms.Our model builds the agents’ memory using a list of

2

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE

behaviors executed in the past time. In fact, the inputinformation that an agent can have are: intention list andevents. This information is time-based or priority-based.Our approach does not execute tasks in parallel, if one taskconflicts with another one, then our model takes a decisionbased on input information presented in section 3.3. Eachtask is represented as a node in the FSM.

Goto restaurant t 10

Figure 2. A node in the memory: the task is goto restaurant, the state of the agent is walkingand this task has started in time 10.

When another task is executed, we create a new node repre-senting a new state and its links.In figure 3 this situation is graphically represented. Whenthe simulation finishes, a complete description of all agentbehavior is completed. The nodes maintain important in-formation about simulation (what agents do and when,where they went, their emotions, when their emotions havechanged, etc).

Figure 3. Creating a transition and a new state

3.1. Task Planning

We deal with a script where we specify the possible actionsto be applied by the agent. There are three action types thatcan be executed by the agents:

Goto The agent walks from its position to some place inthe virtual environment.

Play The agent plays some animation file.

Change The emotional status of agent changes.

[<ID>] <action> [t <time>][p <priority>]<ID> := ’agent_identifier’<action> := Play <what> |

Goto <where>|Change <parameter> <value>

<what> := ’animation_file’<where> := ’Place’<parameter> := Happiness |

Sociability |Communicability |Leadership

<value> := ’int’<time> := ’float’<priority> := ’int’ | ’string’

Figure 4. Task script

< ID > represents the agent identifier indicating whichagent must execute the task, if this identifier is not spec-ified, all agents must execute this task.< action >means one of the possible actions (Play, Goto, Change).< what > is the name of the animation file that the agentmust play. < where > is the place that the agent mustgo. < parameter > represents the emotional status of theagent.The agent has four emotional parameters.Happinessrepre-sents how happy the agent is happy.Sociabilityrepresentshow sociable it is.Communicabilityrepresents the level ofcommunication, and Leadership represent the tendency ofthe agent to be leader. This status is represented by a value< value > between 0 and 1. And< time > correspondsto the time when the task should be executed.The priority and time describe the order of task appli-cations. The priority is represent by semantic words,these words make it easier to write the task list for theanimator. We use decreasing priorities.DANGER, PANICand LIFE IMPORTANCEhave priority 1, IMPORTANThas priority5, the priority value is bigger as it deals whenless important feelings.

• DANGER

• PANIC

• LIFE IMPORTANCE

• IMPORTANT

• VITAL

• NORMAL

• LOW

3.2. Scheduling

We have a time-based or priority-based scheduler to definethe execution order of the task list. It uses priorities whenone or more tasks are conflicting. This fact can be seen infigure 5:

3

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE

Goto restaurant t 12 p VITAL (1)Goto bathroom t 12 p IMPORTANT (2)

Figure 5. Conflicting tasks

The two tasks in figure 5 should be executed in time 12.However the scheduler will select the task with higher pri-ority. The task (1) is pushed in a stack until the task (2)has been completed. When a task is interrupted, we save inthe memory node the time when this interruption happened(tint). When a task is completed, the time information issaved too (tf ).As seen in figure 6, inputs to the scheduler are: the task list,a configuration file (containing the initial information aboutthe agents), the emotional state of the agent and the events.Events are tasks (Goto, Play and Change) sent by user dur-ing the simulation. The user can send tasks to a specificagent or to all agents. The functionality of the scheduler isshown in figure 6. The possible inputs to the scheduler are

Figure 6. Decision module

events, emotional state, task list and configuration file andjust one task will be executed by the agent. The tasks arenot parallel, but are executed sequentially.

3.3. Handling with the Memory

After the simulation, we have a complete description of pastexperiences of all agents stored in their memories. Thus,the user is able to modify these memories using the graph-ical interface, as consequence the DHFSMs can be updated(deleted, modified or created) as well as to define some hi-erarchy to FSMs. The animator can visualize the memoriesand re-simulate them. Afterwards, behavioral rules can begenerated according to the memory of agents. These rulesare time-based and deal with interruption time as well asother information saved in the memory.

4. Results

To see how this system works in practice, we will consider aspecific simulation: there are five agents (Bob, Carol, Lara,John and Patty). The Configuration file is shown in figure 7:

NAgents 5 (1)

Agent Bob { (2)Happiness 1 (3)Sociability 0.5 (4)Communicability 0.5 (5)Leadership 0 (6)Obj Bob.obj (7)

}

Agent Carol { (8)Happiness 1 (9)Sociability 0.5 (10)Communicability 0.5 (11)Leadership 0 (12)Obj Carol.obj (13)

}

Agent Lara { (14)Happiness 1 (15)Sociability 0.5 (16)Communicability 0.5 (17)Leadership 0 (18)Obj Lara.obj (20)

}

Agent John { (21)Happiness 1 (22)Sociability 0.5 (23)Communicability 0.5 (24)Leadership 0 (25)Obj John.obj (26)

}

Agent Patty { (27)Happiness 1 (28)Sociability 0.5 (29)Communicability 0.5 (30)Leadership 0 (31)Obj Patty.obj (32)

}

Figure 7. Configuration file

In (1) we define the number of agents. In other lines (2 -32) we specify the initial emotion parameters and the objfile with the geometric description of the agent (vertex andfaces) for each one. The task list deals with the followinginformation (figure 8):

Bob Goto bank t 7 p IMPORTANTCarol Goto bank t 7 p IMPORTANTJohn Goto bank t 7 p IMPORTANTPatty Goto park t 7 p NORMALLara Goto park t 7 p NORMAL

4

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE

Carol Goto bank t 21 p NORMAL (1)Goto restaurant t 12 p IMPORTANT (2)

Figure 8. Task list

In this script (figure 8), Bob, Carol and John should go tothe bank at time 7. As the same time, Patty and Lara shouldgo to the park. At time 12 all agents go to the restaurant.At time 21 Carol comes back to the bank. At the end of thesimulation the memory of all agents is modelled using FSM(Carol’s memory is shown in figure 9).

Figure 9. Carol’s memory

Although task (1) orders Carol to go to the bank, she is ableto execute this task only when she finishes task (2).The Carol’s memory is represented through the followingautomatically generated rules:

if (t == 1) {State = Waiting

}if (t == 7) {

State = WalkingGoto Bank

}if (t == 10){

State = Waiting

}if (t == 12){

State = WalkingGoto restaurant

}if (t == 21){

State = WalkingGoto Bank

}if (t == 26) {

State = Waiting}

Figure 10. Carol’s memory Rules

Figure 11. System architecture

This memory model is integrated with our framework [4]where visualization is completely isolated from the be-havioural model (figure 11). In fact, visualization is accom-plished by a separate process that runs concurrently with theprocess for the behaviour. In figure 12 we can see one sim-ulation using RTKrowd viewer (It is based on RTK MotionToolkit). We use the rules generated from memory to writea script for the simulation.

Figure 12. Simulation using RTKrowd

The graphical interface to build the agent’s memories isshown in figure 13.

5

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE

Figure 13. Graphical interface to build memo-ries

5. Conclusion

We have described in this paper a model to build ArtificialMemory to autonomous agents using DHFSM created inreal time. A novel idea of this paper is to present the possi-bility of creating and modifying the agents’ memories andto generate a set of time-based rules. These rules can beused to animate the agents in others rule-based systems, asthe HFMS can be used too. To illustrate the memory con-cepts, we presented a simulation involving 5 agents, anddiscussed some aspects.Our designers easily modified DHFSM using the graphicalinterface. For future work we plan to investigate the follow-ing issues:

• How parallel tasks can be executed.

• Building HFSM automatically.

• Creating interaction between agents.

Concerning the scalability of the system, it can become aproblem when there are many agents simulated for a longtime. In this way, as the memories are associated to the in-dividuals, the amount of memory used by the system canincrease until having all memory occupied. As future work,we plan to provide an optimization to the occupied memoryin order to group common memories, for instance generat-ing groups of individuals that share the same knowledge.

References

[1] D. Andre. Evolution of mapmaking: Learning, planning,and memory using genetic programming. InProceedings

of the First IEEE Conference on Evolutionary Computation,New York, NY, 1995. IEEE Press.

[2] N. Badler, W. Becket, and J. Granieri. Towards real-timesimulated human agents, 1995.

[3] N. I. Badler and B. L. Webber. Planning and parallel transi-tion networks: Animation’s new frontiers. InPacific Graph-ics ’95, 1995.

[4] L. M. Barros, T. F. Evers, and S. R. Musse. A frame-work to investigate behavioural models.Journal of WSCG,10(1):40–47, 2002.

[5] S. Brave. The evolution of memory and mental models usinggenetic programming. In J. R. Koza, D. E. Goldberg, D. B.Fogel, and R. L. Riolo, editors,Genetic Programming 1996:Proceedings of the First Annual Conference, pages 261–266,Stanford University, CA, USA, 28–31 1996. MIT Press.

[6] J. Cremer, J. Kearney, and Y. Papelis. Hcsm: A frameworkfor behavior and scenario control in virtual environments,1995.

[7] J. Funge.AI for Games and Animation: A Cognitive Mod-eling Approach. AK Peters Natick Press, Massachusetts,1999.

[8] M. F. Inc. Motivate intelligent digital actor system: Userguide & tutorial, 1990.

[9] J. R. Koza. Genetic Programming: On the Programmingof Computers by Natural Selection. MIT Press, Cambridge,MA, 1992.

[10] J. C. McNamara and A. I. Houston. Memory and effi-cient use of information. Journal of Theorical Biology,(125):385–395, 1987.

[11] G. Moreau and S. Donikian. From psycological and real-time interaction requirements to behavioral simulation. InEurographics Workshop on Computer Animation and Simu-lation ’98. Spring-Verlag, 1998.

[12] S. R. Musse.Human Crowd Modelling with Various Levelsof Behaviour Control. Phd thesis, EPFL, Lausanne, 2000.

[13] H. Noser, O. Renault, D. Thalmann, and N. M. Thalmann.Navigation for digital actors based on synthetic vision,memory, and learning.Computers and Graphics, 19(1):7–19, – 1995.

[14] C. W. Reynolds. Flocks, herds, and schools: A distributedbehavioral model.Computer Graphics, 21(4):25–34, 1987.

[15] S. Shafir and J. Roughgarden. From animal to animats 3. InJ. M. D. Cliff, P. Husbands and S. Wilson, editors,Proceed-ings of the Third International Conference on Simulation ofAdaptative Behaviour, pages 221–227, Seattle, WA, USA,1994. MIT Press.

[16] A. Teller. The evolution of mental models. MIT Press, 1994.[17] D. Thalmann, S. R. Musse, and M. Kallmann. Virtual hu-

mansbehaviour: Individuals, groups, and crowds. InPro-ceedings of Digital MediaFutures. Bradford, 1999.

[18] X. Tu and D. Terzopoulos. Artificial fishes: Physics, loco-motion, perception, behavior.Computer Graphics, 28(An-nual Conference Series):43–50, 1994.

[19] B. Ulicny and D. Thalmann. Crowd simulation for interac-tive virtual environments and vr training systems. InEuro-graphics Workshop on Animation and Simulation ’01, pages163–170. Springer-Verlag, 2001.

6

Proceedings of the Computer Animation 2002 (CA 2002) 1087-4844/02 $17.00 © 2002 IEEE