Specification of an Open Architecture for Interactive Storytelling

Nicolas Szilas1, Thomas Boggini1, Monica Axelrad1, Paolo Petta2, Stefan Rank2

1 TECFA, FPSE, University of Geneva, CH 1211 Genève 4, Switzerland

{Nicolas.Szilas, Thomas.Boggini, Monica.Axelrad}@unige.ch 2 Austrian Research Institute for Artificial Intelligence OFAI,

Freyung 6/6, A 1010 Vienna, Austria {Paolo.Petta, Stefan.Rank}@ofai.at

Abstract. This article introduces OPARIS, an OPen ARchitecture for Interactive Storytelling, which aims at facilitating and fostering the integration of various and heterogeneous Interactive Storytelling components. It is based on a modular decomposition of functionalities and a specification of the various messages that different modules exchange with each other.

Keywords: Interactive Digital Storytelling, Interactive Narrative, Architecture, Narrative Engine, Behaviour Engine, Animation Engine.

1. Motivation for an Open Architecture

This last decade, a significant number of Interactive Digital Storytelling (IDS) systems have been implemented. These systems include: DEFACTO [1], IDtension [2], Teatrix [3], Mimesis [4], Façade [5], Scenejo [6], FearNot [7], EmoEmma [8], Beergarden [9] (see [10] for a review). In order to provide the end-user with influence on the storytelling process, these systems include a core “narrative engine”, that is responsible of choosing/generating appropriate actions during the experience, and various peripheral yet essential technologies for input and output, i.e. for enabling interventions by the end-user and for presenting elements as calculated by the narrative engines: menu-based interfaces [11], text recognition/generation technologies [6,9,12], behaviour engines [13,14], camera management systems [15,16], light engines [17], etc. These components are also relevant for games and pure story generation systems. Research on IDS needs a large range of the above technologies, because when it comes to evaluate the value of a technology, one needs to provide a whole interactive experience to an end-user. Most systems above integrate its own set of technology, within an in-house architecture, developed over a period of several years.

However, there has been limited integration between these systems to date. This lack of integration is problematic because it limits the ability of each research team to experiment/validate/demonstrate their core contribution. Furthermore, with limited integration, the spectrum of available technologies ready for applications remains limited. For example, researchers focused on drama management would use a text-

based rendering or cartoon-like rendering not only by design choice but also according to feasibility constraints. Similarly, researchers on advanced behavioural management would use a simple branching story for similar reasons. Research labs able to cover the full range of the above technologies would still tend to continue with their in-house technology, rather than explore alternative external technology. This lack of integration is certainly related to the variety of approaches that exists in the field, and the large variability in the related concepts and terms [18].

This context has been acknowledged within the European project Integrating Research on Interactive Storytelling (IRIS) [19], started in 2009. Based on IRIS’ technical expertise in various IS technologies, our goal is to propose an open architecture for IDS called OPARIS, with an Application Programming Interface (API), that will enable various IDS components to communicate with each other. Before presenting the architecture’s design, this general goal is expanded on in more detail in the following section using a structured series of motivations.

1.1 Engineering motivations

1.1.1 Interchangeability of modules Develop a better suited module: Starting from an existing IS prototype, it is interesting to replace a module by another one, to better answer a specific need. For example, a project might need to develop a specific “comics engine” to represent behaviours with interactive comics strips. An open architecture should provide an easy way to “unplug” one module and replace it with the newly developed one, without having to modify the rest of the architecture.

Use an existing third-party module: In this case, a relevant module has been developed elsewhere. This external module is possibly not compliant to the API. Nevertheless, it is worth being able to integrate this module with limited effort. This is the case for the many existing systems and components listed above. Note that there are many reasons for replacing a functional module by a third-party one. Either the existing module is replaced by a more powerful one. Or, for technical, political, legal reasons, a module might become unavailable and needs to be replaced by another equivalent module.

Stub module: For making the authoring process easier, it is useful, during the authoring process, to test part of a story with a “stub module”, in order to bypass or at least simplify the authoring related to the replaced module. This need corresponds to the practice of using stubs and mock objects in software testing in order to simplify testing a single module in isolation.

1.1.2 Wide-spread adoption of modules

Well defined API for a module: A given IDS module is more likely to be adopted and used if it is provided with a well-specified and documented API. This paper aims to provide such an API.

Multiple Use: As soon as several IDS modules, based on the current API (and subsequent versions) will be developed within the IDS community, modules that are

compliant with the architecture will naturally find more opportunities for being reused in various contexts.

Development of new functionalities: With a clearly specified API, it makes it easier for a developer to create a new module for a new functionality and to test it within a whole system (e.g. a Music Engine for IDS specifically).

1.2 Theoretical Motivations

Our motivations go beyond the mere interoperability. A new architecture should also support the specificity of IDS and in particular the fact that what is handled is not just data but relates to narrative. Specifying a modular architecture also brings the opportunity to study and clarify the narrative nature of the IDS medium.

1.2.1 Dependence/Independence between the Narrative and the Medium The specification of a principled API between the Narrative Engine on the one hand and media-related modules on the other hand implies a narratological point of view where the narrative level possesses its own existence, independently of the medium used to represent the story. The proposed architecture enables to test this hypothesis, by studying which narrative phenomena can be described within the Narrative Engine, which one cannot and finally which information needs to circulate between modules to generate these phenomena. Examples of such phenomena include parallelism, time dilatations, discourse/story relation, etc.

1.2.2 Fostering Narrative Intelligence Our view on IDS is that it must be guided by narrative theories, should it be “classical” theories, or new medial theories. Therefore, the proposed architecture aims at reflecting this view by using, as much as possible, narrative concepts and terms to describe what circulates between the architecture's components. This aims at encouraging IDS engineers and researchers to reason in these terms, rather than in raw Artificial Intelligence or Computer Science terms. This is in line with the philosophy of Narrative Intelligence, a field where “humanist notions of experience and humanity fruitfully inform technological conceptions" [20].

2 General Design of OPARIS

2.1 Design Principles

2.1.1 Flexibility Building an open architecture for IDS is different from designing a software architecture from scratch. On the one hand we target a unified API, so that any module that respects this API could straightforwardly communicate with each other. On the other hand, we need to take into account the set of existing IDS systems and components and be able to integrate them into the architecture. Therefore, a key

principle of OPARIS is to provide several levels of integration, from an ideal case, where each module works according to the documented API, to cases where modules have followed different specifications.

2.1.2 Simplicity

Designing an API is a critical activity that needs to take into account its users. In our case, users are researchers and developers who carry out an IDS project, and possibly advanced media designers who are assembling modules to create a system for a given project. This population has some knowledge in computing but its focus lies more in narrative, Artificial Intelligence, Human-Computer Interaction, authoring, etc. Therefore, the architecture specification must remain simple, avoiding too much abstraction and not be over-constrained.

Furthermore, it should enable several levels of integration, depending on the invested effort. In other words, it should be possible to:

- Easily integrate a module into the architecture to cover basic needs; - Use more sophisticated communication mechanisms for advanced needs.

2.1.3 Narrativity As explained in Section 1.2.2, we aim at fostering a narrative point of view on IDS via the architecture. Therefore, OPARIS uses an existing set of already defined IDS concepts [21] to specify and name the data circulating between modules.

2.2 Modular decomposition

The modular decomposition chosen for OPARIS is inferred from the various existing IDS architectures developed within each system [5,22,23,24]. Given the variety of underlying research, this decomposition cannot be unique. The granularity of modularity varies between systems. For example, some prototypes clearly identify thee different levels – drama management, behaviour, animation - [5,22], while other merge behaviour and animation together. More specialized research, in lighting [24] or camera management [16], often uses a more detailed architecture that isolates these functionalities from the rest of the architecture. This implies that modules should be differentiated from functionalities. Modules are isolated software components that perform one or more functionalities. Some functionalities are easily isolated from the rest of the system, and are good candidate to form a module. Others are usually integrated within a larger module, as it is the case for light management or rendering.

Fig. 1. Two examples of specific architectures.

Table 1 summarizes the modules and their functionalities as proposed in the

architecture design. The two last modules in Table 1 are composite modules. They gather several of the functionalities carried out by non-composite modules. Several configurations of the architecture can be built, depending on whether a module is used or not and whether some composite modules are used. (See Figure 1.)

Table 1. Modules in the proposed architecture, with the associated typical functionalities.

Module Functionalities Narrative Engine (NE) Calculates actions that must be displayed to the end-user.

Behaviour Engine (BE) Transforms behaviours into series of elementary animations.

Animation Engine (AE) - Animates a 3D or 2D model of a character. - Renders the animation. - Displays other information to the end-user.

User Interface (UI) Takes input from the end-user; In particular, enables the end-user to select the action of his/her character.

Music Engine (ME) Plays background music according to the unfolding of the story.

Theatre (TH) Displays the story events and descriptions and manages the end-user's interaction. In case of 3D, it encompasses functions carried out by a BE, an AE, Camera Management, and a UI.

Small Theatre (ST) Displays the story events and descriptions and manages the end-user's interaction. In case of 3D, it excludes the BE.

2.3 Physical implementation

Each module is an independent software component that communicates with the rest of the architecture with IP sockets. This has several well-known advantages:

• It avoids constraining the users of the open architecture in terms of programming language or operating system.

• It connects easily to commercial technologies such as game engines (Unreal Tournament, Unity3D) or multimedia platforms (Flash, Max/MSP).

• It enables distributing the whole system across several machines. XML has been chosen for coding the data circulating within the architecture, for the following reasons: - XML data are more readable for testing, debugging and authoring. - Many programming languages benefit from existing libraries to code and decode the messages transmitted in XML.

Within XML, a more constrained coding such as XML-RPC or SOAP has been considered. It would have had some benefits for programming languages that are able to easily code and decode these formats. The idea was finally rejected because in the general case it would have made the integration more complex and less flexible.

The specification of the API for each module (see Section 3) is supported by a XML schema (XSD), which is provided to the users of the open architecture.

2.3.2 Centralized orchestration

As described above, the architecture requires each module's output to be compatible with every other module's input. For example, in a recent effort, within the IRIS project, to connect the IDtension narrative engine to a Theatre developed by INRIA, IDtension needed to convert all its output actions, which are scenario dependent, into specific behaviours for the Theatre.

The conversion effort was made on the NE side, which is arbitrary (it could have been done on the BE side). This raises two issues: • If we want to replace the BE, then we need to redevelop the conversion at the

narrative engine level, in order to fit with the new existing behaviours in the new BE. This hinders the interoperability of the whole architecture.

• With an increasing number of modules, the necessity of rewriting conversion code has a stronger impact and becomes a serious problem.

To solve this issue, a new module is introduced. It is a central module to which each module connects. This module follows the design pattern of a Mediator: it “promotes less coupling by keeping objects from referring to each other explicitly, and it lets you vary their interaction independently” [25] This new module has been termed the Director. It performs the three following operations: • It receives all messages from the other modules and redirects them to the proper

module(s), based on the content of the messages. The modules do not indicate to which other module(s) the message is intended to. This approach makes integration easier, by avoiding protocol-related messages.

• It performs a semantic conversion between modules. It translates the scenario-specific vocabulary outputted by the NE to the behaviour's catalogue terms in use by the BE.

• If a module is not compliant with the API, it can perform a syntactic conversion of data from and to this module.

The Director plays a role of both routing and conversion. For the conversion, taking advantage of the XML framework, the Director uses an XML transformation language, XSLT (Extensible Stylesheet Language Transformations), dedicated to transforming an XML document into another XML document. A single “.xslt” file enables to manage all the conversions needed. This enables the building of a specific architecture without entering into the director. The effort of writing this file is lower than entering into the code of one or more specific modules. The complexity of the XSLT sheet depends on the compliance of the modules to the API. If they are compliant, it only needs to deal with semantic conversion of messages.

The Director is realized as an IP server, with all clients connected to it as clients (but see Section 2.3.4). Note that the Director still remains optional: the API can be used without it, by connecting modules directly. The choice of which module is a server or which is a client is left open in that case.

2.3.3 Sharing Narrative Data

In numerous situations, data from one module to another does not need to be transmitted directly as a command but rather should be shared in a central place, accessible to all modules. Therefore, a module called the Shared Narrative States Repository (SNSR) is introduced. It enables any module to store narrative data that is useful to one or more other modules. These modules can either take these data when they need it (pull method) or subscribe to certain data and receive notifications when it changes (push method).

The SNSR is an optional module that is provided as a facility to share narrative data. It comes with an XML-based protocol to access data. The structure of data is left open to the IDS architect. Nevertheless, it is recommended that the shared data follow the conceptualization of IDS concepts mentioned above [21].

2.3.4 Increasing openness

To further increase the openness of OPARIS, a module is introduced, an adapter, that is inserted between a module and the Director. It performs two transformations: • XML wrapping: If a module does not use XML in its communication, the

adapter wraps messages with tags to transform it into an XML message that can be further processed by the Director. And vice-versa, it transforms XML messages from the Director to non XML messages to the module.

• Client-Server inversion: If a module is not a TCP client but a server, the adapter ensures that it can still connect to the rest of the architecture, by acting as a client towards both the module and the Director.

Note that several adapters can be used within the same specific architecture.

3. Modules’ API

Each module described in Table 1 is specified via its API. Since OPARIS is language independent, the API is not described in a specific language (Java, C#, etc.). Instead, it focuses on communication protocols, i.e., all messages sent and received by modules are specified, first in an explicative reference document, then via examples of XML messages and the corresponding XML schema. The XML is accessible online [26]. Messages are coded by their names (in capital letter) and a series of nested parameters.

3.1 Communication between the Narrative Engine and the Behaviour Engine

Table 2 lists the 8 messages outputted from the NE, with their parameters. One of the main communication channels concerns the exchange of messages between the NE and the BE. It uses the ACTION_TO_PERFORM message and, in the other direction, the ACTION_RETURN message. The transmission of the action uses several datatypes: StructuredAction, DescribedAction and PerformanceConstraint.

StructuredAction contains a unique identifier, a type of action (a verb), an actor (described as a string) and, optionally a series of parameters. This reproduces a typical predicate-based structure. This data type is used both for narrative actions and for animations sent to the animation engine (see Table 3).

Here follows an example of a message from the NE to the BE: <?xml version="1.0" encoding="UTF-8"?> <message> <ACTION_TO_PERFORM> <act> <id>34</id> <actionType>talk</actionType> <actor>Amanda</actor> <actionParameters> <actionParameter parameterName="addressee"> John </actionParameter> </actionParameters> </act> </ACTION_TO_PERFORM> </message>

The above message corresponds to an action named “talk”, with the character Amanda as actor and the character John as a parameter of the talk action, the parameter being named “addressee”. (See [26] for the corresponding XML schema).

3.2 Communication between the Narrative Engine and the User Interface

Depending on the nature of the user interface, the information that must be sent by the NE to the UI is different. Therefore, the API contains two kinds of messages for sending possible user actions to the UI. POSSIBLE_USER_ACTION_MNU is used

for menu-based interface for which the unique identifier of the action and its text description are sufficient. POSSIBLE_USER_ACTION_ATT are used for attribute-based interfaces, that is interfaces in which the action is not directly chosen but selected according to some of its attributes. For example, in the commercial game Dark Earth, the user would control the dialog by selecting the mood of the character (light or dark). In these cases, the API sends one or more attributes (e.g. mood, emotional valences), that the UI uses for managing the implicit action selection.

Table 2. List of messages in the API that are sent by the NE. Data types are in italic. Parameters in parenthesis are optional. Parameters with a star can occur repeatedly.

Message name Parameters Description

ACTION_TO_PERFORM StructuredAction act (PerformanceConstraint constr)

Asks the BE to perform a given action.

POSSIBLE_USER_ ACTION_MNU

DescribedAction act Sends an action that the user can play to the UI, in the case of a menu-based GUI.

POSSIBLE_USER_ ACTION_ATT

Integer id (String description) String attribute*

Send an action that the user can play to the UI, in the case of an attribute-based interface

FRAGMENT String fragment String fragmentType

Sends a whole text fragment to the AE. This text is to be displayed directly to the user.

ASSOCIATION String fictDataName Integer actionId

Indicates to a menu-based interface that a fictional data is associated to a possible user action.

NO_ASSOCIATIONS - Invalidates previous associations.

CONTEXTUAL_FILTER String type String value

Provides contextual information that enables the UI to present the available options differently. (e.g. the current addressee).

DISPLAY_ INFORMATION

String type String value

Provides additional data to the UI, e.g. in a history-based UI [11], the annotated lines of the history.

In return, the UI sends back the USER_ACTION message, which only contains the unique identifier of the selected action.

For menu-based interfaces, only receiving the list of action would not support advanced features such as the selection of an action from the fictional elements it embeds, or the filtering of actions according to the addressee. Additional messages are introduced for that purpose: ASSOCIATION, NO_ASSOCIATIONS,

CONTEXTUAL_FILTER and DISPLAY_INFORMATION. This latter is weakly specified, to accommodate any additional information that the UI might need. The FRAGMENT messages are used to convey information as text, bypassing the action-based reasoning. It is useful for introductions, intermissions, and conclusions. Finally, PARANARRATIVE_COMMAND is used by the UI to inform the NE that the user has clicked on a “quit”, “pause”, “restart” button, etc.

3.2 Communication between the Behaviour Engine and the Animation Engine

This communication is similar to the communication between the NE and the BE. The BE sends an animation to the AE via the ANIMATION_TO_PERFORM message, and the AE sends back an ANIMATION_RETURN message. Other communication between the two levels is to be implemented via the SNSR (Section 2.3.3).

4. Examples

So far, the proposed API has been tested with the IDtension system. IDtension can now be used in various configurations. In the “traditional” text-based interface [2,11], the system is split into two independent modules: the NE and the history-based Theatre. In the 3D version, the same NE is used but with a 3D Theatre, based on the Unity3D engine [27]. Also, for testing purpose, this Theatre can be replaced by a “stub” theatre that enables the user to choose actions, to receive actions from the NE (possibly in parallel) and decide whether actions succeed or fail, and when. Another version of this stub theatre enables manually sending XML commands to the NE.

Fig. 2. Three different “theatres” for the same NE and the same story. The history-based theatre (top left), the 3D theatre (top right), the stub theatre (bottom).

5. Conclusion

In order to foster the collaboration between researchers, the reusing of technology, an improved understanding of computational narratology and the acceleration of IDS development, we have proposed OPARIS, an open architecture for IDS that includes:

− A modular and flexible organization of individual IDS software components; − A technical framework for this open platform, in terms of socket-based

communication and specific technical modules (Director, Adapter); − An API for each of these modules, in the form of messages circulating

between modules, these messages being guided by narrative concepts; − The implementation of the API via an XML Schema.

While the open architecture and its API have been tested internally and with partners, one big challenge of such an initiative is its adoption by a wider community. The architecture will become all the more valuable that several and diverse compatible modules are made available. The next step will consist in “advertising” the API, initiating collaborative projects based on it and creating an associated research and development community. At the same time, it is expected that the API will evolve and grow, in particular by adding new modules.

Acknowledgements

This work has been funded (in part) by the European Commission under grant agreement IRIS (FP7-ICT-231824) and the Swiss National Science Foundation (J. Dumas and N. Szilas, principal investigators). The authors wish to thank all IRIS members who participated directly or indirectly to this work.

References

1. Sgouros, N. M.: Dynamic, User-Centered Resolution in Interactive Stories. In: Pollack, M. (ed.) IJCAI'97 Proceedings of the 15th International Joint Conference on Artificial Intelligence. Morgan Kaufmann Publishers, San Francisco (1997)

2. Szilas, N.: IDtension: a narrative engine for Interactive Drama. In: Göbel et al. (eds.) TIDSE’03. Frauenhofer IRB Verlag (2003)

3. Prada, R., Machado, I. and Paiva, A.: TEATRIX: Virtual Environment for Story Creation. In: ITS 2000. Springer, Heidelberg (2000)

4. Young, R. M., Riedl, M. O., Branly, M., Jhala, A., Martin, R.J. Saretto, C. J.: An architecture for integrating plan-based behavior generation with interactive game environments. Journal of Game Development, 1, 1 (2004)

5. Mateas, M. and Stern, A.: Integrating Plot, Character and Natural Language Processing in the Interactive Drama Façade. In: Göbel et al. (eds.) TIDSE’03, 139--151. Frauenhofer IRB Verlag (2003)

6. Spierling, U., Weiß, S.A., Müller, W.: Towards Accessible Authoring Tools for Interactive Storytelling. In: Göbel, Malkewitz, Iurgel (eds.): TIDSE’06. LNCS, vol. 4326, pp. 169-180. Springer, Heidelberg (2006)

7. Cavazza, M., Pizzi, D., Charles, F., Vogt, T., André, E.: Emotional Input for Character-based Interactive Storytelling. In: AAMAS 2009. 313--320. Springer, Heidelberg (2009)

8. Aylett, R., Louchart, S., Dias, J., Paiva, A., Vala, M., Woods, S. & Hall, L.: Unscripted Narrative for affectively driven characters. IEEE Journal of Graphics and Animation. 26(3), 42—52 (2006)

9. Endraß, B., Boegler, M. , Bee, N. and André, E.: What Would You Do in their Shoes? Experiencing Different Perspectives in an Interactive Drama for Multiple Users. In: Iurgel, I., Zagalo, N., Petta, P. (eds.): ICIDS 2009. LNCS vol. 5915, pp. 258—268. Springer, Heidelberg (2009)

10. Interactive Storytelling Systems, http://iris.scm.tees.ac.uk/public-wiki/index.php/IS_Systems

11. Szilas, N. and Kavakli, M.: PastMaster@storytelling: a controlled interface for interactive drama. In: 11th international conference on Intelligent user interfaces (Sydney, Australia, January 29 - February 1, 2006), pp. 288--290. ACM, New-York (2006)

12. Mateas, M., and Stern, A: Natural Language Understanding in Façade: Surface-text Processing. In: Göbel et al. (eds.) TIDSE'04. LNCS, vol. 3105, pp 3—13. Springer, Heidelberg (2004).

13. Szilas, N.: BEcool: Towards an Author Friendly Behaviour Engine. In Cavazza, M., Donikian, S. (eds.) ICVS 2007. LNCS, vol. 4871, pp. 102—113. Springer, Heidelberg (2007)

14. Mateas, M., Stern, A.: A Behavior Language: Joint Action and Behavior Idioms. In: Prendinger, H., Ishizuka, M. (eds.): Life-like Characters: Tools, Affective Functions and Applications, 135--162. Springer, Heidelberg (2004)

15. Lino, C., Christie, M., Lamarche, F., Schofield, G. and Olivier, P.: A Real-time Cinematography System for Interactive 3D Environments. In: SCA'10. (2010).

16. Arnav Jhala, R. Michael Young: Comparing Effects of Different Cinematic Visualization Strategies on Viewer Comprehension. In: Iurgel, I., Zagalo, N., Petta, P. (eds.): ICIDS 2009. LNCS vol. 5915, pp. 26—37. Springer, Heidelberg (2009)

17. El-Nasr, M.S., Horswill, I.: Real-Time Lighting Design for Interactive Narrative. In Balet, O., Subsol, G., Torguet, P. (eds.) ICVS 2003. LNCS, vol. 2897, pp. 12—20. Springer, Heidelberg (2003)

18. Koenitz, H.: Towards a Theoretical Framework for Interactive Digital Narrative. In: Aylett, et al. (eds.): ICIDS 2010. LNCS, vol. 6432, pp. 176—185. Springer, Heidelberg (2010)

19. IRIS, http://iris.scm.tees.ac.uk/ 20. Mateas, M. , Sengers,P. Narrative Intelligence. In AAAI Fall Symposium on Narrative

Intelligence, Technical Report FS-99-01. AAAI Press, Menlo Park (1999) 21. Interactive Storytelling concepts, http://tecfalabs.unige.ch/topincs/ 22. Young, R.M., Riedl, M.O.: Towards an architecture for intelligent control of narrative in

interactive virtual worlds. In: IUI 2003, pp. 310—312. ACM, New York (2003) 23. Szilas, N., Barles, J., Kavakli, M.: An implementation of real-time 3D interactive drama.

Computers in Entertainment 5(1) (Jan 2007) 24. El-Nasr, M.S.: Interactive Narrative Architecture based on Filmmaking Theory. Int. J.

Intell. Games & Simulation 3(1), 29--36 (March 2004) 25. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable

Object-Oriented Software. Addison-Wesley (2000) 26. Oparis API, http://tecfa.unige.ch/~szilas/iris/oparis/oparis.xsd 27. Dumas, J. E., Szilas, N., Richle, U., Boggini, T. Interactive simulations to help teenagers

cope when a parent has a traumatic brain injury. Computers in Entertainment 8(2) (Dec. 2010)