Enhancing Event-Based Semantics in the
Ontology of Picture Books 2
Karen Ang De La Salle University
2401 Taft Avenue, 1004 Manila, Philippines
Ethel Ong De La Salle University
2401 Taft Avenue, 1004 Manila, Philippines
ABSTRACT
Picture Books 2 generates fable-like stories for children age 6 to
8 years old based on an input picture comprising of three scenes.
Currently the system is able to generate stories that show
transitions and movement of characters and objects across the
adjacent scenes of the input picture. However some of the stories
produced are illogical. This study explores an approach to
enhancing the ontology structure and the reasoning engine of the
system to produce stories that follow a logical flow.
Keywords
Picture Books 2, Story Generation, Ontology, Reasoning Engine,
World State
1. INTRODUCTION
1.1 Storytelling for Children Many people use storytelling for various reasons, such as sharing
their beliefs, culture, and experiences. Also, storytelling has been
widely recognized to develop a child’s language and literary
skills. It plays a vital role in shaping a child's creativity skills.
Children get a chance to practice their imaginations and also
relate the stories to their daily lives.
During the early years of children, they use drawings to share
their stories. Storybooks which contain pictures or picture books
are usually the first contact of a child with the world. The
combination of art and words is a powerful experience for them.
This triggers their imagination and introduces concepts for their
cognitive and language development [6]. This idea led to the
development of a story generation system, Picture Books [7].
1.2 Motivation Picture Books 1 [7] takes a single picture as input and generates
a story based on the elements (background, characters and
objects) in the picture story. However, a single picture may not
capture the intended story that has been conceptualized by the
child when he/she drew the picture. This motivated the
development of Picture Books 2 [1], which takes an input picture
containing a sequence of multiple scenes. The system generates
fable-like stories that also feature common animals possessing
traits as the main characters of the generated story. These
character traits comprise the main factor in determining the
theme for the story. Furthermore, the system relies on the causal
chain of events that may occur in the given scenes to generate a
story. The chain of events is defined in the storytelling ontology
which was manually populated with activities and concepts that
are familiar to the target age group.
Validation made by linguists and storytellers on the resulting
stories have led to the conclusion that the design of the ontology
is insufficient to support the reasoning engine of the story
planner. It lacks certain attributes which can contribute to the
logical correlation of events to produce a coherent story.
Although the ontology design is inadequate, it can still be used to
causally connect events. Hence, in this study, the proposed
design of the ontology will still utilize the previous structure
adapted from ConceptNet [8] while appending other attributes,
specifically conditions and parameters needed for an action or
event to happen in the story world.
1.3 World State Picture Books 2 reasons through the ontology by retrieving the
causal chain of events between the initial state (e.g., the main
character arrives at the camp) and the target goal of the story
(e.g., the character learns to be brave). Due to solely relying on a
causal chain of events, the scoring function used to select an
event from a set of candidate events is random. This simplistic
approach in story planning combined with a set of binary
relations defining events in the ontology did not consistently
produce a chain of events that will lead to a coherent story to be
generated. Oftentimes, it is necessary to consider the world state
to constrain the possible events to be included in the story flow.
Tracking the world state as actions and events occur in the story
is currently not supported in Picture Books 2. A world state
contains knowledge about the state of the different characters and
objects in the story and are updated every time a character
executes an action that may change not only his/her state, but
that of the other characters and objects present in the story world.
In [9], this represents the first level of story representation - the
semantic networks for story world knowledge. The first level
relies on the general ontology of semantic network describing the
knowledge about the concepts and possible relations that exist
between them. World states or story world description [9] is an
instantiation of some concepts in the general ontology which is
concerned with the current properties and relationships of a
concept in a certain point of time.
Although the ontology of Picture Books 2 includes properties and
relationships between concepts, no world state has been utilized
and updated as the story progresses. Instead it assumed static
states, states that do not change over time. Because of this, the
scoring function purely relied on a random selection of
81 Proceedings of the 8th National Natural Language Processing Research Symposium, pages 81-84
De La Salle University, Manila, 24-25 November 2011
subsequent events that may not be applicable to the character's or
object's current state, leading to illogical stories.
A logical story is defined to be a story that exhibits consistency
and coherency. Both terms are adapted from Uijling’s [13]
definition of consistency and coherency in a story. In a consistent
story, the character’s actions are believable because the story
planner process of identifying the candidate actions that a
character may perform at a given point in the story is guided by
its knowledge of the current state of the story world. A coherent
story means that the plot has a logical flow of events.
Consider the story below generated by Picture Books 2 [3]. In
line #3, the object marshmallow has been introduced. However,
as the story progresses, the marshmallow was never mentioned
again; it played no part towards the development of the theme
where Danny the dog has to learn to become brave. Instead,
another object, the flashlight, introduced in line #8, was the key
object that led Danny to search for the cause of his fear and to
attain a change in his trait – that of becoming braver.
#1> It was a fine evening.
#2> Danny the dog was in the camp for a trip.
#3> He buys a packed marshmallow.
#4> The camp is very big.
…
#5> He sees a shadow.
#6> Danny the dog feels scared.
#7> He does not know what to do.
#8> He turned on a flashlight.
#9> Danny the dog searched the shadow.
…
#10> Since then, Danny the dog learned to be brave.
The main question that this research seeks to address is to
determine the types of knowledge to be added to the current
ontology that can be used by a reasoning engine to assist the
story planner in coming up with stories that are logical. The rest
of this paper is organized as follows. Section 2 presents the
proposed ontology structure, where the various types of
knowledge needed by a story planner are discussed. Section 3
presents the initial design of the system, as well as the
interaction among the character agent, plot agent and world
agent. The idea for these three agents was taken from the Virtual
Storyteller [12].
2. ONTOLOGY STRUCTURE The main source of knowledge for the generation of the fabula or
story plan in Picture Books 2 comes from three main categories.
These are the narratological knowledge, domain knowledge, and
character frame.
2.1 Narratological Knowledge The narratological knowledge represents concepts on narrative
structures as defined in [4]. It drives the flow of the story.
Themes and author goals are stored in this knowledge. Author
goals represent the premise for the story and include the conflict,
counter action for the conflict, and resolution for each theme.
2.2 Domain Knowledge The semantic description about concepts, objects, and their
relation are contained under the domain knowledge [4]. The
domain knowledge is the static storytelling knowledge
comprising mainly of the concepts and their relations that are
familiar to children, as well as fabula elements, links,
parameters, conditions, and primitives.
Concepts are connected via relations. Adapted from ConceptNet
[8], these relations include isA, receivesAction, usedFor,
hasProperty, and locatedAt relations.
Moreover, these relations aid the story planner generate the
fabula, which is defined as a chronological sequence of narrated
events for the story. The structure of the fabula is adapted from
[11] on the fabula model for story generation, which is comprised
of the fabula elements and links respectively.
The fabula elements define the story elements which comprise
the happenings in the story. These utilize the concepts,
parameters, primitives, and conditions, to define a single fabula
element. Table 1 shows a sample fabula element.
Table 1. Sample Fabula Element
Fabula Element Action
Concept go
Parameter agent, direction
Primitive PTRANS
[precond: agent ISA character &
direction ISA place &
!(agent LOCATEDAT direction)]
[postcond:
agent LOCATEDAT direction]
Pre-condition null
Post-condition null
On the other hand, the links causally connect these elements
together. These elements together with the links form an implicit
hierarchical structure, which form a story. Figure 1 shows a
sample connection of two fabula elements.
Figure 1. Sample Link
The parameters define the roles required for an action to be
executed, such as the agent and the receiver of the action. In
contrast, conditions define the pre-requisites for an event or
action to take place and its consequential results.
Lastly, the primitives consist of the most basic actions and their
corresponding conditions. These primitives are based from the
theory of Roger Schank on conceptual dependency [10]. Table 2
shows a sample set of primitives.
Table 2. Primitives as defined in Schank [10]
Primitive Description
Atrans Transfer of possession
Ptrans Transfer of physical location
Grasp Agent picks up an object/drops an object
Ingest An agent eats an object
82
2.3 Character Frame The character frame contains description of characters in the
domain. It is where all the information about a character is
stored. This includes the sub-characters, personality, desires, and
roles.
Sub-characters contain all possible sub-characters that may
appear in a story. Table 3 illustrates the sample information
stored for sub-characters. Table 3. Sample Sub-character
Name Gender Type
Peggy female pig
Personality, on the other hand, contains the positive and negative
traits of each character. Also as the name implies, desires define
the desires of characters. For example, children like sweets.
Lastly, roles specify the relationship of a character to another. It
also defines the initial emotion of a character to another. Thus,
this emotion is represented dynamically; as the story progresses,
the emotion felt by a character towards another may change. For
instance, Peggy is the sister of Peter and it is defined that Peter
loves Peggy and vice versa. Over the course of the story, Peter
may do something that may cause Peggy to hate Peter. As such,
Peggy’s emotion toward Peter will change from love to hate.
3. ARCHITECTURAL DESIGN From the multi-scene input picture specified through the Story
Editor of Picture Books 2, a corresponding abstract scene
representation [2] is fed to the story planner to generate the
fabula. The fabula is then forwarded to the Sentence Planner of
Picture Books 2 for lexicalization and aggregation of character
goals, which are then transformed into English sentences during
the realization phase to produce the actual story text.
The Story Planner, which is the main focus of this paper, is
responsible for generating the contents of the story (the fabula or
story plan). Figure 2 illustrates the interaction among the three
agents in the Story Planner.
Figure 2. Architectural Design for the Story Planner
3.1 Plot Agent The plot agent serves as the director of the story planner. It
supervises the selection of the story events. The plot agent also
selects the theme for the story and ensures the coherency of the
generated fabula, which must lead to the selected theme of the
story. In order to accomplish these purposes, it has both the
premise generator and the fabula generator.
The concept of a premise is adapted from Egri [5], which defines
it as a guiding force to ensure that the story will not lead to a
meaningless direction. Hence, Egri proposed a premise to be
comprised of three essential parts, which are the character,
conflict, and conclusion. The character is not only described as
the character itself but also the environment in which the
character belongs. The conflict, on the other hand, describes the
primary conflict which holds the action and counter-action that it
evokes; while the conclusion is the final state of the world. These
parts are represented as the setting or the initial state of the
world, the action and reaction pair that describes the primary
conflict, and the final state of the world, respectively.
The theme selector selects the story’s theme, which defines the
conflict and conclusion of the story. The selection is based on
the negative traits of the main character and the number of
matched conditions between the conflict and resolution to the
input scenes. The highest scoring theme will then be sent to the
setting formulator for it to formulate a setting according to the
selected theme. When the setting for the story is successfully
generated, the fabula or story plan for the selected theme and
setting will then be generated.
Since stories revolve around character actions, the fabula
generator would need the help of the character agent. The
character agent approach provides a means for each character in
the story world to decide on what it can do to accomplish its
goals. It generates a set of plan steps to achieve its current goal.
In order to verify the consistency of these plan steps with the
selected theme, the fabula generator will deliberate on these plan
steps.
The deliberation process considers which plan of action is to be
executed. The plot agent can also manipulate the generated
fabula by inserting events which would affect the character thus
influencing its next plan of action. These events are defined as
the unintentional change in the world state by a character.
Generating events that allow the system to force the story to go
with the given theme is handled by the events generator.
Together, they make up the fabula generator which works with
the character agent to produce a story that corresponds to the
selected theme and given input scenes by the user.
If the plot agent cannot produce a fabula for the selected theme,
it reiterates the process of formulating another setting using the
next candidate theme. The next candidate theme is the next
highest scoring theme. This is determined by comparing the
requirements of the theme with the given input scenes. The more
similarities the theme possesses with the given scenes, the more
likely that that theme is possible to happen.
3.2 Character Agent As mentioned earlier, the character agent decides which actions a
character can take to accomplish its goals. It represents the goals
and plans of a character in the story. Characters have goals that
serve as the basis for their actions. These goals are formed by
their existing goals, personality, needs, and desires.
Thus, the character agent renders the set of candidate goals of a
character given its existing goals, personality, desires, and
current world state. These attributes are accessible to the
character agent through the plot agent, which retrieves these
83
from the world agent. These also provide the character a goal to
pursue.
To achieve its goal, a character has to perform a set of actions.
These actions are realized by generating the next action steps
based on its current goal or previous event that happened. It
filters the possible actions to select the candidate set of actions
that are relevant to the accomplishment of its goal. It also verifies
whether the given input scenes fit the candidate actions. These
set of actions are not automatically executed but are passed onto
the fabula generator, which will decide if the plan of action is to
be executed.
3.3 World Agent Lastly, a world is described by the characters and objects present
and also their properties. Thus, a world state is defined as the
state of the world in a particular point in time. It contains
information about the characters and objects present in the
world. Also, these world states change by appending, deleting or
modifying properties of either the characters or objects.
The world agent holds this world state in each timestep, which is
defined as a point in time that a particular event occurs. It is also
responsible for updating the world state in each timestep. This is
done after the plot agent decides to execute an action or event.
After an action is performed, the plot agent utilizes this updated
world state to verify if the next action can be executed given the
current world state. This process repeats until the system
generates the whole fabula for the story.
4. ONGOING WORK Though Picture Books 2 is able to generate stories, it is
important to consider the quality of the output which may affect
the child using the system. The redesign of the ontology structure
and reasoning engine of the system aims to enhance the story
planner of the system to be able to select a more logical story
path.
The ontology structure plays an important role in story
generation. It is the source from which the system generates the
contents of the story. The bigger the amount of relations
available aids the system to generate more flexible stories.
Hence, the ontology structure should be designed efficiently so as
to reduce the needed information to be inputted. Furthermore,
the right blend of the character agent and plot agent must be
achieved to balance the believability of a character and also the
occurring events which guide the story to its theme.
Issues that must be addressed throughout the development of the
system include representing when a character perceives an event
or action of another character. What would be the basis of the
system to decide that a character must act on the changes in its
environment? Also, given a list of goals of a character, how will
the character agent decide on which goal to pursue first?
5. ACKNOWLEDGMENTS We would like to acknowledge the Department of Science and
Technology – Engineering Research and Development for
Technology for their support in the achievement of this project.
6. REFERENCES [1] Ang, K., Antonio, J., Sanchez, D., and Yu, S. 2010. Picture
Books 2. Undergraduate Thesis, De La Salle University,
Manila, Philippines.
[2] Ang, K., Antonio, J., Sanchez, D., Yu, S., and Ong, E. 2010.
Generating Stories for a Multi-Scene Input Picture. In
Proceedings of the 7th National Natural Language
Processing Research Symposium, 21-26, November 19-20
2010, De La Salle University, Manila, Philippines.
[3] Ang, K., Yu, S., and Ong, E. 2011. Theme-Based Cause-
Effect Planning for Multiple-Scene Story Generation. In
Proceedings of the International Conference on
Computational Creativity, 48-53, April 27-29 2011, Mexico
City, Mexico.
[4] Cua, J., Ong, E., and Pease, A. 2011. Using SUMO to
Represent Storytelling Knowledge. Philippine Computing
Journal, 5(2):37-43, December 2010, Computing Society of
the Philippines.
[5] Egri, L. 2007. The Art of Dramatic Writing. Wildside Press.
[6] Hillman, J. 2003. Discovering Children’s Literature, Third
Edition. Columbus, OH: Merrill Prentice Hall.
[7] Hong, A., Siy, J., Solis, C., and Tabirao, E. 2008. Picture
Books: An Automated Story Generator. Undergraduate
Thesis, De La Salle University, Manila, Philippines.
[8] Liu, H., and Singh, P. 2004. Conceptnet - A Practical
Commonsense Reasoning Tool-kit. BT Technology Journal,
22 (4), 211–226.
[9] Oinonen, K., Theune, M., Nijholt, A., and Uijlings, J. 2006.
Designing a Story Database for Use in Automatic Story
Generation. In Proceedings of the 5th International
Conference on Entertainment Computing, 298-301,
Cambridge, UK.
[10] Schank, R. C. 1972. Conceptual Dependency: {A} Theory of
Natural Language Understanding. Cognitive Psychology,
(3)4, 532-631.
[11] Swartjes, I., and Theune, M. 2006. A Fabula Model for
Emergent Narrative. In Proceedings of the Third
International Conference on Technologies for Interactive
Digital Storytelling and Entertainment, (Darmstadt,
Germany, December 04 - 06, 2006).
[12] Theune, M., Faas, S., Nijholt, A., and Heylen, D. 2002. The
Virtual Storyteller. IEEE, 23(2), 20–21.
[13] Uijlings, J.R.R. 2006. Designing the Virtual Environment
for Story Generation. Master’s Thesis, University of
Amsterdam, Amsterdam, Netherlands.
84