Markup of Multimodal Emotion-Sensitive Corpora

Berardina Nadja de Carolis, Univ. BariMarc Schröder, DFKI

Our Background:Generation of ECA Behavior

Believable ECA: multimodality, affect, personality, human-like, etc.

Two approaches:Bari APML (Affective Presentation Markup Language)

NECA RRL (Rich Representation Language)

Overview

Markup Schema Definition What to Annotate

What is a Markup Language What is XML

Dealing with complexity of human communication

Labelling Tools

Markup Scheme Definition

What to annotate? (verbal, non-verbal behavior, emotions, context, …)

Units of consideration (utterances/ turns/ paragraphs/ time units/…)

Descriptive Scheme Categorical Dimensional Functional …

The Question of Coding

How to Share and Access Data Annotation?

Standardization Format Readable for Computers

(statistics, inference, …)

XML-based Markup Languages!

What is a Markup-Language

Dictionary Definition:

a set of symbols and rules for their use when doing a markup of a document nomenclature, terminology, language - a system of words used in a particular discipline;

Computer Domain Examples: HTML LATEX RTF …

But …• proprietary language• not generic• not standard• fixed syntax and semantics

<?XML!>

What is XML

XML (eXtensible Markup Language) is

a language for creating markup languages that describe data thus, a meta language used to define new markup languages (domain- or industry-specific). Simple, open standard, extensible, ready to use, …

XML has a lot of friends that makes computation easy (XSL, DOM, SAX, …) and interoperable (RDF, RDF Schema, OWL, …)

No No No!

I said turn left!!! <pitch level=“high”>

</pitch>

<emotion type=“reprimanding”>

</emotion>

Dealing with complexity of human communication

Communication at different non-exclusive abstraction levels

Mental state (Beliefs, Goals, Intentions, Affective State, …) Discourse (Communicative Goals, Focus, Relations) Meanings (Performative, Affect, Turn-Taking, Emphasis) Signals (Physical Communicative Expressions)

Defining Different Markup Layers

Separate levels of meaning into different layers Define a Markup Language for each layer Define Interaction between layers

Develop Easy-to-use tools

Labelling tools:Text-centered annotation: MMAXhttp://mmax.eml-research.de

Labelling tools:Video-centered annotation: ANVILhttp://www.dfki.de/~kipp/anvil

Feedback into Generation Systems

Annotated Corpora can be used for improving ECA systems:

extracting knowledge markup language for specifying ECA behaviors …

Overview of the session

Introduction to XML structure

Demos Use of APML in an ECA system XML markup in MARY Text-to-speech

Hands-on: The ANVIL tool and the resulting XML markup