![Page 1: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/1.jpg)
Markup of Multimodal Emotion-Sensitive Corpora
Berardina Nadja de Carolis, Univ. BariMarc Schröder, DFKI
![Page 2: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/2.jpg)
Our Background:Generation of ECA Behavior
Believable ECA: multimodality, affect, personality, human-like, etc.
Two approaches:Bari APML (Affective Presentation Markup Language)
NECA RRL (Rich Representation Language)
![Page 3: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/3.jpg)
Overview
Markup Schema Definition What to Annotate
What is a Markup Language What is XML
Dealing with complexity of human communication
Labelling Tools
![Page 4: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/4.jpg)
Markup Scheme Definition
What to annotate? (verbal, non-verbal behavior, emotions, context, …)
Units of consideration (utterances/ turns/ paragraphs/ time units/…)
Descriptive Scheme Categorical Dimensional Functional …
![Page 5: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/5.jpg)
The Question of Coding
How to Share and Access Data Annotation?
Standardization Format Readable for Computers
(statistics, inference, …)
XML-based Markup Languages!
![Page 6: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/6.jpg)
What is a Markup-Language
Dictionary Definition:
a set of symbols and rules for their use when doing a markup of a document nomenclature, terminology, language - a system of words used in a particular discipline;
Computer Domain Examples: HTML LATEX RTF …
But …• proprietary language• not generic• not standard• fixed syntax and semantics
<?XML!>
![Page 7: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/7.jpg)
What is XML
XML (eXtensible Markup Language) is
a language for creating markup languages that describe data thus, a meta language used to define new markup languages (domain- or industry-specific). Simple, open standard, extensible, ready to use, …
XML has a lot of friends that makes computation easy (XSL, DOM, SAX, …) and interoperable (RDF, RDF Schema, OWL, …)
No No No!
I said turn left!!! <pitch level=“high”>
</pitch>
<emotion type=“reprimanding”>
</emotion>
![Page 8: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/8.jpg)
Dealing with complexity of human communication
Communication at different non-exclusive abstraction levels
Mental state (Beliefs, Goals, Intentions, Affective State, …) Discourse (Communicative Goals, Focus, Relations) Meanings (Performative, Affect, Turn-Taking, Emphasis) Signals (Physical Communicative Expressions)
Defining Different Markup Layers
Separate levels of meaning into different layers Define a Markup Language for each layer Define Interaction between layers
Develop Easy-to-use tools
![Page 9: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/9.jpg)
Labelling tools:Text-centered annotation: MMAXhttp://mmax.eml-research.de
![Page 10: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/10.jpg)
Labelling tools:Video-centered annotation: ANVILhttp://www.dfki.de/~kipp/anvil
![Page 11: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/11.jpg)
Feedback into Generation Systems
Annotated Corpora can be used for improving ECA systems:
extracting knowledge markup language for specifying ECA behaviors …
![Page 12: Markup of Multimodal Emotion-Sensitive Corpora Berardina Nadja de Carolis, Univ. Bari Marc Schröder, DFKI](https://reader036.vdocuments.mx/reader036/viewer/2022082407/56649e735503460f94b72606/html5/thumbnails/12.jpg)
Overview of the session
Introduction to XML structure
Demos Use of APML in an ECA system XML markup in MARY Text-to-speech
Hands-on: The ANVIL tool and the resulting XML markup