1
CASPAR Contemporary Arts Testbed:CASPAR Contemporary Arts Testbed:Preservation of
Interactive Multimedia Performances
Kia Ng and Eleni Mikroyannidi
ICSRiM - University of [email protected] www.icsrim.org.uk/caspar
Overview
• Context and Motivation– CASPAR EC IST Project Contemporary Arts Testbed– Interactive Multimedia Performance (IMP)
• Music via Motion (MvM)
• i-Maestro 3D Augmented Mirror (AMIR) System
• ICSRiM Conducting Interface
• Research and Development– IMP dataset– CIDOC-CRM and FRBR ontologies
• Result and Conclusion– ICSRiM IMP Archival System
• Interface and functionalities
• Scenarios
• Validation
Context
• CASPAR EC IST Project–Build a framework to support end-to-end preservation
lifecycle for scientific, artistic and cultural information
• Contemporary performing arts testbed–Interactive Multimedia Performance (IMP)
• Interactive Multimedia–Interactive multimedia performances
–Performance capture/recording and analysis beyond normal AV
Music via Motion (MvM)
• MvM: a framework to integrate interactive multimedia, motion sensing, audio synthesis, VR and AR technologies
– To explore virtual and augmented instruments
– To provide users/performers with real-time control of multimedia events with their physical movements
• Application areas include stage performance and installation arts
IMP Elements and Processes
Example of an IMP production processbased on the Music via Motion (MvM) system, www.kcng.org/mvm
Data Analysis, Classification
and Recognition
Mapping
Multimedia Generation
GUI (for monitor & control)
Data Capture & Processing
Motions
multimodal data
Multimedia output
Mapping Strategy Parameters
i-Maestro 3D Augmented Mirror (AMIR)
• Technology-enhanced learning
• More in-depth recording and understanding
– playing gesture– Intangible heritage– Individual performance
style
• Data stream include: audio, video, 3D motion data, sensor data, analysis
• Multimodal feedbacks with graphical visualisation and sonification
• Max/MSP jitter, c/c++, Lua, SDIF
Excerpt from Bach's Partita No. 3 in E major. BMW 1006 (Preludio)
AMIR (www.i-maestro.org)
ICSRiM Conducting Interface
• A system for tracking and analysing a conductors hand movements– Uses multiple wiimotes to
capture a conductors movements
– These movements are analysed
– This information is then fed back to the user in an entertaining yet educational manner
System Setup
GUI
• 3D environment• Intuitive UI• Multiple modes• Multimodal data capture and
analysis
Complexity of the IMP Preservation
• IMP rely heavily on digital media• A recording (e.g. video) of a performance is not
sufficient for re-performance• Preserving individual components of a
performance is clearly a challenge• However, putting these components together in
one place does not make a performance• They need to be assembled in a logical and
temporal order with their inter-relationships• Preserving this knowledge through time is even
more challenging
• Preservation for– Re-performance at a later time– Historical study/analysis in performing arts in the
future– Better preparation for the future needs
Example IMP interface from the i-Maestro EC IST Project involving 3D motion data, sonification, and graphical feedbacks
Example IMP - from a Music via Motion (MvM) performance involving motion generated music with dance
12
IMP Dataset
• An Interactive Multimedia Performance (IMP) work typically consists of the following digital objects: – Audio– Video– 2D/3D motion data– Sensor data– Max/MSP patches– Configuration files– and others
• For reconstruction of an IMP work– Representation Information– Inter-relationship
• Logical• Temporal
CIDOC-CRM for IMPs
• The CIDOC Conceptual Reference Model (CIDOC-CRM) has been proposed as a standard ontology for enabling interoperability among digital libraries
• It defines a set concepts for physical and temporal entities
Conceptual Objects
Physical Objects
Types
Temporal Entities
Ap
pel
lati
ons
Actors
PlacesTime Spans
are referred/ refine
participate in
within at
affect/ refer to
has location
refer to/ identify
CIDOC-CRM for IMPs (2)
• CIDOC-CRM can be used to describe a performance– at a high level
• More specialised vocabularies are necessary for IMPs– e.g. how different types of equipment are connected
together in the performance.
• CIDOC-CRM – lacks concepts for digital objects
– is designed primarily for documentation of what has happened, whereas in digital preservation, it is also required to document the reconstruction of a past event from preserved components.
CIDOC-CRM FRBR Extension
• Two main objectives in extending CIDOC-CRM for the preservation of an IMP:– To provide a domain specific vocabulary for describing an IMP.
• e.g. details on how the archived performance was carried out and how it can be re-performed
– To provide a vocabulary for describing digital objects, their interrelationships and operations performed on them in the digital preservation context
• FRBR (Functional Requirements for Bibliographic Records) – describes a high level conceptual model of creative works and how
they are represented in the real world
• It is used as extension ontology of the CIDOC-CRM – for describing the conception process of the performance
15
16
Representation Information
• The Representation Information for this performance is in the form of CIDOC-CRM and FRBR instantiation.
F52.PerformanceIMP
F8.PersonKia (Director)Frank (Performer)
E53.PlaceLeeds - UK
E52.Time-Span2hours:5PM-12/02/07
E22.Man-Made ObjectCelloSound MixerComputer System
E73.Information ObjectMusicMusic Score
P16F – use specific object
P16F – use specific object
P7F – took place atP4F – has time span
P14F – carried out by
17
Mapping IMP detailsto CIDOC-CRM and FRBR
• The FRBR concept “F52.Performance” is used to model an IMP.
• The FRBR concept “F8.Person” represents the people involved in the performance, e.g. directors, performers, MAX/MSP programmers, etc.
• The digital objects used (software patches, 3D models, etc) are instances of the “E73.Information Object” CIDOC-CRM class, whilst any physical objects (instruments, computer equipment, etc) are instances of the “E22.Man-Made Object” class.
• The “E52.Time-Span” and “E53.Place” classes are used to represent the time-span and location of the performance respectively.
18
Mapping IMP detailsto CIDOC-CRM and FRBR (contd.)
ICSRiM IMP Archival System
• The ICSRiM IMP Archival System – for the preservation of different IMPs– based on the CASPAR Framework – Integration of the selected CASPAR components
• Communication with the Repository containing the Knowledge base and the DataStore
ArchivalSystem
Knowledge Base
Repository DataStore
communicatesRepInfo
PerformanceFilesCASPAR
Web Services
calls
Web Interface of the IMP Archival System
Cyclops tool
• A tool to capture appropriate Representation Information to enhance Virtualisation and future re-use of the digital object.
• Cyclops provides a usable interface to describe the Performances in the form of graphs
• The graph in the backend is saved as an RDF instantiation of the CIDOC and FRBR ontologies
• Defined terminologies according to various data types
Cyclops Interface
Ingestion and Retrieval of an IMP
• Scenario 1: Ingestion of an IMP– Description of the performance with the use of
Cyclops• Generation of the Representation Information
– Both the metadata and the digital files are ingested and stored in the Repository
• Scenario 2: Access of an IMP– Queries are performed on the metadata – The related objects are returned to the user– Instructions extracted from the RepInfo provide
details on how to assemble the objects
Scenario 1: Ingestion of an IMP
Steps in Screenshots
24
Step 1: Describe the IMP
• Follow the Ingest wizard to complete the ingestion
25
Step 1: Describe the IMP (2)• Create Representation Information with the use of Cyclops tool• It is saved in RDF format. In this way, the graph was transformed into
an RDF instantiation of the CIDOC-CRM and FRBR ontologies.
26
Step 2: RepInfo parsing
• The system detects and parses the RepInfo for defined data types
27
Step 3: Addition of the digital files
• Upload the related digital files of the performance• The files are linked to the RepInfo• The package is stored in the Repository
28
Scenario 2: Retrieval of an IMP
Steps in Screenshots
29
Step1: Search for the IMP
• The search is done by the name of the performance
30
Step1: Search for the IMP (2)
• The system returns a table with the following selected information from the RepInfo– The name of the Performance– Details on the performance– The associated with the performance files– The links to download the files of the performance– Details on the files (e.g. which one is the data file,
the patch file, how to use them etc.)– A link to the RDF description.
31
Step 2: Download Associated Performance files
• Select and download the components of the performance
32
Step 3: Reconstruct the performance
• The user has to understand the connection of the components
• Instructions on the connection is provided in the table• By assembling the components the user is able to see the
performance
33
Retrieval of an IMP
Validation and Feedback• Participant
– A small set of experts who work with IMP
• Validation– Ingestion of an IMP work (their own work)
– Retrieval and reconstruction of an IMP they have no information about it
• Download and connect the components using the instructions
– Fill in a questionnaire with their feedback
• Results– The system is useful to store and preserve the IMPs
– The system provided all the needed instructions for the reconstruction of an IMP
• The appropriate RepInfo has been captured
– Some difficulty in using Cyclops (complexity of the graphs)
• Demo videos and tutorials helped
• Cyclops graph templates to help to get started
CASPAR IMP References
• Selected publications include:– E. Mikroyannidi, B. Ong, D. Giaretta and K. Ng, “Preservation of
Interactive Multimedia Performance with the use of Ontology Models”, in Proc. of the Digital Resources for the Humanities & Arts (DRHA) Conference, Belfast, September 2009
– K. Ng, E. Mikroyannidi, B. Ong, N. Esposito and D. Giaretta, "Interactive Multimedia Systems for Technology-Enhanced Learning and Preservation" in Proc. Of the 15th International Conference on Distributed Multimedia Systems (DMS) 2009, San Francisco, September 2009.
– K. Ng, T.V. Pham, B. Ong, A. Mikroyannidis, and D. Giaretta, "Preservation of Interactive Multimedia Performances", International Journal of Metadata, Semantics and Ontologies, 3(3), 28 February 2009 , pp. 183-196, Inderscience Publishers.
• For latest publications on CASPAR IMP Preservation, see– www.icsrim.org.uk/caspar– CASPAR project website
36
Conclusions
• Preserving knowledge is vital for future reconstruction of interactive multimedia performances
• Proposed IMP preservation process– Made use of standard ontology models, CIDOC-CRM and
FRBR to create Representation Information• Used to define interrelationships between the components
• Creation of a web-based archival system for the preservation of IMPs– Integrated selected components of the CASPAR
Framework
– Current prototype online:• http://www.icsrim.org.uk/imp-preserve
37
Thank you Kia Ng and Eleni Mikroyannidi
ICSRiM – University of Leeds
www.icsrim.org.uk/caspar [email protected]
38