are sensory effects ready for the world wide web?
TRANSCRIPT
Are Sensory EffectsReady for the World Wide Web?
Christian Timmerer, Markus Waltl, and Hermann Hellwagner
Klagenfurt University (UNIKLU) Faculty of Technical Sciences (TEWI)
Department of Information Technology (ITEC) Multimedia Communication (MMC)
http://research.timmerer.com http://blog.timmerer.com mailto:[email protected]
20 May 2010
Acknowledgments. This work was supported in part by the European Commission in the context of the NoEINTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), and the ALICANTE project (FP7-ICT-248652).
Outline
• Background / Introduction
– MPEG-V Media Context and Control– Concept of MPEG-V Sensory Information– Sensory Effect Description Language (SEDL)– Software/Hardware components: SEVino, SESim, SEMP, and
amBX+SDK
• Are Sensory Effects Ready for the World Wide Web?
– How to embed SEM into Web documents?– How to interpret SEM within a browser?– How to synchronize with audio/video content of the Web document?– How to extract color information automatically?
• Conclusions and Future Work
2010/05/20 2Christian Timmerer, Klagenfurt University, Austria
MPEG-V: Media Context and Control (MCC)
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 3
System Architecture
Pt. 1: Architecture
Pt. 3: Sensory Information
Pt. 4: Virtual World Object Characteristics
Pt. 2: Control Information
Pt. 6: Common Types and Tools
Pt. 7: Conformance and Reference Software
Pt. 5: Data Formats for Interaction Devices
http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V
Concept of MPEG-V Sensory Information
• Consumption of multimedia content may stimulate also other senses– Vision or audition– Olfaction, mechanoreception, equilibrioception, thermoception, …
• Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 4
… giving her/him the sensation of being part of the particular media➪ worthwhile, informative user experience
Sensory Effect Description Language (SEDL)
• XML Schema-based language for describing sensory effects– Basic building blocks to describe, e.g., light, wind, fog, vibration, scent– MPEG-V Part 3, Sensory Information– Adopted MPEG-21 DIA tools for adding time information (synchronization)
• Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)– Extensibility: additional effects can be added easily w/o affecting SEDL– Flexibility: each application domain may define its own sensory effects
• Description conforming to SEDL :== Sensory Effect Metadata (SEM)– May be associated to any kind of multimedia content (e.g., movies, music,
Web sites, games)– Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate
mediation device
➪ Increase the experience of the user➪ Worthwhile, informative user experience
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 5
Sensory Effect Description Language (cont’d)
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 6
SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|
Effect|ReferenceEffect)+
Declarations ::= (GroupOfEffects|Effect|Parameter)+
GroupOfEffects ::= timestamp EffectDefinition
EffectDefinition (EffectDefinition)*
Effect ::= timestamp EffectDefinition
EffectDefinition ::= [activate][duration][fade][alt]
[priority][intensity][position]
[adaptability]
SEVino, SESim, SEMP, and amBX
amBX (Ambient Experience) system + SDK• Two fan devices, a wrist rumbler, two sound
speakers, a subwoofer, two lights, and a wall washer
• Everything controlled by SEM descriptionsexcept light effect
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 7
Annotation Tool: SEVino Simulator: SESimPlayer: SEMP
Are Sensory Effects Ready for the World Wide Web?
• How to embed SEM into Web documents?
• How to interpret/parse SEM within a browser?– Directly via JavaScript
– Browser plug-in
• How to synchronize with audio/video content of the Web document?– HTML5’s video and audio elements and corresponding
DOM attributes
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 8
<link rel="alternate" type="text/xml”
href="http://server.org/sem.xml”
media="light, wind, vibration"/>
Are Sensory Effects Ready for the World Wide Web? (cont’d)
• Synchronization with audio/video (cont’d)– currentTimeDOM attribute: current playback
position in seconds– timeupdateevent: indicates that the current
playback position changed as part of normal playback
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 9
Are Sensory Effects Ready for the World Wide Web? (cont’d)
• How to extract color information automatically?
– Used for controlling ambient lights
– currentFramedoes neither exist nor is foreseen in HTML5
– canvas element allows for manipulating the display’s content
(+) Pro: extract the pixel information from the display
(-) Con: needs to be added explicitly during authoring
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 10
Conclusions
• Introduction to MPEG-V Sensory Effects
• Are Sensory Effects Ready for the World Wide Web? – Yes, they are!
• Future work
– Implementation is ongoing
– Subjective quality assessments
2010/05/20 Christian Timmerer, Klagenfurt University, Austria 11
Thank you for your attention
... questions, comments, etc. are welcome …
Ass.-Prof. Dipl.-Ing. Dr. Christian TimmererKlagenfurt University, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, [email protected]
http://research.timmerer.com/Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer
122010/05/20 Christian Timmerer, Klagenfurt University, Austria