mpeg v-awareness event

MPEG-V: a standard for multi-sensorial and immersive

experiences

Marius PREDA* Institut TELECOM / TELECOM SudParis

*and several tens of people that shared their images on Internet

From centuries, we are building two worlds

Physical Informa0onal stories

novels music

films

knowledge

Immersion, presence, stimuli, perception …

making abstraction of the Physical world (remove all possible connection to it) and have stimuli only from the Informational World

Immersion – a straightforward definition

In which world our users are?

The stimuli from the physical world are familiar or at very low intensities

We intentionally block our interfaces with the physical world

Improving immersion feeling

Strategies for blocking the interfaces with the Physical world


… but also Strategies for extending the space of the Informational world stimuli


How far are we advanced with this strategy?

Quite far …



How far are we advanced with this strategy?

Probably too far …

What is the secret of a good immersion in the Informational world?

Is the quality of the stimuli? Not only.

A guess: The “scenarisation” of the experience, or, in MPEG terms “authoring the content”

We should have methods and tools to express this new type of complex content

Multi-sensorial content A natural extension of the more traditional audio video content

This is not new, but it is now the right time to bring it in home environments, therefore interoperability is an issue

MPEG-V is the solution by offering a rich set of tools for representing multi-sensorial content

Covered mainly by Part 2, 3 & 5

Immersion by means of 360° view

Dimension discontinuity problem

2D 3D

Research by kzero.org

- New technologies for creating, representing and visualizing 3D content are now available

- Fast development of high performing 3D graphics cards

- Connecting real and virtual economy

- Increased users demands for rich communication channels

3D Virtual Worlds, a promising trend in the 3DGFX space

Towards 3D content

MPEG was already prepared to this take-off by defining means for GFX representation

Container for personal data, personality, skills, …

Virtual World assets Generic Virtual Objects Avatars

Communication support between users

Interaction support between the user and the virtual environment

MPEG-V

Sharing content between Virtual Worlds becomes possible with MPEG-V

Covered mainly by Part 4

Closed VW vs Open VW

A standardisation effort initiated in 2007 and driven by two complementary forces – multi-sensorial experience and virtual worlds – with the goal of offering a solid technical ground for immersive, multi-dimensional, multimedia applications and services

MPEG-V

Promoted as ISO/IEC 23005 in January 2011

Why the appropriate place for MPEG-V is in MPEG? Answer by Dr. Leonardo Chiariglione

How MPEG is dealing with the interoperability of immersive experience? Answer by Eng. Jean Gelissen (Part 1), Pr. Kyoungro Yoon (Part 2&5), Dr. Christian Timmerer (Part 3) and Dr. Jae Joon Han (Part 4)

How MPEG-V connects to MPEG-4 3D Graphics? Answer by Dr. Minsu Ahn (invited speaker)

What the future reserves us related to Virtual Worlds? Answer by Dr. Yesha Sivan (invited speaker)

Why, How and other more or less philosophical questions

Encouragement Message

Leonardo Chiariglione MPEG Convener

MPEG-V Architecture & Use Case

Jean H.A. Gelissen Philips Research

Eindhoven, The Netherlands

MPEG-V Architecture and Use Cases

MPEG-‐V defines an architecture that provides interoperability for informa<on exchange with virtual worlds. This allows for the simultaneous reac<ons in both worlds to changes in the environment and human behavior. Key words are efficient, effec<ve, intui<ve and entertaining interfaces between both worlds taking the economics, rules and regula<ons into account.

•  Use cases have been crucial in the development of MPEG-V: –  For the requirements gathering, –  During design, development / implementation, –  Basis for validation (supported by reference software and conformance)

•  Representation of Sensory Effects (RoSE) •  Full motion control and navigation of avatar/object with multi-input sources •  Virtual Travel •  Serious gaming for Ambient Assisted Living •  Virtual Traces of Real Places •  Avatar Appearance •  Social presence •  Group Decision-making in the context of Spatial Planning •  Consumer Collaboration in Product Design Processes along the Supply Chain •  Virtual Objects •  Internet Haptic Service - YouTube, Online Chatting •  Next Generation Classroom – Sensation Book •  Immersive Broadcasting – Home Shopping, Fishing Channels •  Entertainment – Game (Second Life, Star Craft), Movie Theater •  Virtual Simulation for Training – Military Task, Medical training •  Motion Effects

Use Case driven Architecture

Architecture (after many iterations)

Architecture instantiation 1 M

appi

ng fo

r Exc

hang

es w

ithin

the

Rea

l Wor

lds

Use Case Example

Single Renderer

A/V

RoSE-‐enabled Mul<media Consump<on for Advanced User Experience RoS

E (R

epre

sent

atio

n of

Sen

sory

Effe

cts)


appi

ng fo

r (bi

dire

ctio

nal)

exch

ange

of

info

rmat

ion

betw

een

real

wor

ld a

nd v

irtua

l wor

ld

Use Cases example Fu

ll m

otio

n co

ntro

l and

nav

igat

ion

of

avat

ar/o

bjec

t with

mul

ti-in

put s

ourc

es

Use Cases example


appi

ng fo

r (bi

dire

ctio

nal)

exch

ange

of

info

rmat

ion

betw

een

virt

ual w

orld

s

Use Cases example


appi

ng fo

r the

con

trol

of a

vata

rs a

nd o

ther

vi

rtua

l obj

ects

by

real

wor

ld s

igna

ls

Use Cases example

Next Generation Classroom – Sensation Book

Using a haptic wristband for Internet browsing


appi

ng fo

r the

Con

trol

of o

bjec

ts b

y si

gnal

s fr

om th

e vi

rtua

l wor

ld

Use Cases example

Various motion chairs and their characteristics

•  Examples of new categories of sensors: –  Bio sensors –  Gas and Dust sensors –  Gaze Tracking sensors –  Attributed Coordinate sensors –  Multi Pointing sensors –  Wind sensors –  Navigation sensors

•  Examples of new Use Cases: –  Mental Health & Food Intake for Lifestyle Management –  Cardiovascular Rehabilitation for Health Management –  Glucose level / Diabetes management for Health Management –  Multipoint interaction devices –  Unmanned Aerial Vehicle Scenario –  Virtual tour guides –  Serious gaming & training (for Ambient Assisted Living) –  Motion tracking –  Facial Animation

Next Steps (Amendment 1)

Thank you for your attention …

Capturing and controlling the real world with MPEG-V

Kyoungro Yoon ([email protected]) School of Computer Science and

Engineering Konkuk University

Contents

•  MPEG-V Architecture •  Scope of Part 2 and Part 5 •  Example Scenarios •  Part 2 Control Information •  Part 5 Data Formats for Interaction Devices •  Conclusion & Discussions

Digital Content Provider (Virtual World, (serious) game, simulator, DVD, …)

Virtual World N

Real World Dev1

Real World Device N

S Sensor Actuator

MPEG-V Architecture

Real World Dev2

Real World Dev3

S A

Enriched content

User In

tera

ction

Adaptation VV

Adaptation VV

Adaptation VR

Adaptation RV/VR

Adaptation RV

A

Adaptation VR Adaptation RV

Areas A & B are targets of MPEG-V

standardization

Virtual World Data Representation V Adaptation VR/RV

Device Commands

Standardization Area B: Sensory Information

(Part 3, 4, …)

Virt

ual W

orld

Dat

a R

epre

sent

atio

n R

R

eal W

orld

Dat

a R

epre

sent

atio

n Standardization Area A: Control Information

(Part 2 & 5)

Two Parts to capture and control the real world

•  ISO/IEC 23005 Part 2 Control Information –  The capability descriptions of actuators (sensory

devices) and sensors in the real world –  The user’s sensory preference information (USEP),

which characterizes devices and users, so that appropriate information to control individual devices (actuators and sensors) for individual users can be generated

•  ISO/IEC 23005 Part 5 Data Formats for Interaction Devices –  Data formats for industry-ready interaction devices:

sensors and actuators

Two Parts to capture and control the real world

Sensor / Actuator

Sensed Information /

Sensory Device Commands

Scope of Part 5

User’s Sensory Preference (Part 2)

Sensory Device Capability (Part 2)

Sensor Device Capability (Part 2)

Virtual Worlds1

Adaptation RV / VR (non-normative)





Virtual Worlds2

Scope

Intention

•  Part 5 alone can provide functionality of capturing and controlling the real-world. –  Device Commands: Intensity provided in percentage with

respect to the maximum intensity that the specific device can generate

–  Sensed Information: Sensed value provided in specific unit of the each individual sensor

•  Part 2 helps to adapt the control to each individual user’s case. –  Device capability description with min/max intensity –  Sensor capability description with min/max value and

accuracy, etc. –  User’s Preferences on Sensory Effects

Example Scenarios

•  RoSE –  Sensory Effect Metadata, Sensory Device Capabilities, Sensory

Device Commands, and User Sensory Preferences are within the scope of standardization and, thus shall be normatively specified. On the other side, the RoSE Engine as well as Provider entities and Consumer Devices are informative and are left open for industry competition.

Virtual Flight

Des0na0on Posi0on RigidBodyMo0on

Mo0onSensor GlobalPosi0onSensor Al0tude Sensor

Bringing sensibility in a virtual world

Sensing in Real World Reflec<ng Environmental context to Virtual World

-‐  Sensed Informa<on (TimeStamp, id, sensorIdRef, linkedlist, groupID, ac0vate, priority) -‐  Sensor Capability (Accuracy, unit, maxValue, minValue, offset, numOfLevels, sensi0vity, SNR)

-‐ Measure Temperature, Humidity, Light, Atmospheric pressure, and etc., in the real world.

"   Environment Sensors -‐ The real-‐0me environmental data of the real world is applied to reflect on the virtual environment. -‐ Light, ambient noise, temperature, humidity, distance, atmospheric pressure, force, torque, pressure sensors.

-‐ Adapt the 0me series sensor data seman0cally to the virtual world based on the sensor capabili0es and adapta0on preferences.

-‐ Brightness changes

-‐ Turn on the light -‐ Change day (night) to night (day)

Courtesy of Samsung

Do what I do

Virtual Naviga0on Virtual Sports

Virtual Games Virtual Concert

"   Mo<on Sensor

-‐ detects/tracks Real 0me 3D mo0on informa0on of a control device

-‐ Posi0on, Velocity, Accelera0on, Orienta0on, Angular velocity, Angular accelera0on sensors "   Can be applicable to mo<on sensor based games.

Courtesy of Samsung

Part 5: Data Formats for Interaction Devices

•  Provide simple interface to the physical world (real world)

Adaptation RV/VR Engine

(non-normative)

Sensory Information

(Part 3)

Sensory Device Capability

Sensor Capability User’s Sensory Preference

Device Commands

Sensed Information

Within Scope of Part 5 Virtual World Object Char.

(Part 4)


•  Interaction Information Description Language (IIDL) •  Device Command Vocabulary •  Sensed Information Vocabulary


  Light   Flash   Heating   Cooling   Wind   Vibration   Scent   Fog

  Sprayer   ColorCorrection   Tactile   Kinesthetic   RigidBodyMotion

Device Command Vocabulary


  LightSensor   AmbientNoiseSensor   TemperatureSensor   HumiditySensor   LenghtSensor   AtmosphericPressureSensor   PositionSensor   VelocitySensor

  AccelerationSensor   OrientationSensor   AngularVelocitySensor   AngularAccelerationSensor   ForceSensor   TorqueSensor   PressureSensor   MotionSensor   IntelligentCamera

Sensed Information Vocabulary


•  Interaction Information Description Language (IIDL) •  Three Root Elements –  InteractionInfo –  DeviceCommand –  SensedInfo


•  Interaction Information Description Language (IIDL) –  Provides base types for data formats for interaction

devices •  DeviceCommandBaseType •  SensedInfoBaseType


•  Interaction Information Description Language (IIDL)

Part 2: Control Information •  Provide Capability and Preference Descriptions for

Fine-Tuned Control of Devices

Adaptation RV/VR Engine

(non-normative)

Sensory Information

(Part 3)

Sensory Device Capability

Sensor Capability User’s Sensory Preference

Device Commands

Sensed Information

Within Scope of Part 2 Virtual World Object Char.

(Part 4)

Part 2: Control Information

•  Control Information Description Language (CIDL) •  Device Capability Description Vocabulary (DCDV) •  Sensor Capability Description Vocabulary (SCDV) •  User’s Sensory Effect Preference Vocabulary (SEPV)


  LightCapability   FlashCapability   HeatingCapability   CoolingCapability   WindCapability   VibrationCapability   ScentCapability   FogCapability

  SprayerCapability   ColorCorrectionCapability   TactileCapability   KinestheticCapability   RigidBodyMotionCapability   MoveTowardCapability   InclineCapability

Device Capability Description Vocabulary


  LightSensorCapability   AmbientNoiseSensorCapability   TemperatureSensorCapability   HumiditySensorCapability   LenghtSensorCapability   PositionSensorCapability   VelocitySensorCapability   AccelerationSensorCapability

  OrientationSensorCapability   AngularVelocitySensorCapability   AngularAccelerationSensorCapability   ForceSensorCapability   TorqueSensorCapability   PressureSensorCapability   MotionSensorCapability   IntelligentCameraCapability   AtmosphericPressureSensorCapability

Sensor Capability Description Vocabulary


  LightPref   FlashPref   HeatingPref   CoolingPref   WindPref   VibrationPref   ScentPref   FogPref   SprayingPref   ColorCorrectionPref

  TactilePref   KinestheticPref   RigidBodyMotionPref   MoveTowardPref   InclinePref   WavePref   CollidePref   TurnPref   ShakePref   SpinPref

User’s Sensory Effect Preference Vocabulary (SEPV)


  Control Information Description Language (CIDL)   Provides basic structure of tools defined in part 2


  Control Information Description Language (CIDL)   Also provides base types for each type of description

  Sensory Device Capability Base Type


  Control Information Description Language (CIDL)   Also provides base types for

each type of description   Sensor Capability Base

Type


  Control Information Description Language (CIDL)   Also provides base types for each type of description

  User Sensory Preference Base Type

Conclusion and Discussions

•  Capturing and Controlling Real-World can be supported by MPEG-V Part 5

•  For the fine-tuned control/capture of real-world, MPEG-V Part 2 can help.

•  For the personalized effects, MPEG-V Part 2 is required.

•  Questions?

•  Thank You.

Immersive Future Media Technologies Sensory Experience

Christian Timmerer

Klagenfurt University (UNI-KLU) Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC) Multimedia Communication (MMC)

http://research.timmerer.com http://blog.timmerer.com mailto:[email protected]

Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-‐Next project (FP7-‐ICT-‐216217), and the ALICANTE project (FP7-‐ICT-‐248652).

Motivation

•  Consumption of multimedia content may stimulate also other senses –  Vision or audition –  Olfaction, mechanoreception, equilibrioception, thermoception, …

•  Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects

… giving her/him the sensa<on of being part of the par<cular media ➪ worthwhile, informa<ve user experience

Outline

•  Background / Introduction –  MPEG-V Media Context and Control –  Sensory Effect Description Language (SEDL) and Sensory

Effect Vocabulary (SEV) –  Software/Hardware components: SEVino, SESim, SEMP,

and amBX+SDK

•  Improving the QoE through Sensory Effects ➪ Sensory Experience –  A Brief Introduction to UME/QoE (UMA/QoS) –  Results from Subjective Tests

•  Conclusions and Future Work

MPEG-V: Media Context and Control

System Architecture

Pt. 1: Architecture

Pt. 3: Sensory Informa0on

Pt. 4: Virtual World Object Characteris0cs

Pt. 2: Control Informa0on

Pt. 6: Common Types and Tools

Pt. 7: Conformance and Reference Sodware

Pt. 5: Data Formats for Interac0on Devices

hep://www.chiariglione.org/mpeg/working_documents.htm#MPEG-‐V

Sensory Effect Description Language (SEDL)

•  XML Schema-based language for describing sensory effects –  Basic building blocks to describe, e.g., light, wind, fog, vibration, scent –  MPEG-V Part 3, Sensory Information –  Adopted MPEG-21 DIA tools for adding time information (synchronization)

•  Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV) –  Extensibility: additional effects can be added easily w/o affecting SEDL –  Flexibility: each application domain may define its own sensory effects

•  Description conforming to SEDL :== Sensory Effect Metadata (SEM) –  May be associated to any kind of multimedia content (e.g., movies, music,

Web sites, games) –  Steer sensory devices like fans, vibration chairs, lamps, etc. via an

appropriate mediation device

➪ Increase the experience of the user ➪ Worthwhile, informative user experience

Sensory Effect Description Language (cont’d)

SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects| Effect|ReferenceEffect)+

Declarations ::= (GroupOfEffects|Effect|Parameter)+

GroupOfEffects ::= timestamp EffectDefinition EffectDefinition (EffectDefinition)*

Effect ::= timestamp EffectDefinition

EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][location] [adaptability]

Sensory Effect Vocabulary (SEV)

•  Extensibility: additional effects can be added easily w/o affecting SEDL •  Flexibility: each application domain may define its own sensory effects •  Sensory Effects

–  Light, colored light, flash light –  Temperature –  Wind –  Vibration –  Water sprayer –  Scent –  Fog –  Color correction –  Rigid body motion –  Passive kinesthetic motion –  Passive kinesthetic force –  Active kinesthetic –  Tactile

Example

<sedl:GroupOfEffects si:pts="3240000" duration="100" fade="15" position="urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front">

<sedl:Effect xsi:type="sev:WindType" intensity="0.0769"/>

<sedl:Effect xsi:type="sev:VibrationType" intensity="0.56"/>

<sedl:Effect xsi:type="sev:LightType" intensity="0.0000077"/>

</sedl:GroupOfEffects>

SEVino, SESim, SEMP, and amBX

amBX (Ambient Experience) system + SDK •  Two fan devices, a wrist rumbler, two sound

speakers, a subwoofer, two lights, and a wall washer

•  Everything controlled by SEM descriptions except light effect

Annota0on Tool: SEVino Simulator: SESim Player: SEMP

Quality of Experience

Factors impacting Quality of Experience

T. Ebrahimi, “Quality of Mul0media Experience: Past, Present and Future”, Keynote at ACM Mul0media 2009, Beijing, China, Oct 22, 2009. hep://www.slideshare.net/touradj_ebrahimi/qoe


(QoE)

Device Network

Content Format

Environment Content

User Expecta<on

Task Applica<on

Technical Factors

Social and Psychological

Factors

User

Context


•  Universal Multimedia Access (UMA) –  Anywhere, anytime, any device + technically feasible –  Main focus on devices and network connectivity issues ➪ QoS

•  Universal Multimedia Experience (UME) –  Take the user into account ➪ QoE

•  Multimedia Adaptation and Quality Models/Metrics –  Single modality (i.e., audio, image, or video only) or a simple combination of

two modalities (i.e., audio and video) •  Triple user characterization model

–  Sensorial, e.g., sharpness, brightness –  Perceptual, e.g., what/where is the content –  Emotional, e.g., feeling, sensation

•  Ambient Intelligence –  Add’l light effects are highly appreciated for both audio and visual content –  Calls for a scientific framework to capture, measure, quantify, judge, and

explain the user experience

F. Pereira, “A triple user characteriza0on model for video adapta0on and quality of experience evalua0on,” Proc. of the 7th Workshop on Mul<media Signal Processing, Shanghai, China, October 2005, pp. 1–4.

B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208. E. Aarts, B. de Ruyter, “New research perspec0ves on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14.

Experiment: Goal & Setup

•  Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the actual genre

•  Tools –  Sensory Effect Media Player (SEMP) –  Test sequences annotated with sensory effects: action (Rambo 4,

Babylon A.D.), news (ZIB Flash), documentary (Earth), commercials (Wo ist Klaus), and sports (Formula 1)

–  Double Stimulus Impairment Scale (DSIS) also known as Degradation Category Rating (DCR)

•  Five-level impairment scale ➪ new five-level enhancement scale •  Procedure

–  First, show reference sequence w/o sensory effects –  Second, the same sequence enriched with sensory effects with a two

second break in between –  Finally, subjects to rate the overall opinion of the audio/video resource

and sensory effect quality

Experiment: Results

Conclusions

•  Sensory effects is a vital tool for enhancing the user experience leading to a unique, worthwhile Sensory Experience –  Action, sports, and documentary genres benefit more

from these additional effects –  Rambo 4 and Babylon A.D. are from the same genre,

the results differ slightly –  Commercial genre can also profit from the additional

effects but not at the same level as documentary –  News genre will not profit from these effects

•  Interoperability through MPEG-V (ISO/IEC 23005)

Acknowledgments

•  EC projects for funding this activity –  NoE INTERMEDIA (NoE 038419)

•  http://intermedia.miralab.ch/ –  P2P-Next project (FP7-ICT-216217)

•  http://www.p2p-next.eu –  ALICANTE project (FP7-ICT-248652)

•  http://www.ict-alicante.eu –  COST ICT Action IC1003

•  QUALINET – European Network on Quality of Experience in Multimedia Systems and Services

•  Markus Waltl for implementing, preparing, conducting, evaluating almost all the experiments

•  Benjamin Rainer for implementing the Firefox plug-in + WWW tests •  Hermann Hellwagner for his advice and feedback •  ISO/IEC MPEG and its participating members for their constructive

feedback during the standardization process

References •  Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of

Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.

•  C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-Malo, France, September 28-30, 2009.

•  M. Waltl, Enriching Multimedia with Sensory Effects, VDM Verlag Dr. Müller, February, 2010.

•  M. Waltl, C. Timmerer and H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”, Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.

•  C. Timmerer, M. Waltl, and H. Hellwagner, “Are Sensory Effects Ready for the World Wide Web?”, Proceedings of the Workshop on Interoperable Social Multimedia Applications (WISMA 2010), Barcelona, Spain, May 19-20, 2010.

•  M. Waltl, C. Timmerer, and H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”, Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway, June 21-23, 2010.

•  M. Waltl, C. Raffelsberger, C. Timmerer, and H. Hellwagner, “Metadata-based Content Management and Sharing System for Improved User Experience”, Proc. of the 4th InterMedia Open Forum (IMOF 2010), Palma de Mallorca, Spain, September 1, 2010.

Thank you for your attention

... questions, comments, etc. are welcome …

Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer Klagenfurt University, Department of Information Technology (ITEC)

Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA [email protected]

http://research.timmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699

© Copyright: Christian Timmerer

Overview of virtual world assets characteristics

(MPEG-V part 4)

Jae Joon Han Samsung Electronics.

2010.1.25

•  Introduction

•  Requirements for standardization

•  Specifications

•  Conclusion

Agenda

Introduction (1/2)

Enable the interoperability between virtual worlds

(Adaptation VV) Provides controllability of virtual worlds with the real world devices

Architecture and specifications of associated information representations of virtual worlds

Introduction (2/2)

Avatars

Virtual Object

"  Characterize virtual world objects by the two elements.  Avatar: a representation of the user inside the virtual environments.  Virtual object: any object except for avatars in the virtual environments.

•  Need to import characters from one virtual world to another virtual world.

•  Provide common formats for interfacing between virtual world and the real world.

Why need the common specifications?

VW 1 VW 2

Import parameters of the created avatar

VW N …

…

Scaling & Rotation of an object

•  Characterize various kinds of objects within the VW. •  referencing the resources

•  Provide an interaction with the VW.

Virtual world object characteristics (VWOC)

sound scent

animation

Resources

references as a resource

Interaction as an event

Interaction Devices

•  Common characteristics for both avatars and virtual objects •  Identification •  A list of sound resources •  A list of scent resources •  A list of control parameters •  A list of events •  A list of behavioral models

•  Inherit the base type to extend the specific aspects of each.

A base type of virtual world object characteristics

Characterizes virtual world objects

Provides Interaction with virtual world objects

•  Describes the ownership, credits, associated user ID.

•  Supports digital rights according to ISO/IEC 21000-5:2004 (MPEG21)

Identification

Name Cello

family Musical Instrument

UserID JohnDoe

Ownership V. School of Art

Rights No duplication

Credits Ms. Jane Sound

Added a cello sound.

Virtual Instruments

shop

Sound and scent type

•  Contains the URLs of the resources. •  Contains the descriptions of the resources

–  ID, intensity, duration, loop, and name.

V- concert Performance

Name CelloSound

Resource http://...

Intensity 50%

duration 2 seconds

loop unlimited

V-‐Cello

Control type

•  Contains the control parameters for position, orientation, and scale factor.

Scaling an object Rotating an object

Event type and behavioral model

•  Event type supports legacy input devices and user defined Input devices.

•  Behavioral model provides the mapping between the in/out event by IDs

Mouse Keyboard User defined input devices

“Wow~”

walk animation (ID: AniID4)

wow sound (ID: SoID5)

Mouse left click (ID: EventID1)

Additional common types

•  Haptic properties

•  Animation description type

Tactile properties Dynamic force effects Material properties

S0ffness

Damping …

Animation Name: RunAndJump, Duration:10 seconds, Loop: 5 repetitions, URL: http://www.animation.com /jumping.ani

Avatar metadata

•  Based on VWOBaseType (ID, Behavioral model, Control, Event, Resources (Sound and Scent))

•  Other resources defined for Avatar –  Appearance: body parts, accessories, and physical

conditions, links to the resources. •  ex.) NoseType = {size, width, upper bridge, lower bridge,

tip angle, hapticIDRef and etc} –  Animation –  Communication skills

•  Supports input/ouput channels. Each channel support verbal and non-verbal communication. (ex. text, voice, sign language, gesture and so on.)

–  Personality –  Control features –  Haptic properties –  Gender

Avatar metadata (animation) •  Composed of various types of animation sequences

–  Ex. idle, greeting, dancing, and fighting animation types

•  Each type has its own animation set defined in its classification scheme.

Resting pose Body noise Default idle

Examples of Idle type

Avatar metadata (personality) •  Based on the OCEAN model which is a set of personality

characteristics. –  Openness, agreeableness, neuroticism, extraversion, and

conscientiousness (ranged between -1 and 1) •  The characteristics can be used for

–  Designing the characteristics of the avatar. –  VW can interpret its inhabitant wishes. –  Adapt the communication to the personality.

Agreeableness = -0.9 Agreeableness = 0.9

No As always

Help me

Can you help me?

Communication Reaction creation based on personality

Avatar metadata (Control feature)

 Supports the feature points of the avatar to control both face and body. –  Placeholder for sensor (sensed information) –  Facial control features are defined by the outlines of each facial

part. –  Body features are defined by the bones of the skeleton.

Head

Upper

Middle

Down

Virtual Object •  Defines the representation of virtual objects inside the

environment –  to characterize various kinds of objects. –  to provide an interaction with the real world devices.

•  Support the following types of data, in addition to the common characteristics. –  Appearance, animation, haptic properties with the base type –  Components: allows to build a virtual object with the combination

of virtual objects. Carrot

-sound : cutting sound, frying sound -smell : carrot juice, carrot soup etc -stiffness : 4th level -components: green part and main part -behavioral model: input: mouse left click output: cutting sound + slicing animation Virtual cooking class

Covered and not yet covered

Covered Not covered yet

Interfaces between virtual worlds

•  Avatar characteristics

- size, clothing, accessories,

•  Personal attributes

- Shape, animation, control

•  VO characteristics

- cars, house, furniture, …

•  User ID, user profile, ownership,

rights and obligation of VO

•  Haptic properties

•  Communication protocols

- Ensure security, trust, privacy

•  Virtual currency exchange

•  Visual context on location and ori

entation information from virtual w

orld

•  Personal attributes

-Movement, behaviors

Covered and not yet covered

Covered Not covered yet

Interfaces between virtual worlds and the physical world

•  Many sensors

•  Mental state

- Emotional, physical condition

•  Control feature points for avatar

•  Input events

•  Behavioral model

•  Not all the sensors,

•  User conditions (preference, atm

osphere, context, and so on),

•  Easy to use privacy /openness c

ontrol,

•  Contextual information (visual im

pression) from real world,

•  Timing constraints for sensors an

d actuators

Conclusion •  MPEG-V part 4: Virtual world object characteristics deals the

high level description of the two elements in the virtual world.

•  The specification describes - identity, - resources description (sound, scent, haptics, animation, and appearance), - real-time direct control (scaling, position, rotation, body skeleton, facial feature points), - behavioral mapping for interaction (input devices and output events).

•  The specification can support –  Easy import of characters from a virtual world to another virtual

world. –  Common formats for interfacing between virtual world and the

real world devices.

Thank you very much!!

mpeg v-awareness event

Technology

issue mpegv

mpeg terms

multisensorial content

d content3d virtual

immersion feelinghow

virtual worlds becomespossible

virtual environment

thephysical world