augmented reality: connecting physical and digital worlds

Download Augmented Reality: Connecting physical and digital worlds

Post on 17-May-2015




2 download

Embed Size (px)


I made this presentation at the MPEG Multimedia Ecosystem 2013 in Incheon. It includes a summary of MPEG technologies related to Augmented Reality and is focuses on the separation between the AR creation and AR consuming. A system architecture for AR is also presented. Photo credits: Lisa Blum, Richard Wetzel, Veronica Scurtu Note: many pictures used in this presentation are downloaded from the Internet; I'll be happy to add credits to the original authors if they let me know



2. From centuries, we are building two different types of worldsPhysicalInformationalImmersion, from a human user point of view, is making abstraction of the Physicalworld (remove all possible connection to it) and have stimuli only from theInformational oneImmersion, presence, stimuli, perception, storiesmusicfilmsknowledge 3. In which world our users are?The stimuli from physical worldsare familiar or at very lowintensities 4. We intentionally block ourinterfaces with thephysical worldImproving immersion feeling 5. Strategies for blockingthe interfaces with thePhysical worldImproving immersion feeling 6. Strategies for extending the space ofthe Informational world stimuliImproving immersion feeling 7. How far weve gone withthis strategy?Relatively far Improving immersion feeling 8. Probably too far Improving immersion feelingHow far? 9. Its the time to connect our worlds 10. Why now? For the first time, smartphones enableeveryone to access Informational worldwhile sensing also the Physical world They change our view of the environment Enhance the rich historyand meaningfulness ofthe real world withpowerful digitalelements 11. Physical & Informational Augment The RealityAn invitation: become architects of a newkind of bridges 12. In AR, almost everything remains to be inventedLas VegasThis place some years ago 13. In AR, almost everything remains to be invented 14. Marketing Tourism Culture Education Serious and less serious games Hardware manufacturers Software vendorsIn AR, almost everything remains to be invented 15. We (and few others) already started and inviteyou to be part ofThe AR RevolutionIn AR, almost everything remains to be invented 16. Tidy City 17. Portal Hunt 18. Elements 19. ARQuiz 20. Augmented Books 21. Summer School games 22. TECHNOLOGY INSIGHTSMPEG-V AND MPEG ARAF 23. MPEG technologies related to ARMPEG-1/2(AV content)1992/4VRML1997 Part 11 - BIFS:-Binarisation of VRML-Extensions for streaming-Extensions for server command-Extensions for 2D graphics- Real time augmentation withaudio & video Part 2 - Visual:- 3D Mesh compression- Face animation1998 Part 2 Visual- Body animation1999MPEG-4 v.1MPEG-4 v.2First form of broadcast signal augmentation 24. MPEG technologies related to ARMPEG-42003AFX 2nd Edition:- Animation bymorphing- Multi-texturing2005 AFX 3rd Edition- WSS for terrainand cities- Frame basedanimation2007MPEG-4MPEG-4 Part 16 - AFX:- A rich set of 3Dgraphics tools- Compression ofgeometry,appearance,animation AFX 4th Edition- Scalable complexitymesh coding2011MPEG-4A rich set of 3D Graphicsrepresentation andcompression tools 25. MPEG technologies related to ARMPEG-42003AFX 2nd Edition:- Animation bymorphing- Multi-texturing2005 AFX 3rd Edition- WSS for terrainand cities- Frame basedanimation2007MPEG-4MPEG-4 Part 16 - AFX:- A rich set of 3Dgraphics tools- Compression ofgeometry,appearance,animation AFX 4th Edition- Scalable complexitymesh coding2011MPEG-42009 Part 25- Compression ofthird-party XML(X3D, COLLADA)MPEG-42004 Part 16- X3DInteractiveProfileMPEG-4 26. MPEG technologies related to ARMPEG-V - MediaContext and Control2011 2nd Edition:- GPS- Biosensors- 3D Camera201x Compressionof video +depth201xMPEG-V- 3D Video 1st Edition- Sensors andactuators- Interoperabilitybetween VirtualWorlds Feature-point baseddescriptors for imagerecognition201xCDVSMPEG-U AdvancedUser Interface2012A rich set of sensors andactuators- 3D AudioMPEG-H 27. MPEG technologies related to ARMPEG-V Media Context and Control 28. ActuatorsLightFlashHeatingCoolingWindVibrationSprayerScentFogColor correctionInitialize color correction parameterRigid body motionTactileKinestheticGlobal position commandSensorsLightAmbient noiseTemperatureHumidityDistanceAtmospheric pressurePositionVelocityAccelerationOrientationAngular velocityAngular accelerationForceTorquePressureMotionIntelligent camera typeMulti Interaction pointGaze trackingWindDustBody heightBody weightBody temperatureBody fatBlood typeBlood pressureBlood sugarBlood oxygenHeart rateElectrographEEG , ECG, EMG, EOG , GSRWeatherFacial expressionFacial morphologyFacial expression characteristicsGeomagneticGlobal positionAltitudeBendGasMPEG technologies related to ARMPEG-V Media Context and Control 29. All AR-related data is available from MPEG standards Real time composition of synthetic and natural objects Access to Remotely/locally stored BIFS/compressed 2D/3D mesh objects Streamed real-time BIFS/compressed 2D/3D mesh objects Inherent object scalability (e.g. for streaming) User interaction & server generated scene changes Physical context Captured by a broad range of standard sensors Affected by a broad range of standard actuatorsMain features of MPEG AR technologies 30. MPEG vision on AR: the MPEG AR BrowserPoint to a URL no need to download new applications for each contextThe browser Retrieves scenario from the internet Starts video acquisition Tracks objects Recognizes objects from visual signatures Recovers camera pose Gets streamed 3D graphics Composes new scenes Gets inputs from various sensors Offers optimal AR experience by constantly adapting interaction possibilities andobjects from a remote server.Industry Maximize number of customers through MPEG-compliant authoring tools andbrowsers No need to develop a new application for each use case and device platform 31. MPEG vision on ARMPEG-4 + MPEG-7 + MPEG-21 ++ MPEG-U+ MPEG-V= ARAFMPEG PlayerCompressionAuthoring ToolProduceDownload 32. Technology insightsModeling the content chain Creation Consumption The pivot : a representation formalism 33. Technology insights: the chainARAFBrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools 34. Technology insights: InfrastructureARAF BrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools 35. The ARAF browserARAF Browser Full scene graph representation (2D and 3D) Static and dynamic objects and scenes Programmable behavior (JS) Access to sensors and cameras Access to remote resources and services (httprequest and web sockets) Support for user interaction Support for compressed resources 36. The ARAF browserARAF BrowserARAF BrowserJavaScript EngineMPEG Totem ServicesWebSocketsServerCommunicationserviceWebSocketsClientARAF BrowserSC3DMCDecoderBBA DecoderARAF FileBIFS StreamSC3DMCStreamBBA StreamBIFS DecoderMPEG-4SceneIndexedFaceSetSBVCAnimationCameraImage dataReferenceImagesImage dataImage dataImage dataAFAR BrowserReferenceSignalDetectionDetectedImages 37. Convert media assets in a common format Describe, index, organize, search digital assets Compressed assets, ready for transmissionA multimedia repository and processing platformsupporting both creation and consumption Media&ServiceServers 38. Technology insights: the formalismARAF PlayerMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools 39. ARAF Description LanguageARAF : an XML representationPlayerCompressionAuthoring ToolProduceDownloadARAF 40. A set of scene graph nodes defined in MPEG-4 Audio, image, video, graphics, programming, communication, userinteractivity, animation Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, CameraCalibration, AugmentedRegionConnection to sensors defined in MPEG-V Orientation, Position, AngularVelocity, Acceleration, GPS, Geomagnetic, Altitude, Local camera(s)Compressed media Image, sound, video, 2D/3D graphicsAugmented Reality Application FormatARAF 41. Technology insights: creationARAFBrowserMediaServersServiceServersUserLocalSensors &ActuatorsRemoteSensors &ActuatorsMPEGARAFLocalReal WorldEnvironmentRemoteReal WorldEnvironmentAuth. Tools 42. Authoring ARAF Specific tools Generic tools TOTEM.Designer TOTEM.ScoutAuth. Tools 43. Specific Authoring Tools Scenario driven approach Tidy City, Portal Hunt, ARQuiz, AR Book, TreasureHuntAuth. Tools 44. Towards a generic Designer Create templates for app. objects Fill structure with data Create GUI Create application logic Export data 45. On site designer 46. Conclusions AR the next digital revolution For making revolutions, one needs weapons: MPEG technologies represent an efficient andpowerful set of weapons Current MPEG activities Improve and enrich the weapons Package them in consistent and easy to use tools