immersive medien und 3d-video - htw berlininka.htw-berlin.de/kui/05/05/docs/ralfschaefer_web.pdf ·...

pa

ge 1

R. Schäfer - Immersive Medien und 3D-Video

page

1

Fraunhofer-InstitutFraunhofer-Institut ffüürrNachrichtentechnikNachrichtentechnik

Heinrich-Hertz-Heinrich-Hertz-InstitutInstitutRalf Ralf SchSchääferfer

[email protected]@hhi.dehttp://ip.hhi.dehttp://ip.hhi.de

Immersive Immersive MedienMedienund 3D-Videound 3D-Video

pa

ge 2


page

2

OutlineOutline

Immersive Media Examples Interactive Media and Free Navigation Free Navigation in Video Scenes Examples Free Viewpoint Video Summary

pa

ge 3


page

3

ImmersiveImmersive Media Media –– CostsCosts vs. Sensation vs. Sensation

CDTV

Sensation

Costs

ImmersiveMedia

VoD

VCR

Interactive TV

AmericanFootball

Theatre

Rock Concert

Cinema

IMAX

Real Live Sensation

pa

ge 4


page

4

ImmersiveImmersive Display (UHDTV, Display (UHDTV, D-CinemaD-Cinema))

pa

ge 5


page

5

ExampleExample: : ImmersiveImmersive Video Video ConferencingConferencing –– im.pointim.point

• Seamless transition between the real and virtual world• Head motion parallax• Life-sized upper body images representing gestures and body language• 3D representation of the remote participants including provision of eye contact

Audio Speakers

Semi-circulartable

61” Plasmadisplay

Cameras

pa

ge 6


page

6

ExampleExample: : LargestLargest Panorama Panorama WorldwideWorldwide

Gasometerin Leipzig

Printed panorama with 60 m diameterand 36 m hight (weight > 1 ton)

pa

ge 7


page

7

pa

ge 8


page

8

CylindricalCylindrical ProjectionProjection withwith HHIHHI‘‘ss CineCardCineCardultraultra

υ

υ

Single level Multiple levels

CineCardultra: Horizontal & vertical soft edge blending Warping for cylindrical and parabolic projection

pa

ge 9


page

9

InteractiveInteractive Media Media Allow to be active in some way instead of just

being a passive consumer Enabled through convergence on

telecommunications, consumer electronics andcomputer science

Well-known: linking of multimedia data forinteractive access

Multimedia DVD, Internet presentations,Multimedia Home Platform (MHP)

page

10


page

10

Free NavigationFree Navigation Arbitrary selection of viewpoint and direction within real

world audio-visual scenes Well-known from

computer graphics, virtual reality, computer games synthetic scenes, objects or static views of real objects or restricted to predefined viewpoints (e.g. Formula 1

on Premiere)

page

11


page

11

Free Navigation in Real Free Navigation in Real ScenesScenes New technology allows generation of intermediate

views in between real existing camera view Image-Based rendering, 3D reconstruction Allows e.g. viewing sports or theatre from arbitrary

viewpoints New technologies for the whole chain:

Acquisition, signal processing, modelling,representation, compression, transmission,rendering, display

page

12


page

12

SceneScene RepresentationsRepresentationsMore geometry -based More image-based

StaticTexture,SingleGeometry

View-dep.Texture,SingleGeometry

StaticTexture,View-dep.Geometry

View-dep.Texture,View-dep.Geometry

Sprites

Layered -Depthimages

Lumigraph

Light-fieldrendering

Incarnations of 3D model

Conventional graphics pipeline Warping Interpolation

page

13


page

13

Systems with curved mirrors www.remotereality.com

Omni-directional VideoOmni-directional Video

page

14


page

14

Courtesy of Prof. Yokoya, Nara Institute of Scienceand Technology

Omni-directional Video Omni-directional Video –– Example 1 Example 1

page

15


page

15

Omni-directional Video Omni-directional Video –– Example 2 Example 2

page

16


page

16

Patches

InteractiveInteractive Navigation in 3D Panoramas Navigation in 3D Panoramas

page

17


page

17

Creation of 3D PanoramasCreation of 3D Panoramas By using several shots of a still camera and stitching techniques By using video and mosaiquing techniques

page

18


page

18

Current view on screen

Visibility sensor

InteractiveInteractive DecodingDecoding and and RenderingRenderingTiles can be streamed asJPEG or JPEG2000 files

page

19


page

19

Capturing a real dynamic scene with N cameras Transformation into a special data representation Interpolation of arbitrary intermediate views Allows free navigation within the scene (within

practical limits)

Free Viewpoint Video (Free Viewpoint Video (FVVFVV))

page

20


page

20

X

Virtual Camera

Path

Example: Eye Vision using tracking camerasExample: Eye Vision using tracking cameras

page

21


page

21

No reconstruction of 3D geometrie => many cameras No individual interaction possible Live broadcast event: Super Bowl 2001, 30 cameras Courtesy of Prof. Takeyo Kanade, Carnegie Mellon Univ.

Eye VisionEye Vision

page

22


page

22

Visual Hull ReconstructionVisual Hull Reconstruction

Synchronized Synchronized multi-video recordingmulti-video recording

Background subtractionBackground subtractionSilhouette extractionSilhouette extraction

Visual Hull Visual Hull reconstructionreconstruction

page

23


page

23

Example: Temple Acquisition by PhotographsExample: Temple Acquisition by Photographs

page

24


page

24

ProcessingProcessing StepStep: 3D : 3D MeshMesh ExtractionExtraction

page

25


page

25

Processing Step: TexturingProcessing Step: Texturing

page

26


page

26

Courtesy ofStephan Wuermlinet al.,ETH Zuerich,Switzerland

Multi camera video acquisitionMulti camera video acquisition

page

27


page

27

SegmentationSegmentationCourtesy of StephanWuermlin et al.,ETH Zuerich,Switzerland

page

28


page

28

Original sequence (1 camera)Original sequence (1 camera)

Source: ETH Zürich

page

29


page

29

Free Viewpoint Video from 16 CamerasFree Viewpoint Video from 16 Cameras Results produced by HHI Same functionality as CG

objects free navigation, can be

viewed from anyviewpoint/direction

Integration into completescenes(virtual/augmented/real)

But: depict appearance,motion, deformation of realworld objects

page

30


page

30

Courtesy of Microsoft Research, Redmont, WA, USA

Example: Depth-based RenderingExample: Depth-based Rendering

page

31


page

31

ConclusionConclusion

Immersive Media is a challengig and excitingfield of research & development

There are many approaches to allowinteractivity and free navigation in video scenes

Free Viewpoint Video (FVV) is a new conceptfor video entertainment which enables the userto navigate in real scenes in the same way asin virtual worlds

page

32


page

32

Fraunhofer Heinrich-Hertz-Institut

We put science into action

Thank you

immersive medien und 3d-video - htw berlininka.htw-berlin.de/kui/05/05/docs/ralfschaefer_web.pdf ·...

Documents