3d realistic broadcast system

8/2/2019 3D Realistic Broadcast System

1/38

Investigate of 3D realistic

broadcast system


2/38

Outline

Introduction

Video processing

Transmission Rendering

What are the most technical challenge?

Conclusion References


3/38

I. Introduction

Trends on broadcasting technology


4/38

Realistic broadcast systems


5/38

Requirements for realistic


6/38

Requirements for realistic

Multi-modal information

- 3D video with depth information at multiple

viewpionts.

- Multi-channel audio.

- CG model.

- Haptic Data. High quality data.

User friendly interaction.


7/38

Main components


8/38

II. Video processing

Multi-view video


9/38

Multi-view camera system


10/38

Multi-view codec system


11/38

Coding of multi-view video


12/38

Multi video coding


13/38

Cu trc don

Dongia cc min khng gian


14/38

Php donda trn H264/VAC


15/38

3D video system


16/38

Depth information


17/38

Depth camera system


18/38

Hybird camera system


19/38

Image of hybird camera


20/38

Problems of depth map


21/38

High quality depth map generation


22/38

III. Transmission


23/38

Cu trc cab m ha H264/AVC


24/38

NAL

Gip d dng nh xdliu H264/AVC titngvnchuynnh: RTP/IP chobtkdchv internet thi gian thc.

nhdng file thng dnglutr, truyndn.

H.23x cho dchvmthoi.

Hthng MPEG-2 cho dchvqung b.

Dliu video c m ha cspxp voNAL units.1 NAL unit gms nguyn lnbyte.


25/38

1 hthngdchvxut


26/38

3D contents generation


27/38

IV. Multiview Video Rendering

Belongs to the broad research field of image-

based rendering.

Multi-stereoscopic video with depth maps:

A number of video sequences captured from

diferent view points.

A single depth map is available to facilitate the

virtual view rendering.


28/38

Multi-stereoscopic video with depth

map

An example multiview data set.


29/38

Rendering and weight map generation

The rendering process from multiview video


30/38

Rendering Process

Split the to be rendered view into light rays.

For each light ray, trace the light ray to the surfaceof the depth map, obtain the intersection, andreprojected the intersection into nearby cameras(Cam 3 & Cam4).

The intensity of the light ray is thus the weightedaverage of the projected light rays in Cam 3 andCam 4.

The weight is the angular difference between thelight ray to be rendered and the light ray beingprojected.


31/38

Rendering Process (Cont)

When the virtual view point moves away from Cam4(where the depth map is given), the will be occlusionsand holes when computing the light ray/geometryintersection.


32/38


Convert the given depth map into 3D mesh surface, whereeach vertex corresponds to one pixel in the depth map.

The mesh surface is then projected to the capturing camerasto compute any potential occlusions in the captured images.

Finally, the mesh is projected to the virtual rendering pointwith multi-texture blending.


33/38


For each vertex being rendered, it is projected

to the nearby captured images to locate the

corresponding texture coordinate.

This process takes into consideration the

occlusion computed earlier: if a vertex is

occluded in a nearby view, its weight for that

camera will be set to zero.


34/38

The weight maps

The weight maps generated by the redering process


35/38

The weight maps (Cont)

The brighter pixels are the ones with largerweights.

During compression of the multiview video,

the pixels with high weights shall be encodedwith high quality, while the rest pixels can beencoded with low quality QuantizationParameter map: portion of the image that havelow weights (and thus high QP) are much morecoarsely quantized


36/38

Quantization Parameter Maps

QP values for weight maps


37/38


A segment of enceded images from cameras 2, 3 and 4


38/38

V. Technically ChallengingDepth/Disparity Estimation

Sub-pixel accuracy Temporal enhancement to reduce flickering effects

Depth map refinement for distorted depth map

Coding of Multiview Video + Depth Map Coding structure

Depth map coding scheme

Bit allocation for depth map coding

Intermediate View Synthesis

View synthesis method for depth map distortion

Filtering along object boundaries

3d realistic broadcast system

Documents