d6.3: 3d media tools – report – version b - ict-bridgetict- ?· fp7-ict-610691 d6.3: 3d media...

Download D6.3: 3D Media Tools – Report – Version B - Ict-Bridgetict- ?· FP7-ICT-610691 D6.3: 3D Media Tools…

Post on 18-Aug-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • D6.3: 3D Media Tools Report Version B

    Project ref. no. FP7-ICT-610691

    Project acronym BRIDGET

    Start date of project (duration) 2013-11-01 (36 months)

    Document due Date: 2016-07-31

    Actual date of delivery 2016-07-30

    Leader of this document Ingo Feldmann

    Reply to

    Document status Public deliverable

  • FP7-ICT-610691 D6.3: 3D Media Tools Report Version B

    D6.3: 3D Media Tools Report Version B 2/70

    Deliverable Identification Sheet

    Project ref. no. FP7-ICT-610691

    Project acronym BRIDGET

    Project full title BRIDging the Gap for Enhanced broadcasT

    Document name D6.3_final.docx

    Security (distribution level) PU

    Contractual date of delivery (none)

    Actual date of delivery 2016-07-30

    Document number

    Type Public deliverable

    Status & version v0.5

    Number of pages 70

    WP / Task responsible WP6

    Other contributors Nicola Piotto, Daniel Berjn Dez, Rafael Pags Scasso, Sergio Garca Lobo, Francisco Morn Bur-gos, Giovanni Cordara, Milos Markovic, Sascha Ebel, Wolfgang Waizenegger, Sylvain Renault, Thomas Ebner

    Author(s) Ingo Feldmann

    Project Officer Alberto Rabbachin

    Abstract This document summarizes the research achieve-ments of WP6 for the tasks of off-line extraction of 3D audio-visual content (T6.1), 3D Media Coding (T6.2) as well as the free viewpoint scene recon-struction and rendering (T6.3). Task T6.4 is not mentioned because it focuses on standardisation (in the context of WP6) and will have its results reported in D9.3 at the end of the project. The document extends the first technical deliverable D6.1.

    Keywords 3D tools, media analysis, 3D reconstruction

    Sent to peer reviewer 2016-07-08

    Peer review completed 2016-07-22

    Circulated to partners 2016-07-22

    Read by partners 2016-07-28

  • FP7-ICT-610691 D6.3: 3D Media Tools Report Version B

    D6.3: 3D Media Tools Report Version B 3/70

    Mgt. Board approval 2016-07-29

    Version Date Reason of change

    0.1 2016-05-24 Ingo Feldmann Initial draft

    0.2 2016-06-15 Ingo Feldmann First fully merged version with input from all part-ners

    0.3 2016-07-08 QAT ready version

    0.4 2016-07-19 Included comments from WP6 internal review

    0.5 2016-07-28 Included QAT comments, final version

  • FP7-ICT-610691 D6.3: 3D Media Tools Report Version B

    D6.3: 3D Media Tools Report Version B 4/70

    Table of Contents 1 Executive summary ............................................................................................................................................... 6 2 Introduction ............................................................................................................................................................. 6

    2.1 WP6 Objective ................................................................................................................................................................ 6 2.2 General algorithmic workflow for 3D video processing ............................................................................... 6 2.3 Structure of the document ........................................................................................................................................ 7

    3 3D Scene Off-line Modelling ............................................................................................................................... 7 3.1 Generation of detailed point clouds using alternative feature descriptors .......................................... 7

    3.1.1 Prioritisation of pairs of images .................................................................................................................... 8 3.1.2 Reduction of per-pair computational cost ................................................................................................ 9

    3.2 3D mesh processing .................................................................................................................................................. 10 3.2.1 Mesh cleaning..................................................................................................................................................... 10 3.2.2 Mesh smoothing and decimation ............................................................................................................... 12

    3.3 Seamless, static multi-texturing ........................................................................................................................... 13 3.3.1 Colour blending ................................................................................................................................................. 13 3.3.2 Occlusion detection ......................................................................................................................................... 13

    3.4 Generation of textured-splat models ................................................................................................................. 15 3.5 Definition of a hybrid SPLAts + meSH (SPLASH) model ............................................................................ 16 3.6 Dense depth based 3D surface reconstruction .............................................................................................. 25

    3.6.1 Automatic camera pre-selection ................................................................................................................ 25 3.6.2 Enhanced depth estimation workflow .................................................................................................... 27

    3.7 3D Audio modeling recorded audio bridget approach ........................................................................... 31 4 3D Media Coding ................................................................................................................................................... 32

    4.1 Coding scheme proposal for hybrid SPLASH models .................................................................................. 32 4.2 Framework for point cloud compression and visualization .................................................................... 34

    4.2.1 State-of-the-Art ................................................................................................................................................. 35 4.2.2 Compression methods for BRIDGET ........................................................................................................ 37 4.2.3 Description of the point cloud compression process ........................................................................ 37 4.2.4 Decompression .................................................................................................................................................. 41 4.2.5 Compression results ........................................................................................................................................ 42 4.2.6 Summary of compression using quantization and octrees ............................................................. 46

    5 3D Scene On-line Reconstruction ................................................................................................................... 47 5.1 Video-to-Point-Cloud Framework ....................................................................................................................... 47 5.2 Stereo SLAM framework ......................................................................................................................................... 47

    5.2.1 Map representation: points and keyframes .......................................................................................... 48 5.2.2 Front-end: tracking .......................................................................................................................................... 49 5.2.3 Back-end: mapping and loop closure ....................................................................................................... 52 5.2.4 Depth map fusion ............................................................................................................................................. 53 5.2.5 Considerations ................................................................................................................................................... 56

    5.3 Rendering of textured splat models ................................................................................................................... 56 5.4 Surface Patch Rendering ......................................................................................................................................... 60 5.5 Mesh post-processing and simplification......................................................................................................... 61 5.6 Web based rendering of compressed point clouds ...................................................................................... 65

  • FP7-ICT-610691 D6.3: 3D Media Tools Report Version B

    D6.3: 3D Media Tools Report Version B 5/70

    5.7 3D Audio reconstruction recorded audio bridget approach ................................................................ 67 6 Conclusion .............................................................................................................................................................. 67 7 References .............................................................................................................................................................. 68

  • FP7-ICT-610691 D6.3: 3D Media Tools Report Version B

    D6.3: 3D Media Tools Report Version B 6/70

    1 Executive summary