acm multimedia systems conference 2018 - distributed systems … · 2018-10-04 · acm multimedia...
TRANSCRIPT
![Page 1: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/1.jpg)
Distribution Systems for 3D Teleimmersive and Video 360
Content: Similarities and Differences
Klara Nahrstedt
Department of Computer Science
University of Illinois at Urbana-Champaign
ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands
![Page 2: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/2.jpg)
Overview
• Motivation
• 3D Teleimmersive Video Representation
• Video 360 Representation
• Similarities and Differences in Content Representation
• Distribution of 3DTI Video
• Distribution of Video 360
• Similarities and Differences in Content Distribution
• Conclusion
![Page 3: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/3.jpg)
3D Teleimmersive (3DTI) Systems
3Source: http://tele-immersion.citris-uc.org; http://monet.cs.illinois.edu/projects/cyphy-multi-modal-teleimmersion-for-tele-physiotherapy/teleimmersion-gallery/
![Page 4: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/4.jpg)
High-End Tele-Presence Environments
2
Traditional telephony and videoconferencing provide some of these elements, including ease of
use and audio quality, yet fail on most others. Our Coliseum effort aims to advance the state of
videoconferencing by applying recent advances in image-based modeling and computer vision to
bring these other elements of face-to-face realism to remote collaboration.
Scene reconstruction, the task of building 3D descriptions using the information contained in
multiple views of a scene, is an established challenge in computer vision [Longuet-Higgins 81].
It has seen remarkable progress over the last few years due to improved algorithms [Seitz 97,
Narayanan 98, Pollefeys 99] and faster computers. The Coliseum system is based on the Image-
Based Visual Hulls (IBVH) image-based rendering scene reconstruction technology of MIT
[Matusik 00]. Our recent Coliseum efforts have shown that the IBVH method can operate at
video rates from multiple camera streams hosted by a single personal computer [Baker 02].
Each Coliseum participant works on a standard PC with LCD monitor and a rig housing five
video cameras spaced at roughly 30 degree increments, as shown in Figure 1. During a
teleconferencing session, Coliseum builds 3D representations of each participant at video rates.
The appropriate views of each participant are rendered for all others and placed in their virtual
environments, one view of which is shown in Figure 2. The impression of a shared space results,
with participants free to move about and express themselves in natural ways, such as through
voice, gesture, and gaze.
Handling five video streams and preparing 3D reprojection views for each of numerous
coparticipating workstations at video rates is a formidable task. Tight control must be exercised
on computation, process organization, and inter-desktop communication. At project inception,
we determined we needed an effective speedup of about one hundred times over the MIT IBVH
processing on a single PC to reach utility. Our purpose in this paper is to detail some of the major
issues in attaining this performance.
Figure 1. The Coliseum immersive videoconferencing system
Cis
co T
ele
-pre
sence
HP
Halo
UN
CH
P C
ole
siu
m
![Page 5: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/5.jpg)
Multi-Camera Live Broadcast Systems
http://www.dailymail.co.uk/sciencetech/article-2336893/New-TV-cameras-bring-Matrix-style-bullet-time-
trickery-live-sports-coverage.html
![Page 6: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/6.jpg)
Multi-Camera Broadcast Systems
https://thegadgetflow.com/portfolio/slingstudio-
multi-camera-broadcaster/
https://www.myslingstudio.com/ https://www.cinfo.es/our-products/synthetrick/multicam
https://www.spiideo.com/sports/
![Page 7: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/7.jpg)
360-Degree Video
7360 Degrees Cameras – CoolPile.com: http://coolpile.com/tag/360-degrees-cameras
![Page 8: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/8.jpg)
3D Teleimmersive Video Representation
![Page 9: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/9.jpg)
3D Teleimmersive Stereo Video and Free Viewpoint Video Capture
![Page 10: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/10.jpg)
3DTI Viewing
Photo courtesy of Prof. Ruzena Bajcsy.
Singapore, 2014
![Page 11: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/11.jpg)
3D Stereo Video Representation
Wu, Ahsan, Kurillo, Agarwal, Nahrstedt, Bajcsy, “Color-plus-Depth Level-of-Detail in 3D Teleimmersive Video: A Psychophysical Approach”, ACM Multimedia 2011
![Page 12: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/12.jpg)
Free-Viewpoint 3D Video Representation
Example of 3D representation captured by different cameras
ca
me
ra-1
Cam
era
-2
Cam
era
-3
Cam
era
-8
![Page 13: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/13.jpg)
camera
direction
source: http://zing.ncsl.nist.gov/~gseidman/vrml/
Angle
θ
View Model
OiOu
![Page 14: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/14.jpg)
3DTI Data Model
• 3D frame for camera i at time t: fi,t
• Each pixel in the frame carries color+depth data and can be independently rendered
• Stream for camera i• Si = { fi,t1 fi,t2 … }
• Macro-frame• Ft = { f1,t f2,t … fn,t }
…
…
1 n
f1,t1 fn,t1Ft1
…f1,t2 fn,t2Ft2
S1 Sn
![Page 15: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/15.jpg)
360-Degree Video Representation
![Page 16: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/16.jpg)
360-Degree VideoUser’s Viewport
Generation of 360-Degree Video • Capturing of multiple 2D videos together with their metadata• Stitching videos together and further editing them in spherical video• Encoding spherical video considering projection, interactivity, storage and delivery formats
(this will impact decoding and rendering processes)
![Page 17: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/17.jpg)
Video 360 Viewing and Navigation
https://en.wikipedia.org/wiki/Head-mounted_display
Controller
Example of HDM (Head-Mounted Displays) – Oculus Rift, Samsung Gear VR, HTC Vive,
![Page 18: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/18.jpg)
360-Degree Video Data Model• Field-of-View or Viewport – display region on the Head-Mounted Display
• Fraction of omnidirectional view of the scene
• Viewport defined by a device-specific viewing angle (typically 120 degrees) which delimits horizontally scene from head direction center, called viewport center
• Viewport Resolution – 4K (3840x2160) pixels• Resolution of full 360-degree video – at least 12K (11520x6480)
• Video Framerate – order of HMD refresh rate 100Hz – 100 fps
• Motion-to-Photon Latency requirement • Less than 20 ms for VR – much smaller than Internet request-reply delay
• Need viewport prediction
• Bitrate – Video 360 vs HEVC (8K video at 60fps is approx. 100 Mbps)
• Tiling- Spatial divide of spherical video into in independent tiles
![Page 19: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/19.jpg)
Tiles and Spherical Maps
Issues with Spherical Mapping to Tiles• Viewport distortion• Spatial quality variance Considerations of sphere-to-plane mapping and viewing probability of tiles are IMPORTANT• Overall spherical distortion of segment is the sum of distortion over all pixels the segment
covers
Xie et al. “360ProbDASH: Improving QoE of 360 Video Streaming Using Tile-based HTTP Adaptive Streaming”, ACM MM 2017
![Page 20: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/20.jpg)
Video 360 Spherical-to-Plane Projections
Carbillon, Simon, Devlic, Chakareski, “Viewport-Adaptive Navigable 360-Degree Video delivery”, May 2017Nasrabadi et al. “Adaptive 360-Degree Video Streaming using Scalable Video Coding”, ACM Multimedia 2017
Video 360 Capture as Spherical Video
Equirectangular Projection – stretches poles and reduces efficiency of codingPyramid Projection – sees degradation on sides Cubemap – maps 90 degree FOV to sides of cube and provides hence less degradation
![Page 21: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/21.jpg)
Encoding and Delivery Formats • Codecs
• AVC/H.264, HEVC/H.265• VP8, VP9
• Delivery Formats• DASH/HLS (Dynamic Adaptive HTTP)
• MPEG-DASH Standard considers tiling
• MPD (Media Presentation Description) –Modified for Video 360
• SRD (Spatial Relation Description) integrated into MPD
• HEVC considers video tiles
• MPEG – Immersive media standard ISO/IEC 23090
• Part 1: Use cases• Part 2: OMAF (Omnidirectional Media
Application Format)• Description of equirectangular projection
format• Metadata for interoperable rendering of
360-degree monoscopic and stereoscopic audio-visual data
• Storage format (ISO base media file format/MP4
• Codecs: HEVC, MPEG0H 3D audio
• Part 3: Immersive video• Part 4: Immersive Audio
Graf, Timmerer, Mueller, “Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP”, ACM MMSys 2017
![Page 22: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/22.jpg)
Similarities and Differences of Representations
![Page 23: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/23.jpg)
Similarity Parameter 3DTI Video 360-Degree Video
Multi-camera Views Yes (view) Yes (viewport)
Joint coordinate system Yes Yes
Bitrate consideration Yes Yes
View change Yes Yes
Difference Parameter 3DTI Video 360-Degree Video
Video Format Color-Plus-Depth Color
Smallest item to adapt 3DTI frame tile
Frame Representation Frame manipulation at Pixel level (RGB, Depth, Polygons)
Frame manipulation at tiles and Region of Interest level
Coding Simple zlip Complex HVEC
Resolution 640x480 or 1080p 4K to 16K
Resolution for diverse devices No Yes
Format for diverse navigation No Yes
![Page 24: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/24.jpg)
Distribution Systems of 3DTI Video
![Page 25: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/25.jpg)
Multi-Camera 3DTI Transmission System
P
camera
av
dis
pla
y
CCR
GG switch
Site -2
A
microphone
camera
av
dis
pla
y
RC
C
Gswitch
Site-1
A
microphoneC = camera
A = microphone
G = gateway
R = renderer
Internet
25
![Page 26: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/26.jpg)
Approach: Multi-stream Hierarchical Adaptation
![Page 27: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/27.jpg)
Multi-stream Adaptation(Stream Selection)
• Camera orientation:
• User view orientation: cos = , , where is the angle between camera and user view
• Selection (SI) – View-Centric Stream Selection
where T is a user specified parameter
cameradirection
Zhenuy Yang, Klara Nahrstedt, Bin Yu, Ruzena Bajcsy, “A Multi-stream Adaptation framework for Bandwidth Management in 3D Teleimmersion”, ACM NOSSDAV 2006, May 2006, Newport, Rhode Island
![Page 28: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/28.jpg)
View-Centric Stream Differentiation
3D capturing
8
4
6 2
3D camera
transmission
8
4
6 2
3D rendering
user viewstreams contributingmore to user view
less important streams
![Page 29: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/29.jpg)
Timing Performance Validation
Macro-Frame Delay at Sender side
Macro-frame Completion Interval at Receiver Side (End-to-End Delay UIUC-UCB)
![Page 30: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/30.jpg)
Immersive View-Centric Multi-View Multi-Party 3DTI
Z. Yang et al. “ViewCast: View Dissemination and Management for Multi-Party 3D Tele-immersive Environments, ACM Multimedia 2007
![Page 31: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/31.jpg)
Multi-Party Multi-View Telepresence
Example of 3D representation captured by 4 camerasca
mer
a-1
Cam
era-
2
Cam
era-
3
Cam
era-
8
c1c2
Camera
c3
c4
c5
c6c7c8
view
Multi-stream contents
Multi-view environment
High resource demand
Multi-stream dependency
Real-time interactivity
![Page 32: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/32.jpg)
Telepresence Session Control
G
G GR
C
C
A
C
C R
A
R
A
C
C
C = camera
A = microphone
G = gateway
R = renderer
Decoupled control and data plane Hierarchical control Global session controller Local session controllers at G
Coordinated global control plane Monitor data plane Configure data plane
Data plane at TI participants Session routing table (SRT) Stream forwarding
Global Session Controller
(SRT)
Matching
Field (ID)
Forwarding
Action Bitrate
Site-X
Site-Y
Site-Z
![Page 33: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/33.jpg)
ViewCast: Middleware (Overlay) FrameworkA three-layer multi-party/multi-stream
management framework
View-aware Stream
Differentiation/Selection
Overlay network
Service Middleware
Network
Tele-immersive Application
ViewCast
![Page 34: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/34.jpg)
V2
V3
V4
V1
U2.wU3.w
U4.w
user view
U2
session
controller
U3
U4
3D capturing
8
4
6 2
3D camera
transmission
8
4
6 2
3D rendering
User/node’s view request
streams contributingmore to user view
less important streams
![Page 35: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/35.jpg)
V2
V3
V4
V1
U2.wU3.w
U4.w
U2
session
controller
U3
U4
U3.w
victim
Why view change a problem?
![Page 36: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/36.jpg)
Streams/View
GC = 100%, Ii (Oi) = 24
average 3.2
better than
MC–3
performance
but with 22%
less rejection
ratio
![Page 37: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/37.jpg)
Immersive and Non-ImmersiveMulti-Party Multi-View (Live Broadcast) Systems
Arefin Ahsan , Zixia Huang, Klara Nahrstedt, Pooja Agarwal, “4D TeleCast: Towards Large Scale Multi-site and Multi-view Dissemination of 3DTI Content”, IEEE ICDCS 2012, Makau, China.
![Page 38: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/38.jpg)
TI Components & Participants
• Immersive Participants
• Tight Interactivity
• Limited ScaleP
camera
avd
isp
lay
CCR
GG switch
SITE -2
A
microphone
Be
rke
ley
camera
avd
isp
lay
RCC
Gswitch
SITE-1
A
microphone
Illin
ois INTERNET
C = camera
A = microphone
G = gateway
R = renderer
S = sensors
S• Non-immersive Participants
• Large Scale
SITE-3
RG
SITE-4
RG
SITE-5
RG
SITE-6
RG
SITE-7
RG
SITE-9
RG
SITE-8
RG
SITE-10
RG
SITE-10
RG
Producers
NI Viewers
Producers
![Page 39: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/39.jpg)
View/Stream Concepts among Immersive Participants
3D capturing
8
4
6 2
3D camera
transmission
8
4
6 2
3D rendering
user viewstreams contributing
more to user view
less important
streams
Content Producer(Immersive Participant)
Content Producer(Immersive Participant)
![Page 40: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/40.jpg)
View/Stream Concept among Non-Immersive Participants
Site-A Site-B1
35
7
1
3
7
Dis
pla
y
view
Viewers
(Non-immersive Participant)
5
Camera
v1
v1 = [ ]
3D streams 3D streams
4D Content
Content
Producers 5
7
5
6 5 4 7 6 5
6 5 47 6 5> > > > >
![Page 41: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/41.jpg)
Site-B1
35
7
1
3
7
Dis
pla
y
view
Viewers
5
Camera
v2
Site-A
Producers
v2 = [ ]4 3 5 6 7 5
4 3 56 7 5> > > > >
Multi-View Video among Non-Immersive Participants
![Page 42: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/42.jpg)
Approach: 4D TeleCast
Producer Tier
Site-A Site-B
Site-C
Viewer Tier
C G ViewerCamera Communication
Gateway
InternetGSC
LSC LSC LSC
GSC – Global Session Controller LSC – Local Session Controller
![Page 43: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/43.jpg)
Vie
wer
Tie
r
G
C
C
AS
R
G
C
C
AS
R
Producer Tier
Site-BSite-A
Site-C
D
Internet
CDNCDN-P2P
Infrastructure Management
[CDN Assisted Peer]
Wang’08, Liu’10, Chang’09
![Page 44: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/44.jpg)
4D TeleCast
Distribution
Core Server
Edge Server
CDN
Request view V1={S1, S2, S3}
s1 s2 s3
Request view: V1={S1, S2, S3}
s1
s3
s2 s1
s2 s3
Request view: V1={S1, S2, S3}
Request view: V1={S1, S2, S3}
s2 s3
s1
![Page 45: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/45.jpg)
Multi-stream Dependency (Problem Description)
S1A S2
A
>dbuffv1 = {S1
A, S2A}
U1
dbuff
Send to
display
time
S1A
S1A
U2
S2A S1
AS2
A
U1
CDN
Victim stream
Maximum allowed delay bound = dbuff
S1A
Violation of delay bound by dbuff Waste of bandwidthVictim streams
U3
![Page 46: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/46.jpg)
Understanding E2E Delay
u2
A
B
CSite-A
Site-BSite-C
Layer-0
Layer-1
Layer-2
τ
Δ
end
-to
-en
d d
elay
Pro
du
cer
u1 s1A s1
B s1C
…
u3
u2
u1
u3
u3
u2
u1
τ = layer size
Δ = distance from source
• Use Delay Layer Hierarchy
00.10.20.30.40.50.60.70.80.9
1
0 2 4 6 8 10 12 14 16 18
Fra
cti
on
of
vie
wers
Maximum layer of accepted streams
![Page 47: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/47.jpg)
Distribution Systems for Video 360
![Page 48: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/48.jpg)
Pipeline of 360-Degree Video
Graf, Timmerer, Mueller, “Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP”, ACM MMSys 2017
![Page 49: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/49.jpg)
Challenges of 360-degree Video Distribution
• Real-Time Stitching
• Simulator Sickness in Interactivity Scenarios• Enable to react to HMD head movements as fast as the HMD refresh rate (120 Hz)
• Viewport extraction in real-time • Challenge: difficult to predict user orientation for more than 3 seconds
• Challenge: if short-term prediction is needed, how do we avoid rebuffering/stall under small playout buffers?
• Avoidance of bandwidth waste (if one downloads viewports that are not needed)
• Tiles prefetching error
![Page 50: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/50.jpg)
MPEG-DASH Video Distribution System for Single 2D Video Stream
dash.js. https://github.com/Dash-Industry-Forum/dash.js/wiki.
![Page 51: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/51.jpg)
MPEG-DASH Video 360 Video Streaming using Tiles
Graf, Timmerer, Mueller, “Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP”, ACM MMSys 2017
![Page 52: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/52.jpg)
360-Video Streaming Systems
• Tiling for Adaptive Streaming• Video divided into tiles• Depending on the mapping of spherical video projection, different tiles will be
streamed• Tiles currently viewed by users are streamed at high quality and the rest with low
resolution
• Personalized Viewport-Only Streaming – Asymmetric Panorama viewing• Also called asymmetric panorama viewport adaptive streaming
• Methods: Truncated Pyramid Projection (TSP), Cubemap• Video divided into segments• When client moves head, the viewport center changes and new viewport must be
display• Decrease of bitrate without decrease of quality of viewport
ISO/IEC JTC1/SC29/WG11/M. 2016. VR/360 Video Truncated Square Pyramid Geometry for OMAF.
![Page 53: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/53.jpg)
Tile-based HTTP Adaptive Streaming and Head Movement PredictionXie, Xu, Ban, Zhang, Guo, “360ProbDASH: Improving QoE of 360 Video Streaming Tile-based HTTP Adaptive Streaming”, ACM Multimedia 2017
![Page 54: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/54.jpg)
Tile-based HTTP Adaptive Streaming for 360 Video
![Page 55: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/55.jpg)
Data Model at 360ProbDASH Server
ERP – Raw Panoramic Video• ERP is divided into video chunks• Each chunk is cropped into N tiles, indexed in raster-scan order• Each tile is encoded into segments with M bit-rate levels• MxN optional segments stored at server and ready for pre-fetching and streaming
![Page 56: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/56.jpg)
360ProbDASH Approach
• Pre-fetch Segments by predicting viewport• Use probabilistic model for prediction
• Leverage Linear Regression Prediction of Orientation
• Distribution of Prediction Errors • Long-term predictions are hard
• 5 users data collection for short term prediction error (3 seconds)
Yourstory.comYaw prediction
Pitch prediction
Roll prediction
Delta = 3 sec
![Page 57: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/57.jpg)
Tile-based Adaptive Video Streaming
• Ochi et al use tile-based streaming where spherical video is mapped to equirectangularvideo and video is cut into 8x8 tiles
• Hosseini and Swaminathan use hexa-face sphere-based tiling of 360-degree video to take into account projection distortion
• Description of tiles with MPEG-DASH Spatial Relation Description
• Quan et al use prediction of head movement to deliver tiles
• Weaknesses of Tiling systems• Time and energy consuming reconstruction • Coding inefficiency due to independent tiling• Server management of files is difficult due to large amount of quality levels and large MPD files• Client selection process is complex• Mixed bit-rate tiles can result in visible border and quality inconsistence in combined-tiles rendering• Multiple Decoders
D. Ochi, Y. Kunita, A. Kameda, A. Kojima, and S. Iwaki. Live streaming system for omnidirectional video. In Proc. of IEEE Virtual Reality (VR), 2015.M. Hosseini and V. Swaminathan. Adaptive 360 vr video streaming: Divide and conquer! In IEEE International Symposium on Multimedia (ISM), 2016.F. Quan, B. Han, L. Ji, and V. Gopalakrishnan. Optimizing 360 video delivery over cellular networks. In ACM SIGCOMM AllThingsCellular, 2016.
![Page 58: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/58.jpg)
QER Viewport-Adaptive StreamingCarbillon, Simon, Devlic, Chakareski, “Viewport-Adaptive Navigable 360-Degree Video delivery”, May 2017
![Page 59: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/59.jpg)
Viewport Adaptive Streaming System
Carbillon, Simon, Devlic, Chakareski, “Viewport-Adaptive Navigable 360-Degree Video delivery”, May 2017
![Page 60: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/60.jpg)
Approach: QER - Quality Emphasized Region • Not only bit-rate adaptation but also QER server adaptation where different regions have
different quality• QER – Quality Enhanced Region
• Each QER is represented by Quality Emphasis Center (QEC)• Full video gets delivered in certain projection representation (equirectangular, cube, ..), but it has different
versions of video QEC• Client device selects the right representation and extracts viewport
• Viewport-adaptive streaming similar to DASH• Client runs adaptation algorithm to select video representation; selects QER and QEC of available QER• QEC selection is based on smallest orthodromic distance
• Orthodromic distance –shortest distance between two points on surface of sphere, measured along surface of sphere
• Video segment length• Temporal Chunk sent from server – 1-10 seconds• Tradeoff between short and long segments
• Expanded MPD• MPD file expanded with new information
• Coordinates of its QEC in degrees• Two angles (0,360) degrees and (-90,90) degrees
• All representations assume the same reference coordinate system
![Page 61: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/61.jpg)
QER-Based Viewport Adaptive Streaming
Carbillon, Simon, Devlic, Chakareski, “Viewport-Adaptive Navigable 360-Degree Video delivery”, May 2017
![Page 62: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/62.jpg)
Examples of Experimental Results• Metric to extract viewport – (1) MS-SSIM: Multi-Scale Structural Similarity and (2) PSNR• Original equirectangular video of full quality - 4K video with 1080p resolution• QEC - in center of face encoded with best quality, other faces at 25% of full quality• Distance - for d = 0, QEC and viewport center match 0.98; as d increases, quality decreases• QEC numbers - With increased QEC number, quality increases; shorter segments are better
![Page 63: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/63.jpg)
Similarities and Differences of Distribution Systems
![Page 64: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/64.jpg)
Similarity Parameter 3DTI Video 360-Degree Video
Dealing with Bandwidth Adapt Views Adapt Viewports
View change yes yes
Navigation Via mouse yes Via mouse yes
Client adaptation yes yes
Streaming Protocols TCP-based TCP-based
Difference Parameter 3DTI Video 360-Degree Video
Dealing with Bandwidth Adapt Views/Streams Adapt Viewports/Tiles
Encoding Standards zlip/some efforts in MPEG/OMAF on 3DTI compression
MPEG-DASH considersomnidirectional video tiles
Distribution Style Real-time view-based telepresencestyle or live view-based broadcast
On-demand DASH-style
Clients homogeneous heterogeneous
Viewing Flat 2D or 3D displays Head-Mounted Displays
Streaming Protocols TCP-Based HTTP-based Standard MPEG-DASH
Navigation Via mouse only Via mouse, head movement, hand movement
![Page 65: ACM Multimedia Systems Conference 2018 - Distributed Systems … · 2018-10-04 · ACM Multimedia Systems, June 12, 2018, Amsterdam, Netherlands. Overview ... AM Multimedia 2011](https://reader033.vdocuments.mx/reader033/viewer/2022042412/5f2bb876e2f6623435770fd7/html5/thumbnails/65.jpg)
Conclusion and Summary• 360-degree video is becoming possible for
• 3D teleimmersive video or • Omnidirectional video
• First solutions are coming up in terms of • capture, encoding and viewing
• But distribution represents challenge• Real-time live streaming or • Near-real-time distribution of 360-degree video
• A lot of presented material will be published in a survey paper • “Scalable 36-Degree Video Streaming: Challenges, Solutions and Opportunities”• Authors: Michael Zink, Ramesh Sitaraman, Klara Nahrstedt • Journal Venue: Proceedings of IEEE Special Issue• Editors: Boris Koldehofe, Ralf Steinmetz, …• Coming up in early 2019