scalable remote rendering with augmented streaming
DESCRIPTION
Scalable Remote Rendering with Augmented Streaming. Dawid Pająk , Robert Herzog, Elmar Eisemann , Karol Myszkowski , Hans-Peter Seidel. Motivation. Service oriented Hardware and location independent computing Great for thin clients - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/1.jpg)
Scalable Remote Rendering with Augmented Streaming
Dawid Pająk, Robert Herzog, Elmar Eisemann, Karol Myszkowski, Hans-Peter Seidel
![Page 2: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/2.jpg)
2
Motivation
Service oriented Hardware and location
independent computing Great for thin clients Code and confidential
data kept in a secure remote environment
Low-cost of maintenance Full-time access
![Page 3: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/3.jpg)
3
Cloud Computing and Rendering
NVidia RealityServer & iRay
![Page 4: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/4.jpg)
4
Cloud Computing and Rendering
![Page 5: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/5.jpg)
5
MotivationServer
InternetInternet
Video Encoding
Client
Full-frameRendering
CG application
Video Decoding
Bandwidth:2-6Mbit per client
![Page 6: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/6.jpg)
6
MotivationServer
InternetInternet
Video Decoding
Video Encoding
Clients
Full-frameRendering
CG applicationVideo
Decoding
Bandwidth:2-6Mbit per client
![Page 7: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/7.jpg)
7
MotivationServer
InternetInternet
Video Decoding
Video Encoding
Clients
Full-frameRendering
CG application
Video Decoding
Out of resources!
Bandwidth:2-6Mbit per client
Design similar to current commercial solutions
Video Decoding
![Page 8: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/8.jpg)
8
The Idea
Server
InternetInternet
Video Encoding
Clients
Low-resolutionFrame Rendering
CG application
Auxiliary StreamEncoding
Similar bandwidth:2-6Mbit per client
Video Decoding
Auxiliary Stream
Decoding
Upsampling
Video Decoding
Auxiliary Stream
Decoding
Upsampling
Video Decoding
Auxiliary Stream
Decoding
Upsampling
![Page 9: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/9.jpg)
9
Motivation
Are thin clients really „thin”? 10 years ago – YES! Now – definitely NOT!
Year: 2000CPU: ARM7 33-75MhzNo FPUNo GPULow-res monochrome display
Year: 2006CPU: ARM11 333MhzFPU, SIMD capableGPU: OpenGL ES 1.1Display: 240x320 24-bit
Year: 2010CPU: Cortex A8 ~1GhzGPU: OpenGL ES 2.0Display: 640x960
Year: 2011 and upCPU: Dual/Quad CoreGPU: multi-core designHD-ready resolution display
![Page 10: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/10.jpg)
10
Our solution
Server
InternetInternet
Video Encoding
Clients
Low-resolutionFrame Rendering
CG application
Auxiliary StreamEncoding
Similar bandwidth:2-6Mbit per client
Video Decoding
Auxiliary Stream
Decoding
Upsampling
Video Decoding
Auxiliary Stream
Decoding
Upsampling
Video Decoding
Auxiliary Stream
Decoding
Upsampling
![Page 11: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/11.jpg)
11
Client
Client-side extra applicationsVideo
Decoding
Auxiliary Stream
Decoding
2x upsampled video usingdepth/motion encoded with H.264
Ours (same bandwidth)
Spatio-temporalUpsampling1
1[Herzog et al. 2010]
![Page 12: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/12.jpg)
12
Our solution – server-side
RendererRenderer
High-res.attributebuffers
High-res.attributebuffers
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Client side
![Page 13: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/13.jpg)
13
Edgedetection
Edgedetection
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
High-res.attributebuffers
High-res.attributebuffers
Client side
![Page 14: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/14.jpg)
14
Client side
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
High-res.attributebuffers
High-res.attributebuffers
J2K ContrastTransducer
J2K ContrastTransducer
Edge visual thresholding
![Page 15: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/15.jpg)
15
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
High-res.attributebuffers
High-res.attributebuffers
QuantizationQuantization
• float→int conversion• non-linear mapping of values• depth – 10 bits• motion – 2x8 bits
• float→int conversion• non-linear mapping of values• depth – 10 bits• motion – 2x8 bits
Client side
![Page 16: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/16.jpg)
16
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization Edge imagediffusion
Edge imagediffusion
High-res.attributebuffers
High-res.attributebuffers
Client side
t-1
![Page 17: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/17.jpg)
17
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
High-res.attributebuffers
High-res.attributebuffers
Our solution – server-side
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 18: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/18.jpg)
18
Client side
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Push Pull
![Page 19: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/19.jpg)
19
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 20: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/20.jpg)
20
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 21: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/21.jpg)
21
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 22: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/22.jpg)
22
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 23: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/23.jpg)
23
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 24: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/24.jpg)
24
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 25: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/25.jpg)
25
Our solution – server-side
Edge imagediffusion
Edge imagediffusion
Client side
Push Pull
![Page 26: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/26.jpg)
26
Client side
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization Edge imagediffusion
Edge imagediffusion
High-res.attributebuffers
High-res.attributebuffers
Diffusion error with respect to ground truth depth buffer
PSNR: 52.42dB PSNR: 48.47dB PSNR: 52.49dB
![Page 27: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/27.jpg)
27
Client side
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization Edge imagediffusion
Edge imagediffusion
Current frameprediction
Current frameprediction
High-res.attributebuffers
High-res.attributebuffers
2[Didyk et al. 2010]
Grid-based warping2
![Page 28: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/28.jpg)
28
Client side
Our solution – server-side
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization Edge imagediffusion
Edge imagediffusion
Current frameprediction
Current frameprediction
Edge imageEncoder
Edge imageEncoder
High-res.attributebuffers
High-res.attributebuffers
Binary edge image Depth and motion residuals
Auxiliary data frame packet
Lossless AC Prediction + Quantization + AC
Edge samplessign
![Page 29: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/29.jpg)
29
Binary edge image coding (edge topology)
Example:
Scanline traversal order
avg. bits per symbol
no edge (0), edge (1)symbol counters
compression ratio
![Page 30: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/30.jpg)
30
Binary edge image coding (edge topology)
x
x
xx
Samples unknown to the decoder!Samples unknown to the decoder!
• Probabilities for symbols depend not only on their frequencies, but also on the neighborhood (context)
Scanline traversal order
![Page 31: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/31.jpg)
31
…
0 0 0 0
0 0 0 0
0 0 0 0
…
Binary edge image coding (edge topology)
• Progressive encoding – pass 0/3
0 0 0
0 x
Samples taken from current frame predictionSamples taken from current frame prediction
![Page 32: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/32.jpg)
32
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
……
Binary edge image coding (edge topology)
• Progressive encoding – pass 1/3
![Page 33: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/33.jpg)
33
……
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
0 2
1
Binary edge image coding (edge topology)
• Progressive encoding – pass 2/3
![Page 34: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/34.jpg)
34
……
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
0 2
3 1
Binary edge image coding (edge topology)
• Progressive encoding – pass 3/3
![Page 35: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/35.jpg)
35
Current frame prediction
Depth and motion value coding
Current frame
Predictor choice based on the minimum residual “history”Predictor choice based on the minimum residual “history”
?
![Page 36: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/36.jpg)
36
RendererRenderer
Low-res.jittered frame
Low-res.jittered frame
H.264EncoderH.264
EncoderCameramatrix
Cameramatrix
Edgedetection
Edgedetection
QuantizationQuantization
Edge imageEncoder
Edge imageEncoder
Edge imagediffusion
Edge imagediffusion
Current frameprediction
Current frameprediction
High-res.attributebuffers
High-res.attributebuffers
36
Client side
Our solution – server-side
![Page 37: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/37.jpg)
3737
Client side
Our solution – client-side
Upsampling+
Other applications
Upsampling+
Other applications
Low-res.jittered frame
Low-res.jittered frame
H.264DecoderH.264
DecoderCameramatrix
Cameramatrix
Edge imageDecoder
Edge imageDecoder
Edge imagediffusion
Edge imagediffusion
Future frameprediction
Future frameprediction
High-res.attributebuffers
High-res.attributebuffers
![Page 38: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/38.jpg)
38
Results – QualityPS
NR
[dB]
Slightly lower depth qualitySlightly lower depth quality
Still better reconstruction!Still better reconstruction!
• x264 software used for H.264 low-res. stream encoding (@2MBit)
• 2x2 upsampling ratio
![Page 39: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/39.jpg)
39
Results – speed / scalabilityFramerate [FPS]
• x264 software used for H.264 low-res. stream encoding (@2MBit)
• 4x4 upsampling ratio• Costly pixel shaders:
SSAO, PCF soft shadow maps…
x2,2x2,2
x2,0x2,0
x1,9x1,9
x3,5x3,5
x3,0x3,0
x2,3x2,3
x3,7x3,7
x3,3x3,3
x2,4x2,4
![Page 40: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/40.jpg)
40
Client-side applications
Client
3D stereo vision Temporal frameinter/extra-polation
No additionalbandwidth cost!
![Page 41: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/41.jpg)
41
Video
![Page 42: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/42.jpg)
42
Limitations
Inherited from upsampling method– multi-sample techniques are problematic: transparency, anti-aliasing, refractions,
reflections
Edge representation– more difficult to control bandwidth than with DCT-based codecs– requires more research for efficient RDO
High-res. ground truth image Upsampled image
![Page 43: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/43.jpg)
43
Conclusions
Our scalable streaming framework– off-loads the server by moving part of the computation to
the client– auxiliary data stream allows for attractive applications on
the client side: 3D Stereo warping, temporal upsampling, in-game advertisement
– total bandwidth usage is comparable to streaming full-res. video using H.264
MC & DCT codecs are not an end-of-story for video compression!
![Page 44: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/44.jpg)
44
Conclusions
Thank You for your attention!
![Page 45: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/45.jpg)
45
Video
![Page 46: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/46.jpg)
46
Backup: Custom encoder
Upsampling requires very precise values of depth and motion – especially at discontinuities
High quality also required for other applications (IBR) MPEG is not well suited for such data
– minimizes visual error globally– encoding hard edges requires lots of space with DCT
transform – encoding both depth and motion in one stream is
problematic (chroma subsampling, luma based MC)– solutions for some problems are defined but not
supported by many software/hardware vendors
H.264 encoder Our custom encoder
![Page 47: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/47.jpg)
47
Temporally Interleaved Sampling
Temporal coherent sampling no new information for static frames Need disjoint sub-pixel sets for each frame
– We use a regular sampling pattern (efficient just jitter frames)
t
4n x 4n pixels
n x n pixels
![Page 48: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/48.jpg)
48
Temporal Reprojection Caching
Naïve blending ghosting artifacts! Compensate for camera and scene motion, Easy!
?
t t+1
motion flow
warp disocclusions
![Page 49: Scalable Remote Rendering with Augmented Streaming](https://reader036.vdocuments.mx/reader036/viewer/2022062520/568156f7550346895dc49ec6/html5/thumbnails/49.jpg)
49
Bibliography
[Herzog et al. 2010] - HERZOG R., EISEMANN E., MYSZKOWSKI K., SEIDEL H.-P.: Spatio-temporal upsampling on the GPU. In Proc. of I3D (2010), ACM, pp. 91–98.
[Didyk et al. 2010] - DIDYK P., EISEMANN E., RITSCHEL T., MYSZKOWSKI K., SEIDEL H.-P.: Perceptually-motivated real-time temporal upsampling of 3D content for high-refreshrate displays. Comp. Graph. Forum (Proc. of Eurographics) 29, 2 (2010), 713–722.
[Morvan et al. 2006] – MORVAN Y., H. N. DE WITH P., FARIN D.: Platelet-based coding of depth maps for the transmission of multi-view images. Proceedings of SPIE, Stereoscopic Displays and Applications (SD&A 2006)vol. 6055 p. 93--100, January 2006, San Jose (CA), USA