![Page 1: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/1.jpg)
Efficient multimediasupport in QtWebKit
on Raspberry Pi
![Page 2: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/2.jpg)
Talk outline
• Raspberry Pi
• WebKit architecture(s) with emphasis on Media playback
• Platform-level improvements for QtWebKit
![Page 3: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/3.jpg)
Raspberry Pi - Model B
• 700MHz single-core ARMv6 CPU• Broadcom BCM2835 VideoCore IV• 512 MB RAM (partly allocated for GPU)
![Page 4: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/4.jpg)
GStreamer OpenMax (gst-omx)
• Hardware decoders for H.264, VP8, Theora
• meta:EGLImage
• Custom audio sinks: HDMI and analog
• Very good integration within playbin
![Page 5: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/5.jpg)
Premises of the project
• Fullscreen browser, no tabs, no WM
• Focus on media consumption web-apps and Youtube TV
• OpenGL for rendering
• Qt5 platform, EGLFS QPA
![Page 6: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/6.jpg)
WebKit rendering - from HTML to the screen
![Page 7: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/7.jpg)
WebKit rendering - the forest
![Page 8: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/8.jpg)
WebKit1 rendering (with OpenGL)
• GraphicsLayers are backed by GL textures
• The video player has its own GraphicsLayer
• RenderLayers are traversed and painted recursively (to their GraphicsLayers): children below -> self -> children above
• Composition: GraphicsLayers are stacked to produce the final result
![Page 9: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/9.jpg)
WebKit2 rendering (OpenGL) - processes
• Rendering split into 2 processes: UI and Web
• Web process renders the layers (similar to WebKit1 rendering)
• UI process composites and shows them
• GraphicsLayers shared using GraphicsSurfaces
• A GraphicsSurface encapsulates a method to share gl textures among processes (EGLImages)
![Page 10: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/10.jpg)
WebKit2 rendering (OpenGL) - Coordinated graphics
![Page 11: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/11.jpg)
The MediaPlayer in WebCore
![Page 12: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/12.jpg)
MediaPlayer - platform level
![Page 13: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/13.jpg)
Video rendering - the inefficient way
• Sink negotiates xRGB or ARGB video caps
• Optional buffer copy in ::render, conversion to Cairo's ARGB pre-multiplied alpha
• Buffer data copy in MediaPlayerPrivate::paint to a WebCore::BitmapImage
• BitmapImage rendered in a QPixmap
• Another approach: GLTextureUpload meta (CPU -> GPU buffer copy)
![Page 14: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/14.jpg)
WebKit2 video rendering (inefficient)
![Page 15: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/15.jpg)
Video sink - improved integration withinthe platform
• Sink negotiates meta:EGLImage with upstream video decoder
• Buffer pool and EGLImage allocator (like in glimagesink)
• EGLImage encapsulated in a GstBuffer passed to MediaPlayerPrivate
• Efficient rendering in the player, different approaches for WebKit1 and WebKit2
![Page 16: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/16.jpg)
Improvements in WebKit2 rendering
![Page 17: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/17.jpg)
Improving media resources loading
![Page 18: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/18.jpg)
Using libsoup within QtWebKit
• The Qt-based resource loader doesn't cope well with large media resources
• The libsoup-based resource loader is more memory efficient and better integrated in WebCore
• Pre-allocation of GstBuffers within our HTTP source element
![Page 19: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/19.jpg)
WebRTC / getUserMedia
![Page 20: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/20.jpg)
Local devices listing/playback
• In JS: webkitGetUserMedia({audio: true}, successCallback, failureCallback);
• function successCallback(stream) { video.src = URL.createObjectURL(stream); video.play(); }
• Pipeline should also allow encoding and RTP payloading if required by webapp (WebRTC PeerConnection)
![Page 21: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/21.jpg)
WebRTC - RPiCam integration
• rpicamsrc: emits H.264 byte stream
• Leverage H.264 hardware rendering
• rpicamsrc ! h264parse ! omxh264dec ! sink
• with a tee downstream rpicamsrc to allow encoding/streaming later on
![Page 22: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/22.jpg)
WebAudio
![Page 23: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/23.jpg)
WebAudio - basics
• JS API for processing and synthesizing audio in web applications
• Pipeline based. Examples of nodes: BufferSource, GainNode, BiQuadFilter, Destination...
• Head-related transfer function: tricking the human ear to spatial sound
![Page 24: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/24.jpg)
WebAudio - decoding incoming files
• First scenario: local file. Leverage decodebin and appsink to create an in-memory representation of the audio file PCM data (FloatArray).
• Second scenario: memory buffer. Decodebin again! With giostreamsrc
• No RPi-specific optimisations
![Page 25: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/25.jpg)
WebAudio - playback
• Playback of WebAudio FloatArrays PCM samples to soundcard
• Custom GStreamer source element encoding internal WebCore audio buffers to WAV
• Again, no RPi-specific optimisations
![Page 26: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/26.jpg)
So which part of the backend needed optimisations?
![Page 27: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/27.jpg)
WebAudio - HRTF Database
• Loaded at AudioContext creation, from a separate thread
• 240 very small (256 frames) WAV files => lots of context switches
• Improvement: concatenate all samples and internally split when building the database
• => only one file to read, less context switches!
![Page 28: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/28.jpg)
WebAudio - FFT experiments
• default WebAudio FFTFrame backend using GstFFT (itself based on KissFFT)
• math functions hot on perf profiles
• GPUFFT library (from rpi-userland): leveraging VideoCore for FFT calculations
• Major drawback: requires allocated RAM, so less RAM for the other libraries
![Page 29: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/29.jpg)
Next tasks
• video/canvas and video/webgl integration
• WebRTC
• ORC ARMv6 backend?
![Page 30: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/30.jpg)
Demo rendering from youtube at 720p
![Page 31: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/31.jpg)
Demo rendering from youtube at 1080p
![Page 32: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/32.jpg)
![Page 33: Efficient multimedia support in QtWebKit on Raspberry Pi (GStreamer Conference 2014)](https://reader033.vdocuments.mx/reader033/viewer/2022042713/54809157b37959932b8b5c72/html5/thumbnails/33.jpg)
Flickr pictures
• http://bit.ly/1p8YHJA
• http://bit.ly/1yz8ctT
• http://bit.ly/1v8nwLh
This is the last slide, let's have lunch now!