using linux media controller for wayland/weston renderer · pdf fileget_capabilities get...

Technology Consulting Company IGEL Co.,Ltd.

Technology Consulting Company Research, Development & Global Standard

Using Linux Media Controller for Wayland/Weston Renderer

Takanari Hayama [email protected] http://www.igel.co.jp/

1

•  Wayland/Weston Overview

•  Porting Weston to R-Car

•  Why Linux Media Controller Renderer?

•  Linux Media Controller Framework

•  V4L2 Renderer Design

•  Conclusions

2

Agenda

WAYLAND/WESTON OVERVIEW �

3

Weston Architecture

Wayland Client

Weston

Wayland Client

Compositor Core

Compositor (e.g. DRM)

Renderer (e.g. GL)

Shell (e.g. desktop)

Shell Client (e.g. desktop-shell)

Wayland IPCLocal API Call

Confidential 4

Weston w/ DRM Backend

5

Wayland Client

Wayland Client Stub

Weston (Wayland Server)

Wayland Server StubOpenGL/ES

OpenGL/ES

compositor-drm

gl-renderer

shell

libgbm/libdrm

compositor-core

pixman-renderer

libpximan

GPU Kernel Driver

DRM/KMS Driver

Qt, EFL, etc.

Weston Components

Wayland Components

Other Components

IPC API Call

User Space

Kernel Space

wa

yla

nd

p

roto

co

l

6

Rendering and Composition: Overview (GL-Renderer)

Wayland Client

OpenGL/ES + wayland_egl

GPU driver for Wayland

Weston / DRM Compositor

OpenGL/ES + WL Ext. GBM

DRM/KMS

GPU

GPU Driver for Wayland

Display

wl_egl_window Type

wl_egl_window Type

RENDERING �

DISPLAYING �

wl_buffer Type

wl_buffer Type

FRAME BUFFERS �gbm_surface Type

CLIENT BUFFERS �wl_buffer Type

FULL SCREEN�wl_buffer Type DRM/KMS BO Type

SCANOUT BUFFERS �

Software�

Hardware�

Wayland/Weston Components

Khronos w/ Wayland Ext.

OSS Components

Hardware Specific

wa

yla

nd

p

roto

co

l

7

Rendering and Composition: Window Composition

Wayland Client





DRM/KMS

GPU


Display

wl_egl_window Type

wl_egl_window Type

RENDERING �

DISPLAYING �

wl_buffer Type

wl_buffer Type




SCANOUT BUFFERS �

Software�

Hardware�



OSS Components

Hardware Specific

1. Render w/OpenGL/ES

2.Commit buffers w/ eglSwapBuffers()

3. Import w/ eglCreateImageKHR()

6. Set composed buffers as KMS BOs.

5. Compose w/ OpenGL/ES

4. Register destination frame buffers allocated with GBM.

wa

yla

nd

p

roto

co

l

8

Rendering and Composition: Full Screen or Sprite Rendering

Wayland Client





DRM/KMS

GPU


Display

wl_egl_window Type

wl_egl_window Type

RENDERING �

DISPLAYING �

wl_buffer Type

wl_buffer Type




SCANOUT BUFFERS �

Software�

Hardware�



OSS Components

Hardware Specific

1. Render w/OpenGL/ES

2.Commit buffers w/ eglSwapBuffers()

3. Import w/ gbm_bo_import()

4. Set composed buffers as KMS BOs.

PORTING WESTON TO R-CAR �

9

1.  OpenGL/ES for Wayland/Weston

2.  Zero Copy Mechanism for Native Buffer

“Typically, hardware enabling includes modesetting/display and EGL/GLES2. On top of that, Wayland needs a way to share buffers efficiently between processes.”http://wayland.freedesktop.org/architecture.html

10

What Are Required?

•  Must support the following Native Display Types for eglGetDisplay(): –  wl_display for clients –  gbm handle for Weston

•  Must support the following EGL_EXTENSIONs: –  EGL_KHR_image_pixmap –  EGL_WL_bind_wayland_display

•  Must support the following Native Pixmap Type for eglCreateImageKHR(): –  EGL_WAYLAND_BUFFER_WL

•  Must support the following Wayland extension APIs: –  eglBindWaylandDisplayWL –  eglUnbindWaylandDisplayWL –  eglQueryWaylandBufferWL

11

Wayland Requirements for OpenGL/ES

WSEGL for Wayland

Weston for Renesas R-Car

12

Wayland Client

Renesas OpenGL/ES

libgbm w/ KMS Backend

wayland-kms


New OSS Components

Standard OSS Components IPC API Call

Wayland Client Support Wayland Server Support


Wayland Server Stub

EGL with Wayland Extension

Wayland Client Stub

libdrm & libkms

Proprietary

wayland-egl

1.  A client creates a wl_surface on the server.

2.  The client attach a wl_buffer to the created surface.

3.  The client submit the wl_buffer to the server.

4.  The server takes the wl_buffer and compose to the screen.

All of above should happen in zero-copy manner!�

13

Wayland Composition Revisited

•  An abstract data type that represents a reference to a pixel buffer.

•  2 open source implementations: –  wl_shm : wayland standard

•  Based on Linux shared memory. Not physically contiguous.

–  wl_drm : Mesa standard •  Based on DRI. Possibly physically contiguous.

•  Weston understands wl_shm only. Wl_drm is Mesa specific. Thus, wl_drm is not handled by Weston, but by Mesa internally.

14

What is wl_buffer by the way?

•  Requirements – End-to-end Buffer Zero Copy – Physically Contiguous Memory

•  wl_drm? –  Implementation is too Mesa dependent.

•  Need more generic implementation.

15

Which wl_buffer implementation to use?

•  KMS BO buffer type. – https://github.com/thayama/wayland-kms – Based on wl_drm in Mesa.

•  Imports DMABUF via PRIME, a dma-buf

interface layer in DRM. – Originally, we used DRM Handle, but we now

use DMABUF instead.

•  Can directly pass video output from V4L2.

16

wl_kms

17

Buffer Zero Copying with wl_kms

Wayland Client

Wayland Client Stub


Wayland Server StubOpenGL/ES gl-renderer

compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU Driver

18


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU DriverBuffer Allocation. Client creates rendering surface with EGL API. WSEGL allocates memory with KMS BO, and make avaialable to GPU.

19


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU DriverRendering to the allocated buffer. GPU gets all details needed about the buffer via WSEGL.

20


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU Driver

When a client calls eglSwapBuffers(), WSEGL commits a buffer to the server via Wayland. The details of the buffer is DMABUF fd, a stride, a size, and a pixelf ormat.

21


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU Driver

When the server receives the buffer, it imports with eglCreateImageKHR().

22


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU Driver

WSEGL gets details of the buffer from wayland-kms, and asks to import the given DMABUF. The buffer is then made available to GPU.

23


Wayland Client

Wayland Client Stub



compositor-drm

KMS BO

User Space

Kernel Space

Wayland GPU Driver

KMS Driver

OpenGL/ESWayland

GPU Driver

Gl-renderer can now refer the buffer passed by the client, and composes a final output.

WHY LINUX MEDIA CONTROLLER RENDERER? �

24

•  Applications are heading towards more and more GPU intensive.

•  People want to use GPU for more advanced UI, rather than a simple window composition. –  On the other hand, some people want to do

more complex composition using GPU. J

•  GPU Offloading is one way. –  https://archive.fosdem.org/2014/schedule/

event/wayland_gpu/ –  But, still premature for real products.

25

Motivation

•  Not many embedded SoC has multiple GPUs.

•  However, they often have sophisticated hardware for video signal processing that allow to do 2D blending.

•  Why not use them?

26

Simpler Approach?

LINUX MEDIA CONTROLLER FRAMEWORK �

27

•  Just a Video4Linux2 device.

•  Make V4L2 media device parameters and pipelines configurable from user space. – http://linuxtv.org/downloads/presentations/

summit_jun_2010/20100206-fosdem.pdf

•  Important keywords: Entities, Pads, and Links.

28

What is Media Controller?

29

Media Controller

Entity

Entity

1

0

0

1

Entity

0

2

1

Link

Entity

Source Pad

Sink Pad

May have multiple sink

pads

•  A standard Linux API to configure complicated media devices from user space.

•  On Renesas R-Car, VSP1, a device for video signal processing, is exposed via Media Controller API in Linux.

•  Zero-copy could be easily achieved via DMABUF. Ideal for our use case. – Can use wl_kms!

30

Why Media Controller?

•  Supported Features – Scaling – Cropping – Pixel Format Conversion – 4 to 1 Blending

•  As pipelines in VSP1 are configurable, best suits to Media Controller Framework!

31

Renesas R-Car VSP1

V4L2 RENDERER FOR WESTON�

32

V4L2 Renderer Support in Weston

33

Wayland Client

Wayland Client Stub


Wayland Server StubOpenGL/ES

v4l2-renderer device for VSPD

compositor-drm

v4l2-renderer

shell

libgbm/libdrm

compositor-core

GPU Kernel Driver

DRM/KMS Driver

Qt, EFL, etc.

Weston Components

Wayland Components

Other Components

IPC API Call

User Space

Kernel Space VSP V4L2

Driver

New Components

New components added to Weston. Plus Minimum changes to the existing compositor-drm to host a v4l2-renderer.

•  Kept as minimum as possible.

•  Changes are to load v4l2-renderer, and pass output buffers to v4l2-renderer. Almost same as those of for pixman-renderer.

•  Minor changes on output buffer allocations; DMABUF export and a read permission to mmap’d output buffer are added.

34

Changes to compositor-drm

•  Media device agnostic layer.

•  Does everything needed to import wl_kms and wl_shm buffer to V4L2 Media Controller arena, i.e. DMABUF.

•  Calculations required to figure out source regions and destination regions are done in v4l2-renderer.

•  Anything that are not media device specific is handled in v4l2-renderer.

35

v4l2-renderer

•  Media device specific layer.

•  Does everything specific to the media devices. –  Media Controller Framework requires the background

knowledge of the underlying media devices.

•  Does actual job to compose surfaces specified as DMABUF from v4l2-renderer.

36

V4L2 renderer device

API � Descriptions�init Initialize a v4l2 media controller device.

create_output Create an output. No buffer passed yet.

set_output_buffer Set an output buffer for the output.

create_surface Create a surface. No buffer passed yet.

attach_buffer Set a buffer for the surface.

begin_compose Begin a new composition.

finish_compose Finish the composition.

draw_view Compose the surface.

get_capabilities Get capabilities of the V4L2 Renderer Device.

37

V4L2 Renderer Device API

•  V4L2 Renderer is not official yet. – https://github.com/thayama/weston

38

Current Status

CONCLUSIONS �

39

•  Yes and No.

•  Configuring parameters and links are not free. Not really great if we need to configure media device very often. – DRI could be alternative.

•  On the other hand, use of the Linux standard features including DMABUF is great for extensibility and flexibility.

40

Was Media Controller the Right Choice?

•  Yes and No.

•  Other renderers help you implementing new renderer.

•  Geometry was the most complex part of Weston. –  Global coordinate, Output local coordinate, and

view coordinate.

•  Renderers are responsible for understanding these coordinates and rendering views to the correct location. –  Pixman-renderer can give you some idea; how

complicated it is.

41

Was implementing new renderer easy?

42

Coordinate System in Weston

Output 0Output 1

opaque region

View position is in Global Coordinate.

Rendering needs to take care of output local coordinate.

Opaque region is specified in View Coordinate.

APPENDIX �

43

Components used in Renesas OpenGL/ES for Wayland

Components � Descriptions�EGL with Wayland Extension

A thin layer to support Wayland specific EGL APIs and a native buffer type for eglCreateImageKHR(). https://github.com/thayama/libegl

wayland-kms A subclass of wl_buffer for to pass KMS BO. Defines a wl_kms wayland protocol, and server side codes. https://github.com/thayama/wayland-kms

libgbm w/ KMS Backend GBM frontend extracted from Mesa and used in Weston with a KMS Backend support. https://github.com/thayama/libgbm

WSEGL for Wayland A bridge component between GPU and Wayland.

44

using linux media controller for wayland/weston renderer · pdf fileget_capabilities get...

Documents