sound open firmware - github pages · 2020-06-05 · drivers are generic and not coupled to any...

Sound Open Firmware

Liam Girdwood

Open Source Summit + Embedded Linux Conference Europe 2018

Sound Open Firmware is an open source audio digital signal

processing firmware and driver infrastructure.

SOF Goals 1 DSP architecture and platform agnostic.

Separate architecture and platform code like Linux*.

Abstract interfaces for architecture APIs (again like Linux).

Host AP architecture and host AP OS agnostic.

Drivers are generic and not coupled to any host architecture.

Nothing in SOF that couples firmware or low level driver to any OS.

Toolchain freedom

Fully support open source tools – GCC, Octave, M4, Qemu.

Also support proprietary tools - Cadence xcc and emulator, Matlab.

Permissive licensing of all firmware and most SDK code.

*Other names and brands may be claimed as the property of others.

SOF Goals 2 Freedom for users to define new audio processing pipelines using open source

components and tools.

Freedom for users to develop and integrate proprietary audio processing algorithms.

All development done in public.

Support for several CI systems to best match device and infrastructure under test.

Cadence Compiler

SDK Birds Eye View 1

Firmware source

ELF

Image

Image

Signed Image

TopologyMetadata

Android HAL XML(*.xml)

/lib/firmware/intelSOF Linux*

Driver

rmbox / trace Tool

Crosstool-NG Tool Chain

rimage Tool

SOF Source Code

NuttX OS

SDK diagram – complete SDK

Topologyfile

(*.conf)

Audio DSP

SOF Firmware

m4ITT

alsaplg

Signing Tool

ALSA SoC Topology IPC/Mailbox

debugfs

Unsigned Image

ELFImage

OR

Cadence Emulator

SOF Firmware

debug

Qemu DSP & Host Emulator

OR

Note: Docker* contains only the open source components

m4 topology file (*m4)

Tool

LegendSOF Source

3rd Party Tool SDK Binary Image

Config DataOptional

Zephyr OS

Xtos OS


Matlab

SDK Birds Eye View 2

Firmware source

ELF

Image

GNU Octave

Alsa kcontrol

Algorithm Performance

Criteria


Coefficients


Tool

LegendSOF Source


Config DataOptional

SOF Linux* Driver

Audio DSP

SOF Firmware

IPC/Mailbox

Firmware source

SOF Test BenchTest Algorithm.


Firmware Overview Architecture agnostic implementation in C with some assembler.

Permissive BSD/MIT licensed code.

Build time selection of features based on target (moving to use Kconfig).

Scales down to ~40kB footprint (text, data and bss).

Support for user defined audio processing pipeline topologies.

Dynamically loaded pipelines at runtime.

Statically stitched into image for host less operation.

Allow general compute and signal processing. Not just an audio FW.

Image generation tools provided to:

Convert ELF to device specific firmware image formats.

Sign firmware images with PKCS#1.5

Firmware ArchitectureSOF Audio Components

mixer

mux

splitter

volume

EQ

SRC

ASRC

buffer queue

tone

speaker protection

AEC/NS

Generic Micro Kernel (XTOS, Zephyr, FreeRTOS, NuttX options)

boot

IRQs

timer

msg queues

schd

threads

work queues

mutex

atomic services

exceptions

memory

semaphores

Platform DriversDMA

DMIC

SSP

HDA

GNA

USB

I2C

SPI

UART GPIO

Audio Tasks

Configurable pipelines

IPC

DSP or Discrete ChipX86 or ARM

Host

ControlData

Linux* ASoCAudio Driver

RTOS/HV Audio Driver

SOF HW/IPC driver

2 of 2 Firmware diagram – complete diagram *Other names and brands may be claimed as the property of others.

Driver Overview Architecture agnostic.

Dual GPL/BSD licensed code.

Hardware and physical IO abstraction, so can work equally well with local MMIO and remote SPI based DSPs.

Supports virtualization through VirtIO backend and frontends (using ACRN*).

Driver has no hard coded assumptions about DSP pipelines or components.

Topologies loaded from FS.

Component configurations can be runtime updated via ALSA kcontrols.


Driver Architecture

DSP or Discrete

ChipIPC

Audio Data

ASoC Machine Driver codec

integrationboard

integration HW config

ASoC PCM Drivertopology PCMs Kcontrols

Generic IPC Drivermixer stream PM

DSP Platform Driver

DAPM

pipeline

doorbell mailbox IRQ

IO PMcode loader

2 of 2 Driver diagram – complete diagram

Physical IO layer can currently beeither PCI, SPI, MMIO or VirtIO.

Pipeline Architecture A pipeline is a set of audio processing components and buffers that share

common scheduling.

Pipelines can currently be defined using M4 in a few lines of code. Plans for GUI.

TopologyBinary

Topologyfile

(*.conf)

alsaplg


M4

EQ coefficients loaded via ALSA binary kcontrols

Standard ALSA volume kcontrols

Continuous Integration - Travis

Firmware source

ELF

Image

Image

Signed Image


rimage Tool

SOF Source Code


Signing Tool

UnsignedImage

ELFImage

Qemu DSP Emulator


Tool

LegendSOF Source


Config DataOptional

SOF Slim Driver

Topologyfile

(*.conf)

alsaplg


Travis CIGithub PR Github

Travis CI &


Continuous Integration - 0day

Firmware source

ELF

Image

Image

Signed Image


rimage Tool

SOF Source Code


Signing Tool

UnsignedImage

ELFImage

Tool

LegendSOF Source


Config DataOptional

SOF Driver

Topologyfile

(*.conf)

alsaplg


Travis CIGithub PR Github

0day

alsabat

SOF Platforms

Vendor Platform DSP Status

Generic Host PC Host PC Testbench.Qemu Host PC Qemu targets Currently only has support for xtensa based devices.Intel Baytrail,

Cherrytrail, Brasswell

1 * Xtensa HiFi2EP Upstream, support for upto 6 I2S DAIs.

Intel Haswell, Broadwell

1 * Xtensa HiFi2EP Upstream, need I2S integration for some codecs.

Intel Apollolake, Geminilake

2 * Xtensa HiFi3 Upstream, support for I2S, DMIC, HDMI and HDA.

Intel Cannonlake, Whiskylake

4 * Xtensa Hifi4 Upstream, support for I2S, DMIC, HDMI and HDA.

Intel Suecreek HAT on Rasberry PI

2 * Xtensa HiFi4 Upstream core, boot over SPI WiP.

Xiaomi ARM M4 CEVA Teaklite DSP Upstreaming WiP.

SOF Processing Components (open source)

Name Generic C available

SIMD support Status

Volume Yes Xtensa HiFiEP, HiFi3 Upstream, can process 1 – 8 channels, 16, 24 and 32bit formats.

SRC Yes Xtensa HiFiEP, HiFi3 Upstream. 24 and 32bit formats. FIR polyphase optimized M/N and M/N x O/P fractional rate conversions.

Configuration tools upstream.FIR Yes Xtensa HiFiEP, HiFi3 16/24/32 bit formats, up to 192 taps.

Configuration tools upstream.IIR Yes Planned 16/24/32 bit formats, up to 11 biquads or 22th order.

Configuration tools upstream.

2015

Driver open sourcedPlatforms: Apollo Lake, Haswell, Broadwell

Features: BSD/GPL driver, topologies

Driver upstreamPlatforms: Cannon Lake, Ice Lake,

Sue Creek, TeakLiteFeatures: Docker* build, SOF on host

Became Linux* Foundation project

Firmware open sourcedPlatforms: Cherry Trail and Braswell

Features: Qemu host

Platforms: Bay TrailFeatures: Qemu Xtensa

SOF 1.0 public release

Inception

2016 2017 2018

HistoryYear ending


SOF Future Plans

Feature Status

GDB stub and tunnel.

Initial firmware and kernel code in developer repository. Upstream in 2019.

RPMSG Code complete, now upstreaming to SOF and Linux.*

Loadable firmware modules

Initial code in github, still WiP. Upstream in 2019.

CMSIS RTOSAbstraction layer

Work in progress.


Supporting Organizations

Xiaomi


Firmware Preference Linux* already has a preference for open source drivers over closed source drivers.

Should this preference also be extended to open source firmware in the future?

Selection criteria would also have to consider:

Features.

Stability.

Portability.

Some open source firmware could still contain binary blobs.

Regulatory reasons (e.g. comply with FCC regulations).

3rd party IP (e.g. some audio processing algorithms).


Q & A

www.sofproject.com https://github.com/thesofproject

Visit the booth for demo.

Qemu Heterogeneous Virtualization

Baytrail DSP Architecture

SSP0

to/fromIOSF2OCP

bridge

JTAGPIF2AXI

JTAG

Interrupts(from shim)

JTAGTrace Logic

Interrupt

HiFi EPCore

EXTI/F PIF

HiFi EP Core

LPEShim

Control & Config signals

AUDIO

SMX

Instruction RAM

Data RAM

LPEDMA_01

32 bit OCP Slave

LPEDMA_00

64 bit AXI Master and Slave

OCPMasterReads

OCP Master Writes

32 bit OCP Slave

OCPMasterReads

OCP Master Writes

Data Cache

Instruction Cache

4kB RFMailbox

OCP (Upstream)

32-bit APB Slave

M/NCLK

SSP132-bit APB Slave

M/NCLK

SSP232-bit APB Slave

M/NCLKSource:

Tangier

VLV

Sonics

Xtensa HiFi 2EP core

96kB Instruction RAM

168kB Data RAM

2 * DMACs

3 * I2S ports

PCI device from host OS

Firmware and host share

SHIM registers

Mailbox memory Must be shared with IA and XtensaVMs

Heterogeneous Virtualization

Firmware must be debugged alongside driver.

Qemu used to virtualize drivers and firmware together.

Host side almost real time.

DSP side emulated.Codec driver

Qemu IO Bridge

IO, DMA IRQs

FW SSP driver

X86 Qemu/KVM

Guest OS *

Audio DSP driver FW DMA driver

Xtensa Qemu

SOF Firmware

FW Audio mixerAudio Userspace

VM #0 VM #1

x86 GDBOn host

xtensa GDB

On host

Heterogeneous IO bridging

PCI BAR 0 /dev/shmX86 Qemu/KVM

Guest OS *

Xtensa Qemu

SOF Firmware

VM #0 VM #1

x86 GDBOn host

xtensa GDB

On host

PCI BAR 1 /dev/shm

IRQ /dev/mqueue

Zero Copy Virtual Audio

Zero Copy Audio Virtualization

SOS buffer SG PHY pagesAudio Buffer (on

SOS)

Audio driver

Audio DMACs

Audio HW

Guest N

IPC virtio Qs

Shared audio buffers

Service OS

Audio Driver

SOS buffer PHY pages shared with guests.

SOS allocates SG PHY pages for guest audio buffers. Audio DMA copies directly from/to these buffers.

Legal notices and disclaimers• This document contains information on products, services and/or processes in development. All information provided here is

subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

• Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure.

• Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

• Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

• Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K.

• The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

• No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

• Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

• Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.

© 2018 Intel Corporation.

http://www.intel.com/performance

sound open firmware - github pages · 2020-06-05 · drivers are generic and not coupled to any...

Documents