multiple agl profiles consolidation with l4re …l4re: consolidating multiple agl profiles march...

Multiple AGL profiles consolidationwith L4RE Hypervisor

By Iot.bzh & KernKonzept

AMM @ Tokyo, March 19

Johann [email protected]

Jakub Jermář[email protected]

March 2019 L4RE: consolidating multiple AGL profiles 2

IoT.bzh team

● Fully dedicated to AGL● https://iot.bzh/en/● http://github.com/iotbzh

Workshop in Lorient

LORIENT

vannes

https://iot.bzh/en/

http://github.com/iotbzh

3L4RE: consolidating multiple AGL profiles March 2019

Typical AGL architecture

● One board, one kernel :

LINUX

APPLICATIONS

SERVICES low-can

CAN

NOTE : for this PoC runs on Renesas Gen3 hardware (H3, M3, Salvator X, ...)


● CAN subsystem isolation using L4Re hypervision:

● RT_CAN instance runs on a RAM disk, IVI on mmc (SD card).

Targeted Architecture

L4Re hypervision

LINUX

APPLICATIONS

SERVICES low-can

LINUX-RT

low-can

CAN

host-IVI

host-RT_CAN

virtio-sock


PoC Goals● Purpose:

– CAN isolation leveraging hypervisor

– Isolate priority service from IVI

– Possible certification of a critical subsystem

● Perspective:,– Better availability of CAN statistics,

– Possible certification of RT_CAN instance.

– Binder for smaller trusted OS (certification)● NewGen RTOS: Zephyr, Mbed● Legacy RTOS: FreeRTOS, VxWorks, QNX● L4Re native server module, native trust zone service, ..

– Binder for other Linux flavours● Android● Adaptive Autosar


Communication between AGL instances

● L4Re provides 2 solutions outside TCP/IP.– Virtio-console:

● No kernel patch needed,● No AGL services change required at source code (config update only),● Require a “socket proxy” (proxso) to emulate sockets over virtio-console.

– L4Re’s VSOCKs:● Impose to patch the kernel to add the virtio_sock module,● Change required to application,● Applications/Services can then rely directly on L4 virtio sockets.


L4Re’s virtio vsocks

4.14.35 kernel + L4 patch

L4 VSOCKS

4.14.35 kernel + L4 patch

L4 VSOCKS

L4Re

UVMM UVMM

CANCAN

App

App

App

App agl-service-can-low-level

Renesas R-Car Gen 3


L4Re virtio console

4.14.35 kernel

virtio-console

4.14.35 kernel

virtio-console

L4Re

UVMM UVMM

CAN

proxso proxso

CAN

App

App App

Appagl-service-can-low-level

Renesas R-Car Gen 3


Not as simple as it should

● Multiple problems, related to :– Actual hardware / firmware (far simpler with QEMU)

– Yocto / AGL

– L4Re● No hard showstopper but:

– Multiple room for improvement (virtio performance/integration)

– Integration cost is far from being negligible

– Portabilty/Maintainability does not come for free


Roadmap of the PoC Around 10 weeks FTE

● Learn L4Re (configuration, build process, …)● Hack binder to support virtio● Generate AGL images (from yocto):

– IVI : ● close to an agl-demo-platform standard image, but without CAN capabilities (driver options disabled

in kernel .config)

– RT_CAN : based on core-image-minimal target in yocto● Features : agl-netboot, agl-appfw-smack ● Apply PREMPT_RT kernel patch● Disable as much options as possible in kernel .config (except CAN)

– Apply L4_VSOCKS kernel patch on both images● Assemble everything in a u-boot image● Run it on a RCar Gen3 board


UBOOT Hard-Firmware issues

● U-Boot : binary (kernel) loaded in memory must be <8MiB (default value), or else u-boot complains.– Size can be configured (CONFIG_SYS_BOOTM_LEN), but

ultimately, the problem is u-boot needs to copy the binary to the “entry point” address declared in the header, them jump to this address (this binary is NOT rellocatable).

– Better solution : load the binary “in place” (i.e. in such a way that no copy is needed). Target address is then:

● entry_point – header_length (u-boot header is 64 bytes)● 0x51 00 00 00 – 0x40 == 0x50 FF FF C0● ext4load 0:1 0x50ffffc0 ${uImage}


RAM Hard-Firmware Issues

● Memory Mapping : some RAM zones are used by “obscure” features like “CMA for lossy comp” (48MiB from 0x54 00 00 00 to 0x57 00 00 00),

● Another zone has been detected further, but not its exact boudaries, nor the feature that uses it,

● Bytes in those zones are “corrupted”, ● Corruption is detected by u-boot (CRC error),● As for now, we believe it’s safe to load uImage after 0x5A 00 00 00● Need to change the entry point address, which is configuration in L4Re


L4Re / Hardware issues (Firmware)

● Problem with mapping the A53 in L4Re startup script.– Original setup :

● IVI: 2 A57 + 2 A53● RT_CAN + L4Re: 2 A53

– Quick hotfix : ● Deploy VMs only on two A57, one VM on each. ● Obviously not a valid option for the IVI instance.

● Nota: Seems to be fixed with newer firmware versions


L4Re / Hardware issues (RAMDISK)

● Problem to deploy ramdisks big files from uImage– Needed because we cannot share access to the SD card (L4Re)

– Related to memory mapping (RAM banks that are not contiguous).

– KernKonzept managed to deploy the 240MiB ramdisk for RT_CAN.

– IoT.bzh approach implements 2nd partition on the SD card. One served by the IVI instance over NBD (agl-netboot):

● NOT an ideal architecture:● RT_CAN can’t boot/work if IVI is down● Somewhat “breaks” isolation


VIRTIO VSOCK patch issue

● Patch L4 VSOCK : only applies on kernels 4.18 and 4.19 – Not applicable out-of-the-box on our 4.14.35 kernel (few API

changes, ...),

– Managed to backport, but still generated a Kernel Oops when using VSOCKs.

– Not a fully standard implementation


Yocto / AGL (1)

● core-image-minimal is still ~520MiB (!!)– After manually stripping down, still ~240MiB…

● 90 packages removed using packaging system● 3000+ files removed manually (including dnf and dependencies)

– A really “minimal” target (BSP+AGL appfw) should be < 128MiB (<64MiB).● Need to define a restricted list of packages ● Each package (at least in this list) should support a minimal installation mode (no

debug symbols, …)● Customization of packages (for instance, no need to have a full HW in udev)● Kernel optimisation, install only needed modules, ...


Preemp_RT for Yocto / AGL

● PREEMPT_RT patch.– Today : nothing to enable / apply it in yocto,

– Very tricky, no version for kernel 4.14.35,

– Renesas kernel is not “vanilla”, which adds difficulties.

Ref: Thierry Dresden presentation on AGL and Real Time https://iot.bzh/en/publications/32-2018/78-agl-real-time-architecture-options-for-critical-profiles

https://iot.bzh/en/publications/32-2018/78-agl-real-time-architecture-options-for-critical-profiles

March 2019

M I C R O K E R N E L M A D E I N G E R M A N Y

L4RE: consolidating multiple AGL profiles

18www.kernkonzept.com

L4Re HypervisorJakub Jermář

Kernkonzept

www.kernkonzept.com

March 2019M I C R O K E R N E L M A D E I N G E R M A N Y


19

L4Re Highlights Mature technology 20+ years of operating systems research and application in industry TU Dresden, Germany Kernkonzept Small & secure Orders of magnitude smaller TCB than Linux

Secure capability-based component architecture Multiplatform 32-bit/64-bit x ARM/MIPS/x86

L4Re Microkernel

VM2

sigma0

moe

uvmm

ned

io

VM1

uvmm

~31 kloc

~1 kloc

~5 kloc

~2.5 kloc

~20 kloc

~11 kloc~3.6 kloc

2+ mloc

l4re libs ~35 klocuclibc

libstdc++

www.kernkonzept.com



20

L4Re Highlights Virtualization Full-virtualization x paravirtualization Hardware pass-through Per-guest untrusted user-level VMM VIRTIO P2P link components Optee scenarios Real-time Multi-server Microkernel L4Re native applications POSIX subset

www.kernkonzept.com



21

L4Re Highlights Power management features Suspend to RAM Enable/disable CPUs Dual-licensing model Open source, CLA Commercial licensing

www.kernkonzept.com



22

L4Re Usecases In automotive EB corbos Hypervisor AGL on L4Re Not only in automotive Security products Intrusion detection systems Safety products Consumer electronics Smart factories IoT

www.kernkonzept.com



23

Open source GitHub https://github.com/kernkonzept Snapshots http://l4re.org/download/snapshots/ Tutorials https://github.com/kernkonzept/manifest/wiki#tutorials QEMU Advent Calendar 2018 http://www.qemu-advent-calendar.org/2018/#day-21 FOSDEM Microkernel devroom

https://github.com/kernkonzept

http://l4re.org/download/snapshots/

https://github.com/kernkonzept/manifest/wiki#tutorials

http://www.qemu-advent-calendar.org/2018/#day-21

www.kernkonzept.com



24

AGL on L4Re Combining AGL and L4Re expertise Over a distance of 1300km Different user experience from x86 desktop virtualization Lots of configuration due to HW pass-through Platforms’ variability Detailed knowledge of the target platform needed Familiarity with the guest kernel is a plus

www.kernkonzept.com



25

Hardware Challenges Issues with the M3/H3 boards Failures to netboot a large image over TFTP All Cortex-A53 CPUs disabled in older M3 firmware Virtualization not enabled on one H3 board Perfect HW partitioning vs. HW dependencies Shared system, reset and clock controllers

www.kernkonzept.com



26

Software Challenges How does the CAN-only VM get its rootfs? AGL image size vs. available memory agl-minimal-image ~520MB Guest drivers ignoring DT configuration

MM_LOSSY_SHARED_MEM_ADDR

www.kernkonzept.com



27

Come to see our Under the Hood of AGL Virtualization on L4Re BoF!


Conclusions

● Good collaboration with KernKonzept● About L4Re :

– works well, despite some limitations in actual implementation

– Ultra-lightweight code base (and good quality, but some core “legacy” components, like Fiasco kernel, sigma0 pager, etc. would deserve a little lifting to take advantage of modern C++ features)

– Significant learning curve when no/limited knowledge on “microkernels”

– A fully compliant VIRTIO VSOCK implementation would help● About Virtualisation

– It open new doors but it’s not a magic wand people dream of

– Integration cost and time should not be ignore

– Add an extra layer to debug that might impose unwanted cross team collaboration

multiple agl profiles consolidation with l4re …l4re: consolidating multiple agl profiles march...

Documents