Dealing with Hardware Heterogeneity Using a Virtualization Framework Tailored to ARM Based Embedded Systems
Prof. Daniel Rossier, PhD
HEIG-VDInstitut REDS, Reconfigurable & Embedded Digital Systemsrte Cheseaux 1, 1400 Yverdon-les-Bainshttp://www.reds.ch/
REDS@HEIG-VD
Outline• Background
• Overview of EmbeddedXEN
• Protection & Memory Isolation
• Domain Interactions
• Device Heterogeneity
• Conclusions & Future Work
2
REDS@HEIG-VD
Background
• HEIG-VD • University of Applied Sciences in Yverdon, Switzerland (CH)
• Reconfigurable Embedded Digital System Institute
Hardware Design, FPGA
Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc.
• Applied Research & Development• ARM based Microcontrolers (v4, v5, v6, v7)
• Low-level interactions between computing cores (CPU, DSP, FPGA, etc.)
• Towards Cortex-A15 HVM
3
REDS@HEIG-VD4
Background• Embedded Virtualization on ARM
• XEN: free & easy access to a stable & evolving hypervisor source code
• Early port of XEN on ARM in 2007 in the context of a Diploma Project
• Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization framework
• Re-use of Linux file organization & build system (Makefiles, scripts, …)
• Focus on realtime aspects (Linux/Xenomai as RTOS)
• Different sources of inspiration• "Fast Secure Virtualization for the ARM Platform", Daniel
Ferstay, Master Thesis, The University of British Columbia, 2006
• XEN ARM port, George G. Davis, MontaVista, 2007
• Secure Xen on ARM Project, Sang-bum SUH, Samsung, 2007
REDS@HEIG-VD
Background
• Focus on heterogeneity of embedded devices• Idea to re-use OS & applications from old devices to recent
devices
• Time-to-market migration to new hardware generation
• Dealing with various cross-compiled binaries (ARM v5-v7)
• Dealing with different peripherals
• Less emphasis on security aspects
• Publicly available• https://sourceforge.net/projects/embeddedxen
5
REDS@HEIG-VD6
Background• Hardware constraints
• Low latency, reactivity (response time)
• ARM cores do not support virtualization mechanisms
not easy to deal with various levels of execution modes
• Para-virtualization remains attractive• About 30 files to be (slightly) adapted
• Low execution overhead
• Efficient processing of downcalls/upcalls with support of domain interactions
Physical interrupts are quickly processed in dom0.
REDS@HEIG-VD7
Overview of EmbeddedXEN
• Limited use of hypercalls
• ARM execution modes (USR/SVC) – pseudo-user mode with double stack handling / downcalls & upcalls are simple jumps to specific (callback) addresses
• No subtle use of domain access control (ARM DACR) to protect memory
• Memory sharing facilitated between primary and secondary OS
Domain Creation and
Setup
Secondary Guest OS known as DomU
VCPU sched_flipScheduler
Migration
Manager
Hardware
Primary Guest OS known as Dom0
EmbeddedXEN Hypervisor Hypercalls
Handling
Xen-guest(pv)
Xen-guest(pv)
PrioritizedUpcalls
Handling
- Full privilege- DomU original
drivers- Frontend drivers
- Full privilege- Dom0 original drivers- Backend drivers- Native drivers
REDS@HEIG-VD
Overview of EmbeddedXEN• Linux-like source tree and build system (Makefiles)
8
embeddedxen
hypervisor-4.0.2
linux-2.6.32-dom0
arch/arm
linux-2.6.26-domU
xen-guest
xen-guest
tools
xen
arch/arm
environnement.conf
include
xen-guest
include
xen-guest
xen-guest
mach-mx35
arch
arm
mach-msm
mach-msm
xenbus
core includ
earch
arm
mach-mx35
mx35_fab4.c
kernel
mm
console
include
xen-guest
Hypervisor
Commonpart
Primary OS
SecondaryOS
Config & build system
REDS@HEIG-VD9
Overview of EmbeddedXEN
• Single binary (multi-kernel) image
• Automatic parsing & image relocation during hypervisor bootstrap
uImage
Boot Head
EmbeddedXEN
Hypervisor
DOM-0EOD
0 (Dom-0)
DOM-UEOD
1 (Dom-U)
vmlinux.dom0 vmlinux.domUvmlinux
Dom-0 Filesystem
Dom-U Filesystem
/home/root/squeezeos.rootfs.img: • Stored separately in sdcard, for example
REDS@HEIG-VD
Protection & Memory Isolation• Memory isolation between domains relies on different
address space isolation.• No further advanced mechanisms to protect hypervisor and guest memory
• The guest OS kernel runs at the same privilege as the hypervisor.
• Each domain receives its own (contiguous) physical memory region during domain set up.
• No pagination of the kernel linear address space is performed.
• Paravirt of memory management is kept minimal.
• Guest OS kernel has access to the whole memory.• No protection mechanism for strong isolation of VM (but is it really necessary?)
10
REDS@HEIG-VD
Protection & Memory Isolation
• Virtual & Physical Memory Layouts
11
3GiB
PAGE_OFFSET 0xC0000000
0x00000000
Initial Virtual Address Space Physical RAM
boot allocator
XEN heap (2 MiB)
frame table
I/OVMALLOC_END
Interrupt Vectors0xffff0000
0xFF000000
Hypervisor
dom0
Hypervisor code
1st-level page table
domU
dom0domU
Linearmapping
Linearmapping
3GiB
dom0
boot allocator
XEN heap (2 MiB)
frame table
I/O
Interrupt Vectors
Hypervisor code
1st-level page table
dom0 Virtual Address Space
REDS@HEIG-VD
Domain Interactions• Virtualization of peripherals in EmbeddedXEN is quite
similar to the existing mechanisms in XEN.• Driver split with frontend & backend drivers
• Communication with xenbus
• Use of grant tables for sharing/copying pages between domains
• However, a revisited (simplified) implementation of these mechanisms have been achieved in EmbeddedXEN.
• XEN store is dynamically allocated at boot time of guest OSes.
• No user space tools are required to manage XEN store entries or peripherals configs.
• Hotplugs & dynamic configs of peripherals are less relevant to embedded systems.
12
REDS@HEIG-VD
Domain Interactions• domU is passed information during bootstrap under control
of dom0.
13
xen-guestsubsys
Hypervisor
OS Subsys
start_info->store_mfnstart_info->store_evtchn
domU
xenstore
xen-guest subsysxenbus
thread
xenbus thread
xenstore thread
frontend drivers
backend drivers
event_channel C/D
event_channel A/B
Page Dom0
prod
cons
Page DomU
via unpause_domU(store_mfn, store_evtchn)
prod
cons
OS Subsys
REDS@HEIG-VD14
Domain Interactions• Grant tables are used in a different way
• Shared pages are possible only in the vmalloc'd area.
• Kernel linear addresses are not shareable; contents needs to be copied using temporary mappings.
3GiB
0xC0000000
0x00000000
RAM
VMALLOC_END
Hypervisor0xFF000000
Hypervisor
dom0
domU
VMALLOC_START
3GiB
Hypervisor
0xC0000000
0x00000000
VMALLOC_END
0xFF000000
VMALLOC_STARTgnttab(dom0)
gnttab(dom0)
mem_map_foreign(domU)
gnttab(dom0) gnttab_foreign[0]
gnttab
mem_map_foreign mem_map_foreign(dom0)
shared pages
Dom0 DomU
gnttab(domU) gnttabgnttab(domU)gnttab_foreign[1]
gnttab(domU)
shared pages
mem_map_foreign
has references to
has references to
has references to
has references to
REDS@HEIG-VD
Device Heterogeneity• Different levels of heterogeneity
• At CPU level: various instructions sets (locks, cache, etc.), various PTEs (MMU) flags, various co-processors, etc.
Compatibility ensured via hypercalls
• At peripherals level: not the same hardware
Compatibility ensured via backend driver processing
15
hypervisor-4.0.2
arch/arm
xen
arch/arm
mach-mx35
mach-msm
kernel
mm
include
xen-guest
mm
built-in.o cache-v6.S cache-v7.S Kconfig Makefile mm.h proc-macros.S tlb-v6.S tlb-v7.S
REDS@HEIG-VD
Device Heterogeneity• Example of ARMv6 running on ARMv7 CPU (iMX35 -> HTC
Desire HD)
16
linux-2.6.26-domU/arch/arm/mm/cache-v6.S:ENTRY(v6_flush_kern_cache_all) mov r0, #0#ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, 0 @ D cache clean+invalidate#ifdef CONFIG_SMP mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate#else b v6_icache_inval_all#endif#else mcr p15, 0, r0, c7, c15, 0 @ Cache clean+invalidate#endif
mov pc, lr
linux-2.6.26-domU/xen-guest/hypervisor.c:void xen_flush_kern_cache_all(void){ struct mmuext_op op;
op.cmd = MMUEXT_FLUSH_CACHE; HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF);}
hypervisor-4.0.2/arch/arm/mm/cache_v7.S:ENTRY(xen_flush_kern_cache_all) stmfd sp!, {r4-r5, r7, r9-r11, lr} bl xen_flush_dcache_all @ much more complex!! mov r0, #0 mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate ldmfd sp!, {r4-r5, r7, r9-r11, lr} mov pc, lrENDPROC(xen_flush_kern_cache_all)
linux-2.6.26-domU/arch/arm/mm/cache-v6.S:.extern xen_flush_kern_cache_allENTRY(v6_flush_kern_cache_all) b xen_flush_kern_cache_all#if 0 /* paravirt */ mov r0, #0
…
mov pc, lr#endif /* 0 */
Original version Paravirt version
REDS@HEIG-VD
Device Heterogeneity• Example of framebuffer device heterogeneity
• Configuration of framebuffer is retrieved from Dom0 via xenbus
17
Dom0 DomUxen-guest
xenfb backend xenfb frontend
User space applications(/dev/fb0)
framebuffercore
User space applications(/dev/fb0)
framebuffercore
dom0 fb memory(allocated by dom0)
fb_event_dom_register fb_event_dom_switch
Query framebuffer params
xenbus thread
domU fb memory(allocated by domU)
FB config is retrieved from dom0
device-specific
framebuffermsmfb_pan_update()
xen-guest
xenstore
xenbus thread
REDS@HEIG-VD
DomUDom0xen-guest
Device Heterogeneity• Example of audio device heterogeneity
• DomU audio buffers are accessed from Dom0 via shared pages.
18
xen-audio backendxen-audio frontend
Sound device (headset, speakers, etc.)
xenstore
User space applications
/dev/msm_pcm_out - or -/dev/pcm/snd/pcm0c0d0p
xenbus thread
User space applications
/dev/pcm/snd/pcm0c0d0p
pcm subsystem
arch-specific audio driver
pcm subsystem
xenvaud_pcm_start xenvaud_pcm_stop
Audio buffers allocated in domU
xen-guest
xenstore
xenbus thread
REDS@HEIG-VD
Conclusions & Future Work• EmbeddedXEN is an embedded virtualization framework
which puts emphasis on efficient and heterogeneous hardware.
• Application environments can be re-used "as such" on modern platforms (Android-based for example) taking advantage of last generation hardware.
• EmbeddedXEN relies on the main principles of XEN, with a revisited lightweight, but less secure architecture.
• A single multikernel binary image, easy to deploy on the target platform without additional tools, makes EmbeddedXEN well tailored to embedded systems.
19
REDS@HEIG-VD
Conclusions & Future Work
• Further investigation projects:
• Elaboration of a domU using a graphical desktop for user applications
• Support of multicore ARM CPUs (cortex-A9, cortex-A15)
• Live migration of domU using remote NFS-filesystem (migration within a cloud)
• Support of hard realtime OS (RTEMS-paravirt)
20
