xpds16: xenbedded: xen-based client virtualization for phones and tablets - chris patterson &...

153 Brooks Road, Rome, NY | 315.336.3306 | http://ainfosec.com

Xenbedded:Xen-based client virtualization for phones and tablets

Kyle TemkinChris PattersonAssured Information Security, Inc.

August 25th, 2016 // XenSummit 2016


Who are we?

Kyle Temkin‘Hardware Hacker’ and Security Researcher

Twitter: @ktemkin, Freenode: ktemkinE-mail: [email protected]

Chris PattersonSenior Computer Engineer

Freenode: cjp256E-mail: [email protected]


Why run Xen on Phones/Tablets?

First, and foremost: it’s cool!

} This is the ‘frontier’ of virtualiztion tech–there’s lots of new stuff to build!

Mobile virtualization also offers more pragmatic benefits:

} Separation of critical and non-critical tasks.

◦ Thnk Qubes or OpenXT for mobile: different VMs for different types of activity.

◦ Potential tools to deal with the whole ‘BYOD’ mess.

Pictured: Google Pixel CPhoto: Google Store

} Potential to extend benefits beyond phones/tablets; e.g. to devices with integrated touchscreens such as car media centers.


Isn’t ARM support enough?

Q: Xen already supports many ARM processors: shouldn’t it be able to run on most phones and tablets?

A: Almost. There are still a few things we need:

◦ Hardware support for ARM’s virtualization extensions.

◦ Bootloader support for launching software in hypervisor mode (HYP/EL2).

• It’s easy to find yourself ‘locked out’ of these modes!

First-StageBootloader(usuallycomeswithSoC;

ofteninROM)

Second-StageBootloader(e.g.u-boot,android

bootloader)

FinalPayload(Linux,Xen)

Alltoooften,theFSBLdropsdowntoSupervisorMode(PL1/EL1)... …leavingnowaybackintoPL2/EL2!


Landing in PL2/EL2

Q: What if the boot chain can’t land me in PL/EL2?

A: If you have to use that device, things will get… interesting.

◦ Best option: convince your vendor to support EL2 on unlocked devices.

◦ Last resort: dust off your hardware hacking skills!

• Samsung CID hacks (developer bootloader)

• Qualcomm TrustZone exploits (chain to get at HVBAR)

- CPU mode All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError, IRQ and FIQ). The CPU must be in either EL2 (RECOMMENDED in order to have access to the virtualisation extensions) or non-secure EL1.


Finding the Right Device

Finding a good phone or tablet to work with can be tricky.

You need a device that:

} Supports the ARM Virtualization Extensions, including a full-featured GIC.

} Provides an “unlockable” bootloader entered in NS-HYP or EL2.

You want a device that:

} Is supported by an open-source Linux variant (for dom0)

} Is closely approximated by an available evaluation board.

} Has an accessible UART “serial port”


Identify Devices with HYP/EL2 ‘Payloads’

If your device runs Linux 3.7+ (arm32) or Linux 4.3+ (arm64), checking for HYP/EL2 is easy:

Otherwise, you can compile a simple kernel module to check:

https://github.com/xenbedded/hyp-mode-checks

# dmesg | grep HYP –A 1 #arm32CPU: All CPU(s) started in HYP mode.CPU: Virtualization extensions available.

# dmesg | grep EL2 #arm64CPU: All CPU(s) started at EL2

# #Note: these only work bare-metal!


Fragmentation of Linux Baselines

Depending on the device, several Linux variants may be available, including mainline and per-device variants.

Not all Linux forks are created equal:

} If possible, select a device that has reasonable mainline support. (e.g. MSI Primo73)

} Failing that, you may want to look for a fork that deviates less from mainline:

◦ A recent kernel version is usually a good hint.

◦ Another good hint: look at the device tree for your target device. Are many of its compatiblestrings supported in mainline?

• Mainline

• Android Variants

• ChromiumOS variants

• linux4tegra

• linux-sunxi

• linux-exynos

• linux-xilinx

• linux-96boards

• … oh, my!


Approximating HW with Dev Boards

≈

Pixel Cimage source: Google Store

NVIDIA Jetson TX1image source: NVIDIA Dev Blogs

Pairing your device with an evaluation board can dramatically speed up development.

◦ It helps a lot to have a “known good” environment to test your hardware in. Is the issue in your code, or with the weird tablet bootloader? Compare and find out!

◦ If you can get one, a JTAG debugger make all the difference!


Finding UART Consoles

Most– but not all– phones and tablets have a UART surfaced somewhere. Friendlier devices have externally accessible UARTs; some devices require soldering.

Externally AccessibleInternally Accessible

The Xen-compatible Primo 73’s UART is accessible both directly on its PCB…

… or, with some configruation, routed through its internal microSD slot.

Embedded Linux and Android communities are a good first resource for UART pinouts: you’ll be surprised how often the community has documented how to get UART access!

Original image.

Credit: ssvb


While the Pixel C has no documented external UART, digging through some of its firmware reveals an internal USB-to-serial-adapter.

} When the USB-C connector’s CC1 and CC2 pins are pulled down with 5.1Ωresistors, the device enters ‘debug mode’, and the convert is connected as below:

} Naturally, we went a little overboard, and our intern (@c4757p) developed a full fledged open-source-hardware debug adapter for these ports:

◦ https://github.com/xenbedded/pixel-c-support/tree/master/hardware/debug_bridge_r2

Example: Pixel C UART


Getting Xen onto Your Device

Once you’ve selected a device, the next step is to get Xen to start.

A basic path forward might be:

} Boot a simple, custom hello-world kernel.

} Boot up the Xen kernel alone– get it to crash!

} Provide Xen with its required images, and see how things behave.


Booting a ‘Hello World’ Kernel

Many devices require their kernels to be encapsulated in payload-specific image formats.

} ChromiumOS: FIT image with specific contents, prefixed with signed header

} Many Android Devices: Android-specific boot.img format

} We’ve seen lots of variants on these formats!

Booting a “Hello World” image helps to verify that you have the particulars of your image format correct.

} Also a great time for a final check for HYP/EL2!

} Good starter example:https://github.com/andreiw/nexus9_demo


(Some People Just Want to)

Watch Xen Crash

Once you can boot an image on your particular device, try swapping out your ‘hello-world’ kernel for Xen.

} Be sure to build Xen with EARLYPRINTK set up for your board!

Launching Xen... - UART enabled –- CPU 00000000 booting –- Current EL 00000008 –- Xen starting at EL2 –- Zero BSS –- Setting up control registers –- Turning on paging –- Ready –(XEN)(XEN) **************************************** (XEN) Panic on CPU 0: (XEN) No valid device tree (XEN) ****************************************(XEN) (XEN) Reboot in five seconds...

= Victory!


Getting Xen its Images

Once you successfully have Xen crashing, you’re ready to try a real boot, which means passing in several necessary ‘images’.

This can be tricky. We expect our bootloader to load:

} The Xen kernel image, typically a zImage/Image;

} A flattened device tree (dtb) describing the target device;

} The dom0 kernel, typically also a Linux zImage/Image; and

} An optional ramdisk (initrd/initramfs).

… but most image formats won’t load all of these!

} We’ll need a next-stage bootloader to make our images available.

◦ u-boot is a popular choice.


Discharge Boot Adapter

We created Discharge, a simple image adapter for phones and tablets.

} Combines the images relevant to a Xen boot into the single-image formats expected by most consumer device loaders (Android/CrOS).

} ‘Faster’ and significantly lower-footprint than u-boot; but also significantly less powerful.

◦ If u-boot better fits your use case, use it!

Full MIT-licensed source code:

http://github.com/xenbedded/discharge


Xen on Your SoC

If things seem broken at first, don’t despair!

You may have to deal with quirks for your particular SoC– or with Xen behaviors (bugs?) that your new configuration has unmasked.

} This is where things can get complicated. We won’t provide a general method for moving forward (this talk is too short!), but we will provide examples.

} In particular, see Julien Grall’s slides from Porting Xen on ARM to a new SoC.


Example: Tegra Legacy Interrupt Controller

On Tegra systems, a set of Legacy Interrupt Controllers must be programmed in addition to the core Generic Interrupt Controller (GIC).

} Idea is straightforward: implement Xen support for the Tegra LIC.Figuring out the best way to do this: less straightforward..

} One working implementation is available online:

We’re currently working to get a proper solution mainlined!

https://github.com/xenbedded/xen/tree/tegra_support_v2

TegraLICLegacyInterruptControllers

GICDistributor

GICCPUInterface

GICCPUInterface

CPU0

CPU1

SPI[n:32] SPI IRQ

IRQ

(Very)SimplifiedViewofTegraInterruptRoutingTheTegraLIC,bydefault,gatesallSharedinterrupts.


Client Virtualization:

New Peripheral Support for Xen

Tablets and phones present a huge variety of new peripherals we may want to virtualize for guest domains:

Virtualizing these devices require three components:

} A device driver running in domain zero; ideally provided by the BSP distribution.

} A backend driver to (e.g.) multiplex access to the physical hardware; and

} A frontend driver to deliver events to the guest kernel.

• Display/GPUs

• Light Sensors

• Flash LEDs

• Touch Sensors

• Cameras

• Wireless/Cellular

• Inertial Sensors

• Fingerprint Sensors

• … and more every day.


Display and Input Multiplexing

To provide our guests with a high-performance display capability, our group has worked to develop a new display technology for Xen.

} Provides a near-native experience for both mobile and desktop devices, including backend and frontend drivers for multi-touch and multi-monitor support.

} Includes a full input stack, including PV drivers.

} We won’t go into detail here. This is covered in full by the next talk in this room (“Display Handler”).

Baseline: X11 and VNC

Display Handler: Native Look


Multi-Touch with openxt-kbdfront

Multi-touch input is handled by an extended version of xen-kbdfront.

} Entirely open; source is GPLv2 and available on our GitHub. (http://github.com/xenbedded/openxt-kbdfront)

} Backwards-compatible with xen-kbd, and thus works with existing backends.

} We may submit our changes as patches to xen-kbdfront, if well received.


Goal: Android Support

Support for Android guests is not far off—basic functionality is working on x86—but work towards this support is still underway.

Challenges still to be solved:

} Blocker: Modern Android versions require* hardware-accelerated video;so we’ll need to implement a solution for multiplexing access to the host GPU.

} Soft req: Android isn’t as useful on a tablet without mobile-centric power mgmt., including intelligent use of host sleep states, paralleling Android’s wakelocks.

} Soft req: Many android applications require services from TrustZone ‘Applets’ (Trustlets), but we now need a guest-aware TrustZone, or equivalent.

*At least, according to Google. Software rendering may be an alternative, but doesn’t work as-is, and likely isn’t worth pursuing for performance reasons.


Android: Accelerated Graphics

Android’s SurfaceFlinger compositor– and many applications– require OpenGL support.

Potential paths forward:

} On x86, HVM guests can run Android under the VirGL virtualized GPU, which is heavily tied to its QEMU implementation.

◦ See Rob Herring’s Android-on-QEMU effort.

◦ To support ARM PVH guests, we’d need to develop a PV transport for the VirGL.

} Raw “API Sharing” solutions – such as the emugllibrary used by the Android emulator – also offer potential solutions.


Android: TrustZone Implications

In the current implementation, only one trusted enclave is available.

} This is definitely not an optimal architecture for per-guest services, such as secure password stores, DRM, and cryptographic services.

◦ As a result, we disable guest SMCs: and lose these capabilities altogether! (“No Netflix?”)

} Consider the implications of existing TrustZone escalations (e.g. the QSEE bug)—do you really want your guests interacting with TrustZone services?

} Potential solution: leverage hypervisor separation to create a service-VMtrusted execution environment (TEE) for per-guest services.

dom0 dom1 dom2

SingleTZ-EnabledEnclaveCoreSystemServices

TEEServiceVMdom0

TEEServiceVMdom1

TEEServiceVMdom2

dom0 dom1 dom2

SingleTZ-EnabledEnclaveUnused?


Questions and Demo

xpds16: xenbedded: xen-based client virtualization for phones and tablets - chris patterson &...

Technology