11th November 2014 - Andrea Gallo

Enabling Server Class Workloadsfor ARM Servers

ARM Progress in Servers

ARM-based servers on track▪ 16 licenses now signed for server applications▪ 2013: Chip development▪ 2014: Software optimization accelerates▪ 2015: First production systems▪ 2018: ARM expects 10-15% share

of total server market

Data Center Workloads are Changing

Today Next 3 Years 5 Years +

Largediscrete

tasks

● Linaro is a collaborative software engineering company funded by members

● Instead of duplicating effort for common open source software, competitors share development costs and software is built once

● The work is carried out in the open, tested and then upstreamed into the relevant open source projects: kernel.org, Gnu, Fedora, Ubuntu, OpenJDK etc.

Linaro Overview

Linaro Organization

● Founded 2010

● Now 30 members

● $50M annual revenue

● >200 OSS engineers

● Global footprint

Technical Steering Committee (TSC)

Offi

ce o

f the

CTO

(OC

TO)

Enterprise Group (LEG)16 companies

Digital Home Group (LHG)8 companies

Mobile Group (LMG)10 companies

Networking Group (LNG)12 companies

Core Development

Kernel

Power Management

Security

Virtualization

Product Technology

LAVA Software Linaro Stable Kernel (LSK) Toolchain

Platform Engineering

Builds & Baselines

Systems

LAVA Lab

QA Services

MemberServices

Club/Core Services

Linaro Developer Technical Support

(LDTS)

Field Engineering

Confidential

CORE

CLUB

GROUP

COMMUNITY

● Linux 3.17 October 2014 ● 12,353 total patch sets

in this release

● 180 known companies contributed

● Linaro is #4 company contributor; in top 5 for last 8 kernel versions (since 3.10)

Linux Kernel Upstreaming

Source: http://www.remword.com/kps_result/3.17_whole.html

Top Linux 3.17 Patchset Contributors by Company

1 Unknown/Hobbyists 3705 (30.00%)

2 Intel 1261 (10.21%)

3 Red Hat 980 (7.93%)

4 Samsung 491 (3.97%)

5 Linaro 412 (3.34%)

6 Vision Engraving & Routing 393 (3.18%)

7 Novell 326 (2.64%)

8 Texas Instruments 306 (2.48%)

9 Renesas Electronics 263 (2.13%)

10 Google 227 (1.84%)

Linaro Projects

Confidential

● OSS for the digital home ● W3C EME Secure

Media playback for RDK and Android

● Middleware and user-space stack

● DRM, DLNA, CVP-2, HTML5

● LSK kernel version for STB/IPTV

● Common media frameworks

● OSS for mobile devices● Android 64 bit, “L”● big.LITTLE power

management● QEMU based Android

development● 64-bit Chromium browser

● Support members, ARM and Google Android development

● OSS for networking ● Real Time Support● Virtualization● Core isolation● OpenDataPlane (ODP)● Big-endian legacy support

● ODP cross-platform support for SoC accelerators

● OSS for ARM servers● UEFI/ACPI● KVM/Xen● ARMv8 optimization● OpenJDK, Hadoop,

OpenStack

● Reduces fragmentation, cost, accelerates time to market

Linaro Segment GroupsDigital Home - LHG Mobile - LMG Networking - LNG Enterprise - LEG

● Formed in November 2012● enable the ARM server ecosystem● share costs of the enablement

engineering● focus on standards, avoid

fragmentation● identify, test and optimise

workloads and core libraries

Linaro Enterprise Group (LEG)

http://wiki.linaro.org/LEG

Engineering (1): enablement

● All patches already upstream or under deep constructive review● Strong engagement with all maintainers

Engineering (2): workloads

● LEG projects agreed and prioritized by LEG Steering Committee● One representative per member, plus Linaro LEG Director

● ARM Boot Architecture● ARM Server Base System Architecture; ACPI, UEFI, Grub; ARM Trusted firmware

● KVM and Xen Virtualization● Stable ARM kernel version based on upstream● LAMP Stack optimization

● HDFS CRC, LibTBB, CRC, Hugepages, OpenSSL

● Facebook HipHop JIT● High performance OpenJDK for ARMv8 with C1 and C2 JIT

Engineering I

● Middleware and user-space stack testing● Hadoop, OpenStack, Ceph

● Testing through multinode CI validation in LAVA● ARMv8 Ubuntu, Fedora/Red Hat and OpenStack builds

● LEG also works with Linaro core working groups and can request resources through the Linaro Technical Steering Committee (TSC)● ARMv8 64 bit toolchain optimization● Multi-core power management

● Security and Secure boot using Trustzone and ARM Trusted Firmware

Engineering II

● UEFI● Aligned ARM Tianocore to other architectures, boot as an EFI application, runtime

services, pass ACPI tables from firmware, support for SMBIOS 3.0● Support GRUB and network boot● Ported GNUEFI to build EFI apps easily● Verified Tianocore Secure boot and ported the Shim layer to ARM● Ported UEFI to EL1 as a guest in KVM/QEMU hypervisor (ongoing for Xen)● Established a monthly UEFI release rebased on Tianocore EDK2 + all Linaro patches,

tested with UEFI SCT suite● ACPI

● Ported ACPICA core kernel support and tools to ARM● Ported key ARM core peripherals to ACPI● Able to boot ARM FVP, Juno, AMD Seattle and Cavium Thunder-X model via ACPI● Developed a PCC driver to support the new CPPC power management model● Ported all reference test suites to ARM, e.g. FWTS, ACPI API test, ACPI ASL test, etc.● Releasing an updated LEG ACPI kernel for every new weekly kernel 3.xx-rc

Achievements (1)

● OpenJDK● Excellent cooperation with Red Hat on OpenJDK C1 and C2 JIT● Driven by OpenJDK8, backported to OpenJDK7 and being forward ported to

OpenJDK9● On par with x86 as JIT performance vs interpreted code● Close to 10,000 tests executed every night: Mauve, JTREG, JCStress, SPECjbb2013● Hadoop TeraSort and Jenkins as functional testing workloads

● Optimisations for the LAMP Webserver workload● ARMv8 assembly tuning for OpenSSL, CRC32, etc. up to 5x-16x speed up● Hugepages, fast_gup, etc.

● OpenStack● Built OpenStack on ARMv8 with KVM/QEMU (ongoing for Xen)● Running the Tempest test suite and the Rally benchmarking suite● Functional test deploying multiple VM’s to run Java JTREG testing in parallel

● Early proof of concept with Docker and Go

Achievements (2)

it’s time for some cloud and server stuff now….

Platform as a Service (PaaS)

Infrastructure as a Service (IaaS)

Software as a Service (SaaS)

Platform as a Service (PaaS)

Infrastructure as a Service (IaaS)

Software as a Service (SaaS)

Database as a Service (DBaaS)Mobile Backend as a service (MBaaS)

Logging as a Service (LaaS)

Desktop as a Service (DaaS)Data as a Service (DaaS)

Payments as a service (PaaS)

Security as a service (SECaaS)Network as a service (NaaS) IT as a service (ITaaS)

Storage as a service (STaaS)

Location as a service (LaaS)

Recovery as a service (RaaS)

Mobility as a Service (MaaS)

Let’s start with one remote machine

Let’s add few more remote machines

It is not sustainable and not scaling

Let’s go back to one control point

Let’s add nodes now

We can now run multiple VM’s per node

We can run multiple runtimes per VM

And one or more apps per runtime

PaaS

SaaS

IaaS

It’s all ...as a Service

OpenStack logical architecture

http://docs.openstack.org/admin-guide-cloud/content/logical-architecture.html

OpenStack conceptual architecture

http://www.solinea.com/blog/openstack-grizzly-architecture-revisited

OpenStack software diagram

http://www.openstack.org/themes/openstack/images/openstack-software-diagram.png

OpenStack modules

http://www.ca.com/us/lpg/ca-technology-exchange/sunny-outlook-for-openstack-clouds.aspx

It’s running on ARM 64-bit now!

http://www.ca.com/us/lpg/ca-technology-exchange/sunny-outlook-for-openstack-clouds.aspx

It’s running on ARM 64-bit now!

● Long cooperation with Canonical and Red Hat engineers● Reusing pre-built DevStack packages from Ubuntu● Leveraging on LEG kernel and Linaro KVM work● Running on real ARM 64-bit hardware,

APM X-Gene and HP Moonshot● Running all applicable Tempest tests

● Ongoing● PackStack packages from Fedora● libvirt running on XEN

● AARCH64 port of OpenJDK● Based on same codebase as x86 OpenJDK● 95%+ code in common with Oracle JDK● Same relative C2 vs interpreter JIT performance within a few % as Oracle JDK● Goal is to track all x86 optimisations / features● No support for ARMv7 or earlier or for 32 bit on ARMv8

● Three releases are supported: JDK7, JDK8 and JDK9

OpenJDK on ARMv8

CI and nightly testsServer results from Aug 22 2014

ARM64 x86

Pass Fail Error Pass Fail Error

Hotspot 562 2 0 562 2 0

langtools 3004 0 11 2979 0 25

JDK 5283 181 46 5273 232 5

Totals 8849 183 57 8814 234 30

● Daily native build of Hadoop from source● Daily benchmark testing

● Client and server run of Terasort to test for performance regression

● Single node testing only.

● Looking at multinode testing

● Optimisation focused on optimising OpenJDK for Hadoop● +24% improvement since FCS release in April

● Some optimisation on Hadoop directly● CRC optimisation using SIMD

● Pushed to list but not yet adopted upstream

Hadoop Testing / Optimisation

Demonstrated at LCU14:OpenStack running OpenJDK JTREG testing with Jenkins

LXC 0..3

OpenStack Cloud Controller

Compute Node 0

Compute Node 1

● 3x APM Mustang systems running Ubuntu Trusty 14.04● All services deployed with juju● Mustang 01 as cloud controller and jenkins master with LXC● Mustang 02 and 03 as OpenStack compute nodes with KVM

● Run a subset of the OpenJDK JTREG test regression harness● jenkins dynamically provisions SIX VM’s instances via OpenStack on Mustang 2 and 3● the VM’s are destroyed upon test completion

Demo Overview

What’s the effort and cost to port OpenStack?

let’s review few open source projects

let’s rely on ohloh.net and the co.co.mo. model

http://csse.usc.edu/csse/research/COCOMOII/cocomo_main.html

OpenStack: lines of code

OpenStack: commits per month

Hadoop: lines of code

Hadoop: commits per month

Linux Kernel: lines of code

Linux Kernel: commits per month

Chrome: lines of code

Chrome: commits per month

Comparison

Project SLOC (M) Commits/month Start date

OpenStack 2 3992 2010*OpenJDK 8 5 983 2007Hadoop 2.5 219 2006mongoDB 0.6 500 2007Linux Kernel 17 6000 2002Google Chrome 8 5000 2008

(*) 1st commit for OpenStack was in 2006 but really picked up in mid 2010

● OpenStack is still relatively small but growing as fast as the Linux Kernel itself…

What about OpenStack on ARM then?

a.k.a. percussive maintenance

Tapping the server with a hammer: $0.00.Knowing where to hit it: $5,000.00.

The Fonzarelli Fix

http://tvtropes.org/pmwiki/pmwiki.php/Main/PercussiveMaintenance

● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af

● libvirt: "AArch64: Add AArch64 architecture to list of valid arches."http://libvirt.org/git/?p=libvirt.git;a=commit;h=670c08afd461cba6164f7a8a643f5a5b41782e8d

● libvirt: "AArch64: Porting of armv7l conditons to run qemu for aarch64."http://libvirt.org/git/?p=libvirt.git;a=commit;h=27e32e0f3d4d5840cee7e361bdff94c68ae181e2

● libvirt: "AArch64: CPU Support for AArch64 (ARMv8 64bit)."http://libvirt.org/git/?p=libvirt.git;a=commit;h=5cb6816715968dc08486307bd1894dc91e22342c

● libvirt: "qemu: Add support for virt machine type with virtio-mmio devices on armv7"http://libvirt.org/git/?p=libvirt.git;a=commit;h=c7ccd2c44be99b81ffce29374faeab3d1644a151

Patches available upstream

● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af

● libvirt: "AArch64: Add AArch64 architecture to list of valid arches."http://libvirt.org/git/?p=libvirt.git;a=commit;h=670c08afd461cba6164f7a8a643f5a5b41782e8d

● libvirt: "AArch64: Porting of armv7l conditons to run qemu for aarch64."http://libvirt.org/git/?p=libvirt.git;a=commit;h=27e32e0f3d4d5840cee7e361bdff94c68ae181e2

● libvirt: "AArch64: CPU Support for AArch64 (ARMv8 64bit)."http://libvirt.org/git/?p=libvirt.git;a=commit;h=5cb6816715968dc08486307bd1894dc91e22342c

● libvirt: "qemu: Add support for virt machine type with virtio-mmio devices on armv7"http://libvirt.org/git/?p=libvirt.git;a=commit;h=c7ccd2c44be99b81ffce29374faeab3d1644a151

LibVirt patches

Thanks to libvirt maintainer Cole Robinson at Red Hat for his support!

● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af

OpenStack patches: only 114 lines in Nova

Thanks to Canonical engineers for their contribution!

DeploymentReady

Software

Developer ARMv8 Server Cluster to Accelerate Deployment-ready Software

Operating System Vendors

+ +

Open Source

Community

Independent Software Vendors

www.linaro.org/leg/servercluster/

● OpenStack is at the heart of the cloud revolution● All workloads run well on Linaro ARMv8 foundations● Linaro fosters the open source collaboration on ARM

Summary

More about Linaro: www.linaro.org/about/Linaro members: www.linaro.org/members