microservices in unikernels
TRANSCRIPT
Microservices in UnikernelsRean Griffith, Madhuri Yechuri
1
AgendaIntroduction - bios
Unikernel Background
Developer/DevOps care about
Metric Set 1: Application lifecycle overhead
CIO cares about
Metric Set 2: Application datacenter footprint
Customer cares about
Metric Set 3: Application performance
Q & A
2
What is a Unikernel?
A single purpose (virtual) appliance (Madhavapeddy et al.)Specialized at compile-time into a standalone kernel
A single-process, single address-space runtime environmentNo fork()
No shared memory
No IPC
Smaller attack surface (potentially) 3
fork() Shared memory
ApplicationIPC
networkingsched Application
networkingthreadsched
servicesvmm
vmm
Unikernel Background
4
Unmodified Legacy App support
Multi-threaded App support
OSv Partial Yes (1: glibc subset, no fork/exec)
Yes* (pthread subset)
Rumprun Yes* (no fork/execve/sigaction/mmap)
Yes (pthread)
MirageOS No* (until non-OCAML language bindings are available, no fork/execve)
Green threads (event loop) only
IncludeOS No Green threads (event loop) only
Developer/DevOps care about
Enterprise Application Lifecycle management
Developer: Time to build app from source code, preferably unmodified
DevOps: Time to configure runtime parameters (ex: TCP port, log file location)
DevOps: Time to deploy application
DevOps: Qualitative ease of managing+debugging long-running (weeks / months) application
DevOps: Time to destroy application (free up its resources)
5
App Lifecycle Experiment EnvironmentMachine
CPU: Intel(R) Core(TM) i3 CPU M 380 @ 2.53GHz
Memory: 4GB RAM
OS: Ubuntu 16.04 LTS
Applications
Web tier: Nginx
Application tier: Tomcat
Deployment Options (local image)
VM: VirtualBox 5.0 (Hashicorp Alpine 3.4), Vagrant 1.8.1, Nginx 1.10.1-r1, Tomcat 7, pre-pulled images, 1GB VM
Container: Docker 1.12.0-rc2, pre-pulled Official Nginx:latest (1.11.1) image, official Tomcat:latest (8.0) image
Unikernel: OSv 0.24, KVM + QEMU 2.5.0, pre-pulled Nginx 1.9.15 image, pre-pulled Tomcat 7.0 image, 2GB VM
6
Metric Set 1: Application LifecycleConvert Code to Image (Hours)
VM 8 (Nginx, Issues: 1 , 2, 3)8 (Tomcat, Issues with Alpine glibc availability)
Container 0 (Nginx)0 (Tomcat)
Unikernel 40 (Nginx, Issues: 1, 2)4 (Tomcat)
7
Unikernel conversion - ukonvrt
Demo
8
Metric Set 1: Application LifecycleConvert Code to Image (Hours)
Start Time(Seconds)
VM 8 (Nginx)8 (Tomcat)
66.557 (Nginx)68.964 (Tomcat)
Container 0 (Nginx)0 (Tomcat)
1.113 (Nginx)4.1 (Tomcat)
Unikernel 40 (Nginx)0 (Tomcat)
0.483 (Nginx)10 (Tomcat)
9
Metric Set 1: Application LifecycleConvert Code to Image (Hours)
Start Time(Seconds)
Stop Time(Seconds)
VM 8 (Nginx)8 (Tomcat)
66.557 (Nginx)68.964 (Tomcat)
7.478 (Nginx)5.418 (Tomcat)
Container 0 (Nginx)0 (Tomcat)
1.113 (Nginx)4.1 (Tomcat)
0.685 (Nginx)0.016 (Tomcat)
Unikernel 40 (Nginx)4 (Tomcat)
0.483 (Nginx)10 (Tomcat)
0.019 (Nginx)0.006 (Tomcat)
10
Metric Set 1: Application LifecycleCode to Image (Hours)
Start Time(Seconds)
Stop Time(Seconds)
Debuggability
VM 8 (Nginx)8 (Tomcat)
66.557 (Nginx)68.964 (Tomcat)
7.478 (Nginx)5.418 (Tomcat)
Container 0 (Nginx)0 (Tomcat)
1.113 (Nginx)4.1 (Tomcat)
0.685 (Nginx)0.016 (Tomcat)
Unikernel 40 (Nginx)4 (Tomcat)
0.483 (Nginx)10 (Tomcat)
0.019 (Nginx)0.006 (Tomcat)
CIO cares about
Consolidation of applications on finite hardware resources
Multi-tenant security isolation amongst applications on a compute node
Multi-tenant Resource Management Manageability, Accounting, Auditability
Infrastructure Power consumption
12
Metric Set 2: Data center footprintImage Size(MB)
VM 143 (Nginx)447 (Tomcat, Issue 1 - Alpine musl vs glibc)
Container 182.8 (Nginx)357.5 (Tomcat)
Unikernel 27.8 (Nginx)106 (Tomcat)
Metric Set 2: Data center footprintImage Size(MB)
Runtime Memory Overhead (MB)
VM 143 (Nginx)447 (Tomcat)
619 (Nginx)878 (Tomcat)(/proc/{vboxpid}/status/{VmSize} - Configured)
Container 182.8 (Nginx)357.5 (Tomcat)
274.4 (Nginx)210.5 (Tomcat)(containerd-shim /proc/{pid}/status/{VmSize})
Unikernel 7.8 (Nginx)106 (Tomcat)
1222 (Nginx)2056 (Tomcat)(/proc/{qemupid}/status/{VmSize} - Configured)
Metric Set 2: Data center footprintImage Size(MB)
Runtime Memory Overhead (MB)
Security (Tenant Isolation)
VM 143 (Nginx)447 (Tomcat)
619 (Nginx)878 (Tomcat)
Strong
Container 182.8 (Nginx)357.5 (Tomcat)
274.4 (Nginx)210.5 (Tomcat)
Weak
Unikernel 7.8 (Nginx)106 (Tomcat)
1222 (Nginx)2056 (Tomcat)
Strong
Metric Set 2: Data center footprintImage Size(MB)
Runtime Memory Overhead (MB)
Security (Tenant Isolation)
Resource Knobs
VM 143 (Nginx)447 (Tomcat)
619 (Nginx)878 (Tomcat)
Strong Strong (Reservation, Limits)
Container 182.8 (Nginx)357.5 (Tomcat)
274.4 (Nginx)210.5 (Tomcat)
Weak Moderate (Limits)
Unikernel 7.8 (Nginx)106 (Tomcat)
1222 (Nginx)2056 (Tomcat)
Strong Moderate (knobs available, not used yet)
Customer cares about
Application PerformanceResource Isolation
Security
Application high-availability
17
Performance Experiment EnvironmentMachine
Lenovo W520, CPU: Intel i7-2760QM CPU 2.40 GHz x 8 logical cores, Memory: 19.5 GB RAM
OS: Ubuntu 16.04.1 LTS (64-bit)
Deployment Options
Linux (host machine) - Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP
VM: VirtualBox (v5.1.2) - Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP, 8GB RAM, 4 vCPUs
Container: Docker (v1.12.0) - Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP
Unikernel: OSv (based on git hash: f53c0c39) - v0.24-176-g2e19ba4 (Ubuntu 16.04.1 LTS, Linux kernel: 4.4.0-34-generic #53-Ubuntu SMP), 4 vCPUs, 2GB RAM
Application: Nginx (v1.10.0) hosting small static html page (88 bytes), worker_processes = 1, 10240 worker connections Apache Tomcat (7.0.54) hosting example servlets (Hello World)
Client Workload: Rain (https://github.com/rean/rain-workload-toolkit/tree/master/src/radlab/rain/workload/nginx)
18
Metric Set 3a: Application Performance (nginx)
19
Metrics Set 3: Throughput Explanationnginx-osv > nginx-linux > nginx-docker > nginx-vm
Baseline: 1 thread/clientNginx-linux (bare metal) ~600 requests/secNginx-vm slightly lower: expected because the client request needs to traverse
two I/O stacks - the hypervisor’s and the Guest OS’sNginx-docker is close to bare metal: expected since the only thing separating the
container from the workload generator is a network bridgeNginx-osv slightly better than bare metal: client requests still have to go through
the unikernel’s I/O stack but the I/O stack for OSV was designed to be light/lower-overhead - influenced by a design based on Van Jacobson’s net channels
10 threadsResults get slightly more than 10X better (this is mostly because of reductions in
average latency - next graph) but the ordering remains the same20
Metrics Set 3: Response Time Explanation
nginx-osv > nginx-linux > nginx-docker > nginx-vm
Overall response times between 1ms and 2msSingle thread case ~1.5ms, and 10 thread case < 1.5msReduction in response time moving 1 to 10 threads is mostly a
result of caching and multiplexing. With multiple threads, more work gets done per-unit time. While thread A is
processing the results of a response, thread B, which was waiting, can quickly be given a cached copy of the static file being served.
21
Metric Set 3b Application Performance (Apache Tomcat)
22
Summary
Many microservice tools can be deployed in a unikernelNginx, Tomcat, JVM, Nodejs, Redis, Memcached etc. (list is growing)
Performance is comparableSmaller “attack” surface (no extraneous services)
Lean network-stack (e.g., OSv)
Lean OS (no kernel-userspace crossing, no context-switching, heavy mem mgt etc.,)
Opportunities in tooling to help flesh out the workflow for planning or effecting UK-deployments
ukonvrt, capstan (build tool from OSv), ukdctl
23
Acknowledgements
Thank you!
OSv: Nadav Har’El
Nirmata: Jim Bugwadia
Microservices and Cloud Native Apps Meetup
Mike Larkin, Carl Waldspurger, Anne Holler
24
Links
Tools: Ukonvrt, ukdctl
Rain Workload Toolkit
Nginx VirtualBox repo
Nginx OSv
OSv networking hack
Performance evaluation of OSv
OSv USENIX paper25
Q & A
Madhuri
GitHub: myechuri
Rean
GitHub: rean 26