containers and virtual machines at scale: a …...containers and virtual machines at scale: a...
TRANSCRIPT
Containers and Virtual Machines at Scale: A Comparative Study
Prateek SharmaLucas ChaufournierPrashant Shenoy
Y.C. Tay
1
Virtualization in Data Centers and Clouds• Data centers and clouds host multiple applications• Virtualization and cluster management frameworks used for
managing and multiplexing resources• Virtualization options: Hardware-level and Operating System-level
2
Virtualization Layer
Cluster Management Framework(OpenStack, Kubernetes…)
Virtualization LayerVirtualization Layer
System Virtualization Today
3
Hardware Operating System
Virtual Machines Containers
Example VMWare, Xen, … Docker, LXC, …
Management OpenStack,… Kubernetes,…
Performance Noticeable overhead “Lightweight virtualization”
Guest OS Operating System
Problem Statement
• How do they compare in multi-tenant data centers and cloud environments?1.What are the tradeoffs in application performance? 2.How do they affect resource management and deployment?3.What is their influence on software development?
4
• Hardware virtualization predominant (especially in public clouds)• OS virtualization (Containers) is being rapidly adopted
Guest OSvs.VM Container
Outline
• Motivation and problem statement• Virtualization background• Performance comparison• Managing and deploying applications• Hybrid virtualization approaches• Related work• Conclusion
5
Hardware Virtualization• Hypervisor exposes virtual hardware (CPU, memory, I/O)• Applications run on top of a guest Operating System• Multiple layers between application and hardware may reduce
performance
6
Hardware
Hypervisor
Virtual H/W Virtual H/W
Guest OS
ApplicationVirtualMachine
Guest OS
Application
Xen, KVM,VMWare…
Linux, Windows…
Operating System Virtualization• Lightweight virtualization of OS resources and interfaces• Applications run in Containers and share the underlying OS• Cgroups in Linux for isolating resources (cpu, mem, I/O,…)• Namespaces to isolate process hierarchies, sockets, filesystem,…
7
Hardware
Operating System
Application
Container
Application
Linux, BSD, Solaris
LXC Jails Zones
VirtualizationLayer
Outline
• Motivation and problem statement• Virtualization background• Performance comparison• Managing and deploying applications• Hybrid virtualization approaches• Related work• Conclusion
8
Performance Evaluation Setup
• KVM for hardware virtualization vs. Linux Containers (LXC)• Containers and VMs given same physical resources using cgroups• CPU, memory, I/O intensive benchmarks• In multi-tenant environments, applications can face interference• Performance depends on type of co-located application:
9
Application 1 2
App-1 App-2
Competing
App-1 App-2
Orthogonal
App-1 App-2
Adversarial(Intentional starvation)
CPU Isolation
• Container and VM CPUs pinned• Containers can face starvation due to shared OS CPU scheduler• Shared OS or hypervisor components can reduce isolation
10
VMs
Containers
Kernel compile slowdown relative to no-interference0 0.2 0.4 0.6 0.8 1 1.2 1.4
Competing Orthogonal Adversarial
Did Not Finish
10-25% slowdown
Kernel-compile File-bench Fork-bomb
• Map applications to servers carefully to minimize interference
I/O Isolation
• Equal allocation of I/O bandwidths using cgroups• Disk I/O handled by hypervisor (virtIO in KVM)• Both VMs and containers have shared I/O
paths, causing interference
11
VM 1
Hypervisor
VM 2
File-bench Kernel-compile Bonnie++
VMs
Containers
File-bench latency0 0.22 0.44 0.66 0.88 1.1 1.32 1.54 1.76 1.98 2.2
Competing Orthogonal Adversarial
High variance
High average
Outline
• Motivation and problem statement• Virtualization background• Performance comparison• Resource management and deployment• Hybrid virtualization approaches• Related work• Conclusion
12
Resource Overcommitment
13
SSecJBB0
5000
10000
15000
20000
25000
30000
ThrouJhSut
lxc vm
• Memory overcommitted by 2x for SpecJBB• Excessive page faults in guest OS,
because VM memory limits are hard• Performance depends on overcommitment
capabilities of virtualization layer
40%
• Cluster managers overcommit resources to increase efficiency• Allocate more resources than are actually available
• Hypervisors support overcommitment mechanisms for VMs • OS overcommits container resources using soft-limits
• Free (but allocated) resources can be used by other applications
Migration• Live-migration used for load balancing and consolidation• VM migration implemented in all hypervisors• Container migration is not production-ready
• Involves careful preservation of OS state (process table, network sockets, open file descriptors,…)
• Containers can instead be checkpointed and restarted
14
Memory size (GB) Container VMKernel Compile 0.42 4
YCSB 4 4SpecJBB 1.7 4File bench 2.2 4
• Container memory state is smaller • VMs must also save guest OS and cache pages
Getting Applications Into Production
15
• Docker: 2-6x fasterTime (s) VM (Vagrant) DockerMySQL 236 129NodeJS 304 49
Base OSStandard Libraries
ApplicationContainer images have only application layerVMs contain entire OS, and have larger images
• Getting applications from development to production involves creating disk images
• Fast image creation enables rapid testing and continuous deployment
Install App Application Image
Base Image(Ubuntu 14.04)
Deploy
Big Picture
• Performance depends on workload and neighbors • Interference because of shared resources
• Different application management and deployment capabilities • Neither is best for all use-cases and scenarios• Can we get the best of both worlds?
16
Image Management Software Engineering
MigrationResource Allocation
Overcommit
MemoryCPU Disk NetworkIsolation
Hardware Virtualization
OSVirtualization
No clear winnerLimits
Outline
• Motivation and problem statement• Virtualization background• Performance comparison• Managing and deploying applications• Hybrid virtualization approaches• Related work• Conclusion
17
Containers Inside VMs
18
.ernel CRmSLleYC6B-read
YC6B-XSdate0.0
0.2
0.4
0.6
0.8
1.0
1.2
RXnn
Lng
TLm
e Re
latLY
e tR
LXC
lxc lxcYm Ym
• Performance comparable to VMs
Hypervisor
ContainerApplication
VM • Run containers inside VMs • Isolation and security of VMs• Image-management, provisioning of
containers
Promising architecture which combines benefits of today’s hardware and OS virtualization
Lightweight VMs• Heavily optimized and stripped-down hardware VMs• Make VMs more like containers• Isolation properties of VMs + low overhead of containers
• Intel ClearLinux, VMware Project Bonneville• Directly access host file system. No virtual disk abstraction.
19
Boot-time (s)
Container 0.3ClearLinux 0.8
KVM ~10
• Fast boot-times enable VMs to be used in container pipelines• Quick deployment tests, rapid
scaling,…
Related Work
• Felter et.al. — Single container vs. VM performance study• Agarwal et.al. — Comparison of memory footprint • Containers in HPC (Ruiz et.al) , data processing (Xavier et. al) • Alternate virtualization architectures — Unikernels, NoHype
(Keller et.al)
20
Conclusion
• Containers and VM represent two choices in system virtualization• Different tradeoffs in performance isolation, deployment• “Right platform” depends on workload, environment, and use-case• Nested containers and lightweight VMs are two promising hybrid
strategies• Fast changing landscape: change in performance and
capabilities will require continuing evaluation
21
23
Backup Slides
Baseline
24
lRad read uSdateYCSB 5edLs RSeratLRns
050
100150200250
Late
ncy
(ms)
lxc Ym
FLOebenFh Throughput
02000400060008000
1000012000
2pe
ratLo
ns/s
eFon
dFLOebenFh /atenFy
0.00.20.40.60.81.0
ms
OxF vm
CPU with different Cgroups setting
25
25% 50% 75%CPU allocation
0
50000
100000
150000
200000
SSec
-%%
thUo
uJhS
utCPU-set CPU-shaUe
More Interference
26
LXC V00.0
0.2
0.4
0.6
0.8
1.0
6Sec
JBB
thrR
XJhS
Xt
ReO
atLv
e tR
nR
Lnte
rfer
ence
CRmSetLnJ 2rthRJRnaO AdverVarLaO
LXC V00.0
0.2
0.4
0.6
0.8
1.0
RXbL
V th
rRXg
hpXt
R
eOat
Lve
tR n
R Ln
terf
eren
ce
CRmpetLng 2rthRgRnaO AdverVarLaO
Today’s Virtualization
• Applications run inside Virtual Machines
• Vmware, Xen, KVM,…• Can choose own guest OS
27
Hardware Virtualization Operating System Virtualization• Applications run inside
Containers• Docker, Jails, LXC,…• Applications share the OS• Lightweight virtualization
Guest OS
VMContainer
• Cluster-wide management tools help users deploy applications:• Placement of applications • Horizontal scaling
Application Deployment
28
• VMs contain entire OS, and have larger images• Docker stores only differences (application layer)
Image size VM LXC DockerMySQL 1.68 GB 0.4 GB 112 KBNodeJS 2.05 GB 0.6 GB 72 KB
ApplicationCopy & Deploy
Disk Image
Docker: 2-6x smaller
Ubuntu 14.04
Base Install app
App
29
Base line performance
Isolation: CPU, Memory, Disk, Network
Resource Overcommitment,migration
Image Creation
SwEngg