vmworld 2015: containers without compromise - persistent storage for docker containers with vmware
TRANSCRIPT
![Page 1: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/1.jpg)
Containers without Compromise:Persistent Storage for Docker Containers with
VMwareVivek Saraswat, VMware, Inc
CNA5860
#CNA5860
![Page 2: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/2.jpg)
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
CONFIDENTIAL 2
![Page 3: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/3.jpg)
CONFIDENTIAL 3
Agenda
1 Intro to Cloud-Native Applications
2 Containers 101
3 Persistent Data in Containers
4 VMware Solutions for Cloud-Native Storage
5 Looking Towards the Future
![Page 4: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/4.jpg)
CONFIDENTIAL 4
What are Cloud-Native Applications?
Developer access via APIs
Microservices, not monolithic stacks
Continuous integration and deployment
App-defined Availability
Built for scale
Decoupled from infrastructure
Application
![Page 5: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/5.jpg)
CONFIDENTIAL
Hardware
OS Kernel
OS File system
Use
rspa
ce
Container
App
pro
cess
App
pro
cess
App
pro
cess
App
pro
cess
App
pro
cess
Container
App
pro
cess
App
pro
cess
Linux Containers
55
• App-level Isolation– Isolation at individual kernel
subsystem level (e.g. filesystem, CPU resources, etc.)
– User-level process (LXC, libcontainer) orchestrates these subsystems to create a container
• Existed for Many Years– Solaris Zones, FreeBSD Jails, etc.
• Why?– Process isolation– Reproducible environment– Enables management at scale
![Page 6: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/6.jpg)
CONFIDENTIAL 6
is a “Shipping Container” for Code
Frictionless deployment andmaximum portability
On developer laptop:
Then on server:
That’s it!!
A natural fit for 3rd Platform, 12 factor, microservicesIt makes DevOps much much easier
Developers because …
~# docker build my_app~# docker push my_app~#
~# docker pull my_app~# docker run my_app~#
![Page 7: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/7.jpg)
CONFIDENTIAL 7
Containers Are Stateless “Cattle”
Source: “CERN Data Centre Evolution”http://www.slideshare.net/gmccance/cern-data-centre-evolution
![Page 8: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/8.jpg)
CONFIDENTIAL 8
But…What about Your Data?• If you start a new container, you might lose
all the data from the old one!• “Stateful” data needs to be accessed and
protected separately
• Original model: Persist stateful data to noncontainerized managed storage
• But, problems arise:– No local control over storage management– Latency/access issues– Issues at scale
![Page 9: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/9.jpg)
CONFIDENTIAL 9
Containerized Storage Apps are Rapidly Increasing
Thousands of DB appsMillions of downloads
![Page 10: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/10.jpg)
CONFIDENTIAL 10
Bring in the Container Data Volumes!• Usage
– Contains persistent data for local containers– Appears as directory within host file system
(e.g. “docker run –v /mount/mydata/”)– Can store on external storage and
mount/unmount from a host
• Benefits– Manage and preserve your stateful data– Utilize storage platform data services
Volumes open up new possibilities for containerized applications!
Host
Container
Storage Platform
Container
DataVolumes
![Page 11: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/11.jpg)
CONFIDENTIAL 11
Container Storage Use CasesUnshared Volumes Shared Volumes Persist to External Storage
Use Case: Running container-based SQL or noSQL DB
Use Case: Sharing a set of tools or content across app instances
Use Case: Object store for retention/archival, DBaaS for config/transactions
Host
C C
Storage Platform
Host
C C
Host
C C
Host
C C
Storage Platform
Host
C C
Host
C C
CloudStorage
API API
![Page 12: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/12.jpg)
CONFIDENTIAL 12
Containerized Storage in Practice• Application composed of series of
loosely coupled microservices– Scheduled by an application orchestrator
(e.g. Kubernetes, Mesos)– Microservices coordinate via REST API
• Each microservice consists of multiple stateless and stateful containers– API frontend container– Database engine container– Actual data is persisted to a container
data volume (supported byexternal storage)
ApplicationOrchestrator
Microservice 1
API Front-End
Database Engine
Data Volume
Microservice 2
API Front-End
Database Engine
Data Volume
Microservice 2
API Front-End
Database Engine
Data Volume
![Page 13: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/13.jpg)
CONFIDENTIAL 13
Container Volumes and Virtual Disks: Look Familiar?
![Page 14: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/14.jpg)
CONFIDENTIAL 14
Announcing vSphere Driver for Flocker 1.0!
![Page 15: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/15.jpg)
Luke Marsden, CTO
![Page 16: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/16.jpg)
CONFIDENTIAL 16
The Problem ClusterHQ’s Flocker Solves
![Page 17: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/17.jpg)
CONFIDENTIAL 17
ClusterHQ Flocker Data Volume Migration (before)
![Page 18: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/18.jpg)
CONFIDENTIAL 18
ClusterHQ Flocker Data Volume Migration (after)
![Page 19: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/19.jpg)
CONFIDENTIAL 19
• Integrate “upwards” to Orchestration Frameworks– Docker Swarm and Compose via Flocker Docker Plugin– Mesosphere/Marathon– Kubernetes (coming later), CoreOS
• Integrate “downwards” to storage vendors– vSphere driver enables awesome coverage– Growing ClusterHQ partner network– Also support OpenStack Cinder– Integrate with public cloud storage backends
• Add depth of storage capabilities– Make data portable: enable migration of data volumes between clouds,
different stages of software development lifecycle– Enable snapshotting, cloning, backup/restore, HA, DR…
Get in touch! clusterhq.com/contact @clusterhq
ClusterHQ Flocker Roadmap
ClusterHQ
![Page 20: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/20.jpg)
CONFIDENTIAL 20
vSphere Driver for Flocker Details• Run containerized stateful apps on your current vSphere deployment using open-source
Flocker software• Move containers + data volumes from one ESX VM to another• Compatible with ALL vSphere storage (VSAN, VVOL, VMFS, NFS)
• Straightforward install/configure/deploy process
• Free! • Available at https://github.com/vmware/vsphere-flocker-driver
![Page 21: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/21.jpg)
21
Native Docker on vSphere
CONFIDENTIAL
ESX VM2
Container DB App
ESX VM1
Container DB App
VMDK1+ Container Volume
VMDK2
ESX VM2
Container DB App
ESX VM1
Container DB App
vSphere + Flocker
MoveContainer
MoveContainer
VMDK1 VMDK2Container VolumeVMDK
When container moves, data volume stays on host VMDK. Database starts on new
VM without any of its data.
Data Volume stored on separate VMDK. When container moves, VMDK moves with
it. Database keeps its data!
![Page 22: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/22.jpg)
vSphere Flocker Demo
22
![Page 23: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/23.jpg)
CONFIDENTIAL 23
VMware Virtual SAN 6.1Radically Simple Hyperconverged Storage for VMs
vSphere + Virtual SAN…
• Software-defined storage optimized for VMs
• Hypervisor-converged architecture
• Runs on any standard x86 server
• Pools HDD/SSD into a shared datastore
• Delivers enterprise-level scalability and performance
• Managed through per-VM storage policies
• Deeply integrated with the VMware stack
Overview
Hard disksSSDHard disks
SSDHard disks
SSD
Virtual SAN Datastore
![Page 24: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/24.jpg)
CONFIDENTIAL 24
But Don’t Take Our Word for It…
“ClusterHQ and VMware supported our evaluation of Flocker with vSphere and Virtual SAN, providing us building blocks for container persistence in our Docker-based PaaS.”
![Page 25: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/25.jpg)
CONFIDENTIAL 25
![Page 26: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/26.jpg)
CONFIDENTIAL 26
Unified Hybrid Platform
vSphere Integrated Containers• Give developers the flexibility, portability
and speed containers deliver• Leverage existing investments in VMware
infrastructure, people, processes and management tools
• DRS, vMotion, HA/DR• Storage and Network Integration (VSAN
and NSX)
• Apply same isolation, data persistence, networking, management and robust service levels you have today
• No rebuilding or re-architecture required
• Full compatibility with broad ecosystem of existing tools
NSX
VSAN
vRealize
vSphere
vSphere Integrated Containers(Instant Clone, Project Bonneville, Photon OS)
![Page 27: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/27.jpg)
Cloud-Native Platform
VMware Photon Platform
CONFIDENTIAL 27
Photon Controller
Photon Machine
Photon Platform
Host controller & scheduler:High scale distributed control plane,includes Lightwave
Compute host:Lightweight hypervisor, based on ESXIncludes Photon OS
Deep integration with modern, open source frameworks & app platforms
![Page 28: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/28.jpg)
CONFIDENTIAL 28
Looking Towards the Future of Storage
• Distributed infrastructure– Global management– Automation friendly
• Scalable troubleshooting– Decentralized analytics– Information gathering, processing,
prediction
• IT-friendly GUI, scripting– Infrastructure operations
• Dev-friendly APIs, CLI– Application integration
Physical Servers
Infrastructures
Pools of Resources
Virtual SAN Datastore
Magnetic Devices
Flash Devices
Magnetic Devices
Flash Devices
Storage Abstraction Distributed storage platform
stor
age
infr
astr
uctu
rem
anag
emen
t
dist
ribut
ed m
onito
ring
UI/A
PIs
![Page 29: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/29.jpg)
CONFIDENTIAL 29
VDI files
A Distributed File System for Cloud-Native Apps• Hyper-converged scale-out file system
• Relies on Object Storage backend– Hardware management– Resource provisioning, discovery– Distributed parallel data path– High availability
• Backend: VSAN, others…– Can span multiple backend “clusters”
• Design Requirements– POSIX file system– Cloud scale: files, clients, clones– Per file / directory access control– O(1) snapshot / clone creation
Distributed storage management service
VSAN (Object) store VSAN (Object) store
Distributed File system
NoSQL
![Page 30: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/30.jpg)
Distributed File System Demo
30
![Page 31: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/31.jpg)
CONFIDENTIAL 31
What’s Now?
• vSphere driver for Flocker 1.0 available now for free at: https://github.com/vmware/vsphere-flocker-driver
• Extending functionality for VMware-based container data volumes• Storage integration with vSphere Integrated Containers• Storage integration with Photon Platform
Follow along on Twitter: @theVSaraswat
What’s Next?
![Page 32: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/32.jpg)
![Page 33: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/33.jpg)
![Page 34: VMworld 2015: Containers without Compromise - Persistent Storage for Docker Containers with VMware](https://reader035.vdocuments.mx/reader035/viewer/2022062904/587458c91a28abd53e8b9221/html5/thumbnails/34.jpg)
Containers without Compromise:Persistent Storage for Docker Containers with
VMwareVivek Saraswat, VMware, Inc
CNA5860
#CNA5860