how to get back from marscontinuouslifecycle.london/wp-content/uploads/2017/... · docker uses a...
TRANSCRIPT
How to Get Back from MarsPersistent Storage for Containers
Julian Topley, Senior Solutions Architect
www.storageos.com
© 2013-2017 StorageOS Ltd. All rights reserved.
My container journey
• Solution Architect for StorageOS helping to deliver persistent storage for containers
• Over 15 years of experience delivering enterprise storage for dev teams at Goldman Sachs
• Helped move the organisation to modern storage methodologies and strategy
• Now I’m helping to solve storage for containers, microservices and cloud
© 2013-2017 StorageOS Ltd. All rights reserved. 2
3
How to get back from Mars
4
We all want simplified, rapid application development and deployment to the cloud, so we’ve turned to containers …
© 2013-2017 StorageOS Ltd. All rights reserved.
Or shall we say Docker
Docker allows users to package, distribute and manage Linux applications within containers
5
Simplicity and faster configurations
Increased productivity
Consistency across environments – package dependencies with your apps
Rapid deployment
Works with any stack
© 2013-2017 StorageOS Ltd. All rights reserved.
Docker has changed …
6
How we develop, deploy
and maintain applications
How we consume infrastructure
© 2013-2017 StorageOS Ltd. All rights reserved.
Containers miss a major mark
7
NO DATA PERSISTENCEcontainers cannot maintain data persistence when rescheduled or destroyed
© 2013-2017 StorageOS Ltd. All rights reserved.
8
… but Docker is held back by traditional, often monolithic, storage infrastructure solutions
© 2013-2017 StorageOS Ltd. All rights reserved.
6 common problems with persistent container storage
9
1.Stateless containers do not support storage
features such as HA, replication and
clones 3. Storage does not scale with or follow
apps and performance can be unpredictable2. Legacy storage
architectures are complex, deliver to infrastructure not apps and lack API
automation
4. Moving data securely between locations and/or
cloud providers is difficult
5. Integrated management and
performance toolsets are
lacking
6. Cost model is geared towards fork lift upgrades, vendor
lock-in (storage or cloud) and complex
refresh cycles
© 2013-2017 StorageOS Ltd. All rights reserved.
10
How have containers changed how DevOps develop and deploy to infrastructure?
• Containers are growing in popularity but still evolving
• There is much experience yet to be gained
• Cut down CI/CD development times
• Significantly improved efficiency over virtualisation
FAST CURRENT
• Allow simple representation of components
• Easily integrated into other platforms
• Docker integrates with tools like Jenkins and GitHub
• Build and test on any platform or infrastructure
BUILT FOR CI/CD API DRIVEN ECOSYSTEM
11
Containers have changed how we deploy and maintain applications
© 2013-2017 StorageOS Ltd. All rights reserved.
Containers have changed how we consume infrastructure
12
• With stateless applications, data needs to be externalized and persisted
DATA EXTERNALISED AND PERSISTED
• Hyper-convergence is driving adoption of commodity compute, networking and storage in virtualized environments and driving up efficiency
• Applications have become loosely coupled and stateless
• Designed to scale and manage failure – it is no longer economical to remediate state
APPS LOOSLEY COUPLED & STATELESS HYPER-CONVERGENCE
© 2013-2017 StorageOS Ltd. All rights reserved.
13
How does Docker manage persistent container storage?
Key technologies behind Docker image and container management
• Docker containers comprise a layered image and a writable ‘Container Layer’• New, or modified data is stored in the writable container layer• When a container is deleted its writable layer is removed leaving just the
underlying image layers behind
14
Stackable ImageLayers
Container Layer Copy-on-write
1 2
© 2013-2017 StorageOS Ltd. All rights reserved.
Starting up and destroying a container
• When you launch a container, Docker uses a copy-on-write overlay to apply changes as a read-write ‘Container Layer’ to the read-only container image
• Once an app is terminated…
15
2
• Different images can share the same layers
• Sharing layers promotes smaller images
$ docker stop mycontainer
$ docker rm mycontainer
1
$ docker run –d --name mycontainer myapp:v2
© 2013-2017 StorageOS Ltd. All rights reserved. Confidential - Not for distribution. 16
Your data is gone
Docker’s solution
17
3. Volume Plugins
2. Named Volumes
1. Directory Mounts
Local Directory Mounts
18
1
$ docker run --rm -v ~/tmp:/data alpine ash -c \"echo hello world > /data/myfile"
• Share data between host and containers
SHARED
• Data persists after a container has been removed
PERSISTENCE
$ sudo cat ~/tmp/myfilehello world
Local Named Volumes
19
2
• Share data between host and containers
• Also works with NFS
PERSISTENCE
• Data persists after a container has been removed
SHARED
$ docker volume create --name mydata
$ docker run --rm -v mydata:/data:rw alpine ash -c \"echo hello world > /data/myfile"
$ sudo cat /var/lib/docker/volumes/mydata/_data/myfilehello world
Volume Plugins
20
3
• Integrates external storage
• Extends ecosystem
INTEGRATED• Data persists
beyond the host life
• Choose from Block, File and Object
EXTERNALISED
$ docker run --rm -v myvol:/data \--volume-driver=storageos \alpine ash -c "echo hello world > /data/myfile"
$ docker volume create --driver storageos --opt size=1 myvol
© 2013-2017 StorageOS Ltd. All rights reserved.
Real World Example – StorageOS Volume Plugin
21
Independent data and control plane architecture
• Manages data access requests
• Pools aggregated storage for presentation
• Runs as a container
DATA PLANE
• Manages config, health, scheduling, policy, provisioning and recovery
• API is accessed by plugins, CLI, GUI
• Runs as a container
CONTROL PLANE
© 2013-2017 StorageOS Ltd. All rights reserved.
22
How can DevOps benefit from volume plugins?
Container storage provides mobility
23
• Provides flexibility and choice to move regardless of infrastructure
• Completely API driven with end-to-end integration across ecosystems
ORCHESTRATION SOFTWARE DEFINED
• Hot spots require rescheduling containers & storage to other nodes
• Requires mobility within a cluster and across clouds and datacenters
PERSISTENT DATA HOT SPOTS
© 2013-2017 StorageOS Ltd. All rights reserved.
Why a new storage paradigm for Docker is needed
24
Not all plugins are equal, some plugins simply provide a thin wrapper around IaaS
Some plugins may not integrate properly into other platforms such as Kubernetes or Mesos
Container storage should address a variety of SLAs, performance, access and cost constraints
Container storage needs to be natively delivered to the platform not infrastructure
Storage needs to deliver repeatable processes into CI/CD pipelines with clones and snapshots
© 2013-2017 StorageOS Ltd. All rights reserved.
Addressing business continuity and HA
25
Asynchronous Replication
Public Cloud
• Reduce complexity• Reduce licensing• Reduce infra
• Highly available and replicated
• Global visibility
CRITICAL APPS REDUCE COSTS
• Operate across on-premises or public cloud infrastructure
• Business continuity maintenance and migrations
DATA ANYWHERE CONSISTENCY
I N F R A S T U C T U R E
OS Kernel
Highly Available Container Storage Pool
Container Runtime
OS Kernel
Container Runtime
OS Kernel
Container Runtime
App Containers App Containers App Containers
© 2013-2017 StorageOS Ltd. All rights reserved.
Different storage services can work together
Applications may have more than one storage requirement:1. Ephemeral, performance storage for application binaries2. Shared persistent storage for configuration 3. Dedicated persistent performance storage4. Cost efficient persistent backup storage
Applications may require different storage features:1. Performance: Replication, Encryption2. Efficiency: Compression, deduplication
© 2013-2017 StorageOS Ltd. All rights reserved. 26
27
© 2013-2017 StorageOS Ltd. All rights reserved.
CI/CD benefits from persistent storage
A continuous delivery pipeline with containerised software is a new paradigm
But it’s not easy to work with Docker volumes when you need to access data between Development, QA, Staging and Production…
CREATE A SNAPSHOT
A B C D
Current File System Snapshot
CREATE A THIN CLONE FROM A SNAPSHOT
A B C D
Snapshot Thin Clone
CHANGES POINT TO NEW DATA BLOCKS
A B C D
Snapshot Thin Clone
E FD+
Deleted data
Modifieddata
New data
unless you have snapshots and thin clones
CI/CD benefits from persistent storage
28© 2013-2017 StorageOS Ltd. All rights reserved.
• Start up thin clones of Jenkins /home
• Run tests with all images and artefacts
JENKINS
• Run unit tests against a database migration
• Start from a known state with the ability to reset and re-test
• Globally visible volumes provide instant snapshot and clones
• Provide data mobility and version control into the build pipeline
DATA MOBILITY & VERSION CONTROL UNIT TESTING
Build TestCode DeployPublish
Where do we go next?
Like getting to Mars, the first part of the journey is always the easy part, getting there – figuring out the hard part comes next
29© 2013-2017 StorageOS Ltd. All rights reserved.
… Not everyone is a rocket scientist
Containers are popular and still evolving
30© 2013-2017 StorageOS Ltd. All rights reserved.
The journey has just begun for Volume, Security and Network plugins
As we get more familiar working with containers and services, the more we’ll demand
While containers started out as simple ephemeral applications, they are now highly evolved, complex and with growing persistence use cases
31
Native Volume Plugins provide the necessary rocket fuel to get us back and complete our journey
© 2013-2017 StorageOS Ltd. All rights reserved.
So how do we complete our journey?
QuestionsJulian Topley, Senior Solutions [email protected]
www.storageos.com© 2013-2017 StorageOS Ltd. All rights reserved.