how to get back from marscontinuouslifecycle.london/wp-content/uploads/2017/... · docker uses a...

How to Get Back from MarsPersistent Storage for Containers

Julian Topley, Senior Solutions Architect

www.storageos.com

© 2013-2017 StorageOS Ltd. All rights reserved.

My container journey

• Solution Architect for StorageOS helping to deliver persistent storage for containers

• Over 15 years of experience delivering enterprise storage for dev teams at Goldman Sachs

• Helped move the organisation to modern storage methodologies and strategy

• Now I’m helping to solve storage for containers, microservices and cloud

© 2013-2017 StorageOS Ltd. All rights reserved. 2

3

How to get back from Mars

4

We all want simplified, rapid application development and deployment to the cloud, so we’ve turned to containers …

© 2013-2017 StorageOS Ltd. All rights reserved.

Or shall we say Docker

Docker allows users to package, distribute and manage Linux applications within containers

5

Simplicity and faster configurations

Increased productivity

Consistency across environments – package dependencies with your apps

Rapid deployment

Works with any stack

© 2013-2017 StorageOS Ltd. All rights reserved.

Docker has changed …

6

How we develop, deploy

and maintain applications

How we consume infrastructure

© 2013-2017 StorageOS Ltd. All rights reserved.

Containers miss a major mark

7

NO DATA PERSISTENCEcontainers cannot maintain data persistence when rescheduled or destroyed

© 2013-2017 StorageOS Ltd. All rights reserved.

8

… but Docker is held back by traditional, often monolithic, storage infrastructure solutions

© 2013-2017 StorageOS Ltd. All rights reserved.

6 common problems with persistent container storage

9

1.Stateless containers do not support storage

features such as HA, replication and

clones 3. Storage does not scale with or follow

apps and performance can be unpredictable2. Legacy storage

architectures are complex, deliver to infrastructure not apps and lack API

automation

4. Moving data securely between locations and/or

cloud providers is difficult

5. Integrated management and

performance toolsets are

lacking

6. Cost model is geared towards fork lift upgrades, vendor

lock-in (storage or cloud) and complex

refresh cycles

© 2013-2017 StorageOS Ltd. All rights reserved.

10

How have containers changed how DevOps develop and deploy to infrastructure?

• Containers are growing in popularity but still evolving

• There is much experience yet to be gained

• Cut down CI/CD development times

• Significantly improved efficiency over virtualisation

FAST CURRENT

• Allow simple representation of components

• Easily integrated into other platforms

• Docker integrates with tools like Jenkins and GitHub

• Build and test on any platform or infrastructure

BUILT FOR CI/CD API DRIVEN ECOSYSTEM

11

Containers have changed how we deploy and maintain applications

© 2013-2017 StorageOS Ltd. All rights reserved.

Containers have changed how we consume infrastructure

12

• With stateless applications, data needs to be externalized and persisted

DATA EXTERNALISED AND PERSISTED

• Hyper-convergence is driving adoption of commodity compute, networking and storage in virtualized environments and driving up efficiency

• Applications have become loosely coupled and stateless

• Designed to scale and manage failure – it is no longer economical to remediate state

APPS LOOSLEY COUPLED & STATELESS HYPER-CONVERGENCE

© 2013-2017 StorageOS Ltd. All rights reserved.

13

How does Docker manage persistent container storage?

Key technologies behind Docker image and container management

• Docker containers comprise a layered image and a writable ‘Container Layer’• New, or modified data is stored in the writable container layer• When a container is deleted its writable layer is removed leaving just the

underlying image layers behind

14

Stackable ImageLayers

Container Layer Copy-on-write

1 2

© 2013-2017 StorageOS Ltd. All rights reserved.

Starting up and destroying a container

• When you launch a container, Docker uses a copy-on-write overlay to apply changes as a read-write ‘Container Layer’ to the read-only container image

• Once an app is terminated…

15

2

• Different images can share the same layers

• Sharing layers promotes smaller images

$ docker stop mycontainer

$ docker rm mycontainer

1

$ docker run –d --name mycontainer myapp:v2

© 2013-2017 StorageOS Ltd. All rights reserved. Confidential - Not for distribution. 16

Your data is gone

Docker’s solution

17

3. Volume Plugins

2. Named Volumes

1. Directory Mounts

Local Directory Mounts

18

1

$ docker run --rm -v ~/tmp:/data alpine ash -c \"echo hello world > /data/myfile"

• Share data between host and containers

SHARED

• Data persists after a container has been removed

PERSISTENCE

$ sudo cat ~/tmp/myfilehello world

Local Named Volumes

19

2

• Share data between host and containers

• Also works with NFS

PERSISTENCE

• Data persists after a container has been removed

SHARED

$ docker volume create --name mydata

$ docker run --rm -v mydata:/data:rw alpine ash -c \"echo hello world > /data/myfile"

$ sudo cat /var/lib/docker/volumes/mydata/_data/myfilehello world

Volume Plugins

20

3

• Integrates external storage

• Extends ecosystem

INTEGRATED• Data persists

beyond the host life

• Choose from Block, File and Object

EXTERNALISED

$ docker run --rm -v myvol:/data \--volume-driver=storageos \alpine ash -c "echo hello world > /data/myfile"

$ docker volume create --driver storageos --opt size=1 myvol

© 2013-2017 StorageOS Ltd. All rights reserved.

Real World Example – StorageOS Volume Plugin

21

Independent data and control plane architecture

• Manages data access requests

• Pools aggregated storage for presentation

• Runs as a container

DATA PLANE

• Manages config, health, scheduling, policy, provisioning and recovery

• API is accessed by plugins, CLI, GUI

• Runs as a container

CONTROL PLANE

© 2013-2017 StorageOS Ltd. All rights reserved.

22

How can DevOps benefit from volume plugins?

Container storage provides mobility

23

• Provides flexibility and choice to move regardless of infrastructure

• Completely API driven with end-to-end integration across ecosystems

ORCHESTRATION SOFTWARE DEFINED

• Hot spots require rescheduling containers & storage to other nodes

• Requires mobility within a cluster and across clouds and datacenters

PERSISTENT DATA HOT SPOTS

© 2013-2017 StorageOS Ltd. All rights reserved.

Why a new storage paradigm for Docker is needed

24

Not all plugins are equal, some plugins simply provide a thin wrapper around IaaS

Some plugins may not integrate properly into other platforms such as Kubernetes or Mesos

Container storage should address a variety of SLAs, performance, access and cost constraints

Container storage needs to be natively delivered to the platform not infrastructure

Storage needs to deliver repeatable processes into CI/CD pipelines with clones and snapshots

© 2013-2017 StorageOS Ltd. All rights reserved.

Addressing business continuity and HA

25

Asynchronous Replication

Public Cloud

• Reduce complexity• Reduce licensing• Reduce infra

• Highly available and replicated

• Global visibility

CRITICAL APPS REDUCE COSTS

• Operate across on-premises or public cloud infrastructure

• Business continuity maintenance and migrations

DATA ANYWHERE CONSISTENCY

I N F R A S T U C T U R E

OS Kernel

Highly Available Container Storage Pool

Container Runtime

OS Kernel

Container Runtime

OS Kernel

Container Runtime

App Containers App Containers App Containers

© 2013-2017 StorageOS Ltd. All rights reserved.

Different storage services can work together

Applications may have more than one storage requirement:1. Ephemeral, performance storage for application binaries2. Shared persistent storage for configuration 3. Dedicated persistent performance storage4. Cost efficient persistent backup storage

Applications may require different storage features:1. Performance: Replication, Encryption2. Efficiency: Compression, deduplication

© 2013-2017 StorageOS Ltd. All rights reserved. 26

27

© 2013-2017 StorageOS Ltd. All rights reserved.

CI/CD benefits from persistent storage

A continuous delivery pipeline with containerised software is a new paradigm

But it’s not easy to work with Docker volumes when you need to access data between Development, QA, Staging and Production…

CREATE A SNAPSHOT

A B C D

Current File System Snapshot

CREATE A THIN CLONE FROM A SNAPSHOT

A B C D

Snapshot Thin Clone

CHANGES POINT TO NEW DATA BLOCKS

A B C D

Snapshot Thin Clone

E FD+

Deleted data

Modifieddata

New data

unless you have snapshots and thin clones

CI/CD benefits from persistent storage

28© 2013-2017 StorageOS Ltd. All rights reserved.

• Start up thin clones of Jenkins /home

• Run tests with all images and artefacts

JENKINS

• Run unit tests against a database migration

• Start from a known state with the ability to reset and re-test

• Globally visible volumes provide instant snapshot and clones

• Provide data mobility and version control into the build pipeline

DATA MOBILITY & VERSION CONTROL UNIT TESTING

Build TestCode DeployPublish

Where do we go next?

Like getting to Mars, the first part of the journey is always the easy part, getting there – figuring out the hard part comes next

29© 2013-2017 StorageOS Ltd. All rights reserved.

… Not everyone is a rocket scientist

Containers are popular and still evolving

30© 2013-2017 StorageOS Ltd. All rights reserved.

The journey has just begun for Volume, Security and Network plugins

As we get more familiar working with containers and services, the more we’ll demand

While containers started out as simple ephemeral applications, they are now highly evolved, complex and with growing persistence use cases

31

Native Volume Plugins provide the necessary rocket fuel to get us back and complete our journey

© 2013-2017 StorageOS Ltd. All rights reserved.

So how do we complete our journey?

QuestionsJulian Topley, Senior Solutions Architectjulian.topley@storageos.com

www.storageos.com© 2013-2017 StorageOS Ltd. All rights reserved.