Anusha Ragunathan

Provisioning and Management

of Storage in the Docker

platform

Sr Software Engineer, Docker

Agenda

Docker Volume Plugins (DVP) on Engine &

Swarm

Docker Enterprise & Kubernetes

Storage Ecosystem

Storage Ops is HardA key demand enterprises have is that developers should be able to store data

in clusters without having to worry about how persistent storage is working

under the hood.

Provision

Volumes

Deploy

AppsHA DR

Understand

Your

use-case

Map

use-case to

plugins

Life of a volume: Single node

CreateVolume

UnMountVolume (from

container)

MountVolume (to container

and format if necessary)

DeleteVolume (after use)

A way to extend the docker engine's volume subsystem on a

single node. An out-of-tree implementation for:

● Create/Remove volumes

● Mount/Unmount volumes

● Get Volume Capabilities

● List volumes

Examples: DVP include NetApp, VMware vSphere,

PureStorage plugins.

Docker Volume Plugins: Engine

Life of a volume: Cluster awareCreateVolume

AttachVolume (to Node)

Format and partition volume, if necessary

MountVolume (to container)

UnMountVolume (from container)

DetachVolume (from Node)

DeleteVolume (after use)

Docker Swarm does not orchestrate volume creation across a

cluster.

• Docker Swarm makes use of volume names instead of

volume ID as unique volume identifier

• Volume requests are simultaneously send to each node in a

Swarm cluster.

• Volume Plugins must run independently on each node in a

Swarm cluster.

These results in issues like race conditions that can result in

the creation of a large number of volumes for a single request

without a clear winner.

Docker Volume Plugins: Swarm

● Around the same time that Docker Swarm was considered

getting re-architected to be volume aware, a new industry

standard was evolving in the container orchestration

community.

● Container Storage Interface (CSI) started and matured by

contributions from Docker, Kubernetes and Mesos.

● Today, CSI specification has reached 1.0. Container

Orchestrators such as Kubernetes and Mesos have

supported for CSI.

● Docker Enterprise is a strong integrator for Kubernetes and

works well with CSI and storage providers.

Container Storage Interface

Docker Enterprise + Kubernetes

● A storage claim made by a

user

● Just like how Pods consume

Compute resources, PVC

consume Volume resources.

● Just like how Pods can

request specific levels CPU &

memory, PVCs can request

specific sizes and access

modes.

● Pods reference PVC

Persistent Volume

Claim (PVC)

Persistent Volume (PV)

● Storage resources in a cluster

● Lifecycle independent of a Pod

Pods

● A set of running containers

representing a workload

● Provisioning is the creation/allocation of Persistent Volumes

● Static and Dynamic Provisioning

○ Static: pre creation of PV

○ Dynamic: automatic creation of PV based on size,

permissions requested.

● Dynamic Provisioning through Storage Classes

○ Provides a way for Admins to describe “classes” of

storage available. Eg, different performance SLAs,

value-add features such as replication, backup, etc

○ Backed by a provisioner

PV Provisioning

● Provisioning is the creation/allocation of Persistent Volumes

● Static and Dynamic Provisioning

○ Static: pre creation of PV

○ Dynamic: automatic creation of PV based on size,

permissions requested.

● Dynamic Provisioning through Storage Classes

○ Provides a way for Admins to describe “classes” of

storage available. Eg, different performance SLAs,

value-add features such as replication, backup, etc

○ Backed by a provisioner

PV Provisioning

Provisioning Workflow (static)

1. Cluster Admin

pre-provisions

volumes and

registers PVs

2. Developer

claims a PV

from the pool

4. Developer

references the

claim in a Pod

3. Controller BINDS

PV to PVC

Pool of Persistent Volumes

NFS PV iSCSI PV EBS PV

Claim

`Claim

ClaimClaim

Pod

Claim

5. Controller inspects

Claim and MOUNTs

the PV into the Pod.

Provisioning Workflow (dynamic)

1. Cluster Admin

registers

Storage Classes

2. Developer defines a

claim by referring a

storage class

5. Developer

references the

claim in a Pod

4. Controller BINDS

PV to PVC

6. Controller

inspects Claim

and MOUNTs the

PV into the Pod.

SSD

Pod

Claim

Claim

Persistent

VolumesStorage

Classes

Slow

Fast

3. Controller

provisions

volumes

on-demand

Demo 1: PV Provisioning using iSCSI

Worker Worker

Master

ISCSI Target Server

High Availability (HA)● HA involves handling Node failures and Storage failures.

● Node failures:

○ Node drain: Maintenance

○ Node failures: Node lost/ kubelet crash

● Kubernetes provides built-in support for node failures. The

attach-detach controller on the master is in charge of

interacting with the volume plugin and moving PVs across

nodes.

● If delays/instability with attach/detach is not desired, use a

Software Defined Storage (SDS) solution.

Plugin Type Pros Cons Future

InTreeNo extra installation

necessary

Release cycles tied to

k8s releasesSuperseded by CSI

External

Provisioner

Code maintained

independently

Limited customization for

attach and mount opsSuperseded by CSI

FlexVolumeHighly customizable code

maintained independently

Host based exec model

reduces portability

Deprecated for Linux

Will be used for Windows

CSI

Highly customizable code

maintained independently

and based on standard

Requires installation and

configuration

The future of storage

plugins

Kubernetes Storage Plugin Ecosystem

In-tree Plugin Scenarios

Cloud Protocol Ephemeral SDS Extensions

AWS EBS NFS local Portworx FlexVolume

GCE PD iSCSI hostpath StorageOS CSI

Azure Disk/File Fibre Channel config_map ScaleIO

Openstack Cinder secret

vSphereVolume

In tree Plugin Architecture

Worker Worker

Master

Storage Service

Kubelet Kubelet

AttachVolume

DetachVolume

CreateVolume

DeleteVolume

Kube Core Components

MountVolume

UnmountVolume

AttachVolume

DetachVolume

CreateVolume

Delete Volume

In-tree Plugins

PV Claim

API Server

External Provisioner Scenarios

In-tree PV source Implementations [not certified with UCP yet]

iSCSINetapp-Trident, Dell/EMC-Isilon, HPE-Nimble, HPE-3PAR, Nutanix,

OpenEBS-iscsi

NFS Netapp-Trident, AWS-EFS, Dell-EMC Isilon

External Provisioner Architecture

Worker Worker

Master

Storage Service

Kubelet Kubelet

External Provisioner (StatefulSet/Deployment Pod)

Kube Core ComponentsExternal components

API Server

MountVolumeUnmountVolume

AttachVolumeDetachVolume

CreateVolume

Delete Volume

In-tree Plugins

PV Claim

PV

CSI Scenarios

Cloud Local/ephemeral

On-prem SAN/NAS SDS

CSI Plugins

Worker Worker

Storage Service

Kubelet Kubelet

External

Provisioner(Deployment/

StatefulSet)

CreateVolume

DeleteVolume

External

Attacher (Deployment/

StatefulSet)

CSI Controller (Deployment/

StatefulSet)

CSI Node(DaemonSet)

Kube Core Components Kube Sidecar containers CSI plugin components

ControllerPublish

ControllerUnPublish

Master

API Server

NodeStage/NodeUnstage NodePublish/NodeUnpublish

PV

PV Claim

volumeattachment

CSI In-

Tree

Future enterprise features in CSI

❖Backup

➢Snapshot/restore

➢Application consistent/triggered snapshots

❖Volume resizing

❖Cloning and Replication

Demo 2: PV Provisioning using CSI AWS EBS

Worker Worker

Master

AWS EBS Service

OS specific considerations

● Cluster may support Linux and Windows nodes

● Cluster-scoped operations can be OS agnostic

▪ Create, Delete

▪ Attach, Detach

● Node-scoped operations need to be OS aware

▪ Device enumeration

▪ Format, Mount, Dismount

OS specific considerations

● File system support varies across OS

▪ NTFS and SMB in Windows

▪ ext, xfs and NFS in Linux

● Support for privileged containers absent in Windows

▪ Plugins running directly on host have no problems

▪ Fully containerized plugins cannot function

▪ Host proxy to support CSI plugins being investigated

Persistent storage needs for containerized workloads is available for different

environments supporting a variety of use cases

The plugin model is standardized through a common industry standard spec -

CSI

Certify as many plugins as possible with UCP

to provide more choice to our customers

Summary

Thank You!

● Creating and Deleting Volumes

○ Handled by external-provisioner sidecar

○ Watches PersistentVolumeClaim objects and triggers CreateVolume and

DeleteVolume operations.

● Attaching and Detaching

○ Handled by external-attacher sidecar, unlike the attach-detach controller on the kube

master in external provisioners.

○ Watches VolumeAttachment objects and triggers ControllerPublish and

ControllerUnpublish against a CSI endpoint

● Mounting and Unmounting (into a Node)

○ Triggered by external attacher

○ NodeStageVolume, NodeUnstageVolume

● Mounting and Unmounting (into a Pod)

○ Triggered by external attacher

○ NodePublishVolume, NodeUnpublishVolume

CSI Operations [backup details]

CSI: Create PVC request

Kube core

external provisioner

CSI controller

CreatePersistentVolumeClaim

API request

Reference CSI plugin as provisioner in claim

Watches ‘PVC’ objects; triggers ‘CreateVolume’ CSI API call

Dispatch request to appropriate provisioner

AWS EBS

Service

API call to actual storage service to Create

the Volumes

EBS Volume

Created

CSI: Pod refers to PVC

Kube core

external attacher

CreatePod API request; Pod refers to PVC

Refer CSI plugin as provisioner in PVC

Watches ‘VolumeAttachment’ objects; triggers

‘ControllerPublishVolume’ CSI API call

Dispatch request to appropriate provisioner

API call to storage service to ‘Attach’ the

Created Volume to a specific node.CSI Controller

CSI Node

1. ControllerPublishVolume CSI API

Call

2. NodeStageVolume CSI API Call

3. NodePublishVolume CSI API Call

Mounts the attached Volume to a

‘Staging-Path’. This path is a global

directory on the node;

Bind mounts volumes from ‘Staging-Path’

to ‘Target-Path’ in a Pod. This enables

multiple Pods to refer to the same volume.

External Provisioner Operations

Operation Invocation

Provision/De-provisionPV claim => External Provisioner => PV object with In-Tree volume source

(iSCSI/NFS)

Attach/Detach AD controller => PV’s volume src plugin interface

Mount/Unmount Kubelet volume manager => PV’s volume src plugin interface

In-tree Plugin Operations

Operation Invocation

Provision/De-provision PVC => PV controller => plugin interface

Attach/Detach AD controller => plugin interface

Mount/Unmount Kubelet volume manager => plugin interface

CSI Operations

Operation Invocation

Provision/De-provision PV claim => CSI External Provisioner => CSI Plugin => PV with CSI volume source

Attach/Detach CSI in-tree => VolumeAttachment => CSI External Attacher => CSI Plugin

Mount/Unmount Kubelet CSI client => CSI Plugin

Backup [Alpha] Volume Snapshot object => CSI External Snapshotter => CSI Plugin

Restore [Alpha] PV claim => CSI External Provisioner => CSI Plugin