docker volumes with beegfs - qnibqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 ·...
TRANSCRIPT
![Page 1: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/1.jpg)
Docker Volumes with BeeGFS
Jasper Lievisse Adriaanse
Docker: Linux Containers to Optimise IT Infrastructure for HPC & BigData
[June 23, 2016]
![Page 2: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/2.jpg)
[Agenda]
Introduction
Docker
VolumesIntroductionTypesPlugins
BeeGFSBeeGFSDockerized stackBeeGFS plugin
Demo
Q&A
![Page 3: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/3.jpg)
[Introduction] Who am I?
CTO for RedCoolBeansOpenBSD developer
[email protected] jasper_la , jasperla
![Page 4: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/4.jpg)
[Introduction] RedCoolBeans
I Secure Docker stacksI CargOS, Dockerlint, Crane
RedCoolBeans @RedCoolBeans RedCoolBeans.com and CargOS.io
![Page 5: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/5.jpg)
[Docker] Adoption Hurdles
Two primary challenges:I Networking
I Storage
![Page 6: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/6.jpg)
[Docker] Adoption Hurdles
Two primary challenges:I NetworkingI Storage
![Page 7: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/7.jpg)
[Docker] Networking
I LinksI Multi-host setups
I Ambassador pattern
![Page 8: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/8.jpg)
[Docker] Networking (cont.)
Two “solutions”:I WeaveI docker-compose
Docker 1.9 network subcommand
![Page 9: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/9.jpg)
[Docker] Data persistence
Containers are ephemeral, data is forever (sort of)Not all containers are stateless
I DatabasesI CMS uploadsI …
![Page 10: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/10.jpg)
[Volumes] Purpose of volumes
I Store data outside of a container
I Share data between containersI Backup/replicationI Utilize external/clustered filesystems
![Page 11: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/11.jpg)
[Volumes] Purpose of volumes
I Store data outside of a containerI Share data between containers
I Backup/replicationI Utilize external/clustered filesystems
![Page 12: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/12.jpg)
[Volumes] Purpose of volumes
I Store data outside of a containerI Share data between containersI Backup/replication
I Utilize external/clustered filesystems
![Page 13: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/13.jpg)
[Volumes] Purpose of volumes
I Store data outside of a containerI Share data between containersI Backup/replicationI Utilize external/clustered filesystems
![Page 14: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/14.jpg)
[Volumes]
Data volumesI docker run -v /data:/data […]
Data volume containersI docker run --volumes-from data-container […]
![Page 15: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/15.jpg)
[Volumes] Types of volumes
Data volumes on host (host volumes)I docker run -v /docker/db:/db […]
![Page 16: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/16.jpg)
[Volumes] Types of volumes (cont.)
Data volumesAnonymous volume
I docker run -v /data […]I Data lifetime matches container’s
Named volumeI docker run -v datavolume:/data […]I Data lifetime exceeds container’s
![Page 17: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/17.jpg)
[Volumes] Types of volumes (cont.)
Data volumesAnonymous volume
I docker run -v /data […]I Data lifetime matches container’s
Named volumeI docker run -v datavolume:/data […]I Data lifetime exceeds container’s
![Page 18: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/18.jpg)
[Volumes] Types of volumes (cont.)
Data volume containersI docker run --volumes-from data-container […]I Store data in containers
![Page 19: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/19.jpg)
[Volumes] Security implications
I No restrictions on sharingI SELinux-labelingI Boot2docker situation
![Page 20: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/20.jpg)
[Volumes] Plugins
I docker volume subcommand/APII Arbitrary backend providerI Plugins for:
I NFS, CIFS, EFSI GlusterFS, IPFS
I BeeGFS
![Page 21: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/21.jpg)
[Volumes] Plugins
I docker volume subcommand/APII Arbitrary backend providerI Plugins for:
I NFS, CIFS, EFSI GlusterFS, IPFSI BeeGFS
![Page 22: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/22.jpg)
[BeeGFS]
I Open source, highly performant cluster filesystem
I Easy to setupI Excels on commodity hardwareI Developed by Fraunhofer institute
![Page 23: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/23.jpg)
[BeeGFS]
I Open source, highly performant cluster filesystemI Easy to setup
I Excels on commodity hardwareI Developed by Fraunhofer institute
![Page 24: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/24.jpg)
[BeeGFS]
I Open source, highly performant cluster filesystemI Easy to setupI Excels on commodity hardware
I Developed by Fraunhofer institute
![Page 25: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/25.jpg)
[BeeGFS]
I Open source, highly performant cluster filesystemI Easy to setupI Excels on commodity hardwareI Developed by Fraunhofer institute
![Page 26: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/26.jpg)
[BeeGFS] Overview
Figure: Architecture overview (source: beegfs.com)
![Page 27: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/27.jpg)
[Dockerized stack] Services
I ManagementI MetadataI Storage
I Client
![Page 28: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/28.jpg)
[Dockerized stack] Services
I ManagementI MetadataI StorageI Client
![Page 29: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/29.jpg)
[Dockerized stack] Availability
I DockerHubI github.com/RedCoolBeans/docker-beegfs
![Page 30: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/30.jpg)
[BeeGFS Plugin]
I github.com/RedCoolBeans/docker-volume-beegfsI Available for:
I RedHat (+CentOS, &c)I Debian (+Ubuntu, &c)
![Page 31: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/31.jpg)
[BeeGFS plugin] Requirements
I Docker 1.8I Plugin service runningI Kernel module loadedI BeeGFS share mounted
![Page 32: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/32.jpg)
[BeeGFS plugin] How does it work?
I Bridge to Docker API (docker volume)I Maps volume name to on-disk nameI Checks underlying filesystem type
![Page 33: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/33.jpg)
[BeeGFS plugin] Use cases
I Multi-host Docker clusterI Redudant storageI (Live-)migrationI CI/CD
![Page 34: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/34.jpg)
[Demo]
Demo
![Page 35: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack](https://reader034.vdocuments.mx/reader034/viewer/2022042223/5ec9a158f5d25d6a6e31bc15/html5/thumbnails/35.jpg)
[Q&A]
Questions?