Managing Containers in Production

Brian GracelyDirector Strategy, Red Hat

@bgracely | bgracely@redhat.com

• Brian Gracely (@bgracely)

• Director Strategy, Red Hat OpenShift

• Co-Host of The Cloudcast | The ServerlessCast

• Formerly EMC {code}, Virtustream, Cisco, NetApp, Linksys, Wikibon

Creating a container on your laptop is easy. Making it work in production can be very complicated. Learn how to plan for Container Registries, CI/CD, Networking, Storage, Security, Scaling and Application Services for your containerized application.

Includes insight on Kubernetes, Cloud Foundry, Docker Swarm and Mesos.*

* Deferred to some URLs for your reading pleasure (on the next slide)

Comparing Container Schedulers

• Kubernetes vs. SwarmKit : https://platform9.com/blog/compare-kubernetes-vs-docker-swarm/

• Kubernetes vs. Mesos : https://platform9.com/blog/compare-kubernetes-vs-mesos/

• Cloud Foundry Diego Architecture : https://docs.cloudfoundry.org/concepts/diego/diego-architecture.html

What’s the Ultimate Goal?

APITraffic

LB

APP1

APP2 Microservices

DATA

DATA

DATA

Container Usage Grows Quickly

Eventually You Need a Platform / Orchestrator

Containers vs. Platforms

The Battle for Container Orchestration

CONFIDENTIAL - FOR INTERNAL USE ONLY

Kubernetes Mesos Others

Cloud Foundry Diego

AWS Blox

Rancher Cattle

VMware Admiral

HashiCorp Nomad

CoreOS Fleet

SwarmKit

PLATFORM ARCHITECTURE

SCM

(GIT)

CI/CD

Platform Architecture

EXISTING

AUTOMATION

TOOLSETS

NEW

AUTOMATION

TOOLSETS

Infrastructure Choices

“Node” Instances – Where Applications Run

Applications Run in Containers

Container

Image

Container

Pod

“Pods” are the Unit of Orchestration

16

“Masters” are the Control Plane

17

Master - API and Authenticator

18

Master – Desired and Current State

19

Master - Orchestration and Scheduling

20

Master – Node Placement by Policy

21

Master – Service Discovery

NETWORKING

23

• Pluggable routing architecture• HAProxy Router• F5 Router

• Multiple-routers with traffic sharding

• Router supported protocols• HTTP/HTTPS• WebSockets• TLS with SNI

• Non-standard ports via cloud load-balancers, ExternalIP, and NodePort

ROUTING AND LOAD-BALANCING

24

ROUTE SPLIT TRAFFIC

SERVICE A

App A App A

SERVICE B

App B App B

ROUTE

10%

traffic

90% traffic

Split Traffic Between Multiple

Services For A/B Testing,

Blue/Green and Canary

Deployments

NODE

IP-3

NODE

IP-2

NODE

IP-1

25

EXTERNAL TRAFFIC WITH EXTERNALIP

SERVICE

EXT: IP-10:8080

INT: INT-IP:8080

EDGE ROUTERS

IP-10, IP-11, IP-12

POD

Port: 8080

POD

Port: 8080

POD

Port: 8080

IP

FAILOVER

POD

IP

FAILOVER

POD

connect

IP-10:8080

CLIENT• Route external traffic to a

service on any TCP/UDP port

• Available on non-cloud clusters

• External IP automatically

assigned from a pre-defined

pool of external IPs

• IP failover pods provide high

availability for the pool of

external IPs

• NodePort exposes a unique port

on all the nodes in the cluster

• Ports in 30K-60K range which

usually differs from the service

• Traffic received on any node redirects to a node with the running service

• Firewall rules must allow traffic to

all nodes on the specific portNODE

IP-3

NODE

IP-2

NODE

IP-1

26

EXTERNAL TRAFFIC WITH NODEPORT

SERVICE

INTERNAL-IP:8080

NODEPORT: 32010

POD

Port: 8080

POD

Port: 8080

POD

Port: 8080

connect

IP-1:32010

CLIENT

27

CONTROL SOURCE IP WITH EGRESS ROUTER

NODE

IP1

EGRESS

ROUTER

POD

IP1

EGRESS

SERVICEINTERNAL-IP:8080

EXTERNAL

SERVICE

Whitelist: IP1

POD

POD

POD

28

• Built-in DNS to enable reaching services by DNS

• Split DNS is supported via SkyDNS• Master answers DNS queries for internal services• Other nameservers serve the rest of the queries

• Software Defined Networking (SDN) for a unified cluster network to enable pod-to-pod communication

• OpenShift follows Kubernetes network plug-in model

(CNI)

• Supported plug-ins

• OpenShift SDN (Open vSwitch or Flannel)

• Nuage SDN (Virtualized Services Platform)

PLATFORM NETWORKING

PERSISTENT STORAGE

30

PERSISTENT STORAGE

• Persistent Volume• Tied to a piece of network storage• Provisioned by an administrator (static or

dynamically)• Allows admins to describe storage and users to

request storage

NFS GlusterFSOpenStack

CinderCeph RBD

AWS

Elastic

Block Store

(EBS)

GCE

Persistent

Disk

iSCSIFibre

Channel

PROJECT

POOL OF PERSISTENT VOLUMES

31

PERSISTENT STORAGE – PERSISTENT VOLUMES (PV)

NFS

PV

iSCSI

PV

NFS

PV

Admin

User

register PV

create claim

NFS

PVGlusterFS

PV

Pod

claim

Pod

claim

Pod

claim

Ceph

RBD

PV

32

DYNAMIC VOLUME PROVISIONING

Admin

User

define StorageClass

create claim: Fastest

SlowAzure-Disk

FastAWS-SSD

FastestNetApp-Flash

NetApp

Provisioner

AWS

Provisioner

Pod

claim

PV

Kubernetes

PV Controller

provision

Azure

Provisioner

bound

LOGGING & METRICS

CENTRALIZED LOG MANAGEMENT

APPLICATION LOGS

OPERATION LOGS

ELASTICELASTIC

34

RHEL

NODE

POD POD

PODPOD

FLU

ENTD

RHEL

NODE

POD POD

PODPOD

FLU

ENTD

ELASTICSEARCH

RHEL

NODE

POD POD

PODPOD

FLU

ENTD

USER

ELASTICELASTIC

KIBANA

ELASTICELASTIC

ELASTICSEARCH

ELASTICELASTIC

KIBANA

ADMIN

35

CONTAINER METRICS

CONTAINER METRICS

36

RHEL

NODE

POD POD

PODPOD

FLU

ENTD

CONTAINER METRICS

RHEL

NODE

POD POD

PODPOD

FLU

ENTD

HEAPSTER

RHEL

NODE

POD POD

PODPOD

CA

DV

ISO

R

HAWKULAR WEB CONSOLE

ELASTICELASTICCASSANDRA

CLOUD

MANAGEMENT

CUSTOM

DASHBOARDS

API

USER

SECURITY

38

TEN LAYERS OF CONTAINER SECURITY

Container Host & Multi-tenancy

Container ContentContainer Registry

Building Containers

Deploying Container

Container Platform

Network Isolation

Storage

API Management

Federated Clusters

NODE

MASTER

• Secure mechanism for holding sensitive data e.g.

• Passwords and credentials• SSH Keys• Certificates

• Secrets are made available as• Environment variables• Volume mounts• Interaction with external systems

• Encrypted in transit

• Never rest on the nodes

39

SECRET MANAGEMENT

Container

Distributed Store

Container

OPERATIONAL MANAGEMENT

41

TOP CHALLENGES OFRUNNING CONTAINERS AT SCALE

SERVICE

HEALTH

SECURITY

&

COMPLIANCE

FINANCIAL

MANAGEMENT

OPERATIONAL

EFFICIENCY

42

OPERATIONAL EFFICIENCY

43

SERVICE HEALTH

44

SECURITY & COMPLIANCE

45

FINANCIAL MANAGEMENT

BUILD & DEPLOYCONTAINER IMAGES

47

BUILD AND DEPLOY CONTAINER IMAGES

DEPLOY YOUR

SOURCE CODE

DEPLOY YOUR

APP BINARY

DEPLOY YOUR

CONTAINER IMAGE

48

DEPLOY YOUR

SOURCE CODE

49

DEPLOY SOURCE CODE

codeGit

Repository

Source-to-Image (S2I)

deploy

CODE

BUILD

DEPLO

Y

Container

Image

Image

Registry

OP

SD

EV

Application

Container

Build Packs

DEPLOY APP BINARY

50

DEPLO

Y

51

DEPLOY APP BINARY

build Application

Binary

(e.g. WAR)

Source-to-Image (S2I)

deploy

Container

Image

Image

Registry

OP

SD

EV

BINARY

BUILD IMAGE

Existing Build

Process

Application

Container

Build Packs

DEPLOY CONTAINER IMAGE

52

DEPLO

Y

53

DEPLOY CONTAINER IMAGE

build

Application

Container

deploy

Application

Image

Image

Registry

OP

SD

EV

BUILD

Existing Image

Build Process

CONTINUOUS INTEGRATION (CI)CONTINUOUS DELIVERY (CD)

CI/CID WITH BUILD AND DEPLOYMENTS

55

BUILDS

• Webhook triggers: build the app image whenever the code changes

• Image trigger: build the app image whenever the base language or app runtime changes

• Build hooks: test the app image before pushing it to an image registry

DEPLOYMENTS

• Deployment triggers: redeploy app containers whenever the image changes in the Kubernetes image registry or upstream registries

56

CONTINUOUS DELIVERY WITH CONTAINERS

source

repository

CI/CD

engine

dev container

physical

virtual

private

cloud

public cloud

CONTINUOUS DELIVERY PIPELINE

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

57

DEVELOPER GIT SERVERARTIFACT

REPOSITORY

OPENSHIFT

CI/CD PIPELINE

(JENKINS)IMAGE BUILD

& DEPLOY

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

NON-PROD PRODDEV

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

58

CONTINUOUS DELIVERY PIPELINE

DEVELOPER GIT SERVERARTIFACT

REPOSITORY

OPENSHIFT

CI/CD PIPELINE

(JENKINS)IMAGE BUILD

& DEPLOY

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

PROMOTE

TO TEST

NON-PROD PRODDEV TEST

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

59

CONTINUOUS DELIVERY PIPELINE

DEVELOPER GIT SERVERARTIFACT

REPOSITORY

OPENSHIFT

CI/CD PIPELINE

(JENKINS)IMAGE BUILD

& DEPLOY

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

PROMOTE

TO TEST

PROMOTE

TO UAT

NON-PROD PRODDEV TEST UAT

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

60

CONTINUOUS DELIVERY PIPELINE

DEVELOPER GIT SERVERARTIFACT

REPOSITORY

OPENSHIFT

CI/CD PIPELINE

(JENKINS)IMAGE BUILD

& DEPLOY

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

GO

LIVE?

PROMOTE

TO TEST

PROMOTE

TO UAT

RELEASE

MANAGER

NON-PROD PROD

☒

☑

DEV TEST UAT

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

61

CONTINUOUS DELIVERY PIPELINE

DEVELOPER GIT SERVERARTIFACT

REPOSITORY

OPENSHIFT

CI/CD PIPELINE

(JENKINS)IMAGE BUILD

& DEPLOY

INTEGRATED

IMAGE

REGISTRY

KUBERNETES

CLUSTER

GO

LIVE?

PROMOTE

TO TEST

PROMOTE

TO UAT

PROMOTE

TO PROD

RELEASE

MANAGER

NON-PROD PRODDEV TEST UAT

☒ ☑

KEY TAKEAWAYS

• Containers are relevant for both developers and operators, but in different ways. Establishes a common language for deployments.

• Containers are the new packaging model for both existing and new applications.

• Container platforms operationalize the management of containers.

• Container workloads need a new way of thinking about networking, storage, monitoring, logging, etc.

Key Takeaways

NEXT STEPS

• Learn about Containers – PluralSight (Docker, Nigel Poulton)

• Learn about Orchestration - https://katacoda.com/ (Kubernetes, OpenShift, Docker, etc.)

• Attend local Meetups to Network – http://meetup.com

• Setup a Local (or cloud) Environment for FREE – Docker for Mac/Windows, MiniKube, MiniShift, AWS Quick Starts, etc.

Next Steps – Getting Smarter

THANK YOU