craig box - google kubernetes at #doxlon

KubernetesGoogle's cluster computer and you

Craig BoxSolutions Engineer

For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet.

Images by Connie Zhou

• Lightweight Linux environment

• Hermetically sealed, deployable application

• Introspectable, runnable artifact

• Recently popularized by Docker

Containers

Resource isolation

Predictability

Quality of Service

Batch vs. interactive

Containers at Google

Resource accounting

Efficient overcommit

Everything at Google runs in a container

We start over 2 billion containers a week

Managed Base OS

Node Container Manager• Common services: log rotation, watchdog restarting

Containers:• System container for shared daemons - statically defined• Dynamically scheduled containers

Cluster Scheduler• Schedules work (tasks) onto nodes• Work specified based on intents• Surfaces data about running tasks, restarts, etc. Managed Base OS

Node Container Manager

System Container

ScheduledContainers

Cluster Scheduler

The Managed Container Stack at Google

Imperative: "Start this container on that server"

Declarative: "Run 100 copies of this container with a target of <= 2 tasks down at any time"

over ImperativeDeclarative

Pro:• Repeatable• "Set it and forget it"• Eventually consistent• Easily updatable

Con:• Tracing action/reaction can be difficult. "I

made a change, is it done?"

2004 2006 2013 2014

Limitedisolation

ReleasedCGroups

ReleasedLMCTFY

cAdvisorKubernetes

Google and containers

Google:• Host bind mounts• Binary and deps built together• Signed static binaries• Interfaces to Container Manager:

Standard locations for logs, API

Docker image and environment:• More hermetic: entire chroot

is explicitly included• Less guaranteed file structure• Leverages OS distributions and package

managers

Image: "Untitled" 802 CC-BY-2.0 http://www.flickr.com/photos/802/1510186897

KubernetesPackaging containers

Why do developers care?

Repeatable, runnable artifact Loosely coupledStatic application

environment


• Portability

• Develop here, run there

• Pick your cloud solely on its merits

• Easier to build and manage

• Compose applications from micro-services

• Mix in and extend third party services

•

• Reliable deployments

• No stress deployment and update

Highly automatable = path to active management

• Efficiency: optimized packing, better scaling

• Performance: active environment tuning

• Continuous integration: easy and reliable

• Robustness: active monitoring, self healing

Why do developers care?

Node container management

Managed Base OS


Scheduled Containers

Cluster Scheduler

Container Optimized VM Image

Standardized Declarative Container Manifest

Container health monitoring and restart

User Experience

Container Env.

User App

Host GCE VM

Google Provided

OS

monitoring agent

logging agent

init / systemd sshd

docker

Managed Base OS


Scheduled Containers

Cluster SchedulerSchedule containers across machines

Replication and resizing

Service naming and discovery

Cluster scheduling

Kubernetes

• Lean, extensible & portable

• Apache 2.0 licensed

• Runs anywhere

• Manage applications, not machines

• Written in Go

Kubernetes

KubernetesMaster/Scheduler

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

Log Roller

Web Server

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Pods

Log RollerFE

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Pods

FE

FE

FE

FE

FE

FE

BE

BE

BE BEBE

BE

BEBE

BE

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

Kubernetes - Master/Scheduler

Pod Overflow

labels: role: frontend

FE

FE

FE

FE

FE

FE

BE

BE

BE BEBE

BE

BEBE

BE

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Labels

labels: role: frontend stage: production

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


FE

FE

FE

FE

FE

FE

BE

BE

BE BEBE

BE

BEBE

BE

Labels

FE FE FE FE

replicas: 4template: ...labels: role: frontend stage: production

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Replica Controller

FE


MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Replica Controller

FE FE FE


MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Replica Controller

id: backend-serviceport: 9000labels: role: backend stage: production

Backend Service

BE BE BE BE

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

MachineHost

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent

ContainerAgent


Service

• Red Hat• CoreOS• Mesosphere• IBM• Microsoft Azure• VMWare• Docker• SaltStack

Partners

• >3600 stars on GitHub

• >100 authors

http://kubernetes.io/

Kubernetes

$500 in Cloud Platform credit to launch your idea!

Build. Store. Analyze.On the same infrastructure

that powers Google.Start building!

Click ‘Apply Now’ and complete the

application with promo code: devops-con

Starter Pack

Offer Description

1

2

3

Go to http://cloud.google.com/starterpack

http://cloud.google.com/starterpack

cloud.google.com


craig box - google kubernetes at #doxlon

Technology