2016/7/28YJ AmericaNorifumi MatsuyaOCP, Kubernetes

YJ America1

2

74%Yahoo! JAPAN

3

Yahoo! JAPAN26%74%

4

26%74%

Yahoo! JAPAN

5

6

2015 (http://www.enecho.meti.go.jp/about/whitepaper/2015gaiyou/whitepaper2015pdf_h26_nenji.pdf): Map of average US residential electricity price by utility service territory (http://en.openei.org/wiki/File:2012_12_14_Electricity_Price-01.jpg)22%

1/67

201410Yahoo! JAPAN2015454

8

201410Yahoo! JAPAN2015454

9

19

10

19OpenStack

OpenStackOpenStack2010

11

19

12

OpenStack13

14

OpenStackOperating system15

16

OpenStack17

YJ America

YJ America18

19

20

21

22

23

0

GoogleFacebook01024

25

26

27

OSS

OSS28

OSS

29

30

31

Hadoop

Y AmericaHadoop32

YJ AmericaHadoop33

34

35

AB

36

Open Compute Project

Facebook201150

Open Compute Project OCP 2011FacebookOCPFacebook37

FacebookOCP

38

25,000/15

Facebook2500039

40

41

OCP

42

43

FacebookOCP44

HadoopOCP

White box switchSwitchOCPServerServerOCPFacebook

OCP

hadoopOCPOCP45

OEM/ODM

46

47

Docker48

: Kubernetes on OpenStack at eBay, Aswin Nair, eBay (https://www.youtube.com/watch?v=l5HpUNhpKwU)

OpenStack Day Seattle 2015Kubernetes on OpenStack @eBay (OpenStack)

OpenStack 49

eBayOpenStackKubernetes: Kubernetes on OpenStack at eBay, Aswin Nair, eBay (https://www.youtube.com/watch?v=l5HpUNhpKwU)

OpenStack Day Seattle 2015Kubernetes on OpenStack @eBay (OpenStack)

OpenStack Day SeattleeBayKubernetes on OpenStackKubernete50

BorgKubernetesGoogle Borg10Google2billion/week launch

: Large-scale cluster management at Google with Borg (https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/43438.pdf)

KubernetesGoogle51

BorgKubernetesGoogle Borg10Google2billion/week launch

: Large-scale cluster management at Google with Borg (https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/43438.pdf)

BorgKubernetes52

Kubernetes

Kubernetes53

Code ControlImage CreationDeploy to Platform

OpenStack54

Code ControlImage CreationDeploy to Platform

55

AppdeployTool chain

Image creationCode controlDeployment Platform

Bare Metal & VMContainer

GitHubOpenstackKubernetedeploy56

Image creationCode controlDeployment Platform

Bare Metal & VMContainer

Tool chain

57

Code CodeManagementInstance

AuthenticationPersistentVolumeContainerEngine Container ClusterManagerContainerNetworkingServiceRegistry

ContainerRegistryInfrastructureProvisioning

APP

APP

APP

loggingMetricsHeapsterCI tool

ServiceMonitoring

Kubernetes on OpenStackGitHubetcdCI toolKubernetesKubernetesOpenStack

Manage application, Not MachinesKubernetesIaaSIaaS

Kubernetesmanage application not machinensOpenstack

58

Image creationCode controlDeployment Platform

Bare Metal & VMContainerContainer Image creation flow

Artifactory

ImagedeployOSk8s

59

MasterSlave

Service Job

DockerfilerepositoryBuild.shdockerbuild

DockerimageDcokerimageDocker HUB

NewDockerimage123458697Container Image creation flowGitHubCodepushJenkinsMasterJenkinsslaveDockerfile repositorycheckoutService JobArtifactorybase docker imageArtifactorydocker imageDocker HubDocker buildimageimageArtifactory

Kubernetesdocker60

Image creationCode controlDeployment Platform

Bare Metal & VMContainerdeployKubernetes

Kubernetes Architecture

deployKuberntes61

VMKubernetes Master

VM

KeystoneCinder

VMetcd proxykubelet

kube proxy

Pod

App

App

App

Pod

App

App

Pod

App

App

App

Pod

App

Kubernetes Node

P

CalicoDocker enginePodetcd proxykubelet

kube-apiserver

P

CalicoDocker enginePodkube-scheduler

Podkube-controller-manager

Pod

VMetcd proxykubelet

kube proxy

Pod

App

App

App

Pod

App

App

Pod

App

App

App

Pod

App

Kubernetes Node

P

CalicoDocker enginePodMasterNodePodKubernetesKeystoneKeystoneKubernetespolicyCinderProject CalicoKubernetes Architecture

62

VM

Kubernetes Master

VM

Loadbalancer Node

VM

Pod

App

App

Kubernetes Node

VM

Pod

App

Kubernetes Node

Pod

App

AppiBGPiBGPiBGPiBGPRoute Reflector & Gateway

Redistribute ClusterIP and Pod IPs to Backbone

Advertise Pod IPs(/26)

Advertise ClusterIP

Advertise PodIPs IP(/26) ClusterIP range : 10.0.0.0/22Pod IP range: 192.168.0.0/22Node IP range: 172.16.0.0/22 Kubernetes NetworkingRoute ReflectorVMiBGPNodePodBlackholeiBGPPodIPPodBackbonePod

63

192.168.0.0/24

ClusterIP: 10.0.0.100

200.0.0.100Src addressDst address200.0.0.10010.0.0.100172.16.0.100192.168.0.100192.168.0.100172.16.0.10010.0.0.100200.0.0.100

VM

Loadbalancer Node

VM

Kubernetes Master

VM

Pod

App

App

Kubernetes Node

Pod IP: 192.168.0.100Node IP : 172.16.0.100ClusterIP range : 10.0.0.0/22Pod IP range: 192.168.0.0/22Node IP range: 172.16.0.0/22 External Service Load BalancingKubernetesExternalInternalExternal

1. ClusterIPNode2. DNAT/SNATSNATKubernetes3. Node4. NAT

External service load balancing64

Docker registry

Jenkins Master

Launch Jenkins slave Podand run commandhookUpload Artifactorypull repository

issue trackingtrack commitbuild resultCI support Kubernetes Cluster

masterdeploy

APP

CinderPersistentVolume

KeystoneAuthTenant Isolation

pushLaunch Pod

pull image

master

APP

CinderPersistentVolume

KeystoneAuthTenant IsolationLaunch Pod

Kubernetes Cluster #1for Datacenter A

Nova

NovaBoot Kubernetes nodeOpenStackAdmin

Dockerbuild

masterBoot Kubernetes nodeKubernetes Cluster #2for Datacenter B

Code ControlImage CreationDeploy to PlatformOpenStack30

OpenStack on Kubernetes on OpenStack

image creationkuberntestool chaintool chainOpenstackOpenstack65

APP

Openstack Cluster #1for Datacenter A

Glance

CI support Kubernetes Cluster

pushOpenStackAdmin

qcow2ImageTeraformPackerCode ControlImage CreationDeploy to Platform

KubernetesImage deploy

APP

Openstack Cluster #2for Datacenter B

Glance

VM image creationkubernetes

66

67

Techblog- Kuberneteshttp://techblog.yahoo.co.jp/infrastructure/os_n_k8s/

- Open Compute Projecthttp://techblog.yahoo.co.jp/operation/2015-10-ocp/

68