georgia ibm power systems user group€¦ · georgia ibm power systems user group january 15, 2019...

GeorgiaIBM POWER Systems User GroupJanuary 15, 2019

IBM Cloud Private 3.1.0Red Hat OpenShift 3.10

Steven [email protected]

What is Kubernetes

• Kubernetes - Open source orchestrator for Docker workloads

• Manages Linux nodes in a cluster

• Master

• Infrastructure

• Worker

• Single node deployment is possible, but a production cluster needs "HA" configuration

• Manages docker container workloads you deploy in the cluster

• You can set limits on cpu, memory consumed

• Assures the number of instances you specify are always kept running

• Supports x86 and Power workloads - (OCP cluster for each, ICP all in single cluster)

• Seeking on-premises, ease of application deployment, common in cloud environments

Both Red Hat OpenShift and IBM Cloud Private share this objective.

• "both have their place in the market" "both support POWER nodes"

2

Enterprises are reconsideringa pure public cloud strategy 38% of enterprises have

moved workloads from the public

cloud back to their data centers

More than half (52%) did so because

of pricing and cost concerns *

* http://www.bmc.com/content/dam/bmc/migration/pdf/Forbes_Avoid_Sticker_Shock_eBook.pdf

$

• Cost more than expected

• Did not scale as hoped

• Financial, medical records

• Competitive data exposure

• 60-80% of data is on-premise

IBM Cloud Private and Red Hat OpenShift Comparison Tables

4

Red Hat OpenShift IBM Cloud Private

Names OpenShift Container Platform (OCP)

OpenShift Enterprise (OSE)

3.10

3.11

V 3.1 ICP

Community (ce) single master, single proxy

Foundation

Cloud Native

Enterprise (ee)

Node Types - Master(s)

- Infra(s)

- Node(s)

- Masters must be known at install time

Nodes can be added or removed from

running cluster

Master(s)

Proxy(s)

Management(s)

Vulnerability Advisor(s) (VA)

Worker(s)

Master, Proxy, Mgmt, VA, must be known at

install time. If separate node(s) for Mgmt, VA are

not specified, function is merged onto Master(s)

Workers can be added or removed from running

cluster

HA Config Pacemaker, corosync prereq install

virtual IP managed by Pacemaker. Neither

product migrates single to HA multi-master

"odd" number of masters (3 or 5) proxies, mgmt,

VA nodes. cluster_vip, and proxy_vip required in

config.yaml at install. Shared /var/lib/registry,

/var/lib/icp/audit. No HA in CE edition


5


Storage NFS

Tech Preview -

• AWS Elastic Block Stores (EBS) (x86)

• GCE Persistent Disks (x86)

• Gluster FS

• iSCSI

• RADOS (Ceph)

• NFS

• Gluster FS

• vSphere

• Host path (likely where we mount GPFS,

Spectrum Scale in the worker node, and point

at such volume as "Host path")

Config files /etc/ansible/hosts (the "inventory" file)

/etc/sysconfig/docker

/opt/icp310/cluster/hosts

/opt/icp310/cluster/config.yaml

/opt/icp310/cluster/images (directory for different

architecture worker images)

/etc/docker/daemon.json

Container Engine 3.1.0 CRI-O optional

3.1.1 CRI-O default

Docker

Tech Preview -

cri containerd on 3.1.0, with Ubuntu 16.04 (?)

OpenStack function Red Hat OpenStack Platform (OSP)

PowerVC interface 1Q2019

Cloud Automation Manager / Terraforms

(CAM/TF) to PowerVC

• From Kubernetes

• Container Runtime Interface - Open Container Initiative compliant

• A lightweight, OCI-compliant container runtime

• Red Hat, Intel, IBM, SUSE and others contributing

6

Minimal and Secure Architecture

Optimized for Kubernetes

Runs any OCI-compliant image (including docker)

OpenShift - optional runtime in OCP 3.10, default OCP 3.11+

7

cri - containerd

from https://medium.com/@huimintai/run-ibm-cloud-private-on-cri-containerd-fb3d112fdca0

Also see https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/installing/setup_containerd.html

https://medium.com/@huimintai/run-ibm-cloud-private-on-cri-containerd-fb3d112fdca0

https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/installing/setup_containerd.html


8


Which to choose? • Fully engaged Red Hat Strategy

• Ansible your default provisioner/compliance

• Pacemaker already on production clusters

• Red Hat on POWER, you prefer/purchase

support from Red Hat

• If you can tolerate Separate Clusters

• x86

• Power

• PowerVM, PowerVC strategy

• Red Hat on POWER, you prefer/purchase

support from IBM

• Single cluster Power, x86, z worker nodes

• If you need cluster running in a week - ICP

Trial 30 day eval at Red Hat Portal (no support) Community Edition (CE) docker pull from

https://hub.docker.com, support via Slack

Pro longer form, more linear documentation Easier Deploy; better packaging, "HA" and

ansible built in, no separate handling

Con More exposed nuts and bolts, multiple repos,

multiple components (Pacemaker, Ansible)

handled separately. Likely more integrated at

RHEL8

Knowledge Center, overly brief articles with

many links to other overly brief articles. A

positive, copy "button" on most syntax examples

Separate Clusters?

Now you might ask,

why do Power?

Next 2 slides

https://hub.docker.com/

IBM Internal and Business Partners Only

Improved Container Density at

Lower Solution Price

1.96XPrice-Performance

IBM Power

S822LC for BD(20-core, 512GB)

HP

DL380(24-core, 512GB)

Server price-3-year warranty

$18,080 $26,711

System Cost-Server + WAS Liberty ND Annual

Subscription @ $4,606 per core (3yrs)

$110,200 $137,255

Total Throughput (tps) 48,780 33,420

Number of containers 120 76

$/container $919/container $1,805/container1.96X better

Acme Air

Open source Dockercontainers

WebsphereApplication Server

Liberty Profile

•Results are based on IBM internal testing of single system and OS image running with Acme Air work load (https://github.com/acmeair) on a private network with a dedicated JMETER driver machine and dedicated MongoDB server machine: One MongoDB instance per 8 WAS containers. Each WAS container was bound to a full core to run with 20 users and a 25ms think time between transactions. The number of containers were

increased for each system until average throughput dropped below 400 transactions/second or latency exceeded 25ms. Tests were run on November 29th, 2016. Individual results will vary depending on individual workloads, configurations and conditions. IBM Power System S822LC for Big Data; 20 cores / 160 threads, POWER8; 2.92GHz, kernel 4.4.0-12-generic, CPU frequency governor of performance, and hardware prefetch

disabled. HP Proliant DL380, 24 cores / 48 threads; Intel E5-2650 v4; 2.2 GHz; CPU frequency governor of performance, and hardware prefetch disabled. Both configurations ran Ubuntu 16.04, had 512 GB memory, included 1TB SATA 7.2K rpm HDD, 10 Gb 4-port, 1 x 16gbps FCA; Websphere Application Server V9.0 Liberty profile;Java options: -Xmx512m -Xms512m -Xgcthreads8 -Xnocompactgc -Xnoclassgc -Xconcurrentlevel0 –

Xdisableexplicitgc; Open source Docker Version: 1.12.0 / API : 1.24 / Go : 1.6.3; Docker storage driver: overlay2 and aufs had similar results. Pricing is based on: S822LC for Big Data http://www-03.ibm.com/systems/power/hardware/linux-lc.html and HP DL380 https://h22174.www2.hp.com/SimplifiedConfig/Index on December 7th, 2016. Pricing for Websphere Application Server based on 20% discount from Passport Advantage pricing

on: https://www-01.ibm.com/software/passportadvantage/pao_customer.html on December 7th, 2016

1.45XThroughput

per Server

1.57XContainers

per Server

WebSphere Application Server V9 Liberty on IBM Power S822LC for BD with open source Docker delivers 1.57X more containers and 1.96X better price-performance than Intel Xeon E5-2650 v4 Broadwell

Modernization: WAS Liberty and Db2

Why Power for Containers, and IBM Cloud Private?

Power LC922 Server: Improved Price-Performance for ClientsBetter Performance and Lower Cost running YCSB with MongoDB than tested Intel Xeon SP servers

1. Based on IBM internal testing of MongoDB 3.6.2 using YCSB workload, Results valid as of 4/11/18 and conducted under laboratory condition with speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks on both

systems, individual results can vary based on workload size, use of storage subsystems & other conditions.

2. IBM Power LC922 (2x22-core/2.6 GHz/256 GB memory) using 2 x internal HDD, 10 GbE two-port, 1 x 16gbps FCA running 4VM’s of Mongo 3.6 and RHEL 7.5 LE for Power9, running 4 VM’s Mongo 3.6 and RHEL 7.5

3. Competitive stack: 2-socket Intel Xeon SP (Skylake) Gold 6150 (2x18-core/2.7 GHz/256 GB memory) using 2 x 300GB SATA 15K rpm HDD, 10 GbE two-port, 1 x 16gbps FCA , running 3 VM’s Mongo 3.6 and RHEL 7.5

4. Pricing is based on Power LC922 http://www-03.ibm.com/systems/power/hardware/linux-lc.html and publicly available x86 pricing.

Intel Xeon SP Gold 6150322,738 Ops/sec

3 VMs @ 107,579 ops/sec

Power LC922

472,927 Ops/sec4 VMs @ 118,232 ops/sec

Power LC922

$21,878

Intel Xeon SP

Gold 6150 server:

$30,587

28%LOWER

Price2,3,4

47%MORE Performance1

Power LC922 Delivers

2XPrice-performance

+

http://www-03.ibm.com/systems/power/hardware/linux-lc.html

11

Install Doc https://docs.openshift.com/container-platform/3.10/install/index.html

Be sure to select the

right version.

Dec 2018, many 3.11

links are blank

Many Sections here.

Take to time to read it

all lightly before you

start

https://docs.openshift.com/container-platform/3.10/install/index.html

Having 2 (or more) Kubernetes based clusters could be reasonable

• Even before OpenShift, IBM Cloud Private was not our only Kubernetes based product. Others -

• Spectrum Conductor Deep Learning Impact

• a cluster focused on machine learning, with a "workflow" nature for a team of Data Scientists

• Uses Apache Spark in data source and flow. Multi-tenancy support.

• Was known as "Spectrum Conductor with Spark - DLI"

• Might still be referred to as "CwS DLI"

• Free trial available, if you have interest.

• Watson Studio Local WSL (formerly Data Science Experience Local)

• a cluster of Jupiter notebook examples, meant for collaboration among a team of Data Scientists

• Again, free trial available

• The two products above, I don’t seek to put ICP or OpenShift in front of them

• One insurer is trialing all three products, ICP, DLI, and DSXL, mid 2018

• With a lot of development and sales team oversight, for now, separate 3 node cluster for each;

• They are not trying to stack one on another.

12

SERVICE CATALOG

(LANGUAGE RUNTIMES, MIDDLEWARE, DATABASES, …)

SELF-SERVICE

APPLICATION LIFECYCLE MANAGEMENT

(CI / CD)

BUILD AUTOMATION DEPLOYMENT AUTOMATION

CONTAINER CONTAINERCONTAINER CONTAINER CONTAINER

NETWORKING SECURITYSTORAGE REGISTRYLOGS &

METRICS

CONTAINER ORCHESTRATION & CLUSTER MANAGEMENT

(KUBERNETES)

ATOMIC HOST /

RED HAT ENTERPRISE LINUX (Power)

OCI CONTAINER RUNTIME & PACKAGING

INFRASTRUCTURE AUTOMATION & COCKPIT

OpenShift = Enterprise Kubernetes+

Build, Deploy and Manage Containerized Apps

OpenShift Notable Observations

• For POWER nodes, POWER cluster only, RHEL Server rpm install; no Atomic

• Much of Red Hat 8 will be running in Containers, CRI-O engine (“Atomic” naming might not continue)

• You might be surprised that these exist; don’t include them

atomic-openshift.ppc64le

atomic-openshift-clients.ppc64le

atomic-openshift-node.ppc64le

• OpenShift on POWER, all masters and nodes share a network, on premise only (ICP equivalent)

• DNS lookup of all nodes required

• ssh public key from master to all masters and nodes; need ssh to all nodes without password

# ssh-keygen –t rsa –f ~/.ssh/id_rsa –P ''

# ssh-copy-id –i ~/.ssh/id_rsa.pub root@thisnode

# ssh-copy-id –i ~/.ssh/id_rsa.pub root@othernode1

# ssh-copy-id –i ~/.ssh/id_rsa.pub root@othernode2

...

# ssh date othernode1

14


• A number of repositories to have subscription / configuration. Red Hat fully documented

# subscription-manager repos \

--enable="rhel-7-for-power-le-rpms" \

--enable="rhel-7-for-power-le-extras-rpms" \

--enable="rhel-7-for-power-le-optional-rpms" \

--enable="rhel-7-server-ansible-2.6-for-power-le-rpms" \

--enable="rhel-7-server-for-power-le-rhscl-rpms" \

--enable="rhel-7-for-power-le-ose-3.10-rpms"

• Some OCP prep and install commands:ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift_facts.yml | tee ~/facts.txt 2>&1

ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml | tee ~/preq.txt 2>&1

ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml | tee ~/deploy5.txt 2>&1

• Typical ICP Enterprise image load and install

# mkdir -p /opt/icp2103/cluster/images ; cd /opt/icp2103/cluster/images

# wget http://10.31.193.224/icp/ibm-cloud-private-ppc64le-2.1.0.3.tar.gz

# tar -Oxvzf ./ibm-cloud-private-ppc64le-2.1.0.3.tar.gz | docker load

# cd /opt/icp2103

# docker run -e LICENSE=accept -v "$(pwd)":/data ibmcom/icp-inception:2.1.0.3-ee cp -r cluster /data

# cd /opt/icp2103/cluster

# docker run -e LICENSE=accept --net=host -t –v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:2.1.0.3-ee install | tee ~/out.txt 2>&1

15

The one ICP Ansible

playbook runs in this

install command


• Typical end of deployPLAY RECAP *********************************************************************

10.31.193.134 : ok=218 changed=71 unreachable=0 failed=0



localhost : ok=182 changed=114 unreachable=0 failed=0

POST DEPLOY MESSAGE ************************************************************

The Dashboard URL: https://10.31.193.134:8443, default username/password is admin/admin

Playbook run took 0 days, 0 hours, 29 minutes, 56 seconds

16

Typical end of

Anslible playbook,

both products

Uniquely IBM Cloud

Private

CONTAINER FUNDAMENTALS

18

A container is the smallest compute unit

CONTAINER

19

containers are created from container images

CONTAINERCONTAINER

IMAGE

BINARY RUNTIME

20

IMAGE REGISTRY

container images are stored in an image registry, or often, a local repository

CONTAINER

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

21

an image repository contains all versions of an image in the image registry

IMAGE REGISTRY

frontend:latest

frontend:2.0

frontend:1.1

frontend:1.0CONTAINER

IMAGE

mongo:latest

mongo:3.7

mongo:3.6

mongo:3.4CONTAINER

IMAGE

myregistry/frontend myregistry/mongo

22

PODPOD

containers are wrapped in pods which are units of deployment and management

CONTAINER CONTAINERCONTAINER

IP: 10.1.0.11

IP: 10.1.0.55

23

pods configuration is defined in a deployment

image name

replicas

labels

cpu

memory

storage

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

DEPLOYMENT

24

services provide internal load-balancing and service discovery across pods

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend

10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110

25

apps can talk to each other via services

InvokeBackend API

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend

10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110

26

POD

routes add services to the external load-balancer and provide readable urls for the app

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

ROUTEapp-prod.mycompany.com

> curl http://app-prod.mycompany.com

27

projects isolate apps across environments, teams, groups and departments

POD

C

POD

C

POD

C

PAYMENT DEV

POD

C

POD

C

POD

C

PAYMENT PROD

POD

C

POD

C

POD

C

CATALOG

POD

C

POD

C

POD

C

INVENTORY

❌

❌❌

MAINTAINING APPLICATION HEALTH

29

AUTO-HEALING FAILED PODS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C

RED HATENTERPRISE LINUX

MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

30

AUTO-HEALING FAILED CONTAINERS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

31


RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

32


RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

33


RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

C

C

c


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

c

34

Linux

RedHat, SLES, Ubuntu

Full IBM ICP Stack

• Access ICP Applications & Middleware

• Available Across Power Server Lines

• Available on Mixed Intel and Power Cloud

via Nutanix – Fastest Time to Cloud

• Extend existing PowerVC Clouds with ICP

ICP and OpenShift Container Platfom (OCP) on IBM Power

Red Hat Linux

Red Hat OpenShift

Kubernetes

RH OpenShift Core

Services & GUI

RH OpenShift Catalog

of Apps

• Access Openshift App Ecosystem ported to Power

• Initially Single Architecture Clusters

• Strategic Goal: Synergy with X86 Scale-Out Cloud

Full Red Hat

Stack18 ISV / OS Apps

Available

28 Apps Available (18 IBM / 10 ISV

& OS) + 13 in Plan, 10 WIP

Hybrid & Multi-cloud management

IBM Cloud Private Editions

Community

Platform

• Kubernetes (+ Helm)

• Core services

• Discover and Try Content

catalog (Containers)

Freely Available in

Docker Hub

Cloud Native Enterprise

IBM Enterprise Software

• Microservice Builder

• WebSphere Liberty

• IBM SDK for node.js

• Cloud Automation

Manager /Terraforms

Platform

• Kubernetes (+ Helm)

• Core services



Platform

• Kubernetes (+Helm)

• Core services

• Discover and Try

Content catalog

(Containers)

IBM Enterprise Software

Cloud Native Edition, plus:

• WAS ND

• MQ Advanced

• API Connect

Professional

* Available only when included with IBM Power Systems servers;

order through PPA today and through AAS after July 24th, 2018

Foundation*

(on Power bundle)

Platform

• Kubernetes (+Helm)

• Core services



Bring your own software

• Available only when

ordered with Power

Systems servers

• Entitles client to core

operational services; Cloud

Automation Manager not

included

• Upgrade to Cloud Native or

Enterprise

• Golden e-configs and sizing

guidance for easy ordering

New!

georgia ibm power systems user group€¦ · georgia ibm power systems user group january 15, 2019...

Documents