deploying an openshift cluster with “vmware cloud assembly”
Post on 23-Jan-2022
26 Views
Preview:
TRANSCRIPT
#vmworld
CODE3455U
Deploying an OpenShift Cluster with “VMware Cloud Assembly”
Sajal Debnath, VMware, Inc. Rafael Brito, VMware, Inc.
#CODE3455U
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
Disclaimer
This presentation may contain product features or functionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
2
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc.
Agenda
3
Introduction
Overview of the overall Solution
Overview of Red Hat OpenShift (OCP)
Ansible Inventory File
“SSH Equiv” Helper for Ansible
OCP Install and Post-Install
Decisions and assumptions for Implementation
Blueprint & Other Configurations
Extensibility Workflows & Scripts
Yet to Do
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 4
• Joined VMware (Office of CTO) in early 2019.
• Before at Citigroup (Containers Global Engineer Lead), NYSE Technologies and Architecture & Engineering.
• Background in OpenShift, GRID, HPC, High Frequency Trading, Linux and TCP/IP.
• Original from Rio de Janeiro, Brazil. Lived 12+ years in NY metro area. In Austin, TX since 2012.
• DadOps, Soccer Fan, Homebrewer and (slow) Runner.
• A professional with 16+ years of experience in Cloud and related technologies
• With VMware for 7+ years. Earlier worked with France Telecom and Hewlett Packard
• Author of the book “Mastering PowerCLI”
• Blog at https://sajaldebnath.com
• Pending patents on Hybrid Cloud Storage
• Reach me at @sajal_debnath
Rafael Brito Sajal Debnath
Introduction
VMworld 2019 Content: Not for publication or distribution
5©2019 VMware, Inc.
Overview of the Solution
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 6
The Problem – Complexity, Inflexibility
• Too many moving parts…
• Too rigid and fixed…
• How can I be Agile
providing Agility to the
deployment and time to
market?
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 7
One of the goals of OpenShift running on VMC is bridging engineering teams to a common ground: IaaS + PaaS
Bringing Down the Silos
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 8
OpenShift Deployment Layers
Infra Layer / VMware Automation – PART - 2
OpenShift Layer / App Layer – PART -1
CAS Blueprint Extensibility Workflows
Master Nodes Infra Nodes App Nodes HA-Proxy NodesAdmin Node
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 9
Deployment
Public Clouds
Deployment Flow
Users
VMware Cloud on AWS
Catalog Items
Private Cloud/Infra
Cloud Assembly
Policy
Tags
Compliance
Authenticate
VMware Cloud Services
OpenShift ClusterK8s Cluster managed by OpenShiftOpenShift UI
PART – 2 – CAS and Automation details
PART – 1 – OpenShift Details
Op
en
Sh
ift Insta
ller +
An
sible
VMworld 2019 Content: Not for publication or distribution
10©2019 VMware, Inc.
Part - 1 OpenShift Layer
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 11
• Red Hat OpenShift Container Platform (“OCP”) is a Platform as a Service (PaaS) based on Kubernetes. Many corporations adopted OpenShift as their enterprise version of k8s.
• Key components of OCP:
• CI/CD (Jenkins, “Source to Image”), Service Catalog, Elastic Search, Prometheus, Docker Registry, Istio, etc.
• Installed and Managed via Ansible
• Most used Release: OCP 3.11 (based on Kubernetes 1.11). Newest Release: OCP 4.1 (based on Kubernetes 1.13). Released in June/2019.
• The scope and code of this session is for OCP 3.11 (applicable to OCP 4.1 only when noted).
• OCP 3.11 is *very* different from OCP 4.1 (based on RHCoreOS, Kubernetes Operator, depends on direct connectivity to Internet). There is no in-place upgrade from OCP 3.X.
Overview of Red Hat OpenShift (OCP)
What is OpenShift ?
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 12
OpenShift Components
Master Node Infra Node
• API
• Scheduler & Controller
• ETCD
• Router POD
App Node
• Your Actual App
Admin Node
• Ansible
Load Balancers
• HA-Proxy
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 13
Interaction Between the OCP Components
ROUTING LAYER
SERVICE LAYER
A
P
I
T
R
A
F
F
I
C
Secured External API Access (optional)
PassthroughLoad Balancer
Container ApplicationTrafficLoad Balancer
Load Balancerwith trusted CA signed cert and
XFF
Operations,App Developers,CI/CD
Internet
Master Nodes (3)
Infra Nodes (3)
App Nodes (N)
Operations,OCP
Cluster Admins
Other Application
Users
SSH / Ansible
Virtual Infrastructure Layer
Operations,ESXi Admins
external-api.example.com
internal-api.example.com
*.apps.example.com
Admin Node (1)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 14
VM sizes to support up to 5,000-10,000 PODs
OpenShift: Recommended Node Sizes for Master/Infra/Admin/LB
MASTERS
3 VMs
8 vCPUs
32GB RAM
4 Disks
50 GB (root filesystem)
50 GB /var
40 GB /var/lib/etcd
50 GB (docker storage)
INFRA
3 VMs
4 vCPUs
16GB RAM
3 Disks
50 GB (root filesystem)
50 GB /var
50 GB (docker storage)
ADMIN
2 vCPUs
8GB RAM
1 Disk
100 GB (root filesystem)
1 VM
HAPROXY
3 VMs
4 vCPUs
16GB RAM
3 Disks
100 GB (root filesystem)
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 15
App node size (horizontal and vertical) depends on multiple variables (application footprint, cluster size, etc.). For this session, each App node VM has 4 vCPUs and 64GB RAM.
OpenShift: App Node Sizes
APP Nodes
Minimum 3 VMs. Maximum 1,000 VMs.
Minimum 4 vCPUs. Maximum Recommended: Max vCPUs under NUMA Node minus ESXi overhead.
Minimum 64GB RAM. Maximum Recommended: Max Memory under NUMA Node minus ESXi overhead.
3 Disks
50 GB (root filesystem)
100 GB /var
150 GB (docker storage)
HORIZONTAL SIZE:NUMBER OF VMs
VERTICAL SIZE:SPECS OF VMs
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 16
Ansible Inventory Host File is horsepower behind any OCP installation. We generate this file with three inputs:
• OCP parameters user input (generated from Cloud Assembly)
• Fixed VM name convention for master/infra/app/lb/admin nodes
• Jinja2 Template File
On the admin node, an Ansible playbook will generate the ansible host file. Before OCP installation, all OCP nodes (master/infra/app/load balancers) must have SSH root equivalency with the admin node.
Ansible Inventory File
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 17
“ocp-parameters.yaml” is a config file generated from user input.
The most critical parameters are “cluster_name” (it will generate DNS entries) “subdomain” and Red Hat subscription credentials (to pull containers and packages for the OCP installation) :
Ansible Inventory File: OCP Cluster Parameters
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 18
This OCP cluster example has 13 VMs. $cluster is the input of the name of the OCP cluster:
• 3 x Master Nodes named $cluster-master-0[1..3]
• 3 x Infra Nodes named $cluster-infra-0[1..3]
• 1 x Admin Node named $cluster-admin-0[1..3]
• 3 x App Nodes named $cluster-app-0[1..3]
• 3 x Load Balancer Nodes named $cluster-haproxy-0[1..3]
– $cluster-haproxy-01 is for passthrough API traffic between nodes
– $cluster-haproxy-02 is for External API traffic
– $cluster-haproxy-03 is for Application traffic (Wild Card DNS)
Ansible Inventory File: VM Name Convention
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 19
Ansible Inventory File: Jinja2 Template
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 20
At Admin Node Boot up:
• Generates root’s SSH key pair
• Runs a non-privileged python web-server serving the public key
• Start “Ansible pinging” all other nodes in a loop
At any other OCP Node Boot up:
• Wget the SSH public key and setup SSH root’s equivalency
“SSH Equiv” Helper
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 21
OCP Install and Post-Install
# =~ 10 minutesansible-playbook -i /root/CLUSTER/ansible-hosts /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml && \
# =~ 45 minutesansible-playbook -i /root/CLUSTER/ansible-hosts /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml && \
# post-install: setup load balancer, cluster-admin and other steps
ansible-playbook -i /root/CLUSTER/ansible-hosts /root/$cluster/config.yml
VMworld 2019 Content: Not for publication or distribution
VMworld 2019 Content: Not for publication or distribution
23©2019 VMware, Inc.
Part - 2 Overview of the Solution - Automation
VMworld 2019 Content: Not for publication or distribution
24©2019 VMware, Inc.
Decisions and Assumptions
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 25
Programmable and Unified provisioning across clouds
VMware Automation: Why?
Catalog & Policy Definitions Continuous Delivery
Policy-Driven Provisioning
Cloud API
Cloud Admin,SRE
Blueprints: Multi-Cloud Templates
Business Users
On Premises: vRealize Automation SaaS: Cloud Automation Services
Developer, DevOps Admin
Application Service
Container Service
Function Service
Third-Party Tools
VCPP / VMC PUBLIC CLOUDSDDC (VCF) EDGE
Hybrid infraVMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 26
AWS Global Infrastructure
VMware Cloud™ on AWSPowered by VMware Cloud Foundation
AWS Global InfrastructureCustomer Data Center
vSphere vSAN NSX
Operational Management
Native AWS Services
Amazon EC2
AmazonS3
AmazonRDS
AWS Direct
Connect
AWS IAMAWS IoT
…
…
…
…
vRealize Suite, vSphere Integrated Containers, ISV Ecosystem
vCentervCenter
Hassle free access to SDDC environment – Anytime Anywhere
VMware Cloud on AWS: Why?
• ESXi on Dedicated Hardware
• Support for VMs and Containers
• vSAN on Flash and EBS Storage
• Replication and DR Orchestration
• NSX Spanning on-premises and Cloud
• Advanced Networking & Security Services
AWS Global Infrastructure
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 27
DNS is very important for the deployment and overall working of the solution.
Two options:
• User has authoritative access to DNS server and hence can use our workflows. It does the following:
• At the deployment time, the VM’s gets IP from DHCP. Post IP assignment, VM’s creates the DNS entries for the respective nodes
• CNAME for the load balancer nodes are also created
• While destroying the VM’s, all the DNS entries are removed
• If the user does not have access to DNS, then they need to pre-create all the DNS entries. In this case current blueprint needs to be updated to get the static IP for all the nodes.
DNS Registration
VMworld 2019 Content: Not for publication or distribution
28©2019 VMware, Inc.
Blueprint & Other Configurations
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 29
Blueprint with/without CloudInit
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 30
Subscriptions
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 31
Set VM Name
vRO Workflow – 1
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 32
OpenShift-on-VMware-CAS-Alpha-1-v1.0
vRO Workflow - 2
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 33
Un Register CAS VM from DNS-DHCP-RHN
vRO Workflow - 3
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 34
Register Un-register to DNS
Major Scripts
registerDns.ps1
registerCname.ps1
unregisterDns.ps1unregisterCname.ps1
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 35
Generic Host Preparation
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 36
Admin Node Setup Script
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 37
SSH Setup in Admin and Other Nodes
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 38
OpenShift Installation
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 39
Configure Docker Storage
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 40
Deployment View
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 41
OpenShift View
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 42
Grafana
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 43
Jenkins
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 44
https://labs.vmware.com/flings/enterprise-openshift-as-a-service-on-cloud-automation-services
Check out the Fling
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 45
• Support for OpenShift 4.1
• Do the installation through Cloudinit and ABX Actions (platform agnostics)
• Finetune and improve performance of the overall solution
• Put more failure checkpoints – increase stability
• Provide a workflow where end users dynamically can choose number of nodes in cluster
• Provide Day-2 action to add more nodes to the already deployed cluster
• Add an existing cluster as an endpoint in CAS and Code Stream (already in beta)
To-Do
VMworld 2019 Content: Not for publication or distribution
©2019 VMware, Inc. 46
Reach out to octo-openshift@vmware.com
Read https://octo.vmware.com/vmware-octo-application-platforms-position-paper/
Let us know your use cases!!
Download the Package @ https://labs.vmware.com/flings/enterprise-openshift-as-a-service-on-cloud-automation-services
Twitter:
• @rafaelbrito
• @sajal_debnath
Next Steps (Call to Action)
VMworld 2019 Content: Not for publication or distribution
VMworld 2019 Content: Not for publication or distribution
VMworld 2019 Content: Not for publication or distribution
top related