openshift container platform 3 - red hat customer portal · 2019-04-27 · openshift container...

OpenShift Container Platform 3.6

Installation and Configuration

OpenShift Container Platform 3.6 Installation and Configuration

Last Updated: 2019-04-27

Legal Notice

Copyright © 2019 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinitylogo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and othercountries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related toor endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and other countriesand are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed orsponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

OpenShift Installation and Configuration topics cover the basics of installing and configuringOpenShift in your environment. Use these topics for the one-time tasks required to get OpenShift upand running.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. OVERVIEW

CHAPTER 2. INSTALLING A CLUSTER2.1. PLANNING

2.1.1. Initial Planning2.1.2. Installation Methods2.1.3. Sizing Considerations2.1.4. Environment Scenarios

2.1.4.1. Single Master and Node on One System2.1.4.2. Single Master and Multiple Nodes2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes2.1.4.4. Multiple Masters Using Native HA2.1.4.5. Stand-alone Registry

2.1.5. RPM Versus Containerized2.2. PREREQUISITES

2.2.1. System Requirements2.2.1.1. Red Hat Subscriptions2.2.1.2. Minimum Hardware Requirements2.2.1.3. Production Level Hardware Requirements2.2.1.4. Configuring Core Usage2.2.1.5. SELinux

Using OverlayFS2.2.1.6. NTP2.2.1.7. Security Warning

2.2.2. Environment Requirements2.2.2.1. DNS

2.2.2.1.1. Configuring Hosts to Use DNS2.2.2.1.2. Configuring a DNS Wildcard

2.2.2.2. Network Access2.2.2.2.1. NetworkManager2.2.2.2.2. Configuring firewalld as the firewall2.2.2.2.3. Required Ports

2.2.2.3. Persistent Storage2.2.2.4. Cloud Provider Considerations

2.2.2.4.1. Overriding Detected IP Addresses and Host Names2.2.2.4.2. Post-Installation Configuration for Cloud Providers

2.2.2.5. Containerized GlusterFS Considerations2.2.2.5.1. Storage Nodes2.2.2.5.2. Required Software Components

2.3. HOST PREPARATION2.3.1. Setting PATH2.3.2. Operating System Requirements2.3.3. Host Registration2.3.4. Installing Base Packages2.3.5. Installing Docker2.3.6. Configuring Docker Storage

2.3.6.1. Configuring OverlayFS2.3.6.2. Configuring Thin Pool Storage2.3.6.3. Reconfiguring Docker Storage2.3.6.4. Enabling Image Signature Support2.3.6.5. Managing Container Logs

20

21212121212222232323242424242425262727272828282829303131313134343536363636373737373839394040434344

Table of Contents

1

2.3.6.6. Viewing Available Container Logs2.3.6.7. Blocking Local Volume Usage

2.3.7. Ensuring Host Access2.3.8. Setting Proxy Overrides2.3.9. What’s Next?

2.4. INSTALLING ON CONTAINERIZED HOSTS2.4.1. RPM Versus Containerized Installation2.4.2. Install Methods for Containerized Hosts2.4.3. Required Images2.4.4. Starting and Stopping Containers2.4.5. File Paths2.4.6. Storage Requirements2.4.7. Open vSwitch SDN Initialization

2.5. QUICK INSTALLATION2.5.1. Overview2.5.2. Before You Begin2.5.3. Running an Interactive Installation2.5.4. Defining an Installation Configuration File2.5.5. Running an Unattended Installation2.5.6. Verifying the Installation2.5.7. Uninstalling OpenShift Container Platform2.5.8. What’s Next?

2.6. ADVANCED INSTALLATION2.6.1. Overview2.6.2. Before You Begin2.6.3. Configuring Ansible Inventory Files

Image Version Policy2.6.3.1. Configuring Cluster Variables2.6.3.2. Configuring Deployment Type2.6.3.3. Configuring Host Variables2.6.3.4. Configuring Master API and Console Ports2.6.3.5. Configuring Cluster Pre-install Checks2.6.3.6. Configuring System Containers

2.6.3.6.1. Running Docker as a System Container2.6.3.6.2. Running etcd as a System Container

2.6.3.7. Configuring a Registry Location2.6.3.8. Configuring the Registry Console

2.6.3.8.1. Configuring Registry StorageOption A: NFS Host GroupOption B: External NFS HostOption C: OpenStack PlatformOption D: AWS or Another S3 Storage Solution

2.6.3.9. Configuring Router Sharding2.6.3.10. Configuring GlusterFS Persistent Storage

2.6.3.10.1. Configuring Containerized GlusterFS Persistent Storage2.6.3.11. Configuring the OpenShift Container Registry

2.6.3.11.1. Configuring a Containerized GlusterFS-Backed Registry2.6.3.12. Configuring Global Proxy Options2.6.3.13. Configuring the Firewall2.6.3.14. Configuring Schedulability on Masters2.6.3.15. Configuring Node Host Labels

2.6.3.15.1. Configuring Dedicated Infrastructure Nodes2.6.3.16. Configuring Session Options

4545464747484848494950505050505151525454555556565657575763636566676869707071717171727272737475767879798081


2

2.6.3.17. Configuring Custom Certificates2.6.3.18. Configuring Certificate Validity2.6.3.19. Configuring Cluster Metrics

2.6.3.19.1. Configuring Metrics StorageOption A: DynamicOption B: NFS Host GroupOption C: External NFS Host

2.6.3.20. Configuring Cluster Logging2.6.3.20.1. Configuring Logging Storage

Option A: DynamicOption B: NFS Host GroupOption C: External NFS Host

2.6.3.21. Enabling the Service Catalog2.6.3.22. Configuring the Ansible Service Broker2.6.3.23. Configuring the Template Service Broker2.6.3.24. Configuring Web Console Customization

2.6.4. Example Inventory Files2.6.4.1. Single Master Examples

Single Master and Multiple NodesSingle Master, Multiple etcd, and Multiple Nodes

2.6.4.2. Multiple Masters ExamplesMultiple Masters with Multiple etcdMultiple Masters with Master and etcd on the Same Host

2.6.5. Running the Advanced Installation2.6.5.1. Running the RPM-based Installer2.6.5.2. Running the Containerized Installer

2.6.5.2.1. Running the Installer as a System Container2.6.5.2.2. Running Other Playbooks2.6.5.2.3. Running the Installer as a Docker Container

2.6.5.3. Deploying the Template Service Broker2.6.6. Verifying the Installation

Verifying Multiple etcd HostsVerifying Multiple Masters Using HAProxy

2.6.7. Optionally Securing Builds2.6.8. Uninstalling OpenShift Container Platform

2.6.8.1. Uninstalling Nodes2.6.9. Known Issues2.6.10. What’s Next?

2.7. DISCONNECTED INSTALLATION2.7.1. Overview2.7.2. Prerequisites2.7.3. Required Software and Components

2.7.3.1. Syncing Repositories2.7.3.2. Syncing Images2.7.3.3. Preparing Images for Export

2.7.4. Repository Server2.7.4.1. Placing the Software

2.7.5. OpenShift Container Platform Systems2.7.5.1. Building Your Hosts2.7.5.2. Connecting the Repositories2.7.5.3. Host Preparation

2.7.6. Installing OpenShift Container Platform2.7.6.1. Importing OpenShift Container Platform Component Images

8182828383838484848484858586878888888990919295969797989899

100102102103103103103104104104104105105105107110111111112112112112113113

Table of Contents

3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.7.6.2. Running the OpenShift Container Platform Installer2.7.6.3. Creating the Internal Docker Registry

2.7.7. Post-Installation Changes2.7.7.1. Re-tagging S2I Builder Images2.7.7.2. Configuring a Registry Location2.7.7.3. Creating an Administrative User2.7.7.4. Modifying the Security Policies2.7.7.5. Editing the Image Stream Definitions2.7.7.6. Loading the Container Images

2.7.8. Installing a Router2.8. INSTALLING A STAND-ALONE DEPLOYMENT OF OPENSHIFT CONTAINER REGISTRY

2.8.1. About OpenShift Container Registry2.8.2. Minimum Hardware Requirements2.8.3. Supported System Topologies2.8.4. Host Preparation2.8.5. Installation Methods

2.8.5.1. Quick Installation for Stand-alone OpenShift Container Registry2.8.5.2. Advanced Installation for Stand-alone OpenShift Container Registry

CHAPTER 3. SETTING UP THE REGISTRY3.1. REGISTRY OVERVIEW

3.1.1. About the Registry3.1.2. Integrated or Stand-alone Registries

3.2. DEPLOYING A REGISTRY ON EXISTING CLUSTERS3.2.1. Overview3.2.2. Deploying the Registry3.2.3. Deploying the Registry as a DaemonSet3.2.4. Registry Compute Resources3.2.5. Storage for the Registry

3.2.5.1. Production Use3.2.5.1.1. Use Amazon S3 as a Storage Back-end

3.2.5.2. Non-Production Use3.2.6. Enabling the Registry Console

3.2.6.1. Deploying the Registry Console3.2.6.2. Securing the Registry Console3.2.6.3. Troubleshooting the Registry Console

3.2.6.3.1. Debug Mode3.2.6.3.2. Display SSL Certificate Path

3.3. ACCESSING THE REGISTRY3.3.1. Viewing Logs3.3.2. File Storage3.3.3. Accessing the Registry Directly

3.3.3.1. User Prerequisites3.3.3.2. Logging in to the Registry3.3.3.3. Pushing and Pulling Images

3.3.4. Accessing Registry Metrics3.4. SECURING AND EXPOSING THE REGISTRY

3.4.1. Overview3.4.2. Manually Securing the Registry3.4.3. Manually Exposing a Secure Registry3.4.4. Manually Exposing a Non-Secure Registry

3.5. EXTENDED REGISTRY CONFIGURATION3.5.1. Maintaining the Registry IP Address

113113113113114115116116117117118118118119119119119120

124124124124124124124125125125126126127127128128130130130130131131133133133134135136136136139141143143


4

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.5.2. Whitelisting Docker Registries3.5.3. Overriding the Registry Configuration3.5.4. Registry Configuration Reference

3.5.4.1. Log3.5.4.2. Hooks3.5.4.3. Storage3.5.4.4. Auth3.5.4.5. Middleware

3.5.4.5.1. CloudFront Middleware3.5.4.5.2. Overriding Middleware Configuration Options3.5.4.5.3. Image Pullthrough3.5.4.5.4. Manifest Schema v2 Support

3.5.4.6. OpenShift3.5.4.7. Reporting3.5.4.8. HTTP3.5.4.9. Notifications3.5.4.10. Redis3.5.4.11. Health3.5.4.12. Proxy

3.6. KNOWN ISSUES3.6.1. Overview3.6.2. Image Push Errors with Scaled Registry Using Shared NFS Volume3.6.3. Pull of Internally Managed Image Fails with "not found" Error3.6.4. Image Push Fails with "500 Internal Server Error" on S3 Storage3.6.5. Image Pruning Fails

CHAPTER 4. SETTING UP A ROUTER4.1. ROUTER OVERVIEW

4.1.1. About Routers4.1.2. Router Service Account

4.1.2.1. Permission to Access Labels4.2. USING THE DEFAULT HAPROXY ROUTER

4.2.1. Overview4.2.2. Creating a Router4.2.3. Other Basic Router Commands4.2.4. Filtering Routes to Specific Routers4.2.5. HAProxy Strict SNI4.2.6. TLS Cipher Suites4.2.7. Highly-Available Routers4.2.8. Customizing the Router Service Ports4.2.9. Working With Multiple Routers4.2.10. Adding a Node Selector to a Deployment Configuration4.2.11. Using Router Shards

4.2.11.1. Creating Router Shards4.2.11.2. Modifying Router Shards4.2.11.3. Using Namespace Router Shards

4.2.12. Finding the Host Name of the Router4.2.13. Customizing the Default Routing Subdomain4.2.14. Forcing Route Host Names to a Custom Routing Subdomain4.2.15. Using Wildcard Certificates4.2.16. Manually Redeploy Certificates4.2.17. Using Secured Routes4.2.18. Using Wildcard Routes (for a Subdomain)

143144146146147147148148148150150151152152152153153153153153153153154154155

156156156156156157157158158159160160160160161161161164166167168169169169170171172

Table of Contents

5

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.2.19. Using the Container Network Stack4.2.20. Exposing Router Metrics4.2.21. Preventing Connection Failures During Restarts4.2.22. ARP Cache Tuning for Large-scale Clusters4.2.23. Protecting Against DDoS Attacks

4.3. DEPLOYING A CUSTOMIZED HAPROXY ROUTER4.3.1. Overview4.3.2. Obtaining the Router Configuration Template4.3.3. Modifying the Router Configuration Template

4.3.3.1. Background4.3.3.2. Go Template Actions4.3.3.3. Router Provided Information4.3.3.4. Annotations4.3.3.5. Environment Variables4.3.3.6. Example Usage

4.3.4. Using a ConfigMap to Replace the Router Configuration Template4.3.5. Using Stick Tables4.3.6. Rebuilding Your Router

4.4. CONFIGURING THE HAPROXY ROUTER TO USE THE PROXY PROTOCOL4.4.1. Overview4.4.2. Why Use the PROXY Protocol?4.4.3. Using the PROXY Protocol

4.5. USING THE F5 ROUTER PLUG-IN4.5.1. Overview4.5.2. Prerequisites and Supportability

4.5.2.1. Configuring the Virtual Servers4.5.3. Deploying the F5 Router4.5.4. F5 Router Partition Paths4.5.5. Setting Up F5 Native Integration

CHAPTER 5. UPGRADING A CLUSTER5.1. OVERVIEW

5.1.1. In-place or Blue-Green UpgradesIn-place UpgradesBlue-green Deployments

5.2. PERFORMING AUTOMATED IN-PLACE CLUSTER UPGRADES5.2.1. Overview5.2.2. Preparing for an Automated Upgrade5.2.3. Using the Installer to Upgrade5.2.4. Running Upgrade Playbooks Directly

5.2.4.1. Upgrading the Control Plane and Nodes in Separate Phases5.2.4.2. Customizing Node Upgrades5.2.4.3. Customizing Upgrades With Ansible Hooks

5.2.4.3.1. Limitations5.2.4.3.2. Using Hooks5.2.4.3.3. Available Upgrade Hooks

5.2.4.4. Upgrading to the Latest OpenShift Container Platform 3.6 Release5.2.5. Upgrading the EFK Logging Stack5.2.6. Upgrading Cluster Metrics5.2.7. Special Considerations for Large-scale Upgrades5.2.8. Special Considerations for Mixed Environments5.2.9. Verifying the Upgrade

5.3. PERFORMING MANUAL IN-PLACE CLUSTER UPGRADES

178178180181182183183183184184184185189190190192193195195195196196200200200202202203204

207207207207207207207208210211211212212212213213214215216217217217218


6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.3.1. Overview5.3.2. Preparing for a Manual Upgrade5.3.3. Upgrading Master Components5.3.4. Updating Policy Definitions5.3.5. Upgrading Nodes5.3.6. Upgrading the Router5.3.7. Upgrading the Registry5.3.8. Updating the Default Image Streams and Templates5.3.9. Importing the Latest Images5.3.10. Upgrading the EFK Logging Stack5.3.11. Upgrading Cluster Metrics5.3.12. Additional Manual Steps Per Release5.3.13. Verifying the Upgrade

5.4. BLUE-GREEN DEPLOYMENTS5.4.1. Overview5.4.2. Preparing for a Blue-Green Upgrade

5.4.2.1. Sharing Software Entitlements5.4.2.2. Labeling Blue Nodes5.4.2.3. Creating and Labeling Green Nodes5.4.2.4. Verifying Green Nodes

5.4.3. Registry and Router Canary Deployments5.4.4. Warming the Green Nodes5.4.5. Evacuating and Decommissioning Blue Nodes

5.5. OPERATING SYSTEM UPDATES AND UPGRADES5.5.1. Updating and Upgrading the Operating System

5.6. MIGRATING ETCD DATA: V2 TO V35.6.1. Overview5.6.2. Before You Begin5.6.3. Running the Automated Migration Playbook5.6.4. Running the Migration Manually5.6.5. Recovering from Migration Issues

5.7. KNOWN ISSUES5.7.1. Overview5.7.2. Orphaned RoleBindingRestriction Objects5.7.3. Orphaned OAuthClientAuthorization Objects

CHAPTER 6. DOWNGRADING OPENSHIFT6.1. OVERVIEW6.2. VERIFYING BACKUPS6.3. SHUTTING DOWN THE CLUSTER6.4. REMOVING RPMS6.5. DOWNGRADING DOCKER6.6. REINSTALLING RPMS6.7. RESTORING ETCD6.8. BRINGING OPENSHIFT CONTAINER PLATFORM SERVICES BACK ONLINE6.9. VERIFYING THE DOWNGRADE

CHAPTER 7. MASTER AND NODE CONFIGURATION7.1. OVERVIEW7.2. MASTER CONFIGURATION FILES

7.2.1. Admission Control Configuration7.2.2. Asset Configuration7.2.3. Authentication and Authorization Configuration

218218220224226230230231233234235236236237237238238239239240241241242243243244244245245245248249249249251

254254254254255255255256256256

258258258258259260

Table of Contents

7

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.2.4. Controller Configuration7.2.5. etcd Configuration7.2.6. Grant Configuration7.2.7. Image Configuration7.2.8. Kubernetes Master Configuration7.2.9. Network Configuration7.2.10. OAuth Authentication Configuration7.2.11. Project Configuration7.2.12. Scheduler Configuration7.2.13. Security Allocator Configuration7.2.14. Service Account Configuration7.2.15. Serving Information Configuration7.2.16. Volume Configuration7.2.17. Audit Configuration

7.3. NODE CONFIGURATION FILES7.3.1. Pod and Node Configuration7.3.2. Docker Configuration7.3.3. Parallel Image Pulls with Docker 1.9+

7.4. PASSWORDS AND OTHER SENSITIVE DATA7.5. CREATING NEW CONFIGURATION FILES7.6. LAUNCHING SERVERS USING CONFIGURATION FILES7.7. CONFIGURING LOGGING LEVELS7.8. RESTARTING OPENSHIFT CONTAINER PLATFORM SERVICES

CHAPTER 8. ADDING HOSTS TO AN EXISTING CLUSTER8.1. OVERVIEW8.2. ADDING HOSTS USING THE QUICK INSTALLER TOOL8.3. ADDING HOSTS USING THE ADVANCED INSTALL

CHAPTER 9. LOADING THE DEFAULT IMAGE STREAMS AND TEMPLATES9.1. OVERVIEW9.2. OFFERINGS BY SUBSCRIPTION TYPE

9.2.1. OpenShift Container Platform Subscription9.2.2. xPaaS Middleware Add-on Subscriptions

9.3. BEFORE YOU BEGIN9.4. PREREQUISITES9.5. CREATING IMAGE STREAMS FOR OPENSHIFT CONTAINER PLATFORM IMAGES9.6. CREATING IMAGE STREAMS FOR XPAAS MIDDLEWARE IMAGES9.7. CREATING DATABASE SERVICE TEMPLATES9.8. CREATING INSTANT APP AND QUICKSTART TEMPLATES9.9. WHAT’S NEXT?

CHAPTER 10. CONFIGURING CUSTOM CERTIFICATES10.1. OVERVIEW10.2. CONFIGURING CUSTOM CERTIFICATES WITH ANSIBLE10.3. CONFIGURING CUSTOM CERTIFICATES10.4. CONFIGURING A CUSTOM WILDCARD CERTIFICATE FOR THE DEFAULT ROUTER10.5. CONFIGURING A CUSTOM CERTIFICATE FOR A LOAD BALANCER

CHAPTER 11. REDEPLOYING CERTIFICATES11.1. OVERVIEW11.2. CHECKING CERTIFICATE EXPIRATIONS

11.2.1. Role Variables11.2.2. Running Certificate Expiration Playbooks

260261261262263263264265265266266267268268270271271272272273274274279

280280280281

284284284284285285285286286286287288

289289289289290291

293293293293294


8

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Other Example Playbooks11.2.3. Output Formats

HTML ReportJSON Report

11.3. REDEPLOYING CERTIFICATES11.3.1. Redeploying All Certificates Using the Current OpenShift Container Platform and etcd CA11.3.2. Redeploying a New or Custom OpenShift Container Platform CA11.3.3. Redeploying a New etcd CA11.3.4. Redeploying Master Certificates Only11.3.5. Redeploying etcd Certificates Only11.3.6. Redeploying Node Certificates Only11.3.7. Redeploying Registry or Router Certificates Only

11.3.7.1. Redeploying Registry Certificates Only11.3.7.2. Redeploying Router Certificates Only

11.3.8. Redeploying Custom Registry or Router Certificates11.3.8.1. Redeploying Registry Certificates Manually11.3.8.2. Redeploying Router Certificates Manually

CHAPTER 12. CONFIGURING AUTHENTICATION AND USER AGENT12.1. OVERVIEW12.2. IDENTITY PROVIDER PARAMETERS12.3. CONFIGURING IDENTITY PROVIDERS

12.3.1. Configuring identity providers with Ansible12.3.2. Configuring identity providers in the master configuration file12.3.3. Configuring an identity provider or method

12.3.3.1. Manually provisioning a user when using the lookup mapping method12.3.4. Allow all12.3.5. Deny all12.3.6. HTPasswd12.3.7. Keystone12.3.8. LDAP authentication12.3.9. Basic authentication (remote)12.3.10. Request header12.3.11. GitHub12.3.12. GitLab12.3.13. Google12.3.14. OpenID connect

12.4. TOKEN OPTIONS12.5. GRANT OPTIONS12.6. SESSION OPTIONS12.7. PREVENTING CLI VERSION MISMATCH WITH USER AGENT

CHAPTER 13. SYNCING GROUPS WITH LDAP13.1. OVERVIEW13.2. CONFIGURING LDAP SYNC

13.2.1. LDAP Client Configuration13.2.2. LDAP Query Definition13.2.3. User-Defined Name Mapping

13.3. RUNNING LDAP SYNC13.4. RUNNING A GROUP PRUNING JOB13.5. SYNC EXAMPLES

13.5.1. RFC 230713.5.1.1. RFC2307 with User-Defined Name Mappings

295295295296296297297298299299299300300300300300302

305305305306307307308308309309310311312315317324325326328330331331333

335335335335336337337338338339341

Table of Contents

9

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13.5.2. RFC 2307 with User-Defined Error Tolerances13.5.3. Active Directory13.5.4. Augmented Active Directory

13.6. NESTED MEMBERSHIP SYNC EXAMPLE13.7. LDAP SYNC CONFIGURATION SPECIFICATION

13.7.1. v1.LDAPSyncConfig13.7.2. v1.StringSource13.7.3. v1.LDAPQuery13.7.4. v1.RFC2307Config13.7.5. v1.ActiveDirectoryConfig13.7.6. v1.AugmentedActiveDirectoryConfig

CHAPTER 14. CONFIGURING LDAP FAILOVER14.1. PREREQUISITES FOR CONFIGURING BASIC REMOTE AUTHENTICATION14.2. GENERATING AND SHARING CERTIFICATES WITH THE REMOTE BASIC AUTHENTICATION SERVER

14.3. CONFIGURING SSSD FOR LDAP FAILOVER14.4. CONFIGURING APACHE TO USE SSSD14.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM TO USE SSSD AS THE BASIC REMOTEAUTHENTICATION SERVER

CHAPTER 15. CONFIGURING THE SDN15.1. OVERVIEW15.2. AVAILABLE SDN PROVIDERS

Installing VMware NSX-T (™) on OpenShift Container Platform15.3. CONFIGURING THE POD NETWORK WITH ANSIBLE15.4. CONFIGURING THE POD NETWORK ON MASTERS15.5. CONFIGURING THE POD NETWORK ON NODES15.6. MIGRATING BETWEEN SDN PLUG-INS

15.6.1. Migrating from ovs-multitenant to ovs-networkpolicy15.7. EXTERNAL ACCESS TO THE CLUSTER NETWORK15.8. USING FLANNEL

CHAPTER 16. CONFIGURING NUAGE SDN16.1. NUAGE SDN AND OPENSHIFT CONTAINER PLATFORM16.2. DEVELOPER WORKFLOW16.3. OPERATIONS WORKFLOW16.4. INSTALLATION

16.4.1. Installation for a Single Master16.4.2. Installation for Multiple Masters (HA)

CHAPTER 17. CONFIGURING FOR AWS17.1. OVERVIEW17.2. PERMISSIONS17.3. CONFIGURING A SECURITY GROUP

17.3.1. Overriding Detected IP Addresses and Host Names17.4. CONFIGURING AWS VARIABLES17.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR AWS

17.5.1. Configuring OpenShift Container Platform for AWS with Ansible17.5.2. Manually Configuring OpenShift Container Platform Masters for AWS17.5.3. Manually Configuring OpenShift Container Platform Nodes for AWS

17.6. SETTING KEY VALUE ACCESS PAIRS17.7. APPLYING CONFIGURATION CHANGES

343345347350353353355356357358359

361361

361362364

367

369369369369369370372372372373373

377377377377377377378

380380380381381382383383384384385385


10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTER 18. CONFIGURING FOR OPENSTACK18.1. OVERVIEW18.2. PERMISSIONS18.3. CONFIGURING A SECURITY GROUP18.4. CONFIGURING OPENSTACK VARIABLES18.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR OPENSTACK

18.5.1. Configuring OpenShift Container Platform for OpenStack with Ansible18.5.2. Manually Configuring OpenShift Container Platform Masters for OpenStack18.5.3. Manually Configuring OpenShift Container Platform Nodes for OpenStack

18.6. APPLYING CONFIGURATION CHANGES

CHAPTER 19. CONFIGURING FOR GCE19.1. OVERVIEW19.2. PERMISSIONS19.3. CONFIGURING MASTERS

19.3.1. Configuring OpenShift Container Platform Masters for GCE with Ansible19.3.2. Manually Configuring OpenShift Container Platform Masters for GCE

19.4. CONFIGURING NODES19.5. CONFIGURING MULTIZONE SUPPORT IN A GCE DEPLOYMENT19.6. APPLYING CONFIGURATION CHANGES

CHAPTER 20. CONFIGURING FOR AZURE20.1. OVERVIEW20.2. PERMISSIONS20.3. THE AZURE CONFIGURATION FILE20.4. CONFIGURING MASTERS20.5. CONFIGURING NODES20.6. APPLYING CONFIGURATION CHANGES

CHAPTER 21. CONFIGURING FOR VMWARE VSPHERE21.1. OVERVIEW21.2. ENABLING VMWARE VSPHERE CLOUD PROVIDER21.3. THE VMWARE VSPHERE CONFIGURATION FILE21.4. CONFIGURING MASTERS21.5. CONFIGURING NODES21.6. APPLYING CONFIGURATION CHANGES21.7. BACKUP OF PERSISTENT VOLUMES

CHAPTER 22. CONFIGURING PERSISTENT STORAGE22.1. OVERVIEW22.2. PERSISTENT STORAGE USING NFS

22.2.1. Overview22.2.2. Provisioning22.2.3. Enforcing Disk Quotas22.2.4. NFS Volume Security

22.2.4.1. Group IDs22.2.4.2. User IDs22.2.4.3. SELinux22.2.4.4. Export Settings

22.2.5. Reclaiming Resources22.2.6. Automation22.2.7. Additional Configuration and Troubleshooting

22.3. PERSISTENT STORAGE USING GLUSTERFS22.3.1. Overview

386386386386387387387388389389

391391391391391391392393393

395395395395395396396

398398398400401401402402

404404404404405406406407408409409410411411411412

Table of Contents

11

22.3.1.1. Containerized Red Hat Gluster Storage22.3.1.2. Container Native Storage Recommendations

22.3.1.2.1. Creation Time of Volumes with Container Native Storage22.3.1.2.2. Deletion Time of Volumes with Container Native Storage22.3.1.2.3. Recommended Memory Requirements for Container Native Storage

22.3.1.3. Dedicated Storage Cluster22.3.2. Support Requirements

22.3.2.1. Supported Operating Systems22.3.2.2. Environment Requirements

22.3.3. Provisioning22.3.3.1. Creating Gluster Endpoints22.3.3.2. Creating the Persistent Volume22.3.3.3. Creating the Persistent Volume Claim

22.3.4. Gluster Volume Security22.3.4.1. Group IDs22.3.4.2. User IDs22.3.4.3. SELinux

22.4. PERSISTENT STORAGE USING OPENSTACK CINDER22.4.1. Overview22.4.2. Provisioning

22.4.2.1. Creating the Persistent Volume22.4.2.2. Volume Format

22.5. PERSISTENT STORAGE USING CEPH RADOS BLOCK DEVICE (RBD)22.5.1. Overview22.5.2. Provisioning

22.5.2.1. Creating the Ceph Secret22.5.2.2. Creating the Persistent Volume

22.5.3. Ceph Volume Security22.6. PERSISTENT STORAGE USING AWS ELASTIC BLOCK STORE

22.6.1. Overview22.6.2. Provisioning

22.6.2.1. Creating the Persistent Volume22.6.2.2. Volume Format22.6.2.3. Maximum Number of EBS Volumes on a Node

22.7. PERSISTENT STORAGE USING GCE PERSISTENT DISK22.7.1. Overview22.7.2. Provisioning


22.8. PERSISTENT STORAGE USING ISCSI22.8.1. Overview22.8.2. Provisioning

22.8.2.1. Enforcing Disk Quotas22.8.2.2. iSCSI Volume Security22.8.2.3. iSCSI Multipathing

22.9. PERSISTENT STORAGE USING FIBRE CHANNEL22.9.1. Overview22.9.2. Provisioning

22.9.2.1. Enforcing Disk Quotas22.9.2.2. Fibre Channel Volume Security

22.10. PERSISTENT STORAGE USING AZURE DISK22.10.1. Overview22.10.2. Prerequisites

412412413413413413414414415415416417418419419420421421421422422423423423424424425426427427428428429429429429430430431431431432432432433433433433434434434434435


12

22.10.3. Provisioning22.10.4. Configuring Azure Disk for regional cloud


22.11. PERSISTENT STORAGE USING AZURE FILE22.11.1. Overview22.11.2. Before you begin22.11.3. Configuring Azure file for regional cloud22.11.4. Creating the Persistent Volume22.11.5. Creating the Azure Storage Account Secret

22.12. PERSISTENT STORAGE USING FLEXVOLUME PLUG-INS22.12.1. Overview22.12.2. Installing FlexVolume Drivers22.12.3. Consuming Storage using FlexVolume22.12.4. FlexVolume Drivers

22.12.4.1. FlexVolume Drivers with Master-initiated Attach/Detach22.12.4.2. FlexVolume Drivers Without Master-initiated Attach/Detach

22.13. USING VMWARE VSPHERE VOLUMES FOR PERSISTENT STORAGE22.13.1. Overview

Prerequisites22.13.2. Provisioning VMware vSphere volumes

22.13.2.1. Creating persistent volumes22.13.2.2. Formatting VMware vSphere volumes

22.14. DYNAMIC PROVISIONING AND CREATING STORAGE CLASSES22.14.1. Overview22.14.2. Available Dynamically Provisioned Plug-ins22.14.3. Defining a StorageClass

22.14.3.1. Basic StorageClass Object Definition22.14.3.2. StorageClass Annotations22.14.3.3. OpenStack Cinder Object Definition22.14.3.4. AWS ElasticBlockStore (EBS) Object Definition22.14.3.5. GCE PersistentDisk (gcePD) Object Definition22.14.3.6. GlusterFS Object Definition22.14.3.7. Ceph RBD Object Definition22.14.3.8. Azure Disk Object Definition22.14.3.9. Trident Object Definition22.14.3.10. VMware vSphere Object Definition

22.14.4. Changing the Default StorageClass22.14.5. Additional Information and Examples

22.15. VOLUME SECURITY22.15.1. Overview22.15.2. SCCs, Defaults, and Allowed Ranges22.15.3. Supplemental Groups22.15.4. fsGroup22.15.5. User IDs22.15.6. SELinux Options

22.16. SELECTOR-LABEL VOLUME BINDING22.16.1. Overview22.16.2. Motivation22.16.3. Deployment

22.16.3.1. Prerequisites22.16.3.2. Define the Persistent Volume and Claim22.16.3.3. Deploy the Persistent Volume and Claim

435435435437437437437437438438439439440440441442444445445445446446447447447448449449450450451451452453454454454455455456456456460463465467468468468469469469470

Table of Contents

13

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22.17. ENABLING CONTROLLER-MANAGED ATTACHMENT AND DETACHMENT22.17.1. Overview22.17.2. Determining What Is Managing Attachment and Detachment22.17.3. Configuring Nodes to Enable Controller-managed Attachment and Detachment

CHAPTER 23. PERSISTENT STORAGE EXAMPLES23.1. OVERVIEW23.2. SHARING AN NFS MOUNT ACROSS TWO PERSISTENT VOLUME CLAIMS

23.2.1. Overview23.2.2. Creating the Persistent Volume23.2.3. Creating the Persistent Volume Claim23.2.4. Ensuring NFS Volume Access23.2.5. Creating the Pod23.2.6. Creating an Additional Pod to Reference the Same PVC

23.3. COMPLETE EXAMPLE USING CEPH RBD23.3.1. Overview23.3.2. Installing the ceph-common Package23.3.3. Creating the Ceph Secret23.3.4. Creating the Persistent Volume23.3.5. Creating the Persistent Volume Claim23.3.6. Creating the Pod23.3.7. Defining Group and Owner IDs (Optional)23.3.8. Setting ceph-user-secret as Default for Projects

23.4. USING CEPH RBD FOR DYNAMIC PROVISIONING23.4.1. Overview23.4.2. Creating a pool for dynamic volumes23.4.3. Using an existing Ceph cluster for dynamic persistent storage23.4.4. Setting ceph-user-secret as the default for projects

23.5. COMPLETE EXAMPLE USING GLUSTERFS23.5.1. Overview23.5.2. Installing the glusterfs-fuse Package23.5.3. Creating the Gluster Endpoints and Gluster Service for Persistence23.5.4. Creating the Persistent Volume23.5.5. Creating the Persistent Volume Claim23.5.6. Defining GlusterFS Volume Access23.5.7. Creating the Pod using NGINX Web Server image

23.6. COMPLETE EXAMPLE OF DYNAMIC PROVISIONING USING CONTAINERIZED GLUSTERFS23.6.1. Overview23.6.2. Verify the Environment and Gather Needed Information23.6.3. Create a Storage Class for Your GlusterFS Dynamic Provisioner23.6.4. Create a PVC to Request Storage for Your Application23.6.5. Create a NGINX Pod That Uses the PVC

23.7. COMPLETE EXAMPLE OF DYNAMIC PROVISIONING USING DEDICATED GLUSTERFS23.7.1. Overview23.7.2. Environment and Prerequisites23.7.3. Installing and Configuring Heketi23.7.4. Loading Topology23.7.5. Dynamically Provision a Volume23.7.6. Creating a NGINX Pod That Uses the PVC

23.8. EXAMPLE: CONTAINERIZED HEKETI FOR MANAGING DEDICATED GLUSTERFS STORAGE23.8.1. Overview23.8.2. Environment and Prerequisites23.8.3. Installing and Configuring Heketi

470470471471

472472472472472473474475479481481481481482483484485485486486486487490491491491491493494495495500500500501502503505505505506507509510511512512512


14

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23.8.4. Loading Topology23.8.5. Dynamically Provision a Volume23.8.6. Creating a NGINX Pod That Uses the PVC

23.9. MOUNTING VOLUMES ON PRIVILEGED PODS23.9.1. Overview23.9.2. Prerequisites23.9.3. Creating the Persistent Volume23.9.4. Creating a Regular User23.9.5. Creating the Persistent Volume Claim23.9.6. Verifying the Setup

23.9.6.1. Checking the Pod SCC23.9.6.2. Verifying the Mount

23.10. BACKING DOCKER REGISTRY WITH GLUSTERFS STORAGE23.10.1. Overview23.10.2. Prerequisites23.10.3. Create the Gluster Persistent Volume23.10.4. Attach the PVC to the Docker Registry23.10.5. Known Issues

23.10.5.1. Pod Cannot Resolve the Volume Host23.11. BINDING PERSISTENT VOLUMES BY LABELS

23.11.1. Overview23.11.1.1. Assumptions

23.11.2. Defining Specifications23.11.2.1. Persistent Volume with Labels23.11.2.2. Persistent Volume Claim with Selectors23.11.2.3. Volume Endpoints23.11.2.4. Deploy the PV, PVC, and Endpoints

23.12. USING STORAGE CLASSES FOR DYNAMIC PROVISIONING23.12.1. Overview23.12.2. Scenario 1: Basic Dynamic Provisioning with Two Types of StorageClasses23.12.3. Scenario 2: How to enable Default StorageClass behavior for a Cluster

23.13. USING STORAGE CLASSES FOR EXISTING LEGACY STORAGE23.13.1. Overview

23.13.1.1. Scenario 1: Link StorageClass to existing Persistent Volume with Legacy Data23.14. CONFIGURING AZURE BLOB STORAGE FOR INTEGRATED DOCKER REGISTRY

23.14.1. Overview23.14.2. Before You Begin23.14.3. Overriding Registry Configuration

CHAPTER 24. WORKING WITH HTTP PROXIES24.1. OVERVIEW24.2. CONFIGURING NO_PROXY24.3. CONFIGURING HOSTS FOR PROXIES24.4. CONFIGURING HOSTS FOR PROXIES USING ANSIBLE24.5. PROXYING DOCKER PULL24.6. USING MAVEN BEHIND A PROXY24.7. CONFIGURING S2I BUILDS FOR PROXIES24.8. CONFIGURING DEFAULT TEMPLATES FOR PROXIES24.9. SETTING PROXY ENVIRONMENT VARIABLES IN PODS24.10. GIT REPOSITORY ACCESS

CHAPTER 25. CONFIGURING GLOBAL BUILD DEFAULTS AND OVERRIDES25.1. OVERVIEW

513515517518518518518519519520520520521521521521522522522523523524524524525525526526526526529533533533536536536536

538538538539540540541541541542542

544544

Table of Contents

15

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25.2. SETTING GLOBAL BUILD DEFAULTS25.2.1. Configuring Global Build Defaults with Ansible25.2.2. Manually Setting Global Build Defaults

25.3. SETTING GLOBAL BUILD OVERRIDES25.3.1. Configuring Global Build Overrides with Ansible25.3.2. Manually Setting Global Build Overrides

CHAPTER 26. CONFIGURING PIPELINE EXECUTION26.1. OVERVIEW

CHAPTER 27. CONFIGURING ROUTE TIMEOUTS

CHAPTER 28. CONFIGURING NATIVE CONTAINER ROUTING28.1. NETWORK OVERVIEW28.2. CONFIGURE NATIVE CONTAINER ROUTING28.3. SETTING UP A NODE FOR CONTAINER NETWORKING28.4. SETTING UP A ROUTER FOR CONTAINER NETWORKING

CHAPTER 29. ROUTING FROM EDGE LOAD BALANCERS29.1. OVERVIEW29.2. INCLUDING THE LOAD BALANCER IN THE SDN29.3. ESTABLISHING A TUNNEL USING A RAMP NODE

29.3.1. Configuring a Highly-Available Ramp Node

CHAPTER 30. AGGREGATING CONTAINER LOGS30.1. OVERVIEW30.2. PRE-DEPLOYMENT CONFIGURATION30.3. SPECIFYING LOGGING ANSIBLE VARIABLES30.4. DEPLOYING THE EFK STACK30.5. UNDERSTANDING AND ADJUSTING THE DEPLOYMENT

30.5.1. Ops Cluster30.5.2. Elasticsearch30.5.3. Fluentd30.5.4. Kibana30.5.5. Curator

30.5.5.1. Creating the Curator Configuration30.6. CLEANUP30.7. TROUBLESHOOTING KIBANA30.8. SENDING LOGS TO AN EXTERNAL ELASTICSEARCH INSTANCE30.9. PERFORMING ADMINISTRATIVE ELASTICSEARCH OPERATIONS30.10. CHANGING THE AGGREGATED LOGGING DRIVER30.11. UPDATING FLUENTD’S LOG SOURCE AFTER A DOCKER LOG DRIVER UPDATE

CHAPTER 31. AGGREGATE LOGGING SIZING GUIDELINES31.1. OVERVIEW31.2. INSTALLATION

31.2.1. Large Clusters31.3. SYSTEMD-JOURNALD AND RSYSLOG31.4. SCALING UP EFK LOGGING31.5. STORAGE CONSIDERATIONS

CHAPTER 32. ENABLING CLUSTER METRICS32.1. OVERVIEW32.2. BEFORE YOU BEGIN32.3. METRICS PROJECT

544545546547547548

550550

552

553553553554554

555555555555558

559559559560568568568569574580581582583583585585586587

589589589591591592593

595595595595


16

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32.4. METRICS DATA STORAGE32.4.1. Persistent Storage32.4.2. Capacity Planning for Cluster Metrics

Recommendations for OpenShift Container Platform Version 3.6Known Issues and Limitations

32.4.3. Non-Persistent Storage32.5. METRICS ANSIBLE ROLE

32.5.1. Specifying Metrics Ansible Variables32.5.2. Using Secrets

32.5.2.1. Providing Your Own Certificates32.6. DEPLOYING THE METRIC COMPONENTS

32.6.1. Metrics Diagnostics32.7. SETTING THE METRICS PUBLIC URL32.8. ACCESSING HAWKULAR METRICS DIRECTLY

32.8.1. OpenShift Container Platform Projects and Hawkular Tenants32.8.2. Authorization

32.9. SCALING OPENSHIFT CONTAINER PLATFORM CLUSTER METRICS PODS32.10. CLEANUP

CHAPTER 33. CUSTOMIZING THE WEB CONSOLE33.1. OVERVIEW33.2. LOADING EXTENSION SCRIPTS AND STYLESHEETS

33.2.1. Setting Extension Properties33.3. EXTENSION OPTION FOR EXTERNAL LOGGING SOLUTIONS33.4. CUSTOMIZING THE LOGO33.5. CHANGING LINKS TO DOCUMENTATION33.6. ADDING OR CHANGING LINKS TO DOWNLOAD THE CLI

33.6.1. Customizing the About Page33.7. CONFIGURING NAVIGATION MENUS

33.7.1. Top Navigation Dropdown Menus33.7.2. Application Launcher33.7.3. Project Left Navigation

33.8. CONFIGURING CATALOG CATEGORIES33.9. CONFIGURING THE CREATE FROM URL NAMESPACE WHITELIST

33.9.1. Enabling Wildcard Routes33.10. ENABLING FEATURES IN TECHNOLOGY PREVIEW33.11. SERVING STATIC FILES

33.11.1. Enabling HTML5 Mode33.12. CUSTOMIZING THE LOGIN PAGE

33.12.1. Example Usage33.13. CUSTOMIZING THE OAUTH ERROR PAGE33.14. CHANGING THE LOGOUT URL33.15. CONFIGURING WEB CONSOLE CUSTOMIZATIONS WITH ANSIBLE

CHAPTER 34. DEPLOYING EXTERNAL PERSISTENT VOLUME PROVISIONERS34.1. OVERVIEW34.2. BEFORE YOU BEGIN

34.2.1. External Provisioners Ansible Role34.2.2. External Provisioners Ansible Variables34.2.3. AWS EFS Provisioner Ansible Variables

34.3. DEPLOYING THE PROVISIONERS34.3.1. Deploying the AWS EFS Provisioner

34.3.1.1. AWS EFS Object Definition

595595596598598599599599603603603604604605605606606606

607607607608608609609610610611611612613615616617618618619619620620620620

623623623623623624625625625

Table of Contents

17

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34.4. CLEANUP

CHAPTER 35. DEPLOYING RED HAT CLOUDFORMS ON OPENSHIFT CONTAINER PLATFORM35.1. OVERVIEW35.2. REQUIREMENTS

35.2.1. Prerequisites35.2.2. Cluster Sizing35.2.3. Other Sizing Considerations35.2.4. Assumptions

35.3. ROLE VARIABLES35.4. PRE-FLIGHT CHECKS35.5. RUNNING THE PLAYBOOK35.6. VERIFYING THE DEPLOYMENT

35.6.1. Describing the CFME Pod35.6.2. Opening a Remote Shell to the CFME Pod

35.7. MANUAL CLEANUP

CHAPTER 36. REVISION HISTORY: INSTALLATION AND CONFIGURATION36.1. MON MAR 12 201836.2. WED MAR 07 201836.3. MON FEB 26 201836.4. WED FEB 21 201836.5. MON FEB 19 201836.6. FRI FEB 16 201836.7. TUE FEB 06 201836.8. THU JAN 25 201836.9. MON JAN 08 201836.10. FRI DEC 22 201736.11. MON DEC 11 201736.12. TUE NOV 21 201736.13. FRI NOV 10 201736.14. FRI NOV 03 201736.15. MON OCT 16 201736.16. WED OCT 11 201736.17. MON OCT 02 201736.18. FRI SEP 22 201736.19. MON SEP 18 201736.20. WED SEP 06 201736.21. TUE AUG 29 201736.22. FRI AUG 25 201736.23. TUE AUG 22 201736.24. MON AUG 14 201736.25. WED AUG 09 2017

626

627627627627627628628628629629629630630634

636636636636636636637638638639640640641641642642642642643643644644645645645645


18

Table of Contents

19

CHAPTER 1. OVERVIEWOpenShift Container Platform Installation and Configuration topics cover the basics of installing andconfiguring OpenShift Container Platform in your environment. Configuration, management, and loggingare also covered. Use these topics for the one-time tasks required to quickly set up your OpenShiftContainer Platform environment and configure it based on your organizational needs.

For day to day cluster administration tasks, see Cluster Administration.


20

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#admin-guide-index

CHAPTER 2. INSTALLING A CLUSTER

2.1. PLANNING

2.1.1. Initial Planning

For production environments, several factors influence installation. Consider the following questions asyou read through the documentation:

Which installation method do you want to use? The Installation Methods section provides someinformation about the quick and advanced installation methods.

How many hosts do you require in the cluster? The Environment Scenarios section providesmultiple examples of Single Master and Multiple Master configurations.

How many pods are required in your cluster? The Sizing Considerations section provides limitsfor nodes and pods so you can calculate how large your environment needs to be.

Is high availability required? High availability is recommended for fault tolerance. In thissituation, you might aim to use the Multiple Masters Using Native HA example as a basis foryour environment.

Which installation type do you want to use: RPM or containerized? Both installations provide aworking OpenShift Container Platform environment, but you might have a preference for aparticular method of installing, managing, and updating your services.

Which identity provider do you use for authentication? If you already use a supported identityprovider, it is a best practice to configure OpenShift Container Platform to use that identityprovider during advanced installation.

Is my installation supported if integrating with other technologies? See the OpenShift ContainerPlatform Tested Integrations for a list of tested integrations.

2.1.2. Installation Methods

Both the quick and advanced installation methods are supported for development and productionenvironments. If you want to quickly get OpenShift Container Platform up and running to try out for thefirst time, use the quick installer and let the interactive CLI guide you through the configuration optionsrelevant to your environment.

For the most control over your cluster’s configuration, you can use the advanced installation method.This method is particularly suited if you are already familiar with Ansible. However, following along withthe OpenShift Container Platform documentation should equip you with enough information to reliablydeploy your cluster and continue to manage its configuration post-deployment using the provided Ansibleplaybooks directly.

If you install initially using the quick installer, you can always further tweak your cluster’s configurationand adjust the number of hosts in the cluster using the same installer tool. If you wanted to later switch tousing the advanced method, you can create an inventory file for your configuration and carry on thatway.

2.1.3. Sizing Considerations


21

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#admin-guide-high-availabilityhttps://access.redhat.com/articles/2176281

Determine how many nodes and pods you require for your OpenShift Container Platform cluster. Clusterscalability correlates to the number of pods in a cluster environment. That number influences the othernumbers in your setup.

The following table provides the maximum sizing limits for nodes and pods:

Type Maximum

Maximum nodes per cluster 2000

Maximum pods per cluster 120,000

Maximum pods per node 250

Maximum pods per core 10

IMPORTANT

Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.

Determine how many pods are expected to fit per node:

Maximum Pods per Cluster / Expected Pods per Node = Total Number of Nodes

Example Scenario

If you want to scope your cluster for 2200 pods per cluster, you would need at least 9 nodes, assumingthat there are 250 maximum pods per node:

2200 / 250 = 8.8

If you increase the number of nodes to 20, then the pod distribution changes to 110 pods per node:

2200 / 20 = 110

2.1.4. Environment Scenarios

This section outlines different examples of scenarios for your OpenShift Container Platform environment.Use these scenarios as a basis for planning your own OpenShift Container Platform cluster, based onyour sizing needs.

NOTE

Moving from a single master cluster to multiple masters after installation is not supported.

For information on updating labels, see Updating Labels on Nodes.

2.1.4.1. Single Master and Node on One System


22

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#disabling-swap-memoryhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#updating-labels-on-nodes

OpenShift Container Platform can be installed on a single system for a development environment only.An all-in-one environment is not considered a production environment.

2.1.4.2. Single Master and Multiple Nodes

The following table describes an example environment for a single master (with embedded etcd) andtwo nodes:

Host Name Infrastructure Component to Install

master.example.com Master and node

node1.example.com Node

node2.example.com

2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes

The following table describes an example environment for a single master, three etcd hosts, and twonodes:


master.example.com Master and node

etcd1.example.com etcd

etcd2.example.com

etcd3.example.com


node2.example.com

NOTE

When specifying multiple etcd hosts, external etcd is installed and configured. Clusteringof OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.4. Multiple Masters Using Native HA

The following describes an example environment for three masters, one HAProxy load balancer, threeetcd hosts, and two nodes using the native HA method:


23

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/developer_guide/#dev-guide-promoting-application-dehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#node


master1.example.com Master (clustered using native HA) and node

master2.example.com

master3.example.com

lb.example.com HAProxy to load balance API master endpoints

etcd1.example.com etcd

etcd2.example.com

etcd3.example.com


node2.example.com

NOTE

When specifying multiple etcd hosts, external etcd is installed and configured. Clusteringof OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.5. Stand-alone Registry

You can also install OpenShift Container Platform to act as a stand-alone registry using the OpenShiftContainer Platform’s integrated registry. See Installing a Stand-alone Registry for details on thisscenario.

2.1.5. RPM Versus Containerized

An RPM installation installs all services through package management and configures services to runwithin the same user space, while a containerized installation installs services using container imagesand runs separate services in individual containers.

See the Installing on Containerized Hosts topic for more details on configuring your installation to usecontainerized services.

2.2. PREREQUISITES

2.2.1. System Requirements

The following sections identify the hardware specifications and system-level requirements of all hostswithin your OpenShift Container Platform environment.

2.2.1.1. Red Hat Subscriptions


24

You must have an active OpenShift Container Platform subscription on your Red Hat account toproceed. If you do not, contact your sales representative for more information.

IMPORTANT

OpenShift Container Platform 3.6 requires Docker 1.12.

2.2.1.2. Minimum Hardware Requirements

The system requirements vary per host type:

MastersPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: RHEL 7.3 or 7.4 with the "Minimal" installation option and the latestpackages from the Extras channel, or RHEL Atomic Host 7.3.6 or later.

Minimum 4 vCPU (additional are strongly recommended).

Minimum 16 GB RAM (additional memory is strongly recommended, especially if etcdis co-located on masters).

Minimum 40 GB hard disk space for the file system containing /var/.

Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

Minimum 1 GB hard disk space for the file system containing the system’s temporary

directory.

Masters with a co-located etcd require a minimum of 4 cores. 2 core systems will notwork.

NodesPhysical or virtual system, or an instance running on a public or private IaaS.

Base OS: RHEL 7.3 or 7.4 with "Minimal" installation option, or RHEL Atomic Host7.3.6 or later.

NetworkManager 1.0 or later.

1 vCPU.

Minimum 8 GB RAM.

Minimum 15 GB hard disk space for the file system containing /var/.

Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

Minimum 1 GB hard disk space for the file system containing the system’s temporary

directory.

An additional minimum 15 GB unallocated space to be used for Docker’s storage backend; see Configuring Docker Storage.


25

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#node

SeparateetcdNodes

Minimum 20 GB hard disk space for etcd data.

Consult Hardware Recommendations to properly size your etcd nodes.

Currently, OpenShift Container Platform stores image, build, and deployment metadatain etcd. You must periodically prune old resources. If you are planning to leverage alarge number of images/builds/deployments, place etcd on machines with largeamounts of memory and fast SSD drives.

Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires making changes tothe default configuration. See Managing Storage with Docker-formatted Containers for instructions onconfiguring this during or after installation.

The system’s temporary directory is determined according to the rules defined in the tempfilemodule in Python’s standard library.

IMPORTANT

OpenShift Container Platform only supports servers with the x86_64 architecture.

2.2.1.3. Production Level Hardware Requirements

Test or sample environments function with the minimum requirements. For production environments, thefollowing recommendations apply:

Master Hosts

In a highly available OpenShift Container Platform cluster with a separate etcd cluster, a master hostshould have, in addition to the minimum requirements in the table above, 1 CPU core and 1.5 GB ofmemory for each 1000 pods. Therefore, the recommended size of a master host in an OpenShiftContainer Platform cluster of 2000 pods would be the minimum requirements of 2 CPU cores and 16GB of RAM, plus 2 CPU cores and 3 GB of RAM, totaling 4 CPU cores and 19 GB of RAM.

When planning an environment with multiple masters, a minimum of three etcd hosts and a load-balancer between the master hosts are required.

The OpenShift Container Platform master caches deserialized versions of resources aggressively toease CPU load. However, in smaller clusters of less than 1000 pods, this cache can waste a lot ofmemory for negligible CPU load reduction. The default cache size is 50,000 entries, which, depending onthe size of your resources, can grow to occupy 1 to 2 GB of memory. This cache size can be reducedusing the following setting the in /etc/origin/master/master-config.yaml:

kubernetesMasterConfig: apiServerArguments: deserialization-cache-size: - "1000"

Node Hosts

The size of a node host depends on the expected size of its workload. As an OpenShift ContainerPlatform cluster administrator, you will need to calculate the expected workload, then add about 10percent for overhead. For production environments, allocate enough resources so that a node hostfailure does not affect your maximum capacity.


26

https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendationshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#admin-guide-pruning-resourceshttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/managing_containers/managing_storage_with_docker_formatted_containershttps://docs.python.org/2/library/tempfile.html#tempfile.tempdir

Use the above with the following table to plan the maximum loads for nodes and pods:

Host Sizing Recommendation

Maximum nodes per cluster 2000

Maximum pods per cluster 120000

Maximum pods per nodes 250

Maximum pods per core 10

IMPORTANT

Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.

2.2.1.4. Configuring Core Usage

By default, OpenShift Container Platform masters and nodes use all available cores in the system theyrun on. You can choose the number of cores you want OpenShift Container Platform to use by settingthe GOMAXPROCS environment variable.

For example, run the following before starting the server to make OpenShift Container Platform only runon one core:

# export GOMAXPROCS=1

2.2.1.5. SELinux

Security-Enhanced Linux (SELinux) must be enabled on all of the servers before installing OpenShiftContainer Platform or the installer will fail. Also, configure SELINUXTYPE=targeted in the/etc/selinux/config file:

# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=enforcing# SELINUXTYPE= can take one of these three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPE=targeted

Using OverlayFSOverlayFS is a union file system that allows you to overlay one file system on top of another.


27

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#disabling-swap-memoryhttps://golang.org/pkg/runtime/

As of Red Hat Enterprise Linux 7.4, you have the option to configure your OpenShift Container Platformenvironment to use OverlayFS. The overlay2 graph driver is fully supported in addition to the older overlay driver. However, Red Hat recommends using overlay2 instead of overlay, because of itsspeed and simple implementation.

Comparing the Overlay Versus Overlay2 Graph Drivers has more information about the overlay andoverlay2 drivers.

See the Overlay Graph Driver section of the Atomic Host documentation for instructions on how to toenable the overlay2 graph driver for the Docker service.

2.2.1.6. NTP

You must enable Network Time Protocol (NTP) to prevent masters and nodes in the cluster from goingout of sync. Set openshift_clock_enabled to true in the Ansible playbook to enable NTP onmasters and nodes in the cluster during Ansible installation.

# openshift_clock_enabled=true

2.2.1.7. Security Warning

OpenShift Container Platform runs containers on your hosts, and in some cases, such as buildoperations and the registry service, it does so using privileged containers. Furthermore, those containersaccess your host’s Docker daemon and perform docker build and docker push operations. Assuch, you should be aware of the inherent security risks associated with performing docker runoperations on arbitrary images as they effectively have root access.

For more information, see these articles:

http://opensource.com/business/14/7/docker-security-selinux

https://docs.docker.com/engine/security/security/

To address these risks, OpenShift Container Platform uses security context constraints that control theactions that pods can perform and what it has the ability to access.

2.2.2. Environment Requirements

The following section defines the requirements of the environment containing your OpenShift ContainerPlatform configuration. This includes networking considerations and access to external services, such asGit repository access, storage, and cloud infrastructure providers.

2.2.2.1. DNS

OpenShift Container Platform requires a fully functional DNS server in the environment. This is ideally aseparate host running DNS software and can provide name resolution to hosts and containers runningon the platform.

IMPORTANT

Adding entries into the /etc/hosts file on each host is not enough. This file is not copiedinto containers running on the platform.


28

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/scaling_and_performance_guide/#comparing-overlay-graph-drivershttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/managing_containers/#overlay_graph_driverhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#containershttp://opensource.com/business/14/7/docker-security-selinuxhttps://docs.docker.com/engine/security/security/https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#security-context-constraints

Key components of OpenShift Container Platform run themselves inside of containers and use thefollowing process for name resolution:

1. By default, containers receive their DNS configuration file (/etc/resolv.conf) from their host.

2. OpenShift Container Platform then inserts one DNS value into the pods (above the node’snameserver values). That value is defined in the /etc/origin/node/node-config.yaml file by the dnsIP parameter, which by default is set to the address of the host node because the host isusing dnsmasq.

3. If the dnsIP parameter is omitted from the node-config.yaml file, then the value defaults to thekubernetes service IP, which is the first nameserver in the pod’s /etc/resolv.conf file.

As of OpenShift Container Platform 3.2, dnsmasq is automatically configured on all masters and nodes.The pods use the nodes as their DNS, and the nodes forward the requests. By default, dnsmasq isconfigured on the nodes to listen on port 53, therefore the nodes cannot run any other type of DNSapplication.

NOTE

NetworkManager is required on the nodes in order to populate dnsmasq with the DNSIP addresses. DNS does not work properly when the network interface for OpenShiftContainer Platform has NM_CONTROLLED=no.

The following is an example set of DNS records for the Single Master and Multiple Nodes scenario:

master A 10.64.33.100node1 A 10.64.33.101node2 A 10.64.33.102

If you do not have a properly functioning DNS environment, you could experience failure with:

Product installation via the reference Ansible-based scripts

Deployment of the infrastructure containers (registry, routers)

Access to the OpenShift Container Platform web console, because it is not accessible via IPaddress alone

2.2.2.1.1. Configuring Hosts to Use DNS

Make sure each host in your environment is configured to resolve hostnames from your DNS server. Theconfiguration for hosts' DNS resolution depend on whether DHCP is enabled. If DHCP is:

Disabled, then configure your network interface to be static, and add DNS nameservers toNetworkManager.

Enabled, then the NetworkManager dispatch script automatically configures DNS based on theDHCP configuration. Optionally, you can add a value to dnsIP in the node-config.yaml file toprepend the pod’s resolv.conf file. The second nameserver is then defined by the host’s firstnameserver. By default, this will be the IP address of the node host.


29

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/administrator_solutions/#node-config-options

NOTE

For most configurations, do not set the openshift_dns_ip option during theadvanced installation of OpenShift Container Platform (using Ansible), becausethis option overrides the default IP address set by dnsIP.

Instead, allow the installer to configure each node to use dnsmasq and forwardrequests to SkyDNS or the external DNS provider. If you do set the openshift_dns_ip option, then it should be set either with a DNS IP thatqueries SkyDNS first, or to the SkyDNS service or endpoint IP (the Kubernetesservice IP).

To verify that hosts can be resolved by your DNS server:

1. Check the contents of /etc/resolv.conf:

$ cat /etc/resolv.conf# Generated by NetworkManagersearch example.comnameserver 10.64.33.1# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

In this example, 10.64.33.1 is the address of our DNS server.

2. Test that the DNS servers listed in /etc/resolv.conf are able to resolve host names to the IPaddresses of all masters and nodes in your OpenShift Container Platform environment:

$ dig @ +short

For example:

$ dig master.example.com @10.64.33.1 +short10.64.33.100$ dig node1.example.com @10.64.33.1 +short10.64.33.101

2.2.2.1.2. Configuring a DNS Wildcard

Optionally, configure a wildcard for the router to use, so that you do not need to update your DNSconfiguration when new routes are added.

A wildcard for a DNS zone must ultimately resolve to the IP address of the OpenShift Container Platformrouter.

For example, create a wildcard DNS entry for cloudapps that has a low time-to-live value (TTL) andpoints to the public IP address of the host where the router will be deployed:

*.cloudapps.example.com. 300 IN A 192.168.133.2

In almost all cases, when referencing VMs you must use host names, and the host names that you usemust match the output of the hostname -f command on each node.


30

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/administrator_solutions/#node-config-optionshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#routers

WARNING

In your /etc/resolv.conf file on each node host, ensure that the DNS server that hasthe wildcard entry is not listed as a nameserver or that the wildcard domain is notlisted in the search list. Otherwise, containers managed by OpenShift ContainerPlatform may fail to resolve host names properly.

2.2.2.2. Network Access

A shared network must exist between the master and node hosts. If you plan to configure multiplemasters for high-availability using the advanced installation method, you must also select an IP to beconfigured as your virtual IP (VIP) during the installation process. The IP that you select must be routablebetween all of your nodes, and if you configure using a FQDN it should resolve on all nodes.

2.2.2.2.1. NetworkManager

NetworkManager, a program for providing detection and configuration for systems to automaticallyconnect to the network, is required. DNS does not work properly when the network interface forOpenShift Container Platform has NM_CONTROLLED=no.

2.2.2.2.2. Configuring firewalld as the firewall

While iptables is the default firewall, firewalld is recommended for new installations. You can enablefirewalld by setting os_firewall_use_firewalld=true in the Ansible inventory file.

[OSEv3:vars]os_firewall_use_firewalld=True

Setting this variable to true opens the required ports and adds rules to the default zone, which ensurethat firewalld is configured correctly.

NOTE

Using the firewalld default configuration comes with limited configuration options, andcannot be overridden. For example, while you can set up a storage network withinterfaces in multiple zones, the interface that nodes communicate on must be in thedefault zone.

2.2.2.2.3. Required Ports

The OpenShift Container Platform installation automatically creates a set of internal firewall rules oneach host using iptables. However, if your network configuration uses an external firewall, such as ahardware-based firewall, you must ensure infrastructure components can communicate with each otherthrough specific ports that act as communication endpoints for certain processes or services.

Ensure the following ports required by OpenShift Container Platform are open on your network andconfigured to allow access between hosts. Some ports are optional depending on your configuration andusage.

Table 2.1. Node to Node


31

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#high-availability-mastershttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/architecture/#master-componentshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.6/html-single/cluster_administration/#overview

4789 UDP Required for SDN communication between pods on separate hosts.

Table 2.2. Nodes to Master

53 or 8053 TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to3.2 or environments upgraded to 3.2 use port 53. New installations will use8053 by default so that dnsmasq may be configured.


443 or 8443 TCP Required for node hosts to communicate to the master API, for the node hoststo post back status, to receive tasks, and so on.

Table 2.3. Master to Node


10250 TCP The master proxies to node hosts via the Kubelet for oc commands.

NOTE

In the following table, (L) indicates the marked port is also used

openshift container platform 3 - red hat customer portal · 2019-04-27 · openshift container...

Documents