-
PivotalContainerService(PKS)
Documentation
v1.6
Published:November14,2019
Copyright©2020VMware,Inc.AllRightsReserved.
Note:ThecontentsofthisPDFmayhavefallenoutofdate.Forcurrentdocumentation,seehttps://docs.pivotal.io/pks/1-6
https://docs.pivotal.io/pks/1-6
-
VMwareEnterprisePKS
In this topic
Overview
WhatEnterprisePKSAddstoKubernetes
Features
EnterprisePKSComponents
EnterprisePKSConcepts
EnterprisePKSPrerequisites
PreparingtoInstallEnterprisePKS
InstallingEnterprisePKS
UpgradingEnterprisePKS
ManagingEnterprisePKS
ManagingKubernetesClustersandWorkloads
BackingUpandRestoringEnterprisePKS
EnterprisePKSSecurity
DiagnosingandTroubleshootingEnterprisePKS
Page last updated:
VMwareEnterprisePKSenablesoperatorstoprovision,operate,andmanageenterprise-gradeKubernetesclustersusingBOSHandPivotalOpsManager.
Overview
EnterprisePKSusestheOn-Demand Broker todeployCloud Foundry Container Runtime ,aBOSHreleasethatoffersauniformwaytoinstantiate,deploy,andmanagehighlyavailableKubernetesclustersonacloudplatformusingBOSH.
AfteroperatorsinstalltheEnterprisePKStileontheOpsManagerInstallationDashboard,developerscanprovisionKubernetesclustersusingthePKSCommandLineInterface(PKSCLI),andruncontainer-basedworkloadsontheclusterswiththeKubernetesCLI,kubectl.
OnPivotal Platform ,youcanrunEnterprisePKSstandaloneoralongsidePivotalApplicationService.
WhatEnterprisePKSAddstoKubernetes
ThefollowingtabledetailsthefeaturesthatEnterprisePKSaddstotheKubernetesplatform.
Feature Included in K8s Included in Enterprise PKS
Singletenantingress ✓ ✓
Securemulti-tenantingress ✓
Statefulsetsofpods ✓ ✓
Multi-containerpods ✓ ✓
Rollingupgradestopods ✓ ✓
Copyright©2020VMware,Inc.AllRightsReserved. 2 1.6
https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://docs-cfcr.cfapps.io/https://docs.pivotal.io/platform
-
Rollingupgradestoclusterinfrastructure ✓
Podscalingandhighavailability ✓ ✓
Clusterprovisioningandscaling ✓
MonitoringandrecoveryofclusterVMsandprocesses ✓
Persistentdisks ✓ ✓
Securecontainerregistry ✓
Embedded,hardenedoperatingsystem ✓
Features
EnterprisePKShasthefollowingfeatures:
Kubernetes compatibility:ConstantcompatibilitywithcurrentstablereleaseofKubernetes
Production-ready:Highlyavailablefromapplicationstoinfrastructure,withnosinglepointsoffailure
BOSH advantages:Built-inhealthchecks,scaling,auto-healingandrollingupgrades
Fully automated operations:Fullyautomateddeploy,scale,patch,andupgradeexperience
Multi-cloud:Consistentoperationalexperienceacrossmultipleclouds
GCP APIs access:TheGoogleCloudPlatform(GCP)ServiceBrokergivesapplicationsaccesstotheGoogleCloudAPIs,andGoogleContainerEngine(GKE)consistencyenablesthetransferofworkloadsfromortoGCP
OnvSphere,EnterprisePKSsupportsdeployingandrunningKubernetesclustersinair-gappedenvironments.
FeatureSupportbyIaaS
AWS Azure GCP vSphere with Flannel vSphere with NSX-TAutomatic Kubernetes Cluster API loadbalancer ✓
HTTP proxy ✓ ✓ ✓
Multi-AZ storage ✓ ✓
Per-namespace subnets ✓Service type:LoadBalancer ✓ ✓ ✓ ✓
Windows worker-based cluster ✓
FormoreinformationaboutconfiguringService type:LoadBalancer onAWS,seetheAccess Workloads Using an Internal AWS LoadBalancersectionofDeployingandExposingBasicLinuxWorkloads.
EnterprisePKSComponents
ThePKScontrolplanecontainsthefollowingcomponents:
AnOn-Demand Broker thatdeploysCloud Foundry Container Runtime (CFCR),anopen-sourceprojectthatprovidesasolutionfordeployingandmanagingKubernetes clustersusingBOSH .
AServiceAdapter
*
*
Copyright©2020VMware,Inc.AllRightsReserved. 3 1.6
https://docs.pivotal.io/svc-sdk/odb/https://docs-cfcr.cfapps.io/https://kubernetes.io/docs/home/https://bosh.io/docs
-
ThePKSAPI
FormoreinformationaboutthePKScontrolplane,seeEnterprise PKS Cluster Management.
ForadetailedlistofcomponentsandsupportedversionsbyaparticularEnterprisePKSrelease,seetheEnterprise PKS ReleaseNotes.
EnterprisePKSConcepts
ForconceptualinformationaboutEnterprisePKS,seeEnterprise PKS Concepts.
EnterprisePKSPrerequisites
ForinformationabouttheresourcerequirementsforinstallingEnterprisePKS,seethetopicthatcorrespondstoyourcloudprovider:
vSphere Prerequisites and Resource Requirements
vSphere with NSX-T Version RequirementsandHardware Requirements for Enterprise PKS on vSphere with NSX-T
GCP Prerequisites and Resource Requirements
AWS Prerequisites and Resource Requirements
Azure Prerequisites and Resource Requirements
PreparingtoInstallEnterprisePKS
ToinstallEnterprisePKS,youmustdeployOpsManager.YouuseOpsManagertoinstallandconfigureEnterprisePKS.
IfyouareinstallingEnterprisePKStovSphere,youcanalsoconfigureintegrationwithNSX-TandHarbor.
Consultthefollowingtableforcompatibilityinformation:
IaaS Ops Manager v2.6.16+ or v2.7.6+ NSX-T Harbor
vSphere Required Available Available
GCP Required NotAvailable Available
AWS Required NotAvailable Available
Azure Required NotAvailable Available
Formoreinformationaboutcompatibilityandcomponentversions,seetheEnterprise PKS Release Notes.
ForinformationaboutpreparingyourenvironmentbeforeinstallingEnterprisePKS,seethetopicthatcorrespondstoyourcloudprovider:
vSphere
vSphere with NSX-T Integration
GCP
AWS
Azure
Copyright©2020VMware,Inc.AllRightsReserved. 4 1.6
-
InstallingEnterprisePKS
ForinformationaboutinstallingEnterprisePKS,seeInstallingEnterprisePKSforyourIaaS:
vSphere
vSphere with NSX-T Integration
Google Cloud Platform (GCP)
Amazon Web Services (AWS)
Microsoft Azure (Azure)
UpgradingEnterprisePKS
ForinformationaboutupgradingtheEnterprisePKStileandEnterprisePKS-deployedKubernetesclusters,seeUpgrading EnterprisePKS.
ManagingEnterprisePKS
Forinformationaboutconfiguringauthentication,creatingusers,andmanagingyourEnterprisePKSdeployment,seeManagingEnterprise PKS.
ManagingKubernetesClustersandWorkloads
ForinformationaboutmanagingEnterprisePKS-provisionedKubernetesclustersanddeployingworkloads,seeManagingKubernetes Clusters and Workloads.
BackingUpandRestoringEnterprisePKS
ForinformationaboutusingBOSHBackupandRestore(BBR)tobackupandrestoreEnterprisePKS,seeBacking Up and RestoringEnterprise PKS.
EnterprisePKSSecurity
ForinformationaboutsecurityinEnterprisePKS,seeEnterprise PKS Security.
DiagnosingandTroubleshootingEnterprisePKS
ForinformationaboutdiagnosingandtroubleshootingissuesinstallingorusingEnterprisePKS,seeDiagnosing andTroubleshooting Enterprise PKS.
Copyright©2020VMware,Inc.AllRightsReserved. 5 1.6
mailto:[email protected]
-
ReleaseNotes
In this topic
v1.6.2Features
ProductSnapshot
vSphereVersionRequirements
UpgradePath
BreakingChanges
KnownIssues
v1.6.1Features
ProductSnapshot
vSphereVersionRequirements
UpgradePath
BreakingChanges
KnownIssues
v1.6.0Features
ProductSnapshot
vSphereVersionRequirements
UpgradePath
BreakingChanges
KnownIssues
EnterprisePKSManagementConsole1.6.2Features
BugFixes
ProductSnapshot
KnownIssues
EnterprisePKSManagementConsole1.6.1Features
BugFixes
ProductSnapshot
KnownIssues
EnterprisePKSManagementConsole1.6.0-rev.3Features
BugFixes
ProductSnapshot
KnownIssues
EnterprisePKSManagementConsolev1.6.0-rev.2Features
ProductSnapshot
KnownIssues
Page last updated:
Copyright©2020VMware,Inc.AllRightsReserved. 6 1.6
-
ThistopiccontainsreleasenotesforVMwareEnterprisePKSv1.6.
v1.6.2
Release Date:April29,2020
Features
Newfeaturesandchangesinthisrelease:
BumpsKubernetestov1.15.10.
BumpsUAAtov73.4.20.
BumpsPerconaXtraDBCluster(PXC)tov0.22.
BumpsWindowsStemcelltov2019.15.
BumpsODBtov0.38.0.
BumpsApacheTomcat(inPKSAPI)tov9.0.31.
[Security Fix]UAAbumpfixesblindSCIMinjectionvulnerability,CVE-2019-11282.
[Security Fix]UAAbumpfixesCSRFattackvulnerability.
[Security Fix]PXCbumpfixescURL/libcURLbufferoverflowvulnerability,CVE-2019-3822.
[Bug Fix]Improvesthebehaviorofthe pks get-kubeconfig and pks get-credentials commandsduringclusterupdatesandupgrades.Youcannowrunthe pks get-kubeconfig commandduringsingle-andmulti-masterclusterupdates.Additionally,youcanrunthe pks get-credentials commandduringmulti-masterclusterupgrades.
[Bug Fix]NewUAAversionincludesApacheTomcatbumpthatfixesSAMLloginissues.
ProductSnapshot
Element Details
Version v1.6.2
Releasedate April29,2020
CompatibleOpsManagerversions SeePivotal Network
Xenialstemcellversion SeePivotal Network
Windowsstemcellversion v2019.15
Kubernetesversion v1.15.10
On-DemandBrokerversion v0.38.0
CompatibleNSX-Tversions v2.5.1,v2.5.0,v2.4.3
NCPversion v2.5.1
Dockerversion v18.09.9
BackupandRestoreSDKversion v1.17.0
warning:BeforeinstallingorupgradingtoEnterprisePKSv1.6,reviewtheBreaking Changesbelow.
Copyright©2020VMware,Inc.AllRightsReserved. 7 1.6
https://network.pivotal.io/products/pivotal-container-service#/releases/631561https://network.pivotal.io/products/pivotal-container-service#/releases/631561
-
UAAversion v73.4.20
vSphereVersionRequirements
ForEnterprisePKSinstallationsonvSphereoronvSpherewithNSX-TDataCenter,refertotheVMware Product InteroperabilityMatrices .
UpgradePath
ThesupportedupgradepathstoEnterprisePKSv1.6.2arefromEnterprisePKSv1.5.0andlater.
BreakingChanges
AllbreakingchangesinEnterprisePKSv1.6.2arealsoinEnterprisePKSv1.6.0.SeeBreaking Changes in Enterprise PKS v1.6.0.
KnownIssues
AllknownissuesinEnterprisePKSv1.6.2arealsoinEnterprisePKSv1.6.0.SeeKnown Issues in Enterprise PKS v1.6.0.
v1.6.1
Release Date:Jan13,2020
Features
Newfeaturesandchangesinthisrelease:
[Security Fix]SecurestrafficintoKubernetesclusterswithup-to-dateTLS(v1.2+)andapprovedciphersuites.
[Security Fix]BumpsUAAtov73.4.16.ThisupdatepreventsloggingofsecureinformationandenablesthePKSUAAtostartwiththe env.no_proxy propertyset.
[Bug Fix]ResolvesanissuewhereifyouareusingOpsManagerv2.7andPKSv1.6asafreshinstall,enablingPlans11,12,or13doesnotenableWindowsworker-basedclusters.ItcreatesLinux-basedclustersonly.Formoreinformation,seeEnterprise PKSCreates a Linux Cluster When You Expect a Windows Cluster.
[Bug Fix]ResolvesanissuewhereapplyingchangestoEnterprisePKSfailsifPlan8isenabledintheEnterprisePKStile.Formoreinformation,seeApplying Changes Fails If Plan 8 Is Enabled.
[Bug Fix]Resolvesanissuewherethe pks update-cluster --network-profile commandsets subnet_prefix to0inthencp.inifileifthenetworkprofiledoesnothave pod_subnet_prefix .Formoreinformation,seeNetwork Profile for “pks update-cluster” Does Not Use the Defaults from the Original Cluster Manifest.
[Bug Fix]ResolvesanissuewheretryingtocreateaclusterwithalongnetworkprofilecausesanerrorData too long for column 'nsxt_network_profile' .
UpdatesthesupportedNCPversiontoNCPv2.5.1.RefertotheNCP Release Notes formoreinformation.
SupportforNSX-Tv2.5.1.
ProductSnapshot
Copyright©2020VMware,Inc.AllRightsReserved. 8 1.6
https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/rn/NSX-Container-Plugin-251-Release-Notes.html
-
Element Details
Version v1.6.1
Releasedate January13,2020
CompatibleOpsManagerversions SeePivotal Network
Xenialstemcellversion SeePivotal Network
Windowsstemcellversion v2019.7
Kubernetesversion v1.15.5
On-DemandBrokerversion v0.29.0
CompatibleNSX-Tversions v2.5.1,v2.5.0,v2.4.3
NCPversion v2.5.1
Dockerversion v18.09.9
BackupandRestoreSDKversion v1.17.0
UAA v73.4.16
vSphereVersionRequirements
ForEnterprisePKSinstallationsonvSphereoronvSpherewithNSX-TDataCenter,refertotheVMware Product InteroperabilityMatrices .
UpgradePath
ThesupportedupgradepathstoEnterprisePKSv1.6.1arefromEnterprisePKSv1.5.0andlater.
BreakingChanges
AllbreakingchangesinEnterprisePKSv1.6.1arealsoinEnterprisePKSv1.6.0.SeeBreaking Changes in Enterprise PKS v1.6.0.
KnownIssues
AllknownissuesinEnterprisePKSv1.6.1arealsoinEnterprisePKSv1.6.0.SeeKnown Issues in Enterprise PKS v1.6.0.
v1.6.0
Release Date:November14,2019
Features
Thissectiondescribesnewfeaturesandchangesinthisrelease.
PKSControlPlaneandAPI
EnterprisePKSv1.6.0updatesinclude:
EnablesoperatorstoupgrademultipleKubernetesclusterssimultaneouslyandtodesignatespecificupgradeclustersascanary
Copyright©2020VMware,Inc.AllRightsReserved. 9 1.6
https://network.pivotal.io/products/pivotal-container-service#/releases/551663https://network.pivotal.io/products/pivotal-container-service#/releases/551663https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=
-
clusters.Formoreinformationaboutmultipleclusterupgrades,seeUpgrade ClustersinUpgradingClusters.
AddsanewUAAscope, pks.clusters.admin.read ,forEnterprisePKSusers.ForinformationaboutUAAscopes,seeUAAScopes for Enterprise PKS UsersandManaging Enterprise PKS Users with UAA.
ProvidesexperimentalintegrationwithTanzuMissionControl.Formoreinformation,seeTanzu Mission Control Integration.
EnablesoperatorstolimitthetotalnumberofclustersausercanprovisioninEnterprisePKS.Formoreinformationaboutquotas,seeManaging Resource Usage with QuotasandViewing Usage Quotas.
EnablesoperatorstoconfigureasingleKubernetesclusterwithaspecificDockerRegistryCAcertificate.FormoreinformationaboutconfiguringaclusterwithaDockerRegistryCAcertificate,seeConfiguring Enterprise PKS Clusters with Private DockerRegistry CA Certificates (Beta).
Updatesthe pks delete-cluster PKSCLIcommandsothatallclusterobjects,includingNSX-Tnetworkingobjects,aredeletedwithouttheneedtousethe bosh delete deployment commandtoremovefailedclusterdeletions.
KubernetesControlPlane
EnterprisePKSv1.6.0updatesinclude:
IncreasestheWorker VM Max in Flightdefaultvaluefrom 1 to 4 inthePKS APIconfigurationpane,whichacceleratesclustercreationbyallowinguptofournewnodestobeprovisionedsimultaneously.TheupdateddefaultvalueisonlyappliedduringnewEnterprisePKSinstallationandisnotappliedduringanEnterprisePKSupgrade.IfyouareupgradingEnterprisePKSfromapreviousversionandwanttoacceleratemulti-clusterprovisioning,youcanincreasethevalueofWorker VM Max inFlightmanually.
PKSMonitoringandLogging
EnterprisePKSv1.6.0updatesinclude:
RedesignstheLoggingandMonitoringpanesoftheEnterprisePKStileandrenamesthemtoHost MonitoringandIn-ClusterMonitoring.Forinformationaboutconfiguringthesepanes,seetheInstallingEnterprisePKStopicforyourIaaS.
AddstheMax Message SizefieldintheHost Monitoringpane.Thisallowsyoutoconfigurethemaximumnumberofcharactersofalogmessagethatisforwardedtoasyslogendpoint.Thisfeaturehelpsensurethatlogmessagesarenottruncatedatthesyslogendpoint.Bydefault,theMax Message Sizefieldis10,000characters.Formoreinformation,seeHost MonitoringintheInstallingEnterprisePKStopicforyourIaaS.
AddstheInclude kubelet metricssetting.ThisenablesoperatorstocollectworkloadmetricsacrossallKubernetesclusters.Formoreinformation,seeHost MonitoringintheInstallingEnterprisePKStopicforyourIaaS.
AddssupportforFluentBitoutputpluginstologsinks.ForinformationaboutconfiguringFluentBitoutputplugins,seeCreate aClusterLogSink or LogSink Resource with a Fluent Bit Output PlugininCreatingandManagingSinkResources.
Addssupportforfilteringlogsandeventsfroma ClusterLogSink or LogSink resource.Formoreinformation,seeFilter SinksinCreatingandManagingSinkResources.
WindowsonPKS
EnterprisePKSv1.6.0updatesinclude:
AddssupportforfloatingWindowsstemcellsonvSphere.ForinformationaboutKubernetesclusterswithWindowsworkersinEnterprisePKS,seeConfiguring Windows Worker-Based Kubernetes Clusters (Beta).
EnablesoperatorstoconfigurethelocationoftheWindowspauseimage.ForinformationaboutconfiguringKubeletcustomization - Windows pause image location,seePlansinConfiguringWindowsWorker-BasedKubernetesClusters(Beta).
Copyright©2020VMware,Inc.AllRightsReserved. 10 1.6
-
PKSwithNSX-TNetworking
EnterprisePKSv1.6.0updatesinclude:
NSXErrorCRDletsclustermanagersandusersviewNSXerrorsinKubernetesresourceannotations,andusethecommandkubectl get nsxerror toviewthehealthstatusofNSX-Tclusternetworkingobjects(NCPv2.5.0+).Formoreinformation,seeViewing the Health Status of Cluster Networking Objects (NSX-T only).
DFWlogcontrolfordroppedtrafficletsclusteradministratorsdefinenetworkprofiletoturnonloggingandloganydroppedorrejectedpacketbyNSX-Tdistributedfirewallrules(NCPv2.5.0+).Formoreinformation,seeDefining Network Profiles for NCPLogging.
LoadbalancerandingressresourcecapacityobservabilityusingtheNSXLoadBalancerMonitorCRDletsclustermanagersandusersusethecommand kubectl get nsxLoadBalancerMonitors toviewahealthscorethatreflectsthecurrentperformanceoftheNSX-Tloadbalancerservice,includingusage,traffic,andcurrentstatus(NCPv2.5.1+).Formoreinformation,seeIngress Scaling(NSX-T only).
IngressscaleoutusingtheLoadBalancerCRDletsclustermanagersscaleouttheNSX-Tloadbalancerforingressrouting(NCPv2.5.1+).Formoreinformation,seeIngress Scaling (NSX-T only).
SupportforIngressURLRewrite.Formoreinformation,seeUsing Ingress URL Rewrite.
SupportforActive–ActiveTier-0routerconfigurationwhenusingaShared-Tier-1 topology.
AbilitytoplacetheloadbalancerandTier-1Active/Standbyroutersondifferentfailuredomains.SeeMultisite Deployment ofNSX-T Data Center formoreinformation.
PKSonAWSNetworking
EnterprisePKSv1.6.0updatesinclude:
SupportforHTTP/HTTPSProxyonAWS.Formoreinformationsee,Using Proxies with Enterprise PKS on AWS.
CustomerExperienceImprovementProgram
EnterprisePKSv1.6.0updatesinclude:
AdministratorscannameEnterprisePKSinstallationssotheyaremoreeasilyrecognizableinreports.Formoreinformation,seeSample Reports.
ComponentUpdates
EnterprisePKSv1.6.0updatesinclude:
BumpsKubernetestov1.15.5.
BumpsUAAtov73.4.8.
BumpsJacksondependenciesinthePKSAPI.
BugFixes
EnterprisePKSv1.6.0includesthefollowingbugfixes:
FixesanissuewhereenablingtheAvailabilitySetsmodeattheBOSHDirector>AzureConfigresultedinthekubeletfailingtostartonprovisioningofaKubernetescluster.
Copyright©2020VMware,Inc.AllRightsReserved. 11 1.6
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/administration/GUID-5D7E3D43-6497-4273-99C1-77613C36AD75.html
-
FixesanissuewherepersistentvolumeattachmentfailedonvSphereinascenariowhereanAZdefinedinOpsManagerdoesnotcontainaresourcepool.
Increases network_profile columnsize.
FixesaTelemetryeventgenerationissuewherethe upgrade_cluster_end eventisnotsentforcompletedclusterupgrades.
FixesanissuewherenetworkingchangesdidnotpropagatewhenupgradingfromEnterprisePKSv1.5orlater.
FixesanissuewheretheIngressIPaddresswasexcludedfromtheEnterprisePKSfloatingIPpool.
FixesanissuewherethePKSOSBProxystartwasdelayedbyscanningallNSX-Tfirewallrules.
FixesanissuewiththePKSclustersupgradeerrandnotpushingthelatestNSX-TcertificatetoKubernetesMasternodes.
FixesanissuewiththePKSOSBProxytakingalongtimetostartduetoscanningallNSX-Tfirewallrules.
FixesanissuewithPKSreleasingfloatingIPaddressesincompletelywhiledeletingclustersunderactive/activemode.
FixesanissuewiththeDNSLookupFeature:INGRESSIPnotkeptoutofPKSFloatingIPpool.
Fixesanissuewiththecommand pks cluster details doesnotdisplayNSGroupIDofmasterVMs.
ChecksthehighavailabilitymodeoftheTier-0routerbeforecreatingPKSacluster.
ProductSnapshot
Element Details
Version v1.6.0
Releasedate November14,2019
CompatibleOpsManagerversions SeePivotal Network
Xenialstemcellversion SeePivotal Network
Windowsstemcellversion v2019.7
Kubernetesversion v1.15.5
On-DemandBrokerversion v0.29.0
CompatibleNSX-Tversions v2.5.0,v2.4.3
NCPversion v2.5.1
Dockerversion v18.09.9
BackupandRestoreSDKversion v1.17.0
UAA v73.4.8
vSphereVersionRequirements
ForEnterprisePKSinstallationsonvSphereoronvSpherewithNSX-TDataCenter,refertotheVMware Product InteroperabilityMatrices .
UpgradePath
ThesupportedupgradepathstoEnterprisePKSv1.6.0arefromEnterprisePKSv1.5.0andlater.
BreakingChanges
EnterprisePKSv1.6.0hasthefollowingbreakingchanges:
Copyright©2020VMware,Inc.AllRightsReserved. 12 1.6
https://network.pivotal.io/products/pivotal-container-service#/releases/501833https://network.pivotal.io/products/pivotal-container-service#/releases/501833https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=
-
PersistentVolumeDataLosswithWorkerReboot
WitholdversionsofOpsManager,PKSworkernodeswithpersistentdiskvolumesmaygetstuckinastartupstateandlosedatawhentheyarerebootedmanuallyfromthedashboardorautomaticallybyvSphereHA.
ThisissueisfixedinthefollowingOpsManagerversions:
v2.8.0+
v2.7.6+
v2.6.16+
ForallPKSinstallationsthathostworkersusingpersistentvolumes,PivotalrecommendsupgradingtooneoftheOpsManagerversionsabove.
EnterprisePKSRemovesSinkCommandsinthePKSCLI
EnterprisePKSremovesthefollowingEnterprisePKSCommandLineInterface(PKSCLI)commands:
pks create-sink
pks sinks
pks delete-sink
YoucanusethefollowingKubernetesCLIcommandsinstead:
kubectl apply -f YOUR-SINK.yml
kubectl get clusterlogsinks
kubectl delete clusterlogsink YOUR-SINK
Formoreinformationaboutdefiningandmanagingsinkresources,seeCreating and Managing Sink Resources.
ChangestoPKSAPIEndpoints
Thisreleasemovesthe clusters , compute-profiles , quotas ,and usages PKSAPIendpointsfrom v1beta1 to v1 . v1beta1 isnolongersupportedfortheseendpoints.Youmustuse v1 .Forexample,insteadof https://YOUR-PKS-API-FQDN:9021/v1beta1/quotas ,usehttps://YOUR-PKS-API-FQDN:9021/v1/quotas .
KnownIssues
EnterprisePKSv1.6.0hasthefollowingknownissues.
YourKubernetesAPIServerCACertificateExpiresUnlessYouRegenerateIt
Symptom
YourKubernetesAPIserver’s tls-kubernetes-2018 certificateisaone-yearcertificateinsteadofafour-yearcertificate.
Explanation
WhenyouupgradedfromPKSv1.2.7toPKSv1.3.1,theupgradeprocessextendedthelifespanofallPKSCAcertificatestofouryears,exceptfortheKubernetesAPIserver’s tls-kubernetes-2018 certificate.The tls-kubernetes-2018 certificateremainedaone-yearcertificate.
Copyright©2020VMware,Inc.AllRightsReserved. 13 1.6
-
Unlessyouregeneratethe tls-kubernetes-2018 certificateitretainsitsone-yearlifespan,eventhroughsubsequentEnterprisePKSupgrades.
Workaround
Ifyouhavenotalreadydoneso,youshouldreplacetheKubernetesAPIserver’sone-year tls-kubernetes-2018 certificatebeforeitexpires.Forinformationaboutgeneratingandapplyinganewfour-year tls-kubernetes-2018 certificate,seeHow to regenerate tls-kubernetes-2018 certificate when it is not regenerated in the upgrade to PKS v1.3.x inthePivotalKnowledgeBase.
ClusterUpgradeDoesNotUpgradeKubernetesVersiononWindowsWorkers
WhenPKSclustersareupgraded,WindowsworkernodesintheclusterdonotupgradetheirKubernetesversion.ThemasterandLinuxworkernodesintheclusterdoupgradetheirKubernetesversionasexpected.
WhentheKubernetesversionofaWindowsworkerdoesnotexactlymatchtheversionofthemasternode,theclusterstillfunctions.
kube-apiserver hasnorestrictiononlaggingpatchbumps.
PKSclustersupgrademanuallywiththe pksupgrade-cluster command,orautomaticallywithPKSupgradeswhentheUpgrade allclusterserrandissettoDefault (On)inthePKStileErrandspane.
NetworkProfilefor“pksupdate-cluster”DoesNotUsetheDefaultsfromtheOriginalClusterManifest
Symptom
TheNetworkprofilefor pksupdate-cluster usescontentsthatarebeingupdatedandnotusingthedefaultsfromtheoriginalclustermanifest.
Explanation
The pksupdate-cluster operationsetsthe subnet_prefix to0inthencp.inifilewhenthenetwork-profilehas pod_ip_block_ids setbutitdoesnothave pod_subnet_prefix .
Workaround
Whencreatingthenetworkprofiletobeusedforupdate,includeallthebelowfields.Thenupdate-clusterwiththenetworkprofileshouldwork.
{"name":"np","parameters":{"t0_router_id":"c501f114-870b-4eda-99ac-966adf464452","fip_pool_ids":["b7acbda8-46de-4195-add2-5fb11ca46cbf"],"pod_ip_block_ids":["b03bff60-854b-4ccb-9b2b-016867b319c9","234c3652-69e7-4365-9627-8e0d8d4a6b86"],"pod_subnet_prefix":24,"single_tier_topology":false}}
AzureDefaultSecurityGroupIsNotAutomaticallyAssignedtoClusterVMs
Note:ThisissueisresolvedinEnterprisePKSv1.6.1.
Copyright©2020VMware,Inc.AllRightsReserved. 14 1.6
https://community.pivotal.io/s/article/How-to-regenerate-tls-kubernetes-2018-certificate-when-it-was-not-regenerated-in-the-upgrade-to-PKS-v1-3-xhttps://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
-
Symptom
Youexperienceissueswhenconfiguringaloadbalancerforamulti-masterKubernetesclusterorcreatingaserviceoftypeLoadBalancer .Additionally,intheAzureportal,theVM>NetworkingpagedoesnotdisplayanyinboundandoutboundtrafficrulesforyourclusterVMs.
Explanation
AspartofconfiguringtheEnterprisePKStileforAzure,youenterDefault Security GroupintheKubernetes Cloud Providerpane.WhenyoucreateaKubernetescluster,EnterprisePKSautomaticallyassignsthissecuritygrouptoeachVMinthecluster.However,onAzuretheautomaticassignmentmaynotoccur.
Asaresult,yourinboundandoutboundtrafficrulesdefinedinthesecuritygrouparenotappliedtotheclusterVMs.
Workaround
Ifyouexperiencethisissue,manuallyassignthedefaultsecuritygrouptoeachVMNICinyourcluster.
ClusterCreationFailsWhenFirstAZRunsOutofResources
Symptom
Ifthefirstavailabilityzone(AZ)usedbyaplanwithmultipleAZsrunsoutofresources,clustercreationfailswithanerrorlikethefollowing:
LError:CPIerror'Bosh::Clouds::CloudError'withmessage'Novalidplacementfoundforrequestedmemory:4096
Explanation
BOSHcreatesVMsforyourEnterprisePKSdeploymentusingaround-robinalgorithm,creatingthefirstVMinthefirstAZthatyourplanuses.IftheAZrunsoutofresources,clustercreationfailsbecauseBOSHcannotcreatetheclusterVM.
Forexample,ifyouhavethreeAZsandyoucreatetwoclusterswithfourworkerVMseach,BOSHdeploysVMsinthefollowingAZs:
AZ1 AZ2 AZ3
Cluster 1 WorkerVM1 WorkerVM2 WorkerVM3
WorkerVM4
Cluster 2 WorkerVM1 WorkerVM2 WorkerVM3
WorkerVM4
Inthisscenario,AZ1hastwiceasmanyVMsasAZ2orAZ3.
ClusterCreationFailswithLongNetworkProfile
Creatingaclusterwithalongnetworkprofile,suchaswithmultiple pod_ip_block_ids values,causesanerrorDatatoolongforcolumn'nsxt_network_profile' .
Note:ThisissueisresolvedinEnterprisePKSv1.6.1.
Copyright©2020VMware,Inc.AllRightsReserved. 15 1.6
-
AzureWorkerNodeCommunicationFailsafterUpgrade
Symptom
OutboundcommunicationfromaworkernodeVMfailsafterupgradingEnterprisePKS.
Explanation
EnterprisePKSusesAzureAvailabilitySetstoimprovetheuptimeofworkloadsandworkernodesintheeventofAzureplatformfailures.WorkernodeVMsaredistributedevenlyacrossAvailabilitySets.
AzureStandardSKULoadBalancersarerecommendedfortheKubernetescontrolplaneandKubernetesingressandegress.ThisloadbalancertypeprovidesanIPaddressforoutboundcommunicationusingSNAT.
Duringanupgrade,whenBOSHrebuildsagivenworkerinstanceinanAvailabilitySet,Azurecantimeoutwhilere-attachingtheworkernodenetworkinterfacetotheback-endpooloftheStandardSKULoadBalancer.
Formoreinformation,seeOutbound connections in Azure intheAzuredocumentation.
Workaround
Youcanmanuallyre-attachtheworkerinstancetotheback-endpooloftheAzureStandardSKULoadBalancerinyourAzureconsole.
ErrorDuringIndividualClusterUpgrades
Symptom
Whilesubmittingalargenumberofclusterupgraderequestsusingthe pksupgrade-cluster command,someofyourKubernetesclustersaremarkedasfailed.
Explanation
BOSHupgradesKubernetesclustersinparallelwithalimitofuptofourconcurrentclusterupgradesbydefault.Ifyouschedulemorethanfourclusterupgrades,EnterprisePKSqueuestheupgradesandwaitsforBOSHtofinishthelastupgrade.WhenBOSHfinishesthelastupgrade,itstartsworkingonthenextupgraderequest.
IfyousubmittoomanyclusterupgradestoBOSH,anerrormayoccur,wheresomeofyourclustersaremarkedas FAILED becauseBOSHcanstarttheupgradeonlywithinthespecifiedtimeout.Thetimeoutissetto168hoursbydefault.However,BOSHdoesnotremovethetaskfromthequeueorstopworkingontheupgradeifithasbeenpickedup.
Solution
IfyouexpectthatupgradingallofyourKubernetesclusterstakesmorethan168hours,donotuseascriptthatsubmitsupgraderequestsforallofyourclustersatonce.ForinformationaboutupgradingKubernetesclustersprovisionedbyEnterprisePKS,seeUpgrading Clusters.
KubectlCLICommandsDoNotWorkafterChanginganExistingPlantoaDifferentAZ
Symptom
AfteryouupdatetheAZofanexistingplan,kubectlCLIcommandsdonotworkforyourclustersassociatedwiththeplan.
Explanation
Copyright©2020VMware,Inc.AllRightsReserved. 16 1.6
https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections
-
ThisissueoccursinIaaSenvironmentsthatdonotsupportattachingadiskacrossmultipleAZs.
WhentheplanofanexistingclusterchangestoadifferentAZ,BOSHmigratestheclusterbycreatingVMsfortheclusterinthenewAZandremovingyourclusterVMsfromtheoriginalAZ.
OnanIaaSthatdoesnotsupportattachingVMdisksacrossAZs,thedisksBOSHattachestothenewVMsdonothavetheoriginalcontent.
Workaround
IfyoucannotrunkubectlCLIcommandsafterreconfiguringtheAZofanexistingcluster,contactSupportforassistance.
ApplyingChangesFailsIfPlan8IsEnabled
Symptom
AfteryouclickApply ChangesontheOpsManagerInstallationDashboard,thefollowingerroroccurs:CannotgeneratemanifestforproductEnterprisePKS
.
Explanation
ThiserroroccursifPlan8isenabledinyourEnterprisePKSv1.6.0tile.
Workaround
DisablePlan8intheEnterprisePKStileandmoveyourplansettingstoaplanthatisavailableforconfiguration,forexample,Plan9or10.
TodisablePlan8:
1. InPlan 8,selectPlan > Inactive.
2. ClickSave.
OnePlanIDLongerthanOtherPlanIDs
Symptom
OneofyourplanIDsisonecharacterlongerthanyourotherplanIDs.
Explanation
InEnterprisePKS,eachplanhasauniqueplanID.AplanIDisnormallyaUUIDconsistingof32alphanumericcharactersand4hyphens.However,thePlan 4IDconsistsof33alphanumericcharactersand4hyphens.
Solution
YoucansafelyconfigureandusePlan 4.ThelengthofthePlan 4IDdoesnotaffectthefunctionalityofPlan 4clusters.
IfyourequireallplanIDstohaveidenticallength,donotactivateorusePlan 4.
Note:ThisissueisresolvedinEnterprisePKSv1.6.1.
Copyright©2020VMware,Inc.AllRightsReserved. 17 1.6
-
KubernetesClusterNameLimitationforTanzuMissionControlIntegration
TanzuMissionControlintegrationcannotattachTanzuMissionControltoKubernetesclustersthathaveuppercaselettersintheirnames.
Symptom
Clustersthatyoucreatewith pkscreate-cluster
donotappearintheTanzuMissionControl,eventhoughyouconfiguredTanzuMission
ControlintegrationasdescribedinIntegrate Tanzu Mission Control.
Explanation
TheregexpatternthatparsesclusternamesinTanzuMissionControlintegrationfailswithnamesthatcontainuppercaseletters.
Solution
Whenrunning pkscreate-cluster
tocreateclustersthatyouwanttotrackinTanzuMissionControl,passinnamesthatcontainonly
lowercaselettersandnumbers.
EnterprisePKSCreatesaLinuxClusterWhenYouExpectaWindowsCluster
Symptom
WhenyoucreateanEnterprisePKSclusterusingeitherPlan11,12or13theclusteriscreatedasaLinuxclusterinsteadofaWindowscluster.
Explanation
WhenyoucreateanEnterprisePKSclusterusingeitherPlan11,12or13aWindowsclustershouldbecreated.IfyouareusingEnterprisePKSv1.6withOperationsManagerv2.7aLinuxclusteriscreatedinstead.
SavingUAATabSettingsFailsWithError:‘InvalidURIErrorbadURI’
Symptom
WhenyousaveyourUAAtabwithLDAP ServerselectedandmultipleLDAPserversspecified,youreceivetheerror:URI::InvalidURIErrorbadURI(isnotURI?):LDAPURLs
.
Explanation
WhenyouconfiguretheUAAtabwithmultipleLDAPserversyoursettingswillfailtovalidatewhenusingthefollowingOpsManagerreleases:
Ops Manager Version Affected ReleasesOps Manager v2.6 OpsManagerv2.6.18andearlierpatchreleases.
Ops Manager v2.7 Allpatchreleases.
Ops Manager v2.8 Allpatchreleases.
Note:ThisissueisresolvedinEnterprisePKSv1.6.1.
Copyright©2020VMware,Inc.AllRightsReserved. 18 1.6
-
Workaround
Toresolvethisissueseethefollowing:
Ops Manager Version Workaround
Ops Manager v2.6
Performoneofthefollowing:UpgradetoOpsManagerv2.6.19orlaterv2.6patchrelease.
Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.
Ops Manager v2.7Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.
Ops Manager v2.8Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.
WindowsWorkerClustersFailtoUpgradetov1.6
Symptoms
DuringyourupgradefromEnterprisePKSv1.5toEnterprisePKSv1.6aWindowsworkerVMfailstoupgrade,asevidencedby:
Thecommandlineoutputsanerror Failed jobs: docker-windows .
TheWindowsworkerVMdisappearsfromtheoutputof kubectl get nodes .
Thecommandlineshowsthestatus failed andtheaction UPGRADE fortheclusterthatcontainstheworker.
Thelogshowsanentry \docker\dockerd.exe: Access is denied .
Explanation
BetweenPKSv1.5andv1.6,thenameoftheDockerservicechangedfrom docker to docker-windows ,butyourenvironmentcontinuestousetheoldDockerservicenameandpaths.TheincompatibleservicenameandpathingcausesaWindowsworkerupgradetofail.
IfyourclusterhasmultipleWindowsworkers,thisissuedoesnotincurdowntime.BeforeBOSHattemptstoupgradeaWindowsworker,itmovestheworker’sappstootherWindowsworkersinthecluster.Whentheupgradefails,BOSHstopstheclusterupgradeprocessandtheotherWindowsworkerscontinuerunningattheearlierversion.
Workaround
AfterupgradingtoEnterprisePKSv1.6andyourWindowsworkerclustershavefailedtoupgrade,completethefollowingsteps:
1. UploadavSpherestemcellv2019.8orlaterforWindowsServerversion2019toyourEnterprisePKStile.
2. ToupgradeyourWindowsworkerclusters,performoneofthefollowing:
EnabletheUpgrade all clusters errandsettinganddeploythePKStile.FormoreinformationaboutconfiguringtheUpgrade all clusters errandanddeployingtheEnterprisePKStile,seeModify Errand Configuration in the EnterprisePKS TileinUpgradingClusters.
Copyright©2020VMware,Inc.AllRightsReserved. 19 1.6
https://kb.vmware.com/s/article/76495https://kb.vmware.com/s/article/76495https://kb.vmware.com/s/article/76495
-
Run pks upgrade-cluster or pks upgrade-clusters onyourfailedWindowsworkercluster(s).FormoreinformationaboutupgradingspecificEnterprisePKSclusters,seeUpgrade ClustersinUpgradingClusters.
502BadGatewayAfterOIDCLogin
Symptom
Youexperiencea“502BadGateway”errorfromtheNSXloadbalancerafteryoulogintoOIDC.
Explanation
AlargeresponseheaderhasexceededyourNSX-Tloadbalancermaximumresponseheadersize.Thedefaultmaximumresponseheadersizeis10,240charactersandshouldberesizedto50,000.
Workaround
Ifyouexperiencethisissue,manuallyreconfigureyourNSX-T request_header_size and response_header_size to50,000characters.ForinformationaboutconfiguringNSX-Tdefaultheadersizes,seeOIDC Response Header Overflow inthePivotalKnowledgeBase.
NSX-TPre-CheckErrandFailsDuetoEdgeNodeConfiguration
Symptom
YouhaveconfiguredyourNSX-TEdgeNodeVMas medium size,andtheNSX-TPre-CheckErrandfailswiththefollowingerror:“ERROR:NSX-TPrecheckfailedduetoEdgeNode…noofcpucoresislessthan8”.
Explanation
TheNSX-TPre-CheckErrandiserroneouslyreturningthe“cpucoresislessthan8”error.
Solution
YoucansafelyconfigureyourNSX-TEdgeNodeVMsas medium sizeandignoretheerror.
CharacterLimitationsinHTTPProxyPassword
ForvSpherewithNSX-T,theHTTPProxypasswordfielddoesnotsupportthefollowingspecialcharacters: & or ; .
EnterprisePKSManagementConsole1.6.2
Release Date:April29,2020
Features
OtherthansupportforEnterprisePKSv1.6.2,EnterprisePKSManagementConsole1.6.2hasnonewfeatures.
BugFixes
EnterprisePKSManagementConsole1.6.2includesnobugfixes.
Copyright©2020VMware,Inc.AllRightsReserved. 20 1.6
https://community.pivotal.io/s/article/OIDC-Response-Header-overflow
-
ProductSnapshot
Element Details
Version v1.6.2
Releasedate April29,2020
InstalledEnterprisePKSversion v1.6.2
InstalledOpsManagerversion v2.8.5
InstalledKubernetesversion v1.15.10
CompatibleNSX-Tversions v2.5.0,v2.4.3
InstalledHarborRegistryversion v1.9.4
KnownIssues
TheEnterprisePKSManagementConsolev1.6.2applianceanduserinterfacehavethesameknown issuesasv1.6.1.
EnterprisePKSManagementConsole1.6.1
Release Date:January23,2020
Features
OtherthansupportforEnterprisePKSv1.6.1,EnterprisePKSManagementConsole1.6.1hasnonewfeatures.
BugFixes
EnterprisePKSManagementConsole1.6.1includesnobugfixes.
ProductSnapshot
Element Details
Version v1.6.1
Releasedate January23,2020
InstalledEnterprisePKSversion v1.6.1
InstalledOpsManagerversion v2.8.0
InstalledKubernetesversion v1.15.5
CompatibleNSX-Tversions v2.5.0,v2.4.3
Note:EnterprisePKSManagementConsoleprovidesanopinionatedinstallationofEnterprisePKS.ThesupportedversionsmaydifferfromorbemorelimitedthanwhatisgenerallysupportedbyEnterprisePKS.
Note:EnterprisePKSManagementConsoleprovidesanopinionatedinstallationofEnterprisePKS.ThesupportedversionsmaydifferfromorbemorelimitedthanwhatisgenerallysupportedbyEnterprisePKS.
Copyright©2020VMware,Inc.AllRightsReserved. 21 1.6
-
InstalledHarborRegistryversion v1.9.3
KnownIssues
TheEnterprisePKSManagementConsolev1.6.1applianceanduserinterfacehavethesameknown issuesasv1.6.0-rev.3andv1.6.0-rev.2.
EnterprisePKSManagementConsole1.6.0-rev.3
Release Date:December19,2019
Features
EnterprisePKSManagementConsole1.6.0-rev.3hasnonewfeatures.
BugFixes
EnterprisePKSManagementConsole1.6.0-rev.3includesthefollowingbugfixes:
FixesUIfailurecausedbymultipledatacentersbeingpresentinvCenterServer.
AddssupportforbothFQDNandIPaddressesinLDAP/LDAPSconfigurationforidentitymanagement.
FixesUIfreezingafterenteringunconventionallyformattedURLsforSAMLprovidermetadata.
AddssupportforUAArole pks.clusters.admin.read inidentityManagementconfiguration.
AddsvalidationforHarborFQDNinlowercase.
FixesmisconfiguredWavefrontHTTPProxywhenfieldisleftempty.
ProductSnapshot
Element Details
Version v1.6.0-rev.3
Releasedate December19,2019
InstalledEnterprisePKSversion v1.6.0
InstalledOpsManagerversion v2.7.3
InstalledKubernetesversion v1.15.5
CompatibleNSX-Tversions v2.5.0,v2.4.3
InstalledHarborRegistryversion v1.9.3
important:TheEnterprisePKSManagementConsole1.6.0-rev.3offlinepatchcanonlybeappliedinanair-gappedenvironment.Itcanonlybeappliedto1.6.0-rev.2andnottoanyotherversion.Forinformationabouthowtoapplythepatch,seePatch Enterprise PKS Management Console Components.
Note:EnterprisePKSManagementConsoleprovidesanopinionatedinstallationofEnterprisePKS.ThesupportedversionsmaydifferfromorbemorelimitedthanwhatisgenerallysupportedbyEnterprisePKS.
Copyright©2020VMware,Inc.AllRightsReserved. 22 1.6
-
KnownIssues
WiththeexceptionoftheBug Fixeslistedabove,theEnterprisePKSManagementConsolev1.6.0-rev.3applianceanduserinterfacehavethesameknown issuesasv1.6.0-rev.2.
EnterprisePKSManagementConsolev1.6.0-rev.2
Release Date:November26,2019
Features
EnterprisePKSManagementConsolev1.6.0-rev.2updatesinclude:
ProvidesexperimentalintegrationwithVMwareTanzuMissionControl.Formoreinformation,seeTanzu Mission ControlIntegration.
ProvidesexperimentalsupportforplansthatuseWindowsworkernodes.Forinformation,seeConfigure Plans.
DeploysHarborregistryv1.9.Forinformation,seeConfigure Harbor.
Addssupportforactive-activemodeonthetier0routerinautomated-NATdeploymentsandNo-NATconfigurationsinBringYourOwnTopologydeployments.Forinformation,seeConfigure Networking.
AddstheabilitytoconfigureproxiesfortheintegrationwithWavefront.Forinformation,seeConfigure a Connection toWavefront.
AddstheabilitytoconfigurethesizeofthePKSAPIVM.Forinformation,seeConfigure Resources and Storage.
Allowsyoutousethemanagementconsoletoupgradetov1.6.0-rev.2.Forinformation,seeUpgrade Enterprise PKSManagement Console.
ProductSnapshot
Element Details
Version v1.6.0-rev.2
Releasedate November26,2019
InstalledEnterprisePKSversion v1.6.0
InstalledOpsManagerversion v2.7.3
InstalledKubernetesversion v1.15.5
CompatibleNSX-Tversions v2.5.0,v2.4.3
InstalledHarborRegistryversion v1.9.3
KnownIssues
ThefollowingknownissuesarespecifictotheEnterprisePKSManagementConsolev1.6.0-rev.2applianceanduserinterface.
YAMLValidationErrorsNotCleared
Note:EnterprisePKSManagementConsoleprovidesanopinionatedinstallationofEnterprisePKS.ThesupportedversionsmaydifferfromorbemorelimitedthanwhatisgenerallysupportedbyEnterprisePKS.
Copyright©2020VMware,Inc.AllRightsReserved. 23 1.6
-
Symptom
IfyouattempttouploadaYAMLconfigurationfileandthedeploymentfailsbecauseofaninvalidmanifest,EnterprisePKSManagementConsoledisplaysanerrornotificationwiththevalidationerror.Ifsubsequentattemptsalsofailbecauseofvalidationissues,thevalidationerrorsareappendedtoeachother.
Explanation
ThevalidationerrorsarenotclearedwhenyouresubmittheYAMLconfigurationfile.
Workaround
None
EnterprisePKSManagementConsoleNotificationsPersist
Symptom
IntheEnterprise PKSviewofEnterprisePKSManagementConsole,errornotificationssometimespersistinmemoryontheClustersandNodespagesafteryouclearthosenotifications.
Explanation
AfterclickingtheXbuttontoclearanotificationitisremoved,butwhenyounavigatebacktothosepagesthenotificationmightshowagain.
Workaround
Useshift+refreshtoreloadthepage.
CannotDeleteEnterprisePKSDeploymentfromManagementConsole
Symptom
IntheEnterprise PKSviewofEnterprisePKSManagementConsole,youcannotusetheDelete Enterprise PKS Deploymentoptionevenafteryouhaveremovedallclusters.
Explanation
Theoptiontodeletethedeploymentisonlyactivatedinthemanagementconsoleashortperiodaftertheclustersaredeleted.
Workaround
Afterremovingclusters,waitforafewminutesbeforeattemptingtousetheDelete Enterprise PKS Deploymentoptionagain.
ConfiguringEnterprisePKSManagementConsoleIntegrationwithVMwarevRealizeLogInsight
Symptom
EnterprisePKSManagementConsoleappliancesendslogstoVMwarevRealizeLogInsightoverHTTP,notHTTPS.
Explanation
Copyright©2020VMware,Inc.AllRightsReserved. 24 1.6
-
WhenyoudeploytheEnterprisePKSManagementConsoleappliancefromtheOVA,ifyourequirelogforwardingtovRealizeLogInsight,youmustprovidetheportonthevRealizeLogInsightserveronwhichitlistensforHTTPtraffic.DonotprovidetheHTTPSport.
Workaround
SetthevRealizeLogInsightporttotheHTTPport.Thisistypicallyport 9000 .
DeployingEnterprisePKStoanUnpreparedNSX-TDataCenterEnvironmentResultsinFlannelError
Symptom
WhenusingthemanagementconsoletodeployEnterprisePKSinNSX-T Data Center (Not prepared for PKS)mode,ifanerroroccursduringthenetworkconfiguration,themessage Unabletosetflannel
environmentisdisplayedinthedeploymentprogresspage.
Explanation
Thenetworkconfigurationhasfailed,buttheerrormessageisincorrect.
Workaround
Toseethecorrectreasonforthefailure,seetheserverlogs.Forinstructionsabouthowtoobtaintheserverlogs,seeTroubleshootingEnterprise PKS Management Console.
UsingBOSHCLIfromOperationsManagerVM
Symptom
TheBOSHCLIclientbashcommandthatyouobtainfromtheDeployment MetadataviewdoesnotworkwhenloggedintotheOperationsManagerVM.
Explanation
TheBOSHCLIclientbashcommandfromtheDeployment MetadataviewisintendedtobeusedfromwithintheEnterprisePKSManagementConsoleappliance.
Workaround
TousetheBOSHCLIfromwithintheOperationsManagerVM,seeConnect to Operations Manager.
FromtheOpsManagerVM,usetheBOSHCLIclientbashcommandfromtheDeployment Metadatapage,withthefollowingmodifications:
Removetheclause BOSH_ALL_PROXY=xxx
Replacethe BOSH_CA_CERT sectionwith BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
Run pks CommandsagainstthePKSAPIServer
Explanation
ThePKSCLIisavailableintheEnterprisePKSManagementConsoleappliance.
Copyright©2020VMware,Inc.AllRightsReserved. 25 1.6
-
Workaround
Tobeabletorun pks commandsagainstthePKSAPIServer,youmustfirstlogtoPKSusingthefollowingcommandsyntaxpkslogin-afqdn_of_pks...
.
Todothis,youmustensureeitherofthefollowing:
TheFQDNconfiguredforthePKSServerisresolvablebytheDNSserverconfiguredfortheEnterprisePKSManagementConsoleappliance,or
AnentrythatmapstheFloatingIPassignedtothePKSServertotheFQDNexistson/etc/hostsintheappliance.Forexample:192.168.160.102 api.pks.local .
Copyright©2020VMware,Inc.AllRightsReserved. 26 1.6
mailto:[email protected]
-
EnterprisePKSConceptsPage last updated:
ThistopicdescribesVMwareEnterprisePKSconcepts.Seethefollowingsections:
Enterprise PKS Cluster Management
PKS API Authentication
Load Balancers in Enterprise PKS
VM Sizing for Enterprise PKS Clusters
Telemetry
Sink Architecture in Enterprise PKS
Copyright©2020VMware,Inc.AllRightsReserved. 27 1.6
mailto:[email protected]
-
EnterprisePKSClusterManagement
In this topic
Overview
ClusterLifecycleManagementPKSControlPlaneOverview
PKSControlPlaneArchitecture
ClusterWorkloadManagement
Page last updated:
ThistopicdescribeshowVMwareEnterprisePKSmanagesthedeploymentofKubernetesclusters.
Overview
UsersinteractwithEnterprisePKSandEnterprisePKS-deployedKubernetesclustersintwoways:
DeployingKubernetesclusterswithBOSHandmanagingtheirlifecycle.ThesetasksareperformedusingthePKSCommandLineInterface(PKSCLI)andthePKScontrolplane.
Deployingandmanagingcontainer-basedworkloadsonKubernetesclusters.ThesetasksareperformedusingtheKubernetesCLI,kubectl .
ClusterLifecycleManagement
ThePKScontrolplaneenablesuserstodeployandmanageKubernetesclusters.
ForcommunicatingwiththePKScontrolplane,EnterprisePKSprovidesacommandlineinterface,thePKSCLI.SeeInstalling thePKS CLIforinstallationinstructions.
PKSControlPlaneOverview
ThePKScontrolplanemanagesthelifecycleofKubernetesclustersdeployedusingEnterprisePKS.ThecontrolplaneallowsuserstodothefollowingthroughthePKSCLI:
Viewclusterplans
Createclusters
Viewinformationaboutclusters
Obtaincredentialstodeployworkloadstoclusters
Scaleclusters
Deleteclusters
CreateandmanagenetworkprofilesforVMwareNSX-T
Inaddition,thePKScontrolplanecanupgradeallexistingclustersusingtheUpgrade all clustersBOSHerrand.Formoreinformation,seeUpgrade Kubernetes ClustersinUpgradingEnterprisePKS.
Copyright©2020VMware,Inc.AllRightsReserved. 28 1.6
-
PKSControlPlaneArchitecture
ThePKScontrolplaneisdeployedonasingleVMthatincludesthefollowingcomponents:
ThePKSAPIserver
ThePKSBroker
AUserAccountandAuthentication(UAA)server
Thefollowingillustrationshowshowthesecomponentsinteract:
ThePKSAPILoadBalancerisusedforAWS,GCP,andvSpherewithoutNSX-Tdeployments.IfEnterprisePKSisdeployedonvSpherewithNSX-T,aDNATruleisconfiguredforthePKSAPIhostsothatitisaccessible.Formoreinformation,seetheShare the PKS APIEndpointsectioninInstallingEnterprisePKSonvSpherewithNSX-TIntegration.
UAA
WhenauserlogsintoorlogsoutofthePKSAPIthroughthePKSCLI,thePKSCLIcommunicateswithUAAtoauthenticatethem.ThePKSAPIpermitsonlyauthenticateduserstomanageKubernetesclusters.Formoreinformationaboutauthenticating,seePKS APIAuthentication.
Copyright©2020VMware,Inc.AllRightsReserved. 29 1.6
-
UAAmustbeconfiguredwiththeappropriateusersanduserpermissions.Formoreinformation,seeManaging Enterprise PKSUsers with UAA.
PKSAPI
ThroughthePKSCLI,usersinstructthePKSAPIservertodeploy,scaleup,anddeleteKubernetesclustersaswellasshowclusterdetailsandplans.ThePKSAPIcanalsowriteKubernetesclustercredentialstoalocalkubeconfigfile,whichenablesuserstoconnecttoaclusterthrough kubectl .
ThePKSAPIsendsallclustermanagementrequests,exceptread-onlyrequests,tothePKSBroker.
PKSBroker
WhenthePKSAPIreceivesarequesttomodifyaKubernetescluster,itinstructsthePKSBrokertomaketherequestedchange.
ThePKSBrokerconsistsofanOn-Demand Service Broker andaServiceAdapter.ThePKSBrokergeneratesaBOSHmanifestandinstructstheBOSHDirectortodeployordeletetheKubernetescluster.
ForEnterprisePKSdeploymentsonvSpherewithNSX-T,thereisanadditionalcomponent,theEnterprisePKSNSX-TProxyBroker.ThePKSAPIcommunicateswiththePKSNSX-TProxyBroker,whichinturncommunicateswiththeNSXManagertoprovisiontheNodeNetworkingresources.ThePKSNSX-TProxyBrokerthenforwardstherequesttotheOn-DemandServiceBrokertodeploythecluster.
ClusterWorkloadManagement
EnterprisePKSusersmanagetheircontainer-basedworkloadsonKubernetesclustersthrough kubectl .Formoreinformationaboutkubectl ,seeOverview of kubectl intheKubernetesdocumentation.
Copyright©2020VMware,Inc.AllRightsReserved. 30 1.6
https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://kubernetes.io/docs/reference/kubectl/overview/mailto:[email protected]
-
PKSAPIAuthentication
In this topic
AuthenticationofPKSAPIRequests
RoutingtothePKSAPIControlPlaneVM
Page last updated:
ThistopicdescribeshowtheVMwareEnterprisePKSAPIworkswithUserAccountandAuthentication(UAA)tomanageauthenticationandauthorizationinyourEnterprisePKSdeployment.
AuthenticationofPKSAPIRequests
BeforeuserscanloginandusethePKSCLI,youmustconfigurePKSAPIaccesswithUAA.Formoreinformation,seeManagingEnterprise PKS Users with UAAandLogging in to Enterprise PKS.
YouusetheUAACommandLineInterface(UAAC)totargettheUAAserverandrequestanaccesstokenfortheUAAadminuser.Ifyourrequestissuccessful,theUAAserverreturnstheaccesstoken.TheUAAadminaccesstokenauthorizesyoutomakerequeststothePKSAPIusingthePKSCLIandgrantclusteraccesstoneworexistingusers.
WhenauserwithclusteraccesslogsintothePKSCLI,theCLIrequestsanaccesstokenfortheuserfromtheUAAserver.Iftherequestissuccessful,theUAAserverreturnsanaccesstokentothePKSCLI.WhentheuserrunsPKSCLIcommands,forexample, pksclusters ,theCLIsendstherequesttothePKSAPIserverandincludestheuser’sUAAtoken.
ThePKSAPIsendsarequesttotheUAAservertovalidatetheuser’stoken.IftheUAAserverconfirmsthatthetokenisvalid,thePKSAPIusestheclusterinformationfromthePKSbrokertorespondtotherequest.Forexample,iftheuserruns pksclusters ,theCLIreturnsalistoftheclustersthattheuserisauthorizedtomanage.
RoutingtothePKSAPIControlPlaneVM
ThePKSAPIserverandtheUAAserverusedifferentportnumbersonthecontrolplaneVM.Forexample,ifyourPKSAPIdomainisapi.pks.example.com ,youcanreachyourPKSAPIandUAAserversatthefollowingURLs:
Server URL
PKSAPI api.pks.example.com:9021
UAA api.pks.example.com:8443
RefertoOps Manager>Enterprise PKS tile>PKS API>API Hostname (FQDN)foryourPKSAPIdomain.
Loadbalancerimplementationsdifferbydeploymentenvironment.ForEnterprisePKSdeploymentsonGCP,AWS,orvSpherewithoutNSX-T,youconfigurealoadbalancertoaccessthePKSAPIwhenyouinstalltheEnterprisePKStile.Forexample,seeConfiguring PKSAPI Load Balancer.
ForoverviewinformationaboutloadbalancersinEnterprisePKS,seeLoad Balancers in Enterprise PKS Deployments withoutNSX-T.
Copyright©2020VMware,Inc.AllRightsReserved. 31 1.6
mailto:[email protected]
-
Copyright©2020VMware,Inc.AllRightsReserved. 32 1.6
-
UAAScopesforEnterprisePKSUsers
In this topic
Overview
UAAScopes
Page last updated:
ThistopicdescribesUserAccountandAuthentication(UAA)scopesthataUAAadmincanassigntoVMwareEnterprisePKSusers.
Overview
UAAistheidentitymanagementserviceforEnterprisePKS.
ByassigningUAAscopes,yougrantuserstheabilitytocreate,manage,andauditKubernetesclustersinEnterprisePKS.
AUAAadminusercanassignthefollowingUAAscopestoEnterprisePKSusers:
pks.clusters.manage :Accountswiththisscopecancreateandaccesstheirownclusters.
pks.clusters.admin :Accountswiththisscopecancreateandaccessallclusters.
pks.clusters.admin.read :Accountswiththisscopecanaccessanyinformationaboutallclustersexceptforclustercredentials.
Youcanassignthesescopestoindividualusers,externalidentityprovidergroups,orclientsforautomationpurposes.
UAAScopes
EachUAAscopegrantsEnterprisePKSusersasetofpermissionsforcreating,managing,andauditingEnterprisePKS-provisionedKubernetesclusters.Forinformationaboutthepermissions,seethetablebelow.
Operation pks.clusters.manage pks.clusters.admin pks.clusters.admin.read
Create, update,resize, and delete acluster
Yes.Userswiththisscopecancreate,modify,anddeleteonlytheirownclusters.
Yes.Userswiththisscopecancreate,modify,anddeleteallclusters.
No.Userswiththisscopecannotcreate,modify,anddeleteclusters.
Get clustercredentials
Yes.Userswiththisscopecanretrieveclustercredentialsonlyfortheirownclusters.
Yes.Userswiththisscopecanretrieveclustercredentialsforallclusters.
No.Userswiththisscopecannotretrieveclustercredentials.
Upgrade clustersYes.Userswiththisscopecanupgradeonlytheirownclusters.
Yes.Userswiththisscopecanupgradeallclusters.
No.Userswiththisscopecannotupgradeclusters.
List clusters Yes.Userswiththisscopecanlistonlytheirownclusters.
Yes.Userswiththisscopecanlistallclusters.
Yes.Userswiththisscopecanlistallclusters.
View cluster detailsYes.Userswiththisscopecanviewclusterdetailsonlyfortheirownclusters.
Yes.Userswiththisscopecanviewclusterdetailsforallclusters.
Yes.Userswiththisscopecanviewclusterdetailsforallclusters.
Copyright©2020VMware,Inc.AllRightsReserved. 33 1.6
-
Create and delete acompute profile
No.Userswiththisscopecannotcreateanddeletecomputeprofiles.
Yes.Userswiththisscopecancreateanddeletecomputeprofiles.
No.Userswiththisscopecannotcreateanddeletecomputeprofiles.
Create and delete anetwork profile
No.Userswiththisscopecannotcreateanddeletenetworkprofiles.
Yes.Userswiththisscopecancreateanddeletenetworkprofiles.
No.Userswiththisscopecannotcreateanddeletenetworkprofiles.
Create, update, anddelete a quota
No.Userswiththisscopecannotcreate,update,anddeletequotas.
Yes.Userswiththisscopecancreate,update,anddeletequotas.
No.Userswiththisscopecannotcreate,update,anddeletequotas.
List Enterprise PKSplans
Yes.Userswiththisscopecanlistallavailableplans.
Yes.Userswiththisscopecanlistallavailableplans.
Yes.Userswiththisscopecanlistallavailableplans.
ToassignUAAscopesinEnterprisePKS,followtheinstructionsinManaging Enterprise PKS Users with UAA.
Copyright©2020VMware,Inc.AllRightsReserved. 34 1.6
mailto:[email protected]
-
LoadBalancersinEnterprisePKS
In this topic
LoadBalancersinEnterprisePKSDeploymentswithoutNSX-TAboutthePKSAPILoadBalancer
AboutKubernetesClusterLoadBalancers
AboutWorkloadLoadBalancers
LoadBalancersinEnterprisePKSDeploymentsonvSpherewithNSX-TResizingLoadBalancers
Page last updated:
ThistopicdescribesthetypesofloadbalancersthatareusedinVMwareEnterprisePKSdeployments.Loadbalancersdifferbythetypeofdeployment.
LoadBalancersinEnterprisePKSDeploymentswithoutNSX-TForEnterprisePKSdeploymentsonGCP,AWS,orvSpherewithoutNSX-T,youcanconfigureloadbalancersforthefollowing:
PKS API:ConfiguringthisloadbalancerenablesyoutorunPKSCommandLineInterface(PKSCLI)commandsfromyourlocalworkstation.
Kubernetes Clusters:ConfiguringaloadbalancerforeachnewclusterenablesyoutorunKubernetesCLI(kubectl)commandsonthecluster.
Workloads:Configuringaloadbalancerforyourapplicationworkloadsenablesexternalaccesstotheservicesthatrunonyourcluster.
Thefollowingdiagram,applicabletoGCP,AWS,andvSpherewithoutNSX-T,showswhereeachoftheaboveloadbalancerscanbeusedwithinyourEnterprisePKSdeployment.
Copyright©2020VMware,Inc.AllRightsReserved. 35 1.6
-
IfyouuseeithervSpherewithoutNSX-TorGCP,youareexpectedtocreateyourownloadbalancerswithinyourcloudproviderconsole.Ifyourcloudproviderdoesnotofferloadbalancing,youcanuseanyexternalTCPorHTTPSloadbalancerofyourchoice.
AboutthePKSAPILoadBalancer
ThePKSAPIloadbalancerenablesyoutoaccessthePKSAPIfromoutsidethenetworkonEnterprisePKSdeploymentsonGCP,AWS,andonvSpherewithoutNSX-T.Forexample,configuringaloadbalancerforthePKSAPIenablesyoutorunPKSCLIcommandsfromyourlocalworkstation.
ForinformationaboutconfiguringthePKSAPIloadbalanceronvSpherewithoutNSX-T,seeConfiguring PKS API Load Balancer.
AboutKubernetesClusterLoadBalancers
WhenyoucreateanEnterprisePKSclusteronGCP,AWS,andonvSpherewithoutNSX-T,youmustconfigureexternalaccesstothe
Copyright©2020VMware,Inc.AllRightsReserved. 36 1.6
-
clusterbycreatinganexternalTCPorHTTPSloadbalancer.TheloadbalancerenablestheKubernetesCLItocommunicatewiththecluster.
Ifyoucreateaclusterinanon-productionenvironment,youcanchoosenottousealoadbalancer.Toenablekubectltoaccesstheclusterwithoutaloadbalancer,youcandooneofthefollowing:
CreateaDNSentrythatpointstothecluster’smasterVM.Forexample:
my-cluster.example.com A 10.0.0.5
Ontheworkstationwhereyourunkubectlcommands,addthemasterIPaddressofyourclusterand kubo.internal tothe/etc/hosts file.Forexample:
10.0.0.5 kubo.internal
Formoreinformationaboutconfiguringaclusterloadbalancer,seethefollowing:
Creating and Configuring a GCP Load Balancer for Enterprise PKS Clusters
Creating and Configuring an AWS Load Balancer for Enterprise PKS Clusters
Creating and Configuring an Azure Load Balancer for Enterprise PKS Clusters
AboutWorkloadLoadBalancers
ToenableexternalaccesstoyourEnterprisePKSapponGCP,AWS,andonvSpherewithoutNSX-T,youcaneithercreatealoadbalancerorexposeastaticportonyourworkload.
Forinformationaboutconfiguringaloadbalancerforyourappworkload,seeDeploying and Exposing Basic Linux Workloads.
IfyouuseAWS,youmustconfigureroutingintheAWSconsolebeforeyoucancreatealoadbalancerforyourworkload.Youmustcreateapublicsubnetineachavailabilityzone(AZ)whereyouaredeployingtheworkloadandtagthepublicsubnetwithyourcluster’suniqueidentifier.
SeetheAWS PrerequisitessectionofDeployingandExposingBasicLinuxWorkloadsbeforeyoucreateaworkloadloadbalancer.
DeployYourWorkloadLoadBalancerwithanIngressController
AKubernetesingresscontrollersitsbehindaloadbalancer,routingHTTPandHTTPSrequestsfromoutsidetheclustertoserviceswithinthecluster.Kubernetesingressresourcescanbeconfiguredtoloadbalancetraffic,provideexternallyreachableURLstoservices,andmanageotheraspectsofnetworktraffic.
IfyouaddaningresscontrollertoyourEnterprisePKSdeployment,trafficroutingiscontrolledbytheingressresourcerulesyoudefine.PivotalrecommendsconfiguringEnterprisePKSdeploymentswithbothaworkloadloadbalancerandaningresscontroller.
ThefollowingdiagramshowshowtheingressroutingcanbeusedwithinyourEnterprisePKSdeployment.
Copyright©2020VMware,Inc.AllRightsReserved. 37 1.6
-
TheloadbalanceronEnterprisePKSonvSpherewithNSX-TisautomaticallyprovisionedwithKubernetesingressresourceswithouttheneedtodeployandconfigureanadditionalingresscontroller.
ForinformationaboutdeployingaloadbalancerconfiguredwithingressroutingonGCP,AWS,Azure,andvSpherewithoutNSX-T,seeConfiguring Ingress Routing.ForinformationaboutingressroutingonvSpherewithNSX-T,seeConfiguring Ingress Resourcesand Load Balancer Services.
LoadBalancersinEnterprisePKSDeploymentsonvSpherewithNSX-TEnterprisePKSdeploymentsonvSpherewithNSX-TdonotrequirealoadbalancerconfiguredtoaccessthePKSAPI.TheyrequireonlyaDNATruleconfiguredsothatthePKSAPIhostisaccessible.Formoreinformation,seeShare the Enterprise PKS EndpointinInstallingEnterprisePKSonvSpherewithNSX-TIntegration.
NSX-Thandlesloadbalancercreation,configuration,anddeletionautomaticallyaspartoftheKubernetesclustercreate,update,anddeleteprocess.WhenanewKubernetesclusteriscreated,NSX-Tcreatesandconfiguresadedicatedloadbalancertiedtoit.Theloadbalancerisasharedresourcedesignedtoprovideefficienttrafficdistributiontomasternodesaswellasservicesdeployedonworkernodes.Eachapplicationserviceismappedtoavirtualserverinstance,carvedoutfromthesameloadbalancer.Formoreinformation,seeLogical Load Balancer intheNSX-Tdocumentation.
Virtualserverinstancesarecreatedontheloadbalancertoprovideaccesstothefollowing:
Kubernetes API and UI services on a Kubernetes cluster.Thisenablesrequeststobeloadbalancedacrossmultiplemasternodes.
Ingress controller.ThisenablesthevirtualserverinstancetodispatchHTTPandHTTPSrequeststoservicesassociatedwithIngressrules.
type:loadbalancer services.ThisenablestheservertohandleTCPconnectionsorUDPflowstowardexposedservices.
Loadbalancersaredeployedinhigh-availabilitymodesothattheyareresilienttopotentialfailuresandabletorecoverquicklyfromcriticalconditions.
Note:The NodePort ServicetypeisnotsupportedforEnterprisePKSdeploymentsonvSpherewithNSX-T.Only
Copyright©2020VMware,Inc.AllRightsReserved. 38 1.6
https://docs.vmware.com/en/VMware-NSX-T/2.1/com.vmware.nsxt.admin.doc/GUID-46567C8D-A5C5-4793-8CDF-858E58FDE3C4.html
-
ResizingLoadBalancers
WhenanewKubernetesclusterisprovisionedusingthePKSAPI,NSX-Tcreatesadedicatedloadbalancerforthatnewcluster.Bydefault,thesizeoftheloadbalancerissettoSmall.
Withnetworkprofiles,youcanchangethesizeoftheloadbalancerdeployedbyNSX-Tatthetimeofclustercreation.Forinformationaboutnetworkprofiles,seeUsing Network Profiles (NSX-T Only).
FormoreinformationaboutthetypesofloadbalancersNSX-Tprovisionsandtheircapacities,seeScaling Load BalancerResources intheNSX-Tdocumentation.
type:LoadBalancer ServicesandServicesassociatedwithIngressrulesaresupportedonvSpherewithNSX-T.
Copyright©2020VMware,Inc.AllRightsReserved. 39 1.6
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.3/com.vmware.nsxt.admin.doc/GUID-19B12230-8BF4-4AF7-9EB7-3701B0A0A439.htmlmailto:[email protected]
-
VMSizingforEnterprisePKSClusters
In this topic
Overview
MasterNodeVMSize
WorkerNodeVMNumberandSizeExampleWorkerNodeRequirementCalculation
CustomizeMasterandWorkerNodeVMSizeandType
Page last updated:
ThistopicdescribeshowVMwareEnterprisePKSrecommendsyouapproachthesizingofVMsforclustercomponents.
Overview
WhenyouconfigureplansintheEnterprisePKStile,youprovideVMsizesforthemasterandworkernodeVMs.Formoreinformationaboutconfiguringplans,seethePlanssectionofInstallingEnterprisePKSforyourIaaS:
vSphere
vSphere with NSX-T Integration
Google Cloud Platform (GCP)
Amazon Web Services (AWS)
Azure
Youselectthenumberofmasternodeswhenyouconfiguretheplan.
ForworkernodeVMs,youselectthenumberandsizebasedontheneedsofyourworkload.ThesizingofmasterandworkernodeVMsishighlydependentonthecharacteristicsoftheworkload.Adapttherecommendationsinthistopicbasedonyourownworkloadrequirements.
MasterNodeVMSize
ThemasternodeVMsizeislinkedtothenumberofworkernodes.TheVMsizingshowninthefollowingtableispermasternode:
TocustomizethesizeoftheKubernetesmasternodeVM,seeCustomize Master and Worker Node VM Size and Type.
Number of Workers CPU RAM (GB)
1-5 1 3.75
6-10 2 7.5
11-100 4 15
101-250 8 30
Note:Iftherearemultiplemasternodes,allmasternodeVMsarethesamesize.Toconfigurethenumberofmasternodes,seethePlanssectionofInstallingEnterprisePKSforyourIaaS.
Copyright©2020VMware,Inc.AllRightsReserved. 40 1.6
-
251-500 16 60
500+ 32 120
Number of Workers CPU RAM (GB)
DonotoverloadyourmasternodeVMsbyexceedingtherecommendedmaximumnumberofworkernodeVMsorbydownsizingfromtherecommendedVMsizingslistedabove.TheserecommendationssupportbothatypicalworkloadmanagedbyaVMandthehigherthanusualworkloadmanagedbytheVMwhileotherVM’sintheclusterareupgrading.
WorkerNodeVMNumberandSize
Amaximumof100podscanrunonasingleworkernode.TheactualnumberofpodsthateachworkernoderunsdependsontheworkloadtypeaswellastheCPUandmemoryrequirementsoftheworkload.
TocalculatethenumberandsizeofworkerVMsyourequire,determinethefollowingforyourworkload:
Maximumnumberofpodsyouexpecttorun[ p ]
Memoryrequirementsperpod[ m ]
CPUrequirementsperpod[ c ]
Usingthevaluesabove,youcancalculatethefollowing:
Minimumnumberofworkers[ W ]= p / 100
MinimumRAMperworker= m * 100
MinimumnumberofCPUsperworker= c * 100
Thiscalculationgivesyoutheminimumnumberofworkernodesyourworkloadrequires.Werecommendthatyouincreasethisvaluetoaccountforfailuresandupgrades.
Forexample,increasethenumberofworkernodesbyatleastonetomaintainworkloaduptimeduringanupgrade.Additionally,increasethenumberofworkernodestofityourownfailuretolerancecriteria.
ThemaximumnumberofworkernodesthatyoucancreateforaplaninanEnterprisePKS-provisionedKubernetesclusterissetbytheMaximum number of workers on a clusterfieldinthePlanspaneoftheEnterprisePKStile.TocustomizethesizeoftheKubernetesworkernodeVM,seeCustomize Master and Worker Node VM Size and Type.
ExampleWorkerNodeRequirementCalculation
Anexampleapphasthefollowingminimumrequirements:
Numberofpods[ p ]=1000
RAMperpod[ m ]=1GB
CPUperpod[ c ]=0.10
TodeterminehowmanyworkernodeVMstheapprequires,dothefollowing:
1. Calculatethenumberofworkersusing p / 100 :
warning:UpgradinganoverloadedKubernetesclustermasternodeVMcanresultindowntime.
Copyright©2020VMware,Inc.AllRightsReserved. 41 1.6
-
1000/100 = 10 workers
2. CalculatetheminimumRAMperworkerusing m * 100 :
1 * 100 = 100 GB
3. CalculatetheminimumnumberofCPUsperworkerusing c * 100 :
0.10 * 100 = 10 CPUs
4. Forupgrades,increasethenumberofworkersbyone:
10 workers + 1 worker = 11 workers
5. Forfailuretolerance,increasethenumberofworkersbytwo:
11 workers + 2 workers = 13 workers
Intotal,thisappworkloadrequires13workerswith10CPUsand100GBRAM.
CustomizeMasterandWorkerNodeVMSizeandType
YouselecttheCPU,memory,anddiskspacefortheKubernetesnodeVMsfromasetlistintheEnterprisePKStile.MasterandworkernodeVMsizesandtypesareselectedonaper-planbasis.Formoreinformation,seethePlanssectionoftheEnterprisePKSinstallationtopicforyourIaaS.Forexample,Installing Enterprise PKS on vSphere with NSX-T.
WhilethelistofavailablenodeVMtypesandsizesisextensive,thelistmaynotprovidetheexacttypeandsizeofVMthatyouwant.YoucanusetheOpsManagerAPItocustomizethesizeandtypesofthemasterandworkernodeVMs.Formoreinformation,seeHowto Create or Remove Custom VM_TYPE Template using the Operations Manager API intheKnowledgeBase.
warning:DonotreducethesizeofyourKubernetesmasternodeVMsbelowtherecommendedsizeslistedinMaster NodeVM Size,above.UpgradinganoverloadedKubernetesclustermasternodeVMcanresultindowntime.
Copyright©2020VMware,Inc.AllRightsReserved. 42 1.6
https://community.pivotal.io/s/article/how-to-create-or-remove-custom-vmtype-template-using-the-ops-manager-apimailto:[email protected]
-
Telemetry
In this topic
OverviewParticipationLevels
ConfigureCEIPandTelemetry
BenefitsoftheEnhancedParticipationLevel
SystemComponents
DataDictionary
SampleReports
Page last updated:
ThistopicdescribestheVMwareCustomerExperienceImprovementProgram(CEIP)andthePivotalTelemetryProgram(Telemetry)usedintheEnterprisePKStile.
Overview
TheCEIPandTelemetryprogramallowVMwareandPivotaltocollectdatafromcustomerinstallationstoimproveyourEnterprisePKSexperience.CollectingdataatscaleenablesustoidentifypatternsandalertyoutowarningsignalsinyourEnterprisePKSinstallation.
ParticipationLevels
YoucanconfigureEnterprisePKStouseoneofthefollowingCEIPandTelemetryparticipationlevels:
None:Thisleveldisablesdatacollection.
Standard:(Default)Thislevelcollectsdataanonymously.YourdataisusedtoinformtheongoingdevelopmentofEnterprisePKS.
Enhanced:ThislevelenablesVMwareandPivotaltowarnyouaboutsecurityvulnerabilitiesandpotentialissueswithyoursoftwareconfigurations.Formoreinformation,seeBenefits of the Enhanced Participation Levelbelow.
ConfigureCEIPandTelemetry
Video:ForinformationaboutconfiguringCEIPandTelemetryparticipation,seetheCEIP Opt-In Walkthrough video onYouTube.
ForinformationaboutconfiguringCEIPandTelemetryparticipation,seetheCEIPandTelemetry sectionoftheinstallationtopicforyourIaaS:
Installing Enterprise PKS on vSphere
Installing Enterprise PKS on vSphere with NSX-T
Installing Enterprise PKS on AWS
Installing Enterprise PKS on Azure
Note:EnterprisePKSdoesnotcollectanypersonallyidentifiableinformation(PII)ateitherparticipationlevel.ForalistofthedataEnterprisePKScollects,seeData Dictionary.
Copyright©2020VMware,Inc.AllRightsReserved. 43 1.6
https://www.youtube.com/embed/RTyq9ODUbU4
-
Installing Enterprise PKS on GCP
BenefitsoftheEnhancedParticipationLevel
BenefitsyoureceivewiththeEnhancedparticipationlevelincludebutarenotlimitedtothefollowing:
Usage data:ThisgivesyouaccesstodataaboutKubernetespodandclusterusageinyourEnterprisePKSinstallation.Seesample reportsbelowformoredetails.
Access to your telemetry data:ThisgivesyouaccesstoconfigurationandusagedataaboutyourEnterprisePKSinstallation.Seesample reportsbelowformoredetails.
Proactive support:ThisenablesVMwareandPivotaltoproactivelywarnyouaboutunhealthypatterns.
Benchmarks:ThisisyourusagerelativetotherestoftheEnterprisePKSuserbase.
ThetablebelowcomparestheStandardandEnhancedparticipationlevels.
Benefit Standard Level Enhanced Level
Usagedata Rawdata Reportsandtrendanalysis
Accesstoyourtelemetrydata No Yes
Proactivesupport No Yes
Benchmarks No Yes
SystemComponents
TheCEIPandTelemetryprogramsusethefollowingcomponentstocollectdata:
Telemetry Server:ThiscomponentrunsonthePKScontrolplane.TheserverreceivestelemetryeventsfromthePKSAPIandmetricsfromTelemetryagentpods.Theserversendseventsandmetricstoadatalakeforarchivingandanalysis.
Telemetry Agent Pod:ThiscomponentrunsineachKubernetesclusterasadeploymentwithonereplica.AgentpodsperiodicallypolltheKubernetesAPIforclustermetricsandsendthemetricstotheTelemetryserver.
Thefollowingdiagramshowshowtelemetrydataflowsthroughthesystemcomponents:
Note:VMwarereservestherighttochangethebenefitsassociatedwiththeEnhancedparticipationlevelatanytime.
Copyright©2020VMware,Inc.AllRightsReserved. 44 1.6
-
DataDictionary
ForinformationaboutPKSTelemetrycollectionandreporting,seethePKS Telemetry Data spreadsheet,hostedonGoogleDrive.
SampleReports
Video:SeetheSample Report: Create Cluster Duration videoonYouTube.
YoucanviewtheinteractiveversionoftheSample Workbook withTableau Reader (freetouse).Clickonthelinksbelowtoseestaticscreenshotsofthereports.
1. Consumption :AsanOperatorofPKS,IneedawaytomonitorpodconsumptionacrossmyPKSenvironmentsovertime,soIcan:
SeewhichenvironmentsandclustersgettheheaviestuseSeetemporalpatternsinpodconsumptionScalecapacityaccordinglyShowandchargebackusersofPKSwithinmyorganization
2. API heartbeats + Cluster heartbeats :AsanOperatorofPKSIneedawaytoseetheversionofPKSeachofmyenvironmentswasrunningovertime,soIcan:
KeeptrackofallmyPKSenvironmentsandclustersIdentifyenvironmentsandclustersinneedofupgrading
3. Cluster creation events :AsanOperatorofPKSIwanttoseehowoftenclustercreationsucceedsacrossmyPKSenvironments,soIcan:
Identifyenvironmentsthatencounterrepeatedfailuresanddebugorinterveneasappropriatetoavoidfrustrationforclusteradminsandusers
Copyright©2020VMware,Inc.AllRightsReserved. 45 1.6
file:///Users/pspinrad/workspace/pdfer/html/docs-pcf-staging.cfapps.io/pks/1-6/images/telemetry-data-flow.pnghttps://drive.google.com/open?id=18UCd1kbhR3xV_XOl6KcEU64GI6ySdkRa3iG_8QAROl8#gid=1858241440https://www.youtube.com/embed/Q41g7uWBvhAhttp://bit.ly/sampleworkbookhttps://www.tableau.com/products/readerhttp://bit.ly/consumptionreporthttp://bit.ly/apiheartbeatshttp://bit.ly/clusterheartbeatshttp://bit.ly/clustercreate
-
4. Cluster creation duration :AsanOperatorofPKSIwanttoseehowlongittakestocreateclusters,soIcan:
Intervenewhenclustercreationsignificantlymoretimethanexpected,andadjustmyplanandnetworkconfigurationasappropriate
5. Cluster creation errors :AsanOperatorofPKS,IwanttoseewhaterrorsarebeingencounteredmostfrequentlyduringclustercreationsoIcan:
Quicklyidentifywidespreadproblemsandremediate(e.g.NSXerrors)
6. Container images :AsanOperatorofPKS,IwanttoseewhichcontainerimagesareinuseacrossmyPKSinstallationssoIcan:
ConductanauditofcontainerimagesandidentifyprohibitedorproblematicimagesInferwhichworkloadsarerunningonPKS,toinformmyplanning,resourcing,andoutreach
Copyright©2020VMware,Inc.AllRightsReserved. 46 1.6
http://bit.ly/createdurationhttp://bit.ly/createerrorshttp://bit.ly/containerimagesmailto:[email protected]
-
InstallingEnterprisePKSPage last updated:
EnterprisePKSManagementConsole(vSphereOnly)
SeethefollowingdocumentationfortheEnterprisePKSManagementConsole,whichdeploysEnterprisePKSasavirtualapplianceonvSpherewithoutPivotalPlatform:
Enterprise PKS Management Console (vSphere Only)
VMwareEnterprisePKSonPivotalPlatform
SeethefollowingdocumentationforhowtoinstallVMwareEnterprisePKSonPivotalPlatform:
vSphere with Flannel
vSphere with NSX-T
GCP
AWS
Azure
Note:EnterprisePKSsupportsair-gappeddeploymentsonvSpherewithorwithoutNSX-Tintegration.
Copyright©2020VMware,Inc.AllRightsReserved. 47 1.6
mailto:[email protected]
-
vSpherePage last updated:
ThistopicliststheprocedurestofollowtoinstallVMwareEnterprisePKSonvSphere.
InstallEnterprisePKSonvSphere
ToinstallEnterprisePKSonvSpherewithoutNSX-T,followtheinstructionsbelow:
Prerequisites and Resource Requirements
Firewall Ports and Protocols Requirements for vSphere without NSX-T
Creating Dedicated Users and Roles for vSphere (Optional)
Installing and Configuring Ops Manager on vSphere
Installing Enterprise PKS on vSphere
Configuring PKS API Load Balancer
Setting Up Enterprise PKS Admin Users on vSphere
(Optional) Integrating VMware Harbor with Enterprise PKS
InstallthePKSandKubernetesCLIs
ThePKSCLIandKubernetesCLIhelpyouinteractwithyourEnterprisePKS-provisionedKubernetesclustersandKubernetesworkloads.
ToinstalltheCLIs,followtheinstructionsbelow:
Installing the PKS CLI
Installing the Kubernetes CLI
Note:VMwareHarborisanenterprise-classregistryserverforcontainerimages.Formoreinformation,seeVMware HarborRegistry inthePivotalPartnerdocumentation.
Copyright©2020VMware,Inc.AllRightsReserved. 48 1.6
https://docs.pivotal.io/partners/vmware-harbor/integrating-pks.htmlhttps://docs.pivotal.io/partners/vmware-harbor/index.htmlmailto:[email protected]
-
vSpherePrerequisitesandResourceRequirements
In this topic
Prerequisites
vSphereVersionRequirements
ResourceRequirements
NetworkCommunicationRequirements
Page last updated:
ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingVMwareEnterprisePKSonvSphere.
ForprerequisitesandresourcerequirementsforinstallingEnterprisePKSonvSpherewithNSX-Tintegration,seevSphere with NSX-T Version RequirementsandHardware Requirements for Enterprise PKS on vSphere with NSX-T.
Prerequisites
BeforeinstallingEnterprisePKS:
1. ReviewthesectionsbelowandtheinstructionsinCreating Dedicated Users and Roles for vSphere (Optional).
2. InstallandconfigureOpsManager.ToinstallOpsManager,followtheinstructionsinInstalling and Configuring Ops Manageron vSphere.
vSphereVersionRequirements
ForEnterprisePKSonvSphereversionrequirements,refertotheVMware Product Interoperability Matrices .
ResourceRequirements
InstallingOpsManagerandEnterprisePKSrequiresthefollowingvirtualmachines(VMs):
VM CPU RAM Storage
PivotalContainerService 2 8GB 16GB
PivotalOpsManager 1 8GB 160GB
BOSHDirector 2 8GB 16GB
StorageRequirementsforLargeNumbersofPods
Ifyouexpecttheclusterworkloadtorunalargenumberofpodscontinuously,thenincreasethesizeofpersistentdiskstorageallocatedtothePivotalContainerServiceVMasfollows:
Number of Pods Storage (Persistent Disk) Requirements
1,000pods 20GB
Copyright©2020VMware,Inc.AllRightsReserved. 49 1.6
https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=
-
5,000pods 100GB
10,000pods 200GB
50,000pods 1,000GB
Number of Pods Storage (Persistent Disk) Requirements
EphemeralVMResources
EachEnterprisePKSdeploymentrequiresephemeralVMsduringinstallationandupgradesofEnterprisePKS.AfteryoudeployEnterprisePKS,BOSHautomaticallydeletestheseVMs.ToenableEnterprisePKStodynamicallycreatetheephemeralVMswhenneeded,ensurethatthefollowingresourcesareavailableinyourvSphereinfrastructurebeforedeployingEnterprisePKS:
Ephemeral VM Number CPU Cores RAM Ephemeral Disk
BOSHCompilationVMs 4 4 4GB 32GB
KubernetesClusterResources
EachKubernetesclusterprovisionedthroughEnterprisePKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.
VM Number CPU Cores RAM Ephemeral Disk Persistent Disk
master 1or3 2 4GB 8GB 5GB
worker 1ormore 2 4GB 8GB 50GB
errand(ephemeral) 1 1 1GB 8GB none
NetworkCommunicationRequirements
ForacompletelistofnetworkcommunicationrequirementsforvSpherewithoutNSX-T,seeFirewall Ports and ProtocolsRequirements for vSphere without NSX-T.
Copyright©2020VMware,Inc.AllRightsReserved. 50 1.6
mailto:[email protected]
-
FirewallPortsandProtocolsRequirementsforvSpherewithoutNSX-TPage last updated:
In this topic
EnterprisePKSPortsandProtocols
EnterprisePKSUsersPortsandProtocols
EnterprisePKSCorePortsandProtocols
VMwarePortsandProtocols
VMwareVirtualInfrastructurePortsandProtocols
VMwareOptionalIntegrationPortsandProtocols
ThistopicdescribesthefirewallportsandprotocolsrequirementsforusingVMwareEnterprisePKSonvSphere.
Firewallsandsecuritypoliciesareusedtofiltertrafficandlimitaccessinenvironmentswithstrictinter-networkaccesscontrolpolicies.
Appsfrequentlyrequiretheabilitytopassinternalcommunicationbetweensystemcomponentsondifferentnetworksandrequireoneormoreconduitsthroughtheenvironment’sfirewalls.FirewallrulesarealsorequiredtoenableinterfacingwithexternalsystemssuchaswithenterpriseappsorappsanddataonthepublicInternet.
ForEnterprisePKS,Pivotalrecommendsthatyoudisablesecuritypoliciesthatfiltertrafficbetweenthenetworkssupportingthesystem.WithEnterprisePKSyoushouldenableaccesstoappsthroughstandardKubernetesload-balancersandingresscontrollertypes.Thisenablesyoutodesignatespecificportsandprotocolsasafirewallconduit.
ForinformationonportsandprotocolrequirementsforvSpherewithNSX-T,seeFirewall Ports and Protocols Requirements forvSphere with NSX-T
Ifyouareunabletoimplementyoursecuritypolicyusingthemethodsdescribedabove,refertothefollowingtable,whichidentifiestheflowsbetweensystemcomponentsinatypicalEnterprisePKSdeployment.
EnterprisePKSPortsandProtocols
ThefollowingtableslistportsandprotocolsrequiredfornetworkcommunicationsbetweenEnterprisePKSv1.5.0andlater,andvSphere6.7andlater.
EnterprisePKSUsersPortsandProtocols
ThefollowingtablelistsportsandprotocolsusedfornetworkcommunicationbetweenEnterprisePKSuserinterfacecomponents.
Source Component Destination Component DestinationProtocolDestinationPort Service
Admin/OperatorConsole AllSystemComponents TCP 22 ssh
Note:Tocontrolwhichgroupsaccessdeployingandscalingyourorganization’sEnterprisePKS-deployedKubernetesclusters,configureyourfirewallsettingsasdescribedontheOperator–>PKSAPIserverlinesbelow.
Copyright©2020VMware,Inc.AllRightsReserved. 51 1.6
-
Admin/OperatorConsole AllSystemComponents TCP 80 http
Admin/OperatorConsole AllSystemComponents TCP 443 https
Admin/OperatorConsole CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi
Admin/OperatorConsolePivotalCloudFoundryOperationsManager
TCP 22 ssh
Admin/OperatorConsolePivotalCloudFoundryOperationsManager
TCP 443 https
Admin/OperatorConsole PKSController TCP 9021 pksapiserver
Admin/OperatorConsole vCenterServer TCP 443 https
Admin/OperatorConsole vCenterServer TCP 5480 vami
Admin/OperatorConsolevSphereESXIHostsMgmt.vmknic
TCP 902 ideafarm-door
Admin/OperatorandDeveloperConsoles
HarborPrivateImageRegistry TCP 80 http
Admin/OperatorandDeveloperConsoles
HarborPrivateImageRegistry TCP 443 https
Admin/OperatorandDeveloperConsoles
HarborPrivateImageRegistry TCP 4443 notary
Admin/OperatorandDeveloperConsoles
KubernetesAppLoad-BalancerSvc
TCP/UDP Varies varieswithapps
Admin/OperatorandDeveloperConsoles
KubernetesClusterAPIServer-LBVIP
TCP 8443 httpsca
Admin/OperatorandDeveloperConsoles
KubernetesClusterIngressController
TCP 80 http
Admin/OperatorandDeveloperConsoles
KubernetesClusterIngressController
TCP 443 https
Admin/OperatorandDeveloperConsoles
KubernetesClusterWorkerNode TCP/UDP 30000-32767kubernetesnodeport
Admin/OperatorandDeveloperConsoles
PKSController TCP 8443 httpsca
AllUserConsoles(Operator,Developer,Consumer)
KubernetesAppLoad-BalancerSvc
TCP/UDP Varies varieswithapps
AllUserConsoles(Operator,Developer,Consumer)
KubernetesClusterIngressController
TCP 80 http
AllUserConsoles(Operator,Developer,Consumer)
KubernetesClusterIngressController
TCP 443 https
AllUserConsoles(Operator,Developer,Consumer)
KubernetesClusterWorkerNode TCP/UDP 30000-32767kubernetesnodeport
Source Component Destination Component DestinationProtocolDestinationPort Service
EnterprisePKSCorePortsandProtocols
Copyright©2020VMware,Inc.AllRightsReserved. 52 1.6
-
ThefollowingtablelistsportsandprotocolsusedfornetworkcommunicationbetweencoreEnterprisePKScomponents.
Source Component Destination Component DestinationProtocolDestinationPort Service
AllSystemComponentsCorporateDomainNameServer
TCP/UDP 53 dns
AllSystemComponents NetworkTimeServer UDP 123 ntp
AllSystemComponents vRealizeLogInsight TCP/UDP 514/1514 syslog/tlssyslog
AllSystemControlPlaneComponents
AD/LDAPDirectoryServer TCP/UDP 389/636 ldap/ldaps
PivotalCloudFoundryOperationsManager
Admin/OperatorConsole TCP 22 ssh
PivotalCloudFoundryOperationsManager
CloudFoundryBOSHDirector TCP 6868 boshagenthttp
PivotalCloudFoundryOperationsManager
CloudFoundryBOSHDirector TCP 8443 httpsca
PivotalCloudFoundryOperationsManager
CloudFoundryBOSHDirector TCP 8844 credhub
PivotalCloudFoundryOperationsManager
CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi
PivotalCloudFoundryOperationsManager
HarborPrivateImageRegistry TCP 22 ssh
PivotalCloudFoundryOperationsManager
KubernetesClusterMaster/EtcdNode
TCP 22 ssh
PivotalCloudFoundryOperationsManager
KubernetesClusterWorkerNode
TCP 22 ssh
PivotalCloudFoundryOperationsManager
PKSController TCP 22 ssh
PivotalCloudFoundryOperationsManager
PKSController TCP 8443 httpsca
PivotalCloudFoundryOperationsManager
vCenterServer TCP 443 https
PivotalCloudFoundryOperationsManager
vSphereESXIHostsMgmt.vmknic
TCP 443 https
CloudFoundryBOSHDirector vCenterServer TCP 443 https
CloudFoundryBOSHDirectorvSphereESXIHostsMgmt.vmknic
TCP 443 https
BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 4222 boshnatsserver
BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 25250 boshblobstore
BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 25923healthmonitordaemon
BOSHCompilationJobVM HarborPrivateImageRegistry TCP 443 https
BOSHCompilationJobVM HarborPrivateImageRegistry TCP 8853 boshdnshealth
Copyright©2020VMware,Inc.AllRightsReserved. 53 1.6
-
PKSController CloudFoundryBOSHDirector TCP 4222 boshnatsserver
PKSController CloudFoundryBOSHDirector TCP 8443 httpsca
PKSController CloudFoundryBOSHDirector TCP 25250 boshblobstore
PKSController CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi
PKSController CloudFoundryBOSHDirector TCP 25923healthmonitordaemon
PKSControllerKubernetesClusterMaster/EtcdNode TCP 8443 httpsca
PKSController vCenterServer TCP 443 https
HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 4222 boshnatsserver
HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 25250 boshblobstore
HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 25923healthmonitordaemon
HarborPrivateImageRegistry IPNASStorageArray TCP 111 nfsrpcportmapper
HarborPrivateImageRegistry IPNASStorageArray TCP 2049 nfs
HarborPrivateImageRegistry PublicCVESourceDatabase TCP 443 https
kube-systempod/telemetry-agent PKSController TCP 24224fluentdout_forward
KubernetesClusterMaster/EtcdNode
CloudFoundryBOSHDirector TCP 4222 boshnatsserver
KubernetesClusterMaster/EtcdNode
CloudFoundryBOSHDirector TCP 25250 boshblobstore
KubernetesClusterMaster/EtcdNode
CloudFoundryBOSHDirector TCP 25923healthmonitordaemon
KubernetesClusterMaster/EtcdNode
KubernetesClusterMaster/EtcdNode
TCP 2379 etcdclent
KubernetesClusterMaster/EtcdNode
KubernetesClusterMaster/EtcdNode
TCP 2380 etcdserver
KubernetesClusterMaster/EtcdNode
KubernetesClusterMaster/EtcdNode
TCP 8443 httpsca
KubernetesClusterMaster/EtcdNode
KubernetesClusterMaster/EtcdNode
TCP 8853 boshdnshealth