pivotal container service (pks) v1 · 1. deploying ops manager on vsphere 2. configuring bosh...

PivotalContainerService(PKS)

Documentation

v1.6

Published:November14,2019

Copyright©2020VMware,Inc.AllRightsReserved.

Note:ThecontentsofthisPDFmayhavefallenoutofdate.Forcurrentdocumentation,seehttps://docs.pivotal.io/pks/1-6

https://docs.pivotal.io/pks/1-6

VMwareEnterprisePKS

In this topic

Overview

WhatEnterprisePKSAddstoKubernetes

Features

EnterprisePKSComponents

EnterprisePKSConcepts

EnterprisePKSPrerequisites

PreparingtoInstallEnterprisePKS

InstallingEnterprisePKS

UpgradingEnterprisePKS

ManagingEnterprisePKS

ManagingKubernetesClustersandWorkloads

BackingUpandRestoringEnterprisePKS

EnterprisePKSSecurity

DiagnosingandTroubleshootingEnterprisePKS

Page last updated:

VMwareEnterprisePKSenablesoperatorstoprovision,operate,andmanageenterprise-gradeKubernetesclustersusingBOSHandPivotalOpsManager.

Overview

EnterprisePKSusestheOn-Demand Broker todeployCloud Foundry Container Runtime ,aBOSHreleasethatoffersauniformwaytoinstantiate,deploy,andmanagehighlyavailableKubernetesclustersonacloudplatformusingBOSH.

AfteroperatorsinstalltheEnterprisePKStileontheOpsManagerInstallationDashboard,developerscanprovisionKubernetesclustersusingthePKSCommandLineInterface(PKSCLI),andruncontainer-basedworkloadsontheclusterswiththeKubernetesCLI,kubectl.

OnPivotal Platform ,youcanrunEnterprisePKSstandaloneoralongsidePivotalApplicationService.

WhatEnterprisePKSAddstoKubernetes

ThefollowingtabledetailsthefeaturesthatEnterprisePKSaddstotheKubernetesplatform.

Feature Included in K8s Included in Enterprise PKS

Singletenantingress ✓ ✓

Securemulti-tenantingress ✓

Statefulsetsofpods ✓ ✓

Multi-containerpods ✓ ✓

Rollingupgradestopods ✓ ✓

Copyright©2020VMware,Inc.AllRightsReserved. 2 1.6

https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://docs-cfcr.cfapps.io/https://docs.pivotal.io/platform

Rollingupgradestoclusterinfrastructure ✓

Podscalingandhighavailability ✓ ✓

Clusterprovisioningandscaling ✓

MonitoringandrecoveryofclusterVMsandprocesses ✓

Persistentdisks ✓ ✓

Securecontainerregistry ✓

Embedded,hardenedoperatingsystem ✓

Features

EnterprisePKShasthefollowingfeatures:

Kubernetes compatibility:ConstantcompatibilitywithcurrentstablereleaseofKubernetes

Production-ready:Highlyavailablefromapplicationstoinfrastructure,withnosinglepointsoffailure

BOSH advantages:Built-inhealthchecks,scaling,auto-healingandrollingupgrades

Fully automated operations:Fullyautomateddeploy,scale,patch,andupgradeexperience

Multi-cloud:Consistentoperationalexperienceacrossmultipleclouds

GCP APIs access:TheGoogleCloudPlatform(GCP)ServiceBrokergivesapplicationsaccesstotheGoogleCloudAPIs,andGoogleContainerEngine(GKE)consistencyenablesthetransferofworkloadsfromortoGCP

OnvSphere,EnterprisePKSsupportsdeployingandrunningKubernetesclustersinair-gappedenvironments.

FeatureSupportbyIaaS

AWS Azure GCP vSphere with Flannel vSphere with NSX-TAutomatic Kubernetes Cluster API loadbalancer ✓

HTTP proxy ✓ ✓ ✓

Multi-AZ storage ✓ ✓

Per-namespace subnets ✓Service type:LoadBalancer ✓ ✓ ✓ ✓

Windows worker-based cluster ✓

FormoreinformationaboutconfiguringService type:LoadBalancer onAWS,seetheAccess Workloads Using an Internal AWS LoadBalancersectionofDeployingandExposingBasicLinuxWorkloads.

EnterprisePKSComponents

ThePKScontrolplanecontainsthefollowingcomponents:

AnOn-Demand Broker thatdeploysCloud Foundry Container Runtime (CFCR),anopen-sourceprojectthatprovidesasolutionfordeployingandmanagingKubernetes clustersusingBOSH .

AServiceAdapter

*

*


https://docs.pivotal.io/svc-sdk/odb/https://docs-cfcr.cfapps.io/https://kubernetes.io/docs/home/https://bosh.io/docs

ThePKSAPI

FormoreinformationaboutthePKScontrolplane,seeEnterprise PKS Cluster Management.

ForadetailedlistofcomponentsandsupportedversionsbyaparticularEnterprisePKSrelease,seetheEnterprise PKS ReleaseNotes.

EnterprisePKSConcepts

ForconceptualinformationaboutEnterprisePKS,seeEnterprise PKS Concepts.

EnterprisePKSPrerequisites

ForinformationabouttheresourcerequirementsforinstallingEnterprisePKS,seethetopicthatcorrespondstoyourcloudprovider:

vSphere Prerequisites and Resource Requirements

vSphere with NSX-T Version RequirementsandHardware Requirements for Enterprise PKS on vSphere with NSX-T

GCP Prerequisites and Resource Requirements

AWS Prerequisites and Resource Requirements

Azure Prerequisites and Resource Requirements

PreparingtoInstallEnterprisePKS

ToinstallEnterprisePKS,youmustdeployOpsManager.YouuseOpsManagertoinstallandconfigureEnterprisePKS.

IfyouareinstallingEnterprisePKStovSphere,youcanalsoconfigureintegrationwithNSX-TandHarbor.

Consultthefollowingtableforcompatibilityinformation:

IaaS Ops Manager v2.6.16+ or v2.7.6+ NSX-T Harbor

vSphere Required Available Available

GCP Required NotAvailable Available

AWS Required NotAvailable Available

Azure Required NotAvailable Available

Formoreinformationaboutcompatibilityandcomponentversions,seetheEnterprise PKS Release Notes.

ForinformationaboutpreparingyourenvironmentbeforeinstallingEnterprisePKS,seethetopicthatcorrespondstoyourcloudprovider:

vSphere

vSphere with NSX-T Integration

GCP

AWS

Azure


InstallingEnterprisePKS

ForinformationaboutinstallingEnterprisePKS,seeInstallingEnterprisePKSforyourIaaS:

vSphere


Google Cloud Platform (GCP)

Amazon Web Services (AWS)

Microsoft Azure (Azure)

UpgradingEnterprisePKS

ForinformationaboutupgradingtheEnterprisePKStileandEnterprisePKS-deployedKubernetesclusters,seeUpgrading EnterprisePKS.

ManagingEnterprisePKS

Forinformationaboutconfiguringauthentication,creatingusers,andmanagingyourEnterprisePKSdeployment,seeManagingEnterprise PKS.

ManagingKubernetesClustersandWorkloads

ForinformationaboutmanagingEnterprisePKS-provisionedKubernetesclustersanddeployingworkloads,seeManagingKubernetes Clusters and Workloads.

BackingUpandRestoringEnterprisePKS

ForinformationaboutusingBOSHBackupandRestore(BBR)tobackupandrestoreEnterprisePKS,seeBacking Up and RestoringEnterprise PKS.

EnterprisePKSSecurity

ForinformationaboutsecurityinEnterprisePKS,seeEnterprise PKS Security.

DiagnosingandTroubleshootingEnterprisePKS

ForinformationaboutdiagnosingandtroubleshootingissuesinstallingorusingEnterprisePKS,seeDiagnosing andTroubleshooting Enterprise PKS.

[email protected].


mailto:[email protected]

ReleaseNotes

In this topic

v1.6.2Features

ProductSnapshot

vSphereVersionRequirements

UpgradePath

BreakingChanges

KnownIssues

v1.6.1Features

ProductSnapshot


UpgradePath

BreakingChanges

KnownIssues

v1.6.0Features

ProductSnapshot


UpgradePath

BreakingChanges

KnownIssues

EnterprisePKSManagementConsole1.6.2Features

BugFixes

ProductSnapshot

KnownIssues

EnterprisePKSManagementConsole1.6.1Features

BugFixes

ProductSnapshot

KnownIssues

EnterprisePKSManagementConsole1.6.0-rev.3Features

BugFixes

ProductSnapshot

KnownIssues

EnterprisePKSManagementConsolev1.6.0-rev.2Features

ProductSnapshot

KnownIssues

Page last updated:


ThistopiccontainsreleasenotesforVMwareEnterprisePKSv1.6.

v1.6.2

Release Date:April29,2020

Features

Newfeaturesandchangesinthisrelease:

BumpsKubernetestov1.15.10.

BumpsUAAtov73.4.20.

BumpsPerconaXtraDBCluster(PXC)tov0.22.

BumpsWindowsStemcelltov2019.15.

BumpsODBtov0.38.0.

BumpsApacheTomcat(inPKSAPI)tov9.0.31.

[Security Fix]UAAbumpfixesblindSCIMinjectionvulnerability,CVE-2019-11282.

[Security Fix]UAAbumpfixesCSRFattackvulnerability.

[Security Fix]PXCbumpfixescURL/libcURLbufferoverflowvulnerability,CVE-2019-3822.

[Bug Fix]Improvesthebehaviorofthe pks get-kubeconfig and pks get-credentials commandsduringclusterupdatesandupgrades.Youcannowrunthe pks get-kubeconfig commandduringsingle-andmulti-masterclusterupdates.Additionally,youcanrunthe pks get-credentials commandduringmulti-masterclusterupgrades.

[Bug Fix]NewUAAversionincludesApacheTomcatbumpthatfixesSAMLloginissues.

ProductSnapshot

Element Details

Version v1.6.2

Releasedate April29,2020

CompatibleOpsManagerversions SeePivotal Network

Xenialstemcellversion SeePivotal Network

Windowsstemcellversion v2019.15

Kubernetesversion v1.15.10

On-DemandBrokerversion v0.38.0

CompatibleNSX-Tversions v2.5.1,v2.5.0,v2.4.3

NCPversion v2.5.1

Dockerversion v18.09.9

BackupandRestoreSDKversion v1.17.0

warning:BeforeinstallingorupgradingtoEnterprisePKSv1.6,reviewtheBreaking Changesbelow.


https://network.pivotal.io/products/pivotal-container-service#/releases/631561https://network.pivotal.io/products/pivotal-container-service#/releases/631561

UAAversion v73.4.20


ForEnterprisePKSinstallationsonvSphereoronvSpherewithNSX-TDataCenter,refertotheVMware Product InteroperabilityMatrices .

UpgradePath

ThesupportedupgradepathstoEnterprisePKSv1.6.2arefromEnterprisePKSv1.5.0andlater.

BreakingChanges

AllbreakingchangesinEnterprisePKSv1.6.2arealsoinEnterprisePKSv1.6.0.SeeBreaking Changes in Enterprise PKS v1.6.0.

KnownIssues

AllknownissuesinEnterprisePKSv1.6.2arealsoinEnterprisePKSv1.6.0.SeeKnown Issues in Enterprise PKS v1.6.0.

v1.6.1

Release Date:Jan13,2020

Features

Newfeaturesandchangesinthisrelease:

[Security Fix]SecurestrafficintoKubernetesclusterswithup-to-dateTLS(v1.2+)andapprovedciphersuites.

[Security Fix]BumpsUAAtov73.4.16.ThisupdatepreventsloggingofsecureinformationandenablesthePKSUAAtostartwiththe env.no_proxy propertyset.

[Bug Fix]ResolvesanissuewhereifyouareusingOpsManagerv2.7andPKSv1.6asafreshinstall,enablingPlans11,12,or13doesnotenableWindowsworker-basedclusters.ItcreatesLinux-basedclustersonly.Formoreinformation,seeEnterprise PKSCreates a Linux Cluster When You Expect a Windows Cluster.

[Bug Fix]ResolvesanissuewhereapplyingchangestoEnterprisePKSfailsifPlan8isenabledintheEnterprisePKStile.Formoreinformation,seeApplying Changes Fails If Plan 8 Is Enabled.

[Bug Fix]Resolvesanissuewherethe pks update-cluster --network-profile commandsets subnet_prefix to0inthencp.inifileifthenetworkprofiledoesnothave pod_subnet_prefix .Formoreinformation,seeNetwork Profile for “pks update-cluster” Does Not Use the Defaults from the Original Cluster Manifest.

[Bug Fix]ResolvesanissuewheretryingtocreateaclusterwithalongnetworkprofilecausesanerrorData too long for column 'nsxt_network_profile' .

UpdatesthesupportedNCPversiontoNCPv2.5.1.RefertotheNCP Release Notes formoreinformation.

SupportforNSX-Tv2.5.1.

ProductSnapshot


https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/rn/NSX-Container-Plugin-251-Release-Notes.html

Element Details

Version v1.6.1

Releasedate January13,2020






CompatibleNSX-Tversions v2.5.1,v2.5.0,v2.4.3

NCPversion v2.5.1



UAA v73.4.16



UpgradePath


BreakingChanges

AllbreakingchangesinEnterprisePKSv1.6.1arealsoinEnterprisePKSv1.6.0.SeeBreaking Changes in Enterprise PKS v1.6.0.

KnownIssues

AllknownissuesinEnterprisePKSv1.6.1arealsoinEnterprisePKSv1.6.0.SeeKnown Issues in Enterprise PKS v1.6.0.

v1.6.0

Release Date:November14,2019

Features

Thissectiondescribesnewfeaturesandchangesinthisrelease.

PKSControlPlaneandAPI

EnterprisePKSv1.6.0updatesinclude:

EnablesoperatorstoupgrademultipleKubernetesclusterssimultaneouslyandtodesignatespecificupgradeclustersascanary


https://network.pivotal.io/products/pivotal-container-service#/releases/551663https://network.pivotal.io/products/pivotal-container-service#/releases/551663https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=

clusters.Formoreinformationaboutmultipleclusterupgrades,seeUpgrade ClustersinUpgradingClusters.

AddsanewUAAscope, pks.clusters.admin.read ,forEnterprisePKSusers.ForinformationaboutUAAscopes,seeUAAScopes for Enterprise PKS UsersandManaging Enterprise PKS Users with UAA.

ProvidesexperimentalintegrationwithTanzuMissionControl.Formoreinformation,seeTanzu Mission Control Integration.

EnablesoperatorstolimitthetotalnumberofclustersausercanprovisioninEnterprisePKS.Formoreinformationaboutquotas,seeManaging Resource Usage with QuotasandViewing Usage Quotas.

EnablesoperatorstoconfigureasingleKubernetesclusterwithaspecificDockerRegistryCAcertificate.FormoreinformationaboutconfiguringaclusterwithaDockerRegistryCAcertificate,seeConfiguring Enterprise PKS Clusters with Private DockerRegistry CA Certificates (Beta).

Updatesthe pks delete-cluster PKSCLIcommandsothatallclusterobjects,includingNSX-Tnetworkingobjects,aredeletedwithouttheneedtousethe bosh delete deployment commandtoremovefailedclusterdeletions.

KubernetesControlPlane


IncreasestheWorker VM Max in Flightdefaultvaluefrom 1 to 4 inthePKS APIconfigurationpane,whichacceleratesclustercreationbyallowinguptofournewnodestobeprovisionedsimultaneously.TheupdateddefaultvalueisonlyappliedduringnewEnterprisePKSinstallationandisnotappliedduringanEnterprisePKSupgrade.IfyouareupgradingEnterprisePKSfromapreviousversionandwanttoacceleratemulti-clusterprovisioning,youcanincreasethevalueofWorker VM Max inFlightmanually.

PKSMonitoringandLogging


RedesignstheLoggingandMonitoringpanesoftheEnterprisePKStileandrenamesthemtoHost MonitoringandIn-ClusterMonitoring.Forinformationaboutconfiguringthesepanes,seetheInstallingEnterprisePKStopicforyourIaaS.

AddstheMax Message SizefieldintheHost Monitoringpane.Thisallowsyoutoconfigurethemaximumnumberofcharactersofalogmessagethatisforwardedtoasyslogendpoint.Thisfeaturehelpsensurethatlogmessagesarenottruncatedatthesyslogendpoint.Bydefault,theMax Message Sizefieldis10,000characters.Formoreinformation,seeHost MonitoringintheInstallingEnterprisePKStopicforyourIaaS.

AddstheInclude kubelet metricssetting.ThisenablesoperatorstocollectworkloadmetricsacrossallKubernetesclusters.Formoreinformation,seeHost MonitoringintheInstallingEnterprisePKStopicforyourIaaS.

AddssupportforFluentBitoutputpluginstologsinks.ForinformationaboutconfiguringFluentBitoutputplugins,seeCreate aClusterLogSink or LogSink Resource with a Fluent Bit Output PlugininCreatingandManagingSinkResources.

Addssupportforfilteringlogsandeventsfroma ClusterLogSink or LogSink resource.Formoreinformation,seeFilter SinksinCreatingandManagingSinkResources.

WindowsonPKS


AddssupportforfloatingWindowsstemcellsonvSphere.ForinformationaboutKubernetesclusterswithWindowsworkersinEnterprisePKS,seeConfiguring Windows Worker-Based Kubernetes Clusters (Beta).

EnablesoperatorstoconfigurethelocationoftheWindowspauseimage.ForinformationaboutconfiguringKubeletcustomization - Windows pause image location,seePlansinConfiguringWindowsWorker-BasedKubernetesClusters(Beta).


PKSwithNSX-TNetworking


NSXErrorCRDletsclustermanagersandusersviewNSXerrorsinKubernetesresourceannotations,andusethecommandkubectl get nsxerror toviewthehealthstatusofNSX-Tclusternetworkingobjects(NCPv2.5.0+).Formoreinformation,seeViewing the Health Status of Cluster Networking Objects (NSX-T only).

DFWlogcontrolfordroppedtrafficletsclusteradministratorsdefinenetworkprofiletoturnonloggingandloganydroppedorrejectedpacketbyNSX-Tdistributedfirewallrules(NCPv2.5.0+).Formoreinformation,seeDefining Network Profiles for NCPLogging.

LoadbalancerandingressresourcecapacityobservabilityusingtheNSXLoadBalancerMonitorCRDletsclustermanagersandusersusethecommand kubectl get nsxLoadBalancerMonitors toviewahealthscorethatreflectsthecurrentperformanceoftheNSX-Tloadbalancerservice,includingusage,traffic,andcurrentstatus(NCPv2.5.1+).Formoreinformation,seeIngress Scaling(NSX-T only).

IngressscaleoutusingtheLoadBalancerCRDletsclustermanagersscaleouttheNSX-Tloadbalancerforingressrouting(NCPv2.5.1+).Formoreinformation,seeIngress Scaling (NSX-T only).

SupportforIngressURLRewrite.Formoreinformation,seeUsing Ingress URL Rewrite.

SupportforActive–ActiveTier-0routerconfigurationwhenusingaShared-Tier-1 topology.

AbilitytoplacetheloadbalancerandTier-1Active/Standbyroutersondifferentfailuredomains.SeeMultisite Deployment ofNSX-T Data Center formoreinformation.

PKSonAWSNetworking


SupportforHTTP/HTTPSProxyonAWS.Formoreinformationsee,Using Proxies with Enterprise PKS on AWS.

CustomerExperienceImprovementProgram


AdministratorscannameEnterprisePKSinstallationssotheyaremoreeasilyrecognizableinreports.Formoreinformation,seeSample Reports.

ComponentUpdates


BumpsKubernetestov1.15.5.

BumpsUAAtov73.4.8.

BumpsJacksondependenciesinthePKSAPI.

BugFixes

EnterprisePKSv1.6.0includesthefollowingbugfixes:

FixesanissuewhereenablingtheAvailabilitySetsmodeattheBOSHDirector>AzureConfigresultedinthekubeletfailingtostartonprovisioningofaKubernetescluster.


https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5/administration/GUID-5D7E3D43-6497-4273-99C1-77613C36AD75.html

FixesanissuewherepersistentvolumeattachmentfailedonvSphereinascenariowhereanAZdefinedinOpsManagerdoesnotcontainaresourcepool.

Increases network_profile columnsize.

FixesaTelemetryeventgenerationissuewherethe upgrade_cluster_end eventisnotsentforcompletedclusterupgrades.

FixesanissuewherenetworkingchangesdidnotpropagatewhenupgradingfromEnterprisePKSv1.5orlater.

FixesanissuewheretheIngressIPaddresswasexcludedfromtheEnterprisePKSfloatingIPpool.

FixesanissuewherethePKSOSBProxystartwasdelayedbyscanningallNSX-Tfirewallrules.

FixesanissuewiththePKSclustersupgradeerrandnotpushingthelatestNSX-TcertificatetoKubernetesMasternodes.

FixesanissuewiththePKSOSBProxytakingalongtimetostartduetoscanningallNSX-Tfirewallrules.

FixesanissuewithPKSreleasingfloatingIPaddressesincompletelywhiledeletingclustersunderactive/activemode.

FixesanissuewiththeDNSLookupFeature:INGRESSIPnotkeptoutofPKSFloatingIPpool.

Fixesanissuewiththecommand pks cluster details doesnotdisplayNSGroupIDofmasterVMs.

ChecksthehighavailabilitymodeoftheTier-0routerbeforecreatingPKSacluster.

ProductSnapshot

Element Details

Version v1.6.0

Releasedate November14,2019






CompatibleNSX-Tversions v2.5.0,v2.4.3

NCPversion v2.5.1



UAA v73.4.8



UpgradePath


BreakingChanges

EnterprisePKSv1.6.0hasthefollowingbreakingchanges:


https://network.pivotal.io/products/pivotal-container-service#/releases/501833https://network.pivotal.io/products/pivotal-container-service#/releases/501833https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=

PersistentVolumeDataLosswithWorkerReboot

WitholdversionsofOpsManager,PKSworkernodeswithpersistentdiskvolumesmaygetstuckinastartupstateandlosedatawhentheyarerebootedmanuallyfromthedashboardorautomaticallybyvSphereHA.

ThisissueisfixedinthefollowingOpsManagerversions:

v2.8.0+

v2.7.6+

v2.6.16+

ForallPKSinstallationsthathostworkersusingpersistentvolumes,PivotalrecommendsupgradingtooneoftheOpsManagerversionsabove.

EnterprisePKSRemovesSinkCommandsinthePKSCLI

EnterprisePKSremovesthefollowingEnterprisePKSCommandLineInterface(PKSCLI)commands:

pks create-sink

pks sinks

pks delete-sink

YoucanusethefollowingKubernetesCLIcommandsinstead:

kubectl apply -f YOUR-SINK.yml

kubectl get clusterlogsinks

kubectl delete clusterlogsink YOUR-SINK

Formoreinformationaboutdefiningandmanagingsinkresources,seeCreating and Managing Sink Resources.

ChangestoPKSAPIEndpoints

Thisreleasemovesthe clusters , compute-profiles , quotas ,and usages PKSAPIendpointsfrom v1beta1 to v1 . v1beta1 isnolongersupportedfortheseendpoints.Youmustuse v1 .Forexample,insteadof https://YOUR-PKS-API-FQDN:9021/v1beta1/quotas ,usehttps://YOUR-PKS-API-FQDN:9021/v1/quotas .

KnownIssues

EnterprisePKSv1.6.0hasthefollowingknownissues.

YourKubernetesAPIServerCACertificateExpiresUnlessYouRegenerateIt

Symptom

YourKubernetesAPIserver’s tls-kubernetes-2018 certificateisaone-yearcertificateinsteadofafour-yearcertificate.

Explanation

WhenyouupgradedfromPKSv1.2.7toPKSv1.3.1,theupgradeprocessextendedthelifespanofallPKSCAcertificatestofouryears,exceptfortheKubernetesAPIserver’s tls-kubernetes-2018 certificate.The tls-kubernetes-2018 certificateremainedaone-yearcertificate.


Unlessyouregeneratethe tls-kubernetes-2018 certificateitretainsitsone-yearlifespan,eventhroughsubsequentEnterprisePKSupgrades.

Workaround

Ifyouhavenotalreadydoneso,youshouldreplacetheKubernetesAPIserver’sone-year tls-kubernetes-2018 certificatebeforeitexpires.Forinformationaboutgeneratingandapplyinganewfour-year tls-kubernetes-2018 certificate,seeHow to regenerate tls-kubernetes-2018 certificate when it is not regenerated in the upgrade to PKS v1.3.x inthePivotalKnowledgeBase.

ClusterUpgradeDoesNotUpgradeKubernetesVersiononWindowsWorkers

WhenPKSclustersareupgraded,WindowsworkernodesintheclusterdonotupgradetheirKubernetesversion.ThemasterandLinuxworkernodesintheclusterdoupgradetheirKubernetesversionasexpected.

WhentheKubernetesversionofaWindowsworkerdoesnotexactlymatchtheversionofthemasternode,theclusterstillfunctions.

kube-apiserver hasnorestrictiononlaggingpatchbumps.

PKSclustersupgrademanuallywiththe pksupgrade-cluster command,orautomaticallywithPKSupgradeswhentheUpgrade allclusterserrandissettoDefault (On)inthePKStileErrandspane.

NetworkProfilefor“pksupdate-cluster”DoesNotUsetheDefaultsfromtheOriginalClusterManifest

Symptom

TheNetworkprofilefor pksupdate-cluster usescontentsthatarebeingupdatedandnotusingthedefaultsfromtheoriginalclustermanifest.

Explanation

The pksupdate-cluster operationsetsthe subnet_prefix to0inthencp.inifilewhenthenetwork-profilehas pod_ip_block_ids setbutitdoesnothave pod_subnet_prefix .

Workaround

Whencreatingthenetworkprofiletobeusedforupdate,includeallthebelowfields.Thenupdate-clusterwiththenetworkprofileshouldwork.

{"name":"np","parameters":{"t0_router_id":"c501f114-870b-4eda-99ac-966adf464452","fip_pool_ids":["b7acbda8-46de-4195-add2-5fb11ca46cbf"],"pod_ip_block_ids":["b03bff60-854b-4ccb-9b2b-016867b319c9","234c3652-69e7-4365-9627-8e0d8d4a6b86"],"pod_subnet_prefix":24,"single_tier_topology":false}}

AzureDefaultSecurityGroupIsNotAutomaticallyAssignedtoClusterVMs

Note:ThisissueisresolvedinEnterprisePKSv1.6.1.


https://community.pivotal.io/s/article/How-to-regenerate-tls-kubernetes-2018-certificate-when-it-was-not-regenerated-in-the-upgrade-to-PKS-v1-3-xhttps://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

Symptom

Youexperienceissueswhenconfiguringaloadbalancerforamulti-masterKubernetesclusterorcreatingaserviceoftypeLoadBalancer .Additionally,intheAzureportal,theVM>NetworkingpagedoesnotdisplayanyinboundandoutboundtrafficrulesforyourclusterVMs.

Explanation

AspartofconfiguringtheEnterprisePKStileforAzure,youenterDefault Security GroupintheKubernetes Cloud Providerpane.WhenyoucreateaKubernetescluster,EnterprisePKSautomaticallyassignsthissecuritygrouptoeachVMinthecluster.However,onAzuretheautomaticassignmentmaynotoccur.

Asaresult,yourinboundandoutboundtrafficrulesdefinedinthesecuritygrouparenotappliedtotheclusterVMs.

Workaround

Ifyouexperiencethisissue,manuallyassignthedefaultsecuritygrouptoeachVMNICinyourcluster.

ClusterCreationFailsWhenFirstAZRunsOutofResources

Symptom

Ifthefirstavailabilityzone(AZ)usedbyaplanwithmultipleAZsrunsoutofresources,clustercreationfailswithanerrorlikethefollowing:

LError:CPIerror'Bosh::Clouds::CloudError'withmessage'Novalidplacementfoundforrequestedmemory:4096

Explanation

BOSHcreatesVMsforyourEnterprisePKSdeploymentusingaround-robinalgorithm,creatingthefirstVMinthefirstAZthatyourplanuses.IftheAZrunsoutofresources,clustercreationfailsbecauseBOSHcannotcreatetheclusterVM.

Forexample,ifyouhavethreeAZsandyoucreatetwoclusterswithfourworkerVMseach,BOSHdeploysVMsinthefollowingAZs:

AZ1 AZ2 AZ3

Cluster 1 WorkerVM1 WorkerVM2 WorkerVM3

WorkerVM4

Cluster 2 WorkerVM1 WorkerVM2 WorkerVM3

WorkerVM4

Inthisscenario,AZ1hastwiceasmanyVMsasAZ2orAZ3.

ClusterCreationFailswithLongNetworkProfile

Creatingaclusterwithalongnetworkprofile,suchaswithmultiple pod_ip_block_ids values,causesanerrorDatatoolongforcolumn'nsxt_network_profile' .



AzureWorkerNodeCommunicationFailsafterUpgrade

Symptom

OutboundcommunicationfromaworkernodeVMfailsafterupgradingEnterprisePKS.

Explanation

EnterprisePKSusesAzureAvailabilitySetstoimprovetheuptimeofworkloadsandworkernodesintheeventofAzureplatformfailures.WorkernodeVMsaredistributedevenlyacrossAvailabilitySets.

AzureStandardSKULoadBalancersarerecommendedfortheKubernetescontrolplaneandKubernetesingressandegress.ThisloadbalancertypeprovidesanIPaddressforoutboundcommunicationusingSNAT.

Duringanupgrade,whenBOSHrebuildsagivenworkerinstanceinanAvailabilitySet,Azurecantimeoutwhilere-attachingtheworkernodenetworkinterfacetotheback-endpooloftheStandardSKULoadBalancer.

Formoreinformation,seeOutbound connections in Azure intheAzuredocumentation.

Workaround

Youcanmanuallyre-attachtheworkerinstancetotheback-endpooloftheAzureStandardSKULoadBalancerinyourAzureconsole.

ErrorDuringIndividualClusterUpgrades

Symptom

Whilesubmittingalargenumberofclusterupgraderequestsusingthe pksupgrade-cluster command,someofyourKubernetesclustersaremarkedasfailed.

Explanation

BOSHupgradesKubernetesclustersinparallelwithalimitofuptofourconcurrentclusterupgradesbydefault.Ifyouschedulemorethanfourclusterupgrades,EnterprisePKSqueuestheupgradesandwaitsforBOSHtofinishthelastupgrade.WhenBOSHfinishesthelastupgrade,itstartsworkingonthenextupgraderequest.

IfyousubmittoomanyclusterupgradestoBOSH,anerrormayoccur,wheresomeofyourclustersaremarkedas FAILED becauseBOSHcanstarttheupgradeonlywithinthespecifiedtimeout.Thetimeoutissetto168hoursbydefault.However,BOSHdoesnotremovethetaskfromthequeueorstopworkingontheupgradeifithasbeenpickedup.

Solution

IfyouexpectthatupgradingallofyourKubernetesclusterstakesmorethan168hours,donotuseascriptthatsubmitsupgraderequestsforallofyourclustersatonce.ForinformationaboutupgradingKubernetesclustersprovisionedbyEnterprisePKS,seeUpgrading Clusters.

KubectlCLICommandsDoNotWorkafterChanginganExistingPlantoaDifferentAZ

Symptom

AfteryouupdatetheAZofanexistingplan,kubectlCLIcommandsdonotworkforyourclustersassociatedwiththeplan.

Explanation


https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections

ThisissueoccursinIaaSenvironmentsthatdonotsupportattachingadiskacrossmultipleAZs.

WhentheplanofanexistingclusterchangestoadifferentAZ,BOSHmigratestheclusterbycreatingVMsfortheclusterinthenewAZandremovingyourclusterVMsfromtheoriginalAZ.

OnanIaaSthatdoesnotsupportattachingVMdisksacrossAZs,thedisksBOSHattachestothenewVMsdonothavetheoriginalcontent.

Workaround

IfyoucannotrunkubectlCLIcommandsafterreconfiguringtheAZofanexistingcluster,contactSupportforassistance.

ApplyingChangesFailsIfPlan8IsEnabled

Symptom

AfteryouclickApply ChangesontheOpsManagerInstallationDashboard,thefollowingerroroccurs:CannotgeneratemanifestforproductEnterprisePKS

.

Explanation

ThiserroroccursifPlan8isenabledinyourEnterprisePKSv1.6.0tile.

Workaround

DisablePlan8intheEnterprisePKStileandmoveyourplansettingstoaplanthatisavailableforconfiguration,forexample,Plan9or10.

TodisablePlan8:

1. InPlan 8,selectPlan > Inactive.

2. ClickSave.

OnePlanIDLongerthanOtherPlanIDs

Symptom

OneofyourplanIDsisonecharacterlongerthanyourotherplanIDs.

Explanation

InEnterprisePKS,eachplanhasauniqueplanID.AplanIDisnormallyaUUIDconsistingof32alphanumericcharactersand4hyphens.However,thePlan 4IDconsistsof33alphanumericcharactersand4hyphens.

Solution

YoucansafelyconfigureandusePlan 4.ThelengthofthePlan 4IDdoesnotaffectthefunctionalityofPlan 4clusters.

IfyourequireallplanIDstohaveidenticallength,donotactivateorusePlan 4.



KubernetesClusterNameLimitationforTanzuMissionControlIntegration

TanzuMissionControlintegrationcannotattachTanzuMissionControltoKubernetesclustersthathaveuppercaselettersintheirnames.

Symptom

Clustersthatyoucreatewith pkscreate-cluster

donotappearintheTanzuMissionControl,eventhoughyouconfiguredTanzuMission

ControlintegrationasdescribedinIntegrate Tanzu Mission Control.

Explanation

TheregexpatternthatparsesclusternamesinTanzuMissionControlintegrationfailswithnamesthatcontainuppercaseletters.

Solution

Whenrunning pkscreate-cluster

tocreateclustersthatyouwanttotrackinTanzuMissionControl,passinnamesthatcontainonly

lowercaselettersandnumbers.

EnterprisePKSCreatesaLinuxClusterWhenYouExpectaWindowsCluster

Symptom

WhenyoucreateanEnterprisePKSclusterusingeitherPlan11,12or13theclusteriscreatedasaLinuxclusterinsteadofaWindowscluster.

Explanation

WhenyoucreateanEnterprisePKSclusterusingeitherPlan11,12or13aWindowsclustershouldbecreated.IfyouareusingEnterprisePKSv1.6withOperationsManagerv2.7aLinuxclusteriscreatedinstead.

SavingUAATabSettingsFailsWithError:‘InvalidURIErrorbadURI’

Symptom

WhenyousaveyourUAAtabwithLDAP ServerselectedandmultipleLDAPserversspecified,youreceivetheerror:URI::InvalidURIErrorbadURI(isnotURI?):LDAPURLs

.

Explanation

WhenyouconfiguretheUAAtabwithmultipleLDAPserversyoursettingswillfailtovalidatewhenusingthefollowingOpsManagerreleases:

Ops Manager Version Affected ReleasesOps Manager v2.6 OpsManagerv2.6.18andearlierpatchreleases.

Ops Manager v2.7 Allpatchreleases.

Ops Manager v2.8 Allpatchreleases.



Workaround

Toresolvethisissueseethefollowing:

Ops Manager Version Workaround

Ops Manager v2.6

Performoneofthefollowing:UpgradetoOpsManagerv2.6.19orlaterv2.6patchrelease.

Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.

Ops Manager v2.7Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.

Ops Manager v2.8Completetheproceduresin UAA authentication tab in PKS 1.6 fails to save with error“URI::InvalidURIError bad URI(is not URI?):LDAP URLs” (76495) inthePivotalSupportKnowledgeBase.

WindowsWorkerClustersFailtoUpgradetov1.6

Symptoms

DuringyourupgradefromEnterprisePKSv1.5toEnterprisePKSv1.6aWindowsworkerVMfailstoupgrade,asevidencedby:

Thecommandlineoutputsanerror Failed jobs: docker-windows .

TheWindowsworkerVMdisappearsfromtheoutputof kubectl get nodes .

Thecommandlineshowsthestatus failed andtheaction UPGRADE fortheclusterthatcontainstheworker.

Thelogshowsanentry \docker\dockerd.exe: Access is denied .

Explanation

BetweenPKSv1.5andv1.6,thenameoftheDockerservicechangedfrom docker to docker-windows ,butyourenvironmentcontinuestousetheoldDockerservicenameandpaths.TheincompatibleservicenameandpathingcausesaWindowsworkerupgradetofail.

IfyourclusterhasmultipleWindowsworkers,thisissuedoesnotincurdowntime.BeforeBOSHattemptstoupgradeaWindowsworker,itmovestheworker’sappstootherWindowsworkersinthecluster.Whentheupgradefails,BOSHstopstheclusterupgradeprocessandtheotherWindowsworkerscontinuerunningattheearlierversion.

Workaround

AfterupgradingtoEnterprisePKSv1.6andyourWindowsworkerclustershavefailedtoupgrade,completethefollowingsteps:

1. UploadavSpherestemcellv2019.8orlaterforWindowsServerversion2019toyourEnterprisePKStile.

2. ToupgradeyourWindowsworkerclusters,performoneofthefollowing:

EnabletheUpgrade all clusters errandsettinganddeploythePKStile.FormoreinformationaboutconfiguringtheUpgrade all clusters errandanddeployingtheEnterprisePKStile,seeModify Errand Configuration in the EnterprisePKS TileinUpgradingClusters.


https://kb.vmware.com/s/article/76495https://kb.vmware.com/s/article/76495https://kb.vmware.com/s/article/76495

Run pks upgrade-cluster or pks upgrade-clusters onyourfailedWindowsworkercluster(s).FormoreinformationaboutupgradingspecificEnterprisePKSclusters,seeUpgrade ClustersinUpgradingClusters.

502BadGatewayAfterOIDCLogin

Symptom

Youexperiencea“502BadGateway”errorfromtheNSXloadbalancerafteryoulogintoOIDC.

Explanation

AlargeresponseheaderhasexceededyourNSX-Tloadbalancermaximumresponseheadersize.Thedefaultmaximumresponseheadersizeis10,240charactersandshouldberesizedto50,000.

Workaround

Ifyouexperiencethisissue,manuallyreconfigureyourNSX-T request_header_size and response_header_size to50,000characters.ForinformationaboutconfiguringNSX-Tdefaultheadersizes,seeOIDC Response Header Overflow inthePivotalKnowledgeBase.

NSX-TPre-CheckErrandFailsDuetoEdgeNodeConfiguration

Symptom

YouhaveconfiguredyourNSX-TEdgeNodeVMas medium size,andtheNSX-TPre-CheckErrandfailswiththefollowingerror:“ERROR:NSX-TPrecheckfailedduetoEdgeNode…noofcpucoresislessthan8”.

Explanation

TheNSX-TPre-CheckErrandiserroneouslyreturningthe“cpucoresislessthan8”error.

Solution

YoucansafelyconfigureyourNSX-TEdgeNodeVMsas medium sizeandignoretheerror.

CharacterLimitationsinHTTPProxyPassword

ForvSpherewithNSX-T,theHTTPProxypasswordfielddoesnotsupportthefollowingspecialcharacters: & or ; .

EnterprisePKSManagementConsole1.6.2

Release Date:April29,2020

Features

OtherthansupportforEnterprisePKSv1.6.2,EnterprisePKSManagementConsole1.6.2hasnonewfeatures.

BugFixes

EnterprisePKSManagementConsole1.6.2includesnobugfixes.


https://community.pivotal.io/s/article/OIDC-Response-Header-overflow

ProductSnapshot

Element Details

Version v1.6.2

Releasedate April29,2020

InstalledEnterprisePKSversion v1.6.2

InstalledOpsManagerversion v2.8.5

InstalledKubernetesversion v1.15.10


InstalledHarborRegistryversion v1.9.4

KnownIssues

TheEnterprisePKSManagementConsolev1.6.2applianceanduserinterfacehavethesameknown issuesasv1.6.1.

EnterprisePKSManagementConsole1.6.1

Release Date:January23,2020

Features

OtherthansupportforEnterprisePKSv1.6.1,EnterprisePKSManagementConsole1.6.1hasnonewfeatures.

BugFixes

EnterprisePKSManagementConsole1.6.1includesnobugfixes.

ProductSnapshot

Element Details

Version v1.6.1

Releasedate January23,2020





Note:EnterprisePKSManagementConsoleprovidesanopinionatedinstallationofEnterprisePKS.ThesupportedversionsmaydifferfromorbemorelimitedthanwhatisgenerallysupportedbyEnterprisePKS.




KnownIssues

TheEnterprisePKSManagementConsolev1.6.1applianceanduserinterfacehavethesameknown issuesasv1.6.0-rev.3andv1.6.0-rev.2.

EnterprisePKSManagementConsole1.6.0-rev.3

Release Date:December19,2019

Features

EnterprisePKSManagementConsole1.6.0-rev.3hasnonewfeatures.

BugFixes

EnterprisePKSManagementConsole1.6.0-rev.3includesthefollowingbugfixes:

FixesUIfailurecausedbymultipledatacentersbeingpresentinvCenterServer.

AddssupportforbothFQDNandIPaddressesinLDAP/LDAPSconfigurationforidentitymanagement.

FixesUIfreezingafterenteringunconventionallyformattedURLsforSAMLprovidermetadata.

AddssupportforUAArole pks.clusters.admin.read inidentityManagementconfiguration.

AddsvalidationforHarborFQDNinlowercase.

FixesmisconfiguredWavefrontHTTPProxywhenfieldisleftempty.

ProductSnapshot

Element Details

Version v1.6.0-rev.3

Releasedate December19,2019






important:TheEnterprisePKSManagementConsole1.6.0-rev.3offlinepatchcanonlybeappliedinanair-gappedenvironment.Itcanonlybeappliedto1.6.0-rev.2andnottoanyotherversion.Forinformationabouthowtoapplythepatch,seePatch Enterprise PKS Management Console Components.



KnownIssues

WiththeexceptionoftheBug Fixeslistedabove,theEnterprisePKSManagementConsolev1.6.0-rev.3applianceanduserinterfacehavethesameknown issuesasv1.6.0-rev.2.

EnterprisePKSManagementConsolev1.6.0-rev.2

Release Date:November26,2019

Features

EnterprisePKSManagementConsolev1.6.0-rev.2updatesinclude:

ProvidesexperimentalintegrationwithVMwareTanzuMissionControl.Formoreinformation,seeTanzu Mission ControlIntegration.

ProvidesexperimentalsupportforplansthatuseWindowsworkernodes.Forinformation,seeConfigure Plans.

DeploysHarborregistryv1.9.Forinformation,seeConfigure Harbor.

Addssupportforactive-activemodeonthetier0routerinautomated-NATdeploymentsandNo-NATconfigurationsinBringYourOwnTopologydeployments.Forinformation,seeConfigure Networking.

AddstheabilitytoconfigureproxiesfortheintegrationwithWavefront.Forinformation,seeConfigure a Connection toWavefront.

AddstheabilitytoconfigurethesizeofthePKSAPIVM.Forinformation,seeConfigure Resources and Storage.

Allowsyoutousethemanagementconsoletoupgradetov1.6.0-rev.2.Forinformation,seeUpgrade Enterprise PKSManagement Console.

ProductSnapshot

Element Details

Version v1.6.0-rev.2

Releasedate November26,2019






KnownIssues

ThefollowingknownissuesarespecifictotheEnterprisePKSManagementConsolev1.6.0-rev.2applianceanduserinterface.

YAMLValidationErrorsNotCleared



Symptom

IfyouattempttouploadaYAMLconfigurationfileandthedeploymentfailsbecauseofaninvalidmanifest,EnterprisePKSManagementConsoledisplaysanerrornotificationwiththevalidationerror.Ifsubsequentattemptsalsofailbecauseofvalidationissues,thevalidationerrorsareappendedtoeachother.

Explanation

ThevalidationerrorsarenotclearedwhenyouresubmittheYAMLconfigurationfile.

Workaround

None

EnterprisePKSManagementConsoleNotificationsPersist

Symptom

IntheEnterprise PKSviewofEnterprisePKSManagementConsole,errornotificationssometimespersistinmemoryontheClustersandNodespagesafteryouclearthosenotifications.

Explanation

AfterclickingtheXbuttontoclearanotificationitisremoved,butwhenyounavigatebacktothosepagesthenotificationmightshowagain.

Workaround

Useshift+refreshtoreloadthepage.

CannotDeleteEnterprisePKSDeploymentfromManagementConsole

Symptom

IntheEnterprise PKSviewofEnterprisePKSManagementConsole,youcannotusetheDelete Enterprise PKS Deploymentoptionevenafteryouhaveremovedallclusters.

Explanation

Theoptiontodeletethedeploymentisonlyactivatedinthemanagementconsoleashortperiodaftertheclustersaredeleted.

Workaround

Afterremovingclusters,waitforafewminutesbeforeattemptingtousetheDelete Enterprise PKS Deploymentoptionagain.

ConfiguringEnterprisePKSManagementConsoleIntegrationwithVMwarevRealizeLogInsight

Symptom

EnterprisePKSManagementConsoleappliancesendslogstoVMwarevRealizeLogInsightoverHTTP,notHTTPS.

Explanation


WhenyoudeploytheEnterprisePKSManagementConsoleappliancefromtheOVA,ifyourequirelogforwardingtovRealizeLogInsight,youmustprovidetheportonthevRealizeLogInsightserveronwhichitlistensforHTTPtraffic.DonotprovidetheHTTPSport.

Workaround

SetthevRealizeLogInsightporttotheHTTPport.Thisistypicallyport 9000 .

DeployingEnterprisePKStoanUnpreparedNSX-TDataCenterEnvironmentResultsinFlannelError

Symptom

WhenusingthemanagementconsoletodeployEnterprisePKSinNSX-T Data Center (Not prepared for PKS)mode,ifanerroroccursduringthenetworkconfiguration,themessage Unabletosetflannel

environmentisdisplayedinthedeploymentprogresspage.

Explanation

Thenetworkconfigurationhasfailed,buttheerrormessageisincorrect.

Workaround

Toseethecorrectreasonforthefailure,seetheserverlogs.Forinstructionsabouthowtoobtaintheserverlogs,seeTroubleshootingEnterprise PKS Management Console.

UsingBOSHCLIfromOperationsManagerVM

Symptom

TheBOSHCLIclientbashcommandthatyouobtainfromtheDeployment MetadataviewdoesnotworkwhenloggedintotheOperationsManagerVM.

Explanation

TheBOSHCLIclientbashcommandfromtheDeployment MetadataviewisintendedtobeusedfromwithintheEnterprisePKSManagementConsoleappliance.

Workaround

TousetheBOSHCLIfromwithintheOperationsManagerVM,seeConnect to Operations Manager.

FromtheOpsManagerVM,usetheBOSHCLIclientbashcommandfromtheDeployment Metadatapage,withthefollowingmodifications:

Removetheclause BOSH_ALL_PROXY=xxx

Replacethe BOSH_CA_CERT sectionwith BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate

Run pks CommandsagainstthePKSAPIServer

Explanation

ThePKSCLIisavailableintheEnterprisePKSManagementConsoleappliance.


Workaround

Tobeabletorun pks commandsagainstthePKSAPIServer,youmustfirstlogtoPKSusingthefollowingcommandsyntaxpkslogin-afqdn_of_pks...

.

Todothis,youmustensureeitherofthefollowing:

TheFQDNconfiguredforthePKSServerisresolvablebytheDNSserverconfiguredfortheEnterprisePKSManagementConsoleappliance,or

AnentrythatmapstheFloatingIPassignedtothePKSServertotheFQDNexistson/etc/hostsintheappliance.Forexample:192.168.160.102 api.pks.local .

[email protected].



EnterprisePKSConceptsPage last updated:

ThistopicdescribesVMwareEnterprisePKSconcepts.Seethefollowingsections:

Enterprise PKS Cluster Management

PKS API Authentication

Load Balancers in Enterprise PKS

VM Sizing for Enterprise PKS Clusters

Telemetry

Sink Architecture in Enterprise PKS

[email protected].



EnterprisePKSClusterManagement

In this topic

Overview

ClusterLifecycleManagementPKSControlPlaneOverview

PKSControlPlaneArchitecture

ClusterWorkloadManagement

Page last updated:

ThistopicdescribeshowVMwareEnterprisePKSmanagesthedeploymentofKubernetesclusters.

Overview

UsersinteractwithEnterprisePKSandEnterprisePKS-deployedKubernetesclustersintwoways:

DeployingKubernetesclusterswithBOSHandmanagingtheirlifecycle.ThesetasksareperformedusingthePKSCommandLineInterface(PKSCLI)andthePKScontrolplane.

Deployingandmanagingcontainer-basedworkloadsonKubernetesclusters.ThesetasksareperformedusingtheKubernetesCLI,kubectl .

ClusterLifecycleManagement

ThePKScontrolplaneenablesuserstodeployandmanageKubernetesclusters.

ForcommunicatingwiththePKScontrolplane,EnterprisePKSprovidesacommandlineinterface,thePKSCLI.SeeInstalling thePKS CLIforinstallationinstructions.

PKSControlPlaneOverview

ThePKScontrolplanemanagesthelifecycleofKubernetesclustersdeployedusingEnterprisePKS.ThecontrolplaneallowsuserstodothefollowingthroughthePKSCLI:

Viewclusterplans

Createclusters

Viewinformationaboutclusters

Obtaincredentialstodeployworkloadstoclusters

Scaleclusters

Deleteclusters

CreateandmanagenetworkprofilesforVMwareNSX-T

Inaddition,thePKScontrolplanecanupgradeallexistingclustersusingtheUpgrade all clustersBOSHerrand.Formoreinformation,seeUpgrade Kubernetes ClustersinUpgradingEnterprisePKS.


PKSControlPlaneArchitecture

ThePKScontrolplaneisdeployedonasingleVMthatincludesthefollowingcomponents:

ThePKSAPIserver

ThePKSBroker

AUserAccountandAuthentication(UAA)server

Thefollowingillustrationshowshowthesecomponentsinteract:

ThePKSAPILoadBalancerisusedforAWS,GCP,andvSpherewithoutNSX-Tdeployments.IfEnterprisePKSisdeployedonvSpherewithNSX-T,aDNATruleisconfiguredforthePKSAPIhostsothatitisaccessible.Formoreinformation,seetheShare the PKS APIEndpointsectioninInstallingEnterprisePKSonvSpherewithNSX-TIntegration.

UAA

WhenauserlogsintoorlogsoutofthePKSAPIthroughthePKSCLI,thePKSCLIcommunicateswithUAAtoauthenticatethem.ThePKSAPIpermitsonlyauthenticateduserstomanageKubernetesclusters.Formoreinformationaboutauthenticating,seePKS APIAuthentication.


UAAmustbeconfiguredwiththeappropriateusersanduserpermissions.Formoreinformation,seeManaging Enterprise PKSUsers with UAA.

PKSAPI

ThroughthePKSCLI,usersinstructthePKSAPIservertodeploy,scaleup,anddeleteKubernetesclustersaswellasshowclusterdetailsandplans.ThePKSAPIcanalsowriteKubernetesclustercredentialstoalocalkubeconfigfile,whichenablesuserstoconnecttoaclusterthrough kubectl .

ThePKSAPIsendsallclustermanagementrequests,exceptread-onlyrequests,tothePKSBroker.

PKSBroker

WhenthePKSAPIreceivesarequesttomodifyaKubernetescluster,itinstructsthePKSBrokertomaketherequestedchange.

ThePKSBrokerconsistsofanOn-Demand Service Broker andaServiceAdapter.ThePKSBrokergeneratesaBOSHmanifestandinstructstheBOSHDirectortodeployordeletetheKubernetescluster.

ForEnterprisePKSdeploymentsonvSpherewithNSX-T,thereisanadditionalcomponent,theEnterprisePKSNSX-TProxyBroker.ThePKSAPIcommunicateswiththePKSNSX-TProxyBroker,whichinturncommunicateswiththeNSXManagertoprovisiontheNodeNetworkingresources.ThePKSNSX-TProxyBrokerthenforwardstherequesttotheOn-DemandServiceBrokertodeploythecluster.

ClusterWorkloadManagement

EnterprisePKSusersmanagetheircontainer-basedworkloadsonKubernetesclustersthrough kubectl .Formoreinformationaboutkubectl ,seeOverview of kubectl intheKubernetesdocumentation.

[email protected].


https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://kubernetes.io/docs/reference/kubectl/overview/mailto:[email protected]

PKSAPIAuthentication

In this topic

AuthenticationofPKSAPIRequests

RoutingtothePKSAPIControlPlaneVM

Page last updated:

ThistopicdescribeshowtheVMwareEnterprisePKSAPIworkswithUserAccountandAuthentication(UAA)tomanageauthenticationandauthorizationinyourEnterprisePKSdeployment.

AuthenticationofPKSAPIRequests

BeforeuserscanloginandusethePKSCLI,youmustconfigurePKSAPIaccesswithUAA.Formoreinformation,seeManagingEnterprise PKS Users with UAAandLogging in to Enterprise PKS.

YouusetheUAACommandLineInterface(UAAC)totargettheUAAserverandrequestanaccesstokenfortheUAAadminuser.Ifyourrequestissuccessful,theUAAserverreturnstheaccesstoken.TheUAAadminaccesstokenauthorizesyoutomakerequeststothePKSAPIusingthePKSCLIandgrantclusteraccesstoneworexistingusers.

WhenauserwithclusteraccesslogsintothePKSCLI,theCLIrequestsanaccesstokenfortheuserfromtheUAAserver.Iftherequestissuccessful,theUAAserverreturnsanaccesstokentothePKSCLI.WhentheuserrunsPKSCLIcommands,forexample, pksclusters ,theCLIsendstherequesttothePKSAPIserverandincludestheuser’sUAAtoken.

ThePKSAPIsendsarequesttotheUAAservertovalidatetheuser’stoken.IftheUAAserverconfirmsthatthetokenisvalid,thePKSAPIusestheclusterinformationfromthePKSbrokertorespondtotherequest.Forexample,iftheuserruns pksclusters ,theCLIreturnsalistoftheclustersthattheuserisauthorizedtomanage.

RoutingtothePKSAPIControlPlaneVM

ThePKSAPIserverandtheUAAserverusedifferentportnumbersonthecontrolplaneVM.Forexample,ifyourPKSAPIdomainisapi.pks.example.com ,youcanreachyourPKSAPIandUAAserversatthefollowingURLs:

Server URL

PKSAPI api.pks.example.com:9021

UAA api.pks.example.com:8443

RefertoOps Manager>Enterprise PKS tile>PKS API>API Hostname (FQDN)foryourPKSAPIdomain.

Loadbalancerimplementationsdifferbydeploymentenvironment.ForEnterprisePKSdeploymentsonGCP,AWS,orvSpherewithoutNSX-T,youconfigurealoadbalancertoaccessthePKSAPIwhenyouinstalltheEnterprisePKStile.Forexample,seeConfiguring PKSAPI Load Balancer.

ForoverviewinformationaboutloadbalancersinEnterprisePKS,seeLoad Balancers in Enterprise PKS Deployments withoutNSX-T.

[email protected].



UAAScopesforEnterprisePKSUsers

In this topic

Overview

UAAScopes

Page last updated:

ThistopicdescribesUserAccountandAuthentication(UAA)scopesthataUAAadmincanassigntoVMwareEnterprisePKSusers.

Overview

UAAistheidentitymanagementserviceforEnterprisePKS.

ByassigningUAAscopes,yougrantuserstheabilitytocreate,manage,andauditKubernetesclustersinEnterprisePKS.

AUAAadminusercanassignthefollowingUAAscopestoEnterprisePKSusers:

pks.clusters.manage :Accountswiththisscopecancreateandaccesstheirownclusters.

pks.clusters.admin :Accountswiththisscopecancreateandaccessallclusters.

pks.clusters.admin.read :Accountswiththisscopecanaccessanyinformationaboutallclustersexceptforclustercredentials.

Youcanassignthesescopestoindividualusers,externalidentityprovidergroups,orclientsforautomationpurposes.

UAAScopes

EachUAAscopegrantsEnterprisePKSusersasetofpermissionsforcreating,managing,andauditingEnterprisePKS-provisionedKubernetesclusters.Forinformationaboutthepermissions,seethetablebelow.

Operation pks.clusters.manage pks.clusters.admin pks.clusters.admin.read

Create, update,resize, and delete acluster

Yes.Userswiththisscopecancreate,modify,anddeleteonlytheirownclusters.

Yes.Userswiththisscopecancreate,modify,anddeleteallclusters.

No.Userswiththisscopecannotcreate,modify,anddeleteclusters.

Get clustercredentials

Yes.Userswiththisscopecanretrieveclustercredentialsonlyfortheirownclusters.

Yes.Userswiththisscopecanretrieveclustercredentialsforallclusters.

No.Userswiththisscopecannotretrieveclustercredentials.

Upgrade clustersYes.Userswiththisscopecanupgradeonlytheirownclusters.

Yes.Userswiththisscopecanupgradeallclusters.

No.Userswiththisscopecannotupgradeclusters.

List clusters Yes.Userswiththisscopecanlistonlytheirownclusters.

Yes.Userswiththisscopecanlistallclusters.

Yes.Userswiththisscopecanlistallclusters.

View cluster detailsYes.Userswiththisscopecanviewclusterdetailsonlyfortheirownclusters.

Yes.Userswiththisscopecanviewclusterdetailsforallclusters.

Yes.Userswiththisscopecanviewclusterdetailsforallclusters.


Create and delete acompute profile

No.Userswiththisscopecannotcreateanddeletecomputeprofiles.

Yes.Userswiththisscopecancreateanddeletecomputeprofiles.

No.Userswiththisscopecannotcreateanddeletecomputeprofiles.

Create and delete anetwork profile

No.Userswiththisscopecannotcreateanddeletenetworkprofiles.

Yes.Userswiththisscopecancreateanddeletenetworkprofiles.

No.Userswiththisscopecannotcreateanddeletenetworkprofiles.

Create, update, anddelete a quota

No.Userswiththisscopecannotcreate,update,anddeletequotas.

Yes.Userswiththisscopecancreate,update,anddeletequotas.

No.Userswiththisscopecannotcreate,update,anddeletequotas.

List Enterprise PKSplans

Yes.Userswiththisscopecanlistallavailableplans.



ToassignUAAscopesinEnterprisePKS,followtheinstructionsinManaging Enterprise PKS Users with UAA.

[email protected].



LoadBalancersinEnterprisePKS

In this topic

LoadBalancersinEnterprisePKSDeploymentswithoutNSX-TAboutthePKSAPILoadBalancer

AboutKubernetesClusterLoadBalancers

AboutWorkloadLoadBalancers

LoadBalancersinEnterprisePKSDeploymentsonvSpherewithNSX-TResizingLoadBalancers

Page last updated:

ThistopicdescribesthetypesofloadbalancersthatareusedinVMwareEnterprisePKSdeployments.Loadbalancersdifferbythetypeofdeployment.

LoadBalancersinEnterprisePKSDeploymentswithoutNSX-TForEnterprisePKSdeploymentsonGCP,AWS,orvSpherewithoutNSX-T,youcanconfigureloadbalancersforthefollowing:

PKS API:ConfiguringthisloadbalancerenablesyoutorunPKSCommandLineInterface(PKSCLI)commandsfromyourlocalworkstation.

Kubernetes Clusters:ConfiguringaloadbalancerforeachnewclusterenablesyoutorunKubernetesCLI(kubectl)commandsonthecluster.

Workloads:Configuringaloadbalancerforyourapplicationworkloadsenablesexternalaccesstotheservicesthatrunonyourcluster.

Thefollowingdiagram,applicabletoGCP,AWS,andvSpherewithoutNSX-T,showswhereeachoftheaboveloadbalancerscanbeusedwithinyourEnterprisePKSdeployment.


IfyouuseeithervSpherewithoutNSX-TorGCP,youareexpectedtocreateyourownloadbalancerswithinyourcloudproviderconsole.Ifyourcloudproviderdoesnotofferloadbalancing,youcanuseanyexternalTCPorHTTPSloadbalancerofyourchoice.

AboutthePKSAPILoadBalancer

ThePKSAPIloadbalancerenablesyoutoaccessthePKSAPIfromoutsidethenetworkonEnterprisePKSdeploymentsonGCP,AWS,andonvSpherewithoutNSX-T.Forexample,configuringaloadbalancerforthePKSAPIenablesyoutorunPKSCLIcommandsfromyourlocalworkstation.

ForinformationaboutconfiguringthePKSAPIloadbalanceronvSpherewithoutNSX-T,seeConfiguring PKS API Load Balancer.

AboutKubernetesClusterLoadBalancers

WhenyoucreateanEnterprisePKSclusteronGCP,AWS,andonvSpherewithoutNSX-T,youmustconfigureexternalaccesstothe


clusterbycreatinganexternalTCPorHTTPSloadbalancer.TheloadbalancerenablestheKubernetesCLItocommunicatewiththecluster.

Ifyoucreateaclusterinanon-productionenvironment,youcanchoosenottousealoadbalancer.Toenablekubectltoaccesstheclusterwithoutaloadbalancer,youcandooneofthefollowing:

CreateaDNSentrythatpointstothecluster’smasterVM.Forexample:

my-cluster.example.com A 10.0.0.5

Ontheworkstationwhereyourunkubectlcommands,addthemasterIPaddressofyourclusterand kubo.internal tothe/etc/hosts file.Forexample:

10.0.0.5 kubo.internal

Formoreinformationaboutconfiguringaclusterloadbalancer,seethefollowing:

Creating and Configuring a GCP Load Balancer for Enterprise PKS Clusters

Creating and Configuring an AWS Load Balancer for Enterprise PKS Clusters

Creating and Configuring an Azure Load Balancer for Enterprise PKS Clusters

AboutWorkloadLoadBalancers

ToenableexternalaccesstoyourEnterprisePKSapponGCP,AWS,andonvSpherewithoutNSX-T,youcaneithercreatealoadbalancerorexposeastaticportonyourworkload.

Forinformationaboutconfiguringaloadbalancerforyourappworkload,seeDeploying and Exposing Basic Linux Workloads.

IfyouuseAWS,youmustconfigureroutingintheAWSconsolebeforeyoucancreatealoadbalancerforyourworkload.Youmustcreateapublicsubnetineachavailabilityzone(AZ)whereyouaredeployingtheworkloadandtagthepublicsubnetwithyourcluster’suniqueidentifier.

SeetheAWS PrerequisitessectionofDeployingandExposingBasicLinuxWorkloadsbeforeyoucreateaworkloadloadbalancer.

DeployYourWorkloadLoadBalancerwithanIngressController

AKubernetesingresscontrollersitsbehindaloadbalancer,routingHTTPandHTTPSrequestsfromoutsidetheclustertoserviceswithinthecluster.Kubernetesingressresourcescanbeconfiguredtoloadbalancetraffic,provideexternallyreachableURLstoservices,andmanageotheraspectsofnetworktraffic.

IfyouaddaningresscontrollertoyourEnterprisePKSdeployment,trafficroutingiscontrolledbytheingressresourcerulesyoudefine.PivotalrecommendsconfiguringEnterprisePKSdeploymentswithbothaworkloadloadbalancerandaningresscontroller.

ThefollowingdiagramshowshowtheingressroutingcanbeusedwithinyourEnterprisePKSdeployment.


TheloadbalanceronEnterprisePKSonvSpherewithNSX-TisautomaticallyprovisionedwithKubernetesingressresourceswithouttheneedtodeployandconfigureanadditionalingresscontroller.

ForinformationaboutdeployingaloadbalancerconfiguredwithingressroutingonGCP,AWS,Azure,andvSpherewithoutNSX-T,seeConfiguring Ingress Routing.ForinformationaboutingressroutingonvSpherewithNSX-T,seeConfiguring Ingress Resourcesand Load Balancer Services.

LoadBalancersinEnterprisePKSDeploymentsonvSpherewithNSX-TEnterprisePKSdeploymentsonvSpherewithNSX-TdonotrequirealoadbalancerconfiguredtoaccessthePKSAPI.TheyrequireonlyaDNATruleconfiguredsothatthePKSAPIhostisaccessible.Formoreinformation,seeShare the Enterprise PKS EndpointinInstallingEnterprisePKSonvSpherewithNSX-TIntegration.

NSX-Thandlesloadbalancercreation,configuration,anddeletionautomaticallyaspartoftheKubernetesclustercreate,update,anddeleteprocess.WhenanewKubernetesclusteriscreated,NSX-Tcreatesandconfiguresadedicatedloadbalancertiedtoit.Theloadbalancerisasharedresourcedesignedtoprovideefficienttrafficdistributiontomasternodesaswellasservicesdeployedonworkernodes.Eachapplicationserviceismappedtoavirtualserverinstance,carvedoutfromthesameloadbalancer.Formoreinformation,seeLogical Load Balancer intheNSX-Tdocumentation.

Virtualserverinstancesarecreatedontheloadbalancertoprovideaccesstothefollowing:

Kubernetes API and UI services on a Kubernetes cluster.Thisenablesrequeststobeloadbalancedacrossmultiplemasternodes.

Ingress controller.ThisenablesthevirtualserverinstancetodispatchHTTPandHTTPSrequeststoservicesassociatedwithIngressrules.

type:loadbalancer services.ThisenablestheservertohandleTCPconnectionsorUDPflowstowardexposedservices.

Loadbalancersaredeployedinhigh-availabilitymodesothattheyareresilienttopotentialfailuresandabletorecoverquicklyfromcriticalconditions.

Note:The NodePort ServicetypeisnotsupportedforEnterprisePKSdeploymentsonvSpherewithNSX-T.Only


https://docs.vmware.com/en/VMware-NSX-T/2.1/com.vmware.nsxt.admin.doc/GUID-46567C8D-A5C5-4793-8CDF-858E58FDE3C4.html

ResizingLoadBalancers

WhenanewKubernetesclusterisprovisionedusingthePKSAPI,NSX-Tcreatesadedicatedloadbalancerforthatnewcluster.Bydefault,thesizeoftheloadbalancerissettoSmall.

Withnetworkprofiles,youcanchangethesizeoftheloadbalancerdeployedbyNSX-Tatthetimeofclustercreation.Forinformationaboutnetworkprofiles,seeUsing Network Profiles (NSX-T Only).

FormoreinformationaboutthetypesofloadbalancersNSX-Tprovisionsandtheircapacities,seeScaling Load BalancerResources intheNSX-Tdocumentation.

[email protected].

type:LoadBalancer ServicesandServicesassociatedwithIngressrulesaresupportedonvSpherewithNSX-T.


https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.3/com.vmware.nsxt.admin.doc/GUID-19B12230-8BF4-4AF7-9EB7-3701B0A0A439.htmlmailto:[email protected]

VMSizingforEnterprisePKSClusters

In this topic

Overview

MasterNodeVMSize

WorkerNodeVMNumberandSizeExampleWorkerNodeRequirementCalculation

CustomizeMasterandWorkerNodeVMSizeandType

Page last updated:

ThistopicdescribeshowVMwareEnterprisePKSrecommendsyouapproachthesizingofVMsforclustercomponents.

Overview

WhenyouconfigureplansintheEnterprisePKStile,youprovideVMsizesforthemasterandworkernodeVMs.Formoreinformationaboutconfiguringplans,seethePlanssectionofInstallingEnterprisePKSforyourIaaS:

vSphere


Google Cloud Platform (GCP)

Amazon Web Services (AWS)

Azure

Youselectthenumberofmasternodeswhenyouconfiguretheplan.

ForworkernodeVMs,youselectthenumberandsizebasedontheneedsofyourworkload.ThesizingofmasterandworkernodeVMsishighlydependentonthecharacteristicsoftheworkload.Adapttherecommendationsinthistopicbasedonyourownworkloadrequirements.

MasterNodeVMSize

ThemasternodeVMsizeislinkedtothenumberofworkernodes.TheVMsizingshowninthefollowingtableispermasternode:

TocustomizethesizeoftheKubernetesmasternodeVM,seeCustomize Master and Worker Node VM Size and Type.

Number of Workers CPU RAM (GB)

1-5 1 3.75

6-10 2 7.5

11-100 4 15

101-250 8 30

Note:Iftherearemultiplemasternodes,allmasternodeVMsarethesamesize.Toconfigurethenumberofmasternodes,seethePlanssectionofInstallingEnterprisePKSforyourIaaS.


251-500 16 60

500+ 32 120

Number of Workers CPU RAM (GB)

DonotoverloadyourmasternodeVMsbyexceedingtherecommendedmaximumnumberofworkernodeVMsorbydownsizingfromtherecommendedVMsizingslistedabove.TheserecommendationssupportbothatypicalworkloadmanagedbyaVMandthehigherthanusualworkloadmanagedbytheVMwhileotherVM’sintheclusterareupgrading.

WorkerNodeVMNumberandSize

Amaximumof100podscanrunonasingleworkernode.TheactualnumberofpodsthateachworkernoderunsdependsontheworkloadtypeaswellastheCPUandmemoryrequirementsoftheworkload.

TocalculatethenumberandsizeofworkerVMsyourequire,determinethefollowingforyourworkload:

Maximumnumberofpodsyouexpecttorun[ p ]

Memoryrequirementsperpod[ m ]

CPUrequirementsperpod[ c ]

Usingthevaluesabove,youcancalculatethefollowing:

Minimumnumberofworkers[ W ]= p / 100

MinimumRAMperworker= m * 100

MinimumnumberofCPUsperworker= c * 100

Thiscalculationgivesyoutheminimumnumberofworkernodesyourworkloadrequires.Werecommendthatyouincreasethisvaluetoaccountforfailuresandupgrades.

Forexample,increasethenumberofworkernodesbyatleastonetomaintainworkloaduptimeduringanupgrade.Additionally,increasethenumberofworkernodestofityourownfailuretolerancecriteria.

ThemaximumnumberofworkernodesthatyoucancreateforaplaninanEnterprisePKS-provisionedKubernetesclusterissetbytheMaximum number of workers on a clusterfieldinthePlanspaneoftheEnterprisePKStile.TocustomizethesizeoftheKubernetesworkernodeVM,seeCustomize Master and Worker Node VM Size and Type.

ExampleWorkerNodeRequirementCalculation

Anexampleapphasthefollowingminimumrequirements:

Numberofpods[ p ]=1000

RAMperpod[ m ]=1GB

CPUperpod[ c ]=0.10

TodeterminehowmanyworkernodeVMstheapprequires,dothefollowing:

1. Calculatethenumberofworkersusing p / 100 :

warning:UpgradinganoverloadedKubernetesclustermasternodeVMcanresultindowntime.


1000/100 = 10 workers

2. CalculatetheminimumRAMperworkerusing m * 100 :

1 * 100 = 100 GB

3. CalculatetheminimumnumberofCPUsperworkerusing c * 100 :

0.10 * 100 = 10 CPUs

4. Forupgrades,increasethenumberofworkersbyone:

10 workers + 1 worker = 11 workers

5. Forfailuretolerance,increasethenumberofworkersbytwo:

11 workers + 2 workers = 13 workers

Intotal,thisappworkloadrequires13workerswith10CPUsand100GBRAM.

CustomizeMasterandWorkerNodeVMSizeandType

YouselecttheCPU,memory,anddiskspacefortheKubernetesnodeVMsfromasetlistintheEnterprisePKStile.MasterandworkernodeVMsizesandtypesareselectedonaper-planbasis.Formoreinformation,seethePlanssectionoftheEnterprisePKSinstallationtopicforyourIaaS.Forexample,Installing Enterprise PKS on vSphere with NSX-T.

WhilethelistofavailablenodeVMtypesandsizesisextensive,thelistmaynotprovidetheexacttypeandsizeofVMthatyouwant.YoucanusetheOpsManagerAPItocustomizethesizeandtypesofthemasterandworkernodeVMs.Formoreinformation,seeHowto Create or Remove Custom VM_TYPE Template using the Operations Manager API intheKnowledgeBase.

[email protected].

warning:DonotreducethesizeofyourKubernetesmasternodeVMsbelowtherecommendedsizeslistedinMaster NodeVM Size,above.UpgradinganoverloadedKubernetesclustermasternodeVMcanresultindowntime.


https://community.pivotal.io/s/article/how-to-create-or-remove-custom-vmtype-template-using-the-ops-manager-apimailto:[email protected]

Telemetry

In this topic

OverviewParticipationLevels

ConfigureCEIPandTelemetry

BenefitsoftheEnhancedParticipationLevel

SystemComponents

DataDictionary

SampleReports

Page last updated:

ThistopicdescribestheVMwareCustomerExperienceImprovementProgram(CEIP)andthePivotalTelemetryProgram(Telemetry)usedintheEnterprisePKStile.

Overview

TheCEIPandTelemetryprogramallowVMwareandPivotaltocollectdatafromcustomerinstallationstoimproveyourEnterprisePKSexperience.CollectingdataatscaleenablesustoidentifypatternsandalertyoutowarningsignalsinyourEnterprisePKSinstallation.

ParticipationLevels

YoucanconfigureEnterprisePKStouseoneofthefollowingCEIPandTelemetryparticipationlevels:

None:Thisleveldisablesdatacollection.

Standard:(Default)Thislevelcollectsdataanonymously.YourdataisusedtoinformtheongoingdevelopmentofEnterprisePKS.

Enhanced:ThislevelenablesVMwareandPivotaltowarnyouaboutsecurityvulnerabilitiesandpotentialissueswithyoursoftwareconfigurations.Formoreinformation,seeBenefits of the Enhanced Participation Levelbelow.

ConfigureCEIPandTelemetry

Video:ForinformationaboutconfiguringCEIPandTelemetryparticipation,seetheCEIP Opt-In Walkthrough video onYouTube.

ForinformationaboutconfiguringCEIPandTelemetryparticipation,seetheCEIPandTelemetry sectionoftheinstallationtopicforyourIaaS:

Installing Enterprise PKS on vSphere

Installing Enterprise PKS on vSphere with NSX-T

Installing Enterprise PKS on AWS

Installing Enterprise PKS on Azure

Note:EnterprisePKSdoesnotcollectanypersonallyidentifiableinformation(PII)ateitherparticipationlevel.ForalistofthedataEnterprisePKScollects,seeData Dictionary.


https://www.youtube.com/embed/RTyq9ODUbU4

Installing Enterprise PKS on GCP

BenefitsoftheEnhancedParticipationLevel

BenefitsyoureceivewiththeEnhancedparticipationlevelincludebutarenotlimitedtothefollowing:

Usage data:ThisgivesyouaccesstodataaboutKubernetespodandclusterusageinyourEnterprisePKSinstallation.Seesample reportsbelowformoredetails.

Access to your telemetry data:ThisgivesyouaccesstoconfigurationandusagedataaboutyourEnterprisePKSinstallation.Seesample reportsbelowformoredetails.

Proactive support:ThisenablesVMwareandPivotaltoproactivelywarnyouaboutunhealthypatterns.

Benchmarks:ThisisyourusagerelativetotherestoftheEnterprisePKSuserbase.

ThetablebelowcomparestheStandardandEnhancedparticipationlevels.

Benefit Standard Level Enhanced Level

Usagedata Rawdata Reportsandtrendanalysis

Accesstoyourtelemetrydata No Yes

Proactivesupport No Yes

Benchmarks No Yes

SystemComponents

TheCEIPandTelemetryprogramsusethefollowingcomponentstocollectdata:

Telemetry Server:ThiscomponentrunsonthePKScontrolplane.TheserverreceivestelemetryeventsfromthePKSAPIandmetricsfromTelemetryagentpods.Theserversendseventsandmetricstoadatalakeforarchivingandanalysis.

Telemetry Agent Pod:ThiscomponentrunsineachKubernetesclusterasadeploymentwithonereplica.AgentpodsperiodicallypolltheKubernetesAPIforclustermetricsandsendthemetricstotheTelemetryserver.

Thefollowingdiagramshowshowtelemetrydataflowsthroughthesystemcomponents:

Note:VMwarereservestherighttochangethebenefitsassociatedwiththeEnhancedparticipationlevelatanytime.


DataDictionary

ForinformationaboutPKSTelemetrycollectionandreporting,seethePKS Telemetry Data spreadsheet,hostedonGoogleDrive.

SampleReports

Video:SeetheSample Report: Create Cluster Duration videoonYouTube.

YoucanviewtheinteractiveversionoftheSample Workbook withTableau Reader (freetouse).Clickonthelinksbelowtoseestaticscreenshotsofthereports.

1. Consumption :AsanOperatorofPKS,IneedawaytomonitorpodconsumptionacrossmyPKSenvironmentsovertime,soIcan:

SeewhichenvironmentsandclustersgettheheaviestuseSeetemporalpatternsinpodconsumptionScalecapacityaccordinglyShowandchargebackusersofPKSwithinmyorganization

2. API heartbeats + Cluster heartbeats :AsanOperatorofPKSIneedawaytoseetheversionofPKSeachofmyenvironmentswasrunningovertime,soIcan:

KeeptrackofallmyPKSenvironmentsandclustersIdentifyenvironmentsandclustersinneedofupgrading

3. Cluster creation events :AsanOperatorofPKSIwanttoseehowoftenclustercreationsucceedsacrossmyPKSenvironments,soIcan:

Identifyenvironmentsthatencounterrepeatedfailuresanddebugorinterveneasappropriatetoavoidfrustrationforclusteradminsandusers


file:///Users/pspinrad/workspace/pdfer/html/docs-pcf-staging.cfapps.io/pks/1-6/images/telemetry-data-flow.pnghttps://drive.google.com/open?id=18UCd1kbhR3xV_XOl6KcEU64GI6ySdkRa3iG_8QAROl8#gid=1858241440https://www.youtube.com/embed/Q41g7uWBvhAhttp://bit.ly/sampleworkbookhttps://www.tableau.com/products/readerhttp://bit.ly/consumptionreporthttp://bit.ly/apiheartbeatshttp://bit.ly/clusterheartbeatshttp://bit.ly/clustercreate

4. Cluster creation duration :AsanOperatorofPKSIwanttoseehowlongittakestocreateclusters,soIcan:

Intervenewhenclustercreationsignificantlymoretimethanexpected,andadjustmyplanandnetworkconfigurationasappropriate

5. Cluster creation errors :AsanOperatorofPKS,IwanttoseewhaterrorsarebeingencounteredmostfrequentlyduringclustercreationsoIcan:

Quicklyidentifywidespreadproblemsandremediate(e.g.NSXerrors)

6. Container images :AsanOperatorofPKS,IwanttoseewhichcontainerimagesareinuseacrossmyPKSinstallationssoIcan:

ConductanauditofcontainerimagesandidentifyprohibitedorproblematicimagesInferwhichworkloadsarerunningonPKS,toinformmyplanning,resourcing,andoutreach

[email protected].


http://bit.ly/createdurationhttp://bit.ly/createerrorshttp://bit.ly/containerimagesmailto:[email protected]

InstallingEnterprisePKSPage last updated:

EnterprisePKSManagementConsole(vSphereOnly)

SeethefollowingdocumentationfortheEnterprisePKSManagementConsole,whichdeploysEnterprisePKSasavirtualapplianceonvSpherewithoutPivotalPlatform:

Enterprise PKS Management Console (vSphere Only)

VMwareEnterprisePKSonPivotalPlatform

SeethefollowingdocumentationforhowtoinstallVMwareEnterprisePKSonPivotalPlatform:

vSphere with Flannel

vSphere with NSX-T

GCP

AWS

Azure

[email protected].

Note:EnterprisePKSsupportsair-gappeddeploymentsonvSpherewithorwithoutNSX-Tintegration.



vSpherePage last updated:

ThistopicliststheprocedurestofollowtoinstallVMwareEnterprisePKSonvSphere.

InstallEnterprisePKSonvSphere

ToinstallEnterprisePKSonvSpherewithoutNSX-T,followtheinstructionsbelow:

Prerequisites and Resource Requirements

Firewall Ports and Protocols Requirements for vSphere without NSX-T

Creating Dedicated Users and Roles for vSphere (Optional)

Installing and Configuring Ops Manager on vSphere

Installing Enterprise PKS on vSphere

Configuring PKS API Load Balancer

Setting Up Enterprise PKS Admin Users on vSphere

(Optional) Integrating VMware Harbor with Enterprise PKS

InstallthePKSandKubernetesCLIs

ThePKSCLIandKubernetesCLIhelpyouinteractwithyourEnterprisePKS-provisionedKubernetesclustersandKubernetesworkloads.

ToinstalltheCLIs,followtheinstructionsbelow:

Installing the PKS CLI

Installing the Kubernetes CLI

[email protected].

Note:VMwareHarborisanenterprise-classregistryserverforcontainerimages.Formoreinformation,seeVMware HarborRegistry inthePivotalPartnerdocumentation.


https://docs.pivotal.io/partners/vmware-harbor/integrating-pks.htmlhttps://docs.pivotal.io/partners/vmware-harbor/index.htmlmailto:[email protected]

vSpherePrerequisitesandResourceRequirements

In this topic

Prerequisites


ResourceRequirements

NetworkCommunicationRequirements

Page last updated:

ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingVMwareEnterprisePKSonvSphere.

ForprerequisitesandresourcerequirementsforinstallingEnterprisePKSonvSpherewithNSX-Tintegration,seevSphere with NSX-T Version RequirementsandHardware Requirements for Enterprise PKS on vSphere with NSX-T.

Prerequisites

BeforeinstallingEnterprisePKS:

1. ReviewthesectionsbelowandtheinstructionsinCreating Dedicated Users and Roles for vSphere (Optional).

2. InstallandconfigureOpsManager.ToinstallOpsManager,followtheinstructionsinInstalling and Configuring Ops Manageron vSphere.


ForEnterprisePKSonvSphereversionrequirements,refertotheVMware Product Interoperability Matrices .

ResourceRequirements

InstallingOpsManagerandEnterprisePKSrequiresthefollowingvirtualmachines(VMs):

VM CPU RAM Storage

PivotalContainerService 2 8GB 16GB

PivotalOpsManager 1 8GB 160GB

BOSHDirector 2 8GB 16GB

StorageRequirementsforLargeNumbersofPods

Ifyouexpecttheclusterworkloadtorunalargenumberofpodscontinuously,thenincreasethesizeofpersistentdiskstorageallocatedtothePivotalContainerServiceVMasfollows:

Number of Pods Storage (Persistent Disk) Requirements

1,000pods 20GB


https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&356=&175=&1=

5,000pods 100GB

10,000pods 200GB

50,000pods 1,000GB

Number of Pods Storage (Persistent Disk) Requirements

EphemeralVMResources

EachEnterprisePKSdeploymentrequiresephemeralVMsduringinstallationandupgradesofEnterprisePKS.AfteryoudeployEnterprisePKS,BOSHautomaticallydeletestheseVMs.ToenableEnterprisePKStodynamicallycreatetheephemeralVMswhenneeded,ensurethatthefollowingresourcesareavailableinyourvSphereinfrastructurebeforedeployingEnterprisePKS:

Ephemeral VM Number CPU Cores RAM Ephemeral Disk

BOSHCompilationVMs 4 4 4GB 32GB

KubernetesClusterResources

EachKubernetesclusterprovisionedthroughEnterprisePKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.

VM Number CPU Cores RAM Ephemeral Disk Persistent Disk

master 1or3 2 4GB 8GB 5GB

worker 1ormore 2 4GB 8GB 50GB

errand(ephemeral) 1 1 1GB 8GB none

NetworkCommunicationRequirements

ForacompletelistofnetworkcommunicationrequirementsforvSpherewithoutNSX-T,seeFirewall Ports and ProtocolsRequirements for vSphere without NSX-T.

[email protected].



FirewallPortsandProtocolsRequirementsforvSpherewithoutNSX-TPage last updated:

In this topic

EnterprisePKSPortsandProtocols

EnterprisePKSUsersPortsandProtocols

EnterprisePKSCorePortsandProtocols

VMwarePortsandProtocols

VMwareVirtualInfrastructurePortsandProtocols

VMwareOptionalIntegrationPortsandProtocols

ThistopicdescribesthefirewallportsandprotocolsrequirementsforusingVMwareEnterprisePKSonvSphere.

Firewallsandsecuritypoliciesareusedtofiltertrafficandlimitaccessinenvironmentswithstrictinter-networkaccesscontrolpolicies.

Appsfrequentlyrequiretheabilitytopassinternalcommunicationbetweensystemcomponentsondifferentnetworksandrequireoneormoreconduitsthroughtheenvironment’sfirewalls.FirewallrulesarealsorequiredtoenableinterfacingwithexternalsystemssuchaswithenterpriseappsorappsanddataonthepublicInternet.

ForEnterprisePKS,Pivotalrecommendsthatyoudisablesecuritypoliciesthatfiltertrafficbetweenthenetworkssupportingthesystem.WithEnterprisePKSyoushouldenableaccesstoappsthroughstandardKubernetesload-balancersandingresscontrollertypes.Thisenablesyoutodesignatespecificportsandprotocolsasafirewallconduit.

ForinformationonportsandprotocolrequirementsforvSpherewithNSX-T,seeFirewall Ports and Protocols Requirements forvSphere with NSX-T

Ifyouareunabletoimplementyoursecuritypolicyusingthemethodsdescribedabove,refertothefollowingtable,whichidentifiestheflowsbetweensystemcomponentsinatypicalEnterprisePKSdeployment.

EnterprisePKSPortsandProtocols

ThefollowingtableslistportsandprotocolsrequiredfornetworkcommunicationsbetweenEnterprisePKSv1.5.0andlater,andvSphere6.7andlater.

EnterprisePKSUsersPortsandProtocols

ThefollowingtablelistsportsandprotocolsusedfornetworkcommunicationbetweenEnterprisePKSuserinterfacecomponents.

Source Component Destination Component DestinationProtocolDestinationPort Service

Admin/OperatorConsole AllSystemComponents TCP 22 ssh

Note:Tocontrolwhichgroupsaccessdeployingandscalingyourorganization’sEnterprisePKS-deployedKubernetesclusters,configureyourfirewallsettingsasdescribedontheOperator–>PKSAPIserverlinesbelow.


Admin/OperatorConsole AllSystemComponents TCP 80 http

Admin/OperatorConsole AllSystemComponents TCP 443 https

Admin/OperatorConsole CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi

Admin/OperatorConsolePivotalCloudFoundryOperationsManager

TCP 22 ssh

Admin/OperatorConsolePivotalCloudFoundryOperationsManager

TCP 443 https

Admin/OperatorConsole PKSController TCP 9021 pksapiserver

Admin/OperatorConsole vCenterServer TCP 443 https

Admin/OperatorConsole vCenterServer TCP 5480 vami

Admin/OperatorConsolevSphereESXIHostsMgmt.vmknic

TCP 902 ideafarm-door

Admin/OperatorandDeveloperConsoles

HarborPrivateImageRegistry TCP 80 http


HarborPrivateImageRegistry TCP 443 https


HarborPrivateImageRegistry TCP 4443 notary


KubernetesAppLoad-BalancerSvc

TCP/UDP Varies varieswithapps


KubernetesClusterAPIServer-LBVIP

TCP 8443 httpsca


KubernetesClusterIngressController

TCP 80 http



TCP 443 https


KubernetesClusterWorkerNode TCP/UDP 30000-32767kubernetesnodeport


PKSController TCP 8443 httpsca

AllUserConsoles(Operator,Developer,Consumer)

KubernetesAppLoad-BalancerSvc

TCP/UDP Varies varieswithapps



TCP 80 http



TCP 443 https


KubernetesClusterWorkerNode TCP/UDP 30000-32767kubernetesnodeport


EnterprisePKSCorePortsandProtocols


ThefollowingtablelistsportsandprotocolsusedfornetworkcommunicationbetweencoreEnterprisePKScomponents.


AllSystemComponentsCorporateDomainNameServer

TCP/UDP 53 dns

AllSystemComponents NetworkTimeServer UDP 123 ntp

AllSystemComponents vRealizeLogInsight TCP/UDP 514/1514 syslog/tlssyslog

AllSystemControlPlaneComponents

AD/LDAPDirectoryServer TCP/UDP 389/636 ldap/ldaps

PivotalCloudFoundryOperationsManager

Admin/OperatorConsole TCP 22 ssh


CloudFoundryBOSHDirector TCP 6868 boshagenthttp


CloudFoundryBOSHDirector TCP 8443 httpsca


CloudFoundryBOSHDirector TCP 8844 credhub


CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi


HarborPrivateImageRegistry TCP 22 ssh


KubernetesClusterMaster/EtcdNode

TCP 22 ssh


KubernetesClusterWorkerNode

TCP 22 ssh


PKSController TCP 22 ssh


PKSController TCP 8443 httpsca


vCenterServer TCP 443 https


vSphereESXIHostsMgmt.vmknic

TCP 443 https

CloudFoundryBOSHDirector vCenterServer TCP 443 https

CloudFoundryBOSHDirectorvSphereESXIHostsMgmt.vmknic

TCP 443 https

BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 4222 boshnatsserver

BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 25250 boshblobstore

BOSHCompilationJobVM CloudFoundryBOSHDirector TCP 25923healthmonitordaemon

BOSHCompilationJobVM HarborPrivateImageRegistry TCP 443 https

BOSHCompilationJobVM HarborPrivateImageRegistry TCP 8853 boshdnshealth


PKSController CloudFoundryBOSHDirector TCP 4222 boshnatsserver

PKSController CloudFoundryBOSHDirector TCP 8443 httpsca

PKSController CloudFoundryBOSHDirector TCP 25250 boshblobstore

PKSController CloudFoundryBOSHDirector TCP 25555boshdirectorrestapi

PKSController CloudFoundryBOSHDirector TCP 25923healthmonitordaemon

PKSControllerKubernetesClusterMaster/EtcdNode TCP 8443 httpsca

PKSController vCenterServer TCP 443 https

HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 4222 boshnatsserver

HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 25250 boshblobstore

HarborPrivateImageRegistry CloudFoundryBOSHDirector TCP 25923healthmonitordaemon

HarborPrivateImageRegistry IPNASStorageArray TCP 111 nfsrpcportmapper

HarborPrivateImageRegistry IPNASStorageArray TCP 2049 nfs

HarborPrivateImageRegistry PublicCVESourceDatabase TCP 443 https

kube-systempod/telemetry-agent PKSController TCP 24224fluentdout_forward


CloudFoundryBOSHDirector TCP 4222 boshnatsserver


CloudFoundryBOSHDirector TCP 25250 boshblobstore


CloudFoundryBOSHDirector TCP 25923healthmonitordaemon



TCP 2379 etcdclent



TCP 2380 etcdserver



TCP 8443 httpsca



TCP 8853 boshdnshealth

pivotal container service (pks) v1 · 1. deploying ops manager on vsphere 2. configuring bosh...

Documents