massively parallel postgres kubernetes operator for...gemfire spark object storage hdfs json, apache...
TRANSCRIPT
© Copyright 2018 Pivotal Software, Inc. All rights Reserved.
Kubernetes Operator for Massively Parallel Postgres
Goutam Tadi (@goutamtadi)Senior Software Engineer, Pivotal Software IncEmail: [email protected]
PGConf India 2019
Agenda
● Intro to Greenplum
● Kubernetes 101
● Greenplum for Kubernetes
○ Components
■ Greenplum Operator
■ Greenplum Cluster
● Demo
Massively Parallel Postgres
Greenplum
Greenplum Data Platform
ANALYTICALAPPLICATIONS
NATIVE INTERFACES
PIVOTALGREENPLUMPLATFORM
MULTI-STRUCTURED DATA
SOURCES &PIPELINES
Structured Data
JDBC, ODBC
SQL
ANSI SQL
FLEXIBLE DEPLOYMENT
LocalStorage
OtherRDBMSes
SparkGemFireCloudObject
StorageHDFS
JSON, Apache AVRO, Apache Parquet and XML
Teradata SQL
Other DB SQL
Apache MADlib
ML/Statistics/Graph
Python. R,Java, Perl, C
Programmatic
Apache SOLR
Text
PostGIS
GeoSpatial
Custom Apps BI / Reporting Machine Learning AI
On-Premises
NEXT GENERATION
DATA PLATFORM
KafkaETLSpringCloud
Data Flow
MassivelyParallel(MPP)
PostgreSQLKernel
PetabyteScale
Loading
QueryOptimizer(GPORCA)
WorkloadManager
PolymorphicStorage
Command Center
SQL Compatibility
(Hyper-Q)
DS AnalystsIT Dev
PublicClouds
PrivateClouds
FullyManaged
Clouds
5
Container Orchestration System
Kubernetes
Greenplum on Kubernetes 101
Kubernetes Master
Greenplum on Kubernetes 101
Kubernetes Master
kubelet kube-proxy docker
Node
kubelet kube-proxy docker
Node
Greenplum on Kubernetes 101
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
Greenplum on Kubernetes 101
Kubernetes Master
Pod
kubelet kube-proxy docker
Node
Pod
kubelet kube-proxy docker
Node
Storage volumes
Greenplum on Kubernetes 101
Kubernetes Master
Pod
Postgres Container
kubelet kube-proxy docker
Node
Pod
Postgres Container
kubelet kube-proxy docker
Node
Storage volumes
Greenplum on Kubernetes 101
Kubernetes MasterGreenplum Service
Pod
Postgres container
kubelet kube-proxy docker
Node
Pod
Postgres Container
kubelet kube-proxy docker
Node
Storage volumes
Greenplum on Kubernetes
Node
Pod
segment-b-0
kubelet kube-proxy docker
Pod
segment-a-0
kubelet kube-proxy docker
Node
Storage volumes
Pod
master-0
kubelet kube-proxy docker
Pod
master-1
kubelet kube-proxy docker
Massively Parallel Postgres on Kubernetes
Greenplum for Kubernetes
Components ● Greenplum Operator
● Greenplum Cluster
Greenplum Operator
apiVersion: "greenplum.pivotal.io/v1"kind: "GreenplumCluster"metadata: name: my-greenplumspec: masterAndStandby: …. Segments: ….
Greenplum Operator
Greenplum Cluster
CREATE / UPDATE / DELETE
Greenplum Cluster
Namespace: defaultKubernetes Cluster
Master StatefulSet
master-0 master-1
Greenplum Cluster
Namespace: defaultKubernetes Cluster
Primary StatefulSet
segment-a-0 segment-a-1
Mirror StatefulSet
segment-b-0 segment-b-1 Master StatefulSet
master-0 master-1
Greenplum Cluster
Namespace: defaultKubernetes Cluster
Primary StatefulSet
segment-a-0 segment-a-1
Mirror StatefulSet
segment-b-0 segment-b-1
ConfigMap
Master StatefulSet
master-0 master-1
Greenplum Cluster
Namespace: defaultKubernetes Cluster
Primary StatefulSet
segment-a-0 segment-a-1
Mirror StatefulSet
segment-b-0 segment-b-1
ConfigMap
Master StatefulSet
master-0 master-1
Greenplum Cluster
Namespace: defaultKubernetes Cluster
Primary StatefulSet
segment-a-0 segment-a-1
Mirror StatefulSet
segment-b-0 segment-b-1
ConfigMap
Master StatefulSet
master-0 master-1
Psql query
Benefits
● Declarative style deployments
● Auto cluster initialization
● Quick and fast deployments
● Easy to expand
● Delete the compute and retain storage for later use
Demohttps://youtu.be/d8X2BXSg07Q
Install Greenplum OperatorInstall Greenplum Cluster
Failover ScenariosExpand Greenplum Cluster
Future
Auto FailoverAuto Cluster Rejoin
PVC SnapshotPod Affinity
Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.25
© Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
Resources
http://greenplum-kubernetes.docs.pivotal.iohttps://network.pivotal.io