when tidb meets kubernetesjun 26, 2017 · kubernetes is the os for the datacenter, but on the...
TRANSCRIPT
![Page 1: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/1.jpg)
When TiDB meets Kubernetes
Dongxu Huang @ PingCAP
![Page 2: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/2.jpg)
About me
● Dongxu Huang, Co-founder & CTO, PingCAP● Infrastructure engineer / Hacker / Open source enthusiast ● Go / Rust / Python● Codis / TiDB / TiKV
![Page 3: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/3.jpg)
Agenda
● The problem we meet● Brief introduction of Kubernetes / TiDB● Operator saves the day● Live without PersistentVolume● The future
![Page 4: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/4.jpg)
Cloud is the future. But database maintenance still sucks.
![Page 5: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/5.jpg)
![Page 6: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/6.jpg)
We were told everything would be scalable, easily.But operating it makes it even harder.
![Page 7: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/7.jpg)
Operating a single-node system
...A P2P distributed system
![Page 8: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/8.jpg)
Services & Components
Unstable network infrastructure
![Page 9: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/9.jpg)
![Page 10: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/10.jpg)
A brief introduction of Kubernetes
● Container-centric cluster management● Service orchestrator● Optimize use of hardware by using only the resources you need● Auto deployment / Auto scaling / Auto healing
![Page 11: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/11.jpg)
A brief introduction of TiDB
● SQL is necessary● Transparent sharding and data movement● 100% OLTP + 80% OLAP
○ Transaction + Complex query
● Compatible with MySQL, at most cases● 24/7 availability, even in case of datacenter outages
○ Thanks to Raft consensus algorithm
● Open source, of course.
![Page 12: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/12.jpg)
A brief introduction of TiDB
TiKV TiKV TiKV TiKV
Raft Raft Raft
TiDB TiDB TiDB
... ......
... ...
Placement Driver (PD)
Control flow:Balance / Failover
Metadata / Timestamp request
Stateless SQL Layer
Distributed Storage Layer
gRPC
gRPC
gRPC
![Page 13: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/13.jpg)
The problem
● It’s easy for stateless applications, but how about stateful?○ Databases: MySQL / PG / TiDB○ Coordination: Etcd / ZooKeeper○ Streaming: Kafka○ Big data: Hadoop / Ceph / GlusterFS○ Search: ElasticSearch○ …
● Or even kubernetes itself.
![Page 14: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/14.jpg)
What is the hard part?
Domain knowledge of the distributed system.
![Page 15: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/15.jpg)
For example,if you want to operate a redis cluster well, you must be
a redis expert.
![Page 16: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/16.jpg)
Operator saves the day
An Operator is software that encodes domain knowledge and extends the Kubernetes API through the third party resources mechanism, enabling users to create, configure, and manage applications.
--- CoreOS
![Page 17: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/17.jpg)
Operator
Deploy
Resize
UpgradeBackup / Restore
Failover
Monitoring
![Page 18: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/18.jpg)
‘Self-driving’ mode
Observation
Act Analysis
![Page 19: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/19.jpg)
TiDB-Operator
● Create● Rolling update● Scale out● Failover● Backup/Restore
Kubernetes
TPR
TiDB-Operator
![Page 20: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/20.jpg)
TiDB-Operator
User1 User2
Load balancer
Kubernetes
Load balancer
![Page 21: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/21.jpg)
The hardest part: Local storage resource
![Page 22: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/22.jpg)
![Page 23: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/23.jpg)
Chill out, bro...
Try to create a TPR to manage local storage resources
![Page 24: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/24.jpg)
Under the hood
1. Create a ConfigMap: tidb-storage
nodes: - name: "172.17.4.101" directories: - "/tikv-storage-dir-1" - "/tikv-storage-dir-2" - name: "172.17.4.102" directories: - "/tikv-storage-dir-3" - "/tikv-storage-dir-4"
![Page 25: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/25.jpg)
2. Create a TPR: tidb-volume.pingcap.com/v1, Like:
name: "172-14-4-101-tikv-dir1"state: "binded"podName: "tikv-1"
3. Create a controller: volume-controller, notify configuration change of tidb-storage, generate tidb-volume resources
Under the hood
![Page 26: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/26.jpg)
Under the hood
4. Add storage attribute to tidb-operator, so that tidb-operator would assign local storage resource to specific tikv instance
…storage: - "172.17.4.101:/tikv-storage-dir-1" - "172.17.4.102:/tikv-storage-dir-2" - "172.17.4.103:/tikv-storage-dir-4"…
5. Add a DaemonSet: tidb-storage-ds, maintain the lifetime of hostPath, when a tikv instance is offline, tidb-storage-ds would reclaim the storage resource.
![Page 27: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/27.jpg)
Open source....coming soon
![Page 28: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/28.jpg)
The future
● ‘Self-driving mode’ for everything○ Circuit breaker and MT is still important, as lifesaver.
● DB as a Service / Serverless● Local storage isn’t necessary, maybe
○ Or another way, in-storage computing○ Or maybe, both
![Page 29: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/29.jpg)
Wrap it up
● Distributed system operation matters● Kubernetes is the OS for the datacenter, but on the storage side,
things become complicated● Operator builds the bridge between domain knowledge and
kubernetes, it’s kindof batch script for DCOS.● TiDB-operator provides the ability to set up/manage large cluster
○ We solve the local storage problem, little hacky, but it works
![Page 30: When TiDB meets KubernetesJun 26, 2017 · Kubernetes is the OS for the datacenter, but on the storage side, things become complicated Operator builds the bridge between domain knowledge](https://reader033.vdocuments.mx/reader033/viewer/2022052104/603fc0d7794fce2d877ee2d6/html5/thumbnails/30.jpg)