gfs/hdfs 15-440 distributed systems. 2 building blocks google workqueue (scheduler) google file...
TRANSCRIPT
![Page 1: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/1.jpg)
GFS/HDFS
15-440 Distributed Systems
![Page 2: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/2.jpg)
2
Building Blocks
• Google WorkQueue (scheduler)• Google File System• Chubby Lock service (paxos-based)• Two other pieces helpful but not required
• Sawzall (languate)• MapReduce
• BigTable: Build a more application-friendly storage service using these parts
![Page 3: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/3.jpg)
Overview
• Google File System (GFS) and Hadoop Distributed File System (HDFS)
• BigTable
3
![Page 4: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/4.jpg)
Google Disk Farm
Early days…
…today
4
![Page 5: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/5.jpg)
Google Platform Characteristics
• Lots of cheap PCs, each with disk and CPU• High aggregate storage capacity• Spread search processing across many CPUs
• How to share data among PCs?
5
![Page 6: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/6.jpg)
Google Platform Characteristics
• 100s to 1000s of PCs in cluster• Many modes of failure for each PC:
• App bugs, OS bugs• Human error• Disk failure, memory failure, net failure, power supply
failure• Connector failure
• Monitoring, fault tolerance, auto-recovery essential
6
![Page 7: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/7.jpg)
7
![Page 8: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/8.jpg)
Data-Center Network
![Page 9: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/9.jpg)
![Page 10: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/10.jpg)
![Page 11: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/11.jpg)
![Page 12: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/12.jpg)
![Page 13: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/13.jpg)
![Page 14: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/14.jpg)
![Page 15: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/15.jpg)
![Page 16: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/16.jpg)
![Page 17: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/17.jpg)
![Page 18: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/18.jpg)
![Page 19: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/19.jpg)
![Page 20: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/20.jpg)
![Page 21: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/21.jpg)
![Page 22: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/22.jpg)
![Page 23: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/23.jpg)
![Page 24: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/24.jpg)
![Page 25: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/25.jpg)
![Page 26: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/26.jpg)
![Page 27: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/27.jpg)
![Page 28: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/28.jpg)
![Page 29: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/29.jpg)
![Page 30: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/30.jpg)
![Page 31: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/31.jpg)
![Page 32: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/32.jpg)
![Page 33: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/33.jpg)
![Page 34: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/34.jpg)
![Page 35: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/35.jpg)
![Page 36: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/36.jpg)
![Page 37: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/37.jpg)
![Page 38: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/38.jpg)
![Page 39: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/39.jpg)
![Page 40: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/40.jpg)
![Page 41: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/41.jpg)
![Page 42: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/42.jpg)
![Page 43: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/43.jpg)
![Page 44: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/44.jpg)
![Page 45: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/45.jpg)
![Page 46: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/46.jpg)
![Page 47: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/47.jpg)
![Page 48: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/48.jpg)
![Page 49: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/49.jpg)
![Page 50: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/50.jpg)
![Page 51: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/51.jpg)
![Page 52: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/52.jpg)
![Page 53: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/53.jpg)
![Page 54: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/54.jpg)
![Page 55: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/55.jpg)
![Page 56: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/56.jpg)
![Page 57: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/57.jpg)
![Page 58: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/58.jpg)
![Page 59: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/59.jpg)
![Page 60: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/60.jpg)
![Page 61: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/61.jpg)
![Page 62: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/62.jpg)
![Page 63: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/63.jpg)
![Page 64: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/64.jpg)
![Page 65: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/65.jpg)
![Page 66: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/66.jpg)
![Page 67: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/67.jpg)
![Page 68: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/68.jpg)
![Page 69: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces](https://reader035.vdocuments.mx/reader035/viewer/2022062807/5697bff61a28abf838cbde55/html5/thumbnails/69.jpg)