introduction to cloud computing jiaheng lu department of computer science renmin university of china
TRANSCRIPT
![Page 1: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/1.jpg)
Introduction to cloud computing
Jiaheng Lu
Department of Computer Science
Renmin University of Chinawww.jiahenglu.net
![Page 2: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/2.jpg)
Cloud computing
![Page 3: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/3.jpg)
![Page 4: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/4.jpg)
Google Cloud computing techniques
![Page 5: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/5.jpg)
The Google File System
![Page 6: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/6.jpg)
The Google File System(GFS)
A scalable distributed file system for large distributed data intensive applications
Multiple GFS clusters are currently deployed.
The largest ones have:1000+ storage nodes
300+ TeraBytes of disk storage
heavily accessed by hundreds of clients on distinct machines
![Page 7: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/7.jpg)
Introduction
Shares many same goals as previous distributed file systems
performance, scalability, reliability, etc
GFS design has been driven by four key observation of Google application workloads and technological environment
![Page 8: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/8.jpg)
Intro: Observations 1
1. Component failures are the normconstant monitoring, error detection, fault tolerance and automatic recovery are integral to the system
2. Huge files (by traditional standards)Multi GB files are common
I/O operations and blocks sizes must be revisited
![Page 9: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/9.jpg)
Intro: Observations 2
3. Most files are mutated by appending new data
This is the focus of performance optimization and atomicity guarantees
4. Co-designing the applications and APIs benefits overall system by increasing flexibility
![Page 10: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/10.jpg)
The Design
Cluster consists of a single master and multiple chunkservers and is accessed by multiple clients
![Page 11: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/11.jpg)
The Master
Maintains all file system metadata.names space, access control info, file to chunk mappings, chunk (including replicas) location, etc.
Periodically communicates with chunkservers in HeartBeat messages to give instructions and check state
![Page 12: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/12.jpg)
The Master
Helps make sophisticated chunk placement and replication decision, using global knowledge
For reading and writing, client contacts Master to get chunk locations, then deals directly with chunkservers
Master is not a bottleneck for reads/writes
![Page 13: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/13.jpg)
Chunkservers
Files are broken into chunks. Each chunk has a immutable globally unique 64-bit chunk-handle.
handle is assigned by the master at chunk creation
Chunk size is 64 MB
Each chunk is replicated on 3 (default) servers
![Page 14: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/14.jpg)
Clients
Linked to apps using the file system API.
Communicates with master and chunkservers for reading and writing
Master interactions only for metadata
Chunkserver interactions for data
Only caches metadata informationData is too large to cache.
![Page 15: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/15.jpg)
Chunk Locations
Master does not keep a persistent record of locations of chunks and replicas.
Polls chunkservers at startup, and when new chunkservers join/leave for this.
Stays up to date by controlling placement of new chunks and through HeartBeat messages (when monitoring chunkservers)
![Page 16: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/16.jpg)
Operation Log
Record of all critical metadata changes
Stored on Master and replicated on other machines
Defines order of concurrent operations
Changes not visible to clients until they propigate to all chunk replicas
Also used to recover the file system state
![Page 17: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/17.jpg)
System Interactions: Leases and Mutation Order
Leases maintain a mutation order across all chunk replicas
Master grants a lease to a replica, called the primary
The primary choses the serial mutation order, and all replicas follow this order
Minimizes management overhead for the Master
![Page 18: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/18.jpg)
System Interactions: Leases and Mutation Order
![Page 19: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/19.jpg)
Atomic Record Append
Client specifies the data to write; GFS chooses and returns the offset it writes to and appends the data to each replica at least once
Heavily used by Google’s Distributed applications.
No need for a distributed lock managerGFS choses the offset, not the client
![Page 20: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/20.jpg)
Atomic Record Append: How?
• Follows similar control flow as mutations
• Primary tells secondary replicas to append at the same offset as the primary
• If a replica append fails at any replica, it is retried by the client.
So replicas of the same chunk may contain different data, including duplicates, whole or in part, of the same record
![Page 21: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/21.jpg)
Atomic Record Append: How?
• GFS does not guarantee that all replicas are bitwise identical.
Only guarantees that data is written at least once in an atomic unit.
Data must be written at the same offset for all chunk replicas for success to be reported.
![Page 22: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/22.jpg)
Replica Placement
Placement policy maximizes data reliability and network bandwidth
Spread replicas not only across machines, but also across racks
Guards against machine failures, and racks getting damaged or going offline
Reads for a chunk exploit aggregate bandwidth of multiple racks
Writes have to flow through multiple rackstradeoff made willingly
![Page 23: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/23.jpg)
Chunk creation
created and placed by master.
placed on chunkservers with below average disk utilization
limit number of recent “creations” on a chunkserver
with creations comes lots of writes
![Page 24: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/24.jpg)
Detecting Stale Replicas
• Master has a chunk version number to distinguish up to date and stale replicas
• Increase version when granting a lease
• If a replica is not available, its version is not increased
• master detects stale replicas when a chunkservers report chunks and versions
• Remove stale replicas during garbage collection
![Page 25: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/25.jpg)
Garbage collection
When a client deletes a file, master logs it like other changes and changes filename to a hidden file.
Master removes files hidden for longer than 3 days when scanning file system name space
metadata is also erased
During HeartBeat messages, the chunkservers send the master a subset of its chunks, and the master tells it which files have no metadata.
Chunkserver removes these files on its own
![Page 26: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/26.jpg)
Fault Tolerance:High Availability
• Fast recoveryMaster and chunkservers can restart in seconds
• Chunk Replication
• Master Replication“shadow” masters provide read-only access when primary master is down
mutations not done until recorded on all master replicas
![Page 27: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/27.jpg)
Fault Tolerance:Data Integrity
Chunkservers use checksums to detect corrupt data
Since replicas are not bitwise identical, chunkservers maintain their own checksums
For reads, chunkserver verifies checksum before sending chunk
Update checksums during writes
![Page 28: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/28.jpg)
Introduction to Introduction to MapReduce MapReduce
![Page 29: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/29.jpg)
MapReduce: InsightMapReduce: Insight
”Consider the problem of counting the
number of occurrences of each word in a large collection of documents”
How would you do it in parallel ?
![Page 30: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/30.jpg)
MapReduce Programming ModelMapReduce Programming Model Inspired from map and reduce operations
commonly used in functional programming languages like Lisp.
Users implement interface of two primary methods:1. Map: (key1, val1) → (key2, val2)2. Reduce: (key2, [val2]) → [val3]
![Page 31: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/31.jpg)
Map operationMap operation Map, a pure function, written by the user, takes
an input key/value pair and produces a set of intermediate key/value pairs. e.g. (doc—id, doc-content)
Draw an analogy to SQL, map can be visualized as group-by clause of an aggregate query.
![Page 32: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/32.jpg)
Reduce operationReduce operation On completion of map phase, all the
intermediate values for a given output key are combined together into a list and given to a reducer.
Can be visualized as aggregate function (e.g., average) that is computed over all the rows with the same group-by attribute.
![Page 33: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/33.jpg)
Pseudo-codePseudo-codemap(String input_key, String input_value): // input_key: document name // input_value: document contents
for each word w in input_value: EmitIntermediate(w, "1");
reduce(String output_key, Iterator intermediate_values): // output_key: a word // output_values: a list of counts
int result = 0; for each v in intermediate_values:
result += ParseInt(v); Emit(AsString(result));
![Page 34: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/34.jpg)
MapReduce: Execution overviewMapReduce: Execution overview
![Page 35: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/35.jpg)
MapReduce: ExampleMapReduce: Example
![Page 36: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/36.jpg)
MapReduce in Parallel: ExampleMapReduce in Parallel: Example
![Page 37: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/37.jpg)
MapReduce: Fault ToleranceMapReduce: Fault Tolerance Handled via re-execution of tasks.
Task completion committed through master
What happens if Mapper fails ? Re-execute completed + in-progress map tasks
What happens if Reducer fails ? Re-execute in progress reduce tasks
What happens if Master fails ? Potential trouble !!
![Page 38: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/38.jpg)
MapReduce: MapReduce:
Walk through of One more Application
![Page 39: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/39.jpg)
![Page 40: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/40.jpg)
MapReduce : PageRankMapReduce : PageRank
PageRank models the behavior of a “random surfer”.
C(t) is the out-degree of t, and (1-d) is a damping factor (random jump)
The “random surfer” keeps clicking on successive links at random not taking content into consideration.
Distributes its pages rank equally among all pages it links to.
The dampening factor takes the surfer “getting bored” and typing arbitrary URL.
n
i i
i
tC
tPRddxPR
1 )(
)()1()(
![Page 41: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/41.jpg)
PageRank : Key InsightsPageRank : Key Insights
Effects at each iteration is local. i+1th iteration
depends only on ith iteration
At iteration i, PageRank for individual nodes can be computed independently
![Page 42: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/42.jpg)
PageRank using MapReducePageRank using MapReduce
Use Sparse matrix representation (M)
Map each row of M to a list of PageRank “credit” to assign to out link neighbours.
These prestige scores are reduced to a single PageRank value for a page by aggregating over them.
![Page 43: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/43.jpg)
PageRank using MapReducePageRank using MapReduceMap: distribute PageRank “credit” to link targets
Reduce: gather up PageRank “credit” from multiple sources to compute new PageRank value
Iterate untilconvergence
Source of Image: Lin 2008
![Page 44: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/44.jpg)
Phase 1: Phase 1: Process HTMLProcess HTML
Map task takes (URL, page-content) pairs
and maps them to (URL, (PRinit
, list-of-urls))PRinit is the “seed” PageRank for URLlist-of-urls contains all pages pointed to by URL
Reduce task is just the identity function
![Page 45: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/45.jpg)
Phase 2: Phase 2: PageRank DistributionPageRank Distribution
Reduce task gets (URL, url_list) and many
(URL, val) valuesSum vals and fix up with d to get new PREmit (URL, (new_rank, url_list))
Check for convergence using non parallel component
![Page 46: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/46.jpg)
MapReduce: Some More AppsMapReduce: Some More Apps
Distributed Grep.
Count of URL Access Frequency.
Clustering (K-means)
Graph Algorithms.
Indexing Systems
MapReduce Programs In Google Source Tree
![Page 47: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/47.jpg)
MapReduce: Extensions and MapReduce: Extensions and similar appssimilar apps PIG (Yahoo)
Hadoop (Apache)
DryadLinq (Microsoft)
![Page 48: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/48.jpg)
Large Scale Systems Architecture using Large Scale Systems Architecture using MapReduceMapReduce
![Page 49: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/49.jpg)
BigTable: A Distributed Storage System for Structured Data
![Page 50: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/50.jpg)
Introduction
BigTable is a distributed storage system for managing structured data.
Designed to scale to a very large size Petabytes of data across thousands of servers
Used for many Google projects Web indexing, Personalized Search, Google Earth,
Google Analytics, Google Finance, … Flexible, high-performance solution for all of
Google’s products
![Page 51: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/51.jpg)
Motivation Lots of (semi-)structured data at Google
URLs: Contents, crawl metadata, links, anchors, pagerank, …
Per-user data: User preference settings, recent queries/search results, …
Geographic locations: Physical entities (shops, restaurants, etc.), roads, satellite
image data, user annotations, … Scale is large
Billions of URLs, many versions/page (~20K/version) Hundreds of millions of users, thousands or q/sec 100TB+ of satellite image data
![Page 52: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/52.jpg)
Why not just use commercial DB?
Scale is too large for most commercial databases
Even if it weren’t, cost would be very high Building internally means system can be applied
across many projects for low incremental cost Low-level storage optimizations help
performance significantly Much harder to do when running on top of a database
layer
![Page 53: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/53.jpg)
Goals
Want asynchronous processes to be continuously updating different pieces of data Want access to most current data at any time
Need to support: Very high read/write rates (millions of ops per second) Efficient scans over all or interesting subsets of data Efficient joins of large one-to-one and one-to-many
datasets Often want to examine data changes over time
E.g. Contents of a web page over multiple crawls
![Page 54: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/54.jpg)
BigTable
Distributed multi-level map Fault-tolerant, persistent Scalable
Thousands of servers Terabytes of in-memory data Petabyte of disk-based data Millions of reads/writes per second, efficient scans
Self-managing Servers can be added/removed dynamically Servers adjust to load imbalance
![Page 55: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/55.jpg)
Building Blocks Building blocks:
Google File System (GFS): Raw storage Scheduler: schedules jobs onto machines Lock service: distributed lock manager MapReduce: simplified large-scale data processing
BigTable uses of building blocks: GFS: stores persistent data (SSTable file format for
storage of data) Scheduler: schedules jobs involved in BigTable
serving Lock service: master election, location bootstrapping Map Reduce: often used to read/write BigTable data
![Page 56: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/56.jpg)
Basic Data Model
A BigTable is a sparse, distributed persistent multi-dimensional sorted map
(row, column, timestamp) -> cell contents
Good match for most Google applications
![Page 57: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/57.jpg)
WebTable Example
Want to keep copy of a large collection of web pages and related information
Use URLs as row keys Various aspects of web page as column names Store contents of web pages in the contents: column
under the timestamps when they were fetched.
![Page 58: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/58.jpg)
Rows
Name is an arbitrary string Access to data in a row is atomic Row creation is implicit upon storing data
Rows ordered lexicographically Rows close together lexicographically usually on
one or a small number of machines
![Page 59: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/59.jpg)
Rows (cont.)
Reads of short row ranges are efficient and typically require communication with a small number of machines.
Can exploit this property by selecting row keys so they get good locality for data access.
Example: math.gatech.edu, math.uga.edu, phys.gatech.edu, phys.uga.edu
VS
edu.gatech.math, edu.gatech.phys, edu.uga.math, edu.uga.phys
![Page 60: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/60.jpg)
Columns
Columns have two-level name structure: family:optional_qualifier
Column family Unit of access control Has associated type information
Qualifier gives unbounded columns Additional levels of indexing, if desired
![Page 61: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/61.jpg)
Timestamps
Used to store different versions of data in a cell New writes default to current time, but timestamps for writes can also be
set explicitly by clients Lookup options:
“Return most recent K values” “Return all values in timestamp range (or all values)”
Column families can be marked w/ attributes: “Only retain most recent K values in a cell” “Keep values until they are older than K seconds”
![Page 62: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/62.jpg)
Implementation – Three Major Components
Library linked into every client One master server
Responsible for: Assigning tablets to tablet servers Detecting addition and expiration of tablet servers Balancing tablet-server load Garbage collection
Many tablet servers Tablet servers handle read and write requests to its
table Splits tablets that have grown too large
![Page 63: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/63.jpg)
Implementation (cont.)
Client data doesn’t move through master server. Clients communicate directly with tablet servers for reads and writes.
Most clients never communicate with the master server, leaving it lightly loaded in practice.
![Page 64: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/64.jpg)
Tablets
Large tables broken into tablets at row boundaries Tablet holds contiguous range of rows
Clients can often choose row keys to achieve locality Aim for ~100MB to 200MB of data per tablet
Serving machine responsible for ~100 tablets Fast recovery:
100 machines each pick up 1 tablet for failed machine Fine-grained load balancing:
Migrate tablets away from overloaded machine Master makes load-balancing decisions
![Page 65: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/65.jpg)
Tablet Location
Since tablets move around from server to server, given a row, how do clients find the right machine? Need to find tablet whose row range covers the
target row
![Page 66: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/66.jpg)
Tablet Assignment Each tablet is assigned to one tablet server at
a time. Master server keeps track of the set of live
tablet servers and current assignments of tablets to servers. Also keeps track of unassigned tablets.
When a tablet is unassigned, master assigns the tablet to an tablet server with sufficient room.
![Page 67: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/67.jpg)
API Metadata operations
Create/delete tables, column families, change metadata Writes (atomic)
Set(): write cells in a row DeleteCells(): delete cells in a row DeleteRow(): delete all cells in a row
Reads Scanner: read arbitrary cells in a bigtable
Each row read is atomic Can restrict returned rows to a particular range Can ask for just data from 1 row, all rows, etc. Can ask for all columns, just certain column families, or specific
columns
![Page 68: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/68.jpg)
Refinements: Locality Groups
Can group multiple column families into a locality group Separate SSTable is created for each locality
group in each tablet. Segregating columns families that are not
typically accessed together enables more efficient reads. In WebTable, page metadata can be in one group
and contents of the page in another group.
![Page 69: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/69.jpg)
Refinements: Compression
Many opportunities for compression Similar values in the same row/column at different
timestamps Similar values in different columns Similar values across adjacent rows
Two-pass custom compressions scheme First pass: compress long common strings across a
large window Second pass: look for repetitions in small window
Speed emphasized, but good space reduction (10-to-1)
![Page 70: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/70.jpg)
Refinements: Bloom Filters
Read operation has to read from disk when desired SSTable isn’t in memory
Reduce number of accesses by specifying a Bloom filter. Allows us ask if an SSTable might contain data for a
specified row/column pair. Small amount of memory for Bloom filters drastically
reduces the number of disk seeks for read operations Use implies that most lookups for non-existent rows or
columns do not need to touch disk
![Page 71: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/71.jpg)
BigTable: A Distributed Storage System for Structured Data
![Page 72: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/72.jpg)
Introduction
BigTable is a distributed storage system for managing structured data.
Designed to scale to a very large size Petabytes of data across thousands of servers
Used for many Google projects Web indexing, Personalized Search, Google Earth,
Google Analytics, Google Finance, … Flexible, high-performance solution for all of
Google’s products
![Page 73: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/73.jpg)
Motivation Lots of (semi-)structured data at Google
URLs: Contents, crawl metadata, links, anchors, pagerank, …
Per-user data: User preference settings, recent queries/search results, …
Geographic locations: Physical entities (shops, restaurants, etc.), roads, satellite
image data, user annotations, … Scale is large
Billions of URLs, many versions/page (~20K/version) Hundreds of millions of users, thousands or q/sec 100TB+ of satellite image data
![Page 74: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/74.jpg)
Why not just use commercial DB?
Scale is too large for most commercial databases
Even if it weren’t, cost would be very high Building internally means system can be applied
across many projects for low incremental cost Low-level storage optimizations help
performance significantly Much harder to do when running on top of a database
layer
![Page 75: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/75.jpg)
Goals
Want asynchronous processes to be continuously updating different pieces of data Want access to most current data at any time
Need to support: Very high read/write rates (millions of ops per second) Efficient scans over all or interesting subsets of data Efficient joins of large one-to-one and one-to-many
datasets Often want to examine data changes over time
E.g. Contents of a web page over multiple crawls
![Page 76: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/76.jpg)
BigTable
Distributed multi-level map Fault-tolerant, persistent Scalable
Thousands of servers Terabytes of in-memory data Petabyte of disk-based data Millions of reads/writes per second, efficient scans
Self-managing Servers can be added/removed dynamically Servers adjust to load imbalance
![Page 77: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/77.jpg)
Building Blocks Building blocks:
Google File System (GFS): Raw storage Scheduler: schedules jobs onto machines Lock service: distributed lock manager MapReduce: simplified large-scale data processing
BigTable uses of building blocks: GFS: stores persistent data (SSTable file format for
storage of data) Scheduler: schedules jobs involved in BigTable
serving Lock service: master election, location bootstrapping Map Reduce: often used to read/write BigTable data
![Page 78: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/78.jpg)
Basic Data Model
A BigTable is a sparse, distributed persistent multi-dimensional sorted map
(row, column, timestamp) -> cell contents
Good match for most Google applications
![Page 79: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/79.jpg)
WebTable Example
Want to keep copy of a large collection of web pages and related information
Use URLs as row keys Various aspects of web page as column names Store contents of web pages in the contents: column
under the timestamps when they were fetched.
![Page 80: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/80.jpg)
Rows
Name is an arbitrary string Access to data in a row is atomic Row creation is implicit upon storing data
Rows ordered lexicographically Rows close together lexicographically usually on
one or a small number of machines
![Page 81: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/81.jpg)
Rows (cont.)
Reads of short row ranges are efficient and typically require communication with a small number of machines.
Can exploit this property by selecting row keys so they get good locality for data access.
Example: math.gatech.edu, math.uga.edu, phys.gatech.edu, phys.uga.edu
VS
edu.gatech.math, edu.gatech.phys, edu.uga.math, edu.uga.phys
![Page 82: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/82.jpg)
Columns
Columns have two-level name structure: family:optional_qualifier
Column family Unit of access control Has associated type information
Qualifier gives unbounded columns Additional levels of indexing, if desired
![Page 83: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/83.jpg)
Timestamps
Used to store different versions of data in a cell New writes default to current time, but timestamps for writes can also be
set explicitly by clients Lookup options:
“Return most recent K values” “Return all values in timestamp range (or all values)”
Column families can be marked w/ attributes: “Only retain most recent K values in a cell” “Keep values until they are older than K seconds”
![Page 84: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/84.jpg)
Implementation – Three Major Components
Library linked into every client One master server
Responsible for: Assigning tablets to tablet servers Detecting addition and expiration of tablet servers Balancing tablet-server load Garbage collection
Many tablet servers Tablet servers handle read and write requests to its
table Splits tablets that have grown too large
![Page 85: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/85.jpg)
Implementation (cont.)
Client data doesn’t move through master server. Clients communicate directly with tablet servers for reads and writes.
Most clients never communicate with the master server, leaving it lightly loaded in practice.
![Page 86: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/86.jpg)
Tablets
Large tables broken into tablets at row boundaries Tablet holds contiguous range of rows
Clients can often choose row keys to achieve locality Aim for ~100MB to 200MB of data per tablet
Serving machine responsible for ~100 tablets Fast recovery:
100 machines each pick up 1 tablet for failed machine Fine-grained load balancing:
Migrate tablets away from overloaded machine Master makes load-balancing decisions
![Page 87: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/87.jpg)
Tablet Location
Since tablets move around from server to server, given a row, how do clients find the right machine? Need to find tablet whose row range covers the
target row
![Page 88: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/88.jpg)
Tablet Assignment Each tablet is assigned to one tablet server at
a time. Master server keeps track of the set of live
tablet servers and current assignments of tablets to servers. Also keeps track of unassigned tablets.
When a tablet is unassigned, master assigns the tablet to an tablet server with sufficient room.
![Page 89: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/89.jpg)
API Metadata operations
Create/delete tables, column families, change metadata Writes (atomic)
Set(): write cells in a row DeleteCells(): delete cells in a row DeleteRow(): delete all cells in a row
Reads Scanner: read arbitrary cells in a bigtable
Each row read is atomic Can restrict returned rows to a particular range Can ask for just data from 1 row, all rows, etc. Can ask for all columns, just certain column families, or specific
columns
![Page 90: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/90.jpg)
Refinements: Locality Groups
Can group multiple column families into a locality group Separate SSTable is created for each locality
group in each tablet. Segregating columns families that are not
typically accessed together enables more efficient reads. In WebTable, page metadata can be in one group
and contents of the page in another group.
![Page 91: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/91.jpg)
Refinements: Compression
Many opportunities for compression Similar values in the same row/column at different
timestamps Similar values in different columns Similar values across adjacent rows
Two-pass custom compressions scheme First pass: compress long common strings across a
large window Second pass: look for repetitions in small window
Speed emphasized, but good space reduction (10-to-1)
![Page 92: Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China](https://reader035.vdocuments.mx/reader035/viewer/2022062515/56649c525503460f948fbe59/html5/thumbnails/92.jpg)
Refinements: Bloom Filters
Read operation has to read from disk when desired SSTable isn’t in memory
Reduce number of accesses by specifying a Bloom filter. Allows us ask if an SSTable might contain data for a
specified row/column pair. Small amount of memory for Bloom filters drastically
reduces the number of disk seeks for read operations Use implies that most lookups for non-existent rows or
columns do not need to touch disk