wei-chiu chuang 10/17/2013 permission to copy/distribute/adapt the work except the figures which are...
TRANSCRIPT
![Page 1: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/1.jpg)
Apache Hadoop YARN: Yet Another Resource
NegotiatorWei-Chiu Chuang
10/17/2013
Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM
![Page 2: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/2.jpg)
Master: JobTracker (JT) Worker: Tasktracker (TT)
◦ Fixed # of map slots and reduce slots
Hadoop
clientMaster Node(JobTracker)
Worker Node(TaskTracker)
Worker Node(TaskTracker)
Worker Node(TaskTracker)
![Page 3: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/3.jpg)
Hadoop is being used for all kinds of tasks beyond its original design
Tight coupling of a specific programming model with the resource management infrastructure
Centralized handling of jobs’ control flow
The Problem
![Page 4: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/4.jpg)
Scalability
Hadoop Design Criteria
![Page 5: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/5.jpg)
Scalability
Multi-tenancy◦ A shared pool of nodes for all jobs◦ Allocate Hadoop clusters of fixed size on the
shared pool. Serviceability
◦ sets up a new cluster for every job◦ old and new Hadoop co-exist◦ Hadoop has short, 3-month release cycle
Hadoop on Demand
![Page 6: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/6.jpg)
Scalability Multi-tenancy Serviceability
Locality Awareness◦ JobTracker tries to place tasks close to the input
data◦ But node allocator is not aware of the locality
Failure of HoD
![Page 7: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/7.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness
High Cluster Utilization◦ HoD does not resize the cluster between stages◦ Users allocate more nodes than needed
Competing for resources results in longer latency to start a job
Failure of HoD
![Page 8: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/8.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness High Cluster Utilization
Reliability/Availability◦ The failure in one job tracker can bring down the
entire cluster◦ Overhead of tracking multiple jobs in a larger,
shared cluster
Problem with Shared Cluster
![Page 9: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/9.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness High Cluster Utilization Reliability/Availability
Secure and auditable operation◦ Authentication
Challenge of Multi-tenancy
![Page 10: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/10.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness High Cluster Utilization Reliability/Availability Secure and auditable operation
Support for Programming Model Diversity◦ Iterative computation ◦ Different communication pattern
Challenge of Multi-tenancy
![Page 11: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/11.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness High Cluster Utilization Reliability/Availability Secure and auditable operation Support for Programming Model Diversity
Flexible Resource Model◦ Hadoop: # of Map/reduce slots are fixed. ◦ Easy, but lower utilization
YARN
![Page 12: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/12.jpg)
Scalability Multi-tenancy Serviceability Locality Awareness High Cluster Utilization Reliability/Availability Secure and auditable operation Support for Programming Model Diversity Flexible Resource Model
Backward Compatibility◦ The system behaves similar to the old Hadoop
YARN
![Page 13: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/13.jpg)
Separating resource management functions from the programming model
MapReduce becomes just one of the application
Dryad, …. Etc Binary compatible/Source compatible
YARN
![Page 14: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/14.jpg)
YARN: Architecture
![Page 15: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/15.jpg)
One per cluster◦ Central, global view◦ Enable global properties
Fairness, capacity, locality Container
◦ Logical bundle of resources (CPU/memory) Job requests are submitted to RM
To start a job, RM finds a container to spawn AM No static resource partitioning
Resource Manager
![Page 16: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/16.jpg)
only handles an overall resource profile for each application◦ Local optimization/internal flow is up to the
application Preemption
◦ Request resources back from an application◦ Checkpoint snapshot instead of explicitly killing
jobs / migrate computation to other containers
Resource Manager (cont’)
![Page 17: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/17.jpg)
The head of a job Runs as a container Request resources from RM
◦ # of containers/ resource per container/ locality … Dynamically changing resource
consumption Can run any user code (Dryad, MapReduce,
Tez, REEF…etc) Requests are “late-binding”
Application Master
![Page 18: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/18.jpg)
Optimizes for locality among map tasks with identical resource requirements◦ Selecting a task with input data close to the
container. AM determines the semantics of the success
or failure of the container
MapReduce AM
![Page 19: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/19.jpg)
The “worker” daemon. Registers with RM One per node Container Launch Context – env var,
commands… Report resources (memory/CPU/etc…) Configure the environment for task execution Garbage collection/ Authentication Auxiliary services
◦ Output intermediate data between map and reduce tasks
Node Manager
![Page 20: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/20.jpg)
1. Submitting the application by passing a CLC for the Application Master to the RM.
2. When RM starts the AM, it should register with the RM and periodically advertise its liveness and requirements over the heartbeat protocol
3. Once the RM allocates a container, AM can construct a CLC to launch the container on the corresponding NM. It may also monitor the status of the running container and stop it when the resource should be reclaimed. Monitoring the progress of work done inside the container is strictly the AM’s responsibility.
4. Once the AM is done with its work, it should unregister from the RM and exit cleanly.
5. Optionally, framework authors may add control flow between their own clients to report job status and expose a control plane.
YARN framework/application writers
![Page 21: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/21.jpg)
RM Failure◦ Recover using persistent storage◦ Kill all containers, including AMs’◦ Relaunch AMs
NM Failure◦ RM detects it, mark the containers as killed,
report to Ams AM Failure
◦ RM kills the container and restarts it. Container Failure
◦ The framework is responsible for recovery
Fault tolerance and availability
![Page 22: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/22.jpg)
In a 2500-node cluster, throughput improves from 77 K jobs/day to 150 K jobs/day
YARN at Yahoo!
![Page 23: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/23.jpg)
In a 2500-node cluster, throughput improves from 4 M tasks/day to 10 M jobs/day
YARN at Yahoo!
![Page 24: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/24.jpg)
Why? ◦ the removal of the static split between map and
reduce slots. Essentially, moving to YARN, the CPU
utilization almost doubled “upgrading to YARN was equivalent to
adding 1000 machines [to this 2500 machines cluster]”
YARN at Yahoo!
![Page 25: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/25.jpg)
Pig, Hive, Oozie◦ Decompose a DAG job into multiple MR jobs
Apache Tez◦ DAG execution framework
Spark Dryad Giraph
◦ Vertice centric graph computation framework◦ fits naturally within YARN model
Storm – distributed real time processing engine (parallel stream processing)
REEF◦ Simplify implementing ApplicationMaster
Haya – Hbase clusters
Applications/Frameworks
![Page 26: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/26.jpg)
Sorting MapReduce benchmarks Preemption W/ Apache Tez REEF
Evaluation
![Page 27: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/27.jpg)
2100 nodes, each w/ two 2.3Ghz hexcore Xeon E5-2630, 64 GB memory, and 12x3TB disks
Record holder
Sorting
![Page 28: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/28.jpg)
Compare Hadoop 2.1.0 (YARN) against 1.2.1 260 node cluster Each slave node is running 2.27GHz Intel(R) Xeon(R) CPU
totalling to 16 cores, has 38GB physicalmemory, and 6x1TB 7200 RPM disks
MapReduce benchmarks
![Page 29: Wei-Chiu Chuang 10/17/2013 Permission to copy/distribute/adapt the work except the figures which are copyrighted by ACM](https://reader035.vdocuments.mx/reader035/viewer/2022062713/56649cee5503460f949bb66c/html5/thumbnails/29.jpg)
Described in more details in the Natjam paper (also in SoCC’13)
Benefit of preemption