automated profiling and resource management of pig programs … · • the language, called pig...
TRANSCRIPT
Automated Profiling and Resource Management ofPig Programs for Meeting Service Level Objectives⇤
Zhuoyao ZhangUniversity of Pennsylvania
Ludmila CherkasovaHewlett-Packard [email protected]
Abhishek VermaUniversity of Illinois [email protected]
Boon Thau LooUniversity of Pennsylvania
ABSTRACTAn increasing number of MapReduce applications associ-ated with live business intelligence require completion timeguarantees. In this paper, we consider the popular Pig frame-work that provides a high-level SQL-like abstraction on topof MapReduce engine for processing large data sets. Pro-grams written in such frameworks are compiled into directedacyclic graphs (DAGs) of MapReduce jobs. There is a lackof performance models and analysis tools for automated per-formance management of such MapReduce jobs. We of-fer a performance modeling environment for Pig programsthat automatically profiles jobs from the past runs and aimsto solve the following inter-related problems: (i) estimatingthe completion time of a Pig program as a function of allo-cated resources; (ii) estimating the amount of resources (anumber of map and reduce slots) required for completing aPig program with a given (soft) deadline. For solving theseproblems, initially, we optimize a Pig program execution byenforcing the optimal schedule of its concurrent jobs. ForDAGs with concurrent jobs, this optimization helps reducingthe program completion time: 10%-27% in our experiments.Moreover, it eliminates possible non-determinism of concur-rent jobs’ execution in the Pig program, and therefore, en-ables a more accurate performance model for Pig programs.We validate our approach using a 66-node Hadoop clusterand a diverse set of workloads: PigMix benchmark, TPC-Hqueries, and customized queries mining a collection of HPLabs’ web proxy logs. The proposed scheduling optimiza-tion leads to significant resource savings (20%-40% in ourexperiments) compared with the original, unoptimized solu-tion, and the predicted program completion times are within10% of the measured ones.
1. INTRODUCTIONThe amount of enterprise data produced daily is ex-
ploding. This is partly due to a new era of automateddata generation and massive event logging of automated⇤This work was largely completed during Z. Zhang’s and A.Verma’s internship at HP Labs. Will appear in Proc. of the9th IEEE International Conference on Autonomic Comput-ing (ICAC’2012), Sept. 14-18, 2012, San Jose, CA, USA.
and digitized business processes; new style customerinteractions that are done entirely via web; a set ofnovel applications used for advanced data analytics incall centers and for information management associatedwith data retention, government compliance rules, e-discovery and litigation issues that require to store andprocess large amount of historical data. Many com-panies are following the new wave of using MapRe-duce [4] and its open-source implementation Hadoopto quickly process large quantities of new data to drivetheir core business. MapReduce o↵ers a scalable andfault-tolerant framework for processing large data sets.However, a single input dataset and simple two-stagedataflow processing schema imposed by MapReduce modelis low level and rigid. To enable programmers to specifymore complex queries in an easier way, several projects,such as Pig [6], Hive [15], Scope [3], and Dryad [9], pro-vide high-level SQL-like abstractions on top of MapRe-duce engines. These frameworks enable complex ana-lytics tasks (expressed as high-level declarative abstrac-tions) to be compiled into directed acyclic graphs (DAGs)of MapReduce jobs.Another technological trend is the shift towards using
MapReduce and the above frameworks for supportinglatency-sensitive applications, e.g., personalized adver-tising, sentiment analysis, spam and fraud detection,real-time event log analysis, etc. These MapReduceapplications are deadline-driven and typically requirecompletion time guarantees and achieving service levelobjectives (SLOs). While there have been some re-search e↵orts [17, 19, 14] towards developing perfor-mance models for MapReduce jobs, these techniques donot apply to complex queries consisting of MapReduceDAGs. To address this limitation, our paper studies thepopular Pig framework [6], and aims to design a perfor-mance modeling environment for Pig programs to o↵ersolutions for the following problems:
• Given a Pig program, estimate its completion time asa function of allocated resources (i.e., allocated mapand reduce slots);
• Given a Pig program with a completion time goal, es-
1
timate the amount of resources (a number of map andreduce slots) required for completing the Pig programwith a given (soft) deadline.
The designed performance framework enables an SLO-driven job scheduler for MapReduce environments thatgiven a Pig program with a completion time goal, itcould allocate the appropriate amount of resources tothe program that it completes within the desired dead-line. This framework utilizes an automated profilingtool and past Pig program runs to extract performanceprofiles of the MapReduce jobs that constitute a givenPig program. We focus on Pig, since it is quickly becom-ing a popular and widely-adopted system for expressinga broad variety of data analysis tasks. With Pig, thedata analysts can specify complex analytics tasks with-out directly writing Map and Reduce functions. In June2009, more than 40% of Hadoop production jobs at Ya-hoo! were Pig programs [6].
While our paper is based on the Pig experience, webelieve that the proposed models and optimizations aregeneral and can be applied for performance modelingand resource allocations of complex analytics tasks thatare expressed as an ensemble (DAG) of MapReducejobs.
The paper makes the following key contributions:
Pig scheduling optimizations. For a Pig programdefined by a DAG of MapReduce jobs, its completiontime might be approximated as the sum of completiontimes of the jobs that constitute this Pig program. How-ever, such model might lead to a higher time estimatethan the actual measured program time. The reason isthat unlike the execution of sequential jobs where dueto data dependencies the next job can only start afterthe previous one is completed, for concurrent jobs inthe DAG, they might be executed in arbitrary orderand their map (and reduce) phases might be pipelined.That is, after one job completes its map phase and be-gins its reduce phase, the other concurrent job can startits map phase execution with the released map resourcesin a pipelined fashion. The performance model shouldtake this “overlap” in executions of concurrent jobs intoaccount. Moreover, this observation suggests that thechosen execution order of concurrent jobs may impactthe “amount of overlap” and the overall program pro-cessing time. Using this observation, we first, optimizea Pig program execution by enforcing the optimal sched-ule of its concurrent jobs. We evaluate optimized Pigprograms and the related performance improvementsusing TPC-H queries and a set of customized queriesmining a collection of HP Labs’ web proxy logs (bothsets are presented by the DAGs with concurrent jobs).Our results show 10%-27% decrease in Pig programcompletion times.
Performance modeling framework. The pro-posed Pig optimization has another useful outcome: it
eliminates existing non-determinism in Pig program ex-ecution of concurrent jobs, and therefore, it enables bet-ter performance predictions. We develop an accurateperformance model for completion time estimates andresource allocations of optimized Pig programs. Theaccuracy of this model is validated using PigMix bench-mark [2] and a combination of TPC-H and web proxylog analysis queries on a 66-node Hadoop cluster. Ourevaluation shows that the predicted completion timesare within 10% of the measured ones. For Pig pro-grams with concurrent jobs, we demonstrate that theproposed approach leads to significant resource savings(20%-40% in our experiments) compared with the orig-inal, non-optimized solution.This paper is organized as follows. Section 2 provides
a background on MapReduce processing and the Pigframework. Section 3 discusses subtleties of concurrentjobs execution in Pig, introduces optimized schedulingof concurrent jobs, and o↵ers a novel performance mod-eling framework for Pig programs. Section 4 describesthe experimental testbed and three workloads used inthe performance study. The models’ accuracy is evalu-ated in Section 5. Section 6 describes the related work.Section 7 presents a summary and future directions.
2. BACKGROUNDThis section provides a basic background on the MapRe-
duce framework [4] and its extension, the Pig system [6].
2.1 MapReduce JobsIn the MapReduce model, computation is expressed
as two functions: map and reduce. The map functiontakes an input pair and produces a list of intermediatekey/value pairs. The reduce function then merges oraggregates all the values associated with the same key.MapReduce jobs are automatically parallelized, dis-
tributed, and executed on a large cluster of commoditymachines. The map and reduce stages are partitionedinto map and reduce tasks respectively. Each map taskprocesses a logical split of input data. The map taskreads the data, applies the user-defined map functionon each record, and bu↵ers the resulting output. Thereduce stage consists of three phases: shu✏e, sort andreduce phase. In the shu✏e phase, the reduce tasksfetch the intermediate data files from the already com-pleted map tasks, thus following the“pull”model. Afterall the intermediate data is shu✏ed (i.e., when the en-tire map stage with all the map tasks is completed), afinal pass is made to merge all these sorted files, henceinterleaving the shu✏e and sort phases. Finally, in thereduce phase, the sorted intermediate data is passedto the user-defined reduce function. The output fromthe reduce function is generally written back to the dis-tributed file system.Job scheduling in Hadoop is performed by a master
2
node, which manages a number of worker nodes in thecluster. Each worker has a fixed number of map slotsand reduce slots, which can run map and reduce tasksrespectively. The number of map and reduce slots isstatically configured (typically, one per core or disk).Slaves periodically send heartbeats to the master to re-port the number of free slots and the progress of tasksthat they are currently running. Based on the availabil-ity of free slots and the scheduling policy, the masterassigns map and reduce tasks to slots in the cluster.
2.2 Pig ProgramsThere are two main components in the Pig system:
• The language, called Pig Latin, that combines high-level declarative style of SQL and the low-level pro-cedural programming of MapReduce. A Pig programis similar to specifying a query execution plan: it rep-resent a sequence of steps, where each one carries asingle data transformation using a high-level data ma-nipulation constructs, like filter, group, join, etc. Inthis way, the Pig program encodes a set of explicitdataflows.
• The execution environment to run Pig programs. ThePig system takes a Pig Latin program as input, com-piles it into a DAG of MapReduce jobs, and coor-dinates their execution on a given Hadoop cluster.Pig relies on underlying Hadoop execution engine forscalability and fault-tolerance properties.
The following specification shows a simple example ofa Pig program. It describes a task that operates over atable URLs that stores data with the three attributes:(url, category, pagerank). This program identifiesfor each category the url with the highest pagerankin that category.
URLs = load ’dataset’ as (url, category, pagerank);groups = group URLs by category;result = foreach groups generate group, max(URLs.pagerank);store result into ’myOutput’
The example Pig program is compiled into a singleMapReduce job. Typically, Pig programs are more com-plex, and can be compiled into an execution plan con-sisting of several stages of MapReduce jobs, some ofwhich can run concurrently. The structure of the ex-ecution plan can be represented by a DAG of MapRe-duce jobs that could contain both concurrent and se-quential branches. Figure 1 shows a possible DAG offive MapReduce jobs {j1, j2, j3, j4, j5}, where each noderepresents a MapReduce job, and the edges between thenodes represent data dependencies between jobs.
To execute the plan, the Pig engine first submits allthe ready jobs (i.e., the jobs that do not have data de-pendency on the other jobs) to Hadoop. After Hadoophas processed these jobs, the Pig system deletes themand the corresponding edges from the processing DAG,
j1
j2
j3
j4
j5 j6
Figure 1: Example of a Pig program’ execution plan
represented as a DAG of MapReduce jobs.
and identifies and submits the next set of ready jobs.This process continues until all the jobs are completed.In this way, the Pig engine partitions the DAG into mul-tiple stages, each containing one or more independentMapReduce jobs that can be executed concurrently.For example, the DAG shown in Figure 1 is parti-
tioned into the following four stages for processing:
• first stage: {j1, j2};• second stage: {j3, j4};• third stage: {j5};• fourth stage: {j6}.In Section 5, we will show some examples based on
TPC-H and web log analysis queries that are represen-tative of such MapReduce DAGs. Note that for stageswith concurrent jobs, there is no specifically defined or-dering in which the jobs are going to be executed byHadoop.
3. PERFORMANCE MODELING FRAME-WORK FOR PIG PROGRAMS
This section introduces our modeling framework forPig programs. First, we outline the profiling and mod-eling technique for a single MapReduce job. Then weanalyze subtleties in execution of concurrent MapRe-duce jobs and demonstrate that the job order has asignificant impact on the program completion time. Weoptimize a Pig program by enforcing the optimal sched-ule of its concurrent jobs, and it enables more accuratePig program performance modeling.
3.1 Performance Model of a Single MapRe-duce Job
As a building block for modeling Pig programs de-fined as DAGs of MapReduce jobs, we apply a slightlymodified approach introduced in [17] for performancemodeling of a single MapReduce job. The proposedMapReduce performance model [17] evaluates lower andupper bounds on the job completion time. It is basedon a general model for computing performance boundson the completion time of a given set of n tasks thatare processed by k servers, (e.g., n map tasks are pro-cessed by k map slots in MapReduce environment). LetT1, T2, . . . , Tn
be the duration of n tasks in a given set.Let k be the number of slots that can each execute onetask at a time. The assignment of tasks to slots is doneusing an online, greedy algorithm: assign each task tothe slot which finished its running task the earliest. Let
3
avg and max be the average and maximum duration ofthe n tasks respectively. Then the completion time of agreedy task assignment is proven to be at least:
T
low = avg · nk
and at most
T
up = avg · (n� 1)
k
+max.
The di↵erence between lower and upper bounds repre-sents the range of possible completion times due to taskscheduling non-determinism (i.e., whether the maximumduration task is scheduled to run last). Note, that theseprovable lower and upper bounds on the completiontime can be easily computed if we know the average andmaximum durations of the set of tasks and the numberof allocated slots. See [17] for detailed proofs on thesebounds.
As motivated by the above model, in order to ap-proximate the overall completion time of a MapReducejob J , we need to estimate the average and maximumtask durations during di↵erent execution phases of thejob, i.e., map, shu✏e/sort, and reduce phases. Thesemeasurements can be obtained from the job executionlogs. By applying the outlined bounds model, we canestimate the completion times of di↵erent processingphases of the job. For example, let job J be partitionedinto N
J
M
map tasks. Then the lower and upper boundson the duration of the entire map stage in the futureexecution with S
J
M
map slots (denoted as T low
M
and T
up
M
respectively) are estimated as follows:
T
low
M
= M
J
avg
·NJ
M
/S
J
M
(1)
T
up
M
= M
J
avg
· (NJ
M
� 1)/SJ
M
+M
J
max
(2)
where M
avg
and M
max
are the average and maximumof the map task durations of the past run respectively.Similarly, we can compute bounds of the execution timeof other processing phases of the job. As a result, wecan express the estimates for the entire job completiontime (lower bound T
low
J
and upper bound T
up
J
) as afunction of allocated map/reduce slots (SJ
M
, S
J
R
) usingthe following equation form:
T
low
J
=A
low
J
S
J
M
+B
low
J
S
J
R
+ C
low
J
. (3)
The equation for T
up
J
can be written in a similar form(see [17] for details and exact expressions of coe�cientsin these equations). Typically, the average of lower andupper bounds (T avg
J
) is a good approximation of the jobcompletion time.
Once we have a technique for predicting the job com-pletion time, it also can be used for solving the inverseproblem: finding the appropriate number of map andreduce slots that could support a given job deadline D
(e.g., if D is used instead of T low
J
in Equation 3). Whenwe consider S
J
M
and S
J
R
as variables in Equation 3 it
yields a hyperbola. All integral points on this hyper-bola are possible allocations of map and reduce slotswhich result in meeting the same deadline D. There isa point where the sum of the required map and reduceslots is minimized. We calculate this minima on thecurve using Lagrange’s multipliers [17], since we wouldlike to conserve the number of map and reduce slots re-quired for the minimum resource allocation per job J
with a given deadline D. Note, that we can use D forfinding the resource allocations from the correspondingequations for upper and lower bounds on the job com-pletion time estimates. In Section 5, we will comparethe outcome of using di↵erent bounds for estimating acompletion time of a Pig program.
3.2 Modeling Concurrent Jobs’ ExecutionsOur goal is to design a model for a Pig program that
can estimate the number of map and reduce slots re-quired for completing a Pig program with a given (soft)deadline. These estimates can be used by the SLO-based scheduler like ARIA [17] to tailor and controlresource allocations to di↵erent applications for achiev-ing their performance goals. When such a schedulerallocates a recommended amount of map/reduce slotsto the Pig program, it uses a FIFO schedule for jobswithin the DAG (see Section 2.2 for how these jobs aresubmitted by the Pig system).There is a subtlety in how concurrent jobs in the DAG
of MapReduce jobs might be executed. In the expla-nations below, we use the following useful abstraction.We represent any MapReduce job as a composition ofnon-overlaping map and reduce stages. Indeed, thereis a barrier between map and reduce stages and anyreduce task may start its execution only after all maptasks complete and all the intermediate data is shu✏ed.However, the shu✏e phase (that we consider as a partof the reduce stage) overlaps with the map stage. Note,that in the ARIA performance model [17] that we usefor estimating a job completion time, the shu✏e phasemeasurements include only non-overlaping portion ofthe latency. These measurements and the model allowus to estmate the duration of map and reduce stages(as a function of allocated map and reduce slots) andsupport a simple abstraction where the job executionis represented as a composition of non-overlaping mapand reduce stages.Let us consider two concurrent MapReduce jobs J1
and J2. Let us also assume that there are no data de-pendencies among the concurrent jobs. Therefore, un-like the execution of sequential jobs where the next jobcan only start after the previous one is entirely finished(shown in Figure 2 (a)), for concurrent jobs, once theprevious job completes its map stage and begins reducestage processing, the next job can start its map stageexecution with the released map resources in a pipelinedfashion (shown in Figure 2 (b)). The Pig performance
4
model should take this “overlap” in executions of con-current jobs into account.
J1M J1
R
J1
J2M J2
R
J2
(a) Sequential execution of two jobs J1 and J2.
J1M J1
R
J1 J2M J2
R
J2
(b) Concurrent execution of two jobs J1 and J2.
Figure 2: Di↵erence in executions of (a) two sequential
MapReduce jobs; (b) two concurrent MapReduce jobs.
We find one more interesting observation about con-current jobs’ execution of the Pig program. The originalHadoop implementation executes concurrent MapRe-duce jobs from the same Pig program in a random order.Some ordering may lead to ine�cient resource usage andan increased processing time. As a motivating example,let us consider two concurrent MapReduce jobs that re-sult in the following map and reduce stage durations:
• Job J1 has a map stage duration of JM
1 = 10s andthe reduce stage duration of JR
1 = 1s.
• Job J2 has a map stage duration of JM
2 = 1s andthe reduce stage duration of JR
2 = 10s.
There are two possible executions shown in Figure 3:J1
M=10s J1R=1s
J1 J2M=1s J2
R=10s
J2
(a) J1 is followed by J2.
J1M=10s J1
R=1s
J1
J2M=1s J2
R=10s
J2
(b) J2 is followed by J1.
Figure 3: Impact of concurrent job schedulingon their completion time.
• J1 is followed by J2 shown in Figure 3(a). The re-duce stage of J1 overlaps with the map stage of J2leading to overlap of only 1s. The total completiontime of processing two jobs is 10s+1s+10s = 21s.
• J2 is followed by J1 shown in Figure 3(b). Thereduce stage of J2 overlaps with the map stage ofJ1 leading to a much better pipelined executionand a larger overlap of 10s. The total makespan is1s+ 10s+ 1s = 12s.
There is a significant di↵erence in the job completiontime (75% in the example above) depending on the ex-ecution order of the jobs. We optimize a Pig programexecution by enforcing the optimal schedule of its con-current jobs. We apply the classic Johnson algorithmfor building the optimal two-stage jobs’ schedule [10].The optimal execution of concurrent jobs leads to im-proved completion time. Moreover, this optimization
eliminates possible non-determinism in Pig program ex-ecution, and enables more accurate completion timepredictions for Pig programs.
3.3 Completion Time Estimates for Pig Pro-grams
Using the model of a single MapReduce job as a build-ing block, we consider a Pig program P that is compiledinto a DAG of |P |MapReduce jobs P = {J1, J2, ...J|P |}.Automated profiling. To automate the construc-tion of all performance models, we build an automatedprofiling tool that extracts the MapReduce job profiles1
of the Pig program from the past program executions.These job profiles represent critical performance char-acteristics of the underlying application during all theexecution phases: map, shu✏e/sort, and reduce phases.For each MapReduce job J
i
(1 i |P |) that consti-tutes Pig program P , in addition to the number of map(NJi
M
) and reduce (NJiR
) tasks, we also extract met-rics that reflect the durations of map and reduce tasks(note that shu✏e phase measurements are included inthe reduce task measurements) 2:
(MJiavg
,M
Jimax
, AvgSize
Jiinput
M
, Selectivity
JiM
)
(RJiavg
, R
Jimax
, Selectivity
JiR
)
• AvgSize
Jiinput
M
is the average amount of input dataper map task of job J
i
(we use it to estimate thenumber of map tasks to be spawned for processing anew dataset).
• Selectivity
JiM
and Selectivity
JiR
refer to the ratio ofthe map (and reduce) output size to the map inputsize. It is used to estimate the amount of intermedi-ate data produced by the map (and reduce) stage ofjob J
i
. This allows to estimate the size of the inputdataset for the next job in the DAG.
As a building block for modeling a Pig program definedas a DAG of MapReduce jobs, we apply the approachintroduced in ARIA [17] for performance modeling ofa single MapReduce job. We extract performance pro-files of all the jobs in the DAG from the past programexecutions. Using these job profiles we can predict thecompletion time of each job (and completion time ofmap and reduce stages) as a function of allocated map1To di↵erentiate MapReduce jobs in the same Pig program, we mod-
ified the Pig system to assign a unique name for each job as follows:queryName-stageID-indexID, where stageID represents the stage inthe DAG that the job belongs to, and indexID represents the indexof jobs within a particular stage.2Unlike prior models [17], we normalize all the collected measure-
ments per record to reflect the processing cost of a single record.This normalized cost is used to approximate the duration of map andreduce tasks when the Pig program executes on a new dataset with alarger/smaller number of records. To reflect a possible skew of recordsper task, we collect an average and maximum number of records pertask. The task durations (average and maximum) are computed bymultiplying the measured per-record time by the number of inputrecords (average and maximum) processed by the task.
5
and reduce slots. We can compute the completion timeusing a lower or upper bound estimates as described inSection 2. For the rest of this section, we use the com-pletion time estimates based on the average of the lowerand upper bounds.
Let us consider a Pig program P that is compiled intoa DAG of MapReduce jobs and consists of S stages.
Note that due to data dependencies within a Pig exe-cution plan, the next stage cannot start until the previ-ous stage finishes. Let T
Si denote the completion timeof stage S
i
. Thus, the completion of a Pig program P
can be estimated as follows:
T
P
=X
1iS
T
Si . (4)
For a stage that consists of a single job J , the stagecompletion time is defined by the job J ’s completiontime.
For a stage that contains concurrent jobs, the stagecompletion time depends on the jobs’ execution order.Suppose there are |S
i
| jobs within a particular stageS
i
and the jobs are executed according to the order{J1, J2, ...J|Si|}. Note, that given a number of allocatedmap/reduce slots (SP
M
, S
P
R
) to the Pig program P , wecan compute for any MapReduce job J
i
(1 i |Si
|)the duration of its map and reduce phases that are re-quired for the Johnson’s algorithm [10] to determine theoptimal schedule of the jobs {J1, J2, ...J|Si|}.
Let us assume, that for each stage with concurrentjobs, we have already determined the optimal job sched-ule that minimizes the completion time of the stage.Now, we introduce the performance model for predict-ing the Pig program P completion time T
P
as a functionof allocated resources (SP
M
, S
P
R
). We use the followingnotations:
timeStartMJithe start time of job Ji’s map phase
timeEndMJithe end time of job Ji’s map phase
timeStartRJithe start time of job Ji’s reduce phase
timeEndRJithe end time of job Ji’s reduce phase
Then the stage completion time can be estimated as
T
Si = timeEnd
R
J|Si|� timeStart
M
J1(5)
We now explain how to estimate the start/end time ofeach job’s map/reduce phase3.
Let TM
Jiand T
R
Jidenote the completion times of map
and reduce phases of job J
i
respectively. Then
timeEnd
M
Ji= timeStart
M
Ji+ T
M
Ji(6)
timeEnd
R
Ji= timeStart
R
Ji+ T
R
Ji(7)
3These computations present the main, typical case when the number
of allocated slots is smaller than the number of tasks that jobs need toprocess, and therefore, the execution of concurrent jobs is pipelined.There are some corner cases with small concurrent jobs when thereare enough resources for processing them at the same time. In thiscase, the designed model over-estimates the stage completion time.We have an additional set of equations that describes these cornercases in a more accurate way. We omit them here for presentationsimplicity.
Figure 4 shows an example of three concurrent jobs ex-ecution in the order J1, J2, J3.
J1M J1
R
J1 J2M J2
R
J2 J3M J3
R
J3
(a)J1
M
J1R
J2M
J2R
J3M
J3R
(b)
Figure 4: Execution of Concurrent Jobs
Note, that Figure 4 (a) can be rearranged to show theexecution of jobs’ map/reduce stages separately (overthe map/reduce slots) as shown in Figure 4 (b). It iseasy to see that since all the concurrent jobs are inde-pendent, the map phase of the next job can start im-mediately once the previous job’s map stage is finished,i.e.,
timeStart
M
Ji= timeEnd
M
Ji�1= timeStart
M
Ji�1+ T
M
Ji�1
(8)The start time timeStart
R
Jiof the reduce stage of the
concurrent job J
i
should satisfy the following two con-ditions:
1. timeStart
R
Ji� timeEnd
M
Ji
2. timeStart
R
Ji� timeEnd
R
Ji�1
Therefore, we have the following equation:
timeStart
R
Ji= max{timeEnd
M
Ji, timeEnd
R
Ji�1} =
= max{timeStart
M
Ji+ T
M
Ji, timeStart
R
Ji�1+ T
R
Ji�1} (9)
Finally, the completion time of the entire Pig programP is defined as the sum of its stages using eq. (4).
3.4 Resource Allocation Estimates for Opti-mized Pig Programs
Let us consider a Pig program P with a given dead-lineD. The optimized execution of concurrent jobs in P
may significantly improve the program completion time.Therefore, P may need to be assigned a smaller amountof resources for meeting a given deadline D compared toits non-optimized execution. First, we explain how toapproximate the resource allocation of a non-optimizedexecution of a Pig program. The completion time ofnon-optimized P can be represented as a sum of com-pletion time of the jobs that comprise the DAG of thisPig program. Thus, for a Pig program P that con-tains |P | jobs, its completion time can be estimated asa function of assigned map and reduce slots (SP
M
, S
P
R
)as follows:
T
P
(SP
M
, S
P
R
) =X
1i|P |
T
Ji(SP
M
, S
P
R
) (10)
6
The unique benefit of this model is that it allows us toexpress the completion time D of a Pig program P via aspecial form equation shown below (similar to eq. (3)):
D =A
P
S
P
M
+A
P
S
P
R
+ C
P (11)
This equation can be used for solving the inverse prob-lem of finding resource allocations (SP
M
, S
P
R
) such thatP completes within time D. This equation yields a hy-perbola if (SP
M
, S
P
R
) are considered as variables. We candirectly calculate the minima on this curve by using La-grange’s multipliers for finding the resource allocationof a single MapReduce job with a given deadline.
The model introduced in Section 3.3 for accurate com-pletion time estimates of an optimized Pig program ismore complex. It requires computing a function maxfor stages with concurrent jobs, and therefore, it can-not be expressed as a single equation for solving theinverse problem of finding the appropriate resource al-location. However, we can use the “over-provisioned”resource allocation defined by eq. (11) as an initial pointfor determining the solution required by the optimizedPig program P . The hyperbola with all the possiblesolutions according to the “over-sized” model is shownin Figure 5 as the red curve, and A(M,R) representsthe point with a minimal number of map and reduceslots (i.e., the pair (M,R) results in the minimal sumof map and reduce slots). We designed the followingalgorithm described below that determines the minimalresource allocation pair (M
min
, R
min
) for an optimizedPig program P with deadline D. This computation isillustrated by Figure 5.
B (M’,R) A (M,R)
D (Mmin,Rmin) C (M,R’)
Nu
mb
er o
f R
edu
ce S
lots
(R
)
Number of Map Slots (M)
Figure 5: Resource allocation estimates for anoptimized Pig program.
First, we find the minimal number of map slots M
0
(i.e., the pair (M 0, R)) such that deadline D can still be
met by the optimized Pig program with the enforced op-timal execution of its concurrent jobs. We do it by fixingthe number of reduce slots to R, and then step-by-stepreducing the allocation of map slots. Specifically, Algo-rithm 1 sets the resource allocation to (M � 1, R) andchecks whether program P can still be completed withintime D (we use T
avg
P
for completion time estimates).
If the answer is positive, then it tries (M � 2, R) asthe next allocation. This process continues until pointB(M 0
, R) (see Figure 5) is found such that the numberM
0 of map slots cannot be further reduced for meetinga given deadline D (lines 1-4 of Algorithm 1).
Algorithm 1 Determining the resource allocation fora Pig program
Input:Job profiles of all the jobs in P = {J1, J2, ...J|Si|}D a given deadline(M,R) the minimum pair of map and reduce slots obtainedfor P and deadline D by applying the basic modelOptimal execution of jobs J1, J2, ...J|Si| based on (M,R)Output:Resource allocation pair (Mmin, Rmin) for optimized P
1: M 0 M , R0 R2: while Tavg
P (M 0, R) D do // From A to B3: M 0 (M 0 � 14: end while5: while Tavg
P (M,R0) D do // From A to C6: R0 ( R0 � 1,7: end while8: Mmin M,Rmin R , Min (M +R)9: for M M 0 + 1 to M do // Explore purple curve B to
C10: R = R� 111: while Tavg
P (M, R) D do
12: R( R� 113: end while14: if M + R < Min then15: Mmin ( M,Rmin ( R,Min (M + R)16: end if17: end for
In the second step, we apply the same process for find-ing the minimal number of reduce slots R0 (i.e., the pair(M,R
0)) such that the deadline D can still be met bythe optimized Pig program P (lines 5-7 of Algorithm 1).In the third step, we determine the intermediate val-
ues on the curve between (M 0, R) and (M,R
0) such thatdeadline D is met by the optimized Pig program P .Starting from point (M 0
, R), we are trying to find theallocation of map slots from M
0 to M , such that theminimal number of reduce slots R should be assigned toP for meeting its deadline (lines 10-12 of Algorithm 1).Finally, (M
min
, R
min
) is the pair on this curve suchthat it results in the the minimal sum of map and reduceslots.
4. EXPERIMENTAL TESTBED AND WORK-LOADS
All experiments are performed on 66 HP DL145 GL3machines. Each machine has four AMD 2.39GHz cores,8 GB RAM and two 160GB hard disks. The machinesare set up in two racks and interconnected with gigabitEthernet. We used Hadoop 0.20.2 and Pig-0.7.0 withtwo machines dedicated as the JobTracker and the Na-meNode, and remaining 64 machines as workers. Eachworker is configured with 2 map and 1 reduce slots.
7
j1
j2
j3
j4
j5 j6 j7 j8
(a) TPC-H Q5
j1
j2
j3
j4
j5
j6
j7 j8 j9 j10
(b) TPC-H Q8
j1
j2
j3 j4 j5 j6
(c) TPC-H Q10
j1
j2
j3
j4
(d) Proxy Q1
j1
j2
j3
j4
j5 j6
(e) Proxy Q2
j1
j2
j3
j4
j5
j6
j7
j8
(f) Proxy Q3
Figure 6: DAGs of Pig programs in the TPC-H and HP Labs Proxy query sets.
The file system blocksize is set to 64MB. The replica-tion level is set to 3. We disabled speculative executionsince it did not lead to significant improvements in ourexperiments.
In our case studies, we use three di↵erent workloadsets: the PigMix benchmark, TPC-H queries, and cus-tomized queries for mining web proxy logs from an en-terprise company. We briefly describe each dataset andour respective modifications:
PigMix. We use the well-known PigMix bench-mark [2] that was created for testing Pig system perfor-mance. It consists of 17 Pig programs (L1-L17), whichuses datasets generated by the default Pigmix data gen-erator. In total, 1TB of data across 8 tables are gener-ated. The PigMix programs cover a wide range of thePig features and operators, and the data set are gen-erated with similar properties to Yahoo’s datasets thatare commonly processed using Pig. With the exceptionof L11 (that contains a stage with 2 concurrent jobs),all PigMix programs involve DAGs of sequential jobs.Therefore, with the PigMix benchmark we mostly eval-uate the accuracy of the proposed performance modelfor completion time estimates and resource allocationdecisions,. The e�ciency of the designed optimizationtechnique for concurrent job execution is evaluated withthe next two workloads in the set.
TPC-H. This workload is based on TPC-H [1], a stan-dard database benchmark for decision-support work-loads. We select three queries Q5, Q8, Q10 out of exist-ing 22 SQL queries and express them as Pig programs.
For each query, we select a logical plan that results ina DAG of concurrent MapReduce jobs shown in Fig. 6 (a),(b),(c)respectively4:
• The TPC-H Q5 query joins 6 tables, and its dataflowresults in 3 concurrent MapReduce jobs.
• The TPC-H Q8 query joins 8 tables, and its dataflowresults in two stages with 4 and 2 concurrent jobs.
• The TPC-H Q10 query joins 4 tables, and its dataflowresults in 2 concurrent MapReduce jobs.
HP Labs’ Web Proxy Query Set. This workloadconsists of a set of Pig programs for analyzing HP Labs’web proxy logs. It contains 6 months access logs to web
4While more e�cient logical plans may exist, our goal here is to create
a DAG with concurrent jobs to stress test our model.
proxy gateway at HP Labs. The fields include informa-tion such as date, time, time-taken, c-ip, cs-host, etc.We intend to create realistic Pig queries executed onreal-world data.
• The Proxy Q1 program investigates the dynamics inaccess frequencies to di↵erent websites per month andcompares them across di↵erent months. The Pig pro-gram results in 3 concurrent MapReduce jobs withthe DAG of the program shown in Figure 6 (d).
• The Proxy Q2 program discovers the co-relationshipbetween two websites from di↵erent sets (tables) ofpopular websites: the first set is created to representthe top 500 popular websites accessed by web userswithin the enterprise. The second set contains thetop 100 popular websites in US according to Alexa’sstatistics5. The program DAG is shown in Figure 6 (e).
• The Proxy Q3 program presents the intersection of100 most popular websites (i.e., websites with high-est access frequencies) accessed both during work andafter work hours. The DAG of the program is shownin Figure 6 (f).
To perform the validation experiments, we create twodi↵erent datasets for each of the three workloads above:
1. A test dataset: It is used for extracting the jobprofiles of the corresponding Pig programs.
• For PigMix benchmark, the test dataset is gen-erated by the default data generator. It con-tains 125 million records for the largest tableand has a total size around 1 TB.
• For TPC-H, the test dataset is generated withscaling factor 9 using the standard data gen-erator. The dataset size is around 9 GB.
• For HP Labs’ Web proxy query set, we use thelogs from February, March and April as thetest dataset with the total input size around9 GB.
2. An experimental dataset: It is used to vali-date our performance models using the profiles ex-tracted from the Pig programs that were executedwith the test dataset. Both the test and experi-mental datasets are formed by the tables with thesame layout but with di↵erent input sizes.
5http://www.alexa.com/topsites
8
• For PigMix benchmark, the input size of theexperimental dataset is 20% larger than thetest dataset (with 150 million records for thelargest table).
• For TPC-H, the experimental dataset is around15 GB (scaling factor 15 using the standarddata generator).
• For HP Labs’ Web proxy query set, we use thelogs from May, June and July as the experi-mental dataset, the total input size is around9 GB.
5. PERFORMANCE EVALUATIONThis section presents the performance evaluation of
the proposed models and optimizations. Since PigMixbenchmark mostly consists of the Pig programs with se-quential DAGs, we use PigMix only for validating theaccuracy of the proposed performance models. BecauseTPC-H and HP Labs’ proxy log queries represent Pigprograms that are defined by the DAGs with concur-rent jobs, we use these two workloads for evaluating theperformance benefits of introduced Pig program opti-mization as well as the models’ validation.
5.1 PigMix Case StudyThis section aims to evaluate the accuracy of the pro-
posed models: i) the bound-based model for estimatingthe completion time of Pig programs as a function ofallocated resources, and ii) the accuracy of the recom-mended resource allocation for meeting the completiontime goals.
First, we run the Pigmix benchmark on the test dataset,and the job profiles of the corresponding Pig programsare built from these executions. By using the extractedjob profiles and the designed Pig model we computethe completion time estimates of Pig programs in thePigMix benchmark for processing these two datasets(test and experimental) as a function of allocated re-sources. Then we validate the predicted completiontimes against the measured ones.
Figure 7 shows the predicted vs measured results forthe PigMix benchmark that processes the test datasetwith 128 map and 64 reduce slots. Given that the com-pletion times of di↵erent programs in PigMix are ina broad range of 100s – 2000s, for presentation pur-poses and easier comparison, we normalize the pre-dicted completion times with respect to the measuredones. The three bars in Figure 7 represent the predictedcompletion times based on the lower (T low) and upper(Tup) bounds, and the average of them (T avg). We ob-serve that the actual completion times (shown as thestraight Measured-CT line) of all 17 programs fall be-tween the lower and upper bound estimates. Moreover,the predicted completion times based on the averageof the upper and lower bounds are within 10% of the
measured results for most cases. The worst prediction(around 20% error) is for the Pig query L11.
��
����
��
����
��
/� /� /� /� /� /� /� /� /� /�� /�� /�� /�� /�� /�� /�� /��
Prog
ram
Com
plet
ion
Tim
e Tlow
Tavg
TupMeasured-CT
Figure 7: Predicted and measured completion time for
PigMix executed with test dataset and 128x64 slots.
Figure 8 shows the results for the PigMix benchmarkthat processes the test dataset with 64 map and 64 re-duce slots. Indeed, our model accurately computes theprogram completion time estimates as a function of al-located resources: the actual completion times of all 17programs are between the computed lower and upperbounds. The predicted completion times based on theaverage of the upper and lower bounds provide the bestresults: 10-12% of the measured results for most cases.
��
����
��
����
��
/� /� /� /� /� /� /� /� /� /�� /�� /�� /�� /�� /�� /�� /��
Prog
ram
Com
plet
ion
Tim
e Tlow
Tavg
TupMeasured-CT
Figure 8: Predicted and measured completion time for
PigMix executed with test dataset and 64x64 slots.
Figure 9 shows the predicted vs measured comple-tion times for the PigMix benchmark that processes thelarger, experimental dataset with 64 map and 64 reduceslots.
��
����
��
����
��
����
/� /� /� /� /� /� /� /� /� /�� /�� /�� /�� /�� /�� /�� /��
Prog
ram
Com
plet
ion
Tim
e Tlow
Tavg
Tup
Measured-CT
Figure 9: Predicted and measured completion time for
PigMix executed with experimental dataset and 64x64 slots.
As shown in Figure 9, our model and computed es-timates are quite accurate. The measured completiontime of all programs is between the low and upper bounds.The predicted completion times that are based on theaverage of the upper and lower bounds provide the bestresults: they are within 10% of the measured results formost cases.
9
In Figures 7, 8, 9 we execute PigMix benchmark threetimes and report the measured completion time aver-aged across 3 runs. A variance for most programs inthese three runs is within 1%-2%, with the largest vari-ance being around 6%. Because the variance is so small,we have omitted the error bars in Figures 7, 8, 9.
Our second set of experiments aims to evaluate thesolution of the inverse problem: the accuracy of a re-source allocation for a Pig program with a completiontime goal, often defined as a part of Service Level Ob-jectives (SLOs).
In this set of experiments, let T denote the Pig pro-gram completion time when the program is processedwith maximum available cluster resources (i.e., whenthe entire cluster is used for program processing). Weset D = 3 · T as a completion time goal. Using the ap-proach described in Section 3.4 we compute the requiredresource allocation, i.e., a tailored number of map andreduce slots that allow the Pig program to be completedwith deadline D on a new experimental dataset. Wecompute resource allocations when D is targeted as ei-ther a lower bound, or upper bound or the average oflower and upper bounds on the completion time. Fig-ure 10 shows the measured program completion timesbased on these three di↵erent resource allocations. Sim-ilar to our earlier results, for presentation purposes, wenormalize the achieved completion times with respectto the given deadline D.
��������������������
����������������
/� /� /� /� /� /� /� /� /� /�� /�� /�� /�� /�� /�� /�� /��
Prog
ram
Com
plet
ion
Tim
e Tlow-basedTavg-basedTup-based
Deadline
Figure 10: PigMix executed with the experimental
dataset: do we meet deadlines?
In most cases, the resource allocation that targetsD as a lower bound is insu�cient for meeting the tar-geted deadline (e.g., the L10 program misses deadlineby more than 20%). However, when we compute theresource allocation based on D as an upper bound – weare always able to meet the required deadline, but inmost cases, we over-provision resources, e.g., L16 andL17 finish more than 20% earlier than a given deadline.
The resource allocations based on the average be-tween lower and upper bounds result in the closest com-pletion time to the targeted program deadlines.
5.2 Optimal Schedule of Concurrent JobsFigure 11 shows the impact of concurrent jobs schedul-
ing on the completion time of TPC-H and Proxy querieswhen each program is processed with 128 map and 64
reduce slots.
��
����
����
����
����
��
����
����
4� 4� 4��
Prog
ram
Com
plet
ion
Tim
e Best-CTWorst-CT
(a) Job completion time
��
����
����
����
����
��
����
����
4� 4� 4��
Stag
e C
ompl
etio
n Ti
me Best-CT
Worst-CT
(b) Stage comple-tion time
��
����
����
����
����
��
����
����
4� 4� 4�
Prog
ram
Com
plet
ion
Tim
e Best-CTWorst-CT
(c) Job completion time
��
����
����
����
����
��
����
����
4� 4� 4�
Stag
e C
ompl
etio
n Ti
me Best-CT
Worst-CT
(d) Stage comple-tion time
Figure 11: Normalized completion time for di↵erent
schedules of concurrent jobs: (a-b) TPC-H, (c-d) HP
Labs proxy queries.
Figures 11 (a) and (c) show two extreme measure-ments: the best program completion time (i.e., whenthe optimal schedule of concurrent jobs is chosen) andthe worst one (i.e., when concurrent jobs are executed inthe “worst” possible order based on our estimates). Forpresentation purposes, the best (optimal) completiontime time is normalized with respect to the worst one.The choice of optimal schedule of concurrent jobs re-duces the completion time by 10%-27% compared withthe worse case ordering.Figures 11 (b) and (d) show completion times of stages
with concurrent jobs under di↵erent schedules for thesame TPC-H and Proxy queries. Performance benefitsat the stage level are even higher: they range between20%-30%.
5.3 Predicting Completion Time and ResourceAllocation of Optimized Pig Programs
Figure 12 shows the Pig program completion timeestimates (we use T
avg
P
in these experiments) based onthe proposed performance model for TPC-H and Proxyqueries with the experimental datasets. Figure 12 showsthe results when each program is processed with 64x64and 32x64 map and reduce slots respectively.
��
����
��
����
4� 4� 4��
Prog
ram
Com
plet
ion
Tim
e CT-Predicted-64x64CT-Predicted-32x64
Measured-CT
(a) TPC-H
��
����
��
����
4� 4� 4�
Prog
ram
Com
plet
ion
Tim
e CT-Predicted-64x64CT-Predicted-32x64
Measured-CT
(b) Proxy’s Queries
Figure 12: Predicted Pig programs completion times
executed with the experimental dataset.
In all cases, the predicted completion times are within10% of the measured ones.Let T denote the Pig program completion time when
10
program P is processed with maximum available clus-ter resources. We set D = 2 · T as a completion timegoal (we use di↵erent deadlines for di↵erent experimentson purpose, in order to validate the accuracy of ourmodels for a variety of parameters). Then we computethe required resource allocation for P executed withthe experimental dataset to meet the deadline D. Fig-ure 13 (a) shows measured completion times achieved bythe TPC-H and Proxy’s queries respectively when theyare assigned the resource allocations computed withthe designed resource allocation model. All the queriescomplete within 10% of the targeted deadlines.
��
����
��
����
Prog
ram
Com
plet
ion
Tim
e Measured-Proxy-QueriesMeasured-TPC-H
Deadline
(a) Can we meet deadlines?
��
����
��
����
Nor
mal
ized
Res
ourc
e R
equi
rem
ents
Proxy-Queries-RATPC-H-RA
Non-Optimized-RA
(b) Resource requirementsof optimized Pig programs.
Figure 13: Resource allocations for optimized Pig pro-
grams executed with the experimental dataset.
Figure 13 (b) compares the amount of resources (thesum of map and reduce slots) for non-optimized andoptimized executions of TPC-H and Proxy’s queries re-spectively. The optimized executions are able to achievetargeted deadlines with much smaller resource alloca-tions (20%-40% smaller) compared to resource alloca-tions for non-optimized Pig programs. Therefore, theproposed optimal schedule of concurrent jobs leads tosignificant resource savings for deadline-driven Pig pro-grams.
6. RELATED WORKWhile performance modeling in the MapReduce frame-
work is a new topic, there are several interesting re-search e↵orts in this direction.
Polo et al. [14] introduce an online job completiontime estimator which can be used in their new Hadoopscheduler for adjusting the resource allocations of di↵er-ent jobs. However, their estimator tracks the progressof the map stage alone, and use a simplistic way forpredicting the job completion time, while skipping theshu✏e/sort phase, and have no information or controlover the reduce stage.
FLEX [19] develops a novel Hadoop scheduler by propos-ing a special slot allocation schema that aims to opti-mize some given scheduling metric. FLEX relies on thespeedup function of the job (for map and reduce stages)that defines the job execution time as a function of al-located slots. However, it is not clear how to derivethis function for di↵erent applications and for di↵erentsizes of input datasets. The authors do not provide adetailed MapReduce performance model for jobs withtargeted job deadlines.
ARIA [17] introduces a deadline-based scheduler forHadoop. This scheduler extracts the job profiles fromthe past executions, and provides a variety of bounds-based models for predicting a job completion time asa function of allocated resources and a solution of theinverse problem. However, these models apply only toa single MapReduce job.Tian and Chen [16] aim to predict performance of a
single MapReduce program from the test runs with asmaller number of nodes. They consider MapReduceprocessing at a fine granularity. For example, the maptask is partitioned in 4 functions: read a block, mapfunction processing of the block, partition and sort ofthe processed data, and the combiner step (if it is used).The reduce task is decomposed in 4 functions as well.The authors use a linear regression technique to approx-imate the cost (duration) of each function. These func-tions are used for predicting the larger dataset process-ing. There are a few simplifying assumptions, e.g., asingle wave in the reduce stage. The problem of findingresource allocations that support given job completiongoals are formulated as an optimization problem thatcan be solved with existing commercial solvers.There is an interesting group of papers that design
a detailed job profiling approach for pursuing a di↵er-ent goal: to optimize the configuration parameters ofboth a Hadoop cluster and a given MapReduce job (orworkflow of jobs) for achieving the improved completiontime. Starfish project [8, 7] applies dynamic instrumen-tation to collect a detailed run-time monitoring infor-mation about job execution at a fine granularity: datareading, map processing, spilling, merging, shu✏ing,sorting, reduce processing and writing. Such a detailedjob profiling information enables the authors to analyzeand predict job execution under di↵erent configurationparameters, and automatically derive an optimized con-figuration. One of the main challenges outlined by theauthors is a design of an e�cient searching strategythrough the high-dimensional space of parameter val-ues. The authors o↵er a workflow-aware scheduler thatcorrelate data (block) placement with task schedulingto optimize the workflow completion time. In our work,we propose complementary optimizations based on op-timal scheduling of concurrent jobs within the DAG tominimize overall completion time.Kambatla et al [11] propose a di↵erent approach for
optimizing the Hadoop configuration parameters (num-ber of map and reduce slots per node) to improve MapRe-duce program performance. A signature-based (fingerprint-based) approach is used to predict the performance of anew MapReduce program using a set of already studiedprograms. The ideas presented in the paper are inter-esting but it is a position paper that does not provideenough details and lacking the extended evaluation ofthe approach.
11
Ganapathi et al. [5] use Kernel Canonical CorrelationAnalysis to predict the performance of Hive queries.However, they do not attempt to model the actual exe-cution of the MapReduce job: the authors discover thefeature vectors through statistical correlation.
CoScan [18] o↵ers a special scheduling framework thatmerges the execution of Pig programs with commondata inputs in such a way that this data is only scannedonce. Authors augment Pig programs with a set of(deadline, reward) options to achieve. Then they for-mulate the schedule as an optimization problem ando↵er a heuristic solution.
Morton et al. [12] propose ParaTimer: the progressestimator for parallel queries expressed as Pig scripts [6].In their earlier work [13], they designed Parallax – aprogress estimator that aims to predict the completiontime of a limited class of Pig queries that translate intoa sequence of MapReduce jobs. In both papers, in-stead of a detailed profiling technique that is designedin our work, the authors rely on earlier debug runs ofthe same query for estimating throughput of map andreduce stages on the input data samples provided bythe user. The approach is based on precomputing theexpected schedule of all the tasks, and therefore iden-tifying all the pipelines (sequences of MapReduce jobs)in the query. The approach relies on a simplified as-sumption that map (reduce) tasks of the same job havethe same duration. This work is closest to ours in pur-suing the completion time estimates for Pig programs.However, the usage of the FIFO scheduler and simpli-fying assumptions limit the approach applicability forprogress estimation of multiple jobs running in the clus-ter with a di↵erent Hadoop scheduler, especially if theamount of resources allocated to a job varies over timeor di↵ers from the debug runs that are used for mea-surements.
7. CONCLUSIONDesign of new job profiling tools and performance
models for MapReduce environments has been an ac-tive research topic in industry and academia duringpast few years. Most of these e↵orts were driven bydesign of new schedulers to satisfy the job specific goalsand to improve cluster resource management. In ourwork, we have introduced a novel performance model-ing framework for processing Pig programs with dead-lines. Our job profiling technique is not intrusive, itdoes not require any modifications or instrumentationof either the application or the underlying Hadoop/Pigexecution engines. The proposed approach enables au-tomated SLO-driven resource sizing and provisioningof complex workflows defined by the DAGs of MapRe-duce jobs. Moreover, our approach o↵ers an optimizedscheduling of concurrent jobs within a DAG that allowsto significantly reduce the overall completion time.
Our performance models are designed for the casewithout node failures. We see a natural extension forincorporating di↵erent failure scenarios and estimatingtheir impact on the application performance and achiev-able “degraded” SLOs. We intend to apply designedmodels for solving a broad set of problems related to ca-pacity planning of MapReduce applications (defined bythe DAGs of MapReduce jobs) and the analysis of var-ious resource allocation trade-o↵s for supporting theirSLOs.
8. REFERENCES[1] TPC Benchmark H (Decision Support), Version 2.8.0,
Transaction Processing Performance Council (TPC),http://www.tpc.org/tpch/, 2008.
[2] Apache. PigMix Benchmark,http://wiki.apache.org/pig/PigMix, 2010.
[3] R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib,S. Weaver, and J. Zhou. Easy and E�cient Parallel Processingof Massive Data Sets. Proc. of the VLDB Endowment, 1(2),2008.
[4] J. Dean and S. Ghemawat. MapReduce: Simplified DataProcessing on Large Clusters. Communications of the ACM,51(1), 2008.
[5] A. Ganapathi, Y. Chen, A. Fox, R. Katz, and D. Patterson.Statistics-driven workload modeling for the cloud. In Proc. of5th International Workshop on Self Managing DatabaseSystems (SMDB), 2010.
[6] A. Gates, O. Natkovich, S. Chopra, P. Kamath, S. Narayanam,C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Buildinga High-Level Dataflow System on Top of Map-Reduce: The PigExperience. Proc. of the VLDB Endowment, 2(2), 2009.
[7] H. Herodotou and S. Babu. Profiling, What-if Analysis, andCostbased Optimization of MapReduce Programs. In Proc. ofthe VLDB Endowment, Vol. 4, No. 11, 2011.
[8] H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. Cetin,and S. Babu. Starfish: A Self-tuning System for Big DataAnalytics. In Proc. of 5th Conf. on Innovative Data SystemsResearch (CIDR), 2011.
[9] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad:Distributed Data-Parallel Programs from Sequential BuildingBlocks. ACM SIGOPS OS Review, 41(3), 2007.
[10] S. Johnson. Optimal Two- and Three-Stage ProductionSchedules with Setup Times Included. Naval Res. Log. Quart.,1954.
[11] K. Kambatla, A. Pathak, and H. Pucha. Towards optimizinghadoop provisioning in the cloud. In Proc. of the FirstWorkshop on Hot Topics in Cloud Computing, 2009.
[12] K. Morton, M. Balazinska, and D. Grossman. ParaTimer: aprogress indicator for MapReduce DAGs. In Proc. ofSIGMOD. ACM, 2010.
[13] K. Morton, A. Friesen, M. Balazinska, and D. Grossman.Estimating the progress of MapReduce pipelines. In Proc. ofICDE, 2010.
[14] J. Polo, D. Carrera, Y. Becerra, J. Torres, E. Ayguade,M. Steinder, and I. Whalley. Performance-Driven TaskCo-Scheduling for MapReduce Environments. In Proc. of the12th IEEE/IFIP Network Operations and ManagementSymposium, 2010.
[15] A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka,S. Anthony, H. Liu, P. Wycko↵, and R. Murthy. Hive - aWarehousing Solution over a Map-Reduce Framework. Proc. ofVLDB, 2009.
[16] F. Tian and K. Chen. Towards Optimal Resource Provisioningfor Running MapReduce Programs in Public Clouds. In Proc.of IEEE Conference on Cloud Computing (CLOUD 2011).
[17] A. Verma, L. Cherkasova, and R. H. Campbell. ARIA:Automatic Resource Inference and Allocation for MapReduceEnvironments. Proc. of the 8th ACM InternationalConference on Autonomic Computing (ICAC’2011), 2011.
[18] X. Wang, C. Olston, A. Sarma, and R. Burns. CoScan:Cooperative Scan Sharing in the Cloud. In Proc. of the ACMSymposium on Cloud Computing,(SOCC’2011), 2011.
[19] J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar,S. Parekh, K.-L. Wu, and A. Balmin. FLEX: A Slot AllocationScheduling Optimizer for MapReduce Workloads. Proc. of the11th ACM/IFIP/USENIX Middleware Conference, 2010.
12