PerfEnforce: A Dynamic Scaling Engine forAnalytics with Performance Guarantees

Jennifer Ortiz†, Brendan Lee†, Magdalena Balazinska†, and Joseph L. Hellerstein‡

†Department of Computer Science & Engineering,‡eScience InstituteUniversity of Washington, Seattle, Washington, USA

{jortiz16, lee33, magda}@cs.washington.edu, jlheller@uw.edu

ABSTRACTIn this paper, we present PerfEnforce, a scaling engine de-signed to enable cloud providers to sell performance levels fordata analytics cloud services. PerfEnforce scales a cluster ofvirtual machines (VMs) allocated to a user in a way that min-imizes cost while probabilistically meeting the query runtimeguarantees offered by a service level agreement (SLA). WithPerfEnforce, we show how to scale a cluster in a way that min-imally disrupts a user’s query session. We further show whento scale the cluster using one of three methods: feedback con-trol, reinforcement learning, or perceptron learning. We findthat perceptron learning outperforms the other two methodswhen making cluster scaling decisions.

1. INTRODUCTIONA variety of systems for data analytics are avail-

able as cloud services today, including Amazon Elas-tic MapReduce (EMR), Amazon Redshift [2], Azure’sHDInsight [4], and several others. While these ser-vices greatly facilitate access to compute resources anddata analytics software, they remain difficult for users totune in terms of cost and performance. Users choose aprice-performance trade-off by selecting a desired num-ber and type of service instances. It is well-known,however, that users have difficulty determining their re-source needs and often attempt many configurations be-fore finding a suitable one [16]. Some systems do notoffer any configuration choices. Google BigQuery [6]is one example. These systems, however, deprive usersof the ability to adjust how much money they want tospend on an analysis at the expense of some loss inperformance. There exist systems that can help selecta cluster configuration [16, 17]. However, these priormethods are specific to MapReduce engines and also re-quire profile runs of each job. In contrast, we target ex-

PerfEnforce!!SLA,Generator

Shared3Nothing,DBMS,

Figure 1: PerfEnforce deployment: PerfEnforce sits on top ofan elastically scalable big data management system in supportof performance-oriented SLAs for cloud data analytics pro-vided by an SLA Generator.

ploratory analytics, where users interactively submit ad-hoc queries and we develop an approach that can easilybe applied to any big data system.

Performance-centric service level agreements(SLAs) [31, 29] have been proposed in responseto the above limitations. With this approach, a userbuys a given performance level (query latency) ratherthan an amount of resources. However, a fundamentalchallenge with performance-centric SLAs, is how toguarantee the performance that the user purchases. Thisproblem is important because, for performance-basedSLAs to be meaningful, they must come with concreteperformance guarantees. For example, the SLA couldspecify that 90 percent of the user queries will executewithin their posted runtime. If the SLA is violated, theuser receives a predefined compensation.

In this paper, we develop a system called PerfEn-force that works with a cloud service to meet the goalsof a performance-based SLA. PerfEnforce is designedfor data management systems that support data analyticworkloads (e.g., Myria [15], Spark [3], Impala [24],EMR [2]). Additionally, PerfEnforce targets cloud ser-vices that follow a model such as that of AmazonEC2 [1] and Azure HDInsight [4], where each user per-forms her analysis using a separate set of virtual ma-chines (VMs). In this paper, we do not address the prob-lem of how to generate a performance SLA, which wasthe focus of prior work including our own [29]. PerfEn-force assumes that the SLA exists and takes as input aset of pairs: (qi, ti), where qi is a query submitted bythe user and ti is the SLA runtime associated with thatspecific query.

Figure 1 shows the system architecture: The user first

1

arX

iv:1

605.

0975

3v1

[cs

.DB

] 3

1 M

ay 2

016

purchases a performance SLA generated by an SLAGenerator. PerfEnforce provisions the cluster of VMson behalf of the user by ingesting the user’s data intothe cluster and monitoring the execution of the user’squeries. To guarantee the query runtimes associatedwith an SLA, PerfEnforce resizes the cluster in betweenqueries either in a proactive or reactive approach. With aproactive approach, PerfEnforce decides to scale basedon how well it met previous SLA deadlines. In a re-active approach, PerfEnforce decides whether to rescalethe cluster before executing each incoming query. Per-fEnforce’s goal is to select the cheapest configurationpossible in order to meet the SLA runtimes.

During the user’s query session, PerfEnforce facestwo key technical challenges: how to rescale the clus-ter and when to rescale it.

Quickly scaling a cluster (either up or down) tomeet SLA guarantees or save costs is not trivial. Re-allocating resources during data analysis can be disrup-tive to the analysis if it requires significant data shuf-fling. At the same time, data replication in preparationfor quick scaling can increase setup costs, which areknown to be highly undesirable [14]. Deployments thatseparate between compute and data nodes to acceleratesetup and cluster configuration changes can either neg-atively impact query runtimes or significantly increasecosts. In this paper, we empirically evaluate a set ofelastic scaling methods and compare them in terms ofinitial setup time, the storage type, time to change thecluster configuration between queries, query executiontime, and total cost. We demonstrate the above chal-lenges associated with inexpensive and rapid scaling andshow that careful data placement and partial replicationoffer a practical solution to the problem.

The second challenge is when to decide to scale thecluster up or down. Several systems have recently stud-ied performance guarantees through dynamic resourceallocation in storage systems [23] using feedback con-trol, or in transaction processing systems [21] using re-inforcement learning. In this paper, we show how toapply feedback control and reinforcement learning tothe problem of query time guarantees for data analytics.We experimentally demonstrate, however, that these ap-proaches do not work well in this context because querytime estimation errors can vary significantly for consec-utive queries and errors can be in either direction (under-or over-estimation of query times). As a result, duringa single user session the system does not converge to asingle cluster size but instead needs to make resource al-location decisions separately for each query. Based onthis observation, we develop a third cluster-scaling al-gorithm. Our approach uses perceptron learning: As theuser executes queries, PerfEnforce continuously updatesits model of query time estimates. Perceptron learning

has the double benefit of quickly adapting to the user’srecent query workload and current system conditions.In addition, we apply this approach without having tobuild an analytical model of the underlying system. Fea-tures of the system are simply fed into the model andquery latencies are adaptively learned. PerfEnforce thenuses this model to select the most appropriate clustersize separately for each query. We show experimentallythat this approach delivers better quality of service andis more cost-effective than either feedback control or re-inforcement learning.

In summary, we make the following contributions:

• We develop PerfEnforce, a dynamic scaling en-gine for data analytics services (Section 3).• We quantitatively evaluate different data place-

ment and cluster re-sizing methods (Section 4).• We adapt well-known resource scaling algorithms

based on feedback control and reinforcementlearning to the problem of query time guaranteesfor data analytics (Section 5.1).• We develop a new resource scaling algorithm

based on perceptron learning (Section 5.2).• We study the performance of the three scaling al-

gorithms through experiments with the Myria [15]shared-nothing DBMS and the Amazon EC2cloud [1] (Section 6).

2. RELATED WORKPerformance Guarantees in Data Analytics Perfor-

mance guarantees have traditionally been the focus ofreal-time database systems [19], where the goal is toschedule queries in a fixed-size cluster to ensure theymeet their deadlines. More recently, dynamic provision-ing and admission control methods have enabled OLAPand OLTP systems to make profitable choices with re-spect to performance guarantees [8, 7, 38], possiblypostponing or even simply rejecting queries. PerfEn-force’s goal instead is to scale the cluster with minimaldelay to meet SLA guarantees.

Multi-Tenant Performance Guarantees An activearea of research in multi-tenant cloud DBMS systemsis tenant packing [11, 26, 25], or how best to colo-cate tenants on a shared set of machines or even DBMSinstances. In contrast, we focus on the independentdatabase user who spins up his own private cluster inthe cloud. We seek to minimize the size of that clusterwhile meeting SLA runtime guarantees.

Query Runtime Prediction Previous work has reliedon classification and regression techniques to determinewhether a query will miss or meet a deadline [38], build-ing gray-box performance models [13], using histori-cal traces of previous workloads [12] or running smallersamples of the workload with a low overhead [36]. Most

2

Table 1: PerfEnforce’s API.

Function name and parameters Returned valueInitialize (D, initc, configs) idQuery (id, q, tsla) voidTerminate (id) void

closely related is work by Herodotou et. al. [16], whichassumes a previously profiled workload from the userin order to predict the runtime of that program againstdifferent sized clusters. Work by Jalaparti et. al. [17] fo-cuses on generating resource combinations given perfor-mance goals from the user. Instead of building a white-box or analytical model, we focus on using a model thatdoes not require an extensive understanding of a singlesystem. We also focus on interactive, ad-hoc queries forwhich there are no prior profiles.

Elasticity Cloud providers offer the ability to scalea database application [2, 4]. However, they requireusers to manually specify scaling conditions throughvendor-specific APIs. This requires expertise and im-poses the risk of resource over-provisioning. Moreover,these scaling features can be costly, as some of these ac-tions are subject to service downtimes and may take sev-eral minutes to complete (such as data rebalancing) [2].

Most academic work on elastic systems focuses onOLTP workloads [9, 34, 37] and thus develops newtechniques for tenant database migration [10], data re-partitioning while maintaining consistency [27] or au-tomated replication [37]. In these systems, the goal isto maximize aggregate system performance, while ourfocus is on a per-query performance guarantees.

3. PERFENFORCE OVERVIEWIn this section, we present an overview of PerfEn-

force: How PerfEnforce interacts with the other com-ponents of a cloud service, what it assumes about thecloud service, what it takes as input, and its internal op-timization goal.

3.1 PerfEnforce APIPerfEnforce is designed to work with a DBMS for

data analytics, an SLA Generator, and a cloud service.Figure 1 illustrates how PerfEnforce interacts with thesecomponents. When a user begins her query session, shefirst purchases a performance-level given by the SLAGenerator. Given the performance-level selected, theSLA Generator provides an initial cluster size to Per-fEnforce, initc, to begin the session. PerfEnforce thenmonitors the query session to rescale if necessary.

PerfEnforce exposes an API with three methods asshown in Table 1. The SLA Generator calls these meth-ods. The Initialize method takes as input theuser’s data D, an initial cluster size initc, and also theset of cluster sizes, configs. This method deploys an

initial set of virtual machines (VMs), starts the DBMS,and ingests the data, D. The method returns a uniquesession identifier, id. Subsequently, each call to themethod Query passes the SQL query, q, to execute inthe session id and the SLA time, tsla associated withthis query. The Terminate call deletes a previouslydeployed cluster. As PerfEnforce scales the cluster dur-ing the query session, it keeps its size within the mini-mum and maximum values specified in the set configs.

In addition, PerfEnforce requires the following func-tionality from the underlying DBMS system: (1) Abilityto add and remove workers dynamically and (2) controlover the way the data is organized in the cluster. Per-fEnforce can still work with a system that does not havecomplete control of the data layout, but this may impactperformance as we explore in Section 4.

3.2 PerfEnforce’s OptimizationGiven a query session Q, with queries q0 through qn

and a set of cluster sizes configs, PerfEnforce optimizeswhat we call the Performance Ratio (PR) of a query ses-sion. We define PR as:

PR(Q) =1

n

n∑q=0

treal(qi)

tsla(qi)(1)

In Equation 1, tsla(qi) and treal(qi) represent theSLA and actual runtimes of a query qi, respectively. Inorder to neither waste cluster resources nor violate SLAruntimes, PerfEnforce’s goal is to maintain PR(Q) asclose to 1 as possible.

In Section 5, we show how different cluster-scaling al-gorithms, yield different treal(qi)

tsla(qi)distributions (See Fig-

ure 9). The best cluster scaling algorithm is one that(1) yields a tight distribution close to 1.0, which ensuresthat most query runtimes stay close to the promised onesfrom the SLA and (2) achieves this goal at a low servicecost. We define the Cost of Service (CS) as:

CS(Q) =

n∑i=0

cost(qi) (2)

For Equation 2, cost(qi) is defined as the cost of vir-tual machines used to execute qi.

4. DATA ORGANIZATIONIn this section, we define and evaluate how to store

data on disk to (1) ingest data quickly in preparation forthe query session and (2) facilitate scaling with mini-mal interruptions during the query session. PerfEnforcetargets cloud services that execute the data managementand analytics software in a separate set of VMs for eachtenant. In that context, data can be stored in the lo-cal storage of each VM or in a separate storage systemavailable over the network.

3

When a user starts her query session, PerfEnforce pre-pares an initial set of initc VMs. Additionally, PerfEn-force prepares the system to resize itself to any clustersize in the set given by configs.

PerfEnforce has many choices to organize the user’sdata and scale resources up and down. First, we intro-duce the available storage types in Amazon AWS andevaluate them on data ingest and data read times. Wethen present different data placement methods and eval-uate them based on latency to first query and disruptiondue to cluster resizing.

4.1 Local, Networked, and Shared StorageA big data system can read and write data from a va-

riety of sources once a set of VMs is provisioned. In theconcrete case of the Amazon cloud, these include, butare not limited to, Amazon Simple Storage Service (S3),Amazon EC2 Instance Store (Ephemeral) and AmazonElastic Block Storage (EBS) [1]. To evaluate these stor-age options, we provision one m3.large node (4 ECU,7.5 GB Memory) and read and write the lineorder tablefrom the TPC-H Star Schema Benchmark (SSB) [28]dataset. For EBS, we use a general purpose SSD (gp2)type. For S3, we make sure that the S3 bucket and VMare within the same region. Figure 2a shows the timeto ingest data. There is no ingest time for S3 as nodescan read data directly from that storage service duringquery execution. Ingest times are nearly identical forthe other two storage systems. Figure 2b and Figure 2cshow the time to read data. While reading an entire tabletakes the same amount of time between ephemeral andEBS, reading a subset, such as one column, is signif-icantly faster when using ephemeral storage comparedto EBS. Reading tables from S3 takes slightly longerand it is not possible to read a single data column fromS3. Kossmann et. al. [22] also report, though in thecontext of OTLP workloads, that EBS and ephemeralstorage achieve similar performance. Compared to S3,both ephemeral and EBS have the additional advantageof caching data locally during the query session andperforming local joins without having to reshuffle datawhen tables are partitioned on their join attribute. Tominimize CS(Q) and maximize query performance, weopt to use ephemeral storage since its price is included inthe VM price and its performance is highest when read-ing subsets of the data. In the remainder of the paper,we use only ephemeral storage. We show next how toensure that data ingest and cluster reconfiguration timesboth remain low with this storage option.

4.2 Data Placement StrategiesPerfEnforce replicates small dimension tables across

all workers (a.k.a. nodes) while partitioning large facttables. Small tables take a negligible amount of time to

copy over to a new worker. As such, any approach forcluster scaling works with small tables. The question ishow to best manage cluster scaling for large tables.

Workers responsible for reading data constitute thedata storage layer of the system. The compute layerare the workers that execute query operators such asjoins or aggregates. We first consider the case whereeach worker serves as both a data and a compute node:i.e., when running queries with N workers, each workerstores and processes 1

N th of the data.Shuffled-Scaling In this method, each large table is

first uniformly partitioned across the initial set, initc,of workers using hash-, range-, or random data parti-tioning. To resize the cluster to a different configura-tion c′, PerfEnforce issues a query that reads the table,shuffles it, and re-materializes it across the updated setof c′ workers. An important optimization is for work-ers to reshuffle only the minimal amount of data neededto rescale. This can be done by using consistent hash-ing [20] or simply using mini partitions as follows: LetPR = {pr0 , pr1 , ..., prn} represent the partitions of rela-tion,R, where n is the number of nodes in configurationinitc. Each partition, pri is assigned to one node fromconfiguration initc and is further split into j mini parti-tions. In order to scale from initc to c′, each node needsto only read and shuffle a fraction of its mini partitions.For example, when resizing from 2 to 4 workers, each ofthe original two workers must reshuffle half of its minipartitions across the two new workers.

Static-Replicated To avoid data re-shuffling uponcluster rescaling, PerfEnforce can ingest multiple copiesof each big table. Each copy is uniformly partitionedacross a subset of machines that corresponds to one con-figuration in configs. For example, one copy of a tableis partitioned across four workers, a second copy is par-titioned across six workers, a third across eight, etc.

As an optimization, instead of ingesting multiple fullcopies of each big table in sequence for each configu-ration in configs, PerfEnforce can, once again, use ei-ther mini-partitions or consistent hashing to only repli-cate a minimum amount of data. For example, assumeconfigs = {2, 4}. PerfEnforce first partitions relationR across four workers as PR = {pr1 , pr2 , pr3 , pr4}. Togenerate a 2-worker partition, PR′ , PerfEnforce copiespr3 and pr4 onto workers r1 and r2 respectively. We callthis approach Static-Replicated Chunks.

Dynamic-Scaling The final approach distinguishesbetween sets of compute nodes, Ccompute, and datanodes, Ddata. PerfEnforce uniformly ingests the tablesinto the number of assigned data nodes,Ddata. The datalayer remains fixed and never changes in size. Instead ofre-materializing a table for a new configuration c′, Per-fEnforce only reads data from Ddata, and shuffles thedata to the Ccompute nodes (s.t. |Ccompute| = c′ ) in or-

4

500MB 1GB 2GB 3GB 4GB 5GB0

100

200

300

400

500Seco

nds

Ephemeral Storage

EBS Storage

(a) Time to Ingest Data

500MB 1GB 2GB 3GB 4GB 5GB0

20406080

100120140

Sec

onds

Ephemeral StorageEBS StorageS3 Storage

(b) Time to Read a Table

500MB 1GB 2GB 3GB 4GB 5GB0

20406080

100

Sec

onds

Ephemeral StorageEBS Storage

(c) Time to Read a Column

Figure 2: Evaluating Storage Options

der to finish the computation of the query. We considertwo cases of Dynamic-Scaling: |Ddata| < |Ccompute|and |Ddata| > |Ccompute|, which we call Dynamic-Small and Dynamic-Large respectively. For example,when provisioning a Dynamic-Small cluster, the systemcan fix the number of data nodes to 4 workers and onlyscale the number of compute nodes to range from 5 to10 nodes. This can be advantageous if the user workloadis CPU-bound. In a Dynamic-Large cluster, the systemspreads the data thinly to many data nodes, which wouldthen shuffle data to a smaller number of compute nodesto finish the query computation. Keeping the data thinis beneficial particularly for IO-bound workloads.

Among the three techniques above, Shuffled-Scalingrisks imposing high overheads when changing betweencluster configurations. Static-Replicated scaling risksslowing down the initial data ingest time. Dynamic-Scaling is more costly (as one has to pay for both dataand compute nodes). We evaluate these techniques next.

4.3 Data Placement EvaluationWe run PerfEnforce on an Amazon EC2 cluster.

Each node is an m3.large (4 ECU, 7.5 GB Memory)type. We consider five types of possible configura-tions, configs = {4, 6, 8, 10, 12}. For our underly-ing database management system, we use Myria [15] asit provides the ability to easily control data placement.Myria uses PostgreSQL as its node-local storage sub-system.

For our dataset, we use the TPC-H Star SchemaBenchmark (SSB) [28]. This dataset consists of onefact table (lineorders) and a set of four smaller dimen-sion tables. In total, the dataset is approximately 10GB,containing 5 tables and 58 attributes. We choose thisdataset size because multiple Hadoop measurement pa-pers report 10GB as a median input dataset analyzed byusers [33]. For our query pool, we generate a set ofapproximately 900 select-project-join queries using ouropen-source PSLAManager tool [29].

Data Ingest Runtime Given that PerfEnforce op-erates as a cloud service, it must prepare and ingestthe data efficiently in order to allow the user to be-gin the query session quickly. We consider the time ittakes for each scaling method to ingest the TPC-H SSBdataset. In Figure 3, we display the runtimes for in-gesting data using Static-Replicated, Static-ReplicatedChunks, Dynamic-Small or Dynamic-Large methods.

Static-Rep.Static-Rep. Chunks

Dynamic-Small (Fixed at 4)

Dynamic-Large (Fixed at 12)0

100200300400500600700

Sec

onds

Figure 3: Time to Ingest Data for Scaling Methods

Ingesting for Static-Replicated takes approximately606 seconds. This method takes the longest as it re-quires five copies of the lineorder table. For Dynamic-Scaling, we show the ingest runtimes for 4 and 12 fixednodes. Ingesting data for the Static-Replicated Chunksmethod is comparable to ingesting data for the small-est configuration in configs (4 workers). The Shuffled-Scaling method (not shown) takes the same time as ei-ther Dynamic-Small or Dynamic-Large, depending onthe number of nodes in initc.

In general, the bottleneck for ingest time largely de-pends on either the fixed number of data nodes selectedfor Dynamic-Scaling or the smallest configuration sizethat exists in configs for Static-Replicated Chunks.Most importantly, the latter method provides the ben-efit of a replicated set of tables without the data ingestoverhead associated with full data replication.

Delay When Changing Between ConfigurationsAnother factor to consider is the time it takes toswitch between configurations in configs. For Static-Replicated and Static-Replicated Chunks, the multiplecopies of the data allow for immediate scaling. ForDynamic-Scaling, the only factor that needs to changeare the number of compute nodes. For both of thesescaling methods, there is no delay when scaling the sys-tem, as no data materialization is required when switch-ing between configurations.

Shuffled-Scaling must re-organize the data beforerunning the next query. In Figure 4a, we show theamount of time it takes for a configuration of 4 work-ers to move data to a configuration c′. Figure 4b showsthe amount of time it takes to change from a set of 12worker nodes to c′. We evaluate two approaches toswitch between configurations for Shuffled-Scaling. Asa first method, PerfEnforce reads the entire table fromdisk in c, shuffles the data, and writes it to c′. We callthis approach Read & Write Full Table from Disk. Asa second method, the system uses the optimization de-

5

4-to-6 4-to-8 4-to-10 4-to-120

50

100

150

200S

econ

dsRead & Write Chunks From DiskRead & Write Full Table From Disk

(a) Time to Change Configurations from 4 Workers

12-to-4 12-to-6 12-to-8 12-to-100

50

100

150

200

Sec

onds

Read & Write Chunks From DiskRead & Write Full Table From Disk

(b) Time to Change Configurations from 12 Workers

Figure 4: Configuration Change Runtimes

scribed above that only reads and writes the minimumamount of data, which we denote with Read & WriteChunks From Disk. The longest configuration switch isfrom 12-to-4, which implies that the bottleneck is in thetime it takes to write data to disk. In all cases, how-ever, data reshuffling creates a visible interruption in thequery session.

Query Processing Time So far, we showed thatShuffled-Scaling imposes too much overhead duringcluster resizing while Static-Replicated can take along time to ingest data. Here, we compare the re-maining competitive methods on their query execu-tion times. Figure 5 shows the query runtime ratiosfor 100 randomly selected queries from our generatedpool of queries. The ratios measure the query timefor Dynamic-Small (4 data and 8 compute nodes) andDynamic-Large (12 data and 8 compute nodes) com-pared with Static-Replicated Chunks (8 nodes shown as8-to-8 static in the figure). In this experiment, we onlymeasure the computation time for each query and donot flush the query results to disk. As the figure shows,Dynamic-Small leads to slower query runtimes than theStatic-Replicated method. In general, we observe inour experiments (results not shown due to space con-straints) that a small set of data nodes can easily becomea bottleneck and nullify any benefit of scaling computenodes. In contrast, Dynamic-Large has excellent per-formance. This latter method, however, is expensive.We find that it delivers high performance only whendata nodes use powerful VMs. When using cheapernodes, the data nodes become a bottleneck again (resultsnot shown). However, if a data node uses a powerfulVM, it has the capability to also run as as a computenode. Shuffled-Scaling, Static-Replicated and Static-Replicated Chunks already co-locate compute and datanodes, and thus are more cost effective compared to dy-namic methods.

Cost of Virtual Machines At the start of the querysession, PerfEnforce launches the number of VMs nec-

0 10 20 30 40 50 60 70 80 90 100Query

0

1

2

3

4

5

Runti

me R

ati

o 8-to-8 Static

4-to-8 Dynamic-Small

12-to-8 Dynamic-Large

Figure 5: Time to run a random set of TPC-H SSB queries instatic and dynamic clustersessary to meet all configuration options in configs.PerfEnforce can help minimize CS(Q) without sig-nificantly penalizing query performance by turning offVMs that are not in use. The time it takes to launch anew virtual machine in PerfEnforce depends on the sizeof the virtual machine. For an m3.large machine, it takesapproximately 17 seconds to launch a machine with anAmazon Linux AMI. Turning off the machine takes 27seconds on average. Turning a machine back on takesonly approximately 10 seconds.

Data Organization Summary Although Dynamic-Scaling can have the lowest data ingest times, this op-tion is costly and risks slowing down query processingif an insufficient number of data nodes are selected asshown in Figure 5. Given this result, the best optionis for PerfEnforce to use the Static-Replicated Chunksscaling method as it provides a quick way to ingest dataand does not incur any runtime penalties when switch-ing between configurations.

5. SCALING ALGORITHMSIn this section, we consider both reactive and proac-

tive methods for PerfEnforce to rescale the user’s clusterduring her query session. We introduced these methodsinitially in a short, demonstration proposal [30]. Thecontribution of this paper lies in the actual study of thesemethods. The goal of these scaling methods is to main-tain PR(Q) as close to 1.0 as possible.

5.1 Reactive Scaling AlgorithmsWe first describe reactive scaling algorithms. These

algorithms take action after they witness either a good orbad event. In PerfEnforce, we implement proportionalintegral control and reinforcement learning as our reac-tive methods because these methods have successfullybeen used in other resource allocation contexts [23, 21].

Proportional Integral Control (PI) Feedback con-trol [18] is a commonly used approach to regulate a sys-tem in order to ensure that it operates at a given ref-erence point. We use a proportional-integral controller(PI) as a method that helps PerfEnforce react based onthe magnitude of the error while avoiding oscillationsover time.

At each time step, t, the controller produces an ac-tuator value u(t) that causes the system to produce anoutput y(t+ 1) at the next time step. The goal is for the

6

system output y(t) to be equal to some desired referenceoutput r(t). In an integral controller, the actuator valuedepends on the accumulation of past errors of the sys-tem. This can be represented as u(t+1) = u(t)+kie(t).Where e(t) = y(t)− r(t), with y(t) being the observedoutput and r(t) being the target system output. ki rep-resents the gain of the integral control. Ideally, this pa-rameter is tuned in such a way that helps drive e(t) to0. In our scenario, the actuator value u(t) is the discretenumber of VMs provisioned.

As for the system output, y(t), we use the averageratio of the real query runtime treal(q) over the queryruntime promised in the SLA, tsla(q), over some timewindow of queriesw as y(t) = 1

|w|∑q∈w

treal(q)tsla(q)

where|w| is the number of queries in w.

Our target operating point is thus r(t) = 1.0 and theerror e(t) = y(t)−r(t) captures a percent error betweenthe current and desired average runtime ratios. Sincethe number of VMs to spin up and remove given sucha percent error depends on the cluster size, we add thatsize to the error computation as follows: e(t) = (y(t)−r(t))u(t).

Integral control alone may be slow to react to changesin the workload. Therefore, we also introduce a propor-tional control component, where kp represents the gainof the proportional error. Our final PI controller thustakes the following form:

u(t+ 1) = u(0) +

t∑x=0

kie(x) + kpe(t) (3)

Reinforcement Learning (RL) As our second re-active method, we use reinforcement learning (RL).This approach has successfully been applied in theTIRAMOLA system, which supports elastic scaling ofNoSQL databases [21].

At each state s, the model makes a probabilistic de-cision to move to another state s′ by taking an actiona. In our case, each state represents a configuration inconfigs and the action is to change to that configura-tion. The goal is to make a series of beneficial action-state moves, as motivated by the rewards at each state,R(s). To explore the search space and learn the optimalaction-state paths, reinforcement learning uses a tech-nique known as Q-learning [35]:

Q(s, a) = Q(s, a)+α[R(s′)+γmaxaQ(s′, a′)−Q(s, a)]

(4)In Equation 4, Q(s,a) is the reward for taking action

a from state s. It is a function of the reward at states′ reached by taking action a and of the actions that cansubsequently be taken from s′. α represents the learningrate, which controls how fast the learning takes place.

At convergence, Q-learning is able to find an optimalaction-state path. In PerfEnforce, our goal is different.Since a user’s query workload is constantly changingthroughout the session, recording the action-state path isunnecessary. Instead, PerfEnforce directly transitions tothe state with the highest reward. We define the rewardfunction to be the real-to-SLA runtime ratio. At eachiteration, we favor states with the reward closest to 1.0,where the real query runtimes are closest to the SLAruntime. As the system transitions to a state s, it updatesthe reward function for that state. We use the followingequation, where R(s) denotes the updated reward forstate s:

R(s) = α ∗ ( treal(q)tsla(q)

−R(s)) +R(s) (5)

At the initialization of the model, each state must be-gin with a defined reward value, R(s). This impliesthat the system must have prior knowledge of the per-formance of the user’s queries for each configuration.Since we do not have such prior knowledge, we setthe reward at each state to 1.0 and force the system tofirst explore states that are closest to initc. To do this,we maintain a set of states called active states. Whenthe query session begins, active states only contains theconfiguration initc. If the reward for the current stategoes above 1.0, we add the next larger cluster size to theactive states. If the reward for the current state goes be-low 1.0, we similarly add the next smaller cluster size.We repeat the process until all possible cluster sizes havebeen added.

Additionally, we observe that rewards for some statesdo not quickly adapt if the user’s workload changes.For example, if a slow query runs on configuration cand misses the deadline, the reward will be updated toa value above 1.0. If a new fast query is introduced,c will not be chosen as the current reward (above 1.0)suggests that the query will miss the deadline. There-fore, as a heuristic, we introduce a linear-drag update.Each state whose reward was not modified by Equa-tion 5 (denoted as state x), receives the following up-date: R(x) = β ∗ ( treal(q)

tsla(q)∗ yz −R(x)) +R(x). Where

β < α and z represents number of VMs of state x. y isthe number of VMs in state s from Equation 5.

5.2 Proactive Scaling AlgorithmsInstead of approaches that react to runtime errors such

as PI and RL, we also explore an approach that makesuse of a predictive model. For each incoming query,PerfEnforce predicts the runtime for the query for eachconfiguration and switches to the closest configurationwhere the ratio is closest to 1.0.

PerfEnforce first builds an offline model for a givencloud data analytics service. For training data, we use

7

the Parallel Data Generation Framework tool [32] togenerate a 10GB dataset with a set of 6120 queries.These training queries are based on the query generatorprovided by the open-source PSLAManager tool [29].Training data consists of query plan features includingthe estimated max cost, estimated number of rows, esti-mated width, and number of workers.

Initially, such an offline model is expected to be inac-curate. However, we can adaptively improve the modelif we incorporate information about the queries the userexecutes on his data. We achieve this goal by using aperceptron learning model: as the user executes queries,PerfEnforce improves the model in an online fashion.We use the MOA (Massive Online Analysis) tool forlearning [5].

Perceptron Online Machine Learning (OML) Theperceptron learning algorithm works by adjustingweights for each new data point. We find that it adaptsmore quickly to new information than an active-learningbased approach. PerfEnforce initiates the perceptronmodel by first learning from the training set. For an in-coming query, PerfEnforce uses this model to predict aruntime for each configuration in configs. The clus-ter size with the closest runtime to the incoming query’sSLA is chosen. Once the system runs the query andlearns about the real runtime, it feeds this informationback into the model. If we predict in parallel for allconfigs, the process takes less than 1 second.

6. SCALING ALGORITHMSEVALUATION

We now evaluate both reactive and proactive scalingalgorithms. We first execute a series of microbench-marks designed to demonstrate fundamental character-istics of each algorithm. We then evaluate each algo-rithm on macrobenchmarks consisting of random work-loads. We keep the same experimental setting as be-fore where we have configs = {4, 6, 8, 10, 12} and theTPC-H SSB dataset. We run queries from our querypool for each cluster configuration and record the exe-cution times. We label each query with the cluster con-figuration that is able to run the query at the runtimeclosest to the query’s SLA. For all experiments, we setinitc to 4.

6.1 Scaling Algorithms MicrobenchmarksFor each algorithm, we consider the following: (1)

How fast does the algorithm converge to a different con-figuration if the current configuration is either too smallto meet the SLA times or is unnecessarily too large?How fast does the algorithm react to a workload changethat requires a different cluster size? (2) How stableis the algorithm in the face of occasional queries thatwould require either a smaller or larger cluster than the

rest? (3) How well does the algorithm handle an oscillat-ing workload where the ideal cluster size is different forconsecutive queries? We use the following three work-loads to help answer these three questions: (1) Micro-W#1: Convergence Speed. (2) Micro-W#2: Stability.(3) Micro-W#3: Workload tracking.

All three scaling algorithms have tuning parameters.In this section, we show the performance of reinforce-ment learning (RL) and PI-control (PI) with best param-eters chosen separately for each workload. The selectedparameters overfit the workload. We describe how weselect the best parameters for PI and RL in Section 6.3.For perceptron online machine learning (OML), we se-lect a learning rate of 0.04. In Section 6.2, we discusshow to tune OML. For Micro-W#1 and Micro-W#2, wealso include an additional overfit line for a differentworkload (shown in blue). The goal is to show howtuning parameters for one workload do not necessarilybenefit scaling for other workloads. We do not show thisfor OML since we use the same parameter value for allworkloads.

Micro Workload 1- Convergence Speed In this firstworkload, we evaluate the speed of convergence foreach technique on the workload shown in Figure 6a. Thesystem starts at initc, a 4-worker configuration. Thequery sequence begins with a set of 10 queries whoseSLA deadline is best met at 12 workers. This is thenfollowed by a set queries whose SLA is more closelymet at 4 workers. We repeat this pattern for a total of50 queries. For PI, the model immediately scales to thelargest cluster size after running the first query in the se-quence. PI is able to converge because we tune the PIcontroller to react quickly with a kp value of 100. Al-though kp is high, it does not oscillate once it convergesto 12 workers since e(t) turns out to be positive for eachof these queries (recall, we initially select queries whoseSLA is best met at 12 workers, but might not necessarilymeet the guarantee at this configuration size). Neverthe-less, there exists a lag between the workload change andthe PI’s reaction to that change. In contrast, OML is ableto track the workload exactly as it correctly predicts therequired cluster size. For RL, scaling does not happen asquickly as seen in PI, as states are incrementally addedto the activeStates set. Therefore, convergence for thefirst set of queries does not occur until the 5th query.Linear-drag updates still take place in this workload, butthe sequences between slow and fast queries are not longenough to be able to see its effect. As a result, RL re-mains in one state.

Micro Workload 2- Stability For this workload, weshow the stability of the scaling algorithms in situationswhere the system runs a fast query among a long se-quence of queries whose guarantee is best met at 12workers. The results are shown in Figure 6b. For the

8

Overfit for Micro-W#2 Overfit for Micro-W#2

(a) Micro-Workload #1: Convergence Speed

Overfit for Micro-W#1 Overfit for Micro-W#1

(b) Micro-Workload #2: Stability in the Face of one Different Query

0 10 20 30 40 50Query

02468

101214

Clu

ster

Siz

e

PI Control

kp = . 5, ki = 0, w= 2 Ideal

0 10 20 30 40 50Query

02468

101214

Clu

ster

Siz

e

Reinforcement Learning

α= . 1, β= . 1 Ideal

0 10 20 30 40 50Query

02468

101214

Clu

ster

Siz

e

Online Learning

LR= . 04 Ideal

(c) Micro-Workload #3: Tracking of a Rapidly Changing Workload

Figure 6: Micro-Benchmarks

PI controller, the best settings are those where ki or wvalue is large. With these settings, the PI controller isstable in face of outlier queries. Observe, however, thatthe PI controller settings are now different compared toW#1. As we still show in Section 6.3.2, the PI con-troller is highly sensitive to its parameter settings. ForRL, we see a similar behavior. The ideal setting usesa high α parameter, the rewards for configurations run-ning the first few queries are updated to a high ratio,since they all miss the SLA deadline. This then makes itdifficult for the model to quickly scale back down laterin the sequence. Once the fast query runs at 12 workers,the model updates the reward for this configuration, butthis state continues to be the closest to 1.0. OML is ableto determine the ideal cluster configurations before run-ning each query. In general, a one-time-only query witha different ideal cluster size does not negatively impactthe result for any method. In this example, PI and RLonly end up over provisioning for these fast queries.

Micro Workload 3 - Workload tracking Finally, wedemonstrate how well each method is able to keep upwith a rapidly changing workload. We demonstrate thisthrough a sawtooth workload where we first run a mixof queries whose SLA is best met at 12 and 10 work-ers, followed by a mix of queries whose SLA is met at4 and 6 workers as shown in Figure 6c. For PI, oneof the parameter combinations that work best for this

workload are kp = .5, ki = 0, w = 2. The model im-mediately scales up after the first window of queries.Similarly to the first workload, PI continues to scale bysimply reacting to the error due to the kp parameter. RLscales to the highest cluster size as before. It temporar-ily scales down to a configuration of 10 queries thanks tolinear-drag, but the model quickly scales back up sinceit under-provisions the 20th query. In OML, the modelover provisions for several queries, by at most one con-figuration size.

In general, reactive methods are able to converge andeven recover in the presence of a sudden change in theworkloads. However, they are difficult to tune especiallyfor rapidly changing workloads. OML is able to keepup with rapidly changing workloads given that it is ableto observe the features for the upcoming query beforechoosing the best cluster configuration.

6.2 Perceptron Learning TuningWe now discuss how to find an optimal learning rate

for OML. Recall, the learning rate for OML determineshow quickly the model adjusts the weights for differ-ent features. If the learning rate value is too low, themodel might not quickly adapt to the user’s new queries.If it is high, learning is faster, but there is a risk thatthe model might never converge, as it will tend to jumpover the optimum. To evaluate the sensitivity, we firsttake three random sets of 100 queries from the TPC-H

9

0.00 0.02 0.04 0.06 0.08 0.10 0.12

Learning Rate

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Avg.

Rela

tive R

MSE

First Dataset

Second Dataset

Third Dataset

Figure 7: Tuning for OML based on the TPC-H SSB Dataset

SSB dataset. For each of these test sets, we also pre-pare a separate group of 400 holdout queries from thesame dataset. For offline training, we use a total of 6120queries from the Parallel Data Generation Frameworkdataset [32].

We first select a learning rate and a test set. For eachquery in the test set, we add it to the training model,update the model, and evaluate the model on the cor-responding holdout set of queries to collect the relativeroot mean squared error (relative RMSE). Once we eval-uate against all the queries in the test set, we calculatethe average relative RMSE. We repeat this process formany learning rates. Figure 7 shows the resulting aver-age relative RMSE on the y-axis for different learningrates on the x-axis. As the figure shows, for all datasets,the learning rate with the lowest average relative RMSEis approximately 0.04, which is the value that we use inall other experiments in this section. Importantly, how-ever, a large range of learning rates [0.02, 0.07] yieldsimilar prediction quality.

6.2.1 Effects of Caching and ContentionIn the previous section, tuned for OML based on a

cold-cache environment. We evaluated a cold-cacheset of test queries against a cold-cache training model,ctrain. In a real query session, PerfEnforce will not clearthe cache after each query runs.

In practice, queries should execute faster in a warmenvironment. We briefly evaluate the effects of datacaching with query runtime predictions. We first gener-ate a warm-cache training model, wtrain in order to ob-serve if the wtrain model outperforms the ctrain modelwhen it comes to predicting the runtimes in a warmquery session. To record the runtimes for the offlinewarm training and testing models, we run each querytwice and only record the runtime of the second query.Figure 8a shows the prediction error for a set of 100queries. We evaluate ctrain against three versions of thetest queries: cold-cache runtimes, warm-cache runtimesand with a 20% additional time (to model contention)above the cold-cache runtime. In general, ctrain is ableto improve the predictions over time for all three sets ofqueries. However, wtrain does not perform as well. Asshown in Figure 8b, the error starts off higher than ctrainat approximately a Relative RMSE of 0.7. Second, al-

0 20 40 60 80 100

Query

0.00.10.20.30.40.50.60.7

Rela

tive R

MSE Cold Runtimes

Warm Runtimes

20% Over Runtimes

(a) Prediction Errors based on ctrain

0 10 20 30 40 50Query

0.00.10.20.30.40.50.60.7

Rela

tive R

MSE Cold Runtimes

Warm Runtimes

20% Over Runtimes

(b) Prediction Errors based on wtrain

Figure 8: Prediction Errors between ctrain and wtrain

though wtrain is able to achieve a low Relative RMSEfor warm test runtimes, there is more significant errorfor cold test runtimes. Overall, an offline model trainedon cold-cache queries is more resilient and can adapt topredicting either cold-cache or warm-cache query times.

6.3 Scaling AlgorithmsMacrobenchmarks

In this section, we focus on the performance of thescaling algorithms on random workloads. We seek toanswer two questions: (1) On random workloads, howwell do different techniques manage to operate at thedesired PR(Q) = 1? (2) What is the cost of operatingat the given set point?

For RL and PI, we first show the performance whenselecting the best parameter settings separately for eachworkload. We call this variant overfitted, since the pa-rameters are completely overfitted to the workload andthus change for each sequence of queries. To find over-fit parameters, we use information from an Oracle. TheOracle is an additional technique that holds all knowl-edge of queries and their corresponding ideal config-urations. The Oracle executes the same workload ofqueries, picking the best cluster configuration for eachquery. We compute the PR(Q) for the workload as exe-cuted by the Oracle. We then iterate through all possiblecombinations of parameters for each technique. For RL,we iterate through α rates from 0 to 1.0 where β = α

dand we vary d from 1 to 100. For PI, we vary kp andki values from 0 to 100, with varying window sizes, w,from 1 to 100. For each parameter combination, we ex-ecute the technique on the given workload. We showthe results for the parameter combination that yields aPR(Q), closest to that of the Oracle. For OML, wecontinue to use the optimal learning rate of 0.04.

6.3.1 Ratio Distributions for Random Workloads

10

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=0.87σ=0.53

PI-ki = 0, kp = 11, w= 15

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=0.87σ=0.47

RL- α= . 8, β= . 05

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=0.85σ=0.32

OML- LR= . 04

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=0.69σ=0.38

Random

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=0.87σ=0.2

Oracle

(a) Workload: With Convergence at 4 Workers

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=1.15σ=1.69

PI-ki = 1. 5, kp = 0, w= 1

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=1.12σ=1.44

RL- α= . 9, β= . 05

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=1.14σ=0.94

OML- LR= . 04

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=1.34σ=1.72

Random

4 3 2 1 0 1 2 3 4

Ratio

0.0

0.2

0.4

0.6

0.8

1.0

Den

sity

µ=1.14σ=0.93

Oracle

(b) Workload: No-Convergence

Figure 9: Ratio Distributions of Random Workloads: Distributions of treal(q)tsla(q)

ratios for each technique on two random work-loads. RL and PI use overfitted parameters. OML uses a learning rate of .04

We first show the distributions of query runtime ra-tios for two concrete random workloads. Each workloadcomprises 100 random queries from our set of TPC-HSSB queries.

Figure 9a shows the ratio distribution of each tech-nique for the first random workload. The x-axis showsthe, treal(q)

tsla(q)ratio. The y-axis shows the density for each

ratio (i.e., the fraction of queries with that ratio). In ad-dition to the Oracle’s distribution, we also show the dis-tribution for a random technique. The random techniquesimply selects a random cluster size to run each query.As the figure shows, the Oracle’s distribution has an av-erage of 0.87, which implies that there are many queriesin the workload where the smallest cluster size availablein configs does not closely meet the query’s SLA. Thesystem could scale down further, but it does not becausewe set the lower limit at four workers.

For PI and RL, we show the distributions based on theoverfit parameters. Both RL and PI run queries that attimes are 2x or 3x slower than the query’s SLA guar-antee. OML is able to follow the Oracle’s distributionmore closely with most of the queries falling betweenthe ratios 0.53 and 1.17. We find that for this randomworkload in particular, many of queries are able to meettheir SLA at a 4 worker configuration. This provides anopportunity for RL and PI to converge to this configura-tion size for a majority of the queries. We refer to thisworkload as the Convergence workload. For this work-load, all techniques achieve a PR(Q) close to that of theOracle. However, the standard deviation is much largerfor PI and RL than for OML.

For the second random workload, we only selectqueries whose SLA is not met at 4 workers. We referto this workload as No-Convergence. We show the dis-tributions for No-Convergence in Figure 9b. The Ora-cle distribution has a higher average and standard devi-

0.2 0.4 0.6 0.8 1.0 1.2

PR(Q)

020406080

100120

CS(Q

)

OMLOracle

PIRLOMLOracle

(a) Parameter Sensitivity for Convergence Workload

1.0 1.2 1.4 1.6 1.8 2.0

PR(Q)

0

50

100

150

200

CS(Q

)

OML Oracle

PIRLOMLOracle

(b) Parameter Sensitivity for No-Convergence Workload

Figure 10: Parameter Value Sensitivity: Each point representsan execution with different parameter values.

ation for this workload. This is due to some queries inthe workload not being able to meet their SLAs even atthe largest configuration size. The overfit parameters forPI and RL are not as close to the Oracle distribution asfor the previous workload. Standard deviations are evenhigher. For both of these techniques, there are queriesthat run up to 3x slower than their assigned SLA run-time. OML is able to produce a similar distribution asthe Oracle.

6.3.2 The Parameter Search SpaceFor the Convergence and No-Convergence workloads,

we previously showed only the distributions for over-fit parameter values. We now show a summary of thedistributions for different parameter values. Figure 10aand Figure 10b show PR(Q) vs. CS(Q) for the result-ing distributions. We also included the resulting distri-bution for OML (based on a learning rate of .04) andthe Oracle. As the figure shows, both PI and RL arehighly sensitive to their parameter settings. Wrong set-

11

PI RL OMLScaling Algorithm

0.00.20.40.60.81.01.21.41.6

PR

(Q)

Oracle′ sPR

(Q)

1)PI RL OML Oracle

Scaling Algorithm

0.00.51.01.52.02.53.03.54.04.5

Rel

ativ

e S

TDE

V

of R

atio

s

2)PI RL OML

Scaling Algorithm

0.00.20.40.60.81.01.21.41.6

CS(Q

)

Oracle′ sCS(Q

)

3)

Figure 11: Performance for Techniques across Many Workloads using Average Parameter Settingstings can yield high query time ratios or high costs (mea-sured in terms of virtual machines). Additionally, wefind that the best settings vary significantly across work-loads. These techniques are thus impractical in our set-ting, where the workload is unknown until the user startsissuing queries.

6.3.3 Performance for Many Random WorkloadsWe evaluate each of the techniques on a set of ten

workloads. Five of those workloads are completely ran-dom, two workloads are random but comprise only largequeries without selection predicates, one workload com-prises only queries that select 10% of the data, and the fi-nal two workloads either comprise a majority of queriesthat have an ideal cluster of size 4 or an ideal cluster ofsize 12. For OML, we continue to use the same learn-ing rate for each workload. For PI and RL, we first findthe overfit parameter values for each workload and thencompute an overall average value for each parameter.We show the performance of the techniques on the aver-age settings, since in practice, the system cannot predictthe workload and set the optimal parameter values foreach workload.

Figure 11 shows the results of using these average pa-rameter values. In Figure 11 (1), we show the distribu-tion of PR(Q) across the ten workloads for each tech-nique relative to the Oracle’s PR(Q). In Figure 11 (2)we show the distribution of the relative standard devia-tions across the workloads for each technique. The rel-ative standard deviation is taken by calculating the stan-dard deviation of the resulting query ratios ( treal(qi)

tsla(qi)) for

each workload and dividing it by the mean. Finally, Fig-ure 11 (3) shows the distribution of the cost of service(CS(Q)) across all workloads for each technique rela-tive to the Oracle’s CS(Q). As the figures show, OMLyields ratio distributions closest to those of the Oracle:Both the PR(Q) and relative standard deviations arecloser to those of the Oracle compared with PI and RL.Both PI and OML yield similar CS(Q) to the Oracle.

6.4 From QoS to SLAIn the previous sections, we evaluated the elastic scal-

ing algorithms in terms of how well they enable thesystem to operate close to the desired set point wherePR(Q) = 1. In this section, we discuss how the re-sults can translate into a concrete SLA. As we showed

W#1 W#2 W#3 W#4 W#5 W#6 W#7 W#8 W#9 W#1075% .82 1.21 .83 .93 1.39 1.11 1.24 1.16 1.04 1.1780% .99 1.24 .85 .99 1.47 1.18 1.27 1.21 1.14 1.2085% 1.18 1.35 1.00 1.06 1.65 1.21 1.43 1.25 1.22 1.3090% 1.22 1.56 1.18 1.11 1.81 1.33 1.66 1.42 1.31 1.37

Table 2: Percentile of Ratios in OML

above, each cluster scaling algorithm produces a distri-bution of these ratios around the desired set point andwe showed that OML yields a distribution close to thatof the Oracle without difficult parameter tuning. In Ta-ble 2, we show the resulting query time ratios for dif-ferent percentiles in the distributions obtained for the 10random workloads. As the table shows, the ratios areconsistent across the workloads and are relatively closeto the desired set point. A cloud provider can thus usePerfEnforce with the OML scaling algorithm and adver-tise a probabilistic SLA, where the cloud provider in-creases the estimated query runtimes by some weight wand then promises thatX% of the queries will meet theirSLA runtimes. That is, the SLA will promise fewer than1−X% SLA violations in a session. Considering the ta-ble, possible SLAs include advertising query times thatare w = 2 times slower than actually anticipated and of-fering fewer than 10% SLA violations. Another optionwould be w = 1.7 with fewer than 15% SLA viola-tions in a session. Depending on SLA violation costs, ofcourse, the cloud may choose to be more or less conser-vative.

7. CONCLUSIONIn this work, we presented the PerfEnforce system.

Based on a user’s performance-centric SLA, PerfEn-force scales the user’s cluster of VMs in order to achievegood QoS at a low cost. We explored different tech-niques to layout the user’s data to enable cluster resizingduring a query session. We found that local, shared-nothing storage with partial data replication offers apractical solution with low setup times, minimal clus-ter resizing overheads, reliable query execution times,and low costs. PerfEnforce further scales a user’s clus-ter during a query session. While different scaling al-gorithms are possible, we find that perceptron learningyields results closest to those of an oracle and withoutdifficult parameter tunings. As future work, we planto combine reactive and proactive scaling techniques,which could prove beneficial in cases where perceptronlearning is not as accurate.

12

Acknowledgements This project was supported inpart by NSF grants IIS-1247469 and IIS-1524535, giftsfrom Amazon, the Intel Science and Technology Centerfor Big Data, and Facebook. J. Ortiz was supported inpart by an NSF graduate fellowship.

8. REFERENCES[1] Amazon EC2. http://aws.amazon.com/ec2/.[2] Amazon AWS. http://aws.amazon.com/.[3] Apache spark: Lightnight-fast cluster computing.

http://spark.apache.org/.[4] Microsoft Azure. http://azure.microsoft.com/en-us/.[5] A. Bifet et al. Moa: Massive online analysis, a framework for stream

classification and clustering. In Journal of Machine Learning Research,11:44âAS50, 2010.

[6] Google BigQuery.https://developers.google.com/bigquery/.

[7] Y. Chi et al. Sla-tree: a framework for efficiently supporting sla-baseddecisions in cloud computing. In Proc. of the EDBT Conf., pages129–140, 2011.

[8] Y. Chi, H. J. Moon, and H. Hacigümüs. icbs: Incremental costbasedscheduling under piecewise linear slas. PVLDB, 4(9):563–574, 2011.

[9] S. Das et al. Albatross: Lightweight elasticity in shared storage databasesfor the cloud using live data migration. PVLDB, 4(8):494–505, 2011.

[10] A. J. Elmore, S. Das, D. Agrawal, and A. El Abbadi. Zephyr: livemigration in shared nothing databases for elastic cloud platforms. InProc. of the SIGMOD Conf., pages 301–312, 2011.

[11] A. J. Elmore, S. Das, A. Pucher, D. Agrawal, A. El Abbadi, and X. Yan.Characterizing tenant behavior for placement and crisis mitigation inmultitenant dbmss. In Proceedings of the ACM SIGMOD InternationalConference on Management of Data, SIGMOD 2013, New York, NY,USA, June 22-27, 2013, pages 517–528, 2013.

[12] A. D. Ferguson, P. Bodík, S. Kandula, E. Boutin, and R. Fonseca. Jockey:guaranteed job latency in data parallel clusters. In EuroSys ’12.

[13] A. Gandhi, P. Dube, A. Kochut, L. Zhang, and S. Thota. Autoscaling forhadoop clusters. In IC2E 2016.

[14] A. Gupta et al. Amazon redshift and the case for simpler datawarehouses. Proc. of the SIGMOD Conf., pages 1917–1923, 2015.

[15] D. Halperin et al. Demonstration of the Myria big data managementservice. In SIGMOD, pages 881–884, 2014.

[16] H. Herodotou et al. No one (cluster) size fits all: automatic cluster sizingfor data-intensive analytics. In Proc. of the Second SoCC Conf., page 18,2011.

[17] V. Jalaparti et al. Bridging the tenant-provider gap in cloud services. InProc. of the 3rd ACM Symp. on Cloud Computing, page 10, 2012.

[18] P. K. Janert. Feedback Control for Computer Systems. O’Reilly Media,Inc., 2013.

[19] B. Kao et al. Advances in real-time systems. chapter An Overview ofReal-time Database Systems, pages 463–486. Prentice-Hall, Inc., 1995.

[20] D. Karger et al. Consistent hashing and random trees: Distributed cachingprotocols for relieving hot spots on the world wide web. STOC, pages654–663, 1997.

[21] I. Konstantinou et al. TIRAMOLA: elastic nosql provisioning through acloud management platform. In Proc. of the SIGMOD Conf., pages725–728, 2012.

[22] D. Kossmann, T. Kraska, and S. Loesing. An evaluation of alternativearchitectures for transaction processing in the cloud. In Proceedings ofthe 2010 ACM SIGMOD International Conference on Management ofData, SIGMOD ’10, pages 579–590, New York, NY, USA, 2010. ACM.

[23] H. Lim et al. Automated control for elastic storage. In ICAC, pages 1–10,2010.

[24] T. Liu and M. Martonosi. Impala: A middleware system for managingautonomic, parallel sensor systems. SIGPLAN Not., 2003.

[25] Z. Liu, H. Hacigümüs, H. J. Moon, Y. Chi, and W.-P. Hsiung. Pmax:Tenant placement in multitenant databases for profit maximization. InProceedings of the 16th International Conference on Extending DatabaseTechnology, EDBT ’13, New York, NY, USA, 2013. ACM.

[26] H. A. Mahmoud, H. J. Moon, Y. Chi, H. Hacigümüs, D. Agrawal, andA. El Abbadi. Cloudoptimizer: multi-tenancy for i/o-bound OLAPworkloads. In Joint 2013 EDBT/ICDT Conferences, EDBT ’13Proceedings, Genoa, Italy, March 18-22, 2013, pages 77–88, 2013.

[27] U. F. Minhas et al. Elastic scale-out for partition-based database systems.In Proceedings of the 2012 IEEE 28th International Conference on DataEngineering Workshops, ICDEW ’12, pages 281–288, Washington, DC,USA, 2012. IEEE Computer Society.

[28] P. O’Neil, E. O’Neil, and X. Chen. The star schema benchmark.http://www.cs.umb.edu/~poneil/StarSchemaB.PDF.

[29] J. Ortiz et al. Changing the face of database cloud services withpersonalized service level agreements. In CIDR, 2015.

[30] J. Ortiz et al. Perfenforce demonstration: Data analytics withperformance guarantees. In SIGMOD, 2016.

[31] O. Papaemmanouil. Supporting extensible performance slas for clouddatabases. In Proc. of the 28th ICDE Conf., pages 123–126, 2012.

[32] T. Rabl, M. Frank, H. M. Sergieh, and H. Kosch. A data generator forcloud-scale benchmarking. TPCTC’10, pages 41–56.

[33] K. Ren, Y. Kwon, M. Balazinska, and B. Howe. Hadoop’s adolescence:An analysis of hadoop usage in scientific workloads. Proc. VLDBEndow., 6(10):853–864, Aug. 2013.

[34] M. Stonebraker and A. Weisberg. The voltdb main memory DBMS. IEEEData Eng. Bull., 36, 2013.

[35] R. S. Sutton and A. G. Barto. Reinforcement learning i: Introduction,1998.

[36] S. Venkataraman, Z. Yang, M. J. Franklin, B. Recht, and I. Stoica.Autoscaling for hadoop clusters. In NSDI 2016.

[37] H. T. Vo, C. Chen, and B. C. Ooi. Towards elastic transactional cloudstorage with range query support. PVLDB, 3(1):506–517, 2010.

[38] P. Xiong et al. Activesla: a profit-oriented admission control frameworkfor database-as-a-service providers. In Proc. of the Second SoCC Conf.,page 15, 2011.

13