Computer Networks 72 (2014) 140155Contents lists available at ScienceDirect

Computer Networks

journal homepage: www.elsevier .com/locate /comnetRED-BL: Evaluating dynamic workload relocation for datacenter networkshttp://dx.doi.org/10.1016/j.comnet.2014.07.0011389-1286/ 2014 Elsevier B.V. All rights reserved.

Corresponding author.E-mail addresses: saqibm@lums.edu.pk (M.S. Ilyas), sraza@ucdavis.

edu (S. Raza), cchchen@ucdavis.edu (C.-C. Chen), zartash@lums.edu.pk(Z.A. Uzmi), chuah@ucdavis.edu (C.-N. Chuah).Muhammad Saqib Ilyas a,, Saqib Raza b, Chao-Chih Chen c, Zartash Afzal Uzmi a,Chen-Nee Chuah c

a SBA School of Science and Engineering, LUMS, Lahore, PakistanbData Center Switching Technology Group, Cisco Systems Inc., San Jose, CA, USAcUniversity of California, Davis, CA, USAa r t i c l e i n f o

Article history:Received 4 April 2013Received in revised form 20 June 2014Accepted 7 July 2014Available online 25 July 2014

Keywords:Data centerElectricity costOptimizationWorkload relocationConfiguration planninga b s t r a c t

In this paper, we present RED-BL (Relocate Energy Demand to Better Locations), a frame-work to minimize the electricity cost for operating data center networks over consecutiveintervals of fixed duration. Within each interval, RED-BL provides a mapping of workload toa set of geographically distributed data centers. To this end, RED-BL uses the geographicaland temporal variations in electricity prices as exhibited by electrical energy markets. Inaddition, we incorporate the transition costs associated with a change in workloadmapping from one interval to the next, over a planning window comprising multiple suchintervals. This results in a sequence of workload mappings that is optimal over the entireplanning window, even though the workload mapping in a given interval may not belocally optimal.Our evaluation of RED-BL uses electricity prices from the US markets and workload

traces from live Internet applications with millions of users. We find that RED-BL canreduce the electric bill by as much as 45% compared to the case when the workload isuniformly distributed. When compared to existing workload relocation solutions, for awide range of data center deployment sizes, RED-BL achieves electricity cost savings thatare 8.28% higher, on average. This seemingly modest reduction can save millions of dollarsfor the operators. The cost of this saving is an inexpensive computation at the start of eachplanning window.

2014 Elsevier B.V. All rights reserved.1. Introduction

Geo-diverse data centers enable robust and low-latencycloud services. The electricity cost for this huge infrastruc-ture is a significant fraction of the operational cost (15%)[1] as well as capital cost [2]. Due to increasing demandfor cloud services and increasing electricity prices, it isessential for data center operators to cut their electricitybill [3,4].

A data centers electricity bill for a particular period oftime is the product of the unit price of electricity and theamount of electrical energy consumed. Hence, two possi-bilities to reduce data center electricity costs are: (i) usecheaper sources of electricity, and (ii) reduce energy con-sumption. Our present work jointly exploits both of thesedimensions.

A geo-diverse data center deployment is dimensionedaccording to peak workload, which occurs for only a shortperiod of time [5]. Therefore, most of the time, the

http://crossmark.crossref.org/dialog/?doi=10.1016/j.comnet.2014.07.001&domain=pdfhttp://dx.doi.org/10.1016/j.comnet.2014.07.001mailto:saqibm@lums.edu.pkmailto:sraza@ucdavis.edumailto:sraza@ucdavis.edumailto:cchchen@ucdavis.edumailto:zartash@lums.edu.pkmailto:chuah@ucdavis.eduhttp://dx.doi.org/10.1016/j.comnet.2014.07.001http://www.sciencedirect.com/science/journal/13891286http://www.elsevier.com/locate/comnet

1 The electric load that may be turned on or off to save electricity costwithout long delays or long-term impact on performance.

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 141workload may be mapped to a subset of the over-provi-sioned infrastructure. During a given period of time, a datacenters energy consumption is an affine function of theworkload it handles [6]. Therefore, geographic diversityin electricity prices [2] may be leveraged to cut electricitycost by directing most of the workload to data centers withcheaper electricity prices.

In addition to geographic diversity, electricity pricesexhibit temporal diversity as well [2], causing the cheapestset of data centers to handle current workload to changewith time. Therefore, an electricity cost reduction tech-nique for geo-diverse data centers must periodicallyupdate its choice of data centers to be used for handlingthe current workload. We define an interval as a period oftime for which electricity prices are fixed and workloadis known. We also define a sequence of consecutive inter-vals as a planning window. An interval may be an hour longand the planning window may be 24 hours, for instance.The problem, then, is to pick the distribution, or mapping,of workload to data centers for each interval in a planningwindow to minimize the electricity cost by exploiting thegeo-temporal variation in electricity prices and temporalvariations in workload. If we define the aggregation ofworkload mapping for all data centers during a particularinterval as a network state, this problem may be viewedas determining a state trajectory that is electricity costoptimal over the corresponding planning window.

The electricity cost savings resulting from workloadrelocation are somewhat limited due to lack of energy pro-portionality in todays data centers [5]. Therefore, it wasproposed to dynamically scale the active infrastructure inresponse to changes in the magnitude of workload [5,7].This capacity scaling is expected to incur some overheadelectricity cost which may be modeled as the state transi-tion cost in the state trajectory problem.

Theoretically, one can benefit from capacity scaling byshutting down elastic load when it is not needed. Thisscheme minimizes the electricity consumption, but con-ventional wisdom suggests that restarts affect equipmentlifetime and operators are generally reluctant to adopt thisapproach. On the other extreme, elastic load may be left inan idle state, but the reduction in electricity consumptionwould be quite small. In between these two extremeswould be Dynamic Voltage and Frequency Scaling (DVFS)techniques. The transition cost would be zero for the idlingscheme since no state-change overhead is incurred. Thetransition costs would be quite high if elastic load is de-activated (and activated later when needed), whereasDVFS would account for transition costs somewherebetween the two extremes [8]. In this work, we experi-ment with the entire spectrum of transition costs in orderto generalize our results.

To the best of our knowledge, prior work has largelytaken amicro-scale view of this problem by scaling the datacenter capacity at the granularity of states of individualservers within a data center [913,7] and, in some cases,has altogether ignored transitioncosts resulting fromcapac-ity scaling. These approaches lack scalability to multi-datacenter scenarios or are sub-optimal. In this paper, weaddress the challenges of scalability aswell as incorporationof transition costs into the optimization problem.In the present work, we approach a scalable solution tothis problem by treating all the elastic electric load1 in adata center as an aggregate and determining this aggregatesstate for each interval in a planning window. For every inter-val in the planning window, our coarse-granularity formula-tion provides the average utilization of the servers within adata center. Relaxing a discrete optimization problem to acontinuous one typically introduces an approximation error.The magnitude of this error is expected to be small for largescale problems such as geo-diverse data centers. Determina-tion of the approximation errors magnitude is beyond thescope of this paper and is left as future work.

To motivate the significance of transition costs inthe dynamic scaling of geo-diverse data center capacityand our scheme for incorporating the transition costs tothe optimization problem, we will use a simple exampleshown in Fig. 1. It depicts an example instance of the statetrajectory problem for a planning window consisting ofthree intervals represented along the horizontal axis. Foreach interval, we show three sample states representedusing rounded rectangles. Each state is labeled with aname in the lower left corner and the corresponding elec-tricity cost in the lower right corner. Moreover, transitionbetween states in consecutive intervals is shown usingarrows and the corresponding transition cost is shown asa label over the arrow. We consider three data centers inthis example, represented using circles numbered 1, 2and 3. In Fig. 1, the relative height of the circles in a giveninterval represents the diversity in electricity prices. Forinstance, in interval 3, data center 3 has the lowest electric-ity price. In a particular state, the workload mapped toeach data center is represented using shading within thecircle. For simplicity of demonstration, we assume thatthe cumulative workload is fixed at a value that equals1.3 times a single data centers workload capacity.

In the absence of transition costs, the optimal state tra-jectory could be obtained by making a greedy choice ofstate in each interval (the path S2! S6! S8 in Fig. 1)[2,7,13]. This is clearly the lowest possible sum of statecosts without considering any transition costs. With transi-tion costs included, however, the greedy solution yields atotal cost of 42. We refer to such a strategy as RelocateEnergy Demand to Cheaper Locations (RED-CL).

One may also consider a static deployment configura-tion where an operator selects the data centers that havethe lowest average electricity price over the planning win-dow. This corresponds to the path S1! S4! S7, with thesum of state costs equal to 42. Since the workload mappingdoes not change, there are no transition costs, and hencethe total solution costs is also 42. In general, dependingon the magnitude of transition costs, the static solutioncould be better or worse compared to the greedy solution.

The optimal solution from Fig. 1 is the pathS3! S5! S9, with a total cost of 39. For this state path,the sum of state costs is 39, which is higher than thecorresponding component for the greedy solution.However, the sum of transition costs is 0 resulting in an

231

21 3

1 32

23

1

2

13

13

2

2

31

2

31

13

2

0

226

6

5

4

6

0

0

04

6 6

4

5

03

Fig. 1. A motivating example that depicts the workload-mapping problem for three consecutive intervals involving three data centers of equal capacity. Forthis example, the workload is assumed to be constant equal to 1.3 times the capacity of a single data center, in all three intervals. Of many possible states ineach interval, we show just three example candidate states along with the electricity cost for being in those states. Cost of transition from one state toanother in the next interval are also labeled on the arrows representing the state transition.

142 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155overall lower total solution cost than both the static andthe greedy strategy. This simple example illustrates thatit is important to consider the costs associated with relo-cating demands in operational data centers.

Our present work uses the overhead incurred in chang-ing the state of the elastic load within a data center as thetransition costs. However, in practice, there can be otherforms of transition costs as well, which would vary fromone deployment to the other. Examples of other sourcesof transition costs include, but may not be limited to, thefollowing:

Convergence time: An operator might change the wayuser traffic is routed to data centers at various layers ofthe network stack. For instance, this could be done bymodifying DNS entries or BGP routing tables. However,these and other workload rerouting schemes wouldhave non-negligible convergence times. For instance,many DNS caches do not honor the time-to-live (TTL)values for DNS entries [14]. Also, BGP routing tablechanges have been shown to take an unpredictableamount of time to reflect globally [15]. Thus, if data cen-ter A is scaled down for interval j 1 and its workloadis redirected to data center B, the former may continueto receive some fraction of the workload during intervalj 1. Some reserve capacity must, therefore, be keptactive at data center A during interval j 1, therebyreducing the electricity cost savings.

Consistency traffic: In order for any request to be han-dled anywhere, the data store for the applications mustbe replicated and consistency must be maintained. Thecost of inter-data center traffic is quite high, hence thisform of transition costs may be quite significant. Themagnitude of such overheads is not easy to predictbecause replication schemes are operator and applica-tion dependent. To the best of our knowledge, the cur-rent body of knowledge lacks a generic model for suchtraffic. Therefore, similar to [7], in the present work,we assume that content is perfectly replicated.

It is clear from the above discussion that the factorscontributing towards transition costs depend not only onhow the data center network is deployed and operatedbut also on the applications being hosted. The utility ofmodeling a specific deployment is limited. The questionthat is significant, however, is the impact of variation inthe magnitude of transition costs relative to the electricitycost of operation within a given interval on the possibleelectricity cost savings. Therefore, we have used a normal-ized and parametrized model for transition costs in ourproblem formulation so that operators can easily plug-inthe parameter values (idling costs, transition costs, numberof data centers and their locations) from their own datacenter deployment.

In this paper, we present Relocate Energy Demand toBetter Locations (RED-BL), a framework for optimizing anoperators electricity costs by dynamically re-assigningworkload to available data centers at discrete intervals in aplanning window. This optimization considers not only theelectricity cost of a particularworkload assignment, but alsothe cost of transition from one network state to another.

We find that using RED-BL workload relocation solu-tions, an operator may save up to 45% of their electric bill,for a wide range of transition costs, compared to the case ofuniformly distributing the workload among data centers.On average, RED-BL is 8.28% better as compared to theexisting RED-CL solutions [2,7,13]. While this percent addi-tional saving may seem modest, it can translate into mil-lions of dollars of annual savings for data centeroperators. To realize these savings, RED-BL requires a quickcomputation at the start of each planning interval.

In a short version of this paper [16], we made the fol-lowing contributions:

1. We proposed RED-BL, the first electric bill minimiza-tion framework for data center operations consider-ing the transition costs.

2. We formulated RED-BL as a network state trajectoryoptimization problem; the solution (RED-BL) picks asequence of network states over a look-ahead plan-ning window.

3. An evaluation of RED-BL and its comparison withRED-CL was presented based on trace-driven simula-tions. Our evaluation used electricity prices from the

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 143US markets and workload data from live Internetapplications. Our simulations spanned a wide varietyof operators (with number of data center varyingfrom 1 to 33), and data centers of varying capacity.We also performed a sensitivity analysis of theRED-BL solution as the cost of activating and deacti-vating a data center changes.

4. To the best of our knowledge, this is the first study toevaluate the sensitivity of workload relocation solu-tions (RED-BL and RED-CL) to workload predictionaccuracy, amount of over provisioning, and geo-graphical diversity.

In this paper, we extend our work published in [16], andmake the following contributions:

1. The electric load for certain type of equipment, suchas air-conditioning and networking, may never becurtailed. We term such equipment as inelastic loadand the rest of the equipment (only servers, in thepresent work) as the elastic load. We evaluate RED-BLbymodeling the overhead of state changes for elas-tic load in a data center deployment. In our study, thestate change ranges from the (theoretical) extreme ofturning servers on and off to the use of DVFS.

2. Workload estimation errors are likely to affect thequality of the RED-BL solution. We propose a slid-ing-window re-optimization scheme that periodi-cally re-invokes the workload estimation andoptimization. We also perform a study of the impactof re-optimization frequency on the amount of sav-ings achievable through RED-BL.

3. Computational resources needed in a given intervaldepend on the peak workload for that interval.While average workload may be predicted quiteaccurately, peak workload within an hour is not eas-ily predictable. To schedule resources according toaverage workload demand and still absorb fluctua-tions, some reserve resources must be kept activeas well. We study the sensitivity of RED-BLs elec-tricity cost savings to the size of reserve computa-tional resources.

4. Granular (de)activation of data center resources islikely to reap more savings in electricity costs. Wepropose a RED-BL framework that is configurable inthe granularity of (de)activation. We study theimpact of (de)activation granularity on the amountof electricity cost savings achievable throughRED-BL.

5. We show that RED-BL problem is NP-Complete andpropose a heuristic algorithmand compare its perfor-mance with the optimal solution using real datasets.

Our solution provides detailed operational planninginformation in the form of:

A list of data centers where elastic load must be keptactive for each interval in the planning window.

The workload distribution amongst these data centers.

The rest of the paper is organized as follows. Section 2discusses related prior work. Section 3 presents theRED-BLmathematical optimization problem, discusses thatthe optimal workload mapping problem itself is NP-Com-plete and provides a heuristic algorithm. Section 4 describesthe experimental setup and the data sets that we used. InSection 5 we present the results of our study, followed bya discussion of future directions in Section 6. In Section 7,we discuss the conclusions resulting from our study.2. Related work

Li et al. determined the electricity cost optimal mappingof workload to geo-diverse data centers by controlling thestate of the individual servers within each data center [9].The state of the servers and their electricity consumptionwas controlled using Dynamic Voltage and Frequency Scal-ing (DVFS) and Dynamic Cluster Server Configuration(DCSC). Since their optimization problem formulation usedMixed Integer Programming (MIP) with decision variablesper server, their approach is effective for small-scale prob-lems such as an individual data center or a fraction thereof.This limitation is evident from the small number of serversused in the simulation-based evaluation in [9]. One way toscale their approach to large distributed data centers is touse coarse granularity in their problem formulation. Forinstance, instead of controlling the state of each serverindependently, all servers in a single rack could be config-ured in the same state at a given time. They used theSoccer World Cup 1998 webserver workload traces andelectricity prices at four different locations to evaluatetheir proposal.

Some researchers have also proposed algorithms for thedata center electricity cost optimization problem. In thecontext of a web-search query processing system hostedon a geo-diverse data center network, Kayaaslan et al. pre-sented a bin-packing type of algorithm for shifting searchquery workload between data centers in [17]. Buchbinderet al. proposed online algorithms for relocating MapReduce jobs between geo-diverse data centers to reducethe electricity bill while considering the cost of inter-datacenter bandwidth [18]. Their proposed algorithms considerthe uncertainty in electricity prices and workload esti-mates while mapping the jobs to data centers. They evalu-ated their algorithms using electricity prices from 30locations across the US and workload data from a 10,000node Map Reduce cluster. Bhaskar and Fleischer proposedonline algorithms for mixed packing and covering, a prob-lem which may be applied to optimally map workload togeo-diverse data centers [19]. For configuring servers in asingle data center with a view of minimizing electricitycosts, Lin et al. presented offline as well as online algo-rithms for dynamic scaling of server computational capac-ity [10]. Urgaonkar et al. proposed an online optimizationalgorithm while proposing to disconnect data centerdevices from mains and running on UPS when the electric-ity prices are high [20]. Their proposed scheme rechargesthe UPS units when the electricity prices are lower.

Investigation of strategies of infrastructure scaling toconserve power in a single data center is reported in[11,2123]. Chen et al. proposed three different solutionsthat either shut down or frequency-scale servers in a

Table 1Data center network model parameters.

Parameter Description

144 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155web hosting data center with the objective of minimizingelectricity and maintenance cost while ensuring SLA com-pliance [11]. The first two of the proposed algorithms werebased on a queuing theoretic and control theoretic analy-sis, respectively, while the third one was a hybrid scheme.While scaling the deployed capacity, their proposedscheme considers the cost of turning the servers on andoff in terms of the resulting wear and tear. Mazzuccoet al. presented similar strategies in [21]. Oh et al. consid-ered a virtualized environment and proposed solutions foroptimally placing VMs on servers and map workload to theVMs such that electricity costs are minimized [22]. In [23],Chase et al. presented policies for resource allocation in ahosting center alongwith a switching infrastructure forrouting requests to servers.

Most of the prior work in this area considers applica-tions with short requestresponse type jobs. In [24], how-ever, Chen et al. considered connection-intensiveapplications such as video streaming, Internet gamingand instant messaging in the context of energy cost awareload dispatch.

Rao et al. considered data center operation in a futureselectricity market and the possibility of hedging againstuncertain electricity prices under a smart grid environ-ment in [25]. The authors used workload data from Googlesearch cluster and evaluated a scenario of an operator withdata centers at two different locations.

Another related theme of research is greening of datacenters. Some examples of work that reports the resultsof efforts towards green mapping of workload to datacenters are: [2633]. On a related note, Sucevic et al. stud-ied various approaches for shutting down end-hosts tominimize the total electricity consumption on participanthosts in a peer-to-peer file download system [34].

All of the above work deals with problems that can becategorized broadly as optimal scheduling problems. Suchproblems arise in many different domains and prior workin such domains is relevant. For instance, System on Chip(SOC) [35], electric power systems and smart grid[3639], WiFi access points [40], wide area networks[41], cellular networks [42] and high performance comput-ing [4348].m Number of data centersn Number of intervals in a planning windowk Duration of an interval in hoursf The ratio between a data centers peak and idle power

consumptionci Normalized workload capacity of data center ir Penalty for activating the elastic load at a unit capacity

data center as a fractionof its energy consumption at full load in one interval

d Penalty for deactivating the elastic load at a unitcapacity data center as a fractionof its energy consumption at full load in one interval

ejiUnit cost of electricity at data center i during interval j

wj Operators workload during interval j

xjiWorkload mapped to data center i during interval j

pji1 if data center i is active (either computing workloadoridling) during interval j; 0 otherwise

bji1 if data center is elastic load is activated at intervalj; 0 otherwise

sji1 if data center is elastic load is deactivated at intervalj; 0 otherwise3. Problem formulation

Some of the electricity load in a data center is inelasticin the sense that it must be on all the time, whereas therest is elastic in that it may be put into low-power mode(hibernation or standby) or de-activated if there is noworkload for the data center to handle. While formulatingthe electricity cost minimization problem, we must focussolely on the elastic electricity load in the data center.

According to [49], the breakup of power draw in atypical data center is servers: 56%, cooling: 30%, powerconditioning: 8%, network: 5% and lighting: 1%. Coolingoften requires a significant ramp up time hence turningon/off cooling may not be a good idea. Power conditioningwould be needed to run the inelastic pool of devices. Net-work equipment may not be turned off due to unpredict-able convergence delays whereas lighting load isnegligible. In this paper, therefore, we put only the serversin the elastic pool of devices.

Table 1 provides a quick summary of the parametersthat we use to develop the RED-BL optimization formulate.We consider a geo-distributed data center infrastructurecomprising m interconnected data centers. At any giventime, the workload is distributed amongst the data centersin the network. For ease of modeling, we assume thatchanges to the distribution of workload amongst data cen-ters may be done at the start of discrete intervals of dura-tion k. We use xji to denote the fraction of workload duringinterval j that is mapped to data center i. We considerworkload that is normalized over its peak, i.e., the work-load values for any interval are between 0 and 1. The work-load capacity of data center i, denoted ci, is also normalizedon the same scale. We assume that the network of datacenters is over-subscribed so that

Pmi1ci > 1.

Let the sum of elastic loads peak and idle power con-sumption over all data centers be Pmax and Pmin, respec-tively. Assuming that the data centers are homogenous,an individual data centers workload capacity is directlyrelated to its peak (or idle) power consumption. Thus,Pmaxi ciP

max and Pmini ciPmin.

Data center power consumption is an affine function ofthe average CPU utilization of the servers [6]. Therefore,the average power consumption at data center i duringinterval j is:

Pji ci Pmin x

jiP

max Pminci

!1

Dividing both sides of the above equation by Pmax givesthe normalized power consumption for data center iduring interval j:bPji fci xji1 f 2where f is the ratio of Pmax to Pmin. If we set xji 0, i.e., datacenter i is not computing any workload during interval j,

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 145then the second term in Eq. (2) goes to zero and the powerconsumption reduces to the first term in Eq. (2) only,which we refer to as idle power consumption. The secondterm in Eq. (2) indicates the workload-dependentcomputational power consumption, which is independentof the data center capacity.

Let r and d be the average power consumption, over asingle interval, required to activate or deactivate, respec-tively, all of the elastic load at a unit capacity data center.Then, the power consumption for activation of the elasticload at data center i is rci. The electricity cost for activatingdata center is elastic load at the start of interval j is, there-fore,2 cie

jir. Here, we are assuming that the elastic load at a

data center may be activated within a single interval. Thevalue of k that we used in our experiments is equal to onehour, which is a sufficiently large interval for serveractivation. Deactivation cost for elastic load may also bederived in a similar manner.

Electricity cost incurred at data center i during interval jis a product of its total power consumption (computing,idling, activation and deactivation), duration of the interval(k) and the unit price of electricity (eji). Hence, the RED-BLoptimization problem formulation may be given as:

minimizeXnj1

Xmi1

cieji p

jik f 1 f

xjici

! bjir s

jid

!3

subject to:

xji 6 ci 8i; 8j 4Xmi1

xji wj 8j 5

pji; bji; s

ji 2 f0;1g 8i; 8j 6

pji P xji 8i; 8j 7

bji P pji p

j1i 8i;2 6 j 6 n 8

sji P pj1i p

ji 8i;2 6 j 6 n 9

b0i p0i ; s0i 0 8i 10

Decision variable pji is 1 if the elastic load in data centeri is active during interval j, or 0 otherwise. Similarly, bji (s

ji)

is 1 if the elastic load in data center i is activated (deacti-vated) at the start of interval j. In Eq. (3), multiplicationof the first two terms by pji ensures that computation andidling costs are accounted for in interval j, only if the elasticload in data center i is active during that interval. Similarly,multiplication of the last two terms in Eq. (3) by bji and s

ji,

respectively, ensures that activation and de-activationcosts contribute to the summation only when the elasticload in a data center is activated or de-activated.

The workload capacity constraint is given in (4). Eq. (5)ensures that all incident workload is handled, while (6) rep-resents the binary-value constraint. Inequality (7) ensuresthat the elastic load in a data center is active wheneverthere is any workload mapped to it. The constraint in Eq.(8) ensures that bji is 1 if the elastic load is inactive in inter-val j 1 and active in the next interval. The involvement of2 Multiplication with the duration of an interval, i.e., k is not necessary,because r is defined as the per interval cost.bji in the minimization objective function ensures that it is 0otherwise. Similarly, the constraint in Eq. (9) ensures that sjitakes on the correct value depending on the deactivation ofelastic load in the data centers. We assume that the elasticload in all data centers is initially inactive, therefore, anactivation may be necessary at the first interval whereasdeactivation in the first interval is not necessary. These con-ditions are ensured by the constraints in Eq. (10). It is easyto customize this constraint such that all data centers areassumed to be initially active.3.1. Problem complexity and a heuristic

The optimal workload relocation problem is identical tothe Unit-Commit problem [50] in distributed electricitygeneration and transmission scenario. In the unit commitproblem, one determines the amount of power to be sup-plied from each generating resource and schedules theactivation (ramp up), deactivation (ramp down) and idling(spinning reserves) of the generating resources, giventime-varying demand for electricity. Due to a one-to-onemapping between the data center-workload mapping andUnit-Commit problems, it follows that if there is apolynomial time solution for the data center-workloadmapping problem, Unit-Commit may also be solved inpolynomial time. However, since the Unit-Commit prob-lem is known to be NP-Complete [50], it follows that sois the workload-mapping problem that we are consideringin this paper. We will show later that we are able to solvereasonably large sized instances of the above NP-Hard MIPformulation for RED-BL using the CPLEX solver. Nonethe-less, we now present a heuristic algorithm for it. The over-all worst case running time3 of the heuristic algorithm,given in Algorithm 1, is O(mn2 + n3 + mlgm).

The pseudo-code of our heuristic algorithm for RED-BLis given in Algorithm 1. Assume that the workload vectorfor the planning window starts at a trough, then rises ina non-decreasing manner to the peak before falling off ina non-increasing manner to another trough. Since the acti-vation/deactivation costs are expected to be significant,our heuristic is designed to select a small number of datacenters to operate in long continuous stretches during agiven day. For the assumed characterization of the work-load, elastic load at a few data centers would be sufficientto handle the workload early (and late) in the planningwindow. As the workload rises gradually, elastic load atsome more data centers would need to be brought online.As the workload starts to fall, elastic load at some data cen-ters may gradually be deactivated until the planning win-dow ends. Our heuristic places two pointers at thebeginning and end of the planning window, determinesthe number of data centers (d1 and d2) needed to handlethe workload corresponding to the two pointers andpicks the smaller of these two values. It then findsmin(d1; d2) best data centers in terms of having the leastaverage electricity price over the planning window. Theelastic load at these data centers will be kept active3 To conserve space, we have omitted the detailed but straightforwardderivation of the running time of our heuristic.

146 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155between the intervals corresponding to the two pointers.Furthermore, our algorithm assigns as much workload aspossible to the selected data centers in ascending orderof average electricity price in the chosen intervals.

As long as the workload in the intervals correspondingto the two pointers may be handled by the same numberof data centers, both of the pointers are moved closer toeach other. Otherwise, the pointer that corresponds tothe interval requiring the smaller number of data centersis moved towards the other pointer. This pointer move-ment is performed until either the pointers cross eachother or the number of data centers required to handlethe workload in the interval corresponding to the movingpointer increases. In the former case, we are done and inthe latter, the algorithm repeats the data center selectionand workload mapping step. The algorithm then activateselastic load at data center(s) to meet the workload require-ment of the pointer corresponding to the interval with thehigher workload. The data center(s) where the elastic loadis activated at this time are the ones that are not usedbefore and have the least average electricity price in theinterval between the two pointers.

Algorithm 1. Heuristic for the RED-BL problemRequire w1 . . .n: Cumulative data center workloadfor the planning window,

e1 . . .m1 . . .n: Electricity prices for all datacenters over the planning window

Ensure z1 . . .m1 . . .n: workload assigned to thedata centers for all intervals

y1 . . .m1 . . .n: Data center status (1 = on/0 = off)over the planning window

1: g1 0; g2 n 1; l w; a 1 . . .m;nc 0;

2: yij 0; zij 0; (8i; 8j)3: repeat4: d1 dwg1=c1e; d2 dwg2=c1e;

nd mind1; d25: if nd > nc then6: Sort a in ascending order of average

electricity price in g1; g27: for all i 2 a do8: for all j 2 g1; g2 do9: yij 1; nc++

10: zij minlj; ci11: lj lj zij12: Remove i from a13: end for14: end for15: end if16: repeat17: g1++18: until dwg1=c1e > nc or g1 > g2

or dwg1=c1e > dwg2=c1e19: repeat20: g2- -21: until dwg2=c1e > nc or g1 > g2

or dwg1=c1e < dwg2=c1e22: until g1 > g24. Experimental setup

In this section, we describe the experimental setup toperform a comparative study of different workload place-ment algorithms under various scenarios.

4.1. Application workload

We used an year-long trace of hourly workload for 3social networking applications, with a subscription baseof over 8 million users [51]. In order to make the datasetrepresentative of a large data center network operator,we aggregated these traces into a week long trace asfollows. We sliced the trace into week-long segments andconsidered each slice as workload for a differentapplication, for the same week. We, then, normalized thesum of these trace vectors so that the peak cumulativeworkload corresponds to a value of 0.9. The normalizedworkload intensity is plotted in Fig. 2. The statisticalcharacteristics of our workload, as plotted in Fig. 3, arequite similar to those reported by Google for thousandsof servers during a six-month interval at a Google datacenter [5].

4.2. Electricity prices

We selected 33 different regions in the USA for whichhourly electricity prices are available online. These regionsbelong to the following Independent System Operators(ISOs): NYISO, CAISO, MISO, ISO-NE and PJM. We usedthe day-ahead prices for these locations, i.e., the electricityprice negotiated for the same hour on the following day. Inall the experiments for this work, we considered an opera-tor with data centers at all 33 locations in our dataset.

4.3. Algorithms for workload distribution/relocation

The workload relocation problem has the followingdimensions based on which different algorithms may beformulated.

For a given interval, the strategy for distribution ofworkload amongst data centers.

For a given interval, the strategy for the state (on/off) of elastic load at a data center which has notbeen assigned any workload. In such cases, there is0 20 40 60 80 100 120 140 160 1800

0.10.20.30.40.50.60.70.80.9

Hour

Nor

mal

ized

wor

kloa

d

Fig. 2. Normalized workload.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Normalized Workload Intensity

Frac

tion

of T

ime

Fig. 3. Workload intensity histogram.

Table 2A comparison of the algorithms studied in this paper.

Workload mapping strategyLI, LD, LS, LO GreedyRED-BL Based on global optimal solutionUNIFORM Workload equally divided amongst all data centers

State of a data center in an interval when it has no workloadLI Active and idlingLD, LO InactiveLS Either inactive or idling, whichever is cheaperRED-BL Based on global optimal solutionUNIFORM Active

Is transition cost reported in the total electricity cost reported?LI N/ALD, LS YesLO NoRED-BL YesUNIFORM N/A

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 147a trade-off between keeping the elastic load on (andincurring idling costs) and deactivating it (whileincurring deactivation overhead and possibly acti-vation overhead if it needs to be brought backonline later in the planning window).

Over the planning window, does the algorithmreport transition costs in the total electricity cost?

In this paper, we report comparative results for sixworkload placement algorithms. The following listdescribes and differentiates these algorithms. The samecomparison is also presented in tabular form in Table 2.

RED-BL: This is our proposed algorithm that determinesthe global optimal cost of electricity over a planningwindow while considering and reporting the transitioncosts. The choice of workload distribution as well as thestate of elastic resources with no workload is governedby the optimal solution as determined by the CPLEXsolver.

Heuristic: This is the heuristic algorithm that we pro-posed in Section 3. The transition costs are reported aspart of the total electricity cost. UNIFORM: This algorithm represents the choice ofthose operators that find an even loading of their datacenters desirable. This algorithm does not deactivateelastic loads and hence does not incur transition costs.

Greedy algorithms: The originally proposed algorithmin [7] distributes workload to data centers such that,for each interval in the planning window, it makes agreedy assignment (in terms of current electricity price)of workload to data centers. Furthermore, this originalalgorithm keeps the elastic load at all data centersactive in all intervals, incurring significant idling costsand hence is naturally disadvantaged against RED-BL.To have a fair comparison with the greedy workloaddistribution strategy, we use several variants of the ori-ginal algorithm as well. Local optimal with Idling (LI): This is the originally

proposed algorithm from [7]. It does not deactivateelastic load.

Local optimal withOut transition costs (LO): Thisvariant of LI was proposed in [7]. It deactivates un-needed elastic load while ignoring the transitioncosts. This algorithm does not report transition costsin the total electricity cost of its proposed workloadmapping for the planning window. This algorithm isvery useful because it defines the lower bound onelectricity cost that any algorithm can ever achieve.

Local optimal with Deactivation (LD): This algo-rithm is similar to LO in all respects except that italso reports the activation/deactivation costs as partof the total cost of its proposed solution. Unlike LO,its results are practically relevant. Its total cost isless than (for all practical cases) LI, which makes itsomewhat competitive to RED-BL.

Local optimal with Selection (LS): In cases wheretransition costs are high compared to idling costs itwould be better to keep the elastic resources at adata center active and incur idling costs if it will beneeded again after the lapse of a small number ofintervals. LS is a variant of LD that is empoweredwith the ability to select whether to deactivateunneeded elastic load at a data center or keep itidling. The cost of LS is never greater than that of LD.

148 M.S. Ilyas et al. / Computer Networks 72 (2014) 1401555. Results

To evaluate the utility of workload relocation for elec-tricity cost minimization, we formulated seven differentscenarios. For each scenario, we ran seven experiments(one for each days workload in our dataset) and reportthe average of the total electricity cost for each algorithm.Each experiment determines an operational plan for aplanning window consisting of 24 consecutive intervals,each with a duration of one-hour.

5.1. Scenario 1 (extent of over provisioning)

In this scenario, we investigate the relationship of theamount of data center capacity over-provisioning withthe electricity cost savings. As we increase the amount ofover-provisioning, each individual data centers capacitywould increase enabling more and more workload to bemapped to data centers at locations with cheaper electric-ity price.

With data centers at all 33 locations in our dataset, wevaried ci between 0.03 and 0.12 (in increments of 0.01).This covers a variety of operators whose workload capacityranges from just over expected peak workload to almost300% over-provisioning.

We computed the total electricity cost for all algorithmswhile setting f r d 0:65. The percentage savings intotal electricity cost by various algorithms compared toUNIFORM are plotted against the data center capacityover-provisioning in Fig. 4. We found that for the widerange of capacity over-provisioning that we considered,LI is able to do only slightly better than the naive UNIFORMalgorithm (about 2%). This is due to the significant idlingcosts incurred by LI under the experiments conditions.For this reason, we have omitted LI from this plot.

The most competitive practical variants of LI, i.e., LS was10.35% off from the ideal lower bound (LO). Meanwhile,RED-BL solution is quite close to the ideal lower bound(LO). The reason for greater savings with RED-BL comparedto the greedy solutions (LS and LD) is that, the transitioncosts being significant, the former does fewer state transi-tions. In several intervals, RED-BL chooses data centerswith relatively higher electricity price than the greedy0 50 100 150 200 250 300 35010

15

20

25

30

35

40

45

50

Overprovisioning Percentage

Perc

enta

ge C

ost S

avin

gs v

s U

NIF

ORM

LDLSREDBLLO

Fig. 4. Percentage electricity cost savings vs over-provisioning (comparedto UNIFORM).solutions, but makes up for the higher computational costby a reduction in the transition costs incurred.

5.2. Scenario 2 (activation/deactivation overhead)

As the magnitude of transition costs relative to the statecost for an interval grows beyond a certain point, the ben-efits of deactivating elastic load at data centers woulddiminish. Accordingly, the electricity cost savings achiev-able by the workload relocation schemes would drop withincrease in transition costs. In this scenario, we determinethe percentage savings in total electricity cost for eachalgorithm compared to UNIFORM, while varying theactivation/deactivation overhead between 0 and 1, inincrements of 0.1.

Since LI does not (de)activate unneeded elastic load, itselectricity cost is independent of the magnitude of transi-tion costs. We observed that it offered a saving of merely1.74% compared to UNIFORM. Fig. 5 shows the electricitycost savings for the other algorithms compared toUNIFORM. The LS and LD variants offered savings that scalealmost linearly to the magnitude of transition costs. Com-pared to LI, both LS and LD bring a factor of 4 reduction inthe electricity cost, on average. RED-BL not only scalesbetter than LS and LD but also achieves electricity cost sav-ing that is fairly close to the ideal lower bound as reportedby LO (only 3% higher, on average).

5.3. Scenario 3 (granular activation/deactivation)

In this scenario, we investigate the potential benefits of(de)activating fixed size subsets of the elastic load in a datacenter instead of an all or nothing (de)activation approach.Granular (de)activation is expected to bring additionalpower savings. For instance, if we are only allowed to(de)activate the entire elastic load in a data center that isoperating at 10% of its capacity, then 90% of its elastic loadis still consuming significant idling costs. However, if wehad the ability to deactivate half of the elastic load at adata center, we could cut idling energy cost significantly.

The size of the portion of the elastic load in a data cen-ter that may be independently (de)activated may bedeployment-dependent or operator-dependent. Possible0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 139

40

41

42

43

44

45

Activation/deactivation Overhead

LDLSREDBLLO

Perc

enta

ge E

lect

ricity

Cos

t Sav

ing

MROFI

NU

otder ap

moC

Fig. 5. Percentage electricity cost savings vs transition overhead (com-pared to UNIFORM).

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 149choices of granularity may be a rack, a pod or one half ofthe elastic load, etc. The RED-BL optimization problem for-mulation with l granular (de)activation levels is given by:

minimizeXnj1

Xmi1

cieji p

jik

fl 1 f x

ji

ci

! b

jirl

sjid

l

!

subject to:

xji 6 ci 8i; 8j 11Xmi1

xji wj 8j 12

pji; bji; s

ji 2 f0;1; . . . ; lg 8i; 8j 13

pji P xji l=ci 8i; 8j 14

bji P pji p

j1i 8i;2 6 j 6 n 15

sji P pj1i p

ji 8i;2 6 j 6 n 16

b0i p0i ; s0i 0 8i 17

There are three primary differences from the vanillaRED-BL formulation. The first difference is in the objectivefunction, where the idling, activation and de-activationcosts depend on the number of granular units involved inthe idling, activation or de-activation process respectively.The second difference is in the domain of pji;m; b

ji and s

ji

(see constraint (13)). The third difference is in the con-straint (14), which ensures that pji takes on an appropriatevalue from 0,1, . . . , l. Since computational cost is indepen-dent of the data center capacity, it is also independent ofthe number of granular units being used at a data centerduring a given interval.

In Fig. 13, we have plotted the percentage savings inelectricity cost vs the granularity of data centers elasticload (de)activation. The savings are compared against thescenario where the entire elastic load in the data centermay only be (de)activated as a whole (l 1). Accordingly,in Fig. 13 we see no savings for l 1. We also see thatthe ability to independently (de)activate half of a datacenters elastic load provides around 2.5% additionalelectricity cost savings on top of what vanilla RED-BL can0.1 0.2 0.3 0.4 0.50.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

DVFS Power Sc

Nor

mal

ized

Pow

er C

onsu

mpt

ion

Fig. 6. If DVFS is used to reduce server power consumption instead of shutting thwould be more acceptable to risk-averse operators. Here, we plot the differenconsumption between 0% (complete shutdown) and 100% (unused servers run oachieve. The electricity cost savings grow almost linearlywhen going to more granular size of independent(de)activation.

5.4. Scenario 4 (DVFS)

Dynamic Voltage and Frequency Scaling (DVFS) offersan alternative to shutting down servers when they arenot needed. We evaluate the power savings achievableby RED-BL with elastic load (de)activation vs the power-reduction possible through DVFS. According to [52], DVFSmay be used to reduce server power consumption to 18%of its peak power consumption. In order to keep ourresults generic, we have experimented with all possiblevalues of power consumption reduction factor for DVFSin conjunction with our dataset. The values of f ; r and dwere fixed at 0.65 in these experiments and the resultsare plotted in Fig. 6. Our results suggest that under theexperimental conditions, if servers at un-utilized data cen-ters were low-powered using DVFS, we could save about60% on electricity costs compared to the case where suchdata centers were allowed to run on idle. The scaling ofpower cost reduction is linear and tapers out when idlingpower consumption is less than the power consumptionoffered by DVFS.

5.5. Scenario 5 (margin for short-term workload variation)

We have used traces of cumulative hourly workload,which is quite smoothed out, whereas there are variationsin instantaneous workload on short time-scales. If capacityis provisioned for average workload, then queues wouldbuild up sometimes with a negative impact on perfor-mance. In order to avoid that, some reserve margin in datacenter capacity may be kept in every interval on top of theexpected workload.

Fig. 7 shows the relationship between the reservemargin and the electricity cost. The reserve margin isvaried between 1% and 10% of a data centers capacity.The base-line is RED-BL cost when no margin is kept, andworkload is well-behaved. The lower line in the graph0.6 0.7 0.8 0.9 1

aling Factor

em down, the electricity cost savings would be reduced, but the approachce in RED-BL electricity cost when DVFS is used to reduce server powern idle).

1 2 3 4 5 6 7 8 9 10x 103Workload margin (normalized)

margin reachedmargin not needed

0

5

10

15

20

25

30

35

40

Dif

fere

nce

in c

ost f

rom

no

mar

gin

case

(%

)

Fig. 7. Some reserve margin of server capacity must be reserved at each data center to handle unexpected workload variations. This figure plots thepercentage increase in RED-BL electricity cost for varying degrees of reserve margin compared to the RED-BL electricity cost in the absence of reservemargin and well-behaved workload. The upper line shows the worst-case, i.e., workload exhausts all reserves. The lower line is the best-case, i.e., reservecapacity is not needed.

4 For workload forecasting, we trained an ARMA(4,4) [54] model on adays workload and used it to forecast the workload for the rest of the week.

150 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155shows the percentage increase from the baseline if theactual average workload is equal to what was estimated,i.e., the reserve capacity did not compute any workloadand only accounted for idling costs. The upper line in thegraph shows the case if workload is so excessive that allreserve capacity is needed to compute workload. Depend-ing upon the magnitude of the spike above the expectedworkload, the actual difference in electricity costcompared to the baseline would be somewhere betweenthe two lines.

5.6. Scenario 6 (Sliding Window Optimization)

All of our prior simulation scenarios were driven byerror-free workload traces. The underpinning assumptionto the corresponding results, therefore, is the availabilityof accurate workload estimates. We opine that this is notsuch a bad assumption given that the cumulative workloadon the granularity of an hour changes slowly from onehour to the next and from one hour on a day to the samehour the next day. Nevertheless, workload forecasting willhave some error, however small.

The state trajectory proposed by RED-BL for a planningwindow would be different if erroneous workload esti-mates were used instead of error-free estimates. In partic-ular, the projected state in a given interval may beinfeasible in the sense that the active resources may beinsufficient for the actual workload. In such a case, somemore resources may need to be activated. Similarly, theprojected state for some interval may be sub-optimal inthe sense that more resources may be active than what isneeded for the actual workload. In such a case, it may bedesirable to deactivate some of the resources. In a realdeployment, therefore, one must periodically correct theprojected state trajectory to be as close to the optimal statetrajectory as possible.

Receding horizon control (RHC) [53] is a strategy that iscommonly used in such situations. At a given interval, RHCmakes a forecast for a number of future intervals, known asthe horizon. Based on the projected horizon, the optimalcurrent state is picked. The forecasting and state correctionis repeated at every interval to get more accurate forecastsfor future intervals by exploiting the availability of morehistoric data about the workload. We call this step ofrepeated forecasting and subsequent generation of aRED-BL plan global trajectory correction.

Since the RED-BL optimization problem is an NP-Com-plete problem, it may not be feasible for very large scaledeployments to invoke it at every interval. For this reason,we formulated a generalization of the RHC which we callsliding window re-optimization. We define a parameter ccalled the window slide latency. At interval number 1,the workload for the next n intervals is forecast4 and a pro-jected RED-BL state trajectory is calculated. The same thingis done c intervals later. In other words, the planning over ann interval horizon is done at a longer time-scale compared toRHC. To accommodate for infeasible or sub-optimal states inthe intervals 1 through c, i.e., on a short time-scale, we per-form a local trajectory correction. The local trajectory correc-tion only looks at the projected state in the current intervaland the actual workload to determine a corrected state forthe current interval only. This avoids computation of statesover an entire planning window reducing the size of theNP-Complete problem that needs to be solved.

The local trajectory correction step is shown in Fig. 11.We start at the initial state S0. Based on workload forecastfor the next n intervals, we project a RED-BL state trajec-tory and transition to the projected state bS1 at the begin-ning of interval 1. However, some time during interval 1,we realize that the actual workload is different from ourestimates. To adapt to this situation, we transition to alocally better state S1. Until c intervals elapse, we willcontinue to perform local trajectory correction. That is,we will assume that the estimate for interval 2 is accurateand transition to the planned state bS2. During interval 2,the local trajectory correction process repeats once we

0 5 10 15 20 250

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Sliding Window Size

Mea

n W

ork

load

Pre

dic

tio

n E

rro

r

Fig. 9. Mean absolute workload prediction error vs sliding window size.

15 10 5 0 5 10 15 20 250

0.05

0.1

0.15

0.2

0.25

Workload estimation error (%)

Frac

tion

of ti

me

Fig. 10. Distribution of workload prediction error for sliding window sizeof 12 hours.

Fig. 11. Local trajectory correction technique for three consecutiveintervals.

Try all possible values forwindow-slide latency

Local trajectory correction

Global trajectory correction

j = j + 1

i < c_ j = 1

i = i + 1

= +1

Fig. 8. Flow for Sliding Window Optimization experiments.

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 151realize that the actual workload is different from theestimated one.

The local trajectory correction for interval j is an optimi-zation problem that attempts to minimize the electricitycost of the corrected state Sj and the cost of transitionbetween the planned state bSj and the corrected state. Themixed integer linear programming formulation for thelocal trajectory correction step for interval j is as follows:

minimizeXmi1

cieji p

jik f 1 f

xjici

!bji b

jirs

ji s

jid

!18

subject to:

xji 6 ci8i 19Xmi1

xji wj 20

pji; bji; s

ji; p

ji; b

ji; s

ji 2 f0;1g8i 21

pji P xji8i 22

bji P pji p

ji8i 23

sji P pji p

ji8i 24

bji P pji p

j1i 8i 25

sji P pj1i p

ji8i 26Given that the planning window size is n intervals, thepossible values for c are 2;3; . . . ; c. Fig. 8 shows the flow ofour experiments. The leftmost dashed polygon representsthe loop that cycles through all possible values of c. Asan example, consider the iteration of the outer loop wherec 2. We start by forecasting the workload for the first

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

1

2

3

4

5

6

Size of independent (de)activation unit as a multiple of a data center capacity

Perc

enta

ge

incr

ease

in e

lect

rici

ty c

ost

sav

ing

sco

mp

ared

to v

anill

a RE

D

BL

Fig. 13. Cost saving vs (de)activation granularity.

0.2 0 0.2 0.4 0.6 0.8 1 1.20

2

4

6

8

10

12

Activation/Deactivation Overhead

Perc

enta

ge

erro

r fo

r Heu

rist

ic 1 f = 0.65

Fig. 14. The minimum, maximum and average percentage differencebetween the cost of our heuristic and RED-BL.

0 5 10 15 20 255

5.5

6

6.5

7

7.5

Sliding Window Size (Hours)C

ost

Est

imat

ion

Err

or (

%)

Fig. 12. Percentage error of sliding window forecasts compared to global optimal with error-free workload.

152 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155n-intervals, denoted by cW 21 w21;1; w22;1; . . . ; w2n;1. Here,w2j;1, for instance, represents the workload forecast forinterval j during the first forecasting operation while the

value of c is 2. Using cW 21 as the expected workload vector,we propose a RED-BL deployment plan for the first n-inter-vals. At the start of the third interval (after the lapse of cintervals), we forecast the workload for the next n inter-vals, leveraging the additional information about the actualworkload for the first two intervals which was notavailable in the first forecast step at t 0. This forecast isdenoted by cW 22 w23;2; w24;2; . . . ; w2n2;2. Then, we computethe RED-BL deployment plan for intervals 3;4; . . . ;n 2 asthe global trajectory correction step. Since the windowslide latency is c and the number of intervals in our exper-iments is n, the number of times the window must slide,for a given value of c is dn=ce.

Having trained an ARMA(4,4) model on the first daysdata, we ran experiments for the last six days workloadin our dataset. We computed the average error of the dailyelectricity cost reported by these experiments compared tothe total daily electricity cost for the same period with per-fect workload estimates. The size of the planning windowwas set to 24 hours.

The first set of results in this scenario is the percentageworkload estimation error for various values of windowslide latency. We see in Fig. 9 that the mean absolute per-centage prediction error is less than 1%. The minimummean error is for a window slide latency of 12 hours. Forthis sliding window size, the distribution of percentageworkload estimation error is plotted in Fig. 10. Most ofthe workload estimates are quite close to error-free, whilea few estimates are as much as 24% off. This low averageerror for c = 12 is expected due to the diurnal cycles inworkload volume.

The difference of the electricity cost resulting from theuse of the sliding window trajectory correction approachcompared to the optimal solution with perfect workloadknowledge is plotted in Fig. 12. We see that the electricitycost achievable with RED-BL in a sliding window fashion iswithin 57% of the optimal cost achievable with perfectworkload estimates for all values of c. Also note that thepure RHC strategy turns out to be the best. This is expectedbecause a RED-BL plan based on knowledge of the currentactual workload as well as forecast for the next n 1intervals is better than a greedy decision based only onthe current actual workload.5.7. Scenario 7 (performance of the heuristic algorithm)

Fig. 14 shows the performance of the heuristic algo-rithm that we proposed in Section 3 compared to the opti-mal solution of the problem for various values of the(de)activation overhead parameters. For each value of theb and s parameters, we have plotted the average error overthe seven days in our workload dataset (the curve) as well

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 153as the minimum and maximum error for any given day(the vertical bars). Since our heuristic is designed to avoidactivation/deactivation, it performs poorly when the valueof b and s parameter is low. As the value of b and sincreases, our heuristics error compared to the optimalsolution drops until it starts a slight rise. When the valueof b and s parameters is higher compared to the value off, it may sometimes be better to activate the elasticresources in a data center a few intervals earlier than theyare needed. Suppose that the resources at a certain datacenter are needed in interval i. There may be some intervali that has a lower electricity price and the sum of theidling costs for intervals i through i 1 plus the activa-tion cost in interval i may be less than the cost of acti-vation in interval i. It is also possible that some timesdelayed deactivation of elastic resources may be better inthe long rum. We observed similar trends for other valuesof f as well, when b and s parameter values are varied from0 to 1.6. Discussion

Our work opens several avenues for further studies.Some of these future directions involve considering morespecific sources of transition costs instead of an abstractmodel of transition costs.

Deactivating the elastic load at data centers with no-load might change the latency to some of the clients.Increased latency is reported to result in loss in revenue[55]. This information could be incorporated into theRED-BL optimization problem to maximize the operatorrevenue instead of simply minimizing the electricitycost.

Due to convergence delays inherent in the relocationmechanism, some clients might notice the change inworkload mapping only after the lapse of considerabledelay following a network state change. Meanwhile,the operator cannot deactivate the old data centersbecause some workload would continue to be routedthere. This poses additional challenges for the RED-BLframework and a more detailed study of the trade-offsbetween energy cost and performance would be useful.

Inter-data center traffic costs are quite high [1]. Main-taining replication amongst data centers will incursome overhead in terms of the cost of replication traffic.Furthermore, if the elastic load at a data center is re-activated after being inactive for several intervals, it isunclear how much cost would be incurred to bringthe replica back to the same level of consistency asthe rest of the network. We think that it would be usefulto investigate how this can be incorporated into thetransition costs.

7. Conclusion

Geo-temporal diversity in electricity prices coupledwith geographic diversity in typically over-provisioneddata center network suggests a possibility of smartlyallocating resources to save electricity costs. Previouslyproposed approaches to such dynamic workload relocationmostly ignored the cost of transitions between data centernetwork states in consecutive intervals. We have providedan extensive simulation study of this idea while consider-ing such transition costs.

Our results indicate that ignoring transition costs canresult in a significant erosion of possible electricity costsavings. Furthermore, our approach of incorporating tran-sition costs in a global optimization, called RED-BL, scalesbetter with the magnitude of transition costs than the pre-viously proposed RED-CL approaches.

Frameworks such as ours rely on accurate estimates ofworkload. To compensate for errors in workload estimates,we propose RED-BL with trajectory correction, wherebywe perform a revision of workload estimates on a slid-ing-window basis, coupled with a local re-optimizationof the network state at every interval in a planning win-dow. Through our experiments, we found that using work-load predicted by an ARMA model, RED-BL with trajectorycorrection achieves an electricity cost that is, on average,around 6% of the cost achievable using perfect workloadestimates.

In another set of experiments, we found that if an oper-ator is able to (de)activate portions of the elastic load intheir data center, they can use RED-BL to cut their electric-ity cost even further. The additional savings increasealmost linearly as the number of units of (de)activationwithin a data center increases.

References

[1] A. Greenberg, J. Hamilton, D.A. Maltz, P. Patel, The cost of a cloud:research problems in data center networks, Comput. Commun. Rev.39 (1) (2009) 6873.

[2] A. Qureshi, Plugging into energy market diversity, in: 7th ACMHotNets, Calgary, Canada, 2008, pp. 16.

[3] K.G. Brill, The invisible crisis in the data center: the economicmeltdown of moores law, Tech. rep., The Uptime Institute, 2007.

[4] C.L. Belady, In the data center, power and cooling costs more thanthe IT equipment it supports, Electron. Cool. 23 (1) (2007).

[5] L.A. Barroso, U. Holzle, The case for energy-proportional computing,Computer 40 (2007) 3337.

[6] X. Fan, W.-D. Weber, L.A. Barroso, Power provisioning for awarehouse-sized computer, in: ISCA, ACM Press, New York, NY,USA, 2007, pp. 1323.

[7] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, B. Maggs, Cutting theelectric bill for internet-scale systems, in: ACM SIGCOMM, Barcelona,Spain, 2009, pp. 123134.

[8] D. Meisner, B.T. Gold, T.F. Wenisch, Powernap: eliminating serveridle power, in: Proceedings of the 14th International Conference onArchitectural Support for Programming Languages and OperatingSystems, ASPLOS XIV, ACM, New York, NY, USA, 2009, pp. 205216.

[9] J. Li, Z. Li, K. Ren, X. Liu, Towards optimal electric demandmanagement for internet data centers, IEEE Trans. Smart Grid 3 (1)(2012) 183192.

[10] M. Lin, A. Wierman, L. Andrew, E. Thereska, Dynamic right-sizing forpower-proportional data centers, in: INFOCOM, 2011, pp.10981106.

[11] Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang, N. Gautam,Managing server energy and operational costs in hosting centers, in:ACM SIGMETRICS, ACM, New York, NY, USA, 2005, pp. 303314.

[12] M. Mazzucco, D. Dyachuk, Balancing electricity bill and performancein server farms with setup costs, Future Gener. Comp. Syst. 28 (2)(2012) 415426.

[13] L. Rao, X. Liu, L. Xie, W. Liu, Minimizing electricity cost: optimizationof distributed internet data centers in a multi-electricity-marketenvironment, in: INFOCOM, San Diego, USA, 2010, pp. 19.

[14] J. Pang, A. Akella, A. Shaikh, B. Krishnamurthy, S. Seshan, On theresponsiveness of DNS-based network control, in: IMC, ACM, 2004,pp. 2126.

http://refhub.elsevier.com/S1389-1286(14)00247-3/h0005http://refhub.elsevier.com/S1389-1286(14)00247-3/h0005http://refhub.elsevier.com/S1389-1286(14)00247-3/h0005http://refhub.elsevier.com/S1389-1286(14)00247-3/h0280http://refhub.elsevier.com/S1389-1286(14)00247-3/h0280http://refhub.elsevier.com/S1389-1286(14)00247-3/h0025http://refhub.elsevier.com/S1389-1286(14)00247-3/h0025http://refhub.elsevier.com/S1389-1286(14)00247-3/h0030http://refhub.elsevier.com/S1389-1286(14)00247-3/h0030http://refhub.elsevier.com/S1389-1286(14)00247-3/h0030http://refhub.elsevier.com/S1389-1286(14)00247-3/h0030http://refhub.elsevier.com/S1389-1286(14)00247-3/h0040http://refhub.elsevier.com/S1389-1286(14)00247-3/h0040http://refhub.elsevier.com/S1389-1286(14)00247-3/h0040http://refhub.elsevier.com/S1389-1286(14)00247-3/h0040http://refhub.elsevier.com/S1389-1286(14)00247-3/h0040http://refhub.elsevier.com/S1389-1286(14)00247-3/h0045http://refhub.elsevier.com/S1389-1286(14)00247-3/h0045http://refhub.elsevier.com/S1389-1286(14)00247-3/h0045http://refhub.elsevier.com/S1389-1286(14)00247-3/h0055http://refhub.elsevier.com/S1389-1286(14)00247-3/h0055http://refhub.elsevier.com/S1389-1286(14)00247-3/h0055http://refhub.elsevier.com/S1389-1286(14)00247-3/h0055http://refhub.elsevier.com/S1389-1286(14)00247-3/h0060http://refhub.elsevier.com/S1389-1286(14)00247-3/h0060http://refhub.elsevier.com/S1389-1286(14)00247-3/h0060http://refhub.elsevier.com/S1389-1286(14)00247-3/h0070http://refhub.elsevier.com/S1389-1286(14)00247-3/h0070http://refhub.elsevier.com/S1389-1286(14)00247-3/h0070http://refhub.elsevier.com/S1389-1286(14)00247-3/h0070

154 M.S. Ilyas et al. / Computer Networks 72 (2014) 140155[15] C. Labovitz, A. Ahuja, A. Bose, F. Jahanian, Delayed internet routingconvergence, in: Proc. of the ACM SIGCOMM, 2000, pp. 175187.

[16] M.S. Ilyas, S. Raza, C.-C. Chen, Z.A. Uzmi, C.-N. Chuah, RED-BL: energysolution for loading data centers, in: INFOCOM12, 2012,pp. 28662870.

[17] E. Kayaaslan, B.B. Cambazoglu, R. Blanco, F.P. Junqueira, C.Aykanat, Energy-price-driven query processing in multi-centerweb search engines, in: 34th ACM SIGIR, ACM, New York, NY,USA, 2011, pp. 983992.

[18] N. Buchbinder, N. Jain, I. Menache, Online job-migration forreducing the electricity bill in the cloud, in: Proceedings of the10th International IFIP TC 6 Conference on Networking VolumePart I, NETWORKING11, Springer-Verlag, Berlin, Heidelberg, 2011,pp. 172185.

[19] U. Bhaskar, L. Fleischer, Online mixed packing and covering, CoRRabs/1203.6695.

[20] R. Urgaonkar, B. Urgaonkar, M.J. Neely, A. Sivasubramaniam, Optimalpower cost management using stored energy in data centers, in:SIGMETRICS, ACM, New York, NY, USA, 2011, pp. 221232.

[21] M. Mazzucco, D. Dyachuk, R. Deters, Maximizing cloud providersrevenues via energy aware allocation policies, CoRR abs/1102.3058.

[22] F.Y.-K. Oh, H.S. Kim, H. Eom, H.Y. Yeom, Enabling consolidation andscaling down to provide power management for cloud computing,in: USENIX HotCloud, USENIX Association, Berkeley, CA, USA, 2011,p. 14.

[23] J.S. Chase, D.C. Anderson, P.N. Thakar, A.M. Vahdat, R.P. Doyle,Managing energy and server resources in hosting centers, SIGOPSOper. Syst. Rev. 35 (5) (2001) 103116.

[24] G. Chen, W. He, J. Liu, S. Nath, L. Rigas, L. Xiao, F. Zhao, Energy-awareserver provisioning and load dispatching for connection-intensiveinternet services, in: USENIX NSDI, USENIX Association, Berkeley,CA, USA, 2008, pp. 337350.

[25] L. Rao, X. Liu, L. Xie, Z. Pang, Hedging against uncertainty: a tale ofinternet data center operations under smart grid environment, IEEETrans. Smart Grid 2 (3) (2011) 555563.

[26] K. Le, R. Bianchini, M. Martonosi, T.D. Nguyen, Cost- and energy-aware load distribution across data centers, in: HotPower, 2009,pp. 15.

[27] X. Zheng, Y. Cai, Energy-aware load dispatching in geographicallylocated internet data centers, Sust. Comput.: Inform. Syst. 1 (4)(2011) 275285.

[28] G. Koutitas, P. Demestichas, Challenges for energy efficiency in localand regional data centers, J. Green Eng. 1 (1) (2010) 132.

[29] Z. Abbasi, T. Mukherjee, G. Varsamopoulos, S. Gupta, Dahm: a greenand dynamic web application hosting manager acrossgeographically distributed data centers, Atlanta 60 (2011) 80.

[30] L. Chiaraviglio, I. Matta, Greencoop: cooperative green routing withenergy-efficient servers, in: Proceedings of the 1st InternationalConference on Energy-Efficient Computing and Networking, ACM,2010, pp. 191194.

[31] D.H. Phan, J. Suzuki, R. Carroll, S. Balasubramaniam, W. Donnelly, D.Botvich, Evolutionary multiobjective optimization for green clouds,in: ACM GECCO 2012, ACM, 2012, pp. 1926.

[32] Z. Liu, M. Lin, A. Wierman, S.H. Low, L. Andrew, Greeninggeographical load balancing, in: ACM SIGMETRICS, ACM, San Jose,California, USA, 2011, pp. 233244.

[33] Z. Liu, M. Lin, A. Wierman, S.H. Low, L. Andrew, Geographical loadbalancing with renewables, in: ACM GREENMETRICS, ACM, 2011,pp. 6266.

[34] A. Sucevic, L.L. Andrew, T.T. Nguyen, Powering down for energyefficient peer-to-peer file distribution, SIGMETRICS Perform. Eval.Rev. 39 (3) (2011) 7276.

[35] Z. Fang, L. Zhao, R.R. Iyer, C.F. Fajardo, G.F. Garcia, S.E. Lee, B. Li, S.R.King, X. Jiang, S. Makineni, Cost-effectively offering private buffers inSoCs and CMPs, in: Proceedings of the International Conference onSupercomputing, ICS 11, ACM,NewYork,NY,USA, 2011, pp. 275284.

[36] F. Javed, N. Arshad, On the use of linear programming in optimizingenergy costs, in: 3rd IWSOS, Springer-Verlag, Berlin, Heidelberg,2008, pp. 305310.

[37] T. Logenthiran, D. Srinivasan, A.M. Khambadkone, Multi-agentsystem for energy resource scheduling of integrated microgrids ina distributed system, Electr. Power Syst. Res. 81 (1) (2011) 138148.

[38] G. Celli, F. Pilo, Optimal distributed generation allocation in mvdistribution networks, in: 22nd IEEE PICA 2001, 2001, pp. 81 86.

[39] F. Javed, N. Arshad, Adopt: an adaptive optimization framework forlarge-scale power distribution systems, IEEE Int. Conf. Self-Adapt.Self-Org. Syst. (2009) 254264.

[40] M.A. Marsan, L. Chiaraviglio, D. Ciullo, M. Meo, A simple analyticalmodel for the energy-efficient activation of access points in densewlans, in: Proceedings of the 1st International Conference onEnergy-Efficient Computing and Networking, e-Energy 10, ACM,New York, NY, USA, 2010, pp. 159168.

[41] C. Cavdar, A. Yayimli, L. Wosinska, How to cut the electric bill inoptical wdm networks with time-zones and time-of-use prices, in:2011 37th European Conference and Exhibition on OpticalCommunication (ECOC), 2011, pp. 13.

[42] C. Peng, S.-B. Lee, S. Lu, H. Luo, H. Li, Traffic-driven power saving inoperational 3G cellular networks, in: 17th MobiCom, ACM, NewYork, NY, USA, 2011, pp. 121132.

[43] S. Lee, S. Sahu, Efficient server consolidation considering intra-cluster traffic, in: GLOBECOM, 2011, pp. 16.

[44] E. Pinheiro, R. Bianchini, E.V. Carrera, T. Heath, Load balancing andunbalancing for power and performance in cluster-based systems,in: Compilers and Operating Systems for Low Power, 2001.

[45] Y. Yao, L.Huang,A. Sharma, L.Golubchik,M.Neely,Data centerspowerreduction: a two time scale approach for delay tolerantworkloads, in:INFOCOM, 2012 Proceedings IEEE, 2012, pp. 1431 1439.

[46] H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F.B. Cetin, S. Babu,Starfish: a self-tuning system for big data analytics, in: 5th BiennialConference on Innovative Data Systems Research (CIDR) 2011, 2011,pp. 261 272.

[47] H. Herodotou, F. Dong, S. Babu, No one (cluster) size fits all:automatic cluster sizing for data-intensive analytics, in: 2nd ACMSOCC, ACM, New York, NY, USA, 2011, pp. 18:118:14.

[48] D.Aikema,R. Simmonds, Electrical cost savingsandcleanenergyusagepotential for hpc workloads, in: 2011 IEEE International Symposiumon Sustainable Systems and Technology (ISSST), 2011, pp. 16.

[49] S. Pelley, D. Meisner, T.F. Wenisch, J.W. VanGilder, Understandingand abstracting total data center power, in: Proc. 2009 Workshop onEnergy Efficient Design (WEED 09).

[50] N. Padhy, Unit commitment-a bibliographical survey, IEEE Trans.Power Syst. 19 (2) (2004) 11961205.

[51] A. Nazir, S. Raza, C.-N. Chuah, Unveiling facebook: a measurementstudy of social network based applications, in: IMC, ACM, New York,NY, USA, 2008, pp. 4356.

[52] A. Gandhi, M. Harchol-Balter, R. Das, C. Lefurgy, Optimal powerallocation in server farms, SIGMETRICS Perform. Eval. Rev. 37 (1)(2009) 157168.

[53] W.H. Kwon, S.H. Han, Receding Horizon Control, Elsevier, 2005.[54] G. Box, G.M. Jenkins, G.C. Reinsel, Time Series Analysis: Forecasting

and Control, Springer, 1994.[55] E. Schurman, J. Brutlag, The User and Business Impact of Server

Delays, Additional Bytes, and HTTP Chunking in Web Search, June2009. (accessed 03.04.13).

Muhammad Saqib Ilyas is a Ph.D. Candidateat the Department of Computer Science at theSchool of Science and Engineering, LUMS,Lahore Pakistan. He holds an MS EE degreefrom Wichita State University, Wichita, KS,USA and a Bachelor of Computer SystemEngineering from N.E.D. University of Engi-neering and Technology, Karachi, Pakistan. Hecurrently focuses on performance and energyoptimization issues in large scale networks.Saqib Raza received the B.S. degree from theLahore University of Management Sciences,Lahore, Pakistan, in 2004, and the M.S. andPh.D. degrees from the University of Califor-nia, Davis, in 2007 and 2010, respectively, allin computer science. He is a Software Engi-neer with the Data Center Switching Tech-nology Group, Cisco Systems, San Jose, CA.

http://refhub.elsevier.com/S1389-1286(14)00247-3/h0085http://refhub.elsevier.com/S1389-1286(14)00247-3/h0085http://refhub.elsevier.com/S1389-1286(14)00247-3/h0085http://refhub.elsevier.com/S1389-1286(14)00247-3/h0085http://refhub.elsevier.com/S1389-1286(14)00247-3/h0085http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0090http://refhub.elsevier.com/S1389-1286(14)00247-3/h0100http://refhub.elsevier.com/S1389-1286(14)00247-3/h0100http://refhub.elsevier.com/S1389-1286(14)00247-3/h0100http://refhub.elsevier.com/S1389-1286(14)00247-3/h0100http://refhub.elsevier.com/S1389-1286(14)00247-3/h0110http://refhub.elsevier.com/S1389-1286(14)00247-3/h0110http://refhub.elsevier.com/S1389-1286(14)00247-3/h0110http://refhub.elsevier.com/S1389-1286(14)00247-3/h0110http://refhub.elsevier.com/S1389-1286(14)00247-3/h0110http://refhub.elsevier.com/S1389-1286(14)00247-3/h0115http://refhub.elsevier.com/S1389-1286(14)00247-3/h0115http://refhub.elsevier.com/S1389-1286(14)00247-3/h0115http://refhub.elsevier.com/S1389-1286(14)00247-3/h0120http://refhub.elsevier.com/S1389-1286(14)00247-3/h0120http://refhub.elsevier.com/S1389-1286(14)00247-3/h0120http://refhub.elsevier.com/S1389-1286(14)00247-3/h0120http://refhub.elsevier.com/S1389-1286(14)00247-3/h0120http://refhub.elsevier.com/S1389-1286(14)00247-3/h0125http://refhub.elsevier.com/S1389-1286(14)00247-3/h0125http://refhub.elsevier.com/S1389-1286(14)00247-3/h0125http://refhub.elsevier.com/S1389-1286(14)00247-3/h0135http://refhub.elsevier.com/S1389-1286(14)00247-3/h0135http://refhub.elsevier.com/S1389-1286(14)00247-3/h0135http://refhub.elsevier.com/S1389-1286(14)00247-3/h0140http://refhub.elsevier.com/S1389-1286(14)00247-3/h0140http://refhub.elsevier.com/S1389-1286(14)00247-3/h0145http://refhub.elsevier.com/S1389-1286(14)00247-3/h0145http://refhub.elsevier.com/S1389-1286(14)00247-3/h0145http://refhub.elsevier.com/S1389-1286(14)00247-3/h0150http://refhub.elsevier.com/S1389-1286(14)00247-3/h0150http://refhub.elsevier.com/S1389-1286(14)00247-3/h0150http://refhub.elsevier.com/S1389-1286(14)00247-3/h0150http://refhub.elsevier.com/S1389-1286(14)00247-3/h0150http://refhub.elsevier.com/S1389-1286(14)00247-3/h0155http://refhub.elsevier.com/S1389-1286(14)00247-3/h0155http://refhub.elsevier.com/S1389-1286(14)00247-3/h0155http://refhub.elsevier.com/S1389-1286(14)00247-3/h0155http://refhub.elsevier.com/S1389-1286(14)00247-3/h0160http://refhub.elsevier.com/S1389-1286(14)00247-3/h0160http://refhub.elsevier.com/S1389-1286(14)00247-3/h0160http://refhub.elsevier.com/S1389-1286(14)00247-3/h0160http://refhub.elsevier.com/S1389-1286(14)00247-3/h0165http://refhub.elsevier.com/S1389-1286(14)00247-3/h0165http://refhub.elsevier.com/S1389-1286(14)00247-3/h0165http://refhub.elsevier.com/S1389-1286(14)00247-3/h0165http://refhub.elsevier.com/S1389-1286(14)00247-3/h0170http://refhub.elsevier.com/S1389-1286(14)00247-3/h0170http://refhub.elsevier.com/S1389-1286(14)00247-3/h0170http://refhub.elsevier.com/S1389-1286(14)00247-3/h0175http://refhub.elsevier.com/S1389-1286(14)00247-3/h0175http://refhub.elsevier.com/S1389-1286(14)00247-3/h0175http://refhub.elsevier.com/S1389-1286(14)00247-3/h0175http://refhub.elsevier.com/S1389-1286(14)00247-3/h0175http://refhub.elsevier.com/S1389-1286(14)00247-3/h0180http://refhub.elsevier.com/S1389-1286(14)00247-3/h0180http://refhub.elsevier.com/S1389-1286(14)00247-3/h0180http://refhub.elsevier.com/S1389-1286(14)00247-3/h0180http://refhub.elsevier.com/S1389-1286(14)00247-3/h0185http://refhub.elsevier.com/S1389-1286(14)00247-3/h0185http://refhub.elsevier.com/S1389-1286(14)00247-3/h0185http://refhub.elsevier.com/S1389-1286(14)00247-3/h0195http://refhub.elsevier.com/S1389-1286(14)00247-3/h0195http://refhub.elsevier.com/S1389-1286(14)00247-3/h0195http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0200http://refhub.elsevier.com/S1389-1286(14)00247-3/h0210http://refhub.elsevier.com/S1389-1286(14)00247-3/h0210http://refhub.elsevier.com/S1389-1286(14)00247-3/h0210http://refhub.elsevier.com/S1389-1286(14)00247-3/h0210http://refhub.elsevier.com/S1389-1286(14)00247-3/h0235http://refhub.elsevier.com/S1389-1286(14)00247-3/h0235http://refhub.elsevier.com/S1389-1286(14)00247-3/h0235http://refhub.elsevier.com/S1389-1286(14)00247-3/h0235http://refhub.elsevier.com/S1389-1286(14)00247-3/h0250http://refhub.elsevier.com/S1389-1286(14)00247-3/h0250http://refhub.elsevier.com/S1389-1286(14)00247-3/h0255http://refhub.elsevier.com/S1389-1286(14)00247-3/h0255http://refhub.elsevier.com/S1389-1286(14)00247-3/h0255http://refhub.elsevier.com/S1389-1286(14)00247-3/h0255http://refhub.elsevier.com/S1389-1286(14)00247-3/h0260http://refhub.elsevier.com/S1389-1286(14)00247-3/h0260http://refhub.elsevier.com/S1389-1286(14)00247-3/h0260http://refhub.elsevier.com/S1389-1286(14)00247-3/h0265http://refhub.elsevier.com/S1389-1286(14)00247-3/h0265http://refhub.elsevier.com/S1389-1286(14)00247-3/h0270http://refhub.elsevier.com/S1389-1286(14)00247-3/h0270http://refhub.elsevier.com/S1389-1286(14)00247-3/h0270http://oreil.ly/fTmYwz

M.S. Ilyas et al. / Computer Networks 72 (2014) 140155 155Chao-Chih Chen received his Ph.D. at Uni-versity of California, Davis, CA and is currentlya software engineer at Windows Azureworking on data center network manage-ment. His research interests include datacenter networks and network management.Zartash Afzal Uzmi is an associate professorof electrical engineering and computer sci-ence at LUMS. He received his B.Sc. degreefrom UET, Taxila, and M.S. and Ph.D. degreesfrom Stanford University, all in electricalengineering. He works in communicationsand networking with a particular focus onMPLS traffic engineering and restorationrouting, connection preemption in multiclassnetworks, routing protocols for infrastructureand ad hoc networks, and signal processingfor communications. Previously, he held

positions at Bell Labs and Nokia Research Center. His current research ison aggregation mechanisms for routing protocols.Chen-Nee Chuah is a Professor in Electricaland Computer Engineering at the Universityof California, Davis. She received her Ph.D. inElectrical Engineering and Computer Sciencesfrom the University of California, Berkeley.Her research interests include Internet mea-surements, network management, anomalydetection, online social networks, and vehic-ular ad hoc networks. Chuah is an ACM Dis-tinguished Scientist. She received the NSFCAREER Award in 2003 and was named aChancellors Fellow of UC Davis in 2008. She

has served on the executive/technical program committee of several ACMand IEEE conferences. She was an Associate Editor for IEEE/ACM Trans-actions on Networking from 2008 to 2012.

RED-BL: Evaluating dynamic workload relocation for data center networks1 Introduction2 Related work3 Problem formulation3.1 Problem complexity and a heuristic

4 Experimental setup4.1 Application workload4.2 Electricity prices4.3 Algorithms for workload distribution/relocation

5 Results5.1 Scenario 1 (extent of over provisioning)5.2 Scenario 2 (activation/deactivation overhead)5.3 Scenario 3 (granular activation/deactivation)5.4 Scenario 4 (DVFS)5.5 Scenario 5 (margin for short-term workload variation)5.6 Scenario 6 (Sliding Window Optimization)5.7 Scenario 7 (performance of the heuristic algorithm)

6 Discussion7 ConclusionReferences