Peter X. Gao, Andrew R. Curtis,Bernard Wong, S. Keshav

Cheriton School of Computer ScienceUniversity of Waterloo

August 15, 2012

1

2

=CO2 of 280,000 cars~1M servers

Datacenters and Request RoutingDC 2

DC 1

Dynamic DNS

3

Where to route?Datacenter Latency Electricity Price Carbon footprint

DC1 (Texas) Low High High

DC2 (Washington) High Low Low

Hydro67%

Nuclear9%

Gas10%

Coal8%

Other6%

WashingtonNuclear

10%

Gas

45%

Coal37%

Other8%

Texas

4

Where to route?Datacenter Latency Electricity Price Carbon footprint

DC1 (Texas) Low High High

DC2 (Washington) High Low Low

5

Datacenter Latency Electricity Price Carbon footprint

DC1 (Texas) Low High Low

DC2 (Washington) High Low High

A.M.

P.M.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

50

100

150

200

250

300

350

400

450

Hours of a day

Carb

on F

ootp

rint (

g/kW

h)

Electricity carbon footprint in California

How to split?

6

DC 1

DC 2

80%

20%

FORTE and its ContributionsFORTE:

Flow Optimization based framework for Request-routing and Traffic Engineering

Contributions:– Principled framework for managing the three-way trade-

off between access latency, electricity cost, and carbon footprint• Green datacenter upgrade plans

– Impact of carbon taxes on datacenter carbon footprint reduction

7

Surprising Results

• FORTE can reduce datacenter carbon footprint by 10% with no increase in electricity cost and access latency

• Carbon Tax is not effective because taxes are only about 5% of electricity price

8Electricity Cost "Carbon Cost"

Outline

• Model• P1: Assigning users to datacenters• P2: Assigning data objects to datacenters• P3: Datacenter upgrade• Evaluation

9

Model

User Groups: ui

Datacenters: nj

Data Objects: dk

Requests r(ui, dk)

NY

LA

DC

10

Carbon emission: c(nj)Electricity price: e(nj)Capacity: cap(nj)

P3 P2P1

Model

User Groups: ui

Datacenters: nj

Data Objects: dk

Requests r(ui, dk)

servesis placed at

NY

LA

DC

11

Access latency:

l(u i, n j, d k)

Carbon emission: c(nj)Electricity price: e(nj)Capacity: cap(nj)

Latency Cost Function

Latency sensitivity

lmax

latency

latency cost: l(ui,nj)

12

Outline

• Model• P1: Assigning users to datacenters• P2: Assigning data objects to datacenters• P3: Datacenter upgrade• Evaluation

13

P1

User Groups: ui

Datacenters: nj

Data Objects: dk

14

Assigning Users to Datacenters

Objective Function

15

n3

n1

d2

User Groups: ui

Datacenters: nj

Data Objects: dk

u1Access latency:

l(u i, n j, d k)

𝐟 (𝐮𝐢 ,𝐧

𝐣 ,𝐝𝐤 )

n1

𝐟 (𝐮 𝐢 ,𝐧 𝐣 ,𝐝𝐤 ) { Weighted Carbon Cost: λ1c(nj)+ Weighted Electricity Cost: λ2e(nj) }+ Weighted Latency Cost: λ0l(ui, nj, dk)

Minimize: ∑

Carbon emission: c(nj)Electricity price: e(nj)

16

Demand Satisfaction Constraints

n3

n1

d2

User Groups: ui

Datacenters: nj

Data Objects: dk

Requests r(ui, dk)

u1Access latency:

l(u i, n j, d k)

𝐟 (𝐮𝐢 ,𝐧

𝐣 ,𝐝𝐤 )

∑𝐧 𝐣

𝐟 (𝐮𝐢 ,𝐧 𝐣 ,𝐝𝐤 )

n3

n1

¿ 𝐫 (𝐮𝐢 ,𝐝𝐤 )

Carbon emission: c(nj)Electricity price: e(nj)

17

Datacenter Capacity Constraints

n4

User Groups: ui

Datacenters: nj

Data Objects: dk

u2

u3

Capacity: cap(nj)

∑𝐮 𝐢 ,𝐝𝐤

𝐟 (𝐮𝐢 ,𝐧 𝐣 ,𝐝𝐤 )

n4

≤𝐜𝐚𝐩 (𝐧 𝐣 )

Scale of Linear Program

• Evaluation problem size:– Over 1 million variables– FORTE can solve it in approximately 2 min

• Actual problem:– Can be over 1 billion variables

18

Fast-FORTE

• Greedy Heuristic• Running time O(N logN) vs Simplex O(~N6)• Reduces running time from 2 minutes to 6

seconds• 0.3% approximation error

19

Outline

• Model• P1: Assigning users to datacenters• P2: Assigning data objects to datacenters• P3: Datacenter upgrade• Evaluation

20

P2

Assigning Data Objects to Datacenters

21

User Groups: ui

Datacenters: nj

Data Objects: dk

Assigning Data Objects to Datacenters

User Groups

DatacentersData Objects

requests

Σ Flow size = 100

Σ Flow size = 1

22

Σ Flow size = 101

Outline

• Model• P1: Assigning users to datacenters• P2: Assigning data objects to datacenters• P3: Datacenter upgrade• Evaluation

23

P3

Using FORTE for upgrading datacenters

24

User Groups: ui

Datacenters: nj

Data Objects: dk

Using FORTE for upgrading datacenters

Datacenter operators need to decide:– Which datacenters should be upgraded?– How many servers in that datacenter should be

upgraded?

The upgrade decisions are based on:– Estimation of future traffic demands– Annual budget on upgrading– Trade-off between cost and benefit

25

Using FORTE for upgrading datacenters

26

User Groups

Datacenters

Data Objects

requests

Can also be used for selecting new datacenter locations by adding zero size datacenters into the network

Outline

• Model• P1: Assigning users to datacenters• P2: Assigning data objects to datacenters• P3: Datacenter upgrade• Evaluation

27

DatasetsAkamai traffic data– Akamai delivers about 15% - 20% Internet traffic– 3 weeks coarse-grained data in U.S.– Aggregated every 5 minutes

U.S. Energy Information Administration– Carbon footprint– Electricity cost

Data Objects: Synthetic with long-tail popularity, 10% latency tolerant

28

Different Level of Carbon Reduction

29

0.7

0.8

0.9

1.0

Date

Carb

on (n

orm

aliz

ed)

Latency Only Small Reduction

Medium Reduction Large Reduction

Three-way Tradeoff

Tradeoff between carbon emissions, average distance, andelectricity costs. 30

Two-way Tradeoff between Carbon Emission and Electricity Cost

31

850 870 890 910 930 950 970 990 1010 1030 10505

5.2

5.4

5.6

5.8

6

6.2

6.4

6.6

6.8

7

Carb

on E

miss

ion

( ton

/hou

r)

(987, 6.5)

(987, 5.83)(1010, 5.73)

Electricity Cost ($/hour)

Will Carbon Taxes or Credits Work?

Akamai uses ~2 * 108 kWh per year

• Electricity cost of 2 * 108 kWh:2 * 108 kWh * 11.2c/kWh = $22.4 M

• “Carbon cost” of 2 * 108 kWh :2 * 108 kWh * 500g/kWh = 105 t

105 t * $10/t = $1 M

32

Electricity Cost "Carbon Cost"

Green Upgrades

33

WA1

CA1

CA2

TX1

NY1

NJ1

NJ2

Year1 Year 2 Year 3

Reduces carbon emission by ~25% compare to carbon oblivious plan

• Use Green Energy

• Use Green Energy• Reduce Access Latency

• Reduce Access Latency

• Low Electricity Price

34

Related Work

• Qureshi et. al., Cutting the electric bill for internet-scale systems, SIGCOMM 09

• Doyle et. al., Server Selection for Carbon Emission Control, GreenNet 11

• Other related work can be found in our paper

FORTE:– Considers data allocation problem– Supports datacenter upgrade– Explores the three-way trade-off

Conclusions

• FORTE is a request routing framework that can reduce carbon emissions by ~10% without affecting latency and electricity cost

• Surprisingly, carbon taxes do not provide sufficient incentives to reduce carbon emissions

• A green upgrade plan can further reduce carbon emissions by ~25% over 3 years

35

Acknowledgement

• We thank Prof. Bruce Maggs for providing us access to Akamai traces

• We thank our shepherd Prof. Fabian Bustamante and the reviewers for their insightful comments

36