Energy Sustainability in Cooperating Clouds

Antonio Celesti, Antonio Puliafito, Francesco Tusa, and Massimo VillariUniversita degli Studi di Messina, Facolta di Ingegneria

Contrada di Dio, S. Agata, 98166 Messina, Italy.e-mail: {ftusa,acelesti,apuliafito,mvillari}@unime.it

Keywords: Cloud Computing, Federation, Virtual Infrastructure Management, Energy Sustainability, PhotovoltaicSystems, Renewable Energy, Energy Optimization.

Abstract: Nowadays, cloud federation is paving the way toward new business scenarios in which it is possible to enforcemore flexible energy management strategies than in the past. Considering independent cloud providers, eachone is exclusively bounded to the specific energy supplier powering its datacenter. The situation radicallychange if we consider a federation of cloud providers each one powered by both a conventional energy supplierand a renewable energy generator. In such a context the opportune relocation of computational workloadamong providers can lead to a global energy sustainability policy for the whole federation. In this work, weinvestigate the advantages, constrains, and issues for the achievement of such a sustainable environment.

1 Introduction

Federation is the next frontier of cloud comput-ing. Throughout the federation, different small andmedium Cloud providers belonging to different orga-nizations, can join each other to achieve a commongoal, usually represented by the optimization of theirresources.

The basic idea is that a Cloud provider has notinfinite resources. In order to achieve target busi-ness scenarios a Cloud provider may need a flexi-ble infrastructure. Federation allows Cloud providersto achieve such a resilient infrastructure asking ad-ditional resources to other federation-enabled CloudProviders. Cloud federation is much more than themere use of resources provided by a mega-provider.

From a political point of view, the term federa-tion refers to a type of system organization character-ized by a joining of partially “self-governing” entitiesunited by a “central government”. In a federation,each self-governing status of the component entitiesis typically independent and may not be altered by aunilateral decision of the “central government”.

Besides cloud mega-providers, alsosmaller/medium providers are becoming populareven though the virtualization infrastructures theyhave deployed in their datacenters cannot directlycompete with the bigger counterparts. A way toovercome these resource limitation is representedby the promotion of federation mechanisms amongsmall/medium cloud providers. This allows to pickup the advantages of other form of economic model

considering societies, universities, research centersand organizations that commonly do not fully use theresources of their own physical infrastructures.

Moreover the traditional market where cloudproviders offer cloud-based services to their clients,federation triggers a new market where cloudproviders can buy and/or sell computing/storage ca-pabilities and services to other clouds. The advantageof transforming a physical datacenter in a cloud vir-tualization infrastructure in the perspective of cloudfederation is twofold. On the one hand, small/mediumcloud providers that rent resources to other providerscan optimize the use of their infrastructure, whichare often underutilized. On the other hand, ex-ternal small/medium cloud providers can elasticallyscale up/down their logical virtualization infrastruc-ture borrowing resources of other providers. Federa-tion enables cloud providers to relocate their servicesin other ones. In our opinion, this allows to plan moreflexible energy sustainability strategies than the past.

In this work, we focus on an innovative sustain-able federated cloud scenario in which resources arerelocated between cloud providers whose datacentersare partially powered by renewable energy generatorsystems. The federation is seen as a way for reduc-ing energy costs (Energy Cost Saving), but at thesame time a possibility to reduce the CO2 emissions(Energy Sustainability). Here we discuss strategiesand policies it is possible to apply in federated cloudenvironments for achieving the just mentioned goals.Specifically our assessment is aimed at the design ofan Energy Manager, to be included in our Virtual In-

frastructure Manager, that is the CLoud-Enabled Vir-tual EnviRonment (CLEVER). Further details on howto achieve such a goal will be provided in Section 4.

The manuscript is organized as follows. Section2 introduce how an energy sustainability strategy canbe applied to a federated cloud environment. The en-ergy consumption of a datacenter is affected by dif-ferent factors including the Power contribution forthe Information Technology (IT) equipment (PIT ), thePower contribution for the Electrical (POW) equip-ments (PPOW ), and the Power contribution for theCooling (COOL) equipments (PCOOL). To this regardsseveral energy considerations about cloud datacentersare discussed in Section 3. As said before, a deci-sion algorithm for sustainable federated clouds is pre-sented in Section 4. Section 5 discusses related works.Section 6 summarizes conclusions and lights to thefuture.

2 Cloud Federation and EnergySustainability

Federation brings new business opportunities forclouds. In fact, besides the traditional market wherecloud providers offer cloud-based services to theirclients, federation triggers a new market where cloudproviders can buy and/or sell computing/storage ca-pabilities and services from/to other clouds. The ad-vantage of transforming a physical datacenter in acloud virtualization infrastructure in the perspectiveof cloud federation is twofold. On the one hand,small/medium cloud providers that rent resources toother providers can optimize the use of their in-frastructure, which are often underutilized. On theother hand, small/medium cloud providers can elas-tically scale up/down their virtualization infrastruc-ture borrowing resources and paying them from otherproviders. A cloud provider can decide to lend re-sources to other clouds when it realizes that its data-center is under-utilized at given times. Typically, dat-acenters are under-utilized during the night and over-utilized during the morning. Therefore, as the data-center cannot be turned off, the cloud provider maydecide to turn the problem into a business opportu-nity. Instead, a cloud might need to buy resourcesfrom other clouds for the following reasons:

• The cloud runs out of its storage/computing re-sources. In order to continue providing cloud-based service to its clients, it decides to buy re-sources from other clouds.

• The cloud needs to deploy a distributed cloud-based service in different geographical locations;

hence, it acquires resources placed in those loca-tions.

• The cloud needs to relocate service instances inother clouds.

As federation enables cloud providers to relocate theirservices on other peers belonging to the system, inour opinion, it is possible to carry out more flexibleenergy-aware scenarios than the past, when we con-sidered independent non-federated clouds. Two pos-sible alternative energy-aware scenarios are:

• Energy Cost Saving. Resources are migrated inexternal cloud providers in order to push downthe energy consumption cost of their datacen-ters. The resources will be migrated in exter-nal cloud providers in which the cost of the en-ergy is cheaper. A possible approach could con-sist into turning off a given datacenter within asite and move the load towards another where theresources renting price is lower than the cost ofcomputing locally.

• Energy Sustainability. Resources are migratedin external cloud providers whose datacenters arepartially or totally powered by renewable energyjust for minimizing the costs due to the environ-mental impact (e.g. reducing the CO2 emissions).

An Energy Sustainability scenario often allows topush down costs, but unfortunately this is not alwaystrue. In fact, by now, energy-aware strategies basedon sustainable green computing environment have notfound so wide diffusion due to the high costs of en-ergy production. In our opinion a sustainable feder-ated cloud environment could push down such costsalso allowing to achieve a green computing environ-ment able to reduce the energy costs. Nevertheless,how to achieve such a sustainable federated cloud en-vironment is not clear at all.

The main contribution of this work is to proposea possible approach for the achievement of such anenvironment. Our approach is based on the followingidea: “moving the computation toward the more sus-tainable available cloud datacenter”. This statementis motivated by the following assumptions:

1. Often, the renewable energy generator systemsproduce more energy than necessary.

2. It is very hard to store the exceeded produced re-newable energy (e.g., in batteries).

3. Alternatively, it is becoming very difficult to putthe exceeded produced renewable energy in pub-lic electric grids. This practice is becoming aproblem for Energy suppliers as it implies uncon-trolled power surges which are hard to be man-aged. As this problem is becoming more and more

Figure 1: Example of Sustainable Federated Cloud Environment.

sensitive, the energy suppliers are becoming to bereluctant to absorb energy produced by private re-newable energy generator systems.

4. As consequence of 1), 2), and 3) often the ex-ceeded produced renewable energy is wasted.

5. Consequently, it is easier to move the computa-tion toward a datacenter powered by a renewableenergy generator system with a high large avail-ability of energy than moving the “green energy”toward the datacenter where the computing has totake place.

If we consider a set of datacenters with these features,a sustainable federated cloud environment can allowto save money, maximizing the use of “green energy”and reducing the level of carbon dioxide.

An example of such a scenario is depicted in Fig-ure 1. The sustainable federated cloud ecosystem in-cludes four cloud providers: Messina, Sidnay, SaoPaulo, and Stuttgard. The electricity suppliers A, B,C, and D are independent companies that provide en-ergy at different costs. In our scenario, the datacenterof each cloud providers is meanly powered by a pri-vate renewable energy generator system as primarysource of energy. When the primary source of energyis not enough to power the datacenter, the cloud usesthe energy of its Electricity Supplier. In addition, eachcloud is federated with each other in order to take theadvantages of service relocation and resource consol-idation. Further details regarding how federate cloudarchitecture are out of scope of this work. Further de-tails can be found in Section 5.

Each cloud provider joins the federation in or-der to move its computation in other federated clouds

where the production of “green energy” is maximum.In simple terms, we move the computation load to-wards the more efficient renewable energy generators(in term of produced electricity) maximizing the uti-lization of the federated clouds in which the workloadhas been transferred. Considering Figure 1, the fourclouds have different latitudes. According to a givenperiod (due to time zone and month), each renew-able energy generator system that primarily powerseach cloud datacenter can have different energy effi-ciency compared to each other. These different con-ditions can depend by different factors according tothe adopted source of renewable energy. For exam-ple the amount of energy produced by a photovoltaicsystem depends on the solar radiance which is differ-ent hour by hour, day by day, and month by month.In addition, the energy production of a photovoltaicsystem is also affected by the weather and climatecondictions. For simplicity, let us consider the timezone, when the time of Messina is 17:00, the time ofSao Paulo is 13:00. In this situation, in Sao Paulo thesolar radiance of the sun is stronger then the one inMessina. On the other hand, considering both citiesin July, the temperature of Messina will be proba-bly higher than the one in Sao Paulo. Further similarconsideration can be made considering the latitude ofthe two cities. This scenario implies that if a cloudprovider wants to enforce energy sustainability poli-cies on its own datacenter, it relocate its services intoother federated cloud providers, chosen according theaforementioned energy consideration. For reasons ofQuality of Service (QoS), let us suppose that eachcloud provider of the federation has replicated in ad-vance part of its services into other federated clouds.

Considering a federation of Infrastructure as Service(IaaS) clouds, service replication means copying Vir-tual Machines (VMs) disk-image into other federatedproviders. In this way the cloud providers that wantsto apply energy sustainability policies can turn off theblade center hosting its VMs and turn on the copiesof these VMs pre-arranged into other federated clouddatacenters, where the renewable energy productionis maximum according to temperature, latitude, andtime zone.

3 Power Consuption Considerationsof a Datacenter

The first step for the achievement of a SustainableCloud Federation is to better understand the main fac-tors affecting the total power consumption of a data-center. As already introduced these factors are PIT ,PPOW , and PCOOL. PIT is related to the total powerconsumption of the IT equipment such as:

• CPUs

• Storage (i.e., Hard Disk, Tapes, Optical Disks,etc.)

• RAM

• Switches and Router

• Monitors

• ...

PPOW regards the total power consumption of theElectrical equipment, for example, including:

• UPS (Uninterruptible Power Supply).

• PSU (Power Supply Unit).

• PDU (Power Distribution Unit).

• Cable (copper wires characterized by an electricalresistance)

• Lights

• Batteries

• ...

PCOOL refers cooling equipment including for exam-ple:

• Chiller. responsible for making the GAP amongthe external (outdoor) and internal (indoor) tem-peratures.

• FANs, regarding the Control Room Air Condi-tioning (CRAC) or to equipment used to discardthe heat in the external ambient.

• Pumps, responsible for moving the refrigerantsubstance (or water) inside the distribution pipes.

• Valves

• Unit of Control

• ...

The entire cooling system of a datacenter can bereferred also as HVAC (i.e, Heating, Ventilating,Air-Conditioning) or HVAC(R) (Heating, Ventilating,Air-Conditioning, and Refrigerating). Consequently,the total power consumption of a datacenter can bedefined as:

PTOT = PIT +PPOW +PCOOL (1)

Figure 2 shows the total amount of energy consump-tion of a datacenter. The percentages of the totalpower spent in a datacenter can be roughly distributedas follows:

PIT = 50%; PPOW = 20%; PCOOL = 30%. (2)

Figure 2: Typical Consumption of Energy inside a Datacen-ter

PIT and PPOW are strongly related to the transistorsperformances. In fact, currently, they have a physicallimits that it is not possible to be overcame. However,recent studies are trying to break such limits and theexpectation is that future innovation can bring to moreperformance equipment from the point of view of theenergy consumption. In this direction, a recent and in-teresting dissertation was conducted in The Optimist,the Pessimist, and the Global Race to Exascale in 20Megawatt (Tolentino and Cameron, 2012). Consid-ering the aforementioned assumption, particular con-sideration deserves PCOOL. We believe that PCOOL willhave a big role in the energy consumption studies inthe ICT field. In fact, at present PCOOL is the parame-ter easier to optimize and how it is described later inthis manuscript, cloud federation can help to achievethis goal.

In order to model a sustainable federated cloud en-vironment, we consider the datacenter of each cloudprovider as a black box which acts an ideal refrigera-tion machine also known as the Carnot Engine. The

performances of this ideal model is only affected bythe temperatures: the black box needs to be cooledas much as depending on the environment where itis placed. Moreover we consider that in such a blackbox there are both energy coming in, and heat that hasto be discarded in the external environment. Figure3 provides a graphic representation of our datacentermodeling. The energy coming from the left-side, inform of electricity is discarded to the environment inform of heat. This assumption seems to be simplistic,but in the bottom part of the Figure it is possible tounderstand why this simplification is valid for a data-center.

Considering the First Law of Thermodynamicabout the conservation of energy, we have the fol-lowing equation (see Eq. 3) that states any compo-nent/device connected to the Electric Grid transformthe energy from one form to other ones (Conservationof Energy).

Pin =W +Q (3)

Where Pin is the input power over the time (i.e., inhours), W is the energy spent for mechanical worksand Q is the energy release as Heat. In a datacen-ter, Pin is the electric power delivered through cop-per cables, W is the energy for producing movements(Compressor of a Chiller, FANs, Hard Disk with ro-tors, Optical Readers, etc.). Finally Q is the Heat pro-duced by components, lights, motors, compressors. Ina datacenter, if we assume the PIT and/or PPOW , thecontribution to W is negligible. The Compressor in achiller (PCOOL) catch a lot of energy, but its work isuseful for expelling Heat from inside the datacenterto the outside (environment).

Figure 3: The first law of thermodynamics; the law of con-servation of energy.

In the perspective of a sustainable federated cloudenvironment the basic idea is to relocate services andresource between cloud providers selecting the moreavailable sustainable datacenter minimizing the totalenergy consumption. Furthermore, it is also impor-tant looking at reducing the impact of incoming re-newable energy both for minimizing costs and maxi-mizing the use of renewable sources. In simple terms,

the objective is to identify a datacenter in which relo-cated services can consume as less as possible, hencein which there is a high availability of sustainable en-ergy that costs as less as possible.

The measurement of goodness of a datacenter isgiven by a number called Power Usage Effective-ness (PUE). It is expressed as the ratio from the totalamount of energy consumed as input respect to goodpart of energy used for IT computations. Values ofPUE equals to 1 correspond to an energy efficiencyof 100%.

PUE =PTOT

PIT(4)

The increasing of the PUE value, corresponds to agreater weight of either PPOW or PCOOL contributions(or both). Typical PUE values for a datacenter aregreater than 1 and corresponds to 2-2.5. A similarevaluation index is the DCiE (Datacenter Infrastruc-ture Efficiency). It is expressed in percentage, usingthe following formula:

DCiE = 100∗ PIT

PTOT(5)

The PUE is:

PUE = 100∗ 1DCiE

(6)

Considering a sustainable federated cloud envi-ronment the PUE of two different cloud datacenterswith the same equipment, but placed in two differentregions, may assume different values. The ambienttemperature of each region affects the resulting PUE(either DCiE) depending on the adopted cooling tech-niques.

4 Decision Algorithm

Considering both the scenario previously de-scribed in Section 2 and the considerations alreadypointed out in this work, in the following we are goingto introduce a simple algorithm designed to addressthe problem of establishing the best site on which agiven service has to be deployed for minimizing bothenergy consumption and costs, on the strength of theactual environmental data collected on each.

We assume a given workload has to be executed inour four-sites federated scenario in a given time: weknow how many resources will be needed to accom-plish that task and how many free computational slotswill be available on each site. For each one, we alsoassume to know the availability of the instantaneousamount of electrical power produced from the photo

Figure 4: The representation of a Datacenter: it is the real installation of the DC in Messina. The APC CUBE uses the HVACcooling

voltaic equipment, the amount of electrical power ab-sorbed from the HVAC(R) (or free cooling system)and that one used for achieving the computation (therewill be also other contribution that will be explainedin the following).

If we want to find a method for optimizing compu-tation respect to costs, we have to identify an analyticapproach able to put together all the parameters char-acterizing the scenario. Since energy providers applytheir fares on the actual amount of consumed kWh,our analytic model will take into account the energycontributions as mean value of the electrical powerover a time interval. In order to have a good snapshotof the energy production/absorption when the com-puting element placement have to be performed, inour case this interval corresponds to one hour.

Looking at the scenario depicted in Figure 1,and paying attention to the aforementioned consider-ations, we might assume the energy consumption ofeach site is represented from the Eq. 7

EGRID(T,G,s) =[EIT (s)+ECOOL(T )+EPOW −EPV (T,G)] (7)

where each energy term represents the pro-duced/absorbed medium power in a time interval ofone hour and is expressed in kWh. EGRID(T,G) is theamount of power grid energy needed from a given siteand is related to the other energy contribution termsappearing in the formula. According to this latter, itwill depend on the external temperature T where thesite operates, the G factor describing the availabilityof the renewable green energy source (e.g, sun radi-

ation, wind intensity, etc.) and the s parameter asso-ciated to the number of computational unit allocated(e.g. virtual machines) on that site.

In this specific work, we are assuming each dat-acenter can rely on a green energy source. For thesake of simplicity, we can restrict our considerationsto a scenario where each site takes advantage of justrenewable photovoltaic energy. As consequence, wecan associate the G factor to the sun radiation factorof the energy plant of a given site. According to theabove mentioned assumption, The remaining terms ofthe expression in order are: EIT , a non-constant fac-tor associated to the energy needed from a data cen-ter to perform the computation for a given servicethat depends on the number s of computational slotto be allocated; ECOOL(T ) an energy factor associatedto the HVAC(R) system (or free cooling system) thatdepends on the external temperature T characterizingthe area where the site is working (it is related to themeasured PUE for that site); EPOW is the constant fac-tor associated to the energy consumption of the powersupply equipment of the data center; finally the lastterm EPV (T,G) is a function of both external temper-ature T and sun radiation factor G, related to the meanenergy amount made available from the photo voltaicsystem of that site in the time interval of one hour.

If we consider that each site retrieves its electri-cal power from a different provider, we might esti-mate the costs needed in terms of electrical power toachieve a workload execution according to the appliedfares for kWh. With this assumption and starting fromthe Eq. 7, you can obtain a new expression 8 relatedto the energy expenses associated to each site func-

Site Temperature(T [°C])

Sun Radi-ation (G[MJ/m2])

Energy GridFare (c [$])

Photo VoltaicEnergyEPV [kWh]

Slots (s) PUE Costs[$]

Site 1 35 20 0.08 100 120 3.7 10Site 2 30 18 0.09 150 90 3.2 13Site 3 18 14 0.07 80 70 1.2 10Site 4 23 15 0.08 80 75 2.5 15

Table 1: Data retrieved from 4 different sites that will be given as input for the VIM Energy Manager.

tioning:

CGRID(T,G,s,c) = EGRID(T,G,s) · c (8)

In order to cut costs, the organization of our sce-nario relying on four different computational sites,can select the one for which the cost values associatedto expression 8 is minimum for a given set of parame-ters (i.e. temperature T , sun radiation G, needed com-putational slots s and fare f ). To achieve this goal,we can consider data related to those parameters iscollected from a sensor network on each site period-ically. Depending on these obtained values a table isbuilt considering the mean values of the retrieved pa-rameters.

Taking into account such an approach, this par-ticular module of the VIM operating on the datacen-ters, will take care of computing the costs for eachsite relying on the physical measurements collectedfrom the available sensors and stored in the table. Anexample is reported in Table 1. A simple algorithmimplemented within the Energy Manager will be ableto choose the most convenient site (in terms of energyconsumption) where allocate the computation associ-ated to a given service.

If two different sites are characterized from thesame cost values in a given time, the algorithm im-plemented within the Energy Manager will evaluatealso the amount of photo voltaic energy made avail-able from each site, finally preferring the one wherethis factor is the greatest. In some situations where en-ergy sustainability is preponderant on cost optimiza-tion, the same algorithm could be applied in a differ-ent way: the site(s) where the photo voltaic energyproduction is the greatest is first chosen and, in thecase of more sites producing the same photo voltaicenergy amount, from the retrieved set, the site wherethe overall costs are the lowest will be finally selectedfor allocating the services.

The algorithm could be implemented creating acomplementary module for the VIM (Energy Man-ager) that retrieves needed data from the sensor net-work available in each computational site, computingthe associated values of energy contributions (con-sidering the instantaneous electrical power values re-

trieved in that moment) and storing them on the tableuntil the next data refresh (after one hour). Althoughthis approach might be implemented in generic VIM,we designed it having in mind CLEVER (it is an IaaSVIM middleware whose details are provided in (Tusaet al., 2010)). This information will be stored in the(central) database through the database manager plu-gin (according to the CLEVER terminology). Whenthe VIM has to allocate a new set of s virtual machinesfor a given service, together with snapshot of physicalresource availability of each site (reported in Table 1),it will also invoke the Energy Manager to retrieve in-formation on the most convenient site to which deploythe allocation either in terms of cost minimization orenergy sustainability. Since each site can offer a lim-ited number of computational slots, if the virtual ma-chine number needed for a given service are greaterthan the maximum availability for a site, the load willbe split across more sites still considering the satis-faction of the same requirements (cost minimizationor energy sustainability).

Looking at Table 1, if we suppose executing a ser-vice that needs 100 virtual machines, in the first caseof cost optimization, the site selected by the EnergyManager will be one between Site 1 or Site 3 (as theyguarantee the lowest energy costs: respectively 10 $and 13 $). The final choice will lead to Site 1 since hisphoto voltaic energy availability (100 kWh) is greaterthan the one offered by Site 3 (80 kWh). The EPVavailability in this situation is preponderant: althoughthe temperature in Site 3 is 18 °C and allows free cool-ing as refrigeration methods, the CGRID costs comingfrom Eq. 8 are still more convenient on Site 1 wherethe “free cost” energy is offered by the photo voltaicequipment. Furthermore, Site 1 has the availabilityof s = 120 computational slots and is able to directlysatisfy the requested service demand. Otherwise thecomputation would be split among textitSite 1 andSite 3.

Still looking at the same table in the alternative sit-uation we have mentioned before (energy sustainabil-ity optimization), the first set of selected sites will beformed by Site 3 and Site 4 (either offering the sameamount of EPV = 80 kWh). This time, the final choice

will fall on Site 3 as it is able to offer lower grid en-ergy costs than Site 4 (10 $ against 15 $). Differentlyfrom the previous case, the available computationalslots of this site is lower than the needed ones. In thiscase, 70 of the requested VMs will be deployed onSite 3 while the remaining ones on Site 4.

As reported in the table, the HVAC(R) system ofeach data center is characterized in terms of efficiencythrough the PUE (the values in the table refers to amean value of the coefficient in the time interval ofone hour). High PUE values are associated to bet-ter refrigerator systems that allows to use less electri-cal power to push out heat from the data center. ThePUE values are tightly related the ECOOL(T ) valuescontributing in Eq. 7. The best efficiency in termsof energy spent for cooling is achieved on the site 3,where thanks to the low external environmental tem-perature, it is possible to use the free cooling tech-nique thus reaching a PUE = 1.2.

5 Related Works

In the section hereby, the early part analyzesworks falling into energy saving and green energytopics aimed at datacenter. While in latter part sev-eral works dealing with cloud and federation are re-ported. The section concludes the survey with GreenIT solutions aimed at clouds.

The work we highlight just below show as theproblematic we are trying to address is an hot topicindeed. Many works dealing with datacenters andsustainability exist in the scientific literature, howeverour contribution tries to give an answer in the are ofcooperating clouds.

The authors in (Moore et al., 2005) for optimiz-ing the energy usage in a datacenter introduced theconcept of temperature aware workload placement.They wrote an interesting dissertation that in-deep an-alyze the effects of cooling cost against the IT com-putations. A markable assessment also regards thereal COP (Coefficient of Performance) determination.The COP curve (with its relative formalization) theypresented in this work represents the real evaluationof a chilled-water CRAC unit at the HP Labs UtilityData Center.

The work in (Wang et al., 2011) highlights an in-novative cooling strategy that leverages thermal stor-age to cut the electricity bill for cooling. The authorsclaimed the system does not cause servers in a dat-acenter to overheat. They worked on ComputationalFluid Dynamics (CFD) to consider the realistic ther-mal dynamics in a datacenter with 1120 servers.

A Workload Distribution for Internet datacenters

is proposed in (Abbasi et al., 2010), where the serverprovisioning algorithm is aware of the temperaturedistribution in a DC. The authors try to find a waywhere the utilization constraints (in term of capac-ity and performance constrains) are satisfied and en-ergy consumption is minimized. The proposed for-mula contains the COP of a datacenter, under theirassessment.

Modeling a thermal behavior of a datacenter is achallenging work due to the high number of physi-cal parameters need to be considered. An interest-ing model along with a close-loop control system isdescribed in (Zhou and Wang, 2011). The authorsassessed a datacenter with many CRACs. The inlettemperature of many racks is investigated for accom-plishing the Partition in Zone of a datacenter for anefficient decentralized control.

5.1 Cloud and Federation

Technological solutions for cloud computing infras-tructures are increasing day by day, and the visionwhere companies use computational facilities accord-ing to a pay-per-use model similarly to other utilitieslike electricity, gas and water is becoming true.

However, the concept of cloud federation is quiterecent. Cloud federation refers to mesh of cloudsthat are interconnected based on open standards toprovide a universal decentralized computing environ-ment where everything is driven by constraints andagreements in a ubiquitous, multi-provider infrastruc-ture.

In this paragraph, we provide an overview of cur-rently existing solutions in the field, taking into ac-count initiatives born in academia, industry and ma-jor research projects. Most of the work in the fieldconcerns the study of architectural models able to ef-ficiently support the collaboration between differentcloud providers focusing on various aspects of thefederation.

The authors in (Buyya and Ranjan, 2010) pro-pose a decentralized technique using a structuredpeer-to-peer network supporting discovery, deploy-ment and output data collection of a (PaaS) mid-dleware (Aneka). The system is structured as aset of Aneka coordinator peers deployed in eachcloud (or in the extreme case on each node of thecloud) offering discovery and coordination mecha-nisms useful for federation. A central point is repre-sented by the Distributed Hash Table (DHT) overlaywhich is adapted in order to take into account multi-dimensional queries (e.g., “find all nodes with Linuxoperating system, two cores with 2.0 GHz and 2GBor RAM”). However it is not clear how this approach

deals with dynamism as discovery and matchmakingare carried out by a third part node (imposed by theDHT) which can be subject to failures.

In our previous work (Celesti et al., 2010)we describe an architectural solution for federationby means of a Cross-Cloud Federation Manager(CCFM), a software component in charge of exe-cuting the three main functionalities required for afederation. In particular, the component explicitlymanages: i) the discovery phase in which informa-tion about other clouds are received and sent, ii) thematch-making phase performing the best choice ofthe provider according to some utility measure andiii) the authentication phase creating a secure channelbetween the federated clouds. These concepts can beextended taking into account green policies applied infederated scenarios.

In (Buyya et al., 2010b), the authors propose amore articulated model for federation composed ofthree main components. A Cloud Coordinator man-ages a specific cloud and acts as interface for the ex-ternal clouds by exposing well-defined cloud opera-tions. The Cloud Exchange component implementsthe functionality of a registry by storing all necessaryinformation characterizing cloud providers togetherwith demands and offers for computational resources.Lastly, the Cloud Broker represents the touch pointfor users to enact the federation process; it interactswith the Cloud Exchange to find appropriate cloudproviders and with the Cloud Coordinator to definethe resource allocation satisfying the needed QoS.

A FP7 European Project focusing on cloud fed-eration is RESERVOIR. In Rochwerger et al (see(Rochwerger et al., 2011)), the authors define aRESERVOIR cloud as decentralized federation ofcollaborating sites. RESERVOIR introduces an ab-straction layer that allows developing a set of highlevel management components that are not tied toany specific environment. In RESERVOIR, severalsites can share physical infrastructure resources onwhich service applications can be executed. Eachsite is partitioned by a virtualization layer into Vir-tual Execution Environments (VEEs). These envi-ronments are fully isolated runtime modules that ab-stract away the physical characteristics of the resourceand enable sharing. The virtualized computational re-sources, alongside with the virtualization layer andall the management enablement components, are re-ferred to as the VEE Host. In RESERVOIR a serviceapplication is a set of software components whichwork to achieve a common goal in which each onecan be deployed in the same or in different sites. ARESERVOIR cloud federation is homogeneous (i.e.,each site has to run the same middleware) and decen-

tralized. It occurs at the IaaS level and needs a statica-priori configuration in each site.

The dissertation in (Kiani et al., 2012) describesthe large-scale context provisioning. The authors re-marked that the adoption of context-aware applica-tions and services has proved elusive so far, due tomulti-faceted challenges in cloud computing area. In-deed existing context aware systems are not ideallyplaced to meet the domain objectives, and facilitatetheir use in the emerging cloud computing scenar-ios. The authors identified what the challenges arein heterogeneous cloud contexts. In particular manyworks are addressed considering the simplified useof a central context management component e.g. acontext (cloud) broker. The use of a predominantfocus upon designing for static topologies of the in-teracting distributed components. Presumptions of asingle administrative domain or authority and con-text provisioning within a single administrative, ge-ographic or network domain. Finally they recognizeda lack of standardisation with respect to simple, flex-ible and extensible context models, for the exchangeof contextual and control information between hetero-geneous actors. We can admit, in the near future, thefederation among clouds still needs extra efforts to beconcretized.

5.2 Datacenter and Virtualization

In the recent past even more datacenters are look-ing to increase their flexibility, in particular exploit-ing the Virtualization Technology using different typeof Virtual Machine Managers (VMMs). The VMM isthe layer in which the hardware virtualization is ac-complished. It hides the physical characteristics of acomputing platform, instead showing another abstractcomputing platform. The software that controls thevirtualization used to be called a “control program”at its origins, but nowadays the term Hypervisor ispreferred. A Virtual Machine is a software programthat emulates a specific hardware system. Each vir-tual machine is like a “machine within the machine”and runs like just a real physical computer. This soft-ware layer emulates the operating system and allo-cates hardware resources such as the CPU, disk, net-work controllers, etc.

The main Hypervisors currently used to virtual-ize hardware resources are: Xen (www.xen.org,2012), KVM (www.linux kvm.org, 2012),VMware (www.vmware.com, 2012), VirtualBox(www.virtualbox.org, 2012), Microsoft Hyper-V (www.microsoft.com, 2012), Oracle VM(www.oracle.com, 2010), IBM POWER Hyper-visor (PR/SM) (publib.boulder.ibm.com, 2010),

Apple Parallel Server (www.apple.com, 2010), etc.For example one type of a VMM can be a physicalmachine with the Xen hypervisor para-virtualizer(en.wikipedia.org, 2012b) controlling it (in this casethe VM are Xen domains), whereas another type canbe a machine with the necessary software to hostKVM (full-virtualizer (en.wikipedia.org, 2012a)),and so on.

Authors in (Krishnan et al., 2011) have studied therun-time behavior of many Virtual Machines achiev-ing a good reference model useful for describing VMworkloads. In particular they introduced several mod-els for characterizing CPUs, RAMs, Disk and I/Owithin VMs under different working conditions (i.e.percentage of load, power consumption, etc.). Thiswork represents a good staring point for evaluatinggreen metrics along with VMs and Hypervisors.

The work in (Mukherjee et al., 2009) uses re-cent technological advances in data center virtualiza-tion and proposes cyber-physical, spatio-temporal ,thermal-aware job scheduling algorithms that mini-mize the energy consumption of the data center un-der certain performance constraints. Authors remarkthat savings are possible by being able to tempo-rally spread the workload, assign it to energy-efficientcomputing equipment. They propose a solution ableto reduce the heat recirculation and therefore the loadon the cooling systems. Their paper provides threecategories of thermal-aware energy-saving schedul-ing techniques that is: first-come first-serve withback-filling, first scheduling algorithm with thermal-aware placement, and offline genetic algorithm forscheduling to minimize thermal cross-interferences.The mathematical models they introduced are welldefined with a detailed description and good finalanalysis.

A complex analysis in energy-efficient and Cloudcomputing has been carried out from Buyya et al.in (Buyya et al., 2010a). The authors in this worktried to define an architectural framework and prin-ciples for energy-efficient Cloud computing. Theyclaimed their work is useful for investigating energy-aware resource provisioning and allocation algo-rithms that provision datacenter resources to clientapplications in a way that improves the energy ef-ficiency of the datacenter. Their solution shouldintroduce autonomic and energy-aware mechanismsthat self-manage changes in the state of resourceseffectively and efficiently to satisfy service obliga-tions and achieve energy efficiency. The investi-gation involves heterogeneous workloads of varioustypes of Cloud applications and develop algorithmsfor energy-efficient mixing and mapping of VMs tosuitable Cloud resources in addition to dynamic con-

solidation of VM resource partitions. However thework the authors presented does not provide an in-depth analytic model of what they assessed.

The work in (Moghaddam et al., 2011) falls inserver consolidation solution leveraging the deploy-ment of VMs inside a datacenter. This VMs con-solidation is aimed to carbon footprint minimization.In particular the authors presented a solution for aLow Carbon VPC (they called it LCVPC), and build acarbon footprint model of a reference Virtual PrivateCloud (VPC). They introduced a model proven ona simulation platform network. The approach lookssimilar to our assessment in considering clouds spreadaround the world and using more energy providers (awide portfolio with more sources), but authors do notprovide any solution on how to apply this model ondifferent clouds.

A similar work to our solution, but a very prelim-inary work has been presented in (Yamagiwa and Ue-hara, 2012) The authors try to construct a cloud plat-form which can operate in an area without the elec-tricity such as a disaster situation and/or in a remoteplace. In their solution there is a battery attached to aPC. The photovoltaic power generation is used to sup-ply this PC. In the paper is not clear what the meaningof cloud is for authors’ point of view. In reality theyused an Ubuntu Linux Machine with an Hypervisor.Several VMs are executed, in our view this testbed isnot a cloud at all.

6 Conclusions

Nowadays, a sensitive problem is finding the rightcombination between high performance datacenterand energy sustainability. In this work, considering ascenario of cloud federation, we proposed a method-ology for enabling sustainable cooperating clouds.Considering photovoltaic energy generation systems,our approach is based on an energy and temperature-driven strategies in which the computation workloadof a cloud is moved toward the most efficient sus-tainable federated cloud. According to such a strat-egy and considering a federated CLEVER-based sce-nario, we defined an algorithm for the management ofVM allocation according to energy and temperature-driven policies. In future works, we plan to consideralso heterogeneous cooperating clouds.

REFERENCES

Abbasi, Z., Varsamopoulos, G., and Gupta, S. K. S. (2010).Thermal aware server provisioning and workload dis-

tribution for internet data centers. In HPDC, pages130–141.

Buyya, R., Beloglazov, A., and Abawajy, J. H. (2010a).Energy-efficient management of data center resourcesfor cloud computing: A vision, architectural elements,and open challenges. CoRR, abs/1006.0308.

Buyya, R. and Ranjan, R. (2010). Special section: Fed-erated resource management in grid and cloud com-puting systems. Future Generation Comp. Syst.,26(8):1189–1191.

Buyya, R., Ranjan, R., and Calheiros, R. N. (2010b). Inter-cloud: Utility-oriented federation of cloud computingenvironments for scaling of application services. InProceedings of the 10th International Conference onAlgorithms and Architectures for Parallel Processing(ICA3PP 2010, pages 21–23. Springer.

Celesti, A., Tusa, F., Villari, M., and Puliafito, A. (2010).Three-phase cross-cloud federation model: The cloudsso authentication. In Proceedings of the 2010 Sec-ond International Conference on Advances in FutureInternet, AFIN ’10, pages 94–101, Washington, DC,USA. IEEE Computer Society.

en.wikipedia.org (2012a). Full Virtualization. Thevirtualization technique used to provide a cer-tain kind of virtual machine environment, thatis a complete simulation of the underlying hard-ware. http://en.wikipedia.org/wiki/Full-virtualization,Jan 2012.

en.wikipedia.org (2012b). Paravirtualization. The vir-tualization technique that presents a software in-terface to virtual machines that is similar butnot identical to that of the underlying hard-ware. http://en.wikipedia.org/wiki/Paravirtualization,Jan 2012.

Kiani, L., Anjum, A., Bessis, N., and Hill, R. (2012). Large-scale context provisioning. In 2012 Sixth Interna-tional Conference on Complex, Intelligent, and Soft-ware Intensive Systems, CISIS 2012.

Krishnan, B., Amur, H., Gavrilovska, A., and Schwan, K.(2011). Vm power metering: feasibility and chal-lenges. SIGMETRICS Perform. Eval. Rev., 38(3):56–60.

Moghaddam, F. F., Cheriet, M., and Nguyen, K. K. (2011).Low carbon virtual private clouds. IEEE 4th Interna-tional Conference on Cloud Computing. IEEE Com-puter Society.

Moore, J., Chase, J., Ranganathan, P., and Sharma, R.(2005). Making scheduling ”cool”: temperature-aware workload placement in data centers. InATEC’05: Proceedings of the USENIX Annual Tech-nical Conference 2005 on USENIX Annual TechnicalConference, page 5, Berkeley, CA, USA. USENIXAssociation.

Mukherjee, T., Banerjee, A., Varsamopoulos, G., Gupta,S. K. S., and Rungta, S. (2009). Spatio-temporalthermal-aware job scheduling to minimize energyconsumption in virtualized heterogeneous data cen-ters. Comput. Netw., 53(17):2888–2904.

publib.boulder.ibm.com (2010). The POWER Hyper-visor, the abstraction layer between the hardware

and firmware and the operating system instances forGX host channel adapter (HCA) implementations.http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/, August 2010.

Rochwerger, B., Breitgand, D., Epstein, A., Hadas, D., Loy,I., Nagin, K., Tordsson, J., Ragusa, C., Villari, M.,Clayman, S., Levy, E., Maraschini, A., Massonet, P.,Munoz, H., and Toffetti, G. (2011). Reservoir - whenone cloud is not enough. Computer, 44:44–51.

Tolentino, M. E. and Cameron, K. W. (2012). The optimist,the pessimist, and the global race to exascale in 20megawatts. IEEE Computer, 45(1):95–97.

Tusa, F., Paone, M., Villari, M., and Puliafito., A. (2010).CLEVER: A CLoud-Enabled Virtual EnviRonment.In 15th IEEE Symposium on Computers and Commu-nications, 2010. ISCC ’10. Riccione.

Wang, Y., Wang, X., and Zhang, Y. (2011). Leveragingthermal storage to cut the electricity bill for datacen-ter cooling. In Proceedings of the 4th Workshop onPower-Aware Computing and Systems, HotPower ’11,pages 8:1–8:5, New York, NY, USA. ACM.

www.apple.com (2010). The Mac: Realversatility through virtualization.http://www.apple.com/business/solutions/it/virtualization.html,August 2010.

www.linux kvm.org (2012). KVM (for Kernel-based Vir-tual Machine) is a full virtualization solution for Linuxon x86 hardware. http://www.linux-kvm.org, August2012.

www.microsoft.com (2012). Microsoft Hyper-VServer 2008 R2, the stand-alone product thatprovides a reliable and optimized virtualiza-tion solution. http://www.microsoft.com/hyper-v-server/en/us/default.aspx August 2012.

www.oracle.com (2010). Oracle VM, ther vir-tualization software that fully supportsboth Oracle and non-Oracle applications,and delivers more efficient performance.http://www.oracle.com/us/technologies/virtualization/oraclevm/,August 2010.

www.virtualbox.org (2012). x86 virtualiza-tion software package developed by Oracle.http://www.virtualbox.org/, Jan 2012.

www.vmware.com (2012). Virtualization - the essential cat-alyst for enabling the transition to secure cloud com-puting: http://www.vmware.com/it/, Jan 2012.

www.xen.org (2012). Xen Hypervisor - Leading opensource hypervisor for servers: http://www.xen.org/,Jan 2012.

Yamagiwa, M. and Uehara, M. (2012). A proposal for de-velopment of cloud platform using solar power gen-eration. In 2012 Sixth International Conference onComplex, Intelligent, and Software Intensive Systems,CISIS 2012.

Zhou, R. and Wang, Z. (2011). Modeling and control forcooling management of data centers with hot aislecontainment. In ASME 2011 International Mechan-ical Engineering Congress & Exposition.