thermal-aware task placement in data centers qinghui tang sandeep k s gupta georgios varsamopoulos...
Post on 29-Jan-2016
217 Views
Preview:
TRANSCRIPT
Thermal-aware Task Thermal-aware Task Placement in Data Placement in Data
CentersCenters
Qinghui TangQinghui Tang
Sandeep K S GuptaSandeep K S Gupta
Georgios Georgios VarsamopoulosVarsamopoulos
IMPACT LabIMPACT Labhttp://impact.asu.edu/http://impact.asu.edu/
Arizona State Arizona State UniversityUniversity
Growth Trends in data Growth Trends in data centerscenters
►Power density increasesPower density increases Circuit density Circuit density increases by a factor of 3 every 2 yearsincreases by a factor of 3 every 2 years Energy efficiency Energy efficiency increases by a factor of 2 every 2 yearsincreases by a factor of 2 every 2 years Effective power density Effective power density increases by a factor of 1.5 every increases by a factor of 1.5 every
2 years2 years[Keneth Brill: The Invisible Crisis in the Data Center][Keneth Brill: The Invisible Crisis in the Data Center]
►Maintenance/TCO risingMaintenance/TCO rising Data Center TCO doubles every three yearsData Center TCO doubles every three years By 2009, the three-year cost of electricity will exceed the By 2009, the three-year cost of electricity will exceed the
purchase cost of the serverpurchase cost of the server Virtualization/Consolidation is a 1-time/short term solutionVirtualization/Consolidation is a 1-time/short term solution
[Uptime Institute][Uptime Institute]
►Thermal management corresponds to an increasing Thermal management corresponds to an increasing portion of expensesportion of expenses Thermal-aware solutions becoming prominentThermal-aware solutions becoming prominent Increasing need for thermal awarenessIncreasing need for thermal awareness
Related Work (extended Related Work (extended domain)domain)
IC Case/chassis room
firmware
O/S
Application
(middleware)
Dynamic voltage scalingDynamic frequency scalingCircuitry redundancy
Fan speed scaling
CPU Load balancing
Thermal-aware VMThermal-aware
data centerjob scheduling
softwaredimension
physicaldimension
Thermal issues inThermal issues indense computer roomsdense computer rooms
(i.e. Data centers, Computer Clusters, Data warehouses)(i.e. Data centers, Computer Clusters, Data warehouses)
► Heat recirculationHeat recirculation Hot air from the equipment air Hot air from the equipment air
outletsoutlets is fed back to the is fed back to the equipment air equipment air inletsinlets
► Hot spotsHot spots Effect of Heat RecirculationEffect of Heat Recirculation Areas in the data center with Areas in the data center with
alarmingly high temperaturealarmingly high temperature
► ConsequenceConsequence Cooling has to be set very low to Cooling has to be set very low to
have have allall inlet inlet temperatures in temperatures in safe operating rangesafe operating range
Cou
rtesy
: Inte
l La
bs
Conceptual overview ofConceptual overview ofthermal-aware task placementthermal-aware task placement
Task placement determinestemperature distribution
Temperature distributiondetermines the equipmentpeak air inlet temperature
Peak air inlet temperaturedetermines upper bound toCRAC temperature setting
CRAC temperature settingdetermines it’s efficiency(Coefficient of Performance)
bottomline
There is a task placement that maximizes cooling efficiency. Find it!
The lower the peak inlet temperaturethe higher the CRAC efficiency
Coefficient of Performance(source: HP)
Prerequisites forPrerequisites forthermal managementthermal management
► Task profilingTask profiling CPU utilization, I/O activity etcCPU utilization, I/O activity etc
► Equipment power profilingEquipment power profiling CPU consumption, disk consumption CPU consumption, disk consumption
etcetc► Heat recirculation modelingHeat recirculation modeling► Task management technologiesTask management technologies
►Need for a comprehensive Need for a comprehensive research frameworkresearch framework
Thermal-awareThermal-awarejob schedulingjob scheduling
On-line job scheduling On-line job scheduling algorithm to minimize peak algorithm to minimize peak air inlet temperature, thus air inlet temperature, thus minimizing the cost of minimizing the cost of cooling.cooling.
Thermal ModelsThermal ModelsTo enable on-line real-time thermal-aware job To enable on-line real-time thermal-aware job schedulingscheduling► fast (analytical, non CFD based)fast (analytical, non CFD based)► non-evasive (machine-learning)non-evasive (machine-learning)
CharacterizationCharacterization
Characterize the power Characterize the power consumption of a given workload consumption of a given workload (CPU, memory, disk etc) on a given (CPU, memory, disk etc) on a given equipmentequipment
Thermal management research Thermal management research frameworkframework
Model the thermal impact of Model the thermal impact of multicore systemsmulticore systems
S en s o r D ataG ath er in g S erv ic e
D a ta C e n te rM o n ito rin g
P er fo rm an c eM o n ito r in g S erv ic e
No n -I n v a s iv eTh e rm a lEv a lu a t io n
F as t T h erm alEv alu atio n S erv ic e
T h erm al/P o w er &P er fo rm an c e C o r r e la tio n
S erv ic e
J o b S c h ed u lin gS erv ic e
C lu s te rM a n a g e m e n t
Po licyEn fo rce m e n t
T h erm al M an ag em en tP o lic y En fo rc em en t
S erv ic e
J o b Q u eu esR es o u rc eQ u eu es
T h erm alC o n tr o l P o lic ies
C o o lin g C o n tro lS erv ic e
Air - f lo w C o n tro lS erv ic e
Fa cilityM a n a g e m e n t
R e s o u rce &S e rv e rM a n a g e m e n t
O S -L ev el S erv ic esP er fo rm an c e
M o n ito r in g
T herm al M anagem ent Infras truc ture& S ervic es fo r D ata C enters
http://impact.asu.edu/
Sandeep GuptaQinghui Tang
Tridib MukherjeeMichael Jonas
Georgios Varsamopoulos
Task ProfilingTask Profilingmeasurements at ASU HPC Data Center measurements at ASU HPC Data Center (one (one
chassis)chassis)
Power Model and ProfilingPower Model and Profiling
► Power Power Consumption Consumption is mainly is mainly affected by affected by the CPU the CPU utilizationutilization
► Power Power consumption consumption is linear to is linear to the CPU the CPU utilizationutilizationPP = a = a UU + + bb
Linear Thermal ModelLinear Thermal Model
► Heat Recirculation Heat Recirculation CoefficientsCoefficients AnalyticalAnalytical Matrix-basedMatrix-based
► Properties of modelProperties of model Granularity at air Granularity at air
inlets inlets (discrete/simplified)(discrete/simplified)
Assumes steadiness Assumes steadiness of air flowof air flow
= + ×
inlettemperatures
supplied airtemperatures
heat distribution
powervector
Tin Tsup D P
N 1 A C
R ecircu la tio n
T su p T in T o u t T A C in
N 2 N 3
1 2 1 3
2 13 1
1 1
Benefit: fast thermal Benefit: fast thermal evaluationevaluation
Give workload Run CFD simulation (days)Extract
temperatures
Give workload Compute vector (seconds)
+×
TinTsupD P
Yieldstemperatur
es
Courtesy: Flometrics
Thermal-awareThermal-awareTask Placement ProblemTask Placement Problem
Given an incoming task, find a task partitioning Given an incoming task, find a task partitioning and placement of subtasks to minimize the and placement of subtasks to minimize the (increase of) peak inlet temperature(increase of) peak inlet temperature
= + ×
inlettemperatures
supplied airtemperatures
heat distribution utilization
vector
Tin Tsup D U
(a+
)
bbbbbbb
XInt AlgorithmXInt AlgorithmApproximation solutionApproximation solution(genetic algorithm)(genetic algorithm)► Take a feasible Take a feasible
solution and perform solution and perform mutations until certain mutations until certain number of iterationsnumber of iterations
PP = a = a UU + + bb
InletTemperature
Contrasted scheduling Contrasted scheduling approachesapproaches
► Uniform Outlet Profile (UOP)Uniform Outlet Profile (UOP) Assigning tasks in a way that tries to Assigning tasks in a way that tries to
achieve uniform outlet temperature achieve uniform outlet temperature distributiondistribution
Assigning more task to nodes with low Assigning more task to nodes with low inlet temperature (water filling process)inlet temperature (water filling process)
► Minimum computing energyMinimum computing energy Assigning tasks in a way that keeps the Assigning tasks in a way that keeps the
number of active (power-on) chassis as number of active (power-on) chassis as few as possiblefew as possible
Server with coolest inlet temperature Server with coolest inlet temperature firstfirst
► Uniform Task (UT)Uniform Task (UT) Assigning all chassis the same amount Assigning all chassis the same amount
of tasks (power consumptions)of tasks (power consumptions) All nodes experience the same power All nodes experience the same power
consumption and temperature riseconsumption and temperature rise
OutletTemperature
Simulated EnvironmentSimulated Environment► Used Flometrics Flovent► Simulated a small scale data
center► physical dimensions
9.6m 8.4m 3.6m► two rows of industry
standard 42U racks arranged► CRAC supply at 8 m3/s► There are 10 racks
each rack is equipped with 5 chassis
► 1000 processors in data center. 232KWatts at full utilization
Performance ResultsPerformance Results► Xint outperforms other algorithmsXint outperforms other algorithms► Data Centers almost never run at 100%Data Centers almost never run at 100%
Plenty of room for benefits!Plenty of room for benefits!
Performance ResultsPerformance Results► Xint outperforms other algorithmsXint outperforms other algorithms► Data Centers almost never run at 100%Data Centers almost never run at 100%
Plenty of room for benefits!Plenty of room for benefits!
Power Vector DistributionPower Vector Distribution
key
Xint contradicts “rule of thumb” placement at bottom
Supply Heat Index (SHI)Supply Heat Index (SHI)
►Supply Heat Index Supply Heat Index Metric developed Metric developed
by HP Labsby HP Labs quantifies the quantifies the
overall heat overall heat recirculation of recirculation of data centerdata center
►Xint consistently Xint consistently has the lowest SHIhas the lowest SHI
ConclusionsConclusions
►Thermal-aware task placement can Thermal-aware task placement can significantly reduce heat recirculationsignificantly reduce heat recirculation XInt performance thrives at around 50% XInt performance thrives at around 50%
CPU utilizationCPU utilization►Not much can be done at 100% utilizationNot much can be done at 100% utilization
Cooling savings can exceed 30%Cooling savings can exceed 30%(in comparison to other schemes)(in comparison to other schemes)
►Cost of operation reduces by 15%Cost of operation reduces by 15%(if initially 1:1 ratio of computing-2-cooling)(if initially 1:1 ratio of computing-2-cooling)
Related Work in ProgressRelated Work in Progress
► Waiving simplifying assumptionsWaiving simplifying assumptions Equipment heterogeneity Equipment heterogeneity [INFOCOM 2008][INFOCOM 2008]
Stochastic task arrivalStochastic task arrival
► Thermal maps thru machine learningThermal maps thru machine learning Automated, non-invasive, cost-effective Automated, non-invasive, cost-effective [GreenCom [GreenCom
2007]2007]
► ImplementationsImplementations Thermal-aware Thermal-aware Moab Moab schedulerscheduler Thermal-aware Thermal-aware SLURMSLURM SiCortexSiCortex product thermal management product thermal management
Algorithm AssumptionsAlgorithm Assumptions
► HPC model in mindHPC model in mind Long-running jobs (finish time is the same Long-running jobs (finish time is the same ——
infinity)infinity)► One-time arrival (starting time is the same)One-time arrival (starting time is the same)► Utilization homogeneityUtilization homogeneity
(same utilization throughout task’s length)(same utilization throughout task’s length)► Non preemptive/movable tasksNon preemptive/movable tasks► Data Center equipment homogeneityData Center equipment homogeneity
power consumptionpower consumption computational capabilitycomputational capability
► Cooling is self-controlledCooling is self-controlled
Thank YouThank You
►Questions?Questions?►Comments?Comments?►Suggestions?Suggestions?
http://impact.asu.edu/
Additional SlidesAdditional Slides
Functional model of Functional model of schedulingscheduling
► Tasks arrive at the data centerTasks arrive at the data center► Scheduler figures out the best placementScheduler figures out the best placement
Placement that has minimal impact on peak Placement that has minimal impact on peak inlet temperaturesinlet temperatures
► Assigns task accordinglyAssigns task accordingly
SchedulerTask
TaskTasks
Architectural ViewArchitectural View
Scheduler(Moab, SLURM)
dispatch
MachineLearning
create/update
provideMonitoringProcesses
ThermalModel
report
control
A simple thermal modelA simple thermal model
► Basic Idea:Basic Idea: We don’t need an We don’t need an
extensive CFD modelextensive CFD model We only need to know the We only need to know the
effect of recirculation at effect of recirculation at specific pointsspecific points
► Express recirculation as Express recirculation as “coefficients”“coefficients”
Cou
rtesy
: Inte
l La
bs
N1
N2
N3
N4
N5
Recirculation coefficients:Recirculation coefficients:a fast thermal modela fast thermal model
►Reduce/Simplify Reduce/Simplify the “thermal the “thermal map” concept to map” concept to points of interest: points of interest: equipment air equipment air inletsinlets
►Can be computed Can be computed from CFD from CFD models/simulatiomodels/simulationsns
Matrix Aaij: portion of heatexhausted from node ithat directly goes to node j
A
recirculation coefficients
Opportunities & ChallengesOpportunities & Challenges► Data centers don’t run at fulll Data centers don’t run at fulll
unitilizationunitilization Can choose among multiple CPUs Can choose among multiple CPUs
to allocate a jobto allocate a job Different thermal impact per CPUDifferent thermal impact per CPU
► Need for fast thermal Need for fast thermal evaluationevaluation
► Temporal and spatial Temporal and spatial Heterogeneity of Data CentersHeterogeneity of Data Centers
In equipmentIn equipment In workloadIn workload
Thermal issuesThermal issues► Heat recirculationHeat recirculation
Increases as equipment density exceeds Increases as equipment density exceeds cooling capacity as plannedcooling capacity as planned
► Hot spotsHot spots Effect of Heat RecirculationEffect of Heat Recirculation
► Impact:Impact:Cooling has to be set low enoughCooling has to be set low enoughto have to have allall inlet inlet temperatures intemperatures insafe operating rangesafe operating range
Data Center Thermal Data Center Thermal ManagementManagement
Increasing need for thermal awarenessIncreasing need for thermal awareness► Power density increasesPower density increases
Circuit density Circuit density increases by a factor of 3 increases by a factor of 3 every 2 yearsevery 2 years
Energy efficiency Energy efficiency increases by a factor of 2 increases by a factor of 2 every 2 yearsevery 2 years
Effective power density Effective power density increases by a increases by a factor of 1.5 every 2 yearsfactor of 1.5 every 2 years
[Keneth Brill: The Invisible Crisis in the Data Center][Keneth Brill: The Invisible Crisis in the Data Center]► Maintenance/TCO risingMaintenance/TCO rising
Data Center TCO doubles every three yearsData Center TCO doubles every three years By 2009, the three-year cost of electricity By 2009, the three-year cost of electricity
will exceed the purchase cost of the serverwill exceed the purchase cost of the server Virtualization/Consolidation is a 1-time/short Virtualization/Consolidation is a 1-time/short
term solutionterm solution► Thermal management corresponds to an Thermal management corresponds to an
increasing portion of expensesincreasing portion of expenses Thermal-aware solutions becoming Thermal-aware solutions becoming
prominentprominent
IC Case/chassis room
firmware
O/S
Application
(middleware)
Dynamic voltage scalingDynamic frequency scalingCircuitry redundancy
Fan speed scaling
CPU Load balancing
Thermal-aware VM
Data centerjob scheduling
softwaredimension
physicaldimension
Thermal-aware solutionsat various levels
A dynamic thermal-A dynamic thermal-aware control platform aware control platform is necessary for online is necessary for online thermal evaluationthermal evaluation
without thermal-awaremanagement
With thermal-awaremanagement
computation
cooling
$1M
$10M
$100M
year
Scheduling Impacts Cooling Scheduling Impacts Cooling SettingSetting
Inlet temperaturedistributionwithout Cooling
25C
25C
Inlet temperaturedistributionwith Cooling
Scheduling 1
Scheduling 2
Different demands for cooling capacity
Results(1)Results(1)►Recirculation CoefficientsRecirculation Coefficients Consistent with datacenter observationsConsistent with datacenter observations Large values are observed along diagonalLarge values are observed along diagonal Strong recirculation among neighboring servers, or Strong recirculation among neighboring servers, or
between bottom servers and top serversbetween bottom servers and top servers
1
2
3
4
5
6
7
8
9
10
top related