IBM Systems and Technology Group Lab Services
Keeping our clients in the race. © 2006 IBM Corporation© 2006 IBM Corporation
Energy Efficient Data Centers
Steven AhladasIBM [email protected]
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation2 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
All trends going the same way ….IT Servers were an asset, tend to be a commodityEnergy was a commodity for servers, tend to be a cost
But…..Data Centers energy cost now showsAppear as a critical limitation to server extension in the data center
Heat increase can be a growth killer by stopping introduction of new servers (DataCenter rooms limitations)
Energy economics around IT servers must be rebalanced with new energy-conscious criteria to stop cost increase
– 1) Stop the trend– 2) Reduce power consumption
$0
$20
$40
$60
$80
$100
$120
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Installed Base(M Units)
Spending(US$B)
New Server SpendingPower and Cooling
0
5
10
15
20
25
30
35
40
45
50
IDC Presentation, The Impact of Power and Cooling on Data Center Infrastructure, Doc #201722, May 2006
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation3 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Energy/Performance Metrics
Perf(capacity for storage) Power Perf/Space Perf/Watt0
10
20
30
40
50
60
70
80
Ann
ual C
hang
e(%
)
High End z xSeries 1U p550 High End Storage
Server/Storage Performance/Energy Trends
Server performance increasing much faster than server power
Source: Roger Schmidt
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation4 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Comparison of typical server utilization rates
UNIX x86Mainframe
Used
Wasted
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation5 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
System 1
APP
1
APP
2
10% busy1KW
System 2
AP
P 3
APP
4
10% busy1KW
Advanced Virtualization
APP
5
APP
6
70% busy4KW
APP
1
APP
2
AP
P 3
APP
4
Energy Management Policy ExampleServer Consolidation Conserves Energy
System 8
APP
7
APP
8
10% busy1KW
APP
7
APP
8…
Server consolidation exploiting virtualization is the most effective tool in reducing energy costs
Total Power 8KW Total Power 4KW
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation6 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
• Virtualization was first introduced by IBM in the 1960s to allow the partitioning of large mainframe environments. System z virtualization remains the gold standard today.
• IBM SAN Volume Controller (SVC). for storage virtualization increases storage utilization rates by 30% while reducing the growth rate of storage capacity by 20%.
• VMware, Microsoft® Virtual Server and Xen offerings are available for IBM System x™ and IBM BladeCenter®
Cool Blue: Virtualization on IBM Systems and Storage
Annual Power Cost per Logical Server0 $100 $200 $300 $400 $500 $600 $700 $800 $900 $1,000
$920.70
$353.14
$550.06
$207.57
$127.23
$73.06
$32.80
Dell PowerEdge 1950
Dell PowerEdge 1950 virtualized
Dell PowerEdge 1955
Dell PowerEdge 1955 virtualized
IBM System z™BC S07 Small
IBM System z BC S07 Expanded
IBM System z EC S18
Source: “Avoiding the $25 Million Server Data Center Power, Cooling, and Growth” report by Sine Nomine Associates, 4/6/07
“This shift has eliminated one floor of servers, cut power and cooling costs by 80 percent, and reduced the administrative staff (and costs) by 10 administrators. No other platform has clearly demonstrated the vertical scaling (scale-up) savings as the mainframe.”
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation7 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Cool Blue: Storage virtualization
Pool and share I/O Resources for:– Improved utilization
– Reduced energy needsCombined utilization
50%
EMCIBM HPIBM IBMHitachi
SAN Volume Controller Combined utilization
90%
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation8 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Cool Blue: Deploy efficient IBM System Storage
10 year TCO example using blended tape and disk best practices—Deploy more power efficient storage—Utilize storage more efficiently
250TB of storage25% growthover 10 years
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation9 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Reduce Consumption Through ProvisioningDe-provision Standby Servers
Standby Servers
125 Servers
Active Servers
875 Servers
Provisioned
De-provisioned
Power SavingsIn Standby Mode
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation10 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Energy Efficient Data Centers
Data Center Power Usage Effectiveness
0
2
4
6
8
10
12
1.53 1.81 2.10 2.38 MoreBuilding / IT Eqt Power
Num
ber o
f Dat
a C
ente
rs
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation11 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Computer Loads38%
UPS Losses6%
Lighting2%
HVAC54%
Total Power = 580 kW
UPS Losses13%
HVAC - Air Movement
9%HVAC -
Chilled Water Plant14%
Lighting1%
Computer Loads63%
Total Power = 1700 kW
Data Center A: 54 % HVAC ; Data Center B: 23 % HVACData Center A could possibly save ~ 180 kW “ Those that did many things well still had room to improve…”
DATA CENTER ENERGY EFFICIENCIES
From LBNL DC energy study, W. Tschudi
DATA CENTER A DATA CENTER B
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation12 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
TYPICAL RAISED FLOOR DATA CENTERS
GOAL: provide specified inlet temperatures to every IT Rack as efficiently as possible
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation13 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
cold aislehot aisle z=0.5 feetz=1.5 feetz=2.5 feetz=3.5 feetz=4.5 feetz=5.5 feet
Hot spot at long aisle
z=6.5 feetz=7.5 feetz=8.5 feet
3D Temperature distributions of IBMsupercomputer
13.01oC
54.47oC
33.74oC
yz
x
Measured 3D temperature maps of a datacenter
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation14 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
0 1 2 3 4 5 6 7 8 9
15
20
25
30
35
40
45
inle
t ai
r te
mpe
ratu
re T
inle
t [o C]
height [feet]
1 2 3 4 5 6 7 8 9 10 11 12
upper specs.
Verticle Temperature Gradients
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation15 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
COOLING EFFICIENCY
#1: INTERMIXING between hot and cold air increases locally Inlet Air Temperatures( excessive low CRAC discharge temperatures; affects Cooling Production Efficiency)#2: RECIRCULATION causes small differences in Return and CRAC Discharge Temperatures( CRAC units are not fully utilized; affects Cooling Delivery Efficiency)
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation16 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
- in many places perf. tiles do not supportfor the required cooling capacity
- get high throughput tiles (50 % open)- remove dampers to enhance air flow
(300 cfm ~ 2 kW)
PERFORATED TILES
56% open,with damper
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0 500 1000 1500 2000 2500 3000
Airflow, CFM
Stat
ic P
ress
ure,
in H
2O
Tate GrateAire 56%
Tate GrateAire 56% w/100% Open Damper
56% open,no damper
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation17 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
UNDER-FLOOR BLOCKAGE
- avoid any blockage in the critical path to perforated tiles- put cabling etc. under hot aisles- possibly deploy under-floor boundaries
- limit underfloor airflow to the boundaries of the zone to be cooled- escaping air results in lower underfloor air pressure and room airflow- use under-floor barriers to limit the airflow to the needed places
cables water pipesExamples for blockage
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation18 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
- block cable cut-outs using the methods shown below- add perforated tiles & use higher throughput tiles- try to have good sealing between the tiles
MatsBrushes Bags Pillows
LEAKS AND CABLE CUTOUTS
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation19 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
LEAKAGE in Sample Raised Floors#
CRACs
total flow/cooling
capacity1
#
perf. tiles
total perf. tile
flow/cooling capacity1
total ceiling tile
flow/cooling capacity1
Not-targeted
flow/cooling2
Total
HW / Room Power
Margin
29500 cfm
200 kW
58 tons
11000 cfm
73 kW
21 tons
#3 17 84000 cfm
560 kW
162 tons
24 6000 cfm
40 kW
12 tons
NA 78000 cfm
520 kW
151 tons
(93 %)
175/230 kW
51/68 tons
38000 cfm
250 kW
72 tons
(57 %)
330 kW
96 tons
(143 %)
46.5 kcfm
310 kW
90 tons
31700 cfm
211 kW
61 tons
(74 %)
90 kW
26 tons
(25 %)
300/360 kW
87/104 tons
110/140 kW
32/41 tons
#4 35 194 kcfm
1294 kW
375 tons
185 NA 137.5 kcfm
918 kW
266 tons
(71 %)
585/730 kW
170/212 tons
145 kW
42 tons
(103 %)
564 kW
164 tons
(77%)
1900 cfm
12 kW
3 tons
2000 cfm
13 kW
4 tons
#1 11 67500 cfm
450 kW
130 tons
118
#2 7 42700 cfm
285 kW
82 tons
43
1.) @ 60F (16oC) CRAC discharge temperature2.) Cable cutouts, other losses
H.Hamman 2/2007
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation20 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Understanding Where Costs Hide is Critical Rear Door Heat eXchanger can remove Rear Door Heat eXchanger can remove over 50% of a rackover 50% of a rack’’s heat outputs heat output
– No new fans or electricity needed.
– Attaches to back of rack (adds 5”) No rearrangement of datacenter
– Cost effective; 1KW cooling = $286
The Cool Blue Heat eXchanger adds The Cool Blue Heat eXchanger adds cooling capacity at ~1/4 the cost of cooling capacity at ~1/4 the cost of traditional methodstraditional methods
Rear Door Heat Exchange
Front Cold
BackHot
Cable Opening
Subfloor
Underfloor Chilled Air
Air flow
Perf tile Tile floor
water lines
Rear Door Heat Xchanger
IBM Enterprise Rack
Front Cold
BackHot
Cable Opening
Subfloor Underfloor Chilled Air
Air flow
Perf tile Tile floor
Normal Operation of Server
Improved Operation of Server
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation21 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Motivation for Liquid Cooling- Increase in heat removal performance:
Superior thermal properties of liquids compared to air
- Design flexibility: Sensible heat transport to locations with available space
- Centralized secondary heat exchanger
- Efficient water-water heat exchanger
Limited heat transport due to fin efficiencyLong distance transport possible
Thermal conductivity[W/(m*K)]
Volumetricheat capacity[kJ/(m3*K)]
Air 0.0245 1.27H2O 0.6 4176
Transport
Primary
Secondary
Disadvantage: Increased complexity
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation22 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Data Center Power Distribution Systems & Efficiency
Feeder Transformer
UPS System Raised FloorTransformer
Distribution(all combined)
Server Bulk PS
VRM Load
-
480V Servers
11.2KV(AC) to 480VAC
480VAC to 400VDC to 480VAC
NA - 480VAC to 400VDC to 12VDC
12VDC to 1VDC
1VDC to heat
480V Watts 1.70 1.67 1.50 1.50 1.47 1.25 1400V DC Servers
11.2KV(AC) to 480VAC
480VAC to 400VDC
NA - 400VDC to 12VDC
12VDC to 1VDC
1VDC to heat
Std Raised Floor
11.2KV(AC) to 480VAC
480VAC to 400VDC to 480VAC
480VAC to 208VAC - 208VAC to
400VDC to 12VDC
12VDC to 1VDC
1VDC to heat
Std Watts 1.79 1.75 1.58 1.52 1.47 1.25 1
400V DC Watts 1.57 1.54 1.46 1.46 1.40 1.25 1
5%
12%
Capital/Cost of Acquisition Savings
One WattOf Chip Power
Requires this much power at the facility level
Power Conversion & Distribution Losses
IBM Systems and Technology Group Lab Services
© 2006 IBM Corporation23 Keeping our clients in the race.
Trends and Best Practices
© 2006 IBM Corporation
Thank You!Questions?
Steve Ahladas,Data Center Services:[email protected]