cooling electronic products from hand-held devices to supercomputers
TRANSCRIPT
May 16, 2003, Rohsenow Symosium
Richard C.Richard C. ChuChu -- IBMIBM
Cambridge, MA
Thermal Management RoadmapThermal Management RoadmapCooling Electronic Products fromCooling Electronic Products from
HandHand--HeldDevices HeldDevices to Supercomputers to Supercomputers
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal Management TechnicalThermal Management TechnicalWorking Group (TWG)Working Group (TWG)
Richard C. Chu, IBM (Chair)
Avi Bar-Cohen, U. of Maryland
Darvin Edwards, TI
Magnus Herrlin, Telcordia
Donald Price, Raytheon
Roger Schmidt, IBM Lian-Tuu Yeh, Boeing
Bahgat Sammakia, SUNY-Binghamton
Larry Mok, IBM
Suresh Garimella, Purdue U.
Gregory M. Chrysler, Intel
Jogenda Joshi (Co-chair)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal Management Roadmap Thermal Management Roadmap -- ScopeScope
• Overview• Review of thermal design requirements• Review of thermal design requirement matrix• Review of current product cooling designs• Review of cooling technologies• Review of advanced cooling technology development activities
Introduction-
Summary and Conclusions-Future Cooling Technologies & Strategy-
Outline of Thermal Technology Needs-• High performance product sector• Cost performance product sector• Telecommunications product sector• Hand held product sector• Harsh environment (automotive) product sector• Harsh environment (military) product sector
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal management will play a pivotal role in the coming decade for all types of electronic products. Increased heat fluxes at all levels of packaging from chip to system to facility pose a major cooling challenge. To meet the challenge significant cooling technology enhancements will be needed in each of the following areas:
l Thermal interfacesl Heat spreadingl Air coolingl Indirect and direct water coolingl Immersion coolingl Refrigeration coolingl Thermoelectric cooling l Equipment-facility interface
OverviewOverview
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal Design RequirementsThermal Design Requirements(Traditional)(Traditional)
Design for Performance
Design for Reliability
Design for Serviceability
Design for Extensibility
Design for Minimal Cost
Design for Minimal Impact on User
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal Design Requirements (New)Thermal Design Requirements (New)
Design for improved coolability at the package level via optimized internal thermal conduction paths.
Design for direct air cooling at the product level viaenhanced convection process over the packages.
Design for special cooling needs at the module levelvia spot cooling devices attached to the packages.
Design for low temperature applications -subambient to cryogenic.
Design for low cost via Computer Aided ThermalEngineering (CATE) and improved manufacturability.
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Thermal Management Thermal Management Requirements MatrixRequirements Matrix
PC/Handheld/Wearable
Air C
oolin
gCo
nduc
tion
(Indir
ect L
iquid)
Dire
ct Liq
uid
(see l
ist)
New
Cooli
ng
Low
Tempe
ratur
e
Heat
Pipe
Therm
o-E
lectric
s
Mid-Size Computers
Storage Subsystems
Large Scale Computers
Super Computers
Workstations
Tech
nolog
y Nee
ded
- State-of-Art
- Likely
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Microprocessor PowerMicroprocessor PowerDissipation TrendsDissipation Trends
Year of Announcement1996 1998 2000 2002 2004 20060
10
20
30
40
50
60
Freeway
GP (1.3 GHz)Pentium 4
Pentium 3
McKinley zSeries
i & pSeries
SunUltraSPARC
Intel
Ch
ip H
eat
Flu
x (W
atts
/sq
cm
)
Freeway
Power4(1.3 GHz)Pentium 4
Pentium 3
McKinley zSeries
i & pSeries
SunUltraSPARC
ITRS 2002 Roadmap "First Introduction"
International Technology Roadmap for Semiconductors2002 Update
7% CGR
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Module Heat Flux ExplosionModule Heat Flux Explosion
2010200019901980197019601950
Year of Announcement
15
10
5
0
Mod
ule
Hea
t Flu
x (W
atts
/cm
)2
Bipolar IBM ES9000
VacuumTube IBM 360
IBM 370 IBM 3033
Honeywell DPS-88
Hitachi S-810
NEC LCMIBM RY3
IBM RY4
IBM RY6
IBM RY5
CMOS
Fujitsu M380
IBM 3081 TCMIBM 4381
CDC Cyber 205
IBM 3090 TCM
Fujitsu M780
NTT
IBM 3090S TCM
Fujitsu VP 2000
Merced
IBM GP
IBM RY7
Pulsar
Pentium II (DSIP)
Steam Iron
5 W/cm2
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
The Uptime Institute, 2000.
Servers & DASD (tall racks)
Tapes
1992 20082006200420022000199819961994 2010
Year of Product Announcement
10,000
60
2,000
4,000
6,0008,000
80100
200
400
600800
1,000
100,000
20,000
40,000
60,00080,000
8001,000
2,000
4,000
6,0008,000
10,000Communication Equipment (f
rames)
Workstations
Hea
t L
oad
per
Mac
hin
e F
loo
r A
rea
-wat
ts /
m2
Hea
t L
oad
per
Mac
hin
e F
loo
r A
rea
-wat
ts /f
t2
System Heat Density TrendsSystem Heat Density Trends
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Year
Chi
p H
eat F
lux
( W
/cm
)2
25
50
75
100
125
150
175
200
225
250
275
Air Cooling
or Immersion
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Cooling (TBD)Future
T = 85 Cchipo
T = 25 Cinlet
o
High PerformanceConduction Cooling
Indirect Water
Cooling
Projected Chip Heat Flux andProjected Chip Heat Flux andCooling Technology LimitsCooling Technology Limits
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Current Product Cooling Designs
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Inlet / Outlet zoom
Outlet
Intake
FAN
HDD
BAY
IBM IBM Thinkpad Thinkpad T23T23Air Flow LayoutAir Flow Layout
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Top
Bottom
CPU
IBM IBM Thinkpad Thinkpad T24T24Push FanPush Fan
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Reservoir Tank
Flexible Tube
Hitachi Water Cooling LaptopHitachi Water Cooling Laptop(Prototype Model)(Prototype Model)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM IBM pSeries pSeries 690 690 (with extra rack)(with extra rack)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Processors 8-32 SMP'sClock speed 64-bit 1.1 to 1.3 GHzMain memory 8 GB to 256 GBOS images 1-16Memory bandwidth 205 GB/secI/O bandwidth 16 GB/secInternal storage 4.66 TB - 8 drawers (with extra rack)PCI adaptors up to 160PCI hot-plug slots yesPCI bus recovery yesPCI bus deallocate yesBattery backup yes
Bulk Power Subsystem (8U)
Primary I/O Drawer (4U)
24" System Rack, 42U
Central Electronics Complex(CEC) (17U)
Optional I/O Drawer (4U)
Optional I/O Drawer 4U)
Media Drawer (required) (1U)
Up to 4 Optional internalbatteries (2U each)
IBM IBM pSeries pSeries 690 690 (continued)(continued)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Mayfly Global Stiffener
Mayfly Board
Power Distribution Board
Front side Stiffeners
8 Way MCM Heatsink/Module Asm.
L3 Heatsink / Module Asm.4 per MCM
MCM Key Guide 0
1
2
3
0
1
3
2
pSeries 681 MCM-L3
Interposer
Local stiffener
Load posts
Laminated spring plate, asm.
Fixed stop actuation screw, drive screw until screw head meets spring plate.
Mayfly board
Insulator
MCM heatsink / module assembly
Local stiffener, backside, 4 position
Socket and L3 Module
Opposed, laminated spring plates.
Fixed stop, actuation screw - Drive screw until screw head meets springplate.
Load Post
Load Column
Heatsink
Insulator
Mayfly board
MCM power ~ 624 wattsProcessor - 2 cores - 156 watts maxProcessor 415 sq mmTj ~ 105C
IBM IBM pSeries pSeries 690 690 (continued)(continued)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
4 Chip MCMw/ SiC Thermal Spreaders
Copper Cap (Lid)
Aluminum Air Cooled Heat Sink - 68 fins - fin width = 0.6 mm - fin spacing = 0.73 mm
Cap / Heat Sink thermal interface - Powerstrate 60 - Al foil with phase change material on both sides
Shroud with gasket
IBM IBM pSeries pSeries 690 MCM690 MCM
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM IBM zSerieszSeries 900 Server 900 Server (with extra rack)(with extra rack)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Power
Modular Cooling Units(MRUs)
Front Side Back Side
Refrigerant(R134A)
Lines
I/O Cage
Evaporator(MCM behind)
Memory Cards
Blowers
IBM IBM zSerieszSeries 900 Server 900 Server (continued)(continued)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM IBM zSerieszSeries 900 Server Evaporator900 Server Evaporator
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Blower
Filter Drier
Compressor
IBM IBM zSerieszSeries 900 Modular Refrigeration Unit 900 Modular Refrigeration Unit
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Large IBM ServersLarge IBM Servers
IBM z900S/390 Mainframe
IBM pSeriesRegatta-H
20KW Dual Bulk Power:
CPU Cage: * 32 way SMP @ 1.3GHz
CPU Cage Cooling Blowers: * High Pressure & Flow * Intelligent Variable Speed
Removable Media Drawer:
I/O and Storage Drawer:
13KW Dual Bulk Power:
Modular CPU Cooling Units: * Cools CPU Chips to 0C Tj * Advanced Refrigeration * Fully Redundant
CPU Cage: * 20 way SMP @ 0.8GHz
4 Memory Books: * 24GB / Memory Book
CPU Cage Cooling Blowers: * High Pressure & Flow * Intelligent Variable Speed
I/O Cage: * 28 I/O Book Slots
High Density Single Frame Systems
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM IBM zSeries zSeries Server Model z900Server Model z900
Bulk Power Supplies
Modular Refrigeration Units(MRU)
Evaporator/MCM
Blowers
Input/OutputBasePlate
Ceramic Substrate
Al/Cu Hat
C-Ring
Chip
DeCap
Thin Film
C4s
Grease
Outlet Inlet
Evaporator
Oil Interface
BasePlate
Ceramic Substrate
Al/Cu Hat
C-Ring
Chip
DeCap
Thin Film
C4s
Grease
Outlet Inlet
Evaporator
Oil Interface
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM’s IBM’s xSeries xSeries Model 335Model 335
Xeon ProcessorSecond
processor spot
4 DIMM slots
PCI-X Slots Fans
Power Supply
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM’s IBM’s zSerieszSeries Model z990: TModel z990: T--RexRex
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Data Center CoolingData Center Cooling
Facilities Chilled Water Side View
CRAC CRAC
CRAC - Computer Room Air Conditioner
CRAC
Top View
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Numberof
Modules
Numberof
Processors
Power (1) (W)
Water Flow
Rate (1) (gpm)
Air Flow
Rate (1) (cfm)
Module n/a 28 40 n/a 6
Board 36 1,008 1,440(2,000)
1(1.5)
216(300)
Cabinet 144 4,032 5,760(8,000)
4(6)
864(1,200)
System 36,864 1,032,192 1,474,560(2,048,000)
1,024(1,536)
221,184(307,200)
Note: 1) Top number pertains to the processors; bottom numbers (in parentheses) pertain to the total package
Super Computer System Summary Super Computer System Summary (studied)(studied)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
board
substrate
cap
Al Piston
Al/ CuCold Plate
Al Hat
Square Header(if necessary)
Super Computer System Cooling Module Super Computer System Cooling Module (studied)(studied)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
MER
Pump
HeatExchanger
BuildingCoolant
Return
Supply
Super Computer Cooling System Super Computer Cooling System (studied)(studied)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
IBM Prototype DataIBM Prototype Data--Storage System: Storage System: StorageStorage--Array BrickArray Brick
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Cooling Technologies
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Vapor Chamber Heat Spreader
Chips(s)Vaporchamber
Air flow
Heat sink
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Examples of Heat Pipes Used in Electronics Cooling
Q
Q
Q
QAir Flow
Air Flow
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Air flow
Q
Example of a Large Air-Cooled Heat Sink forA High Performance Processor Module
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Closed Loop Water Cooling SystemWith Heat Rejection to Air
Air to waterheat exchanger Filter
Pump
Water reservoir
Air flow
Cold plate
Electronic module
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Closed Loop Water Cooling SystemWith Heat Rejection to Air
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Closed Loop Water Cooling SystemWith Heat Rejection to Air
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
InletOutlet
Coolingcap
ChipSubstrate
Liquid Jet Impingement Cooling
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Liquid Spray Cooling
InletOutlet
Coolingcap
Substrate Chip
Board
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
DC Rotary
Accumulator
Flexible Hose
Flexible Hose
Evaporator
Hot Gas Bypass Valve
TX Valve
Filter/DrierCondenser
Compressor
Refrigeration Loop and Components forCooling a High Performance Processor
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
PP P P P PN N N N N N
Substrate
Cap
Epoxyinterfaces
Thermoelectricmodule
Chip
Thermalpaste
Heatsink
Module With a Thermoelectric CoolerCooling Enhancement of an Electronic
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Advanced Cooling Technology Development Activities
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Two-Phase Thermosyphon Test Vehicle
Recent Research (DARPA HERETIC)Microfabrication Alliance
(Georgia Tech/Maryland/Sandia/HP/Thermacore)
Demonstrated for 85 W Intel Pentium 4 Processor in 2001.
“Heat Out of Small Packages”, Y. Joshi, Mechanical Engineering, Vol. 123, pp. 56-58, Dec. 2001.
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/2.html
Recent Research (DARPA HERETIC)Spray Cooling (Carnegie Mellon University)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/4.html
Recent Research (DARPA HERETIC)Droplet Atomization and Microjets (Georgia Tech)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/5.html
Recent Research (DARPA HERETIC)Thermoelectric Coolers for Lasers (JPL)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/6.html
Recent Research (DARPA HERETIC)Thermoacoustic Refrigerators (Rockwell)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/7.html
Recent Research (DARPA HERETIC)Electrokinetic Pumped Loops (Stanford)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/10.html
Recent Research (DARPA HERETIC)Microjets With Liquid/Vapor Phase Change (UCLA)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
http://www.darpa.mil/MTO/HERETIC/projects/11.html
Recent Research (DARPA HERETIC)Soild State Thermionic Coolers (UCSB)
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Summary and ConclusionsSummary and Conclusions
CMOS will continue to be the pervasive semiconductortechnology for both memory and logic.Chip sizes will increase but with a higher correspondingincrease in circuit density resulting in higher heat flux.All new electronic products will most likely be air-cooled,including most computers, for the next few years.Portable (laptop) computers will need enhanced cooling technologyin the near future despite the emphasis on low power dissipation.Power of hand held devices is not increasing with time. Battery lifeposes major restrictions on power dissipation and most applicationsdo not require any thermal management.
Low temperature cooling may get “hot” in the near future.Supercomputers with highly parallel scalable design may require new coolingsystems when node power exceeds current levels, or the number of nodescontinues to increase significantly, resulting in a large system load “explosion”.Cost will be a significant challenge for all future thermal designs and thespeed to accomplish new designs will be vital to their success.
High heat flux cooling capability is required for all high performance electronics.High thermal conductivity interface material is needed for heat sink applications.
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Future Cooling TechnologiesFuture Cooling Technologiesand Strategyand Strategy
Strategy for the Future- Explore all options- Establish a closer working relationship with vendors
- Pool resources to fund cooling technology development
- Get university/research labs involved
Enhanced Air Cooling Technology and System- High performance heat sink- Mini air movers for local enhancement
- Higher pressure air movers and higher volume air flow systems
- Highly parallel flow distribution system
- Active redundancy with control
Other Candidate Cooling Technologies- Direct liquid cooling technology - for high performance applications- Heat pipe and vapor chamber cooling technology - for special situations
- Thermoelectric cooling technology - for special situations
- Low temperature cooling technology - for performance enhancement- Self-contained, low cost liquid cooling technology- Thermal interface enhancement technology
Dedicated to the Advancement of North American Electronics Manufacturing “Draft—”Not for Citation, Publication, or Distribution”
Low cost, high performance, direct immersion cooling technology
Low cost, high performance thermal interface (10X) technology
Low cost, high performance cold plate (5X) technology
Low cost, high performance heat sink (5X) technology
Low cost and low noise (2X), high performance (2X) air/liquid moving device technology
Low cost, high performance, scalable cooling system
Low cost, high performance, future data center cooling concept
Grand Challenges for Electronic CoolingGrand Challenges for Electronic CoolingTechnology in the Coming DecadeTechnology in the Coming Decade