compact packaging and liquid cooling technology for exascale · compact packaging and liquid...

19
by Peter Hopton, Fabien Chaix, e.a.; Iceotope, UK; and FORTH, Heraklion, Crete. Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The work described in this ppt has been conducted within the project EuroEXA. This project has received funding from the European Union’s Horizon 2020 (H2020) research and innovation programme under the Grant Agreement no 754337. This ppt and the content included in it do not represent the opinion of the European Union, and the European Union is not responsible for any use that might be made of its content.” Including Achievements from ExaNeSt and Concepts and Achievements from EuroEXA

Upload: others

Post on 28-Jul-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

by Peter Hopton, Fabien Chaix, e.a.; Iceotope, UK; and FORTH, Heraklion, Crete.

Compact Packaging and Liquid Cooling Technology for Exascale

For the Purpose of Dissemination Funding Allocation; “The work described in this ppt has been conducted within the project EuroEXA. This project has received funding from the European Union’s Horizon 2020 (H2020) research and innovation programme under the Grant Agreement no 754337. This ppt and the content included in it do not represent the opinion of the European Union, and the European Union is not responsible for any use that might be made of its content.”

Including Achievements from ExaNeSt and Concepts and Achievements from EuroEXA

Page 2: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

M.2(NVMe)512GByte

4PS-GTR

FPGAXCZU9EG-ffvc900

16GByteDDR4

SODIMM

64

64MBQSPI

‘Network’ FPGAXCZU9EG-ffvc900

Connector

10GTH

FPGAXCZU9EG-ffvc900

12+12LVDS

‘Storage’ FPGAXCZU9EG-ffvc900

2GTH

12+12LVDS

2GTH

12+12LVDS

2GTH

12+12LVDS

2GTH

12+12LVDS

2GTH

16GByteDDR4

SODIMM

64

16GByteDDR4

SODIMM

64

16GByteDDR4

SODIMM

64

RGMII2xUART

64MBQSPI

64MBQSPI

64MBQSPI

TheQFDBarchitecture

Page 3: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

The QFDB implementation

Rev. A versionàDDR issue (erroneous

swap)à48Và12V regulator issue

Rev. B versionà48Và12V regulator issueàSSD (configuration) issue

Key factsDimension 130 x 120 mmHeight < 30 mmPCB 16 layers, Megtron-6Estimated TDP 100~120W

Page 4: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

M.2(NVMe)512GByte

QFDBI2CSensorsnetwork

ExaNeStConfidential4

FPGAXCZU9EG-ffvc900

Connector

FPGAXCZU9EG-ffvc900

‘Storage’ FPGAXCZU9EG-ffvc900

16GByteDDR4

SODIMMMTA18ASF2G72HZ-2G3x4

12VSens.

VCCINTLPSens.

VCCAUX18Sens.

Clockgen.Si5341

VCCINTFPSens.VCCINTSens.

PSDDR504Sens.

16GByteDDR4

SODIMMMTA18ASF2G72HZ-2G3x4

16GByteDDR4

SODIMMMTA18ASF2G72HZ-2G3x4

16GByteDDR4

SODIMMMTA18ASF2G72HZ-2G3x4

‘Network’ FPGAXCZU9EG-ffvc900

VCCINTFPSens.VCCINTSens.

PSDDR504Sens.

VCCINTFPSens.VCCINTSens.

PSDDR504Sens.

VCCINTFPSens.VCCINTSens.

PSDDR504Sens.

PS_I2C0+PL0

PL2

PL1PL1

PL1PL1

PS_I2C1

PS_I2C0+PL0

PS_I2C0+PL0

PS_I2C0+PL0

ThermalSensor

2.5V

3.3V

1.8V

Top-levelmodule

Bottom-levelFPGAmodule

Page 5: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Current QFDB AchievementsFeature StatusZynq functions Validated individually,DDR-RAM Validated

Linux runs with DDR4-2133Bare-metal test pass with DDR4-2400

QSPI memories Validated on LinuxMinor issues with bare-metal

Gigabit Ethernet ValidatedLVDS links Validated, ~5% of traces are problematicI2C busses Validated all but the M.2 deviceUART links ValidatedClock generators ValidatedM.2 Solid State Disk Validated on Rev. A,

Page 6: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Remote RAM Linux setup• We need Linux to validate board sub-systems:

• PCIe, QSPI, Gigabit Ethernet• Successful test of external RAM to boot.è Accessed memory from another boardè Forward memory access through Network FPGA

Page 7: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

ExaNeSt Tracks

Track 1• 4 QFDBs• 800W Max.• 1.5 per u (b2b)• 45C Inlet• Passive I/O

Track 2• 16 QFDBs• 3.2kW Max• 2 per u (b2b)• 45C Inlet• Onboard Switch

Page 8: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

QFDB x Track-1 Mezzanine

Page 9: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

EuroEXA Concepts

Aisle

Aisle

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

Aisle

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

CoolantandDCPower

Aisle

Aisle

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

Aisle

2.5m ICEOTOPE

CONFIDENTIAL

BatteriesandRectifiersfor250kWofLoad

BatteriesandRectifiersfor250kWofLoad

CoolantandDCPower

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

600x60

0mm

cabine

twith

6

chassis&2xCD

Us

CoolantandDCPower

BatteriesandRectifiersfor250kWofLoad

BatteriesandRectifiersfor250kWofLoad

CoolantandDCPower10.6m

Wall

Wall

2MW In 40ft ISO or 10mx2.4m

• Shipping Container Optimised Cabinets (2 rows)• 2MW per shipping container• Enables 3D stacking to minimise distances

Page 10: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The
Page 11: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Advancing to Track 2

• >4x Thermal Improvement Required• 7x Thermal Improvement Targetted• Addition of an integrated high speed switch• Large thermal and mechanical challenge

Page 12: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

What This Means For Track 2Key facts

Dimension 21” Rack x 762mm Deep

Nodes 16 x QFDB (or Next Gen)

Height 1 O-uPCB Megtron-6Estimated TDP 3.2kWI/O (Uplink) 800Gb/sI/O (Total) 2Tb/s

Page 13: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The
Page 14: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Tackling the 7x Cooling Challenge

Page 15: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Proving it will work!Scope• Track 2 horizontal chassis• Thermal proxy QFDBs x16, nominal

TDP 200W (3.2kW overall)

• Critical conditions• Junction temperature < 100oC• Maximum PC temperature < 70oC

• Three tests defined• Critical SC inlet temperature• Critical TDP duty• Critical SC flow-rate

Page 16: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Proving it will work!

Results at 45C Inlet

Page 17: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Proving it will work!

Results at 45C Inlet

Page 18: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

Thermal Validation SummarySummary

• Based on stated nominal conditions The following critical conditions were determined

• SC feed temperature is 47oC• TDP load factor is 1.1 (1,084W)• SC flow-rate is 1.6

• In each case the Primary Coolant max temperature was found to be the limiting factor

• These findings still need further verification experimentally

Page 19: Compact Packaging and Liquid Cooling Technology for Exascale · Compact Packaging and Liquid Cooling Technology for Exascale For the Purpose of Dissemination Funding Allocation; “The

ManyThanksPleaseConnectatthecoolingguy.me

Presenter– PeterHoptonDisseminationLead,EuroEXA,ExaNeSt

Founder,Iceotopehttp://thecoolingguy.me

© 2017 EuroEXA and ExaNeSt Consortia Member Rights Holders Project ID: 754337