minipack: low power 100g fabric networking architecture ...… · minipack: low power 100g fabric...
TRANSCRIPT
![Page 1: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/1.jpg)
![Page 2: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/2.jpg)
MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design
Sami Khan, Technical Program ManagerJimmy Leung, Mechanical EngineerFacebook, Inc
NETWORKING
![Page 3: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/3.jpg)
Minipack HW Overview
Minipack Mechanical Deep DiveAgenda
![Page 4: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/4.jpg)
Specifications
NETWORKING
Minipack – Next-Generation 128x 100G Switch
Switch ASIC: Broadcom Tomahawk-3Size: 4RU (vs. 8RU Backpack)Power: ~1.4KW budgetary (At full line rate, fully populated with 128x QSFP28 CWDM4 optics)Radix: 128x
Adopt cutting edge switch ASICSupport mature optics – 100G CWDM4Lower Power / Smaller Size• 1/2 height, ~1/2 power compared to existing Tomahawk based design – Backpack
Back
pack
Minipack
Specifications
![Page 5: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/5.jpg)
MiniPack Announced during Global Summit in March 2019 Design formally accepted by OCP in June 2019.
NETWORKING
![Page 6: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/6.jpg)
Minipack System Components
Specifications
Switch Main Board (SMB)
Power Distribution System• Horizontal Bus Bar (HBAR)• Power Distribution Board (PDB): Left and Right
System Control Module (SCM) - FRU
Fan Control Module (FCM): Top and Bottom
Port Interface Module (PIM) - FRU• PIM-16Q: Port Interface Module with 16 x QSFP28 100G
DOM (Digital Optics Monitoring) FPGA
NETWORKING
Design Files
![Page 7: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/7.jpg)
Minipack Chassis Architecture• Orthogonal-direct architecture
• Opens up airflow path for better thermal efficiency
• Supports 100G CWDM4-Lite optics with 55 ℃ case temperature limit
• Shortens PCB traces for lower lossSCM / MiniLake
PIM
TH3 and “radiator”
Fan
• FRU-able, modular PIM (line card)• Vertically oriented• PIM-16Q • Easy to explore other PIM options
• FRU-able SCM (micro-server carrier)
PSU
![Page 8: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/8.jpg)
Where to use Minipack
Specifications
Minipack• FSW – Fabric Switch• SSW – Spine Switch• FA – Fabric Aggregator
48. . .
48
4848
48
15
948
1
26
10
37
11
48
11
1
1
2
3
48
. . .
. . .
. . .
. . .
. . .
4
22
2
1
2
3
48
4
33
3
1
2
3
48
44
44
44
44
1
2
3
48
12
34
12
34
. . .
. . . . . .
. . .
. . .
. . .
. . .
SSW
RSW
FSWSupports multiple generations of rack switches
NETWORKING
Design Files
![Page 9: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/9.jpg)
Minipack : Overall Mechanical Spec.
• Chassis Dimensions: 176.2mm(H) x 440.4mm(W) x 738.1mm(D)
• Weight: Fully loaded chassis with all FRUs is 54kg• Chassis assy with thermal syphon• 8 - Fantray assy• 8 - PIM-16Q• 1 - SCM• 4 -1500w PSU
• Handle: One handle and a handle pocket on each side of the chassis
NETWORKING
Specifications
![Page 10: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/10.jpg)
Specifications
Minipack HW Overview
Minipack Mechanical Deep DiveAgenda
![Page 11: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/11.jpg)
Minipack : FB Modular Switch Platform
• Orthogonal direct architecture• Disaggregated data, control, and management plane design
NETWORKING
Specifications
![Page 12: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/12.jpg)
Minipack : FB Modular Switch Platform
NETWORKING
Specifications
SCM / MiniLake
PIM
TH3 and “radiator”
FanPSU
• Top Cross-Sectional View
![Page 13: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/13.jpg)
Minipack : FB Modular Switch Platform
• Side Cross-Sectional View NETWORKING
Specifications
![Page 14: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/14.jpg)
Minipack : FB Modular Switch Platform
NETWORKING
Specifications
• PIM and SCM slide forward from the chassis
![Page 15: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/15.jpg)
Minipack : FB Modular Switch Platform
• PSU and Fan module slide outside from the chassis NETWORKING
Specifications
![Page 16: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/16.jpg)
Minipack : FB Modular Switch Platform
• Showing front to back vent opening NETWORKING
Specifications
![Page 17: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/17.jpg)
Minipack : FB Modular Switch Platform
• Metal midplane design to allow maximum airflow to past thru the chassis
NETWORKING
Specifications
![Page 18: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/18.jpg)
Innovative Design
• Disaggregated architecture: - Field Replaceable SCM(System Control Module)- Field Replaceable PIM(Port Interface Module)
• Orthogonal Direct Chassis Architecture- All major module cards are designed to be mated orthogonally- Open up more air channel for a better thermal performance- Reduce the PCB trace length for better signal integrity
• A sophisticated thermal design to support low cost 55C CWDM4 optics
NETWORKING
Specifications
![Page 19: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/19.jpg)
Minipack System Major Components• Switch Main Board (SMB)• System Control Module (SCM) • Port Interface Module (PIM)
- PIM-16Q: Port Interface Module with 16 x QSFP28 100G • Fan Control Module (FCM)
- Fan Control Module Top (FCM-T)- Fan Control Module Bottom (FCM-B)
• Power Distribution System- Horizontal Bus Bar (HBAR)- Power Distribution Board Left (PDB-L)- Power Distribution Board Right (PDB-R)
NETWORKING
Specifications
![Page 20: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/20.jpg)
Minipack Chassis: 128 x 100G
NETWORKING
Specifications
![Page 21: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/21.jpg)
Minipack Front View with PIM-16Q
NETWORKING
Specifications
PSU right air channel
ESD Jack
PSU left air channel
Pull Tag
![Page 22: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/22.jpg)
Minipack Rear View
NETWORKING
Specifications
![Page 23: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/23.jpg)
Power Distribution System
• Two layer busbar assy• PDB-Left and PDB-Right
NETWORKING
Specifications
![Page 24: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/24.jpg)
Power Distribution System
NETWORKING
Specifications
![Page 25: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/25.jpg)
Power Distribution System
NETWORKING
Specifications
![Page 26: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/26.jpg)
Power Distribution System
• CoolPower connector also an Orthogonal Connector NETWORKING
Specifications
CoolPowerModule
side connector
CoolPower Busbar side connector
![Page 27: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/27.jpg)
FRUs
• Front of the chassis- SCM(System Control Module)- PIM(Port Interface Module)
• Rear of the chassis- Fantray assy- PSU
NETWORKING
Specifications
![Page 28: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/28.jpg)
System Control Module (SCM)
COMe module• Minilake
SATA M.2 SSD
NETWORKING
Specifications
![Page 29: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/29.jpg)
System Control Module (SCM)
NETWORKING
Specifications
Key to prevent inserting in wrong slot
![Page 30: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/30.jpg)
System Control Module (SCM)
NETWORKING
Specifications
![Page 31: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/31.jpg)
Port Interface Module (PIM)
Type of PIM• PIM-16Q: 16 x QSFP28
One DMO Connector to SMB for both data and control plane signals
CoolPower connector to power bus bar
NETWORKING
Specifications
![Page 32: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/32.jpg)
PIM-16Q Module
NETWORKING
Specifications
Key to prevent inserting in wrong slot
168
![Page 33: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/33.jpg)
PIM-16Q Components
NETWORKING
Specifications
![Page 34: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/34.jpg)
PIM-16Q interface
NETWORKING
Specifications
![Page 35: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/35.jpg)
Fan-tray Assy
• Screw-less latch design for easy maintenance
• Powerful 80 x 80 mm CR fan
• Hot swappable
• LED on rear panel
• Each FCM carries 4 Fan-tray, total 8 fan-tray in minipack chassis
NETWORKING
Specifications
![Page 36: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/36.jpg)
High Performance PSU
• Keying feature on Minipack chassis to prevent PSU to mate in wrong direction
• Power cord retention is built in on the HVAC connector
NETWORKING
Specifications
![Page 37: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/37.jpg)
• Front-to-rear airflow
o Ensures N+1 rotor redundancy
• Supports 100G modules at 3.5W with 55°C case temperatures
o Up to 35°C ambient, 6000ft altitude and under one rotor failure
• Fan control delivers desired margins on all temperature-sensitive components and system-level ΔT targets
o Under typical and high workloads
Thermal DesignNETWORKING
![Page 38: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/38.jpg)
• Separation between system and PSU airflow channels
• Other itemso Energy-efficient fan operation (typically ≤ 5% of system power)
o System-level ΔT targets ensure efficient operation in deployed facilities
o Thresholds defined for critical parameters to sustain operation under unexpected conditions
Thermal DesignNETWORKING
![Page 39: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/39.jpg)
Call to Action• MiniPack design is available for your use. Please share
your experiences with the community.
• Please visit OCP website to learn more on MiniPack designhttps://www.opencompute.org/wiki/Networking/SpecsAndDesigns#Facebook_Minipack
![Page 40: MiniPack: Low Power 100G Fabric NETWORKING Architecture ...… · MiniPack: Low Power 100G Fabric Architecture - Mechanical & Thermal Cooling Design. Sami Khan, Technical Program](https://reader033.vdocuments.mx/reader033/viewer/2022052923/5f0404507e708231d40be7b3/html5/thumbnails/40.jpg)