10g ethernet outlook for hpc

Bringing Speed and Intelligence to the Edge of the Network

© 2011 IBM Corporation

Is 10 Gigabit Ethernet Ready for HPC?


Interconnect Count ShareGigabit Ethernet 282 56.4%10 Gig Ethernet 0 0.0%Infiniband 141 28.2%Myrinet 10 2.0%Others 67 13.4%

HPC Interconnect Landscape

April 10, 2023 2


top500

Mass Market

IB

GigE

10Gig

The bigger picture for HPC

April 10, 2023 3


HPC Forecast: Strong Growth Over Next Five Years ($ Millions)

2007 2012 CAGR

Supercomputer $2,682 $3,512 5.5%

Technical Divisional $1,610 $3,092 13.9%

Technical Departmental $3,384 $5,763 11.2%

Technical Workgroup $2,400 $3,193 5.9%

Total $10,076 $15,617 9.2%

Source: IDC, 2008April 10, 2023 4


10 Gig Ethernet Issues

What’s holding up adoption?

10 Gig NICs

Price of Switches

Switch Scaling

PHY Confusion

Proof of Performance


10 Gig NICs

Prices are dropping fast

Major server vendors are including 10 Gig Ethernet as standard server feature (LOM)

Several NIC vendors proving mature and stable for HPC


Price of 10 Gig Ethernet Switches

Switch ports used to cost more than servers!

10 Gig E switches now list for <$500 / port

Top of Rack 10 Gb Switch

IBM10Gb Blade Switch

April 10, 2023 7


12 Leaf Switches144 Non-blocking ports

6 SPINE Switches

Switch Scaling

Typical CLOS Topology - 144 10GbE Ports•2-tier design scales to 288 ports

April 10, 2023 8


Up to 210 Servers :15 Enclosures6 10GbE uplinks per enclosure1 10GbE switch per enclosure

10.1.1.0 VLAN 1 10.1.2.0 VLAN 2 10.1.3.0 VLAN 3 10.1.4.0 VLAN 4

10.2.1.010.2.15.0Load distribution across the core using OSPF ECMP:

•Separate IP subnet for each enclosure•Separate VLAN and IP subnet for each RackSwitch•Full network path redundancy – not just link level •No Spanning Tree•2.3 to 1 oversubscription

10.1.4.0 VLAN 5 10.1.4.0 VLAN 6

HPC Topology – up to 208 10GbE Servers IBM BladeCenter Design

April 10, 2023 9


PHY Confusion

Optical standard interfaces for 10 Gig E:

–Fixed optics–XENPAK–X2–XFP–SFP+

10GBase-T (i.e. Cat5, RJ45)

Users have been unwilling to bet on a survivor!


And the winner is:SFP+

SFP+ Direct Attach Cables– Passive cables with SFP+ ends– Low cost - $40 – $50– High density – same as RJ45


10GBase-T

The problem was harder than thought

Expensive, power-hungry, and 2.6 usec latency

But – 10GBase-T will eventually become widespread


Performance

10 Gig Ethernet offers:– Same familiar operating environment– Ease of use, debug, and management– Path to 40 and 100 Gig Ethernet– 10x bandwidth and 8x better latency vs. Gig Ethernet– But – do applications run faster !??!?

• Vendors talk about micro-benchmarks• Most users care about execution time


(Version 2008.0)(Version 2008.0)

10GE 32% faster than 1G, equal to IB DDR, for 32 cores

PAM CRASH: Elapsed Time (sec)

April 10, 202314




PAM CRASH: Speed Up

April 10, 202315


10GE 4.25x faster than 1G, equal to IB DDR, for 32 cores

Vienna Ab-initio Simulation PackageMolecular Dynamics

VASP 4.6.28: Elapsed Time (sec)

April 10, 202316


10GE 6.3x faster than 1G, almost equal to IB DDR, for 64 cores

VASP 4.6.28: Speed Up

April 10, 202317


RADIOSS 9.0: Speed up



Finite Element Solver

April 10, 202318


• IBM BNT’s 10GbE outperformed InfiniBand– Significantly higher updates per second – 31% Lower latency than InfiniBand

RMDS Performance: IBM BNT’s 10GbE vs. InfiniBand

*Voltaire and BLADE tests used similar 3 GHz Xeon 5160 based servers with 4MB L2 cache

IB

April 10, 202319


BLADE’s 10GbE outperformed InfiniBand

RMDS PerformanceBLADE’s 10GbE vs. InfiniBand

RDMS Updates Per Second

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

Voltaire InfiniBand

BLADE 10GbE

*Voltaire and BLADE tests used similar 3 GHz Xeon 5160 based servers with 4MB L2 cache

10 GEIB

April 10, 2023 20


10 Gig Ethernet Issues

What’s holding up adoption?

10 Gig NICs

Price of Switches

Switch Scaling

PHY Confusion

Proof of Performance

• Prices are dropping • Under $500 / port• Proof points emerging• Benchmarks emerging


Trademarks and disclaimers

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries./ Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind.

The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Photographs shown may be engineering prototypes. Changes may be incorporated in production models.

© IBM Corporation 1994-2010. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml.

10g ethernet outlook for hpc

Technology