Lustre*  &  Intel®  Omni-­‐Path  Fabric  

Keith Mannthey, Lustre Solutions Architect Intel High Performance Data Division

Copyright ® Intel Corporation 2016. All rights reserved. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. 3

Legal Information

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at http://www.intel.com/content/www/us/en/software/intel-solutions-for-lustre-software.html.

Intel technologies may require enabled hardware, specific software, or services activation. Check with your system manufacturer or retailer.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty- free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

* Other names and brands may be claimed as the property of others.

© 2016 Intel Corporation

Agenda  

4  

•  LNET  Introduc/on  •  Intel®  Omni-­‐Path  Architecture  (OPA)  •  LNET  Intel®  OPA  Considera/ons  •  LNET  Intel®  OPA  Integra/on  into  Exis/ng  Fabrics  

Intel®  Manager  for  Lustre*  

LNET  Introduc?on  

Management    Network  

LNET  High  Performance  Data  Network  (Intel®  OPA  ,  InfiniBand*,  Ethernet)  

Metadata    Servers    (1-­‐10s)  

Object  Storage  Servers    (10s-­‐1000s)  

Lustre  Clients  (1  –  100,000+)  

Object  Storage    Targets  (OSTs)  

Object  Storage    Targets  (OSTs)  

Management  Target  (MGT)  MetadataTarget  (MDT)  

HSM  

Na/ve  Lustre*  Client  for  Intel®  Xeon  Phi™  processor    Intel®  Digital  Security  Surveillance?  (hin/ng)  

Intel®  Omni-­‐Path  Support    Robin  Hood  

Open  ZFS,  RAIDz  Hadoop*  Adapters  

5  

LNET  

6  

•  Designed  to  meet  the  needs  of  large-­‐scale  compu?ng  clusters  –  Op/mized  for  very  large  node  counts,  high  throughput  

•  Works  with  most  networks,  supports  RDMA  –  Intel®  Omni-­‐Path,  Ethernet,  Infiniband*,  ELAN,  Myrinet*,  etc.  

•  LNET  is  independent  of  the  Lustre  file  system  –  Abstracts  network  details  from  Lustre  –  Implemented  as  a  set  of  kernel  modules  

LNET  (con?nued)  

7  

•  Networks  are  given  unique  names  –  o2ib0,  tcp0,  tcp1  

•  Lustre  Network  Iden?fier  (NID)  defines  interfaces  – 10.1.145.16@o2ib0  

•  Includes  na?ve  support  for  mul?ple  networks  –  Accomplished  via  the  Lustre  Network  Driver  (LND)  

•  Infiniband  via  o2ib  verbs  interface,  with  RDMA  support  •  Ethernet  via  TCP/IP  interface  

•  Lustre  -­‐>  Network  RPC  API  -­‐>  LNET  -­‐>  LND  -­‐>  Linux  Driver  

Intel®  Omni-­‐Path  Architecture  

Building  on  the  industry’s  best    technologies  §  Highly  leverage  exis/ng  Aries  and  Intel®  True  Scale  fabric  

§  Adds  innova/ve  new  features  and    capabili/es  to  improve  performance,    reliability,  and  QoS  

§  Re-­‐use  of  exis/ng  OpenFabrics    Alliance*  soeware  

Robust  product  offerings  and  ecosystem  §  End-­‐to-­‐end  Intel  product  line  §  Strong  ecosystem  with  70+  Fabric    Builders  members  

4  8  

HFI  Adapters  Single  port  x8  and  x16  

x16    Adapter    

(100  Gb/s)  

x8  Adapter    (58  Gb/s)  

Edge  Switches  1U  Form  Factor  24  and  48  port  

48-­‐port    Edge   Switch  

24-­‐port    

Edge  Switch  

Director  Switches  QSFP-­‐based  

192  and  768  port  768-­‐port    

Director  Switch    (20U  chassis)  

192-­‐port    

Director  Switch    (7U  chassis)  

Silicon  OEM  custom  designs  HFI  and  Switch  ASICs  

HFI   silicon  Up  to  2  ports  

(50  GB/s  total  b/w)  

Switch  silicon    up  to  48  ports    

(1200  GB/s  total  b/w  

Socware  Open  Source  

Host  Socware  and    Fabric  Manager  

Cables  Third  Party  Vendors    Passive  Copper    Ac?ve  Op?cal  

LNET   Intel®  OPA  Considera?ons  

9  

•  Base  OS  Support  –  Red  Hat  Enterprise  Linux  7  –  SUSE  Linux  Enterprise  Server  12  

•  Lustre  2.7+  Required  for  OS  Support  Server  Side  •  Intel  Fabric  Suite  (IFS)  delivers  Driver  UPDATES  

–  IFS  updates  Base  OS  OFED  components  only  as  required  –  Enables  the  use  of  other  “In  Kernel”  drivers  concurently  

LNET   Intel®  OPA  Considera?ons  (con?nued)  

10  

•  Intel®  OPA  gen  1  Supports  RDMA  Verbs  in  OFED  –  LNET  uses  the  exis/ng  LND  Infiniband*  Driver  

•  Only  LND  and  Driver  TUNING  required  for  opera?on  –  Automated  LND  semngs  at  LNET  install  /me  

Integra?on  into  Exis?ng  Fabrics  

Challenge  :  Intel®  OPA  is  not  directly  compa/ble  with  Infinband*  (IB).  Intel®  OPA  cannot  plug  into  IB  switch  

Solu?on:  LNET  Routers  

OPA  Lustre    Components  

LNet  Routers  

IB  Lustre  Componets  

11  

OPA   Inifiband  

LNET  Routers  Overview  

12  

•  LNET  Routers:  Lustre  Soeware  +  Standard  Hardware  –  Use  off  the  shelf  Hardware  –  Soeware  is  apart  of  standard  LNET/Lustre  –  Clustered  Deployment  Recommended  

•  Supported  Configura/ons  with  Redhat*  7.2  and  Lustre  2.7+  –  Intel®  OPA  -­‐>  Ethernet  –  Intel®  OPA  -­‐>  FDR  (use  in  kernel  drivers  for  IB)  –  Intel®  OPA  -­‐>  EDR  (use  in  kernel  drivers  for  IB)  –  See  Intel®  Enterprise  EdiBon  for  Lustre*  soFware  ConfiguraBon  For  LNET  Routers  

Wrap  up  

13  

•  Lustre  Intel®  OPA  is  in  produc/on  Today  •  LNET  Routers  provided  Flexible  Deployment  Op/ons  •  Learn  More  

– www.intel.com/Lustre