http://openfabrics.org/ ofed - status and process november 2007 tziporet koren

http://openfabrics.org/

OFED - Status and Process

November 2007Tziporet Koren

2http://openfabrics.org/ Mellanox Technologies

Agenda

OFED 1.3 StatusOFED Development ProcessOFED Future Discussion


OpenFabrics Software Stack


OFED – Update Since Sonoma

OFED 1.2 was released on 22-JunKernel base: 2.6.20

OFED 1.2.5 was released on 16-AugKernel base 2.6.22ConnectX support

OFED 1.2.5.3 was released on 14-NovRDS in GA for Oracle 11g RAC

OFED 1.3 – in middle of release phaseKernel base 2.6.24


OFED 1.3 Components

HCA/NIC Drivers Mellanox, QLogic, IBM, Chelsio, NetEffect

Core: verbs, mad, SMA, CMA, SA cache

IPoIB SDP SRP, iSER SRP Target RDS Qlogic_VNIC UDAPL OSM Diagnostic tools

NFS-RDMA – still open

Bonding module MPI Components

MVAPICH Open MPI MVAPICH2

Benchmark tests

Proprietary MPIs: Intel, HP, Scali

Proprietary SMs: Cisco, Voltaire, Qlogic

OFA development Add on

Tested with


Main New Features - General

Base kernel: 2.6.24Install

Minimize integration effort into OS distributionUser level split into separately-installable RPMsBased on feedback from Novell and Redhat

PackageSources arrangement for the end userAdded the make-dist script

Reduce compilation warnings of backports


General Features – Cont.

QoS OSM, CM & CMA, ULPs: SRP, SDP, iSER, IPoIB

Updated SA cache Integrated with sa_query module Use of SA events (GID up/down) to provide additional

synchronization User-space control for dynamic enable/disable and update

Extended RC (XRC) Support New verbs to enable XRC

New modules NES (NetEffect driver) SRP Target: Based on Mellanox IBGD SRPT and interfaces with

SCST scsi target middle level driver


Main New Features – ULPs

IPoIBNAPIStateless offloads – for Datagram modeSmall message performance improvements

RDS: RDMA with Zero copy (API version 3) SDP:

Asynch IOSend Zero Copy

uDAPL 2.0 In addition to uDAPL 1.0


Main New Features – Management

OpenSM Quality of Service manager

QoS implementation of the IBTA annex QoS policy parser and support for SA (PR, MPR)

Scalability/stability improvements Performance improvements of the routing algorithms (min-hops, up/down,

lash) Optimized port and switch tables update policy

“Native” daemon mode Performance manager

Fetches and stores performance counters OpenSM integrated, work in any OpenSM state (standby, disabled) Plug-in interface

Diagnostic tools enhancements IBSIM – IB fabric simulator (new)

Emulates user_mad layer useful for OpenSM and management tools testing


Main New Features - MPI

MVAPICH 1.0 Asynchronous progress Flexible message coalescing APM support Enhanced mpirun_rsh for scalable launching UD-based design Optimized and tuned collectives Support for ConnectX Support for QLogic/PSM

Open MPI 1.2.5 ConnectX support

MVAPICH2 1.0 Message Coalescing Hot-spot avoidance Application-initiated systems-level checkpointing APM support Multi-rail support for iWARP On-demand connection management for iWARP and uDAPL RDMA Read Blocking


OFED 1.3 Release Status

Features freeze: Oct 10, 2007 Alpha: Oct 15, 2007– kernel base: 2.6.23 Beta: Nov 20, 2007– kernel base: 2.6.24-rc2 RC1: Dec 4, 2007 RC2: Dec 18, 2007 RC3: Jan 8, 2008 RC4: Jan 22, 2008 Release: Jan 29, 2008Note: The delay mainly caused by the move to

kernel 2.6.24


OFED 1.3 OS Matrix

kernel.org: kernel 2.6.23 and 2.6.24 Novell

SLES 10 SLES 10 SP1 (up1)

Redhat RHEL 4 (up4 and up5) RHEL 5 (up1)

Free distros Partial support only (no QA) Open SuSE 10.2 Fedore C6 Ubuntu 6.06 (with RPM package installed)

Open: OSVs plans to pick up OFED 1.3, or even 1.2.5


Agenda



OFED – Development Process

Start – release definition Agree on release features Agree on OS matrix and kernel base Decide on target schedule Release definition is placed on the Wiki

https://wiki.openfabrics.org/tiki-index.php?page=OFED+Development+Wiki

Note In all OFED releases there were changes in the above that caused release

slips Tracking meetings

Bi-weekly meetings during the development phase Weekly meetings on the release phase Meeting summaries are placed on the Wiki

https://wiki.openfabrics.org/tiki-index.php?page=Meeting+Minutes Bug tracking in bugzilla

https://bugs.openfabrics.org/


Kernel Development

Have one kernel git tree that is maintained by Vladgit://git.openfabrics.org/ofed_1_3/linux-2.6.git

Cloned from Linux git tree Add-on components (e.g., SDP) are added to the tree New features and bug fixes are added as patches under

the fixes directory Patch convention name: <module>_<number>_<text>.patch

(e.g., ipoib_0010_cq_coalescing.patch) Supporting distros and previous kernels:

Backport directory for each kernel OFED can compile on Add-on directory for headers from Linux

The make-dist script applies first the “fixes” patches and then the appropriate backport patches


User Space Development

Each maintainer has a separate git tree on the serverExamples

git://git.openfabrics.org/~sashak/management.git git://git.openfabrics.org/~shefty/libibcm.git

Each package has its own RPM SPEC file All packages should also be released to the ofa

download sitehttp://www.openfabrics.org/downloads.htm

Approaching a release – a branch with the release name is opened

Userspace component fixes must be committed -- no patching scheme is used


External Packages Development

External packages are provided as source RPMs

Each external package has a directory on the ofa serverExample – the bonding package:

/home/monis/public_html/ofed_1_3/latest.txt file points to the package that should

be usedEx.: ib-bonding-0.9.0-19.src.rpm


OFED Daily Build

An OFED package is created everyday at 6am PSThttp://www.openfabrics.org/builds/ofed-1.3/ This package can be used by all to test the daily changes

This package includes Source RPMs for kernel, all user space components, external

modules OFED scripts (install, etc.) OFED documents

Development phase: the build uses the master branch of all git trees

Release phase: the build is based on the release branch


OFED Release Phase

Starts after the alpha releaseClose tracking of components statusAlpha, beta and RC are done according to

the package maturityBugs severity tracking which bugs must

be fixed for the releaseUpdate of all documents


Agenda



OFED Goals

Enterprise Working Group within Open Fabrics Alliance (OFA)OFED has become a brand name in the industry today

Collaborative effort to test & release OFA software Broader test participation Components – Kernel & User Space Interoperability Add-on components for vendors to differentiate

above OFA

Reduces deployment complexity and cost


Reasons to Continue OFED

Industry synchronization point Set of kernel modules, libraries and applications

that are tested togetherComprehensive tests by many vendorsOSVs rely on our qualification work

Faster new feature development and deployment New drivers for new IHV devicesULP updates for new ISV software packages, e.g.,

Oracle 11g Enables support of new features in existing OSes

Ex.: Bonding to work on Redhat EL4 up5, SLES 10, etc


OFED-based and OS Distro-based

OFED-based and OS Distro-based distribution of OFA software – one or both

Originally, some people intended OFED to be a short-term distribution until mainstream Linux distributions caught up and were able to distribute IB/iWARP support

Today there is synergy between OFED and OS vendors as they rely on the OFED quall and support benefits delivered by commercial IB/iWARP suppliers

Are OSV’s ready to take more quall and support responsibilities at this time?

Discussion …

http://openfabrics.org/ ofed - status and process november 2007 tziporet koren

Documents