IBM Spectrum MPI Version 10.2

Release Notes

IBM

Note

Before using this information and the product it supports, read the information in “Notices” on page17 .

This edition applies to version 10, release 2, modification 0 of IBM® Spectrum MPI (product number 5725-G83) and to allsubsequent releases and modifications until otherwise indicated in new editions.© Copyright International Business Machines Corporation 2018.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

About this information........................................................................................... vConventions and terminology used in this information...............................................................................vPrerequisite and related information..........................................................................................................vi

Overview...............................................................................................................1

Features................................................................................................................3

Limitations and restrictions................................................................................... 7GPU limitations and restrictions................................................................................................................10NVIDIA CUDA limitations and restrictions................................................................................................11

Documentation....................................................................................................13Accessibility features for IBM Spectrum MPI........................................................ 15

Notices................................................................................................................17Programming interface information.......................................................................................................... 18

MPI support statement........................................................................................................................ 18Open MPI license................................................................................................................................. 18

Trademarks................................................................................................................................................ 20Terms and conditions.................................................................................................................................20Privacy policy considerations.................................................................................................................... 20

Index.................................................................................................................. 23

iii

iv

About this information

The release notes include late technical information that is not included in other topics, and alsohighlights new functions for IBM Spectrum™ MPI Version 10.2.

Disclaimer:

The functions or features found herein may not be available on all operating systems or platforms anddo not indicate the availability of these functions or features within the IBM product or future versions ofthe IBM product. The development, release, and timing of any future features or functionality is at IBM'ssole discretion. IBM's plans, directions, and intent are subject to change or withdrawal without notice atIBM's sole discretion. The information mentioned is not a commitment, promise, or legal obligation todeliver any material, code or functionality. The information may not be incorporated into any contractand it should not be relied on in making a purchasing decision.

This document explains parallel programming as it relates to IBM Spectrum, IBM's implementation ofOpen MPI 3.0. It includes information about developing, running, and optimizing parallel applications foruse in a distributed memory environment.

IBM Spectrum MPI is a complete MPI implementation, based on the Open MPI open source project, andis designed to comply with all the requirements of the Message Passing Interface standard, MPI: AMessage-Passing Interface Standard, Version 3.1, University of Tennessee, Knoxville, Tennessee, June 4,2015.

For information about Open MPI, and to obtain official Open MPI documentation, refer to the Open MPIwebsite.

Note: This document can be used along with information that is provided in the Open MPI website. Inmany cases, this document explains a topic at a high level, and then points you to the Open MPI websitefor more detailed information.

Conventions and terminology used in this informationTable 1 on page v shows the conventions used in this information:

Table 1: Conventions.

Convention Usage

bold Environment variables.

monospace Examples and information that the system displays, command line options,file names, pathnames.

bold monospace Command names and parameter names.

italic Italic words or characters represent variable values that you must supply.

Italics are also used for unit titles, the first use of a term, and generalemphasis in text.

<key> Angle brackets (less-than and greater-than) enclose the name of a key on thekeyboard. For example, <Enter> refers to the key on your terminal orworkstation that is labeled with the word Enter.

© Copyright IBM Corp. 2018 v

Table 1: Conventions. (continued)

Convention Usage

\ In command examples, a backslash indicates that the command or codingexample continues on the next line. For example:

mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \-E "PercentTotUsed < 85" -m d "FileSystem space used"

{item} Braces enclose a list from which you must choose an item in format andsyntax descriptions.

[item] Brackets enclose optional items in format and syntax descriptions.

<Ctrl-x> The notation <Ctrl-x> indicates a control character sequence. For example,<Ctrl-c> means that you hold down the control key while pressing <c>.

item... Ellipses indicate that you can repeat the preceding item one or more times.

| • In synopsis statements, vertical lines separate a list of choices. In otherwords, a vertical line means Or.

• In the margin of the document, vertical lines indicate technical changes tothe information.

Prerequisite and related informationIBM Spectrum MPI is a member of the IBM Spectrum Computing family (www.ibm.com/systems/spectrum-computing/).

To access the most recent IBM Spectrum MPI documentation in PDF and HTML format, refer to IBMSpectrum MPI Knowledge Center website.

IBM Spectrum LSF (Load Sharing Facility) also works in conjunction with IBM Spectrum MPI. For the latestLSF® documentation, see the IBM Spectrum LSF Knowledge Center website.

Terminology

For terms and definitions related to IBM Spectrum MPI, see the IBM Terminology website.

vi IBM Spectrum MPI Version 10.2: Release Notes

Overview

IBM Spectrum MPI Version 10.2 is a high-performance, production-quality implementation of theMessage Passing Interface (MPI). It is widely used in the high-performance computing (HPC) industry. Itis considered one of the standards for developing scalable, parallel applications. IBM Spectrum MPIVersion 10.2 is based on the Open MPI Version 3.0 and implements the full MPI 3.1 standard.

IBM Spectrum MPI was previously branded as IBM Platform MPI. IBM Spectrum MPI delivers an OpenMPI implementation for HPC parallel applications with improved performance and scalability.

IBM Spectrum MPI Version 10.2 incorporates advanced CPU affinity features, dynamic selection ofnetwork interface libraries, superior workload manager integrations, and improved performance. Itsupports a broad range of industry-standard platforms, interconnects, and open APIs to help ensure thatparallel applications can run almost anywhere.

IBM Spectrum MPI Version 10.2 delivers an improved, RDMA-capable Parallel Active Messaging Interface(PAMI) using Mellanox OFED on both POWER8® and POWER9™ systems in Little Endian mode. It alsooffers an improved collective MPI library that supports the seamless use of GPU memory buffers for theapplication developer. The library provides advanced logic to select the fastest algorithm of manyimplementations for each MPI collective operation.

© Copyright IBM Corp. 2018 1

2 IBM Spectrum MPI Version 10.2: Release Notes

Features

The following features are provided by IBM Spectrum MPI Version 10.2:64-bit support

IBM Spectrum MPI can be used on 64-bit in Little Endian mode for IBM Power Systems™ servers(8335-GCA and 8335-GTA), with and without GPUs.

Thread safetyThe MPI_THREAD_MULTIPLE (multiple threads that are run within the MPI library) option is fullysupported.

GPU support

IBM Spectrum MPI supports the following GPU-acceleration technologies:

• NVIDIA GPUDirect RDMA on POWER9 systems• CUDA-aware MPI on POWER8 and POWER9

By default, GPU support is turned off. To turn on GPU support for GPUDirect RDMA, you can run thempirun -gpu command. On POWER8 systems, due to hardware limitations, you must disableGPUDirect RDMA and turn on CUDS-aware support by running the mpirun -gpu -disable_gdrcommand. Alternatively, you can specify OMPI_MCA_pml_pami_disable_gdr = 1 andOMPI_MCA_common_pami_disable_gdr = 1 in the /opt/ibm/spectrum_mpi/etc/smpi.conffile to disable GPUDirect RDMA. Setting these options in the /opt/ibm/spectrum_mpi/etc/smpi.conf file is especially useful on POWER8 systems.

Note: To use the mpirun -gpu, you must have the NVIDIA drivers 396.19, or later, installed. Tocheck the version of the NVIDIA drives that are installed, run the cat /proc/driver/nvidia/version command. The output that is displayed when you run this command is similar to thefollowing example.

NVRM version: NVIDIA UNIX ppc64le Kernel Module 396.19 Sun Apr 8 01:06:36 PDT 2018GCC version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)

To downloaded the latest NVIDIA drivers, see the NVIDIA Drivers website.

You must use the CUDA Toolkit version 9.2, or later.

IBM Platform LSFIBM Spectrum MPI supports the IBM Platform Load Sharing Facility (LSF) version 9.1.3 for startingjobs. For more information, see the Running applications with IBM Platform LSF topic.

Debugger supportIBM Spectrum MPI supports the Allinea DDT and TotalView debuggers. For more information, see theDebugging applications topic.

PMIxIBM Spectrum MPI supports and redistributes PMI Exascale (PMIx) version 2.0.0. PMIx extends thePMI standard, including the existing PMI-1 and PMI-2 APIs, to support clusters of exascale size. Formore information about PMIx, see the PMIx Programmer's Manual.

FCA (hcoll) support

For installations that use the InfiniBand interconnect, the Mellanox Fabric Collective Accelerator(FCA), which uses Core-Direct technology, can be used to accelerate collective operations. FCA is alsoknown as hcoll. hcoll 4.0, or later, is required. For more information, see the Mellanox HPC-X SoftwareToolkit website.

© Copyright IBM Corp. 2018 3

Portable Hardware Locality (hwloc)

IBM Spectrum MPI uses hwloc (Portable Hardware Locality), which is an API that navigates thehardware topology of your server. You can view an abbreviated image of the server's hardware byusing the --report-bindings option to the mpirun command. For example:

% mpirun -np 1 --report-bindings ./any_mpi_program.x [ibmgpu01:27613] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..]

In the following example, the end of the output line indicates that the server has two sockets, eachwith eight cores, and that each core has two hyper-threads. This output also shows that the startedMPI process is bound to the first socket:

[BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..]

hwloc provides IBM Spectrum MPI with details about NUMA memory nodes, sockets, shared caches,cores and simultaneous multithreading, system attributes, and the location of I/O devices. You canuse this information to place processes, and the memory associated with them, most efficiently, andfor best performance.

IBM Spectrum MPI includes hwloc version 1.11. For more information about hwloc, see the Open MPIwebsite.

MPI-IO

MPI has a number of subroutines that enable your application program to perform efficient parallelinput/output (I/O) operations. These subroutines (collectively referred to as MPI-IO) allow efficientfile I/O on a data structure that is distributed across several tasks for computation, but is organized ina unified way in a single underlying file. MPI-IO presupposes a single parallel file system thatunderlies all the tasks in the parallel job. For IBM Spectrum MPI, this parallel file system is IBMSpectrum Scale™ version 4.2.1.

For parallel I/O, IBM Spectrum MPI supports ROMIO version 3.1.4 and ROMIO 3.2.1, as a non-defaultsupported option. To understand how ROMIO was built, use the ompi_info command, with thehighest level of verbosity. For example:

$ MPI_ROOT/bin/ompi_info -l 9 --param io romio314 MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v10.1.0) MCA io romio314: --------------------------------------------------- MCA io romio314: parameter "io_romio314_priority" (current value: "40", data source: default, level: 9 dev/all, type: int) Priority of the io romio component MCA io romio314: parameter "io_romio314_delete_priority" (current value: "40", data source: default, level: 9 dev/all, type: int) Delete priority of the io romio component MCA io romio314: informational "io_romio314_version" (current value: "from MPICH v3.1.4", data source: default, level: 9 dev/all, type: string) Version of ROMIO MCA io romio314: informational "io_romio314_user_configure_params" (current value: "", data source: default, level: 9 dev/all, type: string) User-specified command line parameters passed to ROMIO's configure script MCA io romio314: informational "io_romio314_complete_configure_params" (current value: " FROM_OMPI=yes CC='gcc -std=gnu99' CFLAGS='-DNDEBUG -m64 -O3 -Wall -Wundef -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic -Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing -pthread

--disable-aio --disable-weak-symbols --enable-strict", data source: default, level: 9 dev/all, type: string)

4 IBM Spectrum MPI Version 10.2: Release Notes

Complete set of command line parameters passed to ROMIO's configure scrip

In Spectrum MPI v10.2.0.10, ROMIO 3.2.1 was added as another optional MPI-IO implementation. Torequest ROMIO 3.2.1, users must specify -mca io romio321. ROMIO 3.2.1 includes improved MPI-IO implementations when running on Spectrum Scale partitions. Users must specify the followinghints to the ROMIO 3.2.1 system by creating a my_romio321_hints.txt file with the followingcontent:

romio_cb_write enableromio_ds_write enablecb_buffer_size 16777216 cb_nodes <#nodes>

After creating the my_romio321_hints.txt file, users must pass the hints file path to SMPI byusing the ROMIO_HINTS environment variable. For example, ROMIO_HINTS=/path/to/my_romio321_hints.txt.

Features 5

6 IBM Spectrum MPI Version 10.2: Release Notes

Limitations and restrictions

The use of MPI_Issend to SELF followed by MPI_Cancel in HWTM mode is restricted.

Adapter affinity is enabled by default, when you are running MPI jobs over PAMI. This usually results inbetter performance when CPU binding is enabled (the default setting), and instructs each rank to use theInfiniBand adapter that is physically closest to the core where the rank is bound. However, adapter affinitycan lead to either a segmentation violation (SEGV) when you are using -verbsbypass, or a job thathangs in the following circumstances:

• If a user runs a job across nodes that have different numbers of InfiniBand adapters per node (forexample, if some nodes have two adapters, and other nodes have only one adapter).

• If a user runs a job across nodes that have one InfiniBand adapter on one fabric, and another adapter ona different fabric.

In either of these situations, you must disable adapter affinity by specifying the following option with thempirun option.

PAMI_IBV_ADAPTER_AFFINITY=0

Adapter striping is also enabled by default, when you are running MPI jobs over PAMI. However, bothadapter affinity and adapter striping must be disabled before a user runs a job across a set of nodes inwhich one node contains a single link only. In this situation, to disable both adapter affinity and adapterstriping, specify the following environment variables with the mpirun option.

PAMI_ENABLE_STRIPING=0PAMI_IBV_DEVICE_NAME=mlx5_0:1PAMI_IBV_ADAPTER_AFFINITY=0

IBM Spectrum MPI Version 10.2 supports OpenSHMEM. For more information about OpenSHMEM, seethe OpenSHMEM website.

Multithread File I/O is not supported.

By default, PAMI creates InfiniBand connections by using Reliably Connected (RC) queue pairs for eachdestination rank. When the MPI_Finalize API is called, these queue pairs are removed by PAMI. As jobsizes grow, the number of queue pairs also grow, which requires more time in the MPI_Finalize API.

If a job is running on a single compute node, the -pami_noib flag to the mpirun command instructs IBMSpectrum MPI to use a shared memory only version of PAMI. If a job is run on more than one computenode, users can direct PAMI to create the InfiniBand connections for the node by using DynamicallyConnected (DC) queue pairs. To force DC queue pairs, you must add the -x PAMI_IBV_ENABLE_DCT=1flag to the mpirun command. By default, PAMI switches from its default RC to DC queue pairs at certaintask geometries. Directing PAMI to use DC queue pairs at other geometries might impact latency.

The following general limitation and restrictions apply to IBM Spectrum MPI Version 10.2:

• IBM Spectrum MPI is not Application Binary Interface (ABI) compatible with any other MPIimplementations such as Open MPI, Platform MPI, or IBM PE Runtime Edition.

• The IBM Spectrum MPI collectives component (libcollectives or coll_ibm) does not supportintercommunicators. For intercommunicator collective support, IBM Spectrum MPI relies on Open MPIcollective components (coll_inter).

• The SMPI health checks (daxpy, CPU dgemm, GPU dgemm, jlink), that are used to check memorybandwidth, CPU and GPU throughput, and network bandwidth on a quiet cluster, do not support coreisolation. Therefore, when you are running SMPI health checks, do not pass the -core_isolationparameter to the LSF bsub command.

• Do not modify the OMPI_MCA_coll_tuned_priority = -1 parameter in the /opt/ibm/spectrum_mpi/etc/smpi.conf file.

© Copyright IBM Corp. 2018 7

• IBM Spectrum MPI requires Mellanox OFED 4.3-4.0.5.x

Note: Users can override values that are specified in the /opt/ibm/spectrum_mpi/etc/smpi.conffiles by setting the values in your environment where you run the mpirun command. You can alsooverride values that are specified in the /opt/ibm/spectrum_mpi/etc/smpi.conf files from thempirun command line by running the mpirun -x <var>=<value> command.

If administrators want to change these parameters, they can modify the values in the /opt/ibm/spectrum_mpi/etc/smpi.conf file.

• To prevent data integrity problems when you are writing to a file, you must use General Parallel FileSystem (GPFS™) version 4.2.3.7, or later, or version 5.0.0.0, or later.

• The default of HCOLL Allreduce behavior prevents auto rank reordering. This behavior can improve thereproducibility for certain HCOLL Allreduce calls.

• The MPI_Comm_spawn API has reserved keys. When you are calling the MPI_Comm_spawn API, thepath key and the soft key are ignored in Open MPI 3.0 and in IBM Spectrum MPI.

• Dynamic process management (dynamic tasking) is supported on only IBM Power® System servers. Torun applications with dynamic tasking, you must run the mpirun -pami_dt command. When you runthe mpirun -pami_dt command, a different PAMI library is used that supports dynamic tasking buthas a lower performance for non-dynamic tasking communication operations. The MPI_Comm_acceptand MPI_Comm_connect APIs are not supported by dynamic tasking.

If you run applications which use dynamic tasking, you must disable the IBM collectives library by usingthe -mca coll ^IBM option.

If you launch jobs with the mpirun command without LSF, you must specify the launch node in the hostlist file or in the -host parameter. If you do not want to run processes on the launch node, you canspecify slots=0 in the host list file or specify launchnodename:0 for the -host parameter. When youmake these changes to the host list file or to the -host parameter, the value of theMPI_UNIVERSE_SIZE attribute is consistent with the number of processors that are available forrunning jobs that are started with the MPI_Comm_spawn API.

Dynamic tasking in IBM Spectrum MPI is supported similar to the OpenMPI implementation. For moreinformation about keys that are supported, see the MPI_Comm_spawn man page.

• If you are running the ompi_info command, you might see the following warning message:mca_base_component_repository_open: unable to open mca_coll_hcoll: libhcoll.so.1: cannot open shared object file: No such file or directory (ignored)

To avoid this warning message and view additional HCOLL related output, run the ompi_info -HCOLLcommand.

• If you launch many processes per node, the local daemon might exhaust the file descriptor limit on thenode. Administrators can increase the file descriptor limit by running the ulimit -n VALUEcommand, where VALUE is the upper limit of file descriptors you want to define per-process on thenode. You must set the VALUE option on all compute nodes in the system. You can estimate the numberof file descriptors that are needed by using the following formula: 25 + (7 x PPN), where PPN is thenumber of processes that are launched on a single node. For example, if you had eight processes thatare launched on a single node, an approximate value for the VALUE option would be 81 (25+7x8). In thisexample, to set the file descriptor limit to 81, you can run the ulimit -n 81 command.

• If you are using nvprof, the NVIDIA profiler tool, you must use the -restrict_libs none optionwhen you run the mpirun command to avoid unnecessary fork() system calls that can lead to incorrectanswers from memory that is not being pinned as expected. This process is the default behavior.

• The default Parallel Active Messaging Interface (PAMI) settings for multi-host support on POWER9systems, might not be optimal for bandwidth sensitive applications. You can use the following settingsto achieve optimal node aggregate bandwidth:

PAMI_IBV_DEVICE_NAME=mlx5_0:1PAMI_IBV_DEVICE_NAME_1=mlx5_3:1

8 IBM Spectrum MPI Version 10.2: Release Notes

In this example, the settings configure tasks on socket 0 to use HCA port 1 and tasks on socket 1 to useHCA port 2.

• The host list file is a flat text file that contains the names of the hosts on which your applications run.Each host is included on a separate line. For example, the following example shows the contents of asimple host list file called myhosts:

node1.mydomain.comnode2.mydomain.comnode3.mydomain.comnode4.mydomain.com

In a host list file, the order of the nodes in the file is not preserved when you launch processes acrossresources. In the previous example, the host list that is named myhosts and thenode1.mydomain.com entry might not be the first node used, even though it is listed first in the hostlist file. For example, the following might be the order in which the nodes are used:

1. node3.mydomain.com2. node2.mydomain.com3. node1.mydomain.com4. node4.mydomain.com

• A failure might occur when many threads (seven or more) call the MPI_Intercomm_create API.• The use of the MPI_File_set_atomicity() API call is not supported.• While creating n-dimensional topologies by using the MPI_Dims_create() API calls, the ndims value

must be greater than 0.• While running Spectrum MPI over TCP on nodes with a virtual adapter, users must specify the correct

adapter that must be used by using the -netaddr option, because Spectrum MPI does not ignore thevirbr# named devices and the TCP BTL tries to stripe across all devices seen.

• Multithreaded File I/O is not supported.• At the time of this release, the MPI standard has some ambiguity about the meaning of theMPI_Comm_get_info(), MPI_Win_get_info(), and the MPI_File_get_info() APIs. These APIsreturn the current internal value of each info setting, which can differ from values that are provided in aprevious version of these APIs. In the future, the MPI standard is likely to be changed to clarify that theget calls should return the same values that were set. So, the current behavior of these APIs is notcompliant with the expected clarification of the MPI standard.

• If your switch network topology consists of more than one InfiniBand network, IBM Spectrum MPIrequires a unique subnet prefix (also known as the network ID) is assigned to each network. Ifdisconnected networks are assigned the same network ID, jobs might not run.

• IBM Spectrum MPI's use of the libevent v2.0.22 library might cause a perturbation in the randomnumber stream generated by the libc rand() and random() functions. Applications that relydeterministic streams of random numbers should consider using the rand_r() and random_r() APIs.

• Querying or attempting to write the value of a performance variable by using the MPI Tool informationinterface (MPI_T) is not supported.

• Fortran MPI applications require the libgfortran3 runtime library to be installed on each computenode, regardless of which Fortran compiler was used to compile and link the application. This is arequirement because the libmpi_ibm_usempi.so.3 library depends on the libgfortran.so.3library.

• On a node with GPUs, it is recommended that you run the following commands shortly after the nodeboots and before you run any GPU workload:

nvidia-smi -pm 1 # persistence modenvidia-modprobe -u -c=0 # pre-load uvm

• The Open MPI collectives components that are included with IBM Spectrum MPI do not support GPUbuffers. For GPU buffer collective support, you must use the libcollectives library (the default library)through the coll_IBM collective component.

Limitations and restrictions 9

• Using -async with multiple PAMI contexts and transferring large messages (rendezvous protocolutilizing the PAMI_Rget API). PAMI occasionally fails with an assertion when destroying the PAMImemory region.

• HCOLL collectives may hang in multi-threaded MPI applications with a large number of communicatorcreations.

• HCOLLs MPI_Reduce may segfault when used with multiple threads. As a workaround, restrict the useof HCOLLMPI_Reduce when running multi-threaded by setting the environment variable toHCOLL_ML_DISABLE_REDUCE = 1.

• MPI_Iscatter may hang when using HCOLL. As a workaround, either add --async to the mpiruncommand line, or set PAMI_IBV_ENABLE_DCT=1 in the job environment.

GPU limitations and restrictionsConsider the following GPU limitations and restrictions:

• When MPI-IO shared file pointers are used with CUDA buffers, the use of the -async feature isrestricted.

• The use of PAMI_DISABLE_IPC=0 is restricted to applications that do not use managed memory.• The GPU-accelerated applications are supported only if you are using the IBM Spectrum MPI PAMI

backend and the IBM collective library (libcollectives). These are the default options for IBM SpectrumMPI, but if you choose to use a different messaging protocol or collective component, GPUs are notsupported.

• The Open MPI collectives components that are included with IBM Spectrum MPI do not support GPUbuffers. For GPU buffer support, you must use libcollectives library (the default option).

• The behavior of IBM Spectrum MPI is undefined when an MPI application passes a GPU device addressto an MPI API that is not CUDA-aware; the application might trigger a segmentation fault and dumpcore, or it might give incorrect results.

Most MPI APIs are CUDS-aware. However, IBM Spectrum MPI does not supply header files or Fortranmodule files that identify which MPI APIs accept GPU device addresses as arguments. Refer to thepreceding restriction for a list of MPI APIs that might not be used with GPU device addresses.

• On a node with GPUs, it is recommended that you run the following commands immediately after thenode boots and before running any GPU workload:

nvidia-smi -pm 1 # persistence modenvidia-modprobe -u -c=0 # pre-load uvm

• The libcollectives component is the only collective component that can support CUDA buffers. As aresult, when the -gpu option is specified with the mpirun command, the libcollectives library must bethe preferred collective component. Therefore, you cannot use the mpirun -gpu command with any ofthe following options:

– -mxm/-MXM– -mxmc/-MXMC– -tcp/-TCP– -ibv/-IBV– -ib/-IB– -openib/-OPENIB– -FCA– -HCOLL

Note: You can use the -gpu option with the -hcoll option. In this run mode, coll_IBM is a higher prioritythan coll, and might do the pug to host transfer before passing the new host buffer to the hcoll library.

10 IBM Spectrum MPI Version 10.2: Release Notes

NVIDIA CUDA limitations and restrictionsConsider the following NVIDIA CUDA limitations and restrictions:

• If an application is built by using the NVIDIA CUDA Toolkit, the NVIDIA CUDA Toolkit must be installedon the node from which it is launched, and on each compute node. You must use CUDA Toolkit version9.2.

• The following MPI functions are not CUDA-aware:

– MPI_Alltoallw– MPI_Ialltoallw– MPI_Ineighbor_allgather– MPI_Ineighbor_allgatherv– MPI_Ineighbor_alltoall– MPI_Ineighbor_alltoallv– MPI_Ineighbor_alltoallw– MPI_Neighbor_allgather– MPI_Neighbor_allgatherv– MPI_Neighbor_alltoall– MPI_Neighbor_alltoallv– MPI_Neighbor_alltoallw

• If you are using the IBM XL Fortran compiler to build CUDA-aware MPI programs that employ thepartitioned global address space (PGAS) model, you must use XL compiler version 15.1.5, or later.

Limitations and restrictions 11

12 IBM Spectrum MPI Version 10.2: Release Notes

Documentation

To view the most recent IBM Spectrum MPI documentation in the PDF and HTML format, refer to the IBMSpectrum MPI Knowledge Center website.

© Copyright IBM Corp. 2018 13

14 IBM Spectrum MPI Version 10.2: Release Notes

Accessibility features for IBM Spectrum MPI

Accessibility features assist users who have a disability, such as restricted mobility or limited vision, touse information technology content successfully.

Accessibility features

IBM Spectrum MPI includes the following major accessibility features:

• Keyboard-only operation• Operations that use a screen reader

IBM Spectrum MPI uses the latest W3C Standard, WAI-ARIA 1.0 (www.w3.org/TR/wai-aria/), to ensurecompliance with US Section 508 (www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/section-508-standards) and Web Content AccessibilityGuidelines (WCAG) 2.0 (www.w3.org/TR/WCAG20/). To take advantage of accessibility features, use thelatest release of your screen reader and the latest web browser that is supported by IBM Spectrum MPI.

The IBM Spectrum MPI online product documentation in IBM Knowledge Center is enabled foraccessibility. The accessibility features of IBM Knowledge Center are described at http://www.ibm.com/support/knowledgecenter/doc/kc_help.html#accessibility.

Related accessibility information

In addition to standard IBM help desk and support websites, IBM has established a TTY telephoneservice for use by deaf or hard of hearing customers to access sales and support services:

TTY service 800-IBM-3383 (800-426-3383) (within North America)

IBM and accessibility

For more information about the commitment that IBM has to accessibility, see IBM Accessibility(www.ibm.com/able).

© Copyright IBM Corp. 2018 15

16 IBM Spectrum MPI Version 10.2: Release Notes

Notices

This information was developed for products and services offered in the US.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply that onlythat IBM product, program, or service may be used. Any functionally equivalent product, program, orservice that does not infringe any IBM intellectual property right may be used instead. However, it is theuser's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document.The furnishing of this document does not grant you any license to these patents. You can send licenseinquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

For license inquiries regarding double-byte character set (DBCS) information, contact the IBM IntellectualProperty Department in your country or send inquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR APARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not inany manner serve as an endorsement of those websites. The materials at those websites are not part ofthe materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate withoutincurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

© Copyright IBM Corp. 2018 17

The licensed program described in this document and all licensed material available for it are provided byIBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or anyequivalent agreement between us.

The performance data and client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice,and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subject to change withoutnotice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject to change before theproducts described become available.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to actual people or business enterprises isentirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programsin any form without payment to IBM, for the purposes of developing, using, marketing or distributingapplication programs conforming to the application programming interface for the operating platform forwhich the sample programs are written. These examples have not been thoroughly tested under allconditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of theseprograms. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work must include a copyright noticeas follows:© (your company name) (year).

Portions of this code are derived from IBM Corp. Sample Programs.© Copyright IBM Corp. _enter the year or years_.

Programming interface information

MPI support statementThe IBM Spectrum MPI product is a complete MPI implementation, based on the Open MPI open sourceproject, designed to comply with all the requirements of the Message Passing Interface standard, MPI: AMessage-Passing Interface Standard, Version 3.1, University of Tennessee, Knoxville, Tennessee, June 4,2015. If you believe that IBM Spectrum MPI does not comply with the MPI-3.1 standard, please contactIBM Service.

Open MPI licenseMost files in this release are marked with the copyrights of theorganizations who have edited them. The copyrights below are in noparticular order and generally reflect members of the Open MPI coreteam who have contributed code to this release. The copyrights forcode used under license from other parties are included in the

18 Notices

corresponding files.

Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana University Research and Technology Corporation. All rights reserved.Copyright (c) 2004-2010 The University of Tennessee and The University of Tennessee Research Foundation. All rights reserved.Copyright (c) 2004-2010 High Performance Computing Center Stuttgart, University of Stuttgart. All rights reserved.Copyright (c) 2004-2008 The Regents of the University of California. All rights reserved.Copyright (c) 2006-2010 Los Alamos National Security, LLC. All rights reserved.Copyright (c) 2006-2010 Cisco Systems, Inc. All rights reserved.Copyright (c) 2006-2010 Voltaire, Inc. All rights reserved.Copyright (c) 2006-2011 Sandia National Laboratories. All rights reserved.Copyright (c) 2006-2010 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms.Copyright (c) 2006-2010 The University of Houston. All rights reserved.Copyright (c) 2006-2009 Myricom, Inc. All rights reserved.Copyright (c) 2007-2008 UT-Battelle, LLC. All rights reserved.Copyright (c) 2007-2018 IBM Corporation. All rights reserved.Copyright (c) 1998-2005 Forschungszentrum Juelich, Juelich Supercomputing Centre, Federal Republic of GermanyCopyright (c) 2005-2008 ZIH, TU Dresden, Federal Republic of GermanyCopyright (c) 2007 Evergrid, Inc. All rights reserved.Copyright (c) 2008 Chelsio, Inc. All rights reserved.Copyright (c) 2008-2009 Institut National de Recherche en Informatique. All rights reserved.Copyright (c) 2007 Lawrence Livermore National Security, LLC. All rights reserved.Copyright (c) 2007-2009 Mellanox Technologies. All rights reserved.Copyright (c) 2006-2010 QLogic Corporation. All rights reserved.Copyright (c) 2008-2010 Oak Ridge National Labs. All rights reserved.Copyright (c) 2006-2010 Oracle and/or its affiliates. All rights reserved.Copyright (c) 2009 Bull SAS. All rights reserved.Copyright (c) 2010 ARM ltd. All rights reserved.Copyright (c) 2010-2011 Alex Brick . All rights reserved.Copyright (c) 2012 The University of Wisconsin-La Crosse. All rights reserved.Copyright (c) 2013-2016 Intel, Inc. All rights reserved.Copyright (c) 2011-2014 NVIDIA Corporation. All rights reserved.

$COPYRIGHT$

Additional copyrights may follow

$HEADER$

Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions aremet:

- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer listed in this license in the documentation and/or other materials provided with the distribution.

- Neither the name of the copyright holders nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

The copyright holders provide no reassurances that the source codeprovided does not infringe any patent, copyright, or any otherintellectual property rights of third parties. The copyright holdersdisclaim any liability to any recipient for claims brought againstrecipient by any third party for infringement of that partiesintellectual property rights.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT

Notices 19

(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International BusinessMachines Corp., registered in many jurisdictions worldwide. Other product and service names might betrademarks of IBM or other companies. A current list of IBM trademarks is available on the web atCopyright and trademark information at www.ibm.com/legal/copytrade.shtml.

Terms and conditionsPermissions for the use of these publications are granted subject to the following terms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBM website.

Personal use

You may reproduce these publications for your personal, noncommercial use provided that all proprietarynotices are preserved. You may not distribute, display or make derivative work of these publications, orany portion thereof, without the express consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within your enterprise provided thatall proprietary notices are preserved. You may not make derivative works of these publications, orreproduce, distribute or display these publications or any portion thereof outside your enterprise, withoutthe express consent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses or rights are granted, eitherexpress or implied, to the publications or any information, data, software or other intellectual propertycontained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in its discretion, the use ofthe publications is detrimental to its interest or, as determined by IBM, the above instructions are notbeing properly followed.

You may not download, export or re-export this information except in full compliance with all applicablelaws and regulations, including all United States export laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESE PUBLICATIONS. THE PUBLICATIONS AREPROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT,AND FITNESS FOR A PARTICULAR PURPOSE.

Privacy policy considerationsIBM Software products, including software as a service solutions, ("Software Offerings") may use cookiesor other technologies to collect product usage information, to help improve the end user experience, totailor interactions with the end user or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offerings can help enable you tocollect personally identifiable information. If this Software Offering uses cookies to collect personallyidentifiable information, specific information about this offering's use of cookies is set forth below.

20 Notices

This Software Offering does not use cookies or other technologies to collect personally identifiableinformation.

If the configurations deployed for this Software Offering provide you as the customer the ability to collectpersonally identifiable information from end users via cookies and other technologies, you should seekyour own legal advice about any laws applicable to such data collection, including any requirements fornotice and consent.

For more information about the use of various technologies, including cookies, for these purposes, seeIBM's Privacy Policy at http://www.ibm.com/privacy and IBM's Online Privacy Statement at http://www.ibm.com/privacy/details the section entitled "Cookies, Web Beacons and Other Technologies" andthe "IBM Software Products and Software-as-a-Service Privacy Statement" at http://www.ibm.com/software/info/product-privacy.

Notices 21

22 IBM Spectrum MPI Version 10.2: Release Notes

Index

Aaccessibility features for this product 15

Cconventions and terminology v

23

24

IBM®