OpenMPI and OpenSHMEM installation with UCX

UCX installation

Requirements: Autoconf 2.63 and above.

1. Get latest version of the UCX code

$ git clone https://github.com/openucx/ucx.git ucx
$ cd ucx

2. Run autogen:

$ ./autogen.sh

3. (This step is only required for OpenPOWER platforms) Power 8 On Ubuntu platform the config.guess file is a bit outdated and does not have support for power. In order to resolve the issue you have to download an updated config.guess. From the root of the project:

$ wget https://github.com/shamisp/ucx/raw/topic/power8-config/config.guess

4. Configure:

$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path

NOTE: For best performance configuration, use ../contrib/configure-release.
This will strip all debugging and profiling code.

NOTE: For debug/development build configuration, use ../contrib/configure-devel.
This will enable all debugging and profiling code.

5. Build and install:

$ make
$ make install

Open MPI and OpenSHMEM installation

1. Get latest-and-greatest Open MPI version:

$ git clone https://github.com/open-mpi/ompi.git

2. Autogen:

$ cd ompi
$ ./autogen.pl

3. Configure with UCX:

$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path/ --with-ucx=/path_to_ucx_installation

NOTE With Open MPI 4.0 and above, there could be compilation errors from "btl_uct" component. This component is not critical for using UCX; so it could be disabled this way:

$ ./configure ... --enable-mca-no-build=btl-uct ...

4. Build:

$ make
$ make install

Running Open MPI with UCX

Example of the command line (with optional flag to select IB device mlx5_0 port 1):

$ mpirun -np 2 -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 ./app

IMPORTANT NOTE: Recent Open MPI versions contain a BTL component called 'uct', which can cause data corruption when enabled, due to conflict on malloc hooks between OPAL and UCM. In order to work-around this, use one of the following alternatives:

Alternative 1: Disable btl/uct in Open MPI build configuration:

$ ./configure ... --enable-mca-no-build=btl-uct ...

Alternative 2: Disable btl/uct at runtime

$ mpirun -np 2 -mca pml ucx -mca btl ^uct -x UCX_NET_DEVICES=mlx5_0:1 ./app

Open MPI runtime optimizations for UCX

• By default Open MPI enables build-in transports (BTLs), which may result in additional software overheads in the Open MPI progress function. In order to workaround this issue you may try to disable certain BTLs.

$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 ./app

MPI/SHMEM versions tested with UCX master

1. UCX current tarball: https://github.com/openucx/ucx/archive/master.zip

2. The table of MPI and OpenSHMEM distributions that are tested with the HEAD of UCX master

MPI/OpenSHMEM project
Open MPI/OSHMEM	4.0 and above
MPICH	Latest

Running on Cray systems with Aries and Gemini interconnect

In order to run UCX on Cray systems a user has explicitly enabled UCX unified mode: UCX_UNIFIED_MODE=1. For example:

$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_UNIFIED_MODE=1 ./app