mpi

MPI on Isambard

MPI (Message Passing Interface) enables parallel communication across compute nodes. On Isambard-AI and Isambard 3, MPI must communicate with the Slingshot 11 (SS11) high-speed interconnect via libfabric to achieve optimal latency and bandwidth.

MPI application → libfabric → Slingshot 11 NIC → network

Never use mpirun or mpiexec. Always launch MPI applications with srun so you can supply the correct --mpi flag for the PMI type. Using mpirun/mpiexec bypasses Slurm's process management and will not work correctly.

Critical Rules

• Always start MPI jobs with srun on Isambard.
• Never use mpirun or mpiexec on Isambard.
• Always choose the --mpi value that matches the MPI implementation in use.
• Load MPI through a Cray PrgEnv or a Conda/MPI build that is explicitly configured for Slingshot 11.
• Do not mix raw GCC invocations with Cray compiler wrappers in the same build.

---

Default MPI: Cray MPICH

The recommended MPI is Cray MPICH, provided by the cray-mpich module. It is loaded automatically with any Cray Programming Environment:

module load PrgEnv-gnu    # recommended
# or
module load PrgEnv-cray

After loading, verify the MPI link flags:

# With PrgEnv-gnu
mpicc -show
# gcc -I/opt/cray/pe/mpich/.../ofi/gnu/12.3/include -L.../lib -lmpi_gnu_123

# With PrgEnv-cray
mpicc -show
# craycc -I/opt/cray/pe/mpich/.../ofi/cray/17.0/include -L.../lib -lmpi_cray

The -lmpi_gnu_* or -lmpi_cray library is the Cray MPICH build tuned for Slingshot.

aarch64 known issues: Cray MPICH support for aarch64 was recently added and some environment variable workarounds may be needed. Check the Known Issues page for current advice before running.

---

PMI — Process Manager Interface

MPI processes need a Process Manager Interface (PMI) to coordinate ranks across nodes through Slurm. The correct --mpi flag to srun depends on which MPI library is in use.

List available PMI types on the system:

srun --mpi=list
# cray_shasta
# none
# pmi2
# pmix  (pmix_v4)

Choosing the right --mpi flag

MPI library	--mpi flag	Notes
Cray MPICH (default)	cray_shasta	Default; usually works without specifying
OpenMPI ≤ 4.x	pmi2	Required for older OpenMPI
OpenMPI ≥ 5.0	pmix	Required for modern OpenMPI

# Cray MPICH (cray_shasta is the default, but explicit is clearer)
srun --mpi=cray_shasta ./mpi_app

# OpenMPI ≤ 4.x
srun --mpi=pmi2 ./mpi_app

# OpenMPI ≥ 5.0
srun --mpi=pmix ./mpi_app

If you do not need to specify a custom PMI and are using Cray MPICH, srun without --mpi will use cray_shasta by default.

---

Installing a Different MPI Version

The system Cray MPICH is recommended for performance. If you need a different version (e.g. for software that requires OpenMPI), you have two options.

Option 1: Conda (easiest)

See the Python/Conda guide to set up Miniforge first, then:

# MPICH via conda-forge
conda install -c conda-forge mpich

# OpenMPI via conda-forge
conda install -c conda-forge openmpi

Note: Conda-installed MPI will not use the Slingshot 11 interconnect and will have lower multi-node performance than Cray MPICH. Use only when necessary for software compatibility.

Option 2: Build OpenMPI from source

For maximum control or when Slingshot integration is needed with OpenMPI:

• OpenMPI v5 build instructions
• Legacy OpenMPI (≤4.x) build instructions

mpi4py

For Python MPI, see the Python guide — specifically the section on building mpi4py from source against the Cray MPI stack using module load PrgEnv-gnu and pip install --no-binary mpi4py mpi4py.

---

Quick Reference

Goal	Command / setting
Load default MPI (recommended)	module load PrgEnv-gnu
Check MPI link flags	mpicc -show
Run an MPI job (Cray MPICH)	srun --mpi=cray_shasta ./mpi_app
Run an MPI job (OpenMPI ≤4)	srun --mpi=pmi2 ./mpi_app
Run an MPI job (OpenMPI ≥5)	srun --mpi=pmix ./mpi_app
List available PMI types	srun --mpi=list
Install OpenMPI via conda	conda install -c conda-forge openmpi
Install MPICH via conda	conda install -c conda-forge mpich
Install mpi4py (source, Slingshot-aware)	module load PrgEnv-gnu + pip install --no-binary mpi4py mpi4py
MPI in containers (Podman-HPC)	--openmpi-pmi2 flag + srun --mpi=pmi2
MPI in containers (Apptainer)	module load brics/apptainer-multi-node + source /host/adapt.sh

---

Related Resources

• Isambard Known Issues — check for current Cray MPICH aarch64 workarounds
• Isambard Modules and Compilers guide
• Isambard NCCL guide
• Isambard Containers guide
• Cray Programming Environment docs
• Argonne PMI2 paper