Distributed Workflows with MPI
Fuzzball supports distributed jobs that use MPI.
MPI, or “Message Passing Interface”, is a standard in high performance computing that allows for communication between CPU cores within or between compute nodes. A program written against MPI spreads its work over multiple cores, so adding cores can shorten its runtime. Many scientific applications are designed to make use of MPI to improve their performance on HPC systems. Several different implementations of the MPI standard are available. Fuzzball provides an MPI wrapper that sets up MPI jobs on a given number of nodes with a given MPI implementation. Two implementations of MPI are supported covering the vast majority of use cases: MPICH and OpenMPI.
Here is a basic Fuzzfile showing how to use single node MPI
(OpenMPI) on Fuzzball. This example uses the
container
openmpi-hello-world. To see how this
container was created as well as the mpi_hello_world.c source file, see this GitHub
repo.
version: v4
jobs:
mpi-hello-world:
env:
- PATH=/usr/lib64/openmpi/bin:/usr/local/bin:/usr/bin:/bin
image:
uri: oras://docker.io/anderbubble/openmpi-hello-world.sif
script: |
#!/bin/sh
mpi_hello_world
resource:
cpu:
cores: 4
affinity: NUMA
memory:
size: 1GB
multinode:
nodes: 1
implementation: openmpi
The multinode section lets you specify that multiple copies of your command should be distributed
across cores. The nodes option sets the number of compute nodes. This example is slightly
counterintuitive. MPI runs 4 processes on a single node, because we have requested 1 node with
4 cores. Processes per node default to the allocated core count; set multinode.procs-per-node
to override that (see
Workflow Syntax).
The implementation option sets the type of MPI to use: openmpi (or its alias ompi) for
OpenMPI, and mpich for MPICH. The
section index lists the other implementations
Fuzzball accepts.
The MPI program itself is not executed withmpirun. The contents of thescriptorcommandfield are run withmpirunprefixed to them when themultinodefield is used for MPI-based execution. Running a script here in the multi-node example withmpirunwould fail because the effective command line would end up beingmpirun mpirun hello-mpi 100- andmpiruncannot be called recursively.
After running this job you will see logs like these showing the hostname, rank, CPU number, NUMA node, and mount namespace of each MPI rank in the job.
Hello world! Processor mpi-hello-world-0, Rank 2 of 4, CPU 2, NUMA node 0, Namespace mnt:[4026532665]
Hello world! Processor mpi-hello-world-0, Rank 0 of 4, CPU 3, NUMA node 0, Namespace mnt:[4026532665]
Hello world! Processor mpi-hello-world-0, Rank 1 of 4, CPU 1, NUMA node 0, Namespace mnt:[4026532665]
Hello world! Processor mpi-hello-world-0, Rank 3 of 4, CPU 0, NUMA node 0, Namespace mnt:[4026532665]
A similar Fuzzfile running the same workflow with the MPICH flavor of MPI is presented below:
version: v4
jobs:
mpi-hello-world:
env:
- PATH=/usr/lib64/mpich/bin:/usr/local/bin:/usr/bin:/bin
image:
uri: oras://docker.io/anderbubble/mpich-hello-world.sif
script: |
#!/bin/sh
mpi_hello_world
resource:
cpu:
cores: 4
affinity: NUMA
memory:
size: 1GB
multinode:
nodes: 1
implementation: mpich
An example workflow that uses OpenMPI on multiple nodes appears below:
version: v4
jobs:
mpi-hello-world:
env:
- PATH=/usr/lib64/openmpi/bin:/usr/local/bin:/usr/bin:/bin
image:
uri: oras://docker.io/anderbubble/openmpi-hello-world.sif
script: |
#!/bin/sh
mpi_hello_world
resource:
cpu:
cores: 2
affinity: NUMA
memory:
size: 1GB
multinode:
nodes: 4
implementation: openmpi
In this case, we use the multinode.nodes field to request 4 compute nodes. We also use the
resource section to request 2 cores
and 1GB of RAM per compute node. So in total, this job will claim 8 cores and 4GB of memory
across 4 compute nodes.
The logs from this job look like this:
Hello world! Processor mpi-hello-world-0, Rank 0 of 8, CPU 1, NUMA node 0, Namespace mnt:[4026532667]
Hello world! Processor mpi-hello-world-0, Rank 1 of 8, CPU 0, NUMA node 0, Namespace mnt:[4026532667]
Hello world! Processor mpi-hello-world-3, Rank 6 of 8, CPU 0, NUMA node 0, Namespace mnt:[4026532669]
Hello world! Processor mpi-hello-world-2, Rank 3 of 8, CPU 1, NUMA node 0, Namespace mnt:[4026532667]
Hello world! Processor mpi-hello-world-2, Rank 2 of 8, CPU 0, NUMA node 0, Namespace mnt:[4026532667]
Hello world! Processor mpi-hello-world-3, Rank 7 of 8, CPU 0, NUMA node 0, Namespace mnt:[4026532669]
Hello world! Processor mpi-hello-world-1, Rank 4 of 8, CPU 1, NUMA node 0, Namespace mnt:[4026532667]
Hello world! Processor mpi-hello-world-1, Rank 5 of 8, CPU 0, NUMA node 0, Namespace mnt:[4026532667]
Multinode ranks run on the node’s own network stack, where every node carries the same fb0
container bridge address. Open MPI’s TCP transport would treat that bridge as a network shared
by all ranks and send peer traffic to the local bridge, so a program that exchanges data between
nodes starts and then hangs. Fuzzball therefore sets OMPI_MCA_btl_tcp_if_exclude=lo,fb0 in
every multinode job and service regardless of implementation. Only Open MPI honors this
variable, so MPICH and GASNet workloads that hit the same bridge still need their own interface
selection.
To select interfaces yourself, set OMPI_MCA_btl_tcp_if_exclude or OMPI_MCA_btl_tcp_if_include
in the job’s or service’s env; Fuzzball then leaves its default out. It also leaves the default
out when the command, args or script contains btl_tcp_if_, which covers an
mpirun --mca btl_tcp_if_include option and an exported OMPI_MCA_btl_tcp_if_include. Open MPI
disables its TCP transport when both an include list and an exclude list are set, and ranks then
fail to reach each other. Fuzzball does not detect a selection baked into the container image’s
environment or an mca-params.conf file, so repeat such a selection in env.
Running MPI across a high-speed fabric such as InfiniBand or Omni-Path requires the matching user-space libraries inside the container. See Using High-Speed Fabrics with Fuzzball for the packages each fabric needs.
A multinode job fails as a group: if any rank dies, the remaining ranks are stopped and the job fails. See Rank Failure.