Using High-Speed Fabrics with Fuzzball
Fuzzball supports the use of traditional HPC high-speed interconnects like InfiniBand and Omni-Path, as well as more cloud oriented high-speed fabrics like the AWS Elastic Fabric Adapter (EFA). The basic principle is that the appropriate user-space libraries to support the driver must be installed in your container to use high-speed interconnects.
Nothing in the Fuzzfile requests a fabric. There is no
device to name in resource.devices the way a GPU is named, and no annotation to set. If the
node has InfiniBand or Omni-Path hardware, Fuzzball bind-mounts /dev/infiniband into the
container read-only (nosuid, noexec) and grants the container access to each device under
it. The only thing you supply is the matching user-space libraries in the image.
That bind mount happens only for jobs and services that declare amultinodeblock. It is the multinode plugin that sets it up, and the plugin does nothing for a job without one. A single-node job does not see/dev/infinibandeven on a node that has it, so a single-process benchmark cannot be used to check whether the fabric is wired up.
AWS EFA is attached at a different level again: when a provisioner launches an instance whose
type supports EFA, Fuzzball requests the EFA interface as the instance is created. Neither the
workflow nor the provisioner configuration
asks for it. For the EFA instance types whose GPUs are joined by NVLink (p4d.24xlarge,
p4de.24xlarge, p5.48xlarge, p5e.48xlarge and p5en.48xlarge), the provisioner also selects
an AMI that runs NVIDIA Fabric Manager.
Multi-Node NVLink, where the GPUs of several nodes share one NVLink domain, works differently: the job asks for an IMEX channel in its Fuzzfile. See Using GPUs with Fuzzball for the node setup and Distributed GPU Training with NCCL for the job side.
Typically, when using Rocky Linux containers, installing the following packages should enable your jobs to use either InfiniBand or Omni-Path.
$ sudo dnf -y groupinstall "InfiniBand Support" \
&& dnf -y install opa-basic-tools libpsm2 libfabric libfabric-develOther interconnects may require different packages. For example, EFA installation depends on the type of MPI that you will use with it. See the AWS documentation section titled “Install the EFA software” for more information.
A job whose libraries are missing, or whose image was built for a different fabric, usually still runs – the MPI layer falls back to TCP over the ordinary network and the job is simply slower. Because the failure is silent, confirm the fabric from inside a multinode job rather than inferring it from the job succeeding.
Two checks worth running in the job script:
- Is the device visible?
ls /dev/infinibandshould list the node’s devices. An empty or missing directory means either the node has no fabric hardware or the job declared nomultinodeblock. - Can the libraries see it?
fi_info, which ships withlibfabricin the package list above, lists the fabric providers available to the container. A run that reports onlytcporsocketsproviders, with noverbs,psm2, orefaentry, is the case that will fall back.
Consult your MPI implementation’s documentation for how to make it fail loudly instead of falling back, which is worth doing on a first run: both Open MPI and MPICH can be told to require a particular transport.