Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Using High-Speed Fabrics with Fuzzball

Fuzzball supports the use of traditional HPC high-speed interconnects like InfiniBand and Omni-Path, as well as more cloud oriented high-speed fabrics like the AWS Elastic Fabric Adapter (EFA). The basic principle is that the appropriate user-space libraries to support the driver must be installed in your container to use high-speed interconnects.

Requesting a Fabric

Nothing in the Fuzzfile requests a fabric. There is no device to name in resource.devices the way a GPU is named, and no annotation to set. If the node has InfiniBand or Omni-Path hardware, Fuzzball bind-mounts /dev/infiniband into the container read-only (nosuid, noexec) and grants the container access to each device under it. The only thing you supply is the matching user-space libraries in the image.

That bind mount happens only for jobs and services that declare a multinode block. It is the multinode plugin that sets it up, and the plugin does nothing for a job without one. A single-node job does not see /dev/infiniband even on a node that has it, so a single-process benchmark cannot be used to check whether the fabric is wired up.

AWS EFA is attached at a different level again: when a provisioner launches an instance whose type supports EFA, Fuzzball requests the EFA interface as the instance is created. Neither the workflow nor the provisioner configuration asks for it. For the EFA instance types whose GPUs are joined by NVLink (p4d.24xlarge, p4de.24xlarge, p5.48xlarge, p5e.48xlarge and p5en.48xlarge), the provisioner also selects an AMI that runs NVIDIA Fabric Manager.

Multi-Node NVLink, where the GPUs of several nodes share one NVLink domain, works differently: the job asks for an IMEX channel in its Fuzzfile. See Using GPUs with Fuzzball for the node setup and Distributed GPU Training with NCCL for the job side.

Container Libraries

Typically, when using Rocky Linux containers, installing the following packages should enable your jobs to use either InfiniBand or Omni-Path.

$ sudo dnf -y groupinstall "InfiniBand Support" \
    && dnf -y install opa-basic-tools libpsm2 libfabric libfabric-devel

Other interconnects may require different packages. For example, EFA installation depends on the type of MPI that you will use with it. See the AWS documentation section titled “Install the EFA software” for more information.

Confirming the Fabric Is Available

A job whose libraries are missing, or whose image was built for a different fabric, usually still runs – the MPI layer falls back to TCP over the ordinary network and the job is simply slower. Because the failure is silent, confirm the fabric from inside a multinode job rather than inferring it from the job succeeding.

Two checks worth running in the job script:

  • Is the device visible? ls /dev/infiniband should list the node’s devices. An empty or missing directory means either the node has no fabric hardware or the job declared no multinode block.
  • Can the libraries see it? fi_info, which ships with libfabric in the package list above, lists the fabric providers available to the container. A run that reports only tcp or sockets providers, with no verbs, psm2, or efa entry, is the case that will fall back.

Consult your MPI implementation’s documentation for how to make it fail loudly instead of falling back, which is worth doing on a first run: both Open MPI and MPICH can be told to require a particular transport.