Distributed GPU Training with NCCL
The NVIDIA Collective Communications Library (NCCL) is the communication layer that lets multiple GPUs exchange data during distributed jobs – the GPU counterpart to MPI. It is the backbone of multi-GPU AI training.
NCCL is a library, not a process launcher: you do not wire it up directly.
Your framework – almost always PyTorch – uses NCCL internally, and a launcher
such as torchrun starts the processes and assigns ranks. Because NCCL is not a
wire-up mechanism, Fuzzball has no dedicated NCCL multinode.implementation.
Instead you run these workloads with the
generic implementation,
which hands your launcher the per-node rank and head-node address it needs.
A single node with several GPUs needs no multinode block at all. torchrun --standalone launches one process per GPU on the node, and PyTorch uses NCCL to
communicate between them:
version: v4
# Single-node, multi-GPU PyTorch/NCCL run.
#
# A single node with multiple GPUs does NOT need the multinode block: torchrun
# --standalone launches one process per GPU, and PyTorch uses NCCL to
# communicate between them. Fuzzball sets RLIMIT_MEMLOCK to the job's memory
# automatically, which covers NCCL's pinned buffers.
jobs:
torch-allreduce:
image:
uri: docker://nvcr.io/nvidia/pytorch:25.09-py3
env:
- NCCL_DEBUG=INFO
script: |
#!/bin/sh
set -eux
cat > /tmp/allreduce.py <<'PY'
import os, torch
import torch.distributed as dist
dist.init_process_group("nccl")
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
t = torch.ones(1, device="cuda") * (dist.get_rank() + 1)
dist.all_reduce(t)
print(f"rank {dist.get_rank()}/{dist.get_world_size()} sum={t.item()}", flush=True)
dist.destroy_process_group()
PY
# --nproc-per-node must match the GPU count requested below.
torchrun --standalone --nproc-per-node=2 /tmp/allreduce.py
resource:
cpu:
cores: 2
affinity: NUMA
memory:
size: 16GB
devices:
nvidia.com/gpu: 2
Set --nproc-per-node to the GPU count. Fuzzball automatically raises the job’s
locked-memory limit (RLIMIT_MEMLOCK) to its memory allocation, which covers
NCCL’s pinned communication buffers.
To span nodes, use the generic implementation. It exposes the per-node rank as
$RANK and the head node’s address as $MULTINODE_NODE_IP – exactly
torchrun’s --node-rank and --master-addr. The exec_all helper runs
torchrun on every node:
version: v4
# Multi-node PyTorch/NCCL training using the `generic` multinode implementation.
#
# NCCL is not a launcher, so there is no dedicated NCCL implementation. The
# `generic` implementation hands torchrun the per-node rank ($RANK) and the head
# node address ($MULTINODE_NODE_IP) it needs; PyTorch then uses NCCL underneath.
jobs:
torch-train:
image:
uri: docker://nvcr.io/nvidia/pytorch:25.09-py3
env:
- NCCL_DEBUG=INFO
# For InfiniBand/RoCE, pin the fabric so NCCL does not fall back to TCP
# (find the device name with `ibv_devinfo` on a node):
# - NCCL_IB_HCA=mlx5
script: |
#!/bin/sh
. /multinode/generic-helper
# Number of nodes = entries in the hostlist.
NNODES=$(echo "$MULTINODE_HOSTLIST_NOSLOTS" | awk -F, '{print NF}')
# Run torchrun on every node. Each node takes its --node-rank from $RANK;
# all rendezvous at the head node ($1 == MULTINODE_NODE_IP). Replace
# /workspace/train.py with your training script.
exec_all - "$MULTINODE_NODE_IP" "$NNODES" <<'EOF'
torchrun \
--nnodes="$2" \
--node-rank="$RANK" \
--nproc-per-node=2 \
--master-addr="$1" \
--master-port=29500 \
/workspace/train.py
EOF
resource:
cpu:
cores: 2
affinity: NUMA
memory:
size: 32GB
devices:
nvidia.com/gpu: 2 # per node; --nproc-per-node must match
multinode:
nodes: 2
implementation: generic
Each node launches --nproc-per-node processes (one per GPU) and rendezvouses at
the head node; PyTorch coordinates the rest through NCCL. See
Distributed Workflows with Generic Multinode
for the full generic environment and helper-function reference.
Because Fuzzball runs jobs as an unprivileged user, everything must be baked into the image – you cannot install packages at runtime:
- CUDA, NCCL, and your framework (e.g. PyTorch). The NGC PyTorch images bundle
all three, plus
torchrun. - For InfiniBand/RoCE, the NCCL network plugin plus
libibverbs/libfabricmatching your cluster’s fabric.
The GPU code in the image must be built for your GPU’s architecture. A CUDA
toolkit older than the GPU fails at kernel launch with Cuda failure 'invalid argument'.
On a fabric-equipped cluster, pin the device so NCCL uses RDMA instead of falling back to TCP:
env:
- NCCL_IB_HCA=mlx5
If NCCL cannot find a high-speed device it falls back to sockets, visible in the
job log as NET/IB : No device found followed by Using network Socket.
Fuzzball exposes /dev/infiniband to multinode containers automatically and
raises the locked-memory limit RDMA pinning needs; the fabric libraries must be
present in your container, and the interconnect provisioned on the nodes.
On systems with Multi-Node NVLink (MNNVL), the GPUs of several nodes share one
NVLink domain, and NCCL can use NVLink between those nodes instead of the
network. For that, each node’s container needs an IMEX
channel. Ask for one by setting NVIDIA_IMEX_CHANNELS in the job’s env, and
pin the job to the MNNVL nodes:
env:
- NVIDIA_IMEX_CHANNELS=0
resource:
devices:
nvidia.com/gpu: 4 # per node; --nproc-per-node must match
annotations:
nvidia.com/gpu.product: GB200 # a GPU model only the MNNVL nodes have
multinode:
nodes: 2
implementation: generic
The rest of the job is the same as the multinode example above. Fuzzball adds
/dev/nvidia-caps-imex-channels/channel0 to the container on every node, and
places the job’s nodes inside one NVLink domain whenever one has enough free
nodes. When none does, the job spans domains and NCCL uses the network between
them. The nvidia.com/gpu.product annotation matches any product name that
contains its value; without a pin like it, a job that lands on nodes without
IMEX fails when its containers are created.
This needs:
- IMEX set up on the nodes by your administrator; see Multi-Node NVLink.
- An NCCL release in the image that supports Multi-Node NVLink.
On an MNNVL system without IMEX, NCCL 2.27.3 and later stop at startup instead of falling back to the network, because they cannot set up NVLink between nodes. SetNCCL_MNNVL_ENABLE=0in the job’senvto run over the network instead.
If the job fails at startup:
Cuda failure 800 'operation not permitted'means the job cannot write to the IMEX channel. Ask your administrator to check its permissions.- An error that mentions MNNVL means IMEX is missing or not running on the nodes.
Every job that uses channel 0 in an NVLink domain shares it, so a job can read GPU memory that another job in the same domain shares through IMEX. Treat each NVLink domain as one trust domain.
For NCCL programs that use MPI to bootstrap (for example NVIDIA’s
nccl-tests), launch them with mpirun on
the generic implementation rather than torchrun. The
generic command example
shows the pattern – your command invokes mpirun -H $MULTINODE_HOSTLIST --mca plm_rsh_agent $MULTINODE_SSH_WRAPPER -np $MULTINODE_TOTAL_SLOTS <program>, and
the program uses NCCL for the actual GPU communication.