Host Networking for Multinode Services
A service is a container that runs alongside a workflow’s jobs instead of completing like one. This page configures a service that spans several nodes.
Multinode services always run with host networking: every rank runs directly on its node’s
network stack instead of in an isolated network namespace. Fuzzball applies this automatically
to any multinode service that declares a network: block, and a multinode service without one
already runs on the node’s network stack, so network.host: true is never required. A multinode
service cannot opt out: network.host: false is overridden.
Host networking is what lets a distributed service work across nodes: ranks reach each other at node addresses on dynamically negotiated ports, and the workload gets direct access to high-speed interconnects like InfiniBand or RoCE for low-latency, high-bandwidth communication.
A service with host networking shares the node’s network namespace, so it can reach every network interface on the node, including interfaces it would not see in an isolated namespace. Treat the service as running with the node’s own network access.
Declare the multinode block on the service; host networking is applied automatically. The
network.host field may still be set explicitly for clarity:
version: v4
services:
distributed-inference:
image:
uri: docker://myregistry/mpi-inference:latest
script: |
#!/bin/bash
python -m inference_server --model /models/llm
multinode:
nodes: 4
implementation: openmpi
resource:
cpu:
cores: 16
affinity: NUMA
memory:
size: 64GB
devices:
nvidia.com/gpu: 2
persist: true # keep running with nothing depending on it
network:
host: true # optional: implied for multinode services
ports:
- name: http
port: 8080
endpoints:
- name: inference
port-name: http
protocol: https
type: subdomain
scope: group
- network.host: Boolean field to enable host networking. Implied
truefor multinode services; defaultfalseotherwise - multinode.nodes: Number of compute nodes to allocate
- multinode.implementation: Coordination framework (
ompi,openmpi,mpich,gasnet, orgeneric) - resource: Resources per node. A four-node service with 16 cores each is allocated 64 cores in total
- persist: Keeps the service running when nothing depends on it. Without it a service that has no dependents is stopped as soon as it reports ready
In a multinode service:
- Rank 0 is the first node (node 0) and serves as the primary endpoint
- Ranks 1, 2, 3, … are worker nodes that participate in distributed computation
- All network ports and endpoints are exposed only on rank 0
- Clients connect to rank 0, which coordinates with other ranks internally
For example, in a 4-node distributed inference service:
- Rank 0 runs the API server and accepts HTTP requests (rank 0 can also be a worker)
- Ranks 1-3 run workers that process inference requests coordinated by rank 0
- Clients send requests to rank 0’s HTTPS endpoint
version: v4
volumes:
models:
name: vllm-cache
services:
vllm:
cwd: /home/vllm
env:
- MULTINODE_WRAPPER_FORWARD_STREAMS=1
image:
uri: >-
docker://public.ecr.aws/deep-learning-containers/vllm:0.20.0-gpu-py312-cu130-ubuntu22.04-ec2-v1.1-2026-04-29-18-08-36-soci
mounts:
/home/vllm: models
script: |
#!/bin/sh
HOSTNAME=$(hostname)
vllm serve "openai/gpt-oss-20b" \
--dtype auto \
--tool-call-parser openai \
--reasoning-parser openai_gptoss \
--enable-auto-tool-choice \
--tensor-parallel-size 2 \
--nnodes 2 --node-rank 0 \
--master-addr ${MULTINODE_NODE_IP} \
--gpu-memory-utilization 0.9 \
--kv-cache-dtype auto \
--max-num-batched-tokens 2048 \
--max-model-len 131072 \
--host 0.0.0.0 \
--port 8080 &
pid=$!
IFS=','
for HOST in ${MULTINODE_HOSTLIST_NOSLOTS}; do
if [ "$HOSTNAME" != "$HOST" ]; then
$MULTINODE_SSH_WRAPPER $HOST vllm serve "openai/gpt-oss-20b" \
--dtype auto \
--tool-call-parser openai \
--reasoning-parser openai_gptoss \
--enable-auto-tool-choice \
--tensor-parallel-size 2 \
--nnodes 2 --node-rank 1 \
--master-addr ${MULTINODE_NODE_IP} --headless \
--gpu-memory-utilization 0.9 \
--kv-cache-dtype auto \
--max-num-batched-tokens 2048 \
--max-model-len 131072 \
--host 0.0.0.0 \
--port 8080
fi
done
wait $pid
resource:
cpu:
cores: 16
affinity: NUMA
memory:
size: 120GB
devices:
nvidia.com/gpu: 1
annotations:
nvidia.com/gpu.model: NVIDIA L4
multinode:
nodes: 2
implementation: generic
network:
host: true # optional: implied for multinode services
ports:
- name: openai-api
port: 8080
protocol: tcp
endpoints:
- name: openai-vllm
type: subdomain
scope: public
protocol: http
port-name: openai-api
readiness-probe:
tcp-socket:
port: 8080
period-seconds: 30
failure-threshold: 60
success-threshold: 1
initial-delay-seconds: 60
persist: true
This example demonstrates:
- A two-node multinode service with one GPU per node (two GPUs in total)
- Host networking, automatic for multinode services, providing high-speed node-to-node communication
- Rank 0 serves the vLLM API on port 8080
- An HTTP endpoint exposed publicly.
scope: publicrequires no authentication; usescope: organizationto restrict it to your organization
The IFS loop and$MULTINODE_SSH_WRAPPERinvocation in the script above can be replaced with thegeneric-helperfunctionsexec_rank0,exec_rankN, andwait_rank0, which handle these details automatically. See Generic Helper Script for details.
- Distributed Workflows with MPI for MPI job configuration
- Distributed Workflows with Generic Multinode for custom distributed workloads
- Workflow Syntax Guide for the complete YAML reference