Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Host Networking for Multinode Services

A service is a container that runs alongside a workflow’s jobs instead of completing like one. This page configures a service that spans several nodes.

Multinode services always run with host networking: every rank runs directly on its node’s network stack instead of in an isolated network namespace. Fuzzball applies this automatically to any multinode service that declares a network: block, and a multinode service without one already runs on the node’s network stack, so network.host: true is never required. A multinode service cannot opt out: network.host: false is overridden.

Host networking is what lets a distributed service work across nodes: ranks reach each other at node addresses on dynamically negotiated ports, and the workload gets direct access to high-speed interconnects like InfiniBand or RoCE for low-latency, high-bandwidth communication.

A service with host networking shares the node’s network namespace, so it can reach every network interface on the node, including interfaces it would not see in an isolated namespace. Treat the service as running with the node’s own network access.

Configuring a Multinode Service in YAML

Declare the multinode block on the service; host networking is applied automatically. The network.host field may still be set explicitly for clarity:

version: v4

services:
  distributed-inference:
    image:
      uri: docker://myregistry/mpi-inference:latest
    script: |
      #!/bin/bash
      python -m inference_server --model /models/llm
    multinode:
      nodes: 4
      implementation: openmpi
    resource:
      cpu:
        cores: 16
        affinity: NUMA
      memory:
        size: 64GB
      devices:
        nvidia.com/gpu: 2
    persist: true  # keep running with nothing depending on it
    network:
      host: true  # optional: implied for multinode services
      ports:
        - name: http
          port: 8080
      endpoints:
        - name: inference
          port-name: http
          protocol: https
          type: subdomain
          scope: group

Key YAML Fields

  • network.host: Boolean field to enable host networking. Implied true for multinode services; default false otherwise
  • multinode.nodes: Number of compute nodes to allocate
  • multinode.implementation: Coordination framework (ompi, openmpi, mpich, gasnet, or generic)
  • resource: Resources per node. A four-node service with 16 cores each is allocated 64 cores in total
  • persist: Keeps the service running when nothing depends on it. Without it a service that has no dependents is stopped as soon as it reports ready

Understanding Multinode Network Endpoints

In a multinode service:

  • Rank 0 is the first node (node 0) and serves as the primary endpoint
  • Ranks 1, 2, 3, … are worker nodes that participate in distributed computation
  • All network ports and endpoints are exposed only on rank 0
  • Clients connect to rank 0, which coordinates with other ranks internally

For example, in a 4-node distributed inference service:

  • Rank 0 runs the API server and accepts HTTP requests (rank 0 can also be a worker)
  • Ranks 1-3 run workers that process inference requests coordinated by rank 0
  • Clients send requests to rank 0’s HTTPS endpoint

Example: vLLM Distributed Cluster with Host Networking

version: v4

volumes:
  models:
    name: vllm-cache

services:
  vllm:
    cwd: /home/vllm
    env:
      - MULTINODE_WRAPPER_FORWARD_STREAMS=1
    image:
      uri: >-
        docker://public.ecr.aws/deep-learning-containers/vllm:0.20.0-gpu-py312-cu130-ubuntu22.04-ec2-v1.1-2026-04-29-18-08-36-soci
    mounts:
      /home/vllm: models
    script: |
      #!/bin/sh
      HOSTNAME=$(hostname)

      vllm serve "openai/gpt-oss-20b" \
      --dtype auto \
      --tool-call-parser openai \
      --reasoning-parser openai_gptoss \
      --enable-auto-tool-choice \
      --tensor-parallel-size 2 \
      --nnodes 2 --node-rank 0 \
      --master-addr ${MULTINODE_NODE_IP} \
      --gpu-memory-utilization 0.9 \
      --kv-cache-dtype auto \
      --max-num-batched-tokens 2048 \
      --max-model-len 131072 \
      --host 0.0.0.0 \
      --port 8080 &

      pid=$!

      IFS=','

      for HOST in ${MULTINODE_HOSTLIST_NOSLOTS}; do
        if [ "$HOSTNAME" != "$HOST" ]; then
          $MULTINODE_SSH_WRAPPER $HOST vllm serve "openai/gpt-oss-20b" \
                                      --dtype auto \
                                      --tool-call-parser openai \
                                      --reasoning-parser openai_gptoss \
                                      --enable-auto-tool-choice \
                                      --tensor-parallel-size 2 \
                                      --nnodes 2 --node-rank 1 \
                                      --master-addr ${MULTINODE_NODE_IP} --headless \
                                      --gpu-memory-utilization 0.9 \
                                      --kv-cache-dtype auto \
                                      --max-num-batched-tokens 2048 \
                                      --max-model-len 131072 \
                                      --host 0.0.0.0 \
                                      --port 8080
        fi
      done

      wait $pid
    resource:
      cpu:
        cores: 16
        affinity: NUMA
      memory:
        size: 120GB
      devices:
        nvidia.com/gpu: 1
      annotations:
        nvidia.com/gpu.model: NVIDIA L4
    multinode:
      nodes: 2
      implementation: generic
    network:
      host: true        # optional: implied for multinode services
      ports:
        - name: openai-api
          port: 8080
          protocol: tcp
      endpoints:
        - name: openai-vllm
          type: subdomain
          scope: public
          protocol: http
          port-name: openai-api
    readiness-probe:
      tcp-socket:
        port: 8080
      period-seconds: 30
      failure-threshold: 60
      success-threshold: 1
      initial-delay-seconds: 60
    persist: true

This example demonstrates:

  • A two-node multinode service with one GPU per node (two GPUs in total)
  • Host networking, automatic for multinode services, providing high-speed node-to-node communication
  • Rank 0 serves the vLLM API on port 8080
  • An HTTP endpoint exposed publicly. scope: public requires no authentication; use scope: organization to restrict it to your organization
The IFS loop and $MULTINODE_SSH_WRAPPER invocation in the script above can be replaced with the generic-helper functions exec_rank0, exec_rankN, and wait_rank0, which handle these details automatically. See Generic Helper Script for details.

See also