Distributed Workflows with PGAS
Partitioned global address space (PGAS) networks are an alternative to MPI that use a multinode shared memory space to support parallelism. GASNet is an implementation of a PGAS network commonly used to run PGAS-based programming languages like Chapel. PGAS execution in Fuzzball is accessed through the same workflow field that is used to control MPI execution.
The choice is made by the application, not by Fuzzball. Reach for gasnet when:
- Your program is written in a PGAS language or library – Chapel, UPC, UPC++, Legion – whose runtime is built against GASNet. This is the common case, and the deciding one.
- The work has an irregular or data-dependent access pattern, where ranks read and write each other’s memory directly rather than exchanging planned messages. PGAS expresses that as one-sided access to a shared address space; MPI would need explicit two-sided sends and receives, or its own one-sided window calls.
Use openmpi or mpich instead for a program written against MPI, which is the majority of
existing HPC code. See Distributed Workflows with
MPI. A program that carries its own launcher
or coordination mechanism, and needs neither, belongs on generic – see
Distributed Workflows with Generic Multinode.
Observe the following Fuzzfile:
version: v4
jobs:
hello:
image:
uri: docker://wresch/chapel-multinode-gasnet:latest
script: |
#!/bin/sh
/opt/chapel/bin/hello-chapel --tasksPerLocale 2 --numLocales 2
resource:
cpu:
cores: 2
affinity: CORE
memory:
size: 1GB
multinode:
nodes: 2
implementation: gasnet
This will use a container derived from the official Chapel container with the runtime recompiled to enable GASNet. It runs a Hello World example across 2 nodes, with 2 Chapel tasks per node.
Setting the multinode.implementation field to gasnet tells Fuzzball to run the job over the
cluster’s GASNet network.
Read the job’s output with fuzzball workflow log, naming the workflow and the job:
$ fuzzball workflow log WORKFLOW_ID helloWith --numLocales 2 --tasksPerLocale 2, the program runs across both allocated nodes with two
Chapel tasks on each.
For more information about Chapel see the official docs.
A gasnet job is launched differently from an MPI one, which is worth knowing when a launch
misbehaves. For openmpi and mpich, Fuzzball wraps your command with mpirun and that
launcher starts the remote ranks. For gasnet it runs your command directly and lets the GASNet
runtime start the other locales itself, configuring it through the environment:
| Variable | Set to |
|---|---|
GASNET_SPAWNFN | S, selecting GASNet’s SSH-based spawner |
GASNET_SSH_CMD | Fuzzball’s own remote-shell wrapper, so the spawn goes through the workflow’s network rather than a real ssh |
GASNET_SSH_SERVERS | The hostnames of the job’s allocated nodes |
GASNET_MASTERIP | The address of the node acting as the job’s master |
Two consequences follow. Your script should invoke the program itself, with no launcher in front
of it – the Chapel example above calls hello-chapel directly. And because GASNet performs the
spawn, a launch that hangs or fails part-way is usually diagnosed from the GASNet runtime’s own
diagnostics rather than from Fuzzball’s.
A multinode job fails as a group: if any rank dies, the remaining ranks are stopped and the job fails. See Rank Failure.