Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

In-Workflow API Access

Fuzzball injects a workflow token into every job and service container in a workflow, so processes running inside a workflow can call the Fuzzball API. This supports long-running pipeline controllers – for example a Nextflow controller job that submits each pipeline task as a new Fuzzball workflow while it runs – without passing user credentials into the workflow as secrets and without logging in inside the container.

Injected environment variables

VariableDescription
FB_TOKENThe workflow token, minted for the workflow owner
FB_API_HOSTThe host:port of the cluster’s gRPC API, usable as a CLI context address
FB_OPENAPI_URLThe URL of the cluster’s REST (OpenAPI) endpoint, usable with curl
SSL_CERT_DIRThe standard system certificate directories plus the node trust store mounted at /run/fuzzball-substrate/trusted-certs; not set if the workflow defines its own

These are the variables for API access. For every variable Fuzzball sets in workflow containers, see the Environment Variables Reference.

Token scope and lifetime

The token carries the identity of the user who submitted the workflow, acting in the group the workflow was submitted under. Its lifetime is tied to that workflow; its permissions are not:

  • It works against a fixed set of operations, all acting as the submitting user:

    • Workflow API – starting, validating, listing, describing, and stopping workflows; reading workflow stages, stage events, and job logs; listing a workflow’s owners.
    • Endpoints API – listing and reading the workflow service endpoints the owner can reach, and minting endpoint access tokens for them.
    • Elsewhere – resolving cluster names, browsing the application catalog and rendering catalog applications (so fuzzball workflow catalog start works), listing and creating volumes, reading the secret values a submitted workflow references (so secret:// image-pull, SIF-decryption, and environment secrets the owner can read resolve), publishing, reading, listing, and deleting objects in the object cache, and explaining scheduling decisions (fuzzball workflow why and fuzzball queue show --mine).
    • Refused – workflow ownership changes, account-wide reporting, volume updates and deletion, creating or modifying secrets, catalog administration, and cluster administration.
  • Commands that open interactive access to running containers, fuzzball workflow exec, attach, and port-forward, do not accept workflow tokens. fuzzball workflow log does.

  • It stops working as soon as its workflow finishes, fails, or is canceled, regardless of its expiry time.

  • Its expiry depends on the container:

    • Jobs with a walltime – one hour after the job’s execution timeout, so a job can never outlive its token.
    • Jobs with no walltime (policy.timeout.execute: "0") – seven days.
    • Service containers, which have no walltime – seven days.

    A container that runs longer than its token must refresh the token on a schedule, before it expires.

Within those operations the token acts as the submitting user, so it can also list and stop that user’s other workflows – it is not confined to the workflow it was injected into. That is what lets a controller job manage the workflows it submits. To narrow the blast radius further, submit long-running controller workflows as a dedicated user.

Using the token

Inside any workflow container that has the Fuzzball CLI available (see Installing the Fuzzball CLI in a job container), the CLI reads FB_TOKEN and FB_API_HOST when no CLI context is configured:

$ fuzzball workflow start pipeline-step.fz
d0a45cbe-77ad-4a94-97e6-51f5b24b8de2
If a CLI configuration file with an active context is present (for example in a mounted home directory), the configured context takes precedence over the environment variables.

To call the REST API directly instead, pass the token as a bearer credential:

$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/workflows"
The curl and wget examples on this page pass FB_TOKEN as a command-line argument. While the command runs, any process in the container can read the token with ps, and shell tracing writes it to the workflow log in plain text. Do not enable shell tracing (set -x, or sh -x) around commands that use the token.

Trusting the cluster’s certificate authority

Both paths verify the API’s TLS certificate. Alongside the token, every workflow container receives the node’s Fuzzball trust store as a read-only mount at /run/fuzzball-substrate/trusted-certs, and SSL_CERT_DIR is set to that directory plus the standard system certificate directories. The cluster publishes its root CA into that directory on every compute node as root-ca.crt. On a cluster that serves the API with a private certificate authority, the Fuzzball CLI therefore works without any image changes, and the image’s own trust store stays intact.

How a tool picks up the CA depends on its TLS stack:

  • Go-based tools (the Fuzzball CLI included) read every certificate file in the SSL_CERT_DIR directories – they work automatically.
  • OpenSSL-based tools (curl, wget, openssl) look up SSL_CERT_DIR entries by hashed file name and ignore plain PEM files, so pass the CA file explicitly: curl --cacert /run/fuzzball-substrate/trusted-certs/root-ca.crt.
  • Tools with their own bundle (Python requests, for example) take the file through their usual override, such as REQUESTS_CA_BUNDLE=/run/fuzzball-substrate/trusted-certs/root-ca.crt.

Setting SSL_CERT_DIR in a job’s or service’s env section replaces the injected value. Development deployments that serve a certificate not present in the node trust store need --insecure (CLI) or -k (curl).

Installing the Fuzzball CLI in a job container

Every cluster serves a Fuzzball CLI download that matches the cluster’s version. The binary is larger than a job container’s writable layer, so download it into a mounted volume:

version: v4
volumes:
  data:
    use: ephemeral
    size: 1GB
jobs:
  controller:
    image:
      uri: docker://alpine:latest
    mounts:
      /data:
        volume: data
    script: |
      #!/bin/sh
      cd /data
      wget --header "Cookie: fwi.access-token=${FB_TOKEN}" -O fuzzball \
          "https://ui.example.com/api/download/cli/linux/amd64"   # or linux/arm64
      chmod +x fuzzball
      ./fuzzball workflow list

Substitute your deployment’s DNS name for ui.example.com, and pick linux/amd64 or linux/arm64 to match the node architecture. The downloaded CLI reads FB_TOKEN and FB_API_HOST, so it does not require login. Invoke the Fuzzball CLI by path or move it onto PATH.

This recipe passes FB_TOKEN to wget on the command line, where ps and shell tracing can expose it. Do not enable shell tracing (set -x, or sh -x) around the wget command. See Using the token.

The TLS notes above apply here too. On a private-CA cluster, point wget at the mounted CA with --ca-certificate /run/fuzzball-substrate/trusted-certs/root-ca.crt; the downloaded CLI honors the injected SSL_CERT_DIR automatically. With a self-signed development certificate use wget --no-check-certificate and ./fuzzball --insecure.

Accessing workflow service endpoints

The token can also list the workflow service endpoints its owner has access to, and mint endpoint access tokens for them – so a controller workflow can discover the services it started and hand out endpoint credentials. The same CLI commands work unchanged inside a container:

$ fuzzball endpoint list
$ fuzzball endpoint generate-token ENDPOINT_ID

Or over REST:

$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/endpoints"
$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
    "${FB_OPENAPI_URL}/v4/endpoints/ENDPOINT_ID/token" -d '{}'

Access is scoped exactly as for the owner’s own credential: the token sees the endpoints the submitting user can reach across their groups and organization, and is refused for endpoints they cannot reach.

An endpoint token minted this way is a separate credential with its own lifetime: it is not revoked when the minting workflow token expires or when workflow tokens are disabled. Its lifetime is instead capped at the minting workflow token’s own remaining lifetime, so it can never outlive the credential that created it.

An endpoint token does stop working once the endpoint itself is gone, and the endpoint is removed when the workflow that declared it reaches a terminal state. A token minted against a service in another workflow therefore lasts only as long as that workflow does. A persistent endpoint is the exception: it outlives the workflows attached to it and is removed only when it is deleted, so a token minted for it keeps working until it expires, the endpoint is deleted, or the endpoint’s scope is changed.

Refreshing the token

Environment variables cannot be changed in a running container, so a service that outlives its seven-day token has to renew the credential itself. Exchange a still-valid token for a fresh one with the same permissions and a new seven-day expiry:

$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
    "${FB_OPENAPI_URL}/v4/workflows:refreshToken"
{"token":"eyJhbGciOiJFUzI1NiIs...","expiresAt":"2026-08-10T17:04:11Z"}

Nothing restarts, and the previous token keeps working until its original expiry, so there is no cutover to coordinate – but the call only returns the new token, it does not install it, so overwrite FB_TOKEN yourself once you have one in hand. Assigning the result of a failed call would discard a credential that is still valid, and the container cannot recover from that:

new_token=$(curl -sf -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
  "${FB_OPENAPI_URL}/v4/workflows:refreshToken" | jq -r '.token // empty')
if [ -n "$new_token" ]; then
  export FB_TOKEN="$new_token"
else
  echo "workflow token refresh failed; keeping the current token" >&2
fi

The CLI reads FB_TOKEN from the environment on every invocation, so the next fuzzball command picks up the re-exported value with no further configuration. Retrying on the next scheduled run is enough as long as the schedule leaves room inside the seven-day window.

Refresh before the token expires. An expired token is rejected outright – there is no grace period, and no way to deliver a new token into a running container from outside it. A service that needs to run indefinitely should refresh on a schedule comfortably inside the seven-day window (daily, for example), not when expiry is imminent.

Only a workflow token can be refreshed, and it can only refresh itself: the returned token always carries the same user, group, and workflow as the one presented. Your own user credential cannot refresh a workflow’s token, and there is no administrative override. All the usual restrictions still apply – refresh fails once the workflow finishes, fails, or is canceled, and stopping the workflow remains the way to revoke a workflow token. Because a token holder can keep renewing while the workflow runs, the dedicated-user recommendation applies with particular force to service workflows.

Workflow lineage

A workflow submitted with a workflow token records the submitting workflow as its parent; workflows submitted directly by a user have no parent. Parentage therefore separates the workflows a controller job launched from unrelated work in the same group.

fuzzball workflow describe shows the parent in the Parent: row and the number of workflows this workflow submitted in the Children: row. For machine-readable output use fuzzball workflow describe -o json, fuzzball workflow get -o json, or fuzzball workflow list -o json, which all carry the same two values as parentWorkflowId and childCount. To list the workflows a given workflow submitted, pass its ID to --parent:

$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2

A plain workflow list defaults to workflows created in the last seven days, but filtering by --parent searches your full history, so child workflows older than a week are still listed. Add --created-after only when you want to narrow the window:

$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2 --created-after 30d

Disabling injection

Cluster administrators can turn off token injection cluster-wide. On Kubernetes deployments, set the disableWorkflowTokens field in the FuzzballOrchestrate resource:

spec:
  disableWorkflowTokens: true

On deployments where the orchestrator configuration file is managed directly, set the equivalent feature flag:

features:
  disableWorkflowTokens: true

With injection disabled, workflow containers receive none of the variables above, including SSL_CERT_DIR, and the API rejects workflow tokens that were injected before the change – including tokens held by containers that are still running.

The read-only trust-store mount at /run/fuzzball-substrate/trusted-certs is still present. A service behind an endpoint needs the caller-identity signing keys it carries whether or not it holds a workflow token.

See also