In-Workflow API Access
Fuzzball injects a workflow token into every job and service container in a workflow, so processes running inside a workflow can call the Fuzzball API. This supports long-running pipeline controllers – for example a Nextflow controller job that submits each pipeline task as a new Fuzzball workflow while it runs – without passing user credentials into the workflow as secrets and without logging in inside the container.
| Variable | Description |
|---|---|
FB_TOKEN | The workflow token, minted for the workflow owner |
FB_API_HOST | The host:port of the cluster’s gRPC API, usable as a CLI context address |
FB_OPENAPI_URL | The URL of the cluster’s REST (OpenAPI) endpoint, usable with curl |
SSL_CERT_DIR | The standard system certificate directories plus the node trust store mounted at /run/fuzzball-substrate/trusted-certs; not set if the workflow defines its own |
These are the variables for API access. For every variable Fuzzball sets in workflow containers, see the Environment Variables Reference.
The token carries the identity of the user who submitted the workflow, acting in the group the workflow was submitted under. Its lifetime is tied to that workflow; its permissions are not:
It works against a fixed set of operations, all acting as the submitting user:
- Workflow API – starting, validating, listing, describing, and stopping workflows; reading workflow stages, stage events, and job logs; listing a workflow’s owners.
- Endpoints API – listing and reading the workflow service endpoints the owner can reach, and minting endpoint access tokens for them.
- Elsewhere – resolving cluster names, browsing the application catalog and rendering
catalog applications (so
fuzzball workflow catalog startworks), listing and creating volumes, reading the secret values a submitted workflow references (sosecret://image-pull, SIF-decryption, and environment secrets the owner can read resolve), publishing, reading, listing, and deleting objects in the object cache, and explaining scheduling decisions (fuzzball workflow whyandfuzzball queue show --mine). - Refused – workflow ownership changes, account-wide reporting, volume updates and deletion, creating or modifying secrets, catalog administration, and cluster administration.
Commands that open interactive access to running containers,
fuzzball workflow exec,attach, andport-forward, do not accept workflow tokens.fuzzball workflow logdoes.It stops working as soon as its workflow finishes, fails, or is canceled, regardless of its expiry time.
Its expiry depends on the container:
- Jobs with a walltime – one hour after the job’s execution timeout, so a job can never outlive its token.
- Jobs with no walltime (
policy.timeout.execute: "0") – seven days. - Service containers, which have no walltime – seven days.
A container that runs longer than its token must refresh the token on a schedule, before it expires.
Within those operations the token acts as the submitting user, so it can also list and stop that user’s other workflows – it is not confined to the workflow it was injected into. That is what lets a controller job manage the workflows it submits. To narrow the blast radius further, submit long-running controller workflows as a dedicated user.
Inside any workflow container that has the Fuzzball CLI available (see
Installing the Fuzzball CLI in a job container),
the CLI reads FB_TOKEN and FB_API_HOST when no CLI context is configured:
$ fuzzball workflow start pipeline-step.fz
d0a45cbe-77ad-4a94-97e6-51f5b24b8de2If a CLI configuration file with an active context is present (for example in a mounted home directory), the configured context takes precedence over the environment variables.
To call the REST API directly instead, pass the token as a bearer credential:
$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/workflows"Thecurlandwgetexamples on this page passFB_TOKENas a command-line argument. While the command runs, any process in the container can read the token withps, and shell tracing writes it to the workflow log in plain text. Do not enable shell tracing (set -x, orsh -x) around commands that use the token.
Both paths verify the API’s TLS certificate. Alongside the token, every workflow container
receives the node’s Fuzzball trust store as a read-only mount at
/run/fuzzball-substrate/trusted-certs, and SSL_CERT_DIR is set to that directory plus the
standard system certificate directories. The cluster publishes its root CA into that directory
on every compute node as root-ca.crt.
On a cluster that serves the API with a private
certificate authority, the Fuzzball CLI therefore works without any image changes, and the
image’s own trust store stays intact.
How a tool picks up the CA depends on its TLS stack:
- Go-based tools (the Fuzzball CLI included) read every certificate file in the
SSL_CERT_DIRdirectories – they work automatically. - OpenSSL-based tools (
curl,wget,openssl) look upSSL_CERT_DIRentries by hashed file name and ignore plain PEM files, so pass the CA file explicitly:curl --cacert /run/fuzzball-substrate/trusted-certs/root-ca.crt. - Tools with their own bundle (Python
requests, for example) take the file through their usual override, such asREQUESTS_CA_BUNDLE=/run/fuzzball-substrate/trusted-certs/root-ca.crt.
Setting SSL_CERT_DIR in a job’s or service’s env section replaces the injected value.
Development deployments that serve a certificate not present in the node trust store need
--insecure (CLI) or -k (curl).
Every cluster serves a Fuzzball CLI download that matches the cluster’s version. The binary is larger than a job container’s writable layer, so download it into a mounted volume:
version: v4
volumes:
data:
use: ephemeral
size: 1GB
jobs:
controller:
image:
uri: docker://alpine:latest
mounts:
/data:
volume: data
script: |
#!/bin/sh
cd /data
wget --header "Cookie: fwi.access-token=${FB_TOKEN}" -O fuzzball \
"https://ui.example.com/api/download/cli/linux/amd64" # or linux/arm64
chmod +x fuzzball
./fuzzball workflow list
Substitute your deployment’s DNS name for ui.example.com, and pick linux/amd64 or
linux/arm64 to match the node architecture. The downloaded CLI reads FB_TOKEN and
FB_API_HOST, so it does not require login. Invoke the Fuzzball CLI by path or move it onto
PATH.
This recipe passesFB_TOKENtowgeton the command line, wherepsand shell tracing can expose it. Do not enable shell tracing (set -x, orsh -x) around thewgetcommand. See Using the token.
The TLS notes above apply here too. On a
private-CA cluster, point wget at the mounted CA
with --ca-certificate /run/fuzzball-substrate/trusted-certs/root-ca.crt; the downloaded CLI
honors the injected SSL_CERT_DIR automatically. With a self-signed development certificate
use wget --no-check-certificate and ./fuzzball --insecure.
The token can also list the workflow service endpoints its owner has access to, and mint endpoint access tokens for them – so a controller workflow can discover the services it started and hand out endpoint credentials. The same CLI commands work unchanged inside a container:
$ fuzzball endpoint list
$ fuzzball endpoint generate-token ENDPOINT_IDOr over REST:
$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/endpoints"
$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
"${FB_OPENAPI_URL}/v4/endpoints/ENDPOINT_ID/token" -d '{}'Access is scoped exactly as for the owner’s own credential: the token sees the endpoints the submitting user can reach across their groups and organization, and is refused for endpoints they cannot reach.
An endpoint token minted this way is a separate credential with its own lifetime: it is not revoked when the minting workflow token expires or when workflow tokens are disabled. Its lifetime is instead capped at the minting workflow token’s own remaining lifetime, so it can never outlive the credential that created it.
An endpoint token does stop working once the endpoint itself is gone, and the endpoint is removed when the workflow that declared it reaches a terminal state. A token minted against a service in another workflow therefore lasts only as long as that workflow does. A persistent endpoint is the exception: it outlives the workflows attached to it and is removed only when it is deleted, so a token minted for it keeps working until it expires, the endpoint is deleted, or the endpoint’s scope is changed.
Environment variables cannot be changed in a running container, so a service that outlives its seven-day token has to renew the credential itself. Exchange a still-valid token for a fresh one with the same permissions and a new seven-day expiry:
$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
"${FB_OPENAPI_URL}/v4/workflows:refreshToken"
{"token":"eyJhbGciOiJFUzI1NiIs...","expiresAt":"2026-08-10T17:04:11Z"}Nothing restarts, and the previous token keeps working until its original expiry, so there is no
cutover to coordinate – but the call only returns the new token, it does not install it, so
overwrite FB_TOKEN yourself once you have one in hand. Assigning the result of a failed call
would discard a credential that is still valid, and the container cannot recover from that:
new_token=$(curl -sf -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
"${FB_OPENAPI_URL}/v4/workflows:refreshToken" | jq -r '.token // empty')
if [ -n "$new_token" ]; then
export FB_TOKEN="$new_token"
else
echo "workflow token refresh failed; keeping the current token" >&2
fi
The CLI reads FB_TOKEN from the environment on every invocation, so the next fuzzball command
picks up the re-exported value with no further configuration. Retrying on the next scheduled run
is enough as long as the schedule leaves room inside the seven-day window.
Refresh before the token expires. An expired token is rejected outright – there is no grace period, and no way to deliver a new token into a running container from outside it. A service that needs to run indefinitely should refresh on a schedule comfortably inside the seven-day window (daily, for example), not when expiry is imminent.
Only a workflow token can be refreshed, and it can only refresh itself: the returned token always carries the same user, group, and workflow as the one presented. Your own user credential cannot refresh a workflow’s token, and there is no administrative override. All the usual restrictions still apply – refresh fails once the workflow finishes, fails, or is canceled, and stopping the workflow remains the way to revoke a workflow token. Because a token holder can keep renewing while the workflow runs, the dedicated-user recommendation applies with particular force to service workflows.
A workflow submitted with a workflow token records the submitting workflow as its parent; workflows submitted directly by a user have no parent. Parentage therefore separates the workflows a controller job launched from unrelated work in the same group.
fuzzball workflow describe shows the parent in the Parent: row and the number of workflows
this workflow submitted in the Children: row. For machine-readable output use
fuzzball workflow describe -o json, fuzzball workflow get -o json, or
fuzzball workflow list -o json, which all carry the same two values as parentWorkflowId
and childCount. To list the workflows a given workflow submitted, pass its ID to --parent:
$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2A plain workflow list defaults to workflows created in the last seven days, but filtering by
--parent searches your full history, so child workflows older than a week are still listed. Add
--created-after only when you want to narrow the window:
$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2 --created-after 30dCluster administrators can turn off token
injection cluster-wide. On Kubernetes deployments,
set
the disableWorkflowTokens field in the
FuzzballOrchestrate resource:
spec:
disableWorkflowTokens: true
On deployments where the orchestrator configuration file is managed directly, set the equivalent feature flag:
features:
disableWorkflowTokens: true
With injection disabled, workflow containers receive none of the variables above, including
SSL_CERT_DIR, and the API rejects workflow tokens that were injected before the change –
including tokens held by containers that are still running.
The read-only trust-store mount at /run/fuzzball-substrate/trusted-certs is still present.
A service behind an endpoint needs the caller-identity signing keys it carries whether or not it
holds a workflow token.
- Workflow Endpoints for endpoint scopes, access-token lifetimes, and the URLs a workflow token can mint against.
- Getting an Interactive Shell and Executing Commands
for the
exec,attach, andport-forwardcommands that workflow tokens cannot use. - Interacting with Other Users’ Workflows for workflow ownership, which a workflow token can read but not change.
- FuzzballOrchestrate CRD reference
for the cluster-wide
disableWorkflowTokensfield.