Deployment
After fulfilling the requirements, you are ready to deploy Fuzzball. Use the Kubernetes Operator tab for a multi-node, production-style deployment across server and compute nodes, or the Docker Compose tab for a single-host deployment suited to local development and testing.
The fuzzball cluster kubernetes deploy command installs Fuzzball into a Kubernetes cluster you
already run: the Fuzzball Operator, Fuzzball
Orchestrate, and the services they depend on. It
suits a multi-node, production-style deployment. deploy checks the cluster before it changes
anything, waits until the deployment is ready, and sets up the Fuzzball CLI to use it. From then on,
you manage the deployment through the fuzzball cluster kubernetes subcommands.
Compute nodes outside the cluster run Fuzzball Substrate and connect to the deployment. Once it is up, see Substrate Installation to add them.
The deployment brings up these components, each in its own namespace:
| Component | Namespace | Role |
|---|---|---|
| Fuzzball Operator | fuzzball-system | Deploys everything below from a FuzzballOrchestrate resource and keeps it that way |
| Fuzzball Orchestrate | fuzzball | API, scheduler, web UI, JetStream, and the substrate bridge that Compute nodes connect through |
| PostgreSQL | fuzzball-database | State for Keycloak and Orchestrate |
| Keycloak | fuzzball-identity | Sign-in for the CLI and the web UI |
| Kong | ingress | TLS and routing for every Fuzzball name; --existing-ingress uses your ingress controller instead |
| cert-manager, trust-manager | cert-manager | Certificates; --existing-cert-manager uses yours instead |
| Prometheus, Grafana | fuzzball-monitoring | Metrics |
| NFS provisioner | nfs-provisioner | Volumes on your NFS export, as StorageClass nfs-client |
The images all come from CIQ Depot, so the
Kubernetes nodes need access to depot.ciq.com.
Make sure the cluster meets the requirements: Kubernetes 1.30 or later,
a default StorageClass, and an NFS export that the Kubernetes nodes and the Compute nodes can mount,
with the NFS client (nfs-utils) on every Kubernetes node. The RKE2
appendix sets up such a cluster. On a cluster without a default
StorageClass, deploy can install one; see Deploy. Then install the Fuzzball CLI and
deploy.
Run the commands in this tab on any machine with a kubeconfig that has cluster-admin access to the cluster, such as a Server node or your workstation. This guide calls it the CLI host. The commands use the kubeconfig’s current context; pass--kubeconfigor--kube-contextto pick another.
The Fuzzball images and the operator’s Helm chart are pulled from depot.ciq.com. deploy needs the Depot
credentials from the CIQ sales/support team, and hands them to the cluster as an image pull Secret.
Set them in your shell; the commands read the token from DEPOT_ACCESS_TOKEN:
$ DEPOT_USER="" # populate with your username for CIQ Depot
$ export DEPOT_ACCESS_TOKEN="" # populate with the Depot key obtained from the CIQ sales/support teamInstall the fuzzball command-line tool on the CLI host, following the CLI installation
guide. Install the version of Fuzzball you
want to deploy: deploy installs the CLI’s own version unless you pass --version. Once installed,
confirm the binary runs:
$ fuzzball --versionRKE2 installs Traefik as its default ingress controller. Leave it in place. Either Fuzzball installs
its own ingress controller, Kong, whose Ingresses use Kong’s kong class, so Traefik doesn’t serve
them, or Traefik serves Fuzzball, as The cluster’s ingress controller
describes. The RKE2 appendix covers the rest of what Fuzzball needs
from RKE2 nodes, such as their external addresses and the SELinux label for local-path.
Ingress mode. EKS nodes in private subnets have no external addresses, so with
nodeportonly clients inside the VPC can reach Fuzzball. Use--ingress loadbalancer. AWS load balancers publish a host name, so create the DNS records as DNS records for a load balancer describes.StorageClass. EKS 1.30 and later create no default StorageClass. Install the Amazon EBS CSI driver add-on with the configuration values
{"defaultStorageClass":{"enabled":true}}, which makes it create a default StorageClass. The add-on needs an IAM role, for example through EKS Pod Identity.--install-storage-class local-pathwould keep Fuzzball’s volumes on the nodes’ local disks instead.NFS export on Amazon EFS. An EFS file system works as the NFS export: pass its DNS name to
--nfs-serverand/to--nfs-path, and allow TCP port2049from the VPC in the security group of its mount targets. The Amazon Linux 2023 nodes of EKS already have the NFS client.Load balancer settings. Set AWS load balancer annotations in a settings file. This one makes JetStream and DNS internal load balancers, reachable only inside the VPC, and limits the ingress’s public load balancer to the address ranges you list:
spec: ingress: create: proxy: annotations: service.beta.kubernetes.io/load-balancer-source-ranges: "203.0.113.0/24,198.51.100.7/32" fuzzball: jetstream: externalService: annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" substrateBridge: dns: externalService: annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true"List every address that reaches
api.<domain>through the public load balancer: your clients; the NAT gateway that the cluster’s nodes use to reach the internet, because the deployment itself connects toapi.<domain>; and the public or NAT addresses of the Compute nodes, which connect to it too, even from inside the VPC. These annotations are for the load balancer support built into EKS, which creates Classic Load Balancers. If you run the AWS Load Balancer Controller, see its documentation for the annotations it reads.updatewith changed source ranges applies them to the same load balancer, so the DNS records stay valid.Compute nodes. Orchestrate connects to Compute nodes from pod addresses in the VPC; see the network requirements.
deploy creates the operator’s namespace, fuzzball-system, with the Kubernetes Pod Security
baseline profile, which Fuzzball pods need. It does this even on clusters that enforce the
restricted profile by default, such as CIS-hardened clusters.
The operator labels these namespaces. substrate is labeled privileged. fuzzball-federate,
fuzzball-database, fuzzball-identity, the monitoring namespace (default fuzzball-monitoring)
and nfs-provisioner are labeled baseline. fuzzball is labeled baseline, or privileged
when the CoreWeave provisioner is enabled, because that provisioner runs privileged substrate pods
there. A namespace label takes precedence over the cluster default. If you labeled one of these
namespaces by hand, remove your label before upgrading so the operator can set it.
The operator leaves ingress and cert-manager unlabeled; their pods are admitted under the
cluster default. It also does not label the datadog namespace. If you enable Datadog, label that
namespace privileged.
If you run your own Linkerd service mesh, install it with the Linkerd CNI plugin. Without it,
Linkerd injects a linkerd-init container that needs the NET_ADMIN and NET_RAW capabilities,
which the baseline profile does not allow.
The deploy command checks the cluster, installs the operator, and waits until Fuzzball is ready:
$ fuzzball cluster kubernetes deploy --domain fuzzball.example.com \
--owner-email admin@example.com --depot-user "${DEPOT_USER}" \
--nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared--domain is the domain the Fuzzball services are served under; its names must resolve as
Reaching the deployment describes. --owner-email is the email address of the
Fuzzball organization’s owner, the account you first sign in with. On a terminal, deploy prompts
for either one if you leave it out; with --non-interactive, a missing value is an error instead.
A first deployment takes 10 minutes or more. deploy runs the preflight checks
first and stops before it changes anything if one fails. It then installs the Fuzzball Operator as
the Helm release fuzzball-operator in namespace fuzzball-system, stores your Depot credentials
in the image pull Secret registry-image-credentials, and applies the FuzzballOrchestrate
resource fuzzball-orchestrate, which names that Secret in spec.image.credentialsSecretName
rather than holding the credentials itself. While the operator works, deploy shows its progress.
Once the deployment reports Ready, deploy adds a Fuzzball CLI context named after the domain,
trusting the operator’s CA when that CA signs the ingress certificate, and prints how to use it:
To use the context and log in, run:
fuzzball context use fuzzball.example.com
fuzzball context login
'fuzzball cluster kubernetes info --secrets' shows the initial credentials.
It also prints the steps that let Compute nodes take work, which Substrate Installation covers.
Ctrl-C stops the wait, not the deployment: the operator keeps going. Run the same deploy command
again to wait again, or check on it with fuzzball cluster kubernetes status. Running deploy
again also updates an existing deployment in place.
Add--dry-runto print theFuzzballOrchestrateresource and the operator’s chart values without contacting the cluster. If your site changes the cluster only through its own tools, such as Argo CD or Flux, see Alternative: generate the deployment files for your own tools.
Most sites need a few more flags. fuzzball cluster kubernetes deploy --help lists them all.
| Flag | Effect |
|---|---|
--ingress MODE | nodeport (the default) or loadbalancer. See Reaching the deployment. |
--storage-class NAME | Puts Fuzzball’s volumes on StorageClass NAME instead of the cluster’s default. |
--install-storage-class local-path | On a cluster without a default StorageClass, installs local-path-provisioner from Depot as the default. |
--install-metallb --metallb-pool RANGE | With --ingress loadbalancer, installs MetalLB 0.16.1 and gives it the addresses in RANGE: CIDRs or FIRST-LAST ranges, comma-separated. |
--nfs-server, --nfs-path | The NFS export. Set both or neither. |
--nfs-destination DIR | Where Compute nodes mount the NFS export. The default is /mnt/fuzzball-sharedfs. |
--existing-cert-manager | Uses the cert-manager and trust-manager already installed on the cluster. |
--existing-ingress CLASS | Serves Fuzzball through the cluster’s own ingress controller of IngressClass CLASS instead of installing Kong. See The cluster’s ingress controller. |
--version VERSION | Deploys this Fuzzball version instead of the CLI’s own. |
--timeout DURATION | How long to wait for the deployment to report Ready. The default is 1h. |
On a cluster with neither a load balancer nor a default StorageClass, for example, deploy installs
both before the operator:
$ fuzzball cluster kubernetes deploy --domain fuzzball.example.com \
--owner-email admin@example.com --depot-user "${DEPOT_USER}" \
--nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared \
--ingress loadbalancer --install-metallb --metallb-pool 10.0.0.240-10.0.0.250 \
--install-storage-class local-pathMetalLB and local-path-provisioner pull their images from their upstream registries, so the
Kubernetes nodes need access to those as well. On nodes with SELinux in enforcing mode,
local-path-provisioner can’t create volumes until /opt/local-path-provisioner carries a label that
containers can write to. preflight fails until it does and prints the command that sets the label;
see StorageClass in the RKE2 appendix.
deploy runs the preflight checks itself, but you can run them on their own to find problems before
you deploy. Pass preflight the same flags as deploy. It ignores the ones that only deploy
reads, such as --owner-email, so a deploy command line works with preflight in its place:
$ fuzzball cluster kubernetes preflight --domain fuzzball.example.com \
--depot-user "${DEPOT_USER}" \
--nfs-server nfs.example.com --nfs-path /srv/fuzzball/sharedIt checks API access and cluster-admin rights, the Kubernetes version, cert-manager or Kong
installations that would conflict with Fuzzball’s own, or the IngressClass that --existing-ingress
names, image pulls from Depot on every untainted
node, storage, the NFS export, the load balancer, and DNS. Each check passes, fails, or warns, and
prints a fix when it doesn’t pass. The command exits with a non-zero status when a required check
fails. The first run can take several minutes while each node pulls the Fuzzball image. Its test
objects live in a temporary namespace that it deletes when it finishes.
If you deploy with a settings file, pass it to preflight as well.
preflight refuses a file that deploy would refuse. With --ingress loadbalancer, its test load
balancers carry the annotations the file sets on the ingress, JetStream, and DNS Services, so the
test covers settings such as an internal scheme, an address pool, or source ranges. It leaves out
annotations that pin an address or a name, such as metallb.io/loadBalancerIPs, and leaves that
address or name to the deployment’s own Service. On a cluster that already runs Fuzzball, the
test leaves out the Services that the deployment already runs as load balancers, because test
Services would compete with them for addresses; when that is all three, preflight skips the test.
If your cluster already runs cert-manager and trust-manager, pass --existing-cert-manager to both
preflight and deploy. See Deploying with External
cert-manager for what Fuzzball expects of them.
For settings that have no flag, pass deploy a file as its argument: a partial
FuzzballOrchestrate resource in YAML. deploy merges the file over the resource it builds from
the flags. Unless you pass --skip-preflight, the file can’t change the values that preflight
checks, such as the domain, the ingress mode, the Fuzzball version, the NFS export, or the
StorageClass; set those with their flags. The file also must not change metadata.name. See the
FuzzballOrchestrate CRD Reference for the fields.
For example, this file pins the ingress to an address from your MetalLB pool, so that you can use
<address>.nip.io as the domain:
spec:
ingress:
create:
proxy:
annotations:
metallb.io/loadBalancerIPs: 10.0.0.240
$ fuzzball cluster kubernetes deploy fuzzball-settings.yaml --domain 10.0.0.240.nip.io \
--owner-email admin@example.com --depot-user "${DEPOT_USER}" \
--nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared --ingress loadbalancerKeep credentials out of the file. The resource takes the Depot credentials from the pull Secret
that deploy creates, and the inline spec.image.username and spec.image.password fields are
deprecated.
The deployment serves its services under subdomains of --domain: api., auth., ui.,
substrate-bridge., and endpoints., plus the wildcard *.endpoints. for workflow endpoints. These
names must resolve to the deployment on the CLI host, inside the cluster, and on the Compute nodes;
preflight checks the first two. Where they point depends on --ingress:
--ingress | Where the names point | Use it when |
|---|---|---|
nodeport (default) | A Kubernetes node’s address. URLs carry the HTTPS node port, as in https://ui.fuzzball.example.com:30443. | Clients and Compute nodes can reach the Kubernetes nodes directly |
loadbalancer | The ingress’s load balancer address | The cluster has a load balancer, such as MetalLB, kube-vip, or your cloud’s, or its nodes are on a private network |
With nodeport, the ingress listens on node port 30443 for HTTPS and 30080 for HTTP on every
Kubernetes node, and JetStream and the Fuzzball DNS service, which Compute nodes connect to, listen
on node ports too. Point the names at a node address that kubectl get nodes -o wide lists as
INTERNAL-IP or EXTERNAL-IP. --https-node-port and --http-node-port change the ingress ports.
If clients and Compute nodes reach the nodes at addresses that Kubernetes doesn’t list, such as
public addresses in front of private ones, make the nodes report those addresses as their
EXTERNAL-IP first; otherwise preflight fails its DNS checks. On RKE2, that is the
node-external-ip setting that the RKE2 appendix
describes. Nodes with no address that clients can reach, such as EKS nodes in private subnets, need
--ingress loadbalancer instead.
Without DNS records of your own, use a nip.io domain. The public nip.io zone resolves every name
under <address>.nip.io, wildcards included, to <address>, with no setup. Use the address of a
node that clients and Compute nodes can reach:
$ fuzzball cluster kubernetes deploy --domain 10.0.0.11.nip.io [other deploy flags]With loadbalancer, the ingress, JetStream, and DNS each get a LoadBalancer Service. On a cluster
without a load balancer, --install-metallb --metallb-pool RANGE installs MetalLB, as in
Deploy.
A load balancer that assigns IP addresses, such as MetalLB, can take a fixed address from your pool:
pin the ingress to it in a settings file and use <address>.nip.io as
the domain. A load balancer that publishes a host name, such as AWS’s, needs DNS records of your own.
With --ingress loadbalancer, the ingress gets its load balancer address only once the operator has
created the ingress Service, and deploy, info, and status don’t show it. Read it from the
EXTERNAL-IP column while deploy is still waiting:
$ kubectl get service -n ingress kong-kong-proxyPoint the Fuzzball names at it: with A records if it shows an IP address, or with a CNAME record or
Route 53 alias if it shows a host name, as AWS load balancers do. One wildcard record,
*.<domain>, covers every name, including those under endpoints.<domain>.
Create the records right away. The deployment loads the default workflow catalog through
api.<domain>, and that step tries six times, then gives up. When you destroy the deployment and
deploy it again, the new load balancer can have a new address, so update the records then.
Instead of installing Kong, deploy can serve Fuzzball through an ingress controller that the
cluster already runs: ingress-nginx, Kong, or Traefik. Pass its IngressClass, as kubectl get ingressclass lists it, to both preflight and deploy:
$ fuzzball cluster kubernetes deploy --existing-ingress nginx [other deploy flags]Point the Fuzzball names at the address that your controller serves on. --ingress then chooses only
how JetStream and the Fuzzball DNS service reach the Compute nodes, and --http-node-port and
--https-node-port don’t apply. The URLs use port 443. If your controller serves HTTPS on another
port, set spec.ingress.publicPort to it in a settings file.
preflight checks that the IngressClass exists and that ingress-nginx, Kong, or Traefik serves it,
and the operator configures Fuzzball’s Ingresses for that controller. If the class reports a
controller that preflight doesn’t recognize, name the controller in the settings file:
spec:
ingress:
external:
controller: nginx
The FuzzballOrchestrate CRD Reference
describes what each controller needs, such as the proxy buffer size of nginx-based controllers.
destroy and cleanup leave your controller in place.
Substrate on each Compute node connects to two services in the cluster: JetStream, to register with
Orchestrate and receive work, and the Fuzzball DNS service, to resolve the hostnames of jobs. With
--ingress nodeport, both services listen on node ports of the Kubernetes nodes, and Substrate is
given the nodes’ addresses. If the Compute nodes can’t route to those addresses, for example because
the Kubernetes nodes are virtual machines on a private network, Substrate can’t register and job
hostnames don’t resolve.
In that case, deploy with --ingress loadbalancer. Substrate is then given each load balancer
address with the service port: 4222 for JetStream and 5353 for DNS. To have your load balancer
assign addresses on a network the Compute nodes can reach, set annotations in a settings
file. With MetalLB, for example, select an address pool:
spec:
fuzzball:
jetstream:
externalService:
annotations:
metallb.io/address-pool: compute-reachable
substrateBridge:
dns:
externalService:
annotations:
metallb.io/address-pool: compute-reachable
Allow TCP ports 4222 and 5353 to those addresses through any firewall between the Compute nodes
and the load balancer. Each Compute node registers with the address it uses to reach the JetStream
load balancer, which on a host with several network interfaces is the address of the interface that
routes to the load balancer. Orchestrate connects back to each Compute node on that address, so the
cluster must be able to reach it.
The deployment doesn’t finish, and Compute nodes can’t join, until the load balancer has assigned an
address to both services. To see the assigned addresses, check the EXTERNAL-IP column, which
shows a host name for load balancers that publish one, such as AWS’s:
$ kubectl get service -n fuzzball fuzzball-jetstream-external fuzzball-substrate-bridge-dnsdeploy created a Fuzzball CLI context for the deployment. Switch to it and log in:
$ fuzzball context use fuzzball.example.com
$ fuzzball context logincontext login launches the device-code flow in your browser against Keycloak. Sign in as the
organization owner, with the email address you passed to --owner-email and the initial password
that fuzzball cluster kubernetes info --secrets shows. Without --secrets, info shows the
deployment’s URLs, including the web UI’s, and the CA that signs the ingress certificate when it is
the operator’s own CA.
Show the state of the deployment:
$ fuzzball cluster kubernetes statusstatus shows the FuzzballOrchestrate resource’s conditions, the Fuzzball version, and the
rollout of the operator and the services it deploys. The deployment is ready when the Ready
condition is True. status reports the state when you run it; it doesn’t wait for the deployment
to become ready.
Then run the readiness checks. They use your Fuzzball context, so log in first:
$ fuzzball cluster kubernetes validateThe default health tier is read-only. Once a Compute node has joined, --tier functional also
runs small test workloads on it. validate exits with status 1 when a required check fails.
To list the pods in any namespace that aren’t running, or to follow the operator’s log:
$ kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
$ fuzzball cluster kubernetes logs -n fuzzball-system --component operator -fTo update Fuzzball to a new version, install the Fuzzball CLI at that version on the CLI host, then
run update:
$ fuzzball cluster kubernetes update --depot-user "${DEPOT_USER}"An update to another major version runs a one-way database migration that can’t be undone.updatefirst asks you to type the resource’s name.
update upgrades the Fuzzball Operator, sets the new version on the resource, and waits up to
--timeout for the deployment to report Ready at that version. It updates to the CLI’s own version
unless you pass --version, and it needs the Depot credentials, because the operator chart comes
from Depot. It leaves the resource’s other settings as they are, whether deploy created the
resource or you wrote it by hand.
To change settings, pass a partial FuzzballOrchestrate resource in YAML as the argument of
update. update merges it over the running resource, and it may change any field but
metadata.name. --dry-run prints the resource and the chart values that update would apply,
without changing anything.
After an update that moves to a new Fuzzball version, or that changes the domain, the HTTPS node port, the NFS destination, the image registries, or the certificate authorities, refresh each Compute node, as Substrate Installation describes.
destroydeletes every PersistentVolumeClaim in the Fuzzball namespaces, the database included. Their data can’t be recovered.
First stop Substrate on each external Compute node with systemctl disable --now fuzzball-substrate. Then remove the deployment:
$ fuzzball cluster kubernetes destroydestroy asks you to type the resource’s name. It deletes the FuzzballOrchestrate resource and
waits for the operator to delete everything it installed: the Fuzzball services, their namespaces,
and their PersistentVolumeClaims. It then uninstalls the operator, deletes namespace
fuzzball-system and the CRDs of the cert-manager, trust-manager, and Kong that Fuzzball installed,
and deletes the CLI context that deploy added. To run it without a prompt, for example in a
script, pass --non-interactive --force. If --timeout, one hour by default, runs out before the
removal finishes, the operator keeps removing the deployment; run destroy again to wait again.
destroy leaves these in place:
- MetalLB and local-path-provisioner, if
deployinstalled them.destroyprints thehelm uninstallcommands that remove them. - The Kong CRDs, while another Kong still serves an IngressClass on the cluster.
- The released PersistentVolumes of StorageClass
nfs-client, whose data stays on the NFS export.destroylists them. Delete them withkubectl delete pv, and their data on the export, when you no longer need them.
If an interrupted deploy or destroy left pieces behind, such as the Helm release or the
fuzzball-system namespace, fuzzball cluster kubernetes cleanup lists them and removes them once
you confirm. It refuses while a FuzzballOrchestrate resource exists.
deploy changes the cluster itself. If your site changes the cluster only through its own tools,
such as Helm, Argo CD, Flux, or kubectl from a Git repository, the Fuzzball CLI can write the files
for those tools instead. Apply them as shown below and connect the
CLI, then continue at Check the deployment.
Generate the files with fuzzball cluster kubernetes deploy --dry-run --output-dir. The command
builds the FuzzballOrchestrate resource from its flags and doesn’t contact the cluster, so it
needs no kubeconfig. It pulls the operator chart from Depot, so it needs your Depot credentials.
Run fuzzball cluster kubernetes deploy --help for the flags that set the resource.
$ fuzzball cluster kubernetes deploy --dry-run --output-dir fuzzball-deploy \
--domain fuzzball.example.com --owner-email admin@example.com \
--depot-user "${DEPOT_USER}" \
--nfs-server nfs.example.com --nfs-path /srv/fuzzball/sharedThe output directory must be new or empty. The command writes these files to it:
| File | Contents |
|---|---|
namespace.yaml | The fuzzball-system namespace, with the baseline Pod Security profile. The chart doesn’t create it. |
fuzzball-operator-<version>.tgz | The operator Helm chart, as pulled from Depot. |
fuzzball-operator-values.yaml | The chart values: the operator image and the name of its pull Secret. |
fuzzballorchestrate.yaml | The FuzzballOrchestrate resource. |
README.md | The order to apply the files in, the commands that apply them, and notes for Argo CD and Flux. |
The values and the resource name the image pull Secret registry-image-credentials in namespace
fuzzball-system, but they don’t hold your Depot credentials. Create that Secret with your site’s
sealed-secrets or external-secrets tooling, of type kubernetes.io/dockerconfigjson with one
auths entry for depot.ciq.com. Alternatively, add --include-secrets to write it to
secrets/registry-image-credentials.yaml, in a directory only you can read. Don’t commit
secrets/ to Git. With --registry, the images come from your registry and the CLI writes no pull
Secret.
The chart holds the CRDs, so apply it before the resource. Skip kubectl apply -f secrets/ if you
created the pull Secret with your own tooling, or if you passed --registry, as the CLI then writes
no pull Secret.
$ cd fuzzball-deploy
$ kubectl apply -f namespace.yaml
$ kubectl apply -f secrets/
$ helm upgrade --install fuzzball-operator ./fuzzball-operator-v4.4.0-110-g8f0b4aff023d.tgz \
-n fuzzball-system -f fuzzball-operator-values.yaml --wait
$ kubectl wait --for condition=established crd/fuzzballorchestrate.deployment.ciq.com
$ kubectl apply -f fuzzballorchestrate.yamlOn upgrade, the chart’s post-upgrade hook runs manager update, which migrates the stored
resources to the new CRDs. A tool that renders the chart without running Helm hooks must run that
step itself.
- Argo CD maps the
post-upgradehook to aPostSynchook and runs it on every sync. Set theServerSideApply=truesync option, since the operator’s CRDs are too large for client-side apply, and sync the chart before the resource, with sync waves or two Applications. - Flux: a
HelmReleaseruns the chart with Helm, hooks included. - kubectl:
helm templaterenders the chart. Apply it withkubectl apply --server-side, since the CRDs are too large for client-side apply, and the upgrade step is then yours.
When Argo CD or Flux manages the FuzzballOrchestrate resource or the operator Deployment,
fuzzball cluster kubernetes update and destroy warn that the tool may undo their changes. Change
the deployment through the tool instead.
To upgrade, generate the files for the new version and apply them the same way:
fuzzball cluster kubernetes update --dry-run --output-dir DIRreads the runningFuzzballOrchestrateresource, so it needs access to the cluster. It writes the resource at the new version, with the changes of an optionalFILEmerged in.- Without access to the cluster, run
fuzzball cluster kubernetes deploy --dry-run --output-dir DIRagain with the new--versionand the same flags.
Once the deployment reports Ready, which fuzzball cluster kubernetes status shows, add the
Fuzzball CLI context with fuzzball cluster kubernetes info --create-context. It adds the context
as deploy does: named after the domain, and trusting the operator’s CA when that CA signs the
ingress certificate. It leaves an existing context of that name unchanged and doesn’t switch the
active context.
$ fuzzball cluster kubernetes info --create-context
$ fuzzball context use fuzzball.example.com
$ fuzzball context loginShortly afterkubectl applycreates the resource, the operator writes its defaults into it and reconciles again. For about half a minute,statuscan then show the deployment as not ready after it first showed Ready. Checkstatusagain before you rely on the deployment.
You now have a running instance of Fuzzball Orchestrate. Next, follow Substrate Installation to connect Compute nodes to it.
The fuzzball cluster docker-compose deploy command brings up a single-host Fuzzball stack — nginx
(TLS termination and subdomain routing), PostgreSQL, Keycloak, Fuzzball
Orchestrate, and one or more Fuzzball
Substrate nodes — suited to local development,
evaluation, and testing. The deployment is managed entirely through the fuzzball cluster docker-compose subcommands.
For hardware-specific walkthroughs that build on this tab, see Install Fuzzball on an NVIDIA DGX Spark and Install Fuzzball on an AMD Ryzen AI Max (Strix Halo).
The compose stack brings up five services on the host:
| Service | Image | Role |
|---|---|---|
nginx | nginx:1.27-alpine | TLS edge + subdomain routing, NATS/JetStream TCP passthrough |
postgres | postgres:16 | State for Keycloak + Orchestrate |
keycloak | quay.io/phasetwo/keycloak-crdb:24.0.5 | OIDC auth (Phase Two CockroachDB variant) |
fuzzball-orchestrate | depot.ciq.com/fuzzball/fuzzball-images/fuzzball-orchestrate:<tag> | API + scheduler + embedded JetStream + UI |
substrate-localnode-1 | depot.ciq.com/fuzzball/fuzzball-images/fuzzball-substrate-orchestrate:<tag> | Workflow executor (use the -gpu image variant with --gpu) |
The substrate runs workflow jobs as rootless OCI containers directly on the host — no Kubernetes, no
cloud, no external infrastructure. On a host with a GPU, the --gpu flag swaps the
substrate to the GPU-enabled image variant, which bundles the device plugins that pass the GPU
through to workflow containers.
Make sure the host meets the requirements, including a working Docker installation with the Compose plugin. Then install the Fuzzball CLI and deploy the stack.
Run the commands in this tab as a regular user who can rundockerwithoutsudo, usually by being a member of thedockergroup. Run them all as the same user: each deployment is stored in that user’s configuration directory. The few steps that change system files, such as--update-etc-hosts,--setup-dns, anddelete, runsudothemselves and prompt for your password.
The Fuzzball images are pulled from depot.ciq.com. If your Docker daemon isn’t already authenticated to
it, log in first with the Depot credentials from the CIQ sales/support team:
$ DEPOT_USER="" # populate with your username for CIQ Depot
$ ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team$ echo "${ACCESS_KEY}" | docker login depot.ciq.com --username "${DEPOT_USER}" --password-stdinInstall the fuzzball command-line tool, following the CLI installation
guide — use the .deb package matching your host
architecture (amd64 or arm64). Once installed, confirm the binary runs:
$ fuzzball --versionThe deploy command renders the stack’s configuration files to disk, and --up starts the
containers:
$ fuzzball cluster docker-compose deploy --upTo stop and restart the stack later, see Starting and stopping a stack.
Add--dry-runto preview the generateddocker-compose.yamland.envwithout writing anything to disk or starting containers.
If the container images require authentication, deploy prompts for a registry user and
password/token (your CIQ Depot credentials); leave
the user blank to skip the registry login. You can also supply them non-interactively with
--registry-user and --registry-password, and override the registry itself with --registry.
Most other settings — organization name, database and Keycloak credentials, owner email — are
auto-generated or defaulted and can be overridden with their corresponding flags (see deploy --help); generated secrets are recoverable later with info --secrets.
By default the stack runs a single substrate node co-located with Orchestrate. For testing or
demonstrating multi-node workflows on a single host, pass --substrate-nodes N to run N substrate
nodes (default 1, max 255):
$ fuzzball cluster docker-compose deploy --substrate-nodes 2Running multiple substrate nodes on a single host can over-provision it. This is intended for development and demonstration of multi-node workflows, not for production capacity.
The stack can also accept a Fuzzball Substrate node running on a separate host, for compute capacity beyond the Orchestrate host. See the Docker Compose tab of Substrate Installation for the full procedure, including the shared image cache and volume storage that external nodes use.
deploy and update --sync-files rewrite docker-compose.yaml from the CLI’s embedded copy, so
edits made directly to that file are lost the next time either runs.
Put local changes in docker-compose.override.yaml in the deployment directory instead. Fuzzball
never writes or regenerates that file, and applies it last, so its settings win over both the base
file and any Fuzzball-managed overlay:
services:
substrate-localnode:
volumes:
- /srv/fuzzball-shared:/mnt
This is the supported way to add a bind mount, pin an address, or change an image without those changes being reverted.
Keycloak runs with a fixed 1GiB Java heap inside a 1.75GiB container memory limit. To give it more,
raise both values together in docker-compose.override.yaml, keeping roughly 750MiB of the limit
above the heap for the JVM’s non-heap memory:
services:
keycloak:
environment:
JAVA_OPTS: >-
-Xms2048m
-Xmx2048m
-XX:+ExitOnOutOfMemoryError
-XX:MetaspaceSize=96m
-XX:MaxMetaspaceSize=256m
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:MaxGCPauseMillis=200
-Dfile.encoding=UTF-8
-Dsun.stdout.encoding=UTF-8
-Dsun.err.encoding=UTF-8
-Dstdout.encoding=UTF-8
-Dstderr.encoding=UTF-8
-Djava.security.egd=file:/dev/urandom
-XX:FlightRecorderOptions=stackdepth=512
deploy:
resources:
limits:
memory: 2.75g
An override replaces JAVA_OPTS as a whole, so restate every flag, not only the heap size.
The base file also sets JAVA_OPTS_APPEND: -XX:-UseBASE64Intrinsics, which mitigates the
Java4Shell vulnerability (CVE-2026-114938) in the JDK that the Keycloak image ships. Keycloak adds
JAVA_OPTS_APPEND after JAVA_OPTS, so the mitigation stays in place when you override
JAVA_OPTS. If you also set JAVA_OPTS_APPEND in your override, include
-XX:-UseBASE64Intrinsics in it.
The standard deploy brings up a CPU-only stack. On hosts that have GPUs you can opt in to a
GPU-enabled substrate image. The same image bundles both the NVIDIA and AMD device plugins; the
substrate automatically loads whichever matches the GPU present on the host, so one --gpu image
works on either vendor:
$ fuzzball cluster docker-compose deploy --gpu [other deploy flags]Or to update an existing deployment to the GPU-enabled substrate image:
$ fuzzball cluster docker-compose update --gpu- The host’s GPU vendor driver must be installed and loaded (the vendor’s status tool — e.g.
nvidia-smifor NVIDIA — should work on the host). - The vendor-specific device nodes must be present (the driver creates these at boot).
$ fuzzball node list --available-resources
ID | CLUSTER | HOSTNAME | STATUS | CORES | MEMORY (GB) | DEVICES
172.21.0.5/7334 | local-dev | substrate-localnode-2 | Ready | 20 | 127.9 | nvidia.com/gpu:1If a GPU device type (for example nvidia.com/gpu:N or amd.com/gpu:N) shows up in the
DEVICES column, the substrate-side plumbing is correct and the stack can run GPU workflows.
For how a workflow then requests a GPU, and for NVIDIA versus AMD ROCm specifics, see Using GPUs with Fuzzball.
By default the deployment is named default. To run more than one stack on the same host, or to
give a deployment a meaningful name, pass --deployment-name:
$ fuzzball cluster docker-compose deploy --deployment-name my-dev-stackThe deployment name is used as the Docker Compose project name and as the directory name under your
Fuzzball CLI config home. All of the other docker-compose subcommands accept the same
--deployment-name flag to select which stack they act on.
Running more than one stack on the same host at the same time is not recommended — separate stacks compete for the same host ports and resources.
The stack routes traffic to its services through subdomains of a base domain. The ui., api.,
keycloak., endpoints., and nats. subdomains must resolve to the host running the stack before
you can reach the UI, API, or log in.
There is also a wildcard requirement. Workflow service endpoints declared with type: subdomain
are served at a per-endpoint hostname of the form endpoint-<id>.endpoints.<domain>. A new hostname
appears for every endpoint, so they cannot be listed ahead of time, and /etc/hosts has no way to
express a wildcard. Endpoints declared with the default type: path are served under the single
endpoints.<domain> name and are unaffected.
Which approach you need depends on the host:
| Situation | What to do |
|---|---|
| The site has its own DNS | Add a wildcard A record, deploy with --domain |
| No DNS, but the host has internet access | Nothing — this is the default |
| Neither DNS nor internet (air-gapped) | Deploy with --setup-dns |
deploy defaults --domain to <host-ip>.nip.io. The nip.io zone resolves every name beneath it
to the IP address embedded in the name, wildcards included, so
endpoint-<id>.endpoints.10.0.0.5.nip.io resolves to 10.0.0.5 with no configuration on the host
and no sudo. Names resolve both on the host itself and from other machines on the same network.
$ fuzzball cluster docker-compose deploy --upThe domain is fixed when the deployment is created — it is written into the generated TLS certificate and.env, andupdatecannot change it. If the host’s IP address changes, delete the deployment and deploy again. Assign the host a static or reserved address to avoid this.
Use your own domain and add a single wildcard A record pointing at the host:
*.fb.example.internal. IN A 10.0.0.5
Then deploy against it:
$ fuzzball cluster docker-compose deploy --domain fb.example.internalThis is the most robust option: it covers every client on the network with no per-machine setup, and the domain stays stable if the host’s address changes.
When the host has neither DNS infrastructure nor internet access, the stack can serve the wildcard zone itself:
$ fuzzball cluster docker-compose deploy --domain fb.local --setup-dns--wildcard-dns makes Fuzzball Orchestrate
answer every name under the domain with 127.0.0.1, published on 127.0.0.1:5354. The zone is
authoritative only — it never forwards queries — and the published port is bound to the loopback
interface, so it cannot act as a resolver for anything else on the network.
--setup-dns implies --wildcard-dns and additionally configures the host to use that zone. It
requires sudo and writes two things, both removed again by delete:
- A split-DNS entry, so that only
*.<domain>is sent to the stack and every other lookup continues to use the host’s normal upstream resolver:- macOS —
/etc/resolver/<domain> - Linux with
systemd-resolved—/etc/systemd/resolved.conf.d/fuzzball-<deployment>.conf
- macOS —
- An
/etc/hostsentry for the fixed subdomains.
Both are required. The split-DNS entry is what makes the per-endpoint wildcard resolve, but on macOS
it is only visible to programs that resolve names through the system library — browsers and curl,
but not the statically linked fuzzball CLI. The /etc/hosts entry covers the fixed names for
every client, including the CLI. After configuring DNS, deploy confirms that the fixed names
resolve and warns if they do not.
On macOS,
digandnslookupquery name servers directly and never consult/etc/resolver, so they report failure for wildcard names that resolve correctly in a browser. Usedscacheutilinstead:$ dscacheutil -q host -a name ui.fb.local
On Linux hosts without systemd-resolved there is no per-domain routing to configure, so deploy
prints instructions for setting up dnsmasq rather than attempting the change itself. The
equivalent configuration is:
server=/fb.local/127.0.0.1#5354
--update-etc-hosts adds a 127.0.0.1 entry covering the fixed subdomains, without any of the
above. It requires sudo:
$ fuzzball cluster docker-compose deploy --update-etc-hostsOmit the flag and deploy prints the line for you to add yourself:
127.0.0.1 ui.fb.local api.fb.local keycloak.fb.local endpoints.fb.local nats.fb.local
Because hosts files cannot express wildcards, this covers everything except subdomain-style
workflow endpoints. It is unnecessary with the default domain, and unnecessary with --setup-dns,
which writes the same entry itself.
Before the CLI can reach the cluster you must trust the deployment’s self-signed certificate (or
connect with --insecure). fuzzball cluster docker-compose info prints the service URLs, the exact
command to trust the certificate, and the ready-to-paste context commands:
$ fuzzball cluster docker-compose info
Deployment:
Name: default
Directory: /home/user/.config/fuzzball/docker-compose/default
Service URLs:
UI: https://ui.fb.local
API: https://api.fb.local
Keycloak: https://keycloak.fb.local
Endpoints: https://endpoints.fb.local
JetStream: nats://nats.fb.local:4222
Configuration:
... (deployment .env values; passwords hidden unless you pass --secrets)
TLS:
Certs: /home/user/.config/fuzzball/docker-compose/default/certs
To trust the deployment's wildcard certificate:
export SSL_CERT_DIR="/home/user/.config/fuzzball/docker-compose/default/certs"
Connect:
fuzzball context create default https://api.fb.local https://keycloak.fb.local/realms/<realm-id> fuzzball-cli
fuzzball context login
UI:
Open https://ui.fb.local in your browserRun the export SSL_CERT_DIR=... line and the two fuzzball context commands exactly as printed.
context login launches the device-code flow in your browser against Keycloak; sign in as the admin
user admin@fb.local plus an auto-generated password (reveal it again with fuzzball cluster docker-compose info --secrets).
The deploy command generates a self-signed root CA (rootCA.pem) and a wildcard leaf certificate
under the deployment’s certs/ directory. Because the CA is self-signed, browsers and CLI clients
will not trust it by default, and connections to the UI and API will report a certificate warning.
To trust the certificate, either point the fuzzball CLI at the deployment’s certs/ directory or
import the root CA into your operating system or browser trust store. fuzzball cluster docker-compose info prints the certs/ path and the exact command to export, for example:
$ export SSL_CERT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/certs"If you accept the browser warnings instead of importing the root CA, accept them for the UI URL and, when you sign in through Keycloak, for the Keycloak URL the sign-in redirects to. You do not need to accept the API URL: the web UI sends its API requests, including the workflow log stream, through its own server.
The leaf certificate is valid for one year. Renew it in place without redeploying usingfuzzball cluster docker-compose update --tls-renew— see Updating Fuzzball.
Once a stack has been deployed, you can start and stop it without re-rendering its files:
$ fuzzball cluster docker-compose up
$ fuzzball cluster docker-compose downdownstops and removes the stack’s containers but preserves its data volumes. To additionally remove the volumes — which destroys all data in the deployment — passdown --volumes. This prompts for confirmation; add-yto skip the prompt in scripts.
The full set of fuzzball cluster docker-compose subcommands lets you manage deployments once they
exist:
| Subcommand | Description |
|---|---|
deploy | Render and (optionally) start a new deployment. |
up | Start a previously deployed stack. |
down | Stop a stack (optionally removing its volumes). |
update | Update an existing deployment — see Updating Fuzzball. |
status | Show a deployment’s name, version, and running state. |
list | List all Docker Compose deployments on the host. |
info | Show deployment details: service URLs, configuration, TLS trust command, how to connect, and the browser UI endpoint. Secret values (passwords, keys) are masked unless you pass --secrets. |
logs | Show a deployment’s logs (optionally for a single service); add -f to follow. |
delete | Remove a deployment and its generated files. Prompts for confirmation (-y to skip), and offers to remove any /etc/hosts entries it added. |
generate-substrate-config | Generate the configuration an external substrate node needs to join the stack — see Substrate Installation. |
preflight | With node, run on an external Compute node: check the node against its installed configuration before starting Substrate — see Substrate Installation. |
Run any subcommand with --help to see its full set of options.
Before upgrading, back up the deployment
directory. Its certs/ directory holds the
deployment’s root CA and its private key, and .env holds the generated passwords. If the CA key
is lost, renewing the certificate creates a new CA, and every external Compute node must then be
reconfigured. For the default deployment:
$ (umask 077 && tar czf ~/fuzzball-compose-default-$(date +%Y%m%d-%H%M%S).tgz \
-C "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose" default)umask 077 makes the archive readable only by you, and the timestamp gives each backup a new file
rather than reusing an older one that may have looser permissions. Keep the archive somewhere
private: it contains secrets. It doesn’t include the stack’s data, which
lives in Docker volumes such as postgres-data.
To bump the stack to a newer Fuzzball image tag:
$ fuzzball cluster docker-compose update --upgrade--upgrade moves the deployment to the version your current fuzzball CLI binary was built against;
use --version <tag> to pick a specific one instead. The same update command rotates credentials
(--reset-owner-password, --reset-keycloak-password, --reset-database-password) and renews the
wildcard TLS leaf certificate (--tls-renew). See Updating Fuzzball and
fuzzball cluster docker-compose update --help for the full set.
On a deployment created by an earlier CLI, the stack’s own substrate nodes publish an internal CA as
/run/fuzzball-substrate/trusted-certs/root-ca.crt, so workflows on them cannot verify the API’s
certificate (see In-Workflow API Access).
Refresh the generated files while upgrading to fix it. --sync-files overwrites edits made directly
to docker-compose.yaml and the other generated files, so move local changes into an override file
first (see Keeping local changes to the stack):
$ fuzzball cluster docker-compose update --upgrade --sync-filesIf the update’s planned changes do not list IMAGE_TAG – the deployment was already on this
version – the update can leave the stack’s own substrate nodes running with the old CA. Restart the
stack with fuzzball cluster docker-compose down and then up to finish.
A compose deployment scopes its default storage provisioner to the docker segment, so it is
offered only to the compose-managed substrate containers that can actually reach the shared Docker
volume behind it. A node added to the cluster by hand never joins that segment and is therefore
never offered storage it cannot see.
Upgrading does not apply this to an existing deployment. The scoped provisioner is created
under a new name, default-<segment>, and a provisioner’s identity is derived from its name, so it
is a new record rather than a rename – renaming would separate the original provisioner from every
volume created on it. The upgrade therefore leaves the original unsegmented default in place and
adds default-docker alongside it.
Nothing that worked before the upgrade stops working: default keeps its volumes and keeps serving
the compose nodes. But because it carries no segment, it is still advertised to every node in the
cluster, including hand-added ones, so a workflow that selects it and runs on such a node still
fails to start with a missing-directory error.
A fresh deployment is the only way to get the scoping applied automatically. If reinstalling is not practical, retire the old provisioner by hand instead:
Create the volumes you want to keep on
default-docker, and copy their data across.Remove the original:
$ fuzzball volume provisioner remove default
Until one of those happens, treat default as usable only from the compose-managed nodes.
Orchestrate logs a warning at startup whenever it finds an unsegmented default provisioner
alongside a segmented one.
Whether the two provisioners share a directory depends on the release that created
default-docker. Check withfuzzball volume provisioner info default-docker:
- It shows a
subPath.default-dockerkeeps its volumes in the organization’s own subdirectory of the shared volume, anddefaultuses the root. A volume on one is not visible through the other, which is why step 1 recreates each volume ondefault-dockerand copies its data.- It shows no
subPath. Both provisioners serve the same directory, so a volume name that already exists there is refused on the other with an “already exists” error rather than quietly sharing a directory. Deleting a volume through one provisioner removes the data a matching record on the other points at; afuzzball volume provisioner scanclears the stale record.
Stop the stack while preserving its persistent volumes (postgres-data, jetstream-data,
substrate-shared-fs, etc.) — useful when you want to come back to your workflows later:
$ fuzzball cluster docker-compose downTo completely remove the deployment, including its generated files and persistent state:
$ fuzzball cluster docker-compose deleteWhen a Docker Compose deployment misbehaves, it’s often useful to inspect the generated stack files
or to drive docker compose directly. The CLI writes each deployment into its own directory under
your Fuzzball CLI config home and runs docker compose against it on your behalf, so you can do the
same thing manually.
pull access denied for depot.ciq.com/...
Docker isn’t authenticated to the registry. Log in as shown in Container registry access, then re-run the deploy.
Each deployment is stored in a subdirectory named after the deployment, under the docker-compose
folder in your Fuzzball CLI config home:
${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/<deployment-name>/
IfXDG_CONFIG_HOMEis set in your environment, the directory is rooted there ($XDG_CONFIG_HOME/fuzzball/docker-compose/...); otherwise it falls back to$HOME/.config/fuzzball/docker-compose/.... Docker compose deployments always use this per-user config directory location.
The directory holds everything the stack needs:
docker-compose.yaml— the generated Compose file describing every service in the stack..env— the environment file with values such asIMAGE_TAG(the deployed Fuzzball version) that the Compose file substitutes in. SetPULL_POLICY=missingthere to stop the Orchestrate and substrate containers from pulling their images on every start, for example when restarting a host that is offline; the default isalways.certs/— TLS material generated for the deployment.
The CLI invokes docker compose with the deployment name as the Compose project name (-p)
and with the deployment directory as the working directory. To reproduce that yourself, change into
the deployment directory and pass the same -p <deployment-name> flag. For the default deployment:
$ cd "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/"List the containers and their state (equivalent to what fuzzball cluster docker-compose status
inspects):
$ docker compose -p default psTail the logs of a single service — for example, Orchestrate:
$ docker compose -p default logs -f fuzzball-orchestrateInspect the fully-rendered Compose configuration with .env values substituted in, which is helpful
for confirming the image tag and other resolved settings:
$ docker compose -p default configAlways pass-p <deployment-name>so your manualdocker composecommands act on the same project the CLI manages. For any deployment created with--deployment-name <name>, substitute that name fordefaultin both the directory path and the-pflag above.
If browser logins fail but fuzzball commands keep succeeding, suspect the Keycloak realm rather
than the user’s password. The two paths authenticate differently: the CLI uses a device flow
against the public fuzzball-cli client and talks to Keycloak directly, while the Web UI needs
the confidential fuzzball-ui client and looks it up through the organization service. A realm
or client that has gone missing therefore breaks the browser while leaving the CLI working.
Fuzzball creates both clients in each realm when the organization is first bootstrapped. Neither
has Keycloak service accounts enabled, so a grant_type=client_credentials request always fails
against them and cannot be used to test whether a client is healthy. Use the two checks below
instead.
First, confirm the realm exists. This needs no credentials:
$ curl -sk -o /dev/null -w '%{http_code}\n' \
https://<host>/auth/realms/<realm>/.well-known/openid-configuration200 means the realm is present; 404 means it is missing, which breaks every browser login in
that organization.
If the realm is present, check that its fuzzball-ui client is too. This uses the Keycloak admin
credentials from the deployment’s .env:
$ TOKEN=$(curl -sk -X POST https://<host>/auth/realms/master/protocol/openid-connect/token \
-d grant_type=password -d client_id=admin-cli \
-d username=<keycloak-admin> -d password=<keycloak-admin-password> \
| sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
$ curl -sk -H "Authorization: Bearer $TOKEN" \
"https://<host>/auth/admin/realms/<realm>/clients?clientId=fuzzball-ui"A JSON array containing a client object means it is present. An empty array ([]) means it is
missing, which is the condition that breaks Web UI logins while the CLI keeps working.
Orchestrate records the same condition at startup, which is the quickest way to confirm what the
two checks above found. When it cannot reach the realm or its fuzzball-ui client, it logs an
error containing failed to bootstrap default organization:
$ docker compose -p default logs fuzzball-orchestrate | grep "bootstrap default organization"If that line is present, recreate the realm and its clients rather than rotating secrets.
Keycloak realm and client settings come from Orchestrate’s configuration, not from the Compose file. A realm identifier written intodocker-compose.yamlhas no effect on which realm the deployment uses.
A multi-host topology — this host plus remote substrate nodes — is supported via generate-substrate-config,
which exports the orchestrate’s CA, NATS NKey, and connection config for deployment to a separate
machine. See --help on that command and Federation
for the cross-cluster story.
Hardening for production (real TLS via cert-manager / Caddy, external managed Postgres + Keycloak,
backup automation, log shipping) is covered in Cloud Deployment — most of the recommendations there apply equally to a
single-host deployment when you want to move beyond local development.