Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Deployment

After fulfilling the requirements, you are ready to deploy Fuzzball. Use the Kubernetes Operator tab for a multi-node, production-style deployment across server and compute nodes, or the Docker Compose tab for a single-host deployment suited to local development and testing.

Please select your environment.

The fuzzball cluster kubernetes deploy command installs Fuzzball into a Kubernetes cluster you already run: the Fuzzball Operator, Fuzzball Orchestrate, and the services they depend on. It suits a multi-node, production-style deployment. deploy checks the cluster before it changes anything, waits until the deployment is ready, and sets up the Fuzzball CLI to use it. From then on, you manage the deployment through the fuzzball cluster kubernetes subcommands.

Compute nodes outside the cluster run Fuzzball Substrate and connect to the deployment. Once it is up, see Substrate Installation to add them.

What the deployment installs

The deployment brings up these components, each in its own namespace:

ComponentNamespaceRole
Fuzzball Operatorfuzzball-systemDeploys everything below from a FuzzballOrchestrate resource and keeps it that way
Fuzzball OrchestratefuzzballAPI, scheduler, web UI, JetStream, and the substrate bridge that Compute nodes connect through
PostgreSQLfuzzball-databaseState for Keycloak and Orchestrate
Keycloakfuzzball-identitySign-in for the CLI and the web UI
KongingressTLS and routing for every Fuzzball name; --existing-ingress uses your ingress controller instead
cert-manager, trust-managercert-managerCertificates; --existing-cert-manager uses yours instead
Prometheus, Grafanafuzzball-monitoringMetrics
NFS provisionernfs-provisionerVolumes on your NFS export, as StorageClass nfs-client

The images all come from CIQ Depot, so the Kubernetes nodes need access to depot.ciq.com.

Installation

Make sure the cluster meets the requirements: Kubernetes 1.30 or later, a default StorageClass, and an NFS export that the Kubernetes nodes and the Compute nodes can mount, with the NFS client (nfs-utils) on every Kubernetes node. The RKE2 appendix sets up such a cluster. On a cluster without a default StorageClass, deploy can install one; see Deploy. Then install the Fuzzball CLI and deploy.

Run the commands in this tab on any machine with a kubeconfig that has cluster-admin access to the cluster, such as a Server node or your workstation. This guide calls it the CLI host. The commands use the kubeconfig’s current context; pass --kubeconfig or --kube-context to pick another.

Container registry access

The Fuzzball images and the operator’s Helm chart are pulled from depot.ciq.com. deploy needs the Depot credentials from the CIQ sales/support team, and hands them to the cluster as an image pull Secret. Set them in your shell; the commands read the token from DEPOT_ACCESS_TOKEN:

$ DEPOT_USER="" # populate with your username for CIQ Depot

$ export DEPOT_ACCESS_TOKEN="" # populate with the Depot key obtained from the CIQ sales/support team

Fuzzball CLI

Install the fuzzball command-line tool on the CLI host, following the CLI installation guide. Install the version of Fuzzball you want to deploy: deploy installs the CLI’s own version unless you pass --version. Once installed, confirm the binary runs:

$ fuzzball --version

RKE2 clusters

RKE2 installs Traefik as its default ingress controller. Leave it in place. Either Fuzzball installs its own ingress controller, Kong, whose Ingresses use Kong’s kong class, so Traefik doesn’t serve them, or Traefik serves Fuzzball, as The cluster’s ingress controller describes. The RKE2 appendix covers the rest of what Fuzzball needs from RKE2 nodes, such as their external addresses and the SELinux label for local-path.

EKS clusters

  • Ingress mode. EKS nodes in private subnets have no external addresses, so with nodeport only clients inside the VPC can reach Fuzzball. Use --ingress loadbalancer. AWS load balancers publish a host name, so create the DNS records as DNS records for a load balancer describes.

  • StorageClass. EKS 1.30 and later create no default StorageClass. Install the Amazon EBS CSI driver add-on with the configuration values {"defaultStorageClass":{"enabled":true}}, which makes it create a default StorageClass. The add-on needs an IAM role, for example through EKS Pod Identity. --install-storage-class local-path would keep Fuzzball’s volumes on the nodes’ local disks instead.

  • NFS export on Amazon EFS. An EFS file system works as the NFS export: pass its DNS name to --nfs-server and / to --nfs-path, and allow TCP port 2049 from the VPC in the security group of its mount targets. The Amazon Linux 2023 nodes of EKS already have the NFS client.

  • Load balancer settings. Set AWS load balancer annotations in a settings file. This one makes JetStream and DNS internal load balancers, reachable only inside the VPC, and limits the ingress’s public load balancer to the address ranges you list:

    spec:
      ingress:
        create:
          proxy:
            annotations:
              service.beta.kubernetes.io/load-balancer-source-ranges: "203.0.113.0/24,198.51.100.7/32"
      fuzzball:
        jetstream:
          externalService:
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        substrateBridge:
          dns:
            externalService:
              annotations:
                service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    

    List every address that reaches api.<domain> through the public load balancer: your clients; the NAT gateway that the cluster’s nodes use to reach the internet, because the deployment itself connects to api.<domain>; and the public or NAT addresses of the Compute nodes, which connect to it too, even from inside the VPC. These annotations are for the load balancer support built into EKS, which creates Classic Load Balancers. If you run the AWS Load Balancer Controller, see its documentation for the annotations it reads. update with changed source ranges applies them to the same load balancer, so the DNS records stay valid.

  • Compute nodes. Orchestrate connects to Compute nodes from pod addresses in the VPC; see the network requirements.

Pod Security

deploy creates the operator’s namespace, fuzzball-system, with the Kubernetes Pod Security baseline profile, which Fuzzball pods need. It does this even on clusters that enforce the restricted profile by default, such as CIS-hardened clusters.

The operator labels these namespaces. substrate is labeled privileged. fuzzball-federate, fuzzball-database, fuzzball-identity, the monitoring namespace (default fuzzball-monitoring) and nfs-provisioner are labeled baseline. fuzzball is labeled baseline, or privileged when the CoreWeave provisioner is enabled, because that provisioner runs privileged substrate pods there. A namespace label takes precedence over the cluster default. If you labeled one of these namespaces by hand, remove your label before upgrading so the operator can set it.

The operator leaves ingress and cert-manager unlabeled; their pods are admitted under the cluster default. It also does not label the datadog namespace. If you enable Datadog, label that namespace privileged.

If you run your own Linkerd service mesh, install it with the Linkerd CNI plugin. Without it, Linkerd injects a linkerd-init container that needs the NET_ADMIN and NET_RAW capabilities, which the baseline profile does not allow.

Deploy

The deploy command checks the cluster, installs the operator, and waits until Fuzzball is ready:

$ fuzzball cluster kubernetes deploy --domain fuzzball.example.com \
    --owner-email admin@example.com --depot-user "${DEPOT_USER}" \
    --nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared

--domain is the domain the Fuzzball services are served under; its names must resolve as Reaching the deployment describes. --owner-email is the email address of the Fuzzball organization’s owner, the account you first sign in with. On a terminal, deploy prompts for either one if you leave it out; with --non-interactive, a missing value is an error instead.

A first deployment takes 10 minutes or more. deploy runs the preflight checks first and stops before it changes anything if one fails. It then installs the Fuzzball Operator as the Helm release fuzzball-operator in namespace fuzzball-system, stores your Depot credentials in the image pull Secret registry-image-credentials, and applies the FuzzballOrchestrate resource fuzzball-orchestrate, which names that Secret in spec.image.credentialsSecretName rather than holding the credentials itself. While the operator works, deploy shows its progress. Once the deployment reports Ready, deploy adds a Fuzzball CLI context named after the domain, trusting the operator’s CA when that CA signs the ingress certificate, and prints how to use it:

  To use the context and log in, run:

    fuzzball context use fuzzball.example.com
    fuzzball context login

  'fuzzball cluster kubernetes info --secrets' shows the initial credentials.

It also prints the steps that let Compute nodes take work, which Substrate Installation covers.

Ctrl-C stops the wait, not the deployment: the operator keeps going. Run the same deploy command again to wait again, or check on it with fuzzball cluster kubernetes status. Running deploy again also updates an existing deployment in place.

Add --dry-run to print the FuzzballOrchestrate resource and the operator’s chart values without contacting the cluster. If your site changes the cluster only through its own tools, such as Argo CD or Flux, see Alternative: generate the deployment files for your own tools.

Most sites need a few more flags. fuzzball cluster kubernetes deploy --help lists them all.

FlagEffect
--ingress MODEnodeport (the default) or loadbalancer. See Reaching the deployment.
--storage-class NAMEPuts Fuzzball’s volumes on StorageClass NAME instead of the cluster’s default.
--install-storage-class local-pathOn a cluster without a default StorageClass, installs local-path-provisioner from Depot as the default.
--install-metallb --metallb-pool RANGEWith --ingress loadbalancer, installs MetalLB 0.16.1 and gives it the addresses in RANGE: CIDRs or FIRST-LAST ranges, comma-separated.
--nfs-server, --nfs-pathThe NFS export. Set both or neither.
--nfs-destination DIRWhere Compute nodes mount the NFS export. The default is /mnt/fuzzball-sharedfs.
--existing-cert-managerUses the cert-manager and trust-manager already installed on the cluster.
--existing-ingress CLASSServes Fuzzball through the cluster’s own ingress controller of IngressClass CLASS instead of installing Kong. See The cluster’s ingress controller.
--version VERSIONDeploys this Fuzzball version instead of the CLI’s own.
--timeout DURATIONHow long to wait for the deployment to report Ready. The default is 1h.

On a cluster with neither a load balancer nor a default StorageClass, for example, deploy installs both before the operator:

$ fuzzball cluster kubernetes deploy --domain fuzzball.example.com \
    --owner-email admin@example.com --depot-user "${DEPOT_USER}" \
    --nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared \
    --ingress loadbalancer --install-metallb --metallb-pool 10.0.0.240-10.0.0.250 \
    --install-storage-class local-path

MetalLB and local-path-provisioner pull their images from their upstream registries, so the Kubernetes nodes need access to those as well. On nodes with SELinux in enforcing mode, local-path-provisioner can’t create volumes until /opt/local-path-provisioner carries a label that containers can write to. preflight fails until it does and prints the command that sets the label; see StorageClass in the RKE2 appendix.

Checking the cluster first

deploy runs the preflight checks itself, but you can run them on their own to find problems before you deploy. Pass preflight the same flags as deploy. It ignores the ones that only deploy reads, such as --owner-email, so a deploy command line works with preflight in its place:

$ fuzzball cluster kubernetes preflight --domain fuzzball.example.com \
    --depot-user "${DEPOT_USER}" \
    --nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared

It checks API access and cluster-admin rights, the Kubernetes version, cert-manager or Kong installations that would conflict with Fuzzball’s own, or the IngressClass that --existing-ingress names, image pulls from Depot on every untainted node, storage, the NFS export, the load balancer, and DNS. Each check passes, fails, or warns, and prints a fix when it doesn’t pass. The command exits with a non-zero status when a required check fails. The first run can take several minutes while each node pulls the Fuzzball image. Its test objects live in a temporary namespace that it deletes when it finishes.

If you deploy with a settings file, pass it to preflight as well. preflight refuses a file that deploy would refuse. With --ingress loadbalancer, its test load balancers carry the annotations the file sets on the ingress, JetStream, and DNS Services, so the test covers settings such as an internal scheme, an address pool, or source ranges. It leaves out annotations that pin an address or a name, such as metallb.io/loadBalancerIPs, and leaves that address or name to the deployment’s own Service. On a cluster that already runs Fuzzball, the test leaves out the Services that the deployment already runs as load balancers, because test Services would compete with them for addresses; when that is all three, preflight skips the test.

If your cluster already runs cert-manager and trust-manager, pass --existing-cert-manager to both preflight and deploy. See Deploying with External cert-manager for what Fuzzball expects of them.

Settings without a flag

For settings that have no flag, pass deploy a file as its argument: a partial FuzzballOrchestrate resource in YAML. deploy merges the file over the resource it builds from the flags. Unless you pass --skip-preflight, the file can’t change the values that preflight checks, such as the domain, the ingress mode, the Fuzzball version, the NFS export, or the StorageClass; set those with their flags. The file also must not change metadata.name. See the FuzzballOrchestrate CRD Reference for the fields.

For example, this file pins the ingress to an address from your MetalLB pool, so that you can use <address>.nip.io as the domain:

spec:
  ingress:
    create:
      proxy:
        annotations:
          metallb.io/loadBalancerIPs: 10.0.0.240
$ fuzzball cluster kubernetes deploy fuzzball-settings.yaml --domain 10.0.0.240.nip.io \
    --owner-email admin@example.com --depot-user "${DEPOT_USER}" \
    --nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared --ingress loadbalancer

Keep credentials out of the file. The resource takes the Depot credentials from the pull Secret that deploy creates, and the inline spec.image.username and spec.image.password fields are deprecated.

Reaching the deployment

The deployment serves its services under subdomains of --domain: api., auth., ui., substrate-bridge., and endpoints., plus the wildcard *.endpoints. for workflow endpoints. These names must resolve to the deployment on the CLI host, inside the cluster, and on the Compute nodes; preflight checks the first two. Where they point depends on --ingress:

--ingressWhere the names pointUse it when
nodeport (default)A Kubernetes node’s address. URLs carry the HTTPS node port, as in https://ui.fuzzball.example.com:30443.Clients and Compute nodes can reach the Kubernetes nodes directly
loadbalancerThe ingress’s load balancer addressThe cluster has a load balancer, such as MetalLB, kube-vip, or your cloud’s, or its nodes are on a private network

Default: node ports

With nodeport, the ingress listens on node port 30443 for HTTPS and 30080 for HTTP on every Kubernetes node, and JetStream and the Fuzzball DNS service, which Compute nodes connect to, listen on node ports too. Point the names at a node address that kubectl get nodes -o wide lists as INTERNAL-IP or EXTERNAL-IP. --https-node-port and --http-node-port change the ingress ports.

If clients and Compute nodes reach the nodes at addresses that Kubernetes doesn’t list, such as public addresses in front of private ones, make the nodes report those addresses as their EXTERNAL-IP first; otherwise preflight fails its DNS checks. On RKE2, that is the node-external-ip setting that the RKE2 appendix describes. Nodes with no address that clients can reach, such as EKS nodes in private subnets, need --ingress loadbalancer instead.

Without DNS records of your own, use a nip.io domain. The public nip.io zone resolves every name under <address>.nip.io, wildcards included, to <address>, with no setup. Use the address of a node that clients and Compute nodes can reach:

$ fuzzball cluster kubernetes deploy --domain 10.0.0.11.nip.io [other deploy flags]

Load balancers

With loadbalancer, the ingress, JetStream, and DNS each get a LoadBalancer Service. On a cluster without a load balancer, --install-metallb --metallb-pool RANGE installs MetalLB, as in Deploy.

A load balancer that assigns IP addresses, such as MetalLB, can take a fixed address from your pool: pin the ingress to it in a settings file and use <address>.nip.io as the domain. A load balancer that publishes a host name, such as AWS’s, needs DNS records of your own.

DNS records for a load balancer

With --ingress loadbalancer, the ingress gets its load balancer address only once the operator has created the ingress Service, and deploy, info, and status don’t show it. Read it from the EXTERNAL-IP column while deploy is still waiting:

$ kubectl get service -n ingress kong-kong-proxy

Point the Fuzzball names at it: with A records if it shows an IP address, or with a CNAME record or Route 53 alias if it shows a host name, as AWS load balancers do. One wildcard record, *.<domain>, covers every name, including those under endpoints.<domain>.

Create the records right away. The deployment loads the default workflow catalog through api.<domain>, and that step tries six times, then gives up. When you destroy the deployment and deploy it again, the new load balancer can have a new address, so update the records then.

The cluster’s ingress controller

Instead of installing Kong, deploy can serve Fuzzball through an ingress controller that the cluster already runs: ingress-nginx, Kong, or Traefik. Pass its IngressClass, as kubectl get ingressclass lists it, to both preflight and deploy:

$ fuzzball cluster kubernetes deploy --existing-ingress nginx [other deploy flags]

Point the Fuzzball names at the address that your controller serves on. --ingress then chooses only how JetStream and the Fuzzball DNS service reach the Compute nodes, and --http-node-port and --https-node-port don’t apply. The URLs use port 443. If your controller serves HTTPS on another port, set spec.ingress.publicPort to it in a settings file.

preflight checks that the IngressClass exists and that ingress-nginx, Kong, or Traefik serves it, and the operator configures Fuzzball’s Ingresses for that controller. If the class reports a controller that preflight doesn’t recognize, name the controller in the settings file:

spec:
  ingress:
    external:
      controller: nginx

The FuzzballOrchestrate CRD Reference describes what each controller needs, such as the proxy buffer size of nginx-based controllers. destroy and cleanup leave your controller in place.

Compute nodes that can’t reach the Kubernetes nodes

Substrate on each Compute node connects to two services in the cluster: JetStream, to register with Orchestrate and receive work, and the Fuzzball DNS service, to resolve the hostnames of jobs. With --ingress nodeport, both services listen on node ports of the Kubernetes nodes, and Substrate is given the nodes’ addresses. If the Compute nodes can’t route to those addresses, for example because the Kubernetes nodes are virtual machines on a private network, Substrate can’t register and job hostnames don’t resolve.

In that case, deploy with --ingress loadbalancer. Substrate is then given each load balancer address with the service port: 4222 for JetStream and 5353 for DNS. To have your load balancer assign addresses on a network the Compute nodes can reach, set annotations in a settings file. With MetalLB, for example, select an address pool:

spec:
  fuzzball:
    jetstream:
      externalService:
        annotations:
          metallb.io/address-pool: compute-reachable
    substrateBridge:
      dns:
        externalService:
          annotations:
            metallb.io/address-pool: compute-reachable

Allow TCP ports 4222 and 5353 to those addresses through any firewall between the Compute nodes and the load balancer. Each Compute node registers with the address it uses to reach the JetStream load balancer, which on a host with several network interfaces is the address of the interface that routes to the load balancer. Orchestrate connects back to each Compute node on that address, so the cluster must be able to reach it.

The deployment doesn’t finish, and Compute nodes can’t join, until the load balancer has assigned an address to both services. To see the assigned addresses, check the EXTERNAL-IP column, which shows a host name for load balancers that publish one, such as AWS’s:

$ kubectl get service -n fuzzball fuzzball-jetstream-external fuzzball-substrate-bridge-dns

Log in

deploy created a Fuzzball CLI context for the deployment. Switch to it and log in:

$ fuzzball context use fuzzball.example.com

$ fuzzball context login

context login launches the device-code flow in your browser against Keycloak. Sign in as the organization owner, with the email address you passed to --owner-email and the initial password that fuzzball cluster kubernetes info --secrets shows. Without --secrets, info shows the deployment’s URLs, including the web UI’s, and the CA that signs the ingress certificate when it is the operator’s own CA.

Check the deployment

Show the state of the deployment:

$ fuzzball cluster kubernetes status

status shows the FuzzballOrchestrate resource’s conditions, the Fuzzball version, and the rollout of the operator and the services it deploys. The deployment is ready when the Ready condition is True. status reports the state when you run it; it doesn’t wait for the deployment to become ready.

Then run the readiness checks. They use your Fuzzball context, so log in first:

$ fuzzball cluster kubernetes validate

The default health tier is read-only. Once a Compute node has joined, --tier functional also runs small test workloads on it. validate exits with status 1 when a required check fails.

To list the pods in any namespace that aren’t running, or to follow the operator’s log:

$ kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded

$ fuzzball cluster kubernetes logs -n fuzzball-system --component operator -f

Update the deployment

To update Fuzzball to a new version, install the Fuzzball CLI at that version on the CLI host, then run update:

$ fuzzball cluster kubernetes update --depot-user "${DEPOT_USER}"
An update to another major version runs a one-way database migration that can’t be undone. update first asks you to type the resource’s name.

update upgrades the Fuzzball Operator, sets the new version on the resource, and waits up to --timeout for the deployment to report Ready at that version. It updates to the CLI’s own version unless you pass --version, and it needs the Depot credentials, because the operator chart comes from Depot. It leaves the resource’s other settings as they are, whether deploy created the resource or you wrote it by hand.

To change settings, pass a partial FuzzballOrchestrate resource in YAML as the argument of update. update merges it over the running resource, and it may change any field but metadata.name. --dry-run prints the resource and the chart values that update would apply, without changing anything.

After an update that moves to a new Fuzzball version, or that changes the domain, the HTTPS node port, the NFS destination, the image registries, or the certificate authorities, refresh each Compute node, as Substrate Installation describes.

Remove the deployment

destroy deletes every PersistentVolumeClaim in the Fuzzball namespaces, the database included. Their data can’t be recovered.

First stop Substrate on each external Compute node with systemctl disable --now fuzzball-substrate. Then remove the deployment:

$ fuzzball cluster kubernetes destroy

destroy asks you to type the resource’s name. It deletes the FuzzballOrchestrate resource and waits for the operator to delete everything it installed: the Fuzzball services, their namespaces, and their PersistentVolumeClaims. It then uninstalls the operator, deletes namespace fuzzball-system and the CRDs of the cert-manager, trust-manager, and Kong that Fuzzball installed, and deletes the CLI context that deploy added. To run it without a prompt, for example in a script, pass --non-interactive --force. If --timeout, one hour by default, runs out before the removal finishes, the operator keeps removing the deployment; run destroy again to wait again.

destroy leaves these in place:

  • MetalLB and local-path-provisioner, if deploy installed them. destroy prints the helm uninstall commands that remove them.
  • The Kong CRDs, while another Kong still serves an IngressClass on the cluster.
  • The released PersistentVolumes of StorageClass nfs-client, whose data stays on the NFS export. destroy lists them. Delete them with kubectl delete pv, and their data on the export, when you no longer need them.

If an interrupted deploy or destroy left pieces behind, such as the Helm release or the fuzzball-system namespace, fuzzball cluster kubernetes cleanup lists them and removes them once you confirm. It refuses while a FuzzballOrchestrate resource exists.

Alternative: generate the deployment files for your own tools

deploy changes the cluster itself. If your site changes the cluster only through its own tools, such as Helm, Argo CD, Flux, or kubectl from a Git repository, the Fuzzball CLI can write the files for those tools instead. Apply them as shown below and connect the CLI, then continue at Check the deployment.

Generate the files with fuzzball cluster kubernetes deploy --dry-run --output-dir. The command builds the FuzzballOrchestrate resource from its flags and doesn’t contact the cluster, so it needs no kubeconfig. It pulls the operator chart from Depot, so it needs your Depot credentials. Run fuzzball cluster kubernetes deploy --help for the flags that set the resource.

$ fuzzball cluster kubernetes deploy --dry-run --output-dir fuzzball-deploy \
    --domain fuzzball.example.com --owner-email admin@example.com \
    --depot-user "${DEPOT_USER}" \
    --nfs-server nfs.example.com --nfs-path /srv/fuzzball/shared

The output directory must be new or empty. The command writes these files to it:

FileContents
namespace.yamlThe fuzzball-system namespace, with the baseline Pod Security profile. The chart doesn’t create it.
fuzzball-operator-<version>.tgzThe operator Helm chart, as pulled from Depot.
fuzzball-operator-values.yamlThe chart values: the operator image and the name of its pull Secret.
fuzzballorchestrate.yamlThe FuzzballOrchestrate resource.
README.mdThe order to apply the files in, the commands that apply them, and notes for Argo CD and Flux.

The values and the resource name the image pull Secret registry-image-credentials in namespace fuzzball-system, but they don’t hold your Depot credentials. Create that Secret with your site’s sealed-secrets or external-secrets tooling, of type kubernetes.io/dockerconfigjson with one auths entry for depot.ciq.com. Alternatively, add --include-secrets to write it to secrets/registry-image-credentials.yaml, in a directory only you can read. Don’t commit secrets/ to Git. With --registry, the images come from your registry and the CLI writes no pull Secret.

Apply the files with Helm

The chart holds the CRDs, so apply it before the resource. Skip kubectl apply -f secrets/ if you created the pull Secret with your own tooling, or if you passed --registry, as the CLI then writes no pull Secret.

$ cd fuzzball-deploy

$ kubectl apply -f namespace.yaml

$ kubectl apply -f secrets/

$ helm upgrade --install fuzzball-operator ./fuzzball-operator-v4.4.0-110-g8f0b4aff023d.tgz \
  -n fuzzball-system -f fuzzball-operator-values.yaml --wait

$ kubectl wait --for condition=established crd/fuzzballorchestrate.deployment.ciq.com

$ kubectl apply -f fuzzballorchestrate.yaml

Argo CD and Flux

On upgrade, the chart’s post-upgrade hook runs manager update, which migrates the stored resources to the new CRDs. A tool that renders the chart without running Helm hooks must run that step itself.

  • Argo CD maps the post-upgrade hook to a PostSync hook and runs it on every sync. Set the ServerSideApply=true sync option, since the operator’s CRDs are too large for client-side apply, and sync the chart before the resource, with sync waves or two Applications.
  • Flux: a HelmRelease runs the chart with Helm, hooks included.
  • kubectl: helm template renders the chart. Apply it with kubectl apply --server-side, since the CRDs are too large for client-side apply, and the upgrade step is then yours.

When Argo CD or Flux manages the FuzzballOrchestrate resource or the operator Deployment, fuzzball cluster kubernetes update and destroy warn that the tool may undo their changes. Change the deployment through the tool instead.

Upgrade with the deployment files

To upgrade, generate the files for the new version and apply them the same way:

  • fuzzball cluster kubernetes update --dry-run --output-dir DIR reads the running FuzzballOrchestrate resource, so it needs access to the cluster. It writes the resource at the new version, with the changes of an optional FILE merged in.
  • Without access to the cluster, run fuzzball cluster kubernetes deploy --dry-run --output-dir DIR again with the new --version and the same flags.

Connect the CLI

Once the deployment reports Ready, which fuzzball cluster kubernetes status shows, add the Fuzzball CLI context with fuzzball cluster kubernetes info --create-context. It adds the context as deploy does: named after the domain, and trusting the operator’s CA when that CA signs the ingress certificate. It leaves an existing context of that name unchanged and doesn’t switch the active context.

$ fuzzball cluster kubernetes info --create-context

$ fuzzball context use fuzzball.example.com

$ fuzzball context login
Shortly after kubectl apply creates the resource, the operator writes its defaults into it and reconciles again. For about half a minute, status can then show the deployment as not ready after it first showed Ready. Check status again before you rely on the deployment.

Next steps

You now have a running instance of Fuzzball Orchestrate. Next, follow Substrate Installation to connect Compute nodes to it.

The fuzzball cluster docker-compose deploy command brings up a single-host Fuzzball stack — nginx (TLS termination and subdomain routing), PostgreSQL, Keycloak, Fuzzball Orchestrate, and one or more Fuzzball Substrate nodes — suited to local development, evaluation, and testing. The deployment is managed entirely through the fuzzball cluster docker-compose subcommands.

For hardware-specific walkthroughs that build on this tab, see Install Fuzzball on an NVIDIA DGX Spark and Install Fuzzball on an AMD Ryzen AI Max (Strix Halo).

What the stack deploys

The compose stack brings up five services on the host:

ServiceImageRole
nginxnginx:1.27-alpineTLS edge + subdomain routing, NATS/JetStream TCP passthrough
postgrespostgres:16State for Keycloak + Orchestrate
keycloakquay.io/phasetwo/keycloak-crdb:24.0.5OIDC auth (Phase Two CockroachDB variant)
fuzzball-orchestratedepot.ciq.com/fuzzball/fuzzball-images/fuzzball-orchestrate:<tag>API + scheduler + embedded JetStream + UI
substrate-localnode-1depot.ciq.com/fuzzball/fuzzball-images/fuzzball-substrate-orchestrate:<tag>Workflow executor (use the -gpu image variant with --gpu)

The substrate runs workflow jobs as rootless OCI containers directly on the host — no Kubernetes, no cloud, no external infrastructure. On a host with a GPU, the --gpu flag swaps the substrate to the GPU-enabled image variant, which bundles the device plugins that pass the GPU through to workflow containers.

Installation

Make sure the host meets the requirements, including a working Docker installation with the Compose plugin. Then install the Fuzzball CLI and deploy the stack.

Run the commands in this tab as a regular user who can run docker without sudo, usually by being a member of the docker group. Run them all as the same user: each deployment is stored in that user’s configuration directory. The few steps that change system files, such as --update-etc-hosts, --setup-dns, and delete, run sudo themselves and prompt for your password.

Container registry access

The Fuzzball images are pulled from depot.ciq.com. If your Docker daemon isn’t already authenticated to it, log in first with the Depot credentials from the CIQ sales/support team:

$ DEPOT_USER="" # populate with your username for CIQ Depot

$ ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team
$ echo "${ACCESS_KEY}" | docker login depot.ciq.com --username "${DEPOT_USER}" --password-stdin

Fuzzball CLI

Install the fuzzball command-line tool, following the CLI installation guide — use the .deb package matching your host architecture (amd64 or arm64). Once installed, confirm the binary runs:

$ fuzzball --version

Deploy

The deploy command renders the stack’s configuration files to disk, and --up starts the containers:

$ fuzzball cluster docker-compose deploy --up

To stop and restart the stack later, see Starting and stopping a stack.

Add --dry-run to preview the generated docker-compose.yaml and .env without writing anything to disk or starting containers.

If the container images require authentication, deploy prompts for a registry user and password/token (your CIQ Depot credentials); leave the user blank to skip the registry login. You can also supply them non-interactively with --registry-user and --registry-password, and override the registry itself with --registry. Most other settings — organization name, database and Keycloak credentials, owner email — are auto-generated or defaulted and can be overridden with their corresponding flags (see deploy --help); generated secrets are recoverable later with info --secrets.

Multi-node stacks

By default the stack runs a single substrate node co-located with Orchestrate. For testing or demonstrating multi-node workflows on a single host, pass --substrate-nodes N to run N substrate nodes (default 1, max 255):

$ fuzzball cluster docker-compose deploy --substrate-nodes 2
Running multiple substrate nodes on a single host can over-provision it. This is intended for development and demonstration of multi-node workflows, not for production capacity.

Adding an external substrate node

The stack can also accept a Fuzzball Substrate node running on a separate host, for compute capacity beyond the Orchestrate host. See the Docker Compose tab of Substrate Installation for the full procedure, including the shared image cache and volume storage that external nodes use.

Keeping local changes to the stack

deploy and update --sync-files rewrite docker-compose.yaml from the CLI’s embedded copy, so edits made directly to that file are lost the next time either runs.

Put local changes in docker-compose.override.yaml in the deployment directory instead. Fuzzball never writes or regenerates that file, and applies it last, so its settings win over both the base file and any Fuzzball-managed overlay:

services:
  substrate-localnode:
    volumes:
      - /srv/fuzzball-shared:/mnt

This is the supported way to add a bind mount, pin an address, or change an image without those changes being reverted.

Keycloak memory

Keycloak runs with a fixed 1GiB Java heap inside a 1.75GiB container memory limit. To give it more, raise both values together in docker-compose.override.yaml, keeping roughly 750MiB of the limit above the heap for the JVM’s non-heap memory:

services:
  keycloak:
    environment:
      JAVA_OPTS: >-
        -Xms2048m
        -Xmx2048m
        -XX:+ExitOnOutOfMemoryError
        -XX:MetaspaceSize=96m
        -XX:MaxMetaspaceSize=256m
        -XX:+UseG1GC
        -XX:+UseStringDeduplication
        -XX:MaxGCPauseMillis=200
        -Dfile.encoding=UTF-8
        -Dsun.stdout.encoding=UTF-8
        -Dsun.err.encoding=UTF-8
        -Dstdout.encoding=UTF-8
        -Dstderr.encoding=UTF-8
        -Djava.security.egd=file:/dev/urandom
        -XX:FlightRecorderOptions=stackdepth=512
    deploy:
      resources:
        limits:
          memory: 2.75g

An override replaces JAVA_OPTS as a whole, so restate every flag, not only the heap size.

The base file also sets JAVA_OPTS_APPEND: -XX:-UseBASE64Intrinsics, which mitigates the Java4Shell vulnerability (CVE-2026-114938) in the JDK that the Keycloak image ships. Keycloak adds JAVA_OPTS_APPEND after JAVA_OPTS, so the mitigation stays in place when you override JAVA_OPTS. If you also set JAVA_OPTS_APPEND in your override, include -XX:-UseBASE64Intrinsics in it.

GPU stacks

The standard deploy brings up a CPU-only stack. On hosts that have GPUs you can opt in to a GPU-enabled substrate image. The same image bundles both the NVIDIA and AMD device plugins; the substrate automatically loads whichever matches the GPU present on the host, so one --gpu image works on either vendor:

$ fuzzball cluster docker-compose deploy --gpu [other deploy flags]

Or to update an existing deployment to the GPU-enabled substrate image:

$ fuzzball cluster docker-compose update --gpu

Prerequisites

  • The host’s GPU vendor driver must be installed and loaded (the vendor’s status tool — e.g. nvidia-smi for NVIDIA — should work on the host).
  • The vendor-specific device nodes must be present (the driver creates these at boot).

Verifying GPU registration

$ fuzzball node list --available-resources
ID              | CLUSTER   | HOSTNAME              | STATUS | CORES | MEMORY (GB) | DEVICES
172.21.0.5/7334 | local-dev | substrate-localnode-2 | Ready  | 20    | 127.9       | nvidia.com/gpu:1

If a GPU device type (for example nvidia.com/gpu:N or amd.com/gpu:N) shows up in the DEVICES column, the substrate-side plumbing is correct and the stack can run GPU workflows.

For how a workflow then requests a GPU, and for NVIDIA versus AMD ROCm specifics, see Using GPUs with Fuzzball.

Naming a deployment

By default the deployment is named default. To run more than one stack on the same host, or to give a deployment a meaningful name, pass --deployment-name:

$ fuzzball cluster docker-compose deploy --deployment-name my-dev-stack

The deployment name is used as the Docker Compose project name and as the directory name under your Fuzzball CLI config home. All of the other docker-compose subcommands accept the same --deployment-name flag to select which stack they act on.

Running more than one stack on the same host at the same time is not recommended — separate stacks compete for the same host ports and resources.

DNS resolution

The stack routes traffic to its services through subdomains of a base domain. The ui., api., keycloak., endpoints., and nats. subdomains must resolve to the host running the stack before you can reach the UI, API, or log in.

There is also a wildcard requirement. Workflow service endpoints declared with type: subdomain are served at a per-endpoint hostname of the form endpoint-<id>.endpoints.<domain>. A new hostname appears for every endpoint, so they cannot be listed ahead of time, and /etc/hosts has no way to express a wildcard. Endpoints declared with the default type: path are served under the single endpoints.<domain> name and are unaffected.

Which approach you need depends on the host:

SituationWhat to do
The site has its own DNSAdd a wildcard A record, deploy with --domain
No DNS, but the host has internet accessNothing — this is the default
Neither DNS nor internet (air-gapped)Deploy with --setup-dns

Default: a self-resolving domain

deploy defaults --domain to <host-ip>.nip.io. The nip.io zone resolves every name beneath it to the IP address embedded in the name, wildcards included, so endpoint-<id>.endpoints.10.0.0.5.nip.io resolves to 10.0.0.5 with no configuration on the host and no sudo. Names resolve both on the host itself and from other machines on the same network.

$ fuzzball cluster docker-compose deploy --up
The domain is fixed when the deployment is created — it is written into the generated TLS certificate and .env, and update cannot change it. If the host’s IP address changes, delete the deployment and deploy again. Assign the host a static or reserved address to avoid this.

Sites with existing DNS

Use your own domain and add a single wildcard A record pointing at the host:

*.fb.example.internal.  IN  A  10.0.0.5

Then deploy against it:

$ fuzzball cluster docker-compose deploy --domain fb.example.internal

This is the most robust option: it covers every client on the network with no per-machine setup, and the domain stays stable if the host’s address changes.

Air-gapped hosts

When the host has neither DNS infrastructure nor internet access, the stack can serve the wildcard zone itself:

$ fuzzball cluster docker-compose deploy --domain fb.local --setup-dns

--wildcard-dns makes Fuzzball Orchestrate answer every name under the domain with 127.0.0.1, published on 127.0.0.1:5354. The zone is authoritative only — it never forwards queries — and the published port is bound to the loopback interface, so it cannot act as a resolver for anything else on the network.

--setup-dns implies --wildcard-dns and additionally configures the host to use that zone. It requires sudo and writes two things, both removed again by delete:

  1. A split-DNS entry, so that only *.<domain> is sent to the stack and every other lookup continues to use the host’s normal upstream resolver:
    • macOS — /etc/resolver/<domain>
    • Linux with systemd-resolved — /etc/systemd/resolved.conf.d/fuzzball-<deployment>.conf
  2. An /etc/hosts entry for the fixed subdomains.

Both are required. The split-DNS entry is what makes the per-endpoint wildcard resolve, but on macOS it is only visible to programs that resolve names through the system library — browsers and curl, but not the statically linked fuzzball CLI. The /etc/hosts entry covers the fixed names for every client, including the CLI. After configuring DNS, deploy confirms that the fixed names resolve and warns if they do not.

On macOS, dig and nslookup query name servers directly and never consult /etc/resolver, so they report failure for wildcard names that resolve correctly in a browser. Use dscacheutil instead:

$ dscacheutil -q host -a name ui.fb.local

On Linux hosts without systemd-resolved there is no per-domain routing to configure, so deploy prints instructions for setting up dnsmasq rather than attempting the change itself. The equivalent configuration is:

server=/fb.local/127.0.0.1#5354

Fixed subdomains only

--update-etc-hosts adds a 127.0.0.1 entry covering the fixed subdomains, without any of the above. It requires sudo:

$ fuzzball cluster docker-compose deploy --update-etc-hosts

Omit the flag and deploy prints the line for you to add yourself:

127.0.0.1 ui.fb.local api.fb.local keycloak.fb.local endpoints.fb.local nats.fb.local

Because hosts files cannot express wildcards, this covers everything except subdomain-style workflow endpoints. It is unnecessary with the default domain, and unnecessary with --setup-dns, which writes the same entry itself.

Log in with the CLI

Before the CLI can reach the cluster you must trust the deployment’s self-signed certificate (or connect with --insecure). fuzzball cluster docker-compose info prints the service URLs, the exact command to trust the certificate, and the ready-to-paste context commands:

$ fuzzball cluster docker-compose info
Deployment:
  Name:      default
  Directory: /home/user/.config/fuzzball/docker-compose/default

Service URLs:
  UI:        https://ui.fb.local
  API:       https://api.fb.local
  Keycloak:  https://keycloak.fb.local
  Endpoints: https://endpoints.fb.local
  JetStream: nats://nats.fb.local:4222

Configuration:
  ... (deployment .env values; passwords hidden unless you pass --secrets)

TLS:
  Certs: /home/user/.config/fuzzball/docker-compose/default/certs
  To trust the deployment's wildcard certificate:
    export SSL_CERT_DIR="/home/user/.config/fuzzball/docker-compose/default/certs"

Connect:
  fuzzball context create default https://api.fb.local https://keycloak.fb.local/realms/<realm-id> fuzzball-cli
  fuzzball context login

UI:
  Open https://ui.fb.local in your browser

Run the export SSL_CERT_DIR=... line and the two fuzzball context commands exactly as printed. context login launches the device-code flow in your browser against Keycloak; sign in as the admin user admin@fb.local plus an auto-generated password (reveal it again with fuzzball cluster docker-compose info --secrets).

Trusting the TLS certificate

The deploy command generates a self-signed root CA (rootCA.pem) and a wildcard leaf certificate under the deployment’s certs/ directory. Because the CA is self-signed, browsers and CLI clients will not trust it by default, and connections to the UI and API will report a certificate warning.

To trust the certificate, either point the fuzzball CLI at the deployment’s certs/ directory or import the root CA into your operating system or browser trust store. fuzzball cluster docker-compose info prints the certs/ path and the exact command to export, for example:

$ export SSL_CERT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/certs"

If you accept the browser warnings instead of importing the root CA, accept them for the UI URL and, when you sign in through Keycloak, for the Keycloak URL the sign-in redirects to. You do not need to accept the API URL: the web UI sends its API requests, including the workflow log stream, through its own server.

The leaf certificate is valid for one year. Renew it in place without redeploying using fuzzball cluster docker-compose update --tls-renew — see Updating Fuzzball.

Starting and stopping a stack

Once a stack has been deployed, you can start and stop it without re-rendering its files:

$ fuzzball cluster docker-compose up

$ fuzzball cluster docker-compose down
down stops and removes the stack’s containers but preserves its data volumes. To additionally remove the volumes — which destroys all data in the deployment — pass down --volumes. This prompts for confirmation; add -y to skip the prompt in scripts.

Lifecycle commands

The full set of fuzzball cluster docker-compose subcommands lets you manage deployments once they exist:

SubcommandDescription
deployRender and (optionally) start a new deployment.
upStart a previously deployed stack.
downStop a stack (optionally removing its volumes).
updateUpdate an existing deployment — see Updating Fuzzball.
statusShow a deployment’s name, version, and running state.
listList all Docker Compose deployments on the host.
infoShow deployment details: service URLs, configuration, TLS trust command, how to connect, and the browser UI endpoint. Secret values (passwords, keys) are masked unless you pass --secrets.
logsShow a deployment’s logs (optionally for a single service); add -f to follow.
deleteRemove a deployment and its generated files. Prompts for confirmation (-y to skip), and offers to remove any /etc/hosts entries it added.
generate-substrate-configGenerate the configuration an external substrate node needs to join the stack — see Substrate Installation.
preflightWith node, run on an external Compute node: check the node against its installed configuration before starting Substrate — see Substrate Installation.

Run any subcommand with --help to see its full set of options.

Upgrading

Before upgrading, back up the deployment directory. Its certs/ directory holds the deployment’s root CA and its private key, and .env holds the generated passwords. If the CA key is lost, renewing the certificate creates a new CA, and every external Compute node must then be reconfigured. For the default deployment:

$ (umask 077 && tar czf ~/fuzzball-compose-default-$(date +%Y%m%d-%H%M%S).tgz \
    -C "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose" default)

umask 077 makes the archive readable only by you, and the timestamp gives each backup a new file rather than reusing an older one that may have looser permissions. Keep the archive somewhere private: it contains secrets. It doesn’t include the stack’s data, which lives in Docker volumes such as postgres-data.

To bump the stack to a newer Fuzzball image tag:

$ fuzzball cluster docker-compose update --upgrade

--upgrade moves the deployment to the version your current fuzzball CLI binary was built against; use --version <tag> to pick a specific one instead. The same update command rotates credentials (--reset-owner-password, --reset-keycloak-password, --reset-database-password) and renews the wildcard TLS leaf certificate (--tls-renew). See Updating Fuzzball and fuzzball cluster docker-compose update --help for the full set.

On a deployment created by an earlier CLI, the stack’s own substrate nodes publish an internal CA as /run/fuzzball-substrate/trusted-certs/root-ca.crt, so workflows on them cannot verify the API’s certificate (see In-Workflow API Access). Refresh the generated files while upgrading to fix it. --sync-files overwrites edits made directly to docker-compose.yaml and the other generated files, so move local changes into an override file first (see Keeping local changes to the stack):

$ fuzzball cluster docker-compose update --upgrade --sync-files

If the update’s planned changes do not list IMAGE_TAG – the deployment was already on this version – the update can leave the stack’s own substrate nodes running with the old CA. Restart the stack with fuzzball cluster docker-compose down and then up to finish.

Upgrades keep the original default provisioner

A compose deployment scopes its default storage provisioner to the docker segment, so it is offered only to the compose-managed substrate containers that can actually reach the shared Docker volume behind it. A node added to the cluster by hand never joins that segment and is therefore never offered storage it cannot see.

Upgrading does not apply this to an existing deployment. The scoped provisioner is created under a new name, default-<segment>, and a provisioner’s identity is derived from its name, so it is a new record rather than a rename – renaming would separate the original provisioner from every volume created on it. The upgrade therefore leaves the original unsegmented default in place and adds default-docker alongside it.

Nothing that worked before the upgrade stops working: default keeps its volumes and keeps serving the compose nodes. But because it carries no segment, it is still advertised to every node in the cluster, including hand-added ones, so a workflow that selects it and runs on such a node still fails to start with a missing-directory error.

A fresh deployment is the only way to get the scoping applied automatically. If reinstalling is not practical, retire the old provisioner by hand instead:

  1. Create the volumes you want to keep on default-docker, and copy their data across.

  2. Remove the original:

    $ fuzzball volume provisioner remove default

Until one of those happens, treat default as usable only from the compose-managed nodes. Orchestrate logs a warning at startup whenever it finds an unsegmented default provisioner alongside a segmented one.

Whether the two provisioners share a directory depends on the release that created default-docker. Check with fuzzball volume provisioner info default-docker:

  • It shows a subPath. default-docker keeps its volumes in the organization’s own subdirectory of the shared volume, and default uses the root. A volume on one is not visible through the other, which is why step 1 recreates each volume on default-docker and copies its data.
  • It shows no subPath. Both provisioners serve the same directory, so a volume name that already exists there is refused on the other with an “already exists” error rather than quietly sharing a directory. Deleting a volume through one provisioner removes the data a matching record on the other points at; a fuzzball volume provisioner scan clears the stale record.

Tear down

Stop the stack while preserving its persistent volumes (postgres-data, jetstream-data, substrate-shared-fs, etc.) — useful when you want to come back to your workflows later:

$ fuzzball cluster docker-compose down

To completely remove the deployment, including its generated files and persistent state:

$ fuzzball cluster docker-compose delete

Troubleshooting

When a Docker Compose deployment misbehaves, it’s often useful to inspect the generated stack files or to drive docker compose directly. The CLI writes each deployment into its own directory under your Fuzzball CLI config home and runs docker compose against it on your behalf, so you can do the same thing manually.

pull access denied for depot.ciq.com/...

Docker isn’t authenticated to the registry. Log in as shown in Container registry access, then re-run the deploy.

Locating the deployment directory

Each deployment is stored in a subdirectory named after the deployment, under the docker-compose folder in your Fuzzball CLI config home:

${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/<deployment-name>/
If XDG_CONFIG_HOME is set in your environment, the directory is rooted there ($XDG_CONFIG_HOME/fuzzball/docker-compose/...); otherwise it falls back to $HOME/.config/fuzzball/docker-compose/.... Docker compose deployments always use this per-user config directory location.

The directory holds everything the stack needs:

  • docker-compose.yaml — the generated Compose file describing every service in the stack.
  • .env — the environment file with values such as IMAGE_TAG (the deployed Fuzzball version) that the Compose file substitutes in. Set PULL_POLICY=missing there to stop the Orchestrate and substrate containers from pulling their images on every start, for example when restarting a host that is offline; the default is always.
  • certs/ — TLS material generated for the deployment.

Running docker compose commands directly

The CLI invokes docker compose with the deployment name as the Compose project name (-p) and with the deployment directory as the working directory. To reproduce that yourself, change into the deployment directory and pass the same -p <deployment-name> flag. For the default deployment:

$ cd "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/"

List the containers and their state (equivalent to what fuzzball cluster docker-compose status inspects):

$ docker compose -p default ps

Tail the logs of a single service — for example, Orchestrate:

$ docker compose -p default logs -f fuzzball-orchestrate

Inspect the fully-rendered Compose configuration with .env values substituted in, which is helpful for confirming the image tag and other resolved settings:

$ docker compose -p default config
Always pass -p <deployment-name> so your manual docker compose commands act on the same project the CLI manages. For any deployment created with --deployment-name <name>, substitute that name for default in both the directory path and the -p flag above.

Web UI login fails while the CLI still works

If browser logins fail but fuzzball commands keep succeeding, suspect the Keycloak realm rather than the user’s password. The two paths authenticate differently: the CLI uses a device flow against the public fuzzball-cli client and talks to Keycloak directly, while the Web UI needs the confidential fuzzball-ui client and looks it up through the organization service. A realm or client that has gone missing therefore breaks the browser while leaving the CLI working.

Fuzzball creates both clients in each realm when the organization is first bootstrapped. Neither has Keycloak service accounts enabled, so a grant_type=client_credentials request always fails against them and cannot be used to test whether a client is healthy. Use the two checks below instead.

First, confirm the realm exists. This needs no credentials:

$ curl -sk -o /dev/null -w '%{http_code}\n' \
    https://<host>/auth/realms/<realm>/.well-known/openid-configuration

200 means the realm is present; 404 means it is missing, which breaks every browser login in that organization.

If the realm is present, check that its fuzzball-ui client is too. This uses the Keycloak admin credentials from the deployment’s .env:

$ TOKEN=$(curl -sk -X POST https://<host>/auth/realms/master/protocol/openid-connect/token \
    -d grant_type=password -d client_id=admin-cli \
    -d username=<keycloak-admin> -d password=<keycloak-admin-password> \
    | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
$ curl -sk -H "Authorization: Bearer $TOKEN" \
    "https://<host>/auth/admin/realms/<realm>/clients?clientId=fuzzball-ui"

A JSON array containing a client object means it is present. An empty array ([]) means it is missing, which is the condition that breaks Web UI logins while the CLI keeps working.

Orchestrate records the same condition at startup, which is the quickest way to confirm what the two checks above found. When it cannot reach the realm or its fuzzball-ui client, it logs an error containing failed to bootstrap default organization:

$ docker compose -p default logs fuzzball-orchestrate | grep "bootstrap default organization"

If that line is present, recreate the realm and its clients rather than rotating secrets.

Keycloak realm and client settings come from Orchestrate’s configuration, not from the Compose file. A realm identifier written into docker-compose.yaml has no effect on which realm the deployment uses.

Beyond a single host

A multi-host topology — this host plus remote substrate nodes — is supported via generate-substrate-config, which exports the orchestrate’s CA, NATS NKey, and connection config for deployment to a separate machine. See --help on that command and Federation for the cross-cluster story.

Hardening for production (real TLS via cert-manager / Caddy, external managed Postgres + Keycloak, backup automation, log shipping) is covered in Cloud Deployment — most of the recommendations there apply equally to a single-host deployment when you want to move beyond local development.