Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

FuzzballOrchestrate CRD Reference

The FuzzballOrchestrate Custom Resource Definition (CRD) is the primary way to deploy and configure a Fuzzball cluster on Kubernetes. This CRD provides comprehensive control over all aspects of a Fuzzball deployment, from basic infrastructure (database, ingress, authentication) to advanced features (multi-cloud provisioning, billing integration, autoscaling).

Basic Example

The minimal configuration requires image credentials, ingress configuration, database setup, and authentication. This example is suitable for local or on-premises deployments:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  ingress:
    create:
      domain: "10.0.0.99.nip.io"
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.0.99
  database:
    create:
      storage:
        class: longhorn
  keycloak:
    create:
      ingress:
        hostname: auth.10.0.0.99.nip.io
      realmName: Fuzzball
      realmId: 550e8400-e29b-41d4-a716-446655440000
      username: keycloak
      ownerEmail: "admin@example.com"
      createDatabase: true
  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
  fuzzball:
    substrate:
      nfs:
        destination: "/fuzzball/shared"
        server: 10.0.0.10
        path: "/srv/fuzzball/shared"
    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 10Gi

Configuration Sections

Many credential fields (passwords, API keys, TLS certificates, image pull secrets) support references to Kubernetes Secrets as an alternative to inline values. Secret references are recommended for production deployments. See Kubernetes Secret References for the full list of fields and usage examples.

Image Configuration

Controls where container images are pulled from and authentication:

ParameterTypeRequiredDefaultDescription
repositorystringNodepot.ciq.com/fuzzball/fuzzball-imagesContainer image registry
credentialsSecretNamestringNo-Name of a kubernetes.io/dockerconfigjson Secret with registry credentials (recommended)
usernamestringNo-Registry authentication username (deprecated — use credentialsSecretName)
passwordstringNo-Registry authentication password (deprecated — use credentialsSecretName)
exclusivebooleanNotrueIf true, all images must come from specified repository
pullPolicystringNoAlwaysimagePullPolicy for the Fuzzball services, SpiceDB, and the Fuzzball jobs and sidecars: Always, IfNotPresent, or Never

Example with inline credentials:

spec:
  image:
    repository: depot.ciq.com/fuzzball/fuzzball-images
    username: my-depot-user
    password: my-depot-token
    exclusive: false

Example with Secret reference (recommended for production):

spec:
  image:
    credentialsSecretName: depot-pull-secret
    exclusive: false

With the default pullPolicy: Always, every pod start contacts the registry, so a pod that restarts while the registry is unreachable fails with ImagePullBackOff even when the image is already on the node. Set pullPolicy: IfNotPresent to start pods from images already on the node, for example on an air-gapped or intermittently connected site:

spec:
  image:
    pullPolicy: IfNotPresent

This setting does not cover other third-party components (PostgreSQL, Keycloak, the ingress controller, cert-manager, Prometheus, Alertmanager, and Grafana), the OCI DNS webhook, or CoreWeave substrate pods. These keep their own pull policies. It also does not cover the operator’s own Deployment, which the fuzzball-operator Helm chart deploys. Set its pull policies with the chart values image.pullPolicy and kubeRbacProxyImage.pullPolicy.

Ingress Configuration

Defines how the cluster is exposed to the network. Choose either create (new ingress) or external (existing ingress controller).

Option A: Create New Ingress

ParameterTypeRequiredDefaultDescription
domainstringYes-Base domain for all Fuzzball services
proxy.typestringYesLoadBalancerService type: LoadBalancer or NodePort
proxy.annotationsmapNo{}Annotations for the proxy service (e.g., MetalLB config)
proxy.http.nodePortintegerYes (if proxy.type is NodePort)-NodePort for HTTP
proxy.tls.nodePortintegerYes (if proxy.type is NodePort)-NodePort for HTTPS

Example with LoadBalancer:

spec:
  ingress:
    create:
      domain: fuzzball.example.com
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.0.99
          metallb.io/allow-shared-ip: ingress-and-fuzzball
CoreWeave LoadBalancer Annotations

CoreWeave requires specific annotations on the LoadBalancer service for automatic DNS configuration:

AnnotationValueDescription
service.beta.kubernetes.io/coreweave-load-balancer-typepublicCreates an internet-accessible load balancer
service.beta.kubernetes.io/external-hostname*.<domain>Wildcard domain for automatic DNS resolution

CoreWeave example:

spec:
  ingress:
    create:
      domain: a1b2c3-my-cluster.coreweave.app
      proxy:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/coreweave-load-balancer-type: public
          service.beta.kubernetes.io/external-hostname: '*.a1b2c3-my-cluster.coreweave.app'

Example with NodePort:

spec:
  ingress:
    create:
      domain: fuzzball.example.com
      proxy:
        type: NodePort
        http:
          nodePort: 30080
        tls:
          nodePort: 30443

Option B: Use External Ingress

ParameterTypeRequiredDefaultDescription
domainstringYes-Base domain for all Fuzzball services
classNamestringYes-Ingress class name (e.g., nginx, traefik)
controllerstringNoDetectedIngress controller that serves className: nginx (ingress-nginx), kong, or traefik
annotationsmapNo{}Annotations for ingress resources

The operator adds controller-specific settings to the ingresses it creates. For example, it adds the gRPC backend protocol for ingress-nginx. To choose these settings, it reads the controller from the IngressClass named by className. A class that isn’t found, or whose controller the operator doesn’t recognize, is treated as if its name were the controller. Set controller when detection gets it wrong, for example when an ingress-nginx class reports a controller value that does not contain ingress-nginx.

With Traefik, the operator serves the gRPC API through a Traefik IngressRoute and sets no kubernetes.io/ingress.class annotation on it, so a Traefik whose CRD provider has no ingressClass filter picks it up. If your Traefik filters CRD resources by class (providers.kubernetesCRD.ingressClass), set the annotation to that filter’s value in spec.fuzzball.config.apiIngress.annotations. These annotations go on the gRPC API ingress for every controller. Don’t set kubernetes.io/ingress.class here for nginx or Kong: Kubernetes rejects an Ingress whose class annotation differs from its ingressClassName.

spec:
  fuzzball:
    config:
      apiIngress:
        annotations:
          kubernetes.io/ingress.class: traefik-internal

Example:

spec:
  ingress:
    external:
      domain: fuzzball.example.com
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod

Sign-in responses from the Web UI and Keycloak carry cookies that are larger than the default 4 KB header buffer of ingress-nginx, which answers with a 502 error. On the nginx class, the operator sets nginx.ingress.kubernetes.io/proxy-buffer-size to 32k on the Web UI ingress and to 128k on the Keycloak ingress. To use a different size, set the same annotation in spec.fuzzball.ui.ingress.annotations or spec.keycloak.create.ingress.annotations. For any other nginx-based controller, raise its proxy buffer size to at least these values.

Public Port

ParameterTypeRequiredDefaultDescription
publicPortintegerNo443Port clients use to reach the ingress hostnames. Set it when clients connect to the NodePort directly. Setting 443 is the same as leaving it unset

Fuzzball puts this port in the URLs it gives to clients: the Keycloak issuer and OIDC endpoints, the UI and API URLs, and the URLs that compute nodes use to reach the cluster. With proxy.type: NodePort and nothing in front of the node port, set it to proxy.tls.nodePort. Otherwise the CLI and the UI are sent to port 443 and the login fails. Leave it unset when a load balancer or port forward serves the ingress on port 443. It applies to both create and external ingress. It does not change an external Keycloak (spec.keycloak.external), whose issuer comes from its own URL.

spec:
  ingress:
    publicPort: 30443
    create:
      domain: fuzzball.example.com
      proxy:
        type: NodePort
        http:
          nodePort: 30080
        tls:
          nodePort: 30443

Keep the following in mind:

  • Some Fuzzball jobs inside the cluster also use these URLs, so the ingress hostnames must also resolve inside the cluster to an address where the node port is open, such as a node IP.
  • Service endpoint URLs and the monitoring node-scrape discovery URL still use port 443, so they are not reachable through a bare node port.
  • Setting or changing publicPort on a running cluster changes the token issuer. Existing sessions and CLI contexts stop working: users log in again, and CLI contexts are recreated with the port. Any other cluster that lists this cluster’s Keycloak in additionalTrustedIssuers must include the port, for example https://auth.fuzzball.example.com:30443/.
ParameterTypeRequiredDefaultDescription
spec.fuzzball.ui.cookieScopestringNoautoWhich hostnames a browser sends the Web UI login cookie to: auto, domain, or host

The Web UI keeps the login token in a browser cookie. A browser only stores a cookie that is shared across a domain when the page that set it is served from that domain, so the scope has to fit the hostname users open the Web UI on:

  • auto shares the cookie across the ingress domain when the Web UI hostname is that domain or under it, for example ui.fuzzball.example.com with the domain fuzzball.example.com. For any other Web UI hostname, such as fuzzball-ui.example.com, it keeps the cookie on the Web UI hostname.
  • domain always shares the cookie across the ingress domain. If the Web UI hostname is not under that domain, the browser discards the cookie and every login returns to the login page.
  • host always keeps the cookie on the Web UI hostname.

When the cookie stays on the Web UI hostname, browsers do not send it to workflow service endpoints, so opening a service endpoint in a browser does not pick up the Web UI login.

auto decides from the configured spec.fuzzball.ui.ingress.hostname, not from the hostname in the browser’s address bar. If a reverse proxy serves the Web UI under a different hostname that is not under the ingress domain, set cookieScope: host.

A change to cookieScope applies to each user at their next login.

spec:
  fuzzball:
    ui:
      cookieScope: host

Database Configuration

Fuzzball requires a PostgreSQL database. Choose either create (deploys a dedicated PostgreSQL alongside Orchestrate) or external.

Option A: Create Dedicated Database

ParameterTypeRequiredDefaultDescription
enableDebugPodbooleanNofalseDeploy a debug pod with database tools
annotationsmapNo{}Annotations for database resources
storage.classstringNo-StorageClass for database PVC
storage.sizestringNo-Size of database storage
storage.accessModestringNoReadWriteOnceAccess mode for PVC
storage.annotationsmapNo{}Annotations for PVC

Example:

spec:
  database:
    create:
      enableDebugPod: true
      storage:
        class: longhorn
        size: 200Gi
        accessMode: ReadWriteOnce

Option B: Use External Database

ParameterTypeRequiredDefaultDescription
hoststringYes-Database hostname or IP
portstringNo5432Database port
driverstringNopostgresDatabase driver
credentials.userstringYes-Database username
credentials.passwordstringNo-Database password (inline)
credentials.passwordRefobjectNo-Reference to a K8s Secret containing the database password (preferred over inline)
sslModestringNoverify-fullSSL mode: disable, require, verify-ca, verify-full
rdsSecretIdstringNo-AWS Secrets Manager ARN for RDS credentials
certificate.caCertstringNo-CA certificate for SSL verification (inline)
certificate.caCertURLstringNo-URL to download CA certificate
certificate.caCertRefobjectNo-Reference to a K8s Secret containing the CA certificate (preferred over inline)
certificate.clientCertstringNo-Client certificate for mTLS (inline)
certificate.clientCertRefobjectNo-Reference to a K8s Secret containing the client certificate (preferred over inline)
certificate.clientKeystringNo-Client key for mTLS (inline)
certificate.clientKeyRefobjectNo-Reference to a K8s Secret containing the client key (preferred over inline)

Example with external PostgreSQL:

spec:
  database:
    external:
      host: postgres.example.com
      port: "5432"
      driver: postgres
      credentials:
        user: fuzzball
        passwordRef:
          name: db-credentials
          key: password
      sslMode: verify-full

Example with AWS RDS:

spec:
  database:
    external:
      host: mydb.abc123.us-east-1.rds.amazonaws.com
      rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:rds-db-credentials-abc123
      sslMode: verify-full
      certificate:
        caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Keycloak Authentication

For Keycloak-backed deployments, Fuzzball delegates authentication to Keycloak. Choose either create (deploy Keycloak alongside Orchestrate) or external. As of v3.3.0, Keycloak is optional — deployments that use the local-organization flow authenticate directly with Fuzzball without a Keycloak instance.

Option A: Deploy Keycloak

As of v3.3.0, plaintext password and defaultUserPassword fields are deprecated. Use the passwordRef and defaultUserPasswordRef fields to reference passwords stored in Kubernetes Secrets. The plaintext fields continue to work but will be removed in a future release.
ParameterTypeRequiredDefaultDescription
ownerEmailstringYes-Email address for the Fuzzball organization owner. Must not be empty — see Owner email is required
realmIdstringNo(generated)Keycloak realm ID (must be UUID v4, lower case)
realmNamestringNoFuzzballKeycloak realm display name
usernamestringNokeycloakKeycloak admin username
passwordstringNo(generated)Deprecated. Keycloak admin password (plaintext). Use passwordRef instead.
passwordRef.namestringNo-Name of Kubernetes Secret containing the Keycloak admin password
passwordRef.keystringNo-Key within the Secret for the Keycloak admin password
passwordRef.namespacestringNofuzzball-systemNamespace of the Secret; defaults to the Fuzzball system namespace
defaultUserPasswordstringNo-Deprecated. Initial password for the organization owner (plaintext). Use defaultUserPasswordRef instead.
defaultUserPasswordRef.namestringNo-Name of Kubernetes Secret containing the default user password
defaultUserPasswordRef.keystringNo-Key within the Secret for the default user password
defaultUserPasswordRef.namespacestringNofuzzball-systemNamespace of the Secret; defaults to the Fuzzball system namespace
createDatabasebooleanNotrueCreate dedicated database for Keycloak
replicasintegerNo1Number of Keycloak replicas
resources.requestsmapNo{memory: 3Gi}Memory (and optionally CPU) requested per Keycloak pod. See the sizing note below before changing.
resources.limitsmapNo{memory: 3Gi}Memory (and optionally CPU) ceiling per Keycloak pod. Keep equal to resources.requests.
ingress.hostnamestringNoauth.{domain}Hostname for Keycloak UI
ingress.classNamestringNo-Ingress class name
ingress.tls.certstringNo-TLS certificate (inline)
ingress.tls.keystringNo-TLS private key (inline)
ingress.tls.secretNamestringNo-Name of a kubernetes.io/tls Secret containing the TLS cert and key (preferred over inline)
ingress.annotationsmapNo{}Ingress annotations

Sizing Keycloak memory. resources.requests and resources.limits default to the same value on purpose. Setting a memory request at all takes the Keycloak pod out of BestEffort QoS, so the kubelet no longer treats it as the first candidate to evict; the limit gives the JVM a container boundary to size its heap from (roughly 2.1GiB of heap under a 3Gi limit); and keeping the two equal means usage can never exceed the request, which puts the pod in the kubelet’s last-to-evict group. Deployments that ship without them can drift to several GiB per pod and be evicted or exit under memory pressure – which, on a single-replica deployment, takes every in-flight login session with it. Note this is Burstable QoS, not Guaranteed, and it is ranked last for eviction rather than immune to it.

At the 3Gi default, each Keycloak replica reserves 3Gi of schedulable memory – 9Gi across a three-replica deployment. Confirm your nodes have that to give before upgrading, or the pods will stay Pending. Small local clusters should lower both values together.

Always override requests and limits together, and always restate memory in each. The two default independently, and setting either one replaces it entirely rather than merging with the default. Two mistakes are easy to make:

  • Moving only one map. Lowering just limits to 2Gi leaves requests at the 3Gi default, and Kubernetes rejects the pod outright: spec.containers[0].resources.requests: Invalid value: "3Gi": must be less than or equal to memory limit of 2Gi. Raising only requests above 3Gi fails the same way against the default limit. The Keycloak StatefulSet will not update until both agree.
  • Dropping memory from limits. This is the quietest way to undo the fix. Adding a CPU limit as limits: {cpu: "500m"} replaces the whole map, leaving a 3Gi memory request and no memory limit. Nothing catches it – the API server accepts it, the pod schedules, and the JVM goes back to sizing its heap from node memory. Kubernetes copies a limit into a matching missing request but never the reverse, so dropping memory from requests alone is harmless while dropping it from limits alone is not. Do not rely on that asymmetry: restate memory in both.

Override both, in full, at the path they live under:

spec:
  keycloak:
    create:
      resources:
        requests:
          cpu: "500m"
          memory: "3Gi"
        limits:
          cpu: "500m"
          memory: "3Gi"

Changing these values updates the Keycloak StatefulSet pod template, which performs a rolling restart. What that costs depends on replicas:

  • replicas: 1 (the default): the single pod holds every session, so a restart drops them all and any login in progress must start over.
  • Multiple replicas: Keycloak clusters its Infinispan caches across pods and replicates the session caches to two owners, so a rolling restart that replaces one pod at a time preserves them. A user whose request is in flight against the pod being replaced may see one failed request, but their session survives on another owner.

Sessions are never persisted to storage in either case, so losing all replicas at once loses every session regardless of this setting.

Preferred example using SecretKeyReference:

spec:
  keycloak:
    create:
      ownerEmail: admin@example.com
      realmName: MyOrganization
      realmId: 550e8400-e29b-41d4-a716-446655440000
      username: keycloak
      passwordRef:
        name: keycloak-credentials
        key: admin-password
      defaultUserPasswordRef:
        name: keycloak-credentials
        key: default-user-password
      createDatabase: true
      ingress:
        hostname: auth.fuzzball.example.com

Initial Setup

Generating a Realm ID

Generate a UUID v4 for the realm ID:

$ uuidgen

Use the output in your configuration:

realmId: 550e8400-e29b-41d4-a716-446655440000
Setting the Owner Account

The ownerEmail and defaultUserPassword fields create the initial administrator account with full cluster administration privileges. Change the default password after first login.

ownerEmail: admin@example.com
defaultUserPassword: initial-secure-password

LDAP Integration

Keycloak can be configured to federate users from an LDAP/Active Directory server:

ParameterTypeRequiredDescription
ldap.urlstringYesLDAP server URL (ldap:// or ldaps://)
ldap.startTLSbooleanNoUse StartTLS for ldap:// connections
ldap.insecurebooleanNoSkip TLS certificate verification
ldap.vendorstringNoLDAP vendor: ad, rhds, tivoli, edirectory, other
ldap.bindDNstringNoBind DN for LDAP authentication
ldap.bindPasswordstringNoPassword for bind DN (inline)
ldap.bindPasswordRefobjectNoReference to a K8s Secret containing the bind password (preferred over inline)
ldap.searchScopestringNoSearch scope: single or subtree
ldap.users.dnstringYesBase DN for user search
ldap.users.attributes.*stringYesUser attribute mappings
ldap.users.objectClasses[]stringYesUser object classes
ldap.users.filterstringNoLDAP filter for users
ldap.groups.dnstringYesBase DN for group search
ldap.groups.membershipAttributeTypestringYesMembership type: dn or uid
ldap.groups.userGroupsStrategystringYesStrategy for loading groups
ldap.groups.attributes.*stringYesGroup attribute mappings
ldap.groups.objectClasses[]stringYesGroup object classes
ldap.groups.filterstringNoLDAP filter for groups

LDAP example:

spec:
  keycloak:
    create:
      ownerEmail: admin@example.com
      ldap:
        url: ldaps://ldap.example.com
        vendor: ad
        bindDN: cn=fuzzball,ou=service,dc=example,dc=com
        bindPassword: ldap-bind-password
        searchScope: subtree
        users:
          dn: ou=users,dc=example,dc=com
          attributes:
            username: sAMAccountName
            rdn: sAMAccountName
            uuid: objectGUID
            uidNumber: uidNumber
            gidNumber: gidNumber
          objectClasses:
            - person
            - organizationalPerson
          filter: "(memberOf=cn=fuzzball-users,ou=groups,dc=example,dc=com)"
        groups:
          dn: ou=groups,dc=example,dc=com
          membershipAttributeType: dn
          userGroupsStrategy: get_groups_from_user_memberof_attribute
          attributes:
            groupName: cn
            groupMembership: member
            userMembership: sAMAccountName
            memberOf: memberOf
            gidNumber: gidNumber
          objectClasses:
            - group
          filter: "(cn=fuzzball-*)"

Option B: Use External Keycloak

ParameterTypeRequiredDescription
urlstringYesKeycloak server URL
realmIdstringYesExisting realm ID (UUID v4)
realmNamestringYesExisting realm name
usernamestringYesAdmin username
passwordstringNoAdmin password (inline)
passwordRefobjectNoReference to a K8s Secret containing the admin password (preferred over inline)
ownerEmailstringYesOrganization owner email. Must not be empty — see Owner email is required
useLDAPFederationbooleanNoWhether LDAP is configured

Example:

spec:
  keycloak:
    external:
      url: https://keycloak.example.com
      realmId: 550e8400-e29b-41d4-a716-446655440000
      realmName: ExistingRealm
      username: admin
      passwordRef:
        name: keycloak-credentials
        key: admin-password
      ownerEmail: admin@example.com

Owner email is required

ownerEmail must name a real user, on both the create and external Keycloak variants. It becomes the owner of the cluster’s initial organization, so an empty value leaves that organization with no administrator.

The operator enforces this at two different strengths, according to whether there is a running cluster to protect:

  • A resource that has never completed a deploy is rejected. Reconciliation stops, the resource goes Degraded, and a Misconfigured condition names the field to fix. This covers a resource being applied for the first time and one whose first deploy failed — in both cases nothing is serving, so there is nothing an outright refusal can break.
  • A resource that has deployed successfully at least once keeps reconciling. An administrator may already have added an owner by hand, so an upgrade is not blocked. A Misconfigured condition and a warning event record the problem instead. This still applies when the most recent deploy failed: the previous deploy’s workloads are still running.

Correcting the field clears the condition on the next reconciliation.

TLS Configuration

Optional configuration for certificate management and Let’s Encrypt:

cert-manager Deployment

ParameterTypeRequiredDescription
certManager.createobjectNoSet to {} to have the operator deploy cert-manager. Omit create (an empty certManager: {} does not deploy) to use an existing installation
certManager.create.serviceAccount.annotationsmapNoService account annotations (for IRSA)
certManager.clusterResourceNamespacestringNoNamespace where an existing cert-manager reads ClusterIssuer secrets (default: cert-manager). Cannot be combined with certManager.create
trustManager.createobjectNoSet to {} to have the operator deploy trust-manager. Omit create to use an existing installation

When no issuer is configured, the operator generates a self-signed internal CA and issues both internal and ingress certificates from it.

Certificate Issuers

ParameterTypeRequiredDescription
internalIssuerobjectNoInternal certificate issuer configuration
ingressIssuer.createobjectNoCreate an ingress issuer. An empty create: {} uses the internal issuer, the same as omitting it
ingressIssuer.create.letsEncrypt.emailstringNoEmail for Let’s Encrypt notifications
ingressIssuer.create.letsEncrypt.issuerstringNoIssuer name (e.g., letsencrypt-prod, letsencrypt-staging)
ingressIssuer.create.letsEncrypt.solvers[]objectNoACME challenge solvers (dns01 or http01)
ingressIssuer.external.issuerNamestringNoName of existing ClusterIssuer for ingress TLS (preferred)
ingressIssuer.external.internalCAIssuerNamestringNoName of existing ClusterIssuer for ingress TLS (deprecated, use issuerName)
internalIssuer.external.issuerNamestringNoName of existing ClusterIssuer for internal TLS (preferred)
internalIssuer.external.internalCAIssuerNamestringNoName of existing ClusterIssuer for internal TLS (deprecated, use issuerName)
internalIssuer.create.customCA.inline.secretNamestringNoName of a kubernetes.io/tls Secret in cert-manager’s cluster resource namespace (certManager.clusterResourceNamespace, default cert-manager) providing the CA cert/key for the internal issuer (otherwise the operator generates a self-signed CA)
ingressIssuer.create.customCA.inline.secretNamestringNoName of a kubernetes.io/tls Secret in cert-manager’s cluster resource namespace (certManager.clusterResourceNamespace, default cert-manager) providing the CA cert/key for the ingress issuer

A customCA must set either inline.secretName or, deprecated, both inline.cert and inline.key. A non-empty secretName takes precedence when both are set. The operator reads the Secret when it reconciles; editing the Secret alone does not start a reconcile. To roll out a new CA, create a new Secret and point secretName at it; the operator replaces its copy of the CA. Certificates already issued keep the old CA until they are renewed. For the ingress issuer, renew them with cmctl:

$ kubectl get certificates -A \
    -o jsonpath='{range .items[?(@.spec.issuerRef.name=="ingress-ca-issuer")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' \
    | while read ns name; do cmctl renew -n "$ns" "$name"; done
When using external issuers, the operator can automatically extract the CA certificate from the ClusterIssuer and distribute it via trust-manager, assuming trust-manager is running. Enable trustManager.create so the operator deploys it, or install trust-manager separately. When trust-manager is available, manual creation of the trusted-ca-certs-bundle ConfigMap is no longer required. trustManager.create can be set independently of certManager.create when using external issuers.

Example with Let’s Encrypt (DNS-01):

spec:
  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@example.com
          issuer: letsencrypt-prod
          solvers:
            - dns01:
                route53:
                  region: us-east-1
                  hostedZoneID: Z1234567890ABC

Example with your own CA:

Store the CA certificate and private key in a kubernetes.io/tls Secret in the cert-manager namespace, creating the namespace first if it does not exist yet. With an existing cert-manager and certManager.clusterResourceNamespace set, create the Secret in that namespace instead:

$ kubectl create namespace cert-manager
$ kubectl create secret tls my-root-ca \
    --cert=path/to/ca.crt \
    --key=path/to/ca.key \
    -n cert-manager

Keep a copy of the CA outside the cluster: with certManager.create set, the operator manages the cert-manager namespace and deleting the Fuzzball custom resource removes it, including this Secret.

Then reference it from either issuer. This example sets the ingress issuer; use internalIssuer.create.customCA for the internal CA:

spec:
  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
    ingressIssuer:
      create:
        customCA:
          inline:
            secretName: my-root-ca

Example with Let’s Encrypt (HTTP-01):

spec:
  tls:
    certManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@example.com
          issuer: letsencrypt-prod
          solvers:
            - http01:
                ingress:
                  class: nginx

Email Configuration

The optional email section configures the SMTP relay used to send signup verification emails. When omitted, verification tokens are written to the Fuzzball agent’s stdout instead — useful for development but not suitable for production use of the self-signup feature.

ParameterTypeRequiredDescription
email.smtpHoststringYesHostname of the SMTP relay
email.smtpPortintegerYesTCP port of the SMTP relay (587 for STARTTLS, 465 for TLS, 25 for plaintext)
email.fromAddressstringYesEnvelope sender address (e.g., noreply@example.com)
email.tlsModestringNoConnection security: none, starttls (default), or tls
email.credentials.secretNamestringYes (if credentials set)Name of the Kubernetes Secret containing SMTP credentials
email.credentials.usernameKeystringNoKey in the Secret for the SMTP username (default: username)
email.credentials.passwordKeystringNoKey in the Secret for the SMTP password (default: password)

Example — unauthenticated relay:

spec:
  email:
    smtpHost: mailrelay.internal
    smtpPort: 25
    fromAddress: noreply@example.com
    tlsMode: none

Example — STARTTLS with credentials (e.g., AWS SES):

spec:
  email:
    smtpHost: email-smtp.us-east-1.amazonaws.com
    smtpPort: 587
    fromAddress: noreply@example.com
    tlsMode: starttls
    credentials:
      secretName: smtp-credentials
When email is omitted, signup verification tokens are written to the Fuzzball agent’s stdout. An administrator can retrieve the token from the logs and share it with the user out-of-band. This fallback is intended for development deployments only.

Kubernetes Scheduling Configuration

Optional top-level fields that constrain which Kubernetes nodes Fuzzball service pods are scheduled onto. All three fields apply globally to every Fuzzball service pod as defaults, and each can be overridden per service component (see Service-Specific Configuration). A per-service override replaces (does not merge with) the global value for that service.

ParameterTypeRequiredDefaultDescription
nodeSelectormapNo{}Node label selector applied to all Fuzzball service pods; overridable per component
tolerations[]objectNo[]Tolerations applied to all Fuzzball service pods; overridable per component
affinityobjectNo-Affinity/anti-affinity rules applied to all Fuzzball service pods; overridable per component

Some operator-managed workloads have no scheduling fields of their own and inherit them from a related service:

  • Catalog jobs (catalog sync and load-default-catalog) use the agent service’s scheduling
  • The resource-definition setup job uses the orchestrator service’s scheduling
  • The sync workload uses the substrateBridge service’s scheduling
  • The SpiceDB authorization database uses the auth service’s scheduling

The supporting services the operator deploys through Helm charts receive only the global values. When database.create or keycloak.create is set, the top-level nodeSelector, tolerations, and affinity are passed through to the PostgreSQL and Keycloak charts. The optional monitoring stack does not use these fields.

All three fields are optional and were added without changing any defaults, so resources created before they existed deploy exactly as before after an upgrade. Adding, changing, or removing a value triggers a rolling restart of every affected pod. Pods backed by node-local persistent volumes stay on their current node after the restart; see the JetStream example below.

Example: dedicated node pool

Constrain all Fuzzball service pods to nodes labeled workload: fuzzball:

spec:
  nodeSelector:
    workload: fuzzball

Example: tolerating a tainted node

Allow Fuzzball pods to run on nodes tainted fuzzball=system:NoSchedule:

spec:
  tolerations:
    - key: fuzzball
      value: system
      operator: Equal
      effect: NoSchedule

Example: node affinity

Require Fuzzball pods to land in a specific availability zone:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - us-east-1a

Example: per-component override

Schedule all Fuzzball pods on nodes labeled workload: fuzzball, but place JetStream pods on nodes labeled storage: high-iops and tolerate a storage taint — the per-service values replace the global ones for JetStream only:

spec:
  nodeSelector:
    workload: fuzzball
  fuzzball:
    jetstream:
      nodeSelector:
        storage: high-iops
      tolerations:
        - key: storage
          operator: Exists
          effect: NoSchedule

Spreading JetStream Replicas Across Nodes

JetStream is deployed with replicated state: it runs as a three-node NATS cluster by default and keeps three copies of every stream. The operator does not apply anti-affinity of its own, so the Kubernetes scheduler is free to place all three JetStream pods on the same node, which gives no protection against that node failing. Production deployments that want node-failure tolerance should set affinity on the jetstream component.

The preferred form below is safe on any cluster size, including single-node development clusters, because the scheduler still places the pods when it cannot honor the preference:

spec:
  fuzzball:
    jetstream:
      replicas: 3
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: fuzzball-jetstream

On a cluster with at least three nodes eligible for JetStream, you can make the spread a hard rule with the required form. Required terms carry no weight and no podAffinityTerm wrapper, so topologyKey and labelSelector sit directly in the list item:

spec:
  fuzzball:
    jetstream:
      replicas: 3
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: fuzzball-jetstream

If fewer than three eligible nodes are available, the JetStream pods that cannot be placed stay Pending. To spread across availability zones instead of nodes, set topologyKey: topology.kubernetes.io/zone, but only when the JetStream storage class can attach volumes across zones.

  • A per-component affinity replaces the global one. Setting jetstream.affinity discards the top-level spec.affinity for JetStream. If the global value carries a nodeAffinity that pins Fuzzball onto labeled nodes, copy that block into jetstream.affinity next to the podAntiAffinity:

    spec:
      fuzzball:
        jetstream:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: workload
                        operator: In
                        values:
                          - fuzzball
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    topologyKey: kubernetes.io/hostname
                    labelSelector:
                      matchLabels:
                        app.kubernetes.io/name: fuzzball-jetstream
    
  • Existing pods. Adding the field restarts the JetStream pods, but if a pod’s jetstream-data volume is ReadWriteOnce and on a node-local storage class it pins the pod to the node that first scheduled it. The spread takes effect for new pods, or after deleting a pod’s PersistentVolumeClaim and restarting it so it is re-provisioned elsewhere. Do this one pod at a time and wait for the stream to resynchronize before the next. On network block storage the pods can relocate without this step.

  • Running pods are never evicted by this rule. Both forms end in IgnoredDuringExecution: the rule is checked only when a pod is scheduled, never afterwards.

Protecting JetStream Quorum During Node Drains

Draining a kubernetes node with more than one JetStream replica results in evicting all its JetStream replicas at the same time. This can take the JetStream cluster below the minimal number of replicas for quorum. Anti-affinity makes that layout less likely and a PodDisruptionBudget stops the drain from going through if it happens anyway. The two are complementary.

An administrator created disruption budget is a separate Kubernetes resource, attached to pods by a label selector. It persists through Fuzzball upgrades since the Fuzzball operator will not manage or remove it. Create the budget by hand in the Fuzzball namespace (fuzzball for FuzzballOrchestrate, fuzzball-federate for FuzzballFederate).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: fuzzball-jetstream
  namespace: fuzzball
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: fuzzball-jetstream

minAvailable: 2 assumes replicas: 3. Do not create a budget for a single-replica JetStream; a drain of its node would block indefinitely.

See the Kubernetes documentation for full syntax details on tolerations and affinity objects, and the disruptions page for PodDisruptionBudget.

Cluster DNS Domain

Fuzzball builds fully qualified in-cluster names, such as the JetStream cluster routes and the internal Keycloak address, on the cluster’s DNS domain. Most clusters use the Kubernetes default, cluster.local, but some distributions let you choose another (for example k3s and RKE2 with cluster-domain). Both FuzzballOrchestrate and FuzzballFederate support this field.

ParameterTypeRequiredDefaultDescription
clusterDomainstringNoDetected, else cluster.localDNS domain of the Kubernetes cluster, without a leading or trailing dot (e.g. fuzz.internal)

When clusterDomain is unset, the operator reads the domain from the svc.<domain> entry the kubelet writes into the search list of its own pod’s /etc/resolv.conf, and falls back to cluster.local when there is no such entry. The operator logs which domain it uses, and its source, on every reconcile, with a WARNING when it falls back. Set clusterDomain only if that detection is wrong, for example when the operator pod does not use cluster DNS, and set it at install time.

On a cluster whose domain is not cluster.local, a wrong domain leaves the JetStream pods stuck at 0/1 with Waiting for routing to be established in their logs, and the rest of the services are never deployed.

Keep the following in mind:

  • Once set, clusterDomain cannot be changed: the API server rejects the edit. To change it, remove the field so the operator falls back to detection, or recreate the resource.
  • Removing clusterDomain, or a change in the detected domain, re-renders the in-cluster names and restarts the affected services, JetStream included.
  • Custom domains require the operator and the Fuzzball images to come from the same release. An older JetStream image ignores the domain the operator passes it, and its routes fail TLS verification.
  • If the operator runs behind an egress proxy and you set clusterDomain because detection is wrong, also add .<domain> to the operator chart’s proxy.noProxy. The operator adds only the detected domain to its own NO_PROXY.

Example:

spec:
  clusterDomain: fuzz.internal

Egress Proxy and Trusted CA Configuration

For environments where outbound internet traffic must pass through an HTTP(S) proxy, the top-level proxy section injects the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables into the Fuzzball service pods, batch jobs, and the Keycloak identity service. The internal authorization database (SpiceDB) makes no outbound connections and is excluded. Both FuzzballOrchestrate and FuzzballFederate support the same fields.

ParameterTypeRequiredDefaultDescription
proxy.httpProxystringNo""Proxy URL for HTTP requests, e.g. http://proxy.example.com:3128
proxy.httpsProxystringNo""Proxy URL for HTTPS requests
proxy.noProxy[]stringNo[]Additional hosts, domain suffixes (written with a leading dot, e.g. .example.com), and CIDRs that bypass the proxy

Cluster-internal destinations are always excluded from proxying automatically: the operator appends the Kubernetes API endpoint, the .svc and .cluster.local suffixes, the cluster DNS domain, Fuzzball’s internal service names, and the cluster’s ingress domain to NO_PROXY. Add noProxy entries only for destinations specific to your environment.

If your pods send telemetry to a node-local collector (for example a Datadog agent DaemonSet), add your node and pod network CIDRs to noProxy — per-node IP addresses cannot be excluded automatically.

When the proxy intercepts TLS connections and re-signs them with a private certificate authority, add that CA through the trustedCACerts section. The certificates are distributed to every Fuzzball service pod through the cluster trust bundle mounted at /etc/ssl/certs, alongside the default public CAs.

ParameterTypeRequiredDefaultDescription
trustedCACerts.refs[]objectNo[]References to Secrets containing PEM-encoded CA certificates; each entry has name, key, and optional namespace (defaults to fuzzball-system)
trustedCACerts.inline[]stringNo[]PEM-encoded CA certificates provided directly in the resource

Example: deploy behind a TLS-intercepting corporate proxy

spec:
  proxy:
    httpProxy: http://proxy.example.com:3128
    httpsProxy: http://proxy.example.com:3128
    noProxy:
      - .internal.example.com
      - 10.0.0.0/8
  trustedCACerts:
    refs:
      - name: corporate-root-ca
        key: ca.crt

Create the referenced Secret in the fuzzball-system namespace before applying the resource:

$ kubectl create secret generic corporate-root-ca \
  --namespace fuzzball-system \
  --from-file=ca.crt=/path/to/corporate-root-ca.pem

Invalid proxy URLs and certificates that fail PEM parsing are rejected when the operator processes the resource; the running deployment continues unchanged until you correct them.

These fields configure only the workloads the operator deploys. The operator’s own deployment is configured separately through Helm chart values — see Configure an Egress Proxy for the Operator.

Monitoring Stack Configuration

The top-level monitoring section configures the bundled Prometheus, Alertmanager, and Grafana stack, preconfigured against Fuzzball’s own metrics and compute-node exporters, with the node health alert rules, the Alertmanager routes that place node holds, and the Fuzzball dashboards.

The stack is deployed by default, with its recommended configuration, when this section is unset. Set monitoring.enabled: false to deploy nothing – for example, on a site that runs its own monitoring and adopts the published configuration instead. The stack lives in its own namespace (fuzzball-monitoring by default), separate from the Fuzzball namespaces, so it can be removed without touching the control plane.

ParameterTypeRequiredDefaultDescription
monitoring.enabledbooleanNotrueDeploy the bundled monitoring stack. Set false to deploy nothing except what monitoring.alerting.externalReceiver asks for
monitoring.namespacestringNofuzzball-monitoringKubernetes namespace to deploy the stack into
monitoring.retentionDaysintegerNo15How long Prometheus keeps samples. Minimum 1
monitoring.storageSizestringNo20GiPrometheus persistent volume size. Set to the empty string ("") to run Prometheus and Alertmanager on emptyDir instead – no storage class required, history and silences are lost on restart
monitoring.storageClassNamestringNo""StorageClass for the Prometheus and Alertmanager volumes. Empty uses the cluster default. With no default and no storageSize: "", the stack is left out with a MonitoringNotDeployed warning event when monitoring.enabled is unset, and the resource is refused when it is true
monitoring.nodeMetricsModestringNoproxyproxy or direct. Must match nodeMetrics.serviceDiscovery.mode in central config
monitoring.prometheus.valuesOverridestringNo""YAML document of Prometheus Helm chart values merged over the generated ones
monitoring.grafana.enabledbooleanNotrueDeploy Grafana with the bundled Fuzzball dashboards. Set false to run Prometheus and Alertmanager alone
monitoring.grafana.adminSecretRefobjectNo-Secret holding the Grafana admin password under key and the user name under admin-user. Must be in the monitoring namespace (namespace, if set, must name it)
monitoring.grafana.valuesOverridestringNo""YAML document of Grafana Helm chart values merged over the generated ones
monitoring.alerting.webhookURLstringNo""Alertmanager webhook receiver URL. Alerts fire into Alertmanager and stop there when unset, still visible in its UI
monitoring.alerting.configOverridestringNo""Replaces the generated Alertmanager configuration wholesale, for a site whose routing the fields above cannot express
monitoring.alerting.valuesOverridestringNo""YAML document of Alertmanager Helm chart values (resources, replicas, storage) merged over the generated ones. Routing belongs in configOverride
monitoring.alerting.externalReceiver.enabledbooleanNofalsePublish the hold receiver for an Alertmanager the operator does not deploy, and issue it a client certificate. Honored whether or not the bundled stack is enabled
monitoring.alerting.externalReceiver.hostnamestringNoalertmanager-receiver.<domain>Hostname the external Alertmanager uses for the receiver. The receiver’s certificate is issued for it; point its DNS at the fuzzball-alertmanager-receiver Service
monitoring.alerting.externalReceiver.serviceTypestringNoLoadBalancerType of the fuzzball-alertmanager-receiver Service: LoadBalancer or NodePort
monitoring.alerting.externalReceiver.annotationsmapNo{}Annotations on the fuzzball-alertmanager-receiver Service, such as a cloud provider’s internal load balancer annotation

Example – tune the default stack:

spec:
  monitoring:
    retentionDays: 30
    storageSize: 100Gi
    alerting:
      webhookURL: https://alerts.example.com/hook
    prometheus:
      valuesOverride: |
        server:
          resources:
            limits:
              memory: 8Gi
    grafana:
      valuesOverride: |
        ingress:
          enabled: true
          ingressClassName: kong
          annotations:
            # Refuse plain HTTP: Grafana's login page carries the admin password.
            konghq.com/protocols: https
            konghq.com/https-redirect-status-code: "308"
          hosts:
            - grafana.example.com
          tls:
            - hosts:
                - grafana.example.com
              secretName: grafana-tls

Keep the host, the tls block and the two Kong annotations together. Without a host the route is a catch-all on the cluster’s Kong proxy, answering every hostname nothing else claims; without the annotations Kong also serves the login page over plain HTTP. grafana-tls is a certificate Secret in the monitoring namespace – issue it with cert-manager or supply your own. Other ingress controllers need their own equivalent of the Kong annotations.

Example – opt out:

spec:
  monitoring:
    enabled: false

Example – your own Alertmanager instead of the bundled stack:

spec:
  monitoring:
    enabled: false
    alerting:
      externalReceiver:
        enabled: true
        hostname: alertmanager-receiver.example.com

The hold receiver accepts only mutual TLS, so it is published through a fuzzball-alertmanager-receiver Service on port 9443 rather than through the cluster ingress, which terminates TLS. The operator issues the external Alertmanager’s client certificate into the fuzzball-external-alertmanager-client-tls Secret (tls.crt, tls.key), from a client CA of the Fuzzball namespace’s own (a namespaced Issuer, never a ClusterIssuer), and publishes the CA that signs the receiver’s certificate in the fuzzball-alertmanager-receiver-ca ConfigMap (ca.crt), both in the Fuzzball namespace. The client certificate is valid for a year and renewed 30 days before it expires; copy the renewed Secret to the Alertmanager. See Using Your Own Alertmanager.

Each valuesOverride is merged the way Helm layers a values file: maps merge key by key, and anything else, lists included, replaces the generated value. The keys are those of the component’s own chart, so server.resources rather than prometheus.server.resources. These keys are refused, because another field owns them: alertmanager in the Prometheus override (use monitoring.alerting.valuesOverride), and config and enabled in the Alertmanager override (use monitoring.alerting.configOverride and monitoring.enabled) – Alertmanager is what places holds, so it cannot be turned off on its own. A malformed override is refused as a configuration error on the resource before anything is deployed. An override can also undo what the operator generated – for example, replacing serverFiles."prometheus.yml".scrape_configs replaces Fuzzball’s own scrape jobs. To add jobs, use extraScrapeConfigs, which the chart takes as a string holding the job list, so it is a block scalar inside the override:

spec:
  monitoring:
    prometheus:
      valuesOverride: |
        extraScrapeConfigs: |
          - job_name: my-exporter
            static_configs:
              - targets: ["exporter.example.com:9200"]

An override may not set a value to null: Helm would read that as “remove this key”, but it does not reach Helm, so it is refused rather than silently ignored. Set the value you want instead.

The stack installs no CRDs, no DaemonSet, and no host ports, so it can run beside a site’s existing kube-prometheus-stack. Grafana is not exposed outside the cluster unless you enable its ingress through monitoring.grafana.valuesOverride, as in the example above. See Node Health for the dashboards, alert rules, and metric series shipped with the stack.

Workflow API Token Configuration

Fuzzball injects a workflow-scoped API token (FB_TOKEN) and API address into every workflow job and service container so running workflows can call the Fuzzball API — see In-Workflow API Access. The top-level disableWorkflowTokens field turns this off cluster-wide: no tokens are injected into new containers, the accompanying trust-store injection (the read-only node trust-store mount and SSL_CERT_DIR) is skipped, and the API stops accepting workflow tokens, including any already held by running containers.

ParameterTypeRequiredDefaultDescription
disableWorkflowTokensbooleanNofalseStop injecting workflow-scoped API tokens and the accompanying trust-store mount into workflow containers, and reject workflow tokens at the API

Example:

spec:
  disableWorkflowTokens: true

Workflow Image Build Configuration

A workflow can build its own container images in a top-level images: section – see Building Images in a Workflow. A build runs the definition’s post script as root inside a confined container on a compute node. The feature is on by default; the top-level enableWorkflowImageBuilds field turns it off, and a workflow carrying an images: section is then rejected at submission and validation. Turn it off where workflow authors are not trusted to choose code that runs as root in that container.

A FuzzballOrchestrate resource created or edited while Fuzzball v4.3 or a v4.4 pre-release was installed, when this field defaulted to false, has false stored and keeps it after an upgrade; set enableWorkflowImageBuilds: true to allow image builds on such a cluster. A resource not written since before v4.3 has nothing stored and allows them, so set false explicitly to keep them off.

ParameterTypeRequiredDefaultDescription
enableWorkflowImageBuildsbooleanNotrueAllow workflow definitions to build container images on compute nodes through an images: section

Example:

spec:
  enableWorkflowImageBuilds: false

Cluster Token Binding

A Federate and its member clusters call each other with cluster tokens. Each cluster binds every cluster token it sends to the cluster it is sending to, so a cluster that receives one cannot replay it against another. The top-level requireClusterTokenAudience field makes this cluster refuse a cluster token that is not bound to it. A token bound to a different cluster is always refused.

The field is off by default because a cluster running an earlier release sends unbound tokens, and turning it on would lock that cluster out. While it is off, this cluster accepts unbound tokens and counts them in the fuzzball_authn_unbound_cluster_tokens_total metric, labeled by the cluster that sent them. Turn it on once that count has stopped increasing for every sender. Refused cluster tokens are counted in fuzzball_authn_refused_cluster_tokens_total, labeled by sender and reason (bound_elsewhere or unbound), and logged. See Binding Cluster Tokens for the rollout. The field is the same on FuzzballOrchestrate and FuzzballFederate resources.

ParameterTypeRequiredDefaultDescription
requireClusterTokenAudiencebooleanNofalseRefuse cluster tokens that were not minted for this cluster

Example:

spec:
  requireClusterTokenAudience: true

Node Health Configuration

Compute nodes send a periodic heartbeat to the control plane, which is how a live node is told from a silent one — see Node Health Monitoring. The nodeHealth section controls how long a node may stay silent before it is recorded as such. Omit the section to accept the default.

ParameterTypeRequiredDefaultDescription
nodeHealth.heartbeatMissThresholdintegerNo3Consecutive health reports a compute node may miss before its status becomes unknown. Measured against each node’s own reporting interval, so it is a count rather than a duration. Minimum 1, maximum 100

Example:

spec:
  nodeHealth:
    heartbeatMissThreshold: 5

Fuzzball Services Configuration

The fuzzball section controls all Fuzzball services and their configurations.

Global Settings

ParameterTypeDescription
versionstringFuzzball version (defaults to operator version)
cluster.namestringCluster name. When unset, defaults to the deployment’s domain (e.g. example.com); Federate clusters are prefixed with federate. (e.g. federate.example.com).
cluster.kindstringCluster kind/type

Substrate Configuration

The substrate section controls how substrate nodes connect and operate. The scheduler admin guide covers sizing guidance for nodesBucketMaxBytes, the failure mode when the bucket fills, and the Prometheus metrics that track bucket usage — see Node Report Storage.

ParameterTypeDescription
substrate.nfs.destinationstringMount point on substrate nodes, including nodes started by a cloud provisioner
substrate.nfs.serverstringNFS server IP or hostname
substrate.nfs.pathstringNFS export path
substrate.secureRegistries[]stringHTTPS registries, each as host:port, whose certificate substrate nodes trust. See Secure Registries.
substrate.imageProxystringHTTP(S) proxy for image pulling
substrate.imageNoProxy[]stringHosts to exclude from proxy
substrate.mtls.*objectFuzzballOrchestrate only. Custom substrate mTLS CA: inline.secretName (a kubernetes.io/tls Secret in tls.certManager.clusterResourceNamespace, default cert-manager) or, deprecated, base64-encoded inline.cert and inline.key
substrate.nodesBucketMaxBytesintegerMaximum size, in bytes, of the substrate-nodes JetStream key-value bucket. Minimum 67108864 (64 MiB), maximum 2147483648 (2 GiB). Omit the field to use the orchestrate service default of 67108864 (64 MiB); do not set 0, which the minimum rejects at admission.

Example:

spec:
  fuzzball:
    cluster:
      name: my-fuzzball-cluster
      kind: on-premises
    substrate:
      nfs:
        destination: /fuzzball/shared
        server: 10.0.0.10
        path: /srv/fuzzball/shared
      secureRegistries:
        - registry.example.com:5000
      imageProxy: http://proxy.example.com:3128
      imageNoProxy:
        - localhost
        - 127.0.0.1
        - .example.com
      nodesBucketMaxBytes: 1073741824
Secure Registries

When it reconciles, the operator connects to each entry and adds the last certificate in the chain that the registry presents to the substrate configuration it generates, so substrate nodes trust the registry with that certificate. Each entry must include the port, for example registry.example.com:5000.

The list goes into two substrate configurations:

  • The one the operator generates for the provisioner, whether or not substrate.nfs is set. Nodes started by PBS and Slurm use it, and fuzzball cluster kubernetes generate-substrate-config copies it for compute nodes that install their configuration locally.
  • When substrate.nfs is set, the one on that share (substrate/substrate-config.yaml), which nodes load through FB_POLL_CONFIG.

The operator pod connects to each registry itself, so the registry must be reachable from the operator pod, not only from the substrate nodes. If an entry has no port, or the operator can’t connect to it, the whole deployment update fails, not just that entry: the resource goes Degraded, the operator logs while getting registry certificate for <entry>, and the operator retries until the entry is fixed or removed.

If the registry presents only its own certificate, nodes trust only that certificate unless they already trust the CA that issued it, so pulls fail once the registry renews it. To avoid this, configure the registry to present its CA certificate after its own certificate.

Substrate reads this list only when the service starts. After you add or change an entry:

  • On a compute node that installs its configuration locally, regenerate and install the configuration and restart Substrate as in Refresh the configuration after a change.
  • On a node that loads the share through FB_POLL_CONFIG, wait until substrate/substrate-config.yaml on the share lists the entry, then restart Substrate.

Jobs running on a node when Substrate restarts fail, so restart it while the node is idle:

# systemctl restart fuzzball-substrate.service

Service-Specific Configuration

Most Fuzzball services support the following common parameters:

ParameterTypeDescription
replicasintegerNumber of pod replicas
autoscaling.enabledbooleanEnable horizontal pod autoscaling
autoscaling.minReplicasintegerMinimum replicas when autoscaling
autoscaling.maxReplicasintegerMaximum replicas when autoscaling
autoscaling.targetCPUUtilizationintegerTarget CPU percentage (default: 80)
autoscaling.targetMemoryUtilizationintegerTarget memory percentage (default: 80)
resources.requestsmapResource requests (cpu, memory) as key-value pairs
resources.limitsmapResource limits (cpu, memory) as key-value pairs
serviceAccount.annotationsmapService account annotations (for IRSA)
nodeSelectormap[string]stringNode label selector for this component; overrides the top-level nodeSelector when set
tolerations[]objectOverride global tolerations for this service only
affinityobjectOverride global affinity for this service only

Example for orchestrator service:

spec:
  fuzzball:
    orchestrator:
      replicas: 2
      autoscaling:
        enabled: true
        minReplicas: 1
        maxReplicas: 5
        targetCPUUtilization: 70
        targetMemoryUtilization: 75
      resources:
        requests:
          cpu: 500m
          memory: 512Mi
        limits:
          cpu: 2000m
          memory: 2Gi
      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator

Available Configuration Sections

The following configuration sections are available under spec.fuzzball:

Top-Level Configuration:

  • version - Fuzzball version (defaults to operator version)
  • config - Global configuration (shared storage, etc.)
  • cluster - Cluster name and metadata
  • substrate - Substrate node configuration (NFS, registries, proxies, mTLS)

Active Services:

  • agent - Agent service (workflow execution)
  • audit - Audit logging service
  • auth - Authentication service (SpiceDB)
  • billing - Billing and marketplace integration
  • clusterAdmin - Admin UI and cluster setup
  • jetstream - NATS Jetstream message broker
  • objectCache - Object cache service (shared blob storage for workflows)
  • openapi - OpenAPI documentation service
  • orchestrator - Orchestrator service
  • storage - Storage service
  • substrateBridge - Substrate bridge service (DNS, logging, Kubernetes integration)
  • ui - web UI service
  • workflow - Workflow service
  • workflowCatalog - Workflow catalog configuration

The following services have been deprecated as of version v3.0 and will be removed in a future version (use recommended replacements):

  • account - Deprecated (use agent)
  • dns - Deprecated (use substrateBridge)
  • kube - Deprecated (use substrateBridge)
  • log - Deprecated (use substrateBridge)
  • organization - Deprecated (use agent)
  • provision - Deprecated (use orchestrator)
  • schedule - Deprecated (use orchestrator)
  • secret - Deprecated (use agent)
  • user - Deprecated (use agent)
  • workflowEngine - Deprecated (use jetstream)

Jetstream Configuration

NATS Jetstream has specific configuration requirements:

ParameterTypeDefaultDescription
replicasinteger3Number of Jetstream replicas, at least 1 (recommended: 3). 1 runs a single standalone server with no clustering or redundancy, for new deployments only: a standalone server cannot recover streams written by a clustered deployment, so scaling an existing deployment down to 1 leaves Jetstream unready.
storage.classstring-StorageClass for Jetstream PVCs
storage.sizestring10GiStorage size per replica
externalService.typestringClusterIPExternal service type (ClusterIP, NodePort, LoadBalancer). Use LoadBalancer when Substrate hosts can’t reach the cluster nodes’ IP addresses
externalService.nodePortinteger-Static node port when externalService.type is NodePort
externalService.annotationsmap-Annotations for the external Service, for example to choose a load balancer address pool

Example:

spec:
  fuzzball:
    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 20Gi
      externalService:
        type: NodePort

The operator does not spread JetStream replicas across nodes or protect them from node drains by default. See Spreading JetStream Replicas Across Nodes and Protecting JetStream Quorum During Node Drains for the affinity and PodDisruptionBudget configuration a production deployment should add.

Storage Service Configuration

This storage is necessary for internal services. Admins can configure storage for computational jobs later.
ParameterTypeDescription
storage.storage.classstringStorageClass for storage service

Example:

spec:
  fuzzball:
    storage:
      storage:
        class: longhorn

Audit Service Configuration

ParameterTypeDescription
audit.storage.classstringStorageClass for audit logs

Example:

spec:
  fuzzball:
    audit:
      storage:
        class: longhorn

Substrate Bridge Configuration

ParameterTypeDescription
substrateBridge.log.storage.classstringStorageClass for substrate logs
substrateBridge.dns.externalService.typestringDNS service type (ClusterIP, NodePort, LoadBalancer)
substrateBridge.dns.externalService.annotationsmapAnnotations for the DNS Service, for example to choose a load balancer address pool
NodePort is recommended for local deployments. When Substrate hosts can’t reach the cluster nodes' IP addresses, set both this and jetstream.externalService.type to LoadBalancer. See Compute nodes that can’t reach the Server nodes.

Example:

spec:
  fuzzball:
    substrateBridge:
      log:
        storage:
          class: longhorn
      dns:
        externalService:
          type: NodePort

Object Cache Configuration

ParameterTypeDefaultDescription
objectCache.storage.classstring-StorageClass for the object cache PVC
objectCache.storage.sizestring50GiStorage size for the object cache PVC
objectCache.storage.pathstring/storage/objectcache/Mount path for object cache blob storage
objectCache.defaultTTLstring7dDefault time-to-live for cached objects (e.g., 7d, 24h)
objectCache.diskPressureThresholdinteger90Disk usage percentage (0-100) that triggers accelerated eviction
objectCache.diskCriticalThresholdinteger95Disk usage percentage (0-100) that rejects new uploads
objectCache.storageClassName, objectCache.storageSize, and objectCache.storagePath are deprecated in favor of the nested objectCache.storage.class, objectCache.storage.size, and objectCache.storage.path fields above, and will be removed in a future release.

Example:

spec:
  fuzzball:
    objectCache:
      storage:
        class: longhorn
        size: 100Gi
        path: /storage/objectcache/
      defaultTTL: 24h
      diskPressureThreshold: 85
      diskCriticalThreshold: 95

Orchestrator Provisioner Configuration

The orchestrator provisioner enables multi-cloud and HPC integration:

ParameterTypeDescription
enabledbooleanEnable provisioner
substrateComputeDirectorystringWorking directory for substrate operations
awsobjectAWS provisioner configuration
azureobjectAzure provisioner configuration
gcpobjectGCP provisioner configuration
coreweaveobjectCoreWeave provisioner configuration
slurmobjectSlurm provisioner configuration
pbsobjectPBS provisioner configuration
Within a single FuzzballOrchestrate resource, at most one of aws, azure, or gcp may be enabled. The default storage provisioner Fuzzball creates at install time is derived from the enabled cloud, so the choice must be unambiguous. Enabling more than one causes the operator to fail its reconcile with a configuration error. Federating multiple clouds is still supported: deploy a separate FuzzballOrchestrate per cloud and join them under a FuzzballFederate.

GCP Provisioner

The GCP provisioner enables dynamic provisioning of compute instances on Google Cloud Platform.

ParameterTypeDescription
gcp.enabledbooleanEnable GCP provisioner
gcp.projectstringGCP project ID where substrate instances will be created
gcp.regionstringGCP region for substrate instances
gcp.zonestringGCP zone for substrate instances
gcp.networkstringVPC network name
gcp.subnetstringVPC subnet name within the network
gcp.serviceAccountEmailstringService account email for substrate instances
gcp.substrateImageProjectstringGCP project hosting substrate VM images (optional; defaults to CIQ public image project when empty)
gcp.sshEnabledbooleanEnable SSH access to substrate instances
gcp.sshKeyPairNamestringSSH key pair name
gcp.sshPrivateKeyPemstringSSH private key in PEM format
gcp.depotUserstringCIQ Depot username for substrate nodes
gcp.depotAccessTokenstringCIQ Depot access token for substrate nodes
gcp.cloudInitScripts[]stringCustom cloud-init scripts to run on instances
gcp.sharedFsmapShared filesystem configuration (uri, type, opts), required when the GCP provisioner is enabled. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. All three fields must be set and may use only letters, digits and =,_./:@-

Example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        gcp:
          enabled: true
          project: my-gcp-project
          region: us-central1
          zone: us-central1-a
          network: default
          subnet: default
          serviceAccountEmail: fuzzball-substrate@my-project.iam.gserviceaccount.com
          substrateImageProject: my-mirrored-images-project
          sharedFs:
            uri: "10.0.0.2:/imagecache"
            type: nfs
            opts: "nfsvers=3,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"

When to set substrateImageProject:

By default, the GCP provisioner pulls substrate VM images from the CIQ public image project. Set substrateImageProject when substrate images have been mirrored to a customer-owned GCP project. This is required in the following scenarios:

  • Air-gapped environments that cannot access the CIQ public image project
  • Deployments where organizational policies prohibit external image references
  • Environments requiring custom image approval workflows

When substrateImageProject is empty or omitted, the provisioner uses the default CIQ public image project.

AWS Provisioner

ParameterTypeDescription
aws.enabledbooleanEnable AWS provisioner
aws.regionstringAWS region
aws.subnetIDs[]stringVPC subnet IDs
aws.securityGroupIDs[]stringSecurity group IDs
aws.instanceProfileARNstringIAM instance profile ARN
aws.usePublicIPbooleanAssign public IPs
aws.sshEnabledbooleanEnable SSH access
aws.sshKeyPairNamestringEC2 key pair name
aws.sshPrivateKeyPemstringSSH private key
aws.depotUserstringCIQ Depot username for substrate nodes
aws.depotAccessTokenstringCIQ Depot access token for substrate nodes
aws.cloudInitScripts[]stringCustom cloud-init scripts to run on instances
aws.sharedFsmapShared filesystem configuration. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. Its uri may use only letters, digits and =,_./:@-. Without uri, provisioned nodes cache images on local disk

Example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        aws:
          enabled: true
          region: us-east-1
          subnetIDs:
            - subnet-0123456789abcdef0
          securityGroupIDs:
            - sg-0123456789abcdef0
          instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate
          usePublicIP: false

Azure Provisioner

The Azure provisioner enables dynamic provisioning of compute instances on Microsoft Azure.

ParameterTypeDescription
azure.enabledbooleanEnable Azure provisioner
azure.subscriptionIDstringAzure subscription ID
azure.tenantIDstringAzure tenant ID
azure.locationstringAzure region (location) for provisioned resources
azure.resourceGroupNamestringResource group for Azure resources
azure.subnetIDstringSubnet ID for substrate VMs
azure.networkSecurityGroupIDstringNetwork security group ID for substrate VMs
azure.usePublicIPbooleanAssign public IPs
azure.managedIdentityClientIDstringUser-assigned managed identity client ID for provisioned VMs
azure.storageAccountNamestringAzure Files storage account name backing the default v4 storage provisioner (workflowio share)
azure.sshEnabledbooleanEnable SSH access
azure.sshPublicKeystringSSH public key
azure.sshPrivateKeyPemstringSSH private key in PEM format
azure.substrateImageBaseIDstringAzure Managed Image resource ID for the base substrate image (overrides the marketplace image when set)
azure.substrateImageNvidiaIDstringAzure Managed Image resource ID for the NVIDIA/GPU substrate image (overrides the marketplace image when set)
azure.depotUserstringCIQ Depot username for substrate nodes
azure.depotAccessTokenstringCIQ Depot access token for substrate nodes
azure.cloudInitScripts[]stringCustom cloud-init scripts to run on instances
azure.sharedFsmapShared filesystem configuration. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. uri, type and opts may use only letters, digits and =,_./:@-, and if uri is set, type and opts must be set too. Without uri, provisioned nodes cache images on local disk

Example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        azure:
          enabled: true
          subscriptionID: 00000000-0000-0000-0000-000000000000
          tenantID: 00000000-0000-0000-0000-000000000000
          location: eastus
          resourceGroupName: fuzzball-rg
          subnetID: /subscriptions/.../resourceGroups/fuzzball-rg/providers/Microsoft.Network/virtualNetworks/fuzzball-vnet/subnets/substrate
          networkSecurityGroupID: /subscriptions/.../resourceGroups/fuzzball-rg/providers/Microsoft.Network/networkSecurityGroups/fuzzball-nsg
          managedIdentityClientID: 00000000-0000-0000-0000-000000000000
          storageAccountName: fuzzballstorage
          usePublicIP: false

CoreWeave Provisioner

The CoreWeave provisioner enables dynamic provisioning of compute nodes on CoreWeave infrastructure. When enabled, Fuzzball will automatically create and manage CoreWeave NodePool resources and deploy substrate pods as DaemonSets on provisioned nodes.

CoreWeave deployments use two types of shared storage PVCs:

  1. Workflow Data PVC (fuzzball-shared-storage): Mounted at /mnt/shared-storage on substrate nodes and exposed to workflows via the HostPath CSI driver
  2. Image Cache PVC (fuzzball-sharedfs): Mounted at /mnt/fuzzball-sharedfs on substrate nodes for internal container image caching (not exposed to workflows)
CoreWeave’s shared-vast storage requires Native Protocol Limit view policy. Ensure this is configured before deployment.
Workflow Data Storage Parameters
ParameterTypeRequiredDefaultDescription
coreweave.enabledbooleanYes-Enable CoreWeave provisioner
coreweave.storage.classstringNo-StorageClass for CoreWeave shared storage PVCs (typically shared-vast)
coreweave.storage.sizestringNo-Size of shared storage per substrate node
coreweave.storage.accessModestringNoReadWriteOnceAccess mode for shared storage PVC (use ReadWriteMany for multi-node access)
coreweave.storage.annotationsmapNo{}Additional annotations for storage resources

Example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        coreweave:
          enabled: true
          storage:
            class: shared-vast
            size: 100Gi
            accessMode: ReadWriteMany
Workflow Data Storage Sizing
Workload TypeRecommended SizeRationale
Light workflows50GiMinimal data processing
Standard workflows100GiTypical data processing needs
Heavy workflows250Gi+Large datasets, intermediate files
Data-intensive500Gi+Big data processing, ML training
Shared PVC Configuration (fuzzball.config.sharedPVC)

The shared PVC (fuzzball-sharedfs) is mounted at /mnt/fuzzball-sharedfs on substrate nodes for internal container image caching across compute nodes. This improves workflow startup times by avoiding repeated image downloads. This storage is not exposed to workflows.

ParameterTypeRequiredDefaultDescription
fuzzball.config.sharedPVC.accessModestringYes-Volume access mode (must be ReadWriteMany)
fuzzball.config.sharedPVC.classstringYes-Storage class name (must be shared-vast)
fuzzball.config.sharedPVC.sizestringYes-Total shared cache size

Example:

spec:
  fuzzball:
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: shared-vast
        size: 10Gi
Shared PVC Sizing
Environment TypeRecommended SizeRationale
Testing/Development10GiFew container images
Small Production25GiLimited image variety
Large Production50Gi+Many different images
Additional Trusted Issuers (fuzzball.config.additionalTrustedIssuers)

By default a Fuzzball cluster only trusts JWTs issued by its own Keycloak. Setting fuzzball.config.additionalTrustedIssuers adds extra JWT issuer URLs that the cluster’s services will also accept. The primary use case is federation, where you want one cluster to honor tokens minted by another cluster’s Keycloak (see Trusting Tokens from Another Keycloak).

ParameterTypeRequiredDefaultDescription
fuzzball.config.additionalTrustedIssuersstring arrayNo[]Extra JWT issuer URLs to trust alongside this cluster’s own Keycloak. Each entry must be an absolute http(s) URL.

Example:

spec:
  fuzzball:
    config:
      additionalTrustedIssuers:
        - https://auth.federate.example.com/
The trusted-issuer list is keyed on the issuer’s scheme://host; the path and query string are ignored. An additional issuer that shares a scheme and host with the cluster’s own Keycloak is discarded so it cannot override the operator-generated entry.

Slurm Provisioner

See Slurm Integration Documentation for detailed configuration.

ParameterTypeRequiredDefaultDescription
slurm.enabledbooleanYes-Enable Slurm provisioner
slurm.sshHoststringYes-SSH host for remote Slurm instance
slurm.sshPortintegerNo22SSH host port for remote Slurm instance
slurm.usernamestringYes-SSH login username for remote Slurm instance
slurm.passwordstringNo-SSH login password for remote Slurm instance
slurm.sshHostPublicKeystringNo-SSH host public key for remote Slurm instance
slurm.sshPrivateKeyPemstringNo-SSH private key PEM for remote Slurm instance
slurm.sshPrivateKeyPassPhrasestringNo-Passphrase for encrypted SSH private key
slurm.binaryPathstringNo-Custom path to Slurm binaries (if not in $PATH)
slurm.connectionTimeoutintegerNo30SSH connection timeout in seconds
slurm.sudoPathstringNo-Path to sudo binary on compute nodes
slurm.optionsmapNo{}Additional Slurm sbatch options
slurm.skipHostKeyVerificationbooleanNofalseSkip SSH host key verification (not recommended)

Basic example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        slurm:
          enabled: true
          sshHost: slurm-head.example.com
          sshPort: 22
          username: fuzzball-service
          sshPrivateKeyPem: |
            <full private key in PEM format>
          sshHostPublicKey: "slurm-head.example.com ecdsa-sha2-nistp256 AAAAE2..."

PBS Provisioner

See PBS Integration Documentation for detailed configuration.

ParameterTypeRequiredDefaultDescription
pbs.enabledbooleanYes-Enable PBS provisioner
pbs.sshHoststringYes-SSH host for remote PBS instance
pbs.sshPortintegerNo22SSH host port for remote PBS instance
pbs.usernamestringYes-SSH login username for remote PBS instance
pbs.passwordstringNo-SSH login password for remote PBS instance
pbs.sshHostPublicKeystringNo-SSH host public key for remote PBS instance
pbs.sshPrivateKeyPemstringNo-SSH private key PEM for remote PBS instance
pbs.sshPrivateKeyPassPhrasestringNo-Passphrase for encrypted SSH private key
pbs.binaryPathstringNo-Custom path to PBS binaries (if not in $PATH)
pbs.validateSubstratebooleanNofalseValidate substrate before use
pbs.defaultQueuestringNo-Default PBS queue name
pbs.pbsServerstringNo-PBS server hostname
pbs.connectionTimeoutintegerNo30SSH connection timeout in seconds
pbs.optionsmapNo{}Additional PBS qsub options
pbs.sudoPathstringNo-Path to sudo binary on compute nodes
pbs.skipHostKeyVerificationbooleanNofalseSkip SSH host key verification (not recommended)

Basic example:

spec:
  fuzzball:
    orchestrator:
      provisioner:
        enabled: true
        pbs:
          enabled: true
          sshHost: pbs-head.example.com
          sshPort: 22
          username: fuzzball-service
          password: secure-password

LOTA Object Storage

Support for CoreWeave LOTA object storage is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using LOTA with Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

Overview

CoreWeave’s LOTA provides S3-compatible object storage for workflow data ingress and egress. Configure LOTA credentials to enable workflows to read from and write to LOTA buckets.

Configuration Parameters

ParameterTypeRequiredDescription
typestringYesMust be s3 for LOTA
secret.access-key-idstringYesLOTA access key ID
secret.secret-access-keystringYesLOTA secret access key
secret.endpointstringYesLOTA endpoint URL (https://cwlota.com)
secret.regionstringYesLOTA bucket region (e.g., ord1, lga1)

Create LOTA Credentials

Step 1: Create Credentials File

Create a YAML file with your LOTA credentials:

# lota-credentials.yaml
type: s3
secret:
  access-key-id: AKIAIOSFODNN7EXAMPLE
  secret-access-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  endpoint: https://cwlota.com
  region: us-east-02a

Replace placeholder values with your actual LOTA credentials from CoreWeave.

Step 2: Create Secret in Fuzzball

$ fuzzball secret create -s user --from-file \
  lota-credentials lota-credentials.yaml

The -s user flag makes the secret available to all users in your organization.

Use LOTA in Workflows

Reference the LOTA credentials in workflow volume definitions:

volumes:
  shared-storage:
    reference: volume://user/fuzzball-shared-vast/shared-storage
    ingress:
      - source:
          uri: "s3://my-bucket/input/data.txt"
          secret: secret://user/lota-credentials
        destination:
          uri: "file://data.txt"
    egress:
      - source:
          uri: "file://results.tar.gz"
        destination:
          uri: "s3://my-bucket/output/results.tar.gz"
          secret: secret://user/lota-credentials

Replace my-bucket with your LOTA bucket name.

Common Configuration Patterns

Minimal Local Deployment

For testing and development on a local Kubernetes cluster. This deploys the Fuzzball control plane. To run workflows, you’ll need to add shared storage (fuzzball.substrate.nfs or fuzzball.config.sharedPVC) and configure compute nodes.

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token
    exclusive: false

  ingress:
    create:
      domain: localhost.nip.io
      proxy:
        type: NodePort
        http:
          nodePort: 30080
        tls:
          nodePort: 30443

  database:
    create:
      storage:
        class: local-path

  keycloak:
    create:
      ownerEmail: admin@localhost
      createDatabase: true
      # The 3Gi production default is more than a laptop-sized cluster can spare.
      resources:
        requests:
          memory: "1Gi"
        limits:
          memory: "1Gi"

  tls:
    certManager:
      create: {}
    trustManager:
      create: {}

  fuzzball:
    jetstream:
      replicas: 1
      externalService:
        type: NodePort
      storage:
        class: local-path

    substrateBridge: {}

    log:
      storage:
        class: "local-path"

    dns:
      externalService:
        type: NodePort

Production On-Premises Deployment

For production deployments with high availability:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token
    exclusive: true

  ingress:
    create:
      domain: fuzzball.company.com
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/loadBalancerIPs: 10.0.100.50

  database:
    create:
      storage:
        class: longhorn
        size: 500Gi

  keycloak:
    create:
      ownerEmail: admin@company.com
      replicas: 2
      createDatabase: true

  tls:
    certManager:
      create: {}
    trustManager:
      create: {}
    ingressIssuer:
      create:
        letsEncrypt:
          email: admin@company.com
          issuer: letsencrypt-prod
          solvers:
            - dns01:
                route53:
                  region: us-east-1
                  hostedZoneID: Z1234567890ABC

  fuzzball:
    cluster:
      name: production-cluster

    substrate:
      nfs:
        server: nfs.company.com
        path: /fuzzball/shared
        destination: /fuzzball/shared

    orchestrator:
      replicas: 3
      autoscaling:
        enabled: true
        minReplicas: 2
        maxReplicas: 10

    agent:
      replicas: 5
      autoscaling:
        enabled: true
        minReplicas: 3
        maxReplicas: 20

    jetstream:
      replicas: 3
      storage:
        class: longhorn
        size: 50Gi

Cloud Deployment with External Services

Using external database and authentication:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  image:
    username: depot-user
    password: depot-token

  ingress:
    external:
      domain: fuzzball.cloud.com
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod

  database:
    external:
      host: postgres.abc123.us-east-1.rds.amazonaws.com
      rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:fuzzball-db
      credentials:
        user: ""      # Leave empty when using rdsSecretId
        password: ""  # Leave empty when using rdsSecretId
      sslMode: verify-full
      certificate:
        caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

  keycloak:
    external:
      url: https://keycloak.cloud.com
      realmId: 550e8400-e29b-41d4-a716-446655440000
      realmName: FuzzballProd
      username: admin
      password: keycloak-password
      ownerEmail: admin@cloud.com

  fuzzball:
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: efs-sc
        size: 100Gi

    orchestrator:
      provisioner:
        enabled: true
        aws:
          enabled: true
          region: us-east-1
          subnetIDs:
            - subnet-0123456789abcdef0
          securityGroupIDs:
            - sg-0123456789abcdef0
          instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate

      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator

Managing FuzzballOrchestrate Resources

Viewing Configuration

Check deployment status:

$ kubectl get fuzzballorchestrate
NAME                   STATUS    AGE
fuzzball-orchestrate   Running   10m

View the current configuration:

$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o yaml

View status and events:

$ kubectl describe fuzzballorchestrate fuzzball-orchestrate

Updating Configuration

Edit the configuration in-place (changes are applied immediately):

$ kubectl edit fuzzballorchestrate fuzzball-orchestrate

Apply configuration from a file:

$ kubectl apply -f fuzzball.yaml

Monitoring Deployment

The operator reports deploy and destroy progress directly on the custom resource. Use kubectl describe to see real-time status during any operation:

$ kubectl describe fuzzballorchestrate fuzzball-orchestrate
The operator uses Pulumi internally to manage the underlying Kubernetes and cloud resources. As a result, events and .status.failedResources entries identify resources by Pulumi URN — a stable identifier of the form urn:pulumi:<stack>::<provider>::<type>::<name>. You can treat a URN as an opaque label unique to the specific resource Pulumi is tracking; you only need to recognize that the trailing <name> segment is human-readable (e.g., fuzzball-coldboot, fuzzball-load-default-catalog).

Status Conditions

Two lifecycle conditions, Progressing and Ready, carry deploy and destroy state:

ConditionStatusReasonMeaning
ProgressingTrueDeployingDeploy in progress
ReadyFalseDeployingDeploy in progress
ProgressingFalseDeployCompleteDeploy finished successfully
ReadyTrueSucceededDeploy finished successfully
ReadyFalseRolloutIncompleteDeploy finished, but a Deployment is still rolling out
ReadyFalseRolloutStalledDeploy finished, but a Deployment will not finish rolling out on its own
ProgressingFalseDeployFailedDeploy failed (see Message for error)
ReadyFalseDeployFailedDeploy failed
ProgressingTrueDestroyingDestroy in progress
ReadyFalseTerminatingDestroy in progress
ProgressingFalseDestroyCompleteDestroy finished — resource pending garbage collection
ProgressingFalseDestroyFailedDestroy failed (see Message for error)
ReadyFalseDestroyFailedDestroy failed

Each condition also carries a Message field summarizing the state in human-readable form: "Deploy in progress" or "Destroy in progress" while the operation is running, "Deploy completed" or "Destroy completed" on success, or the underlying error string when it failed.

On a FuzzballOrchestrate, Ready turns True only once every Deployment in the fuzzball namespace labeled app.kubernetes.io/managed-by: pulumi has rolled out: all replicas updated and available, and no old pods left. Until then it stays False with reason RolloutIncomplete, and its Message names each Deployment still rolling out. Components installed from Helm charts, such as Keycloak, and StatefulSets are not checked. A pod that becomes unavailable, or is not replaced, after its Deployment finished rolling out does not hold Ready, unless that Deployment has returned to a pod template it ran before (a downgrade, or a setting changed and then changed back); then it does, until the Deployment’s next rollout to a pod template it has not run before. The operator checks the Deployments again every 15 seconds, without redeploying, and sets Ready and Running once they have rolled out. If a Deployment exceeds its progress deadline or is paused, the reason becomes RolloutStalled and Degraded is set, so a rollout blocked by, for example, a volume that is still attached to the old pod does not report as ready. The operator then retries the full deployment every five minutes.

After a successful destroy, Ready is not updated — the operator removes the finalizer and Kubernetes garbage-collects the resource. Only Progressing transitions to False / DestroyComplete before the resource disappears.

A separate condition, Misconfigured, reports a problem with the spec rather than with a deploy in progress. It is set independently of the lifecycle pair and stays on the resource until the spec is corrected, so it survives the deploys that run in spite of it:

ConditionStatusReasonMeaning
MisconfiguredTrueMissingOwnerEmailownerEmail is empty (see Owner email is required)

The condition is removed once the spec is fixed. It is recorded on a best-effort basis, so treat its presence as authoritative and its absence as “nothing reported” rather than as a clean bill of health.

Alongside these, the operator records where its state machine currently is with a one-condition-per-state set. Exactly one of these is True at a time:

ConditionMeaning
CreatingFirst deploy of the resource is running
UpdatingA deploy is running against a resource that already exists
RunningThe resource is deployed and reconciled
DegradedThe last operation failed
TerminatingThe resource is being destroyed

Degraded carries a fixed message, An error occurred - See Operator logs for more information., so the operator logs are where the underlying error is found. This is the condition to look for first when kubectl describe shows a deploy that is not progressing.

Progress Field

While a deploy or destroy is running, .status.progress is populated with real-time counters and the currently active resources:

Status:
  Progress:
    Phase:            Deploying
    Completed:        153
    In Flight:        2
    Failed:           0
    Elapsed Seconds:  310
    Start Time:       2026-04-23T01:00:22Z
    Current Resources:
      Kind:   Command    Op: create    Elapsed Seconds: 0
      Urn:    urn:pulumi:…::fuzzball-coldboot
      Kind:   Job        Op: create    Elapsed Seconds: 0
      Urn:    urn:pulumi:…::fuzzball-load-default-catalog

Phase is either Deploying or Terminating. completed, inFlight, and failed count individual infrastructure resource operations. Once the operation finishes, .status.progress is cleared.

Kubernetes Events

The operator emits events on the resource throughout the operation lifecycle. View them at the bottom of kubectl describe output or with:

$ kubectl get events --field-selector involvedObject.name=fuzzball-orchestrate --sort-by='.lastTimestamp'

Deploy events:

EventTypeDescription
DeployStartedNormalDeploy has begun
DeployCompletedNormalDeploy succeeded — includes elapsed time and resource count
ResourceOpFailedWarningA single tracked resource’s create/update/replace/delete failed
DeployFailedWarningDeploy failed — see .status.failedResources and operator logs
RolloutStalledWarningDeploy succeeded, but a Deployment exceeded its progress deadline or is paused

Destroy events:

EventTypeDescription
DestroyStartedNormalDestroy has begun
DestroyingDependencyNormalA named dependency stack (e.g., Fuzzball, Database) is being torn down
DestroyedDependencyNormalA dependency stack completed teardown
DestroyCompletedNormalAll resources destroyed — includes elapsed time and count
ResourceOpFailedWarningA single tracked resource’s create/update/replace/delete failed
DestroyFailedWarningDestroy failed

ResourceOpFailed is emitted per affected resource (not once per operation), so it can fire multiple times during a single Deploy or Destroy. Its message has the form <Kind> <op> failed: <URN> (for example, Job create failed: urn:pulumi:fuzzball::…::fuzzball-load-default-catalog). It is the most precise signal for pinpointing which resource broke before the rolled-up DeployFailed / DestroyFailed event fires.

The same event patterns apply to both FuzzballOrchestrate and FuzzballFederate resources.

Example — successful Orchestrate deploy:

Events:
  Normal  DeployCompleted  0s   Deploy completed in 2m11s; 93 resources applied

Example — 164-resource teardown:

Events:
  Normal  DestroyStarted        1m41s  Starting Destroy for fuzzball-local-dev
  Normal  DestroyingDependency  1m41s  Destroying Identity
  Normal  DestroyingDependency  1m37s  Destroying Fuzzball
  Normal  DestroyingDependency  53s    Destroying Namespace
  Normal  DestroyingDependency  48s    Destroying Database
  Normal  DestroyingDependency  10s    Destroying TLS
  Normal  DestroyedDependency   0s     Destroyed Database
  Normal  DestroyedDependency   0s     Destroyed Fuzzball
  Normal  DestroyedDependency   0s     Destroyed Identity
  Normal  DestroyedDependency   0s     Destroyed Namespace
  Normal  DestroyedDependency   0s     Destroyed TLS
  Normal  DestroyCompleted      0s     Destroy completed in 1m44s; 164 resources destroyed

Watch operator logs during deployment:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f

Important Notes

  1. Resource Scope: FuzzballOrchestrate is a cluster-scoped resource (not namespace-scoped)
  2. Short Names: Can use fb or fuzz as shortcuts in kubectl commands
  3. Mutual Exclusivity:
    • Database: Use either create or external, not both
    • Ingress: Use either create or external, not both
    • Keycloak: Use either create or external, not both
  4. Autoscaling: When enabled, replicas is ignored in favor of minReplicas/maxReplicas
  5. Storage Classes: Ensure specified storage classes exist in your cluster
  6. UUIDs: realmId must be a valid UUID v4 format
  7. NFS Requirements: NFS server must be accessible from all nodes
  8. Scheduling Overrides: A per-component nodeSelector, tolerations, or affinity replaces the global value for that component; the operator sets no default anti-affinity or PodDisruptionBudget for any component, including JetStream

Troubleshooting

Deployment Fails

When a deploy fails, the Ready and Progressing conditions are both set to False with reason DeployFailed. The Message field on the condition contains the Pulumi diagnostic error:

Status:
  Conditions:
    Type: Progressing    Status: False    Reason: DeployFailed
    Message: an unhandled error occurred: waiting for RPCs: marshaling properties:
             awaiting input property "clientId": while getting fuzzball realm: ...
    Type: Ready          Status: False    Reason: DeployFailed
    Message: (same as above)
Events:
  Warning  DeployFailed  0s   Deploy failed after 5m15s. See .status.failedResources and operator logs.

The Message field is capped at 256 characters; a trailing ... in the output indicates the text was truncated. Consult the operator logs (shown further down) for the full error.

Check .status.failedResources for a structured list of every resource that failed, including the Pulumi URN, the operation type (create, update, replace, or delete), the error message, and how many times it was retried:

$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o jsonpath='{.status.failedResources}' | jq .

Each entry contains:

  • urn — Pulumi URN identifying the specific resource that failed
  • op — operation that was attempted
  • error — the diagnostic error message
  • attemptCount — number of retries before giving up

.status.failedResources is retained on the resource until the next stack operation begins — the operator overwrites the slice with a fresh (empty) one as soon as the next deploy or destroy starts, so capture this output before retrying if you want a post-mortem record.

Check operator logs for additional context:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system --tail=100

Check FuzzballOrchestrate status and events:

$ kubectl describe fuzzballorchestrate fuzzball-orchestrate

Pods Not Starting

Check pod events:

$ kubectl get pods -n fuzzball

$ kubectl describe pod <pod-name> -n fuzzball

Check PVC status:

$ kubectl get pvc -A

Pods Pending with Scheduling Errors

A pod that stays in Pending because of the nodeSelector, tolerations, or affinity settings carries a FailedScheduling event explaining which rule excluded each node:

$ kubectl get events -n fuzzball --field-selector reason=FailedScheduling

$ kubectl describe pod <pod-name> -n fuzzball
Event message containsCauseFix
didn't match Pod's node affinity/selectorThe nodeSelector or nodeAffinity names a label no node carriesCompare the rule against kubectl get nodes --show-labels and fix the label or the rule
had untolerated taintA node taint has no matching tolerationAdd the toleration globally or to the affected component; see kubectl describe node <node> | grep Taints
didn't match pod anti-affinity rulesA requiredDuringSchedulingIgnoredDuringExecution anti-affinity needs more eligible nodes than existUse a preferredDuringSchedulingIgnoredDuringExecution rule or add eligible nodes; do not scale an existing clustered JetStream deployment down to one replica
volume node affinity conflictThe pod’s existing volume is on a node or in a zone the new rules excludeRelax the scheduling rule to allow the existing volume, or back up and migrate the data to compatible storage using a supported recovery procedure; deleting the PVC can permanently destroy its data

If only one component shows the taint error after you added a per-component override, remember that the override replaced the global tolerations for that component; copy the global entries into the override.

To see the scheduling fields the operator actually rendered, inspect the workload rather than the custom resource:

$ kubectl get statefulset fuzzball-jetstream -n fuzzball -o jsonpath='{.spec.template.spec.affinity}' | jq .

$ kubectl get deployment fuzzball-orchestrator -n fuzzball -o jsonpath='{.spec.template.spec.nodeSelector}' | jq .

Substitute tolerations for the field name to check that setting.

Ingress Not Working

Check ingress configuration:

$ kubectl get ingress -A

$ kubectl describe ingress <ingress-name> -n fuzzball

Verify domain DNS resolution and load balancer IP assignment.

See Also