FuzzballOrchestrate CRD Reference
The FuzzballOrchestrate Custom Resource Definition (CRD) is the primary way to deploy and
configure a Fuzzball cluster on Kubernetes. This CRD provides comprehensive control over all aspects
of a Fuzzball deployment, from basic infrastructure (database, ingress, authentication) to advanced
features (multi-cloud provisioning, billing integration, autoscaling).
The minimal configuration requires image credentials, ingress configuration, database setup, and authentication. This example is suitable for local or on-premises deployments:
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
image:
username: ${DEPOT_USER}
password: ${ACCESS_KEY}
exclusive: false
ingress:
create:
domain: "10.0.0.99.nip.io"
proxy:
type: LoadBalancer
annotations:
metallb.io/loadBalancerIPs: 10.0.0.99
database:
create:
storage:
class: longhorn
keycloak:
create:
ingress:
hostname: auth.10.0.0.99.nip.io
realmName: Fuzzball
realmId: 550e8400-e29b-41d4-a716-446655440000
username: keycloak
ownerEmail: "admin@example.com"
createDatabase: true
tls:
certManager:
create: {}
trustManager:
create: {}
fuzzball:
substrate:
nfs:
destination: "/fuzzball/shared"
server: 10.0.0.10
path: "/srv/fuzzball/shared"
jetstream:
replicas: 3
storage:
class: longhorn
size: 10Gi
Many credential fields (passwords, API keys, TLS certificates, image pull secrets) support references to Kubernetes Secrets as an alternative to inline values. Secret references are recommended for production deployments. See Kubernetes Secret References for the full list of fields and usage examples.
Controls where container images are pulled from and authentication:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
repository | string | No | depot.ciq.com/fuzzball/fuzzball-images | Container image registry |
credentialsSecretName | string | No | - | Name of a kubernetes.io/dockerconfigjson Secret with registry credentials (recommended) |
username | string | No | - | Registry authentication username (deprecated — use credentialsSecretName) |
password | string | No | - | Registry authentication password (deprecated — use credentialsSecretName) |
exclusive | boolean | No | true | If true, all images must come from specified repository |
pullPolicy | string | No | Always | imagePullPolicy for the Fuzzball services, SpiceDB, and the Fuzzball jobs and sidecars: Always, IfNotPresent, or Never |
Example with inline credentials:
spec:
image:
repository: depot.ciq.com/fuzzball/fuzzball-images
username: my-depot-user
password: my-depot-token
exclusive: false
Example with Secret reference (recommended for production):
spec:
image:
credentialsSecretName: depot-pull-secret
exclusive: false
With the default pullPolicy: Always, every pod start contacts the registry, so a pod that
restarts while the registry is unreachable fails with ImagePullBackOff even when the image is
already on the node. Set pullPolicy: IfNotPresent to start pods from images already on the
node, for example on an air-gapped or intermittently connected site:
spec:
image:
pullPolicy: IfNotPresent
This setting does not cover other third-party components (PostgreSQL, Keycloak, the ingress
controller, cert-manager, Prometheus, Alertmanager, and Grafana), the OCI DNS webhook, or CoreWeave
substrate pods. These keep their own pull policies. It also does not cover the operator’s own
Deployment, which the fuzzball-operator Helm chart deploys. Set its pull policies with the chart
values image.pullPolicy and kubeRbacProxyImage.pullPolicy.
Defines how the cluster is exposed to the network. Choose either create (new ingress) or
external (existing ingress controller).
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
domain | string | Yes | - | Base domain for all Fuzzball services |
proxy.type | string | Yes | LoadBalancer | Service type: LoadBalancer or NodePort |
proxy.annotations | map | No | {} | Annotations for the proxy service (e.g., MetalLB config) |
proxy.http.nodePort | integer | Yes (if proxy.type is NodePort) | - | NodePort for HTTP |
proxy.tls.nodePort | integer | Yes (if proxy.type is NodePort) | - | NodePort for HTTPS |
Example with LoadBalancer:
spec:
ingress:
create:
domain: fuzzball.example.com
proxy:
type: LoadBalancer
annotations:
metallb.io/loadBalancerIPs: 10.0.0.99
metallb.io/allow-shared-ip: ingress-and-fuzzball
CoreWeave requires specific annotations on the LoadBalancer service for automatic DNS configuration:
| Annotation | Value | Description |
|---|---|---|
service.beta.kubernetes.io/coreweave-load-balancer-type | public | Creates an internet-accessible load balancer |
service.beta.kubernetes.io/external-hostname | *.<domain> | Wildcard domain for automatic DNS resolution |
CoreWeave example:
spec:
ingress:
create:
domain: a1b2c3-my-cluster.coreweave.app
proxy:
type: LoadBalancer
annotations:
service.beta.kubernetes.io/coreweave-load-balancer-type: public
service.beta.kubernetes.io/external-hostname: '*.a1b2c3-my-cluster.coreweave.app'
Example with NodePort:
spec:
ingress:
create:
domain: fuzzball.example.com
proxy:
type: NodePort
http:
nodePort: 30080
tls:
nodePort: 30443
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
domain | string | Yes | - | Base domain for all Fuzzball services |
className | string | Yes | - | Ingress class name (e.g., nginx, traefik) |
controller | string | No | Detected | Ingress controller that serves className: nginx (ingress-nginx), kong, or traefik |
annotations | map | No | {} | Annotations for ingress resources |
The operator adds controller-specific settings to the ingresses it creates. For example, it adds the
gRPC backend protocol for ingress-nginx. To choose these settings, it reads the controller from the
IngressClass named by className. A class that isn’t found, or whose controller the operator doesn’t
recognize, is treated as if its name were the controller. Set controller when detection gets it
wrong, for example when an ingress-nginx class reports a controller value that does not contain
ingress-nginx.
With Traefik, the operator serves the gRPC API through a Traefik IngressRoute and sets no
kubernetes.io/ingress.class annotation on it, so a Traefik whose CRD provider has no
ingressClass filter picks it up. If your Traefik filters CRD resources by class
(providers.kubernetesCRD.ingressClass), set the annotation to that filter’s value in
spec.fuzzball.config.apiIngress.annotations. These annotations go on the gRPC API ingress
for every controller. Don’t set kubernetes.io/ingress.class here for nginx or Kong: Kubernetes
rejects an Ingress whose class annotation differs from its ingressClassName.
spec:
fuzzball:
config:
apiIngress:
annotations:
kubernetes.io/ingress.class: traefik-internal
Example:
spec:
ingress:
external:
domain: fuzzball.example.com
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
Sign-in responses from the Web UI and Keycloak carry cookies that are larger than the default
4 KB header buffer of ingress-nginx, which answers with a 502 error. On the nginx class, the
operator sets nginx.ingress.kubernetes.io/proxy-buffer-size to 32k on the Web UI ingress and to
128k on the Keycloak ingress. To use a different size, set the same annotation in
spec.fuzzball.ui.ingress.annotations or spec.keycloak.create.ingress.annotations. For any other
nginx-based controller, raise its proxy buffer size to at least these values.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
publicPort | integer | No | 443 | Port clients use to reach the ingress hostnames. Set it when clients connect to the NodePort directly. Setting 443 is the same as leaving it unset |
Fuzzball puts this port in the URLs it gives to clients: the Keycloak issuer and OIDC endpoints, the
UI and API URLs, and the URLs that compute nodes use to reach the cluster. With
proxy.type: NodePort and nothing in front of the node port, set it to proxy.tls.nodePort.
Otherwise the CLI and the UI are sent to port 443 and the login fails. Leave it unset when a load
balancer or port forward serves the ingress on port 443. It applies to both create and external
ingress. It does not change an external Keycloak (spec.keycloak.external), whose issuer comes from
its own URL.
spec:
ingress:
publicPort: 30443
create:
domain: fuzzball.example.com
proxy:
type: NodePort
http:
nodePort: 30080
tls:
nodePort: 30443
Keep the following in mind:
- Some Fuzzball jobs inside the cluster also use these URLs, so the ingress hostnames must also resolve inside the cluster to an address where the node port is open, such as a node IP.
- Service endpoint URLs and the monitoring node-scrape discovery URL still use port 443, so they are not reachable through a bare node port.
- Setting or changing
publicPorton a running cluster changes the token issuer. Existing sessions and CLI contexts stop working: users log in again, and CLI contexts are recreated with the port. Any other cluster that lists this cluster’s Keycloak inadditionalTrustedIssuersmust include the port, for examplehttps://auth.fuzzball.example.com:30443/.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
spec.fuzzball.ui.cookieScope | string | No | auto | Which hostnames a browser sends the Web UI login cookie to: auto, domain, or host |
The Web UI keeps the login token in a browser cookie. A browser only stores a cookie that is shared across a domain when the page that set it is served from that domain, so the scope has to fit the hostname users open the Web UI on:
autoshares the cookie across the ingressdomainwhen the Web UI hostname is that domain or under it, for exampleui.fuzzball.example.comwith the domainfuzzball.example.com. For any other Web UI hostname, such asfuzzball-ui.example.com, it keeps the cookie on the Web UI hostname.domainalways shares the cookie across the ingressdomain. If the Web UI hostname is not under that domain, the browser discards the cookie and every login returns to the login page.hostalways keeps the cookie on the Web UI hostname.
When the cookie stays on the Web UI hostname, browsers do not send it to workflow service endpoints, so opening a service endpoint in a browser does not pick up the Web UI login.
auto decides from the configured spec.fuzzball.ui.ingress.hostname, not from the hostname in the
browser’s address bar. If a reverse proxy serves the Web UI under a different hostname that is not
under the ingress domain, set cookieScope: host.
A change to cookieScope applies to each user at their next login.
spec:
fuzzball:
ui:
cookieScope: host
Fuzzball requires a PostgreSQL database. Choose either create (deploys a dedicated PostgreSQL
alongside Orchestrate) or external.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
enableDebugPod | boolean | No | false | Deploy a debug pod with database tools |
annotations | map | No | {} | Annotations for database resources |
storage.class | string | No | - | StorageClass for database PVC |
storage.size | string | No | - | Size of database storage |
storage.accessMode | string | No | ReadWriteOnce | Access mode for PVC |
storage.annotations | map | No | {} | Annotations for PVC |
Example:
spec:
database:
create:
enableDebugPod: true
storage:
class: longhorn
size: 200Gi
accessMode: ReadWriteOnce
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
host | string | Yes | - | Database hostname or IP |
port | string | No | 5432 | Database port |
driver | string | No | postgres | Database driver |
credentials.user | string | Yes | - | Database username |
credentials.password | string | No | - | Database password (inline) |
credentials.passwordRef | object | No | - | Reference to a K8s Secret containing the database password (preferred over inline) |
sslMode | string | No | verify-full | SSL mode: disable, require, verify-ca, verify-full |
rdsSecretId | string | No | - | AWS Secrets Manager ARN for RDS credentials |
certificate.caCert | string | No | - | CA certificate for SSL verification (inline) |
certificate.caCertURL | string | No | - | URL to download CA certificate |
certificate.caCertRef | object | No | - | Reference to a K8s Secret containing the CA certificate (preferred over inline) |
certificate.clientCert | string | No | - | Client certificate for mTLS (inline) |
certificate.clientCertRef | object | No | - | Reference to a K8s Secret containing the client certificate (preferred over inline) |
certificate.clientKey | string | No | - | Client key for mTLS (inline) |
certificate.clientKeyRef | object | No | - | Reference to a K8s Secret containing the client key (preferred over inline) |
Example with external PostgreSQL:
spec:
database:
external:
host: postgres.example.com
port: "5432"
driver: postgres
credentials:
user: fuzzball
passwordRef:
name: db-credentials
key: password
sslMode: verify-full
Example with AWS RDS:
spec:
database:
external:
host: mydb.abc123.us-east-1.rds.amazonaws.com
rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:rds-db-credentials-abc123
sslMode: verify-full
certificate:
caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
For Keycloak-backed deployments, Fuzzball delegates authentication to Keycloak. Choose either
create (deploy Keycloak alongside Orchestrate) or external. As of v3.3.0, Keycloak is optional —
deployments that use the local-organization flow authenticate directly with Fuzzball without a
Keycloak instance.
As of v3.3.0, plaintextpasswordanddefaultUserPasswordfields are deprecated. Use thepasswordRefanddefaultUserPasswordReffields to reference passwords stored in Kubernetes Secrets. The plaintext fields continue to work but will be removed in a future release.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
ownerEmail | string | Yes | - | Email address for the Fuzzball organization owner. Must not be empty — see Owner email is required |
realmId | string | No | (generated) | Keycloak realm ID (must be UUID v4, lower case) |
realmName | string | No | Fuzzball | Keycloak realm display name |
username | string | No | keycloak | Keycloak admin username |
password | string | No | (generated) | Deprecated. Keycloak admin password (plaintext). Use passwordRef instead. |
passwordRef.name | string | No | - | Name of Kubernetes Secret containing the Keycloak admin password |
passwordRef.key | string | No | - | Key within the Secret for the Keycloak admin password |
passwordRef.namespace | string | No | fuzzball-system | Namespace of the Secret; defaults to the Fuzzball system namespace |
defaultUserPassword | string | No | - | Deprecated. Initial password for the organization owner (plaintext). Use defaultUserPasswordRef instead. |
defaultUserPasswordRef.name | string | No | - | Name of Kubernetes Secret containing the default user password |
defaultUserPasswordRef.key | string | No | - | Key within the Secret for the default user password |
defaultUserPasswordRef.namespace | string | No | fuzzball-system | Namespace of the Secret; defaults to the Fuzzball system namespace |
createDatabase | boolean | No | true | Create dedicated database for Keycloak |
replicas | integer | No | 1 | Number of Keycloak replicas |
resources.requests | map | No | {memory: 3Gi} | Memory (and optionally CPU) requested per Keycloak pod. See the sizing note below before changing. |
resources.limits | map | No | {memory: 3Gi} | Memory (and optionally CPU) ceiling per Keycloak pod. Keep equal to resources.requests. |
ingress.hostname | string | No | auth.{domain} | Hostname for Keycloak UI |
ingress.className | string | No | - | Ingress class name |
ingress.tls.cert | string | No | - | TLS certificate (inline) |
ingress.tls.key | string | No | - | TLS private key (inline) |
ingress.tls.secretName | string | No | - | Name of a kubernetes.io/tls Secret containing the TLS cert and key (preferred over inline) |
ingress.annotations | map | No | {} | Ingress annotations |
Sizing Keycloak memory.
resources.requestsandresources.limitsdefault to the same value on purpose. Setting a memory request at all takes the Keycloak pod out of BestEffort QoS, so the kubelet no longer treats it as the first candidate to evict; the limit gives the JVM a container boundary to size its heap from (roughly 2.1GiB of heap under a 3Gi limit); and keeping the two equal means usage can never exceed the request, which puts the pod in the kubelet’s last-to-evict group. Deployments that ship without them can drift to several GiB per pod and be evicted or exit under memory pressure – which, on a single-replica deployment, takes every in-flight login session with it. Note this is Burstable QoS, not Guaranteed, and it is ranked last for eviction rather than immune to it.At the 3Gi default, each Keycloak replica reserves 3Gi of schedulable memory – 9Gi across a three-replica deployment. Confirm your nodes have that to give before upgrading, or the pods will stay
Pending. Small local clusters should lower both values together.Always override
requestsandlimitstogether, and always restatememoryin each. The two default independently, and setting either one replaces it entirely rather than merging with the default. Two mistakes are easy to make:
- Moving only one map. Lowering just
limitsto2Gileavesrequestsat the3Gidefault, and Kubernetes rejects the pod outright:spec.containers[0].resources.requests: Invalid value: "3Gi": must be less than or equal to memory limit of 2Gi. Raising onlyrequestsabove3Gifails the same way against the default limit. The Keycloak StatefulSet will not update until both agree.- Dropping
memoryfromlimits. This is the quietest way to undo the fix. Adding a CPU limit aslimits: {cpu: "500m"}replaces the whole map, leaving a 3Gi memory request and no memory limit. Nothing catches it – the API server accepts it, the pod schedules, and the JVM goes back to sizing its heap from node memory. Kubernetes copies a limit into a matching missing request but never the reverse, so droppingmemoryfromrequestsalone is harmless while dropping it fromlimitsalone is not. Do not rely on that asymmetry: restatememoryin both.Override both, in full, at the path they live under:
spec: keycloak: create: resources: requests: cpu: "500m" memory: "3Gi" limits: cpu: "500m" memory: "3Gi"Changing these values updates the Keycloak StatefulSet pod template, which performs a rolling restart. What that costs depends on
replicas:
replicas: 1(the default): the single pod holds every session, so a restart drops them all and any login in progress must start over.- Multiple replicas: Keycloak clusters its Infinispan caches across pods and replicates the session caches to two owners, so a rolling restart that replaces one pod at a time preserves them. A user whose request is in flight against the pod being replaced may see one failed request, but their session survives on another owner.
Sessions are never persisted to storage in either case, so losing all replicas at once loses every session regardless of this setting.
Preferred example using SecretKeyReference:
spec:
keycloak:
create:
ownerEmail: admin@example.com
realmName: MyOrganization
realmId: 550e8400-e29b-41d4-a716-446655440000
username: keycloak
passwordRef:
name: keycloak-credentials
key: admin-password
defaultUserPasswordRef:
name: keycloak-credentials
key: default-user-password
createDatabase: true
ingress:
hostname: auth.fuzzball.example.com
Generate a UUID v4 for the realm ID:
$ uuidgenUse the output in your configuration:
realmId: 550e8400-e29b-41d4-a716-446655440000
The ownerEmail and defaultUserPassword fields create the initial administrator account with full
cluster administration privileges. Change the default password after first login.
ownerEmail: admin@example.com
defaultUserPassword: initial-secure-password
Keycloak can be configured to federate users from an LDAP/Active Directory server:
| Parameter | Type | Required | Description |
|---|---|---|---|
ldap.url | string | Yes | LDAP server URL (ldap:// or ldaps://) |
ldap.startTLS | boolean | No | Use StartTLS for ldap:// connections |
ldap.insecure | boolean | No | Skip TLS certificate verification |
ldap.vendor | string | No | LDAP vendor: ad, rhds, tivoli, edirectory, other |
ldap.bindDN | string | No | Bind DN for LDAP authentication |
ldap.bindPassword | string | No | Password for bind DN (inline) |
ldap.bindPasswordRef | object | No | Reference to a K8s Secret containing the bind password (preferred over inline) |
ldap.searchScope | string | No | Search scope: single or subtree |
ldap.users.dn | string | Yes | Base DN for user search |
ldap.users.attributes.* | string | Yes | User attribute mappings |
ldap.users.objectClasses | []string | Yes | User object classes |
ldap.users.filter | string | No | LDAP filter for users |
ldap.groups.dn | string | Yes | Base DN for group search |
ldap.groups.membershipAttributeType | string | Yes | Membership type: dn or uid |
ldap.groups.userGroupsStrategy | string | Yes | Strategy for loading groups |
ldap.groups.attributes.* | string | Yes | Group attribute mappings |
ldap.groups.objectClasses | []string | Yes | Group object classes |
ldap.groups.filter | string | No | LDAP filter for groups |
LDAP example:
spec:
keycloak:
create:
ownerEmail: admin@example.com
ldap:
url: ldaps://ldap.example.com
vendor: ad
bindDN: cn=fuzzball,ou=service,dc=example,dc=com
bindPassword: ldap-bind-password
searchScope: subtree
users:
dn: ou=users,dc=example,dc=com
attributes:
username: sAMAccountName
rdn: sAMAccountName
uuid: objectGUID
uidNumber: uidNumber
gidNumber: gidNumber
objectClasses:
- person
- organizationalPerson
filter: "(memberOf=cn=fuzzball-users,ou=groups,dc=example,dc=com)"
groups:
dn: ou=groups,dc=example,dc=com
membershipAttributeType: dn
userGroupsStrategy: get_groups_from_user_memberof_attribute
attributes:
groupName: cn
groupMembership: member
userMembership: sAMAccountName
memberOf: memberOf
gidNumber: gidNumber
objectClasses:
- group
filter: "(cn=fuzzball-*)"
| Parameter | Type | Required | Description |
|---|---|---|---|
url | string | Yes | Keycloak server URL |
realmId | string | Yes | Existing realm ID (UUID v4) |
realmName | string | Yes | Existing realm name |
username | string | Yes | Admin username |
password | string | No | Admin password (inline) |
passwordRef | object | No | Reference to a K8s Secret containing the admin password (preferred over inline) |
ownerEmail | string | Yes | Organization owner email. Must not be empty — see Owner email is required |
useLDAPFederation | boolean | No | Whether LDAP is configured |
Example:
spec:
keycloak:
external:
url: https://keycloak.example.com
realmId: 550e8400-e29b-41d4-a716-446655440000
realmName: ExistingRealm
username: admin
passwordRef:
name: keycloak-credentials
key: admin-password
ownerEmail: admin@example.com
ownerEmail must name a real user, on both the create and external Keycloak variants.
It becomes the owner of the cluster’s initial organization, so an empty value leaves that
organization with no administrator.
The operator enforces this at two different strengths, according to whether there is a running cluster to protect:
- A resource that has never completed a deploy is rejected. Reconciliation stops, the
resource goes
Degraded, and aMisconfiguredcondition names the field to fix. This covers a resource being applied for the first time and one whose first deploy failed — in both cases nothing is serving, so there is nothing an outright refusal can break. - A resource that has deployed successfully at least once keeps reconciling. An
administrator may already have added an owner by hand, so an upgrade is not blocked. A
Misconfiguredcondition and a warning event record the problem instead. This still applies when the most recent deploy failed: the previous deploy’s workloads are still running.
Correcting the field clears the condition on the next reconciliation.
Optional configuration for certificate management and Let’s Encrypt:
| Parameter | Type | Required | Description |
|---|---|---|---|
certManager.create | object | No | Set to {} to have the operator deploy cert-manager. Omit create (an empty certManager: {} does not deploy) to use an existing installation |
certManager.create.serviceAccount.annotations | map | No | Service account annotations (for IRSA) |
certManager.clusterResourceNamespace | string | No | Namespace where an existing cert-manager reads ClusterIssuer secrets (default: cert-manager). Cannot be combined with certManager.create |
trustManager.create | object | No | Set to {} to have the operator deploy trust-manager. Omit create to use an existing installation |
When no issuer is configured, the operator generates a self-signed internal CA and issues both internal and ingress certificates from it.
| Parameter | Type | Required | Description |
|---|---|---|---|
internalIssuer | object | No | Internal certificate issuer configuration |
ingressIssuer.create | object | No | Create an ingress issuer. An empty create: {} uses the internal issuer, the same as omitting it |
ingressIssuer.create.letsEncrypt.email | string | No | Email for Let’s Encrypt notifications |
ingressIssuer.create.letsEncrypt.issuer | string | No | Issuer name (e.g., letsencrypt-prod, letsencrypt-staging) |
ingressIssuer.create.letsEncrypt.solvers | []object | No | ACME challenge solvers (dns01 or http01) |
ingressIssuer.external.issuerName | string | No | Name of existing ClusterIssuer for ingress TLS (preferred) |
ingressIssuer.external.internalCAIssuerName | string | No | Name of existing ClusterIssuer for ingress TLS (deprecated, use issuerName) |
internalIssuer.external.issuerName | string | No | Name of existing ClusterIssuer for internal TLS (preferred) |
internalIssuer.external.internalCAIssuerName | string | No | Name of existing ClusterIssuer for internal TLS (deprecated, use issuerName) |
internalIssuer.create.customCA.inline.secretName | string | No | Name of a kubernetes.io/tls Secret in cert-manager’s cluster resource namespace (certManager.clusterResourceNamespace, default cert-manager) providing the CA cert/key for the internal issuer (otherwise the operator generates a self-signed CA) |
ingressIssuer.create.customCA.inline.secretName | string | No | Name of a kubernetes.io/tls Secret in cert-manager’s cluster resource namespace (certManager.clusterResourceNamespace, default cert-manager) providing the CA cert/key for the ingress issuer |
A customCA must set either inline.secretName or, deprecated, both inline.cert and
inline.key. A non-empty secretName takes precedence when both are set. The operator reads the
Secret when it reconciles; editing the Secret alone does not start a reconcile. To roll out a
new CA, create a new Secret and point secretName at it; the operator replaces its copy of the CA.
Certificates already issued keep the old CA until they are renewed. For the ingress issuer, renew
them with cmctl:
$ kubectl get certificates -A \
-o jsonpath='{range .items[?(@.spec.issuerRef.name=="ingress-ca-issuer")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' \
| while read ns name; do cmctl renew -n "$ns" "$name"; doneWhen using external issuers, the operator can automatically extract the CA certificate from the ClusterIssuer and distribute it via trust-manager, assuming trust-manager is running. EnabletrustManager.createso the operator deploys it, or install trust-manager separately. When trust-manager is available, manual creation of thetrusted-ca-certs-bundleConfigMap is no longer required.trustManager.createcan be set independently ofcertManager.createwhen using external issuers.
Example with Let’s Encrypt (DNS-01):
spec:
tls:
certManager:
create: {}
trustManager:
create: {}
ingressIssuer:
create:
letsEncrypt:
email: admin@example.com
issuer: letsencrypt-prod
solvers:
- dns01:
route53:
region: us-east-1
hostedZoneID: Z1234567890ABC
Example with your own CA:
Store the CA certificate and private key in a kubernetes.io/tls Secret in the cert-manager
namespace, creating the namespace first if it does not exist yet. With an existing cert-manager and
certManager.clusterResourceNamespace set, create the Secret in that namespace instead:
$ kubectl create namespace cert-manager
$ kubectl create secret tls my-root-ca \
--cert=path/to/ca.crt \
--key=path/to/ca.key \
-n cert-managerKeep a copy of the CA outside the cluster: with certManager.create set, the operator manages the
cert-manager namespace and deleting the Fuzzball custom resource removes it, including this
Secret.
Then reference it from either issuer. This example sets the ingress issuer; use
internalIssuer.create.customCA for the internal CA:
spec:
tls:
certManager:
create: {}
trustManager:
create: {}
ingressIssuer:
create:
customCA:
inline:
secretName: my-root-ca
Example with Let’s Encrypt (HTTP-01):
spec:
tls:
certManager:
create: {}
ingressIssuer:
create:
letsEncrypt:
email: admin@example.com
issuer: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
The optional email section configures the SMTP relay used to send signup verification emails.
When omitted, verification tokens are written to the Fuzzball agent’s stdout instead — useful for
development but not suitable for production use of the self-signup feature.
| Parameter | Type | Required | Description |
|---|---|---|---|
email.smtpHost | string | Yes | Hostname of the SMTP relay |
email.smtpPort | integer | Yes | TCP port of the SMTP relay (587 for STARTTLS, 465 for TLS, 25 for plaintext) |
email.fromAddress | string | Yes | Envelope sender address (e.g., noreply@example.com) |
email.tlsMode | string | No | Connection security: none, starttls (default), or tls |
email.credentials.secretName | string | Yes (if credentials set) | Name of the Kubernetes Secret containing SMTP credentials |
email.credentials.usernameKey | string | No | Key in the Secret for the SMTP username (default: username) |
email.credentials.passwordKey | string | No | Key in the Secret for the SMTP password (default: password) |
Example — unauthenticated relay:
spec:
email:
smtpHost: mailrelay.internal
smtpPort: 25
fromAddress: noreply@example.com
tlsMode: none
Example — STARTTLS with credentials (e.g., AWS SES):
spec:
email:
smtpHost: email-smtp.us-east-1.amazonaws.com
smtpPort: 587
fromAddress: noreply@example.com
tlsMode: starttls
credentials:
secretName: smtp-credentials
When
Optional top-level fields that constrain which Kubernetes nodes Fuzzball service pods are scheduled onto. All three fields apply globally to every Fuzzball service pod as defaults, and each can be overridden per service component (see Service-Specific Configuration). A per-service override replaces (does not merge with) the global value for that service.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
nodeSelector | map | No | {} | Node label selector applied to all Fuzzball service pods; overridable per component |
tolerations | []object | No | [] | Tolerations applied to all Fuzzball service pods; overridable per component |
affinity | object | No | - | Affinity/anti-affinity rules applied to all Fuzzball service pods; overridable per component |
Some operator-managed workloads have no scheduling fields of their own and inherit them from a related service:
- Catalog jobs (catalog sync and load-default-catalog) use the
agentservice’s scheduling - The resource-definition setup job uses the
orchestratorservice’s scheduling - The sync workload uses the
substrateBridgeservice’s scheduling - The SpiceDB authorization database uses the
authservice’s scheduling
The supporting services the operator deploys through Helm charts receive only the global values.
When database.create or keycloak.create is set, the top-level nodeSelector, tolerations, and
affinity are passed through to the PostgreSQL and Keycloak charts. The optional monitoring stack
does not use these fields.
All three fields are optional and were added without changing any defaults, so resources created before they existed deploy exactly as before after an upgrade. Adding, changing, or removing a value triggers a rolling restart of every affected pod. Pods backed by node-local persistent volumes stay on their current node after the restart; see the JetStream example below.
Example: dedicated node pool
Constrain all Fuzzball service pods to nodes labeled workload: fuzzball:
spec:
nodeSelector:
workload: fuzzball
Example: tolerating a tainted node
Allow Fuzzball pods to run on nodes tainted fuzzball=system:NoSchedule:
spec:
tolerations:
- key: fuzzball
value: system
operator: Equal
effect: NoSchedule
Example: node affinity
Require Fuzzball pods to land in a specific availability zone:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-1a
Example: per-component override
Schedule all Fuzzball pods on nodes labeled workload: fuzzball, but place JetStream pods on nodes
labeled storage: high-iops and tolerate a storage taint — the per-service values replace the
global ones for JetStream only:
spec:
nodeSelector:
workload: fuzzball
fuzzball:
jetstream:
nodeSelector:
storage: high-iops
tolerations:
- key: storage
operator: Exists
effect: NoSchedule
JetStream is deployed with replicated state: it runs as a three-node NATS cluster by default and
keeps three copies of every stream. The operator does not apply anti-affinity of its own, so the
Kubernetes scheduler is free to place all three JetStream pods on the same node, which gives no
protection against that node failing. Production deployments that want node-failure tolerance should
set affinity on the jetstream component.
The preferred form below is safe on any cluster size, including single-node development clusters, because the scheduler still places the pods when it cannot honor the preference:
spec:
fuzzball:
jetstream:
replicas: 3
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: fuzzball-jetstream
On a cluster with at least three nodes eligible for JetStream, you can make the spread a hard rule
with the required form. Required terms carry no weight and no podAffinityTerm wrapper, so
topologyKey and labelSelector sit directly in the list item:
spec:
fuzzball:
jetstream:
replicas: 3
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: fuzzball-jetstream
If fewer than three eligible nodes are available, the JetStream pods that cannot be placed stay
Pending. To spread across availability zones instead of nodes, set
topologyKey: topology.kubernetes.io/zone, but only when the JetStream storage class can attach
volumes across zones.
A per-component
affinityreplaces the global one. Settingjetstream.affinitydiscards the top-levelspec.affinityfor JetStream. If the global value carries anodeAffinitythat pins Fuzzball onto labeled nodes, copy that block intojetstream.affinitynext to thepodAntiAffinity:spec: fuzzball: jetstream: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload operator: In values: - fuzzball podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: topologyKey: kubernetes.io/hostname labelSelector: matchLabels: app.kubernetes.io/name: fuzzball-jetstreamExisting pods. Adding the field restarts the JetStream pods, but if a pod’s
jetstream-datavolume isReadWriteOnceand on a node-local storage class it pins the pod to the node that first scheduled it. The spread takes effect for new pods, or after deleting a pod’s PersistentVolumeClaim and restarting it so it is re-provisioned elsewhere. Do this one pod at a time and wait for the stream to resynchronize before the next. On network block storage the pods can relocate without this step.Running pods are never evicted by this rule. Both forms end in
IgnoredDuringExecution: the rule is checked only when a pod is scheduled, never afterwards.
Draining a kubernetes node with more than one JetStream replica results in evicting all its
JetStream replicas at the same time. This can take the JetStream cluster below the minimal number of
replicas for quorum. Anti-affinity makes that layout less likely and a PodDisruptionBudget stops
the drain from going through if it happens anyway. The two are complementary.
An administrator created disruption budget is a separate Kubernetes resource, attached to pods by a
label selector. It persists through Fuzzball upgrades since the Fuzzball operator will not manage or
remove it. Create the budget by hand in the Fuzzball namespace (fuzzball for
FuzzballOrchestrate, fuzzball-federate for FuzzballFederate).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: fuzzball-jetstream
namespace: fuzzball
spec:
minAvailable: 2
selector:
matchLabels:
app.kubernetes.io/name: fuzzball-jetstream
minAvailable: 2 assumes replicas: 3. Do not create a budget for a single-replica JetStream; a
drain of its node would block indefinitely.
See the Kubernetes documentation for
full syntax details on tolerations and affinity objects, and the disruptions
page for PodDisruptionBudget.
Fuzzball builds fully qualified in-cluster names, such as the JetStream cluster routes and the
internal Keycloak address, on the cluster’s DNS domain. Most clusters use the Kubernetes default,
cluster.local, but some distributions let you choose another (for example k3s and RKE2 with
cluster-domain). Both FuzzballOrchestrate and FuzzballFederate support this field.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
clusterDomain | string | No | Detected, else cluster.local | DNS domain of the Kubernetes cluster, without a leading or trailing dot (e.g. fuzz.internal) |
When clusterDomain is unset, the operator reads the domain from the svc.<domain> entry the
kubelet writes into the search list of its own pod’s /etc/resolv.conf, and falls back to
cluster.local when there is no such entry. The operator logs which domain it uses, and its
source, on every reconcile, with a WARNING when it falls back. Set clusterDomain only if that
detection is wrong, for example when the operator pod does not use cluster DNS, and set it at
install time.
On a cluster whose domain is not cluster.local, a wrong domain leaves the JetStream pods stuck at
0/1 with Waiting for routing to be established in their logs, and the rest of the services are
never deployed.
Keep the following in mind:
- Once set,
clusterDomaincannot be changed: the API server rejects the edit. To change it, remove the field so the operator falls back to detection, or recreate the resource. - Removing
clusterDomain, or a change in the detected domain, re-renders the in-cluster names and restarts the affected services, JetStream included. - Custom domains require the operator and the Fuzzball images to come from the same release. An older JetStream image ignores the domain the operator passes it, and its routes fail TLS verification.
- If the operator runs behind an egress proxy and you set
clusterDomainbecause detection is wrong, also add.<domain>to the operator chart’sproxy.noProxy. The operator adds only the detected domain to its ownNO_PROXY.
Example:
spec:
clusterDomain: fuzz.internal
For environments where outbound internet traffic must pass through an HTTP(S) proxy, the top-level
proxy section injects the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment
variables into the Fuzzball service pods, batch jobs, and the Keycloak identity service. The
internal authorization database (SpiceDB) makes no outbound connections and is excluded. Both
FuzzballOrchestrate and FuzzballFederate support the same fields.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
proxy.httpProxy | string | No | "" | Proxy URL for HTTP requests, e.g. http://proxy.example.com:3128 |
proxy.httpsProxy | string | No | "" | Proxy URL for HTTPS requests |
proxy.noProxy | []string | No | [] | Additional hosts, domain suffixes (written with a leading dot, e.g. .example.com), and CIDRs that bypass the proxy |
Cluster-internal destinations are always excluded from proxying automatically: the operator appends
the Kubernetes API endpoint, the .svc and .cluster.local suffixes, the cluster DNS
domain, Fuzzball’s internal service names, and the cluster’s ingress domain to
NO_PROXY. Add noProxy entries only for destinations
specific to your environment.
If your pods send telemetry to a node-local collector (for example a Datadog agent DaemonSet), add your node and pod network CIDRs tonoProxy— per-node IP addresses cannot be excluded automatically.
When the proxy intercepts TLS connections and re-signs them with a private certificate authority,
add that CA through the trustedCACerts section. The certificates are distributed to every
Fuzzball service pod through the cluster trust bundle mounted at /etc/ssl/certs, alongside the
default public CAs.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
trustedCACerts.refs | []object | No | [] | References to Secrets containing PEM-encoded CA certificates; each entry has name, key, and optional namespace (defaults to fuzzball-system) |
trustedCACerts.inline | []string | No | [] | PEM-encoded CA certificates provided directly in the resource |
Example: deploy behind a TLS-intercepting corporate proxy
spec:
proxy:
httpProxy: http://proxy.example.com:3128
httpsProxy: http://proxy.example.com:3128
noProxy:
- .internal.example.com
- 10.0.0.0/8
trustedCACerts:
refs:
- name: corporate-root-ca
key: ca.crt
Create the referenced Secret in the fuzzball-system namespace before applying the resource:
$ kubectl create secret generic corporate-root-ca \
--namespace fuzzball-system \
--from-file=ca.crt=/path/to/corporate-root-ca.pemInvalid proxy URLs and certificates that fail PEM parsing are rejected when the operator processes the resource; the running deployment continues unchanged until you correct them.
These fields configure only the workloads the operator deploys. The operator’s own deployment is configured separately through Helm chart values — see Configure an Egress Proxy for the Operator.
The top-level monitoring section configures the bundled Prometheus, Alertmanager, and Grafana
stack, preconfigured against Fuzzball’s own metrics and compute-node exporters, with the node
health alert rules, the Alertmanager routes that place node holds, and the Fuzzball dashboards.
The stack is deployed by default, with its recommended configuration, when this section is unset.
Set monitoring.enabled: false to deploy nothing – for example, on a site that runs its own
monitoring and adopts the
published configuration
instead. The stack lives in its own namespace (fuzzball-monitoring by default), separate from the
Fuzzball namespaces, so it can be removed without touching the control plane.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
monitoring.enabled | boolean | No | true | Deploy the bundled monitoring stack. Set false to deploy nothing except what monitoring.alerting.externalReceiver asks for |
monitoring.namespace | string | No | fuzzball-monitoring | Kubernetes namespace to deploy the stack into |
monitoring.retentionDays | integer | No | 15 | How long Prometheus keeps samples. Minimum 1 |
monitoring.storageSize | string | No | 20Gi | Prometheus persistent volume size. Set to the empty string ("") to run Prometheus and Alertmanager on emptyDir instead – no storage class required, history and silences are lost on restart |
monitoring.storageClassName | string | No | "" | StorageClass for the Prometheus and Alertmanager volumes. Empty uses the cluster default. With no default and no storageSize: "", the stack is left out with a MonitoringNotDeployed warning event when monitoring.enabled is unset, and the resource is refused when it is true |
monitoring.nodeMetricsMode | string | No | proxy | proxy or direct. Must match nodeMetrics.serviceDiscovery.mode in central config |
monitoring.prometheus.valuesOverride | string | No | "" | YAML document of Prometheus Helm chart values merged over the generated ones |
monitoring.grafana.enabled | boolean | No | true | Deploy Grafana with the bundled Fuzzball dashboards. Set false to run Prometheus and Alertmanager alone |
monitoring.grafana.adminSecretRef | object | No | - | Secret holding the Grafana admin password under key and the user name under admin-user. Must be in the monitoring namespace (namespace, if set, must name it) |
monitoring.grafana.valuesOverride | string | No | "" | YAML document of Grafana Helm chart values merged over the generated ones |
monitoring.alerting.webhookURL | string | No | "" | Alertmanager webhook receiver URL. Alerts fire into Alertmanager and stop there when unset, still visible in its UI |
monitoring.alerting.configOverride | string | No | "" | Replaces the generated Alertmanager configuration wholesale, for a site whose routing the fields above cannot express |
monitoring.alerting.valuesOverride | string | No | "" | YAML document of Alertmanager Helm chart values (resources, replicas, storage) merged over the generated ones. Routing belongs in configOverride |
monitoring.alerting.externalReceiver.enabled | boolean | No | false | Publish the hold receiver for an Alertmanager the operator does not deploy, and issue it a client certificate. Honored whether or not the bundled stack is enabled |
monitoring.alerting.externalReceiver.hostname | string | No | alertmanager-receiver.<domain> | Hostname the external Alertmanager uses for the receiver. The receiver’s certificate is issued for it; point its DNS at the fuzzball-alertmanager-receiver Service |
monitoring.alerting.externalReceiver.serviceType | string | No | LoadBalancer | Type of the fuzzball-alertmanager-receiver Service: LoadBalancer or NodePort |
monitoring.alerting.externalReceiver.annotations | map | No | {} | Annotations on the fuzzball-alertmanager-receiver Service, such as a cloud provider’s internal load balancer annotation |
Example – tune the default stack:
spec:
monitoring:
retentionDays: 30
storageSize: 100Gi
alerting:
webhookURL: https://alerts.example.com/hook
prometheus:
valuesOverride: |
server:
resources:
limits:
memory: 8Gi
grafana:
valuesOverride: |
ingress:
enabled: true
ingressClassName: kong
annotations:
# Refuse plain HTTP: Grafana's login page carries the admin password.
konghq.com/protocols: https
konghq.com/https-redirect-status-code: "308"
hosts:
- grafana.example.com
tls:
- hosts:
- grafana.example.com
secretName: grafana-tls
Keep the host, the tls block and the two Kong annotations together. Without a
host the route is a catch-all on the cluster’s Kong proxy, answering every
hostname nothing else claims; without the annotations Kong also serves the login
page over plain HTTP. grafana-tls is a certificate Secret in the monitoring
namespace – issue it with cert-manager or supply your own. Other ingress
controllers need their own equivalent of the Kong annotations.
Example – opt out:
spec:
monitoring:
enabled: false
Example – your own Alertmanager instead of the bundled stack:
spec:
monitoring:
enabled: false
alerting:
externalReceiver:
enabled: true
hostname: alertmanager-receiver.example.com
The hold receiver accepts only mutual TLS, so it is published through a fuzzball-alertmanager-receiver
Service on port 9443 rather than through the cluster ingress, which terminates TLS. The operator
issues the external Alertmanager’s client certificate into the fuzzball-external-alertmanager-client-tls
Secret (tls.crt, tls.key), from a client CA of the Fuzzball namespace’s own (a namespaced
Issuer, never a ClusterIssuer), and publishes the CA that signs the receiver’s certificate in the
fuzzball-alertmanager-receiver-ca ConfigMap (ca.crt), both in the Fuzzball namespace. The client
certificate is valid for a year and renewed 30 days before it expires; copy the renewed Secret to
the Alertmanager. See
Using Your Own Alertmanager.
Each valuesOverride is merged the way Helm layers a values file: maps merge key by key, and
anything else, lists included, replaces the generated value. The keys are those of the component’s
own chart, so server.resources rather than prometheus.server.resources. These keys are refused,
because another field owns them: alertmanager in the Prometheus override (use
monitoring.alerting.valuesOverride), and config and enabled in the Alertmanager override (use
monitoring.alerting.configOverride and monitoring.enabled) – Alertmanager is what places
holds, so it cannot be turned off on its own. A malformed override is refused as a configuration
error on the resource before anything is deployed. An override can also undo what the operator generated – for
example, replacing serverFiles."prometheus.yml".scrape_configs replaces Fuzzball’s own scrape
jobs. To add jobs, use extraScrapeConfigs, which the chart takes as a string holding the job
list, so it is a block scalar inside the override:
spec:
monitoring:
prometheus:
valuesOverride: |
extraScrapeConfigs: |
- job_name: my-exporter
static_configs:
- targets: ["exporter.example.com:9200"]
An override may not set a value to null: Helm would read that as “remove this key”, but it does
not reach Helm, so it is refused rather than silently ignored. Set the value you want instead.
The stack installs no CRDs, no DaemonSet, and no host ports, so it can run beside a site’s existing kube-prometheus-stack. Grafana is not exposed outside the cluster unless you enable its ingress throughmonitoring.grafana.valuesOverride, as in the example above. See Node Health for the dashboards, alert rules, and metric series shipped with the stack.
Fuzzball injects a workflow-scoped API token (FB_TOKEN) and API address into every workflow job
and service container so running workflows can call the Fuzzball API — see
In-Workflow API Access. The
top-level disableWorkflowTokens field turns this off cluster-wide: no tokens are injected into
new containers, the accompanying trust-store injection (the read-only node trust-store mount and
SSL_CERT_DIR) is skipped, and the API stops accepting workflow tokens, including any already
held by running containers.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
disableWorkflowTokens | boolean | No | false | Stop injecting workflow-scoped API tokens and the accompanying trust-store mount into workflow containers, and reject workflow tokens at the API |
Example:
spec:
disableWorkflowTokens: true
A workflow can build its own container images in a top-level images: section – see
Building Images in a Workflow. A build
runs the definition’s post script as root inside a confined container on a compute node. The
feature is on by default; the top-level enableWorkflowImageBuilds field turns it off, and a
workflow carrying an images: section is then rejected at submission and validation. Turn it off
where workflow authors are not trusted to choose code that runs as root in that container.
A FuzzballOrchestrate resource created or edited while Fuzzball v4.3 or a v4.4 pre-release was
installed, when this field defaulted to false, has false stored and keeps it after an upgrade;
set enableWorkflowImageBuilds: true to allow image builds on such a cluster. A resource not
written since before v4.3 has nothing stored and allows them, so set false explicitly to keep
them off.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
enableWorkflowImageBuilds | boolean | No | true | Allow workflow definitions to build container images on compute nodes through an images: section |
Example:
spec:
enableWorkflowImageBuilds: false
A Federate and its member clusters call each other with cluster tokens. Each cluster binds every
cluster token it sends to the cluster it is sending to, so a cluster that receives one cannot replay
it against another. The top-level requireClusterTokenAudience field makes this cluster refuse a
cluster token that is not bound to it. A token bound to a different cluster is always refused.
The field is off by default because a cluster running an earlier release sends unbound tokens, and
turning it on would lock that cluster out. While it is off, this cluster accepts unbound tokens and
counts them in the fuzzball_authn_unbound_cluster_tokens_total metric, labeled by the cluster
that sent them. Turn it on once that count has stopped increasing for every sender. Refused
cluster tokens are counted in fuzzball_authn_refused_cluster_tokens_total, labeled by sender and
reason (bound_elsewhere or unbound), and logged. See
Binding Cluster Tokens
for the rollout. The field is the same on FuzzballOrchestrate and FuzzballFederate resources.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
requireClusterTokenAudience | boolean | No | false | Refuse cluster tokens that were not minted for this cluster |
Example:
spec:
requireClusterTokenAudience: true
Compute nodes send a periodic heartbeat to the control plane, which is how a live
node is told from a silent one — see
Node Health Monitoring.
The nodeHealth section controls how long a node may stay silent before it is
recorded as such. Omit the section to accept the default.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
nodeHealth.heartbeatMissThreshold | integer | No | 3 | Consecutive health reports a compute node may miss before its status becomes unknown. Measured against each node’s own reporting interval, so it is a count rather than a duration. Minimum 1, maximum 100 |
Example:
spec:
nodeHealth:
heartbeatMissThreshold: 5
The fuzzball section controls all Fuzzball services and their configurations.
| Parameter | Type | Description |
|---|---|---|
version | string | Fuzzball version (defaults to operator version) |
cluster.name | string | Cluster name. When unset, defaults to the deployment’s domain (e.g. example.com); Federate clusters are prefixed with federate. (e.g. federate.example.com). |
cluster.kind | string | Cluster kind/type |
The substrate section controls how substrate nodes connect and operate. The
scheduler admin guide covers sizing guidance for nodesBucketMaxBytes, the
failure mode when the bucket fills, and the Prometheus metrics that track bucket
usage — see
Node Report Storage.
| Parameter | Type | Description |
|---|---|---|
substrate.nfs.destination | string | Mount point on substrate nodes, including nodes started by a cloud provisioner |
substrate.nfs.server | string | NFS server IP or hostname |
substrate.nfs.path | string | NFS export path |
substrate.secureRegistries | []string | HTTPS registries, each as host:port, whose certificate substrate nodes trust. See Secure Registries. |
substrate.imageProxy | string | HTTP(S) proxy for image pulling |
substrate.imageNoProxy | []string | Hosts to exclude from proxy |
substrate.mtls.* | object | FuzzballOrchestrate only. Custom substrate mTLS CA: inline.secretName (a kubernetes.io/tls Secret in tls.certManager.clusterResourceNamespace, default cert-manager) or, deprecated, base64-encoded inline.cert and inline.key |
substrate.nodesBucketMaxBytes | integer | Maximum size, in bytes, of the substrate-nodes JetStream key-value bucket. Minimum 67108864 (64 MiB), maximum 2147483648 (2 GiB). Omit the field to use the orchestrate service default of 67108864 (64 MiB); do not set 0, which the minimum rejects at admission. |
Example:
spec:
fuzzball:
cluster:
name: my-fuzzball-cluster
kind: on-premises
substrate:
nfs:
destination: /fuzzball/shared
server: 10.0.0.10
path: /srv/fuzzball/shared
secureRegistries:
- registry.example.com:5000
imageProxy: http://proxy.example.com:3128
imageNoProxy:
- localhost
- 127.0.0.1
- .example.com
nodesBucketMaxBytes: 1073741824
When it reconciles, the operator connects to each entry and adds the last certificate in the
chain that the registry presents to the substrate configuration it generates, so substrate nodes
trust the registry with that certificate. Each entry must include the port, for example
registry.example.com:5000.
The list goes into two substrate configurations:
- The one the operator generates for the provisioner, whether or not
substrate.nfsis set. Nodes started by PBS and Slurm use it, andfuzzball cluster kubernetes generate-substrate-configcopies it for compute nodes that install their configuration locally. - When
substrate.nfsis set, the one on that share (substrate/substrate-config.yaml), which nodes load throughFB_POLL_CONFIG.
The operator pod connects to each registry itself, so the registry must be reachable from the
operator pod, not only from the substrate nodes. If an entry has no port, or the operator can’t connect to it,
the whole deployment update fails, not just that entry: the resource goes Degraded, the operator
logs while getting registry certificate for <entry>, and the operator retries until the entry
is fixed or removed.
If the registry presents only its own certificate, nodes trust only that certificate unless they already trust the CA that issued it, so pulls fail once the registry renews it. To avoid this, configure the registry to present its CA certificate after its own certificate.
Substrate reads this list only when the service starts. After you add or change an entry:
- On a compute node that installs its configuration locally, regenerate and install the configuration and restart Substrate as in Refresh the configuration after a change.
- On a node that loads the share through
FB_POLL_CONFIG, wait untilsubstrate/substrate-config.yamlon the share lists the entry, then restart Substrate.
Jobs running on a node when Substrate restarts fail, so restart it while the node is idle:
# systemctl restart fuzzball-substrate.serviceMost Fuzzball services support the following common parameters:
| Parameter | Type | Description |
|---|---|---|
replicas | integer | Number of pod replicas |
autoscaling.enabled | boolean | Enable horizontal pod autoscaling |
autoscaling.minReplicas | integer | Minimum replicas when autoscaling |
autoscaling.maxReplicas | integer | Maximum replicas when autoscaling |
autoscaling.targetCPUUtilization | integer | Target CPU percentage (default: 80) |
autoscaling.targetMemoryUtilization | integer | Target memory percentage (default: 80) |
resources.requests | map | Resource requests (cpu, memory) as key-value pairs |
resources.limits | map | Resource limits (cpu, memory) as key-value pairs |
serviceAccount.annotations | map | Service account annotations (for IRSA) |
nodeSelector | map[string]string | Node label selector for this component; overrides the top-level nodeSelector when set |
tolerations | []object | Override global tolerations for this service only |
affinity | object | Override global affinity for this service only |
Example for orchestrator service:
spec:
fuzzball:
orchestrator:
replicas: 2
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 5
targetCPUUtilization: 70
targetMemoryUtilization: 75
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator
The following configuration sections are available under spec.fuzzball:
Top-Level Configuration:
version- Fuzzball version (defaults to operator version)config- Global configuration (shared storage, etc.)cluster- Cluster name and metadatasubstrate- Substrate node configuration (NFS, registries, proxies, mTLS)
Active Services:
agent- Agent service (workflow execution)audit- Audit logging serviceauth- Authentication service (SpiceDB)billing- Billing and marketplace integrationclusterAdmin- Admin UI and cluster setupjetstream- NATS Jetstream message brokerobjectCache- Object cache service (shared blob storage for workflows)openapi- OpenAPI documentation serviceorchestrator- Orchestrator servicestorage- Storage servicesubstrateBridge- Substrate bridge service (DNS, logging, Kubernetes integration)ui- web UI serviceworkflow- Workflow serviceworkflowCatalog- Workflow catalog configuration
The following services have been deprecated as of version v3.0 and will be removed in a future version (use recommended replacements):
account- Deprecated (useagent)dns- Deprecated (usesubstrateBridge)kube- Deprecated (usesubstrateBridge)log- Deprecated (usesubstrateBridge)organization- Deprecated (useagent)provision- Deprecated (useorchestrator)schedule- Deprecated (useorchestrator)secret- Deprecated (useagent)user- Deprecated (useagent)workflowEngine- Deprecated (usejetstream)
NATS Jetstream has specific configuration requirements:
| Parameter | Type | Default | Description |
|---|---|---|---|
replicas | integer | 3 | Number of Jetstream replicas, at least 1 (recommended: 3). 1 runs a single standalone server with no clustering or redundancy, for new deployments only: a standalone server cannot recover streams written by a clustered deployment, so scaling an existing deployment down to 1 leaves Jetstream unready. |
storage.class | string | - | StorageClass for Jetstream PVCs |
storage.size | string | 10Gi | Storage size per replica |
externalService.type | string | ClusterIP | External service type (ClusterIP, NodePort, LoadBalancer). Use LoadBalancer when Substrate hosts can’t reach the cluster nodes’ IP addresses |
externalService.nodePort | integer | - | Static node port when externalService.type is NodePort |
externalService.annotations | map | - | Annotations for the external Service, for example to choose a load balancer address pool |
Example:
spec:
fuzzball:
jetstream:
replicas: 3
storage:
class: longhorn
size: 20Gi
externalService:
type: NodePort
The operator does not spread JetStream replicas across nodes or protect them from node drains by
default. See Spreading JetStream Replicas Across Nodes
and Protecting JetStream Quorum During Node Drains
for the affinity and PodDisruptionBudget configuration a production deployment should add.
This storage is necessary for internal services. Admins can configure storage for computational jobs later.
| Parameter | Type | Description |
|---|---|---|
storage.storage.class | string | StorageClass for storage service |
Example:
spec:
fuzzball:
storage:
storage:
class: longhorn
| Parameter | Type | Description |
|---|---|---|
audit.storage.class | string | StorageClass for audit logs |
Example:
spec:
fuzzball:
audit:
storage:
class: longhorn
| Parameter | Type | Description |
|---|---|---|
substrateBridge.log.storage.class | string | StorageClass for substrate logs |
substrateBridge.dns.externalService.type | string | DNS service type (ClusterIP, NodePort, LoadBalancer) |
substrateBridge.dns.externalService.annotations | map | Annotations for the DNS Service, for example to choose a load balancer address pool |
NodePort is recommended for local deployments. When Substrate hosts can’t reach the cluster nodes' IP addresses, set both this andjetstream.externalService.typeto LoadBalancer. See Compute nodes that can’t reach the Server nodes.
Example:
spec:
fuzzball:
substrateBridge:
log:
storage:
class: longhorn
dns:
externalService:
type: NodePort
| Parameter | Type | Default | Description |
|---|---|---|---|
objectCache.storage.class | string | - | StorageClass for the object cache PVC |
objectCache.storage.size | string | 50Gi | Storage size for the object cache PVC |
objectCache.storage.path | string | /storage/objectcache/ | Mount path for object cache blob storage |
objectCache.defaultTTL | string | 7d | Default time-to-live for cached objects (e.g., 7d, 24h) |
objectCache.diskPressureThreshold | integer | 90 | Disk usage percentage (0-100) that triggers accelerated eviction |
objectCache.diskCriticalThreshold | integer | 95 | Disk usage percentage (0-100) that rejects new uploads |
objectCache.storageClassName,objectCache.storageSize, andobjectCache.storagePathare deprecated in favor of the nestedobjectCache.storage.class,objectCache.storage.size, andobjectCache.storage.pathfields above, and will be removed in a future release.
Example:
spec:
fuzzball:
objectCache:
storage:
class: longhorn
size: 100Gi
path: /storage/objectcache/
defaultTTL: 24h
diskPressureThreshold: 85
diskCriticalThreshold: 95
The orchestrator provisioner enables multi-cloud and HPC integration:
| Parameter | Type | Description |
|---|---|---|
enabled | boolean | Enable provisioner |
substrateComputeDirectory | string | Working directory for substrate operations |
aws | object | AWS provisioner configuration |
azure | object | Azure provisioner configuration |
gcp | object | GCP provisioner configuration |
coreweave | object | CoreWeave provisioner configuration |
slurm | object | Slurm provisioner configuration |
pbs | object | PBS provisioner configuration |
Within a singleFuzzballOrchestrateresource, at most one ofaws,azure, orgcpmay be enabled. The default storage provisioner Fuzzball creates at install time is derived from the enabled cloud, so the choice must be unambiguous. Enabling more than one causes the operator to fail its reconcile with a configuration error. Federating multiple clouds is still supported: deploy a separateFuzzballOrchestrateper cloud and join them under aFuzzballFederate.
The GCP provisioner enables dynamic provisioning of compute instances on Google Cloud Platform.
| Parameter | Type | Description |
|---|---|---|
gcp.enabled | boolean | Enable GCP provisioner |
gcp.project | string | GCP project ID where substrate instances will be created |
gcp.region | string | GCP region for substrate instances |
gcp.zone | string | GCP zone for substrate instances |
gcp.network | string | VPC network name |
gcp.subnet | string | VPC subnet name within the network |
gcp.serviceAccountEmail | string | Service account email for substrate instances |
gcp.substrateImageProject | string | GCP project hosting substrate VM images (optional; defaults to CIQ public image project when empty) |
gcp.sshEnabled | boolean | Enable SSH access to substrate instances |
gcp.sshKeyPairName | string | SSH key pair name |
gcp.sshPrivateKeyPem | string | SSH private key in PEM format |
gcp.depotUser | string | CIQ Depot username for substrate nodes |
gcp.depotAccessToken | string | CIQ Depot access token for substrate nodes |
gcp.cloudInitScripts | []string | Custom cloud-init scripts to run on instances |
gcp.sharedFs | map | Shared filesystem configuration (uri, type, opts), required when the GCP provisioner is enabled. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. All three fields must be set and may use only letters, digits and =,_./:@- |
Example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
gcp:
enabled: true
project: my-gcp-project
region: us-central1
zone: us-central1-a
network: default
subnet: default
serviceAccountEmail: fuzzball-substrate@my-project.iam.gserviceaccount.com
substrateImageProject: my-mirrored-images-project
sharedFs:
uri: "10.0.0.2:/imagecache"
type: nfs
opts: "nfsvers=3,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
When to set substrateImageProject:
By default, the GCP provisioner pulls substrate VM images from the CIQ public image project. Set
substrateImageProject when substrate images have been mirrored to a customer-owned GCP project.
This is required in the following scenarios:
- Air-gapped environments that cannot access the CIQ public image project
- Deployments where organizational policies prohibit external image references
- Environments requiring custom image approval workflows
When substrateImageProject is empty or omitted, the provisioner uses the default CIQ public image
project.
| Parameter | Type | Description |
|---|---|---|
aws.enabled | boolean | Enable AWS provisioner |
aws.region | string | AWS region |
aws.subnetIDs | []string | VPC subnet IDs |
aws.securityGroupIDs | []string | Security group IDs |
aws.instanceProfileARN | string | IAM instance profile ARN |
aws.usePublicIP | boolean | Assign public IPs |
aws.sshEnabled | boolean | Enable SSH access |
aws.sshKeyPairName | string | EC2 key pair name |
aws.sshPrivateKeyPem | string | SSH private key |
aws.depotUser | string | CIQ Depot username for substrate nodes |
aws.depotAccessToken | string | CIQ Depot access token for substrate nodes |
aws.cloudInitScripts | []string | Custom cloud-init scripts to run on instances |
aws.sharedFs | map | Shared filesystem configuration. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. Its uri may use only letters, digits and =,_./:@-. Without uri, provisioned nodes cache images on local disk |
Example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
aws:
enabled: true
region: us-east-1
subnetIDs:
- subnet-0123456789abcdef0
securityGroupIDs:
- sg-0123456789abcdef0
instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate
usePublicIP: false
The Azure provisioner enables dynamic provisioning of compute instances on Microsoft Azure.
| Parameter | Type | Description |
|---|---|---|
azure.enabled | boolean | Enable Azure provisioner |
azure.subscriptionID | string | Azure subscription ID |
azure.tenantID | string | Azure tenant ID |
azure.location | string | Azure region (location) for provisioned resources |
azure.resourceGroupName | string | Resource group for Azure resources |
azure.subnetID | string | Subnet ID for substrate VMs |
azure.networkSecurityGroupID | string | Network security group ID for substrate VMs |
azure.usePublicIP | boolean | Assign public IPs |
azure.managedIdentityClientID | string | User-assigned managed identity client ID for provisioned VMs |
azure.storageAccountName | string | Azure Files storage account name backing the default v4 storage provisioner (workflowio share) |
azure.sshEnabled | boolean | Enable SSH access |
azure.sshPublicKey | string | SSH public key |
azure.sshPrivateKeyPem | string | SSH private key in PEM format |
azure.substrateImageBaseID | string | Azure Managed Image resource ID for the base substrate image (overrides the marketplace image when set) |
azure.substrateImageNvidiaID | string | Azure Managed Image resource ID for the NVIDIA/GPU substrate image (overrides the marketplace image when set) |
azure.depotUser | string | CIQ Depot username for substrate nodes |
azure.depotAccessToken | string | CIQ Depot access token for substrate nodes |
azure.cloudInitScripts | []string | Custom cloud-init scripts to run on instances |
azure.sharedFs | map | Shared filesystem configuration. Provisioned nodes mount it at substrate.nfs.destination, or at /mnt/fuzzball-sharedfs when substrate.nfs is unset. uri, type and opts may use only letters, digits and =,_./:@-, and if uri is set, type and opts must be set too. Without uri, provisioned nodes cache images on local disk |
Example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
azure:
enabled: true
subscriptionID: 00000000-0000-0000-0000-000000000000
tenantID: 00000000-0000-0000-0000-000000000000
location: eastus
resourceGroupName: fuzzball-rg
subnetID: /subscriptions/.../resourceGroups/fuzzball-rg/providers/Microsoft.Network/virtualNetworks/fuzzball-vnet/subnets/substrate
networkSecurityGroupID: /subscriptions/.../resourceGroups/fuzzball-rg/providers/Microsoft.Network/networkSecurityGroups/fuzzball-nsg
managedIdentityClientID: 00000000-0000-0000-0000-000000000000
storageAccountName: fuzzballstorage
usePublicIP: false
The CoreWeave provisioner enables dynamic provisioning of compute nodes on CoreWeave infrastructure.
When enabled, Fuzzball will automatically create and manage CoreWeave NodePool resources and
deploy substrate pods as DaemonSets on provisioned nodes.
CoreWeave deployments use two types of shared storage PVCs:
- Workflow Data PVC (
fuzzball-shared-storage): Mounted at/mnt/shared-storageon substrate nodes and exposed to workflows via the HostPath CSI driver - Image Cache PVC (
fuzzball-sharedfs): Mounted at/mnt/fuzzball-sharedfson substrate nodes for internal container image caching (not exposed to workflows)
CoreWeave’sshared-vaststorage requires Native Protocol Limit view policy. Ensure this is configured before deployment.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
coreweave.enabled | boolean | Yes | - | Enable CoreWeave provisioner |
coreweave.storage.class | string | No | - | StorageClass for CoreWeave shared storage PVCs (typically shared-vast) |
coreweave.storage.size | string | No | - | Size of shared storage per substrate node |
coreweave.storage.accessMode | string | No | ReadWriteOnce | Access mode for shared storage PVC (use ReadWriteMany for multi-node access) |
coreweave.storage.annotations | map | No | {} | Additional annotations for storage resources |
Example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
coreweave:
enabled: true
storage:
class: shared-vast
size: 100Gi
accessMode: ReadWriteMany
| Workload Type | Recommended Size | Rationale |
|---|---|---|
| Light workflows | 50Gi | Minimal data processing |
| Standard workflows | 100Gi | Typical data processing needs |
| Heavy workflows | 250Gi+ | Large datasets, intermediate files |
| Data-intensive | 500Gi+ | Big data processing, ML training |
The shared PVC (fuzzball-sharedfs) is mounted at /mnt/fuzzball-sharedfs on substrate nodes for
internal container image caching across compute nodes. This improves workflow startup times by
avoiding repeated image downloads. This storage is not exposed to workflows.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
fuzzball.config.sharedPVC.accessMode | string | Yes | - | Volume access mode (must be ReadWriteMany) |
fuzzball.config.sharedPVC.class | string | Yes | - | Storage class name (must be shared-vast) |
fuzzball.config.sharedPVC.size | string | Yes | - | Total shared cache size |
Example:
spec:
fuzzball:
config:
sharedPVC:
accessMode: ReadWriteMany
class: shared-vast
size: 10Gi
| Environment Type | Recommended Size | Rationale |
|---|---|---|
| Testing/Development | 10Gi | Few container images |
| Small Production | 25Gi | Limited image variety |
| Large Production | 50Gi+ | Many different images |
By default a Fuzzball cluster only trusts JWTs issued by its own Keycloak. Setting
fuzzball.config.additionalTrustedIssuers adds extra JWT issuer URLs that the cluster’s services
will also accept. The primary use case is federation, where you want one cluster to honor tokens
minted by another cluster’s Keycloak (see
Trusting Tokens from Another Keycloak).
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
fuzzball.config.additionalTrustedIssuers | string array | No | [] | Extra JWT issuer URLs to trust alongside this cluster’s own Keycloak. Each entry must be an absolute http(s) URL. |
Example:
spec:
fuzzball:
config:
additionalTrustedIssuers:
- https://auth.federate.example.com/
The trusted-issuer list is keyed on the issuer’sscheme://host; the path and query string are ignored. An additional issuer that shares a scheme and host with the cluster’s own Keycloak is discarded so it cannot override the operator-generated entry.
See Slurm Integration Documentation for detailed configuration.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
slurm.enabled | boolean | Yes | - | Enable Slurm provisioner |
slurm.sshHost | string | Yes | - | SSH host for remote Slurm instance |
slurm.sshPort | integer | No | 22 | SSH host port for remote Slurm instance |
slurm.username | string | Yes | - | SSH login username for remote Slurm instance |
slurm.password | string | No | - | SSH login password for remote Slurm instance |
slurm.sshHostPublicKey | string | No | - | SSH host public key for remote Slurm instance |
slurm.sshPrivateKeyPem | string | No | - | SSH private key PEM for remote Slurm instance |
slurm.sshPrivateKeyPassPhrase | string | No | - | Passphrase for encrypted SSH private key |
slurm.binaryPath | string | No | - | Custom path to Slurm binaries (if not in $PATH) |
slurm.connectionTimeout | integer | No | 30 | SSH connection timeout in seconds |
slurm.sudoPath | string | No | - | Path to sudo binary on compute nodes |
slurm.options | map | No | {} | Additional Slurm sbatch options |
slurm.skipHostKeyVerification | boolean | No | false | Skip SSH host key verification (not recommended) |
Basic example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
slurm:
enabled: true
sshHost: slurm-head.example.com
sshPort: 22
username: fuzzball-service
sshPrivateKeyPem: |
<full private key in PEM format>
sshHostPublicKey: "slurm-head.example.com ecdsa-sha2-nistp256 AAAAE2..."
See PBS Integration Documentation for detailed configuration.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
pbs.enabled | boolean | Yes | - | Enable PBS provisioner |
pbs.sshHost | string | Yes | - | SSH host for remote PBS instance |
pbs.sshPort | integer | No | 22 | SSH host port for remote PBS instance |
pbs.username | string | Yes | - | SSH login username for remote PBS instance |
pbs.password | string | No | - | SSH login password for remote PBS instance |
pbs.sshHostPublicKey | string | No | - | SSH host public key for remote PBS instance |
pbs.sshPrivateKeyPem | string | No | - | SSH private key PEM for remote PBS instance |
pbs.sshPrivateKeyPassPhrase | string | No | - | Passphrase for encrypted SSH private key |
pbs.binaryPath | string | No | - | Custom path to PBS binaries (if not in $PATH) |
pbs.validateSubstrate | boolean | No | false | Validate substrate before use |
pbs.defaultQueue | string | No | - | Default PBS queue name |
pbs.pbsServer | string | No | - | PBS server hostname |
pbs.connectionTimeout | integer | No | 30 | SSH connection timeout in seconds |
pbs.options | map | No | {} | Additional PBS qsub options |
pbs.sudoPath | string | No | - | Path to sudo binary on compute nodes |
pbs.skipHostKeyVerification | boolean | No | false | Skip SSH host key verification (not recommended) |
Basic example:
spec:
fuzzball:
orchestrator:
provisioner:
enabled: true
pbs:
enabled: true
sshHost: pbs-head.example.com
sshPort: 22
username: fuzzball-service
password: secure-password
Support for CoreWeave LOTA object storage is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using LOTA with Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.
CoreWeave’s LOTA provides S3-compatible object storage for workflow data ingress and egress. Configure LOTA credentials to enable workflows to read from and write to LOTA buckets.
| Parameter | Type | Required | Description |
|---|---|---|---|
type | string | Yes | Must be s3 for LOTA |
secret.access-key-id | string | Yes | LOTA access key ID |
secret.secret-access-key | string | Yes | LOTA secret access key |
secret.endpoint | string | Yes | LOTA endpoint URL (https://cwlota.com) |
secret.region | string | Yes | LOTA bucket region (e.g., ord1, lga1) |
Create a YAML file with your LOTA credentials:
# lota-credentials.yaml
type: s3
secret:
access-key-id: AKIAIOSFODNN7EXAMPLE
secret-access-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
endpoint: https://cwlota.com
region: us-east-02a
Replace placeholder values with your actual LOTA credentials from CoreWeave.
$ fuzzball secret create -s user --from-file \
lota-credentials lota-credentials.yamlThe -s user flag makes the secret available to all users in your organization.
Reference the LOTA credentials in workflow volume definitions:
volumes:
shared-storage:
reference: volume://user/fuzzball-shared-vast/shared-storage
ingress:
- source:
uri: "s3://my-bucket/input/data.txt"
secret: secret://user/lota-credentials
destination:
uri: "file://data.txt"
egress:
- source:
uri: "file://results.tar.gz"
destination:
uri: "s3://my-bucket/output/results.tar.gz"
secret: secret://user/lota-credentials
Replace my-bucket with your LOTA bucket name.
For testing and development on a local Kubernetes cluster. This deploys the Fuzzball control plane.
To run workflows, you’ll need to add shared storage (fuzzball.substrate.nfs or
fuzzball.config.sharedPVC) and configure compute nodes.
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
image:
username: depot-user
password: depot-token
exclusive: false
ingress:
create:
domain: localhost.nip.io
proxy:
type: NodePort
http:
nodePort: 30080
tls:
nodePort: 30443
database:
create:
storage:
class: local-path
keycloak:
create:
ownerEmail: admin@localhost
createDatabase: true
# The 3Gi production default is more than a laptop-sized cluster can spare.
resources:
requests:
memory: "1Gi"
limits:
memory: "1Gi"
tls:
certManager:
create: {}
trustManager:
create: {}
fuzzball:
jetstream:
replicas: 1
externalService:
type: NodePort
storage:
class: local-path
substrateBridge: {}
log:
storage:
class: "local-path"
dns:
externalService:
type: NodePort
For production deployments with high availability:
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
image:
username: depot-user
password: depot-token
exclusive: true
ingress:
create:
domain: fuzzball.company.com
proxy:
type: LoadBalancer
annotations:
metallb.io/loadBalancerIPs: 10.0.100.50
database:
create:
storage:
class: longhorn
size: 500Gi
keycloak:
create:
ownerEmail: admin@company.com
replicas: 2
createDatabase: true
tls:
certManager:
create: {}
trustManager:
create: {}
ingressIssuer:
create:
letsEncrypt:
email: admin@company.com
issuer: letsencrypt-prod
solvers:
- dns01:
route53:
region: us-east-1
hostedZoneID: Z1234567890ABC
fuzzball:
cluster:
name: production-cluster
substrate:
nfs:
server: nfs.company.com
path: /fuzzball/shared
destination: /fuzzball/shared
orchestrator:
replicas: 3
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
agent:
replicas: 5
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 20
jetstream:
replicas: 3
storage:
class: longhorn
size: 50Gi
Using external database and authentication:
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
image:
username: depot-user
password: depot-token
ingress:
external:
domain: fuzzball.cloud.com
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
database:
external:
host: postgres.abc123.us-east-1.rds.amazonaws.com
rdsSecretId: arn:aws:secretsmanager:us-east-1:123456789012:secret:fuzzball-db
credentials:
user: "" # Leave empty when using rdsSecretId
password: "" # Leave empty when using rdsSecretId
sslMode: verify-full
certificate:
caCertURL: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
keycloak:
external:
url: https://keycloak.cloud.com
realmId: 550e8400-e29b-41d4-a716-446655440000
realmName: FuzzballProd
username: admin
password: keycloak-password
ownerEmail: admin@cloud.com
fuzzball:
config:
sharedPVC:
accessMode: ReadWriteMany
class: efs-sc
size: 100Gi
orchestrator:
provisioner:
enabled: true
aws:
enabled: true
region: us-east-1
subnetIDs:
- subnet-0123456789abcdef0
securityGroupIDs:
- sg-0123456789abcdef0
instanceProfileARN: arn:aws:iam::123456789012:instance-profile/FuzzballSubstrate
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/fuzzball-orchestrator
Check deployment status:
$ kubectl get fuzzballorchestrate
NAME STATUS AGE
fuzzball-orchestrate Running 10mView the current configuration:
$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o yamlView status and events:
$ kubectl describe fuzzballorchestrate fuzzball-orchestrateEdit the configuration in-place (changes are applied immediately):
$ kubectl edit fuzzballorchestrate fuzzball-orchestrateApply configuration from a file:
$ kubectl apply -f fuzzball.yamlThe operator reports deploy and destroy progress directly on the custom resource. Use
kubectl describe to see real-time status during any operation:
$ kubectl describe fuzzballorchestrate fuzzball-orchestrateThe operator uses Pulumi internally to manage the underlying Kubernetes and cloud resources. As a result, events and.status.failedResourcesentries identify resources by Pulumi URN — a stable identifier of the formurn:pulumi:<stack>::<provider>::<type>::<name>. You can treat a URN as an opaque label unique to the specific resource Pulumi is tracking; you only need to recognize that the trailing<name>segment is human-readable (e.g.,fuzzball-coldboot,fuzzball-load-default-catalog).
Two lifecycle conditions, Progressing and Ready, carry deploy and destroy state:
| Condition | Status | Reason | Meaning |
|---|---|---|---|
Progressing | True | Deploying | Deploy in progress |
Ready | False | Deploying | Deploy in progress |
Progressing | False | DeployComplete | Deploy finished successfully |
Ready | True | Succeeded | Deploy finished successfully |
Ready | False | RolloutIncomplete | Deploy finished, but a Deployment is still rolling out |
Ready | False | RolloutStalled | Deploy finished, but a Deployment will not finish rolling out on its own |
Progressing | False | DeployFailed | Deploy failed (see Message for error) |
Ready | False | DeployFailed | Deploy failed |
Progressing | True | Destroying | Destroy in progress |
Ready | False | Terminating | Destroy in progress |
Progressing | False | DestroyComplete | Destroy finished — resource pending garbage collection |
Progressing | False | DestroyFailed | Destroy failed (see Message for error) |
Ready | False | DestroyFailed | Destroy failed |
Each condition also carries a Message field summarizing the state in human-readable form:
"Deploy in progress" or "Destroy in progress" while the operation is running,
"Deploy completed" or "Destroy completed" on success, or the underlying error string
when it failed.
On a FuzzballOrchestrate, Ready turns True only once every Deployment in the fuzzball
namespace labeled app.kubernetes.io/managed-by: pulumi has rolled out: all replicas updated and
available, and no old pods left. Until then it stays False with reason RolloutIncomplete, and
its Message names each Deployment still rolling out. Components installed from Helm charts, such
as Keycloak, and StatefulSets are not checked.
A pod that becomes unavailable, or is not replaced, after its Deployment finished rolling out
does not hold Ready, unless that Deployment has returned to a pod template it ran before (a
downgrade, or a setting changed and then changed back); then it does, until the Deployment’s next
rollout to a pod template it has not run before.
The operator checks the Deployments again every 15 seconds, without redeploying, and sets Ready
and Running once they have rolled out. If a Deployment exceeds its progress deadline or is
paused, the reason becomes RolloutStalled and Degraded is set, so a rollout blocked by, for
example, a volume that is still attached to the old pod does not report as ready. The operator then
retries the full deployment every five minutes.
After a successful destroy,Readyis not updated — the operator removes the finalizer and Kubernetes garbage-collects the resource. OnlyProgressingtransitions toFalse / DestroyCompletebefore the resource disappears.
A separate condition, Misconfigured, reports a problem with the spec rather than with a
deploy in progress. It is set independently of the lifecycle pair and stays on the resource
until the spec is corrected, so it survives the deploys that run in spite of it:
| Condition | Status | Reason | Meaning |
|---|---|---|---|
Misconfigured | True | MissingOwnerEmail | ownerEmail is empty (see Owner email is required) |
The condition is removed once the spec is fixed. It is recorded on a best-effort basis, so treat its presence as authoritative and its absence as “nothing reported” rather than as a clean bill of health.
Alongside these, the operator records where its state machine currently is with a
one-condition-per-state set. Exactly one of these is True at a time:
| Condition | Meaning |
|---|---|
Creating | First deploy of the resource is running |
Updating | A deploy is running against a resource that already exists |
Running | The resource is deployed and reconciled |
Degraded | The last operation failed |
Terminating | The resource is being destroyed |
Degraded carries a fixed message, An error occurred - See Operator logs for more information.,
so the operator logs are where the underlying error is found. This is the condition to look for
first when kubectl describe shows a deploy that is not progressing.
While a deploy or destroy is running, .status.progress is populated with real-time counters
and the currently active resources:
Status:
Progress:
Phase: Deploying
Completed: 153
In Flight: 2
Failed: 0
Elapsed Seconds: 310
Start Time: 2026-04-23T01:00:22Z
Current Resources:
Kind: Command Op: create Elapsed Seconds: 0
Urn: urn:pulumi:…::fuzzball-coldboot
Kind: Job Op: create Elapsed Seconds: 0
Urn: urn:pulumi:…::fuzzball-load-default-catalog
Phase is either Deploying or Terminating. completed, inFlight, and failed count
individual infrastructure resource operations. Once the operation finishes, .status.progress
is cleared.
The operator emits events on the resource throughout the operation lifecycle. View them at the
bottom of kubectl describe output or with:
$ kubectl get events --field-selector involvedObject.name=fuzzball-orchestrate --sort-by='.lastTimestamp'Deploy events:
| Event | Type | Description |
|---|---|---|
DeployStarted | Normal | Deploy has begun |
DeployCompleted | Normal | Deploy succeeded — includes elapsed time and resource count |
ResourceOpFailed | Warning | A single tracked resource’s create/update/replace/delete failed |
DeployFailed | Warning | Deploy failed — see .status.failedResources and operator logs |
RolloutStalled | Warning | Deploy succeeded, but a Deployment exceeded its progress deadline or is paused |
Destroy events:
| Event | Type | Description |
|---|---|---|
DestroyStarted | Normal | Destroy has begun |
DestroyingDependency | Normal | A named dependency stack (e.g., Fuzzball, Database) is being torn down |
DestroyedDependency | Normal | A dependency stack completed teardown |
DestroyCompleted | Normal | All resources destroyed — includes elapsed time and count |
ResourceOpFailed | Warning | A single tracked resource’s create/update/replace/delete failed |
DestroyFailed | Warning | Destroy failed |
ResourceOpFailed is emitted per affected resource (not once per operation), so it can fire
multiple times during a single Deploy or Destroy. Its message has the form <Kind> <op> failed: <URN> (for example, Job create failed: urn:pulumi:fuzzball::…::fuzzball-load-default-catalog). It
is the most precise signal for pinpointing which resource broke before the rolled-up DeployFailed
/ DestroyFailed event fires.
The same event patterns apply to both FuzzballOrchestrate and FuzzballFederate resources.
Example — successful Orchestrate deploy:
Events:
Normal DeployCompleted 0s Deploy completed in 2m11s; 93 resources applied
Example — 164-resource teardown:
Events:
Normal DestroyStarted 1m41s Starting Destroy for fuzzball-local-dev
Normal DestroyingDependency 1m41s Destroying Identity
Normal DestroyingDependency 1m37s Destroying Fuzzball
Normal DestroyingDependency 53s Destroying Namespace
Normal DestroyingDependency 48s Destroying Database
Normal DestroyingDependency 10s Destroying TLS
Normal DestroyedDependency 0s Destroyed Database
Normal DestroyedDependency 0s Destroyed Fuzzball
Normal DestroyedDependency 0s Destroyed Identity
Normal DestroyedDependency 0s Destroyed Namespace
Normal DestroyedDependency 0s Destroyed TLS
Normal DestroyCompleted 0s Destroy completed in 1m44s; 164 resources destroyed
Watch operator logs during deployment:
$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f- Resource Scope:
FuzzballOrchestrateis a cluster-scoped resource (not namespace-scoped) - Short Names: Can use
fborfuzzas shortcuts in kubectl commands - Mutual Exclusivity:
- Database: Use either
createorexternal, not both - Ingress: Use either
createorexternal, not both - Keycloak: Use either
createorexternal, not both
- Database: Use either
- Autoscaling: When enabled,
replicasis ignored in favor ofminReplicas/maxReplicas - Storage Classes: Ensure specified storage classes exist in your cluster
- UUIDs:
realmIdmust be a valid UUID v4 format - NFS Requirements: NFS server must be accessible from all nodes
- Scheduling Overrides: A per-component
nodeSelector,tolerations, oraffinityreplaces the global value for that component; the operator sets no default anti-affinity orPodDisruptionBudgetfor any component, including JetStream
When a deploy fails, the Ready and Progressing conditions are both set to False with
reason DeployFailed. The Message field on the condition contains the Pulumi diagnostic error:
Status:
Conditions:
Type: Progressing Status: False Reason: DeployFailed
Message: an unhandled error occurred: waiting for RPCs: marshaling properties:
awaiting input property "clientId": while getting fuzzball realm: ...
Type: Ready Status: False Reason: DeployFailed
Message: (same as above)
Events:
Warning DeployFailed 0s Deploy failed after 5m15s. See .status.failedResources and operator logs.
The Message field is capped at 256 characters; a trailing ... in the output indicates the
text was truncated. Consult the operator logs (shown further down) for the full error.
Check .status.failedResources for a structured list of every resource that failed, including
the Pulumi URN, the operation type (create, update, replace, or delete), the error
message, and how many times it was retried:
$ kubectl get fuzzballorchestrate fuzzball-orchestrate -o jsonpath='{.status.failedResources}' | jq .Each entry contains:
urn— Pulumi URN identifying the specific resource that failedop— operation that was attemptederror— the diagnostic error messageattemptCount— number of retries before giving up
.status.failedResources is retained on the resource until the next stack operation begins —
the operator overwrites the slice with a fresh (empty) one as soon as the next deploy or
destroy starts, so capture this output before retrying if you want a post-mortem record.
Check operator logs for additional context:
$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system --tail=100Check FuzzballOrchestrate status and events:
$ kubectl describe fuzzballorchestrate fuzzball-orchestrateCheck pod events:
$ kubectl get pods -n fuzzball
$ kubectl describe pod <pod-name> -n fuzzballCheck PVC status:
$ kubectl get pvc -AA pod that stays in Pending because of the nodeSelector, tolerations, or affinity settings
carries a FailedScheduling event explaining which rule excluded each node:
$ kubectl get events -n fuzzball --field-selector reason=FailedScheduling
$ kubectl describe pod <pod-name> -n fuzzball| Event message contains | Cause | Fix |
|---|---|---|
didn't match Pod's node affinity/selector | The nodeSelector or nodeAffinity names a label no node carries | Compare the rule against kubectl get nodes --show-labels and fix the label or the rule |
had untolerated taint | A node taint has no matching toleration | Add the toleration globally or to the affected component; see kubectl describe node <node> | grep Taints |
didn't match pod anti-affinity rules | A requiredDuringSchedulingIgnoredDuringExecution anti-affinity needs more eligible nodes than exist | Use a preferredDuringSchedulingIgnoredDuringExecution rule or add eligible nodes; do not scale an existing clustered JetStream deployment down to one replica |
volume node affinity conflict | The pod’s existing volume is on a node or in a zone the new rules exclude | Relax the scheduling rule to allow the existing volume, or back up and migrate the data to compatible storage using a supported recovery procedure; deleting the PVC can permanently destroy its data |
If only one component shows the taint error after you added a per-component override, remember
that the override replaced the global tolerations for that component; copy the global entries
into the override.
To see the scheduling fields the operator actually rendered, inspect the workload rather than the custom resource:
$ kubectl get statefulset fuzzball-jetstream -n fuzzball -o jsonpath='{.spec.template.spec.affinity}' | jq .
$ kubectl get deployment fuzzball-orchestrator -n fuzzball -o jsonpath='{.spec.template.spec.nodeSelector}' | jq .Substitute tolerations for the field name to check that setting.
Check ingress configuration:
$ kubectl get ingress -A
$ kubectl describe ingress <ingress-name> -n fuzzballVerify domain DNS resolution and load balancer IP assignment.