Storage Configuration
Fuzzball V4 uses storage provisioners to manage storage. A provisioner combines a built-in storage driver with access policies in a single configuration — there is no need to install separate storage drivers or create storage classes as in V3.
On cloud deployments, Fuzzball creates adefaultprovisioner for the initial organization using the storage backend appropriate to the deployment (EFS on AWS, Azure Files on Azure, Filestore on GCP). Checkfuzzball volume provisioner listbefore creating one manually. See The default provisioner for details.
For detailed reference on provisioner configuration, see the Volume Configuration and Management section. This page covers the basic setup needed to get storage working on a new cluster.
For on-prem deployments, you will likely want to create an NFS provisioner that manages volumes as subdirectories on an NFS export. In addition, you may also want to create hostpath provisioners for shared storage systems mounted on compute nodes.
First, create directories that will be served as NFS shares. On the server node (or the node serving your NFS shares):
# PRIVATE_SUBNET="" # populate this with the proper value for your environment (e.g. 10.0.0.0/20)# mkdir -pv /srv/fuzzball/storage
# echo "/srv/fuzzball/storage ${PRIVATE_SUBNET}(rw,sync,no_subtree_check,no_root_squash)" \
>>/etc/exports
# exportfs -a
# exportfsIt is not necessary to mount the NFS share on compute nodes. Fuzzball’s built-in NFS driver handles mounting automatically when volumes are used in workflows.
Create a provisioner definition file:
$ NFS_SERVER_IP="" # fill in the value of your NFS server IP address (e.g. 10.0.0.4)$ cat >nfs-provisioner.yaml<< EOF
default:
description: "Default NFS storage"
driver:
type: nfs
target: "${NFS_SERVER_IP}:/srv/fuzzball/storage"
version: 4
access: all
create: all
ephemeral: all
EOFApply the provisioner:
$ fuzzball volume provisioner add default -f nfs-provisioner.yaml
$ fuzzball volume provisioner list
ID | NAME | STATUS | DETAIL | CLUSTER
80e204ec-ee2f-3485-9282-33f77a9a3318 | default | Ready | | local-devMany of the CIQ templates in the Workflow Catalog expect a provisioner named
default. Keep this name unless you have a specific reason to change it.A compose deployment is the exception: its default provisioner is scoped to the
dockersegment and nameddefault-dockeraccordingly, so templates namingdefaultdo not resolve there. Either edit the template to namedefault-docker, or add a provisioner calleddefaultthat the deployment’s nodes can serve.
For AWS deployments, Fuzzball’s built-in EFS driver creates and manages volumes as EFS access points. On a fresh AWS deployment, Fuzzball typically bootstraps a default EFS provisioner automatically.
After deployment, check if a default provisioner was created:
$ fuzzball volume provisioner listIf a default provisioner exists and is in Ready status, storage is configured and you
can skip to Next Steps.
A
Readyprovisioner is not necessarily a usable one. Check that it also carries a create policy:$ fuzzball volume provisioner info default -o jsonIf the output has no
createPolicy, volume creation is denied for every group, including the cluster owner — policies are default-deny.Every default provisioner created by the operator on 4.x releases up to and including 4.2.1 is in this state. Volume creation has been gated on a create policy since 4.0.0. Nothing populated one for the default provisioner until 4.1.3-pre1, and from that release onward the operator still did not pass the cluster’s owner email to the orchestrator, so the seeding had no value to work from.
A default provisioner created by the current release grants create to every member of the organization. Upgrading does not change an existing provisioner, so administrator edits are preserved. If yours has no
createPolicy, grant create access once withfuzzball volume provisioner edit.
If no default provisioner exists, create one manually. You can use either BYO mode (provide an existing EFS filesystem ID) or self-provisioned mode (provide subnet IDs and let Fuzzball create the filesystem):
BYO mode — use an existing EFS filesystem:
$ cat >efs-provisioner.yaml<< 'EOF'
default:
description: "Default EFS storage"
driver:
type: efs
filesystemId: fs-XXXXXXXX
region: us-east-1
access: all
create: all
ephemeral: all
EOFSelf-provisioned mode — Fuzzball creates the EFS filesystem:
$ cat >efs-provisioner.yaml<< 'EOF'
default:
description: "Default EFS storage"
driver:
type: efs
region: us-east-1
subnetIds:
- subnet-XXXXXXXX
- subnet-YYYYYYYY
access: all
create: all
ephemeral: all
EOFReplace placeholder values (fs-XXXXXXXX,us-east-1, subnet IDs) with actual values from your AWS deployment. ProvidefilesystemIdorsubnetIds, not both.
Apply the provisioner:
$ fuzzball volume provisioner add default -f efs-provisioner.yaml
$ fuzzball volume provisioner listFor GCP deployments, Fuzzball uses the Filestore driver to connect to a Cloud Filestore instance. The NFS driver also works against Filestore or any other NFS-compatible storage, but the Filestore driver understands the instance’s endpoint format directly and is the better default.
After deployment, check if a default provisioner was created:
$ fuzzball volume provisioner listIf a default provisioner exists and is in Ready status, storage is configured.
If no default provisioner exists, create one pointing to your Filestore endpoint. The
target is the instance’s NFS endpoint, given as either host or host:/shareName;
omitting the share name uses workflowio:
$ cat >filestore-provisioner.yaml<< 'EOF'
default:
description: "Default Filestore storage"
driver:
type: filestore
target: "10.171.0.2:/workflowio"
access: all
create: all
ephemeral: all
EOFApply the provisioner:
$ fuzzball volume provisioner add default -f filestore-provisioner.yaml
$ fuzzball volume provisioner listTo point at NFS-compatible storage that is not Filestore, use the NFS driver instead
and give target in host:/path form.
Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change than other features. Contact CIQ as part of your deployment planning.
For Fuzzball on CoreWeave, use a hostpath provisioner backed by the shared-vast PVC
mounted on substrate nodes at /mnt/shared-storage.
$ cat >cw-provisioner.yaml<< 'EOF'
default:
description: "CoreWeave shared-vast storage"
driver:
type: hostpath
path: /mnt/shared-storage/volumes
access: all
create: all
ephemeral: all
EOFApply the provisioner:
$ fuzzball volume provisioner add default -f cw-provisioner.yaml
$ fuzzball volume provisioner listCoreWeave’sshared-vaststorage requires Native Protocol Limit view policy. Ensure this is configured before deployment. For additional CoreWeave storage options, see the CoreWeave Configuration Guide.
For Oracle Cloud Infrastructure (OCI) deployments, Fuzzball’s built-in OCI File Storage Service (FSS) driver creates and manages volumes as exports on OCI FSS. On a fresh OCI deployment, Fuzzball typically bootstraps a default OCI FSS provisioner automatically.
After deployment, check if a default provisioner was created:
$ fuzzball volume provisioner listIf a default provisioner exists and is in Ready status, storage is configured and you
can skip to Next Steps.
If no default provisioner exists, create one manually. You can use either BYO mode (provide an existing FSS file system and mount target) or self-provisioned mode (provide subnet IDs and let Fuzzball create the file system and mount target):
BYO mode — use an existing FSS file system and mount target:
$ cat >fss-provisioner.yaml<< 'EOF'
default:
description: "Default OCI File Storage"
driver:
type: oci_fss
region: us-ashburn-1
compartmentId: ocid1.compartment.oc1..aaaaaaaa
availabilityDomain: "Uocm:US-ASHBURN-AD-1"
filesystemId: ocid1.filesystem.oc1.iad.aaaaaaaa
mountTargetId: ocid1.mounttarget.oc1.iad.aaaaaaaa
access: all
create: all
ephemeral: all
EOFSelf-provisioned mode — Fuzzball creates the FSS file system and mount target:
$ cat >fss-provisioner.yaml<< 'EOF'
default:
description: "Default OCI File Storage"
driver:
type: oci_fss
region: us-ashburn-1
compartmentId: ocid1.compartment.oc1..aaaaaaaa
availabilityDomain: "Uocm:US-ASHBURN-AD-1"
subnetIds:
- ocid1.subnet.oc1.iad.aaaaaaaa
access: all
create: all
ephemeral: all
EOFReplace placeholder values (OCIDs, region, and availability domain) with actual values from your OCI deployment. For BYO mode provide bothfilesystemIdandmountTargetId; for self-provisioned mode omit them and providesubnetIds.
Apply the provisioner:
$ fuzzball volume provisioner add default -f fss-provisioner.yaml
$ fuzzball volume provisioner listFor the full set of OCI FSS fields and the BYO-vs-self-provisioned distinction, see Driver Types.
After creating a provisioner, verify it is working by scanning for any pre-existing volumes. The scan operation works consistently across all driver types (NFS, hostpath, EFS, OCI FSS):
# fuzzball volume provisioner scan default --dry-runThis shows any existing volumes on the backend without making changes. If volumes are
discovered, run the scan without --dry-run to import them:
# fuzzball volume provisioner scan defaultA scan looks only inside the provisioner’s own storage. For a hostpath or Filestore
provisioner with a subPath, the scan covers only that subdirectory, so data elsewhere on
the share is not found.
For EFS and OCI FSS provisioners, the scan queries the cloud provider API to discover volumes. Ensure that the credentials and permissions configured for the provisioner allow listing EFS access points and OCI FSS exports, respectively.
Every organization gets its own default provisioner when it is created, including
organizations created after startup through self-serve signup or synced down from a
federate. This needs no configuration beyond provisioners: true.
Each organization’s provisioner is scoped to that organization alone:
- On AWS, where the EFS driver can create storage itself, each organization after the one named in the cluster’s configuration gets its own filesystem, created automatically.
- On GCP and for hostpath, where the backing store is shared, each organization gets
its own subdirectory via
subPath. On a new installation, the organization named in the cluster’s configuration also gets its own subdirectory. After an upgrade, that organization’s existing provisioner stays at the root of the share. - When
azure-files,oci-fss, oroci-lustreis the default driver, the default provisioner serves the organization named in the cluster’s configuration. These drivers attach to one existing share, so each additional organization gets storage of its own by adding a provisioner backed by its own share or file system, withfuzzball volume provisioner add.
Every member of the organization can create volumes on its default provisioner. Deleting
a volume still requires being its creator, an account owner, or an organization owner.
Organization owners can narrow who may create volumes with
fuzzball volume provisioner edit.
Storage failures never block organization creation. If a provisioner cannot be created, the organization is still created and the failure is logged; add a provisioner manually afterwards withfuzzball volume provisioner add.
If your cluster includes nodes of different architectures (for example, both amd64 and arm64 compute nodes), be aware that the object cache stores container images with architecture-specific keys. This means:
- The same logical container image (e.g.,
docker://ubuntu:22.04) will occupy separate storage for each architecture present in your cluster. - Cache warming operations should be performed on representative nodes of each architecture to fully pre-populate the cache.
- Storage capacity planning should account for the possibility of duplicate image storage across architectures.
For single-architecture clusters, this behavior has no impact. For details on architecture-specific caching, see the Object Cache documentation.
At this point, you have successfully configured storage for your Fuzzball cluster. You can move on to configuring some initial entities.