Closing Notes and Next Steps
Each workflow can create/use an arbitrary number of volumes and carry out any amount of data ingress and egress. However, it is a good practice to minimize data transfer, which is time-consuming and can be costly. If you find yourself building workflows with multiple ingress and egress operations, it may help to consider a different strategy. Reach out to an administrator or contact CIQ to learn more about building optimized workflows.
As mentioned previously, it is often necessary to use secrets to access online containers and data. Secrets are covered in detail in another section of these guides.
If your workflow submission fails with a volume validation error, check the following:
A volume name must be 1-255 characters and must not contain path separators
(/ or \), path-traversal sequences (. or ..), or null bytes. Letters in
any case, digits, spaces, and other punctuation are allowed. Common issues:
- Path separators: Remove any
/or\from the name (e.g. changedata/settodata-set) - Path traversal: Rename
.,.., or any name containing..(e.g. changemy..datatomy-data) - Empty or too long: Provide a non-empty name no longer than 255 characters
If you reference a persistent volume by name that does not exist, the workflow will be rejected. Verify the volume exists:
$ fuzzball volume listIf the volume does not exist, create it before submitting the workflow:
$ fuzzball volume create <provisioner-name> <volume-name>If you attempt to create an ephemeral volume on a provisioner your group does not have access to, the workflow will be rejected. Contact your cluster administrator to request access or choose a different provisioner.
- The object cache – how Fuzzball caches container images and ingressed objects, and how workflows reuse them.
- Secrets – storing and referencing the credentials that S3 buckets, private repositories, and the HuggingFace Hub require.
- Storage Volumes and Workflows – the full V4 volume syntax, including permissions and ownership.