Using Volumes
HPC jobs typically require data. Volumes allow your jobs to access external data, allow multiple jobs within the same workflow to share data, and provide a way to save data after a workflow is completed.
Fuzzball supports the use of ephemeral volumes that last for the lifetime of a given workflow, or persistent volumes that retain data after workflow completion. Volumes can pull data from web URLs and from API-compliant S3 object stores. The Ingress page lists every source scheme Fuzzball accepts.
Your cluster administrator configures the storage provisioners that back your organization’s volumes and control which groups and users can reach them. You name the volumes you want in your workflow definition, and Fuzzball does the rest – selecting the provisioner, creating ephemeral volumes on demand, and mounting them into your job containers. Persistent volumes must exist before a workflow can use them.
In many cases, it is necessary to provide a password, token, or other secret credential to access volumes such as S3 buckets, private GitHub repositories, and gated HuggingFace repositories. Secrets are saved within the Fuzzball secret store (via the web UI or CLI) and then referenced by scope and name within workflow specification files. Details are provided in the Secrets section of the User manual.