Using Encrypted SIF Images
Fuzzball supports running jobs with encrypted SIF (Singularity Image Format) images. The image is encrypted at rest, and Fuzzball supplies the decryption credentials at runtime from its secret management system.
Encrypting a SIF image keeps its contents – proprietary code, models, or regulated data – private in the registry and readable only by jobs that can supply the decryption key.
SIF images can be encrypted using two methods:
- Passphrase-based encryption: a simpler approach using a password/passphrase
- PEM key-based encryption: a more secure approach using RSA public/private key pairs
Apptainer and Singularity support both methods when building encrypted images. The walkthrough below builds a PEM-encrypted image; see the Apptainer documentation on encrypted containers for the full set of options.
Generate a 4096-bit RSA key pair:
$ openssl genrsa -traditional -out rsa_private.pem 4096
$ openssl rsa -in rsa_private.pem -RSAPublicKey_out -out rsa_public.pemBoth the private and public keys should be in PKCS#1 format for Apptainer:
- Use the
-traditionalflag withopenssl genrsato get-----BEGIN RSA PRIVATE KEY-----(PKCS#1 private key format)- Use the
-RSAPublicKey_outflag to get-----BEGIN RSA PUBLIC KEY-----(PKCS#1 public key format)Without these flags, modern OpenSSL versions default to PKCS#8 format, which Apptainer may not accept.
With a passphrase. The example here is interactive but you can provide the key
as an environment variable ($APPTAINER_ENCRYPTION_PASSPHRASE) instead.
$ cat > app.def <<EOF
Bootstrap: docker
From: alpine:latest
EOF
$ apptainer build --passphrase passphrase_app.sif app.def
Enter encryption passphrase:
INFO: Starting build...
...With the key created above:
$ apptainer build --pem-path rsa_public.pem key_app.sif app.def$ apptainer push key_app.sif oras://registry.example.com/apps/secure-app:v1.0Once the image is pushed, create the SIF decryption secret that holds the matching private key – see SIF Decryption Secrets.
The examples below assume that secret already exists, assecret://user/secure-app-key.
Once you have created a decryption secret, reference it from your
Fuzzfile. Specify the encrypted image URI and name the
decryption secret with the decryption-secret field:
version: v4
jobs:
secure-analysis:
image:
uri: oras://registry.example.com/secure/analysis-tool.sif:v1.0
decryption-secret: secret://user/secure-app-key
script: |
#!/bin/sh
echo 'Running secure analysis...' && ./run-analysis.sh
resource:
cpu:
cores: 4
memory:
size: 8GB
The decryption-secret field uses the same URI format as other secrets:
secret://<scope>/<secret-name>.
If your encrypted SIF image is in a private registry, you need both a registry access secret and a decryption secret:
version: v4
jobs:
proprietary-model:
image:
uri: oras://private-registry.company.com/ml/proprietary-model.sif:latest
secret: secret://group/registry-access-token
decryption-secret: secret://user/secure-app-key
script: |
#!/bin/sh
python3 /app/train_model.py --config /config/model.yaml
mounts:
/data: data
resource:
cpu:
cores: 8
memory:
size: 32GB
devices:
nvidia.com/gpu: 1
volumes:
data:
use: ephemeral
size: 10GB
In this example:
secretprovides credentials to pull the image from the private registry.decryption-secretprovides the key to decrypt the SIF image after it is pulled.
Usesecret(singular) for registry access credentials, with a single string URI:secret: secret://<scope>/<name>. The pluralsecretsmap is not supported in v4 workflows.
Fuzzball normally converts the container images it pulls to EROFS on the compute node and caches the result in the object cache. An encrypted image cannot be converted without its key, so it takes a different path:
- The node downloads the image as it is and keeps it in SIF format. The
decryption-secretis handed to the container when it starts, and the image is decrypted there. The key never leaves the node running the job. - Encrypted images are not cached in the object cache. Every node that runs the job pulls the image from its registry; the node keeps its own copy for later jobs on that node.
- An encrypted image can be an object in the object cache — uploaded with
fuzzball object putand referenced with afuzzball://URI — in which case the node downloads it from there instead. - An encrypted image cannot be the
fromof an image built by a workflow: the build converts its base to EROFS, so afromwith adecryption-secretis rejected when the workflow is submitted.
- Use PEM Keys for Production: PEM key-based encryption is more secure than passphrase-based encryption for production workloads.
- Scope Secrets Appropriately: Use user-scoped secrets for personal work, group-scoped for team projects, and organization-scoped only when necessary.
- Rotate Keys: Periodically update your decryption keys and re-encrypt images with new keys.
- Separate Registry and Decryption Secrets: Keep registry access credentials separate from image decryption credentials.
- Verify the decryption secret contains the correct passphrase or PEM key.
- Ensure the encryption method used for the image matches the secret type (passphrase or PEM).
- Check that the secret scope and name are correctly referenced in the Fuzzfile.
Verify the secret exists. List every secret you can see, or filter by name:
$ fuzzball secret list
$ fuzzball secret list --name 'secure-app-key'- Ensure the secret reference in the Fuzzfile matches the secret’s scope and name exactly.
- Confirm you have permission to access the secret (check with your administrator).