Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Using Encrypted SIF Images

Fuzzball supports running jobs with encrypted SIF (Singularity Image Format) images. The image is encrypted at rest, and Fuzzball supplies the decryption credentials at runtime from its secret management system.

Encrypting a SIF image keeps its contents – proprietary code, models, or regulated data – private in the registry and readable only by jobs that can supply the decryption key.

Encryption Methods

SIF images can be encrypted using two methods:

  1. Passphrase-based encryption: a simpler approach using a password/passphrase
  2. PEM key-based encryption: a more secure approach using RSA public/private key pairs

Apptainer and Singularity support both methods when building encrypted images. The walkthrough below builds a PEM-encrypted image; see the Apptainer documentation on encrypted containers for the full set of options.

Creating an Encrypted SIF Image

Create an RSA Key Pair

Generate a 4096-bit RSA key pair:

$ openssl genrsa -traditional -out rsa_private.pem 4096

$ openssl rsa -in rsa_private.pem -RSAPublicKey_out -out rsa_public.pem

Both the private and public keys should be in PKCS#1 format for Apptainer:

  • Use the -traditional flag with openssl genrsa to get -----BEGIN RSA PRIVATE KEY----- (PKCS#1 private key format)
  • Use the -RSAPublicKey_out flag to get -----BEGIN RSA PUBLIC KEY----- (PKCS#1 public key format)

Without these flags, modern OpenSSL versions default to PKCS#8 format, which Apptainer may not accept.

Build the Image

With a passphrase. The example here is interactive but you can provide the key as an environment variable ($APPTAINER_ENCRYPTION_PASSPHRASE) instead.

$ cat > app.def <<EOF
Bootstrap: docker
From: alpine:latest
EOF

$ apptainer build --passphrase passphrase_app.sif app.def
Enter encryption passphrase:
INFO:    Starting build...
...

With the key created above:

$ apptainer build --pem-path rsa_public.pem key_app.sif app.def

Push It to a Registry

$ apptainer push key_app.sif oras://registry.example.com/apps/secure-app:v1.0

Once the image is pushed, create the SIF decryption secret that holds the matching private key – see SIF Decryption Secrets.

The examples below assume that secret already exists, as secret://user/secure-app-key.

Using an Encrypted Image in a Fuzzfile

Once you have created a decryption secret, reference it from your Fuzzfile. Specify the encrypted image URI and name the decryption secret with the decryption-secret field:

version: v4

jobs:
  secure-analysis:
    image:
      uri: oras://registry.example.com/secure/analysis-tool.sif:v1.0
      decryption-secret: secret://user/secure-app-key
    script: |
      #!/bin/sh
      echo 'Running secure analysis...' && ./run-analysis.sh
    resource:
      cpu:
        cores: 4
      memory:
        size: 8GB

The decryption-secret field uses the same URI format as other secrets: secret://<scope>/<secret-name>.

Complete Example with Private Registry

If your encrypted SIF image is in a private registry, you need both a registry access secret and a decryption secret:

version: v4

jobs:
  proprietary-model:
    image:
      uri: oras://private-registry.company.com/ml/proprietary-model.sif:latest
      secret: secret://group/registry-access-token
      decryption-secret: secret://user/secure-app-key
    script: |
      #!/bin/sh
      python3 /app/train_model.py --config /config/model.yaml
    mounts:
      /data: data
    resource:
      cpu:
        cores: 8
      memory:
        size: 32GB
      devices:
        nvidia.com/gpu: 1

volumes:
  data:
    use: ephemeral
    size: 10GB

In this example:

  • secret provides credentials to pull the image from the private registry.
  • decryption-secret provides the key to decrypt the SIF image after it is pulled.
Use secret (singular) for registry access credentials, with a single string URI: secret: secret://<scope>/<name>. The plural secrets map is not supported in v4 workflows.

How Fuzzball Handles Encrypted Images

Fuzzball normally converts the container images it pulls to EROFS on the compute node and caches the result in the object cache. An encrypted image cannot be converted without its key, so it takes a different path:

  • The node downloads the image as it is and keeps it in SIF format. The decryption-secret is handed to the container when it starts, and the image is decrypted there. The key never leaves the node running the job.
  • Encrypted images are not cached in the object cache. Every node that runs the job pulls the image from its registry; the node keeps its own copy for later jobs on that node.
  • An encrypted image can be an object in the object cache — uploaded with fuzzball object put and referenced with a fuzzball:// URI — in which case the node downloads it from there instead.
  • An encrypted image cannot be the from of an image built by a workflow: the build converts its base to EROFS, so a from with a decryption-secret is rejected when the workflow is submitted.

Best Practices

Security Recommendations

  1. Use PEM Keys for Production: PEM key-based encryption is more secure than passphrase-based encryption for production workloads.
  2. Scope Secrets Appropriately: Use user-scoped secrets for personal work, group-scoped for team projects, and organization-scoped only when necessary.
  3. Rotate Keys: Periodically update your decryption keys and re-encrypt images with new keys.
  4. Separate Registry and Decryption Secrets: Keep registry access credentials separate from image decryption credentials.

Troubleshooting

Error: “failed to decrypt SIF image”

  • Verify the decryption secret contains the correct passphrase or PEM key.
  • Ensure the encryption method used for the image matches the secret type (passphrase or PEM).
  • Check that the secret scope and name are correctly referenced in the Fuzzfile.

Error: “secret not found”

Verify the secret exists. List every secret you can see, or filter by name:

$ fuzzball secret list
$ fuzzball secret list --name 'secure-app-key'
  • Ensure the secret reference in the Fuzzfile matches the secret’s scope and name exactly.
  • Confirm you have permission to access the secret (check with your administrator).

See also