Using Encrypted SIF Images

Fuzzball supports running jobs with encrypted SIF (Singularity Image Format) images. This feature enables you to protect sensitive container images by encrypting them and securely providing decryption credentials at runtime through Fuzzball’s secret management system.

Encrypted SIF images provide several important security benefits:

• Protect Proprietary Code: Keep proprietary algorithms, models, or applications confidential even when stored in registries.
• Compliance Requirements: Meet regulatory requirements that mandate encryption for sensitive workloads.
• Access Control: Ensure that only authorized users with the correct decryption keys can run specific containers.
• Secure Distribution: Share containers across teams or organizations without exposing their contents.

SIF images can be encrypted using two methods:

1. Passphrase-based encryption: a simpler approach using a password/passphrase
2. PEM key-based encryption: a more secure approach using RSA public/private key pairs

Both methods are supported by Apptainer/Singularity when building encrypted images. See the Apptainer documentation on encrypted containers for details on creating encrypted SIF images. A practical example for creating keys and encrypted images is also included in our decryption secret documentation. Here we assume that a SIF decryption secret with a private key (secret://user/secure-app-key) has been created already.

Once you’ve created a decryption secret, you can reference it in your Fuzzfile to use encrypted SIF images in your workflow jobs.

In your Fuzzfile, specify the encrypted image URI and reference the decryption secret using the decryption-secret field:

version: v4

jobs:
  secure-analysis:
    image:
      uri: oras://registry.example.com/secure/analysis-tool.sif:v1.0
      decryption-secret: secret://user/secure-app-key
    script: |
      #!/bin/sh
      echo 'Running secure analysis...' && ./run-analysis.sh
    resource:
      cpu:
        cores: 4
      memory:
        size: 8GB

The decryption-secret field uses the same URI format as other secrets: secret://<scope>/<secret-name>.

If your encrypted SIF image is in a private registry, you’ll need both a registry access secret and a decryption secret:

version: v4

jobs:
  proprietary-model:
    image:
      uri: oras://private-registry.company.com/ml/proprietary-model.sif:latest
      secret: secret://group/registry-access-token
      decryption-secret: secret://user/secure-app-key
    script: |
      #!/bin/sh
      python3 /app/train_model.py --config /config/model.yaml
    mounts:
      data:
        location: /data
    resource:
      cpu:
        cores: 8
      memory:
        size: 32GB
      devices:
        nvidia.com/gpu: 1

volumes:
  data:
    reference: volume://user/ephemeral

In this example:

• secret provides credentials to pull the image from the private registry.
• decryption-secret provides the key to decrypt the SIF image after it’s pulled.

Image Pull Secret Field: The secret field (singular) is used for registry access credentials. Do not use secrets (plural map) — that form is not supported in v4 workflows. The correct syntax is secret: secret://<scope>/<name> (a single string URI).

Fuzzball normally converts the container images it pulls to EROFS on the compute node and caches the result in the object cache. An encrypted image cannot be converted without its key, so it takes a different path:

• The node downloads the image as it is and keeps it in SIF format. The decryption-secret is handed to the container when it starts, and the image is decrypted there. The key never leaves the node running the job.
• Encrypted images are not cached in the object cache. Every node that runs the job pulls the image from its registry; the node keeps its own copy for later jobs on that node.
• An encrypted image can be an object in the object cache — uploaded with fuzzball object put and referenced with a fuzzball:// URI — in which case the node downloads it from there instead.
• An encrypted image cannot be the from of an image built by a workflow: the build converts its base to EROFS, so a from with a decryption-secret is rejected when the workflow is submitted.

1. Use PEM Keys for Production: PEM key-based encryption is more secure than passphrase-based encryption for production workloads.
2. Scope Secrets Appropriately: Use user-scoped secrets for personal work, group-scoped for team projects, and organization-scoped only when necessary.
3. Rotate Keys: Periodically update your decryption keys and re-encrypt images with new keys.
4. Separate Registry and Decryption Secrets: Keep registry access credentials separate from image decryption credentials.

Error: “failed to decrypt SIF image”

• Verify the decryption secret contains the correct passphrase or PEM key.
• Ensure the encryption method used for the image matches the secret type (passphrase vs. PEM).
• Check that the secret scope and name are correctly referenced in the Fuzzfile.

Error: “secret not found”

• Verify the secret exists. --reference accepts shell-style wildcards:

  # List every secret in the user scope
  $ fuzzball secret list --reference 'user/*'
  
  # Find a specific secret across all scopes
  $ fuzzball secret list --reference '*/secure-app-key'
  
  # Inspect every accessible secret
  $ fuzzball secret list

• Ensure the secret reference in the Fuzzfile matches the secret’s scope and name exactly.

• Confirm you have permission to access the secret (check with your administrator).

• Apptainer Encryption Documentation
• Fuzzfile Syntax Guide
• Decryption Secrets Management