Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Per-Workflow Charges

Fuzzball tracks the resources each workflow uses, and the storage and object cache usage between workflows, and reports the charges through the CLI and the accounting API. Use this view to attribute compute, storage, and egress costs to specific workflows, jobs, and users — orthogonal to the cloud-provider billing covered in Tracking AWS Cloud Spend.

What gets measured

Fuzzball records five kinds of charges:

  • Compute — node-hours of the substrate instance(s) running the workflow, priced per provisioner definition.
  • Volume — GiB-hours that any storage volumes remained attached or held data on behalf of the workflow.
  • Egress — bytes leaving the substrate node, bucketed into an egress class and priced per class.
  • Ingress — bytes entering the substrate node. Tracked but not priced today.
  • Object cache — GiB-hours that objects occupy in the object cache, charged to the object’s namespace owner.

Each charge is denominated in credits, the internal accounting unit. Catalog rates are authored against published cloud-provider prices (e.g. AWS EC2 hourly rates, AWS published data-egress rates), so the credit values track those prices directly. There is no automatic conversion to local currency at present.

These catalog rates are list rates. An organization with a price adjustment is charged each one scaled by its multiplier for that kind of charge, at the adjustment in effect when the usage started.

Viewing charges

fuzzball report charges lists every charge you can see, newest first. Without flags it covers your currently selected group over the last seven days, on every cluster:

$ fuzzball report charges

START                 | CATEGORY     | ITEM                    | CLUSTER         | CREDITS
2026-09-07 01:02:00PM | compute      | train/worker            | aws.example.com | 0.04160000 *
2026-09-07 09:30:00AM | egress       | etl/upload              | gcp.example.com | 0.00176000
2026-09-07 09:30:00AM | ingress      | etl/upload              | gcp.example.com | -
2026-09-07 08:00:00AM | volume       | default/data            | aws.example.com | 0.01680000
2026-09-07 07:15:00AM | object cache | group/models/resnet.bin | aws.example.com | 0.00320000

5 charges from 2026-08-31 02:00:00PM to 2026-09-07 02:00:00PM.
* still accruing: counted up to when the report was computed.

Each row is one charge line. ITEM names what the line charges for: the workflow and stage for compute, network, and workflow volumes; the provisioner and volume for a persistent volume’s own storage; and the object’s path for object cache. A - in CREDITS means no rate applies, so the usage is reported without credits; ingress is always in this state today. An * marks a line still accruing.

The default table leaves out the quantity, rate, and units; -o yaml or -o json shows every field (see Output format).

Choosing what to report

Each flag narrows the report. None of them widens what you may see (see Scope and visibility).

FlagReports on
--group GROUPA group, by name or ID, instead of your current group. Repeatable.
--user USEROne user’s charges, by email or ID: the workflows they ran and the persistent volumes they created.
--cluster CLUSTEROne cluster, by name or ID.
--workflow WORKFLOWA workflow’s compute, network, and workflow volume charges, by ID. Repeatable.
--volume VOLUMEA volume’s charges, by name, PROVISIONER/VOLUME, or ID. Repeatable.
--after TIME, --before TIMEThe window, as an RFC3339 time or a duration ago such as 30d or 12h. --before defaults to now and --after to seven days before --before.
--include-personal-objectsAdds the objects in your personal object cache namespace, or the --user’s.

Every charge is clipped to the window, so only the part inside it counts. A workflow that ran across the start of the window contributes only the time after it.

$ fuzzball report charges --group research --user jdoe@example.com \
    --after 2026-09-01T00:00:00Z --before 2026-10-01T00:00:00Z

Totals

--summary totals the same charges by category:

$ fuzzball report charges --summary

Window:    from 2026-08-31 02:00:00PM to 2026-09-07 02:00:00PM
Credits:   0.06336000
Workflows: 2

CATEGORY     | QUANTITY              | CREDITS    | LINES
compute      | 2 hours               | 0.04160000 | 1
egress       | 0.013 GiB             | 0.00176000 | 1
ingress      | 1.9 GiB, 1.9 unpriced | 0.00000000 | 1
object cache | 1.6 GiB-hours         | 0.00320000 | 1
volume       | 168 GiB-hours         | 0.01680000 | 1

Each total is the sum of the lines the listing shows for the same flags. QUANTITY is the category’s whole quantity; when no rate applies to part of it, that part is shown after it as unpriced and contributes no credits.

Federated deployments

On a Federate, the report covers every registered member cluster that is not disabled, and --cluster narrows it to one. If a cluster is not ready, does not answer in time, or refuses, the command warns on standard error and leaves that cluster’s charges out:

warning: this report is missing the charges of azure.example.com, which did not answer (not ready, timed out, or refused)

With -o json or -o yaml, the missing clusters’ IDs are also in unreachable_cluster_ids. If no cluster answers, the command fails. A member cluster running a release from before this report does not serve it, so until it is upgraded it is always among the clusters named.

Lifetime volume charges

A persistent volume appears once in a report, as its own storage line over the part of its life inside the window – not once per workflow that attached it. To see a volume’s spend over its whole life, widen the window to cover it:

$ fuzzball report charges --volume default/data --after 2026-01-01T00:00:00Z --summary

Object cache usage

Objects in the object cache occupy storage from the moment they are cached until they are purged (TTL expiry, disk-pressure eviction, or explicit delete). That usage is metered as GiB-hours against the object’s namespace owner — the group for group/ objects, the user for user/ objects — independent of which workflows read the object.

Object cache usage appears in fuzzball report charges as object cache lines, and in its --summary as their total. It is priced by the cluster-wide objectCache rate in central config:

objectCache:
  costPerGbHour: 0.002

Usage is priced at the rate in effect when it started, scaled by the organization’s price adjustment for object cache. Changing the rate applies to objects cached from then on and never re-prices earlier usage. Overwriting an object closes out its previous content at the size and rate it had, and the new content starts accruing at the current rate. Objects cached while no rate is configured stay unpriced: the GiB-hours quantity is still reported but contributes no credits. Each object’s cached time is clipped to the report’s window.

On a cluster upgraded from a release that priced object cache at report time, usage that was already recorded – including objects still cached – is priced at the first objectCache rate the upgraded cluster has configured, without the organization’s price adjustment, which did not apply to object cache then. An object cached before the upgrade keeps that rate until it is overwritten or purged. If no rate is configured at the upgrade, that usage is reported without credits until one is, and is then priced at it.

Scope and visibility

fuzzball report charges reports on groups: your currently selected group, or the groups named with --group, each of which must be a group you are a member of. Any member of a group sees every charge in it, whoever ran the workflow. --user, --cluster, --workflow, and --volume narrow that; none of them widens it.

Objects in a personal object cache namespace belong to no group, so they are reported only with --include-personal-objects, and only to their owner and organization owners.

The earlier report commands narrow by role instead: regular users see only their own workflows, group owners see the workflows in groups they own, and organization owners see every workflow in their organization.

Charges report API

The accounting API’s charges report itemizes every kind of charge in one list: compute, volume, egress, ingress, and object cache. It is served at GET /v4/accounting/charges, with totals for the same selection at GET /v4/accounting/charges/summary.

  • Who sees what. A report covers groups: by default the caller’s currently selected group, or the groups named in group_ids, each of which must be a group the caller is a member of. Any member sees every charge in the group, whoever ran the workflow. user_id narrows the report to one user’s workflows and the persistent volumes they created.
  • Personal object cache. Objects in a user’s personal object cache namespace belong to no group, so they are reported only when include_personal_objects is set: the caller’s own, or those of the user named in user_id. Only that user and organization owners may read them.
  • Time window. Every line is clipped to the window between after and before, so only the part of a charge inside the window counts. before defaults to the time the report is computed, and after to seven days before before. The pages of one listing all cover the window its first page was computed over.
  • Usage still accruing. A line whose usage is still running ends at the report’s as_of time and is marked ongoing; it grows in later reports. Egress and ingress are measured when a container stops, so a running container has no network lines yet.
  • Units. Each line carries quantity_unit and rate_unit, and for a priced line quantity × rate = credits exactly. Quantities keep at most 10 decimal places. Network quantities are in GiB, the unit egress is priced in; the raw byte count is in bytes.
  • Nodes. Compute and network lines carry the node_id the node service knows the node by, and the host_name the node reported when the usage was recorded.
  • Persistent volumes appear once each, over their own lifetime, not once per workflow that attached them.
  • Federated deployments. On a Federate, a report covers every registered member cluster that is not disabled, unless cluster_id names one. The Federate decides which groups the caller may read, and every cluster reports over the same window. A cluster that is not ready, does not answer in time, or refuses is listed in unreachable_cluster_ids and its charges are missing. Once a cluster is missing from a page of a listing, it stays missing and listed for the rest of that listing; start a new listing to include it. If no cluster answers, the request fails.

Each total in the summary is the sum of the lines the list returns for the same filters and window. To total exactly the lines a listing returned, pass its response’s window.after and window.before.

Egress classes on AWS

When an AWS substrate runs a workflow, each destination address the workload egresses to is classified by walking the source node’s VPC route table. The class determines the rate. Default classes shipped in the AWS egress catalog:

ClassDefault rate (credits/GiB)Triggered when…
intra_az_local0.00Destination is a pod/service in the source node’s own subnet
intra_az_other_subnet0.00Destination is in the source’s AZ but a different VPC subnet
cross_az_intra_vpc0.01Destination is in the same VPC but a different AZ
via_s3_gateway_endpoint0.00Destination resolves to S3 in-region and the VPC has an S3 gateway endpoint
via_dynamodb_gateway_endpoint0.00Same, for DynamoDB
via_privatelink0.01Destination resolves to a configured PrivateLink interface endpoint
cross_region_aws_service0.02Destination is an AWS-published service IP in a region different from the cluster
via_nat_to_internet0.135Default route via NAT gateway (most generic internet egress)
via_igw_to_internet0.09Default route via Internet Gateway (substrate in a public subnet)
unknown0.135Source-instance placement or VPC topology could not be resolved

The unknown class is intentionally priced at the most expensive realistic rate so misclassifications surface as conservative billing rather than free traffic.

On non-AWS deployments (on-prem, Slurm/PBS) the matcher is source-blind: it applies operator-declared CIDR rules from central config instead of walking VPC routing state. The class names emitted in that mode are whatever the operator named in those rules. Rules may use any prefix length, down to a single host (/32), so a specific license server or storage endpoint can carry its own class and rate.

Output format

fuzzball report charges honors the global --output/-o flag, which accepts json or yaml. Omit it for the table, or with --summary the labeled totals. There is no --output table value.

$ fuzzball report charges --workflow <workflow_id> -o yaml

$ fuzzball report charges --after 2026-09-01T00:00:00Z --summary -o json

Structured output of a listing carries every line from every page in one document, with the window it covers and any unreachable_cluster_ids. Its page_token is always empty, because the CLI fetches every page itself. If --max-pages stops it first, the output holds only the pages fetched, and the CLI warns on standard error that more charges exist.

Reading structured charge output

Structured output carries fields the table leaves out, such as each line’s quantity, quantity_unit, rate, and rate_unit.

JSON and YAML carry the same records but are not textually interchangeable, so a jq filter does not port to yq unchanged:

JSONYAML
Field namesworkflowIdworkflow_id
Enum values"CHARGE_CATEGORY_COMPUTE"charge_category_compute
64-bit integers"3" (quoted)3

Zero-valued scalars are emitted rather than dropped, so a false boolean or a 0 count is always present. Unpopulated message fields are dropped, so timestamps still need a presence check: a line still accruing has an end at the report’s as_of, but an open volume lifetime in the earlier commands emits no deleted key.

Pricing values sit at these paths, in the YAML spelling; in JSON each name takes the lowerCamelCase form:

CommandPricing field paths
chargescharges[].priced, charges[].credits
charges --summarycredits, categories[].credits, categories[].unpriced_quantity

Earlier report commands

These commands are deprecated in favor of fuzzball report charges. They still run, print a deprecation notice, and no longer appear in --help. Their --cluster, --group, and --user flags accept names and emails, as fuzzball report charges does. Unlike fuzzball report charges, they report on one cluster at a time: on a Federate each needs --cluster.

Deprecated commandUse instead
fuzzball report charges workflow <workflow_id>fuzzball report charges --workflow <workflow_id>
fuzzball report charges volume <volume>fuzzball report charges --volume <volume>
fuzzball report charge-summary workflow <workflow_id>fuzzball report charges --workflow <workflow_id> --summary
fuzzball report charge-summary listfuzzball report charges
fuzzball report charge-summary aggregatefuzzball report charges --summary

Their output keeps its earlier shape, so existing scripts keep working. Their time flags, --start-from and --start-until, select workflows by start time rather than clipping charges to a window, so totals can differ from fuzzball report charges over the same dates.