Per-Workflow Charges
Fuzzball tracks the resources each workflow uses, and the storage and object cache usage between workflows, and reports the charges through the CLI and the accounting API. Use this view to attribute compute, storage, and egress costs to specific workflows, jobs, and users — orthogonal to the cloud-provider billing covered in Tracking AWS Cloud Spend.
Fuzzball records five kinds of charges:
- Compute — node-hours of the substrate instance(s) running the workflow, priced per provisioner definition.
- Volume — GiB-hours that any storage volumes remained attached or held data on behalf of the workflow.
- Egress — bytes leaving the substrate node, bucketed into an egress class and priced per class.
- Ingress — bytes entering the substrate node. Tracked but not priced today.
- Object cache — GiB-hours that objects occupy in the object cache, charged to the object’s namespace owner.
Each charge is denominated in credits, the internal accounting unit. Catalog
rates are authored against published cloud-provider prices (e.g. AWS EC2 hourly
rates, AWS published data-egress rates), so the credit values track those
prices directly. There is no automatic conversion to local currency at present.
These catalog rates are list rates. An organization with a price adjustment is charged each one scaled by its multiplier for that kind of charge, at the adjustment in effect when the usage started.
fuzzball report charges lists every charge you can see, newest first. Without
flags it covers your currently selected group over the last seven days, on
every cluster:
$ fuzzball report charges
START | CATEGORY | ITEM | CLUSTER | CREDITS
2026-09-07 01:02:00PM | compute | train/worker | aws.example.com | 0.04160000 *
2026-09-07 09:30:00AM | egress | etl/upload | gcp.example.com | 0.00176000
2026-09-07 09:30:00AM | ingress | etl/upload | gcp.example.com | -
2026-09-07 08:00:00AM | volume | default/data | aws.example.com | 0.01680000
2026-09-07 07:15:00AM | object cache | group/models/resnet.bin | aws.example.com | 0.00320000
5 charges from 2026-08-31 02:00:00PM to 2026-09-07 02:00:00PM.
* still accruing: counted up to when the report was computed.Each row is one charge line. ITEM names what the line charges for: the
workflow and stage for compute, network, and workflow volumes; the
provisioner and volume for a persistent volume’s own storage; and the object’s
path for object cache. A - in CREDITS means no rate applies, so the usage
is reported without credits; ingress is always in this state today. An *
marks a line still accruing.
The default table leaves out the quantity, rate, and units; -o yaml or
-o json shows every field (see Output format).
Each flag narrows the report. None of them widens what you may see (see Scope and visibility).
| Flag | Reports on |
|---|---|
--group GROUP | A group, by name or ID, instead of your current group. Repeatable. |
--user USER | One user’s charges, by email or ID: the workflows they ran and the persistent volumes they created. |
--cluster CLUSTER | One cluster, by name or ID. |
--workflow WORKFLOW | A workflow’s compute, network, and workflow volume charges, by ID. Repeatable. |
--volume VOLUME | A volume’s charges, by name, PROVISIONER/VOLUME, or ID. Repeatable. |
--after TIME, --before TIME | The window, as an RFC3339 time or a duration ago such as 30d or 12h. --before defaults to now and --after to seven days before --before. |
--include-personal-objects | Adds the objects in your personal object cache namespace, or the --user’s. |
Every charge is clipped to the window, so only the part inside it counts. A workflow that ran across the start of the window contributes only the time after it.
$ fuzzball report charges --group research --user jdoe@example.com \
--after 2026-09-01T00:00:00Z --before 2026-10-01T00:00:00Z--summary totals the same charges by category:
$ fuzzball report charges --summary
Window: from 2026-08-31 02:00:00PM to 2026-09-07 02:00:00PM
Credits: 0.06336000
Workflows: 2
CATEGORY | QUANTITY | CREDITS | LINES
compute | 2 hours | 0.04160000 | 1
egress | 0.013 GiB | 0.00176000 | 1
ingress | 1.9 GiB, 1.9 unpriced | 0.00000000 | 1
object cache | 1.6 GiB-hours | 0.00320000 | 1
volume | 168 GiB-hours | 0.01680000 | 1Each total is the sum of the lines the listing shows for the same flags.
QUANTITY is the category’s whole quantity; when no rate applies to part of
it, that part is shown after it as unpriced and contributes no credits.
On a Federate,
the report covers every registered member cluster that is not disabled, and
--cluster narrows it to one. If a cluster is not ready, does not answer in
time, or refuses, the command warns on standard error and leaves that
cluster’s charges out:
warning: this report is missing the charges of azure.example.com, which did not answer (not ready, timed out, or refused)With -o json or -o yaml, the missing clusters’ IDs are also in
unreachable_cluster_ids. If no cluster answers, the command fails. A member
cluster running a release from before this report does not serve it, so until
it is upgraded it is always among the clusters named.
A persistent volume appears once in a report, as its own storage line over the part of its life inside the window – not once per workflow that attached it. To see a volume’s spend over its whole life, widen the window to cover it:
$ fuzzball report charges --volume default/data --after 2026-01-01T00:00:00Z --summaryObjects in the object cache occupy
storage from the moment they are cached until they are purged (TTL expiry,
disk-pressure eviction, or explicit delete). That usage is metered as
GiB-hours against the object’s namespace owner — the group for group/
objects, the user for user/ objects — independent of which workflows read
the object.
Object cache usage appears in fuzzball report charges as object cache
lines, and in its --summary as their total. It is priced by the cluster-wide objectCache
rate in central config:
objectCache:
costPerGbHour: 0.002
Usage is priced at the rate in effect when it started, scaled by the organization’s price adjustment for object cache. Changing the rate applies to objects cached from then on and never re-prices earlier usage. Overwriting an object closes out its previous content at the size and rate it had, and the new content starts accruing at the current rate. Objects cached while no rate is configured stay unpriced: the GiB-hours quantity is still reported but contributes no credits. Each object’s cached time is clipped to the report’s window.
On a cluster upgraded from a release that priced object cache at report time,
usage that was already recorded – including objects still cached – is priced
at the first objectCache rate the upgraded cluster has configured, without the
organization’s price adjustment, which did not apply to object cache then. An
object cached before the upgrade keeps that rate until it is overwritten or
purged. If no rate is configured at the upgrade, that usage is reported without
credits until one is, and is then priced at it.
fuzzball report charges reports on groups: your currently selected group, or
the groups named with --group, each of which must be a group you are a member
of. Any member of a group sees every charge in it, whoever ran the workflow.
--user, --cluster, --workflow, and --volume narrow that; none of them
widens it.
Objects in a personal object cache namespace belong to no group, so they are
reported only with --include-personal-objects, and only to their owner and
organization owners.
The earlier report commands narrow by role instead: regular users see only their own workflows, group owners see the workflows in groups they own, and organization owners see every workflow in their organization.
The accounting API’s charges report itemizes every kind of charge in one
list: compute, volume, egress, ingress, and object cache. It is served at
GET /v4/accounting/charges, with totals for the same selection at
GET /v4/accounting/charges/summary.
- Who sees what. A report covers groups: by default the caller’s
currently selected group, or the groups named in
group_ids, each of which must be a group the caller is a member of. Any member sees every charge in the group, whoever ran the workflow.user_idnarrows the report to one user’s workflows and the persistent volumes they created. - Personal object cache. Objects in a user’s personal object cache
namespace belong to no group, so they are reported only when
include_personal_objectsis set: the caller’s own, or those of the user named inuser_id. Only that user and organization owners may read them. - Time window. Every line is clipped to the window between
afterandbefore, so only the part of a charge inside the window counts.beforedefaults to the time the report is computed, andafterto seven days beforebefore. The pages of one listing all cover the window its first page was computed over. - Usage still accruing. A line whose usage is still running ends at the
report’s
as_oftime and is markedongoing; it grows in later reports. Egress and ingress are measured when a container stops, so a running container has no network lines yet. - Units. Each line carries
quantity_unitandrate_unit, and for a priced linequantity×rate=creditsexactly. Quantities keep at most 10 decimal places. Network quantities are in GiB, the unit egress is priced in; the raw byte count is inbytes. - Nodes. Compute and network lines carry the
node_idthe node service knows the node by, and thehost_namethe node reported when the usage was recorded. - Persistent volumes appear once each, over their own lifetime, not once per workflow that attached them.
- Federated deployments. On a
Federate, a
report covers every registered member cluster that is not disabled, unless
cluster_idnames one. The Federate decides which groups the caller may read, and every cluster reports over the same window. A cluster that is not ready, does not answer in time, or refuses is listed inunreachable_cluster_idsand its charges are missing. Once a cluster is missing from a page of a listing, it stays missing and listed for the rest of that listing; start a new listing to include it. If no cluster answers, the request fails.
Each total in the summary is the sum of the lines the list returns for the
same filters and window. To total exactly the lines a listing returned, pass
its response’s window.after and window.before.
When an AWS substrate runs a workflow, each destination address the workload egresses to is classified by walking the source node’s VPC route table. The class determines the rate. Default classes shipped in the AWS egress catalog:
| Class | Default rate (credits/GiB) | Triggered when… |
|---|---|---|
intra_az_local | 0.00 | Destination is a pod/service in the source node’s own subnet |
intra_az_other_subnet | 0.00 | Destination is in the source’s AZ but a different VPC subnet |
cross_az_intra_vpc | 0.01 | Destination is in the same VPC but a different AZ |
via_s3_gateway_endpoint | 0.00 | Destination resolves to S3 in-region and the VPC has an S3 gateway endpoint |
via_dynamodb_gateway_endpoint | 0.00 | Same, for DynamoDB |
via_privatelink | 0.01 | Destination resolves to a configured PrivateLink interface endpoint |
cross_region_aws_service | 0.02 | Destination is an AWS-published service IP in a region different from the cluster |
via_nat_to_internet | 0.135 | Default route via NAT gateway (most generic internet egress) |
via_igw_to_internet | 0.09 | Default route via Internet Gateway (substrate in a public subnet) |
unknown | 0.135 | Source-instance placement or VPC topology could not be resolved |
The unknown class is intentionally priced at the most expensive realistic
rate so misclassifications surface as conservative billing rather than free
traffic.
On non-AWS deployments (on-prem, Slurm/PBS) the matcher is source-blind: it
applies operator-declared CIDR rules from
central config instead of walking VPC routing
state. The class names emitted in that mode are whatever the operator named in
those rules. Rules may use any prefix length, down to a single host (/32),
so a specific license server or storage endpoint can carry its own class and
rate.
fuzzball report charges honors the global --output/-o flag, which
accepts json or yaml. Omit it for the table, or with --summary the
labeled totals. There is no --output table value.
$ fuzzball report charges --workflow <workflow_id> -o yaml
$ fuzzball report charges --after 2026-09-01T00:00:00Z --summary -o jsonStructured output of a listing carries every line from every page in one
document, with the window it covers and any unreachable_cluster_ids. Its
page_token is always empty, because the CLI fetches every page itself. If
--max-pages stops it first, the output holds only the pages fetched, and the
CLI warns on standard error that more charges exist.
Structured output carries fields the table leaves out, such as each line’s
quantity, quantity_unit, rate, and rate_unit.
JSON and YAML carry the same records but are not textually interchangeable, so
a jq filter does not port to yq unchanged:
| JSON | YAML | |
|---|---|---|
| Field names | workflowId | workflow_id |
| Enum values | "CHARGE_CATEGORY_COMPUTE" | charge_category_compute |
| 64-bit integers | "3" (quoted) | 3 |
Zero-valued scalars are emitted rather than dropped, so a false boolean or a
0 count is always present. Unpopulated message fields are dropped, so
timestamps still need a presence check: a line still accruing has an end at
the report’s as_of, but an open volume lifetime in the earlier commands emits
no deleted key.
Pricing values sit at these paths, in the YAML spelling; in JSON each name takes the lowerCamelCase form:
| Command | Pricing field paths |
|---|---|
charges | charges[].priced, charges[].credits |
charges --summary | credits, categories[].credits, categories[].unpriced_quantity |
These commands are deprecated in favor of fuzzball report charges. They still
run, print a deprecation notice, and no longer appear in --help. Their
--cluster, --group, and --user flags accept names and emails, as
fuzzball report charges does. Unlike fuzzball report charges, they report
on one cluster at a time: on a Federate each needs --cluster.
| Deprecated command | Use instead |
|---|---|
fuzzball report charges workflow <workflow_id> | fuzzball report charges --workflow <workflow_id> |
fuzzball report charges volume <volume> | fuzzball report charges --volume <volume> |
fuzzball report charge-summary workflow <workflow_id> | fuzzball report charges --workflow <workflow_id> --summary |
fuzzball report charge-summary list | fuzzball report charges |
fuzzball report charge-summary aggregate | fuzzball report charges --summary |
Their output keeps its earlier shape, so existing scripts keep working. Their
time flags, --start-from and --start-until, select workflows by start time
rather than clipping charges to a window, so totals can differ from
fuzzball report charges over the same dates.