Agent Token Accounting
Each agent turn reports what it spent: prompt tokens, completion tokens, a total,
and the provider model id that produced them, such as gpt-4o. Those are
aggregated onto the session, so a session also carries a running total and a
turn count.
A few things are worth knowing about the numbers:
- They are the provider’s, not ours. The counts are whatever the endpoint reported, stored as given. Providers do not always report a total equal to prompt plus completion, and re-deriving it would disagree with the bill.
- They are counts, not costs. There is no currency figure anywhere; pricing is per-provider and Fuzzball has no source of truth for it.
- They are advisory. Nothing is capped or blocked by them, and a turn never fails because accounting failed.
- A truncated answer still counts. A reply that hit the server’s response cap is recorded with the tokens it actually cost, not as free – it is typically the most expensive turn a session has.
- Disconnecting does not lose them. If you close the CLI or the browser mid-answer, the turn stops and the session is free for your next message at once. The server keeps reading the provider’s stream in the background for up to 10 seconds to collect its usage report, and adds what the call cost to the session. The provider keeps generating, and charging, during that time. If the answer does not finish within it, the turn is marked as described below.
A turn whose cost was never reported is marked, not silently counted as free.
Usage arrives on the provider’s final chunk, so a stream cut before it, one
abandoned while searching for it, or a provider that simply never sends one all
leave a number that is real but short – or zero. Any of those sets
usage_incomplete on the turn, which is the difference between “this turn cost
nothing” and “nobody told us what this turn cost”. OpenAI only sends the chunk
when asked, and a self-hosted endpoint may never send it, so this is the
ordinary case rather than an exotic one.
A count can also be marked for the opposite reason: a count too large to store is clamped, and the clamp sets the same mark. The ceiling is far above anything a real turn can use – roughly twenty-seven times a full turn at the largest context window on offer – so an honest turn never meets it. It exists because the totals are stored in a column that accumulates, and a single absurd report would otherwise poison every later turn on that session. The clamp applies to the sum, so either one enormous report or several large ones together can trip it.
A report with a negative count, or missing any of the three counts, is refused outright, and that is the whole report for that model call – not just the offending number. A report carrying a negative is not trustworthy in its other fields either, so the server keeps none of them: that call contributes nothing and the totals are marked. The turn itself still completes and any other model call in it still counts. (If a valid report arrived for that same call, in either order, it stands and the negative one is ignored.)
A stream the server has to abandon is marked too, whether it was still producing or had gone silent – it cannot tell those apart, and it marks both. If the provider’s usage had already arrived, that number may well be its final answer and the turn is marked anyway.
So the mark does not tell you by how much, or even that there is a gap at all. What it does guarantee is that the number is never an over-count: the true cost is that number or more. Read a marked total as a lower bound and an unmarked one as exact, apart from the gaps listed below.
The mark is sticky on the session: once one turn has set it, the running
total carries it for good, because a total that has absorbed an unvouched-for
turn cannot become exact again. A turn’s completion event reports the flag when
EITHER its own counts or the session totals are affected – the two are
affected independently, and the event carries one flag for both, deliberately
erring towards “approximate”. Session.usage_incomplete is the authoritative
per-session answer for a client that needs them apart.
A turn whose usage may not have been written is not approximated: the session’s totals are marked as a lower bound instead, and a finished turn’s completion event is marked too, with its session totals left at zero. Its turn may not be counted either, so a marked session’s turn count can be short as well as its token totals. A turn cut off because Orchestrate died mid-turn is covered too: its claim on the session outlives it, and once the claim lapses – within 90 seconds of its last refresh – the session reads as a lower bound, and the next turn to take the claim over stores the mark. A turn whose request handler panics is marked before its claim is released – even one that panics after its completion was sent, whose totals were in fact recorded. A crash that ends the process while a turn holds its claim leaves the claim to lapse as above.
Two gaps remain, and in both the stored total is short but reads as exact:
- The database refuses the mark as well. A finished turn’s own completion event
still carries
usage_incomplete, but the session as read back –GetSession, and the session totals on later turns’ events – does not, and a failed or abandoned turn has no event to mark. The server logs this as an error. - Orchestrate restarts, or a background goroutine crashes it, during the few seconds it spends collecting or writing the usage of a turn whose client disconnected. That work outlives the turn’s claim, which is released as the turn returns, so nothing marks the cost it loses. A graceful restart is no different: shutdown does not wait for it.
Short of a restart, that same work only delays the count: a disconnected turn’s tokens are added a few seconds after it ends, and a client reading the session in that window sees the total without them.
A session that already had turns before the upgrade that added these totals starts counting from zero, so the upgrade marks it as a lower bound too. One limit applies during a rolling upgrade: a replica still on the older version neither counts a turn it serves nor stores the mark when it takes over, renews or releases a lapsed claim. A session can then read as exact while short, and a session already reading as a lower bound because of a lapsed claim can read as exact again.
The API reports both: each turn’s completion event carries that turn’s counts
plus the session’s running total and turn count, and the session itself carries
the persisted figures. The fuzzball agent CLI shows them: each turn’s token
line and the status line carry the session’s running total, and /session and
fuzzball agent show add its prompt and completion counts. The web interface
does not show them yet.