Token usage and cost

Six surfaces, six questions:

Surface	Answers
The assistant footer in a thread	what this turn cost
/status	what the last turn cost, as text
Metrics → Usage	what a window cost, per day, per model, and in money
Metrics → Live	what is happening right now
Metrics → Calls	what a call did
Metrics → Approvals	whether the gate is working, or somebody is rubber-stamping
GET /metrics	the same numbers, for Prometheus, plus rates and a latency histogram

Metrics is its own destination in the sidebar. It is not part of Settings: Settings is where a deployment is configured, and none of these numbers is a setting.

Usage and Calls are the pair most easily confused, and they answer different questions on purpose. Usage is about spend: what a window cost, split by day, by source and by model. Calls is about an action: which tool ran, whether it worked, how long it took, and what the gate decided. Both stay.

What a turn cost

Every assistant message carries its own cost in the footer, beside the timestamp:

3.2K in · 412 out · 87% cached · 4.1s

A turn is usually several provider calls — the model asks for a tool, reads the result, and answers — and the figure is the whole turn, summed across all of them.

Two details in that line mean something specific:

• A ~ prefix means part of the total was estimated. Not every provider reports usage. When one does not, nanoinfra counts the tokens with its own tokenizer, and marks the result rather than presenting a guess as a measurement. ~1.8K in · ~240 out is an honest number you should not do arithmetic on to the last token.
• The cache share only appears when the provider reported it. An absent cache metric and a cache that was not hit are different facts. A provider that says nothing about caching gets no percentage, because 0% cached would state something it never said.

The number survives a reload: it is persisted with the turn, so re-opening a thread shows what the live turn showed.

What one step cost

A turn that made 23 calls has one total, and for a while that was all any surface could say: every activity cluster in it read the same 7m 57s, and a 71-second step rendered exactly like a 4-second one.

Each cluster now reports its own span, and the calls inside it report their own cost:

Worked for 47s · 21K in · 96% cached · 1.5K out

The same rules as the turn line, for the same reasons. The cache share is averaged only over the calls that reported one — on the turn above, 3 of the 23 reported no cache metric at all while sitting between neighbours at 99% and 93%, and counting those as zero would have printed a cold cache that never happened.

A cluster with no figures is a cluster whose calls reported none, or a thread recorded before this existed; it reads as it always did.

What a window cost

Metrics → Usage shows a year of daily totals as a heatmap, split by what started the turn:

Source	What it is
user	somebody typed
api	the OpenAI-compatible API
cron	a schedule or a trigger, including the heartbeat
dream	memory consolidation
system	anything internal, and anything unattributed

system rather than user is the default for an unattributed call on purpose: over-counting user would flatter the figure that matters most.

The same five appear as a band above the heatmap, totalled over the window — which is where "what does automation cost me this month" is answered. Before, source reached only the tooltip inside a single heatmap cell, so that question meant opening thirty tooltips and adding them up.

The window control above it — 7, 30, 90 or 365 days — scopes everything below it, including the per-model table and the failure list. It refetches rather than re-slicing: the breakdowns are grouped in SQL and cannot be recomputed in the browser.

Per model

One row per provider and model over the window:

Column	What it is
In / Out	prompt and completion tokens
Cache read / Cache write	priced separately, because they are billed separately
Calls	attempts, with failures after a slash when there were any
Truncated	answers that stopped at the token limit — a length finish, which is neither a success nor an error
TTFT	mean time to first token, over the streamed calls only
Wall clock	mean total duration, which TTFT is not: it includes the wait before the first token
Cost	what it cost in money, or — when the model has no price configured

Truncated is worth knowing about because it used to be counted as a success. An answer cut off mid-sentence is a turn the user has to ask again, and it is the one outcome that looks fine in every total.

A row expands to four measurements that are checks on the ten columns above it:

Expanded	What it is
Reported / estimated	how many of the calls the provider reported usage for, and how many tokens were counted locally instead
Throughput	output tokens per second of generation, which wall clock is not
Measured vs reported out	what the stream actually carried against what the provider said, which is what the token calibration learns from
Streamed / not	the split, and the number of timed calls TTFT above is averaged over

The last one is why TTFT is stated with its denominator: it means nothing for a call that did not stream, and averaging it over every call would report a first-token latency for calls that never had one.

Why calls failed

Under the table, the failures grouped by error kind, HTTP status and provider. The failure count has been on screen for a while; the reason behind it was recorded and never shown, so 16 failed (4%) could not tell a rate limit apart from a bad key.

Whether the gate is working

Metrics → Approvals answers a question about people rather than about tokens: an approval queue with a median of four hours is a queue nobody reads, and an approver who refuses nothing is either gating only safe things or not reading the ask.

Number	What it is
Held for a person	the gate suspended the action and asked
Answered	a person allowed it, on a path other than the one that asked
Refused by a person	a person said no
Expired	the deadline passed and nobody answered
Median answer	how long a person took, from the ask to the answer

Three of those are not the audit log's decision names, and the difference matters if you ever read the log directly. The log's approve is the ask, not the answer — actor is empty on every one of them because nobody has answered yet. The answer is a later allow that names the path a person answered on. And denied holds two different events: a person's refusal, and a policy refusal where nobody was ever asked. The tab counts them apart and shows the second under the first, because merging them claims an approver rejected actions they never saw.

An ask that was neither answered nor expired is called out on its own. Nothing ran, and nothing said why — the most interesting row on the page.

Counts are attributed to the window the gate held the action in, not the window it was answered in, and the tab says so: an approval raised on the 30th and answered on the 31st belongs to the 30th, because "what did this deployment hold for a person" happened then.

This tab carries its own range control rather than sharing the Usage tab's. It reads the gate audit log, whose retention is not llm_calls', and one range over two retentions would mean two different things in the same click.

What a call did

Every tool call writes one row to a tool_calls table in the same database, and Metrics → Calls reads it. A row holds when the call started, which tool, the coarse source above, who the call is attributed to, the capability class the gate keys on, what the gate decided and why, how it ended, how long it took, and — for a failure — a coarse error kind.

The table filters by tool, source, actor, outcome and gate decision, and a row expands to show every field it holds. From an expanded row you can pull up every call in the same turn, or open the conversation it belongs to — offered only when that conversation still exists, because a call row outlives the session it names.

That answers, from one query, what used to take four reads and a correlation by timestamp: who ran execute_on_server yesterday, did it succeed, how long did it take, and who approved it.

Two columns exist here that a plain tool log cannot have:

• gate_decision and gate_reason. A row that reads approved by alberto → allowed is the difference between a log and an audit trail. A deployment with no gate configured still gets rows, and the decision column is empty rather than a fabricated allow — an empty column says the gate did not answer, and allow would say it did.
• outcome, with denied as its own value. A gate refusing an action is the deployment working as configured, not a tool that broke, so a refusal is never counted as an error.

The arguments are addressed, not copied

A row holds session_key, turn_id and seq — the address of the call in the session history that already holds its arguments and its output. It does not hold the arguments. Expanding a row means reading the transcript at that address.

That is deliberate, and it is the same rule the rest of nanoinfra follows: manifests and telemetry carry names and sizes, never content. A metrics database holding every command line would be a second copy of every conversation — including the one with a token in an argument — living somewhere nobody expects it, outside the retention and the compaction that govern the first copy. So there is no column an argument, a path, a file body or an output could be written to, and a test asserts that of the database file rather than of the schema, because a schema can grow a column.

The consequence to know about: a row whose session was compacted or deleted still says what happened and how long it took, and can no longer show you what was passed. That is the trade, and it is the right way round.

The window, and what it took

Tool-call rows are kept for 180 days, against 400 days for the usage rows — one row per call grows much faster than one row per provider attempt. When the window drops rows, it records how many it dropped, because the count is what stops a gap in the history from looking like a quiet week. The Calls tab prints both at the bottom: the retention window, and when the last purge ran and how much it took. Without them an empty page and a purged page read identically, and one of those two is a deployment doing nothing while the other is a deployment whose evidence expired.

What it cost in money

nanoinfra does not ship a price table. Rates change, and a table baked into an agent is wrong the week after it ships — so the rates are yours to state, per model.

Set them where the model is: Settings → Models, open a model, open Pricing. The panel shows what the recorded window would have cost at the rates you typed, before you save, so a rate is checkable against an invoice rather than taken on faith. It also warns when the model reads cached tokens and you have left cache read at 0, because the cost then excludes most of its volume.

For a provider whose models all cost the same — a local one, where that is nothing — Settings → Providers → Default pricing has a free checkbox and sets it once instead of once per model. A model's own rates always win, and the Usage table marks an inherited cost with a *.

In config.json, the same thing:

{
  "pricing": {
    "openai/gpt-4o": {
      "inputPerMtok": 2.5,
      "outputPerMtok": 10.0,
      "cacheReadPerMtok": 1.25,
      "cacheWritePerMtok": 3.125
    }
  }
}

The key is "<provider>/<model>", matching the per-model rows above. All four rates are USD per million tokens, and they are four rather than one for a reason that costs real money: a cached read is billed at a fraction of a fresh prompt token, so a single blended rate applied to the total over-bills a working cache by more than half.

Until a model has a price, the Usage tab says "No prices configured" rather than showing a spend of $0.00, and the per-model Cost column shows —. Zero is a price; a month of real spend rendered as free is worse than a blank.

The same holds for a rate key nanoinfra does not recognise: the entry reads as unpriced rather than as zero, so a typo cannot turn a bill into $0.00. An explicit 0 is a price and means free — the right answer for a local model.

Only priced models contribute to the window total, and the card says how many of the models in the breakdown are priced — so a partial price table reads as partial rather than as a small bill.

What is happening right now

Rows say what happened. They cannot say how many sockets are open, or how many suspended actions are waiting for a person, and those are the numbers an operator watches rather than reviews.

Metrics → Live samples seven of them:

Gauge	What it is
Approvals waiting	suspended actions waiting for a person to answer
WebUI sockets	open WebSocket connections from the WebUI
Active sessions	sessions with recorded history in this workspace
Inbound queue	messages waiting for the agent loop
Outbound queue	replies waiting for a channel
Context used	tokens in the last turn's prompt
Context limit	the active preset's context window

Approvals waiting comes first and is the only one of the seven that means somebody is blocked rather than a level to watch. Inbound and outbound queue depth are the other two worth an alert: a rising inbound number is the agent falling behind, and a rising outbound number is a channel that is not draining.

A dash is not a zero. A gauge whose source could not be read shows —, and that is a different fact from a source that read zero: on the day the approval watcher is down, a dashboard that renders it as 0 waiting is a dashboard that lies about the thing it exists for.

Scraping it

The same sample is served in Prometheus text format at /metrics, on the gateway's own port rather than the WebUI's:

{
  "gateway": {
    "metricsEnabled": true,
    "metricsToken": "a-long-random-string"
  }
}

Off by default, so no deployment gains a scrape surface by upgrading. Two rules on top of that:

• With a token set, a scrape must present it as Authorization: Bearer <token>. A wrong one gets 401. Give Prometheus its own token; it is read by a service out of a config file, and that is not where an admin credential belongs.
• With no token, only a loopback bind is served. "The bind is the authentication" is the Prometheus convention and it holds exactly while the bind is local. A container that sets gateway.host to 0.0.0.0 so a reverse proxy can front the port answers 404 instead of publishing model names, spend volumes and queue depths — and says so at startup rather than leaving you to find out from an empty scrape.

Alongside the gauges it exports counters and a latency histogram, so a scrape can answer a rate and a quantile:

nanoinfra_llm_calls_total{provider,model,outcome}
nanoinfra_llm_tokens_total{provider,model,kind}     kind: input|output|cache_read|cache_write
nanoinfra_tool_calls_total{tool,outcome,gate_decision}
nanoinfra_llm_duration_ms{provider,model}           nine buckets, 500 ms to 300 s

Those are accumulated in memory for the life of the process, not queried. A Prometheus counter must be monotonic, and a count over a table with a purge is not: it falls when the pruner runs, and a rate() over a falling counter is nonsense. A restart resets them, which is what Prometheus expects of a counter and handles with resets().

There is no gate_decisions_total, and it is the one an operator would most want. Gate decisions happen in the executor process, which has no TCP listener by design, so a counter incremented there is invisible to the gateway that serves /metrics — and the gate's own log drops whole expired segments, so counting its records is not monotonic either. The Approvals tab answers that question over the log, as a window rather than as a rate.

No series carries an unbounded label. session_key, turn_id and actor are per-turn values, and a label per turn is how a Prometheus install falls over. Per-call questions are the Calls tab's job.

Reported versus estimated

Every number carries its origin, and the two never mix silently:

• Reported — the provider's own figure. This is what you are billed on.
• Estimated — nanoinfra's tokenizer, used when the provider reported nothing.

A day, a turn or a single call can be part of each; the two halves always sum to the total. The distinction is not cosmetic. nanoinfra/utils/token_calibration.py learns how far the local estimate sits from what a provider actually charged, and it learns only from reported numbers — an estimate corrected against another estimate would teach the correction factor its own output, and drift a little further on every turn a provider happened not to report.

An error is not estimated at all. A failed request cost whatever it cost, and inventing tokens for it would put a made-up number in a cost figure.

What is recorded, and what is not

Usage is recorded once per provider attempt, in a SQLite database at ~/.nanoinfra/llm-usage.sqlite3. A retried call is two rows, which is the only way "how many calls failed and retried, and what did the retry cost" has an answer — a daily total destroys that information rather than omitting it.

A usage row holds: when the call started, how long it took, the provider, the model, the coarse source above, whether it streamed, how it finished, the token counts, time to first token, and — for a failure — a coarse error kind and an HTTP status code.

A usage row holds none of this, by construction rather than by policy:

• no prompts, no responses, no reasoning, no tool arguments or results
• no provider error text — the field most likely to quote your prompt back
• no session keys, chat ids or user identifiers

Finish reasons and error kinds are normalised to a fixed set before they are written: an unrecognised one becomes other rather than being kept verbatim. The database has no column that content could be written to, and a test asserts that.

Bounds: 400 days and 100,000 rows, pruned as rows are written.

The tool_calls table in the same database is the one exception to the third bullet, and the difference is the point. A day's cost is answerable without knowing whose day it was; what one call did is a question about one call in one session, so that table carries the session key, the turn and the position within the turn — and nothing else that a usage row would not carry. It holds no arguments, no output and no error text either, and its window is shorter. See what a call did.

Upgrading from an earlier version

Deployments before this kept daily totals in ~/.nanoinfra/webui/token-usage.json. That file is migrated into the database the first time the gateway starts, and then renamed to token-usage-migrated.json rather than deleted — those numbers were the only copy.

Every day that was visible stays visible, with its totals and its per-source split intact. What was never measured is written as zero: per-attempt rows, cache writes, time to first token, generation time. A zero there says nothing was recorded, which is what happened — those things cannot be reconstructed from a daily sum, and starting a fresh database would say the same thing by deleting the evidence.

What is not here

• Rates you did not state. nanoinfra counts tokens and prices them against the table in your config. It ships no rates of its own, and an unpriced model reads as unpriced rather than as free. See what it cost in money.
• Per-person accounting. The credential store and the usage store belong to the deployment. Two people sharing one share its totals.
• Counters, in the Prometheus sense. See scraping it for why a count from a table with a purge cannot be one.