The token metrics pipeline has three stages:
collect-token-metrics.ts)Runs every 97 minutes (a jeeves-runner schedule). Scans two data sources:
~/.openclaw/agents/main/sessions/~/.claude/projects/For each file, the collector:
{hour, channel, model}.If the collector encounters a model not in the rate card, it refuses to write any buckets and triggers a rate card refresh job.
If SESSIONS_DIR is missing or holds no transcript files (OpenClaw 2026.9+ moved transcripts into its agent SQLite store), the collector logs a warning, skips OpenClaw sources, keeps collecting Claude Code, and exits 0.
Each bucket file is a JSON file at:
/base/state/jeeves-runner/token-metrics/{YYYY}/{MM}/{YYYY-MM-DDTHH}.json
Schema:
{
"hour": "2026-06-01T14",
"channels": {
"slack:channel:C12345": {
"models": {
"anthropic/claude-sonnet-4-6": {
"input": { "count": 50000, "cost": 0.15 },
"output": { "count": 12000, "cost": 0.06 },
"cacheRead": { "count": 80000, "cost": 0.024 },
"cacheWrite": { "count": 5000, "cost": 0.01875 }
}
}
}
}
}
Bucket files are immutable once the hour closes — the collector only writes to completed hours. The flushBuckets function deep-merges new data into existing bucket files (additive, never replaces).
getTokenMetrics, admin-report-token-metrics)Reads bucket files for a time range and aggregates into a Costs report with:
Run the report (JSON on stdout; admin.md):
node "$JEEVES_SCRIPTS_DIR/bin/jeeves-scripts.js" run admin-report-token-metrics --from 2026-06-01 --to 2026-06-03
When ~/.openclaw/agents/main/agent/openclaw-agent.sqlite exists, the collector reads OpenClaw usage from it instead of SESSIONS_DIR:
PRAGMA user_version, currently 23 and 24; 24, from OpenClaw 2026.9.33, only adds fields token metrics doesn't read, so it reuses the schema-23 reader). Any other version makes the collector exit non-zero with expected user_version 23 or 24, found N. It never guesses.src/admin/lib/openclaw-db/schema-v23.ts (payload decoding in schema-v23-payloads.ts): hot transcript_events rows (event_json or checksummed zstd event_zstd), cold archives (sessions/cold/*.jsonl.zst or blob, sha256-verified) and deleted/reset archives (sha256-verified). Legacy *.jsonl.reset.* / *.jsonl.deleted.* files left in SESSIONS_DIR by the migration are read too (legacy-archives.ts).session_transcript_archives is keyed (session_id, generation), seq restarting at 0) is its own transcript with its own cursor (session:<id>#<generation>); generations are never merged by seq. An archive continues from the live session:<id> cursor it was archived from (matched by the generation stamped in that cursor, from transcript_rewrite_watermarks), so usage already counted live is not counted again. A fully read archive (or legacy archive file) is marked complete in the cursor and not re-read on later runs.node:sqlite) is imported only when the agent DB exists. Core requires Node >=24.22.2, which always has node:sqlite.channel-from-meta.ts, names loaded by schema-v23-meta.ts): the session key (session_windows, or session_transcript_archives for deleted/reset sessions), plus names from session_nodes (groupChannel, displayName, delivery.origin.label, label, parent session) and conversations. This gives slack:channel:#name (or the upper-case channel id when no name is recorded), slack:dm:<person>, cron:<label>, telegram:<kind>:<name> and main (an agent's main session, agent:<id>:main).subagent-rollup.ts). The parent chain (session_nodes.parent_session_key / spawnedBy) is followed transitively to the root: a Slack channel/DM, Telegram or cron root gets the subagent's usage. Under a non-channel root (agent:main:main, recovered; the main session's own turns are main, its subagents are not) the topmost subagent names the bucket: runner workers (label worker-<job>, the first 8 characters of the job id) become runner:<job>, other subagents keep subagent:label:<label> (meta-<phase> for meta labels). If the chain can't be resolved (no linkage, a deleted subagent parent, a cycle), the subagent keeps its own label-based name. A deleted Slack parent still resolves from its session key.detectChannel text rules (first 50 events) are used only when metadata is absent or doesn't identify the channel, e.g. recovered sessions and legacy archive files (no longer agent:main:main, which is main). 2026.9 injects runtime-context text into transcripts, which those rules used to misread (slack:dm:<name>-approved-executables-…), so the DM rule now stops at the end of the line. Every key has its whitespace collapsed and trailing punctuation stripped (no more subagent:repo:…/jeeves-tools.).slack:dm:<USERID>) are named from slack-dm-names.json (beside the buckets), then the Slack user cache ({stateDir}/slack/users.json, read-only), then the gateway message tool (member-info); a looked-up name is cached. If nothing resolves the id key is kept.(transcript, seq), stored in runner state under cursors-openclaw-db. Usage in the still-open hour stops that transcript's cursor, so it's counted on a later run and never dropped. Nothing is counted twice.fresh-openclaw-history.ts). It was if the legacy JSONL cursor (cursors) has an entry, or if any bucket file (or .backup-* copy) under {YYYY}/{MM}/ has a channel other than a Claude Code cc: channel. A bucket that can't be read, or has no channels object, counts too, to stay safe. The check stops at the first such bucket, and runs only when there's no DB cursor, so a host that already has one never scans its buckets.
run admin-regenerate-token-metrics --from <upgrade hour> (below). Claude Code collection still runs.# this instance's OpenClaw 2026.9 upgrade hour (required for live runs; no default)
# PowerShell: $env:OPENCLAW_UPGRADE_CUTOFF = '<YYYY-MM-DDTHH:00:00Z>'
export OPENCLAW_UPGRADE_CUTOFF=<YYYY-MM-DDTHH:00:00Z>
# scratch (never touches runner state or the live store; point TOKEN_METRICS_DIR at a dir holding a copied token-rates.json)
node bin/jeeves-scripts.js run admin-regenerate-token-metrics --from <ISO hour> --out /tmp/regen
# live: rebuild [from, last closed hour), back up + replace buckets, REPLACE the DB cursor (bootstrap / full rebuild)
node bin/jeeves-scripts.js run admin-regenerate-token-metrics --from "$OPENCLAW_UPGRADE_CUTOFF" --dry-run
node bin/jeeves-scripts.js run admin-regenerate-token-metrics --from "$OPENCLAW_UPGRADE_CUTOFF"
# live: rebuild a closed range from already-counted events (OpenClaw seq <= DB cursor,
# Claude Code bytes before the CC cursor); DB and CC cursors untouched
node bin/jeeves-scripts.js run admin-regenerate-token-metrics --from <ISO hour> --to <ISO hour>
Pause the admin-collect-token-metrics job while a live regeneration runs.
OPENCLAW_UPGRADE_CUTOFF is a per-instance environment variable with no default: the first UTC hour this instance ran OpenClaw 2026.9+ (ISO 8601). An instance that never ran an earlier OpenClaw uses its first hour of usage. It is read only by live regenerations (including --dry-run):
--allow-pre-upgrade.--from earlier than it: refused, because pre-upgrade hours were counted by the JSONL collector and are never rewritten. --allow-pre-upgrade overrides this check only; use it only with owner approval.--out) cannot rewrite history, so they ignore the cutoff entirely: no variable and no --allow-pre-upgrade are needed to scan pre-upgrade hours into a scratch directory. That holds because an --out that resolves to TOKEN_METRICS_DIR (the same path, or reached through a symlink) is refused before anything is scanned, and an --out already holding buckets in the range is refused too.The hourly collector and the reports never read it.
This is only for hosts upgraded to 2026.9 that already have token-metrics history. A fresh instance starts its DB cursor automatically (above).
schema_meta.updated_at for meta_key = 'primary', or openclaw.json.pre-* backups.--from = the upgrade hour (floor to the hour), and set the OPENCLAW_UPGRADE_CUTOFF environment variable to it. The JSONL collector wrote every bucket before that hour.--out) from a few days before the upgrade, and compare pre-upgrade days with the live store by day × channel × model. OpenClaw deletes transcripts, and the JSONL collector dropped open-hour usage, so pre-upgrade days can legitimately differ. Understand every difference before continuing.When OpenClaw bumps the agent schema, the collector exits non-zero naming the expected and found versions. Then:
src/admin/lib/openclaw-db/schema-vNN.ts beside schema-v23.ts (don't edit v23). Implement the same OpenClawDbSchema contract and register it in open-agent-db.ts.Use admin-regenerate-token-metrics (above) when bucket data needs correction (e.g. after a rate card fix, a collector bug or corrupted bucket files): dry run first, then a scratch --out run to compare, then live. It backs up the buckets it replaces (.backup-{timestamp}.json) and never touches hours outside the range.
Core has no rebuild for hosts still on legacy JSONL transcripts only (pre-2026.9, no agent DB); that one-off recalculation was instance remediation and is not shipped (Decision 36).
admin-regenerate-token-metrics instead.pruneAfter / maxEntries Interact with Data RetentionThe bucket file layout is partitioned by {YYYY}/{MM}/. There is no automatic pruning built into the collector or query layer — bucket files persist indefinitely unless manually removed.
If you need to implement data retention:
rm -rf /base/state/jeeves-runner/token-metrics/2025/) to free disk space. This is safe because the collector never writes to hours in the past.The rate card lives at /base/state/jeeves-runner/token-metrics/token-rates.json.
The template ships a seed card at config/token-rates.seed.json (default instance models plus delivery-mirror at 0, source starts with SEED <date>). admin-refresh-token-rates and admin-collect-token-metrics copy it into place (creating the directory) only when no rate card exists. An existing card is never overwritten, even if it is invalid. Seeding is atomic: the validated seed is written to a temp file in the same directory and hard-linked into place, so readers never see a partial card and a concurrently created card is never clobbered.
admin-refresh-token-rates validates the card before fetching and again after writing. It exits non-zero (the runner records an error) if:
manual entries, below), or any pending model, can't be priced: OpenRouter returns 404, the request fails, or the fetch time budget runs out. Everything that could be priced is still written first, so one bad model never blocks the rest;admin-refresh-token-rates --dry-run fetches and prints the changes without seeding, writing the card, or touching the pending file.
The refresh-token-rates.ts job runs daily at 05:37 UTC (timeout_seconds 120). It is a plain script with no LLM session: for each model on the card it reads base-tier prices from the public OpenRouter model endpoint, GET https://openrouter.ai/api/v1/model/<provider>/<model> (no auth), converts per-token USD to $/MTok, and writes any changed rates atomically in one write (advancing updatedAt and appending a dated note to source).
overrides (higher long-context prices) and input_cache_write_1h are ignored; Anthropic input_cache_write is the 5-minute tier, which is what the card stores. A missing cache price means the provider has none and is stored as 0.anthropic/claude-opus-5-5 returns anthropic/claude-opus-5.5), but some providers have a different prefix there: xai/ is sent as x-ai/ (OPENROUTER_PROVIDER_ALIASES in lib/openrouter-pricing.ts). A model OpenRouter doesn't list fails the run: remove its card entry once no session uses it, or, while sessions still use it, mark it "manual": true (see Manual).openclaw/… and clawdbot/… routing entries (e.g. delivery-mirror) stay at 0."manual": true is never looked up, so its hand-maintained prices are neither overwritten nor reported as a failure.If the collector encounters an unknown model, it:
token-rates.pending.json next to the card (a JSON array of ids).admin-refresh-token-rates fetches each pending id from OpenRouter and adds it to the card once valid prices come back. After writing the card, the refresh re-reads the pending file and drops only the ids now on the card, so ids the collector adds while a refresh is fetching prices are kept. The file isn't locked, so an id added in the brief moment between that re-read and the write can be dropped; the collector records it again on its next run. Ids OpenRouter doesn't know stay in it and fail the run until someone adds them by hand, usually as a manual entry (below). The next collector run (97 minutes later) picks up the updated card and processes the held data.
Edit token-rates.json directly. The schema:
{
"updatedAt": "2026-06-01T00:00:00Z",
"source": "manual",
"unit": "$/MTok",
"models": {
"anthropic/claude-sonnet-4-6": {
"input": 3.0,
"output": 15.0,
"cacheRead": 0.3,
"cacheWrite": 3.75
}
}
}
Rates are in dollars per million tokens ($/MTok). All four categories (input, output, cacheRead, cacheWrite) are required for each model. The model key format is {provider}/{model} matching what appears in transcript data.
Add "manual": true to an entry to maintain its prices by hand. admin-refresh-token-rates then skips it: it never looks the model up on OpenRouter, never overwrites its rates and never fails because OpenRouter doesn't list it. Use this for a model OpenRouter doesn't carry (it otherwise fails every refresh), or to pin a negotiated price. Without the flag, a hand-added entry is refreshed from OpenRouter like any other.
"vendor/private-model": {
"input": 1.0,
"output": 4.0,
"cacheRead": 0.1,
"cacheWrite": 0,
"manual": true
}
After adding a model manually, restart the collector or wait for its next scheduled run. If the model is listed in token-rates.pending.json, the next refresh run clears it once it finds the model on the card.