The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(tokens): bring the coverage expectation table in line with the capture code (#734)
Justin D'Souza0f93deec6ec3
diff --cc cli/beacon/internal/tokens/coverage.goindex 05d15429,daaddcd3..bbfe219b--- a/cli/beacon/internal/tokens/coverage.go+++ b/cli/beacon/internal/tokens/coverage.go@@@ -76,16 -76,38 +76,38 @@@ var usageExpectation = map[string]struc"pi_cli": {ExpectReported, "extension reports usage and cost"},"omp": {ExpectReported, "extension reports usage and cost"},"prime_agent": {ExpectReported, "extension reports usage and cost"},- // Reported by the managed plugin's `llm_output` hook, which is the only OpenClaw surface that- // carries usage. It is a conversation hook, so OpenClaw withholds it from a non-bundled plugin- // until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set -- which is why- // an OpenClaw endpoint can be collecting everything else and reporting no tokens at all.- // `beacon endpoint hooks status --harness openclaw` is where that gap is surfaced.- "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted"},+ // Senpi's extension forwards the same finalized-message usage object as the rest of the Pi+ // family, through the same mapper (cli/beacon-hooks/cmd/pi_family.go). Without its own key it+ // fell to the unrecognized default and was described as a runtime nobody had looked at.+ "omo_senpi": {ExpectReported, "extension reports usage and cost"},+ // Reported live by the managed plugin's `llm_output` hook, the only OpenClaw hook that carries+ // usage. It is a conversation hook, so OpenClaw withholds it from a non-bundled plugin until+ // `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set -- which is why an+ // OpenClaw endpoint can be collecting everything else live and reporting no tokens at all.+ // `beacon endpoint hooks status --harness openclaw` is where that gap is surfaced. The+ // session-file sync reads the usage OpenClaw commits on each assistant message+ // (openclawsession/records.go), and needs no grant.+ "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted; beacon endpoint integrations openclaw sessions sync reads it from session files"},"vercel_fx": {ExpectReported, "session store carries cumulative usage and cost"},"deepseek_harness": {ExpectReported, "native session backfill reports usage when DeepSeek persists it"},- "copilot_cli": {ExpectReported, "session store carries output tokens and cumulative model usage"},+ "copilot_cli": {ExpectReported, "session store carries output tokens, cumulative model usage, and AI-credit cost"},"asymptote_observe": {ExpectReported, "SDK spans carry semconv usage"},+ // Runtimes whose usage reaches Beacon only through a session-store sync. Their hooks carry no+ // token counts, so a window of hook activity with no usage is spend missing from the total --+ // silent, which is right -- but the fix is running the sync rather than repairing a hook, so+ // the note names the command.+ //+ // Hermes keeps running per-session totals, cost included, in its state database, and the sync+ // differences them into token.usage deltas (hermessession/collect.go, mapUsage).+ "hermes": {ExpectReported, "beacon endpoint hermes sync reads session totals and cost; hook payloads carry none"},+ // Factory keeps running totals in each session's settings file, and the sync differences them+ // (factorysession/mapper.go, emitSettingsUsage). Its OTLP export still contributes when it+ // carries the semconv names; the sync is the path Beacon reads Factory's own record from.+ "factory": {ExpectReported, "beacon endpoint factory sync reads session settings totals; OTLP only if it emits GenAI semconv usage"},+ // The factory_droid spelling never comes from the sync, which writes harness "factory" and is+ // not normalized onto this name. It is what a Factory OTLP export can name itself, so it keeps+ // the generic-OTLP expectation rather than pointing at a command that cannot clear it.+ "factory_droid": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},"gemini_cli": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},"vscode_copilot": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},diff --cc docs/cli/token-usage.mdxindex 512e3f82,470e44a6..cf32ed06--- a/docs/cli/token-usage.mdx+++ b/docs/cli/token-usage.mdx@@@ -47,14 -47,19 +47,19 @@@ from. A turn with no token attribution| Pi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes || Oh My Pi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes || Prime Agent | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes |+ | Senpi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes |+ | Hermes Agent | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`): Hermes's actual cost, or its own estimate when it recorded no actual | Yes, from `beacon endpoint hermes sync` | Yes |+ | Factory Droid | input, output, cache read, cache creation, reasoning | Not reported in USD; Factory credits stay in `raw` | Yes, from `beacon endpoint factory sync` | Yes || DeepSeek Harness | input, output, cache read, cache creation | Not reported | Yes, from committed session records | Yes || fx (Vercel Labs) | input, output, cache read, cache creation | Reported (`cost_usd`) | Yes | Yes |-| GitHub Copilot CLI | input, output, cache read, cache creation, reasoning | Not reported | Yes | Yes |+| GitHub Copilot CLI | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) for sessions started on or after 2026-06-01, from AI credits | Yes | Yes || OpenClaw Gateway | input, output, cache read, cache creation | Not reported on the hook | Yes, per model response | Yes |- OpenClaw reports usage on its `llm_output` hook, which is a *conversation* hook -- OpenClaw withholds those from every non-bundled plugin until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set in its config. An OpenClaw endpoint collecting prompts, tools, commands and files while reporting no tokens at all is almost always that key, and `beacon endpoint hooks status --harness openclaw` says so. OpenClaw does compute a per-turn price, from an operator-configured cost table rather than from a provider report, so Beacon does not record it as a reported cost. See [OpenClaw Gateway](/runtimes/openclaw-gateway).+ OpenClaw reports usage on its `llm_output` hook, which is a *conversation* hook -- OpenClaw withholds those from every non-bundled plugin until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set in its config. An OpenClaw endpoint collecting prompts, tools, commands and files while reporting no tokens at all is almost always that key, and `beacon endpoint hooks status --harness openclaw` says so. OpenClaw does compute a per-turn price, from an operator-configured cost table rather than from a provider report, so Beacon does not record it as a reported cost. The session-file sync, `beacon endpoint integrations openclaw sessions sync`, reads the usage OpenClaw commits on each assistant message and needs no grant. See [OpenClaw Gateway](/runtimes/openclaw-gateway).++ Hermes Agent and Factory Droid report usage only through their session-store sync: neither runtime's hooks carry a token count, so their usage arrives when `beacon endpoint hermes sync` or `beacon endpoint factory sync` runs. Both keep running session totals, which the sync differences into deltas. Until the sync runs, the coverage report lists them as `silent`, with a note naming the command.-Cline reports usage once per task rather than per model call, because Beacon's rollups sum `gen_ai.usage` across events and reporting at both granularities would count every token twice. fx reports cumulative totals, which Beacon differences into per-turn deltas for the same reason. GitHub Copilot CLI reports assistant message output tokens and cumulative shutdown model totals; Beacon records the message output directly, then differences shutdown totals for input, cache, reasoning, and any output not already counted.+Cline reports usage once per task rather than per model call, because Beacon's rollups sum `gen_ai.usage` across events and reporting at both granularities would count every token twice. fx reports cumulative totals, which Beacon differences into per-turn deltas for the same reason. GitHub Copilot CLI reports assistant message output tokens and cumulative shutdown model totals; Beacon records the message output directly, then differences shutdown totals for input, cache, reasoning, and any output not already counted. Cost comes from the shutdown's session-wide `totalNanoAiu`, differenced the same way and converted at one AI credit = 10^9 nano-AIU = US$0.01. It is recorded once per shutdown: on the model's usage event when the shutdown names one model, otherwise on one usage event with no model, so a multi-model session is not counted once per model. Sessions started before GitHub moved Copilot to AI credits on 2026-06-01 were billed per premium request, so they record no cost; the raw nano-AIU total stays in `raw.copilot`. Annual Copilot Pro and Pro+ subscribers stay on premium-request billing until their plan renews, and the session record does not say which plan applies, so for them the recorded cost is what the session would cost in AI credits rather than their bill.**Collected through the generic OpenTelemetry path, not yet verified against captured payloads:**