The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(tokens): bring the coverage expectation table in line with the capture code
Claudeae50bf01bacc
diff --git a/cli/beacon/internal/tokens/coverage.go b/cli/beacon/internal/tokens/coverage.goindex f64475cb..bd1cad8f 100644--- a/cli/beacon/internal/tokens/coverage.go+++ b/cli/beacon/internal/tokens/coverage.go@@ -76,21 +76,42 @@ var usageExpectation = map[string]struct {"pi_cli": {ExpectReported, "extension reports usage and cost"},"omp": {ExpectReported, "extension reports usage and cost"},"prime_agent": {ExpectReported, "extension reports usage and cost"},- // Reported by the managed plugin's `llm_output` hook, which is the only OpenClaw surface that- // carries usage. It is a conversation hook, so OpenClaw withholds it from a non-bundled plugin- // until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set -- which is why- // an OpenClaw endpoint can be collecting everything else and reporting no tokens at all.- // `beacon endpoint hooks status --harness openclaw` is where that gap is surfaced.- "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted"},+ // Senpi's extension forwards the same finalized-message usage object as the rest of the Pi+ // family, through the same mapper (cli/beacon-hooks/cmd/pi_family.go). Without its own key it+ // fell to the unrecognized default and was described as a runtime nobody had looked at.+ "omo_senpi": {ExpectReported, "extension reports usage and cost"},+ // Reported live by the managed plugin's `llm_output` hook, the only OpenClaw hook that carries+ // usage. It is a conversation hook, so OpenClaw withholds it from a non-bundled plugin until+ // `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set -- which is why an+ // OpenClaw endpoint can be collecting everything else live and reporting no tokens at all.+ // `beacon endpoint hooks status --harness openclaw` is where that gap is surfaced. The+ // session-file sync reads the usage OpenClaw commits on each assistant message+ // (openclawsession/records.go), and needs no grant.+ "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted; beacon endpoint integrations openclaw sessions sync reads it from session files"},"vercel_fx": {ExpectReported, "session store carries cumulative usage and cost"},"deepseek_harness": {ExpectReported, "native session backfill reports usage when DeepSeek persists it"},"copilot_cli": {ExpectReported, "session store carries output tokens and cumulative model usage"},"asymptote_observe": {ExpectReported, "SDK spans carry semconv usage"},+ // Runtimes whose usage reaches Beacon only through a session-store sync. Their hooks carry no+ // token counts, so a window of hook activity with no usage is spend missing from the total --+ // silent, which is right -- but the fix is running the sync rather than repairing a hook, so+ // the note names the command.+ //+ // Hermes keeps running per-session totals, cost included, in its state database, and the sync+ // differences them into token.usage deltas (hermessession/collect.go, mapUsage).+ "hermes": {ExpectReported, "beacon endpoint hermes sync reads session totals and cost; hook payloads carry none"},+ // Factory keeps running totals in each session's settings file, and the sync differences them+ // (factorysession/mapper.go, emitSettingsUsage). Its OTLP export still contributes when it+ // carries the semconv names; the sync is the path Beacon reads Factory's own record from.+ "factory": {ExpectReported, "beacon endpoint factory sync reads session settings totals; OTLP only if it emits GenAI semconv usage"},+ "factory_droid": {ExpectReported, "beacon endpoint factory sync reads session settings totals; OTLP only if it emits GenAI semconv usage"},"gemini_cli": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},"vscode_copilot": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},- "factory": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},- "factory_droid": {ExpectGenericOTLP, "only if it emits OTel GenAI semconv usage"},+ // goose's chat spans carry the full semconv usage set, but they reach Beacon only over OTLP+ // configured by hand, and its hook adapter carries none. Without its own key it fell to the+ // unrecognized default.+ "goose": {ExpectGenericOTLP, "chat spans carry GenAI semconv usage, but only over manually configured OTLP"},// Grok Bot runs on a Cursor-hosted cloud computer and reaches Beacon only through Cursor's// server-side OpenTelemetry export, so whether usage arrives depends on whether that export// carries the semconv names -- the generic-OTLP case exactly, not a runtime Beacon reads.@@ -99,7 +120,6 @@ var usageExpectation = map[string]struct {"cursor": {ExpectNone, "no usage on any hook payload; preCompact reports context only"},"antigravity_cli": {ExpectNone, "hook payloads carry no token counts"},"grok": {ExpectNone, "hook payloads carry no token counts"},- "hermes": {ExpectNone, "hook payloads carry no token counts"},// Devin reaches the log under three names, and all three need an entry. The hook installer// writes --platform devin-cli and devin-desktop, an older install still writes plain devin,// and NormalizeHarnessName passes all three through unchanged. Keying only "devin" left thediff --git a/cli/beacon/internal/tokens/coverage_test.go b/cli/beacon/internal/tokens/coverage_test.goindex 58eb9a3d..12fd36ba 100644--- a/cli/beacon/internal/tokens/coverage_test.go+++ b/cli/beacon/internal/tokens/coverage_test.go@@ -185,6 +185,15 @@ func TestRenderCoverageTextExplainsSilentRuntimes(t *testing.T) {if !strings.Contains(out, "silent") || !strings.Contains(out, "beacon endpoint diagnostics") {t.Fatalf("silent report did not tell the reader what to do:\n%s", out)}++ // A runtime whose usage arrives only through a session sync is silent until the sync runs;+ // hook and OTLP advice alone would send that reader to the wrong place.+ sb.Reset()+ RenderCoverageText(&sb, Coverage([]schema.Event{plainEvent("hermes")}, nil))+ out = sb.String()+ if !strings.Contains(out, "beacon endpoint hermes sync") || !strings.Contains(out, "usage arrives only when that sync runs") {+ t.Fatalf("silent report for a sync-sourced runtime did not point at the sync:\n%s", out)+ }}// Every expectation entry must key on a name that survives normalization, or the lookup silently@@ -211,9 +220,9 @@ func TestEveryScannedRuntimeHasAnExpectation(t *testing.T) {// Mirrors the runtime identifiers in internal/endpoint/inventory/inventory.go.scanned := []string{"antigravity_cli", "claude_code", "cline", "codex_cli", "copilot_cli", "cursor",- "devin-cli", "devin-desktop", "factory", "gemini_cli", "grok", "hermes",- "kimi_code", "kiro", "muse_code", "omp", "opencode", "openhands", "pi_cli", "qwen_code",- "vscode",+ "deepseek_harness", "devin-cli", "devin-desktop", "factory", "gemini_cli", "grok", "hermes",+ "kimi_code", "kiro", "muse_code", "omo_senpi", "omp", "openclaw_gateway", "opencode",+ "openhands", "pi_cli", "prime_agent", "qwen_code", "vscode",}for _, runtime := range scanned {harness := normalizedHarnessForTest(runtime)@@ -381,3 +390,93 @@ func TestCoverageDoesNotCountContextOnlyEventsAsUsage(t *testing.T) {t.Errorf("covered = %d, want 0", report.Covered)}}++// pollUsageEvent is a token.usage event shaped like the ones the session-store sync commands+// write: collected by poll, carrying a session and canonical usage.+func pollUsageEvent(harness, session string, input, output int64, mutate func(*schema.Event)) schema.Event {+ event := usageEventFixture("2026-06-11T10:05:00Z", harness, session, "claude-opus-4-7", func(e *schema.Event) {+ e.Harness.CollectionMethod = schema.CollectionMethodPoll+ e.GenAI.Usage.InputTokens = int64Ptr(input)+ e.GenAI.Usage.OutputTokens = int64Ptr(output)+ })+ if mutate != nil {+ mutate(&event)+ }+ return event+}++// hookEvent is what a runtime's live hooks write: activity, never usage.+func hookEvent(harness, session string) schema.Event {+ event := plainEvent(harness)+ event.Harness.CollectionMethod = schema.CollectionMethodHook+ event.Session = &schema.SessionInfo{ID: session}+ return event+}++// Some runtimes report usage only through their session-store sync: Hermes Agent's hooks carry+// no token counts, but `beacon endpoint hermes sync` differences the session totals Hermes keeps+// in its state database, cost included; Factory Droid's usage lives in each session's settings+// file, which `beacon endpoint factory sync` reads. Both are runtimes Beacon is built to read+// usage from, so a window with their hook activity and no usage is spend missing from the total+// -- silent, with a note that names the sync -- and their poll-collected usage covers them.+func TestCoverageCountsPollSourcedUsage(t *testing.T) {+ for _, tc := range []struct {+ harness, command string+ }{+ {"hermes", "beacon endpoint hermes sync"},+ {"factory", "beacon endpoint factory sync"},+ } {+ silent := lineFor(t, Coverage([]schema.Event{hookEvent(tc.harness, "s1")}, []string{tc.harness}), tc.harness)+ if silent.Status != CoverageSilent || silent.Expectation != ExpectReported {+ t.Errorf("%s hooks only = %s/%s, want silent/reported -- its usage is read by %s",+ tc.harness, silent.Status, silent.Expectation, tc.command)+ }+ if !strings.Contains(silent.Reason, tc.command) {+ t.Errorf("%s reason = %q, want it to name %q, the only path its usage arrives on",+ tc.harness, silent.Reason, tc.command)+ }++ events := []schema.Event{+ hookEvent(tc.harness, "s1"),+ pollUsageEvent(tc.harness, "s1", 400, 75, func(e *schema.Event) {+ if tc.harness == "hermes" {+ e.GenAI.Usage.CostUSD = float64Ptr(0.25)+ }+ }),+ }+ covered := lineFor(t, Coverage(events, []string{tc.harness}), tc.harness)+ if covered.Status != CoverageCovered || covered.UsageEvents != 1 || covered.Tokens != 475 {+ t.Errorf("%s with synced usage = %+v, want covered with one usage event and 475 tokens", tc.harness, covered)+ }+ }+}++// Every runtime Beacon reads usage from on its own surface names that surface in the note, and+// none falls through to the unrecognized-runtime default, whose note would tell a reader that+// nobody has looked at the runtime.+func TestCoverageExpectationsMatchTheCaptureCode(t *testing.T) {+ for _, tc := range []struct {+ harness, expect, reasonHas string+ }{+ // The Senpi extension forwards message_end usage and cost through the shared Pi-family+ // mapper (cli/beacon-hooks/cmd/pi_family.go), exactly as Pi, Oh My Pi and Prime do.+ {"omo_senpi", ExpectReported, "extension"},+ // The OpenClaw session-file sync reads per-message usage from committed sessions, which+ // does not depend on the plugin's conversation-access grant.+ {"openclaw_gateway", ExpectReported, "beacon endpoint integrations openclaw sessions sync"},+ // goose's chat spans carry the semconv names, but only over OTLP configured by hand.+ {"goose", ExpectGenericOTLP, "OTLP"},+ {"factory_droid", ExpectReported, "beacon endpoint factory sync"},+ } {+ expect, reason := expectationFor(tc.harness)+ if expect != tc.expect {+ t.Errorf("%s expectation = %q, want %q", tc.harness, expect, tc.expect)+ }+ if !strings.Contains(reason, tc.reasonHas) {+ t.Errorf("%s reason = %q, want it to mention %q", tc.harness, reason, tc.reasonHas)+ }+ if _, ok := usageExpectation[tc.harness]; !ok {+ t.Errorf("%s has no usageExpectation entry and falls to the unrecognized default", tc.harness)+ }+ }+}diff --git a/cli/beacon/internal/tokens/report.go b/cli/beacon/internal/tokens/report.goindex b2325f99..0794c9cd 100644--- a/cli/beacon/internal/tokens/report.go+++ b/cli/beacon/internal/tokens/report.go@@ -151,5 +151,6 @@ func RenderCoverageText(w io.Writer, report CoverageReport) {fmt.Fprintln(w, "\nSilent runtimes produced events but no token usage, and Beacon is built to read")fmt.Fprintln(w, "usage from them. Check that the runtime is current and its hooks or OTLP export")fmt.Fprintln(w, "are still configured: beacon endpoint status, beacon endpoint diagnostics.")+ fmt.Fprintln(w, "Where the note names a sync command, usage arrives only when that sync runs.")}}diff --git a/docs/cli/token-usage.mdx b/docs/cli/token-usage.mdxindex bf2e42b2..470e44a6 100644--- a/docs/cli/token-usage.mdx+++ b/docs/cli/token-usage.mdx@@ -47,12 +47,17 @@ from. A turn with no token attribution is a turn whose runtime never reported an| Pi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes || Oh My Pi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes || Prime Agent | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes |+| Senpi | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) | Yes | Yes |+| Hermes Agent | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`): Hermes's actual cost, or its own estimate when it recorded no actual | Yes, from `beacon endpoint hermes sync` | Yes |+| Factory Droid | input, output, cache read, cache creation, reasoning | Not reported in USD; Factory credits stay in `raw` | Yes, from `beacon endpoint factory sync` | Yes || DeepSeek Harness | input, output, cache read, cache creation | Not reported | Yes, from committed session records | Yes || fx (Vercel Labs) | input, output, cache read, cache creation | Reported (`cost_usd`) | Yes | Yes || GitHub Copilot CLI | input, output, cache read, cache creation, reasoning | Not reported | Yes | Yes || OpenClaw Gateway | input, output, cache read, cache creation | Not reported on the hook | Yes, per model response | Yes |-OpenClaw reports usage on its `llm_output` hook, which is a *conversation* hook -- OpenClaw withholds those from every non-bundled plugin until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set in its config. An OpenClaw endpoint collecting prompts, tools, commands and files while reporting no tokens at all is almost always that key, and `beacon endpoint hooks status --harness openclaw` says so. OpenClaw does compute a per-turn price, from an operator-configured cost table rather than from a provider report, so Beacon does not record it as a reported cost. See [OpenClaw Gateway](/runtimes/openclaw-gateway).+OpenClaw reports usage on its `llm_output` hook, which is a *conversation* hook -- OpenClaw withholds those from every non-bundled plugin until `plugins.entries.beacon-endpoint.hooks.allowConversationAccess` is set in its config. An OpenClaw endpoint collecting prompts, tools, commands and files while reporting no tokens at all is almost always that key, and `beacon endpoint hooks status --harness openclaw` says so. OpenClaw does compute a per-turn price, from an operator-configured cost table rather than from a provider report, so Beacon does not record it as a reported cost. The session-file sync, `beacon endpoint integrations openclaw sessions sync`, reads the usage OpenClaw commits on each assistant message and needs no grant. See [OpenClaw Gateway](/runtimes/openclaw-gateway).++Hermes Agent and Factory Droid report usage only through their session-store sync: neither runtime's hooks carry a token count, so their usage arrives when `beacon endpoint hermes sync` or `beacon endpoint factory sync` runs. Both keep running session totals, which the sync differences into deltas. Until the sync runs, the coverage report lists them as `silent`, with a note naming the command.Cline reports usage once per task rather than per model call, because Beacon's rollups sum `gen_ai.usage` across events and reporting at both granularities would count every token twice. fx reports cumulative totals, which Beacon differences into per-turn deltas for the same reason. GitHub Copilot CLI reports assistant message output tokens and cumulative shutdown model totals; Beacon records the message output directly, then differences shutdown totals for input, cache, reasoning, and any output not already counted.@@ -62,7 +67,6 @@ Cline reports usage once per task rather than per model call, because Beacon's r|---------|--------|| Gemini CLI | Usage is captured when the runtime emits the OTel GenAI semconv names (`gen_ai.usage.*` attributes, or a `gen_ai.client.token.usage` metric split by `token_type`). The generic path is tested; these runtimes' own payloads have not been captured and pinned. || VS Code Copilot | As above |-| Factory Droid | As above || Grok Bot | Reaches Beacon only through Cursor's server-side OpenTelemetry export, so usage arrives only if that export carries the semconv names || goose | goose's `chat` spans carry the full `gen_ai.usage.*` set, cache reads and writes included, under the exact semconv names -- that much is read out of the runtime rather than hoped for. What is conditional is whether it arrives: usage reaches Beacon only over manually configured OTLP, so a hooks-only goose setup is correctly silent rather than faulty. goose reports no cost. See [goose](/runtimes/goose) |@@ -89,7 +93,7 @@ A runtime that reports occupancy and no spend still shows as `not_instrumented`| Runtime | Why ||---------|-----|| Cursor | No hook payload carries a token count. Cursor does report context occupancy on `preCompact`, which is recorded separately -- see [Context occupancy](#context-occupancy) below |-| Antigravity CLI, Grok Build, Hermes Agent, Devin CLI, Devin Desktop | No token counts on the hook payloads Beacon receives |+| Antigravity CLI, Grok Build, Devin CLI, Devin Desktop | No token counts on the hook payloads Beacon receives || Muse Code | Usage arrives on `PostLLMCall` and in Muse's own session log, neither of which Beacon reads. See [Muse Code](/runtimes/muse-code) || Qwen Code | The `Stop` payload carries session-level context counters, not per-call usage. Those counters are recorded as context occupancy rather than spend -- see [Context occupancy](#context-occupancy) below. See [Qwen Code](/runtimes/qwen-code) || OpenHands | None of the six hook payloads carries token counts, and OpenHands exposes no other hook that would. See [OpenHands](/runtimes/openhands) |