The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
Count a runtime as installed only where Beacon configured it
Claudecbbc2060f5a7
diff --git a/cli/beacon/cmd/token_usage.go b/cli/beacon/cmd/token_usage.goindex e84cb026..d7de8ad0 100644--- a/cli/beacon/cmd/token_usage.go+++ b/cli/beacon/cmd/token_usage.go@@ -131,10 +131,10 @@ func runTokenCoverage(cmd *cobra.Command, logPath string, query dashboard.EventQ}events = scopedEvents}- // Installed runtimes come from the config scanner rather than from the log, because the- // whole question is which installed runtime is missing from the log. tokens.InstalledRuntimes- // decides which scanner rows are real evidence of an install; the scanner reports files a- // runtime might read, which is not the same thing.+ // Installed runtimes come from the config scanner rather than from the log, because the whole+ // question is which configured runtime is missing from the log. tokens.InstalledRuntimes keeps+ // only the rows Beacon itself wired up; the scanner reports files a runtime might read, which+ // is a different and much larger set.configs := make([]tokens.InstalledConfig, 0)for _, config := range endpointinventory.Scan(endpointinventory.Options{}).Configs {configs = append(configs, tokens.InstalledConfig{diff --git a/cli/beacon/cmd/token_usage_test.go b/cli/beacon/cmd/token_usage_test.goindex 55261f31..9d690723 100644--- a/cli/beacon/cmd/token_usage_test.go+++ b/cli/beacon/cmd/token_usage_test.go@@ -8,7 +8,6 @@ import ("strings""testing"- endpointinventory "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/endpoint/inventory""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/tokens")@@ -227,16 +226,6 @@ func TestTokenUsageCoverageDoesNotTreatAShellProfileAsAnInstall(t *testing.T) {}}-// tokens.ConfigKindProfile duplicates inventory.KindProfile so the tokens package stays-// independent of the scanner. Pinned here, where both are importable, because a silent drift-// would stop excluding shell profiles and quietly refill the report with false installs.-func TestTokenCoverageProfileKindConstantMatchesInventory(t *testing.T) {- if tokens.ConfigKindProfile != endpointinventory.KindProfile {- t.Fatalf("tokens.ConfigKindProfile = %q, inventory.KindProfile = %q; they must agree",- tokens.ConfigKindProfile, endpointinventory.KindProfile)- }-}-// A harness scope narrows the events, so it has to narrow the installed list with them. Without// that, every other installed runtime has no events in the filtered set and is reported inactive// -- which reads as "installed but unused this window" when the truth is only that the readerdiff --git a/cli/beacon/internal/tokens/coverage.go b/cli/beacon/internal/tokens/coverage.goindex 8aac5983..85cd9df3 100644--- a/cli/beacon/internal/tokens/coverage.go+++ b/cli/beacon/internal/tokens/coverage.go@@ -1,7 +1,6 @@package tokensimport (- "path/filepath""sort""strings"@@ -109,21 +108,6 @@ var usageExpectation = map[string]struct {"openhands": {ExpectNone, "hook payloads carry no token counts"},}-// ConfigKindProfile mirrors inventory.KindProfile. It is duplicated rather than imported to keep-// this package independent of the scanner; cmd pins the two equal in a test.-const ConfigKindProfile = "profile"--// sharedConfigBasenames are config files that more than one runtime reads, which the scanner-// nonetheless attributes to a single runtime.-//-// `.mcp.json` is the case: a workspace MCP server list that Claude Code, fx and others all read,-// which the scanner files under fx alone -- deliberately, because attributing one file to several-// runtimes would report one server several times. That is right for an inventory of MCP servers-// and wrong as evidence of an install: a repository whose .mcp.json exists for Claude Code would-// otherwise report fx as installed, and fx would sit in every coverage report as a permanently-// inactive runtime nobody has.-var sharedConfigBasenames = map[string]bool{".mcp.json": true}-// InstalledConfig is the part of a config-scanner row that install detection reads.type InstalledConfig struct {Runtime string@@ -133,35 +117,35 @@ type InstalledConfig struct {Exists bool}-// InstalledRuntimes selects the scanner rows that are real evidence a runtime is installed.+// InstalledRuntimes selects the scanner rows that are evidence a runtime is wired up to report.//-// The scanner answers "which config files exist that some runtime might read", which is a-// different question from "is this runtime installed", and the gap between them is all-// false positives -- each one a permanently inactive row for a product nobody has. A status that-// is wrong on most machines is one people learn to skip, which costs more than the rows are worth.+// Only rows Beacon itself configured count. Everything else -- a config file that merely exists --+// is not evidence of anything this report can use, and three rounds of review found three separate+// ways it lies: a shell profile exists on every machine with a shell, the workspace .mcp.json is+// read by several runtimes but filed under one, and VS Code's settings.json exists wherever VS Code+// does, Copilot Chat or not. Each produced a permanently inactive row for a product nobody had.+// Excluding them one at a time was losing; the shape of the mistake was using the scanner to answer+// a question it does not answer.//-// A row counts when Beacon itself wired the runtime up, since that is unambiguous whatever file it-// lives in. Otherwise the file has to be one only that runtime reads: not a shell profile, which-// exists on nearly every machine, and not a config shared between runtimes.+// The scanner reports which config files exist that some runtime might read. Coverage needs to know+// which runtimes Beacon set up to send telemetry, because that is the only population where silence+// is informative: Beacon wired it, so it should be reporting, so nothing arriving is worth a look.+// A runtime present on the machine but never configured by Beacon would not be reporting either+// way, and saying so every week is noise rather than coverage.//-// Two exclusions today, both found in review rather than by design, which is why the rule is-// written as a rule with a named list instead of a condition per file. A third shared path is-// likelier than not.+// beacon_managed is exactly that signal. It is set when a file carries Beacon's hook command, its+// managed-plugin marker, or an OTLP endpoint pointing at the local collector -- in each case+// because Beacon put it there.+//+// A runtime configured by hand, outside Beacon, is not counted. If it emits events it still appears+// in the report through the log; if it emits nothing, Beacon has no grounds to claim it should have.func InstalledRuntimes(configs []InstalledConfig) []string {seen := map[string]bool{}var out []stringfor _, config := range configs {- if !config.Exists {+ if !config.Exists || !config.BeaconManaged {continue}- if !config.BeaconManaged {- if config.Kind == ConfigKindProfile {- continue- }- if sharedConfigBasenames[strings.ToLower(filepath.Base(config.Path))] {- continue- }- }name := strings.TrimSpace(config.Runtime)if name == "" || seen[name] {continuediff --git a/cli/beacon/internal/tokens/coverage_test.go b/cli/beacon/internal/tokens/coverage_test.goindex 76ee0afb..5842d7b0 100644--- a/cli/beacon/internal/tokens/coverage_test.go+++ b/cli/beacon/internal/tokens/coverage_test.go@@ -263,48 +263,50 @@ func TestNewHarnessesAreClassifiedNotAlerted(t *testing.T) {}}-// Install detection reads the config scanner, which answers "which config files exist that some-// runtime might read" -- a different question from "is this runtime installed". Every row where-// those two diverge is a permanently inactive line for a product nobody has, and a status that is-// wrong on most machines is one people stop reading.-func TestInstalledRuntimesRequiresEvidenceThatIsActuallyTheRuntimes(t *testing.T) {+// Only rows Beacon itself configured count as evidence a runtime is wired up to report.+//+// Three rounds of review found three different files that exist for reasons other than the runtime+// the scanner files them under -- a shell profile, the shared workspace .mcp.json, VS Code's+// settings.json -- each producing a permanently inactive row for a product nobody had. The rule is+// no longer "exclude the known liars" but "only count what Beacon put there".+func TestInstalledRuntimesCountsOnlyWhatBeaconConfigured(t *testing.T) {for _, tc := range []struct {name stringconfig InstalledConfigwant bool}{{- name: "a runtime-exclusive config counts",- config: InstalledConfig{Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", Exists: true},+ name: "a config Beacon wired up counts",+ config: InstalledConfig{Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},want: true,},{- // ~/.zshrc exists on nearly every machine whether or not the product does.- name: "a shell profile alone does not",- config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, Exists: true},+ // The case that started this: ~/.zshrc exists on every machine with a shell.+ name: "a shell profile Beacon did not touch does not",+ config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: "profile", Exists: true},want: false,},{- // Beacon writing the runtime's OTLP export into the profile is unambiguous.- name: "a shell profile Beacon wired up does",- config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, BeaconManaged: true, Exists: true},- want: true,- },- {- // The scanner files the workspace .mcp.json under fx alone, though Claude Code and- // others read it too. Right for an MCP inventory, wrong as proof fx is installed.+ // Read by several runtimes, filed by the scanner under fx alone.name: "a config shared between runtimes does not",config: InstalledConfig{Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},want: false,},{- name: "fx's own MCP file does",- config: InstalledConfig{Runtime: "vercel_fx", Path: "/home/u/.fx/mcp.json", Kind: "native_config", Exists: true},+ // Exists wherever VS Code does, Copilot Chat installed or not.+ name: "VS Code settings alone does not",+ config: InstalledConfig{Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", Exists: true},+ want: false,+ },+ {+ // The same file once Beacon has written the local OTLP endpoint into it.+ name: "VS Code settings Beacon wired up does",+ config: InstalledConfig{Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},want: true,},{- name: "a config that does not exist does not",- config: InstalledConfig{Runtime: "cline", Path: "/home/u/.cline/config.json", Kind: "native_config", Exists: false},+ name: "a managed config that does not exist does not",+ config: InstalledConfig{Runtime: "cline", Path: "/home/u/.cline/config.json", Kind: "native_config", BeaconManaged: true, Exists: false},want: false,},} {@@ -317,32 +319,16 @@ func TestInstalledRuntimesRequiresEvidenceThatIsActuallyTheRuntimes(t *testing.T}}-// A runtime with several scanner rows is one runtime.-func TestInstalledRuntimesDeduplicates(t *testing.T) {+// A runtime with several managed rows is one runtime, and one managed row is enough among many+// unmanaged ones.+func TestInstalledRuntimesDeduplicatesAndNeedsOnlyOneManagedRow(t *testing.T) {got := InstalledRuntimes([]InstalledConfig{- {Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", Exists: true},- {Runtime: "claude_code", Path: "/home/u/.claude.json", Kind: "native_config", Exists: true},- })- if len(got) != 1 || got[0] != "claude_code" {- t.Fatalf("InstalledRuntimes = %v, want [claude_code]", got)- }-}--// One weak row must not be rescued by another weak row for the same runtime, and a strong row-// must still count when a weak one is present.-func TestInstalledRuntimesMixesWeakAndStrongRowsCorrectly(t *testing.T) {- onlyWeak := InstalledRuntimes([]InstalledConfig{- {Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},- {Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, Exists: true},- })- if len(onlyWeak) != 0 {- t.Fatalf("InstalledRuntimes = %v, want none -- both rows are shared files", onlyWeak)- }- withStrong := InstalledRuntimes([]InstalledConfig{- {Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},- {Runtime: "vercel_fx", Path: "/home/u/.fx/settings.json", Kind: "native_config", Exists: true},+ {Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},+ {Runtime: "claude_code", Path: "/home/u/.claude.json", Kind: "native_config", BeaconManaged: true, Exists: true},+ {Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", Exists: true},+ {Runtime: "vscode", Path: "/home/u/.copilot/hooks/beacon.json", Kind: "hook_config", BeaconManaged: true, Exists: true},})- if len(withStrong) != 1 {- t.Fatalf("InstalledRuntimes = %v, want vercel_fx counted on its own settings file", withStrong)+ if len(got) != 2 || got[0] != "claude_code" || got[1] != "vscode" {+ t.Fatalf("InstalledRuntimes = %v, want [claude_code vscode]", got)}}diff --git a/docs/cli/token-usage.mdx b/docs/cli/token-usage.mdxindex e313d1af..8eedfb8f 100644--- a/docs/cli/token-usage.mdx+++ b/docs/cli/token-usage.mdx@@ -65,11 +65,13 @@ The status column is the report:|--------|---------|| `covered` | The runtime reported usage in the window. Nothing to do || `silent` | The runtime produced events but no usage, and Beacon is built to read usage from it. **This is the one to investigate** — a stale runtime build, a hook that stopped firing, or an OTLP export that is no longer configured |-| `inactive` | Installed, but produced no events at all in the window. Usually means it was not used |+| `inactive` | Asymptote configured this runtime, but it produced no events at all in the window. Usually means it was not used; can also mean its hooks or OTLP export stopped firing || `not_instrumented` | The runtime ran, and Beacon cannot read token usage from it. Expected, not a fault. The note column says why |The `not_instrumented` distinction is what makes the report worth reading. Without it Cursor would appear as a problem on every run, and a column that always shows a problem is a column people stop reading.+The `INSTALLED` column means "Asymptote configured this runtime", not "this product is on the machine". Only configuration Asymptote wrote counts — a hook command, a managed plugin marker, or a local OTLP endpoint it set. That is the population where silence is informative: Asymptote wired it up, so it should be reporting. A runtime present but configured by hand would not be reporting through Asymptote either way, so it appears only if it actually emits events.+Coverage applies the `--since`, `--until` and `--harness` scopes, and deliberately ignores `--session`, `--model`, `--repository` and `--run-id`. Those filters select usage-bearing events almost by definition — an event carries no model unless it carried a model call — so a coverage report computed over them would find every runtime covered and prove nothing.Add `--json` for the machine-readable form, which carries the same per-runtime rows plus `silent`, `covered` and `total_events` counts. A `silent` count above zero is the alertable condition.