Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

Count a runtime as installed only where Beacon configured it

Claudecbbc2060f5a7
diff --git a/cli/beacon/cmd/token_usage.go b/cli/beacon/cmd/token_usage.go
index e84cb026..d7de8ad0 100644
--- a/cli/beacon/cmd/token_usage.go
+++ b/cli/beacon/cmd/token_usage.go
@@ -131,10 +131,10 @@ func runTokenCoverage(cmd *cobra.Command, logPath string, query dashboard.EventQ
}
events = scopedEvents
}
- // Installed runtimes come from the config scanner rather than from the log, because the
- // whole question is which installed runtime is missing from the log. tokens.InstalledRuntimes
- // decides which scanner rows are real evidence of an install; the scanner reports files a
- // runtime might read, which is not the same thing.
+ // Installed runtimes come from the config scanner rather than from the log, because the whole
+ // question is which configured runtime is missing from the log. tokens.InstalledRuntimes keeps
+ // only the rows Beacon itself wired up; the scanner reports files a runtime might read, which
+ // is a different and much larger set.
configs := make([]tokens.InstalledConfig, 0)
for _, config := range endpointinventory.Scan(endpointinventory.Options{}).Configs {
configs = append(configs, tokens.InstalledConfig{
diff --git a/cli/beacon/cmd/token_usage_test.go b/cli/beacon/cmd/token_usage_test.go
index 55261f31..9d690723 100644
--- a/cli/beacon/cmd/token_usage_test.go
+++ b/cli/beacon/cmd/token_usage_test.go
@@ -8,7 +8,6 @@ import (
"strings"
"testing"
- endpointinventory "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/endpoint/inventory"
"github.com/asymptote-labs/agent-beacon/cli/beacon/internal/tokens"
)
@@ -227,16 +226,6 @@ func TestTokenUsageCoverageDoesNotTreatAShellProfileAsAnInstall(t *testing.T) {
}
}
-// tokens.ConfigKindProfile duplicates inventory.KindProfile so the tokens package stays
-// independent of the scanner. Pinned here, where both are importable, because a silent drift
-// would stop excluding shell profiles and quietly refill the report with false installs.
-func TestTokenCoverageProfileKindConstantMatchesInventory(t *testing.T) {
- if tokens.ConfigKindProfile != endpointinventory.KindProfile {
- t.Fatalf("tokens.ConfigKindProfile = %q, inventory.KindProfile = %q; they must agree",
- tokens.ConfigKindProfile, endpointinventory.KindProfile)
- }
-}
-
// A harness scope narrows the events, so it has to narrow the installed list with them. Without
// that, every other installed runtime has no events in the filtered set and is reported inactive
// -- which reads as "installed but unused this window" when the truth is only that the reader
diff --git a/cli/beacon/internal/tokens/coverage.go b/cli/beacon/internal/tokens/coverage.go
index 8aac5983..85cd9df3 100644
--- a/cli/beacon/internal/tokens/coverage.go
+++ b/cli/beacon/internal/tokens/coverage.go
@@ -1,7 +1,6 @@
package tokens
import (
- "path/filepath"
"sort"
"strings"
@@ -109,21 +108,6 @@ var usageExpectation = map[string]struct {
"openhands": {ExpectNone, "hook payloads carry no token counts"},
}
-// ConfigKindProfile mirrors inventory.KindProfile. It is duplicated rather than imported to keep
-// this package independent of the scanner; cmd pins the two equal in a test.
-const ConfigKindProfile = "profile"
-
-// sharedConfigBasenames are config files that more than one runtime reads, which the scanner
-// nonetheless attributes to a single runtime.
-//
-// `.mcp.json` is the case: a workspace MCP server list that Claude Code, fx and others all read,
-// which the scanner files under fx alone -- deliberately, because attributing one file to several
-// runtimes would report one server several times. That is right for an inventory of MCP servers
-// and wrong as evidence of an install: a repository whose .mcp.json exists for Claude Code would
-// otherwise report fx as installed, and fx would sit in every coverage report as a permanently
-// inactive runtime nobody has.
-var sharedConfigBasenames = map[string]bool{".mcp.json": true}
-
// InstalledConfig is the part of a config-scanner row that install detection reads.
type InstalledConfig struct {
Runtime string
@@ -133,35 +117,35 @@ type InstalledConfig struct {
Exists bool
}
-// InstalledRuntimes selects the scanner rows that are real evidence a runtime is installed.
+// InstalledRuntimes selects the scanner rows that are evidence a runtime is wired up to report.
//
-// The scanner answers "which config files exist that some runtime might read", which is a
-// different question from "is this runtime installed", and the gap between them is all
-// false positives -- each one a permanently inactive row for a product nobody has. A status that
-// is wrong on most machines is one people learn to skip, which costs more than the rows are worth.
+// Only rows Beacon itself configured count. Everything else -- a config file that merely exists --
+// is not evidence of anything this report can use, and three rounds of review found three separate
+// ways it lies: a shell profile exists on every machine with a shell, the workspace .mcp.json is
+// read by several runtimes but filed under one, and VS Code's settings.json exists wherever VS Code
+// does, Copilot Chat or not. Each produced a permanently inactive row for a product nobody had.
+// Excluding them one at a time was losing; the shape of the mistake was using the scanner to answer
+// a question it does not answer.
//
-// A row counts when Beacon itself wired the runtime up, since that is unambiguous whatever file it
-// lives in. Otherwise the file has to be one only that runtime reads: not a shell profile, which
-// exists on nearly every machine, and not a config shared between runtimes.
+// The scanner reports which config files exist that some runtime might read. Coverage needs to know
+// which runtimes Beacon set up to send telemetry, because that is the only population where silence
+// is informative: Beacon wired it, so it should be reporting, so nothing arriving is worth a look.
+// A runtime present on the machine but never configured by Beacon would not be reporting either
+// way, and saying so every week is noise rather than coverage.
//
-// Two exclusions today, both found in review rather than by design, which is why the rule is
-// written as a rule with a named list instead of a condition per file. A third shared path is
-// likelier than not.
+// beacon_managed is exactly that signal. It is set when a file carries Beacon's hook command, its
+// managed-plugin marker, or an OTLP endpoint pointing at the local collector -- in each case
+// because Beacon put it there.
+//
+// A runtime configured by hand, outside Beacon, is not counted. If it emits events it still appears
+// in the report through the log; if it emits nothing, Beacon has no grounds to claim it should have.
func InstalledRuntimes(configs []InstalledConfig) []string {
seen := map[string]bool{}
var out []string
for _, config := range configs {
- if !config.Exists {
+ if !config.Exists || !config.BeaconManaged {
continue
}
- if !config.BeaconManaged {
- if config.Kind == ConfigKindProfile {
- continue
- }
- if sharedConfigBasenames[strings.ToLower(filepath.Base(config.Path))] {
- continue
- }
- }
name := strings.TrimSpace(config.Runtime)
if name == "" || seen[name] {
continue
diff --git a/cli/beacon/internal/tokens/coverage_test.go b/cli/beacon/internal/tokens/coverage_test.go
index 76ee0afb..5842d7b0 100644
--- a/cli/beacon/internal/tokens/coverage_test.go
+++ b/cli/beacon/internal/tokens/coverage_test.go
@@ -263,48 +263,50 @@ func TestNewHarnessesAreClassifiedNotAlerted(t *testing.T) {
}
}
-// Install detection reads the config scanner, which answers "which config files exist that some
-// runtime might read" -- a different question from "is this runtime installed". Every row where
-// those two diverge is a permanently inactive line for a product nobody has, and a status that is
-// wrong on most machines is one people stop reading.
-func TestInstalledRuntimesRequiresEvidenceThatIsActuallyTheRuntimes(t *testing.T) {
+// Only rows Beacon itself configured count as evidence a runtime is wired up to report.
+//
+// Three rounds of review found three different files that exist for reasons other than the runtime
+// the scanner files them under -- a shell profile, the shared workspace .mcp.json, VS Code's
+// settings.json -- each producing a permanently inactive row for a product nobody had. The rule is
+// no longer "exclude the known liars" but "only count what Beacon put there".
+func TestInstalledRuntimesCountsOnlyWhatBeaconConfigured(t *testing.T) {
for _, tc := range []struct {
name string
config InstalledConfig
want bool
}{
{
- name: "a runtime-exclusive config counts",
- config: InstalledConfig{Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", Exists: true},
+ name: "a config Beacon wired up counts",
+ config: InstalledConfig{Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},
want: true,
},
{
- // ~/.zshrc exists on nearly every machine whether or not the product does.
- name: "a shell profile alone does not",
- config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, Exists: true},
+ // The case that started this: ~/.zshrc exists on every machine with a shell.
+ name: "a shell profile Beacon did not touch does not",
+ config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: "profile", Exists: true},
want: false,
},
{
- // Beacon writing the runtime's OTLP export into the profile is unambiguous.
- name: "a shell profile Beacon wired up does",
- config: InstalledConfig{Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, BeaconManaged: true, Exists: true},
- want: true,
- },
- {
- // The scanner files the workspace .mcp.json under fx alone, though Claude Code and
- // others read it too. Right for an MCP inventory, wrong as proof fx is installed.
+ // Read by several runtimes, filed by the scanner under fx alone.
name: "a config shared between runtimes does not",
config: InstalledConfig{Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},
want: false,
},
{
- name: "fx's own MCP file does",
- config: InstalledConfig{Runtime: "vercel_fx", Path: "/home/u/.fx/mcp.json", Kind: "native_config", Exists: true},
+ // Exists wherever VS Code does, Copilot Chat installed or not.
+ name: "VS Code settings alone does not",
+ config: InstalledConfig{Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", Exists: true},
+ want: false,
+ },
+ {
+ // The same file once Beacon has written the local OTLP endpoint into it.
+ name: "VS Code settings Beacon wired up does",
+ config: InstalledConfig{Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},
want: true,
},
{
- name: "a config that does not exist does not",
- config: InstalledConfig{Runtime: "cline", Path: "/home/u/.cline/config.json", Kind: "native_config", Exists: false},
+ name: "a managed config that does not exist does not",
+ config: InstalledConfig{Runtime: "cline", Path: "/home/u/.cline/config.json", Kind: "native_config", BeaconManaged: true, Exists: false},
want: false,
},
} {
@@ -317,32 +319,16 @@ func TestInstalledRuntimesRequiresEvidenceThatIsActuallyTheRuntimes(t *testing.T
}
}
-// A runtime with several scanner rows is one runtime.
-func TestInstalledRuntimesDeduplicates(t *testing.T) {
+// A runtime with several managed rows is one runtime, and one managed row is enough among many
+// unmanaged ones.
+func TestInstalledRuntimesDeduplicatesAndNeedsOnlyOneManagedRow(t *testing.T) {
got := InstalledRuntimes([]InstalledConfig{
- {Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", Exists: true},
- {Runtime: "claude_code", Path: "/home/u/.claude.json", Kind: "native_config", Exists: true},
- })
- if len(got) != 1 || got[0] != "claude_code" {
- t.Fatalf("InstalledRuntimes = %v, want [claude_code]", got)
- }
-}
-
-// One weak row must not be rescued by another weak row for the same runtime, and a strong row
-// must still count when a weak one is present.
-func TestInstalledRuntimesMixesWeakAndStrongRowsCorrectly(t *testing.T) {
- onlyWeak := InstalledRuntimes([]InstalledConfig{
- {Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},
- {Runtime: "copilot_cli", Path: "/home/u/.zshrc", Kind: ConfigKindProfile, Exists: true},
- })
- if len(onlyWeak) != 0 {
- t.Fatalf("InstalledRuntimes = %v, want none -- both rows are shared files", onlyWeak)
- }
- withStrong := InstalledRuntimes([]InstalledConfig{
- {Runtime: "vercel_fx", Path: "/repo/.mcp.json", Kind: "native_config", Exists: true},
- {Runtime: "vercel_fx", Path: "/home/u/.fx/settings.json", Kind: "native_config", Exists: true},
+ {Runtime: "claude_code", Path: "/home/u/.claude/settings.json", Kind: "native_config", BeaconManaged: true, Exists: true},
+ {Runtime: "claude_code", Path: "/home/u/.claude.json", Kind: "native_config", BeaconManaged: true, Exists: true},
+ {Runtime: "vscode", Path: "/home/u/.config/Code/User/settings.json", Kind: "native_config", Exists: true},
+ {Runtime: "vscode", Path: "/home/u/.copilot/hooks/beacon.json", Kind: "hook_config", BeaconManaged: true, Exists: true},
})
- if len(withStrong) != 1 {
- t.Fatalf("InstalledRuntimes = %v, want vercel_fx counted on its own settings file", withStrong)
+ if len(got) != 2 || got[0] != "claude_code" || got[1] != "vscode" {
+ t.Fatalf("InstalledRuntimes = %v, want [claude_code vscode]", got)
}
}
diff --git a/docs/cli/token-usage.mdx b/docs/cli/token-usage.mdx
index e313d1af..8eedfb8f 100644
--- a/docs/cli/token-usage.mdx
+++ b/docs/cli/token-usage.mdx
@@ -65,11 +65,13 @@ The status column is the report:
|--------|---------|
| `covered` | The runtime reported usage in the window. Nothing to do |
| `silent` | The runtime produced events but no usage, and Beacon is built to read usage from it. **This is the one to investigate** — a stale runtime build, a hook that stopped firing, or an OTLP export that is no longer configured |
-| `inactive` | Installed, but produced no events at all in the window. Usually means it was not used |
+| `inactive` | Asymptote configured this runtime, but it produced no events at all in the window. Usually means it was not used; can also mean its hooks or OTLP export stopped firing |
| `not_instrumented` | The runtime ran, and Beacon cannot read token usage from it. Expected, not a fault. The note column says why |
The `not_instrumented` distinction is what makes the report worth reading. Without it Cursor would appear as a problem on every run, and a column that always shows a problem is a column people stop reading.
+The `INSTALLED` column means "Asymptote configured this runtime", not "this product is on the machine". Only configuration Asymptote wrote counts — a hook command, a managed plugin marker, or a local OTLP endpoint it set. That is the population where silence is informative: Asymptote wired it up, so it should be reporting. A runtime present but configured by hand would not be reporting through Asymptote either way, so it appears only if it actually emits events.
+
Coverage applies the `--since`, `--until` and `--harness` scopes, and deliberately ignores `--session`, `--model`, `--repository` and `--run-id`. Those filters select usage-bearing events almost by definition — an event carries no model unless it carried a model call — so a coverage report computed over them would find every runtime covered and prove nothing.
Add `--json` for the machine-readable form, which carries the same per-runtime rows plus `silent`, `covered` and `total_events` counts. A `silent` count above zero is the alertable condition.