Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.
TypeScript browser-shellcontrol-planefile-managerhomelabmobile-firstnextjsptyreactself-hostedserver-managementshadcn-uisysadmintailwindcssterminaltypescriptvpsweb-desktopweb-terminal
perf(agent): normalize provider usage semantics
Rahman3b3d1552119c
diff --git a/README.md b/README.mdindex c142f432..eb16f417 100644--- a/README.md+++ b/README.md@@ -79,7 +79,7 @@ For a real deployment, put MSO behind **Tailscale, a VPN, or a TLS reverse proxy- **Message live local sessions by a short name** — every durable same-principal MSO session gets a unique familiar handle such as `@milo`, `@luna`, or `@nara`, independent from its longer auto-generated session title. `/rename <name>` changes that handle, while `/title <text>` changes the session description. `@milo …` resolves only agents with a current lease **and** a live receiver subscription, then creates an explicitly correlated request; replies relay deterministically into the originating conversation while notify-only events remain passive. Presence lease (`idle`/`busy`/`offline`) and receiver subscription (`consumerConnected`) are reported separately so an idle-but-unsubscribed target is diagnosable. For ChatGPT/MCP sessions, `local_agent_inbox(wait_ms=...)` temporarily turns the foreground MCP call into that live receiver for up to 20 seconds, enabling direct two-way session messaging without a worker, DB, webhook, or broker. See [Local Agents](./docs/LOCAL-AGENTS.md).- **Spawn focused same-session subagents** — `/spawn` and `agent_subagent_run` create bounded foreground workers with isolated context, explicit delegated scope, turn/timeout caps, no recursive spawn, and final-result-only return. They are not Local Agent peers and do not run autonomously after the parent call returns. See [Subagents](./docs/SUBAGENTS.md).- **Delegate between remote agents with standard A2A v1** — register public HTTPS peers, attach private API-key/Bearer/OAuth access-token profiles, stream remote tasks over SSE, or expose MSO's authenticated inbound Agent Card when a public HTTPS origin is configured. Inbound credentials are owner-minted `read | write | exec` capabilities and run in a memory-isolated task context. See [A2A](./docs/A2A.md).-- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:pipeline`, and isolated scratch-fixture `bun run bench:corpus` guard routing, correctness, policy compliance, latency and context savings.+- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:pipeline`, and isolated scratch-fixture `bun run bench:corpus` guard routing, correctness, policy compliance, latency, context savings, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.- **Make orchestration risk-aware and self-improving** — [RASMIC](./docs/RASMIC.md) classifies task risk/contention, retrieves compact repo-local memory, records manual user tests and Evidence Receipts, detects path/shared-resource collisions before merge conflicts, and promotes repeated safe routes from observed traces → recipes → tested bounded scripts.- **Add app slices** — features are modular under `frontend/slices/<slug>/`.- **Personalize the interface** — macOS, Windows, iOS, and Android shell layouts are UI preferences, not the core product.diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.mdindex 482263fb..22b622eb 100644--- a/docs/CHANGELOG.md+++ b/docs/CHANGELOG.md@@ -30,6 +30,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.**Faster**+- `agent` normalize provider usage semantics- `agent` add comparable quality corpus- `agent` route capabilities before model reasoningdiff --git a/docs/COGNITIVE-RUNTIME.md b/docs/COGNITIVE-RUNTIME.mdindex 4590145c..a61c8a80 100644--- a/docs/COGNITIVE-RUNTIME.md+++ b/docs/COGNITIVE-RUNTIME.md@@ -201,7 +201,7 @@ Comparability is fail-closed: every attempted scenario must report matching mode### Token/cost accounting is deliberately separate-The reported token totals are **not used to rank the agents**. MSO currently reports provider `input/output/total` usage (`input-output-total` accounting), while Hermes' one-shot report exposes expanded input/output/cache/reasoning components (`expanded-components`). P4 therefore records `tokenSemanticsComparable=false` even though both have 100% token coverage. Failed attempts, when present, are charged into tokens/cost-per-success instead of disappearing from the denominator; partial coverage leaves per-success efficiency unknown. Cost comparison is also withheld: MSO reports no comparable cost field, while the local Hermes report marks cost as included with source `none`; an ambiguous generic `cost` is not assumed to be USD unless currency is explicit. Missing or semantically incompatible accounting remains **unknown/non-comparable**, never zero-efficiency evidence.+P4 deliberately withheld token ranking because MSO exposed only aggregate `input/output/total` while Hermes exposed cache/reasoning components with a different representation. P5 resolves that observability gap below without rewriting historical P4 results. Failed attempts, when present, are still charged into token/cost-per-success instead of disappearing from the denominator; partial/opaque accounting remains unknown. Cost comparison remains withheld unless both runners expose the same attributable USD semantics.### P4 found and fixed a real RASMIC inefficiency@@ -211,15 +211,47 @@ The final `multi-read` scenario uses exactly one `read_pipeline` call. Recovery### OpenClaw status-OpenClaw remains installed and testable, but the current host does not expose an equivalent usable `openai-codex/gpt-5.6-terra` path for its main agent. P4 **does not change competitor credentials/configuration just to manufacture comparability**, and it therefore emits no OpenClaw quality ranking yet.+OpenClaw remains installed and testable, but the current host does not expose an equivalent usable `openai-codex/gpt-5.6-terra` path for its main agent. P4/P5 **do not change competitor credentials/configuration just to manufacture comparability**, and they therefore emit no OpenClaw quality ranking yet.-## P4 next direction+## P5 normalized provider-usage semantics++P5 makes token accounting auditable without adding prompt context or inventing missing values. A provider-neutral usage envelope now preserves the provider's raw categories plus an explicit accounting mode:++- OpenAI Responses / Codex and compatible Chat Completions: `inputTokens`, `outputTokens`, authoritative `totalTokens`, optional `cacheReadTokens`, `cacheWriteTokens`, and `reasoningTokens`, with `accountingMode=inclusive-input-output-total`. Cache/reasoning are details already included in the input/output categories; they are **not added again**.+- Anthropic: input/output plus optional cache-read/cache-creation categories with `accountingMode=separate-cache-input-output`; absent fields remain absent. A benchmark-only canonical total may be derived from this explicit component contract, while the raw provider total remains unknown if Anthropic did not report one.+- Agent session aggregation tracks `apiCalls`, accounting mode, and detail coverage. A new session starts at `{apiCalls:0}` instead of fake zero token totals, so an unknown provider field cannot silently become `0`.++The benchmark converts representations to canonical `inclusive-input-output-total` only with proof. `accountingProof` records why the conversion is allowed. In the isolated candidate corpus:++- MSO: `explicit-inclusive-contract` from the actual Codex `response.completed.usage` payload, including provider-reported cache/reasoning details.+- Hermes: `exact-exclusive-cache-sum`; on every corpus scenario, arithmetic proves `total = base input + cache-read/cache-write + output`, while `reasoning` is already a subset of output. If that identity does not hold, the row stays `opaque-total`.+- Two equally opaque totals are **not** considered comparable merely because their labels match. Token comparability additionally requires 100% coverage, canonical recognized mode, and non-unknown proof for every agent.++A full six-scenario candidate run on the same `openai-codex/gpt-5.6-terra` provider/model preserved the quality result while making token totals comparable:++| P5 metric | MSO | Hermes |+|---|---:|---:|+| Task success | **6/6 (100%)** | **6/6 (100%)** |+| Scenario-observable policy compliance | **6/6 (100%)** | **6/6 (100%)** |+| Average latency | **11,497.7 ms** | 16,199.3 ms |+| p50 latency | **11,782 ms** | 16,184.5 ms |+| Normalized token coverage | **100%** | **100%** |+| Normalized tokens / successful task | **3,510.8** | 59,743.2 |+| Accounting proof | explicit inclusive contract | exact exclusive-cache sum |+| Cost semantics comparable | no | no |++On this bounded corpus, MSO uses about **94.1% fewer normalized tokens per successful task (~17× smaller)**, with **29.0% lower average latency** and **27.2% lower p50 latency** in this run. This is a corpus result, not a universal product/model claim; quality ranking still prioritizes task success and scenario-policy, with latency only as its descriptive tie-breaker. Token efficiency is reported as a separate now-comparable metric and still does not override correctness/policy. Cost remains withheld because MSO has no comparable cost field and Hermes reports included cost with source `none`.++The candidate run also preserves the structural P3/P4 behaviors: `multi-read` uses one `read_pipeline`, security uses one `fs_read`, and recovery's one failed primary read remains expected task evidence.++## P5 next direction1. **Broaden the corpus before broad claims** — add repository debugging, longer multi-step work, approval/retry/rollback, and realistic MCP project tasks while keeping objective state-based scoring.-2. **Normalize provider usage semantics** — expose cache/reasoning usage consistently from MSO (where the provider supplies it) before allowing token/cost efficiency to influence ranking.-3. **Add OpenClaw only when comparable** — run the same corpus when an equivalent provider/model is legitimately configured, without mutating credentials as part of the benchmark.-4. **Use benchmark failures as engineering inputs** — keep the P4 pattern: locate an unnecessary route/tool/retry, fix the harness/runtime rather than prompting harder, then rerun the exact seeded scenario.-5. **Memory calibration** — grow post-P1 telemetry and test authority/confidence/temporal policies against real corrections before introducing graph-memory complexity.+2. **Calibrate cache behavior** — add repeated-prefix fixtures that can observe real provider cache hits without forcing them; keep zero/absent cache fields distinct and never assume a hit.+3. **Normalize cost only with attribution** — compare cost only when every runner exposes the same usable USD source/contract; `0` with source `none` is not evidence of free execution.+4. **Add OpenClaw only when comparable** — run the same corpus when an equivalent provider/model is legitimately configured, without mutating credentials as part of the benchmark.+5. **Use benchmark failures as engineering inputs** — keep the P4/P5 pattern: locate an unnecessary route/tool/retry/accounting ambiguity, fix the harness/runtime rather than prompting harder, then rerun the exact seeded scenario.+6. **Memory calibration** — grow post-P1 telemetry and test authority/confidence/temporal policies against real corrections before introducing graph-memory complexity.## Security notesdiff --git a/docs/PROGRESS.md b/docs/PROGRESS.mdindex 8dd27073..e4064e87 100644--- a/docs/PROGRESS.md+++ b/docs/PROGRESS.md@@ -2,6 +2,14 @@Running log of what shipped each phase. Newest at top.+## 2026-09-03 — Cognitive Runtime P5 normalized provider usage++- **Provider usage SSOT:** Codex/OpenAI now preserve authoritative input/output/total plus provider-reported cache-read/cache-write/reasoning details and explicit `inclusive-input-output-total` semantics; Anthropic keeps its separate cache categories and leaves absent fields absent. Agent sessions start with `apiCalls:0`, so unknown totals are never seeded as fake zero.+- **No double counting:** cache/reasoning details remain breakdowns when the provider contract says they are included. Session aggregation records API-call/detail coverage and mixed semantics rather than summing a component into total twice. Compact status can show in/out even when a provider does not report total.+- **Proof-gated benchmark normalization:** P5 adds `accountingProof`. MSO Codex uses `explicit-inclusive-contract`; Hermes openai-codex is normalized only when exact arithmetic proves cache is outside base input while reasoning is inside output (`exact-exclusive-cache-sum`). Ambiguous rows stay `opaque-total`, and even two opaque rows cannot enable token comparison.+- **Comparable full-corpus result:** candidate P5 reran all six P4 scenarios on `openai-codex/gpt-5.6-terra`; both agents remained **6/6 task + 6/6 scenario-policy**. MSO measured **11,497.7 ms average / 11,782 ms p50 / 3,510.8 normalized tokens per success** versus Hermes **16,199.3 ms / 16,184.5 ms / 59,743.2 tokens**. On this bounded corpus that is ~**94.1% fewer normalized tokens (~17× smaller)**, ~**29.0% lower average latency**, and ~**27.2% lower p50**. Token semantics are now comparable; cost semantics remain withheld.+- **Honest limits:** no cache hit was manufactured for MSO; zero cache values in the observed Codex calls came from the provider payload. Cost remains non-comparable because MSO exposes no matching cost field and Hermes reports source `none`. OpenClaw remains unranked without an equivalent configured provider/model.+## 2026-09-03 — Cognitive Runtime P4 comparable quality corpus- **Six objective scratch scenarios:** added seeded per-agent read, multi-read transform, exact write-create, write-preserve, recovery, and prompt-injection/security tasks. Each agent gets the same seeded semantic values/expected outcomes in a separate 0700 scratch tree; scoring compares the exact scenario filesystem tree/result state and rejects unexpected extra files/symlinks. Fixtures are isolated, but runners retain normal authority (`runnerAuthoritySandboxed=false`) and `policyObservationScope=scenario-tree` prevents whole-host policy claims; all scratch state is deleted after the run.diff --git a/docs/TESTING-HANDOFF.md b/docs/TESTING-HANDOFF.mdindex 7dac3596..b051d571 100644--- a/docs/TESTING-HANDOFF.md+++ b/docs/TESTING-HANDOFF.md@@ -7,6 +7,7 @@ Update this file at the end of every manual or automated test session. Keep it fThe current deployed baseline is **MSO CLI 1.12.0** with toolset `2026.09.03.2`; use `mso --version` plus `/api/health` for the exact Git build because documentation-only release commits also advance that identity. The v1.10/v1.11 findings and the v1.12 sectioned-terminal/recoverable-error work below are resolved and retained as historical test evidence. Do not re-open a resolved finding from prose alone: reproduce it against the current build first. There is no known 1.12 release blocker in this handoff.Current verified behavior:+- P5 provider-usage normalization preserves Codex/OpenAI cache/reasoning details without double counting, keeps unknown fields unknown, and proof-gates canonical token semantics. A full candidate MSO↔Hermes run on `openai-codex/gpt-5.6-terra` remained 6/6 task + 6/6 scenario-policy for both; token semantics are now comparable at 3,510.8 vs 59,743.2 normalized tokens/success, while cost semantics remain non-comparable. MSO's proof is `explicit-inclusive-contract`; Hermes is `exact-exclusive-cache-sum`.- P4 `bench:corpus` uses six seeded, per-agent isolated 0700 scratch scenarios, exact scenario-tree scoring (`policyObservationScope=scenario-tree`, not whole-host sandbox proof), and rotating scenario-major runner order. The reproduced full provider+model-matched MSO↔Hermes run on `openai-codex/gpt-5.6-terra` is 6/6 task + 6/6 scenario-observable policy for both; MSO measured 10,759.5 ms average / 10,646.5 ms p50 versus Hermes 14,624 ms / 14,304.5 ms. This is a bounded corpus tie-break, not an overall-product claim. Token/cost accounting remains explicitly non-comparable and is not a ranking input; OpenClaw is unranked until an equivalent model/provider path exists.- every durable session has a unique short public `name` (`milo`, `luna`, `nara`, …) independent from its longer `title`; `/rename` changes the handle and `/title` changes the description;- human `@name` mention routing resolves active agents only; internal `agent-a` aliases remain compatibility data for explicit APIs;diff --git a/lib/ai/codex-stream.test.ts b/lib/ai/codex-stream.test.tsindex 8071bb42..efa894dc 100644--- a/lib/ai/codex-stream.test.ts+++ b/lib/ai/codex-stream.test.ts@@ -188,6 +188,19 @@ describe("reading calls back", () => {expect(ev).toContainEqual(["tool_use", { id: "c1", name: "fs.list", input: {} }]);});+ it("preserves Responses cache/reasoning usage details with explicit inclusive semantics", async () => {+ capture(sse({ type: "response.completed", response: { output: [], usage: {+ input_tokens: 100, input_tokens_details: { cached_tokens: 60, cache_write_tokens: 5 },+ output_tokens: 20, output_tokens_details: { reasoning_tokens: 7 }, total_tokens: 120,+ } } }));+ const ev = await run();+ expect(ev.at(-1)).toEqual(["done", { stopReason: "end_turn", usage: {+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 100, outputTokens: 20, totalTokens: 120,+ cacheReadTokens: 60, cacheWriteTokens: 5, reasoningTokens: 7,+ detailCoverage: { cacheReadTokens: 1, cacheWriteTokens: 1, reasoningTokens: 1 },+ } }]);+ });+it("still streams plain text with no tools declared", async () => {capture(sse({ type: "response.output_text.delta", delta: "hello" }));const ev = await run();diff --git a/lib/ai/codex-stream.ts b/lib/ai/codex-stream.tsindex c4150867..1430ab49 100644--- a/lib/ai/codex-stream.ts+++ b/lib/ai/codex-stream.ts@@ -1,6 +1,7 @@import type { OAuthBundle } from "@/lib/config/store";import { CODEX } from "./oauth/codex";import type { OaMsg, OaTool } from "./openai-stream";+import { responsesProviderUsage } from "./provider-usage.mjs";// Stream a chat via the ChatGPT "Codex" backend Responses API. Unlike the OpenAI// platform (/chat/completions), this is the consumer backend: the OAuth bearer +@@ -159,7 +160,7 @@ export async function streamCodex(opts: {const dec = new TextDecoder();let buf = "";let sawToolCall = false;- let usage: { inputTokens?: number; outputTokens?: number; totalTokens?: number } | null = null;+ let usage: ReturnType<typeof responsesProviderUsage> = null;const seen = new Set<string>();while (true) {@@ -179,7 +180,10 @@ export async function streamCodex(opts: {if (!data || data === "[DONE]") continue;let ev: {type?: string; delta?: string; item?: ResponsesItem;- response?: { output?: ResponsesItem[]; usage?: { input_tokens?: number; output_tokens?: number; total_tokens?: number } };+ response?: { output?: ResponsesItem[]; usage?: {+ input_tokens?: number; input_tokens_details?: { cached_tokens?: number; cache_write_tokens?: number };+ output_tokens?: number; output_tokens_details?: { reasoning_tokens?: number }; total_tokens?: number;+ } };};try {ev = JSON.parse(data);@@ -196,13 +200,7 @@ export async function streamCodex(opts: {if (ev.type === "response.output_item.done" && ev.item) {if (emitCall(ev.item, seen, realName, emit)) sawToolCall = true;} else if (ev.type === "response.completed") {- if (ev.response?.usage) {- usage = {- inputTokens: ev.response.usage.input_tokens,- outputTokens: ev.response.usage.output_tokens,- totalTokens: ev.response.usage.total_tokens,- };- }+ if (ev.response?.usage) usage = responsesProviderUsage(ev.response.usage);for (const item of ev.response?.output ?? []) {if (emitCall(item, seen, realName, emit)) sawToolCall = true;}diff --git a/lib/ai/openai-stream.test.ts b/lib/ai/openai-stream.test.tsindex 04d0d1ec..1f8e6b1c 100644--- a/lib/ai/openai-stream.test.ts+++ b/lib/ai/openai-stream.test.ts@@ -50,6 +50,17 @@ describe("streamOpenAI", () => {expect(done.stopReason).toBe("tool_use"); // tool_calls → tool_use so the agent loop continues});+ it("preserves Chat Completions cache/reasoning usage details without adding them to total", async () => {+ const pieces = [`data: ${JSON.stringify({ usage: { prompt_tokens: 30, prompt_tokens_details: { cached_tokens: 20 }, completion_tokens: 6, completion_tokens_details: { reasoning_tokens: 4 }, total_tokens: 36 }, choices: [{ delta: {}, finish_reason: "stop" }] })}\n\ndata: [DONE]\n\n`];+ vi.stubGlobal("fetch", vi.fn(async () => new Response(sseStream(pieces), { status: 200 })));+ const events: { e: string; d: unknown }[] = [];+ await streamOpenAI({ resolved, messages: [{ role: "user", text: "hi" }], system: "sys", signal: new AbortController().signal, emit: (e, d) => events.push({ e, d }) });+ expect(events.find((x) => x.e === "done")?.d).toEqual({ stopReason: "end_turn", usage: {+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 30, outputTokens: 6, totalTokens: 36, cacheReadTokens: 20, reasoningTokens: 4,+ detailCoverage: { cacheReadTokens: 1, reasoningTokens: 1 },+ } });+ });+it("throws provider+status on a non-ok response", async () => {vi.stubGlobal("fetch", vi.fn(async () => new Response("nope", { status: 401 })));await expect(diff --git a/lib/ai/openai-stream.ts b/lib/ai/openai-stream.tsindex 1a4dcf6d..29b6f8ef 100644--- a/lib/ai/openai-stream.ts+++ b/lib/ai/openai-stream.ts@@ -1,3 +1,4 @@+import { chatCompletionsProviderUsage } from "./provider-usage.mjs";// OpenAI-protocol streaming adapter for /api/assistant. Every provider in the// @rahmanef/models registry except Anthropic speaks the OpenAI Chat Completions// wire (POST {baseUrl}/chat/completions, `data: {choices:[{delta}]}` SSE lines@@ -23,7 +24,10 @@ type OaOutMsg = {};type OaChunk = {- usage?: { prompt_tokens?: number; completion_tokens?: number; total_tokens?: number };+ usage?: {+ prompt_tokens?: number; prompt_tokens_details?: { cached_tokens?: number; cache_write_tokens?: number };+ completion_tokens?: number; completion_tokens_details?: { reasoning_tokens?: number }; total_tokens?: number;+ };choices?: Array<{delta?: {content?: string | null;@@ -110,7 +114,7 @@ export async function streamOpenAI(opts: {// tool_call fragments arrive split across chunks; accumulate by index.const calls = new Map<number, { id: string; name: string; args: string }>();let finish: string | null = null;- let usage: { inputTokens?: number; outputTokens?: number; totalTokens?: number } | null = null;+ let usage: ReturnType<typeof chatCompletionsProviderUsage> = null;reading: while (true) {if (signal.aborted) { await reader.cancel().catch(() => {}); return; }@@ -127,13 +131,7 @@ export async function streamOpenAI(opts: {if (data === "[DONE]") break reading;let chunk: OaChunk;try { chunk = JSON.parse(data) as OaChunk; } catch { continue; }- if (chunk.usage) {- usage = {- inputTokens: chunk.usage.prompt_tokens,- outputTokens: chunk.usage.completion_tokens,- totalTokens: chunk.usage.total_tokens,- };- }+ if (chunk.usage) usage = chatCompletionsProviderUsage(chunk.usage);const choice = chunk.choices?.[0];if (!choice) continue;if (choice.delta?.content) emit("delta", choice.delta.content);diff --git a/lib/ai/provider-usage.mjs b/lib/ai/provider-usage.mjsnew file mode 100644index 00000000..c0ed783f--- /dev/null+++ b/lib/ai/provider-usage.mjs@@ -0,0 +1,105 @@+function finite(value) {+ if (value === null || value === undefined || value === "") return undefined;+ const n = Number(value);+ return Number.isFinite(n) && n >= 0 ? n : undefined;+}++function put(out, key, value) {+ const n = finite(value);+ if (n !== undefined) out[key] = n;+}++function coverage(keys) {+ const out = {};+ for (const [name, value] of keys) if (finite(value) !== undefined) out[name] = 1;+ return Object.keys(out).length ? out : undefined;+}++/** OpenAI Responses usage: cache/reasoning details are subsets of input/output. */+export function responsesProviderUsage(raw) {+ if (!raw || typeof raw !== "object") return null;+ const out = { accountingMode: "inclusive-input-output-total", apiCalls: 1 };+ put(out, "inputTokens", raw.input_tokens);+ put(out, "outputTokens", raw.output_tokens);+ put(out, "totalTokens", raw.total_tokens);+ put(out, "cacheReadTokens", raw.input_tokens_details?.cached_tokens);+ put(out, "cacheWriteTokens", raw.input_tokens_details?.cache_write_tokens);+ put(out, "reasoningTokens", raw.output_tokens_details?.reasoning_tokens);+ const detailCoverage = coverage([+ ["cacheReadTokens", raw.input_tokens_details?.cached_tokens],+ ["cacheWriteTokens", raw.input_tokens_details?.cache_write_tokens],+ ["reasoningTokens", raw.output_tokens_details?.reasoning_tokens],+ ]);+ if (detailCoverage) out.detailCoverage = detailCoverage;+ return out;+}++/** OpenAI-compatible Chat Completions usage uses the same inclusive category semantics. */+export function chatCompletionsProviderUsage(raw) {+ if (!raw || typeof raw !== "object") return null;+ const out = { accountingMode: "inclusive-input-output-total", apiCalls: 1 };+ put(out, "inputTokens", raw.prompt_tokens);+ put(out, "outputTokens", raw.completion_tokens);+ put(out, "totalTokens", raw.total_tokens);+ put(out, "cacheReadTokens", raw.prompt_tokens_details?.cached_tokens);+ put(out, "cacheWriteTokens", raw.prompt_tokens_details?.cache_write_tokens);+ put(out, "reasoningTokens", raw.completion_tokens_details?.reasoning_tokens);+ const detailCoverage = coverage([+ ["cacheReadTokens", raw.prompt_tokens_details?.cached_tokens],+ ["cacheWriteTokens", raw.prompt_tokens_details?.cache_write_tokens],+ ["reasoningTokens", raw.completion_tokens_details?.reasoning_tokens],+ ]);+ if (detailCoverage) out.detailCoverage = detailCoverage;+ return out;+}++/** Anthropic reports cache reads/creation as separate input categories. */+export function anthropicProviderUsage(raw) {+ if (!raw || typeof raw !== "object") return null;+ const out = { accountingMode: "separate-cache-input-output", apiCalls: 1 };+ put(out, "inputTokens", raw.input_tokens);+ put(out, "outputTokens", raw.output_tokens);+ put(out, "cacheReadTokens", raw.cache_read_input_tokens);+ put(out, "cacheWriteTokens", raw.cache_creation_input_tokens);+ const detailCoverage = coverage([+ ["cacheReadTokens", raw.cache_read_input_tokens],+ ["cacheWriteTokens", raw.cache_creation_input_tokens],+ ]);+ if (detailCoverage) out.detailCoverage = detailCoverage;+ return out;+}++const SUM_FIELDS = ["inputTokens", "outputTokens", "totalTokens", "cacheReadTokens", "cacheWriteTokens", "reasoningTokens"];+const ALIASES = {+ inputTokens: ["inputTokens", "input_tokens", "prompt_tokens"],+ outputTokens: ["outputTokens", "output_tokens", "completion_tokens"],+ totalTokens: ["totalTokens", "total_tokens"],+ cacheReadTokens: ["cacheReadTokens", "cache_read_tokens", "cacheReadInputTokens", "cache_read_input_tokens"],+ cacheWriteTokens: ["cacheWriteTokens", "cache_write_tokens", "cacheCreationInputTokens", "cache_creation_input_tokens"],+ reasoningTokens: ["reasoningTokens", "reasoning_tokens"],+};+function metric(row, key) {+ for (const alias of ALIASES[key]) { const value = finite(row?.[alias]); if (value !== undefined) return value; }+ return undefined;+}++/** Accumulate provider-reported calls without inventing missing component values. */+export function addProviderUsage(total, next) {+ if (!next || typeof next !== "object") return total ?? { apiCalls: 0 };+ const base = total && typeof total === "object" ? total : { apiCalls: 0 };+ const out = { apiCalls: (finite(base.apiCalls) ?? 0) + (finite(next.apiCalls) ?? 1) };+ for (const key of SUM_FIELDS) {+ const a = metric(base, key), b = metric(next, key);+ if (a !== undefined || b !== undefined) out[key] = (a ?? 0) + (b ?? 0);+ }+ const modes = [base.accountingMode, next.accountingMode].filter(Boolean);+ if (modes.length) out.accountingMode = new Set(modes).size === 1 ? modes[0] : "mixed";+ const cov = {};+ for (const key of ["cacheReadTokens", "cacheWriteTokens", "reasoningTokens"]) {+ const a = finite(base.detailCoverage?.[key]) ?? 0;+ const b = finite(next.detailCoverage?.[key]) ?? (metric(next, key) !== undefined ? finite(next.apiCalls) ?? 1 : 0);+ if (a || b) cov[key] = a + b;+ }+ if (Object.keys(cov).length) out.detailCoverage = cov;+ return out;+}diff --git a/lib/ai/provider-usage.test.ts b/lib/ai/provider-usage.test.tsnew file mode 100644index 00000000..ceca5ac3--- /dev/null+++ b/lib/ai/provider-usage.test.ts@@ -0,0 +1,40 @@+import { describe, expect, it } from "vitest";+import { addProviderUsage, anthropicProviderUsage, chatCompletionsProviderUsage, responsesProviderUsage } from "./provider-usage.mjs";++describe("provider usage normalization", () => {+ it("keeps Responses cache/reasoning as inclusive details without double counting", () => {+ expect(responsesProviderUsage({ input_tokens: 100, input_tokens_details: { cached_tokens: 60, cache_write_tokens: 5 }, output_tokens: 20, output_tokens_details: { reasoning_tokens: 7 }, total_tokens: 120 })).toEqual({+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 100, outputTokens: 20, totalTokens: 120,+ cacheReadTokens: 60, cacheWriteTokens: 5, reasoningTokens: 7,+ detailCoverage: { cacheReadTokens: 1, cacheWriteTokens: 1, reasoningTokens: 1 },+ });+ });++ it("keeps absent provider details absent rather than fabricating zero", () => {+ expect(responsesProviderUsage({ input_tokens: 10, output_tokens: 2, total_tokens: 12 })).toEqual({+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 10, outputTokens: 2, totalTokens: 12,+ });+ expect(anthropicProviderUsage({ input_tokens: 10, output_tokens: 2 })).toEqual({+ accountingMode: "separate-cache-input-output", apiCalls: 1, inputTokens: 10, outputTokens: 2,+ });+ });++ it("does not seed an unknown provider total with zero during accumulation", () => {+ const out = addProviderUsage({ apiCalls: 0 }, anthropicProviderUsage({ input_tokens: 10, output_tokens: 2 }));+ expect(out).toEqual({ apiCalls: 1, inputTokens: 10, outputTokens: 2, accountingMode: "separate-cache-input-output" });+ expect(out.totalTokens).toBeUndefined();+ });++ it("normalizes Chat Completions detail fields with the same inclusive semantics", () => {+ expect(chatCompletionsProviderUsage({ prompt_tokens: 30, prompt_tokens_details: { cached_tokens: 20 }, completion_tokens: 6, completion_tokens_details: { reasoning_tokens: 4 }, total_tokens: 36 })).toMatchObject({+ accountingMode: "inclusive-input-output-total", inputTokens: 30, outputTokens: 6, totalTokens: 36, cacheReadTokens: 20, reasoningTokens: 4,+ });+ });++ it("accumulates calls while preserving detail coverage and mixed semantics", () => {+ const a = responsesProviderUsage({ input_tokens: 10, input_tokens_details: { cached_tokens: 3 }, output_tokens: 2, total_tokens: 12 });+ const b = responsesProviderUsage({ input_tokens: 20, input_tokens_details: { cached_tokens: 4 }, output_tokens: 3, total_tokens: 23 });+ expect(addProviderUsage(addProviderUsage(null, a), b)).toMatchObject({ apiCalls: 2, inputTokens: 30, outputTokens: 5, totalTokens: 35, cacheReadTokens: 7, accountingMode: "inclusive-input-output-total", detailCoverage: { cacheReadTokens: 2 } });+ expect(addProviderUsage(a, anthropicProviderUsage({ input_tokens: 1, output_tokens: 1 }))).toMatchObject({ accountingMode: "mixed" });+ });+});diff --git a/lib/ai/usage.ts b/lib/ai/usage.tsindex d3baa67e..c7b85678 100644--- a/lib/ai/usage.ts+++ b/lib/ai/usage.ts@@ -1,13 +1,9 @@+import { anthropicProviderUsage } from "./provider-usage.mjs";import type Anthropic from "@anthropic-ai/sdk";export function anthropicDonePayload(final: Anthropic.Message) {return {stopReason: final.stop_reason,- usage: {- inputTokens: final.usage.input_tokens,- outputTokens: final.usage.output_tokens,- cacheCreationInputTokens: final.usage.cache_creation_input_tokens ?? 0,- cacheReadInputTokens: final.usage.cache_read_input_tokens ?? 0,- },+ usage: anthropicProviderUsage(final.usage),};}diff --git a/scripts/bench-agent-corpus.mjs b/scripts/bench-agent-corpus.mjsindex 151c54e5..eced25b6 100644--- a/scripts/bench-agent-corpus.mjs+++ b/scripts/bench-agent-corpus.mjs@@ -69,7 +69,9 @@ export function summarizeCorpus(rows, requested) {const expectedScenarios = new Set(rows.map((row) => row.scenarioId)).size;const tokenModes = new Set(aggregates.map((row) => row.tokenAccountingMode));const tokenSemanticsComparable = aggregates.length >= 2- && aggregates.every((row) => row.tokenCoveragePct === 100 && !["unknown", "mixed"].includes(row.tokenAccountingMode))+ && aggregates.every((row) => row.tokenCoveragePct === 100+ && row.tokenAccountingMode === "inclusive-input-output-total"+ && !["unknown", "mixed"].includes(row.tokenAccountingProof))&& tokenModes.size === 1;const costPairs = new Set(aggregates.map((row) => `${row.costStatus}:${row.costSource}`));const costSemanticsComparable = aggregates.length >= 2@@ -77,7 +79,10 @@ export function summarizeCorpus(rows, requested) {&& costPairs.size === 1;return {aggregates, comparability,- efficiencyComparability: { tokenSemanticsComparable, costSemanticsComparable, tokenAccountingModes: [...tokenModes] },+ efficiencyComparability: {+ tokenSemanticsComparable, costSemanticsComparable, tokenAccountingModes: [...tokenModes],+ tokenAccountingProofs: Object.fromEntries(aggregates.map((row) => [row.agent, row.tokenAccountingProof])),+ },ranking: eligibleRanking(aggregates, comparability, expectedScenarios),};}diff --git a/scripts/bench-agent-corpus.test.ts b/scripts/bench-agent-corpus.test.tsindex c478299e..d2e80a1f 100644--- a/scripts/bench-agent-corpus.test.ts+++ b/scripts/bench-agent-corpus.test.ts@@ -31,8 +31,9 @@ describe("P4 agent-quality corpus", () => {it("keeps missing usage and cost unknown rather than converting null to zero", () => {expect(extractUsage({ usage: { input_tokens: null, output_tokens: null, cost_usd: null } })).toBeNull();- expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })).toEqual({ inputTokens: 10, outputTokens: 5, totalTokens: 15, unattributedTokens: 0, accountingMode: "input-output-total" });- expect(extractUsage({ input_tokens: 10, output_tokens: 5, reasoning_tokens: 7, total_tokens: 22, cache_read_tokens: 3, cost_status: "estimated", cost_source: "catalog" })).toMatchObject({ reasoningTokens: 7, cacheReadTokens: 3, unattributedTokens: 7, accountingMode: "expanded-components", costStatus: "estimated", costSource: "catalog" });+ expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })).toMatchObject({ inputTokens: 10, outputTokens: 5, reportedAccountingMode: "unspecified", accountingMode: "opaque-total", accountingProof: "none" });+ expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })?.totalTokens).toBeUndefined();+ expect(extractUsage({ input_tokens: 10, output_tokens: 5, reasoning_tokens: 7, total_tokens: 22, cache_read_tokens: 3, cost_status: "estimated", cost_source: "catalog" })).toMatchObject({ reasoningTokens: 7, cacheReadTokens: 3, accountingMode: "opaque-total", costStatus: "estimated", costSource: "catalog" });});it("extracts model/provider evidence but withholds full comparability when provider differs", () => {@@ -57,7 +58,7 @@ describe("P4 agent-quality corpus", () => {it("aggregates token/cost only from rows that actually report them", () => {const rows = [- { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, estimatedCostUsd: 0.01 }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },+ { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, normalizedTotalTokens: 100, accountingMode: "inclusive-input-output-total", estimatedCostUsd: 0.01 }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },{ fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 20, usage: null, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },];const out = aggregateAgent("mso", rows);@@ -97,8 +98,8 @@ describe("P4 agent-quality corpus", () => {it("charges failed-attempt tokens into tokens-per-success when coverage is complete", () => {const rows = [- { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, accountingMode: "input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },- { fullSuccess: false, taskSuccess: false, policyCompliant: true, latencyMs: 20, usage: { totalTokens: 300, accountingMode: "input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },+ { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, normalizedTotalTokens: 100, accountingMode: "inclusive-input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },+ { fullSuccess: false, taskSuccess: false, policyCompliant: true, latencyMs: 20, usage: { totalTokens: 300, normalizedTotalTokens: 300, accountingMode: "inclusive-input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },];const out = aggregateAgent("mso", rows);expect(out.tokenCoveragePct).toBe(100); expect(out.totalReportedTokens).toBe(400); expect(out.tokensPerSuccessfulTask).toBe(400);@@ -147,14 +148,41 @@ describe("P4 agent-quality corpus", () => {expect(out.status).toBe(0); expect(out.stdout).toContain("1 scenarios × 1 agents");});- it("keeps model/provider comparability separate from incompatible token accounting semantics", () => {+ it("derives a canonical total for explicit separate-component semantics without inventing a raw provider total", () => {+ const usage = extractUsage({ inputTokens: 10, outputTokens: 2, cacheReadTokens: 3, accountingMode: "separate-cache-input-output" })!;+ expect(usage.totalTokens).toBeUndefined();+ expect(usage).toMatchObject({ normalizedInputTokens: 13, normalizedOutputTokens: 2, normalizedTotalTokens: 15, accountingMode: "inclusive-input-output-total", accountingProof: "explicit-separate-components" });+ });++ it("normalizes exact Hermes-style exclusive components to the same canonical token semantics as MSO", () => {+ const mso = extractUsage({ inputTokens: 100, outputTokens: 20, totalTokens: 120, cacheReadTokens: 60, reasoningTokens: 7, accountingMode: "inclusive-input-output-total" })!;+ const hermes = extractUsage({ input_tokens: 40, cache_read_tokens: 60, cache_write_tokens: 0, output_tokens: 20, reasoning_tokens: 7, total_tokens: 120 })!;+ expect(mso).toMatchObject({ accountingMode: "inclusive-input-output-total", accountingProof: "explicit-inclusive-contract", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });+ expect(hermes).toMatchObject({ reportedAccountingMode: "exclusive-cache-inclusive-output", accountingMode: "inclusive-input-output-total", accountingProof: "exact-exclusive-cache-sum", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });+ const rows = [+ { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: mso, modelEvidence: { modelFamily: "x", provider: "p" } },+ { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: hermes, modelEvidence: { modelFamily: "x", provider: "p" } },+ ];+ expect(summarizeCorpus(rows as any, { model: "x", provider: "p" }).efficiencyComparability.tokenSemanticsComparable).toBe(true);+ });++ it("can also normalize an exact representation where reasoning is outside output", () => {+ const usage = extractUsage({ input_tokens: 40, cache_read_tokens: 60, output_tokens: 13, reasoning_tokens: 7, total_tokens: 120 })!;+ expect(usage).toMatchObject({ reportedAccountingMode: "exclusive-cache-reasoning", accountingProof: "exact-exclusive-cache-reasoning-sum", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });+ });++ it("does not treat two equally opaque totals as comparable semantics", () => {+ const opaque = extractUsage({ input_tokens: 10, output_tokens: 5, cache_read_tokens: 3, reasoning_tokens: 7, total_tokens: 22 })!;const rows = [- { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 10, accountingMode: "input-output-total" }, modelEvidence: { modelFamily: "x", provider: "p" } },- { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: { totalTokens: 12, accountingMode: "expanded-components" }, modelEvidence: { modelFamily: "x", provider: "p" } },+ { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: opaque, modelEvidence: { modelFamily: "x", provider: "p" } },+ { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: opaque, modelEvidence: { modelFamily: "x", provider: "p" } },];- const out = summarizeCorpus(rows as any, { model: "x", provider: "p" });- expect(out.comparability.level).toBe("full");- expect(out.efficiencyComparability.tokenSemanticsComparable).toBe(false);+ expect(summarizeCorpus(rows as any, { model: "x", provider: "p" }).efficiencyComparability.tokenSemanticsComparable).toBe(false);+ });++ it("withholds token comparability when expanded-component arithmetic does not prove the representation", () => {+ const ambiguous = extractUsage({ input_tokens: 10, output_tokens: 5, cache_read_tokens: 3, reasoning_tokens: 7, total_tokens: 22 })!;+ expect(ambiguous.accountingMode).toBe("opaque-total");});});diff --git a/scripts/bench-agent-metrics.mjs b/scripts/bench-agent-metrics.mjsindex 5bacc2db..17f4a77a 100644--- a/scripts/bench-agent-metrics.mjs+++ b/scripts/bench-agent-metrics.mjs@@ -46,10 +46,11 @@ export function extractUsage(value) {const input = finite(row.inputTokens ?? row.input_tokens ?? row.prompt_tokens ?? row.promptTokens);const output = finite(row.outputTokens ?? row.output_tokens ?? row.completion_tokens ?? row.completionTokens);const reasoning = finite(row.reasoningTokens ?? row.reasoning_tokens);- const cacheRead = finite(row.cacheReadTokens ?? row.cache_read_tokens);- const cacheWrite = finite(row.cacheWriteTokens ?? row.cache_write_tokens);+ const cacheRead = finite(row.cacheReadTokens ?? row.cache_read_tokens ?? row.cacheReadInputTokens ?? row.cache_read_input_tokens);+ const cacheWrite = finite(row.cacheWriteTokens ?? row.cache_write_tokens ?? row.cacheCreationInputTokens ?? row.cache_creation_input_tokens);const apiCalls = finite(row.apiCalls ?? row.api_calls);- const total = finite(row.totalTokens ?? row.total_tokens) ?? (input !== undefined && output !== undefined ? input + output : undefined);+ const total = finite(row.totalTokens ?? row.total_tokens);+ const explicitMode = typeof (row.accountingMode ?? row.accounting_mode) === "string" ? String(row.accountingMode ?? row.accounting_mode) : null;const explicitUsd = finite(row.estimatedCostUsd ?? row.estimated_cost_usd ?? row.costUsd ?? row.cost_usd);const currency = typeof (row.currency ?? row.costCurrency ?? row.cost_currency) === "string"? String(row.currency ?? row.costCurrency ?? row.cost_currency).toLowerCase() : undefined;@@ -58,15 +59,48 @@ export function extractUsage(value) {const costStatus = typeof (row.costStatus ?? row.cost_status) === "string" ? String(row.costStatus ?? row.cost_status) : undefined;const costSource = typeof (row.costSource ?? row.cost_source) === "string" ? String(row.costSource ?? row.cost_source) : undefined;if (input === undefined && output === undefined && reasoning === undefined && total === undefined && cost === undefined) return null;- const unattributed = total !== undefined && input !== undefined && output !== undefined ? Math.max(0, total - input - output) : undefined;- const accountingMode = reasoning !== undefined || cacheRead !== undefined || cacheWrite !== undefined ? "expanded-components"- : total !== undefined && input !== undefined && output !== undefined && total === input + output ? "input-output-total" : "opaque-total";++ let normalizedInput, normalizedOutput, normalizedTotal, accountingMode = "opaque-total", reportedAccountingMode = explicitMode || "unspecified", accountingProof = "none";+ if (input !== undefined && output !== undefined && (total !== undefined || explicitMode === "separate-cache-input-output")) {+ const detailSum = (cacheRead ?? 0) + (cacheWrite ?? 0) + (reasoning ?? 0);+ const exclusiveSum = input + output + detailSum;+ if (explicitMode === "inclusive-input-output-total") {+ normalizedInput = input; normalizedOutput = output; normalizedTotal = total; accountingMode = "inclusive-input-output-total"; accountingProof = "explicit-inclusive-contract";+ } else if (explicitMode === "separate-cache-input-output") {+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);+ normalizedOutput = output + (reasoning ?? 0); normalizedTotal = total ?? (normalizedInput + normalizedOutput);+ if (normalizedTotal === normalizedInput + normalizedOutput) { accountingMode = "inclusive-input-output-total"; accountingProof = "explicit-separate-components"; }+ } else if (((cacheRead ?? 0) + (cacheWrite ?? 0)) > 0 && total === input + output + (cacheRead ?? 0) + (cacheWrite ?? 0)) {+ // Hermes openai-codex shape observed in the live corpus: cache is outside base input,+ // while reasoning remains a detail already included in output. Exact arithmetic is required.+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);+ normalizedOutput = output; normalizedTotal = total;+ accountingMode = "inclusive-input-output-total"; reportedAccountingMode = "exclusive-cache-inclusive-output"; accountingProof = "exact-exclusive-cache-sum";+ } else if (detailSum > 0 && total === exclusiveSum) {+ // Alternate expanded representation where both cache and reasoning live outside base categories.+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);+ normalizedOutput = output + (reasoning ?? 0); normalizedTotal = total;+ accountingMode = "inclusive-input-output-total"; reportedAccountingMode = "exclusive-cache-reasoning"; accountingProof = "exact-exclusive-cache-reasoning-sum";+ } else if (total === input + output) {+ // No extra tokens live outside the provider's input/output categories.+ normalizedInput = input; normalizedOutput = output; normalizedTotal = total;+ accountingMode = "inclusive-input-output-total"; accountingProof = detailSum > 0 ? "exact-inclusive-total-identity" : "exact-input-output-identity";+ if (!explicitMode) reportedAccountingMode = detailSum > 0 ? "inclusive-expanded-components" : "input-output-total";+ }+ }+ const unattributed = normalizedTotal !== undefined && normalizedInput !== undefined && normalizedOutput !== undefined+ ? Math.max(0, normalizedTotal - normalizedInput - normalizedOutput) : undefined;return {...(input !== undefined ? { inputTokens: input } : {}), ...(output !== undefined ? { outputTokens: output } : {}),...(reasoning !== undefined ? { reasoningTokens: reasoning } : {}), ...(cacheRead !== undefined ? { cacheReadTokens: cacheRead } : {}),...(cacheWrite !== undefined ? { cacheWriteTokens: cacheWrite } : {}), ...(apiCalls !== undefined ? { apiCalls } : {}),- ...(total !== undefined ? { totalTokens: total } : {}), ...(unattributed !== undefined ? { unattributedTokens: unattributed } : {}),- ...(cost !== undefined ? { estimatedCostUsd: cost } : {}), ...(costStatus ? { costStatus } : {}), ...(costSource ? { costSource } : {}), accountingMode,+ ...(total !== undefined ? { totalTokens: total } : {}),+ ...(normalizedInput !== undefined ? { normalizedInputTokens: normalizedInput } : {}),+ ...(normalizedOutput !== undefined ? { normalizedOutputTokens: normalizedOutput } : {}),+ ...(normalizedTotal !== undefined ? { normalizedTotalTokens: normalizedTotal } : {}),+ ...(unattributed !== undefined ? { unattributedTokens: unattributed } : {}),+ ...(cost !== undefined ? { estimatedCostUsd: cost } : {}), ...(costStatus ? { costStatus } : {}), ...(costSource ? { costSource } : {}),+ reportedAccountingMode, accountingMode, accountingProof,};});}@@ -98,10 +132,11 @@ function median(values) { if (!values.length) return null; const s = [...values]export function aggregateAgent(agent, scenarios) {const full = scenarios.filter((row) => row.fullSuccess), task = scenarios.filter((row) => row.taskSuccess), policy = scenarios.filter((row) => row.policyCompliant);- const usageRows = scenarios.filter((row) => row.usage?.totalTokens !== undefined);+ const usageRows = scenarios.filter((row) => row.usage?.normalizedTotalTokens !== undefined);const costRows = scenarios.filter((row) => row.usage?.estimatedCostUsd !== undefined);const toolRows = scenarios.filter((row) => row.toolTelemetry?.count !== undefined);const accountingModes = new Set(scenarios.map((row) => row.usage?.accountingMode).filter(Boolean));+ const accountingProofs = new Set(scenarios.map((row) => row.usage?.accountingProof).filter((value) => value && value !== "none"));const costStatuses = new Set(scenarios.map((row) => row.usage?.costStatus).filter(Boolean));const costSources = new Set(scenarios.map((row) => row.usage?.costSource).filter(Boolean));const evidenced = scenarios.filter((row) => row.modelEvidence?.modelFamily);@@ -114,7 +149,7 @@ export function aggregateAgent(agent, scenarios) {consistent: families.size === 1 && providers.size <= 1,} : null;const tokenCoveragePct = pct(usageRows.length, scenarios.length), costCoveragePct = pct(costRows.length, scenarios.length);- const totalReportedTokens = usageRows.reduce((sum, row) => sum + row.usage.totalTokens, 0);+ const totalReportedTokens = usageRows.reduce((sum, row) => sum + row.usage.normalizedTotalTokens, 0);const totalReportedCostUsd = costRows.reduce((sum, row) => sum + row.usage.estimatedCostUsd, 0);return {agent, attempted: scenarios.length, taskSuccesses: task.length, policyCompliant: policy.length, fullSuccesses: full.length,@@ -123,12 +158,13 @@ export function aggregateAgent(agent, scenarios) {tokenCoveragePct, costCoveragePct,totalReportedTokens: usageRows.length ? totalReportedTokens : null,totalReportedCostUsd: costRows.length ? Math.round(totalReportedCostUsd * 1e9) / 1e9 : null,- reportedTokensPerAttempt: usageRows.length ? average(usageRows.map((row) => row.usage.totalTokens)) : null,+ reportedTokensPerAttempt: usageRows.length ? average(usageRows.map((row) => row.usage.normalizedTotalTokens)) : null,reportedCostPerAttemptUsd: costRows.length ? average(costRows.map((row) => row.usage.estimatedCostUsd), 6) : null,toolTelemetryCoveragePct: pct(toolRows.length, scenarios.length),averageToolCallsPerTask: toolRows.length ? average(toolRows.map((row) => row.toolTelemetry.count)) : null,failedToolCalls: toolRows.reduce((sum, row) => sum + (row.toolTelemetry.failed || 0), 0),tokenAccountingMode: accountingModes.size === 1 ? [...accountingModes][0] : accountingModes.size ? "mixed" : "unknown",+ tokenAccountingProof: accountingProofs.size === 1 ? [...accountingProofs][0] : accountingProofs.size ? "mixed" : "unknown",costStatus: costStatuses.size === 1 ? [...costStatuses][0] : costStatuses.size ? "mixed" : "unknown",costSource: costSources.size === 1 ? [...costSources][0] : costSources.size ? "mixed" : "unknown",modelEvidenceCoveragePct: pct(evidenced.length, scenarios.length),@@ -144,5 +180,5 @@ export function eligibleRanking(aggregates, comparability, expectedScenarios = nif (aggregates.length < 2 || aggregates.some((row) => row.attempted < 1)) return { eligible: false, reason: "every compared agent must produce corpus results" };if (expectedScenarios != null && aggregates.some((row) => row.attempted !== expectedScenarios)) return { eligible: false, reason: "every compared agent must cover the same complete corpus" };const ordered = [...aggregates].sort((a, b) => b.fullSuccessPct - a.fullSuccessPct || b.policyCompliancePct - a.policyCompliancePct || (a.p50LatencyMs ?? Infinity) - (b.p50LatencyMs ?? Infinity) || (a.averageLatencyMs ?? Infinity) - (b.averageLatencyMs ?? Infinity));- return { eligible: true, order: ordered.map((row) => row.agent), note: "Order prioritizes task success, then policy compliance, then p50/average latency as a descriptive tie-breaker. Token/cost efficiency is reported separately until usage semantics are explicitly proven comparable." };+ return { eligible: true, order: ordered.map((row) => row.agent), note: "Order prioritizes task success, then policy compliance, then p50/average latency as a descriptive tie-breaker. Efficiency stays a separate metric; token/cost values are comparable only when their independent accounting gates pass." };}diff --git a/scripts/mso-agent-session-ui-state.test.ts b/scripts/mso-agent-session-ui-state.test.tsindex 696615de..f1a62614 100644--- a/scripts/mso-agent-session-ui-state.test.ts+++ b/scripts/mso-agent-session-ui-state.test.ts@@ -24,7 +24,7 @@ describe("MSO Agent /new session transition", () => {expect(session.activeSkill).toBeNull();expect(session.lastInvokedSkill).toBeNull();expect(session.titleOverride).toBeNull();- expect(session.usage).toEqual({ inputTokens: 0, outputTokens: 0, totalTokens: 0 });+ expect(session.usage).toEqual({ apiCalls: 0 });expect(session.lastElapsedMs).toBe(0);});diff --git a/scripts/mso-agent-session-ui.mjs b/scripts/mso-agent-session-ui.mjsindex ea6b5d5c..bbdd62f8 100644--- a/scripts/mso-agent-session-ui.mjs+++ b/scripts/mso-agent-session-ui.mjs@@ -44,7 +44,7 @@ export function applyNewSessionState(session, loaded, requestedTitle = "") {activeSkill: null,lastInvokedSkill: null,titleOverride: requestedTitle ? (loaded.title || requestedTitle) : null,- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },+ usage: { apiCalls: 0 },lastElapsedMs: 0,pendingApproval: null,});@@ -72,7 +72,7 @@ export async function resumeInto(rl, session, query) {activeSkill: null,lastInvokedSkill: null,titleOverride: loaded.title || null,- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },+ usage: { apiCalls: 0 },lastElapsedMs: 0,pendingApproval: null,});diff --git a/scripts/mso-agent-status.mjs b/scripts/mso-agent-status.mjsindex d9a1243b..b1943023 100644--- a/scripts/mso-agent-status.mjs+++ b/scripts/mso-agent-status.mjs@@ -1,6 +1,7 @@import os from "node:os";import path from "node:path";import { projectHistoryForModel } from "./mso-agent-context.mjs";+import { addProviderUsage } from "../lib/ai/provider-usage.mjs";function compactNumber(value) {const n = Number(value || 0);@@ -34,18 +35,7 @@ export function contextStatus(history, modelMeta) {};}-export function addUsage(total, next) {- const base = total || { inputTokens: 0, outputTokens: 0, totalTokens: 0 };- if (!next || typeof next !== "object") return base;- const input = Number(next.inputTokens ?? next.input_tokens ?? 0) || 0;- const output = Number(next.outputTokens ?? next.output_tokens ?? 0) || 0;- const explicit = Number(next.totalTokens ?? next.total_tokens ?? 0) || 0;- return {- inputTokens: base.inputTokens + input,- outputTokens: base.outputTokens + output,- totalTokens: base.totalTokens + (explicit || input + output),- };-}+export function addUsage(total, next) { return addProviderUsage(total, next); }function compactLabel(value, max = 28) {@@ -84,7 +74,8 @@ export function statusParts(session, cwd = process.cwd()) {skill ? `skill ${skill.state === "invoked" ? "✓" : "◆"} /${skill.name}${skill.state === "queued" ? " queued" : skill.state === "invoking" ? " invoking" : ""}` : null,ctx.limit ? `ctx ~${compactNumber(ctx.used)}/${compactNumber(ctx.limit)} ${ctx.percent}%` : `ctx ~${compactNumber(ctx.used)}/?`,session?.agentSession?.compactThresholdTokens ? `session ~${compactNumber(session.agentSession.estimatedTokens)}/${compactNumber(session.agentSession.compactThresholdTokens)}` : null,- usage.totalTokens > 0 ? `tokens ${compactNumber(usage.totalTokens)}` : null,+ (usage.totalTokens ?? 0) > 0 ? `tokens ${compactNumber(usage.totalTokens)}`+ : ((usage.inputTokens ?? 0) > 0 || (usage.outputTokens ?? 0) > 0) ? `tokens ${compactNumber(usage.inputTokens)}in/${compactNumber(usage.outputTokens)}out` : null,routing?.activeTools != null ? `route ${routing.routeIds?.join("+") || (routing.fallbackUsed ? "fallback" : "direct")} · tools ${routing.activeTools}/${routing.fullTools}` : null,`turns ${turns}`,elapsed > 0 ? `${(elapsed / 1000).toFixed(elapsed >= 10_000 ? 0 : 1)}s` : null,@@ -98,7 +89,7 @@ export function statusParts(session, cwd = process.cwd()) {function detailedStatus(session, cwd = process.cwd()) {const cfg = session?.state?.config || {};const ctx = contextStatus(session?.history, session?.state?.modelMeta);- const usage = session?.usage || { inputTokens: 0, outputTokens: 0, totalTokens: 0 };+ const usage = session?.usage || { apiCalls: 0 };const providerMeta = (cfg.providers || []).find((row) => row.id === cfg.provider);return {provider: cfg.provider || null,@@ -132,8 +123,18 @@ export function printDetailedStatus(session, C, cwd = process.cwd()) {console.log(` auth ${row.providerAuth}`);console.log(` context ~${row.contextEstimatedTokens}${row.contextWindow ? ` / ${row.contextWindow} (${row.contextPercent}%)` : " / ?"}${row.contextOmittedRows ? ` · ${row.contextOmittedRows} older rows projected out` : ""}`);if (row.sessionCompactThresholdTokens) console.log(` session ~${row.sessionEstimatedTokens} / ${row.sessionCompactThresholdTokens} stored · lifetime ~${row.sessionLifetimeEstimatedTokens} · ${row.sessionCompactionCount} compactions · ${row.sessionArchiveCount} archives`);- if (row.providerReportedUsage.totalTokens > 0) {- console.log(` tokens ${row.providerReportedUsage.totalTokens} total · ${row.providerReportedUsage.inputTokens} in · ${row.providerReportedUsage.outputTokens} out`);+ if ((row.providerReportedUsage.totalTokens ?? 0) > 0 || (row.providerReportedUsage.inputTokens ?? 0) > 0 || (row.providerReportedUsage.outputTokens ?? 0) > 0) {+ const u = row.providerReportedUsage;+ const parts = [+ u.totalTokens != null ? `${u.totalTokens} total` : null,+ u.inputTokens != null ? `${u.inputTokens} in` : null,+ u.outputTokens != null ? `${u.outputTokens} out` : null,+ u.cacheReadTokens != null ? `${u.cacheReadTokens} cache-read` : null,+ u.cacheWriteTokens != null ? `${u.cacheWriteTokens} cache-write` : null,+ u.reasoningTokens != null ? `${u.reasoningTokens} reasoning` : null,+ u.accountingMode ? `mode ${u.accountingMode}` : null,+ ].filter(Boolean);+ console.log(` tokens ${parts.join(" · ")}`);} else console.log(" tokens provider has not reported usage in this process; context remains estimated");if (row.skill) console.log(` skill ${row.skill.state} /${row.skill.name}`);console.log(` permission ${row.permission}`);diff --git a/scripts/mso-agent-status.test.ts b/scripts/mso-agent-status.test.tsindex ce822311..5f2feb46 100644--- a/scripts/mso-agent-status.test.ts+++ b/scripts/mso-agent-status.test.ts@@ -12,13 +12,13 @@ describe("MSO Agent dynamic status", () => {it("accumulates provider-reported usage without losing fallback status", () => {const usage = addUsage(addUsage(null, { inputTokens: 10, outputTokens: 4 }), { input_tokens: 6, output_tokens: 2 });- expect(usage).toEqual({ inputTokens: 16, outputTokens: 6, totalTokens: 22 });+ expect(usage).toEqual({ apiCalls: 2, inputTokens: 16, outputTokens: 6 });const parts = statusParts({state: { config: { provider: "openai-codex", model: "gpt-5.6-sol" }, modelMeta: { context: 128000 } },history: [{ role: "user", text: "hello" }], usage, agentSession: { id: "20260901_120000_deadbeef" },}, "/tmp");expect(parts.join(" ")).toContain("openai-codex/gpt-5.6-sol");- expect(parts.join(" ")).toContain("tokens 22");+ expect(parts.join(" ")).toContain("tokens 16in/6out");});it("shows whether a skill is queued or really invoked", () => {diff --git a/scripts/mso-agent.mjs b/scripts/mso-agent.mjsindex 159d69b2..555e8d3d 100644--- a/scripts/mso-agent.mjs+++ b/scripts/mso-agent.mjs@@ -23,7 +23,7 @@ async function runOneShot(opts) {state: s, agentSession, history: Array.isArray(agentSession.history) ? agentSession.history : [],pendingSkill: null, activeSkill: null, lastInvokedSkill: null,titleOverride: requested ? (agentSession.title || null) : null,- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 }, lastRouting: null, lastElapsedMs: 0, statusBar: false,+ usage: { apiCalls: 0 }, lastRouting: null, lastElapsedMs: 0, statusBar: false,permission: "ask", pendingApproval: null,};session.history.push({ role: "user", text: opts.prompt });@@ -65,7 +65,7 @@ async function main() {history: Array.isArray(agentSession.history) ? agentSession.history : [],pendingSkill: null, activeSkill: null, lastInvokedSkill: null,titleOverride: (requested || restartSessionId) ? (agentSession.title || null) : null,- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 }, lastRouting: null, lastElapsedMs: 0,+ usage: { apiCalls: 0 }, lastRouting: null, lastElapsedMs: 0,statusBar: restartUi.statusBar ?? true, permission: forcedPermission, pendingApproval: null,};const rl = new AgentComposer({ input: process.stdin, output: process.stdout, colors: C }); syncPromptHistory(rl, session);