Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

perf(agent): normalize provider usage semantics

Rahman3b3d1552119c
diff --git a/README.md b/README.md
index c142f432..eb16f417 100644
--- a/README.md
+++ b/README.md
@@ -79,7 +79,7 @@ For a real deployment, put MSO behind **Tailscale, a VPN, or a TLS reverse proxy
- **Message live local sessions by a short name** — every durable same-principal MSO session gets a unique familiar handle such as `@milo`, `@luna`, or `@nara`, independent from its longer auto-generated session title. `/rename <name>` changes that handle, while `/title <text>` changes the session description. `@milo …` resolves only agents with a current lease **and** a live receiver subscription, then creates an explicitly correlated request; replies relay deterministically into the originating conversation while notify-only events remain passive. Presence lease (`idle`/`busy`/`offline`) and receiver subscription (`consumerConnected`) are reported separately so an idle-but-unsubscribed target is diagnosable. For ChatGPT/MCP sessions, `local_agent_inbox(wait_ms=...)` temporarily turns the foreground MCP call into that live receiver for up to 20 seconds, enabling direct two-way session messaging without a worker, DB, webhook, or broker. See [Local Agents](./docs/LOCAL-AGENTS.md).
- **Spawn focused same-session subagents** — `/spawn` and `agent_subagent_run` create bounded foreground workers with isolated context, explicit delegated scope, turn/timeout caps, no recursive spawn, and final-result-only return. They are not Local Agent peers and do not run autonomously after the parent call returns. See [Subagents](./docs/SUBAGENTS.md).
- **Delegate between remote agents with standard A2A v1** — register public HTTPS peers, attach private API-key/Bearer/OAuth access-token profiles, stream remote tasks over SSE, or expose MSO's authenticated inbound Agent Card when a public HTTPS origin is configured. Inbound credentials are owner-minted `read | write | exec` capabilities and run in a memory-isolated task context. See [A2A](./docs/A2A.md).
-- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:pipeline`, and isolated scratch-fixture `bun run bench:corpus` guard routing, correctness, policy compliance, latency and context savings.
+- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:pipeline`, and isolated scratch-fixture `bun run bench:corpus` guard routing, correctness, policy compliance, latency, context savings, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.
- **Make orchestration risk-aware and self-improving** — [RASMIC](./docs/RASMIC.md) classifies task risk/contention, retrieves compact repo-local memory, records manual user tests and Evidence Receipts, detects path/shared-resource collisions before merge conflicts, and promotes repeated safe routes from observed traces → recipes → tested bounded scripts.
- **Add app slices** — features are modular under `frontend/slices/<slug>/`.
- **Personalize the interface** — macOS, Windows, iOS, and Android shell layouts are UI preferences, not the core product.
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 482263fb..22b622eb 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -30,6 +30,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.
**Faster**
+- `agent` normalize provider usage semantics
- `agent` add comparable quality corpus
- `agent` route capabilities before model reasoning
diff --git a/docs/COGNITIVE-RUNTIME.md b/docs/COGNITIVE-RUNTIME.md
index 4590145c..a61c8a80 100644
--- a/docs/COGNITIVE-RUNTIME.md
+++ b/docs/COGNITIVE-RUNTIME.md
@@ -201,7 +201,7 @@ Comparability is fail-closed: every attempted scenario must report matching mode
### Token/cost accounting is deliberately separate
-The reported token totals are **not used to rank the agents**. MSO currently reports provider `input/output/total` usage (`input-output-total` accounting), while Hermes' one-shot report exposes expanded input/output/cache/reasoning components (`expanded-components`). P4 therefore records `tokenSemanticsComparable=false` even though both have 100% token coverage. Failed attempts, when present, are charged into tokens/cost-per-success instead of disappearing from the denominator; partial coverage leaves per-success efficiency unknown. Cost comparison is also withheld: MSO reports no comparable cost field, while the local Hermes report marks cost as included with source `none`; an ambiguous generic `cost` is not assumed to be USD unless currency is explicit. Missing or semantically incompatible accounting remains **unknown/non-comparable**, never zero-efficiency evidence.
+P4 deliberately withheld token ranking because MSO exposed only aggregate `input/output/total` while Hermes exposed cache/reasoning components with a different representation. P5 resolves that observability gap below without rewriting historical P4 results. Failed attempts, when present, are still charged into token/cost-per-success instead of disappearing from the denominator; partial/opaque accounting remains unknown. Cost comparison remains withheld unless both runners expose the same attributable USD semantics.
### P4 found and fixed a real RASMIC inefficiency
@@ -211,15 +211,47 @@ The final `multi-read` scenario uses exactly one `read_pipeline` call. Recovery
### OpenClaw status
-OpenClaw remains installed and testable, but the current host does not expose an equivalent usable `openai-codex/gpt-5.6-terra` path for its main agent. P4 **does not change competitor credentials/configuration just to manufacture comparability**, and it therefore emits no OpenClaw quality ranking yet.
+OpenClaw remains installed and testable, but the current host does not expose an equivalent usable `openai-codex/gpt-5.6-terra` path for its main agent. P4/P5 **do not change competitor credentials/configuration just to manufacture comparability**, and they therefore emit no OpenClaw quality ranking yet.
-## P4 next direction
+## P5 normalized provider-usage semantics
+
+P5 makes token accounting auditable without adding prompt context or inventing missing values. A provider-neutral usage envelope now preserves the provider's raw categories plus an explicit accounting mode:
+
+- OpenAI Responses / Codex and compatible Chat Completions: `inputTokens`, `outputTokens`, authoritative `totalTokens`, optional `cacheReadTokens`, `cacheWriteTokens`, and `reasoningTokens`, with `accountingMode=inclusive-input-output-total`. Cache/reasoning are details already included in the input/output categories; they are **not added again**.
+- Anthropic: input/output plus optional cache-read/cache-creation categories with `accountingMode=separate-cache-input-output`; absent fields remain absent. A benchmark-only canonical total may be derived from this explicit component contract, while the raw provider total remains unknown if Anthropic did not report one.
+- Agent session aggregation tracks `apiCalls`, accounting mode, and detail coverage. A new session starts at `{apiCalls:0}` instead of fake zero token totals, so an unknown provider field cannot silently become `0`.
+
+The benchmark converts representations to canonical `inclusive-input-output-total` only with proof. `accountingProof` records why the conversion is allowed. In the isolated candidate corpus:
+
+- MSO: `explicit-inclusive-contract` from the actual Codex `response.completed.usage` payload, including provider-reported cache/reasoning details.
+- Hermes: `exact-exclusive-cache-sum`; on every corpus scenario, arithmetic proves `total = base input + cache-read/cache-write + output`, while `reasoning` is already a subset of output. If that identity does not hold, the row stays `opaque-total`.
+- Two equally opaque totals are **not** considered comparable merely because their labels match. Token comparability additionally requires 100% coverage, canonical recognized mode, and non-unknown proof for every agent.
+
+A full six-scenario candidate run on the same `openai-codex/gpt-5.6-terra` provider/model preserved the quality result while making token totals comparable:
+
+| P5 metric | MSO | Hermes |
+|---|---:|---:|
+| Task success | **6/6 (100%)** | **6/6 (100%)** |
+| Scenario-observable policy compliance | **6/6 (100%)** | **6/6 (100%)** |
+| Average latency | **11,497.7 ms** | 16,199.3 ms |
+| p50 latency | **11,782 ms** | 16,184.5 ms |
+| Normalized token coverage | **100%** | **100%** |
+| Normalized tokens / successful task | **3,510.8** | 59,743.2 |
+| Accounting proof | explicit inclusive contract | exact exclusive-cache sum |
+| Cost semantics comparable | no | no |
+
+On this bounded corpus, MSO uses about **94.1% fewer normalized tokens per successful task (~17× smaller)**, with **29.0% lower average latency** and **27.2% lower p50 latency** in this run. This is a corpus result, not a universal product/model claim; quality ranking still prioritizes task success and scenario-policy, with latency only as its descriptive tie-breaker. Token efficiency is reported as a separate now-comparable metric and still does not override correctness/policy. Cost remains withheld because MSO has no comparable cost field and Hermes reports included cost with source `none`.
+
+The candidate run also preserves the structural P3/P4 behaviors: `multi-read` uses one `read_pipeline`, security uses one `fs_read`, and recovery's one failed primary read remains expected task evidence.
+
+## P5 next direction
1. **Broaden the corpus before broad claims** — add repository debugging, longer multi-step work, approval/retry/rollback, and realistic MCP project tasks while keeping objective state-based scoring.
-2. **Normalize provider usage semantics** — expose cache/reasoning usage consistently from MSO (where the provider supplies it) before allowing token/cost efficiency to influence ranking.
-3. **Add OpenClaw only when comparable** — run the same corpus when an equivalent provider/model is legitimately configured, without mutating credentials as part of the benchmark.
-4. **Use benchmark failures as engineering inputs** — keep the P4 pattern: locate an unnecessary route/tool/retry, fix the harness/runtime rather than prompting harder, then rerun the exact seeded scenario.
-5. **Memory calibration** — grow post-P1 telemetry and test authority/confidence/temporal policies against real corrections before introducing graph-memory complexity.
+2. **Calibrate cache behavior** — add repeated-prefix fixtures that can observe real provider cache hits without forcing them; keep zero/absent cache fields distinct and never assume a hit.
+3. **Normalize cost only with attribution** — compare cost only when every runner exposes the same usable USD source/contract; `0` with source `none` is not evidence of free execution.
+4. **Add OpenClaw only when comparable** — run the same corpus when an equivalent provider/model is legitimately configured, without mutating credentials as part of the benchmark.
+5. **Use benchmark failures as engineering inputs** — keep the P4/P5 pattern: locate an unnecessary route/tool/retry/accounting ambiguity, fix the harness/runtime rather than prompting harder, then rerun the exact seeded scenario.
+6. **Memory calibration** — grow post-P1 telemetry and test authority/confidence/temporal policies against real corrections before introducing graph-memory complexity.
## Security notes
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index 8dd27073..e4064e87 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -2,6 +2,14 @@
Running log of what shipped each phase. Newest at top.
+## 2026-09-03 — Cognitive Runtime P5 normalized provider usage
+
+- **Provider usage SSOT:** Codex/OpenAI now preserve authoritative input/output/total plus provider-reported cache-read/cache-write/reasoning details and explicit `inclusive-input-output-total` semantics; Anthropic keeps its separate cache categories and leaves absent fields absent. Agent sessions start with `apiCalls:0`, so unknown totals are never seeded as fake zero.
+- **No double counting:** cache/reasoning details remain breakdowns when the provider contract says they are included. Session aggregation records API-call/detail coverage and mixed semantics rather than summing a component into total twice. Compact status can show in/out even when a provider does not report total.
+- **Proof-gated benchmark normalization:** P5 adds `accountingProof`. MSO Codex uses `explicit-inclusive-contract`; Hermes openai-codex is normalized only when exact arithmetic proves cache is outside base input while reasoning is inside output (`exact-exclusive-cache-sum`). Ambiguous rows stay `opaque-total`, and even two opaque rows cannot enable token comparison.
+- **Comparable full-corpus result:** candidate P5 reran all six P4 scenarios on `openai-codex/gpt-5.6-terra`; both agents remained **6/6 task + 6/6 scenario-policy**. MSO measured **11,497.7 ms average / 11,782 ms p50 / 3,510.8 normalized tokens per success** versus Hermes **16,199.3 ms / 16,184.5 ms / 59,743.2 tokens**. On this bounded corpus that is ~**94.1% fewer normalized tokens (~17× smaller)**, ~**29.0% lower average latency**, and ~**27.2% lower p50**. Token semantics are now comparable; cost semantics remain withheld.
+- **Honest limits:** no cache hit was manufactured for MSO; zero cache values in the observed Codex calls came from the provider payload. Cost remains non-comparable because MSO exposes no matching cost field and Hermes reports source `none`. OpenClaw remains unranked without an equivalent configured provider/model.
+
## 2026-09-03 — Cognitive Runtime P4 comparable quality corpus
- **Six objective scratch scenarios:** added seeded per-agent read, multi-read transform, exact write-create, write-preserve, recovery, and prompt-injection/security tasks. Each agent gets the same seeded semantic values/expected outcomes in a separate 0700 scratch tree; scoring compares the exact scenario filesystem tree/result state and rejects unexpected extra files/symlinks. Fixtures are isolated, but runners retain normal authority (`runnerAuthoritySandboxed=false`) and `policyObservationScope=scenario-tree` prevents whole-host policy claims; all scratch state is deleted after the run.
diff --git a/docs/TESTING-HANDOFF.md b/docs/TESTING-HANDOFF.md
index 7dac3596..b051d571 100644
--- a/docs/TESTING-HANDOFF.md
+++ b/docs/TESTING-HANDOFF.md
@@ -7,6 +7,7 @@ Update this file at the end of every manual or automated test session. Keep it f
The current deployed baseline is **MSO CLI 1.12.0** with toolset `2026.09.03.2`; use `mso --version` plus `/api/health` for the exact Git build because documentation-only release commits also advance that identity. The v1.10/v1.11 findings and the v1.12 sectioned-terminal/recoverable-error work below are resolved and retained as historical test evidence. Do not re-open a resolved finding from prose alone: reproduce it against the current build first. There is no known 1.12 release blocker in this handoff.
Current verified behavior:
+- P5 provider-usage normalization preserves Codex/OpenAI cache/reasoning details without double counting, keeps unknown fields unknown, and proof-gates canonical token semantics. A full candidate MSO↔Hermes run on `openai-codex/gpt-5.6-terra` remained 6/6 task + 6/6 scenario-policy for both; token semantics are now comparable at 3,510.8 vs 59,743.2 normalized tokens/success, while cost semantics remain non-comparable. MSO's proof is `explicit-inclusive-contract`; Hermes is `exact-exclusive-cache-sum`.
- P4 `bench:corpus` uses six seeded, per-agent isolated 0700 scratch scenarios, exact scenario-tree scoring (`policyObservationScope=scenario-tree`, not whole-host sandbox proof), and rotating scenario-major runner order. The reproduced full provider+model-matched MSO↔Hermes run on `openai-codex/gpt-5.6-terra` is 6/6 task + 6/6 scenario-observable policy for both; MSO measured 10,759.5 ms average / 10,646.5 ms p50 versus Hermes 14,624 ms / 14,304.5 ms. This is a bounded corpus tie-break, not an overall-product claim. Token/cost accounting remains explicitly non-comparable and is not a ranking input; OpenClaw is unranked until an equivalent model/provider path exists.
- every durable session has a unique short public `name` (`milo`, `luna`, `nara`, …) independent from its longer `title`; `/rename` changes the handle and `/title` changes the description;
- human `@name` mention routing resolves active agents only; internal `agent-a` aliases remain compatibility data for explicit APIs;
diff --git a/lib/ai/codex-stream.test.ts b/lib/ai/codex-stream.test.ts
index 8071bb42..efa894dc 100644
--- a/lib/ai/codex-stream.test.ts
+++ b/lib/ai/codex-stream.test.ts
@@ -188,6 +188,19 @@ describe("reading calls back", () => {
expect(ev).toContainEqual(["tool_use", { id: "c1", name: "fs.list", input: {} }]);
});
+ it("preserves Responses cache/reasoning usage details with explicit inclusive semantics", async () => {
+ capture(sse({ type: "response.completed", response: { output: [], usage: {
+ input_tokens: 100, input_tokens_details: { cached_tokens: 60, cache_write_tokens: 5 },
+ output_tokens: 20, output_tokens_details: { reasoning_tokens: 7 }, total_tokens: 120,
+ } } }));
+ const ev = await run();
+ expect(ev.at(-1)).toEqual(["done", { stopReason: "end_turn", usage: {
+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 100, outputTokens: 20, totalTokens: 120,
+ cacheReadTokens: 60, cacheWriteTokens: 5, reasoningTokens: 7,
+ detailCoverage: { cacheReadTokens: 1, cacheWriteTokens: 1, reasoningTokens: 1 },
+ } }]);
+ });
+
it("still streams plain text with no tools declared", async () => {
capture(sse({ type: "response.output_text.delta", delta: "hello" }));
const ev = await run();
diff --git a/lib/ai/codex-stream.ts b/lib/ai/codex-stream.ts
index c4150867..1430ab49 100644
--- a/lib/ai/codex-stream.ts
+++ b/lib/ai/codex-stream.ts
@@ -1,6 +1,7 @@
import type { OAuthBundle } from "@/lib/config/store";
import { CODEX } from "./oauth/codex";
import type { OaMsg, OaTool } from "./openai-stream";
+import { responsesProviderUsage } from "./provider-usage.mjs";
// Stream a chat via the ChatGPT "Codex" backend Responses API. Unlike the OpenAI
// platform (/chat/completions), this is the consumer backend: the OAuth bearer +
@@ -159,7 +160,7 @@ export async function streamCodex(opts: {
const dec = new TextDecoder();
let buf = "";
let sawToolCall = false;
- let usage: { inputTokens?: number; outputTokens?: number; totalTokens?: number } | null = null;
+ let usage: ReturnType<typeof responsesProviderUsage> = null;
const seen = new Set<string>();
while (true) {
@@ -179,7 +180,10 @@ export async function streamCodex(opts: {
if (!data || data === "[DONE]") continue;
let ev: {
type?: string; delta?: string; item?: ResponsesItem;
- response?: { output?: ResponsesItem[]; usage?: { input_tokens?: number; output_tokens?: number; total_tokens?: number } };
+ response?: { output?: ResponsesItem[]; usage?: {
+ input_tokens?: number; input_tokens_details?: { cached_tokens?: number; cache_write_tokens?: number };
+ output_tokens?: number; output_tokens_details?: { reasoning_tokens?: number }; total_tokens?: number;
+ } };
};
try {
ev = JSON.parse(data);
@@ -196,13 +200,7 @@ export async function streamCodex(opts: {
if (ev.type === "response.output_item.done" && ev.item) {
if (emitCall(ev.item, seen, realName, emit)) sawToolCall = true;
} else if (ev.type === "response.completed") {
- if (ev.response?.usage) {
- usage = {
- inputTokens: ev.response.usage.input_tokens,
- outputTokens: ev.response.usage.output_tokens,
- totalTokens: ev.response.usage.total_tokens,
- };
- }
+ if (ev.response?.usage) usage = responsesProviderUsage(ev.response.usage);
for (const item of ev.response?.output ?? []) {
if (emitCall(item, seen, realName, emit)) sawToolCall = true;
}
diff --git a/lib/ai/openai-stream.test.ts b/lib/ai/openai-stream.test.ts
index 04d0d1ec..1f8e6b1c 100644
--- a/lib/ai/openai-stream.test.ts
+++ b/lib/ai/openai-stream.test.ts
@@ -50,6 +50,17 @@ describe("streamOpenAI", () => {
expect(done.stopReason).toBe("tool_use"); // tool_calls → tool_use so the agent loop continues
});
+ it("preserves Chat Completions cache/reasoning usage details without adding them to total", async () => {
+ const pieces = [`data: ${JSON.stringify({ usage: { prompt_tokens: 30, prompt_tokens_details: { cached_tokens: 20 }, completion_tokens: 6, completion_tokens_details: { reasoning_tokens: 4 }, total_tokens: 36 }, choices: [{ delta: {}, finish_reason: "stop" }] })}\n\ndata: [DONE]\n\n`];
+ vi.stubGlobal("fetch", vi.fn(async () => new Response(sseStream(pieces), { status: 200 })));
+ const events: { e: string; d: unknown }[] = [];
+ await streamOpenAI({ resolved, messages: [{ role: "user", text: "hi" }], system: "sys", signal: new AbortController().signal, emit: (e, d) => events.push({ e, d }) });
+ expect(events.find((x) => x.e === "done")?.d).toEqual({ stopReason: "end_turn", usage: {
+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 30, outputTokens: 6, totalTokens: 36, cacheReadTokens: 20, reasoningTokens: 4,
+ detailCoverage: { cacheReadTokens: 1, reasoningTokens: 1 },
+ } });
+ });
+
it("throws provider+status on a non-ok response", async () => {
vi.stubGlobal("fetch", vi.fn(async () => new Response("nope", { status: 401 })));
await expect(
diff --git a/lib/ai/openai-stream.ts b/lib/ai/openai-stream.ts
index 1a4dcf6d..29b6f8ef 100644
--- a/lib/ai/openai-stream.ts
+++ b/lib/ai/openai-stream.ts
@@ -1,3 +1,4 @@
+import { chatCompletionsProviderUsage } from "./provider-usage.mjs";
// OpenAI-protocol streaming adapter for /api/assistant. Every provider in the
// @rahmanef/models registry except Anthropic speaks the OpenAI Chat Completions
// wire (POST {baseUrl}/chat/completions, `data: {choices:[{delta}]}` SSE lines
@@ -23,7 +24,10 @@ type OaOutMsg = {
};
type OaChunk = {
- usage?: { prompt_tokens?: number; completion_tokens?: number; total_tokens?: number };
+ usage?: {
+ prompt_tokens?: number; prompt_tokens_details?: { cached_tokens?: number; cache_write_tokens?: number };
+ completion_tokens?: number; completion_tokens_details?: { reasoning_tokens?: number }; total_tokens?: number;
+ };
choices?: Array<{
delta?: {
content?: string | null;
@@ -110,7 +114,7 @@ export async function streamOpenAI(opts: {
// tool_call fragments arrive split across chunks; accumulate by index.
const calls = new Map<number, { id: string; name: string; args: string }>();
let finish: string | null = null;
- let usage: { inputTokens?: number; outputTokens?: number; totalTokens?: number } | null = null;
+ let usage: ReturnType<typeof chatCompletionsProviderUsage> = null;
reading: while (true) {
if (signal.aborted) { await reader.cancel().catch(() => {}); return; }
@@ -127,13 +131,7 @@ export async function streamOpenAI(opts: {
if (data === "[DONE]") break reading;
let chunk: OaChunk;
try { chunk = JSON.parse(data) as OaChunk; } catch { continue; }
- if (chunk.usage) {
- usage = {
- inputTokens: chunk.usage.prompt_tokens,
- outputTokens: chunk.usage.completion_tokens,
- totalTokens: chunk.usage.total_tokens,
- };
- }
+ if (chunk.usage) usage = chatCompletionsProviderUsage(chunk.usage);
const choice = chunk.choices?.[0];
if (!choice) continue;
if (choice.delta?.content) emit("delta", choice.delta.content);
diff --git a/lib/ai/provider-usage.mjs b/lib/ai/provider-usage.mjs
new file mode 100644
index 00000000..c0ed783f
--- /dev/null
+++ b/lib/ai/provider-usage.mjs
@@ -0,0 +1,105 @@
+function finite(value) {
+ if (value === null || value === undefined || value === "") return undefined;
+ const n = Number(value);
+ return Number.isFinite(n) && n >= 0 ? n : undefined;
+}
+
+function put(out, key, value) {
+ const n = finite(value);
+ if (n !== undefined) out[key] = n;
+}
+
+function coverage(keys) {
+ const out = {};
+ for (const [name, value] of keys) if (finite(value) !== undefined) out[name] = 1;
+ return Object.keys(out).length ? out : undefined;
+}
+
+/** OpenAI Responses usage: cache/reasoning details are subsets of input/output. */
+export function responsesProviderUsage(raw) {
+ if (!raw || typeof raw !== "object") return null;
+ const out = { accountingMode: "inclusive-input-output-total", apiCalls: 1 };
+ put(out, "inputTokens", raw.input_tokens);
+ put(out, "outputTokens", raw.output_tokens);
+ put(out, "totalTokens", raw.total_tokens);
+ put(out, "cacheReadTokens", raw.input_tokens_details?.cached_tokens);
+ put(out, "cacheWriteTokens", raw.input_tokens_details?.cache_write_tokens);
+ put(out, "reasoningTokens", raw.output_tokens_details?.reasoning_tokens);
+ const detailCoverage = coverage([
+ ["cacheReadTokens", raw.input_tokens_details?.cached_tokens],
+ ["cacheWriteTokens", raw.input_tokens_details?.cache_write_tokens],
+ ["reasoningTokens", raw.output_tokens_details?.reasoning_tokens],
+ ]);
+ if (detailCoverage) out.detailCoverage = detailCoverage;
+ return out;
+}
+
+/** OpenAI-compatible Chat Completions usage uses the same inclusive category semantics. */
+export function chatCompletionsProviderUsage(raw) {
+ if (!raw || typeof raw !== "object") return null;
+ const out = { accountingMode: "inclusive-input-output-total", apiCalls: 1 };
+ put(out, "inputTokens", raw.prompt_tokens);
+ put(out, "outputTokens", raw.completion_tokens);
+ put(out, "totalTokens", raw.total_tokens);
+ put(out, "cacheReadTokens", raw.prompt_tokens_details?.cached_tokens);
+ put(out, "cacheWriteTokens", raw.prompt_tokens_details?.cache_write_tokens);
+ put(out, "reasoningTokens", raw.completion_tokens_details?.reasoning_tokens);
+ const detailCoverage = coverage([
+ ["cacheReadTokens", raw.prompt_tokens_details?.cached_tokens],
+ ["cacheWriteTokens", raw.prompt_tokens_details?.cache_write_tokens],
+ ["reasoningTokens", raw.completion_tokens_details?.reasoning_tokens],
+ ]);
+ if (detailCoverage) out.detailCoverage = detailCoverage;
+ return out;
+}
+
+/** Anthropic reports cache reads/creation as separate input categories. */
+export function anthropicProviderUsage(raw) {
+ if (!raw || typeof raw !== "object") return null;
+ const out = { accountingMode: "separate-cache-input-output", apiCalls: 1 };
+ put(out, "inputTokens", raw.input_tokens);
+ put(out, "outputTokens", raw.output_tokens);
+ put(out, "cacheReadTokens", raw.cache_read_input_tokens);
+ put(out, "cacheWriteTokens", raw.cache_creation_input_tokens);
+ const detailCoverage = coverage([
+ ["cacheReadTokens", raw.cache_read_input_tokens],
+ ["cacheWriteTokens", raw.cache_creation_input_tokens],
+ ]);
+ if (detailCoverage) out.detailCoverage = detailCoverage;
+ return out;
+}
+
+const SUM_FIELDS = ["inputTokens", "outputTokens", "totalTokens", "cacheReadTokens", "cacheWriteTokens", "reasoningTokens"];
+const ALIASES = {
+ inputTokens: ["inputTokens", "input_tokens", "prompt_tokens"],
+ outputTokens: ["outputTokens", "output_tokens", "completion_tokens"],
+ totalTokens: ["totalTokens", "total_tokens"],
+ cacheReadTokens: ["cacheReadTokens", "cache_read_tokens", "cacheReadInputTokens", "cache_read_input_tokens"],
+ cacheWriteTokens: ["cacheWriteTokens", "cache_write_tokens", "cacheCreationInputTokens", "cache_creation_input_tokens"],
+ reasoningTokens: ["reasoningTokens", "reasoning_tokens"],
+};
+function metric(row, key) {
+ for (const alias of ALIASES[key]) { const value = finite(row?.[alias]); if (value !== undefined) return value; }
+ return undefined;
+}
+
+/** Accumulate provider-reported calls without inventing missing component values. */
+export function addProviderUsage(total, next) {
+ if (!next || typeof next !== "object") return total ?? { apiCalls: 0 };
+ const base = total && typeof total === "object" ? total : { apiCalls: 0 };
+ const out = { apiCalls: (finite(base.apiCalls) ?? 0) + (finite(next.apiCalls) ?? 1) };
+ for (const key of SUM_FIELDS) {
+ const a = metric(base, key), b = metric(next, key);
+ if (a !== undefined || b !== undefined) out[key] = (a ?? 0) + (b ?? 0);
+ }
+ const modes = [base.accountingMode, next.accountingMode].filter(Boolean);
+ if (modes.length) out.accountingMode = new Set(modes).size === 1 ? modes[0] : "mixed";
+ const cov = {};
+ for (const key of ["cacheReadTokens", "cacheWriteTokens", "reasoningTokens"]) {
+ const a = finite(base.detailCoverage?.[key]) ?? 0;
+ const b = finite(next.detailCoverage?.[key]) ?? (metric(next, key) !== undefined ? finite(next.apiCalls) ?? 1 : 0);
+ if (a || b) cov[key] = a + b;
+ }
+ if (Object.keys(cov).length) out.detailCoverage = cov;
+ return out;
+}
diff --git a/lib/ai/provider-usage.test.ts b/lib/ai/provider-usage.test.ts
new file mode 100644
index 00000000..ceca5ac3
--- /dev/null
+++ b/lib/ai/provider-usage.test.ts
@@ -0,0 +1,40 @@
+import { describe, expect, it } from "vitest";
+import { addProviderUsage, anthropicProviderUsage, chatCompletionsProviderUsage, responsesProviderUsage } from "./provider-usage.mjs";
+
+describe("provider usage normalization", () => {
+ it("keeps Responses cache/reasoning as inclusive details without double counting", () => {
+ expect(responsesProviderUsage({ input_tokens: 100, input_tokens_details: { cached_tokens: 60, cache_write_tokens: 5 }, output_tokens: 20, output_tokens_details: { reasoning_tokens: 7 }, total_tokens: 120 })).toEqual({
+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 100, outputTokens: 20, totalTokens: 120,
+ cacheReadTokens: 60, cacheWriteTokens: 5, reasoningTokens: 7,
+ detailCoverage: { cacheReadTokens: 1, cacheWriteTokens: 1, reasoningTokens: 1 },
+ });
+ });
+
+ it("keeps absent provider details absent rather than fabricating zero", () => {
+ expect(responsesProviderUsage({ input_tokens: 10, output_tokens: 2, total_tokens: 12 })).toEqual({
+ accountingMode: "inclusive-input-output-total", apiCalls: 1, inputTokens: 10, outputTokens: 2, totalTokens: 12,
+ });
+ expect(anthropicProviderUsage({ input_tokens: 10, output_tokens: 2 })).toEqual({
+ accountingMode: "separate-cache-input-output", apiCalls: 1, inputTokens: 10, outputTokens: 2,
+ });
+ });
+
+ it("does not seed an unknown provider total with zero during accumulation", () => {
+ const out = addProviderUsage({ apiCalls: 0 }, anthropicProviderUsage({ input_tokens: 10, output_tokens: 2 }));
+ expect(out).toEqual({ apiCalls: 1, inputTokens: 10, outputTokens: 2, accountingMode: "separate-cache-input-output" });
+ expect(out.totalTokens).toBeUndefined();
+ });
+
+ it("normalizes Chat Completions detail fields with the same inclusive semantics", () => {
+ expect(chatCompletionsProviderUsage({ prompt_tokens: 30, prompt_tokens_details: { cached_tokens: 20 }, completion_tokens: 6, completion_tokens_details: { reasoning_tokens: 4 }, total_tokens: 36 })).toMatchObject({
+ accountingMode: "inclusive-input-output-total", inputTokens: 30, outputTokens: 6, totalTokens: 36, cacheReadTokens: 20, reasoningTokens: 4,
+ });
+ });
+
+ it("accumulates calls while preserving detail coverage and mixed semantics", () => {
+ const a = responsesProviderUsage({ input_tokens: 10, input_tokens_details: { cached_tokens: 3 }, output_tokens: 2, total_tokens: 12 });
+ const b = responsesProviderUsage({ input_tokens: 20, input_tokens_details: { cached_tokens: 4 }, output_tokens: 3, total_tokens: 23 });
+ expect(addProviderUsage(addProviderUsage(null, a), b)).toMatchObject({ apiCalls: 2, inputTokens: 30, outputTokens: 5, totalTokens: 35, cacheReadTokens: 7, accountingMode: "inclusive-input-output-total", detailCoverage: { cacheReadTokens: 2 } });
+ expect(addProviderUsage(a, anthropicProviderUsage({ input_tokens: 1, output_tokens: 1 }))).toMatchObject({ accountingMode: "mixed" });
+ });
+});
diff --git a/lib/ai/usage.ts b/lib/ai/usage.ts
index d3baa67e..c7b85678 100644
--- a/lib/ai/usage.ts
+++ b/lib/ai/usage.ts
@@ -1,13 +1,9 @@
+import { anthropicProviderUsage } from "./provider-usage.mjs";
import type Anthropic from "@anthropic-ai/sdk";
export function anthropicDonePayload(final: Anthropic.Message) {
return {
stopReason: final.stop_reason,
- usage: {
- inputTokens: final.usage.input_tokens,
- outputTokens: final.usage.output_tokens,
- cacheCreationInputTokens: final.usage.cache_creation_input_tokens ?? 0,
- cacheReadInputTokens: final.usage.cache_read_input_tokens ?? 0,
- },
+ usage: anthropicProviderUsage(final.usage),
};
}
diff --git a/scripts/bench-agent-corpus.mjs b/scripts/bench-agent-corpus.mjs
index 151c54e5..eced25b6 100644
--- a/scripts/bench-agent-corpus.mjs
+++ b/scripts/bench-agent-corpus.mjs
@@ -69,7 +69,9 @@ export function summarizeCorpus(rows, requested) {
const expectedScenarios = new Set(rows.map((row) => row.scenarioId)).size;
const tokenModes = new Set(aggregates.map((row) => row.tokenAccountingMode));
const tokenSemanticsComparable = aggregates.length >= 2
- && aggregates.every((row) => row.tokenCoveragePct === 100 && !["unknown", "mixed"].includes(row.tokenAccountingMode))
+ && aggregates.every((row) => row.tokenCoveragePct === 100
+ && row.tokenAccountingMode === "inclusive-input-output-total"
+ && !["unknown", "mixed"].includes(row.tokenAccountingProof))
&& tokenModes.size === 1;
const costPairs = new Set(aggregates.map((row) => `${row.costStatus}:${row.costSource}`));
const costSemanticsComparable = aggregates.length >= 2
@@ -77,7 +79,10 @@ export function summarizeCorpus(rows, requested) {
&& costPairs.size === 1;
return {
aggregates, comparability,
- efficiencyComparability: { tokenSemanticsComparable, costSemanticsComparable, tokenAccountingModes: [...tokenModes] },
+ efficiencyComparability: {
+ tokenSemanticsComparable, costSemanticsComparable, tokenAccountingModes: [...tokenModes],
+ tokenAccountingProofs: Object.fromEntries(aggregates.map((row) => [row.agent, row.tokenAccountingProof])),
+ },
ranking: eligibleRanking(aggregates, comparability, expectedScenarios),
};
}
diff --git a/scripts/bench-agent-corpus.test.ts b/scripts/bench-agent-corpus.test.ts
index c478299e..d2e80a1f 100644
--- a/scripts/bench-agent-corpus.test.ts
+++ b/scripts/bench-agent-corpus.test.ts
@@ -31,8 +31,9 @@ describe("P4 agent-quality corpus", () => {
it("keeps missing usage and cost unknown rather than converting null to zero", () => {
expect(extractUsage({ usage: { input_tokens: null, output_tokens: null, cost_usd: null } })).toBeNull();
- expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })).toEqual({ inputTokens: 10, outputTokens: 5, totalTokens: 15, unattributedTokens: 0, accountingMode: "input-output-total" });
- expect(extractUsage({ input_tokens: 10, output_tokens: 5, reasoning_tokens: 7, total_tokens: 22, cache_read_tokens: 3, cost_status: "estimated", cost_source: "catalog" })).toMatchObject({ reasoningTokens: 7, cacheReadTokens: 3, unattributedTokens: 7, accountingMode: "expanded-components", costStatus: "estimated", costSource: "catalog" });
+ expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })).toMatchObject({ inputTokens: 10, outputTokens: 5, reportedAccountingMode: "unspecified", accountingMode: "opaque-total", accountingProof: "none" });
+ expect(extractUsage({ usage: { input_tokens: 10, output_tokens: 5 } })?.totalTokens).toBeUndefined();
+ expect(extractUsage({ input_tokens: 10, output_tokens: 5, reasoning_tokens: 7, total_tokens: 22, cache_read_tokens: 3, cost_status: "estimated", cost_source: "catalog" })).toMatchObject({ reasoningTokens: 7, cacheReadTokens: 3, accountingMode: "opaque-total", costStatus: "estimated", costSource: "catalog" });
});
it("extracts model/provider evidence but withholds full comparability when provider differs", () => {
@@ -57,7 +58,7 @@ describe("P4 agent-quality corpus", () => {
it("aggregates token/cost only from rows that actually report them", () => {
const rows = [
- { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, estimatedCostUsd: 0.01 }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
+ { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, normalizedTotalTokens: 100, accountingMode: "inclusive-input-output-total", estimatedCostUsd: 0.01 }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
{ fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 20, usage: null, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
];
const out = aggregateAgent("mso", rows);
@@ -97,8 +98,8 @@ describe("P4 agent-quality corpus", () => {
it("charges failed-attempt tokens into tokens-per-success when coverage is complete", () => {
const rows = [
- { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, accountingMode: "input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
- { fullSuccess: false, taskSuccess: false, policyCompliant: true, latencyMs: 20, usage: { totalTokens: 300, accountingMode: "input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
+ { fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 100, normalizedTotalTokens: 100, accountingMode: "inclusive-input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
+ { fullSuccess: false, taskSuccess: false, policyCompliant: true, latencyMs: 20, usage: { totalTokens: 300, normalizedTotalTokens: 300, accountingMode: "inclusive-input-output-total" }, modelEvidence: { model: "p/x", modelFamily: "x", provider: "p" } },
];
const out = aggregateAgent("mso", rows);
expect(out.tokenCoveragePct).toBe(100); expect(out.totalReportedTokens).toBe(400); expect(out.tokensPerSuccessfulTask).toBe(400);
@@ -147,14 +148,41 @@ describe("P4 agent-quality corpus", () => {
expect(out.status).toBe(0); expect(out.stdout).toContain("1 scenarios × 1 agents");
});
- it("keeps model/provider comparability separate from incompatible token accounting semantics", () => {
+ it("derives a canonical total for explicit separate-component semantics without inventing a raw provider total", () => {
+ const usage = extractUsage({ inputTokens: 10, outputTokens: 2, cacheReadTokens: 3, accountingMode: "separate-cache-input-output" })!;
+ expect(usage.totalTokens).toBeUndefined();
+ expect(usage).toMatchObject({ normalizedInputTokens: 13, normalizedOutputTokens: 2, normalizedTotalTokens: 15, accountingMode: "inclusive-input-output-total", accountingProof: "explicit-separate-components" });
+ });
+
+ it("normalizes exact Hermes-style exclusive components to the same canonical token semantics as MSO", () => {
+ const mso = extractUsage({ inputTokens: 100, outputTokens: 20, totalTokens: 120, cacheReadTokens: 60, reasoningTokens: 7, accountingMode: "inclusive-input-output-total" })!;
+ const hermes = extractUsage({ input_tokens: 40, cache_read_tokens: 60, cache_write_tokens: 0, output_tokens: 20, reasoning_tokens: 7, total_tokens: 120 })!;
+ expect(mso).toMatchObject({ accountingMode: "inclusive-input-output-total", accountingProof: "explicit-inclusive-contract", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });
+ expect(hermes).toMatchObject({ reportedAccountingMode: "exclusive-cache-inclusive-output", accountingMode: "inclusive-input-output-total", accountingProof: "exact-exclusive-cache-sum", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });
+ const rows = [
+ { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: mso, modelEvidence: { modelFamily: "x", provider: "p" } },
+ { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: hermes, modelEvidence: { modelFamily: "x", provider: "p" } },
+ ];
+ expect(summarizeCorpus(rows as any, { model: "x", provider: "p" }).efficiencyComparability.tokenSemanticsComparable).toBe(true);
+ });
+
+ it("can also normalize an exact representation where reasoning is outside output", () => {
+ const usage = extractUsage({ input_tokens: 40, cache_read_tokens: 60, output_tokens: 13, reasoning_tokens: 7, total_tokens: 120 })!;
+ expect(usage).toMatchObject({ reportedAccountingMode: "exclusive-cache-reasoning", accountingProof: "exact-exclusive-cache-reasoning-sum", normalizedInputTokens: 100, normalizedOutputTokens: 20, normalizedTotalTokens: 120 });
+ });
+
+ it("does not treat two equally opaque totals as comparable semantics", () => {
+ const opaque = extractUsage({ input_tokens: 10, output_tokens: 5, cache_read_tokens: 3, reasoning_tokens: 7, total_tokens: 22 })!;
const rows = [
- { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: { totalTokens: 10, accountingMode: "input-output-total" }, modelEvidence: { modelFamily: "x", provider: "p" } },
- { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: { totalTokens: 12, accountingMode: "expanded-components" }, modelEvidence: { modelFamily: "x", provider: "p" } },
+ { agent: "a", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 10, usage: opaque, modelEvidence: { modelFamily: "x", provider: "p" } },
+ { agent: "b", scenarioId: "x", fullSuccess: true, taskSuccess: true, policyCompliant: true, latencyMs: 12, usage: opaque, modelEvidence: { modelFamily: "x", provider: "p" } },
];
- const out = summarizeCorpus(rows as any, { model: "x", provider: "p" });
- expect(out.comparability.level).toBe("full");
- expect(out.efficiencyComparability.tokenSemanticsComparable).toBe(false);
+ expect(summarizeCorpus(rows as any, { model: "x", provider: "p" }).efficiencyComparability.tokenSemanticsComparable).toBe(false);
+ });
+
+ it("withholds token comparability when expanded-component arithmetic does not prove the representation", () => {
+ const ambiguous = extractUsage({ input_tokens: 10, output_tokens: 5, cache_read_tokens: 3, reasoning_tokens: 7, total_tokens: 22 })!;
+ expect(ambiguous.accountingMode).toBe("opaque-total");
});
});
diff --git a/scripts/bench-agent-metrics.mjs b/scripts/bench-agent-metrics.mjs
index 5bacc2db..17f4a77a 100644
--- a/scripts/bench-agent-metrics.mjs
+++ b/scripts/bench-agent-metrics.mjs
@@ -46,10 +46,11 @@ export function extractUsage(value) {
const input = finite(row.inputTokens ?? row.input_tokens ?? row.prompt_tokens ?? row.promptTokens);
const output = finite(row.outputTokens ?? row.output_tokens ?? row.completion_tokens ?? row.completionTokens);
const reasoning = finite(row.reasoningTokens ?? row.reasoning_tokens);
- const cacheRead = finite(row.cacheReadTokens ?? row.cache_read_tokens);
- const cacheWrite = finite(row.cacheWriteTokens ?? row.cache_write_tokens);
+ const cacheRead = finite(row.cacheReadTokens ?? row.cache_read_tokens ?? row.cacheReadInputTokens ?? row.cache_read_input_tokens);
+ const cacheWrite = finite(row.cacheWriteTokens ?? row.cache_write_tokens ?? row.cacheCreationInputTokens ?? row.cache_creation_input_tokens);
const apiCalls = finite(row.apiCalls ?? row.api_calls);
- const total = finite(row.totalTokens ?? row.total_tokens) ?? (input !== undefined && output !== undefined ? input + output : undefined);
+ const total = finite(row.totalTokens ?? row.total_tokens);
+ const explicitMode = typeof (row.accountingMode ?? row.accounting_mode) === "string" ? String(row.accountingMode ?? row.accounting_mode) : null;
const explicitUsd = finite(row.estimatedCostUsd ?? row.estimated_cost_usd ?? row.costUsd ?? row.cost_usd);
const currency = typeof (row.currency ?? row.costCurrency ?? row.cost_currency) === "string"
? String(row.currency ?? row.costCurrency ?? row.cost_currency).toLowerCase() : undefined;
@@ -58,15 +59,48 @@ export function extractUsage(value) {
const costStatus = typeof (row.costStatus ?? row.cost_status) === "string" ? String(row.costStatus ?? row.cost_status) : undefined;
const costSource = typeof (row.costSource ?? row.cost_source) === "string" ? String(row.costSource ?? row.cost_source) : undefined;
if (input === undefined && output === undefined && reasoning === undefined && total === undefined && cost === undefined) return null;
- const unattributed = total !== undefined && input !== undefined && output !== undefined ? Math.max(0, total - input - output) : undefined;
- const accountingMode = reasoning !== undefined || cacheRead !== undefined || cacheWrite !== undefined ? "expanded-components"
- : total !== undefined && input !== undefined && output !== undefined && total === input + output ? "input-output-total" : "opaque-total";
+
+ let normalizedInput, normalizedOutput, normalizedTotal, accountingMode = "opaque-total", reportedAccountingMode = explicitMode || "unspecified", accountingProof = "none";
+ if (input !== undefined && output !== undefined && (total !== undefined || explicitMode === "separate-cache-input-output")) {
+ const detailSum = (cacheRead ?? 0) + (cacheWrite ?? 0) + (reasoning ?? 0);
+ const exclusiveSum = input + output + detailSum;
+ if (explicitMode === "inclusive-input-output-total") {
+ normalizedInput = input; normalizedOutput = output; normalizedTotal = total; accountingMode = "inclusive-input-output-total"; accountingProof = "explicit-inclusive-contract";
+ } else if (explicitMode === "separate-cache-input-output") {
+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);
+ normalizedOutput = output + (reasoning ?? 0); normalizedTotal = total ?? (normalizedInput + normalizedOutput);
+ if (normalizedTotal === normalizedInput + normalizedOutput) { accountingMode = "inclusive-input-output-total"; accountingProof = "explicit-separate-components"; }
+ } else if (((cacheRead ?? 0) + (cacheWrite ?? 0)) > 0 && total === input + output + (cacheRead ?? 0) + (cacheWrite ?? 0)) {
+ // Hermes openai-codex shape observed in the live corpus: cache is outside base input,
+ // while reasoning remains a detail already included in output. Exact arithmetic is required.
+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);
+ normalizedOutput = output; normalizedTotal = total;
+ accountingMode = "inclusive-input-output-total"; reportedAccountingMode = "exclusive-cache-inclusive-output"; accountingProof = "exact-exclusive-cache-sum";
+ } else if (detailSum > 0 && total === exclusiveSum) {
+ // Alternate expanded representation where both cache and reasoning live outside base categories.
+ normalizedInput = input + (cacheRead ?? 0) + (cacheWrite ?? 0);
+ normalizedOutput = output + (reasoning ?? 0); normalizedTotal = total;
+ accountingMode = "inclusive-input-output-total"; reportedAccountingMode = "exclusive-cache-reasoning"; accountingProof = "exact-exclusive-cache-reasoning-sum";
+ } else if (total === input + output) {
+ // No extra tokens live outside the provider's input/output categories.
+ normalizedInput = input; normalizedOutput = output; normalizedTotal = total;
+ accountingMode = "inclusive-input-output-total"; accountingProof = detailSum > 0 ? "exact-inclusive-total-identity" : "exact-input-output-identity";
+ if (!explicitMode) reportedAccountingMode = detailSum > 0 ? "inclusive-expanded-components" : "input-output-total";
+ }
+ }
+ const unattributed = normalizedTotal !== undefined && normalizedInput !== undefined && normalizedOutput !== undefined
+ ? Math.max(0, normalizedTotal - normalizedInput - normalizedOutput) : undefined;
return {
...(input !== undefined ? { inputTokens: input } : {}), ...(output !== undefined ? { outputTokens: output } : {}),
...(reasoning !== undefined ? { reasoningTokens: reasoning } : {}), ...(cacheRead !== undefined ? { cacheReadTokens: cacheRead } : {}),
...(cacheWrite !== undefined ? { cacheWriteTokens: cacheWrite } : {}), ...(apiCalls !== undefined ? { apiCalls } : {}),
- ...(total !== undefined ? { totalTokens: total } : {}), ...(unattributed !== undefined ? { unattributedTokens: unattributed } : {}),
- ...(cost !== undefined ? { estimatedCostUsd: cost } : {}), ...(costStatus ? { costStatus } : {}), ...(costSource ? { costSource } : {}), accountingMode,
+ ...(total !== undefined ? { totalTokens: total } : {}),
+ ...(normalizedInput !== undefined ? { normalizedInputTokens: normalizedInput } : {}),
+ ...(normalizedOutput !== undefined ? { normalizedOutputTokens: normalizedOutput } : {}),
+ ...(normalizedTotal !== undefined ? { normalizedTotalTokens: normalizedTotal } : {}),
+ ...(unattributed !== undefined ? { unattributedTokens: unattributed } : {}),
+ ...(cost !== undefined ? { estimatedCostUsd: cost } : {}), ...(costStatus ? { costStatus } : {}), ...(costSource ? { costSource } : {}),
+ reportedAccountingMode, accountingMode, accountingProof,
};
});
}
@@ -98,10 +132,11 @@ function median(values) { if (!values.length) return null; const s = [...values]
export function aggregateAgent(agent, scenarios) {
const full = scenarios.filter((row) => row.fullSuccess), task = scenarios.filter((row) => row.taskSuccess), policy = scenarios.filter((row) => row.policyCompliant);
- const usageRows = scenarios.filter((row) => row.usage?.totalTokens !== undefined);
+ const usageRows = scenarios.filter((row) => row.usage?.normalizedTotalTokens !== undefined);
const costRows = scenarios.filter((row) => row.usage?.estimatedCostUsd !== undefined);
const toolRows = scenarios.filter((row) => row.toolTelemetry?.count !== undefined);
const accountingModes = new Set(scenarios.map((row) => row.usage?.accountingMode).filter(Boolean));
+ const accountingProofs = new Set(scenarios.map((row) => row.usage?.accountingProof).filter((value) => value && value !== "none"));
const costStatuses = new Set(scenarios.map((row) => row.usage?.costStatus).filter(Boolean));
const costSources = new Set(scenarios.map((row) => row.usage?.costSource).filter(Boolean));
const evidenced = scenarios.filter((row) => row.modelEvidence?.modelFamily);
@@ -114,7 +149,7 @@ export function aggregateAgent(agent, scenarios) {
consistent: families.size === 1 && providers.size <= 1,
} : null;
const tokenCoveragePct = pct(usageRows.length, scenarios.length), costCoveragePct = pct(costRows.length, scenarios.length);
- const totalReportedTokens = usageRows.reduce((sum, row) => sum + row.usage.totalTokens, 0);
+ const totalReportedTokens = usageRows.reduce((sum, row) => sum + row.usage.normalizedTotalTokens, 0);
const totalReportedCostUsd = costRows.reduce((sum, row) => sum + row.usage.estimatedCostUsd, 0);
return {
agent, attempted: scenarios.length, taskSuccesses: task.length, policyCompliant: policy.length, fullSuccesses: full.length,
@@ -123,12 +158,13 @@ export function aggregateAgent(agent, scenarios) {
tokenCoveragePct, costCoveragePct,
totalReportedTokens: usageRows.length ? totalReportedTokens : null,
totalReportedCostUsd: costRows.length ? Math.round(totalReportedCostUsd * 1e9) / 1e9 : null,
- reportedTokensPerAttempt: usageRows.length ? average(usageRows.map((row) => row.usage.totalTokens)) : null,
+ reportedTokensPerAttempt: usageRows.length ? average(usageRows.map((row) => row.usage.normalizedTotalTokens)) : null,
reportedCostPerAttemptUsd: costRows.length ? average(costRows.map((row) => row.usage.estimatedCostUsd), 6) : null,
toolTelemetryCoveragePct: pct(toolRows.length, scenarios.length),
averageToolCallsPerTask: toolRows.length ? average(toolRows.map((row) => row.toolTelemetry.count)) : null,
failedToolCalls: toolRows.reduce((sum, row) => sum + (row.toolTelemetry.failed || 0), 0),
tokenAccountingMode: accountingModes.size === 1 ? [...accountingModes][0] : accountingModes.size ? "mixed" : "unknown",
+ tokenAccountingProof: accountingProofs.size === 1 ? [...accountingProofs][0] : accountingProofs.size ? "mixed" : "unknown",
costStatus: costStatuses.size === 1 ? [...costStatuses][0] : costStatuses.size ? "mixed" : "unknown",
costSource: costSources.size === 1 ? [...costSources][0] : costSources.size ? "mixed" : "unknown",
modelEvidenceCoveragePct: pct(evidenced.length, scenarios.length),
@@ -144,5 +180,5 @@ export function eligibleRanking(aggregates, comparability, expectedScenarios = n
if (aggregates.length < 2 || aggregates.some((row) => row.attempted < 1)) return { eligible: false, reason: "every compared agent must produce corpus results" };
if (expectedScenarios != null && aggregates.some((row) => row.attempted !== expectedScenarios)) return { eligible: false, reason: "every compared agent must cover the same complete corpus" };
const ordered = [...aggregates].sort((a, b) => b.fullSuccessPct - a.fullSuccessPct || b.policyCompliancePct - a.policyCompliancePct || (a.p50LatencyMs ?? Infinity) - (b.p50LatencyMs ?? Infinity) || (a.averageLatencyMs ?? Infinity) - (b.averageLatencyMs ?? Infinity));
- return { eligible: true, order: ordered.map((row) => row.agent), note: "Order prioritizes task success, then policy compliance, then p50/average latency as a descriptive tie-breaker. Token/cost efficiency is reported separately until usage semantics are explicitly proven comparable." };
+ return { eligible: true, order: ordered.map((row) => row.agent), note: "Order prioritizes task success, then policy compliance, then p50/average latency as a descriptive tie-breaker. Efficiency stays a separate metric; token/cost values are comparable only when their independent accounting gates pass." };
}
diff --git a/scripts/mso-agent-session-ui-state.test.ts b/scripts/mso-agent-session-ui-state.test.ts
index 696615de..f1a62614 100644
--- a/scripts/mso-agent-session-ui-state.test.ts
+++ b/scripts/mso-agent-session-ui-state.test.ts
@@ -24,7 +24,7 @@ describe("MSO Agent /new session transition", () => {
expect(session.activeSkill).toBeNull();
expect(session.lastInvokedSkill).toBeNull();
expect(session.titleOverride).toBeNull();
- expect(session.usage).toEqual({ inputTokens: 0, outputTokens: 0, totalTokens: 0 });
+ expect(session.usage).toEqual({ apiCalls: 0 });
expect(session.lastElapsedMs).toBe(0);
});
diff --git a/scripts/mso-agent-session-ui.mjs b/scripts/mso-agent-session-ui.mjs
index ea6b5d5c..bbdd62f8 100644
--- a/scripts/mso-agent-session-ui.mjs
+++ b/scripts/mso-agent-session-ui.mjs
@@ -44,7 +44,7 @@ export function applyNewSessionState(session, loaded, requestedTitle = "") {
activeSkill: null,
lastInvokedSkill: null,
titleOverride: requestedTitle ? (loaded.title || requestedTitle) : null,
- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },
+ usage: { apiCalls: 0 },
lastElapsedMs: 0,
pendingApproval: null,
});
@@ -72,7 +72,7 @@ export async function resumeInto(rl, session, query) {
activeSkill: null,
lastInvokedSkill: null,
titleOverride: loaded.title || null,
- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },
+ usage: { apiCalls: 0 },
lastElapsedMs: 0,
pendingApproval: null,
});
diff --git a/scripts/mso-agent-status.mjs b/scripts/mso-agent-status.mjs
index d9a1243b..b1943023 100644
--- a/scripts/mso-agent-status.mjs
+++ b/scripts/mso-agent-status.mjs
@@ -1,6 +1,7 @@
import os from "node:os";
import path from "node:path";
import { projectHistoryForModel } from "./mso-agent-context.mjs";
+import { addProviderUsage } from "../lib/ai/provider-usage.mjs";
function compactNumber(value) {
const n = Number(value || 0);
@@ -34,18 +35,7 @@ export function contextStatus(history, modelMeta) {
};
}
-export function addUsage(total, next) {
- const base = total || { inputTokens: 0, outputTokens: 0, totalTokens: 0 };
- if (!next || typeof next !== "object") return base;
- const input = Number(next.inputTokens ?? next.input_tokens ?? 0) || 0;
- const output = Number(next.outputTokens ?? next.output_tokens ?? 0) || 0;
- const explicit = Number(next.totalTokens ?? next.total_tokens ?? 0) || 0;
- return {
- inputTokens: base.inputTokens + input,
- outputTokens: base.outputTokens + output,
- totalTokens: base.totalTokens + (explicit || input + output),
- };
-}
+export function addUsage(total, next) { return addProviderUsage(total, next); }
function compactLabel(value, max = 28) {
@@ -84,7 +74,8 @@ export function statusParts(session, cwd = process.cwd()) {
skill ? `skill ${skill.state === "invoked" ? "✓" : "◆"} /${skill.name}${skill.state === "queued" ? " queued" : skill.state === "invoking" ? " invoking" : ""}` : null,
ctx.limit ? `ctx ~${compactNumber(ctx.used)}/${compactNumber(ctx.limit)} ${ctx.percent}%` : `ctx ~${compactNumber(ctx.used)}/?`,
session?.agentSession?.compactThresholdTokens ? `session ~${compactNumber(session.agentSession.estimatedTokens)}/${compactNumber(session.agentSession.compactThresholdTokens)}` : null,
- usage.totalTokens > 0 ? `tokens ${compactNumber(usage.totalTokens)}` : null,
+ (usage.totalTokens ?? 0) > 0 ? `tokens ${compactNumber(usage.totalTokens)}`
+ : ((usage.inputTokens ?? 0) > 0 || (usage.outputTokens ?? 0) > 0) ? `tokens ${compactNumber(usage.inputTokens)}in/${compactNumber(usage.outputTokens)}out` : null,
routing?.activeTools != null ? `route ${routing.routeIds?.join("+") || (routing.fallbackUsed ? "fallback" : "direct")} · tools ${routing.activeTools}/${routing.fullTools}` : null,
`turns ${turns}`,
elapsed > 0 ? `${(elapsed / 1000).toFixed(elapsed >= 10_000 ? 0 : 1)}s` : null,
@@ -98,7 +89,7 @@ export function statusParts(session, cwd = process.cwd()) {
function detailedStatus(session, cwd = process.cwd()) {
const cfg = session?.state?.config || {};
const ctx = contextStatus(session?.history, session?.state?.modelMeta);
- const usage = session?.usage || { inputTokens: 0, outputTokens: 0, totalTokens: 0 };
+ const usage = session?.usage || { apiCalls: 0 };
const providerMeta = (cfg.providers || []).find((row) => row.id === cfg.provider);
return {
provider: cfg.provider || null,
@@ -132,8 +123,18 @@ export function printDetailedStatus(session, C, cwd = process.cwd()) {
console.log(` auth ${row.providerAuth}`);
console.log(` context ~${row.contextEstimatedTokens}${row.contextWindow ? ` / ${row.contextWindow} (${row.contextPercent}%)` : " / ?"}${row.contextOmittedRows ? ` · ${row.contextOmittedRows} older rows projected out` : ""}`);
if (row.sessionCompactThresholdTokens) console.log(` session ~${row.sessionEstimatedTokens} / ${row.sessionCompactThresholdTokens} stored · lifetime ~${row.sessionLifetimeEstimatedTokens} · ${row.sessionCompactionCount} compactions · ${row.sessionArchiveCount} archives`);
- if (row.providerReportedUsage.totalTokens > 0) {
- console.log(` tokens ${row.providerReportedUsage.totalTokens} total · ${row.providerReportedUsage.inputTokens} in · ${row.providerReportedUsage.outputTokens} out`);
+ if ((row.providerReportedUsage.totalTokens ?? 0) > 0 || (row.providerReportedUsage.inputTokens ?? 0) > 0 || (row.providerReportedUsage.outputTokens ?? 0) > 0) {
+ const u = row.providerReportedUsage;
+ const parts = [
+ u.totalTokens != null ? `${u.totalTokens} total` : null,
+ u.inputTokens != null ? `${u.inputTokens} in` : null,
+ u.outputTokens != null ? `${u.outputTokens} out` : null,
+ u.cacheReadTokens != null ? `${u.cacheReadTokens} cache-read` : null,
+ u.cacheWriteTokens != null ? `${u.cacheWriteTokens} cache-write` : null,
+ u.reasoningTokens != null ? `${u.reasoningTokens} reasoning` : null,
+ u.accountingMode ? `mode ${u.accountingMode}` : null,
+ ].filter(Boolean);
+ console.log(` tokens ${parts.join(" · ")}`);
} else console.log(" tokens provider has not reported usage in this process; context remains estimated");
if (row.skill) console.log(` skill ${row.skill.state} /${row.skill.name}`);
console.log(` permission ${row.permission}`);
diff --git a/scripts/mso-agent-status.test.ts b/scripts/mso-agent-status.test.ts
index ce822311..5f2feb46 100644
--- a/scripts/mso-agent-status.test.ts
+++ b/scripts/mso-agent-status.test.ts
@@ -12,13 +12,13 @@ describe("MSO Agent dynamic status", () => {
it("accumulates provider-reported usage without losing fallback status", () => {
const usage = addUsage(addUsage(null, { inputTokens: 10, outputTokens: 4 }), { input_tokens: 6, output_tokens: 2 });
- expect(usage).toEqual({ inputTokens: 16, outputTokens: 6, totalTokens: 22 });
+ expect(usage).toEqual({ apiCalls: 2, inputTokens: 16, outputTokens: 6 });
const parts = statusParts({
state: { config: { provider: "openai-codex", model: "gpt-5.6-sol" }, modelMeta: { context: 128000 } },
history: [{ role: "user", text: "hello" }], usage, agentSession: { id: "20260901_120000_deadbeef" },
}, "/tmp");
expect(parts.join(" ")).toContain("openai-codex/gpt-5.6-sol");
- expect(parts.join(" ")).toContain("tokens 22");
+ expect(parts.join(" ")).toContain("tokens 16in/6out");
});
it("shows whether a skill is queued or really invoked", () => {
diff --git a/scripts/mso-agent.mjs b/scripts/mso-agent.mjs
index 159d69b2..555e8d3d 100644
--- a/scripts/mso-agent.mjs
+++ b/scripts/mso-agent.mjs
@@ -23,7 +23,7 @@ async function runOneShot(opts) {
state: s, agentSession, history: Array.isArray(agentSession.history) ? agentSession.history : [],
pendingSkill: null, activeSkill: null, lastInvokedSkill: null,
titleOverride: requested ? (agentSession.title || null) : null,
- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 }, lastRouting: null, lastElapsedMs: 0, statusBar: false,
+ usage: { apiCalls: 0 }, lastRouting: null, lastElapsedMs: 0, statusBar: false,
permission: "ask", pendingApproval: null,
};
session.history.push({ role: "user", text: opts.prompt });
@@ -65,7 +65,7 @@ async function main() {
history: Array.isArray(agentSession.history) ? agentSession.history : [],
pendingSkill: null, activeSkill: null, lastInvokedSkill: null,
titleOverride: (requested || restartSessionId) ? (agentSession.title || null) : null,
- usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 }, lastRouting: null, lastElapsedMs: 0,
+ usage: { apiCalls: 0 }, lastRouting: null, lastElapsedMs: 0,
statusBar: restartUi.statusBar ?? true, permission: forcedPermission, pendingApproval: null,
};
const rl = new AgentComposer({ input: process.stdin, output: process.stdout, colors: C }); syncPromptHistory(rl, session);