Skip to content

armgit / mso

Public Imported from GitHub
0

Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.

test(agent): calibrate local semantic memory retrieval

Rahman42ee0fc33a9e
diff --git a/README.md b/README.md
index c9febc56..0a454ed1 100644
--- a/README.md
+++ b/README.md
@@ -79,7 +79,7 @@ For a real deployment, put MSO behind **Tailscale, a VPN, or a TLS reverse proxy
- **Message live local sessions by a short name** — every durable same-principal MSO session gets a unique familiar handle such as `@milo`, `@luna`, or `@nara`, independent from its longer auto-generated session title. `/rename <name>` changes that handle, while `/title <text>` changes the session description. `@milo …` resolves only agents with a current lease **and** a live receiver subscription, then creates an explicitly correlated request; replies relay deterministically into the originating conversation while notify-only events remain passive. Presence lease (`idle`/`busy`/`offline`) and receiver subscription (`consumerConnected`) are reported separately so an idle-but-unsubscribed target is diagnosable. For ChatGPT/MCP sessions, `local_agent_inbox(wait_ms=...)` temporarily turns the foreground MCP call into that live receiver for up to 20 seconds, enabling direct two-way session messaging without a worker, DB, webhook, or broker. See [Local Agents](./docs/LOCAL-AGENTS.md).
- **Spawn focused same-session subagents** — `/spawn` and `agent_subagent_run` create bounded foreground workers with isolated context, explicit delegated scope, turn/timeout caps, no recursive spawn, and final-result-only return. They are not Local Agent peers and do not run autonomously after the parent call returns. See [Subagents](./docs/SUBAGENTS.md).
- **Delegate between remote agents with standard A2A v1** — register public HTTPS peers, attach private API-key/Bearer/OAuth access-token profiles, stream remote tasks over SSE, or expose MSO's authenticated inbound Agent Card when a public HTTPS origin is configured. Inbound credentials are owner-minted `read | write | exec` capabilities and run in a memory-isolated task context. See [A2A](./docs/A2A.md).
-- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:memory:calibration`, `bun run bench:memory:lifecycle`, `bun run bench:memory:retrieval-calibration`, `bun run bench:pipeline`, isolated scratch-fixture `bun run bench:corpus`, deterministic repeated `bun run bench:corpus:repeat`, and observation-only `bun run bench:cache-calibration` guard routing, memory correction/privacy/retention semantics, frozen-session behavior, lexical/relationship retrieval evidence gates, correctness, policy compliance, repeat coverage/spread, latency, context savings, cache telemetry, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.
+- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:memory:calibration`, `bun run bench:memory:lifecycle`, `bun run bench:memory:retrieval-calibration`, `bun run bench:pipeline`, isolated scratch-fixture `bun run bench:corpus`, deterministic repeated `bun run bench:corpus:repeat`, and observation-only `bun run bench:cache-calibration` guard routing, memory correction/privacy/retention semantics, frozen-session behavior, lexical/local-semantic/relationship retrieval evidence gates, correctness, policy compliance, repeat coverage/spread, latency, context savings, cache telemetry, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.
- **Make orchestration risk-aware and self-improving** — [RASMIC](./docs/RASMIC.md) classifies task risk/contention, retrieves compact repo-local memory, records manual user tests and Evidence Receipts, detects path/shared-resource collisions before merge conflicts, and promotes repeated safe routes from observed traces → recipes → tested bounded scripts.
- **Add app slices** — features are modular under `frontend/slices/<slug>/`.
- **Personalize the interface** — macOS, Windows, iOS, and Android shell layouts are UI preferences, not the core product.
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 8885abd0..a622fa5d 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -31,6 +31,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.
**Tests**
+- `agent` calibrate local semantic memory retrieval
- separate corpus accounting ownership
- keep Fresh 3 within architecture ratchets
- `gateway` await complete browser argv capture
diff --git a/docs/COGNITIVE-RUNTIME.md b/docs/COGNITIVE-RUNTIME.md
index 4ac71ad3..fc5715a3 100644
--- a/docs/COGNITIVE-RUNTIME.md
+++ b/docs/COGNITIVE-RUNTIME.md
@@ -381,7 +381,7 @@ Archive segments are bounded (up to 400 records / 768 KiB), owner-only (`0700` d
The lifecycle fixture also closed one deterministic lexical defect. A history search for `Runtime-1` could previously be crowded out by substring matches such as `Runtime-10` through `Runtime-19` before the result limit. Search scoring now prioritizes exact phrase/token matches and only uses bounded alphabetic-prefix matching for ordinary word stems. The existing 8-scenario `bench:memory` suite remains 100%.
-`bun run bench:memory:retrieval-calibration` deliberately probes cases that literal lexical retrieval may not solve. The bounded fixture currently passes **4/6 lexical cases**: paraphrase-with-overlap, long-value tail retrieval, similar-key disambiguation and exact domain phrase pass; `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office` remain objective misses. That is evidence of a semantic gap, but **not yet evidence for vector infrastructure**: no local embedding approach has demonstrated a measured recall benefit with acceptable privacy, latency and context cost. Vector/embedding storage therefore remains blocked rather than being added speculatively.
+`bun run bench:memory:retrieval-calibration` deliberately probes cases that literal lexical retrieval may not solve. The bounded fixture currently passes **4/6 lexical cases**: paraphrase-with-overlap, long-value tail retrieval, similar-key disambiguation and exact domain phrase pass; `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office` remain objective misses. P10D then calibrates the repo's already-existing local skill-routing encoder, **`mso-local-hybrid-v1`**, against the exact same memory rows with no network call and no persisted memory vectors. That candidate scores **3/6**: it does not fix either synonym/bilingual miss and additionally ranks `Deployment region` above the exact-domain rollback record. Because the local semantic candidate is **one case worse than lexical (−1/6)**, reusing it for memory would add vector/index complexity without recall benefit. Production vector/embedding storage therefore remains blocked; a future encoder must first beat the lexical baseline on a broader objective fixture before privacy, latency and index-lifecycle costs even become relevant.
The same calibration passes **3/3 two-hop relationship fixtures** by retrieving the first edge and then the intermediate-node edge with the existing keyed records. This proves only that the tested relationship chains are reachable without graph storage; it is not a universal graph-reasoning claim. Graph memory remains blocked until a reproduced task cannot be solved with bounded keyed retrieval and a smaller relationship layer measurably fixes it.
diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md
index 1917b3a1..ee6c2357 100644
--- a/docs/PROGRESS.md
+++ b/docs/PROGRESS.md
@@ -8,7 +8,7 @@
- **Frozen sessions now have explicit regression coverage:** the isolated lifecycle fixture proves session A retains its captured old memory after persistent memory changes, session B captures the new value, and resuming A returns A's frozen snapshot. No hidden mutable global memory context was added.
- **Threshold calibration:** `bench:memory:lifecycle` is **8/8 deterministic**. A synthetic 1,400-correction ledger compacts to **1,000 live + 401 archived records / two segments**, with a 56-byte resolved projection. The same fixture verifies archived-history retrieval, historical temporal resolution, archive permissions, archive-before-ledger crash semantics, backdated replacement over immutable cold records, and frozen sessions without touching real operator memory.
- **Retrieval evidence gate found and fixed exact-match crowding:** history query `Runtime-1` could be displaced by substring matches like `Runtime-10` before the result limit. Deterministic scoring now prioritizes exact phrases/tokens and keeps bounded alphabetic-prefix stemming. Existing `bench:memory` stays 8/8 / 100%.
-- **Vector and graph complexity remain evidence-blocked:** realistic retrieval calibration is **4/6 lexical**, with objective misses only for `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office`; no embedding layer is added because no candidate has yet proven better recall with acceptable privacy/latency/context cost. Relationship calibration is **3/3 two-hop**, so current keyed retrieval does not justify graph storage.
+- **Vector and graph complexity remain evidence-blocked:** realistic retrieval calibration is **4/6 lexical**, with objective misses only for `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office`. P10D compares the existing local `mso-local-hybrid-v1` semantic encoder against the same rows: it is deterministic but only **3/6**, fails the same two semantic cases and regresses the exact-domain rollback case. The candidate runs ephemerally with zero network calls / zero persisted memory vectors, so this is a direct architecture gate rather than an infrastructure limitation: adding a memory vector index now would make retrieval more complex without improving recall. Relationship calibration remains **3/3 two-hop**, so current keyed retrieval still does not justify graph storage.
- **P9 privacy semantics preserved:** 200-correction `bench:memory:calibration` remains 6/6 and future-scheduled claims still cannot resurrect after `forget`. Cache remains observation-only, cost remains attribution-blocked, OpenClaw remains unranked without a legitimate matched provider/model path, and five-run reliability remains descriptive.
- **MCP Apps navigation hardening (2026-09-03):** server `1.8.2` / toolset `2026.09.03.7` bumps all four UI resources to v2 and makes `Open in MSO` self-diagnosing: official `openExternal` + `setOpenInAppUrl`, visible success/failure feedback, and a user-clickable direct-link fallback. No tool count or security-scope changes.
## 2026-09-03 — Cognitive Runtime P9 memory calibration
diff --git a/docs/TESTING-HANDOFF.md b/docs/TESTING-HANDOFF.md
index ef7107a9..b3fbce99 100644
--- a/docs/TESTING-HANDOFF.md
+++ b/docs/TESTING-HANDOFF.md
@@ -8,7 +8,7 @@ The current release target is **MSO CLI 1.12.0**, MCP server `1.8.2`, toolset `2
Current verified behavior:
- Fresh 3 current MCP contract: server `1.8.2` / toolset `2026.09.03.7`; ChatGPT receives **62 transport tools = 61 model/operator actions (32 read / 17 write / 12 exec) + app-only `workflow_status`**, while the full transport catalog is 86 tools / 85 model tools. Every ChatGPT action advertises `outputSchema`, the dedicated MCP Apps widget origin is `https://mso-ui.rahmanef.com`, and five official digest-verified Skills are exposed through the MCP Skills/resource extension. The four MCP Apps use v2 resource URIs and `Open in MSO` now has visible bridge status, `setOpenInAppUrl`, and a direct-link fallback for host navigation failures.
-- P10 memory retention/lifecycle: isolated fixtures now archive only cold finished claims before the hard 2,000-record / 2 MiB live-ledger limits. Trigger/target are 1,400 -> <=1,000 live records (or ~1.4 MiB -> ~1 MiB); archive segments are content-addressed, owner-only and archive-before-ledger atomic. `bench:memory:lifecycle` is **8/8 deterministic** with a 1,400-correction seed ending at **1,000 live + 401 archived / two segments** and a 56-byte resolved projection. It verifies explicit archived-history retrieval, historical temporal resolution, aggregate-only no-content telemetry, `0700/0600` permissions, interrupted-commit live-wins semantics, backdated replacement over immutable archive, and frozen session A / fresh session B / resume-copy behavior. The P10 retrieval gate is **4/6 lexical** (synonym + Indonesian/English misses remain explicit), vector storage stays blocked without measured embedding benefit, and **3/3 two-hop relationship fixtures** keep graph storage blocked. The exact-history crowding bug (`Runtime-1` vs `Runtime-10...`) is fixed by exact phrase/token scoring; legacy `bench:memory` remains 100%.
+- P10 memory retention/lifecycle: isolated fixtures now archive only cold finished claims before the hard 2,000-record / 2 MiB live-ledger limits. Trigger/target are 1,400 -> <=1,000 live records (or ~1.4 MiB -> ~1 MiB); archive segments are content-addressed, owner-only and archive-before-ledger atomic. `bench:memory:lifecycle` is **8/8 deterministic** with a 1,400-correction seed ending at **1,000 live + 401 archived / two segments** and a 56-byte resolved projection. It verifies explicit archived-history retrieval, historical temporal resolution, aggregate-only no-content telemetry, `0700/0600` permissions, interrupted-commit live-wins semantics, backdated replacement over immutable archive, and frozen session A / fresh session B / resume-copy behavior. The P10 retrieval gate is **4/6 lexical** (synonym + Indonesian/English misses remain explicit); P10D also calibrates the existing local `mso-local-hybrid-v1` encoder on the same rows and gets **3/6** with zero network calls / zero persisted memory vectors, so production vector storage stays blocked by measured non-improvement rather than assumption. **3/3 two-hop relationship fixtures** keep graph storage blocked. The exact-history crowding bug (`Runtime-1` vs `Runtime-10...`) is fixed by exact phrase/token scoring; legacy `bench:memory` remains 100%.
- P9 memory calibration: isolated temporary-memory fixtures now cover repeated corrections, explicit-vs-observed authority, confidence/recency, future validity, forget, and history integrity. P9 fixed a real temporal privacy bug where `forget` could leave a future-scheduled claim that resurrected later; forget now closes all current/future-capable intervals while preserving already-ended history. Final 200-correction stress from source `d249da4`: **6/6 calibration scenarios**, existing memory benchmark **100%**, Cognitive Runtime memory gates **100%**; 200 ledger rows / 135,180 bytes resolve to 1 row / 58-byte compatibility projection (**99.96% smaller**) and history retrieval remains capped at 100. No calibrated case requires graph memory.
- P8 calibration: repeat aggregation now reports descriptive per-run spread without inferential claims; three new complete nine-scenario runs from source `51ae31f` were **27/27 task + 27/27 scenario-policy and 3/3 perfect runs for both agents**, with 54/54 row identity/provider/model/token-arithmetic checks passing. Combined with P7, both agents are **5/5 perfect**; five-run means are MSO vs Hermes 16,907.6 vs 66,791.1 normalized tokens/attempt (~74.7% lower for MSO), 18,985.9 vs 21,779.8 ms average latency, and 14,130.8 vs 21,457.2 ms per-run p50. New `bench:cache-calibration` final 4-round/12K-prefix run was 8/8 success; MSO cache-read was zero on 4/4 while Hermes naturally reported positive cache reads on 2/4 (9,728 + 19,968), but cache frequency is unranked because request envelopes differ. P8 also fixes false token non-comparability when independently valid normalized proof shapes vary across cache/no-cache rows. Cost remains non-comparable because attribution is asymmetric.
- P7 repeatability gate: `bench:corpus:repeat` adds explicit/deterministic seeds, alternating starting-agent order, and strict per-run/per-agent scenario + provider/model identity checks before any cross-run ranking. The first two complete 9-scenario runs from source `5bd5788` on `openai-codex/gpt-5.6-terra` were 18/18 task + 18/18 scenario-policy for both agents and 2/2 perfect runs each. Aggregate MSO vs Hermes: 17,271.3 vs 65,073.3 normalized tokens/attempt (~73.5% lower for MSO), 18,922.6 vs 21,077.4 ms average latency, and 13,761 vs 20,513.5 ms p50. All 36 token-proof arithmetic checks passed. This is a bounded two-run baseline, not statistical reliability proof; cost remains non-comparable.
diff --git a/scripts/bench-memory-retrieval-calibration.mjs b/scripts/bench-memory-retrieval-calibration.mjs
index 197360fd..031e5f03 100644
--- a/scripts/bench-memory-retrieval-calibration.mjs
+++ b/scripts/bench-memory-retrieval-calibration.mjs
@@ -1,12 +1,24 @@
#!/usr/bin/env bun
import { queryMemoryLedger } from "../lib/agent/memory-query.ts";
+import { hybridSemanticScore, prepareSemanticQuery, SKILL_EMBEDDING_VERSION } from "../lib/skills/semantic.ts";
const AT = "2026-09-03T00:00:00.000Z";
-let seq = 0;
-function rec(key, value) {
- seq += 1;
- return {
- id: `p10_retrieval_${seq}`,
+
+function ledger(records) { return { schemaVersion: 1, updatedAt: AT, records }; }
+function lexicalTopKey(records, query) { return queryMemoryLedger(ledger(records), { query, at: AT }).records[0]?.record.key ?? null; }
+function semanticTopKey(records, query) {
+ const prepared = prepareSemanticQuery(query);
+ const ranked = records
+ .map((record) => ({ key: record.key, score: hybridSemanticScore(prepared, `${record.key}\n${record.value}`) }))
+ .sort((a, b) => b.score - a.score || a.key.localeCompare(b.key));
+ return ranked[0]?.score > 0 ? ranked[0].key : null;
+}
+function pct(n, d) { return d ? Math.round(n / d * 1000) / 10 : 0; }
+
+export function runMemoryRetrievalCalibration() {
+ let seq = 0;
+ const rec = (key, value) => ({
+ id: `p10_retrieval_${++seq}`,
document: "MEMORY.md",
key,
value,
@@ -16,96 +28,120 @@ function rec(key, value) {
validFrom: "2026-01-01T00:00:00.000Z",
createdAt: "2026-01-01T00:00:00.000Z",
provenance: { authority: "explicit", channel: "system", observedAt: "2026-01-01T00:00:00.000Z" },
- };
-}
-function ledger(records) { return { schemaVersion: 1, updatedAt: AT, records }; }
-function topKey(records, query) { return queryMemoryLedger(ledger(records), { query, at: AT }).records[0]?.record.key ?? null; }
+ });
+
+ const retrievalCases = [
+ {
+ id: "paraphrase-same-concept",
+ class: "paraphrase",
+ records: [rec("Deployment rollback policy", "Verify service health before rolling back a release"), rec("Frontend convention", "Use Svelte 5 runes")],
+ query: "undo release safely",
+ expected: "Deployment rollback policy",
+ },
+ {
+ id: "synonym-ide-editor",
+ class: "synonym",
+ records: [rec("Primary editor", "VS Code"), rec("Primary shell", "zsh")],
+ query: "IDE preference",
+ expected: "Primary editor",
+ },
+ {
+ id: "bilingual-office-kantor",
+ class: "bilingual",
+ records: [rec("Office location", "Jakarta"), rec("Release region", "Singapore")],
+ query: "kantor utama",
+ expected: "Office location",
+ },
+ {
+ id: "long-value-tail",
+ class: "long-value",
+ records: [rec("Release handbook", `${"Operational notes. ".repeat(80)}Before rollback always verify service health and preserve evidence.`), rec("Design handbook", "Use compact mobile navigation")],
+ query: "verify service health",
+ expected: "Release handbook",
+ },
+ {
+ id: "similar-keys-disambiguation",
+ class: "similar-keys",
+ records: [rec("Primary database", "PostgreSQL"), rec("Analytics database", "ClickHouse"), rec("Archive database", "SQLite")],
+ query: "analytics database",
+ expected: "Analytics database",
+ },
+ {
+ id: "exact-domain-phrase",
+ class: "lexical-control",
+ records: [rec("Deployment rollback policy", "Verify service health"), rec("Deployment region", "Jakarta")],
+ query: "deployment rollback",
+ expected: "Deployment rollback policy",
+ },
+ ];
-const retrievalCases = [
- {
- id: "paraphrase-same-concept",
- class: "paraphrase",
- records: [rec("Deployment rollback policy", "Verify service health before rolling back a release"), rec("Frontend convention", "Use Svelte 5 runes")],
- query: "undo release safely",
- expected: "Deployment rollback policy",
- },
- {
- id: "synonym-ide-editor",
- class: "synonym",
- records: [rec("Primary editor", "VS Code"), rec("Primary shell", "zsh")],
- query: "IDE preference",
- expected: "Primary editor",
- },
- {
- id: "bilingual-office-kantor",
- class: "bilingual",
- records: [rec("Office location", "Jakarta"), rec("Release region", "Singapore")],
- query: "kantor utama",
- expected: "Office location",
- },
- {
- id: "long-value-tail",
- class: "long-value",
- records: [rec("Release handbook", `${"Operational notes. ".repeat(80)}Before rollback always verify service health and preserve evidence.`), rec("Design handbook", "Use compact mobile navigation")],
- query: "verify service health",
- expected: "Release handbook",
- },
- {
- id: "similar-keys-disambiguation",
- class: "similar-keys",
- records: [rec("Primary database", "PostgreSQL"), rec("Analytics database", "ClickHouse"), rec("Archive database", "SQLite")],
- query: "analytics database",
- expected: "Analytics database",
- },
- {
- id: "exact-domain-phrase",
- class: "lexical-control",
- records: [rec("Deployment rollback policy", "Verify service health"), rec("Deployment region", "Jakarta")],
- query: "deployment rollback",
- expected: "Deployment rollback policy",
- },
-];
+ const retrieval = retrievalCases.map((row) => {
+ const first = lexicalTopKey(row.records, row.query), second = lexicalTopKey(row.records, row.query);
+ return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };
+ });
+ const semantic = retrievalCases.map((row) => {
+ const first = semanticTopKey(row.records, row.query), second = semanticTopKey(row.records, row.query);
+ return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };
+ });
-const retrieval = retrievalCases.map((row) => {
- const first = topKey(row.records, row.query), second = topKey(row.records, row.query);
- return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };
-});
+ const graphCases = [
+ { id: "ownership-dependency", first: "Atlas", intermediate: "Beacon", second: "Core", records: [rec("Atlas ownership", "Atlas owns Beacon"), rec("Beacon dependency", "Beacon depends on Core"), rec("Unrelated preference", "Compact UI")] },
+ { id: "service-team", first: "Orchid", intermediate: "Platform", second: "Infra", records: [rec("Orchid owner", "Orchid is owned by Platform"), rec("Platform dependency", "Platform depends on Infra"), rec("Theme", "Dark") ] },
+ { id: "module-package", first: "Renderer", intermediate: "Scene", second: "Assets", records: [rec("Renderer relation", "Renderer uses Scene"), rec("Scene relation", "Scene consumes Assets"), rec("Editor", "VS Code") ] },
+ ];
-const graphCases = [
- { id: "ownership-dependency", first: "Atlas", intermediate: "Beacon", second: "Core", records: [rec("Atlas ownership", "Atlas owns Beacon"), rec("Beacon dependency", "Beacon depends on Core"), rec("Unrelated preference", "Compact UI")] },
- { id: "service-team", first: "Orchid", intermediate: "Platform", second: "Infra", records: [rec("Orchid owner", "Orchid is owned by Platform"), rec("Platform dependency", "Platform depends on Infra"), rec("Theme", "Dark") ] },
- { id: "module-package", first: "Renderer", intermediate: "Scene", second: "Assets", records: [rec("Renderer relation", "Renderer uses Scene"), rec("Scene relation", "Scene consumes Assets"), rec("Editor", "VS Code") ] },
-];
+ const graph = graphCases.map((row) => {
+ const firstHop = queryMemoryLedger(ledger(row.records), { query: row.first, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);
+ const secondHop = queryMemoryLedger(ledger(row.records), { query: row.intermediate, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);
+ const pass = firstHop.some((text) => text.includes(row.intermediate)) && secondHop.some((text) => text.includes(row.second));
+ return { id: row.id, pass, hops: 2, firstHopRows: firstHop.length, secondHopRows: secondHop.length };
+ });
-const graph = graphCases.map((row) => {
- const firstHop = queryMemoryLedger(ledger(row.records), { query: row.first, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);
- const secondHop = queryMemoryLedger(ledger(row.records), { query: row.intermediate, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);
- const pass = firstHop.some((text) => text.includes(row.intermediate)) && secondHop.some((text) => text.includes(row.second));
- return { id: row.id, pass, hops: 2, firstHopRows: firstHop.length, secondHopRows: secondHop.length };
-});
+ const retrievalPassed = retrieval.filter((row) => row.pass && row.deterministic).length;
+ const semanticPassed = semantic.filter((row) => row.pass && row.deterministic).length;
+ const lexicalMisses = retrieval.filter((row) => !row.pass).map((row) => row.id);
+ const semanticMisses = semantic.filter((row) => !row.pass).map((row) => row.id);
+ const graphPassed = graph.filter((row) => row.pass).length;
+ const candidateImprovesRecall = semanticPassed > retrievalPassed;
-const retrievalPassed = retrieval.filter((row) => row.pass && row.deterministic).length;
-const semanticMisses = retrieval.filter((row) => !row.pass).map((row) => row.id);
-const graphPassed = graph.filter((row) => row.pass).length;
-const result = {
- calibrationVersion: "mso-memory-retrieval-calibration-v1",
- retrieval: { passed: retrievalPassed, total: retrieval.length, accuracyPct: Math.round(retrievalPassed / retrieval.length * 1000) / 10, deterministic: retrieval.every((row) => row.deterministic), rows: retrieval },
- semanticEvidenceGate: {
- objectiveMisses: semanticMisses,
- vectorLayerRequired: false,
- reason: semanticMisses.length
- ? "The fixture exposes semantic gaps, but no embedding approach has yet proven a measurable benefit with acceptable privacy/token/latency cost. Keep vector infrastructure blocked until that comparison exists."
- : "The deterministic lexical fixture is adequate; no vector layer is justified.",
- },
- graph: { passed: graphPassed, total: graph.length, rows: graph, graphStorageRequired: graphPassed !== graph.length },
- note: "This is a bounded synthetic evidence gate. Graph rows prove whether small relationship chains remain reachable through explicit two-step retrieval; they do not claim general graph reasoning reliability.",
-};
+ return {
+ calibrationVersion: "mso-memory-retrieval-calibration-v2",
+ retrieval: { passed: retrievalPassed, total: retrieval.length, accuracyPct: pct(retrievalPassed, retrieval.length), deterministic: retrieval.every((row) => row.deterministic), rows: retrieval },
+ semanticCandidate: {
+ encoder: SKILL_EMBEDDING_VERSION,
+ mode: "existing-local-ephemeral-hybrid",
+ networkCalls: 0,
+ persistedMemoryVectors: 0,
+ passed: semanticPassed,
+ total: semantic.length,
+ accuracyPct: pct(semanticPassed, semantic.length),
+ deterministic: semantic.every((row) => row.deterministic),
+ deltaPassedVsLexical: semanticPassed - retrievalPassed,
+ rows: semantic,
+ },
+ semanticEvidenceGate: {
+ objectiveLexicalMisses: lexicalMisses,
+ candidateMisses: semanticMisses,
+ candidateImprovesRecall,
+ vectorLayerRequired: false,
+ reason: candidateImprovesRecall
+ ? "The existing local semantic encoder improves this bounded fixture, but production memory vectors remain blocked until privacy, latency and index lifecycle costs are calibrated."
+ : "The existing local semantic encoder does not improve this bounded memory fixture, so reusing it for production memory retrieval would add vector/index complexity without a recall benefit.",
+ },
+ graph: { passed: graphPassed, total: graph.length, rows: graph, graphStorageRequired: graphPassed !== graph.length },
+ note: "This is a bounded synthetic evidence gate. The semantic candidate is the repo's existing local skill-routing encoder and is evaluated ephemerally with no network call or persisted memory vector. Graph rows prove only whether these small relationship chains remain reachable through explicit two-step retrieval; neither result is a universal retrieval/reasoning claim.",
+ };
+}
-if (process.argv.includes("--json")) console.log(JSON.stringify(result, null, 2));
-else {
- console.log(`MSO memory retrieval calibration · lexical ${result.retrieval.passed}/${result.retrieval.total} · deterministic=${result.retrieval.deterministic}`);
- for (const row of result.retrieval.rows) console.log(` ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);
- console.log(` semantic gate: ${result.semanticEvidenceGate.vectorLayerRequired ? "vector candidate" : "vector blocked"}`);
- console.log(` graph gate: ${result.graph.passed}/${result.graph.total} two-hop fixtures · graph storage ${result.graph.graphStorageRequired ? "candidate" : "blocked"}`);
+if (import.meta.main) {
+ const result = runMemoryRetrievalCalibration();
+ if (process.argv.includes("--json")) console.log(JSON.stringify(result, null, 2));
+ else {
+ console.log(`MSO memory retrieval calibration · lexical ${result.retrieval.passed}/${result.retrieval.total} · local semantic ${result.semanticCandidate.passed}/${result.semanticCandidate.total}`);
+ for (const row of result.retrieval.rows) console.log(` lexical ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);
+ for (const row of result.semanticCandidate.rows) console.log(` semantic ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);
+ console.log(` semantic gate: ${result.semanticEvidenceGate.vectorLayerRequired ? "vector candidate" : "vector blocked"} · delta=${result.semanticCandidate.deltaPassedVsLexical}`);
+ console.log(` graph gate: ${result.graph.passed}/${result.graph.total} two-hop fixtures · graph storage ${result.graph.graphStorageRequired ? "candidate" : "blocked"}`);
+ }
+ if (!result.retrieval.deterministic || !result.semanticCandidate.deterministic || result.graph.passed !== result.graph.total) process.exitCode = 1;
}
-if (!result.retrieval.deterministic || result.graph.passed !== result.graph.total) process.exitCode = 1;
diff --git a/scripts/bench-memory-retrieval-calibration.test.ts b/scripts/bench-memory-retrieval-calibration.test.ts
new file mode 100644
index 00000000..95758933
--- /dev/null
+++ b/scripts/bench-memory-retrieval-calibration.test.ts
@@ -0,0 +1,21 @@
+import { describe, expect, it } from "vitest";
+import { runMemoryRetrievalCalibration } from "./bench-memory-retrieval-calibration.mjs";
+
+describe("P10D memory retrieval evidence gate", () => {
+ it("compares the existing local semantic encoder against lexical retrieval without adding a memory vector index", () => {
+ const result = runMemoryRetrievalCalibration();
+ expect(result.retrieval.passed).toBe(4);
+ expect(result.retrieval.total).toBe(6);
+ expect(result.semanticCandidate.encoder).toBe("mso-local-hybrid-v1");
+ expect(result.semanticCandidate.passed).toBe(3);
+ expect(result.semanticCandidate.total).toBe(6);
+ expect(result.semanticCandidate.deltaPassedVsLexical).toBe(-1);
+ expect(result.semanticCandidate.networkCalls).toBe(0);
+ expect(result.semanticCandidate.persistedMemoryVectors).toBe(0);
+ expect(result.semanticEvidenceGate.candidateImprovesRecall).toBe(false);
+ expect(result.semanticEvidenceGate.vectorLayerRequired).toBe(false);
+ expect(result.graph.passed).toBe(3);
+ expect(result.graph.total).toBe(3);
+ expect(result.graph.graphStorageRequired).toBe(false);
+ });
+});