Browser-based graphical shell and control plane for a Linux server you own — real terminal, file manager, live metrics and BYOK AI in one mobile-first workspace. Next.js 16 · React 19 · node-pty · Tailwind v4 · shadcn/ui. Public alpha, self-hosted.
TypeScript browser-shellcontrol-planefile-managerhomelabmobile-firstnextjsptyreactself-hostedserver-managementshadcn-uisysadmintailwindcssterminaltypescriptvpsweb-desktopweb-terminal
test(agent): calibrate local semantic memory retrieval
Rahman42ee0fc33a9e
diff --git a/README.md b/README.mdindex c9febc56..0a454ed1 100644--- a/README.md+++ b/README.md@@ -79,7 +79,7 @@ For a real deployment, put MSO behind **Tailscale, a VPN, or a TLS reverse proxy- **Message live local sessions by a short name** — every durable same-principal MSO session gets a unique familiar handle such as `@milo`, `@luna`, or `@nara`, independent from its longer auto-generated session title. `/rename <name>` changes that handle, while `/title <text>` changes the session description. `@milo …` resolves only agents with a current lease **and** a live receiver subscription, then creates an explicitly correlated request; replies relay deterministically into the originating conversation while notify-only events remain passive. Presence lease (`idle`/`busy`/`offline`) and receiver subscription (`consumerConnected`) are reported separately so an idle-but-unsubscribed target is diagnosable. For ChatGPT/MCP sessions, `local_agent_inbox(wait_ms=...)` temporarily turns the foreground MCP call into that live receiver for up to 20 seconds, enabling direct two-way session messaging without a worker, DB, webhook, or broker. See [Local Agents](./docs/LOCAL-AGENTS.md).- **Spawn focused same-session subagents** — `/spawn` and `agent_subagent_run` create bounded foreground workers with isolated context, explicit delegated scope, turn/timeout caps, no recursive spawn, and final-result-only return. They are not Local Agent peers and do not run autonomously after the parent call returns. See [Subagents](./docs/SUBAGENTS.md).- **Delegate between remote agents with standard A2A v1** — register public HTTPS peers, attach private API-key/Bearer/OAuth access-token profiles, stream remote tasks over SSE, or expose MSO's authenticated inbound Agent Card when a public HTTPS origin is configured. Inbound credentials are owner-minted `read | write | exec` capabilities and run in a memory-isolated task context. See [A2A](./docs/A2A.md).-- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:memory:calibration`, `bun run bench:memory:lifecycle`, `bun run bench:memory:retrieval-calibration`, `bun run bench:pipeline`, isolated scratch-fixture `bun run bench:corpus`, deterministic repeated `bun run bench:corpus:repeat`, and observation-only `bun run bench:cache-calibration` guard routing, memory correction/privacy/retention semantics, frozen-session behavior, lexical/relationship retrieval evidence gates, correctness, policy compliance, repeat coverage/spread, latency, context savings, cache telemetry, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.+- **Keep agents efficient independently of the model provider** — the MCP-first [Cognitive Runtime](./docs/COGNITIVE-RUNTIME.md) isolates conversations from OAuth identity, budgets model context/tool output, uses the RASMIC catalog-first router to select a tiny per-turn capability pack without weakening scopes, compacts durable sessions with sanitized 30-day backup archives, learns only verified workflow recipes, promotes repeated procedures only through the eval-gated Tool Forge, and batches/filter/aggregates independent reads through `read_pipeline` before raw data reaches the model. `bun run bench:cognitive`, `bun run bench:memory:calibration`, `bun run bench:memory:lifecycle`, `bun run bench:memory:retrieval-calibration`, `bun run bench:pipeline`, isolated scratch-fixture `bun run bench:corpus`, deterministic repeated `bun run bench:corpus:repeat`, and observation-only `bun run bench:cache-calibration` guard routing, memory correction/privacy/retention semantics, frozen-session behavior, lexical/local-semantic/relationship retrieval evidence gates, correctness, policy compliance, repeat coverage/spread, latency, context savings, cache telemetry, and proof-gated normalized provider usage so cache/reasoning details are never double-counted or guessed.- **Make orchestration risk-aware and self-improving** — [RASMIC](./docs/RASMIC.md) classifies task risk/contention, retrieves compact repo-local memory, records manual user tests and Evidence Receipts, detects path/shared-resource collisions before merge conflicts, and promotes repeated safe routes from observed traces → recipes → tested bounded scripts.- **Add app slices** — features are modular under `frontend/slices/<slug>/`.- **Personalize the interface** — macOS, Windows, iOS, and Android shell layouts are UI preferences, not the core product.diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.mdindex 8885abd0..a622fa5d 100644--- a/docs/CHANGELOG.md+++ b/docs/CHANGELOG.md@@ -31,6 +31,7 @@ this is the *what*, and it is what Settings → About shows as “What's new”.**Tests**+- `agent` calibrate local semantic memory retrieval- separate corpus accounting ownership- keep Fresh 3 within architecture ratchets- `gateway` await complete browser argv capturediff --git a/docs/COGNITIVE-RUNTIME.md b/docs/COGNITIVE-RUNTIME.mdindex 4ac71ad3..fc5715a3 100644--- a/docs/COGNITIVE-RUNTIME.md+++ b/docs/COGNITIVE-RUNTIME.md@@ -381,7 +381,7 @@ Archive segments are bounded (up to 400 records / 768 KiB), owner-only (`0700` dThe lifecycle fixture also closed one deterministic lexical defect. A history search for `Runtime-1` could previously be crowded out by substring matches such as `Runtime-10` through `Runtime-19` before the result limit. Search scoring now prioritizes exact phrase/token matches and only uses bounded alphabetic-prefix matching for ordinary word stems. The existing 8-scenario `bench:memory` suite remains 100%.-`bun run bench:memory:retrieval-calibration` deliberately probes cases that literal lexical retrieval may not solve. The bounded fixture currently passes **4/6 lexical cases**: paraphrase-with-overlap, long-value tail retrieval, similar-key disambiguation and exact domain phrase pass; `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office` remain objective misses. That is evidence of a semantic gap, but **not yet evidence for vector infrastructure**: no local embedding approach has demonstrated a measured recall benefit with acceptable privacy, latency and context cost. Vector/embedding storage therefore remains blocked rather than being added speculatively.+`bun run bench:memory:retrieval-calibration` deliberately probes cases that literal lexical retrieval may not solve. The bounded fixture currently passes **4/6 lexical cases**: paraphrase-with-overlap, long-value tail retrieval, similar-key disambiguation and exact domain phrase pass; `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office` remain objective misses. P10D then calibrates the repo's already-existing local skill-routing encoder, **`mso-local-hybrid-v1`**, against the exact same memory rows with no network call and no persisted memory vectors. That candidate scores **3/6**: it does not fix either synonym/bilingual miss and additionally ranks `Deployment region` above the exact-domain rollback record. Because the local semantic candidate is **one case worse than lexical (−1/6)**, reusing it for memory would add vector/index complexity without recall benefit. Production vector/embedding storage therefore remains blocked; a future encoder must first beat the lexical baseline on a broader objective fixture before privacy, latency and index-lifecycle costs even become relevant.The same calibration passes **3/3 two-hop relationship fixtures** by retrieving the first edge and then the intermediate-node edge with the existing keyed records. This proves only that the tested relationship chains are reachable without graph storage; it is not a universal graph-reasoning claim. Graph memory remains blocked until a reproduced task cannot be solved with bounded keyed retrieval and a smaller relationship layer measurably fixes it.diff --git a/docs/PROGRESS.md b/docs/PROGRESS.mdindex 1917b3a1..ee6c2357 100644--- a/docs/PROGRESS.md+++ b/docs/PROGRESS.md@@ -8,7 +8,7 @@- **Frozen sessions now have explicit regression coverage:** the isolated lifecycle fixture proves session A retains its captured old memory after persistent memory changes, session B captures the new value, and resuming A returns A's frozen snapshot. No hidden mutable global memory context was added.- **Threshold calibration:** `bench:memory:lifecycle` is **8/8 deterministic**. A synthetic 1,400-correction ledger compacts to **1,000 live + 401 archived records / two segments**, with a 56-byte resolved projection. The same fixture verifies archived-history retrieval, historical temporal resolution, archive permissions, archive-before-ledger crash semantics, backdated replacement over immutable cold records, and frozen sessions without touching real operator memory.- **Retrieval evidence gate found and fixed exact-match crowding:** history query `Runtime-1` could be displaced by substring matches like `Runtime-10` before the result limit. Deterministic scoring now prioritizes exact phrases/tokens and keeps bounded alphabetic-prefix stemming. Existing `bench:memory` stays 8/8 / 100%.-- **Vector and graph complexity remain evidence-blocked:** realistic retrieval calibration is **4/6 lexical**, with objective misses only for `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office`; no embedding layer is added because no candidate has yet proven better recall with acceptable privacy/latency/context cost. Relationship calibration is **3/3 two-hop**, so current keyed retrieval does not justify graph storage.+- **Vector and graph complexity remain evidence-blocked:** realistic retrieval calibration is **4/6 lexical**, with objective misses only for `IDE`↔`editor` synonym and Indonesian `kantor`↔English `office`. P10D compares the existing local `mso-local-hybrid-v1` semantic encoder against the same rows: it is deterministic but only **3/6**, fails the same two semantic cases and regresses the exact-domain rollback case. The candidate runs ephemerally with zero network calls / zero persisted memory vectors, so this is a direct architecture gate rather than an infrastructure limitation: adding a memory vector index now would make retrieval more complex without improving recall. Relationship calibration remains **3/3 two-hop**, so current keyed retrieval still does not justify graph storage.- **P9 privacy semantics preserved:** 200-correction `bench:memory:calibration` remains 6/6 and future-scheduled claims still cannot resurrect after `forget`. Cache remains observation-only, cost remains attribution-blocked, OpenClaw remains unranked without a legitimate matched provider/model path, and five-run reliability remains descriptive.- **MCP Apps navigation hardening (2026-09-03):** server `1.8.2` / toolset `2026.09.03.7` bumps all four UI resources to v2 and makes `Open in MSO` self-diagnosing: official `openExternal` + `setOpenInAppUrl`, visible success/failure feedback, and a user-clickable direct-link fallback. No tool count or security-scope changes.## 2026-09-03 — Cognitive Runtime P9 memory calibrationdiff --git a/docs/TESTING-HANDOFF.md b/docs/TESTING-HANDOFF.mdindex ef7107a9..b3fbce99 100644--- a/docs/TESTING-HANDOFF.md+++ b/docs/TESTING-HANDOFF.md@@ -8,7 +8,7 @@ The current release target is **MSO CLI 1.12.0**, MCP server `1.8.2`, toolset `2Current verified behavior:- Fresh 3 current MCP contract: server `1.8.2` / toolset `2026.09.03.7`; ChatGPT receives **62 transport tools = 61 model/operator actions (32 read / 17 write / 12 exec) + app-only `workflow_status`**, while the full transport catalog is 86 tools / 85 model tools. Every ChatGPT action advertises `outputSchema`, the dedicated MCP Apps widget origin is `https://mso-ui.rahmanef.com`, and five official digest-verified Skills are exposed through the MCP Skills/resource extension. The four MCP Apps use v2 resource URIs and `Open in MSO` now has visible bridge status, `setOpenInAppUrl`, and a direct-link fallback for host navigation failures.-- P10 memory retention/lifecycle: isolated fixtures now archive only cold finished claims before the hard 2,000-record / 2 MiB live-ledger limits. Trigger/target are 1,400 -> <=1,000 live records (or ~1.4 MiB -> ~1 MiB); archive segments are content-addressed, owner-only and archive-before-ledger atomic. `bench:memory:lifecycle` is **8/8 deterministic** with a 1,400-correction seed ending at **1,000 live + 401 archived / two segments** and a 56-byte resolved projection. It verifies explicit archived-history retrieval, historical temporal resolution, aggregate-only no-content telemetry, `0700/0600` permissions, interrupted-commit live-wins semantics, backdated replacement over immutable archive, and frozen session A / fresh session B / resume-copy behavior. The P10 retrieval gate is **4/6 lexical** (synonym + Indonesian/English misses remain explicit), vector storage stays blocked without measured embedding benefit, and **3/3 two-hop relationship fixtures** keep graph storage blocked. The exact-history crowding bug (`Runtime-1` vs `Runtime-10...`) is fixed by exact phrase/token scoring; legacy `bench:memory` remains 100%.+- P10 memory retention/lifecycle: isolated fixtures now archive only cold finished claims before the hard 2,000-record / 2 MiB live-ledger limits. Trigger/target are 1,400 -> <=1,000 live records (or ~1.4 MiB -> ~1 MiB); archive segments are content-addressed, owner-only and archive-before-ledger atomic. `bench:memory:lifecycle` is **8/8 deterministic** with a 1,400-correction seed ending at **1,000 live + 401 archived / two segments** and a 56-byte resolved projection. It verifies explicit archived-history retrieval, historical temporal resolution, aggregate-only no-content telemetry, `0700/0600` permissions, interrupted-commit live-wins semantics, backdated replacement over immutable archive, and frozen session A / fresh session B / resume-copy behavior. The P10 retrieval gate is **4/6 lexical** (synonym + Indonesian/English misses remain explicit); P10D also calibrates the existing local `mso-local-hybrid-v1` encoder on the same rows and gets **3/6** with zero network calls / zero persisted memory vectors, so production vector storage stays blocked by measured non-improvement rather than assumption. **3/3 two-hop relationship fixtures** keep graph storage blocked. The exact-history crowding bug (`Runtime-1` vs `Runtime-10...`) is fixed by exact phrase/token scoring; legacy `bench:memory` remains 100%.- P9 memory calibration: isolated temporary-memory fixtures now cover repeated corrections, explicit-vs-observed authority, confidence/recency, future validity, forget, and history integrity. P9 fixed a real temporal privacy bug where `forget` could leave a future-scheduled claim that resurrected later; forget now closes all current/future-capable intervals while preserving already-ended history. Final 200-correction stress from source `d249da4`: **6/6 calibration scenarios**, existing memory benchmark **100%**, Cognitive Runtime memory gates **100%**; 200 ledger rows / 135,180 bytes resolve to 1 row / 58-byte compatibility projection (**99.96% smaller**) and history retrieval remains capped at 100. No calibrated case requires graph memory.- P8 calibration: repeat aggregation now reports descriptive per-run spread without inferential claims; three new complete nine-scenario runs from source `51ae31f` were **27/27 task + 27/27 scenario-policy and 3/3 perfect runs for both agents**, with 54/54 row identity/provider/model/token-arithmetic checks passing. Combined with P7, both agents are **5/5 perfect**; five-run means are MSO vs Hermes 16,907.6 vs 66,791.1 normalized tokens/attempt (~74.7% lower for MSO), 18,985.9 vs 21,779.8 ms average latency, and 14,130.8 vs 21,457.2 ms per-run p50. New `bench:cache-calibration` final 4-round/12K-prefix run was 8/8 success; MSO cache-read was zero on 4/4 while Hermes naturally reported positive cache reads on 2/4 (9,728 + 19,968), but cache frequency is unranked because request envelopes differ. P8 also fixes false token non-comparability when independently valid normalized proof shapes vary across cache/no-cache rows. Cost remains non-comparable because attribution is asymmetric.- P7 repeatability gate: `bench:corpus:repeat` adds explicit/deterministic seeds, alternating starting-agent order, and strict per-run/per-agent scenario + provider/model identity checks before any cross-run ranking. The first two complete 9-scenario runs from source `5bd5788` on `openai-codex/gpt-5.6-terra` were 18/18 task + 18/18 scenario-policy for both agents and 2/2 perfect runs each. Aggregate MSO vs Hermes: 17,271.3 vs 65,073.3 normalized tokens/attempt (~73.5% lower for MSO), 18,922.6 vs 21,077.4 ms average latency, and 13,761 vs 20,513.5 ms p50. All 36 token-proof arithmetic checks passed. This is a bounded two-run baseline, not statistical reliability proof; cost remains non-comparable.diff --git a/scripts/bench-memory-retrieval-calibration.mjs b/scripts/bench-memory-retrieval-calibration.mjsindex 197360fd..031e5f03 100644--- a/scripts/bench-memory-retrieval-calibration.mjs+++ b/scripts/bench-memory-retrieval-calibration.mjs@@ -1,12 +1,24 @@#!/usr/bin/env bunimport { queryMemoryLedger } from "../lib/agent/memory-query.ts";+import { hybridSemanticScore, prepareSemanticQuery, SKILL_EMBEDDING_VERSION } from "../lib/skills/semantic.ts";const AT = "2026-09-03T00:00:00.000Z";-let seq = 0;-function rec(key, value) {- seq += 1;- return {- id: `p10_retrieval_${seq}`,++function ledger(records) { return { schemaVersion: 1, updatedAt: AT, records }; }+function lexicalTopKey(records, query) { return queryMemoryLedger(ledger(records), { query, at: AT }).records[0]?.record.key ?? null; }+function semanticTopKey(records, query) {+ const prepared = prepareSemanticQuery(query);+ const ranked = records+ .map((record) => ({ key: record.key, score: hybridSemanticScore(prepared, `${record.key}\n${record.value}`) }))+ .sort((a, b) => b.score - a.score || a.key.localeCompare(b.key));+ return ranked[0]?.score > 0 ? ranked[0].key : null;+}+function pct(n, d) { return d ? Math.round(n / d * 1000) / 10 : 0; }++export function runMemoryRetrievalCalibration() {+ let seq = 0;+ const rec = (key, value) => ({+ id: `p10_retrieval_${++seq}`,document: "MEMORY.md",key,value,@@ -16,96 +28,120 @@ function rec(key, value) {validFrom: "2026-01-01T00:00:00.000Z",createdAt: "2026-01-01T00:00:00.000Z",provenance: { authority: "explicit", channel: "system", observedAt: "2026-01-01T00:00:00.000Z" },- };-}-function ledger(records) { return { schemaVersion: 1, updatedAt: AT, records }; }-function topKey(records, query) { return queryMemoryLedger(ledger(records), { query, at: AT }).records[0]?.record.key ?? null; }+ });++ const retrievalCases = [+ {+ id: "paraphrase-same-concept",+ class: "paraphrase",+ records: [rec("Deployment rollback policy", "Verify service health before rolling back a release"), rec("Frontend convention", "Use Svelte 5 runes")],+ query: "undo release safely",+ expected: "Deployment rollback policy",+ },+ {+ id: "synonym-ide-editor",+ class: "synonym",+ records: [rec("Primary editor", "VS Code"), rec("Primary shell", "zsh")],+ query: "IDE preference",+ expected: "Primary editor",+ },+ {+ id: "bilingual-office-kantor",+ class: "bilingual",+ records: [rec("Office location", "Jakarta"), rec("Release region", "Singapore")],+ query: "kantor utama",+ expected: "Office location",+ },+ {+ id: "long-value-tail",+ class: "long-value",+ records: [rec("Release handbook", `${"Operational notes. ".repeat(80)}Before rollback always verify service health and preserve evidence.`), rec("Design handbook", "Use compact mobile navigation")],+ query: "verify service health",+ expected: "Release handbook",+ },+ {+ id: "similar-keys-disambiguation",+ class: "similar-keys",+ records: [rec("Primary database", "PostgreSQL"), rec("Analytics database", "ClickHouse"), rec("Archive database", "SQLite")],+ query: "analytics database",+ expected: "Analytics database",+ },+ {+ id: "exact-domain-phrase",+ class: "lexical-control",+ records: [rec("Deployment rollback policy", "Verify service health"), rec("Deployment region", "Jakarta")],+ query: "deployment rollback",+ expected: "Deployment rollback policy",+ },+ ];-const retrievalCases = [- {- id: "paraphrase-same-concept",- class: "paraphrase",- records: [rec("Deployment rollback policy", "Verify service health before rolling back a release"), rec("Frontend convention", "Use Svelte 5 runes")],- query: "undo release safely",- expected: "Deployment rollback policy",- },- {- id: "synonym-ide-editor",- class: "synonym",- records: [rec("Primary editor", "VS Code"), rec("Primary shell", "zsh")],- query: "IDE preference",- expected: "Primary editor",- },- {- id: "bilingual-office-kantor",- class: "bilingual",- records: [rec("Office location", "Jakarta"), rec("Release region", "Singapore")],- query: "kantor utama",- expected: "Office location",- },- {- id: "long-value-tail",- class: "long-value",- records: [rec("Release handbook", `${"Operational notes. ".repeat(80)}Before rollback always verify service health and preserve evidence.`), rec("Design handbook", "Use compact mobile navigation")],- query: "verify service health",- expected: "Release handbook",- },- {- id: "similar-keys-disambiguation",- class: "similar-keys",- records: [rec("Primary database", "PostgreSQL"), rec("Analytics database", "ClickHouse"), rec("Archive database", "SQLite")],- query: "analytics database",- expected: "Analytics database",- },- {- id: "exact-domain-phrase",- class: "lexical-control",- records: [rec("Deployment rollback policy", "Verify service health"), rec("Deployment region", "Jakarta")],- query: "deployment rollback",- expected: "Deployment rollback policy",- },-];+ const retrieval = retrievalCases.map((row) => {+ const first = lexicalTopKey(row.records, row.query), second = lexicalTopKey(row.records, row.query);+ return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };+ });+ const semantic = retrievalCases.map((row) => {+ const first = semanticTopKey(row.records, row.query), second = semanticTopKey(row.records, row.query);+ return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };+ });-const retrieval = retrievalCases.map((row) => {- const first = topKey(row.records, row.query), second = topKey(row.records, row.query);- return { id: row.id, class: row.class, pass: first === row.expected, deterministic: first === second, topKey: first };-});+ const graphCases = [+ { id: "ownership-dependency", first: "Atlas", intermediate: "Beacon", second: "Core", records: [rec("Atlas ownership", "Atlas owns Beacon"), rec("Beacon dependency", "Beacon depends on Core"), rec("Unrelated preference", "Compact UI")] },+ { id: "service-team", first: "Orchid", intermediate: "Platform", second: "Infra", records: [rec("Orchid owner", "Orchid is owned by Platform"), rec("Platform dependency", "Platform depends on Infra"), rec("Theme", "Dark") ] },+ { id: "module-package", first: "Renderer", intermediate: "Scene", second: "Assets", records: [rec("Renderer relation", "Renderer uses Scene"), rec("Scene relation", "Scene consumes Assets"), rec("Editor", "VS Code") ] },+ ];-const graphCases = [- { id: "ownership-dependency", first: "Atlas", intermediate: "Beacon", second: "Core", records: [rec("Atlas ownership", "Atlas owns Beacon"), rec("Beacon dependency", "Beacon depends on Core"), rec("Unrelated preference", "Compact UI")] },- { id: "service-team", first: "Orchid", intermediate: "Platform", second: "Infra", records: [rec("Orchid owner", "Orchid is owned by Platform"), rec("Platform dependency", "Platform depends on Infra"), rec("Theme", "Dark") ] },- { id: "module-package", first: "Renderer", intermediate: "Scene", second: "Assets", records: [rec("Renderer relation", "Renderer uses Scene"), rec("Scene relation", "Scene consumes Assets"), rec("Editor", "VS Code") ] },-];+ const graph = graphCases.map((row) => {+ const firstHop = queryMemoryLedger(ledger(row.records), { query: row.first, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);+ const secondHop = queryMemoryLedger(ledger(row.records), { query: row.intermediate, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);+ const pass = firstHop.some((text) => text.includes(row.intermediate)) && secondHop.some((text) => text.includes(row.second));+ return { id: row.id, pass, hops: 2, firstHopRows: firstHop.length, secondHopRows: secondHop.length };+ });-const graph = graphCases.map((row) => {- const firstHop = queryMemoryLedger(ledger(row.records), { query: row.first, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);- const secondHop = queryMemoryLedger(ledger(row.records), { query: row.intermediate, at: AT }).records.map((item) => `${item.record.key} ${item.record.value}`);- const pass = firstHop.some((text) => text.includes(row.intermediate)) && secondHop.some((text) => text.includes(row.second));- return { id: row.id, pass, hops: 2, firstHopRows: firstHop.length, secondHopRows: secondHop.length };-});+ const retrievalPassed = retrieval.filter((row) => row.pass && row.deterministic).length;+ const semanticPassed = semantic.filter((row) => row.pass && row.deterministic).length;+ const lexicalMisses = retrieval.filter((row) => !row.pass).map((row) => row.id);+ const semanticMisses = semantic.filter((row) => !row.pass).map((row) => row.id);+ const graphPassed = graph.filter((row) => row.pass).length;+ const candidateImprovesRecall = semanticPassed > retrievalPassed;-const retrievalPassed = retrieval.filter((row) => row.pass && row.deterministic).length;-const semanticMisses = retrieval.filter((row) => !row.pass).map((row) => row.id);-const graphPassed = graph.filter((row) => row.pass).length;-const result = {- calibrationVersion: "mso-memory-retrieval-calibration-v1",- retrieval: { passed: retrievalPassed, total: retrieval.length, accuracyPct: Math.round(retrievalPassed / retrieval.length * 1000) / 10, deterministic: retrieval.every((row) => row.deterministic), rows: retrieval },- semanticEvidenceGate: {- objectiveMisses: semanticMisses,- vectorLayerRequired: false,- reason: semanticMisses.length- ? "The fixture exposes semantic gaps, but no embedding approach has yet proven a measurable benefit with acceptable privacy/token/latency cost. Keep vector infrastructure blocked until that comparison exists."- : "The deterministic lexical fixture is adequate; no vector layer is justified.",- },- graph: { passed: graphPassed, total: graph.length, rows: graph, graphStorageRequired: graphPassed !== graph.length },- note: "This is a bounded synthetic evidence gate. Graph rows prove whether small relationship chains remain reachable through explicit two-step retrieval; they do not claim general graph reasoning reliability.",-};+ return {+ calibrationVersion: "mso-memory-retrieval-calibration-v2",+ retrieval: { passed: retrievalPassed, total: retrieval.length, accuracyPct: pct(retrievalPassed, retrieval.length), deterministic: retrieval.every((row) => row.deterministic), rows: retrieval },+ semanticCandidate: {+ encoder: SKILL_EMBEDDING_VERSION,+ mode: "existing-local-ephemeral-hybrid",+ networkCalls: 0,+ persistedMemoryVectors: 0,+ passed: semanticPassed,+ total: semantic.length,+ accuracyPct: pct(semanticPassed, semantic.length),+ deterministic: semantic.every((row) => row.deterministic),+ deltaPassedVsLexical: semanticPassed - retrievalPassed,+ rows: semantic,+ },+ semanticEvidenceGate: {+ objectiveLexicalMisses: lexicalMisses,+ candidateMisses: semanticMisses,+ candidateImprovesRecall,+ vectorLayerRequired: false,+ reason: candidateImprovesRecall+ ? "The existing local semantic encoder improves this bounded fixture, but production memory vectors remain blocked until privacy, latency and index lifecycle costs are calibrated."+ : "The existing local semantic encoder does not improve this bounded memory fixture, so reusing it for production memory retrieval would add vector/index complexity without a recall benefit.",+ },+ graph: { passed: graphPassed, total: graph.length, rows: graph, graphStorageRequired: graphPassed !== graph.length },+ note: "This is a bounded synthetic evidence gate. The semantic candidate is the repo's existing local skill-routing encoder and is evaluated ephemerally with no network call or persisted memory vector. Graph rows prove only whether these small relationship chains remain reachable through explicit two-step retrieval; neither result is a universal retrieval/reasoning claim.",+ };+}-if (process.argv.includes("--json")) console.log(JSON.stringify(result, null, 2));-else {- console.log(`MSO memory retrieval calibration · lexical ${result.retrieval.passed}/${result.retrieval.total} · deterministic=${result.retrieval.deterministic}`);- for (const row of result.retrieval.rows) console.log(` ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);- console.log(` semantic gate: ${result.semanticEvidenceGate.vectorLayerRequired ? "vector candidate" : "vector blocked"}`);- console.log(` graph gate: ${result.graph.passed}/${result.graph.total} two-hop fixtures · graph storage ${result.graph.graphStorageRequired ? "candidate" : "blocked"}`);+if (import.meta.main) {+ const result = runMemoryRetrievalCalibration();+ if (process.argv.includes("--json")) console.log(JSON.stringify(result, null, 2));+ else {+ console.log(`MSO memory retrieval calibration · lexical ${result.retrieval.passed}/${result.retrieval.total} · local semantic ${result.semanticCandidate.passed}/${result.semanticCandidate.total}`);+ for (const row of result.retrieval.rows) console.log(` lexical ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);+ for (const row of result.semanticCandidate.rows) console.log(` semantic ${row.pass && row.deterministic ? "PASS" : "MISS"} ${row.id}`);+ console.log(` semantic gate: ${result.semanticEvidenceGate.vectorLayerRequired ? "vector candidate" : "vector blocked"} · delta=${result.semanticCandidate.deltaPassedVsLexical}`);+ console.log(` graph gate: ${result.graph.passed}/${result.graph.total} two-hop fixtures · graph storage ${result.graph.graphStorageRequired ? "candidate" : "blocked"}`);+ }+ if (!result.retrieval.deterministic || !result.semanticCandidate.deterministic || result.graph.passed !== result.graph.total) process.exitCode = 1;}-if (!result.retrieval.deterministic || result.graph.passed !== result.graph.total) process.exitCode = 1;diff --git a/scripts/bench-memory-retrieval-calibration.test.ts b/scripts/bench-memory-retrieval-calibration.test.tsnew file mode 100644index 00000000..95758933--- /dev/null+++ b/scripts/bench-memory-retrieval-calibration.test.ts@@ -0,0 +1,21 @@+import { describe, expect, it } from "vitest";+import { runMemoryRetrievalCalibration } from "./bench-memory-retrieval-calibration.mjs";++describe("P10D memory retrieval evidence gate", () => {+ it("compares the existing local semantic encoder against lexical retrieval without adding a memory vector index", () => {+ const result = runMemoryRetrievalCalibration();+ expect(result.retrieval.passed).toBe(4);+ expect(result.retrieval.total).toBe(6);+ expect(result.semanticCandidate.encoder).toBe("mso-local-hybrid-v1");+ expect(result.semanticCandidate.passed).toBe(3);+ expect(result.semanticCandidate.total).toBe(6);+ expect(result.semanticCandidate.deltaPassedVsLexical).toBe(-1);+ expect(result.semanticCandidate.networkCalls).toBe(0);+ expect(result.semanticCandidate.persistedMemoryVectors).toBe(0);+ expect(result.semanticEvidenceGate.candidateImprovesRecall).toBe(false);+ expect(result.semanticEvidenceGate.vectorLayerRequired).toBe(false);+ expect(result.graph.passed).toBe(3);+ expect(result.graph.total).toBe(3);+ expect(result.graph.graphStorageRequired).toBe(false);+ });+});