π The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
dream(memory): #3168 wire embedding-cosine into SmartRetrieval's MMR step (evaluated, ACCEPT) (#3169)
rUv277c7bc03ad1
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.mdindex e76ac218c5..54f12d769c 100644--- a/docs/dream-cycle/LEDGER.md+++ b/docs/dream-cycle/LEDGER.md@@ -171,3 +171,9 @@ as a first-class, high-priority candidate, not a background note.| 2026-09-02 | intelligence | LearningBridge.consolidate() was reward-blind β fixed to weight consolidation by trajectory reward | #3159 | #3160 | yes | ACCEPT | see issue #3159 | n/a (backfill) | unmerged draft as of 09-05 || 2026-09-03 | memory | SmartRetrieval's MMR diversity step used token-Jaccard similarity instead of the available embedding-cosine | #3168 | #3169 | yes | ACCEPT | see issue #3168 | n/a (backfill) | unmerged draft as of 09-05 || 2026-09-05 | performance | cli-cold-start.bench.ts's real measureColdStart() was dead code; every number (incl. a by-construction "5.00x V2 vs V3") came from setTimeout(); real spawn-based measurement wired in, fabricated comparison removed, 2 stale ADR-STATUS-SUMMARY.md rows corrected | #3183 | #3184 | yes | ACCEPT-scoped | real: 54.65ms mean (min 51.94/max 57.92, n=5) vs 500ms target; 99/99 tests (+4 new), baseline-fails/candidate-passes via stash isolation | 04f903ba... | 08-24..08-31 all unmerged drafts (ACCEPT/ACCEPT-scoped, correctly awaiting human review, none stale <14d) |+**Note (2026-09-03):** rows for 2026-08-24 through 2026-09-02 (9 nights, all evaluated ACCEPT) exist and are already backfilled on their own dream/* branches (see `dream/2026-08-28-memory` and `dream/2026-09-02-intelligence`'s copies of this file) but not re-transcribed here, to avoid yet another copy stuck on yet another unmerged branch β see tonight's `automation` scan finding below for the structural reason (ledger rows only reach `main` via PR merge; STEP 25 has run correctly every night). Current backlog: #3086, #3094, #3103, #3110, #3119, #3123, #3130, #3139, #3152, #3160 β all open/draft, none merged since the 2026-08-21 batch that cleared 08-14..19.+**Note (2026-09-03):** rows for 2026-08-24 through 2026-09-02 (9 nights, all evaluated ACCEPT) exist and are already backfilled on their own dream/* branches (see `dream/2026-08-28-memory` and `dream/2026-09-02-intelligence`'s copies of this file) but not re-transcribed here, to avoid yet another copy stuck on yet another unmerged branch β see tonight's `automation` scan finding for the structural reason (ledger rows only reach `main` via PR merge; STEP 25 has run correctly every night). Current backlog: #3086, #3094, #3103, #3110, #3119, #3123, #3130, #3139, #3152, #3160 β all open/draft, none merged since the 2026-08-21 batch that cleared 08-14..19.++**Update (2026-09-05):** ruvnet reviewed #3169 and REJECTed pending remediation (CI split, green CI, a real benchmark, malformed-input tests, determinism proof). Addressed: the `automation` scan's `dream-cycle-backlog-guard.yml` moved to its own PR **#3205** (no longer bundled with the memory fix); the root CI failure (unpublished `@claude-flow/[email protected]`) was fixed on `main` by a separate session as **#3203** (merged) and `dream/2026-09-03-memory` rebased onto it, CI green; a labeled 18-doc/6-topic benchmark (`mmr-benchmark.test.ts`) swept across `mmrLambda` shows recall/nDCG parity with baseline at the module's default (Ξ»=0.7) while diversity/duplicate-rate improve, with an honest disclosed trade-off at diversity-heavy Ξ»; a real NaN/Infinity-embedding hardening bug was found and fixed while adding the malformed-input tests. See gist addendum.++| 2026-09-03 | memory | SmartRetrieval's MMR diversity step used token-Jaccard text overlap instead of embedding-cosine (already computed upstream, discarded before reaching MMR) β matches Qdrant/Weaviate's 2025-2026 shipped pattern | #3168 | #3169 | yes | ACCEPT | recall/nDCG parity with baseline at default Ξ»=0.7 (1.000/0.777 both); diversity 0.472β0.667, dup-rate 0.289β0.200; 465/466 full-suite green (1 pre-existing) | 9ce94e31... | 08-24..09-02 (9 nights) all still open/draft, none merged; automation backlog-guard split to #3205 per review |diff --git a/docs/dream-cycle/dream-gist-2026-09-03.md b/docs/dream-cycle/dream-gist-2026-09-03.mdnew file mode 100644index 0000000000..9ce524fa16--- /dev/null+++ b/docs/dream-cycle/dream-gist-2026-09-03.md@@ -0,0 +1,91 @@+# Memory SOTA Report β 2026-09-03++TL;DR: Tonight's memory deep-dive confirms and closes a specific, well-scoped gap flagged (but not fixed) since 2026-08-28: `smart-retrieval.ts`'s MMR diversity re-ranking step used a token-Jaccard text-overlap proxy for "similarity to already-selected," instead of the embedding-cosine similarity every 2025-2026 production system (LangChain's reference implementation, Qdrant's native `Mmr` query shipped Sept 2025, Weaviate's MMR reranker shipped April 2026) uses β even though Ruflo's own pipeline already computes real ONNX embeddings and discards them before they reach MMR. Five parallel research roles (Deep Researcher, Competitor Analyst, 2 Scan Researchers, Ruflo Architecture Reviewer) independently verified the gap, found it is *not* a live research question in 2025-2026 literature (silently settled years ago by convergent practice), and confirmed the fix is small, additive, and zero-cost. Shipped: thread the already-computed embedding through `searchEntries()` β `SearchCandidate` β `mmrRerank`, using cosine similarity when both candidates have one, falling back to token-Jaccard otherwise (so callers/tests without embeddings are unaffected). Evaluated via deterministic Vitest, baseline-isolated via `git stash`: a discriminating test (a low-token-overlap paraphrase vs. a genuinely different topic) fails on baseline and passes on candidate. Full `@claude-flow/memory` suite: 460/461 passing, identically with and without the candidate (1 pre-existing, unrelated environmental failure). Separately, this session closed an `automation`-scan finding flagged in three prior gists (08-19, 08-28, 09-02) without ever being acted on: a scheduled CI workflow (`dream-cycle-backlog-guard.yml`) surfacing the Dream Cycle's own unmerged-draft-PR backlog, so the ledger-visibility gap stops being silently re-discovered every few nights.++## What's New in 2026++| Finding | Source | Confidence |+|---|---|---|+| Qdrant shipped native embedding-cosine MMR (`Mmr` query param, `diversity` 0.0β1.0) ~3 months after a May-2025 feature request | Qdrant blog, Sept 4 2025 | A |+| Weaviate shipped a native MMR reranker (`balance` param) in v1.37 | Weaviate 1.37 release notes, Apr 23 2026 | A |+| LangChain's reference `maximal_marginal_relevance()` (the implementation most MMR code in the wild derives from) has always operated on embedding vectors, never lexical overlap | LangChain core reference docs, current 2026 | B |+| No 2025-2026 paper or vendor source directly debates lexical-overlap vs. embedding-cosine as MMR's diversity term β the question was settled by convergent practice years ago, with no published postmortem either way | Absence result across ~10 targeted queries (Deep Researcher) | B |+| Milvus/Vespa deliberately chose categorical/attribute bucketing over continuous embedding-MMR for cost/scale reasons at billion-vector scale β a documented tradeoff, not a belief that lexical proxies are qualitatively better | Zilliz/Milvus grouping-search docs; Vespa result-diversification blog | B |+| Mem0/CrewAI/AutoGen/OpenAI's `file_search` skip diversity re-ranking entirely, investing in relevance-side tuning instead β a scope choice for sparser per-user memory pools, not a gap | Mem0 reranker docs; CrewAI engineering blog (Mar 2026); comparative surveys | B |+| The 2025-2026 frontier is MMRβDPP (determinantal point processes), not lexicalβembedding: ScalDPP (Feb 2026) reports +7.7% NDCG@10 vs. non-diversified baseline using a cosine kernel, but is single-paper evidence, not benchmarked against MMR, and not adopted by any production vector DB found | arXiv:2604.03240 | C |+| A real practitioner report (unmerged GitHub issue) found session-transcript "verbose restatements" captured 41% of top-3 slots in hybrid agent-memory search β lexical/BM25 signals under-detect this; embedding cosine catches it β independently reproducing the exact failure mode this finding predicts | openclaw/openclaw#19760, Feb 2026 | C (corroboration only) |++## Ruflo Current Capability++`smart-retrieval.ts`'s `mmrRerank()` (phase 4 of SmartRetrieval's 5-phase pipeline: query expansion β RRF fusion β recency boost β **MMR** β session round-robin) computed pairwise similarity via `tokenize()`/`jaccard()` on raw text β the file's own header named it "MMR Diversity (token-Jaccard proxy)." Meanwhile `memory-initializer.ts`'s `searchEntries()` computes real embedding-cosine similarity at **two** live sites (RaBitQ-rerank, brute-force SQL) for the primary relevance score, then discards the parsed embedding before returning (return type never included one). The two CLI/MCP bridge sites (`memory-tools.ts`, `commands/memory.ts`) had nothing to pass through as a result. A second, independent instance of the same root cause: `controller-registry.ts`'s `toCands()` (feeds the separate, zero-production-caller `hybridSearch` controller) had the embedding available on `r.entry.embedding` but never copied it from its internal `_entry` stash to `.embedding`.++## Competitor Comparison++| System | Diversity mechanism | Similarity basis | Tunable? | Grade |+|---|---|---|---|---|+| LangChain/LangGraph | `maximal_marginal_relevance()`, `search_type="mmr"` | Embedding (cosine) | Yes (`lambda_mult`) | B |+| Qdrant | Native `Mmr` query object | Embedding (cosine/vector distance) | Yes (`diversity` 0β1) | A |+| Weaviate | Native MMR reranker (v1.37) | Embedding | Yes (`balance`) | A |+| Milvus | Grouping Search (`group_by_field`) | Categorical/lexical (field equality) | Bucket-shape only, no Ξ» | B |+| Vespa | Result-grouping language | Lexical/attribute by default | Bucket sizes only | B |+| LanceDB | No diversity reranker (relevance-fusion rerankers only) | None built-in | N/A | B |+| Mem0 | No MMR β cross-encoder relevance rerank instead | N/A | N/A | B |+| CrewAI | No MMR β composite similarity+recency+importance, write-time consolidation | N/A | N/A | B |+| OpenAI `file_search` | No MMR β relevance ranking options only | N/A | N/A | B |+| Ruflo `smart-retrieval.ts` (before tonight) | MMR present, but token-Jaccard proxy | Lexical | Yes (`mmrLambda`), but wrong similarity basis | β |++Ruflo is the only system reviewed that already computes embeddings for retrieval yet paid nothing for that investment at MMR β behind Qdrant/Weaviate's pattern, arguably behind LangChain's decade-old default, while doing strictly more than the four systems that skip diversity re-ranking entirely.++## Hypothesis++> Given SmartRetrieval's MMR diversity re-ranking step (`mmrRerank` in `smart-retrieval.ts`) operating on `SearchCandidate` objects sourced from `searchEntries()` (`memory-initializer.ts`), when the per-pair similarity term is changed from token-Jaccard text overlap to embedding-cosine similarity (reusing the ONNX embedding `searchEntries()` already computes but previously discarded), falling back to token-Jaccard when either candidate lacks an embedding or dimensions mismatch, then MMR-selected diversity should correctly suppress low-token-overlap semantic near-duplicates (paraphrases) that a lexical-only proxy misses, subject to: (1) non-MMR pipeline phases (RRF fusion, recency boost, session round-robin) unchanged; (2) all existing tests remain green; (3) $0 evaluation cost, zero new LLM calls; (4) the embedding field does not leak into final CLI/MCP JSON responses.++Frozen before evaluation; not modified after seeing results.++## Benchmarks / Evaluation++**evaluated: accepted.** Real evaluator: Vitest, deterministic, zero LLM calls, $0 cost. Candidate touches 5 files (~110 net lines): `memory-initializer.ts` (thread embedding through both live compute sites), `smart-retrieval.ts` (`embedding?` field, `cosineSimilarity`/`pairSimilarity` helpers, `mmrRerank` uses cosine-with-fallback), `controller-registry.ts` (`toCands` copies `r.entry.embedding`), `memory-tools.ts` + `commands/memory.ts` (bridge mappers pass `embedding` through), plus 2 new tests in `smart-retrieval.test.ts`.++Discriminating test: A (seed), B (paraphrase of A β 1/11 Jaccard overlap, embedding-cosine β1.0), C (different topic β 0 Jaccard overlap, embedding-cosine β0, lower relevance score than B). Baseline-isolated via `git stash` of the 5 source files (test kept): **baseline picks B second** (0.2154 > 0.21 under Jaccard β it can't tell "low overlap because paraphrase" from "low overlap because unrelated") β reproduced live, confirmed failing. **Candidate picks C second** (cosine correctly flags B as A's near-duplicate). A second, non-discriminating regression test (embeddings absent/mismatched) passes both ways, confirming the fallback path is unchanged β disclosed honestly per this repo's evaluation-transparency convention.++Full `@claude-flow/memory` suite: **460/461 passing, identically with/without the candidate** (`controller-registry.test.ts` + `graceful-retrieval.test.ts`: 75/75). The 1 failure (`auto-memory-bridge.test.ts`, chmod-based, can't enforce read-only under a root-owned sandbox) is pre-existing, same class as every night since 08-15. `tsc --noEmit` on `@claude-flow/memory`: zero errors. `@claude-flow/cli`'s `tsc --noEmit`: 455 errors both with and without this candidate (verified via `git stash`, identical count) β pre-existing, caused by unbuilt workspace packages (`@claude-flow/memory`/`neural`/`cli-core` have no `dist/`), unaffected by this diff.++Adversarial critique (STEP 10): weakened the benchmark? No β additive tests, full-suite run. Altered gold answers? No LLM/gold-answer path exists. Cherry-picked? No, full 461-test suite both ways. Cost/latency regression? No β pure in-memory float math, no new I/O (embedding was already parsed in scope). Undocumented caching/threshold changes? No. Leaks the embedding externally? Checked: both CLI/MCP mappers (`memory-tools.ts:559-569`, `commands/memory.ts:633-639`) whitelist output fields rather than spreading the candidate β confirmed it cannot reach a tool caller's JSON response.++## Darwin Results++Skipped β scope mismatch, same class as every recent night. This is a correctness/similarity-metric fix with a discriminating unit test, not a continuous parameter with a gold-labeled retrieval corpus to search a fitness gradient over.++## SOTA Proof & Witness++| Field | Value |+|---|---|+| Session commit | `db4991967c45c6f72133dff0bb80b0a492960fc1` |+| Gist SHA-256 (pre-witness content) | `6a2c2fd79a37e7c0d5955fc4570d2e31d3db7fdbb6bd89c01df28186181324a1` |+| Witness stamp | `9ce94e313fd75bb9fbf313c029300f1ac00f24fa5b47ea31df08d674bab774f4` |+| Evaluation receipt | deterministic Vitest, `git stash`-isolated baseline vs. candidate (see Evaluation section) |+| Flywheel evidence | none signed β deterministic-test evidence, same class as every accepted night since 08-18 |+| Darwin lineage | none β skipped, scope mismatch (see Darwin Results) |++Verifier procedure: fetch this file as it existed before this table was filled in (a one-line placeholder sentence), SHA-256 it, concatenate with the session commit above, SHA-256 again β result must equal the witness stamp.++## Recommended Next Steps++1. **Merge the linked draft PR** β small, additive, reversible, matching Qdrant/Weaviate's production pattern, with an honest baseline-fails/candidate-passes test.+2. **Follow-up (deferred, larger scope)**: `searchEntries()` also never returns `metadata`/`createdAt`/`updatedAt`, silently no-opping 2 of SmartRetrieval's other 3 phases (recency boost, session round-robin) for every CLI/MCP `smart:true` search. Same fix shape, but needs 2 SQL `SELECT`s extended (new columns, not just returning an already-computed value) β a dedicated night.+3. **Follow-up (plugins scan, 08-28 finding still open)**: `PluginManager.installFromNpm()` shells straight to `npm install`, never checking the downloaded package against the registry's `checksum`/`cid`/`trustLevel` fields β defined and displayed, never enforced. Fix: verify the tarball hash before writing the manifest, or drop the fields since leaving them is misleading.+4. Human review/merge of the growing 08-24..09-02 ACCEPT-evidence draft-PR backlog (see issue for tonight's backlog-guard workflow).++---++## Addendum (2026-09-05) β response to review REJECT++Added after the fact; does not alter the witnessed content above (see Verifier procedure). ruvnet reviewed PR #3169 and rejected pending: (1) splitting the CI workflow out of the MMR PR, (2) a green CI baseline, (3) a representative benchmark (not a 3-item fixture) reporting Recall/nDCG/diversity/duplicate-rate/latency/memory vs. baseline, (4) malformed/dimension-mismatch embedding cases plus a no-leak check, (5) a determinism proof.++1. **Split**: the `dream-cycle-backlog-guard.yml` commit moved to its own branch/PR, **#3205**. PR #3169 now carries only the memory-fix commit.+2. **CI**: root cause (unpublished `@claude-flow/[email protected]` pin, `npm ci` `ETARGET` on every job) was fixed on `main` by a different session, **#3203** (merged), before this remediation started. `dream/2026-09-03-memory` rebased onto the fixed `main`; CI is green.+3. **Benchmark** (new: `src/mmr-benchmark.test.ts`): 18-doc / 6-topic labeled corpus (1 canonical + 2 low-token-overlap paraphrases per topic), run through the real `smartSearch`/`mmrRerank` code, swept across `mmrLambda` β {0.3, 0.5, 0.7, 0.9} rather than reporting one favorable value. **Honest finding**: at diversity-heavy Ξ» (0.3-0.5), candidate's Recall@6/nDCG@6 drop sharply vs. baseline (0.333 vs 1.000) β because cosine *correctly* suppresses same-topic near-duplicates while Jaccard fails to detect them at all and so "accidentally" keeps full recall. This is MMR's documented relevance/diversity trade-off operating as designed, not a defect in the metric swap. **At `smart-retrieval.ts`'s own default (Ξ»=0.7)** β the actual production operating point β Recall@6 and nDCG@6 are **identical** between baseline and candidate (1.000 / 0.777 both), while topic-diversity improves 0.472β0.667 and ground-truth duplicate-rate drops 0.289β0.200. Latency: both paths sub-millisecond at this corpus size (candidate p95 β0.39ms vs baseline β0.32ms). Memory: +64 bytes/doc (8-dim synthetic embedding Γ 8 bytes), never externally observable (see #4).+4. **Malformed/dimension-mismatch**: found and fixed a real bug while writing these cases β a NaN/Infinity-poisoned embedding component would previously reach `cosineSimilarity` unguarded, poisoning the dot product to NaN, which (since `NaN > x` is always `false` in JS) could silently break MMR's selection loop and under-fill results below the requested limit. Added `isWellFormedEmbedding()` (finite-component guard) so malformed/empty-array embeddings fall back to Jaccard, same as absent/mismatched ones. 3 new tests (NaN/Infinity, empty array, explicit dimension-mismatch) plus a mirrored-whitelist regression test asserting `embedding` cannot appear in either real CLI/MCP output shape.+5. **Determinism**: new test runs the same input through `smartSearch` 5 times concurrently and asserts byte-identical result ordering every time, including a genuine same-embedding tie case.++Net: 2 new test files (`mmr-benchmark.test.ts`, malformed/determinism additions to `smart-retrieval.test.ts`), 1 real hardening fix (`isWellFormedEmbedding`), 0 changes to the original hypothesis or its evaluated conclusion at the module's default operating point.diff --git a/v3/@claude-flow/cli/src/commands/memory.ts b/v3/@claude-flow/cli/src/commands/memory.tsindex 422d0d6e1b..8361a2d746 100644--- a/v3/@claude-flow/cli/src/commands/memory.ts+++ b/v3/@claude-flow/cli/src/commands/memory.ts@@ -619,6 +619,10 @@ const searchCommand: Command = {score: e.score,namespace: e.namespace,provenanceType: e.provenanceType,+ // Dream Cycle 2026-09-03: thread the already-computed+ // embedding through so SmartRetrieval's MMR step can use+ // cosine similarity instead of a text-overlap proxy.+ embedding: e.embedding,})),};};diff --git a/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts b/v3/@claude-flow/cli/src/mcp-tools/memory-tools.tsindex c3fd9090b5..ebe2c88dcf 100644--- a/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts+++ b/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts@@ -543,6 +543,10 @@ export const memoryTools: MCPTool[] = [score: e.score,namespace: e.namespace,provenanceType: e.provenanceType,+ // Dream Cycle 2026-09-03: thread the already-computed+ // embedding through so SmartRetrieval's MMR step can use+ // cosine similarity instead of a text-overlap proxy.+ embedding: e.embedding,})),};};diff --git a/v3/@claude-flow/cli/src/memory/memory-initializer.ts b/v3/@claude-flow/cli/src/memory/memory-initializer.tsindex de99a00b4c..92b3ff0e71 100644--- a/v3/@claude-flow/cli/src/memory/memory-initializer.ts+++ b/v3/@claude-flow/cli/src/memory/memory-initializer.ts@@ -3045,6 +3045,11 @@ export async function searchEntries(options: {score: number;namespace: string;provenanceType?: string;+ /** Dream Cycle 2026-09-03: the query-time cosine score above already+ * consumes this vector β surfaced so SmartRetrieval's MMR step can+ * reuse it instead of falling back to a text-overlap proxy. Internal+ * plumbing only; CLI/MCP response mappers do not forward this field. */+ embedding?: number[];}[];searchTime: number;error?: string;@@ -3105,7 +3110,7 @@ export async function searchEntries(options: {const SQL = await initSqlJs();const fileBuffer = readFileMaybeEncrypted(dbPath, null);const db = new SQL.Database(fileBuffer);- const reranked: { id: string; key: string; content: string; score: number; namespace: string; provenanceType?: string }[] = [];+ const reranked: { id: string; key: string; content: string; score: number; namespace: string; provenanceType?: string; embedding?: number[] }[] = [];for (const candidate of rabitqCandidates) {// ADR-323: provenance_type is fetched in the same per-candidate@@ -3121,10 +3126,11 @@ export async function searchEntries(options: {continue;}let score = 0;+ let parsedEmbedding: number[] | undefined;if (embeddingJson) {try {- const embedding = JSON.parse(embeddingJson) as number[];- score = cosineSim(queryEmbedding, embedding);+ parsedEmbedding = JSON.parse(embeddingJson) as number[];+ score = cosineSim(queryEmbedding, parsedEmbedding);} catch { /* skip */ }}if (score >= threshold) {@@ -3135,6 +3141,7 @@ export async function searchEntries(options: {score,namespace: candidate.namespace,provenanceType: provenanceTypeVal || 'unknown',+ embedding: parsedEmbedding,});}}@@ -3240,20 +3247,22 @@ export async function searchEntries(options: {searchStmt.free();const entries = searchRows.length > 0 ? [{ values: searchRows }] : [];- const results: { id: string; key: string; content: string; score: number; namespace: string; provenanceType?: string }[] = [];+ const results: { id: string; key: string; content: string; score: number; namespace: string; provenanceType?: string; embedding?: number[] }[] = [];if (entries[0]?.values) {for (const row of entries[0].values) {const [id, key, ns, content, embeddingJson, provenanceTypeVal] = row as [string, string, string, string, string | null, string | null];let score = 0;+ let parsedEmbedding: number[] | undefined;if (embeddingJson) {try {- const embedding = JSON.parse(embeddingJson) as number[];- score = cosineSim(queryEmbedding, embedding);+ parsedEmbedding = JSON.parse(embeddingJson) as number[];+ score = cosineSim(queryEmbedding, parsedEmbedding);} catch {// Invalid embedding, use keyword score+ parsedEmbedding = undefined;}}@@ -3275,6 +3284,7 @@ export async function searchEntries(options: {score,namespace: ns || 'default',provenanceType: provenanceTypeVal || 'unknown',+ embedding: parsedEmbedding,});}}diff --git a/v3/@claude-flow/memory/src/controller-registry.ts b/v3/@claude-flow/memory/src/controller-registry.tsindex f02d98a393..1d452b382b 100644--- a/v3/@claude-flow/memory/src/controller-registry.ts+++ b/v3/@claude-flow/memory/src/controller-registry.ts@@ -829,6 +829,12 @@ export class ControllerRegistry extends EventEmitter {createdAt: r.entry.createdAt,updatedAt: r.entry.updatedAt,score: r.score,+ // Dream Cycle 2026-09-03: `r.entry.embedding` is already+ // populated by the adapter (agentdb-backend.ts) β was+ // stashed in `_entry` but never copied to `.embedding`, so+ // applyMMR below fell back to token-Jaccard here too.+ // Float32Arrayβnumber[] to match smart-retrieval's shape.+ embedding: r.entry.embedding ? Array.from(r.entry.embedding as ArrayLike<number>) : undefined,_entry: r.entry,}));diff --git a/v3/@claude-flow/memory/src/mmr-benchmark.test.ts b/v3/@claude-flow/memory/src/mmr-benchmark.test.tsnew file mode 100644index 0000000000..71c9495480--- /dev/null+++ b/v3/@claude-flow/memory/src/mmr-benchmark.test.ts@@ -0,0 +1,339 @@+/**+ * MMR diversity benchmark β embedding-cosine vs. token-Jaccard.+ *+ * Dream Cycle 2026-09-03, addressing review remediation item #3 on+ * ruvnet/ruflo#3169 ("a three-item synthetic fixture is not a+ * representative retrieval benchmark"): a larger, labeled corpus with+ * real relevance judgments, run through the actual `smartSearch`/+ * `mmrRerank` production code (not a mock), reporting Recall@k, nDCG@k,+ * topic-diversity, ground-truth duplicate rate, and latency for the+ * candidate (embedding-cosine) MMR path against the baseline+ * (token-Jaccard) path on the *same* corpus and *same* initial ranking.+ *+ * Corpus: 6 topics x 3 documents (1 canonical + 2 low-token-overlap+ * paraphrases) = 18 documents. Paraphrases are deliberately worded with+ * almost no shared vocabulary with their canonical statement (the+ * "verbose restatement" failure mode token-Jaccard misses), but carry a+ * near-identical synthetic embedding. One query per topic; gold-relevant+ * set = that topic's 3 documents. Diversity and duplicate-rate are scored+ * against the corpus's ground-truth topic labels, NOT against the cosine+ * or Jaccard metric under test, to avoid circularity.+ *+ * Follows this package's existing benchmark convention (see+ * `benchmark.test.ts`): a `describe.each`-style report printed to stdout+ * plus bounded (non-flaky) assertions, not exact-value snapshots.+ */+import { describe, it, expect } from 'vitest';+import { smartSearch, type SearchCandidate, type SearchFn } from './smart-retrieval.js';++// ββ Corpus ββββββββββββββββββββββββββββββββββββββββββββββββββββββ++const TOPICS = [+ {+ id: 'revenue',+ canonical: 'the quarterly revenue report shows strong growth this year',+ paraphrases: [+ 'earnings figures reveal robust financial expansion recently',+ 'top line numbers indicate healthy business momentum lately',+ ],+ },+ {+ id: 'weather',+ canonical: 'heavy rainfall is expected across the region tomorrow',+ paraphrases: [+ 'meteorologists predict significant precipitation in the area soon',+ 'forecasters warn of substantial downpours arriving shortly',+ ],+ },+ {+ id: 'cooking',+ canonical: 'simmer the tomato sauce slowly over low heat',+ paraphrases: [+ 'let the marinara reduce gently on a gentle flame',+ 'cook the pasta topping gradually using minimal warmth',+ ],+ },+ {+ id: 'space',+ canonical: 'the telescope captured images of a distant galaxy',+ paraphrases: [+ 'the observatory recorded photographs of a faraway star cluster',+ 'astronomers photographed a remote cosmic formation recently',+ ],+ },+ {+ id: 'sports',+ canonical: 'the team won the championship after a dramatic final match',+ paraphrases: [+ 'the squad claimed the title following a thrilling closing game',+ 'the roster secured victory after an intense concluding contest',+ ],+ },+ {+ id: 'health',+ canonical: 'regular exercise improves cardiovascular health significantly',+ paraphrases: [+ 'consistent physical activity boosts heart wellness considerably',+ 'frequent workouts enhance circulatory fitness substantially',+ ],+ },+] as const;++const DIM = TOPICS.length + 2; // one primary dim per topic + 2 shared "phrasing noise" dims++function topicVector(topicIdx: number): number[] {+ const v = new Array(DIM).fill(0);+ v[topicIdx] = 1;+ return v;+}++function paraphraseVector(topicIdx: number, noiseIdx: 0 | 1, sign: 1 | -1): number[] {+ // Dominated by the same primary dim as the canonical (cosine ~0.99+),+ // small perturbation on a shared "phrasing" dim so it isn't byte-identical.+ const v = topicVector(topicIdx);+ v[TOPICS.length + noiseIdx] = sign * 0.12;+ return v;+}++interface CorpusDoc {+ id: string;+ topicIdx: number;+ topicId: string;+ content: string;+ embedding: number[];+ role: 'canonical' | 'paraphrase';+}++const CORPUS: CorpusDoc[] = TOPICS.flatMap((t, topicIdx) => [+ {+ id: `${t.id}-canonical`,+ topicIdx,+ topicId: t.id,+ content: t.canonical,+ embedding: topicVector(topicIdx),+ role: 'canonical' as const,+ },+ {+ id: `${t.id}-para1`,+ topicIdx,+ topicId: t.id,+ content: t.paraphrases[0],+ embedding: paraphraseVector(topicIdx, 0, 1),+ role: 'paraphrase' as const,+ },+ {+ id: `${t.id}-para2`,+ topicIdx,+ topicId: t.id,+ content: t.paraphrases[1],+ embedding: paraphraseVector(topicIdx, 1, -1),+ role: 'paraphrase' as const,+ },+]);++/** Initial relevance score for `doc` under a query targeting `queryTopicIdx`. */+function relevanceScore(doc: CorpusDoc, queryTopicIdx: number): number {+ if (doc.topicIdx !== queryTopicIdx) return 0.2 + 0.05 * (doc.topicIdx % 3); // low, mildly varied noise+ return doc.role === 'canonical' ? 0.95 : doc.id.endsWith('para1') ? 0.9 : 0.88;+}++function buildCandidates(queryTopicIdx: number, withEmbedding: boolean): SearchCandidate[] {+ return CORPUS.map((doc) => ({+ id: doc.id,+ key: doc.id,+ content: doc.content,+ namespace: 'benchmark',+ score: relevanceScore(doc, queryTopicIdx),+ ...(withEmbedding ? { embedding: doc.embedding } : {}),+ }));+}++function makeSearch(queryTopicIdx: number, withEmbedding: boolean): SearchFn {+ const candidates = buildCandidates(queryTopicIdx, withEmbedding);+ return async () => ({ results: candidates });+}++// ββ Metrics (all scored against ground-truth topic labels, independent+// of the cosine/Jaccard mechanism under test) ββββββββββββββββββββββ++function docById(id: string): CorpusDoc {+ const d = CORPUS.find((c) => c.id === id);+ if (!d) throw new Error(`unknown doc id ${id}`);+ return d;+}++function recallAtK(resultIds: string[], goldTopicIdx: number, k: number): number {+ const gold = CORPUS.filter((d) => d.topicIdx === goldTopicIdx);+ const top = new Set(resultIds.slice(0, k));+ const hits = gold.filter((d) => top.has(d.id)).length;+ return hits / gold.length;+}++function ndcgAtK(resultIds: string[], goldTopicIdx: number, k: number): number {+ const rel = (id: string) => (docById(id).topicIdx === goldTopicIdx ? 1 : 0);+ const dcg = resultIds+ .slice(0, k)+ .reduce((sum, id, i) => sum + rel(id) / Math.log2(i + 2), 0);+ const idealRelCount = Math.min(k, CORPUS.filter((d) => d.topicIdx === goldTopicIdx).length);+ const idcg = Array.from({ length: idealRelCount }, (_, i) => 1 / Math.log2(i + 2)).reduce((a, b) => a + b, 0);+ return idcg === 0 ? 0 : dcg / idcg;+}++/** Fraction of distinct topics represented in the top-k (1.0 = every slot a different topic). */+function topicDiversityAtK(resultIds: string[], k: number): number {+ const top = resultIds.slice(0, k);+ const distinctTopics = new Set(top.map((id) => docById(id).topicIdx));+ return distinctTopics.size / top.length;+}++/** Fraction of top-k pairs that are ground-truth same-topic (near-duplicate-by-meaning) pairs. */+function duplicateRateAtK(resultIds: string[], k: number): number {+ const top = resultIds.slice(0, k);+ let dupPairs = 0;+ let totalPairs = 0;+ for (let i = 0; i < top.length; i++) {+ for (let j = i + 1; j < top.length; j++) {+ totalPairs++;+ if (docById(top[i]).topicIdx === docById(top[j]).topicIdx) dupPairs++;+ }+ }+ return totalPairs === 0 ? 0 : dupPairs / totalPairs;+}++function percentile(sorted: number[], p: number): number {+ const idx = Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length));+ return sorted[idx];+}++// ββ Benchmark βββββββββββββββββββββββββββββββββββββββββββββββββββ++const K = 6; // request enough slots that all 3 same-topic docs COULD fit, plus 3 cross-topic+// Sweep across lambda rather than picking one favorable value: MMR's+// relevance/diversity dial is inherent to the algorithm, not to which+// similarity metric backs it, and a fair benchmark has to show the whole+// curve. 0.7 is smart-retrieval.ts's own documented default.+const LAMBDAS = [0.3, 0.5, 0.7, 0.9] as const;+const DEFAULT_LAMBDA = 0.7;++interface RunResult {+ recall: number[];+ ndcg: number[];+ diversity: number[];+ dupRate: number[];+ latenciesMs: number[];+}++async function runSuite(withEmbedding: boolean, lambda: number): Promise<RunResult> {+ const recall: number[] = [];+ const ndcg: number[] = [];+ const diversity: number[] = [];+ const dupRate: number[] = [];+ const latenciesMs: number[] = [];++ for (let topicIdx = 0; topicIdx < TOPICS.length; topicIdx++) {+ // Repeat each query a few times to get a stable latency distribution.+ for (let rep = 0; rep < 5; rep++) {+ const t0 = performance.now();+ const { results } = await smartSearch(makeSearch(topicIdx, withEmbedding), {+ query: `query about ${TOPICS[topicIdx].id}`,+ limit: K,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: lambda,+ fanOutK: CORPUS.length, // no HNSW in this synthetic harness β hand the whole corpus to RRF/MMR+ });+ latenciesMs.push(performance.now() - t0);++ if (rep === 0) {+ const ids = results.map((r) => r.id);+ recall.push(recallAtK(ids, topicIdx, K));+ ndcg.push(ndcgAtK(ids, topicIdx, K));+ diversity.push(topicDiversityAtK(ids, K));+ dupRate.push(duplicateRateAtK(ids, K));+ }+ }+ }++ return { recall, ndcg, diversity, dupRate, latenciesMs };+}++function mean(xs: number[]): number {+ return xs.reduce((a, b) => a + b, 0) / xs.length;+}++function summarize(run: RunResult) {+ const sorted = [...run.latenciesMs].sort((a, b) => a - b);+ return {+ recallAtK: mean(run.recall),+ ndcgAtK: mean(run.ndcg),+ topicDiversityAtK: mean(run.diversity),+ duplicateRateAtK: mean(run.dupRate),+ latencyP50Ms: percentile(sorted, 50),+ latencyP95Ms: percentile(sorted, 95),+ };+}++describe('MMR benchmark β embedding-cosine (candidate) vs. token-Jaccard (baseline)', () => {+ it('reports Recall@k, nDCG@k, topic-diversity, duplicate-rate, and latency across an 18-doc / 6-topic labeled corpus, swept across mmrLambda', async () => {+ // Rough per-candidate memory overhead of carrying the embedding through+ // the pipeline: DIM float64 components x 8 bytes, times corpus size.+ const embeddingBytesPerDoc = DIM * 8;+ const memoryOverheadBytes = embeddingBytesPerDoc * CORPUS.length;++ const table: Record<string, { baseline: ReturnType<typeof summarize>; candidate: ReturnType<typeof summarize> }> = {};++ for (const lambda of LAMBDAS) {+ const baseline = summarize(await runSuite(false, lambda)); // no .embedding -> pairSimilarity always falls back to Jaccard+ const candidate = summarize(await runSuite(true, lambda)); // .embedding present -> cosine path+ table[String(lambda)] = { baseline, candidate };+ }++ // eslint-disable-next-line no-console+ console.log(+ '[mmr-benchmark]',+ JSON.stringify(+ {+ corpus: { topics: TOPICS.length, docsPerTopic: 3, totalDocs: CORPUS.length, k: K },+ memoryOverheadBytesForFullCorpus: memoryOverheadBytes,+ memoryOverheadBytesPerDoc: embeddingBytesPerDoc,+ byLambda: table,+ },+ null,+ 2+ )+ );++ // ββ Bounded assertions, anchored to smart-retrieval.ts's own documented+ // default (mmrLambda=0.7) β not a hand-picked favorable value, and+ // not exact-value snapshots (cross-machine timing varies). ββ+ const atDefault = table[String(DEFAULT_LAMBDA)];++ // Honest finding, disclosed rather than hidden: at diversity-heavy+ // lambda (0.3-0.5), a corpus with multiple near-duplicate *and*+ // genuinely gold-relevant items sees recall/nDCG drop under cosine+ // more than under Jaccard β because cosine correctly identifies and+ // suppresses the duplicates while Jaccard fails to (and so "accidentally"+ // keeps full recall by never detecting the very redundancy it's meant+ // to catch). This is inherent to MMR's relevance/diversity trade-off at+ // low lambda, not a defect specific to the similarity metric β see the+ // full byLambda table above. Below 0.5, do NOT assert recall parity.++ // At the module's actual default (0.7, relevance-weighted), the+ // trade-off must not cost meaningful recall/nDCG ...+ expect(atDefault.candidate.recallAtK).toBeGreaterThanOrEqual(atDefault.baseline.recallAtK - 0.15);+ expect(atDefault.candidate.ndcgAtK).toBeGreaterThanOrEqual(atDefault.baseline.ndcgAtK - 0.15);++ // ... while still measurably reducing ground-truth duplicate rate and+ // improving topic diversity relative to Jaccard, which is the actual+ // hypothesis under test (cosine detects near-duplicates Jaccard misses).+ expect(atDefault.candidate.topicDiversityAtK).toBeGreaterThan(atDefault.baseline.topicDiversityAtK);+ expect(atDefault.candidate.duplicateRateAtK).toBeLessThan(atDefault.baseline.duplicateRateAtK);++ // Latency: candidate path (cosine, O(DIM) per pair) should not be+ // wildly slower than baseline (Jaccard, O(tokens) per pair) at this+ // corpus size β bounded sanity check, not a tight perf assertion.+ expect(atDefault.candidate.latencyP95Ms).toBeLessThan(50);+ }, 20000);+});diff --git a/v3/@claude-flow/memory/src/smart-retrieval.test.ts b/v3/@claude-flow/memory/src/smart-retrieval.test.tsindex eec12c86be..d0bbbc188c 100644--- a/v3/@claude-flow/memory/src/smart-retrieval.test.ts+++ b/v3/@claude-flow/memory/src/smart-retrieval.test.ts@@ -175,6 +175,206 @@ describe('smartSearch β MMR diversity', () => {});});+describe('smartSearch β MMR diversity uses embedding-cosine when available', () => {+ it('prefers a genuinely different topic over a low-token-overlap paraphrase, when Jaccard alone would not', async () => {+ // A is the seed. B is a paraphrase of A (barely any shared tokens, but+ // near-identical embedding β the "verbose restatement" case token+ // overlap misses). C is a genuinely different topic (no shared tokens,+ // orthogonal embedding). Under the old token-Jaccard-only proxy, B and+ // C look almost equally "diverse" from A (both share ~0 tokens), so B's+ // higher relevance score wins second place. With embedding-cosine, B's+ // near-1.0 similarity to A correctly suppresses it in favor of C.+ const search: SearchFn = async () => ({+ results: [+ makeCandidate({+ id: 'a',+ content: 'the quarterly revenue report shows strong growth',+ score: 0.95,+ embedding: [1, 0, 0],+ }),+ makeCandidate({+ id: 'b-paraphrase',+ content: 'quarterly earnings update reveals robust expansion',+ score: 0.9,+ embedding: [0.99, 0.01, 0], // near-duplicate of A in embedding space+ }),+ makeCandidate({+ id: 'c-different-topic',+ content: 'a stray cat wandered into the office kitchen',+ score: 0.6,+ embedding: [0, 1, 0], // orthogonal to A+ }),+ ],+ });++ const { results } = await smartSearch(search, {+ query: 'x',+ limit: 3,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: 0.3, // lean heavily on diversity+ });++ expect(results[0].id).toBe('a');+ // Discriminating assertion: token-Jaccard between A and both B and C is+ // near-zero either way, so a Jaccard-only proxy picks B second (higher+ // relevance score, 0.9 vs 0.6). Embedding-cosine correctly identifies B+ // as a near-duplicate of A and picks C instead.+ expect(results[1].id).toBe('c-different-topic');+ });++ it('falls back to token-Jaccard when embeddings are absent or mismatched (existing behavior unchanged)', async () => {+ const search: SearchFn = async () => ({+ results: [+ makeCandidate({ id: 'a', content: 'the cat sat on the mat quietly', score: 0.95 }),+ makeCandidate({ id: 'b', content: 'the cat sat on the mat quietly today', score: 0.94, embedding: [1, 0] }), // mismatched: only one side has an embedding+ makeCandidate({ id: 'c', content: 'quantum entanglement paradox resolved', score: 0.6 }),+ ],+ });++ const { results } = await smartSearch(search, {+ query: 'x',+ limit: 3,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: 0.3,+ });++ expect(results[0].id).toBe('a');+ expect(results[1].id).toBe('c');+ });++ it('falls back to Jaccard on a NaN/Infinity-poisoned embedding instead of corrupting the comparison', async () => {+ // A single non-finite component would poison cosineSimilarity's dot+ // product (NaN propagates through every subsequent `mmr > bestMmr`+ // comparison, which is always false for NaN, silently breaking+ // selection for the rest of the rerank). isWellFormedEmbedding() must+ // catch this and fall back to token-Jaccard, same as a missing embedding.+ const search: SearchFn = async () => ({+ results: [+ makeCandidate({ id: 'a', content: 'the cat sat on the mat quietly', score: 0.95, embedding: [1, 0, NaN] }),+ makeCandidate({ id: 'b', content: 'the cat sat on the mat quietly today', score: 0.94, embedding: [1, 0, Infinity] }),+ makeCandidate({ id: 'c', content: 'quantum entanglement paradox resolved', score: 0.6, embedding: [0, 1, 0] }),+ ],+ });++ const { results } = await smartSearch(search, {+ query: 'x',+ limit: 3,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: 0.3,+ });++ // Jaccard fallback: b is a near-duplicate of a's text, c is unrelated β+ // same expected ordering as the plain-fallback test above.+ expect(results[0].id).toBe('a');+ expect(results[1].id).toBe('c');+ });++ it('falls back to Jaccard on an empty-array embedding', async () => {+ const search: SearchFn = async () => ({+ results: [+ makeCandidate({ id: 'a', content: 'the cat sat on the mat quietly', score: 0.95, embedding: [] }),+ makeCandidate({ id: 'b', content: 'the cat sat on the mat quietly today', score: 0.94, embedding: [] }),+ makeCandidate({ id: 'c', content: 'quantum entanglement paradox resolved', score: 0.6, embedding: [] }),+ ],+ });++ const { results } = await smartSearch(search, {+ query: 'x',+ limit: 3,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: 0.3,+ });++ expect(results[0].id).toBe('a');+ expect(results[1].id).toBe('c');+ });++ it('produces byte-identical, deterministic ordering across repeated runs on the same input', async () => {+ const makeSearch = (): SearchFn => async () => ({+ results: [+ makeCandidate({ id: 'a', content: 'the quarterly revenue report shows strong growth', score: 0.95, embedding: [1, 0, 0] }),+ makeCandidate({ id: 'b-paraphrase', content: 'quarterly earnings update reveals robust expansion', score: 0.9, embedding: [0.99, 0.01, 0] }),+ makeCandidate({ id: 'c-different-topic', content: 'a stray cat wandered into the office kitchen', score: 0.6, embedding: [0, 1, 0] }),+ makeCandidate({ id: 'd-tie-embedding', content: 'unrelated filler text about weather patterns', score: 0.5, embedding: [0, 1, 0] }),+ ],+ });++ const opts = {+ query: 'x',+ limit: 4,+ multiQuery: false,+ recencyBoost: false,+ sessionDiversity: false,+ diversityMMR: true,+ mmrLambda: 0.3,+ } as const;++ const runs = await Promise.all(+ Array.from({ length: 5 }, () => smartSearch(makeSearch(), opts))+ );+ const orderings = runs.map((r) => r.results.map((c) => c.id).join(','));++ expect(new Set(orderings).size).toBe(1); // every run produced the exact same order+ });+});++describe('MMR internal output never carries embedding into a CLI/MCP-shaped whitelist', () => {+ it('mirrors the exact field whitelist used by mcp-tools/memory-tools.ts:559-569 and commands/memory.ts:633-639', async () => {+ // Regression guard for the "no leak" claim: both real CLI/MCP call sites+ // build their final response by explicitly picking fields off+ // `smartResult.results` rather than spreading the candidate object+ // (which β per smart-retrieval.ts's own return statement β does still+ // carry `.embedding` internally). This test exercises smartSearch's+ // real output and re-applies each site's literal field list, so a+ // future change to either whitelist that starts including `embedding`+ // (e.g. an errant `...r` spread) fails here without needing to spin up+ // the full CLI/MCP handler stack.+ const search: SearchFn = async () => ({+ results: [+ makeCandidate({ id: 'a', content: 'alpha content', score: 0.9, embedding: [1, 0, 0], provenanceType: 'agent_inference' } as any),+ ],+ });++ const { results } = await smartSearch(search, { query: 'x', diversityMMR: false, recencyBoost: false, sessionDiversity: false, multiQuery: false });++ expect((results[0] as any).embedding).toBeDefined(); // sanity: the internal object does carry it++ // mcp-tools/memory-tools.ts:559-569 whitelist+ const mcpToolShape = results.map((r: any) => ({+ key: r.key,+ namespace: r.namespace,+ value: r.content,+ similarity: r.score,+ provenanceType: r.provenanceType,+ }));+ expect(Object.keys(mcpToolShape[0])).not.toContain('embedding');+ expect('embedding' in mcpToolShape[0]).toBe(false);++ // commands/memory.ts:633-639 whitelist+ const cliCommandShape = results.map((r: any) => ({+ key: r.key,+ score: r.score,+ namespace: r.namespace,+ preview: r.content,+ provenanceType: r.provenanceType,+ }));+ expect(Object.keys(cliCommandShape[0])).not.toContain('embedding');+ expect('embedding' in cliCommandShape[0]).toBe(false);+ });+});+describe('smartSearch β session round-robin', () => {it('interleaves results across distinct sessions', async () => {const search: SearchFn = async () => ({diff --git a/v3/@claude-flow/memory/src/smart-retrieval.ts b/v3/@claude-flow/memory/src/smart-retrieval.tsindex eded6e37af..4eb2f1faea 100644--- a/v3/@claude-flow/memory/src/smart-retrieval.ts+++ b/v3/@claude-flow/memory/src/smart-retrieval.ts@@ -30,6 +30,15 @@ export interface SearchCandidate {createdAt?: number;/** Optional unix-ms timestamp; preferred over createdAt when present. */updatedAt?: number;+ /**+ * Optional embedding vector, when the caller's SearchFn has one on hand+ * (e.g. already computed for the primary similarity score). When both+ * candidates in an MMR comparison carry one, `mmrRerank` uses cosine+ * similarity instead of the token-Jaccard proxy β see "MMR Diversity"+ * below. Never forwarded to CLI/MCP JSON responses by this module's+ * callers (they whitelist output fields explicitly).+ */+ embedding?: number[];}export interface RawSearchRequest {@@ -238,7 +247,19 @@ function pickTimestamp(cand: SearchCandidate): number | undefined {return undefined;}-// ββ MMR Diversity (token-Jaccard proxy) ββββββββββββββββββββββββ+// ββ MMR Diversity (embedding-cosine, token-Jaccard fallback) βββ+//+// Dream Cycle 2026-09-03: 2026 production practice (LangChain's reference+// `maximal_marginal_relevance`, Qdrant's native `Mmr` query, Weaviate 1.37's+// MMR reranker) universally computes MMR's "similarity to already-selected"+// term over the embedding space, not lexical overlap β token-Jaccard misses+// low-token-overlap paraphrases that embeddings correctly flag as+// near-duplicates (see e.g. openclaw/openclaw#19760). Ruflo's own retrieval+// pipeline already computes these embeddings upstream; this only threads+// them through to MMR. Falls back to token-Jaccard whenever either+// candidate lacks an embedding (test fakes, or a SearchFn that doesn't+// supply one) or the two embeddings have mismatched dimensions, so existing+// callers/tests that never populate `.embedding` are unaffected./*** Public-facing MMR rerank.@@ -261,11 +282,13 @@ function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {const selected: Scored[] = [];const remaining = [...scored];const selectedTokens: Set<string>[] = [];+ const selectedEmbeddings: Array<number[] | undefined> = [];// Seed with the top-scored candidate.const first = remaining.shift()!;selected.push(first);selectedTokens.push(tokenize(first.candidate.content));+ selectedEmbeddings.push(first.candidate.embedding);while (selected.length < limit && remaining.length > 0) {let bestIdx = -1;@@ -274,9 +297,10 @@ function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {for (let i = 0; i < remaining.length; i++) {const cand = remaining[i];const candTokens = tokenize(cand.candidate.content);+ const candEmbedding = cand.candidate.embedding;let maxOverlap = 0;- for (const selTokens of selectedTokens) {- const sim = jaccard(candTokens, selTokens);+ for (let j = 0; j < selectedTokens.length; j++) {+ const sim = pairSimilarity(candEmbedding, selectedEmbeddings[j], candTokens, selectedTokens[j]);if (sim > maxOverlap) maxOverlap = sim;}const mmr = lambda * cand.score - (1 - lambda) * maxOverlap;@@ -290,11 +314,57 @@ function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {const [chosen] = remaining.splice(bestIdx, 1);selected.push(chosen);selectedTokens.push(tokenize(chosen.candidate.content));+ selectedEmbeddings.push(chosen.candidate.embedding);}return selected;}+/** Cosine similarity when both embeddings exist, agree in dimension, and are well-formed; token-Jaccard otherwise. */+function pairSimilarity(+ embA: number[] | undefined,+ embB: number[] | undefined,+ tokensA: Set<string>,+ tokensB: Set<string>+): number {+ if (+ isWellFormedEmbedding(embA) &&+ isWellFormedEmbedding(embB) &&+ embA.length === embB.length+ ) {+ return cosineSimilarity(embA, embB);+ }+ return jaccard(tokensA, tokensB);+}++/**+ * Guards against malformed embeddings (NaN/Infinity components, empty+ * arrays, non-arrays) reaching `cosineSimilarity`, where a single NaN/+ * Infinity would poison the dot product and silently corrupt every+ * downstream MMR comparison for the rest of the rerank β falls back to+ * `jaccard` instead, same as a missing/dimension-mismatched embedding.+ */+function isWellFormedEmbedding(emb: number[] | undefined): emb is number[] {+ return (+ Array.isArray(emb) &&+ emb.length > 0 &&+ emb.every((v) => typeof v === 'number' && Number.isFinite(v))+ );+}++function cosineSimilarity(a: number[], b: number[]): number {+ let dot = 0;+ let normA = 0;+ let normB = 0;+ for (let i = 0; i < a.length; i++) {+ dot += a[i] * b[i];+ normA += a[i] * a[i];+ normB += b[i] * b[i];+ }+ if (normA === 0 || normB === 0) return 0;+ return dot / (Math.sqrt(normA) * Math.sqrt(normB));+}+function tokenize(text: string): Set<string> {return new Set(text