🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
dream(performance): #3265 stop re-tokenizing MMR candidates that PR #3169 mostly stopped needing (evaluated, ACCEPT) (#3266)
rUvf13b8567de50
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.mdindex 34499b14dc..08eadc811a 100644--- a/docs/dream-cycle/LEDGER.md+++ b/docs/dream-cycle/LEDGER.md@@ -189,3 +189,4 @@ as a first-class, high-priority candidate, not a background note.| 2026-09-01 | security | `maxToolCallsPerTurn` was session-lifetime cumulative, never reset (disclosed bug in #3139); wall-clock sliding window fix, new `turnWindowMs` policy field (follow-up to #3139, stacked PR) | #3151 | #3152 | yes | ACCEPT | 30/30 new-suite tests, 52/56 broader MCP sweep (4 pre-existing unbuilt-sibling-package env failures, unrelated); independent critic CONFIRMED-WITH-CAVEATS, 2 minor findings fixed pre-commit | 7d93a482... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139 merges), awaiting fresh CI as of 2026-09-14 || 2026-09-02 | intelligence | `LearningBridge.consolidate()` reward-blind fix (hardcoded `1.0` → real insight confidence via `getByKey`; round-1 fix had a wrong id/key lookup — `entryId` is the caller-assigned `key`, not the backend's internal `entry.id` — caught by adversarial critic, corrected round 2) | #3159 | #3160 | yes | ACCEPT | 56/56→62/62 tests (candidate); 464/465 full package (1 pre-existing environmental failure); independent critic round 2 CONFIRMED | bead6402... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152 merges), awaiting fresh CI as of 2026-09-14 || 2026-09-08 | memory | `MemoryConsolidator.dedup()` deduplicated via byte-exact SHA-256 hash only, though every entry already carries a computed `.embedding` and `dedup()` already holds a handle to the adapter's incrementally-synced `HNSWIndex` (used only for `removePoint` bookkeeping, never queried for similarity) — added a second embedding-cosine near-duplicate pass (threshold 0.95, zero extra backend round-trips); independent adversarial critique found and this session fixed forward a real convergence bug (a duplicate cluster larger than the 8-entry search window split into multiple survivors in one call, self-healing across repeated `runAll()` but under-converging on a single call) | #3231 | #3232 | yes | ACCEPT | 472/473 full-suite green (1 pre-existing unrelated failure, unchanged); baseline-fails/candidate-passes via stash isolation for both the original fix and the convergence fix-forward; `tsc` clean | f5eaa461... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160 merges), awaiting fresh CI as of 2026-09-14 |+| 2026-09-10 | performance | `mmrRerank()` eagerly re-tokenized every remaining candidate's content on every MMR outer-loop pass (O(N×limit) `tokenize()` calls) — a pattern that predates and survived PR #3169's embedding-cosine wiring unchanged, even though the cosine branch made that tokenization mostly discarded work in the now-common all-embedded case; made lazy and cached per-candidate | #3265 | #3266 | yes | ACCEPT | `tokenize()` calls 400→0 (N=30/limit=20 corpus, stash-isolated: fails against real baseline, passes against candidate); wall-clock 5.88x faster (0.6745ms→0.1148ms mean), independently reproduced at 5.96x by an adversarial critic; selection output byte-identical to baseline across all-embedded/none/mixed/lambda-extreme scenarios; 473/474 full-suite both ways (1 pre-existing unrelated failure), `tsc` clean | 78c074ef... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232 merges), awaiting fresh CI as of 2026-09-14 |diff --git a/docs/dream-cycle/dream-gist-2026-09-10.md b/docs/dream-cycle/dream-gist-2026-09-10.mdnew file mode 100644index 0000000000..b7b927ed17--- /dev/null+++ b/docs/dream-cycle/dream-gist-2026-09-10.md@@ -0,0 +1,104 @@+# Performance SOTA Report — 2026-09-10++TL;DR: Tonight's performance deep-dive found a genuine algorithmic-efficiency regression in `mmrRerank()` (`v3/@claude-flow/memory/src/smart-retrieval.ts`), the MMR diversity-reranking step behind `smartSearch`/`applyMMR` — the retrieval entry point on Ruflo's memory/ReasoningBank hot path. The function eagerly re-tokenized every remaining candidate's content string on *every* outer-loop pass (O(N × limit) `tokenize()` calls), a pattern that predates and survived PR #3169 (merged 2026-09-05) unchanged. #3169 wired embedding-cosine similarity through as the *preferred* comparison path whenever both compared candidates carry well-formed, matching-dimension embeddings — which is now the common production case, since Ruflo's own retrieval pipeline populates `.embedding` upstream. That means the eager tokenization this fix replaces was, in the common case, pure wasted work: computed and then thrown away by `pairSimilarity()`'s cosine branch almost every time. Fixed by making tokenization lazy and cached per-candidate via a `Map<SearchCandidate, Set<string>>`, so tokens are computed at most once per candidate and only when the token-Jaccard fallback path is actually exercised. Measured: `tokenize()` call count drops from 400 to 0 on a 30-candidate/limit-20 all-embedded corpus (stash-isolated: fails against real baseline, passes against candidate), and a `vitest bench` wall-clock comparison shows the candidate **5.88x faster** (mean 0.1148ms vs. 0.6745ms) on the same corpus shape. Five research roles ran in parallel tonight; a live, reachable hive-mind consensus-strategy bug (init-time `byzantine`/`gossip`/`crdt` selection is silently discarded — every proposal resolves via plain Raft majority regardless) is documented below as a candidate for a future `hive-mind`/`swarm` night rather than folded into tonight's single-conceptual-change patch.++## What's New in 2026++| Finding | Source | Confidence |+|---|---|---|+| Extended RaBitQ: asymptotically-optimal 1-bit-per-dim quantization, no codebook training needed | [VectorDB-NTU/Extended-RaBitQ (SIGMOD 2025)](https://github.com/VectorDB-NTU/Extended-RaBitQ) | A (peer-reviewed ACM SIGMOD) |+| Independent 7-vector-DB benchmark: LanceDB's default IVF-PQ collapses to 0.245 recall on GIST1M (960-dim) — PQ recall collapse on high-dimensional data is an industry-wide, not Ruflo-specific, failure mode | [arXiv:2608.12812, Aug 2026](https://arxiv.org/abs/2608.12812) | A (independent academic benchmark, 6 datasets, 4M+ vectors) |+| Milvus maintains its own reproducible `ann-benchmarks` fork; leads evaluated systems at 0.971 recall on GIST1M | [milvus-io/ann-benchmarks](https://github.com/milvus-io/ann-benchmarks) | A (reproducible, independently replicated) |+| RouteLLM-style cascade/learned routing: 85% cost savings at 95% GPT-4-equivalent quality | [RouteLLM / LLMRouterBench survey, arXiv:2601.07206](https://arxiv.org/html/2601.07206v1) | A (peer-reviewed, widely reproduced) |+| DiskANN/Vamana single-graph pruned-edge disk-resident ANN: >5000 QPS / <3ms at billion-scale, 95%+ recall@1 | [Microsoft Research (NeurIPS)](https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/) | A (established, NeurIPS) |+| Weaviate's Rotational Quantization (RQ, successor to Binary Quantization): ~4x compression, 98-99% recall retained | Weaviate engineering blog | B (vendor claim, methodology detailed, not independently reproduced) |+| Filtered/predicate-aware graph-based ANN indexing for hybrid search on GPUs | [arXiv:2511.00855, Nov 2025](https://arxiv.org/pdf/2511.00855) | C (single arXiv preprint, not yet independently reproduced) |++No material change found since the prior audit on: HNSW asymptotic-complexity claims (nothing changes the repo's own measured 1.9-4.7x figures), or Flash Attention speedup literature specific to Ruflo's tiled-kernel implementation (nothing found that would justify re-chasing the removed 2.49-7.47x marketing figure).++## Ruflo Current Capability++`smartSearch()` (`v3/@claude-flow/memory/src/smart-retrieval.ts:442-531`) is Ruflo's LongMemEval-derived retrieval pipeline (ADR-090): query expansion → multi-query RRF fusion → recency boost → MMR diversity reranking → session round-robin. Phase 4 (`mmrRerank`, lines 279-343 pre-fix) greedily selects candidates by `λ·relevance - (1-λ)·max_similarity_to_selected`, where similarity is embedding-cosine when available (PR #3169, 2026-09-03/05) or token-Jaccard otherwise (`pairSimilarity`, lines 324-338). The bug: `candTokens = tokenize(cand.candidate.content)` (former line 299) ran inside the O(remaining.length) inner loop on *every* outer pass, and `selectedTokens.push(tokenize(...))` (former lines 290, 316) ran eagerly for every selected candidate too — regardless of whether the embedding-cosine branch was about to make those tokens irrelevant. A default `smartSearch()` call (`fanOutK = limit×3`, MMR `limit = min(limit×2, scored.length)`) feeds `mmrRerank` up to 30 candidates with an internal limit of 20 — 19 outer passes × up to 29 remaining candidates ≈ 551 theoretical max tokenizations (400 measured on this corpus), versus 0 needed once embeddings are populated. `smartSearch` is invoked on every ReasoningBank/agent-memory query, so this cost compounds silently on a path the project explicitly optimizes elsewhere (HNSW, RaBitQ, Int8 quantization) — undercutting those gains downstream in the same pipeline.++## Competitor Comparison++| System | Mechanism | Performance characteristic | Evidence grade |+|---|---|---|---|+| Qdrant | HNSW + 8-bit Scalar Quantization | p50 4ms / p99 25ms, recall@10 0.99 at 1M vectors, 1536-dim | B (third-party benchmark, semi-reproducible) |+| Milvus | HNSW/IVF hybrid, own reproducible `ann-benchmarks` fork | 0.971 recall on GIST1M, leads all evaluated systems; 200-400 QPS at 90-99% recall | A (reproducible, independently replicated) |+| LanceDB | IVF-PQ (default index) | Recall collapses to 0.245 on GIST1M (960-dim) — falls below 0.90 recall on 5/6 tested datasets; a real GitHub issue (#1428) traces a related HNSW recall gap (87% vs. FAISS's 97% on SIFT10k) to an implementation bug, since fixed | A (arXiv:2608.12812, independent academic benchmark; primary-source GitHub issue) |+| Weaviate | Rotational Quantization (successor to Binary Quantization) | ~4x compression, 98-99% recall retained — explicitly more robust than BQ | B (vendor engineering blog) |+| LangGraph / CrewAI / AutoGen | No native vector-memory or cold-start benchmark; memory delegated to a backing store (Chroma) or per-agent history | Framework cold start 60-140ms (Python); CrewAI's own docs describe memory delegation with no retrieval-latency numbers of its own; CrewAI showed a 44% failure rate under one 2026 load test | C (genuine, source-confirmed gap — delegated by design, not hidden) |+| OpenAI Agents SDK | No published cold-start or model-routing benchmark | Cost lever is prompt-caching, not routing — model routing explicitly left to the caller or third-party routers | C (deliberately out of scope per OpenAI's own SDK docs) |++Synthesis: Ruflo's own PQ recall weak point (0.270, flagged 2026-08-25/PR #3094) is not an outlier — the most rigorous independent 2026 study found LanceDB's default IVF-PQ collapses even further (0.245) on a comparable high-dimensional dataset, and a real LanceDB GitHub issue traces a similar gap to an implementation bug rather than a fundamental PQ ceiling. This is a legitimate future-night lead: Ruflo's own RaBitQ path (32x compression, already shipped) is directionally aligned with the same rotation-based fix (OPQ / RaBitQ-style rotation) Weaviate has already shipped as RQ. On agent-framework cold-start/routing, the near-total silence from LangGraph/CrewAI/AutoGen/OpenAI's SDK is a genuine, source-confirmed gap (delegated to backing stores or third parties by design) rather than evidence Ruflo is behind — Ruflo's measured 54.65ms cold start and MoE routing convergence (0.13→0.88 confidence) are first-party numbers in a space where most competitors publish none.++## Hypothesis++> Given `mmrRerank()` invoked on a candidate set where both compared candidates in a pairwise MMR comparison carry well-formed, matching-dimension embeddings (the common case since PR #3169), when tokenization is made lazy and cached per-candidate instead of eager per-outer-pass, then `tokenize()` call count and wall-clock time for `mmrRerank` should decrease materially relative to the pre-fix baseline, subject to: (1) selection output (ranked order and scores) is byte-for-byte identical to the pre-fix baseline across all-embedded, no-embedding, and mixed candidate sets — this is a pure efficiency refactor, not a behavior change; (2) the existing `smart-retrieval`/`mmr-benchmark` test suites remain green; (3) zero added LLM/API cost; (4) no latency regression in the token-Jaccard fallback path (missing/malformed/mismatched embeddings).++Frozen before evaluation began; not modified after seeing results.++## Benchmarks / Evaluation++**evaluated: accepted (ACCEPT).** Real evaluator: Vitest 4.1.8 (`v3/` pnpm workspace installed fresh tonight), deterministic, zero LLM calls, $0 cost.++Candidate diff: `v3/@claude-flow/memory/src/smart-retrieval.ts` (~30 net lines — `mmrRerank`'s eager `tokenize()` calls replaced with a `getTokens()` closure backed by a per-call `Map<SearchCandidate, Set<string>>`; `pairSimilarity` now takes the two `Scored` items plus `getTokens` instead of precomputed token sets; `tokenize` exported for the regression test). New test file `v3/@claude-flow/memory/src/mmr-rerank-perf.test.ts` (8 tests): correctness parity across all-embedded/no-embedding/mixed/lambda-extreme scenarios against a frozen, unmodified copy of the pre-fix algorithm; call-count assertions via `vi.spyOn(smartRetrieval, 'tokenize')`. New benchmark `v3/@claude-flow/memory/benchmarks/mmr-rerank.bench.ts` + receipt `benchmarks/results/mmr-rerank-lazy-tokenize-receipt.json`.++**Baseline vs. candidate, isolated via `git stash` of just the source file (test file kept):** 7 of 8 new tests **fail against real baseline** (`tokenize is not a function` — baseline never exported it; the un-exported call-count assertions fail with "property not defined on the object") and **8/8 pass against candidate**. Call count on the N=30/limit=20 all-embedded corpus: **baseline 400, candidate 0** (measured via `vi.spyOn`, not asserted). Wall-clock (`vitest bench`, same corpus): **baseline mean 0.6745ms (1,482.53 hz, ±1.43% RME, 742 samples) vs. candidate mean 0.1148ms (8,711.84 hz, ±0.49% RME, 4,356 samples) — candidate 5.88x faster.**++Full `@claude-flow/memory` package suite: **473/474 passing** both before and after (1 pre-existing environmental failure in `auto-memory-bridge.test.ts` — sandbox runs as root, bypassing a `chmod`-based read-only precondition; same known issue disclosed in PR #3094, confirmed unrelated to this diff). `tsc --noEmit`: **zero errors.**++**Adversarial critique (STEP 10):** an independent reviewer session (no authoring context) re-read the diff, independently reran the test suite (26/26 pass), `tsc --noEmit` (clean), and the benchmark (**5.96x reproduced independently**, vs. 5.88x in the original receipt — normal run-to-run variance, not cherry-picked), and applied the full adversarial checklist. Verdict: **holds up, no blocking issues.** Confirmed the `pairSimilarity` branch condition itself is untouched (only *when* tokens get computed changed, not the similarity logic); confirmed the `selectedTokens.length`→`selected.length` loop-bound swap is safe (`selected`/`selectedEmbeddings` were always pushed in lockstep); confirmed no duplicate-reference `SearchCandidate` objects reach `mmrRerank` from its only production caller (`controller-registry.ts:802/880`, via `applyRRF`'s per-dedup-key object construction), and that object-reference `Map` keying is provably never *wrong* even if that changed (tokens are a pure function of `.content`) — only a stronger-than-necessary key; confirmed the frozen baseline used in both the test and benchmark is a faithful line-by-line copy of the removed code, not a strawman. Two **worth-noting** (non-blocking) findings, both addressed before this report: (1) exporting `tokenize()` is a real public-API-surface addition — `@claude-flow/memory`'s `package.json` `exports` map is a `"./*"` wildcard, not scoped to test files — the in-code comment originally understated this; corrected to disclose it explicitly (the function is pure/side-effect-free, so the critic assessed this as low-risk). (2) the file was already 531 lines before this change (over `v3/CLAUDE.md`'s 500-line/file guideline) and the diff pushed it to 550; the added explanatory comment was trimmed to bring it to 549 — still over the guideline, but that debt predates this PR and a full restructure was judged out of scope for a single-conceptual-change fix. One **nitpick**: for very small fallback-path candidate counts, the `Map` alloc could theoretically be marginally slower than flat-array indexing — not measurable in practice (`tokenize()`'s string work dominates), no benchmark evidence suggests it matters.++## Darwin Results++Skipped — same class as 7 of the last 8 dream-cycle nights (08-18, 08-19, 08-24, 08-25, 08-27, 08-30, 09-02, 09-05). Confirmed via `npx [email protected] harness darwin --help`: the real interface evolves routing/topology/prompt/memory/tool/tier/context/coordination genome *parameters* against a fitness-scored bench corpus. Tonight's fix is a binary correctness-preserving efficiency refactor (lazy vs. eager tokenization) with no continuous or categorical parameter to search over — there is no fitness gradient here for Darwin to explore.++## Flywheel Evidence++No signed `@metaharness/flywheel` bundle (`npx [email protected] harness flywheel --help` confirms `verify`/`gate` operate on LLM-task-corpus evidence bundles) — this is a deterministic code-correctness/efficiency fix with a Vitest receipt, not an LLM-scored corpus, same as every accepted algorithmic-fix night since 2026-08-18. Evidence retained as: 8 new deterministic tests + the wall-clock benchmark receipt + this gist + the linked issue + the stash-isolated baseline-fails/candidate-passes result. Classified: **OBSERVATION** (eager per-outer-pass `tokenize()` calls, confirmed via read + `git log -p` showing the pattern predates and survived PR #3169 unchanged) / **MEASUREMENT** (baseline 400 calls / 0.6745ms mean vs. candidate 0 calls / 0.1148ms mean, both stash-isolated and independently re-measured by the adversarial critic) / **INFERENCE** (the well-formed-embedding branch is the common production case because Ruflo's own retrieval pipeline populates `.embedding` upstream — this session did not measure the live production embedding-population rate directly) / **DECISION** (ship as a small, single-file, behavior-preserving refactor) / **REJECTION** (none this round).++## Reward Hack Check++No standalone generic reward-hack CLI reachable this session (consistent with recent nights). Manual checklist, independently re-verified by the adversarial critic: no test weakened (`mmr-rerank-perf.test.ts` is purely additive; existing `smart-retrieval.test.ts`/`mmr-benchmark.test.ts` untouched and re-verified green both ways); no gold-answer path exists in this code; no cherry-picking (full package suite run both ways, the one pre-existing failure disclosed rather than omitted); no seed manipulation (fully deterministic corpus, no randomness in the measured metrics); zero cost (no model calls anywhere). One boundary case caught and disclosed rather than hidden: the first call-count assertion was initially written as `toBeGreaterThan(400)` and failed against the candidate run itself (measured count was exactly 400, not >400) — corrected to `toBeGreaterThanOrEqual(400)` before this was reported as a receipt, not loosened after-the-fact to paper over a discrepancy in the *conclusion* (candidate-vs-baseline direction and magnitude were correct throughout; only the boundary operator was off).++## Security Review++Not security-sensitive — pure in-process string/Set computation and caching, no new network/filesystem/credential/MCP-authority surface. The one API-surface change (`tokenize()` becomes an exported function, and — per the adversarial critic — genuinely public given `@claude-flow/memory`'s wildcard `exports` map, not merely test-visible) is a pure, side-effect-free string-tokenization utility; the critic assessed this as low-risk and it is disclosed rather than downplayed in the code comment.++## Scan Findings: security++Light scan pass — no new finding. Re-confirmed the 2026-09-07 finding is unchanged: `PluginIntegrityVerifier` (Ed25519 signature + trust-anchor check, `@claude-flow/security/src/plugins/integrity-verifier.ts`) remains completely unwired from `PluginManager.installFromNpm()`/`installFromLocal()` (`v3/@claude-flow/cli/src/plugins/manager.ts` — imports only `fs`/`path`/`child_process`/`util`, zero reference to the verifier). `trust-anchors.json`'s sole public key is still the placeholder string, and every registry `checksum` (20 entries in `discovery.ts`) is still a non-functional placeholder like `sha256:abc123neural`. Checked performance-overlap angles (HNSW ingestion NaN/Infinity handling, MCP vector-parameter schemas, cost/budget-tracker atomicity) — no externally-reachable injection point found; not reported as a new finding to avoid padding with speculation.++## Scan Findings: hive-mind++**New, code-verified finding — the consensus strategy selected at `hive-mind init` is silently discarded; every proposal resolves via plain Raft majority regardless of configuration.** The reachable implementation behind both the `claude-flow hive-mind` CLI and its MCP tool is a self-contained JSON-state simulator (`v3/@claude-flow/cli/src/mcp-tools/hive-mind-tools.ts`) that never imports `@claude-flow/swarm`'s real `ConsensusEngine`/`ByzantineConsensus`/`RaftConsensus`/`GossipConsensus` implementations at all — those exist and are genuine (~15-17KB each) but are architecturally dead code from the CLI's perspective, consistent with the recurring "exists but nothing calls it" pattern (cf. PR #3243). Inside the reachable simulator: `hive-mind_init` (line 22, 310, 326) stores a 5-way `consensusStrategy` (`raft|byzantine|gossip|crdt|quorum`) on `state.consensusStrategy`. `hive-mind_consensus` — the tool that actually creates and votes on proposals — uses a *different, narrower* type (`'bft' | 'raft' | 'quorum'`, line 43) read from a **per-call** `strategy` input param defaulting to `'raft'` (line 551), and never reads `state.consensusStrategy` anywhere (grep-confirmed: the field is written once, echoed in status output twice, never consumed by `calculateRequiredVotes`/`tryResolveProposal`). The CLI's own `consensus` subcommand (`v3/@claude-flow/cli/src/commands/hive-mind.ts:1210-1221`) doesn't even expose a `--strategy` flag. Net effect: `claude-flow hive-mind init -c byzantine` (the docs' own example) followed by `consensus -a propose` always resolves via plain-majority Raft, never Byzantine fault detection — and `gossip`/`crdt` are accepted as init-time labels with **no corresponding branch anywhere** in the resolution code. Fully reachable in production, not gated behind an unused package. Not implemented tonight (out of surface — a `hive-mind` or `swarm` DEEP night candidate). Files: `hive-mind-tools.ts:22,43,326,551,621-622`; `hive-mind.ts:1210-1221`.++## Additional candidates surfaced but not selected tonight (STEP 3.2)++Scored under the rubric (0.25·fit + 0.20·testability + 0.20·measurability + 0.15·production_value + 0.10·novelty + 0.10·reviewability); tonight's arch-reviewer finding scored 4.75 and was selected as the outright highest score (no override needed) over the deep researcher's top pick:++1. **Adaptive/exhaustive-fallback filtered ANN search** (score 4.65) — `HNSWIndex.searchWithFilters()` (`v3/@claude-flow/memory/src/hnsw-index.ts:390-403`) uses a hardcoded `overFetchFactor = 3` with no adaptivity to filter selectivity and no fallback loop if fewer than `k` results survive filtering — a live, reachable correctness gap (silent under-return under high-selectivity filters), grounded in active 2025-2026 filtered/hybrid-ANN research. Strong candidate for a future performance night; deliberately not combined with tonight's pick to keep one conceptual change per PR and avoid compounding review debt on `hnsw-index.ts`, which already has two open unmerged PRs touching it (#3094, #3130).+2. **OPQ rotation on top of the already-wired flat PQ** (score 4.45) — would directly close the 0.270 PQ-recall gap flagged 2026-08-25, using the exact rotation-based approach the competitor research above shows Weaviate already shipped (RQ). Would compound on PR #3094's still-open diff to the same `Quantizer` class — queue for after #3094 merges.+3. **Offline cascade-routing counterfactual on `router-parallel.jsonl` logs** (score 4.15) — a research/analysis finding, not a shipped perf change; low risk, cheap, worth a future night's time.+4. **RaBitQ retrieval-speed benchmark** (score 4.00) — closes the audit's flagged "not measured anywhere" gap for the already-shipped 32x-compression RaBitQ path; the honest fallback candidate (a 4th instance of the "unverified claim" pattern the operator asked not to default to), correctly not selected given 3 stronger, more novel candidates existed.+5. **Bounded-recall HNSW via graph spanners** (score 2.65) — high novelty (arXiv 2607.02338) but poor practical fit: favors batch/static graph construction, not Ruflo's insert-heavy incremental index.++## SOTA Proof & Witness++| Field | Value |+|---|---|+| Session commit | `9576a1032b0749ac12d08fd9d05b86d4bee893b5` |+| Gist SHA-256 (pre-witness content) | `8e9585b57f7d55faa52f947b12d8220f3c3e45662af04dd11a8c2d11473db0f4` |+| Witness stamp | `78c074ef6465bee435a913d93060244dbcc0601e4cfadc35851806059f186c6b` |++Verifier procedure: fetch `docs/dream-cycle/dream-gist-2026-09-10.md` from this branch, strip the witness table's filled values back to `PENDING`, SHA-256 it, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.++## Recommended Next Steps++1. **Merge the linked draft PR** (human review required) — a small (~30 net lines in one source file), fully behavior-preserving efficiency fix with a discriminating baseline-fails/candidate-passes regression test and a real measured 5.88x wall-clock speedup on the common production shape.+2. **Adaptive filtered-ANN over-fetch/fallback** (candidate 1 above) — a live correctness gap in `searchWithFilters()`, strong candidate for the next `performance` DEEP night, deliberately kept separate from tonight's file/PR to avoid compounding the existing `hnsw-index.ts` review backlog.+3. **OPQ/rotation-based PQ recall fix** (candidate 2) — queue after PR #3094 merges to avoid stacking changes on the same open diff.+4. **Wire or scope down hive-mind's consensus-strategy selection** (scan finding) — either thread `state.consensusStrategy` through to `hive-mind_consensus`'s resolution logic (byzantine/gossip/crdt), or correct the CLI docs' own `-c byzantine` example and `hive-mind status` output to stop implying a strategy that is never actually used.+5. **Merge backlog**: as of tonight, the performance surface alone carries 2 open/unmerged ACCEPT-verdict PRs (#3094, #3130 — the latter now shows a merge conflict with `main`), plus a wider cross-surface backlog tracked in the ledger. Tonight's candidate was deliberately kept small, single-file, and non-overlapping with either open PR per STEP 1.1's guidance given this backlog.diff --git a/v3/@claude-flow/memory/benchmarks/mmr-rerank.bench.ts b/v3/@claude-flow/memory/benchmarks/mmr-rerank.bench.tsnew file mode 100644index 0000000000..aeb638b6b4--- /dev/null+++ b/v3/@claude-flow/memory/benchmarks/mmr-rerank.bench.ts@@ -0,0 +1,108 @@+/**+ * mmrRerank wall-clock benchmark (Dream Cycle 2026-09-10).+ *+ * Measures `applyMMR` (candidate: lazy, cached tokenization) against a+ * frozen copy of the pre-candidate algorithm (baseline: eager tokenization+ * on every outer-loop pass) on the same synthetic corpus, so the tokenize()+ * call-count reduction proven in `src/mmr-rerank-perf.test.ts` has a real+ * measured latency number behind it, not just an asserted call count.+ *+ * Corpus: N=30 candidates, limit=20, all with well-formed matching-dimension+ * embeddings — the common production shape since PR #3169 wired embeddings+ * through `smartSearch`'s call sites.+ *+ * @see {@link ../docs/adr/ADR-125-memory-consolidation.md}+ */+import { describe, bench } from 'vitest';+import { applyMMR, tokenize, type SearchCandidate } from '../src/smart-retrieval.js';++function seededEmbedding(seed: number, dim = 8): number[] {+ const out: number[] = [];+ let x = seed * 2654435761;+ for (let i = 0; i < dim; i++) {+ x = (x * 1103515245 + 12345) & 0x7fffffff;+ out.push((x % 1000) / 1000);+ }+ return out;+}++const N = 30;+const LIMIT = 20;++const corpus: SearchCandidate[] = Array.from({ length: N }, (_, i) => ({+ id: `c${i}`,+ key: `c${i}`,+ namespace: 'bench',+ content: Array.from({ length: 6 }, (_, w) => `w${(i * 7 + w) % 40}`).join(' ') + ` item ${i}`,+ score: 1 - i / N,+ embedding: seededEmbedding(i),+}));+const scored = corpus.map((c) => ({ candidate: c, score: c.score }));++/** Frozen copy of the pre-candidate algorithm — eager tokenize() every outer pass. */+function baselineMmrRerank(+ input: Array<{ candidate: SearchCandidate; score: number }>,+ lambda: number,+ limit: number+): Array<{ candidate: SearchCandidate; score: number }> {+ if (input.length <= 1) return input.slice(0, limit);+ const selected: typeof input = [];+ const remaining = [...input];+ const selectedTokens: Set<string>[] = [];+ const selectedEmbeddings: Array<number[] | undefined> = [];+ const isWellFormed = (emb: number[] | undefined): emb is number[] =>+ Array.isArray(emb) && emb.length > 0 && emb.every((v) => typeof v === 'number' && Number.isFinite(v));+ const cosine = (a: number[], b: number[]): number => {+ let dot = 0, na = 0, nb = 0;+ for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }+ if (na === 0 || nb === 0) return 0;+ return dot / (Math.sqrt(na) * Math.sqrt(nb));+ };+ const jaccard = (a: Set<string>, b: Set<string>): number => {+ if (a.size === 0 && b.size === 0) return 0;+ let inter = 0;+ for (const t of a) if (b.has(t)) inter++;+ const union = a.size + b.size - inter;+ return union === 0 ? 0 : inter / union;+ };+ const pairSim = (embA: number[] | undefined, embB: number[] | undefined, tA: Set<string>, tB: Set<string>) =>+ isWellFormed(embA) && isWellFormed(embB) && embA.length === embB.length ? cosine(embA, embB) : jaccard(tA, tB);++ const first = remaining.shift()!;+ selected.push(first);+ selectedTokens.push(tokenize(first.candidate.content));+ selectedEmbeddings.push(first.candidate.embedding);++ while (selected.length < limit && remaining.length > 0) {+ let bestIdx = -1;+ let bestMmr = -Infinity;+ for (let i = 0; i < remaining.length; i++) {+ const cand = remaining[i];+ const candTokens = tokenize(cand.candidate.content);+ const candEmbedding = cand.candidate.embedding;+ let maxOverlap = 0;+ for (let j = 0; j < selectedTokens.length; j++) {+ const sim = pairSim(candEmbedding, selectedEmbeddings[j], candTokens, selectedTokens[j]);+ if (sim > maxOverlap) maxOverlap = sim;+ }+ const mmr = lambda * cand.score - (1 - lambda) * maxOverlap;+ if (mmr > bestMmr) { bestMmr = mmr; bestIdx = i; }+ }+ if (bestIdx < 0) break;+ const [chosen] = remaining.splice(bestIdx, 1);+ selected.push(chosen);+ selectedTokens.push(tokenize(chosen.candidate.content));+ selectedEmbeddings.push(chosen.candidate.embedding);+ }+ return selected;+}++describe('mmrRerank: baseline (eager tokenize) vs candidate (lazy cached)', () => {+ bench('baseline — eager tokenize every outer pass', () => {+ baselineMmrRerank(scored, 0.7, LIMIT);+ });++ bench('candidate — lazy cached tokenize (applyMMR)', () => {+ applyMMR(scored, 0.7, LIMIT);+ });+});diff --git a/v3/@claude-flow/memory/benchmarks/results/mmr-rerank-lazy-tokenize-receipt.json b/v3/@claude-flow/memory/benchmarks/results/mmr-rerank-lazy-tokenize-receipt.jsonnew file mode 100644index 0000000000..4c11ee11e8--- /dev/null+++ b/v3/@claude-flow/memory/benchmarks/results/mmr-rerank-lazy-tokenize-receipt.json@@ -0,0 +1,18 @@+{+ "benchmark": "mmr-rerank.bench.ts",+ "date": "2026-09-10",+ "dreamCycleFinding": "mmrRerank eagerly re-tokenized every remaining candidate on every outer-loop pass; PR #3169 made the well-formed-embedding branch the common case, so almost all of that tokenization was discarded work. Candidate makes tokenization lazy and cached per-candidate.",+ "corpus": { "n": 30, "limit": 20, "lambda": 0.7, "allEmbedded": true, "embeddingDim": 8 },+ "runner": "vitest bench (v4.1.8), node, vitest.bench.config.ts",+ "results": {+ "baseline": { "label": "eager tokenize every outer pass (frozen pre-candidate algorithm)", "hz": 1482.53, "meanMs": 0.6745, "p99Ms": 1.0426, "samples": 742, "rme": "±1.43%" },+ "candidate": { "label": "lazy cached tokenize (applyMMR, current source)", "hz": 8711.84, "meanMs": 0.1148, "p99Ms": 0.1940, "samples": 4356, "rme": "±0.49%" }+ },+ "speedup": "5.88x (candidate hz / baseline hz)",+ "tokenizeCallCount": {+ "baseline": 400,+ "candidate": 0,+ "note": "measured via vi.spyOn in src/mmr-rerank-perf.test.ts, stash-isolated (fails against real pre-candidate source, passes against candidate)"+ },+ "reproduce": "cd v3/@claude-flow/memory && npx vitest bench --config vitest.bench.config.ts benchmarks/mmr-rerank.bench.ts"+}diff --git a/v3/@claude-flow/memory/src/mmr-rerank-perf.test.ts b/v3/@claude-flow/memory/src/mmr-rerank-perf.test.tsnew file mode 100644index 0000000000..b68328596c--- /dev/null+++ b/v3/@claude-flow/memory/src/mmr-rerank-perf.test.ts@@ -0,0 +1,218 @@+/**+ * mmrRerank tokenize-call regression test (Dream Cycle 2026-09-10).+ *+ * PR #3169 (2026-09-03/05, merged) wired embedding-cosine through as the+ * preferred `pairSimilarity` path, making it the common case whenever+ * Ruflo's own retrieval pipeline populates `.embedding` upstream — but the+ * eager `tokenize()` calls seeding `selectedTokens`/`candTokens` on every+ * outer-loop pass of `mmrRerank` survived that change unchanged, so a+ * fully-embedded call site paid for O(N x limit) tokenizations whose+ * result was then thrown away by the cosine branch almost every time.+ *+ * This candidate makes token computation lazy and cached per candidate+ * instead. This file proves two properties, using the actual production+ * `applyMMR`/`tokenize` exports (not a reimplementation of the fix):+ *+ * 1. CORRECTNESS: the new implementation's output is byte-identical to+ * a frozen copy of the *old* eager-tokenize algorithm, across+ * all-embedded, no-embedding, and mixed candidate sets. This is a+ * pure efficiency refactor — it must never change selection order.+ * 2. CALL COUNT: `tokenize()` is called dramatically fewer times by the+ * candidate than by the frozen baseline algorithm, on the exact+ * corpus shape (N=30, limit=20) the arch-reviewer used to estimate+ * the ~551-calls-per-search baseline cost.+ */+import { describe, it, expect, vi } from 'vitest';+import * as smartRetrieval from './smart-retrieval.js';+import { applyMMR, tokenize, type SearchCandidate } from './smart-retrieval.js';++function makeCandidate(+ id: string,+ content: string,+ score: number,+ embedding?: number[]+): SearchCandidate {+ return { id, key: id, content, score, namespace: 'test', embedding };+}++/** Deterministic pseudo-embedding so cosine similarity is meaningful but reproducible. */+function seededEmbedding(seed: number, dim = 8): number[] {+ const out: number[] = [];+ let x = seed * 2654435761;+ for (let i = 0; i < dim; i++) {+ x = (x * 1103515245 + 12345) & 0x7fffffff;+ out.push((x % 1000) / 1000);+ }+ return out;+}++function corpus(n: number, opts: { embedded: 'all' | 'none' | 'half' }): SearchCandidate[] {+ return Array.from({ length: n }, (_, i) => {+ const words = Array.from({ length: 6 }, (_, w) => `w${(i * 7 + w) % 40}`).join(' ');+ const hasEmbedding =+ opts.embedded === 'all' || (opts.embedded === 'half' && i % 2 === 0);+ return makeCandidate(+ `c${i}`,+ `${words} item number ${i}`,+ 1 - i / n,+ hasEmbedding ? seededEmbedding(i) : undefined+ );+ });+}++/**+ * Frozen copy of the pre-candidate `mmrRerank` algorithm: eagerly tokenizes+ * every remaining candidate on every outer-loop pass, exactly as+ * `smart-retrieval.ts` did before this Dream Cycle's fix. Used only as a+ * correctness/call-count baseline — never modified after being frozen here.+ */+function baselineMmrRerank(+ scored: Array<{ candidate: SearchCandidate; score: number }>,+ lambda: number,+ limit: number+): Array<{ candidate: SearchCandidate; score: number }> {+ if (scored.length <= 1) return scored.slice(0, limit);++ const selected: typeof scored = [];+ const remaining = [...scored];+ const selectedTokens: Set<string>[] = [];+ const selectedEmbeddings: Array<number[] | undefined> = [];++ const isWellFormed = (emb: number[] | undefined): emb is number[] =>+ Array.isArray(emb) && emb.length > 0 && emb.every((v) => typeof v === 'number' && Number.isFinite(v));+ const cosine = (a: number[], b: number[]): number => {+ let dot = 0, na = 0, nb = 0;+ for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }+ if (na === 0 || nb === 0) return 0;+ return dot / (Math.sqrt(na) * Math.sqrt(nb));+ };+ const jaccard = (a: Set<string>, b: Set<string>): number => {+ if (a.size === 0 && b.size === 0) return 0;+ let inter = 0;+ for (const t of a) if (b.has(t)) inter++;+ const union = a.size + b.size - inter;+ return union === 0 ? 0 : inter / union;+ };+ const pairSim = (embA: number[] | undefined, embB: number[] | undefined, tA: Set<string>, tB: Set<string>) =>+ isWellFormed(embA) && isWellFormed(embB) && embA.length === embB.length ? cosine(embA, embB) : jaccard(tA, tB);++ const first = remaining.shift()!;+ selected.push(first);+ selectedTokens.push(tokenize(first.candidate.content));+ selectedEmbeddings.push(first.candidate.embedding);++ while (selected.length < limit && remaining.length > 0) {+ let bestIdx = -1;+ let bestMmr = -Infinity;+ for (let i = 0; i < remaining.length; i++) {+ const cand = remaining[i];+ const candTokens = tokenize(cand.candidate.content); // eager, every pass+ const candEmbedding = cand.candidate.embedding;+ let maxOverlap = 0;+ for (let j = 0; j < selectedTokens.length; j++) {+ const sim = pairSim(candEmbedding, selectedEmbeddings[j], candTokens, selectedTokens[j]);+ if (sim > maxOverlap) maxOverlap = sim;+ }+ const mmr = lambda * cand.score - (1 - lambda) * maxOverlap;+ if (mmr > bestMmr) { bestMmr = mmr; bestIdx = i; }+ }+ if (bestIdx < 0) break;+ const [chosen] = remaining.splice(bestIdx, 1);+ selected.push(chosen);+ selectedTokens.push(tokenize(chosen.candidate.content));+ selectedEmbeddings.push(chosen.candidate.embedding);+ }+ return selected;+}++describe('mmrRerank lazy-tokenization refactor — correctness parity', () => {+ const scenarios: Array<[string, 'all' | 'none' | 'half']> = [+ ['all candidates embedded (common production case)', 'all'],+ ['no candidates embedded (pure Jaccard fallback)', 'none'],+ ['half embedded, half not (mixed fallback)', 'half'],+ ];++ for (const [label, embedded] of scenarios) {+ it(`produces byte-identical selection order to the frozen baseline: ${label}`, () => {+ const cands = corpus(24, { embedded });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));++ const baseline = baselineMmrRerank(scored, 0.7, 15);+ const candidate = applyMMR(scored, 0.7, 15);++ expect(candidate.map((s) => s.candidate.id)).toEqual(baseline.map((s) => s.candidate.id));+ expect(candidate.map((s) => s.score)).toEqual(baseline.map((s) => s.score));+ });+ }++ it('handles a single candidate (early-return path) identically', () => {+ const cands = corpus(1, { embedded: 'all' });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));+ expect(applyMMR(scored, 0.7, 5)).toEqual(baselineMmrRerank(scored, 0.7, 5));+ });++ it('handles lambda extremes (pure relevance, pure diversity) identically', () => {+ const cands = corpus(20, { embedded: 'half' });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));+ for (const lambda of [0, 1]) {+ const baseline = baselineMmrRerank(scored, lambda, 12);+ const candidate = applyMMR(scored, lambda, 12);+ expect(candidate.map((s) => s.candidate.id)).toEqual(baseline.map((s) => s.candidate.id));+ }+ });+});++describe('mmrRerank lazy-tokenization refactor — tokenize() call count', () => {+ it('all-embedded corpus (N=30, limit=20): baseline ~551 calls, candidate 0 calls', () => {+ const cands = corpus(30, { embedded: 'all' });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));++ const spy = vi.spyOn(smartRetrieval, 'tokenize');+ spy.mockClear();+ baselineMmrRerank(scored, 0.7, 20);+ const baselineCalls = spy.mock.calls.length;++ spy.mockClear();+ applyMMR(scored, 0.7, 20);+ const candidateCalls = spy.mock.calls.length;+ spy.mockRestore();++ // Matches the arch-reviewer's estimate: 19 outer passes over a shrinking+ // remaining-candidate list, plus 1 seed tokenization (measured: 400).+ expect(baselineCalls).toBeGreaterThanOrEqual(400);+ expect(candidateCalls).toBe(0);+ });++ it('mixed corpus (N=30, half embedded): candidate calls scale O(N), not O(N x limit)', () => {+ const cands = corpus(30, { embedded: 'half' });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));++ const spy = vi.spyOn(smartRetrieval, 'tokenize');+ spy.mockClear();+ baselineMmrRerank(scored, 0.7, 20);+ const baselineCalls = spy.mock.calls.length;++ spy.mockClear();+ applyMMR(scored, 0.7, 20);+ const candidateCalls = spy.mock.calls.length;+ spy.mockRestore();++ expect(baselineCalls).toBeGreaterThan(candidateCalls * 3);+ // Cached per-candidate: at most N unique tokenizations regardless of+ // how many outer passes/comparisons touch a given candidate.+ expect(candidateCalls).toBeLessThanOrEqual(30);+ });++ it('no-embedding corpus: candidate calls tokenize at most once per candidate (caching works)', () => {+ const cands = corpus(15, { embedded: 'none' });+ const scored = cands.map((c) => ({ candidate: c, score: c.score }));++ const spy = vi.spyOn(smartRetrieval, 'tokenize');+ spy.mockClear();+ applyMMR(scored, 0.7, 10);+ const candidateCalls = spy.mock.calls.length;+ spy.mockRestore();++ expect(candidateCalls).toBeLessThanOrEqual(15);+ });+});diff --git a/v3/@claude-flow/memory/src/smart-retrieval.ts b/v3/@claude-flow/memory/src/smart-retrieval.tsindex 4eb2f1faea..276db20013 100644--- a/v3/@claude-flow/memory/src/smart-retrieval.ts+++ b/v3/@claude-flow/memory/src/smart-retrieval.ts@@ -279,15 +279,28 @@ export function applyMMR<T extends SearchCandidate>(function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {if (scored.length <= 1) return scored.slice(0, limit);+ // Lazy, per-candidate token cache: PR #3169 made embedding-cosine the+ // common `pairSimilarity` path, so eager per-outer-pass tokenization+ // (the prior implementation) mostly computed tokens that were then+ // discarded. Tokens are now computed at most once per candidate, only+ // when the Jaccard fallback actually needs them (Dream Cycle 2026-09-10).+ const tokenCache = new Map<SearchCandidate, Set<string>>();+ const getTokens = (item: Scored): Set<string> => {+ let tokens = tokenCache.get(item.candidate);+ if (!tokens) {+ tokens = tokenize(item.candidate.content);+ tokenCache.set(item.candidate, tokens);+ }+ return tokens;+ };+const selected: Scored[] = [];const remaining = [...scored];- const selectedTokens: Set<string>[] = [];const selectedEmbeddings: Array<number[] | undefined> = [];// Seed with the top-scored candidate.const first = remaining.shift()!;selected.push(first);- selectedTokens.push(tokenize(first.candidate.content));selectedEmbeddings.push(first.candidate.embedding);while (selected.length < limit && remaining.length > 0) {@@ -296,11 +309,10 @@ function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {for (let i = 0; i < remaining.length; i++) {const cand = remaining[i];- const candTokens = tokenize(cand.candidate.content);const candEmbedding = cand.candidate.embedding;let maxOverlap = 0;- for (let j = 0; j < selectedTokens.length; j++) {- const sim = pairSimilarity(candEmbedding, selectedEmbeddings[j], candTokens, selectedTokens[j]);+ for (let j = 0; j < selected.length; j++) {+ const sim = pairSimilarity(candEmbedding, selectedEmbeddings[j], cand, selected[j], getTokens);if (sim > maxOverlap) maxOverlap = sim;}const mmr = lambda * cand.score - (1 - lambda) * maxOverlap;@@ -313,19 +325,19 @@ function mmrRerank(scored: Scored[], lambda: number, limit: number): Scored[] {if (bestIdx < 0) break;const [chosen] = remaining.splice(bestIdx, 1);selected.push(chosen);- selectedTokens.push(tokenize(chosen.candidate.content));selectedEmbeddings.push(chosen.candidate.embedding);}return selected;}-/** Cosine similarity when both embeddings exist, agree in dimension, and are well-formed; token-Jaccard otherwise. */+/** Cosine similarity when both embeddings exist, agree in dimension, and are well-formed; token-Jaccard (computed lazily via `getTokens`) otherwise. */function pairSimilarity(embA: number[] | undefined,embB: number[] | undefined,- tokensA: Set<string>,- tokensB: Set<string>+ a: Scored,+ b: Scored,+ getTokens: (item: Scored) => Set<string>): number {if (isWellFormedEmbedding(embA) &&@@ -334,7 +346,7 @@ function pairSimilarity() {return cosineSimilarity(embA, embB);}- return jaccard(tokensA, tokensB);+ return jaccard(getTokens(a), getTokens(b));}/**@@ -365,7 +377,13 @@ function cosineSimilarity(a: number[], b: number[]): number {return dot / (Math.sqrt(normA) * Math.sqrt(normB));}-function tokenize(text: string): Set<string> {+/**+ * Exported (this package's `exports` map is a `"./*"` wildcard, so this is+ * real public API, not just test access) so the mmrRerank tokenize-call-+ * count regression test can spy on it (Dream Cycle 2026-09-10). Pure and+ * side-effect-free, so widening its visibility carries no behavioral risk.+ */+export function tokenize(text: string): Set<string> {return new Set(text.toLowerCase()