π The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
dream(performance): #3093 wire productQuantizeDistance() into HNSW search path (evaluated, ACCEPT) (#3094)
rUv2d6227005e37
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.mdindex a99a186b44..e6bd5b207f 100644--- a/docs/dream-cycle/LEDGER.md+++ b/docs/dream-cycle/LEDGER.md@@ -159,8 +159,8 @@ finding" to an explicit recommendation: a future `automation`/`meta` DEEPnight should treat "harden STEP 25's ledger-append against silent failure"as a first-class, high-priority candidate, not a background note.-| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable β no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false β weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | 08-16,17,18,19 MERGED (4/7); 08-14,15 correctly unmerged REJECT evidence; 08-20..23 confirmed real gap (no branch/PR, not a ledger-append failure) |-| 2026-08-25 | performance | productQuantizeDistance() implemented but never dispatched in HNSW search path β wired in | #3093 | #3094 | yes | ACCEPT | see issue #3093 | n/a (backfill) | unmerged draft as of 09-05 |+| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable β no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false β weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | MERGED 2026-09-14 |+| 2026-08-25 | performance | HNSW product-quantization: productQuantizeDistance() correctly implemented but zero callers β generic distance ran on raw PQ centroid indices, numerically meaningless; now dispatched, with a disclosed residual isValidPQEncoding() plausibility-heuristic caveat | #3093 | #3094 | yes | ACCEPT | recall@10 0.097 (pre-fix) β 0.270 (post-fix) on deterministic synthetic corpus, ~2.8x; 461/462 tests, 0 regressions | 300da158... | awaiting final CI/merge as of 2026-09-14 || 2026-08-26 | security | authorizeMcpTool() trusted unsigned caller identity (ASI07) β ADR-377 verifyInvocationToken bound into live chokepoint | #3102 | #3103 | yes | ACCEPT-with-caveats | see issue #3102 | n/a (backfill) | unmerged draft as of 09-05 || 2026-08-27 | intelligence | distillLearning() EWC gate read 1 of 384 Fisher dims via length-collapsed getPenalty() call; rewired to computeConfidencePenalty()/updateFisherFromConfidences() | #3109 | #3110 | yes | ACCEPT-scoped | 152/152 tests (+4 new), baseline-fails/candidate-passes via stash isolation | 89946458... | unmerged draft as of 09-05 || 2026-08-28 | memory | HybridBackend's dead `weights` field now drives real weighted-RRF fusion (was computed, never read by merge helpers) | #3118 | #3119 | yes | ACCEPT | see issue #3118 | n/a (backfill) | unmerged draft as of 09-05 |diff --git a/docs/dream-cycle/dream-gist-2026-08-25.md b/docs/dream-cycle/dream-gist-2026-08-25.mdnew file mode 100644index 0000000000..cc783efe79--- /dev/null+++ b/docs/dream-cycle/dream-gist-2026-08-25.md@@ -0,0 +1,78 @@+# Performance SOTA Report β 2026-08-25++TL;DR: Tonight's performance deep-dive found that Ruflo's HNSW product-quantization (PQ) feature (`quantization: { type: 'product' }`) has been silently broken since it was written: `Quantizer.productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance (`hnsw-index.ts:1209-1360`), but the actual search path (`distance()`/`distanceOptimized()`) never called `productQuantizeDistance()` β it ran generic cosine/euclidean/dot/manhattan distance directly on raw PQ centroid-index arrays, which is numerically meaningless (index 5 vs. index 200 has no relation to embedding-space proximity). This is the same "computed-but-never-wired" bug class this pipeline has repeatedly shipped (2026-08-18 hybridSearch, 2026-08-19 MessageBus retry, 2026-08-24 weightedConsensus). Fix: dispatch to `productQuantizeDistance()` when PQ is active and codebooks are trained, guarded by a new `isValidPQEncoding()` check to avoid crashing/corrupting on vectors added before codebook training completes (a separate, disclosed, pre-existing limitation this patch does not fully solve). Measured on a deterministic synthetic clustered corpus: recall@10 recovers from 0.097 (pre-fix) to 0.270 (post-fix) β a real, reproducible ~2.8x improvement β while remaining honest that flat PQ trained globally on multi-modal data has a known ceiling well below "near-perfect," which this fix does not and should not claim to lift.++## What's New in 2026++| Finding | Source | Confidence |+|---|---|---|+| Two independent SIGMOD 2026 papers (Ada-ef, DARTH) converge on per-query *declarative-recall* adaptive `efSearch` as the fix for the exact static-ef recall/latency tradeoff that killed 2026-08-15's rejected candidate β strongest external evidence of the night, but scoped out of tonight's candidate for patch-size/risk reasons (see Recommended Next Steps) | arXiv:2512.06636 (SIGMOD 2026), arXiv:2505.19001 + github.com/MChatzakis/DARTH (SIGMOD 2026) | A |+| CVE-2026-25536: `@modelcontextprotocol/sdk` cross-client data leak via `StreamableHTTPServerTransport` reuse (1.10.0-1.25.3, fixed 1.26.0) β checked against Ruflo, not exploitable here (stdio transport, and `ruflo/src/ruvocal` already pinned to the fixed version) | GHSA-345p-7cg4-v4c7, cross-checked SentinelOne/Tenable/GitLab | A |+| OWASP Top 10 for Agentic Applications 2026 formalizes ASI07 "Insecure Inter-Agent Communication" β directly relevant to a dormant Ruflo control found tonight (see Scan Findings: security) | genai.owasp.org | A |+| SWARM+ (arXiv:2603.19431) validates at 990-agent scale that federation-style consensus quorums must shrink as members go stale/unreachable β directly on-point for a new hive-mind scan finding tonight | arXiv:2603.19431 | B |+| No production multi-agent framework (LangGraph, AutoGen/AG2, CrewAI, OpenAI Agents SDK) ships queue-aware/cost-aware model routing as a first-party feature β a genuine, well-documented open gap (layering convention, not a solved-but-unpublished capability), per InfraMIND (arXiv:2606.11440) | arXiv:2606.11440 + 4 official framework docs, checked 2026-08-25 | B |++## Ruflo Current Capability++`v3/@claude-flow/memory/src/hnsw-index.ts`: `Quantizer` supports `binary`/`scalar`/`product` quantization types. For `product`, `productQuantize()` (1209-1269) trains PQ codebooks via k-means and stores centroid indices; `productQuantizeDistance()` (1337-1360, pre-fix) correctly implements codebook-aware distance β but had **zero callers** anywhere in `v3/` (confirmed via repo-wide grep) prior to tonight. `addPoint()`/`search()` stored/queried via `this.quantizer.encode()` but then ran the generic `distance()` switch (cosine/euclidean/dot/manhattan) directly on the resulting centroid-index arrays, and even normalized those index arrays for the cosine fast-path β both operations meaningless on PQ codes. `type: 'product'` appeared in exactly one test fixture repo-wide and was exercised by zero recall/correctness tests before tonight.++Also confirmed, unrelated to tonight's fix: `binary`/`scalar` quantization have a related-but-distinct issue β `scalarQuantize()`'s output is `[min, range, ...levels]` (the prefix pollutes generic distance the same way PQ indices do), and `binaryQuantize()`'s packed-bit floats aren't cosine/Euclidean/dot-compatible either. **Disclosed, not fixed tonight** β kept out of scope to keep this patch one conceptual change (see Recommended Next Steps).++## Competitor Comparison++| Framework | Caching / KV-cache reuse | Adaptive/dynamic topology or scheduling | Queue-aware / cost-aware routing | Published perf benchmarks |+|---|---|---|---|---|+| LangGraph (v1.1.3) | Node-level output cache (`CachePolicy`, TTL) β not KV-cache | Conditional edges are task-logic-driven, not latency/cost-driven | None | Conflicting single-source claims (2.2x vs. 5.75x vs. losing to CrewAI) β B/C, not reproducible from a shared harness |+| AutoGen/AG2 | `cache_seed` exact-match completion cache | `GroupChat` speaker selection is who-talks-next logic, not a perf scheduler | None (cost/token tracking exists, unused for routing) | "8-9x token-efficiency vs LangChain" (B, academic cross-framework study); no first-party latency benchmark |+| CrewAI | Per-tool `cache_function` (developer-defined) | `Flows` branching is static, developer-defined | None β docs point to bolting on LiteLLM/Requesty | Vendor-adjacent "5.76x faster than LangGraph" contradicts the LangGraph row's own claim β neither independently reproducible (C) |+| OpenAI Agents SDK | Platform-level prompt caching (0.1x read cost, β₯1024 tokens) β real and quantified (A) but API-layer, not SDK-specific | `Handoffs` are developer-authored, not adaptive | Load-balancing exists but is infra-level, invisible to orchestration logic | OpenHands' agent SDK measured 38.8% cache-hit rate vs. reference `codex` agent's 94.8% β a **2.7-3.4x cost inflation from one missing `prompt_cache_key` parameter** (B, concrete cautionary data point) |++The "None" pattern for queue-aware/cost-aware routing is a genuine open gap, not solved-but-unpublished: InfraMIND (arXiv:2606.11440) reports +7.6pp accuracy / 7x lower latency / 99.9% vs <50% SLO compliance when routing is made infra-aware. The gap exists because all four frameworks deliberately push load-balancing/cache-affinity/cost signals down to a gateway layer (LiteLLM, vLLM Router, provider backend) to stay provider-portable β a real design tradeoff, not an oversight. Real opportunity for Ruflo, but the OpenHands `prompt_cache_key` incident is a live warning: caching is trivial to silently break in a multi-agent context, so any cache-aware routing claim needs a hit-rate-regression benchmark per release, not just a feature flag.++## Hypothesis++> Given an HNSWIndex configured with `quantization: { type: 'product' }`, where `productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance but are never invoked by the search path, when `distance()`/`distanceOptimized()` are patched to dispatch to `quantizer.productQuantizeDistance()` whenever product quantization is active and codebooks are trained (falling back to today's generic distance during the pre-training bootstrap window, unchanged), and vector normalization is skipped for product-quantized vectors, then recall@10 of a product-quantized HNSWIndex should recover from the current near-chance-level baseline to a level materially closer to the unquantized index's recall@10, subject to: (1) non-`'product'` quantization paths are byte-identical to today's; (2) existing HNSW/memory test suite remains green; (3) `getCompressionRatio()` is unchanged; (4) zero LLM/API cost.++Frozen before evaluation began; not modified after seeing results.++## Benchmarks++Deterministic, $0, zero-LLM vitest additions: `v3/@claude-flow/memory/src/hnsw-quantization.test.ts`. Synthetic clustered corpus (8 well-separated Gaussian-ish blobs, N=1500, dim=64, seeded xorshift32 PRNG for full reproducibility), 30 held-out queries, recall@10 against brute-force ground truth computed on the *original unquantized* vectors (never leaked into the candidate).++Disclosed for transparency: initial corpus parameters (N=400, `codebookSize=16`) were adjusted twice β N=400 put an unrepresentatively large fraction of the corpus inside the PQ pre-training bootstrap window; `codebookSize=16` was arbitrarily small. `codebookSize=256` is the code's actual documented default, not a tuned value. The pass-bar (`>= 0.25`) was set *after* measuring both baseline and candidate at the final N=1500/codebookSize=256 config, disclosed in the test file's comment with both real numbers.++Darwin (`metaharness-darwin evolve --bench <suite.json>`) confirmed out of scope again tonight: its real interface evolves routing/topology/prompt/memory/tool/tier/context/coordination genome parameters against an LLM-scored bench corpus β no analog for a scoped TS distance-dispatch fix. Same class of skip as 4 of the last 5 dream-cycle nights.++## Evaluation++**evaluated: accepted.**++Baseline captured via `git stash` on `hnsw-index.ts` only (test file kept in place):++| Config | Recall@10 |+|---|---|+| Unquantized index (sanity check on corpus/harness) | 1.000 |+| Product-quantized, **pre-fix** (generic distance on raw PQ indices) | **0.097** |+| Product-quantized, **post-fix** (dispatched to `productQuantizeDistance`) | **0.270** |++A ~2.8x recall@10 improvement, fully reproducible (all seeds fixed). A diagnostic run (not part of the committed suite) isolated PQ metric quality from HNSW graph-traversal quality: brute-force search using `productQuantizeDistance()` directly gave the *same* 0.270 as the full graph search, confirming the fix's ceiling is genuine PQ fidelity (flat PQ trained globally under-resolves fine-grained same-cluster ranking on multi-modal data β a real, known PQ characteristic, not a new bug), not an HNSW graph-construction artifact compounding on top of it.++Full package suite: 461/462 passing pre- and post-fix (the 1 failure, `auto-memory-bridge.test.ts`'s read-only-file error-handling test, is environmental β this sandbox runs as root, which bypasses the `chmod`-based read-only precondition the test depends on; confirmed unrelated to this diff by identical failure on baseline). `tsc --noEmit` shows zero new errors attributable to this diff (all remaining errors are pre-existing `@types/node`/environment noise unrelated to `hnsw-index.ts`'s logic).++**Independent adversarial critic** (fresh session, no authoring context) independently re-derived the 0.097β0.270 numbers via its own `git stash` isolation (confirmed exact match), re-ran the full suite, checked the `Uint8Array`β`Float32Array` signature change against every caller in `v3/` (found one, `@claude-flow/cli/__tests__/pq-validation.test.ts`, unaffected β 7/7 tests pass, `any`-typed accessor), and generalization-tested beyond the shipped corpus shape (128d/16 clusters, 16d/4 clusters, 64d/1 cluster β improvement direction and rough magnitude held in all three, pre-fix 0.068-0.088 β post-fix 0.264-0.296). **Verdict: CONFIRMED-WITH-CAVEATS.** The one real caveat: `isValidPQEncoding()` is a *plausibility heuristic, not a proof of provenance* β the critic built a reproduction where an all-zero vector added pre-training passes the guard and gets silently (not-crashing, not-correctly-falling-back) mis-dispatched as if it were a real centroid assignment. This is a narrower manifestation of the same disclosed pre-training-fallback limitation, not a new hole; closing it fully needs per-node encoding-provenance tracking (out of scope tonight). Disclosed in-code and here rather than treated as fully solved.++## Darwin Results++Skipped β scope mismatch (see Benchmarks). Consistent with 08-14, 08-15, 08-18, 08-19, 08-24.++## SOTA Proof & Witness++(Populated in the Witness section of tonight's issue/PR; see below.)++## Recommended Next Steps++1. **Merge this candidate** (human review required) β closes a real, zero-cost correctness bug in a documented compression feature, with a permanent deterministic regression test.+2. **Follow up on 2026-08-15's rejected `efSearch` finding using tonight's Grade-A evidence** (Ada-ef/DARTH, SIGMOD 2026): per-query declarative-recall adaptive `efSearch`, materially different and better-evidenced than a `sqrt(N)` heuristic. Scoped out tonight for patch-size/risk (needs a calibration mechanism plus `deserialize()` persistence) β strongest lead of the night, deserves a dedicated future night.+3. **Wire `verifyInvocationToken` into `AgentAuthorizationPropagator.checkToolCall`** (`v3/@claude-flow/security/src/mcp-caller-identity.ts`) β an ADR-377 Ed25519 caller-identity control dormant since written, freshly relevant given OWASP's 2026 ASI07 category. Small, deterministic, env-gated.+4. **Fix `FederationHub.vote()`'s stale-vote/quorum-denominator mismatch** (`v3/@claude-flow/swarm/src/federation-hub.ts:700-728`) β a departed swarm's vote is never purged while the quorum denominator shrinks. Independent of 08-24's weightedConsensus fix, anchored by SWARM+ (arXiv:2603.19431).+5. **Harden or extend `scalar`/`binary` quantization distance dispatch** β same bug class as tonight's fix, disclosed but not fixed.diff --git a/v3/@claude-flow/memory/src/hnsw-index.ts b/v3/@claude-flow/memory/src/hnsw-index.tsindex 881d835826..5c5cf9a761 100644--- a/v3/@claude-flow/memory/src/hnsw-index.ts+++ b/v3/@claude-flow/memory/src/hnsw-index.ts@@ -270,8 +270,11 @@ export class HNSWIndex extends EventEmitter {? this.quantizer.encode(vector): vector;- // Pre-normalize vector for O(1) cosine similarity- const normalizedVector = this.config.metric === 'cosine'+ // Pre-normalize vector for O(1) cosine similarity.+ // Skipped for product quantization: storedVector holds PQ centroid+ // indices, not embedding coordinates, so normalizing it is meaningless+ // (and productQuantizeDistance() below never consults normalizedVector).+ const normalizedVector = this.config.metric === 'cosine' && !this.isProductQuantized()? this.normalizeVector(storedVector): null;@@ -335,8 +338,9 @@ export class HNSWIndex extends EventEmitter {? this.quantizer.encode(query): query;- // Pre-normalize query for O(1) cosine similarity- const normalizedQuery = this.config.metric === 'cosine'+ // Pre-normalize query for O(1) cosine similarity (see addPoint() for why+ // this is skipped under product quantization).+ const normalizedQuery = this.config.metric === 'cosine' && !this.isProductQuantized()? this.normalizeVector(queryVector): null;@@ -993,7 +997,49 @@ export class HNSWIndex extends EventEmitter {}}+ /**+ * True when this index uses product quantization with trained codebooks.+ * Stored vectors are then PQ centroid indices, not embedding coordinates β+ * generic metric distances (cosine/euclidean/dot/manhattan) computed+ * directly on those indices are meaningless (index 5 vs index 200 has no+ * relation to embedding-space proximity). Falls back to false during the+ * pre-training bootstrap window (see Quantizer.productQuantize()), where+ * stored vectors are still sub-vector-mean floats and generic distance,+ * while imprecise, is not nonsensical β matching prior behavior exactly.+ */+ private isProductQuantized(): boolean {+ return (+ this.quantizer !== null &&+ this.config.quantization?.type === 'product' &&+ this.quantizer.isPQTrained+ );+ }+private distance(a: Float32Array, b: Float32Array): number {+ // Pre-existing, disclosed limitation (not introduced by this fix): any+ // vector added before PQ codebook training completes is stored under a+ // sub-vector-mean fallback encoding (see Quantizer.productQuantize())+ // and is never re-encoded once training finishes β the original raw+ // vector isn't retained. isValidPQEncoding() is a *plausibility*+ // heuristic, not a proof of provenance: it structurally guards the+ // crash/out-of-bounds case (fallback values outside codebook range, or+ // non-integer), but a fallback encoding whose sub-vector means+ // coincidentally land on small non-negative integers (most concretely,+ // an all-zero or otherwise degenerate vector added pre-training) can+ // still pass the check and be silently mis-dispatched to+ // productQuantizeDistance() as if it were a real centroid assignment.+ // Closing this fully would need per-node encoding-provenance tracking+ // (e.g. a sentinel/flag threaded through HNSWNode and its persistence+ // format), out of scope for this patch β flagged as a known residual+ // risk, not treated as fully solved.+ if (+ this.isProductQuantized() &&+ this.quantizer!.isValidPQEncoding(a) &&+ this.quantizer!.isValidPQEncoding(b)+ ) {+ return this.quantizer!.productQuantizeDistance(a, b);+ }+switch (this.config.metric) {case 'cosine':return this.cosineDistance(a, b);@@ -1312,14 +1358,17 @@ class Quantizer {* Compute approximate squared Euclidean distance between two PQ-encoded* vectors using their centroid indices and the shared codebooks.*- * @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers)+ * @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers).+ * Typed Float32Array to match productQuantize()'s actual return type β+ * indices are small integers (0..codebookSize-1) exactly representable+ * in float32.* @param encoded2 - Centroid indices for vector 2 (length = numSubquantizers)* @param codebooks - Trained codebooks from trainProductQuantizer()* @returns Approximate squared Euclidean distance*/productQuantizeDistance(- encoded1: Uint8Array,- encoded2: Uint8Array,+ encoded1: Float32Array,+ encoded2: Float32Array,codebooks?: number[][][],): number {const cb = codebooks || this.codebooks;@@ -1412,6 +1461,23 @@ class Quantizer {getCodebooks(): number[][][] | null {return this.codebooks;}++ /**+ * True when `encoded` looks like a real trained-codebook PQ encoding+ * (length matches numSubquantizers, every value is a non-negative integer+ * in range for its sub-quantizer's codebook). False for the pre-training+ * sub-vector-mean fallback (see productQuantize()), which cannot be+ * compared via productQuantizeDistance() without corrupting results or+ * indexing out of bounds.+ */+ isValidPQEncoding(encoded: Float32Array): boolean {+ if (!this.codebooks || encoded.length !== this.codebooks.length) return false;+ for (let m = 0; m < encoded.length; m++) {+ const idx = encoded[m];+ if (!Number.isInteger(idx) || idx < 0 || idx >= this.codebooks[m].length) return false;+ }+ return true;+ }}// ===== Persistence Helpers (ADR-125 Phase 3) =====diff --git a/v3/@claude-flow/memory/src/hnsw-quantization.test.ts b/v3/@claude-flow/memory/src/hnsw-quantization.test.tsnew file mode 100644index 0000000000..0b81522bea--- /dev/null+++ b/v3/@claude-flow/memory/src/hnsw-quantization.test.ts@@ -0,0 +1,170 @@+/**+ * Dream Cycle 2026-08-25 β HNSW product-quantization distance dispatch.+ *+ * `Quantizer.productQuantize()` / `productQuantizeDistance()` correctly+ * implement codebook-aware PQ encoding and distance, but nothing in the+ * search path (`distance()` / `distanceOptimized()`) ever called+ * `productQuantizeDistance()` β generic cosine/euclidean/dot/manhattan+ * distance was computed directly on raw PQ centroid-index arrays, which is+ * numerically meaningless (adjacent centroid indices are not adjacent in+ * embedding space). This test builds a synthetic clustered corpus, computes+ * brute-force ground-truth top-10 neighbors on the *unquantized* vectors,+ * and asserts recall@10 for a product-quantized index stays well above+ * chance level. On the pre-fix code this assertion fails (recall collapses+ * toward chance); post-fix it passes.+ */++import { describe, it, expect } from 'vitest';+import { HNSWIndex } from './hnsw-index.js';++function xorshift32(seed: number): () => number {+ let s = seed || 1;+ return () => {+ s ^= s << 13;+ s ^= s >>> 17;+ s ^= s << 5;+ return ((s >>> 0) / 4294967296);+ };+}++/** Deterministic clustered corpus: `numClusters` well-separated Gaussian blobs. */+function buildClusteredCorpus(+ n: number,+ dim: number,+ numClusters: number,+ seed: number,+): Float32Array[] {+ const rand = xorshift32(seed);+ const centers: Float32Array[] = [];+ for (let c = 0; c < numClusters; c++) {+ const center = new Float32Array(dim);+ for (let d = 0; d < dim; d++) {+ // Spread centers far apart so clusters are well-separated.+ center[d] = (rand() - 0.5) * 20;+ }+ centers.push(center);+ }++ const vectors: Float32Array[] = [];+ for (let i = 0; i < n; i++) {+ const center = centers[i % numClusters];+ const v = new Float32Array(dim);+ for (let d = 0; d < dim; d++) {+ v[d] = center[d] + (rand() - 0.5) * 1.5; // small intra-cluster noise+ }+ vectors.push(v);+ }+ return vectors;+}++function bruteForceTopK(+ query: Float32Array,+ corpus: Float32Array[],+ ids: string[],+ k: number,+): Set<string> {+ const scored = corpus.map((v, i) => {+ let sum = 0;+ for (let d = 0; d < v.length; d++) {+ const diff = v[d] - query[d];+ sum += diff * diff;+ }+ return { id: ids[i], dist: sum };+ });+ scored.sort((a, b) => a.dist - b.dist);+ return new Set(scored.slice(0, k).map((s) => s.id));+}++async function recallAtK(config: {+ dim: number;+ n: number;+ numClusters: number;+ numQueries: number;+ k: number;+ quantization?: { type: 'product'; subquantizers: number; codebookSize: number };+}): Promise<number> {+ const corpus = buildClusteredCorpus(config.n, config.dim, config.numClusters, 42);+ const ids = corpus.map((_, i) => `v-${i}`);++ const index = new HNSWIndex({+ dimensions: config.dim,+ metric: 'euclidean',+ M: 16,+ efConstruction: 100,+ ...(config.quantization ? { quantization: config.quantization } : {}),+ });++ for (let i = 0; i < corpus.length; i++) {+ await index.addPoint(ids[i], corpus[i]);+ }++ const rand = xorshift32(1337);+ let totalRecall = 0;+ for (let q = 0; q < config.numQueries; q++) {+ const queryIdx = Math.floor(rand() * config.n);+ const query = corpus[queryIdx];++ const groundTruth = bruteForceTopK(query, corpus, ids, config.k);+ const results = await index.search(query, config.k, 200);+ const returned = new Set(results.map((r) => r.id));++ let hits = 0;+ for (const id of returned) if (groundTruth.has(id)) hits++;+ totalRecall += hits / config.k;+ }++ return totalRecall / config.numQueries;+}++describe('HNSWIndex product quantization β distance dispatch (Dream Cycle 2026-08-25)', () => {+ const DIM = 64;+ // pqTrainingThreshold is 256; N well above it keeps the pre-training+ // bootstrap window (see isValidPQEncoding()) a small fraction of the+ // corpus, representative of real usage rather than a pathological+ // near-100%-cold-start case.+ const N = 1500;+ const NUM_CLUSTERS = 8;+ const NUM_QUERIES = 30;+ const K = 10;++ it('unquantized baseline recovers near-perfect recall@10 (sanity check on the corpus/harness itself)', async () => {+ const recall = await recallAtK({ dim: DIM, n: N, numClusters: NUM_CLUSTERS, numQueries: NUM_QUERIES, k: K });+ // eslint-disable-next-line no-console+ console.log(`[dream-cycle] unquantized recall@10 = ${recall.toFixed(3)}`);+ expect(recall).toBeGreaterThan(0.85);+ });++ it('product-quantized index recall@10 recovers materially vs. the pre-fix baseline', async () => {+ const recall = await recallAtK({+ dim: DIM,+ n: N,+ numClusters: NUM_CLUSTERS,+ numQueries: NUM_QUERIES,+ k: K,+ quantization: { type: 'product', subquantizers: 8, codebookSize: 256 },+ });+ // eslint-disable-next-line no-console+ console.log(`[dream-cycle] product-quantized recall@10 = ${recall.toFixed(3)}`);+ // Measured on this exact deterministic corpus (git stash on hnsw-index.ts+ // only, test file kept in place):+ // pre-fix (generic distance on raw PQ centroid indices): recall@10 = 0.097+ // post-fix (productQuantizeDistance dispatched correctly): recall@10 = 0.270+ // Flat PQ trained globally on multi-modal (clustered) data is known to+ // under-resolve fine-grained same-cluster ranking β recall@10 in the+ // 0.2-0.3 range here reflects that real PQ characteristic, not a bug;+ // the assertion checks the fix's effect (recall recovers well above the+ // measured buggy baseline), not an absolute "good PQ" bar.+ expect(recall).toBeGreaterThanOrEqual(0.25);+ });++ it('getCompressionRatio() is unaffected by the distance-dispatch fix', () => {+ const index = new HNSWIndex({+ dimensions: DIM,+ metric: 'euclidean',+ quantization: { type: 'product', subquantizers: 8, codebookSize: 16 },+ });+ // 8 subquantizers -> 8x compression, unchanged by this fix (only the+ // distance computation changed, not the stored representation).+ expect((index as unknown as { quantizer: { getCompressionRatio(): number } }).quantizer.getCompressionRatio()).toBe(8);+ });+});