Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

dream(performance): #3093 wire productQuantizeDistance() into HNSW search path (evaluated, ACCEPT) (#3094)

rUv2d6227005e37
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md
index a99a186b44..e6bd5b207f 100644
--- a/docs/dream-cycle/LEDGER.md
+++ b/docs/dream-cycle/LEDGER.md
@@ -159,8 +159,8 @@ finding" to an explicit recommendation: a future `automation`/`meta` DEEP
night should treat "harden STEP 25's ledger-append against silent failure"
as a first-class, high-priority candidate, not a background note.
-| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable β€” no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false β†’ weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | 08-16,17,18,19 MERGED (4/7); 08-14,15 correctly unmerged REJECT evidence; 08-20..23 confirmed real gap (no branch/PR, not a ledger-append failure) |
-| 2026-08-25 | performance | productQuantizeDistance() implemented but never dispatched in HNSW search path β€” wired in | #3093 | #3094 | yes | ACCEPT | see issue #3093 | n/a (backfill) | unmerged draft as of 09-05 |
+| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable β€” no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false β†’ weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | MERGED 2026-09-14 |
+| 2026-08-25 | performance | HNSW product-quantization: productQuantizeDistance() correctly implemented but zero callers β€” generic distance ran on raw PQ centroid indices, numerically meaningless; now dispatched, with a disclosed residual isValidPQEncoding() plausibility-heuristic caveat | #3093 | #3094 | yes | ACCEPT | recall@10 0.097 (pre-fix) β†’ 0.270 (post-fix) on deterministic synthetic corpus, ~2.8x; 461/462 tests, 0 regressions | 300da158... | awaiting final CI/merge as of 2026-09-14 |
| 2026-08-26 | security | authorizeMcpTool() trusted unsigned caller identity (ASI07) β€” ADR-377 verifyInvocationToken bound into live chokepoint | #3102 | #3103 | yes | ACCEPT-with-caveats | see issue #3102 | n/a (backfill) | unmerged draft as of 09-05 |
| 2026-08-27 | intelligence | distillLearning() EWC gate read 1 of 384 Fisher dims via length-collapsed getPenalty() call; rewired to computeConfidencePenalty()/updateFisherFromConfidences() | #3109 | #3110 | yes | ACCEPT-scoped | 152/152 tests (+4 new), baseline-fails/candidate-passes via stash isolation | 89946458... | unmerged draft as of 09-05 |
| 2026-08-28 | memory | HybridBackend's dead `weights` field now drives real weighted-RRF fusion (was computed, never read by merge helpers) | #3118 | #3119 | yes | ACCEPT | see issue #3118 | n/a (backfill) | unmerged draft as of 09-05 |
diff --git a/docs/dream-cycle/dream-gist-2026-08-25.md b/docs/dream-cycle/dream-gist-2026-08-25.md
new file mode 100644
index 0000000000..cc783efe79
--- /dev/null
+++ b/docs/dream-cycle/dream-gist-2026-08-25.md
@@ -0,0 +1,78 @@
+# Performance SOTA Report β€” 2026-08-25
+
+TL;DR: Tonight's performance deep-dive found that Ruflo's HNSW product-quantization (PQ) feature (`quantization: { type: 'product' }`) has been silently broken since it was written: `Quantizer.productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance (`hnsw-index.ts:1209-1360`), but the actual search path (`distance()`/`distanceOptimized()`) never called `productQuantizeDistance()` β€” it ran generic cosine/euclidean/dot/manhattan distance directly on raw PQ centroid-index arrays, which is numerically meaningless (index 5 vs. index 200 has no relation to embedding-space proximity). This is the same "computed-but-never-wired" bug class this pipeline has repeatedly shipped (2026-08-18 hybridSearch, 2026-08-19 MessageBus retry, 2026-08-24 weightedConsensus). Fix: dispatch to `productQuantizeDistance()` when PQ is active and codebooks are trained, guarded by a new `isValidPQEncoding()` check to avoid crashing/corrupting on vectors added before codebook training completes (a separate, disclosed, pre-existing limitation this patch does not fully solve). Measured on a deterministic synthetic clustered corpus: recall@10 recovers from 0.097 (pre-fix) to 0.270 (post-fix) β€” a real, reproducible ~2.8x improvement β€” while remaining honest that flat PQ trained globally on multi-modal data has a known ceiling well below "near-perfect," which this fix does not and should not claim to lift.
+
+## What's New in 2026
+
+| Finding | Source | Confidence |
+|---|---|---|
+| Two independent SIGMOD 2026 papers (Ada-ef, DARTH) converge on per-query *declarative-recall* adaptive `efSearch` as the fix for the exact static-ef recall/latency tradeoff that killed 2026-08-15's rejected candidate β€” strongest external evidence of the night, but scoped out of tonight's candidate for patch-size/risk reasons (see Recommended Next Steps) | arXiv:2512.06636 (SIGMOD 2026), arXiv:2505.19001 + github.com/MChatzakis/DARTH (SIGMOD 2026) | A |
+| CVE-2026-25536: `@modelcontextprotocol/sdk` cross-client data leak via `StreamableHTTPServerTransport` reuse (1.10.0-1.25.3, fixed 1.26.0) β€” checked against Ruflo, not exploitable here (stdio transport, and `ruflo/src/ruvocal` already pinned to the fixed version) | GHSA-345p-7cg4-v4c7, cross-checked SentinelOne/Tenable/GitLab | A |
+| OWASP Top 10 for Agentic Applications 2026 formalizes ASI07 "Insecure Inter-Agent Communication" β€” directly relevant to a dormant Ruflo control found tonight (see Scan Findings: security) | genai.owasp.org | A |
+| SWARM+ (arXiv:2603.19431) validates at 990-agent scale that federation-style consensus quorums must shrink as members go stale/unreachable β€” directly on-point for a new hive-mind scan finding tonight | arXiv:2603.19431 | B |
+| No production multi-agent framework (LangGraph, AutoGen/AG2, CrewAI, OpenAI Agents SDK) ships queue-aware/cost-aware model routing as a first-party feature β€” a genuine, well-documented open gap (layering convention, not a solved-but-unpublished capability), per InfraMIND (arXiv:2606.11440) | arXiv:2606.11440 + 4 official framework docs, checked 2026-08-25 | B |
+
+## Ruflo Current Capability
+
+`v3/@claude-flow/memory/src/hnsw-index.ts`: `Quantizer` supports `binary`/`scalar`/`product` quantization types. For `product`, `productQuantize()` (1209-1269) trains PQ codebooks via k-means and stores centroid indices; `productQuantizeDistance()` (1337-1360, pre-fix) correctly implements codebook-aware distance β€” but had **zero callers** anywhere in `v3/` (confirmed via repo-wide grep) prior to tonight. `addPoint()`/`search()` stored/queried via `this.quantizer.encode()` but then ran the generic `distance()` switch (cosine/euclidean/dot/manhattan) directly on the resulting centroid-index arrays, and even normalized those index arrays for the cosine fast-path β€” both operations meaningless on PQ codes. `type: 'product'` appeared in exactly one test fixture repo-wide and was exercised by zero recall/correctness tests before tonight.
+
+Also confirmed, unrelated to tonight's fix: `binary`/`scalar` quantization have a related-but-distinct issue β€” `scalarQuantize()`'s output is `[min, range, ...levels]` (the prefix pollutes generic distance the same way PQ indices do), and `binaryQuantize()`'s packed-bit floats aren't cosine/Euclidean/dot-compatible either. **Disclosed, not fixed tonight** β€” kept out of scope to keep this patch one conceptual change (see Recommended Next Steps).
+
+## Competitor Comparison
+
+| Framework | Caching / KV-cache reuse | Adaptive/dynamic topology or scheduling | Queue-aware / cost-aware routing | Published perf benchmarks |
+|---|---|---|---|---|
+| LangGraph (v1.1.3) | Node-level output cache (`CachePolicy`, TTL) β€” not KV-cache | Conditional edges are task-logic-driven, not latency/cost-driven | None | Conflicting single-source claims (2.2x vs. 5.75x vs. losing to CrewAI) β€” B/C, not reproducible from a shared harness |
+| AutoGen/AG2 | `cache_seed` exact-match completion cache | `GroupChat` speaker selection is who-talks-next logic, not a perf scheduler | None (cost/token tracking exists, unused for routing) | "8-9x token-efficiency vs LangChain" (B, academic cross-framework study); no first-party latency benchmark |
+| CrewAI | Per-tool `cache_function` (developer-defined) | `Flows` branching is static, developer-defined | None β€” docs point to bolting on LiteLLM/Requesty | Vendor-adjacent "5.76x faster than LangGraph" contradicts the LangGraph row's own claim β€” neither independently reproducible (C) |
+| OpenAI Agents SDK | Platform-level prompt caching (0.1x read cost, β‰₯1024 tokens) β€” real and quantified (A) but API-layer, not SDK-specific | `Handoffs` are developer-authored, not adaptive | Load-balancing exists but is infra-level, invisible to orchestration logic | OpenHands' agent SDK measured 38.8% cache-hit rate vs. reference `codex` agent's 94.8% β€” a **2.7-3.4x cost inflation from one missing `prompt_cache_key` parameter** (B, concrete cautionary data point) |
+
+The "None" pattern for queue-aware/cost-aware routing is a genuine open gap, not solved-but-unpublished: InfraMIND (arXiv:2606.11440) reports +7.6pp accuracy / 7x lower latency / 99.9% vs <50% SLO compliance when routing is made infra-aware. The gap exists because all four frameworks deliberately push load-balancing/cache-affinity/cost signals down to a gateway layer (LiteLLM, vLLM Router, provider backend) to stay provider-portable β€” a real design tradeoff, not an oversight. Real opportunity for Ruflo, but the OpenHands `prompt_cache_key` incident is a live warning: caching is trivial to silently break in a multi-agent context, so any cache-aware routing claim needs a hit-rate-regression benchmark per release, not just a feature flag.
+
+## Hypothesis
+
+> Given an HNSWIndex configured with `quantization: { type: 'product' }`, where `productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance but are never invoked by the search path, when `distance()`/`distanceOptimized()` are patched to dispatch to `quantizer.productQuantizeDistance()` whenever product quantization is active and codebooks are trained (falling back to today's generic distance during the pre-training bootstrap window, unchanged), and vector normalization is skipped for product-quantized vectors, then recall@10 of a product-quantized HNSWIndex should recover from the current near-chance-level baseline to a level materially closer to the unquantized index's recall@10, subject to: (1) non-`'product'` quantization paths are byte-identical to today's; (2) existing HNSW/memory test suite remains green; (3) `getCompressionRatio()` is unchanged; (4) zero LLM/API cost.
+
+Frozen before evaluation began; not modified after seeing results.
+
+## Benchmarks
+
+Deterministic, $0, zero-LLM vitest additions: `v3/@claude-flow/memory/src/hnsw-quantization.test.ts`. Synthetic clustered corpus (8 well-separated Gaussian-ish blobs, N=1500, dim=64, seeded xorshift32 PRNG for full reproducibility), 30 held-out queries, recall@10 against brute-force ground truth computed on the *original unquantized* vectors (never leaked into the candidate).
+
+Disclosed for transparency: initial corpus parameters (N=400, `codebookSize=16`) were adjusted twice β€” N=400 put an unrepresentatively large fraction of the corpus inside the PQ pre-training bootstrap window; `codebookSize=16` was arbitrarily small. `codebookSize=256` is the code's actual documented default, not a tuned value. The pass-bar (`>= 0.25`) was set *after* measuring both baseline and candidate at the final N=1500/codebookSize=256 config, disclosed in the test file's comment with both real numbers.
+
+Darwin (`metaharness-darwin evolve --bench <suite.json>`) confirmed out of scope again tonight: its real interface evolves routing/topology/prompt/memory/tool/tier/context/coordination genome parameters against an LLM-scored bench corpus β€” no analog for a scoped TS distance-dispatch fix. Same class of skip as 4 of the last 5 dream-cycle nights.
+
+## Evaluation
+
+**evaluated: accepted.**
+
+Baseline captured via `git stash` on `hnsw-index.ts` only (test file kept in place):
+
+| Config | Recall@10 |
+|---|---|
+| Unquantized index (sanity check on corpus/harness) | 1.000 |
+| Product-quantized, **pre-fix** (generic distance on raw PQ indices) | **0.097** |
+| Product-quantized, **post-fix** (dispatched to `productQuantizeDistance`) | **0.270** |
+
+A ~2.8x recall@10 improvement, fully reproducible (all seeds fixed). A diagnostic run (not part of the committed suite) isolated PQ metric quality from HNSW graph-traversal quality: brute-force search using `productQuantizeDistance()` directly gave the *same* 0.270 as the full graph search, confirming the fix's ceiling is genuine PQ fidelity (flat PQ trained globally under-resolves fine-grained same-cluster ranking on multi-modal data β€” a real, known PQ characteristic, not a new bug), not an HNSW graph-construction artifact compounding on top of it.
+
+Full package suite: 461/462 passing pre- and post-fix (the 1 failure, `auto-memory-bridge.test.ts`'s read-only-file error-handling test, is environmental β€” this sandbox runs as root, which bypasses the `chmod`-based read-only precondition the test depends on; confirmed unrelated to this diff by identical failure on baseline). `tsc --noEmit` shows zero new errors attributable to this diff (all remaining errors are pre-existing `@types/node`/environment noise unrelated to `hnsw-index.ts`'s logic).
+
+**Independent adversarial critic** (fresh session, no authoring context) independently re-derived the 0.097β†’0.270 numbers via its own `git stash` isolation (confirmed exact match), re-ran the full suite, checked the `Uint8Array`β†’`Float32Array` signature change against every caller in `v3/` (found one, `@claude-flow/cli/__tests__/pq-validation.test.ts`, unaffected β€” 7/7 tests pass, `any`-typed accessor), and generalization-tested beyond the shipped corpus shape (128d/16 clusters, 16d/4 clusters, 64d/1 cluster β€” improvement direction and rough magnitude held in all three, pre-fix 0.068-0.088 β†’ post-fix 0.264-0.296). **Verdict: CONFIRMED-WITH-CAVEATS.** The one real caveat: `isValidPQEncoding()` is a *plausibility heuristic, not a proof of provenance* β€” the critic built a reproduction where an all-zero vector added pre-training passes the guard and gets silently (not-crashing, not-correctly-falling-back) mis-dispatched as if it were a real centroid assignment. This is a narrower manifestation of the same disclosed pre-training-fallback limitation, not a new hole; closing it fully needs per-node encoding-provenance tracking (out of scope tonight). Disclosed in-code and here rather than treated as fully solved.
+
+## Darwin Results
+
+Skipped β€” scope mismatch (see Benchmarks). Consistent with 08-14, 08-15, 08-18, 08-19, 08-24.
+
+## SOTA Proof & Witness
+
+(Populated in the Witness section of tonight's issue/PR; see below.)
+
+## Recommended Next Steps
+
+1. **Merge this candidate** (human review required) β€” closes a real, zero-cost correctness bug in a documented compression feature, with a permanent deterministic regression test.
+2. **Follow up on 2026-08-15's rejected `efSearch` finding using tonight's Grade-A evidence** (Ada-ef/DARTH, SIGMOD 2026): per-query declarative-recall adaptive `efSearch`, materially different and better-evidenced than a `sqrt(N)` heuristic. Scoped out tonight for patch-size/risk (needs a calibration mechanism plus `deserialize()` persistence) β€” strongest lead of the night, deserves a dedicated future night.
+3. **Wire `verifyInvocationToken` into `AgentAuthorizationPropagator.checkToolCall`** (`v3/@claude-flow/security/src/mcp-caller-identity.ts`) β€” an ADR-377 Ed25519 caller-identity control dormant since written, freshly relevant given OWASP's 2026 ASI07 category. Small, deterministic, env-gated.
+4. **Fix `FederationHub.vote()`'s stale-vote/quorum-denominator mismatch** (`v3/@claude-flow/swarm/src/federation-hub.ts:700-728`) β€” a departed swarm's vote is never purged while the quorum denominator shrinks. Independent of 08-24's weightedConsensus fix, anchored by SWARM+ (arXiv:2603.19431).
+5. **Harden or extend `scalar`/`binary` quantization distance dispatch** β€” same bug class as tonight's fix, disclosed but not fixed.
diff --git a/v3/@claude-flow/memory/src/hnsw-index.ts b/v3/@claude-flow/memory/src/hnsw-index.ts
index 881d835826..5c5cf9a761 100644
--- a/v3/@claude-flow/memory/src/hnsw-index.ts
+++ b/v3/@claude-flow/memory/src/hnsw-index.ts
@@ -270,8 +270,11 @@ export class HNSWIndex extends EventEmitter {
? this.quantizer.encode(vector)
: vector;
- // Pre-normalize vector for O(1) cosine similarity
- const normalizedVector = this.config.metric === 'cosine'
+ // Pre-normalize vector for O(1) cosine similarity.
+ // Skipped for product quantization: storedVector holds PQ centroid
+ // indices, not embedding coordinates, so normalizing it is meaningless
+ // (and productQuantizeDistance() below never consults normalizedVector).
+ const normalizedVector = this.config.metric === 'cosine' && !this.isProductQuantized()
? this.normalizeVector(storedVector)
: null;
@@ -335,8 +338,9 @@ export class HNSWIndex extends EventEmitter {
? this.quantizer.encode(query)
: query;
- // Pre-normalize query for O(1) cosine similarity
- const normalizedQuery = this.config.metric === 'cosine'
+ // Pre-normalize query for O(1) cosine similarity (see addPoint() for why
+ // this is skipped under product quantization).
+ const normalizedQuery = this.config.metric === 'cosine' && !this.isProductQuantized()
? this.normalizeVector(queryVector)
: null;
@@ -993,7 +997,49 @@ export class HNSWIndex extends EventEmitter {
}
}
+ /**
+ * True when this index uses product quantization with trained codebooks.
+ * Stored vectors are then PQ centroid indices, not embedding coordinates β€”
+ * generic metric distances (cosine/euclidean/dot/manhattan) computed
+ * directly on those indices are meaningless (index 5 vs index 200 has no
+ * relation to embedding-space proximity). Falls back to false during the
+ * pre-training bootstrap window (see Quantizer.productQuantize()), where
+ * stored vectors are still sub-vector-mean floats and generic distance,
+ * while imprecise, is not nonsensical β€” matching prior behavior exactly.
+ */
+ private isProductQuantized(): boolean {
+ return (
+ this.quantizer !== null &&
+ this.config.quantization?.type === 'product' &&
+ this.quantizer.isPQTrained
+ );
+ }
+
private distance(a: Float32Array, b: Float32Array): number {
+ // Pre-existing, disclosed limitation (not introduced by this fix): any
+ // vector added before PQ codebook training completes is stored under a
+ // sub-vector-mean fallback encoding (see Quantizer.productQuantize())
+ // and is never re-encoded once training finishes β€” the original raw
+ // vector isn't retained. isValidPQEncoding() is a *plausibility*
+ // heuristic, not a proof of provenance: it structurally guards the
+ // crash/out-of-bounds case (fallback values outside codebook range, or
+ // non-integer), but a fallback encoding whose sub-vector means
+ // coincidentally land on small non-negative integers (most concretely,
+ // an all-zero or otherwise degenerate vector added pre-training) can
+ // still pass the check and be silently mis-dispatched to
+ // productQuantizeDistance() as if it were a real centroid assignment.
+ // Closing this fully would need per-node encoding-provenance tracking
+ // (e.g. a sentinel/flag threaded through HNSWNode and its persistence
+ // format), out of scope for this patch β€” flagged as a known residual
+ // risk, not treated as fully solved.
+ if (
+ this.isProductQuantized() &&
+ this.quantizer!.isValidPQEncoding(a) &&
+ this.quantizer!.isValidPQEncoding(b)
+ ) {
+ return this.quantizer!.productQuantizeDistance(a, b);
+ }
+
switch (this.config.metric) {
case 'cosine':
return this.cosineDistance(a, b);
@@ -1312,14 +1358,17 @@ class Quantizer {
* Compute approximate squared Euclidean distance between two PQ-encoded
* vectors using their centroid indices and the shared codebooks.
*
- * @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers)
+ * @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers).
+ * Typed Float32Array to match productQuantize()'s actual return type β€”
+ * indices are small integers (0..codebookSize-1) exactly representable
+ * in float32.
* @param encoded2 - Centroid indices for vector 2 (length = numSubquantizers)
* @param codebooks - Trained codebooks from trainProductQuantizer()
* @returns Approximate squared Euclidean distance
*/
productQuantizeDistance(
- encoded1: Uint8Array,
- encoded2: Uint8Array,
+ encoded1: Float32Array,
+ encoded2: Float32Array,
codebooks?: number[][][],
): number {
const cb = codebooks || this.codebooks;
@@ -1412,6 +1461,23 @@ class Quantizer {
getCodebooks(): number[][][] | null {
return this.codebooks;
}
+
+ /**
+ * True when `encoded` looks like a real trained-codebook PQ encoding
+ * (length matches numSubquantizers, every value is a non-negative integer
+ * in range for its sub-quantizer's codebook). False for the pre-training
+ * sub-vector-mean fallback (see productQuantize()), which cannot be
+ * compared via productQuantizeDistance() without corrupting results or
+ * indexing out of bounds.
+ */
+ isValidPQEncoding(encoded: Float32Array): boolean {
+ if (!this.codebooks || encoded.length !== this.codebooks.length) return false;
+ for (let m = 0; m < encoded.length; m++) {
+ const idx = encoded[m];
+ if (!Number.isInteger(idx) || idx < 0 || idx >= this.codebooks[m].length) return false;
+ }
+ return true;
+ }
}
// ===== Persistence Helpers (ADR-125 Phase 3) =====
diff --git a/v3/@claude-flow/memory/src/hnsw-quantization.test.ts b/v3/@claude-flow/memory/src/hnsw-quantization.test.ts
new file mode 100644
index 0000000000..0b81522bea
--- /dev/null
+++ b/v3/@claude-flow/memory/src/hnsw-quantization.test.ts
@@ -0,0 +1,170 @@
+/**
+ * Dream Cycle 2026-08-25 β€” HNSW product-quantization distance dispatch.
+ *
+ * `Quantizer.productQuantize()` / `productQuantizeDistance()` correctly
+ * implement codebook-aware PQ encoding and distance, but nothing in the
+ * search path (`distance()` / `distanceOptimized()`) ever called
+ * `productQuantizeDistance()` β€” generic cosine/euclidean/dot/manhattan
+ * distance was computed directly on raw PQ centroid-index arrays, which is
+ * numerically meaningless (adjacent centroid indices are not adjacent in
+ * embedding space). This test builds a synthetic clustered corpus, computes
+ * brute-force ground-truth top-10 neighbors on the *unquantized* vectors,
+ * and asserts recall@10 for a product-quantized index stays well above
+ * chance level. On the pre-fix code this assertion fails (recall collapses
+ * toward chance); post-fix it passes.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { HNSWIndex } from './hnsw-index.js';
+
+function xorshift32(seed: number): () => number {
+ let s = seed || 1;
+ return () => {
+ s ^= s << 13;
+ s ^= s >>> 17;
+ s ^= s << 5;
+ return ((s >>> 0) / 4294967296);
+ };
+}
+
+/** Deterministic clustered corpus: `numClusters` well-separated Gaussian blobs. */
+function buildClusteredCorpus(
+ n: number,
+ dim: number,
+ numClusters: number,
+ seed: number,
+): Float32Array[] {
+ const rand = xorshift32(seed);
+ const centers: Float32Array[] = [];
+ for (let c = 0; c < numClusters; c++) {
+ const center = new Float32Array(dim);
+ for (let d = 0; d < dim; d++) {
+ // Spread centers far apart so clusters are well-separated.
+ center[d] = (rand() - 0.5) * 20;
+ }
+ centers.push(center);
+ }
+
+ const vectors: Float32Array[] = [];
+ for (let i = 0; i < n; i++) {
+ const center = centers[i % numClusters];
+ const v = new Float32Array(dim);
+ for (let d = 0; d < dim; d++) {
+ v[d] = center[d] + (rand() - 0.5) * 1.5; // small intra-cluster noise
+ }
+ vectors.push(v);
+ }
+ return vectors;
+}
+
+function bruteForceTopK(
+ query: Float32Array,
+ corpus: Float32Array[],
+ ids: string[],
+ k: number,
+): Set<string> {
+ const scored = corpus.map((v, i) => {
+ let sum = 0;
+ for (let d = 0; d < v.length; d++) {
+ const diff = v[d] - query[d];
+ sum += diff * diff;
+ }
+ return { id: ids[i], dist: sum };
+ });
+ scored.sort((a, b) => a.dist - b.dist);
+ return new Set(scored.slice(0, k).map((s) => s.id));
+}
+
+async function recallAtK(config: {
+ dim: number;
+ n: number;
+ numClusters: number;
+ numQueries: number;
+ k: number;
+ quantization?: { type: 'product'; subquantizers: number; codebookSize: number };
+}): Promise<number> {
+ const corpus = buildClusteredCorpus(config.n, config.dim, config.numClusters, 42);
+ const ids = corpus.map((_, i) => `v-${i}`);
+
+ const index = new HNSWIndex({
+ dimensions: config.dim,
+ metric: 'euclidean',
+ M: 16,
+ efConstruction: 100,
+ ...(config.quantization ? { quantization: config.quantization } : {}),
+ });
+
+ for (let i = 0; i < corpus.length; i++) {
+ await index.addPoint(ids[i], corpus[i]);
+ }
+
+ const rand = xorshift32(1337);
+ let totalRecall = 0;
+ for (let q = 0; q < config.numQueries; q++) {
+ const queryIdx = Math.floor(rand() * config.n);
+ const query = corpus[queryIdx];
+
+ const groundTruth = bruteForceTopK(query, corpus, ids, config.k);
+ const results = await index.search(query, config.k, 200);
+ const returned = new Set(results.map((r) => r.id));
+
+ let hits = 0;
+ for (const id of returned) if (groundTruth.has(id)) hits++;
+ totalRecall += hits / config.k;
+ }
+
+ return totalRecall / config.numQueries;
+}
+
+describe('HNSWIndex product quantization β€” distance dispatch (Dream Cycle 2026-08-25)', () => {
+ const DIM = 64;
+ // pqTrainingThreshold is 256; N well above it keeps the pre-training
+ // bootstrap window (see isValidPQEncoding()) a small fraction of the
+ // corpus, representative of real usage rather than a pathological
+ // near-100%-cold-start case.
+ const N = 1500;
+ const NUM_CLUSTERS = 8;
+ const NUM_QUERIES = 30;
+ const K = 10;
+
+ it('unquantized baseline recovers near-perfect recall@10 (sanity check on the corpus/harness itself)', async () => {
+ const recall = await recallAtK({ dim: DIM, n: N, numClusters: NUM_CLUSTERS, numQueries: NUM_QUERIES, k: K });
+ // eslint-disable-next-line no-console
+ console.log(`[dream-cycle] unquantized recall@10 = ${recall.toFixed(3)}`);
+ expect(recall).toBeGreaterThan(0.85);
+ });
+
+ it('product-quantized index recall@10 recovers materially vs. the pre-fix baseline', async () => {
+ const recall = await recallAtK({
+ dim: DIM,
+ n: N,
+ numClusters: NUM_CLUSTERS,
+ numQueries: NUM_QUERIES,
+ k: K,
+ quantization: { type: 'product', subquantizers: 8, codebookSize: 256 },
+ });
+ // eslint-disable-next-line no-console
+ console.log(`[dream-cycle] product-quantized recall@10 = ${recall.toFixed(3)}`);
+ // Measured on this exact deterministic corpus (git stash on hnsw-index.ts
+ // only, test file kept in place):
+ // pre-fix (generic distance on raw PQ centroid indices): recall@10 = 0.097
+ // post-fix (productQuantizeDistance dispatched correctly): recall@10 = 0.270
+ // Flat PQ trained globally on multi-modal (clustered) data is known to
+ // under-resolve fine-grained same-cluster ranking β€” recall@10 in the
+ // 0.2-0.3 range here reflects that real PQ characteristic, not a bug;
+ // the assertion checks the fix's effect (recall recovers well above the
+ // measured buggy baseline), not an absolute "good PQ" bar.
+ expect(recall).toBeGreaterThanOrEqual(0.25);
+ });
+
+ it('getCompressionRatio() is unaffected by the distance-dispatch fix', () => {
+ const index = new HNSWIndex({
+ dimensions: DIM,
+ metric: 'euclidean',
+ quantization: { type: 'product', subquantizers: 8, codebookSize: 16 },
+ });
+ // 8 subquantizers -> 8x compression, unchanged by this fix (only the
+ // distance computation changed, not the stored representation).
+ expect((index as unknown as { quantizer: { getCompressionRatio(): number } }).quantizer.getCompressionRatio()).toBe(8);
+ });
+});