Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

dream(memory): #3351 fix AgentDBAdapter.store() same-key upsert orphaning (evaluated, ACCEPT-scoped) (#3352)

rUv34e4bb400a31
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md
index 12ded52316..6eee761c5d 100644
--- a/docs/dream-cycle/LEDGER.md
+++ b/docs/dream-cycle/LEDGER.md
@@ -192,6 +192,11 @@ as a first-class, high-priority candidate, not a background note.
| 2026-09-10 | performance | `mmrRerank()` eagerly re-tokenized every remaining candidate's content on every MMR outer-loop pass (O(N×limit) `tokenize()` calls) — a pattern that predates and survived PR #3169's embedding-cosine wiring unchanged, even though the cosine branch made that tokenization mostly discarded work in the now-common all-embedded case; made lazy and cached per-candidate | #3265 | #3266 | yes | ACCEPT | `tokenize()` calls 400→0 (N=30/limit=20 corpus, stash-isolated: fails against real baseline, passes against candidate); wall-clock 5.88x faster (0.6745ms→0.1148ms mean), independently reproduced at 5.96x by an adversarial critic; selection output byte-identical to baseline across all-embedded/none/mixed/lambda-extreme scenarios; 473/474 full-suite both ways (1 pre-existing unrelated failure), `tsc` clean | 78c074ef... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232 merges), awaiting fresh CI as of 2026-09-14 |
| 2026-09-11 | security | `hive-mind_consensus` accepted votes from any caller-supplied `voterId` with no check against `state.workers` (a single caller could cross any strategy's quorum by voting under fabricated identities — a Sybil attack on consensus); fail-closed roster-membership check added. **Update (ruvnet review):** REJECTed the initial scoped fix on the caveat the adversarial critic had already disclosed (`hive-mind_join`/`leave` fully unauthenticated — the roster itself could be Sybil'd) plus a required-CI cold-install failure (`@claude-flow/security` missing a `prepare` script, so root `npm ci` never built its dist/). Both addressed same-session: `join`/`leave`/`vote` now capability-bound to a `hiveToken` minted by `hive-mind_init` (32-byte random, constant-time comparison, fail-closed — zero state change for denied calls, verified via a fresh state.json reload simulating restart; a same-machine-only token reader lets the CLI's own subcommands keep working without exposing the token over any MCP response); `@claude-flow/security`'s `package.json` gained a `prepare: npm run build` so root `npm ci` builds it automatically for every consumer, not just this CI job | #3290 | #3291 | yes | ACCEPT-scoped (post-review) | stash-isolated 2/3 fail baseline (forged votes silently accepted) → 3/3 pass candidate; full `@claude-flow/cli` suite failures 46→28 after the CI fix (remainder pre-existing/unrelated); adversarial critic CONFIRMED-WITH-CAVEATS pre-review, both caveats closed post-review | 55d67b4a... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232/#3266 merges), awaiting fresh CI as of 2026-09-14 |
| 2026-09-12 | intelligence | `LocalReasoningBank.findSimilar()` overwrote every returned pattern's `confidence` (learned reliability) with the per-query cosine similarity score instead of the already-declared `PatternMatch.similarity` field, inverting `distillLearning()`'s "only distill from high-confidence matches" gate (it tested query-similarity, not reliability) and forcing `findSimilarPatterns()`'s public API into an unsafe cast that made `confidence`/`similarity` always identical; `findSimilar()` now preserves real `confidence` and returns `similarity` separately, with the one affected consumer (`memory-bridge.ts`'s search-ranking score) migrated accordingly | #3301 | #3302 | yes | ACCEPT | 4 new discriminating tests, baseline-fails/candidate-passes via stash isolation; full suite 90-vs-91-failed-files identical except the new test file; `tsc --noEmit` byte-identical (463 errors both ways); independent adversarial critique CONFIRMED, no blocking caveats | cb60b43b... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232/#3266/#3291 merges), awaiting fresh CI as of 2026-09-14 |
+**No-run gap (2026-09-13, 2026-09-14):** no branch/PR/issue exists for either date (verified via `git ls-remote --heads origin "dream/*"` and `search_pull_requests`, 2026-09-18). Tonight's automation SCAN confirms no in-repo cron/workflow drives this trigger, so the gap is genuinely unexplained from repo config, not a local misconfiguration — same open question as the 2026-08-20..23 gap.
+| 2026-09-15 | performance | wire MoE gate's load-balance loss into the actual weight update (title-derived backfill; full PR body not re-verified tonight) | #3329 | #3330 | yes | ACCEPT-scoped | see issue #3329 / PR #3330 | n/a (backfill) | unmerged draft as of 2026-09-18, 3d old |
+| 2026-09-16 | security | gate `hive-mind_spawn`/`propose`/`broadcast`/`shutdown`/`memory`/`optimize-memory` + `hive-mind_init`'s own credential issuance, round 2 (title-derived backfill; full PR body not re-verified tonight) | #3338 | #3339 | yes | ACCEPT | see issue #3338 / PR #3339 | n/a (backfill) | unmerged draft as of 2026-09-18, 2d old |
+| 2026-09-17 | intelligence | wire `CLAUDE_FLOW_PRIOR_DECAY` env override for `ModelRouter`'s dormant discounted-Thompson-sampling decay primitive (#3049), recommended by 2026-09-12's own gist; real benchmark receipt (low-complexity bucket: recovery 26.5→21.9 rounds, t=7.0; med-complexity bucket: stationary Δ=−0.08pp, t=−3.01, accepted within pre-declared ±1pp tolerance) | #3349 | #3350 | yes | ACCEPT-with-caveats | 16/16 tests; full suite byte-identical 221 pre-existing failures both ways; independent adversarial critic CONFIRMED-WITH-CAVEATS (net-positive low-complexity, negligible-but-real med-complexity cost) | 91d8edc3... | unmerged draft as of 2026-09-18, 1d old |
+| 2026-09-18 | memory | `AgentDBAdapter.store()` (default in-process write path) never checked for an existing entry under the same `(namespace,key)` before writing — a fresh random id every call left the prior occupant an orphan, unreachable via `getByKey()` but still live in `entries`/`namespaceIndex`/`tagIndex` and, for embedded entries, still a point in the HNSW index, so `search()`/`semanticSearch()` returned stale duplicates forever; fixed by evicting the prior occupant via a shared `evictEntry()` primitive before storing. Independently flagged by two prior nightly scans (09-12, 09-17) as a future-night candidate. **Post-review (ruvnet, same night):** two further public-path gaps found and fixed — `bulkInsert()` bypassed the same-key dedup entirely (fixed: pre-existing + intra-batch duplicate-key eviction), and `store()` deleted the prior occupant *before* validating the replacement embedding, risking data loss on a rejected write (fixed: validate/index first, evict on success) | #3351 | #3352 | yes | ACCEPT-scoped (post-review) | 9 discriminating tests total (4 original + 5 post-review, incl. a hand-built two-gate harness proving `setTimeout` can't reproduce the disclosed concurrency race); full suite 517→526 passed (1 pre-existing unrelated failure, unchanged); `tsc --noEmit` clean; independent adversarial critique CONFIRMED-WITH-CAVEATS pre-review, human review found 2 further blocking gaps post-review, both closed same night — concurrent-store race (not bulkInsert/atomicity) remains the one disclosed, still-open follow-up | d4d02062... | 09-08..09-12 (5 nights) MERGED in a 2026-09-14/15 batch; 09-15/09-16/09-17 still open/draft, none stale (<4d old); 09-13/09-14 no-run gap noted above |
**Note (2026-09-17):** verified via GitHub MCP tools (not inferred) before tonight's own row: **2026-09-13 and 2026-09-14 are genuine no-run gaps** — no `dream/2026-09-13-*`/`dream/2026-09-14-*` branch, no matching issue/PR, same class as the earlier 2026-08-20..23 gap. **2026-09-15 ran** (`performance`, #3329 — MoE gate load-balance loss computed but never applied to weights) and **2026-09-16 ran** (`security`, #3338 — hive-mind consensus bypass); neither transcribed as its own row here (out of scope for tonight's `intelligence` slot to backfill in full — flagging their existence is enough to keep STEP 1's gap-detection honest). Trailing-14 PR merge rate: **14/14 MERGED** — a bulk human-review sweep on 2026-09-14/15 cleared the entire backlog from #3086 through #3302 in one pass. This is the opposite extreme from the "0 of last 14" signal earlier nights tracked, so tonight did **not** bias toward an artificially small candidate on that basis — but the bulk-sweep nature (many PRs merged together) is a weaker scrutiny signal than a steady one-by-one trickle, noted for future nights' calibration.
diff --git a/docs/dream-cycle/dream-gist-2026-09-18.md b/docs/dream-cycle/dream-gist-2026-09-18.md
new file mode 100644
index 0000000000..90368a51cb
--- /dev/null
+++ b/docs/dream-cycle/dream-gist-2026-09-18.md
@@ -0,0 +1,103 @@
+# Memory SOTA Report — 2026-09-18
+
+**TL;DR**: `AgentDBAdapter.store()` (`v3/@claude-flow/memory/src/agentdb-adapter.ts`) — the default in-process write path behind `storeEntry()` — never checked for an existing entry under the same `(namespace, key)` before writing. Since `entry.id` is always a fresh random id (`generateMemoryId()`), a second `store()` call for the same logical key left the prior occupant as a permanent orphan: unreachable via `getByKey()`, but still live in `entries`/`namespaceIndex`/`tagIndex` and, for embedded entries, still a point in the HNSW index — so `search()`/`semanticSearch()` returned stale duplicates forever. This exact gap was independently flagged by two prior nightly scans (2026-09-12 memory scan, 2026-09-17 memory scan) as "a strong candidate for a future memory DEEP night." Tonight closes it by reusing the adapter's own already-tested `delete()` path to evict the prior occupant before storing the new entry — a ~15-line, single-file, single-conceptual-change fix.
+
+## What's New in 2026
+
+| Finding | Source | Confidence |
+|---|---|---|
+| Qdrant/Weaviate's documented idempotent-upsert pattern: deterministic id derived from the natural key (UUIDv5), not a random id | Qdrant Points docs, Weaviate `generate_uuid5` docs (current) | A |
+| Milvus `upsert()` = transactional delete-by-PK + insert; Sept-2026 change (PR #53158/#53511) made Full-AutoID upsert *preserve* the existing PK instead of reallocating | milvus.io upsert docs + linked PRs, Sept 2026 | A |
+| Mem0 has an open TOCTOU race in its own hash-based dedup: `add()` snapshots existing memories, checks the stale snapshot after an LLM round-trip — two concurrent calls for the same fact both insert | github.com/mem0ai/mem0#6531, filed 2026-07-23, open | A |
+| Cross-vendor pattern for quantized-vector distance (Qdrant, Weaviate, OpenSearch): Asymmetric Distance Computation — keep the query unquantized, compare against quantized document vectors | OpenSearch ADC blog (2025-12-16), Qdrant/Weaviate quantization docs | A |
+| RaBitQ is real and shipped — in `@claude-flow/cli/src/memory/rabitq-index.ts` (WASM, 32x compression), not `@claude-flow/memory`; CLAUDE.md's claim isn't fabricated, just cross-package | Direct source read tonight | A |
+
+## Ruflo Current Capability
+
+Four backends implement `IMemoryBackend`: `AgentDBAdapter` (default, in-process, HNSW-backed), `sqlite-backend.ts`, `sqljs-backend.ts`, `hybrid-backend.ts` (delegates reads-by-key to sqlite only). Tonight's architecture review confirmed the bug end-to-end: `AgentDBAdapter.store()` had no key-dedup; `sqlite-backend.ts`'s schema has no `UNIQUE(namespace,key)` (only `sqljs-backend.ts` does — a cross-backend inconsistency); `hybrid-backend.getByKey()` is hard-wired to the unordered SQLite read. A working precedent already existed in-repo: `v3/@claude-flow/cli/src/memory/memory-initializer.ts` has both `UNIQUE(namespace,key)` and a wired `removeHNSWEntriesByKey()` — the newer `@claude-flow/memory` package never carried it forward. Tonight closes the default-adapter gap; the SQLite-schema half (candidate C, 4.00) and the HNSW binary/scalar quantization dispatch bug (candidate B, 4.35, same class as merged PR #3093/#3094) remain open.
+
+## Competitor Comparison
+
+| Competitor | Idempotent upsert-by-key | Quantized-distance correctness (ADC) | Grade |
+|---|---|---|---|
+| Qdrant | Deterministic UUIDv5-from-key is the documented pattern | Yes — genuine asymmetric quantization since v1.15 | A |
+| Weaviate | `generate_uuid5(key)` re-write replaces the object | Yes, explicit rescore/oversampling for SQ/BQ/RQ | A |
+| Milvus | PK-based upsert (delete+insert); Sept-2026 fix preserves PK under AutoID | — | A |
+| Mem0 (agent memory) | Intent is right (content-hash key) but has an open TOCTOU dedup race (#6531) | — | A |
+| LangGraph / CrewAI | Client-supplied key (LangGraph) or LLM-mediated semantic consolidation, no hash-of-key (CrewAI, deliberate — natural-language facts rarely repeat verbatim) | — | B |
+
+Synthesis: Ruflo's `auto-memory-bridge.ts` already does the *right* thing structurally (content-hash dedup key, pre-fetch existing hashes) — the same shape Mem0 just got a TOCTOU bug filed against; tonight's fix doesn't touch that concurrency question, which is a disclosed follow-up risk, not solved tonight. On quantized-distance correctness, Ruflo's merged PQ dispatch (`productQuantizeDistance()`) is genuinely at parity with Qdrant/Weaviate's ADC pattern; binary/scalar remain unfixed (candidate B, not selected tonight).
+
+## Hypothesis
+
+> Given `AgentDBAdapter.store()`, when a second `store()` call targets an entry sharing `(namespace, key)` with an already-stored entry, then the prior entry should be fully evicted (`entries`, `namespaceIndex`, `tagIndex`, `keyIndex`, and — when embedded — the HNSW index) before the new entry is stored, relative to baseline where the prior entry persists forever as an orphaned duplicate returned by `search()`/`semanticSearch()`, subject to: (1) a genuinely new `(namespace,key)` is unaffected; (2) reuses `delete()`'s existing tested cleanup rather than duplicating it; (3) no change to public entry/search-result shapes; (4) all existing tests remain green; (5) $0, fully deterministic evaluation.
+
+Frozen before evaluation; not modified after seeing results.
+
+## Benchmarks / Evaluation
+
+**evaluated: accepted.** Real evaluator: Vitest 4.1.8, deterministic, $0, zero LLM calls. New file `v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts` (4 tests).
+
+Baseline (pre-diff): 2 of 4 new tests fail for exactly the predicted reason (`get(first.id)` returns the stale entry instead of `null`; `search()` returns both `'stale-fact'` and `'fresh-fact'`). The other 2 (new key, cross-namespace) pass both ways — correct controls. Candidate: 4/4 pass. Full `@claude-flow/memory` suite: baseline 517 passed/1 failed (518, 25 files) → candidate 521 passed/1 failed (522, 26 files) — the 1 failure is `auto-memory-bridge.test.ts`'s pre-existing chmod-based read-only test, identical both ways (sandbox runs as root, bypasses the `chmod`). `tsc --noEmit`: 0 errors both ways.
+
+**Independent adversarial critique** (separate subagent, no authoring context, re-verified the stash-isolation result itself): **CONFIRMED-WITH-CAVEATS**. Confirmed diff scope (only the 2 named files touched), confirmed `storeEntry()` also benefits (no bypass path), confirmed embedding-present/absent transitions and `entry.id === existingId` are handled correctly, confirmed O(1) added cost, confirmed zero new I/O/network/credential surface. **Real disclosed gap**: no mutex — under truly *concurrent* `store()` calls for the same `(namespace,key)` (e.g. `Promise.all([...])`), both can read `keyIndex.get()` before either writes, and both insert, reproducing the same orphan bug for the concurrent case. All 4 new tests are sequential and don't exercise this. Verdict scoped to **ACCEPT for sequential re-stores** (today's only exercised production pattern); concurrent-store safety is an explicit, disclosed follow-up, not fixed tonight.
+
+## Darwin Results
+
+Confirmed real interface: `npx [email protected] harness darwin <config> --execute`. Skipped — binary correctness fix, no continuous/categorical fitness-gradient parameter. Same class as most recent nights.
+
+## Flywheel Evidence
+
+Confirmed real interface: `npx [email protected] harness flywheel {verify <bundle>|gate <evidence>}` — targets signed LLM-scored replay bundles, not deterministic code fixes; none exists for this candidate. Evidence retained as: 4 new tests, this gist, the linked issue, stash-isolated baseline/candidate comparison, adversarial critique — consistent with every accepted night since 2026-08-18.
+
+## Reward Hack Check
+
+No standalone reward-hack CLI reachable. Manual checklist: no existing test weakened (new file only); no gold data touched; no cherry-picking (full suite reported both ways, the 1 pre-existing failure disclosed); no seed manipulation; zero cost.
+
+## Security Review
+
+Not security-sensitive: pure in-process bookkeeping (reuse of the existing `delete()` path), no new I/O/network/credential/filesystem surface. The disclosed concurrency gap (above) is a correctness/data-integrity risk under concurrent writers, not a security boundary issue.
+
+## Post-review addendum (2026-09-18, same night)
+
+ruvnet reviewed at `eb499928c` and found two further public-path gaps the first pass missed: (1) `AgentDBAdapter.bulkInsert()` — which `UnifiedMemoryService.bulkInsert()` forwards to — bypassed the same-key dedup entirely, so a batch with duplicate `(namespace,key)` values or one replacing an already-stored key still left old ids reachable; (2) `store()` deleted the prior occupant *before* validating the replacement via `HNSWIndex.addPoint()` (fallible: dimension mismatch, full index), so a rejected write lost the prior value outright instead of merely duplicating it. Both fixed in `8636bbac5`: `bulkInsert()` now computes superseded ids (pre-existing occupant + intra-batch duplicate-key losers) from a pre-mutation snapshot and evicts them via a shared `evictEntry()` primitive after the batch is indexed; `store()` now runs `addPoint()` before eviction, so a rejected replacement fails clean. 5 new tests added per the review's request, including a hand-built two-promise-gate harness for the disclosed concurrent-store race (a plain `setTimeout` delay does not actually interleave two async calls in Node — confirmed empirically, the first attempt at this test could not reproduce the race at all). Full suite: 521→526 passed (+5), same 1 pre-existing unrelated failure, `tsc --noEmit` clean.
+
+## SOTA Proof & Witness
+
+| Field | Value |
+|---|---|
+| Session commit | `e558f0c0fc29c1a658085f6e6f80ad27d4fe811f` |
+| Gist SHA-256 (pre-witness content) | `c5c9c86da1136fd9280a8a1a40b58679e993bd02574c10756103ba3f3663650a` |
+| Witness stamp | `d4d020625c6468c7c955e0081ab7eb74ef2259ca4e4dd0a50a0e02f511bbf82f` |
+
+Verifier procedure: fetch this gist, strip the witness table's filled values back to `PENDING`, SHA-256 the result, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.
+
+## Scan Findings: plugins
+
+`PluginManager.installFromNpm` (`manager.ts:149-221`) validates only the package-name string and runs `npm install` unconditionally — never reads `PluginEntry.permissions`/`.trustLevel`/`.securityAudit` against `PluginStoreConfig`, despite these being fully modeled and named in SECURITY.md as the mitigation. `--verify` (checksum) is parsed and never used. New tonight: `trust/trust-anchors.json` declares an "ADR-145 PluginIntegrityVerifier" schema with a placeholder all-zero key and `owner: "PLACEHOLDER"` — zero TS references anywhere; the control was never implemented. Comparison: npm ships `--provenance`/Sigstore attestations verifiable pre-install; Ruflo's installer resembles VS Code Marketplace's weaker identity-only model (malicious-extension detections nearly quadrupled there in 2025, per ReversingLabs). Not selected tonight (SCAN); real gap for a future capabilities/security night.
+
+## Scan Findings: automation
+
+No in-repo cron/workflow drives the nightly dream-cycle trigger — it lives outside this repo, so tonight's confirmed 2-night no-run gap (2026-09-13/14, no branch/PR/issue for either date, verified via `git ls-remote`) is genuinely unexplained from repo config. The one visibility fix for this class of gap, `dream-cycle-backlog-guard.yml`, has itself sat as draft PR #3205 since 2026-09-05 (13+ days, zero activity) — the detector for "things go stale unmerged" is itself stale and unmerged. Separately: `hooks/README.md` documents 12 named background workers; the actual `ALLOWED_WORKERS` allowlist in `src/workers/index.ts` has 11 completely different names, zero overlap, and only 5 worker files exist on disk. Not selected tonight (SCAN); merge #3205 first, then treat the doc/implementation drift as a themed future `automation` night.
+
+## Competitors Reviewed
+
+Qdrant, Weaviate, Milvus, LanceDB, Mem0, Zep/Graphiti, Letta, LangGraph, Microsoft Agent Framework/AutoGen, CrewAI, OpenAI Agents SDK (memory-deep comparison); npm provenance/Sigstore, VS Code Marketplace (plugins scan); GitHub Actions scheduled-workflow reliability (automation scan).
+
+## Additional candidates surfaced but not selected tonight (STEP 3.2)
+
+Scored under `0.25·Ruflo_fit + 0.20·testability + 0.20·measurability + 0.15·production_value + 0.10·novelty + 0.10·reviewability`. Selected candidate (A, same-key upsert) scored **4.70**, highest of five.
+
+1. **[4.35] B — HNSW binary/scalar quantization distance dispatch bug**: same class as merged PQ fix (#3093/#3094); no `binaryQuantizeDistance()`/`scalarQuantizeDistance()` exists, so those types fall through to generic cosine/euclidean on packed bitmasks/min-range-polluted arrays. Dead code today (never configured/tested), lowering urgency, not correctness risk. Strongest standalone second choice.
+2. **[4.00] C — `sqlite-backend.ts` `UNIQUE(namespace,key)` + `ON CONFLICT DO UPDATE`**: closes the other backend's half of tonight's bug class; not the default path, left for follow-up.
+3. **[3.35] D — Architectural**: two unequal quantizers exist (`@claude-flow/cli`'s real WASM RaBitQ vs. `@claude-flow/memory`'s broken hand-rolled binary/scalar) — gate the broken one or port the real one.
+4. **[2.95] E — Consolidator threshold calibration**: replace the fixed 0.95 cosine cutoff with a calibrated percentile (arXiv:2605.08538 uses 0.559 at p99) or Mem0-style write-time classification. Needs a labeled corpus — out of scope tonight.
+
+## Recommended Next Steps
+
+1. **Merge tonight's linked draft PR** (human review required).
+2. **Add a per-key async lock to `AgentDBAdapter.store()`/`bulkInsert()`** — the still-open, now-doubly-disclosed gap (adversarial critic, then human review): sequential re-stores and same-batch/batch-vs-pre-existing replacements are now safe, concurrent `Promise.all()` re-stores of the same key across separate calls are not.
+3. **Fix candidate B** (HNSW binary/scalar quantization distance dispatch) — same mechanical shape as the already-reviewed PQ fix.
+4. **Check `auto-memory-bridge.ts`'s hash-dedup path for the same TOCTOU race Mem0 just filed** (#6531) — structurally similar snapshot-then-insert shape, not verified either way tonight.
+5. **Merge draft PR #3205** (`dream-cycle-backlog-guard.yml`) — the missing heartbeat for exactly the no-run-gap class tonight's automation scan re-confirmed.
+6. **Reconcile `hooks/README.md`'s 12-worker roster against the actual 11-name `ALLOWED_WORKERS` allowlist** — a future `automation` night should resolve which is wrong.
diff --git a/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts b/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts
new file mode 100644
index 0000000000..9ace00c91a
--- /dev/null
+++ b/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts
@@ -0,0 +1,252 @@
+/**
+ * Dream Cycle 2026-09-18 (memory) — regression tests for AgentDBAdapter.store()
+ * same-(namespace,key) idempotency.
+ *
+ * Baseline bug: store() minted a fresh random id per call (createDefaultEntry ->
+ * generateMemoryId()) and never looked up an existing entry under the same
+ * (namespace, key) before writing. The prior occupant was left as an orphan:
+ * unreachable via getByKey()/keyIndex, but still present in entries/
+ * namespaceIndex/tagIndex, and — for embedded entries — still a live point in
+ * the HNSW index, so search()/semanticSearch() returned stale duplicates
+ * forever. Fixed by evicting the prior (namespace,key) occupant via a shared
+ * evictEntry() primitive (also used by delete() and bulkInsert()) before
+ * storing the new entry.
+ *
+ * Post-review hardening (same night): an adversarial human review found two
+ * further public-path gaps the first pass missed — bulkInsert() bypassed the
+ * same-key dedup entirely, and store() originally evicted the prior occupant
+ * BEFORE validating the replacement's embedding, so a rejected addPoint()
+ * (dimension mismatch, index full) would delete the old value and still fail,
+ * losing data outright. Both are fixed below; the tests in this second block
+ * cover exactly the five cases the review asked for.
+ */
+import { existsSync, mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { AgentDBAdapter } from './agentdb-adapter.js';
+import { createDefaultEntry } from './types.js';
+
+function unitVector(dims: number, hotIndex: number): Float32Array {
+ const v = new Float32Array(dims);
+ v[hotIndex % dims] = 1;
+ return v;
+}
+
+function allInNamespace(namespace: string) {
+ return { type: 'prefix' as const, keyPrefix: '', namespace, limit: 1000 };
+}
+
+describe('AgentDBAdapter.store() — same-key upsert idempotency (Dream Cycle 2026-09-18)', () => {
+ let adapter: AgentDBAdapter;
+
+ beforeEach(async () => {
+ adapter = new AgentDBAdapter({ dimensions: 8, cacheEnabled: true });
+ await adapter.initialize();
+ });
+
+ it('a second store() under the same (namespace,key) does not leave two entries reachable', async () => {
+ const first = createDefaultEntry({ key: 'profile', namespace: 'ns1', content: 'v1' });
+ await adapter.store(first);
+
+ const second = createDefaultEntry({ key: 'profile', namespace: 'ns1', content: 'v2' });
+ await adapter.store(second);
+
+ expect(first.id).not.toBe(second.id); // ids are independently random, as today
+
+ const byKey = await adapter.getByKey('ns1', 'profile');
+ expect(byKey?.content).toBe('v2');
+
+ // The prior occupant must be fully gone, not just unreachable via getByKey.
+ const stale = await adapter.get(first.id);
+ expect(stale).toBeNull();
+
+ const all = await adapter.query(allInNamespace('ns1'));
+ expect(all.filter((e) => e.key === 'profile')).toHaveLength(1);
+ });
+
+ it('a second store() under the same (namespace,key) evicts the stale HNSW point, so search() never returns the old content', async () => {
+ const first = createDefaultEntry({ key: 'fact', namespace: 'ns1', content: 'stale-fact' });
+ first.embedding = unitVector(8, 0);
+ await adapter.store(first);
+
+ const second = createDefaultEntry({ key: 'fact', namespace: 'ns1', content: 'fresh-fact' });
+ second.embedding = unitVector(8, 0); // same embedding direction, different content/id
+ await adapter.store(second);
+
+ const results = await adapter.search(unitVector(8, 0), { k: 10 });
+ const contents = results.map((r) => r.entry.content);
+ expect(contents).toContain('fresh-fact');
+ expect(contents).not.toContain('stale-fact');
+ expect(contents.filter((c) => c === 'fresh-fact')).toHaveLength(1);
+ });
+
+ it('storing a genuinely new (namespace,key) is unaffected (no eviction, both entries reachable)', async () => {
+ const a = createDefaultEntry({ key: 'a', namespace: 'ns1', content: 'A' });
+ const b = createDefaultEntry({ key: 'b', namespace: 'ns1', content: 'B' });
+ await adapter.store(a);
+ await adapter.store(b);
+
+ expect((await adapter.get(a.id))?.content).toBe('A');
+ expect((await adapter.get(b.id))?.content).toBe('B');
+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(2);
+ });
+
+ it('the same (namespace,key) in two different namespaces are independent (no cross-namespace eviction)', async () => {
+ const a = createDefaultEntry({ key: 'shared', namespace: 'ns1', content: 'from-ns1' });
+ const b = createDefaultEntry({ key: 'shared', namespace: 'ns2', content: 'from-ns2' });
+ await adapter.store(a);
+ await adapter.store(b);
+
+ expect((await adapter.getByKey('ns1', 'shared'))?.content).toBe('from-ns1');
+ expect((await adapter.getByKey('ns2', 'shared'))?.content).toBe('from-ns2');
+ });
+});
+
+describe('AgentDBAdapter same-key upsert — post-review hardening (Dream Cycle 2026-09-18)', () => {
+ let adapter: AgentDBAdapter;
+
+ beforeEach(async () => {
+ adapter = new AgentDBAdapter({ dimensions: 8, cacheEnabled: true });
+ await adapter.initialize();
+ });
+
+ it('a replacement rejected by HNSWIndex.addPoint() (dimension mismatch) leaves the prior value intact', async () => {
+ const first = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'original' });
+ first.embedding = unitVector(8, 0);
+ await adapter.store(first);
+
+ const bad = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'malformed' });
+ bad.embedding = new Float32Array(4); // wrong dimension for this 8-dim adapter
+
+ await expect(adapter.store(bad)).rejects.toThrow(/dimension mismatch/i);
+
+ // The prior value must survive a rejected replacement -- not be deleted
+ // and then fail to be replaced (the exact data-loss bug the review found).
+ const byKey = await adapter.getByKey('ns1', 'k');
+ expect(byKey?.content).toBe('original');
+ expect(byKey?.id).toBe(first.id);
+ expect(await adapter.get(first.id)).not.toBeNull();
+ });
+
+ it('bulkInsert() evicts a duplicate key within the same batch, last entry wins', async () => {
+ const loser = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'first-in-batch' });
+ const winner = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'last-in-batch' });
+
+ await adapter.bulkInsert([loser, winner]);
+
+ const byKey = await adapter.getByKey('ns1', 'k');
+ expect(byKey?.content).toBe('last-in-batch');
+ expect(byKey?.id).toBe(winner.id);
+
+ expect(await adapter.get(loser.id)).toBeNull();
+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(1);
+ });
+
+ it('bulkInsert() evicts a pre-existing store()d entry that a batch replaces', async () => {
+ const preExisting = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'pre-existing' });
+ preExisting.embedding = unitVector(8, 0);
+ await adapter.store(preExisting);
+
+ const replacement = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-batch' });
+ replacement.embedding = unitVector(8, 0);
+ await adapter.bulkInsert([replacement]);
+
+ expect(await adapter.get(preExisting.id)).toBeNull();
+ const results = await adapter.search(unitVector(8, 0), { k: 10 });
+ const contents = results.map((r) => r.entry.content);
+ expect(contents).toContain('from-batch');
+ expect(contents).not.toContain('pre-existing');
+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(1);
+ });
+
+ it('a same-key replacement survives a persisted save/reload -- only the winner comes back', async () => {
+ const dir = mkdtempSync(join(tmpdir(), 'agentdb-adapter-upsert-'));
+ const persistencePath = join(dir, 'store');
+ try {
+ const first = new AgentDBAdapter({
+ dimensions: 8,
+ cacheEnabled: true,
+ persistenceEnabled: true,
+ persistencePath,
+ });
+ await first.initialize();
+
+ const original = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'v1' });
+ await first.store(original);
+ const replacement = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'v2' });
+ await first.store(replacement);
+ await first.shutdown(); // persists to disk
+
+ const reloaded = new AgentDBAdapter({
+ dimensions: 8,
+ cacheEnabled: true,
+ persistenceEnabled: true,
+ persistencePath,
+ });
+ await reloaded.initialize(); // loads from disk
+
+ const byKey = await reloaded.getByKey('ns1', 'k');
+ expect(byKey?.content).toBe('v2');
+ expect(await reloaded.get(original.id)).toBeNull();
+ expect(await reloaded.query(allInNamespace('ns1'))).toHaveLength(1);
+ } finally {
+ if (existsSync(dir)) rmSync(dir, { recursive: true, force: true });
+ }
+ });
+
+ it('KNOWN LIMITATION (disclosed, not fixed tonight): truly concurrent store() calls for the same key can both proceed and leave two reachable entries', async () => {
+ // No mutex/lock exists around the keyIndex.get() check + writes. Two
+ // store() calls for the same (namespace,key), both parked mid-flight on
+ // an embeddingGenerator await and released in the same tick, can each
+ // resume and read keyIndex before the other has written to it, so
+ // neither evicts the other. A plain setTimeout-based delay does NOT
+ // reliably reproduce this: Node drains the microtask queue (including
+ // every further await inside one resumed call) before the next timer
+ // macrotask runs, so two independently-timed calls tend to fully
+ // serialize instead of interleaving at the check (confirmed: an earlier
+ // version of this test using setTimeout could not reproduce the gap).
+ // Two externally-controlled gates, released back-to-back in the same
+ // synchronous tick after both calls are already parked, force the
+ // actual interleaving this test needs to demonstrate.
+ let releaseA!: () => void;
+ let releaseB!: () => void;
+ const gateA = new Promise<void>((resolve) => { releaseA = resolve; });
+ const gateB = new Promise<void>((resolve) => { releaseB = resolve; });
+
+ const raced = new AgentDBAdapter({
+ dimensions: 8,
+ cacheEnabled: true,
+ embeddingGenerator: (content) => (content === 'from-a' ? gateA : gateB).then(() => unitVector(8, 0)),
+ });
+ await raced.initialize();
+
+ const a = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-a' });
+ const b = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-b' });
+
+ // Both calls run synchronously up to their embeddingGenerator await and
+ // are now genuinely parked -- store() hasn't read keyIndex yet for
+ // either. Releasing both gates back-to-back interleaves their resumption.
+ const pA = raced.store(a);
+ const pB = raced.store(b);
+ releaseA();
+ releaseB();
+ await Promise.all([pA, pB]);
+
+ // query()'s prefix search iterates keyIndex, which by construction holds
+ // exactly one id per (namespace,key) -- it structurally cannot see an
+ // orphan still sitting in `entries` under a different id, so it's the
+ // wrong probe here (an earlier version of this test used it and always
+ // saw "1", regardless of whether the race actually happened). get(id)
+ // against each participant's own id is the correct probe, matching how
+ // this file's very first test detects the sequential-case orphan.
+ const aStillPresent = (await raced.get(a.id)) !== null;
+ const bStillPresent = (await raced.get(b.id)) !== null;
+ // Today: both survive concurrently (the bug this test documents) --
+ // neither call's isReplacement check saw the other's write in time. If
+ // this assertion ever starts failing because only one is still present,
+ // the concurrency gap has been fixed -- update this test to assert the
+ // fixed behavior instead of loosening it.
+ expect([aStillPresent, bStillPresent]).toEqual([true, true]);
+ });
+});
diff --git a/v3/@claude-flow/memory/src/agentdb-adapter.ts b/v3/@claude-flow/memory/src/agentdb-adapter.ts
index 793e2498ab..e14457d03e 100644
--- a/v3/@claude-flow/memory/src/agentdb-adapter.ts
+++ b/v3/@claude-flow/memory/src/agentdb-adapter.ts
@@ -182,18 +182,50 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
entry.embedding = await this.config.embeddingGenerator(entry.content);
}
+ // Namespace is resolved once, up front, so the dedup lookup below and the
+ // index updates further down agree on the same value.
+ const namespace = entry.namespace || this.config.defaultNamespace;
+ const keyIndexKey = `${namespace}:${entry.key}`;
+
+ // Idempotent upsert-by-key (Dream Cycle 2026-09-18, hardened post-review):
+ // entry.id is always a fresh random id (generateMemoryId()), so a second
+ // store() under the same (namespace, key) previously left the prior
+ // occupant as an orphan — unreachable via getByKey()/keyIndex, but still
+ // live in entries/namespaceIndex/tagIndex and, for embedded entries,
+ // still a point in the HNSW index, so search()/semanticSearch() returned
+ // stale duplicates forever.
+ const existingId = this.keyIndex.get(keyIndexKey);
+ const isReplacement = existingId !== undefined && existingId !== entry.id;
+
+ // Fallible precondition first: HNSWIndex.addPoint() can throw (dimension
+ // mismatch, index full). Doing this BEFORE evicting the prior occupant
+ // means a rejected write fails cleanly with the prior value still intact,
+ // instead of deleting it and only then discovering the replacement can't
+ // be indexed (a real data-loss bug an adversarial review caught).
+ if (entry.embedding) {
+ await this.index.addPoint(entry.id, entry.embedding);
+ }
+
+ // Evict the prior (namespace,key) occupant via the shared primitive also
+ // used by bulkInsert() and delete(). keyIndex is intentionally left
+ // alone here — it's overwritten to the new id a few lines down, and
+ // clearing it in evictEntry() first would just be redone work (worse,
+ // if evictEntry() read a since-mutated keyIndex it could delete the
+ // mapping this call is about to set).
+ if (isReplacement) {
+ await this.evictEntry(existingId!, { touchKeyIndex: false });
+ }
+
// Store in main storage
this.entries.set(entry.id, entry);
// Update namespace index
- const namespace = entry.namespace || this.config.defaultNamespace;
if (!this.namespaceIndex.has(namespace)) {
this.namespaceIndex.set(namespace, new Set());
}
this.namespaceIndex.get(namespace)!.add(entry.id);
// Update key index
- const keyIndexKey = `${namespace}:${entry.key}`;
this.keyIndex.set(keyIndexKey, entry.id);
// Update tag index
@@ -204,11 +236,6 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
this.tagIndex.get(tag)!.add(entry.id);
}
- // Index embedding if available
- if (entry.embedding) {
- await this.index.addPoint(entry.id, entry.embedding);
- }
-
// Update cache
if (this.config.cacheEnabled) {
this.cache.set(entry.id, entry);
@@ -320,6 +347,19 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
* Delete a memory entry
*/
async delete(id: string): Promise<boolean> {
+ return this.evictEntry(id, { touchKeyIndex: true });
+ }
+
+ /**
+ * Shared eviction primitive behind delete() and the same-key upsert paths
+ * in store()/bulkInsert() (Dream Cycle 2026-09-18, post-review). Removes
+ * an entry from entries/namespaceIndex/tagIndex/HNSW/cache. touchKeyIndex
+ * is false when the caller is itself about to overwrite the
+ * (namespace,key) -> id mapping to point at a replacement entry —
+ * clearing it here first would either be redone work or, if the mapping
+ * had already been repointed, would incorrectly delete the new mapping.
+ */
+ private async evictEntry(id: string, { touchKeyIndex }: { touchKeyIndex: boolean }): Promise<boolean> {
const entry = this.entries.get(id);
if (!entry) return false;
@@ -330,8 +370,10 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
this.namespaceIndex.get(entry.namespace)?.delete(id);
// Remove from key index
- const keyIndexKey = `${entry.namespace}:${entry.key}`;
- this.keyIndex.delete(keyIndexKey);
+ if (touchKeyIndex) {
+ const keyIndexKey = `${entry.namespace}:${entry.key}`;
+ this.keyIndex.delete(keyIndexKey);
+ }
// Remove from tag index
for (const tag of entry.tags) {
@@ -458,6 +500,35 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
}
}
+ // Same-key upsert (Dream Cycle 2026-09-18, post-review): bulkInsert()
+ // bypassed store()'s per-key dedup entirely, so a batch containing
+ // duplicate (namespace,key) values -- or replacing an already-stored
+ // key -- left old ids reachable via query()/search() while only
+ // keyIndex pointed at the newest one. Computed from the pre-mutation
+ // keyIndex snapshot and input order, before any Map below is touched:
+ // the *last* entry per key in this batch wins (matching what N
+ // sequential store() calls for the same inputs would produce), and any
+ // id that isn't the winner for its key -- a pre-existing occupant or an
+ // earlier same-key entry within this very batch -- is scheduled for
+ // eviction once the winners are safely indexed.
+ const keyOf = (e: MemoryEntry) => `${e.namespace || this.config.defaultNamespace}:${e.key}`;
+ const winnerIdForKey = new Map<string, string>();
+ for (const entry of entries) {
+ winnerIdForKey.set(keyOf(entry), entry.id);
+ }
+ const idsToEvict = new Set<string>();
+ for (const [key, winnerId] of winnerIdForKey) {
+ const existingId = this.keyIndex.get(key);
+ if (existingId && existingId !== winnerId) {
+ idsToEvict.add(existingId);
+ }
+ }
+ for (const entry of entries) {
+ if (entry.id !== winnerIdForKey.get(keyOf(entry))) {
+ idsToEvict.add(entry.id);
+ }
+ }
+
// Phase 2: Store all entries (skip individual cache updates)
const embeddings: Array<{ id: string; embedding: Float32Array }> = [];
@@ -472,9 +543,8 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
}
this.namespaceIndex.get(namespace)!.add(entry.id);
- // Update key index
- const keyIndexKey = `${namespace}:${entry.key}`;
- this.keyIndex.set(keyIndexKey, entry.id);
+ // Update key index (last entry per key wins, per winnerIdForKey above)
+ this.keyIndex.set(keyOf(entry), entry.id);
// Update tag index
for (const tag of entry.tags) {
@@ -490,16 +560,30 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {
}
}
- // Phase 3: Batch index embeddings
+ // Phase 3: Batch index embeddings. Deliberately BEFORE Phase 3.5's
+ // eviction below, mirroring store()'s ordering: if addPoint() throws
+ // (dimension mismatch, index full) for any winner, nothing superseded
+ // has been evicted yet, so a failed bulkInsert() doesn't lose data that
+ // a failed single store() wouldn't have lost either.
for (let i = 0; i < embeddings.length; i += batchSize) {
const batch = embeddings.slice(i, i + batchSize);
await Promise.all(batch.map(({ id, embedding }) => this.index.addPoint(id, embedding)));
}
- // Phase 4: Batch cache update (only populate hot entries)
+ // Phase 3.5: evict superseded occupants (pre-existing and intra-batch),
+ // now that every winner is fully written and indexed. keyIndex is left
+ // alone -- Phase 2 already points it at the correct winner for each key.
+ for (const id of idsToEvict) {
+ await this.evictEntry(id, { touchKeyIndex: false });
+ }
+
+ // Phase 4: Batch cache update (only populate hot entries, winners only —
+ // an evicted loser must not be left warm in the cache under its own id)
if (this.config.cacheEnabled && entries.length <= this.config.cacheSize) {
for (const entry of entries) {
- this.cache.set(entry.id, entry);
+ if (!idsToEvict.has(entry.id)) {
+ this.cache.set(entry.id, entry);
+ }
}
}