🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
dream(memory): #3351 fix AgentDBAdapter.store() same-key upsert orphaning (evaluated, ACCEPT-scoped) (#3352)
rUv34e4bb400a31
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.mdindex 12ded52316..6eee761c5d 100644--- a/docs/dream-cycle/LEDGER.md+++ b/docs/dream-cycle/LEDGER.md@@ -192,6 +192,11 @@ as a first-class, high-priority candidate, not a background note.| 2026-09-10 | performance | `mmrRerank()` eagerly re-tokenized every remaining candidate's content on every MMR outer-loop pass (O(N×limit) `tokenize()` calls) — a pattern that predates and survived PR #3169's embedding-cosine wiring unchanged, even though the cosine branch made that tokenization mostly discarded work in the now-common all-embedded case; made lazy and cached per-candidate | #3265 | #3266 | yes | ACCEPT | `tokenize()` calls 400→0 (N=30/limit=20 corpus, stash-isolated: fails against real baseline, passes against candidate); wall-clock 5.88x faster (0.6745ms→0.1148ms mean), independently reproduced at 5.96x by an adversarial critic; selection output byte-identical to baseline across all-embedded/none/mixed/lambda-extreme scenarios; 473/474 full-suite both ways (1 pre-existing unrelated failure), `tsc` clean | 78c074ef... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232 merges), awaiting fresh CI as of 2026-09-14 || 2026-09-11 | security | `hive-mind_consensus` accepted votes from any caller-supplied `voterId` with no check against `state.workers` (a single caller could cross any strategy's quorum by voting under fabricated identities — a Sybil attack on consensus); fail-closed roster-membership check added. **Update (ruvnet review):** REJECTed the initial scoped fix on the caveat the adversarial critic had already disclosed (`hive-mind_join`/`leave` fully unauthenticated — the roster itself could be Sybil'd) plus a required-CI cold-install failure (`@claude-flow/security` missing a `prepare` script, so root `npm ci` never built its dist/). Both addressed same-session: `join`/`leave`/`vote` now capability-bound to a `hiveToken` minted by `hive-mind_init` (32-byte random, constant-time comparison, fail-closed — zero state change for denied calls, verified via a fresh state.json reload simulating restart; a same-machine-only token reader lets the CLI's own subcommands keep working without exposing the token over any MCP response); `@claude-flow/security`'s `package.json` gained a `prepare: npm run build` so root `npm ci` builds it automatically for every consumer, not just this CI job | #3290 | #3291 | yes | ACCEPT-scoped (post-review) | stash-isolated 2/3 fail baseline (forged votes silently accepted) → 3/3 pass candidate; full `@claude-flow/cli` suite failures 46→28 after the CI fix (remainder pre-existing/unrelated); adversarial critic CONFIRMED-WITH-CAVEATS pre-review, both caveats closed post-review | 55d67b4a... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232/#3266 merges), awaiting fresh CI as of 2026-09-14 || 2026-09-12 | intelligence | `LocalReasoningBank.findSimilar()` overwrote every returned pattern's `confidence` (learned reliability) with the per-query cosine similarity score instead of the already-declared `PatternMatch.similarity` field, inverting `distillLearning()`'s "only distill from high-confidence matches" gate (it tested query-similarity, not reliability) and forcing `findSimilarPatterns()`'s public API into an unsafe cast that made `confidence`/`similarity` always identical; `findSimilar()` now preserves real `confidence` and returns `similarity` separately, with the one affected consumer (`memory-bridge.ts`'s search-ranking score) migrated accordingly | #3301 | #3302 | yes | ACCEPT | 4 new discriminating tests, baseline-fails/candidate-passes via stash isolation; full suite 90-vs-91-failed-files identical except the new test file; `tsc --noEmit` byte-identical (463 errors both ways); independent adversarial critique CONFIRMED, no blocking caveats | cb60b43b... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160/#3232/#3266/#3291 merges), awaiting fresh CI as of 2026-09-14 |+**No-run gap (2026-09-13, 2026-09-14):** no branch/PR/issue exists for either date (verified via `git ls-remote --heads origin "dream/*"` and `search_pull_requests`, 2026-09-18). Tonight's automation SCAN confirms no in-repo cron/workflow drives this trigger, so the gap is genuinely unexplained from repo config, not a local misconfiguration — same open question as the 2026-08-20..23 gap.+| 2026-09-15 | performance | wire MoE gate's load-balance loss into the actual weight update (title-derived backfill; full PR body not re-verified tonight) | #3329 | #3330 | yes | ACCEPT-scoped | see issue #3329 / PR #3330 | n/a (backfill) | unmerged draft as of 2026-09-18, 3d old |+| 2026-09-16 | security | gate `hive-mind_spawn`/`propose`/`broadcast`/`shutdown`/`memory`/`optimize-memory` + `hive-mind_init`'s own credential issuance, round 2 (title-derived backfill; full PR body not re-verified tonight) | #3338 | #3339 | yes | ACCEPT | see issue #3338 / PR #3339 | n/a (backfill) | unmerged draft as of 2026-09-18, 2d old |+| 2026-09-17 | intelligence | wire `CLAUDE_FLOW_PRIOR_DECAY` env override for `ModelRouter`'s dormant discounted-Thompson-sampling decay primitive (#3049), recommended by 2026-09-12's own gist; real benchmark receipt (low-complexity bucket: recovery 26.5→21.9 rounds, t=7.0; med-complexity bucket: stationary Δ=−0.08pp, t=−3.01, accepted within pre-declared ±1pp tolerance) | #3349 | #3350 | yes | ACCEPT-with-caveats | 16/16 tests; full suite byte-identical 221 pre-existing failures both ways; independent adversarial critic CONFIRMED-WITH-CAVEATS (net-positive low-complexity, negligible-but-real med-complexity cost) | 91d8edc3... | unmerged draft as of 2026-09-18, 1d old |+| 2026-09-18 | memory | `AgentDBAdapter.store()` (default in-process write path) never checked for an existing entry under the same `(namespace,key)` before writing — a fresh random id every call left the prior occupant an orphan, unreachable via `getByKey()` but still live in `entries`/`namespaceIndex`/`tagIndex` and, for embedded entries, still a point in the HNSW index, so `search()`/`semanticSearch()` returned stale duplicates forever; fixed by evicting the prior occupant via a shared `evictEntry()` primitive before storing. Independently flagged by two prior nightly scans (09-12, 09-17) as a future-night candidate. **Post-review (ruvnet, same night):** two further public-path gaps found and fixed — `bulkInsert()` bypassed the same-key dedup entirely (fixed: pre-existing + intra-batch duplicate-key eviction), and `store()` deleted the prior occupant *before* validating the replacement embedding, risking data loss on a rejected write (fixed: validate/index first, evict on success) | #3351 | #3352 | yes | ACCEPT-scoped (post-review) | 9 discriminating tests total (4 original + 5 post-review, incl. a hand-built two-gate harness proving `setTimeout` can't reproduce the disclosed concurrency race); full suite 517→526 passed (1 pre-existing unrelated failure, unchanged); `tsc --noEmit` clean; independent adversarial critique CONFIRMED-WITH-CAVEATS pre-review, human review found 2 further blocking gaps post-review, both closed same night — concurrent-store race (not bulkInsert/atomicity) remains the one disclosed, still-open follow-up | d4d02062... | 09-08..09-12 (5 nights) MERGED in a 2026-09-14/15 batch; 09-15/09-16/09-17 still open/draft, none stale (<4d old); 09-13/09-14 no-run gap noted above |**Note (2026-09-17):** verified via GitHub MCP tools (not inferred) before tonight's own row: **2026-09-13 and 2026-09-14 are genuine no-run gaps** — no `dream/2026-09-13-*`/`dream/2026-09-14-*` branch, no matching issue/PR, same class as the earlier 2026-08-20..23 gap. **2026-09-15 ran** (`performance`, #3329 — MoE gate load-balance loss computed but never applied to weights) and **2026-09-16 ran** (`security`, #3338 — hive-mind consensus bypass); neither transcribed as its own row here (out of scope for tonight's `intelligence` slot to backfill in full — flagging their existence is enough to keep STEP 1's gap-detection honest). Trailing-14 PR merge rate: **14/14 MERGED** — a bulk human-review sweep on 2026-09-14/15 cleared the entire backlog from #3086 through #3302 in one pass. This is the opposite extreme from the "0 of last 14" signal earlier nights tracked, so tonight did **not** bias toward an artificially small candidate on that basis — but the bulk-sweep nature (many PRs merged together) is a weaker scrutiny signal than a steady one-by-one trickle, noted for future nights' calibration.diff --git a/docs/dream-cycle/dream-gist-2026-09-18.md b/docs/dream-cycle/dream-gist-2026-09-18.mdnew file mode 100644index 0000000000..90368a51cb--- /dev/null+++ b/docs/dream-cycle/dream-gist-2026-09-18.md@@ -0,0 +1,103 @@+# Memory SOTA Report — 2026-09-18++**TL;DR**: `AgentDBAdapter.store()` (`v3/@claude-flow/memory/src/agentdb-adapter.ts`) — the default in-process write path behind `storeEntry()` — never checked for an existing entry under the same `(namespace, key)` before writing. Since `entry.id` is always a fresh random id (`generateMemoryId()`), a second `store()` call for the same logical key left the prior occupant as a permanent orphan: unreachable via `getByKey()`, but still live in `entries`/`namespaceIndex`/`tagIndex` and, for embedded entries, still a point in the HNSW index — so `search()`/`semanticSearch()` returned stale duplicates forever. This exact gap was independently flagged by two prior nightly scans (2026-09-12 memory scan, 2026-09-17 memory scan) as "a strong candidate for a future memory DEEP night." Tonight closes it by reusing the adapter's own already-tested `delete()` path to evict the prior occupant before storing the new entry — a ~15-line, single-file, single-conceptual-change fix.++## What's New in 2026++| Finding | Source | Confidence |+|---|---|---|+| Qdrant/Weaviate's documented idempotent-upsert pattern: deterministic id derived from the natural key (UUIDv5), not a random id | Qdrant Points docs, Weaviate `generate_uuid5` docs (current) | A |+| Milvus `upsert()` = transactional delete-by-PK + insert; Sept-2026 change (PR #53158/#53511) made Full-AutoID upsert *preserve* the existing PK instead of reallocating | milvus.io upsert docs + linked PRs, Sept 2026 | A |+| Mem0 has an open TOCTOU race in its own hash-based dedup: `add()` snapshots existing memories, checks the stale snapshot after an LLM round-trip — two concurrent calls for the same fact both insert | github.com/mem0ai/mem0#6531, filed 2026-07-23, open | A |+| Cross-vendor pattern for quantized-vector distance (Qdrant, Weaviate, OpenSearch): Asymmetric Distance Computation — keep the query unquantized, compare against quantized document vectors | OpenSearch ADC blog (2025-12-16), Qdrant/Weaviate quantization docs | A |+| RaBitQ is real and shipped — in `@claude-flow/cli/src/memory/rabitq-index.ts` (WASM, 32x compression), not `@claude-flow/memory`; CLAUDE.md's claim isn't fabricated, just cross-package | Direct source read tonight | A |++## Ruflo Current Capability++Four backends implement `IMemoryBackend`: `AgentDBAdapter` (default, in-process, HNSW-backed), `sqlite-backend.ts`, `sqljs-backend.ts`, `hybrid-backend.ts` (delegates reads-by-key to sqlite only). Tonight's architecture review confirmed the bug end-to-end: `AgentDBAdapter.store()` had no key-dedup; `sqlite-backend.ts`'s schema has no `UNIQUE(namespace,key)` (only `sqljs-backend.ts` does — a cross-backend inconsistency); `hybrid-backend.getByKey()` is hard-wired to the unordered SQLite read. A working precedent already existed in-repo: `v3/@claude-flow/cli/src/memory/memory-initializer.ts` has both `UNIQUE(namespace,key)` and a wired `removeHNSWEntriesByKey()` — the newer `@claude-flow/memory` package never carried it forward. Tonight closes the default-adapter gap; the SQLite-schema half (candidate C, 4.00) and the HNSW binary/scalar quantization dispatch bug (candidate B, 4.35, same class as merged PR #3093/#3094) remain open.++## Competitor Comparison++| Competitor | Idempotent upsert-by-key | Quantized-distance correctness (ADC) | Grade |+|---|---|---|---|+| Qdrant | Deterministic UUIDv5-from-key is the documented pattern | Yes — genuine asymmetric quantization since v1.15 | A |+| Weaviate | `generate_uuid5(key)` re-write replaces the object | Yes, explicit rescore/oversampling for SQ/BQ/RQ | A |+| Milvus | PK-based upsert (delete+insert); Sept-2026 fix preserves PK under AutoID | — | A |+| Mem0 (agent memory) | Intent is right (content-hash key) but has an open TOCTOU dedup race (#6531) | — | A |+| LangGraph / CrewAI | Client-supplied key (LangGraph) or LLM-mediated semantic consolidation, no hash-of-key (CrewAI, deliberate — natural-language facts rarely repeat verbatim) | — | B |++Synthesis: Ruflo's `auto-memory-bridge.ts` already does the *right* thing structurally (content-hash dedup key, pre-fetch existing hashes) — the same shape Mem0 just got a TOCTOU bug filed against; tonight's fix doesn't touch that concurrency question, which is a disclosed follow-up risk, not solved tonight. On quantized-distance correctness, Ruflo's merged PQ dispatch (`productQuantizeDistance()`) is genuinely at parity with Qdrant/Weaviate's ADC pattern; binary/scalar remain unfixed (candidate B, not selected tonight).++## Hypothesis++> Given `AgentDBAdapter.store()`, when a second `store()` call targets an entry sharing `(namespace, key)` with an already-stored entry, then the prior entry should be fully evicted (`entries`, `namespaceIndex`, `tagIndex`, `keyIndex`, and — when embedded — the HNSW index) before the new entry is stored, relative to baseline where the prior entry persists forever as an orphaned duplicate returned by `search()`/`semanticSearch()`, subject to: (1) a genuinely new `(namespace,key)` is unaffected; (2) reuses `delete()`'s existing tested cleanup rather than duplicating it; (3) no change to public entry/search-result shapes; (4) all existing tests remain green; (5) $0, fully deterministic evaluation.++Frozen before evaluation; not modified after seeing results.++## Benchmarks / Evaluation++**evaluated: accepted.** Real evaluator: Vitest 4.1.8, deterministic, $0, zero LLM calls. New file `v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts` (4 tests).++Baseline (pre-diff): 2 of 4 new tests fail for exactly the predicted reason (`get(first.id)` returns the stale entry instead of `null`; `search()` returns both `'stale-fact'` and `'fresh-fact'`). The other 2 (new key, cross-namespace) pass both ways — correct controls. Candidate: 4/4 pass. Full `@claude-flow/memory` suite: baseline 517 passed/1 failed (518, 25 files) → candidate 521 passed/1 failed (522, 26 files) — the 1 failure is `auto-memory-bridge.test.ts`'s pre-existing chmod-based read-only test, identical both ways (sandbox runs as root, bypasses the `chmod`). `tsc --noEmit`: 0 errors both ways.++**Independent adversarial critique** (separate subagent, no authoring context, re-verified the stash-isolation result itself): **CONFIRMED-WITH-CAVEATS**. Confirmed diff scope (only the 2 named files touched), confirmed `storeEntry()` also benefits (no bypass path), confirmed embedding-present/absent transitions and `entry.id === existingId` are handled correctly, confirmed O(1) added cost, confirmed zero new I/O/network/credential surface. **Real disclosed gap**: no mutex — under truly *concurrent* `store()` calls for the same `(namespace,key)` (e.g. `Promise.all([...])`), both can read `keyIndex.get()` before either writes, and both insert, reproducing the same orphan bug for the concurrent case. All 4 new tests are sequential and don't exercise this. Verdict scoped to **ACCEPT for sequential re-stores** (today's only exercised production pattern); concurrent-store safety is an explicit, disclosed follow-up, not fixed tonight.++## Darwin Results++Confirmed real interface: `npx [email protected] harness darwin <config> --execute`. Skipped — binary correctness fix, no continuous/categorical fitness-gradient parameter. Same class as most recent nights.++## Flywheel Evidence++Confirmed real interface: `npx [email protected] harness flywheel {verify <bundle>|gate <evidence>}` — targets signed LLM-scored replay bundles, not deterministic code fixes; none exists for this candidate. Evidence retained as: 4 new tests, this gist, the linked issue, stash-isolated baseline/candidate comparison, adversarial critique — consistent with every accepted night since 2026-08-18.++## Reward Hack Check++No standalone reward-hack CLI reachable. Manual checklist: no existing test weakened (new file only); no gold data touched; no cherry-picking (full suite reported both ways, the 1 pre-existing failure disclosed); no seed manipulation; zero cost.++## Security Review++Not security-sensitive: pure in-process bookkeeping (reuse of the existing `delete()` path), no new I/O/network/credential/filesystem surface. The disclosed concurrency gap (above) is a correctness/data-integrity risk under concurrent writers, not a security boundary issue.++## Post-review addendum (2026-09-18, same night)++ruvnet reviewed at `eb499928c` and found two further public-path gaps the first pass missed: (1) `AgentDBAdapter.bulkInsert()` — which `UnifiedMemoryService.bulkInsert()` forwards to — bypassed the same-key dedup entirely, so a batch with duplicate `(namespace,key)` values or one replacing an already-stored key still left old ids reachable; (2) `store()` deleted the prior occupant *before* validating the replacement via `HNSWIndex.addPoint()` (fallible: dimension mismatch, full index), so a rejected write lost the prior value outright instead of merely duplicating it. Both fixed in `8636bbac5`: `bulkInsert()` now computes superseded ids (pre-existing occupant + intra-batch duplicate-key losers) from a pre-mutation snapshot and evicts them via a shared `evictEntry()` primitive after the batch is indexed; `store()` now runs `addPoint()` before eviction, so a rejected replacement fails clean. 5 new tests added per the review's request, including a hand-built two-promise-gate harness for the disclosed concurrent-store race (a plain `setTimeout` delay does not actually interleave two async calls in Node — confirmed empirically, the first attempt at this test could not reproduce the race at all). Full suite: 521→526 passed (+5), same 1 pre-existing unrelated failure, `tsc --noEmit` clean.++## SOTA Proof & Witness++| Field | Value |+|---|---|+| Session commit | `e558f0c0fc29c1a658085f6e6f80ad27d4fe811f` |+| Gist SHA-256 (pre-witness content) | `c5c9c86da1136fd9280a8a1a40b58679e993bd02574c10756103ba3f3663650a` |+| Witness stamp | `d4d020625c6468c7c955e0081ab7eb74ef2259ca4e4dd0a50a0e02f511bbf82f` |++Verifier procedure: fetch this gist, strip the witness table's filled values back to `PENDING`, SHA-256 the result, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.++## Scan Findings: plugins++`PluginManager.installFromNpm` (`manager.ts:149-221`) validates only the package-name string and runs `npm install` unconditionally — never reads `PluginEntry.permissions`/`.trustLevel`/`.securityAudit` against `PluginStoreConfig`, despite these being fully modeled and named in SECURITY.md as the mitigation. `--verify` (checksum) is parsed and never used. New tonight: `trust/trust-anchors.json` declares an "ADR-145 PluginIntegrityVerifier" schema with a placeholder all-zero key and `owner: "PLACEHOLDER"` — zero TS references anywhere; the control was never implemented. Comparison: npm ships `--provenance`/Sigstore attestations verifiable pre-install; Ruflo's installer resembles VS Code Marketplace's weaker identity-only model (malicious-extension detections nearly quadrupled there in 2025, per ReversingLabs). Not selected tonight (SCAN); real gap for a future capabilities/security night.++## Scan Findings: automation++No in-repo cron/workflow drives the nightly dream-cycle trigger — it lives outside this repo, so tonight's confirmed 2-night no-run gap (2026-09-13/14, no branch/PR/issue for either date, verified via `git ls-remote`) is genuinely unexplained from repo config. The one visibility fix for this class of gap, `dream-cycle-backlog-guard.yml`, has itself sat as draft PR #3205 since 2026-09-05 (13+ days, zero activity) — the detector for "things go stale unmerged" is itself stale and unmerged. Separately: `hooks/README.md` documents 12 named background workers; the actual `ALLOWED_WORKERS` allowlist in `src/workers/index.ts` has 11 completely different names, zero overlap, and only 5 worker files exist on disk. Not selected tonight (SCAN); merge #3205 first, then treat the doc/implementation drift as a themed future `automation` night.++## Competitors Reviewed++Qdrant, Weaviate, Milvus, LanceDB, Mem0, Zep/Graphiti, Letta, LangGraph, Microsoft Agent Framework/AutoGen, CrewAI, OpenAI Agents SDK (memory-deep comparison); npm provenance/Sigstore, VS Code Marketplace (plugins scan); GitHub Actions scheduled-workflow reliability (automation scan).++## Additional candidates surfaced but not selected tonight (STEP 3.2)++Scored under `0.25·Ruflo_fit + 0.20·testability + 0.20·measurability + 0.15·production_value + 0.10·novelty + 0.10·reviewability`. Selected candidate (A, same-key upsert) scored **4.70**, highest of five.++1. **[4.35] B — HNSW binary/scalar quantization distance dispatch bug**: same class as merged PQ fix (#3093/#3094); no `binaryQuantizeDistance()`/`scalarQuantizeDistance()` exists, so those types fall through to generic cosine/euclidean on packed bitmasks/min-range-polluted arrays. Dead code today (never configured/tested), lowering urgency, not correctness risk. Strongest standalone second choice.+2. **[4.00] C — `sqlite-backend.ts` `UNIQUE(namespace,key)` + `ON CONFLICT DO UPDATE`**: closes the other backend's half of tonight's bug class; not the default path, left for follow-up.+3. **[3.35] D — Architectural**: two unequal quantizers exist (`@claude-flow/cli`'s real WASM RaBitQ vs. `@claude-flow/memory`'s broken hand-rolled binary/scalar) — gate the broken one or port the real one.+4. **[2.95] E — Consolidator threshold calibration**: replace the fixed 0.95 cosine cutoff with a calibrated percentile (arXiv:2605.08538 uses 0.559 at p99) or Mem0-style write-time classification. Needs a labeled corpus — out of scope tonight.++## Recommended Next Steps++1. **Merge tonight's linked draft PR** (human review required).+2. **Add a per-key async lock to `AgentDBAdapter.store()`/`bulkInsert()`** — the still-open, now-doubly-disclosed gap (adversarial critic, then human review): sequential re-stores and same-batch/batch-vs-pre-existing replacements are now safe, concurrent `Promise.all()` re-stores of the same key across separate calls are not.+3. **Fix candidate B** (HNSW binary/scalar quantization distance dispatch) — same mechanical shape as the already-reviewed PQ fix.+4. **Check `auto-memory-bridge.ts`'s hash-dedup path for the same TOCTOU race Mem0 just filed** (#6531) — structurally similar snapshot-then-insert shape, not verified either way tonight.+5. **Merge draft PR #3205** (`dream-cycle-backlog-guard.yml`) — the missing heartbeat for exactly the no-run-gap class tonight's automation scan re-confirmed.+6. **Reconcile `hooks/README.md`'s 12-worker roster against the actual 11-name `ALLOWED_WORKERS` allowlist** — a future `automation` night should resolve which is wrong.diff --git a/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts b/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.tsnew file mode 100644index 0000000000..9ace00c91a--- /dev/null+++ b/v3/@claude-flow/memory/src/agentdb-adapter-upsert.test.ts@@ -0,0 +1,252 @@+/**+ * Dream Cycle 2026-09-18 (memory) — regression tests for AgentDBAdapter.store()+ * same-(namespace,key) idempotency.+ *+ * Baseline bug: store() minted a fresh random id per call (createDefaultEntry ->+ * generateMemoryId()) and never looked up an existing entry under the same+ * (namespace, key) before writing. The prior occupant was left as an orphan:+ * unreachable via getByKey()/keyIndex, but still present in entries/+ * namespaceIndex/tagIndex, and — for embedded entries — still a live point in+ * the HNSW index, so search()/semanticSearch() returned stale duplicates+ * forever. Fixed by evicting the prior (namespace,key) occupant via a shared+ * evictEntry() primitive (also used by delete() and bulkInsert()) before+ * storing the new entry.+ *+ * Post-review hardening (same night): an adversarial human review found two+ * further public-path gaps the first pass missed — bulkInsert() bypassed the+ * same-key dedup entirely, and store() originally evicted the prior occupant+ * BEFORE validating the replacement's embedding, so a rejected addPoint()+ * (dimension mismatch, index full) would delete the old value and still fail,+ * losing data outright. Both are fixed below; the tests in this second block+ * cover exactly the five cases the review asked for.+ */+import { existsSync, mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { describe, it, expect, beforeEach, afterEach } from 'vitest';+import { AgentDBAdapter } from './agentdb-adapter.js';+import { createDefaultEntry } from './types.js';++function unitVector(dims: number, hotIndex: number): Float32Array {+ const v = new Float32Array(dims);+ v[hotIndex % dims] = 1;+ return v;+}++function allInNamespace(namespace: string) {+ return { type: 'prefix' as const, keyPrefix: '', namespace, limit: 1000 };+}++describe('AgentDBAdapter.store() — same-key upsert idempotency (Dream Cycle 2026-09-18)', () => {+ let adapter: AgentDBAdapter;++ beforeEach(async () => {+ adapter = new AgentDBAdapter({ dimensions: 8, cacheEnabled: true });+ await adapter.initialize();+ });++ it('a second store() under the same (namespace,key) does not leave two entries reachable', async () => {+ const first = createDefaultEntry({ key: 'profile', namespace: 'ns1', content: 'v1' });+ await adapter.store(first);++ const second = createDefaultEntry({ key: 'profile', namespace: 'ns1', content: 'v2' });+ await adapter.store(second);++ expect(first.id).not.toBe(second.id); // ids are independently random, as today++ const byKey = await adapter.getByKey('ns1', 'profile');+ expect(byKey?.content).toBe('v2');++ // The prior occupant must be fully gone, not just unreachable via getByKey.+ const stale = await adapter.get(first.id);+ expect(stale).toBeNull();++ const all = await adapter.query(allInNamespace('ns1'));+ expect(all.filter((e) => e.key === 'profile')).toHaveLength(1);+ });++ it('a second store() under the same (namespace,key) evicts the stale HNSW point, so search() never returns the old content', async () => {+ const first = createDefaultEntry({ key: 'fact', namespace: 'ns1', content: 'stale-fact' });+ first.embedding = unitVector(8, 0);+ await adapter.store(first);++ const second = createDefaultEntry({ key: 'fact', namespace: 'ns1', content: 'fresh-fact' });+ second.embedding = unitVector(8, 0); // same embedding direction, different content/id+ await adapter.store(second);++ const results = await adapter.search(unitVector(8, 0), { k: 10 });+ const contents = results.map((r) => r.entry.content);+ expect(contents).toContain('fresh-fact');+ expect(contents).not.toContain('stale-fact');+ expect(contents.filter((c) => c === 'fresh-fact')).toHaveLength(1);+ });++ it('storing a genuinely new (namespace,key) is unaffected (no eviction, both entries reachable)', async () => {+ const a = createDefaultEntry({ key: 'a', namespace: 'ns1', content: 'A' });+ const b = createDefaultEntry({ key: 'b', namespace: 'ns1', content: 'B' });+ await adapter.store(a);+ await adapter.store(b);++ expect((await adapter.get(a.id))?.content).toBe('A');+ expect((await adapter.get(b.id))?.content).toBe('B');+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(2);+ });++ it('the same (namespace,key) in two different namespaces are independent (no cross-namespace eviction)', async () => {+ const a = createDefaultEntry({ key: 'shared', namespace: 'ns1', content: 'from-ns1' });+ const b = createDefaultEntry({ key: 'shared', namespace: 'ns2', content: 'from-ns2' });+ await adapter.store(a);+ await adapter.store(b);++ expect((await adapter.getByKey('ns1', 'shared'))?.content).toBe('from-ns1');+ expect((await adapter.getByKey('ns2', 'shared'))?.content).toBe('from-ns2');+ });+});++describe('AgentDBAdapter same-key upsert — post-review hardening (Dream Cycle 2026-09-18)', () => {+ let adapter: AgentDBAdapter;++ beforeEach(async () => {+ adapter = new AgentDBAdapter({ dimensions: 8, cacheEnabled: true });+ await adapter.initialize();+ });++ it('a replacement rejected by HNSWIndex.addPoint() (dimension mismatch) leaves the prior value intact', async () => {+ const first = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'original' });+ first.embedding = unitVector(8, 0);+ await adapter.store(first);++ const bad = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'malformed' });+ bad.embedding = new Float32Array(4); // wrong dimension for this 8-dim adapter++ await expect(adapter.store(bad)).rejects.toThrow(/dimension mismatch/i);++ // The prior value must survive a rejected replacement -- not be deleted+ // and then fail to be replaced (the exact data-loss bug the review found).+ const byKey = await adapter.getByKey('ns1', 'k');+ expect(byKey?.content).toBe('original');+ expect(byKey?.id).toBe(first.id);+ expect(await adapter.get(first.id)).not.toBeNull();+ });++ it('bulkInsert() evicts a duplicate key within the same batch, last entry wins', async () => {+ const loser = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'first-in-batch' });+ const winner = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'last-in-batch' });++ await adapter.bulkInsert([loser, winner]);++ const byKey = await adapter.getByKey('ns1', 'k');+ expect(byKey?.content).toBe('last-in-batch');+ expect(byKey?.id).toBe(winner.id);++ expect(await adapter.get(loser.id)).toBeNull();+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(1);+ });++ it('bulkInsert() evicts a pre-existing store()d entry that a batch replaces', async () => {+ const preExisting = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'pre-existing' });+ preExisting.embedding = unitVector(8, 0);+ await adapter.store(preExisting);++ const replacement = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-batch' });+ replacement.embedding = unitVector(8, 0);+ await adapter.bulkInsert([replacement]);++ expect(await adapter.get(preExisting.id)).toBeNull();+ const results = await adapter.search(unitVector(8, 0), { k: 10 });+ const contents = results.map((r) => r.entry.content);+ expect(contents).toContain('from-batch');+ expect(contents).not.toContain('pre-existing');+ expect(await adapter.query(allInNamespace('ns1'))).toHaveLength(1);+ });++ it('a same-key replacement survives a persisted save/reload -- only the winner comes back', async () => {+ const dir = mkdtempSync(join(tmpdir(), 'agentdb-adapter-upsert-'));+ const persistencePath = join(dir, 'store');+ try {+ const first = new AgentDBAdapter({+ dimensions: 8,+ cacheEnabled: true,+ persistenceEnabled: true,+ persistencePath,+ });+ await first.initialize();++ const original = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'v1' });+ await first.store(original);+ const replacement = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'v2' });+ await first.store(replacement);+ await first.shutdown(); // persists to disk++ const reloaded = new AgentDBAdapter({+ dimensions: 8,+ cacheEnabled: true,+ persistenceEnabled: true,+ persistencePath,+ });+ await reloaded.initialize(); // loads from disk++ const byKey = await reloaded.getByKey('ns1', 'k');+ expect(byKey?.content).toBe('v2');+ expect(await reloaded.get(original.id)).toBeNull();+ expect(await reloaded.query(allInNamespace('ns1'))).toHaveLength(1);+ } finally {+ if (existsSync(dir)) rmSync(dir, { recursive: true, force: true });+ }+ });++ it('KNOWN LIMITATION (disclosed, not fixed tonight): truly concurrent store() calls for the same key can both proceed and leave two reachable entries', async () => {+ // No mutex/lock exists around the keyIndex.get() check + writes. Two+ // store() calls for the same (namespace,key), both parked mid-flight on+ // an embeddingGenerator await and released in the same tick, can each+ // resume and read keyIndex before the other has written to it, so+ // neither evicts the other. A plain setTimeout-based delay does NOT+ // reliably reproduce this: Node drains the microtask queue (including+ // every further await inside one resumed call) before the next timer+ // macrotask runs, so two independently-timed calls tend to fully+ // serialize instead of interleaving at the check (confirmed: an earlier+ // version of this test using setTimeout could not reproduce the gap).+ // Two externally-controlled gates, released back-to-back in the same+ // synchronous tick after both calls are already parked, force the+ // actual interleaving this test needs to demonstrate.+ let releaseA!: () => void;+ let releaseB!: () => void;+ const gateA = new Promise<void>((resolve) => { releaseA = resolve; });+ const gateB = new Promise<void>((resolve) => { releaseB = resolve; });++ const raced = new AgentDBAdapter({+ dimensions: 8,+ cacheEnabled: true,+ embeddingGenerator: (content) => (content === 'from-a' ? gateA : gateB).then(() => unitVector(8, 0)),+ });+ await raced.initialize();++ const a = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-a' });+ const b = createDefaultEntry({ key: 'k', namespace: 'ns1', content: 'from-b' });++ // Both calls run synchronously up to their embeddingGenerator await and+ // are now genuinely parked -- store() hasn't read keyIndex yet for+ // either. Releasing both gates back-to-back interleaves their resumption.+ const pA = raced.store(a);+ const pB = raced.store(b);+ releaseA();+ releaseB();+ await Promise.all([pA, pB]);++ // query()'s prefix search iterates keyIndex, which by construction holds+ // exactly one id per (namespace,key) -- it structurally cannot see an+ // orphan still sitting in `entries` under a different id, so it's the+ // wrong probe here (an earlier version of this test used it and always+ // saw "1", regardless of whether the race actually happened). get(id)+ // against each participant's own id is the correct probe, matching how+ // this file's very first test detects the sequential-case orphan.+ const aStillPresent = (await raced.get(a.id)) !== null;+ const bStillPresent = (await raced.get(b.id)) !== null;+ // Today: both survive concurrently (the bug this test documents) --+ // neither call's isReplacement check saw the other's write in time. If+ // this assertion ever starts failing because only one is still present,+ // the concurrency gap has been fixed -- update this test to assert the+ // fixed behavior instead of loosening it.+ expect([aStillPresent, bStillPresent]).toEqual([true, true]);+ });+});diff --git a/v3/@claude-flow/memory/src/agentdb-adapter.ts b/v3/@claude-flow/memory/src/agentdb-adapter.tsindex 793e2498ab..e14457d03e 100644--- a/v3/@claude-flow/memory/src/agentdb-adapter.ts+++ b/v3/@claude-flow/memory/src/agentdb-adapter.ts@@ -182,18 +182,50 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {entry.embedding = await this.config.embeddingGenerator(entry.content);}+ // Namespace is resolved once, up front, so the dedup lookup below and the+ // index updates further down agree on the same value.+ const namespace = entry.namespace || this.config.defaultNamespace;+ const keyIndexKey = `${namespace}:${entry.key}`;++ // Idempotent upsert-by-key (Dream Cycle 2026-09-18, hardened post-review):+ // entry.id is always a fresh random id (generateMemoryId()), so a second+ // store() under the same (namespace, key) previously left the prior+ // occupant as an orphan — unreachable via getByKey()/keyIndex, but still+ // live in entries/namespaceIndex/tagIndex and, for embedded entries,+ // still a point in the HNSW index, so search()/semanticSearch() returned+ // stale duplicates forever.+ const existingId = this.keyIndex.get(keyIndexKey);+ const isReplacement = existingId !== undefined && existingId !== entry.id;++ // Fallible precondition first: HNSWIndex.addPoint() can throw (dimension+ // mismatch, index full). Doing this BEFORE evicting the prior occupant+ // means a rejected write fails cleanly with the prior value still intact,+ // instead of deleting it and only then discovering the replacement can't+ // be indexed (a real data-loss bug an adversarial review caught).+ if (entry.embedding) {+ await this.index.addPoint(entry.id, entry.embedding);+ }++ // Evict the prior (namespace,key) occupant via the shared primitive also+ // used by bulkInsert() and delete(). keyIndex is intentionally left+ // alone here — it's overwritten to the new id a few lines down, and+ // clearing it in evictEntry() first would just be redone work (worse,+ // if evictEntry() read a since-mutated keyIndex it could delete the+ // mapping this call is about to set).+ if (isReplacement) {+ await this.evictEntry(existingId!, { touchKeyIndex: false });+ }+// Store in main storagethis.entries.set(entry.id, entry);// Update namespace index- const namespace = entry.namespace || this.config.defaultNamespace;if (!this.namespaceIndex.has(namespace)) {this.namespaceIndex.set(namespace, new Set());}this.namespaceIndex.get(namespace)!.add(entry.id);// Update key index- const keyIndexKey = `${namespace}:${entry.key}`;this.keyIndex.set(keyIndexKey, entry.id);// Update tag index@@ -204,11 +236,6 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {this.tagIndex.get(tag)!.add(entry.id);}- // Index embedding if available- if (entry.embedding) {- await this.index.addPoint(entry.id, entry.embedding);- }-// Update cacheif (this.config.cacheEnabled) {this.cache.set(entry.id, entry);@@ -320,6 +347,19 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {* Delete a memory entry*/async delete(id: string): Promise<boolean> {+ return this.evictEntry(id, { touchKeyIndex: true });+ }++ /**+ * Shared eviction primitive behind delete() and the same-key upsert paths+ * in store()/bulkInsert() (Dream Cycle 2026-09-18, post-review). Removes+ * an entry from entries/namespaceIndex/tagIndex/HNSW/cache. touchKeyIndex+ * is false when the caller is itself about to overwrite the+ * (namespace,key) -> id mapping to point at a replacement entry —+ * clearing it here first would either be redone work or, if the mapping+ * had already been repointed, would incorrectly delete the new mapping.+ */+ private async evictEntry(id: string, { touchKeyIndex }: { touchKeyIndex: boolean }): Promise<boolean> {const entry = this.entries.get(id);if (!entry) return false;@@ -330,8 +370,10 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {this.namespaceIndex.get(entry.namespace)?.delete(id);// Remove from key index- const keyIndexKey = `${entry.namespace}:${entry.key}`;- this.keyIndex.delete(keyIndexKey);+ if (touchKeyIndex) {+ const keyIndexKey = `${entry.namespace}:${entry.key}`;+ this.keyIndex.delete(keyIndexKey);+ }// Remove from tag indexfor (const tag of entry.tags) {@@ -458,6 +500,35 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {}}+ // Same-key upsert (Dream Cycle 2026-09-18, post-review): bulkInsert()+ // bypassed store()'s per-key dedup entirely, so a batch containing+ // duplicate (namespace,key) values -- or replacing an already-stored+ // key -- left old ids reachable via query()/search() while only+ // keyIndex pointed at the newest one. Computed from the pre-mutation+ // keyIndex snapshot and input order, before any Map below is touched:+ // the *last* entry per key in this batch wins (matching what N+ // sequential store() calls for the same inputs would produce), and any+ // id that isn't the winner for its key -- a pre-existing occupant or an+ // earlier same-key entry within this very batch -- is scheduled for+ // eviction once the winners are safely indexed.+ const keyOf = (e: MemoryEntry) => `${e.namespace || this.config.defaultNamespace}:${e.key}`;+ const winnerIdForKey = new Map<string, string>();+ for (const entry of entries) {+ winnerIdForKey.set(keyOf(entry), entry.id);+ }+ const idsToEvict = new Set<string>();+ for (const [key, winnerId] of winnerIdForKey) {+ const existingId = this.keyIndex.get(key);+ if (existingId && existingId !== winnerId) {+ idsToEvict.add(existingId);+ }+ }+ for (const entry of entries) {+ if (entry.id !== winnerIdForKey.get(keyOf(entry))) {+ idsToEvict.add(entry.id);+ }+ }+// Phase 2: Store all entries (skip individual cache updates)const embeddings: Array<{ id: string; embedding: Float32Array }> = [];@@ -472,9 +543,8 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {}this.namespaceIndex.get(namespace)!.add(entry.id);- // Update key index- const keyIndexKey = `${namespace}:${entry.key}`;- this.keyIndex.set(keyIndexKey, entry.id);+ // Update key index (last entry per key wins, per winnerIdForKey above)+ this.keyIndex.set(keyOf(entry), entry.id);// Update tag indexfor (const tag of entry.tags) {@@ -490,16 +560,30 @@ export class AgentDBAdapter extends EventEmitter implements IMemoryBackend {}}- // Phase 3: Batch index embeddings+ // Phase 3: Batch index embeddings. Deliberately BEFORE Phase 3.5's+ // eviction below, mirroring store()'s ordering: if addPoint() throws+ // (dimension mismatch, index full) for any winner, nothing superseded+ // has been evicted yet, so a failed bulkInsert() doesn't lose data that+ // a failed single store() wouldn't have lost either.for (let i = 0; i < embeddings.length; i += batchSize) {const batch = embeddings.slice(i, i + batchSize);await Promise.all(batch.map(({ id, embedding }) => this.index.addPoint(id, embedding)));}- // Phase 4: Batch cache update (only populate hot entries)+ // Phase 3.5: evict superseded occupants (pre-existing and intra-batch),+ // now that every winner is fully written and indexed. keyIndex is left+ // alone -- Phase 2 already points it at the correct winner for each key.+ for (const id of idsToEvict) {+ await this.evictEntry(id, { touchKeyIndex: false });+ }++ // Phase 4: Batch cache update (only populate hot entries, winners only —+ // an evicted loser must not be left warm in the cache under its own id)if (this.config.cacheEnabled && entries.length <= this.config.cacheSize) {for (const entry of entries) {- this.cache.set(entry.id, entry);+ if (!idsToEvict.has(entry.id)) {+ this.cache.set(entry.id, entry);+ }}}