Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

dream(memory): #4137 wire delete-sync into AutoMemoryBridge.importFromAutoMemory() (evaluated, ACCEPT) (#4138)

rUv9cf2ad57ba36
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md
index 8e82d30fcd..7b87569eef 100644
--- a/docs/dream-cycle/LEDGER.md
+++ b/docs/dream-cycle/LEDGER.md
@@ -231,6 +231,7 @@ as a first-class, high-priority candidate, not a background note.
| 2026-10-09 | swarm | `MessageBus.handleDeliveryError()`'s retry re-queue used the broadcast sentinel `message.to === 'broadcast'` as the queue key instead of the real subscriber `agentId` `enqueue()` had fanned the message out to β€” `processQueues()` skips any unsubscribed queue, so a failing broadcast subscriber's retries and eventual `message.failed` were silently dropped into an orphaned queue leaking until `shutdown()`; explicitly disclosed as a "KNOWN LIMITATION (pre-existing, not fixed here)" by the immediately-preceding #3562/#3563 rewrite, closed tonight by threading `subscription.agentId` through `deliverMessage()`β†’`handleDeliveryError()`β†’the re-queue | #3928 | #3929 | yes | ACCEPT | 2 new discriminating tests, baseline-fails/candidate-passes via stash isolation (independently reproduced byte-identical by an adversarial critic's own stash/pop cycle); full `@claude-flow/swarm` suite 274/274β†’276/276 (+2 new), 0 regressions, `tsc --noEmit` clean both ways; healthy co-subscriber in the same broadcast unaffected (explicit regression guard); independent adversarial critique CONFIRMED, no blocking issues | 1b7e818e... | 10-01 #3599 MERGED 2026-10-07; 10-02..10-03/10-05..10-08 open/draft or closed-not-merged (see 2026-10-09 note above); 10-04 no-run gap; also newly confirmed MERGED this session: #3339 (09-16), #3539 (09-29) |
| 2026-10-10 | performance | `fallbackLouvain()`'s `communityTotal` map was rebuilt via a full O(n) scan of every node, once per node, once per local-moving pass β€” O(nΒ²) per pass β€” instead of maintained incrementally like networkx's `Stot`/python-louvain's `status.degrees` (O(1) per accepted move); this is the *only* community-detection path that ever runs (`loadRuVector()`'s native-acceleration checks for `hooks_graph_mincut`/`hooks_graph_cluster`/`@ruvector/wasm`'s `GraphAnalyzer`, none of which exist in any published `ruvector`/`@ruvector/wasm` version, confirmed by direct inspection of the installed `[email protected]`'s real exports β€” correcting 2026-10-09's own "wire the real exports in" recommendation, which had the wrong premise) | #3976 | #3977 | yes | ACCEPT | 4 new discriminating tests (3 correctness + 1 complexity-growth) with a frozen pre-fix copy inline for CI-safe comparison; stash-isolated baseline-fails/candidate-passes on the growth assertion, independently reproduced by an adversarial critic's own stash/pop cycle; standalone `tsx` probe: baseline 10.37msβ†’837.29ms (~81x growth, N=200β†’1600), candidate 5.00msβ†’22.33ms (~4.5x growth), per-N speedup 2.1xβ†’37.5x; full `@claude-flow/cli` suite failed-file set byte-identical baseline vs. candidate (200 vs. 199 β€” the one difference is the new test file itself), `tsc --noEmit` byte-identical (498 pre-existing errors); independent adversarial critique CONFIRMED, no blocking issues | 81e21481... | 10-01 #3599 MERGED; 10-02/10-03/10-06/10-07/10-08 open/draft, none stale; 10-04 no-run gap; 10-05 #3754 closed as duplicate of #3517 (not a fresh rejection β€” same fix had already shipped) |
| 2026-10-11 | security | settings.json risk scanner (#3043) never ran on the one path that fires automatically on every real session boot β€” `hook-handler.cjs`'s `session-restore` command (Claude Code's own `SessionStart` hook) β€” a gap #3835 (10-06) itself disclosed as a future-night item; ported the TS scanner's logic (no build step for `.cjs`) into `scanSessionStartSettingsForRisk()`, validated against a real 2026 npm-worm campaign ("Mini Shai-Hulud") and CVE-2026-25725, both of which use this exact persistence mechanism | #4052 | #4053 | yes | ACCEPT-scoped (post-critique) | 12 new discriminating tests, stash-isolated 10/12 fail baseline β†’ 12/12 pass candidate; adversarial critic found and this session fixed a non-SyntaxError `JSON.parse` failure that would have aborted session-restore itself (narrowed catch copied from the opt-in #3835 MCP path onto an unconditional-boot path); related suites 72/72 pass; full `@claude-flow/cli` suite 207 pre-existing environmental failures unchanged (unbuilt sibling packages), zero touching this change | 016fe07f... | 10-01 #3599 MERGED; 10-02/10-03/10-06/10-07/10-08 open/draft, none stale; 10-04 no-run gap; 10-05 #3754 closed as duplicate; 10-09 #3929, 10-10 #3977 open/draft, not stale. Prior gist (10-10, performance) scored 10/10 (2+2+2+2+1(1489 words)+1) |
+| 2026-10-13 | memory | `AutoMemoryBridge.importFromAutoMemory()` was additive-only β€” never deleted an AgentDB `auto-memory` entry whose source heading, or whole source file, was removed, so deleted content stayed retrievable via semantic search indefinitely; fixed via a per-source-file key diff against the existing single bulk query (zero schema change, mirrors `agentdb-adapter.ts`'s 09-18 same-key eviction pattern) plus a second pass for whole-file removal, each with a false-full-wipe guard. Handed in by 10-12's own memory scan (#4080 Β§11); SCAN: plugins (`PluginStoreConfig`'s documented security defaults never read anywhere; confirmed last night's `plugin.register` test-harness gap), automation (confirmed no in-repo cron drives the nightly trigger; found an un-hardened `worker/dispatch` MCP tool in `@claude-flow/mcp` fabricating audit/benchmark results with no disclosure, unlike its hardened CLI sibling) | #4137 | #4138 | yes | ACCEPT-scoped (post-critique) | 7 new discriminating tests, stash-isolated: baseline fails all 7, candidate passes all 7; full `@claude-flow/memory` suite baseline 606/607 β†’ candidate 611/614 (+7 new; 2 extra failures are CPU-contention-sensitive timing tests in untouched files, confirmed non-deterministic by not reproducing on a repeat baseline run); `tsc --noEmit` 0 errors both ways; independent adversarial critique CONFIRMED-WITH-CAVEATS β€” critic found and this session fixed forward a real undisclosed gap (whole-file-removed-from-disk entries were a permanent orphan) plus a vacuous test assertion, before this commit | e32bb677... | 10-09 #3929 MERGED; 10-10 #3977 MERGED; 10-11 #4053 MERGED; 10-12 #4080/#4081 open/draft, not stale (0d old). Prior gist (10-12, intelligence) scored 10/10 (2+2+2+2+1+1) |
**Note (2026-10-14):** 2026-10-12 (intelligence, #4080/#4081 β€” SONA learning-state persistence wiring into `LearningBridge`, ACCEPT) and 2026-10-13 (memory, #4137/#4138 β€” `AutoMemoryBridge` delete-sync wiring, ACCEPT) both ran (real branches+issues+draft-PRs confirmed via `git ls-remote --heads origin "dream/*"` and `pull_request_read`, not inferred) but aren't transcribed as their own rows here β€” same "ledger rows only reach `main` via PR merge" structural gap noted since 2026-08-19. Both open/draft, neither stale (2d/1d old respectively). Prior gist (10-13, memory) scored 10/10 (2+2+2+2+1(~1490 words)+1). GitHub-verified fates for the full trailing week before tonight's own row: 10-05 #3754 **closed, not merged** (duplicate of #3517); 10-06 #3835, 10-07 #3879, 10-08 #3909 open/draft, none stale; 10-09 #3929, 10-10 #3977, 10-11 #4053 **MERGED**; 10-12 #4081, 10-13 #4138 open/draft, not stale. Swarm-surface historical merge rate: 6/6 (08-24, 08-29, 09-09, 09-19, 09-29, 10-09) once reviewed.
| 2026-10-14 | swarm | `TaskOrchestrator.getQueuePosition()`'s priority map used keys `critical/high/normal/low/background` β€” `TaskPriority` is `critical/high/medium/low`, and `createTask()` defaults to `'medium'`, so the lookup always fell through to 0, ranking default-priority tasks behind even `'low'` in the `taskQueuedEvent` position field; real dispatch (`getNextTask`/`getPriorityQueue`) uses a separate, already-correct map and was unaffected. Scan also resolved a direct contradiction between the 10-09 and 10-10 nights: real `[email protected]` graph exports (`minCut`/`louvainCommunities`/`buildGraph`) DO exist under different names than `graph-analyzer.ts`'s dead detection checks β€” 10-09 was right, 10-10's "nothing real exists" was wrong | #4173 | #4174 | yes | ACCEPT | 3 new discriminating tests, stash-isolated baseline 278/280 (2 predicted failures) β†’ candidate 280/280, 0 other regressions; `tsc --noEmit` clean both ways (fix also adds compile-time exhaustiveness via `Record<TaskPriority, number>`); independent adversarial critique CONFIRMED, no blocking issues | fab09db0... | 10-05 closed-not-merged (dup); 10-06/10-07/10-08 open/draft, none stale; 10-09/10-10/10-11 MERGED; 10-12/10-13 open/draft, not stale |
diff --git a/docs/dream-cycle/dream-gist-2026-10-13.md b/docs/dream-cycle/dream-gist-2026-10-13.md
new file mode 100644
index 0000000000..5dd9721ec0
--- /dev/null
+++ b/docs/dream-cycle/dream-gist-2026-10-13.md
@@ -0,0 +1,172 @@
+# Memory SOTA Report β€” 2026-10-13
+
+TL;DR: Ruflo's `AutoMemoryBridge.importFromAutoMemory()` (the production path
+wired into every Claude Code session via `SessionStart`) is additive-only β€”
+it never deletes an AgentDB entry whose source markdown heading was removed.
+Deleted content stays retrievable via semantic search indefinitely. This is
+genuinely behind 2026 practice, not merely typical of it: LangChain's
+`index()` API ships a direct, official blueprint (`cleanup="scoped_full"`)
+for exactly this per-source diff-and-delete pattern. Fixed tonight with a
+~100-line, zero-schema-migration patch that mirrors an existing Ruflo
+precedent (`agentdb-adapter.ts`'s same-key eviction logic).
+
+## What's New in 2026
+
+| Finding | Source | Confidence |
+|---|---|---|
+| LangChain's `index()` `cleanup="scoped_full"` mode is the official, reproducible pattern for per-source diff-and-delete sync into a vector store | langchain docs, maintained through 2025/26 | A |
+| Zep/Graphiti's bi-temporal "never hard-delete, mark `invalid_at`" is the considered 2026 agent-memory answer to unreliable LLM-judged deletes | arXiv:2501.13956 (Jan 2025); Thoughtworks Radar "Trial" Apr 2026 | A |
+| Mem0's own repro shows ~25% malformed-JSON failures when an LLM is trusted to decide ADD/UPDATE/DELETE β€” the stated reason the field avoids naive hard-delete-on-diff | arXiv:2606.15903 (2026) | B |
+| No mainstream vector DB (Qdrant/Weaviate/Milvus/LanceDB/Vespa) auto-detects source deletion β€” a deliberate layering choice, not an oversight (the DB has no concept of "source") | Qdrant/Weaviate/Vespa official docs, 2026 | B |
+
+## Ruflo Current Capability
+
+`importFromAutoMemory()` (`v3/@claude-flow/memory/src/auto-memory-bridge.ts:380-495`)
+keys each entry `auto-memory:${file}:${heading}` and tags
+`metadata.sourceFile`/`metadata.heading`, but only ever *inserts*: it diffs
+new content against a content-hash set to skip/insert, and never computes
+"keys AgentDB has for this file that this pass didn't re-see." Confirmed
+by code reading and grep (zero `delete`/`bulkDelete` calls in the file,
+despite 15+ other files in the package already using
+`IMemoryBackend.bulkDelete`). `curateIndex()`'s `MEMORY.md` rebuild reads
+topic files directly, so it's unaffected β€” the leak is specifically
+semantic search / `memory query` against the `auto-memory` namespace
+surfacing content a human deleted, forever.
+
+## Competitor Comparison
+
+| System | Handles source-deletion sync? | Mechanism | Grade |
+|---|---|---|---|
+| LangChain `index()` API | **Yes** | content-hash diff + `cleanup="full"`/`scoped_full"` deletes anything the latest pass didn't return | A |
+| Zep / Graphiti | Yes (by design, non-destructive) | bi-temporal edges; contradicted facts marked `invalid_at`, never hard-deleted | A |
+| Mem0 | Partial, fragile | LLM judges ADD/UPDATE/DELETE per call; ~25% malformed-JSON in independent repro | B |
+| Qdrant / Weaviate / Milvus / LanceDB / Vespa | No (deliberate) | deletion primitives exist (TTL, filter-delete, tombstone+compaction) but no source-diff logic β€” "no concept of source" is the stated design boundary | A/B |
+| LangGraph store | TTL only, not source-aware | `store.ttl`/`sweep_interval_minutes`, not retroactive | A |
+| AutoGen / OpenAI Agents SDK | No native answer | delegated to third-party bolt-ons (Mem0 etc.) | β€” |
+
+Ruflo's gap is genuinely behind the honest 2026 baseline at the layer that
+matters: raw vector DBs are no worse off, but Ruflo's bridge *is* the
+orchestration/agent-memory layer, where this is an active, published,
+already-has-reference-designs problem (LangChain ships the recipe).
+
+## Hypothesis
+
+> Given a workload of N previously-imported auto-memory markdown files in
+> AgentDB's `auto-memory` namespace, when `importFromAutoMemory()` is
+> extended to diff each file's current parsed heading-keys against the
+> AgentDB entries previously tagged with that file's `metadata.sourceFile`
+> and delete entries no longer present, then entries for headings removed
+> from the source file should be deleted from AgentDB within one import
+> call (baseline: they persist indefinitely), subject to: (a) entries in
+> other namespaces or other source files are unaffected; (b) a per-file
+> parse yielding zero entries for non-empty content does not trigger
+> deletion (false-full-wipe guard); (c) a per-file read failure does not
+> abort processing of other files and does not trigger deletion for that
+> file; (d) all existing tests remain green.
+
+Frozen before evaluation; not modified after.
+
+## Benchmarks
+
+Not a performance candidate β€” this is a correctness/wiring fix (same class
+as nearly every ACCEPT in the ledger since 2026-08-18). No benchmark corpus
+applies; evaluation is via discriminating unit tests + full-suite
+regression, as for every prior wiring/correctness fix in this ledger.
+
+## Evaluation
+
+Real evaluator: Vitest 4.1.8, deterministic, $0, zero LLM calls.
+
+- `auto-memory-bridge.test.ts`: **7 new discriminating tests** (delete on
+ removed heading; no-op when heading set unchanged; per-file scoping β€”
+ same heading text in another file untouched; false-wipe guard on
+ non-empty-zero-entry parse; per-file read failure caught without
+ aborting the batch or deleting that file's own entries; delete-sync for
+ a file removed entirely from disk, not just emptied; suspicious-empty-
+ directory guard on the same). Stash-isolated (`git diff`/`checkout`/
+ `apply` on the source file only, tests left in place): **baseline fails
+ all 7**, candidate passes all 7.
+- Full `@claude-flow/memory` suite: baseline 606/607 (1 pre-existing
+ chmod-under-root flake); candidate 611/614 (+7 new tests; same chmod
+ flake plus 2 CPU-contention-sensitive timing tests β€”
+ `consolidator-embedding-benchmark.test.ts` and
+ `controller-registry.test.ts`'s 500ms init-time assertion β€” in files
+ this diff does not touch; reproduced failing in isolation, and
+ independently confirmed non-deterministic by *not* reproducing on a
+ second full-suite baseline run in this same sandbox).
+- `tsc --noEmit`: 0 errors, both ways.
+
+**Adversarial critique**: independent critic reviewed the actual diff
+(not just the design), verdict **CONFIRMED-WITH-CAVEATS**, then
+re-verified after caveats were addressed:
+1. *Fixed*: a source file deleted entirely from disk (not emptied) was
+ never visited by the per-file loop, so its entries were a permanent,
+ undisclosed orphan β€” closed by a second delete-sync pass over
+ `existingByFile` (already fetched, no extra query) diffed against the
+ current directory listing, with its own suspicious-empty-directory
+ guard.
+2. *Fixed*: the try/catch around the per-file read did not also cover
+ `parseMarkdownEntries()` β€” widened to cover both.
+3. *Fixed*: the read-failure test asserted "doesn't delete that file's
+ entries" without ever seeding an old entry for that file, so the
+ assertion was vacuously true β€” now seeds `broken.md` with a stale
+ entry first.
+4. *Disclosed, not fixed*: `fetchExistingAutoMemoryIndex()`'s pre-existing
+ `limit: 10_000` cap (unchanged by this diff) now also bounds
+ delete-sync visibility at that same scale β€” inherited, not introduced,
+ flagged for a future night if the `auto-memory` namespace grows past it.
+
+## Darwin Results
+
+Skipped β€” binary correctness/wiring fix, no continuous parameter with a
+fitness gradient for Darwin's real interface to search over. Same skip
+class as nearly every accepted fix since 2026-08-18.
+
+## SOTA Proof & Witness
+
+```
+Session commit: 55e2c9011c20d25e347a175427ca8084dacde40c
+Gist SHA-256: b377c4bfed43cceb8589ac54c054f17642bc2944c6f67019072ce85b2e96ded1
+Witness stamp: e32bb6776f3d0d96a23dd989a8249de23ed68f757a750d21bd4d4b449c8bfd7b
+```
+
+Verifier: strip the three values above back to `PENDING`, SHA-256 this
+file, concatenate with the session commit, SHA-256 again β€” must equal the
+witness stamp. See the paired issue's own Witness section for the filled
+values, computed once this file reached its final content.
+
+## Recommended Next Steps
+
+1. **Port the same diff-and-delete pattern to the sibling "edited-content"
+ leak**: editing (not removing) a heading's body also leaves the
+ pre-edit row forever retrievable under a stale content-hash β€” same root
+ cause, same code path, flagged by tonight's deep-research pass as
+ candidate #2 (composite 4.00).
+2. **Fix the `scripts/live-ruflo-mods-options.sh` single-plugin-dir gap**
+ (plugins SCAN): `plugin.register` structurally cannot fire under the
+ current live-verification script because it only ever loads one plugin.
+ Point `--plugin-dir` at the parent `plugins/` directory (or add a
+ second flag) to resolve last night's disclosed ADR-451 ambiguity.
+3. **Reconcile or wire up `PluginStoreConfig`'s dead security fields**
+ (plugins SCAN): `minTrustLevel`, `blockedPlugins`,
+ `requirePermissionPrompt` etc. are documented in `SECURITY.md` as
+ default-secure behavior but are never read anywhere β€” the real
+ enforcement path (`trust-policy.ts`, #3557) ignores them entirely.
+ Either wire them in or correct the documentation.
+4. **Harden or retire the un-hardened `worker/dispatch` MCP tool**
+ (automation SCAN): `@claude-flow/mcp`'s `worker/dispatch`/`worker/status`
+ (distinct from the already-hardened `hooks_worker-dispatch` in
+ `@claude-flow/cli`) returns hardcoded, input-independent fabricated
+ results for all 12 workers (e.g. `executeAudit()` always reports
+ `2 high / 5 medium / 8 low` vulnerabilities regardless of the real repo
+ state) with no disclosure field β€” the exact dishonesty class #1700/#1845
+ already fixed on the sibling tool, un-ported to this second live surface.
+ Worth its own `automation` DEEP night.
+5. **Add a same-day "missed dream-cycle run" alert** (automation SCAN):
+ no in-repo cron/workflow drives the nightly trigger (confirmed by grep,
+ again, tonight), so a fully-silent no-run night (seen 10 times since
+ 2026-08-20, including 2026-10-04) is currently only discovered days
+ later by a human/agent re-deriving it from `git ls-remote`. A small
+ daily-scheduled workflow checking for today's `dream/<date>-*` branch
+ would close that discovery gap, even though it can't fix the external
+ scheduler itself.
diff --git a/v3/@claude-flow/cli/package.json b/v3/@claude-flow/cli/package.json
index 56bfb8fc6a..593c345a62 100644
--- a/v3/@claude-flow/cli/package.json
+++ b/v3/@claude-flow/cli/package.json
@@ -126,11 +126,11 @@
"ws": "^8.21.0",
"yaml": "^2.8.0",
"zod": "^3.22.0",
- "@claude-flow/memory": "3.0.6"
+ "@claude-flow/memory": "3.0.7"
},
"optionalDependencies": {
"@agntcy/slim-bindings": "2.0.0-alpha.5",
- "@claude-flow/memory": "3.0.6",
+ "@claude-flow/memory": "3.0.7",
"@metaharness/darwin": "~0.12.0",
"@metaharness/flywheel": "~0.1.10",
"@metaharness/radio": "~0.1.0",
diff --git a/v3/@claude-flow/memory/.gitignore b/v3/@claude-flow/memory/.gitignore
index bf106f213a..fd7c739e3c 100644
--- a/v3/@claude-flow/memory/.gitignore
+++ b/v3/@claude-flow/memory/.gitignore
@@ -28,3 +28,4 @@ benchmarks/longmemeval/results/*.md
# Node modules (defensive β€” should never appear here in a workspace)
node_modules/
+agentdb.rvf.idmap.json
diff --git a/v3/@claude-flow/memory/package.json b/v3/@claude-flow/memory/package.json
index 85d0a81cd9..d4c8a5ea59 100644
--- a/v3/@claude-flow/memory/package.json
+++ b/v3/@claude-flow/memory/package.json
@@ -1,6 +1,6 @@
{
"name": "@claude-flow/memory",
- "version": "3.0.6",
+ "version": "3.0.7",
"type": "module",
"description": "Memory module - AgentDB unification, HNSW indexing, vector search, hybrid SQLite+AgentDB backend (ADR-009)",
"main": "dist/index.js",
diff --git a/v3/@claude-flow/memory/src/auto-memory-bridge.test.ts b/v3/@claude-flow/memory/src/auto-memory-bridge.test.ts
index f47eeae6c1..6e9aa154ab 100644
--- a/v3/@claude-flow/memory/src/auto-memory-bridge.test.ts
+++ b/v3/@claude-flow/memory/src/auto-memory-bridge.test.ts
@@ -1015,6 +1015,227 @@ Already in DB
});
});
+ describe('importFromAutoMemory - delete-sync (Dream Cycle 2026-10-13)', () => {
+ // #4080 (2026-10-12) flagged importFromAutoMemory() as additive-only: it
+ // never deletes an AgentDB entry whose source heading was removed from
+ // the markdown file. These tests are the baseline-fails/candidate-passes
+ // discriminating set for that fix.
+
+ it('deletes the AgentDB entry for a heading removed from its source file', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'old-1',
+ key: 'auto-memory:test.md:Old',
+ metadata: { sourceFile: 'test.md', heading: 'Old', memoryDir: testDir, contentHash: 'stale-hash' },
+ },
+ ]);
+ (backend.bulkDelete as any).mockImplementation(async (ids: string[]) => ids.length);
+
+ fsSync.writeFileSync(path.join(testDir, 'test.md'), '## New\nNew content\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).toHaveBeenCalledWith(['old-1']);
+ expect(result.deleted).toBe(1);
+ });
+
+ it('does not delete an entry for an unchanged heading (no-op case)', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'x-1',
+ key: 'auto-memory:same.md:Keep',
+ metadata: { sourceFile: 'same.md', heading: 'Keep', memoryDir: testDir, contentHash: hashContent('Keep me') },
+ },
+ ]);
+
+ fsSync.writeFileSync(path.join(testDir, 'same.md'), '## Keep\nKeep me\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ expect(result.deleted).toBe(0);
+ });
+
+ it('scopes deletion per source file β€” a same-named heading in another file is untouched', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'a-1',
+ key: 'auto-memory:a.md:Shared',
+ metadata: { sourceFile: 'a.md', heading: 'Shared', memoryDir: testDir, contentHash: hashContent('Content A') },
+ },
+ {
+ id: 'b-1',
+ key: 'auto-memory:b.md:Shared',
+ metadata: { sourceFile: 'b.md', heading: 'Shared', memoryDir: testDir, contentHash: 'stale-b-hash' },
+ },
+ ]);
+ (backend.bulkDelete as any).mockImplementation(async (ids: string[]) => ids.length);
+
+ // a.md keeps its '## Shared' heading; b.md's was removed.
+ fsSync.writeFileSync(path.join(testDir, 'a.md'), '## Shared\nContent A\n', 'utf-8');
+ fsSync.writeFileSync(path.join(testDir, 'b.md'), '## Other\nContent B\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).toHaveBeenCalledWith(['b-1']);
+ expect(result.deleted).toBe(1);
+ });
+
+ it('refuses to delete-sync when a non-empty file parses to zero entries (guards against a false full wipe)', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'ghost-1',
+ key: 'auto-memory:ghost.md:Something',
+ metadata: { sourceFile: 'ghost.md', heading: 'Something', memoryDir: testDir, contentHash: 'h' },
+ },
+ ]);
+
+ // Frontmatter-only file: non-empty raw content, but the body parses to
+ // zero entries (no '##' headings, nothing left after trimming).
+ fsSync.writeFileSync(path.join(testDir, 'ghost.md'), '---\nname: ghost\n---\n\n', 'utf-8');
+
+ let skipEvent: any;
+ bridge.on('import:delete-skipped', (e) => { skipEvent = e; });
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ expect(result.deleted).toBe(0);
+ expect(skipEvent).toEqual({ file: 'ghost.md', reason: 'suspicious-empty-parse' });
+ });
+
+ it('catches a per-file read failure without deleting that file\'s entries or aborting the rest of the batch', async () => {
+ // A directory named *.md matches the readdirSync filter but throws
+ // EISDIR on readFile β€” a read failure that is NOT permission-dependent
+ // (unlike chmod, which no-ops when the test runner is root).
+ // Pre-seed an existing entry for 'broken.md' itself, so "doesn't delete
+ // that file's entries" is actually exercised rather than vacuously true
+ // (adversarial critic finding: the prior version of this test had no
+ // old entries for broken.md, so the bulkDelete-not-called assertion
+ // would have passed even without the catch block).
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'broken-1',
+ key: 'auto-memory:broken.md:Stale',
+ metadata: { sourceFile: 'broken.md', heading: 'Stale', memoryDir: testDir, contentHash: 'h' },
+ },
+ ]);
+ fsSync.mkdirSync(path.join(testDir, 'broken.md'));
+ fsSync.writeFileSync(path.join(testDir, 'good.md'), '## Still Here\nContent\n', 'utf-8');
+
+ let errorEvent: any;
+ bridge.on('import:file-error', (e) => { errorEvent = e; });
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(errorEvent).toBeDefined();
+ expect(errorEvent.file).toBe('broken.md');
+ expect(result.files).toContain('good.md');
+ expect(result.files).not.toContain('broken.md');
+ // broken.md IS still present on disk (as a directory) β€” it's just
+ // unreadable β€” so it must NOT be treated as "removed from disk" either.
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ });
+
+ it('deletes all of a source file\'s entries when the file is removed entirely from disk (not just emptied)', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'gone-1',
+ key: 'auto-memory:vanished.md:A',
+ metadata: { sourceFile: 'vanished.md', heading: 'A', memoryDir: testDir, contentHash: 'h1' },
+ },
+ {
+ id: 'gone-2',
+ key: 'auto-memory:vanished.md:B',
+ metadata: { sourceFile: 'vanished.md', heading: 'B', memoryDir: testDir, contentHash: 'h2' },
+ },
+ {
+ id: 'kept-1',
+ key: 'auto-memory:present.md:C',
+ metadata: { sourceFile: 'present.md', heading: 'C', memoryDir: testDir, contentHash: hashContent('still here') },
+ },
+ ]);
+ (backend.bulkDelete as any).mockImplementation(async (ids: string[]) => ids.length);
+
+ // Only present.md exists on disk this run; vanished.md was deleted.
+ fsSync.writeFileSync(path.join(testDir, 'present.md'), '## C\nstill here\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ const deletedIds = (backend.bulkDelete as any).mock.calls[0][0];
+ expect(new Set(deletedIds)).toEqual(new Set(['gone-1', 'gone-2']));
+ expect(result.deleted).toBe(2);
+ });
+
+ it('refuses the whole-file-removed delete-sync when the memory dir looks suspiciously empty', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'x-1',
+ key: 'auto-memory:a.md:Heading',
+ metadata: { sourceFile: 'a.md', heading: 'Heading', memoryDir: testDir, contentHash: 'h' },
+ },
+ ]);
+
+ // No .md files written to testDir at all this run.
+ let skipEvent: any;
+ bridge.on('import:delete-skipped', (e) => { skipEvent = e; });
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ expect(result.deleted).toBe(0);
+ expect(skipEvent).toEqual({ file: '*', reason: 'suspicious-empty-directory' });
+ });
+
+ it('scopes delete-sync to this memoryDir β€” an entry written by a different memoryDir sharing the same backend is untouched (review finding)', async () => {
+ // The same backend can be shared across multiple memory dirs β€” e.g. the
+ // SessionStart hook's main dir vs. a worktree dir from
+ // worktreeMemoryDir(). Importing one dir must never treat the other
+ // dir's same-named file/heading as "gone" and delete it.
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'other-dir-1',
+ key: 'auto-memory:shared.md:Heading',
+ metadata: {
+ sourceFile: 'shared.md',
+ heading: 'Heading',
+ memoryDir: '/some/other/worktree/memory/dir',
+ contentHash: 'h',
+ },
+ },
+ ]);
+
+ // testDir has no 'shared.md' at all β€” if the other dir's entry were
+ // eligible, both the per-file diff and the whole-file-removed pass
+ // would delete it.
+ fsSync.writeFileSync(path.join(testDir, 'present.md'), '## Still Here\nContent\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ expect(result.deleted).toBe(0);
+ });
+
+ it('a legacy entry with no recorded memoryDir is never delete-sync eligible', async () => {
+ (backend.query as any).mockResolvedValue([
+ {
+ id: 'legacy-1',
+ key: 'auto-memory:legacy.md:Old',
+ metadata: { sourceFile: 'legacy.md', heading: 'Old', contentHash: 'h' }, // no memoryDir field
+ },
+ ]);
+
+ // legacy.md is absent from disk β€” it would read as "removed from disk"
+ // if the legacy entry were eligible for delete-sync.
+ fsSync.writeFileSync(path.join(testDir, 'present.md'), '## Still Here\nContent\n', 'utf-8');
+
+ const result = await bridge.importFromAutoMemory();
+
+ expect(backend.bulkDelete).not.toHaveBeenCalled();
+ expect(result.deleted).toBe(0);
+ });
+ });
+
describe('curateIndex - edge cases', () => {
it('should skip index write when topic files have no extractable summaries', async () => {
// #1556: curateIndex must be non-destructive when there's nothing to
diff --git a/v3/@claude-flow/memory/src/auto-memory-bridge.ts b/v3/@claude-flow/memory/src/auto-memory-bridge.ts
index 5fb958a50d..6526be72a0 100644
--- a/v3/@claude-flow/memory/src/auto-memory-bridge.ts
+++ b/v3/@claude-flow/memory/src/auto-memory-bridge.ts
@@ -128,6 +128,9 @@ export interface ImportResult {
/** Number of entries skipped (already in AgentDB) */
skipped: number;
+ /** Number of AgentDB entries deleted because their source heading was removed */
+ deleted: number;
+
/** Files processed */
files: string[];
@@ -379,7 +382,7 @@ export class AutoMemoryBridge extends EventEmitter {
const memoryDir = this.config.memoryDir;
if (!existsSync(memoryDir)) {
- return { imported: 0, skipped: 0, files: [], durationMs: 0 };
+ return { imported: 0, skipped: 0, deleted: 0, files: [], durationMs: 0 };
}
let imported = 0;
@@ -388,19 +391,46 @@ export class AutoMemoryBridge extends EventEmitter {
const files = readdirSync(memoryDir).filter(f => f.endsWith('.md'));
- // Pre-fetch existing content hashes to avoid N queries
- const existingHashes = await this.fetchExistingContentHashes();
+ // Pre-fetch existing content hashes + per-file key->id index to avoid N queries
+ const { hashes: existingHashes, byFile: existingByFile } =
+ await this.fetchExistingAutoMemoryIndex();
- // Batch entries for bulk insert
+ // Batch entries for bulk insert / stale ids for bulk delete
const batch: MemoryEntry[] = [];
+ const deleteBatch: string[] = [];
for (const file of files) {
const filePath = path.join(memoryDir, file);
- const content = await fs.readFile(filePath, 'utf-8');
- const entries = parseMarkdownEntries(content);
+ const oldKeysForFile = existingByFile.get(file);
+
+ let content: string;
+ let entries: ParsedEntry[];
+ try {
+ content = await fs.readFile(filePath, 'utf-8');
+ entries = parseMarkdownEntries(content);
+ } catch (err) {
+ // A transient read/parse failure must never be treated as "the file
+ // is now empty" β€” that would delete-sync every entry this file ever
+ // wrote. Skip delete-sync for this file only; keep processing the rest.
+ this.emit('import:file-error', { file, reason: 'read-failed', error: err });
+ continue;
+ }
+
+ const newKeysForFile = new Set<string>();
+
+ if (entries.length === 0 && content.trim().length > 0 && oldKeysForFile && oldKeysForFile.size > 0) {
+ // Non-empty source parsed to zero entries while AgentDB holds entries
+ // for this file already β€” more likely a parser/encoding edge case
+ // than "every heading was really removed." Refuse to delete-sync.
+ this.emit('import:delete-skipped', { file, reason: 'suspicious-empty-parse' });
+ processedFiles.push(file);
+ continue;
+ }
for (const entry of entries) {
const contentHash = hashContent(entry.content);
+ const key = `auto-memory:${file}:${entry.heading}`;
+ newKeysForFile.add(key);
if (existingHashes.has(contentHash)) {
skipped++;
@@ -408,7 +438,7 @@ export class AutoMemoryBridge extends EventEmitter {
}
const input: MemoryEntryInput = {
- key: `auto-memory:${file}:${entry.heading}`,
+ key,
content: entry.content,
namespace: 'auto-memory',
type: 'semantic',
@@ -416,6 +446,7 @@ export class AutoMemoryBridge extends EventEmitter {
metadata: {
sourceFile: file,
heading: entry.heading,
+ memoryDir: this.config.memoryDir,
importedAt: new Date().toISOString(),
contentHash,
},
@@ -426,22 +457,57 @@ export class AutoMemoryBridge extends EventEmitter {
imported++;
}
+ if (oldKeysForFile) {
+ for (const [oldKey, oldId] of oldKeysForFile) {
+ if (!newKeysForFile.has(oldKey)) {
+ deleteBatch.push(oldId);
+ }
+ }
+ }
+
processedFiles.push(file);
}
+ // Delete-sync, part 2: a source file removed entirely from disk (not
+ // just emptied) is never visited by the loop above, so its entries would
+ // otherwise become permanent orphans. `existingByFile` came from AgentDB,
+ // independent of the current directory listing, so this is free β€” no
+ // extra query. Same false-wipe concern as the per-file guard: if the
+ // directory listing itself looks suspiciously empty against non-trivial
+ // prior history, skip rather than treat it as "every file was deleted."
+ if (files.length === 0 && existingByFile.size > 0) {
+ this.emit('import:delete-skipped', { file: '*', reason: 'suspicious-empty-directory' });
+ } else {
+ const currentFileSet = new Set(files);
+ for (const [oldFile, keyIndex] of existingByFile) {
+ if (!currentFileSet.has(oldFile)) {
+ for (const oldId of keyIndex.values()) {
+ deleteBatch.push(oldId);
+ }
+ }
+ }
+ }
+
// Bulk insert all at once
if (batch.length > 0) {
await this.backend.bulkInsert(batch);
}
+ // Delete-sync: remove entries whose source heading no longer exists.
+ let deleted = 0;
+ if (deleteBatch.length > 0) {
+ deleted = await this.backend.bulkDelete(deleteBatch);
+ }
+
// ADR-049: Build knowledge graph from imported entries
- if (this.memoryGraph && batch.length > 0) {
+ if (this.memoryGraph && (batch.length > 0 || deleted > 0)) {
await this.memoryGraph.buildFromBackend(this.backend, 'auto-memory');
}
const result: ImportResult = {
imported,
skipped,
+ deleted,
files: processedFiles,
durationMs: Date.now() - startTime,
};
@@ -760,8 +826,15 @@ export class AutoMemoryBridge extends EventEmitter {
await this.writeInsightToFiles(insight);
}
- /** Fetch all existing content hashes from the auto-memory namespace in one query */
- private async fetchExistingContentHashes(): Promise<Set<string>> {
+ /**
+ * Fetch all existing auto-memory entries in one query, returning both the
+ * content-hash set (dedup-on-import) and a per-source-file key->id index
+ * (delete-sync: find entries whose heading no longer exists in the file).
+ */
+ private async fetchExistingAutoMemoryIndex(): Promise<{
+ hashes: Set<string>;
+ byFile: Map<string, Map<string, string>>;
+ }> {
try {
const entries = await this.backend.query({
type: 'hybrid',
@@ -769,13 +842,32 @@ export class AutoMemoryBridge extends EventEmitter {
limit: 10_000,
});
const hashes = new Set<string>();
+ const byFile = new Map<string, Map<string, string>>();
for (const entry of entries) {
const hash = entry.metadata?.contentHash as string | undefined;
if (hash) hashes.add(hash);
+
+ // Delete-sync eligibility is scoped to entries written by THIS
+ // memoryDir. The same backend can be shared across multiple memory
+ // dirs (e.g. the SessionStart hook's main dir vs. a worktree dir
+ // from worktreeMemoryDir()) β€” without this check, importing one dir
+ // would see the other dir's same-named file/heading as "gone" and
+ // delete it. Legacy entries written before this field existed have
+ // no `memoryDir` and are therefore never delete-sync eligible.
+ const sourceFile = entry.metadata?.sourceFile as string | undefined;
+ const entryMemoryDir = entry.metadata?.memoryDir as string | undefined;
+ if (sourceFile && entry.key && entry.id && entryMemoryDir === this.config.memoryDir) {
+ let fileIndex = byFile.get(sourceFile);
+ if (!fileIndex) {
+ fileIndex = new Map<string, string>();
+ byFile.set(sourceFile, fileIndex);
+ }
+ fileIndex.set(entry.key, entry.id);
+ }
}
- return hashes;
+ return { hashes, byFile };
} catch {
- return new Set();
+ return { hashes: new Set(), byFile: new Map() };
}
}
diff --git a/v3/pnpm-lock.yaml b/v3/pnpm-lock.yaml
index f8cb98f6e9..e8be0e2132 100644
--- a/v3/pnpm-lock.yaml
+++ b/v3/pnpm-lock.yaml
@@ -212,7 +212,7 @@ importers:
specifier: 2.0.0-alpha.5
version: 2.0.0-alpha.5
'@claude-flow/memory':
- specifier: 3.0.6
+ specifier: 3.0.7
version: link:../memory
'@metaharness/darwin':
specifier: ~0.12.0