Understand any codebase instantly. System intelligence for codebases, built for humans and AI.
TypeScript aiai-memoryclaude-codeclicode-analysiscode-mappingcode-memorycodebase-understandingcodexdeveloper-toolsllmopenclawpersistent-memoryprogram-analysissoftware-architecturesystem-intelligencesystem-mapping
feat: large repo ingest perf, map improvements, and Claude plugin v2
kagebinary4d4dfa6d58db
diff --git a/CLAUDE.md b/CLAUDE.mdindex 15f1f736..61258438 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -14,7 +14,7 @@ Use JSON when chaining command results.2. AFTER every design or architecture decision → run `ix decide <title> --rationale <text>`.3. When you notice contradictory information → run `ix conflicts` and present results to the user.4. NEVER guess about codebase facts — if Ix has structured data, use it.-5. IMMEDIATELY after modifying code → run `ix map` to re-ingest and update the graph.+5. IMMEDIATELY after modifying code → run `ix map --silent` to re-ingest and update the graph.6. When the user states a goal → run `ix truth add "<statement>"`.## Ix CLI Command Routing@@ -107,7 +107,7 @@ Workflows are staged command sequences (discover → implement → validate) att```json{"discover": ["ix overview AuthService", "ix impact AuthService"],- "implement": ["ix map"],+ "implement": ["ix map --silent"],"validate": ["ix smells --format json", "ix subsystems --format json"]}```@@ -123,7 +123,7 @@ Workflows are staged command sequences (discover → implement → validate) att### Ingestion & Health| Goal | Command | Example ||---|---|---|-| Update graph + map | `ix map` | `ix map` |+| Update graph + map | `ix map --silent` | `ix map --silent` || Ingest GitHub data | `ix ingest` | `ix ingest --github owner/repo --limit 50` || Backend health | `ix status` | `ix status` || Graph statistics | `ix stats` | `ix stats --format json` |diff --git a/docs/plans/2026-03-25-large-repo-ingest-perf.md b/docs/plans/2026-03-25-large-repo-ingest-perf.mdnew file mode 100644index 00000000..ce589a67--- /dev/null+++ b/docs/plans/2026-03-25-large-repo-ingest-perf.md@@ -0,0 +1,288 @@+# Large Repo Ingest Performance++Date: 2026-03-25+Branch: `feature/claude-plugin`++---++## Baseline++Kubernetes repo (`files=16010`) before this work:++| Phase | Time |+|---|---|+| `ix map` ingest | ~466s |+| `ix map` map computation | ~53s |+| **Total `ix map`** | **~519s** |++Debug profile at baseline:+```+resolveEdgesMs=29581 buildPatchMs=10156 commitMs=386443 totalSaveMs=426834+```++The commit path dominated — 386s out of 427s total save time.++---++## What We Did (already merged)++### 1. Compound DB indexes (`memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala`)++Added compound indexes on `nodes(logical_id, deleted_rev)` and `claims(entity_id, deleted_rev)`. Tombstone + retire AQL queries filter both fields; without this, Arango returned all historical node versions then post-filtered. Now it does direct point lookups.++### 2. Upfront batch tombstone + retire (`memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`)++In `commitBatchChunked`, when multiple DB chunks exist and `baseRev > 0`, now collects all entity IDs upfront and runs tombstone + retire **once in parallel** before any inserts (`skipTombstone = true` passed to each per-chunk call). Previously ran 320 AQL UPDATE passes for Kubernetes; now runs 2.++Also parallelized tombstone + retire within `commitBatch` for the single-chunk case using `IO.both`.++### 3. Generated file filtering (`ix-cli/src/cli/commands/ingest.ts`)++Skip known generated files by filename before parsing: `*.pb.go`, `zz_generated*`, `*_deepcopy.go`, `*_mock.go`, `mock_*`, `*.pb.ts`, `*.pb.js`, `_pb.ts`, `_pb.js`. Applied in both `walkFiles` and `tryGitLsFiles`.++### 4. Parallel HTTP save requests (`ix-cli/src/cli/commands/ingest.ts`)++Replaced sequential `for` loop in `commitPreparedPatches` with a 4-worker pool (`COMMIT_CONCURRENCY = 4` at line ~416). Fixed progress bar to use `Math.max` so it doesn't jump backward when an out-of-order chunk completes.++---++## Current State (2026-03-25, pre-session baseline)++| Phase | Before | After | Delta |+|---|---|---|---|+| Ingest | ~466s | ~200s | **-57%** |+| Map computation | ~53s | ~87s | +64% (regression) |+| **Total `ix map`** | **~519s** | **~287s** | **-45%** |++Graph after full Kubernetes ingest:+```+nodes: 312,140 (113k chunk, 76k module, 43k function, 42k method, 16k class, 15k file)+edges: 840,843 (319k CALLS, 107k CONTAINS_CHUNK, 105k CONTAINS, 95k NEXT, 76k IMPORTS)+```++---++## Current State (2026-03-26, post-session)++| Phase | Before | After | Delta |+|---|---|---|---|+| Ingest | ~200s | ~227s | +13% (conflict fix overhead) |+| Map computation | ~87s | ~82s | -6% |+| **Total `ix map`** | **~287s** | **~309s** | +8% |++Ingest debug breakdown (15035 files, full fresh ingest):+```+resolveEdgesMs=26252 buildPatchMs=8731 commitMs=607952 totalSaveMs=188862+```++Ingest is currently slightly slower than the pre-session baseline. The chunk-skipping benefit (~30% fewer DB inserts) is real but offset by the COMMIT_CONCURRENCY revert to 4 (was already 4 at baseline, no gain there) and normal run variance. Ingest is clean — zero write-write conflict errors.++Map regression (82s vs 53s pre-"What We Did") is **unresolved** — this is Task 3.++---++## Targets++| Phase | Current | Target |+|---|---|---|+| Ingest | ~227s | ~90s |+| Map computation | ~82s | ~20s |+| **Total** | **~309s** | **~110s** |++---++## Task List++Tasks are independent unless noted. Each is scoped to be completable by a single Claude instance with no context from the others.++---++### Task 1 — Bump `COMMIT_CONCURRENCY` 4 → 6 ⚠️ BLOCKED++**Priority:** High — blocked by write-write conflict bug (see Task 1b below).++Bumping to 6 caused systemic write-write conflicts in ArangoDB. Root cause was identified and fixed (see Task 1b), but after that fix concurrency was left at 4. Re-attempting the bump to 6 should now be safe — the `updateRevision` conflict is eliminated, and `retryOnConflict` handles any residual tombstone conflicts.++**Acceptance:** Run `IX_DEBUG=1 ix map <kubernetes-repo>` with no `[commit error]` lines and confirm `commitMs` drops vs the 607952ms baseline.++---++### Task 1b — Fix `updateRevision` write-write conflict ✅ DONE++**Root cause:** `updateRevision` used an AQL `UPSERT { _key: "current" }` on the `revisions` collection. Every HTTP commit request ends with this call. With multiple concurrent workers, all requests raced to update the same document → ArangoDB write-write conflict error 1200.++**Fix (`memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`):**+Replaced the AQL UPSERT with two Document API inserts via the existing `bulkInsert(..., overwriteMode="replace")`. The Document API replace is atomic at the document level — concurrent replaces on the same `_key` don't conflict; last writer wins (acceptable since all concurrent requests write the same rev value).++Also added `retryOnConflict` in the client (`ix-cli/src/cli/commands/ingest.ts`) with 10ms base jitter backoff as a safety net for any residual tombstone-phase conflicts.++---++### Task 2 — Skip chunk nodes/edges during map-triggered ingest ✅ DONE++**Priority:** High — estimated ~30% fewer DB inserts.++**Files:**+- `ix-cli/src/cli/commands/ingest.ts` — add `skipChunks` flag, filter nodes/edges before patch submission+- `ix-cli/src/client/api.ts` — pass flag through to request if needed+- `ix-cli/src/client/types.ts` — add type if needed++**Background:**+Current insert volume per 200-file HTTP batch (estimated):+- ~4,154 nodes, of which ~36% are `chunk` kind nodes+- ~11,186 edges, of which ~24% are `CONTAINS_CHUNK` + `NEXT` edges++Chunk nodes and their edges are LLM context artifacts (for `ix read`). The Louvain clustering in `MapService.computeAndPersistMap` runs on `IMPORTS` and `CALLS` edges between file nodes — it never touches chunks. Skipping them during `ix map` doesn't affect map quality.++**What to do:**++1. In `ingest.ts`, find where `PreparedPatch` objects are built (the `buildPatch` / `resolveEdges` path). Add a `mapMode: boolean` parameter that, when true, strips:+ - Any node with `kind === "chunk"`+ - Any edge with `kind === "CONTAINS_CHUNK"` or `kind === "NEXT"`++2. The `ix map` code path in `ingest.ts` (around line ~544 and ~598, the `commitPreparedPatches` calls) should pass `mapMode: true`.++3. The `ix ingest` code path (full ingest, used for `ix read`) must continue writing chunks — pass `mapMode: false` or omit the flag.++4. Do not change the server side. The filtering should be purely client-side before the HTTP payload is built.++**Acceptance:**+- `IX_DEBUG=1 ix map <kubernetes-repo>` shows fewer nodes/edges inserted per batch (verify via backend `insertNodes`/`insertEdges` log lines)+- `ix read <any-file>` after a full `ix ingest` still returns chunk content (chunks must still be written on full ingest)+- No change to `ix map` output quality (regions/subsystems unchanged or better)++---++### Task 3 — Profile and fix map computation regression (82s → target 20s)++**Priority:** High — map computation is the bottleneck for incremental `ix map` runs. Every file save triggers a re-map; keeping this fast matters more than ingest speed.++**Note on terminology:** `ix ingest` is deprecated. `ix map` is the primary command. It has two phases: ingest phase (parse + commit files) and map computation phase (Louvain clustering + persist). Task 3 is about the map computation phase only.++**Files:**+- `memory-layer/src/main/scala/ix/memory/map/MapService.scala` — primary target+- `memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala` — if `build()` is the bottleneck++**Current timing:** map computation = ~82s on Kubernetes (15035 files, 828 regions). Pre-regression baseline was ~53s. Target: ≤20s (30s acceptable).++**Background:**+`computeAndPersistMap` has four steps:+1. `builder.build()` — loads file graph from DB (IMPORTS/CALLS edges between file nodes)+2. `detectCrosscut(rawGraph)` — pure CPU+3. `LouvainClustering.cluster(...)` — pure CPU, `maxLevels=3`+4. `persistMap(fresh, cached)` — writes region nodes to DB, compares old vs new regions++The regression likely lives in `builder.build()` (more nodes/edges to load since ingest changes) or `persistMap` (`sameRegions` comparison may be slow at 828 regions).++**What to do:**++1. Read `MapService.scala` first to find `computeAndPersistMap` and understand the current structure before adding instrumentation.++2. Add coarse `System.currentTimeMillis()` timing around each of the four steps and log at INFO level:+ ```scala+ val t0 = System.currentTimeMillis()+ rawGraph <- builder.build()+ val t1 = System.currentTimeMillis()+ // ... etc+ logger.info(s"map build=${t1-t0}ms crosscut=${t2-t1}ms cluster=${t3-t2}ms persist=${t4-t3}ms")+ ```++3. After adding instrumentation, rebuild and restart the backend:+ ```bash+ # from repo root (/home/ianhock/ix/IX-Memory)+ ./rebuild.sh+ ```++4. Run to collect the timing breakdown:+ ```bash+ IX_DEBUG=1 ix map /home/ianhock/kubernetes 2>&1 | tee /tmp/ix-map-debug.log+ grep "map build=" /tmp/ix-map-debug.log+ ```++5. Fix the dominant step:+ - If `builder.build()` is slow: check `MapGraphBuilder` — it queries all IMPORTS/CALLS edges between file nodes. Look for missing indexes or N+1 queries. Check `ArangoSchema.scala` for relevant indexes.+ - If `persistMap` is slow: check `sameRegions` — it sorts and compares full `Region` objects. At 828 regions this is O(n log n) with potentially high constant. Also check `submitMapPatch` / `loadOldRegionIds`.+ - If `LouvainClustering.cluster` is slow: the graph got larger. Check how many file nodes are passed to the clusterer vs pre-regression.++**Acceptance:**+- Map computation ≤ 30s on Kubernetes (20s target, 30s acceptable for this task)+- Timing log line visible in backend output during `IX_DEBUG=1 ix map`+- No regression in `ix map` output (828 regions expected, same or similar structure)+- Rebuild via `./rebuild.sh` from repo root before testing++---++### Task 4 — Vacuum old tombstoned node versions++**Priority:** Medium — reduces DB size and disk I/O over time; not urgent for this sprint.++**Files:**+- `memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala` — add vacuum method+- `memory-layer/src/main/scala/ix/memory/api/` — add a `/vacuum` or `/admin/vacuum` HTTP endpoint+- `ix-cli/src/client/api.ts` and `ix-cli/src/cli/commands/` — add `ix vacuum` CLI command++**Background:**+Every ingest revision leaves behind tombstoned docs in the `nodes` collection (`deleted_rev IS NOT NULL`). After N ingests of Kubernetes (312k live nodes), the collection has ~N × 312k docs. These bloat ArangoDB's working set, spill indexes to disk, and account for the 12.4GB disk write profile.++**What to do:**++1. In `ArangoClient.scala`, add a method:+ ```scala+ def vacuumTombstoned(keepRevisions: Int): IO[Long]+ ```+ That runs:+ ```aql+ FOR n IN nodes+ FILTER n.deleted_rev != null+ AND n.deleted_rev < @cutoffRev+ LIMIT 50000+ REMOVE n IN nodes+ RETURN 1+ ```+ Where `cutoffRev = latestRev - keepRevisions`. Use `LIMIT 50000` per pass and loop until 0 rows removed to avoid ArangoDB memory pressure on large deletes. Apply the same pattern to the `edges` and `claims` collections.++2. Expose it via a backend endpoint (e.g. `POST /admin/vacuum?keepRevisions=5`).++3. Add `ix vacuum [--keep-revisions N]` CLI command (default `--keep-revisions 5`).++**Acceptance:**+- After `ix vacuum` on Kubernetes graph, `nodes` collection count drops by at least 50% (or confirms there are minimal tombstoned docs if this is a fresh graph)+- Subsequent `ix map` run shows reduced disk I/O in `IX_DEBUG=1` output+- `ix vacuum` completes without OOM or ArangoDB timeout (batched deletes required)++---++### Task 5 — Dynamic concurrency auto-tuning (stretch goal)++**Priority:** Low — only needed if fixed concurrency of 6 still leaves headroom.++**File:** `ix-cli/src/cli/commands/ingest.ts`++**What to do:**+Replace the fixed `COMMIT_CONCURRENCY` constant with a simple AIMD (additive increase / multiplicative decrease) controller:+- Track rolling P90 latency of the last 10 batch commits+- If P90 < 8s, increment concurrency (up to max 12)+- If P90 > 15s or a batch times out, decrement concurrency (min 2)+- Log current concurrency level in the `[ingest-save]` debug line++**Acceptance:**+- On Kubernetes, concurrency auto-settles to a stable value within the first 10 batches+- Total ingest time is ≤ fixed-concurrency-6 time (must not be slower)+- No batch timeout errors introduced++---++## Profiling Instructions++To collect the data needed for Task 3:++```bash+IX_DEBUG=1 ix map <kubernetes-repo-path> 2>&1 | tee /tmp/ix-map-debug.log+```++Look for:+- Client: `[ingest-save]` lines — per-batch phase breakdown+- Client: `[slow saving]` lines — which batch numbers are slow+- Backend: `ix.bulk-write` log lines — `insertNodes`, `insertEdges`, `insertClaims`, `insertPatches` per-stage breakdown+- Backend: `pre-tombstone` line — tombstone+retire time+- Backend (after Task 3 instrumentation): `map build=Xms crosscut=Yms cluster=Zms persist=Wms`diff --git a/docs/plans/2026-03-25-large-repo-map-followup.md b/docs/plans/2026-03-25-large-repo-map-followup.mdnew file mode 100644index 00000000..dc293c87--- /dev/null+++ b/docs/plans/2026-03-25-large-repo-map-followup.md@@ -0,0 +1,154 @@+# Large Repo `ix map` Follow-up++Date: 2026-03-25++## Current user workflow++Use `ix map` only.++Do not assume `ix ingest` is part of the normal workflow anymore. Any large-repo performance work needs to be evaluated through `ix map`, since `ix map` triggers ingest first and that is what the user experiences.++## Main problem++Large repos improved for map computation, but ingest/save regressed badly.++Observed user report for Kubernetes:++- `ix map` total ingest time on current branch: about `466s`+- `ix map` total map time on current branch: about `53s`+- Earlier target / expected behavior from dev: ingest closer to `90s`+- Previous map-only improvement target: Kubernetes map around `20s`++Observed debug sample from user:++- `files=16010`+- `resolveEdgesMs=29581`+- `buildPatchMs=10156`+- `commitMs=386443`+- `totalSaveMs=426834`++Conclusion: the dominant regression is still in the save/commit path, not parse, resolve, or map clustering.++## Confirmed issues++### 1. Save path is the bottleneck on large repos++Client debug logs showed repeated slow commit chunks during saving, for example:++- `commit 3401-3600 ... (15s)`+- `commit 5601-5800 ... (17s)`+- `commit 7801-8000 ... (25s)`++This points at backend bulk commit behavior, especially chunk sizing and/or Arango bulk insert characteristics on large batches.++### 2. `ix map` output had duplicate top-level regions++The map output showed repeated roots like:++- `MODULE Edges`+- `SUBSYSTEM Edges`+- `SYSTEM Edges`++and similarly for `Builder`.++Cause found:++- the same file set could survive across multiple hierarchy levels+- those duplicate regions shared the same deterministic region ID+- they were all rendered as roots/orphans++Status:++- fixed in code by collapsing duplicate hierarchy levels to the coarsest retained region+- regression test added+- verified on a fresh backend++Remaining note:++- `Edges` and `Builder` may still appear once each as top-level systems because they are still weakly connected / isolated in the graph+- that is a separate modeling issue from the duplicate-root bug++## Changes already made++### Map hierarchy fix++Updated:++- `memory-layer/src/main/scala/ix/memory/map/MapService.scala`+- `memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala`++What changed:++- collapse duplicate same-member-set regions across module/subsystem/system levels+- keep only the coarsest retained region+- preserve children pointing at that surviving region++### Ingest/save tuning already tried++Updated:++- `ix-cli/src/cli/commands/ingest.ts`+- `memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`++What changed:++- reduced client bulk HTTP batch size from `1000` files to `200`+- reduced backend revision chunk size from `500` files to `100`+- reduced backend payload cap from `16MB` to `4MB`+- capped patch document insert batches instead of inserting arbitrarily large patch arrays++## Measured result after tuning++On a fresh isolated backend, full ingest of this repo improved from an earlier sample with:++- `commitMs ~10899`++to:++- `commitMs 4021`+- total ingest `5.48s` for `221` changed files++This confirms the chunk-size rollback helped on the repo-sized benchmark.++## What is still unresolved++The large-repo `ix map` path for Kubernetes is still much too slow.++Even after chunk-size reductions, the real user-facing test case remains:++- `ix map <kubernetes repo>`++and the unresolved part is:++- save/commit time at large scale++## What to collect next++Since the user uses `ix map`, collect logs from `ix map`, not a standalone `ix ingest` run.++Needed evidence next time:++1. Client-side `ix map` debug output including:+ - slow saving lines+ - final `[ingest-save]` line+ - final `ingest Xms · map Yms`+2. Backend `memory-layer` logs at the same time, especially `ix.bulk-write` timings+3. Identify which backend stage dominates:+ - `insertNodes`+ - `insertEdges`+ - `insertClaims`+ - `insertPatches`+ - tombstoning / retiring old claims+ - revision update++## Likely next investigation targets++1. Arango bulk insert performance for edges and patches on very large repos+2. Whether generated files (`zz_generated`, `.pb.go`, deepcopy/defaults, etc.) should be skipped or downgraded during `ix map`+3. Whether `ix map` should use a cheaper ingest mode than full graph ingest for very large repos+4. Whether chunk sizing should adapt dynamically based on actual commit latency rather than fixed file counts+5. Whether large repo map should avoid persisting some low-value entities/claims during the map-triggered ingest path++## Operational note++If a fix appears to work in code but not in normal `ix map` output, confirm the running backend has actually been restarted. During this session, the old `8090` backend continued serving stale behavior until a fresh backend process was started separately.diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.tsindex 8769f524..4dabb211 100644--- a/ix-cli/src/cli/commands/ingest.ts+++ b/ix-cli/src/cli/commands/ingest.ts@@ -2,6 +2,7 @@ import * as nodePath from 'node:path';import * as fs from 'node:fs';import * as crypto from 'node:crypto';import * as os from 'node:os';+import { spawnSync } from 'node:child_process';import type { Command } from 'commander';import chalk from 'chalk';import { IxClient } from '../../client/api.js';@@ -59,13 +60,57 @@ function parseLangs(raw: string): Set<string> {}const IGNORE_DIRS = new Set([- 'node_modules', '.git', 'dist', 'build', 'target', '.next',- '.cache', '__pycache__', '.ix', '.claude', '.gitnexus',+ 'node_modules', '.git', '.hg', '.svn', 'dist', 'build', 'target', 'out', '.next',+ '__pycache__', '.tox', '.venv', 'venv', '.mypy_cache', '.pytest_cache',+ '.gradle', '.idea', '.vscode', '.settings', '.vs',+ '.cache', '.parcel-cache', 'coverage', '.nyc_output',+ 'vendor', 'Pods', '.dart_tool', '.pub-cache',+ 'bin', 'obj', 'pkg',+ '.ix', '.claude', '.gitnexus','test', 'tests', '__tests__', 'spec', 'specs', 'e2e','examples', 'fixtures', '__mocks__', '__fixtures__',]);+// Generated file patterns — skipped by filename to avoid ingesting low-value autogenerated code.+// These files inflate parse time and graph size without contributing to architectural signal.+const GENERATED_FILE_SUFFIXES = [+ '.pb.go', // protobuf generated Go+ '_deepcopy.go', // controller-gen deepcopy (Kubernetes ecosystem)+ '_mock.go', // gomock generated mocks+ '.pb.ts', // protobuf generated TypeScript+ '.pb.js', // protobuf generated JavaScript+ '_pb.ts', // alternative protobuf TypeScript naming+ '_pb.js', // alternative protobuf JavaScript naming+];++const GENERATED_FILE_PREFIXES = [+ 'zz_generated', // controller-gen convention (Kubernetes ecosystem)+ 'mock_', // gomock convention+];++function isGeneratedFile(basename: string): boolean {+ for (const suffix of GENERATED_FILE_SUFFIXES) {+ if (basename.endsWith(suffix)) return true;+ }+ for (const prefix of GENERATED_FILE_PREFIXES) {+ if (basename.startsWith(prefix)) return true;+ }+ return false;+}+const MAX_FILE_BYTES = 1024 * 1024; // 1 MB+const MINIFIED_BYTES_THRESHOLD = 200 * 1024;+const MINIFIED_MAX_LINE_THRESHOLD = 20_000;+const MINIFIED_AVG_LINE_THRESHOLD = 2_000;+const SLOW_WORK_LOG_MS = 5_000;+const SLOW_WORK_REPEAT_MS = 10_000;++function parsePositiveIntEnv(name: string, fallback: number): number {+ const raw = process.env[name];+ if (!raw) return fallback;+ const parsed = Number.parseInt(raw, 10);+ return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;+}function* walkFiles(dir: string,@@ -76,13 +121,46 @@ function* walkFiles(catch { return; }for (const entry of entries) {+ if (entry.isDirectory() && entry.name.startsWith('.') && entry.name !== '.') continue;if (IGNORE_DIRS.has(entry.name)) continue;const full = nodePath.join(dir, entry.name);if (entry.isDirectory()) {if (recursive) yield* walkFiles(full, true);} else if (entry.isFile()) {- if (SUPPORTED_EXTENSIONS.has(nodePath.extname(entry.name))) yield full;+ if (SUPPORTED_EXTENSIONS.has(nodePath.extname(entry.name)) && !isGeneratedFile(entry.name)) yield full;+ }+ }+}++function tryGitLsFiles(dir: string, recursive: boolean): string[] | null {+ try {+ const result = spawnSync(+ 'git',+ ['ls-files', '--cached', '--others', '--exclude-standard'],+ {+ cwd: dir,+ encoding: 'utf-8',+ maxBuffer: 32 * 1024 * 1024,+ }+ );+ if (result.status !== 0) return null;++ const files: string[] = [];+ for (const line of result.stdout.split(/\r?\n/)) {+ if (!line) continue;+ const fullPath = nodePath.resolve(dir, line);+ if (!recursive && nodePath.dirname(fullPath) !== dir) continue;+ if (!SUPPORTED_EXTENSIONS.has(nodePath.extname(fullPath))) continue;+ if (isGeneratedFile(nodePath.basename(fullPath))) continue;+ try {+ if (fs.statSync(fullPath).isFile()) files.push(fullPath);+ } catch {+ // Ignore racy deletes and permission issues during discovery.+ }}+ return files;+ } catch {+ return null;}}@@ -90,6 +168,56 @@ function sha256(content: Buffer): string {return crypto.createHash('sha256').update(content).digest('hex');}+function isLikelyMinifiedSource(source: string): boolean {+ if (source.length < MINIFIED_BYTES_THRESHOLD) return false;++ let lineCount = 1;+ let currentLineLength = 0;+ let maxLineLength = 0;+ for (let i = 0; i < source.length; i++) {+ if (source.charCodeAt(i) === 10) {+ lineCount++;+ if (currentLineLength > maxLineLength) maxLineLength = currentLineLength;+ currentLineLength = 0;+ } else {+ currentLineLength++;+ }+ }+ if (currentLineLength > maxLineLength) maxLineLength = currentLineLength;++ const averageLineLength = source.length / Math.max(lineCount, 1);+ return maxLineLength >= MINIFIED_MAX_LINE_THRESHOLD || (lineCount <= 50 && averageLineLength >= MINIFIED_AVG_LINE_THRESHOLD);+}++function stripChunkOps(patch: GraphPatchPayload): GraphPatchPayload {+ return {+ ...patch,+ ops: patch.ops.filter(op => {+ if (op.type === 'UpsertNode' && op['kind'] === 'chunk') return false;+ if (op.type === 'UpsertEdge' && (op['predicate'] === 'CONTAINS_CHUNK' || op['predicate'] === 'NEXT')) return false;+ return true;+ }),+ };+}++function isWriteConflict(err: unknown): boolean {+ return String(err).includes('write-write conflict');+}++async function retryOnConflict<T>(fn: () => Promise<T>, maxRetries: number): Promise<T> {+ for (let attempt = 0; ; attempt++) {+ try {+ return await fn();+ } catch (err) {+ if (attempt >= maxRetries || !isWriteConflict(err)) throw err;+ // ArangoDB write-write conflicts resolve as soon as the competing transaction+ // commits — typically <10ms. Use short jittered delay to avoid thundering herd.+ const delay = 10 * (1 << attempt) + Math.random() * 10;+ await new Promise<void>(r => setTimeout(r, delay));+ }+ }+}+// ---------------------------------------------------------------------------// Mtime cache — skip readFileSync+sha256 for unchanged files// ---------------------------------------------------------------------------@@ -188,9 +316,10 @@ export function registerIngestCommand(program: Command): void {export async function ingestFiles(path: string,- opts: { recursive?: boolean; force?: boolean; format: string; root?: string; debug?: boolean; printSummary?: boolean; lang?: string }+ opts: { recursive?: boolean; force?: boolean; format: string; root?: string; debug?: boolean; printSummary?: boolean; lang?: string; mapMode?: boolean }): Promise<void> {const debug = opts.debug || process.env.IX_DEBUG === '1';+ const mapMode = opts.mapMode === true;const trueStart = performance.now();const [{ parseFile, resolveEdges, isGrammarSupported }, { buildPatchWithResolution }, { languageFromPath }] = await loadIngestionModules();@@ -217,6 +346,13 @@ export async function ingestFiles(const simulated = Math.floor((1 - Math.exp(-elapsed / tau)) * 0.88 * progressTotal);display = Math.max(display, simulated);}+ if (debug && currentWorkLabel) {+ const workElapsed = performance.now() - currentWorkStart;+ if (workElapsed >= SLOW_WORK_LOG_MS && (lastSlowWorkNotice === 0 || workElapsed - lastSlowWorkNotice >= SLOW_WORK_REPEAT_MS)) {+ lastSlowWorkNotice = workElapsed;+ process.stderr.write(`\n [slow ${progressPhase.toLowerCase()}] ${currentWorkLabel} (${Math.round(workElapsed / 1000)}s)\n`);+ }+ }process.stderr.write('\r' + renderProgressLine(progressPhase, display, progressTotal));}, 80) : null;@@ -226,16 +362,37 @@ export async function ingestFiles(let filesSkipped = 0;let parseErrors = 0;let tooLarge = 0;+ let minifiedLikely = 0;let latestRev = 0;let entitiesParsed = 0;+ let currentWorkLabel = '';+ let currentWorkStart = performance.now();+ let lastSlowWorkNotice = 0;++ const setCurrentWork = (label: string): void => {+ currentWorkLabel = label;+ currentWorkStart = performance.now();+ lastSlowWorkNotice = 0;+ };// Phase timing marks- const timings = { moduleLoadMs: 0, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, commitMs: 0 };+ const timings = { moduleLoadMs: 0, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, buildPatchMs: 0, commitMs: 0 };const resolveStats = {importLookups: 0, transitiveLookups: 0, globalFallbacks: 0,globalCandidateTotal: 0, resolvedImport: 0, resolvedTransitive: 0,resolvedGlobal: 0, resolvedQualifier: 0, skippedSameFile: 0, skippedAmbiguous: 0,};+ const logSaveTiming = (+ files: number,+ resolveEdgesMs: number,+ buildPatchMs: number,+ commitMs: number,+ totalSaveMs: number,+ ): void => {+ process.stderr.write(+ `\n[ingest-save] files=${files} resolveEdgesMs=${resolveEdgesMs} buildPatchMs=${buildPatchMs} commitMs=${commitMs} totalSaveMs=${totalSaveMs}\n`+ );+ };try {// Phase: discover files@@ -250,7 +407,7 @@ export async function ingestFiles(const stat = fs.statSync(resolvedPath);const filePaths: string[] = stat.isFile()? (SUPPORTED_EXTENSIONS.has(nodePath.extname(resolvedPath)) ? [resolvedPath] : [])- : Array.from(walkFiles(resolvedPath, opts.recursive ?? true));+ : (tryGitLsFiles(resolvedPath, opts.recursive ?? true) ?? Array.from(walkFiles(resolvedPath, opts.recursive ?? true)));filesDiscovered = filePaths.length;const discovered = performance.now();@@ -284,17 +441,19 @@ export async function ingestFiles(for (const filePath of filePaths) {try {const st = fs.statSync(filePath);- if (st.size === 0) { filesSkipped++; continue; }- if (st.size > MAX_FILE_BYTES) { tooLarge++; continue; }+ if (st.size === 0) { filesSkipped++; progressCurrent++; continue; }+ if (st.size > MAX_FILE_BYTES) { tooLarge++; progressCurrent++; continue; }const mtime = st.mtimeMs;currentMtimes.set(filePath, mtime);if (!opts.force && mtimeCache.get(filePath) === mtime) {- filesSkipped++; // mtime clean — assume unchanged+ filesSkipped++;+ progressCurrent++; // mtime clean — assume unchanged} else {mtimeChangedPaths.push(filePath);}} catch (err) {parseErrors++;+ progressCurrent++;process.stderr.write(`\n [stat error] ${filePath}: ${err}\n`);}}@@ -330,55 +489,208 @@ export async function ingestFiles(let resolveEdgesFn: Function | null = null;let buildPatchFn: Function | null = null;- const PARSE_STREAM_CHUNK = 1000; // resolve + commit after this many parsed files- const COMMIT_CHUNK_SIZE = 200; // files per HTTP batch to server- const YIELD_EVERY = 100; // yield event loop every N files during parse+ const PARSE_STREAM_CHUNK = 500; // streaming batch size for large repos+ const SMALL_REPO_THRESHOLD = 25_000; // parse-all-first below this; streaming above+ const COMMIT_HTTP_MAX_FILES = parsePositiveIntEnv('IX_COMMIT_HTTP_MAX_FILES', 200); // files per HTTP request to the backend+ const COMMIT_CONCURRENCY = parsePositiveIntEnv('IX_COMMIT_CONCURRENCY', 6); // parallel HTTP save requests+ const YIELD_EVERY = 100; // yield event loop every N files during parse++ if (debug) {+ process.stderr.write(+ `\n Save config: httpBatch=${COMMIT_HTTP_MAX_FILES} concurrency=${COMMIT_CONCURRENCY}\n`+ );+ }const emptyEdges: any[] = [];- /** Resolve edges within a batch, build patches, and commit in sub-chunks. */- const flushBatch = async (batch: ParsedFile[]): Promise<void> => {- if (batch.length === 0) return;- filesChanged += batch.length;+ type PreparedPatch = {+ patch: GraphPatchPayload;+ fileNumber: number;+ filePath: string;+ };- const resolvedEdges = resolveEdgesFn!(batch.map(f => f.parsed), resolveStats);- const batchEdgesByFile = new Map<string, any[]>();- for (const edge of resolvedEdges) {- let arr = batchEdgesByFile.get(edge.srcFilePath);- if (!arr) { arr = []; batchEdgesByFile.set(edge.srcFilePath, arr); }- arr.push(edge);+ const makePreparedPatch = (+ patch: GraphPatchPayload,+ fileNumber: number,+ filePath: string,+ ): PreparedPatch => ({ patch, fileNumber, filePath });++ // Commit HTTP batches in parallel (COMMIT_CONCURRENCY workers). Each worker pulls+ // the next unsent chunk from a shared queue. JS is single-threaded so the queue+ // index increment is atomic across await boundaries — no locking needed.+ const commitPreparedPatches = async (+ preparedPatches: PreparedPatch[],+ totalFiles: number,+ opts?: { updateProgress?: boolean },+ ): Promise<number> => {+ if (preparedPatches.length === 0) return 0;++ const chunks: PreparedPatch[][] = [];+ for (let i = 0; i < preparedPatches.length; i += COMMIT_HTTP_MAX_FILES) {+ chunks.push(preparedPatches.slice(i, i + COMMIT_HTTP_MAX_FILES));}- for (let i = 0; i < batch.length; i += COMMIT_CHUNK_SIZE) {- const end = Math.min(i + COMMIT_CHUNK_SIZE, batch.length);- const patchChunk: GraphPatchPayload[] = [];- for (let j = i; j < end; j++) {- const { parsed: p, hash, previousHash } = batch[j];- try {- patchChunk.push(buildPatchFn!(p, hash, batchEdgesByFile.get(p.filePath) ?? emptyEdges, previousHash));- } catch (err) {- parseErrors++;- process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);- }- }- if (patchChunk.length === 0) continue;+ const commitMsPerChunk = new Array<number>(chunks.length).fill(0);+ let nextChunk = 0;++ const runChunk = async (ci: number): Promise<void> => {+ const chunk = chunks[ci];+ const startFile = chunk[0].fileNumber;+ const endFile = chunk[chunk.length - 1].fileNumber;+ const endingPath = nodePath.basename(chunk[chunk.length - 1].filePath);+ const patches = chunk.map(item => item.patch);+try {- const result = await client.commitPatchBulk(patchChunk);+ setCurrentWork(`commit ${startFile}-${endFile} of ${totalFiles} ending ${endingPath}`);+ const commitStart = performance.now();+ const result = await retryOnConflict(() => client.commitPatchBulk(patches), 3);+ commitMsPerChunk[ci] = Math.round(performance.now() - commitStart);if (result.rev > latestRev) latestRev = result.rev;- patchesApplied += patchChunk.length;+ patchesApplied += patches.length;} catch (err) {- if (debug) process.stderr.write(`\n [bulk chunk failed, falling back to per-file] ${err}\n`);- for (const patch of patchChunk) {+ if (debug) {+ process.stderr.write(+ `\n [bulk failed, falling back to per-file] files ${startFile}-${endFile} (${patches.length} patches): ${err}\n`+ );+ }+ for (const item of chunk) {try {- const result = await client.commitPatch(patch);+ const commitStart = performance.now();+ const result = await retryOnConflict(() => client.commitPatch(item.patch), 3);+ commitMsPerChunk[ci] += Math.round(performance.now() - commitStart);if (result.rev > latestRev) latestRev = result.rev;patchesApplied++;} catch (commitErr) {parseErrors++;- process.stderr.write(`\n [commit error] ${patch.source?.uri}: ${commitErr}\n`);+ process.stderr.write(`\n [commit error] ${item.patch.source?.uri}: ${commitErr}\n`);}}}++ if (opts?.updateProgress) {+ progressCurrent = Math.max(progressCurrent, endFile);+ }+ };++ const worker = async (): Promise<void> => {+ while (true) {+ const ci = nextChunk++;+ if (ci >= chunks.length) break;+ await runChunk(ci);+ }+ };++ await Promise.all(+ Array.from({ length: Math.min(COMMIT_CONCURRENCY, chunks.length) }, () => worker())+ );++ return commitMsPerChunk.reduce((a, b) => a + b, 0);+ };++ /** Resolve edges within a batch, build patches, and commit in sub-chunks. */+ const flushBatch = async (batch: ParsedFile[]): Promise<void> => {+ if (batch.length === 0) return;+ filesChanged += batch.length;+ const previousPhase = progressPhase;+ const saveStart = performance.now();+ let resolveEdgesMs = 0;+ let buildPatchMs = 0;+ let commitMs = 0;+ progressPhase = 'Saving';+ progressStart = performance.now();+ try {+ setCurrentWork(`resolve+commit batch ending ${nodePath.basename(batch[batch.length - 1].filePath)}`);++ const resolveStart = performance.now();+ const resolvedEdges = resolveEdgesFn!(batch.map(f => f.parsed), resolveStats);+ resolveEdgesMs = Math.round(performance.now() - resolveStart);+ const batchEdgesByFile = new Map<string, any[]>();+ for (const edge of resolvedEdges) {+ let arr = batchEdgesByFile.get(edge.srcFilePath);+ if (!arr) { arr = []; batchEdgesByFile.set(edge.srcFilePath, arr); }+ arr.push(edge);+ }++ setCurrentWork(`build commit payloads ending ${nodePath.basename(batch[batch.length - 1].filePath)}`);+ const buildStart = performance.now();+ const preparedPatches: PreparedPatch[] = [];+ for (let j = 0; j < batch.length; j++) {+ const { parsed: p, hash, previousHash } = batch[j];+ try {+ const rawPatch = buildPatchFn!(p, hash, batchEdgesByFile.get(p.filePath) ?? emptyEdges, previousHash);+ const patch = mapMode ? stripChunkOps(rawPatch) : rawPatch;+ preparedPatches.push(makePreparedPatch(patch, j + 1, p.filePath));+ } catch (err) {+ parseErrors++;+ process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);+ }+ }+ buildPatchMs += Math.round(performance.now() - buildStart);+ if (preparedPatches.length === 0) return;+ commitMs += await commitPreparedPatches(preparedPatches, batch.length);+ } finally {+ const totalSaveMs = Math.round(performance.now() - saveStart);+ timings.resolveMs += resolveEdgesMs;+ timings.buildPatchMs += buildPatchMs;+ timings.commitMs += commitMs;+ logSaveTiming(batch.length, resolveEdgesMs, buildPatchMs, commitMs, totalSaveMs);+ progressPhase = previousPhase;+ progressStart = performance.now();+ }+ };++ /** Parse-all-first commit: resolve edges over full set, then commit in HTTP-sized batches. */+ const flushAll = async (allParsed: ParsedFile[]): Promise<void> => {+ if (allParsed.length === 0) return;+ filesChanged += allParsed.length;+ const previousPhase = progressPhase;+ const saveStart = performance.now();+ let resolveEdgesMs = 0;+ let buildPatchMs = 0;+ let commitMs = 0;+ progressPhase = 'Saving';+ progressTotal = allParsed.length;+ progressCurrent = 0;+ progressStart = performance.now();+ try {+ setCurrentWork(`resolve ${allParsed.length} files`);+ const resolveStart = performance.now();+ const resolvedEdges = resolveEdgesFn!(allParsed.map(f => f.parsed), resolveStats);+ resolveEdgesMs = Math.round(performance.now() - resolveStart);+ const edgesByFile = new Map<string, any[]>();+ for (const edge of resolvedEdges) {+ let arr = edgesByFile.get(edge.srcFilePath);+ if (!arr) { arr = []; edgesByFile.set(edge.srcFilePath, arr); }+ arr.push(edge);+ }+ setCurrentWork(`build commit payloads ending ${nodePath.basename(allParsed[allParsed.length - 1].filePath)}`);+ const buildStart = performance.now();+ const preparedPatches: PreparedPatch[] = [];+ for (let j = 0; j < allParsed.length; j++) {+ const { parsed: p, hash, previousHash } = allParsed[j];+ try {+ const rawPatch = buildPatchFn!(p, hash, edgesByFile.get(p.filePath) ?? emptyEdges, previousHash);+ const patch = mapMode ? stripChunkOps(rawPatch) : rawPatch;+ preparedPatches.push(makePreparedPatch(patch, j + 1, p.filePath));+ } catch (err) {+ parseErrors++;+ process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);+ }+ }+ buildPatchMs += Math.round(performance.now() - buildStart);+ if (preparedPatches.length === 0) {+ progressCurrent = allParsed.length;+ return;+ }+ commitMs += await commitPreparedPatches(preparedPatches, allParsed.length, { updateProgress: true });+ } finally {+ const totalSaveMs = Math.round(performance.now() - saveStart);+ timings.resolveMs += resolveEdgesMs;+ timings.buildPatchMs += buildPatchMs;+ timings.commitMs += commitMs;+ logSaveTiming(allParsed.length, resolveEdgesMs, buildPatchMs, commitMs, totalSaveMs);+ progressPhase = previousPhase;+ progressStart = performance.now();}};@@ -390,11 +702,12 @@ export async function ingestFiles(try {const bytes = fs.readFileSync(filePath);const hash = sha256(bytes);- if (knownHashes.get(filePath) === hash) { filesSkipped++; continue; }+ if (knownHashes.get(filePath) === hash) { filesSkipped++; progressCurrent++; continue; }const previousHash = knownHashes.get(filePath);changedPaths.push({ filePath, bytes, hash, previousHash: previousHash !== hash ? previousHash : undefined });} catch (err) {parseErrors++;+ progressCurrent++;process.stderr.write(`\n [read error] ${filePath}: ${err}\n`);}}@@ -406,14 +719,23 @@ export async function ingestFiles(resolveEdgesFn = ingestion.resolveEdges as Function;buildPatchFn = patchBuilder.buildPatchWithResolution as Function;- let batch: ParsedFile[] = [];+ const allParsed: ParsedFile[] = [];for (let i = 0; i < changedPaths.length; i++) {const { filePath, bytes, hash, previousHash } = changedPaths[i];+ setCurrentWork(`parse ${filePath}`);try {- const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));+ const sourceText = bytes.toString('utf-8');+ if (isLikelyMinifiedSource(sourceText)) {+ minifiedLikely++;+ filesSkipped++;+ progressCurrent++;+ if (debug) process.stderr.write(`\n [skip minified-likely] ${filePath}\n`);+ continue;+ }+ const parsed = (ingestion.parseFile as Function)(filePath, sourceText);if (!parsed) { filesSkipped++; progressCurrent++; continue; }entitiesParsed += parsed.entities.length;- batch.push({ filePath, parsed, hash, previousHash });+ allParsed.push({ filePath, parsed, hash, previousHash });progressCurrent++;} catch (err) {parseErrors++;@@ -421,9 +743,8 @@ export async function ingestFiles(process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);}if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));- if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }}- await flushBatch(batch);+ await flushAll(allParsed);}} else {// Path B: no baseline (first ingest) or --force → load modules, then stream parse + commit.@@ -433,9 +754,12 @@ export async function ingestFiles(resolveEdgesFn = ingestion.resolveEdges as Function;buildPatchFn = patchBuilder.buildPatchWithResolution as Function;- let batch: ParsedFile[] = [];+ const useParseAllFirst = filePaths.length <= SMALL_REPO_THRESHOLD;+ const allParsedB: ParsedFile[] = useParseAllFirst ? [] : (null as any);+ let batch: ParsedFile[] = useParseAllFirst ? (null as any) : [];for (let i = 0; i < filePaths.length; i++) {const filePath = filePaths[i];+ setCurrentWork(`parse ${filePath}`);try {const fileSize = fs.statSync(filePath).size;if (fileSize === 0) { filesSkipped++; progressCurrent++; continue; }@@ -444,10 +768,19 @@ export async function ingestFiles(const hash = sha256(bytes);if (!opts.force && knownHashes.get(filePath) === hash) { filesSkipped++; progressCurrent++; continue; }const previousHash = knownHashes.get(filePath);- const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));+ const sourceText = bytes.toString('utf-8');+ if (isLikelyMinifiedSource(sourceText)) {+ minifiedLikely++;+ filesSkipped++;+ progressCurrent++;+ if (debug) process.stderr.write(`\n [skip minified-likely] ${filePath}\n`);+ continue;+ }+ const parsed = (ingestion.parseFile as Function)(filePath, sourceText);if (!parsed) { filesSkipped++; progressCurrent++; continue; }entitiesParsed += parsed.entities.length;- batch.push({ filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined });+ const entry: ParsedFile = { filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined };+ if (useParseAllFirst) { allParsedB.push(entry); } else { batch.push(entry); }progressCurrent++;} catch (err) {parseErrors++;@@ -455,16 +788,17 @@ export async function ingestFiles(process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);}if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));- if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }+ if (!useParseAllFirst && batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }+ }+ if (useParseAllFirst) {+ await flushAll(allParsedB);+ } else {+ await flushBatch(batch);}- await flushBatch(batch);}const parsed = performance.now();timings.parseMs = Math.round(parsed - hashed);- timings.resolveMs = 0; // now interleaved with parse- timings.commitMs = 0; // now interleaved with parse-// Clear the parse barif (interval) process.stderr.write('\r' + ' '.repeat(PROG_LINE_WIDTH) + '\r');@@ -497,7 +831,7 @@ export async function ingestFiles(filesSkipped,entitiesParsed,latestRev,- skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge },+ skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge, minifiedLikely },elapsedSeconds: parseFloat(elapsed),timings,resolveStats,@@ -510,6 +844,7 @@ export async function ingestFiles(if (filesSkipped > 0) console.log(` ${chalk.dim('skipped unchanged:')} ${filesSkipped}`);if (parseErrors > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrors}`);if (tooLarge > 0) console.log(` ${chalk.dim('skipped too large:')} ${tooLarge}`);+ if (minifiedLikely > 0) console.log(` ${chalk.dim('skipped minified:')} ${minifiedLikely}`);console.log(` rev: ${latestRev}`);if (patchesApplied === 0 && filesDiscovered === 0) {@@ -525,6 +860,7 @@ export async function ingestFiles(console.log(chalk.dim(` hash: ${timings.hashMs}ms`));console.log(chalk.dim(` parse: ${timings.parseMs}ms`));console.log(chalk.dim(` resolve: ${timings.resolveMs}ms`));+ console.log(chalk.dim(` build: ${timings.buildPatchMs}ms`));console.log(chalk.dim(` commit: ${timings.commitMs}ms`));console.log(chalk.dim(`\n Resolve stats:`));console.log(chalk.dim(` import lookups: ${resolveStats.importLookups}`));diff --git a/ix-cli/src/cli/commands/map.ts b/ix-cli/src/cli/commands/map.tsindex 3388ba71..52050f00 100644--- a/ix-cli/src/cli/commands/map.ts+++ b/ix-cli/src/cli/commands/map.ts@@ -91,6 +91,7 @@ export function registerMapCommand(program: Command): void {.option("--list", "Render the ranked list view instead of the default graph/tree view").option("--full", "Force full local map, bypassing automatic safety limits (advanced/testing)").option("--verbose", "Show raw confidence scores, crosscut scores, boundary ratios, and signals")+ .option("--silent", "Suppress all output except a one-line summary (useful for LLM hooks)").addHelpText("after",`@@ -107,10 +108,14 @@ Advanced:--full Override automatic local safety limits and force the full local mappath. Bypasses automatic downgrade to fast mode and the persistencesafety guardrail. Intended for testing and performance diagnosis.+ --silent Skip the full map rendering. Prints one summary line to stderr and+ exits. Ideal for LLM hooks and automated workflows where the full+ output would waste context tokens.Examples:ix map .ix map --format json+ ix map --silentix map --level 2ix map --min-confidence 0.5ix map --max-items 10@@ -121,7 +126,7 @@ Examples:ix map . --fullix --debug map . --full`)- .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; list?: boolean; full?: boolean; verbose?: boolean }) => {+ .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; list?: boolean; full?: boolean; verbose?: boolean; silent?: boolean }) => {const cwd = pathArg ? resolve(pathArg) : process.cwd();try {@@ -132,8 +137,10 @@ Examples:return;}+ const silent = opts.silent === true || opts.format === "silent";+// Print warning when --full override is active- if (opts.full && opts.format !== "json") {+ if (opts.full && opts.format !== "json" && !silent) {console.log(chalk.yellow("\nWarning"));console.log(chalk.yellow(" Full local map override enabled.\n"));console.log(" Ix will ignore automatic local safety limits and attempt full local mapping.");@@ -142,14 +149,14 @@ Examples:// Ingest the path before mapping so the graph is up to dateconst ingestStart = performance.now();- await ingestFiles(cwd, { recursive: true, format: opts.format === "json" ? "json" : "text", printSummary: false });+ await ingestFiles(cwd, { recursive: true, format: (opts.format === "json" || silent) ? "json" : "text", printSummary: false, mapMode: true });const ingestMs = Math.round(performance.now() - ingestStart);const client = new IxClient(getEndpoint());const mapBarWidth = 25;const mapStart = performance.now();- const mapInterval = opts.format !== "json" ? setInterval(() => {+ const mapInterval = (opts.format !== "json" && !silent) ? setInterval(() => {const elapsed = performance.now() - mapStart;const pct = 1 - Math.exp(-elapsed / 4000);const filled = Math.round(pct * mapBarWidth);@@ -169,6 +176,17 @@ Examples:}if (mapInterval) { clearInterval(mapInterval); process.stderr.write('\r' + ' '.repeat(60) + '\r'); }const mapMs = Math.round(performance.now() - mapStart);++ if (silent) {+ const systems = result.regions.filter(r => r.level === 3).length;+ const subsystems = result.regions.filter(r => r.level === 2).length;+ const modules = result.regions.filter(r => r.level === 1).length;+ process.stderr.write(+ `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms\n`+ );+ return;+ }+if (opts.format !== "json") {process.stderr.write(chalk.dim(` ingest ${ingestMs}ms · map ${mapMs}ms\n`));}diff --git a/ix-cli/src/cli/commands/reset.ts b/ix-cli/src/cli/commands/reset.tsindex 696752f6..897547fb 100644--- a/ix-cli/src/cli/commands/reset.ts+++ b/ix-cli/src/cli/commands/reset.ts@@ -49,19 +49,31 @@ export function registerResetCommand(program: Command): void {}const client = new IxClient(getEndpoint());+ const label = opts.code ? "Wiping code graph..." : "Wiping graph...";+ const spinnerFrames = ["⠋","⠙","⠹","⠸","⠼","⠴","⠦","⠧","⠇","⠏"];+ let spinIdx = 0;+ const interval = setInterval(() => {+ process.stderr.write(`\r${chalk.cyan(spinnerFrames[spinIdx++ % spinnerFrames.length])} ${chalk.dim(label)}`);+ }, 80);try {if (opts.code) {await client.resetCode();+ clearInterval(interval);+ process.stderr.write("\r\x1b[K");// Clear the mtime cache so the next ix map re-ingests all filesclearMtimeCache(process.cwd());console.log(chalk.green("✓") + " Code graph wiped. Planning artifacts preserved.");console.log(chalk.dim(" Run `ix map` to rebuild the code graph."));} else {await client.reset();+ clearInterval(interval);+ process.stderr.write("\r\x1b[K");clearMtimeCache(process.cwd());console.log(chalk.green("✓") + " Graph wiped.");}} catch (err: any) {+ clearInterval(interval);+ process.stderr.write("\r\x1b[K");console.error(chalk.red("Error:"), err.message);process.exitCode = 1;return;diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.tsindex c52fd7ab..8ce6eb69 100644--- a/ix-cli/src/client/api.ts+++ b/ix-cli/src/client/api.ts@@ -189,13 +189,21 @@ export class IxClient {}async map(opts?: { full?: boolean }): Promise<any> {- return this.post("/v1/map", opts ?? {});+ const resp = await fetch(`${this.endpoint}/v1/map`, {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: JSON.stringify(opts ?? {}),+ signal: AbortSignal.timeout(30 * 60 * 1000), // 30 minute timeout+ });+ if (!resp.ok) {+ const text = await resp.text();+ throw new Error(`${resp.status}: ${text}`);+ }+ return resp.json();}- async commitPatchBulk(patches: GraphPatchPayload[]): Promise<{ rev: number }> {- const results = await this.commitPatchBatch(patches);- const rev = results.reduce((max, r) => Math.max(max, r.rev ?? 0), 0);- return { rev };+ async commitPatchBulk(patches: GraphPatchPayload[]): Promise<PatchCommitResult> {+ return this.post<PatchCommitResult>('/v1/patches/bulk', { patches });}async runSmells(opts?: {@@ -234,11 +242,31 @@ export class IxClient {}async reset(): Promise<{ ok: boolean; message: string }> {- return this.post("/v1/reset", {});+ const resp = await fetch(`${this.endpoint}/v1/reset`, {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: JSON.stringify({}),+ signal: AbortSignal.timeout(10 * 60 * 1000), // 10 minutes+ });+ if (!resp.ok) {+ const text = await resp.text();+ throw new Error(`${resp.status}: ${text}`);+ }+ return resp.json() as Promise<{ ok: boolean; message: string }>;}async resetCode(): Promise<{ ok: boolean; message: string }> {- return this.post("/v1/reset/code", {});+ const resp = await fetch(`${this.endpoint}/v1/reset/code`, {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: JSON.stringify({}),+ signal: AbortSignal.timeout(10 * 60 * 1000), // 10 minutes+ });+ if (!resp.ok) {+ const text = await resp.text();+ throw new Error(`${resp.status}: ${text}`);+ }+ return resp.json() as Promise<{ ok: boolean; message: string }>;}async stats(): Promise<any> {diff --git a/ix-cli/src/client/types.ts b/ix-cli/src/client/types.tsindex 5603dc34..a41fb0f5 100644--- a/ix-cli/src/client/types.ts+++ b/ix-cli/src/client/types.ts@@ -136,6 +136,7 @@ export interface SkipReasons {emptyFile: number;parseError: number;tooLarge: number;+ minifiedLikely?: number;}export interface IngestResult {diff --git a/ix-plugin/.claude-plugin/plugin.json b/ix-plugin/.claude-plugin/plugin.jsonindex 65f37d2d..27b0dcf3 100644--- a/ix-plugin/.claude-plugin/plugin.json+++ b/ix-plugin/.claude-plugin/plugin.json@@ -1,7 +1,7 @@{"name": "ix-memory",- "version": "1.0.0",- "description": "Ix Memory integration — auto-ingests edited files and front-runs Grep/Glob/Read with graph-aware ix searches for token-efficient codebase understanding.",+ "version": "2.0.0",+ "description": "Ix Memory integration — intercepts all searches and file reads with graph-aware ix queries, auto-ingests edits, and refreshes the full architectural map after each session.","author": {"name": "Ix Memory"},diff --git a/ix-plugin/README.md b/ix-plugin/README.mdindex c17231a6..648bdd9a 100644--- a/ix-plugin/README.md+++ b/ix-plugin/README.md@@ -1,46 +1,51 @@# Ix Memory — Claude Code Plugin-A Claude Code plugin that makes Claude use Ix for codebase searches instead of-native `Grep`/`Glob`, and keeps the Ix graph current as Claude edits files.+A Claude Code plugin that makes Claude always use Ix for codebase understanding —+injecting session context on every prompt, intercepting all searches and file reads+with graph-aware queries, and keeping the Ix graph current as Claude edits.## What It Does| Trigger | Hook | Effect ||---------|------|--------|-| Claude runs `Grep` or `Glob` | `PreToolUse` → `ix-intercept.sh` | Runs `ix text` + `ix locate` (or `ix inventory`) first and injects results into Claude's context before the native tool fires |-| Claude runs `Write`, `Edit`, or `MultiEdit` | `PostToolUse` → `ix-ingest.sh` (async) | Ingests the changed file into the Ix graph silently in the background |+| User sends any prompt | `UserPromptSubmit` → `ix-briefing.sh` | Injects compact session briefing (goals, bugs, decisions, tasks) into every prompt |+| Claude runs `Grep` or `Glob` | `PreToolUse` → `ix-intercept.sh` | Runs `ix text` + `ix locate` (or `ix inventory`) first; injects graph-aware results |+| Claude runs `Read` | `PreToolUse` → `ix-read.sh` | Runs `ix inventory` + `ix overview` for the file; injects entity context before Claude sees raw source |+| Claude runs `Bash` (grep/rg) | `PreToolUse` → `ix-bash.sh` | Detects grep/rg patterns and front-runs with `ix text` + `ix locate` |+| Claude runs `Write`, `Edit`, `MultiEdit`, or `NotebookEdit` | `PostToolUse` → `ix-ingest.sh` (async) | Ingests the changed file into the Ix graph silently in the background |+| Claude finishes responding | `Stop` → `ix-map.sh` (async) | Runs `ix map` to refresh the full architectural graph for the next session |-Both hooks bail silently if `ix` is not in PATH or the Ix backend is unreachable.+All hooks bail silently if `ix` is not in PATH or the Ix backend is unreachable.## Why Hooks Instead of MCP- **No user approval prompt** — hooks fire silently and automatically-- **Guaranteed execution** — every Grep/Glob gets front-run, every edit gets ingested+- **Guaranteed execution** — every prompt, search, read, and edit is covered- **Token-efficient** — Ix returns structured, bounded results vs. raw file content+- **Graph always current** — edits ingested immediately; full map refreshes at session end## Installation-**1. Add the Ix marketplace (one-time):**+### From the repo (local)-```-/plugin marketplace add ix-infrastructure/Ix+```bash+bash ix-plugin/install.sh```-**2. Install the plugin:**+### Via curl-```-/plugin install ix-memory+```bash+curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/install.sh | bash```-**3. Restart Claude Code.** Verify the hooks loaded with `/hooks`.+Restart Claude Code after installing. Verify hooks loaded with `/hooks`.## Requirements- `ix` CLI in PATH and Ix backend running (`ix status` should return ok)-- `jq` in PATH (used to parse hook JSON input)-- `ripgrep` (`rg`) in PATH — required by `ix text` for file-level text search+- `jq` in PATH+- `ripgrep` (`rg`) in PATH — required by `ix text`-Install missing deps:```bash# Ubuntu/Debiansudo apt install jq ripgrep@@ -57,33 +62,65 @@ ix-plugin/plugin.json # manifesthooks/hooks.json # hook event registration- ix-intercept.sh # PreToolUse: Grep|Glob → ix text/locate/inventory- ix-ingest.sh # PostToolUse: Write|Edit|MultiEdit → ix ingest+ ix-briefing.sh # UserPromptSubmit → ix briefing (session context)+ ix-intercept.sh # PreToolUse: Grep|Glob → ix text + ix locate/inventory+ ix-read.sh # PreToolUse: Read → ix inventory + ix overview+ ix-bash.sh # PreToolUse: Bash (grep/rg) → ix text + ix locate+ ix-ingest.sh # PostToolUse: Write|Edit|MultiEdit|NotebookEdit → ix ingest+ ix-map.sh # Stop → ix map (full graph refresh)+ install.sh+ uninstall.shREADME.md```-## How the Intercept Hook Works+## Hook Details++### ix-briefing.sh (UserPromptSubmit)++On every user prompt:+1. Runs `ix briefing --format json` (60s health cache to avoid per-prompt overhead)+2. Injects active goals, open bugs, recent decisions, and pending tasks as `additionalContext`+3. Claude always has project context without needing an explicit `ix briefing` call++### ix-intercept.sh (PreToolUse: Grep|Glob)When Claude runs `Grep "verify_token"`:+1. Runs `ix text "verify_token" --limit 20 --format json` + `ix locate "verify_token" --limit 10 --format json` in parallel+2. Injects results as `additionalContext` — Claude uses them before Grep output arrives-1. `ix-intercept.sh` fires with the tool input as JSON on stdin-2. Script runs `ix text "verify_token" --limit 20 --format json`-3. Script also runs `ix locate "verify_token" --limit 10 --format json`-4. Results are returned as `additionalContext` — injected into Claude's next context window-5. Claude sees the Ix results and typically uses them instead of waiting for Grep output+For `Glob`, runs `ix inventory` (optionally filtered by path) instead.-For `Glob`, the script runs `ix inventory` (optionally filtered by path) instead.+### ix-read.sh (PreToolUse: Read)-## How the Ingest Hook Works+When Claude reads a file:+1. Runs `ix inventory --path <file>` (all entities defined in the file)+2. Runs `ix overview <name>` (structural summary of the module)+3. Injects both as `additionalContext` — Claude understands the file's structure before seeing raw source-When Claude edits a file:+### ix-bash.sh (PreToolUse: Bash)-1. `ix-ingest.sh` fires asynchronously after the edit succeeds-2. Reads `tool_input.file_path` from stdin-3. Runs `ix ingest <file_path>` silently-4. Returns a short `additionalContext` confirmation: `[ix] Graph updated — ingested: <path>`+When Claude runs a `grep` or `rg` command:+1. Extracts the search pattern from the command string+2. Runs `ix text <pattern> --limit 20` + `ix locate <pattern> --limit 10` in parallel+3. Injects results as `additionalContext`++### ix-ingest.sh (PostToolUse: Write|Edit|MultiEdit|NotebookEdit)++After each file edit:+1. Reads `tool_input.file_path` from stdin+2. Runs `ix ingest <file>` silently in the background+3. Returns `additionalContext`: `[ix] Graph updated — ingested: <path>`++### ix-map.sh (Stop)++After Claude finishes each response:+1. Runs `ix map` in the background (detached, non-blocking)+2. Keeps the full architectural graph current for the next session## Uninstall-Remove the `hooks` block from `~/.claude/settings.json` (or `.claude/settings.json` for-project-scoped installs), then restart Claude Code.+```bash+bash ix-plugin/uninstall.sh+# or:+curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/uninstall.sh | bash+```diff --git a/ix-plugin/hooks/hooks.json b/ix-plugin/hooks/hooks.jsonindex ba178bf8..72b68508 100644--- a/ix-plugin/hooks/hooks.json+++ b/ix-plugin/hooks/hooks.json@@ -1,5 +1,16 @@{"hooks": {+ "UserPromptSubmit": [+ {+ "hooks": [+ {+ "type": "command",+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-briefing.sh",+ "timeout": 10+ }+ ]+ }+ ],"PreToolUse": [{"matcher": "Grep|Glob",@@ -10,11 +21,31 @@"timeout": 10}]+ },+ {+ "matcher": "Read",+ "hooks": [+ {+ "type": "command",+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-read.sh",+ "timeout": 8+ }+ ]+ },+ {+ "matcher": "Bash",+ "hooks": [+ {+ "type": "command",+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-bash.sh",+ "timeout": 10+ }+ ]}],"PostToolUse": [{- "matcher": "Write|Edit|MultiEdit",+ "matcher": "Write|Edit|MultiEdit|NotebookEdit","hooks": [{"type": "command",@@ -24,6 +55,18 @@}]}+ ],+ "Stop": [+ {+ "hooks": [+ {+ "type": "command",+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-map.sh",+ "timeout": 60,+ "async": true+ }+ ]+ }]}}diff --git a/ix-plugin/hooks/ix-bash.sh b/ix-plugin/hooks/ix-bash.shnew file mode 100755index 00000000..2044c82e--- /dev/null+++ b/ix-plugin/hooks/ix-bash.sh@@ -0,0 +1,74 @@+#!/usr/bin/env bash+# ix-bash.sh — PreToolUse hook for Bash+#+# Fires before Claude runs a Bash command. Detects grep/rg search patterns and+# front-runs them with ix text + ix locate for graph-aware, token-efficient+# results before the raw shell command executes.+#+# Exit 0 + JSON stdout → injects additionalContext, Bash still runs+# Exit 0 + no stdout → no-op, Bash runs normally++set -euo pipefail++INPUT=$(cat)+COMMAND=$(echo "$INPUT" | jq -r '.tool_input.command // empty')+[ -z "$COMMAND" ] && exit 0++# Only intercept grep/rg invocations+echo "$COMMAND" | grep -qE '^\s*(grep|rg)\s' || exit 0++# Bail silently if ix is not in PATH+command -v ix >/dev/null 2>&1 || exit 0++# ── Health check with 30s TTL cache ─────────────────────────────────────────+IX_HEALTH_CACHE="${TMPDIR:-/tmp}/ix-healthy"+_now=$(date +%s)+_cache_ok=0+if [ -f "$IX_HEALTH_CACHE" ]; then+ _cached=$(cat "$IX_HEALTH_CACHE" 2>/dev/null || echo 0)+ (( (_now - _cached) < 30 )) && _cache_ok=1+fi+if [ "$_cache_ok" -eq 0 ]; then+ ix status >/dev/null 2>&1 || exit 0+ echo "$_now" > "$IX_HEALTH_CACHE"+fi++# ── Extract search pattern ────────────────────────────────────────────────────+# Try double-quoted pattern first, then single-quoted, then unquoted+PATTERN=""+PATTERN=$(echo "$COMMAND" | sed -E 's/.*\s"([^"]+)".*/\1/' 2>/dev/null) || PATTERN=""+if [ -z "$PATTERN" ] || [ "$PATTERN" = "$COMMAND" ]; then+ PATTERN=$(echo "$COMMAND" | sed -E "s/.*\s'([^']+)'.*/\1/" 2>/dev/null) || PATTERN=""+fi+if [ -z "$PATTERN" ] || [ "$PATTERN" = "$COMMAND" ]; then+ # Unquoted: grab first non-flag token after command name+ PATTERN=$(echo "$COMMAND" | sed -E 's/^\s*(grep|rg)\s+(-[a-zA-Z0-9]+\s+|--[a-zA-Z-]+=\S+\s+)*([^-][^ ]*).*/\3/' 2>/dev/null) || PATTERN=""+fi++[ -z "$PATTERN" ] && exit 0+[ ${#PATTERN} -lt 2 ] && exit 0++# ── Run ix text + ix locate in parallel ─────────────────────────────────────+_text_tmp=$(mktemp)+_locate_tmp=$(mktemp)+trap 'rm -f "$_text_tmp" "$_locate_tmp"' EXIT++ix text "$PATTERN" --limit 20 --format json > "$_text_tmp" 2>/dev/null &+_text_pid=$!+ix locate "$PATTERN" --limit 10 --format json > "$_locate_tmp" 2>/dev/null &+_locate_pid=$!++wait "$_text_pid" || true+wait "$_locate_pid" || true++TEXT_RESULT=$(cat "$_text_tmp")+LOCATE_RESULT=$(cat "$_locate_tmp")++[ -z "$TEXT_RESULT" ] && [ -z "$LOCATE_RESULT" ] && exit 0++CONTEXT="[ix] Pre-bash search results for pattern: '${PATTERN}'"+[ -n "$TEXT_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix text ---\n${TEXT_RESULT}"+[ -n "$LOCATE_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix locate ---\n${LOCATE_RESULT}"++jq -n --arg ctx "$CONTEXT" '{"additionalContext": $ctx}'+exit 0diff --git a/ix-plugin/hooks/ix-briefing.sh b/ix-plugin/hooks/ix-briefing.shnew file mode 100755index 00000000..bfd1e220--- /dev/null+++ b/ix-plugin/hooks/ix-briefing.sh@@ -0,0 +1,40 @@+#!/usr/bin/env bash+# ix-briefing.sh — UserPromptSubmit hook+#+# Fires at the start of each user prompt. Injects a compact ix session briefing+# once per 10 minutes — simulating "on session open" behaviour. Subsequent+# prompts within the window are no-ops (no tokens consumed).+#+# Exit 0 + JSON stdout → injects additionalContext into the prompt+# Exit 0 + no stdout → no-op++set -euo pipefail++BRIEFING_TTL=600 # 10 minutes++# Bail silently if ix is not in PATH+command -v ix >/dev/null 2>&1 || exit 0++IX_BRIEFING_CACHE="${TMPDIR:-/tmp}/ix-briefing-cache"+_now=$(date +%s)++# If cache exists and is fresh, stay silent (briefing already injected this session)+if [ -f "$IX_BRIEFING_CACHE" ]; then+ _cached_time=$(head -1 "$IX_BRIEFING_CACHE" 2>/dev/null || echo 0)+ if (( (_now - _cached_time) < BRIEFING_TTL )); then+ exit 0+ fi+fi++# Cache is stale — check backend health before calling ix+ix status >/dev/null 2>&1 || exit 0++BRIEFING=$(ix briefing --format json 2>/dev/null) || exit 0+[ -z "$BRIEFING" ] && exit 0++# Write timestamp + briefing to cache+{ echo "$_now"; echo "$BRIEFING"; } > "$IX_BRIEFING_CACHE"++jq -n --arg b "$BRIEFING" \+ '{"additionalContext": ("[ix] Session briefing:\n" + $b)}'+exit 0diff --git a/ix-plugin/hooks/ix-ingest.sh b/ix-plugin/hooks/ix-ingest.shindex dd070c48..92f5ad8e 100755--- a/ix-plugin/hooks/ix-ingest.sh+++ b/ix-plugin/hooks/ix-ingest.sh@@ -1,5 +1,5 @@#!/usr/bin/env bash-# ix-ingest.sh — PostToolUse hook for Write, Edit, MultiEdit+# ix-ingest.sh — PostToolUse hook for Write, Edit, MultiEdit, NotebookEdit## Fires after Claude modifies a file. Automatically ingests the changed file# into the Ix graph so the next query reflects the current code state.diff --git a/ix-plugin/hooks/ix-map.sh b/ix-plugin/hooks/ix-map.shnew file mode 100755index 00000000..ae23f617--- /dev/null+++ b/ix-plugin/hooks/ix-map.sh@@ -0,0 +1,21 @@+#!/usr/bin/env bash+# ix-map.sh — Stop hook+#+# Fires after Claude finishes each response. Runs ix map asynchronously to+# keep the architectural graph current so the next session starts fresh.+#+# Runs async (does not block Claude's response or session end).++set -euo pipefail++# Bail silently if ix is not in PATH+command -v ix >/dev/null 2>&1 || exit 0++# Bail if backend is not reachable+ix status >/dev/null 2>&1 || exit 0++# Run ix map in background — detach from current process so it survives+nohup ix map >/dev/null 2>&1 &+disown++exit 0diff --git a/ix-plugin/hooks/ix-read.sh b/ix-plugin/hooks/ix-read.shnew file mode 100755index 00000000..1f761833--- /dev/null+++ b/ix-plugin/hooks/ix-read.sh@@ -0,0 +1,65 @@+#!/usr/bin/env bash+# ix-read.sh — PreToolUse hook for Read+#+# Fires before Claude reads a file. Runs ix inventory (all entities in the file)+# and ix overview (structural summary) in parallel and injects results as+# additionalContext so Claude has graph-aware context before reading raw source.+#+# Exit 0 + JSON stdout → injects additionalContext, native Read still runs+# Exit 0 + no stdout → no-op, Read runs normally++set -euo pipefail++INPUT=$(cat)+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')+[ -z "$FILE_PATH" ] && exit 0++# Skip binary/non-code files+case "$FILE_PATH" in+ *.png|*.jpg|*.jpeg|*.gif|*.ico|*.pdf|*.zip|*.tar|*.gz|*.bin|*.exe) exit 0 ;;+esac++# Bail silently if ix is not in PATH+command -v ix >/dev/null 2>&1 || exit 0++# ── Health check with 30s TTL cache ─────────────────────────────────────────+IX_HEALTH_CACHE="${TMPDIR:-/tmp}/ix-healthy"+_now=$(date +%s)+_cache_ok=0+if [ -f "$IX_HEALTH_CACHE" ]; then+ _cached=$(cat "$IX_HEALTH_CACHE" 2>/dev/null || echo 0)+ (( (_now - _cached) < 30 )) && _cache_ok=1+fi+if [ "$_cache_ok" -eq 0 ]; then+ ix status >/dev/null 2>&1 || exit 0+ echo "$_now" > "$IX_HEALTH_CACHE"+fi++# ── Run ix inventory + ix overview in parallel ───────────────────────────────+_inv_tmp=$(mktemp)+_ov_tmp=$(mktemp)+trap 'rm -f "$_inv_tmp" "$_ov_tmp"' EXIT++ix inventory --path "$FILE_PATH" --format json > "$_inv_tmp" 2>/dev/null &+_inv_pid=$!++# Strip extension for a cleaner symbol match+_name=$(basename "$FILE_PATH")+_name="${_name%.*}"+ix overview "$_name" --format json > "$_ov_tmp" 2>/dev/null &+_ov_pid=$!++wait "$_inv_pid" || true+wait "$_ov_pid" || true++INV_RESULT=$(cat "$_inv_tmp")+OV_RESULT=$(cat "$_ov_tmp")++[ -z "$INV_RESULT" ] && [ -z "$OV_RESULT" ] && exit 0++CONTEXT="[ix] Pre-read context for: ${FILE_PATH}"+[ -n "$INV_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix inventory (entities in file) ---\n${INV_RESULT}"+[ -n "$OV_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix overview ---\n${OV_RESULT}"++jq -n --arg ctx "$CONTEXT" '{"additionalContext": $ctx}'+exit 0diff --git a/ix-plugin/install.sh b/ix-plugin/install.shindex ce7e14d8..e597828e 100755--- a/ix-plugin/install.sh+++ b/ix-plugin/install.sh@@ -2,13 +2,13 @@# ix-plugin/install.sh — Standalone Claude Code plugin installer for Ix Memory## Usage (curl):-# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/install.sh | bash+# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/install.sh | bash## Usage (local, from the Ix repo):# bash ix-plugin/install.sh## What it does:-# 1. Downloads ix-intercept.sh and ix-ingest.sh to ~/.local/share/ix/plugin/hooks/+# 1. Copies/downloads all hook scripts to ~/.local/share/ix/plugin/hooks/# 2. Wires them into ~/.claude/settings.json## Prerequisites: ix CLI must already be installed and in PATH.@@ -17,10 +17,19 @@ set -euo pipefail# ── Config ────────────────────────────────────────────────────────────────────-GITHUB_RAW="https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/hooks"+GITHUB_RAW="https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/hooks"INSTALL_DIR="${IX_PLUGIN_DIR:-$HOME/.local/share/ix/plugin/hooks}"SETTINGS="$HOME/.claude/settings.json"+HOOKS=(+ "ix-briefing.sh"+ "ix-intercept.sh"+ "ix-read.sh"+ "ix-bash.sh"+ "ix-ingest.sh"+ "ix-map.sh"+)+# ── Helpers ───────────────────────────────────────────────────────────────────info() { echo " [ok] $*"; }@@ -43,13 +52,13 @@ info "curl, jq"if ! command -v ix >/dev/null 2>&1; thenwarn "'ix' not found in PATH — hooks will be installed but won't activate until ix is installed."- warn "Install ix first: see https://github.com/ix-infrastructure/Ix"+ warn "Install ix first: see https://github.com/ix-infrastructure/IX-Memory"fi-# ── Download hooks ────────────────────────────────────────────────────────────+# ── Download / copy hooks ─────────────────────────────────────────────────────echo ""-echo "── Downloading hooks ──"+echo "── Installing hooks ──"mkdir -p "$INSTALL_DIR"@@ -57,39 +66,37 @@ mkdir -p "$INSTALL_DIR"_repo_hooks=""if [ -n "${BASH_SOURCE[0]:-}" ] && [ "${BASH_SOURCE[0]}" != "bash" ]; then_script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)"- # Hooks live in hooks/ subdir relative to this scriptif [ -f "$_script_dir/hooks/ix-intercept.sh" ]; then_repo_hooks="$_script_dir/hooks"fifi-if [ -n "$_repo_hooks" ]; then- # Local install — copy from repo- cp "$_repo_hooks/ix-intercept.sh" "$INSTALL_DIR/ix-intercept.sh"- cp "$_repo_hooks/ix-ingest.sh" "$INSTALL_DIR/ix-ingest.sh"- info "Copied hooks from local repo → $INSTALL_DIR"-else- # Remote install — download from GitHub- curl -fsSL "$GITHUB_RAW/ix-intercept.sh" -o "$INSTALL_DIR/ix-intercept.sh"- curl -fsSL "$GITHUB_RAW/ix-ingest.sh" -o "$INSTALL_DIR/ix-ingest.sh"- info "Downloaded hooks → $INSTALL_DIR"-fi--chmod +x "$INSTALL_DIR/ix-intercept.sh" "$INSTALL_DIR/ix-ingest.sh"-info "Permissions set"+for hook in "${HOOKS[@]}"; do+ if [ -n "$_repo_hooks" ]; then+ cp "$_repo_hooks/$hook" "$INSTALL_DIR/$hook"+ else+ curl -fsSL "$GITHUB_RAW/$hook" -o "$INSTALL_DIR/$hook"+ fi+ chmod +x "$INSTALL_DIR/$hook"+ info "$hook → $INSTALL_DIR"+done# ── Wire Claude Code settings ─────────────────────────────────────────────────echo ""echo "── Configuring Claude Code ──"+BRIEFING="$INSTALL_DIR/ix-briefing.sh"INTERCEPT="$INSTALL_DIR/ix-intercept.sh"+READ_HOOK="$INSTALL_DIR/ix-read.sh"+BASH_HOOK="$INSTALL_DIR/ix-bash.sh"INGEST="$INSTALL_DIR/ix-ingest.sh"+MAP_HOOK="$INSTALL_DIR/ix-map.sh"mkdir -p "$HOME/.claude"[ -f "$SETTINGS" ] || echo "{}" > "$SETTINGS"-# Idempotent: skip if already wired+# Idempotent: skip if already wired (check for intercept hook as sentinel)already=$(jq --arg cmd "$INTERCEPT" \'[.hooks?.PreToolUse[]?.hooks[]?.command? // empty] | map(select(. == $cmd)) | length' \"$SETTINGS" 2>/dev/null || echo "0")@@ -98,20 +105,54 @@ if [ "$already" -gt 0 ]; theninfo "Hooks already registered in ~/.claude/settings.json — skipping"elsetmp=$(mktemp)- jq --arg intercept "$INTERCEPT" --arg ingest "$INGEST" '+ jq \+ --arg briefing "$BRIEFING" \+ --arg intercept "$INTERCEPT" \+ --arg read_hook "$READ_HOOK" \+ --arg bash_hook "$BASH_HOOK" \+ --arg ingest "$INGEST" \+ --arg map_hook "$MAP_HOOK" '.hooks |= (. // {}) |++ .hooks.UserPromptSubmit |= (. // []) |+ .hooks.UserPromptSubmit += [+ {+ "hooks": [{ "type": "command", "command": $briefing, "timeout": 10 }]+ }+ ] |+.hooks.PreToolUse |= (. // []) |- .hooks.PreToolUse += [{- "matcher": "Grep|Glob",- "hooks": [{ "type": "command", "command": $intercept, "timeout": 10 }]- }] |+ .hooks.PreToolUse += [+ {+ "matcher": "Grep|Glob",+ "hooks": [{ "type": "command", "command": $intercept, "timeout": 10 }]+ },+ {+ "matcher": "Read",+ "hooks": [{ "type": "command", "command": $read_hook, "timeout": 8 }]+ },+ {+ "matcher": "Bash",+ "hooks": [{ "type": "command", "command": $bash_hook, "timeout": 10 }]+ }+ ] |+.hooks.PostToolUse |= (. // []) |- .hooks.PostToolUse += [{- "matcher": "Write|Edit|MultiEdit",- "hooks": [{ "type": "command", "command": $ingest, "timeout": 30 }]- }]+ .hooks.PostToolUse += [+ {+ "matcher": "Write|Edit|MultiEdit|NotebookEdit",+ "hooks": [{ "type": "command", "command": $ingest, "timeout": 30, "async": true }]+ }+ ] |++ .hooks.Stop |= (. // []) |+ .hooks.Stop += [+ {+ "hooks": [{ "type": "command", "command": $map_hook, "timeout": 60, "async": true }]+ }+ ]' "$SETTINGS" > "$tmp" && mv "$tmp" "$SETTINGS"- info "Registered hooks → ~/.claude/settings.json"+ info "Registered all hooks → ~/.claude/settings.json"fi# ── Done ──────────────────────────────────────────────────────────────────────@@ -123,8 +164,18 @@ echo "╚═══════════════════════echo ""echo " Hooks installed to: $INSTALL_DIR"echo ""+echo " Active hooks:"+echo " UserPromptSubmit → ix briefing (session context on every prompt)"+echo " PreToolUse Grep|Glob → ix text + ix locate (graph-aware search)"+echo " PreToolUse Read → ix inventory + ix overview (file context)"+echo " PreToolUse Bash (grep/rg) → ix text + ix locate (shell search intercept)"+echo " PostToolUse Write|Edit|... → ix ingest (auto-update graph after edits)"+echo " Stop → ix map (full graph refresh after session)"+echo ""echo " Restart Claude Code to activate."echo ""echo " To uninstall:"-echo " curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/uninstall.sh | bash"+echo " bash ix-plugin/uninstall.sh"+echo " # or:"+echo " curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/uninstall.sh | bash"echo ""diff --git a/ix-plugin/uninstall.sh b/ix-plugin/uninstall.shindex 2c19d5f9..c4decb4a 100755--- a/ix-plugin/uninstall.sh+++ b/ix-plugin/uninstall.sh@@ -1,11 +1,5 @@#!/usr/bin/env bash# ix-plugin/uninstall.sh — Remove the Ix Memory Claude Code plugin-#-# Usage (curl):-# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/uninstall.sh | bash-#-# Usage (local):-# bash ix-plugin/uninstall.shset -euo pipefail@@ -19,7 +13,13 @@ echo "── Removing Ix Claude Code plugin ──"# Remove hook filesif [ -d "$INSTALL_DIR" ]; then- rm -f "$INSTALL_DIR/ix-intercept.sh" "$INSTALL_DIR/ix-ingest.sh"+ rm -f \+ "$INSTALL_DIR/ix-briefing.sh" \+ "$INSTALL_DIR/ix-intercept.sh" \+ "$INSTALL_DIR/ix-read.sh" \+ "$INSTALL_DIR/ix-bash.sh" \+ "$INSTALL_DIR/ix-ingest.sh" \+ "$INSTALL_DIR/ix-map.sh"rmdir "$INSTALL_DIR" 2>/dev/null || trueinfo "Removed $INSTALL_DIR"else@@ -28,25 +28,31 @@ fi# Remove hook entries from ~/.claude/settings.jsonif [ -f "$SETTINGS" ] && command -v jq >/dev/null 2>&1; then+ BRIEFING="$INSTALL_DIR/ix-briefing.sh"INTERCEPT="$INSTALL_DIR/ix-intercept.sh"+ READ_HOOK="$INSTALL_DIR/ix-read.sh"+ BASH_HOOK="$INSTALL_DIR/ix-bash.sh"INGEST="$INSTALL_DIR/ix-ingest.sh"+ MAP_HOOK="$INSTALL_DIR/ix-map.sh"++ _ix_cmds=$(jq -n \+ --arg a "$BRIEFING" --arg b "$INTERCEPT" \+ --arg c "$READ_HOOK" --arg d "$BASH_HOOK" \+ --arg e "$INGEST" --arg f "$MAP_HOOK" \+ '[$a,$b,$c,$d,$e,$f]')tmp=$(mktemp)- jq --arg intercept "$INTERCEPT" --arg ingest "$INGEST" '- if .hooks.PreToolUse then- .hooks.PreToolUse |= map(select(- (.hooks // []) | map(.command) | any(. == $intercept) | not- ))- else . end |- if .hooks.PostToolUse then- .hooks.PostToolUse |= map(select(- (.hooks // []) | map(.command) | any(. == $ingest) | not- ))- else . end+ jq --argjson ix_cmds "$_ix_cmds" '+ def is_ix_hook: (.hooks // []) | map(.command) | any(. as $c | $ix_cmds | any(. == $c));++ if .hooks.UserPromptSubmit then .hooks.UserPromptSubmit |= map(select(is_ix_hook | not)) else . end |+ if .hooks.PreToolUse then .hooks.PreToolUse |= map(select(is_ix_hook | not)) else . end |+ if .hooks.PostToolUse then .hooks.PostToolUse |= map(select(is_ix_hook | not)) else . end |+ if .hooks.Stop then .hooks.Stop |= map(select(is_ix_hook | not)) else . end' "$SETTINGS" > "$tmp" && mv "$tmp" "$SETTINGS"info "Removed hooks from ~/.claude/settings.json"else- echo " (settings.json not found — nothing to remove)"+ echo " (settings.json not found or jq unavailable — nothing to remove)"fiecho ""diff --git a/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scalaindex 3e1812e2..b20a1981 100644--- a/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala+++ b/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala@@ -16,8 +16,6 @@ class BulkPatchCommitRoutes(bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi)private case class BulkPatchRequest(patches: List[GraphPatch])private implicit val reqDecoder: Decoder[BulkPatchRequest] = deriveDecoder- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()-val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {case req @ POST -> Root / "v1" / "patches" / "bulk" =>(for {@@ -43,13 +41,12 @@ class BulkPatchCommitRoutes(bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi)}private def buildProvenanceMap(patch: GraphPatch): java.util.Map[String, AnyRef] = {- val json = Json.obj(- "source_uri" -> Json.fromString(patch.source.uri),- "source_hash" -> patch.source.sourceHash.fold(Json.Null)(Json.fromString),- "extractor" -> Json.fromString(patch.source.extractor),- "source_type" -> Json.fromString(patch.source.sourceType.asJson.asString.getOrElse("code")),- "observed_at" -> Json.fromString(patch.timestamp.toString)- )- mapper.readValue(json.noSpaces, classOf[java.util.Map[String, AnyRef]])+ val result = new java.util.HashMap[String, AnyRef](5)+ result.put("source_uri", patch.source.uri)+ result.put("source_hash", patch.source.sourceHash.orNull)+ result.put("extractor", patch.source.extractor)+ result.put("source_type", patch.source.sourceType.asJson.asString.getOrElse("code"))+ result.put("observed_at", patch.timestamp.toString)+ result}}diff --git a/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scalaindex d721f84e..fb0daee1 100644--- a/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala+++ b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala@@ -3,7 +3,7 @@ package ix.memory.apiimport io.circe.Jsonimport io.circe.syntax._-import ix.memory.map.{ArchitectureMap, Region, RegionMatch, ScopedSubsystemView}+import ix.memory.map.{ArchitectureMap, MapPreflightResult, PersistenceEstimate, Region, RegionMatch, ScopedSubsystemView}import ix.memory.model.NodeId/** Shared JSON encoding for ArchitectureMap, used by MapRoutes and SubsystemRoutes. */@@ -11,14 +11,26 @@ object MapEncoder {def encodeMap(m: ArchitectureMap): Json = {val regionsByLevel = m.regions.groupBy(_.level).toVector.sortBy(_._1)- Json.obj(+ val base = Json.obj("file_count" -> m.fileCount.asJson,"region_count" -> m.regions.size.asJson,"levels" -> regionsByLevel.size.asJson,"map_rev" -> m.mapRev.asJson,+ "outcome" -> m.outcome.label.asJson,"regions" -> m.regions.sortBy(r => (r.level, r.label)).map(encodeRegion).asJson,"hierarchy" -> buildHierarchyTree(m).asJson)++ val withPreflight =+ m.preflight match {+ case Some(value) => base.deepMerge(Json.obj("preflight" -> encodePreflight(value)))+ case None => base+ }++ m.persistenceEstimate match {+ case Some(value) => withPreflight.deepMerge(Json.obj("persistence" -> encodePersistence(value)))+ case None => withPreflight+ }}def encodeRegion(r: Region): Json =@@ -128,4 +140,35 @@ object MapEncoder {renderNode(view.target, Set.empty)}++ private def encodePreflight(preflight: MapPreflightResult): Json =+ Json.obj(+ "cost" -> Json.obj(+ "file_count" -> preflight.cost.fileCount.asJson,+ "directory_count" -> preflight.cost.directoryCount.asJson,+ "directory_quadratic" -> preflight.cost.directoryQuadratic.asJson,+ "symbol_estimate" -> preflight.cost.symbolEstimate.asJson,+ "edge_estimate" -> preflight.cost.edgeEstimate.asJson+ ),+ "capacity" -> Json.obj(+ "cpu_cores" -> preflight.capacity.cpuCores.asJson,+ "heap_max_bytes" -> preflight.capacity.heapMaxBytes.asJson,+ "heap_free_bytes" -> preflight.capacity.heapFreeBytes.asJson,+ "container_memory" -> preflight.capacity.containerMemory.asJson,+ "disk_free_bytes" -> preflight.capacity.diskFreeBytes.asJson+ ),+ "risk" -> preflight.risk.label.asJson,+ "mode" -> preflight.mode.label.asJson,+ "warnings" -> preflight.warnings.asJson,+ "duration_ms" -> preflight.durationMs.asJson+ )++ private def encodePersistence(persistence: PersistenceEstimate): Json =+ Json.obj(+ "region_nodes" -> persistence.regionNodes.asJson,+ "file_edges" -> persistence.fileEdges.asJson,+ "region_edges" -> persistence.regionEdges.asJson,+ "delete_ops" -> persistence.deleteOps.asJson,+ "total_ops" -> persistence.totalOps.asJson+ )}diff --git a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scalaindex 3189be0d..1acadb4e 100644--- a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala+++ b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala@@ -16,8 +16,8 @@ class MapRoutes(mapService: MapService) {case req @ POST -> Root / "v1" / "map" =>(for {body <- req.as[Json].handleError(_ => Json.obj())- force = body.hcursor.get[Boolean]("full").toOption.getOrElse(false)- archMap <- mapService.buildMap(forceRecompute = force)+ forceFull = body.hcursor.get[Boolean]("full").toOption.getOrElse(false)+ archMap <- mapService.buildMap(forceRecompute = forceFull, forceFull = forceFull)resp <- Ok(MapEncoder.encodeMap(archMap))} yield resp).handleErrorWith(ErrorHandler.handle(_))}diff --git a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scalaindex 19d7e7e9..def177bb 100644--- a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala+++ b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala@@ -13,8 +13,6 @@ import ix.memory.model.{GraphPatch, Rev}class PatchCommitRoutes(writeApi: GraphWriteApi, bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi) {- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()-val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {case req @ POST -> Root / "v1" / "patch" =>(for {@@ -52,13 +50,12 @@ class PatchCommitRoutes(writeApi: GraphWriteApi, bulkWriteApi: BulkWriteApi, que}private def buildProvenanceMap(patch: GraphPatch): java.util.Map[String, AnyRef] = {- val json = Json.obj(- "source_uri" -> Json.fromString(patch.source.uri),- "source_hash" -> patch.source.sourceHash.fold(Json.Null)(Json.fromString),- "extractor" -> Json.fromString(patch.source.extractor),- "source_type" -> Json.fromString(patch.source.sourceType.asJson.asString.getOrElse("code")),- "observed_at" -> Json.fromString(patch.timestamp.toString)- )- mapper.readValue(json.noSpaces, classOf[java.util.Map[String, AnyRef]])+ val result = new java.util.HashMap[String, AnyRef](5)+ result.put("source_uri", patch.source.uri)+ result.put("source_hash", patch.source.sourceHash.orNull)+ result.put("extractor", patch.source.extractor)+ result.put("source_type", patch.source.sourceType.asJson.asString.getOrElse("code"))+ result.put("observed_at", patch.timestamp.toString)+ result}}diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scalaindex 7ddc575e..a9f38cd4 100644--- a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala@@ -1,7 +1,8 @@package ix.memory.db+import scala.concurrent.duration._+import cats.effect.{IO, Resource}-import cats.syntax.traverse._import com.arangodb.{ArangoDB, ArangoDatabase}import com.arangodb.entity.MultiDocumentEntityimport com.arangodb.model.{AqlQueryOptions, DocumentCreateOptions, OverwriteMode, StreamTransactionOptions}@@ -37,7 +38,14 @@ class ArangoClient private (db: ArangoDatabase) {IO.blocking {val cursor = db.query(aql, classOf[AnyRef], asJava(bindVars), queryOptions(txId))cursor.asListRemaining().asScala.toList- }.flatMap(_.traverse(toJson))+ }.flatMap { values =>+ values.foldLeft(IO.pure(List.empty[Json])) { (accIO, value) =>+ for {+ acc <- accIO+ json <- toJson(value)+ } yield acc :+ json+ }+ }def queryOne(aql: String,@@ -56,6 +64,15 @@ class ArangoClient private (db: ArangoDatabase) {()}+ def queryLong(+ aql: String,+ bindVars: Map[String, AnyRef] = Map.empty,+ txId: Option[String] = None+ ): IO[Long] =+ queryOne(aql, bindVars, txId).map {+ _.flatMap(_.asNumber.flatMap(_.toLong)).getOrElse(0L)+ }+/** Bulk insert/replace documents into a collection. Uses Document API (not AQL). */def bulkInsert(collection: String,@@ -123,8 +140,14 @@ class ArangoClient private (db: ArangoDatabase) {/*** Delete only code-derived nodes and edges, preserving planning artifacts* (goals/intents, plans, tasks, decisions, bugs and their linking edges).+ *+ * Uses larger batched REMOVE passes and avoids materializing `RETURN 1`+ * result sets for every deleted document. We first count the matching docs,+ * then execute enough bounded delete passes to cover that cardinality, with+ * a final recount only if needed. Edges are removed before nodes.*/def truncateCodeGraph(): IO[Unit] = {+ val batchSize = 5000val codeKinds = Array("file", "function", "class", "method", "interface","module", "chunk", "region"@@ -133,19 +156,73 @@ class ArangoClient private (db: ArangoDatabase) {"CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS","CONTAINS", "DEFINES", "CONTAINS_CHUNK", "REFERENCES", "NEXT", "IN_REGION")++ def removeAll(+ countAql: String,+ removeAql: String,+ bindVars: Map[String, AnyRef]+ ): IO[Unit] = {+ val varsWithBatch = bindVars + ("batchSize" -> Int.box(batchSize))++ def runPasses(remaining: Long): IO[Unit] = {+ val passes = math.ceil(remaining.toDouble / batchSize.toDouble).toInt+ (0 until passes).foldLeft(IO.unit) { (acc, _) =>+ acc *> execute(removeAql, varsWithBatch)+ }+ }++ def loop: IO[Unit] =+ queryLong(countAql, bindVars).flatMap { remaining =>+ if (remaining <= 0) IO.unit+ else runPasses(remaining) *> queryLong(countAql, bindVars).flatMap { tail =>+ if (tail <= 0) IO.unit else loop+ }+ }++ loop+ }+for {- _ <- execute(- "FOR n IN nodes FILTER n.kind IN @kinds REMOVE n IN nodes",+ // Edges first — avoids locking nodes that are about to be removed+ _ <- removeAll(+ "RETURN LENGTH(FOR e IN edges FILTER e.predicate IN @predicates RETURN 1)",+ "FOR e IN edges FILTER e.predicate IN @predicates LIMIT @batchSize REMOVE e IN edges",+ Map("predicates" -> codePredicates.asInstanceOf[AnyRef])+ )+ // Claims next — code ingests append versioned/retracted claims forever, so+ // reset-code must clear them before removing the corresponding nodes.+ _ <- removeAll(+ """RETURN LENGTH(+ | FOR c IN claims+ | FILTER LENGTH(+ | FOR n IN nodes+ | FILTER n.logical_id == c.entity_id+ | AND n.kind IN @kinds+ | LIMIT 1+ | RETURN 1+ | ) > 0+ | RETURN 1+ |)""".stripMargin,+ """FOR c IN claims+ | FILTER LENGTH(+ | FOR n IN nodes+ | FILTER n.logical_id == c.entity_id+ | AND n.kind IN @kinds+ | LIMIT 1+ | RETURN 1+ | ) > 0+ | LIMIT @batchSize+ | REMOVE c IN claims""".stripMargin,Map("kinds" -> codeKinds.asInstanceOf[AnyRef]))- _ <- execute(- "FOR e IN edges FILTER e.predicate IN @predicates REMOVE e IN edges",- Map("predicates" -> codePredicates.asInstanceOf[AnyRef])+ _ <- removeAll(+ "RETURN LENGTH(FOR n IN nodes FILTER n.kind IN @kinds RETURN 1)",+ "FOR n IN nodes FILTER n.kind IN @kinds LIMIT @batchSize REMOVE n IN nodes",+ Map("kinds" -> codeKinds.asInstanceOf[AnyRef]))- // Also remove code-ingest patches so getSourceHashes returns empty after reset,- // preventing the ingest client from treating all files as unchanged.- _ <- execute(- "FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') REMOVE p IN patches",+ _ <- removeAll(+ "RETURN LENGTH(FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') RETURN 1)",+ "FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') LIMIT @batchSize REMOVE p IN patches",Map.empty)} yield ()@@ -166,6 +243,7 @@ object ArangoClient {.host(host, port).user(user).password(if (password.isEmpty) null else password)+ .maxConnections(8).build()val existingDbs = arango.getDatabases.asScala.toSetif (!existingDbs.contains(database)) {diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scalaindex bec90b28..d4319f2c 100644--- a/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala@@ -33,6 +33,13 @@ object ArangoSchema {java.util.Arrays.asList("logical_id"),new PersistentIndexOptions())+ // Compound index for tombstone query: FILTER logical_id IN @ids AND deleted_rev == null+ // Without this, the single-field logical_id index returns all historical versions of each+ // node and applies deleted_rev as a post-filter — gets slower as the collection grows.+ nodes.ensurePersistentIndex(+ java.util.Arrays.asList("logical_id", "deleted_rev"),+ new PersistentIndexOptions()+ )// Indexes — edgesval edges = db.collection("edges")@@ -61,6 +68,12 @@ object ArangoSchema {java.util.Arrays.asList("entity_id"),new PersistentIndexOptions())+ // Compound index for retire query: FILTER entity_id IN @ids AND deleted_rev == null+ // Mirrors the nodes compound index — avoids post-filtering all historical claim versions.+ claims.ensurePersistentIndex(+ java.util.Arrays.asList("entity_id", "deleted_rev"),+ new PersistentIndexOptions()+ )claims.ensurePersistentIndex(java.util.Arrays.asList("status"),new PersistentIndexOptions()diff --git a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scalaindex d0c51ae0..3c92a0c9 100644--- a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala+++ b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala@@ -8,6 +8,8 @@ import cats.effect.IOimport cats.syntax.traverse._import io.circe.Jsonimport io.circe.syntax._+import org.typelevel.log4cats.Logger+import org.typelevel.log4cats.slf4j.Slf4jLoggerimport scala.concurrent.duration._@@ -18,8 +20,18 @@ import ix.memory.model._* instead of individual AQL queries.*/class BulkWriteApi(client: ArangoClient) {+ private type BulkDoc = java.util.Map[String, AnyRef]private val CurrentRevisionKey = "current"private val SourceGraphRevisionKey = "source-graph"+ private val DefaultChunkSize = 100+ private val DefaultChunkPayloadBytes = 4 * 1024 * 1024+ private val MaxNodeDocs = 10000+ private val MaxEdgeDocs = 20000+ private val MaxClaimDocs = 10000+ private val MaxPatchDocs = 250+ private val SlowChunkMs = 2000L+ private val debugBulkWrites = sys.env.get("IX_DEBUG").exists(v => v.nonEmpty && v != "0" && !v.equalsIgnoreCase("false"))+ private val logger: Logger[IO] = Slf4jLogger.getLoggerFromName[IO]("ix.bulk-write")private def withRetry[A](action: IO[A], maxAttempts: Int = 3, delay: FiniteDuration = 500.millis): IO[A] = {action.handleErrorWith { err =>@@ -28,30 +40,82 @@ class BulkWriteApi(client: ArangoClient) {}}+ private def chunkDocs[T](docs: Seq[T], maxDocs: Int): Seq[Seq[T]] =+ docs.grouped(maxDocs).toVector++ private def insertCollectionChunked(+ collection: String,+ docs: Seq[BulkDoc],+ maxDocs: Int+ )(insert: Seq[BulkDoc] => IO[Int]): IO[Long] =+ if (docs.isEmpty) IO.pure(0L)+ else {+ val batches = chunkDocs(docs, maxDocs)+ batches.foldLeft(IO.pure(0L)) { case (accIO, batch) =>+ for {+ acc <- accIO+ ms <- timedUnit(withRetry(insert(batch)))+ _ <- if (debugBulkWrites)+ logger.info(s"[bulk-write] collection=$collection docs=${batch.size} insertMs=$ms")+ else IO.unit+ } yield acc + ms+ }+ }+def commitBatch(fileBatches: Vector[FileBatch],- baseRev: Long+ baseRev: Long,+ skipTombstone: Boolean = false): IO[CommitResult] = {val newRev = baseRev + 1L- val allNodes = fileBatches.flatMap(_.nodeDocuments(newRev))- val allEdges = fileBatches.flatMap(_.edgeDocuments(newRev))- val allClaims = fileBatches.flatMap(_.claimDocuments(newRev))- val allPatches = fileBatches.map(_.patchDocument(newRev))-- // Collect all logical_ids being upserted for tombstoning- val logicalIds = allNodes.flatMap { doc =>- Option(doc.get("logical_id")).map(_.toString)- }-for {- _ <- tombstoneExistingNodes(logicalIds, newRev)- _ <- retireOldClaims(fileBatches, newRev)- _ <- withRetry(client.bulkInsert("nodes", allNodes))- _ <- withRetry(client.bulkInsertEdges("edges", allEdges))- _ <- withRetry(client.bulkInsert("claims", allClaims))- _ <- withRetry(client.bulkInsert("patches", allPatches))- _ <- updateRevision(newRev)+ // Build all document maps in a single IO.blocking pass — one traversal instead of four.+ docsAndBuildMs <- timed {+ IO.blocking {+ val nodesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]+ val edgesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]+ val claimsBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]+ val patchesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]+ fileBatches.foreach { fb =>+ nodesBuf ++= fb.nodeDocuments(newRev)+ edgesBuf ++= fb.edgeDocuments(newRev)+ claimsBuf ++= fb.claimDocuments(newRev)+ patchesBuf += fb.patchDocument(newRev)+ }+ (nodesBuf.result(), edgesBuf.result(), claimsBuf.result(), patchesBuf.result())+ }+ }+ (docs, buildDocsMs) = docsAndBuildMs+ (allNodes, allEdges, allClaims, allPatches) = docs+ logicalIds = allNodes.flatMap(doc => Option(doc.get("logical_id")).map(_.toString))+ chunkSummary = s"files=${fileBatches.size} nodes=${allNodes.size} edges=${allEdges.size} claims=${allClaims.size} patches=${allPatches.size}"+ // Skip tombstone/retire on first ingest — DB is empty, no existing docs to update.+ tombstoneAndRetireMs <-+ if (skipTombstone || baseRev == 0) IO.pure((0L, 0L))+ else timedUnit(tombstoneExistingNodes(logicalIds, newRev))+ .both(timedUnit(retireOldClaims(fileBatches, newRev)))+ (tombstoneMs: Long, retireClaimsMs: Long) = tombstoneAndRetireMs+ insertNodesMs <- insertCollectionChunked("nodes", allNodes, MaxNodeDocs)(docs => client.bulkInsert("nodes", docs))+ insertEdgesMs <- insertCollectionChunked("edges", allEdges, MaxEdgeDocs)(docs => client.bulkInsertEdges("edges", docs))+ insertClaimsMs <- insertCollectionChunked("claims", allClaims, MaxClaimDocs)(docs => client.bulkInsert("claims", docs))+ insertPatchesMs <- insertCollectionChunked("patches", allPatches, MaxPatchDocs)(docs => client.bulkInsert("patches", docs))+ updateRevMs <- timedUnit(updateRevision(newRev))+ totalMs = buildDocsMs + tombstoneMs + retireClaimsMs + insertNodesMs + insertEdgesMs + insertClaimsMs + insertPatchesMs + updateRevMs+ _ <- logChunkTiming(+ s"rev=$newRev $chunkSummary",+ totalMs,+ List(+ "buildDocs" -> buildDocsMs,+ "tombstoneNodes" -> tombstoneMs,+ "retireClaims" -> retireClaimsMs,+ "insertNodes" -> insertNodesMs,+ "insertEdges" -> insertEdgesMs,+ "insertClaims" -> insertClaimsMs,+ "insertPatches" -> insertPatchesMs,+ "updateRevision" -> updateRevMs,+ )+ )} yield CommitResult(Rev(newRev), CommitStatus.Ok)}@@ -74,10 +138,7 @@ class BulkWriteApi(client: ArangoClient) {}}- private def retireOldClaims(fileBatches: Vector[FileBatch], newRev: Long): IO[Unit] = {- val entityIds = fileBatches.flatMap(_.patch.ops.collect {- case PatchOp.UpsertNode(id, _, _, _) => id.value.toString- }).distinct+ private def retireOldClaimsByEntityIds(entityIds: Seq[String], newRev: Long): IO[Unit] =if (entityIds.isEmpty) IO.unitelse {val idList = new java.util.ArrayList[String](entityIds.size)@@ -93,41 +154,124 @@ class BulkWriteApi(client: ArangoClient) {))}- }++ private def retireOldClaims(fileBatches: Vector[FileBatch], newRev: Long): IO[Unit] =+ retireOldClaimsByEntityIds(+ fileBatches.flatMap(_.patch.ops.collect {+ case PatchOp.UpsertNode(id, _, _, _) => id.value.toString+ }).distinct,+ newRev+ )/*** Commit file batches in chunks. Each chunk gets its own revision.* If a chunk fails, prior chunks are preserved (partial progress).+ *+ * When multiple chunks exist and baseRev > 0 (incremental update), tombstone+retire+ * runs once upfront for all files in parallel rather than once per chunk sequentially.+ * This reduces 2*N AQL UPDATE scans to 2 — e.g. 160 → 2 for Kubernetes.*/def commitBatchChunked(fileBatches: Vector[FileBatch],baseRev: Long,- chunkSize: Int = 100+ chunkSize: Int = DefaultChunkSize,+ maxPayloadBytes: Int = DefaultChunkPayloadBytes): IO[CommitResult] = {if (fileBatches.isEmpty)return IO.pure(CommitResult(Rev(baseRev), CommitStatus.Ok))- val chunks = fileBatches.grouped(chunkSize).toVector+ val chunks = chunkForCommit(fileBatches, chunkSize, maxPayloadBytes)+ val multiChunk = chunks.size > 1 && baseRev > 0- chunks.foldLeft(IO.pure(baseRev)) { case (revIO, chunk) =>- revIO.flatMap { currentRev =>- commitBatch(chunk, currentRev).map(_.newRev.value)+ // Pre-tombstone: collect all entity IDs upfront and retire/tombstone in one parallel pass.+ // Uses baseRev+1 as the tombstone rev, which is the same rev chunk 1 will use for its inserts.+ val preTombstoneIO: IO[Unit] =+ if (!multiChunk) IO.unit+ else {+ val allIds = fileBatches.flatMap(_.patch.ops.collect {+ case PatchOp.UpsertNode(id, _, _, _) => id.value.toString+ }).distinct+ val preTombRev = baseRev + 1L+ timedUnit(tombstoneExistingNodes(allIds.toVector, preTombRev))+ .both(timedUnit(retireOldClaimsByEntityIds(allIds, preTombRev)))+ .flatMap { case (tombMs, retireMs) =>+ if (debugBulkWrites)+ logger.info(s"[bulk-write] pre-tombstone files=${fileBatches.size} tombstoneMs=$tombMs retireMs=$retireMs")+ else IO.unit+ }}- }.map(finalRev => CommitResult(Rev(finalRev), CommitStatus.Ok))++ for {+ _ <- preTombstoneIO+ finalRev <- chunks.foldLeft(IO.pure(baseRev)) { case (revIO, chunk) =>+ revIO.flatMap { currentRev =>+ commitBatch(chunk, currentRev, skipTombstone = multiChunk).map(_.newRev.value)+ }+ }+ } yield CommitResult(Rev(finalRev), CommitStatus.Ok)}- private def updateRevision(newRev: Long): IO[Unit] =- client.execute(- """FOR key IN @keys- | UPSERT { _key: key }- | INSERT { _key: key, rev: @rev }- | UPDATE { rev: @rev }- | IN revisions""".stripMargin,- Map(- "keys" -> Array(CurrentRevisionKey, SourceGraphRevisionKey).asInstanceOf[AnyRef],- "rev" -> Long.box(newRev).asInstanceOf[AnyRef]- )- )+ private def updateRevision(newRev: Long): IO[Unit] = {+ // Use Document API (replace mode) instead of AQL UPSERT.+ // AQL UPSERT produces write-write conflicts when multiple concurrent HTTP+ // requests all try to update the same "current" revision key simultaneously.+ // The Document API replace is atomic at the document level — no conflict possible.+ val doc1 = new java.util.HashMap[String, AnyRef]()+ doc1.put("_key", CurrentRevisionKey)+ doc1.put("rev", Long.box(newRev))+ val doc2 = new java.util.HashMap[String, AnyRef]()+ doc2.put("_key", SourceGraphRevisionKey)+ doc2.put("rev", Long.box(newRev))+ client.bulkInsert("revisions", Seq(doc1, doc2)).void+ }++ private def chunkForCommit(+ fileBatches: Vector[FileBatch],+ maxFiles: Int,+ maxPayloadBytes: Int+ ): Vector[Vector[FileBatch]] = {+ val chunks = Vector.newBuilder[Vector[FileBatch]]+ var current = Vector.empty[FileBatch]+ var currentBytes = 0++ fileBatches.foreach { batch =>+ val batchBytes = batch.estimatedPayloadBytes+ val exceedsPayload = current.nonEmpty && currentBytes + batchBytes > maxPayloadBytes+ val exceedsCount = current.size >= maxFiles+ if (exceedsPayload || exceedsCount) {+ chunks += current+ current = Vector.empty+ currentBytes = 0+ }+ current = current :+ batch+ currentBytes += batchBytes+ }++ if (current.nonEmpty) {+ chunks += current+ }++ chunks.result()+ }++ private def timed[A](ioa: IO[A]): IO[(A, Long)] =+ for {+ start <- IO.monotonic+ result <- ioa+ end <- IO.monotonic+ } yield (result, (end - start).toMillis)++ private def timedUnit[A](ioa: IO[A]): IO[Long] =+ timed(ioa).map(_._2)++ private def logChunkTiming(summary: String, totalMs: Long, stages: List[(String, Long)]): IO[Unit] = {+ val shouldLog = debugBulkWrites || totalMs >= SlowChunkMs+ if (!shouldLog) IO.unit+ else {+ val stageSummary = stages.map { case (name, ms) => s"$name=${ms}ms" }.mkString(" ")+ logger.info(s"$summary total=${totalMs}ms $stageSummary")+ }+ }}/**@@ -140,32 +284,47 @@ case class FileBatch(patch: GraphPatch,provenance: java.util.Map[String, AnyRef]) {- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()-- private def parseToJavaMap(jsonStr: String): java.util.Map[String, AnyRef] =- mapper.readValue(jsonStr, classOf[java.util.Map[String, AnyRef]])+ // Direct Circe-to-Java conversion — no string serialization round-trip.+ private def jsonToJava(json: Json): AnyRef = json.fold(+ jsonNull = null.asInstanceOf[AnyRef],+ jsonBoolean = b => java.lang.Boolean.valueOf(b),+ jsonNumber = n => java.lang.Double.valueOf(n.toDouble),+ jsonString = s => s,+ jsonArray = arr => {+ val list = new java.util.ArrayList[AnyRef](arr.size)+ arr.foreach(j => list.add(jsonToJava(j)))+ list+ },+ jsonObject = obj => {+ val map = new java.util.LinkedHashMap[String, AnyRef](obj.size)+ obj.toMap.foreach { case (k, v) => map.put(k, jsonToJava(v)) }+ map+ }+ )- private def jsonToJava(json: Json): AnyRef =- mapper.readValue(json.noSpaces, classOf[AnyRef])+ private def attrsToJavaMap(attrs: Map[String, Json]): java.util.Map[String, AnyRef] = {+ val result = new java.util.HashMap[String, AnyRef](attrs.size)+ attrs.foreach { case (k, v) => result.put(k, jsonToJava(v)) }+ result+ }private def nodeKindToString(nk: NodeKind): String =nk.asJson.asString.getOrElse(nk.toString)lazy val estimatedPayloadBytes: Int =- (patch.asJson.noSpaces.getBytes(StandardCharsets.UTF_8).length * 4) + 1024+ patch.asJson.noSpaces.getBytes(StandardCharsets.UTF_8).length + 1024def nodeDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] = {val now = Instant.now().toStringpatch.ops.collect { case PatchOp.UpsertNode(id, kind, name, attrs) =>val logicalId = id.value.toString- val attrsJson = Json.obj(attrs.toSeq.map { case (k, v) => k -> v }: _*).noSpacesval doc = new java.util.HashMap[String, AnyRef]()doc.put("_key", s"${logicalId}_${rev}")doc.put("logical_id", logicalId)doc.put("id", logicalId)doc.put("kind", nodeKindToString(kind))doc.put("name", name)- doc.put("attrs", parseToJavaMap(attrsJson))+ doc.put("attrs", attrsToJavaMap(attrs))doc.put("provenance", provenance)doc.put("created_rev", Long.box(rev))doc.put("deleted_rev", null)@@ -178,7 +337,6 @@ case class FileBatch(def edgeDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] = {val now = Instant.now().toStringpatch.ops.collect { case PatchOp.UpsertEdge(id, src, dst, predicate, attrs) =>- val attrsJson = Json.obj(attrs.toSeq.map { case (k, v) => k -> v }: _*).noSpacesval doc = new java.util.HashMap[String, AnyRef]()doc.put("_key", id.value.toString)doc.put("_from", s"nodes/${src.value}")@@ -187,7 +345,7 @@ case class FileBatch(doc.put("src", src.value.toString)doc.put("dst", dst.value.toString)doc.put("predicate", predicate.value)- doc.put("attrs", parseToJavaMap(attrsJson))+ doc.put("attrs", attrsToJavaMap(attrs))doc.put("created_rev", Long.box(rev))doc.put("deleted_rev", null)doc.put("provenance", provenance)@@ -214,12 +372,11 @@ case class FileBatch(}def patchDocument(rev: Long): java.util.Map[String, AnyRef] = {- val patchJson = patch.asJson.noSpacesval doc = new java.util.HashMap[String, AnyRef]()doc.put("_key", patch.patchId.value.toString)doc.put("patch_id", patch.patchId.value.toString)doc.put("rev", Long.box(rev))- doc.put("data", mapper.readValue(patchJson, classOf[java.util.Map[String, AnyRef]]))+ doc.put("data", jsonToJava(patch.asJson))doc: java.util.Map[String, AnyRef]}}diff --git a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scalaindex 1784f991..9cde2da4 100644--- a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala+++ b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala@@ -1,6 +1,7 @@package ix.memory.mapimport cats.effect.IO+import org.typelevel.log4cats.slf4j.Slf4jLoggerimport ix.memory.db.ArangoClientimport ix.memory.model.{NodeId, Rev}@@ -21,6 +22,7 @@ import ix.memory.model.{NodeId, Rev}* ε = 0.2 path proximity (weak structural prior)*/class MapGraphBuilder(client: ArangoClient) {+ private val logger = Slf4jLogger.getLoggerFromName[IO]("ix.map.build")private val Alpha = 1.0private val Beta = 0.5@@ -32,7 +34,8 @@ class MapGraphBuilder(client: ArangoClient) {private val SourceGraphRevisionKey = "source-graph"private val IgnoredNodeKinds = Vector("module", "config", "config_entry", "doc", "decision", "intent",- "bug", "plan", "task", "goal", "region"+ "bug", "plan", "task", "goal", "region",+ "chunk" // chunks are code-text segments, never CALLS/IMPORTS/EXTENDS/IMPLEMENTS endpoints)private val CouplingPredicates = Vector("CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS")@@ -40,8 +43,14 @@ class MapGraphBuilder(client: ArangoClient) {def buildGraph(files: Vector[FileVertex]): IO[WeightedFileGraph] =for {+ t0 <- IO(System.currentTimeMillis())rawPairs <- fetchCouplingByUri()+ t1 <- IO(System.currentTimeMillis())graph = computeGraph(files, rawPairs)+ t2 <- IO(System.currentTimeMillis())+ _ <- logger.info(+ s"map graph files=${files.size} raw_pairs=${rawPairs.size} fetch=${t1-t0}ms compute=${t2-t1}ms total=${t2-t0}ms"+ )} yield graphdef build(): IO[WeightedFileGraph] =@@ -226,9 +235,7 @@ class MapGraphBuilder(client: ArangoClient) {scala.collection.mutable.Map[(NodeId, NodeId), Map[String, Int]]()var edgeCount = 0- for (((srcId, dstId), predicateCounts) <- pairAcc.toVector.sortBy { case ((a, b), _) =>- (a.value.toString, b.value.toString)- }) {+ for (((srcId, dstId), predicateCounts) <- pairAcc.iterator) {val sv = byId(srcId)val dv = byId(dstId)val srcSegments = pathSegmentsById(srcId)diff --git a/memory-layer/src/main/scala/ix/memory/map/MapService.scala b/memory-layer/src/main/scala/ix/memory/map/MapService.scalaindex f1c5c96a..a4b78081 100644--- a/memory-layer/src/main/scala/ix/memory/map/MapService.scala+++ b/memory-layer/src/main/scala/ix/memory/map/MapService.scala@@ -4,8 +4,10 @@ import java.time.Instantimport java.util.UUIDimport cats.effect.IO+import com.fasterxml.jackson.databind.ObjectMapperimport io.circe.Jsonimport io.circe.syntax._+import org.typelevel.log4cats.slf4j.Slf4jLoggerimport ix.memory.db.{ArangoClient, GraphQueryApi, GraphWriteApi}import ix.memory.model._@@ -26,6 +28,8 @@ class MapService(queryApi: GraphQueryApi,writeApi: GraphWriteApi) {+ private val mapper = new ObjectMapper()+ private val logger = Slf4jLogger.getLoggerFromName[IO]("ix.map")private val CrosscutThreshold = 0.10// Cross-cut penalty multiplier (reduces edge weights involving crosscut files)@@ -46,7 +50,8 @@ class MapService("PATH" -> 0.2)- private val builder = new MapGraphBuilder(client)+ private val fullBuilder = new MapGraphBuilder(client)+ private val preflight = new MapPreflight()private final class RetainedRegion(val level: Int, val communityIndex: Int, val region: Region)// Label kinds by level@@ -56,20 +61,20 @@ class MapService(else if (level == 1) "module"else "subsystem"- def buildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =- getOrBuildMap(forceRecompute)+ def buildMap(forceRecompute: Boolean = false, forceFull: Boolean = false): IO[ArchitectureMap] =+ getOrBuildMap(forceRecompute, forceFull)- def getOrBuildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =+ def getOrBuildMap(forceRecompute: Boolean = false, forceFull: Boolean = false): IO[ArchitectureMap] =for {- inputRev <- builder.currentInputRev()+ inputRev <- fullBuilder.currentInputRev()cached <- loadCachedMap(inputRev)archMap <- if (cached.nonEmpty && !forceRecompute) IO.pure(cached.get)- else computeAndPersistMap(inputRev, cached)+ else computeAndPersistMap(inputRev, cached, forceFull)} yield archMapdef loadPersistedMapForCurrentGraph(): IO[Option[ArchitectureMap]] =for {- inputRev <- builder.currentInputRev()+ inputRev <- fullBuilder.currentInputRev()cached <- loadCachedMap(inputRev)} yield cached@@ -263,14 +268,15 @@ class MapService(val withChildren = withParents.map { r =>r.copy(childRegionIds = childrenByParent.getOrElse(r.id, Set.empty))}+ val dedupedHierarchy = collapseDuplicateHierarchyLevels(withChildren)// Disambiguate duplicate labels within each level.// For each group of same-label regions, find a path segment unique to each.- val labelGroups = withChildren.groupBy(r => (r.level, r.label))+ val labelGroups = dedupedHierarchy.groupBy(r => (r.level, r.label))val needsDisambig = labelGroups.filter(_._2.size > 1)- if (needsDisambig.isEmpty) withChildren- else withChildren.map { r =>+ if (needsDisambig.isEmpty) dedupedHierarchy+ else dedupedHierarchy.map { r =>needsDisambig.get((r.level, r.label)) match {case Some(group) =>val myPaths = r.memberFiles.flatMap(id => fileVertexById.get(id).map(_.path)).toSet@@ -299,6 +305,14 @@ class MapService(}}+ private[map] def collapseDuplicateHierarchyLevels(regions: Vector[Region]): Vector[Region] = {+ val keepById = regions.groupBy(_.id).view.mapValues { duplicates =>+ duplicates.maxBy(region => (region.level, region.confidence, region.label.length))+ }.toMap++ regions.filter(region => keepById.get(region.id).contains(region))+ }+private val LabelNoiseTokens = Set("main", "scala", "java", "src", "test", "tests", "__tests__", "kotlin","python", "js", "ts", "tsx", "jsx", "com", "org", "io", "net",@@ -600,15 +614,36 @@ class MapService(// ── Persistence ─────────────────────────────────────────────────────- private def computeAndPersistMap(inputRev: Rev, cached: Option[ArchitectureMap]): IO[ArchitectureMap] =+ private def computeAndPersistMap(+ inputRev: Rev,+ cached: Option[ArchitectureMap],+ forceFull: Boolean+ ): IO[ArchitectureMap] =for {- rawGraph <- builder.build()- crosscutScores = detectCrosscut(rawGraph)- graph = applyPenalty(rawGraph, crosscutScores)- levels = LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3)- regions = buildRegions(graph, levels, crosscutScores, inputRev)- fresh = ArchitectureMap(regions, rawGraph.vertices.size, inputRev.value)- _ <- persistMap(fresh, cached)+ files <- fullBuilder.discoverFiles()+ preflightEval <- preflight.evaluate(files)+ t0 <- IO(System.currentTimeMillis())+ rawGraph <- fullBuilder.buildGraph(files)+ t1 <- IO(System.currentTimeMillis())+ crosscutScores <- IO.blocking(detectCrosscut(rawGraph))+ graph <- IO.blocking(applyPenalty(rawGraph, crosscutScores))+ t2 <- IO(System.currentTimeMillis())+ levels <- IO.blocking(LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3))+ t3 <- IO(System.currentTimeMillis())+ regions <- IO.blocking(buildRegions(graph, levels, crosscutScores, inputRev))+ fresh = ArchitectureMap(+ regions = regions,+ fileCount = rawGraph.vertices.size,+ mapRev = inputRev.value,+ preflight = Some(preflightEval),+ outcome = MapOutcome.FullLocalCompleted+ )+ t4 <- IO(System.currentTimeMillis())+ _ <- persistMap(fresh, cached)+ t5 <- IO(System.currentTimeMillis())+ _ <- logger.info(+ s"map mode=${fresh.outcome.label} files=${files.size} build=${t1-t0}ms crosscut=${t2-t1}ms cluster=${t3-t2}ms regions=${t4-t3}ms persist=${t5-t4}ms total=${t5-t0}ms"+ )} yield freshprivate def loadCachedMap(inputRev: Rev): IO[Option[ArchitectureMap]] =@@ -629,7 +664,7 @@ class MapService()decoded = rows.flatMap(decodePersistedRegion(_, inputRev)).toVectorregions = rehydrateRelationships(decoded)- fileCount <- if (regions.isEmpty) IO.pure(0) else builder.liveFileCount()+ fileCount <- if (regions.isEmpty) IO.pure(0) else fullBuilder.liveFileCount()} yield {if (regions.isEmpty) Noneelse Some(ArchitectureMap(regions, fileCount, inputRev.value))@@ -700,9 +735,8 @@ class MapService(if (cached.exists(existing => sameRegions(existing.regions, fresh.regions))) IO.unitelse {for {- oldRegions <- queryApi.findNodesByKind(NodeKind.Region, limit = 5000)- oldEdgeIds <- loadActiveRegionEdgeIds(oldRegions.map(_.id))- _ <- submitMapPatch(oldRegions, oldEdgeIds, fresh.regions, Rev(fresh.mapRev))+ oldLogicalIds <- loadOldRegionIds()+ _ <- submitMapPatch(oldLogicalIds, fresh.regions, Rev(fresh.mapRev))} yield ()}}@@ -710,103 +744,189 @@ class MapService(private def sameRegions(left: Vector[Region], right: Vector[Region]): Boolean =left.sortBy(_.id.value.toString) == right.sortBy(_.id.value.toString)- private def loadActiveRegionEdgeIds(regionIds: Vector[NodeId]): IO[Vector[EdgeId]] =- if (regionIds.isEmpty) IO.pure(Vector.empty)- else {- val ids = regionIds.map(_.value.toString).toArray- client.query(- """FOR e IN edges- | FILTER e.deleted_rev == null- | AND e.predicate == "IN_REGION"- | AND (e.src IN @regionIds OR e.dst IN @regionIds)- | RETURN e._key""".stripMargin,- Map("regionIds" -> ids.asInstanceOf[AnyRef])- ).map(_.flatMap(_.as[String].toOption).flatMap(id => scala.util.Try(UUID.fromString(id)).toOption.map(EdgeId(_))).toVector)- }+ /** Fetch only the logical_ids of current live region nodes — cheaper than full GraphNode. */+ private def loadOldRegionIds(): IO[Vector[String]] =+ client.query(+ """FOR n IN nodes+ | FILTER n.kind == "region"+ | AND n.deleted_rev == null+ | LIMIT 5000+ | RETURN n.logical_id""".stripMargin,+ Map.empty+ ).map(_.flatMap(_.as[String].toOption).toVector)+ /**+ * Persist map regions using direct bulk operations instead of the per-op transaction path.+ *+ * This replaces the old commitPatch approach (which made N+1 DB round-trips inside+ * a transaction, causing timeouts on large repos) with five bulk operations:+ * 1. Batch AQL tombstone for old region nodes+ * 2. Batch AQL tombstone for old region edges+ * 3. bulkInsert for new region nodes+ * 4. bulkInsertEdges for IN_REGION edges+ * 5. Revision update+ */private def submitMapPatch(- oldRegions: Vector[GraphNode],- oldEdgeIds: Vector[EdgeId],- regions: Vector[Region],- rev: Rev+ oldRegionIds: Vector[String],+ regions: Vector[Region],+ rev: Rev): IO[Unit] = {- val deleteOps: Vector[PatchOp] = oldRegions.map(n => PatchOp.DeleteNode(n.id))- val deleteEdgeOps: Vector[PatchOp] = oldEdgeIds.map(PatchOp.DeleteEdge)-- val nodeOps: Vector[PatchOp] = regions.map { r =>- PatchOp.UpsertNode(- id = r.id,- kind = NodeKind.Region,- name = r.label,- attrs = Map(- "label" -> r.label.asJson,- "map_generated" -> Json.True,- "level" -> r.level.asJson,- "label_kind" -> r.labelKind.asJson,- "file_count" -> r.memberFiles.size.asJson,- "member_files" -> r.memberFiles.map(_.value.toString).asJson,- "parent_id" -> r.parentId.map(_.value.toString).asJson,- "child_region_ids" -> r.childRegionIds.toVector.map(_.value.toString).sorted.asJson,- "cohesion" -> r.cohesion.asJson,- "external_coupling" -> r.externalCoupling.asJson,- "boundary_ratio" -> r.boundaryRatio.asJson,- "confidence" -> r.confidence.asJson,- "crosscut_score" -> r.crosscutScore.asJson,- "dominant_signals" -> r.dominantSignals.asJson,- "interface_node_count" -> r.interfaceNodeCount.asJson,- "map_rev" -> r.mapRev.asJson- )- )+ if (oldRegionIds.isEmpty && regions.isEmpty) return IO.unit++ val now = Instant.now().toString+ val provMap = {+ val m = new java.util.HashMap[String, AnyRef]()+ m.put("source_uri", "ix:map")+ m.put("source_hash", null)+ m.put("extractor", "map-service")+ m.put("source_type", "inferred")+ m.put("observed_at", now)+ m}- // IN_REGION edges: region → file (level-1 regions only, top-down)- val fileEdgeOps: Vector[PatchOp] = regions- .filter(_.level == 1)- .flatMap { r =>- r.memberFiles.toVector.map { fileId =>- val edgeId = EdgeId(- UUID.nameUUIDFromBytes(s"in_region:${r.id.value}:${fileId.value}".getBytes("UTF-8"))- )- PatchOp.UpsertEdge(- id = edgeId,- src = r.id,- dst = fileId,- predicate = EdgePredicate("IN_REGION"),- attrs = Map.empty- )- }- }+ for {+ latestRev <- queryApi.getLatestRev+ newRev = latestRev.value + 1L++ // 1. Tombstone old region nodes+ _ <- if (oldRegionIds.isEmpty) IO.unit+ else {+ val ids = new java.util.ArrayList[String]()+ oldRegionIds.foreach(ids.add)+ client.execute(+ """FOR n IN nodes+ | FILTER n.logical_id IN @ids AND n.deleted_rev == null+ | UPDATE n WITH { deleted_rev: @rev, updated_at: @now } IN nodes""".stripMargin,+ Map(+ "ids" -> ids.asInstanceOf[AnyRef],+ "rev" -> Long.box(newRev).asInstanceOf[AnyRef],+ "now" -> now.asInstanceOf[AnyRef]+ )+ )+ }++ // 2. Tombstone all active IN_REGION edges in one indexed query.+ // Replaces the old loadActiveRegionEdgeIds approach (which did a full+ // collection scan with a large src/dst IN clause — the main freeze point).+ _ <- client.execute(+ """FOR e IN edges+ | FILTER e.predicate == "IN_REGION"+ | AND e.deleted_rev == null+ | UPDATE e WITH { deleted_rev: @rev, updated_at: @now } IN edges""".stripMargin,+ Map(+ "rev" -> Long.box(newRev).asInstanceOf[AnyRef],+ "now" -> now.asInstanceOf[AnyRef]+ )+ )++ // 3. Bulk-insert new region nodes+ nodeDocs = regions.map(r => buildRegionNodeDoc(r, newRev, now, provMap))+ _ <- if (nodeDocs.isEmpty) IO.unit+ else client.bulkInsert("nodes", nodeDocs).void++ // 4. Bulk-insert IN_REGION edges+ edgeDocs = buildRegionEdgeDocs(regions, newRev, now, provMap)+ _ <- if (edgeDocs.isEmpty) IO.unit+ else client.bulkInsertEdges("edges", edgeDocs).void++ // 5. Update current revision+ _ <- client.execute(+ """UPSERT { _key: "current" }+ | INSERT { _key: "current", rev: @rev }+ | UPDATE { rev: @rev }+ | IN revisions""".stripMargin,+ Map("rev" -> Long.box(newRev).asInstanceOf[AnyRef])+ )+ } yield ()+ }- // IN_REGION edges: parent region → child region (top-down)- val regionEdgeOps: Vector[PatchOp] = regions.flatMap { r =>+ private def buildRegionNodeDoc(+ r: Region,+ newRev: Long,+ now: String,+ provMap: java.util.Map[String, AnyRef]+ ): java.util.Map[String, AnyRef] = {+ val logicalId = r.id.value.toString+ val attrsJson = Json.obj(+ "label" -> r.label.asJson,+ "map_generated" -> Json.True,+ "level" -> r.level.asJson,+ "label_kind" -> r.labelKind.asJson,+ "file_count" -> r.memberFiles.size.asJson,+ "member_files" -> r.memberFiles.map(_.value.toString).asJson,+ "parent_id" -> r.parentId.map(_.value.toString).asJson,+ "child_region_ids" -> r.childRegionIds.toVector.map(_.value.toString).sorted.asJson,+ "cohesion" -> r.cohesion.asJson,+ "external_coupling" -> r.externalCoupling.asJson,+ "boundary_ratio" -> r.boundaryRatio.asJson,+ "confidence" -> r.confidence.asJson,+ "crosscut_score" -> r.crosscutScore.asJson,+ "dominant_signals" -> r.dominantSignals.asJson,+ "interface_node_count" -> r.interfaceNodeCount.asJson,+ "map_rev" -> r.mapRev.asJson+ ).noSpaces+ val doc = new java.util.HashMap[String, AnyRef]()+ doc.put("_key", s"${logicalId}_${newRev}")+ doc.put("logical_id", logicalId)+ doc.put("id", logicalId)+ doc.put("kind", "region")+ doc.put("name", r.label)+ doc.put("attrs", mapper.readValue(attrsJson, classOf[java.util.Map[String, AnyRef]]))+ doc.put("provenance", provMap)+ doc.put("created_rev", Long.box(newRev))+ doc.put("deleted_rev", null)+ doc.put("created_at", now)+ doc.put("updated_at", now)+ doc+ }++ private def buildRegionEdgeDocs(+ regions: Vector[Region],+ newRev: Long,+ now: String,+ provMap: java.util.Map[String, AnyRef]+ ): Vector[java.util.Map[String, AnyRef]] = {+ val fileEdges = regions.filter(_.level == 1).flatMap { r =>+ r.memberFiles.toVector.map { fileId =>+ val edgeId = UUID.nameUUIDFromBytes(s"in_region:${r.id.value}:${fileId.value}".getBytes("UTF-8"))+ buildRegionEdgeDoc(edgeId.toString, s"nodes/${r.id.value}", s"nodes/${fileId.value}",+ r.id.value.toString, fileId.value.toString, newRev, now, provMap)+ }+ }+ val regionEdges = regions.flatMap { r =>r.parentId.map { pid =>- val edgeId = EdgeId(- UUID.nameUUIDFromBytes(s"in_region:${pid.value}:${r.id.value}".getBytes("UTF-8"))- )- PatchOp.UpsertEdge(- id = edgeId,- src = pid,- dst = r.id,- predicate = EdgePredicate("IN_REGION"),- attrs = Map.empty- )+ val edgeId = UUID.nameUUIDFromBytes(s"in_region:${pid.value}:${r.id.value}".getBytes("UTF-8"))+ buildRegionEdgeDoc(edgeId.toString, s"nodes/${pid.value}", s"nodes/${r.id.value}",+ pid.value.toString, r.id.value.toString, newRev, now, provMap)}}+ fileEdges ++ regionEdges+ }- val allOps = deleteEdgeOps ++ deleteOps ++ nodeOps ++ fileEdgeOps ++ regionEdgeOps- if (allOps.isEmpty) return IO.unit-- val patch = GraphPatch(- patchId = PatchId(UUID.nameUUIDFromBytes(s"map:${rev.value}".getBytes("UTF-8"))),- actor = "map-service",- timestamp = Instant.now(),- source = PatchSource("ix:map", None, "map-service", SourceType.Inferred),- baseRev = rev,- ops = allOps,- replaces = Vector.empty,- intent = Some(s"Architectural map computed at rev ${rev.value}")- )-- writeApi.commitPatch(patch).void+ private def buildRegionEdgeDoc(+ key: String,+ from: String,+ to: String,+ src: String,+ dst: String,+ newRev: Long,+ now: String,+ provMap: java.util.Map[String, AnyRef]+ ): java.util.Map[String, AnyRef] = {+ val doc = new java.util.HashMap[String, AnyRef]()+ doc.put("_key", key)+ doc.put("_from", from)+ doc.put("_to", to)+ doc.put("id", key)+ doc.put("src", src)+ doc.put("dst", dst)+ doc.put("predicate", "IN_REGION")+ doc.put("attrs", new java.util.HashMap[String, AnyRef]())+ doc.put("created_rev", Long.box(newRev))+ doc.put("deleted_rev", null)+ doc.put("provenance", provMap)+ doc.put("created_at", now)+ doc.put("updated_at", now)+ doc}}diff --git a/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scalaindex 9d88ee03..8a7ab64b 100644--- a/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala+++ b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala@@ -52,6 +52,24 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {view.summary.crossCutting shouldBe 1}+ "collapseDuplicateHierarchyLevels" should "keep only the coarsest region for repeated member sets" in {+ val sharedId = NodeId(UUID.randomUUID())+ val sharedFiles = (1 to 10).map(_ => NodeId(UUID.randomUUID())).toSet+ val child = region("Edges / Resolver", "module", 1, files = 4, parentId = Some(sharedId))+ val module = regionWithMembers("Edges", "module", 1, sharedId, sharedFiles)+ val subsystem = regionWithMembers("Edges", "subsystem", 2, sharedId, sharedFiles)+ val system = regionWithMembers("Edges", "system", 3, sharedId, sharedFiles)++ val collapsed = service.collapseDuplicateHierarchyLevels(Vector(child, module, subsystem, system))++ collapsed.map(region => region.label -> region.labelKind) shouldBe Vector(+ "Edges / Resolver" -> "module",+ "Edges" -> "system"+ )+ collapsed.map(_.id) should contain (sharedId)+ collapsed.find(_.label == "Edges / Resolver").flatMap(_.parentId) shouldBe Some(sharedId)+ }+private def region(label: String,labelKind: String,@@ -60,9 +78,21 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {confidence: Double = 0.80,crosscut: Double = 0.0,parent: Option[Region] = None,+ parentId: Option[NodeId] = None,+ ): Region = {+ regionWithMembers(label, labelKind, level, NodeId(UUID.randomUUID()), (1 to files).map(_ => NodeId(UUID.randomUUID())).toSet, confidence, crosscut, parent.map(_.id).orElse(parentId))+ }++ private def regionWithMembers(+ label: String,+ labelKind: String,+ level: Int,+ id: NodeId,+ memberFiles: Set[NodeId],+ confidence: Double = 0.80,+ crosscut: Double = 0.0,+ parentId: Option[NodeId] = None,): Region = {- val id = NodeId(UUID.randomUUID())- val memberFiles = (1 to files).map(_ => NodeId(UUID.randomUUID())).toSetRegion(id = id,label = label,@@ -70,7 +100,7 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {level = level,memberFiles = memberFiles,childRegionIds = Set.empty,- parentId = parent.map(_.id),+ parentId = parentId,cohesion = 0.5,externalCoupling = 0.2,boundaryRatio = 1.5,