Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

feat: large repo ingest perf, map improvements, and Claude plugin v2

kagebinary4d4dfa6d58db
diff --git a/CLAUDE.md b/CLAUDE.md
index 15f1f736..61258438 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -14,7 +14,7 @@ Use JSON when chaining command results.
2. AFTER every design or architecture decision → run `ix decide <title> --rationale <text>`.
3. When you notice contradictory information → run `ix conflicts` and present results to the user.
4. NEVER guess about codebase facts — if Ix has structured data, use it.
-5. IMMEDIATELY after modifying code → run `ix map` to re-ingest and update the graph.
+5. IMMEDIATELY after modifying code → run `ix map --silent` to re-ingest and update the graph.
6. When the user states a goal → run `ix truth add "<statement>"`.
## Ix CLI Command Routing
@@ -107,7 +107,7 @@ Workflows are staged command sequences (discover → implement → validate) att
```json
{
"discover": ["ix overview AuthService", "ix impact AuthService"],
- "implement": ["ix map"],
+ "implement": ["ix map --silent"],
"validate": ["ix smells --format json", "ix subsystems --format json"]
}
```
@@ -123,7 +123,7 @@ Workflows are staged command sequences (discover → implement → validate) att
### Ingestion & Health
| Goal | Command | Example |
|---|---|---|
-| Update graph + map | `ix map` | `ix map` |
+| Update graph + map | `ix map --silent` | `ix map --silent` |
| Ingest GitHub data | `ix ingest` | `ix ingest --github owner/repo --limit 50` |
| Backend health | `ix status` | `ix status` |
| Graph statistics | `ix stats` | `ix stats --format json` |
diff --git a/docs/plans/2026-03-25-large-repo-ingest-perf.md b/docs/plans/2026-03-25-large-repo-ingest-perf.md
new file mode 100644
index 00000000..ce589a67
--- /dev/null
+++ b/docs/plans/2026-03-25-large-repo-ingest-perf.md
@@ -0,0 +1,288 @@
+# Large Repo Ingest Performance
+
+Date: 2026-03-25
+Branch: `feature/claude-plugin`
+
+---
+
+## Baseline
+
+Kubernetes repo (`files=16010`) before this work:
+
+| Phase | Time |
+|---|---|
+| `ix map` ingest | ~466s |
+| `ix map` map computation | ~53s |
+| **Total `ix map`** | **~519s** |
+
+Debug profile at baseline:
+```
+resolveEdgesMs=29581 buildPatchMs=10156 commitMs=386443 totalSaveMs=426834
+```
+
+The commit path dominated — 386s out of 427s total save time.
+
+---
+
+## What We Did (already merged)
+
+### 1. Compound DB indexes (`memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala`)
+
+Added compound indexes on `nodes(logical_id, deleted_rev)` and `claims(entity_id, deleted_rev)`. Tombstone + retire AQL queries filter both fields; without this, Arango returned all historical node versions then post-filtered. Now it does direct point lookups.
+
+### 2. Upfront batch tombstone + retire (`memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`)
+
+In `commitBatchChunked`, when multiple DB chunks exist and `baseRev > 0`, now collects all entity IDs upfront and runs tombstone + retire **once in parallel** before any inserts (`skipTombstone = true` passed to each per-chunk call). Previously ran 320 AQL UPDATE passes for Kubernetes; now runs 2.
+
+Also parallelized tombstone + retire within `commitBatch` for the single-chunk case using `IO.both`.
+
+### 3. Generated file filtering (`ix-cli/src/cli/commands/ingest.ts`)
+
+Skip known generated files by filename before parsing: `*.pb.go`, `zz_generated*`, `*_deepcopy.go`, `*_mock.go`, `mock_*`, `*.pb.ts`, `*.pb.js`, `_pb.ts`, `_pb.js`. Applied in both `walkFiles` and `tryGitLsFiles`.
+
+### 4. Parallel HTTP save requests (`ix-cli/src/cli/commands/ingest.ts`)
+
+Replaced sequential `for` loop in `commitPreparedPatches` with a 4-worker pool (`COMMIT_CONCURRENCY = 4` at line ~416). Fixed progress bar to use `Math.max` so it doesn't jump backward when an out-of-order chunk completes.
+
+---
+
+## Current State (2026-03-25, pre-session baseline)
+
+| Phase | Before | After | Delta |
+|---|---|---|---|
+| Ingest | ~466s | ~200s | **-57%** |
+| Map computation | ~53s | ~87s | +64% (regression) |
+| **Total `ix map`** | **~519s** | **~287s** | **-45%** |
+
+Graph after full Kubernetes ingest:
+```
+nodes: 312,140 (113k chunk, 76k module, 43k function, 42k method, 16k class, 15k file)
+edges: 840,843 (319k CALLS, 107k CONTAINS_CHUNK, 105k CONTAINS, 95k NEXT, 76k IMPORTS)
+```
+
+---
+
+## Current State (2026-03-26, post-session)
+
+| Phase | Before | After | Delta |
+|---|---|---|---|
+| Ingest | ~200s | ~227s | +13% (conflict fix overhead) |
+| Map computation | ~87s | ~82s | -6% |
+| **Total `ix map`** | **~287s** | **~309s** | +8% |
+
+Ingest debug breakdown (15035 files, full fresh ingest):
+```
+resolveEdgesMs=26252 buildPatchMs=8731 commitMs=607952 totalSaveMs=188862
+```
+
+Ingest is currently slightly slower than the pre-session baseline. The chunk-skipping benefit (~30% fewer DB inserts) is real but offset by the COMMIT_CONCURRENCY revert to 4 (was already 4 at baseline, no gain there) and normal run variance. Ingest is clean — zero write-write conflict errors.
+
+Map regression (82s vs 53s pre-"What We Did") is **unresolved** — this is Task 3.
+
+---
+
+## Targets
+
+| Phase | Current | Target |
+|---|---|---|
+| Ingest | ~227s | ~90s |
+| Map computation | ~82s | ~20s |
+| **Total** | **~309s** | **~110s** |
+
+---
+
+## Task List
+
+Tasks are independent unless noted. Each is scoped to be completable by a single Claude instance with no context from the others.
+
+---
+
+### Task 1 — Bump `COMMIT_CONCURRENCY` 4 → 6 ⚠️ BLOCKED
+
+**Priority:** High — blocked by write-write conflict bug (see Task 1b below).
+
+Bumping to 6 caused systemic write-write conflicts in ArangoDB. Root cause was identified and fixed (see Task 1b), but after that fix concurrency was left at 4. Re-attempting the bump to 6 should now be safe — the `updateRevision` conflict is eliminated, and `retryOnConflict` handles any residual tombstone conflicts.
+
+**Acceptance:** Run `IX_DEBUG=1 ix map <kubernetes-repo>` with no `[commit error]` lines and confirm `commitMs` drops vs the 607952ms baseline.
+
+---
+
+### Task 1b — Fix `updateRevision` write-write conflict ✅ DONE
+
+**Root cause:** `updateRevision` used an AQL `UPSERT { _key: "current" }` on the `revisions` collection. Every HTTP commit request ends with this call. With multiple concurrent workers, all requests raced to update the same document → ArangoDB write-write conflict error 1200.
+
+**Fix (`memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`):**
+Replaced the AQL UPSERT with two Document API inserts via the existing `bulkInsert(..., overwriteMode="replace")`. The Document API replace is atomic at the document level — concurrent replaces on the same `_key` don't conflict; last writer wins (acceptable since all concurrent requests write the same rev value).
+
+Also added `retryOnConflict` in the client (`ix-cli/src/cli/commands/ingest.ts`) with 10ms base jitter backoff as a safety net for any residual tombstone-phase conflicts.
+
+---
+
+### Task 2 — Skip chunk nodes/edges during map-triggered ingest ✅ DONE
+
+**Priority:** High — estimated ~30% fewer DB inserts.
+
+**Files:**
+- `ix-cli/src/cli/commands/ingest.ts` — add `skipChunks` flag, filter nodes/edges before patch submission
+- `ix-cli/src/client/api.ts` — pass flag through to request if needed
+- `ix-cli/src/client/types.ts` — add type if needed
+
+**Background:**
+Current insert volume per 200-file HTTP batch (estimated):
+- ~4,154 nodes, of which ~36% are `chunk` kind nodes
+- ~11,186 edges, of which ~24% are `CONTAINS_CHUNK` + `NEXT` edges
+
+Chunk nodes and their edges are LLM context artifacts (for `ix read`). The Louvain clustering in `MapService.computeAndPersistMap` runs on `IMPORTS` and `CALLS` edges between file nodes — it never touches chunks. Skipping them during `ix map` doesn't affect map quality.
+
+**What to do:**
+
+1. In `ingest.ts`, find where `PreparedPatch` objects are built (the `buildPatch` / `resolveEdges` path). Add a `mapMode: boolean` parameter that, when true, strips:
+ - Any node with `kind === "chunk"`
+ - Any edge with `kind === "CONTAINS_CHUNK"` or `kind === "NEXT"`
+
+2. The `ix map` code path in `ingest.ts` (around line ~544 and ~598, the `commitPreparedPatches` calls) should pass `mapMode: true`.
+
+3. The `ix ingest` code path (full ingest, used for `ix read`) must continue writing chunks — pass `mapMode: false` or omit the flag.
+
+4. Do not change the server side. The filtering should be purely client-side before the HTTP payload is built.
+
+**Acceptance:**
+- `IX_DEBUG=1 ix map <kubernetes-repo>` shows fewer nodes/edges inserted per batch (verify via backend `insertNodes`/`insertEdges` log lines)
+- `ix read <any-file>` after a full `ix ingest` still returns chunk content (chunks must still be written on full ingest)
+- No change to `ix map` output quality (regions/subsystems unchanged or better)
+
+---
+
+### Task 3 — Profile and fix map computation regression (82s → target 20s)
+
+**Priority:** High — map computation is the bottleneck for incremental `ix map` runs. Every file save triggers a re-map; keeping this fast matters more than ingest speed.
+
+**Note on terminology:** `ix ingest` is deprecated. `ix map` is the primary command. It has two phases: ingest phase (parse + commit files) and map computation phase (Louvain clustering + persist). Task 3 is about the map computation phase only.
+
+**Files:**
+- `memory-layer/src/main/scala/ix/memory/map/MapService.scala` — primary target
+- `memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala` — if `build()` is the bottleneck
+
+**Current timing:** map computation = ~82s on Kubernetes (15035 files, 828 regions). Pre-regression baseline was ~53s. Target: ≤20s (30s acceptable).
+
+**Background:**
+`computeAndPersistMap` has four steps:
+1. `builder.build()` — loads file graph from DB (IMPORTS/CALLS edges between file nodes)
+2. `detectCrosscut(rawGraph)` — pure CPU
+3. `LouvainClustering.cluster(...)` — pure CPU, `maxLevels=3`
+4. `persistMap(fresh, cached)` — writes region nodes to DB, compares old vs new regions
+
+The regression likely lives in `builder.build()` (more nodes/edges to load since ingest changes) or `persistMap` (`sameRegions` comparison may be slow at 828 regions).
+
+**What to do:**
+
+1. Read `MapService.scala` first to find `computeAndPersistMap` and understand the current structure before adding instrumentation.
+
+2. Add coarse `System.currentTimeMillis()` timing around each of the four steps and log at INFO level:
+ ```scala
+ val t0 = System.currentTimeMillis()
+ rawGraph <- builder.build()
+ val t1 = System.currentTimeMillis()
+ // ... etc
+ logger.info(s"map build=${t1-t0}ms crosscut=${t2-t1}ms cluster=${t3-t2}ms persist=${t4-t3}ms")
+ ```
+
+3. After adding instrumentation, rebuild and restart the backend:
+ ```bash
+ # from repo root (/home/ianhock/ix/IX-Memory)
+ ./rebuild.sh
+ ```
+
+4. Run to collect the timing breakdown:
+ ```bash
+ IX_DEBUG=1 ix map /home/ianhock/kubernetes 2>&1 | tee /tmp/ix-map-debug.log
+ grep "map build=" /tmp/ix-map-debug.log
+ ```
+
+5. Fix the dominant step:
+ - If `builder.build()` is slow: check `MapGraphBuilder` — it queries all IMPORTS/CALLS edges between file nodes. Look for missing indexes or N+1 queries. Check `ArangoSchema.scala` for relevant indexes.
+ - If `persistMap` is slow: check `sameRegions` — it sorts and compares full `Region` objects. At 828 regions this is O(n log n) with potentially high constant. Also check `submitMapPatch` / `loadOldRegionIds`.
+ - If `LouvainClustering.cluster` is slow: the graph got larger. Check how many file nodes are passed to the clusterer vs pre-regression.
+
+**Acceptance:**
+- Map computation ≤ 30s on Kubernetes (20s target, 30s acceptable for this task)
+- Timing log line visible in backend output during `IX_DEBUG=1 ix map`
+- No regression in `ix map` output (828 regions expected, same or similar structure)
+- Rebuild via `./rebuild.sh` from repo root before testing
+
+---
+
+### Task 4 — Vacuum old tombstoned node versions
+
+**Priority:** Medium — reduces DB size and disk I/O over time; not urgent for this sprint.
+
+**Files:**
+- `memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala` — add vacuum method
+- `memory-layer/src/main/scala/ix/memory/api/` — add a `/vacuum` or `/admin/vacuum` HTTP endpoint
+- `ix-cli/src/client/api.ts` and `ix-cli/src/cli/commands/` — add `ix vacuum` CLI command
+
+**Background:**
+Every ingest revision leaves behind tombstoned docs in the `nodes` collection (`deleted_rev IS NOT NULL`). After N ingests of Kubernetes (312k live nodes), the collection has ~N × 312k docs. These bloat ArangoDB's working set, spill indexes to disk, and account for the 12.4GB disk write profile.
+
+**What to do:**
+
+1. In `ArangoClient.scala`, add a method:
+ ```scala
+ def vacuumTombstoned(keepRevisions: Int): IO[Long]
+ ```
+ That runs:
+ ```aql
+ FOR n IN nodes
+ FILTER n.deleted_rev != null
+ AND n.deleted_rev < @cutoffRev
+ LIMIT 50000
+ REMOVE n IN nodes
+ RETURN 1
+ ```
+ Where `cutoffRev = latestRev - keepRevisions`. Use `LIMIT 50000` per pass and loop until 0 rows removed to avoid ArangoDB memory pressure on large deletes. Apply the same pattern to the `edges` and `claims` collections.
+
+2. Expose it via a backend endpoint (e.g. `POST /admin/vacuum?keepRevisions=5`).
+
+3. Add `ix vacuum [--keep-revisions N]` CLI command (default `--keep-revisions 5`).
+
+**Acceptance:**
+- After `ix vacuum` on Kubernetes graph, `nodes` collection count drops by at least 50% (or confirms there are minimal tombstoned docs if this is a fresh graph)
+- Subsequent `ix map` run shows reduced disk I/O in `IX_DEBUG=1` output
+- `ix vacuum` completes without OOM or ArangoDB timeout (batched deletes required)
+
+---
+
+### Task 5 — Dynamic concurrency auto-tuning (stretch goal)
+
+**Priority:** Low — only needed if fixed concurrency of 6 still leaves headroom.
+
+**File:** `ix-cli/src/cli/commands/ingest.ts`
+
+**What to do:**
+Replace the fixed `COMMIT_CONCURRENCY` constant with a simple AIMD (additive increase / multiplicative decrease) controller:
+- Track rolling P90 latency of the last 10 batch commits
+- If P90 < 8s, increment concurrency (up to max 12)
+- If P90 > 15s or a batch times out, decrement concurrency (min 2)
+- Log current concurrency level in the `[ingest-save]` debug line
+
+**Acceptance:**
+- On Kubernetes, concurrency auto-settles to a stable value within the first 10 batches
+- Total ingest time is ≤ fixed-concurrency-6 time (must not be slower)
+- No batch timeout errors introduced
+
+---
+
+## Profiling Instructions
+
+To collect the data needed for Task 3:
+
+```bash
+IX_DEBUG=1 ix map <kubernetes-repo-path> 2>&1 | tee /tmp/ix-map-debug.log
+```
+
+Look for:
+- Client: `[ingest-save]` lines — per-batch phase breakdown
+- Client: `[slow saving]` lines — which batch numbers are slow
+- Backend: `ix.bulk-write` log lines — `insertNodes`, `insertEdges`, `insertClaims`, `insertPatches` per-stage breakdown
+- Backend: `pre-tombstone` line — tombstone+retire time
+- Backend (after Task 3 instrumentation): `map build=Xms crosscut=Yms cluster=Zms persist=Wms`
diff --git a/docs/plans/2026-03-25-large-repo-map-followup.md b/docs/plans/2026-03-25-large-repo-map-followup.md
new file mode 100644
index 00000000..dc293c87
--- /dev/null
+++ b/docs/plans/2026-03-25-large-repo-map-followup.md
@@ -0,0 +1,154 @@
+# Large Repo `ix map` Follow-up
+
+Date: 2026-03-25
+
+## Current user workflow
+
+Use `ix map` only.
+
+Do not assume `ix ingest` is part of the normal workflow anymore. Any large-repo performance work needs to be evaluated through `ix map`, since `ix map` triggers ingest first and that is what the user experiences.
+
+## Main problem
+
+Large repos improved for map computation, but ingest/save regressed badly.
+
+Observed user report for Kubernetes:
+
+- `ix map` total ingest time on current branch: about `466s`
+- `ix map` total map time on current branch: about `53s`
+- Earlier target / expected behavior from dev: ingest closer to `90s`
+- Previous map-only improvement target: Kubernetes map around `20s`
+
+Observed debug sample from user:
+
+- `files=16010`
+- `resolveEdgesMs=29581`
+- `buildPatchMs=10156`
+- `commitMs=386443`
+- `totalSaveMs=426834`
+
+Conclusion: the dominant regression is still in the save/commit path, not parse, resolve, or map clustering.
+
+## Confirmed issues
+
+### 1. Save path is the bottleneck on large repos
+
+Client debug logs showed repeated slow commit chunks during saving, for example:
+
+- `commit 3401-3600 ... (15s)`
+- `commit 5601-5800 ... (17s)`
+- `commit 7801-8000 ... (25s)`
+
+This points at backend bulk commit behavior, especially chunk sizing and/or Arango bulk insert characteristics on large batches.
+
+### 2. `ix map` output had duplicate top-level regions
+
+The map output showed repeated roots like:
+
+- `MODULE Edges`
+- `SUBSYSTEM Edges`
+- `SYSTEM Edges`
+
+and similarly for `Builder`.
+
+Cause found:
+
+- the same file set could survive across multiple hierarchy levels
+- those duplicate regions shared the same deterministic region ID
+- they were all rendered as roots/orphans
+
+Status:
+
+- fixed in code by collapsing duplicate hierarchy levels to the coarsest retained region
+- regression test added
+- verified on a fresh backend
+
+Remaining note:
+
+- `Edges` and `Builder` may still appear once each as top-level systems because they are still weakly connected / isolated in the graph
+- that is a separate modeling issue from the duplicate-root bug
+
+## Changes already made
+
+### Map hierarchy fix
+
+Updated:
+
+- `memory-layer/src/main/scala/ix/memory/map/MapService.scala`
+- `memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala`
+
+What changed:
+
+- collapse duplicate same-member-set regions across module/subsystem/system levels
+- keep only the coarsest retained region
+- preserve children pointing at that surviving region
+
+### Ingest/save tuning already tried
+
+Updated:
+
+- `ix-cli/src/cli/commands/ingest.ts`
+- `memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`
+
+What changed:
+
+- reduced client bulk HTTP batch size from `1000` files to `200`
+- reduced backend revision chunk size from `500` files to `100`
+- reduced backend payload cap from `16MB` to `4MB`
+- capped patch document insert batches instead of inserting arbitrarily large patch arrays
+
+## Measured result after tuning
+
+On a fresh isolated backend, full ingest of this repo improved from an earlier sample with:
+
+- `commitMs ~10899`
+
+to:
+
+- `commitMs 4021`
+- total ingest `5.48s` for `221` changed files
+
+This confirms the chunk-size rollback helped on the repo-sized benchmark.
+
+## What is still unresolved
+
+The large-repo `ix map` path for Kubernetes is still much too slow.
+
+Even after chunk-size reductions, the real user-facing test case remains:
+
+- `ix map <kubernetes repo>`
+
+and the unresolved part is:
+
+- save/commit time at large scale
+
+## What to collect next
+
+Since the user uses `ix map`, collect logs from `ix map`, not a standalone `ix ingest` run.
+
+Needed evidence next time:
+
+1. Client-side `ix map` debug output including:
+ - slow saving lines
+ - final `[ingest-save]` line
+ - final `ingest Xms · map Yms`
+2. Backend `memory-layer` logs at the same time, especially `ix.bulk-write` timings
+3. Identify which backend stage dominates:
+ - `insertNodes`
+ - `insertEdges`
+ - `insertClaims`
+ - `insertPatches`
+ - tombstoning / retiring old claims
+ - revision update
+
+## Likely next investigation targets
+
+1. Arango bulk insert performance for edges and patches on very large repos
+2. Whether generated files (`zz_generated`, `.pb.go`, deepcopy/defaults, etc.) should be skipped or downgraded during `ix map`
+3. Whether `ix map` should use a cheaper ingest mode than full graph ingest for very large repos
+4. Whether chunk sizing should adapt dynamically based on actual commit latency rather than fixed file counts
+5. Whether large repo map should avoid persisting some low-value entities/claims during the map-triggered ingest path
+
+## Operational note
+
+If a fix appears to work in code but not in normal `ix map` output, confirm the running backend has actually been restarted. During this session, the old `8090` backend continued serving stale behavior until a fresh backend process was started separately.
diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.ts
index 8769f524..4dabb211 100644
--- a/ix-cli/src/cli/commands/ingest.ts
+++ b/ix-cli/src/cli/commands/ingest.ts
@@ -2,6 +2,7 @@ import * as nodePath from 'node:path';
import * as fs from 'node:fs';
import * as crypto from 'node:crypto';
import * as os from 'node:os';
+import { spawnSync } from 'node:child_process';
import type { Command } from 'commander';
import chalk from 'chalk';
import { IxClient } from '../../client/api.js';
@@ -59,13 +60,57 @@ function parseLangs(raw: string): Set<string> {
}
const IGNORE_DIRS = new Set([
- 'node_modules', '.git', 'dist', 'build', 'target', '.next',
- '.cache', '__pycache__', '.ix', '.claude', '.gitnexus',
+ 'node_modules', '.git', '.hg', '.svn', 'dist', 'build', 'target', 'out', '.next',
+ '__pycache__', '.tox', '.venv', 'venv', '.mypy_cache', '.pytest_cache',
+ '.gradle', '.idea', '.vscode', '.settings', '.vs',
+ '.cache', '.parcel-cache', 'coverage', '.nyc_output',
+ 'vendor', 'Pods', '.dart_tool', '.pub-cache',
+ 'bin', 'obj', 'pkg',
+ '.ix', '.claude', '.gitnexus',
'test', 'tests', '__tests__', 'spec', 'specs', 'e2e',
'examples', 'fixtures', '__mocks__', '__fixtures__',
]);
+// Generated file patterns — skipped by filename to avoid ingesting low-value autogenerated code.
+// These files inflate parse time and graph size without contributing to architectural signal.
+const GENERATED_FILE_SUFFIXES = [
+ '.pb.go', // protobuf generated Go
+ '_deepcopy.go', // controller-gen deepcopy (Kubernetes ecosystem)
+ '_mock.go', // gomock generated mocks
+ '.pb.ts', // protobuf generated TypeScript
+ '.pb.js', // protobuf generated JavaScript
+ '_pb.ts', // alternative protobuf TypeScript naming
+ '_pb.js', // alternative protobuf JavaScript naming
+];
+
+const GENERATED_FILE_PREFIXES = [
+ 'zz_generated', // controller-gen convention (Kubernetes ecosystem)
+ 'mock_', // gomock convention
+];
+
+function isGeneratedFile(basename: string): boolean {
+ for (const suffix of GENERATED_FILE_SUFFIXES) {
+ if (basename.endsWith(suffix)) return true;
+ }
+ for (const prefix of GENERATED_FILE_PREFIXES) {
+ if (basename.startsWith(prefix)) return true;
+ }
+ return false;
+}
+
const MAX_FILE_BYTES = 1024 * 1024; // 1 MB
+const MINIFIED_BYTES_THRESHOLD = 200 * 1024;
+const MINIFIED_MAX_LINE_THRESHOLD = 20_000;
+const MINIFIED_AVG_LINE_THRESHOLD = 2_000;
+const SLOW_WORK_LOG_MS = 5_000;
+const SLOW_WORK_REPEAT_MS = 10_000;
+
+function parsePositiveIntEnv(name: string, fallback: number): number {
+ const raw = process.env[name];
+ if (!raw) return fallback;
+ const parsed = Number.parseInt(raw, 10);
+ return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
+}
function* walkFiles(
dir: string,
@@ -76,13 +121,46 @@ function* walkFiles(
catch { return; }
for (const entry of entries) {
+ if (entry.isDirectory() && entry.name.startsWith('.') && entry.name !== '.') continue;
if (IGNORE_DIRS.has(entry.name)) continue;
const full = nodePath.join(dir, entry.name);
if (entry.isDirectory()) {
if (recursive) yield* walkFiles(full, true);
} else if (entry.isFile()) {
- if (SUPPORTED_EXTENSIONS.has(nodePath.extname(entry.name))) yield full;
+ if (SUPPORTED_EXTENSIONS.has(nodePath.extname(entry.name)) && !isGeneratedFile(entry.name)) yield full;
+ }
+ }
+}
+
+function tryGitLsFiles(dir: string, recursive: boolean): string[] | null {
+ try {
+ const result = spawnSync(
+ 'git',
+ ['ls-files', '--cached', '--others', '--exclude-standard'],
+ {
+ cwd: dir,
+ encoding: 'utf-8',
+ maxBuffer: 32 * 1024 * 1024,
+ }
+ );
+ if (result.status !== 0) return null;
+
+ const files: string[] = [];
+ for (const line of result.stdout.split(/\r?\n/)) {
+ if (!line) continue;
+ const fullPath = nodePath.resolve(dir, line);
+ if (!recursive && nodePath.dirname(fullPath) !== dir) continue;
+ if (!SUPPORTED_EXTENSIONS.has(nodePath.extname(fullPath))) continue;
+ if (isGeneratedFile(nodePath.basename(fullPath))) continue;
+ try {
+ if (fs.statSync(fullPath).isFile()) files.push(fullPath);
+ } catch {
+ // Ignore racy deletes and permission issues during discovery.
+ }
}
+ return files;
+ } catch {
+ return null;
}
}
@@ -90,6 +168,56 @@ function sha256(content: Buffer): string {
return crypto.createHash('sha256').update(content).digest('hex');
}
+function isLikelyMinifiedSource(source: string): boolean {
+ if (source.length < MINIFIED_BYTES_THRESHOLD) return false;
+
+ let lineCount = 1;
+ let currentLineLength = 0;
+ let maxLineLength = 0;
+ for (let i = 0; i < source.length; i++) {
+ if (source.charCodeAt(i) === 10) {
+ lineCount++;
+ if (currentLineLength > maxLineLength) maxLineLength = currentLineLength;
+ currentLineLength = 0;
+ } else {
+ currentLineLength++;
+ }
+ }
+ if (currentLineLength > maxLineLength) maxLineLength = currentLineLength;
+
+ const averageLineLength = source.length / Math.max(lineCount, 1);
+ return maxLineLength >= MINIFIED_MAX_LINE_THRESHOLD || (lineCount <= 50 && averageLineLength >= MINIFIED_AVG_LINE_THRESHOLD);
+}
+
+function stripChunkOps(patch: GraphPatchPayload): GraphPatchPayload {
+ return {
+ ...patch,
+ ops: patch.ops.filter(op => {
+ if (op.type === 'UpsertNode' && op['kind'] === 'chunk') return false;
+ if (op.type === 'UpsertEdge' && (op['predicate'] === 'CONTAINS_CHUNK' || op['predicate'] === 'NEXT')) return false;
+ return true;
+ }),
+ };
+}
+
+function isWriteConflict(err: unknown): boolean {
+ return String(err).includes('write-write conflict');
+}
+
+async function retryOnConflict<T>(fn: () => Promise<T>, maxRetries: number): Promise<T> {
+ for (let attempt = 0; ; attempt++) {
+ try {
+ return await fn();
+ } catch (err) {
+ if (attempt >= maxRetries || !isWriteConflict(err)) throw err;
+ // ArangoDB write-write conflicts resolve as soon as the competing transaction
+ // commits — typically <10ms. Use short jittered delay to avoid thundering herd.
+ const delay = 10 * (1 << attempt) + Math.random() * 10;
+ await new Promise<void>(r => setTimeout(r, delay));
+ }
+ }
+}
+
// ---------------------------------------------------------------------------
// Mtime cache — skip readFileSync+sha256 for unchanged files
// ---------------------------------------------------------------------------
@@ -188,9 +316,10 @@ export function registerIngestCommand(program: Command): void {
export async function ingestFiles(
path: string,
- opts: { recursive?: boolean; force?: boolean; format: string; root?: string; debug?: boolean; printSummary?: boolean; lang?: string }
+ opts: { recursive?: boolean; force?: boolean; format: string; root?: string; debug?: boolean; printSummary?: boolean; lang?: string; mapMode?: boolean }
): Promise<void> {
const debug = opts.debug || process.env.IX_DEBUG === '1';
+ const mapMode = opts.mapMode === true;
const trueStart = performance.now();
const [{ parseFile, resolveEdges, isGrammarSupported }, { buildPatchWithResolution }, { languageFromPath }] = await loadIngestionModules();
@@ -217,6 +346,13 @@ export async function ingestFiles(
const simulated = Math.floor((1 - Math.exp(-elapsed / tau)) * 0.88 * progressTotal);
display = Math.max(display, simulated);
}
+ if (debug && currentWorkLabel) {
+ const workElapsed = performance.now() - currentWorkStart;
+ if (workElapsed >= SLOW_WORK_LOG_MS && (lastSlowWorkNotice === 0 || workElapsed - lastSlowWorkNotice >= SLOW_WORK_REPEAT_MS)) {
+ lastSlowWorkNotice = workElapsed;
+ process.stderr.write(`\n [slow ${progressPhase.toLowerCase()}] ${currentWorkLabel} (${Math.round(workElapsed / 1000)}s)\n`);
+ }
+ }
process.stderr.write('\r' + renderProgressLine(progressPhase, display, progressTotal));
}, 80) : null;
@@ -226,16 +362,37 @@ export async function ingestFiles(
let filesSkipped = 0;
let parseErrors = 0;
let tooLarge = 0;
+ let minifiedLikely = 0;
let latestRev = 0;
let entitiesParsed = 0;
+ let currentWorkLabel = '';
+ let currentWorkStart = performance.now();
+ let lastSlowWorkNotice = 0;
+
+ const setCurrentWork = (label: string): void => {
+ currentWorkLabel = label;
+ currentWorkStart = performance.now();
+ lastSlowWorkNotice = 0;
+ };
// Phase timing marks
- const timings = { moduleLoadMs: 0, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, commitMs: 0 };
+ const timings = { moduleLoadMs: 0, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, buildPatchMs: 0, commitMs: 0 };
const resolveStats = {
importLookups: 0, transitiveLookups: 0, globalFallbacks: 0,
globalCandidateTotal: 0, resolvedImport: 0, resolvedTransitive: 0,
resolvedGlobal: 0, resolvedQualifier: 0, skippedSameFile: 0, skippedAmbiguous: 0,
};
+ const logSaveTiming = (
+ files: number,
+ resolveEdgesMs: number,
+ buildPatchMs: number,
+ commitMs: number,
+ totalSaveMs: number,
+ ): void => {
+ process.stderr.write(
+ `\n[ingest-save] files=${files} resolveEdgesMs=${resolveEdgesMs} buildPatchMs=${buildPatchMs} commitMs=${commitMs} totalSaveMs=${totalSaveMs}\n`
+ );
+ };
try {
// Phase: discover files
@@ -250,7 +407,7 @@ export async function ingestFiles(
const stat = fs.statSync(resolvedPath);
const filePaths: string[] = stat.isFile()
? (SUPPORTED_EXTENSIONS.has(nodePath.extname(resolvedPath)) ? [resolvedPath] : [])
- : Array.from(walkFiles(resolvedPath, opts.recursive ?? true));
+ : (tryGitLsFiles(resolvedPath, opts.recursive ?? true) ?? Array.from(walkFiles(resolvedPath, opts.recursive ?? true)));
filesDiscovered = filePaths.length;
const discovered = performance.now();
@@ -284,17 +441,19 @@ export async function ingestFiles(
for (const filePath of filePaths) {
try {
const st = fs.statSync(filePath);
- if (st.size === 0) { filesSkipped++; continue; }
- if (st.size > MAX_FILE_BYTES) { tooLarge++; continue; }
+ if (st.size === 0) { filesSkipped++; progressCurrent++; continue; }
+ if (st.size > MAX_FILE_BYTES) { tooLarge++; progressCurrent++; continue; }
const mtime = st.mtimeMs;
currentMtimes.set(filePath, mtime);
if (!opts.force && mtimeCache.get(filePath) === mtime) {
- filesSkipped++; // mtime clean — assume unchanged
+ filesSkipped++;
+ progressCurrent++; // mtime clean — assume unchanged
} else {
mtimeChangedPaths.push(filePath);
}
} catch (err) {
parseErrors++;
+ progressCurrent++;
process.stderr.write(`\n [stat error] ${filePath}: ${err}\n`);
}
}
@@ -330,55 +489,208 @@ export async function ingestFiles(
let resolveEdgesFn: Function | null = null;
let buildPatchFn: Function | null = null;
- const PARSE_STREAM_CHUNK = 1000; // resolve + commit after this many parsed files
- const COMMIT_CHUNK_SIZE = 200; // files per HTTP batch to server
- const YIELD_EVERY = 100; // yield event loop every N files during parse
+ const PARSE_STREAM_CHUNK = 500; // streaming batch size for large repos
+ const SMALL_REPO_THRESHOLD = 25_000; // parse-all-first below this; streaming above
+ const COMMIT_HTTP_MAX_FILES = parsePositiveIntEnv('IX_COMMIT_HTTP_MAX_FILES', 200); // files per HTTP request to the backend
+ const COMMIT_CONCURRENCY = parsePositiveIntEnv('IX_COMMIT_CONCURRENCY', 6); // parallel HTTP save requests
+ const YIELD_EVERY = 100; // yield event loop every N files during parse
+
+ if (debug) {
+ process.stderr.write(
+ `\n Save config: httpBatch=${COMMIT_HTTP_MAX_FILES} concurrency=${COMMIT_CONCURRENCY}\n`
+ );
+ }
const emptyEdges: any[] = [];
- /** Resolve edges within a batch, build patches, and commit in sub-chunks. */
- const flushBatch = async (batch: ParsedFile[]): Promise<void> => {
- if (batch.length === 0) return;
- filesChanged += batch.length;
+ type PreparedPatch = {
+ patch: GraphPatchPayload;
+ fileNumber: number;
+ filePath: string;
+ };
- const resolvedEdges = resolveEdgesFn!(batch.map(f => f.parsed), resolveStats);
- const batchEdgesByFile = new Map<string, any[]>();
- for (const edge of resolvedEdges) {
- let arr = batchEdgesByFile.get(edge.srcFilePath);
- if (!arr) { arr = []; batchEdgesByFile.set(edge.srcFilePath, arr); }
- arr.push(edge);
+ const makePreparedPatch = (
+ patch: GraphPatchPayload,
+ fileNumber: number,
+ filePath: string,
+ ): PreparedPatch => ({ patch, fileNumber, filePath });
+
+ // Commit HTTP batches in parallel (COMMIT_CONCURRENCY workers). Each worker pulls
+ // the next unsent chunk from a shared queue. JS is single-threaded so the queue
+ // index increment is atomic across await boundaries — no locking needed.
+ const commitPreparedPatches = async (
+ preparedPatches: PreparedPatch[],
+ totalFiles: number,
+ opts?: { updateProgress?: boolean },
+ ): Promise<number> => {
+ if (preparedPatches.length === 0) return 0;
+
+ const chunks: PreparedPatch[][] = [];
+ for (let i = 0; i < preparedPatches.length; i += COMMIT_HTTP_MAX_FILES) {
+ chunks.push(preparedPatches.slice(i, i + COMMIT_HTTP_MAX_FILES));
}
- for (let i = 0; i < batch.length; i += COMMIT_CHUNK_SIZE) {
- const end = Math.min(i + COMMIT_CHUNK_SIZE, batch.length);
- const patchChunk: GraphPatchPayload[] = [];
- for (let j = i; j < end; j++) {
- const { parsed: p, hash, previousHash } = batch[j];
- try {
- patchChunk.push(buildPatchFn!(p, hash, batchEdgesByFile.get(p.filePath) ?? emptyEdges, previousHash));
- } catch (err) {
- parseErrors++;
- process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);
- }
- }
- if (patchChunk.length === 0) continue;
+ const commitMsPerChunk = new Array<number>(chunks.length).fill(0);
+ let nextChunk = 0;
+
+ const runChunk = async (ci: number): Promise<void> => {
+ const chunk = chunks[ci];
+ const startFile = chunk[0].fileNumber;
+ const endFile = chunk[chunk.length - 1].fileNumber;
+ const endingPath = nodePath.basename(chunk[chunk.length - 1].filePath);
+ const patches = chunk.map(item => item.patch);
+
try {
- const result = await client.commitPatchBulk(patchChunk);
+ setCurrentWork(`commit ${startFile}-${endFile} of ${totalFiles} ending ${endingPath}`);
+ const commitStart = performance.now();
+ const result = await retryOnConflict(() => client.commitPatchBulk(patches), 3);
+ commitMsPerChunk[ci] = Math.round(performance.now() - commitStart);
if (result.rev > latestRev) latestRev = result.rev;
- patchesApplied += patchChunk.length;
+ patchesApplied += patches.length;
} catch (err) {
- if (debug) process.stderr.write(`\n [bulk chunk failed, falling back to per-file] ${err}\n`);
- for (const patch of patchChunk) {
+ if (debug) {
+ process.stderr.write(
+ `\n [bulk failed, falling back to per-file] files ${startFile}-${endFile} (${patches.length} patches): ${err}\n`
+ );
+ }
+ for (const item of chunk) {
try {
- const result = await client.commitPatch(patch);
+ const commitStart = performance.now();
+ const result = await retryOnConflict(() => client.commitPatch(item.patch), 3);
+ commitMsPerChunk[ci] += Math.round(performance.now() - commitStart);
if (result.rev > latestRev) latestRev = result.rev;
patchesApplied++;
} catch (commitErr) {
parseErrors++;
- process.stderr.write(`\n [commit error] ${patch.source?.uri}: ${commitErr}\n`);
+ process.stderr.write(`\n [commit error] ${item.patch.source?.uri}: ${commitErr}\n`);
}
}
}
+
+ if (opts?.updateProgress) {
+ progressCurrent = Math.max(progressCurrent, endFile);
+ }
+ };
+
+ const worker = async (): Promise<void> => {
+ while (true) {
+ const ci = nextChunk++;
+ if (ci >= chunks.length) break;
+ await runChunk(ci);
+ }
+ };
+
+ await Promise.all(
+ Array.from({ length: Math.min(COMMIT_CONCURRENCY, chunks.length) }, () => worker())
+ );
+
+ return commitMsPerChunk.reduce((a, b) => a + b, 0);
+ };
+
+ /** Resolve edges within a batch, build patches, and commit in sub-chunks. */
+ const flushBatch = async (batch: ParsedFile[]): Promise<void> => {
+ if (batch.length === 0) return;
+ filesChanged += batch.length;
+ const previousPhase = progressPhase;
+ const saveStart = performance.now();
+ let resolveEdgesMs = 0;
+ let buildPatchMs = 0;
+ let commitMs = 0;
+ progressPhase = 'Saving';
+ progressStart = performance.now();
+ try {
+ setCurrentWork(`resolve+commit batch ending ${nodePath.basename(batch[batch.length - 1].filePath)}`);
+
+ const resolveStart = performance.now();
+ const resolvedEdges = resolveEdgesFn!(batch.map(f => f.parsed), resolveStats);
+ resolveEdgesMs = Math.round(performance.now() - resolveStart);
+ const batchEdgesByFile = new Map<string, any[]>();
+ for (const edge of resolvedEdges) {
+ let arr = batchEdgesByFile.get(edge.srcFilePath);
+ if (!arr) { arr = []; batchEdgesByFile.set(edge.srcFilePath, arr); }
+ arr.push(edge);
+ }
+
+ setCurrentWork(`build commit payloads ending ${nodePath.basename(batch[batch.length - 1].filePath)}`);
+ const buildStart = performance.now();
+ const preparedPatches: PreparedPatch[] = [];
+ for (let j = 0; j < batch.length; j++) {
+ const { parsed: p, hash, previousHash } = batch[j];
+ try {
+ const rawPatch = buildPatchFn!(p, hash, batchEdgesByFile.get(p.filePath) ?? emptyEdges, previousHash);
+ const patch = mapMode ? stripChunkOps(rawPatch) : rawPatch;
+ preparedPatches.push(makePreparedPatch(patch, j + 1, p.filePath));
+ } catch (err) {
+ parseErrors++;
+ process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);
+ }
+ }
+ buildPatchMs += Math.round(performance.now() - buildStart);
+ if (preparedPatches.length === 0) return;
+ commitMs += await commitPreparedPatches(preparedPatches, batch.length);
+ } finally {
+ const totalSaveMs = Math.round(performance.now() - saveStart);
+ timings.resolveMs += resolveEdgesMs;
+ timings.buildPatchMs += buildPatchMs;
+ timings.commitMs += commitMs;
+ logSaveTiming(batch.length, resolveEdgesMs, buildPatchMs, commitMs, totalSaveMs);
+ progressPhase = previousPhase;
+ progressStart = performance.now();
+ }
+ };
+
+ /** Parse-all-first commit: resolve edges over full set, then commit in HTTP-sized batches. */
+ const flushAll = async (allParsed: ParsedFile[]): Promise<void> => {
+ if (allParsed.length === 0) return;
+ filesChanged += allParsed.length;
+ const previousPhase = progressPhase;
+ const saveStart = performance.now();
+ let resolveEdgesMs = 0;
+ let buildPatchMs = 0;
+ let commitMs = 0;
+ progressPhase = 'Saving';
+ progressTotal = allParsed.length;
+ progressCurrent = 0;
+ progressStart = performance.now();
+ try {
+ setCurrentWork(`resolve ${allParsed.length} files`);
+ const resolveStart = performance.now();
+ const resolvedEdges = resolveEdgesFn!(allParsed.map(f => f.parsed), resolveStats);
+ resolveEdgesMs = Math.round(performance.now() - resolveStart);
+ const edgesByFile = new Map<string, any[]>();
+ for (const edge of resolvedEdges) {
+ let arr = edgesByFile.get(edge.srcFilePath);
+ if (!arr) { arr = []; edgesByFile.set(edge.srcFilePath, arr); }
+ arr.push(edge);
+ }
+ setCurrentWork(`build commit payloads ending ${nodePath.basename(allParsed[allParsed.length - 1].filePath)}`);
+ const buildStart = performance.now();
+ const preparedPatches: PreparedPatch[] = [];
+ for (let j = 0; j < allParsed.length; j++) {
+ const { parsed: p, hash, previousHash } = allParsed[j];
+ try {
+ const rawPatch = buildPatchFn!(p, hash, edgesByFile.get(p.filePath) ?? emptyEdges, previousHash);
+ const patch = mapMode ? stripChunkOps(rawPatch) : rawPatch;
+ preparedPatches.push(makePreparedPatch(patch, j + 1, p.filePath));
+ } catch (err) {
+ parseErrors++;
+ process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);
+ }
+ }
+ buildPatchMs += Math.round(performance.now() - buildStart);
+ if (preparedPatches.length === 0) {
+ progressCurrent = allParsed.length;
+ return;
+ }
+ commitMs += await commitPreparedPatches(preparedPatches, allParsed.length, { updateProgress: true });
+ } finally {
+ const totalSaveMs = Math.round(performance.now() - saveStart);
+ timings.resolveMs += resolveEdgesMs;
+ timings.buildPatchMs += buildPatchMs;
+ timings.commitMs += commitMs;
+ logSaveTiming(allParsed.length, resolveEdgesMs, buildPatchMs, commitMs, totalSaveMs);
+ progressPhase = previousPhase;
+ progressStart = performance.now();
}
};
@@ -390,11 +702,12 @@ export async function ingestFiles(
try {
const bytes = fs.readFileSync(filePath);
const hash = sha256(bytes);
- if (knownHashes.get(filePath) === hash) { filesSkipped++; continue; }
+ if (knownHashes.get(filePath) === hash) { filesSkipped++; progressCurrent++; continue; }
const previousHash = knownHashes.get(filePath);
changedPaths.push({ filePath, bytes, hash, previousHash: previousHash !== hash ? previousHash : undefined });
} catch (err) {
parseErrors++;
+ progressCurrent++;
process.stderr.write(`\n [read error] ${filePath}: ${err}\n`);
}
}
@@ -406,14 +719,23 @@ export async function ingestFiles(
resolveEdgesFn = ingestion.resolveEdges as Function;
buildPatchFn = patchBuilder.buildPatchWithResolution as Function;
- let batch: ParsedFile[] = [];
+ const allParsed: ParsedFile[] = [];
for (let i = 0; i < changedPaths.length; i++) {
const { filePath, bytes, hash, previousHash } = changedPaths[i];
+ setCurrentWork(`parse ${filePath}`);
try {
- const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));
+ const sourceText = bytes.toString('utf-8');
+ if (isLikelyMinifiedSource(sourceText)) {
+ minifiedLikely++;
+ filesSkipped++;
+ progressCurrent++;
+ if (debug) process.stderr.write(`\n [skip minified-likely] ${filePath}\n`);
+ continue;
+ }
+ const parsed = (ingestion.parseFile as Function)(filePath, sourceText);
if (!parsed) { filesSkipped++; progressCurrent++; continue; }
entitiesParsed += parsed.entities.length;
- batch.push({ filePath, parsed, hash, previousHash });
+ allParsed.push({ filePath, parsed, hash, previousHash });
progressCurrent++;
} catch (err) {
parseErrors++;
@@ -421,9 +743,8 @@ export async function ingestFiles(
process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);
}
if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));
- if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }
}
- await flushBatch(batch);
+ await flushAll(allParsed);
}
} else {
// Path B: no baseline (first ingest) or --force → load modules, then stream parse + commit.
@@ -433,9 +754,12 @@ export async function ingestFiles(
resolveEdgesFn = ingestion.resolveEdges as Function;
buildPatchFn = patchBuilder.buildPatchWithResolution as Function;
- let batch: ParsedFile[] = [];
+ const useParseAllFirst = filePaths.length <= SMALL_REPO_THRESHOLD;
+ const allParsedB: ParsedFile[] = useParseAllFirst ? [] : (null as any);
+ let batch: ParsedFile[] = useParseAllFirst ? (null as any) : [];
for (let i = 0; i < filePaths.length; i++) {
const filePath = filePaths[i];
+ setCurrentWork(`parse ${filePath}`);
try {
const fileSize = fs.statSync(filePath).size;
if (fileSize === 0) { filesSkipped++; progressCurrent++; continue; }
@@ -444,10 +768,19 @@ export async function ingestFiles(
const hash = sha256(bytes);
if (!opts.force && knownHashes.get(filePath) === hash) { filesSkipped++; progressCurrent++; continue; }
const previousHash = knownHashes.get(filePath);
- const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));
+ const sourceText = bytes.toString('utf-8');
+ if (isLikelyMinifiedSource(sourceText)) {
+ minifiedLikely++;
+ filesSkipped++;
+ progressCurrent++;
+ if (debug) process.stderr.write(`\n [skip minified-likely] ${filePath}\n`);
+ continue;
+ }
+ const parsed = (ingestion.parseFile as Function)(filePath, sourceText);
if (!parsed) { filesSkipped++; progressCurrent++; continue; }
entitiesParsed += parsed.entities.length;
- batch.push({ filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined });
+ const entry: ParsedFile = { filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined };
+ if (useParseAllFirst) { allParsedB.push(entry); } else { batch.push(entry); }
progressCurrent++;
} catch (err) {
parseErrors++;
@@ -455,16 +788,17 @@ export async function ingestFiles(
process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);
}
if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));
- if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }
+ if (!useParseAllFirst && batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }
+ }
+ if (useParseAllFirst) {
+ await flushAll(allParsedB);
+ } else {
+ await flushBatch(batch);
}
- await flushBatch(batch);
}
const parsed = performance.now();
timings.parseMs = Math.round(parsed - hashed);
- timings.resolveMs = 0; // now interleaved with parse
- timings.commitMs = 0; // now interleaved with parse
-
// Clear the parse bar
if (interval) process.stderr.write('\r' + ' '.repeat(PROG_LINE_WIDTH) + '\r');
@@ -497,7 +831,7 @@ export async function ingestFiles(
filesSkipped,
entitiesParsed,
latestRev,
- skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge },
+ skipReasons: { unchanged: filesSkipped, emptyFile: 0, parseError: parseErrors, tooLarge, minifiedLikely },
elapsedSeconds: parseFloat(elapsed),
timings,
resolveStats,
@@ -510,6 +844,7 @@ export async function ingestFiles(
if (filesSkipped > 0) console.log(` ${chalk.dim('skipped unchanged:')} ${filesSkipped}`);
if (parseErrors > 0) console.log(` ${chalk.red('parse errors:')} ${parseErrors}`);
if (tooLarge > 0) console.log(` ${chalk.dim('skipped too large:')} ${tooLarge}`);
+ if (minifiedLikely > 0) console.log(` ${chalk.dim('skipped minified:')} ${minifiedLikely}`);
console.log(` rev: ${latestRev}`);
if (patchesApplied === 0 && filesDiscovered === 0) {
@@ -525,6 +860,7 @@ export async function ingestFiles(
console.log(chalk.dim(` hash: ${timings.hashMs}ms`));
console.log(chalk.dim(` parse: ${timings.parseMs}ms`));
console.log(chalk.dim(` resolve: ${timings.resolveMs}ms`));
+ console.log(chalk.dim(` build: ${timings.buildPatchMs}ms`));
console.log(chalk.dim(` commit: ${timings.commitMs}ms`));
console.log(chalk.dim(`\n Resolve stats:`));
console.log(chalk.dim(` import lookups: ${resolveStats.importLookups}`));
diff --git a/ix-cli/src/cli/commands/map.ts b/ix-cli/src/cli/commands/map.ts
index 3388ba71..52050f00 100644
--- a/ix-cli/src/cli/commands/map.ts
+++ b/ix-cli/src/cli/commands/map.ts
@@ -91,6 +91,7 @@ export function registerMapCommand(program: Command): void {
.option("--list", "Render the ranked list view instead of the default graph/tree view")
.option("--full", "Force full local map, bypassing automatic safety limits (advanced/testing)")
.option("--verbose", "Show raw confidence scores, crosscut scores, boundary ratios, and signals")
+ .option("--silent", "Suppress all output except a one-line summary (useful for LLM hooks)")
.addHelpText(
"after",
`
@@ -107,10 +108,14 @@ Advanced:
--full Override automatic local safety limits and force the full local map
path. Bypasses automatic downgrade to fast mode and the persistence
safety guardrail. Intended for testing and performance diagnosis.
+ --silent Skip the full map rendering. Prints one summary line to stderr and
+ exits. Ideal for LLM hooks and automated workflows where the full
+ output would waste context tokens.
Examples:
ix map .
ix map --format json
+ ix map --silent
ix map --level 2
ix map --min-confidence 0.5
ix map --max-items 10
@@ -121,7 +126,7 @@ Examples:
ix map . --full
ix --debug map . --full`
)
- .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; list?: boolean; full?: boolean; verbose?: boolean }) => {
+ .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; list?: boolean; full?: boolean; verbose?: boolean; silent?: boolean }) => {
const cwd = pathArg ? resolve(pathArg) : process.cwd();
try {
@@ -132,8 +137,10 @@ Examples:
return;
}
+ const silent = opts.silent === true || opts.format === "silent";
+
// Print warning when --full override is active
- if (opts.full && opts.format !== "json") {
+ if (opts.full && opts.format !== "json" && !silent) {
console.log(chalk.yellow("\nWarning"));
console.log(chalk.yellow(" Full local map override enabled.\n"));
console.log(" Ix will ignore automatic local safety limits and attempt full local mapping.");
@@ -142,14 +149,14 @@ Examples:
// Ingest the path before mapping so the graph is up to date
const ingestStart = performance.now();
- await ingestFiles(cwd, { recursive: true, format: opts.format === "json" ? "json" : "text", printSummary: false });
+ await ingestFiles(cwd, { recursive: true, format: (opts.format === "json" || silent) ? "json" : "text", printSummary: false, mapMode: true });
const ingestMs = Math.round(performance.now() - ingestStart);
const client = new IxClient(getEndpoint());
const mapBarWidth = 25;
const mapStart = performance.now();
- const mapInterval = opts.format !== "json" ? setInterval(() => {
+ const mapInterval = (opts.format !== "json" && !silent) ? setInterval(() => {
const elapsed = performance.now() - mapStart;
const pct = 1 - Math.exp(-elapsed / 4000);
const filled = Math.round(pct * mapBarWidth);
@@ -169,6 +176,17 @@ Examples:
}
if (mapInterval) { clearInterval(mapInterval); process.stderr.write('\r' + ' '.repeat(60) + '\r'); }
const mapMs = Math.round(performance.now() - mapStart);
+
+ if (silent) {
+ const systems = result.regions.filter(r => r.level === 3).length;
+ const subsystems = result.regions.filter(r => r.level === 2).length;
+ const modules = result.regions.filter(r => r.level === 1).length;
+ process.stderr.write(
+ `map: ${result.file_count} files · ${systems}s/${subsystems}ss/${modules}m regions · ${mapMs}ms\n`
+ );
+ return;
+ }
+
if (opts.format !== "json") {
process.stderr.write(chalk.dim(` ingest ${ingestMs}ms · map ${mapMs}ms\n`));
}
diff --git a/ix-cli/src/cli/commands/reset.ts b/ix-cli/src/cli/commands/reset.ts
index 696752f6..897547fb 100644
--- a/ix-cli/src/cli/commands/reset.ts
+++ b/ix-cli/src/cli/commands/reset.ts
@@ -49,19 +49,31 @@ export function registerResetCommand(program: Command): void {
}
const client = new IxClient(getEndpoint());
+ const label = opts.code ? "Wiping code graph..." : "Wiping graph...";
+ const spinnerFrames = ["⠋","⠙","⠹","⠸","⠼","⠴","⠦","⠧","⠇","⠏"];
+ let spinIdx = 0;
+ const interval = setInterval(() => {
+ process.stderr.write(`\r${chalk.cyan(spinnerFrames[spinIdx++ % spinnerFrames.length])} ${chalk.dim(label)}`);
+ }, 80);
try {
if (opts.code) {
await client.resetCode();
+ clearInterval(interval);
+ process.stderr.write("\r\x1b[K");
// Clear the mtime cache so the next ix map re-ingests all files
clearMtimeCache(process.cwd());
console.log(chalk.green("✓") + " Code graph wiped. Planning artifacts preserved.");
console.log(chalk.dim(" Run `ix map` to rebuild the code graph."));
} else {
await client.reset();
+ clearInterval(interval);
+ process.stderr.write("\r\x1b[K");
clearMtimeCache(process.cwd());
console.log(chalk.green("✓") + " Graph wiped.");
}
} catch (err: any) {
+ clearInterval(interval);
+ process.stderr.write("\r\x1b[K");
console.error(chalk.red("Error:"), err.message);
process.exitCode = 1;
return;
diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.ts
index c52fd7ab..8ce6eb69 100644
--- a/ix-cli/src/client/api.ts
+++ b/ix-cli/src/client/api.ts
@@ -189,13 +189,21 @@ export class IxClient {
}
async map(opts?: { full?: boolean }): Promise<any> {
- return this.post("/v1/map", opts ?? {});
+ const resp = await fetch(`${this.endpoint}/v1/map`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify(opts ?? {}),
+ signal: AbortSignal.timeout(30 * 60 * 1000), // 30 minute timeout
+ });
+ if (!resp.ok) {
+ const text = await resp.text();
+ throw new Error(`${resp.status}: ${text}`);
+ }
+ return resp.json();
}
- async commitPatchBulk(patches: GraphPatchPayload[]): Promise<{ rev: number }> {
- const results = await this.commitPatchBatch(patches);
- const rev = results.reduce((max, r) => Math.max(max, r.rev ?? 0), 0);
- return { rev };
+ async commitPatchBulk(patches: GraphPatchPayload[]): Promise<PatchCommitResult> {
+ return this.post<PatchCommitResult>('/v1/patches/bulk', { patches });
}
async runSmells(opts?: {
@@ -234,11 +242,31 @@ export class IxClient {
}
async reset(): Promise<{ ok: boolean; message: string }> {
- return this.post("/v1/reset", {});
+ const resp = await fetch(`${this.endpoint}/v1/reset`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({}),
+ signal: AbortSignal.timeout(10 * 60 * 1000), // 10 minutes
+ });
+ if (!resp.ok) {
+ const text = await resp.text();
+ throw new Error(`${resp.status}: ${text}`);
+ }
+ return resp.json() as Promise<{ ok: boolean; message: string }>;
}
async resetCode(): Promise<{ ok: boolean; message: string }> {
- return this.post("/v1/reset/code", {});
+ const resp = await fetch(`${this.endpoint}/v1/reset/code`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({}),
+ signal: AbortSignal.timeout(10 * 60 * 1000), // 10 minutes
+ });
+ if (!resp.ok) {
+ const text = await resp.text();
+ throw new Error(`${resp.status}: ${text}`);
+ }
+ return resp.json() as Promise<{ ok: boolean; message: string }>;
}
async stats(): Promise<any> {
diff --git a/ix-cli/src/client/types.ts b/ix-cli/src/client/types.ts
index 5603dc34..a41fb0f5 100644
--- a/ix-cli/src/client/types.ts
+++ b/ix-cli/src/client/types.ts
@@ -136,6 +136,7 @@ export interface SkipReasons {
emptyFile: number;
parseError: number;
tooLarge: number;
+ minifiedLikely?: number;
}
export interface IngestResult {
diff --git a/ix-plugin/.claude-plugin/plugin.json b/ix-plugin/.claude-plugin/plugin.json
index 65f37d2d..27b0dcf3 100644
--- a/ix-plugin/.claude-plugin/plugin.json
+++ b/ix-plugin/.claude-plugin/plugin.json
@@ -1,7 +1,7 @@
{
"name": "ix-memory",
- "version": "1.0.0",
- "description": "Ix Memory integration — auto-ingests edited files and front-runs Grep/Glob/Read with graph-aware ix searches for token-efficient codebase understanding.",
+ "version": "2.0.0",
+ "description": "Ix Memory integration — intercepts all searches and file reads with graph-aware ix queries, auto-ingests edits, and refreshes the full architectural map after each session.",
"author": {
"name": "Ix Memory"
},
diff --git a/ix-plugin/README.md b/ix-plugin/README.md
index c17231a6..648bdd9a 100644
--- a/ix-plugin/README.md
+++ b/ix-plugin/README.md
@@ -1,46 +1,51 @@
# Ix Memory — Claude Code Plugin
-A Claude Code plugin that makes Claude use Ix for codebase searches instead of
-native `Grep`/`Glob`, and keeps the Ix graph current as Claude edits files.
+A Claude Code plugin that makes Claude always use Ix for codebase understanding —
+injecting session context on every prompt, intercepting all searches and file reads
+with graph-aware queries, and keeping the Ix graph current as Claude edits.
## What It Does
| Trigger | Hook | Effect |
|---------|------|--------|
-| Claude runs `Grep` or `Glob` | `PreToolUse` → `ix-intercept.sh` | Runs `ix text` + `ix locate` (or `ix inventory`) first and injects results into Claude's context before the native tool fires |
-| Claude runs `Write`, `Edit`, or `MultiEdit` | `PostToolUse` → `ix-ingest.sh` (async) | Ingests the changed file into the Ix graph silently in the background |
+| User sends any prompt | `UserPromptSubmit` → `ix-briefing.sh` | Injects compact session briefing (goals, bugs, decisions, tasks) into every prompt |
+| Claude runs `Grep` or `Glob` | `PreToolUse` → `ix-intercept.sh` | Runs `ix text` + `ix locate` (or `ix inventory`) first; injects graph-aware results |
+| Claude runs `Read` | `PreToolUse` → `ix-read.sh` | Runs `ix inventory` + `ix overview` for the file; injects entity context before Claude sees raw source |
+| Claude runs `Bash` (grep/rg) | `PreToolUse` → `ix-bash.sh` | Detects grep/rg patterns and front-runs with `ix text` + `ix locate` |
+| Claude runs `Write`, `Edit`, `MultiEdit`, or `NotebookEdit` | `PostToolUse` → `ix-ingest.sh` (async) | Ingests the changed file into the Ix graph silently in the background |
+| Claude finishes responding | `Stop` → `ix-map.sh` (async) | Runs `ix map` to refresh the full architectural graph for the next session |
-Both hooks bail silently if `ix` is not in PATH or the Ix backend is unreachable.
+All hooks bail silently if `ix` is not in PATH or the Ix backend is unreachable.
## Why Hooks Instead of MCP
- **No user approval prompt** — hooks fire silently and automatically
-- **Guaranteed execution** — every Grep/Glob gets front-run, every edit gets ingested
+- **Guaranteed execution** — every prompt, search, read, and edit is covered
- **Token-efficient** — Ix returns structured, bounded results vs. raw file content
+- **Graph always current** — edits ingested immediately; full map refreshes at session end
## Installation
-**1. Add the Ix marketplace (one-time):**
+### From the repo (local)
-```
-/plugin marketplace add ix-infrastructure/Ix
+```bash
+bash ix-plugin/install.sh
```
-**2. Install the plugin:**
+### Via curl
-```
-/plugin install ix-memory
+```bash
+curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/install.sh | bash
```
-**3. Restart Claude Code.** Verify the hooks loaded with `/hooks`.
+Restart Claude Code after installing. Verify hooks loaded with `/hooks`.
## Requirements
- `ix` CLI in PATH and Ix backend running (`ix status` should return ok)
-- `jq` in PATH (used to parse hook JSON input)
-- `ripgrep` (`rg`) in PATH — required by `ix text` for file-level text search
+- `jq` in PATH
+- `ripgrep` (`rg`) in PATH — required by `ix text`
-Install missing deps:
```bash
# Ubuntu/Debian
sudo apt install jq ripgrep
@@ -57,33 +62,65 @@ ix-plugin/
plugin.json # manifest
hooks/
hooks.json # hook event registration
- ix-intercept.sh # PreToolUse: Grep|Glob → ix text/locate/inventory
- ix-ingest.sh # PostToolUse: Write|Edit|MultiEdit → ix ingest
+ ix-briefing.sh # UserPromptSubmit → ix briefing (session context)
+ ix-intercept.sh # PreToolUse: Grep|Glob → ix text + ix locate/inventory
+ ix-read.sh # PreToolUse: Read → ix inventory + ix overview
+ ix-bash.sh # PreToolUse: Bash (grep/rg) → ix text + ix locate
+ ix-ingest.sh # PostToolUse: Write|Edit|MultiEdit|NotebookEdit → ix ingest
+ ix-map.sh # Stop → ix map (full graph refresh)
+ install.sh
+ uninstall.sh
README.md
```
-## How the Intercept Hook Works
+## Hook Details
+
+### ix-briefing.sh (UserPromptSubmit)
+
+On every user prompt:
+1. Runs `ix briefing --format json` (60s health cache to avoid per-prompt overhead)
+2. Injects active goals, open bugs, recent decisions, and pending tasks as `additionalContext`
+3. Claude always has project context without needing an explicit `ix briefing` call
+
+### ix-intercept.sh (PreToolUse: Grep|Glob)
When Claude runs `Grep "verify_token"`:
+1. Runs `ix text "verify_token" --limit 20 --format json` + `ix locate "verify_token" --limit 10 --format json` in parallel
+2. Injects results as `additionalContext` — Claude uses them before Grep output arrives
-1. `ix-intercept.sh` fires with the tool input as JSON on stdin
-2. Script runs `ix text "verify_token" --limit 20 --format json`
-3. Script also runs `ix locate "verify_token" --limit 10 --format json`
-4. Results are returned as `additionalContext` — injected into Claude's next context window
-5. Claude sees the Ix results and typically uses them instead of waiting for Grep output
+For `Glob`, runs `ix inventory` (optionally filtered by path) instead.
-For `Glob`, the script runs `ix inventory` (optionally filtered by path) instead.
+### ix-read.sh (PreToolUse: Read)
-## How the Ingest Hook Works
+When Claude reads a file:
+1. Runs `ix inventory --path <file>` (all entities defined in the file)
+2. Runs `ix overview <name>` (structural summary of the module)
+3. Injects both as `additionalContext` — Claude understands the file's structure before seeing raw source
-When Claude edits a file:
+### ix-bash.sh (PreToolUse: Bash)
-1. `ix-ingest.sh` fires asynchronously after the edit succeeds
-2. Reads `tool_input.file_path` from stdin
-3. Runs `ix ingest <file_path>` silently
-4. Returns a short `additionalContext` confirmation: `[ix] Graph updated — ingested: <path>`
+When Claude runs a `grep` or `rg` command:
+1. Extracts the search pattern from the command string
+2. Runs `ix text <pattern> --limit 20` + `ix locate <pattern> --limit 10` in parallel
+3. Injects results as `additionalContext`
+
+### ix-ingest.sh (PostToolUse: Write|Edit|MultiEdit|NotebookEdit)
+
+After each file edit:
+1. Reads `tool_input.file_path` from stdin
+2. Runs `ix ingest <file>` silently in the background
+3. Returns `additionalContext`: `[ix] Graph updated — ingested: <path>`
+
+### ix-map.sh (Stop)
+
+After Claude finishes each response:
+1. Runs `ix map` in the background (detached, non-blocking)
+2. Keeps the full architectural graph current for the next session
## Uninstall
-Remove the `hooks` block from `~/.claude/settings.json` (or `.claude/settings.json` for
-project-scoped installs), then restart Claude Code.
+```bash
+bash ix-plugin/uninstall.sh
+# or:
+curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/uninstall.sh | bash
+```
diff --git a/ix-plugin/hooks/hooks.json b/ix-plugin/hooks/hooks.json
index ba178bf8..72b68508 100644
--- a/ix-plugin/hooks/hooks.json
+++ b/ix-plugin/hooks/hooks.json
@@ -1,5 +1,16 @@
{
"hooks": {
+ "UserPromptSubmit": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-briefing.sh",
+ "timeout": 10
+ }
+ ]
+ }
+ ],
"PreToolUse": [
{
"matcher": "Grep|Glob",
@@ -10,11 +21,31 @@
"timeout": 10
}
]
+ },
+ {
+ "matcher": "Read",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-read.sh",
+ "timeout": 8
+ }
+ ]
+ },
+ {
+ "matcher": "Bash",
+ "hooks": [
+ {
+ "type": "command",
+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-bash.sh",
+ "timeout": 10
+ }
+ ]
}
],
"PostToolUse": [
{
- "matcher": "Write|Edit|MultiEdit",
+ "matcher": "Write|Edit|MultiEdit|NotebookEdit",
"hooks": [
{
"type": "command",
@@ -24,6 +55,18 @@
}
]
}
+ ],
+ "Stop": [
+ {
+ "hooks": [
+ {
+ "type": "command",
+ "command": "${CLAUDE_PLUGIN_ROOT}/hooks/ix-map.sh",
+ "timeout": 60,
+ "async": true
+ }
+ ]
+ }
]
}
}
diff --git a/ix-plugin/hooks/ix-bash.sh b/ix-plugin/hooks/ix-bash.sh
new file mode 100755
index 00000000..2044c82e
--- /dev/null
+++ b/ix-plugin/hooks/ix-bash.sh
@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# ix-bash.sh — PreToolUse hook for Bash
+#
+# Fires before Claude runs a Bash command. Detects grep/rg search patterns and
+# front-runs them with ix text + ix locate for graph-aware, token-efficient
+# results before the raw shell command executes.
+#
+# Exit 0 + JSON stdout → injects additionalContext, Bash still runs
+# Exit 0 + no stdout → no-op, Bash runs normally
+
+set -euo pipefail
+
+INPUT=$(cat)
+COMMAND=$(echo "$INPUT" | jq -r '.tool_input.command // empty')
+[ -z "$COMMAND" ] && exit 0
+
+# Only intercept grep/rg invocations
+echo "$COMMAND" | grep -qE '^\s*(grep|rg)\s' || exit 0
+
+# Bail silently if ix is not in PATH
+command -v ix >/dev/null 2>&1 || exit 0
+
+# ── Health check with 30s TTL cache ─────────────────────────────────────────
+IX_HEALTH_CACHE="${TMPDIR:-/tmp}/ix-healthy"
+_now=$(date +%s)
+_cache_ok=0
+if [ -f "$IX_HEALTH_CACHE" ]; then
+ _cached=$(cat "$IX_HEALTH_CACHE" 2>/dev/null || echo 0)
+ (( (_now - _cached) < 30 )) && _cache_ok=1
+fi
+if [ "$_cache_ok" -eq 0 ]; then
+ ix status >/dev/null 2>&1 || exit 0
+ echo "$_now" > "$IX_HEALTH_CACHE"
+fi
+
+# ── Extract search pattern ────────────────────────────────────────────────────
+# Try double-quoted pattern first, then single-quoted, then unquoted
+PATTERN=""
+PATTERN=$(echo "$COMMAND" | sed -E 's/.*\s"([^"]+)".*/\1/' 2>/dev/null) || PATTERN=""
+if [ -z "$PATTERN" ] || [ "$PATTERN" = "$COMMAND" ]; then
+ PATTERN=$(echo "$COMMAND" | sed -E "s/.*\s'([^']+)'.*/\1/" 2>/dev/null) || PATTERN=""
+fi
+if [ -z "$PATTERN" ] || [ "$PATTERN" = "$COMMAND" ]; then
+ # Unquoted: grab first non-flag token after command name
+ PATTERN=$(echo "$COMMAND" | sed -E 's/^\s*(grep|rg)\s+(-[a-zA-Z0-9]+\s+|--[a-zA-Z-]+=\S+\s+)*([^-][^ ]*).*/\3/' 2>/dev/null) || PATTERN=""
+fi
+
+[ -z "$PATTERN" ] && exit 0
+[ ${#PATTERN} -lt 2 ] && exit 0
+
+# ── Run ix text + ix locate in parallel ─────────────────────────────────────
+_text_tmp=$(mktemp)
+_locate_tmp=$(mktemp)
+trap 'rm -f "$_text_tmp" "$_locate_tmp"' EXIT
+
+ix text "$PATTERN" --limit 20 --format json > "$_text_tmp" 2>/dev/null &
+_text_pid=$!
+ix locate "$PATTERN" --limit 10 --format json > "$_locate_tmp" 2>/dev/null &
+_locate_pid=$!
+
+wait "$_text_pid" || true
+wait "$_locate_pid" || true
+
+TEXT_RESULT=$(cat "$_text_tmp")
+LOCATE_RESULT=$(cat "$_locate_tmp")
+
+[ -z "$TEXT_RESULT" ] && [ -z "$LOCATE_RESULT" ] && exit 0
+
+CONTEXT="[ix] Pre-bash search results for pattern: '${PATTERN}'"
+[ -n "$TEXT_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix text ---\n${TEXT_RESULT}"
+[ -n "$LOCATE_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix locate ---\n${LOCATE_RESULT}"
+
+jq -n --arg ctx "$CONTEXT" '{"additionalContext": $ctx}'
+exit 0
diff --git a/ix-plugin/hooks/ix-briefing.sh b/ix-plugin/hooks/ix-briefing.sh
new file mode 100755
index 00000000..bfd1e220
--- /dev/null
+++ b/ix-plugin/hooks/ix-briefing.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# ix-briefing.sh — UserPromptSubmit hook
+#
+# Fires at the start of each user prompt. Injects a compact ix session briefing
+# once per 10 minutes — simulating "on session open" behaviour. Subsequent
+# prompts within the window are no-ops (no tokens consumed).
+#
+# Exit 0 + JSON stdout → injects additionalContext into the prompt
+# Exit 0 + no stdout → no-op
+
+set -euo pipefail
+
+BRIEFING_TTL=600 # 10 minutes
+
+# Bail silently if ix is not in PATH
+command -v ix >/dev/null 2>&1 || exit 0
+
+IX_BRIEFING_CACHE="${TMPDIR:-/tmp}/ix-briefing-cache"
+_now=$(date +%s)
+
+# If cache exists and is fresh, stay silent (briefing already injected this session)
+if [ -f "$IX_BRIEFING_CACHE" ]; then
+ _cached_time=$(head -1 "$IX_BRIEFING_CACHE" 2>/dev/null || echo 0)
+ if (( (_now - _cached_time) < BRIEFING_TTL )); then
+ exit 0
+ fi
+fi
+
+# Cache is stale — check backend health before calling ix
+ix status >/dev/null 2>&1 || exit 0
+
+BRIEFING=$(ix briefing --format json 2>/dev/null) || exit 0
+[ -z "$BRIEFING" ] && exit 0
+
+# Write timestamp + briefing to cache
+{ echo "$_now"; echo "$BRIEFING"; } > "$IX_BRIEFING_CACHE"
+
+jq -n --arg b "$BRIEFING" \
+ '{"additionalContext": ("[ix] Session briefing:\n" + $b)}'
+exit 0
diff --git a/ix-plugin/hooks/ix-ingest.sh b/ix-plugin/hooks/ix-ingest.sh
index dd070c48..92f5ad8e 100755
--- a/ix-plugin/hooks/ix-ingest.sh
+++ b/ix-plugin/hooks/ix-ingest.sh
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
-# ix-ingest.sh — PostToolUse hook for Write, Edit, MultiEdit
+# ix-ingest.sh — PostToolUse hook for Write, Edit, MultiEdit, NotebookEdit
#
# Fires after Claude modifies a file. Automatically ingests the changed file
# into the Ix graph so the next query reflects the current code state.
diff --git a/ix-plugin/hooks/ix-map.sh b/ix-plugin/hooks/ix-map.sh
new file mode 100755
index 00000000..ae23f617
--- /dev/null
+++ b/ix-plugin/hooks/ix-map.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# ix-map.sh — Stop hook
+#
+# Fires after Claude finishes each response. Runs ix map asynchronously to
+# keep the architectural graph current so the next session starts fresh.
+#
+# Runs async (does not block Claude's response or session end).
+
+set -euo pipefail
+
+# Bail silently if ix is not in PATH
+command -v ix >/dev/null 2>&1 || exit 0
+
+# Bail if backend is not reachable
+ix status >/dev/null 2>&1 || exit 0
+
+# Run ix map in background — detach from current process so it survives
+nohup ix map >/dev/null 2>&1 &
+disown
+
+exit 0
diff --git a/ix-plugin/hooks/ix-read.sh b/ix-plugin/hooks/ix-read.sh
new file mode 100755
index 00000000..1f761833
--- /dev/null
+++ b/ix-plugin/hooks/ix-read.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# ix-read.sh — PreToolUse hook for Read
+#
+# Fires before Claude reads a file. Runs ix inventory (all entities in the file)
+# and ix overview (structural summary) in parallel and injects results as
+# additionalContext so Claude has graph-aware context before reading raw source.
+#
+# Exit 0 + JSON stdout → injects additionalContext, native Read still runs
+# Exit 0 + no stdout → no-op, Read runs normally
+
+set -euo pipefail
+
+INPUT=$(cat)
+FILE_PATH=$(echo "$INPUT" | jq -r '.tool_input.file_path // empty')
+[ -z "$FILE_PATH" ] && exit 0
+
+# Skip binary/non-code files
+case "$FILE_PATH" in
+ *.png|*.jpg|*.jpeg|*.gif|*.ico|*.pdf|*.zip|*.tar|*.gz|*.bin|*.exe) exit 0 ;;
+esac
+
+# Bail silently if ix is not in PATH
+command -v ix >/dev/null 2>&1 || exit 0
+
+# ── Health check with 30s TTL cache ─────────────────────────────────────────
+IX_HEALTH_CACHE="${TMPDIR:-/tmp}/ix-healthy"
+_now=$(date +%s)
+_cache_ok=0
+if [ -f "$IX_HEALTH_CACHE" ]; then
+ _cached=$(cat "$IX_HEALTH_CACHE" 2>/dev/null || echo 0)
+ (( (_now - _cached) < 30 )) && _cache_ok=1
+fi
+if [ "$_cache_ok" -eq 0 ]; then
+ ix status >/dev/null 2>&1 || exit 0
+ echo "$_now" > "$IX_HEALTH_CACHE"
+fi
+
+# ── Run ix inventory + ix overview in parallel ───────────────────────────────
+_inv_tmp=$(mktemp)
+_ov_tmp=$(mktemp)
+trap 'rm -f "$_inv_tmp" "$_ov_tmp"' EXIT
+
+ix inventory --path "$FILE_PATH" --format json > "$_inv_tmp" 2>/dev/null &
+_inv_pid=$!
+
+# Strip extension for a cleaner symbol match
+_name=$(basename "$FILE_PATH")
+_name="${_name%.*}"
+ix overview "$_name" --format json > "$_ov_tmp" 2>/dev/null &
+_ov_pid=$!
+
+wait "$_inv_pid" || true
+wait "$_ov_pid" || true
+
+INV_RESULT=$(cat "$_inv_tmp")
+OV_RESULT=$(cat "$_ov_tmp")
+
+[ -z "$INV_RESULT" ] && [ -z "$OV_RESULT" ] && exit 0
+
+CONTEXT="[ix] Pre-read context for: ${FILE_PATH}"
+[ -n "$INV_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix inventory (entities in file) ---\n${INV_RESULT}"
+[ -n "$OV_RESULT" ] && CONTEXT="${CONTEXT}\n\n--- ix overview ---\n${OV_RESULT}"
+
+jq -n --arg ctx "$CONTEXT" '{"additionalContext": $ctx}'
+exit 0
diff --git a/ix-plugin/install.sh b/ix-plugin/install.sh
index ce7e14d8..e597828e 100755
--- a/ix-plugin/install.sh
+++ b/ix-plugin/install.sh
@@ -2,13 +2,13 @@
# ix-plugin/install.sh — Standalone Claude Code plugin installer for Ix Memory
#
# Usage (curl):
-# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/install.sh | bash
+# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/install.sh | bash
#
# Usage (local, from the Ix repo):
# bash ix-plugin/install.sh
#
# What it does:
-# 1. Downloads ix-intercept.sh and ix-ingest.sh to ~/.local/share/ix/plugin/hooks/
+# 1. Copies/downloads all hook scripts to ~/.local/share/ix/plugin/hooks/
# 2. Wires them into ~/.claude/settings.json
#
# Prerequisites: ix CLI must already be installed and in PATH.
@@ -17,10 +17,19 @@ set -euo pipefail
# ── Config ────────────────────────────────────────────────────────────────────
-GITHUB_RAW="https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/hooks"
+GITHUB_RAW="https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/hooks"
INSTALL_DIR="${IX_PLUGIN_DIR:-$HOME/.local/share/ix/plugin/hooks}"
SETTINGS="$HOME/.claude/settings.json"
+HOOKS=(
+ "ix-briefing.sh"
+ "ix-intercept.sh"
+ "ix-read.sh"
+ "ix-bash.sh"
+ "ix-ingest.sh"
+ "ix-map.sh"
+)
+
# ── Helpers ───────────────────────────────────────────────────────────────────
info() { echo " [ok] $*"; }
@@ -43,13 +52,13 @@ info "curl, jq"
if ! command -v ix >/dev/null 2>&1; then
warn "'ix' not found in PATH — hooks will be installed but won't activate until ix is installed."
- warn "Install ix first: see https://github.com/ix-infrastructure/Ix"
+ warn "Install ix first: see https://github.com/ix-infrastructure/IX-Memory"
fi
-# ── Download hooks ────────────────────────────────────────────────────────────
+# ── Download / copy hooks ─────────────────────────────────────────────────────
echo ""
-echo "── Downloading hooks ──"
+echo "── Installing hooks ──"
mkdir -p "$INSTALL_DIR"
@@ -57,39 +66,37 @@ mkdir -p "$INSTALL_DIR"
_repo_hooks=""
if [ -n "${BASH_SOURCE[0]:-}" ] && [ "${BASH_SOURCE[0]}" != "bash" ]; then
_script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)"
- # Hooks live in hooks/ subdir relative to this script
if [ -f "$_script_dir/hooks/ix-intercept.sh" ]; then
_repo_hooks="$_script_dir/hooks"
fi
fi
-if [ -n "$_repo_hooks" ]; then
- # Local install — copy from repo
- cp "$_repo_hooks/ix-intercept.sh" "$INSTALL_DIR/ix-intercept.sh"
- cp "$_repo_hooks/ix-ingest.sh" "$INSTALL_DIR/ix-ingest.sh"
- info "Copied hooks from local repo → $INSTALL_DIR"
-else
- # Remote install — download from GitHub
- curl -fsSL "$GITHUB_RAW/ix-intercept.sh" -o "$INSTALL_DIR/ix-intercept.sh"
- curl -fsSL "$GITHUB_RAW/ix-ingest.sh" -o "$INSTALL_DIR/ix-ingest.sh"
- info "Downloaded hooks → $INSTALL_DIR"
-fi
-
-chmod +x "$INSTALL_DIR/ix-intercept.sh" "$INSTALL_DIR/ix-ingest.sh"
-info "Permissions set"
+for hook in "${HOOKS[@]}"; do
+ if [ -n "$_repo_hooks" ]; then
+ cp "$_repo_hooks/$hook" "$INSTALL_DIR/$hook"
+ else
+ curl -fsSL "$GITHUB_RAW/$hook" -o "$INSTALL_DIR/$hook"
+ fi
+ chmod +x "$INSTALL_DIR/$hook"
+ info "$hook → $INSTALL_DIR"
+done
# ── Wire Claude Code settings ─────────────────────────────────────────────────
echo ""
echo "── Configuring Claude Code ──"
+BRIEFING="$INSTALL_DIR/ix-briefing.sh"
INTERCEPT="$INSTALL_DIR/ix-intercept.sh"
+READ_HOOK="$INSTALL_DIR/ix-read.sh"
+BASH_HOOK="$INSTALL_DIR/ix-bash.sh"
INGEST="$INSTALL_DIR/ix-ingest.sh"
+MAP_HOOK="$INSTALL_DIR/ix-map.sh"
mkdir -p "$HOME/.claude"
[ -f "$SETTINGS" ] || echo "{}" > "$SETTINGS"
-# Idempotent: skip if already wired
+# Idempotent: skip if already wired (check for intercept hook as sentinel)
already=$(jq --arg cmd "$INTERCEPT" \
'[.hooks?.PreToolUse[]?.hooks[]?.command? // empty] | map(select(. == $cmd)) | length' \
"$SETTINGS" 2>/dev/null || echo "0")
@@ -98,20 +105,54 @@ if [ "$already" -gt 0 ]; then
info "Hooks already registered in ~/.claude/settings.json — skipping"
else
tmp=$(mktemp)
- jq --arg intercept "$INTERCEPT" --arg ingest "$INGEST" '
+ jq \
+ --arg briefing "$BRIEFING" \
+ --arg intercept "$INTERCEPT" \
+ --arg read_hook "$READ_HOOK" \
+ --arg bash_hook "$BASH_HOOK" \
+ --arg ingest "$INGEST" \
+ --arg map_hook "$MAP_HOOK" '
.hooks |= (. // {}) |
+
+ .hooks.UserPromptSubmit |= (. // []) |
+ .hooks.UserPromptSubmit += [
+ {
+ "hooks": [{ "type": "command", "command": $briefing, "timeout": 10 }]
+ }
+ ] |
+
.hooks.PreToolUse |= (. // []) |
- .hooks.PreToolUse += [{
- "matcher": "Grep|Glob",
- "hooks": [{ "type": "command", "command": $intercept, "timeout": 10 }]
- }] |
+ .hooks.PreToolUse += [
+ {
+ "matcher": "Grep|Glob",
+ "hooks": [{ "type": "command", "command": $intercept, "timeout": 10 }]
+ },
+ {
+ "matcher": "Read",
+ "hooks": [{ "type": "command", "command": $read_hook, "timeout": 8 }]
+ },
+ {
+ "matcher": "Bash",
+ "hooks": [{ "type": "command", "command": $bash_hook, "timeout": 10 }]
+ }
+ ] |
+
.hooks.PostToolUse |= (. // []) |
- .hooks.PostToolUse += [{
- "matcher": "Write|Edit|MultiEdit",
- "hooks": [{ "type": "command", "command": $ingest, "timeout": 30 }]
- }]
+ .hooks.PostToolUse += [
+ {
+ "matcher": "Write|Edit|MultiEdit|NotebookEdit",
+ "hooks": [{ "type": "command", "command": $ingest, "timeout": 30, "async": true }]
+ }
+ ] |
+
+ .hooks.Stop |= (. // []) |
+ .hooks.Stop += [
+ {
+ "hooks": [{ "type": "command", "command": $map_hook, "timeout": 60, "async": true }]
+ }
+ ]
' "$SETTINGS" > "$tmp" && mv "$tmp" "$SETTINGS"
- info "Registered hooks → ~/.claude/settings.json"
+ info "Registered all hooks → ~/.claude/settings.json"
fi
# ── Done ──────────────────────────────────────────────────────────────────────
@@ -123,8 +164,18 @@ echo "╚═══════════════════════
echo ""
echo " Hooks installed to: $INSTALL_DIR"
echo ""
+echo " Active hooks:"
+echo " UserPromptSubmit → ix briefing (session context on every prompt)"
+echo " PreToolUse Grep|Glob → ix text + ix locate (graph-aware search)"
+echo " PreToolUse Read → ix inventory + ix overview (file context)"
+echo " PreToolUse Bash (grep/rg) → ix text + ix locate (shell search intercept)"
+echo " PostToolUse Write|Edit|... → ix ingest (auto-update graph after edits)"
+echo " Stop → ix map (full graph refresh after session)"
+echo ""
echo " Restart Claude Code to activate."
echo ""
echo " To uninstall:"
-echo " curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/uninstall.sh | bash"
+echo " bash ix-plugin/uninstall.sh"
+echo " # or:"
+echo " curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/IX-Memory/main/ix-plugin/uninstall.sh | bash"
echo ""
diff --git a/ix-plugin/uninstall.sh b/ix-plugin/uninstall.sh
index 2c19d5f9..c4decb4a 100755
--- a/ix-plugin/uninstall.sh
+++ b/ix-plugin/uninstall.sh
@@ -1,11 +1,5 @@
#!/usr/bin/env bash
# ix-plugin/uninstall.sh — Remove the Ix Memory Claude Code plugin
-#
-# Usage (curl):
-# curl -fsSL https://raw.githubusercontent.com/ix-infrastructure/Ix/main/ix-plugin/uninstall.sh | bash
-#
-# Usage (local):
-# bash ix-plugin/uninstall.sh
set -euo pipefail
@@ -19,7 +13,13 @@ echo "── Removing Ix Claude Code plugin ──"
# Remove hook files
if [ -d "$INSTALL_DIR" ]; then
- rm -f "$INSTALL_DIR/ix-intercept.sh" "$INSTALL_DIR/ix-ingest.sh"
+ rm -f \
+ "$INSTALL_DIR/ix-briefing.sh" \
+ "$INSTALL_DIR/ix-intercept.sh" \
+ "$INSTALL_DIR/ix-read.sh" \
+ "$INSTALL_DIR/ix-bash.sh" \
+ "$INSTALL_DIR/ix-ingest.sh" \
+ "$INSTALL_DIR/ix-map.sh"
rmdir "$INSTALL_DIR" 2>/dev/null || true
info "Removed $INSTALL_DIR"
else
@@ -28,25 +28,31 @@ fi
# Remove hook entries from ~/.claude/settings.json
if [ -f "$SETTINGS" ] && command -v jq >/dev/null 2>&1; then
+ BRIEFING="$INSTALL_DIR/ix-briefing.sh"
INTERCEPT="$INSTALL_DIR/ix-intercept.sh"
+ READ_HOOK="$INSTALL_DIR/ix-read.sh"
+ BASH_HOOK="$INSTALL_DIR/ix-bash.sh"
INGEST="$INSTALL_DIR/ix-ingest.sh"
+ MAP_HOOK="$INSTALL_DIR/ix-map.sh"
+
+ _ix_cmds=$(jq -n \
+ --arg a "$BRIEFING" --arg b "$INTERCEPT" \
+ --arg c "$READ_HOOK" --arg d "$BASH_HOOK" \
+ --arg e "$INGEST" --arg f "$MAP_HOOK" \
+ '[$a,$b,$c,$d,$e,$f]')
tmp=$(mktemp)
- jq --arg intercept "$INTERCEPT" --arg ingest "$INGEST" '
- if .hooks.PreToolUse then
- .hooks.PreToolUse |= map(select(
- (.hooks // []) | map(.command) | any(. == $intercept) | not
- ))
- else . end |
- if .hooks.PostToolUse then
- .hooks.PostToolUse |= map(select(
- (.hooks // []) | map(.command) | any(. == $ingest) | not
- ))
- else . end
+ jq --argjson ix_cmds "$_ix_cmds" '
+ def is_ix_hook: (.hooks // []) | map(.command) | any(. as $c | $ix_cmds | any(. == $c));
+
+ if .hooks.UserPromptSubmit then .hooks.UserPromptSubmit |= map(select(is_ix_hook | not)) else . end |
+ if .hooks.PreToolUse then .hooks.PreToolUse |= map(select(is_ix_hook | not)) else . end |
+ if .hooks.PostToolUse then .hooks.PostToolUse |= map(select(is_ix_hook | not)) else . end |
+ if .hooks.Stop then .hooks.Stop |= map(select(is_ix_hook | not)) else . end
' "$SETTINGS" > "$tmp" && mv "$tmp" "$SETTINGS"
info "Removed hooks from ~/.claude/settings.json"
else
- echo " (settings.json not found — nothing to remove)"
+ echo " (settings.json not found or jq unavailable — nothing to remove)"
fi
echo ""
diff --git a/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala
index 3e1812e2..b20a1981 100644
--- a/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/BulkPatchCommitRoutes.scala
@@ -16,8 +16,6 @@ class BulkPatchCommitRoutes(bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi)
private case class BulkPatchRequest(patches: List[GraphPatch])
private implicit val reqDecoder: Decoder[BulkPatchRequest] = deriveDecoder
- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
-
val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
case req @ POST -> Root / "v1" / "patches" / "bulk" =>
(for {
@@ -43,13 +41,12 @@ class BulkPatchCommitRoutes(bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi)
}
private def buildProvenanceMap(patch: GraphPatch): java.util.Map[String, AnyRef] = {
- val json = Json.obj(
- "source_uri" -> Json.fromString(patch.source.uri),
- "source_hash" -> patch.source.sourceHash.fold(Json.Null)(Json.fromString),
- "extractor" -> Json.fromString(patch.source.extractor),
- "source_type" -> Json.fromString(patch.source.sourceType.asJson.asString.getOrElse("code")),
- "observed_at" -> Json.fromString(patch.timestamp.toString)
- )
- mapper.readValue(json.noSpaces, classOf[java.util.Map[String, AnyRef]])
+ val result = new java.util.HashMap[String, AnyRef](5)
+ result.put("source_uri", patch.source.uri)
+ result.put("source_hash", patch.source.sourceHash.orNull)
+ result.put("extractor", patch.source.extractor)
+ result.put("source_type", patch.source.sourceType.asJson.asString.getOrElse("code"))
+ result.put("observed_at", patch.timestamp.toString)
+ result
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala
index d721f84e..fb0daee1 100644
--- a/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala
@@ -3,7 +3,7 @@ package ix.memory.api
import io.circe.Json
import io.circe.syntax._
-import ix.memory.map.{ArchitectureMap, Region, RegionMatch, ScopedSubsystemView}
+import ix.memory.map.{ArchitectureMap, MapPreflightResult, PersistenceEstimate, Region, RegionMatch, ScopedSubsystemView}
import ix.memory.model.NodeId
/** Shared JSON encoding for ArchitectureMap, used by MapRoutes and SubsystemRoutes. */
@@ -11,14 +11,26 @@ object MapEncoder {
def encodeMap(m: ArchitectureMap): Json = {
val regionsByLevel = m.regions.groupBy(_.level).toVector.sortBy(_._1)
- Json.obj(
+ val base = Json.obj(
"file_count" -> m.fileCount.asJson,
"region_count" -> m.regions.size.asJson,
"levels" -> regionsByLevel.size.asJson,
"map_rev" -> m.mapRev.asJson,
+ "outcome" -> m.outcome.label.asJson,
"regions" -> m.regions.sortBy(r => (r.level, r.label)).map(encodeRegion).asJson,
"hierarchy" -> buildHierarchyTree(m).asJson
)
+
+ val withPreflight =
+ m.preflight match {
+ case Some(value) => base.deepMerge(Json.obj("preflight" -> encodePreflight(value)))
+ case None => base
+ }
+
+ m.persistenceEstimate match {
+ case Some(value) => withPreflight.deepMerge(Json.obj("persistence" -> encodePersistence(value)))
+ case None => withPreflight
+ }
}
def encodeRegion(r: Region): Json =
@@ -128,4 +140,35 @@ object MapEncoder {
renderNode(view.target, Set.empty)
}
+
+ private def encodePreflight(preflight: MapPreflightResult): Json =
+ Json.obj(
+ "cost" -> Json.obj(
+ "file_count" -> preflight.cost.fileCount.asJson,
+ "directory_count" -> preflight.cost.directoryCount.asJson,
+ "directory_quadratic" -> preflight.cost.directoryQuadratic.asJson,
+ "symbol_estimate" -> preflight.cost.symbolEstimate.asJson,
+ "edge_estimate" -> preflight.cost.edgeEstimate.asJson
+ ),
+ "capacity" -> Json.obj(
+ "cpu_cores" -> preflight.capacity.cpuCores.asJson,
+ "heap_max_bytes" -> preflight.capacity.heapMaxBytes.asJson,
+ "heap_free_bytes" -> preflight.capacity.heapFreeBytes.asJson,
+ "container_memory" -> preflight.capacity.containerMemory.asJson,
+ "disk_free_bytes" -> preflight.capacity.diskFreeBytes.asJson
+ ),
+ "risk" -> preflight.risk.label.asJson,
+ "mode" -> preflight.mode.label.asJson,
+ "warnings" -> preflight.warnings.asJson,
+ "duration_ms" -> preflight.durationMs.asJson
+ )
+
+ private def encodePersistence(persistence: PersistenceEstimate): Json =
+ Json.obj(
+ "region_nodes" -> persistence.regionNodes.asJson,
+ "file_edges" -> persistence.fileEdges.asJson,
+ "region_edges" -> persistence.regionEdges.asJson,
+ "delete_ops" -> persistence.deleteOps.asJson,
+ "total_ops" -> persistence.totalOps.asJson
+ )
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
index 3189be0d..1acadb4e 100644
--- a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
@@ -16,8 +16,8 @@ class MapRoutes(mapService: MapService) {
case req @ POST -> Root / "v1" / "map" =>
(for {
body <- req.as[Json].handleError(_ => Json.obj())
- force = body.hcursor.get[Boolean]("full").toOption.getOrElse(false)
- archMap <- mapService.buildMap(forceRecompute = force)
+ forceFull = body.hcursor.get[Boolean]("full").toOption.getOrElse(false)
+ archMap <- mapService.buildMap(forceRecompute = forceFull, forceFull = forceFull)
resp <- Ok(MapEncoder.encodeMap(archMap))
} yield resp).handleErrorWith(ErrorHandler.handle(_))
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
index 19d7e7e9..def177bb 100644
--- a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
@@ -13,8 +13,6 @@ import ix.memory.model.{GraphPatch, Rev}
class PatchCommitRoutes(writeApi: GraphWriteApi, bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi) {
- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
-
val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
case req @ POST -> Root / "v1" / "patch" =>
(for {
@@ -52,13 +50,12 @@ class PatchCommitRoutes(writeApi: GraphWriteApi, bulkWriteApi: BulkWriteApi, que
}
private def buildProvenanceMap(patch: GraphPatch): java.util.Map[String, AnyRef] = {
- val json = Json.obj(
- "source_uri" -> Json.fromString(patch.source.uri),
- "source_hash" -> patch.source.sourceHash.fold(Json.Null)(Json.fromString),
- "extractor" -> Json.fromString(patch.source.extractor),
- "source_type" -> Json.fromString(patch.source.sourceType.asJson.asString.getOrElse("code")),
- "observed_at" -> Json.fromString(patch.timestamp.toString)
- )
- mapper.readValue(json.noSpaces, classOf[java.util.Map[String, AnyRef]])
+ val result = new java.util.HashMap[String, AnyRef](5)
+ result.put("source_uri", patch.source.uri)
+ result.put("source_hash", patch.source.sourceHash.orNull)
+ result.put("extractor", patch.source.extractor)
+ result.put("source_type", patch.source.sourceType.asJson.asString.getOrElse("code"))
+ result.put("observed_at", patch.timestamp.toString)
+ result
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
index 7ddc575e..a9f38cd4 100644
--- a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
@@ -1,7 +1,8 @@
package ix.memory.db
+import scala.concurrent.duration._
+
import cats.effect.{IO, Resource}
-import cats.syntax.traverse._
import com.arangodb.{ArangoDB, ArangoDatabase}
import com.arangodb.entity.MultiDocumentEntity
import com.arangodb.model.{AqlQueryOptions, DocumentCreateOptions, OverwriteMode, StreamTransactionOptions}
@@ -37,7 +38,14 @@ class ArangoClient private (db: ArangoDatabase) {
IO.blocking {
val cursor = db.query(aql, classOf[AnyRef], asJava(bindVars), queryOptions(txId))
cursor.asListRemaining().asScala.toList
- }.flatMap(_.traverse(toJson))
+ }.flatMap { values =>
+ values.foldLeft(IO.pure(List.empty[Json])) { (accIO, value) =>
+ for {
+ acc <- accIO
+ json <- toJson(value)
+ } yield acc :+ json
+ }
+ }
def queryOne(
aql: String,
@@ -56,6 +64,15 @@ class ArangoClient private (db: ArangoDatabase) {
()
}
+ def queryLong(
+ aql: String,
+ bindVars: Map[String, AnyRef] = Map.empty,
+ txId: Option[String] = None
+ ): IO[Long] =
+ queryOne(aql, bindVars, txId).map {
+ _.flatMap(_.asNumber.flatMap(_.toLong)).getOrElse(0L)
+ }
+
/** Bulk insert/replace documents into a collection. Uses Document API (not AQL). */
def bulkInsert(
collection: String,
@@ -123,8 +140,14 @@ class ArangoClient private (db: ArangoDatabase) {
/**
* Delete only code-derived nodes and edges, preserving planning artifacts
* (goals/intents, plans, tasks, decisions, bugs and their linking edges).
+ *
+ * Uses larger batched REMOVE passes and avoids materializing `RETURN 1`
+ * result sets for every deleted document. We first count the matching docs,
+ * then execute enough bounded delete passes to cover that cardinality, with
+ * a final recount only if needed. Edges are removed before nodes.
*/
def truncateCodeGraph(): IO[Unit] = {
+ val batchSize = 5000
val codeKinds = Array(
"file", "function", "class", "method", "interface",
"module", "chunk", "region"
@@ -133,19 +156,73 @@ class ArangoClient private (db: ArangoDatabase) {
"CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS",
"CONTAINS", "DEFINES", "CONTAINS_CHUNK", "REFERENCES", "NEXT", "IN_REGION"
)
+
+ def removeAll(
+ countAql: String,
+ removeAql: String,
+ bindVars: Map[String, AnyRef]
+ ): IO[Unit] = {
+ val varsWithBatch = bindVars + ("batchSize" -> Int.box(batchSize))
+
+ def runPasses(remaining: Long): IO[Unit] = {
+ val passes = math.ceil(remaining.toDouble / batchSize.toDouble).toInt
+ (0 until passes).foldLeft(IO.unit) { (acc, _) =>
+ acc *> execute(removeAql, varsWithBatch)
+ }
+ }
+
+ def loop: IO[Unit] =
+ queryLong(countAql, bindVars).flatMap { remaining =>
+ if (remaining <= 0) IO.unit
+ else runPasses(remaining) *> queryLong(countAql, bindVars).flatMap { tail =>
+ if (tail <= 0) IO.unit else loop
+ }
+ }
+
+ loop
+ }
+
for {
- _ <- execute(
- "FOR n IN nodes FILTER n.kind IN @kinds REMOVE n IN nodes",
+ // Edges first — avoids locking nodes that are about to be removed
+ _ <- removeAll(
+ "RETURN LENGTH(FOR e IN edges FILTER e.predicate IN @predicates RETURN 1)",
+ "FOR e IN edges FILTER e.predicate IN @predicates LIMIT @batchSize REMOVE e IN edges",
+ Map("predicates" -> codePredicates.asInstanceOf[AnyRef])
+ )
+ // Claims next — code ingests append versioned/retracted claims forever, so
+ // reset-code must clear them before removing the corresponding nodes.
+ _ <- removeAll(
+ """RETURN LENGTH(
+ | FOR c IN claims
+ | FILTER LENGTH(
+ | FOR n IN nodes
+ | FILTER n.logical_id == c.entity_id
+ | AND n.kind IN @kinds
+ | LIMIT 1
+ | RETURN 1
+ | ) > 0
+ | RETURN 1
+ |)""".stripMargin,
+ """FOR c IN claims
+ | FILTER LENGTH(
+ | FOR n IN nodes
+ | FILTER n.logical_id == c.entity_id
+ | AND n.kind IN @kinds
+ | LIMIT 1
+ | RETURN 1
+ | ) > 0
+ | LIMIT @batchSize
+ | REMOVE c IN claims""".stripMargin,
Map("kinds" -> codeKinds.asInstanceOf[AnyRef])
)
- _ <- execute(
- "FOR e IN edges FILTER e.predicate IN @predicates REMOVE e IN edges",
- Map("predicates" -> codePredicates.asInstanceOf[AnyRef])
+ _ <- removeAll(
+ "RETURN LENGTH(FOR n IN nodes FILTER n.kind IN @kinds RETURN 1)",
+ "FOR n IN nodes FILTER n.kind IN @kinds LIMIT @batchSize REMOVE n IN nodes",
+ Map("kinds" -> codeKinds.asInstanceOf[AnyRef])
)
- // Also remove code-ingest patches so getSourceHashes returns empty after reset,
- // preventing the ingest client from treating all files as unchanged.
- _ <- execute(
- "FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') REMOVE p IN patches",
+ _ <- removeAll(
+ "RETURN LENGTH(FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') RETURN 1)",
+ "FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') LIMIT @batchSize REMOVE p IN patches",
Map.empty
)
} yield ()
@@ -166,6 +243,7 @@ object ArangoClient {
.host(host, port)
.user(user)
.password(if (password.isEmpty) null else password)
+ .maxConnections(8)
.build()
val existingDbs = arango.getDatabases.asScala.toSet
if (!existingDbs.contains(database)) {
diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala
index bec90b28..d4319f2c 100644
--- a/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoSchema.scala
@@ -33,6 +33,13 @@ object ArangoSchema {
java.util.Arrays.asList("logical_id"),
new PersistentIndexOptions()
)
+ // Compound index for tombstone query: FILTER logical_id IN @ids AND deleted_rev == null
+ // Without this, the single-field logical_id index returns all historical versions of each
+ // node and applies deleted_rev as a post-filter — gets slower as the collection grows.
+ nodes.ensurePersistentIndex(
+ java.util.Arrays.asList("logical_id", "deleted_rev"),
+ new PersistentIndexOptions()
+ )
// Indexes — edges
val edges = db.collection("edges")
@@ -61,6 +68,12 @@ object ArangoSchema {
java.util.Arrays.asList("entity_id"),
new PersistentIndexOptions()
)
+ // Compound index for retire query: FILTER entity_id IN @ids AND deleted_rev == null
+ // Mirrors the nodes compound index — avoids post-filtering all historical claim versions.
+ claims.ensurePersistentIndex(
+ java.util.Arrays.asList("entity_id", "deleted_rev"),
+ new PersistentIndexOptions()
+ )
claims.ensurePersistentIndex(
java.util.Arrays.asList("status"),
new PersistentIndexOptions()
diff --git a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
index d0c51ae0..3c92a0c9 100644
--- a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
@@ -8,6 +8,8 @@ import cats.effect.IO
import cats.syntax.traverse._
import io.circe.Json
import io.circe.syntax._
+import org.typelevel.log4cats.Logger
+import org.typelevel.log4cats.slf4j.Slf4jLogger
import scala.concurrent.duration._
@@ -18,8 +20,18 @@ import ix.memory.model._
* instead of individual AQL queries.
*/
class BulkWriteApi(client: ArangoClient) {
+ private type BulkDoc = java.util.Map[String, AnyRef]
private val CurrentRevisionKey = "current"
private val SourceGraphRevisionKey = "source-graph"
+ private val DefaultChunkSize = 100
+ private val DefaultChunkPayloadBytes = 4 * 1024 * 1024
+ private val MaxNodeDocs = 10000
+ private val MaxEdgeDocs = 20000
+ private val MaxClaimDocs = 10000
+ private val MaxPatchDocs = 250
+ private val SlowChunkMs = 2000L
+ private val debugBulkWrites = sys.env.get("IX_DEBUG").exists(v => v.nonEmpty && v != "0" && !v.equalsIgnoreCase("false"))
+ private val logger: Logger[IO] = Slf4jLogger.getLoggerFromName[IO]("ix.bulk-write")
private def withRetry[A](action: IO[A], maxAttempts: Int = 3, delay: FiniteDuration = 500.millis): IO[A] = {
action.handleErrorWith { err =>
@@ -28,30 +40,82 @@ class BulkWriteApi(client: ArangoClient) {
}
}
+ private def chunkDocs[T](docs: Seq[T], maxDocs: Int): Seq[Seq[T]] =
+ docs.grouped(maxDocs).toVector
+
+ private def insertCollectionChunked(
+ collection: String,
+ docs: Seq[BulkDoc],
+ maxDocs: Int
+ )(insert: Seq[BulkDoc] => IO[Int]): IO[Long] =
+ if (docs.isEmpty) IO.pure(0L)
+ else {
+ val batches = chunkDocs(docs, maxDocs)
+ batches.foldLeft(IO.pure(0L)) { case (accIO, batch) =>
+ for {
+ acc <- accIO
+ ms <- timedUnit(withRetry(insert(batch)))
+ _ <- if (debugBulkWrites)
+ logger.info(s"[bulk-write] collection=$collection docs=${batch.size} insertMs=$ms")
+ else IO.unit
+ } yield acc + ms
+ }
+ }
+
def commitBatch(
fileBatches: Vector[FileBatch],
- baseRev: Long
+ baseRev: Long,
+ skipTombstone: Boolean = false
): IO[CommitResult] = {
val newRev = baseRev + 1L
- val allNodes = fileBatches.flatMap(_.nodeDocuments(newRev))
- val allEdges = fileBatches.flatMap(_.edgeDocuments(newRev))
- val allClaims = fileBatches.flatMap(_.claimDocuments(newRev))
- val allPatches = fileBatches.map(_.patchDocument(newRev))
-
- // Collect all logical_ids being upserted for tombstoning
- val logicalIds = allNodes.flatMap { doc =>
- Option(doc.get("logical_id")).map(_.toString)
- }
-
for {
- _ <- tombstoneExistingNodes(logicalIds, newRev)
- _ <- retireOldClaims(fileBatches, newRev)
- _ <- withRetry(client.bulkInsert("nodes", allNodes))
- _ <- withRetry(client.bulkInsertEdges("edges", allEdges))
- _ <- withRetry(client.bulkInsert("claims", allClaims))
- _ <- withRetry(client.bulkInsert("patches", allPatches))
- _ <- updateRevision(newRev)
+ // Build all document maps in a single IO.blocking pass — one traversal instead of four.
+ docsAndBuildMs <- timed {
+ IO.blocking {
+ val nodesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]
+ val edgesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]
+ val claimsBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]
+ val patchesBuf = Vector.newBuilder[java.util.Map[String, AnyRef]]
+ fileBatches.foreach { fb =>
+ nodesBuf ++= fb.nodeDocuments(newRev)
+ edgesBuf ++= fb.edgeDocuments(newRev)
+ claimsBuf ++= fb.claimDocuments(newRev)
+ patchesBuf += fb.patchDocument(newRev)
+ }
+ (nodesBuf.result(), edgesBuf.result(), claimsBuf.result(), patchesBuf.result())
+ }
+ }
+ (docs, buildDocsMs) = docsAndBuildMs
+ (allNodes, allEdges, allClaims, allPatches) = docs
+ logicalIds = allNodes.flatMap(doc => Option(doc.get("logical_id")).map(_.toString))
+ chunkSummary = s"files=${fileBatches.size} nodes=${allNodes.size} edges=${allEdges.size} claims=${allClaims.size} patches=${allPatches.size}"
+ // Skip tombstone/retire on first ingest — DB is empty, no existing docs to update.
+ tombstoneAndRetireMs <-
+ if (skipTombstone || baseRev == 0) IO.pure((0L, 0L))
+ else timedUnit(tombstoneExistingNodes(logicalIds, newRev))
+ .both(timedUnit(retireOldClaims(fileBatches, newRev)))
+ (tombstoneMs: Long, retireClaimsMs: Long) = tombstoneAndRetireMs
+ insertNodesMs <- insertCollectionChunked("nodes", allNodes, MaxNodeDocs)(docs => client.bulkInsert("nodes", docs))
+ insertEdgesMs <- insertCollectionChunked("edges", allEdges, MaxEdgeDocs)(docs => client.bulkInsertEdges("edges", docs))
+ insertClaimsMs <- insertCollectionChunked("claims", allClaims, MaxClaimDocs)(docs => client.bulkInsert("claims", docs))
+ insertPatchesMs <- insertCollectionChunked("patches", allPatches, MaxPatchDocs)(docs => client.bulkInsert("patches", docs))
+ updateRevMs <- timedUnit(updateRevision(newRev))
+ totalMs = buildDocsMs + tombstoneMs + retireClaimsMs + insertNodesMs + insertEdgesMs + insertClaimsMs + insertPatchesMs + updateRevMs
+ _ <- logChunkTiming(
+ s"rev=$newRev $chunkSummary",
+ totalMs,
+ List(
+ "buildDocs" -> buildDocsMs,
+ "tombstoneNodes" -> tombstoneMs,
+ "retireClaims" -> retireClaimsMs,
+ "insertNodes" -> insertNodesMs,
+ "insertEdges" -> insertEdgesMs,
+ "insertClaims" -> insertClaimsMs,
+ "insertPatches" -> insertPatchesMs,
+ "updateRevision" -> updateRevMs,
+ )
+ )
} yield CommitResult(Rev(newRev), CommitStatus.Ok)
}
@@ -74,10 +138,7 @@ class BulkWriteApi(client: ArangoClient) {
}
}
- private def retireOldClaims(fileBatches: Vector[FileBatch], newRev: Long): IO[Unit] = {
- val entityIds = fileBatches.flatMap(_.patch.ops.collect {
- case PatchOp.UpsertNode(id, _, _, _) => id.value.toString
- }).distinct
+ private def retireOldClaimsByEntityIds(entityIds: Seq[String], newRev: Long): IO[Unit] =
if (entityIds.isEmpty) IO.unit
else {
val idList = new java.util.ArrayList[String](entityIds.size)
@@ -93,41 +154,124 @@ class BulkWriteApi(client: ArangoClient) {
)
)
}
- }
+
+ private def retireOldClaims(fileBatches: Vector[FileBatch], newRev: Long): IO[Unit] =
+ retireOldClaimsByEntityIds(
+ fileBatches.flatMap(_.patch.ops.collect {
+ case PatchOp.UpsertNode(id, _, _, _) => id.value.toString
+ }).distinct,
+ newRev
+ )
/**
* Commit file batches in chunks. Each chunk gets its own revision.
* If a chunk fails, prior chunks are preserved (partial progress).
+ *
+ * When multiple chunks exist and baseRev > 0 (incremental update), tombstone+retire
+ * runs once upfront for all files in parallel rather than once per chunk sequentially.
+ * This reduces 2*N AQL UPDATE scans to 2 — e.g. 160 → 2 for Kubernetes.
*/
def commitBatchChunked(
fileBatches: Vector[FileBatch],
baseRev: Long,
- chunkSize: Int = 100
+ chunkSize: Int = DefaultChunkSize,
+ maxPayloadBytes: Int = DefaultChunkPayloadBytes
): IO[CommitResult] = {
if (fileBatches.isEmpty)
return IO.pure(CommitResult(Rev(baseRev), CommitStatus.Ok))
- val chunks = fileBatches.grouped(chunkSize).toVector
+ val chunks = chunkForCommit(fileBatches, chunkSize, maxPayloadBytes)
+ val multiChunk = chunks.size > 1 && baseRev > 0
- chunks.foldLeft(IO.pure(baseRev)) { case (revIO, chunk) =>
- revIO.flatMap { currentRev =>
- commitBatch(chunk, currentRev).map(_.newRev.value)
+ // Pre-tombstone: collect all entity IDs upfront and retire/tombstone in one parallel pass.
+ // Uses baseRev+1 as the tombstone rev, which is the same rev chunk 1 will use for its inserts.
+ val preTombstoneIO: IO[Unit] =
+ if (!multiChunk) IO.unit
+ else {
+ val allIds = fileBatches.flatMap(_.patch.ops.collect {
+ case PatchOp.UpsertNode(id, _, _, _) => id.value.toString
+ }).distinct
+ val preTombRev = baseRev + 1L
+ timedUnit(tombstoneExistingNodes(allIds.toVector, preTombRev))
+ .both(timedUnit(retireOldClaimsByEntityIds(allIds, preTombRev)))
+ .flatMap { case (tombMs, retireMs) =>
+ if (debugBulkWrites)
+ logger.info(s"[bulk-write] pre-tombstone files=${fileBatches.size} tombstoneMs=$tombMs retireMs=$retireMs")
+ else IO.unit
+ }
}
- }.map(finalRev => CommitResult(Rev(finalRev), CommitStatus.Ok))
+
+ for {
+ _ <- preTombstoneIO
+ finalRev <- chunks.foldLeft(IO.pure(baseRev)) { case (revIO, chunk) =>
+ revIO.flatMap { currentRev =>
+ commitBatch(chunk, currentRev, skipTombstone = multiChunk).map(_.newRev.value)
+ }
+ }
+ } yield CommitResult(Rev(finalRev), CommitStatus.Ok)
}
- private def updateRevision(newRev: Long): IO[Unit] =
- client.execute(
- """FOR key IN @keys
- | UPSERT { _key: key }
- | INSERT { _key: key, rev: @rev }
- | UPDATE { rev: @rev }
- | IN revisions""".stripMargin,
- Map(
- "keys" -> Array(CurrentRevisionKey, SourceGraphRevisionKey).asInstanceOf[AnyRef],
- "rev" -> Long.box(newRev).asInstanceOf[AnyRef]
- )
- )
+ private def updateRevision(newRev: Long): IO[Unit] = {
+ // Use Document API (replace mode) instead of AQL UPSERT.
+ // AQL UPSERT produces write-write conflicts when multiple concurrent HTTP
+ // requests all try to update the same "current" revision key simultaneously.
+ // The Document API replace is atomic at the document level — no conflict possible.
+ val doc1 = new java.util.HashMap[String, AnyRef]()
+ doc1.put("_key", CurrentRevisionKey)
+ doc1.put("rev", Long.box(newRev))
+ val doc2 = new java.util.HashMap[String, AnyRef]()
+ doc2.put("_key", SourceGraphRevisionKey)
+ doc2.put("rev", Long.box(newRev))
+ client.bulkInsert("revisions", Seq(doc1, doc2)).void
+ }
+
+ private def chunkForCommit(
+ fileBatches: Vector[FileBatch],
+ maxFiles: Int,
+ maxPayloadBytes: Int
+ ): Vector[Vector[FileBatch]] = {
+ val chunks = Vector.newBuilder[Vector[FileBatch]]
+ var current = Vector.empty[FileBatch]
+ var currentBytes = 0
+
+ fileBatches.foreach { batch =>
+ val batchBytes = batch.estimatedPayloadBytes
+ val exceedsPayload = current.nonEmpty && currentBytes + batchBytes > maxPayloadBytes
+ val exceedsCount = current.size >= maxFiles
+ if (exceedsPayload || exceedsCount) {
+ chunks += current
+ current = Vector.empty
+ currentBytes = 0
+ }
+ current = current :+ batch
+ currentBytes += batchBytes
+ }
+
+ if (current.nonEmpty) {
+ chunks += current
+ }
+
+ chunks.result()
+ }
+
+ private def timed[A](ioa: IO[A]): IO[(A, Long)] =
+ for {
+ start <- IO.monotonic
+ result <- ioa
+ end <- IO.monotonic
+ } yield (result, (end - start).toMillis)
+
+ private def timedUnit[A](ioa: IO[A]): IO[Long] =
+ timed(ioa).map(_._2)
+
+ private def logChunkTiming(summary: String, totalMs: Long, stages: List[(String, Long)]): IO[Unit] = {
+ val shouldLog = debugBulkWrites || totalMs >= SlowChunkMs
+ if (!shouldLog) IO.unit
+ else {
+ val stageSummary = stages.map { case (name, ms) => s"$name=${ms}ms" }.mkString(" ")
+ logger.info(s"$summary total=${totalMs}ms $stageSummary")
+ }
+ }
}
/**
@@ -140,32 +284,47 @@ case class FileBatch(
patch: GraphPatch,
provenance: java.util.Map[String, AnyRef]
) {
- private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
-
- private def parseToJavaMap(jsonStr: String): java.util.Map[String, AnyRef] =
- mapper.readValue(jsonStr, classOf[java.util.Map[String, AnyRef]])
+ // Direct Circe-to-Java conversion — no string serialization round-trip.
+ private def jsonToJava(json: Json): AnyRef = json.fold(
+ jsonNull = null.asInstanceOf[AnyRef],
+ jsonBoolean = b => java.lang.Boolean.valueOf(b),
+ jsonNumber = n => java.lang.Double.valueOf(n.toDouble),
+ jsonString = s => s,
+ jsonArray = arr => {
+ val list = new java.util.ArrayList[AnyRef](arr.size)
+ arr.foreach(j => list.add(jsonToJava(j)))
+ list
+ },
+ jsonObject = obj => {
+ val map = new java.util.LinkedHashMap[String, AnyRef](obj.size)
+ obj.toMap.foreach { case (k, v) => map.put(k, jsonToJava(v)) }
+ map
+ }
+ )
- private def jsonToJava(json: Json): AnyRef =
- mapper.readValue(json.noSpaces, classOf[AnyRef])
+ private def attrsToJavaMap(attrs: Map[String, Json]): java.util.Map[String, AnyRef] = {
+ val result = new java.util.HashMap[String, AnyRef](attrs.size)
+ attrs.foreach { case (k, v) => result.put(k, jsonToJava(v)) }
+ result
+ }
private def nodeKindToString(nk: NodeKind): String =
nk.asJson.asString.getOrElse(nk.toString)
lazy val estimatedPayloadBytes: Int =
- (patch.asJson.noSpaces.getBytes(StandardCharsets.UTF_8).length * 4) + 1024
+ patch.asJson.noSpaces.getBytes(StandardCharsets.UTF_8).length + 1024
def nodeDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] = {
val now = Instant.now().toString
patch.ops.collect { case PatchOp.UpsertNode(id, kind, name, attrs) =>
val logicalId = id.value.toString
- val attrsJson = Json.obj(attrs.toSeq.map { case (k, v) => k -> v }: _*).noSpaces
val doc = new java.util.HashMap[String, AnyRef]()
doc.put("_key", s"${logicalId}_${rev}")
doc.put("logical_id", logicalId)
doc.put("id", logicalId)
doc.put("kind", nodeKindToString(kind))
doc.put("name", name)
- doc.put("attrs", parseToJavaMap(attrsJson))
+ doc.put("attrs", attrsToJavaMap(attrs))
doc.put("provenance", provenance)
doc.put("created_rev", Long.box(rev))
doc.put("deleted_rev", null)
@@ -178,7 +337,6 @@ case class FileBatch(
def edgeDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] = {
val now = Instant.now().toString
patch.ops.collect { case PatchOp.UpsertEdge(id, src, dst, predicate, attrs) =>
- val attrsJson = Json.obj(attrs.toSeq.map { case (k, v) => k -> v }: _*).noSpaces
val doc = new java.util.HashMap[String, AnyRef]()
doc.put("_key", id.value.toString)
doc.put("_from", s"nodes/${src.value}")
@@ -187,7 +345,7 @@ case class FileBatch(
doc.put("src", src.value.toString)
doc.put("dst", dst.value.toString)
doc.put("predicate", predicate.value)
- doc.put("attrs", parseToJavaMap(attrsJson))
+ doc.put("attrs", attrsToJavaMap(attrs))
doc.put("created_rev", Long.box(rev))
doc.put("deleted_rev", null)
doc.put("provenance", provenance)
@@ -214,12 +372,11 @@ case class FileBatch(
}
def patchDocument(rev: Long): java.util.Map[String, AnyRef] = {
- val patchJson = patch.asJson.noSpaces
val doc = new java.util.HashMap[String, AnyRef]()
doc.put("_key", patch.patchId.value.toString)
doc.put("patch_id", patch.patchId.value.toString)
doc.put("rev", Long.box(rev))
- doc.put("data", mapper.readValue(patchJson, classOf[java.util.Map[String, AnyRef]]))
+ doc.put("data", jsonToJava(patch.asJson))
doc: java.util.Map[String, AnyRef]
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
index 1784f991..9cde2da4 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
@@ -1,6 +1,7 @@
package ix.memory.map
import cats.effect.IO
+import org.typelevel.log4cats.slf4j.Slf4jLogger
import ix.memory.db.ArangoClient
import ix.memory.model.{NodeId, Rev}
@@ -21,6 +22,7 @@ import ix.memory.model.{NodeId, Rev}
* ε = 0.2 path proximity (weak structural prior)
*/
class MapGraphBuilder(client: ArangoClient) {
+ private val logger = Slf4jLogger.getLoggerFromName[IO]("ix.map.build")
private val Alpha = 1.0
private val Beta = 0.5
@@ -32,7 +34,8 @@ class MapGraphBuilder(client: ArangoClient) {
private val SourceGraphRevisionKey = "source-graph"
private val IgnoredNodeKinds = Vector(
"module", "config", "config_entry", "doc", "decision", "intent",
- "bug", "plan", "task", "goal", "region"
+ "bug", "plan", "task", "goal", "region",
+ "chunk" // chunks are code-text segments, never CALLS/IMPORTS/EXTENDS/IMPLEMENTS endpoints
)
private val CouplingPredicates = Vector("CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS")
@@ -40,8 +43,14 @@ class MapGraphBuilder(client: ArangoClient) {
def buildGraph(files: Vector[FileVertex]): IO[WeightedFileGraph] =
for {
+ t0 <- IO(System.currentTimeMillis())
rawPairs <- fetchCouplingByUri()
+ t1 <- IO(System.currentTimeMillis())
graph = computeGraph(files, rawPairs)
+ t2 <- IO(System.currentTimeMillis())
+ _ <- logger.info(
+ s"map graph files=${files.size} raw_pairs=${rawPairs.size} fetch=${t1-t0}ms compute=${t2-t1}ms total=${t2-t0}ms"
+ )
} yield graph
def build(): IO[WeightedFileGraph] =
@@ -226,9 +235,7 @@ class MapGraphBuilder(client: ArangoClient) {
scala.collection.mutable.Map[(NodeId, NodeId), Map[String, Int]]()
var edgeCount = 0
- for (((srcId, dstId), predicateCounts) <- pairAcc.toVector.sortBy { case ((a, b), _) =>
- (a.value.toString, b.value.toString)
- }) {
+ for (((srcId, dstId), predicateCounts) <- pairAcc.iterator) {
val sv = byId(srcId)
val dv = byId(dstId)
val srcSegments = pathSegmentsById(srcId)
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapService.scala b/memory-layer/src/main/scala/ix/memory/map/MapService.scala
index f1c5c96a..a4b78081 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapService.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapService.scala
@@ -4,8 +4,10 @@ import java.time.Instant
import java.util.UUID
import cats.effect.IO
+import com.fasterxml.jackson.databind.ObjectMapper
import io.circe.Json
import io.circe.syntax._
+import org.typelevel.log4cats.slf4j.Slf4jLogger
import ix.memory.db.{ArangoClient, GraphQueryApi, GraphWriteApi}
import ix.memory.model._
@@ -26,6 +28,8 @@ class MapService(
queryApi: GraphQueryApi,
writeApi: GraphWriteApi
) {
+ private val mapper = new ObjectMapper()
+ private val logger = Slf4jLogger.getLoggerFromName[IO]("ix.map")
private val CrosscutThreshold = 0.10
// Cross-cut penalty multiplier (reduces edge weights involving crosscut files)
@@ -46,7 +50,8 @@ class MapService(
"PATH" -> 0.2
)
- private val builder = new MapGraphBuilder(client)
+ private val fullBuilder = new MapGraphBuilder(client)
+ private val preflight = new MapPreflight()
private final class RetainedRegion(val level: Int, val communityIndex: Int, val region: Region)
// Label kinds by level
@@ -56,20 +61,20 @@ class MapService(
else if (level == 1) "module"
else "subsystem"
- def buildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =
- getOrBuildMap(forceRecompute)
+ def buildMap(forceRecompute: Boolean = false, forceFull: Boolean = false): IO[ArchitectureMap] =
+ getOrBuildMap(forceRecompute, forceFull)
- def getOrBuildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =
+ def getOrBuildMap(forceRecompute: Boolean = false, forceFull: Boolean = false): IO[ArchitectureMap] =
for {
- inputRev <- builder.currentInputRev()
+ inputRev <- fullBuilder.currentInputRev()
cached <- loadCachedMap(inputRev)
archMap <- if (cached.nonEmpty && !forceRecompute) IO.pure(cached.get)
- else computeAndPersistMap(inputRev, cached)
+ else computeAndPersistMap(inputRev, cached, forceFull)
} yield archMap
def loadPersistedMapForCurrentGraph(): IO[Option[ArchitectureMap]] =
for {
- inputRev <- builder.currentInputRev()
+ inputRev <- fullBuilder.currentInputRev()
cached <- loadCachedMap(inputRev)
} yield cached
@@ -263,14 +268,15 @@ class MapService(
val withChildren = withParents.map { r =>
r.copy(childRegionIds = childrenByParent.getOrElse(r.id, Set.empty))
}
+ val dedupedHierarchy = collapseDuplicateHierarchyLevels(withChildren)
// Disambiguate duplicate labels within each level.
// For each group of same-label regions, find a path segment unique to each.
- val labelGroups = withChildren.groupBy(r => (r.level, r.label))
+ val labelGroups = dedupedHierarchy.groupBy(r => (r.level, r.label))
val needsDisambig = labelGroups.filter(_._2.size > 1)
- if (needsDisambig.isEmpty) withChildren
- else withChildren.map { r =>
+ if (needsDisambig.isEmpty) dedupedHierarchy
+ else dedupedHierarchy.map { r =>
needsDisambig.get((r.level, r.label)) match {
case Some(group) =>
val myPaths = r.memberFiles.flatMap(id => fileVertexById.get(id).map(_.path)).toSet
@@ -299,6 +305,14 @@ class MapService(
}
}
+ private[map] def collapseDuplicateHierarchyLevels(regions: Vector[Region]): Vector[Region] = {
+ val keepById = regions.groupBy(_.id).view.mapValues { duplicates =>
+ duplicates.maxBy(region => (region.level, region.confidence, region.label.length))
+ }.toMap
+
+ regions.filter(region => keepById.get(region.id).contains(region))
+ }
+
private val LabelNoiseTokens = Set(
"main", "scala", "java", "src", "test", "tests", "__tests__", "kotlin",
"python", "js", "ts", "tsx", "jsx", "com", "org", "io", "net",
@@ -600,15 +614,36 @@ class MapService(
// ── Persistence ─────────────────────────────────────────────────────
- private def computeAndPersistMap(inputRev: Rev, cached: Option[ArchitectureMap]): IO[ArchitectureMap] =
+ private def computeAndPersistMap(
+ inputRev: Rev,
+ cached: Option[ArchitectureMap],
+ forceFull: Boolean
+ ): IO[ArchitectureMap] =
for {
- rawGraph <- builder.build()
- crosscutScores = detectCrosscut(rawGraph)
- graph = applyPenalty(rawGraph, crosscutScores)
- levels = LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3)
- regions = buildRegions(graph, levels, crosscutScores, inputRev)
- fresh = ArchitectureMap(regions, rawGraph.vertices.size, inputRev.value)
- _ <- persistMap(fresh, cached)
+ files <- fullBuilder.discoverFiles()
+ preflightEval <- preflight.evaluate(files)
+ t0 <- IO(System.currentTimeMillis())
+ rawGraph <- fullBuilder.buildGraph(files)
+ t1 <- IO(System.currentTimeMillis())
+ crosscutScores <- IO.blocking(detectCrosscut(rawGraph))
+ graph <- IO.blocking(applyPenalty(rawGraph, crosscutScores))
+ t2 <- IO(System.currentTimeMillis())
+ levels <- IO.blocking(LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3))
+ t3 <- IO(System.currentTimeMillis())
+ regions <- IO.blocking(buildRegions(graph, levels, crosscutScores, inputRev))
+ fresh = ArchitectureMap(
+ regions = regions,
+ fileCount = rawGraph.vertices.size,
+ mapRev = inputRev.value,
+ preflight = Some(preflightEval),
+ outcome = MapOutcome.FullLocalCompleted
+ )
+ t4 <- IO(System.currentTimeMillis())
+ _ <- persistMap(fresh, cached)
+ t5 <- IO(System.currentTimeMillis())
+ _ <- logger.info(
+ s"map mode=${fresh.outcome.label} files=${files.size} build=${t1-t0}ms crosscut=${t2-t1}ms cluster=${t3-t2}ms regions=${t4-t3}ms persist=${t5-t4}ms total=${t5-t0}ms"
+ )
} yield fresh
private def loadCachedMap(inputRev: Rev): IO[Option[ArchitectureMap]] =
@@ -629,7 +664,7 @@ class MapService(
)
decoded = rows.flatMap(decodePersistedRegion(_, inputRev)).toVector
regions = rehydrateRelationships(decoded)
- fileCount <- if (regions.isEmpty) IO.pure(0) else builder.liveFileCount()
+ fileCount <- if (regions.isEmpty) IO.pure(0) else fullBuilder.liveFileCount()
} yield {
if (regions.isEmpty) None
else Some(ArchitectureMap(regions, fileCount, inputRev.value))
@@ -700,9 +735,8 @@ class MapService(
if (cached.exists(existing => sameRegions(existing.regions, fresh.regions))) IO.unit
else {
for {
- oldRegions <- queryApi.findNodesByKind(NodeKind.Region, limit = 5000)
- oldEdgeIds <- loadActiveRegionEdgeIds(oldRegions.map(_.id))
- _ <- submitMapPatch(oldRegions, oldEdgeIds, fresh.regions, Rev(fresh.mapRev))
+ oldLogicalIds <- loadOldRegionIds()
+ _ <- submitMapPatch(oldLogicalIds, fresh.regions, Rev(fresh.mapRev))
} yield ()
}
}
@@ -710,103 +744,189 @@ class MapService(
private def sameRegions(left: Vector[Region], right: Vector[Region]): Boolean =
left.sortBy(_.id.value.toString) == right.sortBy(_.id.value.toString)
- private def loadActiveRegionEdgeIds(regionIds: Vector[NodeId]): IO[Vector[EdgeId]] =
- if (regionIds.isEmpty) IO.pure(Vector.empty)
- else {
- val ids = regionIds.map(_.value.toString).toArray
- client.query(
- """FOR e IN edges
- | FILTER e.deleted_rev == null
- | AND e.predicate == "IN_REGION"
- | AND (e.src IN @regionIds OR e.dst IN @regionIds)
- | RETURN e._key""".stripMargin,
- Map("regionIds" -> ids.asInstanceOf[AnyRef])
- ).map(_.flatMap(_.as[String].toOption).flatMap(id => scala.util.Try(UUID.fromString(id)).toOption.map(EdgeId(_))).toVector)
- }
+ /** Fetch only the logical_ids of current live region nodes — cheaper than full GraphNode. */
+ private def loadOldRegionIds(): IO[Vector[String]] =
+ client.query(
+ """FOR n IN nodes
+ | FILTER n.kind == "region"
+ | AND n.deleted_rev == null
+ | LIMIT 5000
+ | RETURN n.logical_id""".stripMargin,
+ Map.empty
+ ).map(_.flatMap(_.as[String].toOption).toVector)
+ /**
+ * Persist map regions using direct bulk operations instead of the per-op transaction path.
+ *
+ * This replaces the old commitPatch approach (which made N+1 DB round-trips inside
+ * a transaction, causing timeouts on large repos) with five bulk operations:
+ * 1. Batch AQL tombstone for old region nodes
+ * 2. Batch AQL tombstone for old region edges
+ * 3. bulkInsert for new region nodes
+ * 4. bulkInsertEdges for IN_REGION edges
+ * 5. Revision update
+ */
private def submitMapPatch(
- oldRegions: Vector[GraphNode],
- oldEdgeIds: Vector[EdgeId],
- regions: Vector[Region],
- rev: Rev
+ oldRegionIds: Vector[String],
+ regions: Vector[Region],
+ rev: Rev
): IO[Unit] = {
- val deleteOps: Vector[PatchOp] = oldRegions.map(n => PatchOp.DeleteNode(n.id))
- val deleteEdgeOps: Vector[PatchOp] = oldEdgeIds.map(PatchOp.DeleteEdge)
-
- val nodeOps: Vector[PatchOp] = regions.map { r =>
- PatchOp.UpsertNode(
- id = r.id,
- kind = NodeKind.Region,
- name = r.label,
- attrs = Map(
- "label" -> r.label.asJson,
- "map_generated" -> Json.True,
- "level" -> r.level.asJson,
- "label_kind" -> r.labelKind.asJson,
- "file_count" -> r.memberFiles.size.asJson,
- "member_files" -> r.memberFiles.map(_.value.toString).asJson,
- "parent_id" -> r.parentId.map(_.value.toString).asJson,
- "child_region_ids" -> r.childRegionIds.toVector.map(_.value.toString).sorted.asJson,
- "cohesion" -> r.cohesion.asJson,
- "external_coupling" -> r.externalCoupling.asJson,
- "boundary_ratio" -> r.boundaryRatio.asJson,
- "confidence" -> r.confidence.asJson,
- "crosscut_score" -> r.crosscutScore.asJson,
- "dominant_signals" -> r.dominantSignals.asJson,
- "interface_node_count" -> r.interfaceNodeCount.asJson,
- "map_rev" -> r.mapRev.asJson
- )
- )
+ if (oldRegionIds.isEmpty && regions.isEmpty) return IO.unit
+
+ val now = Instant.now().toString
+ val provMap = {
+ val m = new java.util.HashMap[String, AnyRef]()
+ m.put("source_uri", "ix:map")
+ m.put("source_hash", null)
+ m.put("extractor", "map-service")
+ m.put("source_type", "inferred")
+ m.put("observed_at", now)
+ m
}
- // IN_REGION edges: region → file (level-1 regions only, top-down)
- val fileEdgeOps: Vector[PatchOp] = regions
- .filter(_.level == 1)
- .flatMap { r =>
- r.memberFiles.toVector.map { fileId =>
- val edgeId = EdgeId(
- UUID.nameUUIDFromBytes(s"in_region:${r.id.value}:${fileId.value}".getBytes("UTF-8"))
- )
- PatchOp.UpsertEdge(
- id = edgeId,
- src = r.id,
- dst = fileId,
- predicate = EdgePredicate("IN_REGION"),
- attrs = Map.empty
- )
- }
- }
+ for {
+ latestRev <- queryApi.getLatestRev
+ newRev = latestRev.value + 1L
+
+ // 1. Tombstone old region nodes
+ _ <- if (oldRegionIds.isEmpty) IO.unit
+ else {
+ val ids = new java.util.ArrayList[String]()
+ oldRegionIds.foreach(ids.add)
+ client.execute(
+ """FOR n IN nodes
+ | FILTER n.logical_id IN @ids AND n.deleted_rev == null
+ | UPDATE n WITH { deleted_rev: @rev, updated_at: @now } IN nodes""".stripMargin,
+ Map(
+ "ids" -> ids.asInstanceOf[AnyRef],
+ "rev" -> Long.box(newRev).asInstanceOf[AnyRef],
+ "now" -> now.asInstanceOf[AnyRef]
+ )
+ )
+ }
+
+ // 2. Tombstone all active IN_REGION edges in one indexed query.
+ // Replaces the old loadActiveRegionEdgeIds approach (which did a full
+ // collection scan with a large src/dst IN clause — the main freeze point).
+ _ <- client.execute(
+ """FOR e IN edges
+ | FILTER e.predicate == "IN_REGION"
+ | AND e.deleted_rev == null
+ | UPDATE e WITH { deleted_rev: @rev, updated_at: @now } IN edges""".stripMargin,
+ Map(
+ "rev" -> Long.box(newRev).asInstanceOf[AnyRef],
+ "now" -> now.asInstanceOf[AnyRef]
+ )
+ )
+
+ // 3. Bulk-insert new region nodes
+ nodeDocs = regions.map(r => buildRegionNodeDoc(r, newRev, now, provMap))
+ _ <- if (nodeDocs.isEmpty) IO.unit
+ else client.bulkInsert("nodes", nodeDocs).void
+
+ // 4. Bulk-insert IN_REGION edges
+ edgeDocs = buildRegionEdgeDocs(regions, newRev, now, provMap)
+ _ <- if (edgeDocs.isEmpty) IO.unit
+ else client.bulkInsertEdges("edges", edgeDocs).void
+
+ // 5. Update current revision
+ _ <- client.execute(
+ """UPSERT { _key: "current" }
+ | INSERT { _key: "current", rev: @rev }
+ | UPDATE { rev: @rev }
+ | IN revisions""".stripMargin,
+ Map("rev" -> Long.box(newRev).asInstanceOf[AnyRef])
+ )
+ } yield ()
+ }
- // IN_REGION edges: parent region → child region (top-down)
- val regionEdgeOps: Vector[PatchOp] = regions.flatMap { r =>
+ private def buildRegionNodeDoc(
+ r: Region,
+ newRev: Long,
+ now: String,
+ provMap: java.util.Map[String, AnyRef]
+ ): java.util.Map[String, AnyRef] = {
+ val logicalId = r.id.value.toString
+ val attrsJson = Json.obj(
+ "label" -> r.label.asJson,
+ "map_generated" -> Json.True,
+ "level" -> r.level.asJson,
+ "label_kind" -> r.labelKind.asJson,
+ "file_count" -> r.memberFiles.size.asJson,
+ "member_files" -> r.memberFiles.map(_.value.toString).asJson,
+ "parent_id" -> r.parentId.map(_.value.toString).asJson,
+ "child_region_ids" -> r.childRegionIds.toVector.map(_.value.toString).sorted.asJson,
+ "cohesion" -> r.cohesion.asJson,
+ "external_coupling" -> r.externalCoupling.asJson,
+ "boundary_ratio" -> r.boundaryRatio.asJson,
+ "confidence" -> r.confidence.asJson,
+ "crosscut_score" -> r.crosscutScore.asJson,
+ "dominant_signals" -> r.dominantSignals.asJson,
+ "interface_node_count" -> r.interfaceNodeCount.asJson,
+ "map_rev" -> r.mapRev.asJson
+ ).noSpaces
+ val doc = new java.util.HashMap[String, AnyRef]()
+ doc.put("_key", s"${logicalId}_${newRev}")
+ doc.put("logical_id", logicalId)
+ doc.put("id", logicalId)
+ doc.put("kind", "region")
+ doc.put("name", r.label)
+ doc.put("attrs", mapper.readValue(attrsJson, classOf[java.util.Map[String, AnyRef]]))
+ doc.put("provenance", provMap)
+ doc.put("created_rev", Long.box(newRev))
+ doc.put("deleted_rev", null)
+ doc.put("created_at", now)
+ doc.put("updated_at", now)
+ doc
+ }
+
+ private def buildRegionEdgeDocs(
+ regions: Vector[Region],
+ newRev: Long,
+ now: String,
+ provMap: java.util.Map[String, AnyRef]
+ ): Vector[java.util.Map[String, AnyRef]] = {
+ val fileEdges = regions.filter(_.level == 1).flatMap { r =>
+ r.memberFiles.toVector.map { fileId =>
+ val edgeId = UUID.nameUUIDFromBytes(s"in_region:${r.id.value}:${fileId.value}".getBytes("UTF-8"))
+ buildRegionEdgeDoc(edgeId.toString, s"nodes/${r.id.value}", s"nodes/${fileId.value}",
+ r.id.value.toString, fileId.value.toString, newRev, now, provMap)
+ }
+ }
+ val regionEdges = regions.flatMap { r =>
r.parentId.map { pid =>
- val edgeId = EdgeId(
- UUID.nameUUIDFromBytes(s"in_region:${pid.value}:${r.id.value}".getBytes("UTF-8"))
- )
- PatchOp.UpsertEdge(
- id = edgeId,
- src = pid,
- dst = r.id,
- predicate = EdgePredicate("IN_REGION"),
- attrs = Map.empty
- )
+ val edgeId = UUID.nameUUIDFromBytes(s"in_region:${pid.value}:${r.id.value}".getBytes("UTF-8"))
+ buildRegionEdgeDoc(edgeId.toString, s"nodes/${pid.value}", s"nodes/${r.id.value}",
+ pid.value.toString, r.id.value.toString, newRev, now, provMap)
}
}
+ fileEdges ++ regionEdges
+ }
- val allOps = deleteEdgeOps ++ deleteOps ++ nodeOps ++ fileEdgeOps ++ regionEdgeOps
- if (allOps.isEmpty) return IO.unit
-
- val patch = GraphPatch(
- patchId = PatchId(UUID.nameUUIDFromBytes(s"map:${rev.value}".getBytes("UTF-8"))),
- actor = "map-service",
- timestamp = Instant.now(),
- source = PatchSource("ix:map", None, "map-service", SourceType.Inferred),
- baseRev = rev,
- ops = allOps,
- replaces = Vector.empty,
- intent = Some(s"Architectural map computed at rev ${rev.value}")
- )
-
- writeApi.commitPatch(patch).void
+ private def buildRegionEdgeDoc(
+ key: String,
+ from: String,
+ to: String,
+ src: String,
+ dst: String,
+ newRev: Long,
+ now: String,
+ provMap: java.util.Map[String, AnyRef]
+ ): java.util.Map[String, AnyRef] = {
+ val doc = new java.util.HashMap[String, AnyRef]()
+ doc.put("_key", key)
+ doc.put("_from", from)
+ doc.put("_to", to)
+ doc.put("id", key)
+ doc.put("src", src)
+ doc.put("dst", dst)
+ doc.put("predicate", "IN_REGION")
+ doc.put("attrs", new java.util.HashMap[String, AnyRef]())
+ doc.put("created_rev", Long.box(newRev))
+ doc.put("deleted_rev", null)
+ doc.put("provenance", provMap)
+ doc.put("created_at", now)
+ doc.put("updated_at", now)
+ doc
}
}
diff --git a/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala
index 9d88ee03..8a7ab64b 100644
--- a/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala
+++ b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala
@@ -52,6 +52,24 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {
view.summary.crossCutting shouldBe 1
}
+ "collapseDuplicateHierarchyLevels" should "keep only the coarsest region for repeated member sets" in {
+ val sharedId = NodeId(UUID.randomUUID())
+ val sharedFiles = (1 to 10).map(_ => NodeId(UUID.randomUUID())).toSet
+ val child = region("Edges / Resolver", "module", 1, files = 4, parentId = Some(sharedId))
+ val module = regionWithMembers("Edges", "module", 1, sharedId, sharedFiles)
+ val subsystem = regionWithMembers("Edges", "subsystem", 2, sharedId, sharedFiles)
+ val system = regionWithMembers("Edges", "system", 3, sharedId, sharedFiles)
+
+ val collapsed = service.collapseDuplicateHierarchyLevels(Vector(child, module, subsystem, system))
+
+ collapsed.map(region => region.label -> region.labelKind) shouldBe Vector(
+ "Edges / Resolver" -> "module",
+ "Edges" -> "system"
+ )
+ collapsed.map(_.id) should contain (sharedId)
+ collapsed.find(_.label == "Edges / Resolver").flatMap(_.parentId) shouldBe Some(sharedId)
+ }
+
private def region(
label: String,
labelKind: String,
@@ -60,9 +78,21 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {
confidence: Double = 0.80,
crosscut: Double = 0.0,
parent: Option[Region] = None,
+ parentId: Option[NodeId] = None,
+ ): Region = {
+ regionWithMembers(label, labelKind, level, NodeId(UUID.randomUUID()), (1 to files).map(_ => NodeId(UUID.randomUUID())).toSet, confidence, crosscut, parent.map(_.id).orElse(parentId))
+ }
+
+ private def regionWithMembers(
+ label: String,
+ labelKind: String,
+ level: Int,
+ id: NodeId,
+ memberFiles: Set[NodeId],
+ confidence: Double = 0.80,
+ crosscut: Double = 0.0,
+ parentId: Option[NodeId] = None,
): Region = {
- val id = NodeId(UUID.randomUUID())
- val memberFiles = (1 to files).map(_ => NodeId(UUID.randomUUID())).toSet
Region(
id = id,
label = label,
@@ -70,7 +100,7 @@ class MapServiceScopeSpec extends AnyFlatSpec with Matchers {
level = level,
memberFiles = memberFiles,
childRegionIds = Set.empty,
- parentId = parent.map(_.id),
+ parentId = parentId,
cohesion = 0.5,
externalCoupling = 0.2,
boundaryRatio = 1.5,