Skip to content

armgit / Ix

Public Imported from GitHub
0

Understand any codebase instantly. System intelligence for codebases, built for humans and AI.

merge large merge into main

Riley Thompsone38a87a11726
diff --git a/AGENTS.md b/AGENTS.md
index 0e72fb17..1fe58ccd 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -89,6 +89,8 @@ ix bug show <id> --format json # Bug details
- "What are the most critical methods?" → `ix rank --by callers --kind method --top 10`
- "What changed between rev 5 and 10?" → `ix diff 5 10 --summary`
- "Show me all changes" → `ix diff 5 10 --full --format json`
+- "Are there architectural problems?" → `ix smells --format json`
+- "Which subsystems are unhealthy?" → `ix subsystems --format json`
### Low-level structural primitives
@@ -119,9 +121,24 @@ ix imported-by <symbol> --format json
ix depends <symbol> --format json
```
-### Ingesting data
+### Architecture analysis
```
-ix ingest ./src --recursive --format json
+ix smells --format json # Detect orphan files, god modules, weak components
+ix smells --list --format json # List previously stored smell claims
+ix smells --god-module-chunks 15 --god-module-fan 10 # Custom thresholds
+ix subsystems --format json # Score subsystems by health (0–1)
+ix subsystems --list --format json # List stored scores
+ix subsystems --level 2 --format json # Filter to subsystem level
+```
+
+**When to use which:**
+- "Are there any architectural problems?" → `ix smells`
+- "Which subsystems are unhealthy?" → `ix subsystems --format json`
+- "Show me god modules" → `ix smells --format json | jq '.candidates[] | select(.smell == "has_smell.god_module")'`
+
+### Updating the graph
+```
+ix map --format json
ix ingest --github owner/repo --format json --limit 50
ix ingest --github owner/repo --since 2026-01-01 --format json
```
@@ -135,27 +152,72 @@ ix diff <from> <to> --format json # Full diff (default limit 100)
ix diff <from> <to> --full --format json # All changes, no limit
```
-### Planning & decisions
+### Planning & decisions (Pro)
```
+# Goals
ix goal create "GitHub ingestion pipeline" --format json
-ix goal list --format json
-ix plan create "GitHub Ingestion" --goal <goal-id> --format json
+ix goal show <id-or-name> --format json
+ix goal list --status active --format json
+ix goals --format json # shorthand list
+
+# Plans
+ix plan create "GitHub Ingestion" --goal <goal-id> --responds-to <bugId> --format json
+ix plan status <plan-id> --format json # task list + critical path + open bug count
+ix plan next <plan-id> --with-workflow --format json # next actionable task + workflow
+ix plan next <plan-id> --run-workflow --stage discover --format json # execute one stage
+ix plans --format json # shorthand list
+
+# Tasks
ix plan task "API fetch layer" --plan <plan-id> --depends-on <task-id> --format json
-ix plan task "title" --plan <id> --workflow "cmd1,cmd2" --format json # Task with flat workflow
-ix plan task "title" --plan <id> --workflow-staged '{"discover":["cmd1"],"implement":["cmd2"],"validate":["cmd3"]}' --format json # Staged workflow
-ix plan task "title" --plan <id> --resolves <bugId> --format json # Task resolving a bug
-ix plan status <plan-id> --format json # Includes openBugCount in summary
-ix plan next <plan-id> --format json # Explains reason if no actionable task
-ix plan next <plan-id> --with-workflow --format json # Next task with staged workflow display
-ix tasks --format json # List all tasks across plans
-ix tasks --status pending --format json # Filter tasks by status
-ix tasks --plan <plan-id> --format json # Tasks in a specific plan
-ix task show <task-id> --with-workflow --format json # Task details with workflow
+ix plan task "title" --plan <id> --workflow-staged '{"discover":["ix overview X"],"implement":["ix map"],"validate":["ix smells"]}' --format json
+ix plan task "title" --plan <id> --resolves <bugId> --format json
+ix tasks --format json # list all tasks across plans
+ix tasks --status pending --plan <plan-id> --format json
+ix task show <task-id> --with-workflow --format json
ix task update <task-id> --status done --format json
+ix task update <task-id> --run-workflow implement --format json # run a workflow stage
+
+# Decisions
ix decide "Use client-side patch" --rationale "..." --affects IngestionService --format json
-ix decide "Use JWT" --rationale "..." --responds-to <bugId> --format json # Decision responding to a bug
+ix decide "Use JWT" --rationale "..." --responds-to <bugId> --format json
+
+# Bugs
+ix bug create "title" --severity high --affects Entity --format json
+ix bug update <id> --status resolved --format json
+ix bug show <id> --format json
+ix bugs --status open --format json
+
+# Misc
+ix patches --limit 20 --format json # recent graph patches
+ix truth add "Support 100k file repos"
+ix truth list --format json
```
+### Workflows (Pro)
+Staged command sequences attached to tasks, plans, or decisions. Stages: `discover → implement → validate`. All commands must start with `ix ` — no arbitrary shell.
+```
+ix workflow attach task <id> --file workflow.json # attach from JSON file
+ix workflow show task <id> --format json # view attached workflow
+ix workflow validate task <id> --format json # check structure
+ix workflow run task <id> --stage implement --format json # execute one stage
+```
+
+**Workflow JSON format:**
+```json
+{
+ "discover": ["ix overview AuthService", "ix impact AuthService"],
+ "implement": ["ix map"],
+ "validate": ["ix smells --format json"]
+}
+```
+
+**When to use which:**
+- "What work is planned?" → `ix plans --format json` then `ix plan status <id>`
+- "What should I do next?" → `ix plan next <id> --with-workflow`
+- "Execute the next task's discovery phase" → `ix plan next <id> --run-workflow --stage discover`
+- "Mark a task done" → `ix task update <id> --status done`
+- "Link a decision to a bug" → `ix decide "title" --rationale "..." --responds-to <bugId>`
+
## Semantic boundaries
- **decision** — a choice between alternatives, with rationale. Use `ix decide`.
diff --git a/CLAUDE.md b/CLAUDE.md
index d5051404..15f1f736 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -14,7 +14,7 @@ Use JSON when chaining command results.
2. AFTER every design or architecture decision → run `ix decide <title> --rationale <text>`.
3. When you notice contradictory information → run `ix conflicts` and present results to the user.
4. NEVER guess about codebase facts — if Ix has structured data, use it.
-5. IMMEDIATELY after modifying code → run `ix ingest <path>` on changed files.
+5. IMMEDIATELY after modifying code → run `ix map` to re-ingest and update the graph.
6. When the user states a goal → run `ix truth add "<statement>"`.
## Ix CLI Command Routing
@@ -49,7 +49,7 @@ Underlying structural commands — useful for debugging or fine-grained inspecti
| Read source code | `ix read` | `ix read src/auth.py:10-50` or `ix read verify_token` |
| Full entity details | `ix entity` | `ix entity <id> --format json` |
| Fast text search | `ix text` | `ix text "verify_token" --language python --limit 20` |
-| Find symbol (graph+text) | `ix locate` | `ix locate AuthProvider --limit 10` |
+| Find symbol (graph+text) | `ix locate` | `ix locate AuthProvider --kind class` |
#### Navigating Relationships
| Goal | Command | Example |
@@ -72,13 +72,58 @@ Underlying structural commands — useful for debugging or fine-grained inspecti
| Record a goal | `ix truth add` | `ix truth add "Support 100k file repos"` |
| List goals | `ix truth list` | `ix truth list --format json` |
| Bug tracking | `ix bug create` | `ix bug create "title" --severity high --affects Entity` |
-| Bug listing | `ix bugs` | `ix bugs --format json` |
+| Update bug status | `ix bug update` | `ix bug update <id> --status resolved` |
+| Bug listing | `ix bugs` | `ix bugs --status open --format json` |
| Bug details | `ix bug show` | `ix bug show <id> --format json` |
+| List recent patches | `ix patches` | `ix patches --limit 20 --format json` |
+
+### Planning (Pro)
+| Goal | Command | Example |
+|---|---|---|
+| Create a goal | `ix goal create` | `ix goal create "Support GitHub" --format json` |
+| List goals | `ix goals` | `ix goals --status active --format json` |
+| Create a plan | `ix plan create` | `ix plan create "Fix auth" --goal <id> --responds-to <bugId> --format json` |
+| Add a task | `ix plan task` | `ix plan task "Step 1" --plan <id> --depends-on <taskId> --resolves <bugId> --workflow-staged '{"discover":["ix overview X"],"implement":["ix map"],"validate":["ix smells"]}' --format json` |
+| Plan status | `ix plan status` | `ix plan status <id> --format json` |
+| Next actionable task | `ix plan next` | `ix plan next <id> --with-workflow --format json` |
+| Run next task workflow | `ix plan next` | `ix plan next <id> --run-workflow --stage discover --format json` |
+| List all plans | `ix plans` | `ix plans --format json` |
+| List tasks | `ix tasks` | `ix tasks --status pending --plan <id> --format json` |
+| Task details | `ix task show` | `ix task show <id> --with-workflow --format json` |
+| Update task | `ix task update` | `ix task update <id> --status done --format json` |
+| Run task workflow stage | `ix task update` | `ix task update <id> --run-workflow discover --format json` |
+
+### Workflows (Pro)
+Workflows are staged command sequences (discover → implement → validate) attached to tasks, plans, or decisions. All commands must start with `ix ` — no shell operators.
+
+| Goal | Command | Example |
+|---|---|---|
+| Attach workflow | `ix workflow attach` | `ix workflow attach task <id> --file workflow.json` |
+| Show workflow | `ix workflow show` | `ix workflow show task <id> --format json` |
+| Validate workflow | `ix workflow validate` | `ix workflow validate task <id>` |
+| Run workflow | `ix workflow run` | `ix workflow run task <id> --stage implement --format json` |
+
+**Workflow JSON format:**
+```json
+{
+ "discover": ["ix overview AuthService", "ix impact AuthService"],
+ "implement": ["ix map"],
+ "validate": ["ix smells --format json", "ix subsystems --format json"]
+}
+```
+
+### Architecture Analysis
+| Goal | Command | Example |
+|---|---|---|
+| Detect code smells | `ix smells` | `ix smells --format json` |
+| Score subsystems | `ix subsystems` | `ix subsystems --level 2 --format json` |
+| List smell claims | `ix smells --list` | `ix smells --list --format json` |
+| List subsystem scores | `ix subsystems --list` | `ix subsystems --list --format json` |
### Ingestion & Health
| Goal | Command | Example |
|---|---|---|
-| Ingest files | `ix ingest` | `ix ingest ./src --recursive` |
+| Update graph + map | `ix map` | `ix map` |
| Ingest GitHub data | `ix ingest` | `ix ingest --github owner/repo --limit 50` |
| Backend health | `ix status` | `ix status` |
| Graph statistics | `ix stats` | `ix stats --format json` |
@@ -137,6 +182,6 @@ Use the right entity type for the right purpose:
## Confidence Scores
Ix returns confidence scores with results. When data has low confidence:
- Mention the uncertainty to the user
-- Suggest re-ingesting the relevant files
+- Suggest re-running `ix map` to refresh the graph
- Never present low-confidence data as established fact
<!-- IX-MEMORY END -->
diff --git a/core-ingestion/dist/index.d.ts b/core-ingestion/dist/index.d.ts
index 1a7c51d5..d24611fc 100644
--- a/core-ingestion/dist/index.d.ts
+++ b/core-ingestion/dist/index.d.ts
@@ -9,6 +9,22 @@ export interface ParsedEntity {
/** Direct enclosing class/interface/trait, if any (undefined for file-level entities). */
container?: string;
}
+/** A semantic code span extracted from the AST. The primary LLM retrieval unit. */
+export interface ParsedChunk {
+ /** Semantic name of the chunk (function/class/trait name), or null for file_body. */
+ name: string | null;
+ /** Chunk kind: "function" | "method" | "class" | "interface" | "trait" | "module_block" | "file_body" */
+ chunkKind: string;
+ lineStart: number;
+ lineEnd: number;
+ startByte: number;
+ endByte: number;
+ /** SHA-256 of the chunk source text for change detection and stable identity. */
+ contentHash: string;
+ language: string;
+ /** Name of the directly enclosing class/trait/interface, if any. */
+ container?: string;
+}
export interface ParsedRelationship {
srcName: string;
dstName: string;
@@ -18,6 +34,7 @@ export interface FileParseResult {
filePath: string;
language: SupportedLanguages;
entities: ParsedEntity[];
+ chunks: ParsedChunk[];
relationships: ParsedRelationship[];
fileRole: RoleClassification;
}
diff --git a/core-ingestion/dist/index.js b/core-ingestion/dist/index.js
index 712b01de..d3e96d77 100644
--- a/core-ingestion/dist/index.js
+++ b/core-ingestion/dist/index.js
@@ -1,4 +1,5 @@
import * as nodePath from 'node:path';
+import * as crypto from 'node:crypto';
// @ts-ignore — tree-sitter has no bundled types
import Parser from 'tree-sitter';
// @ts-ignore
@@ -124,14 +125,12 @@ export function isGrammarSupported(filePath) {
// Parser instance (reused across calls)
// ---------------------------------------------------------------------------
let _parser = null;
+const _queryCache = new Map();
function getParser() {
if (!_parser)
_parser = new Parser();
return _parser;
}
-// Per-process cache for compiled tree-sitter queries, keyed by language.
-// Avoids recompiling the same query string for every file of the same language.
-const _queryCache = new Map();
function getCachedQuery(language, grammar, querySource) {
const cached = _queryCache.get(language);
if (cached && cached.grammar === grammar)
@@ -163,16 +162,17 @@ export function parseFile(filePath, source) {
const query = getCachedQuery(cacheKey, grammar, queries);
const matches = query.matches(tree.rootNode);
const fileName = nodePath.basename(filePath);
- // Count newlines instead of allocating a full string array
- let lineCount = 1;
+ let sourceLineCount = 1;
for (let i = 0; i < source.length; i++) {
if (source.charCodeAt(i) === 10)
- lineCount++;
+ sourceLineCount++;
}
const entities = [
- { name: fileName, kind: 'file', lineStart: 1, lineEnd: lineCount, language },
+ { name: fileName, kind: 'file', lineStart: 1, lineEnd: sourceLineCount, language },
];
+ const chunks = [];
const relationships = [];
+ const pendingChunks = [];
// Track class ranges for containment: [name, startLine, endLine]
const classRanges = [];
// Track seen calls per enclosing scope to avoid duplicate CALLS edges
@@ -193,6 +193,8 @@ export function parseFile(filePath, source) {
const defNode = defCapture.node;
const lineStart = defNode.startPosition.row + 1;
const lineEnd = defNode.endPosition.row + 1;
+ const startByte = defNode.startIndex;
+ const endByte = defNode.endIndex;
// Containment: file CONTAINS or class CONTAINS
const enclosing = findEnclosing(classRanges, lineStart, name);
entities.push({
@@ -203,6 +205,16 @@ export function parseFile(filePath, source) {
language,
container: enclosing ?? undefined,
});
+ pendingChunks.push({
+ name,
+ chunkKind: kind,
+ lineStart,
+ lineEnd,
+ startByte,
+ endByte,
+ language,
+ container: enclosing ?? undefined,
+ });
if (kind === 'class' || kind === 'interface' || kind === 'trait') {
classRanges.push({ name, start: lineStart, end: lineEnd });
}
@@ -345,7 +357,29 @@ export function parseFile(filePath, source) {
continue;
}
}
- return { filePath, language, entities, relationships, fileRole: classifyFileRole(filePath, source) };
+ for (const pendingChunk of pendingChunks) {
+ const chunkText = source.slice(pendingChunk.startByte, pendingChunk.endByte);
+ const contentHash = crypto.createHash('sha256').update(chunkText).digest('hex').slice(0, 16);
+ chunks.push({
+ ...pendingChunk,
+ contentHash,
+ });
+ }
+ // Fallback: if no semantic chunks found, emit one file_body chunk covering the whole file
+ if (chunks.length === 0) {
+ const contentHash = crypto.createHash('sha256').update(source).digest('hex').slice(0, 16);
+ chunks.push({
+ name: null,
+ chunkKind: 'file_body',
+ lineStart: 1,
+ lineEnd: sourceLineCount,
+ startByte: 0,
+ endByte: source.length,
+ contentHash,
+ language,
+ });
+ }
+ return { filePath, language, entities, chunks, relationships, fileRole: classifyFileRole(filePath, source) };
}
catch {
return null;
@@ -367,18 +401,33 @@ function bestQKey(fileQKeys, filePath, plainName) {
const qks = [...new Set(fileQKeys.get(filePath)?.get(plainName) ?? [])];
return qks.length === 1 ? qks[0] : null;
}
-export function resolveEdges(results, statsOut) {
- const stats = statsOut ?? {
- importLookups: 0, transitiveLookups: 0, globalFallbacks: 0,
- globalCandidateTotal: 0, resolvedImport: 0, resolvedTransitive: 0,
- resolvedGlobal: 0, resolvedQualifier: 0, skippedSameFile: 0, skippedAmbiguous: 0,
- };
- // ── Index structures (built once) ──────────────────────────────────
- // Direct file lookup: filePath → FileParseResult (replaces results.find())
- const resultByPath = new Map();
- for (const r of results) {
- resultByPath.set(r.filePath, r);
- }
+/**
+ * Resolves CALLS and EXTENDS relationships to their cross-file targets by
+ * building a symbol table and import map over the full batch of parsed files.
+ *
+ * Tiers (in priority order):
+ * 0.9 — import-scoped: dst is in a file the src explicitly imports
+ * 0.8 — transitive import-scoped: one re-export hop away
+ * 0.5 — global fallback: dst exists in exactly one other file (unambiguous)
+ *
+ * Same-file edges are already handled correctly by buildPatch and are not
+ * emitted here.
+ *
+ * Import resolution handles:
+ * - Stem-based: './payments' → 'payments.ts'
+ * - Path-aliased: '@/lib/payments' → 'payments.ts' (last segment)
+ * - Directory index: './services' → 'services/index.ts'
+ * - Bare aliases: '@components' → 'components.ts' (strip leading non-word chars)
+ * - Dotted paths: 'ix.memory.model.Edge' → stem 'Edge' → 'Edge.scala'
+ * (handles Scala/Java package imports where tree-sitter
+ * emits each identifier separately)
+ *
+ * Qualified-key resolution:
+ * - Module-level function/class: dstQualifiedKey === dstName
+ * - Unambiguous class method: dstQualifiedKey === 'ClassName.method'
+ * - Ambiguous (two entities share the same plain name): edge not emitted
+ */
+export function resolveEdges(results) {
// fileQKeys: filePath → (plainName → qualifiedKeys[])
// Mirrors the entityQKey computation in buildPatch so nodeIds match exactly.
const fileQKeys = new Map();
@@ -399,22 +448,15 @@ export function resolveEdges(results, statsOut) {
for (const [fp, qkMap] of fileQKeys) {
fileHasSymbol.set(fp, new Set(qkMap.keys()));
}
- // Precompute language per file (avoids calling languageFromPath in hot loops)
- const fileLanguage = new Map();
- for (const r of results) {
- fileLanguage.set(r.filePath, r.language);
- }
- // Inverted symbol index: symbolName → Array<{filePath, language}>
- // Replaces the O(N) global fallback scan with O(candidates) lookup.
+ // resultsByPath: O(1) lookup replacing results.find() in transitive import loop
+ const resultsByPath = new Map(results.map(r => [r.filePath, r]));
+ // symbolToFiles: plainName → filePath[] — replaces O(F) full-scan in tier-3 and qualifier fallback
const symbolToFiles = new Map();
for (const [fp, symbols] of fileHasSymbol) {
for (const sym of symbols) {
- let arr = symbolToFiles.get(sym);
- if (!arr) {
- arr = [];
- symbolToFiles.set(sym, arr);
- }
- arr.push(fp);
+ const list = symbolToFiles.get(sym) ?? [];
+ list.push(fp);
+ symbolToFiles.set(sym, list);
}
}
// stemToFiles: basename-without-extension → filePath[]
@@ -458,7 +500,6 @@ export function resolveEdges(results, statsOut) {
packageToFiles.set(pkg, list);
}
}
- // ── Helpers ────────────────────────────────────────────────────────
/** Resolve a module name to matching file paths in the batch. */
function modNameToFiles(modName, excludeFp) {
const fps = [];
@@ -496,12 +537,10 @@ export function resolveEdges(results, statsOut) {
}
return fps;
}
- // ── Main resolution loop ───────────────────────────────────────────
const resolved = [];
for (const result of results) {
const srcFilePath = result.filePath;
const srcSymbols = fileHasSymbol.get(srcFilePath);
- const srcLanguage = result.language;
// Build the set of file paths this file explicitly imports.
const importedFilePaths = new Set();
for (const rel of result.relationships) {
@@ -515,7 +554,7 @@ export function resolveEdges(results, statsOut) {
// Handles re-exports: baz.ts → index.ts (re-exports from bar.ts) → bar.ts
const transitiveFilePaths = new Set();
for (const fp of importedFilePaths) {
- const fpResult = resultByPath.get(fp); // O(1) instead of results.find()
+ const fpResult = resultsByPath.get(fp);
if (!fpResult)
continue;
for (const rel of fpResult.relationships) {
@@ -556,7 +595,6 @@ export function resolveEdges(results, statsOut) {
if (dstQualifiedKey === null)
continue;
resolved.push({ srcFilePath, srcName: rel.srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: 'IMPORTS', confidence: 0.9 });
- stats.resolvedImport++;
continue;
}
const dstName = rel.dstName;
@@ -583,23 +621,18 @@ export function resolveEdges(results, statsOut) {
if (dstQualifiedKey !== null) {
// dstName must match rel.dstName so buildPatchWithResolution can look it up
resolved.push({ srcFilePath, srcName, dstFilePath: qfp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.9 });
- stats.resolvedQualifier++;
}
}
continue;
}
- // Global qualifier fallback — use inverted index instead of scanning all files
- const qualCandidates = symbolToFiles.get(qualifierPart) ?? [];
- const qualGlobalMatches = qualCandidates.filter(fp => fp !== srcFilePath);
- stats.globalFallbacks++;
- stats.globalCandidateTotal += qualGlobalMatches.length;
+ // Global qualifier fallback — use symbolToFiles index instead of full scan
+ const qualGlobalMatches = (symbolToFiles.get(qualifierPart) ?? []).filter(fp => fp !== srcFilePath);
if (qualGlobalMatches.length === 1) {
const qfp = qualGlobalMatches[0];
if (fileHasSymbol.get(qfp)?.has(memberPart)) {
const dstQualifiedKey = bestQKey(fileQKeys, qfp, memberPart);
if (dstQualifiedKey !== null) {
resolved.push({ srcFilePath, srcName, dstFilePath: qfp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.7 });
- stats.resolvedQualifier++;
}
}
}
@@ -607,12 +640,9 @@ export function resolveEdges(results, statsOut) {
}
}
// Tier 1: same-file — already correct in buildPatch, skip here
- if (srcSymbols.has(dstName)) {
- stats.skippedSameFile++;
+ if (srcSymbols.has(dstName))
continue;
- }
// Tier 2: import-scoped (confidence 0.9)
- stats.importLookups++;
const importMatches = [];
for (const fp of importedFilePaths) {
if (fileHasSymbol.get(fp)?.has(dstName))
@@ -624,15 +654,11 @@ export function resolveEdges(results, statsOut) {
if (dstQualifiedKey === null)
continue; // ambiguous — do not emit bad nodeId
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.9 });
- stats.resolvedImport++;
continue;
}
- if (importMatches.length > 1) {
- stats.skippedAmbiguous++;
- continue;
- } // ambiguous file — do not emit
+ if (importMatches.length > 1)
+ continue; // ambiguous file — do not emit
// Tier 2.5: transitive import-scoped (confidence 0.8) — one re-export hop away
- stats.transitiveLookups++;
const transitiveMatches = [];
for (const fp of transitiveFilePaths) {
if (fileHasSymbol.get(fp)?.has(dstName))
@@ -644,26 +670,21 @@ export function resolveEdges(results, statsOut) {
if (dstQualifiedKey === null)
continue;
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.8 });
- stats.resolvedTransitive++;
continue;
}
- if (transitiveMatches.length > 1) {
- stats.skippedAmbiguous++;
- continue;
- } // ambiguous — do not emit
- // Tier 3: global fallback (confidence 0.5) — uses inverted symbol index
- // instead of scanning all files.
- stats.globalFallbacks++;
- const candidates = symbolToFiles.get(dstName) ?? [];
- const globalMatches = candidates.filter(fp => fp !== srcFilePath && fileLanguage.get(fp) === srcLanguage);
- stats.globalCandidateTotal += globalMatches.length;
+ if (transitiveMatches.length > 1)
+ continue; // ambiguous — do not emit
+ // Tier 3: global fallback (confidence 0.5) — exactly one other file defines it
+ // Only match files in the same language as the caller to avoid cross-language false positives.
+ const srcLanguage = result.language;
+ const globalMatches = (symbolToFiles.get(dstName) ?? [])
+ .filter(fp => fp !== srcFilePath && languageFromPath(fp) === srcLanguage);
if (globalMatches.length === 1) {
const fp = globalMatches[0];
const dstQualifiedKey = bestQKey(fileQKeys, fp, dstName);
if (dstQualifiedKey === null)
continue; // ambiguous — do not emit bad nodeId
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.5 });
- stats.resolvedGlobal++;
}
// 0 or >1 global matches — leave as dangling edge, do not emit
}
diff --git a/core-ingestion/dist/patch-builder.js b/core-ingestion/dist/patch-builder.js
index 684c9d48..89e88c19 100644
--- a/core-ingestion/dist/patch-builder.js
+++ b/core-ingestion/dist/patch-builder.js
@@ -22,6 +22,10 @@ function nodeId(filePath, name) {
function edgeId(filePath, src, dst, predicate) {
return deterministicId(`${normalizePath(filePath)}:${src}:${dst}:${predicate}`);
}
+/** Deterministic ID for a chunk — keyed on file + kind + name + start line to survive minor edits. */
+function chunkId(filePath, chunkKind, name, startLine) {
+ return deterministicId(`${normalizePath(filePath)}:chunk:${chunkKind}:${name ?? 'file_body'}:${startLine}`);
+}
// ---------------------------------------------------------------------------
// Source type from file extension
// ---------------------------------------------------------------------------
@@ -34,10 +38,10 @@ function sourceType(filePath) {
return 'code';
}
export function extractorName() {
- return `tree-sitter/1.12`;
+ return `tree-sitter/1.13`;
}
/** Previous extractor versions — their patches are superseded when re-ingesting. */
-export const PREVIOUS_EXTRACTORS = ['tree-sitter/1.11', 'tree-sitter/1.10', 'tree-sitter/1.9', 'tree-sitter/1.8', 'tree-sitter/1.7', 'tree-sitter/1.6', 'tree-sitter/1.5', 'tree-sitter/1.4', 'tree-sitter/1.3', 'tree-sitter/1.2', 'tree-sitter/1.1'];
+export const PREVIOUS_EXTRACTORS = ['tree-sitter/1.12', 'tree-sitter/1.11', 'tree-sitter/1.10', 'tree-sitter/1.9', 'tree-sitter/1.8', 'tree-sitter/1.7', 'tree-sitter/1.6', 'tree-sitter/1.5', 'tree-sitter/1.4', 'tree-sitter/1.3', 'tree-sitter/1.2', 'tree-sitter/1.1'];
/** Compute a patchId for a (filePath, sourceHash, extractorVersion) triple. */
function computePatchId(filePath, sourceHash, extractor) {
return deterministicId(`${normalizePath(filePath)}:${sourceHash}:${extractor}`);
@@ -50,7 +54,7 @@ function legacyPatchId(filePath, sourceHash) {
// Build a GraphPatchPayload from a FileParseResult
// ---------------------------------------------------------------------------
export function buildPatch(result, sourceHash, previousSourceHash) {
- const { filePath, entities, relationships } = result;
+ const { filePath, entities, chunks, relationships } = result;
const ops = [];
// Build a qualified-key map so that same-named entities in different
// enclosing classes within the same file get distinct nodeIds.
@@ -107,6 +111,71 @@ export function buildPatch(result, sourceHash, previousSourceHash) {
});
}
}
+ // UpsertNode + edges for each chunk
+ const fileNodeId = nodeId(filePath, entities.find(e => e.kind === 'file')?.name ?? filePath);
+ for (const chunk of chunks) {
+ const cid = chunkId(filePath, chunk.chunkKind, chunk.name, chunk.lineStart);
+ const chunkName = chunk.name ?? `file_body:${chunk.lineStart}`;
+ ops.push({
+ type: 'UpsertNode',
+ id: cid,
+ kind: 'chunk',
+ name: chunkName,
+ attrs: {
+ file_uri: filePath,
+ language: chunk.language,
+ chunk_kind: chunk.chunkKind,
+ start_line: chunk.lineStart,
+ end_line: chunk.lineEnd,
+ start_byte: chunk.startByte,
+ end_byte: chunk.endByte,
+ content_hash: chunk.contentHash,
+ parser_version: extractorName(),
+ },
+ });
+ // File -[CONTAINS]-> Chunk
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, 'file', chunkName, 'CONTAINS_CHUNK'),
+ src: fileNodeId,
+ dst: cid,
+ predicate: 'CONTAINS_CHUNK',
+ attrs: {},
+ });
+ // Chunk -[DEFINES]-> Symbol (only for named chunks)
+ if (chunk.name !== null) {
+ const symbolKey = chunk.container ? `${chunk.container}.${chunk.name}` : chunk.name;
+ const symbolNid = nodeId(filePath, symbolKey);
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, chunkName, symbolKey, 'DEFINES'),
+ src: cid,
+ dst: symbolNid,
+ predicate: 'DEFINES',
+ attrs: {},
+ });
+ }
+ }
+ // NEXT edges for source-order chunk adjacency
+ for (let i = 0; i + 1 < chunks.length; i++) {
+ const a = chunks[i];
+ const b = chunks[i + 1];
+ // Only link top-level chunks (no container) to avoid intra-class ordering noise
+ if (a.container == null && b.container == null) {
+ const aid = chunkId(filePath, a.chunkKind, a.name, a.lineStart);
+ const bid = chunkId(filePath, b.chunkKind, b.name, b.lineStart);
+ const aName = a.name ?? `file_body:${a.lineStart}`;
+ const bName = b.name ?? `file_body:${b.lineStart}`;
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, aName, bName, 'NEXT'),
+ src: aid,
+ dst: bid,
+ predicate: 'NEXT',
+ attrs: {},
+ });
+ }
+ }
// UpsertEdge for each relationship
for (const r of relationships) {
// For CONTAINS edges, srcName is the container of dstName — use that to disambiguate.
@@ -177,7 +246,7 @@ export function buildPatchWithResolution(result, sourceHash, resolvedEdges, prev
dstQualifiedKey: edge.dstQualifiedKey,
});
}
- const { filePath, entities, relationships } = result;
+ const { filePath, entities, chunks, relationships } = result;
const ops = [];
const entityQKey = new Map();
for (const e of entities) {
@@ -218,6 +287,67 @@ export function buildPatchWithResolution(result, sourceHash, resolvedEdges, prev
});
}
}
+ // UpsertNode + edges for each chunk (same logic as buildPatch)
+ const fileNodeId2 = nodeId(filePath, entities.find(e => e.kind === 'file')?.name ?? filePath);
+ for (const chunk of chunks) {
+ const cid = chunkId(filePath, chunk.chunkKind, chunk.name, chunk.lineStart);
+ const chunkName = chunk.name ?? `file_body:${chunk.lineStart}`;
+ ops.push({
+ type: 'UpsertNode',
+ id: cid,
+ kind: 'chunk',
+ name: chunkName,
+ attrs: {
+ file_uri: filePath,
+ language: chunk.language,
+ chunk_kind: chunk.chunkKind,
+ start_line: chunk.lineStart,
+ end_line: chunk.lineEnd,
+ start_byte: chunk.startByte,
+ end_byte: chunk.endByte,
+ content_hash: chunk.contentHash,
+ parser_version: extractorName(),
+ },
+ });
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, 'file', chunkName, 'CONTAINS_CHUNK'),
+ src: fileNodeId2,
+ dst: cid,
+ predicate: 'CONTAINS_CHUNK',
+ attrs: {},
+ });
+ if (chunk.name !== null) {
+ const symbolKey = chunk.container ? `${chunk.container}.${chunk.name}` : chunk.name;
+ const symbolNid = nodeId(filePath, symbolKey);
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, chunkName, symbolKey, 'DEFINES'),
+ src: cid,
+ dst: symbolNid,
+ predicate: 'DEFINES',
+ attrs: {},
+ });
+ }
+ }
+ for (let i = 0; i + 1 < chunks.length; i++) {
+ const a = chunks[i];
+ const b = chunks[i + 1];
+ if (a.container == null && b.container == null) {
+ const aid = chunkId(filePath, a.chunkKind, a.name, a.lineStart);
+ const bid = chunkId(filePath, b.chunkKind, b.name, b.lineStart);
+ const aName = a.name ?? `file_body:${a.lineStart}`;
+ const bName = b.name ?? `file_body:${b.lineStart}`;
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, aName, bName, 'NEXT'),
+ src: aid,
+ dst: bid,
+ predicate: 'NEXT',
+ attrs: {},
+ });
+ }
+ }
for (const r of relationships) {
const srcKey = resolveKey(r.srcName);
const dstKey = r.predicate === 'CONTAINS'
diff --git a/core-ingestion/src/__tests__/__snapshots__/parseFile.test.ts.snap b/core-ingestion/src/__tests__/__snapshots__/parseFile.test.ts.snap
deleted file mode 100644
index ed6ed9e8..00000000
--- a/core-ingestion/src/__tests__/__snapshots__/parseFile.test.ts.snap
+++ /dev/null
@@ -1,1347 +0,0 @@
-// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
-
-exports[`parseFile snapshots > snapshots Identifiers.scala 1`] = `
-{
- "entities": [
- {
- "kind": "file",
- "language": "scala",
- "lineEnd": 42,
- "lineStart": 1,
- "name": "Identifiers.scala",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 7,
- "lineStart": 7,
- "name": "NodeId",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 11,
- "lineStart": 8,
- "name": "NodeId",
- },
- {
- "container": "NodeId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 9,
- "lineStart": 9,
- "name": "encoder",
- },
- {
- "container": "NodeId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 10,
- "lineStart": 10,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 13,
- "lineStart": 13,
- "name": "EdgeId",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 17,
- "lineStart": 14,
- "name": "EdgeId",
- },
- {
- "container": "EdgeId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 15,
- "lineStart": 15,
- "name": "encoder",
- },
- {
- "container": "EdgeId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 16,
- "lineStart": 16,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 19,
- "lineStart": 19,
- "name": "PatchId",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 23,
- "lineStart": 20,
- "name": "PatchId",
- },
- {
- "container": "PatchId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 21,
- "lineStart": 21,
- "name": "encoder",
- },
- {
- "container": "PatchId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 22,
- "lineStart": 22,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 25,
- "lineStart": 25,
- "name": "ClaimId",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 29,
- "lineStart": 26,
- "name": "ClaimId",
- },
- {
- "container": "ClaimId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 27,
- "lineStart": 27,
- "name": "encoder",
- },
- {
- "container": "ClaimId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 28,
- "lineStart": 28,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 31,
- "lineStart": 31,
- "name": "ConflictId",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 35,
- "lineStart": 32,
- "name": "ConflictId",
- },
- {
- "container": "ConflictId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 33,
- "lineStart": 33,
- "name": "encoder",
- },
- {
- "container": "ConflictId",
- "kind": "function",
- "language": "scala",
- "lineEnd": 34,
- "lineStart": 34,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 37,
- "lineStart": 37,
- "name": "Rev",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 41,
- "lineStart": 38,
- "name": "Rev",
- },
- {
- "container": "Rev",
- "kind": "function",
- "language": "scala",
- "lineEnd": 39,
- "lineStart": 39,
- "name": "encoder",
- },
- {
- "container": "Rev",
- "kind": "function",
- "language": "scala",
- "lineEnd": 40,
- "lineStart": 40,
- "name": "decoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 3,
- "lineStart": 3,
- "name": "java.util.UUID",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 5,
- "lineStart": 5,
- "name": "io.circe.Decoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 5,
- "lineStart": 5,
- "name": "io.circe.Encoder",
- },
- ],
- "filePath": "memory-layer/src/main/scala/ix/memory/model/Identifiers.scala",
- "fileRole": {
- "role": "production",
- "role_confidence": 0.5,
- "role_signals": [],
- },
- "language": "scala",
- "relationships": [
- {
- "dstName": "NodeId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "NodeId",
- },
- {
- "dstName": "NodeId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "NodeId",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "NodeId",
- },
- {
- "dstName": "EdgeId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "EdgeId",
- },
- {
- "dstName": "EdgeId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "EdgeId",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "EdgeId",
- },
- {
- "dstName": "PatchId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "PatchId",
- },
- {
- "dstName": "PatchId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "PatchId",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "PatchId",
- },
- {
- "dstName": "ClaimId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "ClaimId",
- },
- {
- "dstName": "ClaimId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "ClaimId",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "ClaimId",
- },
- {
- "dstName": "ConflictId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "ConflictId",
- },
- {
- "dstName": "ConflictId",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "ConflictId",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "ConflictId",
- },
- {
- "dstName": "Rev",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "AnyVal",
- "predicate": "EXTENDS",
- "srcName": "Rev",
- },
- {
- "dstName": "Rev",
- "predicate": "CONTAINS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "Rev",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "Rev",
- },
- {
- "dstName": "java.util.UUID",
- "predicate": "IMPORTS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "io.circe.Decoder",
- "predicate": "IMPORTS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "io.circe.Encoder",
- "predicate": "IMPORTS",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "UUID",
- "predicate": "REFERENCES",
- "srcName": "NodeId",
- },
- {
- "dstName": "NodeId",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "NodeId.encoder",
- },
- {
- "dstName": "NodeId",
- "predicate": "CALLS",
- "srcName": "NodeId.decoder",
- },
- {
- "dstName": "UUID",
- "predicate": "REFERENCES",
- "srcName": "EdgeId",
- },
- {
- "dstName": "EdgeId",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "EdgeId.encoder",
- },
- {
- "dstName": "EdgeId",
- "predicate": "CALLS",
- "srcName": "EdgeId.decoder",
- },
- {
- "dstName": "UUID",
- "predicate": "REFERENCES",
- "srcName": "PatchId",
- },
- {
- "dstName": "PatchId",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "PatchId.encoder",
- },
- {
- "dstName": "PatchId",
- "predicate": "CALLS",
- "srcName": "PatchId.decoder",
- },
- {
- "dstName": "UUID",
- "predicate": "REFERENCES",
- "srcName": "ClaimId",
- },
- {
- "dstName": "ClaimId",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "ClaimId.encoder",
- },
- {
- "dstName": "ClaimId",
- "predicate": "CALLS",
- "srcName": "ClaimId.decoder",
- },
- {
- "dstName": "UUID",
- "predicate": "REFERENCES",
- "srcName": "ConflictId",
- },
- {
- "dstName": "ConflictId",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "ConflictId.encoder",
- },
- {
- "dstName": "ConflictId",
- "predicate": "CALLS",
- "srcName": "ConflictId.decoder",
- },
- {
- "dstName": "Rev",
- "predicate": "REFERENCES",
- "srcName": "Identifiers.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "Rev.encoder",
- },
- {
- "dstName": "Rev",
- "predicate": "CALLS",
- "srcName": "Rev.decoder",
- },
- ],
-}
-`;
-
-exports[`parseFile snapshots > snapshots Node.scala 1`] = `
-{
- "entities": [
- {
- "kind": "file",
- "language": "scala",
- "lineEnd": 97,
- "lineStart": 1,
- "name": "Node.scala",
- },
- {
- "container": undefined,
- "kind": "interface",
- "language": "scala",
- "lineEnd": 8,
- "lineStart": 8,
- "name": "NodeKind",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 77,
- "lineStart": 9,
- "name": "NodeKind",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 10,
- "lineStart": 10,
- "name": "Module",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 11,
- "lineStart": 11,
- "name": "File",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 12,
- "lineStart": 12,
- "name": "Class",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 13,
- "lineStart": 13,
- "name": "Function",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 14,
- "lineStart": 14,
- "name": "Variable",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 15,
- "lineStart": 15,
- "name": "Config",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 16,
- "lineStart": 16,
- "name": "ConfigEntry",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 17,
- "lineStart": 17,
- "name": "Service",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 18,
- "lineStart": 18,
- "name": "Endpoint",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 19,
- "lineStart": 19,
- "name": "Intent",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 20,
- "lineStart": 20,
- "name": "Decision",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 21,
- "lineStart": 21,
- "name": "Doc",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 22,
- "lineStart": 22,
- "name": "Object",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 23,
- "lineStart": 23,
- "name": "Trait",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 24,
- "lineStart": 24,
- "name": "Interface",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 25,
- "lineStart": 25,
- "name": "Method",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 26,
- "lineStart": 26,
- "name": "Plan",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 27,
- "lineStart": 27,
- "name": "Task",
- },
- {
- "container": "NodeKind",
- "kind": "class",
- "language": "scala",
- "lineEnd": 28,
- "lineStart": 28,
- "name": "Bug",
- },
- {
- "container": "NodeKind",
- "kind": "function",
- "language": "scala",
- "lineEnd": 50,
- "lineStart": 30,
- "name": "nameMap",
- },
- {
- "container": "NodeKind",
- "kind": "function",
- "language": "scala",
- "lineEnd": 72,
- "lineStart": 52,
- "name": "encoder",
- },
- {
- "container": "NodeKind",
- "kind": "function",
- "language": "scala",
- "lineEnd": 76,
- "lineStart": 74,
- "name": "decoder",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 89,
- "lineStart": 79,
- "name": "GraphNode",
- },
- {
- "container": undefined,
- "kind": "class",
- "language": "scala",
- "lineEnd": 96,
- "lineStart": 91,
- "name": "GraphNode",
- },
- {
- "container": "GraphNode",
- "kind": "function",
- "language": "scala",
- "lineEnd": 94,
- "lineStart": 94,
- "name": "encoder",
- },
- {
- "container": "GraphNode",
- "kind": "function",
- "language": "scala",
- "lineEnd": 95,
- "lineStart": 95,
- "name": "decoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 3,
- "lineStart": 3,
- "name": "java.time.Instant",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 5,
- "lineStart": 5,
- "name": "io.circe.Decoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 5,
- "lineStart": 5,
- "name": "io.circe.Encoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 5,
- "lineStart": 5,
- "name": "io.circe.Json",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 6,
- "lineStart": 6,
- "name": "io.circe.generic.semiauto.deriveDecoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 6,
- "lineStart": 6,
- "name": "io.circe.generic.semiauto.deriveEncoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 92,
- "lineStart": 92,
- "name": "Provenance.instantEncoder",
- },
- {
- "kind": "module",
- "language": "scala",
- "lineEnd": 92,
- "lineStart": 92,
- "name": "Provenance.instantDecoder",
- },
- ],
- "filePath": "memory-layer/src/main/scala/ix/memory/model/Node.scala",
- "fileRole": {
- "role": "production",
- "role_confidence": 0.5,
- "role_signals": [],
- },
- "language": "scala",
- "relationships": [
- {
- "dstName": "NodeKind",
- "predicate": "CONTAINS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "NodeKind",
- "predicate": "CONTAINS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "Module",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Module",
- },
- {
- "dstName": "File",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "File",
- },
- {
- "dstName": "Class",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Class",
- },
- {
- "dstName": "Function",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Function",
- },
- {
- "dstName": "Variable",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Variable",
- },
- {
- "dstName": "Config",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Config",
- },
- {
- "dstName": "ConfigEntry",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "ConfigEntry",
- },
- {
- "dstName": "Service",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Service",
- },
- {
- "dstName": "Endpoint",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Endpoint",
- },
- {
- "dstName": "Intent",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Intent",
- },
- {
- "dstName": "Decision",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Decision",
- },
- {
- "dstName": "Doc",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Doc",
- },
- {
- "dstName": "Object",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Object",
- },
- {
- "dstName": "Trait",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Trait",
- },
- {
- "dstName": "Interface",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Interface",
- },
- {
- "dstName": "Method",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Method",
- },
- {
- "dstName": "Plan",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Plan",
- },
- {
- "dstName": "Task",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Task",
- },
- {
- "dstName": "Bug",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "NodeKind",
- "predicate": "EXTENDS",
- "srcName": "Bug",
- },
- {
- "dstName": "nameMap",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "NodeKind",
- },
- {
- "dstName": "GraphNode",
- "predicate": "CONTAINS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "GraphNode",
- "predicate": "CONTAINS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "encoder",
- "predicate": "CONTAINS",
- "srcName": "GraphNode",
- },
- {
- "dstName": "decoder",
- "predicate": "CONTAINS",
- "srcName": "GraphNode",
- },
- {
- "dstName": "java.time.Instant",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "io.circe.Decoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "io.circe.Encoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "io.circe.Json",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "io.circe.generic.semiauto.deriveDecoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "io.circe.generic.semiauto.deriveEncoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "NodeKind",
- "predicate": "REFERENCES",
- "srcName": "Node.scala",
- },
- {
- "dstName": "contramap",
- "predicate": "CALLS",
- "srcName": "NodeKind.encoder",
- },
- {
- "dstName": "emap",
- "predicate": "CALLS",
- "srcName": "NodeKind.decoder",
- },
- {
- "dstName": "get",
- "predicate": "CALLS",
- "srcName": "NodeKind.decoder",
- },
- {
- "dstName": "toRight",
- "predicate": "CALLS",
- "srcName": "NodeKind.decoder",
- },
- {
- "dstName": "NodeId",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "NodeKind",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "Json",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "Provenance",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "Rev",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "Instant",
- "predicate": "REFERENCES",
- "srcName": "GraphNode",
- },
- {
- "dstName": "Provenance.instantEncoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "Provenance.instantDecoder",
- "predicate": "IMPORTS",
- "srcName": "Node.scala",
- },
- {
- "dstName": "GraphNode",
- "predicate": "REFERENCES",
- "srcName": "Node.scala",
- },
- ],
-}
-`;
-
-exports[`parseFile snapshots > snapshots doctor.ts 1`] = `
-{
- "entities": [
- {
- "kind": "file",
- "language": "typescript",
- "lineEnd": 99,
- "lineStart": 1,
- "name": "doctor.ts",
- },
- {
- "container": undefined,
- "kind": "interface",
- "language": "typescript",
- "lineEnd": 9,
- "lineStart": 6,
- "name": "Check",
- },
- {
- "container": undefined,
- "kind": "function",
- "language": "typescript",
- "lineEnd": 98,
- "lineStart": 11,
- "name": "registerDoctorCommand",
- },
- {
- "kind": "module",
- "language": "typescript",
- "lineEnd": 1,
- "lineStart": 1,
- "name": "commander",
- },
- {
- "kind": "module",
- "language": "typescript",
- "lineEnd": 2,
- "lineStart": 2,
- "name": "chalk",
- },
- {
- "kind": "module",
- "language": "typescript",
- "lineEnd": 3,
- "lineStart": 3,
- "name": "api.js",
- },
- {
- "kind": "module",
- "language": "typescript",
- "lineEnd": 4,
- "lineStart": 4,
- "name": "config.js",
- },
- ],
- "filePath": "ix-cli/src/cli/commands/doctor.ts",
- "fileRole": {
- "role": "production",
- "role_confidence": 0.5,
- "role_signals": [],
- },
- "language": "typescript",
- "relationships": [
- {
- "dstName": "Check",
- "predicate": "CONTAINS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "registerDoctorCommand",
- "predicate": "CONTAINS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "commander",
- "predicate": "IMPORTS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "chalk",
- "predicate": "IMPORTS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "api.js",
- "predicate": "IMPORTS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "config.js",
- "predicate": "IMPORTS",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "Command",
- "predicate": "REFERENCES",
- "srcName": "doctor.ts",
- },
- {
- "dstName": "command",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "description",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "option",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "action",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "getEndpoint",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "IxClient",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "health",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "stats",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "conflicts",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "isArray",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "run",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "push",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "every",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "stringify",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "bold",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "green",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "red",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- {
- "dstName": "dim",
- "predicate": "CALLS",
- "srcName": "registerDoctorCommand",
- },
- ],
-}
-`;
diff --git a/core-ingestion/src/__tests__/chunks.test.ts b/core-ingestion/src/__tests__/chunks.test.ts
new file mode 100644
index 00000000..7f190b46
--- /dev/null
+++ b/core-ingestion/src/__tests__/chunks.test.ts
@@ -0,0 +1,118 @@
+import { describe, expect, it } from 'vitest';
+import { parseFile, type ParsedChunk } from '../index.js';
+
+// ---------------------------------------------------------------------------
+// Helper to parse a synthetic source string as a given file extension
+// ---------------------------------------------------------------------------
+function parse(source: string, ext = '.ts') {
+ const filePath = `/repo/fixture${ext}`;
+ return parseFile(filePath, source);
+}
+
+describe('Chunk extraction — M1 acceptance criteria', () => {
+ it('a file with 3 top-level functions produces 3 chunk nodes', () => {
+ const src = `
+function alpha() { return 1; }
+function beta() { return 2; }
+function gamma() { return 3; }
+`.trim();
+ const result = parse(src);
+ expect(result).not.toBeNull();
+ const chunks = result!.chunks.filter(c => c.name !== null);
+ expect(chunks.length).toBe(3);
+ expect(chunks.map(c => c.name)).toEqual(expect.arrayContaining(['alpha', 'beta', 'gamma']));
+ expect(chunks.every(c => c.chunkKind === 'function')).toBe(true);
+ });
+
+ it('a class file produces at least a class chunk and/or method chunks', () => {
+ const src = `
+class MyService {
+ greet() { return 'hello'; }
+ farewell() { return 'bye'; }
+}
+`.trim();
+ const result = parse(src);
+ expect(result).not.toBeNull();
+ const chunks = result!.chunks.filter(c => c.name !== null);
+ // Must have at least the class chunk or method chunks
+ expect(chunks.length).toBeGreaterThanOrEqual(1);
+ const names = chunks.map(c => c.name);
+ expect(names.some(n => n === 'MyService' || n === 'greet' || n === 'farewell')).toBe(true);
+ });
+
+ it('a file with no extractable semantic units produces one file_body chunk', () => {
+ // A comment-only file has no definitions
+ const src = `// This file has no definitions\nconst x = 1;\n`;
+ const result = parseFile('/repo/fixture.txt', src);
+ // .txt is not a supported language, so we test with an empty TS file
+ const emptyTs = parseFile('/repo/empty.ts', '// empty\n');
+ // Even if parse fails gracefully, test the fallback chunk on a real parsed file
+ if (emptyTs) {
+ expect(emptyTs.chunks.length).toBeGreaterThanOrEqual(1);
+ if (emptyTs.chunks.every(c => c.name === null)) {
+ expect(emptyTs.chunks[0].chunkKind).toBe('file_body');
+ }
+ }
+ });
+
+ it('each chunk has required fields: startByte, endByte, contentHash, language', () => {
+ const src = `function foo() { return 42; }`;
+ const result = parse(src);
+ expect(result).not.toBeNull();
+ for (const chunk of result!.chunks) {
+ expect(typeof chunk.startByte).toBe('number');
+ expect(typeof chunk.endByte).toBe('number');
+ expect(chunk.endByte).toBeGreaterThan(chunk.startByte);
+ expect(typeof chunk.contentHash).toBe('string');
+ expect(chunk.contentHash.length).toBeGreaterThan(0);
+ expect(typeof chunk.language).toBe('string');
+ }
+ });
+
+ it('chunks have matching lineStart/lineEnd within file bounds', () => {
+ const src = `function a() {}\nfunction b() {}\n`;
+ const result = parse(src);
+ expect(result).not.toBeNull();
+ const lines = src.split('\n').length;
+ for (const chunk of result!.chunks) {
+ expect(chunk.lineStart).toBeGreaterThanOrEqual(1);
+ expect(chunk.lineEnd).toBeLessThanOrEqual(lines);
+ }
+ });
+
+ it('method chunks carry a container field matching their enclosing class', () => {
+ const src = `
+class Calculator {
+ add(a: number, b: number) { return a + b; }
+}
+`.trim();
+ const result = parse(src);
+ expect(result).not.toBeNull();
+ const methodChunk = result!.chunks.find(c => c.name === 'add');
+ if (methodChunk) {
+ expect(methodChunk.container).toBe('Calculator');
+ }
+ });
+
+ it('re-parsing the same file produces identical chunks (stable IDs)', () => {
+ const src = `function stableFunc() { return true; }`;
+ const r1 = parse(src);
+ const r2 = parse(src);
+ expect(r1).not.toBeNull();
+ expect(r2).not.toBeNull();
+ expect(r1!.chunks).toEqual(r2!.chunks);
+ });
+
+ it('Scala file produces chunks for class/trait/def definitions', () => {
+ const src = `
+class GraphService {
+ def findNode(id: String): Option[Node] = None
+ def saveNode(node: Node): Unit = ()
+}
+`.trim();
+ const result = parseFile('/repo/fixture.scala', src);
+ expect(result).not.toBeNull();
+ const chunks = result!.chunks.filter(c => c.name !== null);
+ expect(chunks.length).toBeGreaterThanOrEqual(1);
+ });
+});
diff --git a/core-ingestion/src/__tests__/resolveEdges.property.test.ts b/core-ingestion/src/__tests__/resolveEdges.property.test.ts
index ac34f372..578cc8fd 100644
--- a/core-ingestion/src/__tests__/resolveEdges.property.test.ts
+++ b/core-ingestion/src/__tests__/resolveEdges.property.test.ts
@@ -38,6 +38,7 @@ describe('resolveEdges property invariants', () => {
makeEntity(def.symbol, def.language),
...(def.includeCall ? [makeEntity(`caller${index}`, def.language)] : []),
],
+ chunks: [],
relationships: def.includeCall
? [{ srcName: `caller${index}`, dstName: def.symbol, predicate: 'CALLS' }]
: [],
@@ -69,6 +70,7 @@ describe('resolveEdges property invariants', () => {
{ name: 'caller.ts', kind: 'file', lineStart: 1, lineEnd: 1, language: SupportedLanguages.TypeScript },
makeEntity('caller', SupportedLanguages.TypeScript),
],
+ chunks: [],
relationships: [{ srcName: 'caller', dstName: 'missing', predicate: 'CALLS' }],
fileRole: defaultFileRole,
},
@@ -79,6 +81,7 @@ describe('resolveEdges property invariants', () => {
{ name: 'other.scala', kind: 'file', lineStart: 1, lineEnd: 1, language: SupportedLanguages.Scala },
makeEntity('missing', SupportedLanguages.Scala),
],
+ chunks: [],
relationships: [],
fileRole: defaultFileRole,
},
@@ -94,6 +97,7 @@ describe('resolveEdges property invariants', () => {
{ name: 'caller.ts', kind: 'file', lineStart: 1, lineEnd: 1, language: SupportedLanguages.TypeScript },
makeEntity('caller', SupportedLanguages.TypeScript),
],
+ chunks: [],
relationships: [{ srcName: 'caller', dstName: 'run', predicate: 'CALLS' }],
fileRole: defaultFileRole,
},
@@ -104,6 +108,7 @@ describe('resolveEdges property invariants', () => {
{ name: 'a.ts', kind: 'file', lineStart: 1, lineEnd: 1, language: SupportedLanguages.TypeScript },
makeEntity('run', SupportedLanguages.TypeScript),
],
+ chunks: [],
relationships: [],
fileRole: defaultFileRole,
},
@@ -114,6 +119,7 @@ describe('resolveEdges property invariants', () => {
{ name: 'b.ts', kind: 'file', lineStart: 1, lineEnd: 1, language: SupportedLanguages.TypeScript },
makeEntity('run', SupportedLanguages.TypeScript),
],
+ chunks: [],
relationships: [],
fileRole: defaultFileRole,
},
diff --git a/core-ingestion/src/__tests__/resolveEdges.test.ts b/core-ingestion/src/__tests__/resolveEdges.test.ts
index c6ee50a7..cb1e3961 100644
--- a/core-ingestion/src/__tests__/resolveEdges.test.ts
+++ b/core-ingestion/src/__tests__/resolveEdges.test.ts
@@ -34,6 +34,7 @@ function fileResult(
{ name: filePath.split(/[\\/]/).pop() ?? filePath, kind: 'file', lineStart: 1, lineEnd: 1, language },
...entities,
],
+ chunks: [],
relationships,
fileRole: defaultFileRole,
};
diff --git a/core-ingestion/src/index.ts b/core-ingestion/src/index.ts
index a541d383..93d090b4 100644
--- a/core-ingestion/src/index.ts
+++ b/core-ingestion/src/index.ts
@@ -1,4 +1,5 @@
import * as nodePath from 'node:path';
+import * as crypto from 'node:crypto';
// @ts-ignore — tree-sitter has no bundled types
import Parser from 'tree-sitter';
// @ts-ignore
@@ -45,6 +46,23 @@ export interface ParsedEntity {
container?: string;
}
+/** A semantic code span extracted from the AST. The primary LLM retrieval unit. */
+export interface ParsedChunk {
+ /** Semantic name of the chunk (function/class/trait name), or null for file_body. */
+ name: string | null;
+ /** Chunk kind: "function" | "method" | "class" | "interface" | "trait" | "module_block" | "file_body" */
+ chunkKind: string;
+ lineStart: number;
+ lineEnd: number;
+ startByte: number;
+ endByte: number;
+ /** SHA-256 of the chunk source text for change detection and stable identity. */
+ contentHash: string;
+ language: string;
+ /** Name of the directly enclosing class/trait/interface, if any. */
+ container?: string;
+}
+
export interface ParsedRelationship {
srcName: string;
dstName: string;
@@ -55,6 +73,7 @@ export interface FileParseResult {
filePath: string;
language: SupportedLanguages;
entities: ParsedEntity[];
+ chunks: ParsedChunk[];
relationships: ParsedRelationship[];
fileRole: RoleClassification;
}
@@ -160,17 +179,18 @@ export function isGrammarSupported(filePath: string): boolean {
// ---------------------------------------------------------------------------
let _parser: Parser | null = null;
+const _queryCache = new Map<SupportedLanguages | 'tsx', { grammar: any; query: any }>();
function getParser(): Parser {
if (!_parser) _parser = new Parser();
return _parser;
}
-// Per-process cache for compiled tree-sitter queries, keyed by language.
-// Avoids recompiling the same query string for every file of the same language.
-const _queryCache = new Map<SupportedLanguages | 'tsx', { grammar: any; query: any }>();
-
-function getCachedQuery(language: SupportedLanguages | 'tsx', grammar: any, querySource: string): any {
+function getCachedQuery(
+ language: SupportedLanguages | 'tsx',
+ grammar: any,
+ querySource: string,
+): any {
const cached = _queryCache.get(language);
if (cached && cached.grammar === grammar) return cached.query;
const query = new Parser.Query(grammar, querySource);
@@ -203,14 +223,26 @@ export function parseFile(filePath: string, source: string): FileParseResult | n
const matches = query.matches(tree.rootNode);
const fileName = nodePath.basename(filePath);
- // Count newlines instead of allocating a full string array
- let lineCount = 1;
- for (let i = 0; i < source.length; i++) { if (source.charCodeAt(i) === 10) lineCount++; }
+ let sourceLineCount = 1;
+ for (let i = 0; i < source.length; i++) {
+ if (source.charCodeAt(i) === 10) sourceLineCount++;
+ }
const entities: ParsedEntity[] = [
- { name: fileName, kind: 'file', lineStart: 1, lineEnd: lineCount, language },
+ { name: fileName, kind: 'file', lineStart: 1, lineEnd: sourceLineCount, language },
];
+ const chunks: ParsedChunk[] = [];
const relationships: ParsedRelationship[] = [];
+ const pendingChunks: Array<{
+ name: string;
+ chunkKind: string;
+ lineStart: number;
+ lineEnd: number;
+ startByte: number;
+ endByte: number;
+ language: SupportedLanguages;
+ container?: string;
+ }> = [];
// Track class ranges for containment: [name, startLine, endLine]
const classRanges: Array<{ name: string; start: number; end: number }> = [];
@@ -236,6 +268,8 @@ export function parseFile(filePath: string, source: string): FileParseResult | n
const defNode = defCapture.node;
const lineStart = defNode.startPosition.row + 1;
const lineEnd = defNode.endPosition.row + 1;
+ const startByte = defNode.startIndex;
+ const endByte = defNode.endIndex;
// Containment: file CONTAINS or class CONTAINS
const enclosing = findEnclosing(classRanges, lineStart, name);
@@ -249,6 +283,17 @@ export function parseFile(filePath: string, source: string): FileParseResult | n
container: enclosing ?? undefined,
});
+ pendingChunks.push({
+ name,
+ chunkKind: kind,
+ lineStart,
+ lineEnd,
+ startByte,
+ endByte,
+ language,
+ container: enclosing ?? undefined,
+ });
+
if (kind === 'class' || kind === 'interface' || kind === 'trait') {
classRanges.push({ name, start: lineStart, end: lineEnd });
}
@@ -404,7 +449,31 @@ export function parseFile(filePath: string, source: string): FileParseResult | n
}
}
- return { filePath, language, entities, relationships, fileRole: classifyFileRole(filePath, source) };
+ for (const pendingChunk of pendingChunks) {
+ const chunkText = source.slice(pendingChunk.startByte, pendingChunk.endByte);
+ const contentHash = crypto.createHash('sha256').update(chunkText).digest('hex').slice(0, 16);
+ chunks.push({
+ ...pendingChunk,
+ contentHash,
+ });
+ }
+
+ // Fallback: if no semantic chunks found, emit one file_body chunk covering the whole file
+ if (chunks.length === 0) {
+ const contentHash = crypto.createHash('sha256').update(source).digest('hex').slice(0, 16);
+ chunks.push({
+ name: null,
+ chunkKind: 'file_body',
+ lineStart: 1,
+ lineEnd: sourceLineCount,
+ startByte: 0,
+ endByte: source.length,
+ contentHash,
+ language,
+ });
+ }
+
+ return { filePath, language, entities, chunks, relationships, fileRole: classifyFileRole(filePath, source) };
} catch {
return null;
}
@@ -485,37 +554,7 @@ function bestQKey(
* - Unambiguous class method: dstQualifiedKey === 'ClassName.method'
* - Ambiguous (two entities share the same plain name): edge not emitted
*/
-/** Stats collected during resolution (exposed for debug instrumentation). */
-export interface ResolveStats {
- importLookups: number;
- transitiveLookups: number;
- globalFallbacks: number;
- globalCandidateTotal: number;
- resolvedImport: number;
- resolvedTransitive: number;
- resolvedGlobal: number;
- resolvedQualifier: number;
- skippedSameFile: number;
- skippedAmbiguous: number;
-}
-
-export function resolveEdges(results: FileParseResult[]): ResolvedEdge[];
-export function resolveEdges(results: FileParseResult[], statsOut: ResolveStats): ResolvedEdge[];
-export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats): ResolvedEdge[] {
- const stats: ResolveStats = statsOut ?? {
- importLookups: 0, transitiveLookups: 0, globalFallbacks: 0,
- globalCandidateTotal: 0, resolvedImport: 0, resolvedTransitive: 0,
- resolvedGlobal: 0, resolvedQualifier: 0, skippedSameFile: 0, skippedAmbiguous: 0,
- };
-
- // ── Index structures (built once) ──────────────────────────────────
-
- // Direct file lookup: filePath → FileParseResult (replaces results.find())
- const resultByPath = new Map<string, FileParseResult>();
- for (const r of results) {
- resultByPath.set(r.filePath, r);
- }
-
+export function resolveEdges(results: FileParseResult[]): ResolvedEdge[] {
// fileQKeys: filePath → (plainName → qualifiedKeys[])
// Mirrors the entityQKey computation in buildPatch so nodeIds match exactly.
const fileQKeys = new Map<string, Map<string, string[]>>();
@@ -537,20 +576,16 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
fileHasSymbol.set(fp, new Set(qkMap.keys()));
}
- // Precompute language per file (avoids calling languageFromPath in hot loops)
- const fileLanguage = new Map<string, SupportedLanguages | null>();
- for (const r of results) {
- fileLanguage.set(r.filePath, r.language);
- }
+ // resultsByPath: O(1) lookup replacing results.find() in transitive import loop
+ const resultsByPath = new Map<string, FileParseResult>(results.map(r => [r.filePath, r]));
- // Inverted symbol index: symbolName → Array<{filePath, language}>
- // Replaces the O(N) global fallback scan with O(candidates) lookup.
- const symbolToFiles = new Map<string, Array<string>>();
+ // symbolToFiles: plainName → filePath[] — replaces O(F) full-scan in tier-3 and qualifier fallback
+ const symbolToFiles = new Map<string, string[]>();
for (const [fp, symbols] of fileHasSymbol) {
for (const sym of symbols) {
- let arr = symbolToFiles.get(sym);
- if (!arr) { arr = []; symbolToFiles.set(sym, arr); }
- arr.push(fp);
+ const list = symbolToFiles.get(sym) ?? [];
+ list.push(fp);
+ symbolToFiles.set(sym, list);
}
}
@@ -596,8 +631,6 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
}
}
- // ── Helpers ────────────────────────────────────────────────────────
-
/** Resolve a module name to matching file paths in the batch. */
function modNameToFiles(modName: string, excludeFp: string): string[] {
const fps: string[] = [];
@@ -630,14 +663,11 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
return fps;
}
- // ── Main resolution loop ───────────────────────────────────────────
-
const resolved: ResolvedEdge[] = [];
for (const result of results) {
const srcFilePath = result.filePath;
const srcSymbols = fileHasSymbol.get(srcFilePath)!;
- const srcLanguage = result.language;
// Build the set of file paths this file explicitly imports.
const importedFilePaths = new Set<string>();
@@ -652,7 +682,7 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
// Handles re-exports: baz.ts → index.ts (re-exports from bar.ts) → bar.ts
const transitiveFilePaths = new Set<string>();
for (const fp of importedFilePaths) {
- const fpResult = resultByPath.get(fp); // O(1) instead of results.find()
+ const fpResult = resultsByPath.get(fp);
if (!fpResult) continue;
for (const rel of fpResult.relationships) {
if (rel.predicate !== 'IMPORTS') continue;
@@ -686,7 +716,6 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
const dstQualifiedKey = bestQKey(fileQKeys, fp, entityName);
if (dstQualifiedKey === null) continue;
resolved.push({ srcFilePath, srcName: rel.srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: 'IMPORTS', confidence: 0.9 });
- stats.resolvedImport++;
continue;
}
@@ -714,23 +743,18 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
if (dstQualifiedKey !== null) {
// dstName must match rel.dstName so buildPatchWithResolution can look it up
resolved.push({ srcFilePath, srcName, dstFilePath: qfp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.9 });
- stats.resolvedQualifier++;
}
}
continue;
}
- // Global qualifier fallback — use inverted index instead of scanning all files
- const qualCandidates = symbolToFiles.get(qualifierPart) ?? [];
- const qualGlobalMatches = qualCandidates.filter(fp => fp !== srcFilePath);
- stats.globalFallbacks++;
- stats.globalCandidateTotal += qualGlobalMatches.length;
+ // Global qualifier fallback — use symbolToFiles index instead of full scan
+ const qualGlobalMatches = (symbolToFiles.get(qualifierPart) ?? []).filter(fp => fp !== srcFilePath);
if (qualGlobalMatches.length === 1) {
const qfp = qualGlobalMatches[0];
if (fileHasSymbol.get(qfp)?.has(memberPart)) {
const dstQualifiedKey = bestQKey(fileQKeys, qfp, memberPart);
if (dstQualifiedKey !== null) {
resolved.push({ srcFilePath, srcName, dstFilePath: qfp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.7 });
- stats.resolvedQualifier++;
}
}
}
@@ -739,10 +763,9 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
}
// Tier 1: same-file — already correct in buildPatch, skip here
- if (srcSymbols.has(dstName)) { stats.skippedSameFile++; continue; }
+ if (srcSymbols.has(dstName)) continue;
// Tier 2: import-scoped (confidence 0.9)
- stats.importLookups++;
const importMatches: string[] = [];
for (const fp of importedFilePaths) {
if (fileHasSymbol.get(fp)?.has(dstName)) importMatches.push(fp);
@@ -753,13 +776,11 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
const dstQualifiedKey = bestQKey(fileQKeys, fp, dstName);
if (dstQualifiedKey === null) continue; // ambiguous — do not emit bad nodeId
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.9 });
- stats.resolvedImport++;
continue;
}
- if (importMatches.length > 1) { stats.skippedAmbiguous++; continue; } // ambiguous file — do not emit
+ if (importMatches.length > 1) continue; // ambiguous file — do not emit
// Tier 2.5: transitive import-scoped (confidence 0.8) — one re-export hop away
- stats.transitiveLookups++;
const transitiveMatches: string[] = [];
for (const fp of transitiveFilePaths) {
if (fileHasSymbol.get(fp)?.has(dstName)) transitiveMatches.push(fp);
@@ -770,24 +791,21 @@ export function resolveEdges(results: FileParseResult[], statsOut?: ResolveStats
const dstQualifiedKey = bestQKey(fileQKeys, fp, dstName);
if (dstQualifiedKey === null) continue;
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.8 });
- stats.resolvedTransitive++;
continue;
}
- if (transitiveMatches.length > 1) { stats.skippedAmbiguous++; continue; } // ambiguous — do not emit
+ if (transitiveMatches.length > 1) continue; // ambiguous — do not emit
- // Tier 3: global fallback (confidence 0.5) — uses inverted symbol index
- // instead of scanning all files.
- stats.globalFallbacks++;
- const candidates = symbolToFiles.get(dstName) ?? [];
- const globalMatches = candidates.filter(fp => fp !== srcFilePath && fileLanguage.get(fp) === srcLanguage);
- stats.globalCandidateTotal += globalMatches.length;
+ // Tier 3: global fallback (confidence 0.5) — exactly one other file defines it
+ // Only match files in the same language as the caller to avoid cross-language false positives.
+ const srcLanguage = result.language;
+ const globalMatches = (symbolToFiles.get(dstName) ?? [])
+ .filter(fp => fp !== srcFilePath && languageFromPath(fp) === srcLanguage);
if (globalMatches.length === 1) {
const fp = globalMatches[0];
const dstQualifiedKey = bestQKey(fileQKeys, fp, dstName);
if (dstQualifiedKey === null) continue; // ambiguous — do not emit bad nodeId
resolved.push({ srcFilePath, srcName, dstFilePath: fp, dstName, dstQualifiedKey, predicate: rel.predicate, confidence: 0.5 });
- stats.resolvedGlobal++;
}
// 0 or >1 global matches — leave as dangling edge, do not emit
}
diff --git a/core-ingestion/src/parse-worker.ts b/core-ingestion/src/parse-worker.ts
new file mode 100644
index 00000000..3485d01e
--- /dev/null
+++ b/core-ingestion/src/parse-worker.ts
@@ -0,0 +1,19 @@
+/**
+ * Worker thread entry point for parallel file parsing.
+ * Each worker maintains its own Parser singleton (safe — module state is per-thread).
+ * Receives: { filePath: string, source: string }
+ * Posts: { ok: true, result: FileParseResult } | { ok: false }
+ */
+import { parentPort } from 'node:worker_threads';
+import { parseFile } from './index.js';
+
+if (!parentPort) throw new Error('parse-worker must run inside a worker thread');
+
+parentPort.on('message', ({ filePath, source }: { filePath: string; source: string }) => {
+ try {
+ const result = parseFile(filePath, source);
+ parentPort!.postMessage({ ok: result !== null, result: result ?? null });
+ } catch {
+ parentPort!.postMessage({ ok: false, result: null });
+ }
+});
diff --git a/core-ingestion/src/patch-builder.ts b/core-ingestion/src/patch-builder.ts
index 09b5941b..d06f6553 100644
--- a/core-ingestion/src/patch-builder.ts
+++ b/core-ingestion/src/patch-builder.ts
@@ -30,6 +30,11 @@ function edgeId(filePath: string, src: string, dst: string, predicate: string):
return deterministicId(`${normalizePath(filePath)}:${src}:${dst}:${predicate}`);
}
+/** Deterministic ID for a chunk — keyed on file + kind + name + start line to survive minor edits. */
+function chunkId(filePath: string, chunkKind: string, name: string | null, startLine: number): string {
+ return deterministicId(`${normalizePath(filePath)}:chunk:${chunkKind}:${name ?? 'file_body'}:${startLine}`);
+}
+
// ---------------------------------------------------------------------------
// Source type from file extension
// ---------------------------------------------------------------------------
@@ -42,11 +47,11 @@ function sourceType(filePath: string): string {
}
export function extractorName(): string {
- return `tree-sitter/1.12`;
+ return `tree-sitter/1.13`;
}
/** Previous extractor versions — their patches are superseded when re-ingesting. */
-export const PREVIOUS_EXTRACTORS = ['tree-sitter/1.11', 'tree-sitter/1.10', 'tree-sitter/1.9', 'tree-sitter/1.8', 'tree-sitter/1.7', 'tree-sitter/1.6', 'tree-sitter/1.5', 'tree-sitter/1.4', 'tree-sitter/1.3', 'tree-sitter/1.2', 'tree-sitter/1.1'];
+export const PREVIOUS_EXTRACTORS = ['tree-sitter/1.12', 'tree-sitter/1.11', 'tree-sitter/1.10', 'tree-sitter/1.9', 'tree-sitter/1.8', 'tree-sitter/1.7', 'tree-sitter/1.6', 'tree-sitter/1.5', 'tree-sitter/1.4', 'tree-sitter/1.3', 'tree-sitter/1.2', 'tree-sitter/1.1'];
/** Compute a patchId for a (filePath, sourceHash, extractorVersion) triple. */
function computePatchId(filePath: string, sourceHash: string, extractor: string): string {
@@ -67,7 +72,7 @@ export function buildPatch(
sourceHash: string,
previousSourceHash?: string
): GraphPatchPayload {
- const { filePath, entities, relationships } = result;
+ const { filePath, entities, chunks, relationships } = result;
const ops: PatchOp[] = [];
// Build a qualified-key map so that same-named entities in different
@@ -127,6 +132,73 @@ export function buildPatch(
}
}
+ // UpsertNode + edges for each chunk
+ const fileNodeId = nodeId(filePath, entities.find(e => e.kind === 'file')?.name ?? filePath);
+ for (const chunk of chunks) {
+ const cid = chunkId(filePath, chunk.chunkKind, chunk.name, chunk.lineStart);
+ const chunkName = chunk.name ?? `file_body:${chunk.lineStart}`;
+ ops.push({
+ type: 'UpsertNode',
+ id: cid,
+ kind: 'chunk',
+ name: chunkName,
+ attrs: {
+ file_uri: filePath,
+ language: chunk.language,
+ chunk_kind: chunk.chunkKind,
+ start_line: chunk.lineStart,
+ end_line: chunk.lineEnd,
+ start_byte: chunk.startByte,
+ end_byte: chunk.endByte,
+ content_hash: chunk.contentHash,
+ parser_version: extractorName(),
+ },
+ });
+ // File -[CONTAINS]-> Chunk
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, 'file', chunkName, 'CONTAINS_CHUNK'),
+ src: fileNodeId,
+ dst: cid,
+ predicate: 'CONTAINS_CHUNK',
+ attrs: {},
+ });
+ // Chunk -[DEFINES]-> Symbol (only for named chunks)
+ if (chunk.name !== null) {
+ const symbolKey = chunk.container ? `${chunk.container}.${chunk.name}` : chunk.name;
+ const symbolNid = nodeId(filePath, symbolKey);
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, chunkName, symbolKey, 'DEFINES'),
+ src: cid,
+ dst: symbolNid,
+ predicate: 'DEFINES',
+ attrs: {},
+ });
+ }
+ }
+
+ // NEXT edges for source-order chunk adjacency
+ for (let i = 0; i + 1 < chunks.length; i++) {
+ const a = chunks[i];
+ const b = chunks[i + 1];
+ // Only link top-level chunks (no container) to avoid intra-class ordering noise
+ if (a.container == null && b.container == null) {
+ const aid = chunkId(filePath, a.chunkKind, a.name, a.lineStart);
+ const bid = chunkId(filePath, b.chunkKind, b.name, b.lineStart);
+ const aName = a.name ?? `file_body:${a.lineStart}`;
+ const bName = b.name ?? `file_body:${b.lineStart}`;
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, aName, bName, 'NEXT'),
+ src: aid,
+ dst: bid,
+ predicate: 'NEXT',
+ attrs: {},
+ });
+ }
+ }
+
// UpsertEdge for each relationship
for (const r of relationships) {
// For CONTAINS edges, srcName is the container of dstName — use that to disambiguate.
@@ -208,7 +280,7 @@ export function buildPatchWithResolution(
});
}
- const { filePath, entities, relationships } = result;
+ const { filePath, entities, chunks, relationships } = result;
const ops: PatchOp[] = [];
const entityQKey = new Map<ParsedEntity, string>();
@@ -252,6 +324,69 @@ export function buildPatchWithResolution(
}
}
+ // UpsertNode + edges for each chunk (same logic as buildPatch)
+ const fileNodeId2 = nodeId(filePath, entities.find(e => e.kind === 'file')?.name ?? filePath);
+ for (const chunk of chunks) {
+ const cid = chunkId(filePath, chunk.chunkKind, chunk.name, chunk.lineStart);
+ const chunkName = chunk.name ?? `file_body:${chunk.lineStart}`;
+ ops.push({
+ type: 'UpsertNode',
+ id: cid,
+ kind: 'chunk',
+ name: chunkName,
+ attrs: {
+ file_uri: filePath,
+ language: chunk.language,
+ chunk_kind: chunk.chunkKind,
+ start_line: chunk.lineStart,
+ end_line: chunk.lineEnd,
+ start_byte: chunk.startByte,
+ end_byte: chunk.endByte,
+ content_hash: chunk.contentHash,
+ parser_version: extractorName(),
+ },
+ });
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, 'file', chunkName, 'CONTAINS_CHUNK'),
+ src: fileNodeId2,
+ dst: cid,
+ predicate: 'CONTAINS_CHUNK',
+ attrs: {},
+ });
+ if (chunk.name !== null) {
+ const symbolKey = chunk.container ? `${chunk.container}.${chunk.name}` : chunk.name;
+ const symbolNid = nodeId(filePath, symbolKey);
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, chunkName, symbolKey, 'DEFINES'),
+ src: cid,
+ dst: symbolNid,
+ predicate: 'DEFINES',
+ attrs: {},
+ });
+ }
+ }
+
+ for (let i = 0; i + 1 < chunks.length; i++) {
+ const a = chunks[i];
+ const b = chunks[i + 1];
+ if (a.container == null && b.container == null) {
+ const aid = chunkId(filePath, a.chunkKind, a.name, a.lineStart);
+ const bid = chunkId(filePath, b.chunkKind, b.name, b.lineStart);
+ const aName = a.name ?? `file_body:${a.lineStart}`;
+ const bName = b.name ?? `file_body:${b.lineStart}`;
+ ops.push({
+ type: 'UpsertEdge',
+ id: edgeId(filePath, aName, bName, 'NEXT'),
+ src: aid,
+ dst: bid,
+ predicate: 'NEXT',
+ attrs: {},
+ });
+ }
+ }
+
for (const r of relationships) {
const srcKey = resolveKey(r.srcName);
const dstKey = r.predicate === 'CONTAINS'
diff --git a/decisions-backup.md b/decisions-backup.md
new file mode 100644
index 00000000..dc0c19b8
--- /dev/null
+++ b/decisions-backup.md
@@ -0,0 +1,14 @@
+# Graph Decisions Backup
+# Re-record these after reset with: ix decide "<name>" --rationale "<rationale>"
+
+1. ix decide "CONTAINS_CHUNK predicate for File→Chunk edges (separate from CONTAINS)" --rationale "CONTAINS is already used for File→Symbol and Class→Method hierarchy. Using a distinct CONTAINS_CHUNK predicate for File→Chunk edges avoids ambiguity in graph traversal and keeps chunk subgraph cleanly separable from symbol hierarchy queries."
+
+2. ix decide "inference_version is a required field on all inferred claims" --rationale "Observed claims (from parsing) are versioned implicitly via the extractor field in provenance. Inferred claims (subsystem membership, smells, summaries) need an explicit inference_version string (e.g. smell_v1) so they can be rerun selectively without touching observed facts. Missing = legacy for backward compat."
+
+3. ix decide "Chunk nodes are first-class observed nodes, not derived from symbols" --rationale "Symbols (class/function/method) are named entities. Chunks are code spans — the retrieval unit for LLM context. A chunk DEFINES one or more symbols and is CONTAINED by a file. Separating them allows chunk-granularity retrieval without blowing up symbol addressing."
+
+4. ix decide "Chunk granularity: AST-based semantic chunks" --rationale "Fixed-line windows break semantics. Symbol-only is too fragmented for LLM context. AST-based chunks (function body, class, top-level logical block) are the right unit. Hierarchy: File -> Chunk -> Symbol(s), with edges next_chunk, CALLS, REFERENCES between chunks. This enables context expansion: retrieve chunk + neighbors instead of whole file."
+
+5. ix decide "Summaries as claims with has_summary.* predicates" --rationale "Summary nodes would be a new abstraction with no benefit. Claims with predicates has_summary.purpose, has_summary.api, has_summary.dependencies, has_summary.responsibility are replaceable, versionable via source field (e.g. llm_v1), allow competing summaries to coexist, and make retrieval trivial with no extra joins."
+
+6. ix decide "SubsystemCandidate as claim not node" --rationale "Subsystem membership is belief, not fact. Using candidate_member_of claims on existing Subsystem nodes avoids graph bloat, reuses confidence/provenance/evidence infrastructure, supports conflicting memberships and overlapping subsystems, and allows re-scoring without rewiring the graph. Subsystems are real entities (nodes); membership is belief (claims)."
diff --git a/docs/map-update/README.md b/docs/map-update/README.md
new file mode 100644
index 00000000..2cbea245
--- /dev/null
+++ b/docs/map-update/README.md
@@ -0,0 +1,100 @@
+# Map Update — Feature Overview
+
+This document describes the `ix map` overhaul and the two new analysis commands (`ix smells`, `ix subsystems`) added in the `feature/map-update` branch.
+
+## What Changed
+
+### `ix map` — Progress Bar + Timing
+
+The map command now shows a live progress bar while computing the architectural map and prints ingest/map timing on completion.
+
+```
+ Computing map... ████████████░░░░░░░░░░░░░ 51%
+ ingest 1240ms · map 3820ms
+```
+
+Key changes in `ix-cli/src/cli/commands/map.ts`:
+- Exponential-decay progress bar using `setInterval`
+- Ingest and map durations printed to stderr (text mode only)
+- `--full` flag preserved for bypassing incremental map cache
+
+### `ix smells` — Architecture Smell Detection
+
+Runs graph-query-based smell detection and stores results as versioned claims on `File` nodes.
+
+```bash
+ix smells # run detection with default thresholds
+ix smells --format json # machine-readable output
+ix smells --list # show previously stored claims
+ix smells --god-module-chunks 15 # tune threshold
+```
+
+**Detected smells:**
+
+| Smell | Field | Description |
+|-------|-------|-------------|
+| Orphan file | `has_smell.orphan_file` | File with no import/call connections |
+| God module | `has_smell.god_module` | File with high chunk count or fan-in/out |
+| Weak component member | `has_smell.weak_component_member` | File connected to very few others |
+
+Claims are stored with `inference_version=smell_v1` and are versioned — re-running retires old claims before writing new ones.
+
+**Backend:** `SmellService.scala` + `SmellRoutes.scala`
+
+### `ix subsystems` — Subsystem Health Scoring
+
+Scores each architectural region produced by `ix map` on a 0–1 health scale.
+
+```bash
+ix subsystems # compute and store scores
+ix subsystems --list # show stored scores
+ix subsystems --level 2 # filter to subsystem level
+ix subsystems --format json # machine-readable
+```
+
+**Health score formula:**
+
+```
+health = 0.4 × (1 − smell_rate)
+ + 0.3 × tanh(chunk_density / 5)
+ + 0.3 × confidence
+```
+
+| Signal | Meaning |
+|--------|---------|
+| `smell_rate` | Fraction of files in the region with smell claims (lower = healthier) |
+| `chunk_density` | Avg `CONTAINS_CHUNK` edges per file (higher = richer structure) |
+| `confidence` | MapService cohesion/boundary ratio score |
+
+Claims stored with `inference_version=subsystem_v1`.
+
+**Backend:** `SubsystemService.scala` + `SubsystemRoutes.scala`
+
+## New Files
+
+| File | Purpose |
+|------|---------|
+| `ix-cli/src/cli/commands/smells.ts` | CLI command for smell detection |
+| `ix-cli/src/cli/commands/subsystems.ts` | CLI command for subsystem scoring |
+| `memory-layer/.../smell/SmellService.scala` | Smell detection logic (AQL queries + claim storage) |
+| `memory-layer/.../smell/SubsystemService.scala` | Subsystem scoring logic |
+| `memory-layer/.../api/SmellRoutes.scala` | HTTP routes for smell endpoints |
+| `memory-layer/.../api/SubsystemRoutes.scala` | HTTP routes for subsystem endpoints |
+
+## Recommended Workflow
+
+After running `ix map`, use the new commands to get a quality picture:
+
+```bash
+ix map # build architectural map
+ix smells --format json # find problem files
+ix subsystems --format json # score each region
+```
+
+Pipe results into `ix decide` to record findings:
+
+```bash
+ix decide "Refactor auth module" \
+ --rationale "god_module smell + health score 0.31" \
+ --affects AuthModule
+```
diff --git a/docs/map-update/map-experience-improvements.md b/docs/map-update/map-experience-improvements.md
new file mode 100644
index 00000000..456517ba
--- /dev/null
+++ b/docs/map-update/map-experience-improvements.md
@@ -0,0 +1,393 @@
+# ix map Experience Improvements — Full Implementation Spec
+
+---
+
+## TL;DR (Start Here)
+
+**Implement Phase 1 only (CLI renderer rewrite).**
+
+Do:
+
+* Replace flat output with tree layout
+* Group modules under subsystems
+* Add health summary line
+* Preserve JSON output exactly
+
+Do NOT:
+
+* Modify backend
+* Change API contracts
+
+Primary file:
+
+* `ix-cli/src/cli/commands/map.ts`
+
+---
+
+## Goals
+
+Improve `ix map` across 3 dimensions:
+
+1. **Clarity** → readable, hierarchical output
+2. **Exploration** → drill into subsystems/modules
+3. **Performance** → avoid unnecessary recomputation
+
+All changes must be:
+
+* Incremental
+* Backwards compatible
+* Independently shippable
+
+---
+
+## Current Behavior (Simplified)
+
+Pipeline:
+
+```
+ix map
+ → ingest
+ → POST /v1/map
+ → build graph + cluster (Louvain)
+ → persist regions
+ → return JSON
+ → CLI renders flat table
+```
+
+---
+
+## Data Model (IMPORTANT)
+
+The CLI receives:
+
+```json
+{
+ "regions": [
+ {
+ "id": "r1",
+ "label": "Cli / Client",
+ "level": 2,
+ "parent_id": null,
+ "file_count": 105,
+ "confidence": 0.72,
+ "is_cross_cutting": false
+ },
+ {
+ "id": "r2",
+ "label": "Queries",
+ "level": 1,
+ "parent_id": "r1",
+ "file_count": 10,
+ "confidence": 0.66,
+ "is_cross_cutting": true
+ }
+ ]
+}
+```
+
+Levels:
+
+* 3 = System
+* 2 = Subsystem
+* 1 = Module
+
+---
+
+## Problems
+
+### Output
+
+* Flat structure → no hierarchy
+* Hard to answer “what’s inside X?”
+* Truncation breaks readability
+* Cross-cutting unclear
+* Verbose mode overwhelming
+
+### Exploration
+
+* No drill-down
+* No file visibility
+* No explain integration
+
+### Performance
+
+* Always recomputes map
+* Always persists
+* Always runs ingest
+
+---
+
+## Target Design
+
+---
+
+## 1. Default Output (Tree Layout)
+
+Replace flat output with:
+
+```
+Architectural Map 225 files · 4 subsystems · 14 modules
+
+ Cli / Client · 105 files · Moderate (72%)
+ ├─ Cli / Client 53 files Well-defined
+ ├─ Cli / Github 22 files Moderate
+ ├─ Queries 10 files Moderate ⚠
+ ╰─ Role 5 files Fuzzy ⚠
+
+ Context · 56 files · Moderate (65%)
+ ├─ Ingestion 12 files Well-defined
+ ╰─ Model 19 files Fuzzy
+```
+
+---
+
+## 2. Health Summary Line
+
+Add at top:
+
+```
+3 well-defined · 8 moderate · 3 fuzzy · 3 cross-cutting
+```
+
+---
+
+## 3. Verbose Mode
+
+Keep tree, add right-aligned metrics:
+
+```
+Cli / Client · 105 files conf=72% br=651 xcut=0.03
+├─ Queries 10 files conf=66% xcut=0.15 ⚠
+```
+
+---
+
+## 4. Rendering Algorithm (IMPLEMENT THIS)
+
+```ts
+const subsystems = regions.filter(r => r.level === 2);
+const modules = regions.filter(r => r.level === 1);
+
+const modulesByParent = groupBy(modules, m => m.parent_id);
+
+for (const subsystem of subsystems) {
+ printSubsystem(subsystem);
+
+ const children = modulesByParent[subsystem.id] || [];
+
+ children.forEach((child, i) => {
+ const isLast = i === children.length - 1;
+ const prefix = isLast ? "╰─" : "├─";
+ printModule(child, prefix);
+ });
+}
+```
+
+---
+
+## 5. Rendering Rules
+
+* No label truncation
+* Use terminal width (`process.stdout.columns`)
+* Indentation = 2 spaces
+* Tree chars: `├─`, `╰─`
+* Cross-cutting = `⚠`
+* Confidence shown as % in headers
+
+---
+
+## 6. Flags (Future Phases)
+
+### `--subsystem <name>`
+
+Show only one subsystem + modules
+
+### `--module <name>`
+
+Show module + files
+
+### `--files`
+
+Expand full file list
+
+### Positional shorthand
+
+```
+ix map Ingestion
+ix map "Cli / Client"
+```
+
+---
+
+## 7. File Listing (Future)
+
+Add to backend:
+
+```json
+"member_paths": ["src/cli/map.ts"]
+```
+
+Only for level-1 regions.
+
+---
+
+## 8. Explain Integration (Future)
+
+### CLI Prompt Template
+
+```
+Explain the following subsystem:
+
+Name: {label}
+Files: {file_count}
+Confidence: {confidence}
+Child modules: {children}
+
+Boundary files:
+{top_files}
+
+Smells:
+{smells}
+
+Summarize:
+1. Purpose
+2. Structure quality
+3. Risks
+```
+
+---
+
+## 9. Performance Plan
+
+### Priority
+
+1. Cache map by graph revision (HIGH)
+2. Skip persist if unchanged (HIGH)
+3. Skip ingest if fresh (MEDIUM)
+4. Optimize parent wiring (LOW)
+
+---
+
+### 9a. Cache Map
+
+If `map_rev == graph_rev` → return stored map
+
+---
+
+### 9b. Skip Persist
+
+```
+if storedRev == currentRev:
+ skip write
+```
+
+---
+
+### 9c. Skip Ingest
+
+If freshness = false → skip ingest
+
+---
+
+## 10. Implementation Phases
+
+---
+
+### Phase 1 — CLI Output (DO THIS FIRST)
+
+Scope:
+
+* Tree rendering
+* Health summary
+* Verbose redesign
+
+Files:
+
+* `map.ts`
+
+Constraints:
+
+* JSON output unchanged
+* No backend changes
+
+---
+
+### Phase 2 — Drill-down
+
+* Add flags
+* Add member_paths
+
+---
+
+### Phase 3 — Explain
+
+* CLI context builder
+* Call explain endpoint
+
+---
+
+### Phase 4 — Performance
+
+* Cache map
+* Skip persist
+* Skip ingest
+
+---
+
+### Phase 5 — Future
+
+* Incremental Louvain
+* `/map/explain` endpoint
+* TUI interface
+
+---
+
+## 11. Acceptance Criteria (Phase 1)
+
+Must pass:
+
+* `ix map` → tree structure
+* `ix map --verbose` → tree + metrics
+* `ix map --level 3` → systems only
+* `ix map --level 1` → modules only
+* `ix map --format json` unchanged
+* No truncation up to 40 chars
+* Health summary present
+
+---
+
+## 12. Risks
+
+| Risk | Mitigation |
+| --------------- | --------------- |
+| Narrow terminal | fallback layout |
+| Name ambiguity | disambiguation |
+| Cache stale | `--full` flag |
+| Payload size | level-1 only |
+| Explain quality | improve prompt |
+
+---
+
+## 13. Key Files
+
+* CLI: `ix-cli/src/cli/commands/map.ts`
+* API: `MapRoutes.scala`
+* Service: `MapService.scala`
+* Graph: `MapGraphBuilder.scala`
+* Clustering: `LouvainClustering.scala`
+
+---
+
+## Final Instruction for Implementer
+
+Start with Phase 1.
+
+Do not over-engineer.
+
+Focus on:
+
+* Correct grouping
+* Clean tree rendering
+* Zero regression in JSON output
+
+Then stop and validate before continuing.
diff --git a/docs/plans/ingestion-perf-fix.md b/docs/plans/ingestion-perf-fix.md
new file mode 100644
index 00000000..1d07a5ec
--- /dev/null
+++ b/docs/plans/ingestion-perf-fix.md
@@ -0,0 +1,140 @@
+# Ingestion Performance Fix
+
+## Context
+
+`core-ingestion/src/index.ts` on this branch added chunk extraction (`ParsedChunk`) which is
+great, but introduced 3 regressions vs the fast baseline. Fix all three.
+
+## Change Tracking
+
+This section is the handoff record for follow-up review and bug fixing.
+
+### Files changed
+
+- `core-ingestion/src/index.ts`
+ Restored a module-level tree-sitter query cache, replaced `source.split('\n')`
+ with a character-scan line counter, and moved per-definition chunk hash work
+ into a post-match pass using a `pendingChunks` buffer. Also added one explicit
+ `(s: string)` type annotation in existing import parsing code so the package
+ still builds under the current TypeScript settings.
+- `docs/plans/ingestion-perf-fix.md`
+ Added this change-tracking section and will keep verification notes here.
+
+### Verification notes
+
+- `core-ingestion`: `npm run build` passed on 2026-03-22
+- `ix map` cold-ingest check: not run; skipped because `ix reset --code -y`
+ mutates indexed state and was not necessary to complete the code change
+
+---
+
+## Fix 1 — Restore the per-process query cache (CRITICAL)
+
+**File:** `core-ingestion/src/index.ts`
+
+**Problem:** `parseFile()` currently calls `new Parser.Query(grammar, queries)` on every
+invocation. Compiling a tree-sitter query (S-expression → automata) is expensive. With
+100 TypeScript files, it compiles the same TS query 100 times.
+
+**Fix:** Add a module-level query cache (already existed in the baseline) and use it inside
+`parseFile()`.
+
+Add this block near the existing `_parser` singleton (around line 180):
+
+```typescript
+// Per-process cache for compiled tree-sitter queries, keyed by language.
+// Avoids recompiling the same query string for every file of the same language.
+const _queryCache = new Map<SupportedLanguages | 'tsx', { grammar: any; query: any }>();
+
+function getCachedQuery(language: SupportedLanguages | 'tsx', grammar: any, querySource: string): any {
+ const cached = _queryCache.get(language);
+ if (cached && cached.grammar === grammar) return cached.query;
+ const query = new Parser.Query(grammar, querySource);
+ _queryCache.set(language, { grammar, query });
+ return query;
+}
+```
+
+Then inside `parseFile()`, replace:
+```typescript
+const query = new Parser.Query(grammar, queries);
+```
+with:
+```typescript
+const cacheKey = isTsx ? 'tsx' as const : language;
+const query = getCachedQuery(cacheKey, grammar, queries);
+```
+
+---
+
+## Fix 2 — Replace `source.split('\n')` with a char scan
+
+**File:** `core-ingestion/src/index.ts`
+
+**Problem:** `const lines = source.split('\n')` allocates a full string array only used for
+`lines.length` and `lines.length` in the `file_body` fallback chunk. Wasteful on large files.
+
+**Fix:** Replace the split with a char scan for the line count, and keep a separate
+`sourceLineCount` variable. Use it wherever `lines.length` is currently referenced.
+
+Replace:
+```typescript
+const lines = source.split('\n');
+```
+with:
+```typescript
+let sourceLineCount = 1;
+for (let i = 0; i < source.length; i++) { if (source.charCodeAt(i) === 10) sourceLineCount++; }
+```
+
+Then replace every reference to `lines.length` with `sourceLineCount`.
+
+---
+
+## Fix 3 — Move chunk SHA-256 hashing out of the hot definition loop
+
+**File:** `core-ingestion/src/index.ts`
+
+**Problem:** Inside the first-pass definition loop, every definition does:
+```typescript
+const chunkText = source.slice(startByte, endByte);
+const contentHash = crypto.createHash('sha256').update(chunkText).digest('hex').slice(0, 16);
+```
+This runs a full SHA-256 for every function/class/method at parse time. For a file with
+20 definitions that's 20 crypto operations, plus the `source.slice()` allocations.
+
+**Fix:** Keep the chunk data (startByte, endByte) in a temporary structure during the loop,
+then compute all content hashes in a single pass after the loop completes. This doesn't
+change output — just batches the hashing after the tree-sitter work is done so the two
+phases don't interleave.
+
+Concretely: collect `{ name, kind, lineStart, lineEnd, startByte, endByte, language, container }`
+into a `pendingChunks` array during the loop, then after the loop iterate `pendingChunks`
+to compute `contentHash` and push into `chunks[]`. The `file_body` fallback should also
+keep its hash computation (it's only one hash for the whole file when no chunks exist).
+
+---
+
+## Do NOT change
+
+- The `ParsedChunk` interface and its fields — keep as-is.
+- The chunk ops in `patch-builder.ts` (`CONTAINS_CHUNK`, `DEFINES`, `NEXT` edges) — keep as-is.
+- The `buildPatchWithResolution` function — keep as-is.
+- `resolveEdges` in `index.ts` — keep as-is.
+- The `_parser` singleton — keep as-is.
+- All query string constants in `queries.ts` — keep as-is.
+
+---
+
+## Verification
+
+After the changes, run:
+
+```bash
+cd core-ingestion && npm run build
+ix reset --code -y && IX_DEBUG=1 ix map
+```
+
+Expected: parse phase timing should drop from ~9s to ~2-3s for 200 files. The
+`Source hash lookup: 0 known hashes` line confirms a clean cold ingest.
+The map output should still show chunk-based regions (the visual output should be unchanged).
diff --git a/docs/plans/map-algorithm-overhaul.md b/docs/plans/map-algorithm-overhaul.md
new file mode 100644
index 00000000..aa4cef10
--- /dev/null
+++ b/docs/plans/map-algorithm-overhaul.md
@@ -0,0 +1,256 @@
+# ix map Algorithm: Speed & Accuracy Overhaul
+
+**Plan ID:** `46ecf20c-8bb2-9e54-121b-f5388adc8156`
+**Goal ID:** `ec6a17f4-3d75-4a18-a614-c23e3fd709ae`
+
+---
+
+## Overview
+
+`ix map` runs a multi-level Louvain community detection algorithm over a weighted file-coupling graph built from the knowledge graph. This document tracks planned improvements to make it both faster and more accurate.
+
+Pipeline stages:
+
+```
+ArangoDB → WeightedFileGraph → (crosscut penalty) → Louvain → Regions → Labels → Confidence → Persist
+```
+
+---
+
+## Implementation Status
+
+Status: implemented in code.
+
+What shipped:
+
+- `MapGraphBuilder.fetchCouplingByUri` no longer builds a giant `MERGE` map. It now resolves edge endpoints with direct `nodes` lookups keyed by `logical_id` and aggregates with `COLLECT ... WITH COUNT`.
+- `MapGraphBuilder.liveFileCount` now uses `COLLECT WITH COUNT` instead of loading all file rows.
+- `MapGraphBuilder.computeGraph` now precomputes directory segments once, short-circuits same-directory path proximity to `1.0`, and caps synthetic structural edge growth with `MaxEdges`.
+- `WeightedFileGraph` now carries `predicatePairs`, which restores dominant signal computation downstream.
+- `LouvainClustering.cluster` now runs three deterministic seeds, applies adaptive resolution by repository size, stops on modularity-gain convergence instead of a fixed 30-pass cap, and uses in-place Fisher-Yates shuffle.
+- `MapService` now uses the tuned signal weights:
+ - `CALLS = 1.0`
+ - `IMPORTS = 0.5`
+ - `EXTENDS/IMPLEMENTS = 1.2`
+ - `PATH = 0.2`
+- The CLI text renderer now defaults to the hierarchy graph view, with the ranked list preserved behind `ix map --list`.
+- Cross-cut token detection was expanded with layer/framework tokens including `service`, `controller`, `repository`, `handler`, `middleware`, `model`, `entity`, `dto`, `interface`, `types`, and `errors`.
+- Confidence scoring now uses:
+
+```scala
+val brScore = tanh(boundaryRatio / 2.5)
+val cohScore = cohesion.min(1.0)
+val sizeBonus = tanh(size / 8.0)
+val crossPenalty = crosscutScore * 0.4
+```
+
+- Labeling now prefers filtered suffixes first, then discriminative directory scoring, then filename-token fallback. Weak suffix tokens like `api`, `service`, and `common` no longer win by default.
+- `dominant_signals` is now populated from real intra-region predicate counts instead of being emitted empty.
+
+Validation summary:
+
+- `sbt memoryLayer/compile` passed after the changes.
+- A fresh backend using the new build returned `empty_dominant_signals = 0`.
+- Sequential timing checks on this repo remained within the target range for a repo of this size, though end-to-end `ix map` time here is heavily influenced by ingest/persist overhead rather than the clustering math alone.
+
+---
+
+## Current Known Issues
+
+### Speed
+
+| Location | Issue | Severity |
+|---|---|---|
+| `MapGraphBuilder.fetchCouplingByUri` | AQL `MERGE` over all live nodes to build `uri_map` — full collection scan on every call | High |
+| `MapGraphBuilder.fetchFiles` | Full table scan, no index on `kind` | Medium |
+| `MapGraphBuilder.currentInputRev` | Two separate full table scans (nodes + edges) | Medium |
+| `MapGraphBuilder.liveFileCount` | Calls `fetchFiles()` just to count — fetches all data unnecessarily | Low |
+| `MapGraphBuilder.computeGraph` (path-proximity pass) | O(n²) within each directory — 100-file directory = 4,950 pairs; `pathProximity` called per pair with string splits | High |
+| `MapGraphBuilder.pathProximity` | Splits path strings on every call, no precomputation | Medium |
+| `LouvainClustering.louvainPass` | `rng.shuffle(nodes)` on every pass — allocates new list up to 30× | Low |
+| `MapService.applyPenalty` | Copies full adjacency map — could be done in-place since graph is not reused | Low |
+| `MapService.loadCachedMap` → `liveFileCount` | Separate DB round trip just to populate `fileCount` | Low |
+
+### Accuracy
+
+| Location | Issue | Severity |
+|---|---|---|
+| `MapService.buildOneRegion` | `dominantSignals` always `[]` — per-predicate counts lost after graph construction | Medium |
+| Signal weights (α=1.0 β=0.7 γ=0.9 ε=0.3) | Arbitrary, not empirically validated | Medium |
+| `crosscutTokens` set | Missing common framework terms: "service", "controller", "repository", "handler", "middleware" | Medium |
+| `computeConfidence` weights (0.4/0.3/0.3) | Ad-hoc formula, not validated against real-world repo quality | Low |
+| `inferLabel` | Strategy 1 (longest common dir suffix) fires before the better strategy 2 (discriminative segment scoring) — can produce misleading labels for deeply nested monorepos | Medium |
+| Louvain resolution fixed at 1.0 | Higher resolution → smaller communities; lower → larger. Should adapt to repo size or be tunable | Medium |
+| Louvain seed fixed at 42 | Single deterministic run may hit local optima — multiple seeds with best-modularity selection improves stability | Low |
+| `minCommunitySize` hardcoded | 3 for 3-level repos, 2 otherwise — should scale with repo size | Low |
+
+---
+
+## Tasks
+
+### Task 1 — Optimize AQL queries in MapGraphBuilder
+**ID:** `adb844f1-62a3-98f2-a27e-33cc50364ae0`
+
+The two costliest DB operations:
+
+**`fetchCouplingByUri` MERGE approach:**
+```aql
+LET uri_map = MERGE(
+ FOR n IN nodes FILTER ... RETURN {[n.logical_id]: n.provenance.source_uri}
+)
+```
+`MERGE` over thousands of nodes is slow and memory-heavy. Replace with a direct join:
+```aql
+FOR e IN edges
+ FILTER e.predicate IN @predicates AND e.deleted_rev == null
+ LET sn = FIRST(FOR n IN nodes FILTER n.logical_id == e.src ... RETURN n.provenance.source_uri)
+ LET dn = FIRST(FOR n IN nodes FILTER n.logical_id == e.dst ... RETURN n.provenance.source_uri)
+ FILTER sn != null AND dn != null AND sn != dn
+ COLLECT srcUri = sn, dstUri = dn, pred = e.predicate WITH COUNT INTO cnt
+ RETURN {srcUri, dstUri, predicate: pred, count: cnt}
+```
+Or with `AQL FOR` subqueries if the ArangoDB index on `logical_id` is usable (it should be, as it's the document key).
+
+**`liveFileCount`:** Replace with a `COLLECT WITH COUNT` query instead of fetching all file data.
+
+**`currentInputRev`:** Benchmark whether combining into one AQL pass is faster.
+
+---
+
+### Task 2 — Fix O(n²) path-proximity pass
+**ID:** `9544f83f-e0bb-8973-aa1c-a4d642a63ef0`
+
+**Problem:** The same-directory edge pass is:
+```scala
+for (i <- dirFiles.indices; j <- (i + 1) until dirFiles.size) {
+ pathProximity(sv.path, dv.path) // splits strings every call
+}
+```
+For a directory with D files, this is O(D²) calls. Monorepos can have 50+ files in src/.
+
+**Fix:**
+1. Precompute `dirSegments: Map[NodeId, Array[String]]` for all files once.
+2. Replace `pathProximity(a, b)` with an index lookup using precomputed segments.
+3. For same-directory files the result is always `1.0 / (1.0 + 0)` = 1.0 — can hardcode for the same-dir case.
+4. Add an `adjMut.size > MaxEdges` safety cap to prevent quadratic blowup in dense directories.
+
+---
+
+### Task 3 — Improve Louvain convergence and stability
+**ID:** `ca2b0225-c97d-77bf-021f-32b6e1b51884`
+
+**Multi-seed best-of selection:**
+```scala
+val runs = (1 to 3).map(seed => louvainPass(graph, resolution, new Random(seed)))
+val best = runs.maxBy(a => modularity(graph, a))
+```
+3 runs with different seeds, pick the one with highest modularity. Cost: 3× but more stable results.
+
+**Adaptive resolution:**
+- Small repos (< 50 files): resolution 1.2 (prefers smaller, tighter communities)
+- Medium repos (50–500 files): resolution 1.0 (default)
+- Large repos (> 500 files): resolution 0.8 (allows larger communities, avoids over-fragmentation)
+
+**Convergence check:** The 30-pass hard cap is arbitrary. Add a modularity delta check — stop when ΔQ < 1e-6 rather than forcing 30 passes.
+
+**`rng.shuffle` allocation:** Use Fisher-Yates in-place shuffle on an `Array[NodeId]` instead of `rng.shuffle(nodes)` which allocates a new `Vector` each pass.
+
+---
+
+### Task 4 — Tune signal weights, crosscut detection, and confidence formula
+**ID:** `fc9c3ec1-5b6e-3889-8974-ecc9d6dc8995`
+
+**Signal weights — proposed tuned values:**
+
+| Signal | Current | Proposed | Rationale |
+|---|---|---|---|
+| α (CALLS) | 1.0 | 1.0 | Keep as anchor |
+| β (IMPORTS) | 0.7 | 0.5 | Imports are noisier than calls in TypeScript/Python; over-weighted |
+| γ (EXTENDS/IMPLEMENTS) | 0.9 | 1.2 | Type hierarchy is a very strong cohesion signal, especially in Scala/Java |
+| ε (path proximity) | 0.3 | 0.2 | Structural prior should be weak; files in same dir ≠ same subsystem |
+
+**Crosscut token expansion:**
+```scala
+CrosscutTokens = Set(
+ // existing
+ "util", "utils", "helper", "helpers", "common", "shared", "base",
+ "logging", "log", "logger", "config", "conf", "constants", "constant",
+ "extensions", "ext", "mixin", "mixins", "core", "framework",
+ // add: framework layer names
+ "service", "services", "controller", "controllers",
+ "repository", "repositories", "repo", "handler", "handlers",
+ "middleware", "interceptor", "filter", "filters",
+ "model", "models", "entity", "entities", "dto", "dtos",
+ "interface", "interfaces", "types", "errors", "exceptions"
+)
+```
+These are all cross-cutting by nature in layered architectures.
+
+**`computeConfidence` — boundary ratio saturation:**
+
+Current `brScore = tanh(br / 3.0)` saturates at br ≈ 9. Real repos often have br in 0.5–5 range, so the curve is well-placed. But the size bonus `tanh(size / 10)` gives half-credit to 7-file regions. Proposed:
+```scala
+val brScore = math.tanh(boundaryRatio / 2.5) // slightly tighter saturation
+val cohScore = cohesion.min(1.0)
+val sizeBonus = math.tanh(size / 8.0) // penalize tiny regions more
+val crossPenalty = crosscutScore * 0.4 // stronger crosscut penalty
+```
+
+---
+
+### Task 5 — Restore dominant_signals and improve label inference
+**ID:** `8c897b67-c639-5705-c0d3-4278b4fa54f3`
+
+**Dominant signals:**
+The `dominantSignals` field was cleared in the bug-fix review because `signalCounts` was never populated. The right fix is to thread per-predicate counts through from `MapGraphBuilder` into `buildOneRegion`.
+
+Option A: Add `predicateCounts: Map[(NodeId,NodeId), Map[String,Int]]` to `WeightedFileGraph` (keeps graph builder responsible).
+
+Option B: In `buildOneRegion`, iterate the adjacency and compute a signal breakdown from the raw weights by inverting the weight formula (approximate).
+
+Option A is cleaner. Add to `WeightedFileGraph`:
+```scala
+final case class WeightedFileGraph(
+ vertices: Vector[FileVertex],
+ adjMatrix: Map[NodeId, Map[NodeId, Double]],
+ degrees: Map[NodeId, Double],
+ totalWeight: Double,
+ predicatePairs: Map[(NodeId, NodeId), Map[String, Int]] // NEW
+)
+```
+
+**Label inference — fix strategy order:**
+
+Strategy 1 (longest common suffix) fires even when the common segment is a noise token like "api" or "db". Proposed:
+
+1. Try longest common suffix after filtering noise tokens (current).
+2. If result length < 3 chars, fall through to strategy 2.
+3. Strategy 2 (discriminative depth-weighted segments) should be weighted by _intra-cluster_ frequency vs. _corpus_ frequency — a segment appearing in 80% of this cluster but only 10% of all files is highly discriminative.
+4. Fallback: most frequent camelCase token from filenames.
+
+---
+
+## Performance Benchmarks Needed
+
+Before tuning, establish baselines on this repo:
+
+```bash
+time ix map --format json > /dev/null # end-to-end
+time ix map --format json > /dev/null # second run (should hit cache)
+```
+
+Target metrics:
+- Cold map (no cache): < 5s for this repo (~150 files)
+- Cache hit: < 500ms
+- Large repo (2000+ files): < 30s cold, < 1s cached
+
+---
+
+## Files Affected
+
+| File | Changes |
+|---|---|
+| `memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala` | Tasks 1, 2 |
+| `memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala` | Task 3 |
+| `memory-layer/src/main/scala/ix/memory/map/MapService.scala` | Task 4, 5 |
+| `memory-layer/src/main/scala/ix/memory/map/MapTypes.scala` | Task 5 (add `predicatePairs` to `WeightedFileGraph`) |
diff --git a/docs/plans/map-algorithm-review.md b/docs/plans/map-algorithm-review.md
new file mode 100644
index 00000000..748fe938
--- /dev/null
+++ b/docs/plans/map-algorithm-review.md
@@ -0,0 +1,334 @@
+# ix map Performance + Scalability Fix Plan
+
+**Files reviewed:** `MapGraphBuilder.scala`, `MapService.scala`, `LouvainClustering.scala`, `MapTypes.scala`
+**Date:** 2026-03-22
+
+---
+
+## Change Tracking
+
+This section is the running handoff record for follow-up review and bug fixing.
+
+### Files changed
+
+- `memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala`
+ Rewrote `fetchCouplingByUri()` to build a single `logical_id -> source_uri`
+ lookup map inside AQL, pushed source-extension filtering into `fetchFiles()`,
+ added directory-size guardrails for path-proximity edges, and changed
+ `currentInputRev()` to use a dedicated `source-graph` revision with a scan
+ fallback for older data.
+- `memory-layer/src/main/scala/ix/memory/map/MapService.scala`
+ Replaced O(R²) parent wiring in `buildRegions()` with partition-based lookup,
+ removed subset-scan recomputation from `rehydrateRelationships()`, trimmed
+ `CrosscutTokens`, replaced the max-degree heuristic with a real percentile
+ cutoff, and precomputed signal adjacency so dominant-signal scoring only
+ touches region-local incident pairs.
+- `memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala`
+ Added seed gating for small graphs and reused the community-weight map inside
+ the hot Louvain loop instead of allocating it per node.
+- `memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`
+ Updated revision writes so bulk ingestion advances both the global revision
+ and the dedicated `source-graph` revision.
+- `memory-layer/src/main/scala/ix/memory/db/ArangoGraphWriteApi.scala`
+ Added source-graph revision tracking for patch commits that touch map inputs,
+ while leaving derived writes on the global revision only.
+- `docs/plans/map-algorithm-review.md`
+ Revised the recommendations so they match the current codebase, added this
+ tracking section, and recorded implementation and verification status.
+
+### Implementation notes
+
+- `currentInputRev()` needs a dedicated source-graph revision, not just the
+ global revision, because non-map-input writes also advance the main revision
+ counter.
+- `buildRegions()` can use partition assignments directly, but only after
+ accounting for communities filtered out by `minCommunitySize`.
+- `CrosscutTokens` cleanup should be paired with a fix to the current
+ pseudo-percentile degree threshold.
+
+### Verification notes
+
+- Build: `sbt memoryLayer/compile` passed on 2026-03-22
+- Runtime benchmark: not run
+- `ix map` end-to-end validation: not run in this pass
+
+---
+
+## TL;DR
+
+Primary bottlenecks:
+
+1. O(E × N) database query in `fetchCouplingByUri()`
+2. Full graph scan per run in `currentInputRev()`
+3. O(R²) region construction in `buildRegions()`
+
+These issues cause `ix map` to degrade from near-linear toward quadratic behavior, leading to Docker crashes and unusable performance on large codebases.
+
+Fixing these should reduce runtime from minutes to seconds for 500-2,000 file repos.
+
+---
+
+## Summary
+
+The algorithm is architecturally correct: multi-level Louvain plus weighted signals is the right direction.
+
+The problem is **scalability**, not correctness.
+
+**Severity legend:** Critical / High / Medium / Low
+
+---
+
+# 1. Critical Bottlenecks
+
+## `fetchCouplingByUri()` - O(E × N) correlated subqueries
+
+Each edge performs two full scans over `nodes` to resolve `src` and `dst`. Without an index on `logical_id`, this is O(E × N) - for 5,000 edges and 50,000 nodes that's 250M comparisons per `ix map`.
+
+**Fix (preferred):** build the URI map once, then join:
+
+```aql
+LET uriMap = (
+ FOR n IN nodes
+ FILTER n.deleted_rev == null
+ AND n.provenance.source_uri != null
+ AND n.kind NOT IN @ignoredKinds
+ RETURN { id: n.logical_id, uri: n.provenance.source_uri }
+)
+
+LET byId = ZIP(uriMap[*].id, uriMap[*].uri)
+
+FOR e IN edges
+ FILTER e.predicate IN @predicates
+ AND e.deleted_rev == null
+ LET su = byId[e.src]
+ LET du = byId[e.dst]
+ FILTER su != null AND du != null AND su != du
+ COLLECT srcUri = su, dstUri = du, pred = e.predicate WITH COUNT INTO cnt
+ RETURN { srcUri, dstUri, predicate: pred, count: cnt }
+```
+
+**Alternative:** add a persistent index on `nodes.logical_id`. Cuts each subquery from O(N) to O(log N) without rewriting the query.
+
+---
+
+## `currentInputRev()` - full graph scan per run
+
+Runs a MAX aggregation over all nodes and edges on every `ix map` call, even when the graph is unchanged and the cached map will be reused.
+
+**Fix:** maintain a dedicated source-graph revision entry updated only when source-graph inputs change. Do not reuse the global revision counter directly, because map persistence and other derived writes also advance it.
+
+One viable implementation is a separate revisions entry:
+
+```json
+{ "_key": "source-graph", "rev": 123 }
+```
+
+Then `currentInputRev()` becomes a single key lookup, with a fallback full scan only for older databases that do not yet have the dedicated metadata:
+
+```aql
+FOR doc IN revisions
+ FILTER doc._key == "source-graph"
+ LIMIT 1
+ RETURN doc.rev
+```
+
+Result: O(1) instead of O(N + E).
+
+---
+
+# 2. High Priority Fixes
+
+## `buildRegions()` - O(R²) parent-child wiring
+
+Currently calls `region.memberFiles.subsetOf(p.memberFiles)` for every `(region, parent)` pair. Replace with a direct lookup from the Louvain partition, but preserve correctness when some communities are filtered out by `minCommunitySize`:
+
+```scala
+val fileToParent: Map[NodeId, Int] = coarseLevel.assignment
+
+val parentOpt = memberFiles.headOption
+ .flatMap(fileToParent.get)
+ .flatMap(coarseRegionByCommunity.get)
+```
+
+Where `coarseRegionByCommunity` only contains retained coarse communities, and equal-set coarse regions should still be treated as "no parent" to match the current hierarchy behavior.
+
+---
+
+## `rehydrateRelationships()` - remove recomputation
+
+`parentId` and `childRegionIds` are already persisted on region nodes. Do not rerun the subset scan on cache load. Wire children directly from the loaded data:
+
+```scala
+val childrenByParent = regions
+ .flatMap(r => r.parentId.map(pid => pid -> r.id))
+ .groupBy(_._1)
+ .view
+ .mapValues(_.map(_._2).toSet)
+ .toMap
+```
+
+---
+
+## `CrosscutTokens` - too many false positives
+
+Tokens like `"service"`, `"controller"`, `"repository"`, `"model"`, `"entity"`, `"dto"` are standard naming conventions, not cross-cutting concerns. Flagging them applies a penalty to the coupling signals that should cluster those files together.
+
+Also fix the degree-side heuristic: the current code uses `maxDeg * 0.9`, which is not actually "top 10% by degree". If percentile behavior matters, compute it from the observed degree distribution instead of scaling the maximum.
+
+Remove architectural terms. Keep only genuine horizontal concerns:
+
+```scala
+private val CrosscutTokens = Set(
+ "util", "utils", "helper", "helpers", "common", "shared",
+ "logging", "config", "constants", "base"
+)
+```
+
+---
+
+# 3. Medium Fixes
+
+## `computeDominantSignals()` - O(total edges) per region
+
+Iterates all `graph.predicatePairs` for each region. Build a per-file signal index once before the region loop so each region only touches incident pairs for its own members.
+
+```scala
+Map[NodeId, Vector[(NodeId, Map[String, Int])]]
+```
+
+---
+
+## Path proximity - O(N²) per directory
+
+Only run the path proximity pass for directories with between 2 and 50 files. Large flat directories (generated code, fixtures) add noise and inflate the graph without contributing architectural signal.
+
+Also push source-extension filtering into the `fetchFiles()` AQL so the JVM does not deserialize non-source file rows it will drop anyway.
+
+---
+
+## Louvain optimizations
+
+**Seed gating** - skip the extra two seeds for small graphs:
+
+```scala
+val numSeeds = if (graph.vertices.size < 30) 1 else 3
+```
+
+**Avoid map allocation in hot loop** - allocate once and clear per node:
+
+```scala
+val wToComm = mutable.HashMap[Int, Double]()
+// inside the loop:
+wToComm.clear()
+```
+
+---
+
+# 4. Graph Guardrails
+
+Hard limits to prevent runaway memory and time on large repos:
+
+**Node limits:**
+- V > 5,000: disable path proximity edges; cap soft edges per node at 5
+- V > 10,000: switch to coarse mode (directory-level clustering only)
+
+**Density limits:**
+- E / V > 50: prune lowest-weight edges before clustering
+
+**Edge budget by type:**
+- Structural edges (imports, calls): unlimited
+- Path similarity: max 3 per node
+
+---
+
+# 5. Execution Plan
+
+## Phase 1 - Database
+
+- Fix `fetchCouplingByUri()` query
+- Replace `currentInputRev()` with dedicated source-graph revision metadata
+
+**Validate:** identical output, runtime measurably reduced.
+
+## Phase 2 - Region construction
+
+- Replace `buildRegions()` subset scan with direct lookup
+- Remove recomputation in `rehydrateRelationships()`
+- Precompute dominant-signal adjacency once per graph
+
+**Validate:** same region hierarchy, no missing parents.
+
+## Phase 3 - Graph density controls
+
+- Limit path proximity to small/medium directories
+- Push source-extension filtering into the DB query
+
+**Validate:** lower edge count on large flat trees with no missing file vertices.
+
+## Phase 4 - Louvain optimization
+
+- Seed gating by graph size
+- Reuse `wToComm` map across iterations
+
+**Validate:** clustering stable across runs, runtime improved.
+
+## Phase 5 - Cross-cut cleanup
+
+- Fix `CrosscutTokens`
+- Replace the pseudo-percentile threshold with a real percentile cutoff
+
+## Phase 6 - Scaling safeguards
+
+- Implement graph guardrails
+- Add coarse mode for V > 10,000
+
+---
+
+# 6. Constraints
+
+- Output of `ix map` should remain materially consistent: same broad region membership and hierarchy unless a heuristic fix explicitly justifies a behavior change
+- Do not introduce new edge types
+- Do not alter signal weights without justification
+
+---
+
+# 7. Anti-Patterns
+
+- O(N²) similarity computations across the whole repo
+- Full collection scans inside per-edge or per-node loops
+- Per-node or per-edge database queries
+- Rebuilding the full graph on every cache lookup
+- Dense adjacency matrices
+
+---
+
+# 8. Success Criteria
+
+| Metric | Target |
+|---|---|
+| `fetchCouplingByUri` | < 100ms for 10k edges |
+| `currentInputRev` | < 5ms |
+| 500 files | < 5s end-to-end |
+| 2,000 files | < 10s end-to-end |
+
+---
+
+# 9. Priority Order
+
+| Priority | Fix |
+|---|---|
+| 1 | `fetchCouplingByUri` query rewrite |
+| 2 | `currentInputRev` dedicated source-graph revision |
+| 3 | Path proximity directory-size guardrail |
+| 4 | `CrosscutTokens` cleanup + real percentile threshold |
+| 5 | `rehydrateRelationships` recompute removal |
+| 6 | `buildRegions` O(R²) replacement |
+| 7 | Louvain seed gating + map reuse |
+
+---
+
+## Final Note
+
+Target complexity: **O(V + E)**.
+
+If performance degrades superlinearly, the graph is too dense or queries are unindexed. Diagnose with `IX_DEBUG=1 ix map` and inspect per-phase timing before adding more signal types or hierarchy levels.
diff --git a/docs/plans/subsystem-explain.md b/docs/plans/subsystem-explain.md
new file mode 100644
index 00000000..a9077221
--- /dev/null
+++ b/docs/plans/subsystem-explain.md
@@ -0,0 +1,514 @@
+# `ix subsystems <target> --explain`
+
+**Date:** 2026-03-22
+
+---
+
+## Overview
+
+`ix explain` produces a narrative breakdown of an individual symbol (function, class, file).
+`ix subsystems <target> --explain` does the same thing one level up — for an architectural
+region produced by `ix map`.
+
+The goal is to answer: *"What is this subsystem, what does it contain, how healthy is it, and
+why does it matter?"* — in plain English, with enough structural context that a developer new
+to the codebase can orient themselves immediately.
+
+---
+
+## Reality Check
+
+The basic shape of this plan is correct, but the current codebase has two gaps the original
+draft did not account for:
+
+1. The existing scoped subsystem payload is sufficient for most of `--explain`, but it does
+ **not** currently expose enough revision metadata to support a reliable
+ `"health score may be stale"` note.
+2. Large-repo ingestion is currently fragile for architectural reasons that are adjacent to
+ this feature work: the ingest path materializes too much in memory, queries too much state
+ up front, and emits oversized patch payloads for non-code files.
+
+This document now covers both:
+- the concrete changes required for `ix subsystems <target> --explain`
+- the concrete changes required to stop ingestion from failing on large repositories
+
+---
+
+## Change Tracking
+
+### Files changed
+
+- `docs/plans/subsystem-explain.md`
+ Expanded the plan so it reflects the current subsystem APIs and the large-repo ingestion
+ failure modes discovered during code review.
+
+### Verification notes
+
+- Doc-only update; no code or tests run in this pass.
+
+---
+
+## CLI Interface
+
+```
+ix subsystems <target> --explain
+ix subsystems <target> --explain --format json
+ix subsystems <target> --explain --pick <n>
+```
+
+`--explain` is a new flag on the existing `subsystems` command, layered on top of the current
+target-scoping behaviour. It is mutually exclusive with `--list`.
+
+**Examples:**
+
+```
+ix subsystems api --explain
+ix subsystems "Ingestion Layer" --explain
+ix subsystems auth --explain --pick 2
+ix subsystems auth --explain --format json
+```
+
+`--pick` resolves ambiguous matches the same way it does today.
+
+---
+
+## Data Sources
+
+Two existing server calls are combined — no new endpoints required.
+
+| Data | Source |
+|---|---|
+| Region label, level, parent, children, files, confidence, dominant signals, cross-cutting flag | `GET /v1/subsystems/map?target=…` → `ScopedSubsystemResult` |
+| Health score, chunk density, smell rate, smell files, total chunks | `GET /v1/subsystems` → `SubsystemScore[]`, filtered to the resolved region id |
+
+If the health score has not yet been computed for the region (e.g. `ix map` was run but
+`ix subsystems` has not been run since), the Health section is omitted and a note is printed:
+`Run 'ix subsystems' to compute health scores.`
+
+### Current Data Contract Gap
+
+The current score payload is enough to render Health, but not enough to prove staleness.
+
+Today:
+- `GET /v1/subsystems/map` returns scoped region structure without score metadata.
+- `GET /v1/subsystems` returns health scores without a `map_rev` or score revision field.
+
+That means this note from the original draft is **not implementable as written**:
+
+`Health score may be stale — run 'ix subsystems' to refresh.`
+
+To support that note cleanly, one of these must change:
+
+1. Add `map_rev` to each stored subsystem score result.
+2. Add score `created_rev` or score computation revision to the list response.
+3. Remove the stale-health note from v1 and only keep the `"Run 'ix subsystems' to compute health scores."` note when the score is missing.
+
+Recommended v1 choice:
+- Keep the feature server-light.
+- Do **not** add a stale-health note yet.
+- Only add it after score revision metadata is exposed.
+
+---
+
+## Output Sections
+
+### Text Format
+
+```
+Explanation
+ `Ingestion Layer` serves as a subsystem spanning 34 files. It groups components
+ responsible for parsing source files and committing patches to the graph. It sits
+ inside the Core System and is well-defined with import coupling as its primary
+ structural signal.
+
+Context
+ Level: subsystem (level 2)
+ Parent: Core System (system)
+ Files: 34
+ Children: 4 modules
+ Confidence: Well-defined (88%)
+ Signals: import coupling · path proximity
+
+Composition
+ ● MODULE File Parser Well-defined 91% 12 files
+ ● MODULE Patch Builder Well-defined 84% 9 files
+ ● MODULE Bulk Writer Moderate 66% 8 files
+ ● MODULE Ingestion Loader Fuzzy 44% 5 files
+ 3 well-defined · 1 moderate · 0 fuzzy · 0 cross-cutting
+
+Health
+ Score: ████░ 0.74
+ Chunk density: 5.1 chunks/file
+ Smell rate: 6% (2 files flagged)
+ Status: Good — rich structure, low smell coverage
+
+Why it matters
+ Ingestion Layer is the primary entry path for all source data into the graph.
+ Its high chunk density indicates thorough code coverage in the knowledge graph.
+ 2 files have active smell claims — run `ix smells` to review.
+ Run `ix impact "Ingestion Layer"` to see what depends on this subsystem.
+```
+
+### JSON Format
+
+`--format json` returns a single object:
+
+```json
+{
+ "resolvedTarget": {
+ "id": "uuid",
+ "label": "Ingestion Layer",
+ "level": 2,
+ "label_kind": "subsystem"
+ },
+ "explanation": "...",
+ "context": {
+ "level": 2,
+ "label_kind": "subsystem",
+ "parent": { "id": "uuid", "label": "Core System", "label_kind": "system" },
+ "file_count": 34,
+ "child_count": 4,
+ "confidence": 0.88,
+ "dominant_signals": ["import coupling", "path proximity"],
+ "is_cross_cutting": false
+ },
+ "composition": {
+ "children": [
+ { "label": "File Parser", "level": 1, "label_kind": "module", "file_count": 12, "confidence": 0.91, "is_cross_cutting": false }
+ ],
+ "summary": { "well_defined": 3, "moderate": 1, "fuzzy": 0, "cross_cutting": 0 }
+ },
+ "health": {
+ "score": 0.74,
+ "chunk_density": 5.1,
+ "smell_rate": 0.06,
+ "smell_files": 2,
+ "total_chunks": 173,
+ "status": "good"
+ },
+ "whyItMatters": "...",
+ "notes": []
+}
+```
+
+`health` is `null` if no score has been computed yet.
+
+---
+
+## Section Definitions
+
+### Explanation
+
+One to two sentence prose summary synthesised from the region label, level, parent name,
+dominant signals, file count, and cross-cutting flag. No LLM — constructed from templates,
+the same way `ix explain`'s `describeRoleLocal` and `synthesizeSystemRole` work.
+
+Template inputs:
+
+| Signal | Mapped meaning |
+|---|---|
+| `level === 3` | "system" — top-level architectural boundary |
+| `level === 2` | "subsystem" — a bounded functional area |
+| `level === 1` | "module" — a focused implementation cluster |
+| `is_cross_cutting` | adds "…and is cross-cutting — it spans multiple subsystems" |
+| `dominant_signals` | "Its primary structural signal is import coupling" |
+| `parent.label` | "It sits inside the `<parent>` system" |
+| `confidence ≥ 0.75` | "well-defined" |
+| `confidence ≥ 0.50` | "moderately defined" |
+| `confidence < 0.50` | "loosely defined" |
+
+### Context
+
+Tabular structural facts — mirrors the `ix explain` Context section:
+
+```
+Level: subsystem (level 2)
+Parent: Core System (system)
+Files: 34
+Children: 4 modules
+Confidence: Well-defined (88%)
+Signals: import coupling · path proximity
+```
+
+Omit `Children` line when there are none (leaf modules).
+Omit `Parent` line for top-level systems with no parent.
+
+### Composition
+
+Renders the immediate child regions as a mini tree, sorted by file count descending
+(same comparator as `renderScopedTreeNode`). Capped at 10 children; if more exist,
+append ` … N more. Use 'ix subsystems <target>' for the full tree.`
+
+Followed by the existing well-defined / moderate / fuzzy / cross-cutting tally from
+`ScopedSubsystemResult.summary`.
+
+Omit the entire section for leaf regions (no children).
+
+### Health
+
+Uses the `SubsystemScore` for the resolved region id.
+
+```
+Score: ████░ 0.74 (healthBar using 5-block scale, same as --list)
+Chunk density: 5.1 chunks/file
+Smell rate: 6% (2 files flagged)
+Status: <status phrase>
+```
+
+**Status phrases** (derived from score thresholds):
+
+| Score | Status phrase |
+|---|---|
+| ≥ 0.80 | `Healthy — rich structure, minimal technical debt` |
+| ≥ 0.65 | `Good — rich structure, low smell coverage` |
+| ≥ 0.45 | `Moderate — structure is present, smell coverage is elevated` |
+| < 0.45 | `Needs attention — sparse structure or significant smell coverage` |
+
+Chunk density context lines (appended to Status line as a note):
+
+| Density | Note |
+|---|---|
+| ≥ 5.0 | `High chunk density indicates thorough code coverage in the knowledge graph.` |
+| ≥ 2.0 | *(no note)* |
+| < 2.0 | `Low chunk density — run 'ix ingest' to improve coverage.` |
+
+### Why it matters
+
+Single prose paragraph synthesised from health, level, cross-cutting status, and parent
+context. Mirrors the structure of `ix explain`'s `renderWhyItMatters`.
+
+Decision tree:
+
+1. **Cross-cutting + fuzzy confidence** → architectural drift warning:
+ `<label> spans multiple subsystems and has low clustering confidence, suggesting architectural drift. Consider decomposing or clarifying its boundaries.`
+
+2. **High health (≥ 0.80) + top-level (level 3)** → stable anchor:
+ `<label> is a stable, well-defined system boundary. Changes within it are well-contained. Run 'ix impact "<label>"' to review cross-system dependencies.`
+
+3. **Low health (< 0.45) + smell files > 0** → needs attention:
+ `<label> has <N> files with active smell claims and a health score of <score>. Prioritize refactoring here before expanding the subsystem. Run 'ix smells' to review.`
+
+4. **Normal** → role + callers suggestion:
+ `<label> is a <status> <level label> <parent phrase>. Run 'ix impact "<label>"' to understand its downstream blast radius.`
+
+### Notes
+
+Mirrors `ix explain`'s Notes section. Sources:
+- Health score missing: `Run 'ix subsystems' to compute health scores for this region.`
+- Health score computed at a different revision than current graph:
+ `Health score may be stale — run 'ix subsystems' to refresh.`
+
+Implementation note:
+- This second note requires new score revision metadata and should be treated as optional
+ follow-up work, not part of the minimum implementation.
+
+---
+
+## Ambiguity Handling
+
+Identical to the current `ix subsystems <target>` behaviour. If the target is ambiguous,
+the existing `renderSubsystemError` output is shown and `--explain` is ignored:
+
+```
+Ambiguous target "auth". Multiple architecture regions matched:
+
+1. Auth Middleware subsystem 12 files
+2. Auth Tests module 8 files
+
+Run:
+ ix subsystems auth --pick <n> --explain
+```
+
+---
+
+## Implementation Plan
+
+### Phase 1 — Client (subsystems.ts)
+
+1. Add `--explain` flag to the command definition.
+2. When `--explain` is set and a `ScopedSubsystemResult` is returned:
+ a. Make a second call to `client.listSubsystems()` and find the matching `SubsystemScore`
+ by `region_id`.
+ b. Call a new `renderSubsystemExplanation(scoped, score | null)` function.
+3. `renderSubsystemExplanation` implements the five sections above using template functions
+ (no new server endpoints, no LLM).
+4. If `--format json` is set, return a purpose-built explanation object instead of the current
+ scoped map JSON.
+
+New functions (all in `subsystems.ts` or a new `explain/subsystem.ts`):
+
+| Function | Purpose |
+|---|---|
+| `synthesizeSubsystemExplanation(region, parent)` | Returns the explanation string |
+| `healthStatus(score)` | Returns the status phrase |
+| `renderSubsystemExplanation(scoped, score)` | Assembles all five sections |
+| `renderSubsystemExplanationJson(scoped, score)` | Returns the JSON object |
+
+Primary files to change:
+
+- `ix-cli/src/cli/commands/subsystems.ts`
+ Add the new flag, target validation, second score lookup, text rendering, and JSON rendering.
+- `ix-cli/src/client/api.ts`
+ Likely no required API changes for v1, but tighten typings for `listSubsystems()` and
+ `getSubsystemMap()` if the implementation needs stricter data contracts.
+- `memory-layer/src/main/scala/ix/memory/api/SubsystemRoutes.scala`
+ No new endpoint required for v1. Only change this file if revision metadata is added later.
+- `memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala`
+ No required change for v1 text mode. Only change this file if scoped JSON needs extra fields
+ or if you choose to expose `map_rev`.
+- `memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala`
+ Only required if you want the stale-health note in v1 by exposing score revision metadata.
+
+### Phase 2 — Optional score metadata
+
+Only needed if stale-health notes are required:
+
+1. Extend stored score payloads to include `map_rev` or score creation revision.
+2. Extend `listScores()` so the CLI can compare the score revision to the current scoped region.
+3. Add the stale-health note only when the score metadata proves drift.
+
+Suggested file targets:
+
+- `memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala`
+- `memory-layer/src/main/scala/ix/memory/api/SubsystemRoutes.scala`
+- `ix-cli/src/client/api.ts`
+- `ix-cli/src/cli/commands/subsystems.ts`
+
+### Phase 3 — Validation
+
+- Smoke test against two regions: one with a health score, one without.
+- Confirm `--format json` produces valid parseable output.
+- Confirm `--explain` + ambiguous target shows the error, not a stack trace.
+- Confirm `--explain` without a target produces the intended CLI error.
+- If stale-health metadata is implemented, verify both fresh and stale note paths.
+
+---
+
+## Large-Repo Ingestion Failures
+
+This is separate from `subsystems --explain`, but it is the other major issue discovered while
+reviewing the current architecture and should be fixed in parallel.
+
+### Failure Mode 1 — all parse results are retained in memory before any write
+
+Current behavior:
+- `BulkIngestionService.ingestPath()` discovers all files.
+- Loads all existing hashes for all files.
+- Parses all files with `parTraverseN`.
+- Keeps every `Parsed(FileBatch)` in memory.
+- Only then starts chunked writes.
+
+That makes the current chunked commit logic too late to protect memory.
+
+Required change:
+- Convert ingestion into a bounded pipeline:
+ 1. discover files
+ 2. slice into file batches
+ 3. load hashes for that slice
+ 4. parse only that slice
+ 5. commit only that slice
+ 6. release memory before moving to the next slice
+
+Primary file:
+- `memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala`
+
+### Failure Mode 2 — source hash lookup is one giant query
+
+Current behavior:
+- `getSourceHashes(sourceUris)` is called with the full discovered file list.
+- The query scans `patches`, sorts by revision, and groups by URI in one shot.
+
+On large repos, the bind payload and query work both become too large.
+
+Required change:
+- Chunk source-hash lookups to the same batch size as ingestion.
+- Keep hash lookup local to each parse/commit slice.
+- If needed later, add an indexed materialized source-hash table rather than scanning `patches`.
+
+Primary files:
+- `memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala`
+- `memory-layer/src/main/scala/ix/memory/db/ArangoGraphQueryApi.scala`
+
+### Failure Mode 3 — discovery is broader than parser support
+
+Current behavior:
+- File discovery includes many docs, configs, scripts, build files, and other text-like files.
+- Parser routing only has dedicated parsers for Python, TypeScript, Scala, config, and Markdown.
+- Unsupported files fall back to generic text ingestion.
+
+That means large repos do not just ingest code; they also ingest a wide tail of raw text.
+
+Required change:
+- Narrow default discovery to parser-supported file types for normal ingestion.
+- Optionally add an explicit `--include-generic-text` or similar opt-in later.
+- At minimum, stop treating unsupported text as full-content graph facts by default.
+
+Primary files:
+- `memory-layer/src/main/scala/ix/memory/ingestion/FileDiscovery.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/ParserRouter.scala`
+- `memory-layer/src/main/scala/ix/memory/api/IngestionRoutes.scala`
+- `ix-cli/src/client/api.ts`
+
+### Failure Mode 4 — generic text path stores full content and explodes patch size
+
+Current behavior:
+- Generic text fallback stores `"content"` in entity attrs.
+- `GraphPatchBuilder` converts attrs into claims.
+- Bulk ingestion persists node docs, claim docs, and the full patch document.
+
+This duplicates large text payloads across multiple persisted artifacts.
+
+Required change:
+- Remove full raw content from generic-text attrs by default.
+- Replace it with compact metadata only:
+ - file name
+ - line count
+ - maybe a short preview
+ - content fingerprint / source hash
+- Do not emit claims for raw file content.
+
+Primary files:
+- `memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/GraphPatchBuilder.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/ParseResult.scala`
+
+### Failure Mode 5 — chunking is by file count, not payload size
+
+Current behavior:
+- Bulk writes use a fixed `chunkSize = 100`.
+- A 100-file batch can still serialize into an oversized bulk request.
+
+Required change:
+- Chunk by estimated payload size, not just file count.
+- Keep a file-count cap as a secondary guardrail.
+- Emit progress based on committed slices, not only discovered totals.
+
+Primary files:
+- `memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala`
+- `memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala`
+- `memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala`
+
+### Recommended ingestion order
+
+1. Refactor bulk ingestion into bounded discovery/hash/parse/commit slices.
+2. Chunk `getSourceHashes()` by slice.
+3. Remove raw-content attrs and claims from generic text ingestion.
+4. Narrow default discovery to parser-supported files.
+5. Add payload-size-aware commit chunking.
+
+This order addresses the actual failure mode first: unbounded memory retention.
+
+---
+
+## Constraints
+
+- No new server endpoints.
+- No LLM calls — explanations are template-driven, same as `ix explain`.
+- `--explain` is ignored when `--list` is also passed (error message).
+- `--explain` without a `<target>` is an error: `--explain requires a target.`
+- Output is intentionally terse — one screen of text for the full explanation.
+- Do not claim stale health in the output unless the score payload actually exposes revision
+ metadata that proves staleness.
+- Large-repo ingestion fixes should prefer bounded-memory behavior over preserving the current
+ exact batching structure.
diff --git a/ix-cli/src/cli/__tests__/subsystem-explain.test.ts b/ix-cli/src/cli/__tests__/subsystem-explain.test.ts
new file mode 100644
index 00000000..ce292954
--- /dev/null
+++ b/ix-cli/src/cli/__tests__/subsystem-explain.test.ts
@@ -0,0 +1,115 @@
+import { describe, expect, it } from "vitest";
+import {
+ renderSubsystemExplanationJson,
+ renderSubsystemExplanationText,
+ type ScopedSubsystemResult,
+ type ScopedSubsystemRegion,
+ type SubsystemScore,
+} from "../explain/subsystem.js";
+
+function region(overrides: Partial<ScopedSubsystemRegion> = {}): ScopedSubsystemRegion {
+ return {
+ id: overrides.id ?? "region-1",
+ label: overrides.label ?? "Ingestion Layer",
+ level: overrides.level ?? 2,
+ label_kind: overrides.label_kind ?? "subsystem",
+ parent_id: overrides.parent_id ?? "parent-1",
+ file_count: overrides.file_count ?? 34,
+ confidence: overrides.confidence ?? 0.88,
+ is_cross_cutting: overrides.is_cross_cutting ?? false,
+ dominant_signals: overrides.dominant_signals ?? ["import coupling", "path proximity"],
+ children: overrides.children,
+ };
+}
+
+function scopedResult(overrides: Partial<ScopedSubsystemResult> = {}): ScopedSubsystemResult {
+ return {
+ target: overrides.target ?? region(),
+ parent: overrides.parent ?? {
+ id: "parent-1",
+ label: "Core System",
+ level: 3,
+ label_kind: "system",
+ },
+ summary: overrides.summary ?? {
+ well_defined: 3,
+ moderate: 1,
+ fuzzy: 0,
+ cross_cutting: 0,
+ },
+ children: overrides.children ?? [
+ region({ id: "child-1", label: "File Parser", level: 1, label_kind: "module", file_count: 12, confidence: 0.91 }),
+ region({ id: "child-2", label: "Patch Builder", level: 1, label_kind: "module", file_count: 9, confidence: 0.84 }),
+ region({ id: "child-3", label: "Bulk Writer", level: 1, label_kind: "module", file_count: 8, confidence: 0.66 }),
+ region({ id: "child-4", label: "Loader", level: 1, label_kind: "module", file_count: 5, confidence: 0.44 }),
+ ],
+ hierarchy: overrides.hierarchy ?? region({
+ children: overrides.children ?? [
+ region({ id: "child-1", label: "File Parser", level: 1, label_kind: "module", file_count: 12, confidence: 0.91 }),
+ ],
+ }),
+ };
+}
+
+function score(overrides: Partial<SubsystemScore> = {}): SubsystemScore {
+ return {
+ region_id: overrides.region_id ?? "region-1",
+ name: overrides.name ?? "Ingestion Layer",
+ level: overrides.level ?? 2,
+ label_kind: overrides.label_kind ?? "subsystem",
+ file_count: overrides.file_count ?? 34,
+ health_score: overrides.health_score ?? 0.74,
+ chunk_density: overrides.chunk_density ?? 5.1,
+ smell_rate: overrides.smell_rate ?? 0.06,
+ smell_files: overrides.smell_files ?? 2,
+ total_chunks: overrides.total_chunks ?? 173,
+ confidence: overrides.confidence ?? 0.88,
+ inference_version: overrides.inference_version ?? "subsystem_v1",
+ };
+}
+
+describe("subsystem explanation rendering", () => {
+ it("renders structured json with composition and health", () => {
+ const rendered = renderSubsystemExplanationJson(scopedResult(), score());
+
+ expect(rendered.resolvedTarget.label).toBe("Ingestion Layer");
+ expect(rendered.context.parent?.label).toBe("Core System");
+ expect(rendered.composition?.children).toHaveLength(4);
+ expect(rendered.health?.status).toBe("good");
+ expect(rendered.notes).toEqual([]);
+ expect(rendered.whyItMatters).toContain("ix impact");
+ });
+
+ it("omits health and adds a note when scores are missing", () => {
+ const rendered = renderSubsystemExplanationJson(scopedResult(), null);
+
+ expect(rendered.health).toBeNull();
+ expect(rendered.notes).toEqual(["Run 'ix subsystems' to compute health scores for this region."]);
+ });
+
+ it("flags cross-cutting fuzzy subsystems as drift risk", () => {
+ const rendered = renderSubsystemExplanationJson(
+ scopedResult({
+ target: region({
+ label: "Shared Auth",
+ confidence: 0.41,
+ is_cross_cutting: true,
+ }),
+ }),
+ score({ health_score: 0.39, smell_files: 3 }),
+ );
+
+ expect(rendered.whyItMatters).toContain("architectural drift");
+ });
+
+ it("renders text sections for a healthy subsystem explanation", () => {
+ const rendered = renderSubsystemExplanationText(scopedResult(), score());
+
+ expect(rendered).toContain("Explanation");
+ expect(rendered).toContain("Context");
+ expect(rendered).toContain("Composition");
+ expect(rendered).toContain("Health");
+ expect(rendered).toContain("Why it matters");
+ expect(rendered).toContain("Ingestion Layer");
+ });
+});
diff --git a/ix-cli/src/cli/commands/ingest.ts b/ix-cli/src/cli/commands/ingest.ts
index 966fa21f..0b53ed2e 100644
--- a/ix-cli/src/cli/commands/ingest.ts
+++ b/ix-cli/src/cli/commands/ingest.ts
@@ -1,6 +1,7 @@
import * as nodePath from 'node:path';
import * as fs from 'node:fs';
import * as crypto from 'node:crypto';
+import * as os from 'node:os';
import type { Command } from 'commander';
import chalk from 'chalk';
import { IxClient } from '../../client/api.js';
@@ -21,6 +22,15 @@ import {
// File discovery
// ---------------------------------------------------------------------------
+// Inline extension set — mirrors core-ingestion/dist/languages.js EXT_MAP.
+// Kept here so file discovery does NOT require loading tree-sitter grammars.
+const SUPPORTED_EXTENSIONS = new Set([
+ '.ts', '.tsx', '.js', '.jsx', '.mjs', '.cjs',
+ '.py', '.java', '.c', '.h', '.cpp', '.cc', '.cxx', '.hpp',
+ '.cs', '.go', '.rb', '.rs', '.php', '.kt', '.kts', '.swift',
+ '.scala', '.sc',
+]);
+
const IGNORE_DIRS = new Set([
'node_modules', '.git', 'dist', 'build', 'target', '.next',
'.cache', '__pycache__', '.ix', '.claude', '.gitnexus',
@@ -31,7 +41,6 @@ const MAX_FILE_BYTES = 1024 * 1024; // 1 MB
function* walkFiles(
dir: string,
recursive: boolean,
- supportsFile: (fileName: string) => boolean
): Generator<string> {
let entries: fs.Dirent[];
try { entries = fs.readdirSync(dir, { withFileTypes: true }); }
@@ -41,9 +50,9 @@ function* walkFiles(
if (IGNORE_DIRS.has(entry.name)) continue;
const full = nodePath.join(dir, entry.name);
if (entry.isDirectory()) {
- if (recursive) yield* walkFiles(full, true, supportsFile);
+ if (recursive) yield* walkFiles(full, true);
} else if (entry.isFile()) {
- if (supportsFile(entry.name)) yield full;
+ if (SUPPORTED_EXTENSIONS.has(nodePath.extname(entry.name))) yield full;
}
}
}
@@ -52,6 +61,42 @@ function sha256(content: Buffer): string {
return crypto.createHash('sha256').update(content).digest('hex');
}
+// ---------------------------------------------------------------------------
+// Mtime cache — skip readFileSync+sha256 for unchanged files
+// ---------------------------------------------------------------------------
+
+interface MtimeCache {
+ root: string;
+ files: Record<string, number>; // absolute path → mtime (ms)
+}
+
+function mtimeCachePath(projectRoot: string): string {
+ const key = crypto.createHash('sha256').update(projectRoot).digest('hex').slice(0, 12);
+ return nodePath.join(os.homedir(), '.ix', `ingest_mtimes_${key}.json`);
+}
+
+function loadMtimeCache(projectRoot: string): Map<string, number> {
+ try {
+ const raw = fs.readFileSync(mtimeCachePath(projectRoot), 'utf-8');
+ const data = JSON.parse(raw) as MtimeCache;
+ if (data.root !== projectRoot) return new Map();
+ return new Map(Object.entries(data.files));
+ } catch {
+ return new Map();
+ }
+}
+
+function saveMtimeCache(projectRoot: string, mtimes: Map<string, number>): void {
+ try {
+ const dir = nodePath.join(os.homedir(), '.ix');
+ fs.mkdirSync(dir, { recursive: true });
+ const data: MtimeCache = { root: projectRoot, files: Object.fromEntries(mtimes) };
+ fs.writeFileSync(mtimeCachePath(projectRoot), JSON.stringify(data));
+ } catch {
+ // Non-critical: ignore write errors
+ }
+}
+
// ---------------------------------------------------------------------------
// Progress bar
// ---------------------------------------------------------------------------
@@ -118,9 +163,6 @@ export async function ingestFiles(
const debug = opts.debug || process.env.IX_DEBUG === '1';
const trueStart = performance.now();
- const [{ parseFile, resolveEdges, isGrammarSupported }, { buildPatchWithResolution }] = await loadIngestionModules();
- const moduleLoadMs = Math.round(performance.now() - trueStart);
-
const resolvedPath = nodePath.isAbsolute(path)
? path
: nodePath.resolve(resolveWorkspaceRoot(opts.root), path);
@@ -131,8 +173,17 @@ export async function ingestFiles(
let progressPhase = 'Scanning';
let progressCurrent = 0;
let progressTotal = 0;
+ let progressStart = performance.now();
+
const interval = opts.format === 'text' ? setInterval(() => {
- process.stderr.write('\r' + renderProgressLine(progressPhase, progressCurrent, progressTotal));
+ const elapsed = performance.now() - progressStart;
+ const tau = progressPhase === 'Saving' ? 3000 : 2000;
+ let display = progressCurrent;
+ if (progressTotal > 0 && progressCurrent < progressTotal) {
+ const simulated = Math.floor((1 - Math.exp(-elapsed / tau)) * 0.88 * progressTotal);
+ display = Math.max(display, simulated);
+ }
+ process.stderr.write('\r' + renderProgressLine(progressPhase, display, progressTotal));
}, 80) : null;
let filesDiscovered = 0;
@@ -145,7 +196,7 @@ export async function ingestFiles(
let entitiesParsed = 0;
// Phase timing marks
- const timings = { moduleLoadMs, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, commitMs: 0 };
+ const timings = { moduleLoadMs: 0, discoverMs: 0, hashMs: 0, parseMs: 0, resolveMs: 0, commitMs: 0 };
const resolveStats = {
importLookups: 0, transitiveLookups: 0, globalFallbacks: 0,
globalCandidateTotal: 0, resolvedImport: 0, resolvedTransitive: 0,
@@ -156,8 +207,8 @@ export async function ingestFiles(
// Phase: discover files
const stat = fs.statSync(resolvedPath);
const filePaths: string[] = stat.isFile()
- ? [resolvedPath]
- : Array.from(walkFiles(resolvedPath, opts.recursive ?? true, isGrammarSupported));
+ ? (SUPPORTED_EXTENSIONS.has(nodePath.extname(resolvedPath)) ? [resolvedPath] : [])
+ : Array.from(walkFiles(resolvedPath, opts.recursive ?? true));
filesDiscovered = filePaths.length;
const discovered = performance.now();
@@ -166,120 +217,222 @@ export async function ingestFiles(
progressPhase = 'Parsing';
progressTotal = filePaths.length;
progressCurrent = 0;
-
- // Phase: hash lookup (skipped on first ingest when no baseline exists)
- let knownHashes: Map<string, string>;
- let baselineSkipped = false;
- if (opts.force) {
- knownHashes = new Map();
- } else {
- const hasBaseline = await checkIngestBaseline(client, debug);
- if (hasBaseline) {
- if (debug) process.stderr.write('\n Existing ingest baseline found — performing source hash comparison\n');
- knownHashes = await loadExistingHashes(client, filePaths, debug);
- } else {
- if (debug) process.stderr.write('\n No existing ingest baseline found — skipping source hash lookup\n');
- knownHashes = new Map();
- baselineSkipped = true;
+ progressStart = performance.now();
+
+ // Phase: mtime pre-filter — skip readFileSync+sha256 for files whose mtime
+ // is unchanged since the last successful ingest (common "ix map" re-run case).
+ const projectRoot = fs.statSync(resolvedPath).isDirectory() ? resolvedPath : nodePath.dirname(resolvedPath);
+ const mtimeCache = opts.force ? new Map<string, number>() : loadMtimeCache(projectRoot);
+ const currentMtimes = new Map<string, number>();
+
+ // DB-reset guard: if the mtime cache has entries but the server returns no hashes
+ // for a small sample, the DB was wiped (e.g. `ix reset` run from a different dir
+ // so the cache wasn't cleared). Invalidate the cache so files are re-ingested.
+ if (!opts.force && mtimeCache.size > 0) {
+ const samplePaths = [...mtimeCache.keys()].slice(0, 5);
+ const sampleHashes = await loadExistingHashes(client, samplePaths, debug);
+ if (sampleHashes.size === 0) {
+ mtimeCache.clear();
+ if (debug) process.stderr.write(`\n DB reset detected — invalidating mtime cache\n`);
}
}
- const hashed = performance.now();
- timings.hashMs = Math.round(hashed - discovered);
-
- // Phase: parse all files
- type ParsedFile = { filePath: string; parsed: any; hash: string; previousHash: string | undefined };
- const parsedFiles: ParsedFile[] = [];
+ // Stat all files and partition into mtime-clean (skip) and mtime-changed (need hash check).
+ const mtimeChangedPaths: string[] = [];
for (const filePath of filePaths) {
- progressCurrent++;
- if (interval) process.stderr.write('\r' + renderProgressLine(progressPhase, progressCurrent, progressTotal));
try {
- const fileSize = fs.statSync(filePath).size;
- if (fileSize === 0) { filesSkipped++; continue; }
- if (fileSize > MAX_FILE_BYTES) { tooLarge++; continue; }
-
- const bytes = fs.readFileSync(filePath);
- const hash = sha256(bytes);
-
- if (!opts.force && knownHashes.get(filePath) === hash) {
- filesSkipped++;
- continue;
- }
-
- const source = bytes.toString('utf-8');
- const parsed = parseFile(filePath, source);
-
- if (!parsed) {
- filesSkipped++;
- continue;
+ const st = fs.statSync(filePath);
+ if (st.size === 0) { filesSkipped++; continue; }
+ if (st.size > MAX_FILE_BYTES) { tooLarge++; continue; }
+ const mtime = st.mtimeMs;
+ currentMtimes.set(filePath, mtime);
+ if (!opts.force && mtimeCache.get(filePath) === mtime) {
+ filesSkipped++; // mtime clean — assume unchanged
+ } else {
+ mtimeChangedPaths.push(filePath);
}
-
- entitiesParsed += parsed.entities.length;
- const previousHash = knownHashes.get(filePath);
- parsedFiles.push({ filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined });
} catch (err) {
parseErrors++;
- process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);
+ process.stderr.write(`\n [stat error] ${filePath}: ${err}\n`);
}
}
- filesChanged = parsedFiles.length;
- const parsed = performance.now();
- timings.parseMs = Math.round(parsed - hashed);
-
- // Phase: cross-file CALLS + EXTENDS resolution over the full batch
- const resolvedEdges = (resolveEdges as Function)(parsedFiles.map(f => f.parsed), resolveStats);
- const resolved = performance.now();
- timings.resolveMs = Math.round(resolved - parsed);
-
- // Pre-group resolved edges by source file for O(1) lookup in patch building
- const edgesByFile = new Map<string, typeof resolvedEdges>();
- for (const edge of resolvedEdges) {
- let arr = edgesByFile.get(edge.srcFilePath);
- if (!arr) { arr = []; edgesByFile.set(edge.srcFilePath, arr); }
- arr.push(edge);
+ // Phase: hash lookup — only needed when mtime-changed files exist.
+ let knownHashes: Map<string, string>;
+ if (opts.force || mtimeChangedPaths.length > 0) {
+ knownHashes = await loadExistingHashes(client, opts.force ? filePaths : mtimeChangedPaths, debug);
+ if (debug) process.stderr.write(`\n Source hash lookup: ${knownHashes.size} known hashes (${mtimeChangedPaths.length} mtime-changed)\n`);
+ } else {
+ // All files are mtime-clean — skip server round-trip entirely.
+ knownHashes = new Map();
+ if (debug) process.stderr.write(`\n All ${filePaths.length} files mtime-clean — skipping hash lookup\n`);
}
- const emptyEdges: typeof resolvedEdges = [];
+ const hashed = performance.now();
+ timings.hashMs = Math.round(hashed - discovered);
- // Phase: build patches and bulk commit
- const patches: GraphPatchPayload[] = [];
- for (const { parsed: p, hash, previousHash } of parsedFiles) {
- try {
- patches.push(buildPatchWithResolution(p, hash, edgesByFile.get(p.filePath) ?? emptyEdges, previousHash));
- } catch (err) {
- parseErrors++;
- process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);
+ // Phase: parse + commit — streaming to bound peak memory.
+ //
+ // Files are parsed in PARSE_STREAM_CHUNK-sized batches. After each batch,
+ // resolveEdges runs on that batch and patches are committed before the next
+ // batch is parsed. This keeps heap usage at O(chunk) instead of O(total).
+ //
+ // YIELD_EVERY yields the Node.js event loop so the progress bar timer can
+ // fire even during the synchronous tree-sitter parse calls.
+ //
+ // Two paths:
+ // A) Has baseline or mtime-changed files → pre-scan first, skip module load
+ // entirely if nothing changed (common "ix map" re-run case).
+ // B) No baseline / force → load modules, parse + stream-commit all files.
+
+ type ParsedFile = { filePath: string; parsed: any; hash: string; previousHash: string | undefined };
+ let resolveEdgesFn: Function | null = null;
+ let buildPatchFn: Function | null = null;
+
+ const PARSE_STREAM_CHUNK = 1000; // resolve + commit after this many parsed files
+ const COMMIT_CHUNK_SIZE = 200; // files per HTTP batch to server
+ const YIELD_EVERY = 100; // yield event loop every N files during parse
+
+ const emptyEdges: any[] = [];
+
+ /** Resolve edges within a batch, build patches, and commit in sub-chunks. */
+ const flushBatch = async (batch: ParsedFile[]): Promise<void> => {
+ if (batch.length === 0) return;
+ filesChanged += batch.length;
+
+ const resolvedEdges = resolveEdgesFn!(batch.map(f => f.parsed), resolveStats);
+ const batchEdgesByFile = new Map<string, any[]>();
+ for (const edge of resolvedEdges) {
+ let arr = batchEdgesByFile.get(edge.srcFilePath);
+ if (!arr) { arr = []; batchEdgesByFile.set(edge.srcFilePath, arr); }
+ arr.push(edge);
}
- }
- if (patches.length > 0) {
- progressPhase = 'Saving';
- progressTotal = patches.length;
- progressCurrent = 0;
- try {
- const result = await client.commitPatchBulk(patches);
- latestRev = result.rev;
- patchesApplied = patches.length;
- progressCurrent = patches.length;
- } catch (err) {
- // Fallback: try per-file commits if bulk endpoint fails
- if (debug) process.stderr.write(`\n [bulk commit failed, falling back to per-file] ${err}\n`);
- for (const patch of patches) {
+ for (let i = 0; i < batch.length; i += COMMIT_CHUNK_SIZE) {
+ const end = Math.min(i + COMMIT_CHUNK_SIZE, batch.length);
+ const patchChunk: GraphPatchPayload[] = [];
+ for (let j = i; j < end; j++) {
+ const { parsed: p, hash, previousHash } = batch[j];
try {
- const result = await client.commitPatch(patch);
- if (result.rev > latestRev) latestRev = result.rev;
- patchesApplied++;
+ patchChunk.push(buildPatchFn!(p, hash, batchEdgesByFile.get(p.filePath) ?? emptyEdges, previousHash));
+ } catch (err) {
+ parseErrors++;
+ process.stderr.write(`\n [patch build error] ${p.filePath}: ${err}\n`);
+ }
+ }
+ if (patchChunk.length === 0) continue;
+ try {
+ const result = await client.commitPatchBulk(patchChunk);
+ if (result.rev > latestRev) latestRev = result.rev;
+ patchesApplied += patchChunk.length;
+ } catch (err) {
+ if (debug) process.stderr.write(`\n [bulk chunk failed, falling back to per-file] ${err}\n`);
+ for (const patch of patchChunk) {
+ try {
+ const result = await client.commitPatch(patch);
+ if (result.rev > latestRev) latestRev = result.rev;
+ patchesApplied++;
+ } catch (commitErr) {
+ parseErrors++;
+ process.stderr.write(`\n [commit error] ${patch.source?.uri}: ${commitErr}\n`);
+ }
+ }
+ }
+ }
+ };
+
+ if ((knownHashes.size > 0 || mtimeChangedPaths.length === 0) && !opts.force) {
+ // Path A: has baseline or all mtime-clean → pre-scan to detect changes before loading modules.
+ // If nothing changed, module load is skipped entirely.
+ const changedPaths: Array<{ filePath: string; bytes: Buffer; hash: string; previousHash: string | undefined }> = [];
+ for (const filePath of mtimeChangedPaths) {
+ try {
+ const bytes = fs.readFileSync(filePath);
+ const hash = sha256(bytes);
+ if (knownHashes.get(filePath) === hash) { filesSkipped++; continue; }
+ const previousHash = knownHashes.get(filePath);
+ changedPaths.push({ filePath, bytes, hash, previousHash: previousHash !== hash ? previousHash : undefined });
+ } catch (err) {
+ parseErrors++;
+ process.stderr.write(`\n [read error] ${filePath}: ${err}\n`);
+ }
+ }
+
+ if (changedPaths.length > 0) {
+ const moduleStart = performance.now();
+ const [ingestion, patchBuilder] = await loadIngestionModules();
+ timings.moduleLoadMs = Math.round(performance.now() - moduleStart);
+ resolveEdgesFn = ingestion.resolveEdges as Function;
+ buildPatchFn = patchBuilder.buildPatchWithResolution as Function;
+
+ let batch: ParsedFile[] = [];
+ for (let i = 0; i < changedPaths.length; i++) {
+ const { filePath, bytes, hash, previousHash } = changedPaths[i];
+ try {
+ const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));
+ if (!parsed) { filesSkipped++; progressCurrent++; continue; }
+ entitiesParsed += parsed.entities.length;
+ batch.push({ filePath, parsed, hash, previousHash });
progressCurrent++;
- } catch (commitErr) {
+ } catch (err) {
parseErrors++;
- process.stderr.write(`\n [commit error] ${patch.source?.uri}: ${commitErr}\n`);
+ progressCurrent++;
+ process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);
}
+ if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));
+ if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }
}
+ await flushBatch(batch);
}
+ } else {
+ // Path B: no baseline (first ingest) or --force → load modules, then stream parse + commit.
+ const moduleStart = performance.now();
+ const [ingestion, patchBuilder] = await loadIngestionModules();
+ timings.moduleLoadMs = Math.round(performance.now() - moduleStart);
+ resolveEdgesFn = ingestion.resolveEdges as Function;
+ buildPatchFn = patchBuilder.buildPatchWithResolution as Function;
+
+ let batch: ParsedFile[] = [];
+ for (let i = 0; i < filePaths.length; i++) {
+ const filePath = filePaths[i];
+ try {
+ const fileSize = fs.statSync(filePath).size;
+ if (fileSize === 0) { filesSkipped++; progressCurrent++; continue; }
+ if (fileSize > MAX_FILE_BYTES) { tooLarge++; progressCurrent++; continue; }
+ const bytes = fs.readFileSync(filePath);
+ const hash = sha256(bytes);
+ if (!opts.force && knownHashes.get(filePath) === hash) { filesSkipped++; progressCurrent++; continue; }
+ const previousHash = knownHashes.get(filePath);
+ const parsed = (ingestion.parseFile as Function)(filePath, bytes.toString('utf-8'));
+ if (!parsed) { filesSkipped++; progressCurrent++; continue; }
+ entitiesParsed += parsed.entities.length;
+ batch.push({ filePath, parsed, hash, previousHash: previousHash !== hash ? previousHash : undefined });
+ progressCurrent++;
+ } catch (err) {
+ parseErrors++;
+ progressCurrent++;
+ process.stderr.write(`\n [parse error] ${filePath}: ${err}\n`);
+ }
+ if ((i + 1) % YIELD_EVERY === 0) await new Promise<void>(resolve => setImmediate(resolve));
+ if (batch.length >= PARSE_STREAM_CHUNK) { await flushBatch(batch); batch = []; }
+ }
+ await flushBatch(batch);
}
+ const parsed = performance.now();
+ timings.parseMs = Math.round(parsed - hashed);
+ timings.resolveMs = 0; // now interleaved with parse
+ timings.commitMs = 0; // now interleaved with parse
+
+ // Clear the parse bar
+ if (interval) process.stderr.write('\r' + ' '.repeat(PROG_LINE_WIDTH) + '\r');
+
const committed = performance.now();
- timings.commitMs = Math.round(committed - resolved);
+
+ // Persist mtime cache so next run can skip unchanged files quickly.
+ // Only save when no parse errors (avoid poisoning cache on partial failures).
+ if (!opts.force && parseErrors === 0 && currentMtimes.size > 0) {
+ saveMtimeCache(projectRoot, currentMtimes);
+ }
} finally {
if (interval) {
clearInterval(interval);
@@ -287,6 +440,8 @@ export async function ingestFiles(
process.stderr.write('\r' + renderProgressLine(progressPhase, progressTotal, progressTotal));
}
process.stderr.write('\r' + ' '.repeat(PROG_LINE_WIDTH) + '\r');
+ const elapsedSec = ((performance.now() - start) / 1000).toFixed(1);
+ process.stderr.write(chalk.dim(` Ingested in ${elapsedSec}s\n`));
}
}
@@ -350,15 +505,6 @@ export async function ingestFiles(
// Load existing hashes from the server for change detection
// ---------------------------------------------------------------------------
-async function checkIngestBaseline(client: IxClient, debug = false): Promise<boolean> {
- try {
- return await client.hasIngestBaseline();
- } catch (err) {
- if (debug) process.stderr.write(`\n [baseline check failed, falling back to full hash lookup] ${err}\n`);
- return true; // Safe fallback: assume baseline exists, do the full lookup
- }
-}
-
async function loadExistingHashes(client: IxClient, filePaths: string[], debug = false): Promise<Map<string, string>> {
try {
return await client.getSourceHashes(filePaths);
diff --git a/ix-cli/src/cli/commands/map.ts b/ix-cli/src/cli/commands/map.ts
index 8486aa0f..3388ba71 100644
--- a/ix-cli/src/cli/commands/map.ts
+++ b/ix-cli/src/cli/commands/map.ts
@@ -5,9 +5,8 @@ import { IxClient } from "../../client/api.js";
import { getEndpoint } from "../config.js";
import { bootstrap } from "../bootstrap.js";
import { ingestFiles } from "./ingest.js";
-import { parseBackendError, renderStructuredError } from "../errors.js";
-interface MapRegion {
+export interface MapRegion {
id: string;
label: string;
label_kind: string;
@@ -54,7 +53,7 @@ interface MapPersistence {
total_ops: number;
}
-interface MapResult {
+export interface MapResult {
file_count: number;
region_count: number;
levels: number;
@@ -66,6 +65,18 @@ interface MapResult {
persistence?: MapPersistence;
}
+type MapSortMode = "importance" | "confidence" | "size" | "alpha";
+export interface MapTextRenderOptions {
+ level?: string;
+ minConfidence: string;
+ maxItems: string;
+ allItems?: boolean;
+ sort: string;
+ graph?: boolean;
+ list?: boolean;
+ verbose?: boolean;
+}
+
export function registerMapCommand(program: Command): void {
program
.command("map [path]")
@@ -73,6 +84,11 @@ export function registerMapCommand(program: Command): void {
.option("--format <fmt>", "Output format (text|json)", "text")
.option("--level <n>", "Show only regions at this level (1=finest, higher=coarser)")
.option("--min-confidence <n>", "Only show regions above this confidence threshold (0-1)", "0")
+ .option("--max-items <n>", "Max items to show per section in text output (default: 10)", "10")
+ .option("--all-items", "Show all items in each section (overrides --max-items)")
+ .option("--sort <mode>", "Sort mode for text output (importance|confidence|size|alpha)", "importance")
+ .option("--graph", "Render the hierarchy as a graph/tree view (default)")
+ .option("--list", "Render the ranked list view instead of the default graph/tree view")
.option("--full", "Force full local map, bypassing automatic safety limits (advanced/testing)")
.option("--verbose", "Show raw confidence scores, crosscut scores, boundary ratios, and signals")
.addHelpText(
@@ -97,10 +113,15 @@ Examples:
ix map --format json
ix map --level 2
ix map --min-confidence 0.5
+ ix map --max-items 10
+ ix map --sort confidence
+ ix map --graph
+ ix map --list
+ ix map --all-items
ix map . --full
ix --debug map . --full`
)
- .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; full?: boolean; verbose?: boolean }) => {
+ .action(async (pathArg: string | undefined, opts: { format: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; list?: boolean; full?: boolean; verbose?: boolean }) => {
const cwd = pathArg ? resolve(pathArg) : process.cwd();
try {
@@ -120,31 +141,43 @@ Examples:
}
// Ingest the path before mapping so the graph is up to date
+ const ingestStart = performance.now();
await ingestFiles(cwd, { recursive: true, format: opts.format === "json" ? "json" : "text", printSummary: false });
+ const ingestMs = Math.round(performance.now() - ingestStart);
const client = new IxClient(getEndpoint());
- if (opts.format !== "json") {
- process.stderr.write(chalk.dim(" Computing map...\n"));
- }
+ const mapBarWidth = 25;
+ const mapStart = performance.now();
+ const mapInterval = opts.format !== "json" ? setInterval(() => {
+ const elapsed = performance.now() - mapStart;
+ const pct = 1 - Math.exp(-elapsed / 4000);
+ const filled = Math.round(pct * mapBarWidth);
+ const bar = chalk.cyan('█'.repeat(filled)) + chalk.dim('░'.repeat(mapBarWidth - filled));
+ const pctStr = chalk.cyan(`${Math.min(Math.round(pct * 100), 99)}%`.padStart(4));
+ process.stderr.write(`\r Computing map... ${bar} ${pctStr}`);
+ }, 80) : null;
let result: MapResult;
try {
result = await client.map({ full: opts.full }) as MapResult;
} catch (err: any) {
- const msg: string = err.message ?? "";
- const structured = parseBackendError(msg);
- if (structured) {
- renderStructuredError(structured);
- } else {
- console.error(chalk.red("Error:"), msg);
- }
+ if (mapInterval) { clearInterval(mapInterval); process.stderr.write('\r' + ' '.repeat(60) + '\r'); }
+ console.error(chalk.red("Error:"), err.message ?? err);
process.exitCode = 1;
return;
}
+ if (mapInterval) { clearInterval(mapInterval); process.stderr.write('\r' + ' '.repeat(60) + '\r'); }
+ const mapMs = Math.round(performance.now() - mapStart);
+ if (opts.format !== "json") {
+ process.stderr.write(chalk.dim(` ingest ${ingestMs}ms · map ${mapMs}ms\n`));
+ }
const minConf = parseFloat(opts.minConfidence ?? "0");
const levelFilter = opts.level ? parseInt(opts.level, 10) : null;
+ const parsedMaxItems = parseInt(opts.maxItems ?? "10", 10);
+ const maxItems = Number.isFinite(parsedMaxItems) && parsedMaxItems > 0 ? parsedMaxItems : 10;
+ const sortMode = normalizeSortMode(opts.sort);
let regions = result.regions;
if (levelFilter !== null) regions = regions.filter(r => r.level === levelFilter);
@@ -154,105 +187,82 @@ Examples:
console.log(JSON.stringify({ ...result, regions }, null, 2));
return;
}
+ renderMapText(result, cwd, opts);
+ });
+}
- // Text output
- console.log(
- `\n${chalk.bold("Architectural Map")} — ` +
- `${result.file_count} files · ${result.region_count} regions`
- );
+export function renderMapText(result: MapResult, cwd: string, opts: MapTextRenderOptions): void {
+ const minConf = parseFloat(opts.minConfidence ?? "0");
+ const levelFilter = opts.level ? parseInt(opts.level, 10) : null;
+ const parsedMaxItems = parseInt(opts.maxItems ?? "10", 10);
+ const maxItems = Number.isFinite(parsedMaxItems) && parsedMaxItems > 0 ? parsedMaxItems : 10;
+ const sortMode = normalizeSortMode(opts.sort);
+ const showGraph = !opts.list;
- if (result.outcome === "fast_local_completed") {
- console.log(chalk.yellow(" Large system detected") + chalk.dim(" — using Fast Map"));
- console.log(chalk.dim(" Reduced coupling model with full region hierarchy output."));
- }
+ let regions = result.regions;
+ if (levelFilter !== null) regions = regions.filter(r => r.level === levelFilter);
+ if (minConf > 0) regions = regions.filter(r => r.confidence >= minConf);
- if (regions.length === 0) {
- console.log(chalk.dim("\n No regions found matching filters."));
- return;
- }
+ console.log(
+ `\n${chalk.bold("Architectural Map")} — ` +
+ `${result.file_count} files · ${result.region_count} regions`
+ );
+ const topSystem = pickTopSystemName(result.regions, cwd);
+ if (topSystem) {
+ console.log(chalk.dim(`System: ${topSystem}`));
+ }
- // Build id→region lookup for parent label resolution
- const regionById = new Map(result.regions.map(r => [r.id, r]));
-
- const KIND_ORDER = ["system", "subsystem", "module"];
- const KIND_HEADERS: Record<string, string> = {
- system: "Systems",
- subsystem: "Subsystems",
- module: "Modules",
- };
- const CROSSCUT_THRESHOLD = 0.10;
- const RULE = chalk.dim("─".repeat(58));
-
- // Group by label_kind
- const byKind = new Map<string, MapRegion[]>();
- for (const r of regions) {
- if (!byKind.has(r.label_kind)) byKind.set(r.label_kind, []);
- byKind.get(r.label_kind)!.push(r);
- }
+ if (result.outcome === "fast_local_completed") {
+ console.log(chalk.yellow(" Large system detected") + chalk.dim(" — using Fast Map"));
+ console.log(chalk.dim(" Reduced coupling model with full region hierarchy output."));
+ }
- for (const kind of KIND_ORDER) {
- const kindRegions = byKind.get(kind);
- if (!kindRegions) continue;
-
- // Sort: systems/subsystems by file_count desc; modules: problematic first, then parent+size
- if (kind !== "module") {
- kindRegions.sort((a, b) => b.file_count - a.file_count);
- } else {
- const isProblematic = (r: MapRegion) =>
- r.crosscut_score > CROSSCUT_THRESHOLD || r.confidence < 0.50;
- kindRegions.sort((a, b) => {
- const ap = isProblematic(a) ? 1 : 0;
- const bp = isProblematic(b) ? 1 : 0;
- if (bp !== ap) return bp - ap;
- const aParent = a.parent_id ? (regionById.get(a.parent_id)?.label ?? "") : "";
- const bParent = b.parent_id ? (regionById.get(b.parent_id)?.label ?? "") : "";
- if (aParent !== bParent) return aParent.localeCompare(bParent);
- return b.file_count - a.file_count;
- });
- }
-
- const colHeader = kind === "module"
- ? chalk.dim(` ${"Name".padEnd(22)} ${"Files".padStart(8)} ${"Clarity".padEnd(13)} System`)
- : chalk.dim(` ${"Name".padEnd(22)} ${"Files".padStart(8)} Clarity`);
-
- console.log(`\n${chalk.bold(KIND_HEADERS[kind] ?? kind)}`);
- console.log(colHeader);
- console.log(RULE);
-
- if (opts.verbose) {
- for (const r of kindRegions) {
- const bar = confidenceBar(r.confidence);
- const name = chalk.bold(r.label.padEnd(22));
- const files = chalk.dim(`${r.file_count} files`.padStart(8));
- const confPct = chalk.cyan(`${Math.round(r.confidence * 100)}%`.padStart(4));
- const xcut = chalk.dim(`xcut=${r.crosscut_score.toFixed(2)}`);
- const br = chalk.dim(`br=${Math.min(r.boundary_ratio, 999.9).toFixed(1)}`);
- const signals = chalk.dim(r.dominant_signals.slice(0, 3).join("+") || "—");
- const parentLabel = r.parent_id ? regionById.get(r.parent_id)?.label : null;
- const parentStr = (kind === "module" && parentLabel) ? chalk.dim(` (${parentLabel})`) : "";
- const crossStr = r.crosscut_score > CROSSCUT_THRESHOLD ? chalk.yellow(" ⚠ cross-cutting") : "";
- console.log(` ${bar} ${name} ${files} ${confPct} ${xcut} ${br} ${signals}${parentStr}${crossStr}`);
- }
- } else {
- for (const r of kindRegions) {
- const label = confidenceLabel(r.confidence);
- const labelColor = r.confidence >= 0.75 ? chalk.green : r.confidence >= 0.50 ? chalk.yellow : chalk.red;
- const name = chalk.bold(r.label.padEnd(22));
- const files = chalk.dim(`${r.file_count} files`.padStart(8));
- const parentLabel = r.parent_id ? regionById.get(r.parent_id)?.label : null;
- const parentStr = (kind === "module" && parentLabel) ? chalk.dim(` (${parentLabel})`) : "";
- const crossStr = r.crosscut_score > CROSSCUT_THRESHOLD ? chalk.yellow(" ⚠ cross-cutting") : "";
- console.log(` ${name} ${files} ${labelColor(label.padEnd(13))}${parentStr}${crossStr}`);
- }
- }
- }
+ if (regions.length === 0) {
+ console.log(chalk.dim("\n No regions found matching filters."));
+ return;
+ }
+
+ const regionById = new Map(result.regions.map(r => [r.id, r]));
+ const CROSSCUT_THRESHOLD = 0.10;
+ const systemsCount = regions.filter(r => r.level === 3).length;
+ const subsystemsCount = regions.filter(r => r.level === 2).length;
+ const modulesCount = regions.filter(r => r.level === 1).length;
+ const wellDefined = regions.filter(r => r.confidence >= 0.75).length;
+ const moderate = regions.filter(r => r.confidence >= 0.50 && r.confidence < 0.75).length;
+ const fuzzy = regions.filter(r => r.confidence < 0.50).length;
+ const crossCutting = regions.filter(r => r.crosscut_score > CROSSCUT_THRESHOLD).length;
- console.log(chalk.dim(`\nLegend: cross-cutting = spans multiple subsystems.`));
- if (!opts.verbose) {
- console.log(chalk.dim(`Run 'ix map --verbose' for confidence scores and raw metrics.`));
+ console.log(chalk.dim(
+ `Scope: ${systemsCount} systems · ${subsystemsCount} subsystems · ${modulesCount} modules`
+ ));
+ console.log(chalk.dim(
+ `Clarity: ${wellDefined} well-defined · ${moderate} moderate · ${fuzzy} fuzzy · ${crossCutting} cross-cutting`
+ ));
+
+ if (levelFilter === null) {
+ if (showGraph) {
+ if (!opts.allItems) {
+ console.log(chalk.dim(`Showing up to ${maxItems} branches per level. Use --all-items to show everything.`));
}
- console.log();
- });
+ renderMapTree(regions, maxItems, Boolean(opts.allItems), Boolean(opts.verbose), sortMode);
+ } else {
+ if (!opts.allItems) {
+ console.log(chalk.dim(`Showing the top ${maxItems} subsystems and the top ${maxItems} modules drawn from those subsystems. Use --all-items to show everything.`));
+ }
+ renderRankedList(regions, regionById, maxItems, Boolean(opts.allItems), Boolean(opts.verbose), sortMode);
+ }
+ } else {
+ if (!opts.allItems) {
+ console.log(chalk.dim(`Showing up to ${maxItems} items. Use --all-items to show everything.`));
+ }
+ renderLevelList(regions, regionById, levelFilter, maxItems, Boolean(opts.allItems), Boolean(opts.verbose), sortMode);
+ }
+
+ console.log(chalk.dim(`\nLegend: cross-cutting = spans multiple subsystems.`));
+ if (!opts.verbose) {
+ console.log(chalk.dim(`Run 'ix map --verbose' for confidence scores and raw metrics. Use --list for the ranked view.`));
+ }
+ console.log();
}
/** Render a confidence score as a compact bar: ████░░ (used in --verbose mode) */
@@ -269,3 +279,272 @@ function confidenceLabel(conf: number): string {
if (conf >= 0.50) return "Moderate";
return "Fuzzy";
}
+
+function pickTopSystemName(regions: MapRegion[], cwd: string): string {
+ const systems = regions
+ .filter(r => r.level === 3 || r.label_kind === "system")
+ .slice()
+ .sort((a, b) => b.file_count - a.file_count || b.confidence - a.confidence);
+
+ if (systems.length > 0 && systems[0].label.trim().length > 0) {
+ return systems[0].label;
+ }
+
+ const inferred = cwd.split(/[\\/]/).filter(Boolean).pop() ?? "System";
+ const compact = inferred.replace(/[^a-zA-Z0-9]+/g, " ").trim();
+ if (compact.length === 0) return "System";
+ const parts = compact.split(/\s+/);
+ if (parts.some(p => p.toLowerCase() === "ix")) return "IX";
+ if (compact.length <= 4) return compact.toUpperCase();
+ return parts
+ .map(part => part[0].toUpperCase() + part.slice(1).toLowerCase())
+ .join(" ");
+}
+
+function normalizeSortMode(input: string | undefined): MapSortMode {
+ switch ((input ?? "importance").toLowerCase()) {
+ case "confidence":
+ return "confidence";
+ case "size":
+ return "size";
+ case "alpha":
+ return "alpha";
+ default:
+ return "importance";
+ }
+}
+
+function compareRegions(a: MapRegion, b: MapRegion, mode: MapSortMode): number {
+ if (mode === "alpha") {
+ return a.label.localeCompare(b.label) || b.file_count - a.file_count || b.confidence - a.confidence;
+ }
+ if (mode === "confidence") {
+ return b.confidence - a.confidence || b.file_count - a.file_count || a.label.localeCompare(b.label);
+ }
+ return b.file_count - a.file_count || b.confidence - a.confidence || a.label.localeCompare(b.label);
+}
+
+function sortRegions(regions: MapRegion[], mode: MapSortMode): MapRegion[] {
+ return [...regions].sort((a, b) => compareRegions(a, b, mode));
+}
+
+function renderMapTree(
+ regions: MapRegion[],
+ maxItems: number,
+ allItems: boolean,
+ verbose: boolean,
+ sortMode: MapSortMode,
+): void {
+ const regionById = new Map(regions.map(region => [region.id, region]));
+ const childrenByParent = new Map<string, MapRegion[]>();
+ for (const region of regions) {
+ if (!region.parent_id) continue;
+ const existing = childrenByParent.get(region.parent_id) ?? [];
+ existing.push(region);
+ childrenByParent.set(region.parent_id, existing);
+ }
+
+ const roots = sortRegions(
+ regions.filter(region => region.level === 3 || region.parent_id === null || !regionById.has(region.parent_id)),
+ sortMode,
+ );
+ const shownRoots = allItems ? roots : roots.slice(0, maxItems);
+
+ console.log(`\n${chalk.bold("Architecture Graph")}`);
+ console.log(chalk.dim(" System → subsystem → module"));
+
+ for (const root of shownRoots) {
+ console.log();
+ renderTreeNode(root, "", true, 0, childrenByParent, maxItems, allItems, verbose, sortMode);
+ }
+
+ if (!allItems && roots.length > shownRoots.length) {
+ const remaining = roots.length - shownRoots.length;
+ console.log(chalk.dim(`\n ... ${remaining} more top-level branches. Use --all-items to show all.`));
+ }
+}
+
+function renderTreeNode(
+ region: MapRegion,
+ prefix: string,
+ isLast: boolean,
+ depth: number,
+ childrenByParent: Map<string, MapRegion[]>,
+ maxItems: number,
+ allItems: boolean,
+ verbose: boolean,
+ sortMode: MapSortMode,
+): void {
+ const branch = depth === 0 ? "●" : isLast ? "└─" : "├─";
+ console.log(`${prefix}${branch} ${formatRegionLine(region, verbose, depth)}`);
+
+ const children = sortRegions(childrenByParent.get(region.id) ?? [], sortMode);
+ const shownChildren = allItems ? children : children.slice(0, maxItems);
+ const nextPrefix = depth === 0 ? " " : `${prefix}${isLast ? " " : "│ "}`;
+
+ shownChildren.forEach((child, index) => {
+ renderTreeNode(
+ child,
+ nextPrefix,
+ index === shownChildren.length - 1,
+ depth + 1,
+ childrenByParent,
+ maxItems,
+ allItems,
+ verbose,
+ sortMode,
+ );
+ });
+
+ if (!allItems && children.length > shownChildren.length) {
+ const remaining = children.length - shownChildren.length;
+ console.log(chalk.dim(`${nextPrefix}… ${remaining} more branches under ${region.label}`));
+ }
+}
+
+function renderRankedList(
+ regions: MapRegion[],
+ regionById: Map<string, MapRegion>,
+ maxItems: number,
+ allItems: boolean,
+ verbose: boolean,
+ sortMode: MapSortMode,
+): void {
+ const subsystems = sortRegions(regions.filter(r => r.level === 2), sortMode);
+ const shownSubsystems = allItems ? subsystems : subsystems.slice(0, maxItems);
+ const shownSubsystemIds = new Set(shownSubsystems.map(region => region.id));
+ const candidateModules = sortRegions(
+ regions.filter(r => r.level === 1 && (
+ shownSubsystemIds.size === 0 ||
+ (r.parent_id !== null && shownSubsystemIds.has(r.parent_id))
+ )),
+ sortMode,
+ );
+ const shownModules = allItems ? candidateModules : candidateModules.slice(0, maxItems);
+
+ if (shownSubsystems.length > 0) {
+ console.log(`\n${chalk.bold("Subsystems")}`);
+ printAlignedRows(shownSubsystems.map(region => ({ region, parentLabel: null })), verbose, false);
+ if (!allItems && subsystems.length > shownSubsystems.length) {
+ console.log(chalk.dim(` ... ${subsystems.length - shownSubsystems.length} more subsystems. Use --all-items to show all.`));
+ }
+ }
+
+ if (shownModules.length > 0) {
+ console.log(`\n${chalk.bold("Modules")}`);
+ printAlignedRows(
+ shownModules.map(region => ({
+ region,
+ parentLabel: region.parent_id ? regionById.get(region.parent_id)?.label ?? "Unknown" : "Unknown",
+ })),
+ verbose,
+ true,
+ );
+ if (!allItems && candidateModules.length > shownModules.length) {
+ console.log(chalk.dim(` ... ${candidateModules.length - shownModules.length} more modules from the shown subsystems. Use --all-items to show all.`));
+ }
+ }
+
+ if (shownSubsystems.length === 0 && shownModules.length === 0) {
+ console.log(chalk.dim("\n No ranked subsystem or module regions found."));
+ }
+}
+
+function renderLevelList(
+ regions: MapRegion[],
+ regionById: Map<string, MapRegion>,
+ level: number,
+ maxItems: number,
+ allItems: boolean,
+ verbose: boolean,
+ sortMode: MapSortMode,
+): void {
+ const label = level === 3 ? "Systems" : level === 2 ? "Subsystems" : "Modules";
+ const ranked = sortRegions(regions, sortMode);
+ const shown = allItems ? ranked : ranked.slice(0, maxItems);
+
+ console.log(`\n${chalk.bold(label)}`);
+ printAlignedRows(
+ shown.map(region => ({
+ region,
+ parentLabel: level === 1 && region.parent_id ? (regionById.get(region.parent_id)?.label ?? "Unknown") : null,
+ })),
+ verbose,
+ level === 1,
+ );
+
+ if (!allItems && ranked.length > shown.length) {
+ const remaining = ranked.length - shown.length;
+ console.log(chalk.dim(` ... ${remaining} more ${label.toLowerCase()}. Use --all-items to show all.`));
+ }
+}
+
+function printAlignedRows(
+ rows: Array<{ region: MapRegion; parentLabel: string | null }>,
+ verbose: boolean,
+ includeParent: boolean,
+): void {
+ const nameWidth = Math.max("Name".length, ...rows.map(({ region }) => region.label.length));
+ const filesWidth = Math.max("Files".length, ...rows.map(({ region }) => String(region.file_count).length));
+ const confidenceWidth = Math.max(
+ "Confidence".length,
+ ...rows.map(({ region }) => `${Math.round(region.confidence * 100)}% ${confidenceLabel(region.confidence)}`.length),
+ );
+ const parentWidth = includeParent
+ ? Math.max("Parent".length, ...rows.map(({ parentLabel }) => (parentLabel ?? "-").length))
+ : 0;
+ const crossWidth = Math.max("Cross".length, ...rows.map(({ region }) => region.crosscut_score > 0.10 ? 3 : 2));
+
+ let header = ` ${"Name".padEnd(nameWidth)} ${"Files".padStart(filesWidth)} ${"Confidence".padEnd(confidenceWidth)}`;
+ if (includeParent) {
+ header += ` ${"Parent".padEnd(parentWidth)}`;
+ }
+ header += ` ${"Cross".padEnd(crossWidth)}`;
+ console.log(chalk.dim(header));
+ console.log(chalk.dim(` ${"─".repeat(Math.max(20, visibleWidth(header.trim())))}`));
+
+ for (const { region, parentLabel } of rows) {
+ const confidenceText = `${Math.round(region.confidence * 100)}% ${confidenceLabel(region.confidence)}`;
+ const confidenceColor = region.confidence >= 0.75 ? chalk.green : region.confidence >= 0.50 ? chalk.yellow : chalk.red;
+ const crossText = region.crosscut_score > 0.10 ? "yes" : "no";
+
+ let line = ` ${chalk.bold(region.label.padEnd(nameWidth))} ${chalk.dim(String(region.file_count).padStart(filesWidth))} ${confidenceColor(confidenceText.padEnd(confidenceWidth))}`;
+ if (includeParent) {
+ line += ` ${chalk.dim((parentLabel ?? "-").padEnd(parentWidth))}`;
+ }
+ line += ` ${(crossText === "yes" ? chalk.yellow(crossText.padEnd(crossWidth)) : chalk.dim(crossText.padEnd(crossWidth)))}`;
+
+ if (verbose) {
+ line += chalk.dim(` br=${Math.min(region.boundary_ratio, 999.9).toFixed(1)} xcut=${region.crosscut_score.toFixed(2)}`);
+ }
+
+ console.log(line);
+ }
+}
+
+function visibleWidth(text: string): number {
+ return text.replace(/\x1B\[[0-9;]*m/g, "").length;
+}
+
+function formatRegionLine(region: MapRegion, verbose: boolean, depth = 0): string {
+ const clarity = confidenceLabel(region.confidence);
+ const clarityColor = region.confidence >= 0.75 ? chalk.green : region.confidence >= 0.50 ? chalk.yellow : chalk.red;
+ const confPct = Math.round(region.confidence * 100);
+ const crosscut = region.crosscut_score > 0.10 ? chalk.yellow(" shared") : "";
+ const levelTag = region.level === 3 ? "system" : region.level === 2 ? "subsystem" : "module";
+ const badge = chalk.bgBlackBright.white(` ${levelTag.toUpperCase()} `);
+ const signals = region.dominant_signals.slice(0, 2).join(" · ");
+
+ if (verbose) {
+ const bar = confidenceBar(region.confidence);
+ const metrics = chalk.dim(
+ `conf=${confPct}% br=${Math.min(region.boundary_ratio, 999.9).toFixed(1)} xcut=${region.crosscut_score.toFixed(2)}`
+ );
+ const signalText = signals.length > 0 ? chalk.dim(` ${signals}`) : "";
+ return `${badge} ${bar} ${chalk.bold(region.label)} ${chalk.dim(`${region.file_count} files`)} ${clarityColor(`${clarity} (${confPct}%)`)} ${metrics}${signalText}${crosscut}`;
+ }
+
+ const fileText = depth === 0 ? chalk.dim(`${region.file_count} files`) : chalk.dim(`${region.file_count}`);
+ const signalText = signals.length > 0 ? chalk.dim(` ${signals}`) : "";
+ return `${badge} ${chalk.bold(region.label)} ${fileText} ${clarityColor(`${clarity} ${confPct}%`)}${signalText}${crosscut}`;
+}
diff --git a/ix-cli/src/cli/commands/parse-pool.ts b/ix-cli/src/cli/commands/parse-pool.ts
new file mode 100644
index 00000000..324e2aba
--- /dev/null
+++ b/ix-cli/src/cli/commands/parse-pool.ts
@@ -0,0 +1,87 @@
+/**
+ * Bounded worker-thread pool for parallel file parsing.
+ * Uses the core-ingestion parse-worker entry compiled to dist/.
+ *
+ * - Bounded to `concurrency` simultaneous workers.
+ * - One crashed worker is replaced; its in-flight task resolves as null (treated as parse failure).
+ * - parse() returns a Promise that resolves to FileParseResult | null.
+ * - Results collected via Promise.all preserve input ordering.
+ */
+import { Worker } from 'node:worker_threads';
+
+type Task = {
+ filePath: string;
+ source: string;
+ resolve: (result: unknown) => void;
+};
+
+export class ParsePool {
+ private workers: Worker[] = [];
+ private idle: Worker[] = [];
+ private queue: Task[] = [];
+ private active = new Map<Worker, Task>();
+
+ constructor(private workerPath: string, private concurrency: number) {}
+
+ init(): void {
+ for (let i = 0; i < this.concurrency; i++) {
+ this.spawnWorker();
+ }
+ }
+
+ parse(filePath: string, source: string): Promise<unknown> {
+ return new Promise((resolve) => {
+ this.queue.push({ filePath, source, resolve });
+ this.drain();
+ });
+ }
+
+ async destroy(): Promise<void> {
+ await Promise.all(this.workers.map(w => w.terminate()));
+ this.workers = [];
+ this.idle = [];
+ }
+
+ private spawnWorker(): Worker {
+ const w = new Worker(this.workerPath);
+ w.on('message', (msg: { ok: boolean; result: unknown }) => this.onResult(w, msg));
+ w.on('error', (err) => this.onError(w, err));
+ this.workers.push(w);
+ this.idle.push(w);
+ return w;
+ }
+
+ private drain(): void {
+ while (this.idle.length > 0 && this.queue.length > 0) {
+ const w = this.idle.pop()!;
+ const task = this.queue.shift()!;
+ this.active.set(w, task);
+ w.postMessage({ filePath: task.filePath, source: task.source });
+ }
+ }
+
+ private onResult(w: Worker, msg: { ok: boolean; result: unknown }): void {
+ const task = this.active.get(w);
+ if (!task) return;
+ this.active.delete(w);
+ task.resolve(msg.ok ? msg.result : null);
+ this.idle.push(w);
+ this.drain();
+ }
+
+ private onError(w: Worker, _err: Error): void {
+ const task = this.active.get(w);
+ if (task) {
+ this.active.delete(w);
+ task.resolve(null); // isolate: failed file = null parse result
+ }
+ // Replace the crashed worker
+ const idx = this.workers.indexOf(w);
+ if (idx !== -1) {
+ w.terminate().catch(() => {});
+ this.workers.splice(idx, 1);
+ this.spawnWorker();
+ this.drain();
+ }
+ }
+}
diff --git a/ix-cli/src/cli/commands/reset.ts b/ix-cli/src/cli/commands/reset.ts
index e9cb4d00..696752f6 100644
--- a/ix-cli/src/cli/commands/reset.ts
+++ b/ix-cli/src/cli/commands/reset.ts
@@ -1,20 +1,42 @@
+import * as nodePath from "node:path";
+import * as fs from "node:fs";
+import * as crypto from "node:crypto";
+import * as os from "node:os";
import type { Command } from "commander";
import chalk from "chalk";
import { spawnSync } from "child_process";
import { IxClient } from "../../client/api.js";
import { getEndpoint } from "../config.js";
+function mtimeCachePath(projectRoot: string): string {
+ const key = crypto.createHash("sha256").update(projectRoot).digest("hex").slice(0, 12);
+ return nodePath.join(os.homedir(), ".ix", `ingest_mtimes_${key}.json`);
+}
+
+function clearMtimeCache(projectRoot: string): void {
+ try {
+ fs.rmSync(mtimeCachePath(projectRoot), { force: true });
+ } catch {
+ // Non-critical
+ }
+}
+
export function registerResetCommand(program: Command): void {
program
.command("reset")
- .description("Wipe all graph data (nodes + edges)")
+ .description("Wipe graph data")
.option("-y, --yes", "Skip confirmation prompt")
- .option("--ingest", "Re-ingest current directory after wiping")
- .action(async (opts: { yes?: boolean; ingest?: boolean }) => {
+ .option("--code", "Reset only code graph (files, functions, classes, regions); preserve goals, plans, tasks, bugs, and decisions")
+ .option("--ingest", "Re-run ix map after wiping (rebuilds the code graph)")
+ .action(async (opts: { yes?: boolean; code?: boolean; ingest?: boolean }) => {
+ const scope = opts.code ? "code graph" : "all graph data";
+ const warning = opts.code
+ ? "This will delete all code nodes and edges (files, functions, classes, regions).\nPlanning artifacts (goals, plans, tasks, bugs, decisions) will be preserved."
+ : "This will delete ALL nodes and edges including planning artifacts.";
+
if (!opts.yes) {
- process.stdout.write(
- chalk.yellow("This will delete all nodes and edges. Are you sure? (y/N) ")
- );
+ console.log(chalk.yellow(warning));
+ process.stdout.write(chalk.yellow(`Reset ${scope}? (y/N) `));
const answer = await new Promise<string>(resolve => {
process.stdin.setEncoding("utf8");
process.stdin.once("data", (chunk: string) => resolve(chunk.trim()));
@@ -28,8 +50,17 @@ export function registerResetCommand(program: Command): void {
const client = new IxClient(getEndpoint());
try {
- await client.reset();
- console.log(chalk.green("✓") + " Graph wiped.");
+ if (opts.code) {
+ await client.resetCode();
+ // Clear the mtime cache so the next ix map re-ingests all files
+ clearMtimeCache(process.cwd());
+ console.log(chalk.green("✓") + " Code graph wiped. Planning artifacts preserved.");
+ console.log(chalk.dim(" Run `ix map` to rebuild the code graph."));
+ } else {
+ await client.reset();
+ clearMtimeCache(process.cwd());
+ console.log(chalk.green("✓") + " Graph wiped.");
+ }
} catch (err: any) {
console.error(chalk.red("Error:"), err.message);
process.exitCode = 1;
@@ -37,8 +68,8 @@ export function registerResetCommand(program: Command): void {
}
if (opts.ingest) {
- console.log(chalk.dim("Re-ingesting..."));
- const result = spawnSync(process.argv[0], [process.argv[1], "ingest", "."], {
+ console.log(chalk.dim("Rebuilding..."));
+ const result = spawnSync(process.argv[0], [process.argv[1], "map"], {
stdio: "inherit",
cwd: process.cwd(),
});
diff --git a/ix-cli/src/cli/commands/smells.ts b/ix-cli/src/cli/commands/smells.ts
new file mode 100644
index 00000000..81445982
--- /dev/null
+++ b/ix-cli/src/cli/commands/smells.ts
@@ -0,0 +1,169 @@
+import type { Command } from "commander";
+import chalk from "chalk";
+import { IxClient } from "../../client/api.js";
+import { getEndpoint } from "../config.js";
+
+interface SmellCandidate {
+ file_id: string;
+ file: string;
+ smell: string;
+ confidence: number;
+ signals: Record<string, unknown>;
+ inference_version: string;
+}
+
+interface SmellReport {
+ rev: number;
+ run_at: string;
+ count: number;
+ candidates: SmellCandidate[];
+}
+
+const SMELL_LABELS: Record<string, string> = {
+ "has_smell.orphan_file": "Orphan File",
+ "has_smell.god_module": "God Module",
+ "has_smell.weak_component_member":"Weak Component",
+};
+
+export function registerSmellsCommand(program: Command): void {
+ program
+ .command("smells")
+ .description("Detect architecture smells in the codebase")
+ .option("--format <fmt>", "Output format (text|json)", "text")
+ .option("--orphan-max-connections <n>","Max connections for orphan files", "0")
+ .option("--god-module-chunks <n>", "Min chunks for god module", "20")
+ .option("--god-module-fan <n>", "Min fan-in/out for god module", "15")
+ .option("--weak-max-neighbors <n>", "Max neighbors for weak component", "1")
+ .option("--list", "List existing smell claims without rerunning")
+ .addHelpText("after", `
+Runs graph-query-based architecture smell detection and stores results as
+versioned claims (inference_version=smell_v1) on affected File nodes.
+
+Smells detected:
+ orphan_file Files with no import or call connections
+ god_module Files with high chunk count or fan-in/fan-out
+ weak_component_member Files connected to very few others
+
+Examples:
+ ix smells
+ ix smells --god-module-chunks 15 --god-module-fan 10
+ ix smells --format json
+ ix smells --list`)
+ .action(async (opts: {
+ format: string;
+ orphanMaxConnections: string;
+ godModuleChunks: string;
+ godModuleFan: string;
+ weakMaxNeighbors: string;
+ list?: boolean;
+ }) => {
+ const client = new IxClient(getEndpoint());
+
+ if (opts.list) {
+ let result: any;
+ try {
+ result = await client.listSmells();
+ } catch (err: any) {
+ console.error(chalk.red("Error:"), err.message);
+ process.exitCode = 1;
+ return;
+ }
+ if (opts.format === "json") {
+ console.log(JSON.stringify(result, null, 2));
+ return;
+ }
+ const smells = result.smells ?? [];
+ if (smells.length === 0) {
+ console.log(chalk.dim("No smell claims found. Run `ix smells` to detect."));
+ return;
+ }
+ console.log(`\n${chalk.bold("Smell Claims")} ${chalk.dim(`(${smells.length} total)`)}\n`);
+ for (const s of smells) {
+ const label = SMELL_LABELS[s.smell] ?? s.smell;
+ console.log(` ${chalk.yellow(label.padEnd(20))} ${chalk.dim(s.entity_id?.slice(0, 8))} ${chalk.dim(s.inference_version ?? "")}`);
+ }
+ return;
+ }
+
+ if (opts.format !== "json") {
+ process.stderr.write(chalk.dim("Running smell detection...\n"));
+ }
+
+ let report: SmellReport;
+ try {
+ report = await client.runSmells({
+ orphanMaxConnections: parseInt(opts.orphanMaxConnections, 10),
+ godModuleChunks: parseInt(opts.godModuleChunks, 10),
+ godModuleFan: parseInt(opts.godModuleFan, 10),
+ weakMaxNeighbors: parseInt(opts.weakMaxNeighbors, 10),
+ }) as SmellReport;
+ } catch (err: any) {
+ console.error(chalk.red("Error:"), err.message);
+ process.exitCode = 1;
+ return;
+ }
+
+ if (opts.format === "json") {
+ console.log(JSON.stringify(report, null, 2));
+ return;
+ }
+
+ console.log(
+ `\n${chalk.bold("Architecture Smells")} ` +
+ chalk.dim(`rev ${report.rev} · ${report.count} candidates`)
+ );
+
+ if (report.count === 0) {
+ console.log(chalk.green("\n No smell candidates found."));
+ return;
+ }
+
+ // Group by smell kind
+ const byKind = new Map<string, SmellCandidate[]>();
+ for (const c of report.candidates) {
+ if (!byKind.has(c.smell)) byKind.set(c.smell, []);
+ byKind.get(c.smell)!.push(c);
+ }
+
+ for (const [kind, candidates] of byKind) {
+ const label = SMELL_LABELS[kind] ?? kind;
+ console.log(`\n${chalk.bold(label)} ${chalk.dim(`(${candidates.length})`)}`);
+
+ const sorted = [...candidates].sort((a, b) => b.confidence - a.confidence);
+ for (const c of sorted) {
+ const conf = confidenceBar(c.confidence);
+ const fileName = c.file.length > 45
+ ? "…" + c.file.slice(-44)
+ : c.file.padEnd(45);
+ const signalStr = formatSignals(c.smell, c.signals);
+ console.log(` ${conf} ${chalk.dim(fileName)} ${chalk.dim(signalStr)}`);
+ }
+ }
+
+ console.log(chalk.dim(`\nSmell claims stored with inference_version=smell_v1. Use 'ix smells --list' to retrieve.`));
+ });
+}
+
+function confidenceBar(conf: number): string {
+ const filled = Math.round(conf * 5);
+ const bar = "█".repeat(filled) + "░".repeat(5 - filled);
+ const color = conf >= 0.75 ? chalk.red : conf >= 0.5 ? chalk.yellow : chalk.dim;
+ return color(bar);
+}
+
+function formatSignals(smellKind: string, signals: Record<string, unknown>): string {
+ switch (smellKind) {
+ case "has_smell.orphan_file":
+ return `connections=${signals["connectivity"] ?? 0}`;
+ case "has_smell.god_module":
+ return [
+ signals["chunks"] !== undefined ? `chunks=${signals["chunks"]}` : null,
+ signals["fan_in"] !== undefined ? `fan_in=${signals["fan_in"]}` : null,
+ signals["fan_out"] !== undefined ? `fan_out=${signals["fan_out"]}` : null,
+ ].filter(Boolean).join(" ");
+ case "has_smell.weak_component_member":
+ return `neighbors=${signals["neighbor_count"] ?? 0}`;
+ default:
+ return "";
+ }
+}
diff --git a/ix-cli/src/cli/commands/subsystems.ts b/ix-cli/src/cli/commands/subsystems.ts
new file mode 100644
index 00000000..923e1f0b
--- /dev/null
+++ b/ix-cli/src/cli/commands/subsystems.ts
@@ -0,0 +1,384 @@
+import type { Command } from "commander";
+import chalk from "chalk";
+import { IxClient } from "../../client/api.js";
+import { getEndpoint } from "../config.js";
+import { renderMapText, type MapResult } from "./map.js";
+import {
+ renderSubsystemExplanationJson,
+ renderSubsystemExplanationText,
+ type ScopedSubsystemRegion,
+ type ScopedSubsystemResult,
+ type SubsystemScore,
+} from "../explain/subsystem.js";
+
+interface AmbiguousSubsystemResult {
+ error: "ambiguous_target";
+ target_query: string;
+ candidates: Array<{
+ pick: number;
+ id: string;
+ label: string;
+ level: number;
+ label_kind: string;
+ file_count: number;
+ parent?: string | null;
+ }>;
+}
+
+interface UnknownSubsystemTargetResult {
+ error: "unknown_target";
+ target_query: string;
+ message: string;
+ suggestions?: string[];
+}
+
+export function registerSubsystemsCommand(program: Command): void {
+ program
+ .command("subsystems [target]")
+ .description("Show the persisted architectural map saved by 'ix map'")
+ .option("--format <fmt>", "Output format (text|json)", "text")
+ .option("--list", "List stored subsystem health scores instead of the persisted architecture map")
+ .option("--target <target>", "Scope subsystem output to a persisted architecture region")
+ .option("--pick <n>", "Resolve an ambiguous region target by numbered candidate")
+ .option("--level <n>", "Filter to level (1=module, 2=subsystem, 3=system)")
+ .option("--min-confidence <n>", "Only show regions above this confidence threshold (0-1)", "0")
+ .option("--max-items <n>", "Max items to show per section in text output (default: 10)", "10")
+ .option("--all-items", "Show all items in each section (overrides --max-items)")
+ .option("--sort <mode>", "Sort mode for text output (importance|confidence|size|alpha)", "importance")
+ .option("--graph", "Render the hierarchy as a graph/tree view instead of the default ranked list")
+ .option("--verbose", "Show raw confidence scores, crosscut scores, boundary ratios, and signals")
+ .option("--explain", "Explain a scoped subsystem region in plain English")
+ .addHelpText("after", `
+Reads the persisted architectural map saved by 'ix map' from the graph and
+renders it with the same output formatter as 'ix map', without rerunning map
+ingestion or clustering.
+
+Examples:
+ ix subsystems
+ ix subsystems --list
+ ix subsystems api
+ ix subsystems --target "Cli / Client"
+ ix subsystems api --pick 2
+ ix subsystems api --explain
+ ix subsystems --level 2
+ ix subsystems --graph
+ ix subsystems --format json`)
+ .action(async (positionalTarget: string | undefined, opts: { format: string; list?: boolean; target?: string; pick?: string; level?: string; minConfidence: string; maxItems: string; allItems?: boolean; sort: string; graph?: boolean; verbose?: boolean; explain?: boolean }) => {
+ const client = new IxClient(getEndpoint());
+ const target = resolveSubsystemTarget(positionalTarget, opts.target);
+ const pick = parsePickOption(opts.pick);
+
+ if (!target.error && pick.error) {
+ console.error(chalk.red("Error:"), pick.error);
+ process.exitCode = 1;
+ return;
+ }
+ if (target.error) {
+ console.error(chalk.red("Error:"), target.error);
+ process.exitCode = 1;
+ return;
+ }
+ if (opts.list && target.value) {
+ console.error(chalk.red("Error:"), "Scoped targets are not supported with --list.");
+ process.exitCode = 1;
+ return;
+ }
+ if (opts.list && opts.explain) {
+ console.error(chalk.red("Error:"), "--explain cannot be used with --list.");
+ process.exitCode = 1;
+ return;
+ }
+ if (!target.value && pick.value !== undefined) {
+ console.error(chalk.red("Error:"), "--pick requires a target.");
+ process.exitCode = 1;
+ return;
+ }
+ if (opts.explain && !target.value) {
+ console.error(chalk.red("Error:"), "--explain requires a target.");
+ process.exitCode = 1;
+ return;
+ }
+
+ if (opts.list) {
+ let result: any;
+ try {
+ result = await client.listSubsystems();
+ } catch (err: any) {
+ console.error(chalk.red("Error:"), err.message);
+ process.exitCode = 1;
+ return;
+ }
+ const scores: SubsystemScore[] = result.scores ?? [];
+ const filtered = opts.level
+ ? scores.filter(s => s.level === parseInt(opts.level!, 10))
+ : scores;
+
+ if (opts.format === "json") {
+ console.log(JSON.stringify({ scores: filtered }, null, 2));
+ return;
+ }
+ printScores(filtered);
+ return;
+ }
+
+ let result: MapResult | ScopedSubsystemResult;
+ try {
+ result = await client.getSubsystemMap({
+ target: target.value,
+ pick: pick.value,
+ }) as MapResult;
+ } catch (err: any) {
+ const body = parseErrorBody(err);
+ if (body) {
+ if (opts.format === "json") {
+ console.log(JSON.stringify(body, null, 2));
+ process.exitCode = 1;
+ return;
+ }
+ renderSubsystemError(body, Boolean(opts.explain));
+ process.exitCode = 1;
+ return;
+ }
+ console.error(chalk.red("Error:"), err.message);
+ process.exitCode = 1;
+ return;
+ }
+
+ if (opts.explain) {
+ if (!isScopedSubsystemResult(result)) {
+ console.error(chalk.red("Error:"), "--explain requires a scoped subsystem target.");
+ process.exitCode = 1;
+ return;
+ }
+
+ let scoreResult: { scores?: SubsystemScore[] };
+ try {
+ scoreResult = await client.listSubsystems();
+ } catch (err: any) {
+ console.error(chalk.red("Error:"), err.message);
+ process.exitCode = 1;
+ return;
+ }
+
+ const score = (scoreResult.scores ?? []).find((candidate) => candidate.region_id === result.target.id) ?? null;
+ if (opts.format === "json") {
+ console.log(JSON.stringify(renderSubsystemExplanationJson(result, score), null, 2));
+ return;
+ }
+
+ console.log(renderSubsystemExplanationText(result, score));
+ return;
+ }
+
+ if (opts.format === "json") {
+ console.log(JSON.stringify(result, null, 2));
+ return;
+ }
+
+ if (isScopedSubsystemResult(result)) {
+ renderScopedSubsystemText(result, Boolean(opts.verbose));
+ return;
+ }
+
+ renderMapText(result, process.cwd(), {
+ level: opts.level,
+ minConfidence: opts.minConfidence,
+ maxItems: opts.maxItems,
+ allItems: opts.allItems,
+ sort: opts.sort,
+ graph: opts.graph,
+ verbose: opts.verbose,
+ });
+ });
+}
+
+function printScores(scores: SubsystemScore[]): void {
+ if (scores.length === 0) {
+ console.log(chalk.dim("No subsystem scores found. Run 'ix map' then 'ix subsystems'."));
+ return;
+ }
+
+ // Group by level
+ const byLevel = new Map<number, SubsystemScore[]>();
+ for (const s of scores) {
+ if (!byLevel.has(s.level)) byLevel.set(s.level, []);
+ byLevel.get(s.level)!.push(s);
+ }
+
+ const levelLabels: Record<number, string> = { 1: "Modules", 2: "Subsystems", 3: "Systems" };
+ const sorted = [...byLevel.entries()].sort((a, b) => b[0] - a[0]);
+
+ for (const [level, group] of sorted) {
+ const label = levelLabels[level] ?? `Level ${level}`;
+ console.log(`\n${chalk.bold(label)}`);
+
+ const ranked = [...group].sort((a, b) => b.health_score - a.health_score);
+ for (const s of ranked) {
+ const bar = healthBar(s.health_score);
+ const name = s.name.length > 32 ? s.name.slice(0, 31) + "…" : s.name.padEnd(32);
+ const files = chalk.dim(`${s.file_count}f`);
+ const chunks = chalk.dim(`${s.chunk_density.toFixed(1)}c/f`);
+ const smells = s.smell_files > 0
+ ? chalk.yellow(`${s.smell_files}⚠`)
+ : chalk.green("✓");
+ console.log(` ${bar} ${chalk.bold(name)} ${files} ${chunks} ${smells}`);
+ }
+ }
+}
+
+function healthBar(score: number): string {
+ const filled = Math.round(score * 5);
+ const bar = "█".repeat(filled) + "░".repeat(5 - filled);
+ const color = score >= 0.7 ? chalk.green : score >= 0.4 ? chalk.yellow : chalk.red;
+ return color(bar);
+}
+
+function parseErrorBody(err: { message?: string }): unknown | null {
+ const match = /^\d+: (.+)$/.exec(err.message ?? "");
+ if (!match) return null;
+ try {
+ return JSON.parse(match[1]);
+ } catch {
+ return null;
+ }
+}
+
+function isScopedSubsystemResult(value: unknown): value is ScopedSubsystemResult {
+ return typeof value === "object" && value !== null && "target" in value && "summary" in value && "children" in value;
+}
+
+function isAmbiguousSubsystemResult(value: unknown): value is AmbiguousSubsystemResult {
+ return typeof value === "object" && value !== null && (value as { error?: string }).error === "ambiguous_target";
+}
+
+function isUnknownSubsystemTargetResult(value: unknown): value is UnknownSubsystemTargetResult {
+ return typeof value === "object" && value !== null && (value as { error?: string }).error === "unknown_target";
+}
+
+function resolveSubsystemTarget(
+ positionalTarget: string | undefined,
+ explicitTarget: string | undefined,
+): { value?: string; error?: string } {
+ const positional = positionalTarget?.trim();
+ const explicit = explicitTarget?.trim();
+ if (positional && explicit && positional !== explicit) {
+ return { error: "Pass either a positional target or --target, not both." };
+ }
+ const value = positional || explicit;
+ return value ? { value } : {};
+}
+
+function parsePickOption(rawPick: string | undefined): { value?: number; error?: string } {
+ if (rawPick === undefined) return {};
+ const value = Number.parseInt(rawPick, 10);
+ if (!Number.isFinite(value) || value <= 0) {
+ return { error: "Invalid --pick value." };
+ }
+ return { value };
+}
+
+function renderSubsystemError(body: unknown, explain: boolean = false): void {
+ if (isAmbiguousSubsystemResult(body)) {
+ console.error(`Ambiguous target "${body.target_query}". Multiple architecture regions matched:\n`);
+ for (const candidate of body.candidates) {
+ const parent = candidate.parent ? ` parent: ${candidate.parent}` : "";
+ console.error(
+ `${candidate.pick}. ${candidate.label.padEnd(18)} ${candidate.label_kind.padEnd(11)} ${String(candidate.file_count).padStart(3)} files${parent}`
+ );
+ }
+ const explainFlag = explain ? " --explain" : "";
+ console.error(`\nRun:\n ix subsystems ${JSON.stringify(body.target_query)} --pick <n>${explainFlag}`);
+ return;
+ }
+
+ if (isUnknownSubsystemTargetResult(body)) {
+ console.error(body.message);
+ if (body.suggestions && body.suggestions.length > 0) {
+ console.error("\nTry:");
+ for (const suggestion of body.suggestions) {
+ console.error(`- ${suggestion}`);
+ }
+ }
+ return;
+ }
+
+ if (typeof body === "object" && body !== null && "error" in body) {
+ console.error(chalk.red("Error:"), String((body as { error: unknown }).error));
+ return;
+ }
+
+ console.error(chalk.red("Error:"), "Request failed.");
+}
+
+function renderScopedSubsystemText(result: ScopedSubsystemResult, verbose: boolean): void {
+ const target = result.target;
+ const lines = [
+ `${chalk.bold("Subsystem Scope")}: ${target.label}`,
+ `Level: ${target.label_kind}`,
+ result.parent ? `Parent: ${result.parent.label}` : null,
+ `Files: ${target.file_count}`,
+ `Confidence: ${target.confidence.toFixed(2)}`,
+ target.level === 1 ? `Cross-cutting: ${target.is_cross_cutting ? "yes" : "no"}` : null,
+ target.dominant_signals.length > 0 ? `Signals: ${target.dominant_signals.join(", ")}` : null,
+ ].filter((line): line is string => line !== null);
+
+ console.log(lines.join("\n"));
+ console.log(`\n${chalk.bold("Health")}:`);
+ console.log(
+ `${result.summary.well_defined} well-defined · ` +
+ `${result.summary.moderate} moderate · ` +
+ `${result.summary.fuzzy} fuzzy · ` +
+ `${result.summary.cross_cutting} cross-cutting`
+ );
+
+ console.log(`\n${chalk.bold("Focused Graph")}`);
+ console.log(chalk.dim(" Scoped architecture tree"));
+ console.log();
+ renderScopedTreeNode(result.hierarchy, "", true, 0, verbose);
+ console.log();
+}
+
+function renderScopedTreeNode(
+ region: ScopedSubsystemRegion,
+ prefix: string,
+ isLast: boolean,
+ depth: number,
+ verbose: boolean,
+): void {
+ const branch = depth === 0 ? "●" : isLast ? "└─" : "├─";
+ console.log(`${prefix}${branch} ${formatScopedRegionLine(region, verbose, depth)}`);
+
+ const children = [...(region.children ?? [])].sort(compareScopedRegions);
+ const nextPrefix = depth === 0 ? " " : `${prefix}${isLast ? " " : "│ "}`;
+ children.forEach((child, index) => {
+ renderScopedTreeNode(child, nextPrefix, index === children.length - 1, depth + 1, verbose);
+ });
+}
+
+function compareScopedRegions(a: ScopedSubsystemRegion, b: ScopedSubsystemRegion): number {
+ return b.file_count - a.file_count || b.confidence - a.confidence || a.label.localeCompare(b.label);
+}
+
+function formatScopedRegionLine(region: ScopedSubsystemRegion, verbose: boolean, depth: number): string {
+ const clarity = confidenceLabel(region.confidence);
+ const clarityColor = region.confidence >= 0.75 ? chalk.green : region.confidence >= 0.50 ? chalk.yellow : chalk.red;
+ const confPct = Math.round(region.confidence * 100);
+ const crosscut = region.is_cross_cutting ? chalk.yellow(" shared") : "";
+ const badge = chalk.bgBlackBright.white(` ${region.label_kind.toUpperCase()} `);
+ const signals = region.dominant_signals.slice(0, 2).join(" · ");
+
+ if (verbose) {
+ const signalText = signals.length > 0 ? chalk.dim(` ${signals}`) : "";
+ return `${badge} ${chalk.bold(region.label)} ${chalk.dim(`${region.file_count} files`)} ${clarityColor(`${clarity} (${confPct}%)`)}${signalText}${crosscut}`;
+ }
+
+ const fileText = depth === 0 ? chalk.dim(`${region.file_count} files`) : chalk.dim(`${region.file_count}`);
+ const signalText = signals.length > 0 ? chalk.dim(` ${signals}`) : "";
+ return `${badge} ${chalk.bold(region.label)} ${fileText} ${clarityColor(`${clarity} ${confPct}%`)}${signalText}${crosscut}`;
+}
+
+function confidenceLabel(confidence: number): string {
+ if (confidence >= 0.75) return "Well-defined";
+ if (confidence >= 0.50) return "Moderate";
+ return "Fuzzy";
+}
diff --git a/ix-cli/src/cli/errors.ts b/ix-cli/src/cli/errors.ts
index 50698358..5d73eddc 100644
--- a/ix-cli/src/cli/errors.ts
+++ b/ix-cli/src/cli/errors.ts
@@ -48,3 +48,47 @@ export function renderStructuredError(err: StructuredError): void {
}
console.error("");
}
+
+export class CliUsageError extends Error {
+ constructor(
+ message: string,
+ public readonly hint?: string,
+ ) {
+ super(message);
+ this.name = "CliUsageError";
+ }
+}
+
+export class CliResolutionError extends Error {
+ constructor(
+ message: string,
+ public readonly hint?: string,
+ public readonly detail?: string,
+ ) {
+ super(message);
+ this.name = "CliResolutionError";
+ }
+}
+
+export function renderCliError(err: unknown, debug = false): void {
+ if (err instanceof CliUsageError || err instanceof CliResolutionError) {
+ process.stderr.write(chalk.red(`Error: ${err.message}\n`));
+ if (err.hint) {
+ process.stderr.write(chalk.dim(`${err.hint}\n`));
+ }
+ if (debug && err instanceof CliResolutionError && err.detail) {
+ process.stderr.write(chalk.dim(`Detail: ${err.detail}\n`));
+ }
+ process.exit(1);
+ }
+
+ const e = err as any;
+ const msg = e?.message ?? String(err);
+ process.stderr.write(chalk.red(`Error: ${msg}\n`));
+
+ if (debug && e?.stack) {
+ process.stderr.write(chalk.dim(`${e.stack}\n`));
+ }
+
+ process.exit(1);
+}
diff --git a/ix-cli/src/cli/explain/subsystem.ts b/ix-cli/src/cli/explain/subsystem.ts
new file mode 100644
index 00000000..8fe19052
--- /dev/null
+++ b/ix-cli/src/cli/explain/subsystem.ts
@@ -0,0 +1,376 @@
+export interface SubsystemScore {
+ region_id: string;
+ name: string;
+ level: number;
+ label_kind: string;
+ file_count: number;
+ health_score: number;
+ chunk_density: number;
+ smell_rate: number;
+ smell_files: number;
+ total_chunks: number;
+ confidence: number;
+ inference_version: string;
+}
+
+export interface ScopedSubsystemRegion {
+ id: string;
+ label: string;
+ level: number;
+ label_kind: string;
+ parent_id: string | null;
+ file_count: number;
+ confidence: number;
+ is_cross_cutting: boolean;
+ dominant_signals: string[];
+ children?: ScopedSubsystemRegion[];
+}
+
+export interface ScopedSubsystemResult {
+ target: ScopedSubsystemRegion;
+ parent?: {
+ id: string;
+ label: string;
+ level: number;
+ label_kind: string;
+ } | null;
+ summary: {
+ well_defined: number;
+ moderate: number;
+ fuzzy: number;
+ cross_cutting: number;
+ };
+ children: ScopedSubsystemRegion[];
+ hierarchy: ScopedSubsystemRegion;
+}
+
+export interface SubsystemExplanationJson {
+ resolvedTarget: {
+ id: string;
+ label: string;
+ level: number;
+ label_kind: string;
+ };
+ explanation: string;
+ context: {
+ level: number;
+ label_kind: string;
+ parent: {
+ id: string;
+ label: string;
+ label_kind: string;
+ } | null;
+ file_count: number;
+ child_count: number;
+ confidence: number;
+ dominant_signals: string[];
+ is_cross_cutting: boolean;
+ };
+ composition: {
+ children: Array<{
+ label: string;
+ level: number;
+ label_kind: string;
+ file_count: number;
+ confidence: number;
+ is_cross_cutting: boolean;
+ }>;
+ summary: {
+ well_defined: number;
+ moderate: number;
+ fuzzy: number;
+ cross_cutting: number;
+ };
+ omitted_child_count: number;
+ } | null;
+ health: {
+ score: number;
+ chunk_density: number;
+ smell_rate: number;
+ smell_files: number;
+ total_chunks: number;
+ status: "healthy" | "good" | "moderate" | "needs_attention";
+ } | null;
+ whyItMatters: string;
+ notes: string[];
+}
+
+const MAX_COMPOSITION_CHILDREN = 10;
+
+export function renderSubsystemExplanationJson(
+ scoped: ScopedSubsystemResult,
+ score: SubsystemScore | null,
+): SubsystemExplanationJson {
+ const target = scoped.target;
+ const children = sortRegions(scoped.children);
+ const visibleChildren = children.slice(0, MAX_COMPOSITION_CHILDREN);
+ const notes = score ? [] : ["Run 'ix subsystems' to compute health scores for this region."];
+ const health = score ? renderHealthJson(score) : null;
+
+ return {
+ resolvedTarget: {
+ id: target.id,
+ label: target.label,
+ level: target.level,
+ label_kind: target.label_kind,
+ },
+ explanation: synthesizeSubsystemExplanation(target, scoped.parent ?? null),
+ context: {
+ level: target.level,
+ label_kind: target.label_kind,
+ parent: scoped.parent
+ ? {
+ id: scoped.parent.id,
+ label: scoped.parent.label,
+ label_kind: scoped.parent.label_kind,
+ }
+ : null,
+ file_count: target.file_count,
+ child_count: scoped.children.length,
+ confidence: target.confidence,
+ dominant_signals: target.dominant_signals,
+ is_cross_cutting: target.is_cross_cutting,
+ },
+ composition: scoped.children.length > 0
+ ? {
+ children: visibleChildren.map((child) => ({
+ label: child.label,
+ level: child.level,
+ label_kind: child.label_kind,
+ file_count: child.file_count,
+ confidence: child.confidence,
+ is_cross_cutting: child.is_cross_cutting,
+ })),
+ summary: { ...scoped.summary },
+ omitted_child_count: Math.max(children.length - visibleChildren.length, 0),
+ }
+ : null,
+ health,
+ whyItMatters: renderWhyItMatters(target, scoped.parent ?? null, score),
+ notes,
+ };
+}
+
+export function renderSubsystemExplanationText(
+ scoped: ScopedSubsystemResult,
+ score: SubsystemScore | null,
+): string {
+ const rendered = renderSubsystemExplanationJson(scoped, score);
+ const lines: string[] = [];
+
+ lines.push("Explanation");
+ lines.push(` ${rendered.explanation}`);
+ lines.push("");
+ lines.push("Context");
+ lines.push(` Level: ${rendered.context.label_kind} (level ${rendered.context.level})`);
+ if (rendered.context.parent) {
+ lines.push(` Parent: ${rendered.context.parent.label} (${rendered.context.parent.label_kind})`);
+ }
+ lines.push(` Files: ${rendered.context.file_count}`);
+ if (rendered.context.child_count > 0) {
+ lines.push(` Children: ${rendered.context.child_count} ${pluralize(rendered.context.child_count, childKindLabel(scoped.children))}`);
+ }
+ lines.push(` Confidence: ${confidenceLabel(rendered.context.confidence)} (${Math.round(rendered.context.confidence * 100)}%)`);
+ if (rendered.context.dominant_signals.length > 0) {
+ lines.push(` Signals: ${rendered.context.dominant_signals.join(" · ")}`);
+ }
+ if (rendered.context.is_cross_cutting) {
+ lines.push(" Scope: Cross-cutting");
+ }
+
+ if (rendered.composition) {
+ lines.push("");
+ lines.push("Composition");
+ for (const child of rendered.composition.children) {
+ lines.push(` ${formatCompositionLine(child)}`);
+ }
+ if (rendered.composition.omitted_child_count > 0) {
+ lines.push(` ... ${rendered.composition.omitted_child_count} more. Use 'ix subsystems ${JSON.stringify(scoped.target.label)}' for the full tree.`);
+ }
+ lines.push(
+ ` ${rendered.composition.summary.well_defined} well-defined · ` +
+ `${rendered.composition.summary.moderate} moderate · ` +
+ `${rendered.composition.summary.fuzzy} fuzzy · ` +
+ `${rendered.composition.summary.cross_cutting} cross-cutting`
+ );
+ }
+
+ if (rendered.health) {
+ lines.push("");
+ lines.push("Health");
+ lines.push(` Score: ${plainHealthBar(rendered.health.score)} ${rendered.health.score.toFixed(2)}`);
+ lines.push(` Chunk density: ${rendered.health.chunk_density.toFixed(1)} chunks/file`);
+ lines.push(` Smell rate: ${Math.round(rendered.health.smell_rate * 100)}% (${rendered.health.smell_files} files flagged)`);
+ lines.push(` Status: ${healthStatusPhrase(rendered.health.score)}${densityNote(rendered.health.chunk_density)}`);
+ }
+
+ lines.push("");
+ lines.push("Why it matters");
+ lines.push(` ${rendered.whyItMatters}`);
+
+ if (rendered.notes.length > 0) {
+ lines.push("");
+ lines.push(rendered.notes.length === 1 ? "Note" : "Notes");
+ for (const note of rendered.notes) {
+ lines.push(` ${note}`);
+ }
+ }
+
+ return lines.join("\n");
+}
+
+export function synthesizeSubsystemExplanation(
+ region: ScopedSubsystemRegion,
+ parent: ScopedSubsystemResult["parent"] | null,
+): string {
+ const regionNoun = articleFor(region.label_kind) + region.label_kind;
+ const parentPhrase = parent ? ` inside ${parent.label}` : "";
+ const confidencePhrase = confidenceSentence(region.confidence);
+ const crossCuttingPhrase = region.is_cross_cutting
+ ? " It is cross-cutting and spans multiple neighboring regions."
+ : "";
+ const signalPhrase = primarySignalSentence(region.dominant_signals);
+
+ return (
+ `\`${region.label}\` serves as ${regionNoun} spanning ${region.file_count} ` +
+ `${pluralize(region.file_count, "file")}${parentPhrase}. ` +
+ `It is ${confidencePhrase}.${signalPhrase}${crossCuttingPhrase}`
+ ).replace(/\s+/g, " ").trim();
+}
+
+function renderWhyItMatters(
+ region: ScopedSubsystemRegion,
+ parent: ScopedSubsystemResult["parent"] | null,
+ score: SubsystemScore | null,
+): string {
+ if (region.is_cross_cutting && region.confidence < 0.5) {
+ return (
+ `${region.label} spans multiple subsystems and has low clustering confidence, ` +
+ "which suggests architectural drift. Consider clarifying or decomposing its boundaries."
+ );
+ }
+
+ if (score && score.health_score >= 0.8 && region.level === 3) {
+ return (
+ `${region.label} is a stable, well-defined system boundary. Changes within it are usually ` +
+ `well-contained. Run \`ix impact ${JSON.stringify(region.label)}\` to review cross-system dependencies.`
+ );
+ }
+
+ if (score && score.health_score < 0.45 && score.smell_files > 0) {
+ return (
+ `${region.label} has ${score.smell_files} ${pluralize(score.smell_files, "file")} with active smell claims ` +
+ `and a health score of ${score.health_score.toFixed(2)}. Prioritize refactoring here before expanding the subsystem. ` +
+ "Run `ix smells` to review."
+ );
+ }
+
+ const parentPhrase = parent ? ` inside ${parent.label}` : "";
+ const densityPhrase = score
+ ? densityNote(score.chunk_density).trimStart()
+ : "";
+ return (
+ `${region.label} is a ${confidenceLabel(region.confidence).toLowerCase()} ${region.label_kind}${parentPhrase}. ` +
+ `Run \`ix impact ${JSON.stringify(region.label)}\` to understand its downstream blast radius.${densityPhrase}`
+ ).trim();
+}
+
+function renderHealthJson(score: SubsystemScore): NonNullable<SubsystemExplanationJson["health"]> {
+ return {
+ score: score.health_score,
+ chunk_density: score.chunk_density,
+ smell_rate: score.smell_rate,
+ smell_files: score.smell_files,
+ total_chunks: score.total_chunks,
+ status: healthStatus(score.health_score),
+ };
+}
+
+function healthStatus(score: number): "healthy" | "good" | "moderate" | "needs_attention" {
+ if (score >= 0.8) return "healthy";
+ if (score >= 0.65) return "good";
+ if (score >= 0.45) return "moderate";
+ return "needs_attention";
+}
+
+function healthStatusPhrase(score: number): string {
+ switch (healthStatus(score)) {
+ case "healthy":
+ return "Healthy - rich structure, minimal technical debt";
+ case "good":
+ return "Good - rich structure, low smell coverage";
+ case "moderate":
+ return "Moderate - structure is present, smell coverage is elevated";
+ case "needs_attention":
+ return "Needs attention - sparse structure or significant smell coverage";
+ }
+}
+
+function densityNote(chunkDensity: number): string {
+ if (chunkDensity >= 5.0) {
+ return " High chunk density indicates thorough code coverage in the knowledge graph.";
+ }
+ if (chunkDensity < 2.0) {
+ return " Low chunk density - run 'ix ingest' to improve coverage.";
+ }
+ return "";
+}
+
+function primarySignalSentence(signals: string[]): string {
+ if (signals.length === 0) return "";
+ if (signals.length === 1) {
+ return ` Its primary structural signal is ${signals[0]}.`;
+ }
+ return ` Its primary structural signals are ${signals.slice(0, 2).join(" and ")}.`;
+}
+
+function confidenceSentence(confidence: number): string {
+ if (confidence >= 0.75) return "well-defined";
+ if (confidence >= 0.5) return "moderately defined";
+ return "loosely defined";
+}
+
+function confidenceLabel(confidence: number): string {
+ if (confidence >= 0.75) return "Well-defined";
+ if (confidence >= 0.5) return "Moderate";
+ return "Fuzzy";
+}
+
+function plainHealthBar(score: number): string {
+ const filled = Math.round(score * 5);
+ return "█".repeat(filled) + "░".repeat(5 - filled);
+}
+
+function formatCompositionLine(region: {
+ label: string;
+ label_kind: string;
+ file_count: number;
+ confidence: number;
+ is_cross_cutting: boolean;
+}): string {
+ const kind = region.label_kind.toUpperCase().padEnd(9);
+ const label = region.label.padEnd(24);
+ const confidence = `${confidenceLabel(region.confidence)} ${Math.round(region.confidence * 100)}%`.padEnd(17);
+ const scope = region.is_cross_cutting ? " shared" : "";
+ return `● ${kind}${label}${confidence}${String(region.file_count).padStart(4)} ${pluralize(region.file_count, "file")}${scope}`;
+}
+
+function childKindLabel(children: ScopedSubsystemRegion[]): string {
+ const first = children[0]?.label_kind;
+ if (!first || !children.every((child) => child.label_kind === first)) {
+ return "child";
+ }
+ return first;
+}
+
+function sortRegions(children: ScopedSubsystemRegion[]): ScopedSubsystemRegion[] {
+ return [...children].sort((a, b) => b.file_count - a.file_count || b.confidence - a.confidence || a.label.localeCompare(b.label));
+}
+
+function pluralize(count: number, noun: string): string {
+ return count === 1 ? noun : `${noun}s`;
+}
+
+function articleFor(noun: string): string {
+ return /^[aeiou]/i.test(noun) ? "an " : "a ";
+}
diff --git a/ix-cli/src/cli/register/oss.ts b/ix-cli/src/cli/register/oss.ts
index 083b3f61..6839b6a1 100644
--- a/ix-cli/src/cli/register/oss.ts
+++ b/ix-cli/src/cli/register/oss.ts
@@ -29,6 +29,8 @@ import { registerMapCommand } from "../commands/map.js";
import { registerResetCommand } from "../commands/reset.js";
import { registerConfigCommand } from "../commands/config.js";
import { registerTraceCommand } from "../commands/trace.js";
+import { registerSmellsCommand } from "../commands/smells.js";
+import { registerSubsystemsCommand } from "../commands/subsystems.js";
import { registerUpgradeCommand } from "../commands/upgrade.js";
const PRO_COMMANDS: { name: string; desc: string }[] = [
@@ -84,6 +86,8 @@ export function registerOssCommands(program: Command): void {
registerDockerCommand(program);
registerWorkflowsHelpCommand(program);
registerMapCommand(program);
+ registerSmellsCommand(program);
+ registerSubsystemsCommand(program);
registerResetCommand(program);
registerConfigCommand(program);
registerTraceCommand(program);
diff --git a/ix-cli/src/client/api.ts b/ix-cli/src/client/api.ts
index eea5dd5a..c52fd7ab 100644
--- a/ix-cli/src/client/api.ts
+++ b/ix-cli/src/client/api.ts
@@ -179,23 +179,68 @@ export class IxClient {
return result.exists;
}
+ async commitPatchBatch(patches: GraphPatchPayload[]): Promise<PatchCommitResult[]> {
+ return this.post("/v1/patches/batch", patches);
+ }
+
async getSourceHashes(filePaths: string[]): Promise<Map<string, string>> {
const result = await this.post<Record<string, string>>('/v1/source-hashes', { uris: filePaths });
return new Map(Object.entries(result));
}
- async commitPatchBulk(patches: GraphPatchPayload[]): Promise<PatchCommitResult> {
- return this.post<PatchCommitResult>('/v1/patches/bulk', { patches });
+ async map(opts?: { full?: boolean }): Promise<any> {
+ return this.post("/v1/map", opts ?? {});
}
- async map(opts?: { full?: boolean }): Promise<any> {
- return this.post("/v1/map", { full: opts?.full });
+ async commitPatchBulk(patches: GraphPatchPayload[]): Promise<{ rev: number }> {
+ const results = await this.commitPatchBatch(patches);
+ const rev = results.reduce((max, r) => Math.max(max, r.rev ?? 0), 0);
+ return { rev };
+ }
+
+ async runSmells(opts?: {
+ orphanMaxConnections?: number;
+ godModuleChunks?: number;
+ godModuleFan?: number;
+ weakMaxNeighbors?: number;
+ }): Promise<any> {
+ const params = new URLSearchParams();
+ if (opts?.orphanMaxConnections !== undefined) params.set("orphan-max-connections", String(opts.orphanMaxConnections));
+ if (opts?.godModuleChunks !== undefined) params.set("god-module-chunks", String(opts.godModuleChunks));
+ if (opts?.godModuleFan !== undefined) params.set("god-module-fan", String(opts.godModuleFan));
+ if (opts?.weakMaxNeighbors !== undefined) params.set("weak-max-neighbors", String(opts.weakMaxNeighbors));
+ const qs = params.toString();
+ return this.post(qs ? `/v1/smells?${qs}` : "/v1/smells", {});
+ }
+
+ async listSmells(): Promise<any> {
+ return this.get("/v1/smells");
+ }
+
+ async scoreSubsystems(): Promise<any> {
+ return this.post("/v1/subsystems/score", {});
+ }
+
+ async listSubsystems(): Promise<any> {
+ return this.get("/v1/subsystems");
+ }
+
+ async getSubsystemMap(opts?: { target?: string; pick?: number }): Promise<any> {
+ const params = new URLSearchParams();
+ if (opts?.target) params.set("target", opts.target);
+ if (opts?.pick !== undefined) params.set("pick", String(opts.pick));
+ const qs = params.toString();
+ return this.get(`/v1/subsystems/map${qs ? `?${qs}` : ""}`);
}
async reset(): Promise<{ ok: boolean; message: string }> {
return this.post("/v1/reset", {});
}
+ async resetCode(): Promise<{ ok: boolean; message: string }> {
+ return this.post("/v1/reset/code", {});
+ }
+
async stats(): Promise<any> {
return this.get("/v1/stats");
}
diff --git a/memory-layer/src/main/scala/ix/memory/Main.scala b/memory-layer/src/main/scala/ix/memory/Main.scala
index 6e3366bc..9ce6d3db 100644
--- a/memory-layer/src/main/scala/ix/memory/Main.scala
+++ b/memory-layer/src/main/scala/ix/memory/Main.scala
@@ -11,6 +11,8 @@ import ix.memory.context._
import ix.memory.db._
import ix.memory.ingestion._
import ix.memory.map.MapService
+import ix.memory.smell.SmellService
+import ix.memory.subsystem.SubsystemScoringService
object Main extends IOApp.Simple {
@@ -44,9 +46,11 @@ object Main extends IOApp.Simple {
claimCollector, confidenceScorer, conflictDetector)
conflictService = new ConflictService(client, queryApi, writeApi)
mapService = new MapService(client, queryApi, writeApi)
+ smellService = new SmellService(client, writeApi)
+ subsystemService = new SubsystemScoringService(client, writeApi, mapService)
// 4. HTTP routes
- routes = Routes.all(contextService, ingestionService, bulkIngestionService, queryApi, writeApi, conflictService, client, mapService, bulkWriteApi)
+ routes = Routes.all(contextService, ingestionService, bulkIngestionService, queryApi, writeApi, conflictService, client, mapService, bulkWriteApi, smellService, subsystemService)
// 5. Server
server <- EmberServerBuilder
diff --git a/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala
new file mode 100644
index 00000000..d721f84e
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/api/MapEncoder.scala
@@ -0,0 +1,131 @@
+package ix.memory.api
+
+import io.circe.Json
+import io.circe.syntax._
+
+import ix.memory.map.{ArchitectureMap, Region, RegionMatch, ScopedSubsystemView}
+import ix.memory.model.NodeId
+
+/** Shared JSON encoding for ArchitectureMap, used by MapRoutes and SubsystemRoutes. */
+object MapEncoder {
+
+ def encodeMap(m: ArchitectureMap): Json = {
+ val regionsByLevel = m.regions.groupBy(_.level).toVector.sortBy(_._1)
+ Json.obj(
+ "file_count" -> m.fileCount.asJson,
+ "region_count" -> m.regions.size.asJson,
+ "levels" -> regionsByLevel.size.asJson,
+ "map_rev" -> m.mapRev.asJson,
+ "regions" -> m.regions.sortBy(r => (r.level, r.label)).map(encodeRegion).asJson,
+ "hierarchy" -> buildHierarchyTree(m).asJson
+ )
+ }
+
+ def encodeRegion(r: Region): Json =
+ Json.obj(
+ "id" -> r.id.value.toString.asJson,
+ "label" -> r.label.asJson,
+ "label_kind" -> r.labelKind.asJson,
+ "level" -> r.level.asJson,
+ "file_count" -> r.memberFiles.size.asJson,
+ "child_region_count" -> r.childRegionIds.size.asJson,
+ "parent_id" -> r.parentId.map(_.value.toString).asJson,
+ "cohesion" -> r.cohesion.asJson,
+ "external_coupling" -> r.externalCoupling.asJson,
+ "boundary_ratio" -> r.boundaryRatio.asJson,
+ "confidence" -> r.confidence.asJson,
+ "crosscut_score" -> r.crosscutScore.asJson,
+ "dominant_signals" -> r.dominantSignals.asJson,
+ "interface_node_count" -> r.interfaceNodeCount.asJson
+ )
+
+ def encodeScopedView(view: ScopedSubsystemView, isCrossCutting: Region => Boolean): Json =
+ Json.obj(
+ "target" -> encodeScopedRegion(view.target, isCrossCutting).asJson,
+ "parent" -> view.parent.map(parent =>
+ Json.obj(
+ "id" -> parent.id.value.toString.asJson,
+ "label" -> parent.label.asJson,
+ "level" -> parent.level.asJson,
+ "label_kind" -> parent.labelKind.asJson
+ )
+ ).asJson,
+ "summary" -> Json.obj(
+ "well_defined" -> view.summary.wellDefined.asJson,
+ "moderate" -> view.summary.moderate.asJson,
+ "fuzzy" -> view.summary.fuzzy.asJson,
+ "cross_cutting" -> view.summary.crossCutting.asJson
+ ),
+ "children" -> view.children.map(child => encodeScopedRegion(child, isCrossCutting)).asJson,
+ "hierarchy" -> buildScopedTree(view, isCrossCutting).asJson
+ )
+
+ def encodeAmbiguousTarget(target: String, candidates: Vector[RegionMatch], parentOf: Region => Option[Region]): Json =
+ Json.obj(
+ "error" -> "ambiguous_target".asJson,
+ "target_query" -> target.asJson,
+ "candidates" -> candidates.zipWithIndex.map { case (candidate, idx) =>
+ val region = candidate.region
+ Json.obj(
+ "pick" -> (idx + 1).asJson,
+ "id" -> region.id.value.toString.asJson,
+ "label" -> region.label.asJson,
+ "level" -> region.level.asJson,
+ "label_kind" -> region.labelKind.asJson,
+ "file_count" -> region.memberFiles.size.asJson,
+ "parent" -> parentOf(region).map(_.label).asJson
+ )
+ }.asJson
+ )
+
+ def buildHierarchyTree(m: ArchitectureMap): Json = {
+ val regionById: Map[NodeId, Region] = m.regions.map(r => r.id -> r).toMap
+
+ // Guard against cycles (can arise when Louvain levels have identical partitions)
+ def renderNode(r: Region, visited: Set[NodeId]): Json = {
+ val children =
+ if (visited.contains(r.id)) Vector.empty
+ else r.childRegionIds.toVector
+ .flatMap(cid => regionById.get(cid))
+ .filter(c => c.level < r.level)
+ .sortBy(_.label)
+ .map(c => renderNode(c, visited + r.id))
+
+ encodeRegion(r).deepMerge(Json.obj("children" -> children.asJson))
+ }
+
+ val roots = m.regions.filter(_.parentId.isEmpty).sortBy(-_.level)
+ roots.map(r => renderNode(r, Set.empty)).asJson
+ }
+
+ private def encodeScopedRegion(region: Region, isCrossCutting: Region => Boolean): Json =
+ Json.obj(
+ "id" -> region.id.value.toString.asJson,
+ "label" -> region.label.asJson,
+ "level" -> region.level.asJson,
+ "label_kind" -> region.labelKind.asJson,
+ "parent_id" -> region.parentId.map(_.value.toString).asJson,
+ "file_count" -> region.memberFiles.size.asJson,
+ "confidence" -> region.confidence.asJson,
+ "is_cross_cutting" -> isCrossCutting(region).asJson,
+ "dominant_signals" -> region.dominantSignals.asJson
+ )
+
+ private def buildScopedTree(view: ScopedSubsystemView, isCrossCutting: Region => Boolean): Json = {
+ val regionsById = view.subtree.map(region => region.id -> region).toMap
+
+ def renderNode(region: Region, visited: Set[NodeId]): Json = {
+ val children =
+ if (visited.contains(region.id)) Vector.empty
+ else region.childRegionIds.toVector
+ .flatMap(regionsById.get)
+ .filter(_.level < region.level)
+ .sortBy(child => (-child.memberFiles.size, -child.confidence, child.label.toLowerCase))
+ .map(child => renderNode(child, visited + region.id))
+
+ encodeScopedRegion(region, isCrossCutting).deepMerge(Json.obj("children" -> children.asJson))
+ }
+
+ renderNode(view.target, Set.empty)
+ }
+}
diff --git a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
index 96c06599..3189be0d 100644
--- a/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/MapRoutes.scala
@@ -1,128 +1,24 @@
package ix.memory.api
import cats.effect.IO
-import io.circe.{Decoder, Json}
-import io.circe.generic.semiauto.deriveDecoder
-import io.circe.syntax._
+import io.circe.Json
import org.http4s.HttpRoutes
import org.http4s.circe.CirceEntityCodec._
import org.http4s.dsl.io._
-import ix.memory.map.{ArchitectureMap, MapPreflightResult, MapService, PersistenceEstimate, Region}
-import ix.memory.model.NodeId
+import ix.memory.map.MapService
class MapRoutes(mapService: MapService) {
- private case class MapRequest(full: Option[Boolean])
- private implicit val mapReqDecoder: Decoder[MapRequest] = deriveDecoder
-
val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
/** POST /v1/map — Run the full map pipeline and return the hierarchy. */
case req @ POST -> Root / "v1" / "map" =>
(for {
- body <- req.as[MapRequest].handleErrorWith(_ => IO.pure(MapRequest(None)))
- archMap <- mapService.buildMap(forceFullLocal = body.full.getOrElse(false))
- resp <- Ok(encodeMap(archMap))
+ body <- req.as[Json].handleError(_ => Json.obj())
+ force = body.hcursor.get[Boolean]("full").toOption.getOrElse(false)
+ archMap <- mapService.buildMap(forceRecompute = force)
+ resp <- Ok(MapEncoder.encodeMap(archMap))
} yield resp).handleErrorWith(ErrorHandler.handle(_))
}
-
- private def encodeMap(m: ArchitectureMap): Json = {
- val regionsByLevel = m.regions.groupBy(_.level).toVector.sortBy(_._1)
-
- val base = Json.obj(
- "file_count" -> m.fileCount.asJson,
- "region_count" -> m.regions.size.asJson,
- "levels" -> regionsByLevel.size.asJson,
- "map_rev" -> m.mapRev.asJson,
- "regions" -> m.regions.sortBy(r => (r.level, r.label)).map(encodeRegion).asJson,
- "hierarchy" -> buildHierarchyTree(m).asJson
- )
-
- val withOutcome = base.deepMerge(Json.obj("outcome" -> m.outcome.label.asJson))
-
- val withPreflight = m.preflight match {
- case Some(pf) => withOutcome.deepMerge(Json.obj("preflight" -> encodePreflight(pf)))
- case None => withOutcome
- }
-
- m.persistenceEstimate match {
- case Some(pe) => withPreflight.deepMerge(Json.obj("persistence" -> encodePersistence(pe)))
- case None => withPreflight
- }
- }
-
- private def encodePreflight(pf: MapPreflightResult): Json =
- Json.obj(
- "cost" -> Json.obj(
- "file_count" -> pf.cost.fileCount.asJson,
- "directory_count" -> pf.cost.directoryCount.asJson,
- "directory_quadratic" -> pf.cost.directoryQuadratic.asJson,
- "symbol_estimate" -> pf.cost.symbolEstimate.asJson,
- "edge_estimate" -> pf.cost.edgeEstimate.asJson
- ),
- "capacity" -> Json.obj(
- "cpu_cores" -> pf.capacity.cpuCores.asJson,
- "heap_max_bytes" -> pf.capacity.heapMaxBytes.asJson,
- "heap_free_bytes" -> pf.capacity.heapFreeBytes.asJson,
- "container_memory" -> pf.capacity.containerMemory.asJson,
- "disk_free_bytes" -> pf.capacity.diskFreeBytes.asJson
- ),
- "risk" -> pf.risk.label.asJson,
- "mode" -> pf.mode.label.asJson,
- "warnings" -> pf.warnings.asJson,
- "duration_ms" -> pf.durationMs.asJson
- )
-
- private def encodePersistence(pe: PersistenceEstimate): Json =
- Json.obj(
- "region_nodes" -> pe.regionNodes.asJson,
- "file_edges" -> pe.fileEdges.asJson,
- "region_edges" -> pe.regionEdges.asJson,
- "delete_ops" -> pe.deleteOps.asJson,
- "total_ops" -> pe.totalOps.asJson
- )
-
- private def encodeRegion(r: Region): Json =
- Json.obj(
- "id" -> r.id.value.toString.asJson,
- "label" -> r.label.asJson,
- "label_kind" -> r.labelKind.asJson,
- "level" -> r.level.asJson,
- "file_count" -> r.memberFiles.size.asJson,
- "child_region_count" -> r.childRegionIds.size.asJson,
- "parent_id" -> r.parentId.map(_.value.toString).asJson,
- "cohesion" -> r.cohesion.asJson,
- "external_coupling" -> r.externalCoupling.asJson,
- "boundary_ratio" -> r.boundaryRatio.asJson,
- "confidence" -> r.confidence.asJson,
- "crosscut_score" -> r.crosscutScore.asJson,
- "dominant_signals" -> r.dominantSignals.asJson,
- "interface_node_count" -> r.interfaceNodeCount.asJson
- )
-
- /**
- * Build a tree structure: top-level regions contain nested children.
- * Returns a JSON array of root regions (no parent), each with a recursive
- * `children` field.
- */
- private def buildHierarchyTree(m: ArchitectureMap): Json = {
- val regionById: Map[NodeId, Region] = m.regions.map(r => r.id -> r).toMap
-
- // Guard against cycles (can arise when Louvain levels have identical partitions)
- def renderNode(r: Region, visited: Set[NodeId]): Json = {
- val children =
- if (visited.contains(r.id)) Vector.empty
- else r.childRegionIds.toVector
- .flatMap(cid => regionById.get(cid))
- .filter(c => c.level < r.level) // strict level ordering: child must be finer
- .sortBy(_.label)
- .map(c => renderNode(c, visited + r.id))
-
- encodeRegion(r).deepMerge(Json.obj("children" -> children.asJson))
- }
-
- val roots = m.regions.filter(_.parentId.isEmpty).sortBy(-_.level)
- roots.map(r => renderNode(r, Set.empty)).asJson
- }
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
index 4b2818e0..19d7e7e9 100644
--- a/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/PatchCommitRoutes.scala
@@ -1,16 +1,19 @@
package ix.memory.api
import cats.effect.IO
+import cats.syntax.traverse._
import io.circe.syntax._
import io.circe.Json
import org.http4s.HttpRoutes
import org.http4s.circe.CirceEntityCodec._
import org.http4s.dsl.io._
-import ix.memory.db.GraphWriteApi
-import ix.memory.model.GraphPatch
+import ix.memory.db.{BulkWriteApi, CommitResult, CommitStatus, FileBatch, GraphQueryApi, GraphWriteApi}
+import ix.memory.model.{GraphPatch, Rev}
-class PatchCommitRoutes(writeApi: GraphWriteApi) {
+class PatchCommitRoutes(writeApi: GraphWriteApi, bulkWriteApi: BulkWriteApi, queryApi: GraphQueryApi) {
+
+ private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
case req @ POST -> Root / "v1" / "patch" =>
@@ -22,5 +25,40 @@ class PatchCommitRoutes(writeApi: GraphWriteApi) {
"rev" -> result.newRev.value.asJson
))
} yield resp).handleErrorWith(ErrorHandler.handle(_))
+
+ // Batch endpoint: use BulkWriteApi (bulk document inserts, no per-patch transaction)
+ // instead of traverse(commitPatch) which was N serial ArangoDB transactions.
+ case req @ POST -> Root / "v1" / "patches" / "batch" =>
+ (for {
+ patches <- req.as[Vector[GraphPatch]]
+ latestRev <- queryApi.getLatestRev
+ result <- if (patches.isEmpty)
+ IO.pure(CommitResult(latestRev, CommitStatus.Ok))
+ else {
+ val fileBatches = patches.map(p => FileBatch(
+ filePath = p.source.uri,
+ sourceHash = p.source.sourceHash,
+ patch = p,
+ provenance = buildProvenanceMap(p)
+ ))
+ bulkWriteApi.commitBatchChunked(fileBatches, latestRev.value)
+ }
+ // Return one result entry per patch (same rev for all — one bulk revision)
+ resp <- Ok(patches.map(_ => Json.obj(
+ "status" -> result.status.toString.asJson,
+ "rev" -> result.newRev.value.asJson
+ )).asJson)
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+ }
+
+ private def buildProvenanceMap(patch: GraphPatch): java.util.Map[String, AnyRef] = {
+ val json = Json.obj(
+ "source_uri" -> Json.fromString(patch.source.uri),
+ "source_hash" -> patch.source.sourceHash.fold(Json.Null)(Json.fromString),
+ "extractor" -> Json.fromString(patch.source.extractor),
+ "source_type" -> Json.fromString(patch.source.sourceType.asJson.asString.getOrElse("code")),
+ "observed_at" -> Json.fromString(patch.timestamp.toString)
+ )
+ mapper.readValue(json.noSpaces, classOf[java.util.Map[String, AnyRef]])
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/PatchRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/PatchRoutes.scala
index b050aa46..2a33a4e4 100644
--- a/memory-layer/src/main/scala/ix/memory/api/PatchRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/PatchRoutes.scala
@@ -4,16 +4,24 @@ import java.util.UUID
import cats.effect.IO
import io.circe.syntax._
+import io.circe.{Decoder, Encoder}
+import io.circe.generic.semiauto._
import org.http4s.HttpRoutes
import org.http4s.circe.CirceEntityCodec._
import org.http4s.dsl.io._
import org.http4s.dsl.impl.OptionalQueryParamDecoderMatcher
-import ix.memory.db.ArangoClient
+import ix.memory.db.{ArangoClient, GraphQueryApi}
object LimitParam extends OptionalQueryParamDecoderMatcher[Int]("limit")
-class PatchRoutes(client: ArangoClient) {
+case class SourceHashesRequest(uris: List[String])
+object SourceHashesRequest {
+ implicit val decoder: Decoder[SourceHashesRequest] = deriveDecoder
+ implicit val encoder: Encoder[SourceHashesRequest] = deriveEncoder
+}
+
+class PatchRoutes(client: ArangoClient, queryApi: GraphQueryApi) {
val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
case GET -> Root / "v1" / "patches" :? LimitParam(limit) =>
@@ -46,5 +54,12 @@ class PatchRoutes(client: ArangoClient) {
)
resp <- Ok(patch)
} yield resp).handleErrorWith(ErrorHandler.handle(_))
+
+ case req @ POST -> Root / "v1" / "source-hashes" =>
+ (for {
+ body <- req.as[SourceHashesRequest]
+ hashes <- queryApi.getSourceHashes(body.uris)
+ resp <- Ok(hashes.asJson)
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/ResetRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/ResetRoutes.scala
index 9b1847e1..898d445b 100644
--- a/memory-layer/src/main/scala/ix/memory/api/ResetRoutes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/ResetRoutes.scala
@@ -19,5 +19,12 @@ class ResetRoutes(client: ArangoClient) {
_ <- client.truncateGraph()
r <- Ok(Json.obj("ok" -> true.asJson, "message" -> "Graph reset. All nodes and edges deleted.".asJson))
} yield r).handleErrorWith(ErrorHandler.handle(_))
+
+ /** POST /v1/reset/code — Delete only code-derived nodes/edges; preserve planning artifacts. */
+ case POST -> Root / "v1" / "reset" / "code" =>
+ (for {
+ _ <- client.truncateCodeGraph()
+ r <- Ok(Json.obj("ok" -> true.asJson, "message" -> "Code graph reset. Planning artifacts (goals, plans, tasks, bugs, decisions) preserved.".asJson))
+ } yield r).handleErrorWith(ErrorHandler.handle(_))
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/Routes.scala b/memory-layer/src/main/scala/ix/memory/api/Routes.scala
index 638b714f..376a16c9 100644
--- a/memory-layer/src/main/scala/ix/memory/api/Routes.scala
+++ b/memory-layer/src/main/scala/ix/memory/api/Routes.scala
@@ -13,6 +13,8 @@ import ix.memory.context.ContextService
import ix.memory.db.{ArangoClient, BulkWriteApi, GraphQueryApi, GraphWriteApi}
import ix.memory.ingestion.{BulkIngestionService, IngestionService}
import ix.memory.map.MapService
+import ix.memory.smell.SmellService
+import ix.memory.subsystem.SubsystemScoringService
object Routes {
@@ -25,7 +27,9 @@ object Routes {
conflictService: ConflictService,
client: ArangoClient,
mapService: MapService,
- bulkWriteApi: BulkWriteApi
+ bulkWriteApi: BulkWriteApi,
+ smellService: SmellService,
+ subsystemService: SubsystemScoringService
): HttpRoutes[IO] = {
val health = HttpRoutes.of[IO] {
@@ -42,21 +46,24 @@ object Routes {
val searchRoutes = new SearchRoutes(queryApi).routes
val truthRoutes = new TruthRoutes(writeApi, queryApi).routes
val goalRoutes = new GoalRoutes(writeApi, queryApi).routes
- val patchRoutes = new PatchRoutes(client).routes
+ val patchRoutes = new PatchRoutes(client, queryApi).routes
val decisionListRoutes = new DecisionRoutes(queryApi).routes
val expandRoutes = new ExpandRoutes(queryApi).routes
val statsRoutes = new StatsRoutes(client).routes
- val patchCommitRoutes = new PatchCommitRoutes(writeApi).routes
+ val patchCommitRoutes = new PatchCommitRoutes(writeApi, bulkWriteApi, queryApi).routes
val listRoutes = new ListRoutes(queryApi).routes
val mapRoutes = new MapRoutes(mapService).routes
val resetRoutes = new ResetRoutes(client).routes
val sourceHashRoutes = new SourceHashRoutes(queryApi).routes
val bulkPatchCommitRoutes = new BulkPatchCommitRoutes(bulkWriteApi, queryApi).routes
+ val smellRoutes = new SmellRoutes(smellService).routes
+ val subsystemRoutes = new SubsystemRoutes(subsystemService, mapService).routes
health <+> contextRoutes <+> ingestionRoutes <+> entityRoutes <+>
diffRoutes <+> conflictRoutes <+> decideRoutes <+> searchRoutes <+>
truthRoutes <+> patchRoutes <+> decisionListRoutes <+> expandRoutes <+>
statsRoutes <+> patchCommitRoutes <+> listRoutes <+> goalRoutes <+>
- mapRoutes <+> resetRoutes <+> sourceHashRoutes <+> bulkPatchCommitRoutes
+ mapRoutes <+> resetRoutes <+> sourceHashRoutes <+> bulkPatchCommitRoutes <+>
+ smellRoutes <+> subsystemRoutes
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/api/SmellRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/SmellRoutes.scala
new file mode 100644
index 00000000..2ed44f05
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/api/SmellRoutes.scala
@@ -0,0 +1,61 @@
+package ix.memory.api
+
+import cats.effect.IO
+import io.circe.Json
+import io.circe.syntax._
+import org.http4s.HttpRoutes
+import org.http4s.circe.CirceEntityCodec._
+import org.http4s.dsl.io._
+import org.http4s.dsl.impl.OptionalQueryParamDecoderMatcher
+
+import ix.memory.smell.{SmellConfig, SmellService}
+
+object OrphanThresholdParam extends OptionalQueryParamDecoderMatcher[Int]("orphan-max-connections")
+object GodChunkParam extends OptionalQueryParamDecoderMatcher[Int]("god-module-chunks")
+object GodFanParam extends OptionalQueryParamDecoderMatcher[Int]("god-module-fan")
+object WeakNeighborParam extends OptionalQueryParamDecoderMatcher[Int]("weak-max-neighbors")
+
+class SmellRoutes(smellService: SmellService) {
+
+ val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
+
+ /** POST /v1/smells — run smell detection and persist claims. */
+ case req @ POST -> Root / "v1" / "smells"
+ :? OrphanThresholdParam(orphanMax)
+ +& GodChunkParam(godChunks)
+ +& GodFanParam(godFan)
+ +& WeakNeighborParam(weakNeighbors) =>
+
+ val config = SmellConfig(
+ orphanMaxConnections = orphanMax.getOrElse(0),
+ godModuleChunkThreshold = godChunks.getOrElse(20),
+ godModuleFanThreshold = godFan.getOrElse(15),
+ weakComponentMaxNeighbors = weakNeighbors.getOrElse(1)
+ )
+ (for {
+ report <- smellService.run(config)
+ resp <- Ok(Json.obj(
+ "rev" -> report.rev.asJson,
+ "run_at" -> report.runAt.asJson,
+ "count" -> report.candidates.size.asJson,
+ "candidates" -> report.candidates.map { c =>
+ Json.obj(
+ "file_id" -> c.fileId.asJson,
+ "file" -> c.fileName.asJson,
+ "smell" -> c.smellKind.field.asJson,
+ "confidence" -> c.confidence.asJson,
+ "signals" -> c.signals.asJson,
+ "inference_version" -> "smell_v1".asJson
+ )
+ }.asJson
+ ))
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+
+ /** GET /v1/smells — retrieve existing smell claims without rerunning. */
+ case GET -> Root / "v1" / "smells" =>
+ (for {
+ smells <- smellService.listSmells()
+ resp <- Ok(Json.obj("smells" -> smells.asJson))
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+ }
+}
diff --git a/memory-layer/src/main/scala/ix/memory/api/SubsystemRoutes.scala b/memory-layer/src/main/scala/ix/memory/api/SubsystemRoutes.scala
new file mode 100644
index 00000000..91d95d59
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/api/SubsystemRoutes.scala
@@ -0,0 +1,118 @@
+package ix.memory.api
+
+import cats.effect.IO
+import io.circe.Json
+import io.circe.syntax._
+import org.http4s.HttpRoutes
+import org.http4s.circe.CirceEntityCodec._
+import org.http4s.dsl.io._
+
+import ix.memory.map.MapService
+import ix.memory.subsystem.SubsystemScoringService
+
+class SubsystemRoutes(
+ scoringService: SubsystemScoringService,
+ mapService: MapService
+) {
+
+ val routes: HttpRoutes[IO] = HttpRoutes.of[IO] {
+
+ /** GET /v1/subsystems/map — retrieve the persisted architecture map or a scoped region view. */
+ case req @ GET -> Root / "v1" / "subsystems" / "map" =>
+ val params = req.uri.query.params
+ val target = params.get("target").map(_.trim).filter(_.nonEmpty)
+ val pickRaw = params.get("pick").map(_.trim).filter(_.nonEmpty)
+ (for {
+ resp <- parsePick(pickRaw) match {
+ case Left(message) =>
+ BadRequest(Json.obj("error" -> message.asJson))
+ case Right(pick) =>
+ for {
+ archMap <- mapService.loadCachedMap().flatMap {
+ case Some(cached) => IO.pure(cached)
+ case None => mapService.getOrBuildMap()
+ }
+ response <- target match {
+ case None =>
+ if (pick.isDefined) {
+ BadRequest(Json.obj("error" -> "--pick requires a target.".asJson))
+ } else {
+ Ok(MapEncoder.encodeMap(archMap))
+ }
+ case Some(query) =>
+ val matches = mapService.resolveRegion(archMap, query)
+ if (matches.isEmpty) {
+ NotFound(Json.obj(
+ "error" -> "unknown_target".asJson,
+ "target_query" -> query.asJson,
+ "message" -> s"""No architecture region matched "$query".""".asJson,
+ "suggestions" -> Vector("ix subsystems", "ix map").asJson
+ ))
+ } else {
+ selectMatch(matches, pick) match {
+ case Left(message) =>
+ BadRequest(Json.obj("error" -> message.asJson))
+ case Right(None) =>
+ BadRequest(MapEncoder.encodeAmbiguousTarget(query, matches, region => mapService.parentOf(archMap, region)))
+ case Right(Some(selected)) =>
+ Ok(MapEncoder.encodeScopedView(mapService.scopedView(archMap, selected.region), mapService.isCrossCutting))
+ }
+ }
+ }
+ } yield response
+ }
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+
+ /** POST /v1/subsystems/score — run health scoring and persist claims. */
+ case POST -> Root / "v1" / "subsystems" / "score" =>
+ (for {
+ scores <- scoringService.score()
+ resp <- Ok(Json.obj(
+ "count" -> scores.size.asJson,
+ "scores" -> scores.map { s =>
+ Json.obj(
+ "region_id" -> s.regionId.asJson,
+ "name" -> s.regionName.asJson,
+ "level" -> s.level.asJson,
+ "label_kind" -> s.labelKind.asJson,
+ "file_count" -> s.fileCount.asJson,
+ "health_score" -> s.healthScore.asJson,
+ "chunk_density"-> s.chunkDensity.asJson,
+ "smell_rate" -> s.smellRate.asJson,
+ "smell_files" -> s.smellFiles.asJson,
+ "total_chunks" -> s.totalChunks.asJson,
+ "confidence" -> s.confidence.asJson,
+ "inference_version" -> "subsystem_v1".asJson
+ )
+ }.asJson
+ ))
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+
+ /** GET /v1/subsystems — retrieve stored health scores. */
+ case GET -> Root / "v1" / "subsystems" =>
+ (for {
+ scores <- scoringService.listScores()
+ resp <- Ok(Json.obj("scores" -> scores.asJson))
+ } yield resp).handleErrorWith(ErrorHandler.handle(_))
+ }
+
+ private def parsePick(pick: Option[String]): Either[String, Option[Int]] =
+ pick match {
+ case None => Right(None)
+ case Some(value) =>
+ scala.util.Try(value.toInt).toOption match {
+ case Some(parsed) if parsed > 0 => Right(Some(parsed))
+ case _ => Left("Invalid --pick value.")
+ }
+ }
+
+ private def selectMatch(matches: Vector[ix.memory.map.RegionMatch], pick: Option[Int]): Either[String, Option[ix.memory.map.RegionMatch]] =
+ pick match {
+ case None if matches.size == 1 => Right(matches.headOption)
+ case None => Right(None)
+ case Some(index) if index <= matches.size =>
+ Right(matches.lift(index - 1))
+ case Some(index) =>
+ Left(s"Invalid --pick value: $index\nThere are only ${matches.size} matches for the requested target.")
+ }
+}
diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
index ddee578a..7ddc575e 100644
--- a/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala
@@ -119,6 +119,37 @@ class ArangoClient private (db: ArangoDatabase) {
db.collection("conflict_sets").truncate()
()
}
+
+ /**
+ * Delete only code-derived nodes and edges, preserving planning artifacts
+ * (goals/intents, plans, tasks, decisions, bugs and their linking edges).
+ */
+ def truncateCodeGraph(): IO[Unit] = {
+ val codeKinds = Array(
+ "file", "function", "class", "method", "interface",
+ "module", "chunk", "region"
+ )
+ val codePredicates = Array(
+ "CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS",
+ "CONTAINS", "DEFINES", "CONTAINS_CHUNK", "REFERENCES", "NEXT", "IN_REGION"
+ )
+ for {
+ _ <- execute(
+ "FOR n IN nodes FILTER n.kind IN @kinds REMOVE n IN nodes",
+ Map("kinds" -> codeKinds.asInstanceOf[AnyRef])
+ )
+ _ <- execute(
+ "FOR e IN edges FILTER e.predicate IN @predicates REMOVE e IN edges",
+ Map("predicates" -> codePredicates.asInstanceOf[AnyRef])
+ )
+ // Also remove code-ingest patches so getSourceHashes returns empty after reset,
+ // preventing the ingest client from treating all files as unchanged.
+ _ <- execute(
+ "FOR p IN patches FILTER STARTS_WITH(p.data.source.extractor, 'tree-sitter') REMOVE p IN patches",
+ Map.empty
+ )
+ } yield ()
+ }
}
object ArangoClient {
diff --git a/memory-layer/src/main/scala/ix/memory/db/ArangoGraphWriteApi.scala b/memory-layer/src/main/scala/ix/memory/db/ArangoGraphWriteApi.scala
index 33f6c44e..5900f8fa 100644
--- a/memory-layer/src/main/scala/ix/memory/db/ArangoGraphWriteApi.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/ArangoGraphWriteApi.scala
@@ -3,6 +3,8 @@ package ix.memory.db
import java.time.Instant
import java.util.UUID
+import scala.collection.mutable
+
import cats.effect.IO
import cats.syntax.traverse._
import cats.syntax.foldable._
@@ -16,6 +18,22 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
// I1 fix: single ObjectMapper instance shared across all calls
private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
+ private val CurrentRevisionKey = "current"
+ private val SourceGraphRevisionKey = "source-graph"
+ private val SourceGraphIgnoredKinds: Set[NodeKind] = Set(
+ NodeKind.Module,
+ NodeKind.Config,
+ NodeKind.ConfigEntry,
+ NodeKind.Doc,
+ NodeKind.Decision,
+ NodeKind.Intent,
+ NodeKind.Bug,
+ NodeKind.Plan,
+ NodeKind.Task,
+ NodeKind.Goal,
+ NodeKind.Region
+ )
+ private val SourceGraphPredicates = Set("CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS")
// ── Collections involved in a commit transaction ──────────────────
@@ -58,7 +76,11 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
private def idempotencyKey(patchId: PatchId): String =
patchId.value.toString
- private def doCommit(patch: GraphPatch, txId: String): IO[CommitResult] =
+ private def doCommit(patch: GraphPatch, txId: String): IO[CommitResult] = {
+ val entityIdsForClaims = patch.ops.collect {
+ case PatchOp.AssertClaim(eid, _, _, _, _) => eid.value.toString
+ }.toSet
+
for {
// Step 2: Load latest_rev (inside transaction)
latestRev <- loadLatestRev(txId)
@@ -71,23 +93,28 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
IO.pure(CommitResult(Rev(latestRev), CommitStatus.BaseRevMismatch))
} else {
val newRev = latestRev + 1L
+ val advanceSourceGraphRev = shouldAdvanceSourceGraphRev(patch)
for {
+ // Step 4b: Pre-load active claims for all entities in this patch (one query)
+ // Eliminates per-claim hasIdenticalActiveClaim DB round-trips
+ claimCache <- loadActiveClaimsCache(entityIdsForClaims, txId)
// Step 5: Execute each op
- _ <- patch.ops.traverse_(op => executeOp(op, patch, newRev, txId))
- // Step 5b: Retire absent claims (claims from same extractor/entity no longer emitted)
- _ <- retireAbsentClaims(patch, newRev, txId)
+ _ <- patch.ops.traverse_(op => executeOp(op, patch, newRev, txId, claimCache))
+ // Step 5b: Retire absent claims — single batch query instead of one per entity
+ _ <- retireAbsentClaimsBatch(patch, newRev, txId)
// Step 6: Store patch
_ <- storePatch(patch, newRev, txId)
// Step 7: Store idempotency key
_ <- storeIdempotencyKey(patch.patchId, newRev, txId)
// Step 8: Update revision
- _ <- updateRevision(newRev, txId)
+ _ <- updateRevision(newRev, txId, advanceSourceGraphRev)
// Step 9: Commit the transaction
_ <- client.commitTransaction(txId)
} yield CommitResult(Rev(newRev), CommitStatus.Ok)
}
}
} yield result
+ }
private def loadLatestRev(txId: String): IO[Long] =
client.queryOne(
@@ -98,7 +125,7 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
txId = Some(txId)
).map(_.flatMap(_.as[Long].toOption).getOrElse(0L))
- private def executeOp(op: PatchOp, patch: GraphPatch, newRev: Long, txId: String): IO[Unit] = op match {
+ private def executeOp(op: PatchOp, patch: GraphPatch, newRev: Long, txId: String, claimCache: mutable.Set[(String, String, String)]): IO[Unit] = op match {
case PatchOp.UpsertNode(id, kind, name, attrs) =>
val logicalId = id.value.toString
val now = Instant.now().toString
@@ -273,41 +300,48 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
txId = Some(txId)
)
- case PatchOp.AssertClaim(entityId, field, value, confidence) =>
+ case PatchOp.AssertClaim(entityId, field, value, confidence, inferenceVersion) =>
val provenanceJson = buildProvenance(patch)
+ val eid = entityId.value.toString
+ val valueStr = value.noSpaces
for {
// Step 1: Retire active claims for same (entity_id, field) with different value
- _ <- retireClaims(entityId.value.toString, field, value, newRev, txId)
- // Step 2: Check if identical active claim already exists
- hasDuplicate <- hasIdenticalActiveClaim(entityId.value.toString, field, value, txId)
- // Step 3: Only insert if no duplicate
- _ <- if (hasDuplicate) IO.unit
+ _ <- retireClaims(eid, field, value, newRev, txId)
+ // Mirror retirement in the cache: remove entries for (eid, field) with different value
+ _ = claimCache.filterInPlace { case (e, f, v) => !(e == eid && f == field && v != valueStr) }
+ // Step 2: Check in-memory cache — avoids a DB round-trip per claim
+ _ <- if (claimCache.contains((eid, field, valueStr))) IO.unit
else {
val claimKey = UUID.randomUUID().toString
- client.execute(
- """INSERT {
- | _key: @key,
- | entity_id: @entity_id,
- | field: @field,
- | value: @value,
- | confidence: @confidence,
- | status: @status,
- | created_rev: @created_rev,
- | deleted_rev: null,
- | provenance: @provenance
- |} INTO claims""".stripMargin,
- Map(
- "key" -> claimKey.asInstanceOf[AnyRef],
- "entity_id" -> entityId.value.toString.asInstanceOf[AnyRef],
- "field" -> field.asInstanceOf[AnyRef],
- "value" -> jsonToJava(value).asInstanceOf[AnyRef],
- "confidence" -> confidence.map(Double.box).orNull.asInstanceOf[AnyRef],
- "status" -> "active".asInstanceOf[AnyRef],
- "created_rev" -> Long.box(newRev).asInstanceOf[AnyRef],
- "provenance" -> parseToJavaMap(provenanceJson.noSpaces).asInstanceOf[AnyRef]
- ),
- txId = Some(txId)
- )
+ for {
+ _ <- client.execute(
+ """INSERT {
+ | _key: @key,
+ | entity_id: @entity_id,
+ | field: @field,
+ | value: @value,
+ | confidence: @confidence,
+ | status: @status,
+ | created_rev: @created_rev,
+ | deleted_rev: null,
+ | provenance: @provenance,
+ | inference_version: @inference_version
+ |} INTO claims""".stripMargin,
+ Map(
+ "key" -> claimKey.asInstanceOf[AnyRef],
+ "entity_id" -> eid.asInstanceOf[AnyRef],
+ "field" -> field.asInstanceOf[AnyRef],
+ "value" -> jsonToJava(value).asInstanceOf[AnyRef],
+ "confidence" -> confidence.map(Double.box).orNull.asInstanceOf[AnyRef],
+ "status" -> "active".asInstanceOf[AnyRef],
+ "created_rev" -> Long.box(newRev).asInstanceOf[AnyRef],
+ "provenance" -> parseToJavaMap(provenanceJson.noSpaces).asInstanceOf[AnyRef],
+ "inference_version" -> inferenceVersion.orNull.asInstanceOf[AnyRef]
+ ),
+ txId = Some(txId)
+ )
+ _ = claimCache.add((eid, field, valueStr))
+ } yield ()
}
} yield ()
@@ -327,12 +361,10 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
private def storePatch(patch: GraphPatch, newRev: Long, txId: String): IO[Unit] = {
val patchJson = patch.asJson.noSpaces
client.execute(
- """INSERT {
- | _key: @key,
- | patch_id: @patch_id,
- | rev: @rev,
- | data: @data
- |} INTO patches""".stripMargin,
+ """UPSERT { _key: @key }
+ | INSERT { _key: @key, patch_id: @patch_id, rev: @rev, data: @data }
+ | UPDATE {}
+ | IN patches""".stripMargin,
Map(
"key" -> patch.patchId.value.toString.asInstanceOf[AnyRef],
"patch_id" -> patch.patchId.value.toString.asInstanceOf[AnyRef],
@@ -357,14 +389,32 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
txId = Some(txId)
)
- private def updateRevision(newRev: Long, txId: String): IO[Unit] =
+ private def shouldAdvanceSourceGraphRev(patch: GraphPatch): Boolean =
+ patch.ops.exists {
+ case PatchOp.UpsertNode(_, kind, _, _) =>
+ !SourceGraphIgnoredKinds.contains(kind)
+ case PatchOp.UpsertEdge(_, _, _, predicate, _) =>
+ SourceGraphPredicates.contains(predicate.value)
+ case PatchOp.DeleteNode(_) | PatchOp.DeleteEdge(_) =>
+ patch.source.sourceType == SourceType.Code || patch.source.sourceType == SourceType.Test
+ case _ =>
+ false
+ }
+
+ private def updateRevision(newRev: Long, txId: String, advanceSourceGraphRev: Boolean): IO[Unit] =
client.execute(
- """UPSERT { _key: @key }
- | INSERT { _key: @key, rev: @rev }
+ """FOR key IN @keys
+ | UPSERT { _key: key }
+ | INSERT { _key: key, rev: @rev }
| UPDATE { rev: @rev }
| IN revisions""".stripMargin,
Map(
- "key" -> "current".asInstanceOf[AnyRef],
+ "keys" -> {
+ val keys =
+ if (advanceSourceGraphRev) Array(CurrentRevisionKey, SourceGraphRevisionKey)
+ else Array(CurrentRevisionKey)
+ keys.asInstanceOf[AnyRef]
+ },
"rev" -> Long.box(newRev).asInstanceOf[AnyRef]
),
txId = Some(txId)
@@ -388,32 +438,44 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
txId = Some(txId)
)
- /** Check if an identical active claim already exists (same entity, field, value). */
- private def hasIdenticalActiveClaim(entityId: String, field: String, value: Json, txId: String): IO[Boolean] =
- client.queryOne(
- """FOR c IN claims
- | FILTER c.entity_id == @entity_id
- | AND c.field == @field
- | AND c.value == @value
- | AND c.deleted_rev == null
- | LIMIT 1
- | RETURN 1""".stripMargin,
- Map(
- "entity_id" -> entityId.asInstanceOf[AnyRef],
- "field" -> field.asInstanceOf[AnyRef],
- "value" -> jsonToJava(value).asInstanceOf[AnyRef]
- ),
- txId = Some(txId)
- ).map(_.isDefined)
+ /**
+ * Pre-load all active claims for the given entity IDs in one DB query.
+ * Returns a mutable set of (entityId, field, valueNoSpaces) triples used
+ * as an in-memory duplicate-check cache throughout a single patch commit.
+ */
+ private def loadActiveClaimsCache(entityIds: Set[String], txId: String): IO[mutable.Set[(String, String, String)]] =
+ if (entityIds.isEmpty) IO.pure(mutable.Set.empty)
+ else {
+ val eids = new java.util.ArrayList[String]()
+ entityIds.foreach(eids.add)
+ client.query(
+ """FOR c IN claims
+ | FILTER c.entity_id IN @entityIds AND c.deleted_rev == null
+ | RETURN { entity_id: c.entity_id, field: c.field, value: c.value }""".stripMargin,
+ Map("entityIds" -> eids.asInstanceOf[AnyRef]),
+ txId = Some(txId)
+ ).map { results =>
+ val cache = mutable.Set.empty[(String, String, String)]
+ results.foreach { json =>
+ val parsed = for {
+ eid <- json.hcursor.get[String]("entity_id").toOption
+ field <- json.hcursor.get[String]("field").toOption
+ value <- json.hcursor.downField("value").focus
+ } yield (eid, field, value.noSpaces)
+ parsed.foreach(cache.add)
+ }
+ cache
+ }
+ }
/**
* After all ops execute, retire claims from the same extractor/entity that are
- * no longer emitted (the "absent claims" policy). This handles the case where
- * a re-parsed file no longer contains a previously asserted claim.
+ * no longer emitted (the "absent claims" policy). Uses a single batch AQL query
+ * across all entities instead of one query per entity.
*/
- private def retireAbsentClaims(patch: GraphPatch, newRev: Long, txId: String): IO[Unit] = {
+ private def retireAbsentClaimsBatch(patch: GraphPatch, newRev: Long, txId: String): IO[Unit] = {
val assertedClaims = patch.ops.collect {
- case PatchOp.AssertClaim(entityId, field, _, _) => (entityId.value.toString, field)
+ case PatchOp.AssertClaim(entityId, field, _, _, _) => (entityId.value.toString, field)
}
if (assertedClaims.isEmpty) IO.unit
@@ -422,27 +484,34 @@ class ArangoGraphWriteApi(client: ArangoClient) extends GraphWriteApi {
.groupBy(_._1)
.map { case (eid, pairs) => eid -> pairs.map(_._2).toSet }
- val extractor = patch.source.extractor
-
- fieldsByEntity.toVector.traverse_ { case (entityId, fields) =>
- val fieldList = new java.util.ArrayList[String]()
- fields.foreach(fieldList.add)
- client.execute(
- """FOR c IN claims
- | FILTER c.entity_id == @entity_id
- | AND c.field NOT IN @fields
- | AND c.deleted_rev == null
- | AND c.provenance.extractor == @extractor
- | UPDATE c WITH { status: "retracted", deleted_rev: @deleted_rev } IN claims""".stripMargin,
- Map(
- "entity_id" -> entityId.asInstanceOf[AnyRef],
- "fields" -> fieldList.asInstanceOf[AnyRef],
- "extractor" -> extractor.asInstanceOf[AnyRef],
- "deleted_rev" -> Long.box(newRev).asInstanceOf[AnyRef]
- ),
- txId = Some(txId)
- )
+ // Build Java list of {entity_id, fields} for AQL — one query instead of N
+ val entityFieldMap = new java.util.ArrayList[java.util.Map[String, AnyRef]]()
+ fieldsByEntity.foreach { case (eid, fields) =>
+ val m = new java.util.HashMap[String, AnyRef]()
+ val fl = new java.util.ArrayList[String]()
+ fields.foreach(fl.add)
+ m.put("entity_id", eid)
+ m.put("fields", fl)
+ entityFieldMap.add(m)
}
+
+ client.execute(
+ """LET fieldMap = MERGE(
+ | FOR ef IN @entityFieldMap RETURN { [ef.entity_id]: ef.fields }
+ |)
+ |FOR c IN claims
+ | FILTER c.entity_id IN ATTRIBUTES(fieldMap)
+ | AND c.field NOT IN fieldMap[c.entity_id]
+ | AND c.deleted_rev == null
+ | AND c.provenance.extractor == @extractor
+ | UPDATE c WITH { status: "retracted", deleted_rev: @deleted_rev } IN claims""".stripMargin,
+ Map(
+ "entityFieldMap" -> entityFieldMap.asInstanceOf[AnyRef],
+ "extractor" -> patch.source.extractor.asInstanceOf[AnyRef],
+ "deleted_rev" -> Long.box(newRev).asInstanceOf[AnyRef]
+ ),
+ txId = Some(txId)
+ )
}
}
diff --git a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
index 982308c6..d0c51ae0 100644
--- a/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
+++ b/memory-layer/src/main/scala/ix/memory/db/BulkWriteApi.scala
@@ -1,5 +1,6 @@
package ix.memory.db
+import java.nio.charset.StandardCharsets
import java.time.Instant
import java.util.UUID
@@ -17,6 +18,8 @@ import ix.memory.model._
* instead of individual AQL queries.
*/
class BulkWriteApi(client: ArangoClient) {
+ private val CurrentRevisionKey = "current"
+ private val SourceGraphRevisionKey = "source-graph"
private def withRetry[A](action: IO[A], maxAttempts: Int = 3, delay: FiniteDuration = 500.millis): IO[A] = {
action.handleErrorWith { err =>
@@ -115,12 +118,13 @@ class BulkWriteApi(client: ArangoClient) {
private def updateRevision(newRev: Long): IO[Unit] =
client.execute(
- """UPSERT { _key: @key }
- | INSERT { _key: @key, rev: @rev }
+ """FOR key IN @keys
+ | UPSERT { _key: key }
+ | INSERT { _key: key, rev: @rev }
| UPDATE { rev: @rev }
| IN revisions""".stripMargin,
Map(
- "key" -> "current".asInstanceOf[AnyRef],
+ "keys" -> Array(CurrentRevisionKey, SourceGraphRevisionKey).asInstanceOf[AnyRef],
"rev" -> Long.box(newRev).asInstanceOf[AnyRef]
)
)
@@ -147,6 +151,9 @@ case class FileBatch(
private def nodeKindToString(nk: NodeKind): String =
nk.asJson.asString.getOrElse(nk.toString)
+ lazy val estimatedPayloadBytes: Int =
+ (patch.asJson.noSpaces.getBytes(StandardCharsets.UTF_8).length * 4) + 1024
+
def nodeDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] = {
val now = Instant.now().toString
patch.ops.collect { case PatchOp.UpsertNode(id, kind, name, attrs) =>
@@ -191,7 +198,7 @@ case class FileBatch(
}
def claimDocuments(rev: Long): Vector[java.util.Map[String, AnyRef]] =
- patch.ops.collect { case PatchOp.AssertClaim(entityId, field, value, confidence) =>
+ patch.ops.collect { case PatchOp.AssertClaim(entityId, field, value, confidence, inferenceVersion) =>
val doc = new java.util.HashMap[String, AnyRef]()
doc.put("_key", UUID.randomUUID().toString)
doc.put("entity_id", entityId.value.toString)
@@ -202,6 +209,7 @@ case class FileBatch(
doc.put("created_rev", Long.box(rev))
doc.put("deleted_rev", null)
doc.put("provenance", provenance)
+ doc.put("inference_version", inferenceVersion.orNull)
doc: java.util.Map[String, AnyRef]
}
diff --git a/memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala b/memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala
index 3d685cf8..d94e39fe 100644
--- a/memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala
+++ b/memory-layer/src/main/scala/ix/memory/ingestion/BulkIngestionService.scala
@@ -10,7 +10,7 @@ import io.circe.syntax._
import org.typelevel.log4cats.slf4j.Slf4jLogger
import org.typelevel.log4cats.Logger
-import ix.memory.db.{BulkWriteApi, CommitResult, CommitStatus, FileBatch, GraphQueryApi}
+import ix.memory.db.{BulkWriteApi, FileBatch, GraphQueryApi}
import ix.memory.model._
private[ingestion] sealed trait ParseOutcome
@@ -32,6 +32,9 @@ class BulkIngestionService(
private val parallelism = Runtime.getRuntime.availableProcessors.max(2)
private val MaxFileBytes: Long = 1024 * 1024 // 1 MB
+ private val ParseBatchSize = 100
+ private val MaxCommitFiles = 100
+ private val MaxCommitPayloadBytes = 1024 * 1024
private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
private val logger: Logger[IO] = Slf4jLogger.getLoggerFromName[IO]("ix.ingestion")
@@ -46,55 +49,69 @@ class BulkIngestionService(
force: Boolean = false
): IO[IngestionResult] = {
for {
- files <- discoverFiles(path, language, recursive)
- _ <- onProgress(IngestionProgress(files.size, 0, 0, 0))
- hashMap <- if (force) IO.pure(Map.empty[String, String]) else loadExistingHashes(files)
- outcomes <- files.parTraverseN(parallelism) { f =>
- parseFile(f, hashMap).handleErrorWith { err =>
- logger.warn(s"Parse error: ${f.toString} — ${err.getMessage}") *>
- IO.pure(Skipped("parseError"): ParseOutcome)
- }
+ files <- discoverFiles(path, language, recursive)
+ _ <- onProgress(IngestionProgress(files.size, 0, 0, 0))
+ latestRev <- queryApi.getLatestRev
+ state <- files.grouped(ParseBatchSize).toVector.foldLeft(IO.pure(BulkIngestionState.empty(latestRev.value))) {
+ case (stateIO, batch) =>
+ stateIO.flatMap(processBatch(batch, force, files.size, onProgress, _))
}
- validBatches = outcomes.collect { case Parsed(b) => b }
- unchangedCount = outcomes.count { case Skipped("unchanged") => true; case _ => false }
- emptyCount = outcomes.count { case Skipped("emptyFile") => true; case _ => false }
- errorCount = outcomes.count { case Skipped("parseError") => true; case _ => false }
- tooLargeCount = outcomes.count { case Skipped("tooLarge") => true; case _ => false }
- skippedCount = unchangedCount + emptyCount + errorCount + tooLargeCount
- totalChunks = if (validBatches.isEmpty) 0 else (validBatches.size + 99) / 100
- _ <- onProgress(IngestionProgress(files.size, outcomes.size, 0, totalChunks))
- latestRev <- queryApi.getLatestRev
- result <- if (validBatches.isEmpty) IO.pure(CommitResult(latestRev, CommitStatus.Ok))
- else commitChunkedWithProgress(validBatches.toVector, latestRev.value, onProgress, files.size, outcomes.size)
} yield IngestionResult(
filesProcessed = files.size,
- patchesApplied = validBatches.size,
- filesSkipped = skippedCount,
- entitiesCreated = validBatches.flatMap(_.patch.ops.collect { case _: PatchOp.UpsertNode => 1 }).size,
- latestRev = result.newRev,
- skipReasons = SkipReasons(unchanged = unchangedCount, emptyFile = emptyCount, parseError = errorCount, tooLarge = tooLargeCount)
+ patchesApplied = state.patchesApplied,
+ filesSkipped = state.skipped.total,
+ entitiesCreated = state.entitiesCreated,
+ latestRev = Rev(state.currentRev),
+ skipReasons = state.skipped
)
}
- private def commitChunkedWithProgress(
- batches: Vector[FileBatch],
- baseRev: Long,
- onProgress: IngestionProgress => IO[Unit],
+ private def processBatch(
+ files: List[Path],
+ force: Boolean,
discovered: Int,
- parsed: Int
- ): IO[CommitResult] = {
- val chunkSize = 100
- val chunks = batches.grouped(chunkSize).toVector
- val totalChunks = chunks.size
-
- chunks.zipWithIndex.foldLeft(IO.pure(baseRev)) { case (revIO, (chunk, idx)) =>
- revIO.flatMap { currentRev =>
- bulkWriteApi.commitBatch(chunk, currentRev).flatMap { result =>
- onProgress(IngestionProgress(discovered, parsed, idx + 1, totalChunks)) *>
- IO.pure(result.newRev.value)
+ onProgress: IngestionProgress => IO[Unit],
+ state: BulkIngestionState
+ ): IO[BulkIngestionState] = {
+ for {
+ hashMap <- if (force) IO.pure(Map.empty[String, String]) else loadExistingHashes(files)
+ outcomes <- files.parTraverseN(parallelism) { f =>
+ parseFile(f, hashMap).handleErrorWith { err =>
+ logger.warn(s"Parse error: ${f.toString} - ${err.getMessage}") *>
+ IO.pure(Skipped("parseError"): ParseOutcome)
}
}
- }.map(finalRev => CommitResult(Rev(finalRev), CommitStatus.Ok))
+ validBatches = outcomes.collect { case Parsed(batch) => batch }.toVector
+ batchSkipped = SkipReasons(
+ unchanged = outcomes.count { case Skipped("unchanged") => true; case _ => false },
+ emptyFile = outcomes.count { case Skipped("emptyFile") => true; case _ => false },
+ parseError = outcomes.count { case Skipped("parseError") => true; case _ => false },
+ tooLarge = outcomes.count { case Skipped("tooLarge") => true; case _ => false }
+ )
+ parsedFiles = state.filesParsed + outcomes.size
+ commitChunks = chunkForCommit(validBatches)
+ totalChunks = state.totalChunksPlanned + commitChunks.size
+ _ <- onProgress(IngestionProgress(discovered, parsedFiles, state.chunksWritten, totalChunks))
+ committed <- commitChunks.foldLeft(IO.pure((state.currentRev, state.chunksWritten))) {
+ case (revIO, chunk) =>
+ revIO.flatMap { case (currentRev, chunksWritten) =>
+ bulkWriteApi.commitBatch(chunk, currentRev).flatMap { result =>
+ val updatedChunks = chunksWritten + 1
+ onProgress(IngestionProgress(discovered, parsedFiles, updatedChunks, totalChunks)) *>
+ IO.pure((result.newRev.value, updatedChunks))
+ }
+ }
+ }
+ entitiesCreated = validBatches.flatMap(_.patch.ops.collect { case _: PatchOp.UpsertNode => 1 }).size
+ } yield state.copy(
+ currentRev = committed._1,
+ filesParsed = parsedFiles,
+ patchesApplied = state.patchesApplied + validBatches.size,
+ chunksWritten = committed._2,
+ totalChunksPlanned = totalChunks,
+ entitiesCreated = state.entitiesCreated + entitiesCreated,
+ skipped = combineSkipReasons(state.skipped, batchSkipped)
+ )
}
/**
@@ -121,7 +138,7 @@ class BulkIngestionService(
parserOpt = parserRouter.parserFor(filePath.toString)
parseResult = parserOpt match {
case Some(p) => p.parse(filePath.getFileName.toString, source)
- case None => genericTextParse(filePath.getFileName.toString, source)
+ case None => GenericTextFallback.parse(filePath.getFileName.toString, source)
}
patch = GraphPatchBuilder.build(filePath.toString, Some(hash), parseResult)
_ <- IO.fromEither(PatchValidator.validate(patch).left.map(msg => new IllegalStateException(msg)))
@@ -133,7 +150,7 @@ class BulkIngestionService(
} yield result
}
- private def loadExistingHashes(files: List[Path]): IO[Map[String, String]] = {
+ private def loadExistingHashes(files: Seq[Path]): IO[Map[String, String]] = {
val paths = files.map(_.toString)
queryApi.getSourceHashes(paths)
}
@@ -156,7 +173,7 @@ class BulkIngestionService(
): IO[List[Path]] = IO.blocking {
val extensions = FileDiscovery.extensionsFor(language)
if (Files.isRegularFile(path)) {
- if (FileDiscovery.matchesFilter(path, extensions)) List(path) else List.empty
+ if (language.isEmpty || FileDiscovery.matchesFilter(path, extensions)) List(path) else List.empty
} else if (Files.isDirectory(path)) {
// Try git ls-files first (respects .gitignore), fall back to manual walk
FileDiscovery.tryGitLsFiles(path, recursive, extensions)
@@ -164,20 +181,65 @@ class BulkIngestionService(
} else List.empty
}
+ private def chunkForCommit(batches: Vector[FileBatch]): Vector[Vector[FileBatch]] = {
+ val chunks = Vector.newBuilder[Vector[FileBatch]]
+ var current = Vector.empty[FileBatch]
+ var currentBytes = 0
+
+ batches.foreach { batch =>
+ val batchBytes = batch.estimatedPayloadBytes
+ val exceedsPayload = current.nonEmpty && currentBytes + batchBytes > MaxCommitPayloadBytes
+ val exceedsCount = current.size >= MaxCommitFiles
+ if (exceedsPayload || exceedsCount) {
+ chunks += current
+ current = Vector.empty
+ currentBytes = 0
+ }
+ current = current :+ batch
+ currentBytes += batchBytes
+ }
+
+ if (current.nonEmpty) {
+ chunks += current
+ }
+
+ chunks.result()
+ }
+
+ private def combineSkipReasons(left: SkipReasons, right: SkipReasons): SkipReasons =
+ SkipReasons(
+ unchanged = left.unchanged + right.unchanged,
+ emptyFile = left.emptyFile + right.emptyFile,
+ parseError = left.parseError + right.parseError,
+ tooLarge = left.tooLarge + right.tooLarge
+ )
+
private def sha256Bytes(bytes: Array[Byte]): String = {
val md = java.security.MessageDigest.getInstance("SHA-256")
md.digest(bytes).map("%02x".format(_)).mkString
}
- private def genericTextParse(fileName: String, source: String): ParseResult = {
- val lines = if (source.isEmpty) 1 else source.count(_ == '\n') + 1
- ParseResult(
- entities = Vector(ParsedEntity(
- name = fileName, kind = NodeKind.File,
- attrs = Map("content" -> Json.fromString(source)),
- lineStart = 1, lineEnd = lines
- )),
- relationships = Vector.empty
+}
+
+private[ingestion] final case class BulkIngestionState(
+ currentRev: Long,
+ filesParsed: Int,
+ patchesApplied: Int,
+ chunksWritten: Int,
+ totalChunksPlanned: Int,
+ entitiesCreated: Int,
+ skipped: SkipReasons
+)
+
+private[ingestion] object BulkIngestionState {
+ def empty(baseRev: Long): BulkIngestionState =
+ BulkIngestionState(
+ currentRev = baseRev,
+ filesParsed = 0,
+ patchesApplied = 0,
+ chunksWritten = 0,
+ totalChunksPlanned = 0,
+ entitiesCreated = 0,
+ skipped = SkipReasons()
)
- }
}
diff --git a/memory-layer/src/main/scala/ix/memory/ingestion/FileDiscovery.scala b/memory-layer/src/main/scala/ix/memory/ingestion/FileDiscovery.scala
index 2a20186a..26e1d212 100644
--- a/memory-layer/src/main/scala/ix/memory/ingestion/FileDiscovery.scala
+++ b/memory-layer/src/main/scala/ix/memory/ingestion/FileDiscovery.scala
@@ -11,32 +11,10 @@ import java.nio.file.{Files, Path}
object FileDiscovery {
private val defaultExtensions = Set(
- // code
- ".py", ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs",
- ".java", ".scala", ".sc", ".go", ".rs", ".c", ".h", ".cc", ".cpp", ".hpp",
- ".kt", ".kts", ".swift", ".rb", ".php", ".cs", ".ex", ".exs",
- ".lua", ".zig", ".nim", ".jl", ".r", ".R", ".pl", ".pm",
- ".sh", ".bash", ".zsh", ".fish",
-
- // config / data
- ".json", ".yaml", ".yml", ".toml", ".ini", ".conf", ".env", ".properties",
- ".xml", ".plist", ".gradle",
-
- // docs
- ".md", ".mdx", ".rst", ".txt",
-
- // build
- ".sbt", ".gradle", ".cmake"
- )
-
- /** Files matched by name regardless of extension. */
- private val specialFiles = Set(
- "Dockerfile", "Makefile", "Rakefile", "Gemfile", "Procfile",
- "README", "LICENSE", "NOTICE", "CHANGELOG",
- "Cargo.toml", "go.mod", "go.sum",
- "package.json", "tsconfig.json", "pyproject.toml", "setup.py", "setup.cfg",
- "build.sbt", "build.gradle", "pom.xml", "CMakeLists.txt",
- ".gitignore", ".dockerignore", ".editorconfig"
+ ".py", ".ts", ".tsx",
+ ".scala", ".sc",
+ ".json", ".yaml", ".yml", ".toml",
+ ".md"
)
def extensionsFor(language: Option[String]): Set[String] = language match {
@@ -55,7 +33,7 @@ object FileDiscovery {
def matchesFilter(file: Path, extensions: Set[String]): Boolean = {
val name = file.getFileName.toString
- specialFiles.contains(name) || extensions.exists(name.endsWith)
+ extensions.exists(name.endsWith)
}
/**
diff --git a/memory-layer/src/main/scala/ix/memory/ingestion/GenericTextFallback.scala b/memory-layer/src/main/scala/ix/memory/ingestion/GenericTextFallback.scala
new file mode 100644
index 00000000..29d670ea
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/ingestion/GenericTextFallback.scala
@@ -0,0 +1,32 @@
+package ix.memory.ingestion
+
+import io.circe.Json
+import ix.memory.model.NodeKind
+
+object GenericTextFallback {
+ private val PreviewLimit = 240
+
+ def parse(fileName: String, source: String): ParseResult = {
+ val lines = if (source.isEmpty) 1 else source.count(_ == '\n') + 1
+ val preview = source.linesIterator.find(_.trim.nonEmpty).map(_.trim.take(PreviewLimit))
+
+ val attrs = Map(
+ "file_name" -> Json.fromString(fileName),
+ "line_count" -> Json.fromInt(lines)
+ ) ++ preview.filter(_.nonEmpty).map(value => "preview" -> Json.fromString(value))
+
+ ParseResult(
+ entities = Vector(
+ ParsedEntity(
+ name = fileName,
+ kind = NodeKind.File,
+ attrs = attrs,
+ lineStart = 1,
+ lineEnd = lines,
+ contentFingerprint = Some(Fingerprint.compute(fileName, source))
+ )
+ ),
+ relationships = Vector.empty
+ )
+ }
+}
diff --git a/memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala b/memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala
index 61c4d9c4..363fbc0d 100644
--- a/memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala
+++ b/memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala
@@ -8,8 +8,6 @@ import cats.effect.IO
import cats.syntax.all._
import ix.memory.db.{CommitResult, GraphQueryApi, GraphWriteApi}
import ix.memory.model._
-import io.circe.Json
-
/**
* Orchestrates file ingestion: reads source files, parses them,
* builds graph patches, and commits them through the write API.
@@ -35,7 +33,7 @@ class IngestionService(parserRouter: ParserRouter, writeApi: GraphWriteApi, quer
parserOpt = parserRouter.parserFor(filePath.toString)
result = parserOpt match {
case Some(p) => p.parse(filePath.getFileName.toString, source)
- case None => genericTextParse(filePath.getFileName.toString, source)
+ case None => GenericTextFallback.parse(filePath.getFileName.toString, source)
}
val fp = filePath.toString
extractor = if (fp.endsWith(".py")) "tree-sitter-python/1.0"
@@ -103,7 +101,7 @@ class IngestionService(parserRouter: ParserRouter, writeApi: GraphWriteApi, quer
): IO[List[Path]] = IO.blocking {
val extensions = FileDiscovery.extensionsFor(language)
if (Files.isRegularFile(path)) {
- if (FileDiscovery.matchesFilter(path, extensions)) List(path) else List.empty
+ if (language.isEmpty || FileDiscovery.matchesFilter(path, extensions)) List(path) else List.empty
} else if (Files.isDirectory(path)) {
FileDiscovery.tryGitLsFiles(path, recursive, extensions)
.getOrElse(FileDiscovery.walkFiles(path, recursive, extensions))
@@ -230,28 +228,6 @@ class IngestionService(parserRouter: ParserRouter, writeApi: GraphWriteApi, quer
val md = java.security.MessageDigest.getInstance("SHA-256")
md.digest(bytes).map("%02x".format(_)).mkString
}
-
- /**
- * Generic fallback parse: ensures every text file produces at least a File entity with content.
- * This avoids needing a bespoke parser for every dev/text extension.
- */
- private def genericTextParse(fileName: String, source: String): ParseResult = {
- val lines = if (source.isEmpty) 1 else source.count(_ == '\n') + 1
- ParseResult(
- entities = Vector(
- ParsedEntity(
- name = fileName,
- kind = NodeKind.File,
- attrs = Map(
- "content" -> Json.fromString(source)
- ),
- lineStart = 1,
- lineEnd = lines
- )
- ),
- relationships = Vector.empty
- )
- }
}
case class IngestionProgress(
@@ -266,7 +242,9 @@ case class SkipReasons(
emptyFile: Int = 0,
parseError: Int = 0,
tooLarge: Int = 0
-)
+) {
+ def total: Int = unchanged + emptyFile + parseError + tooLarge
+}
case class IngestionResult(
filesProcessed: Int,
diff --git a/memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala b/memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala
index b357b6ec..60e7693d 100644
--- a/memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala
@@ -48,15 +48,42 @@ object LouvainClustering {
def cluster(
graph: WeightedFileGraph,
maxLevels: Int = 4,
- resolution: Double = 1.0,
+ resolution: Double = -1.0,
minCommunitySize: Int = 2,
seed: Long = 42L
): Vector[LevelPartition] = {
if (graph.vertices.isEmpty || graph.totalWeight == 0.0) return Vector.empty
+ val effectiveResolution =
+ if (resolution > 0.0) resolution else adaptiveResolution(graph.vertices.size)
+
+ val numSeeds = if (graph.vertices.size < 30) 1L else 3L
+ val runs = (0L until numSeeds).map { offset =>
+ val levels = clusterOnce(
+ graph = graph,
+ maxLevels = maxLevels,
+ resolution = effectiveResolution,
+ minCommunitySize = minCommunitySize,
+ seed = seed + offset
+ )
+ val score = levels.headOption
+ .map(level => modularity(graph, level.assignment, effectiveResolution))
+ .getOrElse(Double.NegativeInfinity)
+ levels -> score
+ }
+
+ runs.maxBy(_._2)._1
+ }
+
+ private def clusterOnce(
+ graph: WeightedFileGraph,
+ maxLevels: Int,
+ resolution: Double,
+ minCommunitySize: Int,
+ seed: Long
+ ): Vector[LevelPartition] = {
val rng = new Random(seed)
- // nodeExpansion: for the current working graph, maps each node → original file IDs it represents
var nodeExpansion: Map[NodeId, Set[NodeId]] =
graph.vertices.map(v => v.id -> Set(v.id)).toMap
@@ -66,19 +93,14 @@ object LouvainClustering {
var level = 0
while (level < maxLevels && currentGraph.vertices.size > 1) {
val rawAssignment = louvainPass(currentGraph, resolution, rng)
-
- // Project raw assignment (current node ids) → original file id communities
- val projected = projectPartition(rawAssignment, nodeExpansion)
-
- val meaningful = projected.communities.count(_.size >= minCommunitySize)
+ val projected = projectPartition(rawAssignment, nodeExpansion)
+ val meaningful = projected.communities.count(_.size >= minCommunitySize)
if (meaningful <= 1 && levels.nonEmpty) {
- // No further useful partitioning — stop
level = maxLevels
} else {
levels += projected
- // Coarsen: collapse each community into a supernode
val (coarsenedGraph, newExpansion) =
coarsen(currentGraph, rawAssignment, nodeExpansion)
@@ -93,6 +115,11 @@ object LouvainClustering {
levels.toVector
}
+ private def adaptiveResolution(fileCount: Int): Double =
+ if (fileCount < 50) 1.2
+ else if (fileCount <= 500) 1.0
+ else 0.8
+
// ── Single Louvain pass ────────────────────────────────────────────
/**
@@ -104,7 +131,7 @@ object LouvainClustering {
resolution: Double,
rng: Random
): Map[NodeId, Int] = {
- val nodes = graph.vertices.map(_.id)
+ val nodes = graph.vertices.map(_.id).toArray
val adj = graph.adjMatrix
val m = graph.totalWeight.max(1e-12)
@@ -113,32 +140,34 @@ object LouvainClustering {
val sumTot = scala.collection.mutable.Map(
nodes.zipWithIndex.map { case (n, i) => i -> graph.degrees.getOrElse(n, 0.0) }: _*
).withDefaultValue(0.0)
- val sumIn = scala.collection.mutable.Map[Int, Double]().withDefaultValue(0.0)
// communities: commId → members (mutable set)
val members = scala.collection.mutable.Map(
nodes.zipWithIndex.map { case (n, i) => i -> scala.collection.mutable.Set(n) }: _*
)
- var improved = true
- var passes = 0
+ var continue = true
+ val wToComm = scala.collection.mutable.HashMap.empty[Int, Double]
- while (improved && passes < 30) {
- improved = false
- passes += 1
+ while (continue) {
+ shuffleInPlace(nodes, rng)
+ var improved = false
+ var passGain = 0.0
- for (node <- rng.shuffle(nodes)) {
+ var idx = 0
+ while (idx < nodes.length) {
+ val node = nodes(idx)
val ci = commOf(node)
val ki = graph.degrees.getOrElse(node, 0.0)
// Compute weight from this node to each neighboring community
- val wToComm = scala.collection.mutable.Map[Int, Double]().withDefaultValue(0.0)
+ wToComm.clear()
for ((neighbor, w) <- adj.getOrElse(node, Map.empty)) {
- wToComm(commOf(neighbor)) += w
+ val community = commOf(neighbor)
+ wToComm.update(community, wToComm.getOrElse(community, 0.0) + w)
}
// Remove node from current community
val kiInCi = wToComm.getOrElse(ci, 0.0)
- sumIn(ci) -= 2.0 * kiInCi
sumTot(ci) -= ki
members(ci) -= node
// Temporarily isolate: treat as its own community (gains = 0 baseline)
@@ -159,11 +188,16 @@ object LouvainClustering {
commOf(node) = bestComm
members.getOrElseUpdate(bestComm, scala.collection.mutable.Set()) += node
val kiInBest = wToComm.getOrElse(bestComm, 0.0)
- sumIn(bestComm) += 2.0 * kiInBest
sumTot(bestComm) += ki
- if (bestComm != ci) improved = true
+ if (bestComm != ci) {
+ improved = true
+ passGain += bestGain
+ }
+ idx += 1
}
+
+ continue = improved && passGain >= 1e-6
}
// Re-index communities from 0 to N-1 (dense)
@@ -176,6 +210,38 @@ object LouvainClustering {
finalAssign
}
+ private def shuffleInPlace(nodes: Array[NodeId], rng: Random): Unit = {
+ var i = nodes.length - 1
+ while (i > 0) {
+ val j = rng.nextInt(i + 1)
+ val tmp = nodes(i)
+ nodes(i) = nodes(j)
+ nodes(j) = tmp
+ i -= 1
+ }
+ }
+
+ private def modularity(
+ graph: WeightedFileGraph,
+ assignment: Map[NodeId, Int],
+ resolution: Double
+ ): Double = {
+ val communities = assignment.groupBy(_._2).map { case (idx, entries) => idx -> entries.keySet }
+ val m = graph.totalWeight.max(1e-12)
+
+ communities.values.foldLeft(0.0) { (acc, members) =>
+ val totalDegree = members.iterator.map(node => graph.degrees.getOrElse(node, 0.0)).sum
+ var internalWeight = 0.0
+ for {
+ src <- members
+ (dst, weight) <- graph.adjMatrix.getOrElse(src, Map.empty)
+ if members.contains(dst) && src.value.compareTo(dst.value) < 0
+ } internalWeight += weight
+
+ acc + (internalWeight / m) - resolution * math.pow(totalDegree / (2.0 * m), 2.0)
+ }
+ }
+
// ── Projection ─────────────────────────────────────────────────────
private def projectPartition(
@@ -262,7 +328,8 @@ object LouvainClustering {
superVertices,
newAdjFinal,
newDegrees.toMap,
- newTotalWeight
+ newTotalWeight,
+ Map.empty
)
(coarsened, newExpansion)
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
index 691c4ad3..1784f991 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilder.scala
@@ -3,7 +3,7 @@ package ix.memory.map
import cats.effect.IO
import ix.memory.db.ArangoClient
-import ix.memory.model.NodeId
+import ix.memory.model.{NodeId, Rev}
/**
* Builds a weighted undirected file-level coupling graph from ArangoDB.
@@ -16,16 +16,25 @@ import ix.memory.model.NodeId
*
* Signal weights (from spec):
* α = 1.0 call coupling (highest importance)
- * β = 0.7 import coupling
- * γ = 0.9 extends/implements (type/interface cohesion proxy)
- * ε = 0.3 path proximity (weak structural prior)
+ * β = 0.5 import coupling
+ * γ = 1.2 extends/implements (type/interface cohesion proxy)
+ * ε = 0.2 path proximity (weak structural prior)
*/
class MapGraphBuilder(client: ArangoClient) {
private val Alpha = 1.0
- private val Beta = 0.7
- private val Gamma = 0.9
- private val Epsilon = 0.3
+ private val Beta = 0.5
+ private val Gamma = 1.2
+ private val Epsilon = 0.2
+ private val MaxEdges = 200000
+ private val PathProximityMinFiles = 2
+ private val PathProximityMaxFiles = 50
+ private val SourceGraphRevisionKey = "source-graph"
+ private val IgnoredNodeKinds = Vector(
+ "module", "config", "config_entry", "doc", "decision", "intent",
+ "bug", "plan", "task", "goal", "region"
+ )
+ private val CouplingPredicates = Vector("CALLS", "IMPORTS", "EXTENDS", "IMPLEMENTS")
def discoverFiles(): IO[Vector[FileVertex]] = fetchFiles()
@@ -41,6 +50,70 @@ class MapGraphBuilder(client: ArangoClient) {
graph <- buildGraph(files)
} yield graph
+ /**
+ * Revision of the graph inputs that materially affect architecture mapping.
+ * This intentionally ignores derived artifacts like persisted region nodes or
+ * scoring claims so cached maps stay reusable across unchanged source graphs.
+ */
+ def currentInputRev(): IO[Rev] =
+ loadSourceGraphRevision().flatMap {
+ case Some(rev) => IO.pure(rev)
+ case None => scanCurrentInputRev()
+ }
+
+ private def loadSourceGraphRevision(): IO[Option[Rev]] =
+ client.queryOne(
+ """FOR r IN revisions
+ | FILTER r._key == @key
+ | LIMIT 1
+ | RETURN r.rev""".stripMargin,
+ Map("key" -> SourceGraphRevisionKey.asInstanceOf[AnyRef])
+ ).map(_.flatMap(_.as[Long].toOption).map(Rev(_)))
+
+ private def scanCurrentInputRev(): IO[Rev] =
+ client.queryOne(
+ """LET node_rev = MAX(
+ | FOR n IN nodes
+ | FILTER n.provenance.source_uri != null
+ | AND n.kind NOT IN @ignoredKinds
+ | RETURN MAX([
+ | TO_NUMBER(n.created_rev),
+ | n.deleted_rev != null ? TO_NUMBER(n.deleted_rev) : 0
+ | ])
+ |)
+ |LET edge_rev = MAX(
+ | FOR e IN edges
+ | FILTER e.predicate IN @predicates
+ | RETURN MAX([
+ | TO_NUMBER(e.created_rev),
+ | e.deleted_rev != null ? TO_NUMBER(e.deleted_rev) : 0
+ | ])
+ |)
+ |RETURN MAX([
+ | node_rev != null ? node_rev : 0,
+ | edge_rev != null ? edge_rev : 0
+ |])""".stripMargin,
+ Map(
+ "ignoredKinds" -> IgnoredNodeKinds.toArray.asInstanceOf[AnyRef],
+ "predicates" -> CouplingPredicates.toArray.asInstanceOf[AnyRef]
+ )
+ ).map { json =>
+ Rev(json.flatMap(_.as[Long].toOption).getOrElse(0L))
+ }
+
+ def liveFileCount(): IO[Int] =
+ client.queryOne(
+ """FOR n IN nodes
+ | FILTER n.kind == "file"
+ | AND n.deleted_rev == null
+ | AND n.provenance.source_uri != null
+ | LET ext = LOWER(LAST(SPLIT(n.provenance.source_uri, ".")))
+ | FILTER ext IN @extensions
+ | COLLECT WITH COUNT INTO cnt
+ | RETURN cnt""".stripMargin,
+ Map("extensions" -> SourceExtensions.toArray.asInstanceOf[AnyRef])
+ ).map(_.flatMap(_.as[Int].toOption).getOrElse(0))
+
// ── ArangoDB queries ───────────────────────────────────────────────
/**
@@ -58,16 +131,16 @@ class MapGraphBuilder(client: ArangoClient) {
| FILTER n.kind == "file"
| AND n.deleted_rev == null
| AND n.provenance.source_uri != null
+ | LET ext = LOWER(LAST(SPLIT(n.provenance.source_uri, ".")))
+ | FILTER ext IN @extensions
| RETURN {id: n.logical_id, path: n.provenance.source_uri}""".stripMargin,
- Map.empty
+ Map("extensions" -> SourceExtensions.toArray.asInstanceOf[AnyRef])
).map { rows =>
rows.flatMap { json =>
val c = json.hcursor
for {
idStr <- c.get[String]("id").toOption
path <- c.get[String]("path").toOption
- ext = path.split("\\.").lastOption.getOrElse("").toLowerCase
- if SourceExtensions.contains(ext)
uuid <- try Some(java.util.UUID.fromString(idStr)) catch { case _: Exception => None }
} yield FileVertex(NodeId(uuid), path)
}.toVector
@@ -83,25 +156,27 @@ class MapGraphBuilder(client: ArangoClient) {
*/
private def fetchCouplingByUri(): IO[Vector[RawFilePair]] =
client.query(
- """LET uri_map = MERGE(
+ """LET uriMap = (
| FOR n IN nodes
| FILTER n.deleted_rev == null
| AND n.provenance.source_uri != null
- | AND n.kind NOT IN ["module","config","config_entry",
- | "doc","decision","intent","bug","plan",
- | "task","goal","region"]
- | RETURN {[n.logical_id]: n.provenance.source_uri}
+ | AND n.kind NOT IN @ignoredKinds
+ | RETURN { id: n.logical_id, uri: n.provenance.source_uri }
|)
+ |LET byId = ZIP(uriMap[*].id, uriMap[*].uri)
|FOR e IN edges
- | FILTER e.predicate IN ["CALLS","IMPORTS","EXTENDS","IMPLEMENTS"]
+ | FILTER e.predicate IN @predicates
| AND e.deleted_rev == null
- | LET su = HAS(uri_map, e.src) ? uri_map[e.src] : null
- | LET du = HAS(uri_map, e.dst) ? uri_map[e.dst] : null
+ | LET su = byId[e.src]
+ | LET du = byId[e.dst]
| FILTER su != null AND du != null AND su != du
| COLLECT srcUri = su, dstUri = du, pred = e.predicate
| WITH COUNT INTO cnt
| RETURN {srcUri, dstUri, predicate: pred, count: cnt}""".stripMargin,
- Map.empty
+ Map(
+ "ignoredKinds" -> IgnoredNodeKinds.toArray.asInstanceOf[AnyRef],
+ "predicates" -> CouplingPredicates.toArray.asInstanceOf[AnyRef]
+ )
).map { rows =>
rows.flatMap { json =>
val c = json.hcursor
@@ -124,18 +199,20 @@ class MapGraphBuilder(client: ArangoClient) {
// Index files by their absolute source_uri path
val byUri: Map[String, FileVertex] = files.map(v => v.path -> v).toMap
+ val byId: Map[NodeId, FileVertex] = files.map(v => v.id -> v).toMap
+ val pathSegmentsById: Map[NodeId, Array[String]] =
+ files.map(v => v.id -> directorySegments(v.path)).toMap
- // Accumulate coupling counts per (srcUri, dstUri) canonical pair
+ // Accumulate coupling counts per canonical file-id pair.
val pairAcc =
- scala.collection.mutable.Map[(String, String), scala.collection.mutable.Map[String, Int]]()
+ scala.collection.mutable.Map[(NodeId, NodeId), scala.collection.mutable.Map[String, Int]]()
for (pair <- rawPairs) {
val sv = byUri.get(pair.srcId) // srcId is source_uri here
val dv = byUri.get(pair.dstId)
(sv, dv) match {
case (Some(s), Some(d)) if s.id != d.id =>
- // Canonical undirected key: lexicographically smaller path first
- val key = if (s.path < d.path) (s.path, d.path) else (d.path, s.path)
+ val key = canonicalPair(s.id, d.id)
val m = pairAcc.getOrElseUpdate(key, scala.collection.mutable.Map())
m(pair.predicate) = m.getOrElse(pair.predicate, 0) + pair.count
case _ =>
@@ -145,43 +222,65 @@ class MapGraphBuilder(client: ArangoClient) {
// Build adjacency with composite weights
val adjMut =
scala.collection.mutable.Map[NodeId, scala.collection.mutable.Map[NodeId, Double]]()
+ val predicatePairs =
+ scala.collection.mutable.Map[(NodeId, NodeId), Map[String, Int]]()
+ var edgeCount = 0
- for (((sp, dp), predicateCounts) <- pairAcc) {
- val sv = byUri(sp)
- val dv = byUri(dp)
+ for (((srcId, dstId), predicateCounts) <- pairAcc.toVector.sortBy { case ((a, b), _) =>
+ (a.value.toString, b.value.toString)
+ }) {
+ val sv = byId(srcId)
+ val dv = byId(dstId)
+ val srcSegments = pathSegmentsById(srcId)
+ val dstSegments = pathSegmentsById(dstId)
+ val sameDir = isSameDirectory(srcSegments, dstSegments)
val callCount = predicateCounts.getOrElse("CALLS", 0)
val importCount = predicateCounts.getOrElse("IMPORTS", 0)
val typeCount = predicateCounts.getOrElse("EXTENDS", 0) +
predicateCounts.getOrElse("IMPLEMENTS", 0)
+ val pairSignals =
+ if (sameDir) predicateCounts.toMap.updated("PATH", predicateCounts.getOrElse("PATH", 0) + 1)
+ else predicateCounts.toMap
val sCall = Alpha * math.log1p(callCount)
val sImport = Beta * math.log1p(importCount)
val sType = Gamma * math.log1p(typeCount)
- val sPath = Epsilon * pathProximity(sv.path, dv.path)
+ val sPath = Epsilon * pathProximity(srcSegments, dstSegments, sameDir)
val w = sCall + sImport + sType + sPath
if (w > 0.0) {
adjMut.getOrElseUpdate(sv.id, scala.collection.mutable.Map())(dv.id) = w
adjMut.getOrElseUpdate(dv.id, scala.collection.mutable.Map())(sv.id) = w
+ predicatePairs((srcId, dstId)) = pairSignals
+ edgeCount += 1
}
}
// Path-proximity pass: add structural edges for same-directory file pairs
// that have no existing coupling edge. This ensures isolated files (no
// CALLS/IMPORTS/EXTENDS edges) can still be clustered by directory.
- val filesByDir = files.groupBy(v => v.path.split("[/\\\\]").dropRight(1).mkString("/"))
- for ((_, dirFiles) <- filesByDir if dirFiles.size >= 2) {
- for (i <- dirFiles.indices; j <- (i + 1) until dirFiles.size) {
- val sv = dirFiles(i)
- val dv = dirFiles(j)
- if (!adjMut.get(sv.id).exists(_.contains(dv.id))) {
- val w = Epsilon * pathProximity(sv.path, dv.path)
- if (w > 0.0) {
- adjMut.getOrElseUpdate(sv.id, scala.collection.mutable.Map())(dv.id) = w
- adjMut.getOrElseUpdate(dv.id, scala.collection.mutable.Map())(sv.id) = w
+ val filesByDir =
+ files.groupBy(v => pathSegmentsById(v.id).mkString("/")).toVector.sortBy(_._1)
+ for ((_, dirFiles) <- filesByDir if dirFiles.size >= PathProximityMinFiles &&
+ dirFiles.size <= PathProximityMaxFiles &&
+ edgeCount < MaxEdges) {
+ val ordered = dirFiles.sortBy(_.path)
+ var i = 0
+ while (i < ordered.length - 1 && edgeCount < MaxEdges) {
+ var j = i + 1
+ while (j < ordered.length && edgeCount < MaxEdges) {
+ val sv = ordered(i)
+ val dv = ordered(j)
+ if (!adjMut.get(sv.id).exists(_.contains(dv.id))) {
+ adjMut.getOrElseUpdate(sv.id, scala.collection.mutable.Map())(dv.id) = Epsilon
+ adjMut.getOrElseUpdate(dv.id, scala.collection.mutable.Map())(sv.id) = Epsilon
+ predicatePairs(canonicalPair(sv.id, dv.id)) = Map("PATH" -> 1)
+ edgeCount += 1
}
+ j += 1
}
+ i += 1
}
}
@@ -189,14 +288,34 @@ class MapGraphBuilder(client: ArangoClient) {
val degrees = adj.map { case (k, m) => k -> m.values.sum }
val totalWeight = degrees.values.sum / 2.0
- WeightedFileGraph(files, adj, degrees.toMap, totalWeight)
+ WeightedFileGraph(files, adj, degrees.toMap, totalWeight, predicatePairs.toMap)
+ }
+
+ private def canonicalPair(a: NodeId, b: NodeId): (NodeId, NodeId) =
+ if (a.value.compareTo(b.value) <= 0) (a, b) else (b, a)
+
+ private def directorySegments(path: String): Array[String] =
+ path.split("[/\\\\]").dropRight(1)
+
+ private def isSameDirectory(partsA: Array[String], partsB: Array[String]): Boolean = {
+ if (partsA.length != partsB.length) return false
+ var i = 0
+ while (i < partsA.length) {
+ if (partsA(i) != partsB(i)) return false
+ i += 1
+ }
+ true
}
/** Path distance heuristic: deeper common prefix → higher proximity. */
- private def pathProximity(a: String, b: String): Double = {
- val partsA = a.split("[/\\\\]").dropRight(1)
- val partsB = b.split("[/\\\\]").dropRight(1)
- val common = partsA.zip(partsB).takeWhile { case (x, y) => x == y }.length
+ private def pathProximity(partsA: Array[String], partsB: Array[String], sameDir: Boolean): Double = {
+ if (sameDir) return 1.0
+
+ val maxLen = math.min(partsA.length, partsB.length)
+ var common = 0
+ while (common < maxLen && partsA(common) == partsB(common)) {
+ common += 1
+ }
val dist = (partsA.length - common) + (partsB.length - common)
1.0 / (1.0 + dist)
}
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilderFast.scala b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilderFast.scala
index acd21d41..8db07619 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilderFast.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapGraphBuilderFast.scala
@@ -145,7 +145,7 @@ class MapGraphBuilderFast(client: ArangoClient) {
val degrees = adj.map { case (k, m) => k -> m.values.sum }
val totalWeight = degrees.values.sum / 2.0
- WeightedFileGraph(files, adj, degrees.toMap, totalWeight)
+ WeightedFileGraph(files, adj, degrees.toMap, totalWeight, Map.empty)
}
private def pathProximity(a: String, b: String): Double = {
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapService.scala b/memory-layer/src/main/scala/ix/memory/map/MapService.scala
index d2b6426e..f1c5c96a 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapService.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapService.scala
@@ -6,7 +6,6 @@ import java.util.UUID
import cats.effect.IO
import io.circe.Json
import io.circe.syntax._
-import org.typelevel.log4cats.slf4j.Slf4jLogger
import ix.memory.db.{ArangoClient, GraphQueryApi, GraphWriteApi}
import ix.memory.model._
@@ -27,6 +26,7 @@ class MapService(
queryApi: GraphQueryApi,
writeApi: GraphWriteApi
) {
+ private val CrosscutThreshold = 0.10
// Cross-cut penalty multiplier (reduces edge weights involving crosscut files)
private val CrosscutLambda = 0.5
@@ -36,16 +36,18 @@ class MapService(
private val CrosscutTokens = Set(
"util", "utils", "helper", "helpers", "common", "shared", "base",
"logging", "log", "logger", "config", "conf", "constants", "constant",
- "extensions", "ext", "mixin", "mixins", "core", "framework"
+ "extensions", "ext"
+ )
+ private val SignalWeights = Map(
+ "CALLS" -> 1.0,
+ "IMPORTS" -> 0.5,
+ "EXTENDS" -> 1.2,
+ "IMPLEMENTS" -> 1.2,
+ "PATH" -> 0.2
)
- private val logger = Slf4jLogger.getLoggerFromName[IO]("ix.map.service")
- private val builder = new MapGraphBuilder(client)
- private val fastBuilder = new MapGraphBuilderFast(client)
- private val preflight = new MapPreflight
-
- // Persistence guardrail: max total patch ops before we refuse to commit
- private val MaxSafePatchOps = 10000
+ private val builder = new MapGraphBuilder(client)
+ private final class RetainedRegion(val level: Int, val communityIndex: Int, val region: Region)
// Label kinds by level
private def labelKind(level: Int, maxLevel: Int): String =
@@ -54,65 +56,104 @@ class MapService(
else if (level == 1) "module"
else "subsystem"
- def buildMap(forceFullLocal: Boolean = false): IO[ArchitectureMap] =
+ def buildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =
+ getOrBuildMap(forceRecompute)
+
+ def getOrBuildMap(forceRecompute: Boolean = false): IO[ArchitectureMap] =
for {
- rev <- queryApi.getLatestRev
- files <- builder.discoverFiles()
- pf <- preflight.evaluate(files)
- mode = if (forceFullLocal) MapExecutionMode.FullLocal else pf.mode
- result <- mode match {
- case MapExecutionMode.FullLocal => buildFullLocal(files, pf, rev, forceFullLocal)
- case MapExecutionMode.FastLocal => buildFastLocal(files, pf, rev)
- }
- } yield result
-
- private def buildFullLocal(
- files: Vector[FileVertex],
- pf: MapPreflightResult,
- rev: Rev,
- bypassPersistenceGuard: Boolean = false
- ): IO[ArchitectureMap] =
+ inputRev <- builder.currentInputRev()
+ cached <- loadCachedMap(inputRev)
+ archMap <- if (cached.nonEmpty && !forceRecompute) IO.pure(cached.get)
+ else computeAndPersistMap(inputRev, cached)
+ } yield archMap
+
+ def loadPersistedMapForCurrentGraph(): IO[Option[ArchitectureMap]] =
for {
- rawGraph <- builder.buildGraph(files)
- crosscutScores = detectCrosscut(rawGraph)
- graph = applyPenalty(rawGraph, crosscutScores)
- levels = LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3)
- regions = buildRegions(graph, levels, crosscutScores, rev)
- pe <- guardAndPersist(regions, rev, bypassPersistenceGuard)
- _ <- logOutcome(MapOutcome.FullLocalCompleted, pf, pe)
- } yield ArchitectureMap(
- regions, rawGraph.vertices.size, rev.value,
- preflight = Some(pf),
- outcome = MapOutcome.FullLocalCompleted,
- persistenceEstimate = Some(pe)
+ inputRev <- builder.currentInputRev()
+ cached <- loadCachedMap(inputRev)
+ } yield cached
+
+ def loadCachedMap(): IO[Option[ArchitectureMap]] =
+ loadPersistedMapForCurrentGraph()
+
+ def resolveRegion(map: ArchitectureMap, target: String): Vector[RegionMatch] = {
+ val query = target.trim
+ val normalizedQuery = normalizeLabelQuery(query)
+ if (query.isEmpty || normalizedQuery.isEmpty) Vector.empty
+ else {
+ map.regions.flatMap { region =>
+ matchQuality(region.label, query).map { quality =>
+ RegionMatch(region, quality, normalizeLabelQuery(region.label))
+ }
+ }.sortBy { m =>
+ (
+ m.matchQuality,
+ levelPreference(m.region),
+ -m.region.memberFiles.size,
+ -m.region.confidence,
+ m.normalizedLabel,
+ m.region.id.value.toString
+ )
+ }
+ }
+ }
+
+ def childrenOf(map: ArchitectureMap, region: Region): Vector[Region] = {
+ val regionById = map.regions.map(r => r.id -> r).toMap
+ val directChildren = region.childRegionIds.toVector
+ .flatMap(regionById.get)
+ .filter(_.level < region.level)
+
+ val children =
+ if (directChildren.nonEmpty) directChildren
+ else map.regions.filter(_.parentId.contains(region.id))
+
+ sortBySignificance(children.distinct)
+ }
+
+ def parentOf(map: ArchitectureMap, region: Region): Option[Region] = {
+ val regionById = map.regions.map(r => r.id -> r).toMap
+ region.parentId.flatMap(regionById.get)
+ }
+
+ def subtreeOf(map: ArchitectureMap, region: Region): Vector[Region] = {
+ def loop(current: Region): Vector[Region] = {
+ val directChildren = childrenOf(map, current)
+ current +: directChildren.flatMap(loop)
+ }
+ loop(region)
+ }
+
+ def scopedView(map: ArchitectureMap, region: Region): ScopedSubsystemView = {
+ val scopeRegions =
+ if (region.labelKind == "module") Vector(region)
+ else subtreeOf(map, region)
+
+ val summary = ScopedHealthSummary(
+ wellDefined = scopeRegions.count(_.confidence >= 0.75),
+ moderate = scopeRegions.count(r => r.confidence >= 0.50 && r.confidence < 0.75),
+ fuzzy = scopeRegions.count(_.confidence < 0.50),
+ crossCutting = scopeRegions.count(isCrossCutting)
)
- private def buildFastLocal(
- files: Vector[FileVertex],
- pf: MapPreflightResult,
- rev: Rev
- ): IO[ArchitectureMap] =
- for {
- rawGraph <- fastBuilder.buildGraph(files)
- crosscutScores = detectCrosscut(rawGraph)
- graph = applyPenalty(rawGraph, crosscutScores)
- levels = LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3)
- regions = buildRegions(graph, levels, crosscutScores, rev)
- pe <- guardAndPersist(regions, rev)
- _ <- logOutcome(MapOutcome.FastLocalCompleted, pf, pe)
- } yield ArchitectureMap(
- regions, rawGraph.vertices.size, rev.value,
- preflight = Some(pf),
- outcome = MapOutcome.FastLocalCompleted,
- persistenceEstimate = Some(pe)
+ ScopedSubsystemView(
+ target = region,
+ parent = parentOf(map, region),
+ children = if (region.labelKind == "module") Vector.empty else childrenOf(map, region),
+ subtree = scopeRegions,
+ summary = summary
)
+ }
+
+ def isCrossCutting(region: Region): Boolean =
+ region.crosscutScore > CrosscutThreshold
// ── Cross-cutting detection ─────────────────────────────────────────
private def detectCrosscut(graph: WeightedFileGraph): Map[NodeId, Double] = {
if (graph.degrees.isEmpty) return Map.empty
val maxDeg = graph.degrees.values.max.max(1e-10)
- val threshold = maxDeg * (1.0 - CrosscutDegreeTop)
+ val threshold = highDegreeThreshold(graph)
graph.vertices.map { v =>
val deg = graph.degrees.getOrElse(v.id, 0.0)
@@ -127,6 +168,16 @@ class MapService(
}.toMap
}
+ private def highDegreeThreshold(graph: WeightedFileGraph): Double = {
+ val sortedDegrees = graph.degrees.values.toVector.sorted
+ if (sortedDegrees.isEmpty) 0.0
+ else {
+ val cutoffIndex = math.ceil(sortedDegrees.size * (1.0 - CrosscutDegreeTop)).toInt - 1
+ val clamped = cutoffIndex.max(0).min(sortedDegrees.size - 1)
+ sortedDegrees(clamped)
+ }
+ }
+
private def applyPenalty(
graph: WeightedFileGraph,
scores: Map[NodeId, Double]
@@ -161,34 +212,46 @@ class MapService(
// Compute common absolute-path prefix to strip for label inference
val allPaths = graph.vertices.map(_.path)
val commonPfxLen = commonPathPrefixLength(allPaths)
+ val relativePaths = allPaths.map(path => relativizePath(path, commonPfxLen))
+ val globalDirTokenCounts = directoryTokenCounts(relativePaths)
+ val signalAdjacency = buildSignalAdjacency(graph)
// Build regions for each level, finest first.
// Apply minCommunitySize at every level — singleton/doubleton regions are noise.
val minSize = if (maxLevel >= 3) 3 else 2
- val allRegions = levels.zipWithIndex.flatMap { case (partition, levelIdx) =>
+ val retainedRegions = levels.zipWithIndex.flatMap { case (partition, levelIdx) =>
val level = levelIdx + 1
- partition.communities.filter(_.size >= minSize).map { memberFiles =>
- buildOneRegion(
+ partition.communities.zipWithIndex.collect {
+ case (memberFiles, communityIndex) if memberFiles.size >= minSize =>
+ new RetainedRegion(
+ level = level,
+ communityIndex = communityIndex,
+ region = buildOneRegion(
memberFiles, level, labelKind(level, maxLevel),
- graph, fileVertexById, crosscutScores, rev, commonPfxLen
- )
+ graph, fileVertexById, crosscutScores, signalAdjacency, rev, commonPfxLen, globalDirTokenCounts
+ )
+ )
}
}
+ val retainedByLevel = retainedRegions.groupBy(_.level)
+ .view
+ .mapValues(_.map(retained => retained.communityIndex -> retained.region).toMap)
+ .toMap
// Wire parent/child relationships
- // For each region at level N, find which level-(N+1) region contains its files
- val byLevel: Map[Int, Vector[Region]] =
- allRegions.groupBy(_.level)
-
- val withParents = allRegions.map { region =>
- val parentOpt = byLevel.get(region.level + 1).flatMap { parents =>
- // Parent must be strictly coarser: strictly larger member set (no equal sets)
- parents
- .filter(p => p.memberFiles.size > region.memberFiles.size &&
- region.memberFiles.subsetOf(p.memberFiles))
- .minByOption(_.memberFiles.size) // pick tightest enclosing region
+ val withParents = retainedRegions.map { retained =>
+ val parentOpt = levels.lift(retained.level).flatMap { coarseLevel =>
+ val candidateCommunities = retained.region.memberFiles.iterator
+ .flatMap(fileId => coarseLevel.assignment.get(fileId))
+ .toSet
+ if (candidateCommunities.size != 1) None
+ else {
+ retainedByLevel.get(retained.level + 1)
+ .flatMap(_.get(candidateCommunities.head))
+ .filter(_.memberFiles.size > retained.region.memberFiles.size)
+ }
}
- region.copy(parentId = parentOpt.map(_.id))
+ retained.region.copy(parentId = parentOpt.map(_.id))
}
// Fill childRegionIds
@@ -222,10 +285,14 @@ class MapService(
val otherSegs = dirSegs(otherPaths)
val exclusive = (mySegs -- otherSegs).filterNot(LabelNoiseTokens.contains)
- // Prefer shorter tokens (more likely to be a meaningful module name)
+ // Prefer shorter tokens (more likely to be a meaningful module name).
+ // Fallback: rank by size — largest keeps the clean label, smaller ones get "(2)", "(3)", etc.
val suffix = exclusive.toVector.sortBy(s => (s.length, s)).headOption
.map(s => s" / ${s.capitalize}")
- .getOrElse(s" (${r.memberFiles.size}f)")
+ .getOrElse {
+ val rank = group.sortBy(-_.memberFiles.size).indexWhere(_.id == r.id) + 1
+ if (rank <= 1) "" else s" ($rank)"
+ }
r.copy(label = r.label + suffix)
case None => r
}
@@ -240,6 +307,13 @@ class MapService(
"ix-memory", "memory-layer", "ix-cli", "core-ingestion", "ix-plugin",
"projects", "claude"
)
+ private val WeakLabelTokens = Set(
+ "api", "apis", "service", "services", "controller", "controllers",
+ "repository", "repositories", "handler", "handlers", "module", "modules",
+ "common", "shared", "core", "base", "internal", "external", "model",
+ "models", "entity", "entities", "dto", "dtos", "types", "errors",
+ "interface", "interfaces"
+ )
private def buildOneRegion(
memberFiles: Set[NodeId],
@@ -248,8 +322,10 @@ class MapService(
graph: WeightedFileGraph,
fileVertexById: Map[NodeId, FileVertex],
crosscutScores: Map[NodeId, Double],
+ signalAdjacency: Map[NodeId, Vector[(NodeId, Map[String, Int])]],
rev: Rev,
- commonPfxLen: Int = 0
+ commonPfxLen: Int = 0,
+ globalDirTokenCounts: Map[String, Int] = Map.empty
): Region = {
// Deterministic ID: stable across re-runs with same file set
val idKey = "map:region:" + memberFiles.map(_.value.toString).toList.sorted.mkString(",")
@@ -259,7 +335,6 @@ class MapService(
var internalWeight = 0.0
var externalWeight = 0.0
var boundaryNodes = Set.empty[NodeId]
- var signalCounts = Map("calls" -> 0.0, "imports" -> 0.0, "type" -> 0.0)
for (fileId <- memberFiles) {
for ((neighbor, w) <- graph.adjMatrix.getOrElse(fileId, Map.empty)) {
@@ -284,8 +359,9 @@ class MapService(
val crosscutScore =
memberFiles.toList.map(f => crosscutScores.getOrElse(f, 0.0)).sum / n
- val label = inferLabel(memberFiles, fileVertexById, commonPfxLen)
+ val label = inferLabel(memberFiles, fileVertexById, commonPfxLen, globalDirTokenCounts)
val confidence = computeConfidence(cohesion, boundaryRatio, memberFiles.size, crosscutScore)
+ val dominantSignals = computeDominantSignals(memberFiles, signalAdjacency)
Region(
id = nodeId,
@@ -300,7 +376,7 @@ class MapService(
boundaryRatio = boundaryRatio,
confidence = confidence,
crosscutScore = crosscutScore,
- dominantSignals = List("calls", "imports", "type").sortBy(s => -signalCounts.getOrElse(s, 0.0)),
+ dominantSignals = dominantSignals,
interfaceNodeCount = boundaryNodes.size,
mapRev = rev.value
)
@@ -321,51 +397,28 @@ class MapService(
private def inferLabel(
memberFiles: Set[NodeId],
fileVertexById: Map[NodeId, FileVertex],
- commonPfxLen: Int = 0
+ commonPfxLen: Int = 0,
+ globalDirTokenCounts: Map[String, Int] = Map.empty
): String = {
val rawPaths = memberFiles.flatMap(f => fileVertexById.get(f).map(_.path)).toVector
if (rawPaths.isEmpty) return "Unknown"
- // Strip common absolute prefix (e.g. /home/user/project/) so labels are project-relative
- val paths = rawPaths.map { p =>
- val segs = p.split("[/\\\\]")
- if (commonPfxLen > 0 && commonPfxLen < segs.length) segs.drop(commonPfxLen).mkString("/")
- else p
- }
+ val paths = rawPaths.map(path => relativizePath(path, commonPfxLen))
val noiseTokens = LabelNoiseTokens
- // Strategy 1: Longest common directory suffix across all paths
- val dirSegments = paths.map(p => p.split("[/\\\\]").dropRight(1).toVector)
- val commonDir = longestCommonSuffix(dirSegments)
- .filterNot(s => noiseTokens.contains(s.toLowerCase))
- .headOption
-
- if (commonDir.isDefined) return commonDir.get.capitalize
-
- // Strategy 2: Most discriminative non-noise directory segment.
- // Prefer deeper (more specific) segments by scoring: frequency × depth-weight.
- // Split each path into (depth, segment) pairs so we can weight deeper dirs higher.
- val allDirsWithDepth: Vector[(Int, String)] = paths.flatMap { p =>
- val segs = p.split("[/\\\\]").dropRight(1)
- segs.zipWithIndex.map { case (s, i) => (i, s) }
- .filter { case (_, s) => !noiseTokens.contains(s.toLowerCase) && s.nonEmpty }
- }
- // Score: count * (1 + depth / maxDepth) — deeper segments score higher
- val maxDepth = allDirsWithDepth.map(_._1).maxOption.getOrElse(1).toDouble.max(1.0)
- val scored = allDirsWithDepth.groupBy(_._2).map { case (seg, entries) =>
- seg -> entries.length * (1.0 + entries.map(_._1 / maxDepth).sum / entries.length)
- }.toVector.sortBy(-_._2)
-
- scored.headOption.map(_._1.capitalize).getOrElse {
- // Strategy 3: Most frequent meaningful filename token
- val fileTokens = paths.flatMap { p =>
- val name = p.split("[/\\\\]").lastOption.getOrElse("")
- val noExt = name.replaceAll("\\.[^.]+$", "")
- splitCamel(noExt).filter(t => t.length > 3).map(_.toLowerCase)
- }
- fileTokens.filterNot(noiseTokens.contains)
- .groupBy(identity).maxByOption(_._2.size)
- .map(_._1.capitalize).getOrElse("Region")
+ val suffixLabel = longestCommonSuffix(paths.map(path => directorySegments(path).toVector))
+ .iterator
+ .map(_.toLowerCase)
+ .find(token => token.nonEmpty && !noiseTokens.contains(token))
+
+ suffixLabel match {
+ case Some(token) if !WeakLabelTokens.contains(token) =>
+ humanizeLabel(token)
+ case _ =>
+ bestDiscriminativeDirectoryToken(paths, globalDirTokenCounts)
+ .orElse(bestFilenameToken(paths))
+ .map(humanizeLabel)
+ .getOrElse("Region")
}
}
@@ -390,6 +443,145 @@ class MapService(
private def splitCamel(s: String): Array[String] =
s.split("(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|[_\\-.]")
+ private def relativizePath(path: String, commonPfxLen: Int): String = {
+ val segs = path.split("[/\\\\]")
+ if (commonPfxLen > 0 && commonPfxLen < segs.length) segs.drop(commonPfxLen).mkString("/")
+ else path
+ }
+
+ private def directorySegments(path: String): Vector[String] =
+ path.split("[/\\\\]").dropRight(1).toVector
+
+ private def directoryTokenCounts(paths: Vector[String]): Map[String, Int] =
+ paths.flatMap(directorySegments)
+ .map(_.toLowerCase)
+ .filter(token => token.nonEmpty && !LabelNoiseTokens.contains(token))
+ .groupBy(identity)
+ .map { case (token, entries) => token -> entries.size }
+
+ private def bestDiscriminativeDirectoryToken(
+ paths: Vector[String],
+ globalDirTokenCounts: Map[String, Int]
+ ): Option[String] = {
+ val clusterCounts = directoryTokenCounts(paths)
+ clusterCounts.toVector
+ .map { case (token, intraCount) =>
+ val globalCount = globalDirTokenCounts.getOrElse(token, intraCount).max(intraCount)
+ val score = intraCount.toDouble * (intraCount.toDouble / globalCount.toDouble)
+ (token, score, intraCount, globalCount)
+ }
+ .sortBy { case (token, score, intraCount, globalCount) =>
+ (-score, globalCount, -intraCount, token)
+ }
+ .headOption
+ .map(_._1)
+ }
+
+ private def bestFilenameToken(paths: Vector[String]): Option[String] = {
+ val fileTokens = paths.flatMap { p =>
+ val name = p.split("[/\\\\]").lastOption.getOrElse("")
+ val noExt = name.replaceAll("\\.[^.]+$", "")
+ splitCamel(noExt)
+ .map(_.toLowerCase)
+ .filter(token => token.length > 3 && !LabelNoiseTokens.contains(token))
+ }
+
+ fileTokens.groupBy(identity)
+ .toVector
+ .sortBy { case (token, entries) => (-entries.size, token) }
+ .headOption
+ .map(_._1)
+ }
+
+ private def humanizeLabel(token: String): String =
+ token.split("[-_\\s]+").filter(_.nonEmpty).map(_.capitalize).mkString(" ")
+
+ private def normalizeWhitespace(input: String): String =
+ input.trim.toLowerCase.replaceAll("\\s+", " ")
+
+ private def normalizeLabelQuery(input: String): String =
+ normalizeWhitespace(input.replaceAll("[/_-]+", " "))
+
+ private def matchQuality(label: String, query: String): Option[Int] = {
+ val trimmedLabel = label.trim
+ val trimmedQuery = query.trim
+ val whitespaceLabel = normalizeWhitespace(trimmedLabel)
+ val whitespaceQuery = normalizeWhitespace(trimmedQuery)
+ val normalizedLabel = normalizeLabelQuery(trimmedLabel)
+ val normalizedQuery = normalizeLabelQuery(trimmedQuery)
+
+ if (trimmedLabel.equalsIgnoreCase(trimmedQuery)) Some(0)
+ else if (whitespaceLabel == whitespaceQuery) Some(1)
+ else if (normalizedLabel == normalizedQuery) Some(2)
+ else if (normalizedLabel.startsWith(normalizedQuery)) Some(3)
+ else if (normalizedLabel.contains(normalizedQuery)) Some(4)
+ else None
+ }
+
+ private def levelPreference(region: Region): Int =
+ region.labelKind match {
+ case "subsystem" => 0
+ case "module" => 1
+ case "system" => 2
+ case _ => 3
+ }
+
+ private def sortBySignificance(regions: Seq[Region]): Vector[Region] =
+ regions.toVector.sortBy(r => (-r.memberFiles.size, -r.confidence, r.label.toLowerCase, r.id.value.toString))
+
+ private def buildSignalAdjacency(
+ graph: WeightedFileGraph
+ ): Map[NodeId, Vector[(NodeId, Map[String, Int])]] = {
+ val adjacency = scala.collection.mutable.Map.empty[NodeId, Vector[(NodeId, Map[String, Int])]]
+ for (((src, dst), predicateCounts) <- graph.predicatePairs) {
+ adjacency.update(src, adjacency.getOrElse(src, Vector.empty) :+ (dst -> predicateCounts))
+ adjacency.update(dst, adjacency.getOrElse(dst, Vector.empty) :+ (src -> predicateCounts))
+ }
+ adjacency.toMap
+ }
+
+ private def computeDominantSignals(
+ memberFiles: Set[NodeId],
+ signalAdjacency: Map[NodeId, Vector[(NodeId, Map[String, Int])]]
+ ): List[String] = {
+ val signalScores = scala.collection.mutable.Map(
+ "calls" -> 0.0,
+ "imports" -> 0.0,
+ "type" -> 0.0,
+ "path" -> 0.0
+ ).withDefaultValue(0.0)
+
+ for {
+ src <- memberFiles.iterator
+ (dst, predicateCounts) <- signalAdjacency.getOrElse(src, Vector.empty)
+ if memberFiles.contains(dst)
+ if src.value.compareTo(dst.value) < 0
+ (predicate, count) <- predicateCounts
+ if count > 0
+ } {
+ predicate match {
+ case "CALLS" =>
+ signalScores("calls") += SignalWeights("CALLS") * math.log1p(count.toDouble)
+ case "IMPORTS" =>
+ signalScores("imports") += SignalWeights("IMPORTS") * math.log1p(count.toDouble)
+ case "EXTENDS" | "IMPLEMENTS" =>
+ signalScores("type") += SignalWeights(predicate) * math.log1p(count.toDouble)
+ case "PATH" =>
+ signalScores("path") += SignalWeights("PATH") * count.toDouble
+ case _ =>
+ }
+ }
+
+ val ranked = signalScores.toVector
+ .filter(_._2 > 0.0)
+ .sortBy { case (name, score) => (-score, name) }
+ .map(_._1)
+ .take(3)
+ .toList
+
+ if (ranked.nonEmpty) ranked else List("path")
+ }
+
// ── Confidence scoring ──────────────────────────────────────────────
private def computeConfidence(
@@ -398,72 +590,148 @@ class MapService(
size: Int,
crosscutScore: Double
): Double = {
- // Normalize boundary ratio: high ratio → high confidence
- val brScore = math.tanh(boundaryRatio / 3.0) // saturates around ratio = 6
+ val brScore = math.tanh(boundaryRatio / 2.5)
val cohScore = cohesion.min(1.0)
- val sizeBonus = math.tanh(size / 10.0) // more files → more signal
- val crossPenalty = crosscutScore * 0.3
+ val sizeBonus = math.tanh(size / 8.0)
+ val crossPenalty = crosscutScore * 0.4
((0.4 * brScore + 0.3 * cohScore + 0.3 * sizeBonus) - crossPenalty).max(0.0).min(1.0)
}
- // ── Persistence guardrail + logging ─────────────────────────────────
+ // ── Persistence ─────────────────────────────────────────────────────
- private def guardAndPersist(
- regions: Vector[Region],
- rev: Rev,
- bypassGuard: Boolean = false
- ): IO[PersistenceEstimate] = {
- if (regions.isEmpty) return IO.pure(PersistenceEstimate(0, 0, 0, 0, 0))
+ private def computeAndPersistMap(inputRev: Rev, cached: Option[ArchitectureMap]): IO[ArchitectureMap] =
+ for {
+ rawGraph <- builder.build()
+ crosscutScores = detectCrosscut(rawGraph)
+ graph = applyPenalty(rawGraph, crosscutScores)
+ levels = LouvainClustering.cluster(graph, maxLevels = 3, minCommunitySize = 3)
+ regions = buildRegions(graph, levels, crosscutScores, inputRev)
+ fresh = ArchitectureMap(regions, rawGraph.vertices.size, inputRev.value)
+ _ <- persistMap(fresh, cached)
+ } yield fresh
+
+ private def loadCachedMap(inputRev: Rev): IO[Option[ArchitectureMap]] =
+ for {
+ rows <- client.query(
+ """FOR n IN nodes
+ | FILTER n.kind == "region"
+ | AND n.deleted_rev == null
+ | AND n.attrs.map_generated == true
+ | AND TO_NUMBER(n.attrs.map_rev) == @mapRev
+ | SORT TO_NUMBER(n.attrs.level) DESC, n.name ASC
+ | RETURN {
+ | id: n.logical_id,
+ | name: n.name,
+ | attrs: n.attrs
+ | }""".stripMargin,
+ Map("mapRev" -> Long.box(inputRev.value).asInstanceOf[AnyRef])
+ )
+ decoded = rows.flatMap(decodePersistedRegion(_, inputRev)).toVector
+ regions = rehydrateRelationships(decoded)
+ fileCount <- if (regions.isEmpty) IO.pure(0) else builder.liveFileCount()
+ } yield {
+ if (regions.isEmpty) None
+ else Some(ArchitectureMap(regions, fileCount, inputRev.value))
+ }
+
+ private def decodePersistedRegion(json: Json, inputRev: Rev): Option[Region] = {
+ val c = json.hcursor
+ val attrs = c.downField("attrs")
for {
- oldRegions <- queryApi.findNodesByKind(NodeKind.Region, limit = 5000)
- pe = estimatePersistence(regions, oldRegions.size)
- _ <- logger.debug(
- s"Persistence estimate: ${pe.totalOps} ops " +
- s"(${pe.regionNodes} regions, ${pe.fileEdges} file edges, " +
- s"${pe.regionEdges} region edges, ${pe.deleteOps} deletes)" +
- (if (bypassGuard) " [guard bypassed via --full]" else "")
+ idStr <- c.get[String]("id").toOption
+ label <- c.get[String]("name").toOption
+ id <- parseNodeId(idStr)
+ level <- attrs.get[Int]("level").toOption
+ labelKind <- attrs.get[String]("label_kind").toOption
+ } yield {
+ val memberFiles = attrs.get[Vector[String]]("member_files").getOrElse(Vector.empty).flatMap(parseNodeId).toSet
+ val childIds = attrs.get[Vector[String]]("child_region_ids").getOrElse(Vector.empty).flatMap(parseNodeId).toSet
+ val parentId = attrs.get[Option[String]]("parent_id").getOrElse(None).flatMap(parseNodeId)
+
+ Region(
+ id = id,
+ label = label,
+ labelKind = labelKind,
+ level = level,
+ memberFiles = memberFiles,
+ childRegionIds = childIds,
+ parentId = parentId,
+ cohesion = attrs.get[Double]("cohesion").getOrElse(0.0),
+ externalCoupling = attrs.get[Double]("external_coupling").getOrElse(0.0),
+ boundaryRatio = attrs.get[Double]("boundary_ratio").getOrElse(0.0),
+ confidence = attrs.get[Double]("confidence").getOrElse(0.0),
+ crosscutScore = attrs.get[Double]("crosscut_score").getOrElse(0.0),
+ dominantSignals = attrs.get[List[String]]("dominant_signals").getOrElse(Nil),
+ interfaceNodeCount = attrs.get[Int]("interface_node_count").getOrElse(0),
+ mapRev = attrs.get[Long]("map_rev").getOrElse(inputRev.value)
)
- _ <- if (!bypassGuard && pe.totalOps > MaxSafePatchOps)
- IO.raiseError(new MapCapacityException(
- outcome = MapOutcome.LocalMapTooLarge,
- userMessage = "Local map output is too large to persist safely.",
- next = "For full system mapping at this scale, use Ix Cloud."
- ))
- else if (bypassGuard && pe.totalOps > MaxSafePatchOps)
- logger.info(
- s"Persistence guard bypassed (--full): ${pe.totalOps} ops exceeds " +
- s"limit of $MaxSafePatchOps, continuing anyway"
- )
- else IO.unit
- _ <- submitMapPatch(oldRegions, regions, rev)
- } yield pe
+ }
}
- private def estimatePersistence(regions: Vector[Region], oldRegionCount: Int): PersistenceEstimate = {
- val regionNodes = regions.size
- val fileEdges = regions.filter(_.level == 1).map(_.memberFiles.size).sum
- val regionEdges = regions.count(_.parentId.isDefined)
- val deleteOps = oldRegionCount
- val totalOps = deleteOps + regionNodes + fileEdges + regionEdges
- PersistenceEstimate(regionNodes, fileEdges, regionEdges, deleteOps, totalOps)
+ private def parseNodeId(id: String): Option[NodeId] =
+ scala.util.Try(UUID.fromString(id)).toOption.map(NodeId(_))
+
+ private def rehydrateRelationships(regions: Vector[Region]): Vector[Region] = {
+ if (regions.isEmpty) return regions
+
+ val parentByChild = regions.iterator
+ .flatMap(region => region.childRegionIds.iterator.map(childId => childId -> region.id))
+ .toMap
+
+ val childrenByParent = regions
+ .flatMap(region => region.parentId.map(parentId => parentId -> region.id))
+ .groupBy(_._1)
+ .view
+ .mapValues(_.map(_._2).toSet)
+ .toMap
+
+ regions.map { region =>
+ val parentId = region.parentId.orElse(parentByChild.get(region.id))
+ val childIds =
+ if (region.childRegionIds.nonEmpty) region.childRegionIds
+ else childrenByParent.getOrElse(region.id, Set.empty)
+ region.copy(parentId = parentId, childRegionIds = childIds)
+ }
}
- private def logOutcome(
- outcome: MapOutcome,
- pf: MapPreflightResult,
- pe: PersistenceEstimate
- ): IO[Unit] =
- logger.info(
- s"Map completed: outcome=${outcome.label} " +
- s"mode=${pf.mode.label} risk=${pf.risk.label} " +
- s"files=${pf.cost.fileCount} regions=${pe.regionNodes} " +
- s"patchOps=${pe.totalOps} preflightMs=${pf.durationMs}"
- )
+ private def persistMap(fresh: ArchitectureMap, cached: Option[ArchitectureMap]): IO[Unit] = {
+ if (cached.exists(existing => sameRegions(existing.regions, fresh.regions))) IO.unit
+ else {
+ for {
+ oldRegions <- queryApi.findNodesByKind(NodeKind.Region, limit = 5000)
+ oldEdgeIds <- loadActiveRegionEdgeIds(oldRegions.map(_.id))
+ _ <- submitMapPatch(oldRegions, oldEdgeIds, fresh.regions, Rev(fresh.mapRev))
+ } yield ()
+ }
+ }
+
+ private def sameRegions(left: Vector[Region], right: Vector[Region]): Boolean =
+ left.sortBy(_.id.value.toString) == right.sortBy(_.id.value.toString)
+
+ private def loadActiveRegionEdgeIds(regionIds: Vector[NodeId]): IO[Vector[EdgeId]] =
+ if (regionIds.isEmpty) IO.pure(Vector.empty)
+ else {
+ val ids = regionIds.map(_.value.toString).toArray
+ client.query(
+ """FOR e IN edges
+ | FILTER e.deleted_rev == null
+ | AND e.predicate == "IN_REGION"
+ | AND (e.src IN @regionIds OR e.dst IN @regionIds)
+ | RETURN e._key""".stripMargin,
+ Map("regionIds" -> ids.asInstanceOf[AnyRef])
+ ).map(_.flatMap(_.as[String].toOption).flatMap(id => scala.util.Try(UUID.fromString(id)).toOption.map(EdgeId(_))).toVector)
+ }
- private def submitMapPatch(oldRegions: Vector[GraphNode], regions: Vector[Region], rev: Rev): IO[Unit] = {
+ private def submitMapPatch(
+ oldRegions: Vector[GraphNode],
+ oldEdgeIds: Vector[EdgeId],
+ regions: Vector[Region],
+ rev: Rev
+ ): IO[Unit] = {
val deleteOps: Vector[PatchOp] = oldRegions.map(n => PatchOp.DeleteNode(n.id))
+ val deleteEdgeOps: Vector[PatchOp] = oldEdgeIds.map(PatchOp.DeleteEdge)
val nodeOps: Vector[PatchOp] = regions.map { r =>
PatchOp.UpsertNode(
@@ -471,11 +739,14 @@ class MapService(
kind = NodeKind.Region,
name = r.label,
attrs = Map(
+ "label" -> r.label.asJson,
"map_generated" -> Json.True,
"level" -> r.level.asJson,
"label_kind" -> r.labelKind.asJson,
"file_count" -> r.memberFiles.size.asJson,
"member_files" -> r.memberFiles.map(_.value.toString).asJson,
+ "parent_id" -> r.parentId.map(_.value.toString).asJson,
+ "child_region_ids" -> r.childRegionIds.toVector.map(_.value.toString).sorted.asJson,
"cohesion" -> r.cohesion.asJson,
"external_coupling" -> r.externalCoupling.asJson,
"boundary_ratio" -> r.boundaryRatio.asJson,
@@ -522,7 +793,9 @@ class MapService(
}
}
- val allOps = deleteOps ++ nodeOps ++ fileEdgeOps ++ regionEdgeOps
+ val allOps = deleteEdgeOps ++ deleteOps ++ nodeOps ++ fileEdgeOps ++ regionEdgeOps
+ if (allOps.isEmpty) return IO.unit
+
val patch = GraphPatch(
patchId = PatchId(UUID.nameUUIDFromBytes(s"map:${rev.value}".getBytes("UTF-8"))),
actor = "map-service",
diff --git a/memory-layer/src/main/scala/ix/memory/map/MapTypes.scala b/memory-layer/src/main/scala/ix/memory/map/MapTypes.scala
index 924e514f..f19e7101 100644
--- a/memory-layer/src/main/scala/ix/memory/map/MapTypes.scala
+++ b/memory-layer/src/main/scala/ix/memory/map/MapTypes.scala
@@ -21,12 +21,13 @@ final case class WeightedFileGraph(
vertices: Vector[FileVertex],
adjMatrix: Map[NodeId, Map[NodeId, Double]],
degrees: Map[NodeId, Double],
- totalWeight: Double
+ totalWeight: Double,
+ predicatePairs: Map[(NodeId, NodeId), Map[String, Int]]
)
object WeightedFileGraph {
val empty: WeightedFileGraph =
- WeightedFileGraph(Vector.empty, Map.empty, Map.empty, 0.0)
+ WeightedFileGraph(Vector.empty, Map.empty, Map.empty, 0.0, Map.empty)
}
/**
@@ -134,3 +135,24 @@ final case class ArchitectureMap(
outcome: MapOutcome = MapOutcome.FullLocalCompleted,
persistenceEstimate: Option[PersistenceEstimate] = None
)
+
+final case class RegionMatch(
+ region: Region,
+ matchQuality: Int,
+ normalizedLabel: String
+)
+
+final case class ScopedHealthSummary(
+ wellDefined: Int,
+ moderate: Int,
+ fuzzy: Int,
+ crossCutting: Int
+)
+
+final case class ScopedSubsystemView(
+ target: Region,
+ parent: Option[Region],
+ children: Vector[Region],
+ subtree: Vector[Region],
+ summary: ScopedHealthSummary
+)
diff --git a/memory-layer/src/main/scala/ix/memory/model/Claim.scala b/memory-layer/src/main/scala/ix/memory/model/Claim.scala
index 64571d6b..c3523dfa 100644
--- a/memory-layer/src/main/scala/ix/memory/model/Claim.scala
+++ b/memory-layer/src/main/scala/ix/memory/model/Claim.scala
@@ -27,15 +27,18 @@ object ClaimStatus {
}
final case class Claim(
- id: ClaimId,
- entityId: NodeId,
- statement: String,
- value: Json,
- confidence: Option[Double],
- status: ClaimStatus,
- provenance: Provenance,
- createdRev: Rev,
- deletedRev: Option[Rev]
+ id: ClaimId,
+ entityId: NodeId,
+ statement: String,
+ value: Json,
+ confidence: Option[Double],
+ status: ClaimStatus,
+ provenance: Provenance,
+ createdRev: Rev,
+ deletedRev: Option[Rev],
+ /** Identifies the inference pass that produced this claim (e.g. "smell_v1", "subsystem_v1").
+ * Absent for observed facts extracted from source code. Missing legacy claims are treated as "legacy". */
+ inferenceVersion: Option[String] = None
)
object Claim {
diff --git a/memory-layer/src/main/scala/ix/memory/model/GraphPatch.scala b/memory-layer/src/main/scala/ix/memory/model/GraphPatch.scala
index 05dceabc..7c0b7881 100644
--- a/memory-layer/src/main/scala/ix/memory/model/GraphPatch.scala
+++ b/memory-layer/src/main/scala/ix/memory/model/GraphPatch.scala
@@ -12,7 +12,14 @@ object PatchOp {
final case class UpsertEdge(id: EdgeId, src: NodeId, dst: NodeId, predicate: EdgePredicate, attrs: Map[String, Json]) extends PatchOp
final case class DeleteNode(id: NodeId) extends PatchOp
final case class DeleteEdge(id: EdgeId) extends PatchOp
- final case class AssertClaim(entityId: NodeId, field: String, value: Json, confidence: Option[Double]) extends PatchOp
+ final case class AssertClaim(
+ entityId: NodeId,
+ field: String,
+ value: Json,
+ confidence: Option[Double],
+ /** Inference pass identifier. Absent for observed facts; required for inferred claims. */
+ inferenceVersion: Option[String] = None
+ ) extends PatchOp
final case class RetractClaim(claimId: ClaimId) extends PatchOp
implicit val encoder: Encoder[PatchOp] = Encoder.instance {
@@ -22,8 +29,8 @@ object PatchOp {
Json.obj("type" -> "UpsertEdge".asJson, "id" -> id.asJson, "src" -> src.asJson, "dst" -> dst.asJson, "predicate" -> predicate.asJson, "attrs" -> attrs.asJson)
case DeleteNode(id) => Json.obj("type" -> "DeleteNode".asJson, "id" -> id.asJson)
case DeleteEdge(id) => Json.obj("type" -> "DeleteEdge".asJson, "id" -> id.asJson)
- case AssertClaim(entityId, field, value, confidence) =>
- Json.obj("type" -> "AssertClaim".asJson, "entityId" -> entityId.asJson, "field" -> field.asJson, "value" -> value.asJson, "confidence" -> confidence.asJson)
+ case AssertClaim(entityId, field, value, confidence, inferenceVersion) =>
+ Json.obj("type" -> "AssertClaim".asJson, "entityId" -> entityId.asJson, "field" -> field.asJson, "value" -> value.asJson, "confidence" -> confidence.asJson, "inferenceVersion" -> inferenceVersion.asJson)
case RetractClaim(claimId) => Json.obj("type" -> "RetractClaim".asJson, "claimId" -> claimId.asJson)
}
@@ -48,11 +55,12 @@ object PatchOp {
case "DeleteEdge" => c.downField("id").as[EdgeId].map(DeleteEdge)
case "AssertClaim" =>
for {
- entityId <- c.downField("entityId").as[NodeId]
- field <- c.downField("field").as[String]
- value <- c.downField("value").as[Json]
- confidence <- c.downField("confidence").as[Option[Double]]
- } yield AssertClaim(entityId, field, value, confidence)
+ entityId <- c.downField("entityId").as[NodeId]
+ field <- c.downField("field").as[String]
+ value <- c.downField("value").as[Json]
+ confidence <- c.downField("confidence").as[Option[Double]]
+ inferenceVersion <- c.downField("inferenceVersion").as[Option[String]]
+ } yield AssertClaim(entityId, field, value, confidence, inferenceVersion)
case "RetractClaim" => c.downField("claimId").as[ClaimId].map(RetractClaim)
case other => Left(DecodingFailure(s"Unknown PatchOp type: $other", c.history))
}
diff --git a/memory-layer/src/main/scala/ix/memory/model/Node.scala b/memory-layer/src/main/scala/ix/memory/model/Node.scala
index 164417b6..38ca41bc 100644
--- a/memory-layer/src/main/scala/ix/memory/model/Node.scala
+++ b/memory-layer/src/main/scala/ix/memory/model/Node.scala
@@ -28,6 +28,7 @@ object NodeKind {
case object Bug extends NodeKind
case object Goal extends NodeKind
case object Region extends NodeKind
+ case object Chunk extends NodeKind
private val nameMap: Map[String, NodeKind] = Map(
"module" -> Module,
@@ -50,7 +51,8 @@ object NodeKind {
"task" -> Task,
"bug" -> Bug,
"goal" -> Goal,
- "region" -> Region
+ "region" -> Region,
+ "chunk" -> Chunk
)
implicit val encoder: Encoder[NodeKind] = Encoder[String].contramap {
@@ -75,6 +77,7 @@ object NodeKind {
case Bug => "bug"
case Goal => "goal"
case Region => "region"
+ case Chunk => "chunk"
}
implicit val decoder: Decoder[NodeKind] = Decoder[String].emap { s =>
diff --git a/memory-layer/src/main/scala/ix/memory/smell/SmellService.scala b/memory-layer/src/main/scala/ix/memory/smell/SmellService.scala
new file mode 100644
index 00000000..7a2c3d92
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/smell/SmellService.scala
@@ -0,0 +1,254 @@
+package ix.memory.smell
+
+import java.time.Instant
+import java.util.UUID
+
+import cats.effect.IO
+import cats.syntax.traverse._
+import io.circe.Json
+import io.circe.syntax._
+
+import ix.memory.db.{ArangoClient, GraphWriteApi}
+import ix.memory.model._
+
+// ---------------------------------------------------------------------------
+// Data types
+// ---------------------------------------------------------------------------
+
+sealed trait SmellKind { def field: String }
+object SmellKind {
+ case object OrphanFile extends SmellKind { val field = "has_smell.orphan_file" }
+ case object GodModule extends SmellKind { val field = "has_smell.god_module" }
+ case object WeakComponent extends SmellKind { val field = "has_smell.weak_component_member" }
+}
+
+final case class SmellCandidate(
+ fileId: String, // logical_id UUID
+ fileName: String,
+ smellKind: SmellKind,
+ confidence: Double,
+ signals: Map[String, Json] // evidence key → value
+)
+
+final case class SmellConfig(
+ /** Max total edge connectivity for a file to be an orphan candidate. */
+ orphanMaxConnections: Int = 0,
+ /** Min chunk count to flag a god module. */
+ godModuleChunkThreshold: Int = 20,
+ /** Min fan-in or fan-out (IMPORTS edges) to flag a god module. */
+ godModuleFanThreshold: Int = 15,
+ /** Max unique file neighbors for a weak-component candidate. */
+ weakComponentMaxNeighbors: Int = 1
+)
+
+final case class SmellReport(
+ candidates: Vector[SmellCandidate],
+ rev: Long,
+ runAt: String
+)
+
+// ---------------------------------------------------------------------------
+// Service
+// ---------------------------------------------------------------------------
+
+class SmellService(client: ArangoClient, writeApi: GraphWriteApi) {
+
+ private val InferenceVersion = "smell_v1"
+ private val Actor = "ix/smell"
+
+ def run(config: SmellConfig = SmellConfig()): IO[SmellReport] =
+ for {
+ orphans <- detectOrphans(config)
+ godMods <- detectGodModules(config)
+ weakComps <- detectWeakComponents(config)
+ candidates = orphans ++ godMods ++ weakComps
+ rev <- commitSmells(candidates)
+ } yield SmellReport(candidates, rev, Instant.now().toString)
+
+ /** Retrieve existing smell claims from the graph without rerunning detection. */
+ def listSmells(): IO[Vector[Json]] =
+ client.query(
+ """FOR c IN claims
+ | FILTER STARTS_WITH(c.field, "has_smell.")
+ | AND c.status == "active"
+ | AND c.deleted_rev == null
+ | SORT c.entity_id, c.field
+ | RETURN {
+ | entity_id: c.entity_id,
+ | smell: c.field,
+ | value: c.value,
+ | inference_version: c.inference_version
+ | }""".stripMargin,
+ Map.empty
+ ).map(_.toVector)
+
+ // ── Detectors ────────────────────────────────────────────────────────────
+
+ private def detectOrphans(cfg: SmellConfig): IO[Vector[SmellCandidate]] =
+ client.query(
+ """FOR f IN nodes
+ | FILTER f.kind == "file" AND f.deleted_rev == null
+ | LET imports_out = LENGTH(
+ | FOR e IN edges FILTER e.src == f.logical_id AND e.predicate == "IMPORTS" AND e.deleted_rev == null RETURN 1)
+ | LET imports_in = LENGTH(
+ | FOR e IN edges FILTER e.dst == f.logical_id AND e.predicate == "IMPORTS" AND e.deleted_rev == null RETURN 1)
+ | LET calls_out = LENGTH(
+ | FOR e IN edges FILTER e.src == f.logical_id AND e.predicate == "CALLS" AND e.deleted_rev == null RETURN 1)
+ | LET calls_in = LENGTH(
+ | FOR e IN edges FILTER e.dst == f.logical_id AND e.predicate == "CALLS" AND e.deleted_rev == null RETURN 1)
+ | LET connectivity = imports_out + imports_in + calls_out + calls_in
+ | FILTER connectivity <= @threshold
+ | RETURN {
+ | logical_id: f.logical_id, name: f.name,
+ | imports_out, imports_in, calls_out, calls_in, connectivity
+ | }""".stripMargin,
+ Map("threshold" -> Int.box(cfg.orphanMaxConnections).asInstanceOf[AnyRef])
+ ).map(_.flatMap { json =>
+ for {
+ id <- json.hcursor.get[String]("logical_id").toOption
+ name <- json.hcursor.get[String]("name").toOption
+ connectivity <- json.hcursor.get[Int]("connectivity").toOption
+ importsOut <- json.hcursor.get[Int]("imports_out").toOption
+ importsIn <- json.hcursor.get[Int]("imports_in").toOption
+ callsOut <- json.hcursor.get[Int]("calls_out").toOption
+ callsIn <- json.hcursor.get[Int]("calls_in").toOption
+ } yield SmellCandidate(
+ fileId = id,
+ fileName = name,
+ smellKind = SmellKind.OrphanFile,
+ confidence = if (connectivity == 0) 0.85 else 0.6,
+ signals = Map(
+ "connectivity" -> connectivity.asJson,
+ "imports_out" -> importsOut.asJson,
+ "imports_in" -> importsIn.asJson,
+ "calls_out" -> callsOut.asJson,
+ "calls_in" -> callsIn.asJson
+ )
+ )
+ }.toVector)
+
+ private def detectGodModules(cfg: SmellConfig): IO[Vector[SmellCandidate]] =
+ client.query(
+ """FOR f IN nodes
+ | FILTER f.kind == "file" AND f.deleted_rev == null
+ | LET chunks = LENGTH(
+ | FOR e IN edges FILTER e.src == f.logical_id AND e.predicate == "CONTAINS_CHUNK" AND e.deleted_rev == null RETURN 1)
+ | LET symbols = LENGTH(
+ | FOR e IN edges FILTER e.src == f.logical_id AND e.predicate == "CONTAINS" AND e.deleted_rev == null RETURN 1)
+ | LET fan_in = LENGTH(
+ | FOR e IN edges FILTER e.dst == f.logical_id AND e.predicate == "IMPORTS" AND e.deleted_rev == null RETURN 1)
+ | LET fan_out = LENGTH(
+ | FOR e IN edges FILTER e.src == f.logical_id AND e.predicate == "IMPORTS" AND e.deleted_rev == null RETURN 1)
+ | FILTER chunks >= @chunkThreshold OR fan_in >= @fanThreshold OR fan_out >= @fanThreshold
+ | RETURN { logical_id: f.logical_id, name: f.name, chunks, symbols, fan_in, fan_out }""".stripMargin,
+ Map(
+ "chunkThreshold" -> Int.box(cfg.godModuleChunkThreshold).asInstanceOf[AnyRef],
+ "fanThreshold" -> Int.box(cfg.godModuleFanThreshold).asInstanceOf[AnyRef]
+ )
+ ).map(_.flatMap { json =>
+ for {
+ id <- json.hcursor.get[String]("logical_id").toOption
+ name <- json.hcursor.get[String]("name").toOption
+ chunks <- json.hcursor.get[Int]("chunks").toOption
+ symbols <- json.hcursor.get[Int]("symbols").toOption
+ fanIn <- json.hcursor.get[Int]("fan_in").toOption
+ fanOut <- json.hcursor.get[Int]("fan_out").toOption
+ } yield {
+ val triggeredSignals = Seq(
+ Option.when(chunks >= cfg.godModuleChunkThreshold)(s"chunks=$chunks"),
+ Option.when(fanIn >= cfg.godModuleFanThreshold)(s"fan_in=$fanIn"),
+ Option.when(fanOut >= cfg.godModuleFanThreshold)(s"fan_out=$fanOut")
+ ).flatten
+ SmellCandidate(
+ fileId = id,
+ fileName = name,
+ smellKind = SmellKind.GodModule,
+ confidence = Math.min(0.5 + triggeredSignals.size * 0.15, 0.9),
+ signals = Map(
+ "chunks" -> chunks.asJson,
+ "symbols" -> symbols.asJson,
+ "fan_in" -> fanIn.asJson,
+ "fan_out" -> fanOut.asJson,
+ "triggers" -> triggeredSignals.asJson
+ )
+ )
+ }
+ }.toVector)
+
+ private def detectWeakComponents(cfg: SmellConfig): IO[Vector[SmellCandidate]] =
+ client.query(
+ """FOR f IN nodes
+ | FILTER f.kind == "file" AND f.deleted_rev == null
+ | LET neighbor_ids = UNIQUE(APPEND(
+ | (FOR e IN edges
+ | FILTER e.src == f.logical_id
+ | AND e.predicate IN ["IMPORTS","CALLS"]
+ | AND e.deleted_rev == null
+ | RETURN e.dst),
+ | (FOR e IN edges
+ | FILTER e.dst == f.logical_id
+ | AND e.predicate IN ["IMPORTS","CALLS"]
+ | AND e.deleted_rev == null
+ | RETURN e.src)
+ | ))
+ | LET neighbor_count = LENGTH(neighbor_ids)
+ | FILTER neighbor_count > 0 AND neighbor_count <= @maxNeighbors
+ | RETURN { logical_id: f.logical_id, name: f.name, neighbor_count }""".stripMargin,
+ Map("maxNeighbors" -> Int.box(cfg.weakComponentMaxNeighbors).asInstanceOf[AnyRef])
+ ).map(_.flatMap { json =>
+ for {
+ id <- json.hcursor.get[String]("logical_id").toOption
+ name <- json.hcursor.get[String]("name").toOption
+ neighborCount <- json.hcursor.get[Int]("neighbor_count").toOption
+ } yield SmellCandidate(
+ fileId = id,
+ fileName = name,
+ smellKind = SmellKind.WeakComponent,
+ confidence = 0.55,
+ signals = Map("neighbor_count" -> neighborCount.asJson)
+ )
+ }.toVector)
+
+ // ── Claim persistence ────────────────────────────────────────────────────
+
+ private def commitSmells(candidates: Vector[SmellCandidate]): IO[Long] = {
+ if (candidates.isEmpty) return client.query(
+ "FOR rev IN latest_rev SORT rev.value DESC LIMIT 1 RETURN rev.value",
+ Map.empty
+ ).map(_.headOption.flatMap(_.as[Long].toOption).getOrElse(0L))
+
+ val ops: Vector[PatchOp] = candidates.map { c =>
+ PatchOp.AssertClaim(
+ entityId = NodeId(UUID.fromString(c.fileId)),
+ field = c.smellKind.field,
+ value = Json.obj(
+ "confidence" -> c.confidence.asJson,
+ "signals" -> c.signals.asJson,
+ "file" -> c.fileName.asJson
+ ),
+ confidence = Some(c.confidence),
+ inferenceVersion = Some(InferenceVersion)
+ )
+ }
+
+ val patchId = PatchId(UUID.randomUUID())
+ val sourceHash = UUID.nameUUIDFromBytes(s"smell_run:${Instant.now()}".getBytes("UTF-8")).toString
+ val patch = GraphPatch(
+ patchId = patchId,
+ actor = Actor,
+ timestamp = Instant.now(),
+ source = PatchSource(
+ uri = "ix://smell-detector",
+ sourceHash = Some(sourceHash),
+ extractor = s"smell-detector/$InferenceVersion",
+ sourceType = SourceType.Code
+ ),
+ baseRev = Rev(0L),
+ ops = ops,
+ replaces = Vector.empty,
+ intent = Some(s"Smell detection: ${candidates.size} candidates")
+ )
+
+ writeApi.commitPatch(patch).map(_.newRev.value)
+ }
+}
diff --git a/memory-layer/src/main/scala/ix/memory/smell/SubsystemService.scala b/memory-layer/src/main/scala/ix/memory/smell/SubsystemService.scala
new file mode 100644
index 00000000..5aff04db
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/smell/SubsystemService.scala
@@ -0,0 +1,195 @@
+package ix.memory.smell
+
+import java.util.UUID
+
+import cats.effect.IO
+import io.circe.Json
+import io.circe.syntax._
+
+import ix.memory.db.{ArangoClient, GraphQueryApi}
+
+class SubsystemService(client: ArangoClient, queryApi: GraphQueryApi) {
+
+ private val InferenceVersion = "subsystem_v1"
+ private val mapper = new com.fasterxml.jackson.databind.ObjectMapper()
+
+ // ── Public API ──────────────────────────────────────────────────────
+
+ def scoreSubsystems(): IO[Json] =
+ for {
+ rows <- queryRawData()
+ scores = rows.map(computeScore)
+ rev <- queryApi.getLatestRev
+ _ <- retireOldScores()
+ _ <- storeScores(scores, rev.value)
+ } yield Json.obj("scores" -> scores.map(encodeScore).asJson)
+
+ def listSubsystems(): IO[Json] =
+ client.query(
+ """FOR c IN claims
+ | FILTER c.field == "subsystem_health" AND c.deleted_rev == null
+ | RETURN {
+ | region_id: c.entity_id,
+ | name: c.value.name,
+ | level: c.value.level,
+ | label_kind: c.value.label_kind,
+ | file_count: c.value.file_count,
+ | health_score: c.value.health_score,
+ | chunk_density: c.value.chunk_density,
+ | smell_rate: c.value.smell_rate,
+ | smell_files: c.value.smell_files,
+ | total_chunks: c.value.total_chunks,
+ | confidence: c.value.confidence,
+ | inference_version: c.value.inference_version
+ | }""".stripMargin,
+ Map.empty
+ ).map(scores => Json.obj("scores" -> scores.asJson))
+
+ // ── Internal data model ─────────────────────────────────────────────
+
+ private case class RawData(
+ regionId: String,
+ name: String,
+ level: Int,
+ labelKind: String,
+ fileCount: Int,
+ totalChunks: Int,
+ smellFiles: Int,
+ confidence: Double
+ )
+
+ private case class Score(
+ regionId: String,
+ name: String,
+ level: Int,
+ labelKind: String,
+ fileCount: Int,
+ healthScore: Double,
+ chunkDensity: Double,
+ smellRate: Double,
+ smellFiles: Int,
+ totalChunks: Int,
+ confidence: Double
+ )
+
+ // ── Raw data query ──────────────────────────────────────────────────
+
+ /**
+ * Single AQL query that joins regions with smell claims and CONTAINS_CHUNK
+ * edge counts. The MERGE + SUM approach avoids N+1 queries.
+ */
+ private def queryRawData(): IO[List[RawData]] =
+ client.query(
+ """LET smell_ids = (
+ | FOR c IN claims
+ | FILTER STARTS_WITH(c.field, "has_smell.") AND c.deleted_rev == null
+ | RETURN DISTINCT c.entity_id
+ |)
+ |LET chunk_map = MERGE(
+ | FOR e IN edges
+ | FILTER e.predicate == "CONTAINS" AND e.deleted_rev == null
+ | COLLECT src = e.src INTO g
+ | RETURN { [src]: LENGTH(g) }
+ |)
+ |FOR n IN nodes
+ | FILTER n.kind == "region" AND n.deleted_rev == null
+ | LET mf = n.attrs.member_files != null ? n.attrs.member_files : []
+ | FILTER LENGTH(mf) > 0
+ | LET file_count = LENGTH(mf)
+ | LET total_chunks = SUM(
+ | FOR f IN mf RETURN HAS(chunk_map, f) ? chunk_map[f] : 0
+ | )
+ | LET smell_count = LENGTH(
+ | FOR f IN mf FILTER f IN smell_ids RETURN 1
+ | )
+ | RETURN {
+ | region_id: n.logical_id,
+ | name: n.name,
+ | level: n.attrs.level,
+ | label_kind: n.attrs.label_kind,
+ | file_count: file_count,
+ | total_chunks: total_chunks,
+ | smell_files: smell_count,
+ | confidence: n.attrs.confidence != null ? n.attrs.confidence : 0.0
+ | }""".stripMargin,
+ Map.empty
+ ).map(_.flatMap { json =>
+ val c = json.hcursor
+ for {
+ regionId <- c.get[String]("region_id").toOption
+ name <- c.get[String]("name").toOption
+ level <- c.get[Int]("level").toOption
+ labelKind <- c.get[String]("label_kind").toOption
+ fileCount <- c.get[Int]("file_count").toOption
+ totalChunks <- c.get[Int]("total_chunks").toOption
+ smellFiles <- c.get[Int]("smell_files").toOption
+ confidence <- c.get[Double]("confidence").toOption
+ } yield RawData(regionId, name, level, labelKind, fileCount, totalChunks, smellFiles, confidence)
+ })
+
+ // ── Scoring formula ─────────────────────────────────────────────────
+
+ /**
+ * Health score formula (from spec):
+ * 0.4 × (1 − smell_rate) + 0.3 × tanh(chunk_density / 5) + 0.3 × confidence
+ */
+ private def computeScore(raw: RawData): Score = {
+ val n = raw.fileCount.toDouble.max(1)
+ val chunkDensity = raw.totalChunks.toDouble / n
+ val smellRate = raw.smellFiles.toDouble / n
+ val healthScore = (0.4 * (1 - smellRate) +
+ 0.3 * Math.tanh(chunkDensity / 5) +
+ 0.3 * raw.confidence).max(0.0).min(1.0)
+ Score(raw.regionId, raw.name, raw.level, raw.labelKind,
+ raw.fileCount, healthScore, chunkDensity, smellRate,
+ raw.smellFiles, raw.totalChunks, raw.confidence)
+ }
+
+ // ── Persistence ─────────────────────────────────────────────────────
+
+ private def retireOldScores(): IO[Unit] =
+ client.execute(
+ """FOR c IN claims
+ | FILTER c.field == "subsystem_health" AND c.deleted_rev == null
+ | REMOVE c IN claims""".stripMargin,
+ Map.empty
+ )
+
+ private def storeScores(scores: List[Score], rev: Long): IO[Unit] =
+ if (scores.isEmpty) IO.unit
+ else {
+ val docs = scores.map { s =>
+ val key = UUID.nameUUIDFromBytes(s"subsystem:${s.regionId}".getBytes("UTF-8")).toString
+ val value = encodeScore(s)
+ val doc = new java.util.HashMap[String, AnyRef]()
+ doc.put("_key", key)
+ doc.put("entity_id", s.regionId)
+ doc.put("field", "subsystem_health")
+ doc.put("value", mapper.readValue(value.noSpaces, classOf[AnyRef]))
+ doc.put("confidence", java.lang.Double.valueOf(s.healthScore))
+ doc.put("status", "active")
+ doc.put("created_rev", Long.box(rev))
+ doc.put("deleted_rev", null)
+ doc: java.util.Map[String, AnyRef]
+ }
+ client.bulkInsert("claims", docs, "replace").void
+ }
+
+ // ── Encoding ────────────────────────────────────────────────────────
+
+ private def encodeScore(s: Score): Json =
+ Json.obj(
+ "region_id" -> s.regionId.asJson,
+ "name" -> s.name.asJson,
+ "level" -> s.level.asJson,
+ "label_kind" -> s.labelKind.asJson,
+ "file_count" -> s.fileCount.asJson,
+ "health_score" -> s.healthScore.asJson,
+ "chunk_density" -> s.chunkDensity.asJson,
+ "smell_rate" -> s.smellRate.asJson,
+ "smell_files" -> s.smellFiles.asJson,
+ "total_chunks" -> s.totalChunks.asJson,
+ "confidence" -> s.confidence.asJson,
+ "inference_version" -> InferenceVersion.asJson
+ )
+}
diff --git a/memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala b/memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala
new file mode 100644
index 00000000..c6013199
--- /dev/null
+++ b/memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala
@@ -0,0 +1,224 @@
+package ix.memory.subsystem
+
+import java.time.Instant
+import java.util.UUID
+
+import cats.effect.IO
+import io.circe.Json
+import io.circe.syntax._
+
+import ix.memory.db.{ArangoClient, GraphWriteApi}
+import ix.memory.map.MapService
+import ix.memory.model._
+
+// ---------------------------------------------------------------------------
+// Data types
+// ---------------------------------------------------------------------------
+
+final case class SubsystemScore(
+ regionId: String,
+ regionName: String,
+ level: Int,
+ labelKind: String,
+ fileCount: Int,
+ totalChunks: Int,
+ smellFiles: Int,
+ confidence: Double,
+ chunkDensity: Double,
+ smellRate: Double,
+ healthScore: Double
+)
+
+// ---------------------------------------------------------------------------
+// Service
+// ---------------------------------------------------------------------------
+
+/**
+ * Computes evidence-based health scores for each architectural region (subsystem).
+ *
+ * Score formula (range 0–1):
+ * health = 0.4*(1 − smellRate) + 0.3*tanh(chunkDensity/5) + 0.3*confidence
+ *
+ * Signals:
+ * chunkDensity — avg CONTAINS_CHUNK edges per member file (more = richer structure)
+ * smellRate — fraction of member files that have an active smell claim (lower = healthier)
+ * confidence — MapService cohesion/boundary score carried from the region node
+ *
+ * Results stored as `has_score.subsystem_health` claims on Region nodes,
+ * with inferenceVersion = "subsystem_v1".
+ */
+class SubsystemScoringService(
+ client: ArangoClient,
+ writeApi: GraphWriteApi,
+ mapService: MapService
+) {
+
+ private val InferenceVersion = "subsystem_v1"
+ private val Actor = "ix/subsystem-scorer"
+ private val ScoreField = "has_score.subsystem_health"
+
+ /** Run scoring: query chunk/smell metrics, compute health, persist claims. */
+ def score(): IO[Vector[SubsystemScore]] =
+ for {
+ archMap <- mapService.getOrBuildMap()
+ rows <- queryRegionMetrics(archMap.regions.map(_.id.value.toString), archMap.mapRev)
+ scores = rows.flatMap(computeScore)
+ _ <- persistScores(scores)
+ } yield scores
+
+ /** Retrieve previously stored health-score claims joined with region metadata. */
+ def listScores(): IO[Vector[Json]] =
+ client.query(
+ """FOR r IN nodes
+ | FILTER r.kind == "region" AND r.deleted_rev == null
+ | LET score_claim = FIRST(
+ | FOR c IN claims
+ | FILTER c.entity_id == r.logical_id
+ | AND c.field == "has_score.subsystem_health"
+ | AND c.status == "active"
+ | AND c.deleted_rev == null
+ | SORT c.created_rev DESC
+ | LIMIT 1
+ | RETURN c
+ | )
+ | FILTER score_claim != null
+ | SORT score_claim.value.health_score DESC
+ | RETURN {
+ | region_id: r.logical_id,
+ | name: r.name,
+ | level: r.attrs.level,
+ | label_kind: r.attrs.label_kind,
+ | file_count: r.attrs.file_count,
+ | health_score: score_claim.value.health_score,
+ | chunk_density: score_claim.value.chunk_density,
+ | smell_rate: score_claim.value.smell_rate,
+ | smell_files: score_claim.value.smell_files,
+ | total_chunks: score_claim.value.total_chunks,
+ | confidence: score_claim.value.confidence,
+ | inference_version: score_claim.inference_version
+ | }""".stripMargin,
+ Map.empty
+ ).map(_.toVector)
+
+ // ── Internals ─────────────────────────────────────────────────────────────
+
+ /**
+ * Single AQL pass: for each region node, count CONTAINS_CHUNK edges across
+ * member files, and count distinct member files that have any smell claim.
+ */
+ private def queryRegionMetrics(regionIds: Vector[String], mapRev: Long): IO[Vector[Json]] =
+ if (regionIds.isEmpty) IO.pure(Vector.empty)
+ else
+ client.query(
+ """FOR r IN nodes
+ | FILTER r.kind == "region" AND r.deleted_rev == null
+ | AND r.logical_id IN @regionIds
+ | AND TO_NUMBER(r.attrs.map_rev) == @mapRev
+ | LET chunk_count = LENGTH(
+ | FOR e IN edges
+ | FILTER e.src IN r.attrs.member_files
+ | AND e.predicate == "CONTAINS_CHUNK"
+ | AND e.deleted_rev == null
+ | RETURN 1
+ | )
+ | LET smell_files = LENGTH(UNIQUE(
+ | FOR c IN claims
+ | FILTER c.entity_id IN r.attrs.member_files
+ | AND STARTS_WITH(c.field, "has_smell.")
+ | AND c.status == "active"
+ | AND c.deleted_rev == null
+ | RETURN c.entity_id
+ | ))
+ | RETURN {
+ | id: r.logical_id,
+ | name: r.name,
+ | level: r.attrs.level,
+ | label_kind: r.attrs.label_kind,
+ | file_count: r.attrs.file_count,
+ | confidence: r.attrs.confidence,
+ | chunk_count: chunk_count,
+ | smell_files: smell_files
+ | }""".stripMargin,
+ Map(
+ "regionIds" -> regionIds.toArray.asInstanceOf[AnyRef],
+ "mapRev" -> Long.box(mapRev).asInstanceOf[AnyRef]
+ )
+ ).map(_.toVector)
+
+ private def computeScore(json: Json): Option[SubsystemScore] = {
+ val c = json.hcursor
+ for {
+ regionId <- c.get[String]("id").toOption
+ regionName <- c.get[String]("name").toOption
+ } yield {
+ val level = c.get[Int]("level").getOrElse(1)
+ val labelKind = c.get[String]("label_kind").getOrElse("module")
+ val fileCount = c.get[Int]("file_count").getOrElse(1).max(1)
+ val confidence = c.get[Double]("confidence").getOrElse(0.0)
+ val totalChunks = c.get[Int]("chunk_count").getOrElse(0)
+ val smellFiles = c.get[Int]("smell_files").getOrElse(0)
+
+ val chunkDensity = totalChunks.toDouble / fileCount
+ val smellRate = smellFiles.toDouble / fileCount
+ val normDensity = Math.tanh(chunkDensity / 5.0)
+ val healthScore = (0.4 * (1.0 - smellRate) + 0.3 * normDensity + 0.3 * confidence)
+ .max(0.0).min(1.0)
+
+ SubsystemScore(
+ regionId = regionId,
+ regionName = regionName,
+ level = level,
+ labelKind = labelKind,
+ fileCount = fileCount,
+ totalChunks = totalChunks,
+ smellFiles = smellFiles,
+ confidence = confidence,
+ chunkDensity = chunkDensity,
+ smellRate = smellRate,
+ healthScore = healthScore
+ )
+ }
+ }
+
+ private def persistScores(scores: Vector[SubsystemScore]): IO[Long] = {
+ if (scores.isEmpty) return IO.pure(0L)
+
+ val ops: Vector[PatchOp] = scores.flatMap { s =>
+ scala.util.Try(UUID.fromString(s.regionId)).toOption.map { uuid =>
+ PatchOp.AssertClaim(
+ entityId = NodeId(uuid),
+ field = ScoreField,
+ value = Json.obj(
+ "health_score" -> s.healthScore.asJson,
+ "chunk_density" -> s.chunkDensity.asJson,
+ "smell_rate" -> s.smellRate.asJson,
+ "smell_files" -> s.smellFiles.asJson,
+ "total_chunks" -> s.totalChunks.asJson,
+ "file_count" -> s.fileCount.asJson,
+ "confidence" -> s.confidence.asJson
+ ),
+ confidence = Some(s.healthScore),
+ inferenceVersion = Some(InferenceVersion)
+ )
+ }
+ }
+
+ val patch = GraphPatch(
+ patchId = PatchId(UUID.randomUUID()),
+ actor = Actor,
+ timestamp = Instant.now(),
+ source = PatchSource(
+ uri = "ix://subsystem-scorer",
+ sourceHash = None,
+ extractor = s"subsystem-scorer/$InferenceVersion",
+ sourceType = SourceType.Inferred
+ ),
+ baseRev = Rev(0L),
+ ops = ops,
+ replaces = Vector.empty,
+ intent = Some(s"Subsystem health scoring: ${scores.size} regions")
+ )
+
+ writeApi.commitPatch(patch).map(_.newRev.value)
+ }
+}
diff --git a/memory-layer/src/test/scala/ix/memory/api/RoutesSpec.scala b/memory-layer/src/test/scala/ix/memory/api/RoutesSpec.scala
index dfc965ed..35ec4613 100644
--- a/memory-layer/src/test/scala/ix/memory/api/RoutesSpec.scala
+++ b/memory-layer/src/test/scala/ix/memory/api/RoutesSpec.scala
@@ -20,6 +20,8 @@ import ix.memory.db._
import ix.memory.ingestion._
import ix.memory.map.MapService
import ix.memory.model._
+import ix.memory.smell.SmellService
+import ix.memory.subsystem.SubsystemScoringService
class RoutesSpec extends AsyncFlatSpec with AsyncIOSpec with Matchers with TestDbHelper {
@@ -68,8 +70,22 @@ class RoutesSpec extends AsyncFlatSpec with AsyncIOSpec with Matchers with TestD
val bulkWriteApi = new BulkWriteApi(client)
val bulkIngestionService = new BulkIngestionService(parserRouter, bulkWriteApi, queryApi)
val mapService = new MapService(client, queryApi, writeApi)
+ val smellService = new SmellService(client, writeApi)
+ val subsystemService = new SubsystemScoringService(client, writeApi, mapService)
- Routes.all(contextService, ingestionService, bulkIngestionService, queryApi, writeApi, conflictService, client, mapService, bulkWriteApi).orNotFound
+ Routes.all(
+ contextService,
+ ingestionService,
+ bulkIngestionService,
+ queryApi,
+ writeApi,
+ conflictService,
+ client,
+ mapService,
+ bulkWriteApi,
+ smellService,
+ subsystemService
+ ).orNotFound
}
"Routes" should "return 200 OK for health check" in {
diff --git a/memory-layer/src/test/scala/ix/memory/ingestion/FileDiscoverySpec.scala b/memory-layer/src/test/scala/ix/memory/ingestion/FileDiscoverySpec.scala
new file mode 100644
index 00000000..806e275c
--- /dev/null
+++ b/memory-layer/src/test/scala/ix/memory/ingestion/FileDiscoverySpec.scala
@@ -0,0 +1,26 @@
+package ix.memory.ingestion
+
+import java.nio.file.Paths
+
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class FileDiscoverySpec extends AnyFlatSpec with Matchers {
+
+ "extensionsFor(None)" should "default to parser-supported file types" in {
+ val extensions = FileDiscovery.extensionsFor(None)
+
+ extensions should contain allOf (".py", ".ts", ".tsx", ".scala", ".sc", ".json", ".yaml", ".yml", ".toml", ".md")
+ extensions should not contain ".txt"
+ extensions should not contain ".ini"
+ extensions should not contain ".sbt"
+ }
+
+ "matchesFilter" should "exclude unsupported generic-text files by default" in {
+ val extensions = FileDiscovery.extensionsFor(None)
+
+ FileDiscovery.matchesFilter(Paths.get("README.md"), extensions) shouldBe true
+ FileDiscovery.matchesFilter(Paths.get("Dockerfile"), extensions) shouldBe false
+ FileDiscovery.matchesFilter(Paths.get("notes.txt"), extensions) shouldBe false
+ }
+}
diff --git a/memory-layer/src/test/scala/ix/memory/ingestion/GenericTextFallbackSpec.scala b/memory-layer/src/test/scala/ix/memory/ingestion/GenericTextFallbackSpec.scala
new file mode 100644
index 00000000..14fca5c8
--- /dev/null
+++ b/memory-layer/src/test/scala/ix/memory/ingestion/GenericTextFallbackSpec.scala
@@ -0,0 +1,17 @@
+package ix.memory.ingestion
+
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class GenericTextFallbackSpec extends AnyFlatSpec with Matchers {
+
+ "GenericTextFallback" should "store compact metadata instead of raw content" in {
+ val parsed = GenericTextFallback.parse("notes.txt", "first line\nsecond line\n")
+ val entity = parsed.entities.head
+
+ entity.attrs.keySet should contain allOf ("file_name", "line_count", "preview")
+ entity.attrs.keySet should not contain "content"
+ entity.attrs("preview").asString shouldBe Some("first line")
+ entity.contentFingerprint should not be empty
+ }
+}
diff --git a/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala
new file mode 100644
index 00000000..9d88ee03
--- /dev/null
+++ b/memory-layer/src/test/scala/ix/memory/map/MapServiceScopeSpec.scala
@@ -0,0 +1,84 @@
+package ix.memory.map
+
+import java.util.UUID
+
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+import ix.memory.model.NodeId
+
+class MapServiceScopeSpec extends AnyFlatSpec with Matchers {
+
+ private val service = new MapService(null, null, null)
+
+ "resolveRegion" should "order ambiguous matches by match quality, level preference, and significance" in {
+ val context = region("Context", "system", 3, files = 120)
+ val apiSubsystem = region("Api", "subsystem", 2, files = 35, parent = Some(context))
+ val apiModule = region("Api", "module", 1, files = 12, parent = Some(context))
+ val apiClient = region("Api / Client", "system", 3, files = 90)
+
+ val map = ArchitectureMap(
+ regions = Vector(context, apiSubsystem, apiModule, apiClient),
+ fileCount = 257,
+ mapRev = 42L
+ )
+
+ val matches = service.resolveRegion(map, "api")
+
+ matches.map(_.region.label) shouldBe Vector("Api", "Api", "Api / Client")
+ matches.map(_.region.labelKind) shouldBe Vector("subsystem", "module", "system")
+ }
+
+ "scopedView" should "summarize subtree health while returning only direct children" in {
+ val system = region("Cli", "system", 3, files = 140, confidence = 0.82)
+ val subsystem = region("Cli / Client", "subsystem", 2, files = 105, confidence = 0.72, parent = Some(system))
+ val moduleA = region("Cli / Client", "module", 1, files = 53, confidence = 0.91, parent = Some(subsystem))
+ val moduleB = region("Queries", "module", 1, files = 10, confidence = 0.42, crosscut = 0.18, parent = Some(subsystem))
+ val moduleC = region("Github", "module", 1, files = 22, confidence = 0.63, parent = Some(subsystem))
+
+ val map = ArchitectureMap(
+ regions = Vector(system, subsystem, moduleA, moduleB, moduleC),
+ fileCount = 140,
+ mapRev = 42L
+ )
+
+ val view = service.scopedView(map, subsystem)
+
+ view.parent.map(_.label) shouldBe Some("Cli")
+ view.children.map(_.label) shouldBe Vector("Cli / Client", "Github", "Queries")
+ view.summary.wellDefined shouldBe 1
+ view.summary.moderate shouldBe 2
+ view.summary.fuzzy shouldBe 1
+ view.summary.crossCutting shouldBe 1
+ }
+
+ private def region(
+ label: String,
+ labelKind: String,
+ level: Int,
+ files: Int,
+ confidence: Double = 0.80,
+ crosscut: Double = 0.0,
+ parent: Option[Region] = None,
+ ): Region = {
+ val id = NodeId(UUID.randomUUID())
+ val memberFiles = (1 to files).map(_ => NodeId(UUID.randomUUID())).toSet
+ Region(
+ id = id,
+ label = label,
+ labelKind = labelKind,
+ level = level,
+ memberFiles = memberFiles,
+ childRegionIds = Set.empty,
+ parentId = parent.map(_.id),
+ cohesion = 0.5,
+ externalCoupling = 0.2,
+ boundaryRatio = 1.5,
+ confidence = confidence,
+ crosscutScore = crosscut,
+ dominantSignals = List("calls", "imports"),
+ interfaceNodeCount = 1,
+ mapRev = 42L
+ )
+ }
+}
diff --git a/node_modules/.vite/vitest/results.json b/node_modules/.vite/vitest/results.json
new file mode 100644
index 00000000..e075ed9c
--- /dev/null
+++ b/node_modules/.vite/vitest/results.json
@@ -0,0 +1 @@
+{"version":"1.6.1","results":[[":ix-cli/src/cli/__tests__/subsystem-explain.test.ts",{"duration":18,"failed":false}]]}
\ No newline at end of file
diff --git a/rebuild.sh b/rebuild.sh
new file mode 100755
index 00000000..00715c16
--- /dev/null
+++ b/rebuild.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$SCRIPT_DIR"
+
+echo "==> Shutting down..."
+./shutdown.sh
+
+echo "==> Building backend (sbt assembly)..."
+sbt assembly
+
+echo "==> Building CLI (npm run build)..."
+cd ix-cli
+npm run build
+cd ..
+
+echo "==> Running setup..."
+./setup.sh
+
+echo "==> Done."
diff --git a/skills/ix-api-contract-change/SKILL.md b/skills/ix-api-contract-change/SKILL.md
new file mode 100644
index 00000000..ecefb9c6
--- /dev/null
+++ b/skills/ix-api-contract-change/SKILL.md
@@ -0,0 +1,56 @@
+---
+name: ix-api-contract-change
+description: Use when adding or changing memory-layer routes or ix-cli client methods so API contracts stay aligned across Scala routes, models, and TypeScript IxClient calls.
+---
+
+# Ix API Contract Change
+
+Use this when touching:
+- `memory-layer/src/main/scala/ix/memory/api/*Routes.scala`
+- `memory-layer/src/main/scala/ix/memory/model/*`
+- `ix-cli/src/client/api.ts`
+
+## Workflow
+
+1. Map blast radius:
+```bash
+ix impact IxClient --format json
+ix search Routes --kind class --format json --limit 30
+```
+
+2. Inspect changed route surface:
+```bash
+ix overview <RouteClassName> --format json
+ix read <absolute/path/to/RouteFile.scala:1-220> --format json
+```
+
+3. Align client boundary:
+```bash
+ix overview IxClient --format json
+ix read <absolute/path/to/ix-cli/src/client/api.ts:1-260> --format json
+```
+
+4. Check for shared request/response model impact:
+```bash
+ix search <ModelName> --kind class --format json --limit 20
+ix impact <ModelName> --format json
+```
+
+5. Validate expected tests were considered:
+```bash
+ix search RoutesSpec --kind class --format json --include-tests --limit 20
+```
+
+## Contract Checklist
+
+- Route path and HTTP verb match client usage.
+- Request fields, optionality, and defaults are consistent.
+- Response shape and error behavior are consistent.
+- New route is wired into aggregate route registration.
+
+## Guardrails
+
+- Treat `IxClient` as a high-risk boundary.
+- If you changed a route signature, update caller methods in `IxClient` in the same task.
+- Do not leave route/client drift for a follow-up task unless explicitly planned.
+
diff --git a/skills/ix-api-contract-change/agents/openai.yaml b/skills/ix-api-contract-change/agents/openai.yaml
new file mode 100644
index 00000000..4aa140a5
--- /dev/null
+++ b/skills/ix-api-contract-change/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX API Contract Change"
+ short_description: "Keep Scala routes and IxClient in lockstep"
+ default_prompt: "Use $ix-api-contract-change to implement this endpoint/client change and verify contract alignment."
+
+policy:
+ allow_implicit_invocation: true
diff --git a/skills/ix-arango-query-change/SKILL.md b/skills/ix-arango-query-change/SKILL.md
new file mode 100644
index 00000000..16103a44
--- /dev/null
+++ b/skills/ix-arango-query-change/SKILL.md
@@ -0,0 +1,47 @@
+---
+name: ix-arango-query-change
+description: Use when modifying Arango-backed query/write behavior so schema, AQL, transaction semantics, and DB-facing tests remain aligned.
+---
+
+# Ix Arango Query Change
+
+Use this when touching:
+- `memory-layer/src/main/scala/ix/memory/db/ArangoClient.scala`
+- DB query/write APIs and AQL construction
+- schema or transaction behavior
+
+## Workflow
+
+1. Inspect DB surface:
+```bash
+ix overview ArangoClient --format json
+ix impact ArangoClient --format json
+```
+
+2. Read query/transaction hot paths:
+```bash
+ix read <absolute/path/to/ArangoClient.scala:1-320> --format json
+```
+
+3. Locate nearby DB behavior and specs:
+```bash
+ix search GraphQueryApiSpec --kind class --format json --include-tests --limit 20
+ix search GraphWriteApiSpec --kind class --format json --include-tests --limit 20
+ix search SearchAqlSpec --kind class --format json --include-tests --limit 20
+ix search ArangoClientSpec --kind class --format json --include-tests --limit 20
+```
+
+## DB Checklist
+
+- Query semantics match intended node/edge filters.
+- Sort/limit/pagination behavior is explicit and stable.
+- Transactions remain atomic for multi-step writes.
+- Schema bootstrap/ensure behavior is still safe and idempotent.
+- Error mapping preserves actionable failure details.
+
+## Guardrails
+
+- Do not change query semantics without corresponding test updates.
+- Treat default limits and sort order changes as behavior changes, not refactors.
+- Keep API-level callers in sync when return shapes or cardinality change.
+
diff --git a/skills/ix-arango-query-change/agents/openai.yaml b/skills/ix-arango-query-change/agents/openai.yaml
new file mode 100644
index 00000000..6cd29fe2
--- /dev/null
+++ b/skills/ix-arango-query-change/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX Arango Query Change"
+ short_description: "Control AQL/schema/transaction behavior drift"
+ default_prompt: "Use $ix-arango-query-change to update Arango query logic and keep DB semantics and tests aligned."
+
+policy:
+ allow_implicit_invocation: true
diff --git a/skills/ix-ingestion-parser-change/SKILL.md b/skills/ix-ingestion-parser-change/SKILL.md
new file mode 100644
index 00000000..9344d320
--- /dev/null
+++ b/skills/ix-ingestion-parser-change/SKILL.md
@@ -0,0 +1,58 @@
+---
+name: ix-ingestion-parser-change
+description: Use when changing ingestion or parser behavior so parser routing, extracted entities/relationships, and graph patch generation stay coherent and testable.
+---
+
+# Ix Ingestion Parser Change
+
+Use this when touching:
+- `memory-layer/src/main/scala/ix/memory/ingestion/IngestionService.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/ParserRouter.scala`
+- `memory-layer/src/main/scala/ix/memory/ingestion/parsers/*`
+- `memory-layer/src/main/scala/ix/memory/ingestion/GraphPatchBuilder.scala`
+
+## Workflow
+
+1. Confirm impact:
+```bash
+ix impact IngestionService --format json
+ix overview ParserRouter --format json
+ix overview GraphPatchBuilder --format json
+```
+
+2. Locate parser family and routing points:
+```bash
+ix search Parser --kind class --format json --limit 30
+ix read <absolute/path/to/ParserRouter.scala:1-260> --format json
+```
+
+3. Verify parse output contract:
+```bash
+ix overview ParseResult --format json
+ix read <absolute/path/to/ParseResult.scala:1-220> --format json
+```
+
+4. Verify graph patch contract:
+```bash
+ix read <absolute/path/to/GraphPatchBuilder.scala:1-260> --format json
+```
+
+5. Validate parser and ingestion tests were considered:
+```bash
+ix search Spec --kind class --format json --include-tests --limit 60
+```
+
+## Change Checklist
+
+- Parser selection logic and file-type routing stay deterministic.
+- New extracted entity/relationship kinds are represented consistently.
+- IDs/hashes remain stable where expected.
+- Import-edge and named-edge resolution still converges on resolvable nodes.
+- Graph patch operations still satisfy downstream commit assumptions.
+
+## Guardrails
+
+- Avoid parser-only changes that silently break patch building.
+- If parser output shape changes, update parse result consumers in the same task.
+- Record intended behavior shifts as an explicit decision via `ix decide`.
+
diff --git a/skills/ix-ingestion-parser-change/agents/openai.yaml b/skills/ix-ingestion-parser-change/agents/openai.yaml
new file mode 100644
index 00000000..1228260b
--- /dev/null
+++ b/skills/ix-ingestion-parser-change/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX Ingestion Parser Change"
+ short_description: "Guard parser routing and patch generation"
+ default_prompt: "Use $ix-ingestion-parser-change to make this ingestion/parser update without breaking graph patch behavior."
+
+policy:
+ allow_implicit_invocation: true
diff --git a/skills/ix-map-pipeline-change/SKILL.md b/skills/ix-map-pipeline-change/SKILL.md
new file mode 100644
index 00000000..fa455cf8
--- /dev/null
+++ b/skills/ix-map-pipeline-change/SKILL.md
@@ -0,0 +1,81 @@
+---
+name: ix-map-pipeline-change
+description: Use when changing the architectural analysis pipeline — MapService, SmellService, SubsystemScoringService, or their routes/CLI commands — so the map → smells → subsystems chain stays coherent.
+---
+
+# Ix Map Pipeline Change
+
+Use this when touching:
+- `memory-layer/src/main/scala/ix/memory/map/MapService.scala`
+- `memory-layer/src/main/scala/ix/memory/map/MapGraphBuilderFast.scala`
+- `memory-layer/src/main/scala/ix/memory/map/MapPreflight.scala`
+- `memory-layer/src/main/scala/ix/memory/map/LouvainClustering.scala`
+- `memory-layer/src/main/scala/ix/memory/smell/SmellService.scala`
+- `memory-layer/src/main/scala/ix/memory/subsystem/SubsystemScoringService.scala`
+- `ix-cli/src/cli/commands/map.ts`
+- `ix-cli/src/cli/commands/smells.ts`
+- `ix-cli/src/cli/commands/subsystems.ts`
+
+## Pipeline Overview
+
+```
+ix map → MapService (Louvain clustering → region hierarchy stored as REGION nodes)
+ix smells → SmellService (AQL graph queries → smell claims on File nodes)
+ix subsystems → SubsystemScoringService (join regions + smell claims → health scores)
+```
+
+Each stage stores versioned claims. Re-running retires old claims before writing new ones.
+
+## Workflow
+
+1. Understand current pipeline state:
+```bash
+ix briefing --format json
+ix overview MapService --format json
+ix impact MapService --format json
+```
+
+2. Check smell and subsystem dependencies:
+```bash
+ix overview SmellService --format json
+ix overview SubsystemScoringService --format json
+ix callers MapService --format json
+```
+
+3. Inspect the map types contract (shared across all three stages):
+```bash
+ix read memory-layer/src/main/scala/ix/memory/map/MapTypes.scala
+```
+
+4. After changes, verify all three stages still chain:
+```bash
+ix map --format json | head -20
+ix smells --format json | head -20
+ix subsystems --format json | head -20
+```
+
+5. Check region/claim data in graph:
+```bash
+ix stats --format json
+ix smells --list --format json
+ix subsystems --list --format json
+```
+
+6. Record the decision:
+```bash
+ix decide "<change title>" --rationale "<why>" --affects MapService --format json
+```
+
+## Key Invariants
+
+- `MapService` stores results as `REGION` nodes and `CONTAINS_FILE` / `CONTAINS_REGION` edges. SmellService and SubsystemScoringService query these — don't change the node/edge kinds without updating all three.
+- Smell claims use field prefix `has_smell.*` and `inference_version=smell_v1`. Subsystem scores use `field=subsystem_health` and `inference_version=subsystem_v1`.
+- Both services call `retireOld*()` before storing new claims — this is idempotent by design.
+- `SubsystemScoringService(client: ArangoClient, writeApi: GraphWriteApi)` — uses `writeApi` not `queryApi`.
+- `SmellService(client: ArangoClient, writeApi: GraphWriteApi)` — uses `writeApi` not `queryApi`.
+
+## Guardrails
+
+- Run `ix map` before `ix smells` or `ix subsystems` — they depend on region data being current.
+- If you change region granularity in MapService, SubsystemScoringService scores will shift — document the expected delta as a decision.
+- Do not change claim field names (`has_smell.*`, `subsystem_health`) without updating the CLI commands that parse them.
diff --git a/skills/ix-new-command/SKILL.md b/skills/ix-new-command/SKILL.md
new file mode 100644
index 00000000..015cd31b
--- /dev/null
+++ b/skills/ix-new-command/SKILL.md
@@ -0,0 +1,89 @@
+---
+name: ix-new-command
+description: Use when adding a new ix CLI command end-to-end — Scala route, IxClient method, CLI command file, oss.ts registration, and doc updates must all stay in sync.
+---
+
+# Ix New Command
+
+Use this when adding a new `ix <command>` from scratch or extending an existing command with a new subcommand.
+
+## Files that must all change together
+
+| Layer | File |
+|-------|------|
+| Backend route | `memory-layer/src/main/scala/ix/memory/api/<Name>Routes.scala` |
+| Route wiring | `memory-layer/src/main/scala/ix/memory/api/Routes.scala` |
+| Server wiring | `memory-layer/src/main/scala/ix/memory/Main.scala` |
+| TypeScript client | `ix-cli/src/client/api.ts` |
+| CLI command | `ix-cli/src/cli/commands/<name>.ts` |
+| OSS registration | `ix-cli/src/cli/register/oss.ts` |
+| Docs | `CLAUDE.md`, `AGENTS.md` |
+
+## Workflow
+
+1. Understand current command surface:
+```bash
+ix overview IxClient --format json
+ix search Routes --kind class --format json --limit 30
+ix impact Routes --format json
+```
+
+2. Read existing route and client as pattern references:
+```bash
+ix read memory-layer/src/main/scala/ix/memory/api/SmellRoutes.scala
+ix read ix-cli/src/cli/commands/smells.ts
+```
+
+3. After writing the new route file, read constructor patterns to wire correctly:
+```bash
+ix search <ServiceName> --kind class --format json --limit 5
+ix read memory-layer/src/main/scala/ix/memory/api/Routes.scala
+ix read memory-layer/src/main/scala/ix/memory/Main.scala
+```
+
+4. After writing the CLI command, verify registration isn't duplicated:
+```bash
+ix read ix-cli/src/cli/register/oss.ts
+```
+
+5. Compile-check the backend:
+```bash
+sbt "memoryLayer/compile" 2>&1 | grep -E "error|warning|success"
+```
+
+6. Build the CLI:
+```bash
+cd ix-cli && npm run build 2>&1 | grep -E "error TS|error:"
+```
+
+7. Smoke-test the command:
+```bash
+ix <newcommand> --format json
+```
+
+8. Update docs and ingest:
+```bash
+# Edit CLAUDE.md and AGENTS.md to add the command to the appropriate table
+ix ingest ./ix-cli/src ./memory-layer/src
+```
+
+9. Record the decision:
+```bash
+ix decide "Add ix <name> command" --rationale "<why this command exists>" --affects IxClient --format json
+```
+
+## Common Wiring Errors
+
+- `Routes.scala`: wrong constructor args — always read the `class <Name>Routes(...)` signature before instantiating
+- `Routes.scala`: referencing a class from the wrong package — check import path matches where the service lives
+- `oss.ts`: registering the same command twice — grep for the register call before adding it
+- `api.ts`: method returns `Record<string,string>` but caller expects `Map` — prefer `Map` and use `new Map(Object.entries(result))`
+- `api.ts`: method signature accepts no args but CLI passes options — always add `opts?: {...}` if the command has flags
+
+## Done Criteria
+
+- `sbt "memoryLayer/compile"` passes with no errors.
+- `cd ix-cli && npm run build` passes with no errors.
+- `ix <newcommand> --format json` returns a valid JSON response.
+- Command appears in `CLAUDE.md` and `AGENTS.md`.
+- No duplicate registration in `oss.ts`.
diff --git a/skills/ix-pro-command-extension/SKILL.md b/skills/ix-pro-command-extension/SKILL.md
new file mode 100644
index 00000000..23a9ce67
--- /dev/null
+++ b/skills/ix-pro-command-extension/SKILL.md
@@ -0,0 +1,56 @@
+---
+name: ix-pro-command-extension
+description: Use when adding or modifying pro CLI commands to keep registration, dynamic loading, patch-building semantics, and command safety coherent across ix-cli and Ix-pro.
+---
+
+# Ix Pro Command Extension
+
+Use this when changing pro command implementation or wiring.
+
+## Scope
+
+- `Ix-pro/src/cli/register.ts`
+- `Ix-pro/src/cli/commands/*`
+- `IX-Memory/ix-cli/src/cli/register/pro-loader.ts`
+- Any shared client patch payload behavior (`GraphPatchPayload`, `PatchOp`)
+
+## Workflow
+
+1. Confirm command wiring:
+```bash
+ix read /home/ianhock/ix/Ix-pro/src/cli/register.ts:1-260 --format json
+ix read /home/ianhock/ix/IX-Memory/ix-cli/src/cli/register/pro-loader.ts:1-260 --format json
+```
+
+2. Inspect changed command module:
+```bash
+ix read /home/ianhock/ix/Ix-pro/src/cli/commands/<command>.ts:1-320 --format json
+ix overview IxClient --format json
+```
+
+3. Validate patch intent and edge semantics:
+```bash
+ix search build<Command>Patch --kind function --format json --limit 20
+ix impact IxClient --format json
+```
+
+4. Verify discoverability and operator UX:
+```bash
+ix search registerProCommands --kind function --format json --limit 10
+ix search tryLoadProCommands --kind function --format json --limit 10
+```
+
+## Checklist
+
+- New command is registered in `registerProCommands`.
+- Loader compatibility with `@ix/pro/register` is preserved.
+- `--format json` output remains stable for automation.
+- Patch ops (`UpsertNode`, `UpsertEdge`, `AssertClaim`) reflect intended graph semantics.
+- Validation blocks unsafe command input where applicable (example: workflow command safety checks).
+
+## Guardrails
+
+- Treat command output schemas as APIs.
+- Do not add command behavior that bypasses entity resolution safeguards.
+- Keep edge predicates explicit and consistent with existing pro model.
+
diff --git a/skills/ix-pro-command-extension/agents/openai.yaml b/skills/ix-pro-command-extension/agents/openai.yaml
new file mode 100644
index 00000000..deb0c36b
--- /dev/null
+++ b/skills/ix-pro-command-extension/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX Pro Command Extension"
+ short_description: "Extend pro CLI safely across loader and commands"
+ default_prompt: "Use $ix-pro-command-extension to add or refactor a pro CLI command without breaking registration or patch semantics."
+
+policy:
+ allow_implicit_invocation: true
diff --git a/skills/ix-pro-delivery-loop/SKILL.md b/skills/ix-pro-delivery-loop/SKILL.md
new file mode 100644
index 00000000..89ad9651
--- /dev/null
+++ b/skills/ix-pro-delivery-loop/SKILL.md
@@ -0,0 +1,63 @@
+---
+name: ix-pro-delivery-loop
+description: Use when executing work with pro CLI objects (goals, plans, tasks, bugs, decisions, workflows) to keep delivery state explicit and machine-queryable.
+---
+
+# Ix Pro Delivery Loop
+
+Use this when a task should be tracked through pro objects, not ad-hoc notes.
+
+## Trigger
+
+Use when at least one applies:
+- New work should be sequenced into tasks.
+- A bug should be linked to affected entities and a responding plan.
+- A repeatable staged workflow should be attached and run.
+
+## Core Loop
+
+1. Resume and inspect:
+```bash
+ix briefing --format json
+ix tasks --status pending --format json
+ix bugs --format json
+```
+
+2. Create or select plan context:
+```bash
+ix goal create "<goal>" --format json
+ix plan create "<plan>" --goal <goal-id> --format json
+ix plan next <plan-id> --with-workflow --format json
+```
+
+3. Add executable tasks:
+```bash
+ix plan task "<title>" --plan <plan-id> --workflow-staged '{"discover":["ix overview <x> --format json"],"implement":["ix impact <x> --format json"],"validate":["ix tasks --plan <plan-id> --format json"]}' --format json
+```
+
+4. Track defects and linkage:
+```bash
+ix bug create "<title>" --severity high --affects <EntityName> --format json
+ix plan create "<plan>" --goal <goal-id> --responds-to <bug-id> --format json
+ix plan task "<title>" --plan <plan-id> --resolves <bug-id> --format json
+```
+
+5. Attach and run workflows:
+```bash
+ix workflow attach task <task-id> --file <workflow.json> --format json
+ix workflow validate task <task-id> --format json
+ix workflow run task <task-id> --stage discover --format json
+```
+
+6. Close loop:
+```bash
+ix task update <task-id> --status done --format json
+ix bug update <bug-id> --status resolved --format json
+```
+
+## Guardrails
+
+- Workflows must contain `ix` commands only; no shell operators.
+- Keep dependency edges explicit with `--depends-on`.
+- Prefer linking bugs/plans/tasks (`RESPONDS_TO`, `TASK_RESOLVES_BUG`) in the same session.
+
diff --git a/skills/ix-pro-delivery-loop/agents/openai.yaml b/skills/ix-pro-delivery-loop/agents/openai.yaml
new file mode 100644
index 00000000..f32d98bd
--- /dev/null
+++ b/skills/ix-pro-delivery-loop/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX Pro Delivery Loop"
+ short_description: "Operate goals/plans/tasks/bugs/workflows together"
+ default_prompt: "Use $ix-pro-delivery-loop to drive this work through plan/task/bug/workflow lifecycle with linked entities."
+
+policy:
+ allow_implicit_invocation: true
diff --git a/skills/ix-rebuild/SKILL.md b/skills/ix-rebuild/SKILL.md
new file mode 100644
index 00000000..c9d237ac
--- /dev/null
+++ b/skills/ix-rebuild/SKILL.md
@@ -0,0 +1,111 @@
+---
+name: ix-rebuild
+description: Use when running or recovering from a full ix rebuild — sbt assembly + CLI build. Covers how to interpret compile errors and common failure patterns in this codebase.
+---
+
+# Ix Rebuild
+
+Use this when running `./rebuild.sh` or fixing compile errors after merging changes.
+
+## Build Order
+
+```
+sbt assembly (Scala backend) → npm run build (TypeScript CLI) → Docker restart
+```
+
+`./rebuild.sh` runs all three. For faster iteration use the individual steps below.
+
+## Fast Iteration Commands
+
+```bash
+# Scala only (fastest feedback on backend changes)
+sbt "memoryLayer/compile" 2>&1 | grep -E "error|warning|success"
+
+# Full Scala assembly (needed before Docker restart)
+sbt assembly 2>&1 | grep -E "error|warning|success|Built"
+
+# TypeScript CLI only
+cd ix-cli && npm run build 2>&1 | grep -E "error TS|error:"
+
+# Both + Docker restart
+./rebuild.sh
+```
+
+## Common Scala Errors and Fixes
+
+### Wrong constructor args
+```
+not enough arguments for constructor Foo: (a: A, b: B)
+```
+→ Read the class signature: `ix search Foo --kind class --format json --limit 5`
+→ Then: `ix read <path-to-file>` to see the actual constructor.
+
+### Type not found
+```
+not found: type BulkPatchCommitRoutes
+```
+→ The class doesn't exist. Search for it: `ix locate BulkPatchCommitRoutes --limit 5`
+→ Check if it was renamed or removed from the route aggregator.
+
+### Type mismatch on service
+```
+found: ix.memory.smell.SubsystemService
+required: ix.memory.subsystem.SubsystemScoringService
+```
+→ The stash/upstream used different packages. Check which class actually exists on disk:
+ `find memory-layer/src -name "Subsystem*.scala"`
+→ Update the import and usage to match what exists.
+
+### Wrong queryApi vs writeApi
+```
+found: ArangoGraphQueryApi
+required: GraphWriteApi
+```
+→ Check the service constructor: `grep "^class <Name>" memory-layer/src/main/scala/**/*.scala`
+→ Services that store claims need `writeApi`; read-only services need `queryApi`.
+
+## Common TypeScript Errors and Fixes
+
+### Missing method on IxClient
+```
+Property 'commitPatchBulk' does not exist on type 'IxClient'
+```
+→ Add the method to `ix-cli/src/client/api.ts`.
+→ Check what endpoint it should call by reading the corresponding Scala route.
+
+### Missing export from module
+```
+Module '"../errors.js"' has no exported member 'parseBackendError'
+```
+→ Read `ix-cli/src/cli/errors.ts` to see what's actually exported.
+→ Remove the import and inline the logic, or add the export if it belongs there.
+
+### Wrong argument count
+```
+Expected 0 arguments, but got 1
+```
+→ The CLI passes options the api.ts method doesn't accept yet.
+→ Add `opts?: { ... }` to the method signature in `api.ts`.
+
+### Duplicate command registration
+```
+Error: cannot add command 'smells' as already have command 'smells'
+```
+→ Open `ix-cli/src/cli/register/oss.ts` and remove the duplicate `register<Name>Command(program)` call.
+
+## Smoke Test After Rebuild
+
+```bash
+ix status # backend alive
+ix stats --format json # graph accessible
+ix map --format json | head -5 # map pipeline works
+ix smells --format json | head -5 # smell pipeline works
+ix subsystems --format json | head -5 # subsystem pipeline works
+```
+
+## Done Criteria
+
+- `sbt "memoryLayer/compile"` exits with `[success]`.
+- `cd ix-cli && npm run build` exits cleanly (no `error TS` lines).
+- `./rebuild.sh` prints `IX backend is ready!`.
+- `ix status` reports `ok`.
diff --git a/skills/ix-session-memory/SKILL.md b/skills/ix-session-memory/SKILL.md
new file mode 100644
index 00000000..9da6c4f6
--- /dev/null
+++ b/skills/ix-session-memory/SKILL.md
@@ -0,0 +1,69 @@
+---
+name: ix-session-memory
+description: Use this for any non-trivial IX-Memory task to keep context synchronized in ix (briefing, impact, planning, decisions, and bugs) and to produce resumable work.
+---
+
+# Ix Session Memory
+
+Use this skill at the start of multi-step work and keep using it until the task is done.
+
+## Trigger
+
+Use when the task has at least one of:
+- More than one file or subsystem
+- A risky change with dependency impact
+- Work that should be resumable by another agent later
+
+## Workflow
+
+1. Resume context:
+```bash
+ix briefing --format json
+```
+
+2. Scope impact:
+```bash
+ix overview <target> --format json
+ix impact <target> --format json
+```
+
+3. If no active goal/plan exists, create one:
+```bash
+ix goal create "<outcome statement>" --format json
+ix plan create "<plan title>" --goal <goal-id> --format json
+ix plan task "<first task>" --plan <plan-id> --format json
+```
+
+4. Pull next actionable work:
+```bash
+ix plan next <plan-id> --with-workflow --format json
+```
+
+5. Track major reasoning:
+```bash
+ix decide "<decision>" --rationale "<why>" --affects <EntityName> --format json
+```
+
+6. Track defects explicitly:
+```bash
+ix bug create "<bug title>" --affects <EntityName> --format json
+```
+
+7. Mark completed tasks:
+```bash
+ix task update <task-id> --status done --format json
+```
+
+## Guardrails
+
+- Always use `--format json` for any command that is parsed or chained.
+- Prefer `overview` and `impact` before low-level traversal.
+- If resolution is ambiguous, rerun with `--kind`.
+- Never follow unresolved `rawId`; use `ix text` or `ix read` instead.
+
+## Done Criteria
+
+- The change is represented in at least one plan/task update.
+- Any architectural tradeoff is captured as a decision.
+- Any discovered defect is captured as a bug or fixed in-task.
+
diff --git a/skills/ix-session-memory/agents/openai.yaml b/skills/ix-session-memory/agents/openai.yaml
new file mode 100644
index 00000000..5f14d114
--- /dev/null
+++ b/skills/ix-session-memory/agents/openai.yaml
@@ -0,0 +1,7 @@
+interface:
+ display_name: "IX Session Memory"
+ short_description: "Keep ix state synced across multi-step work"
+ default_prompt: "Use $ix-session-memory to resume context, scope impact, and track this task in goals/plans/decisions."
+
+policy:
+ allow_implicit_invocation: true