Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

fix: index initialized git submodules

yescineded2783eb292
diff --git a/README.md b/README.md
index eff821fa..60f4fa1e 100644
--- a/README.md
+++ b/README.md
@@ -215,7 +215,7 @@ flowchart LR
Every pass is cached by content hash — the LLM ones and the tree-sitter parse alike. Re-running only touches the files that changed, so the second build is fast and cheap (on this repo, 124 files: 0.74s cold, 0.18s after one edited file, 0.18s with nothing changed). `graft build --no-reuse` forces a cold re-parse.
-That cheapness is what lets **every query refresh the graph before it answers**. A retrieval call stats the tree against the last build's fingerprint (~3ms), and rebuilds only if something moved — so `ask`/`grep`/`callers`/`skeleton`/`map` describe the code as it is right now, including edits that are unsaved to git: uncommitted, unstaged, or staged all look the same to graft, which never reads git at all. The refresh is structural and `$0`; it never calls the LLM. Turn it off per-command with `--no-refresh`, or everywhere with `GRAFT_NO_REFRESH=1`.
+That cheapness is what lets **every query refresh the graph before it answers**. A retrieval call stats the tree against the last build's fingerprint (~3ms), and rebuilds only if something moved — so `ask`/`grep`/`callers`/`skeleton`/`map` describe the code as it is right now, including edits that are unsaved to git: uncommitted, unstaged, or staged all look the same to graft. Git determines the visible file set; freshness compares the working-tree bytes rather than commit or index state. The refresh is structural and `$0`; it never calls the LLM. Turn it off per-command with `--no-refresh`, or everywhere with `GRAFT_NO_REFRESH=1`.
Alongside the markdown graph, `graft build` builds `graft/.graph/wiring.json` — a per-symbol code graph — plus a per-file wiring card mirroring your source tree. Tier 1 is pure tree-sitter (every function, class, and call edge; deterministic, no model, no network), which is why plain `graft build` needs no key. The `--deep` pass adds a one-line summary and a crux excerpt per symbol, cached by body hash.
@@ -419,9 +419,9 @@ scripts/ 2 files · 0 symbols
hotspots: contextDirFor · function · src/context/node-file.ts:L100-L103 · 21← wiringPath · function · src/graph/write.ts:L20-L22 · 14← buildGraph · function · src/graph/build.ts:L104-L218 · 11← ...
```
-## Monorepos & multi-repo folders
+## Monorepos, submodules & multi-repo folders
-Graft handles two shapes without any config:
+Graft handles these layouts without any config:
- **A monorepo with one `.git`** (a `pnpm-workspace.yaml`/`package.json`
`workspaces`, or per-package `go.mod`/`pyproject.toml`/`Cargo.toml`) —
@@ -429,13 +429,18 @@ Graft handles two shapes without any config:
rank every scope on its own terms and fuse the results, so the biggest
sub-project can't drown a small one; hits carry `[scope/]` labels, and
`graft map` groups its directory clusters by scope first.
+- **A Git superproject with initialized submodules** — `graft build` follows
+ initialized gitlinks into one graph, prefixing child paths (for example,
+ `deps/parser/src/index.ts`) while honoring each submodule's own Git ignore
+ rules. Visible untracked files are included too; uninitialized submodules
+ remain absent until `git submodule update --init` checks them out.
- **A folder of separate git repos** (no `.git` at the top) — `graft build`
auto-splits: each child gets its own (git-ignored) `graft/`, and the parent
gets a `graft/workspace.json` index. Queries from the parent federate across
every child, always labeled `<child>/`. Run `graft build` inside a child to
work on just that repo.
-Either way, narrow to one sub-project with `graft ask "<task>" --in <scope>/`
+In every layout, narrow to one sub-project with `graft ask "<task>" --in <scope>/`
once you know where you're working.
## Visualize it (`graft viz`)
diff --git a/src/ingest/fs.ts b/src/ingest/fs.ts
index 3d072e2c..040b1f83 100644
--- a/src/ingest/fs.ts
+++ b/src/ingest/fs.ts
@@ -2,8 +2,8 @@
* Filesystem walking used by `init`/`check` to enumerate a repo's source files.
*/
import { spawnSync } from "node:child_process";
-import { lstatSync, readdirSync, statSync } from "node:fs";
-import { join, resolve } from "node:path";
+import { existsSync, lstatSync, readdirSync, statSync } from "node:fs";
+import { join, relative, resolve } from "node:path";
/** Directories that are dependency/build output, never source. */
export const SKIP_DIRS = new Set([
@@ -53,7 +53,10 @@ export function shouldSkipDir(name: string, includes?: ReadonlySet<string>): boo
* `includes`, and files over 1 MB.
* In a Git worktree, tracked files plus untracked, non-ignored files come from
* `git ls-files`; this gives indexing exactly Git's nested `.gitignore`,
- * negation, and global-exclude semantics. Non-Git directories retain the plain
+ * negation, and global-exclude semantics. Initialized submodules are enumerated
+ * recursively as repositories of their own, so their tracked and visible
+ * untracked files share one graph while keeping each child's ignore rules.
+ * Uninitialized submodules remain absent. Non-Git directories retain the plain
* filesystem walk. `includes` lifts only the built-in skip list — it never
* overrides Git's ignore rules (un-ignore or `git add -f` a directory to
* index it, the same contract as tracked-but-ignored files).
@@ -65,11 +68,23 @@ export function walkDir(dir: string, includes?: ReadonlySet<string>): string[] {
/** Git's canonical working-tree file set, relative to `dir`. Tracked files are
* deliberately included even when a later ignore rule matches them; `.gitignore`
* only controls untracked files in Git, and graft follows the same contract. */
-function gitVisibleFiles(dir: string, includes?: ReadonlySet<string>): string[] | null {
+function gitVisibleFiles(
+ dir: string,
+ includes?: ReadonlySet<string>,
+ traversal?: { topRoot: string; activeRoots: Set<string> },
+): string[] | null {
const root = resolve(dir);
+ const state = traversal ?? { topRoot: root, activeRoots: new Set<string>() };
+ const rootKey = process.platform === "win32" ? root.toLowerCase() : root;
+ if (state.activeRoots.has(rootKey)) return [];
+ state.activeRoots.add(rootKey);
+
+ // `-t --stage` makes the cached/other output unambiguous in one process:
+ // tracked entries are `H <mode> <oid> <stage>\t<path>`, while untracked
+ // entries are `? <path>`. `-z` keeps either form safe for unusual paths.
const result = spawnSync(
"git",
- ["ls-files", "--cached", "--others", "--exclude-standard", "-z", "--"],
+ ["ls-files", "-t", "--stage", "--cached", "--others", "--exclude-standard", "-z", "--"],
{
cwd: root,
encoding: "utf8",
@@ -77,12 +92,63 @@ function gitVisibleFiles(dir: string, includes?: ReadonlySet<string>): string[]
maxBuffer: 64 * 1024 * 1024,
},
);
- if (result.status !== 0 || result.error || typeof result.stdout !== "string") return null;
+ if (result.status !== 0 || result.error || typeof result.stdout !== "string") {
+ state.activeRoots.delete(rootKey);
+ return null;
+ }
- const out: string[] = [];
- for (const rel of result.stdout.split("\0")) {
- if (!rel || skippedPath(rel, includes)) continue;
+ // A path can have multiple stage records during a merge. Collapse those
+ // records and remember if any of them identifies the path as a gitlink.
+ const entries = new Map<string, { gitlink: boolean }>();
+ for (const record of result.stdout.split("\0")) {
+ if (!record || record.length < 2 || record[1] !== " ") continue;
+
+ const tag = record[0];
+ const body = record.slice(2);
+ let rel: string;
+ let gitlink = false;
+ if (tag === "?") {
+ rel = body;
+ } else {
+ const tab = body.indexOf("\t");
+ if (tab === -1) continue;
+ gitlink = body.startsWith("160000 ");
+ rel = body.slice(tab + 1);
+ }
+ if (!rel) continue;
+ const prior = entries.get(rel);
+ entries.set(rel, { gitlink: gitlink || prior?.gitlink === true });
+ }
+
+ const out = new Set<string>();
+ for (const [rel, entry] of entries) {
const abs = resolve(root, rel);
+ // Filter against the original superproject path. Otherwise a submodule
+ // mounted at vendor/ or build/ would bypass the parent's skip policy when
+ // recursion resets its relative root.
+ if (skippedPath(relative(state.topRoot, abs), includes)) continue;
+
+ if (entry.gitlink) {
+ // A deinitialized gitlink can resolve upward to the superproject when Git
+ // runs inside it. Requiring the child's own .git prevents duplicate,
+ // mis-prefixed parent files and recursive loops.
+ if (!existsSync(join(abs, ".git"))) continue;
+ let childFiles = gitVisibleFiles(abs, includes, state);
+ if (childFiles === null) {
+ // Do not let one broken child recreate a "healthy but incomplete"
+ // graph. Match the top-level fail-soft contract locally: fall back to
+ // the filesystem for this child only, preserving the parent repo's Git
+ // visibility and every built-in skip/size guard. Child Git ignore rules
+ // are unavailable in this exceptional path, just as they are when the
+ // top-level Git command fails and walkDir uses its filesystem fallback.
+ // Let an unreadable filesystem fallback surface rather than claiming a
+ // healthy graph that silently omitted the child again.
+ childFiles = walkFilesystem(abs, includes);
+ }
+ for (const file of childFiles) out.add(file);
+ continue;
+ }
+
try {
const stat = lstatSync(abs);
if (!stat.isFile() || stat.size > MAX_FILE_BYTES) continue;
@@ -91,9 +157,11 @@ function gitVisibleFiles(dir: string, includes?: ReadonlySet<string>): string[]
// `--cached`; absence means it is not part of the current source set.
continue;
}
- out.push(abs);
+ out.add(abs);
}
- return out;
+
+ state.activeRoots.delete(rootKey);
+ return [...out].sort();
}
/** A path (git-relative, either separator) is skipped when any of its
diff --git a/test/graph-incremental.test.ts b/test/graph-incremental.test.ts
index 1b63285e..b00a4269 100644
--- a/test/graph-incremental.test.ts
+++ b/test/graph-incremental.test.ts
@@ -38,6 +38,19 @@ function repo(): string {
return d;
}
+function gitRun(root: string, args: string[]): void {
+ execFileSync("git", args, { cwd: root, stdio: "ignore" });
+}
+
+function commitRepo(root: string, message: string): void {
+ gitRun(root, ["add", "-A"]);
+ gitRun(root, [
+ "-c", "user.name=Graft Tests",
+ "commit", "-qm", message,
+ ]);
+}
+
const outOf = (d: string): string => join(d, "graft");
const wiringOf = (d: string): string => readFileSync(wiringPath(outOf(d)), "utf8");
@@ -179,6 +192,66 @@ test("every file on disk lands in the fingerprint", async () => {
assert.deepEqual(Object.keys(fp!.files).sort(), ["src/app.ts", "src/math.ts"]);
});
+test("initialized submodule files share the graph and freshness fingerprint (#74)", async () => {
+ const parent = repo();
+ const child = mkdtempSync(join(tmpdir(), "graft-incr-submodule-"));
+ try {
+ execFileSync("git", ["init", "-q"], { cwd: parent });
+ commitRepo(parent, "parent fixture");
+
+ execFileSync("git", ["init", "-q"], { cwd: child });
+ mkdirSync(join(child, "src"), { recursive: true });
+ writeFileSync(join(child, ".gitignore"), "*.generated.ts\n");
+ writeFileSync(join(child, "src", "sub.ts"), "export function subValue(): number { return 1; }\n");
+ commitRepo(child, "child fixture");
+
+ gitRun(parent, [
+ "-c", "protocol.file.allow=always",
+ "submodule", "add", "-q", child.replace(/\\/g, "/"), "modules/child",
+ ]);
+ commitRepo(parent, "add child submodule");
+
+ const checkout = join(parent, "modules", "child", "src");
+ writeFileSync(join(checkout, "visible.ts"), "export const visible = 1;\n");
+ writeFileSync(join(checkout, "ignored.generated.ts"), "export const ignored = 1;\n");
+
+ const first = await buildGraph(parent);
+ assert.equal(first.files, 4);
+ const graph = readGraph(wiringPath(outOf(parent))) as GraphV1;
+ assert.ok(graph.nodes.some((n) => n.id === "modules/child/src/sub.ts#subValue"));
+ assert.ok(graph.nodes.some((n) => n.path === "modules/child/src/visible.ts"));
+ assert.ok(!graph.nodes.some((n) => n.path.endsWith("ignored.generated.ts")));
+ assert.deepEqual(Object.keys(readFingerprint(outOf(parent))!.files).sort(), [
+ "modules/child/src/sub.ts",
+ "modules/child/src/visible.ts",
+ "src/app.ts",
+ "src/math.ts",
+ ]);
+ assert.ok(isClean(probeDrift(parent, outOf(parent))!));
+
+ writeFileSync(join(checkout, "sub.ts"), "export function changedSubValue(): number { return 2; }\n");
+ writeFileSync(join(checkout, "added.ts"), "export const added = 1;\n");
+ rmSync(join(checkout, "visible.ts"));
+ writeFileSync(join(checkout, "another.generated.ts"), "export const stillIgnored = 1;\n");
+
+ assert.deepEqual(probeDrift(parent, outOf(parent)), {
+ changed: ["modules/child/src/sub.ts"],
+ added: ["modules/child/src/added.ts"],
+ removed: ["modules/child/src/visible.ts"],
+ });
+
+ await buildGraph(parent);
+ const refreshed = readGraph(wiringPath(outOf(parent))) as GraphV1;
+ assert.ok(refreshed.nodes.some((n) => n.id === "modules/child/src/sub.ts#changedSubValue"));
+ assert.ok(refreshed.nodes.some((n) => n.path === "modules/child/src/added.ts"));
+ assert.ok(!refreshed.nodes.some((n) => n.path === "modules/child/src/visible.ts"));
+ assert.ok(isClean(probeDrift(parent, outOf(parent))!));
+ } finally {
+ rmSync(parent, { recursive: true, force: true });
+ rmSync(child, { recursive: true, force: true });
+ }
+});
+
test("gitignored generated files stay out of the graph, fingerprint, and drift probe (#39)", async () => {
const d = repo();
try {
diff --git a/test/ingest-fs.test.ts b/test/ingest-fs.test.ts
index 928e8c0b..f3514cf6 100644
--- a/test/ingest-fs.test.ts
+++ b/test/ingest-fs.test.ts
@@ -1,7 +1,7 @@
import { execFileSync } from "node:child_process";
import { test } from "node:test";
import assert from "node:assert/strict";
-import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
+import { mkdtempSync, mkdirSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, relative } from "node:path";
import { shouldSkipDir, walkDir, SKIP_DIRS } from "../src/ingest/fs.js";
@@ -18,6 +18,27 @@ function write(root: string, path: string, content = "export const value = 1;\n"
writeFileSync(join(root, path), content);
}
+function runGit(root: string, args: string[]): void {
+ execFileSync("git", args, { cwd: root, stdio: "ignore" });
+}
+
+function commitAll(root: string, message: string, forcePaths: string[] = []): void {
+ runGit(root, ["add", "-A"]);
+ if (forcePaths.length > 0) runGit(root, ["add", "-f", "--", ...forcePaths]);
+ runGit(root, [
+ "-c", "user.name=Graft Tests",
+ "commit", "-qm", message,
+ ]);
+}
+
+function addLocalSubmodule(parent: string, source: string, path: string): void {
+ runGit(parent, [
+ "-c", "protocol.file.allow=always",
+ "submodule", "add", "-q", source.replace(/\\/g, "/"), path,
+ ]);
+}
+
function walked(root: string, includes?: ReadonlySet<string>): string[] {
return walkDir(root, includes)
.map((path) => relative(root, path).replace(/\\/g, "/"))
@@ -61,6 +82,78 @@ test("walkDir keeps tracked files that match an ignore rule and untracked visibl
}
});
+test("walkDir recursively includes initialized submodules with their own Git visibility (#74)", () => {
+ const parent = fixture("submodule-parent");
+ const child = fixture("submodule-child");
+ const nested = fixture("submodule-nested");
+ try {
+ write(nested, ".gitignore", "*.generated.ts\n");
+ write(nested, "src/nested.ts");
+ commitAll(nested, "nested fixture");
+
+ write(child, ".gitignore", "*.generated.ts\n");
+ write(child, "src/tracked.ts");
+ write(child, "src/forced.generated.ts");
+ write(child, "build/handwritten.ts");
+ commitAll(child, "child fixture", ["src/forced.generated.ts"]);
+ addLocalSubmodule(child, nested, "components/nested module");
+ commitAll(child, "add nested submodule");
+
+ write(parent, ".gitignore", "deps/child module/src/untracked.ts\n");
+ write(parent, "src/root.ts");
+ commitAll(parent, "parent fixture");
+ addLocalSubmodule(parent, child, "deps/child module");
+ addLocalSubmodule(parent, child, "vendor/skipped child");
+ commitAll(parent, "add child submodules");
+ runGit(parent, [
+ "-c", "protocol.file.allow=always",
+ "submodule", "update", "--init", "--recursive", "--", "deps/child module",
+ ]);
+
+ const childCheckout = join(parent, "deps", "child module");
+ const nestedCheckout = join(childCheckout, "components", "nested module");
+ write(childCheckout, "src/untracked.ts");
+ write(childCheckout, "src/untracked.generated.ts");
+ write(nestedCheckout, "src/nested-untracked.ts");
+ write(nestedCheckout, "src/nested-untracked.generated.ts");
+
+ assert.deepEqual(walked(parent), [
+ "deps/child module/components/nested module/src/nested-untracked.ts",
+ "deps/child module/components/nested module/src/nested.ts",
+ "deps/child module/src/forced.generated.ts",
+ "deps/child module/src/tracked.ts",
+ "deps/child module/src/untracked.ts",
+ "src/root.ts",
+ ]);
+
+ const withVendor = walked(parent, new Set(["vendor"]));
+ assert.ok(withVendor.includes("vendor/skipped child/src/tracked.ts"), "an initialized gitlink is included when its mount's built-in skip is lifted");
+ assert.ok(!withVendor.includes("vendor/skipped child/build/handwritten.ts"), "the child's own build/ directory remains skipped");
+ assert.ok(!withVendor.some((path) => path.endsWith("untracked.generated.ts")), "--include-dir never overrides a child's Git ignore rules");
+
+ const withVendorAndBuild = walked(parent, new Set(["vendor", "build"]));
+ assert.ok(withVendorAndBuild.includes("vendor/skipped child/build/handwritten.ts"));
+
+ const childGitFile = join(childCheckout, ".git");
+ const childGitBackup = join(childCheckout, "git-pointer.backup");
+ renameSync(childGitFile, childGitBackup);
+ writeFileSync(childGitFile, "gitdir: missing-gitdir\n");
+ assert.ok(
+ walked(parent).includes("deps/child module/src/tracked.ts"),
+ "a child Git failure falls back locally instead of producing a healthy but incomplete graph",
+ );
+ rmSync(childGitFile, { force: true });
+ renameSync(childGitBackup, childGitFile);
+
+ runGit(parent, ["submodule", "deinit", "-f", "--all"]);
+ assert.deepEqual(walked(parent), ["src/root.ts"], "deinitialized gitlinks must not resolve upward and duplicate the parent");
+ } finally {
+ rmSync(parent, { recursive: true, force: true });
+ rmSync(child, { recursive: true, force: true });
+ rmSync(nested, { recursive: true, force: true });
+ }
+});
+
test("walkDir retains fixed skips and filesystem fallback outside Git", () => {
const dir = mkdtempSync(join(tmpdir(), "graft-walk-nongit-"));
try {