Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

Add code-repo summarization ingest, Claude Code hooks, and PR ingestion

anirudhkumar-nanonets29a387f364cf
diff --git a/README.md b/README.md
index 9519de93..3eb70ef0 100644
--- a/README.md
+++ b/README.md
@@ -35,7 +35,7 @@ context-graph-web
# → http://localhost:4680
```
-Open the page, point **“Ingest a folder of documents”** at any folder of `.pdf` / `.md` / `.txt` files, and watch it become a knowledge graph. Then ask questions: answers come back with numbered citations, and every fact wears a **provenance receipt** — how many times it's been observed, how confident the graph is, and which sources say so. Use **“Teach it something”** to add a fact the docs don't capture, and watch the graph reinforce in real time.
+Open the page, drop a folder of `.pdf` / `.md` / `.txt` files on the **“Add documents”** dropzone, and watch it become a knowledge graph. Then ask questions: answers come back with numbered citations, and every fact wears a **provenance receipt** — how many times it's been observed, how confident the graph is, and which sources say so. Use **“Add a fact”** to record something the docs don't capture, and watch the graph reinforce in real time.
Everything runs on your machine (the server binds to localhost only). Set `OPENROUTER_API_KEY` to upgrade to LLM-written answers and cloud extraction; without it, retrieval and the graph stay fully local.
@@ -102,6 +102,15 @@ context-graph ingest ./docs/*.md ./handbook.pdf
context-graph ingest-dir ./docs # a whole folder, recursively
echo "some notes" | context-graph ingest-text --title "Notes"
+# Ingest a code repository — as summaries, not raw code (see "Code repositories")
+context-graph repo .
+
+# Ingest GitHub pull requests (title, description, review discussion) via the gh CLI
+context-graph ingest-prs --state merged --limit 50
+
+# Wire the graph into Claude Code (see "Claude Code hooks")
+context-graph install-hooks
+
# Visualize the graph (interactive HTML; also --format json|mermaid)
context-graph export --out graph.html && open graph.html
@@ -155,7 +164,8 @@ Tools exposed:
| `context_contribute` | Write a learning back into the shared graph |
| `context_ingest` | Ingest a document (raw text) into the graph |
| `context_ingest_file` | Ingest files from disk, including **PDFs** (parsed automatically) |
-| `context_ingest_dir` | Ingest a whole **directory** of docs (PDF/MD/TXT), recursively |
+| `context_ingest_dir` | Ingest a whole **directory** of docs (PDF/MD/TXT by default; pass `extensions` to widen), recursively |
+| `context_ingest_repo` | Ingest a **code repository** as per-file prose summaries (never raw code); incremental across runs |
| `context_export` | Write the graph to an interactive **HTML** visualization |
| `context_sync` | Team sharing (git mode): import + re-merge the shared `graph.jsonl`, then write it back |
| `context_stats` | Report how much the graph currently holds |
@@ -164,6 +174,44 @@ A natural agent workflow: **read context → do the task → contribute what you
---
+## Code repositories
+
+`context-graph repo <dir>` (or the `context_ingest_repo` MCP tool) ingests a codebase — but deliberately **never feeds raw source code to the entity extractor**. Running LLM extraction over thousands of code chunks is the most expensive way to compute what `grep` and tree-sitter give agents for free, and it fills the graph with noisy, duplicated symbol entities. (Every serious code-context system — aider, Cursor, Sourcegraph, Claude Code — derives code *structure* statically, and uses LLMs only for prose.)
+
+Instead, repo ingestion:
+
+1. Walks the repo for code files (`.ts`, `.py`, `.go`, … — `node_modules`/`dist`/etc. and files over 1 MB are skipped).
+2. Writes **one LLM prose summary per file**: purpose, key exports, dependencies, design decisions.
+3. Groups summaries into one document per top-level directory and ingests those through the normal pipeline — so the graph gains *module-level* entities ("the chunker is paragraph-aware", "the CLI requires the GitHub CLI for PR ingestion") with provenance, not ten thousand function names.
+
+It is **incremental**: summaries are cached by content hash in `repo-summaries.json` next to the db, so re-running after edits only re-summarizes changed files, and a changed file's module document *replaces* its predecessor rather than piling up stale duplicates. An unchanged repo re-ingests with **zero** LLM calls.
+
+The same replace-on-change behavior now applies to every file ingest: re-ingesting a modified doc replaces the old version instead of leaving both in the graph.
+
+For a ~30-file repo expect ~30 small summary calls on first run; the graph stays module-scale (a handful of documents) rather than chunk-soup-scale. Agents should still read raw code with their own tools — the graph holds what code can't say: intent, decisions, and gotchas.
+
+## Claude Code hooks
+
+`context-graph install-hooks` wires the graph into Claude Code via `.claude/settings.json` (merges with what's already there):
+
+- **SessionStart** — every new session begins with relevant graph context (conventions, decisions, gotchas) auto-injected, plus a reminder that `context_contribute` exists.
+- **Stop** — when the agent finishes, it's nudged once to record any durable learnings from the session before ending. Pass `--no-stop` to skip this hook if you find the nudge too chatty.
+
+Both hooks stay silent when there's no graph yet, so installing them in a fresh project is harmless. Combined with the MCP server this closes the loop: sessions **start** with team knowledge and **end** by adding to it.
+
+## GitHub pull requests
+
+PR descriptions and review threads are where decisions actually get written down. `context-graph ingest-prs` pulls them into the graph via the [GitHub CLI](https://cli.github.com) (no extra auth needed if `gh` is logged in):
+
+```bash
+context-graph ingest-prs # merged PRs of the current repo (default 50)
+context-graph ingest-prs -R owner/name -n 100 --state all
+```
+
+Each PR becomes one document (title, description, review discussion) with source `pr:owner/name#123`. Re-runs are cheap: unchanged PRs are skipped by hash, edited ones replace their old version.
+
+---
+
## Team sharing
A solo graph is one SQLite file. To share one across a team, pick a mode. All three keep the local-first default intact (team sync is **opt-in**).
@@ -177,7 +225,7 @@ docker compose up -d
cat ./data/share-link.txt # the share link, with access token
```
-The graph persists to `./data/graph.db`. Ingest documents two ways: click **Choose folder…** in the UI to upload a folder straight from your computer (works for remote teammates), or drop files into `./docs` on the server and ingest the mounted `/docs` path. When exposed beyond localhost the API requires an access token. It's generated once, saved to `./data/share-link.txt` (and printed to `docker compose logs`), and **stays stable across restarts** so shared links keep working. Pin your own with `CONTEXT_GRAPH_WEB_TOKEN`.
+The graph persists to `./data/graph.db`. Ingest documents two ways: click **Choose a folder…** in the UI to upload a folder straight from your computer (works for remote teammates), or drop files into `./docs` on the server and ingest the mounted `/docs` path. When exposed beyond localhost the API requires an access token. It's generated once, saved to `./data/share-link.txt` (and printed to `docker compose logs`), and **stays stable across restarts** so shared links keep working. Pin your own with `CONTEXT_GRAPH_WEB_TOKEN`.
No server to put it on? Run it on any machine and share it over [Tailscale](https://tailscale.com) without opening a port to the internet:
@@ -271,12 +319,13 @@ new ContextGraphEngine({
The engine depends only on small interfaces, so you can replace any piece:
```ts
-import { ContextGraphEngine, type Embedder, type Extractor, type GraphStore } from "context-graph-engine";
+import { ContextGraphEngine, type Embedder, type Extractor, type GraphStore, type Summarizer } from "context-graph-engine";
new ContextGraphEngine({
- store: myCustomStore, // e.g. Postgres/pgvector
- embedder: myCustomEmbedder, // any embedding model
- extractor: myCustomExtractor // any LLM, or a rules-based extractor
+ store: myCustomStore, // e.g. Postgres/pgvector
+ embedder: myCustomEmbedder, // any embedding model
+ extractor: myCustomExtractor, // any LLM, or a rules-based extractor
+ summarizer: myCustomSummarizer // code-file summaries for ingestRepo
});
```
@@ -291,6 +340,7 @@ class ContextGraphEngine {
ingest(text: string, opts?: { title?; source? }): Promise<IngestResult>;
ingestFile(path: string, opts?): Promise<IngestResult>; // .pdf parsed automatically
ingestDir(dir: string, opts?): Promise<IngestResult[]>; // recursive; PDF/MD/TXT
+ ingestRepo(dir: string, opts?): Promise<RepoIngestResult>; // code → per-file summaries, incremental
read(query: string, opts?: { maxNodes?; maxChunks?; expand? }): Promise<ContextBundle>;
contribute(learning: string, opts?: { agentId?; source? }): Promise<ContributeResult>;
exportGraph(): GraphExport; // -> toHtml / toMermaid
diff --git a/src/ai/providers.ts b/src/ai/providers.ts
index 1dc79e84..f51a8b96 100644
--- a/src/ai/providers.ts
+++ b/src/ai/providers.ts
@@ -1,5 +1,6 @@
import type { Extraction } from "../graph/types.js";
import type { GraphStore } from "../graph/store.js";
+import type { Summarizer } from "./summarize.js";
/** Turns text into embedding vectors for semantic search and dedup. */
export interface Embedder {
@@ -67,6 +68,8 @@ export interface EngineConfig {
embedder?: Embedder;
/** Override the extractor (defaults to OpenRouter, or local Ollama with no key). */
extractor?: Extractor;
+ /** Override the code summarizer used by repo ingestion (same fallbacks as extractor). */
+ summarizer?: Summarizer;
}
/** Fully-resolved configuration with all defaults applied. */
@@ -87,6 +90,7 @@ export interface ResolvedConfig {
store?: GraphStore;
embedder?: Embedder;
extractor?: Extractor;
+ summarizer?: Summarizer;
}
export const DEFAULTS = {
@@ -136,5 +140,6 @@ export function resolveConfig(config: EngineConfig = {}): ResolvedConfig {
store: config.store,
embedder: config.embedder,
extractor: config.extractor,
+ summarizer: config.summarizer,
};
}
diff --git a/src/ai/summarize.ts b/src/ai/summarize.ts
new file mode 100644
index 00000000..72e02c2e
--- /dev/null
+++ b/src/ai/summarize.ts
@@ -0,0 +1,103 @@
+import OpenAI from "openai";
+
+/**
+ * Turns one source-code file into a short prose summary for the knowledge graph.
+ *
+ * Code repos are deliberately NOT fed through the per-chunk entity extractor:
+ * static structure is better served by grep/tree-sitter in the consuming agent,
+ * and LLM extraction over raw code produces noisy, duplicated entities. Instead
+ * the repo ingest path summarizes each file into plain English (purpose, key
+ * exports, dependencies, design decisions) and ingests those summaries — prose
+ * the existing extraction pipeline is actually good at.
+ */
+export interface Summarizer {
+ summarize(code: string, opts: { path: string }): Promise<string>;
+}
+
+const SYSTEM_PROMPT = `You document source code for a team knowledge base. Given one source file, write a compact plain-English summary covering:
+1. The purpose of the file — what it exists to do.
+2. The key exported functions/classes/types and what each is for.
+3. Important dependencies: internal modules it builds on, external libraries or services it talks to.
+4. Notable design decisions, constraints, or gotchas evident in the code.
+
+Write 3-8 sentences of flowing prose. Name concrete identifiers (modules, classes, services) so they can become graph entities. No code blocks, no line-by-line narration, no filler.`;
+
+/** Cap the code sent per file so a single giant file can't blow the context. */
+const MAX_CODE_CHARS = 24_000;
+
+function userContent(code: string, path: string): string {
+ const clipped =
+ code.length > MAX_CODE_CHARS
+ ? `${code.slice(0, MAX_CODE_CHARS)}\n… (truncated at ${MAX_CODE_CHARS} characters)`
+ : code;
+ return `File: ${path}\n\n${clipped}`;
+}
+
+/** Summarizer backed by OpenRouter's OpenAI-compatible chat API. */
+export class OpenRouterSummarizer implements Summarizer {
+ private client: OpenAI;
+ private model: string;
+
+ constructor(apiKey: string, model: string, baseUrl = "https://openrouter.ai/api/v1") {
+ this.client = new OpenAI({
+ apiKey,
+ baseURL: baseUrl,
+ defaultHeaders: { "X-Title": "Context Graph Engine" },
+ });
+ this.model = model;
+ }
+
+ async summarize(code: string, opts: { path: string }): Promise<string> {
+ const response = await this.client.chat.completions.create({
+ model: this.model,
+ temperature: 0,
+ messages: [
+ { role: "system", content: SYSTEM_PROMPT },
+ { role: "user", content: userContent(code, opts.path) },
+ ],
+ });
+ return response.choices[0]?.message?.content?.trim() ?? "";
+ }
+}
+
+/** Summarizer backed by a locally running Ollama instance (key-free fallback). */
+export class OllamaSummarizer implements Summarizer {
+ private baseUrl: string;
+ private model: string;
+
+ constructor(model = "llama3.2", baseUrl = "http://localhost:11434") {
+ this.model = model;
+ this.baseUrl = baseUrl.replace(/\/$/, "");
+ }
+
+ async summarize(code: string, opts: { path: string }): Promise<string> {
+ let res: Response;
+ try {
+ res = await fetch(`${this.baseUrl}/api/chat`, {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify({
+ model: this.model,
+ stream: false,
+ options: { temperature: 0 },
+ messages: [
+ { role: "system", content: SYSTEM_PROMPT },
+ { role: "user", content: userContent(code, opts.path) },
+ ],
+ }),
+ });
+ } catch (err) {
+ throw new Error(
+ `Could not reach Ollama at ${this.baseUrl}. Start it (https://ollama.com) and pull a model ` +
+ `(e.g. \`ollama pull ${this.model}\`), or set OPENROUTER_API_KEY to use cloud summarization. ` +
+ `Underlying error: ${err instanceof Error ? err.message : String(err)}`,
+ );
+ }
+ if (!res.ok) {
+ const body = await res.text().catch(() => "");
+ throw new Error(`Ollama returned ${res.status}: ${body}`);
+ }
+ const data = (await res.json()) as { message?: { content?: string } };
+ return data.message?.content?.trim() ?? "";
+ }
+}
diff --git a/src/cli.ts b/src/cli.ts
index a401bd1c..6ee9bd38 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -12,8 +12,12 @@
*/
import "dotenv/config";
import { Command } from "commander";
-import { readFileSync, writeFileSync, existsSync } from "node:fs";
+import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
+import { basename, dirname, join, resolve } from "node:path";
import { ContextGraphEngine } from "./engine.js";
+import { resolveConfig } from "./ai/providers.js";
import { toHtml, toMermaid } from "./graph/export.js";
const program = new Command();
@@ -132,6 +136,220 @@ program
}
});
+program
+ .command("repo")
+ .description(
+ "Ingest a code repository as prose summaries (one LLM summary per file, grouped per top-level " +
+ "directory). Raw code is never fed to the extractor. Incremental: only changed files are re-summarized.",
+ )
+ .argument("<dir>", "repository root")
+ .option("-e, --extensions <exts...>", 'code extensions to include (e.g. ".ts" ".py")')
+ .action(async (dir: string, opts: { extensions?: string[] }) => {
+ const engine = engineFrom();
+ try {
+ const r = await engine.ingestRepo(dir, {
+ extensions: opts.extensions,
+ onProgress: ({ phase, index, total, file }) =>
+ process.stderr.write(
+ `\r${phase === "summarize" ? "summarizing" : "ingesting"} ${index + 1}/${total}: ${file.slice(0, 60).padEnd(60)}`,
+ ),
+ });
+ process.stderr.write("\n");
+ console.log(
+ `✓ ${r.files} code files — ${r.summarized} summarized, ${r.cached} unchanged (cache hits)`,
+ );
+ for (const m of r.modules) {
+ console.log(
+ m.skipped
+ ? `• ${m.title} — unchanged, skipped`
+ : `✓ ${m.title} — ${m.chunks} chunks, +${m.nodesCreated} nodes, +${m.edgesCreated} edges`,
+ );
+ }
+ for (const e of r.errors) console.error(`✗ ${e}`);
+ } finally {
+ await engine.close();
+ }
+ });
+
+const ghExec = promisify(execFile);
+
+/** Run a `gh` command and parse its JSON output. */
+async function gh<T>(args: string[]): Promise<T> {
+ try {
+ const { stdout } = await ghExec("gh", args, { maxBuffer: 32_000_000 });
+ return JSON.parse(stdout) as T;
+ } catch (err) {
+ const e = err as NodeJS.ErrnoException & { stderr?: string };
+ if (e.code === "ENOENT") {
+ throw new Error("GitHub CLI (`gh`) not found. Install it from https://cli.github.com and run `gh auth login`.");
+ }
+ throw new Error(`gh ${args.slice(0, 3).join(" ")} failed: ${e.stderr?.trim() || e.message}`);
+ }
+}
+
+interface PrDetail {
+ number: number;
+ title: string;
+ body: string;
+ url: string;
+ state: string;
+ mergedAt: string | null;
+ author: { login: string } | null;
+ comments: Array<{ author: { login: string } | null; body: string }>;
+ reviews: Array<{ author: { login: string } | null; state: string; body: string }>;
+}
+
+program
+ .command("ingest-prs")
+ .description("Ingest GitHub pull requests (title, description, review comments) via the gh CLI")
+ .option("-R, --repo <owner/name>", "GitHub repository (default: the current directory's repo)")
+ .option("-n, --limit <n>", "max PRs to ingest", (v) => parseInt(v, 10), 50)
+ .option("-s, --state <state>", "open | closed | merged | all", "merged")
+ .action(async (opts: { repo?: string; limit: number; state: string }) => {
+ const repo =
+ opts.repo ??
+ (await gh<{ nameWithOwner: string }>(["repo", "view", "--json", "nameWithOwner"])).nameWithOwner;
+ const list = await gh<Array<{ number: number }>>([
+ "pr", "list", "--repo", repo, "--state", opts.state,
+ "--limit", String(opts.limit), "--json", "number",
+ ]);
+ if (list.length === 0) {
+ console.log(`No ${opts.state} PRs found in ${repo}.`);
+ return;
+ }
+ const engine = engineFrom();
+ try {
+ for (const { number } of list) {
+ const pr = await gh<PrDetail>([
+ "pr", "view", String(number), "--repo", repo,
+ "--json", "number,title,body,url,state,mergedAt,author,comments,reviews",
+ ]);
+ const discussion = [
+ ...pr.reviews
+ .filter((r) => r.body?.trim())
+ .map((r) => `- ${r.author?.login ?? "unknown"} (review, ${r.state}): ${r.body.trim()}`),
+ ...pr.comments
+ .filter((c) => c.body?.trim())
+ .map((c) => `- ${c.author?.login ?? "unknown"}: ${c.body.trim()}`),
+ ];
+ const text = [
+ `# PR #${pr.number}: ${pr.title}`,
+ `Repo: ${repo}. Author: ${pr.author?.login ?? "unknown"}. State: ${pr.state}${pr.mergedAt ? ` (merged ${pr.mergedAt.slice(0, 10)})` : ""}. ${pr.url}`,
+ `## Description\n\n${pr.body?.trim() || "(no description)"}`,
+ ...(discussion.length ? [`## Discussion\n\n${discussion.join("\n")}`] : []),
+ ].join("\n\n");
+ const r = await engine.ingest(text, {
+ title: `PR #${pr.number}: ${pr.title}`,
+ source: `pr:${repo}#${pr.number}`,
+ });
+ console.log(
+ r.skipped
+ ? `• PR #${pr.number} — already ingested, skipped`
+ : `✓ PR #${pr.number}: ${pr.title} — +${r.nodesCreated} nodes, +${r.edgesCreated} edges`,
+ );
+ }
+ } finally {
+ await engine.close();
+ }
+ });
+
+/** The two hook commands install-hooks wires into Claude Code. */
+const SESSION_START_CMD = "context-graph hook session-start";
+const STOP_CMD = "context-graph hook stop";
+
+program
+ .command("install-hooks")
+ .description("Install Claude Code hooks: inject graph context at session start, nudge a contribution at stop")
+ .option("--dir <path>", "project directory (default: current directory)")
+ .option("--no-stop", "skip the Stop hook (the contribution nudge)")
+ .action(async (opts: { dir?: string; stop: boolean }) => {
+ const projectDir = resolve(opts.dir ?? ".");
+ const settingsPath = join(projectDir, ".claude", "settings.json");
+ let settings: Record<string, unknown> = {};
+ if (existsSync(settingsPath)) {
+ settings = JSON.parse(readFileSync(settingsPath, "utf8")) as Record<string, unknown>;
+ }
+ const hooks = (settings.hooks ??= {}) as Record<
+ string,
+ Array<{ matcher?: string; hooks: Array<{ type: string; command: string }> }>
+ >;
+ const addHook = (event: string, command: string): boolean => {
+ const entries = (hooks[event] ??= []);
+ if (entries.some((e) => e.hooks?.some((h) => h.command === command))) return false;
+ entries.push({ hooks: [{ type: "command", command }] });
+ return true;
+ };
+ const added: string[] = [];
+ if (addHook("SessionStart", SESSION_START_CMD)) added.push("SessionStart");
+ if (opts.stop && addHook("Stop", STOP_CMD)) added.push("Stop");
+ mkdirSync(dirname(settingsPath), { recursive: true });
+ writeFileSync(settingsPath, JSON.stringify(settings, null, 2) + "\n");
+ console.log(
+ added.length
+ ? `✓ installed ${added.join(" + ")} hook${added.length > 1 ? "s" : ""} in ${settingsPath}`
+ : `• hooks already installed in ${settingsPath}`,
+ );
+ console.log(" SessionStart injects relevant graph context; Stop nudges the agent to contribute learnings.");
+ });
+
+const hook = program
+ .command("hook")
+ .description("Claude Code hook entry points (installed by `install-hooks`; not usually run by hand)");
+
+hook
+ .command("session-start")
+ .description("Print relevant graph context for injection at Claude Code session start")
+ .action(async () => {
+ // No graph yet → stay silent rather than erroring inside the hook.
+ const cfg = resolveConfig({ dbPath: program.opts<{ db?: string }>().db });
+ if (cfg.dbPath !== ":memory:" && !existsSync(cfg.dbPath)) return;
+ const engine = engineFrom();
+ try {
+ const stats = await engine.stats();
+ if (stats.nodes === 0) return;
+ const project = basename(process.cwd());
+ const bundle = await engine.read(
+ `${project}: conventions, architecture decisions, gotchas, current work`,
+ { maxNodes: 8, maxChunks: 4 },
+ );
+ if (bundle.nodes.length === 0 && bundle.chunks.length === 0) return;
+ console.log("## Team context graph (auto-injected at session start)\n");
+ console.log(bundle.prompt);
+ console.log(
+ "\nWhen you learn something durable this session (a decision, gotcha, or convention), " +
+ "record it with the context_contribute MCP tool.",
+ );
+ } finally {
+ await engine.close();
+ }
+ });
+
+hook
+ .command("stop")
+ .description("Nudge the agent to contribute session learnings before finishing")
+ .action(async () => {
+ // Reads the Stop-hook JSON from stdin. `stop_hook_active` means this stop
+ // was already extended by us once — let it through or we'd loop forever.
+ let input: { stop_hook_active?: boolean } = {};
+ try {
+ input = JSON.parse(readFileSync(0, "utf8")) as { stop_hook_active?: boolean };
+ } catch {
+ /* no stdin — treat as a plain stop */
+ }
+ if (input.stop_hook_active) return;
+ const cfg = resolveConfig({ dbPath: program.opts<{ db?: string }>().db });
+ if (cfg.dbPath !== ":memory:" && !existsSync(cfg.dbPath)) return;
+ console.log(
+ JSON.stringify({
+ decision: "block",
+ reason:
+ "Before finishing: if this session produced a durable, non-obvious learning (a decision made, " +
+ "a gotcha hit, a convention discovered, a fix future agents should know about), record each one " +
+ "with the context_contribute MCP tool. If nothing is worth recording, finish without contributing.",
+ }),
+ );
+ });
+
program
.command("export")
.description("Export the whole graph for viewing (interactive HTML, JSON, or Mermaid)")
diff --git a/src/engine.ts b/src/engine.ts
index 24358a45..2eb9d7d7 100644
--- a/src/engine.ts
+++ b/src/engine.ts
@@ -1,5 +1,5 @@
-import { readFileSync, readdirSync } from "node:fs";
-import { basename, dirname, join } from "node:path";
+import { readFileSync, readdirSync, statSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
+import { basename, dirname, join, relative, resolve, sep } from "node:path";
import { extractPdfText, isPdfPath } from "./ingest/pdf.js";
import { buildGraphExport, type GraphExport } from "./graph/export.js";
import type { GraphStore } from "./graph/store.js";
@@ -9,6 +9,7 @@ import { resolveConfig } from "./ai/providers.js";
import { OpenRouterExtractor } from "./ai/openrouter.js";
import { OpenAIEmbedder } from "./ai/openai.js";
import { LocalEmbedder, OllamaExtractor } from "./ai/local.js";
+import { OllamaSummarizer, OpenRouterSummarizer, type Summarizer } from "./ai/summarize.js";
import { chunkText } from "./ingest/chunker.js";
import { mergeExtraction } from "./graph/merge.js";
import { serializeGraph, importGraph, type ImportResult } from "./graph/serialize.js";
@@ -50,6 +51,42 @@ export interface ContributeResult {
edgesUpdated: number;
}
+/** Extensions treated as source code by {@link ContextGraphEngine.ingestRepo}. */
+export const CODE_EXTENSIONS = [
+ ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs",
+ ".py", ".go", ".rs", ".java", ".kt", ".scala",
+ ".rb", ".php", ".c", ".h", ".cpp", ".hpp", ".cc",
+ ".cs", ".swift", ".sql", ".sh", ".proto",
+];
+
+export interface RepoIngestOptions {
+ /** File extensions to treat as code. Default: {@link CODE_EXTENSIONS}. */
+ extensions?: string[];
+ /** Called before each file summary and each module ingest, for progress UIs. */
+ onProgress?: (info: {
+ phase: "summarize" | "ingest";
+ index: number;
+ total: number;
+ file: string;
+ }) => void;
+}
+
+export interface RepoIngestResult {
+ /** Code files considered. */
+ files: number;
+ /** Files freshly summarized with an LLM call this run. */
+ summarized: number;
+ /** Files whose summary was reused from the content-hash cache. */
+ cached: number;
+ /** One ingest result per module (top-level directory) document. */
+ modules: IngestResult[];
+ /** Files that could not be read or summarized (path: reason). */
+ errors: string[];
+}
+
+/** file path → { content hash, cached summary } */
+type RepoSummaryCache = Record<string, { hash: string; summary: string }>;
+
/**
* The Context Graph Engine.
*
@@ -62,6 +99,8 @@ export class ContextGraphEngine {
private _store?: GraphStore;
private _embedder?: Embedder;
private _extractor?: Extractor;
+ private _summarizer?: Summarizer;
+ private _memRepoCache: RepoSummaryCache = {};
constructor(config: EngineConfig = {}) {
this.cfg = resolveConfig(config);
@@ -111,6 +150,23 @@ export class ContextGraphEngine {
return this._extractor;
}
+ private get summarizer(): Summarizer {
+ if (!this._summarizer) {
+ if (this.cfg.summarizer) {
+ this._summarizer = this.cfg.summarizer;
+ } else if (!this.cfg.forceLocal && this.cfg.openrouterApiKey) {
+ this._summarizer = new OpenRouterSummarizer(
+ this.cfg.openrouterApiKey,
+ this.cfg.openrouterModel,
+ this.cfg.openrouterBaseUrl,
+ );
+ } else {
+ this._summarizer = new OllamaSummarizer(this.cfg.ollamaModel, this.cfg.ollamaBaseUrl);
+ }
+ }
+ return this._summarizer;
+ }
+
/**
* Ingest a document from disk. PDFs (`.pdf`) are parsed to text automatically;
* anything else is read as UTF-8. Title defaults to the file name.
@@ -141,6 +197,16 @@ export class ContextGraphEngine {
};
}
+ // Same source, new content: the file changed, so replace its stale
+ // document(s) instead of piling up superseded chunks. "inline" is the
+ // catch-all source for pasted text and is exempt — many unrelated
+ // documents legitimately share it.
+ if (source !== "inline") {
+ for (const prior of await this.store.documentsBySource(source)) {
+ await this.store.deleteDocument(prior.id);
+ }
+ }
+
const documentId = newId("doc");
await this.store.insertDocument({ id: documentId, title, source, hash, createdAt: now });
@@ -211,6 +277,119 @@ export class ContextGraphEngine {
return results;
}
+ /**
+ * Ingest a code repository as **summaries, not raw code**.
+ *
+ * Raw source is deliberately kept out of the extraction pipeline (LLM entity
+ * extraction over code is noisy and expensive; agents already grep code
+ * live). Instead each code file gets one LLM-written prose summary; the
+ * summaries are grouped into one document per top-level directory and fed
+ * through the normal {@link ingest} pipeline.
+ *
+ * Incremental: summaries are cached by file content hash (in
+ * `repo-summaries.json` next to the db), so re-running after edits only
+ * re-summarizes changed files, and unchanged module documents are skipped by
+ * the ordinary document-hash dedup.
+ */
+ async ingestRepo(dir: string, opts: RepoIngestOptions = {}): Promise<RepoIngestResult> {
+ const root = resolve(dir);
+ const exts = opts.extensions ?? CODE_EXTENSIONS;
+ const files = walkDir(root).filter((f) => exts.some((e) => f.toLowerCase().endsWith(e)));
+
+ const cache = this.loadRepoCache();
+ const result: RepoIngestResult = {
+ files: files.length,
+ summarized: 0,
+ cached: 0,
+ modules: [],
+ errors: [],
+ };
+
+ // Phase 1: one summary per file, concurrent, content-hash cached.
+ const summaries = await mapWithConcurrency(files, 8, async (file, i) => {
+ const rel = relative(root, file);
+ opts.onProgress?.({ phase: "summarize", index: i, total: files.length, file: rel });
+ let code: string;
+ try {
+ code = readFileSync(file, "utf8");
+ } catch (err) {
+ result.errors.push(`${rel}: ${err instanceof Error ? err.message : String(err)}`);
+ return undefined;
+ }
+ const hash = contentHash(code);
+ const hit = cache[file];
+ if (hit && hit.hash === hash) {
+ result.cached++;
+ return { rel, summary: hit.summary };
+ }
+ try {
+ const summary = await this.summarizer.summarize(code, { path: rel });
+ if (!summary) return undefined;
+ cache[file] = { hash, summary };
+ result.summarized++;
+ return { rel, summary };
+ } catch (err) {
+ result.errors.push(`${rel}: ${err instanceof Error ? err.message : String(err)}`);
+ return undefined;
+ }
+ });
+ this.saveRepoCache(cache);
+
+ // Phase 2: group summaries into one document per top-level directory and
+ // run them through the normal ingest pipeline. The stable `repo:` source
+ // makes re-ingestion replace a module's stale document when it changed.
+ const repoName = basename(root);
+ const modules = new Map<string, Array<{ rel: string; summary: string }>>();
+ for (const s of summaries) {
+ if (!s) continue;
+ const module = s.rel.includes(sep) ? s.rel.split(sep)[0] : "(root)";
+ (modules.get(module) ?? modules.set(module, []).get(module)!).push(s);
+ }
+
+ let i = 0;
+ for (const [module, entries] of modules) {
+ opts.onProgress?.({ phase: "ingest", index: i++, total: modules.size, file: module });
+ const text = [
+ `# Code summary: ${repoName}/${module}`,
+ ...entries.map((e) => `## ${e.rel}\n\n${e.summary}`),
+ ].join("\n\n");
+ result.modules.push(
+ await this.ingest(text, {
+ title: `${repoName}/${module} (code summary)`,
+ source: `repo:${root}#${module}`,
+ }),
+ );
+ }
+ return result;
+ }
+
+ /** Path of the file-summary cache, or undefined for in-memory graphs. */
+ private get repoCachePath(): string | undefined {
+ if (this.cfg.dbPath === ":memory:") return undefined;
+ return join(dirname(this.cfg.dbPath), "repo-summaries.json");
+ }
+
+ private loadRepoCache(): RepoSummaryCache {
+ const path = this.repoCachePath;
+ if (!path) return this._memRepoCache;
+ if (!existsSync(path)) return {};
+ try {
+ return JSON.parse(readFileSync(path, "utf8")) as RepoSummaryCache;
+ } catch {
+ return {};
+ }
+ }
+
+ private saveRepoCache(cache: RepoSummaryCache): void {
+ const path = this.repoCachePath;
+ if (!path) {
+ this._memRepoCache = cache;
+ return;
+ }
+ mkdirSync(dirname(path), { recursive: true });
+ writeFileSync(path, JSON.stringify(cache, null, 2));
+ }
+
/** Read the graph for a query, returning a structured context bundle. */
async read(query: string, opts: RetrieveOptions = {}): Promise<ContextBundle> {
return retrieve(this.store, this.embedder, query, opts);
@@ -327,14 +506,42 @@ export class ContextGraphEngine {
}
}
-/** Recursively list all files under a directory (skips dot-directories). */
+/** Directories that are dependency/build output, never knowledge. */
+const SKIP_DIRS = new Set([
+ "node_modules",
+ "dist",
+ "build",
+ "out",
+ "target",
+ "vendor",
+ "coverage",
+ "__pycache__",
+ "venv",
+]);
+
+/** Files above this size are generated/vendored in practice, not written docs or code. */
+const MAX_FILE_BYTES = 1_000_000;
+
+/**
+ * Recursively list all files under a directory. Skips dot-directories,
+ * dependency/build directories (node_modules, dist, …), and files over 1 MB.
+ */
function walkDir(dir: string): string[] {
const out: string[] = [];
for (const entry of readdirSync(dir, { withFileTypes: true })) {
if (entry.name.startsWith(".")) continue;
const full = join(dir, entry.name);
- if (entry.isDirectory()) out.push(...walkDir(full));
- else if (entry.isFile()) out.push(full);
+ if (entry.isDirectory()) {
+ if (SKIP_DIRS.has(entry.name)) continue;
+ out.push(...walkDir(full));
+ } else if (entry.isFile()) {
+ try {
+ if (statSync(full).size > MAX_FILE_BYTES) continue;
+ } catch {
+ continue;
+ }
+ out.push(full);
+ }
}
return out;
}
diff --git a/src/graph/sqlite-store.ts b/src/graph/sqlite-store.ts
index 09d805fc..94a14b13 100644
--- a/src/graph/sqlite-store.ts
+++ b/src/graph/sqlite-store.ts
@@ -160,6 +160,18 @@ export class SqliteStore implements GraphStore {
return row ? this.rowToDocument(row) : undefined;
}
+ async documentsBySource(source: string): Promise<GraphDocument[]> {
+ const rows = this.db
+ .prepare(`SELECT * FROM documents WHERE source = ? ORDER BY created_at DESC`)
+ .all(source) as Array<Record<string, unknown>>;
+ return rows.map((r) => this.rowToDocument(r));
+ }
+
+ async deleteDocument(id: string): Promise<void> {
+ // Chunks cascade via the document_id foreign key.
+ this.db.prepare(`DELETE FROM documents WHERE id = ?`).run(id);
+ }
+
async insertDocument(doc: GraphDocument): Promise<void> {
this.db
.prepare(
diff --git a/src/graph/store.ts b/src/graph/store.ts
index 72cc0309..666f8da5 100644
--- a/src/graph/store.ts
+++ b/src/graph/store.ts
@@ -22,7 +22,15 @@ export interface GraphStore {
// --- documents ---
getDocumentByHash(hash: string): Promise<GraphDocument | undefined>;
getDocumentById(id: string): Promise<GraphDocument | undefined>;
+ /** All documents recorded with the given source (path/URL/label), newest first. */
+ documentsBySource(source: string): Promise<GraphDocument[]>;
insertDocument(doc: GraphDocument): Promise<void>;
+ /**
+ * Delete a document and its chunks (e.g. when re-ingesting a changed file).
+ * Nodes/edges extracted from it keep their observations — the grow-only CRDT
+ * counters never shrink.
+ */
+ deleteDocument(id: string): Promise<void>;
/** Every ingested document, newest first (for UI listings). */
allDocuments(): Promise<GraphDocument[]>;
diff --git a/src/index.ts b/src/index.ts
index 9dce2c89..7aa5daf4 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -16,12 +16,14 @@
* });
* ```
*/
-export { ContextGraphEngine } from "./engine.js";
+export { ContextGraphEngine, CODE_EXTENSIONS } from "./engine.js";
export type {
IngestOptions,
IngestResult,
ContributeOptions,
ContributeResult,
+ RepoIngestOptions,
+ RepoIngestResult,
} from "./engine.js";
export type {
@@ -67,3 +69,5 @@ export type {
export { OpenRouterExtractor } from "./ai/openrouter.js";
export { OpenAIEmbedder } from "./ai/openai.js";
export { LocalEmbedder, OllamaExtractor } from "./ai/local.js";
+export { OpenRouterSummarizer, OllamaSummarizer } from "./ai/summarize.js";
+export type { Summarizer } from "./ai/summarize.js";
diff --git a/src/mcp.ts b/src/mcp.ts
index 9a066857..d5782979 100644
--- a/src/mcp.ts
+++ b/src/mcp.ts
@@ -127,9 +127,13 @@ server.registerTool(
"Ingest every supported document (.pdf, .md, .markdown, .txt) in a directory, recursively, into the shared context graph. Point this at a folder of docs and it builds the graph in one call.",
inputSchema: {
dir: z.string().describe("Absolute path to the directory of documents to ingest."),
+ extensions: z
+ .array(z.string())
+ .optional()
+ .describe('File extensions to include (e.g. [".md", ".rst"]). Default: .pdf .md .markdown .txt'),
},
},
- async ({ dir }, extra) => {
+ async ({ dir, extensions }, extra) => {
// Report progress per file. If the client supplied a progressToken (and
// resets its request timeout on progress, as Claude Code does), this keeps
// long multi-file ingests from tripping the default 60s tool timeout.
@@ -147,7 +151,7 @@ server.registerTool(
});
}
: undefined;
- const results = await engine.ingestDir(dir, { onProgress });
+ const results = await engine.ingestDir(dir, { extensions, onProgress });
if (results.length === 0) {
return { content: [{ type: "text", text: `No supported files found under ${dir}.` }] };
}
@@ -161,6 +165,52 @@ server.registerTool(
},
);
+server.registerTool(
+ "context_ingest_repo",
+ {
+ title: "Ingest a code repository (as summaries)",
+ description:
+ "Ingest a code repository into the shared context graph. Each source file is summarized " +
+ "into prose (purpose, key exports, dependencies, design decisions) and the summaries are " +
+ "ingested — raw code is never fed to the extractor. Incremental: re-running only " +
+ "re-summarizes files whose content changed.",
+ inputSchema: {
+ dir: z.string().describe("Absolute path to the repository root."),
+ extensions: z
+ .array(z.string())
+ .optional()
+ .describe('File extensions to treat as code (default: common languages, e.g. ".ts", ".py", ".go").'),
+ },
+ },
+ async ({ dir, extensions }, extra) => {
+ const progressToken = extra?._meta?.progressToken;
+ const onProgress = progressToken
+ ? ({ phase, index, total, file }: { phase: string; index: number; total: number; file: string }) => {
+ void extra.sendNotification({
+ method: "notifications/progress",
+ params: {
+ progressToken,
+ progress: index,
+ total,
+ message: `${phase === "summarize" ? "Summarizing" : "Ingesting"} ${file} (${index + 1}/${total})`,
+ },
+ });
+ }
+ : undefined;
+ const r = await engine.ingestRepo(dir, { extensions, onProgress });
+ const lines = [
+ `Repo ingested: ${r.files} code files → ${r.summarized} summarized, ${r.cached} unchanged (cache hits).`,
+ ...r.modules.map((m) =>
+ m.skipped
+ ? `• ${m.title}: unchanged, skipped`
+ : `✓ ${m.title}: ${m.chunks} chunks, +${m.nodesCreated} entities, +${m.edgesCreated} relationships`,
+ ),
+ ...r.errors.map((e) => `✗ ${e}`),
+ ];
+ return { content: [{ type: "text", text: lines.join("\n") }] };
+ },
+);
+
server.registerTool(
"context_export",
{
diff --git a/src/web.ts b/src/web.ts
index 6a67905b..6ea58d7b 100644
--- a/src/web.ts
+++ b/src/web.ts
@@ -287,7 +287,10 @@ const server = createServer(async (req, res) => {
if (req.method === "POST" && url.pathname === "/api/ingest-dir") {
// Accepts a directory or a single file on the server's disk.
- const { dir } = JSON.parse(await readBody(req)) as { dir?: string };
+ const { dir, extensions } = JSON.parse(await readBody(req)) as {
+ dir?: string;
+ extensions?: string[];
+ };
if (!dir?.trim()) return json(res, 400, { error: "dir is required" });
const abs = resolve(dir);
if (!existsSync(abs)) {
@@ -301,6 +304,7 @@ const server = createServer(async (req, res) => {
try {
const results = isDir
? await engine.ingestDir(abs, {
+ extensions,
onProgress: (info) => emit({ type: "progress", ...info }),
})
: [await engine.ingestFile(abs)];
diff --git a/test/repo.test.ts b/test/repo.test.ts
new file mode 100644
index 00000000..a6a2b92b
--- /dev/null
+++ b/test/repo.test.ts
@@ -0,0 +1,96 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { ContextGraphEngine, type Summarizer } from "../src/index.js";
+import { fakeProviders } from "./helpers.ts";
+
+/** Deterministic summarizer: counts calls, emits bracket markup the BracketExtractor understands. */
+class FakeSummarizer implements Summarizer {
+ calls: string[] = [];
+ async summarize(code: string, opts: { path: string }): Promise<string> {
+ this.calls.push(opts.path);
+ return `[[${opts.path}]] is a module in this repo. It contains: ${code.trim()}`;
+ }
+}
+
+function repoFixture(): string {
+ const dir = mkdtempSync(join(tmpdir(), "cge-repo-"));
+ mkdirSync(join(dir, "src"));
+ mkdirSync(join(dir, "lib"));
+ mkdirSync(join(dir, "node_modules", "leftpad"), { recursive: true });
+ writeFileSync(join(dir, "src", "a.ts"), "export const a = 1;\n");
+ writeFileSync(join(dir, "src", "b.ts"), "export const b = 2;\n");
+ writeFileSync(join(dir, "lib", "c.py"), "c = 3\n");
+ writeFileSync(join(dir, "main.ts"), "console.log('hi');\n");
+ writeFileSync(join(dir, "notes.md"), "# not code\n"); // ignored: not a code extension
+ writeFileSync(join(dir, "node_modules", "leftpad", "index.js"), "junk"); // ignored: skip dir
+ return dir;
+}
+
+test("ingest replaces a changed document with the same source", async () => {
+ const e = new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders() });
+ const v1 = await e.ingest("[[Alpha]] version one", { source: "docs/x.md" });
+ assert.equal(v1.skipped, false);
+
+ // Identical content → skipped, nothing replaced.
+ const again = await e.ingest("[[Alpha]] version one", { source: "docs/x.md" });
+ assert.equal(again.skipped, true);
+ assert.equal((await e.store.allDocuments()).length, 1);
+
+ // Changed content, same source → old document (and its chunks) replaced.
+ const v2 = await e.ingest("[[Alpha]] version two", { source: "docs/x.md" });
+ assert.equal(v2.skipped, false);
+ const docs = await e.store.allDocuments();
+ assert.equal(docs.length, 1);
+ assert.equal(docs[0].id, v2.documentId);
+ const chunks = await e.store.allChunks();
+ assert.ok(chunks.every((c) => c.documentId === v2.documentId));
+
+ // "inline" sources are exempt — unrelated pastes must not clobber each other.
+ await e.ingest("[[Beta]] one thing");
+ await e.ingest("[[Gamma]] another thing");
+ assert.equal((await e.store.allDocuments()).length, 3);
+ await e.close();
+});
+
+test("ingestRepo summarizes code files into module documents, incrementally", async () => {
+ const dir = repoFixture();
+ const summarizer = new FakeSummarizer();
+ const e = new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders(), summarizer });
+ try {
+ const r1 = await e.ingestRepo(dir);
+ // 4 code files (a.ts, b.ts, c.py, main.ts) — .md and node_modules excluded.
+ assert.equal(r1.files, 4);
+ assert.equal(r1.summarized, 4);
+ assert.equal(r1.cached, 0);
+ assert.deepEqual(r1.errors, []);
+ // 3 modules: src, lib, and (root) for main.ts.
+ assert.equal(r1.modules.length, 3);
+ assert.ok(r1.modules.every((m) => !m.skipped));
+ // Summaries flowed through extraction into the graph.
+ const nodes = await e.store.findNodesByName(join("src", "a.ts").toLowerCase());
+ assert.equal(nodes.length, 1);
+
+ // Unchanged re-run: all cache hits, all module docs skipped by hash dedup.
+ const r2 = await e.ingestRepo(dir);
+ assert.equal(r2.summarized, 0);
+ assert.equal(r2.cached, 4);
+ assert.ok(r2.modules.every((m) => m.skipped));
+ assert.equal(summarizer.calls.length, 4);
+
+ // Touch one file: only it is re-summarized; only its module re-ingests.
+ writeFileSync(join(dir, "src", "a.ts"), "export const a = 42;\n");
+ const r3 = await e.ingestRepo(dir);
+ assert.equal(r3.summarized, 1);
+ assert.equal(r3.cached, 3);
+ assert.equal(r3.modules.filter((m) => !m.skipped).length, 1);
+ // The changed module document replaced its predecessor (no stale duplicate).
+ const srcDocs = (await e.store.allDocuments()).filter((d) => d.source.endsWith("#src"));
+ assert.equal(srcDocs.length, 1);
+ } finally {
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+ }
+});