Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.
TypeScript ai-agentsanthropicclaude-codeclicode-graphcodexcontext-engineeringcursordeveloper-toolsgeminiknowledge-graphllmmcpmcp-serveropen-sourceopenaitree-sitter
Add code-repo summarization ingest, Claude Code hooks, and PR ingestion
anirudhkumar-nanonets29a387f364cf
diff --git a/README.md b/README.mdindex 9519de93..3eb70ef0 100644--- a/README.md+++ b/README.md@@ -35,7 +35,7 @@ context-graph-web# → http://localhost:4680```-Open the page, point **“Ingest a folder of documents”** at any folder of `.pdf` / `.md` / `.txt` files, and watch it become a knowledge graph. Then ask questions: answers come back with numbered citations, and every fact wears a **provenance receipt** — how many times it's been observed, how confident the graph is, and which sources say so. Use **“Teach it something”** to add a fact the docs don't capture, and watch the graph reinforce in real time.+Open the page, drop a folder of `.pdf` / `.md` / `.txt` files on the **“Add documents”** dropzone, and watch it become a knowledge graph. Then ask questions: answers come back with numbered citations, and every fact wears a **provenance receipt** — how many times it's been observed, how confident the graph is, and which sources say so. Use **“Add a fact”** to record something the docs don't capture, and watch the graph reinforce in real time.Everything runs on your machine (the server binds to localhost only). Set `OPENROUTER_API_KEY` to upgrade to LLM-written answers and cloud extraction; without it, retrieval and the graph stay fully local.@@ -102,6 +102,15 @@ context-graph ingest ./docs/*.md ./handbook.pdfcontext-graph ingest-dir ./docs # a whole folder, recursivelyecho "some notes" | context-graph ingest-text --title "Notes"+# Ingest a code repository — as summaries, not raw code (see "Code repositories")+context-graph repo .++# Ingest GitHub pull requests (title, description, review discussion) via the gh CLI+context-graph ingest-prs --state merged --limit 50++# Wire the graph into Claude Code (see "Claude Code hooks")+context-graph install-hooks+# Visualize the graph (interactive HTML; also --format json|mermaid)context-graph export --out graph.html && open graph.html@@ -155,7 +164,8 @@ Tools exposed:| `context_contribute` | Write a learning back into the shared graph || `context_ingest` | Ingest a document (raw text) into the graph || `context_ingest_file` | Ingest files from disk, including **PDFs** (parsed automatically) |-| `context_ingest_dir` | Ingest a whole **directory** of docs (PDF/MD/TXT), recursively |+| `context_ingest_dir` | Ingest a whole **directory** of docs (PDF/MD/TXT by default; pass `extensions` to widen), recursively |+| `context_ingest_repo` | Ingest a **code repository** as per-file prose summaries (never raw code); incremental across runs || `context_export` | Write the graph to an interactive **HTML** visualization || `context_sync` | Team sharing (git mode): import + re-merge the shared `graph.jsonl`, then write it back || `context_stats` | Report how much the graph currently holds |@@ -164,6 +174,44 @@ A natural agent workflow: **read context → do the task → contribute what you---+## Code repositories++`context-graph repo <dir>` (or the `context_ingest_repo` MCP tool) ingests a codebase — but deliberately **never feeds raw source code to the entity extractor**. Running LLM extraction over thousands of code chunks is the most expensive way to compute what `grep` and tree-sitter give agents for free, and it fills the graph with noisy, duplicated symbol entities. (Every serious code-context system — aider, Cursor, Sourcegraph, Claude Code — derives code *structure* statically, and uses LLMs only for prose.)++Instead, repo ingestion:++1. Walks the repo for code files (`.ts`, `.py`, `.go`, … — `node_modules`/`dist`/etc. and files over 1 MB are skipped).+2. Writes **one LLM prose summary per file**: purpose, key exports, dependencies, design decisions.+3. Groups summaries into one document per top-level directory and ingests those through the normal pipeline — so the graph gains *module-level* entities ("the chunker is paragraph-aware", "the CLI requires the GitHub CLI for PR ingestion") with provenance, not ten thousand function names.++It is **incremental**: summaries are cached by content hash in `repo-summaries.json` next to the db, so re-running after edits only re-summarizes changed files, and a changed file's module document *replaces* its predecessor rather than piling up stale duplicates. An unchanged repo re-ingests with **zero** LLM calls.++The same replace-on-change behavior now applies to every file ingest: re-ingesting a modified doc replaces the old version instead of leaving both in the graph.++For a ~30-file repo expect ~30 small summary calls on first run; the graph stays module-scale (a handful of documents) rather than chunk-soup-scale. Agents should still read raw code with their own tools — the graph holds what code can't say: intent, decisions, and gotchas.++## Claude Code hooks++`context-graph install-hooks` wires the graph into Claude Code via `.claude/settings.json` (merges with what's already there):++- **SessionStart** — every new session begins with relevant graph context (conventions, decisions, gotchas) auto-injected, plus a reminder that `context_contribute` exists.+- **Stop** — when the agent finishes, it's nudged once to record any durable learnings from the session before ending. Pass `--no-stop` to skip this hook if you find the nudge too chatty.++Both hooks stay silent when there's no graph yet, so installing them in a fresh project is harmless. Combined with the MCP server this closes the loop: sessions **start** with team knowledge and **end** by adding to it.++## GitHub pull requests++PR descriptions and review threads are where decisions actually get written down. `context-graph ingest-prs` pulls them into the graph via the [GitHub CLI](https://cli.github.com) (no extra auth needed if `gh` is logged in):++```bash+context-graph ingest-prs # merged PRs of the current repo (default 50)+context-graph ingest-prs -R owner/name -n 100 --state all+```++Each PR becomes one document (title, description, review discussion) with source `pr:owner/name#123`. Re-runs are cheap: unchanged PRs are skipped by hash, edited ones replace their old version.++---+## Team sharingA solo graph is one SQLite file. To share one across a team, pick a mode. All three keep the local-first default intact (team sync is **opt-in**).@@ -177,7 +225,7 @@ docker compose up -dcat ./data/share-link.txt # the share link, with access token```-The graph persists to `./data/graph.db`. Ingest documents two ways: click **Choose folder…** in the UI to upload a folder straight from your computer (works for remote teammates), or drop files into `./docs` on the server and ingest the mounted `/docs` path. When exposed beyond localhost the API requires an access token. It's generated once, saved to `./data/share-link.txt` (and printed to `docker compose logs`), and **stays stable across restarts** so shared links keep working. Pin your own with `CONTEXT_GRAPH_WEB_TOKEN`.+The graph persists to `./data/graph.db`. Ingest documents two ways: click **Choose a folder…** in the UI to upload a folder straight from your computer (works for remote teammates), or drop files into `./docs` on the server and ingest the mounted `/docs` path. When exposed beyond localhost the API requires an access token. It's generated once, saved to `./data/share-link.txt` (and printed to `docker compose logs`), and **stays stable across restarts** so shared links keep working. Pin your own with `CONTEXT_GRAPH_WEB_TOKEN`.No server to put it on? Run it on any machine and share it over [Tailscale](https://tailscale.com) without opening a port to the internet:@@ -271,12 +319,13 @@ new ContextGraphEngine({The engine depends only on small interfaces, so you can replace any piece:```ts-import { ContextGraphEngine, type Embedder, type Extractor, type GraphStore } from "context-graph-engine";+import { ContextGraphEngine, type Embedder, type Extractor, type GraphStore, type Summarizer } from "context-graph-engine";new ContextGraphEngine({- store: myCustomStore, // e.g. Postgres/pgvector- embedder: myCustomEmbedder, // any embedding model- extractor: myCustomExtractor // any LLM, or a rules-based extractor+ store: myCustomStore, // e.g. Postgres/pgvector+ embedder: myCustomEmbedder, // any embedding model+ extractor: myCustomExtractor, // any LLM, or a rules-based extractor+ summarizer: myCustomSummarizer // code-file summaries for ingestRepo});```@@ -291,6 +340,7 @@ class ContextGraphEngine {ingest(text: string, opts?: { title?; source? }): Promise<IngestResult>;ingestFile(path: string, opts?): Promise<IngestResult>; // .pdf parsed automaticallyingestDir(dir: string, opts?): Promise<IngestResult[]>; // recursive; PDF/MD/TXT+ ingestRepo(dir: string, opts?): Promise<RepoIngestResult>; // code → per-file summaries, incrementalread(query: string, opts?: { maxNodes?; maxChunks?; expand? }): Promise<ContextBundle>;contribute(learning: string, opts?: { agentId?; source? }): Promise<ContributeResult>;exportGraph(): GraphExport; // -> toHtml / toMermaiddiff --git a/src/ai/providers.ts b/src/ai/providers.tsindex 1dc79e84..f51a8b96 100644--- a/src/ai/providers.ts+++ b/src/ai/providers.ts@@ -1,5 +1,6 @@import type { Extraction } from "../graph/types.js";import type { GraphStore } from "../graph/store.js";+import type { Summarizer } from "./summarize.js";/** Turns text into embedding vectors for semantic search and dedup. */export interface Embedder {@@ -67,6 +68,8 @@ export interface EngineConfig {embedder?: Embedder;/** Override the extractor (defaults to OpenRouter, or local Ollama with no key). */extractor?: Extractor;+ /** Override the code summarizer used by repo ingestion (same fallbacks as extractor). */+ summarizer?: Summarizer;}/** Fully-resolved configuration with all defaults applied. */@@ -87,6 +90,7 @@ export interface ResolvedConfig {store?: GraphStore;embedder?: Embedder;extractor?: Extractor;+ summarizer?: Summarizer;}export const DEFAULTS = {@@ -136,5 +140,6 @@ export function resolveConfig(config: EngineConfig = {}): ResolvedConfig {store: config.store,embedder: config.embedder,extractor: config.extractor,+ summarizer: config.summarizer,};}diff --git a/src/ai/summarize.ts b/src/ai/summarize.tsnew file mode 100644index 00000000..72e02c2e--- /dev/null+++ b/src/ai/summarize.ts@@ -0,0 +1,103 @@+import OpenAI from "openai";++/**+ * Turns one source-code file into a short prose summary for the knowledge graph.+ *+ * Code repos are deliberately NOT fed through the per-chunk entity extractor:+ * static structure is better served by grep/tree-sitter in the consuming agent,+ * and LLM extraction over raw code produces noisy, duplicated entities. Instead+ * the repo ingest path summarizes each file into plain English (purpose, key+ * exports, dependencies, design decisions) and ingests those summaries — prose+ * the existing extraction pipeline is actually good at.+ */+export interface Summarizer {+ summarize(code: string, opts: { path: string }): Promise<string>;+}++const SYSTEM_PROMPT = `You document source code for a team knowledge base. Given one source file, write a compact plain-English summary covering:+1. The purpose of the file — what it exists to do.+2. The key exported functions/classes/types and what each is for.+3. Important dependencies: internal modules it builds on, external libraries or services it talks to.+4. Notable design decisions, constraints, or gotchas evident in the code.++Write 3-8 sentences of flowing prose. Name concrete identifiers (modules, classes, services) so they can become graph entities. No code blocks, no line-by-line narration, no filler.`;++/** Cap the code sent per file so a single giant file can't blow the context. */+const MAX_CODE_CHARS = 24_000;++function userContent(code: string, path: string): string {+ const clipped =+ code.length > MAX_CODE_CHARS+ ? `${code.slice(0, MAX_CODE_CHARS)}\n… (truncated at ${MAX_CODE_CHARS} characters)`+ : code;+ return `File: ${path}\n\n${clipped}`;+}++/** Summarizer backed by OpenRouter's OpenAI-compatible chat API. */+export class OpenRouterSummarizer implements Summarizer {+ private client: OpenAI;+ private model: string;++ constructor(apiKey: string, model: string, baseUrl = "https://openrouter.ai/api/v1") {+ this.client = new OpenAI({+ apiKey,+ baseURL: baseUrl,+ defaultHeaders: { "X-Title": "Context Graph Engine" },+ });+ this.model = model;+ }++ async summarize(code: string, opts: { path: string }): Promise<string> {+ const response = await this.client.chat.completions.create({+ model: this.model,+ temperature: 0,+ messages: [+ { role: "system", content: SYSTEM_PROMPT },+ { role: "user", content: userContent(code, opts.path) },+ ],+ });+ return response.choices[0]?.message?.content?.trim() ?? "";+ }+}++/** Summarizer backed by a locally running Ollama instance (key-free fallback). */+export class OllamaSummarizer implements Summarizer {+ private baseUrl: string;+ private model: string;++ constructor(model = "llama3.2", baseUrl = "http://localhost:11434") {+ this.model = model;+ this.baseUrl = baseUrl.replace(/\/$/, "");+ }++ async summarize(code: string, opts: { path: string }): Promise<string> {+ let res: Response;+ try {+ res = await fetch(`${this.baseUrl}/api/chat`, {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({+ model: this.model,+ stream: false,+ options: { temperature: 0 },+ messages: [+ { role: "system", content: SYSTEM_PROMPT },+ { role: "user", content: userContent(code, opts.path) },+ ],+ }),+ });+ } catch (err) {+ throw new Error(+ `Could not reach Ollama at ${this.baseUrl}. Start it (https://ollama.com) and pull a model ` ++ `(e.g. \`ollama pull ${this.model}\`), or set OPENROUTER_API_KEY to use cloud summarization. ` ++ `Underlying error: ${err instanceof Error ? err.message : String(err)}`,+ );+ }+ if (!res.ok) {+ const body = await res.text().catch(() => "");+ throw new Error(`Ollama returned ${res.status}: ${body}`);+ }+ const data = (await res.json()) as { message?: { content?: string } };+ return data.message?.content?.trim() ?? "";+ }+}diff --git a/src/cli.ts b/src/cli.tsindex a401bd1c..6ee9bd38 100644--- a/src/cli.ts+++ b/src/cli.ts@@ -12,8 +12,12 @@*/import "dotenv/config";import { Command } from "commander";-import { readFileSync, writeFileSync, existsSync } from "node:fs";+import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";+import { execFile } from "node:child_process";+import { promisify } from "node:util";+import { basename, dirname, join, resolve } from "node:path";import { ContextGraphEngine } from "./engine.js";+import { resolveConfig } from "./ai/providers.js";import { toHtml, toMermaid } from "./graph/export.js";const program = new Command();@@ -132,6 +136,220 @@ program}});+program+ .command("repo")+ .description(+ "Ingest a code repository as prose summaries (one LLM summary per file, grouped per top-level " ++ "directory). Raw code is never fed to the extractor. Incremental: only changed files are re-summarized.",+ )+ .argument("<dir>", "repository root")+ .option("-e, --extensions <exts...>", 'code extensions to include (e.g. ".ts" ".py")')+ .action(async (dir: string, opts: { extensions?: string[] }) => {+ const engine = engineFrom();+ try {+ const r = await engine.ingestRepo(dir, {+ extensions: opts.extensions,+ onProgress: ({ phase, index, total, file }) =>+ process.stderr.write(+ `\r${phase === "summarize" ? "summarizing" : "ingesting"} ${index + 1}/${total}: ${file.slice(0, 60).padEnd(60)}`,+ ),+ });+ process.stderr.write("\n");+ console.log(+ `✓ ${r.files} code files — ${r.summarized} summarized, ${r.cached} unchanged (cache hits)`,+ );+ for (const m of r.modules) {+ console.log(+ m.skipped+ ? `• ${m.title} — unchanged, skipped`+ : `✓ ${m.title} — ${m.chunks} chunks, +${m.nodesCreated} nodes, +${m.edgesCreated} edges`,+ );+ }+ for (const e of r.errors) console.error(`✗ ${e}`);+ } finally {+ await engine.close();+ }+ });++const ghExec = promisify(execFile);++/** Run a `gh` command and parse its JSON output. */+async function gh<T>(args: string[]): Promise<T> {+ try {+ const { stdout } = await ghExec("gh", args, { maxBuffer: 32_000_000 });+ return JSON.parse(stdout) as T;+ } catch (err) {+ const e = err as NodeJS.ErrnoException & { stderr?: string };+ if (e.code === "ENOENT") {+ throw new Error("GitHub CLI (`gh`) not found. Install it from https://cli.github.com and run `gh auth login`.");+ }+ throw new Error(`gh ${args.slice(0, 3).join(" ")} failed: ${e.stderr?.trim() || e.message}`);+ }+}++interface PrDetail {+ number: number;+ title: string;+ body: string;+ url: string;+ state: string;+ mergedAt: string | null;+ author: { login: string } | null;+ comments: Array<{ author: { login: string } | null; body: string }>;+ reviews: Array<{ author: { login: string } | null; state: string; body: string }>;+}++program+ .command("ingest-prs")+ .description("Ingest GitHub pull requests (title, description, review comments) via the gh CLI")+ .option("-R, --repo <owner/name>", "GitHub repository (default: the current directory's repo)")+ .option("-n, --limit <n>", "max PRs to ingest", (v) => parseInt(v, 10), 50)+ .option("-s, --state <state>", "open | closed | merged | all", "merged")+ .action(async (opts: { repo?: string; limit: number; state: string }) => {+ const repo =+ opts.repo ??+ (await gh<{ nameWithOwner: string }>(["repo", "view", "--json", "nameWithOwner"])).nameWithOwner;+ const list = await gh<Array<{ number: number }>>([+ "pr", "list", "--repo", repo, "--state", opts.state,+ "--limit", String(opts.limit), "--json", "number",+ ]);+ if (list.length === 0) {+ console.log(`No ${opts.state} PRs found in ${repo}.`);+ return;+ }+ const engine = engineFrom();+ try {+ for (const { number } of list) {+ const pr = await gh<PrDetail>([+ "pr", "view", String(number), "--repo", repo,+ "--json", "number,title,body,url,state,mergedAt,author,comments,reviews",+ ]);+ const discussion = [+ ...pr.reviews+ .filter((r) => r.body?.trim())+ .map((r) => `- ${r.author?.login ?? "unknown"} (review, ${r.state}): ${r.body.trim()}`),+ ...pr.comments+ .filter((c) => c.body?.trim())+ .map((c) => `- ${c.author?.login ?? "unknown"}: ${c.body.trim()}`),+ ];+ const text = [+ `# PR #${pr.number}: ${pr.title}`,+ `Repo: ${repo}. Author: ${pr.author?.login ?? "unknown"}. State: ${pr.state}${pr.mergedAt ? ` (merged ${pr.mergedAt.slice(0, 10)})` : ""}. ${pr.url}`,+ `## Description\n\n${pr.body?.trim() || "(no description)"}`,+ ...(discussion.length ? [`## Discussion\n\n${discussion.join("\n")}`] : []),+ ].join("\n\n");+ const r = await engine.ingest(text, {+ title: `PR #${pr.number}: ${pr.title}`,+ source: `pr:${repo}#${pr.number}`,+ });+ console.log(+ r.skipped+ ? `• PR #${pr.number} — already ingested, skipped`+ : `✓ PR #${pr.number}: ${pr.title} — +${r.nodesCreated} nodes, +${r.edgesCreated} edges`,+ );+ }+ } finally {+ await engine.close();+ }+ });++/** The two hook commands install-hooks wires into Claude Code. */+const SESSION_START_CMD = "context-graph hook session-start";+const STOP_CMD = "context-graph hook stop";++program+ .command("install-hooks")+ .description("Install Claude Code hooks: inject graph context at session start, nudge a contribution at stop")+ .option("--dir <path>", "project directory (default: current directory)")+ .option("--no-stop", "skip the Stop hook (the contribution nudge)")+ .action(async (opts: { dir?: string; stop: boolean }) => {+ const projectDir = resolve(opts.dir ?? ".");+ const settingsPath = join(projectDir, ".claude", "settings.json");+ let settings: Record<string, unknown> = {};+ if (existsSync(settingsPath)) {+ settings = JSON.parse(readFileSync(settingsPath, "utf8")) as Record<string, unknown>;+ }+ const hooks = (settings.hooks ??= {}) as Record<+ string,+ Array<{ matcher?: string; hooks: Array<{ type: string; command: string }> }>+ >;+ const addHook = (event: string, command: string): boolean => {+ const entries = (hooks[event] ??= []);+ if (entries.some((e) => e.hooks?.some((h) => h.command === command))) return false;+ entries.push({ hooks: [{ type: "command", command }] });+ return true;+ };+ const added: string[] = [];+ if (addHook("SessionStart", SESSION_START_CMD)) added.push("SessionStart");+ if (opts.stop && addHook("Stop", STOP_CMD)) added.push("Stop");+ mkdirSync(dirname(settingsPath), { recursive: true });+ writeFileSync(settingsPath, JSON.stringify(settings, null, 2) + "\n");+ console.log(+ added.length+ ? `✓ installed ${added.join(" + ")} hook${added.length > 1 ? "s" : ""} in ${settingsPath}`+ : `• hooks already installed in ${settingsPath}`,+ );+ console.log(" SessionStart injects relevant graph context; Stop nudges the agent to contribute learnings.");+ });++const hook = program+ .command("hook")+ .description("Claude Code hook entry points (installed by `install-hooks`; not usually run by hand)");++hook+ .command("session-start")+ .description("Print relevant graph context for injection at Claude Code session start")+ .action(async () => {+ // No graph yet → stay silent rather than erroring inside the hook.+ const cfg = resolveConfig({ dbPath: program.opts<{ db?: string }>().db });+ if (cfg.dbPath !== ":memory:" && !existsSync(cfg.dbPath)) return;+ const engine = engineFrom();+ try {+ const stats = await engine.stats();+ if (stats.nodes === 0) return;+ const project = basename(process.cwd());+ const bundle = await engine.read(+ `${project}: conventions, architecture decisions, gotchas, current work`,+ { maxNodes: 8, maxChunks: 4 },+ );+ if (bundle.nodes.length === 0 && bundle.chunks.length === 0) return;+ console.log("## Team context graph (auto-injected at session start)\n");+ console.log(bundle.prompt);+ console.log(+ "\nWhen you learn something durable this session (a decision, gotcha, or convention), " ++ "record it with the context_contribute MCP tool.",+ );+ } finally {+ await engine.close();+ }+ });++hook+ .command("stop")+ .description("Nudge the agent to contribute session learnings before finishing")+ .action(async () => {+ // Reads the Stop-hook JSON from stdin. `stop_hook_active` means this stop+ // was already extended by us once — let it through or we'd loop forever.+ let input: { stop_hook_active?: boolean } = {};+ try {+ input = JSON.parse(readFileSync(0, "utf8")) as { stop_hook_active?: boolean };+ } catch {+ /* no stdin — treat as a plain stop */+ }+ if (input.stop_hook_active) return;+ const cfg = resolveConfig({ dbPath: program.opts<{ db?: string }>().db });+ if (cfg.dbPath !== ":memory:" && !existsSync(cfg.dbPath)) return;+ console.log(+ JSON.stringify({+ decision: "block",+ reason:+ "Before finishing: if this session produced a durable, non-obvious learning (a decision made, " ++ "a gotcha hit, a convention discovered, a fix future agents should know about), record each one " ++ "with the context_contribute MCP tool. If nothing is worth recording, finish without contributing.",+ }),+ );+ });+program.command("export").description("Export the whole graph for viewing (interactive HTML, JSON, or Mermaid)")diff --git a/src/engine.ts b/src/engine.tsindex 24358a45..2eb9d7d7 100644--- a/src/engine.ts+++ b/src/engine.ts@@ -1,5 +1,5 @@-import { readFileSync, readdirSync } from "node:fs";-import { basename, dirname, join } from "node:path";+import { readFileSync, readdirSync, statSync, writeFileSync, existsSync, mkdirSync } from "node:fs";+import { basename, dirname, join, relative, resolve, sep } from "node:path";import { extractPdfText, isPdfPath } from "./ingest/pdf.js";import { buildGraphExport, type GraphExport } from "./graph/export.js";import type { GraphStore } from "./graph/store.js";@@ -9,6 +9,7 @@ import { resolveConfig } from "./ai/providers.js";import { OpenRouterExtractor } from "./ai/openrouter.js";import { OpenAIEmbedder } from "./ai/openai.js";import { LocalEmbedder, OllamaExtractor } from "./ai/local.js";+import { OllamaSummarizer, OpenRouterSummarizer, type Summarizer } from "./ai/summarize.js";import { chunkText } from "./ingest/chunker.js";import { mergeExtraction } from "./graph/merge.js";import { serializeGraph, importGraph, type ImportResult } from "./graph/serialize.js";@@ -50,6 +51,42 @@ export interface ContributeResult {edgesUpdated: number;}+/** Extensions treated as source code by {@link ContextGraphEngine.ingestRepo}. */+export const CODE_EXTENSIONS = [+ ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs",+ ".py", ".go", ".rs", ".java", ".kt", ".scala",+ ".rb", ".php", ".c", ".h", ".cpp", ".hpp", ".cc",+ ".cs", ".swift", ".sql", ".sh", ".proto",+];++export interface RepoIngestOptions {+ /** File extensions to treat as code. Default: {@link CODE_EXTENSIONS}. */+ extensions?: string[];+ /** Called before each file summary and each module ingest, for progress UIs. */+ onProgress?: (info: {+ phase: "summarize" | "ingest";+ index: number;+ total: number;+ file: string;+ }) => void;+}++export interface RepoIngestResult {+ /** Code files considered. */+ files: number;+ /** Files freshly summarized with an LLM call this run. */+ summarized: number;+ /** Files whose summary was reused from the content-hash cache. */+ cached: number;+ /** One ingest result per module (top-level directory) document. */+ modules: IngestResult[];+ /** Files that could not be read or summarized (path: reason). */+ errors: string[];+}++/** file path → { content hash, cached summary } */+type RepoSummaryCache = Record<string, { hash: string; summary: string }>;+/*** The Context Graph Engine.*@@ -62,6 +99,8 @@ export class ContextGraphEngine {private _store?: GraphStore;private _embedder?: Embedder;private _extractor?: Extractor;+ private _summarizer?: Summarizer;+ private _memRepoCache: RepoSummaryCache = {};constructor(config: EngineConfig = {}) {this.cfg = resolveConfig(config);@@ -111,6 +150,23 @@ export class ContextGraphEngine {return this._extractor;}+ private get summarizer(): Summarizer {+ if (!this._summarizer) {+ if (this.cfg.summarizer) {+ this._summarizer = this.cfg.summarizer;+ } else if (!this.cfg.forceLocal && this.cfg.openrouterApiKey) {+ this._summarizer = new OpenRouterSummarizer(+ this.cfg.openrouterApiKey,+ this.cfg.openrouterModel,+ this.cfg.openrouterBaseUrl,+ );+ } else {+ this._summarizer = new OllamaSummarizer(this.cfg.ollamaModel, this.cfg.ollamaBaseUrl);+ }+ }+ return this._summarizer;+ }+/*** Ingest a document from disk. PDFs (`.pdf`) are parsed to text automatically;* anything else is read as UTF-8. Title defaults to the file name.@@ -141,6 +197,16 @@ export class ContextGraphEngine {};}+ // Same source, new content: the file changed, so replace its stale+ // document(s) instead of piling up superseded chunks. "inline" is the+ // catch-all source for pasted text and is exempt — many unrelated+ // documents legitimately share it.+ if (source !== "inline") {+ for (const prior of await this.store.documentsBySource(source)) {+ await this.store.deleteDocument(prior.id);+ }+ }+const documentId = newId("doc");await this.store.insertDocument({ id: documentId, title, source, hash, createdAt: now });@@ -211,6 +277,119 @@ export class ContextGraphEngine {return results;}+ /**+ * Ingest a code repository as **summaries, not raw code**.+ *+ * Raw source is deliberately kept out of the extraction pipeline (LLM entity+ * extraction over code is noisy and expensive; agents already grep code+ * live). Instead each code file gets one LLM-written prose summary; the+ * summaries are grouped into one document per top-level directory and fed+ * through the normal {@link ingest} pipeline.+ *+ * Incremental: summaries are cached by file content hash (in+ * `repo-summaries.json` next to the db), so re-running after edits only+ * re-summarizes changed files, and unchanged module documents are skipped by+ * the ordinary document-hash dedup.+ */+ async ingestRepo(dir: string, opts: RepoIngestOptions = {}): Promise<RepoIngestResult> {+ const root = resolve(dir);+ const exts = opts.extensions ?? CODE_EXTENSIONS;+ const files = walkDir(root).filter((f) => exts.some((e) => f.toLowerCase().endsWith(e)));++ const cache = this.loadRepoCache();+ const result: RepoIngestResult = {+ files: files.length,+ summarized: 0,+ cached: 0,+ modules: [],+ errors: [],+ };++ // Phase 1: one summary per file, concurrent, content-hash cached.+ const summaries = await mapWithConcurrency(files, 8, async (file, i) => {+ const rel = relative(root, file);+ opts.onProgress?.({ phase: "summarize", index: i, total: files.length, file: rel });+ let code: string;+ try {+ code = readFileSync(file, "utf8");+ } catch (err) {+ result.errors.push(`${rel}: ${err instanceof Error ? err.message : String(err)}`);+ return undefined;+ }+ const hash = contentHash(code);+ const hit = cache[file];+ if (hit && hit.hash === hash) {+ result.cached++;+ return { rel, summary: hit.summary };+ }+ try {+ const summary = await this.summarizer.summarize(code, { path: rel });+ if (!summary) return undefined;+ cache[file] = { hash, summary };+ result.summarized++;+ return { rel, summary };+ } catch (err) {+ result.errors.push(`${rel}: ${err instanceof Error ? err.message : String(err)}`);+ return undefined;+ }+ });+ this.saveRepoCache(cache);++ // Phase 2: group summaries into one document per top-level directory and+ // run them through the normal ingest pipeline. The stable `repo:` source+ // makes re-ingestion replace a module's stale document when it changed.+ const repoName = basename(root);+ const modules = new Map<string, Array<{ rel: string; summary: string }>>();+ for (const s of summaries) {+ if (!s) continue;+ const module = s.rel.includes(sep) ? s.rel.split(sep)[0] : "(root)";+ (modules.get(module) ?? modules.set(module, []).get(module)!).push(s);+ }++ let i = 0;+ for (const [module, entries] of modules) {+ opts.onProgress?.({ phase: "ingest", index: i++, total: modules.size, file: module });+ const text = [+ `# Code summary: ${repoName}/${module}`,+ ...entries.map((e) => `## ${e.rel}\n\n${e.summary}`),+ ].join("\n\n");+ result.modules.push(+ await this.ingest(text, {+ title: `${repoName}/${module} (code summary)`,+ source: `repo:${root}#${module}`,+ }),+ );+ }+ return result;+ }++ /** Path of the file-summary cache, or undefined for in-memory graphs. */+ private get repoCachePath(): string | undefined {+ if (this.cfg.dbPath === ":memory:") return undefined;+ return join(dirname(this.cfg.dbPath), "repo-summaries.json");+ }++ private loadRepoCache(): RepoSummaryCache {+ const path = this.repoCachePath;+ if (!path) return this._memRepoCache;+ if (!existsSync(path)) return {};+ try {+ return JSON.parse(readFileSync(path, "utf8")) as RepoSummaryCache;+ } catch {+ return {};+ }+ }++ private saveRepoCache(cache: RepoSummaryCache): void {+ const path = this.repoCachePath;+ if (!path) {+ this._memRepoCache = cache;+ return;+ }+ mkdirSync(dirname(path), { recursive: true });+ writeFileSync(path, JSON.stringify(cache, null, 2));+ }+/** Read the graph for a query, returning a structured context bundle. */async read(query: string, opts: RetrieveOptions = {}): Promise<ContextBundle> {return retrieve(this.store, this.embedder, query, opts);@@ -327,14 +506,42 @@ export class ContextGraphEngine {}}-/** Recursively list all files under a directory (skips dot-directories). */+/** Directories that are dependency/build output, never knowledge. */+const SKIP_DIRS = new Set([+ "node_modules",+ "dist",+ "build",+ "out",+ "target",+ "vendor",+ "coverage",+ "__pycache__",+ "venv",+]);++/** Files above this size are generated/vendored in practice, not written docs or code. */+const MAX_FILE_BYTES = 1_000_000;++/**+ * Recursively list all files under a directory. Skips dot-directories,+ * dependency/build directories (node_modules, dist, …), and files over 1 MB.+ */function walkDir(dir: string): string[] {const out: string[] = [];for (const entry of readdirSync(dir, { withFileTypes: true })) {if (entry.name.startsWith(".")) continue;const full = join(dir, entry.name);- if (entry.isDirectory()) out.push(...walkDir(full));- else if (entry.isFile()) out.push(full);+ if (entry.isDirectory()) {+ if (SKIP_DIRS.has(entry.name)) continue;+ out.push(...walkDir(full));+ } else if (entry.isFile()) {+ try {+ if (statSync(full).size > MAX_FILE_BYTES) continue;+ } catch {+ continue;+ }+ out.push(full);+ }}return out;}diff --git a/src/graph/sqlite-store.ts b/src/graph/sqlite-store.tsindex 09d805fc..94a14b13 100644--- a/src/graph/sqlite-store.ts+++ b/src/graph/sqlite-store.ts@@ -160,6 +160,18 @@ export class SqliteStore implements GraphStore {return row ? this.rowToDocument(row) : undefined;}+ async documentsBySource(source: string): Promise<GraphDocument[]> {+ const rows = this.db+ .prepare(`SELECT * FROM documents WHERE source = ? ORDER BY created_at DESC`)+ .all(source) as Array<Record<string, unknown>>;+ return rows.map((r) => this.rowToDocument(r));+ }++ async deleteDocument(id: string): Promise<void> {+ // Chunks cascade via the document_id foreign key.+ this.db.prepare(`DELETE FROM documents WHERE id = ?`).run(id);+ }+async insertDocument(doc: GraphDocument): Promise<void> {this.db.prepare(diff --git a/src/graph/store.ts b/src/graph/store.tsindex 72cc0309..666f8da5 100644--- a/src/graph/store.ts+++ b/src/graph/store.ts@@ -22,7 +22,15 @@ export interface GraphStore {// --- documents ---getDocumentByHash(hash: string): Promise<GraphDocument | undefined>;getDocumentById(id: string): Promise<GraphDocument | undefined>;+ /** All documents recorded with the given source (path/URL/label), newest first. */+ documentsBySource(source: string): Promise<GraphDocument[]>;insertDocument(doc: GraphDocument): Promise<void>;+ /**+ * Delete a document and its chunks (e.g. when re-ingesting a changed file).+ * Nodes/edges extracted from it keep their observations — the grow-only CRDT+ * counters never shrink.+ */+ deleteDocument(id: string): Promise<void>;/** Every ingested document, newest first (for UI listings). */allDocuments(): Promise<GraphDocument[]>;diff --git a/src/index.ts b/src/index.tsindex 9dce2c89..7aa5daf4 100644--- a/src/index.ts+++ b/src/index.ts@@ -16,12 +16,14 @@* });* ```*/-export { ContextGraphEngine } from "./engine.js";+export { ContextGraphEngine, CODE_EXTENSIONS } from "./engine.js";export type {IngestOptions,IngestResult,ContributeOptions,ContributeResult,+ RepoIngestOptions,+ RepoIngestResult,} from "./engine.js";export type {@@ -67,3 +69,5 @@ export type {export { OpenRouterExtractor } from "./ai/openrouter.js";export { OpenAIEmbedder } from "./ai/openai.js";export { LocalEmbedder, OllamaExtractor } from "./ai/local.js";+export { OpenRouterSummarizer, OllamaSummarizer } from "./ai/summarize.js";+export type { Summarizer } from "./ai/summarize.js";diff --git a/src/mcp.ts b/src/mcp.tsindex 9a066857..d5782979 100644--- a/src/mcp.ts+++ b/src/mcp.ts@@ -127,9 +127,13 @@ server.registerTool("Ingest every supported document (.pdf, .md, .markdown, .txt) in a directory, recursively, into the shared context graph. Point this at a folder of docs and it builds the graph in one call.",inputSchema: {dir: z.string().describe("Absolute path to the directory of documents to ingest."),+ extensions: z+ .array(z.string())+ .optional()+ .describe('File extensions to include (e.g. [".md", ".rst"]). Default: .pdf .md .markdown .txt'),},},- async ({ dir }, extra) => {+ async ({ dir, extensions }, extra) => {// Report progress per file. If the client supplied a progressToken (and// resets its request timeout on progress, as Claude Code does), this keeps// long multi-file ingests from tripping the default 60s tool timeout.@@ -147,7 +151,7 @@ server.registerTool(});}: undefined;- const results = await engine.ingestDir(dir, { onProgress });+ const results = await engine.ingestDir(dir, { extensions, onProgress });if (results.length === 0) {return { content: [{ type: "text", text: `No supported files found under ${dir}.` }] };}@@ -161,6 +165,52 @@ server.registerTool(},);+server.registerTool(+ "context_ingest_repo",+ {+ title: "Ingest a code repository (as summaries)",+ description:+ "Ingest a code repository into the shared context graph. Each source file is summarized " ++ "into prose (purpose, key exports, dependencies, design decisions) and the summaries are " ++ "ingested — raw code is never fed to the extractor. Incremental: re-running only " ++ "re-summarizes files whose content changed.",+ inputSchema: {+ dir: z.string().describe("Absolute path to the repository root."),+ extensions: z+ .array(z.string())+ .optional()+ .describe('File extensions to treat as code (default: common languages, e.g. ".ts", ".py", ".go").'),+ },+ },+ async ({ dir, extensions }, extra) => {+ const progressToken = extra?._meta?.progressToken;+ const onProgress = progressToken+ ? ({ phase, index, total, file }: { phase: string; index: number; total: number; file: string }) => {+ void extra.sendNotification({+ method: "notifications/progress",+ params: {+ progressToken,+ progress: index,+ total,+ message: `${phase === "summarize" ? "Summarizing" : "Ingesting"} ${file} (${index + 1}/${total})`,+ },+ });+ }+ : undefined;+ const r = await engine.ingestRepo(dir, { extensions, onProgress });+ const lines = [+ `Repo ingested: ${r.files} code files → ${r.summarized} summarized, ${r.cached} unchanged (cache hits).`,+ ...r.modules.map((m) =>+ m.skipped+ ? `• ${m.title}: unchanged, skipped`+ : `✓ ${m.title}: ${m.chunks} chunks, +${m.nodesCreated} entities, +${m.edgesCreated} relationships`,+ ),+ ...r.errors.map((e) => `✗ ${e}`),+ ];+ return { content: [{ type: "text", text: lines.join("\n") }] };+ },+);+server.registerTool("context_export",{diff --git a/src/web.ts b/src/web.tsindex 6a67905b..6ea58d7b 100644--- a/src/web.ts+++ b/src/web.ts@@ -287,7 +287,10 @@ const server = createServer(async (req, res) => {if (req.method === "POST" && url.pathname === "/api/ingest-dir") {// Accepts a directory or a single file on the server's disk.- const { dir } = JSON.parse(await readBody(req)) as { dir?: string };+ const { dir, extensions } = JSON.parse(await readBody(req)) as {+ dir?: string;+ extensions?: string[];+ };if (!dir?.trim()) return json(res, 400, { error: "dir is required" });const abs = resolve(dir);if (!existsSync(abs)) {@@ -301,6 +304,7 @@ const server = createServer(async (req, res) => {try {const results = isDir? await engine.ingestDir(abs, {+ extensions,onProgress: (info) => emit({ type: "progress", ...info }),}): [await engine.ingestFile(abs)];diff --git a/test/repo.test.ts b/test/repo.test.tsnew file mode 100644index 00000000..a6a2b92b--- /dev/null+++ b/test/repo.test.ts@@ -0,0 +1,96 @@+import { test } from "node:test";+import assert from "node:assert/strict";+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { ContextGraphEngine, type Summarizer } from "../src/index.js";+import { fakeProviders } from "./helpers.ts";++/** Deterministic summarizer: counts calls, emits bracket markup the BracketExtractor understands. */+class FakeSummarizer implements Summarizer {+ calls: string[] = [];+ async summarize(code: string, opts: { path: string }): Promise<string> {+ this.calls.push(opts.path);+ return `[[${opts.path}]] is a module in this repo. It contains: ${code.trim()}`;+ }+}++function repoFixture(): string {+ const dir = mkdtempSync(join(tmpdir(), "cge-repo-"));+ mkdirSync(join(dir, "src"));+ mkdirSync(join(dir, "lib"));+ mkdirSync(join(dir, "node_modules", "leftpad"), { recursive: true });+ writeFileSync(join(dir, "src", "a.ts"), "export const a = 1;\n");+ writeFileSync(join(dir, "src", "b.ts"), "export const b = 2;\n");+ writeFileSync(join(dir, "lib", "c.py"), "c = 3\n");+ writeFileSync(join(dir, "main.ts"), "console.log('hi');\n");+ writeFileSync(join(dir, "notes.md"), "# not code\n"); // ignored: not a code extension+ writeFileSync(join(dir, "node_modules", "leftpad", "index.js"), "junk"); // ignored: skip dir+ return dir;+}++test("ingest replaces a changed document with the same source", async () => {+ const e = new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders() });+ const v1 = await e.ingest("[[Alpha]] version one", { source: "docs/x.md" });+ assert.equal(v1.skipped, false);++ // Identical content → skipped, nothing replaced.+ const again = await e.ingest("[[Alpha]] version one", { source: "docs/x.md" });+ assert.equal(again.skipped, true);+ assert.equal((await e.store.allDocuments()).length, 1);++ // Changed content, same source → old document (and its chunks) replaced.+ const v2 = await e.ingest("[[Alpha]] version two", { source: "docs/x.md" });+ assert.equal(v2.skipped, false);+ const docs = await e.store.allDocuments();+ assert.equal(docs.length, 1);+ assert.equal(docs[0].id, v2.documentId);+ const chunks = await e.store.allChunks();+ assert.ok(chunks.every((c) => c.documentId === v2.documentId));++ // "inline" sources are exempt — unrelated pastes must not clobber each other.+ await e.ingest("[[Beta]] one thing");+ await e.ingest("[[Gamma]] another thing");+ assert.equal((await e.store.allDocuments()).length, 3);+ await e.close();+});++test("ingestRepo summarizes code files into module documents, incrementally", async () => {+ const dir = repoFixture();+ const summarizer = new FakeSummarizer();+ const e = new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders(), summarizer });+ try {+ const r1 = await e.ingestRepo(dir);+ // 4 code files (a.ts, b.ts, c.py, main.ts) — .md and node_modules excluded.+ assert.equal(r1.files, 4);+ assert.equal(r1.summarized, 4);+ assert.equal(r1.cached, 0);+ assert.deepEqual(r1.errors, []);+ // 3 modules: src, lib, and (root) for main.ts.+ assert.equal(r1.modules.length, 3);+ assert.ok(r1.modules.every((m) => !m.skipped));+ // Summaries flowed through extraction into the graph.+ const nodes = await e.store.findNodesByName(join("src", "a.ts").toLowerCase());+ assert.equal(nodes.length, 1);++ // Unchanged re-run: all cache hits, all module docs skipped by hash dedup.+ const r2 = await e.ingestRepo(dir);+ assert.equal(r2.summarized, 0);+ assert.equal(r2.cached, 4);+ assert.ok(r2.modules.every((m) => m.skipped));+ assert.equal(summarizer.calls.length, 4);++ // Touch one file: only it is re-summarized; only its module re-ingests.+ writeFileSync(join(dir, "src", "a.ts"), "export const a = 42;\n");+ const r3 = await e.ingestRepo(dir);+ assert.equal(r3.summarized, 1);+ assert.equal(r3.cached, 3);+ assert.equal(r3.modules.filter((m) => !m.skipped).length, 1);+ // The changed module document replaced its predecessor (no stale duplicate).+ const srcDocs = (await e.store.allDocuments()).filter((d) => d.source.endsWith("#src"));+ assert.equal(srcDocs.length, 1);+ } finally {+ await e.close();+ rmSync(dir, { recursive: true, force: true });+ }+});