Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

Add auto-watch: folders keep syncing into an evolving graph

anirudhkumar-nanonetsb417c4226c22
diff --git a/README.md b/README.md
index 3eb70ef0..2232b6d1 100644
--- a/README.md
+++ b/README.md
@@ -102,6 +102,12 @@ context-graph ingest ./docs/*.md ./handbook.pdf
context-graph ingest-dir ./docs # a whole folder, recursively
echo "some notes" | context-graph ingest-text --title "Notes"
+# Keep the graph evolving as a folder changes (see "Auto-watch")
+context-graph watch ./docs # register + watch; re-run with no args to resume
+context-graph ingest-dir ./docs --watch # ingest now, register for watching
+context-graph watch-status # list registered folders
+context-graph unwatch ./docs
+
# Ingest a code repository — as summaries, not raw code (see "Code repositories")
context-graph repo .
@@ -166,6 +172,9 @@ Tools exposed:
| `context_ingest_file` | Ingest files from disk, including **PDFs** (parsed automatically) |
| `context_ingest_dir` | Ingest a whole **directory** of docs (PDF/MD/TXT by default; pass `extensions` to widen), recursively |
| `context_ingest_repo` | Ingest a **code repository** as per-file prose summaries (never raw code); incremental across runs |
+| `context_watch_dir` | Connect a folder for **auto-watching**: ingest it now and re-ingest files as they change (see "Auto-watch") |
+| `context_unwatch_dir` | Disconnect a watched folder (its knowledge stays in the graph) |
+| `context_watch_status` | List connected folders and the ingest queue |
| `context_export` | Write the graph to an interactive **HTML** visualization |
| `context_sync` | Team sharing (git mode): import + re-merge the shared `graph.jsonl`, then write it back |
| `context_stats` | Report how much the graph currently holds |
@@ -190,6 +199,26 @@ The same replace-on-change behavior now applies to every file ingest: re-ingesti
For a ~30-file repo expect ~30 small summary calls on first run; the graph stays module-scale (a handful of documents) rather than chunk-soup-scale. Agents should still read raw code with their own tools — the graph holds what code can't say: intent, decisions, and gotchas.
+## Auto-watch — an evolving graph
+
+Connect a folder once and the graph tracks it: files you add or edit are re-ingested automatically, so the graph evolves with your notes instead of freezing at ingest time.
+
+```bash
+context-graph watch ~/notes # catch up on the folder, then watch it live
+# … later, in a fresh shell — no args needed, registered folders resume:
+context-graph watch
+```
+
+How it behaves:
+
+- **Debounced and cheap.** Saves are coalesced (~1.5 s quiet time per file) and unchanged files are skipped by content hash, so a daemon restart over a big unchanged folder costs zero LLM calls. Edited files *replace* their old document rather than piling up copies.
+- **Append-only.** Deleting a file never removes knowledge already learned from it — the graph is a memory, not a mirror. The deletion is logged and nothing else happens.
+- **Same rules as `ingest-dir`.** Default extensions `.pdf .md .markdown .txt` (override with `--ext`), dot-dirs / `node_modules` / files over 1 MB are ignored.
+- **Registered folders** live in `watched-dirs.json` next to the db (machine-local — it never rides the git-synced graph). `ingest-dir --watch` registers while ingesting; `watch-status` / `unwatch` manage the list.
+- **One watcher per db.** Run a single `watch` daemon per graph; MCP sessions and the web UI can read concurrently while it runs.
+
+Elsewhere: the `context_watch_dir` MCP tool connects a folder for the life of the agent session (set `CONTEXT_GRAPH_AUTOWATCH=1` on the MCP server to auto-resume registered folders), and `CONTEXT_GRAPH_WATCH=1` does the same for `context-graph-web` — the UI picks up growth live.
+
## Claude Code hooks
`context-graph install-hooks` wires the graph into Claude Code via `.claude/settings.json` (merges with what's already there):
diff --git a/package-lock.json b/package-lock.json
index 6cb07793..9f4672e6 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -12,6 +12,7 @@
"@huggingface/transformers": "^3.8.1",
"@modelcontextprotocol/sdk": "^1.29.0",
"better-sqlite3": "^12.11.1",
+ "chokidar": "^4.0.3",
"commander": "^15.0.0",
"dotenv": "^17.4.2",
"openai": "^6.45.0",
@@ -1348,6 +1349,21 @@
"url": "https://github.com/sponsors/ljharb"
}
},
+ "node_modules/chokidar": {
+ "version": "4.0.3",
+ "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
+ "integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==",
+ "license": "MIT",
+ "dependencies": {
+ "readdirp": "^4.0.1"
+ },
+ "engines": {
+ "node": ">= 14.16.0"
+ },
+ "funding": {
+ "url": "https://paulmillr.com/funding/"
+ }
+ },
"node_modules/chownr": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
@@ -2714,6 +2730,19 @@
"node": ">= 6"
}
},
+ "node_modules/readdirp": {
+ "version": "4.1.2",
+ "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz",
+ "integrity": "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 14.18.0"
+ },
+ "funding": {
+ "type": "individual",
+ "url": "https://paulmillr.com/funding/"
+ }
+ },
"node_modules/require-from-string": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
diff --git a/package.json b/package.json
index 88e48116..25d381bb 100644
--- a/package.json
+++ b/package.json
@@ -50,6 +50,7 @@
"@huggingface/transformers": "^3.8.1",
"@modelcontextprotocol/sdk": "^1.29.0",
"better-sqlite3": "^12.11.1",
+ "chokidar": "^4.0.3",
"commander": "^15.0.0",
"dotenv": "^17.4.2",
"openai": "^6.45.0",
diff --git a/src/cli.ts b/src/cli.ts
index 6ee9bd38..8889cc67 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -17,6 +17,7 @@ import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { basename, dirname, join, resolve } from "node:path";
import { ContextGraphEngine } from "./engine.js";
+import { GraphWatcher, type WatchEvent } from "./watch.js";
import { resolveConfig } from "./ai/providers.js";
import { toHtml, toMermaid } from "./graph/export.js";
@@ -116,13 +117,13 @@ program
.command("ingest-dir")
.description("Ingest every supported doc (.pdf .md .txt) in a directory, recursively")
.argument("<dir>", "directory to ingest")
- .action(async (dir: string) => {
+ .option("-w, --watch", "also register the directory for auto-watching")
+ .action(async (dir: string, opts: { watch?: boolean }) => {
const engine = engineFrom();
try {
const results = await engine.ingestDir(dir);
if (results.length === 0) {
console.log(`No supported files found under ${dir}`);
- return;
}
for (const r of results) {
console.log(
@@ -131,6 +132,126 @@ program
: `✓ ${r.title} — ${r.chunks} chunks, +${r.nodesCreated} nodes, +${r.edgesCreated} edges`,
);
}
+ if (opts.watch) {
+ const w = engine.addWatchedDir(dir);
+ console.log(`✓ registered ${w.dir} for auto-watch — run \`context-graph watch\` to start the daemon`);
+ }
+ } finally {
+ await engine.close();
+ }
+ });
+
+/** One log line per watcher event, mirroring the ingest commands' ✓/•/✗ style. */
+function logWatchEvent(event: WatchEvent): void {
+ switch (event.type) {
+ case "watching":
+ console.log(`watching ${event.dir} (${event.files} files to catch up on)`);
+ break;
+ case "queued":
+ if (event.reason !== "initial") console.log(` queued ${event.file} (${event.reason})`);
+ break;
+ case "ingested":
+ console.log(
+ event.result.skipped
+ ? `• ${event.result.title} — unchanged, skipped`
+ : `✓ ${event.result.title} — ${event.result.chunks} chunks, +${event.result.nodesCreated} nodes, +${event.result.edgesCreated} edges`,
+ );
+ break;
+ case "deleted":
+ console.log(`• ${event.file} — deleted (knowledge kept, graph is append-only)`);
+ break;
+ case "error":
+ console.error(`✗ ${event.file}: ${event.error}`);
+ break;
+ }
+}
+
+program
+ .command("watch")
+ .description(
+ "Watch folders and keep the graph evolving: new and edited docs are re-ingested automatically. " +
+ "With no arguments, resumes previously registered folders. Ctrl-C to stop.",
+ )
+ .argument("[dirs...]", "directories to watch (registered for future runs unless --no-register)")
+ .option("-e, --ext <exts...>", 'extensions to ingest (default: ".pdf" ".md" ".markdown" ".txt")')
+ .option("--debounce <ms>", "quiet time after a save before re-ingesting", (v) => parseInt(v, 10), 1500)
+ .option("--no-register", "don't persist the given directories to the watch registry")
+ .option("--no-initial-scan", "skip the catch-up scan of existing files at startup")
+ .action(
+ async (
+ dirs: string[],
+ opts: { ext?: string[]; debounce: number; register: boolean; initialScan: boolean },
+ ) => {
+ const engine = engineFrom();
+ if (opts.register) {
+ for (const dir of dirs) engine.addWatchedDir(dir, opts.ext);
+ }
+ const targets = new Set(dirs.map((d) => resolve(d)));
+ for (const w of engine.listWatchedDirs()) targets.add(w.dir);
+ if (targets.size === 0) {
+ console.error("No watched folders. Run: context-graph watch <dir>");
+ await engine.close();
+ process.exit(1);
+ }
+
+ const watcher = new GraphWatcher(engine, {
+ extensions: opts.ext,
+ debounceMs: opts.debounce,
+ initialScan: opts.initialScan,
+ onEvent: logWatchEvent,
+ });
+ // The daemon owns the engine for its lifetime — closed on signal, not
+ // in a finally like the one-shot commands.
+ let stopping = false;
+ const stop = async () => {
+ if (stopping) return;
+ stopping = true;
+ console.log("\nstopping…");
+ await watcher.close();
+ await engine.close();
+ process.exit(0);
+ };
+ process.on("SIGINT", () => void stop());
+ process.on("SIGTERM", () => void stop());
+
+ for (const dir of targets) await watcher.add(dir);
+ await watcher.idle();
+ console.log("caught up — watching for changes (Ctrl-C to stop)");
+ },
+ );
+
+program
+ .command("watch-status")
+ .description("List folders registered for auto-watching")
+ .action(async () => {
+ const engine = engineFrom();
+ try {
+ const dirs = engine.listWatchedDirs();
+ if (dirs.length === 0) {
+ console.log("No watched folders. Register one with: context-graph watch <dir>");
+ return;
+ }
+ for (const w of dirs) {
+ const exts = w.extensions?.join(" ") ?? "default extensions";
+ console.log(`${w.dir} (${exts}, since ${w.addedAt.slice(0, 10)})`);
+ }
+ } finally {
+ await engine.close();
+ }
+ });
+
+program
+ .command("unwatch")
+ .description("Remove a folder from the auto-watch registry")
+ .argument("<dir>", "directory to stop watching")
+ .action(async (dir: string) => {
+ const engine = engineFrom();
+ try {
+ console.log(
+ engine.removeWatchedDir(dir)
+ ? `✓ removed ${resolve(dir)} from the watch registry`
+ : `• ${resolve(dir)} was not registered`,
+ );
} finally {
await engine.close();
}
diff --git a/src/engine.ts b/src/engine.ts
index 2eb9d7d7..73b7ce87 100644
--- a/src/engine.ts
+++ b/src/engine.ts
@@ -1,6 +1,7 @@
-import { readFileSync, readdirSync, statSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
+import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
import { basename, dirname, join, relative, resolve, sep } from "node:path";
import { extractPdfText, isPdfPath } from "./ingest/pdf.js";
+import { DOC_EXTENSIONS, walkDir } from "./ingest/fs.js";
import { buildGraphExport, type GraphExport } from "./graph/export.js";
import type { GraphStore } from "./graph/store.js";
import { SqliteStore } from "./graph/sqlite-store.js";
@@ -87,6 +88,19 @@ export interface RepoIngestResult {
/** file path → { content hash, cached summary } */
type RepoSummaryCache = Record<string, { hash: string; summary: string }>;
+/** A folder registered for auto-watching (see `context-graph watch`). */
+export interface WatchedDir {
+ /** Absolute path of the watched directory. */
+ dir: string;
+ /** Extensions to ingest; undefined means {@link DOC_EXTENSIONS}. */
+ extensions?: string[];
+ /** ISO timestamp of when the folder was first registered. */
+ addedAt: string;
+}
+
+/** absolute dir → registry entry (persisted as watched-dirs.json next to the db) */
+type WatchRegistry = Record<string, { extensions?: string[]; addedAt: string }>;
+
/**
* The Context Graph Engine.
*
@@ -101,6 +115,7 @@ export class ContextGraphEngine {
private _extractor?: Extractor;
private _summarizer?: Summarizer;
private _memRepoCache: RepoSummaryCache = {};
+ private _memWatchRegistry: WatchRegistry = {};
constructor(config: EngineConfig = {}) {
this.cfg = resolveConfig(config);
@@ -267,7 +282,7 @@ export class ContextGraphEngine {
onProgress?: (info: { index: number; total: number; file: string }) => void;
} = {},
): Promise<IngestResult[]> {
- const exts = opts.extensions ?? [".pdf", ".md", ".markdown", ".txt"];
+ const exts = opts.extensions ?? DOC_EXTENSIONS;
const files = walkDir(dir).filter((f) => exts.some((e) => f.toLowerCase().endsWith(e)));
const results: IngestResult[] = [];
for (let i = 0; i < files.length; i++) {
@@ -363,6 +378,66 @@ export class ContextGraphEngine {
return result;
}
+ /** Path of the watch registry, or undefined for in-memory graphs. */
+ private get watchRegistryPath(): string | undefined {
+ if (this.cfg.dbPath === ":memory:") return undefined;
+ return join(dirname(this.cfg.dbPath), "watched-dirs.json");
+ }
+
+ private loadWatchRegistry(): WatchRegistry {
+ const path = this.watchRegistryPath;
+ if (!path) return this._memWatchRegistry;
+ if (!existsSync(path)) return {};
+ try {
+ return JSON.parse(readFileSync(path, "utf8")) as WatchRegistry;
+ } catch {
+ return {};
+ }
+ }
+
+ private saveWatchRegistry(registry: WatchRegistry): void {
+ const path = this.watchRegistryPath;
+ if (!path) {
+ this._memWatchRegistry = registry;
+ return;
+ }
+ mkdirSync(dirname(path), { recursive: true });
+ writeFileSync(path, JSON.stringify(registry, null, 2));
+ }
+
+ /** Folders registered for auto-watching, in registration order. */
+ listWatchedDirs(): WatchedDir[] {
+ return Object.entries(this.loadWatchRegistry()).map(([dir, entry]) => ({ dir, ...entry }));
+ }
+
+ /**
+ * Register a folder for auto-watching (see `context-graph watch`). The
+ * registry lives in `watched-dirs.json` next to the db — machine-local
+ * paths deliberately stay out of the git-synced graph. Idempotent; the
+ * extensions of an existing entry are updated in place.
+ */
+ addWatchedDir(dir: string, extensions?: string[]): WatchedDir {
+ const abs = resolve(dir);
+ const registry = this.loadWatchRegistry();
+ const entry = {
+ extensions: extensions ?? registry[abs]?.extensions,
+ addedAt: registry[abs]?.addedAt ?? new Date().toISOString(),
+ };
+ registry[abs] = entry;
+ this.saveWatchRegistry(registry);
+ return { dir: abs, ...entry };
+ }
+
+ /** Unregister a watched folder. Returns false if it wasn't registered. */
+ removeWatchedDir(dir: string): boolean {
+ const abs = resolve(dir);
+ const registry = this.loadWatchRegistry();
+ if (!(abs in registry)) return false;
+ delete registry[abs];
+ this.saveWatchRegistry(registry);
+ return true;
+ }
+
/** Path of the file-summary cache, or undefined for in-memory graphs. */
private get repoCachePath(): string | undefined {
if (this.cfg.dbPath === ":memory:") return undefined;
@@ -506,46 +581,6 @@ export class ContextGraphEngine {
}
}
-/** Directories that are dependency/build output, never knowledge. */
-const SKIP_DIRS = new Set([
- "node_modules",
- "dist",
- "build",
- "out",
- "target",
- "vendor",
- "coverage",
- "__pycache__",
- "venv",
-]);
-
-/** Files above this size are generated/vendored in practice, not written docs or code. */
-const MAX_FILE_BYTES = 1_000_000;
-
-/**
- * Recursively list all files under a directory. Skips dot-directories,
- * dependency/build directories (node_modules, dist, …), and files over 1 MB.
- */
-function walkDir(dir: string): string[] {
- const out: string[] = [];
- for (const entry of readdirSync(dir, { withFileTypes: true })) {
- if (entry.name.startsWith(".")) continue;
- const full = join(dir, entry.name);
- if (entry.isDirectory()) {
- if (SKIP_DIRS.has(entry.name)) continue;
- out.push(...walkDir(full));
- } else if (entry.isFile()) {
- try {
- if (statSync(full).size > MAX_FILE_BYTES) continue;
- } catch {
- continue;
- }
- out.push(full);
- }
- }
- return out;
-}
-
/** Run `fn` over `items` with at most `limit` in flight at once, preserving order. */
async function mapWithConcurrency<T, R>(
items: T[],
diff --git a/src/index.ts b/src/index.ts
index 7aa5daf4..0892216d 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -24,8 +24,13 @@ export type {
ContributeResult,
RepoIngestOptions,
RepoIngestResult,
+ WatchedDir,
} from "./engine.js";
+export { DOC_EXTENSIONS } from "./ingest/fs.js";
+export { GraphWatcher } from "./watch.js";
+export type { WatchOptions, WatchEvent } from "./watch.js";
+
export type {
EngineConfig,
ResolvedConfig,
diff --git a/src/ingest/fs.ts b/src/ingest/fs.ts
new file mode 100644
index 00000000..c58749cf
--- /dev/null
+++ b/src/ingest/fs.ts
@@ -0,0 +1,95 @@
+/**
+ * Filesystem walking/filtering rules shared by directory ingestion and the
+ * auto-watch daemon, so both always agree on which files are knowledge.
+ */
+import { readdirSync, statSync } from "node:fs";
+import { join, relative, sep } from "node:path";
+
+/** Extensions treated as documents by `ingestDir` and the watcher. */
+export const DOC_EXTENSIONS = [".pdf", ".md", ".markdown", ".txt"];
+
+/** Directories that are dependency/build output, never knowledge. */
+export const SKIP_DIRS = new Set([
+ "node_modules",
+ "dist",
+ "build",
+ "out",
+ "target",
+ "vendor",
+ "coverage",
+ "__pycache__",
+ "venv",
+]);
+
+/** Files above this size are generated/vendored in practice, not written docs or code. */
+export const MAX_FILE_BYTES = 1_000_000;
+
+/**
+ * Recursively list all files under a directory. Skips dot-directories,
+ * dependency/build directories (node_modules, dist, …), and files over 1 MB.
+ */
+export function walkDir(dir: string): string[] {
+ const out: string[] = [];
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
+ if (entry.name.startsWith(".")) continue;
+ const full = join(dir, entry.name);
+ if (entry.isDirectory()) {
+ if (SKIP_DIRS.has(entry.name)) continue;
+ out.push(...walkDir(full));
+ } else if (entry.isFile()) {
+ try {
+ if (statSync(full).size > MAX_FILE_BYTES) continue;
+ } catch {
+ continue;
+ }
+ out.push(full);
+ }
+ }
+ return out;
+}
+
+/** True if any path segment is a dot-entry or a SKIP_DIRS directory. */
+function hasSkippedSegment(path: string): boolean {
+ return path
+ .split(sep)
+ .some((seg) => seg.length > 0 && (seg.startsWith(".") || SKIP_DIRS.has(seg)));
+}
+
+/**
+ * True if this path passes the same rules {@link walkDir} applies below
+ * `root`: a matching extension, no dot-file/dot-dir or SKIP_DIRS segment
+ * under the root, and at most 1 MB. The root itself is exempt from the
+ * segment rules — watching `~/.notes` is legitimate, just as `walkDir` never
+ * inspects the directory it was pointed at.
+ */
+export function isIngestablePath(
+ path: string,
+ root: string,
+ extensions: string[] = DOC_EXTENSIONS,
+): boolean {
+ if (hasSkippedSegment(relative(root, path))) return false;
+ const lower = path.toLowerCase();
+ if (!extensions.some((e) => lower.endsWith(e))) return false;
+ try {
+ if (statSync(path).size > MAX_FILE_BYTES) return false;
+ } catch {
+ return false;
+ }
+ return true;
+}
+
+/**
+ * Predicate for a file watcher's `ignored` option: true for dot/SKIP_DIRS
+ * entries below `root` (so they are never descended into) and for oversized
+ * files. Extension filtering is left to the event handler — watchers hand us
+ * stats only sometimes, and directories have no extension to test.
+ */
+export function shouldIgnorePath(
+ path: string,
+ root: string,
+ stats?: { isDirectory(): boolean; size: number },
+): boolean {
+ if (hasSkippedSegment(relative(root, path))) return true;
+ if (stats && !stats.isDirectory() && stats.size > MAX_FILE_BYTES) return true;
+ return false;
+}
diff --git a/src/mcp.ts b/src/mcp.ts
index d5782979..28a37dda 100644
--- a/src/mcp.ts
+++ b/src/mcp.ts
@@ -12,10 +12,32 @@ import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"
import { z } from "zod";
import { writeFileSync, readFileSync, existsSync } from "node:fs";
import { ContextGraphEngine } from "./engine.js";
+import { GraphWatcher, type WatchEvent } from "./watch.js";
import { toHtml } from "./graph/export.js";
const engine = new ContextGraphEngine();
+// One lazily-created watcher for the life of this MCP process. Folders added
+// via context_watch_dir keep updating the graph in the background for as long
+// as the client session (and thus this process) lives. context_watch_dir
+// temporarily points `progressSink` at its own progress notifications while
+// the catch-up scan runs.
+let watcher: GraphWatcher | undefined;
+let progressSink: ((event: WatchEvent) => void) | undefined;
+
+function getWatcher(): GraphWatcher {
+ return (watcher ??= new GraphWatcher(engine, {
+ onEvent: (event) => {
+ // stderr only — stdout is the JSON-RPC channel.
+ if (event.type === "error") console.error(`watch: ✗ ${event.file}: ${event.error}`);
+ else if (event.type === "ingested" && !event.result.skipped) {
+ console.error(`watch: ✓ ${event.result.title} (+${event.result.nodesCreated} entities)`);
+ }
+ progressSink?.(event);
+ },
+ }));
+}
+
const server = new McpServer({
name: "context-graph-engine",
version: "0.1.0",
@@ -211,6 +233,126 @@ server.registerTool(
},
);
+server.registerTool(
+ "context_watch_dir",
+ {
+ title: "Watch a folder (evolving graph)",
+ description:
+ "Connect a folder to the graph and keep it evolving: existing documents are ingested now, " +
+ "and any file added or edited later is re-ingested automatically for as long as this session " +
+ "lives. The folder is registered, so `context-graph watch` (or CONTEXT_GRAPH_AUTOWATCH=1) " +
+ "resumes it in future sessions. Append-only: deleting a file never removes learned knowledge.",
+ inputSchema: {
+ dir: z.string().describe("Absolute path to the folder to watch."),
+ extensions: z
+ .array(z.string())
+ .optional()
+ .describe('File extensions to include (e.g. [".md", ".rst"]). Default: .pdf .md .markdown .txt'),
+ },
+ },
+ async ({ dir, extensions }, extra) => {
+ const w = getWatcher();
+ // Extensions are fixed per watcher process; a custom set is recorded in
+ // the registry and takes effect when a daemon starts from it.
+ engine.addWatchedDir(dir, extensions);
+
+ const progressToken = extra?._meta?.progressToken;
+ const lines: string[] = [];
+ let ingested = 0;
+ let queued = 0;
+ progressSink = (event) => {
+ if (event.type === "queued") queued++;
+ if (event.type === "ingested") {
+ ingested++;
+ lines.push(
+ event.result.skipped
+ ? `• ${event.result.title}: unchanged, skipped`
+ : `✓ ${event.result.title}: +${event.result.nodesCreated} entities, +${event.result.edgesCreated} relationships`,
+ );
+ if (progressToken) {
+ void extra.sendNotification({
+ method: "notifications/progress",
+ params: {
+ progressToken,
+ progress: ingested,
+ total: queued,
+ message: `Ingesting ${event.file.split("/").pop()} (${ingested}/${queued})`,
+ },
+ });
+ }
+ }
+ if (event.type === "error") lines.push(`✗ ${event.file}: ${event.error}`);
+ };
+ try {
+ await w.add(dir);
+ await w.idle();
+ } finally {
+ progressSink = undefined;
+ }
+ lines.push(
+ `\nWatching ${dir} — new and edited files will be ingested automatically while this session lives.`,
+ );
+ return { content: [{ type: "text", text: lines.join("\n") }] };
+ },
+);
+
+server.registerTool(
+ "context_unwatch_dir",
+ {
+ title: "Stop watching a folder",
+ description:
+ "Stop auto-ingesting a folder and remove it from the watch registry. Knowledge already " +
+ "learned from it stays in the graph.",
+ inputSchema: {
+ dir: z.string().describe("Absolute path of the watched folder to disconnect."),
+ },
+ },
+ async ({ dir }) => {
+ await watcher?.remove(dir);
+ const removed = engine.removeWatchedDir(dir);
+ return {
+ content: [
+ {
+ type: "text",
+ text: removed
+ ? `Stopped watching ${dir}. Its knowledge remains in the graph.`
+ : `${dir} was not being watched.`,
+ },
+ ],
+ };
+ },
+);
+
+server.registerTool(
+ "context_watch_status",
+ {
+ title: "Watched folders",
+ description:
+ "List folders connected for auto-watching: registered folders, which are live in this session, and the ingest queue size.",
+ inputSchema: {},
+ },
+ async () => {
+ const registered = engine.listWatchedDirs();
+ const live = new Set(watcher?.dirs() ?? []);
+ if (registered.length === 0 && live.size === 0) {
+ return {
+ content: [
+ { type: "text", text: "No watched folders. Connect one with context_watch_dir." },
+ ],
+ };
+ }
+ const lines = registered.map(
+ (wd) => `${live.has(wd.dir) ? "● live" : "○ registered"} ${wd.dir}`,
+ );
+ for (const dir of live) {
+ if (!registered.some((wd) => wd.dir === dir)) lines.push(`● live (unregistered) ${dir}`);
+ }
+ const pending = watcher?.pendingCount() ?? 0;
+ if (pending > 0) lines.push(`\n${pending} file(s) queued for ingestion.`);
+ return { content: [{ type: "text", text: lines.join("\n") }] };
+ },
+);
+
server.registerTool(
"context_export",
{
@@ -296,6 +438,22 @@ async function main(): Promise<void> {
await server.connect(transport);
// Log to stderr so we don't corrupt the stdio JSON-RPC channel.
console.error("context-graph-engine MCP server running on stdio");
+
+ // Opt-in: resume registered folders in the background. Off by default —
+ // every client session spawns its own MCP process, and silent LLM spend on
+ // file saves should be something the user chose.
+ if (process.env.CONTEXT_GRAPH_AUTOWATCH === "1") {
+ const dirs = engine.listWatchedDirs();
+ if (dirs.length > 0) {
+ const w = getWatcher();
+ for (const wd of dirs) {
+ w.add(wd.dir).catch((err) =>
+ console.error(`watch: ✗ ${wd.dir}: ${err instanceof Error ? err.message : String(err)}`),
+ );
+ }
+ console.error(`auto-watching ${dirs.length} registered folder(s)`);
+ }
+ }
}
main().catch((err) => {
diff --git a/src/watch.ts b/src/watch.ts
new file mode 100644
index 00000000..2db7ec0d
--- /dev/null
+++ b/src/watch.ts
@@ -0,0 +1,221 @@
+/**
+ * Auto-watch: keep the graph evolving as connected folders change.
+ *
+ * A {@link GraphWatcher} watches document folders and re-ingests files as they
+ * are added or edited, so the graph tracks the folder without manual re-runs
+ * of `ingest-dir`. Deliberately append-only: deleting a file never removes
+ * knowledge already learned from it (the grow-only observation model is
+ * untouched); the deletion is surfaced as an event and nothing more.
+ *
+ * Rapid saves are absorbed by a per-file debounce, and files are ingested by a
+ * single serial worker — each ingest already fans out to concurrent LLM
+ * extraction internally, so one document at a time keeps spend predictable.
+ */
+import { watch, type FSWatcher } from "chokidar";
+import type { Stats } from "node:fs";
+import { resolve, sep } from "node:path";
+import type { ContextGraphEngine, IngestResult } from "./engine.js";
+import { DOC_EXTENSIONS, isIngestablePath, shouldIgnorePath, walkDir } from "./ingest/fs.js";
+
+export interface WatchOptions {
+ /** Extensions to ingest. Default: {@link DOC_EXTENSIONS}. */
+ extensions?: string[];
+ /** Quiet time after a file's last event before it is ingested. Default 1500ms. */
+ debounceMs?: number;
+ /** Catch up on the folder's current contents when it is added. Default true. */
+ initialScan?: boolean;
+ /** Called for every lifecycle event, for progress UIs and logging. */
+ onEvent?: (event: WatchEvent) => void;
+}
+
+export type WatchEvent =
+ | { type: "watching"; dir: string; files: number }
+ | { type: "queued"; file: string; reason: "add" | "change" | "initial" }
+ | { type: "ingested"; file: string; result: IngestResult }
+ | { type: "deleted"; file: string }
+ | { type: "error"; file: string; error: string };
+
+/**
+ * Watches folders and streams their document changes into an engine.
+ *
+ * One watcher process per db is the supported mode — concurrent graph writes
+ * from multiple processes are safe (WAL) but their merges are not transactional.
+ */
+export class GraphWatcher {
+ private watchers = new Map<string, FSWatcher>();
+ private debounces = new Map<string, NodeJS.Timeout>();
+ private pending = new Set<string>();
+ private inFlight?: string;
+ private working = false;
+ private closed = false;
+ private idleResolvers: Array<() => void> = [];
+
+ constructor(
+ private engine: ContextGraphEngine,
+ private opts: WatchOptions = {},
+ ) {}
+
+ private get extensions(): string[] {
+ return this.opts.extensions ?? DOC_EXTENSIONS;
+ }
+
+ private get debounceMs(): number {
+ return this.opts.debounceMs ?? 1500;
+ }
+
+ private emit(event: WatchEvent): void {
+ this.opts.onEvent?.(event);
+ }
+
+ /** Watch a folder. Resolves once watching is established (and the initial catch-up scan is queued). */
+ async add(dir: string): Promise<void> {
+ const root = resolve(dir);
+ if (this.watchers.has(root)) return;
+
+ const watcher = watch(root, {
+ ignored: (path: string, stats?: Stats) => shouldIgnorePath(path, root, stats),
+ ignoreInitial: true,
+ // Don't ingest a file mid-write: wait for its size to hold still, so a
+ // large PDF that takes a moment to save arrives whole.
+ awaitWriteFinish: { stabilityThreshold: 500, pollInterval: 100 },
+ });
+ this.watchers.set(root, watcher);
+
+ const onFile = (reason: "add" | "change") => (path: string) => {
+ if (this.closed || !isIngestablePath(path, root, this.extensions)) return;
+ this.debounce(path, reason);
+ };
+ watcher.on("add", onFile("add"));
+ watcher.on("change", onFile("change"));
+ watcher.on("unlink", (path: string) => {
+ // Append-only by design: the file is gone but what it taught us stays.
+ this.debounces.delete(path);
+ this.pending.delete(path);
+ this.emit({ type: "deleted", file: path });
+ this.checkIdle();
+ });
+ watcher.on("error", (err) => {
+ this.emit({ type: "error", file: root, error: err instanceof Error ? err.message : String(err) });
+ });
+
+ await new Promise<void>((res, rej) => {
+ watcher.once("ready", res);
+ watcher.once("error", rej);
+ });
+
+ let files = 0;
+ if (this.opts.initialScan !== false) {
+ for (const file of walkDir(root)) {
+ if (!isIngestablePath(file, root, this.extensions)) continue;
+ files++;
+ this.pending.add(file);
+ this.emit({ type: "queued", file, reason: "initial" });
+ }
+ this.kick();
+ }
+ this.emit({ type: "watching", dir: root, files });
+ }
+
+ /** Stop watching a folder and drop its queued (not in-flight) files. */
+ async remove(dir: string): Promise<void> {
+ const root = resolve(dir);
+ const watcher = this.watchers.get(root);
+ if (!watcher) return;
+ this.watchers.delete(root);
+ await watcher.close();
+ const under = (path: string) => path === root || path.startsWith(root + sep);
+ for (const [path, timer] of this.debounces) {
+ if (under(path)) {
+ clearTimeout(timer);
+ this.debounces.delete(path);
+ }
+ }
+ for (const path of this.pending) {
+ if (under(path)) this.pending.delete(path);
+ }
+ this.checkIdle();
+ }
+
+ /** Folders currently being watched. */
+ dirs(): string[] {
+ return [...this.watchers.keys()];
+ }
+
+ /** Files queued or being ingested right now. */
+ pendingCount(): number {
+ return this.pending.size + this.debounces.size + (this.inFlight ? 1 : 0);
+ }
+
+ /** Resolves once every queued and debouncing file has been ingested. */
+ idle(): Promise<void> {
+ if (this.pendingCount() === 0) return Promise.resolve();
+ return new Promise((res) => this.idleResolvers.push(res));
+ }
+
+ /** Stop watching everything, drop queued work, and wait out the in-flight ingest. */
+ async close(): Promise<void> {
+ this.closed = true;
+ await Promise.all([...this.watchers.values()].map((w) => w.close()));
+ this.watchers.clear();
+ for (const timer of this.debounces.values()) clearTimeout(timer);
+ this.debounces.clear();
+ this.pending.clear();
+ await this.idle();
+ }
+
+ private debounce(path: string, reason: "add" | "change"): void {
+ const existing = this.debounces.get(path);
+ if (existing) clearTimeout(existing);
+ else this.emit({ type: "queued", file: path, reason });
+ this.debounces.set(
+ path,
+ setTimeout(() => {
+ this.debounces.delete(path);
+ if (this.closed) return this.checkIdle();
+ this.pending.add(path);
+ this.kick();
+ }, this.debounceMs),
+ );
+ }
+
+ private checkIdle(): void {
+ if (this.pendingCount() > 0) return;
+ const resolvers = this.idleResolvers;
+ this.idleResolvers = [];
+ for (const res of resolvers) res();
+ }
+
+ private kick(): void {
+ if (this.working) return;
+ this.working = true;
+ void this.work().finally(() => {
+ this.working = false;
+ this.checkIdle();
+ });
+ }
+
+ private async work(): Promise<void> {
+ for (let next = this.takeNext(); next; next = this.takeNext()) {
+ this.inFlight = next;
+ try {
+ const result = await this.engine.ingestFile(next);
+ this.emit({ type: "ingested", file: next, result });
+ } catch (err) {
+ // The file vanished between the event and the ingest — that's a delete.
+ if ((err as NodeJS.ErrnoException).code === "ENOENT") {
+ this.emit({ type: "deleted", file: next });
+ } else {
+ this.emit({ type: "error", file: next, error: err instanceof Error ? err.message : String(err) });
+ }
+ } finally {
+ this.inFlight = undefined;
+ }
+ }
+ }
+
+ private takeNext(): string | undefined {
+ const next = this.pending.values().next().value as string | undefined;
+ if (next !== undefined) this.pending.delete(next);
+ return next;
+ }
+}
diff --git a/src/web.ts b/src/web.ts
index 6ea58d7b..91f7edeb 100644
--- a/src/web.ts
+++ b/src/web.ts
@@ -22,6 +22,7 @@ import { networkInterfaces } from "node:os";
import { resolve, dirname, join } from "node:path";
import OpenAI from "openai";
import { ContextGraphEngine } from "./engine.js";
+import { GraphWatcher } from "./watch.js";
import { resolveConfig } from "./ai/providers.js";
import { extractPdfTextFromData, isPdfPath } from "./ingest/pdf.js";
@@ -262,6 +263,15 @@ const server = createServer(async (req, res) => {
return;
}
+ if (req.method === "GET" && url.pathname === "/api/watched") {
+ json(res, 200, {
+ registered: engine.listWatchedDirs(),
+ live: webWatcher?.dirs() ?? [],
+ pending: webWatcher?.pendingCount() ?? 0,
+ });
+ return;
+ }
+
if (req.method === "POST" && url.pathname === "/api/ask") {
const { question } = JSON.parse(await readBody(req)) as { question?: string };
if (!question?.trim()) return json(res, 400, { error: "question is required" });
@@ -286,10 +296,13 @@ const server = createServer(async (req, res) => {
}
if (req.method === "POST" && url.pathname === "/api/ingest-dir") {
- // Accepts a directory or a single file on the server's disk.
- const { dir, extensions } = JSON.parse(await readBody(req)) as {
+ // Accepts a directory or a single file on the server's disk. With
+ // watch:true the directory is also registered and watched in-process,
+ // so future edits keep flowing into the graph while the server runs.
+ const { dir, extensions, watch } = JSON.parse(await readBody(req)) as {
dir?: string;
extensions?: string[];
+ watch?: boolean;
};
if (!dir?.trim()) return json(res, 400, { error: "dir is required" });
const abs = resolve(dir);
@@ -308,7 +321,17 @@ const server = createServer(async (req, res) => {
onProgress: (info) => emit({ type: "progress", ...info }),
})
: [await engine.ingestFile(abs)];
- emit({ type: "done", results, stats: await engine.stats() });
+ let watching = false;
+ if (watch && isDir) {
+ engine.addWatchedDir(abs, extensions);
+ // The just-finished ingest makes the watcher's catch-up scan a
+ // pure hash-dedup pass — no LLM calls. Don't hold the response.
+ void ensureWebWatcher()
+ .add(abs)
+ .catch((e) => console.error(`watch: ✗ ${abs}: ${e instanceof Error ? e.message : String(e)}`));
+ watching = true;
+ }
+ emit({ type: "done", results, watching, stats: await engine.stats() });
} catch (e) {
emit({ type: "error", error: e instanceof Error ? e.message : String(e) });
}
@@ -320,9 +343,13 @@ const server = createServer(async (req, res) => {
// One uploaded file at a time (the browser folder picker can't hand us a
// server path, so it ships file contents instead). Text files arrive as
// `text`; PDFs and other binaries arrive base64-encoded in `dataBase64`.
+ // `relPath` (folder/sub/file.md) keys the document when present, so two
+ // same-named files in different folders don't replace each other and a
+ // synced folder's re-uploads update the right document.
// 60 MB cap leaves room for a large PDF plus base64's ~33% overhead.
- const { name, text, dataBase64 } = JSON.parse(await readBody(req, 60_000_000)) as {
+ const { name, relPath, text, dataBase64 } = JSON.parse(await readBody(req, 60_000_000)) as {
name?: string;
+ relPath?: string;
text?: string;
dataBase64?: string;
};
@@ -337,7 +364,10 @@ const server = createServer(async (req, res) => {
}
if (!content.trim()) return json(res, 200, { skipped: true, name, empty: true });
- const result = await engine.ingest(content, { title: name, source: name });
+ const result = await engine.ingest(content, {
+ title: name,
+ source: relPath?.trim() || name,
+ });
json(res, 200, { result, stats: await engine.stats() });
return;
}
@@ -359,6 +389,40 @@ const server = createServer(async (req, res) => {
}
});
+// In-process auto-watch. Created lazily: by CONTEXT_GRAPH_WATCH=1 at startup
+// (resumes registered folders), or the first time the UI ingests a server
+// path with "keep watching" enabled.
+let webWatcher: GraphWatcher | undefined;
+
+function ensureWebWatcher(): GraphWatcher {
+ return (webWatcher ??= new GraphWatcher(engine, {
+ onEvent: (event) => {
+ if (event.type === "ingested" && !event.result.skipped) {
+ console.log(`watch: ✓ ${event.result.title} (+${event.result.nodesCreated} entities)`);
+ } else if (event.type === "deleted") {
+ console.log(`watch: • ${event.file} deleted (knowledge kept)`);
+ } else if (event.type === "error") {
+ console.error(`watch: ✗ ${event.file}: ${event.error}`);
+ }
+ },
+ }));
+}
+
+if (process.env.CONTEXT_GRAPH_WATCH === "1") {
+ const dirs = engine.listWatchedDirs();
+ if (dirs.length > 0) {
+ for (const wd of dirs) {
+ ensureWebWatcher()
+ .add(wd.dir)
+ .catch((err) =>
+ console.error(`watch: ✗ ${wd.dir}: ${err instanceof Error ? err.message : String(err)}`),
+ );
+ }
+ } else {
+ console.log("CONTEXT_GRAPH_WATCH=1 but no folders are registered — run: context-graph watch <dir>");
+ }
+}
+
server.listen(PORT, HOST, () => {
const p = providerStatus();
console.log(`Context Graph web UI → http://localhost:${PORT}`);
diff --git a/src/web/ui.html b/src/web/ui.html
index 24c5e5d7..6d828ef4 100644
--- a/src/web/ui.html
+++ b/src/web/ui.html
@@ -345,6 +345,24 @@
details.server-path .grow-hint code { font-family: var(--mono); font-size: 11.5px; }
details.server-path[open] { padding-bottom: var(--s1); }
details.server-path > *:not(summary) { margin-top: var(--s2); }
+ .watch-opt {
+ display: flex; align-items: center; gap: 7px; font-size: 12.5px; color: var(--ink-soft);
+ cursor: pointer; user-select: none;
+ }
+ .watch-opt input { accent-color: var(--verdant); }
+
+ .sync-pill {
+ display: flex; align-items: center; gap: 8px; margin-top: var(--s3);
+ padding: 8px 12px; border: 1px solid var(--verdant); border-radius: var(--r-sm);
+ background: var(--verdant-soft); font-size: 12.5px; color: var(--ink-soft);
+ }
+ .sync-pill .dot {
+ width: 8px; height: 8px; border-radius: 50%; background: var(--verdant); flex: none;
+ animation: sync-pulse 2s ease-in-out infinite;
+ }
+ .sync-pill #folder-sync-text { flex: 1; min-width: 0; }
+ .sync-pill .btn { padding: 3px 10px; font-size: 12px; }
+ @keyframes sync-pulse { 0%, 100% { opacity: 1; } 50% { opacity: .35; } }
.doc-row {
display: flex; gap: var(--s2); align-items: baseline; padding: 7px 4px;
@@ -488,12 +506,18 @@
<input type="file" id="files-input" multiple accept=".pdf,.md,.markdown,.txt" hidden />
<input type="file" id="folder-input" webkitdirectory multiple hidden />
<div class="picker-row">
- <button class="btn quiet" id="choose-folder-btn" type="button">Choose a folder…</button>
+ <button class="btn quiet" id="choose-folder-btn" type="button" title="The folder keeps syncing while this tab is open">Connect a folder…</button>
+ </div>
+ <div id="folder-sync" class="sync-pill hidden">
+ <span class="dot"></span>
+ <span id="folder-sync-text"></span>
+ <button class="btn quiet" id="folder-sync-stop" type="button">Stop</button>
</div>
<details class="server-path">
<summary>Ingest a path on the server</summary>
<p class="grow-hint">Reads a file or folder from the server's own disk — in Docker, that's the mounted <code>/docs</code>.</p>
<input class="dir" id="ingest-input" placeholder="/path/to/docs or /path/to/file.md" />
+ <label class="watch-opt"><input type="checkbox" id="watch-check" /> Keep watching — future changes are ingested automatically</label>
<button class="btn quiet" id="ingest-btn" type="button">Ingest server path</button>
</details>
<div id="ingest-progress" class="hidden">
@@ -1127,7 +1151,7 @@ $("ingest-btn").addEventListener("click", async () => {
const r = await api("/api/ingest-dir", {
method: "POST",
headers: { "content-type": "application/json" },
- body: JSON.stringify({ dir }),
+ body: JSON.stringify({ dir, watch: $("watch-check").checked }),
});
if (!r.ok) {
const err = await r.json().catch(() => ({}));
@@ -1160,7 +1184,8 @@ $("ingest-btn").addEventListener("click", async () => {
status.textContent =
`✓ ${fresh.length} document${fresh.length === 1 ? "" : "s"} added` +
(skipped ? ` (${skipped} already known, skipped)` : "") +
- `, ${nodes} facts written or reinforced`;
+ `, ${nodes} facts written or reinforced` +
+ (ev.watching ? " — now watching this folder for changes" : "");
} else if (ev.type === "error") {
throw new Error(ev.error);
}
@@ -1182,16 +1207,19 @@ $("ingest-btn").addEventListener("click", async () => {
const UPLOAD_ACCEPT = /\.(pdf|md|markdown|txt)$/i;
// Base64-encode file bytes in chunks (fromCharCode.apply chokes on huge arrays).
+// The relative path (from a folder pick or live sync) keys the document on the
+// server, so same-named files in different subfolders stay distinct.
async function fileToPayload(file) {
+ const relPath = file._relPath || file.webkitRelativePath || undefined;
if (/\.pdf$/i.test(file.name)) {
const bytes = new Uint8Array(await file.arrayBuffer());
let binary = "";
for (let i = 0; i < bytes.length; i += 0x8000) {
binary += String.fromCharCode.apply(null, bytes.subarray(i, i + 0x8000));
}
- return { name: file.name, dataBase64: btoa(binary) };
+ return { name: file.name, relPath, dataBase64: btoa(binary) };
}
- return { name: file.name, text: await file.text() };
+ return { name: file.name, relPath, text: await file.text() };
}
$("dropzone").addEventListener("click", () => $("files-input").click());
@@ -1206,7 +1234,20 @@ $("dropzone").addEventListener("drop", (e) => {
if (ingestBusy) return;
uploadFiles(Array.from(e.dataTransfer.files || []), "files");
});
-$("choose-folder-btn").addEventListener("click", () => $("folder-input").click());
+$("choose-folder-btn").addEventListener("click", async () => {
+ // Chromium: keep a handle to the picked folder and live-sync it while the
+ // tab is open. Elsewhere: fall back to the classic one-shot folder upload.
+ if (!window.showDirectoryPicker) return $("folder-input").click();
+ try {
+ const handle = await showDirectoryPicker({ mode: "read" });
+ await startFolderSync(handle);
+ } catch (e) {
+ if (e.name !== "AbortError") {
+ $("ingest-status").className = "grow-status err";
+ $("ingest-status").textContent = "Could not open the folder: " + e.message;
+ }
+ }
+});
$("files-input").addEventListener("change", (e) => uploadPicked(e, "files"));
$("folder-input").addEventListener("change", (e) => uploadPicked(e, "folder"));
@@ -1276,6 +1317,105 @@ async function uploadFiles(all, kind) {
}
}
+/* ---------- live folder sync (File System Access API, Chromium) ----------
+ The picked folder's handle stays with the page: every few seconds it is
+ re-scanned and files whose size/mtime moved are re-uploaded, so the graph
+ keeps evolving while the tab is open. Deletions are ignored on purpose —
+ the graph is append-only. */
+const SYNC_INTERVAL_MS = 5000;
+const SYNC_SKIP = new Set(["node_modules", "dist", "build", "out", "target", "vendor", "coverage", "__pycache__", "venv"]);
+const SYNC_MAX_BYTES = 1_000_000;
+let folderSync = null; // { handle, name, state: Map<rel, sig>, timer, busy }
+
+async function collectFolderFiles(handle, name) {
+ const files = [];
+ const state = new Map();
+ async function walk(dir, prefix) {
+ for await (const entry of dir.values()) {
+ if (entry.name.startsWith(".") || SYNC_SKIP.has(entry.name)) continue;
+ if (entry.kind === "directory") {
+ await walk(entry, prefix + entry.name + "/");
+ } else if (UPLOAD_ACCEPT.test(entry.name)) {
+ const f = await entry.getFile();
+ if (f.size > SYNC_MAX_BYTES) continue;
+ const rel = prefix + entry.name;
+ state.set(rel, f.size + ":" + f.lastModified);
+ f._relPath = name + "/" + rel;
+ f._rel = rel;
+ files.push(f);
+ }
+ }
+ }
+ await walk(handle, "");
+ return { files, state };
+}
+
+function renderSyncPill(msg) {
+ $("folder-sync-text").textContent = msg;
+ $("folder-sync").classList.remove("hidden");
+}
+
+async function startFolderSync(handle) {
+ stopFolderSync();
+ const session = { handle, name: handle.name, state: new Map(), timer: null, busy: false };
+ folderSync = session;
+ renderSyncPill(`Scanning “${handle.name}”…`);
+ const { files, state } = await collectFolderFiles(handle, handle.name);
+ session.state = state;
+ await uploadFiles(files, "folder"); // the usual progress UI for the first sweep
+ if (folderSync !== session) return; // stopped or replaced mid-sweep
+ session.timer = setInterval(syncTick, SYNC_INTERVAL_MS);
+ renderSyncPill(`Watching “${handle.name}” — ${state.size} files. Edits sync while this tab is open.`);
+}
+
+function stopFolderSync() {
+ if (!folderSync) return;
+ clearInterval(folderSync.timer);
+ folderSync = null;
+ $("folder-sync").classList.add("hidden");
+}
+$("folder-sync-stop").addEventListener("click", stopFolderSync);
+
+async function syncTick() {
+ const session = folderSync;
+ if (!session || session.busy || ingestBusy) return;
+ session.busy = true;
+ try {
+ const { files, state } = await collectFolderFiles(session.handle, session.name);
+ const changed = files.filter((f) => session.state.get(f._rel) !== state.get(f._rel));
+ session.state = state; // forgets deleted entries too — harmless, append-only
+ let synced = 0;
+ for (const f of changed) {
+ if (folderSync !== session) return; // stopped mid-batch
+ const r = await api("/api/ingest-upload", {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify(await fileToPayload(f)),
+ });
+ if (r.ok) {
+ const data = await r.json();
+ if (data.result && !data.result.skipped) synced++;
+ }
+ }
+ if (synced && folderSync === session) {
+ refreshStats().catch(() => {});
+ refreshGraph({ diff: true }).catch(() => {});
+ refreshDocs().catch(() => {});
+ const at = new Date().toLocaleTimeString();
+ renderSyncPill(`Watching “${session.name}” — synced ${synced} change${synced === 1 ? "" : "s"} at ${at}.`);
+ }
+ } catch (e) {
+ // Permission revoked or the folder went away — stop cleanly, not noisily.
+ if (folderSync === session) {
+ clearInterval(session.timer);
+ folderSync = null;
+ renderSyncPill(`Sync stopped: ${e.message}`);
+ }
+ return;
+ }
+ session.busy = false;
+}
+
/* ---------- documents ---------- */
const DOCS_PREVIEW = 10;
let docsExpanded = false;
@@ -1329,6 +1469,25 @@ async function refreshDocs() {
$("ask-form").requestSubmit();
}
})();
+
+/* ---------- live updates ----------
+ When the server runs with an auto-watcher (CONTEXT_GRAPH_WATCH=1), the graph
+ grows on its own as watched folders change. A cheap stats poll notices and
+ refreshes the heavier graph/docs views only when the counts actually moved. */
+let liveCounts = null;
+setInterval(async () => {
+ try {
+ const s = await refreshStats();
+ const counts = `${s.documents}|${s.nodes}|${s.edges}|${s.chunks}`;
+ if (liveCounts !== null && counts !== liveCounts) {
+ refreshGraph({ diff: true }).catch(() => {});
+ refreshDocs().catch(() => {});
+ }
+ liveCounts = counts;
+ } catch {
+ /* offline or token prompt pending — try again next tick */
+ }
+}, 10_000);
</script>
</body>
</html>
diff --git a/test/watch.test.ts b/test/watch.test.ts
new file mode 100644
index 00000000..3db97491
--- /dev/null
+++ b/test/watch.test.ts
@@ -0,0 +1,216 @@
+/**
+ * Auto-watch: the graph evolves as a connected folder changes. All tests run
+ * on the offline fake providers; FS events come from real chokidar watchers on
+ * temp directories, so generous polling deadlines absorb platform latency.
+ */
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { ContextGraphEngine, GraphWatcher, type WatchEvent } from "../src/index.js";
+import { fakeProviders } from "./helpers.ts";
+
+function engine() {
+ return new ContextGraphEngine({ dbPath: ":memory:", ...fakeProviders() });
+}
+
+function tempDir(): string {
+ return mkdtempSync(join(tmpdir(), "cg-watch-"));
+}
+
+/** Poll until `cond` holds — FS events arrive with platform-dependent latency. */
+async function waitFor(cond: () => boolean, what: string, ms = 10_000): Promise<void> {
+ const start = Date.now();
+ while (!cond()) {
+ if (Date.now() - start > ms) throw new Error(`timed out waiting for ${what}`);
+ await new Promise((r) => setTimeout(r, 50));
+ }
+}
+
+function collector() {
+ const events: WatchEvent[] = [];
+ const of = (type: WatchEvent["type"]) => events.filter((e) => e.type === type);
+ return { events, of, onEvent: (e: WatchEvent) => events.push(e) };
+}
+
+test("initial scan ingests existing files; a second watcher skips them all", async () => {
+ const dir = tempDir();
+ const e = engine();
+ writeFileSync(join(dir, "auth.md"), "[[Auth Service]] ==uses==> [[OAuth]]");
+ writeFileSync(join(dir, "deploy.md"), "[[Deploys]] ==run_on==> [[CI]]");
+
+ const first = collector();
+ const w1 = new GraphWatcher(e, { debounceMs: 50, onEvent: first.onEvent });
+ await w1.add(dir);
+ await w1.idle();
+ await w1.close();
+
+ assert.equal(first.of("ingested").length, 2);
+ assert.ok(first.of("ingested").every((ev) => ev.type === "ingested" && !ev.result.skipped));
+ assert.equal((await e.stats()).documents, 2);
+
+ // A fresh watcher over the same unchanged folder is a cheap no-op.
+ const second = collector();
+ const w2 = new GraphWatcher(e, { debounceMs: 50, onEvent: second.onEvent });
+ await w2.add(dir);
+ await w2.idle();
+ await w2.close();
+
+ assert.equal(second.of("ingested").length, 2);
+ assert.ok(second.of("ingested").every((ev) => ev.type === "ingested" && ev.result.skipped));
+ assert.equal((await e.stats()).documents, 2);
+
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("a file created after watching starts is ingested automatically", async () => {
+ const dir = tempDir();
+ const e = engine();
+ const c = collector();
+ const w = new GraphWatcher(e, { debounceMs: 50, onEvent: c.onEvent });
+ await w.add(dir);
+
+ writeFileSync(join(dir, "new.md"), "[[Billing]] ==bills==> [[Customers]]");
+ await waitFor(() => c.of("ingested").length === 1, "the new file to be ingested");
+
+ assert.equal((await e.stats()).documents, 1);
+ const [node] = await e.store.findNodesByName("billing");
+ assert.ok(node, "entity from the new file reached the graph");
+
+ await w.close();
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("rapid successive saves are debounced into one ingest", async () => {
+ const dir = tempDir();
+ const e = engine();
+ const c = collector();
+ const w = new GraphWatcher(e, { debounceMs: 400, onEvent: c.onEvent });
+ await w.add(dir);
+
+ const file = join(dir, "notes.md");
+ writeFileSync(file, "[[Draft]] v1");
+ writeFileSync(file, "[[Draft]] v2");
+ writeFileSync(file, "[[Draft]] final version");
+ await waitFor(() => c.of("ingested").length >= 1, "the burst of saves to be ingested");
+ // Give any straggler event time to (wrongly) trigger a second ingest.
+ await new Promise((r) => setTimeout(r, 700));
+ await w.idle();
+
+ assert.equal(c.of("ingested").length, 1, "one ingest for the whole burst");
+ const docs = await e.store.documentsBySource(file);
+ assert.equal(docs.length, 1);
+
+ await w.close();
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("editing a file replaces its document instead of piling up copies", async () => {
+ const dir = tempDir();
+ const e = engine();
+ const c = collector();
+ const w = new GraphWatcher(e, { debounceMs: 50, onEvent: c.onEvent });
+ const file = join(dir, "auth.md");
+ writeFileSync(file, "[[Auth Service]] ==uses==> [[OAuth]]");
+ await w.add(dir);
+ await w.idle();
+ const before = await e.store.documentsBySource(file);
+ assert.equal(before.length, 1);
+
+ writeFileSync(file, "[[Auth Service]] ==rotates==> [[Tokens]] every 15 minutes");
+ await waitFor(() => c.of("ingested").length === 2, "the edited file to be re-ingested");
+
+ const after = await e.store.documentsBySource(file);
+ assert.equal(after.length, 1, "still one document for the source");
+ assert.notEqual(after[0].hash, before[0].hash, "the document reflects the new content");
+ assert.equal((await e.stats()).documents, 1);
+
+ await w.close();
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("deleting a file keeps its knowledge (append-only)", async () => {
+ const dir = tempDir();
+ const e = engine();
+ const c = collector();
+ const w = new GraphWatcher(e, { debounceMs: 50, onEvent: c.onEvent });
+ const file = join(dir, "legacy.md");
+ writeFileSync(file, "[[Legacy System]] ==replaced_by==> [[New System]]");
+ await w.add(dir);
+ await w.idle();
+ const statsBefore = await e.stats();
+
+ rmSync(file);
+ await waitFor(() => c.of("deleted").length === 1, "the deletion to be observed");
+ await w.idle();
+
+ assert.deepEqual(await e.stats(), statsBefore, "nothing is pruned on delete");
+ assert.equal(c.of("ingested").length, 1, "the delete triggered no ingest");
+
+ await w.close();
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("wrong extensions, oversized files, and skip-dirs are ignored", async () => {
+ const dir = tempDir();
+ const e = engine();
+ const c = collector();
+ writeFileSync(join(dir, "keep.md"), "[[Kept]]");
+ writeFileSync(join(dir, "skip.log"), "[[Log Noise]]");
+ writeFileSync(join(dir, "huge.md"), "x".repeat(1_000_001));
+ mkdirSync(join(dir, "node_modules"));
+ writeFileSync(join(dir, "node_modules", "readme.md"), "[[Vendored]]");
+
+ const w = new GraphWatcher(e, { debounceMs: 50, onEvent: c.onEvent });
+ await w.add(dir);
+ await w.idle();
+
+ assert.equal(c.of("ingested").length, 1);
+ assert.equal((await e.stats()).documents, 1);
+ assert.equal((await e.store.findNodesByName("log noise")).length, 0);
+ assert.equal((await e.store.findNodesByName("vendored")).length, 0);
+
+ // Live events under a skip-dir are ignored too.
+ writeFileSync(join(dir, "node_modules", "later.md"), "[[Vendored Later]]");
+ writeFileSync(join(dir, "later.txt"), "[[Arrived Later]]");
+ await waitFor(() => c.of("ingested").length === 2, "only the valid live file to be ingested");
+ await w.idle();
+ assert.equal((await e.store.findNodesByName("vendored later")).length, 0);
+
+ await w.close();
+ await e.close();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+test("the watch registry survives engine restarts", async () => {
+ const home = tempDir();
+ const dbPath = join(home, "graph.db");
+ const notes = join(home, "notes");
+ mkdirSync(notes);
+
+ const e1 = new ContextGraphEngine({ dbPath, ...fakeProviders() });
+ const added = e1.addWatchedDir(notes, [".md"]);
+ assert.equal(added.dir, notes);
+ // Re-registering is idempotent and keeps the original timestamp.
+ assert.equal(e1.addWatchedDir(notes).addedAt, added.addedAt);
+ await e1.close();
+
+ const e2 = new ContextGraphEngine({ dbPath, ...fakeProviders() });
+ const listed = e2.listWatchedDirs();
+ assert.equal(listed.length, 1);
+ assert.equal(listed[0].dir, notes);
+ assert.deepEqual(listed[0].extensions, [".md"]);
+
+ assert.equal(e2.removeWatchedDir(notes), true);
+ assert.equal(e2.removeWatchedDir(notes), false);
+ assert.equal(e2.listWatchedDirs().length, 0);
+ await e2.close();
+
+ rmSync(home, { recursive: true, force: true });
+});