Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.
TypeScript ai-agentsanthropicclaude-codeclicode-graphcodexcontext-engineeringcursordeveloper-toolsgeminiknowledge-graphllmmcpmcp-serveropen-sourceopenaitree-sitter
Add benchmark harness (bench/) with tasks, agent, judge, and reporting
anirudhkumar-nanonets2ccebd348dc2
diff --git a/.gitignore b/.gitignoreindex 2c3b9659..844a9347 100644--- a/.gitignore+++ b/.gitignore@@ -15,6 +15,9 @@ data/.context-graph/*!.context-graph/graph.jsonl+# Benchmark run artifacts (raw rows + summaries are regenerated per run)+bench/results/+# Env.env.env.localdiff --git a/.mcp.json b/.mcp.jsonnew file mode 100644index 00000000..3c7cf8d9--- /dev/null+++ b/.mcp.json@@ -0,0 +1,14 @@+{+ "mcpServers": {+ "context-graph": {+ "type": "stdio",+ "command": "node",+ "args": [+ "/Users/anirudh/Documents/context-engine/dist/mcp.js"+ ],+ "env": {+ "CONTEXT_GRAPH_DB": "/Users/anirudh/Documents/context-engine/.context-graph/demo.db"+ }+ }+ }+}\ No newline at end of filediff --git a/GTM-AND-PRODUCT-SPEC.md b/GTM-AND-PRODUCT-SPEC.mdnew file mode 100644index 00000000..31266e71--- /dev/null+++ b/GTM-AND-PRODUCT-SPEC.md@@ -0,0 +1,125 @@+# Context Graph Engine — Product Spec & GTM Strategy++> **Internal only — gitignored.** Working draft for management review.+> Companion docs: `BUSINESS-CASE.md` (why this idea), `PRODUCT-DESIGNS.md` (design sketches).+> Last updated: 2026-07-09++---++## 1. One-paragraph summary (the elevator answer)++A self-hosted engine that turns a team's documents into a living knowledge graph that AI agents read before doing work and write learnings back to — so context compounds instead of being re-discovered every session. Bring your own LLM API key (OpenRouter/OpenAI) for high-quality extraction; everything else — the graph, embeddings, retrieval — runs on your own machine, and a fully local mode (Ollama) exists for teams that can't send data to a cloud model. Plugs into any agent via MCP.++**Positioning note:** the recommended path is API-key-first (quality), with local models as the privacy/air-gapped fallback — not the other way around.++## 2. Problem & who has it++**The agent problem (primary, why now):** an AI agent session starts from zero. Claude Code / Cursor re-discovers the same architecture, conventions, and gotchas on every task — burning tokens and latency re-reading files — and throws the learning away when the session ends. Vector RAG gives agents a bag of chunks, not structure they can reason over, and no write path to contribute learnings back.++**The staleness problem (why existing fixes fail):** any knowledge base built once is wrong within days. Repos and folders change daily; nobody re-runs ingestion. The harder problem than indexing is *keeping the index alive*: knowing when a file change actually warrants re-extraction, pruning facts that no longer hold, and doing it in the background without anyone babysitting it.++**Our scope (v1):** local repos and folders — the material closest to where agents work and where changes are detectable. Connectors to Slack, Notion, Drive etc. are a later addition, not v1. Users can also explicitly add files/facts via the web UI to a team or personal graph.++**The payoff:** agents get a compact, fresh context bundle instead of re-exploring , lower latency, fewer tokens per task; humans get a readable, citable map of their own repos and folders.++**Who feels it most:** teams (and individuals) using coding agents daily on real codebases, who prefer self-hosted tooling and won't ship their source to a vendor's cloud index.++## 3. What the product is (spec)++### 3.1 Core capabilities++The engine is a loop, not a one-shot indexer:++1. **Ingest** — point it at repos and folders (plus drag-and-drop of individual files in the web UI). Prose is chunked and extracted into entities + typed relationships, grounded in source passages. Code repos go through a per-file summarization pass first, then extraction over the summaries.+2. **Stay fresh (the differentiator)** — watched folders and repos keep syncing into the graph through background processes. The engine decides *when* a file change actually warrants re-extraction (not every save does), *what* to update incrementally, and *what to prune* — facts whose source files were deleted or contradicted decay and drop out. Nobody babysits the index; the graph is as current as the filesystem.+3. **Read** — an agent (or human) queries before a task and gets a compact context bundle: relevant entities, relationships, and cited source passages, sized to drop into a prompt. Reading the bundle instead of re-exploring the repo is where the latency and token savings come from.+4. **Contribute** — agents and humans write learnings back. New facts are deduplicated against existing knowledge; repeated observations reinforce confidence instead of duplicating. Every fact carries a provenance receipt: observation count, confidence, sources.++### 3.2 Surfaces++- **Graph-as-files (zero-dependency surface)** — the graph materializes *into the tracked repo/folder itself*, in a format and location the user picks per repo: a single compact `CONTEXT.md`, a directory of interlinked markdown files (one per entity/module), or machine-readable JSONL; committed or gitignored. Any agent can read it natively — no MCP call, no server running — and it's grep-able, git-versioned, and PR-reviewable. Agents can even contribute back by writing to a designated learnings file that the watcher ingests. The background sync keeps these files current; the graph database remains the source of truth.+- **MCP server** — the richer agent surface for targeted queries: compact context bundles sized for a prompt, plus structured contribute. Works with Claude Code, Cursor, and anything MCP-speaking, including Claude Code hooks and PR ingestion.+- **Web UI** — the human surface: drop files, browse a readable map of your repos and folders, ask cited questions, add facts by hand to a team or personal graph.+- **CLI & library** — for scripting and embedding in other tools.+- **Team sync** — a git-native graph file (merge-safe under concurrent edits) or a shared Docker deployment at one URL.++### 3.3 What we optimize for++- **Freshness** — the graph tracks the filesystem without human effort; stale facts get pruned, not accumulated.+- **Latency & token savings** — a task that costs an agent minutes of cold file exploration costs one graph read instead: on a real 344-file repo, benchmarks showed 31% fewer tool calls, −23% cost, and −17% latency. Measured, not promised — see `bench/`.+- **Human readability** — the graph doubles as living documentation: a new teammate can browse it and understand a repo's structure, owners, and gotchas.++### 3.4 Explicitly out of scope for v1++- Cloud connectors (Slack, Notion, Drive) — local repos and folders first; connectors come later.+- Hosted/SaaS offering — self-hosted only for now.+- Fine-grained access control within a graph — a graph is shared wholesale or personal.++## 4. Target users & ICP++**1. The individual power-user of coding agents (beachhead).** A developer running Claude Code / Cursor daily on a handful of repos. They already maintain a CLAUDE.md by hand and feel it decaying; they want their agent to stop re-exploring the same codebase every session. Zero-friction adoption: one install, point at a repo, no team buy-in needed. Value lands in the first session.++**2. Teams of 5–50 engineers using agents on a shared codebase.** The individual's graph becomes the team's when it's committed to the repo or served from one Docker URL. The motion: one champion adopts, the committed `.context/` files make it visible to everyone who clones, teammates onboard by browsing the graph. This is where shared agent memory and human-readable living documentation compound.++**3. Knowledge-heavy teams and individuals beyond code.** Folders of PDFs, notes, research, runbooks, contracts — the same engine, pointed at documents instead of repos: watched folders stay fresh, answers come back cited, and the graph is a browsable map of what the team knows. Same self-hosted, local-first appeal (r/selfhosted, HN crowd) for people who won't ship their documents to a vendor's cloud.++All three run on the same product — no separate editions. The segments differ only in what they point the engine at.++## 5. Positioning & competition++**One-liner:** *the self-maintaining context graph for your repos and folders — agents read it, write back to it, and it never goes stale.*++How we differ, by neighbor:++- **Context files (CLAUDE.md, AGENTS.md, .cursor/rules)** — today's default, generated and appended to by agents themselves. But nothing *manages* them: no change detection when the codebase moves underneath, no dedup or pruning (they bloat and drift), no provenance on any line, and the whole flat file is loaded every session regardless of the task. We are the management layer this pattern is missing — a structured graph behind it that stays fresh, with the context file as one generated, always-current output rather than an append-only scratchpad.+- **Vector RAG stacks (LlamaIndex-style pipelines, codebase embeddings in Cursor)** — retrieval over chunks. No structure to reason over, no relationships, no confidence/provenance, no contribute path, and re-indexing is on you. We're a graph with typed relationships and a reinforcement loop, kept fresh automatically.+- **Agent memory layers (mem0, Zep, Letta)** — memory of *conversations*, usually cloud-hosted, per-agent. We're memory of *your filesystem and codebase*, self-hosted, shared across every agent and human on the team. Complementary more than competitive.+- **GraphRAG (Microsoft) and knowledge-graph research tools** — batch, one-shot, expensive to rebuild, no freshness story. Our whole thesis is the maintenance loop they lack.+- **Enterprise search (Glean)** — per-seat SaaS, connector catalog, your documents in their cloud. We're self-hosted, free to start, repos-and-folders first. When we're compared to Glean, the answer is: different buyer, different trust model.++**The moat we're building:** not extraction (everyone has an LLM) — the *maintenance intelligence*: knowing when to update, what to prune, and how confidence should evolve. That's accumulated product logic, and the graph itself compounds — the longer a team runs it, the more contributed learnings it holds that exist nowhere else.++## 6. Open source strategy & licensing++**Yes — the engine is open source, MIT-licensed, deliberately.**++**Why OSS is the distribution strategy:** our target users (agent power-users, self-hosters) adopt from GitHub, HN, and word-of-mouth, and they specifically distrust closed tools that touch their source code. Open source is what earns the "your data never leaves your machine" claim credibility — anyone can verify it. It's also how a two-person project gets a connector/integration ecosystem it could never build alone.++**The open-core line (when monetization starts):** the engine — ingest, freshness loop, graph, MCP, web UI, CLI — stays MIT forever; that's the promise that makes adoption safe. What can be proprietary later: the **hosted cloud version** (we run it for you), and **team/enterprise features** around the core (auth, roles, SSO, audit, central admin). This is the standard open-core playbook (GitLab, Supabase, PostHog) and it doesn't require relicensing anything we shipped.++## 7. Teams & multi-user support++_Current state (Docker single-URL sharing, git-native graph sync), what real "teams" means — auth, roles, permissions, hosted option — and when we'd build it. TODO._++## 8. Release plan & milestones++**Phase 0 — Harden the wedge.** Ship the things the pitch depends on: pruning/decay so the freshness claim is fully true, graph-as-files output, and a token/latency benchmark on 2–3 real repos. Exit criterion: a stranger can install, point at a repo, and get value in under 5 minutes without us in the room.++**Phase 1 — Private alpha.** 10–20 hand-picked internal Nanonets users from segment 1 (coding-agent power-users). Goal is watching where onboarding breaks and whether people *keep* the watcher running after week one. Exit criterion: users still active after 10 days without prompting.++**Phase 2 — Public OSS launch.** "Released" means: public repo, one-line install, docs good enough that issues are feature requests rather than confusion. Plan in §8.1.++**Phase 3 — Team features.** Driven by what alpha/launch users actually ask for.++### 8.1 Public launch plan++**Assets:** README with a demo GIF + benchmark table, 2-min demo video, launch blog post, tested 5-min quickstart.++**Channels:**+- **Show HN** — *"Show HN: A context graph for your repos that AI agents read and keep fresh"*+- **MCP directories** — official registry, Smithery, PulseMCP, mcp.so, Glama. Targeted, compounds over time.+- **Awesome lists** — awesome-mcp-servers, awesome-selfhosted, awesome-claude-code.+- **Reddit** — r/selfhosted, r/LocalLLaMA, r/ClaudeAI, r/cursor — each with its own angle.+- **Communities & creators** — Claude Code/Cursor/MCP Discords, TLDR/Console.dev newsletters, a few agent-tooling YouTubers.++## 9. Business model & monetization path++_Free OSS core → what we could charge for (hosted, team/enterprise features, support). When monetization becomes relevant. TODO._++## 10. Metrics & what winning looks like++_Stars/installs/weekly active graphs, contribution-loop engagement, and the 6-month success criteria. TODO._++## 11. Risks & open questions++_Honest list: adoption risk, moat, model-cost dependency, and questions we want management input on. TODO._diff --git a/bench/README.md b/bench/README.mdnew file mode 100644index 00000000..9bf39367--- /dev/null+++ b/bench/README.md@@ -0,0 +1,61 @@+# Benchmark harness++Measures the product's central claim: **an agent that reads the context graph before a task+costs less (tokens + latency) than an agent that explores from cold — without losing correctness.**++## What it does++For each corpus it ingests the graph once, then runs every task through two arms of the **same**+Claude Sonnet 5 agent loop with the **same** filesystem tools (`read_file`, `grep`, `glob`, `list_dir`):++- **Cold** — the agent starts from zero and explores to find the answer.+- **Graph** — the agent additionally gets the `engine.read(question)` context bundle injected up front.++Every run records input/output tokens, tool-call count, wall-clock, and a correctness score. Correctness+is scored by an **Opus 4.8 judge** (against a reference answer) gated by a **required-keyword** floor, so a+fast-but-wrong answer can't score a win. Each task runs N trials to average out agent stochasticity.++The result is a per-corpus table (cold vs graph, with deltas) — the artifact for the README/launch.++## Requirements++- `OPENROUTER_API_KEY` — that's it. The agent (`anthropic/claude-sonnet-5`), judge+ (`anthropic/claude-opus-4.8`), and graph extraction all run through OpenRouter. Embeddings run+ locally in-process. Override the models with `BENCH_AGENT_MODEL` / `BENCH_JUDGE_MODEL`.++## Run++```bash+npm run bench -- --smoke # 1 corpus, 1 task, 1 trial — plumbing check+npm run bench # full: all corpora, all tasks, 3 trials+npm run bench -- --corpora unified-accounts-login-server --tasks 3 --trials 2+npm run bench -- --arms graph # a single arm+```++Results are written to `bench/results/<timestamp>.json` (raw per-trial rows) and `<timestamp>.md`+(the summary table, also printed to stdout).++## Corpora++Defined in `bench/tasks.ts`. The two code repos are expected as **siblings** of this repo under the+same parent directory; override any path with `BENCH_REPO_<ID_UPPER>` (dashes → underscores), e.g.+`BENCH_REPO_NEW_WEBSITE=/path/to/site`.++- `unified-accounts-login-server` — the Nanonets unified auth service (Node/Express). Small, so cheap to+ ingest; ground truth authored from the source (Auth0 flow, session JWT, redirect logic, config).+- `new-website` — the Nanonets marketing site (Next.js App Router, ~342 `.tsx`). Its README is+ `create-next-app` boilerplate, so every answer requires reading code — a clean cold-arm test.+- `northwind-docs` — the demo PDFs in `examples/demo-docs`; ground truth from `scripts/make-demo-pdfs.mjs`.+ A knowledge-folder data point (segment 3).++**Fairness note:** `ingestRepo` summarizes only code files (`CODE_EXTENSIONS`), so README / markdown /+`package.json` / CSS never enter the graph, while the cold agent *can* read them. Tasks are therefore+written to require multi-file **code** understanding, not README/config lookups.++To add another repo (e.g. `assign`): clone/point at it, add a `kind: "repo"` corpus entry in+`bench/tasks.ts` with tasks whose answers you can verify from its docs or code.++## Reading the result honestly++The claim holds only if the graph arm is **both** materially cheaper/faster **and** at least as correct.+If it's cheaper but less correct, that's a real finding to report — not something the harness hides.diff --git a/bench/agent.ts b/bench/agent.tsnew file mode 100644index 00000000..1231f473--- /dev/null+++ b/bench/agent.ts@@ -0,0 +1,322 @@+/**+ * The benchmark agent: a minimal, honest ReAct-style loop with filesystem+ * tools, driven by Claude Sonnet 5 (served via OpenRouter). Both benchmark arms+ * use this exact loop — the only difference is whether a pre-computed graph+ * context bundle is injected up front (graph arm) or not (cold arm). Keeping the+ * loop identical is what makes the token/latency/correctness comparison fair.+ *+ * We run a *manual* loop so we can read `usage` off every turn and sum it, so+ * the reported token cost is exact rather than estimated.+ */+import type OpenAI from "openai";+import { readFileSync, statSync, readdirSync, realpathSync } from "node:fs";+import { resolve, relative, join, isAbsolute, sep } from "node:path";+import { SKIP_DIRS, MAX_FILE_BYTES } from "../src/ingest/fs.js";+import { makeClient, AGENT_MODEL } from "./llm.js";++export { AGENT_MODEL };++export interface AgentResult {+ answer: string;+ tokens: { input: number; output: number; cacheRead: number; cacheCreate: number; total: number };+ toolCalls: number;+ iterations: number;+ wallMs: number;+ stopReason: string | null;+ toolLog: Array<{ name: string; input: unknown }>;+}++export interface RunAgentOptions {+ /** Directory the filesystem tools are confined to. */+ root: string;+ question: string;+ /** When set (graph arm), this bundle is injected before the question. */+ contextBundle?: string;+ maxIterations?: number;+ client?: OpenAI;+}++const SYSTEM_PROMPT =+ "You are a software engineer answering a factual question about a codebase or a folder of documents. " ++ "The files live under the tool root. Use read_file, grep, glob, and list_dir to find the answer, then " ++ "state it directly. Investigate enough to be correct, but stop exploring once you can answer. " ++ "When you are ready, give your final answer as plain text with no further tool calls.";++/** Resolve a model-supplied path and confine it to `root` (rejects `..`, absolute escapes, symlink escapes). */+export function safePath(root: string, p: string): string {+ const abs = resolve(root, p);+ let rel = relative(root, abs);+ if (rel.startsWith("..") || isAbsolute(rel)) throw new Error(`path escapes root: ${p}`);+ try {+ const real = realpathSync(abs);+ const realRoot = realpathSync(root);+ rel = relative(realRoot, real);+ if (rel.startsWith("..") || isAbsolute(rel)) throw new Error(`path escapes root: ${p}`);+ return real;+ } catch (e) {+ if (e instanceof Error && e.message.startsWith("path escapes")) throw e;+ return abs;+ }+}++function listFiles(root: string): string[] {+ const out: string[] = [];+ const walk = (dir: string) => {+ let entries;+ try {+ entries = readdirSync(dir, { withFileTypes: true });+ } catch {+ return;+ }+ for (const e of entries) {+ if (e.name.startsWith(".")) continue;+ const full = join(dir, e.name);+ if (e.isDirectory()) {+ if (SKIP_DIRS.has(e.name)) continue;+ walk(full);+ } else if (e.isFile()) {+ try {+ if (statSync(full).size > MAX_FILE_BYTES) continue;+ } catch {+ continue;+ }+ out.push(full);+ }+ }+ };+ walk(root);+ return out;+}++/** Very small glob → RegExp (supports **, *, ?), matched against root-relative POSIX paths. */+function globToRegExp(glob: string): RegExp {+ let re = "";+ for (let i = 0; i < glob.length; i++) {+ const c = glob[i];+ if (c === "*") {+ if (glob[i + 1] === "*") {+ re += ".*";+ i++;+ } else re += "[^/]*";+ } else if (c === "?") re += "[^/]";+ else if (".+^${}()|[]\\".includes(c)) re += "\\" + c;+ else re += c;+ }+ return new RegExp("^" + re + "$");+}++/** Tool definitions in OpenAI/OpenRouter function-calling format. */+const TOOLS: OpenAI.Chat.Completions.ChatCompletionTool[] = [+ {+ type: "function",+ function: {+ name: "read_file",+ description: "Read a UTF-8 text file under the root. Optionally limit to a line range.",+ parameters: {+ type: "object",+ properties: {+ path: { type: "string", description: "Path relative to the root" },+ start_line: { type: "integer", description: "1-based first line (optional)" },+ end_line: { type: "integer", description: "1-based last line (optional)" },+ },+ required: ["path"],+ },+ },+ },+ {+ type: "function",+ function: {+ name: "grep",+ description: "Search file contents with a JavaScript regular expression. Returns matching lines as path:line: text.",+ parameters: {+ type: "object",+ properties: {+ pattern: { type: "string", description: "Regular expression" },+ path: { type: "string", description: "Restrict to this subdirectory or file (optional)" },+ ignore_case: { type: "boolean", description: "Case-insensitive match (optional)" },+ },+ required: ["pattern"],+ },+ },+ },+ {+ type: "function",+ function: {+ name: "glob",+ description: "List files whose root-relative path matches a glob (e.g. src/**/*.ts).",+ parameters: { type: "object", properties: { pattern: { type: "string" } }, required: ["pattern"] },+ },+ },+ {+ type: "function",+ function: {+ name: "list_dir",+ description: "List the entries of a directory under the root.",+ parameters: {+ type: "object",+ properties: { path: { type: "string", description: "Directory relative to the root (default: root)" } },+ required: [],+ },+ },+ },+];++const MAX_GREP_MATCHES = 100;+const MAX_GLOB_RESULTS = 200;+const MAX_READ_LINES = 400;++function runTool(root: string, name: string, input: any): string {+ try {+ if (name === "read_file") {+ const file = safePath(root, String(input.path));+ const lines = readFileSync(file, "utf8").split("\n");+ const start = input.start_line ? Math.max(1, input.start_line) : 1;+ const end = input.end_line ? Math.min(lines.length, input.end_line) : Math.min(lines.length, start - 1 + MAX_READ_LINES);+ const slice = lines.slice(start - 1, end);+ const numbered = slice.map((l: string, i: number) => `${start + i}\t${l}`).join("\n");+ const more = end < lines.length ? `\n… (${lines.length - end} more lines)` : "";+ return numbered + more || "(empty file)";+ }+ if (name === "list_dir") {+ const dir = input.path ? safePath(root, String(input.path)) : root;+ const entries = readdirSync(dir, { withFileTypes: true })+ .filter((e) => !e.name.startsWith(".") && !SKIP_DIRS.has(e.name))+ .map((e) => (e.isDirectory() ? e.name + "/" : e.name));+ return entries.length ? entries.join("\n") : "(empty directory)";+ }+ if (name === "glob") {+ const re = globToRegExp(String(input.pattern));+ const hits = listFiles(root)+ .map((f) => relative(root, f).split(sep).join("/"))+ .filter((rel) => re.test(rel))+ .slice(0, MAX_GLOB_RESULTS);+ return hits.length ? hits.join("\n") : "(no files matched)";+ }+ if (name === "grep") {+ const flags = input.ignore_case ? "i" : "";+ let re: RegExp;+ try {+ re = new RegExp(String(input.pattern), flags);+ } catch (e) {+ return `invalid regex: ${e instanceof Error ? e.message : String(e)}`;+ }+ const base = input.path ? safePath(root, String(input.path)) : root;+ const files = statSync(base).isDirectory() ? listFiles(base) : [base];+ const out: string[] = [];+ for (const f of files) {+ let text: string;+ try {+ text = readFileSync(f, "utf8");+ } catch {+ continue;+ }+ const rel = relative(root, f).split(sep).join("/");+ const fl = text.split("\n");+ for (let i = 0; i < fl.length; i++) {+ if (re.test(fl[i])) {+ out.push(`${rel}:${i + 1}: ${fl[i].trim().slice(0, 200)}`);+ if (out.length >= MAX_GREP_MATCHES) return out.join("\n") + "\n… (truncated)";+ }+ }+ }+ return out.length ? out.join("\n") : "(no matches)";+ }+ return `unknown tool: ${name}`;+ } catch (e) {+ return `error: ${e instanceof Error ? e.message : String(e)}`;+ }+}++export async function runAgent(opts: RunAgentOptions): Promise<AgentResult> {+ const client = opts.client ?? makeClient();+ const root = resolve(opts.root);+ const maxIterations = opts.maxIterations ?? 15;++ const userText = opts.contextBundle+ ? `A knowledge graph of these files was queried for this task and returned the context below. ` ++ `Use it to answer; you may still use the tools to confirm or fill gaps.\n\n` ++ `<graph_context>\n${opts.contextBundle}\n</graph_context>\n\nQuestion: ${opts.question}`+ : `Question: ${opts.question}`;++ // Content is expressed as parts so we can attach Anthropic prompt-caching+ // breakpoints (OpenRouter forwards `cache_control` to Anthropic). The system+ // + tools + the (possibly large) bundle are a stable head, cached once; a+ // breakpoint then slides onto the newest tool result each turn so the entire+ // prior transcript is a cache read rather than re-billed at full price —+ // which is how Claude Code actually runs, and what makes the token numbers fair.+ const cc = { type: "ephemeral" as const };+ const messages: any[] = [+ { role: "system", content: [{ type: "text", text: SYSTEM_PROMPT, cache_control: cc }] },+ { role: "user", content: [{ type: "text", text: userText, cache_control: cc }] },+ ];++ /** Keep the cache breakpoint on the most recent tool result (strip it from older ones). */+ const slideCacheBreakpoint = () => {+ const toolMsgs = messages.filter((m) => m.role === "tool");+ toolMsgs.forEach((m, i) => {+ const part = Array.isArray(m.content) ? m.content[m.content.length - 1] : null;+ if (part) part.cache_control = i === toolMsgs.length - 1 ? cc : undefined;+ });+ };+ const tokens = { input: 0, output: 0, cacheRead: 0, cacheCreate: 0, total: 0 };+ const toolLog: AgentResult["toolLog"] = [];+ let toolCalls = 0;+ let iterations = 0;+ let stopReason: string | null = null;+ let answer = "";++ const started = Date.now();+ for (iterations = 1; iterations <= maxIterations; iterations++) {+ const resp = await client.chat.completions.create({+ model: AGENT_MODEL,+ max_tokens: 4096,+ messages,+ tools: TOOLS,+ });+ const u: any = resp.usage ?? {};+ const cached = u.prompt_tokens_details?.cached_tokens ?? 0;+ tokens.cacheRead += cached;+ tokens.input += Math.max(0, (u.prompt_tokens ?? 0) - cached); // uncached input only+ tokens.output += u.completion_tokens ?? 0;+ const choice = resp.choices[0];+ stopReason = choice?.finish_reason ?? null;+ const msg = choice?.message;+ if (!msg) break;++ messages.push(msg); // preserve the assistant turn (with any tool_calls) for the next request++ const toolCallList = msg.tool_calls ?? [];+ if (toolCallList.length === 0) {+ answer = (msg.content ?? "").trim();+ break;+ }++ for (const tc of toolCallList) {+ if (tc.type !== "function") continue;+ toolCalls++;+ let input: any = {};+ try {+ input = JSON.parse(tc.function.arguments || "{}");+ } catch {+ /* leave input empty; runTool will error informatively */+ }+ toolLog.push({ name: tc.function.name, input });+ const result = runTool(root, tc.function.name, input);+ messages.push({ role: "tool", tool_call_id: tc.id, content: [{ type: "text", text: result }] });+ }+ slideCacheBreakpoint();+ }++ if (!answer) {+ // Loop hit the cap — take the last assistant text we have.+ for (let i = messages.length - 1; i >= 0; i--) {+ if (messages[i].role === "assistant" && typeof messages[i].content === "string" && messages[i].content.trim()) {+ answer = messages[i].content.trim();+ break;+ }+ }+ }++ tokens.total = tokens.input + tokens.output + tokens.cacheRead;+ return { answer, tokens, toolCalls, iterations, wallMs: Date.now() - started, stopReason, toolLog };+}diff --git a/bench/judge.ts b/bench/judge.tsnew file mode 100644index 00000000..4458c495--- /dev/null+++ b/bench/judge.ts@@ -0,0 +1,87 @@+/**+ * Correctness scoring. Two layers, deliberately:+ *+ * 1. A deterministic required-keyword floor — the answer must contain every+ * `requiredKeywords` substring (case-insensitive). Cheap, can't be gamed by+ * a confident-but-wrong answer, and catches an over-generous LLM judge.+ * 2. An LLM judge (Claude Opus 4.8, via OpenRouter — deliberately stronger than+ * the Sonnet 5 agent, so it isn't grading itself) that scores the answer+ * against a reference and returns JSON.+ *+ * Final correctness = judge says correct AND the keyword floor passed.+ */+import type OpenAI from "openai";+import { makeClient, JUDGE_MODEL } from "./llm.js";++export { JUDGE_MODEL };++export interface Verdict {+ score: number; // 0..1 from the judge+ judgeCorrect: boolean;+ keywordPass: boolean; // deterministic floor+ correct: boolean; // judgeCorrect && keywordPass+ reasoning: string;+}++export interface JudgeInput {+ question: string;+ referenceAnswer: string;+ agentAnswer: string;+ requiredKeywords: string[];+ client?: OpenAI;+}++/** Pull the first {...} JSON object out of a string (models sometimes wrap JSON in prose/fences). */+function extractJson(text: string): any {+ try {+ return JSON.parse(text);+ } catch {+ const m = text.match(/\{[\s\S]*\}/);+ if (m) {+ try {+ return JSON.parse(m[0]);+ } catch {+ /* fall through */+ }+ }+ return null;+ }+}++export async function judge(input: JudgeInput): Promise<Verdict> {+ const client = input.client ?? makeClient();++ const kw = input.requiredKeywords ?? [];+ const hay = input.agentAnswer.toLowerCase();+ const keywordPass = kw.every((k) => hay.includes(k.toLowerCase()));++ const prompt =+ `You are grading an AI agent's answer to a factual question about a codebase/document set. ` ++ `Respond with ONLY a JSON object of the form {"correct": boolean, "score": number, "reasoning": string} ` ++ `where score is 0.0 (wrong/irrelevant) to 1.0 (fully correct and complete). No prose, no code fences.\n\n` ++ `Question:\n${input.question}\n\n` ++ `Reference answer (ground truth):\n${input.referenceAnswer}\n\n` ++ `Agent's answer:\n${input.agentAnswer}\n\n` ++ `Grade whether the agent's answer is factually correct and responsive, judged against the reference. ` ++ `Extra correct detail is fine; contradicting the reference or missing the core fact is not.`;++ const resp = await client.chat.completions.create({+ model: JUDGE_MODEL,+ max_tokens: 1024,+ messages: [{ role: "user", content: prompt }],+ response_format: { type: "json_object" },+ });++ const text = resp.choices[0]?.message?.content ?? "";+ const parsed = extractJson(text) ?? { correct: false, score: 0, reasoning: `unparseable judge output: ${text.slice(0, 200)}` };++ const score = Math.max(0, Math.min(1, Number(parsed.score) || 0));+ const judgeCorrect = Boolean(parsed.correct);+ return {+ score,+ judgeCorrect,+ keywordPass,+ correct: judgeCorrect && keywordPass,+ reasoning: String(parsed.reasoning ?? ""),+ };+}diff --git a/bench/llm.ts b/bench/llm.tsnew file mode 100644index 00000000..370ef1fe--- /dev/null+++ b/bench/llm.ts@@ -0,0 +1,18 @@+/**+ * LLM plumbing for the benchmark. Routed through OpenRouter (the same key the+ * engine uses for extraction) via the OpenAI-compatible SDK, so no separate+ * Anthropic API key is needed. Models are still Claude — just served by+ * OpenRouter — and overridable via env.+ */+import OpenAI from "openai";++export const OPENROUTER_BASE_URL = process.env.OPENROUTER_BASE_URL ?? "https://openrouter.ai/api/v1";+export const AGENT_MODEL = process.env.BENCH_AGENT_MODEL ?? "anthropic/claude-sonnet-5";+export const JUDGE_MODEL = process.env.BENCH_JUDGE_MODEL ?? "anthropic/claude-opus-4.8";++/** A configured OpenRouter client. Throws if the key is missing (the cold arm can't run without it either). */+export function makeClient(): OpenAI {+ const apiKey = process.env.OPENROUTER_API_KEY;+ if (!apiKey) throw new Error("OPENROUTER_API_KEY is required (agent + judge run through OpenRouter).");+ return new OpenAI({ apiKey, baseURL: OPENROUTER_BASE_URL });+}diff --git a/bench/report.ts b/bench/report.tsnew file mode 100644index 00000000..3f8af25c--- /dev/null+++ b/bench/report.ts@@ -0,0 +1,125 @@+/**+ * Aggregation and reporting. Turns the raw per-trial rows into the summary+ * table that is the actual launch deliverable: per corpus, cold vs graph, with+ * the token / latency / correctness deltas that either back the pitch or don't.+ */+export interface Row {+ corpus: string;+ taskId: string;+ arm: "cold" | "graph";+ trial: number;+ tokensInput: number;+ tokensOutput: number;+ tokensTotal: number;+ cacheRead: number;+ cacheCreate: number;+ toolCalls: number;+ wallMs: number;+ correct: boolean;+ score: number;+ keywordPass: boolean;+ judgeCorrect: boolean;+ iterations: number;+ stopReason: string | null;+ answer: string;+ reasoning: string;+}++// Sonnet 5 pricing per MTok (standard sticker; intro is $2/$10 through 2026-08-31).+const IN_PER_TOK = 3 / 1_000_000;+const OUT_PER_TOK = 15 / 1_000_000;++/** Approximate USD cost of one agent run, accounting for cache read/write multipliers. */+export function costOf(r: { tokensInput: number; tokensOutput: number; cacheRead: number; cacheCreate: number }): number {+ return (+ (r.tokensInput + r.cacheCreate * 1.25 + r.cacheRead * 0.1) * IN_PER_TOK ++ r.tokensOutput * OUT_PER_TOK+ );+}++function mean(xs: number[]): number {+ return xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0;+}++interface ArmAgg {+ n: number;+ tokensTotal: number;+ tokensInput: number;+ tokensOutput: number;+ toolCalls: number;+ wallMs: number;+ correctness: number; // fraction 0..1+ score: number;+ cost: number;+}++function aggregate(rows: Row[]): ArmAgg {+ return {+ n: rows.length,+ tokensTotal: mean(rows.map((r) => r.tokensTotal)),+ tokensInput: mean(rows.map((r) => r.tokensInput)),+ tokensOutput: mean(rows.map((r) => r.tokensOutput)),+ toolCalls: mean(rows.map((r) => r.toolCalls)),+ wallMs: mean(rows.map((r) => r.wallMs)),+ correctness: mean(rows.map((r) => (r.correct ? 1 : 0))),+ score: mean(rows.map((r) => r.score)),+ cost: mean(rows.map((r) => costOf(r))),+ };+}++function pctDelta(cold: number, graph: number): string {+ if (cold === 0) return "n/a";+ const d = ((graph - cold) / cold) * 100;+ return (d > 0 ? "+" : "") + d.toFixed(0) + "%";+}++function fmt(n: number): string {+ return n >= 1000 ? Math.round(n).toLocaleString() : n.toFixed(n < 10 ? 1 : 0);+}++export function buildMarkdown(rows: Row[]): string {+ const corpora = [...new Set(rows.map((r) => r.corpus))];+ const lines: string[] = [];+ lines.push("# Context Graph Engine — Benchmark Results", "");+ lines.push(+ `Agent: Claude Sonnet 5 · Judge: Claude Opus 4.8 · ${rows.length} agent runs total.`,+ "",+ "Each cell is the mean across all tasks × trials for that corpus/arm. " ++ "**Cold** = agent explores with filesystem tools from zero. **Graph** = same agent, " ++ "same tools, plus the graph context bundle injected up front.",+ "",+ );++ for (const corpus of corpora) {+ const cold = rows.filter((r) => r.corpus === corpus && r.arm === "cold");+ const graph = rows.filter((r) => r.corpus === corpus && r.arm === "graph");+ const c = aggregate(cold);+ const g = aggregate(graph);++ lines.push(`## ${corpus}`, "");+ lines.push("| Metric | Cold | Graph | Δ |", "|---|---|---|---|");+ lines.push(`| Total tokens (incl. cached) | ${fmt(c.tokensTotal)} | ${fmt(g.tokensTotal)} | ${pctDelta(c.tokensTotal, g.tokensTotal)} |`);+ lines.push(`| Uncached input tokens | ${fmt(c.tokensInput)} | ${fmt(g.tokensInput)} | ${pctDelta(c.tokensInput, g.tokensInput)} |`);+ lines.push(`| Output tokens | ${fmt(c.tokensOutput)} | ${fmt(g.tokensOutput)} | ${pctDelta(c.tokensOutput, g.tokensOutput)} |`);+ lines.push(`| Tool calls | ${fmt(c.toolCalls)} | ${fmt(g.toolCalls)} | ${pctDelta(c.toolCalls, g.toolCalls)} |`);+ lines.push(`| Wall-clock (s) | ${(c.wallMs / 1000).toFixed(1)} | ${(g.wallMs / 1000).toFixed(1)} | ${pctDelta(c.wallMs, g.wallMs)} |`);+ lines.push(`| Cost / task ($) | ${c.cost.toFixed(4)} | ${g.cost.toFixed(4)} | ${pctDelta(c.cost, g.cost)} |`);+ lines.push(`| Correctness | ${(c.correctness * 100).toFixed(0)}% | ${(g.correctness * 100).toFixed(0)}% | ${(g.correctness * 100 - c.correctness * 100).toFixed(0)} pts |`);+ lines.push(`| Judge score | ${c.score.toFixed(2)} | ${g.score.toFixed(2)} | ${(g.score - c.score).toFixed(2)} |`);+ lines.push("");++ // Judge on COST (cache-aware) + correctness — the economic reality. "Total tokens"+ // includes cache reads billed at ~0.1×, so it overstates the graph arm's true cost.+ const cheaper = g.cost < c.cost;+ const correctnessHeld = g.correctness >= c.correctness - 0.001;+ const verdict = cheaper && correctnessHeld+ ? `✅ Graph is cheaper (${pctDelta(c.cost, g.cost)} cost) and correctness held or improved.`+ : !cheaper && g.correctness > c.correctness + 0.001+ ? `➖ Graph costs more (${pctDelta(c.cost, g.cost)}) but is more correct (+${((g.correctness - c.correctness) * 100).toFixed(0)} pts) — a quality win, not a cost win.`+ : cheaper+ ? `⚠️ Graph is cheaper but correctness dropped ${((c.correctness - g.correctness) * 100).toFixed(0)} pts — not a clean win.`+ : `❌ Graph costs more with no correctness gain on this corpus.`;+ lines.push(verdict, "");+ }+ return lines.join("\n");+}diff --git a/bench/run.ts b/bench/run.tsnew file mode 100644index 00000000..ec676c72--- /dev/null+++ b/bench/run.ts@@ -0,0 +1,206 @@+/**+ * Benchmark orchestrator.+ *+ * npm run bench # full run: all corpora, all tasks, 3 trials+ * npm run bench -- --smoke # 1 corpus, 1 task, 1 trial (plumbing check)+ * npm run bench -- --corpora context-engine --tasks 3 --trials 2+ * npm run bench -- --arms graph # one arm only+ *+ * Needs OPENROUTER_API_KEY only — the agent (Sonnet 5), judge (Opus 4.8), and+ * graph extraction all run through OpenRouter. Override models with+ * BENCH_AGENT_MODEL / BENCH_JUDGE_MODEL.+ */+import "dotenv/config";+import { mkdtempSync, mkdirSync, writeFileSync, copyFileSync, readdirSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join, dirname, resolve, basename } from "node:path";+import { fileURLToPath } from "node:url";+import { ContextGraphEngine } from "../src/index.js";+import { DOC_EXTENSIONS } from "../src/ingest/fs.js";+import { extractPdfText, isPdfPath } from "../src/ingest/pdf.js";+import { CORPORA, type Corpus } from "./tasks.js";+import { runAgent } from "./agent.js";+import { judge } from "./judge.js";+import { makeClient } from "./llm.js";+import { buildMarkdown, type Row } from "./report.js";++const here = dirname(fileURLToPath(import.meta.url));++interface Args {+ corpora?: string[];+ tasks?: number;+ trials: number;+ arms: Array<"cold" | "graph">;+ concurrency: number;+}++function parseArgs(argv: string[]): Args {+ const a: Args = { trials: 3, arms: ["cold", "graph"], concurrency: 6 };+ for (let i = 0; i < argv.length; i++) {+ const arg = argv[i];+ if (arg === "--smoke") {+ a.corpora = ["unified-accounts-login-server"]; // smallest corpus — cheapest plumbing check+ a.tasks = 1;+ a.trials = 1;+ } else if (arg === "--corpora" || arg === "--repos") a.corpora = argv[++i].split(",");+ else if (arg === "--tasks") a.tasks = Number(argv[++i]);+ else if (arg === "--trials") a.trials = Number(argv[++i]);+ else if (arg === "--arms") a.arms = argv[++i].split(",") as Array<"cold" | "graph">;+ else if (arg === "--concurrency") a.concurrency = Math.max(1, Number(argv[++i]));+ }+ return a;+}++/** Run items through `fn` with at most `n` in flight. Independent agent/judge calls, so safe to parallelize. */+async function pool<T, R>(items: T[], n: number, fn: (item: T) => Promise<R>): Promise<R[]> {+ const results: R[] = new Array(items.length);+ let next = 0;+ async function worker() {+ while (next < items.length) {+ const idx = next++;+ results[idx] = await fn(items[idx]);+ }+ }+ await Promise.all(Array.from({ length: Math.min(n, items.length) }, worker));+ return results;+}++/** For a docs corpus, build a temp dir of plaintext the cold agent can read/grep (PDFs → .txt). */+async function makeDocsWorkdir(corpus: Corpus): Promise<string> {+ const dir = mkdtempSync(join(tmpdir(), "cge-bench-docs-"));+ const files = readdirSync(corpus.path).filter((f) => DOC_EXTENSIONS.some((e) => f.toLowerCase().endsWith(e)));+ for (const f of files) {+ const src = join(corpus.path, f);+ if (isPdfPath(f)) {+ const text = await extractPdfText(src);+ writeFileSync(join(dir, f.replace(/\.pdf$/i, ".txt")), text);+ } else {+ copyFileSync(src, join(dir, f));+ }+ }+ return dir;+}++async function main() {+ const args = parseArgs(process.argv.slice(2));+ if (!process.env.OPENROUTER_API_KEY) {+ console.error("Set OPENROUTER_API_KEY (agent, judge, and graph extraction all run through OpenRouter).");+ process.exit(1);+ }+ const client = makeClient();++ const corpora = CORPORA.filter((c) => !args.corpora || args.corpora.includes(c.id));+ if (corpora.length === 0) {+ console.error(`No matching corpora. Available: ${CORPORA.map((c) => c.id).join(", ")}`);+ process.exit(1);+ }++ const rows: Row[] = [];+ const tmpDbDirs: string[] = [];++ for (const corpus of corpora) {+ const tasks = args.tasks ? corpus.tasks.slice(0, args.tasks) : corpus.tasks;+ console.log(`\n=== corpus: ${corpus.id} (${corpus.kind}) — ${tasks.length} tasks × ${args.trials} trials ===`);++ // Fresh temp-backed engine; ingest once and reuse for the whole corpus.+ const dbDir = mkdtempSync(join(tmpdir(), "cge-bench-db-"));+ tmpDbDirs.push(dbDir);+ const engine = new ContextGraphEngine({ dbPath: join(dbDir, "graph.db") });++ let agentRoot = corpus.path;+ let docsWorkdir: string | undefined;++ try {+ const t0 = Date.now();+ if (corpus.kind === "repo") {+ const r = await engine.ingestRepo(corpus.path, {+ onProgress: (p) => process.stdout.write(`\r ingest: ${p.phase} ${p.index + 1}/${p.total} `),+ });+ console.log(`\n ingested repo: ${r.files} files (${r.summarized} summarized, ${r.cached} cached) in ${((Date.now() - t0) / 1000).toFixed(1)}s`);+ if (r.errors.length) console.log(` ingest errors: ${r.errors.length} (first: ${r.errors[0]})`);+ } else {+ docsWorkdir = await makeDocsWorkdir(corpus);+ agentRoot = docsWorkdir;+ const res = await engine.ingestDir(corpus.path);+ console.log(` ingested docs: ${res.length} files in ${((Date.now() - t0) / 1000).toFixed(1)}s`);+ }+ const stats = await engine.stats();+ console.log(` graph: ${stats.nodes} entities, ${stats.edges} relationships, ${stats.chunks} chunks`);++ // Precompute each task's graph bundle once (reused across arms/trials). engine.read+ // is fast and local; do it up front so the parallel pool below never touches the engine.+ const bundles = new Map<string, string>();+ for (const task of tasks) {+ bundles.set(task.id, args.arms.includes("graph") ? (await engine.read(task.question)).prompt : "");+ }++ // Every (task, arm, trial) is an independent unit — run them through a bounded pool.+ const units: Array<{ task: (typeof tasks)[number]; arm: "cold" | "graph"; trial: number }> = [];+ for (const task of tasks) for (const arm of args.arms) for (let trial = 1; trial <= args.trials; trial++) units.push({ task, arm, trial });++ let done = 0;+ const total = units.length;+ const corpusRows = await pool(units, args.concurrency, async ({ task, arm, trial }) => {+ try {+ const ar = await runAgent({+ client,+ root: agentRoot,+ question: task.question,+ contextBundle: arm === "graph" ? bundles.get(task.id) : undefined,+ });+ const v = await judge({+ client,+ question: task.question,+ referenceAnswer: task.referenceAnswer,+ agentAnswer: ar.answer,+ requiredKeywords: task.requiredKeywords,+ });+ console.log(` (${++done}/${total}) [${task.id}] ${arm} t${trial}: ${v.correct ? "✓" : "✗"} ${ar.tokens.total} tok, ${ar.toolCalls} tools, ${(ar.wallMs / 1000).toFixed(1)}s`);+ return {+ corpus: corpus.id, taskId: task.id, arm, trial,+ tokensInput: ar.tokens.input, tokensOutput: ar.tokens.output, tokensTotal: ar.tokens.total,+ cacheRead: ar.tokens.cacheRead, cacheCreate: ar.tokens.cacheCreate,+ toolCalls: ar.toolCalls, wallMs: ar.wallMs,+ correct: v.correct, score: v.score, keywordPass: v.keywordPass, judgeCorrect: v.judgeCorrect,+ iterations: ar.iterations, stopReason: ar.stopReason, answer: ar.answer, reasoning: v.reasoning,+ } as Row;+ } catch (e) {+ // Non-fatal: record the failure and keep going so one error can't sink the run.+ console.log(` (${++done}/${total}) [${task.id}] ${arm} t${trial}: ERROR ${e instanceof Error ? e.message : String(e)}`);+ return {+ corpus: corpus.id, taskId: task.id, arm, trial,+ tokensInput: 0, tokensOutput: 0, tokensTotal: 0, cacheRead: 0, cacheCreate: 0,+ toolCalls: 0, wallMs: 0, correct: false, score: 0, keywordPass: false,+ judgeCorrect: false, iterations: 0, stopReason: "error", answer: "",+ reasoning: `run error: ${e instanceof Error ? e.message : String(e)}`,+ } as Row;+ }+ });+ rows.push(...corpusRows);+ } finally {+ await engine.close();+ if (docsWorkdir) rmSync(docsWorkdir, { recursive: true, force: true });+ }+ }++ // Write results + summary. Timestamp comes from Date (fine in a standalone script).+ const stamp = new Date().toISOString().replace(/[:.]/g, "-");+ const outDir = join(here, "results");+ mkdirSync(outDir, { recursive: true });+ const jsonPath = join(outDir, `${stamp}.json`);+ const mdPath = join(outDir, `${stamp}.md`);+ writeFileSync(jsonPath, JSON.stringify({ generatedAt: stamp, rows }, null, 2));+ const md = buildMarkdown(rows);+ writeFileSync(mdPath, md);++ console.log("\n" + md);+ console.log(`\nRaw rows → ${jsonPath}`);+ console.log(`Summary → ${mdPath}`);++ for (const d of tmpDbDirs) rmSync(d, { recursive: true, force: true });+}++main().catch((err) => {+ console.error(err);+ process.exit(1);+});diff --git a/bench/selfcheck.ts b/bench/selfcheck.tsnew file mode 100644index 00000000..af3a650d--- /dev/null+++ b/bench/selfcheck.ts@@ -0,0 +1,119 @@+/**+ * Offline verification of the harness plumbing — no API key needed. Drives the+ * agent loop, tool execution, judge, and report with a stubbed Claude client so+ * we can confirm control flow end-to-end before spending on a live run.+ *+ * npx tsx bench/selfcheck.ts+ */+import { mkdtempSync, writeFileSync, mkdirSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import assert from "node:assert/strict";+import { runAgent } from "./agent.js";+import { judge } from "./judge.js";+import { buildMarkdown, type Row } from "./report.js";++const usage = { prompt_tokens: 100, completion_tokens: 50, total_tokens: 150 };++/** Stub that mimics the OpenAI/OpenRouter chat.completions shape our code reads. Judge turns pass `response_format`. */+function makeStubClient() {+ let agentTurn = 0;+ return {+ chat: {+ completions: {+ create: async (params: any) => {+ if (params.response_format) {+ // Judge call → JSON verdict.+ return {+ choices: [{ message: { role: "assistant", content: JSON.stringify({ correct: true, score: 0.9, reasoning: "matches reference" }) }, finish_reason: "stop" }],+ usage,+ };+ }+ // Agent call. First turn: use grep. Second turn: final answer.+ agentTurn++;+ if (agentTurn === 1) {+ return {+ choices: [{ message: { role: "assistant", content: null, tool_calls: [{ id: "tc_1", type: "function", function: { name: "grep", arguments: JSON.stringify({ pattern: "SECRET_TOKEN" }) } }] }, finish_reason: "tool_calls" }],+ usage,+ };+ }+ return {+ choices: [{ message: { role: "assistant", content: "The secret token is SECRET_TOKEN, defined in config.ts." }, finish_reason: "stop" }],+ usage,+ };+ },+ },+ },+ } as any;+}++async function main() {+ const root = mkdtempSync(join(tmpdir(), "cge-selfcheck-"));+ mkdirSync(join(root, "src"), { recursive: true });+ writeFileSync(join(root, "src", "config.ts"), "export const token = 'SECRET_TOKEN';\n");++ const client = makeStubClient();++ // 1. Agent loop: should call the grep tool once, then answer.+ const ar = await runAgent({ client, root, question: "What is the secret token?" });+ assert.equal(ar.toolCalls, 1, "expected exactly one tool call");+ assert.equal(ar.toolLog[0].name, "grep", "expected the grep tool to run");+ assert.ok(ar.answer.includes("SECRET_TOKEN"), "answer should contain the fact");+ assert.equal(ar.tokens.total, 300, "tokens should sum across 2 turns (150 each)");+ assert.equal(ar.stopReason, "stop");+ console.log(`✓ agent loop: ${ar.toolCalls} tool call, ${ar.tokens.total} tokens, answer="${ar.answer}"`);++ // 2. Real tool execution actually found the file (grep ran against the temp repo).+ const graphAr = await runAgent({+ client: makeStubClient(),+ root,+ question: "What is the secret token?",+ contextBundle: "# Context\nSECRET_TOKEN lives in config.ts",+ });+ assert.ok(graphAr.answer.includes("SECRET_TOKEN"));+ console.log(`✓ graph arm (bundle injected): answer="${graphAr.answer}"`);++ // 3. Judge: keyword floor + parsed verdict.+ const v = await judge({+ client,+ question: "What is the secret token?",+ referenceAnswer: "SECRET_TOKEN",+ agentAnswer: ar.answer,+ requiredKeywords: ["SECRET_TOKEN"],+ });+ assert.equal(v.keywordPass, true);+ assert.equal(v.judgeCorrect, true);+ assert.equal(v.correct, true);+ assert.equal(v.score, 0.9);+ console.log(`✓ judge: correct=${v.correct} score=${v.score} keywordPass=${v.keywordPass}`);++ // 3b. Keyword floor overrides an over-generous judge.+ const vMiss = await judge({+ client,+ question: "q",+ referenceAnswer: "SECRET_TOKEN",+ agentAnswer: "I could not find it.",+ requiredKeywords: ["SECRET_TOKEN"],+ });+ assert.equal(vMiss.keywordPass, false);+ assert.equal(vMiss.correct, false, "keyword floor must veto the judge's 'correct'");+ console.log(`✓ keyword floor vetoes judge: correct=${vMiss.correct}`);++ // 4. Report renders.+ const rows: Row[] = [+ { corpus: "demo", taskId: "t1", arm: "cold", trial: 1, tokensInput: 5000, tokensOutput: 800, tokensTotal: 5800, cacheRead: 0, cacheCreate: 0, toolCalls: 6, wallMs: 12000, correct: true, score: 0.9, keywordPass: true, judgeCorrect: true, iterations: 4, stopReason: "end_turn", answer: "a", reasoning: "r" },+ { corpus: "demo", taskId: "t1", arm: "graph", trial: 1, tokensInput: 2000, tokensOutput: 400, tokensTotal: 2400, cacheRead: 0, cacheCreate: 0, toolCalls: 1, wallMs: 5000, correct: true, score: 0.92, keywordPass: true, judgeCorrect: true, iterations: 2, stopReason: "end_turn", answer: "a", reasoning: "r" },+ ];+ const md = buildMarkdown(rows);+ assert.ok(md.includes("## demo"));+ assert.ok(md.includes("Total tokens"));+ assert.ok(md.includes("✅"), "graph is cheaper + correctness held → should show a clean win");+ console.log("✓ report renders with cold-vs-graph deltas and verdict");++ console.log("\nAll harness self-checks passed. (Live runs use OPENROUTER_API_KEY.)");+}++main().catch((e) => {+ console.error(e);+ process.exit(1);+});diff --git a/bench/tasks.ts b/bench/tasks.tsnew file mode 100644index 00000000..e747790e--- /dev/null+++ b/bench/tasks.ts@@ -0,0 +1,280 @@+/**+ * The benchmark corpus and task set. This file *is* the benchmark's validity:+ * every task must have a verifiable answer, and (for a fair cold-arm test)+ * require reading more than one file to answer confidently. Reference answers+ * here are grounded in the actual source (cited during authoring), not guessed.+ *+ * Corpora are real external repos expected as siblings of this one under the+ * same parent directory (override a path with BENCH_REPO_<ID_UPPER> if needed):+ * - unified-accounts-login-server — the Nanonets unified auth service (Node/Express).+ * - new-website — the Nanonets marketing site (Next.js App Router).+ * Plus the checked-in demo docs (northwind-docs) as a knowledge-folder data point.+ *+ * Note on fairness: ingestRepo summarizes only CODE files (CODE_EXTENSIONS), so+ * README/markdown/package.json/CSS do NOT enter the graph — while the cold agent+ * can read them. Questions therefore target multi-file *code* understanding,+ * where the graph bundle should actually help, not README/config lookups.+ */+import { dirname, resolve } from "node:path";+import { fileURLToPath } from "node:url";++const here = dirname(fileURLToPath(import.meta.url));+const repoRoot = resolve(here, "..");+const parent = resolve(repoRoot, "..");++/** Sibling repo path by default; overridable via BENCH_REPO_<ID_UPPER> (dashes → underscores). */+function repoPath(id: string): string {+ const envKey = "BENCH_REPO_" + id.toUpperCase().replace(/-/g, "_");+ return process.env[envKey] ?? resolve(parent, id);+}++export interface Task {+ id: string;+ question: string;+ referenceAnswer: string;+ /** Case-insensitive substrings the answer must contain (deterministic floor). */+ requiredKeywords: string[];+}++export interface Corpus {+ id: string;+ /** "repo" → ingestRepo + agent reads source directly. "docs" → ingestDir + agent reads extracted text. */+ kind: "repo" | "docs";+ path: string;+ tasks: Task[];+}++const UNIFIED: Corpus = {+ id: "unified-accounts-login-server",+ kind: "repo",+ path: repoPath("unified-accounts-login-server"),+ tasks: [+ {+ id: "auth-provider-flows",+ question: "What authentication provider does this service use, and what login flows does it support?",+ referenceAnswer:+ "Auth0. Two flows: an Authorization Code flow for Google social login (connection google-oauth2), and a Resource Owner Password Grant (password-realm, realm Username-Password-Authentication) for on-page email/password login.",+ requiredKeywords: ["auth0"],+ },+ {+ id: "redirect-precedence",+ question: "After a successful login, how does the service decide which app to redirect the user to?",+ referenceAnswer:+ "splitService.getRedirectDestination resolves in order: a valid return_url first, then redirect_to (agents→AGENTS_URL, context→CONTEXT_URL/atlas.nanonets.ai, app→APP_NANONETS_URL), defaulting to APP_NANONETS_URL. return_url is validated to a hostname ending in nanonets.com, nanonets.ai, or exactly localhost.",+ requiredKeywords: ["return_url"],+ },+ {+ id: "session-cookie",+ question: "What is the session cookie called and what are its flags?",+ referenceAnswer:+ "The cookie is session_token: httpOnly false (frontends read it in JS), secure only in production, sameSite lax in dev / none in prod, domain unset in dev / .nanonets.com in prod, path '/'.",+ requiredKeywords: ["session_token"],+ },+ {+ id: "session-jwt",+ question: "How is the session token created and validated?",+ referenceAnswer:+ "sessionService mints a JWT signed with HS256 using SESSION_SECRET_KEY, with claims including session_id, email, user_id, auth_type 'auth0', redirect_app, and type:'session'; default expiry 24h applied to both the JWT exp and the cookie maxAge. Validation rejects any token whose type is not 'session'.",+ requiredKeywords: ["session"],+ },+ {+ id: "callback-csrf",+ question: "How does the /callback route protect against CSRF before exchanging the Auth0 code?",+ referenceAnswer:+ "It verifies the Auth0 `state` returned in the query against the value stored in the `auth_state` cookie, and only exchanges the code for tokens if they match.",+ requiredKeywords: ["auth_state"],+ },+ {+ id: "app-handoff",+ question: "When app-handoff is enabled, how does the server decide whether to send a user to the app or to agents?",+ referenceAnswer:+ "When appHandoff is enabled and there is no explicit return_url and redirect_to is empty or 'agents', it makes an HMAC-signed server-to-server call to app's user-exists endpoint; existing app users are 302'd to app via a one-time app_handoff code, while unknown/new users go to AGENTS_URL. On any error/timeout it defaults to agents. The logic exists twice (in /callback and in finalizeAuthSession for the password path).",+ requiredKeywords: ["handoff"],+ },+ {+ id: "required-config",+ question: "Which configuration values are required for the server to boot, and what happens if they are missing?",+ referenceAnswer:+ "validateConfig() runs at startup and process.exit(1)s if AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, or SESSION_SECRET_KEY are missing (plus the app-handoff secrets when handoff is enabled).",+ requiredKeywords: ["auth0_client_id", "session_secret_key"],+ },+ {+ id: "product-preference",+ question: "How is a user's product preference stored and where is it set?",+ referenceAnswer:+ "It is stored in a SQLite table user_preferences (email primary key, preferred_app). The POST /choose-product route writes it via userService.setUserPreference and only accepts product in {app, agents}.",+ requiredKeywords: ["preference"],+ },+ {+ id: "signup-password-stash",+ question: "During signup with email verification, how are the user's credentials handled while waiting for verification?",+ referenceAnswer:+ "signupIntentService stores the signup password AES-256-GCM-encrypted with an HKDF-derived key from SESSION_SECRET_KEY, with a 30-minute TTL; consumption is single-use and uses crypto.timingSafeEqual.",+ requiredKeywords: ["encrypt"],+ },+ {+ id: "entry-and-port",+ question: "What is the server's entry point and what port does it listen on by default?",+ referenceAnswer:+ "The entry point is src/app.js (an Express app); the default port is 5001 (env PORT). Note the README/.env.example mention 3000, but the code default is 5001.",+ requiredKeywords: ["5001"],+ },+ ],+};++const NEW_WEBSITE: Corpus = {+ id: "new-website",+ kind: "repo",+ path: repoPath("new-website"),+ tasks: [+ {+ id: "framework",+ question: "What framework and routing approach does this site use?",+ referenceAnswer:+ "Next.js (v16) with the App Router (file-based routing under src/app), React 19, and TypeScript.",+ requiredKeywords: ["next.js"],+ },+ {+ id: "dynamic-route",+ question: "Is there any dynamic route, and what does it do?",+ referenceAnswer:+ "Yes — /customers/unlock/[slug], a password-gated case-study route. It calls notFound() unless the slug is in PROTECTED_SLUGS, and marks itself robots noindex.",+ requiredKeywords: ["unlock"],+ },+ {+ id: "case-study-auth",+ question: "How does the case-study password gate actually work, and is it secure?",+ referenceAnswer:+ "It is client-side only: passwords are a hardcoded slug→password map in src/lib/case-study-passwords.ts, and UnlockForm compares the typed password in the browser. The passwords are shipped to the client, so it is not real security.",+ requiredKeywords: ["client"],+ },+ {+ id: "redirects",+ question: "Where are URL redirects configured and give an example.",+ referenceAnswer:+ "In next.config.ts, which defines several permanent redirects (e.g. /agent → /agents, /products/nano → /agent/nano, /case-studies/roche → /customers/roche). middleware.ts is a no-op.",+ requiredKeywords: ["redirect"],+ },+ {+ id: "analytics-stack",+ question: "What analytics and tracking tools are wired into the site?",+ referenceAnswer:+ "PostHog plus Google Tag Manager, Google Analytics (gtag), LinkedIn Insight, Intercom, and Termly consent — mostly gated on production. PostHog is set up in src/components/analytics.tsx.",+ requiredKeywords: ["posthog"],+ },+ {+ id: "posthog-proxy",+ question: "How are PostHog analytics requests routed?",+ referenceAnswer:+ "next.config.ts rewrites /ingest/* to https://events.nanonets.com (a self-hosted PostHog proxy); the PostHog host defaults to events.nanonets.com.",+ requiredKeywords: ["ingest"],+ },+ {+ id: "primary-cta",+ question: "What is the site's primary call-to-action and how is it implemented?",+ referenceAnswer:+ "A 'Book a demo' button (BookDemoButton) that opens a Calendly scheduling URL and fires a LinkedIn conversion on calendly.event_scheduled; it is used across most pages.",+ requiredKeywords: ["calendly"],+ },+ {+ id: "forms",+ question: "How are lead-capture forms (e.g. the AP cost calculator) served?",+ referenceAnswer:+ "As static HTML files under public/forms/, wired via next.config.ts rewrites (/forms/<name> → /forms/<name>.html). There are no Next API routes.",+ requiredKeywords: ["forms"],+ },+ {+ id: "styling",+ question: "What styling system does the site use?",+ referenceAnswer:+ "Tailwind CSS v4 via @tailwindcss/postcss, with no tailwind.config file — theme tokens are defined in CSS (@theme) in globals.css. Font is DM Sans via next/font.",+ requiredKeywords: ["tailwind"],+ },+ {+ id: "content-source",+ question: "Where does the page content come from — a CMS, markdown, or code?",+ referenceAnswer:+ "Content is hardcoded in TSX components/pages; there is no CMS and no MDX/markdown content source, and no data-fetching calls.",+ requiredKeywords: [],+ },+ ],+};++const NORTHWIND_DOCS: Corpus = {+ id: "northwind-docs",+ kind: "docs",+ path: resolve(repoRoot, "examples", "demo-docs"),+ tasks: [+ {+ id: "failed-charge-flow",+ question: "What happens when a customer charge fails in the Northwind billing system?",+ referenceAnswer:+ "The Payments Service emits payment_failed; the Dunning Worker retries up to 3 times with exponential backoff (1h, 6h, 24h). After the third failed retry the subscription is marked past_due, and after 7 days past_due it is automatically suspended.",+ requiredKeywords: ["past_due"],+ },+ {+ id: "token-expiry",+ question: "How long do Northwind access tokens last, and what backs the Auth Service?",+ referenceAnswer:+ "Access tokens expire after 15 minutes and refresh tokens after 30 days; the Auth Service is backed by Redis for token revocation lists.",+ requiredKeywords: ["redis"],+ },+ {+ id: "databases",+ question: "Which databases do the Orders and Inventory services use?",+ referenceAnswer:+ "The Orders Service uses a PostgreSQL database named orders_db and the Inventory Service uses a separate PostgreSQL database named inventory_db.",+ requiredKeywords: ["orders_db", "inventory_db"],+ },+ {+ id: "billing-escalation",+ question: "Who do you escalate a Stripe outage to in billing, and where do you post updates?",+ referenceAnswer:+ "Escalate to the Payments team lead, Dana Whitfield, and post an update in the #billing-incidents Slack channel; page on-call via PagerDuty service 'northwind-billing'.",+ requiredKeywords: ["dana"],+ },+ {+ id: "charge-success-alert",+ question: "What is the target charge success rate, and when does an alert fire?",+ referenceAnswer:+ "Target charge success rate is above 97%; an alert fires if it drops below 95% over any 30-minute window.",+ requiredKeywords: ["97"],+ },+ {+ id: "notifications-providers",+ question: "How does Northwind send transactional email and SMS?",+ referenceAnswer: "The Notifications Service sends transactional email through SendGrid and SMS through Twilio.",+ requiredKeywords: ["sendgrid", "twilio"],+ },+ {+ id: "deployment",+ question: "Where and how is Northwind deployed?",+ referenceAnswer:+ "On Kubernetes in AWS us-east-1; each service is a separate Deployment with a minimum of 3 replicas; images are built by GitHub Actions and pushed to Amazon ECR; production deploys require Platform-team approval.",+ requiredKeywords: ["us-east-1"],+ },+ {+ id: "refund-approval",+ question: "What approval is needed to issue a large refund?",+ referenceAnswer:+ "Refunds above $500 require approval from a Finance team lead; all refunds are logged to the audit_log table in payments_db and reflected in Stripe within 24 hours.",+ requiredKeywords: ["500"],+ },+ {+ id: "org-structure",+ question: "How is Northwind engineering organized and who is the VP of Engineering?",+ referenceAnswer:+ "Four teams — Platform (infra/CI-CD), Payments (Payments Service and billing), Orders (Orders and Inventory), and Growth (Notifications and customer-facing). The VP of Engineering is Priya Nair.",+ requiredKeywords: ["priya"],+ },+ {+ id: "local-setup",+ question: "What are the steps to set up the Northwind platform locally?",+ referenceAnswer:+ "Clone the northwind-inc/platform monorepo, install Node.js 20 and Docker, run 'make bootstrap' to start local Postgres/Redis/Kafka via docker-compose, copy .env.sample to .env with your API keys, and run 'make test'.",+ requiredKeywords: ["bootstrap"],+ },+ ],+};++export const CORPORA: Corpus[] = [UNIFIED, NEW_WEBSITE, NORTHWIND_DOCS];diff --git a/package-lock.json b/package-lock.jsonindex 9f4672e6..62ed6d78 100644--- a/package-lock.json+++ b/package-lock.json@@ -22,6 +22,7 @@"bin": {"context-graph": "dist/cli.js","context-graph-mcp": "dist/mcp.js",+ "context-graph-serve": "dist/serve.js","context-graph-web": "dist/web.js"},"devDependencies": {diff --git a/package.json b/package.jsonindex 530a5d0d..1ab91c42 100644--- a/package.json+++ b/package.json@@ -41,6 +41,7 @@"web": "tsx src/web.ts","serve": "tsx src/serve.ts","example": "tsx examples/quickstart.ts",+ "bench": "tsx bench/run.ts","test": "node --import tsx --test test/*.test.ts","prepare": "npm run build","prepublishOnly": "npm run build"diff --git a/test/bench-paths.test.ts b/test/bench-paths.test.tsnew file mode 100644index 00000000..f0b89f1d--- /dev/null+++ b/test/bench-paths.test.ts@@ -0,0 +1,30 @@+/**+ * The benchmark agent's filesystem tools run model-chosen paths. safePath is+ * the confinement boundary — assert it rejects escapes and accepts in-root paths.+ */+import { test } from "node:test";+import assert from "node:assert/strict";+import { tmpdir } from "node:os";+import { mkdtempSync, writeFileSync, mkdirSync } from "node:fs";+import { join } from "node:path";+import { safePath } from "../bench/agent.js";++const root = mkdtempSync(join(tmpdir(), "cge-safepath-"));+mkdirSync(join(root, "sub"), { recursive: true });+writeFileSync(join(root, "a.txt"), "hi");+writeFileSync(join(root, "sub", "b.txt"), "yo");++test("accepts paths inside the root", () => {+ assert.ok(safePath(root, "a.txt").endsWith("a.txt"));+ assert.ok(safePath(root, "sub/b.txt").endsWith("b.txt"));+ assert.ok(safePath(root, "./sub/../a.txt").endsWith("a.txt"));+});++test("rejects parent-directory escapes", () => {+ assert.throws(() => safePath(root, "../../etc/passwd"), /escapes root/);+ assert.throws(() => safePath(root, "sub/../../secret"), /escapes root/);+});++test("rejects absolute paths outside the root", () => {+ assert.throws(() => safePath(root, "/etc/passwd"), /escapes root/);+});