Skip to content

Turn any codebase, with its docs, SQL schemas, configs, and PDFs, into a queryable knowledge graph. A /graphify skill for Claude Code, Cursor, Codex, and Gemini CLI: local deterministic AST parsing, every edge explained, no vector store.

shorten README - cut design principles, files section, marketing padding

Safi1cd52f191ea5
diff --git a/README.md b/README.md
index 06bcbb07f..6e43f6be1 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,7 @@
**A Claude Code skill.** Type `/graphify` in Claude Code - it reads your files, builds a knowledge graph, and gives you back structure you didn't know was there.
-> Andrej Karpathy keeps a `/raw` folder where he drops papers, tweets, screenshots, and notes. The problem: that folder becomes opaque. You forget what's in it. You can't see what connects. graphify is the answer to that problem.
+> Andrej Karpathy keeps a `/raw` folder where he drops papers, tweets, screenshots, and notes. graphify is the answer to that problem - 71.5x fewer tokens per query vs reading the raw files, persistent across sessions, honest about what it found vs guessed.
```
/graphify ./raw
@@ -12,51 +12,22 @@
```
graphify-out/
-├── graph.html interactive graph - click nodes, search, filter by community, open in any browser
-├── obsidian/ open as Obsidian vault - visual graph, wikilinks, filter by community
-├── GRAPH_REPORT.md what the graph found: god nodes, surprising connections, suggested questions
-├── graph.json persistent graph - query it weeks later without re-reading anything
-├── cache/ per-file SHA256 cache - re-runs only process changed files
-└── memory/ Q&A results filed back in - what you ask grows the graph on next --update
+├── graph.html interactive graph - click nodes, search, filter by community
+├── obsidian/ open as Obsidian vault
+├── GRAPH_REPORT.md god nodes, surprising connections, suggested questions
+├── graph.json persistent graph - query weeks later without re-reading
+└── cache/ SHA256 cache - re-runs only process changed files
```
-## Why this exists
-
-graphify takes that observation and builds the missing infrastructure:
-
-| His problem | What graphify adds |
-|---|---|
-| Folder becomes opaque | Community detection surfaces structure automatically |
-| Forget what's in it | Persistent `graph.json` - query weeks later without re-reading |
-| Can't see connections | Cross-community surprising connections as a first-class output |
-| Claude hallucinates missing links | `EXTRACTED` / `INFERRED` / `AMBIGUOUS` - honest about what was found vs guessed |
-| Context resets every session | Memory feedback loop - what you ask grows the graph on `--update` |
-| Only works on text | PDFs, images, screenshots, tweets, any language via vision |
-| Reading everything costs tokens | **71.5x token reduction** on large mixed corpora - query the graph, not the files |
-
-**What LLMs get wrong without it:** Naive summarization fills every gap confidently. You get output that sounds complete but you can't tell what was actually in the files vs invented. And next session, it's all gone.
-
-**What graphify does differently:**
-
-- **71.5x token reduction** - on a mixed corpus (Karpathy repos + papers + images), querying the graph costs 71.5x fewer tokens than reading the raw files. The benchmark runs automatically after every `/graphify` run.
-- **Persistent graph** - relationships stored in `graphify-out/graph.json`, survive across sessions. Query weeks later without re-reading anything.
-- **Honest audit trail** - every edge tagged `EXTRACTED` (explicitly stated), `INFERRED` (call-graph or reasonable deduction), or `AMBIGUOUS` (flagged for review). You always know what was found vs invented.
-- **Cross-document surprise** - Leiden community detection finds clusters, then surfaces cross-community connections: the things you would never think to ask about directly.
-- **Feedback loop** - every query answer saved to `graphify-out/memory/`. On next `--update`, that Q&A becomes a node. The graph grows from what you ask, not just what you add.
-
-The result: a navigable map of your corpus that is honest about what it knows and what it guessed.
-
## Install
-**Requires:** [Claude Code](https://claude.ai/code) (the CLI or desktop app) and Python 3.10+
+**Requires:** [Claude Code](https://claude.ai/code) and Python 3.10+
```bash
pip install graphifyy && graphify install
```
-> **Note:** The PyPI package is temporarily named `graphifyy` while the `graphify` name is being reclaimed. The CLI, skill command, and everything else is still called `graphify` - only `pip install` uses the extra `y`.
-
-This copies the skill file into `~/.claude/skills/graphify/` and registers it in `~/.claude/CLAUDE.md`. The Python package and all dependencies install automatically on first `/graphify` run - you never touch pip again.
+> The PyPI package is temporarily named `graphifyy` while the `graphify` name is being reclaimed. The CLI and skill command are still `graphify`.
Then open Claude Code in any directory and type:
@@ -67,17 +38,13 @@ Then open Claude Code in any directory and type:
<details>
<summary>Manual install (curl)</summary>
-**Step 1 - copy the skill file**
-
```bash
mkdir -p ~/.claude/skills/graphify
curl -fsSL https://raw.githubusercontent.com/safishamsi/graphify/v1/skills/graphify/skill.md \
> ~/.claude/skills/graphify/SKILL.md
```
-**Step 2 - register it in Claude Code**
-
-Add this to `~/.claude/CLAUDE.md` (create the file if it doesn't exist):
+Add to `~/.claude/CLAUDE.md`:
```
- **graphify** (`~/.claude/skills/graphify/SKILL.md`) - any input to knowledge graph. Trigger: `/graphify`
@@ -88,157 +55,67 @@ When the user types `/graphify`, invoke the Skill tool with `skill: "graphify"`
## Usage
-All commands are typed inside Claude Code:
-
```
/graphify # run on current directory
/graphify ./raw # run on a specific folder
/graphify ./raw --mode deep # more aggressive INFERRED edge extraction
/graphify ./raw --update # re-extract only changed files, merge into existing graph
-/graphify ./raw --watch # notify when new files appear
/graphify add https://arxiv.org/abs/1706.03762 # fetch a paper, save, update graph
/graphify add https://x.com/karpathy/status/... # fetch a tweet
-/graphify add <url> --author "Karpathy" --contributor "safi"
-/graphify query "what connects attention to the optimizer?" # BFS - broad context
-/graphify query "how does the encoder reach the loss?" --dfs # DFS - trace a path
-/graphify query "..." --budget 1500 # cap at N tokens
+/graphify query "what connects attention to the optimizer?"
+/graphify path "DigestAuth" "Response"
+/graphify explain "SwinTransformer"
-/graphify path "DigestAuth" "Response" # shortest path between two concepts
-/graphify explain "SwinTransformer" # plain-language node explanation
-
-/graphify ./raw --svg # also export graph.svg (embeds in Notion, GitHub)
-/graphify ./raw --graphml # also export graph.graphml (Gephi, yEd, any GraphML tool)
-/graphify ./raw --neo4j # generate cypher.txt for Neo4j import
-/graphify ./raw --mcp # start MCP stdio server for agent access
+/graphify ./raw --svg # export graph.svg
+/graphify ./raw --graphml # export graph.graphml (Gephi, yEd)
+/graphify ./raw --neo4j # generate cypher.txt for Neo4j
+/graphify ./raw --mcp # start MCP stdio server
```
-Works with any mix of file types in the same folder:
+Works with any mix of file types:
-| Type | Extensions | How it's extracted |
-|------|-----------|-------------------|
-| Code | `.py .ts .tsx .js .go .rs .java .c .cpp .rb .cs .kt .scala .php` | AST via tree-sitter (deterministic) + call-graph pass (INFERRED) |
-| Documents | `.md .txt .rst` | Concepts + relationships via Claude |
+| Type | Extensions | Extraction |
+|------|-----------|------------|
+| Code | `.py .ts .js .go .rs .java .c .cpp .rb .cs .kt .scala .php` | AST via tree-sitter + call-graph pass |
+| Docs | `.md .txt .rst` | Concepts + relationships via Claude |
| Papers | `.pdf` | Citation mining + concept extraction |
-| Images | `.png .jpg .webp .gif .svg` | Claude vision - screenshots, charts, whiteboards, any language |
+| Images | `.png .jpg .webp .gif` | Claude vision - screenshots, diagrams, any language |
## What you get
-After running, Claude outputs three things directly in chat:
-
**God nodes** - highest-degree concepts (what everything connects through)
-**Surprising connections** - ranked by a composite surprise score, not just confidence. A code-paper edge scores higher than code-code. A cross-repo connection scores higher than same-repo. Each result includes a plain-English `why` explaining what makes it non-obvious.
-
-**Suggested questions** - 4-5 questions the graph is uniquely positioned to answer, with the reason why (which bridge node makes it interesting, which community boundary it crosses)
-
-The full GRAPH_REPORT.md adds community summaries with cohesion scores and a list of ambiguous edges for review.
-
-**Token reduction benchmark** - automatically printed after every run on corpora over 5,000 words. Shows how many fewer tokens querying the graph costs vs reading the raw files directly.
-
-## Key files explained
+**Surprising connections** - ranked by composite score. Code-paper edges rank higher than code-code. Each result includes a plain-English why.
-| File | Purpose |
-|------|---------|
-| `graph.html` | Interactive vis.js graph. Node size = degree. Click any node for details + clickable neighbors. Search by name. Filter by community. Opens in any browser. |
-| `GRAPH_REPORT.md` | The audit report. God nodes, surprising connections, community cohesion scores, ambiguous edge list, suggested questions. |
-| `graph.json` | Persistent graph in node-link format. Load it with NetworkX or push to Neo4j. Survives sessions. |
-| `obsidian/` | Wikilink vault. Open in Obsidian → enable graph view → see communities as clusters. Filter by tag, search across everything. |
-| `graphify-out/cache/` | SHA256-based per-file cache. A re-run on an unchanged corpus takes seconds. |
-| `graphify-out/memory/` | Q&A feedback loop. Every `/graphify query` answer is saved here. Next `--update` extracts it into the graph. |
+**Suggested questions** - 4-5 questions the graph is uniquely positioned to answer
-## What this skill will NOT do
+**Token benchmark** - printed automatically after every run. On a mixed corpus (Karpathy repos + papers + images): **71.5x** fewer tokens per query vs reading raw files.
-- **Won't invent edges** - `AMBIGUOUS` exists so uncertain relationships are flagged, not hidden. If the connection isn't clear, it's tagged, not fabricated.
-- **Won't claim the graph is useful when it isn't** - a corpus over 2M words or 200 files gets a cost warning before proceeding.
-- **Won't re-extract unchanged files** - SHA256 cache ensures warm re-runs skip everything that hasn't changed.
-- **Won't visualize graphs over 5,000 nodes** - use `--no-viz` or query instead.
-- **Won't download datasets or set up infrastructure** - graphify reads your files. What you put in the folder is what it works with.
-- **Won't implement baselines or run experiments** - it reads and maps. Analysis is yours.
-
-## Design principles
-
-1. **Extraction quality is everything** - clustering is downstream of it. A bad graph clusters into bad communities. The AST + call-graph pass exists because deterministic beats probabilistic for code.
-2. **Show the numbers** - cohesion is `0.91`, not "good". Token cost is always printed. You know what you spent.
-3. **The best output is what you didn't know** - Surprising Connections is not optional. God nodes you probably already suspected. Cross-community edges are what you came for.
-4. **The graph earns its complexity** - below a certain density, just use Claude directly. The graph adds value when you have more than you can hold in context across sessions.
-5. **What you ask grows the graph** - query results are filed back in automatically. The corpus is not static.
-6. **Honest uncertainty** - `EXTRACTED`, `INFERRED`, `AMBIGUOUS` are not cosmetic labels. They are the difference between trusting the graph and being misled by it.
-
-## Contributing
-
-**Adding worked examples**
-
-Worked examples are the most trust-building part of this project. To add one:
-
-1. Pick a real corpus (people should be able to verify the output)
-2. Run the skill: `/graphify <path>`
-3. Save the full output to `worked/{corpus_slug}/`
-4. Write a `review.md` that honestly evaluates:
- - What the graph got right
- - What edges it correctly flagged AMBIGUOUS
- - Any mistakes or missed connections
- - Any surprising connections that were genuinely surprising
-5. Submit a PR with all of the above
-
-**Improving extraction**
-
-If you find a file type or language where extraction is poor, open an issue with a minimal reproduction case. The best bug reports include: the input file, the extraction output (`graphify-out/cache/` entry), and what was missed or invented.
-
-**Adding domain knowledge**
-
-If corpora in your domain consistently contain structures graphify doesn't extract well (e.g., legal documents, lab notebooks, musical scores), open a discussion with examples.
+Every edge is tagged `EXTRACTED`, `INFERRED`, or `AMBIGUOUS` - you always know what was found vs guessed.
## Worked examples
-| Corpus | Type | Reduction | Eval report |
-|--------|------|-----------|-------------|
-| Karpathy repos + 5 research papers + 4 images | Mixed (code + papers + images) | **71.5x** | [`worked/karpathy-repos/review.md`](worked/karpathy-repos/review.md) |
-| httpx (Python HTTP client) | Codebase (6 files) | small corpus¹ | [`worked/httpx/review.md`](worked/httpx/review.md) + [`GRAPH_REPORT.md`](worked/httpx/GRAPH_REPORT.md) |
-| Mixed corpus (code + paper + Arabic image) | Multi-type (5 files) | small corpus¹ | [`worked/mixed-corpus/review.md`](worked/mixed-corpus/review.md) |
-
-¹ Small corpora fit in a single context window - graph value is structural clarity, not token reduction. Reduction ratios grow with corpus size.
+| Corpus | Type | Reduction | Eval |
+|--------|------|-----------|------|
+| Karpathy repos + 5 papers + 4 images | Mixed | **71.5x** | [`worked/karpathy-repos/review.md`](worked/karpathy-repos/review.md) |
+| httpx (Python HTTP client) | Code | small corpus¹ | [`worked/httpx/review.md`](worked/httpx/review.md) |
+| Code + paper + Arabic image | Multi-type | small corpus¹ | [`worked/mixed-corpus/review.md`](worked/mixed-corpus/review.md) |
-Each includes the full graph output and an honest evaluation of what the skill got right and wrong.
+¹ Small corpora fit in one context window - graph value is structural clarity, not compression.
## Tech stack
-| Layer | Library | Why |
-|-------|---------|-----|
-| Graph | NetworkX | Pure Python, same internals as MS GraphRAG |
-| Community detection | Leiden via graspologic | Better than K-means for sparse graphs |
-| Code parsing | tree-sitter | Multi-language AST, deterministic, zero hallucination |
-| Extraction | Claude (parallel subagents) | Reads anything, outputs structured graph data |
-| Visualization | vis.js (HTML) + Obsidian vault | Interactive browser graph + wikilink vault, no server needed |
+NetworkX + Leiden (graspologic) + tree-sitter + Claude + vis.js. No Neo4j required, no server, runs entirely locally.
-No Neo4j required. No dashboards. No server. Runs entirely locally.
+<details>
+<summary>Contributing</summary>
-## Files
+**Worked examples** are the most trust-building contribution. Run `/graphify` on a real corpus, save output to `worked/{slug}/`, write an honest `review.md` evaluating what the graph got right and wrong, submit a PR.
-```
-graphify/
-├── detect.py detect file types, auto-exclude venvs/caches/node_modules; scan graphify-out/memory/
-├── extract.py AST extraction (13 languages via tree-sitter) + call-graph pass (INFERRED edges)
-├── build.py assemble NetworkX graph from extraction JSON; schema-validates before assembly
-├── cluster.py Leiden community detection, cohesion scoring
-├── analyze.py god nodes, bridge nodes, surprising connections, suggested questions, graph diff
-├── report.py render GRAPH_REPORT.md
-├── export.py Obsidian vault, graph.json, graph.html (vis.js), graph.svg, graph.graphml, Neo4j Cypher, Canvas
-├── ingest.py fetch URLs (arXiv, Twitter/X, PDF, any webpage); save Q&A to graphify-out/memory/
-├── cache.py SHA256-based per-file extraction cache; check_semantic_cache / save_semantic_cache
-├── security.py URL validation (http/https only), safe fetch with size cap, path guards, label sanitisation
-├── validate.py JSON schema checks on extraction output
-├── serve.py MCP stdio server - query_graph, get_node, get_neighbors, shortest_path, god_nodes
-├── benchmark.py token reduction benchmark - corpus tokens vs graph query tokens
-└── watch.py fs watcher, writes flag file when new files appear
-
-skills/graphify/
-└── skill.md the Claude Code skill - the full pipeline the agent runs step by step
-
-ARCHITECTURE.md module responsibilities, extraction schema, how to add a language
-SECURITY.md threat model, mitigations, vulnerability reporting
-worked/ eval reports from real corpora (karpathy-repos, httpx, mixed-corpus)
-tests/ 223 tests, one file per module
-pyproject.toml pip install graphifyy | pip install graphifyy[mcp,neo4j,pdf,watch]
-```
+**Extraction bugs** - open an issue with the input file, the cache entry (`graphify-out/cache/`), and what was missed or invented.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for module responsibilities and how to add a language.
+
+</details>