Skip to content

Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.

docs: graphs

undefined-ui39ee7bb025e2
diff --git a/README.md b/README.md
index ca9a2fc..70c6626 100644
--- a/README.md
+++ b/README.md
@@ -114,7 +114,7 @@ you can see exactly what is coming and open a pull request against any of it.
| [Setup](docs/02-setup/README.md) | Obsidian, Claude Code, `CLAUDE.md`, MCP, projects, git | **written** |
| [Ingestion](docs/03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** |
| [Structuring](docs/04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** |
-| [Graphs](docs/05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress |
+| [Graphs](docs/05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | **written** |
| [Agents](docs/06-agents/README.md) | roles, schedules, hooks, guardrails | in progress |
| [Retrieval](docs/07-retrieval/README.md) | query patterns, search, context budget | in progress |
| [Outputs](docs/08-outputs/README.md) | writing, reports, publishing, learning | in progress |
diff --git a/docs/05-graphs/README.md b/docs/05-graphs/README.md
index 5d9bcb2..7483e67 100644
--- a/docs/05-graphs/README.md
+++ b/docs/05-graphs/README.md
@@ -1,18 +1,25 @@
# Graphs
-The part most second-brain guides skip: what the graph is for and how to make it do work.
+The part most second-brain guides skip. They tell you to open the graph view,
+say it looks like a brain, and move on.
-## Planned
+The graph is not decoration. It is the data structure that makes a wiki answer
+questions a search index cannot, and it has properties you can measure, failure
+shapes you can recognise, and uses beyond looking at it.
-Being written. Outlines for each are in the [roadmap](../ROADMAP.md), and pull requests are welcome.
+This section covers what the graph is actually for, how to read it, how to make
+links carry meaning, and when to move part of it into something more capable
+than markdown.
-- Graph basics for a knowledge base
-- Reading the Obsidian graph view
-- Typed links
-- Graph vs embeddings
-- GraphRAG and where it fits
-- Exporting the graph
-- Graph metrics worth tracking
+## Pages
+
+- [Graph basics for a knowledge base](graph-basics.md)
+- [Reading the Obsidian graph view](obsidian-graph-view.md)
+- [Typed links](typed-links.md)
+- [Graph vs embeddings](graph-vs-vectors.md)
+- [GraphRAG and where it fits](graphrag.md)
+- [Exporting the graph](exporting-your-graph.md)
+- [Graph metrics worth tracking](metrics.md)
---
diff --git a/docs/05-graphs/exporting-your-graph.md b/docs/05-graphs/exporting-your-graph.md
new file mode 100644
index 0000000..a174a2a
--- /dev/null
+++ b/docs/05-graphs/exporting-your-graph.md
@@ -0,0 +1,76 @@
+# Exporting the graph
+
+Obsidian's graph view is a viewer. To compute anything, you need the graph
+outside it.
+
+## Export
+
+```bash
+python3 scripts/graph_export.py ~/brain edges.csv
+python3 scripts/graph_export.py ~/brain graph.graphml --format graphml
+```
+
+The script parses every wikilink into an edge list, carries the page `type` as a
+node attribute, and has no dependencies. CSV for code, GraphML for Gephi.
+
+## Analysis
+
+```python
+import networkx as nx, csv
+
+g = nx.DiGraph()
+with open("edges.csv") as f:
+ next(f)
+ g.add_edges_from(tuple(r) for r in csv.reader(f))
+
+print(nx.number_weakly_connected_components(g))
+print(sorted(nx.pagerank(g).items(), key=lambda kv: -kv[1])[:10])
+print([n for n in g if g.in_degree(n) == 0])
+```
+
+Three questions in ten lines: how fragmented is the vault, which pages are
+structurally central, and what is unreachable.
+
+PageRank on a knowledge graph is worth running once. The top pages are what your
+vault is actually about, which is reliably different from what you would say it
+is about.
+
+## Betweenness, the one worth the compute
+
+```python
+b = nx.betweenness_centrality(g)
+```
+
+Betweenness measures how often a page sits on the shortest path between two
+others. High-betweenness pages are the bridges between clusters.
+
+They matter for two reasons. They are your most valuable pages, holding
+connections no single source contained. And they are the fragile points: if a
+bridge page is wrong or gets deleted in a merge, two areas of the vault silently
+stop informing each other.
+
+## Graph databases
+
+Kuzu is embedded, no server, reads CSV directly, and Cypher queries over your
+own notes are genuinely fun for an evening. Neo4j is the full version and is
+overkill unless you are building something on top.
+
+Be clear about what this is: an analysis layer, not a replacement. The markdown
+stays canonical, the database is a rebuildable derivative. The moment you start
+writing knowledge that exists only in the database, you have lost the
+portability that made the whole approach worth choosing.
+
+## Gephi
+
+For looking rather than computing. Load the GraphML, run ForceAtlas2, size nodes
+by degree, colour by type. It renders large graphs in a way Obsidian cannot, and
+it is what to use if you want a screenshot worth showing.
+
+## Keep it in sync
+
+The export is a snapshot and it is stale immediately. Regenerate it rather than
+maintaining it, and never edit the exported graph expecting the vault to follow.
+
+## Next
+
+[Graph metrics worth tracking](metrics.md)
diff --git a/docs/05-graphs/graph-basics.md b/docs/05-graphs/graph-basics.md
new file mode 100644
index 0000000..6c153ae
--- /dev/null
+++ b/docs/05-graphs/graph-basics.md
@@ -0,0 +1,62 @@
+# Graph basics for a knowledge base
+
+Your vault is a graph whether you plan it or not. Pages are nodes, wikilinks are
+edges. Every structural decision in the previous section was a decision about
+what shape that graph takes.
+
+Four terms are enough to reason about it.
+
+**Degree** is how many links a page has. In-degree is how many point at it,
+out-degree how many it points out. A page with zero in-degree is unreachable by
+following links, which is what makes orphans a problem rather than a cosmetic
+issue.
+
+**Path length** is how many hops separate two pages. This is the one that
+matters for answers: the agent following links from a starting page can only
+reach what is a few hops away before the context budget runs out. Two related
+ideas eight hops apart will never be connected in an answer.
+
+**Clustering** is how much a page's neighbours link to each other. High
+clustering means a genuine topic area. Low clustering with high degree usually
+means a page that collects unrelated things.
+
+**Components** are the disconnected islands. A vault in five components is five
+vaults, and nothing in one will ever inform an answer about another.
+
+## The shapes that go wrong
+
+**The star.** Everything links to two or three hub pages and to nothing else.
+Path length between any two leaves is short, but only through the hub, so the
+graph tells you nothing beyond "these are all in my vault". Usually caused by
+linking to broad concepts instead of specific ones.
+
+**The chain.** Sources link to the concept they introduced and nothing links
+sideways. Common after a bulk import that ran without checking existing pages.
+The vault grows and never connects.
+
+**Islands.** Distinct topic clusters with no bridges. Sometimes legitimate: your
+cooking notes and your database notes genuinely do not connect. Often not: two
+clusters about related things that were built under different vocabulary and
+never linked because the aliases were missing.
+
+## What a healthy graph looks like
+
+Dense within topics, sparse but present between them. Most pages reachable from
+most pages within three or four hops. Hubs that exist but do not dominate. A
+small number of bridge pages that connect clusters, which are usually your most
+valuable pages, because they hold the connections you could not have made from
+one source alone.
+
+## Why any of this matters
+
+When you ask the vault something, the agent starts at the pages that match and
+walks outward. The graph is the search space. A well-shaped graph means the walk
+reaches the relevant material in a few hops; a badly shaped one means it either
+reaches nothing or reaches everything.
+
+That is the whole argument for the linking rules. They are not tidiness, they
+are what determines whether retrieval works.
+
+## Next
+
+[Reading the Obsidian graph view](obsidian-graph-view.md)
diff --git a/docs/05-graphs/graph-vs-vectors.md b/docs/05-graphs/graph-vs-vectors.md
new file mode 100644
index 0000000..b56a0d0
--- /dev/null
+++ b/docs/05-graphs/graph-vs-vectors.md
@@ -0,0 +1,67 @@
+# Graph vs embeddings
+
+The usual advice for asking questions of your own documents is to embed them and
+retrieve by similarity. This guide does not, and the reason is worth
+understanding rather than taking on faith.
+
+## What each is good at
+
+**Embeddings** find text that means something similar to your query, including
+when the wording is completely different. They are good at recall, they need no
+structure, and they work on a pile of documents you have never organised.
+
+**A graph** finds things that are explicitly connected, in a direction you can
+follow, with a reason attached if the links are typed. It is good at
+multi-hop questions and at telling you why two things are related.
+
+## Where similarity fails
+
+The failure is specific: similarity retrieves chunks that resemble the question,
+and the answer to a good question often does not resemble it.
+
+Ask what connects two ideas, and the passages that establish the connection may
+share no vocabulary with either. Ask what you have changed your mind about, and
+no chunk is similar to that, because the answer lives in the relationship
+between two pages written a year apart.
+
+There is also the ranking problem. Twenty chunks that all half-answer the
+question crowd out the one page that answers it fully, because similarity has no
+notion of completeness.
+
+## Where the graph fails
+
+Recall. If a page was never linked, the graph cannot reach it, and the linking
+rules are exactly the thing that degrades quietly when ingestion gets sloppy.
+
+The graph also cannot find the thing you half-remember but cannot name. That is
+a search problem, and full-text search on a vault of a few thousand pages is
+instant, which is why `ripgrep` covers most of this in practice.
+
+## Why a personal vault leans graph
+
+Three reasons specific to this scale.
+
+Your corpus is small. A few thousand pages is not a retrieval problem, it is a
+folder. The infrastructure that embeddings require pays off at millions of
+documents and costs more than it returns below that.
+
+Your corpus is already chunked. Atomic pages are semantic units written by
+something that understood them. Splitting them again into fixed-size chunks
+destroys the structure you paid to create.
+
+Your questions are relational. What do I know about X, what connects X and Y,
+what argues against Z. Those are graph queries.
+
+## Hybrid, when you get there
+
+The combination that makes sense: full-text search or embeddings to find
+entry points, the graph to expand from them.
+
+That way recall comes from search and structure comes from the graph, each doing
+what it is good at. See [adding RAG when you need
+it](../07-retrieval/rag-on-top.md) for the threshold where it starts to be worth
+the maintenance.
+
+## Next
+
+[GraphRAG and where it fits](graphrag.md)
diff --git a/docs/05-graphs/graphrag.md b/docs/05-graphs/graphrag.md
new file mode 100644
index 0000000..29508fc
--- /dev/null
+++ b/docs/05-graphs/graphrag.md
@@ -0,0 +1,61 @@
+# GraphRAG and where it fits
+
+GraphRAG is the approach of building a graph from a corpus first, then answering
+questions against the graph rather than against raw chunks. Microsoft Research
+published the best-known version, "From Local to Global: A Graph RAG Approach to
+Query-Focused Summarization".
+
+The pipeline, in outline: extract entities and relationships from every
+document, build a graph, detect communities of densely connected entities,
+summarise each community, and answer questions by drawing on the community
+summaries rather than on retrieved passages.
+
+## Why it exists
+
+It targets a specific failure of chunk retrieval: questions about the corpus as
+a whole. "What are the main themes here" cannot be answered by retrieving the
+five most similar chunks, because no chunk contains the answer. It has to be
+assembled from everything.
+
+That class of question is exactly what people want from a second brain and
+exactly what similarity search is worst at.
+
+## What your vault already has
+
+Read the pipeline again against a well-maintained wiki and most of it is already
+built.
+
+Entity extraction is what ingestion does when it writes entity pages.
+Relationships are the wikilinks, with types if you added them. The graph is the
+vault. What is missing is the community layer: the detection of clusters and the
+summaries over them.
+
+And that missing piece has a manual equivalent you are already writing.
+Synthesis pages are community summaries, produced by a model that read the
+sources rather than by an algorithm clustering entities.
+
+## Whether to run it
+
+Probably not, at personal scale, and the honest reason is size. The approach
+earns its cost on corpora too large to have been curated by hand. A few thousand
+curated pages with real links do not need automated community detection to find
+their own themes; you can see the clusters in the graph view.
+
+There is a case for it as an audit rather than as a retrieval system. Running
+community detection over your exported graph and comparing the algorithmic
+clusters against your synthesis pages tells you where the vault has themes you
+have not noticed. That is a once-a-quarter exercise, not infrastructure.
+
+## Related research worth reading
+
+**HippoRAG** applies personalised PageRank over a knowledge graph for multi-hop
+questions, which is a more direct analogue of what an agent does walking links
+outward from a starting page.
+
+Read the papers themselves rather than summaries of them. Both are specific
+about the conditions their results hold under, and those conditions are what
+decide whether the approach transfers to your situation.
+
+## Next
+
+[Exporting the graph](exporting-your-graph.md)
diff --git a/docs/05-graphs/metrics.md b/docs/05-graphs/metrics.md
new file mode 100644
index 0000000..2684b67
--- /dev/null
+++ b/docs/05-graphs/metrics.md
@@ -0,0 +1,74 @@
+# Graph metrics worth tracking
+
+Four numbers, checked monthly. Their direction over time tells you whether the
+system is working, which is not obvious from using it day to day.
+
+```bash
+python3 scripts/vault_stats.py ~/brain
+```
+
+## Orphan rate
+
+Pages with no inbound links, as a share of all pages.
+
+The single most diagnostic number in the vault. A rising orphan rate means
+ingestion is writing pages without connecting them, which is the exact failure
+mode that turns a wiki back into a folder.
+
+Under five percent is healthy. Over fifteen and something is broken in the
+ingest instructions, not in your habits.
+
+## Average degree
+
+Links per page.
+
+Below two, the graph is barely connected and retrieval cannot do better than
+search. Between three and eight is the working range for a personal vault. Much
+above ten, check whether links are being added decoratively, which inflates the
+number while making the graph less useful.
+
+Watch it alongside page count. Degree that stays flat while the vault doubles is
+fine; degree that falls means new pages are attaching to less than the existing
+ones did.
+
+## Component count
+
+How many disconnected islands.
+
+Ideally one large component holding most pages, plus a few genuinely separate
+topics. If the main component holds less than about eighty percent of pages, the
+vault is fragmented and answers will be systematically incomplete without
+anything appearing wrong.
+
+## Stale page rate
+
+Concept pages whose `updated` date is older than ninety days, as a share of
+concept pages.
+
+Some staleness is correct: a well-written page on a settled idea does not need
+touching. But a vault where most concept pages have not been touched in months
+while sources keep arriving means new material is landing in new pages instead
+of updating existing ones. That is how you end up with four pages about one
+thing.
+
+```dataview
+TABLE updated FROM "wiki/concepts"
+WHERE updated < date(today) - dur(90 days)
+SORT updated ASC
+```
+
+## What not to track
+
+Page count and word count. Both go up regardless of whether the vault is getting
+better, which makes them satisfying and useless. A vault that doubled its pages
+while doubling its orphan rate got worse.
+
+## Recording them
+
+Append the four numbers to a note monthly. Trend is the information; a single
+reading tells you almost nothing, and you will not remember what last quarter's
+orphan rate was.
+
+## Next
+
+[Agents](../06-agents/README.md), for running all of this without you.
diff --git a/docs/05-graphs/obsidian-graph-view.md b/docs/05-graphs/obsidian-graph-view.md
new file mode 100644
index 0000000..7d273ec
--- /dev/null
+++ b/docs/05-graphs/obsidian-graph-view.md
@@ -0,0 +1,58 @@
+# Reading the Obsidian graph view
+
+The global graph is the screenshot everyone posts. It is the least useful view
+in the app.
+
+At two hundred pages it is a hairball, and a hairball tells you nothing except
+that you have pages. What it is good for is spotting shape at a glance: isolated
+clumps, a single dominant hub, a scatter of unconnected dots around the edge.
+
+## Local graph
+
+This is the view that does work. Open a page, show the local graph, set depth to
+two.
+
+You are looking for two things. What is one hop away should be obviously
+related; anything surprising there is either a decorative link to delete or a
+connection worth reading. What is missing at two hops tells you where the page
+is under-connected.
+
+Reviewing the local graph of a page you just wrote is the fastest quality check
+there is, and it takes five seconds.
+
+## Filters that make the global view usable
+
+- **Filter by type.** `-path:sources` hides source pages and leaves the concept
+ graph, which is the part with meaning in it. Sources outnumber concepts
+ heavily and drown them visually.
+- **Groups by colour.** Colour concepts, entities and synthesis differently. A
+ cluster with no concept pages in it is a pile of sources nobody has processed.
+- **Orphan toggle.** Turn orphans on deliberately, look at what appears, then
+ fix or delete them.
+- **Depth and forces.** Lower link force spreads clusters apart enough to see
+ bridges.
+
+## What to look for
+
+**Dots at the edge, connected to one thing.** Failed ingests, usually. The
+source page was written and the concepts never linked back.
+
+**One giant node in the middle.** A hub that has swallowed the graph. Split it.
+
+**Two clusters with a single connecting page.** That page is doing real work.
+It is also fragile: if it is wrong, both clusters are wrong about each other.
+
+**A cluster you do not recognise.** Usually the most interesting thing on
+screen. It means the vault has accumulated material on a topic you have not
+consciously been tracking.
+
+## What the view cannot tell you
+
+It shows that pages are connected, never why. Every edge looks the same whether
+it means "supports", "contradicts" or "mentioned once in passing".
+
+That limitation is the entire argument for the next page.
+
+## Next
+
+[Typed links](typed-links.md)
diff --git a/docs/05-graphs/typed-links.md b/docs/05-graphs/typed-links.md
new file mode 100644
index 0000000..d9cb13e
--- /dev/null
+++ b/docs/05-graphs/typed-links.md
@@ -0,0 +1,83 @@
+# Typed links
+
+A plain `[[wikilink]]` says two pages are related. It does not say how, and at
+scale that is a real loss.
+
+You cannot ask an untyped graph what contradicts a claim, what evidence supports
+it, or which pages extend a method. Every edge is the same edge, so every
+question collapses to "what is nearby".
+
+## A minimal vocabulary
+
+Small enough to remember, expressive enough to be worth it:
+
+- **supports** - this page is evidence for that one
+- **contradicts** - this page argues against it
+- **extends** - builds on it, adds to it
+- **part-of** - a component of a larger thing
+- **applies** - a use of a method in practice
+
+Five is deliberately few. Every additional type is one the agent has to choose
+between correctly on every link, and a vocabulary of twenty produces
+inconsistent tagging, which is worse than none.
+
+## Two ways to encode it
+
+**Inline, next to the link:**
+
+```markdown
+The position sensitivity result (supports:: [[Lost in the Middle]]) suggests
+context length alone does not fix retrieval.
+```
+
+Dataview reads inline fields, so this is queryable inside Obsidian without extra
+tooling.
+
+**In frontmatter, per page:**
+
+```yaml
+contradicts: ["[[Long context replaces RAG]]"]
+extends: ["[[Retrieval augmented generation]]"]
+```
+
+Cleaner to query, further from the sentence the link came from. Frontmatter for
+page-level relationships, inline for claim-level ones.
+
+## What it buys you
+
+```dataview
+LIST FROM "wiki/concepts"
+WHERE contains(contradicts, this.file.link)
+```
+
+That is "everything in my vault that argues against this page", which is a
+question a plain graph cannot answer at all.
+
+At the agent level it is stronger still. An ingest that records a new source as
+`contradicts` an existing page has done the work of flagging the disagreement
+structurally, rather than burying it in prose that only surfaces if someone
+reads the page.
+
+## Do not type everything
+
+Most links are plain mentions and should stay plain. Typing every link makes the
+vocabulary meaningless and makes ingestion slower and more error-prone.
+
+Type the links where the relationship is the point: evidence, disagreement,
+extension. Roughly one in five links, in practice.
+
+## The instruction
+
+```
+When a source supports, contradicts or extends a claim on an existing page,
+record the relationship with a typed link. Plain wikilinks for ordinary
+mentions. Do not type a relationship you are inferring; only type what the
+source states.
+```
+
+That last sentence matters. An agent guessing at relationships produces a graph
+that looks rigorous and encodes its own assumptions.
+
+## Next
+
+[Graph vs embeddings](graph-vs-vectors.md)
diff --git a/docs/README.md b/docs/README.md
index 1f4cf1a..81895a6 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -10,7 +10,7 @@ Setup is written in full. The rest is being written section by section; see the
| [Setup](02-setup/README.md) | Obsidian, Claude Code, CLAUDE.md, MCP, projects, git | **written** |
| [Ingestion](03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** |
| [Structuring](04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** |
-| [Graphs](05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress |
+| [Graphs](05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | **written** |
| [Agents](06-agents/README.md) | roles, schedules, hooks, guardrails | in progress |
| [Retrieval](07-retrieval/README.md) | query patterns, search, context budget | in progress |
| [Outputs](08-outputs/README.md) | writing, reports, publishing, learning | in progress |
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index 86400cc..83a5c03 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -171,41 +171,41 @@ The rules that decide whether your vault becomes a graph or a landfill.
- Using the log to audit and roll back
- Keeping both files from growing unbounded
-## Graphs (outline)
+## Graphs (done)
The part most second-brain guides skip: what the graph is for and how to make it do work.
-- [ ] **Graph basics for a knowledge base**
+- [x] **Graph basics for a knowledge base**
- Nodes, edges, direction and weight in plain terms
- Why a knowledge base is a graph whether you plan it or not
- Density, clustering, path length and what they tell you
- The failure shapes: stars, chains and islands
-- [ ] **Reading the Obsidian graph view**
+- [x] **Reading the Obsidian graph view**
- Local vs global graph
- Filters and groups that make it useful
- Spotting orphans and over-linked hubs
- Why a pretty graph is not a healthy one
-- [ ] **Typed links**
+- [x] **Typed links**
- Untyped wikilinks lose meaning at scale
- A minimal relation vocabulary: supports, contradicts, extends, part-of
- Encoding types in frontmatter or inline
- Querying by relation type
-- [ ] **Graph vs embeddings**
+- [x] **Graph vs embeddings**
- What each retrieval method is actually good at
- Why similarity misses reasoning across sources
- Hybrid setups
- Cost and maintenance compared
-- [ ] **GraphRAG and where it fits**
+- [x] **GraphRAG and where it fits**
- The idea in one page
- Entity and relation extraction from your own vault
- Community detection and summary layers
- Whether a personal vault is big enough to need it
-- [ ] **Exporting the graph**
+- [x] **Exporting the graph**
- Parsing wikilinks into an edge list
- Loading into Kuzu, Neo4j or NetworkX
- Visualizing outside Obsidian
- Keeping the export in sync
-- [ ] **Graph metrics worth tracking**
+- [x] **Graph metrics worth tracking**
- Orphan rate
- Average degree and its healthy range
- Clusters as evidence of your real interests