Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.
JavaScript agent-skillsai-agentsai-engineeringclaudeclaude-codeclaude-skillsknowledge-graphknowledge-managementllmmcpobsidianpersonal-knowledge-basepkmsecond-brain
docs: graphs
undefined-ui39ee7bb025e2
diff --git a/README.md b/README.mdindex ca9a2fc..70c6626 100644--- a/README.md+++ b/README.md@@ -114,7 +114,7 @@ you can see exactly what is coming and open a pull request against any of it.| [Setup](docs/02-setup/README.md) | Obsidian, Claude Code, `CLAUDE.md`, MCP, projects, git | **written** || [Ingestion](docs/03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** || [Structuring](docs/04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** |-| [Graphs](docs/05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress |+| [Graphs](docs/05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | **written** || [Agents](docs/06-agents/README.md) | roles, schedules, hooks, guardrails | in progress || [Retrieval](docs/07-retrieval/README.md) | query patterns, search, context budget | in progress || [Outputs](docs/08-outputs/README.md) | writing, reports, publishing, learning | in progress |diff --git a/docs/05-graphs/README.md b/docs/05-graphs/README.mdindex 5d9bcb2..7483e67 100644--- a/docs/05-graphs/README.md+++ b/docs/05-graphs/README.md@@ -1,18 +1,25 @@# Graphs-The part most second-brain guides skip: what the graph is for and how to make it do work.+The part most second-brain guides skip. They tell you to open the graph view,+say it looks like a brain, and move on.-## Planned+The graph is not decoration. It is the data structure that makes a wiki answer+questions a search index cannot, and it has properties you can measure, failure+shapes you can recognise, and uses beyond looking at it.-Being written. Outlines for each are in the [roadmap](../ROADMAP.md), and pull requests are welcome.+This section covers what the graph is actually for, how to read it, how to make+links carry meaning, and when to move part of it into something more capable+than markdown.-- Graph basics for a knowledge base-- Reading the Obsidian graph view-- Typed links-- Graph vs embeddings-- GraphRAG and where it fits-- Exporting the graph-- Graph metrics worth tracking+## Pages++- [Graph basics for a knowledge base](graph-basics.md)+- [Reading the Obsidian graph view](obsidian-graph-view.md)+- [Typed links](typed-links.md)+- [Graph vs embeddings](graph-vs-vectors.md)+- [GraphRAG and where it fits](graphrag.md)+- [Exporting the graph](exporting-your-graph.md)+- [Graph metrics worth tracking](metrics.md)---diff --git a/docs/05-graphs/exporting-your-graph.md b/docs/05-graphs/exporting-your-graph.mdnew file mode 100644index 0000000..a174a2a--- /dev/null+++ b/docs/05-graphs/exporting-your-graph.md@@ -0,0 +1,76 @@+# Exporting the graph++Obsidian's graph view is a viewer. To compute anything, you need the graph+outside it.++## Export++```bash+python3 scripts/graph_export.py ~/brain edges.csv+python3 scripts/graph_export.py ~/brain graph.graphml --format graphml+```++The script parses every wikilink into an edge list, carries the page `type` as a+node attribute, and has no dependencies. CSV for code, GraphML for Gephi.++## Analysis++```python+import networkx as nx, csv++g = nx.DiGraph()+with open("edges.csv") as f:+ next(f)+ g.add_edges_from(tuple(r) for r in csv.reader(f))++print(nx.number_weakly_connected_components(g))+print(sorted(nx.pagerank(g).items(), key=lambda kv: -kv[1])[:10])+print([n for n in g if g.in_degree(n) == 0])+```++Three questions in ten lines: how fragmented is the vault, which pages are+structurally central, and what is unreachable.++PageRank on a knowledge graph is worth running once. The top pages are what your+vault is actually about, which is reliably different from what you would say it+is about.++## Betweenness, the one worth the compute++```python+b = nx.betweenness_centrality(g)+```++Betweenness measures how often a page sits on the shortest path between two+others. High-betweenness pages are the bridges between clusters.++They matter for two reasons. They are your most valuable pages, holding+connections no single source contained. And they are the fragile points: if a+bridge page is wrong or gets deleted in a merge, two areas of the vault silently+stop informing each other.++## Graph databases++Kuzu is embedded, no server, reads CSV directly, and Cypher queries over your+own notes are genuinely fun for an evening. Neo4j is the full version and is+overkill unless you are building something on top.++Be clear about what this is: an analysis layer, not a replacement. The markdown+stays canonical, the database is a rebuildable derivative. The moment you start+writing knowledge that exists only in the database, you have lost the+portability that made the whole approach worth choosing.++## Gephi++For looking rather than computing. Load the GraphML, run ForceAtlas2, size nodes+by degree, colour by type. It renders large graphs in a way Obsidian cannot, and+it is what to use if you want a screenshot worth showing.++## Keep it in sync++The export is a snapshot and it is stale immediately. Regenerate it rather than+maintaining it, and never edit the exported graph expecting the vault to follow.++## Next++[Graph metrics worth tracking](metrics.md)diff --git a/docs/05-graphs/graph-basics.md b/docs/05-graphs/graph-basics.mdnew file mode 100644index 0000000..6c153ae--- /dev/null+++ b/docs/05-graphs/graph-basics.md@@ -0,0 +1,62 @@+# Graph basics for a knowledge base++Your vault is a graph whether you plan it or not. Pages are nodes, wikilinks are+edges. Every structural decision in the previous section was a decision about+what shape that graph takes.++Four terms are enough to reason about it.++**Degree** is how many links a page has. In-degree is how many point at it,+out-degree how many it points out. A page with zero in-degree is unreachable by+following links, which is what makes orphans a problem rather than a cosmetic+issue.++**Path length** is how many hops separate two pages. This is the one that+matters for answers: the agent following links from a starting page can only+reach what is a few hops away before the context budget runs out. Two related+ideas eight hops apart will never be connected in an answer.++**Clustering** is how much a page's neighbours link to each other. High+clustering means a genuine topic area. Low clustering with high degree usually+means a page that collects unrelated things.++**Components** are the disconnected islands. A vault in five components is five+vaults, and nothing in one will ever inform an answer about another.++## The shapes that go wrong++**The star.** Everything links to two or three hub pages and to nothing else.+Path length between any two leaves is short, but only through the hub, so the+graph tells you nothing beyond "these are all in my vault". Usually caused by+linking to broad concepts instead of specific ones.++**The chain.** Sources link to the concept they introduced and nothing links+sideways. Common after a bulk import that ran without checking existing pages.+The vault grows and never connects.++**Islands.** Distinct topic clusters with no bridges. Sometimes legitimate: your+cooking notes and your database notes genuinely do not connect. Often not: two+clusters about related things that were built under different vocabulary and+never linked because the aliases were missing.++## What a healthy graph looks like++Dense within topics, sparse but present between them. Most pages reachable from+most pages within three or four hops. Hubs that exist but do not dominate. A+small number of bridge pages that connect clusters, which are usually your most+valuable pages, because they hold the connections you could not have made from+one source alone.++## Why any of this matters++When you ask the vault something, the agent starts at the pages that match and+walks outward. The graph is the search space. A well-shaped graph means the walk+reaches the relevant material in a few hops; a badly shaped one means it either+reaches nothing or reaches everything.++That is the whole argument for the linking rules. They are not tidiness, they+are what determines whether retrieval works.++## Next++[Reading the Obsidian graph view](obsidian-graph-view.md)diff --git a/docs/05-graphs/graph-vs-vectors.md b/docs/05-graphs/graph-vs-vectors.mdnew file mode 100644index 0000000..b56a0d0--- /dev/null+++ b/docs/05-graphs/graph-vs-vectors.md@@ -0,0 +1,67 @@+# Graph vs embeddings++The usual advice for asking questions of your own documents is to embed them and+retrieve by similarity. This guide does not, and the reason is worth+understanding rather than taking on faith.++## What each is good at++**Embeddings** find text that means something similar to your query, including+when the wording is completely different. They are good at recall, they need no+structure, and they work on a pile of documents you have never organised.++**A graph** finds things that are explicitly connected, in a direction you can+follow, with a reason attached if the links are typed. It is good at+multi-hop questions and at telling you why two things are related.++## Where similarity fails++The failure is specific: similarity retrieves chunks that resemble the question,+and the answer to a good question often does not resemble it.++Ask what connects two ideas, and the passages that establish the connection may+share no vocabulary with either. Ask what you have changed your mind about, and+no chunk is similar to that, because the answer lives in the relationship+between two pages written a year apart.++There is also the ranking problem. Twenty chunks that all half-answer the+question crowd out the one page that answers it fully, because similarity has no+notion of completeness.++## Where the graph fails++Recall. If a page was never linked, the graph cannot reach it, and the linking+rules are exactly the thing that degrades quietly when ingestion gets sloppy.++The graph also cannot find the thing you half-remember but cannot name. That is+a search problem, and full-text search on a vault of a few thousand pages is+instant, which is why `ripgrep` covers most of this in practice.++## Why a personal vault leans graph++Three reasons specific to this scale.++Your corpus is small. A few thousand pages is not a retrieval problem, it is a+folder. The infrastructure that embeddings require pays off at millions of+documents and costs more than it returns below that.++Your corpus is already chunked. Atomic pages are semantic units written by+something that understood them. Splitting them again into fixed-size chunks+destroys the structure you paid to create.++Your questions are relational. What do I know about X, what connects X and Y,+what argues against Z. Those are graph queries.++## Hybrid, when you get there++The combination that makes sense: full-text search or embeddings to find+entry points, the graph to expand from them.++That way recall comes from search and structure comes from the graph, each doing+what it is good at. See [adding RAG when you need+it](../07-retrieval/rag-on-top.md) for the threshold where it starts to be worth+the maintenance.++## Next++[GraphRAG and where it fits](graphrag.md)diff --git a/docs/05-graphs/graphrag.md b/docs/05-graphs/graphrag.mdnew file mode 100644index 0000000..29508fc--- /dev/null+++ b/docs/05-graphs/graphrag.md@@ -0,0 +1,61 @@+# GraphRAG and where it fits++GraphRAG is the approach of building a graph from a corpus first, then answering+questions against the graph rather than against raw chunks. Microsoft Research+published the best-known version, "From Local to Global: A Graph RAG Approach to+Query-Focused Summarization".++The pipeline, in outline: extract entities and relationships from every+document, build a graph, detect communities of densely connected entities,+summarise each community, and answer questions by drawing on the community+summaries rather than on retrieved passages.++## Why it exists++It targets a specific failure of chunk retrieval: questions about the corpus as+a whole. "What are the main themes here" cannot be answered by retrieving the+five most similar chunks, because no chunk contains the answer. It has to be+assembled from everything.++That class of question is exactly what people want from a second brain and+exactly what similarity search is worst at.++## What your vault already has++Read the pipeline again against a well-maintained wiki and most of it is already+built.++Entity extraction is what ingestion does when it writes entity pages.+Relationships are the wikilinks, with types if you added them. The graph is the+vault. What is missing is the community layer: the detection of clusters and the+summaries over them.++And that missing piece has a manual equivalent you are already writing.+Synthesis pages are community summaries, produced by a model that read the+sources rather than by an algorithm clustering entities.++## Whether to run it++Probably not, at personal scale, and the honest reason is size. The approach+earns its cost on corpora too large to have been curated by hand. A few thousand+curated pages with real links do not need automated community detection to find+their own themes; you can see the clusters in the graph view.++There is a case for it as an audit rather than as a retrieval system. Running+community detection over your exported graph and comparing the algorithmic+clusters against your synthesis pages tells you where the vault has themes you+have not noticed. That is a once-a-quarter exercise, not infrastructure.++## Related research worth reading++**HippoRAG** applies personalised PageRank over a knowledge graph for multi-hop+questions, which is a more direct analogue of what an agent does walking links+outward from a starting page.++Read the papers themselves rather than summaries of them. Both are specific+about the conditions their results hold under, and those conditions are what+decide whether the approach transfers to your situation.++## Next++[Exporting the graph](exporting-your-graph.md)diff --git a/docs/05-graphs/metrics.md b/docs/05-graphs/metrics.mdnew file mode 100644index 0000000..2684b67--- /dev/null+++ b/docs/05-graphs/metrics.md@@ -0,0 +1,74 @@+# Graph metrics worth tracking++Four numbers, checked monthly. Their direction over time tells you whether the+system is working, which is not obvious from using it day to day.++```bash+python3 scripts/vault_stats.py ~/brain+```++## Orphan rate++Pages with no inbound links, as a share of all pages.++The single most diagnostic number in the vault. A rising orphan rate means+ingestion is writing pages without connecting them, which is the exact failure+mode that turns a wiki back into a folder.++Under five percent is healthy. Over fifteen and something is broken in the+ingest instructions, not in your habits.++## Average degree++Links per page.++Below two, the graph is barely connected and retrieval cannot do better than+search. Between three and eight is the working range for a personal vault. Much+above ten, check whether links are being added decoratively, which inflates the+number while making the graph less useful.++Watch it alongside page count. Degree that stays flat while the vault doubles is+fine; degree that falls means new pages are attaching to less than the existing+ones did.++## Component count++How many disconnected islands.++Ideally one large component holding most pages, plus a few genuinely separate+topics. If the main component holds less than about eighty percent of pages, the+vault is fragmented and answers will be systematically incomplete without+anything appearing wrong.++## Stale page rate++Concept pages whose `updated` date is older than ninety days, as a share of+concept pages.++Some staleness is correct: a well-written page on a settled idea does not need+touching. But a vault where most concept pages have not been touched in months+while sources keep arriving means new material is landing in new pages instead+of updating existing ones. That is how you end up with four pages about one+thing.++```dataview+TABLE updated FROM "wiki/concepts"+WHERE updated < date(today) - dur(90 days)+SORT updated ASC+```++## What not to track++Page count and word count. Both go up regardless of whether the vault is getting+better, which makes them satisfying and useless. A vault that doubled its pages+while doubling its orphan rate got worse.++## Recording them++Append the four numbers to a note monthly. Trend is the information; a single+reading tells you almost nothing, and you will not remember what last quarter's+orphan rate was.++## Next++[Agents](../06-agents/README.md), for running all of this without you.diff --git a/docs/05-graphs/obsidian-graph-view.md b/docs/05-graphs/obsidian-graph-view.mdnew file mode 100644index 0000000..7d273ec--- /dev/null+++ b/docs/05-graphs/obsidian-graph-view.md@@ -0,0 +1,58 @@+# Reading the Obsidian graph view++The global graph is the screenshot everyone posts. It is the least useful view+in the app.++At two hundred pages it is a hairball, and a hairball tells you nothing except+that you have pages. What it is good for is spotting shape at a glance: isolated+clumps, a single dominant hub, a scatter of unconnected dots around the edge.++## Local graph++This is the view that does work. Open a page, show the local graph, set depth to+two.++You are looking for two things. What is one hop away should be obviously+related; anything surprising there is either a decorative link to delete or a+connection worth reading. What is missing at two hops tells you where the page+is under-connected.++Reviewing the local graph of a page you just wrote is the fastest quality check+there is, and it takes five seconds.++## Filters that make the global view usable++- **Filter by type.** `-path:sources` hides source pages and leaves the concept+ graph, which is the part with meaning in it. Sources outnumber concepts+ heavily and drown them visually.+- **Groups by colour.** Colour concepts, entities and synthesis differently. A+ cluster with no concept pages in it is a pile of sources nobody has processed.+- **Orphan toggle.** Turn orphans on deliberately, look at what appears, then+ fix or delete them.+- **Depth and forces.** Lower link force spreads clusters apart enough to see+ bridges.++## What to look for++**Dots at the edge, connected to one thing.** Failed ingests, usually. The+source page was written and the concepts never linked back.++**One giant node in the middle.** A hub that has swallowed the graph. Split it.++**Two clusters with a single connecting page.** That page is doing real work.+It is also fragile: if it is wrong, both clusters are wrong about each other.++**A cluster you do not recognise.** Usually the most interesting thing on+screen. It means the vault has accumulated material on a topic you have not+consciously been tracking.++## What the view cannot tell you++It shows that pages are connected, never why. Every edge looks the same whether+it means "supports", "contradicts" or "mentioned once in passing".++That limitation is the entire argument for the next page.++## Next++[Typed links](typed-links.md)diff --git a/docs/05-graphs/typed-links.md b/docs/05-graphs/typed-links.mdnew file mode 100644index 0000000..d9cb13e--- /dev/null+++ b/docs/05-graphs/typed-links.md@@ -0,0 +1,83 @@+# Typed links++A plain `[[wikilink]]` says two pages are related. It does not say how, and at+scale that is a real loss.++You cannot ask an untyped graph what contradicts a claim, what evidence supports+it, or which pages extend a method. Every edge is the same edge, so every+question collapses to "what is nearby".++## A minimal vocabulary++Small enough to remember, expressive enough to be worth it:++- **supports** - this page is evidence for that one+- **contradicts** - this page argues against it+- **extends** - builds on it, adds to it+- **part-of** - a component of a larger thing+- **applies** - a use of a method in practice++Five is deliberately few. Every additional type is one the agent has to choose+between correctly on every link, and a vocabulary of twenty produces+inconsistent tagging, which is worse than none.++## Two ways to encode it++**Inline, next to the link:**++```markdown+The position sensitivity result (supports:: [[Lost in the Middle]]) suggests+context length alone does not fix retrieval.+```++Dataview reads inline fields, so this is queryable inside Obsidian without extra+tooling.++**In frontmatter, per page:**++```yaml+contradicts: ["[[Long context replaces RAG]]"]+extends: ["[[Retrieval augmented generation]]"]+```++Cleaner to query, further from the sentence the link came from. Frontmatter for+page-level relationships, inline for claim-level ones.++## What it buys you++```dataview+LIST FROM "wiki/concepts"+WHERE contains(contradicts, this.file.link)+```++That is "everything in my vault that argues against this page", which is a+question a plain graph cannot answer at all.++At the agent level it is stronger still. An ingest that records a new source as+`contradicts` an existing page has done the work of flagging the disagreement+structurally, rather than burying it in prose that only surfaces if someone+reads the page.++## Do not type everything++Most links are plain mentions and should stay plain. Typing every link makes the+vocabulary meaningless and makes ingestion slower and more error-prone.++Type the links where the relationship is the point: evidence, disagreement,+extension. Roughly one in five links, in practice.++## The instruction++```+When a source supports, contradicts or extends a claim on an existing page,+record the relationship with a typed link. Plain wikilinks for ordinary+mentions. Do not type a relationship you are inferring; only type what the+source states.+```++That last sentence matters. An agent guessing at relationships produces a graph+that looks rigorous and encodes its own assumptions.++## Next++[Graph vs embeddings](graph-vs-vectors.md)diff --git a/docs/README.md b/docs/README.mdindex 1f4cf1a..81895a6 100644--- a/docs/README.md+++ b/docs/README.md@@ -10,7 +10,7 @@ Setup is written in full. The rest is being written section by section; see the| [Setup](02-setup/README.md) | Obsidian, Claude Code, CLAUDE.md, MCP, projects, git | **written** || [Ingestion](03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** || [Structuring](04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** |-| [Graphs](05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress |+| [Graphs](05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | **written** || [Agents](06-agents/README.md) | roles, schedules, hooks, guardrails | in progress || [Retrieval](07-retrieval/README.md) | query patterns, search, context budget | in progress || [Outputs](08-outputs/README.md) | writing, reports, publishing, learning | in progress |diff --git a/docs/ROADMAP.md b/docs/ROADMAP.mdindex 86400cc..83a5c03 100644--- a/docs/ROADMAP.md+++ b/docs/ROADMAP.md@@ -171,41 +171,41 @@ The rules that decide whether your vault becomes a graph or a landfill.- Using the log to audit and roll back- Keeping both files from growing unbounded-## Graphs (outline)+## Graphs (done)The part most second-brain guides skip: what the graph is for and how to make it do work.-- [ ] **Graph basics for a knowledge base**+- [x] **Graph basics for a knowledge base**- Nodes, edges, direction and weight in plain terms- Why a knowledge base is a graph whether you plan it or not- Density, clustering, path length and what they tell you- The failure shapes: stars, chains and islands-- [ ] **Reading the Obsidian graph view**+- [x] **Reading the Obsidian graph view**- Local vs global graph- Filters and groups that make it useful- Spotting orphans and over-linked hubs- Why a pretty graph is not a healthy one-- [ ] **Typed links**+- [x] **Typed links**- Untyped wikilinks lose meaning at scale- A minimal relation vocabulary: supports, contradicts, extends, part-of- Encoding types in frontmatter or inline- Querying by relation type-- [ ] **Graph vs embeddings**+- [x] **Graph vs embeddings**- What each retrieval method is actually good at- Why similarity misses reasoning across sources- Hybrid setups- Cost and maintenance compared-- [ ] **GraphRAG and where it fits**+- [x] **GraphRAG and where it fits**- The idea in one page- Entity and relation extraction from your own vault- Community detection and summary layers- Whether a personal vault is big enough to need it-- [ ] **Exporting the graph**+- [x] **Exporting the graph**- Parsing wikilinks into an edge list- Loading into Kuzu, Neo4j or NetworkX- Visualizing outside Obsidian- Keeping the export in sync-- [ ] **Graph metrics worth tracking**+- [x] **Graph metrics worth tracking**- Orphan rate- Average degree and its healthy range- Clusters as evidence of your real interests