Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.
JavaScript agent-skillsai-agentsai-engineeringclaudeclaude-codeclaude-skillsknowledge-graphknowledge-managementllmmcpobsidianpersonal-knowledge-basepkmsecond-brain
docs: structuring
undefined-ui4ceaa56d5d80
diff --git a/README.md b/README.mdindex 78ddb43..ca9a2fc 100644--- a/README.md+++ b/README.md@@ -113,7 +113,7 @@ you can see exactly what is coming and open a pull request against any of it.| [Concepts](docs/01-concepts/README.md) | what the pattern is and why the old note systems died | partial || [Setup](docs/02-setup/README.md) | Obsidian, Claude Code, `CLAUDE.md`, MCP, projects, git | **written** || [Ingestion](docs/03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** |-| [Structuring](docs/04-structuring/README.md) | page types, linking rules, schema, contradictions | in progress |+| [Structuring](docs/04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** || [Graphs](docs/05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress || [Agents](docs/06-agents/README.md) | roles, schedules, hooks, guardrails | in progress || [Retrieval](docs/07-retrieval/README.md) | query patterns, search, context budget | in progress |diff --git a/docs/04-structuring/README.md b/docs/04-structuring/README.mdindex 0c64fe5..15cc229 100644--- a/docs/04-structuring/README.md+++ b/docs/04-structuring/README.md@@ -2,18 +2,25 @@The rules that decide whether your vault becomes a graph or a landfill.-## Planned+Ingestion determines what goes in. Structure determines whether any of it comes+back out. Two vaults holding the same thousand sources can be a research+partner or an unsearchable pile, and the difference is entirely in the rules+below.-Being written. Outlines for each are in the [roadmap](../ROADMAP.md), and pull requests are welcome.+Everything here belongs in your `CLAUDE.md`, because a rule the agent reads once+in a chat lasts one session, and a rule in `CLAUDE.md` applies to every page it+ever writes.-- One page, one idea-- The four page types-- Linking rules-- Frontmatter schema-- Naming and aliases-- Deduplication and merging-- Contradictions and supersession-- index.md and log.md+## Pages++- [One page, one idea](atomic-pages.md)+- [The four page types](page-types.md)+- [Linking rules](linking-rules.md)+- [Frontmatter schema](frontmatter-schema.md)+- [Naming and aliases](naming-and-aliases.md)+- [Deduplication and merging](dedupe-and-merge.md)+- [Contradictions and supersession](contradictions-and-supersession.md)+- [index.md and log.md](index-and-log.md)---diff --git a/docs/04-structuring/atomic-pages.md b/docs/04-structuring/atomic-pages.mdnew file mode 100644index 0000000..b8f342b--- /dev/null+++ b/docs/04-structuring/atomic-pages.md@@ -0,0 +1,55 @@+# One page, one idea++A page holds one idea, explained well enough that someone who has never seen the+source understands it.++The rule exists because of how retrieval works. When the agent answers a+question, it pulls in whole pages. A page covering four topics drags three+irrelevant ones into context every time it is touched. A page covering one gets+linked precisely, retrieved precisely, and updated without collateral damage.++## What atomic means here++Not short. A concept page can run several hundred words. Atomic means it has one+subject and you could give it an accurate title without using "and".++If the title needs "and", or the page has section headings for unrelated+subjects, it is two pages.++## Splitting a source++One article usually yields one to three concepts worth their own page, plus the+source page recording what the article itself said.++The test for whether something deserves a page: can it be explained without+referring back to this one source? An idea that only makes sense as "what this+article argued" belongs inside the source page. An idea that will recur across+other sources needs its own page, because that is where the other sources will+attach.++## The failure in both directions++**Over-splitting** produces ten pages from one article, each a restated+paragraph, each linking only back to the source. The graph looks dense and says+nothing. This is the more common failure with agents, because splitting feels+like thoroughness.++**Under-splitting** produces source pages that are article summaries and a wiki+with no concepts at all. You end up with a searchable archive, which you already+had.++If you are unsure, write fewer pages. Merging later is cheap; untangling a+hundred fragments is not.++## Length++A concept page under about eighty words is usually a stub that failed, not a+page that was efficient. Either the source did not really cover it, or the agent+gave up. Both are worth catching in a lint pass.++A page over roughly a thousand words is usually two ideas that grew together.+Read it and look for the seam.++## Next++[The four page types](page-types.md)diff --git a/docs/04-structuring/contradictions-and-supersession.md b/docs/04-structuring/contradictions-and-supersession.mdnew file mode 100644index 0000000..3ee20b9--- /dev/null+++ b/docs/04-structuring/contradictions-and-supersession.md@@ -0,0 +1,86 @@+# Contradictions and supersession++This is the rule that separates a second brain from a search engine, and the one+most implementations get wrong.++When a new source disagrees with what a page already says, the page records+both. It does not overwrite.++## Why overwriting is the worst failure mode++A vault that silently updates to the latest claim has no memory of what you+believed and why. It always presents the most recent thing you read as the+truth, which is a ranking by recency, not by evidence.++Worse, it is invisible. Nothing errors. The page simply says something different+than it did last month, with the same confidence, and you have no way to notice+or to check what changed.++The history is the asset. Knowing that you believed X because of a 2024 source,+then revised to Y after better evidence, is more valuable than either claim+alone, because it tells you how firm Y actually is.++## What a contradiction looks like on a page++Both positions, each attributed and dated:++```markdown+## Current position++Retrieval quality degrades past roughly 50k tokens of context+([[Lost in the Middle]], 2023).++## Contested++[[Some 2026 benchmark]] reports no degradation up to 200k on the same task+family. The two use different evaluation setups: the first measures position+sensitivity, the second end-task accuracy. Unresolved.+```++The reader can act on that. A page that had silently adopted whichever was+ingested last cannot be acted on at all.++## Supersession++Sometimes it is not a genuine disagreement, it is old information. A tool+changed its API, a company was acquired, a paper was retracted.++Mark the old claim as superseded and keep it:++```markdown+~~Requires Node 18 or later~~ superseded 2026-05: the native installer no+longer requires Node ([[source]]).+```++Deleting it means that when you read something written before the change, you+have no way to reconcile it with what the vault now says.++## Confidence++Not everything ingested deserves equal weight. A tag or a frontmatter field is+enough:++- **stated** by a primary source you trust+- **reported** by secondary coverage+- **self-reported** by the party it benefits+- **unverified**++Self-reported numbers in particular should never lose that label. A company's+own performance claim repeated as fact in a concept page is how a vault becomes+confidently wrong.++## The instruction++In `CLAUDE.md`, worded so it generalises rather than covering only the obvious+case:++```+When a new source disagrees with an existing page, do not overwrite. Record both+positions with sources and dates, and say what would settle it. If the new+source is clearly better evidence, mark the older claim superseded rather than+deleting it. Never resolve a contradiction by picking the more recent source.+```++## Next++[index.md and log.md](index-and-log.md)diff --git a/docs/04-structuring/dedupe-and-merge.md b/docs/04-structuring/dedupe-and-merge.mdnew file mode 100644index 0000000..8ae0341--- /dev/null+++ b/docs/04-structuring/dedupe-and-merge.md@@ -0,0 +1,61 @@+# Deduplication and merging++Duplicates are inevitable. Two sources use different words for the same idea, an+alias was missing, a bulk import ran before the vault had the concept. The+question is not how to prevent them entirely but how to catch them before they+both accumulate links.++## Finding them++Signals, in order of reliability:++- **Similar titles.** Cheap to check, catches the plural and acronym cases.+- **Overlapping aliases.** Two pages claiming the same alias is always a bug.+- **Shared inbound links.** Two pages that the same five sources both link to+ are usually one page.+- **Content similarity.** Expensive, but catches the cases where the names look+ unrelated.++`second-brain-lint` runs the first three and proposes merges. It does not+perform them, which is deliberate.++## Why merging is not automatic++A merge is irreversible in the way that matters: once two pages are one, you+cannot tell which claim came from which, and the agent's judgement about "these+are the same thing" is exactly the judgement that goes wrong.++Two ideas that look identical from their summaries are often a general case and+a specific one, and merging them loses the distinction permanently.++So: the agent proposes, you decide. Reviewing a merge takes fifteen seconds.++## The procedure++When you approve one:++1. **Pick the survivor** by canonical name, not by which is longer.+2. **Merge content,** keeping every distinct claim with its source. Overlapping+ claims collapse, differing ones both stay.+3. **Merge aliases,** including the dead page's title as an alias on the+ survivor.+4. **Redirect inbound links** to the survivor.+5. **Leave a stub** at the old path only if it was linked from outside the+ vault. Otherwise delete it.+6. **Record it in `log.md`** with both names, so a page that vanishes is+ traceable.++Step 6 is the one agents skip and the one you will need. A page that disappears+without a record is indistinguishable from a page that was deleted by mistake.++## Do not dedupe live++Merging during an ingest run means the agent is restructuring the vault while+adding to it, and a bad merge lands in the middle of otherwise good work.++Run dedupe as its own pass, weekly or after a large import, on a clean git+state. Then a wrong merge is one `git checkout` away from undone.++## Next++[Contradictions and supersession](contradictions-and-supersession.md)diff --git a/docs/04-structuring/frontmatter-schema.md b/docs/04-structuring/frontmatter-schema.mdnew file mode 100644index 0000000..7fcd753--- /dev/null+++ b/docs/04-structuring/frontmatter-schema.md@@ -0,0 +1,75 @@+# Frontmatter schema++Frontmatter is what makes the vault queryable by something other than text+search. It costs nothing to write and it is what Dataview, scripts and the agent+itself use to find things by shape.++## The schema++```yaml+---+title: Canonical name+type: source | entity | concept | synthesis+created: YYYY-MM-DD+updated: YYYY-MM-DD+aliases: [other names this is known by]+tags: [two or three]+---+```++Source pages add:++```yaml+url: https://...+author:+published: YYYY-MM-DD+```++Entity pages add `kind: person | org | product | tool`.++## Why each field earns its place++`type` drives every structural query and every lint check. Without it you cannot+ask how many concepts you have, or find sources with no concepts attached.++`updated` is what makes review possible. Stale concept pages are the main way a+vault rots, and you cannot find them without a date.++`aliases` is what makes wikilinks resolve when the same thing has three names.+See [naming and aliases](naming-and-aliases.md).++`url` on a source page is what separates a citation from a claim. A source page+without it is unverifiable.++## Tags++Two or three per page, from a small controlled vocabulary you actually maintain.++Tags fail in one specific way: the vocabulary grows until it is meaningless.+Fifty tags used once each are worse than no tags, because they create the+impression of a taxonomy while doing none of the work.++Keep the list in `CLAUDE.md` and instruct the agent to use existing tags or+propose a new one explicitly rather than inventing them silently.++Folders and links carry the structural load. Tags are for cross-cutting+qualities that do not fit either: `unverified`, `to-revisit`, `disputed`.++## Empty fields++Leave a field out rather than filling it with a guess. An absent `published`+means unknown, which the agent can say honestly. A guessed date becomes a fact+in a concept page three weeks later.++## Keep it stable++Adding a field to the schema means either backfilling every page or living with+a mixed vault where queries miss the older half.++Decide the schema early, keep it small, and change it rarely. When you do change+it, run a backfill pass over the whole vault in one go rather than letting the+change apply only to new pages.++## Next++[Naming and aliases](naming-and-aliases.md)diff --git a/docs/04-structuring/index-and-log.md b/docs/04-structuring/index-and-log.mdnew file mode 100644index 0000000..a844fcb--- /dev/null+++ b/docs/04-structuring/index-and-log.md@@ -0,0 +1,78 @@+# index.md and log.md++Two files that are not knowledge, but without which the knowledge is hard to+reach and impossible to audit.++## index.md++The catalog. Every page, grouped by type, with the gaps listed at the bottom.++Its real job is retrieval economy. When the agent gets a question, reading the+index first tells it what exists and which pages are worth opening. Without it,+the alternative is a full-text sweep, which is slower, costs more context, and+misses pages whose wording differs from the question.++Keep it navigable rather than exhaustive-looking:++```markdown+## Concepts++### Retrieval+- [[Retrieval augmented generation]] - chunking, embeddings, when it beats a wiki+- [[GraphRAG]] - entity graph plus community summaries++### Agents+- [[Agent harness]] - the loop around a model+```++One line of description per entry. That line is what lets the agent decide+whether to open the page, and it is worth the tokens.++## Gaps++At the bottom of the index, the pages that are linked and do not exist.++This section is the most actionable thing in the vault. It is a list of what+your own material keeps referring to and you have never written down, which is+usually a precise map of what to read next.++## log.md++One line per operation, appended, newest at the bottom.++```+2026-09-07 ingest raw/karpathy-gist.md -> 1 source, 2 concepts, 1 entity, 9 links+2026-09-07 merge [[RAG]] <- [[Retrieval-augmented generation]]+2026-09-08 lint -> 3 broken links fixed, 2 orphans flagged+```++The log is what makes an agent-maintained vault auditable. When a page says+something you do not recognise, the log tells you which run produced it. When+something disappears, the log is the difference between a traceable merge and a+mystery.++It also gives the [review skill](../../skills/README.md) its raw material: what+changed this week comes from here, not from guessing at file timestamps.++## Keeping both from bloating++The index grows with the vault, which is fine, but flat lists stop being+navigable somewhere around two hundred entries. Group by theme, and once a theme+has more than about thirty entries, give it its own index page and link it from+the main one.++The log grows forever and nobody reads old entries. Roll entries older than a+few months into one summary line per month, or archive them into a dated file.+Keep recent detail, compress history.++## The rule that matters++Both files update in the same run as the work they describe.++An index that lags is worse than no index, because the agent trusts it and stops+looking for pages it does not list. A log written later is a reconstruction, not+a record.++## Next++[Graphs](../05-graphs/README.md), where these structures start paying off.diff --git a/docs/04-structuring/linking-rules.md b/docs/04-structuring/linking-rules.mdnew file mode 100644index 0000000..a9a8a53--- /dev/null+++ b/docs/04-structuring/linking-rules.md@@ -0,0 +1,73 @@+# Linking rules++Links are the entire value of the structure. Without them you have a folder of+files with good filenames, which is a search index, and search indexes already+exist.++## The rule++Link the first mention of any concept or entity, on every page.++That is it. Applied consistently, it produces the graph on its own, with no+separate linking pass and no maintenance ritual.++## Link, do not restate++If you are explaining something that already has a page, link it and move on.++This is what keeps knowledge in one place. When the explanation of a concept+lives on four pages in four slightly different versions, updating it means+finding all four, and you will not. Linked, it lives once and improves once.++## Both directions++A new page links out to what it references. What it references links back.++One-directional linking produces pages that reference the vault but that the+vault never reaches. They exist, they are correct, and nothing finds them.+Obsidian shows unlinked backlinks, which helps you spot these, but the agent+should be closing the loop at ingest rather than leaving it to you.++## Links that do not exist yet++Write the link anyway, then either create the target page in the same run or+record it under Gaps in `index.md`.++Gaps are useful information. A page that three sources have linked to and that+does not exist yet is exactly what to write next, or exactly what to go read+about.++What is not acceptable is a permanent gap. Links to pages that will never exist+make the graph lie about its own coverage.++## How many++There is no target number, but the shape matters. A page with no outbound links+is a dead end. A page with forty is usually a list pretending to be a page.++Watch the average across the vault rather than any single page. Below roughly+two links per page, the graph is not connected enough to answer anything the+individual pages could not. Track it with `scripts/vault_stats.py`.++## Hub pages++Some pages accumulate links until everything routes through them. A general+concept that every source touches becomes a hub, and once it is a hub, the graph+around it stops being informative: everything is two hops from everything.++The fix is to split the hub into the specific ideas people are actually linking+to. If forty pages link to a single broad concept, they are linking to four+different aspects of it.++## Decorative links++A link is a claim that two pages are related in a way worth following. Links+added because a word happened to appear are noise, and enough of them make the+graph unusable in exactly the way an empty graph is unusable.++The test: would you follow this link while researching the topic? If not,+delete it.++## Next++[Frontmatter schema](frontmatter-schema.md)diff --git a/docs/04-structuring/naming-and-aliases.md b/docs/04-structuring/naming-and-aliases.mdnew file mode 100644index 0000000..d38a12a--- /dev/null+++ b/docs/04-structuring/naming-and-aliases.md@@ -0,0 +1,69 @@+# Naming and aliases++Wikilinks resolve by name, so the name is not cosmetic. It is the key the whole+graph is built on.++## Canonical titles++One page, one canonical title, and it is the name you would use talking to+someone in the field.++Practical rules that avoid most of the pain:++- **Singular, not plural.** `Vector database`, not `Vector databases`.+- **Spell out acronyms in the title, put the acronym in aliases.** The title is+ for the reader, the alias is for the link.+- **No dates, no versions in the title** unless the thing genuinely is a+ version. A page called `MCP spec 2026-07` becomes wrong in a month; a page+ called `MCP` with a version history does not.+- **No your-vault-specific prefixes.** `Concept - RAG` fights the link syntax+ for no benefit.++## Aliases++Everything the same subject gets called, listed in frontmatter:++```yaml+title: Retrieval augmented generation+aliases: [RAG, retrieval-augmented generation]+```++Aliases are what stop the vault from quietly growing three pages for one thing.+Without the alias, a source that says RAG creates `RAG`, and a source that+spells it out creates a second page, and neither knows about the other.++Add aliases at creation, not later. By the time you notice the duplicate, both+pages have inbound links.++## Two things with the same name++Disambiguate in the title, not with a suffix nobody would type:++```+Mercury (planet)+Mercury (element)+```++Then alias the bare word to whichever one is dominant in your vault, or to+neither if both matter, which forces the agent to pick explicitly rather than+guessing.++Same applies to people who share a name and to your own projects that collide+with a well-known product.++## Renaming++Obsidian updates links when you rename inside the app. The agent renaming a file+directly does not, and that is the single most common source of broken links in+an agent-maintained vault.++Rule for `CLAUDE.md`: renames go through a documented procedure. Update the+file, update every inbound link, add the old name as an alias, and record it in+`log.md`. The alias matters because it keeps external references and your own+memory working.++Then run `scripts/link_check.py` to confirm nothing dangled.++## Next++[Deduplication and merging](dedupe-and-merge.md)diff --git a/docs/04-structuring/page-types.md b/docs/04-structuring/page-types.mdnew file mode 100644index 0000000..c39460f--- /dev/null+++ b/docs/04-structuring/page-types.md@@ -0,0 +1,64 @@+# The four page types++Four types, each with a job. Every page declares its type in frontmatter, which+is what lets you query the vault by shape rather than reading everything.++## Sources++One page per ingested item. What this article, video, paper or conversation+said.++Records claims as claims: this source says X, with the URL, author and date+attached. It does not evaluate. It does not merge in what other sources think.++Source pages are the audit trail. When a concept page says something surprising,+the source pages behind it are how you check whether it was ever true.++## Entities++People, organisations, products, tools. What it is, why it appears in your+vault, and everything that mentions it.++Entity pages are cheap to create and pay off in a way that is not obvious until+the vault is large. They become the natural join points: three unrelated sources+that all mention the same company suddenly connect through it, and that+connection is often the thing worth noticing.++Do not write biographies. An entity page holds what is relevant to your vault,+not what Wikipedia holds.++## Concepts++Ideas, methods, frameworks, arguments. These are the pages that compound.++A concept page explains the idea in plain language, says where it came from,+what supports it, what argues against it, and what is still unclear. Written+well, it is readable by someone who has seen none of your sources, including+you in a year.++This is the only page type where the agent should be synthesising across+sources rather than recording.++## Synthesis++Comparisons, themes, open questions. A synthesis page exists only when it says+something no single source did: two sources disagree, three describe the same+pattern, a question keeps recurring across unrelated material.++The rule that keeps these honest: do not write a synthesis page to have one. An+agent asked for synthesis will always produce something, and manufactured themes+are the fastest way to fill a wiki with confident nonsense.++## Where the types blur++Some things are legitimately two types. A methodology named after its author is+both an entity and a concept. Pick one, usually the concept, and give the other+an alias pointing at it.++Some things are neither. A recipe, a config snippet, a checklist. Those are+reference material, not knowledge, and they belong in a project folder rather+than the wiki.++## Next++[Linking rules](linking-rules.md)diff --git a/docs/README.md b/docs/README.mdindex dd6338d..1f4cf1a 100644--- a/docs/README.md+++ b/docs/README.md@@ -9,7 +9,7 @@ Setup is written in full. The rest is being written section by section; see the| [Concepts](01-concepts/README.md) | what the pattern is and why the old note systems died | partial || [Setup](02-setup/README.md) | Obsidian, Claude Code, CLAUDE.md, MCP, projects, git | **written** || [Ingestion](03-ingestion/README.md) | articles, video, PDFs, chat exports, voice, backfilling | **written** |-| [Structuring](04-structuring/README.md) | page types, linking rules, schema, contradictions | in progress |+| [Structuring](04-structuring/README.md) | page types, linking rules, schema, contradictions | **written** || [Graphs](05-graphs/README.md) | what the graph is for, typed links, GraphRAG, metrics | in progress || [Agents](06-agents/README.md) | roles, schedules, hooks, guardrails | in progress || [Retrieval](07-retrieval/README.md) | query patterns, search, context budget | in progress |diff --git a/docs/ROADMAP.md b/docs/ROADMAP.mdindex 60d6fd0..86400cc 100644--- a/docs/ROADMAP.md+++ b/docs/ROADMAP.md@@ -126,46 +126,46 @@ Getting everything you read, watch and write into raw/ with as little friction a- Cost control on large imports- Checkpoints and resuming-## Structuring (outline)+## Structuring (done)The rules that decide whether your vault becomes a graph or a landfill.-- [ ] **One page, one idea**+- [x] **One page, one idea**- What atomic means in practice- Splitting a source into concepts- How long a page should be- When to merge instead of split-- [ ] **The four page types**+- [x] **The four page types**- Sources: one page per ingested item- Entities: people, orgs, products, tools- Concepts: ideas, frameworks, methods- Synthesis: comparisons, themes, open questions-- [ ] **Linking rules**+- [x] **Linking rules**- Link on first mention, every page type- Minimum and maximum links per page- Links as claims, not decoration- Preventing hub pages from swallowing the graph-- [ ] **Frontmatter schema**+- [x] **Frontmatter schema**- Required fields per page type- Dates, source URLs, confidence- Tags: a small controlled vocabulary- Keeping the schema stable as the vault grows-- [ ] **Naming and aliases**+- [x] **Naming and aliases**- Canonical titles and why they matter for wikilinks- Aliases for acronyms and alternate names- Disambiguating two things with the same name- Renaming without breaking links-- [ ] **Deduplication and merging**+- [x] **Deduplication and merging**- Detecting near-duplicate pages- Merge procedure that preserves backlinks- Redirect stubs- Scheduling dedupe rather than doing it live-- [ ] **Contradictions and supersession**+- [x] **Contradictions and supersession**- Two sources disagree: what the page should say- Marking superseded claims instead of deleting them- Confidence and provenance fields- Why silent overwrites are the worst failure mode-- [ ] **index.md and log.md**+- [x] **index.md and log.md**- The index as a navigable catalog, not a dump- What the agent must record in the log- Using the log to audit and roll back