Skip to content

Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.

13.1 KB
Raw Blame

Roadmap

Every section is written. This page stays as a map of what each page covers, and as the place to propose new ones.

Pull requests against any outline are welcome. Read CONTRIBUTING.md first: every factual claim needs a primary source.


Concepts (done)

Why an LLM-maintained knowledge base beats the note system you abandoned, and what the pattern actually is.

  • What a second brain actually is
    • The definition that matters here: one local, plain-text place your agent can read and write
    • What it is not: a prettier Notion, a chatbot with your PDFs attached
    • The three jobs it does: capture, connect, answer
    • The compounding effect and why it only shows up after ~50 sources
  • The save-for-later paradox
    • Why you save things you never reopen
    • Cost of retrieval vs cost of capture
    • Where bookmarks, read-later apps and screenshots break down
    • What changes when something else does the filing
  • The LLM wiki pattern
    • Karpathy's llm-wiki gist (published April 4, 2026) and what it specifies
    • Compilation over summarization: the wiki is primary, raw sources are the archive
    • Ingest, lint, query as the three operations
    • Where the gist stops and implementation begins
  • Why markdown and plain text
    • Agents read and write files better than they drive APIs
    • Portability: no lock-in, no export step, git-friendly
    • Wikilinks as a graph format that survives any tool
    • Limits of plain text and when to add a database
  • Zettelkasten, PARA, evergreen notes
    • What each system got right and what it demanded from you
    • Why maintenance load is the thing that kills note systems
    • Which parts of the old methods still apply
    • What the LLM removes from the workload and what it does not
  • Wiki vs RAG
    • Why a personal corpus rarely needs a vector database
    • Retrieval quality: structured pages vs chunk similarity
    • Cost, latency and debuggability compared
    • When RAG genuinely earns its place
  • What good looks like
    • Concrete success criteria for a working vault
    • Health signals: orphan rate, link density, page freshness
    • Failure signals you should catch early
    • A realistic timeline for the first month

Setup (done)

From zero to a vault your agent maintains, in one evening.

  • Obsidian and your first vault
    • Install and create a local vault
    • Why the vault is just a folder
    • Settings worth changing on day one
    • Mobile and sync considerations
  • Vault structure
    • raw/, wiki/, output/ and what belongs in each
    • Page-type folders: sources, entities, concepts, synthesis
    • index.md and log.md as the agent's control surface
    • Structures to avoid and why deep folder trees fight the graph
  • Claude Code in the vault
    • Installing and starting Claude Code inside the vault folder
    • Permissions and what to allow
    • First run: pointing it at the structure
    • Working sessions vs scheduled runs
  • Writing your CLAUDE.md
    • What belongs in the instruction file and what does not
    • Page contracts: shape of each page type
    • Linking rules the agent must follow every time
    • Iterating on CLAUDE.md as the vault grows
  • Plugins worth installing
    • Web Clipper
    • Dataview and what it buys you
    • Templater
    • Plugins that conflict with agent-maintained files
  • MCP for Obsidian
    • When you need MCP and when files are enough
    • Local REST API plugin plus claude mcp add
    • What MCP unlocks: working while Obsidian is closed
    • Security notes for a local server
  • Git, sync and backups
    • Why a vault is an ideal git repo
    • What to commit and what to ignore
    • Sync options across machines
    • Recovering from a bad agent run

Ingestion (done)

Getting everything you read, watch and write into raw/ with as little friction as possible.

  • Web articles
    • Obsidian Web Clipper setup and templates
    • Clipping to raw/ with consistent frontmatter
    • Paywalls, newsletters and dynamic pages
    • What to clip and what to skip
  • YouTube and podcasts
    • Pulling transcripts reliably
    • Timestamps and why to keep them
    • Long videos: chunking before ingest
    • Audio-only sources and transcription options
  • PDFs, papers and books
    • Text extraction that preserves structure
    • Scanned documents and OCR
    • Highlights from ebook readers
    • Handling papers with figures and tables
  • Chat exports
    • Exporting Claude and ChatGPT history
    • Converting a JSON export into per-conversation markdown
    • Filtering noise: which conversations are worth ingesting
    • Privacy pass before anything enters the vault
  • Voice notes and meetings
    • Capture on mobile
    • Transcription pipeline
    • Turning a meeting into decisions and entities
    • Keeping personal audio out of shared vaults
  • Newsletters and email
    • A dedicated capture address
    • Automated forwarding into raw/
    • Deduplicating recurring newsletters
    • When email is not worth ingesting
  • Backfilling years of material
    • Ordering the backfill so early pages do not get overwritten
    • Batching: how many sources per run
    • Cost control on large imports
    • Checkpoints and resuming

Structuring (done)

The rules that decide whether your vault becomes a graph or a landfill.

  • One page, one idea
    • What atomic means in practice
    • Splitting a source into concepts
    • How long a page should be
    • When to merge instead of split
  • The four page types
    • Sources: one page per ingested item
    • Entities: people, orgs, products, tools
    • Concepts: ideas, frameworks, methods
    • Synthesis: comparisons, themes, open questions
  • Linking rules
    • Link on first mention, every page type
    • Minimum and maximum links per page
    • Links as claims, not decoration
    • Preventing hub pages from swallowing the graph
  • Frontmatter schema
    • Required fields per page type
    • Dates, source URLs, confidence
    • Tags: a small controlled vocabulary
    • Keeping the schema stable as the vault grows
  • Naming and aliases
    • Canonical titles and why they matter for wikilinks
    • Aliases for acronyms and alternate names
    • Disambiguating two things with the same name
    • Renaming without breaking links
  • Deduplication and merging
    • Detecting near-duplicate pages
    • Merge procedure that preserves backlinks
    • Redirect stubs
    • Scheduling dedupe rather than doing it live
  • Contradictions and supersession
    • Two sources disagree: what the page should say
    • Marking superseded claims instead of deleting them
    • Confidence and provenance fields
    • Why silent overwrites are the worst failure mode
  • index.md and log.md
    • The index as a navigable catalog, not a dump
    • What the agent must record in the log
    • Using the log to audit and roll back
    • Keeping both files from growing unbounded

Graphs (done)

The part most second-brain guides skip: what the graph is for and how to make it do work.

  • Graph basics for a knowledge base
    • Nodes, edges, direction and weight in plain terms
    • Why a knowledge base is a graph whether you plan it or not
    • Density, clustering, path length and what they tell you
    • The failure shapes: stars, chains and islands
  • Reading the Obsidian graph view
    • Local vs global graph
    • Filters and groups that make it useful
    • Spotting orphans and over-linked hubs
    • Why a pretty graph is not a healthy one
  • Typed links
    • Untyped wikilinks lose meaning at scale
    • A minimal relation vocabulary: supports, contradicts, extends, part-of
    • Encoding types in frontmatter or inline
    • Querying by relation type
  • Graph vs embeddings
    • What each retrieval method is actually good at
    • Why similarity misses reasoning across sources
    • Hybrid setups
    • Cost and maintenance compared
  • GraphRAG and where it fits
    • The idea in one page
    • Entity and relation extraction from your own vault
    • Community detection and summary layers
    • Whether a personal vault is big enough to need it
  • Exporting the graph
    • Parsing wikilinks into an edge list
    • Loading into Ladybug, Neo4j or NetworkX
    • Visualizing outside Obsidian
    • Keeping the export in sync
  • Graph metrics worth tracking
    • Orphan rate
    • Average degree and its healthy range
    • Clusters as evidence of your real interests
    • Tracking metrics over time

Agents (done)

Turning a one-off setup into a system that maintains itself.

  • Agent roles
    • Ingestor, linker, reviewer, researcher
    • One job per agent and why
    • Handoffs between agents
    • Roles that are not worth automating
  • Scheduling
    • Cadence per job: hourly, daily, weekly
    • Scheduled tasks, cron and CI options
    • What a run should produce
    • Detecting silent failures
  • Subagents and parallel work
    • When to fan out
    • Context isolation between subagents
    • Merging parallel edits without conflicts
    • Cost of parallelism
  • Hooks
    • Triggering on file changes in raw/
    • Pre-write validation
    • Post-run linting
    • Keeping hooks fast
  • Skills and slash commands
    • Skills for repeated behavior, commands for repeated tasks
    • Where to store them so any agent can use them
    • Versioning your skills alongside the vault
    • Testing a skill before trusting it with the vault
  • Guardrails
    • Never let an agent delete without a log entry
    • Write scopes: which folders are off limits
    • Dry-run mode
    • Reviewing agent output in git diffs

Retrieval (done)

How to actually ask your vault things and get answers worth trusting.

  • Asking the vault
    • Question shapes that work
    • Anchoring answers to pages and sources
    • Requiring citations from your own vault
    • What to do when the answer is not in there
  • Query patterns
    • What do I know about X
    • What connects X and Y
    • What have I changed my mind about
    • What is missing from my understanding of X
  • Search tools
    • ripgrep as the default
    • Dataview queries
    • Full-text index options
    • When search is slow enough to matter
  • Adding RAG when you need it
    • The size threshold where it starts to pay
    • Chunking a wiki that is already chunked
    • Keeping the index fresh
    • Hybrid: graph for structure, vectors for recall
  • Context budget
    • Why you should not paste the whole vault
    • Progressive disclosure: index first, pages second
    • Summary layers
    • Measuring what the agent actually read

Outputs (done)

The vault earns its keep when things come out of it.

  • Writing from the vault
    • Outlining from concept pages
    • Keeping citations to your own sources
    • Voice: your notes, not the model's
    • Publishing loop back into the vault
  • Research reports
    • Scoping a question against existing pages
    • Filling gaps with new ingestion
    • Report structure that survives review
    • Storing outputs so they become sources
  • Publishing and export
    • Static site options for a public wiki
    • Deciding what stays private
    • HTML and PDF export
    • Keeping public and private in one vault
  • Learning from your own vault
    • Generating questions from concept pages
    • Spaced repetition against your notes
    • Finding what you saved but never understood
    • Measuring learning instead of collecting

Maintenance (done)

What keeps the thing alive after the novelty wears off.

  • Linting the vault
    • Broken links, orphan pages, empty stubs
    • Schema violations
    • Automating the lint run
    • Reading a lint report
  • Review cadence
    • Weekly review that takes ten minutes
    • Monthly structural review
    • What to prune and what to archive
    • Reviews that the agent prepares for you
  • Versioning
    • Commit granularity for agent runs
    • Reading diffs of generated pages
    • Reverting a bad ingest
    • Branching for experiments
  • Backups and portability
    • Local, cloud and offline copies
    • Testing a restore
    • Moving to another editor or agent
    • Avoiding formats that trap you
  • Privacy
    • Deciding what never enters the vault
    • Redaction before ingest
    • Local models for sensitive material
    • Sharing a vault safely
  • Scaling past a thousand pages
    • Where the plain-text approach strains
    • Splitting vaults by domain
    • Summary layers and hierarchical indexes
    • Performance tuning

Troubleshooting (done)

Concrete failures, with fixes.

  • The failures everyone hits
    • Vault stalls after two weeks
    • Pages that are summaries of summaries
    • The agent rewrites work you did by hand
    • Ingest quality drops as the vault grows
  • The agent writes garbage
    • Diagnosing whether it is CLAUDE.md or the source
    • Tightening page contracts
    • Adding examples instead of adding rules
    • Rolling back and re-running
  • Broken links and orphans
    • Why renames break wikilinks
    • Bulk fixing safely
    • Preventing orphans at ingest time
    • Auditing after a big import
  • The vault got too big
    • Symptoms
    • Splitting vs summarizing
    • Archiving cold material
    • Rebuilding the index
  • FAQ
    • Do I need MCP
    • Does this work with other agents
    • How much does it cost to run
    • Can I use a local model

Back to the guide