Fastest way to scaffold your LLM powered personal knowledge base - Obsidian, Claude Code, qmd, Github
JavaScript aiampclaude-codecodexcursorgithubknowledge-baseknowledge-graphknowledge-managementllmobsidianobsidian-vaultqmdsecond-brain
feat: add /brain-refresh and /brain-rebuild skills
jessepinkman99008150df53a093
diff --git a/.claude/skills/pack-test/SKILL.md b/.claude/skills/pack-test/SKILL.mdindex 9f4fea8..b7faa16 100644--- a/.claude/skills/pack-test/SKILL.md+++ b/.claude/skills/pack-test/SKILL.md@@ -45,7 +45,7 @@ ls test-vault/wiki/# Sources dirsls test-vault/sources/-# Claude Code skills (expect: brain-ingest, brain-search, lint, qmd-cli, setup)+# Claude Code skills (expect: brain-ingest, brain-search, brain-refresh, brain-rebuild, lint, qmd-cli, setup)ls test-vault/.claude/skills/# qmd scripts@@ -60,9 +60,11 @@ grep "const DB" test-vault/scripts/qmd/setup.ts# Global skills must be installedls ~/.claude/skills/brain-ingest/ls ~/.claude/skills/brain-search/+ls ~/.claude/skills/brain-refresh/# Global skill content must have the absolute INDEX_PATHgrep "INDEX_PATH=" ~/.claude/skills/brain-ingest/SKILL.md+grep "INDEX_PATH=" ~/.claude/skills/brain-refresh/SKILL.md```Report results against this checklist:@@ -73,13 +75,14 @@ Report results against this checklist:| `wiki/index.md`, `log.md`, `overview.md` | All present | || `wiki/sources/` and `wiki/qa/` | Present | || `sources/articles/`, `pdfs/`, `personal/` | Present | |-| `.claude/skills/` — 5 subdirs | brain-ingest, brain-search, lint, qmd-cli, setup | |+| `.claude/skills/` — 7 subdirs | brain-ingest, brain-search, brain-refresh, brain-rebuild, lint, qmd-cli, setup | || `scripts/qmd/setup.ts`, `reindex.ts`, `package.json` | All present | || `CLAUDE.md`, `mise.toml` | Present | || `CLAUDE.md` INDEX_PATH | Absolute path (not `qmd.sqlite`) | || `scripts/qmd/setup.ts` DB const | Hardcoded absolute path (not `join(VAULT, ...)`) | || `~/.claude/skills/brain-ingest/SKILL.md` | Present with absolute INDEX_PATH | || `~/.claude/skills/brain-search/SKILL.md` | Present with absolute INDEX_PATH | |+| `~/.claude/skills/brain-refresh/SKILL.md` | Present with absolute INDEX_PATH | |**Step 4 — Clean up**@@ -88,10 +91,11 @@ rm -rf test-vaultrm claude-second-brain-$(node -p "require('./package.json').version").tgzrm -rf ~/.claude/skills/brain-ingestrm -rf ~/.claude/skills/brain-search+rm -rf ~/.claude/skills/brain-refreshgh repo delete test-vault --yes```-> **Note:** If you had pre-existing `brain-ingest` / `brain-search` global skills from a real vault, they were overwritten by this test. Re-run `/setup` inside your real vault to restore them.+> **Note:** If you had pre-existing `brain-ingest` / `brain-search` / `brain-refresh` global skills from a real vault, they were overwritten by this test. Re-run `/setup` inside your real vault to restore them.## What failure meansdiff --git a/.github/workflows/pack-test.yml b/.github/workflows/pack-test.ymlindex a99606d..c8d6cff 100644--- a/.github/workflows/pack-test.yml+++ b/.github/workflows/pack-test.yml@@ -58,6 +58,8 @@ jobs:check ".claude/skills/setup" test-vault/.claude/skills/setupcheck ".claude/skills/brain-ingest" test-vault/.claude/skills/brain-ingestcheck ".claude/skills/brain-search" test-vault/.claude/skills/brain-search+ check ".claude/skills/brain-refresh" test-vault/.claude/skills/brain-refresh+ check ".claude/skills/brain-rebuild" test-vault/.claude/skills/brain-rebuildcheck ".claude/skills/lint" test-vault/.claude/skills/lintcheck ".claude/skills/qmd-cli" test-vault/.claude/skills/qmd-clicheck "scripts/qmd/setup.ts" test-vault/scripts/qmd/setup.ts@@ -65,8 +67,9 @@ jobs:check "scripts/qmd/package.json" test-vault/scripts/qmd/package.jsoncheck "CLAUDE.md" test-vault/CLAUDE.mdcheck "mise.toml" test-vault/mise.toml- check "~/.claude/skills/brain-ingest/SKILL.md" "$HOME/.claude/skills/brain-ingest/SKILL.md"- check "~/.claude/skills/brain-search/SKILL.md" "$HOME/.claude/skills/brain-search/SKILL.md"+ check "~/.claude/skills/brain-ingest/SKILL.md" "$HOME/.claude/skills/brain-ingest/SKILL.md"+ check "~/.claude/skills/brain-search/SKILL.md" "$HOME/.claude/skills/brain-search/SKILL.md"+ check "~/.claude/skills/brain-refresh/SKILL.md" "$HOME/.claude/skills/brain-refresh/SKILL.md"grep_check() {local label="$1"@@ -80,9 +83,11 @@ jobs:fi}- grep_check "CLAUDE.md INDEX_PATH is absolute" "INDEX_PATH=/" test-vault/CLAUDE.md- grep_check "setup.ts DB is hardcoded absolute path" 'const DB = "/' test-vault/scripts/qmd/setup.ts- grep_check "global brain-ingest INDEX_PATH is absolute" "INDEX_PATH=/" "$HOME/.claude/skills/brain-ingest/SKILL.md"+ grep_check "CLAUDE.md INDEX_PATH is absolute" "INDEX_PATH=/" test-vault/CLAUDE.md+ grep_check "setup.ts DB is hardcoded absolute path" 'const DB = "/' test-vault/scripts/qmd/setup.ts+ grep_check "global brain-ingest INDEX_PATH is absolute" "INDEX_PATH=/" "$HOME/.claude/skills/brain-ingest/SKILL.md"+ grep_check "global brain-search INDEX_PATH is absolute" "INDEX_PATH=/" "$HOME/.claude/skills/brain-search/SKILL.md"+ grep_check "global brain-refresh INDEX_PATH is absolute" "INDEX_PATH=/" "$HOME/.claude/skills/brain-refresh/SKILL.md"if [ "$fail" -ne 0 ]; thenecho ""diff --git a/CLAUDE.md b/CLAUDE.mdindex 517c68b..04379c0 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -229,3 +229,21 @@ Add `--json` for structured output. Omit `-c wiki` to search all collections (wibun scripts/qmd/reindex.ts```Do NOT re-index after every single file edit.++---++## Repo Maintenance (dev-only, not shipped to user vaults)++This section applies only when working on the `claude-second-brain` repo itself — it is not part of the template copied into user vaults.++Whenever you **add, remove, or rename a skill** under `template/.claude/skills/`, you MUST update the following in the same change:++1. **`README.md`** (root) — the "Claude Code skills included" section and the "Installing and updating skills" block. Bump the skill count ("N wiki skills") in the install commands.+2. **`template/README.md`** — the "Your Claude Code skills" section and the "Installing and updating skills" block. Bump the skill count.+3. **`bin/create.js`** — the `installGlobalSkills()` array (lines ~65) if the skill should be installed globally (`~/.claude/skills/`). `brain-ingest`, `brain-search`, `brain-refresh` are global today; `brain-rebuild`, `lint`, `setup`, `qmd-cli` are vault-local.+4. **`.github/workflows/pack-test.yml`** — add a `check` line for the new vault-local skill directory, and if the skill is global, add a `check` line for `$HOME/.claude/skills/<name>/SKILL.md` plus a `grep_check` for `INDEX_PATH=/`.+5. **`.claude/skills/pack-test/SKILL.md`** — update the expected-skills comment, the `ls` and `grep` commands in Step 3, the checklist table (including the "N subdirs" count), and the cleanup / overwrite-warning note in Step 4.++Whenever you **change a qmd collection name, context path, or CLI invocation** in `scripts/qmd/setup.ts` or in skill workflows, you MUST also update every `-c <collection>` reference in `template/CLAUDE.md`, all `SKILL.md` files under `template/.claude/skills/`, and both READMEs.++Use `__QMD_PATH__` as the placeholder in any template file that references the qmd index path — `bin/create.js::patchVault()` substitutes it at scaffold time.diff --git a/README.md b/README.mdindex 2b333a5..c671002 100644--- a/README.md+++ b/README.md@@ -96,7 +96,9 @@ Open `my-brain/` as a vault in Obsidian — the folder is already a valid Obsidi## Claude Code skills included-The wiki ships with two global slash commands that cover the full workflow. No manual prompting, no copy-pasting.+The wiki ships with a set of slash commands that cover the full workflow. No manual prompting, no copy-pasting.++### Daily workflow**`/brain-ingest`** — Drop a file into `sources/articles/`, `sources/pdfs/`, or `sources/personal/`. Run `/brain-ingest`. Claude summarizes the source, asks what matters most to you, creates a `wiki/sources/` page, updates or creates related topic pages, flags any contradictions with existing knowledge, and logs everything.@@ -104,6 +106,16 @@ The wiki ships with two global slash commands that cover the full workflow. No m**`/lint`** — Health-check the wiki. Surfaces orphan pages, broken links, unresolved contradictions, and data gaps. Reports findings and applies fixes where possible.+### Maintenance++**`/brain-refresh`** — Re-scan the vault for new or changed files and regenerate vector embeddings. Run after a bulk ingest session or manual edits. Pass `force` to re-embed every chunk (e.g. after changing the embedding model).++**`/brain-rebuild`** — **Destructive.** Redesigns the qmd schema: analyzes the wiki, proposes new collections and contexts, waits for your approval, then patches `scripts/qmd/setup.ts`, drops the old index, and rebuilds embeddings from scratch. Use only when the current structure no longer fits how you search.++### Setup++**`/setup`** — First-time initialization. Registers the qmd collections and generates local vector embeddings. Run once after scaffolding.+---## Access from anywhere@@ -176,14 +188,14 @@ my-brain/## Installing and updating skills-Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md`. `/brain-ingest` and `/brain-search` are installed globally to `~/.claude/skills/` during setup so they work in any Claude Code session. `/lint`, `/setup`, and `/qmd-cli` are installed into the vault itself.+Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md`. `/brain-ingest`, `/brain-search`, and `/brain-refresh` install globally to `~/.claude/skills/` during setup so they work in any Claude Code session. `/brain-rebuild`, `/lint`, `/setup`, and `/qmd-cli` live inside the vault at `.claude/skills/`.### Update the built-in wiki skills-The wiki's own skills (`/brain-ingest`, `/brain-search`, `/lint`, `/setup`, `/qmd-cli`) are scaffolded at creation time. To pull in improvements, use `npx skills` pointing to the template skills directory in this repo:+The wiki's own skills are scaffolded at creation time. To pull in improvements, use `npx skills` pointing to the template skills directory in this repo:```bash-# Install or update all 5 wiki skills from the latest template+# Install or update all 7 wiki skills from the latest templatenpx skills add https://github.com/jessepinkman9900/claude-second-brain/tree/main/template/.claude/skills -a claude-code -y# Or update a specific skill@@ -200,7 +212,7 @@ npx skills update -a claude-code## Roadmap-- [ ] GitHub Actions for scheduled re-indexing+- [ ] GitHub Actions to invoke `/brain-refresh` on a schedule- [ ] GitHub agentic workflows — auto-ingest on push, scheduled lint, auto-summary on new sources- [ ] More Claude Code skills (source discovery, topic clustering)diff --git a/bin/create.js b/bin/create.jsindex befff11..43094b9 100644--- a/bin/create.js+++ b/bin/create.js@@ -62,7 +62,7 @@ async function patchVault(targetDir, qmdPath, brainName) {async function installGlobalSkills(qmdPath) {const globalSkillsDir = join(homedir(), ".claude", "skills")- await Promise.all(["brain-ingest", "brain-search"].map(async skillName => {+ await Promise.all(["brain-ingest", "brain-search", "brain-refresh"].map(async skillName => {const srcFile = join(TEMPLATE, ".claude/skills", skillName, "SKILL.md")const destDir = join(globalSkillsDir, skillName)diff --git a/template/.claude/skills/brain-rebuild/SKILL.md b/template/.claude/skills/brain-rebuild/SKILL.mdnew file mode 100644index 0000000..ae5bf4c--- /dev/null+++ b/template/.claude/skills/brain-rebuild/SKILL.md@@ -0,0 +1,119 @@+---+name: brain-rebuild+description: "Redesign the qmd schema for this vault — analyze the current wiki, recommend a new set of collections and contexts, update scripts/qmd/setup.ts, then tear down and rebuild the index from scratch. Destructive. Trigger phrases: /brain-rebuild, rebuild the brain, restructure qmd, redesign collections, redo contexts, recompute schema."+argument-hint: "Optional: a hint about the desired structure (e.g. 'split wiki by domain', 'add a separate collection for qa pages')"+---++# Brain Rebuild++Redesigns the qmd schema based on what the wiki actually contains today, then rebuilds the index from scratch. **Destructive** — wipes existing collections/contexts and their embeddings. Always get explicit user approval before applying changes.++## When to Use++- The wiki has grown and the current single `wiki` collection no longer matches how the user searches+- The user wants finer-grained contexts (e.g. distinct context descriptions per sub-folder)+- After significant reorganization of `wiki/` or `sources/` folder structure+- Never as a routine refresh — for that, use `/brain-refresh`++## Procedure++All commands run from the vault root.++### Step 1 — Survey the current state++- Read `scripts/qmd/setup.ts` to see the current collections and contexts+- Read `CLAUDE.md` to understand the documented schema and any references to collection names+- List current state from qmd:+ ```bash+ INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd collection list+ INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd context list+ INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd status+ ```++### Step 2 — Analyze the wiki++- Glob `wiki/**/*.md` and `sources/**/*.md`+- Read enough pages (especially `wiki/index.md` and `wiki/overview.md`) to understand actual topic clusters, page-type distribution, and folder structure+- Identify natural groupings: by domain (e.g. ml, distributed-systems, finance), by page type (topics vs entities vs qa), by source provenance, etc.++### Step 3 — Propose a new schema++Produce a recommendation that includes:++- **Collections** — name, root path (relative to vault), file glob pattern. Justify each split.+- **Contexts** — global context, plus per-collection and per-subpath context descriptions. Each context should be a one-sentence description that helps a future search disambiguate between similar pages.+- **Migration impact** — what changes for `/brain-ingest`, `/brain-search`, `/lint`, and `CLAUDE.md` (e.g. if collection name `wiki` becomes `wiki-topics`, all `-c wiki` invocations break)++Present the recommendation as a clear diff against the current schema. Do not modify any files yet.++### Step 4 — Get explicit approval++Ask the user: "Apply this schema? This will drop existing collections and their embeddings, then rebuild from scratch. Re-embedding the full wiki may take significant time and download GGUF models if not cached."++Do not proceed without an explicit yes.++### Step 5 — Update `scripts/qmd/setup.ts`++Edit `scripts/qmd/setup.ts` to reflect the new schema:++- Update the `ensureCollection` calls (add, remove, or rename)+- Update `setGlobalContext` and the `addContext` calls+- Keep the `__QMD_PATH__` / `DB` line untouched — it was already substituted at install time+- Keep the script idempotent++### Step 6 — Update related references++If collection names changed:++- Update every occurrence in `CLAUDE.md` (e.g. `qmd query -c wiki` → new name)+- Update `.claude/skills/brain-ingest/SKILL.md`, `.claude/skills/brain-search/SKILL.md`, `.claude/skills/lint/SKILL.md`, `.claude/skills/qmd-cli/SKILL.md` — anywhere collection names are hard-coded+- Use Grep to find any remaining stale references before moving on++### Step 7 — Tear down the old schema++For each existing collection that no longer fits the new schema:++```bash+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd collection remove <old-name>+```++For each obsolete context:++```bash+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd context rm <path>+```++### Step 8 — Register the new schema++Run:+```bash+bun scripts/qmd/setup.ts+```++This is idempotent — collections that already exist (e.g. ones you kept) are skipped; new ones are added; contexts are upserted.++### Step 9 — Rebuild embeddings from scratch++Run:+```bash+bun scripts/qmd/reindex.ts+```++This indexes all files under the new collections and generates fresh embeddings.++### Step 10 — Verify and report++```bash+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd collection list+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd context list+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd status+```++Confirm the new collections appear, contexts match the plan, and document/embedding counts are non-zero. Report a summary of what changed (collections added/removed/renamed, contexts updated, files re-indexed, references updated in CLAUDE.md and skills).++## Hard Rules++- Never apply schema changes without explicit user approval at Step 4+- Never delete a collection without first showing the user what will be dropped+- Always update `CLAUDE.md` and skill files in lockstep with collection renames — stale `-c <name>` references will silently break `/brain-search`+- Do not touch `wiki/`, `sources/`, or any user content — this skill changes the index schema, not the datadiff --git a/template/.claude/skills/brain-refresh/SKILL.md b/template/.claude/skills/brain-refresh/SKILL.mdnew file mode 100644index 0000000..7689dea--- /dev/null+++ b/template/.claude/skills/brain-refresh/SKILL.md@@ -0,0 +1,63 @@+---+name: brain-refresh+description: "Refresh the qmd index for this vault — re-scans all collections for new/changed files and regenerates vector embeddings. Use after a bulk ingest session, after manual file edits, or whenever search feels stale. Trigger phrases: /brain-refresh, refresh the brain, refresh embeddings, re-embed wiki, update qmd index, refresh search index."+argument-hint: "Optional: 'force' to force re-embedding of every chunk (slow, useful after embedding model changes)"+---++# Brain Refresh++Refreshes the qmd index so search reflects the current state of the vault. Wraps `bun scripts/qmd/reindex.ts` (incremental) and the qmd CLI's force-embed flag.++## When to Use++- After a `/brain-ingest` session (or several) — batch the refresh, don't run after every file edit+- After manual edits to `wiki/` or `sources/` files+- When `/brain-search` results feel stale or miss recently added content+- After upgrading `@tobilu/qmd` or changing the embedding model — use `force` mode++## Procedure++All commands run from the vault root.++### Default — Incremental Refresh++Run:+```bash+bun scripts/qmd/reindex.ts+```++This script does two things:+1. **Update** — scans collections for new, changed, or deleted files and updates the index+2. **Embed** — generates vector embeddings for any chunks that need them++Only chunks that changed get re-embedded, so this is fast on subsequent runs.++### Force Mode (`force` argument)++When the user passes `force`, re-embed **every** chunk — not just the changed ones. Use after embedding-model changes or if embeddings appear corrupted.++```bash+# Step 1 — update the file index first+bun scripts/qmd/reindex.ts++# Step 2 — force re-embed everything+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd embed -f+```++Force mode is slow — confirm with the user before proceeding if the wiki is large.++## Verify++After refresh, confirm with:++```bash+INDEX_PATH=__QMD_PATH__ bunx @tobilu/qmd status+```++Document and embedding counts should be non-zero and reflect recent activity. If embeddings show as `0` or far below document count, re-run the refresh.++## Notes++- First run downloads ~2GB of GGUF models — expected, one-time+- Do not run this after every single file edit — batch it after a session+- This skill does **not** change the qmd schema (collections, contexts). For that, use `/brain-rebuild`diff --git a/template/README.md b/template/README.mdindex 6aea1ce..a3b59d4 100644--- a/template/README.md+++ b/template/README.md@@ -34,12 +34,24 @@ Registers the qmd collections and generates local vector embeddings. First run d## Your Claude Code skills+### Daily workflow+**`/brain-ingest`** — Add a file to `sources/articles/`, `sources/pdfs/`, or `sources/personal/`, then run `/brain-ingest`. Claude summarizes the source, asks what aspects matter most, updates related wiki pages, flags contradictions, and logs everything.**`/brain-search`** — Ask anything: `what do I know about [topic]?` Claude searches the wiki semantically and returns a cited answer. If it synthesizes multiple pages in a useful way, it offers to file it as a permanent `wiki/qa/` entry.**`/lint`** — Health-check the wiki. Finds orphan pages, broken links, unresolved contradictions, and data gaps. Reports findings and fixes what it can.+### Maintenance++**`/brain-refresh`** — Re-scan the vault for new or changed files and regenerate vector embeddings. Run after a bulk ingest session or manual edits. Pass `force` to re-embed every chunk.++**`/brain-rebuild`** — **Destructive.** Redesigns the qmd schema: analyzes the wiki, proposes new collections and contexts, waits for your approval, then patches `scripts/qmd/setup.ts`, drops the old index, and rebuilds embeddings from scratch.++### Setup++**`/setup`** — First-time initialization. Registers the qmd collections and generates local vector embeddings. Run once after scaffolding.+---## Obsidian Mobile@@ -86,14 +98,14 @@ claude-second-brain/## Installing and updating skills-Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md` in this vault. The wiki ships with `/brain-ingest`, `/brain-search`, `/lint`, `/setup`, and `/qmd-cli` pre-installed.+Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md` in this vault. The wiki ships with `/brain-ingest`, `/brain-search`, `/brain-refresh`, `/brain-rebuild`, `/lint`, `/setup`, and `/qmd-cli` pre-installed.### Update built-in wiki skills-Pull the latest `/brain-ingest`, `/brain-search`, `/lint`, `/setup`, and `/qmd-cli` from the upstream template:+Pull the latest skills from the upstream template:```bash-# Install or update all 5 wiki skills+# Install or update all 7 wiki skillsnpx skills add https://github.com/jessepinkman9900/claude-second-brain/tree/main/template/.claude/skills -a claude-code -y# Or update a specific skill@@ -112,11 +124,11 @@ npx skills update -a claude-codeAfter a bulk ingest session, re-index to keep search current:-```bash-bun scripts/qmd/reindex.ts+```+/brain-refresh```-Or run `/setup` again inside Claude Code.+This wraps `bun scripts/qmd/reindex.ts` — you can also run that command directly if you're not inside Claude Code. Pass `force` to `/brain-refresh` to re-embed every chunk (e.g. after changing the embedding model).---