Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
Skills mandate incremental KB checkpointing (PRD-7050) (#1962)
tim-inkeep10456a1be25d
diff --git a/.changeset/skill-incremental-checkpointing.md b/.changeset/skill-incremental-checkpointing.mdnew file mode 100644index 000000000..2065f8836--- /dev/null+++ b/.changeset/skill-incremental-checkpointing.md@@ -0,0 +1,5 @@+---+"@inkeep/open-knowledge": patch+---++Open Knowledge skill guidance now mandates incremental checkpointing: agents write completed work into the knowledge base as they go — per section, per source, per doc — instead of holding finished findings in context for one final write. A new "Persist incrementally — the knowledge base IS your checkpoint" rule lands in the project skill's Writing section, and the `research` and `consolidate` workflow guides now instruct agents to create the article skeleton early and fill it section-by-section as each source is analyzed. This prevents the worst-case "got rate limited and discarded its findings" outcome, where completed (paid-for) research work is lost to a mid-session rate limit, crash, or context compaction.diff --git a/packages/server/assets/skills/project/SKILL.md b/packages/server/assets/skills/project/SKILL.mdindex cabbbde0e..26ac29e3c 100644--- a/packages/server/assets/skills/project/SKILL.md+++ b/packages/server/assets/skills/project/SKILL.md@@ -113,6 +113,8 @@ OK Electron and `ok ui` share `ui.lock`; when a second UI binds a different portCall `write` / `edit` as soon as you have content. Native `Edit` / `sed` / direct `Write` on in-scope markdown is forbidden — it bypasses the CRDT and loses agent attribution in the shadow repo.+**Persist incrementally — the knowledge base IS your checkpoint (MUST).** On any multi-step or long-running task — a research sweep, a multi-source synthesis, a batch of docs — write completed work to the KB as you finish each unit: per section, per source, per doc. Never hold finished findings only in your context waiting for one final write at the end. A rate limit, crash, or context compaction mid-task discards everything still unwritten; work already persisted to a doc survives, and you resume by reading the doc back instead of redoing the work. Create the target doc early (skeleton + frontmatter), then `edit` each section in as it firms up. Completed, paid-for work that existed only in context and got dropped on a rate limit is the exact failure this rule prevents — the KB is where work goes to be safe, not a thing you save to once at the finish line.+**Pass a `summary` on every content write (SHOULD).** `write`, `edit`, and `move` each take a one-line `summary` (≤80 chars) describing the user-facing outcome of the change — "Add gear list and permit info", not "edited trip doc". It renders as a bullet under your name in the document timeline and is the only human-readable change-note persisted to the shadow-repo history; omit it and the timeline shows *that* you wrote but not *what changed*. Write it from the reader's perspective, keep it specific, and avoid secrets or PII (it lands in git history). Each entry in the batch `documents:` form carries its own `summary`.**Reach for visual structure where it aids comprehension.** Default to the right OK primitive over flat prose: a Callout (`> [!NOTE]`) for a key caveat, a ` ```mermaid ` diagram for a process or relationship, a table for options or comparisons, an `html preview` chart for numbers. **Call the `palette` MCP tool as you draft** (and `palette({ components })` for a canonical's JSX schema) — it returns copy-ready markdown-native forms, themed `html preview` embed starters, and the theme tokens, so the visual lands themed and in the content graph instead of hand-rolled. Don't decorate — use a visual only when it carries the point better than prose would. Full catalog: §Components.@@ -398,6 +400,8 @@ Body is just the wiki-embed. PDFs/opaque attachments render as a click-dispatchiWhen you make a multi-step change (batch of new docs, folder restructure), pause between steps to let the browser preview catch up. The CRDT edit streams live; the preview follows your edit cadence. Don't batch 10 writes in a row — interleave the writes so the user watching the browser sees the narrative progress.+This does not conflict with *Persist incrementally* (§Writing): a checkpoint-write per section/source is naturally spaced by the work that produces that unit (read a source → write its findings → read the next), so those writes *are* the interleaved cadence. The anti-pattern is firing many writes back-to-back with no intervening work — not persisting completed work as you go. When in tension, durability wins: never hold finished work back from the KB to smooth cadence.+This is primarily a human-watchability concern — the user watches edits land in the preview; interleaved cadence makes the narrative legible. When the batch is done, navigate the preview to the primary deliverable (see "End a turn on the deliverable" in §Preview).**Hub docs.** Don't *create* `INDEX.md` / `README.md` hub files solely to catalog children — `exec("ls -A <folder>")` returns the same view live, with per-file frontmatter + backlink counts. But if a hub doc *already exists* from prior work, keep it updated as children change — interleave: write child → update hub → write next child, rather than batching five child edits and a single trailing hub update.diff --git a/packages/server/src/mcp/tools/consolidate-body.ts b/packages/server/src/mcp/tools/consolidate-body.tsindex 8ecf9d004..1a080e687 100644--- a/packages/server/src/mcp/tools/consolidate-body.ts+++ b/packages/server/src/mcp/tools/consolidate-body.ts@@ -58,6 +58,8 @@ You already confirmed the decision at the STOP gate at the top of this workflow.### 3. Write the canonical article+**Persist as you go (MUST).** For a large consolidation drawing on several research docs, create the article skeleton — frontmatter + the headings below — first, then \`edit\` each section in as you finish it; don't hold the whole synthesis in context for one final write. A rate limit or crash mid-synthesis then costs you one section, not the entire article. (The OK skill's §Writing carries this rule for all long-running work: the knowledge base is your checkpoint.)+Save inside the content directory (\`${contentDir}\`). Path convention depends on the project:- If the project uses the three-tier lifecycle (external-sources → research → articles), save under an \`articles/\` folder relative to the content dir, grouped by topic subfolder when the area is broad (e.g., \`articles/editor/crdt-architecture.md\`)diff --git a/packages/server/src/mcp/tools/research-body.ts b/packages/server/src/mcp/tools/research-body.tsindex 72c430aa7..830113a98 100644--- a/packages/server/src/mcp/tools/research-body.ts+++ b/packages/server/src/mcp/tools/research-body.ts@@ -52,6 +52,17 @@ Research articles default to **3P/external framing** — investigating third-par---+## Persist as you go — the article IS your checkpoint++⛔ **PERSIST AS YOU GO — crash-safe checkpoint rule.** The single most expensive failure this workflow has produced is completed research lost to a mid-session rate limit or crash — analysis held in context, never written, discarded when the session died. The user paid for work that vanished. The knowledge base is the checkpoint; two rules make every step crash-safe:++- **\`ingest\` each source the moment you fetch it (Step 3), one at a time** — never fetch all sources and ingest them in a trailing batch. An ingested source survives a crash; a fetched-but-unwritten one does not.+- **Create the article skeleton early and fill it section-by-section as you read (Steps 4–5), not in one final write at the end.** After you analyze each source, \`edit\` its findings into the article before moving to the next. A crash after reading five of eight sources then leaves five sections safely in the KB; you resume by reading the partial article back, not by re-running the whole sweep.++Structured notes that live only in your context are not persisted work. If a finding is worth keeping, it belongs in an ingested source or in the article — written, not held.++---+## Step 0: Create workflow checkpoint tasks⛔ **ALWAYS THE FIRST ACTION.** Before any read, any scan, any fetch — create tasks. They persist across context compaction, make skipped steps immediately visible, and show progress to the user.@@ -194,6 +205,8 @@ Take structured notes:- **Unknowns** and open questions — the boundary of what you know- **Relevance** to the specific decision at hand+**Write these notes into the article as you take them, not after (MUST — see *Persist as you go* above).** Create the article skeleton — frontmatter + the Step 5 section headings — before you start reading, then \`edit\` each source's findings into the relevant section the moment you finish analyzing it. The "notes" ARE the article's Findings section in progress; don't hold them in context to transcribe in one pass at Step 5. A rate limit between here and Step 5 must not be able to discard analysis you've already done. By the time you reach Step 5 the article is mostly written, and Step 5 becomes finalize-and-polish.+### Grounding disciplineEvery factual claim in the article must cite its source inline. No unsourced speculation. If you don't have evidence: (a) run another search and cite it, (b) mark inline \`(TODO: needs source)\`, or (c) don't write the claim. Never fabricate.@@ -202,6 +215,8 @@ Every factual claim in the article must cite its source inline. No unsourced spe## Step 5: Write the research article (Path A only)+If you followed *Persist as you go*, the article already exists and is substantially filled from Step 4 — this step **finalizes** it (fill any remaining sections, tighten the recommendation, run the structure + validation checks below) rather than writing from a blank doc. **If it does not exist or is thin, and you are resuming after an interruption: \`exec("cat <path>")\` the partial article back first and fill only the missing sections** — and note that any Step 4 analysis that was never written to the KB was lost when the session broke, so re-derive only what's actually missing. Creating the doc from scratch here means the incremental rule was skipped; that's the failure mode, not the happy path.+Save a markdown document inside the content directory. Path convention:- If the project adopted the three-tier lifecycle (external-sources → research → articles), save under \`<content-dir>/research/<slug>.md\`.