AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(second-brain): verified brain-CLI slice — demo corpus, MCP brain tools, sniffer + harness fixes
xerj-orga6dc79355e37
diff --git a/demo/usecases/second-brain/README.md b/demo/usecases/second-brain/README.mdnew file mode 100644index 000000000..b0e1f98ec--- /dev/null+++ b/demo/usecases/second-brain/README.md@@ -0,0 +1,74 @@+# Second-brain demo — corpus + verification harness++One command turns a folder of notes into a queryable, replayable link index+with a dashboard. This directory holds the demo vault generator and the+scripts that PROVE the three claims against a live server — nothing here is+mocked, and every number the scripts print is read from a live response.++> XERJ doesn't understand your notes — it makes the structure you wrote+> queryable, timestamped, and evidence-backed, deterministically. It is a+> search engine with a graph-shaped index over its own documents, **not a+> graph database** and not a brain.++## The demo vault++`gen-corpus.sh` writes a realistic 39-file personal knowledge base (bread+baking + a bakery-launch project) shaped so every link detector has honest+work to do:++| folder | files | what it exercises |+|---|---|---|+| `notes/` | 15 | `[[wikilinks]]` — authored links between zettels (one deliberately dangling: counted, never invented) |+| `projects/bakery-launch/` | 6 | relative markdown links, incl. external URLs that must NOT become links |+| `clippings/` | 4 | saved HTML pages with `<a href>` to corpus files |+| `journal/` | 12 | **no authored links at all** — every edge is structural (files sitting together), the corner of the vault that makes the dashboard's AUTHORED vs STRUCTURAL split honest |+| `reading/` | 1 | one long note that splits into sections → section-order links |++File bytes and mtimes are pinned, and detector links take their+believed-since time from mtime — so regenerating the corpus and re-running+`xerj brain` reproduces the **same edge ids every time**. (Only `created_at`+— when the brain learned each link — is wall clock, by contract.)++## Run it++```bash+./boot-and-brain.sh # generate the vault (if absent), run `xerj brain`,+ # verify the server answers → ports 9331-9333+./transcript.sh # the API proof (see below) → PASS/FAIL summary+./mcp-smoke.sh # the agent surface over MCP → PASS/FAIL summary+```++Defaults: working root `${TMPDIR:-/tmp}/xerj-second-brain-demo`, es-compat+port `9331` (override with `XERJ_BRAIN_DEMO_ROOT` / `XERJ_BRAIN_DEMO_PORT`),+binaries from `engine/target/release/`. All scripts are idempotent; stop the+booted server with `kill $(cat <root>/data/server.pid)`.++## What the transcript proves++1. **Overview** — brain exists, live/retired counts, all five detectors+ fired, the `lexical-feature-hash` honesty marker, `not_shown` accounting.+2. **Ego** — one note's neighborhood: every link annotated with hop,+ direction, and the evidence that taught it; node previews hydrated;+ clipping honestly counted.+3. **Link** — assert a manual edge *with the quote you are relying on*;+ identical re-assert returns `created:false` with the same edge id+ (deterministic identity ⇒ safe to retry).+4. **Unlink** — retire, never delete: the response carries both clocks+ (when it stopped being true vs. when the system recorded that).+5. **Replay** — ask again with `as_of` inside the belief interval: the+ retired link is still there in the past, absent now, and the exclusion is+ *counted* in `not_shown.expired_excluded`.+6. **Refusals** — `hops=3` → 400 with the not-a-graph-database wording;+ self-links → 400; unknown brain → 404 `exists:false`.++## What this demo does NOT claim++- No semantic understanding: detectors are lexical, the embedder is+ feature hashing — never "neural".+- No hidden-connection discovery: authored links are re-indexed from what+ you wrote; same-folder/section-order links are structural priors and are+ labeled as such.+- No graph-database features: no path queries, no PageRank, no unbounded+ traversal — expansion is capped at 2 hops per call, by design.+- No performance numbers: this harness measures correctness, not speed.+ Anything not measured here is "not measured".diff --git a/demo/usecases/second-brain/boot-and-brain.sh b/demo/usecases/second-brain/boot-and-brain.shnew file mode 100755index 000000000..e2597ae52--- /dev/null+++ b/demo/usecases/second-brain/boot-and-brain.sh@@ -0,0 +1,60 @@+#!/usr/bin/env bash+# boot-and-brain.sh — one command from the demo vault to a running brain.+#+# Runs `xerj brain` on the generated corpus. That single command boots (or+# attaches to) a local xerj server, indexes the vault with link detection on,+# and prints the console URL — this script only stages the inputs and then+# verifies the server actually answers.+#+# Idempotent: a second run attaches to the already-running server and the+# indexing converges (same edge_ids — detector valid_at comes from pinned+# mtimes). Ports: default 9331 (the booted server also takes +1/+2 for+# rest/grpc — i.e. 9331-9333, inside the 9330-9345 demo range; never+# 9200/9310).+#+# Env overrides:+# XERJ_BRAIN_DEMO_ROOT working root (default ${TMPDIR:-/tmp}/xerj-second-brain-demo)+# XERJ_BRAIN_DEMO_PORT es-compat port (default 9331)+# XERJ_BIN xerj binary (default <repo>/engine/target/release/xerj)+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+REPO="$(cd "$HERE/../../.." && pwd)"++DEMO_ROOT="${XERJ_BRAIN_DEMO_ROOT:-${TMPDIR:-/tmp}/xerj-second-brain-demo}"+PORT="${XERJ_BRAIN_DEMO_PORT:-9331}"+XERJ_BIN="${XERJ_BIN:-$REPO/engine/target/release/xerj}"+BRAIN="vault"+URL="http://localhost:$PORT"+CORPUS="$DEMO_ROOT/vault"+DATA_DIR="$DEMO_ROOT/data"++fail() { echo "FAIL: $*" >&2; exit 1; }++[ -x "$XERJ_BIN" ] || fail "xerj binary not found at $XERJ_BIN — build it: (cd $REPO/engine && cargo build --release -j 32 -p xerj-server)"++# Corpus: regenerate only when absent, so mtimes (= edge valid_at) stay put+# across re-runs of this script.+if [ ! -d "$CORPUS" ]; then+ "$HERE/gen-corpus.sh" "$CORPUS"+fi++echo "── xerj brain $CORPUS (brain '$BRAIN', $URL) ──"+"$XERJ_BIN" brain "$CORPUS" --brain "$BRAIN" --url "$URL" \+ --data-dir "$DATA_DIR" --no-open++# `xerj brain` already waited for readiness; verify independently anyway.+for _ in $(seq 1 30); do+ code=$(curl -s -o /dev/null -w '%{http_code}' "$URL/health/ready" || true)+ [ "$code" = "200" ] && break+ sleep 2+done+[ "$code" = "200" ] || fail "server at $URL not ready (last probe HTTP $code)"++KEY_FILE="$DATA_DIR/admin.key"+[ -s "$KEY_FILE" ] || echo "note: no $KEY_FILE — server may be running unauthenticated or was booted elsewhere"++echo "PASS: brain '$BRAIN' live at $URL (api key: $KEY_FILE, data: $DATA_DIR)"+echo "next: ./transcript.sh # the API proof"+echo " ./mcp-smoke.sh # the agent surface"+echo "stop: kill \$(cat $DATA_DIR/server.pid)"diff --git a/demo/usecases/second-brain/gen-corpus.sh b/demo/usecases/second-brain/gen-corpus.shnew file mode 100755index 000000000..d8cb53027--- /dev/null+++ b/demo/usecases/second-brain/gen-corpus.sh@@ -0,0 +1,486 @@+#!/usr/bin/env bash+# gen-corpus.sh [DEST] — write the second-brain demo vault: a realistic+# personal knowledge base (bread baking + a bakery-launch project) that+# exercises every edge detector:+#+# notes/ 15 zettels, [[wikilink]]-rich → wikilink@1+# projects/bakery-launch/ 6 files, relative markdown links → mdlink@1+# clippings/ 4 saved HTML pages, <a href> → href@1+# journal/ 12 dated entries, NO authored links → structural only+# reading/ 1 long note (>2 sections) → sequence@1+# (every directory with 2+ files also chains) → samedir@1+#+# Determinism: file bytes are fixed and every mtime is pinned with `touch -d`.+# Detector edges take `valid_at` from mtime, so re-running this generator and+# re-running `xerj brain` reproduces the SAME edge_ids every time (only+# `created_at` — when the brain learned it — is wall clock, by contract).+#+# One wikilink is deliberately dangling ([[banana-bread]] has no note): the+# run summary must count it as unresolved, never invent an edge for it.+#+# Idempotent: wipes and rewrites DEST on every run.+set -euo pipefail++DEMO_ROOT="${XERJ_BRAIN_DEMO_ROOT:-${TMPDIR:-/tmp}/xerj-second-brain-demo}"+DEST="${1:-$DEMO_ROOT/vault}"++rm -rf "$DEST"+mkdir -p "$DEST/notes" "$DEST/projects/bakery-launch" "$DEST/clippings" \+ "$DEST/journal" "$DEST/reading"++# ── notes/ — the zettel cluster (wikilinks) ──────────────────────────────+cat > "$DEST/notes/sourdough-starter.md" <<'EOF'+# Sourdough starter++A starter is a stable culture of wild yeast and lactobacilli in a flour-water+paste. Mine is 100% hydration, fed daily — see [[starter-maintenance]] for the+schedule and [[hydration]] for what the percentage means.++A young, sweet starter builds a milder [[levain]]; an older, more acidic one+brings tang but weakens gluten. Starting a new culture goes fastest with some+[[rye-flour]] in the first feeds — more wild yeast lives on the bran.+EOF++cat > "$DEST/notes/hydration.md" <<'EOF'+# Hydration++Hydration is water as a percentage of flour weight, the core of+[[baker-percentages]]. 65% is a tight sandwich loaf; 75-80% is where open+[[crumb-structure]] lives; past 85% the dough demands real skill.++Higher hydration = looser dough, longer bulk, bigger holes. It is the single+most consequential number in the formula.+EOF++cat > "$DEST/notes/autolyse.md" <<'EOF'+# Autolyse++Mix just flour and water and rest 20-60 minutes before adding salt and+starter. Enzymes begin [[gluten-development]] with zero kneading, and the+flour hydrates fully — see [[hydration]].++Skip it for rye-heavy doughs; rye has little gluten to develop anyway.+EOF++cat > "$DEST/notes/bulk-fermentation.md" <<'EOF'+# Bulk fermentation++The first rise, from mixing until divide. This is where flavour and strength+happen: stretch-and-folds every 30-45 minutes build [[gluten-development]]+without a mixer.++Judge by the dough, not the clock — 50% growth, domed edges, a few surface+bubbles. What follows is shaping and final [[proofing]].+EOF++cat > "$DEST/notes/gluten-development.md" <<'EOF'+# Gluten development++Glutenin and gliadin cross-link into the network that traps gas. Windowpane+test: stretch a piece thin enough to read through without tearing.++Strong development plus good fermentation gives the open, glossy+[[crumb-structure]] everyone photographs.+EOF++cat > "$DEST/notes/levain.md" <<'EOF'+# Levain++An offshoot of the [[sourdough-starter]] built for one specific bake, so the+mother culture is never depleted. Build it 4-12 hours ahead, use it at peak.++The levain's ripeness sets the tempo of [[bulk-fermentation]]: a ripe, active+levain can cut bulk time by a third.+EOF++cat > "$DEST/notes/scoring.md" <<'EOF'+# Scoring++The lame cut that tells the loaf where to open. Shallow angled cuts make ears;+deep vertical cuts make width. Score decisively — hesitation drags.++Scoring is the release valve for [[oven-spring]]: an unscored high-hydration+loaf bursts wherever the crust is weakest.+EOF++cat > "$DEST/notes/oven-spring.md" <<'EOF'+# Oven spring++The dramatic rise in the first 15 minutes of the bake: trapped CO2 expands,+water flashes to steam, and yeast has one last party before 60°C.++Steam is everything — it keeps the crust soft long enough to expand, which is+the whole point of the [[dutch-oven-method]]. Good [[scoring]] directs it.+EOF++cat > "$DEST/notes/baker-percentages.md" <<'EOF'+# Baker's percentages++Everything is expressed relative to total flour = 100%. Water 75%, salt 2%,+starter 20% — a formula, not a recipe, so it scales to any batch size.++[[hydration]] is the percentage that changes the bread most.+EOF++cat > "$DEST/notes/dutch-oven-method.md" <<'EOF'+# Dutch oven method++A preheated cast-iron pot is a home oven's steam injection: the loaf's own+moisture stays trapped for the first 20 minutes, maximising [[oven-spring]].++Lid on 20 min at 250°C, lid off 20-25 min at 230°C. One day I should try the+same trick for [[banana-bread]], though that is a quick bread and probably+does not care.+EOF++cat > "$DEST/notes/rye-flour.md" <<'EOF'+# Rye flour++Low gluten, high enzymes, ferments fast. Even 10% rye darkens flavour+noticeably; 100% rye is a different craft entirely (dense, pentosan-bound).++Compared with [[whole-wheat]], rye brings more sourness and stickier dough.+It is the best fuel for a young [[sourdough-starter]].+EOF++cat > "$DEST/notes/whole-wheat.md" <<'EOF'+# Whole wheat++Bran and germ included: more flavour, more nutrition, and sharp bran edges+that cut gluten strands. Compensate with more water — add 5% [[hydration]]+for every 20% whole grain — and gentler handling than white flour.++See [[rye-flour]] for the other whole-grain regular in my rotation.+EOF++cat > "$DEST/notes/crumb-structure.md" <<'EOF'+# Crumb structure++The pattern of holes in the slice — the bake's report card. Even, open,+glossy alveoli need three things at once: enough [[hydration]], full+[[gluten-development]], and fermentation stopped at the right moment.++Dense streaks near the bottom usually mean underproofing, not underbaking.+EOF++cat > "$DEST/notes/proofing.md" <<'EOF'+# Proofing++The final rise after shaping, seam-side up in a banneton. Poke test: a slow,+partial spring-back means ready.++Overproofed dough has spent its gas — it flattens when turned out. When in+doubt, cold-retard overnight; the fridge slows [[bulk-fermentation]] chemistry+to a crawl and makes [[scoring]] far easier on the firm surface.+EOF++cat > "$DEST/notes/starter-maintenance.md" <<'EOF'+# Starter maintenance++Daily at room temperature: discard to 20g, feed 1:4:4. Weekly if refrigerated;+revive with two feeds before baking. Hooch on top just means hungry.++The mother [[sourdough-starter]] never goes into dough directly — build a+[[levain]] instead and keep the culture safe.+EOF++# ── vault root — the entry note ──────────────────────────────────────────+cat > "$DEST/README.md" <<'EOF'+# Baking vault++My second brain for bread. The science lives in the zettels — start from+[[sourdough-starter]] and [[bulk-fermentation]] — the bakery plan lives in+[the project folder](projects/bakery-launch/01-business-plan.md), and every+bake gets a line in the journal.+EOF++# ── projects/bakery-launch/ — relative markdown links ────────────────────+cat > "$DEST/projects/bakery-launch/01-business-plan.md" <<'EOF'+# Micro-bakery: business plan++Weekend micro-bakery, 40 loaves a batch, subscription-first. The product is+the naturally-leavened loaf my [starter culture](../../notes/sourdough-starter.md)+already makes — the plan is only about repeating it at scale, which is worked+out in [recipes at scale](03-recipes-at-scale.md).++Revenue target: 40 loaves × 2 bakes × €6.50 = €520/week before costs.+EOF++cat > "$DEST/projects/bakery-launch/02-equipment.md" <<'EOF'+# Equipment list++The deck oven replaces the [dutch oven method](../../notes/dutch-oven-method.md)+with real steam injection. Until it arrives, batch-baking continues in three+cast-iron pots.++- Rofco B40 deck oven — quoted €1,850 (see https://www.rofco.be for specs)+- Spiral mixer, 20L+- Proofing retarder (a converted fridge for now)+EOF++cat > "$DEST/projects/bakery-launch/03-recipes-at-scale.md" <<'EOF'+# Recipes at scale++Everything converts through [baker's percentages](../../notes/baker-percentages.md):+the formula is batch-size-free by construction. The house loaf stays at 78%+[hydration](../../notes/hydration.md) — customers buy the open crumb.++Scale test log: 4kg flour batch ferments ~20% faster than 1kg (thermal mass);+pull bulk earlier.+EOF++cat > "$DEST/projects/bakery-launch/04-permits.md" <<'EOF'+# Permits and registration++Cottage food registration first, full food-business licence only if the+weekly volume passes the exemption threshold. Kitchen inspection checklist+requested from the municipality (https://example.gov/food-business).++Insurance quote pending. No links to the science here on purpose — this file+is pure bureaucracy.+EOF++cat > "$DEST/projects/bakery-launch/05-launch-checklist.md" <<'EOF'+# Launch checklist++- [ ] Sign off the [business plan](01-business-plan.md)+- [ ] Close out [permits](04-permits.md)+- [ ] Deck oven delivered and cured+- [ ] Subscription page live+- [ ] Dry run: two full batches back to back+EOF++cat > "$DEST/projects/bakery-launch/notes-from-mentor.md" <<'EOF'+# Notes from the mentor call++Talked an hour with Ana (ex-Poilâne). Her rules, verbatim:++"Volume changes fermentation more than any recipe tweak — re-learn+[bulk fermentation](../../notes/bulk-fermentation.md) at every batch size."++"Sell the schedule, not the bread. People subscribe to Saturday."+EOF++# ── clippings/ — saved HTML pages (href) ─────────────────────────────────+cat > "$DEST/clippings/maillard-reaction.html" <<'EOF'+<!DOCTYPE html>+<html>+<head><title>The Maillard reaction in bread crust</title></head>+<body>+<h1>The Maillard reaction in bread crust</h1>+<p>Browning is chemistry between amino acids and reducing sugars above+140°C. In a lean dough it is concentrated at the crust, which is why+steam timing — and with it <a href="../notes/oven-spring.md">oven+spring</a> — decides both colour and flavour.</p>+<p>Source: <a href="https://en.wikipedia.org/wiki/Maillard_reaction">the+Wikipedia article</a> this clip summarises.</p>+</body>+</html>+EOF++cat > "$DEST/clippings/flour-protein-guide.html" <<'EOF'+<!DOCTYPE html>+<html>+<head><title>Flour protein content guide</title></head>+<body>+<h1>Flour protein, by the numbers</h1>+<p>Bread flour 12-14%, all-purpose 10-12%, pastry 8-9%. Whole-grain flours+report high protein but behave weaker — my notes on+<a href="../notes/whole-wheat.md">whole wheat</a> and+<a href="../notes/rye-flour.md">rye</a> cover why the bran matters more than+the number.</p>+</body>+</html>+EOF++cat > "$DEST/clippings/steam-injection.html" <<'EOF'+<!DOCTYPE html>+<html>+<head><title>Steam in professional deck ovens</title></head>+<body>+<h1>Steam in professional deck ovens</h1>+<p>Commercial deck ovens inject steam for the first minutes of the bake; the+home equivalent is the <a href="../notes/dutch-oven-method.md">dutch oven+method</a>. Overdo steam and the crust turns leathery instead of crisp.</p>+</body>+</html>+EOF++cat > "$DEST/clippings/banneton-care.html" <<'EOF'+<!DOCTYPE html>+<html>+<head><title>Banneton care</title></head>+<body>+<h1>Banneton care</h1>+<p>Flour the basket with rice flour (it does not hydrate into glue), never+wash it, dry it after every use. A well-kept banneton releases the loaf+cleanly at the end of <a href="../notes/proofing.md">proofing</a>.</p>+</body>+</html>+EOF++# ── journal/ — dated entries, deliberately link-free ─────────────────────+# The structural-only corner of the vault: every edge here comes from+# samedir@1 (files sitting together), none from authored links — this is+# what the dashboard's AUTHORED vs STRUCTURAL split makes visible.+cat > "$DEST/journal/2026-05-03.md" <<'EOF'+Saturday bake #31. 75% hydration, 20% levain, 2% salt. Bulk 4h30 at 24C.+Best ear so far — the new blade angle works. Crumb slightly tight at the+bottom, probably shaped too cold.+EOF+cat > "$DEST/journal/2026-05-10.md" <<'EOF'+Bake #32. Pushed hydration to 78%. Dough was a puddle after mix but three+folds saved it. Open crumb, glossy, the best slice photo yet. Neighbour+claimed a whole loaf.+EOF+cat > "$DEST/journal/2026-05-17.md" <<'EOF'+Bake #33. Tried 15% rye. Fermentation went visibly faster — pulled bulk at+3h45 and it was already at the edge. Tangier than usual, crumb tighter.+Would do 10% next time.+EOF+cat > "$DEST/journal/2026-05-24.md" <<'EOF'+Bake #34. Cold-retarded overnight for the first time in weeks. Scoring on+fridge-cold dough is a different sport — clean single slash, textbook ear.+Flavour deeper. This is the schedule now.+EOF+cat > "$DEST/journal/2026-05-31.md" <<'EOF'+Bake #35. Disaster bake. Forgot salt until the second fold, dough never+tightened, loaf spread flat. Still tasted fine toasted. Lesson: mise en+place, always.+EOF+cat > "$DEST/journal/2026-06-07.md" <<'EOF'+Bake #36. Back to the standard formula to recalibrate. Solid, unremarkable,+exactly what a control bake should be. Starter smells great since the rye+feeds.+EOF+cat > "$DEST/journal/2026-06-14.md" <<'EOF'+Bake #37. Doubled the batch to four loaves as a scale test for the bakery+idea. The mixing bowl is now officially too small. Fermentation ran hotter+with the bigger dough mass — pulled bulk 30 min early.+EOF+cat > "$DEST/journal/2026-06-21.md" <<'EOF'+Bake #38. Four loaves again, staggered bakes. Second pair overproofed while+waiting for the oven — need a retarder, added it to the equipment list.+Subscribers (mum, two colleagues) unbothered.+EOF+cat > "$DEST/journal/2026-06-28.md" <<'EOF'+Bake #39. Whole-wheat at 30%, water up 7%. Nutty, moist, keeps better than+the white loaf. A keeper formula for the winter menu.+EOF+cat > "$DEST/journal/2026-07-05.md" <<'EOF'+Bake #40. Milestone bake, gave loaves away at the park run. Two people asked+if they could order. Wrote both emails down. This might actually be a+business.+EOF+cat > "$DEST/journal/2026-07-12.md" <<'EOF'+Bake #41. First dry run of the Saturday production schedule: levain at 06:00,+mix at 07:30, bake windows 13:00 and 14:00. Made it with 20 minutes to+spare. Feet hurt.+EOF+cat > "$DEST/journal/2026-07-19.md" <<'EOF'+Bake #42. Second dry run, smoother. Timed every step for the checklist.+Best batch consistency yet — all eight loaves within 40g of each other.+Ready to sign off the plan.+EOF++# ── reading/ — one long note that splits into sections (sequence) ────────+# ~3.3KB of prose — past the 2KB section target, so autoindex splits it,+# and each section s_{i} gets a sequence@1 edge from s_{i-1} — the+# "document's own narrative order survives the split" signal.+cat > "$DEST/reading/bread-science-notes.md" <<'EOF'+# Reading notes: bread science, one long braid++Working notes from three books read back to back — Hamelman's "Bread",+Forkish's "Flour Water Salt Yeast", and Robertson's "Tartine Bread" — folded+together into one narrative so future-me stops re-reading the same chapters.+The through-line of all three: bread is four ingredients and three control+knobs (time, temperature, hydration), and every technique in every book is+just a different grip on those knobs.++Hamelman is the engineer. His chapters on fermentation read like process+control: the dough is a bioreactor, the baker's job is to keep it inside an+envelope, and every visible sign — doming, jiggle, surface bubbles — is+telemetry, not folklore. His preferments taxonomy (poolish, biga, levain,+pate fermentee) finally made the terminology stick: they differ in hydration+and yeast source, nothing else. The rest is marketing by tradition. What I+took into my own process: read the dough at fixed checkpoints instead of+checking the clock, and log what the checkpoints looked like. That is the+whole reason the journal folder in this vault exists.++Forkish is the schedule. His entire system is built so that a person with a+job can bake serious bread: overnight bulk at cool temperatures, tiny levain+percentages, everything tuned so the dough's slow chemistry does the work+while you sleep. The insight that transfers: levain percentage is a throttle.+Five percent ripe levain and a 78-degree kitchen gives the same fermentation+arc as twenty percent in a cold one — pick the percentage that makes the+timeline fit your life, not the recipe's. His iron discipline about dough+temperature (he measures water to the degree) looked fussy until the+four-loaf scale test proved him right: mass changes everything, and the only+way to keep the arc constant across batch sizes is to hit the same dough+temperature every time.++Robertson is the palate. Tartine's method is the least precise of the three+and the most opinionated about what the loaf should BE: custardy open crumb,+blistered mahogany crust, acidity in balance rather than in charge. His young+levain doctrine — use the starter hours before it peaks, when it smells of+yogurt rather than vinegar — directly contradicts the older sour-is-authentic+school, and my own side-by-side bakes came down firmly on Robertson's side.+The same bake also taught me that his shaping technique carries more of the+open-crumb result than his formula does: surface tension built during+shaping is what lets a wet dough stand tall through oven spring instead of+relaxing into a pancake.++Where the three books disagree, the disagreement is itself the lesson.+Hamelman wants strong development early (improved mix), Forkish wants almost+none (folds only), Robertson is in between — and all three produce great+bread, because development and fermentation time trade off against each+other. Strong early gluten tolerates shorter bulk; lazy mixing demands the+long slow build that also happens to develop more flavour. Pick the pair+that fits the schedule. There is no single correct process, only internally+consistent ones — which is the most liberating sentence in any of the three+books, and the reason this note exists as one braid instead of three+summaries.+EOF++# ── pin every mtime (valid_at = mtime; determinism depends on this) ──────+t() { touch -d "$1" "$DEST/$2"; }++t "2026-05-01 09:00:00 UTC" README.md+t "2026-05-02 10:00:00 UTC" notes/sourdough-starter.md+t "2026-05-04 10:00:00 UTC" notes/starter-maintenance.md+t "2026-05-06 10:00:00 UTC" notes/hydration.md+t "2026-05-08 10:00:00 UTC" notes/baker-percentages.md+t "2026-05-11 10:00:00 UTC" notes/levain.md+t "2026-05-14 10:00:00 UTC" notes/autolyse.md+t "2026-05-18 10:00:00 UTC" notes/bulk-fermentation.md+t "2026-05-21 10:00:00 UTC" notes/gluten-development.md+t "2026-05-25 10:00:00 UTC" notes/crumb-structure.md+t "2026-05-28 10:00:00 UTC" notes/proofing.md+t "2026-06-01 10:00:00 UTC" notes/scoring.md+t "2026-06-04 10:00:00 UTC" notes/oven-spring.md+t "2026-06-08 10:00:00 UTC" notes/dutch-oven-method.md+t "2026-06-10 10:00:00 UTC" notes/rye-flour.md+t "2026-06-11 10:00:00 UTC" notes/whole-wheat.md++t "2026-06-12 12:00:00 UTC" clippings/maillard-reaction.html+t "2026-06-14 12:00:00 UTC" clippings/flour-protein-guide.html+t "2026-06-16 12:00:00 UTC" clippings/steam-injection.html+t "2026-06-18 12:00:00 UTC" clippings/banneton-care.html++for d in 2026-05-03 2026-05-10 2026-05-17 2026-05-24 2026-05-31 \+ 2026-06-07 2026-06-14 2026-06-21 2026-06-28 \+ 2026-07-05 2026-07-12 2026-07-19; do+ t "$d 20:00:00 UTC" "journal/$d.md"+done++t "2026-06-20 09:00:00 UTC" projects/bakery-launch/01-business-plan.md+t "2026-06-23 09:00:00 UTC" projects/bakery-launch/02-equipment.md+t "2026-06-26 09:00:00 UTC" projects/bakery-launch/03-recipes-at-scale.md+t "2026-06-29 09:00:00 UTC" projects/bakery-launch/04-permits.md+t "2026-07-02 09:00:00 UTC" projects/bakery-launch/05-launch-checklist.md+t "2026-07-03 09:00:00 UTC" projects/bakery-launch/notes-from-mentor.md++t "2026-07-06 21:00:00 UTC" reading/bread-science-notes.md++n=$(find "$DEST" -type f | wc -l)+echo "corpus: $n files under $DEST (bytes and mtimes pinned — re-runs are identical)"diff --git a/demo/usecases/second-brain/mcp-smoke.sh b/demo/usecases/second-brain/mcp-smoke.shnew file mode 100755index 000000000..3a5c3310d--- /dev/null+++ b/demo/usecases/second-brain/mcp-smoke.sh@@ -0,0 +1,86 @@+#!/usr/bin/env bash+# mcp-smoke.sh — smoke-test the agent surface: the xerj-mcp stdio proxy+# pointed (via XERJ_URL) at the demo brain boot-and-brain.sh started.+#+# Checks that the four brain tools are listed (xerj_brain_ego, xerj_brain_link,+# xerj_brain_unlink, xerj_brain_overview) and that a real tools/call round-trip+# against the live brain succeeds. If the brain tools are not present, that is+# reported as a FAIL — never papered over.+set -uo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+REPO="$(cd "$HERE/../../.." && pwd)"++DEMO_ROOT="${XERJ_BRAIN_DEMO_ROOT:-${TMPDIR:-/tmp}/xerj-second-brain-demo}"+PORT="${XERJ_BRAIN_DEMO_PORT:-9331}"+MCP_BIN="${XERJ_MCP_BIN:-$REPO/engine/target/release/xerj-mcp}"+BRAIN="vault"+KEY_FILE="$DEMO_ROOT/data/admin.key"+WORK="$(mktemp -d)"+trap 'rm -rf "$WORK"' EXIT++if [ ! -x "$MCP_BIN" ]; then+ echo "FAIL: xerj-mcp binary not found at $MCP_BIN"+ echo " build it: (cd $REPO/engine && cargo build --release -j 32 -p xerj-mcp)"+ exit 2+fi++export XERJ_URL="http://localhost:$PORT"+[ -s "$KEY_FILE" ] && export XERJ_AUTH="ApiKey $(cat "$KEY_FILE")"++PASSED=0 FAILED=0+ok() { echo " PASS: $*"; PASSED=$((PASSED + 1)); }+bad() { echo " FAIL: $*"; FAILED=$((FAILED + 1)); }++echo "── xerj-mcp smoke · XERJ_URL=$XERJ_URL ──"++# Line-delimited JSON-RPC over stdio: initialize → tools/list → one real+# tools/call against the live brain.+{+ printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"mcp-smoke","version":"0"}}}'+ printf '%s\n' '{"jsonrpc":"2.0","method":"notifications/initialized"}'+ printf '%s\n' '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'+ printf '%s\n' "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"tools/call\",\"params\":{\"name\":\"xerj_brain_overview\",\"arguments\":{\"brain\":\"$BRAIN\"}}}"+ sleep 2+} | timeout 30 "$MCP_BIN" > "$WORK/out.jsonl" 2> "$WORK/err.log"++if [ ! -s "$WORK/out.jsonl" ]; then+ echo "FAIL: xerj-mcp produced no output; stderr:"+ sed 's/^/ /' "$WORK/err.log"+ exit 1+fi++jq -s '.[] | select(.id == 1)' "$WORK/out.jsonl" > "$WORK/init.json"+jq -s '.[] | select(.id == 2)' "$WORK/out.jsonl" > "$WORK/tools.json"+jq -s '.[] | select(.id == 3)' "$WORK/out.jsonl" > "$WORK/call.json"++jq -e '.result.serverInfo' "$WORK/init.json" >/dev/null 2>&1 \+ && ok "initialize handshake" || bad "initialize handshake"++TOOLS=$(jq -r '.result.tools[].name' "$WORK/tools.json" 2>/dev/null | sort)+echo " tools listed: $(echo "$TOOLS" | tr '\n' ' ')"+for t in xerj_brain_ego xerj_brain_link xerj_brain_unlink xerj_brain_overview; do+ echo "$TOOLS" | grep -qx "$t" && ok "tool $t listed" || bad "tool $t listed"+done++# Honesty strings the design mandates in every brain-tool description.+jq -e '[.result.tools[] | select(.name | startswith("xerj_brain_"))] as $b+ | ($b | length) > 0 and ($b | all(.description | test("not a graph database")))' \+ "$WORK/tools.json" >/dev/null 2>&1 \+ && ok "brain tools carry the not-a-graph-database honesty string" \+ || bad "brain tools carry the not-a-graph-database honesty string"++if jq -e '.result and (.result.isError != true)' "$WORK/call.json" >/dev/null 2>&1; then+ ok "tools/call xerj_brain_overview succeeded"+ jq -e '.result.content[0].text | fromjson | .exists == true and .edges.total >= 1' \+ "$WORK/call.json" >/dev/null 2>&1 \+ && ok "overview payload: exists:true with live edge counts" \+ || bad "overview payload: exists:true with live edge counts"+else+ bad "tools/call xerj_brain_overview succeeded"+fi++echo "──"+echo "mcp-smoke: $PASSED passed, $FAILED failed"+[ "$FAILED" -eq 0 ] && echo "RESULT: PASS" || echo "RESULT: FAIL"+exit $((FAILED > 0))diff --git a/demo/usecases/second-brain/transcript.sh b/demo/usecases/second-brain/transcript.shnew file mode 100755index 000000000..588a31c3e--- /dev/null+++ b/demo/usecases/second-brain/transcript.sh@@ -0,0 +1,150 @@+#!/usr/bin/env bash+# transcript.sh — the API proof for the second-brain demo, against the server+# boot-and-brain.sh started. Every claim in the case study that touches the+# HTTP surface is exercised here; every number printed is read from a live+# response, never hard-coded.+#+# Covers: overview (counts, detectors, honesty fields) · ego (evidence,+# node hydration, not_shown) · link (idempotent assert with evidence) ·+# unlink (retire, never delete) · as_of replay (the retired link is still+# believed in the past) · the two normative 400s (hops cap wording,+# self-edge) · unknown-brain 404.+#+# Idempotent: safe to re-run; the manual edge it asserts has a fixed+# valid_at, so re-runs converge on the same edge_id.+set -uo pipefail++DEMO_ROOT="${XERJ_BRAIN_DEMO_ROOT:-${TMPDIR:-/tmp}/xerj-second-brain-demo}"+PORT="${XERJ_BRAIN_DEMO_PORT:-9331}"+URL="http://localhost:$PORT"+BRAIN="vault"+KEY_FILE="$DEMO_ROOT/data/admin.key"+WORK="$(mktemp -d)"+trap 'rm -rf "$WORK"' EXIT++AUTH=()+[ -s "$KEY_FILE" ] && AUTH=(-H "Authorization: ApiKey $(cat "$KEY_FILE")")++PASSED=0 FAILED=0+ok() { echo " PASS: $*"; PASSED=$((PASSED + 1)); }+bad() { echo " FAIL: $*"; FAILED=$((FAILED + 1)); }+ck() { local desc="$1"; shift; if "$@" >/dev/null 2>&1; then ok "$desc"; else bad "$desc"; fi; }++# req METHOD PATH [BODY] [-G-style query pairs after --] → status in $STATUS,+# body in $WORK/resp.json+req() {+ local method="$1" path="$2" body="${3:-}"+ shift; shift; [ $# -gt 0 ] && shift+ local args=(-s -o "$WORK/resp.json" -w '%{http_code}' -X "$method" "${AUTH[@]}")+ if [ -n "$body" ]; then+ args+=(-H 'Content-Type: application/json' -d "$body")+ fi+ local q=()+ for kv in "$@"; do q+=(--data-urlencode "$kv"); done+ if [ ${#q[@]} -gt 0 ]; then+ STATUS=$(curl "${args[@]}" -G "${q[@]}" "$URL$path")+ else+ STATUS=$(curl "${args[@]}" "$URL$path")+ fi+}+J() { jq -e "$@" "$WORK/resp.json"; }++echo "── second-brain API transcript · brain '$BRAIN' at $URL ──"++code=$(curl -s -o /dev/null -w '%{http_code}' "$URL/health/ready" || true)+if [ "$code" != "200" ]; then+ echo "FAIL: no server at $URL (HTTP $code) — run ./boot-and-brain.sh first"+ exit 1+fi++# ── 1. overview: the brain-level truth ───────────────────────────────────+echo "[1] GET /_graph/$BRAIN/overview"+req GET "/_graph/$BRAIN/overview"+ck "HTTP 200" test "$STATUS" = 200+ck "exists == true" J '.exists == true'+ck "contract xerj-second-brain/1" J '.contract == "xerj-second-brain/1"'+ck "embedder honesty marker is lexical" J '.embedder == "lexical-feature-hash"'+ck "edges.total >= 1" J '.edges.total >= 1'+ck "invalidated == total - live" J '.edges.invalidated == .edges.total - .edges.live'+for tag in wikilink@1 mdlink@1 href@1 sequence@1 samedir@1; do+ ck "detector $tag has live edges" J --arg t "$tag" '[.detectors[] | select(.detector == $t and .live > 0)] | length == 1'+done+ck "not_shown accounting present" J '.not_shown | has("types_not_listed") and has("hubs_out_not_listed")'+echo " measured: $(jq -c '{edges, types: [.types[] | {(.type): .live}] | add}' "$WORK/resp.json")"++HUB=$(jq -r '.hubs.out[0].id' "$WORK/resp.json")+ck "an out-hub exists" test -n "$HUB" -a "$HUB" != null+echo " hub (most outbound links): $HUB"++# ── 2. ego: one note's annotated neighborhood ────────────────────────────+echo "[2] GET /_graph/$BRAIN/ego (hub, 2 hops, nodes hydrated)"+req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "hops=2" "include_nodes=true"+ck "HTTP 200" test "$STATUS" = 200+ck "contract xerj-second-brain/1" J '.contract == "xerj-second-brain/1"'+ck "edges returned" J '.edges | length >= 1'+ck "every edge carries hop + direction" J 'all(.edges[]; .hop >= 1 and (.direction == "out" or .direction == "in"))'+ck "detected edges carry evidence quotes" J 'all(.edges[] | select(.detector != "manual@1"); .evidence.quote | type == "string")'+ck "node summaries hydrated" J '.nodes | type == "object" and length >= 1'+ck "not_shown accounting present" J '.not_shown | has("edges_clipped") and has("expired_excluded") and has("dangling_nodes")'+echo " measured: $(jq -c '{edges: (.edges | length), neighbors: (.neighbors | length), not_shown}' "$WORK/resp.json")"++DST=$(jq -r '.neighbors[0].id' "$WORK/resp.json")+ck "a neighbor exists to link against" test -n "$DST" -a "$DST" != null++# ── 3. link: idempotent assert with evidence ─────────────────────────────+echo "[3] POST /_graph/$BRAIN/link (manual edge, fixed valid_at)"+LINK_BODY=$(jq -nc --arg src "$HUB" --arg dst "$DST" '{+ src: $src, dst: $dst, type: "references",+ valid_at: "2026-07-01T00:00:00Z",+ confidence: 0.9,+ evidence: {quote: "asserted by transcript.sh for the retire-then-replay proof",+ source: "transcript.sh", offset: 0}}')+req POST "/_graph/$BRAIN/link" "$LINK_BODY"+ck "HTTP 200 or 201" test "$STATUS" = 200 -o "$STATUS" = 201+EDGE_ID=$(jq -r '.edge_id' "$WORK/resp.json")+ck "edge_id returned" test -n "$EDGE_ID" -a "$EDGE_ID" != null+req POST "/_graph/$BRAIN/link" "$LINK_BODY"+ck "identical re-assert → created:false" J '.created == false'+ck "identical re-assert → same edge_id" J --arg e "$EDGE_ID" '.edge_id == $e'+echo " edge_id: $EDGE_ID"+curl -s -o /dev/null "${AUTH[@]}" -X POST "$URL/.xerj-memory-$BRAIN-edges/_refresh" || true++req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "types=references"+ck "asserted link visible live" J --arg e "$EDGE_ID" '[.edges[] | select(.edge_id == $e)] | length == 1'++# ── 4. unlink: retire, never delete ──────────────────────────────────────+echo "[4] DELETE /_graph/$BRAIN/link/$EDGE_ID (retire as of 2026-07-25)"+req DELETE "/_graph/$BRAIN/link/$EDGE_ID" "" -- "invalid_at=2026-07-25T00:00:00Z"+ck "HTTP 200" test "$STATUS" = 200+ck "invalidated == true" J '.invalidated == true'+ck "expired_at recorded (system clock)" J '.expired_at | type == "number"'+curl -s -o /dev/null "${AUTH[@]}" -X POST "$URL/.xerj-memory-$BRAIN-edges/_refresh" || true++# ── 5. time travel: the retired link is still believed in the past ───────+echo "[5] as_of replay"+req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "types=references" "as_of=2026-07-10T00:00:00Z"+ck "as_of 2026-07-10 → link WAS believed" J --arg e "$EDGE_ID" '[.edges[] | select(.edge_id == $e)] | length == 1'+req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "types=references"+ck "now → link no longer believed" J --arg e "$EDGE_ID" '[.edges[] | select(.edge_id == $e)] | length == 0'+ck "…and the exclusion is COUNTED" J '.not_shown.expired_excluded >= 1'+req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "types=references" "include_expired=true"+ck "include_expired → still queryable, with invalid_at" \+ J --arg e "$EDGE_ID" '[.edges[] | select(.edge_id == $e and (.invalid_at | type == "number"))] | length == 1'++# ── 6. the normative refusals ────────────────────────────────────────────+echo "[6] normative errors"+req GET "/_graph/$BRAIN/ego" "" -- "node=$HUB" "hops=3"+ck "hops=3 → HTTP 400" test "$STATUS" = 400+ck "…with the not-a-graph-database wording" J '.error.reason | contains("hops is capped at 2") and contains("not a graph")'+req POST "/_graph/$BRAIN/link" '{"src":"a","dst":"a","type":"references"}'+ck "self-edge → HTTP 400" test "$STATUS" = 400+ck "…with the normative reason" J '.error.reason | contains("self-edges are not allowed (src == dst)")'+req GET "/_graph/no-such-brain/overview"+ck "unknown brain → HTTP 404" test "$STATUS" = 404+ck "…body says exists:false" J '.exists == false'++# ── summary ──────────────────────────────────────────────────────────────+echo "──"+echo "transcript: $PASSED passed, $FAILED failed"+[ "$FAILED" -eq 0 ] && echo "RESULT: PASS" || echo "RESULT: FAIL"+exit $((FAILED > 0))diff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.mdindex 7cb6cd2a7..f48b52713 100644--- a/docs/design/SECOND_BRAIN_SPEC.md+++ b/docs/design/SECOND_BRAIN_SPEC.md@@ -964,7 +964,12 @@ export const secondBrain = {name: 'Second Brain',render: ({ data, time }) => ({title: 'SECOND BRAIN',- kicker: 'WHAT YOUR NOTES BELIEVE · EVERY LINK HAS A QUOTE · REPLAYABLE AT ANY MOMENT',+ // Amended 2026-07-30 (integrator, honesty review): was "EVERY LINK HAS A+ // QUOTE". The schema does not enforce a quote — §4.1 `evidence` is+ // optional (manual/agent asserts may omit it) and structural detectors+ // record a rationale, not note text. The surface may not claim more than+ // the schema enforces.+ kicker: 'WHAT YOUR NOTES BELIEVE · EVERY LINK SHOWS ITS EVIDENCE · REPLAYABLE AT ANY MOMENT',meta: [time, 'XERJ-GRAPH'],panels: [{ id: 'edgesLive', eyebrow: 'BELIEVED AT THIS MOMENT', cols: 3, type: 'metric' },diff --git a/engine/crates/xerj-autoindex/src/extract/yaml_x.rs b/engine/crates/xerj-autoindex/src/extract/yaml_x.rsindex 1f9df5eda..91cf97165 100644--- a/engine/crates/xerj-autoindex/src/extract/yaml_x.rs+++ b/engine/crates/xerj-autoindex/src/extract/yaml_x.rs@@ -20,8 +20,15 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {let yv: serde_yaml::Value = match serde_yaml::Value::deserialize(de) {Ok(v) => v,Err(_) => {+ // MUST break, never continue: serde_yaml's multi-document+ // iterator does not advance past a document that fails to+ // parse — `continue` re-yields the same broken document+ // forever (live hang: a markdown task list `- [ ] …` sniffed+ // as YAML spun autoindex phase A indefinitely). Junk-file the+ // rest of the stream instead; the loss is recorded, not+ // hidden.stats.junk += 1;- continue;+ break;}};any = true;@@ -109,3 +116,47 @@ fn yaml_to_json(v: serde_yaml::Value) -> Value {serde_yaml::Value::Tagged(t) => yaml_to_json(t.value),}}++#[cfg(test)]+mod tests {+ use super::*;++ fn extract_str(text: &str) -> (ExtractStats, usize) {+ let dir = std::env::temp_dir().join(format!(+ "xerj-yamlx-test-{}-{:?}",+ std::process::id(),+ std::thread::current().id()+ ));+ std::fs::create_dir_all(&dir).unwrap();+ let path = dir.join("t.yaml");+ std::fs::write(&path, text).unwrap();+ let mut n = 0usize;+ let stats = extract(&path, false, &mut |_r| {+ n += 1;+ true+ })+ .unwrap();+ std::fs::remove_dir_all(&dir).ok();+ (stats, n)+ }++ /// Regression for a live hang: serde_yaml's multi-document iterator does+ /// not advance past a document that fails to parse, so `continue` on the+ /// error re-yielded the same broken document forever. A markdown task+ /// list sniffed as YAML (`- [ ] …` is invalid YAML) spun autoindex phase+ /// A indefinitely. The extractor must terminate and junk-file instead.+ #[test]+ fn broken_document_terminates_as_junk_not_forever() {+ let (stats, n) = extract_str("# Launch checklist\n- [ ] alpha\n- [ ] beta\n");+ assert_eq!(n, 0, "invalid YAML must not fabricate records");+ assert!(stats.junk >= 1, "the unparseable stream is junk-filed");+ }++ #[test]+ fn valid_multi_document_stream_still_yields_records() {+ let (stats, n) = extract_str("a: 1\nb: two\n---\nc: 3\n");+ assert_eq!(n, 2);+ assert_eq!(stats.records, 2);+ assert_eq!(stats.junk, 0);+ }+}diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex 1bffda1ca..c456e037b 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -228,6 +228,11 @@ fn event_count(state_dir: &Path, kind: &str) -> usize {#[test]fn fresh_publication_skips_delete_and_noop_resume_does_not_append_plan() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();fs::write(@@ -255,6 +260,11 @@ fn fresh_publication_skips_delete_and_noop_resume_does_not_append_plan() {#[test]fn legacy_plan_without_completion_cleans_possible_partial_visibility() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let first_state = tempfile::tempdir().unwrap();let legacy_state = tempfile::tempdir().unwrap();@@ -327,6 +337,11 @@ fn legacy_plan_without_completion_cleans_possible_partial_visibility() {#[test]fn legacy_key_collision_fails_before_visibility_with_scoped_guidance() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();let mut owner = vec![b'x'; 65_537];@@ -393,6 +408,11 @@ fn legacy_key_collision_fails_before_visibility_with_scoped_guidance() {#[test]fn existing_completion_is_invalidated_and_partial_visibility_is_deleted_on_resume() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();// 1: failure immediately after delete; 2: one complete 5,000-doc bulk// then a partial final bulk; 3: two complete bulks then a partial final// bulk immediately before the file_done journal append.@@ -490,6 +510,11 @@ fn existing_completion_is_invalidated_and_partial_visibility_is_deleted_on_resum#[test]fn resume_repairs_kills_after_plan_delete_and_final_bulk_before_file_done() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();for boundary in [1u8, 2, 4] {let rows = 6_001usize;let corpus = tempfile::tempdir().unwrap();@@ -592,6 +617,11 @@ fn file_done_append_and_fsync_failures_are_fatal_and_resume_repairs() {#[test]fn pending_generation_b_is_superseded_when_source_changes_to_c() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();let path = corpus.path().join("records.csv");@@ -657,6 +687,11 @@ fn pending_generation_b_is_superseded_when_source_changes_to_c() {#[test]fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();let original = "id,value\n0,original\n1,original\n";@@ -718,6 +753,11 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {#[test]fn deleting_an_entire_duplicate_group_strands_no_pending_replacement() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ // run_index reaches Journal::file_done; the file_done IO failpoint is+ // a global one-shot, so every file_done-reaching test must hold its+ // lock (armer or not) or it can steal an armed injection. Acquired+ // after FAILPOINT_TEST_LOCK everywhere — one consistent order.+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();fs::write(corpus.path().join("dup-a.csv"), "id,value\n0,dup\n1,dup\n").unwrap();diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex 5f7a2831f..559b0d187 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -293,7 +293,17 @@ fn yaml_line_ratio(nonblank: &[&str]) -> f64 {let re = regex::Regex::new(r"^\s*(- )?[\w.@/-]+:(\s|$)").unwrap();let hits = nonblank.iter()- .filter(|l| re.is_match(l) || l.trim_start().starts_with("- "))+ .filter(|l| {+ let t = l.trim_start();+ // Markdown task-list items (`- [ ]` / `- [x]`) are NOT YAML+ // evidence: `- [ ] text` is invalid YAML (flow sequence followed+ // by a scalar), while checklists are everywhere in real notes.+ // Counting them here routed whole checklist files into the YAML+ // extractor, which can only junk-file them.+ let checkbox =+ t.starts_with("- [ ] ") || t.starts_with("- [x] ") || t.starts_with("- [X] ");+ re.is_match(l) || (t.starts_with("- ") && !checkbox)+ }).count();hits as f64 / nonblank.len() as f64}@@ -335,9 +345,34 @@ fn txt_kind(nonblank: &[&str]) -> Family {if sentence_ratio >= 0.40 {return Family::TxtProse;}++ // Hard-wrapped markdown rescue. Sentence density per LINE undercounts+ // prose whose author wraps at ~70-80 columns: sentences end mid-line, so+ // a five-paragraph note with a `# Title` scores ~0.30 and landed in+ // TxtLines — which silently cost it its title, its section anchors, and+ // (second brain) its wikilink detection. Content evidence, not the+ // extension: a file that OPENS with an ATX heading and shows either+ // markdown link syntax or some terminal punctuation is a markdown+ // document. The heading-opener requirement keeps shebang'd scripts+ // (`#!…`) and most code/config out; the second signal keeps out comment+ // banners over pure record streams.+ let md_link = nonblank+ .iter()+ .any(|l| l.contains("[[") || l.contains("]("));+ if md_heading(nonblank[0]) && (md_link || sentence_ratio >= 0.20) {+ return Family::TxtProse;+ }Family::TxtLines}+/// `# Title` … `###### Title` — an ATX markdown heading (1-6 `#`, a space,+/// then text). `#!/bin/sh` and bare `#` fail the space-then-text rule.+fn md_heading(line: &str) -> bool {+ let t = line.trim_start();+ let hashes = t.bytes().take_while(|&b| b == b'#').count();+ (1..=6).contains(&hashes) && t.as_bytes().get(hashes) == Some(&b' ') && t.len() > hashes + 1+}+/// Quote-aware field split (supports " and ' quoting).fn split_quoted(line: &str, delim: u8) -> Vec<String> {let mut fields = Vec::new();@@ -582,4 +617,50 @@ mod text_family_tests {.join("\n");assert_eq!(kind(&t), Family::TxtProse);}++ /// Regression: a markdown checklist (`- [ ]` task items under a heading)+ /// used to sniff as YAML — 2 of 3 nonblank lines start with `- ` — and+ /// the YAML extractor then junk-filed it (and, before the yaml_x+ /// non-progress fix, hung on it). Checkbox items are invalid YAML and+ /// must not count as YAML evidence.+ #[test]+ fn markdown_checklist_is_prose_not_yaml() {+ let md = "# Launch checklist\n\n\+ - [ ] Sign off the [business plan](01-business-plan.md)\n\+ - [x] Close out permits\n\+ - [ ] Dry run: two full batches back to back\n";+ assert_eq!(classify_full(md), Family::TxtProse);+ // A real YAML list is still YAML.+ let yaml = "- alpha\n- beta\n- gamma\n";+ assert_eq!(classify_full(yaml), Family::Yaml);+ }++ /// Regression (second-brain demo vault, live 2026-07-30): a markdown+ /// note hard-wrapped at ~75 columns averages < 60 chars/line and ends+ /// most lines mid-sentence, so it scored below the 0.40 sentence ratio+ /// and landed in TxtLines — silently losing its title, its `s0` section+ /// anchor, and its wikilink detection (8 of 39 vault files). A file that+ /// opens with an ATX heading and shows markdown link syntax (or some+ /// terminal punctuation) is markdown prose.+ #[test]+ fn hard_wrapped_markdown_note_is_prose_not_lines() {+ let md = "# Hydration\n\n\+ Hydration is water as a percentage of flour weight, the core of\n\+ [[baker-percentages]]. 65% is a tight sandwich loaf; 75-80% is where open\n\+ [[crumb-structure]] lives; past 85% the dough demands real skill.\n\n\+ Higher hydration = looser dough, longer bulk, bigger holes. It is the single\n\+ most consequential number in the formula.\n";+ assert_eq!(classify_full(md), Family::TxtProse);+ // No heading opener → the rescue must not fire: a Python file whose+ // comment banner starts mid-file stays wherever the base heuristics+ // put it, and a shebang is not a heading.+ let sh = "#!/usr/bin/env bash\n\+ set -euo pipefail\n\+ for f in a b c; do\n\+ echo one\n\+ echo two\n\+ echo three\n\+ done\n";+ assert_eq!(classify_full(sh), Family::TxtLines);+ }}diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rsindex 363fceb70..da59f5eb7 100644--- a/engine/crates/xerj-autoindex/src/state.rs+++ b/engine/crates/xerj-autoindex/src/state.rs@@ -541,6 +541,12 @@ mod compatibility_tests {#[test]fn replacement_start_durably_invalidates_done_until_a_durable_commit() {+ // The failpoint is a global one-shot: any test that reaches+ // `file_done` while another test has it armed would consume the+ // injection meant for the armer (observed live 2026-07-30 as a+ // parallel-run flake + poison cascade). Every file_done-reaching+ // test holds this lock, armer or not.+ let _guard = FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let dir = tempfile::tempdir().unwrap();let mut journal =Journal::open(dir.path(), "root", "http://engine", "ax", 300, false).unwrap();@@ -592,6 +598,8 @@ mod compatibility_tests {#[test]fn torn_file_done_tail_is_truncated_before_repair_commit_is_appended() {+ // Reaches file_done → must hold the failpoint lock (see above).+ let _guard = FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let dir = tempfile::tempdir().unwrap();let mut journal =Journal::open(dir.path(), "root", "http://engine", "ax", 300, false).unwrap();@@ -687,6 +695,8 @@ mod compatibility_tests {#[test]fn stale_generation_completion_cannot_clear_newer_pending_replacement() {+ // Reaches file_done → must hold the failpoint lock (see above).+ let _guard = FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let dir = tempfile::tempdir().unwrap();let mut journal =Journal::open(dir.path(), "root", "http://engine", "ax", 300, false).unwrap();diff --git a/engine/crates/xerj-mcp/Cargo.toml b/engine/crates/xerj-mcp/Cargo.tomlindex 62dc5d92d..61b7bc6b0 100644--- a/engine/crates/xerj-mcp/Cargo.toml+++ b/engine/crates/xerj-mcp/Cargo.toml@@ -4,7 +4,7 @@ version.workspace = trueedition.workspace = truelicense.workspace = trueauthors.workspace = true-description = "Model Context Protocol (MCP) stdio server exposing the 6 canonical XERJ agent operations as thin proxies to an ES-compatible XERJ endpoint"+description = "Model Context Protocol (MCP) stdio server exposing the 10 canonical XERJ agent operations (search, memory, second-brain graph) as thin proxies to an ES-compatible XERJ endpoint"[[bin]]name = "xerj-mcp"diff --git a/engine/crates/xerj-mcp/src/main.rs b/engine/crates/xerj-mcp/src/main.rsindex a5023e0ed..cf1e5831d 100644--- a/engine/crates/xerj-mcp/src/main.rs+++ b/engine/crates/xerj-mcp/src/main.rs@@ -2,12 +2,12 @@//!//! This binary speaks the MCP **stdio transport**: newline-delimited//! JSON-RPC 2.0 messages on stdin/stdout. It exposes XERJ to any MCP-capable-//! agent host (Claude Desktop, IDE agents, custom orchestrators) as six+//! agent host (Claude Desktop, IDE agents, custom orchestrators) as ten//! tools that map 1:1 onto XERJ's real, verified REST surface. Every tool is-//! a *thin proxy*: it constructs exactly the request body the running engine+//! a *thin proxy*: it constructs exactly the request the running engine//! already accepts and forwards it to a configurable base URL.//!-//! ## The six canonical agent operations+//! ## The ten canonical agent operations//!//! | MCP tool | XERJ endpoint | Real capability |//! |------------------------|---------------------------------------|-----------------|@@ -16,7 +16,11 @@//! | `xerj_vector_search` | `POST /{index}/_search` (`knn`) | kNN over a `dense_vector` field (HNSW-served unfiltered, exact filtered) |//! | `xerj_hybrid_search` | `POST /{index}/_search` (`hybrid`) | RRF or linear fusion of sub-queries |//! | `xerj_memory_store` | `POST /_memory/{ns}` | namespaced agent-memory write |-//! | `xerj_memory_recall` | `POST /_memory/{ns}/_recall` | recall by meaning (BM25 / semantic / vector) |+//! | `xerj_memory_recall` | `POST /_memory/{ns}/_recall` | recall by meaning (BM25 / semantic / vector), optionally graph-coupled |+//! | `xerj_brain_ego` | `GET /_graph/{brain}/ego` | one node's evidence-backed, bi-temporal link neighborhood |+//! | `xerj_brain_link` | `POST /_graph/{brain}/link` | assert a link (idempotent, deterministic edge_id) |+//! | `xerj_brain_unlink` | `DELETE /_graph/{brain}/link/{id}` | retire a link (soft-invalidate; never deletes) |+//! | `xerj_brain_overview` | `GET /_graph/{brain}/overview` | whole-brain orientation: counts, hubs, detectors, timeline |//!//! ## Honesty notes (must match the engine, never oversell)//!@@ -27,6 +31,11 @@//! * `hybrid` supports `fusion: "rrf"` and `"linear"` only. `"learned"` is//! forwarded verbatim and the engine rejects it loudly; the schema therefore//! advertises only `rrf`/`linear`.+//! * The `xerj_brain_*` tools work a **link index, not a graph database**:+//! deterministic lexical detection plus explicit assertions — no neural+//! understanding, no query language, no unbounded traversal. Every brain+//! tool description says both things, and engine refusals (e.g. the hops+//! cap) are passed back to the agent verbatim.//! * This proxy adds **no** capabilities of its own — whatever the engine//! returns (including errors) is passed straight back to the agent.//!@@ -191,7 +200,12 @@ fn initialize_result(msg: &Value) -> Value {field (embedding is server-side, no key), xerj_vector_search for \kNN over a dense_vector field, xerj_hybrid_search to fuse \lexical + vector results (rrf|linear), and xerj_memory_store / \- xerj_memory_recall for durable agent memory recalled by meaning.",+ xerj_memory_recall for durable agent memory recalled by meaning. \+ The xerj_brain_* tools work a second brain (a bi-temporal, \+ evidence-carrying link index built by `xerj brain <folder>`): \+ xerj_brain_overview to orient, then xerj_brain_ego for one node's \+ neighborhood, then xerj_brain_link / xerj_brain_unlink to assert \+ or retire links — retired links stay replayable via as_of.",})}@@ -219,7 +233,14 @@ fn tool_text(text: impl Into<String>, is_error: bool) -> Value {// ─────────────────────────────── Tools ─────────────────────────────────────-/// The six tool specifications advertised via `tools/list`. Input schemas are+/// The two honesty sentences every `xerj_brain_*` tool description MUST carry+/// (SECOND_BRAIN_SPEC honesty rule; mirrors the crate's kNN precedent above).+const BRAIN_HONESTY: &str = " Honesty: links come from deterministic lexical \+ detection plus explicit assertions — not neural understanding. XERJ is a \+ search engine with a graph-shaped index over its own documents, not a \+ graph database.";++/// The ten tool specifications advertised via `tools/list`. Input schemas are/// plain JSON Schema; every property maps onto a field the engine accepts.fn tool_specs() -> Value {json!([@@ -375,10 +396,134 @@ fn tool_specs() -> Value {"recency_weight": {"type": "number","description": "Blend relevance with recency in [0,1]; 0=pure relevance, 1=pure recency."+ },+ "graph": {+ "type": "object",+ "description":+ "Optional graph coupling with the namespace's second-brain links \+ (edges index `.xerj-memory-{ns}-edges`). `restrict` recalls only \+ within graph reach of the seeds; `blend` lets graph proximity pull \+ related memories up. Degrades gracefully (`no_edges_index:true` in \+ the response) when the namespace has no links yet.",+ "properties": {+ "mode": { "type": "string", "enum": ["restrict", "blend"] },+ "seeds": {+ "type": "array",+ "items": { "type": "string" },+ "description": "Seed node ids. Required for restrict; blend defaults to the top-5 base-recall hits."+ },+ "hops": { "type": "integer", "description": "1 (default) or 2." },+ "types": {+ "type": "array",+ "items": { "type": "string" },+ "description": "Link-type allowlist."+ },+ "weight": { "type": "number", "description": "Blend weight in [0,1], default 0.3. Ignored for restrict." },+ "as_of": { "description": "Bi-temporal cut: epoch-ms number or RFC3339 string (default now)." }+ },+ "required": ["mode"]}},"required": ["namespace"]}+ },+ {+ "name": "xerj_brain_ego",+ "description": format!(+ "One node's annotated neighborhood in a second brain: every link with \+ direction, hop, type, confidence, and the evidence that justified it, \+ plus node previews and a `not_shown` accounting of anything clipped. \+ Links are believed-since/retired (`valid_at`/`invalid_at`); pass `as_of` \+ to replay a past moment. Hops cap at 2 by design — iterate from \+ `reachable`. Proxies GET /_graph/{{brain}}/ego.{BRAIN_HONESTY}"),+ "inputSchema": {+ "type": "object",+ "properties": {+ "brain": { "type": "string", "description": "Brain name (as created by `xerj brain <folder>` or the link tool)." },+ "node": { "type": "string", "description": "Node id to expand from (a document _id in the brain's nodes index)." },+ "hops": { "type": "integer", "enum": [1, 2], "description": "Neighborhood radius: 1 (default) or 2. Capped at 2 by design." },+ "direction": { "type": "string", "enum": ["out", "in", "both"], "description": "Which links to follow (default both)." },+ "types": {+ "type": "array",+ "items": { "type": "string" },+ "description": "Link-type allowlist (e.g. [\"references\"]); absent = all types."+ },+ "limit": { "type": "integer", "description": "Max returned links; engine clamps to 1..=1000 (default 100). Clipped remainder is reported in `not_shown`." },+ "as_of": { "description": "Bi-temporal cut: epoch-ms number or RFC3339 string. Replays what the brain believed at that moment (default now)." },+ "include_expired": { "type": "boolean", "description": "Also return retired links; their `invalid_at` says when they stopped being believed (default false)." },+ "include_nodes": { "type": "boolean", "description": "Hydrate node title/preview summaries (default true here — the proxy sets it explicitly)." }+ },+ "required": ["brain", "node"]+ }+ },+ {+ "name": "xerj_brain_link",+ "description": format!(+ "Assert a link between two nodes in a second brain, with the evidence \+ that justifies it — pass the exact quote you are relying on. \+ edge_id is deterministic over (src, type, dst, valid_at), so \+ re-asserting the same fact is idempotent (`created:true/false`); \+ `valid_at` defaults to server-now, so pass it explicitly when a \+ retry must dedupe. Proxies POST \+ /_graph/{{brain}}/link.{BRAIN_HONESTY}"),+ "inputSchema": {+ "type": "object",+ "properties": {+ "brain": { "type": "string", "description": "Brain name. The edges index is created lazily on first link." },+ "src": { "type": "string", "description": "Source node id (a document _id)." },+ "dst": { "type": "string", "description": "Destination node id. Must differ from `src` (no self-links)." },+ "type": { "type": "string", "description": "Link type, e.g. references, mentions, contradicts." },+ "evidence": {+ "type": "object",+ "description": "Why this link exists: {quote, source, offset} — the exact text relied on, where it came from, and its byte offset. Optional, but a link without evidence is shown as asserted-not-detected.",+ "properties": {+ "quote": { "type": "string" },+ "source": { "type": "string" },+ "offset": { "type": "integer" }+ }+ },+ "weight": { "type": "number", "description": "Link strength in [0,1], default 1.0." },+ "confidence": { "type": "number", "description": "Assertion confidence in [0,1], default 1.0." },+ "valid_at": { "description": "When the fact became true: epoch-ms number or RFC3339 string (default now). Part of the deterministic edge_id." }+ },+ "required": ["brain", "src", "dst", "type"]+ }+ },+ {+ "name": "xerj_brain_unlink",+ "description": format!(+ "Retire a link in a second brain — never delete: the edge is \+ soft-invalidated and stays queryable at past `as_of` moments. Get \+ `edge_id` from ego responses. Idempotent: retiring an already-retired \+ link reports `already_invalid_at`. Proxies \+ DELETE /_graph/{{brain}}/link/{{edge_id}}.{BRAIN_HONESTY}"),+ "inputSchema": {+ "type": "object",+ "properties": {+ "brain": { "type": "string", "description": "Brain name." },+ "edge_id": { "type": "string", "description": "The link's id, from an ego response." },+ "invalid_at": { "description": "When the fact stopped being true: epoch-ms number or RFC3339 string (default: server now). The server separately records when it learned this (`expired_at`)." }+ },+ "required": ["brain", "edge_id"]+ }+ },+ {+ "name": "xerj_brain_overview",+ "description": format!(+ "Orientation call for a second brain: does it exist, live vs retired \+ link counts, hub nodes (most linked), what taught it (per-detector \+ counts), link types, and a created-over-time timeline. Cheap bounded \+ aggregations, always current; pass `as_of` to summarize a past moment. \+ Proxies GET /_graph/{{brain}}/overview.{BRAIN_HONESTY}"),+ "inputSchema": {+ "type": "object",+ "properties": {+ "brain": { "type": "string", "description": "Brain name." },+ "as_of": { "description": "Bi-temporal cut: epoch-ms number or RFC3339 string (default now)." },+ "top": { "type": "integer", "description": "Size of every top-N list; engine clamps to 1..=50 (default 10)." }+ },+ "required": ["brain"]+ }}])}@@ -407,15 +552,32 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {"xerj_hybrid_search" => build_hybrid(&args),"xerj_memory_store" => build_memory_store(&args),"xerj_memory_recall" => build_memory_recall(&args),+ "xerj_brain_ego" => build_brain_ego(&args),+ "xerj_brain_link" => build_brain_link(&args),+ "xerj_brain_unlink" => build_brain_unlink(&args),+ "xerj_brain_overview" => build_brain_overview(&args),other => return tool_text(format!("unknown tool: {other}"), true),};match result {- Ok((path, body)) => engine_post(ctx, &path, body).await,+ Ok((method, path, body)) => engine_request(ctx, method, &path, body).await,Err(msg) => tool_text(msg, true),}}+/// HTTP method of a proxied engine request. The engine's graph surface is not+/// POST-only (ego/overview are GET, unlink is DELETE), so builders carry it.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum Method {+ Get,+ Post,+ Delete,+}++/// What every `build_*` function returns: the request to proxy, fully formed+/// (query params already encoded into the path) — pure and unit-testable.+type BuiltRequest = (Method, String, Option<Value>);+// ── Argument helpers ────────────────────────────────────────────────────────fn req_str<'a>(args: &'a Value, key: &str) -> Result<&'a str, String> {@@ -434,9 +596,69 @@ fn copy_opt(args: &Value, body: &mut serde_json::Map<String, Value>, key: &str)}}-// ── Per-tool request builders → (path, body) ────────────────────────────────+/// Percent-encode one path segment or query value (RFC 3986 unreserved set+/// passes through; everything else, including `/`, is `%XX`-encoded). Node+/// ids are arbitrary document ids, so this is not optional.+fn enc(s: &str) -> String {+ let mut out = String::with_capacity(s.len());+ for b in s.bytes() {+ match b {+ b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'.' | b'_' | b'~' => {+ out.push(b as char);+ }+ _ => out.push_str(&format!("%{b:02X}")),+ }+ }+ out+}-fn build_search(args: &Value) -> Result<(String, Value), String> {+/// Join `(key, value)` pairs into a query string, percent-encoding values+/// (keys are fixed identifiers). Empty input → empty string (no `?`).+fn qs(pairs: &[(&str, String)]) -> String {+ if pairs.is_empty() {+ return String::new();+ }+ let joined: Vec<String> = pairs+ .iter()+ .map(|(k, v)| format!("{k}={}", enc(v)))+ .collect();+ format!("?{}", joined.join("&"))+}++/// Read an optional bi-temporal instant argument: an epoch-ms number or an+/// RFC3339 string, stringified for the query line (the engine parses both).+fn opt_instant(args: &Value, key: &str) -> Result<Option<String>, String> {+ match args.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(Value::Number(n)) => Ok(Some(n.to_string())),+ Some(Value::String(s)) if !s.is_empty() => Ok(Some(s.clone())),+ Some(_) => Err(format!(+ "`{key}` must be an epoch-ms number or an RFC3339 string"+ )),+ }+}++/// Read an optional argument of one exact JSON type. Present-but-mistyped is a+/// builder error, never a silent drop: an agent that sends+/// `include_expired: "true"` must be told, not handed the default and left+/// believing it saw history. (Same policy `opt_instant` already applies.)+fn opt_typed<'a, T>(+ args: &'a Value,+ key: &str,+ want: &str,+ read: impl Fn(&'a Value) -> Option<T>,+) -> Result<Option<T>, String> {+ match args.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(v) => read(v)+ .map(Some)+ .ok_or_else(|| format!("`{key}` must be {want}")),+ }+}++// ── Per-tool request builders → (method, path, body) ────────────────────────++fn build_search(args: &Value) -> Result<BuiltRequest, String> {let index = req_str(args, "index")?;let mut body = serde_json::Map::new();match args.get("query") {@@ -450,10 +672,14 @@ fn build_search(args: &Value) -> Result<(String, Value), String> {for k in ["size", "from", "sort", "_source"] {copy_opt(args, &mut body, k);}- Ok((format!("/{index}/_search"), Value::Object(body)))+ Ok((+ Method::Post,+ format!("/{index}/_search"),+ Some(Value::Object(body)),+ ))}-fn build_semantic(args: &Value) -> Result<(String, Value), String> {+fn build_semantic(args: &Value) -> Result<BuiltRequest, String> {let index = req_str(args, "index")?;let field = req_str(args, "field")?;let query = req_str(args, "query")?;@@ -467,10 +693,10 @@ fn build_semantic(args: &Value) -> Result<(String, Value), String> {}// Also cap `size` at k so the response isn't padded past the requested set.let body = json!({ "query": { "semantic": semantic }, "size": k });- Ok((format!("/{index}/_search"), body))+ Ok((Method::Post, format!("/{index}/_search"), Some(body)))}-fn build_vector(args: &Value) -> Result<(String, Value), String> {+fn build_vector(args: &Value) -> Result<BuiltRequest, String> {let index = req_str(args, "index")?;let field = req_str(args, "field")?;let vector = args@@ -492,10 +718,10 @@ fn build_vector(args: &Value) -> Result<(String, Value), String> {}}let body = json!({ "knn": Value::Object(knn), "size": k });- Ok((format!("/{index}/_search"), body))+ Ok((Method::Post, format!("/{index}/_search"), Some(body)))}-fn build_hybrid(args: &Value) -> Result<(String, Value), String> {+fn build_hybrid(args: &Value) -> Result<BuiltRequest, String> {let index = req_str(args, "index")?;let queries = args.get("queries")@@ -513,10 +739,14 @@ fn build_hybrid(args: &Value) -> Result<(String, Value), String> {let mut body = serde_json::Map::new();body.insert("query".into(), json!({ "hybrid": Value::Object(hybrid) }));copy_opt(args, &mut body, "size");- Ok((format!("/{index}/_search"), Value::Object(body)))+ Ok((+ Method::Post,+ format!("/{index}/_search"),+ Some(Value::Object(body)),+ ))}-fn build_memory_store(args: &Value) -> Result<(String, Value), String> {+fn build_memory_store(args: &Value) -> Result<BuiltRequest, String> {let namespace = req_str(args, "namespace")?;// `text` is required unless a raw `vector` is supplied; the engine enforces// "non-empty text or a vector", so mirror that leniently here.@@ -541,10 +771,14 @@ fn build_memory_store(args: &Value) -> Result<(String, Value), String> {] {copy_opt(args, &mut body, k);}- Ok((format!("/_memory/{namespace}"), Value::Object(body)))+ Ok((+ Method::Post,+ format!("/_memory/{namespace}"),+ Some(Value::Object(body)),+ ))}-fn build_memory_recall(args: &Value) -> Result<(String, Value), String> {+fn build_memory_recall(args: &Value) -> Result<BuiltRequest, String> {let namespace = req_str(args, "namespace")?;let mut body = serde_json::Map::new();for k in [@@ -554,20 +788,128 @@ fn build_memory_recall(args: &Value) -> Result<(String, Value), String> {"k","filter","recency_weight",+ "graph",] {copy_opt(args, &mut body, k);}- Ok((format!("/_memory/{namespace}/_recall"), Value::Object(body)))+ Ok((+ Method::Post,+ format!("/_memory/{namespace}/_recall"),+ Some(Value::Object(body)),+ ))+}++// ── Second-brain builders (graph surface: GET/POST/DELETE) ──────────────────++fn build_brain_ego(args: &Value) -> Result<BuiltRequest, String> {+ let brain = req_str(args, "brain")?;+ let node = req_str(args, "node")?;++ let mut q: Vec<(&str, String)> = vec![("node", node.to_string())];+ if let Some(h) = opt_typed(args, "hops", "an integer (1 or 2)", Value::as_u64)? {+ q.push(("hops", h.to_string()));+ }+ if let Some(d) = opt_typed(args, "direction", "a string (out|in|both)", Value::as_str)? {+ q.push(("direction", d.to_string()));+ }+ // The HTTP surface takes a comma-separated list; the schema takes an+ // array (better for models). Join here.+ if let Some(types) = opt_typed(args, "types", "an array of strings", |v| {+ v.as_array()+ .and_then(|a| a.iter().map(Value::as_str).collect::<Option<Vec<_>>>())+ })? {+ if !types.is_empty() {+ q.push(("types", types.join(",")));+ }+ }+ if let Some(l) = opt_typed(args, "limit", "an integer", Value::as_u64)? {+ q.push(("limit", l.to_string()));+ }+ if let Some(a) = opt_instant(args, "as_of")? {+ q.push(("as_of", a));+ }+ if let Some(b) = opt_typed(args, "include_expired", "a boolean", Value::as_bool)? {+ q.push(("include_expired", b.to_string()));+ }+ // Deliberate default flip vs the HTTP surface: agents nearly always want+ // node previews, so the proxy sets include_nodes explicitly, default true.+ let include_nodes =+ opt_typed(args, "include_nodes", "a boolean", Value::as_bool)?.unwrap_or(true);+ q.push(("include_nodes", include_nodes.to_string()));++ Ok((+ Method::Get,+ format!("/_graph/{}/ego{}", enc(brain), qs(&q)),+ None,+ ))+}++fn build_brain_link(args: &Value) -> Result<BuiltRequest, String> {+ let brain = req_str(args, "brain")?;+ let src = req_str(args, "src")?;+ let dst = req_str(args, "dst")?;+ let edge_type = req_str(args, "type")?;++ let mut body = serde_json::Map::new();+ body.insert("src".into(), json!(src));+ body.insert("dst".into(), json!(dst));+ body.insert("type".into(), json!(edge_type));+ for k in ["evidence", "weight", "confidence", "valid_at"] {+ copy_opt(args, &mut body, k);+ }+ Ok((+ Method::Post,+ format!("/_graph/{}/link", enc(brain)),+ Some(Value::Object(body)),+ ))+}++fn build_brain_unlink(args: &Value) -> Result<BuiltRequest, String> {+ let brain = req_str(args, "brain")?;+ let edge_id = req_str(args, "edge_id")?;+ let mut q: Vec<(&str, String)> = Vec::new();+ if let Some(at) = opt_instant(args, "invalid_at")? {+ q.push(("invalid_at", at));+ }+ Ok((+ Method::Delete,+ format!("/_graph/{}/link/{}{}", enc(brain), enc(edge_id), qs(&q)),+ None,+ ))+}++fn build_brain_overview(args: &Value) -> Result<BuiltRequest, String> {+ let brain = req_str(args, "brain")?;+ let mut q: Vec<(&str, String)> = Vec::new();+ if let Some(a) = opt_instant(args, "as_of")? {+ q.push(("as_of", a));+ }+ if let Some(t) = opt_typed(args, "top", "an integer", Value::as_u64)? {+ q.push(("top", t.to_string()));+ }+ Ok((+ Method::Get,+ format!("/_graph/{}/overview{}", enc(brain), qs(&q)),+ None,+ ))}// ── Engine transport ─────────────────────────────────────────────────────────-/// POST `body` to `path` on the configured XERJ base URL and wrap the response-/// as an MCP tool result. Non-2xx responses (and transport errors) come back as-/// `isError:true` so the agent sees exactly what the engine said.-async fn engine_post(ctx: &Ctx, path: &str, body: Value) -> Value {+/// Send one request to the configured XERJ base URL and wrap the response as+/// an MCP tool result. Non-2xx responses (and transport errors) come back as+/// `isError:true` with the engine's text verbatim, so the agent sees exactly+/// what the engine said (including its "not a graph database" refusals).+async fn engine_request(ctx: &Ctx, method: Method, path: &str, body: Option<Value>) -> Value {let url = format!("{}{}", ctx.base_url, path);- let mut req = ctx.client.post(&url).json(&body);+ let mut req = match method {+ Method::Get => ctx.client.get(&url),+ Method::Post => ctx.client.post(&url),+ Method::Delete => ctx.client.delete(&url),+ };+ if let Some(body) = &body {+ req = req.json(body);+ }if let Some(auth) = &ctx.auth {req = req.header("Authorization", auth);}@@ -592,21 +934,28 @@ async fn engine_post(ctx: &Ctx, path: &str, body: Value) -> Value {mod tests {use super::*;+ /// Unwrap a builder result into `(method, path, body)` with a non-optional+ /// body, for the POST builders that always carry one.+ fn built(r: Result<BuiltRequest, String>) -> (Method, String, Value) {+ let (m, p, b) = r.unwrap();+ (m, p, b.expect("builder should carry a body"))+ }+#[test]fn search_defaults_to_match_all() {- let (path, body) = build_search(&json!({ "index": "docs" })).unwrap();+ let (method, path, body) = built(build_search(&json!({ "index": "docs" })));+ assert_eq!(method, Method::Post);assert_eq!(path, "/docs/_search");assert_eq!(body["query"], json!({ "match_all": {} }));}#[test]fn search_passes_query_and_size() {- let (_, body) = build_search(&json!({+ let (_, _, body) = built(build_search(&json!({"index": "docs","query": { "match": { "body": "rust" } },"size": 5- }))- .unwrap();+ })));assert_eq!(body["query"]["match"]["body"], "rust");assert_eq!(body["size"], 5);}@@ -618,10 +967,10 @@ mod tests {#[test]fn semantic_builds_semantic_query_and_caps_size() {- let (path, body) = build_semantic(&json!({+ let (method, path, body) = built(build_semantic(&json!({"index": "kb", "field": "content", "query": "how to reset", "k": 3- }))- .unwrap();+ })));+ assert_eq!(method, Method::Post);assert_eq!(path, "/kb/_search");assert_eq!(body["query"]["semantic"]["field"], "content");assert_eq!(body["query"]["semantic"]["query"], "how to reset");@@ -631,10 +980,9 @@ mod tests {#[test]fn vector_builds_top_level_knn() {- let (path, body) = build_vector(&json!({+ let (_, path, body) = built(build_vector(&json!({"index": "emb", "field": "vec", "query_vector": [0.1, 0.2, 0.3], "k": 4- }))- .unwrap();+ })));assert_eq!(path, "/emb/_search");assert_eq!(body["knn"]["field"], "vec");assert_eq!(body["knn"]["query_vector"], json!([0.1, 0.2, 0.3]));@@ -649,7 +997,7 @@ mod tests {#[test]fn hybrid_wraps_queries_and_fusion() {- let (path, body) = build_hybrid(&json!({+ let (_, path, body) = built(build_hybrid(&json!({"index": "h","queries": [{ "query": { "match": { "body": "cats" } }, "weight": 1.0 },@@ -657,8 +1005,7 @@ mod tests {],"fusion": "rrf","size": 10- }))- .unwrap();+ })));assert_eq!(path, "/h/_search");assert_eq!(body["query"]["hybrid"]["fusion"], "rrf");assert_eq!(@@ -671,23 +1018,204 @@ mod tests {#[test]fn memory_store_requires_text_or_vector() {assert!(build_memory_store(&json!({ "namespace": "n" })).is_err());- let (path, body) =- build_memory_store(&json!({ "namespace": "n", "text": "remember this" })).unwrap();+ let (_, path, body) = built(build_memory_store(+ &json!({ "namespace": "n", "text": "remember this" }),+ ));assert_eq!(path, "/_memory/n");assert_eq!(body["text"], "remember this");}#[test]fn memory_recall_path_and_passthrough() {- let (path, body) = build_memory_recall(&json!({+ let (_, path, body) = built(build_memory_recall(&json!({"namespace": "n", "query": "what did I say", "semantic": true, "k": 5- }))- .unwrap();+ })));assert_eq!(path, "/_memory/n/_recall");assert_eq!(body["semantic"], true);assert_eq!(body["k"], 5);}+ #[test]+ fn memory_recall_passes_graph_coupling_verbatim() {+ let graph = json!({+ "mode": "restrict",+ "seeds": ["note-a"],+ "hops": 2,+ "as_of": 1750000000000_i64+ });+ let (_, path, body) = built(build_memory_recall(&json!({+ "namespace": "n", "query": "deploy steps", "graph": graph+ })));+ assert_eq!(path, "/_memory/n/_recall");+ assert_eq!(body["graph"], graph);+ // Absent → key absent (graph-less recall stays bit-identical).+ let (_, _, body) = built(build_memory_recall(+ &json!({ "namespace": "n", "query": "x" }),+ ));+ assert!(body.get("graph").is_none());+ }++ // ── second-brain builders ───────────────────────────────────────────────++ #[test]+ fn brain_ego_minimal_sets_include_nodes_true() {+ let (method, path, body) =+ build_brain_ego(&json!({ "brain": "notes", "node": "note-a" })).unwrap();+ assert_eq!(method, Method::Get);+ assert_eq!(path, "/_graph/notes/ego?node=note-a&include_nodes=true");+ assert!(body.is_none(), "GET must carry no body");+ }++ #[test]+ fn brain_ego_include_nodes_false_is_respected() {+ let (_, path, _) = build_brain_ego(&json!({+ "brain": "notes", "node": "note-a", "include_nodes": false+ }))+ .unwrap();+ assert_eq!(path, "/_graph/notes/ego?node=note-a&include_nodes=false");+ }++ #[test]+ fn brain_ego_encodes_all_params() {+ let (_, path, _) = build_brain_ego(&json!({+ "brain": "notes",+ "node": "dir/some note.md",+ "hops": 2,+ "direction": "out",+ "types": ["references", "mentions"],+ "limit": 50,+ "as_of": 1750000000000_i64,+ "include_expired": true+ }))+ .unwrap();+ assert_eq!(+ path,+ "/_graph/notes/ego?node=dir%2Fsome%20note.md&hops=2&direction=out\+ &types=references%2Cmentions&limit=50&as_of=1750000000000\+ &include_expired=true&include_nodes=true"+ );+ }++ #[test]+ fn brain_ego_accepts_rfc3339_as_of() {+ let (_, path, _) = build_brain_ego(&json!({+ "brain": "b", "node": "n", "as_of": "2026-05-02T00:00:00Z"+ }))+ .unwrap();+ assert!(path.contains("as_of=2026-05-02T00%3A00%3A00Z"));+ // Wrong type is a builder-side error, not a silent drop.+ assert!(build_brain_ego(&json!({ "brain": "b", "node": "n", "as_of": true })).is_err());+ }++ #[test]+ fn brain_ego_requires_brain_and_node() {+ assert!(build_brain_ego(&json!({ "node": "n" })).is_err());+ assert!(build_brain_ego(&json!({ "brain": "b" })).is_err());+ }++ #[test]+ fn brain_link_builds_post_body() {+ let (method, path, body) = built(build_brain_link(&json!({+ "brain": "notes",+ "src": "note-a",+ "dst": "note-b",+ "type": "references",+ "evidence": { "quote": "see [[note-b]]", "source": "note-a", "offset": 120 },+ "confidence": 0.8,+ "valid_at": 1750000000000_i64+ })));+ assert_eq!(method, Method::Post);+ assert_eq!(path, "/_graph/notes/link");+ assert_eq!(body["src"], "note-a");+ assert_eq!(body["dst"], "note-b");+ assert_eq!(body["type"], "references");+ assert_eq!(body["evidence"]["quote"], "see [[note-b]]");+ assert_eq!(body["confidence"], 0.8);+ assert_eq!(body["valid_at"], 1750000000000_i64);+ }++ #[test]+ fn brain_link_requires_src_dst_type() {+ let base = json!({ "brain": "b", "src": "a", "dst": "c", "type": "t" });+ assert!(build_brain_link(&base).is_ok());+ for missing in ["brain", "src", "dst", "type"] {+ let mut args = base.clone();+ args.as_object_mut().unwrap().remove(missing);+ assert!(build_brain_link(&args).is_err(), "should require {missing}");+ }+ }++ #[test]+ fn brain_unlink_builds_delete_with_invalid_at() {+ let (method, path, body) = build_brain_unlink(&json!({+ "brain": "notes", "edge_id": "bef814a75bd3d914c3e561f610154304"+ }))+ .unwrap();+ assert_eq!(method, Method::Delete);+ assert_eq!(path, "/_graph/notes/link/bef814a75bd3d914c3e561f610154304");+ assert!(body.is_none());++ let (_, path, _) = build_brain_unlink(&json!({+ "brain": "notes",+ "edge_id": "bef814a75bd3d914c3e561f610154304",+ "invalid_at": 1750000000000_i64+ }))+ .unwrap();+ assert_eq!(+ path,+ "/_graph/notes/link/bef814a75bd3d914c3e561f610154304?invalid_at=1750000000000"+ );+ }++ #[test]+ fn brain_unlink_requires_edge_id() {+ assert!(build_brain_unlink(&json!({ "brain": "b" })).is_err());+ }++ #[test]+ fn brain_overview_builds_get_with_params() {+ let (method, path, body) = build_brain_overview(&json!({ "brain": "notes" })).unwrap();+ assert_eq!(method, Method::Get);+ assert_eq!(path, "/_graph/notes/overview");+ assert!(body.is_none());++ let (_, path, _) = build_brain_overview(&json!({+ "brain": "notes", "as_of": "2026-05-02T00:00:00Z", "top": 5+ }))+ .unwrap();+ assert_eq!(+ path,+ "/_graph/notes/overview?as_of=2026-05-02T00%3A00%3A00Z&top=5"+ );+ }++ #[test]+ fn brain_builders_reject_mistyped_args_never_drop() {+ // Each of these was once silently dropped (defaults applied) — an+ // honesty bug: the agent believed it filtered/replayed but did not.+ let cases = [+ json!({ "brain": "b", "node": "n", "hops": "2" }),+ json!({ "brain": "b", "node": "n", "direction": 5 }),+ json!({ "brain": "b", "node": "n", "types": "references" }),+ json!({ "brain": "b", "node": "n", "types": [1] }),+ json!({ "brain": "b", "node": "n", "limit": "50" }),+ json!({ "brain": "b", "node": "n", "include_expired": "true" }),+ json!({ "brain": "b", "node": "n", "include_nodes": "false" }),+ ];+ for args in &cases {+ assert!(build_brain_ego(args).is_err(), "should reject {args}");+ }+ assert!(build_brain_overview(&json!({ "brain": "b", "top": "5" })).is_err());+ // Explicit null still means "absent", not an error.+ assert!(build_brain_ego(&json!({ "brain": "b", "node": "n", "hops": null })).is_ok());+ }++ #[test]+ fn enc_percent_encodes_reserved_bytes() {+ assert_eq!(enc("plain-id_0.9~x"), "plain-id_0.9~x");+ assert_eq!(enc("a/b c?d&e=f"), "a%2Fb%20c%3Fd%26e%3Df");+ }+#[test]fn initialize_echoes_client_protocol_version() {let msg = json!({ "params": { "protocolVersion": "2024-11-05" } });@@ -697,7 +1225,7 @@ mod tests {}#[test]- fn tools_list_has_all_six() {+ fn tools_list_has_all_ten() {let specs = tool_specs();let names: Vec<&str> = specs.as_array()@@ -705,7 +1233,7 @@ mod tests {.iter().map(|t| t["name"].as_str().unwrap()).collect();- assert_eq!(names.len(), 6);+ assert_eq!(names.len(), 10);for n in ["xerj_search","xerj_semantic_search",@@ -713,8 +1241,39 @@ mod tests {"xerj_hybrid_search","xerj_memory_store","xerj_memory_recall",+ "xerj_brain_ego",+ "xerj_brain_link",+ "xerj_brain_unlink",+ "xerj_brain_overview",] {assert!(names.contains(&n), "missing tool {n}");}+ // Deliberately NOT exposed: raw traversal, bulk dumps, operator actions.+ for forbidden in ["graph_expand", "autoindex", "dump"] {+ assert!(+ !names.iter().any(|n| n.contains(forbidden)),+ "must not expose {forbidden}"+ );+ }+ }++ #[test]+ fn every_brain_tool_carries_both_honesty_strings() {+ let specs = tool_specs();+ for tool in specs.as_array().unwrap() {+ let name = tool["name"].as_str().unwrap();+ if !name.starts_with("xerj_brain_") {+ continue;+ }+ let desc = tool["description"].as_str().unwrap();+ assert!(+ desc.contains("not neural understanding"),+ "{name} missing the determinism honesty string"+ );+ assert!(+ desc.contains("not a graph database"),+ "{name} missing the not-a-graph-database honesty string"+ );+ }}}diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rsindex 04d19dc1f..68f2a25d9 100644--- a/engine/crates/xerj-server/src/brain.rs+++ b/engine/crates/xerj-server/src/brain.rs@@ -121,7 +121,7 @@ fn parse(args: Vec<String>) -> Result<Option<BrainCfg>, String> {"--data-dir" => {data_dir = Some(PathBuf::from(it.next().ok_or("--data-dir needs a value")?))}- "--api-key" => api_key = it.next(),+ "--api-key" => api_key = Some(it.next().ok_or("--api-key needs a value")?),"--fresh" => fresh = true,"--no-open" => no_open = true,"--help" | "-h" => return Ok(None),@@ -370,6 +370,7 @@ fn run(cfg: BrainCfg) -> Result<i32> {t0.elapsed().as_secs_f64());println!(" → {console_url}");+ println!(" agents: XERJ_URL={} xerj-mcp", cfg.url);if let Some(link) = &setup_link {println!(" one-time passkey setup (open once, valid 30 min):");println!(" → {link}");