AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(design): the second-brain implementation contract
xerj-org6e38b2ea00a5
diff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.mdnew file mode 100644index 000000000..61a6deb89--- /dev/null+++ b/docs/design/SECOND_BRAIN_SPEC.md@@ -0,0 +1,1205 @@+# XERJ SECOND BRAIN — IMPLEMENTATION CONTRACT (v1)++Status: FROZEN. Every stream codes against this file. If reality contradicts this+file, fix the file first (one integrator-approved edit), then the code — never+silently diverge. Field names, route paths, JSON shapes, constants, and the+fixture in §8 are normative.++Contract version string: `"xerj-second-brain/1"` (returned by `GET /_graph/{brain}/overview` as `contract`).++---++## 0. Scope, invariants, stream ownership++### What this is+A relationship layer over documents that already exist. Nodes are ordinary XERJ+documents (memory entries, autoindex section docs — any doc, anywhere). Edges+are ordinary documents in one reserved index per brain. Traversal is a bounded,+batched, columnar expansion — not a graph query language.++### Hard invariants (violating any of these is a defect, not a trade-off)+1. **Edges are ordinary documents.** No storage-format change. Edges ride the+ existing WAL → memtable → segment → doc-values path. Boot replay, flush,+ merge, snapshots all work on edges because they are just docs.+2. **Bi-temporal, soft-delete only.** An edge is never physically removed by the+ API. Invalidation sets `invalid_at`/`expired_at`. "What did it believe last+ Tuesday" stays answerable via `as_of`.+3. **Hop path never hydrates `_source`.** Expansion reads doc-values columns+ only. Evidence hydration (ego endpoint) happens AFTER traversal on the+ bounded result set via an `ids` query, never during it.+4. **One column scan per segment per hop for the WHOLE frontier** (two scans+ for direction `both`: one over `src`, one over `dst`). O(1) hash-set+ membership per row. Cost is frontier-size-independent.+5. **Hops hard-capped at 2.** `hops > 2` is a 400 whose message tells the+ caller to iterate: expansion composes — feed `reachable` back in as the next+ frontier.+6. **NOT a graph database.** No Cypher, no shortest-path, no PageRank, no+ variable-depth traversal. Docs and error messages say this explicitly (exact+ wording in §4.6).+7. **Detectors are deterministic and versioned.** No LLM extraction. Same+ corpus in → byte-identical edge set out (given unchanged mtimes). Every+ derived edge carries evidence and a `detector` tag like `wikilink@1`.+8. **Honesty in-band.** Every response carries a `not_shown` object counting+ what was withheld: clipped frontiers, clipped edges, dangling node refs,+ segments skipped, expired edges excluded, agg tails. The default embedder is+ LEXICAL feature-hashing — no surface may imply neural semantics.++### Stream ownership (files each stream may touch — no overlaps)+| Stream | Files |+|---|---|+| ENGINE | `engine/crates/xerj-engine/src/index.rs` (ONE new contiguous section, see §3.1) + `engine/crates/xerj-engine/tests/` |+| API | `engine/crates/xerj-api/src/graph_api.rs` (new), `engine/crates/xerj-api/src/lib.rs` (one `pub mod graph_api;` line), `engine/crates/xerj-api/src/router.rs` (one route block), `engine/crates/xerj-api/src/memory_api.rs` (recall extension §5 + `-edges` suffix guard §1), `engine/crates/xerj-api/Cargo.toml` (add `xxhash-rust.workspace = true`) |+| AUTOINDEX | `engine/crates/xerj-autoindex/src/detect/` (new module), `engine/crates/xerj-autoindex/src/lib.rs` (hooks), `cli.rs` (flags), `Cargo.toml` if needed (xxhash-rust already present) |+| UX | `xerj-ux/src/dashboards/second-brain.js` (new), `xerj-ux/src/dashboards/registry.js`, `engine/crates/xerj-console-api/src/seed.rs` |++Build scoped only, per repo directive:+```+cargo build --release -j 32 -p xerj-engine # engine stream+cargo build --release -j 32 -p xerj-api # api stream (pulls engine)+cargo build --release -j 32 -p xerj-autoindex # autoindex stream+cargo build --release -j 32 -p xerj-server # integrator smoke+```+NEVER workspace-wide, NEVER `cargo clean`. rustfmt + `clippy -D warnings` must pass.++---++## 1. Naming, validation, index layout++- A **brain** is a namespace string. Validation: identical rules to+ `memory_api::validate_namespace` (lowercase start, `[a-z0-9._-]`, ≤200 chars,+ no `..`) **plus one new rule: the name must not end with `-edges`**.+- Edges index for brain `B`: **`.xerj-memory-{B}-edges`**. The leading dot+ passes `IndexName::validate` (xerj-common/src/types.rs:96 explicitly allows+ it) and is unreachable through user index APIs.+- Default nodes index for brain `B`: **`.xerj-memory-{B}`** (the memory+ namespace of the same name). Autoindex brains override this via the meta doc+ (§6.6); every read endpoint also accepts an explicit `nodes_index` request+ parameter (any index expression the ES-compat search path accepts: name,+ comma-list, wildcard).+- **Collision guard (touches memory_api):** `memory_api::validate_namespace`+ gains the same `-edges` suffix rejection with reason+ `"namespace suffix '-edges' is reserved for graph edge indices"`. Without+ this, `POST /_memory/kb-edges` would write memories into brain `kb`'s edge+ index. This is a deliberate breaking change; record it in the release notes.+- Node ids (`src`/`dst`) are **opaque strings** — document `_id`s in whatever+ index the caller's nodes live in. The graph layer never interprets them on+ the hop path.++---++## 2. EDGE SCHEMA++### 2.1 Exact index mapping (normative JSON)++Created lazily by `graph_api::link` on first write, and by autoindex+`ensure_index` — **both must send exactly this body** (single shared constant+per crate; keep byte-identical):++```json+{+ "mappings": {+ "properties": {+ "edge_id": { "type": "keyword" },+ "src": { "type": "keyword" },+ "dst": { "type": "keyword" },+ "type": { "type": "keyword" },+ "weight": { "type": "float" },+ "valid_at": { "type": "date", "format": "epoch_millis" },+ "invalid_at": { "type": "date", "format": "epoch_millis" },+ "created_at": { "type": "date", "format": "epoch_millis" },+ "expired_at": { "type": "date", "format": "epoch_millis" },+ "detector": { "type": "keyword" },+ "confidence": { "type": "float" },+ "schema_version": { "type": "integer" },+ "src_file": { "type": "keyword" },+ "evidence": {+ "properties": {+ "quote": { "type": "text" },+ "source": { "type": "keyword" },+ "offset": { "type": "long" }+ }+ }+ }+ }+}+```++### 2.2 Stored-document type discipline (LOAD-BEARING)++`build_doc_value_columns` (index.rs:12820) types columns from the **raw JSON+values**, not the mapping, and a field that is an array in ANY doc of a segment+ships NO column for that segment. Therefore every edge writer MUST emit:++- `edge_id`, `src`, `dst`, `type`, `detector`, `src_file` — JSON **string**, never array. → `KeywordColumn`.+- `weight`, `confidence` — JSON **number**. → `NumericColumn` (f64 bits).+- `valid_at`, `invalid_at`, `created_at`, `expired_at` — JSON **number, epoch+ milliseconds UTC**. Never ISO strings (a string would make it a+ KeywordColumn and kill the as-of compare). The explicit+ `"format": "epoch_millis"` in the mapping exempts these fields from the+ engine's default-date-format ingest validation (see `date_format_exclusions`,+ index.rs:1586).+- `invalid_at` / `expired_at` / `src_file`: **omit the key entirely when+ unset** — never JSON `null`. Omission lands the row in the column's+ `null_bitmap`, which is exactly the "still valid" signal the hop reads.+- `evidence` — JSON object. Objects produce no doc-values column (by design);+ evidence is read only on hydration paths, never on the hop.+- `schema_version` — the integer `1` for every edge this contract produces.++Every edge document's **`_id` is exactly its `edge_id` field value**. This is+what lets the hop path return edge ids straight from the `edge_id` keyword+column with zero `_source` access, and makes re-writes idempotent overwrites.++### 2.3 edge_id computation (normative)++Hash fn: `xxhash_rust::xxh3::xxh3_128` — already a workspace dependency+(`engine/Cargo.toml:61`, `xxhash-rust = { version = "0.8", features = ["xxh3"] }`),+already used for identity in `xerj-autoindex/src/ids.rs`. The API stream adds+`xxhash-rust.workspace = true` to `engine/crates/xerj-api/Cargo.toml`.++Canonical input, mirroring the `ids::doc_id` framing convention (`ax1` → `xg1`):++```rust+/// edge_id = xxh3_128("xg1\0" src "\0" type "\0" dst "\0" decimal(valid_at_ms)),+/// rendered as 32 lowercase hex chars ({:032x}).+pub fn edge_id(src: &str, edge_type: &str, dst: &str, valid_at_ms: i64) -> String {+ use xxhash_rust::xxh3::xxh3_128;+ let mut input = Vec::with_capacity(16 + src.len() + edge_type.len() + dst.len());+ input.extend_from_slice(b"xg1\x00");+ input.extend_from_slice(src.as_bytes());+ input.push(0);+ input.extend_from_slice(edge_type.as_bytes());+ input.push(0);+ input.extend_from_slice(dst.as_bytes());+ input.push(0);+ input.extend_from_slice(valid_at_ms.to_string().as_bytes());+ format!("{:032x}", xxh3_128(&input))+}+```++Both xerj-api (`graph_api.rs`) and xerj-autoindex (`detect/mod.rs`) implement+this function **byte-identically** (each crate has its own copy — do NOT+introduce a shared crate for one function; a unit test in each crate pins the+fixture vector below). Pin test vector:+`edge_id("note-alpha", "wikilink", "note-beta", 1753600000000) == "bef814a75bd3d914c3e561f610154304"`.++Consequences: same (src, type, dst, valid_at) → same `_id` → bulk `index`+actions overwrite instead of duplicating; re-running a detector over an+unchanged corpus converges (kill -9 safe, same property as `ids::doc_id`).+Re-asserting the same fact at a DIFFERENT `valid_at` creates a distinct edge —+that is the bi-temporal design, not a bug.++### 2.4 Exact stored edge document (normative example)++The wikilink edge `note-alpha → note-beta` from the fixture (§8), as stored:++```json+{+ "edge_id": "bef814a75bd3d914c3e561f610154304",+ "src": "note-alpha",+ "dst": "note-beta",+ "type": "wikilink",+ "weight": 1.0,+ "valid_at": 1753600000000,+ "created_at": 1753600000000,+ "detector": "wikilink@1",+ "confidence": 0.95,+ "schema_version": 1,+ "src_file": "alpha.md",+ "evidence": {+ "quote": "Alpha is the hub note. It links to [[beta]] and [[gamma]].",+ "source": "alpha.md",+ "offset": 35+ }+}+```+(`invalid_at`/`expired_at` absent = edge is live.) After soft invalidation the+same doc is re-indexed under the same `_id` with two added fields, e.g.+`"invalid_at": 1753700000000, "expired_at": 1753700000000`.++### 2.5 Brain meta document++One reserved doc per edges index, `_id` = `__xerj-brain-meta`:++```json+{+ "meta_version": 1,+ "brain": "notes",+ "nodes_index": ".xerj-memory-notes",+ "created_at": 1753600000000+}+```++It has **no** `src`/`dst`/`type`/`edge_id` fields, so it is invisible to the+hop (null `src` → null_bitmap skip) and to every count in §4 (all counts filter+on `exists src`). `graph_api::link` writes it (nodes_index = default) when it+creates the index; autoindex writes it with `nodes_index` = its dataset index+pattern (§6.6). Writers use `create`-if-absent semantics (never clobber an+existing meta doc).++---++## 3. ENGINE API (xerj-engine)++### 3.1 Placement++All engine-side graph code lives in **`engine/crates/xerj-engine/src/index.rs`**+in ONE new contiguous section appended after the existing cache-helper impl+(after the `impl Index` block that ends the prefilter helpers), delimited:++```rust+// ─────────────────────────────────────────────────────────────────────────────+// Graph expansion (second brain) — batched, columnar, bounded+// ─────────────────────────────────────────────────────────────────────────────+```++Rationale (do not re-litigate): the hop needs `dv_columns_for`,+`ghost_positions_for`, `self.store`, `self.memtable`, `self.data_dir` — all+private to the `index` module. A sibling `graph.rs` would force visibility+churn on ~6 fields/helpers; a single in-file section costs nothing and only the+ENGINE stream touches index.rs, so no cross-stream conflicts.++### 3.2 Public types and constants (exact signatures)++```rust+/// Hard ceiling on expansion depth. Depth beyond this composes client-side:+/// feed `reachable` back in as the next call's frontier.+pub const GRAPH_MAX_HOPS: u8 = 2;+/// Frontier ids admitted per hop; excess is dropped and counted+/// (`stats.frontier_clipped`). 4096 ords is a ~32KB hash set — cheap — while+/// bounding the per-segment FST-probe phase.+pub const GRAPH_MAX_FRONTIER: usize = 4096;+/// Result-edge ceiling per request; collection stops (scan continues to the+/// end of the current segment, then aborts remaining segments) and+/// `stats.edges_clipped` reports the overflow.+pub const GRAPH_MAX_RESULT_EDGES: usize = 100_000;++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum GraphDirection {+ /// Follow src → dst (scan the `src` column).+ Out,+ /// Follow dst → src (scan the `dst` column).+ In,+ /// Both directions (two column scans per segment). The default everywhere.+ Both,+}++#[derive(Debug, Clone)]+pub struct GraphExpandRequest {+ /// Node ids to expand from. Deduped internally; clipped to+ /// GRAPH_MAX_FRONTIER (clip counted, never silent).+ pub frontier: Vec<String>,+ /// 1 or 2. 0 or >2 → XerjError::InvalidQuery (see §3.5).+ pub hops: u8,+ pub direction: GraphDirection,+ /// Edge-type allowlist; None = all types.+ pub types: Option<Vec<String>>,+ /// Bi-temporal cut: an edge is visible iff+ /// valid_at <= as_of_ms AND (invalid_at absent OR invalid_at > as_of_ms).+ pub as_of_ms: i64,+ /// true = ignore invalid_at (return soft-deleted edges too; used by+ /// ego?include_expired=true). Excluded edges are counted either way.+ pub include_expired: bool,+ /// Per-request result cap, clamped to 1..=GRAPH_MAX_RESULT_EDGES.+ pub max_result_edges: usize,+}++/// One edge as read ENTIRELY from doc-values columns — no `_source` access.+/// `weight` is f64 because numeric columns store f64 bits (index.rs:12869);+/// timestamps are exact for |v| < 2^53 (epoch-ms until year ~287396).+#[derive(Debug, Clone)]+pub struct GraphEdgeLite {+ pub edge_id: String,+ pub src: String,+ pub dst: String,+ pub edge_type: String,+ pub weight: f64,+ pub valid_at_ms: i64,+ pub invalid_at_ms: Option<i64>,+ /// Which hop discovered this edge (1 or 2).+ pub hop: u8,+}++/// In-band honesty: what the expansion did NOT show, and what it cost.+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct GraphExpandStats {+ pub frontier_clipped: u64,+ pub edges_clipped: u64,+ /// Edges skipped by the bi-temporal cut (valid_at in the future of as_of,+ /// or invalidated at/before as_of) — NOT counting include_expired=true.+ pub expired_excluded: u64,+ /// Edges skipped by the `types` allowlist.+ pub type_filtered: u64,+ pub segments_scanned: u64,+ /// Segments skipped because they lack a dv sidecar or a `src`/`dst`+ /// keyword column (legacy segment / meta-only segment / poisoned column).+ /// The hop NEVER falls back to hydration — skipped means reported.+ pub segments_without_columns: u64,+ pub memtable_docs_scanned: u64,+}++#[derive(Debug, Clone)]+pub struct GraphExpandResult {+ /// Deduped by edge_id, sorted (hop asc, weight desc, edge_id asc) — the+ /// stable order every response surface and the fixture assert on.+ pub edges: Vec<GraphEdgeLite>,+ /// Frontier ids (that were admitted) first, in input order, then newly+ /// reached ids in first-discovery order following the sorted edge list.+ /// Deduped.+ pub reachable: Vec<String>,+ pub stats: GraphExpandStats,+}++impl Index {+ /// Batched graph expansion over an edges index (§2 schema).+ ///+ /// Sync fn (no await points): all reads are memtable walks + doc-values+ /// column access — the same blocking pattern the columnar agg/search fast+ /// paths already use from async handlers.+ pub fn graph_expand(&self, req: &GraphExpandRequest) -> xerj_common::Result<GraphExpandResult>;+}+```++### 3.3 Batching contract (the algorithm, normative)++Per hop `h` in `1..=req.hops`, with `frontier_h` (hop 1 = admitted request+frontier; hop 2 = new endpoints discovered in hop 1, minus already-visited,+clipped to `GRAPH_MAX_FRONTIER` with the clip counted):++1. `snapshot = self.store_snapshot()` once per request (index.rs:12357);+ `segments_dir = self.data_dir.join("segments")` (the derivation every+ columnar path uses, e.g. index.rs:9821).+2. `any_ghosts = self.store.version_map.ghost_events() > 0` once per request+ (xerj-storage/src/version_map.rs:523).+3. **Per segment** in `snapshot.segments`:+ a. `let Some(cols) = self.dv_columns_for(&segments_dir, &meta.id)` — the+ cached sidecar read (index.rs:14150). None → `segments_without_columns += 1`, skip.+ b. Pull needed columns from the `DocValueMap` (BTreeMap, index.rs:44):+ `src`, `dst` (per direction), `edge_id`, `type`, `weight`, `valid_at`,+ `invalid_at` — via `Column::Keyword(k)` / `Column::Numeric(n)` matches.+ A missing/`Numeric`-typed `src` (resp. `dst` for In/Both) →+ `segments_without_columns += 1`, skip segment. Missing `type`/`weight`/+ `valid_at`/`edge_id` columns → same skip (schema-corrupt segment; honesty+ over guessing). A missing `invalid_at` column is NORMAL (segment where no+ edge was ever invalidated) and means "no edge here is invalidated".+ c. Build the frontier ordinal set: for each id in `frontier_h`,+ `k_src.ord_for_term(id)` (O(log T) FST probe, doc_values.rs:443); collect+ `HashSet<u32>`. Empty set → skip segment without scanning.+ d. **THE scan** — one pass for the whole frontier:+ `for (pos, &ord) in k_src.ords.iter().enumerate()`:+ skip if `k_src.null_bitmap.contains(pos as u32)`;+ admit iff `frontier_ords.contains(&ord)` (O(1)).+ Direction `Both` runs this twice (once over `src` ords, once over `dst`+ ords) with the same admit logic; an edge admitted by both scans is+ emitted once (dedupe on edge_id).+ e. **Per admitted position** (random access, still no `_source`):+ - Liveness: if `any_ghosts`, fetch once per segment+ `self.ghost_positions_for(&meta.id, meta.doc_count)` (index.rs:16873,+ `Option<Arc<Vec<u64>>>` word-bitmap; None → conservatively+ `segments_without_columns += 1`, skip segment); skip positions whose+ bit is set (deleted OR superseded rows — Lucene ghost semantics, same+ source of truth `scored_columnar` uses at index.rs:14570).+ - Type filter: `k_type.ord_for(pos)` → `term_for_ord` compare against the+ allowlist (resolve allowlist→ord-set per segment first; absent term =+ nothing matches). Filtered → `type_filtered += 1`.+ - Bi-temporal cut: `n_valid.get(pos)` (doc_values.rs:147; recover+ `f64::from_bits(v as u64) as i64`); `n_invalid.get(pos)` if the column+ exists. Apply the §3.2 visibility rule → else `expired_excluded += 1`.+ - Emit `GraphEdgeLite` with `edge_id/src/dst/type` read via+ `ord_for(pos)` + `term_for_ord` on their keyword columns, `hop = h`.+ - Stop collecting at `max_result_edges` (count the rest of this+ segment's admits + skip remaining segments into `edges_clipped`).+4. **Memtable** (unflushed edges — correctness, not speed): one bounded walk+ per hop over every shard:+ `for s in 0..self.memtable.shard_count() { self.memtable.with_shard(s, |m| m.all_docs_with_sources_arc()) }`+ (memtable.rs:858). Per (id, source): `memtable_docs_scanned += 1`; skip if+ `self.store.version_map.get(&id)` says deleted; admit if+ `source["src"]` (and/or `"dst"` per direction) is a JSON string in the+ frontier hash set; apply the same type/bi-temporal filters from source+ fields; emit with values read from the source. The memtable is bounded by+ the flush threshold (~10k docs), so O(memtable) per hop is fine and is+ reported in-band.+5. After the final hop: global dedupe by edge_id (a memtable row wins over any+ segment row with the same edge_id — it is the newer version), sort+ (hop asc, weight desc, edge_id asc), build `reachable`, return.++Explicitly forbidden on this path: `StoredSlices` access, `simd_json` parsing,+`hydrate_prefiltered_unsorted`, per-frontier-id queries, and any per-edge+`_source` read.++### 3.4 Engine helpers reused (verified names — cite these in doc comments)++| Helper | Where | Role in the hop |+|---|---|---|+| `Index::dv_columns_for(&self, segments_dir: &Path, segment_id: &str) -> Option<Resident<DocValueMap>>` | index.rs:14150 | cached per-segment doc-values sidecar (`Resident<T> = Arc<CacheResident<T>>`, index.rs:43; `DocValueMap = BTreeMap<String, Column>`, index.rs:44) |+| `KeywordColumn::{ord_for_term, ord_for, term_for_ord}`, `.ords`, `.null_bitmap`, `.per_ord_count` | xerj-storage/src/doc_values.rs:343–460 | frontier→ordinal resolution, THE scan, dst/type/edge_id random access |+| `NumericColumn::get(&self, doc_id: u32) -> Option<i64>` (+ `f64::from_bits(v as u64)`) | doc_values.rs:147 | weight / valid_at / invalid_at reads (numeric columns store f64 BITS — never use the i64 raw) |+| `Index::ghost_positions_for(&self, seg_id: &str, expect_docs: u64) -> Option<Arc<Vec<u64>>>` | index.rs:16873 | liveness (deleted + superseded rows) without hydration |+| `VersionMap::ghost_events() -> u64` | xerj-storage/src/version_map.rs:523 | request-level "any ghosts?" gate (same gate as `scored_columnar`, index.rs:14521) |+| `Index::store_snapshot() -> IndexSnapshot` | index.rs:12357 | immutable segment list (`segments: Vec<SegmentMeta>`, segment.rs:324) |+| `ShardedFtsMemtable::{shard_count, with_shard}` + `FtsMemtable::all_docs_with_sources_arc()` | xerj-engine/src/memtable.rs:681/695/858 | unflushed-edge coverage |+| `Index::row_seqs_for(&self, seg_id, expect_docs)` | index.rs:16244 | NOT needed if ghost bitmaps are used (they subsume the supersede check); listed so nobody reinvents it |++### 3.5 Errors++`graph_expand` returns `Err(XerjError::InvalidQuery { reason })`+(xerj-common/src/error.rs:38) for: `hops == 0` or `hops > GRAPH_MAX_HOPS`+(reason MUST contain the §4.6 not-a-graph-database sentence), empty `frontier`+after dedup. Everything else is a normal result with honest `stats` — a brain+with no edges expands to zero edges, zero error.++### 3.6 Engine tests (fixture-bound)++`engine/crates/xerj-engine/tests/graph_expand.rs`: build the §8 fixture edges+via `index_document` on a `.xerj-memory-notes-edges` index; assert (a) exact+§8.4 expansion pre-flush (memtable path), (b) byte-identical result post+`flush()` (columnar path), (c) post-soft-invalidate `as_of` time travel:+expanding at `as_of < invalid_at` still sees the edge, at `>= invalid_at` does+not and `expired_excluded == 1`, (d) hop-2 + caps: `max_result_edges = 2`+yields `edges_clipped > 0`.++---++## 4. HTTP API (xerj-api)++New module `engine/crates/xerj-api/src/graph_api.rs`, modeled line-for-line on+the `memory_api.rs` adapter discipline: thin handlers that compose+`es_compat::{create_index, index_doc, get_doc, search}` ++`Engine::get_index(...)` + `Index::graph_expand(...)`; reuse the local+`drain_json` / `OptionalJson` / error-shape patterns (copy the private fns —+do not make memory_api's pub).++Route registration in `router.rs`, appended directly under the existing+"Agent-Memory API" block (~line 753):++```rust+// ── Second-Brain Graph API ─────────────────────────────────────────────────+// Edges are ordinary documents in reserved `.xerj-memory-{brain}-edges`+// indices; traversal is a bounded columnar expansion (Index::graph_expand),+// NOT a graph query language. See graph_api.rs.+.route("/_graph/:brain/link", post(graph_api::link))+.route("/_graph/:brain/link/:edge_id", delete(graph_api::unlink))+.route("/_graph/:brain/ego", get(graph_api::ego))+.route("/_graph/:brain/overview", get(graph_api::overview))+```++Shared conventions:+- All timestamps in request/response JSON are **epoch milliseconds (numbers)**.+ Inputs additionally accept RFC3339 strings (parsed with `chrono`); outputs+ are always numbers.+- Error body shape (identical to memory_api but typed `graph_error`):+ `{"error": {"type": "graph_error", "reason": "<msg>"}, "status": <n>}`.+- Auth: process-wide API-key middleware only, same caveat block as+ memory_api's module docs (no per-brain authz; copy the warning).+- Unknown brain on read endpoints → 404. `link` creates lazily (like+ `ensure_backing_index`).++### 4.1 POST /_graph/{brain}/link — assert an edge++Request (`deny_unknown_fields`):+```json+{+ "src": "note-alpha", // required, non-empty string+ "dst": "note-beta", // required, non-empty, != src+ "type": "wikilink", // required, non-empty+ "weight": 1.0, // optional, default 1.0, finite, clamped [0,1]+ "valid_at": 1753600000000, // optional, default = server now+ "created_at": 1753600000000, // optional, default = server now (fixtures/imports)+ "confidence": 0.95, // optional, default 1.0, clamped [0,1]+ "detector": "manual@1", // optional, default "manual@1"+ "evidence": { // optional+ "quote": "…", "source": "alpha.md", "offset": 35+ }+}+```++Handler: validate brain (§1) and body (400 on violation; self-edge reason:+`"self-edges are not allowed (src == dst)"`); ensure edges index exists with+the §2.1 mapping + §2.5 meta doc; compute `edge_id` (§2.3); build the §2.4+document; `es_compat::index_doc` with `_id = edge_id`.++Response — 201 when the ES-compat result is `created`, 200 when `updated`:+```json+{+ "brain": "notes",+ "edge_id": "bef814a75bd3d914c3e561f610154304",+ "created": true,+ "edge": { /* the exact stored document, §2.4 */ }+}+```++### 4.2 DELETE /_graph/{brain}/link/{edge_id} — soft invalidate++Query params: `invalid_at` (optional epoch-ms/RFC3339; default = server now).+NEVER removes the document.++Handler: `es_compat::get_doc` on the edges index → 404+(`"edge '<id>' does not exist in brain '<brain>'"`) if absent. If the doc+already has `invalid_at` → 200:+```json+{ "brain": "notes", "edge_id": "…", "invalidated": false, "already_invalid_at": 1753650000000 }+```+Else re-index the full source under the same `_id` with+`invalid_at = <param>` and `expired_at = <server now>` added → 200:+```json+{ "brain": "notes", "edge_id": "…", "invalidated": true, "invalid_at": 1753700000000, "expired_at": 1753700000000 }+```+`invalid_at` = when the fact stopped being true (caller-suppliable);+`expired_at` = when the system recorded that (always server now). This is the+bi-temporal pair; both are set on the same call.++### 4.3 GET /_graph/{brain}/ego — neighborhood of one node++Query params:+| param | type | default | notes |+|---|---|---|---|+| `node` | string | REQUIRED | 400 if missing/empty |+| `hops` | int | 1 | 1 or 2; >2 → 400 with §4.6 wording |+| `direction` | `out`\|`in`\|`both` | `both` | |+| `types` | comma-list | all | |+| `limit` | int | 100 | max returned edges, clamped 1..=1000 |+| `as_of` | epoch-ms/RFC3339 | now | |+| `include_expired` | bool | false | |+| `include_nodes` | bool | false | hydrate node summaries |+| `nodes_index` | string | meta doc's `nodes_index`, else `.xerj-memory-{brain}` | |+| `include_evidence` | bool | true | hydrate evidence for RETURNED edges |++Handler: brain/index existence check (404) → `graph_expand` with+`frontier = [node]`, `max_result_edges = limit` → post-traversal hydration:+- `include_evidence`: ONE `es_compat::search` on the edges index with+ `{"query": {"ids": {"values": [<returned edge_ids>]}}, "size": <n>}`+ (bounded ≤1000; rides the existing `build_ids_prefilter_cached` fast path,+ index.rs:16282, for free) → attach `evidence`, `created_at`, `detector`,+ `confidence`, `expired_at` to each edge.+- `include_nodes`: ONE `es_compat::search` on `nodes_index` with an `ids`+ query for all reachable ids → summaries. Ids that resolve nowhere are+ **dangling** (edge kept, honesty counted, ids listed up to 50).++Response (fixture-exact instance in §8.5):+```json+{+ "brain": "notes",+ "contract": "xerj-second-brain/1",+ "node": "note-alpha",+ "as_of": 1753700000000,+ "hops": 1,+ "direction": "both",+ "edges": [+ {+ "edge_id": "…", "src": "…", "dst": "…", "type": "wikilink",+ "weight": 1.0, "hop": 1, "direction": "out",+ "valid_at": 1753600000000, "invalid_at": null,+ "detector": "wikilink@1", "confidence": 0.95,+ "evidence": { "quote": "…", "source": "alpha.md", "offset": 35 }+ }+ ],+ "neighbors": [+ { "id": "note-gamma", "hop": 1, "via_edge": "<edge_id of first sorted edge that reached it>" }+ ],+ "nodes": { "note-beta": { "title": "…", "preview": "…", "index": ".xerj-memory-notes" } },+ "not_shown": {+ "edges_clipped": 0,+ "frontier_clipped": 0,+ "expired_excluded": 0,+ "type_filtered": 0,+ "segments_without_columns": 0,+ "memtable_docs_scanned": 0,+ "dangling_nodes": 0,+ "dangling_ids": []+ }+}+```+Rules: `edges` in the §3.2 stable order, each annotated with+`direction: "out"|"in"` relative to the expansion (an edge admitted by both+scans reports `"out"`); `neighbors` excludes `node` itself, first-discovery+order; `edge.invalid_at` is `null` in JSON when unset (response-side null is+fine — §2.2's omit-rule binds stored docs, not responses); `nodes` present+only when `include_nodes=true` — summary = `{title, preview, index}` where+`title` = `_source.title` if string else null, `preview` = first 160 chars of+`_source.text` else `_source.body` else null, `index` = the index the doc was+found in. `not_shown` maps `GraphExpandStats` 1:1 plus the dangling fields;+`segments_scanned`/`memtable_docs_scanned` ride along as cost honesty.++### 4.4 GET /_graph/{brain}/overview — brain-level stats (dashboard feed)++Query params: `as_of` (default now), `top` (default 10, clamp 1..=50 — size of+every top-N list), `histogram_interval` (`"day"` default | `"hour"`).++Handler composes exactly three `es_compat::search` calls on the edges index+(size 0, `track_total_hits: true`):+1. totals: `{"query": {"exists": {"field": "src"}}}` → `edges.total`.+2. live slice: query = `exists src` AND live-at-as_of+ (`bool.filter[exists src, range valid_at lte as_of].must_not[range invalid_at lte as_of]`)+ with aggs `{"by_type": {"terms": {"field": "type", "size": top}}, "by_detector": {"terms": {"field": "detector", "size": top}}, "top_src": {"terms": {"field": "src", "size": top}}, "top_dst": {"terms": {"field": "dst", "size": top}}}`+ → live count, per-type/per-detector live counts, hub lists.+3. timeline: `exists src` with+ `{"created": {"date_histogram": {"field": "created_at", "calendar_interval": "<interval>"}}}`.++Response:+```json+{+ "brain": "notes",+ "contract": "xerj-second-brain/1",+ "exists": true,+ "as_of": 1753700000000,+ "nodes_index": ".xerj-memory-notes",+ "embedder": "lexical-feature-hash",+ "edges": { "total": 8, "live": 8, "invalidated": 0 },+ "types": [ { "type": "wikilink", "live": 4 }, { "type": "same_dir", "live": 4 } ],+ "detectors": [ { "detector": "wikilink@1", "live": 4 }, { "detector": "samedir@1", "live": 4 } ],+ "hubs": {+ "out": [ { "id": "note-alpha", "live_edges": 3 } ],+ "in": [ { "id": "note-gamma", "live_edges": 3 } ]+ },+ "created_over_time": [ { "t": 1753574400000, "count": 8 } ],+ "not_shown": {+ "types_not_listed": 0,+ "detectors_not_listed": 0,+ "hubs_out_not_listed": 0,+ "hubs_in_not_listed": 0+ }+}+```+`invalidated = total - live`. Each `*_not_listed` = that terms agg's+`sum_other_doc_count`. `embedder` is the honesty marker for §0-invariant 8 —+the literal string `"lexical-feature-hash"` unless the node backing store is+configured with an external embedder (then its configured id verbatim). Unknown+brain → `{"brain": "...", "contract": "...", "exists": false}` with 404 status.++### 4.5 (moved to §5 — recall integration lives in memory_api.rs)++### 4.6 Not-a-graph-database wording (normative strings)++- 400 for `hops > 2` (both engine reason and HTTP): `"hops is capped at 2:+ XERJ's second brain is a relationship layer over documents, not a graph+ database (no Cypher, no shortest-path, no variable-depth traversal). Iterate:+ expand again from this response's 'reachable' ids."`+- graph_api.rs module doc + docs pages MUST contain: `"This is not a graph+ database. There is no query language, no shortest-path, no PageRank, and no+ unbounded traversal — by design. Edges are ordinary documents; expansion is+ a bounded, batched read (≤2 hops per call)."`++---++## 5. RECALL INTEGRATION — POST /_memory/{ns}/_recall graph modes++Extends `memory_api::RecallBody` (the struct is `deny_unknown_fields`; the new+field is additive and optional — absent means behavior is bit-identical to+today):++```rust+/// Optional graph coupling for recall. The namespace's brain is the namespace+/// itself: edges live in `.xerj-memory-{ns}-edges`.+#[serde(default)]+pub graph: Option<GraphRecallOpts>,++#[derive(Debug, Deserialize)]+#[serde(deny_unknown_fields)]+pub struct GraphRecallOpts {+ /// "restrict" | "blend" (required).+ pub mode: String,+ /// Seed node ids. REQUIRED for restrict (400 if absent/empty).+ /// Optional for blend (default: ids of the top-5 base-recall hits).+ #[serde(default)]+ pub seeds: Option<Vec<String>>,+ /// 1 (default) or 2.+ #[serde(default)]+ pub hops: Option<u8>,+ /// Edge-type allowlist.+ #[serde(default)]+ pub types: Option<Vec<String>>,+ /// Blend weight in [0,1], default 0.3. Ignored for restrict.+ #[serde(default)]+ pub weight: Option<f32>,+ /// Bi-temporal cut, default now. Epoch-ms number or RFC3339 string.+ #[serde(default)]+ pub as_of: Option<serde_json::Value>,+}+```++Both modes call `Index::graph_expand` on the edges index with+`direction: Both`. Constant: `MAX_RESTRICT_IDS: usize = 10_000`.++**restrict** — "recall only within reach of these seeds":+1. Expand seeds (hops, types, as_of) → `reachable` (includes seeds).+2. Clip `reachable` to `MAX_RESTRICT_IDS` (count clipped).+3. Run the EXISTING recall exactly as today, with the reachable set folded into+ the filter: `filter_out = {"bool": {"filter": [<caller filter if any>, {"ids": {"values": [<reachable>]}}]}}`.+ (The `ids` query is already supported by the parser; nothing new in+ xerj-query.)+4. Empty reachable set → normal empty-hits response (plus the `graph` object),+ not an error.++**blend** — "let graph proximity pull related memories up":+1. If `seeds` absent: run base recall first with `fetch = max(k*4, 50)` (the+ existing recency-blend over-fetch convention, memory_api.rs:486), seeds =+ top-5 hit ids. If present: same over-fetch, seeds as given.+2. Expand seeds → per-node proximity, computed during expansion:+ `proximity(seed) = 1.0`;+ reached at hop h via edge e: `proximity = max(proximity, 0.5^h * clamp(e.weight, 0, 1))`+ (max over all paths; deterministic).+3. Re-rank the fetched candidates:+ `blended = (1 - w) * norm_score + w * proximity`, where `norm_score` is the+ same min-max normalization `blend_recency` uses (memory_api.rs:606) and+ nodes absent from the proximity map get 0. Truncate to k. Hits keep their+ original relevance `score`; each gains `"graph_proximity": <f64>`.+4. Composition with `recency_weight`: apply the graph blend FIRST, then the+ existing recency blend over the graph-blended ordering (each re-rank is+ order+truncate only, so they compose; document this in the handler).++Response gains a top-level `graph` object (both modes):+```json+"graph": {+ "mode": "restrict",+ "seeds": ["note-alpha"],+ "hops": 1,+ "as_of": 1753700000000,+ "reachable": 4,+ "not_shown": {+ "frontier_clipped": 0, "edges_clipped": 0, "expired_excluded": 0,+ "type_filtered": 0, "segments_without_columns": 0,+ "reachable_clipped": 0+ }+}+```++Errors (400, memory_error shape — this endpoint keeps its existing error type):+unknown `mode`; restrict without seeds+(`"graph.mode 'restrict' requires non-empty graph.seeds"`); `hops` 0 or >2+(§4.6 wording); non-existent edges index → NOT an error: recall proceeds+ungated with `"graph": {"mode": …, "reachable": 0, "not_shown": {…, "no_edges_index": true}}`+(an agent must be able to opt into graph recall before its first link exists).++---++## 6. AUTOINDEX CONTRACT (xerj-autoindex)++### 6.1 CLI surface++`xerj autoindex <folder>` grows:+- `--brain <name>` — brain name; default `sanitize_slug(<root folder basename>)`+ (dataset.rs:82). Validation as §1 (reject `-edges` suffix).+- `--no-graph` — disable edge detection entirely.+- Graph detection is **ON by default** (a behavior change: creates+ `.xerj-memory-{brain}-edges`; release notes + `--no-graph` escape hatch).+- `autoindex map` / `status` gain one summary line each (edges written /+ edge count via `Es::count` on the edges index with an `exists src` query).++### 6.2 Module layout++```+engine/crates/xerj-autoindex/src/detect/+ mod.rs // trait, ctx types, edge_id (§2.3 copy), registry, bulk builder+ wikilink.rs // [[target]] type "wikilink" w 1.0 conf 0.95+ mdlink.rs // [text](relative/path.md) type "mdlink" w 0.9 conf 0.9+ href.rs // <a href="relative-or-corpus-url">type "href" w 0.7 conf 0.85+ sequence.rs // section s_i -> s_{i+1}, same file type "sequence" w 0.8 conf 0.99+ samedir.rs // chain over sorted files per dir type "same_dir" w 0.3 conf 0.4+```++### 6.3 Detector trait (exact shape)++```rust+/// Edge-schema version stamped into every emitted edge (`schema_version`).+pub const EDGE_SCHEMA_VERSION: u32 = 1;++/// Corpus-wide resolution table, built once after Phase A (plan assignment):+/// every indexed file's rel path → identity + the doc id of its first section.+pub struct CorpusIndex {+ /// key: root-relative path with forward slashes (`FileEntry.rel`).+ pub files: std::collections::BTreeMap<String, CorpusFile>,+ /// lowercase file-stem → rel paths bearing it (wikilink resolution;+ /// BTreeMap so ambiguity resolution below is deterministic).+ pub by_stem: std::collections::BTreeMap<String, Vec<String>>,+}+pub struct CorpusFile {+ pub rel: String,+ pub file_key: String, // ids::file_key output+ pub dataset_slug: String,+ /// ids::doc_id(dataset_slug, file_key, "s0") — the file's anchor node.+ pub anchor_doc_id: String,+ pub mtime_ms: i64,+ pub dir: String, // parent dir rel path, "" for root+}++/// One detected edge before identity/envelope assembly.+pub struct EdgeDraft {+ pub src: String, // node doc id (usually the section containing the evidence)+ pub dst: String, // node doc id (target file's anchor_doc_id)+ pub edge_type: &'static str,+ pub weight: f32,+ pub confidence: f32,+ pub valid_at_ms: i64, // §6.4 determinism rule+ pub src_file: String, // rel path that taught this edge (top-level keyword)+ pub quote: String, // evidence.quote (≤240 chars, trimmed)+ pub offset: u64, // evidence.offset (byte offset in section text; 0 for structural)+}++/// Per-section textual context. `text` is the exact section string that became+/// the node doc's `body` (post `split_sections`).+pub struct SectionCtx<'a> {+ pub corpus: &'a CorpusIndex,+ pub file: &'a CorpusFile,+ pub section_ordinal: u32,+ pub section_doc_id: &'a str,+ pub text: &'a str,+}++/// Deterministic, versioned edge detector. NO network, NO LLM, NO clock reads+/// (all time comes from ctx mtimes) — same inputs must yield the same drafts.+pub trait EdgeDetector: Sync {+ /// Versioned tag stored in `detector`, e.g. "wikilink@1". Bump the @N on+ /// ANY behavior change.+ fn tag(&self) -> &'static str;+ /// Per-section textual detection. Default: no-op.+ fn detect_text(&self, _ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {}+ /// Corpus-structural detection, called once after Phase A. Default: no-op.+ fn detect_structure(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}+}++/// The registry. Order is normative (fixture edge ordering depends on it only+/// via edge sort, so this is cosmetic — but keep it).+pub fn default_detectors() -> Vec<Box<dyn EdgeDetector>>; // wikilink, mdlink, href, sequence, samedir+```++### 6.4 Determinism + edge identity rules++- `valid_at_ms` = **mtime of `src_file`** (from the walk metadata), truncated+ to ms. Re-running over an unchanged corpus reproduces identical `valid_at` →+ identical `edge_id` (§2.3) → bulk `index` overwrites; the run converges+ exactly like `ids::doc_id` does for node docs (ids.rs module docs).+- `created_at` = run wall-clock at bulk-assembly time (the ONE non-deterministic+ field; it does not participate in `edge_id`).+- Self-edges (src == dst) are dropped by the assembler, silently — with a+ per-run counter surfaced in the summary (`edges_self_dropped`).++### 6.5 Per-detector emission rules (normative)++- **wikilink@1**: scan section text for `[[target]]` / `[[target|alias]]`+ (target = text before `|`, trimmed). Resolve: exact rel-path match first+ (with or without extension), else `by_stem[target.to_lowercase()]`; if+ multiple candidates, pick the lexicographically smallest rel (deterministic;+ count the ambiguity in the run summary as `edges_ambiguous`). Unresolvable →+ no edge, count `edges_unresolved`. src = section doc id; dst = target's+ `anchor_doc_id`; quote = the full trimmed line containing the link (≤240+ chars); offset = byte offset of the `[[` within the section text.+- **mdlink@1**: `[text](url)` where url has no scheme and no leading `//`;+ strip fragment/query; resolve relative to the containing file's dir, then as+ root-relative; must resolve to a corpus file. Same evidence rules (offset of+ the `[`).+- **href@1**: `<a href="…">` (html-extracted files only, case-insensitive+ attr); same scheme-less resolution as mdlink. Offset = offset of the `href`+ attribute value start in the section text (best effort; 0 when the extractor+ lost byte positions).+- **sequence@1** (structural per file, emitted from `detect_text` on section+ i>0 to avoid a second pass): for each multi-section file, edge+ `section s_{i-1} doc id → section s_i doc id`, quote =+ `"section {i-1} precedes section {i} of {rel}"`, offset 0.+- **samedir@1** (structural): per directory with ≥2 indexed files, sort files+ by rel; emit a CHAIN — one edge per consecutive pair+ (`left.anchor_doc_id → right.anchor_doc_id`). A chain, not a clique: cliques+ are O(n²) and would drown real signal; chain keeps directory cohesion+ reachable within the 2-hop cap for small dirs at O(n). quote =+ `"{left.rel} and {right.rel} share directory {dir}"` (dir `"."` for root),+ offset 0, `src_file` = left.rel, `valid_at_ms` = left file's mtime.++### 6.6 Pipeline hooks (where, exactly)++All hooks live in `lib.rs::run_index`:+1. **After Phase A / plan finalization** (once `FileAssignment`s + dataset+ slugs + file_keys exist, before the Phase B par-iter): build `CorpusIndex`;+ run every detector's `detect_structure`; `Es::ensure_index` the edges index+ with the §2.1 mapping; `create` the §2.5 meta doc with `nodes_index` =+ comma-list of this run's dataset index names (e.g. `"ax-notes-docs"`);+ bulk-write structural edges (§6.7).+2. **Inside the Phase B per-file sink** (lib.rs ~917, the closure that stages+ each `RawRecord`): after the node action is staged for a text-family record,+ run `detect_text` for each detector with the section text+ (`fields["body"]`), section ordinal (parsed from `rec.locator` `"s{i}"`),+ and the just-computed node `id`. Append resulting edge actions to a+ per-worker edge buffer (NOT the node staging file — different target index).+3. **Replacement invalidation** (same site as the existing+ `delete_by_query(index, {"term": {"ax_file": key}})` cleanup, lib.rs ~1000):+ when a file is being replaced, soft-invalidate its prior edges FIRST:+ `Es::search` the edges index for+ `{"query": {"bool": {"filter": [{"term": {"src_file": "<rel>"}}], "must_not": [{"exists": {"field": "invalid_at"}}]}}, "size": 1000, "_source": true}`+ (page via repeat until empty), re-index each hit's full source with+ `invalid_at = expired_at = now`. Top-level `src_file` exists precisely so+ this query rides the keyword doc-values prefilter instead of a brute+ `evidence.source` object scan.+4. **Summary line** (the final printed report): edges written / unresolved /+ ambiguous / self-dropped / invalidated, per detector tag.++### 6.7 Bulk write contract++Edge actions use the exact node-side format (lib.rs ~949):+```+{"index":{"_index":".xerj-memory-{brain}-edges","_id":"<edge_id>"}}+{<edge document, §2.4 shape>}+```+Buffered per worker, cut at the same `--bulk-mb` threshold, sent via the+existing `Es::bulk` + `record_bulk_outcome` (lib.rs:88) error discipline.+Edges for a file are sent only after that file's node bulk has been accepted+(same happens-before the staging file already provides). Cross-file dangling+windows (edge lands before its dst file's docs) are permitted — final state+converges, and readers surface dangling honestly (§4.3).++### 6.8 Autoindex tests++`detect/mod.rs` unit tests: the §2.3 pin vector; wikilink/mdlink resolution+incl. ambiguity determinism; samedir chain (not clique) on the §8 fixture+folder; sequence on a 3-section file. Integration (behind the existing+live-server test pattern): index the §8.1 folder twice, assert the edge set is+byte-identical across runs and matches §8.3's (src, dst, type, detector)+tuples with autoindex-derived node ids.++---++## 7. UX DATA CONTRACT (xerj-ux + xerj-console-api)++### 7.1 Dashboard module — `xerj-ux/src/dashboards/second-brain.js`++```js+export const secondBrain = {+ id: 'second-brain',+ name: 'Second Brain',+ render: ({ data, time }) => ({+ title: 'SECOND BRAIN',+ kicker: 'GRAPH MEMORY · EVIDENCE · TIME TRAVEL',+ meta: [time, 'XERJ-GRAPH'],+ panels: [+ { id: 'edgesLive', eyebrow: 'LIVE EDGES', cols: 3, type: 'metric', /* Num */ },+ { id: 'edgesTotal', eyebrow: 'TOTAL ASSERTED', cols: 3, type: 'metric', /* Num */ },+ { id: 'invalidated', eyebrow: 'INVALIDATED (KEPT)',cols: 3, type: 'metric', /* Num */ },+ { id: 'detectors', eyebrow: 'DETECTORS', cols: 3, type: 'metric', /* Num */ },+ { id: 'typeDist', eyebrow: 'EDGE TYPES', cols: 6, type: 'dist', /* Dist */ },+ { id: 'edgeTimeline', eyebrow: 'EDGES OVER TIME', cols: 6, type: 'series', /* Series */ },+ { id: 'ego', eyebrow: 'EGO · 1 HOP FROM SELECTED NODE', cols: 12, type: 'ego' },+ { id: 'hubs', eyebrow: 'HUBS · IN / OUT', cols: 6, type: 'topn', /* TopN */ },+ { id: 'notShown', eyebrow: 'WHAT THIS VIEW DID NOT SHOW', cols: 6, type: 'honesty' },+ ],+ }),+};+```+Import surface identical to vector-index.js (`Num` from `../ux/text.js`,+`Dist`/`Series` from `../ux/charts.js`, `TopN` from `../ux/layout.js`). Two+new panel renderers are UX-stream-owned: `ego` (node-link view of the §4.3+response — render `edges`+`neighbors` only, cap 100 drawn edges, show the cap)+and `honesty` (definition list over `not_shown`; MUST always render, showing+zeros — the panel exists to prove absence). The subtitle under `edgeTimeline`+must carry the embedder honesty string when `overview.embedder` is+`"lexical-feature-hash"`: `"recall is lexical (feature hashing) — not neural"`.++### 7.2 registry.js (exact edits)++```js+import { secondBrain } from './second-brain.js'; // with the other imports+// DEFAULT_GROUP gains:+'second-brain': 'ai',+// the for-loop array and `all` array gain `secondBrain`, appended after+// `agentMemory` (AI group order: ai-overview, rag-quality, vector-index,+// agent-memory, second-brain).+```++### 7.3 `data` shape the render receives (mock now, live later)++Per seed.rs honesty rules, panels bind mock data via builtin keys first; the+mock MUST be shaped as the live payloads so the flip is a fetch swap:++```js+data = {+ brain: 'notes',+ overview: /* EXACTLY the §4.4 response body */,+ ego: /* EXACTLY the §4.3 response body for the currently selected node */,+ metrics: { // derived by the data layer from `overview`, pre-formatted+ edgesLive: { formatted: '8', hint: 'visible at as-of' },+ edgesTotal: { formatted: '8', hint: 'incl. invalidated' },+ invalidated: { formatted: '0', hint: 'soft-deleted, kept for time travel' },+ detectors: { formatted: '2', hint: 'deterministic, versioned' },+ },+ series: { created: overview.created_over_time.map(b => b.count) },+};+```+The mock fixture data file MUST be the §8 fixture verbatim (same ids, same+counts) so UX screenshots and backend tests display the same world.++### 7.4 seed.rs (exact DashboardSpec entry)++Appended to the specs list in `default_specs()` (after the agent-memory+entry), panels mirroring §7.1 ids/kinds/titles/cols 1:1:++```rust+DashboardSpec {+ registry_id: "second-brain",+ name: "Second Brain",+ section: Some("dashboards"),+ group: Some("ai"),+ panels: vec![+ p("edgesLive", "metric", "LIVE EDGES", 3),+ p("edgesTotal", "metric", "TOTAL ASSERTED", 3),+ p("invalidated", "metric", "INVALIDATED (KEPT)", 3),+ p("detectors", "metric", "DETECTORS", 3),+ p("typeDist", "dist", "EDGE TYPES", 6),+ p("edgeTimeline", "series", "EDGES OVER TIME", 6),+ p("ego", "ego", "EGO · 1 HOP FROM SELECTED NODE", 12),+ p("hubs", "topn", "HUBS · IN / OUT", 6),+ p("notShown", "honesty", "WHAT THIS VIEW DID NOT SHOW", 6),+ ],+},+```+**Do NOT bump `SEED_REVISION`** — adding a new default id is covered by the+absent-id-creates rule on every boot (seed.rs module docs); the migration+revision is only for changing EXISTING skeletons.++---++## 8. FIXTURE — the one worked example every stream tests against++Brain: `notes`. Edges index: `.xerj-memory-notes-edges`. All fixture writes go+through `POST /_graph/notes/link` with explicit ids/timestamps so the fixture+runs without autoindex; §8.3's table is ALSO the expected autoindex output+modulo node-id derivation (autoindex integration asserts the same+(src rel, dst rel, type, detector) tuples with `ids::doc_id`-derived node ids).++### 8.1 The five notes (folder `notes/`, all mtime = 1753600000000)++| file | node id (fixture) | body |+|---|---|---|+| `alpha.md` | `note-alpha` | `Alpha is the hub note. It links to [[beta]] and [[gamma]].` |+| `beta.md` | `note-beta` | `Beta continues the thread and references [[gamma]].` |+| `gamma.md` | `note-gamma` | `Gamma is the sink note with no outgoing links.` |+| `delta.md` | `note-delta` | `Delta cites [[alpha]] as its source.` |+| `epsilon.md` | `note-epsilon` | `Epsilon stands alone.` |++### 8.2 Wiki-link occurrences (exact byte offsets in the bodies above)++alpha→beta @35, alpha→gamma @48, beta→gamma @41, delta→alpha @12.+epsilon has none — it participates only via same_dir.++### 8.3 The exact edges produced (valid_at = created_at = 1753600000000)++samedir@1 chains sorted rels: alpha.md → beta.md → delta.md → epsilon.md → gamma.md.++| # | edge_id | src | dst | type | weight | conf | detector | evidence.quote | evidence.source | offset |+|---|---|---|---|---|---|---|---|---|---|---|+| 1 | `bef814a75bd3d914c3e561f610154304` | note-alpha | note-beta | wikilink | 1.0 | 0.95 | wikilink@1 | `Alpha is the hub note. It links to [[beta]] and [[gamma]].` | alpha.md | 35 |+| 2 | `11c2d0ef216cd6e99a3907a0b53c1452` | note-alpha | note-gamma | wikilink | 1.0 | 0.95 | wikilink@1 | same line as #1 | alpha.md | 48 |+| 3 | `9bbf7d2068321ac0fa71d95e21fae2fd` | note-beta | note-gamma | wikilink | 1.0 | 0.95 | wikilink@1 | `Beta continues the thread and references [[gamma]].` | beta.md | 41 |+| 4 | `cead55986c364ad5ff6f0894daf61f77` | note-delta | note-alpha | wikilink | 1.0 | 0.95 | wikilink@1 | `Delta cites [[alpha]] as its source.` | delta.md | 12 |+| 5 | `63b747655365aa16d38188aa49966f40` | note-alpha | note-beta | same_dir | 0.3 | 0.4 | samedir@1 | `alpha.md and beta.md share directory .` | alpha.md | 0 |+| 6 | `a61e6caacb5e485baf6d45184f23ec67` | note-beta | note-delta | same_dir | 0.3 | 0.4 | samedir@1 | `beta.md and delta.md share directory .` | beta.md | 0 |+| 7 | `3efff61b58c978943e6fd2a1e4eeaee8` | note-delta | note-epsilon | same_dir | 0.3 | 0.4 | samedir@1 | `delta.md and epsilon.md share directory .` | delta.md | 0 |+| 8 | `7c07cdc441f0a3faa29be8946df3e7a4` | note-epsilon | note-gamma | same_dir | 0.3 | 0.4 | samedir@1 | `epsilon.md and gamma.md share directory .` | epsilon.md | 0 |++(Every edge_id above was computed with the §2.3 function; #5's src_file is+alpha.md, etc. — src_file = evidence.source for all fixture edges. Note edges+#1 and #5 share (src,dst) but differ in type → distinct edge_ids: parallel+edges of different types are legal and expected.)++### 8.4 Expected `graph_expand` (engine-level)++Request: `frontier=["note-alpha"], hops=1, direction=Both, types=None,+as_of_ms=1753700000000, include_expired=false, max_result_edges=1000`.++Expansion admits only edges touching the frontier: from `note-alpha`, hop 1,+direction both, that is #1 and #2 and #5 (src side) plus #4 (dst side) — and+NOT #3 (beta→gamma touches no frontier id at hop 1). Result edges in EXACTLY+this order (hop asc, weight desc, edge_id asc):++1. `11c2d0ef216cd6e99a3907a0b53c1452` alpha→gamma wikilink 1.0 hop 1+2. `bef814a75bd3d914c3e561f610154304` alpha→beta wikilink 1.0 hop 1+3. `cead55986c364ad5ff6f0894daf61f77` delta→alpha wikilink 1.0 hop 1+4. `63b747655365aa16d38188aa49966f40` alpha→beta same_dir 0.3 hop 1++`reachable = ["note-alpha", "note-gamma", "note-beta", "note-delta"]`.+`stats = { frontier_clipped: 0, edges_clipped: 0, expired_excluded: 0,+type_filtered: 0, segments_scanned: <n>, segments_without_columns: 0,+memtable_docs_scanned: <m> }` (`<n>`/`<m>` depend on flush state; every other+field is exact). The same request with `hops: 2` additionally returns, at hop+2 (frontier = gamma, beta, delta): #3 (beta→gamma), #6 (beta→delta),+#7 (delta→epsilon), #8 (epsilon→gamma) — hop-2 block sorted+`9bbf… (w 1.0), 3eff…, 7c07…, a61e…` — and reachable gains `note-epsilon`.++### 8.5 Expected ego response (HTTP-level, normative JSON)++`GET /_graph/notes/ego?node=note-alpha&hops=1&as_of=1753700000000` (defaults:+direction both, limit 100, include_evidence true, include_nodes false):++```json+{+ "brain": "notes",+ "contract": "xerj-second-brain/1",+ "node": "note-alpha",+ "as_of": 1753700000000,+ "hops": 1,+ "direction": "both",+ "edges": [+ { "edge_id": "11c2d0ef216cd6e99a3907a0b53c1452", "src": "note-alpha", "dst": "note-gamma",+ "type": "wikilink", "weight": 1.0, "hop": 1, "direction": "out",+ "valid_at": 1753600000000, "invalid_at": null, "created_at": 1753600000000,+ "detector": "wikilink@1", "confidence": 0.95,+ "evidence": { "quote": "Alpha is the hub note. It links to [[beta]] and [[gamma]].", "source": "alpha.md", "offset": 48 } },+ { "edge_id": "bef814a75bd3d914c3e561f610154304", "src": "note-alpha", "dst": "note-beta",+ "type": "wikilink", "weight": 1.0, "hop": 1, "direction": "out",+ "valid_at": 1753600000000, "invalid_at": null, "created_at": 1753600000000,+ "detector": "wikilink@1", "confidence": 0.95,+ "evidence": { "quote": "Alpha is the hub note. It links to [[beta]] and [[gamma]].", "source": "alpha.md", "offset": 35 } },+ { "edge_id": "cead55986c364ad5ff6f0894daf61f77", "src": "note-delta", "dst": "note-alpha",+ "type": "wikilink", "weight": 1.0, "hop": 1, "direction": "in",+ "valid_at": 1753600000000, "invalid_at": null, "created_at": 1753600000000,+ "detector": "wikilink@1", "confidence": 0.95,+ "evidence": { "quote": "Delta cites [[alpha]] as its source.", "source": "delta.md", "offset": 12 } },+ { "edge_id": "63b747655365aa16d38188aa49966f40", "src": "note-alpha", "dst": "note-beta",+ "type": "same_dir", "weight": 0.3, "hop": 1, "direction": "out",+ "valid_at": 1753600000000, "invalid_at": null, "created_at": 1753600000000,+ "detector": "samedir@1", "confidence": 0.4,+ "evidence": { "quote": "alpha.md and beta.md share directory .", "source": "alpha.md", "offset": 0 } }+ ],+ "neighbors": [+ { "id": "note-gamma", "hop": 1, "via_edge": "11c2d0ef216cd6e99a3907a0b53c1452" },+ { "id": "note-beta", "hop": 1, "via_edge": "bef814a75bd3d914c3e561f610154304" },+ { "id": "note-delta", "hop": 1, "via_edge": "cead55986c364ad5ff6f0894daf61f77" }+ ],+ "not_shown": {+ "edges_clipped": 0, "frontier_clipped": 0, "expired_excluded": 0,+ "type_filtered": 0, "segments_without_columns": 0,+ "memtable_docs_scanned": 0, "dangling_nodes": 0, "dangling_ids": []+ }+}+```+(`memtable_docs_scanned` is asserted as ≥0, exact-zero only post-flush; every+other byte is normative. When the fixture runs WITHOUT node docs stored,+`include_nodes=true` must return `"nodes": {}` and+`"dangling_nodes": 4, "dangling_ids": ["note-alpha","note-beta","note-delta","note-gamma"]`+— dangling honesty is part of the fixture.)++### 8.6 Fixture time-travel assertion (shared)++After `DELETE /_graph/notes/link/bef814a75bd3d914c3e561f610154304?invalid_at=1753650000000`:+- ego at `as_of=1753640000000` → all four §8.5 edges (belief last Tuesday intact);+- ego at `as_of=1753700000000` → three edges (`bef814…` gone),+ `not_shown.expired_excluded: 1`;+- ego with `include_expired=true` at the same as_of → four edges again, the+ invalidated one carrying `"invalid_at": 1753650000000, "expired_at": <set>`.+- recall restrict from seed `note-delta`, hops 1, at `as_of=1753700000000`:+ both-direction hop-1 admits #4 (src=delta → alpha), #7 (src=delta →+ epsilon), #6 (dst=delta → beta); sorted edge order is #4 (`cead…`, w 1.0),+ #7 (`3eff…`, w 0.3), #6 (`a61e…`, w 0.3), so discovery order gives+ `reachable = ["note-delta","note-alpha","note-epsilon","note-beta"]`,+ `"reachable": 4` in the response's `graph` object.++---++## 9. Verification gates (integrator)++1. `cargo test -p xerj-engine graph_` — §3.6 suite green pre- and post-flush.+2. `cargo test -p xerj-api graph_api` + memory_api recall-graph tests+ (restrict/blend/no-edges-index) green — handler tests follow the existing+ in-process `test_state()` pattern (memory_api.rs:841).+3. `cargo test -p xerj-autoindex detect` green; double-run convergence test.+4. Live smoke against a booted server (per xerj-sandbox-boot memory): the §8+ curl sequence, then `GET overview` matches §4.4 counts; ES-YAML suite is+ untouched surface but run the `search` + `bulk` suites as a regression+ canary since memory_api::validate_namespace changed.+5. rustfmt + clippy -D warnings on the three touched crates.+6. Every response of every new endpoint contains `not_shown` — grep-able+ invariant: a new endpoint without it does not merge.