AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Corpus Hub v2 — implementation plan
Status: plan. Discussion: https://github.com/xerj-org/xerj/discussions/1374
Target: xerj "<question>" answers from a hub corpus in one turn, with no clone and
no indexing on the user's machine. Packs are pre-indexed by CI, signed, pulled on
demand. The corpus is never required in the question and always named in the answer.
This document is written so that an implementer can follow it step by step without deciding anything that is already decided here. Every step names the files, the types, the tests and the condition that makes it done. Steps are ordered; do not skip ahead. Where the plan says "check", the implementer reads the named file and adapts, but does not change the contract.
Repository rules that apply to every step: scoped builds only
(cd engine && cargo build --release -j 32 -p <crate>), never cargo clean; the
ES-YAML conformance suite stays at 0 failed; commits are authored as xerj-org with
a full body; shipped docs state what is true of this binary, anchored to a version,
never a promise about a future release
(engine/crates/xerj-engine/tests/docs_do_not_promise_the_next_release.rs lists the
banned phrasings and fails the build on them). Before touching storage or snapshot code, read how the existing code
does it: xerj code xerj-storage "snapshot restore manifest" and the files named
below.
0. Where things are today (read before coding)
| Concern | Where | What exists |
|---|---|---|
| CLI dispatch | engine/crates/xerj-server/src/main.rs ~L1850 (mcp), ~L2293 (code), ~L2303 (corpus) |
each subcommand calls into a crate and returns an exit code |
| Corpus lifecycle | engine/crates/xerj-autoindex/src/xc.rs |
run_corpus_cli, run_code_cli, code_root() (~/.xerj-code or $XERJ_CODE_HOME), corpora/<name>/corpus.json, state/<name>.json, index names xc-<name>-b<build>-*, a Node trait with a FakeNode test double |
| Shared code-query contract | engine/crates/xerj-common/src/xccode/ |
Mode, CodeParams, CodeOutcome, run_code_query(&impl XcHttp, …), render::render, used identically by the CLI and the MCP tool |
| MCP tools | engine/crates/xerj-mcp/src/lib.rs |
tool_specs() ("xerj_code_search" => run_code_search (run_code_search (~L1752) |
| Snapshots | engine/crates/xerj-engine/src/engine.rs ~L4315 create_snapshot(repo_path, name, indices), ~L4414 restore_snapshot(repo_path, name, indices); HTTP /_snapshot/:repo and /_snapshot/:repo/:snapshot in engine/crates/xerj-api/src/router.rs ~L604 |
a snapshot is a directory per index (WAL + segments + schema) plus manifest.json; restore loads it into a running node without re-indexing |
| Segment bundles | engine/crates/xerj-storage/src/bundle.rs (pack, unpack, parse_footer, CATALOG_KEY = "snapshot.json"), index_store.rs (fetch_and_hydrate_segment) |
the object-storage transport; not used by v1 of this plan (see step 2, "why snapshots") |
| Signing | engine/crates/xerj-autoindex/src/harvest/sign.rs |
generate_keypair, sign_pack(dir, seed_hex), verify_pack(dir, public_hex), verify_sig_at — ed25519 over SHA256SUMS |
| Definitions | engine/crates/xerj-autoindex/src/extract/code.rs |
private struct Symbol { name, kind, line, end_line, code } per definition; the index field defs holds "kind name" lines; locator code:<line>:<name> |
| HTTP client | engine/crates/xerj-autoindex/Cargo.toml |
reqwest (blocking) is already a dependency |
| gRPC | engine/proto/xerj.proto (package xerj.v1; service XerjSearch), engine/crates/xerj-server/src/grpc.rs (XerjSearchServer, Server::builder().add_service) |
tonic 0.12 / prost 0.13 |
| Auth | engine/crates/xerj-api/src/auth.rs auth_middleware |
API-key auth on the node |
| Pack publishing CI | .github/workflows/pack-publish.yml |
builds, signs and uploads the rust-vulns pack as a dated GitHub Release |
1. New crate xerj-hub and shared types in xerj-common
Why a crate. xc.rs is already large and owns the clone-and-index path. The hub
path is different (download, verify, restore, route) and must be testable alone.
Do
- Add
engine/crates/xerj-hubtoengine/Cargo.tomlmembers. Dependencies:xerj-common,xerj-autoindex(forxc::code_rootand the state file writer),reqwest(blocking, workspace),serde,serde_json,toml,zstd,tar,sha2,ed25519-dalek(whicheverharvest/sign.rsalready uses — reuse that crate, do not add a second one),anyhow,base64. - Add
xerj-hubtoengine/crates/xerj-server/Cargo.toml. - Put every type that crosses crates into
engine/crates/xerj-common/src/hub.rs(new module,pub mod hub;inlib.rs).xerj-commonmust not depend on any other xerj crate.
Types (exact; serde with #[serde(deny_unknown_fields)] on everything that is
read from the network)
// engine/crates/xerj-common/src/hub.rs
pub const CATALOGUE_FORMAT_VERSION: u32 = 1;
pub const PACK_FORMAT_VERSION: u32 = 1;
pub const DEFAULT_HUB_URL: &str = "https://hub.xerj.org";
pub const AUTO_PULL_MAX_BYTES: u64 = 50 * 1024 * 1024; // decision 2 in the discussion
pub const DISK_BUDGET_BYTES: u64 = 4 * 1024 * 1024 * 1024;
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
#[serde(rename_all = "kebab-case")]
pub enum LicenceClass { AdaptWithAttribution, ApproachOnly, Excluded }
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct CatalogueEntry {
pub slug: String,
pub domain: String, // SEC, DB, NET, CRYPTO, WEB, API, OPS, LAW, HEALTH, AI, WRITE
pub topics: Vec<String>,
pub ask_for: String, // the registry's one-line "an agent working on … queries this for …"
pub pin_sha: String, // full 40-hex commit
pub pin_date: String, // YYYY-MM-DD
pub refresh_cadence_days: u32, // 7 for code, 1 for feeds
pub licence_spdx: String,
pub licence_class: LicenceClass,
pub pack_bytes: u64,
pub pack_url: String, // https://github.com/xerj-org/xerj/releases/download/pack-<slug>-<pin8>/<slug>-<pin8>.hubpack.tar.zst
pub pack_sha256: String,
pub sig_url: String, // same name + ".sig"
pub dependency_names: Vec<String>, // crate / npm / go module / pypi names this corpus answers for
pub quality_grade: Option<String>, // "G7 4/5" etc, or None — never invent one
pub term_sketch: Sketch,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Sketch { pub m_bits: u32, pub k_hashes: u32, pub bits_base64: String }
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Catalogue {
pub format_version: u32,
pub generated_at: String, // RFC 3339
pub corpora: Vec<CatalogueEntry>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct PackManifest { // pack.json inside the archive
pub format_version: u32,
pub slug: String,
pub pin_sha: String,
pub pin_date: String,
pub built_at: String,
pub builder_version: String, // `xerj --version` of the builder
pub index_prefix: String, // "xc-<slug>-b<build>"
pub indices: Vec<PackIndex>, // every index in the snapshot, docs index and defs index included
pub snapshot_name: String, // the name used with create_snapshot
pub licence_spdx: String,
pub licence_class: LicenceClass,
pub sources: Vec<PackSource>, // copied from corpus.json repos[]
pub defs_count: u64,
pub files_count: u64,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct PackIndex { pub name: String, pub docs: u64, pub bytes: u64 }
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct PackSource { pub repo: String, pub url: String, pub sha: String, pub licence: String }
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct DefRecord { // one line of defs.jsonl and one document of the -defs index
pub symbol: String,
pub kind: String, // as extract/code.rs names it: function, const, define, struct, …
pub lang: String,
pub file: String, // path relative to the corpus root, forward slashes
pub line: u32, // 1-based line the NAME sits on
pub text: String, // the declaration, signature only (Symbol.code), max 400 chars
pub value: Option<String>, // for constants/macros/defines: the literal, see step 3
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Answer { // what every surface returns; JSON for --json, MCP and HTTP
pub corpus: String,
pub pin_sha: String,
pub pin_date: String,
pub age_days: u32,
pub licence_spdx: String,
pub licence_class: LicenceClass,
pub answered_from: AnsweredFrom,
pub mode: AnswerMode,
pub hits: Vec<Hit>,
pub truncated: bool,
pub warnings: Vec<String>, // e.g. "pin is 23 days old; question asks for the current source"
}
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
#[serde(rename_all = "snake_case")]
pub enum AnsweredFrom { LocalPack, Hosted, LocalIndex }
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
#[serde(rename_all = "snake_case")]
pub enum AnswerMode { Def, How, Files }
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct Hit {
pub file: String, pub line: u32, pub kind: String, pub text: String,
pub value: Option<String>, pub score: f32, pub licence_warning: Option<String>,
}
Text rendering (exact, so every surface prints the same thing):
<corpus> pin <pin_date> (<age_days> day[s] old) <licence_spdx> <file>:<line>
<text>
[value: <value>] # only in def mode when value is Some
answered from: <local pack|hosted hub|local index>
[warning: …] # one line per warning
For several hits, repeat the first three lines per hit; print the answered from
line once at the end.
Tests (xerj-common/src/hub.rs #[cfg(test)]): serde round-trip for every
type; deny_unknown_fields rejects an extra field; LicenceClass kebab-case
strings; the renderer output for one hit and for three hits matches a literal
expected string (keep the expected strings in the test, not in a fixture file).
Done when cargo test -p xerj-common hub:: passes and xerj-hub builds empty.
2. Pack build and pull (xerj hub build-pack, xerj hub pull)
Why snapshots, not bundles. The engine already has create_snapshot /
restore_snapshot and their HTTP routes; a restored snapshot is queryable with no
re-indexing. The bundle/snapshot.json transport in xerj-storage would need a
new read-only HTTP StorageBackend. Use snapshots for v1. Record this decision in
the pack format (format_version: 1 means "snapshot layout").
Pack layout (<slug>-<pin8>.hubpack.tar.zst, where pin8 is the first 8 hex of pin_sha):
pack.json PackManifest
SHA256SUMS sha256 of every other file, "hash path" lines, sorted by path
snapshot/ exactly what create_snapshot wrote: manifest.json + one dir per index
defs.jsonl one DefRecord per line, sorted by (file, line)
files.jsonl {"path","lang","bytes","licence"} per indexed file, sorted by path
The signature is a separate file <slug>-<pin8>.hubpack.tar.zst.sig (ed25519 over
the archive bytes), because a signature cannot sit inside the file it signs. Use
harvest/sign.rs's key format; add sign_bytes(bytes, seed_hex) -> String and
verify_bytes(bytes, sig, public_hex) -> Result<()> next to the existing functions.
Builder: xerj hub build-pack <slug> --registry <dir> --out <dir> --url <node> --sign-key <seed-file>
- Read
<registry>/<slug>.json(the existing hub manifest;xerj corpus add --fromalready parses it — call the same function, do not write a second parser). corpus add --fromandcorpus indexinto the builder node at--url(callxc::run_corpus_cliwith the same arguments the CLI would get). After it,state/<slug>.jsonholds theindex_prefix.- Build the defs index (step 3) as
<index_prefix>-defs. PUT /_snapshot/hub-build {"type":"fs","settings":{"location":"<out>/<slug>/snapshot"}}thenPUT /_snapshot/hub-build/<slug>-<pin8>?wait_for_completion=truewithindices: "<index_prefix>-*". Checkput_snapshot_repoinengine/crates/xerj-api/src/es_compat.rs(~L30736) for where a repository location is allowed to live; if the node restricts it to a configured root, start the builder node with that root set to<out>— do not weaken the check.- Write
pack.json,defs.jsonl,files.jsonl,SHA256SUMS; tar + zstd (level 9); sign; write<slug>-<pin8>.hubpack.tar.zstand.sig. - Print one line:
pack <slug> <pin8> <bytes> <indices> <defs>.
Client: xerj hub pull <slug> [--quiet] [--url <node>]
- Load the catalogue (step 4). Unknown slug → exit 2, message names the catalogue date.
- Download
pack_urlandsig_urlto~/.xerj/hub/packs/<slug>/<pin8>.tmp/. Verifysha256(archive) == pack_sha256, then the signature against the embedded hub public key (xerj_common::hub::HUB_PUBKEY_HEX, overridable byXERJ_HUB_PUBKEYfor tests only). Any failure: delete the temp dir, exit 3, message says which check failed. Nothing is extracted before both checks pass. - Extract to
~/.xerj/hub/packs/<slug>/<pin8>/. Verify everySHA256SUMSline. Readpack.json; refuseformat_version != 1. - Register and restore:
PUT /_snapshot/hub-<slug> {"type":"fs","settings":{"location":"<that dir>/snapshot"}},POST /_snapshot/hub-<slug>/<snapshot_name>/_restore?wait_for_completion=true. If indices with the sameindex_prefixalready exist (same pin pulled before), skip restore and say so. - Write
~/.xerj-code/state/<slug>.jsonthe waycorpus indexwrites it (reuse its writer inxccode/state.rs), with the pack'sindex_prefix,pin_sha,pin_date, and a new field"source": "hubpack".xerj code <slug>must now work unchanged. - Print one line:
pulled <slug> <pin8> <bytes> → <n> indices, <defs> definitions.
Staleness rule change (in xccode/mod.rs, the 30-day refusal): a state file
with "source": "hubpack" is never refused for age; instead the answer gets a
warning when the catalogue lists a newer pin_sha. The 30-day refusal stays for
locally indexed corpora.
Node for the client. pull needs a node like corpus index does. Use the
same --url / XERJ_URL / localhost:9200 resolution as xc.rs (resolve_url).
Step 5 adds the automatic node.
Tests (engine/crates/xerj-hub/tests/pack_roundtrip.rs, start a node the way
engine/crates/xerj-api/tests/ask_endpoint.rs does):
- Fixture corpus at
engine/crates/xerj-hub/tests/fixtures/mini-c/(five C files with#defines and functions, plus aLICENSEwith MIT text), plus a registry manifest for it with a fakepin_shaandurlpointing at a local bare git repo created in the test. Build a pack with a test keypair; assert the archive lists exactly the files above; assertSHA256SUMScovers all of them. - Pull into a second, empty data directory:
xerj code mini-c "…"answers and the defs index hasdefs_countdocuments; the state file hassource: hubpack. - Flip one byte of the archive: pull exits 3 before anything is extracted (assert the pack dir does not exist).
- Pull the same pin twice: second run says "already present" and restores nothing.
format_version: 2inpack.json: refused with exit 3.
Done when the roundtrip test passes against the real binary, and a fresh
machine (HOME empty) can xerj hub pull lz4 and answer the benchmark's lz4
questions through xerj code lz4 --def without corpus add or corpus index.
3. Definitions table and def mode
Do
- In
engine/crates/xerj-autoindex/src/extract/code.rs, make the per-file symbol extraction reachable:pub fn definitions(path: &Path, lang: &str, text: &str) -> Vec<DefRecord>built from the existingSymbolvalues (name → symbol,kind,line,code → texttruncated to 400 chars). Do not change howdefs/symbolsare indexed today. valueextraction, applied totext, first match wins, all anchored at the start after optional whitespace and attributes; the captured value is trimmed and has a trailing;or,removed:- C/C++/ObjC:
^\s*#\s*define\s+(\w+)\s+(.+?)\s*(?:/[/*].*)?$→ group 2; also\b(?:static\s+)?const\s+\w[\w\s\*]*\b(\w+)\s*=\s*(.+?)\s*;→ group 2. - Rust:
^\s*(?:pub(?:\([^)]*\))?\s+)?(?:const|static)\s+(\w+)\s*:\s*[^=]+=\s*(.+?)\s*;→ group 2. - Go:
^\s*(?:const\s+)?(\w+)\s*(?:\w+\s*)?=\s*(.+?)\s*$only insideconst (blocks or lines starting withconst. - Python/Ruby/Shell:
^\s*([A-Z_][A-Z0-9_]*)\s*=\s*(.+?)\s*$. - Anything else:
None.
- C/C++/ObjC:
- Builder (step 2) writes
defs.jsonland indexes it into<index_prefix>-defswith this mapping:symbol: keywordplussymbol_text: text(same value, analyzed),kind: keyword,lang: keyword,file: keyword,line: integer,text: text,value: keyword. deflookup, one function used by CLI, MCP and HTTP:xerj_hub::def::lookup(http: &impl XcHttp, slug, symbol, k) -> Answer: term query onsymbol(exact), fallbackmatchonsymbol_text;kdefault 5; hits sorted exact-first then by file path. Zero hits is a normal answer with an emptyhitsand the textno definition named <symbol> in <corpus> (pin <date>).- Add
AnswerMode::Defto the CLI:xerj code <slug> --def <SYMBOL>and toCodeParamsasdef: Option<String>; when set,run_code_querycallslookupand renders with the step-1 renderer instead of passages.
Tests
- Unit (
extract/code.rs): the regexes on these literal lines return exactly:#define LZ4_VERSION_RELEASE 0 /* for tweaks */→"0";pub(crate) const DEFAULT_IDLE_TIMEOUT: Duration = Duration::from_secs(30);→"Duration::from_secs(30)";MDB_MAXKEYSIZE = ((MDB_DEVEL) ? 0 : 511)inside a C#define→ the full expression;batchSize = 16 << 20(Go const block) →"16 << 20";fn main() {→None. - Integration: build the fixture pack;
lookup(…, "MINI_LIMIT")returns one hit withvalue: Some("128")andlineequal to the line in the fixture file. - Benchmark acceptance: with the five benchmark packs pulled, run the 50 registered
questions (
benchmarks/corpus-tasks/tasks/<slug>.jsonon branchbench/corpus-tasks) throughxerj code <slug> --def <name> --jsonand grade with the suite'scheck.py: 50/50, every answer under 80 output tokens when rendered.
Done when the three test groups pass.
4. Catalogue (xerj hub build-catalogue, client cache, signature)
Do
- Builder:
xerj hub build-catalogue --registry <dir> --packs <dir> --out <file> --sign-key <seed-file>. OneCatalogueEntryper live corpus inbacklog/backlog-100.jsonthat has a pack in--packs; fields from the manifest, the backlog row andpack.json;dependency_namesfrom a new optional manifest field"dependency_names": [...](empty if absent — do not guess names);quality_gradeonly from the graded G7 files the site generator already reads (tools/corpus-hub-site/gen.py,parse_graded_suites), elseNone. - Term sketch: Bloom filter,
m_bits = 8 × max(1024, 10 × distinct_terms)rounded up to a multiple of 64,k_hashes = 7, double hashing with FNV-1a 64 (h1) and FNV-1a 64 of the reversed bytes (h2),h_i = h1 + i·h2. Terms: everyDefRecord.symbol, every path component offiles.jsonlwithout extension, and every lowercase word of 3+ letters inask_forandtopics. Deterministic: the same inputs produce byte-identicalbits_base64. - Signature:
catalogue.json.sig= ed25519 over the exact bytes ofcatalogue.json. The site deploy (deploy-pages.yml, corpus-hub branch) publishes both athttps://hub.xerj.org/api/v1/catalogue.jsonand….sig. - Client:
xerj_hub::catalogue::load() -> Cataloguereads~/.xerj/hub/catalogue.jsonif younger than 24 h; otherwiseGETwithIf-None-Match(store the ETag in~/.xerj/hub/catalogue.etag), verify the signature before replacing the cached copy, keep the old copy on any failure and add a warningcatalogue is <n> days old.--offlinenever fetches.
Tests (xerj-hub/src/catalogue.rs, xerj-hub/src/sketch.rs):
- Bloom:
containsis true for every inserted term; false-positive rate on 10,000 random terms is below 2%; byte-identical across two builds. - Catalogue builder on the fixture registry produces the expected entry (compare to a literal JSON in the test).
- Signature tamper: a catalogue with one changed byte is not accepted and the cached copy is kept.
- Size: the catalogue built from the real registry (515 corpora) is under 2 MB (run in CI on the corpus-hub branch as a check, not a unit test).
Done when GET https://hub.xerj.org/api/v1/catalogue.json returns a signed
catalogue that the client loads, and the tests pass.
5. The router and xerj "<question>"
CLI surface (xerj_hub::cli, dispatched from main.rs next to code / corpus):
xerj ask "<question>" [--corpus <slug>] [--def <SYMBOL> | --how | --files]
[--offline] [--no-fetch] [--hosted on|off|auto]
[--max-tokens N] [--json] [--url <node>]
xerj "<question>" # same: argv[1] is not a known subcommand, does not start with "-", and contains a space
xerj hub pull <slug> | status | sync [--install-timer] | update | rm <slug> | config hosted on|off
Router (xerj_hub::router):
pub struct Evidence { pub slug: String, pub score: f32, pub source: EvidenceSource }
pub enum EvidenceSource { ProjectDependency(String), Hosted, Sketch }
pub enum Routing { Confident(String), Ambiguous(Vec<Evidence>), NoCorpus }
pub fn route(question: &str, cwd: &Path, cat: &Catalogue, hosted: Option<&dyn HostedSearch>, cached: &[String]) -> Routing
Order and scoring, exactly:
Project dependencies: parse, if present in
cwdor any parent up to the git root:Cargo.lock([[package]]name = "…"lines),package.json(keys ofdependenciesanddevDependencies),go.mod(requirepaths, last path segment and full path),requirements.txt(name before==,>=,[),pyproject.toml([project] dependenciesnames). Any corpus whosedependency_namesintersects gets score 1.0 withProjectDependency(name).Hosted federated search (step 6) when
hostedisSome: each corpus in the response gets score0.9 × (its best hit score / the top hit score).Sketch: tokenize the question into identifiers (
[A-Za-z_][A-Za-z0-9_]{2,}) and lowercase words; for each corpus, score = (tokens present in its sketch) / (tokens total), ×0.8; tokens present in more than 25% of all sketches count half.Merge by slug taking the max score.
Confidentif top ≥ 0.6 and top ≥ 1.5 × second (or no second).Ambiguousif two or more are ≥ 0.3 and the best reaches 0.35 (a full overlap of common tokens scores 0.4 and stays a real tie). Two or more between 0.3 and 0.35 isNoCoverage: a weak overlap, not a choice. ElseNoCorpus. A question that names a project no entry contains (Phaser,setInteractive) isNoCoverageeven when another token ties several corpora.xerj askexits 1 for bothNoCorpusandNoCoverage(the latter namesxerj autoindexandxerj hub find); exit 4 is only a real ambiguity.--corpusskips routing. Corpora already cached (cached) get +0.05.A question that names a language or a technology is decided before those four steps, from the catalogue entry itself (slug, topics, a language word in
ask_for, then the rest ofask_for). An exact slug the question spells out still wins first. The sketch path above still decides when no distinctive name matches. Seexerj_hub::router.
Answer flow (xerj_hub::ask):
- If the question names a symbol (
--def, or the whole question is one identifier, or it matcheswhat is the (exact )?value of (\w+)), mode isDefwith that symbol; elseHow;--filesforcesFiles. - Route.
Ambiguous→ print the candidates (slug — ask_for, one per line) and exit 4 (the same "needs a decision" code autoindex uses); the agent re-asks with--corpus.NoCorpus→ exit 1 withno hub corpus matches; see xerj.org/llms.txt.NoCoverage→ exit 1 withno hub corpus covers this topicand the local-index hint (xerj autoindex <path>, thenxerj codeorxerj search;xerj hub findto see what the hub does have). Not exit 4. - If the pack is cached locally → answer locally (
AnsweredFrom::LocalPack). - Else if hosted is allowed → answer through the hosted node now; then, unless
--no-fetch/--offline, start the pull in the background: spawncurrent_exe() hub pull <slug> --quiet, detached, stdout/stderr to~/.xerj/hub/pulls/<slug>.log, PID to~/.xerj/hub/pulls/<slug>.pid. Policy: pull only ifpack_bytes ≤ AUTO_PULL_MAX_BYTESandlicence_class != Excluded; above the limit printrun: xerj hub pull <slug> (<MB> MB)instead; forApproachOnlyadd the warningapproach-only licence (<spdx>): read, do not copy. - Else (no cache, no hosted): if
--offline→ exit 1 saying which pack to pull; otherwise pull synchronously (same policy), then answer locally. - Disk budget: before any pull, if the packs dir exceeds
DISK_BUDGET_BYTESafter adding this pack, remove least-recently-used packs (by thelast_usedfile each pack dir gets on every answer) until it fits, never the one being pulled. - Always print the step-1 rendering;
--jsonprintsAnswer.
Hosted switch: --hosted auto (default) is on if ~/.xerj/hub/config.toml has
hosted = "on"; otherwise off when cwd is inside a git work tree, on when it is
not. Decision 1 in the discussion; the first time the switch resolves to off inside a
repo, print once: hosted answers are off in git repositories; xerj hub config hosted on.
Node (xerj_hub::node): resolve --url/XERJ_URL/localhost:9200; if the
health check fails, start current_exe() --insecure --port <free port ≥ 9300> --data-dir ~/.xerj/hub/node
detached, PID to ~/.xerj/hub/node.pid, wait for green (max 60 s), remember the port
in ~/.xerj/hub/node.port. Never start a node on a port that answers. xerj hub status
shows it; xerj hub stop stops only that PID.
A node the user started themselves needs no config for xerj hub pull to restore
into it: by default a server allows snapshot restores (never creates) from the
pack cache of the user it runs as (limits.hub_pack_restore, xerj_common::hub::hub_packs_dir,
refused if that directory is a symlink or owned by another user). A server that does
not share the cache (other user, container, other machine) refuses, and the pull
error prints the exact snapshot_repo_allowlist line to add.
Tests (xerj-hub/tests/routing.rs, xerj-hub/tests/ask_flow.rs):
- Labelled routing set
engine/crates/xerj-hub/tests/fixtures/routing.jsonl: at least 100 lines{"q":…, "deps":[…], "expected":"<slug>"}across 20 corpora of the real catalogue, written by hand from the registry'sask_forlines (the author of the test writes them; the implementer does not generate them from the sketch). Pass criteria: top-1 ≥ 90% withdeps, ≥ 80% without; everyAmbiguousresult contains the expected slug. askwith a cached pack never touches the network (assert with a hosted stub that panics on call).askwith hosted on and no cache returnsHostedand leaves a pull PID file; the pull completes; the secondaskanswersLocalPack.- Policy: a catalogue entry with
pack_bytesabove the limit is not pulled and the message names the size; anExcludedentry is never pulled;ApproachOnlyadds the warning. --offlinewith no cache exits 1 with the pull hint;--no-fetchanswers hosted without starting a pull.- Disk budget: three fake packs over budget → the oldest
last_usedis removed first. - Node: with nothing on 9200,
askstarts a node on 9300+, answers, andxerj hub stopstops exactly that PID (assert another dummy process named like the binary survives).
Done when an agent transcript (the harness in demo/playbooks/AGENT_FTX_HARNESS.md,
shell persona) shows one tool call per question in steady state.
6. Hosted endpoints (xerj-api/src/hub_api.rs) and the public node
Do
- New module
engine/crates/xerj-api/src/hub_api.rs, routes added inrouter.rsnext to/_memory:GET /_hub/v1/catalogue→ bytes of the file at confighub.catalogue_path, withETag= its sha256 andCache-Control: public, max-age=3600.GET /_hub/v1/def/{slug}/{symbol}?k=→Answerfrom step-3lookup.GET /_hub/v1/code/{slug}?q=&k=&max_tokens=&mode=→Answerfromrun_code_query(the same function the CLI uses; implementXcHttpfor the in-process case by calling the node's own loopback address with the admin key — simplest; document it in the module doc).GET /_hub/v1/search?q=&k=→multi_matchonbody,defs,symbol_textacrossxc-*-docs,xc-*-defs, group by corpus (parse the slug from the index name with thexc-<name>-b<build>rule inxc.rs), return{corpora: [{slug, best_score, best_hit}]}sorted by score, at most 10.
- All four are
GET, read-only, slug validated against the catalogue (unknown → 404),symbolandqlimited to 512 bytes,k ≤ 50. A config keyhub.public = truemakes exactly these four routes skipauth_middleware; everything else keeps auth. - Rate limit: if a per-key limiter exists in
xerj-apireuse it; otherwise a token bucket per client IP, 60 requests/minute, 429 withRetry-After. - The public node: one node started with
hub.public = trueand every live pack restored (xerj hub pullfor each slug against its own URL), running the way the reference-coding node runs on the maintainer machine (a systemd user unit; keep its data directory under its own path, never a shared one).hub.xerj.org/api/v1/*is a reverse proxy to it; the staticcatalogue.jsonstays on the site so the catalogue is reachable even when the node is not.
Tests (engine/crates/xerj-api/tests/hub_api.rs, same harness as the other
endpoint tests):
- With the fixture pack restored:
defreturns the fixture value;codereturns at least one hit withfile:line;searchgroups by the fixture slug. - Unknown slug → 404;
k=1000→ clamped to 50; a 600-byteq→ 400. hub.public=false→ the routes require the key;true→ they do not, andPOST /_hub/v1/…is 405.- Rate limit: the 61st request in a minute from one IP is 429.
Done when curl https://hub.xerj.org/api/v1/def/lz4/LZ4_VERSION_RELEASE returns
value: "0" with the pin date, and the test file passes.
7. MCP tools and gRPC
MCP (engine/crates/xerj-mcp/src/lib.rs): add to tool_specs() and the dispatch:
xerj_hub_find {question: string, k?: integer} → the step-6 /search grouping
xerj_hub_def {slug: string, symbol: string, k?: integer} → Answer (def)
xerj_hub_ask {question: string, corpus?: string, mode?: "def"|"how"|"files", max_tokens?: integer} → Answer, via the step-5 flow
Local packs first, hosted second, exactly like the CLI (call xerj_hub::ask; do not
reimplement). XERJ_HUB_URL overrides the hosted base URL. The tool result text is
the step-1 rendering; the structured content is the Answer JSON.
gRPC (engine/proto/xerj.proto, engine/crates/xerj-server/src/grpc.rs):
service XerjHub {
rpc Ask(HubAskRequest) returns (HubAnswer);
rpc Def(HubDefRequest) returns (HubAnswer);
rpc Catalogue(HubCatalogueRequest) returns (HubCatalogueResponse);
}
message HubAskRequest { string question = 1; string corpus = 2; string mode = 3; uint32 max_tokens = 4; }
message HubDefRequest { string slug = 1; string symbol = 2; uint32 k = 3; }
message HubCatalogueRequest {}
message HubCatalogueResponse { bytes catalogue_json = 1; bytes signature = 2; }
message HubHit { string file = 1; uint32 line = 2; string kind = 3; string text = 4; string value = 5; float score = 6; string licence_warning = 7; }
message HubAnswer {
string corpus = 1; string pin_sha = 2; string pin_date = 3; uint32 age_days = 4;
string licence_spdx = 5; string licence_class = 6; string answered_from = 7; string mode = 8;
repeated HubHit hits = 9; bool truncated = 10; repeated string warnings = 11;
}
Register XerjHubServer with add_service next to XerjSearchServer, behind the
same interceptor. Field numbers are fixed by this document; never renumber.
Tests: MCP tools/list contains the three names; xerj_hub_def against the
fixture pack returns the fixture value; gRPC Def returns the same HubAnswer as
the HTTP route (compare field by field), and the existing
grpc_h2c_fail_closed.rs still passes.
Done when the tests pass and xerj init (step 9) prints the MCP registration.
8. Lockfile, freshness, sync, status
Do
xerj.lock(TOML) in the project root (first parent ofcwdcontaining.git, elsecwd):
Written the first time an answer is served in that project; appended per corpus; never rewritten silently.version = 1 [[corpus]] slug = "quinn" pin_sha = "…" pin_date = "2026-10-12" pack_sha256 = "…"--locked(onask) exits 2 if the catalogue's pin differs from the lock.xerj hub updateprints the diff and rewrites the lock when--writeis given.- Freshness warning: if the question matches
\b(current|latest|newest|now)\bandage_days > refresh_cadence_days, add the warningpin is <n> days old and the question asks for the current source. xerj hub sync: refresh the catalogue; for every cached pack whose catalogue pin is newer, pull the new pin and remove the old one after the new one is restored; verify signatures; prune to the disk budget; write~/.xerj/hub/last_sync.--install-timerwrites a systemd user timer (Linux) or a launchd plist (macOS) runningxerj hub syncevery 6 hours; print the unit path; nothing else.xerj hub status: cached packs (slug, pin8, age, bytes, last used), node (running or not, port, PID), hosted setting, last sync, catalogue age. Plain text, one line per item.- CI (corpus-hub branch, new
.github/workflows/hub-packs.yml, copy the job shape ofpack-publish.ymlincluding its signing secret): on push, for each live corpus whose manifest changed, build the pack (step 2) and upload it as a releasepack-<slug>-<pin8>; then rebuild and deploy the catalogue (step 4). A scheduled job (weekly for code domains, daily for feed corpora withrefresh_cadence_days = 1) re-pins: rungit ls-remotefor the manifest's URL, and if the default branch head differs frompin_sha, open a PR that updates the manifest (a human merges; packs follow on merge).
Tests: lock written and appended; --locked mismatch exits 2; the freshness
warning fires only with both conditions; sync with a newer pin replaces the pack
and the old dir is gone; status output lists exactly the packs on disk.
Done when two machines with the same xerj.lock and --locked give identical
Answer JSON for the fixture question.
9. Agent-facing surface
Do
landing/llms.txtsection "Reference corpora: hub or local" becomes: the one-line command, the MCP call, the--defform, when to use it, the three success lines, thexerj hub pullform for large packs, and nothing about manifests. Longest line under 300 characters.landing/llms-full.txt§11 updated to match. Followdocs/research/llms-txt-2026-09/README.mdbefore editing either.xerj initinstalls a second skill.claude/skills/xerj-hub/SKILL.md(and the Cursor rule) with: the decision rule (hub for public material you cannot quote, local for your own files, neither for what you already know), the one line,--deffor exact questions, "stop only the node you started", and the MCP registration per client (claude mcp add …,.mcp.jsonmcpServers, VS Codeservers).- Hub pages (
tools/corpus-hub-site/gen.py, corpus-hub branch): the copy prompt and the "Use it" block become the one line plus the--defform; the catalogue link is on every page.
Tests: the docs drift tests pass; python3 scripts/seo/*.py --check gates pass;
the first-time-agent harness (demo/playbooks/AGENT_FTX_HARNESS.md) shell-less
persona reaches an answer through xerj_hub_ask in one call.
10. Measurement (before anything is published)
Pre-registered plan in benchmarks/hub-pov/PLAN.md on branch bench/corpus-tasks
(write it before running). Arms: memory only; normal agent with git; normal agent
with web only (no git; requests counted; a fetch that returns a bot-protection page
or 429 is recorded as a failed fetch); hosted regex code search; a web search API;
XERJ today (corpus add path); XERJ v2 first use; XERJ v2 steady state (pack cached).
Task classes: the registered exact-lookup suites (control; a tie with git is
expected), how-does-it-work questions across files, non-git multi-source data,
ten-question sessions per corpus. Metrics: correct, cost, output tokens, turns,
bytes downloaded, wall time on a quiet machine with load recorded, failed fetches,
routing accuracy. At least three repeats per cell. Every agent runs in the sandbox
recipe in benchmarks/hub-pov/sandbox.sh (bubblewrap, own PID namespace, read-only
filesystem outside its home). Results go to the discussion, win or not; the site
shows them only if XERJ v2 beats both normal-agent arms on cost and output tokens at
equal or better accuracy.
11. Order, and what not to do
Order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10. Steps 2 and 3 can be developed together; 6 needs 2 and 3; 5 needs 4 and 6 (hosted search) but can ship with the sketch router first.
Do not:
- clone or index a hub corpus on the user's machine (that is the v1 path; v2 pulls);
- require a corpus name in the question, or omit it from the answer;
- extract or restore anything before the sha256 and the signature both verify;
- block an answer on a download when a hosted answer is allowed;
- pull above
AUTO_PULL_MAX_BYTES, or anExcludedlicence class, or send the question to the hosted node when hosted resolves to off; - print passages when
Defmode answers the question; - start a node on a port that already answers, or stop any process other than the PID this tool recorded;
- renumber proto fields, change
format_versionsemantics, or add a second JSON parser for the registry manifest; - publish benchmark numbers unless the step-10 condition holds.
12. Release gate: backfill, binaries only, end-to-end
Steps 1–11 describe how one pack is built and used. This section defines what must
be true before a release advertises Hub v2 (xerj ask, xerj "<question>", the MCP
hub tools, the hub pages' one line). Until every item below holds, a release ships
the hub code switched off and says nothing about it.
Prerequisite: the key. HUB_PUBKEY_HEX holds the public half of the
HUB_SIGNING_SEED secret, and the hub key and secret in place? job in
hub-packs.yml passes. Without it nothing below can run.
Who routes a question to a corpus: the client. xerj_hub::router runs inside the
user's xerj binary. It routes from the project's lockfiles and from the per-corpus
term sketches in the signed catalogue.json. That file is static, so hub.xerj.org can
stay a static site, and routing works with no hub server. The hosted search and answer
routes (step 6) need a live public node, which does not exist. For this release,
hosted is OUT of scope: the client must work, and the gate is passed, with hosted
unreachable. A static host answers unknown paths with the HTML home page and status
200 (today /api/v1/catalogue.json and /api/v1/search both do), so the client must
treat a non-JSON or unsigned response as "unavailable", never as data.
12.1 Backfill: every existing corpus becomes a pre-indexed pack.
- Run
hub-packs.yml(workflow_dispatch,corpora: all) in capped batches until every live corpus inbacklog/backlog-100.jsonhas a release for its current pin. "No release for the current pin" already covers the first run, so the backfill needs no new code. - A corpus whose licence class is
Excludedgets no pack and leaves the catalogue. - A corpus whose build fails is listed with its error in the backfill report
(
benchmarks/hub-backfill/REPORT.md: slug, pin8, bytes, defs, result) and is fixed or delisted on purpose. Never let it disappear silently. - The two record packs from
pack-publish.yml(rust-vulnsandagent-session-trajectories, signed JSON record archives) are rebuilt as hubpacks (format_version: 1, snapshot layout) under the same pipeline and key.
12.2 Binaries only. Users get pre-indexed packs, never data to index themselves.
- Hosted downloads are the signed
.hubpack.tar.zst+.sigrelease assets. The only JSON served is the signedcatalogue.json(+.sig), which is the index of the packs and not corpus data. - The hub pages and
llms.txtstop linking per-corpus manifest JSON and stop printingxerj corpus add --from <url>. The registry manifests stay in the repo as build inputs forbuild-pack(step 2), and are not a consumer path. - The JSON record-pack releases (
pack-rust-vulns-*,pack-agent-session-trajectories-*) are retired once their hubpacks verify, andpack-publish.ymlstops publishing them.xerj corpus add --from <url>keeps working for anyone's own manifests, but the hub no longer points to it.
12.3 End-to-end, per pack (a CI gate in hub-packs.yml). Each pack job, after
Verify the signature and the layout and BEFORE Publish the draft, runs the
consumer path with the real binary:
- A fresh node with an empty data dir and an empty
HOME. xerj hub pull <slug>from the draft release URL, verified against the embedded key.xerj code <slug> --def <symbol> --json, for one symbol taken from the pack's owndefs.jsonl, must return that definition withanswered_from: local pack.xerj code <slug> "<a word from ask_for>" --jsonmust return at least one hit withfile:line.- Any failure leaves the release a draft and fails the job. A pack that has not passed this is never published.
12.4 End-to-end, whole hub (the release gate). Before tagging the release that
advertises v2, run from a clean machine (the sandbox recipe in
benchmarks/hub-pov/sandbox.sh; empty HOME; the release-candidate binary;
production hub.xerj.org):
- The catalogue loads and verifies, and its corpus count equals the number of published packs.
- For EVERY catalogued corpus,
xerj hub pull+ one--def+ one question pass (12.3 again, against the published assets and not the drafts). Results go into the backfill report. The gate is 100% of catalogued corpora; a failing corpus is delisted before the release, not shipped broken. xerj "<question>"with no corpus named routes and answers for the lz4 benchmark questions (step 2's "done when") and forcurl https://hub.xerj.org/api/v1/def/lz4/LZ4_VERSION_RELEASE(step 6's).- Hosted off, hosted unreachable (including an HTML 200 from the static site),
offline (
--offline) and over-the-cap (AUTO_PULL_MAX_BYTES) paths each give their documented message. Thecurl …/api/v1/def/…check above applies only once a public node exists. - The shell-less persona of the first-time-agent harness answers through
xerj_hub_askin one call (step 9). - A tampered pack (one flipped byte) and a tampered catalogue are refused, with nothing extracted.
Runnable as tools/hub-packs/whole_hub_e2e.sh (.github/workflows/hub-e2e.yml,
workflow_dispatch: a release tag or build, and the hub URL). It covers every
item above except the curl …/api/v1/def/… check (no public node) and the
shell-less persona (the agent harness). Per-corpus results are backfill-report rows
(kind: "e2e"); the gate-level checks go to GATE.md and the job summary. Until a
signed catalogue is deployed, --rehearse <pack tags> runs the same checks against a
loopback catalogue of published packs re-signed with a throwaway key; their real
signatures are checked separately, and every check says which key it ran against.
Only after 12.1–12.4 pass do the held agent-surface PRs merge and the release notes mention Hub v2. Step 10's measurement is separate and still governs any published performance claim.