AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Corpus packs — build, sign, publish, consume
A corpus pack is a portable directory of records — advisories, datasets,
any structured data, not source code — built by a declarative recipe, with
per-file checksums, licence and provenance on every record, and an optional
detached ed25519 signature that proves who built it. Install one with
xerj corpus add --from <pack> and it becomes a reference corpus that
xerj code and plain ES-compat search both query.
This is the records half of the "corpus hub" roadmap item: the pack carries records, and indexing them still costs each consumer CPU-hours — the pre-indexed half is tracked separately (#1030).
The tool is domain-agnostic by design: what a record is, which fields are identity, and how duplicates merge all live in the recipe, never in the code. One recipe indexes advisories, another would index mail or genomes. Licence terms ride along as record data; the tool does not enforce them — the pack author owns what goes in a pack.
Every statement below points at the code that makes it true:
| Area | File |
|---|---|
Pipeline orchestration, corpus build CLI |
engine/crates/xerj-autoindex/src/harvest/mod.rs |
| Recipe parsing + strict validation | engine/crates/xerj-autoindex/src/harvest/recipe.rs |
Source adapters (dir, http-zip, http-json, git) |
engine/crates/xerj-autoindex/src/harvest/source.rs |
Input-format normalisation (osv, rustsec-md, flat) |
engine/crates/xerj-autoindex/src/harvest/normalize.rs |
| Content-addressed store (presence = dedup) | engine/crates/xerj-autoindex/src/harvest/store.rs |
| Union-find identity resolution | engine/crates/xerj-autoindex/src/harvest/identity.rs |
| Deterministic pack emit + checksums | engine/crates/xerj-autoindex/src/harvest/pack.rs |
| Mapping/relation suggestions | engine/crates/xerj-autoindex/src/harvest/suggest.rs |
| ed25519 keygen / sign / verify | engine/crates/xerj-autoindex/src/harvest/sign.rs |
corpus add --from <pack> + --verify-sig |
engine/crates/xerj-autoindex/src/xc.rs |
| Claims pinned to a measured build | engine/crates/xerj-autoindex/tests/pack_claims.rs |
| Showcase recipes + stats | tools/packs/rust-vulns/, tools/packs/cisa-kev/ |
| Scheduled build/sign/publish | .github/workflows/pack-publish.yml |
Quick start
# consume the published, signed rust-vulns pack (see its README for the
# exact curl lines) — verification refuses the pack whole on a bad signature:
xerj corpus add rust-vulns --from rust-vulns-pack.zip --verify-sig rust-vulns.pub
xerj corpus index rust-vulns
xerj code rust-vulns "smallvec insert_many buffer overflow"
# or build the same pack yourself — the recipe is the whole build:
xerj corpus build rust-vulns --recipe tools/packs/rust-vulns/recipe.toml
xerj corpus add rust-vulns --from ~/.xerj-code/builds/rust-vulns/pack/rust-vulns
The recipe
A recipe is a commented TOML file — licence reasoning belongs in comments, and the recipe ships verbatim inside the pack as provenance. Sections:
[[sources]]—slug,kind(dir|http-zip|http-json|git),url/path,format(the normalisation adapter),licence, optionalglob,records_field(to unwrap a top-level JSON object's array field), and watermark. Sources are advisory-listed upstream trees, HTTP zip exports, single-URL JSON feeds, or git repos fetched shallow at a pinned ref.[envelope]— which normalised fields become the universal envelope:id_from,title_from,body_join,defs_from.defs(space-joined symbol paths) is what feedsxerj code's definition search.[identity]—edgesover field values ({ field = "id" }, or{ field = "aliases", each = true }to union on every array element) andcanonical_source_order, which decides the canonical id when sources disagree. Union-find closes alias chains for free: A↔CVE↔B merges A and B even with no direct A↔B edge.[merge]—precedence(which source wins a disagreeing scalar; arrays are always unioned, never chosen). The merged record'slicenceis the most restrictive of its contributing sources, andsources[]names them all.[[derived]]— pure post-merge ops (regex_extract,present, …) for fields likecve_idsoris_withdrawn.[emit]— pack shape (shards).
Validation is strict: an unknown key is a loud error, not a silent ignore.
The full example, with per-source licence reasoning in comments, is
tools/packs/rust-vulns/recipe.toml.
Incremental by construction
There is no --sync mode to get wrong. Every normalised record is stored at
store/<slug>.<xxh3-of-canonical-json>.json; a re-run recomputes the key,
sees the file, and skips. Only genuinely new or changed records are written,
and the pack is re-emitted deterministically (fixed shard count, records
ordered by id, a uniform key set on every record) from whatever the store
holds. --fresh discards the build state for a full re-harvest. A dir
source whose path is missing, is not a directory, or cannot be listed fails
the build and does not prune envelopes already stored for that slug; an
existing empty directory is still a real empty upstream and does prune.
A previous manifest.json that is not valid JSON, or that has no integer
generation, fails the same way — --fresh is the recovery. Records whose
id is empty or not a string are refused rather than shipped under one
collapsed id.
Build state lives under ~/.xerj-code/builds/<name>/ (override the root with
XERJ_CODE_HOME) — persistent across reboots, never under /tmp.
The pack format
pack/<name>/
manifest.json # format_version (readers refuse unknown), counts,
# sources + watermarks, per-file sha256
records-*.jsonl # the records, sharded by hash(id), id-ordered
recipe.toml # verbatim, as provenance
mapping.suggested.json # suggested field mapping — never applied
suggestions.md # human-readable suggestions + join candidates
relations.jsonl # candidate relations with cardinality
SHA256SUMS # sha256 of every file above
SHA256SUMS.sig # optional: raw 64-byte ed25519 over SHA256SUMS
corpus add --from <pack> verifies every checksum before anything is
materialized, and refuses an unknown format_version outright — a published
pack outlives the release that built it, so readers gate on the version they
were written for. The suggestions are suggestions, not decisions: the
index infers the real mapping at index time, as it does for any folder.
Signatures
Checksums prove a pack arrived intact. A signature proves who built it. The two answer different attacks, and the difference is pinned by a test: a self-consistent rebuild (tampered records with honestly rewritten checksums) passes every checksum and fails the signature — that is the attack the signature exists for.
# keygen: writes <prefix>.key (SECRET — the seed) and <prefix>.pub
xerj corpus keygen --out rust-vulns
# sign (CI or wherever the seed lives; the build itself never signs)
xerj corpus sign <pack-dir> --key rust-vulns.key
# verify on the way in — before anything is materialized or indexed
xerj corpus add <name> --from <pack> --verify-sig rust-vulns.pub
Design rules, each load-bearing:
- The public key travels out of band — committed beside the recipe, listed on the project site, never inside the pack. A key shipped beside its own signature verifies nothing.
- Build ≠ publish.
corpus buildemits; it never signs. The seed exists only where signing happens (a CI secret in our workflow), and the workflow verifies its own output against the committed.pubbefore publishing — a half-rotated key fails the build instead of shipping an unverifiable pack. - The signature covers SHA256SUMS, which covers every pack file; the signature itself is deliberately not listed in SUMS (it would be circular).
- A signature proves origin, not safety. A verified pack is exactly what the key holder built — indexing it is still your decision, and a pack remains untrusted input to the same degree an indexed folder is.
The released pack is signed with a single v1 release key; the rotation
procedure is in
tools/packs/rust-vulns/README.md.
Publishing
The pack publish workflow rebuilds
and signs rust-vulns daily and attaches the zip, the loose checksums, the
signature, and a freshness file to a GitHub Release per build day — dated
tags pack-rust-vulns-YYYY-MM-DD, newest first at
releases?q=pack-rust-vulns.
This repository has immutable releases (release tags are single-use), which
is why the pack ships one release per day instead of one rolling URL —
each day's release is exactly the bytes that shipped, pinned by its
signature, and the workflow keeps only the newest 7.
Freshness is an operational promise the schedule keeps, not a README claim:
the pack's own 30-day staleness refusal applies to our published corpus
exactly as it does to a user's reference corpora.
Publishing your own pack needs none of this — a pack directory is
self-contained, and corpus add --from <dir|zip> is the whole install. The
release machinery exists because our pack promises freshness.
Fresh-machine portability (2026-10-23)
Checked on one Linux x86_64 host with the published xerj v1.0.0-rc.98
tarball (xerj-1.0.0-rc.98-x86_64-unknown-linux-gnu.tar.gz, sha256sum -c
against the release .sha256 exited 0) and the public pack release
pack-rust-vulns-2026-10-22. This was not a build of the tree that produced
the pack.
- Loose
rust-vulns-SHA256SUMSwas byte-identical toSHA256SUMSinsiderust-vulns-pack.zip.sha256sum -con that file exited 0 (every named file printedOK). - The public key fetched from
https://raw.githubusercontent.com/xerj-org/xerj/main/tools/packs/keys/rust-vulns.pubmatchedtools/packs/keys/rust-vulns.pubin this checkout. xerj corpus add rust-vulns --from rust-vulns-pack.zip --verify-sig rust-vulns.pubprintedsignature verifiedand exited 0. It reported 1297rustsecrecords and 748osvrecords, 2045 in total, pack generation 1. The same 2045 is the count in that release'srust-vulns-freshness.json(2045 advisories, built 2026-10-22, gen 1).- A public key with one flipped nibble:
corpus addexited 2 withsignature does not verify, and the corpus directory stayed empty. Runningcorpus indexafter that refusal (exit status discarded) exited 2 withno corpusand still wrote nothing. - A zip whose
records-00000.jsonlhad one byte flipped, checksums not rewritten:corpus add --verify-sigexited 2 withchecksum mismatchand wrote nothing. - Against
xerj --insecure --port 19200,xerj corpus index rust-vulnsexited 0. Thexerj-doneline wasok=true exit=0 reason=completed wall=6.9s files=2 records=2045 generation=1. xerj code rust-vulns "smallvec insert_many buffer overflow" --url http://127.0.0.1:19200exited 0. The top passage was the RustSec record titled "Buffer overflow in SmallVec::insert_many" (score 27.50). The hit was labelledline unknown.
cisa-kev was not part of this run. gh release list -R xerj-org/xerj -L 200
on 2026-10-23 returned no tag containing cisa or kev, and
tools/packs/cisa-kev/README.md says that pack is not published or signed.
There is no public URL to download, checksum, or query.
Honest limits
- Indexing still costs each consumer the same CPU-hours. The pack distributes records, not a built index — the pre-indexed half is #1030.
- A pack is only as good as its sources and its recipe. Identity
resolution merges what the recipe's edges say to merge; a wrong edge is a
wrong merge, visibly, in
sources[]. - Not a security product. The showcase pack is a searchable corpus of advisories, not a scanner, and makes no completeness claim beyond "what its sources carried at build time".
- The suggester never mutates records. Field types, joins and relations are hints computed from a sample; the index decides at index time.