AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(site): link hub.xerj.org — the registry now has a browsable site
xerj-org5e8432ad3f71
diff --git a/landing/blog/the-corpus-hub.html b/landing/blog/the-corpus-hub.htmlindex 360b177f3..d75d1f879 100644--- a/landing/blog/the-corpus-hub.html+++ b/landing/blog/the-corpus-hub.html@@ -138,6 +138,13 @@source pinned to SHAs, licences reviewed by a human per source, record packs checksummed anded25519-signed. It ships with four reference corpora, one signed daily pack, a measured use case,and one result we did not want and are publishing anyway.</p>++ <p class="lede"><strong>Update, 2026-10-02:</strong> the registry has outgrown this post. The+ launch four are now <strong>75 live corpora</strong> (104 backlog rows) across seven categories —+ including non-IT domains mined from real demand: eCFR titles 12/26/29 and UK legislation — and the+ registry has its own site: browse every corpus with its licence verdict, pin, size, use case and+ measured retrieval scores at <a href="https://hub.xerj.org" rel="external">hub.xerj.org</a>.+ What follows is the post as written at launch.</p></div><section class="scene-section">diff --git a/landing/docs/corpus-hub.html b/landing/docs/corpus-hub.htmlindex 6f6b22cb4..7f4d1e30e 100644--- a/landing/docs/corpus-hub.html+++ b/landing/docs/corpus-hub.html@@ -224,11 +224,14 @@packs — checksummed and signed. It lives on the<a href="https://github.com/xerj-org/xerj/tree/corpus-hub" rel="external"><code class="inline">corpus-hub</code>branch</a> of xerj-org/xerj, and contributions arrive as pull requests against that branch.+ Browse the whole registry as a site — every corpus with its category, use case, licence verdict,+ pin, size and measured retrieval scores — at <a href="https://hub.xerj.org" rel="external">hub.xerj.org</a>;+ that site is generated from the branch, so it is never stale relative to the registry.This page is the walkthrough; the branch's own<a href="https://github.com/xerj-org/xerj/blob/corpus-hub/tools/xerj-code/hub/CONTRIBUTING.md" rel="external">CONTRIBUTING.md</a>is the source of record for the rules.</p>- <p class="source-note">Source · tools/xerj-code/hub/CONTRIBUTING.md (corpus-hub branch) · launch post: /blog/the-corpus-hub</p>+ <p class="source-note">Source · tools/xerj-code/hub/CONTRIBUTING.md (corpus-hub branch) · launch post: /blog/the-corpus-hub · browse: hub.xerj.org</p><h2 id="consume">Consuming a corpus</h2><p>Three commands, no build step — the binary indexes from the pinned manifest directly:</p>diff --git a/landing/llms.txt b/landing/llms.txtindex c57e47f82..229d84586 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -48,7 +48,7 @@ Then, from an unknown folder to an answered question: `xerj autoindex <folder>`> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and memory recall use a deterministic feature-hashing embedder (384-d, L2-normalised → cosine): hybrid lexical+vector retrieval — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that ships in the binary and auto-downloads its model (~90 MB) on first use — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those is actually running.> * **`--embed-mode` is a server-start flag that applies AT INGEST.** Text is embedded as it is indexed, so the mode in force when a document was written is the mode that document keeps. Switching lexical → neural does **not** re-embed anything already indexed: you have to re-index every corpus you care about. **Decide before you index your first corpus.**> * **What the neural download actually is:** the model files (`config.json`, `tokenizer.json`, `model.safetensors`) are fetched over HTTPS from the **HuggingFace Hub** (`sentence-transformers/all-MiniLM-L6-v2`) into `~/.cache/huggingface` on first use, then reused. **XERJ pins no checksum of its own** for them — the integrity guarantee is TLS plus the Hub, not a digest XERJ verifies. Air-gapped or policy-constrained: pre-download the three files, point `embedding.local_model_dir` at the folder, and nothing is fetched.-> - **Not implemented — do not plan around these.** Alerting is PARTIAL, and exactly one watch shape is real (since v1.0.0-rc.80, [#1082](https://github.com/xerj-org/xerj/pull/1082) closing [#1062](https://github.com/xerj-org/xerj/issues/1062); an earlier version of this line said nothing ever evaluates a watch — that was true then and is not now): `PUT /_watcher/watch/<id>` with `trigger.schedule.interval` (>= 1s), `input.search` over exactly one index, `condition.xerj_decide` (question, positive_label, p_min, k) and an `index_alert` action runs a background evaluator — it prefilters with the watch's own query (keyset paging, 100/page), runs each rendered question through the real `/_decide`, and writes a fire record into `.xerj_alert_fires` when the positive label wins non-abstaining at or above `p_min`. Every other shape — cron schedules, transforms, other action types, multi-index input — is refused with a 501 naming what is not evaluated (malformed supported shapes are 400). Honest limits: a restart does not resume watches; the per-pass decide budget is 1000. User-supplied ingest plugins: transforms are built-in native code only (no wasmtime backend in the tree). Downloadable pre-indexed corpus packs: still none — but record packs are real since v1.0.0-rc.78 (2026-09-29, [#1046](https://github.com/xerj-org/xerj/pull/1046), [#1048](https://github.com/xerj-org/xerj/pull/1048); this line said "planned, no code" before): `xerj corpus build` turns a declarative recipe into a checksummed portable pack of records (`format_version` refused when unknown, licence + provenance on every record), `corpus add --from <pack>` verifies the per-file SHA-256 and `--verify-sig` a detached ed25519 signature over SHA256SUMS, and the first pack — `rust-vulns` — is rebuilt, signed and published daily as a dated GitHub Release. Still unbuilt: the PRE-INDEXED half (an already-indexed bundle you mount instead of indexing records yourself). The hub directory beyond that one pack is real since 2026-10-01: the Corpus Hub, a `corpus-hub` branch on this repository carrying four reference-corpus manifests — search / vector / storage / columnar domains (lucene, tantivy, quickwit, meilisearch, sonic, elasticsearch · qdrant, usearch, instant-distance, hnswlib · sled, fjall, redb · clickhouse), every repo pinned to the full SHA the build used and human licence-reviewed — plus the rust-vulns pack recipe; consume any of them directly with `xerj corpus add --from <manifest-url>`, and new corpora arrive as PRs against that branch ([CONTRIBUTING.md](https://github.com/xerj-org/xerj/blob/corpus-hub/tools/xerj-code/hub/CONTRIBUTING.md) · guide: [xerj.org/docs/corpus-hub](https://xerj.org/docs/corpus-hub)). `percolate` **is** real — and so, since v1.0.0-rc.77 (2026-09-21, [#1008](https://github.com/xerj-org/xerj/pull/1008) closing [#965](https://github.com/xerj-org/xerj/issues/965)), **is** object storage (an earlier version of this page said the opposite): `storage.backend = "s3"` publishes each segment family as ONE immutable ZBM1 bundle object plus a per-index `snapshot.json` catalogue, merges publish before retiring their inputs, and a fresh node adopts the bucket — a real S3-compatible client (Cloudflare R2, MinIO, AWS S3 over `aws-sdk-s3`; the old local-directory simulation survives only as a test double). The one remaining startup error is `storage.backend = "s3" requires storage.s3_bucket to name an existing bucket`. `xerj autoindex s3://bucket/prefix` is the other direction — it reads documents OUT of a bucket as a source, it does not store the index there. Statuses, each with the file that proves it: [ROADMAP.md — the zero-token direction](https://github.com/xerj-org/xerj/blob/main/ROADMAP.md#the-zero-token-direction) · [OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/OBJECT_STORAGE.md) · [does XERJ support S3, alerting or custom plugins?](https://xerj.org/answers/does-xerj-support-s3-alerting-plugins.md) · [CORPUS_PACKS.md](https://github.com/xerj-org/xerj/blob/main/docs/CORPUS_PACKS.md)+> - **Not implemented — do not plan around these.** Alerting is PARTIAL, and exactly one watch shape is real (since v1.0.0-rc.80, [#1082](https://github.com/xerj-org/xerj/pull/1082) closing [#1062](https://github.com/xerj-org/xerj/issues/1062); an earlier version of this line said nothing ever evaluates a watch — that was true then and is not now): `PUT /_watcher/watch/<id>` with `trigger.schedule.interval` (>= 1s), `input.search` over exactly one index, `condition.xerj_decide` (question, positive_label, p_min, k) and an `index_alert` action runs a background evaluator — it prefilters with the watch's own query (keyset paging, 100/page), runs each rendered question through the real `/_decide`, and writes a fire record into `.xerj_alert_fires` when the positive label wins non-abstaining at or above `p_min`. Every other shape — cron schedules, transforms, other action types, multi-index input — is refused with a 501 naming what is not evaluated (malformed supported shapes are 400). Honest limits: a restart does not resume watches; the per-pass decide budget is 1000. User-supplied ingest plugins: transforms are built-in native code only (no wasmtime backend in the tree). Downloadable pre-indexed corpus packs: still none — but record packs are real since v1.0.0-rc.78 (2026-09-29, [#1046](https://github.com/xerj-org/xerj/pull/1046), [#1048](https://github.com/xerj-org/xerj/pull/1048); this line said "planned, no code" before): `xerj corpus build` turns a declarative recipe into a checksummed portable pack of records (`format_version` refused when unknown, licence + provenance on every record), `corpus add --from <pack>` verifies the per-file SHA-256 and `--verify-sig` a detached ed25519 signature over SHA256SUMS, and the first pack — `rust-vulns` — is rebuilt, signed and published daily as a dated GitHub Release. Still unbuilt: the PRE-INDEXED half (an already-indexed bundle you mount instead of indexing records yourself). The hub directory beyond that one pack is real since 2026-10-01 (and browsable since 2026-10-02 at [hub.xerj.org](https://hub.xerj.org), a site generated straight from the registry): the Corpus Hub, a `corpus-hub` branch on this repository carrying 75 live reference corpora (104 backlog rows) across seven categories — security precedent, standards and regulations, production knowledge, design guidance, failure precedent, and data- and net-engine reference implementations, plus non-IT regulation corpora mined from real demand (eCFR titles 12/26/29, UK legislation) — every repo pinned to the full SHA the build used and human licence-reviewed — plus the rust-vulns pack recipe; consume any of them directly with `xerj corpus add --from <manifest-url>`, and new corpora arrive as PRs against that branch ([CONTRIBUTING.md](https://github.com/xerj-org/xerj/blob/corpus-hub/tools/xerj-code/hub/CONTRIBUTING.md) · guide: [xerj.org/docs/corpus-hub](https://xerj.org/docs/corpus-hub) · browse: [hub.xerj.org](https://hub.xerj.org)). `percolate` **is** real — and so, since v1.0.0-rc.77 (2026-09-21, [#1008](https://github.com/xerj-org/xerj/pull/1008) closing [#965](https://github.com/xerj-org/xerj/issues/965)), **is** object storage (an earlier version of this page said the opposite): `storage.backend = "s3"` publishes each segment family as ONE immutable ZBM1 bundle object plus a per-index `snapshot.json` catalogue, merges publish before retiring their inputs, and a fresh node adopts the bucket — a real S3-compatible client (Cloudflare R2, MinIO, AWS S3 over `aws-sdk-s3`; the old local-directory simulation survives only as a test double). The one remaining startup error is `storage.backend = "s3" requires storage.s3_bucket to name an existing bucket`. `xerj autoindex s3://bucket/prefix` is the other direction — it reads documents OUT of a bucket as a source, it does not store the index there. Statuses, each with the file that proves it: [ROADMAP.md — the zero-token direction](https://github.com/xerj-org/xerj/blob/main/ROADMAP.md#the-zero-token-direction) · [OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/OBJECT_STORAGE.md) · [does XERJ support S3, alerting or custom plugins?](https://xerj.org/answers/does-xerj-support-s3-alerting-plugins.md) · [CORPUS_PACKS.md](https://github.com/xerj-org/xerj/blob/main/docs/CORPUS_PACKS.md)> - **Default `_search` source is compact.** With `_source` omitted, ordinary fields are returned but engine-generated `<field>_vector` and `<field>_vector_chunks` companions are omitted. Add `"_source": true` when the vectors are needed.> - **Default `_search` returns whole documents; project `_source` and ask for `_passage` to shrink it.** With no `_source`/`fields` in the request, every hit's `_source` is the complete stored document — for source files or long documents that is megabyte-scale for a handful of hits (measured: 5 hits over indexed source files = 416,630 bytes). Add `"fields":["_passage"]` to any query (lexical, or a single semantic/kNN clause — omit it for hybrid/multi-kNN fusion, which 400s naming the fix) to get just the matching snippet — `{field,ordinal,start_offset,end_offset,text,page?}` sliced from the original field — and project `_source` (e.g. `["title","ax_path"]`, or `false`) to drop the rest: the same query measured 12,122 bytes that way, a 34× reduction. `_passage` is opt-in and appears in no `_mapping` or default response, so a response that would benefit says so itself — an additive `_xerj.hints` entry carrying a ready-to-send corrected request lands ahead of `hits` whenever `_source` is unprojected and `_passage` wasn't requested (never inside `hits`/`total`/`_score`, so existing ES clients are unaffected).> - **Ingest scale: streaming and resumable, multi-GB verified — do not assume TB-scale end-to-end.** The `autoindex` client streams with flat memory and is resumable after `kill -9`. The SERVER's #948 RSS-runaway is FIXED ([#1002](https://github.com/xerj-org/xerj/pull/1002), v1.0.0-rc.76: at-rest retention after a 100k-doc ingest fell ~677 → ~108 MB; a 1 GB mailbox completes under its 16 GiB cap) — but the server still retains heap per indexed doc: three per-segment caches (`stored_value_cache`, `dv_cache`, `id_pos_cache`) are unbounded until a merge retires their segment ([#1032](https://github.com/xerj-org/xerj/issues/1032)). Corpora of many millions of documents can still OOM the server.diff --git a/landing/sitemap.xml b/landing/sitemap.xmlindex f62b55063..07c62be97 100644--- a/landing/sitemap.xml+++ b/landing/sitemap.xml@@ -334,7 +334,7 @@</url><url><loc>https://xerj.org/blog/</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/blog/cutting-the-index-28-percent</loc>@@ -462,7 +462,7 @@</url><url><loc>https://xerj.org/docs/</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/agents/</loc>@@ -478,39 +478,39 @@</url><url><loc>https://xerj.org/docs/aggregations</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/analyzers</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/api-es-compat</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/api-native</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/backup-restore</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/cli</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/clustering</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/compression</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/config</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/corpus-hub</loc>@@ -518,55 +518,55 @@</url><url><loc>https://xerj.org/docs/env</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/ingest</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/install</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/metrics</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/migration-from-es</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/operations</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/playbooks/full-text</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/playbooks/log-analytics</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/playbooks/observability</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/playbooks/siem</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/playbooks/vector-search</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/queries</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/quickstart</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/recipes/</loc>@@ -626,23 +626,23 @@</url><url><loc>https://xerj.org/docs/security</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/storage</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/troubleshooting</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/upgrades</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/docs/vectors</loc>- <lastmod>2026-09-30</lastmod>+ <lastmod>2026-10-01</lastmod></url><url><loc>https://xerj.org/for-agents</loc>