Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: roll the corpus-pack status in llms.txt / llms-full.txt — record packs shipped in rc.78

xerj-orgdd9f3858fc48
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 64522e04a..d7847e55c 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -287,7 +287,10 @@ For a complete disconnected deployment procedure, including current GitHub relea
pipeline naming an unimplemented processor is stored as unrunnable and every
ingest through it is refused.
* A log-specific index mode (the xerj-logs crate has zero call sites outside its
- tests) and downloadable signed corpus packs: planned, no code.
+ tests). Corpus-pack records left this list in v1.0.0-rc.78 (paragraph below);
+ what stays on it is the pre-indexed half - an already-indexed bundle you mount
+ instead of spending your own CPU-hours - and a hub directory of packs beyond
+ the first.
Object storage left this list in v1.0.0-rc.77 (2026-09-21, PR #1008, issue #965;
ROADMAP's own row for it was last checked there 2026-09-18 and still says unwired):
`storage.backend = "s3"` really stores indices in a bucket — one immutable ZBM1
@@ -299,6 +302,17 @@ For a complete disconnected deployment procedure, including current GitHub relea
requires storage.s3_bucket to name an existing bucket` (xerj-common/src/config.rs).
Still true of this build: single-writer, the WAL stays local, bundles are packed in
memory, and bucket orphans have no GC command (docs/OBJECT_STORAGE.md).
+ Corpus-pack records left this list in v1.0.0-rc.78 (2026-09-29, PRs #1046, #1047,
+ #1048, publish hardening #1049-#1051; this list said "planned, no code" before):
+ `xerj corpus build` turns a declarative TOML recipe (sources with licences,
+ identity edges, merge precedence) into a deterministic, checksummed pack of
+ records — the manifest carries a `format_version` readers refuse when unknown,
+ licence and provenance ride on every record, `xerj corpus add --from <dir|zip>`
+ verifies the per-file SHA-256, and `--verify-sig <pubkey>` checks a detached
+ ed25519 signature over SHA256SUMS before anything is materialized. The first
+ pack is `rust-vulns` (4,107 osv.dev + RustSec filings identity-resolved to
+ 1,950 records), rebuilt, signed and published by a scheduled workflow as a
+ dated GitHub Release per build day (docs/CORPUS_PACKS.md).
The opt-in neural path (`--embed-mode neural`) had four measured limits, each a
filed issue (#938, #939, #940, #937 in the order below) with a reproduction under
benchmarks/neural-path-triage/ — all four FIXED on `main` before the v1.0.0-rc.77
diff --git a/landing/llms.txt b/landing/llms.txt
index b0c3c408d..ba6ce7786 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -47,7 +47,7 @@ Then, from an unknown folder to an answered question: `xerj autoindex <folder>`
> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and memory recall use a deterministic feature-hashing embedder (384-d, L2-normalised → cosine): hybrid lexical+vector retrieval — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that ships in the binary and auto-downloads its model (~90 MB) on first use — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those is actually running.
> * **`--embed-mode` is a server-start flag that applies AT INGEST.** Text is embedded as it is indexed, so the mode in force when a document was written is the mode that document keeps. Switching lexical → neural does **not** re-embed anything already indexed: you have to re-index every corpus you care about. **Decide before you index your first corpus.**
> * **What the neural download actually is:** the model files (`config.json`, `tokenizer.json`, `model.safetensors`) are fetched over HTTPS from the **HuggingFace Hub** (`sentence-transformers/all-MiniLM-L6-v2`) into `~/.cache/huggingface` on first use, then reused. **XERJ pins no checksum of its own** for them — the integrity guarantee is TLS plus the Hub, not a digest XERJ verifies. Air-gapped or policy-constrained: pre-download the three files, point `embedding.local_model_dir` at the folder, and nothing is fetched.
-> - **Not implemented — do not plan around these.** Alerting: there is none — `_watcher` stores a watch, answers `condition.met: true`, and nothing ever evaluates it; the console alert-rule indices have schemas and no evaluator. User-supplied ingest plugins: transforms are built-in native code only (no wasmtime backend in the tree). Downloadable pre-indexed corpus packs: planned, no code. `percolate` **is** real — and so, since v1.0.0-rc.77 (2026-09-21, [#1008](https://github.com/xerj-org/xerj/pull/1008) closing [#965](https://github.com/xerj-org/xerj/issues/965)), **is** object storage (an earlier version of this page said the opposite): `storage.backend = "s3"` publishes each segment family as ONE immutable ZBM1 bundle object plus a per-index `snapshot.json` catalogue, merges publish before retiring their inputs, and a fresh node adopts the bucket — a real S3-compatible client (Cloudflare R2, MinIO, AWS S3 over `aws-sdk-s3`; the old local-directory simulation survives only as a test double). The one remaining startup error is `storage.backend = "s3" requires storage.s3_bucket to name an existing bucket`. `xerj autoindex s3://bucket/prefix` is the other direction — it reads documents OUT of a bucket as a source, it does not store the index there. Statuses, each with the file that proves it: [ROADMAP.md — the zero-token direction](https://github.com/xerj-org/xerj/blob/main/ROADMAP.md#the-zero-token-direction) · [OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/OBJECT_STORAGE.md) · [does XERJ support S3, alerting or custom plugins?](https://xerj.org/answers/does-xerj-support-s3-alerting-plugins.md)
+> - **Not implemented — do not plan around these.** Alerting: there is none — `_watcher` stores a watch, answers `condition.met: true`, and nothing ever evaluates it; the console alert-rule indices have schemas and no evaluator. User-supplied ingest plugins: transforms are built-in native code only (no wasmtime backend in the tree). Downloadable pre-indexed corpus packs: still none — but record packs are real since v1.0.0-rc.78 (2026-09-29, [#1046](https://github.com/xerj-org/xerj/pull/1046), [#1048](https://github.com/xerj-org/xerj/pull/1048); this line said "planned, no code" before): `xerj corpus build` turns a declarative recipe into a checksummed portable pack of records (`format_version` refused when unknown, licence + provenance on every record), `corpus add --from <pack>` verifies the per-file SHA-256 and `--verify-sig` a detached ed25519 signature over SHA256SUMS, and the first pack — `rust-vulns` — is rebuilt, signed and published daily as a dated GitHub Release. Still unbuilt: the PRE-INDEXED half (an already-indexed bundle you mount instead of indexing records yourself) and a hub directory beyond that one pack. `percolate` **is** real — and so, since v1.0.0-rc.77 (2026-09-21, [#1008](https://github.com/xerj-org/xerj/pull/1008) closing [#965](https://github.com/xerj-org/xerj/issues/965)), **is** object storage (an earlier version of this page said the opposite): `storage.backend = "s3"` publishes each segment family as ONE immutable ZBM1 bundle object plus a per-index `snapshot.json` catalogue, merges publish before retiring their inputs, and a fresh node adopts the bucket — a real S3-compatible client (Cloudflare R2, MinIO, AWS S3 over `aws-sdk-s3`; the old local-directory simulation survives only as a test double). The one remaining startup error is `storage.backend = "s3" requires storage.s3_bucket to name an existing bucket`. `xerj autoindex s3://bucket/prefix` is the other direction — it reads documents OUT of a bucket as a source, it does not store the index there. Statuses, each with the file that proves it: [ROADMAP.md — the zero-token direction](https://github.com/xerj-org/xerj/blob/main/ROADMAP.md#the-zero-token-direction) · [OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/OBJECT_STORAGE.md) · [does XERJ support S3, alerting or custom plugins?](https://xerj.org/answers/does-xerj-support-s3-alerting-plugins.md) · [CORPUS_PACKS.md](https://github.com/xerj-org/xerj/blob/main/docs/CORPUS_PACKS.md)
> - **Default `_search` source is compact.** With `_source` omitted, ordinary fields are returned but engine-generated `<field>_vector` and `<field>_vector_chunks` companions are omitted. Add `"_source": true` when the vectors are needed.
> - **Default `_search` returns whole documents; project `_source` and ask for `_passage` to shrink it.** With no `_source`/`fields` in the request, every hit's `_source` is the complete stored document — for source files or long documents that is megabyte-scale for a handful of hits (measured: 5 hits over indexed source files = 416,630 bytes). Add `"fields":["_passage"]` to any query (lexical, or a single semantic/kNN clause — omit it for hybrid/multi-kNN fusion, which 400s naming the fix) to get just the matching snippet — `{field,ordinal,start_offset,end_offset,text,page?}` sliced from the original field — and project `_source` (e.g. `["title","ax_path"]`, or `false`) to drop the rest: the same query measured 12,122 bytes that way, a 34× reduction. `_passage` is opt-in and appears in no `_mapping` or default response, so a response that would benefit says so itself — an additive `_xerj.hints` entry carrying a ready-to-send corrected request lands ahead of `hits` whenever `_source` is unprojected and `_passage` wasn't requested (never inside `hits`/`total`/`_score`, so existing ES clients are unaffected).
> - **Ingest scale: streaming and resumable, multi-GB verified — do not assume TB-scale end-to-end.** The `autoindex` client streams with flat memory and is resumable after `kill -9`. The SERVER's #948 RSS-runaway is FIXED ([#1002](https://github.com/xerj-org/xerj/pull/1002), v1.0.0-rc.76: at-rest retention after a 100k-doc ingest fell ~677 → ~108 MB; a 1 GB mailbox completes under its 16 GiB cap) — but the server still retains heap per indexed doc: three per-segment caches (`stored_value_cache`, `dv_cache`, `id_pos_cache`) are unbounded until a merge retires their segment ([#1032](https://github.com/xerj-org/xerj/issues/1032)). Corpora of many millions of documents can still OOM the server.