Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

release: cut 1.0.0-rc.14

xerj-org9b020903f704
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 73d7ca4b6..9160e83f7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,8 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+## [1.0.0-rc.14] - 2026-08-10
+
### Changed
+- **One documented core and memory policy, and the worker knobs are live**
+ ([#240](https://github.com/xerj-org/xerj/issues/240)). XERJ had no single
+ answer to "how much of this machine may I take?", and the scattered answers it
+ did have disagreed with each other — and, on Darwin, with the OS.
+ `engine.flush_workers`, `engine.merge_workers` and `engine.search_workers`
+ were documented but inert; they now drive real pool widths, and a value of `0`
+ refuses startup instead of being silently reinterpreted. `--bulk-mb` above 24
+ and `--workers 0` are likewise refused rather than clamped, so a request the
+ engine will not honour fails loudly. Only hand-written configs are affected:
+ the shipped `xerj.default.toml` has no `[engine]` section.
+
+- **One 16 MiB file no longer costs 65 GB of RSS**
+ ([#239](https://github.com/xerj-org/xerj/issues/239)). `split_sections` was
+ quadratic in both time and memory: a 16 MiB text file with no blank line
+ needed **65.6 GB of peak RSS and 16.3 s** to produce 16.8 MB of sections.
+ Now linear — the same input takes **8.3 ms and 35 MB**.
+
- **BREAKING — `server.bind_address` now defaults to `127.0.0.1`, and a
cleartext node refuses to publish itself to the network**
([#228](https://github.com/xerj-org/xerj/issues/228)). The old default was
@@ -54,8 +73,78 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
instead, after the data directory, the `.xerj_*` system indices, the master
key and a printed first-run `admin.key` already existed.
+### Security
+
+- **API key secrets are no longer stored in the clear**
+ ([#201](https://github.com/xerj-org/xerj/issues/201)). `api_keys.json` held
+ the secret verbatim. 0600 plus atomic rename is right for a secret a *process*
+ owns, but a file has more readers than a process: a backup, a snapshot, a
+ container layer, a support bundle, a decommissioned disk. Secrets are now
+ stored as a salted SHA-256 (`$ssha256$<salt>$<digest>`) and compared in
+ constant time, with existing plaintext records migrated on load. The fast hash
+ is deliberate: the secret is two v4 UUIDs — 244 bits of CSPRNG output, never
+ human-chosen — so offline guessing is out of reach whatever the hash costs,
+ while this comparison runs on *every authenticated request*, where an Argon2id
+ would add tens of milliseconds per request and hand out a free CPU-exhaustion
+ DoS. The same release makes `_security/_authenticate` report the key's real
+ identity instead of a hardcoded `superuser`, and gives the audit chain durable
+ storage.
+
+- **The node refuses to start when TLS is on but gRPC would answer in cleartext
+ off-loopback** ([#229](https://github.com/xerj-org/xerj/issues/229)).
+ `tls.enabled = true` encrypted two of the three data-plane listeners; the
+ third never participated, because the gRPC server is built without tonic's
+ `tls` feature. Nothing surfaced it — gRPC clients connected and worked
+ identically either way — so an operator who enabled TLS and bound a network
+ interface got no error, no failed handshake and no symptom, while API keys and
+ document bodies crossed the network in the clear. Auth was never the gap;
+ confidentiality was. Boot now fails closed unless
+ `tls.allow_insecure_grpc_h2c` says otherwise.
+
### Fixed
+- **`multi_match` returns the same hit set before and after `_flush`**
+ ([#218](https://github.com/xerj-org/xerj/issues/218)). The memtable and
+ segment paths disagreed on multi-token semantics, so flushing an index could
+ change which documents a query matched — verified live on a one-document
+ index where an OR query returned 0 hits before `_flush` and 1 after, with ES
+ returning 1 throughout.
+
+- **Dynamic string fields get Elasticsearch's default `.keyword` multi-field**
+ ([#209](https://github.com/xerj-org/xerj/issues/209)). Dynamic mapping
+ inferred every string as bare `text`, so `GET _mapping` never showed a
+ `.keyword` sub-field and `_field_caps` never listed one. Anyone arriving from
+ Elasticsearch — or any tool that reads `_field_caps` for field discovery,
+ Kibana included — could not see that `category.keyword` existed.
+
+- **`min_score` is honoured at `size:0`, and an external scalar N keeps its
+ ghosts** ([#193](https://github.com/xerj-org/xerj/issues/193)). A
+ `size:0 + min_score` body never materialised a hit, so the threshold had
+ nothing to filter and `hits.total` reported the raw match count — measured at
+ **65** where the same query at `size:100` counted **5**. The tie-break
+ comparator item from that issue is deliberately not included here; it belongs
+ with [#191](https://github.com/xerj-org/xerj/issues/191).
+
+- **Index blocks can be removed, `read_only_allow_delete` means what its name
+ says, and a blocked write answers 403 rather than 500.** Three compounding
+ defects: `PUT /_settings` wrote only the display-side map while enforcement
+ read the per-index `settings.json`, so a block was acknowledged, invisible to
+ `GET /_settings`, and — with `_block` registered PUT-only — removable only by
+ restarting with a hand-edited file. The one block that fires automatically did
+ the inverse of its name, and clients hitting any of it were told the server
+ had faulted.
+
+- **`llms.txt` no longer tells agents six things the binary does not do.** The
+ "Running an index for a human" section is executed, not just read: an agent
+ follows it literally and then reports the result to its user. Corrected: the
+ job-size line is not the first line printed; `--dry-run` does not write
+ *nothing* (it creates and locks the state dir and appends a journal record);
+ a dry run reuses a frozen plan instead of re-costing the job, so it is not a
+ floor for the real run; exit `1` is the catch-all for every error, not
+ "endpoint unreachable"; `status` needs the same explicit `--state-dir` the run
+ used; and `--workers`/`--pdf-workers` bound the client-side extractor, not the
+ server (with `--pdf-workers` capped to 1–4).
+
- **A failed index can now be inspected, deleted and retried without stopping
the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index
directory that refused to open at boot was recorded in a map that only
@@ -143,6 +232,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **`autoindex` reports real progress, an honest percent and an ETA — no run is
+ silent** ([#241](https://github.com/xerj-org/xerj/issues/241)). Between 47%
+ and 64% of a typical run produced no output at all, which is the worst
+ possible behaviour for the two audiences that matter: a person watching a
+ laptop they cannot use, and an AI agent that has gone quiet on that person's
+ behalf. Progress now covers phase, items done and total, rate and ETA, in both
+ modes — a terminal gets a live redrawn line, a pipe gets periodic structured
+ `xerj-progress` lines an agent can parse, and every run ends with a single
+ `xerj-done` summary. The percent is bytes-based so a large file cannot make it
+ stall at 99%, the first ETA is withheld for five seconds and labelled `rough`
+ until there is enough evidence for it, and the line names the file currently
+ being waited on. `--quiet` still produces exactly nothing on either stream,
+ and `--progress none` is honoured everywhere — the resource policy's own
+ startup notes were moved onto the same surface so they cannot leak into
+ `--progress json`'s single parseable stream.
+
- **`/get` and `/get.ps1` are counted.** `functions/get.js` (a Cloudflare Pages
Function) serves the installer straight out of `landing/get`, unchanged, and
records the request. The install path had never had a counter of any kind —
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 0c98c057d..3b18ec0b4 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -6378,7 +6378,7 @@ dependencies = [
[[package]]
name = "xerj-ai"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"candle-core",
@@ -6399,7 +6399,7 @@ dependencies = [
[[package]]
name = "xerj-api"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"async-trait",
@@ -6429,7 +6429,7 @@ dependencies = [
[[package]]
name = "xerj-autoindex"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"chrono",
@@ -6471,7 +6471,7 @@ dependencies = [
[[package]]
name = "xerj-cluster"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"async-trait",
@@ -6496,7 +6496,7 @@ dependencies = [
[[package]]
name = "xerj-common"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"byteorder",
@@ -6515,7 +6515,7 @@ dependencies = [
[[package]]
name = "xerj-compress"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"byteorder",
"bytes",
@@ -6528,7 +6528,7 @@ dependencies = [
[[package]]
name = "xerj-console-api"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"arc-swap",
@@ -6564,7 +6564,7 @@ dependencies = [
[[package]]
name = "xerj-engine"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"arc-swap",
@@ -6603,7 +6603,7 @@ dependencies = [
[[package]]
name = "xerj-fts"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"bitpacking",
@@ -6629,7 +6629,7 @@ dependencies = [
[[package]]
name = "xerj-logs"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"byteorder",
@@ -6647,7 +6647,7 @@ dependencies = [
[[package]]
name = "xerj-mcp"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"reqwest",
@@ -6657,7 +6657,7 @@ dependencies = [
[[package]]
name = "xerj-query"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"base64 0.22.1",
@@ -6672,7 +6672,7 @@ dependencies = [
[[package]]
name = "xerj-server"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"axum",
@@ -6718,7 +6718,7 @@ dependencies = [
[[package]]
name = "xerj-storage"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"arc-swap",
@@ -6748,7 +6748,7 @@ dependencies = [
[[package]]
name = "xerj-vector"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"byteorder",
@@ -6766,7 +6766,7 @@ dependencies = [
[[package]]
name = "xerj-wasm"
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
dependencies = [
"anyhow",
"bytes",
diff --git a/engine/Cargo.toml b/engine/Cargo.toml
index ffe748be7..bbf394233 100644
--- a/engine/Cargo.toml
+++ b/engine/Cargo.toml
@@ -21,7 +21,7 @@ members = [
]
[workspace.package]
-version = "1.0.0-rc.13"
+version = "1.0.0-rc.14"
edition = "2021"
license = "Apache-2.0"
authors = ["xerj Contributors"]