Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge pull request #281 from xerj-org/feat/ux-estimate-gate

xerj-orgfe5148f45bfe
diff --cc CHANGELOG.md
index 825563493,8f89724ff..fddf2bbe8
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -184,111 -56,73 +184,177 @@@ and this project adheres to [Semantic V
tolerance made the enforced floor `interval/2` shorter — 12.5 s at the
shipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phase
gaps under 15 s (min 10.0 s) before, none after (min 19.3 s).
+- **`/{index}/_refresh` reports segment-publication failures instead of
+ claiming every shard succeeded.** The endpoint previously discarded every
+ error returned by the synchronous engine flush and always answered HTTP 200
+ with `successful == total`. It now includes the index, status, and underlying
+ reason in `_shards.failures`, attempts every resolved index, and returns the
+ first failed shard's HTTP status as Elasticsearch 8.13.4 does for refresh.
+
+### Changed
+
+- **`merge.strategy = "log_structured"` is now refused at startup instead of
+ silently running the size-tiered policy**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). Nothing in the tree
+ ever read `merge.strategy`; `run_merge_once` always builds a
+ `SizeTieredMergePolicy`. An operator who chose a levelled policy for its read
+ amplification got the other one, quietly. This follows the `storage.backend`
+ and `vector.default_quantization` guards already in `Config::validate`.
+
+- **The three dormant `[merge]` settings now say so, in the code, in the shipped
+ TOML, on the docs site, and in the log at startup**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). The issue reported
+ merge I/O rate limiting as "claimed but dormant" and it is: the `RateLimiter`
+ that honours `io_rate_mb_per_sec` is wired only into `xerj-storage`'s legacy
+ `MergeExecutor`, which the engine never constructs. `min_segments` and
+ `max_concurrent` are unread too — the real per-tier trigger is
+ `min_merge_count`, and merge parallelism comes from `XERJ_MERGE_PARALLELISM`.
+ `MergeConfig::dormant_overrides` names any of the three the operator has moved
+ off its default and `xerj-server` logs each at WARN, so the operator throttling
+ merges to protect query latency finds out instead of guessing. They stay
+ accepted rather than becoming hard errors because the wrong value costs
+ latency, not data, and `io_rate_mb_per_sec = 100` has shipped in
+ `xerj.default.toml` since v0.1. Wiring a real throttle into `run_merge_once`
+ is the fix that makes this list shorter.
+
+- **`engine/xerj.default.toml` disagreed with the defaults it claims to
+ document, in four places.** `max_segment_mb` said 5120 against a real 8192,
+ `wal_max_size_mb` 512 against 1024, `flush_size_mb` 256 against 512, and
+ `default_quantization` `"scalar8"` against `"none"` — so `cp xerj.default.toml
+ xerj.toml`, the documented first step, silently changed four engine behaviours
+ including turning on 8-bit vector quantization. The file is corrected and
+ `shipped_default_config_documents_the_real_defaults` now diffs every leaf key
+ against `Config::default()`, so it cannot drift again.
+
+- **The docs site had drifted the same way, and now has the same guard.**
+ `landing/docs/config.html` shipped `wal_max_size_mb = 512`,
+ `flush_size_mb = 256` and `default_quantization = "scalar8"` in its
+ copy-pasteable `[storage]` and `[vector]` blocks while its own DEFAULT
+ table — thirty lines up, on the same page — said 1024, 512 and `"none"`;
+ `landing/docs/storage.html` said the WAL rolls at 512 MiB;
+ `landing/docs/vectors.html` called `scalar8` "the default" and listed a
+ `scalar4` mode that no config or mapping value can reach; and
+ `engine/README.md` put `flush_size_mb` at 256 MiB. All four are corrected,
+ along with the docs search-index blob duplicated across 44 landing pages,
+ which offered `scalar4` as a selectable quantization mode.
+ A second guard, `the_docs_site_config_page_agrees_with_the_real_defaults`,
+ now diffs *both* halves of that page — every DEFAULT cell in the reference
+ table and every assignment in the example blocks — against
+ `Config::default()`. An example that deliberately differs (the blocks whose
+ point is switching TLS or cluster mode on) has to carry `# not a default` on
+ the line, so the reader is told as well as the test.
+ - **Known gap, found while correcting the `scalar4` prose and disclosed
+ rather than quietly fixed:** the guard reads `config.html` against
+ `Config::default()`, so it covers the *config* half only. Nothing checks a
+ *mapping*, and a mapping is not validated either —
+ `es_compat.rs` matches `"scalar8" | "int8" | "none"` on a `dense_vector`
+ field's `quantization` and falls through `_ => {}`, so
+ `"quantization": "scalar4"` (or `"binary"`, or a typo like `"sq8"`) is
+ accepted with a 200, echoed back verbatim by `GET /_mapping`, and ignored
+ — the field is stored at full-precision f32 while the mapping says it is
+ quantized. Measured through the ES-compat router at this commit. It is
+ pre-existing and outside this diff; it is another instance of the
+ accepted-and-ignored class in
+ [#204](https://github.com/xerj-org/xerj/issues/204), is filed with the
+ repro and a proposed 400 as
+ [#275](https://github.com/xerj-org/xerj/issues/275), and
+ `landing/docs/vectors.html` now says so on the page instead of leaving
+ "not a mode you can select" to imply the value is rejected.
+
+- **Four other claims from [#207](https://github.com/xerj-org/xerj/issues/207)
+ now match the source.**
+ - *Settings count.* `config.rs` said 38 in its header and 56 twenty lines
+ down while the count test asserted 60 — and the test asserted it by
+ re-adding a hardcoded sum (`5 + 2 + 3 + … - 1 == 38`), an identity that
+ could not fail whatever `Config` held. There was a second copy of the same
+ non-test in `config.rs` itself (`count_user_facing_settings`, `61 == 61`).
+ Both now serialise `Config::default()` and count leaf keys, section by
+ section. The measured figure is **105** (103 when this work was done; the
+ rc.14 `server.allow_insecure_network_bind` key made it 104 and #247's
+ `lifecycle.tick_interval_secs` makes it 105 — which is the point: it is
+ measured, so it moves with the code), and 38 / 50 /
+ 56 / 60 / 61 / "<50"
+ are corrected in `config.rs`, `xerj-common/src/lib.rs`, `engine/README.md`,
+ `xerj.default.toml` and the feedback responses, along with the stale
+ per-sub-config annotations (limits 3 → 13, storage 5 → 10, embedding 4 → 19,
+ merge 5 → 8, cluster 4 → 5, tls 3 → 4, auth 2 → 3). This closes item 10 of
+ `user-feedback/ROADMAP-TO-GA.md`. 105 versus 3,000+ is still the winning
+ story, told truthfully.
+ - *Wire-compatibility test framing.* `journey_es_migration` was described as
+ proving "the same curl commands" work. It calls the Rust engine API
+ directly and never makes an HTTP request. The response-shape assertions are
+ real and stay; the comment now says what the test does and points at the
+ ES-compat YAML conformance suite, which is what actually covers the wire.
+ - *OpenAPI spec.* `landing/openapi.json` declared `version: 1.0` while
+ specifying 7 routes out of the 200-plus the router registers. It is now
+ titled and versioned as a partial spec, and says so.
+ - **`autoindex` estimates the job on the user's own machine and hands the
+ decision back before it takes their laptop.** Phase A already reads and
+ parses every file to sniff and sample it, so it now *times* that work per
+ format family and turns it into a range with its basis printed next to it —
+ `code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s` —
+ rather than a constant calibrated on someone else's hardware. A family is
+ priced only from files phase A **provably** read end to end (whole-file
+ parsers always; streaming parsers only when the file was under the sampling
+ byte cap *and* stopped short of the record cap; never `sqlite`, never
+ gzip), and everything else is named under `unmeasured_families` with its
+ bytes instead of being priced at another family's rate. If nothing could be
+ measured there is no number at all and no gate. The two ends are the
+ classical list-scheduling bounds (Graham, 1969) over the phase-B worker
+ count that #240's resource policy chose.
+ The number is a **floor**, labelled as one on every surface: it covers
+ client-side extraction only, because measuring the server, the network or
+ embedding would mean writing to the index the estimate exists to ask
+ permission for. Measured on a 68 MB source tree the floor was 0.1 s against
+ a real 8.9 s run; on a 793 MB one, 64.2 s against ~350 s. The gate therefore
+ under-asks and never over-asks, and says so where a reader would otherwise
+ mistake silence for a promise.
+ - **`--max-minutes` (default 10) stops the run and asks instead of deciding.**
+ Past the threshold with no answer, autoindex indexes nothing, writes a JSON
+ decision request to stdout and exits **4** — a code of its own, because exit
+ 1 is already the catch-all for every real failure and an agent must be able
+ to tell "choose something" from "your endpoint is down". The payload carries
+ the estimate and its basis, file/byte counts, the per-band work order, the
+ heaviest directories with real byte counts (flagged when they match the
+ vendored/generated rule), and four options: `proceed`, `fast`, `narrower`,
+ `cancel`. Answer with `--approve <id>` (`--yes` = proceed), which skips the
+ gate; `--max-minutes 0` disables it. A person at a terminal gets the same
+ facts as prose plus a prompt; a piped or agent-driven run is **never**
+ blocked on stdin, and an unanswerable prompt is a cancel, never consent.
+ `--approve fast` really applies `--no-semantic --no-graph`, and `--approve
+ narrower` is refused with instructions rather than accepted and ignored
+ (#204). The `fast` option states no speed-up factor: it reports the datasets
+ and file count it changes and says plainly that this run did not measure the
+ factor.
+ The gate governs the phase-B route. An **incremental reconcile of an already
+ committed generation** (a `--no-graph` re-run over a folder that already has
+ one) processes only what changed and publishes from a sealed snapshot, so
+ `--max-minutes` does not apply there — and that route now *says so* on
+ stderr rather than accepting the flag and quietly not honouring it.
+
+ ### Changed
+
+ - **`autoindex` indexes what matters first.** Phase B's queue was sorted by
+ size alone — right about scheduling, silent about value, so a user who
+ stopped early got whatever was largest, which on a source tree is
+ `node_modules`. Work is now ordered by value band (source and documents →
+ configuration → structured data → logs and line files → vendored, generated
+ and minified paths), with the old biggest-first rule kept *inside* each band
+ so a large file still runs alongside its band instead of becoming the tail;
+ a single-worker run goes smallest-first, where there is no tail to hide in.
+ One exception keeps the new order from costing wall clock: a file whose own
+ extraction outlasts everything ranked above it (`size × workers > the rest`)
+ starts first regardless of band. The bands, their file/byte counts and the
+ reason each sits where it does are printed with the plan and included in the
+ decision payload as `priority_order` — an unexplained order is
+ indistinguishable from an arbitrary one. Verified live on a 28 MB mixed
+ corpus at two workers: the first second of phase B was spent on
+ `src/mod_*.rs`, and the 1.7 MB `node_modules/**` files — the largest in the
+ corpus, and therefore *first* under the old rule — drained last.
+ Bytes-based progress and its percent are unaffected: reordering does not
+ change the denominator.
+
## [1.0.0-rc.15] - 2026-08-10
### Known issues in this release