AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge pull request #281 from xerj-org/feat/ux-estimate-gate
xerj-orgfe5148f45bfe
diff --cc CHANGELOG.mdindex 825563493,8f89724ff..fddf2bbe8--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -184,111 -56,73 +184,177 @@@ and this project adheres to [Semantic Vtolerance made the enforced floor `interval/2` shorter — 12.5 s at theshipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phasegaps under 15 s (min 10.0 s) before, none after (min 19.3 s).+- **`/{index}/_refresh` reports segment-publication failures instead of+ claiming every shard succeeded.** The endpoint previously discarded every+ error returned by the synchronous engine flush and always answered HTTP 200+ with `successful == total`. It now includes the index, status, and underlying+ reason in `_shards.failures`, attempts every resolved index, and returns the+ first failed shard's HTTP status as Elasticsearch 8.13.4 does for refresh.++### Changed++- **`merge.strategy = "log_structured"` is now refused at startup instead of+ silently running the size-tiered policy**+ ([#207](https://github.com/xerj-org/xerj/issues/207)). Nothing in the tree+ ever read `merge.strategy`; `run_merge_once` always builds a+ `SizeTieredMergePolicy`. An operator who chose a levelled policy for its read+ amplification got the other one, quietly. This follows the `storage.backend`+ and `vector.default_quantization` guards already in `Config::validate`.++- **The three dormant `[merge]` settings now say so, in the code, in the shipped+ TOML, on the docs site, and in the log at startup**+ ([#207](https://github.com/xerj-org/xerj/issues/207)). The issue reported+ merge I/O rate limiting as "claimed but dormant" and it is: the `RateLimiter`+ that honours `io_rate_mb_per_sec` is wired only into `xerj-storage`'s legacy+ `MergeExecutor`, which the engine never constructs. `min_segments` and+ `max_concurrent` are unread too — the real per-tier trigger is+ `min_merge_count`, and merge parallelism comes from `XERJ_MERGE_PARALLELISM`.+ `MergeConfig::dormant_overrides` names any of the three the operator has moved+ off its default and `xerj-server` logs each at WARN, so the operator throttling+ merges to protect query latency finds out instead of guessing. They stay+ accepted rather than becoming hard errors because the wrong value costs+ latency, not data, and `io_rate_mb_per_sec = 100` has shipped in+ `xerj.default.toml` since v0.1. Wiring a real throttle into `run_merge_once`+ is the fix that makes this list shorter.++- **`engine/xerj.default.toml` disagreed with the defaults it claims to+ document, in four places.** `max_segment_mb` said 5120 against a real 8192,+ `wal_max_size_mb` 512 against 1024, `flush_size_mb` 256 against 512, and+ `default_quantization` `"scalar8"` against `"none"` — so `cp xerj.default.toml+ xerj.toml`, the documented first step, silently changed four engine behaviours+ including turning on 8-bit vector quantization. The file is corrected and+ `shipped_default_config_documents_the_real_defaults` now diffs every leaf key+ against `Config::default()`, so it cannot drift again.++- **The docs site had drifted the same way, and now has the same guard.**+ `landing/docs/config.html` shipped `wal_max_size_mb = 512`,+ `flush_size_mb = 256` and `default_quantization = "scalar8"` in its+ copy-pasteable `[storage]` and `[vector]` blocks while its own DEFAULT+ table — thirty lines up, on the same page — said 1024, 512 and `"none"`;+ `landing/docs/storage.html` said the WAL rolls at 512 MiB;+ `landing/docs/vectors.html` called `scalar8` "the default" and listed a+ `scalar4` mode that no config or mapping value can reach; and+ `engine/README.md` put `flush_size_mb` at 256 MiB. All four are corrected,+ along with the docs search-index blob duplicated across 44 landing pages,+ which offered `scalar4` as a selectable quantization mode.+ A second guard, `the_docs_site_config_page_agrees_with_the_real_defaults`,+ now diffs *both* halves of that page — every DEFAULT cell in the reference+ table and every assignment in the example blocks — against+ `Config::default()`. An example that deliberately differs (the blocks whose+ point is switching TLS or cluster mode on) has to carry `# not a default` on+ the line, so the reader is told as well as the test.+ - **Known gap, found while correcting the `scalar4` prose and disclosed+ rather than quietly fixed:** the guard reads `config.html` against+ `Config::default()`, so it covers the *config* half only. Nothing checks a+ *mapping*, and a mapping is not validated either —+ `es_compat.rs` matches `"scalar8" | "int8" | "none"` on a `dense_vector`+ field's `quantization` and falls through `_ => {}`, so+ `"quantization": "scalar4"` (or `"binary"`, or a typo like `"sq8"`) is+ accepted with a 200, echoed back verbatim by `GET /_mapping`, and ignored+ — the field is stored at full-precision f32 while the mapping says it is+ quantized. Measured through the ES-compat router at this commit. It is+ pre-existing and outside this diff; it is another instance of the+ accepted-and-ignored class in+ [#204](https://github.com/xerj-org/xerj/issues/204), is filed with the+ repro and a proposed 400 as+ [#275](https://github.com/xerj-org/xerj/issues/275), and+ `landing/docs/vectors.html` now says so on the page instead of leaving+ "not a mode you can select" to imply the value is rejected.++- **Four other claims from [#207](https://github.com/xerj-org/xerj/issues/207)+ now match the source.**+ - *Settings count.* `config.rs` said 38 in its header and 56 twenty lines+ down while the count test asserted 60 — and the test asserted it by+ re-adding a hardcoded sum (`5 + 2 + 3 + … - 1 == 38`), an identity that+ could not fail whatever `Config` held. There was a second copy of the same+ non-test in `config.rs` itself (`count_user_facing_settings`, `61 == 61`).+ Both now serialise `Config::default()` and count leaf keys, section by+ section. The measured figure is **105** (103 when this work was done; the+ rc.14 `server.allow_insecure_network_bind` key made it 104 and #247's+ `lifecycle.tick_interval_secs` makes it 105 — which is the point: it is+ measured, so it moves with the code), and 38 / 50 /+ 56 / 60 / 61 / "<50"+ are corrected in `config.rs`, `xerj-common/src/lib.rs`, `engine/README.md`,+ `xerj.default.toml` and the feedback responses, along with the stale+ per-sub-config annotations (limits 3 → 13, storage 5 → 10, embedding 4 → 19,+ merge 5 → 8, cluster 4 → 5, tls 3 → 4, auth 2 → 3). This closes item 10 of+ `user-feedback/ROADMAP-TO-GA.md`. 105 versus 3,000+ is still the winning+ story, told truthfully.+ - *Wire-compatibility test framing.* `journey_es_migration` was described as+ proving "the same curl commands" work. It calls the Rust engine API+ directly and never makes an HTTP request. The response-shape assertions are+ real and stay; the comment now says what the test does and points at the+ ES-compat YAML conformance suite, which is what actually covers the wire.+ - *OpenAPI spec.* `landing/openapi.json` declared `version: 1.0` while+ specifying 7 routes out of the 200-plus the router registers. It is now+ titled and versioned as a partial spec, and says so.+ - **`autoindex` estimates the job on the user's own machine and hands the+ decision back before it takes their laptop.** Phase A already reads and+ parses every file to sniff and sample it, so it now *times* that work per+ format family and turns it into a range with its basis printed next to it —+ `code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s` —+ rather than a constant calibrated on someone else's hardware. A family is+ priced only from files phase A **provably** read end to end (whole-file+ parsers always; streaming parsers only when the file was under the sampling+ byte cap *and* stopped short of the record cap; never `sqlite`, never+ gzip), and everything else is named under `unmeasured_families` with its+ bytes instead of being priced at another family's rate. If nothing could be+ measured there is no number at all and no gate. The two ends are the+ classical list-scheduling bounds (Graham, 1969) over the phase-B worker+ count that #240's resource policy chose.+ The number is a **floor**, labelled as one on every surface: it covers+ client-side extraction only, because measuring the server, the network or+ embedding would mean writing to the index the estimate exists to ask+ permission for. Measured on a 68 MB source tree the floor was 0.1 s against+ a real 8.9 s run; on a 793 MB one, 64.2 s against ~350 s. The gate therefore+ under-asks and never over-asks, and says so where a reader would otherwise+ mistake silence for a promise.+ - **`--max-minutes` (default 10) stops the run and asks instead of deciding.**+ Past the threshold with no answer, autoindex indexes nothing, writes a JSON+ decision request to stdout and exits **4** — a code of its own, because exit+ 1 is already the catch-all for every real failure and an agent must be able+ to tell "choose something" from "your endpoint is down". The payload carries+ the estimate and its basis, file/byte counts, the per-band work order, the+ heaviest directories with real byte counts (flagged when they match the+ vendored/generated rule), and four options: `proceed`, `fast`, `narrower`,+ `cancel`. Answer with `--approve <id>` (`--yes` = proceed), which skips the+ gate; `--max-minutes 0` disables it. A person at a terminal gets the same+ facts as prose plus a prompt; a piped or agent-driven run is **never**+ blocked on stdin, and an unanswerable prompt is a cancel, never consent.+ `--approve fast` really applies `--no-semantic --no-graph`, and `--approve+ narrower` is refused with instructions rather than accepted and ignored+ (#204). The `fast` option states no speed-up factor: it reports the datasets+ and file count it changes and says plainly that this run did not measure the+ factor.+ The gate governs the phase-B route. An **incremental reconcile of an already+ committed generation** (a `--no-graph` re-run over a folder that already has+ one) processes only what changed and publishes from a sealed snapshot, so+ `--max-minutes` does not apply there — and that route now *says so* on+ stderr rather than accepting the flag and quietly not honouring it.++ ### Changed++ - **`autoindex` indexes what matters first.** Phase B's queue was sorted by+ size alone — right about scheduling, silent about value, so a user who+ stopped early got whatever was largest, which on a source tree is+ `node_modules`. Work is now ordered by value band (source and documents →+ configuration → structured data → logs and line files → vendored, generated+ and minified paths), with the old biggest-first rule kept *inside* each band+ so a large file still runs alongside its band instead of becoming the tail;+ a single-worker run goes smallest-first, where there is no tail to hide in.+ One exception keeps the new order from costing wall clock: a file whose own+ extraction outlasts everything ranked above it (`size × workers > the rest`)+ starts first regardless of band. The bands, their file/byte counts and the+ reason each sits where it does are printed with the plan and included in the+ decision payload as `priority_order` — an unexplained order is+ indistinguishable from an arbitrary one. Verified live on a 28 MB mixed+ corpus at two workers: the first second of phase B was spent on+ `src/mod_*.rs`, and the 1.7 MB `node_modules/**` files — the largest in the+ corpus, and therefore *first* under the old rule — drained last.+ Bytes-based progress and its percent are unaffected: reordering does not+ change the denominator.+## [1.0.0-rc.15] - 2026-08-10### Known issues in this release