AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge origin/main into feat/ux-estimate-gate
xerj-orgf0c19ea4cb23
diff --cc CHANGELOG.mdindex cc9bce3a0,23cc2c8fc..8f89724ff--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,74 -7,138 +7,204 @@@ and this project adheres to [Semantic V## [Unreleased]+ ### Fixed++ - **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detaches+ the index, and the detach survives a restart** (#282, ported from #262). The+ null previously fell through a string-only settings reader, so the operator+ got `200 acknowledged` while the index stayed attached — to a policy whose+ delete phase then destroyed it. A detach now removes the execution cursor,+ writes a persisted tombstone (`ism_managed_indices.json` grew a+ `managed`/`detached` envelope; the old bare-map file is still read), and+ scrubs the stale `index.lifecycle.name` from the stored settings.+ - **The lifecycle delete action gained #262's safety rails.** It now refuses —+ visibly, in `explain`, never silently — to delete a dot-prefixed internal+ index (`.ds-*` backing indices exempt), a data stream's current write index,+ or an index whose age cannot be established from its execution cursor.+ - **An ILM policy naming an action the engine cannot execute is refused at PUT+ time with the action named** (400), instead of being stored with the action+ silently dropped — the accepted-and-ignored class (#204). Executable ILM+ actions are `rollover`, `delete`, `readonly`.+### Added+ - **`GET /_ilm/status`, `POST /_ilm/start`, `POST /_ilm/stop`** (#282): the+ operator can halt lifecycle execution without stopping the node; a stopped+ engine's tick touches nothing. The flag is in-memory — a restart resumes+ execution — while the per-index detach tombstone is the durable stop.+ - **`POST /{index}/_ilm/remove`** (#282): ES's own detach verb; goes through+ the same tombstoned detach path as the settings-null route. A literal name+ that is not an index answers 404 rather than writing a tombstone for a+ ghost.++ - **A crafted filename can no longer forge records on the agent progress+ stream** (the first rc.15 known issue below). Every externally-controlled+ string — in-flight paths, the paths and error text interpolated into human+ notes, the terminal line's reason — is stripped of control characters, bidi+ overrides and zero-width characters and bounded in length before it reaches+ any progress surface. Measured on a corpus holding crafted names: a run that+ previously emitted a forged `xerj-done ok=true exit=0 reason=completed`+ 2.2 s ahead of its real terminal line now emits one terminal line and no+ control characters at all. The same sanitisation covers the walker's+ "skipping unreadable entry" warning, which renders a path to the same stderr.+ - **`--progress json` paces the bar like `--progress plain`.** The `bar` field+ bypassed the spacing slot entirely: measured at `--progress-interval 1`, 178+ of 178 ticks carried a bar against 18 bars on the plain surface. It is now a+ string on exactly the ticks that owe a bar and `null` in between — 26 of 320+ ticks on the same corpus, against 16 on plain.+ - **The bar spacing floor is the 15 s the documentation states.** A half-tick+ tolerance made the enforced floor `interval/2` shorter — 12.5 s at the+ shipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phase+ gaps under 15 s (min 10.0 s) before, none after (min 19.3 s).++- **`autoindex` estimates the job on the user's own machine and hands the+ decision back before it takes their laptop.** Phase A already reads and+ parses every file to sniff and sample it, so it now *times* that work per+ format family and turns it into a range with its basis printed next to it —+ `code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s` —+ rather than a constant calibrated on someone else's hardware. A family is+ priced only from files phase A **provably** read end to end (whole-file+ parsers always; streaming parsers only when the file was under the sampling+ byte cap *and* stopped short of the record cap; never `sqlite`, never+ gzip), and everything else is named under `unmeasured_families` with its+ bytes instead of being priced at another family's rate. If nothing could be+ measured there is no number at all and no gate. The two ends are the+ classical list-scheduling bounds (Graham, 1969) over the phase-B worker+ count that #240's resource policy chose.+ The number is a **floor**, labelled as one on every surface: it covers+ client-side extraction only, because measuring the server, the network or+ embedding would mean writing to the index the estimate exists to ask+ permission for. Measured on a 68 MB source tree the floor was 0.1 s against+ a real 8.9 s run; on a 793 MB one, 64.2 s against ~350 s. The gate therefore+ under-asks and never over-asks, and says so where a reader would otherwise+ mistake silence for a promise.+- **`--max-minutes` (default 10) stops the run and asks instead of deciding.**+ Past the threshold with no answer, autoindex indexes nothing, writes a JSON+ decision request to stdout and exits **4** — a code of its own, because exit+ 1 is already the catch-all for every real failure and an agent must be able+ to tell "choose something" from "your endpoint is down". The payload carries+ the estimate and its basis, file/byte counts, the per-band work order, the+ heaviest directories with real byte counts (flagged when they match the+ vendored/generated rule), and four options: `proceed`, `fast`, `narrower`,+ `cancel`. Answer with `--approve <id>` (`--yes` = proceed), which skips the+ gate; `--max-minutes 0` disables it. A person at a terminal gets the same+ facts as prose plus a prompt; a piped or agent-driven run is **never**+ blocked on stdin, and an unanswerable prompt is a cancel, never consent.+ `--approve fast` really applies `--no-semantic --no-graph`, and `--approve+ narrower` is refused with instructions rather than accepted and ignored+ (#204). The `fast` option states no speed-up factor: it reports the datasets+ and file count it changes and says plainly that this run did not measure the+ factor.+ The gate governs the phase-B route. An **incremental reconcile of an already+ committed generation** (a `--no-graph` re-run over a folder that already has+ one) processes only what changed and publishes from a sealed snapshot, so+ `--max-minutes` does not apply there — and that route now *says so* on+ stderr rather than accepting the flag and quietly not honouring it.++### Changed++- **`autoindex` indexes what matters first.** Phase B's queue was sorted by+ size alone — right about scheduling, silent about value, so a user who+ stopped early got whatever was largest, which on a source tree is+ `node_modules`. Work is now ordered by value band (source and documents →+ configuration → structured data → logs and line files → vendored, generated+ and minified paths), with the old biggest-first rule kept *inside* each band+ so a large file still runs alongside its band instead of becoming the tail;+ a single-worker run goes smallest-first, where there is no tail to hide in.+ One exception keeps the new order from costing wall clock: a file whose own+ extraction outlasts everything ranked above it (`size × workers > the rest`)+ starts first regardless of band. The bands, their file/byte counts and the+ reason each sits where it does are printed with the plan and included in the+ decision payload as `priority_order` — an unexplained order is+ indistinguishable from an arbitrary one. Verified live on a 28 MB mixed+ corpus at two workers: the first second of phase B was spent on+ `src/mod_*.rs`, and the 1.7 MB `node_modules/**` files — the largest in the+ corpus, and therefore *first* under the old rule — drained last.+ Bytes-based progress and its percent are unaffected: reordering does not+ change the denominator.++ ## [1.0.0-rc.15] - 2026-08-10++ ### Known issues in this release++ Both were found by adversarial review *after* their pull requests merged, so+ they are present in these binaries. Fixes are in flight for rc.16.++ - **A crafted filename can forge records on the agent progress stream.** The+ in-flight path is rendered onto the progress surface without control-character+ sanitisation, so a filename containing a newline can inject something that+ looks like a genuine `xerj-progress` or `xerj-done` record — including a false+ `ok=true` completion — into the stream this release tells AI agents to parse.+ Indexing a repository whose filenames you do not control is enough to trigger+ it. Until the fix lands, do not treat that stream as trustworthy when the+ corpus is untrusted.+ - **An outer repository's `.gitignore` judges files inside a nested checkout.**+ The ignore lookup walks every layer with no repository-boundary stop, unlike+ git itself, so a vendored or submoduled repository is filtered by rules that+ have no authority over it. Files you expect to be indexed may be skipped.+ `--no-ignore` bypasses it.++ Also unfixed and worth knowing: `--progress json` does not pace the bar (a JSON+ consumer sees roughly ten times as many bar lines as a plain consumer), and the+ documented "at most one bar per 15 s" floor is really 12.5 s.+++ ### Added++ - **Index lifecycle policies are executed, not just stored**+ ([#199](https://github.com/xerj-org/xerj/issues/199), contributed by+ @Vinz2168). A single internal engine modeled on OpenSearch's ISM state machine+ — named states, ordered actions, ordered transitions — exposed through two+ REST surfaces: native ISM at `_plugins/_ism/*` and the Elasticsearch-shaped+ `_ilm/*`. `spawn_lifecycle_manager()` runs the tick on+ `lifecycle.tick_interval_secs` (default 300s, matching OpenSearch ISM's own+ job interval), and the managed-index cursor survives restart. Retention now+ actually deletes and rolls over instead of being acknowledged and forgotten.+ One documented divergence: `min_age` is measured from when an index entered+ its current state rather than from rollover time, so a policy that rolls over+ *and* has a downstream phase meant to be measured from the rollover will+ advance on a different clock than Elasticsearch uses.++ - **`xerj autoindex` honours `.gitignore` and `.xerjignore`**+ ([#276](https://github.com/xerj-org/xerj/issues/276)). Nested ignore files and+ negation patterns follow git's own precedence, with build-output defaults+ (`node_modules`, `vendor`, `target`, `dist`, `build`, `.venv`, `__pycache__`)+ on top; `--no-ignore` and `--no-default-ignores` turn them off. Pointing xerj+ directly at an ignored path still indexes it — an explicit instruction beats a+ rule the user did not write for this purpose. `--dry-run` reports what was+ skipped and by which rule, because a user whose files did not appear needs to+ know why. Measured on this repository: 274,826 files walked in 1.18s becomes+ 1,385 in 0.25s.++ - **Progress is relayed to an AI agent as a drawn bar.** The stream surface+ carries a rendered bar alongside its machine-readable `key=value` fields, so+ an agent driving `xerj autoindex` on someone's machine can show a real+ progress line rather than going silent for minutes. `--progress json` keeps a+ single parseable stream; `--quiet` still writes nothing on either.++ ### Fixed++ - **Tied scores have one total order — a bounded page is a prefix of the full+ page** ([#191](https://github.com/xerj-org/xerj/issues/191)). A `size:5` page+ and a `size:1000` page disagreed about which tied document came back. The+ memtable's bounded candidates are now ranked by the same page key as the+ segment path, so the two agree. The remaining `constant_score` and post-sort+ re-sort cases are tracked separately in+ [#270](https://github.com/xerj-org/xerj/issues/270).++ - **`date_histogram` no longer aborts the process on a multi-byte `time_zone`**+ ([#272](https://github.com/xerj-org/xerj/issues/272), contributed by+ @Vinz2168). `aggs::parse_time_zone_offset` sliced a `&str` on a byte index+ that could land mid-character, and with `panic=abort` that took the whole node+ down — an unauthenticated request was enough.++ - **The legacy path-parameter form of scroll continuation works again**+ (contributed by @Vinz2168), and **`_cat/*?format=json` plus `_data_stream/**`+ wildcards** are supported (also @Vinz2168) — both are shapes real+ Elasticsearch clients send.++ ### Changed+- **`autoindex` parses each PDF once per fresh run instead of twice.** Phase Aalready paid a *complete* parse for every PDF — `extract::extract` routes`Family::Pdf` straight to the isolated worker and drops the sampling limit,diff --cc engine/crates/xerj-autoindex/src/cli.rsindex e5217dbe1,88cfab672..27cb40c8d--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@@ -1,6 -1,6 +1,7 @@@//! Hand-rolled arg parser (house style of xerj-server — no clap).+use crate::gate::{Approval, DEFAULT_MAX_MINUTES};+ use crate::ignore_rules::IgnoreOptions;use crate::progress::ProgressMode;use std::path::PathBuf;use std::time::Duration;@@@ -287,8 -260,9 +326,9 @@@ pub fn help_text() -> String2 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\refused unsafe state transition\n",fresh_help = FRESH_HELP,- resume_policy_help = RESUME_POLICY_HELP+ resume_policy_help = RESUME_POLICY_HELP,+ defaults = crate::ignore_rules::DEFAULT_IGNORE_PATTERNS.join(" ")- );+ )}pub fn parse(args: Vec<String>) -> Result<Cmd, String> {@@@ -541,29 -483,20 +600,42 @@@);}+ // `--approve fast` is not a hint: it is the answer "index everything, but+ // without the two expensive features", and the run has to actually apply+ // them. Accepting the word and indexing semantically anyway is precisely+ // the accepted-and-silently-ignored class from #204.+ if approve == Some(Approval::Fast) {+ no_semantic = true;+ no_graph = true;+ }+ if approve == Some(Approval::Cancel) && dry_run {+ return Err(+ "--approve cancel and --dry-run contradict each other: a dry run already indexes \+ nothing. Drop one of the two"+ .into(),+ );+ }++ // `map` reads the catalog off the server and `status` reads the local+ // journal; neither walks a filesystem, so an ignore flag on either cannot+ // change one byte of the output. Measured before this check existed:+ // `xerj autoindex map --no-ignore` was byte-identical to `xerj autoindex+ // map`. Accepting it was the #204 accept-and-ignore class (#279).+ let ignore_flags_used: Vec<&str> = [+ ("--no-ignore", no_ignore),+ ("--no-default-ignores", no_default_ignores),+ ]+ .into_iter()+ .filter_map(|(name, used)| used.then_some(name))+ .collect();+match (sub.as_deref(), folder) {+ (Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {+ Err(format!(+ "--max-minutes/--approve/--yes apply only to indexing, not `autoindex {}`",+ sub.as_deref().unwrap_or_default()+ ))+ }(Some("map"), _) | (Some("status"), _) if progress_explicit => Err(format!("--progress/--progress-interval apply only to indexing, not `autoindex {}`",sub.as_deref().unwrap_or_default()diff --cc engine/crates/xerj-autoindex/src/lib.rsindex f98d8f654,0e57afdd5..1bd3673a2--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@@ -18,8 -16,8 +18,9 @@@ mod generation_catalog#[cfg(test)]mod generation_catalog_http_tests;pub mod ids;+ pub mod ignore_rules;pub mod infer;+pub mod order;pub mod pool;pub mod progress;mod reconcile_plan;