Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge origin/main into feat/ux-estimate-gate

xerj-orgf0c19ea4cb23
diff --cc CHANGELOG.md
index cc9bce3a0,23cc2c8fc..8f89724ff
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,74 -7,138 +7,204 @@@ and this project adheres to [Semantic V
## [Unreleased]
+ ### Fixed
+
+ - **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detaches
+ the index, and the detach survives a restart** (#282, ported from #262). The
+ null previously fell through a string-only settings reader, so the operator
+ got `200 acknowledged` while the index stayed attached — to a policy whose
+ delete phase then destroyed it. A detach now removes the execution cursor,
+ writes a persisted tombstone (`ism_managed_indices.json` grew a
+ `managed`/`detached` envelope; the old bare-map file is still read), and
+ scrubs the stale `index.lifecycle.name` from the stored settings.
+ - **The lifecycle delete action gained #262's safety rails.** It now refuses —
+ visibly, in `explain`, never silently — to delete a dot-prefixed internal
+ index (`.ds-*` backing indices exempt), a data stream's current write index,
+ or an index whose age cannot be established from its execution cursor.
+ - **An ILM policy naming an action the engine cannot execute is refused at PUT
+ time with the action named** (400), instead of being stored with the action
+ silently dropped — the accepted-and-ignored class (#204). Executable ILM
+ actions are `rollover`, `delete`, `readonly`.
+
### Added
+ - **`GET /_ilm/status`, `POST /_ilm/start`, `POST /_ilm/stop`** (#282): the
+ operator can halt lifecycle execution without stopping the node; a stopped
+ engine's tick touches nothing. The flag is in-memory — a restart resumes
+ execution — while the per-index detach tombstone is the durable stop.
+ - **`POST /{index}/_ilm/remove`** (#282): ES's own detach verb; goes through
+ the same tombstoned detach path as the settings-null route. A literal name
+ that is not an index answers 404 rather than writing a tombstone for a
+ ghost.
+
+ - **A crafted filename can no longer forge records on the agent progress
+ stream** (the first rc.15 known issue below). Every externally-controlled
+ string — in-flight paths, the paths and error text interpolated into human
+ notes, the terminal line's reason — is stripped of control characters, bidi
+ overrides and zero-width characters and bounded in length before it reaches
+ any progress surface. Measured on a corpus holding crafted names: a run that
+ previously emitted a forged `xerj-done ok=true exit=0 reason=completed`
+ 2.2 s ahead of its real terminal line now emits one terminal line and no
+ control characters at all. The same sanitisation covers the walker's
+ "skipping unreadable entry" warning, which renders a path to the same stderr.
+ - **`--progress json` paces the bar like `--progress plain`.** The `bar` field
+ bypassed the spacing slot entirely: measured at `--progress-interval 1`, 178
+ of 178 ticks carried a bar against 18 bars on the plain surface. It is now a
+ string on exactly the ticks that owe a bar and `null` in between — 26 of 320
+ ticks on the same corpus, against 16 on plain.
+ - **The bar spacing floor is the 15 s the documentation states.** A half-tick
+ tolerance made the enforced floor `interval/2` shorter — 12.5 s at the
+ shipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phase
+ gaps under 15 s (min 10.0 s) before, none after (min 19.3 s).
+
+- **`autoindex` estimates the job on the user's own machine and hands the
+ decision back before it takes their laptop.** Phase A already reads and
+ parses every file to sniff and sample it, so it now *times* that work per
+ format family and turns it into a range with its basis printed next to it —
+ `code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s` —
+ rather than a constant calibrated on someone else's hardware. A family is
+ priced only from files phase A **provably** read end to end (whole-file
+ parsers always; streaming parsers only when the file was under the sampling
+ byte cap *and* stopped short of the record cap; never `sqlite`, never
+ gzip), and everything else is named under `unmeasured_families` with its
+ bytes instead of being priced at another family's rate. If nothing could be
+ measured there is no number at all and no gate. The two ends are the
+ classical list-scheduling bounds (Graham, 1969) over the phase-B worker
+ count that #240's resource policy chose.
+ The number is a **floor**, labelled as one on every surface: it covers
+ client-side extraction only, because measuring the server, the network or
+ embedding would mean writing to the index the estimate exists to ask
+ permission for. Measured on a 68 MB source tree the floor was 0.1 s against
+ a real 8.9 s run; on a 793 MB one, 64.2 s against ~350 s. The gate therefore
+ under-asks and never over-asks, and says so where a reader would otherwise
+ mistake silence for a promise.
+- **`--max-minutes` (default 10) stops the run and asks instead of deciding.**
+ Past the threshold with no answer, autoindex indexes nothing, writes a JSON
+ decision request to stdout and exits **4** — a code of its own, because exit
+ 1 is already the catch-all for every real failure and an agent must be able
+ to tell "choose something" from "your endpoint is down". The payload carries
+ the estimate and its basis, file/byte counts, the per-band work order, the
+ heaviest directories with real byte counts (flagged when they match the
+ vendored/generated rule), and four options: `proceed`, `fast`, `narrower`,
+ `cancel`. Answer with `--approve <id>` (`--yes` = proceed), which skips the
+ gate; `--max-minutes 0` disables it. A person at a terminal gets the same
+ facts as prose plus a prompt; a piped or agent-driven run is **never**
+ blocked on stdin, and an unanswerable prompt is a cancel, never consent.
+ `--approve fast` really applies `--no-semantic --no-graph`, and `--approve
+ narrower` is refused with instructions rather than accepted and ignored
+ (#204). The `fast` option states no speed-up factor: it reports the datasets
+ and file count it changes and says plainly that this run did not measure the
+ factor.
+ The gate governs the phase-B route. An **incremental reconcile of an already
+ committed generation** (a `--no-graph` re-run over a folder that already has
+ one) processes only what changed and publishes from a sealed snapshot, so
+ `--max-minutes` does not apply there — and that route now *says so* on
+ stderr rather than accepting the flag and quietly not honouring it.
+
+### Changed
+
+- **`autoindex` indexes what matters first.** Phase B's queue was sorted by
+ size alone — right about scheduling, silent about value, so a user who
+ stopped early got whatever was largest, which on a source tree is
+ `node_modules`. Work is now ordered by value band (source and documents →
+ configuration → structured data → logs and line files → vendored, generated
+ and minified paths), with the old biggest-first rule kept *inside* each band
+ so a large file still runs alongside its band instead of becoming the tail;
+ a single-worker run goes smallest-first, where there is no tail to hide in.
+ One exception keeps the new order from costing wall clock: a file whose own
+ extraction outlasts everything ranked above it (`size × workers > the rest`)
+ starts first regardless of band. The bands, their file/byte counts and the
+ reason each sits where it does are printed with the plan and included in the
+ decision payload as `priority_order` — an unexplained order is
+ indistinguishable from an arbitrary one. Verified live on a 28 MB mixed
+ corpus at two workers: the first second of phase B was spent on
+ `src/mod_*.rs`, and the 1.7 MB `node_modules/**` files — the largest in the
+ corpus, and therefore *first* under the old rule — drained last.
+ Bytes-based progress and its percent are unaffected: reordering does not
+ change the denominator.
+
+ ## [1.0.0-rc.15] - 2026-08-10
+
+ ### Known issues in this release
+
+ Both were found by adversarial review *after* their pull requests merged, so
+ they are present in these binaries. Fixes are in flight for rc.16.
+
+ - **A crafted filename can forge records on the agent progress stream.** The
+ in-flight path is rendered onto the progress surface without control-character
+ sanitisation, so a filename containing a newline can inject something that
+ looks like a genuine `xerj-progress` or `xerj-done` record — including a false
+ `ok=true` completion — into the stream this release tells AI agents to parse.
+ Indexing a repository whose filenames you do not control is enough to trigger
+ it. Until the fix lands, do not treat that stream as trustworthy when the
+ corpus is untrusted.
+ - **An outer repository's `.gitignore` judges files inside a nested checkout.**
+ The ignore lookup walks every layer with no repository-boundary stop, unlike
+ git itself, so a vendored or submoduled repository is filtered by rules that
+ have no authority over it. Files you expect to be indexed may be skipped.
+ `--no-ignore` bypasses it.
+
+ Also unfixed and worth knowing: `--progress json` does not pace the bar (a JSON
+ consumer sees roughly ten times as many bar lines as a plain consumer), and the
+ documented "at most one bar per 15 s" floor is really 12.5 s.
+
+
+ ### Added
+
+ - **Index lifecycle policies are executed, not just stored**
+ ([#199](https://github.com/xerj-org/xerj/issues/199), contributed by
+ @Vinz2168). A single internal engine modeled on OpenSearch's ISM state machine
+ — named states, ordered actions, ordered transitions — exposed through two
+ REST surfaces: native ISM at `_plugins/_ism/*` and the Elasticsearch-shaped
+ `_ilm/*`. `spawn_lifecycle_manager()` runs the tick on
+ `lifecycle.tick_interval_secs` (default 300s, matching OpenSearch ISM's own
+ job interval), and the managed-index cursor survives restart. Retention now
+ actually deletes and rolls over instead of being acknowledged and forgotten.
+ One documented divergence: `min_age` is measured from when an index entered
+ its current state rather than from rollover time, so a policy that rolls over
+ *and* has a downstream phase meant to be measured from the rollover will
+ advance on a different clock than Elasticsearch uses.
+
+ - **`xerj autoindex` honours `.gitignore` and `.xerjignore`**
+ ([#276](https://github.com/xerj-org/xerj/issues/276)). Nested ignore files and
+ negation patterns follow git's own precedence, with build-output defaults
+ (`node_modules`, `vendor`, `target`, `dist`, `build`, `.venv`, `__pycache__`)
+ on top; `--no-ignore` and `--no-default-ignores` turn them off. Pointing xerj
+ directly at an ignored path still indexes it — an explicit instruction beats a
+ rule the user did not write for this purpose. `--dry-run` reports what was
+ skipped and by which rule, because a user whose files did not appear needs to
+ know why. Measured on this repository: 274,826 files walked in 1.18s becomes
+ 1,385 in 0.25s.
+
+ - **Progress is relayed to an AI agent as a drawn bar.** The stream surface
+ carries a rendered bar alongside its machine-readable `key=value` fields, so
+ an agent driving `xerj autoindex` on someone's machine can show a real
+ progress line rather than going silent for minutes. `--progress json` keeps a
+ single parseable stream; `--quiet` still writes nothing on either.
+
+ ### Fixed
+
+ - **Tied scores have one total order — a bounded page is a prefix of the full
+ page** ([#191](https://github.com/xerj-org/xerj/issues/191)). A `size:5` page
+ and a `size:1000` page disagreed about which tied document came back. The
+ memtable's bounded candidates are now ranked by the same page key as the
+ segment path, so the two agree. The remaining `constant_score` and post-sort
+ re-sort cases are tracked separately in
+ [#270](https://github.com/xerj-org/xerj/issues/270).
+
+ - **`date_histogram` no longer aborts the process on a multi-byte `time_zone`**
+ ([#272](https://github.com/xerj-org/xerj/issues/272), contributed by
+ @Vinz2168). `aggs::parse_time_zone_offset` sliced a `&str` on a byte index
+ that could land mid-character, and with `panic=abort` that took the whole node
+ down — an unauthenticated request was enough.
+
+ - **The legacy path-parameter form of scroll continuation works again**
+ (contributed by @Vinz2168), and **`_cat/*?format=json` plus `_data_stream/**`
+ wildcards** are supported (also @Vinz2168) — both are shapes real
+ Elasticsearch clients send.
+
+ ### Changed
+
- **`autoindex` parses each PDF once per fresh run instead of twice.** Phase A
already paid a *complete* parse for every PDF — `extract::extract` routes
`Family::Pdf` straight to the isolated worker and drops the sampling limit,
diff --cc engine/crates/xerj-autoindex/src/cli.rs
index e5217dbe1,88cfab672..27cb40c8d
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@@ -1,6 -1,6 +1,7 @@@
//! Hand-rolled arg parser (house style of xerj-server — no clap).
+use crate::gate::{Approval, DEFAULT_MAX_MINUTES};
+ use crate::ignore_rules::IgnoreOptions;
use crate::progress::ProgressMode;
use std::path::PathBuf;
use std::time::Duration;
@@@ -287,8 -260,9 +326,9 @@@ pub fn help_text() -> String
2 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\
refused unsafe state transition\n",
fresh_help = FRESH_HELP,
- resume_policy_help = RESUME_POLICY_HELP
+ resume_policy_help = RESUME_POLICY_HELP,
+ defaults = crate::ignore_rules::DEFAULT_IGNORE_PATTERNS.join(" ")
- );
+ )
}
pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
@@@ -541,29 -483,20 +600,42 @@@
);
}
+ // `--approve fast` is not a hint: it is the answer "index everything, but
+ // without the two expensive features", and the run has to actually apply
+ // them. Accepting the word and indexing semantically anyway is precisely
+ // the accepted-and-silently-ignored class from #204.
+ if approve == Some(Approval::Fast) {
+ no_semantic = true;
+ no_graph = true;
+ }
+ if approve == Some(Approval::Cancel) && dry_run {
+ return Err(
+ "--approve cancel and --dry-run contradict each other: a dry run already indexes \
+ nothing. Drop one of the two"
+ .into(),
+ );
+ }
+
+ // `map` reads the catalog off the server and `status` reads the local
+ // journal; neither walks a filesystem, so an ignore flag on either cannot
+ // change one byte of the output. Measured before this check existed:
+ // `xerj autoindex map --no-ignore` was byte-identical to `xerj autoindex
+ // map`. Accepting it was the #204 accept-and-ignore class (#279).
+ let ignore_flags_used: Vec<&str> = [
+ ("--no-ignore", no_ignore),
+ ("--no-default-ignores", no_default_ignores),
+ ]
+ .into_iter()
+ .filter_map(|(name, used)| used.then_some(name))
+ .collect();
+
match (sub.as_deref(), folder) {
+ (Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {
+ Err(format!(
+ "--max-minutes/--approve/--yes apply only to indexing, not `autoindex {}`",
+ sub.as_deref().unwrap_or_default()
+ ))
+ }
(Some("map"), _) | (Some("status"), _) if progress_explicit => Err(format!(
"--progress/--progress-interval apply only to indexing, not `autoindex {}`",
sub.as_deref().unwrap_or_default()
diff --cc engine/crates/xerj-autoindex/src/lib.rs
index f98d8f654,0e57afdd5..1bd3673a2
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@@ -18,8 -16,8 +18,9 @@@ mod generation_catalog
#[cfg(test)]
mod generation_catalog_http_tests;
pub mod ids;
+ pub mod ignore_rules;
pub mod infer;
+pub mod order;
pub mod pool;
pub mod progress;
mod reconcile_plan;