Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into unblock-258-local

xerj-orgcf301e3c0867
diff --cc CHANGELOG.md
index b0671888b,e380144d1..0277b7723
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -9,138 -9,26 +9,158 @@@ and this project adheres to [Semantic V
### Changed
+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot
+ honour, instead of accepting it and analysing differently**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an
+ existing index is opened, where a hard failure would brick it), so an
+ unresolvable tokenizer silently became `standard` and an unresolvable token
+ filter was dropped — the index was created, `GET /_settings` echoed the block
+ back, and the field was analysed in a way nobody asked for. Index creation
+ now validates the block and answers `400` with the exact constructs it cannot
+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused
+ (it only ever changed what `GET /_settings` echoed; the registry is built
+ once, at create/open, and there is no rebuild path — Elasticsearch also
+ refuses analysis updates on an open index).
+
+ **This rejects settings blocks that XERJ used to accept with a `200`, and the
+ ES-YAML conformance suite does not cover them — CI being green is not
+ evidence that your index still creates.** What is honoured:
+
+ | | honoured (still `200`) |
+ |---|---|
+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |
+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |
+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |
+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |
+ | `analysis.normalizer` | nothing — never built |
+
+ Anything else in those positions is now a `400`. There is one important
+ exception to read first: XERJ resolves a declared filter or tokenizer by
+ **name**, not by `type`, so a declaration whose *name* is a built-in is still
+ honoured whatever its type says. Measured, all still `200`:
+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,
+ `{"lowercase": {"type": "lowercase"}}`,
+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`
+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which
+ has a regression test pinning it. Naming a built-in analyzer on a field
+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.
+
+ What now fails is a declaration under a name that does NOT resolve — which is
+ the usual ES habit of naming filters after what they do in *your* index.
+ Measured blocks that returned `200` in rc.14 and return `400` from this
+ release:
+
+ - a `keyword` field's `normalizer` (any);
+ - a `char_filter` (any, including `html_strip` and `mapping`), declared
+ either under `analysis.char_filter` or as an analyzer's `char_filter`;
+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,
+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,
+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,
+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,
+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,
+ which never lowercased anything, because the analyzer referencing
+ `my_lowercase` found no such filter and dropped it;
+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,
+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,
+ `letter`, `lowercase` or `simple_pattern`;
+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,
+ `fingerprint`, or a language other than `english`;
+ - a `synonym` filter whose `synonyms` is a bare string rather than an array
+ (it built a filter with zero rules).
+
+ **Upgrading:** there is no opt-out flag. Either remove the analysis block and
+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only
+ the constructs in the table. The `400` names every offending construct in one
+ response, so one round-trip is enough to see the whole list. **Two paths are
+ not covered and still accept these blocks silently: an index template's
+ `template.settings.analysis` is dropped wholesale before the gate ever sees
+ it, and option-level divergence inside a supported type (`synonyms_path`,
+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**
+
+ - **`autoindex` parses each PDF once per fresh run instead of twice.** Phase A
+ already paid a *complete* parse for every PDF — `extract::extract` routes
+ `Family::Pdf` straight to the isolated worker and drops the sampling limit,
+ so only delivery ever stopped early — and phase B then spawned the worker
+ again for the same bytes. Phase A now retains each validated worker response
+ in an anonymous temporary file under `--state-dir` and phase B replays it.
+ Reuse is an optional accelerator, never correctness-critical state: it is
+ bounded to 384 MiB of retained-plus-in-flight bytes and to an artifact-handle
+ cap derived from live `RLIMIT_NOFILE` and open-descriptor counts, both
+ re-probed at every admission; a 4 GiB-or-half-free filesystem floor is
+ reserved for the journal and phase-B staging first; and every artifact is
+ verified (physical length, content digest, JSON decode, worker-protocol
+ identity) before a single record is published. Anything that cannot be
+ measured conservatively declines and reparses — which is why the optimization
+ is **Linux-only** today: other platforms have no live descriptor evidence.
+ A restart has a frozen plan but no trusted handle, so it parses again; the
+ spool is deliberately not journal state. `--json` gains a
+ `pdf_extraction_reuse` block so the behaviour is observable rather than
+ inferred from timing. Original work by Leonid Bugaev (@buger).
+
+### Fixed
+
+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and
+ every write path enforces it**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only
+ warn-log a compile failure, so a pipeline that could never run was
+ acknowledged and echoed back by `GET`. It now compiles first, with three
+ outcomes: compiled → stored and acknowledged; a processor config we refuse →
+ `400`; a processor or option this build cannot honour → accepted (as
+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses
+ rather than storing an untransformed document.
+
+ Defects this surfaced, each of which answered `200`/`201` and did something
+ smaller than asked: **every** ES `rename` processor failed to compile (the
+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and
+ renamed nothing; ES `append` was executed as `set`, replacing the field
+ instead of extending it, and disagreeing with `_simulate`, which implemented
+ a real append; `set.override` defaulted to `false` where Elasticsearch
+ defaults it to `true`; an unknown `pii_redaction` type built an empty pattern
+ set and redacted nothing; an unknown `grok` pattern name silently became the
+ catch-all; an unknown `convert` target left the field unconverted per
+ document; array-typed keys given as a bare string were dropped and the stage
+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the
+ definition while leaving the compiled pipeline live under the same name.
+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`
+ rather than being refused.
+
+ Two more of the same, at the definition's own edges: a **pipeline-level**
+ `on_failure` block (the top-level body key, not the processor-level one) was
+ accepted, echoed by `GET`, and never run — the pipeline is now recorded
+ unrunnable, exactly as its processor-level twin already was; and `set` with
+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted
+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:
+ missing value`, which told the caller their valid ES pipeline was malformed.
+
+ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**
+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,
+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`
+ previously read the instruction and dropped it — a redaction pipeline that
+ the single-document endpoint refused was ignored in bulk, and the document
+ was stored verbatim under `errors: false`. All four now run the pipeline, and
+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the
+ name before touching a single document, so a bad pipeline fails the request
+ instead of half of it.
+
+ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`
+ copied at most 10,000 documents and swallowed every per-document write error
+ before answering `{"acknowledged": true}` (now full keyset paging, and any
+ failed write — or a failed source flush — fails the request); and a failed
+ `_bulk` item no longer reports `"result": "deleted"` alongside its own
+ `status: 400`, which claimed a deletion that never happened.
+
+ **Known gaps, stated rather than closed:** `_simulate` interprets the stored
+ ES body directly and covers processors the compiled path does not, so a
+ pipeline recorded as unrunnable can still show a transformation under
+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`
+ returns the ES body with no runnability signal, so an operator only learns at
+ write time; a write addressed to an **alias** does not pick up the backing
+ index's `default_pipeline`; and `_update_by_query` reports a `drop`
+ processor as a per-document failure, because it updates in place and has no
+ delete path.
+
## [1.0.0-rc.14] - 2026-08-10
### Changed