AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Changelog
All notable changes to XERJ are documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased
Added
Opt-in hub answer evidence receipts (Refs #1789, #1786): Rust callers can bind exact UTF-8 excerpts to pinned source bytes and export SHA-256/JCS trace reports. MCP ask/def accepts trace identities through
evidence; unavailable source bytes or calibration explicitly abstain and escalate. Default answer and hit wire fields are unchanged.A pure Rust cyber scope guard (Refs #2767, #2769) with exact hostname/IP allowlists, normalized CIDRs, inclusive authorization windows, explicit repository-text taint refusal, and scope echoes. This is a shared planning primitive, not a network scanner or a wired cyber recipe.
Opt-in advisory Claude Code search hints with
xerj init --hookson macOS/Linux (Refs #2642): Grep/Glob and repeated Read can suggest a cited search scoped to the indexed folder and configured loopback node, with a per-session quota.--hooks --removepreserves other hooks and session memory.The same advisory search hints for Gemini CLI with
xerj init --hooks --host geminion macOS/Linux (Refs #2642). It writes anAfterToolgroup to.gemini/settings.json, becauseBeforeToolhas no field that reaches the model, and a millisecondtimeoutas that host expects. Matched tools aregrep_search,search_file_content,globandread_file, anchored so MCP tool names are not caught. Setup is refused when the host has hooks disabled. Quotas, counters and backups are per host, and either host can be configured or removed without touching the other.Opt-in Claude Code session handoff with
xerj init --memoryon macOS/Linux (Refs #2645). SessionEnd queues a background redacting import and lexical autoindex; SessionStart returns bounded, cited historical excerpts from a project memory namespace, with an offline cache.--memory --removeremoves only the owned hooks and retains data. Ordinary init and manual session import are unchanged; Codex lifecycle hooks are not included in this first version. Each queued delivery is isolated: terminal input refusals are retained in a private rejected queue with recovery reasons, while transient failures retry. Memory excerpts and displayed briefs filter Unicode controls and format marks.A second Claude plugin in the xerj marketplace,
xerj-knowledge, for document search rather than code: a skill and/xerj-knowledge:indexand/xerj-knowledge:findcommands that search local folders, notes, PDFs, Office files and mail exports and cite the file and page. Samexerj mcpserver; the existingxerjplugin is unchanged. Untested in Claude Cowork so far. (#2646)A Nix flake:
nix run github:xerj-org/xerjbuilds and runsxerjfrom source,nix buildproduces it, andnix developopens a shell with the Rust toolchain. The package builds the engine and embeds the Console assets with no network access during the build. (#2479)xerj autoindex map --html <FILE>and--md <FILE>export an offline report with datasets, format counts, top symbols, graph file hubs, catalog exclusions and suggested searches.--redact-pathshides file paths and skip reasons. Existing files are never overwritten; ordinary map and JSON output retain their behavior (#2643).xerj autoindex --exclude <glob>leaves matching paths out of the index without editing an ignore file or deleting anything. Repeatable, same glob syntax as--stub— a pattern without/matches a file name at any depth, one with/is anchored at the root. The run reports how many paths it left out (#2483).Offline
xerj sessions import --format claude-memexports a claude-mem store's prompts, observations and session summaries through the existing text redactor (#2874). It readsclaude-mem.dbread-only in one snapshot, including rows still in a running worker's WAL, joins each row to its Claude Code session UUID, and marks the model-written observations and summariesis_summary: true.Offline
xerj sessions import --format mempalaceexports a MemPalace palace's drawers verbatim, with their wing, room and hall, through the existing text redactor (#2874). It reads thesqlite_exactbackend read-only in one snapshot without creating side files on a clean store, skips registry and sentinel rows and closets, decodes MemPalace's escaped text, and refuses a palace on the default Chroma backend.Offline
xerj sessions import --format codexexports a documented subset of one archived Codex rollout through the existing text redactor. It skips tools and reasoning, retains physical source locations and archive event identities, and refuses unsupported formats without publishing output.Corpus Hub manifest
pathsentries may name regular files as well as directories (#2486). A file selection checks out that file without its siblings; symlinks and submodules are refused.MCP
xerj_statusreports node reachability, cluster health and visible indices before a search. It distinguishes a missing autoindex catalog from other indexed data and preserves failed or malformed probes as errors rather than reporting a successful empty node (#2515).Empty autoindex maps now explain that managed corpus slugs and autoindex dataset/index names belong to separate discovery surfaces. CLI diagnostics, MCP empty responses and the
xerj_mapdescription point to corpus discovery without claiming a corpus exists. Missing catalogs remain errors; auth refusals and successful dataset maps retain their behavior (#2412).Public Agent Skills Discovery v0.2.0 index and a documentation-routing skill at
/.well-known/agent-skills/. The index is generated from the published skill's metadata and exact-byte SHA-256, with a CI drift check.xerj autoindexreads an agent skill'sSKILL.mdfrontmatter: every passage of the file is titled with the skill'snameand carries itsdescriptionas a field of its own, whether the file is read as prose or as line windows. Before, a proseSKILL.mdwas titled by the first line after its frontmatter and a line-window one had no title at all. Only files named exactlySKILL.mdchange; the body, record ids and every other Markdown file index as before. A frontmatter block strict YAML refuses, such as one with anargument-hint: [path] [--dry-run]line, still gives its one-linenameanddescription. Unchanged files are not re-extracted, so an existing index gains the fields as each skill changes, or all at once withxerj autoindex --fresh. Part of #2644.xerj initteaches agents to find an installed skill by task: index the skills folder once withxerj autoindex <skills folder> --prefix skills, then runxerj search --prefix skills "<what you want to do>". The generatedSKILL.mdand theAGENTS.mdsection it writes both say so, and say the match is on words, not meaning. The init files version moves to 7, so a re-run upgrades aSKILL.md(and.cursor/rules/xerj.mdc) that an earlier init stamped and nobody edited. An existingAGENTS.mdthat already mentionsxerj searchorxerj def, which includes every one an earlier init wrote, is kept as it is, so only anAGENTS.mdwithout XERJ text gets the new lines. A new recipe,docs/recipes/find-a-skill.md, walks the loop with measured output. Part of #2644.xerj autoindex --skillsfinds the folders agents keep installed skills in (.claude/skills,.codex/skillsand.agents/skillsin the home directory, and.claude/skillsin the current folder), skips the ones that do not exist, and indexes each under its own prefix below one stem (skills-claude,skills-codex,skills-agents,skills-project; pick the stem with--prefix).xerj search --prefix skills "<what you want to do>"searches them all, and when the hits come from several folders each prints its full path.--watchwatches every folder. One prefix still owns one folder.xerj initwrites teaching files version 8, which name--skills, and the recipedocs/recipes/find-a-skill.mduses it. Part of #2644.bash scripts/bench/skills-intent.shis the eval #2644 asks for: it installs 388 skills from a pinned MIT library into a throwaway home, runsxerj autoindex --skills, and ranks the right skill for 26 labelled task queries written blind. On 2026-10-31 the right skill was first for 13 and in the top 3 for 22; every rank, misses included, is inbenchmarks/skills-intent/results/. Part of #2644.
Fixed
A plain debug
cargo test --workspaceno longer aborts on a stack overflow, and the127.0.0.2loopback test names the missing macOS alias (#2903). Five tests that run the ES-compat router on multi-thread workers overflowed tokio's 2 MiB worker stack in a debug build, killing their whole test binary: one inxerj-mcp'slocal_def, one inxerj-api'sautoindex_changed_file_drops_stale_symbols, and three inhub_api. They now use the server's 4 MiB, as the other in-process router tests do. On a host without127.0.0.2,loopback_proxyfails up front withsudo ifconfig lo0 alias 127.0.0.2 upinstead of a bare connection error, through the newxerj_common::loopback_test::require_loopback_alias. CONTRIBUTING documents both, and that CI runs--profile ci-test._bulkrefuses a malformed action line the way Elasticsearch 8 does: the whole request is a 400illegal_argument_exceptionnaming the line (Malformed action/metadata line [3], expected FIELD_NAME but found [END_OBJECT],Action/metadata line [1] contains an unknown parameter [_type]), and nothing in the body is written. XERJ answered 200 with a per-item error for that line and wrote every other action. An unknown parameter (_type,_version,_routing, a typo) used to be ignored and is now refused, as in ES 8; a client sending the 7.x REST compatibility header (compatible-with=7) may still send_type, as ES 8 allows. A line the check cannot read with confidence (JSON ES's own parser rejects, an escaped key, a byte order mark) stops the check there, and the rest of the body is answered as before.Data-stream
_rolloverhonoursdry_runand itsconditionsinstead of ignoring the body:max_docs/max_ageare ORed and reported inconditions, and a dry run or an unmet condition creates no backing index. The response names the real old/new backing indices, and the new backing is marked as the write index so writes through the stream continue after a rollover. Size-based conditions and alias-rollover body fields are refused rather than silently ignored.A hub catalogue already on disk is verified against its sibling signature before it is served, including a fresh cache and a
304 Not Modified(#2904). Replacingcatalogue.jsonused to be enough:load_catalogueparsed it with no signature check, and a future mtime kept that file fresh. A304then re-touched the replacement. The cache is dropped and refetched when the sibling signature does not verify.xerj mcpno longer answers a JSON-RPC notification, and runs everytools/calloff the stdio read loop (#2898). Atools/callwith noid, or withid: null, used to be awaited on the read loop, so a stalled search held uppingand every later request as in #2390; the id-less one was also answered withid: null. Now both run on their own task, the notification gets no reply, andinitialize,pingandtools/listsent as notifications get none either.Tests name the cause when loopback networking is unavailable (#2886). Every test that binds an ephemeral 127.0.0.1 port, in all crates, goes through
xerj_common::loopback_test::expect_loopback, so a sandbox that denies the bind fails with "this test starts a fake server on 127.0.0.1 and needs loopback networking" instead of hundreds of barePermissionDeniedpanics./leadrejects non-string email values with HTTP 400 before storage. Arrays no longer become accepted addresses through string coercion, and objects with a non-callabletoStringno longer cause HTTP 500.xerj autoindexnow records Dartextensionandextension typedeclarations asclassdefinitions, soxerj def <ExtensionName>finds them. Methods inside an extension were already indexed; the extension's own name was not, because the Dart query adapted from tree-sitter-dart'stags.scmhad dropped both forms.The field-report HTTP fallback now enforces its 16 KiB body limit on raw bytes before decoding or storage access. Multibyte JSON without a reliable
Content-Lengthno longer bypasses the limit by having fewer characters.xerj autoindexno longer stalls near the end of a run at high--workers. When all 8 bulk slots were busy, a worker whose chunk crossed the flush size parked; if another worker then shipped the shared buffer, that chunk included, the first one woke to an empty buffer and parked again every 30 s for good, so the run never finished. It now returns once the bulk carrying its chunk has landed. The same loop could also return while that bulk was still on the wire, so a file could be journaled complete before, or even though, one of its bulks failed; it now waits. On a 26-file, 28 MB scientific CSV folder, rc.100 stalled at 21 of 26 files with--workers 32and 19 of 26 with--workers 16; both now finish all 26 files and 179,374 records (#2841).weighted_avgnow executes field-based requests (value/weight, each with afieldand optional numericmissing) instead of returning HTTP 200 with anunsupported aggregation typeerror in the aggregations body. Ascripton either side, a malformed side, a missingfield, and a non-numericmissingreturn a 400 naming the side. Multi-index searches recombine by weighted sum, not by averaging the per-index values. A multi-valued weight field is skipped rather than refused, because the aggregation evaluator has no error channel; that case stays incatch-lenient.txt(#204).Re-indexing the same doc ids no longer grows
hnsw/graph.binwithout bound. Each overwrite tombstoned the old graph node and nothing ever removed one, so the graph (on disk and in memory) kept every vector ever written: 87.7 MB for 1,000 live vectors. The graph save now compacts the graph once dead nodes reach the live count (and at least 1,024): it rebuilds from the live vectors off-lock, replays writes that land during the rebuild, and swaps the result in, so the graph stays within about twice the live set. (#2796)Code-extraction size skips now report exact file/limit byte counts and explain that
--max-file-gb/--max-file-mb/--max-file-bytesdo not raise the separate fixed 2 MiB AST source-file cap. The reason no longer assumes that an oversized file must be generated. Help and Perl boundary regressions cover this diagnostic follow-up to #2666 (Refs #2745); the cap and skipped-file outcomes are unchanged.The version map takes about 23% less memory per document (#2427). A 32-character lowercase hex id (what
xerj autoindexwrites) or a lowercase UUID is now packed into 16 bytes inside its bucket instead of paying a separate 48 B heap allocation, other ids up to 24 bytes are stored inline, and the tombstone flag shares the version word. On the 2.46 M-entry perf-gate corpus the map drops from about 347 MB to 268 MB.A
_delete_by_querywhose matches fit in one scroll page no longer flushes the whole memtable before deleting.autoindex --freshsends one such delete-before-replace per file, and each one used to cut a new tiny segment out of every other worker's pending bulks (about 1,050 segments on the #2640 corpus). Larger deletes keep the flush-then-page path (#2640).A search no longer hangs forever when a cold segment's stored fields are being warmed in the background. A search-pool thread that waited for the warm's decode could block the decode it waited for, so every later search of that segment timed out at 30 s after ingest stopped. Search-pool threads now wait at most 200 ms and then decode the segment themselves (#2427).
A small merge write admitted after a run of merge yields is now checked against the memory watermark instead of the process limit, so it can no longer take a node from below the watermark into the parent breaker band and turn ingest into HTTP 429s. The relief for a search node resting between the merge-pool ceiling and the watermark is kept (#2427).
The HNSW doc-id maps now share one allocation per graphed document between the forward and reverse maps, instead of storing every id as two
Strings, and their bucket arrays and id bytes count toward thevector_graphingest memory category, so the breaker budget sees them. On the cap-2048 ingest run these maps held about 110 MB of live heap that no category reported (Refs #2427).xerj autoindexno longer stalls for up to 10 minutes per large dataset infinalize-count. The dataset time range was read withmin/maxaggregations, which make the node copy every document first; on a 393 k-record dataset that priced at 768 MB against the 512 MBlimits.max_query_memory_mbdefault and the client re-asked for its full 600 s patience. The bounds now come from two sorted size-1 searches, and a 429 from the per-query memory guard is no longer retried, since the same query is refused again (#2427).The
xerj-autoindexcorpus-lock tests no longer fail at random in CI (#2520, Refs #2427). One test put-c core.fsmonitor=falseafter the git subcommand, so git exited 129 at once and the test only passed by winning a race against that exit; and under the test harness a lock another test just released can stay held for 10-30 ms by a sibling test's forked child until it execs. Test builds now wait out a busy lock for up to 500 ms; a realcorpus addstill refuses at once.A search that decodes a segment's doc-values sidecar the segment cache has no room to keep now reserves the decoded size in the read transient budget before decoding, and holds it until the query drops the columns. Before, such decodes were neither cached nor reserved, so concurrent searches during heavy ingest could stack decoded sidecars (about 100 MB late in the cap-2048 run) that no memory budget counted (Refs #2427).
A memory-pressure drain no longer spikes RSS past the cap while it flushes (#2427). The breaker drain and the cache-release path flushed each index with every non-empty memtable shard building its segment at once, and that working set (parsed documents, columns, postings, stored blocks) is not charged to the memtable budget. On a 1,536 MB-capped node one drain took RSS from 1,460 MB to 2,323 MB in about a second. Memory-release flushes now build one shard at a time; ordinary flushes still run in parallel.
The per-index document version map is smaller and now visible in the ingest memory trace. Its keys are
Box<str>instead ofString, which saves 8 of 64 bytes per hash bucket and drops any spare capacity carried in from the incoming id. The map's estimated heap (hash tables plus key bytes) is reported as the newversion_mapcategory. On the 1 GB gate's search node the map held ~0.5 GB that no budget saw. (#2427)xerj autoindex: Vue (.vue) and Svelte (.svelte) single-file components are indexed as code instead of being junk-filed as malformed XML. Their<script>blocks are parsed with the TypeScript or JavaScript grammar (bylang), so their declarations are symbols on the component's own lines, and the record'slanguageisvueorsvelte. (#2732)The alias APIs reject an update that would leave more than one
is_write_index: truemember (POST /_aliasesandPUT /{index}/_alias/{name}), including a wildcard add that expands to several indices. The check is on the final state of a batch, so an atomic switch (add the new writer, clear the old one) still works; a rejected update changes nothing instead of leaving an alias whose every write answers 400._reindexappliessource._sourcefield filtering to the documents it copies, so an excluded secret or large field no longer reaches the destination._source:falseis refused with ES's sentence, and ascriptbody is refused before any work instead of silently copying the untransformed document (this build has no reindex script context).A document written through the per-document path (single-doc index and the
_bulkper-document path) is now held once in memory until the next flush instead of twice: the storage memtable and the full-text memtable share one parsed_sourcerather than each keeping its own deep copy. On a 1 GB_bulkload a late heap profile had the two copies at ~470 MB of ~1.75 GB in use (#2427).Publishing a flushed or merged segment no longer decompresses its whole stored section when the result cannot be kept. The publish-time warm now sizes the stored slices from the section framing first and skips the warm when the segment hydration cache has no room for them, or when keeping them would carry the node to the memory watermark. Before, a merged segment's section (hundreds of MB on the 1 GB autoindex gate) was decoded and then refused, a transient spike that pushed ingesting nodes over the breaker. Queries hydrate such a segment on demand, as before (#2427).
The in-memory HNSW graph no longer stores every vector in one contiguous buffer. That buffer doubled on growth, so near
393K 384-dim vectors a single 384 MiB -> 768 MiB realloc kept both copies live (1.15 GB transient) outside any memory accounting. Vectors now live in 4 MiB chunks (no realloc copy, at most one chunk of slack), and the graph's estimated heap is reported as the newvector_graphcategory in the ingest memory trace. (#2427)A node near its memory cap no longer warms flushed and merged segments into the segment-hydration cache. Publish-time warming used to stop only while the memory breaker was engaged. It now stops once RSS reaches the breaker's release level (85% of the watermark by default), so warming no longer spends the headroom the breaker has just won back. On the 1 GiB release gate (cap 3072 MB) the ingest node only received
_bulk, yet it held 200-580 MB of warmed stored slices between 40 breaker engages. Queries still hydrate a segment on first read. (Refs #2427)A cold full-text query no longer starts one background whole-section stored decode per cold segment at once (refs #2427). On the 1 GB gate's
pg-docs(54 cold segments) one 5-hit query drove anon RSS from ~1.0 to ~4.7 GB after it had returned, with most decodes refused by the hydration budget at publish and thrown away. One background warm per index now runs at a time and the others are skipped. Query answers are unchanged.simple_query_stringwith"fields": ["*"]is answered from the inverted index instead of a stored-document scan of every segment (refs #2427). The bare*pattern lowered to a literal"*"field that has no index projection, so each query parsed every flushed document; on the 1 GB autoindex release gate a 5-hit query walked ~1.2M stored docs and ran into the 30 s request deadline (raw-surface p95 ≈ 30 s).["*"]now lowers like an absentfieldslist, which is what ES means by it: unquoted terms are matched over every mapped searchable text field. A boost on the bare pattern ("*^2") is dropped, and quoted phrases are unchanged. On a flushed segment this also applies the searchable-field rule, so["*"]no longer matches through anindex: falsefield.query_stringwith["*"]is unchanged.A restarted node that receives no writes no longer runs catch-up merges when its indexes open. Inherited merge debt is now paid at the first segment-set change (a flush, refresh or delete) or on
_forcemerge. Before, a search-only node in the 1 GB gate started merging straight after the restart, while it was serving reads under the memory breaker. This supersedes the 120 s open quiet period in #2492, which only delayed the same merges (Refs #2427).Vector ingest no longer runs on one core per index: the HNSW construction search (greedy descent,
ef_constructionbeam and neighbour selection) now runs under a shared lock, and only the link writes take the graph's write lock. A bulk batch plans its inserts in parallel on the ingest pool, and concurrent batches into one index no longer queue behind each other's inserts (#2427)._bulkrequests of ordinary text and code are no longer refused with a 429 while the node has memory to spare. Each raw bulk reserved 28 bytes per input byte for analysis, a rate sized for the densest text, so a bulk of prose reserved about 3x and a bulk of code about 4x what parsing and analyzing it allocates. The reservation now follows the estimated token count of each document, and dense text keeps the old rate (Refs #2427)._bulkadmission no longer counts analysis memory twice. A held analyze reservation was added on top of live RSS even after its batch had allocated most of it, so a node with several bulks mid-analysis refused writes at about half its watermark. Held reservations are now counted from the RSS they were admitted at (Refs #2427).A node whose resting memory sits between the merge-pool ceiling and the memory watermark merges again. Every background merge's output write was refused there, so small batches were retried every 5 s without ever completing, and each retry decoded its inputs again. After three yields in a row with nothing merged, a batch of at most 64 MB now checks its write against the process memory limit instead of the merge-pool ceiling. (Refs #2427)
_bulkthroughput with explicit_ids:indexactions that carry an_id(and preparedsemantic_textdocuments) are now published as one batch per index instead of oneindex_documentcall each, which took the WAL lock, copied the source into the storage memtable and opened a publication bracket per document. Per-item results are unchanged:created/updated,_version, and per-item date and size errors that keep their_id. (Refs #2427)xerj autoindex --freshover an already-indexed folder no longer pays one edges-index search, bulk and refresh per replaced file (and again per superseded anchor) to soft-invalidate prior relationship edges. The graph phase now invalidates them in batched, terms-sliced sweeps with one refresh per sweep. On a 1,050-file markdown corpus with 14,724 prior edges the--freshrun went from 152.0 s to 26.7 s (edges refreshes 1,052 → 3), with the same 14,724 edges invalidated (#2640).A background merge no longer starts alone when its own memory reservation would carry the node past the merge-pool ceiling. It waits (at most 30 s) while the next batches are planned small enough to fit, so merging still makes progress without the overshoot behind the 1 GB gate's 3.5 GB ingest peak on a 3,072 MB cap (Refs #2427).
Search under a memory cap: when the segment hydration cache is full, a query that has to build a segment's stored slices or FTS reader outside the cache now reserves that memory from a separate read budget (
XERJ_READ_TRANSIENT_MB, default a tenth ofXERJ_MAX_PROCESS_MEMORY_MB). It waits up toXERJ_READ_TRANSIENT_WAIT_MS(default 5 s) for room instead of stacking whole decompressed sections on top of each other. Background cache warms that the full cache would refuse are skipped. While the memory breaker is engaged, its drain still flushes and purges on every tick but releases the read caches at most once perXERJ_BREAKER_CACHE_RELEASE_SECS(default 30 s), so queries no longer re-hydrate cold every two seconds.GET /_nodes/statsreports the budget underindices.read_transient_budget. (Refs #2427, #1122)Opt-in lexical
judgenow matches CJK characters and adjacent pairs inside uninterrupted Chinese/Japanese/Korean text instead of treating the whole run as one word (refs #2580). Single-character evidence is retained, pairs add adjacency evidence, and non-CJK word tokenization stays unchanged. This changes judged CJK scores/threshold outcomes; it does not change the first-stage analyzer or recover candidates absent from its result page.xerj corpus --help,xerj autoindex --help(--watch) and the XERJ.code README now say that corpora and folders do not follow their git remotes, and give the refresh recipe:git pullin each checkout, re-runxerj corpus addsocorpus.jsonrecords the new commit, thenxerj corpus index. (#2416)xerj_batch_search(andxerj batch-search): a batch whose combined_msearchresponse passes the 32 MiB cap no longer fails every item. The request is retried in halves, and only an item whose own response is over the cap fails, withresponse_too_large. (#2694)xerj autoindexandxerj autoindex map: when the node is unreachable at alocalhostURL, the error now also suggests the same URL with127.0.0.1, since the node binds the IPv4 loopback by default andlocalhostcan resolve to::1first (seen on Windows). (#2688)The node binds its ES-compat and REST listeners before it opens its indices, and on plain-HTTP ports answers every request with
503 node_starting_exception(withRetry-After: 1) until the engine is up./health/liveanswers 200{"status":"starting"}and/health/readyanswers 503, so a health check during a long startup no longer sees "connection refused" and takes the node for dead. TLS ports bind early and queue connections until the node is ready (#2575).A body sent without a JSON
Content-Type(thecurl -ddefault,application/x-www-form-urlencoded) is still refused with 415, as Elasticsearch 8 does, but the reason now names the fix:-H 'Content-Type: application/json'. The curl recipes that left the header out now send it (#2575).xerj autoindexnow says why a source file over the 2 MiB code cap is skipped (#2483). Such a file (generated stubs, minified bundles) used to be reported as "no records extracted (code candidate family, 1 junk lines)", which read like a parse failure. The reason now names the cap, the file size, and.xerjignoreas the way to leave it out of the plan._reindexhonoursdest.op_type: create,conflicts, andmax_docsinstead of silently dropping them.createno longer overwrites an existing destination document: the default abort returns 409 with ES's version-conflict failure, andconflicts: proceedskips and counts it.max_docscaps successful writes (the query-parameter form is accepted too); invalid values are refused before any source or destination work.Boolean text queries with one
mustmatch now keepfilterandmust_notclauses non-scoring before a flush and on mixed memory/segment indices. Their retained hits use the same BM25 scores as the bare match (#2403).Reindex preflights every source alias member before creating or writing the destination. Aliases cannot hide a source/destination overlap; a missing source, invalid query or source flush failure leaves destination data intact. An alias with
is_write_index: falserefuses writes even with one member.TS/JS factory constants exported by a separate local export clause or default identifier now get definition records at their declaration lines (refs #2579). Inline
export conststays supported; private constants, function-local bindings, type-only exports and remote re-exports stay out. Existing unchanged files need re-extraction to gain these definitions.Corpus cloning validates existing checkouts and serializes writers (#2453). New checkouts are staged before publication. Ambiguous existing directories are preserved for recovery rather than deleted or reset automatically.
MCP
xerj_searchadvertises plain text inqueryand structured clauses inquery_dsl. Legacy object-valuedqueryand omitted-query browsing remain supported; conflicting fields fail before HTTP requests (#2549).Search payloads and batch retrieval default to 8,000 UAX #29 words when
max_tokensis absent or null (#2491). This is not a byte or model-token cap. Explicit larger budgets remain available; errors and mandatory search staleness hints stay outside the payload budget.GitHub field-report submission uses the supported explicit-repository
gh repo fork --clone=falseinvocation and adds the fork remote itself (#2596). The rollback test now verifies that the report was committed before a failed push, rather than passing after an earlier fork failure.Local admin-key discovery reports the source of the selected key and warns when its directory is outside the working tree (#2578). Packaged installs have their own provenance, and MCP guidance names
XERJ_AUTH. Discovery priority and explicit credential overrides are unchanged.xerj autoindexfails a run when one dataset index holds no documents although the journal says records were written to it, even when the other datasets are live (#2407, defect 2). The check summed live documents over every dataset, so a populated-jsonindex let a re-run printok: indexed and searchableover an empty-docsindex. The error names each empty index and its contributing files; record totals are reported only when those files target that dataset alone. Unknown sibling counts do not hide independently confirmed missing data.SQL source extraction no longer slices statement heads inside a UTF-8 character, so unrecognized CJK statement/comment heads do not panic (#2547).
Short Markdown files with YAML frontmatter retain their prose body even when bullets or
Key: valuelines also parse as YAML (#2447). Existing frontmatter-title behavior and structured CSV/JSON body detection remain.Empty files keep separate inventory identities instead of being reported as content duplicates of unrelated paths (#2449). Non-empty byte-identical files still deduplicate.
A search with both
queryand a top-levelknnreturnssizehits, notk(#2561). The page was widened tok, sosize: 5, k: 20returned 20 hits.kstill sets how many nearest neighbours join the candidate set, andfrom/sizepage through the union, as in Elasticsearch.Index deletion no longer waits for unrelated index statistics while resolving its selectors. Index listings and health checks also release their index-map locks before awaiting statistics, so a stalled schema update does not hold those locks against DELETE (part of #2405; extracted from #2489). Merge/flush cancellation and the process memory cap remain separate follow-ups.
xerj autoindexcompresses per-file phase-B stages with zstd level 1 and uses temporary files cleaned up by the operating system when the client exits (#2493, part of #2427). It finishes buffered compression before deleting previous records, so a stage finalization error preserves the old records and an ordinary resume can complete the replacement. Compression savings depend on the data; this does not impose a disk or server-memory cap.autoindexbulk back-pressure patience is now measured since the last progress, not since the first round. A window the server drains a few records at a time (per-item 429resource_exhaustedon an embedding deadline or admission refusal) used to abort withbulk backend failedonce the rounds together passed the 600 sTHROTTLE_PATIENCE, even though every round settled records. Each round that settles at least one item now restarts the patience clock and the backoff; rounds that settle nothing still count, so a server that answers 429 forever fails at the same limits as before.xerj autoindextitles a prose file that opens with YAML frontmatter from itstitle:key, or from the first line after the closing---when there is none (#2556). Every such file (Jekyll, Hugo, Obsidian) used to be titled---, its opening fence. A leading---block that is not parseable frontmatter (unclosed, not a YAML mapping, over 64 KiB) takes the first line that is not a bare---; the body, the frontmatter text in it andstart_lineare unchanged. An unchanged file is not re-extracted, so re-index withxerj autoindex --freshto retitle documents already in an index.A
--no-graphre-index whose options differ from the committed generation (for example--sampleor--follow-symlinks) is refused before anything is copied, and the error names the option. It used to copy and prepare the whole folder, then report "could not derive generation N; the durable generation state is not internally consistent", although the state was intact and the original options still updated the corpus (#2577).xerj autoindex --no-graph --dry-runnow refuses a changed option the way the real run does, naming the option, instead of printing a plan and exiting 0 for a run that would then be refused. The--max-minutesnote on this route recommends--dry-runas the way to check a plan first (#2684).xerj searchandxerj defwith no hits on a prefix whose autoindex indices exist but whose first run has not finished now say so: search again once the run finishes, or rerun the same command if it stopped. They used to print the node's other indices and "the catalog lists no prefixes; --prefix is the index name before the dataset slug", which read as a wrong--prefix(#2670).xerj autoindexreads a spreadsheet sheet with no header row and no text, such as an all-numbers export, as a table: one record per row, columns namedcol_A,col_Band so on, values keeping their cell types. It used to index such a sheet as one text document, which the 4,096-section cap cut off: in the #2720 reproduction a 200,000-row sheet kept rows up to 72,724. A sheet whose first 20 rows hold a text cell, uneven rows, one column or only one row is still indexed as a document (#2720).A
--no-graphrerun no longer aborts when a file gains a part the committed generation has no dataset for, for example a new or renamed spreadsheet sheet, or a sheet that gains a header row and moves from the document reading to the table reading. The file is set aside with a rebuild message (exit 3) and every other file is still indexed. It used to stop the whole run with "frozen assignment has 0 matches for group" (#2720).xerj autoindexindexes HTML partials and template files instead of junk-filing them as malformed XML. A file that opens with markup and has closing tags was read as XML, so a folder of 16 or more such files became an XML dataset and every file was junked at its first<%,&&or unclosed tag, with the text after that point lost..htmland.htmare now read as HTML, and.hbs,.handlebars,.mustache,.ejs,.erb,.jsp,.aspx,.cshtml,.razor,.twig,.jinja,.jinja2,.j2,.njk,.liquid,.astro,.markoand.riotas text, also when gzipped. Real XML, including.xml,.xamland.svg, is unchanged. An index built before this fix keeps its junked files until each one changes or the folder is indexed again with--fresh(#2740).When the server refuses a record or a graph edge,
xerj autoindexnow names the file it came from and does not mark that file done. It used to mark every file in the same bulk done, so the rerun the abort message asks for resumed past the file, reported success and never sent the refused record (#2807).A file whose record the server refuses as over its 8 MiB per-document limit is now set aside as junk, with its file name and the server's reason, and the run completes with exit 3. Its other records are removed and it is retried on the next run. It used to abort the whole run, every rerun, until
--fresh. On asemantic_textfield the vectors the server adds count toward that limit: a 2 MB generated.d.tsfile grew to 16.6 MB this way.--no-semanticmay index such a file (#2807).xerj corpus indexandxerj autoindex --no-graphnow name the file a refused record came from, and for a size refusal they say how to index the rest: under a new--prefixand--state-dir, with the file left out (--exclude) or with--no-semantic. That route fixes its file list before it sends anything, so it cannot set a refused file aside. It used to say only "prepared bulk contained 1 rejected items" (#2807).xerj corpus indexandxerj autoindex --no-graphnow size each record with the vectors the server will add, before they fix the file list, and set aside a record that would not fit as a junk record. The run completes with exit 3, the file's other records are indexed, and reruns complete too. The 2 MB generated.d.tsfile from #2807 loses only its whole-file record; its type records are still indexed. That route used to stop on the server's refusal, and every rerun stopped on the same record. A run that an earlier version already stopped this way replays the list it fixed and stops again; index that folder under a new--prefixand--state-dir(#2807).
[1.0.0-rc.100] - 2026-10-26
rc.100: faster autoindex and search, shaped by your field reports. This
release is about speed. Background merges skip the zstd-19 stored encode, so
indexing under a memory cap runs at full speed again (#2441). Cold searches
read stored fields about 6x faster and decode only the hits and fields they
return (#2443, #2434, #2454, #2465). Search keeps its own share of the memory
budget while merges run (#2484, #2485), autoindex plans large folders in
bounded memory with fewer, larger datasets (#2417, #2430), and a
semantic_text test segment shrinks from 20.1 MB to 1.8 MB on disk
(#2471). A local performance harness adds autoindex and search budgets and an
Elasticsearch floor (#2432); release-workflow integration remains open in
#2433. Thank you to everyone
who filed a field report: they shaped this release. The memory-ceiling work
continues in #2427 and #1122, which stay open.
Upgrade notes
Stored
semantic_textvectors (PR #2471). Segments written by this build no longer store the vectors the built-in lexical embedder derives for asemantic_textfield (<target>and<target>_chunks) in the stored_source; readers rebuild them from the text. Older segments open unchanged and lose their stored copy when a merge rewrites them. There is no format-marker change, so an older binary still opens the index, but it reads those rows without vectors:_sourcelacks them and semantic and kNN recall on them drops until the index is reindexed. Before downgrading, restart this build withXERJ_STORE_SEMANTIC_VECTORS=1and runPOST /{index}/_forcemerge?max_num_segments=1on each index with asemantic_textfield: with the variable set, flushes store full vectors and a merge writes them back for rows stored without them. Vectors from neural, ONNX or proxy embedders, and vectors supplied by the caller, were always kept and are not affected.#2422: an existing autoindex keeps the datasets it already has. Later runs on the same folder use the new rules only for new files. To get the smaller set of indices, index the folder again with a new
--prefixand a new--state-dir, check it, then delete the old indices withxerj autoindex remove --prefix <old>.--freshdoes not do this for a--no-graphrun that has already committed: it refuses, as before (PR #2430).
Added
scripts/bench/neural-bulk-ab.sh CONTROL_SHA CANDIDATE_SHA [N]is an interleaved A/B harness for neural-embedding_bulkingest. It builds each SHA in a worktree, runs the realxerj autoindexclient against a frozen Okta corpus on a CPU-capped throwaway node, and reports median, p95 and the control arm's spread per metric. It measures the harness only; no performance claim is made. Seebenchmarks/ingest-neural-bulk/README.md.xerj sessions import --format claude-code --input FILE --output NEW.jsonlprepares an explicit Claude Code JSONL transcript for ordinaryautoindex. It emits text-bearing user/assistant records through the existing native PII plugin before writing, preserving UUIDs, parent links, source filename, line/byte offset and explicitly referenced file paths. Tool bodies, thinking and metadata records are omitted. The export is an offline, new-file snapshot: malformed input, unsupported turn shapes, repeated UUIDs or a changed source refuse publication; existing output is never replaced. This covers one preparation format for #1793, not automatic session discovery, incremental synchronization or universal secret removal. See the import guide (PR #2406).With
[cluster] enabled = true,POST /v1/indices/{name}/docs(one document, or a JSON array with one document per element),DELETE /v1/indices/{name}/docs/{id}, andPOST /v1/indices/{name}/docs/_bulkrun on the shard primary, whichever member accepted the request. The primary copies the stored document or the delete onto the in-sync replicas and acknowledges it with the same quorum asPUT /{index}/_doc/{id}. The caller still gets the native envelope:data.seq_nois the engine's 1-based sequence (the ES_seq_noplus one), a successful index or update is HTTP 201, and a delete of a document that is already absent is HTTP 200deleted. A primary that does not answer is the native error body with HTTP 503, and the accepting node does not store the document. A copy that misses quorum is HTTP 503replica_not_acknowledged: the primary has the write. The index must already exist; this path does not create one. A JSON array on this endpoint is a batch. It used to be stored as one value, because the single-document shape was parsed first and accepts any JSON.POST /v1/indices/{name}/ingestis unchanged and still writes on the receiving node, as do logs, OTLP, syslog, and turbo. A write still needs the primary.[cluster] enabled = falsedoes not take this path (part of #1170; the issue stays open) (PR #2380).xerj autoindexnames a file family in the run summary when every file of that family was junked, or when junked files are a strict majority (junk_files * 2 > files; stale-plan skips are not counted). The human summary and the progress stream print onewarning:line per family with the count and the most common junk reason.--jsonand the catalog run document gainfamily_junk_warningsonly when that happens; existing summary fields are unchanged and exit codes are unchanged.xerj autoindex mapand the MCPxerj_mapresponse include the same warnings (#2370) (PR #2377).xerj autoindex map <folder>previews a folder that has not been indexed. It runs the same discovery asxerj autoindex <folder> --dry-run(walk, sniff, infer, and print the plan) and writes nothing to an index.xerj autoindex mapwith no folder still prints the catalog of what is already indexed (#2368).With
[cluster] enabled = true,POST /{index}/_delete_by_queryruns on every shard primary, whichever member accepted the request. Each primary deletes the documents in its own shard and copies those deletes to the in-sync replicas, with the same quorum asDELETE /{index}/_doc/{id}.max_docsis one budget for the index, spent across primaries in node-id order, not once per shard. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is HTTP 503replica_not_acknowledged: the primaries that ran have the delete, and the response still reportsdeletedandfailures.wait_for_completion=falseuses the same path. Native/v1logs, OTLP, syslog, turbo, and pipeline ingest still write on the receiving node. A write still needs the primary.[cluster] enabled = falsedoes not take this path (part of #1170; the issue stays open).Release performance gate for autoindex and search. It adds
gen_corpus.py,autoindex_release_gate.shandgate.py, with per-phase budgets, an Elasticsearch floor and an rc.99 comparison harness (#2432). Release-workflow wiring and the live-tree memory follow-up remain pending in #2433; the harness is currently available for local runs.xerj initprints the Kimi CLI MCP config path (~/.kimi-code/mcp.json), and the README marks the Kimi CLI row as tested (#2386).Read-only post-deployment agent-discovery verifier for final GET/HEAD Link headers, documentation targets and optional public-skill digest checks. CI exercises local HTTP fixtures rather than contacting production (PR #2299).
Changed
- A
semantic_textfield with the built-in lexical embedder no longer stores its embedding twice. The pooled and per-passage vectors were kept in the stored_sourceas well as in the vector index; the columnar stored encoder now drops them when they rebuild bit-exactly from the text, and every reader (_source,GET, kNN, reindex, snapshot) rebuilds them, so responses are unchanged. On a 6,000-document test the segment file went from 20.1 MB to 1.8 MB and the index from 3.67x to 1.63x its source size. The saving is on disk; the HNSW graph in memory is unchanged. Sections under 128 documents keep their vectors.XERJ_STORE_SEMANTIC_VECTORS=1turns this off; see the upgrade note (part of #1122 and #2433).
Fixed
A
semantic_textfield declared withoutdimensionsis now sized at the width the server's embedding backend actually produces, when that width is confirmed (#2407, defect 1). It used to be always 384, so behind aproxyreturning another width (768, 1024, …) every document failed ingest with "embedding backend returned width N, mapping requires 384", and the new index stayed green with no semantic documents. Index creation,PUT _mappingand templates (including indices that_bulkcreates automatically) write the width into the mapping, soGET _mappingshows it. An explicitdimensionsstill wins. Indices created before this change keep their stored 384 mapping. Re-declaring a baresemantic_texton one of them while running a backend of another width is now refused by the immutable embedding contract. When the width cannot be confirmed, the field is still sized at 384, and the server logs a warning.xerj searchnames an unknown flag on one line and exits 2 instead of following "unexpected argument" with the whole usage text (#2458). An agent that guessed--indexand--max-hitsread the help dump as the answer and gave up on the CLI. Those guesses, and--size,--limit,--top,--top-k,-nand-i, now point at--prefix <P>or-k <N>.xerj_batch_search(andxerj batch-search) accept a plain string inqueriesas shorthand for{"query": <string>}(#2497). A string item used to fail alone with "arguments must be a JSON object", so an agent that sent["a", "b"]lost a call; the advertised schema now allows it._reindexfollows aliases on both sides: a source alias is searched across every backing index and each member's alias filter is applied, and a destination alias writes to its single write index (an ambiguous alias is refused withillegal_argument_exception). Previously only the first backing index was read or written and filters were ignored, so the call reported success over the wrong data.xerj corpus addaccepts a local Windows path such asF:\Projects\Foo(#2472). The repo name was the text after the last/, so a\-separated path stayed whole and was refused as an unsafe repo name; it now names the repoFoo, asF:/Projects/Fooalready did.xerj codeshows the part of a long definition that matched the query, not its first--fullcharacters (#2464). The matching text could sit thousands of characters below the signature, so a hit named the right function but showed none of the query terms. Such passages now end with... [function name: showing N of M chars from offset O, window centred on the match]; the@ line Nlabel and citations are unchanged, and a definition whose head is already the best window still endstruncated at N chars. Hub ask passages (xerj ask) and hub search API hits change the same way (PR #2474).A segment merge's output-write step (doc-values build, stored encode, FTS side-car build) is now charged to the merge budget before it starts, and a background batch yields instead of starting it when it would take RSS to the process limit (#2427). That step runs after the last yield checkpoint, and neither admission nor the yield checks saw it: large pg-docs merges (29k-166k docs) took a 3,072 MB search node to 4.4-4.9 GB about 5 s before they completed. The step is estimated at 6x the merged stored JSON or 2x the FTS build input, whichever is larger. An admitted step grows the batch's merge reservation, so concurrent merge admission sees it.
_forcemergebatches are charged but never refused, and neither is a small batch that the yield-streak rule (#2468) lets run to completion (PR #2470).The parent memory breaker no longer flaps, and background merges and flush finalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run engaged the breaker 20 times in 16 minutes, often releasing 100-500 ms later, and RSS still reached 3,808 MB because merges, FST builds and flush buffers kept allocating after writes were getting 429s. The breaker now releases below 85% of the watermark (
XERJ_MEMORY_BREAKER_RELEASE_PCT, 50-99) and only after it has been engaged for 2 s (XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS). RSS that stays flat inside that band for 30 s still releases, so a live set no drain can shrink does not block writes forever. The breaker never releases while RSS is at or above the watermark: the 1 GB release gate (watermark 2,918 MB) released at 5,121 MB through the old flat-footprint release (#1981), re-engaged 100 ms later and was OOM-killed at 8.4 GB. A footprint that climbs while the breaker is engaged is drained again within 2 s instead of waiting out the drain backoff. While the breaker is engaged, searches run one at a time, and at or above the process memory limit new searches get a 429circuit_breaking_exception. Every segment merge reserves its estimated working set (8x its inputs) against the watermark in one process-wide ledger. Background merges wait while the breaker is engaged or when the reservation does not fit, and run on the next merge check. One merge always runs when nothing else is merging and the breaker is not engaged._forcemergemay run alone while engaged. While the breaker is engaged, flush finalizes (segment, FST and doc-values builds) run one at a time. A background merge batch that is already running stops once the breaker is engaged and RSS reaches the process limit, keeps its inputs, and halves the input cap of later batches until batches complete again. On the 1 GB gate corpus one admitted batch (JSON segments that compress 10-20x) grew a 3,072 MBxerj autoindexnode to 8,122 MB in 18 s while writes were getting 429s; plain_bulkinto one index peaked at 4,967 MB (PR #2452).xerj autoindexno longer aborts a long run withResource temporarily unavailable (os error 11)when the machine refuses to start a thread. The sync replay used to start a new thread for every operation (hundreds of thousands on a large folder); it now starts at most the window width of workers once per replay and reuses them. Starting a replay worker or an indexing worker now retries with a short backoff when thread creation is refused, and the replay finishes on the calling thread if no worker can be started, so a refusal is no longer fatal (#2428) (PR #2437)._bulknow checks the memory limiter before it parses the request, not only when each index's batch is written. A bulk that arrived while the memory or memtable breaker was engaged still split and copied its whole body first. Several concurrent bulks between two RSS samples were each admitted on the same old sample, which could carry the node past its memory cap. Each request now counts its body against live RSS plus every other in-flight reservation until its parse ends, and is answered with one HTTP 429 per action (top-level 429,Retry-After: 1) when that would reach the watermark or a breaker is engaged, so clients back off and re-offer every record (part of #2427) (PR #2440).xerj autoindexno longer runs out of memory while planning a large folder. It used to keep the sampled values of every file in memory at once before grouping them into datasets, and kept them all until the end of the run; on a 5 GB folder of about 146,000 files that was several GB on the client before anything was indexed. Files are now grouped as they are read, and only what the end-of-run key correlation needs is kept. The datasets and index names it plans are unchanged (part of #1122) (PR #2417).xerj autoindexno longer fails a run because a dataset name built from a long path (for example a Go module path) is over the 255-character index name limit. Names longer than 128 characters are shortened and end in a short hash, so they stay the same on every run and never clash (part of #1122).xerj autoindexno longer keeps a second full copy of your files in its state directory. Each sync used to keep every indexed file's bytes undersync-snapshots/next to the prepared records, so the state directory grew larger than the folder it indexed. Those copies are now deleted as soon as the file's prepared records are safely on disk; resume and crash recovery still replay from the verified prepared records. On a 344 MB test corpus the state directory went from 503 MB to 242 MB after a sync (#2421) (PR #2435).xerj searchchecks whether the index is stale while the search runs instead of after it. The check reads the autoindex catalog, which cost about 0.3 s on the first search after a restart, and that time was added to every cold search (#2422) (PR #2443).Cold searches read stored fields about 6x faster. Picking a few hits out of a stored segment read the compressed data one byte at a time; it now reads it in 64 KB chunks. Loading one hit from a 20,000-document segment went from 1.3 s to 0.2 s, with the same results (#2422).
Loading a whole segment's stored documents, which a search that returns full
_sourcedoes the first time it touches a segment, is about 1.7x faster. The stored columns are now unpacked side by side instead of one after another, and the documents are rebuilt in parallel. On a 100 MB segment with embeddings this went from 0.41 s to 0.24 s, with identical output (#2422).The first searches after a restart, or after a segment left the cache, no longer read every stored document to return a page of hits. When the request names its
_sourcefields (asxerj searchdoes), only the winning rows and only the needed fields are read, and the full document cache is filled in the background. On a 344 MB autoindexed corpus the firstxerj searchafter a restart went from about 2.6 s to about 1.4 s.fields: ["_passage"]no longer makes a filtered_sourceload the whole document. (#2422, #2423) (PR #2434)autoindexno longer turns a large nested array of mostly numbers, such as a flight trace or a coordinate matrix, into one long text field. The record keeps a<field>_countnumber instead. Such text fields were up to 64 KB per record, made every search hit slower to load, and matched queries on stray digits. Re-index to drop them from existing indices. (#2423)xerj autoindexno longer makes one index for every small table or JSON file (#2422). On a 5 GB benchmark tree the old rules made 454 indices, two thirds of them under 16 rows. On a 300 MB test tree the same run now makes 34 indices instead of 228. Now:- A JSON, YAML, XML or HTML dataset with fewer than 16 rows, in at most
8 files, goes into the folder's document index (
<prefix>-docs, or<prefix>-<repo>-docsinside a repository) and is searchable there as text. Every record still carries its file path. An HTML page indexed this way keeps its title and headings, and its table rows are in the body. - JSON files with several arrays (for example
grammar.jsonor OpenAPI specs) now count toward the limit of 12 small datasets per folder or repository. Before, each array of such a file was its own index with no limit. A file's arrays stay together: all of them stay datasets, or the whole file goes to the document index. - One run makes at most 32 datasets. Past that, the smallest JSON, YAML, XML and HTML datasets go to the document index, and small CSV, JSONL and log datasets of the same folder or repository share one index per format, so their rows stay rows. Datasets with 1024 rows or more, SQL tables and spreadsheet sheets always keep their own index, so a run can still go over 32 with those. Rows of large CSV, JSONL and log files are counted in full for this, not only the 500 rows read as a sample.
- A file is moved whole or not at all, so a dataset that stays never loses rows (PR #2430).
- A JSON, YAML, XML or HTML dataset with fewer than 16 rows, in at most
8 files, goes into the folder's document index (
Indexing is fast again under a memory cap (issue #2438, part of #2423). Since rc.78 every background merge also tried a very slow, very small compression (zstd level
- on the stored documents. That made each merge ~35x slower in that step,
kept memory at the limit for the whole merge, and made
_bulkanswer 429 until it ended, so indexing was about 5x slower and a loader could give up and drop documents. Background merges now use the normal compression level. An explicit_forcemergestill uses the slow, small one, so the force-merged size is unchanged; setXERJ_MERGE_MAX_EFFORT=trueto use it for every merge. No data format change (PR #2441).
- on the stored documents. That made each merge ~35x slower in that step,
kept memory at the limit for the whole merge, and made
The helper thread that frees memory now starts when memory is high.
xerj-drain-freewas refused with "memory is at the governor watermark", which is exactly when it is needed._forcemerge?max_num_segments=1now reaches one segment when every segment is larger than half the per-merge size cap (it used to stop at two).Server startup now reports the resolved data directory, its configuration source and whether this launch created it. New-directory guidance points operators back to their previous
--data-diror--configwhen they expected existing indices/accounts. Defaults, flag precedence, auto-port relocation and existing data are preserved; no other data directories are scanned (#2409) (PR #2414).POST /{index}/_searchappliespost_filterto the hits, as Elasticsearch does. It used to be accepted and ignored: every document the query matched came back andhits.totalcounted them all. Aggregations still see the unfiltered query, which is the point of the parameter: a faceted UI keeps offering a facet's other values while one is selected. A request that aggregates now runs as two searches, one for the filtered hits and one for the aggregations.post_filterbeside a hybrid query is still a 400 (#943)._msearch, async search, scroll and a?q=query still ignorepost_filter(#2400) (PR #2404).xerj mcpno longer holds every message behind a slow tool call. Eachtools/callruns on its own task, soping,tools/listand other calls are answered while one call waits on the node. A ping sent 0.2 s after a call to a stalled node was answered after 60 s; it is now answered at once.notifications/cancellednow aborts the call it names, and no response is sent for that call, as the MCP spec asks; it used to be ignored. Replies can go out in any order (they are matched byid); a batch is still answered one line per request (#2390) (PR #2394).xerj autoindex remove --rootalso deletes indices an aborted or failed build of that folder left behind when the build never wrote a catalog run. The resume journal names the folder and the index names; a catalog walk cannot see them. A cataloged index is left in place — the working copy the abort preserved, and any index another folder still names.xerj autoindex statusmarks a live leftover of any prefixORPHANand prints the remove command (#2391) (PR #2393).xerj mcp: a proxied tool call whose response body cannot be read in full (the connection drops partway, or the request deadline fires mid-body) is now an error that names the URL and the cause. It used to come back as an empty successful result, which reads as "no matches" (PR #2388).MCP calls to a loopback node (
127.0.0.1,localhost,::1) no longer followALL_PROXY,HTTP_PROXY, orHTTPS_PROXY. reqwest sends those addresses through the proxy unless the client opts out, so a socks or HTTP proxy in the environment answered HTTP 503 while the node was healthy andcurl --noproxyreached it.xerj autoindex,xerj search,xerj def, and the other commands that share the node HTTP client skip the proxy for the same hosts. A non-loopback URL still uses the process proxy, includingNO_PROXY. When a proxy is used and returns HTTP 502, 503, or 504, or the connection to it fails, the tool error names the proxy (request to … went through proxy http://…, which returned HTTP 503) instead ofXERJ returned HTTP 503(#2379) (PR #2381).xerj mcp:xerj_code_searchnow gives up on a node that does not answer. Its requests went through a fresh HTTP client without the 30 s deadline every other tool uses, so a node that accepted the connection and then stalled held the call forever, and with it every later request on the stdio connection. The call now fails with an error after 30 s, like the other tools (PR #2366).SpawnedThreadisSendandSyncon musl.libc::pthread_tis*mut c_voidthere and an integer on glibc, so the handle was!Sendonly on musl.MemoryShardRpc::callholds it across.await, andxerj sharemoves the cloudflared log-reader handles onto another thread, so thex86_64-unknown-linux-muslandaarch64-unknown-linux-muslrelease builds ofxerj-serverfailed to compile (release.yml run 40133568141). POSIX treatspthread_tas an opaque id that any thread may join or detach, and this handle still has exactly one owner (#2376).A graph-path
xerj autoindexresume whose plan predates files now on disk names those files in the summary. The human text under theok:line, the lines immediately beforexerj-done, the run document, and the--progress jsonterminal object list the first five paths (skipped_appeared_files), how many were not listed (skipped_appeared_not_listed, only when the list was capped), and say to re-run with--fresh.skipped_appearedstays the full count. Exit code 3 andreason=stale-resume-planare unchanged. The plainxerj-doneline stayskey=valuenumbers, because a path may contain spaces (#2372).xerj autoindexindexes.tsand.tsxsources under Next.js route directories such asapp/[locale]/booking/[token]/. Gitignore reads[locale]as a character class, so a line written as the directory name (!app/[locale]/**orapp/[locale]/) did not name that directory and a broader*left the sources out of the docs index with no file named. Each ignore-file line that contains an unescaped bracket is also applied to the literal directory name. A real class such as*.[ch]is unchanged. A route-like class is explained in the ignore summary, a pruned[…]directory is named, and a code file that is still skipped is named with the rule (#2357).xerj autoindexextracts JavaScript definitions from inline<script>blocks in HTML pages.function openDrawer()inside a<script>in a.htmlor.htmfile becomes a code symbol (defs/symbols, plus a per-declaration record), soxerj def,xerj search, and the MCP def and search tools cite that function at its line in the HTML file. Asrcscript, a non-JavaScript type (application/ld+json, a template), and<style>are not definitions. The page body stays the visible text. A page taken as a data table (five or more consistent rows) stays row records. An HTML file already sealed in a generation keeps those records until the file bytes change or the tree is indexed under a new--state-dir(#2356).Proxy embeddings (
--embed-mode proxy,[embedding] default_endpoint) accept an OpenAI-style base URL. A path ending in/v1or/v1/, and a bare host, have/embeddingsappended. A URL that already ends in/embeddingsis POSTed unchanged, and a trailing slash is dropped. Any other path refuses to start inmode = "proxy"and names the URL that was set plus the{origin}/v1/embeddingspath to use. Inmode = "auto"that same path is an init failure and keeps the existing lexical fallback (#1189) instead of being called. A local vLLM base such ashttp://localhost:8000/v1used to be POSTed verbatim, so every embedding call 404'd. The embedder identity stays the configured string, so a corpus indexed with the full path does not become the/v1base (#2355).xerj autoindexno longer aborts the whole folder when a file disappears between the walk and the hash, or cannot be opened because another process holds it locked on Windows (sharing violation 32 / lock violation 33). The file is oneskippedcatalog row with a reason, the same way a file that changes size mid-run is skipped, and the run finishes (exit 3 when that skip is the junk). A locked file that was already indexed keeps the previous generation's copy on--no-graphand is not treated as an exclusion on the graph path. Unix errno 32 stays a real error (#2352).A failed FTS side-car publish no longer aborts indexing when the partial files can be removed, and building a field's FST no longer panics the process. The logged error names the source path, the destination path, and the OS error (
os error N). Two side-car names that a case-folding volume would store as one file (ASCII case, plusµ/μandς/σ) are refused before the rename, with both paths, instead of failing the batch as a not-found error with no file name or silently keeping only one field. The segment is still published: queries scan its stored section, and a later merge rebuilds full text. A leftover side-car that cannot be removed still fails the flush, and that error names the path and the OS error. A data-dir format-marker preflight failure still fails the flush and leaves the documents uncommitted, so a retry can write the marker and the encoded side-cars. This is the failure reported on Windows rc.98, where every broadxerj autoindexaborted at FTS side-car publish on the aggregate-docsindex, and the fst-build panic from rc.97 (#2349).xerj searchand MCPxerj_searchprint a hint when the autoindex of a folder on this machine is older than the files there. The hint names the last index time andxerj autoindex <folder> --prefix <name>. Passages are printed along with the hint.xerj code's 30-day ledger refusal is unchanged. The check reads the catalog run (summary_generated_at, orstartedwhen the summary stamp is absent) and stats at most 4096 entries, newest directory first, stopping after 100ms. Hidden names,.gitignore/.xerjignore/ built-in ignores, and a folder that is not on this machine produce no hint (#2345).The node no longer aborts when a search or a flush has to create a tokio blocking thread, and
xerj autoindexno longer aborts when its phase-A scan pool or object-fetch pool cannot map a worker's alternate signal stack (part of #1122; the issue stays open).std::thread::Builder::spawnreturns success and then panics in the child onENOMEM; the release profile ispanic = "abort". The server runtime caps that blocking pool (twice the worker count, plus a slice, at most 1024, or one thread past the worker count when that is already above 1024), keeps an idle thread for a day, and starts the spare threads at boot so a laterspawn_blockingorblock_in_placequeues. Tokio adds the worker count on top ofmax_blocking_threads, so the value passed in is only the non-worker slots. The scan and fetch pools start workers withpthread_createand, if the OS refuses, run the work on the calling thread. The object-store I/O runtime is driven on the caller; if it cannot be built, that operation fails and the index stays open. A global-allocator abort (memory allocation of N bytes failed) is unchanged.Search keeps its own share of the memory budget. Background merges now stop at a merge-pool ceiling 20% below the process limit, bytes held by merges are not charged to reads, and a search that arrives during a short memory peak waits up to 3 s for the drain before it answers (#2484, part of #2427, which stays open).
A lone background merge batch waits for headroom under the merge-pool ceiling before it starts (at most 30 s;
_forcemergenever waits), and merges check whether to yield every 1,024 documents, so ingest stays within its memory cap (#2485, part of #2427, which stays open).Merges are reserved on their on-disk footprint and held
_sourcetrees are released early, keeping the ingest node within its cap (#2462, part of #2427, which stays open).Cold full-text top-k walks on identities and hydrates only the winners on the page (#2454).
The cold stored scan decodes only observable columns (#2465).
The full-text projection skips should-leaves on unmapped fields, so queries that name unmapped fields stay fast (#2467).
ES compatibility: a
size:0probe skips the zero-hitunknown_valuesample, so ingest time stays proportional to the files indexed (#2457, #2451).A comma-separated
--prefix a,bgets a clear error instead of being joined (#2418). This comes from field report #2405, which stays open.A
timed_outfinalize-verify window is retried, so the run completes (#2419).xerj autoindexupdates a brain'snodes_indexwhen this run's dataset indexes differ from the stored set, sobrain overviewlists the prefix just indexed instead of a staleax-*list left by an earlier run of the same folder.created_atis kept. Code and SQL records also storestart_line(the same value aslineon a symbol,1on the file document) andend_line.pathon a code symbol is the repo-relative path already stored inax_path; a log or CSVpathcolumn is left as the source wrote it. A_sourceinclude ofpath,start_line, andend_linelocates records this version writes.lineandax_pathstay.xerj autoindex mapand MCPxerj_mapname those locator fields, including for an index written by an older build (#2351) (PR #2353).Writing through a multi-index alias follows ES's write-index rule: one backing index, or exactly one member carrying
is_write_index: true; an ambiguous alias is refused withillegal_argument_exceptioninstead of silently writing to the first backing index. The same resolution covers_bulkitems and the id-addressed GET/HEAD/DELETE, so a document written through the alias is readable through it, and the response_indexnow names the concrete index (PR #2326).Site: Search Console 404s redirect to their pages,
/api/is disallowed inrobots.txt, and installer and video assets carrynoindex(#2425).
Security
- zstd is bumped for RUSTSEC-2026-0391, RUSTSEC-2026-0398, RUSTSEC-2026-0403 and RUSTSEC-2026-0410 (#2466).
[1.0.0-rc.99] - 2026-10-24
Window since v1.0.0-rc.98: 116 merged pull requests. x86_64 release binaries
run on CPUs without AVX2. Cluster mode replicates index metadata, routes a
document write to the primary named at create, copies an acknowledged write
onto replicas placed at create, acknowledges that write once a quorum of
the copies has confirmed, reloads the shard plan from an on-disk Raft log
after a restart, installs a compacted Raft snapshot on a lagging member,
copies the primary's live documents onto a replica that is behind, and runs
_bulk, _update and _update_by_query on the shard primary;
#1170 stays open. ctutils
0.4.5 closes RUSTSEC-2026-0385. Autoindex, corpus and search fixes landed,
including partial work on
#1122; that issue stays open.
MCP gained local definition lookup and ordered batch retrieval. A DeepSeek
Harness profile bundle ships in dsh/. Contributors in this window:
Jarvis636431, Tong-bit-art, redwolf2019, ArkNill, pallidev,
samuelmbabhazi, elewashy, WYSTAF, kamwoh, mgtf, mariomarcolongo and Jun516.
Upgrade notes
- No downgrade after a case-distinguished field write (PR
#2261). Opening a data
directory does not by itself change its format. The marker moves from
2 to 3 only when this binary writes a field whose Windows case-identity
is not the name itself: an ASCII name with an uppercase letter
(
page_Id), orı,ſ,K, orß. rc.98 supports format 2 and refuses a higher marker, so after that write it will not open the directory. Back up the data directory before upgrading if you may need the older binary. A directory that has not taken that write still opens on rc.98. v1 and v2 segments still open on rc.99. - Packed embeddings (PR
#2240). A semantic vector
this binary writes is stored as one
xerj:f32:hex string. That does not advance the data-dir format marker, so an older binary still opens the directory. It does not unpack the string: kNN does not score that vector, and a source read returns the hex instead of a number array. Vectors already stored as number arrays stay that way until rc.99 rewrites them. - Hub catalogues (PR
#2249). rc.99 writes
catalogue format 2, which adds optional
upstream(owner/name). rc.98 and older clients accept only format 1, so they refuse a catalogue this binary writes. This binary still reads format 1. The published catalogue is still format 1 until a binary with this code rebuilds it, soupstreamis empty there. - Re-index to pick up new extraction (PRs
#2278,
#2342,
#2247). Documents already
in an index keep the extraction and settings of the binary that wrote
them. An unchanged file is not re-extracted.
xerj autoindex --freshrebuilds the index. That is what applies the AND match operator and the asctime date mapping on a logs dataset (#2278), indexes.sqlschema and routines as code (#2342), and skips catalog phrase text, parses phone and RFC 5424 syslog, and elects the english analyzer for a logs dataset (#2247). - Templates apply at create (PR
#2283). An index that
already exists keeps the mapping and
index.default_pipelineit was created with. A matching template'ssemantic_textand default pipeline are merged in when an index is created, including the first write that auto-creates one. Indices created before this are unchanged. - CPU baseline (PR
#2253). The four x86_64
release targets are
x86-64-v2(SSE4.2). The installers no longer refuse a CPU without AVX2.XERJ_SKIP_CPU_CHECK, the installer bypass added in PR #2178, is gone. A CPU without SSE4.2 still cannot run the binary. - Defaults that moved:
- A host under 16 GiB gets a process cap of half its RAM. On a budget
that cannot hold the old 512 MiB merge input, the merge batch is
budget/64(64 MiB at a 4 GiB cap) and concurrent merges share a quarter of the budget. Hosts at 16 GiB and above keep the previous steps. This improves #1122; the issue stays open (PR #2340). - On Windows the cap is then lowered to what this process can still commit, not only to physical RAM. Linux and macOS are unchanged. This improves #1122; the issue stays open (PR #2236).
- On a logs index that carries
index.query.default_match_operator: and, an omittedmatchrequires every token. An index without that setting, and an explicitororand, are unchanged (PR #2278). Existing indices do not gain the setting until they are recreated. - When a request highlights a
semantic_textfield on a semantic query, the default highlighter is the semantic one: chunk fragments, not<em>tags. A request that does not ask for highlight is unchanged. An explicit type other thansemanticstays on the lexical highlighter (PR #2227). - A
semanticquery on a wildcard,_all, an alias, or a comma list returns HTTP 200 and records the indices that lack the field under_shards.failed, instead of failing the whole request with 400. If every searched index is skipped, the request is still 400. One concrete index that lacks the field is still 400 (PR #2308). POST /{index}/_validate/queryreportsvalid: falsefor a body_searchrejects: a non-numeric value on a numeric field, or a field that is neither indexed nor has doc values. A selector that resolves to no index stays parse-only (PR #2263).- A question with three or more content tokens ranks a hit that covers more of the question above a one-word symbol. A one-token lookup is unchanged (PR #2329).
- Plain-string MCP
xerj_search, when the caller does not set_source, also returnsax_fileandax_locator. An explicit_sourcelist is left alone (PR #2255). - One indexed JSON document is capped at 8 MiB. That ceiling is a
constant, not a setting (PR
#2256).
PUT /{index}/_doc/{id}andPUT /{index}/_create/{id}refuse a larger body before parsing it (PR #2315). An_updatebody over two 8 MiB documents plus 256 KiB of envelope is refused beforeserde_json(PR #2322). This improves #1122; the issue stays open. - A bulk whose token analysis would cross the memory watermark is HTTP 429 before the documents are parsed (PR #2302). This improves #1122; the issue stays open.
- SIGTERM and SIGINT answer new HTTP requests with 503, leave
/health/liveat 200, and keep the ports bound until the final flush finishes. Connection refusal is the signal that the flush is done (PR #2246). engine.flush_admit_timeout_secs(default 120) andengine.flush_timeout_secs(default 300) bound how long a flush waits for a permit and how long refresh waits for one index.XERJ_FLUSH_ADMIT_TIMEOUT_SECSandXERJ_FLUSH_TIMEOUT_SECSoverride them. A refresh that used to wait now returns when the budget is spent (PR #2129).
- A host under 16 GiB gets a process cap of half its RAM. On a budget
that cannot hold the old 512 MiB merge input, the merge batch is
Security
ctutils0.4.5 (RUSTSEC-2026-0385, published 2026-10-23).Choicemask generation and 32-bitu128comparisons before 0.4.4 can branch on secret values. Nothing in the workspace names the crate. It is transitive:digest0.11 depends onctutils ^0.4, and that line is whatsha10.11,sha20.11,hmac0.13 andmd-5use, reachingxerj-autoindexandxerj-storagethrough the AWS SDK checksum path andpdf_oxide.cargo update -p ctutilslocked 0.4.5 (0.4.4 is the advisory floor).cmovstayed at 0.5.4. Lockfile only (PR #2268).
Added
Cluster replication, nine slices of #1170. The issue stays open. With
[cluster] enabled = falseevery one of these paths is the previous local call.- Index create, delete, mapping and settings are proposed through the Raft log. The HTTP call succeeds only after this node has applied the committed entry. A follower forwards to the leader. No elected leader, a timeout, or leadership lost before commit is HTTP 503, and the index is not created locally. The accepting node applies index templates once and replicates the effective schema. Console system indices created before the handle is attached stay local. Document writes stay on the accepting node. #2257 left that log in memory. The on-disk log is the slice below (PR #2257).
- The create entry names one primary per member (member ids sorted, shard
ion theith).PUT/POST /{index}/_doc,PUT /{index}/_create/{id},GETandDELETE /{index}/_doc/{id}run on that primary. A search of one concrete index asks every owner and merges hits by per-shard score, then appliesfrom/size. A forward that fails is 503 and is not also written locally (PR #2271). index.number_of_replicasat create (nested, flat or dotted; absent means 0) places that many copies, capped atmembers - 1, in a hidden.xerj_replica_{index}_{shard}index so a search of the user index does not count the copy twice. This pull request waited for every placed replica before acknowledging the caller. The next slice acknowledges a quorum instead. GET, and a search whose primary does not answer, read a replica. There is no promotion: a write still needs the primary. A laterPUT /{index}/_settingsstores a new replica count and does not move copies (PR #2287).- With
[cluster] enabled = true, a single-document index or delete is acknowledged when a quorum of the shard's copies has confirmed, not only when every placed replica has. Copies are the primary plus the replicas named byindex.number_of_replicasat create. A quorum iscopies / 2 + 1, and the primary already counts, so one replica out of two is enough. The replica that did not confirm is removed from the in-sync set through the Raft log and is not used for a later read until the backfill slice below has copied the primary's live documents onto it.number_of_replicas0 or 1 is unchanged: there is no replica to wait for, or that one replica must confirm. Fewer confirmations than the quorum is still HTTP 503replica_not_acknowledged: the primary has the write, and retrying the index copies it to the replicas still in sync. Single-node mode does not take this path (PR #2303). #1170 stays open: there is still nowait_for_active_shardsand no replica promotion. - With
[cluster] enabled = true, the server opens an on-disk Raft log at{data_dir}/_raftand replays its committed prefix before it serves. A restart restores the shard plan and the in-sync set, so a later document write is routed to the primary again. A member that starts with an empty log copies the leader's log and applies it, which installs index metadata (and an empty replica store when this node is a replica). Documents written before it joined are copied by the backfill slice below. Term and vote are stored with the log, so a restarted node does not grant a second vote in the same term.[cluster] enabled = falsedoes not open the log. Log compaction and InstallSnapshot are the next slice. This improves #1170; the issue stays open (PR #2313). - A member compacts its Raft log once 64 metadata entries have been
applied since the previous snapshot. The compacted prefix is stored in
{data_dir}/_raft/snapshot.metaand removed fromraft.log. A peer whose log does not overlap that prefix receives InstallSnapshot and applies those commands, which installs the same index metadata an AppendEntries catch-up would have. The snapshot is that command prefix, not a smaller folded image and not the documents. A history that does not fit in 8 MiB stays in the log. Restart replays the snapshot and then the entries after it. Copying the documents themselves is the backfill slice below. Part of #1170; the issue stays open (PR #2320). - A replica that does not have the primary's live documents copies them
and then rejoins the in-sync set. That covers a member that starts
after the writes (the create entry opens an empty replica store), a
replica removed from the in-sync set because it missed an
acknowledgement, and a process that stops and later opens the same
data directory: documents inserted, updated, or deleted while it was
down are applied to its replica store before the enlarged set is
published. The primary sends the live documents, drops ids the replica
still has that the primary no longer does, and publishes the enlarged
in-sync set through the Raft log. The primary holds that shard's
replica lock across the copy, so a write that arrives during the copy
is replicated to the new set when the lock is released. A later
single-document index or delete is copied to that replica with the
others. A write still needs the primary. Changing membership does not
move copies.
[cluster] enabled = falsedoes not do this. Part of #1170; the issue stays open (PR #2328). The outage case (insert, update and delete, then reopen the same data directory) is covered by the test-only pull request #2341. POST /_bulk,POST /{index}/_bulkandPOST /{index}/_update/{id}run on the shard primary, whichever member accepted the request. A bulk body is split by primary. An action whose primary is another member is forwarded there and is not indexed on the receiver. The primary then copies the stored document (or the delete) onto the in-sync replicas and acknowledges it with the same quorum asPUT /{index}/_doc/{id}._bulkcovers index, create, update and delete._updatecovers a partial document and a script; the script runs on the primary, where the current source is. A missing_idon index or create is chosen before the owner is selected, so the stored id is the id that was routed. An update that does not change the source is not copied again. A copy that misses quorum is503 replica_not_acknowledgedfor that item: the primary has the write. Native/v1ingest still writes on the receiving node._update_by_queryis the next slice. A write still needs the primary.[cluster] enabled = falsedoes not take this path. Part of #1170; the issue stays open (PR #2344).POST /{index}/_update_by_queryruns on every shard primary, whichever member accepted the request. Each primary updates the documents in its own shard and copies those updates to the in-sync replicas, with the same quorum asPUT /{index}/_doc/{id}.max_docsis one budget for the index, spent across primaries in node-id order, not once per shard. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is HTTP 503replica_not_acknowledged: the primaries that ran have the update, and the response still reportsupdatedandfailures.?pipeline=runs on the primary; a missing or unrunnable pipeline there is HTTP 400 and writes nothing. Pipelines are not replicated.wait_for_completion=falseuses the same path._delete_by_queryand native/v1ingest still write on the receiving node. A write still needs the primary.[cluster] enabled = falsedoes not take this path. Part of #1170; the issue stays open (PR #2347).- What #1170 still does not do:
wait_for_active_shards; replica promotion (a write still needs the primary); native/v1logs, OTLP, syslog, turbo, and pipeline ingest (they stay on the receiving node);_reindex; replicating pipelines; moving copies when membership changes; merging aggregations, suggest, collapse, rescore, profile,search_afteror scroll across owners; node-lifecycle APIs. The snapshot is the command prefix, not the documents, and a history that does not fit in 8 MiB stays in the log.
A DeepSeek Harness profile bundle in
dsh/(xerj-dsh, Apache-2.0, version1.0.0-rc.99) inserts the README'smcp-xerjstdio row via@deepseek-ai/dsh-mcp-clientat 0.2.1-rc.3. The row is still untested on DSH. The English, Korean and Simplified Chinese READMEs link the bundle.landing/brandbook/xerj-mark-400.pngis a 400×400 PNG of the letter mark for directory submissions (PR #2298).xerj batch-search --input <file|->and MCPxerj_batch_searchretrieve up to 50 IDs or questions in input order through one_msearchcall. Each item returns cited evidence, an explicit empty-result abstention, or an independent error. Exact IDs require an indexed keyword field. Optional whole-response token budgets preserve outcomes and at least one citation per matched item, or refuse explicitly. This is retrieval, not generated answers or calibrated confidence. Partial implementation of #1801 (PR #2333, Jarvis636431).MCP
xerj_defexposes local definition lookup, completing #2080 (PR #2166, Jarvis636431). It sharesxerj def --jsonquery construction, mapping guards, ranking and location deduplication; acceptssymbol,prefix/index,kind,lang,k(1–50) andmax_tokens. No match is a successful empty hit list; invalid arguments are rejected before HTTP.MCP
xerj_searchacceptsprefixas an alternative toindex, matchingxerj search --prefix:projectsearchesproject-*. Conflicting scope arguments and empty or non-string prefixes are rejected before HTTP requests. The default remainsax-*(PR #2137, Jarvis636431; the prefix half of #2080).
Fixed
xerj autoindexno longer junks every PDF on an OS that rejectssetrlimit(RLIMIT_AS)(macOS 27 returnsEINVAL; a hard limit below 1536 MiB returnsEPERM) (#2336, PR #2338). The address-space cap is crash isolation: the refusal is logged once and the worker still starts. Linux still applies the 1536 MiB cap. The same refusal is ignored for the isolated.xlsreader. If every PDF fails because the worker could not start, the run finishesok=false exit=1 reason=pdf-worker-unavailableinstead of looking like completed-with-junk.xerj autoindexindexes.sqlschema, migration and routine files as code (#2337, PR #2342).CREATE TABLE/VIEW/FUNCTION/PROCEDURE/INDEX/TRIGGER/TYPE/SCHEMA/SEQUENCEbecome symbols, and the source stays searchable. A comment, a quoted literal, a dollar-quoted body, or anINSERT/COPYnested inside another statement is not a dump. Those files used to be sniffed as an empty SQL dump and skipped (no records extracted), which dropped hand-written DDL. A file with a top-levelINSERT … VALUESis still a row dump. A data-onlyCOPY … FROM stdindump, which the row parser does not read, is skipped with that reason; when the same file also contains schema statements, those statements are indexed as code.A full process memtable budget now flushes instead of leaving
xerj autoindexon HTTP 429 until it gives up (#2321, PR #2325). The rejection requests a drain (largest memtable first) rather than waiting for the next memory sample, and a search issued while that breaker is engaged publishes those memtables before it scans them.xerj search/xerj defsayingno indices named …names the prefixes on the node in that same line, including a prefix recorded only on a catalog dataset document.The default process memory cap on a host under 16 GiB is half of that RAM, and segment merges follow it. This improves #1122; the issue stays open (PR #2340). An 8 GiB machine used to take the 8 GiB step, so the budget was the whole machine; a merge storm on Windows 10 stayed up only after
XERJ_MAX_PROCESS_MEMORY_MB=4096. That host now gets a 4 GiB cap. A merge batch on a budget that cannot hold the historical 512 MiB input (8× working set plus the 2 GiB memtable floor) is sized tobudget / 64(64 MiB at 4 GiB) and concurrent merges share a quarter of the budget. A batch that would still cross the RSS watermark waits for the next check; one batch still runs, because a merge is what retires per-segment caches. Hosts at 16 GiB and above keep the 8 / 16 / 32 GiB steps, the 512 MiB batch, and the configured merge parallelism. A single JSON document over 50 MiB is still refused at the 8 MiB ceiling, before it is parsed, and the rest of the bulk is indexed. The memory-ceiling class and the WSL2 VM death are not addressed.A multi-term question no longer ranks a one-word symbol definition (
type Token) above a hit that covers more of the question (#2327, PR #2329).xerj search,xerj def,xerj codeand the plain-stringxerj_searchtool add a coverage score when the question has three or more content tokens, and drop a one-termname+kindhit behind those results. A one-token lookup (xerj def Token) is unchanged. A nested test callback larger than a passage is no longer returned whole: the line-snapped window is kept, with its line. An archived path (archive/,archived/, orkeep_archived) is marked on thexerj codeheader and onxerj search.Graph-mode
xerj autoindexre-indexes a folder after files are added or deleted (#2314, PR #2317). An added file that fits a frozen dataset is indexed on the rerun; a file that needs a new dataset stays unindexed until--fresh. Documents, catalog rows and graph edges for a file that left the folder are removed, and the run names that file, including when--freshrebuilds the plan.xerj autoindex removedeletes that folder's resume journal andsync-snapshots, and an explicit--state-dironremovedeletes the journal there too. A file whose size changes while a run is reading it is skipped with a warning instead of aborting the run, on the graph path and on--no-graphsnapshot sealing. The catalog canonical/alias count refreshes and recounts before it aborts a generation. A rerun that removes documents for files that left the folder finishes exit 3 (reason=files-removed); the next unchanged rerun is exit 0.Scoped autoindex threads and
POST /{index}/_update/{id}no longer abort the process when a thread stack or a large body cannot be allocated. This improves #1122; the issue stays open (PR #2322). Indexing workers, sync replay, object-watch fetches and mailbox parsers start withpthread_createand do not map an alternate signal stack. A spawn the OS refuses is an error: the job does not run, workers that already started keep going, and the caller runs the work on the calling thread or records the failure. An update body over two 8 MiB documents plus 256 KiB of envelope is refused beforeserde_json, and the existing document is left as it was. The memory-ceiling class and the WSL2 VM death are not addressed.An idle node whose resident set sits flat at or above the memory watermark no longer drains every 2 seconds for the rest of the process (#1981, PR #2232). The breaker releases once RSS has not fallen by more than 4 MiB for 2 seconds, and does not re-arm inside that band (page-sized
ri_phys_footprintnoise is not a new working set); a climb past it engages again. While the breaker stays engaged, drain requests double their spacing up to 15 minutes when a drain does not lower RSS by more than 4 MiB.corpus index/autoindexreporteta_quality=stalledwhen a phase with a known total completes nothing (including 0/N), and when a phase with no total has been silent for 10 minutes.Semantic
xerj autoindexagainst--embed-mode proxyresumes after a mid-run failure and does not re-embed documents already indexed, when the backend, configured model, endpoint and confirmed vector width are unchanged (#2306, PR #2307). The proxy digest is still the model and endpoint (#533); the width is learned with one probe (and remembered under the data directory) and compared on resume, so a successful probe does not change the hash an existing index was recorded under. A proxy whose endpoint has not confirmed a width stays non-resumable, and the refusal says to make the endpoint reachable and rerun — already-indexed documents are kept — or to rebuild with a new--state-dirand--prefixonly to discard the corpus. A provider that swaps weights without changing the name, endpoint or width is still not detectable.The node no longer aborts when a helper thread or a merge buffer cannot be allocated. This improves #1122; the issue stays open (PR #2315). An in-process shard call starts its 8 MiB thread with
pthread_createand, if the OS refuses, returnscould not start the shard callwithout running the handler.PUT /{index}/_doc/{id}andPUT /{index}/_create/{id}refuse a body over the 8 MiB document ceiling before parsing it, the same wayPOST /{index}/_docalready did. A merge that cannot reserve its output buffer, document map, or the next survivor keeps the input segments and stays up. PDF and.xlsoutput readers, the autoindex progress ticker, and the finalize-verify search use the same fallible spawn: a refusal fails that file, skips the ticker, or runs the search on the calling thread. The memory-ceiling class and the WSL2 VM death are not addressed.POST /_aliasesstores and enforces the alias metadata it is given:filter, routing,is_write_indexandis_hidden(PR #2319, Tong-bit-art). Those fields were dropped by the action parser, so a filtered alias created through the standard API exposed every document while the same alias created withPUT /{index}/_alias/{name}worked._cat/aliasesprinted-for the filter; it now renders*for a filtered alias. Removing an alias, or re-adding it without a filter, clears the stored metadata, so a stale filter cannot outlive its definition.xerj corpus add --ownandxerj corpus own <name>(--offclears it) mark a corpus as your own first-party code ("own": truein itscorpus.json) (#2339, PR #2343).xerj codeand MCPxerj_code_searchstill name an unidentified licence (UNKNOWNorNONE-FOUND) on each hit and omit the do-not-copy banner for that unidentified licence. A licence the detector actually named still warns. A local path is not marked own unless the flag is passed, and a--frommanifest cannot set the marker.xerj corpus indexprints a per-file-class indexed vs ignored-data summary at the end.An existing index's persisted
index.default_pipelineruns again after a clean restart (#2294, PR #2296).GET /_settingsstill reported the binding, and the pipeline definition was still there, but an ordinary document write no longer copied the field: the in-memory binding was filled on create and on settings update, and opening the index from disk did not rebuild it. Boot, a failed-index retry, and snapshot restore now copysettings.jsonback into that map. Explicit?pipeline=was already unaffected.A bulk reserves memory for its tokens before the documents are parsed, and uncompressed JSON larger than 64 MiB is streamed one value at a time. This improves #1122; the issue stays open (PR #2302). A bulk whose analysis would cross the memory watermark is rejected with HTTP 429 before the documents are parsed, so several in-flight bulks cannot allocate their token strings together. A value over the fixed 8 MiB document ceiling is skipped with a note that names the file, its size, and that
--max-file-mbandlimits.max_process_memory_mbdo not raise that ceiling. A file over the scan cap names its size and--max-file-mb. The memory-ceiling class and the WSL2 VM death are not addressed.POST /{index}/_refreshmakes recent writes searchable without waiting for a background segment merge to finish, and a multi-index or wildcard refresh is not serialized behind one busy index (#2234, PR #2245). The route used to call the durability flush (snapshot fsync and WAL prune) once per index; autoindex finalize sent that as onePOST /<prefix>-*/_refreshand stalled at the end of the run when a large matching index had a merge in progress._refreshnow publishes an unflushed memtable and returns immediately when the memtable is already empty._flushis unchanged and still checkpoints the WAL. A merge encode no longer occupies a tokio blocking thread for the whole encode, so a refresh that does have to publish a segment is not queued behind that merge. Autoindex finalize refreshes each dataset index (and the edges index) on its own, names the index it is waiting on, and does not retry a timed-out refresh.Neural
xerj autoindexprints an estimated embedding time before ingest, from a probe measured when the neural model loads and the bytes of files that feed asemantic_textdataset (#1612, PR #2250). The line names no duration when the probe did not run, and it points at the faster lexical default.--embed-modehelp and the startup warning state that the mode is fixed per index at ingest and that changing it requires a re-index. Creating or updating a mapping warns wheninference_id,model_id, orinference_endpointnames a different embedder than the server is running; those keys do not switch the embedder.A
semanticquery posted at a wildcard,_all, an alias, or a comma-separated list skips an index whose mapping has nosemantic_textfield for that name, instead of failing the whole request with HTTP 400 (#2305, PR #2308). Those indices contribute no hits. The response stays HTTP 200, with_shards.failedand a failures entry naming the index. If every searched index is skipped, the request is still a 400, so a wrong field name is not an empty success. A query that names one concrete index is still a 400 when that index lacks the field. The same skip applies on_msearch,_search/template,_msearch/template, and scroll fan-out. A hybrid query whose semantic leg returns this error skips that index rather than failing the request.x86_64 release binaries no longer require AVX2 (#2161, PR #2253). The four x86_64 targets pin
x86-64-v2(SSE4.2: Nehalem / Bulldozer and later). The HNSW eight-accumulator dot product — the kernelx86-64-v3was actually vectorizing — is compiled again underavx2+fmaand selected at runtime when the CPU has both features. Rust does not contractacc += x * yinto FMA, so the scalar compilation and the AVX2 compilation are bit-identical. Pre-Nehalem (no SSE4.2) still cannot run the binary. Earlier in this window the installers refused a CPU without AVX2, BMI2 and FMA and left the build atx86-64-v3(PR #2178, samuelmbabhazi, from field report #2158). #2253 supersedes that refusal: the installers install on CPUs without AVX2, andXERJ_SKIP_CPU_CHECKis gone.A matching index template's
semantic_textproperties andindex.default_pipelineare applied when an index is created, including the bodyxerj autoindexsends and the first bulk or single-document write that auto-creates the index. The template already matched (akeywordfield from it was mapped), butsemantic_textwas stored asobjectand the default pipeline was dropped, so the pipeline never ran (#2281, PR #2283).xerj --helplistsxerj hubnext toxerj askwhen the build has a hub signing key. Thexerj askline namesxerj hub stopas the way to stop the node that command started. A build with no hub key still does not advertisexerj hub(#2280, PR #2282).POST /{index}/_counthonoursmin_score. The handler rebuilds asize: 0search body from the query alone, so the threshold used to be dropped: for{"query":{"match":…},"min_score":10}_searchreturned 0 while_countreturned every match. Both the single-index and the multi-index/alias paths now set it on the request, and a non-numeric value is the same 400parse_exception_searchgives (PR #2277, Tong-bit-art).On an autoindexed log index, a
matchquery requires every token. Omitted tokens used to count as a hit, and a date-typed xferlog stamp is kept as a date (#1504, PR #2278).MCP
xerj_searchcites the matched source line. Each hit can carrymatch_line(the absolute 1-based line of the best-scoring query term inside the passage) and a cappedmatch_lines. Both are omitted when no term is located or the passage start line is unknown (#2266, PR #2269).MCP
xerj_code_searchtoken budgets preserve adjacent definitions and distinct snippets at the same citation. Rendered text is deduped only when both its locator and displayed passage match an already-shown block; display labels, separators and match citations no longer stand in for source ranges (PR #2272, Jarvis636431).MCP JSON search responses with
max_tokensno longer deduplicate rows with distinctax_locatorvalues or hits from different indices merely because their relative paths and passage line numbers match. Deduplication includes the available source index, file identity and record locator; overlapping file and code-symbol passages in the same source still deduplicate. Plain-stringxerj_searchrequests the provenance fields needed for this distinction by default (PR #2255, Jarvis636431).xerj search --prefix,xerj def --prefix, and MCPxerj_search'sprefixargument resolve through one function: the prefixxerj autoindexstores.My_Projsearchesmy-proj-*, and a trailing-*is not part of the prefix. A prefix that cannot name an index is refused before any request. Omitting bothindexandprefixonxerj_searchstill searchesax-*(PR #2235; #2080).SIGTERM and SIGINT keep the HTTP ports bound until the final memtable flush has finished writing the data directory. New HTTP requests during that window get 503, except
/health/live, which stays 200 so a liveness probe does not restart the process mid-flush. Connection refusal is the signal that the process is done with the data directory; closing the port first let a wipe that keys off that signal delete a segment the flush was still publishing (PR #2246).A
node.lockleft by a process that is no longer running no longer blocks the next start (#2233, PR #2246). The data-directory lock is still the OS file lock, so a live node blocks a second start. When that lock is busy and the PID recorded in the file is not running — a Windows crash can leaveLockFileExstuck after the process is gone — the file is replaced and the reclaim is logged. A PID that cannot be read or checked is left alone.--auto-portdoes not treat that stale lock as a reason to start on an empty sibling directory.xerj corpus indexkilled mid-index resumes that build's journal instead of starting a legacy index under a new prefix. When the journal still has a generation in progress, indexing continues at the uncommitted operations and files already applied are not hashed again. A resume that verifies records the build. A resume that cannot verify is recorded as unverified, or left unrecorded with the journal kept, instead of a ledger that names no build (#2239, PR #2243).xerj corpus indexdrains when the memtable is under pressure, so a corpus that used to sit in a breaker loop (writes 429, nothing indexed) makes progress (#2238, PR #2244)._validate/querynow asks every resolved index the same buildability question_searchdoes: a query naming a field that is neither indexed nor has doc values, or a non-numeric value against a numeric field, reportsvalid: false(witherror/explanations) instead of claiming a body_searchrejects is valid. The parse-only answer is kept for a selector that resolves to no index (#1284, #204, PR #2263, Tong-bit-art).A
semanticquery on asemantic_textfield returnshighlightfragments when the request asks for them (#1947, PR #2227). The fragments are the field's best-scoring chunks for the query embedding, in score order, limited bynumber_of_fragments(default 5;0returns the whole field).order: noneemits that same selection in document order."type": "semantic"selects this highlighter, and it is the default for asemantic_textfield on a semantic query. Chunks are not wrapped in<em>tags. A lexicalmatchon the same field still highlights query terms.A single JSON document over 8 MiB is not indexed, so one file of 50 MB or more cannot abort the node during autoindex or
_bulk. This is a partial fix; #1122 stays open (PR #2256; the helper-thread half is PR #2226; the Windows commit-space half is PR #2236). Autoindex streams a top-level array, or a top-level array of records inside a larger object, one element at a time and skips only the values that exceed the limit. The skip names the file. A bulk item over the limit is HTTP 413 and its neighbours still index. The ceiling is fixed, not a new setting. This pull request still did not read a file over 64 MiB; the skip named that cap. PR #2302, above, streams uncompressed JSON past it. #1122 stays open.Opening an index whose snapshot names a segment file that is already gone drops that segment instead of failing every later
_searchwithSegment … not found. A sibling segment that is still on disk stays readable, and sequence numbers from the missing segment are not reused. Object-store mode is unchanged: a catalogue whose bundle is missing still refuses to open. This is what letxerj brain --freshfinish its catalog read-back after a data directory was only partly removed (PR #2256).A best-effort helper thread that cannot be created no longer aborts the process. Partial fix of #1122; the issue stays open (PR #2226). On Unix those threads are started with
pthread_createand do not map an alternate signal stack, so the allocation failure that used to panic inside the child (failed to allocate an alternative stack, os error 12) comes back as an error. Drain-free drops the generation on the caller; pool work runs inline; WAL fsync, background merge, and the ingest-memory trace skip the thread and retry or stay off; share-tunnel and ONNX init fail that startup step. Each site logs once.panic = "abort"is unchanged.On Windows, the process memory cap follows commit space, not only physical RAM. The AUTO tier is still chosen from physical RAM, then lowered to
process commit + available commitminus a reserve (one quarter of that ceiling, clamped to 512 MiB..2 GiB). The sampler compares the max of working set and commit charge with the watermark, and engages the breaker when available commit falls under the reserve. The HTTP 429 names the commit limit and how to enlarge the pagefile. Raisingindices.breaker.total.limitorXERJ_MAX_PROCESS_MEMORY_MBcannot grant commit Windows will not give. Linux and macOS budgets are unchanged. This is a partial fix; #1122 stays open: the memory-ceiling class and the WSL2 VM death are not addressed (PR #2236, from field report #2174).A flood-stage write block is explained in plain text.
POST /_memory/{ns},xerj memory storeand MCPxerj_memory_storesay the disk holding the data dir is nearly full, with free space, used percent, the threshold, the data-dir path, and that freeing space or restarting with--data-diron another drive is the recovery. The previous error was only a nestedcluster_block_exception. An explicitindex.blocks.write, and a manualread_only_allow_deletethat does not mention flood-stage, stay the original error (#2284, PR #2290).xerj corpus add <name> <local-path>copies the checkout's root.xerjignoreinto the clone when that file is untracked or has uncommitted edits.git cloneonly checks out committed files, so the ignore file was dropped and the index used built-in rules only (#2199, PR #2204). A pinned--fromrebuild stays at the recorded SHA and does not copy it; it says so when the working tree's root.xerjignorediffers. Nested.xerjignorefiles are not carried.A repo pin harvests the paths the pin names, not the whole tree (#1842, PR #2228).
xerj autoindexno longer wedges or runs out of memory on a JSON object made of several top-level arrays of records (#1856, #1978, PR #2131). When the largest array is longer than a config document (32 records), each top-level array of objects becomes its own dataset instead of being copied onto every row of the largest one. A shared value over 4 KiB is stored once. A data column that would besemantic_textfalls back to plain text — still searchable — when the real record count exceeds 100,000 or the elected text exceeds 8 MiB, and the plan says so, including--no-semantic. The pre-index budget counts records in.jsonfiles from 256 KiB up to 64 MiB and can refuse on a structural memory ceiling. That ceiling is not a measured store ratio, and no new disk ratio was invented for.json.Semantic embeddings for a JSONL autoindex are packed so resident memory stays bounded. One small-record JSONL used to cost on the order of a couple of hundred kilobytes of node RSS per record (#1940, PR #2240).
existson a text field is answered without hydrating the stored corpus (PR #2130).An embedding width that contradicts the
semantic_textmapping is a per-item 400, not a retryable 500 (#2191, PR #2217).Case-aliased field sidecars no longer abort publish (#2254, PR #2261). A side-car build or publish that fails now reports its cause (PR #2260, samuelmbabhazi).
POST /{index}/_refreshand autoindexfinalize-refreshno longer wait forever when a shard flush cannot get a finalize-gate or flush-semaphore permit. Admission gives up afterengine.flush_admit_timeout_secs(default 120 seconds, overrideXERJ_FLUSH_ADMIT_TIMEOUT_SECS) and does not freeze the memtable. The caller gives up afterengine.flush_timeout_secs(default 300 seconds, overrideXERJ_FLUSH_TIMEOUT_SECS) and leaves a publication that has already frozen its memtable running. The refresh response names the stuck index under_shards.failuresand continues with the other indices. Autoindex reports that failure instead of treating the refresh as success (#1635, PR #2129). Flush deadlines are measured on the wall clock, not a tokio clock that a test can pause (PR #2167, same issue)._cat/indicesand_cat/shardsuse the bare content count, sodocs.countagrees with_countand no longer includes file-card anchors (PR #2141, Tong-bit-art).docs.deletedon_catand the stats surfaces is the real deleted count (PR #2242, Tong-bit-art). Both close #2140.xerj index --file <dir>is refused before any index state is written, instead of dying inmmapwithInvalid argumentafter creating an empty index (#1818, PR #2172, elewashy).Autoindex and corpus, the rest of this window:
- A dead node is reported in full, and
_bulkretries stop (#2177, PR #2192). - A watched file can grow a field without failing the pass (#2176, PR #2193).
- Replacing a file drops the previous symbol generation (#2139, PR #2197).
xerj autoindex --watchdrops events on ignored files before they start a pass (#2180, PR #2198, samuelmbabhazi).- A changed file in a field-capped dataset is kept (#1815, PR #2215).
--langaccepts a file extension and stores the language name (#2205, PR #2219).- A dataless macOS placeholder is skipped instead of blocking the walk (#2058, PR #2223).
- A graph-path run whose catalog rows are not visible fails (#1977, PR #2224).
finalize-verifystops when its target index is deleted (#2222, PR #2225).- A resumed catalog read-back stays on the changed set. Same-generation
resume confirms the generation with one
size: 0count instead of paging every catalog document (#1267, PR #2231). finalize-verifycan capture between catalog writes (#1761, PR #2237).- Catalog phrase text is not indexed as a code symbol, and a phone-format syslog line parses (PR #2247).
- An unreadable corpus source fails the build instead of pruning every stored envelope for that slug. Empty ids, and a torn final record whose checksums were recomputed, are refused (#1110, PR #2258).
- A dead node is reported in full, and
xerj codeand Corpus Hub, the rest of this window:- Lower-case
license.txtandLicense.mdare detected (#2201, PR #2202). - Python passages cite a real start line, from symbols or the checkout (#2195, PR #2203).
- A weak nearest-neighbour page is labeled (#1986, PR #2221).
- An empty corpus ledger names the autoindex prefixes that are actually indexed (#2181, PR #2230).
xerj hub findprintsslug — owner/name — descriptionwhen the catalogue hasupstream. This binary writes catalogue format 2 and still reads format 1, where that field is absent and prints empty. The published catalogue is still format 1, so owner/name is empty until a binary with this code rebuilds it. rc.98 and older clients refuse a format other than 1 (PR #2249).- A question whose topic no hub corpus covers is
NoCoverage, not an ambiguous tie.xerj askprints that no corpus covers the topic, points atxerj autoindexandxerj hub find, and exits 1. Exit 4 is unchanged when several corpora actually match. This is a partial fix of #2179: the issue stays open, because no Phaser, pandas or charting pack was added and a slug that names a different project is still not explained (PR #2267).
- Lower-case
Testing
- A dropped replica is refilled to the primary's live set before it rejoins: insert, update and delete during the outage, then reopen the same data directory. Test only; no production change. It exercises the backfill in PR #2328. Part of #1170; the issue stays open (PR #2341).
- The MCP tool-count assertion was updated in the same change as local definition lookup (PR #2166).
Documentation & site
/productwas rewritten against the v1.0.0-rc.98 binary (PR #1926). Eight answer pages from the 2026-10-20 agent-query captures (PR #2251). The Day/Night choice is kept across pages (#1973, PR #2169, samuelmbabhazi). HomepageLinkheaders advertise the API documentation (PR #2291, mariomarcolongo). The Cline row in the English, Korean and Simplified Chinese README MCP tables is marked tested. Cline CLI 3.0.73 on Linux loaded the READMEmcpServersblock againstxerjv1.0.0-rc.98;xerj_searchwas in the tool list and one call ran. The IDE Configure button was not part of that run (PR #2300). The primary nav's plain GITHUB link is a stars badge: the GitHub mark and the star count, linking to the repository. The shipped count is 14,225, read 2026-10-24T02:05:55Z.theme.jsrefreshes it from the GitHub API, cached in the browser for 10 minutes, and keeps the shipped number if that call fails (PR #2311). The site states the MCP tool count main registers: 22 with a hub signing key (xerj_defandxerj_batch_searchincluded), 19 when the threexerj_hub_*tools are dropped. The publishedmcp-tools.jsonalready lists 22. The v1.0.0-rc.98 figure stays 20 and now names both tools that binary lacks (PR #2350). Canonical description, synced figures, FAQPage JSON-LD,llms.txt, an IndexNow key, and MCP listing manifests (server.jsonand the mcpb manifest)..claude-plugin/marketplace.jsonis a Claude Code plugin marketplace namedxerj(plugin source./plugins/xerj).packaging/docker-mcp/Dockerfilebuilds the Docker MCP imagexerj-mcp(stdioxerj mcp; the file's defaultXERJ_VERSIONis1.0.0-rc.98).landing/robots.txtaddsContent-Signal: search=yes, ai-input=yes, ai-train=yes. The registry manifest names the tagged release v1.0.0-rc.98 (PR #2323). The mcpb package identifier points at the prerelease assetreleases/download/mcpb-v1.0.0-rc.98/xerj-mcp-1.0.0-rc.98.mcpb, notreleases/download/v1.0.0-rc.98/(PR #2364).- A ready-to-paste DeepSeek Harness MCP config, with
envandcwd(#2286, PR #2288). Both Windows install directories are named:~/.local/binfor theshinstaller, including Git Bash, MSYS and Cygwin, and%LOCALAPPDATA%\Programs\xerjfor PowerShell (#2285, PR #2289). - AGENTS.md listed twenty MCP tools at the time (PR
#2150, redwolf2019).
After PR #2166
(
xerj_def) and PR #2333 (xerj_batch_search) the count main registers is 22, or 19 when the threexerj_hub_*tools are dropped. The Simplified Chinese README matches the English one (PR #2149, redwolf2019). A Korean README translation (PR #2164, pallidev).
Community & field reports
- Field reports: multi-array JSON autoindex (PR
#2132); MCP prefix isolation
(PR #2138, Jarvis636431);
Windows 16 GB, three autoindex aborts, WAL replay restores the corpora (PR
#2174, WYSTAF); the #1940
JSONL RSS reproduction (PR
#2241); an rc.98
xerj askof lz4 (PR #2252); consuming the published rust-vulns pack (PR #2259); MCP budget identity (PR #2265, Jarvis636431);corpus addplus a hub-manifest pull request (PR #2270, kamwoh); a Cline MCP setup from the README block, CLI only (PR #2301); batch retrieval against a fresh isolated node with synthetic advisories — usage only, not a quality or performance evaluation (PR #2334, Jarvis636431, written with OpenAI Codex). - CLA signatures: ArkNill (PR #2153), pallidev (PR #2165), samuelmbabhazi (PR #2168), elewashy (PR #2173), WYSTAF (PR #2175), kamwoh (PR #2264), mgtf (PR #2275), mariomarcolongo (PR #2292). Jun516 joined the contributor list (PR #2335).
Housekeeping
- Release-download counters landed as
[skip ci]chores, not pull requests: 2026-10-22 (30,604 assets, 25,047 binaries), 2026-10-23 (31,411 assets, 25,591 binaries) and 2026-10-24 (33,978 assets, 27,993 binaries). The figures are the commit subjects.
[1.0.0-rc.98] - 2026-10-21
Window since v1.0.0-rc.97: a console-API security fix (CVE-2026-97406), a
budget-aware xerj autoindex that estimates the index and checks it against
the node's flood-stage watermark before hashing (with --plan, --yes and
--max-index-size; a reduced plan never deletes already-indexed data without
--drop-indexed), xerj autoindex remove for dropping one folder safely,
--walk-only, a fail-fast on a full node disk, ES
request validation and query/aggregation fixes, highlighting, search ranking
and citation fixes, Corpus Hub routing and ratings, and the field-report
footer honouring the opt-out. Contributors in this window: amartya1523,
Tong-bit-art, roedyrustam, jacksonjp0311-gif, MrUranium1, jayhemnani9910,
leninworld, gyanu2507, dhkts1, Jarvis636431, ravi-arnan, rudycelekli, AK-Lmn,
miku1130, tree-lancer, Zacharon, tizerluo, olituks,
tripleaceme, MokuLakshithReddy, HiChat-fog, eltociear, omertahaoztop,
ronitgupta138 and Jitin2102.
Security
- Console API: a bearer API token can no longer mint credentials
(CVE-2026-97406,
CERT VU#645818, published 2026-10-20).
POST /_xerj-console/api/v1/auth/api-tokensaccepted anyAuthSession, and a passkey cookie andAuthorization: Bearer <api token>both resolved to that type. Holding one API token was enough to mint independent successors — revoking the parent left them working — and an owner token could issue an invitation, which the attacker redeemed in their own browser to enroll their own passkey. Creating and revoking API tokens, issuing invitations, deleting passkeys, and logout now require a passkey-backed browser session and return 403 for a bearer token. Ordinary console calls (/me, prefs, dashboards, and the other data routes) still accept bearer tokens. No token lineage is stored: the only insert of a new API token row refuses bearer auth before it writes, so a revoked token has no descendants (PR #2003).
Added
xerj autoindex remove --prefix P [--root R]and the MCP toolxerj_autoindex_removedrop one autoindexed folder. They list the indices, catalog rows, doc counts and sizes first and delete nothing without--yes, ayat the terminal, orconfirm: true. An index another folder or a sibling prefix still names is kept, andautoindex-catalogis never deleted.DELETEof*,_allor a wildcard is now refused unlessaction.destructive_requires_nameis set to false, as in ES 8. The agent docs and tool descriptions say not to delete an index the agent did not create (PR #2126, from field report #2114).GET /_memorylists the agent-memory namespaces (PR #2022, ravi-arnan).xerj autoindex --walk-onlystops after the walk. It prints each top-level folder's file count and size, what the ignore rules would drop and which rule, and the totals. It does not hash or sniff file contents.--dry-runstill hashes, sniffs and prints the plan (#1912).
Fixed
xerj autoindexno longer deletes an existing index when a later run uses a reduced disk-budget plan (--plan code-and-docs, or--yeswhen the default plan does not fit). The plan's keep-set was applied to the walk before reconcile, so files the plan left out looked removed: the--no-graphgeneration emittedDeletefor their groups, and the graph path swept them as exclusions. A re-run that would drop already-indexed files now exits 4 and lists those files and the datasets that would be left empty.--drop-indexedis the confirmation that deletes them.--planand--yeschoose what to index; they do not agree to the deletion. The regression came from #2102 and never shipped in a release (PR #2128).POST /_ask(xerj_plan) grounds a slash-bearing path in the prompt as a keywordprefixonax_path(thenpath,filepath,file,ax_file). A directory gets a trailing slash socrates/app/src/does not matchcrates/app/src2/. An index with none of those keyword fields is a 422 that names the path, instead of dropping it as an ignored lowercase modifier.xerj_maplists theax_*locators stamped on every record, and when the live mapping can be read itsfield_countis the catalog fields union the top-level mapping fields — the same set the plan counts, labeled on both sides. Cardinality of a locator the profiler never sampled stays unmeasured. Two slash-joined words such astcp/ipstay prose (#2120, PR #2124).A bare hub question that names a language or a technology (
xerj "how do I use async let in Swift",xerj "Swift Result type map") routes from the catalogue entry — slug, topics, a language word in the description, then the rest of the description — instead of the term sketch. On the 468-corpus hub the sketch's false positives tied swift-book with unrelated corpora (qwik, docker-docs, harper, oss-review-toolkit) and both questions exited 4. One clear winner is searched on its own. Two or three corpora that share the winning name are searched together and named on oneusing:line. Exit 4 stays for a genuine tie and lists at most five candidates, best first.An MCP tool call that cannot connect to
XERJ_URLnow says that no XERJ node is listening there, thatxerj mcpdoes not start one, and printsxerj --insecure --data-dir ~/xerj-data. A bare connection error was the whole tool result, so agents abandoned xerj for the session. HTTP responses, including 401, are unchanged (#2079).xerj autoindexon the graph path no longer counts files that appeared after the resume plan was frozen as junk. A resume whose plan predates files now on disk still cannot re-cluster that plan — dataset identity and graph anchors stay put, and the--no-graphroute is what reconciles additions. The resume finishesreason=stale-resume-plan(exit 3, recorded, not a failure), leavesjunk_filesfor unusable input, setsskipped_appearedto the count on every resume while those files are unindexed, and names--freshas the flag that rebuilds the plan in place and includes them (#2076).xerj autoindexchecks the node's data volume against the disk flood-stage watermark before walking or hashing. A volume already at or over the watermark (default 95%) used to fail only at the first write, after the prepare. The run now stops immediately and reports the free space, the threshold, andlimits.disk_flood_stage_percent. The probe reads the node (GET /_cat/allocation?format=json), not the client's disk. A node that does not report the threshold is unchanged, and--dry-run(which writes nothing) is never refused for disk.xerj autoindexcounts the walked folder and estimates the index before hashing. The free-space budget reuses that same allocation row (free bytes under the reported flood-stage watermark; the threshold is not guessed), and--max-index-sizecan lower it. When the default plan does not fit, stdout is a JSON plan list and the run exits 4 until--yestakes the recommended plan that fits or--plannames one; a named plan runs even over the estimate, with a warning, since the estimate errs high. A kind with no measured ratio stays unmeasured and does not refuse the run. A digit-leading--prefixis rejected before the walk (#2057). Skipped files are named after theok:line, and a PDF reuse-fallback line says the PDF is still indexed (#2056). A folder already indexed under another prefix is named before the walk (#2078). MCP:xerj_index_budgetreturns the same estimate and does not index.Playground user dashboards resolve their inherited template for chart data and panel editing. Cloning a user dashboard retains the underlying template; missing or cyclic template references keep the blank fallback. Custom dashboard layout overrides also survive reloading the page.
Landing-page copy buttons report
COPY FAILEDwhen clipboard access is denied or the legacy copy command fails, and retain their original label after repeated clicks. A failed copy can be retried.Playground PNG exports preserve computed SVG paint and text styles, including the embedding query crosshair whose CSS-variable stroke disappeared when rasterized outside the page stylesheet. The live chart is left unchanged.
Release-download snapshots now refuse malformed existing JSONL history without replacing its contents. Repair the history file and rerun the snapshot command.
Playground saved views immediately restore the current dashboard, replace stale URL filters when navigating, and preserve custom time-range endpoints.
Debian release-candidate package versions now sort before the corresponding final release, preserving normal upgrade order without renaming release assets.
Debian Rust coverage reports now include valid zero-overlap measurements instead of exiting before printing counts. Empty source indexes remain an error.
Hub pack planning reports malformed manifest JSON shapes as per-corpus warnings and continues planning healthy corpora, rather than crashing on an invalid entry.
Hub re-pin automation only supersedes generated proposal branches for the exact corpus, preserving proposals for corpora with longer overlapping names.
xerj searchand a plain-stringxerj_searchrank a file that contains the query as an exact phrase (a pasted error message or log line) at the top. The bag-of-words leg alone let a shortdefslist sharing two of the words outrank the long file holding all of them in order. On 100 exact string literals from XERJ's own sources the defining file went from rank 1 on 16 to rank 1 on 91, with symbol lookups unchanged (harness).xerj autoindex --helpnow states the existing--bulk-mbmaximum of 24.ES API compatibility, validated like ES with ES's own messages: aggregation parameters and placements (PR #1861), composite aggregation definitions (PR #1854), misplaced pipeline aggregations and bad
buckets_paths (PR #1824), index-creation, mapping, alias and template requests (PR #1870), and the per-index search request limits (PR #1859), all amartya1523. A bad index name is aninvalid_index_name_exception(PR #1666). A failed_bulkitem omits versioning fields (PR #1672)._cat/indiceshides.xerj_*rows by default (PR #1660), and 404 index hints stop suggesting system indices (PR #1661). A mid-name wildcard matches in index patterns (PR #1937) and_msearchexpands wildcard indices (PR #1939).Queries and aggregations: range queries honour
time_zone(PR #1744, jayhemnani9910). A range matches every value of a multi-valued field (PR #1776, amartya1523), and range aggregations bucket every value (PR #1803, Tong-bit-art).case_insensitiveworks on prefix and regexp queries (PR #1816, MrUranium1).filter/filtersaggregations count what the same clause counts as a query (PR #1752, Tong-bit-art). Histogram data buckets outsideextended_boundsare kept (PR #1823, amartya1523).Highlighting: whole words and every query term in a fragment are tagged (PR #1949),
highlight.fieldsis accepted as an array of field objects (PR #1950), and_msearchitems carry highlight like_search(PR #1952).Search: a file that holds the exact query phrase ranks first (PR #1759, jacksonjp0311-gif). A data row is cited by its locator, not by a line inside the row (PR #1944).
xerj codecites real file lines for prose and Markdown hits (PR #1908), and prose sections carry the file line they start on (PR #2013, Tong-bit-art). A plain-stringxerj_searchreads the target mapping first (PR #1976, Tong-bit-art). A plain-string search over a mixed index pattern no longer reports a field as unknown when another matched index maps it (PR #2123). On a code corpus, plain-wordxerj searchandxerj_searchapplyxerj def's penalty to test, example, fuzz and bench paths. Prose corpora, DSL queries and explicit sorts keep the engine's order (PR #2125).Autoindex: a heartbeat while an extract slot is full (PR #1657). An open section overlaps on a word (PR #1700). The help documents the bulk size maximum (PR #1746, leninworld). Duplicate content is ranked by native path identity, and an unprojectable plan is refused before the snapshot is sealed (PRs #1780, #1781, amartya1523). The summary leads with an
ok:line that spells out exit 3 (PR #1913). A rewritten--prefixwarns and names the wildcard collision (PR #1914), and a literal--in a prefix is kept soowner--repocorpora verify (PR #1906). The help says a.xerjignore!patterncannot re-include a hidden directory (PR #1916). A file that changes mid-run is skipped and recorded instead of aborting (PR #1917). Schema-only SQL is indexed as text (PR #1987), and a README whose first line is badges is not sniffed as JSON (PR #1988). Semantic verify errors include counts and datasets (PR #2016, Jarvis636431). The field-report footer honours--disable-feedbackandXERJ_DISABLE_FEEDBACK(PR #1907). The flood-stage fail-fast is PR #2087, the stale-resume fix is PR #2086,--walk-onlyis PR #2006, and the budget is PR #2102.Corpus Hub and corpora: the record corpora are published as Hub v2 hubpacks (PR #1736). Agents rate hub corpora with
xerj_corpus_feedback,xerj hub feedbackand automatic reviews (PR #1886).xerj askexits 6 when the hub catalogue cannot be loaded (PR #1905). A named language routes to its corpora (PR #2104). The hub test harness runs cross-platform, with Windows file locking (PR #1764, roedyrustam).corpus addandxerj codeuse a completed licence review when the detector reads UNKNOWN (PR #1910), and a BSD-3-Clause checkout no longer warns as BSD (PR #1674). A corpus name that its index prefix would rewrite is refused (PR #1919). A reference corpus skips CSV, HDF and simulation data (PR #2014, Tong-bit-art).MCP, CLI and installer:
xerj_field_reportfills version and os when the agent omits them (PR #2005), and a downXERJ_URLexplains how to start a node (PR #2085). The field-report verdict cap is 1000 characters (PR #1847), and unfilled optional placeholders are dropped on--submit(PR #1851). Thexerj --helpfeedback blurb names--submit(PR #1849).xerj initstops printing a lone backslash in the MCP registration (PR #1848). The installer resolves the latest release without api.github.com (PR #1852) and uses the absolutexerjpath in its next steps whenxerjis not on PATH (PR #1868). The console tells an expired magic link how to recover (PR #1675). A document-style write to/_memoryis explained (PR #1683), and a share link claim acceptsshare_id(PR #1684).Platform: macOS indexing no longer stalls under a memory cap, and shutdown no longer hangs (PR #1677, dhkts1). Native Windows probes for system RAM, process RSS and disk (PR #1766, roedyrustam). A bind failure says how to move each listener, including Windows error 10013 (PR #1779, amartya1523). Startup is refused when the neural model cannot load (PR #1877).
Testing
- Pinned filter
match/term/match_phraseanalyzer parity (PR #1748, gyanu2507). Thepdf_workertests that mutate the environment are serialized to fix a flake (PR #1743). The §12.4 whole-hub gate is green, correct and runnable (PR #1858), and the sandboxed hub-e2e gate runs on a CI runner (PR #1876). Formatting after #1804 (PR #1960).
Documentation & site
- Corpus Hub v2 docs for
ask,huband the MCP hub tools (PR #1728), and the llms.txt agent surface (PR #2048). The README keeps the data directory outside the indexed tree (PR #1686) and opens with a one-line "what it is" (PR #1923). A table of 26 supported agent harnesses (PR #2109, AK-Lmn). MCP tool counts (PRs #1850, #2118). An MCP Registryserver.jsonand an MCPB bundle builder (PR #1922). - Site: a light theme by default (PR #1920), answer-first answer and compare pages (PRs #1963, #1966), new answer and compare pages (PRs #1964, #1968), the Corpus Hub v2 post and one cover series for the blog (PRs #1915, #1918), sitemap content dates and noindex on markdown twins (PR #1992), a shorter home copy prompt (PR #1953), and a GA4 analytics tag on xerj.org (PR #1962; the site only, nothing in the binary).
Community & field reports
- Field report: the ES-YAML conformance run on a debug node (PR #2127).
- Field reports: the HTTP, MCP and console dogfood (PR #1688); autoindex and search on a Windows Electron/Vue project (PR #1809, miku1130); reference-coding natural-language false positives (PR #1936, tree-lancer); Windows MCP setup over an OSS checkout (PR #1943, Zacharon); semantic verification diagnostics on macOS (PR #2017, Jarvis636431); the autoindex store-size calibration (PR #2103).
- CLA signatures: HiChat-fog (#1600), eltociear (#1608), omertahaoztop (#1619), tizerluo (#1753), olituks (#1757), jacksonjp0311-gif (#1758), tripleaceme (#1765), amartya1523 (#1767), MokuLakshithReddy (#1806), miku1130 (#1810), roedyrustam (#1813), MrUranium1 (#1817), tree-lancer (#1938), Zacharon (#1942), ronitgupta138 (#1995), Jitin2102 (#2007), ravi-arnan (#2021), rudycelekli (#2053), AK-Lmn (#2108).
Housekeeping
- Dependencies:
zip9 for RUSTSEC-2026-0367/0368/0369 (PR #1869) andrustls-webpkifor RUSTSEC-2026-0382 (PR #2008).
[1.0.0-rc.97] - 2026-10-19
Window since v1.0.0-rc.96: the first release that ships Corpus Hub v2 switched
on, plus a fix sweep driven by field reports. The fixes cover xerj search /
xerj def / xerj code on corpora that are not code-shaped, autoindex
dataset sprawl and finalize robustness, searches bounded by their deadline
while a merge publishes, ES cluster-settings semantics, the #1122
thread-spawn class in three more pools, and a stricter ES-YAML runner.
Contributors in this window: Tong-bit-art, jayhemnani9910, leninworld,
agammann, YongxinFu, bryanbans, Jarvis636431, eltociear, todie (carried in
#1503), jmbentolila, RumbleOnex, shyamsivakumar, seanpowless-dotcom,
HiChat-fog, omertahaoztop, camen-collab and dhkts1, with eval lessons from
chunxiaoxx (#1138).
Corpus Hub v2
TODO(§12 gate), do not tag with this block as written. Under
docs/CORPUS_HUB_V2_IMPLEMENTATION.md§12, these notes may mention Hub v2 only after §12.1–12.4 pass. Two numbers below can come only from those runs: the pack count (§12.1,benchmarks/hub-backfill/REPORT.mdoncorpus-hub) and the whole-hub pass rate (§12.4). Do not estimate them. If the gate does not pass, replace this section with the rc.96-style "not a feature of this release" paragraph and setHUB_PUBKEY_HEXback to empty for the cut.
- Ask a pinned reference corpus, answered from a pre-indexed pack.
xerj "<question>"(the same asxerj ask "<question>") picks one catalogued corpus for the question and answers from a signed, pre-indexed pack on this machine. The client routes the question itself, using the project's lockfiles and the per-corpus term sketches in the signedcatalogue.json, so no hub server is involved. A symbol question (--def, a bare identifier, or "what is the value of X") returns the definition; other questions return passages withfile:line. Every answer names the corpus, its pin and its licence, and approach-only licences carry a read-do-not-copy warning on every hit. An ambiguous question lists the candidates and exits 4, and--corpus <slug>picks one. Nothing is cloned or indexed on the user's machine: a pack is a snapshot of an index that was already built. Answers need a local XERJ node. If none answers on the configured URL,xerj askstarts one on a free port from 9300 up, andxerj hub stopstops only that node. - What the catalogue holds. hub.xerj.org lists TODO(§12.1): N packs.
From a clean machine, against the published assets, TODO(§12.4): N of N
catalogued corpora passed
xerj hub pullplus one--deflookup plus one question. The gate is 100%; a corpus that fails is delisted, not shipped. - Signed end to end. The binary embeds the hub's ed25519 public key (PR #1505). The catalogue and every pack must pass the signature and sha256 checks before anything is extracted, and a tampered pack or catalogue is refused. A catalogue URL that serves an HTML page with status 200, which is what a static host does for an unknown path, reads as "catalogue unavailable" and never as data (PR #1587).
- Pulls are bounded.
xerj hub pull <slug>fetches a pack;xerj hub sync,status,stopandconfigmanage them.xerj askpulls on its own only packs of 50 MB or less, and never a pack whose licence class is Excluded. For a larger pack it prints thexerj hub pullline with the size. The packs directory has a 4 GB budget, and the least recently used packs are evicted first.--offlinenever touches the network. - No hosted node in this release. The hosted search and answer routes
are out of scope for this release (§12). Answers come from local packs
only, and
--hosted onsays that this build has no hosted client. - MCP tools never download.
xerj_hub_find,xerj_hub_defandxerj_hub_askrun the same answer flow as the CLI, with two differences: a tool call never pulls a pack and never fetches the catalogue. A corpus that is not on the machine is answered with thexerj hub pull <slug>line to run, and with no cached catalogue the tool says to runxerj hub sync. The tools, and the hub filesxerj initwrites, appear only in a build that has a hub key (PR #1629). - What ranks the answer. Passage answers come from the same BM25
pipeline as
xerj code. The default embedder is lexical feature-hashing, so nothing on the hub path is neural unless the node runs with--embed-mode neural. No performance or token-saving figure is claimed here. The step-10 measurement (#1385) governs any such claim and has not been published. - How packs are built and gated (CI and operator tooling):
xerj hub build-catalogue, the signed catalogue builder (PR #1513). Inhub-packs.yml, each pack is pulled into an emptyHOMEand must answer one--defand one question before its release is published (§12.3, PR #1514). Every run records a backfill-report row for every corpus, failures included (§12.1, PR #1522). The delisting guard refuses a "previous catalogue" that is not one (PRs #1582, #1588). The release gate itself is written down in §12 (PR #1485).
Fixed
match/term/termson anindex: false+doc_values: truetextfield answer the same before and after a flush (issue #1509, PR #1517, leninworld). Before the fix, the memtable answered on tokens or not at all, while the flushed segment answered on the whole value. Both now follow the ES whole-value, case-sensitive doc-values semantics. Coverage spans 15 direct and wrapped query shapes,size: 0totals included.- A search blocked behind a merge publication stops at its deadline
(PR #1638). It now returns
HTTP 200 with
timed_out: trueandhits.total.relation: gte. Before, it returned a 500 that retrying clients turned into 76–300 s waits on a node merging a 1.2 M-document dataset. This bounds the wait; it does not make the merge faster. PUT _cluster/settingsmerges into the stored settings, as ES does (PR #1595, jayhemnani9910). One update used to replace the wholepersistent/transientobject, and the nested form was stored nested and never applied. Settings are now keyed by their flat name, andnullremoves one.cluster.routing.allocation.disk.threshold_enabled: falseis honoured (PR #1596, jayhemnani9910): it lifts the flood-stageread_only_allow_deleteblock, as ES does.- ES-compat responses —
GET /{index}and_settingsreport the persistedcreation_dateinstead of the current time on every request (PR #1589, Tong-bit-art; issue #1573 is still open on the tracker at the cut).GET /{concrete-missing}/_mappingreturns 404index_not_found_exceptioninstead of200 {}, and a wildcard miss is unchanged (PR #1611). A bare_countno longer counts autoindex file-card anchors, while explicit queries still see them (PR #1624). A badhybridclause error names the{query, weight}shape (PR #1628). The slow-query log prints the effective body (match_all,knn, or both) instead of an emptyquery=(PR #1630). xerj searchandxerj defwork on corpora that are not code-shaped (issue #1601, PR #1602). On a markdown or log corpus, every query was an HTTP 400 because the client nameddefs/name/codefields that autoindex's strict mappings did not have. The client now reads the mapping and drops the clauses for fields that are not mapped. A field that any index maps as numeric is dropped too, inxerj code(issue #1284, PR #1605) and insearch/def(PR #1617), so onelongtitleno longer fails the whole wildcard query. A prefix that matches nothing exits 1 and lists the prefixes the node has (PR #1618).xerj defranks the declaration first (PR #1614). Structs, classes, functions and other preferred kinds outside tests, examples, fuzz and benches, andpub/exportitems, now outrank type aliases andmodlines. On sled,pub struct Dbmoved from 7th to 1st.xerj code—--mode semanticqueries the field each index actually embeds instead of onlybody(PR #1621), andxerj corpus indexno longer indexes the rootcorpus.jsonmanifest (PR #1622). A page cut short by the node's search deadline is marked partial instead of being printed as complete (PR #1637). The vector leg says that the default embedder is lexical feature-hashing, and translation catalogs and minified bundles sort behind implementing files (PR #1603, partial for field report #1504, which stays open).- autoindex no longer turns a config-heavy repo into one index per file
(PR #1610). Small JSON, YAML
and XML clusters fold into their scope's document dataset, and each scope
keeps at most 12 small structured datasets.
autoindex mapandxerj_mappage their field detail. On the engine tree this went from 161 datasets and a 376 KB map to 9 datasets and a 34 KB map. - autoindex finalize and ingest robustness. A count that times out or
errors is no longer read as 0 and no longer aborts a run that wrote its
records (PR #1639). A nested
JSON value stored as one field is capped at 64 KiB, after a 12 MB value
wedged ingest (PR #1640).
The zero-live refusal names the journal directory, which a new
--data-dirreuses (PR #1616). Prose sections are cut on word boundaries and carrystart_line(PR #1623).autoindex statusmarks another endpoint's journal asother endpoint <url>(PR #1630). - autoindex output agents read correctly. A second folder says "one
folder per run" and shows the two-run
--prefixform (issue #1586, PR #1590, Tong-bit-art). A usage error prints the error and one--helppointer instead of 390 lines of help on stdout (PR #1592, Tong-bit-art). Exit 3 is followed by a line saying that it means success (PR #1643). xerj initteaches--prefixon every surface it writes: the AGENTS.md section (PR #1584), the project skill (PR #1585,INIT_FILES_VERSION2), and the hub skill,--helpand closing text (PR #1606, version 3). A pristine older copy is upgraded with a.bak, and an edited one is kept (issue #1580 still open on the tracker at the cut).- CLI. Local admin-key discovery finds
~/xerj-dataand./.xerj-data, the node records a per-port pointer to its key, and a 401/403 says "authentication failed" instead of "no node reachable" (PR #1625). A flag written before the subcommand (xerj --url URL corpus index …) is accepted (PR #1642). Wrapped help lines keep their indent (PR #1626).xerj feedbackscopes its catalog sentence to the latest run and keeps--agentin the rerun line (PR #1627). The installers close with~/xerj-datainstead of./data,xerj mcp --helpcounts the tools it serves, and an unknown query type links a page that exists (PR #1629). xerj_mapfilters index globs before its 100-dataset cap (PR #1591, agammann). Before, 100 larger unrelated datasets could hide a matching one and report a falsematched: 0. This entry was under[Unreleased].- A failed thread spawn degrades instead of aborting the node, in three
more places (partial for issue
#1122, which stays open).
The governor threads and drain-free spawn use the new
try_spawnhelper and spawn gate (PR #1460). The autoindex object-fetch pool falls back to sequential fetches (PR #1520). The ingest, background and merge rayon pools retry at one thread and then run inline (PR #1604). - Decision write-back keeps growing on split votes (issue
#1094, closed during the
window; PR #1503, carries
todie's #1467). With the local decide tier armed, a history vote whose
winning share is below the new
[decisions] escalate_belowsetting (default 0.8) goes to the local head. The answer is written back with one cached document per decided text, so repetition cannot out-vote the history. With no local tier armed, nothing changes. The Banking77 replay gate has not been re-measured. The setting count goes from 132 to 133.
Added
- autoindex reads journalctl / ISO-timestamp syslog and vsftpd
xferlogas per-line log records with parsed fields (PR #1508, Tong-bit-art, from field report #1504, which stays open for its other suggestions). Before, both were shredded into prose chunks.
Security
- The server no longer prints the admin API key on first boot (PR
#1633). The key is written
to
<data_dir>/admin.key(mode 0600) first, and the notice names the file. An agent that captured server stdout used to end up holding the credential.
Testing
- The ES-YAML runner enforces
catch:(PR #1583). It used to skip every expected-error step, so "0 failed" said nothing about requests that ES rejects. On origin/main the strict run had 151 failures (91 aggregations, 37 search, 12 indices, 7 scroll, 2 bulk, 2 vectors). They are listed incatch-lenient.txtand keep their old behaviour, and every other case is now strict. Four new cases record known divergences and wait for fixes:null_value(PR #1553),enabled: false(PR #1574),_sourceincludes/excludes (PR #1575), and dynamic-template mapping types (PR #1576).
Documentation & site
- The agentic-memory recipe, in the repo and on xerj.org, gains "Before you
trust a recall number", chunxiaoxx's eval lessons from #1138 with credit
(PRs #1527,
#1535; issue #1138 stays
open). A Japanese README (PR
#1607, eltociear). The
contributing guide describes the conformance gate as it runs (PR
#1498, Jarvis636431). The
dead
xerj admin magic-linkis gone from the demo and design docs (PR #1615). Sitemap refresh (PR #1632). - Accuracy polish, one claim per PR. Site and docs pages: #1530, #1532,
#1536, #1537, #1539, #1545, #1546, #1547, #1549, #1551, #1555, #1557,
#1559, #1562, #1563, #1571. Markdown: #1525, #1528, #1529, #1566, #1567,
#1568, #1578.
SECURITY_MODEL.mdline references: #1538, #1540, #1541, #1542, #1543, #1579, #1581. Rustdoc: #1544, #1552, #1556, #1560, #1564, #1569, #1570, #1572. Scripts: #1533, #1534, #1548, #1550, #1554, #1558, #1561, #1577.
Community & field reports
- Field reports: a reference-coding probe on unfamiliar Rust (PR
#1511, jmbentolila);
reference coding and autoindex on Windows (PR
#1518, RumbleOnex); Qabot
reference-coding retrieval on rc.95 (PR
#1519, shyamsivakumar);
search vs
git grepon cudarc and candle (PR #1598, seanpowless-dotcom), which led to #1633; a ZCode (GLM) workstation setup (PR #1599, HiChat-fog); local repos plus hub corpora (PR #1613, omertahaoztop); the engine and sled dogfood (PR #1620); and autoindex at defaults stalling a 13.8k-file repo on rc.96 (PR #1636, camen-collab). That stall is issue #1635, which stays open and is not fixed in this release. - CLA signatures: jmbentolila (#1512), jayhemnani9910 (#1594), agammann
(#1597), cursoragent (#1609), dhkts1 (#1646). A
funding.jsonmanifest for FLOSS/fund (PR #1676).
Housekeeping
- CI: the workflow-timeout guard rejects job declarations it cannot parse (PR #1472, YongxinFu). SEO dates do not trust shallow-clone history (PR #1476, bryanbans). Docs-only PRs skip the heavy Rust jobs and fail open (PR #1477). The 8-target release matrix runs nightly and on tags, not on every push to main (PR #1565).
- The metrics worker source is synced with the deployed worker, and its
/trending-historyroute is documented (PRs #1524, #1531).
[1.0.0-rc.96] - 2026-10-15
Window since v1.0.0-rc.95: a fix-and-honesty sweep across search tie order,
dynamic-mapping doc values, autoindex progress and resume reporting, the
finalize gate, Windows home-directory resolution and xerj init, plus two
dependency security updates. Contributors in this window: Tong-bit-art,
todie, fjjjuv, redwolf2019, riazrahaman, sebollin, Tyagiquamar, driphtyio and
mininxd, whelanh and Jarvis636431. Parts of
this section's wording for the PRs merged before 2026-10-14 are adapted from
todie's draft release PR #1397, with thanks.
Corpus Hub v2 is not a feature of this release. Its code is in the tree
(PR #1423), but this build has
no hub signing key compiled in, so no pre-indexed pack can be verified and
none is downloaded. PR #1491
makes that explicit on every hub surface: xerj ask, xerj "<question>",
xerj hub sync and xerj hub pull exit 5 with one plain message instead of a
catalogue parse error, the MCP hub tools return the same message, xerj init
no longer writes the hub skill or advertises xerj "<question>", and
xerj --help says the command is unavailable in this build. The
xerj code / xerj corpus reference-coding path is unchanged and works as
before.
Fixed
- Inferred
textfields take the ES dynamic-mappingdoc_values: falsedefault (PR #1329, Tong-bit-art) — an inferredtextfield keptdoc_values: truewhile a declared one defaulted tofalse, so at flush the segment wrote a whole-value keyword column andtermsanswered against raw untokenized strings (another instance of the "answers change at flush" class). Inferredtextnow matches declaredtextand ES;.keywordsub-fields keep their column. Test follow-ups: PR #1331 (agg_bucket_cap_terms) and PR #1356 (Tong-bit-art, thegraph_expandfixture declares itsedgesmapping). term/termson atextfield withdoc_values: trueno longer drop to 0 hits after a flush (PR #1489, Tong-bit-art) — the doc-values term prefilter answered against the whole-value column for analyzed text; it now declines analyzed text and the query takes the full, correct path.- Total tie order (#191) held on two more
termroutes (issue #1359, PR #1393, todie) — atermon atextfield without doc values (the post-#1329 default) and a standalonetermon akeywordfield across an unflushed memtable plus flushed segments could return pages that were not prefixes of each other. Bounded pages now keepscore DESC, seq_no ASC, _id ASCacross segments. xerj autoindexaccepts a whole-file size cap below 1 GB (issue #1233, PR #1321) — new--max-file-mb/--max-file-bytesbeside--max-file-gb(default still 2 GB); two size flags together are a usage error. Existing indexes resume with the same prepared-records identity.- An aborted autoindex run no longer prints the "searchable" success
footer (PR #1318, partial
for issue #1183, since
closed) — the salvage path now says the records are present but coverage is
not guaranteed, and points at
xerj corpus index <name> --fresh. - Finalize progress and old-server detection for corpus resumes (issue
#1267, still open) — the
catalog read-back walk is its own
finalize-catalog-readbackprogress phase with a real denominator instead of a frozen N/N bar (PR #1320), and the client times one verify window before the walk and warns, without aborting, when the server looks older than the client's fast-path generation (PR #1430). The server-side walk cost is not fixed by either; the issue stays open. _refresh/ flush skip empty shards before the shared finalize gate (issue #1226) — PR #1319 skips empty shards when building the flush/refresh fan-out; PR #1456 (carries todie's #1399) also checks insidedo_flush_shardbefore taking the process-wide gate, and resolves concrete index names and aliases in O(1) instead of listing (and computing stats for) every open index. No latency figure is claimed here; each change is pinned by a regression test.- Multi-index ingest against the memory breaker (issue
#1250, closed during the
window) — the parent breaker watermark is live-tunable through the ES
dynamic cluster setting
indices.breaker.total.limit(a percentage of machine memory or an absolute size), without a restart, and the 429 message names the knob (PR #1323). The autoindex bulk client damps its AIMD oscillation (a 15 s episode cooldown, an ssthresh, a permit per HTTP attempt) and the breaker's drain churn is reduced (PR #1482, carries todie's #1400). Per-index admission and a server-side shed signal are not part of this. - Native
/v1/cluster/healthreports the configured topology (PR #1316, partial for issue #1170) — it hardcoded one node; it now agrees with_cluster/healthand flags a degraded transport. This is the configured floor, not live ring state: the Raft ring is still not wired to the engine and #1170 stays open. - Corpus builder refuses two silent wrong-result paths (PR
#1315, partial for issue
#1110, which stays open) —
a missing source root used to prune every stored envelope with exit 0, and
records with an empty id shipped together and collapsed to one
_id. Both now fail loudly. PR #1500 (carries todie's #1468) also fails the build on an unreadable or invalid stored envelope and deletes stale shards before writing new ones, and adds acisa-kevpack definition (not published or signed). The fresh-machine portability matrix of #1110 is still open. - Native Windows: home directory falls back to
%USERPROFILE%whenHOMEis unset (PR #1427) —xerj initskipped every home-scoped agent config and caches resolved relative to the working directory. The shared helper is now used by the MCP server, autoindex, the server and local auth too (PRs #1436, #1437, #1438, #1439). xerj initnever edits.mcp.jsonorAGENTS.mdwithout a backup (PR #1411) — a failed backup copy was ignored and an unreadableAGENTS.mdwas replaced; both now leave the file untouched. Teaching files are version-marked (PR #1444): a re-run upgrades an unedited older copy (with a.bak), keeps an edited one and says so, and treats an unmarked file as the user's.xerj coderenders a symbol hit's code, not its provenance digest (PR #1431, carries todie's #1390) —*-symbolshits printed the 42-charax_filedigest as the passage;ax_*keys are now excluded from the fallback, andcodeis used only on symbol-shaped records so a short raw-JSONcodefield (an error code, a CWE id) does not regress.xerj autoindex map --prefixis honoured (PR #1459, carries todie's #1391) — the flag was ignored and every corpus printed; scoping now happens in the server query as well as client-side, and a blankwall_secondsno longer renders aswall s.- Console bootstrap banner names a recovery that exists (issue
#1424, PR
#1426, Tong-bit-art) — it
suggested
xerj admin magic-link, which the binary rejects; it now says to restart the node, which prints a fresh link before enrolment. - Console: a superseded render never paints over a newer route (issue #1425, PR #1428) — a slow view could snap back over the one the user had moved to; also the root cause of a flaky browser test.
xerj searchon multi-word natural-language queries (partial for issue #1364, which stays open) — for queries of three or more distinct tokens, short glue words (EN/ES/PT/FR/IT/DE) are dropped from the match clauses so a symbol nameddeno longer wins, and passages on autoindex code documents prefer thecode/bodyfield over a bare name (PR #1484, carries todie's #1396).- Whole-file
xerj search/xerj defhits print a 1-based line (issue #1365, PR #1502, carries todie's #1392) — the engine also recomputes a refined code passage's start line. A hit whose best passage is adefs/titlefield still prints the bare path, a residual left open. - A failed thread spawn degrades instead of aborting the process
(partial for issue #1122,
which stays open) — under memory exhaustion
std::thread::spawnpanics, and the engine ispanic=abort. These paths now handle the error: thexerj sharetunnel fails on its own (PR #1479), an extractor reader thread fails its file rather than the whole run (PR #1480), and autoindex index workers, sync replay and object fetch degrade (PR #1481), and the phase-A scan pool falls back to the calling thread (PR #1501). Spawn sites elsewhere are tracked by the issue. xerjandxerj indexrefuse a missing or non-numeric flag value (issue #1435, PR #1452) —xerj --data-dirwith no value booted on./dataand swallowed the next flag; it is now a usage error.xerj --helpstates the served MCP tool count (issue #1443, PR #1455) — computed from the served tool list instead of a stale constant.
Added
xerj code --fuse-symbols, opt-in (issue #1237, PR #1457, carries todie's #1401) — autoindex symbol records now carryend_lineand a searchableimplbody slice (capped at 16 KiB); with the flag, a second BM25 leg over symbol bodies is fused with the file ranking by Reciprocal Rank Fusion (k = 60). Without the flag the default ranking is unchanged. Pre-existing state dirs resume without a rebuild.--auto-port, opt-in (PR #1487, mininxd) — with--auto-port,XERJ_AUTO_PORT=trueor[server] auto_port = true, a node whose ports are taken binds the next free ones, and a data dir held by another live node moves to a<data_dir>-Nsibling, so several agents can start nodes side by side. Off by default: a taken port stays a fatal startup error.
Performance
- autoindex indexes the plan once per snapshot instead of rescanning it for every file (PR #1483, carries part of todie's #1394). Also from #1394: read-lease counting moved off writer-hot cache lines and the graveyard lock (PR #1496), and the PDF worker supervisor wakes on pipe EOF instead of a 20 ms sleep poll (PR #1486). No figures are claimed here.
Security
quinn-proto0.11.14 → 0.11.19 (PR #1357, Tong-bit-art) —RUSTSEC-2026-0338through-0342.calamine0.26.1 → 0.36.1 andzip2 → 8 (issue #1358, PR #1398, todie) — removesquick-xml0.31.0 (RUSTSEC-2026-0097,-0098) from the dependency graph. The.xlspath stays process-isolated as in rc.95.
Measured
- The #1062 wrong-and-confident detection gate was run through the real
_watcherpath and it FAILS (issue #1108, PR #1322) — the unmeasured half of the rc.80_watcherwork. On the rc.94 release binary, a realPUT /_watcher/watchwithcondition.xerj_decide(one detection watch per label, one-vs-rest) evaluated a 120-email boundary tier and wrote 120 fire records into.xerj_alert_fires; the measurement reads only those fires. Wrong-and-confident = 0.2449 (24 of 98 fires with confidence ≥ 0.8 were wrong) against the ≤ 0.05 bar. The confidence compared is the raw/_decideconfidence the watcher fires on;p_calwas absent (the vote history has no labelled outcome pairs to fit), and because 18 of the 24 confident errors sit at rawp= 1.000 a monotonicp_calcould not change the verdict anyway. Published under the loses-does-not-ship rule: the_watchermechanism is real, but its default vote is not good enough to ship a detection watch as a "confidently correct" default, and no doc claims it is. Raw results, corpus generator and harness inbenchmarks/systemone-watcher-gate/. Closes #1108.
Documentation & site
- Docs that named commands which do not exist were corrected: the
troubleshooting page's
xerj support-bundle(issue #1433, PR #1434), the upgrades page'sxerj migrate/xerj verify(issue #1440, PR #1451), the rerank egress table's--interval(issue #1441, PR #1442), and the reference-coding opener prompt's removedxcscripts (PR #1450); the engine README's License link now resolves (issue #1445, PR #1446). llms.txtreaches the existingxerj corpus/xerj codecorpus path with commands that work (PR #1371); top-nav HUB, X and Discord links (PR #1352); a Simplified Chinese README (PR #1341, redwolf2019); the Trendshift badge (PR #1458).- Planning documents, not shipped features: the Corpus Hub v2 implementation plan (PR #1376) and its pre-registered proof-of-value measurement (PR #1385); a blog note that the #1138 trajectory corpus gap has been seeded (PR #1314, issue #1138 stays open).
Community & field reports
- Field reports: ES-compat flush-parity verification of rc.95 (PR
#1330, Tong-bit-art), Hermes
agent semantic search over an agent kanban board (PR
#1354, riazrahaman),
natural-language search on a Spanish-language R package (PR
#1367, sebollin),
shared-cluster retrieval and an
llms.txtaudit (PR #1373, todie), Windows autoindex + MCP (PR #1420, fjjjuv), a Hermes agent autoindexing skills and blog corpora (PR #1492, driphtyio), and two reference-coding reports, end to end and across the Omarchy-port and Fedora repos (PRs #1463, #1465, whelanh). - CLA signatures: Tyagiquamar (#1327), Tong-bit-art (#1328), redwolf2019 (#1342), riazrahaman (#1355), sebollin (#1366), fjjjuv (#1421), todie (#1429), mininxd (#1488), Jarvis636431 (#1499), and leninworld, whelanh, bryanbans, YongxinFu and ChenShuo2004 (#1495). The CLA check no longer deadlocks on a contributor's own self-signature PR (PR #1432, carries todie's #1404, with the inputs pinned to the event SHA).
Housekeeping
- Dead-code and comment cleanups: PRs #1447, #1448, #1449, #1453, #1454.
- CI and test reliability: PR #1422 (fjjjuv, transport auth tests retry a
port bind race); PR #1490 (the fuzz workspace's lockfile is seeded from
engine/Cargo.lock, because everyzip8.x release was yanked upstream and a fresh resolve failed); PR #1494 adds a manual workflow that prints the public half of the hub signing seed. It bakes no key into the binary:HUB_PUBKEY_HEXis still empty in rc.96. PR #1506 repairs two merge skews that made main red during the cut: theauto_porttests (#1487) against #1452'sResult-returning argument parser, and a comma the #1499 merge dropped from.contributors.
[1.0.0-rc.95] - 2026-10-12
Window since v1.0.0-rc.94: legacy Microsoft Office support in autoindex, hardened against hostile input.
Added
- Legacy Microsoft Office formats in
xerj autoindex(PR #1324) — OLE2 Compound File attachments, detected by content (never by extension), are now indexed instead of dropped as binary:.xls(BIFF, viacalamine) — one dataset per worksheet, one typed record per row, mirroring the.xlsxextractor..doc(Word 97+) — document text recovered through the FIB piece table, titled fromSummaryInformation. Text-only by design: no formatting, tables-as-rows, headers/footers, footnotes, or pre-Word-97 binaries (those fall back to a lossy read or are skipped).- Other OLE2 containers (e.g.
.epf1C data-processors) are named and skipped, not mis-parsed.
Security
.xlsparsing is process-isolated (PR #1324) —calamine0.26.1 is not panic-safe on hostile BIFF (craftedDIMENSIONS/SSTrecords force multi-terabyte allocations; malformedLbl/ExternSheetrecords panic), and the engine ispanic=abort, so a single hostile spreadsheet in an untrusted dump could abort the whole indexer. The BIFF read now runs in a short-livedxerj __extract-xlsworker underRLIMIT_AS/RLIMIT_CPU+ a wall-clock watchdog + process-group kill; the parent reads records back as NDJSON and degrades any worker crash/timeout to a skipped file, never callingcalamineon the raw bytes. Abiff_allocations_boundedpre-filter rejects the known over-allocation shapes before the spawn. Adversarially verified: the indexer survives every crafted vector while benign workbooks still extract. (Windows keeps the in-process pre-filter as a documented best-effort.)
Housekeeping
- Landing footer version stamps swept to rc.94 (PR #1325); the sweep is now part of the release-cut recipe.
Known gaps
- OpenDocument
.ods/.odt(zip-based) are detected and skipped (no extractor yet);.rtfindexes as raw text. Tracked for a follow-up.
[1.0.0-rc.94] - 2026-10-11
Window since v1.0.0-rc.93: a wave of external-contributor fixes landed alongside a backlog-fix sweep. Contributors in this window: Vinz2168, thomas-villani (Tom Villani), yogthos, and pierreaubert.
Fixed
- Hybrid
query+knnkept a page-local ranking on delete-bearing indexes (issue #1303, PR #1312) — a single historical overwrite/delete flipped the pinned-kNN compound tree off the exact index-wide BM25 path onto the stored-doc page heuristic, so the lexical half scored against a page sample instead of corpus-wide document frequency: correct on a handful of docs, unrelated at scale. The!deletes_presentgate on the pinned-kNN FTS projection is removed; the append-only path is byte-identical, with a regression test that uses enough documents that page-local and corpus-wide statistics diverge. - Bulk auto-id loads rejected the whole batch for one lone UTF-16 surrogate
(issue #1304, PR
#1311) — the turbo index path
validated each line with
serde_jsonIgnoredAny, which does not decode\uXXXXescapes, so a lone surrogate slipped through to the deferred whole-batchValueparse and failed every item. The bad document is now rejected per-item with a 400; the rest of the batch lands. - autoindex
--watchallowed two live owners of one journal/destination (issue #1232, PR #1307) — a session now holds a session-long exclusive lock (.autoindex.watch.lock); a second watcher for the same state dir is refused with a bounded, non-zero-exit message. - autoindex incremental
--no-graphreconcile aborted on sibling forks (issue #1230, PR #1309) — the frozen plan never persisted the per-repository scope genesis clustering uses, so byte-identical sibling repos could not be told apart; the scope is now persisted. - Second Brain console rendered a notes/beliefs persona on non-note corpora
(issue #1100, PR
#1310) — the console now picks a
neutral records/active frame vs the notes persona from the real
ax_formattally, so autoindexing a code tree no longer frames it as personal notes. - memtable
match/match_phrasetokenized with the wrong analyzer (issue #1280, PR #1295, thomas-villani) — the memtable now tokenizes with the analyzer the segment uses, so hit sets are stable across a flush. query_stringfield groups and negation (issues #1298, #1299, PRs #1300, #1301, Vinz2168) —field:( … )groups are scoped to the field and the classic parser builds one boolean per level with correct occur semantics.- A non-numeric value on a numeric field is a 400, honouring
lenient(issue #1284, PR #1293, Vinz2168). highlight.fieldsexpands field-name patterns (issue #1281, PR #1294, Vinz2168)._msearchreports the envelope-leveltook(issue #1288, PR #1289, Vinz2168).termsorder resolves percentile keys and rejects unknown paths (issues #1282, #1283, PR #1290, Vinz2168).- No empty
Warning: 299header on every response (issue #1285, PR #1291, Vinz2168). embedding.timeout_msreaches the proxy client in milliseconds (PR #1276, thomas-villani).index.max_result_windowabove 10,000 works;limits.max_result_windowis the node default (PR #1277, thomas-villani).- autoindex CLI argument hygiene —
xerj def --kind/--langstill requires the name to match (PR #1272), each watch route refuses flags it cannot read (PR #1273), the legacy-journal rebuild argv carries every index-shaping flag (PR #1274), and map-only--datasetis refused outsideautoindex map(issue #1231, PR #1308). thomas-villani / xerj-org. - Row-exact fast aggregations under resident ghosts (issue
#1267, PR
#1270) — bounds the 803 s
fast-agg finalize-verify window. Partial: #1267 stays open for the
finalize-catalog read-back walk and the
size > 0doc-lane residual, which still brute-scan.
Added
- autoindex AST extraction for Clojure and Scheme (PR #1305, yogthos).
Changed
- Dormant config keys named at startup (PR #1275, thomas-villani) — every accepted-but-unread key is announced as DORMANT at startup and no longer documented as working.
Docs
- Config/cluster/env/WAL docs describe the behavior the binary actually has
(PR #1278, thomas-villani);
--helpnames the CORS config and the real native-port default (issue #1286, PR #1292, Vinz2168). Field reports against live ES 8.13.4 and a reference-coding corpus index landed from Vinz2168 (PRs #1287, #1302) and pierreaubert (PR #1297).
Housekeeping
- Landing footer/version-stamp sweep so the
Landing constantsguard onmainis green (PR #1271), and the yogthos CLA signature (PR #1306). - Closed as already-fixed on
mainafter verification: the sorted-scan liveness bug #1212 (via #1013), the raw-JSON own-field recall gap #1244 (via #1246/#1251), and the corpus-index resume honesty bug #1173 (via #1174).
[1.0.0-rc.93] - 2026-10-08
Fixed
The autoindex finalize-verify probe fell off the columnar fast path (issue #1260, PR #1262) — two walls sat between the verify query shape (
bool.filter [terms …, exists <field>]) and the fast path:existshad no arm in the predicate compiler, and the deletes bail keyed on the MONOTONICghost_eventscounter, so an index that had ever seen an overwrite — the catalog is rewritten by every finalize-catalog — never re-qualified, even after merges compacted every ghost away.existsnow compiles to a columnar predicate and admission is per-segment via the ghost-position bitmap: empty bitmaps proceed (live == physical), dirty ones still bail. Measured on the live 402,814-doc autoindex-catalog crawl node: the per-digest probe ran 3.4 s on the pre-fix server and 2.7 ms warm on the fixed one (~1,250×; the first post-restart probe pays an 8.8 s cold page-in, verified to be one-time by a warm repeat). New integration filefast_aggs_exists_and_deletes.rs: six tests pinning fast-path/brute agreement across present / JSON-null / absent / empty-string exists shapes, the meta-field bail, the unmerged-ghost bail, merged-history re-qualify, and object-valued memtable docs. The served/bailed assertions read the process-globalFAST_PATH_AGGS_SERVEDcounter as a before/after delta, and CI run 37816544794 caught the six tests racing on it — the file now serializes behind one tokio Mutex (the engine half was never implicated: serialized, the meta-field query bails exactly as required).pack publishcould not survive a same-day rebuild under immutable releases (PR #1268) — the re-run path deleted today's PUBLISHED pack release and recreated the same tag, but a burned tag can never name another release (run 37826006767 hit the exact 36358399114 error pair and left the day without a published rust-vulns pack; repaired by hand under-r2). Both pack jobs now pick the first unused tag (date, then-r2,-r3, … by release + git-ref probe), retag past an immutable-release refusal instead of failing, and never delete a published release for a rebuild. The retention step also finally works:gh release list --jsonwas an unsupported flag, so it had silently matched zero releases since M6 — it now sorts bycreated_atvia the API (a-r2suffix breaks lexicographic tag order) and includes abandoned drafts.
Performance
- The finalize-verify barrier read back one changed group at a time — three
serial searches per file (PR
#1259, the #1183 count lane)
— cve-records projected ~50 h and vuln-fix-commits ~120 h of verify tail
against index phases measured in hours. Verify now batches per (index ×
1,024 content digests): record windows (terms filter + exact terms
aggregation) run 63 ms cold / 6 ms warm on the columnar fast path, the
catalog leg
_source-restricted paged fetch-count runs 31 ms cold / 14 ms warm, delete-aware, with mid-window total changes and count/total mismatches failing loud. ~1.2 M serial searches collapse to ~800 windows for cve-records. Failure messages and the first-disagreement group are unchanged; the progress denominator is now windows, so the line never sits at0/Nfor a walk's duration.cargo test -p xerj-autoindex --lib: 1299 passed, 0 failed. First live proof: the vuln-fix-commits rebuild completed on this client — 328,891 records, 97 batched windows verified.
Added
agent-session-trajectories, the first external-contributor corpus pack (PRs #1263, #1265; recipe and stats on corpus-hub in #1255 and #1264) — 70 sanitized multi-session agent trajectories from chunxiaoxx / Nautilus (demand anchored to #1138, wishlist slot #1210), CC-BY-4.0, 70 envelopes → 70 records with failure turns kept verbatim. The pack ships with its own ed25519 keypair:.pubcommitted attools/packs/keys/agent-session-trajectories.pub, the seed as thePACK_SIGNING_SEED_ASTActions secret, and a dedicated job in the scheduledpack publishworkflow — build, sign, consumer-path verify against the committed key, dated immutable release. First releasepack-agent-session-trajectories-2026-10-08published and consumer- verified from the public release URL (70 records, signature checked against theraw.githubusercontent.comkey).vuln-fix-commitslive on the hub with graded evidence (corpus-hub PR #1266) — project-kb fix commits: 328,891 records, 97 batched verify windows passed, and the pre-registered G7 suite blind-graded 4/5 PASS (three rank-1 needle hits, one rank-3, one miss where the 2020-era sleuthkityaffsfs_istatpayload lost to newer buffer-overflow payloads on generic terms). Graded evidence committed attools/xerj-code/hub/backlog/g7-vuln-fix-commits-2026-10-08-graded.json. The hub stands at 103 manifests / 105 live rows.
[1.0.0-rc.92] - 2026-10-08
Fixed
A corpus whose records carry prose in schema-named fields was invisible to
xerj code(issue #1244, PR #1246) — the pre-registered g7-cve-records-2026-10-08 suite measured 0/7 becauseresolve_fieldssent only the synthesizedtext^0.5: one shard mappedtextand its presence kept the own-field fallback from ever firing, while the records' actual prose lived in fields likecontainers_cna_descriptions(rank 1 at 18.76 queried directly). Own text fields now join themulti_matchas^0.5recall legs beside the standard fields — safe becausebest_fieldsexecutes as dis_max, so a leg can only win by genuinely outranking the content fields at half weight — gated to fields mapped by at least a quarter of the union's indices. Measured spectrum the gate sits on has an empty middle: schema prose repeats across shards (cve-records 37-96% of 118 indices) while README-frontmatter junk does not (exploit-pocs-2026: 2,143 own text fields, all ≤1% — the old alphabetical 24-cap kept a literal$commentand cutSummary). The mechanism fix is complete; #1244 itself stays open until the pre-registered G7 suites pass on the rebuilt corpora — the cve-records and vuln-fix-commits rebuilds were in flight at release time.A multi-family corpus still lost its minority family to that same gate (issue #1244, PR #1251) — vuln-fix-commits is project-kb repo source (body/defs/title) beside 29,240 fix-commit payloads whose prose lives in
message, mapped by 4 of 26 union indices against a gate demanding 7: the corpus answered every query from its tooling files while every payload was unsearchable (the needle was rank 1 at 36.35 queried onmessagedirectly, zero payload hits through the client). A field now joins when it passes the quarter gate OR any index that maps none of the standard content fields maps it, and the 24-leg cap ranks by mapping coverage instead of name. Exact-signature family grouping was measured and rejected: the mapper types only fields present in a shard's documents, so one family splits intomessage,patch,textandfiles,message,textshards and each half fails its own majority rule. Regression-checked on exploit-pocs-2026 (pinned query "MCPJam inspector 23744": needle README still rank 1, POC.py rank 3), whose 689 own-only indices (12% of the corpus) are newly reachable. #1244 stays open for the same completion condition as above: the suite reruns on the rebuilt corpora close it, not this code.A corpus whose primary content rides the plain-text family was invisible to
xerj code(issue #1254, PR #1258) — thetext^0.5recall-leg weight is the #1238 calibration for corpora where the text family carries mirror noise beside code-familybodyrecords, but otel-proto's primary content IS the text family (every.protodefinition is a txt-lines record), so the discount hid the corpus's point: the pre-registered G7 suite's four proto needles ranked 19/absent/39/23 behind semantic-conventions prose. Two earlier mechanisms were measured and disproven first — absence (a filename wildcard onax_file, which holds content digests; the correctax_pathprobe shows all 79 records) and txt-lines chunk-splitting (a simulated whole-document route ranked the needle 25th, worse than the real chunks' 19th: length dilution beats mass concentration). A corpus may now declarequery.text_weightin itscorpus.json(bounds 0.25..=4.0 enforced; outside them the client warns and keeps the 0.5 default), hub pins carry the block throughcorpus add --from, and the query path applies it to thetext^slot. Measured on the unchanged index bytes: all four needles 19/absent/39/23 → rank 1, the suite re-grades 5/5 against a ≥3/5 bar, and the no-hint path is byte-identical old vs new binary (diffed on cisa and exploit-pocs-2026). #1254 stays open only for the minorcode_files=0labeling gap in the index terminal line.
[1.0.0-rc.91] - 2026-10-08
Added
- The exploit-hub corpus, tested on three post-pin CVEs with and without it (PR
#1240) — the A/B eval write-up the
cyber-exploits group was built for: one analyst, two arms (plain Claude Code vs
Claude Code + the corpus), three CVEs published after the corpus pin, sealed ground
truth. On the needle (CVE-2026-23744, 36 indexed PoC repos) one
multi_matchround over 1,093,749 docs namedPOST /api/mcp/connectandserverConfig.commandand the one-request trigger, cited — the advisory alone could not; both arms then predicted the wrong patch and that loss stays in the table. On the analogy (CVE-2026-105844, no PoC anywhere) Exploit-DB entry 52528 supplied the denylist segments and both mechanisms the sealed 3.88.0 diff shipped. The empty stratum (CVE-2026-97332) is counted, not dropped. The negative control too: 23 s retrieval vs 2.1 s clone+grep when repo names are already known — the corpus is the discovery layer, not a fetch accelerator. n=3, one analyst: a case study with measured arms, not statistics, and the post says so. The same PR caught live client defect #1238 during the eval. Also ships thexerj-blogpostshouse-writing corpus at 8 records (this post included; the registry pin moves with the rebuild in #1242).
Fixed
existsinside a filter-contextboolhad no arm in the columnar filter executor, so the whole plan bailed and source-scanned every row (issue #1183, PR #1234) — keywordexistslowers to an empty-prefix dictionary range, numeric/boolean to an unbounded window (the same lowerings the standalone rootexistsuses), and text/semantic_textfields get a new source-backedSourceExistsleaf that pays one stored-source parse per term survivor instead of per row. Meta fields stay on the brute path. Measured on the standing node's 91,143-doc segment: the conjunction fell 9,638 ms -> handled by the plan. The finalize-verify crawl itself is larger than this fix: the count-context gate still excludesSourceExistsplans and a single tombstone disables the term-count fast path, so the exact-semantic-count shape on the 572,992-doc index still brute-scans — measured 2026-10-08 and left tracked in #1183, which stays open until a full xerj-search rebuild passes finalize with verified numbers.One sibling repo could fill a whole
xerj codepage, and the plain-text extraction family could outscore code-family hits (issue #1238, PR #1241) — two diversification fixes inxccode. A per-source-INDEX cap (MAX_PER_INDEX = 1, engaged only when the fan-out has at least as many distinct indices as the page wants, so a homogeneous corpus's handful of shard indices is untouched): measured on the live exploit group (query "MCPJam inspector 23744", 5,644 indices, 36 needle PoC repos), one sibling-CVE demo repo took 6 of the top 10 slots with a different file each — the per-FILE cap of #1137 cannot see that wall — and the capped page held 5 needle records where the old page held 1. Andtextjoins themulti_matchas a^0.5recall leg (same posture asdefs_expanded^0.5): at full weight one plain-text demo index outscored every code-family hit; at^0.5the family stays searchable (same 39 total hits) while needle docs keep their scores.
[1.0.0-rc.90] - 2026-10-07
Added
- A second community contribution from Vinz2168: the agent field report for the #1091
diagnosis session (PR
#1219) — one report under
user-feedback/16-agent-field-reports/, written by the AI coding agent that ran the session and reviewed by its operator, alongside the #1217 fix that session produced (rc.89). Every number in it comes from a command run in that session (the A/B harness and the ES-YAML runner output quoted in #1217). Filing it closed the tracker #1091 it diagnoses.
Fixed
The kNN exact scan could silently drop a document it had already captured (issue #1220, PR #1223) — two holes, both fixed. The scan read the memtable and the store snapshot with no generation check between them, so a concurrent flush or merge publication could land mid-capture and the "captured" set was half old generation, half new; and the admission predicate consulted the live version map, so a post-capture update of the same id made the captured copy look stale and the scan forgot it. Both walkers now bracket the capture (generation-checked) and admit candidates against the capture-scoped view, not the live one.
A no-match
delete_by_query/update_by_queryrun no longer flushes the index (issue #1222, PR #1227) — by-query runs flushed the member index FIRST, unconditionally, as the #1019 paging precondition; a run whose query matched nothing still paid it. Asize:0probe through the ordinary search path now runs first and a zero-match returns the zero response without flushing; a matching unflushed doc still counts and takes the flush arm exactly as before. This also corrects the #1222 root cause on record: the idle-flush fingerprint loop was already dead (Index::flush()clears the idle-probe pair since #878) — the measured "6,848 one-doc segments for ~3,005 writes" was this unconditional flush firing under the corpus apply loop's per-op defensive deletes. Live A/B on a throwaway node, one doc written withrefresh=false: old binary, no-match delete → 1-doc segment created; this build → 0 segments, and a matching delete still deletes (total 1 / deleted 1 / batches 1).The corpus apply loop asks before deleting: the per-group defensive
delete_by_queryis gated on a visibility probe (one leg of #1224, PR #1225) — every upsert used to fire a defensivedelete_by_query?refresh=trueeven when the index held nothing of that content, and one-record-per-file corpora pay one upsert per FILE: cvelistV5's cve-records build is 402,744 groups, i.e. 402,744 no-match deletes at a measured 5–16 s each under concurrent bulk load (the forcedrefresh=true), against asize:0term probe that answers in <1 ms. The delete now fires only when the probe sees something. #1224 stays open pending the end-to-end re-measure: the O(1) journal fix below removed the measured client-CPU ceiling, and the tracker closes on the completed apply-run numbers, not on the individual legs.
Performance
- Corpus apply: O(1) journal state writes and per-replay lookups (the client-CPU leg of
#1224, PR
#1228) —
Journal::sync_operation_stateran twice per replayed operation and cloned the entirePendingSync(the desired manifest with all 402,695 groups, the 402,744-entry operations Vec, the growing operation_states map) to append one small durable line: ~350 ms × 2 per file, measured on cve-records as apply at 1.18 items/s with 0.72 s client CPU per file, one core 82% user, RSS 5.7 GB. State writes are now validate → durable append → record in place (the journal line stays the transaction boundary);PendingSynccarries a lazily-built membership index; the replay path resolves groups and artifacts through a once-per-replay lookup table instead of O(402k) scans.xerj-autoindexlib suite 1299 passed / 0 failed; journal failpoint suites unchanged (same events, same order). The after-rate posts on #1224 when the in-flight re-measure completes.
[1.0.0-rc.89] - 2026-10-07
Added
- A guest post on the blog: "Six eval scars from a production agent-memory system" (PR #1211) — the community contribution from Chunxiao Wang (Assay / nautilus-compass), landed as a guest post with every measurement his and attributed: topic slicing beating token windows on recall@1 by double digits, an MTEB-leading embedder losing his embedder bakeoff on recall@1, qid grouping against leakage, three-valued retrieval judged with Brier 0.069, cached-input variance attribution, and the paired-vs-unpaired overstatement (a 4× claim collapsing under a paired re-run against an always-yes baseline). The XERJ-side section answers the question the post asks back, checked rather than guessed: 0 of 99 live hub corpora and 0 of 132 backlog rows are session or trajectory data, our chunk unit is file/section, and the memory features store curated facts retrieved by BM25 — so the eval shapes the post scars describe are not yet measurable on this engine, which is the honest answer.
Performance
- Exact semantic/kNN scans on passage-chunked corpora no longer re-parse every stored
document per query (issue
#1091, PR
#1217) — a
semantic_textfield whose documents carry passage vectors stays on the exact scan by design (309f0b6f), and that scan parsed each segment's stored documents (vectors included, ~53 KB a document on FiQA) to read two fields per candidate; they overflowed the hydration budget, the RSS spike tripped the memory-watermark drain, and every query paid the whole decode again. An unfiltered scan now ranks from per-segment vector columns that also carry_id/_seq_no, builds a missing column from the stored section's typed kNN projection (no_sourcereconstructed), and hydrates only the ranked winners row-selectively — without the generated embedding companions when the default_sourcewould drop them anyway. Answers are unchanged (bit-identical scores against the existing clone-everything oracle; 50/50 identical ranked ids on FiQA). FiQA, 57,638 docs, lexical embedder, same machine:semanticp50 3,865 → 261 ms,hybridp50 5,772 → 289 ms, memory-watermark crossings 25 → 0 (benchmarks/beir-hybrid/results/2026-10-07-issue-1091/). Filtered andscalar8scans keep the stored-document path; a segment stored as LZ4 (under 128 documents) still builds its column and hydrates its winners from parsed documents. The issue stays open: this is the latency half under the lexical embedder — the 13-26 s readings were taken under--embed-mode neuralon a loaded box, not re-measured here, and the capture-scoping follow-up the review surfaced is separately tracked in #1220.
Fixed
- The autoindex page walkers no longer read a timed-out partial page as end-of-data (part of issue
#1212, PR
#1213) — the engine applies a 30 s cooperative deadline
to any search whose body carries no timeout and answers a missed deadline with HTTP 200, PARTIAL hits
and
timed_out: true, which is legal on the wire; the xerj-search finalize read-back walked into one on a breaker-drained node (a sorted page answered 441 hits with the flag set) andcatalog_generation's short-page break read it as the last page — 56,441 of 58,568 documents certified.Es::search_pagenow sends every page with an explicit 120 s timeout (added only when the caller set none; above the slowest sorted page measured on that node) and retries the SAME page, bounded at six attempts, then fails loudly naming the index; all three walkers (catalog_generation,detect::scan_by_id, thedelete_catalog_hits_wheresweep) use it. The engine-side question — whether the 30 s default should stand — stays open on #1212. xerj corpus indexrefuses a run it cannot resume over a recorded build (issue #1214, PR #1215) — a state file that records a build pins the corpus to one url, one prefix and one state dir, but a failed conjunct (observed live:localhostvs the recorded127.0.0.1) fell through to legacy mode silently, and autoindex then opened its default hash-of-root journal — a DIFFERENT journal holding a stale pending sync from a retired attempt — which began re-applying 46,308 operations into the legacy namespace beside the recorded, still-unverified build. Becausexerj codereads the corpus namespace as one wildcard, both generations served and every hit was duplicated across incompatible groupings. Any failed conjunct now exits 2, printing each recorded value that did not hold and the two recovery routes (matching--urlto resume;--freshto rebuild beside it); no autoindex invocation runs from the refusal path.- The finalize catalog read-back re-walks while it falls short of the projection (part of issue
#1212, PR
#1216) — the standing rebuild's read-back walk read 9
full 1,000-hit pages, then an EMPTY page with
timed_outabsent: 9,000 of 58,568 documents, no flag, no error, while a terms aggregation on the same field counted all 58,568 and the same walk on an idle node minutes later read all 59 pages clean. Neither existing defense can see that shape (a 5xx retry only re-sends failed requests — this answer was 200; the #1213 page check keys ontimed_out— the flag was absent; page answers reporttotal {relation: "gte"}under the track_total_hits cap, so total cannot validate a page either). The one signal that distinguishes a starved walk from a complete one is observed-count-versus-expected, which the barrier already computes:catalog_generation_complete()now refreshes and re-walks while the observation is short of the sealed projection's document count, bounded at three attempts, and returns the LAST observation when all fall short sovalidate_observedstill produces its loud, precise mismatch. The #971 prior-run stray sweep is deliberately not re-walked: it is a subset check, an unflagged truncated page can only under-report strays, and a false pass is impossible. The engine-side truncation itself is fixed in this release by PR #1218; #1212 stays open until the standing xerj-search rebuild verifies against the released fix. - Scans no longer drop their captured hits when a merge or flush publishes mid-walk (part of issue
#1212, PR
#1218) — the #1013 capture bracket validates a
writer-free instant and the scan then runs on frozen state (segment snapshot, read lease, captured
memtable), but three per-hit liveness checks read the LIVE version map and dropped any hit whose
entry no longer named its own segment: a merge publishing after the capture repoints surviving docs
at the merged segment (
snapshot.rcu+ version-repoints), a flush repoints off__memtable__, and the frozen hits then failed the equality check and were silently dropped — a short or empty page, HTTP 200,timed_outabsent, self-healing on the next capture. Exactly the read-back shape the client legs above defend around, now fixed at the source. A scan may only apply version entries that describe the world it captured (CapturedEntryVerdict): an entry on__memtable__or a captured segment decides live/invisible as before; an entry naming a real segment outside the captured set is post-capture and invisible in both directions. The design follows tantivy's reader-generation pinning (SearcherInner behind an ArcSwap swap, segment readers under META_LOCK, live generations pinned via Inventory — the snapshot as the only scan authority, adapted not copied). Park-hook regression tests park a scan between capture and walk across a merge and a flush publication; both lost their captured hits before the fix. The memtable ghost filter, the scored walk and the fused-bool residual walk are all capture-scoped now. #1212 stays open until this ships in a release and the standing xerj-search rebuild completes against it; the kNN exact scan's admission predicate still reads the live map (a pre-existing predicate #1217 extracted into a helper, window ~15× narrower for it) and is separately tracked in #1220.
[1.0.0-rc.88] - 2026-10-07
Added
xerj autoindex --code-analyzer standard|code(issue #1198, landed as PR #1205) —codedeclares the engine's built-in identifier-aware analyzer (standard tokenizer + word-delimiter + lowercase:snake_case,camelCaseand dotted identifiers gain their sub-words, the original token is kept) on newly created datasets that hold at least one source file, so a query that says "connection pool" reachesget_connection_poolthrough BM25. Defaultstandardkeeps today's election unchanged. Withcodeelected the #1059 stemmer defers for that dataset. It rides the #1059 create-body path: committedindex_identitydigests do not move and existing indexes keep their analyzer. Measured on SWE-bench Lite file localization: acc@5 +0.11 [+0.03, +0.19] on a pre-registered held-out sample; cost ~15% index size and ~2.4× text-analysis time._cat/indicesand_cat/segmentsaccepth,vandbytes(issue #1201, PR #1208) — all three output parameters were silently ignored on both endpoints. A shared renderer now serves them the way ES does:hselects comma-split columns (*expands, unknown names drop out, and aformat=jsonobject carries exactly the selected columns as strings),vturns on text padding — which is the only mode that pads, a plain response joins columns with one space and still prints the header over zero rows — andbytesresizes the byte columns, wherebis raw and the one-letter units are decimal (10³) while the two-letter ones are binary (2¹⁰); an unknown value for either parameter is a 400 that names the valid set rather than a silently ignored flag.
Fixed
_cat/segmentsreturns real segment rows (issue #1202, PR #1208) — the endpoint used to answer 405/empty. It now reads per-segment truth from the store: id, physicaldocs.count,docs.deleteddecoded from the segment's own ZTB2 tombstone section, size, and the remaining ES columns, over a 14-column ES-compatible header, including the whole-cluster/_cat/segmentsform. The per-segment counts follow the store's real geometry, pinned by a test: a delete's tombstone persists in a later segment than the document it kills, so a segment'sdocs.deletedis the tombstones persisted in that segment and the live count is the sum invariantΣ docs.count − Σ docs.deleted— the same pre-merge over-count_cat/indiceshas always had for superseded update copies, now visible and documented instead of guessed at. Row count matches_statssegments.count(also pinned).- The rc.87 landing sweep is on main, and the landing constants guard is green again (PR
#1206) — the rc.87 roll stamped the engine and
Cargo.lock but not the 92
V1.0.0-RC.86landing-page download stamps the rc.86 release commit had included, so the newest-tag comparison in CI's landing-constants guard went red for a day after the rc.87 tag. The sweep moved all 91V1.0.0-RC.86stamps (plus the lowercasedata-latest-tagspan on the index) to RC.87 and regenerated the sitemap (55 lastmods moved). This is also the rule going forward: every rc roll includes the landing stamp sweep as its own commit, which this rc.88 roll follows (54 pages swept to RC.88, sitemap regen, 183 URLs). - The finalize-catalog read-back no longer trips the server's query-memory breaker (part of issue
#1183, PR
#1207) — the catalog read-back sent
size:0 + track_total_hits + min/max aggson each dataset's time field, and an agg with no columnar fast path makes the engine deep-clone every matching document into owned Values (~2 KB a doc) billed againstlimits.max_query_memory_mb. On the rc.86 xerj-search rebuild one ~570 k-record dataset estimated 1.1 GB against the 512 MB default, the server answered429 circuit_breaking_exception: [request] query allocation (1.1gb) would exceed limits.max_query_memory_mb=512MB at [aggregation corpus materialisation], the client retried for its full 600 s envelope and honestly aborted (build b1791330708, 1,010,833 records indexed, existing corpus untouched) — retrying could never succeed because the request itself was the problem. The count now carries no aggs (a count needs no corpus) and the time bounds come from twosize:1searches sorted asc/desc on the time field under the dataset's own filter — a sort reads doc values and clones nothing, and the first hit IS the extreme, so it is exact for the same reason the agg was.epoch_ms_to_iso8601_utcmoved toxerj_common::schemaso client and engine render numeric timestamps through one helper and catalog documents keep the agg'svalue_as_stringshape. The finalize refresh — previously 1,479 serial per-index round trips that read as a stall for the better part of an hour against a node whose memory breaker was draining between them — is windowed: 50 comma-joined index names per request, one progress tick per index. This is the read-back half only: #1183 stays open until the standing xerj-search rebuild completes end-to-end against this client, which is the proof the fix was cut for.
[1.0.0-rc.87] - 2026-10-07
Performance
- Phase-B bulk bodies are coalesced across files (issue #1147, PR #1195, merged just past the rc.86 tag) — the per-file round-trip ceiling on corpus ingest, measured at ~10⁵ records/node-day on the ghsa-scale mirror class. Each file's records used to ship as their own
_bulkbody, so ingest throughput on a mirror of many single-record files was bounded by per-file round trips, not document volume. The shared bulk pipe now coalesces bodies across files: extraction and sending overlap, up to 8 bodies ride the wire at once (matching the server governor'smax_concurrent_bulks), a worker about to park ships what is buffered once half the live workers are parked, the all-parked flush stays as the nothing-more-is-coming drain, and submit parks under backpressure when every wire slot is busy. Per-file ordering is preserved (a file's own cut chunks are sent synchronously by its worker). A/B'd on two corpus shapes with no regression — rustsec (1,965 files / 6,829 records): whole run 10.1 s → 9.7 s with phase B at parity; pysec (7,718 mostly-one-record files): 17.4 s → 16.9 s — and the structural win (one send per flush trigger, not one per file) shows at #1147's scale, which the standing corpus rebuild re-measures.
Fixed
xerj corpus indexnow proves the store answers a search before it says "searchable" (issue #1183, PR #1197) — a resume over an index left with a dangling segment by a killed finalize printedcorpus 'xerj-search' searchable: 642 recordsand exited 0 while the store answered every_search— and the salvage path's owndelete_by_query— withstore_exception: Segment … not found._countis metadata-only and returned 642 over that same store, so every verification built on counts certified a corpus that could not be queried. Verification now has a third leg: onesize:0_searchover exactly the globxerj codereads ({prefix}-*), with the node's own reason surfaced when it fails. In the build arm a build whose verification search fails is not verified — nothing is switched, the existing index stays whatxerj codeserves, and the unsearchable build's own indices are retired by exact name (an unsearchable index left under the namespace is what poisoned the live one); the #367 salvage can no longer record a store that cannot be searched. The update arm refuses to rewrite the ledger over a live corpus that cannot be searched, and the legacy arm — the exact live failure — refuses to print any verdict at all, because one unsearchable index anywhere under the wildcard makes "the corpus is queryable" untrue whatever the counts said. The probe is asize:0search, which the server already classifies as machine traffic in the audit log (#1109), soxerj gainstill measures only the user's queries. The finalize-catalog deadlock that produces the dangling segment (item 1 of #1183) remains open, tracked in the same issue.- A request retrying after a transport error or a 5xx now says so on stderr (part of issue
#1183, PR
#1199) — the autoindex client's retry envelope
is bounded (six attempts at a 300 s request timeout each, ≈31 min worst case) but was
completely silent on everything except HTTP 429, which is how the #1183 incident read as a
deadlock: a finalize-catalog request against a memory-pressured node spent eight minutes inside
that envelope with zero output, and since
reqwest::blockingparks the caller on a oneshot, every thread's/procwchan readfutex_do_wait— indistinguishable from a lock cycle at that granularity. Diagnosis in the issue: it was never a deadlock; the silence was the defect. A loading run (announce on) now printsautoindex: search failed: …; retrying (attempt 2 of 6, … s in this request so far)on the first failure and at most once per 30 s after, sharing the 429 notice's clock so alternating failures cannot double-announce; probes, MCP clients and--quietstay silent. The server-side stall root cause (the reference node running at its memory breaker during large rebuilds) remains open under #1183 and the #1122 memory-ceiling class. - A JSON record with no prose now carries a synthesized
textpassage, and a dataset that maps no text-searchable field says so at plan time (the build-time half of issue #1158, PR #1200; the query-time half — falling back to an index's own text-typed fields — shipped in rc.86 as PR #1185). Raw-JSON corpora indexed as typed records (ids, enums, numbers, dates) were invisible toxerj codepassage search: 36,263 GHSA advisories indexed to 193k records that answeredNo passage matcheswhile the same index returned 10,000+ hits over its typed fields. Now a record whose flattened fields carry no prose-like string (≥ 24 chars and ≥ 4 tokens — below that a string is an id, an enum or a date, and mapskeyword) gains atextpassage rendered from its own fields (key: valuelines, values capped at 256 chars, the passage at 4,096, scalar arrays joined so identifiers stay matchable); a record that already carries prose is left alone — the query side already searches it, and a rendered key/value soup would only outrank the real prose — and a record that owns a field namedtextkeeps it untouched. Separately, any dataset whose sampled mapping contains no text-searchable field (a csv of enums, a keyword-only XML set) now warns at plan time:dataset 'x' maps no text-searchable field — 'xerj code' passage search will find nothing in it (sampled fields: …), through the progress surface so--quietstays silent. Scoped to the JSON family (whole-file and JSONL) — the measured class is raw-JSON mirrors; the same helper can ride the csv/yaml/xml data sites when a real corpus needs it. Existing state dirs re-provision their JSON datasets once to gain the field; every already-built corpus keeps working, and the 100 hub corpora are unaffected (their graded snapshots map standard fields).
[1.0.0-rc.86] - 2026-10-06
Security
- Audit records now carry a server-derived
actorclass (user/machine) (issue #1109, PR #1194) — the audit log's search entries said WHO ran a query but not WHO IT SERVED, so the only wayxerj gaincould exclude autoindex's own machine traffic (#1105) was to re-derive it client-side from the note string (size=0probes) and the resource name (catalog sweeps): a heuristic a future note-format change would silently break, and one any caller writing audit-shaped entries could satisfy. Every audit entry now carries anactorfield derived at append time by the SERVER from what the request itself is — asize:0counting probe or a search of theautoindex-catalogmaintenance index ismachine; everything else isuser. Nothing a client can set (header, principal, body) is consulted — the spoofableX-Xerj-Internalheader proposal from #1107 stays rejected. The class is part of the hash chain: flipping an entry fromusertomachine— the dodge that would remove unfavourable traffic from thexerj gainhit-rate denominator — breaks the chain like any other edit. Entries written by older binaries deserialize withactor: "user"(which is what they all were), so mixed-version logs still verify, andxerj gainprefers the field on new entries while keeping the note/resource derivation as the documented legacy fallback for old ones.
Fixed
Raw-JSON corpora answered "No passage matches" for every query (issue #1158, PR #1185) — a raw-JSON corpus (advisory mirrors, OSV records) maps none of the standard content fields (
body,text, …); thexerj codefield resolver then fell to its "never an empty list" floor,["body"]— a field no index in the corpus maps, and on this engine an unmappedmulti_matchfield collapses a multi-token query to zero hits with no error. The corpus built clean (193k docs) and was then invisible: 0 hits underxerj codewhile the same indices returned 10,000+ hits queried directly (ghsa-db round-1 G7 0/5 was this). When no standard field is mapped, the floor is now the corpus's OWN text-typed properties (ax_*provenance excluded, sorted, capped at 24), and the passage renderer's last resort becomes the source's longest non-empty string field instead of an empty passage under a healthy header. Measured A/B on an advisory-shaped mapping:fields=["body"]→ 0 hits (the defect);fields=["details","summary"]→ 8 hits (the fix). Works retroactively on already-built corpora; the build-time half (extractor-side passage + a zero-text warning at index time) stays open on the issue.embedding.mode = "proxy"with an unusable endpoint silently ran lexical (issue #1189, PR #1193) — an emptydefault_endpointor a typo'd URL (htp://…) produced a startup WARN and a lexical embedder; indexing and semantic search then "succeeded" while writing lexical feature-hash vectors into the index, and/v1/embedding/identityreportedbackend: lexicalfor a config that saidproxy. An explicitmode = "proxy"now fails closed at startup with an actionable error in both the embedder and the identity path (including the reporter's scheme typo, which carries the underlyinghttp or httpsmessage); the auto path (endpoint set, mode not pinned) keeps the documented warn-plus-lexical fallback.xerj autoindexnow names the live embedding backend in its run output so a degraded backend is visible where the work happens.embedding.batch_sizewas never read (issue #1190, PR #1193) — the setting documented "Maximum documents per embedding API call (default: 64)" but no code read it: one_bulkof 20 documents left the proxy as a single request with 20 inputs, and an autoindex run was observed sending a 192-input request. Providers cap the requestinputarray (OpenAI 2048, self-hosted gateways often far lower), so a large bulk could fail on request shape, not load. The proxy now splits every batch into concurrent requests of at mostbatch_sizeinputs (each with its own retry/backoff and concurrency permit; the semaphore now limits in-flight HTTP requests, which was its purpose) and reassembles the vectors in input order. Verified by a mock-endpoint test: 10 texts withbatch_size: 4leave as three requests of 4/4/2 inputs and the returned vectors keep input order across chunk boundaries.Long BM25 queries silently fell back to per-segment IDF/avgdl (issue #1186, PR #1188) — the index-wide statistics pre-pass (#188) budgeted
(fields + terms) × segments ≤ 4096, so a many-term query on a multi-segment index crossed the cap, the pre-pass silently declined, and every segment scored against its owndf/avgdl: padding a query with 1,500 terms that match NO document reordered the whole top-5 (reported1-1:1.6797521→8-4:1.6904112; after_forcemerge?max_num_segments=1both queries agreed — the merge had nothing to do with it, it just brought the probe count back under the cap). The budget is now split by cost class: field probes (one mmap'd FST per segment × field — the widequery_stringexplosion the cap existed for) stay at 4096, term probes (one FST seek per segment × term — the same order of cost execution itself pays for postings) get their own 1,048,576 budget, and every decline now logs a warning instead of changing the ranking silently. The reporter's literal repro (1,502 terms over a 32-core box's ~128 flush segments, ≈192k term probes) and a 4-shard/32-segment variant both now score identically to their unpadded forms, bit-for-bit.xerj code --mode hybriddropped lexical-only indices from the BM25 leg (issue #1146, PR #1177, Vincenzo Lombardo) — on a corpus where only some indices mapbodyassemantic_text, the fused request went to the capable set alone (a semantic leg 400s on plain text), so every lexical-only index vanished from both legs while the note said "BM25 over N index(es)": 1 hit under hybrid where--mode bm25returned 20. The lexical-only indices now get the same nativehybridwith its BM25 leg alone; the engine fuses per index, so both responses carry RRF scores on one scale and are merged by_scorethe way the engine merges indices — no client-side fusion. The note names them ("lexical-only (BM25 leg only, merged by score)"). Same repro, rc.83 build: hybrid -k 20 → 20 hits (19 lexical-only + 1 semantic).An index's analysis configuration could not be verified over the wire (issue #1092, PR #1184, Vincenzo Lombardo) — search honoured a declared
analysis.analyzer.default, but nothing an agent can read said so.GET /{index}/_settingsandGET /{index}replayed an in-memory copy of the create body: after a restart it was empty, so the analysis block (and every other create-time setting) vanished, and a top-levelanalysisbeside anindexblock was never echoed at all. Both now reportindex.analysisas the block the index's analyzer registry was actually built from — none for a pre-#204 index whose declared block is not honoured — and fall back to the persistedsettings.jsonwhen the in-memory copy is gone.POST /{index}/_analyzewas an inline lowercase splitter that ignored the index; it now runs the index's real pipelines: noanalyzer→ the index's default,field→ the analyzer that field is indexed with, a named analyzer from the index's own registry. Ad-hoctokenizer/filter/char_filterchains (names or inline definitions) are built by the same registry code index creation uses, withkeywordas the tokenizer when only filters are given. Unknown analyzers, tokenizers and filters,fieldon the global endpoint andnormalizerare 400illegal_argument_exception. A missing index is 404. Checked side by side against OpenSearch 3.9.0: status codes, error types, tokens, offsets and positions match on all 24 probed requests. Only the wording of the reason differs on 3 of the 400s.sort: [{"_doc": ...}]is arrival order, not_idorder (PR #1127, Saurav Kumar) — the_docsort key projected the document id string, so_docpages ranked lexicographically by_id(matching neither ES index order nor XERJ's own stored order). The key is now the document'sseq_no: numeric arrival order, unique per live doc (no ties), and a workingsearch_aftercursor._idsort still echoes the id string.
Community
- Two field reports, two fixes, and one first-time contributor landed in
this window. Vinz2168 reported the hybrid BM25-leg defect on a real
mixed-mapping corpus and verified the fix on it
(#1178); the same reporter's
_settings/_analyzework (#1092 above) went in through a maintainer rebase (#1184). Thomas Villani measuredxerj codeas a file-localization retriever over SWE-bench Lite and published the numbers (#1187). Saurav Kumar (Ravandevil25, whose_docfield report and CLA landed in rc.85) had his_docarrival-order fix (#1127 above) merge after its CLA wait — his first code contribution to the project. - Release-download metrics chore for 2026-10-06 (3,114 assets, 1,730
binaries — direct commit,
[skip ci]).
[1.0.0-rc.85] - 2026-10-06
The cluster-ring-works window. One defect fix and one docs correction, both born from the rc.83 three-node ring tests: the election churn that made a ring with a dead peer elect a new leader every ~250 ms forever is root-caused and fixed, and the clustering page now describes what the ring actually does instead of an unshipped replication design.
Fixed
- A ring with a dead peer churned elections forever (issue
#1168, PR
#1179) — the documented
convention lists the full membership in
[cluster] peerson every node, self included, and nothing removed this node's own entry before the membership reached Raft. Two compounding consequences: the peer count included self, so a three-node ring computed a four-member cluster whose majority was three — unanimity, zero fault tolerance — and self was also a routable transport peer, so each candidate sent its ownRequestVoteto itself over loopback TCP where a re-grant counted the self vote twice. After the leader died, no survivor could assemble three real votes, and the double-counted self votes let the surviving pair hand each other phantom wins while leapfrogging terms indefinitely. Self is now excluded at three layers (config parse inmain.rs,RaftNode::new, andTcpTransport::new), each logging a warning when it fires. Measured on a three-node loopback ring (tick_ms=50, kill the leader, 240 s observation): rc.83 churned through 94 terms and never stopped; with the transport starvation fix alone (per-peer outbound queues, first commit on the PR) the churn got faster — 909 terms, which is what pinned the real root cause; with self-exclusion, a new leader is elected ~250 ms after the kill (one split-vote retry) and the ring then holds zero elections for the remaining 239 s, with the second survivor never even timing out. Three new tests, including the ring acceptance scenarioring_with_full_membership_lists_survives_leader_deathdriving threeClusterNodes constructed exactly as the server passes the full-membership config.
Documentation
- The clustering page described an unshipped replication design
(PR
#1180, issue
#1170 context) —
docs/clusteringclaimed metadata replication, document replication, node drain/activate/remove endpoints, region split/merge, and aNodeStateAPI, none of which exist in the code; it even invented/_cluster/healthoutput. The page now states what the ring does today — leader election (~250 ms failover, citing the #1168 fix), an HMAC-authenticated cluster transport with dead-peer isolation, and a durable local Raft log — and says plainly that metadata and document replication are roadmap (#1170), with a "when not to cluster: today, almost always" section. The same sweep corrected the docs-index cards across 29 pages (145 replacements), the troubleshooting election-flapping entry (sustained churn is a defect now, not a tuning knob), the rolling-upgrade page (the cluster wire has no version negotiation — full stop/start, not rolling), and the clustered backup section (indexes are node-local today).
[1.0.0-rc.84] - 2026-10-06
The tell-the-truth window. Every signal an operator reads — cluster health, node counts, corpus-index exit codes, the landing counts — now reports what actually happened instead of what was configured or hoped for. A small engine window on top of rc.83's corpus-hub hundred: two cluster-honesty PRs, one corpus-tooling honesty PR, and two docs PRs (#1167, and #1163 by thomas-villani, his fourth merged PR).
Fixed
- Cluster endpoints fabricated topology (issue
#1169, PR
#1172) —
GET /_cluster/health?wait_for_nodes=NechoedNback asnumber_of_nodeswithstatus: greenandtimed_out: false, so the multinode smoke suite "converged" on a single node that had no cluster at all (the handler comment said so out loud). Health now reports the real configured membership (1when cluster mode is off, else the distinct own + peer addresses), an unsatisfiablewait_for_nodestimes out honestly (408,timed_out: true, count stays real), and upper bounds (<=N,<N,le(N)) are enforced instead of collapsed to "always satisfied"._cat/nodeslists every configured member (peers with-placeholders, no invented metrics),_nodestotals the membership whilesuccessful: 1reflects that only self answered, and_cluster/statereports this node's actual cluster address instead of the never-bound 9300 default andmaster_node: nullon a configured ring rather than every node claiming mastership. The multinode smoke case now skips with a recorded reason instead of passing vacuously;cluster.health/20_request_timeoutpasses through the real timeout path. Cluster config that could only fail silently is refused at load (issue #1171): a peer label differing from its address (which made the node lose every election with no diagnostic of its own) andcluster.portcolliding with this node's own rest/grpc/es_compat port (which degraded the node to single-node with green health). - A degraded cluster boot was invisible after the first log line
(issue #1171, PR
#1175) — a configured
ring whose TCP transport failed to bind, or whose cluster-state storage
was unavailable, degraded to single-node with one ERROR line at boot
and then
status: greenforever on/_cluster/health: the operator believed the node was clustered and no API could say otherwise. The boot outcome is now recorded on the application state (ClusterTransportStatus: Disabled / Running / Degraded) before the first router exists, and health answers from it:status: yellow,number_of_nodes: 1(the node actually serving — counting the unreachable configured peers would be the same fabrication #1169 removed), and an additivexerj_cluster_transport: "degraded"marker that says why it is yellow, present exactly when true. Waits stay honest from the degraded side too (wait_for_nodes=2of the one serving node → 408 with the real count), and red still outranks the degraded-yellow (an unopenable index remains the headline, with the transport marker beside it). xerj corpus indexreported a converged state as a failure and an uncountable node as "no new records" (issue #1173, honesty half — the crawl-throughput half stays open, PR #1174) — the resume cleanup's_delete_by_queryagainst an index a namespace swap had already retired answered 404 and killed the run (a missing index cannot hold the stale documents the cleanup exists to remove); 404 is now the answer "deleted nothing", the same rule_countand_cat/indicesalready follow. And when a failed run's before/after counts could not be read from the node, the tool asserted "wrote no new records" — that claim now requires both counts to be known; an unreadable count prints UNKNOWN.- The hub counts went stale again (PR
#1167) —
llms.txt:51and the corpus-hub blog's dated update note still said75 live / 104 backlog rowsfrom 2026-10-02 while the registry had reached 100 live / 119 backlog on 2026-10-05. Both surfaces now say 100/119 and name the rc.83 skill-table milestone — the same line, the same drift class, as the llms.txt:50 pack-status fix queued after rc.82. - EPUB and notebook answers that lagged the extractors (PR
#1163 by
thomas-villani) — the notebook
search and EPUB answer pages were rewritten from fresh captures on the
rc.83 build (the first published binary with both extractors), moving
the fact-check matrix rows for notebooks and EPUB to GREEN; the
Parquet row widened to Parquet/Arrow/HDF5/NumPy/pickle (named with an
export hint, still no reader). The man-page fact-check aliases had held
a literal
\x08since a scripted edit in #1130's docs commit decoded\binto the control character — they now use real word boundaries, andfactcheck.py --self-testfails if any alias or rule pattern holds a control character, so the class cannot silently return.
[1.0.0-rc.83] - 2026-10-06
The corpus-hub hundred release. The engine window is small — one retrieval
fix and three extractor PRs — while the reference-coding catalogue reached
100 live, licence-reviewed, G7-graded corpora at
hub.xerj.org, every one of them now named in the
xerj-code skill's domain-selection table (PR
#1164; the five SKILL-row
additions of the rc.82 window rode in as PR
#1157).
Fixed
- The version roll that missed itself (PR
#1155) — the rc.82 release
PR bumped only the download link's
data-latest-tag; the 91 footer brand spans across the landing pages still readRC.81, and the release's own CI could not see the stale ones because the newest tag at push time was still rc.81. All spans now roll with the release and the guard compares against the release being cut, not the last one tagged.xxhashis bumped to 0.8.19 in the same PR (the dependabot PR it supersedes was #1152, closed). - The vector leg of
xerj codehybrid/semantic retrieval (issues #1145 and #1148, PR #1153 by thomas-villani) — the semantic clause went out with nok, so the parser default (10) capped the vector pool whatever the request size said:--mode semantic -k 20returned 10 hits, and hybrid fused BM25@50 with vector@10. The clause now carries the fetch window ask. And--langsat beside the semantic clause inbool.must, a shape the engine's vector dispatch does not peel — the request silently fell through to the lexical path and answered 200 with zero hits. The language constraint now rides inbool.filter, on both hybrid legs, keeping the semantic clause the bool's singlemust— the exact shape the dispatcher peels. - Corpus generation hygiene (issue
#1136) — stale index
generations no longer survive beside a corpus, silently poisoning its
namespace. Three changes in
xerj corpus index/xerj code: a legacy-mode run (a state file that names no build) now retires the stamped generations it supersedes — the measured shape wastldr-pagescarrying an unstamped 38,554-document generation beside two stamped ones, mixed into everyxerj codeanswer; a build whose namespace listing the node never answered says so loudly and retires nothing instead of reading the empty list as "nothing to retire"; and every verified--freshswap ends with an epilogue naming whatever still sits underxc-<corpus>-outside the build readers are pinned to.xerj codenow also prints a stderr notice when a corpus has no verified build recorded and the query is widening to the whole namespace. - HTML tables and paragraph boundaries in autoindex (PR
#1154 by
thomas-villani) — a table on a
page indexed as one document lost its text twice over: it was not the
dominant table (so not emitted as rows) and cell text never reached
the body. Small tables — option lists, version matrices, the common
case on documentation pages — were silently unindexed. The rows now
go to the body in place, one
|-joined line per row, the shape PPTX tables already use. And<p>/<div>/list ends now end a paragraph, sosplit_sectionsbreaks pages at paragraph ends instead of hard-cutting every 2 KB mid-word behind a heading emitted as its own empty section. - EPUB books in autoindex (PR
#1159 by
thomas-villani) — one record per
spine document in reading order, cut at the table-of-contents anchors,
with the book's
title/author/languageand each chapter's own title from the nav or NCX. DRM-encrypted parts are detected and skipped, and per-part and total byte budgets apply, so a hostile container cannot balloon the index. - A section the next one repeats whole is not emitted (PR #1160 by thomas-villani) — a short paragraph (a chapter heading, seen on a Gutenberg EPUB as "CHAPTER 26. Knights and Squires." alone) followed by one too big to join it was emitted as a section of its own, and the next section then carried it whole as its overlap: a record with no text of its own. A section shorter than the overlap now stays and leads its successor instead.
- Jupyter notebooks, and data files that cannot be read, in
autoindex (PR #1161
by thomas-villani, landed via the
maintainer rebase
#1162) — a notebook is
grouped the way a reader sees it: a section starts at each Markdown
heading and holds every cell to the next one, code fenced in the
kernel's language (R and Julia notebooks say so), text outputs capped
and ANSI-stripped, images skipped, nothing ever executed; the
cellfield gives the 1-based index an agent needs to find the section in Jupyter. Parquet, Arrow, NumPy, HDF5 and pickle files — binary shapes no extractor should parse — are now named records that say which tool exports them to an indexable form.
[1.0.0-rc.82] - 2026-10-05
The documents-people-actually-have release. Three extractor PRs from a
first-time contributor extend autoindex to the formats that were still
read-as-text: Unix man pages, and the two spreadsheet shapes real
workbooks ship — vertically merged cells and two-row grouped headers.
Every code change in the window came through the community surface:
the extractor work is
thomas-villani's first contributions
to the project, and the window also carries the xerj code retrieval
fixes and two field reports.
Added
- Man page extraction — roff
man(7)sources read as documents, one record per.SHsection: the.THtitle (LS(1)), the NAME summary, font escapes dropped,\(em-class specials become the characters they name, no-fill blocks keep their layout, and a tbl table becomes one|-joined line per row. Enough of the roff request language is read (.dsstrings,.if/.ie/.elconditionals,.demacro bodies) that pod2man's preamble needs no special case; mdoc(7) pages and.sostubs stay text, as before. Ported from the man parser in all2md (parsers/man.py, MIT), reduced to plain text with attribution (PR #1134, thomas-villani). - XLSX merged cells fill down — a cell merged DOWN over several rows
(a pandas MultiIndex export, a category label beside its line items)
stores its value only in the top row; every covered row now gets it, so
region: Eastmatches all of East's rows and atermsaggregation counts them. Merges across columns are deliberately not expanded (PR #1132, thomas-villani). - XLSX two-row grouped headers — a header two rows deep (group labels
above the column names, pandas' MultiIndex columns) names each column
from both rows:
Q1_Jan,Q1_Feb. Three-or-more-row headers keep their two lowest rows; on sheets over 64 MB the sample keeps one (PR #1133, thomas-villani, stacked on #1132). - Docs: extracted formats —
.pptx/.xlsxlisted as extracted formats across the docs and the retired "no XLSX/PPTX extractor" caveats removed (PR #1130, thomas-villani). - The xerj-code skill points at the whole hub — per-project corpus selection now walks the live registry instead of a hand-listed few (PR #1126), and the domain-selection table covers every live hub corpus (PRs #1143, #1149, #1150).
Fixed
xerj codetop-k crowding — one large file's adjacent 2 KB chunks no longer crowd every other source out of the top-k: at most 2 records per source file (issue #1137, fixed by PR #1140 — the failure class was measured as G7 retrieval fails before the fix).xerj codetxt-family search — the txt extraction family puts its content in atextfield the BM25 leg never searched, so passages rendered empty; the field is searched now (issue #1139, fixed by PR #1140).- Markdown sniffing — markdown whose body starts with YAML-shaped keys or an HTML prefix classified as Yaml or Xml and indexed as junk; it is Txt now, with guards on the sniffs that mistook it (PRs #1142 and #1144).
.odpadvice — the unsupported-format hint now tells OpenDocument presentation owners to save as.pptx(one record per slide, with slide titles and notes) rather than export to PDF (PR #1131, thomas-villani).
Community
- Field reports: a second reference-coding report from
alessandropcostabr
(PR #1141) and a
doc-sort field report from
Ravandevil25
(PR #1128), whose CLA
signature landed with it (PR
#1129) — the
_docarrival-order fix that report describes is reviewed and waiting on its own CLA signature (PR #1127). - Release-download metrics chore for 2026-10-04 (2,956 assets, 1,651
binaries — direct commit,
[skip ci]).
[1.0.0-rc.81] - 2026-10-03
The agent-intake release — everything between rc.80 and this cut serves the
same turn: an agent (or a first-time user) points XERJ at a folder or a
document and gets a searchable index without being told "unzip it first" or
silently indexing junk. The window's spine is the zero-experience E2E
harness (#1106):
demo/playbooks/AGENT_FTX_HARNESS.md runs a no-prior-knowledge agent
through folder onboarding end to end with a reference run recorded — and its
findings were fixed as defects, not filed and forgotten: the naive path's
four E2E failures (#1107,
issues #1102–#1105), JSONL with a fat first line indexing as one junk
record (#1113), and Office /
OpenDocument / EPUB archives named instead of advised-to-unzip
(#1114). Two new extractors
join the autoindex family on the same one-record-per-unit rule:
PowerPoint decks, one record per slide
(#1117), and Excel workbooks,
one dataset per sheet (#1124,
carrying the community contribution of #1120 rebased). The engine fix the
window is proudest of came from our own corpus-indexing crash: under memory
exhaustion a background std::thread::spawn panics (never returns
Err), and with panic = "abort" one ENOMEM at flush time took the whole
node down (issue #1122) —
#1123 adds a governor-backed
thread_spawn_safe gate so the drain-free path drops inline instead of
aborting the process. The Corpus Hub shipped publicly at
hub.xerj.org — registry-driven, every live corpus
G7-graded against pre-registered suites
(#1119) — and the corpus
A/B benchmark behind its honesty rule landed with the tie published
(#1116, issue #1111: no X
finding traces to corpus content; 33 confirmed defect classes, 2 shared).
Strict unknown-field refusal (#1112)
closed defect #1093: opt-in strict mode 4xx's unknown lookup fields, and
autoindex-created indices are strict at create. CLA signature
#1115 is process, not product.
Added
- Zero-experience E2E harness —
AGENT_FTX_HARNESS.mdplaybook with the rc.80 reference run; re-runnable on shell, MCP-only and HTTP-only personas (PR #1106). - PowerPoint extraction —
.pptxautoindexed as one record per slide (title, notes, shape text), joining DOCX/PDF/EPUB in the document family (PR #1117). - Excel extraction —
.xlsxautoindexed as one dataset per sheet with headered columns typed from the first data row (PR #1124; community contribution #1120 rebased onto main). - Corpus Hub — public registry site at hub.xerj.org, generated purely from the corpus-hub branch registry; live status requires a G7 grade against a suite pre-registered before retrieval (PR #1119).
Fixed
- #1122 — node abort under memory pressure.
std::thread::Builder::spawnpanics (notErr) when the alternate signal stack cannot be mmap'd;panic = "abort"turned that into SIGABRT for the whole process at flush time under RSS exhaustion.ResourceGovernor::thread_spawn_safenow gates the drain-free spawn: breaker engaged or live RSS ≥ watermark → drop inline (PR #1123; root-caused from the persistent server log, repro and three fix directions on the issue). - Zero-experience naive path — the four E2E defects the FTX harness filed: wrong index-name derivation, progress-stream shapes a first-time agent cannot parse, a misleading success on empty folders, and the shell-assumption in the onboarding prose (PR #1107, issues #1102–#1105).
- JSONL fat-first-line — a first line larger than the sniff window sent the whole file to one junk record; JSONL now indexes per line (PR #1113).
- Office/OpenDocument/EPUB zips named — autoindex extracts them itself; the "run unzip" advice is gone (PR #1114).
- #1093 — unknown lookup fields silently 0-hit. Opt-in strict mode
returns a 4xx for unknown fields in
term/match/range/exists; autoindex indices are strict at create (PR #1112).
Benchmarks
- Corpus A/B (#1111) — fair two-arm protocol over 71 corpora; the tie published: no X-arm finding traces to corpus content, P 13 vs X 16 dynamic-exploitable tasks, 33 confirmed defect classes of which 2 shared (PR #1116).
[1.0.0-rc.80] - 2026-09-30
The measured-gates release — the ten-item program to the first official
release (discussion #1054,
issues #1055–#1064) executed the way it was filed: every item landed with its
acceptance criterion measured on the release binary, and the failures ship in
these notes as failures. Three gates passed outright (/_ask 0.9975 macro-F1
at 1.129 ms p50; xerj_map unknown-field 400s 100% reduction; calibration
ECE 0.3109 → 0.0088 held-out), three failed and stay opt-in or behind a flag
with the numbers published (the lexical judge, the flywheel cache share, the
tier-2 decide head's SMS/AG News bars), one split (stemming: +0.0160/+0.0179
BM25 nDCG@10, zero-hits 25 → 15 against ≤ 10). The decide ladder is the
release's spine — tier-2 local head (#1072),
CLI arming + default-feature fix (#1073,
#1097), calibration p_cal beside
every p_raw (#1080,
#1087), the answer flywheel
(#1075) and _watcher evaluating
one real watch shape (#1082) — and
the agent surface grew the twelfth and thirteenth MCP tools (xerj_map,
xerj_plan behind the new POST /_ask) plus token-budgeted MCP responses.
The console closes the user gate the UX reviews kept failing: a knowledge
surface that shows what is indexed using only the engine's own numbers
(PR #1095, carrying the
#1098/#1099 honest-claims fixes from its own screenshot review). rc.79 was folded into
this cut by direction — no rc.79 tag exists.
Added
Console: one knowledge surface — size, structure, relations and capabilities after indexing (PR #1095). The moment
xerj brainfinishes, the Console's landing view (CORPUS) now answers the three questions a person has, in one session-authorized read —GET /_xerj-console/api/v1/knowledge(xerj-console-api/src/knowledge.rs): whole-corpus totals (datasets/records/files/source bytes plus the engine's live doc count), every catalog dataset with its FULL field list (type, semantic flag, cardinality, null%, coverage, example values — the sameFieldSpecdataxerj autoindex mapserves agents) joined with live doc counts and on-disk store bytes, the cross-dataset relations autoindex actually inferred (key overlaps, time alignments — none inferred is shown as none, never guessed), other user indices, and a capability strip grounded in real surfaces (console routes, CLI commands,POST /_ask,POST /_decide;graphonly when a readable brain exists,ask/maponly when a catalog exists). Simplifications in the same PR: the fabricated clusters/index/field mocks behind the DATA section are deleted (real-only, failures yield empty lists), its field table drops type-guessed encoding/ratio columns, Second Brain is reworded into the graph layer of the corpus rather than a notes-flavored mode, andxerj mcphelp now says 13 tools (it said 11). Pinned on payloads byxerj-console-api/tests/knowledge_surface.rs(5) andxerj-ux/test/knowledge-surface.test.mjs(7). Two honest-claims fixes from that PR's screenshot review, riding the same branch (issues #1098 and #1099): the CORPUS fields table now leads with the corpus's DOMINANT fields, ranked by the catalog's own coverage × avg_len with autoindex plumbing last (a type-first rank made a 96%-code corpus open with 0.6%-coverage email fields and pushed itscodefield out of the table), unmeasured coverage renders as "—" on both card paths (never a defaulted 100%), and the whole table is one ranked list through the ALL-FIELDS disclosure; and the capability strip gained a server-computedsemanticentry carrying the node's own facts — field, companion vector field, dims, similarity and the embedder's own label ("lexical feature-hash (built-in, 384-dim, non-neural)" on a default node, with the--embed-mode neuralpointer) — with NO vector count, because the engine exposes none on any stats surface; an index whose elected semantic field carries no embedding says so in as many words.rc.80 decide-gate measurements: tier-2 quality, serving latency, and the flywheel replay, all through live nodes (PR #1088). The rc.79/rc.80 gates on the System One decide ladder, measured on commit
a179e1ef3(xerj v1.0.0-rc.78) inbenchmarks/decisions-as-retrieval/results/2026-09-30-rc80gates/. #1057 quality FAIL (bars: SMS acc ≥0.95 / ECE ≤0.05, AG News ≥0.85): SMS 0.9193 / ECE 0.287 on the 1,574-row seed-7 held-out split; AG News 0.2599 on the full 7,600-row test set (chance 0.25 — the artifact was never trained on it); Banking77 reported, not gated: 0.1185 / ECE 0.0717, which reproduces the eval card exactly through the node's choice path. #1057 latency PASS: p50 8.4 ms for 30/_decidenoul questions with the node pinned to 8 cores (5-way choice 20.0 ms; the full 77-way vocabulary 282.6 ms); the tier downloads 0 MB — there is no download path — and the artifact is 28.5 MiB of F32 (not quantized; candle 0.9 has no quantized safetensors loader). #1061 flywheel replay FAIL: as shipped, the write-back caches only 12 answers in 2,000 requests (once any cached doc gives BM25 support the history vote answers everything, and history answers are never cached), then answers 99.7% of later traffic from that frozen cache at 0.033 accuracy — 4.91% of later traffic at ≥0.8 confidence (bar 80%) at 0.0377 accuracy (bar 0.97); with the gate's premise instantiated (2,000 gold answers cached), 19.72% of later traffic clears 0.8 confidence at 0.9953 accuracy — the accuracy bar holds, the 80% share bar fails 4× under at k=10. Release-cut fact recorded beside the numbers: the stock release binary is built without thedecide-localcargo feature —--decide-mode localon it logs a warning and tier 2 stays off, so the tier ships absent, not merely flagged off, until release.yml enables the feature.xerj-decide-v1: the open tier-2 decide model — training/export harness, measured eval card, upload-ready bundle (#1064, PR #1084). The decide ladder's tier 2 (--decide-mode local --decide-model-dir <dir>) until now had a loader and no artifact to load.benchmarks/decide-model/is the deterministic harness that trains one: stdlib-plus-candle (CPU only), the samecandle_transformersModernBERTloadcall the server loads with — trainer and server run one forward, not two that might drift — splitmix64 for every random choice including weight init (candle's CPU RNG cannot be seeded), per-epochstream_seed, no clock or environment read in the artifact path, and the one process-unstable step (the tokenizers crate's wordpiece fit; measured 8,740/8,741/8,742 across runs) shipped as a frozen committed input,pinned/tokenizer.json. The artifact — 7,419,907 params (256×6×4, FFN 768, vocab 8,742), 29,683,844 bytes of F32 safetensors — is verified by loading it back through--decide-model-diron a booted node; the weights exceed the 5 MB blob ceiling so the repo carries the export script, manifest, config, tokenizer and VERSION, andscripts/publish_bundle.shassembles the complete upload bundle (Apache-2.0 licence, digest-checked manifest, the card as README) with the exacthuggingface-cli/curlcommands — publication is PENDING operator credentials, and no published URL is claimed. Every number is measured and indocs/DECIDE_MODEL.md: full-test-split Banking77 77-way choice 0.1185 acc / 0.072 ECE; SMS held-out noul 0.8850 acc / 0.310 ECE / F1 0.686; leave-one-out zero-shot ≈ chance in both directions (Banking77 0.0094 vs 1/77 ≈ 0.013 chance; SMS 0.1982) — the honest boundary of the "no history" claim; serving latency on a booted private-port node (release build, CPU, sequential client):/_decidenoul p50 13.0 ms / p99 13.9,/v1/systemonenoul 13.2/14.3, 5-way choice 21.6/23.5, full 77-way 228.1/237.1; against the history vote's published 0.819 BM25 / 0.937 hybrid (Banking77) and 0.983 BM25 (SMS), tier 1 stays ahead on its home ground, as it should. Wide-choice ranking and calibration at this size are named as unsolved in the card, not buried. Reference-coding corpora are absent in this sandbox and were not consulted.benchmarks:
xerj_mapunknown-field-400 gate measured — 20/30 → 0/30 (PASS) (#1055, PR #1086). The rc.79 gate forxerj_map("unknown-field 400s from MCP-issued DSL down ≥ 80 % on the reference-coding harness") is now a measured number with raw results (benchmarks/map-gate/), per the plan's packaging rule. On the release binary against a reference-coding corpus (the three small pinnedxerj-vectorrepos, 9 datasets, 3472 records), 30 MCP-shaped query intents issued twice — once with field names guessed the way an ES-fluent agent writes them first, once with field choices derived mechanically from the realxerj mcp→xerj_mapstdio response: 20 of 30 guessed-DSL requests failed with an unknown-field HTTP 400 (9 sort, 5 semantic, 6 knn/hybrid-leg), 0 of 30 map-informed requests did — a 100 % reduction against the ≥ 80 % threshold. Discovery recorded beside it: on this engine a wrong field interm/match/range/existsis a silent 0-hit 200, not a 400; the 400 class is exactlysortwithoutunmapped_type,knnon a non-vector field, andsemanticon a non-semantic_textfield. The silent-miss class is reported separately (5 vs 1), not folded into the gate number. The live model-driven agent loop (whether an agent consults the map unprompted) is designed but not run — live-model cost not approved for this wave; design and ~$120 estimate in the benchmark README.systemone: calibration layer —
p_calbeside everyp_raw, published reliability curve (#1063, PR #1080). A ladder probability ranks before it odds — on the FiQA rerank baseline a hosted noul of 0.93 meant relevance 34% of the time (ECE 0.3109). The new[decisions] calibration = isotonic | temperaturesetting (defaultnone, the section's 8th setting) fits a correction on a deterministic held-out fifth of the node's recorded outcomes — outcome documents carry the truth label, thepthat was served for that label, and NOsourcefield (flywheel answers are predictions; a fit on predictions would learn the model is right by construction), 10 labelled pairs minimum — and every probability-shipping path on the decide surface carries the calibrated probability BESIDE the raw one:/_decideanswersp_raw(the positive label's share, the quantity a noul answers) plusp_cal;/v1/systemonecarries both indecisions.evidenceper noul question with the strictly-parsed answer objects untouched; choice per-option shares are a different quantity and are not calibrated (the block says"scope": "noul"). No calibration configured → nop_calfield at all; configured but unfitted →p_cal: nullwith a published, actionable reason; fitted → the fit's value, never a copy ofp_rawpretending to be calibrated — and whenever ap_calexists, acalibrationblock rides beside it with the method, the fit's ECE raw and calibrated, and the fitted-on/held-out counts, so no probability ships without an ECE beside it.GET /_decide/_calibrationpublishes the whole reliability curve (binnedp_rawvs empirical frequency with bin counts, raw and calibrated), both ECEs, the fit's parameters (isotonic knots or the temperature), the pair counts, and the fitted documents' date range; it always refits, so it doubles as the operator's refresh (decide requests read a per-index cache refreshed at most once a minute). Isotonic is PAVA with linear interpolation and end-clamping; temperature is one scalar on the log-odds fitted by golden-section NLL search; both are implemented from first principles inxerj_common::calibrationwith method citations (retrieval skipped: the reference corpora are not present in this sandbox). Measured against the FiQA baseline's own retained curve (benchmarks/decisions-calibration, stdlib Python mirroring the Rust exactly — both reproduce the published raw 0.3109 and agree to the fourth decimal): raw held-out ECE 0.3831 → isotonic 0.0330 (gate ≤ 0.10 MET); temperature 0.3903, honestly failing — one scalar cannot map 0.93 → 0.34 while keeping 0.05 → 0.0002. Scope of that measurement: bin-level held-out (the run retained 10 binned aggregates, not the 19,440 raw pairs); the pair-level re-run needs the model's raw scores and is queued as the rc.80 final form.confidencekeeps its raw meaning and the abstain gate keeps applying to it.The #1063 gate in its rc.80 final form: pair-level, measured, PASS (#1063, PR #1087). FiQA re-scored at pair level — 19,440 raw (query, doc,
p_raw, gold) rows, committed — instead of the binned reconstruction. Method: BM25 top-30 shortlists regenerated on the release binary (xerj v1.0.0-rc.78, lexical embedder, private port, throwaway data dir) with the harness validating nDCG@10 = 0.2382 bit-for-bit against the 2026-09-20 baseline before any paid call; then the baseline's fixed request shape verbatim (jev-1.13.0), 648 paid calls (cap 50,000 enforced), 5.06 M input tokens, $0.2125, zero fails. Deterministic query-level 80/20 split (sha256(qid) % 5, no RNG, no clock — #940), deployment-faithful (fit on past queries, apply to new ones); pair-level hash split reported as the secondary row. Measured (benchmarks/decisions-calibration/results/2026-09-30-pairlevel-fiqa/): raw all-pairs ECE 0.3109 — the published baseline number reproduced exactly on a fresh paid run; held-out raw 0.2872 (Brier 0.1625); isotonic held-out 0.0088 (Brier 0.0256) — gate ≤ 0.10 MET; temperature 0.3163 (T = 1.4589), failing as before. Honest scope, stated in the README: ECE flatters on a 96.9 %-negative dataset (92 % of held-out pairs calibrate below 0.1); the substance is the Brier halving and the top-bin remap — raw 0.90–0.98 becomes 0.24–0.53 against an actual held-out relevance rate of 0.4208. The binned-smoothing caveat closes in the gate's favour: pair-level (0.0088) beats binned (0.0330), because 15,840 real pairs give the fit ~98 knots where the binned form gave it 5.POST /_ask+xerj_planMCP tool: prompt in, validated query DSL out (#1056, PR #1076).POST /_ask(ES-compat) andPOST /v1/ask(native) take{index or pattern, prompt}and return{query, plan, confidence, indices}wherequeryis an ES query clause that has ALREADY passed this node's ownxerj_query::parse_request— zero invalid DSL can leave the node, and a phrase the index cannot ground is a 422 naming it, never a silent guess. The planner is deterministic and rule-based (not an LLM): date literals and year phrases, comparator vocabulary, units, scale words and a concept lexicon resolve field phrases against the index's real schema; equality values resolve against ONE batchedtermsagg (≤1,000 buckets/field, ≤24 fields), with a BM25matchprobe for fields whose values overflow the agg. Numeric and date bounds are only ever the prompt's own literals. Responses are byte-identical across repeats;ax-*patterns route by theautoindex-catalogdataset description read through the engine's own index APIs (no xerj-autoindex dependency in xerj-api).xerj_plan, the 12th MCP tool, proxies it (indexdefaults toax-*likexerj_search); the publishedmcp-tools.jsonschema is regenerated. Measured by thebenchmarks/ask-planharness (PR #1071, unmodified): 230 pairs × 3 runs, macro result-set F1 0.9975 (round 1: 0.9309),invalid_dsl_out0,non_deterministic0 — gate ≥ 0.9 over ≥ 200 pairs passed; the three sub-1.0 pairs are the "from A through B" date pairs, where this PR reads "through" as inclusive by design. ES-YAML on this tree: 1380/0/3. The p50 ≤ 300 ms CPU budget is a release-time gate and is not claimed here.The #1056 latency gate measured:
POST /_askwall-clock p50 1.129 ms against the 300 ms bar (PR #1079). The ask-plan harness scored F1 but never recorded milliseconds;scripts/latency_arm.pyis the missing arm (one warmup pass + N measured passes over the same 230pairs.jsonlprompts on one booted node, per-request wall-clock raw, nearest-rank quantiles, stdlib-only, honest stop on endpoint-absence) andrun.shstep 5 now captures it. Measured on the release build off0b96d634(private port, throwaway data dir): 690 requests, all HTTP 200, p50 1.129 ms, p90 1.883, p99 2.852, max 155.7 ms — the deterministic rule-based planner costs microseconds of CPU, as designed, and the 300 ms release bar is met at 266× margin. Results committed verbatim inbenchmarks/ask-plan/results/2026-09-30-latency-gate/; the token arm of the same gate remains designed-not-run (live-model cost not approved).Decision cache flywheel: every tier-2+ answer written back,
sourceper answer, human corrections weighted ≥ 2x (#1061, PR #1075)./v1/systemoneand/_decidenow close the loop PR #1072 opened: every tier-2 (local head) answer — and every tier-3 answer when that tier exists — is cached into the configured[decisions]index as an ordinary document carrying the configured text/label fields plusp,sourceandts, so the answers a node computes become the labelled history that answers the next request (pinned by test: the same request twice is answered by history from its own cached answer). The write-back is spawned, never awaited by the request, and cannot fail it — an unreachable index is a log line; a configured-but-missing index is created by the write-back itself, which is how a local-tier node bootstraps its history. History-tier answers are never re-written (they are the index already), and/_decideabstains are never cached. Every answer now names its tier in asourcefield (/_decidetop level,/v1/systemoneper-question indecisions.evidence.*.source) with the strictly-parsed answer objects untouched. Human corrections are ordinary history documents carryinghuman: true(the existing write path, no new endpoint) and weigh the new[decisions] human_weight× (default 2.0, the issue's ≥ 2x floor; validated positive at config load) their reciprocal rank in the vote — everything else keeps the 1/rank arithmetic the published measurements used. The issue's Banking77 replay gate and the TypeSafe-terms question on retaining provider outputs are release-time gates, deliberately not claimed here.Local
judgestage on_search:_p_relevantper hit,min_ppruning, zero tokens, zero egress (#1060, PR #1077). A_searchbody may now carry"judge": {"local": true, "min_p": 0.5}: the node re-scores the hits it was about to return against the question in process — no provider, no key, no network call — and each judged hit carries_p_relevant, a 0..1 relevance probability on an absolute scale. The engine's_scoreis NOT replaced (the one deliberate difference fromrerank): a client sorting by_scorekeeps the engine's number while a client thresholding_p_relevantgets a cut a BM25 score can never give.min_pdrops hits below the threshold and counts them in the response'sjudgedblock ({applied, scorer, query, judged, kept, dropped, took_ms}). Which judge judged is on the wire:judged.scorerislexicalormodel. The always-compiled arm is a deterministic lexical scorer (window-local BM25 saturation, page-local IDF, K1/B from xerj-fts) with NO semantic signal — named as such, never passed off as a model; the model arm (cargo featuredecide-local) reuses the #1072 decide loader and is unmeasured in this sandbox — the issue's BEIR bar (0.699 SciFact / 0.345 NFCorpus, FiQA ≥ 0.30) and ≤ 40 ms p50 top-30 gates bind at release. Strictness mirrors the rerank stage:sort/search_after/collapse/scroll/size:0/_source:false-without-fields refused by name before the search runs, and every surface that does not run the stage (scroll continuation,_msearch, the search templates,_async_search,_rank_eval, the native/v1search API, the gRPC Search RPC) refuses the block by name instead of silently returning unjudged hits under a 200. The MCP toolsxerj_searchandxerj_hybrid_searchtake the samejudgeargument (query required on hybrid, carried from the plain string otherwise), their descriptions state the honest ceiling and the zero-egress difference fromrerank, and the published schema is regenerated.--decide-mode/--decide-model-dir: the tier-2 decide head is armable from the command line (PR #1073). The local decide head landed with PR #1072 env-only (XERJ_DECIDE_MODE/XERJ_DECIDE_MODEL_DIR); the server CLI now parses--decide-mode history|local(defaulthistory) and--decide-model-dir <path>with the same flag-wins-over-env shape as--embed-mode, resolving through the sameDecideSettings::resolveinto thestate.decideseam (xerj-api unchanged). Also adds thedecide-localcargo feature to xerj-server (off by default, matching xerj-api's gating) — the rebuildresolve's own warning text had been naming all along. A default build asked for--decide-mode localboots on the history vote and says exactly which rebuild arms the tier; verified by booting both builds (armed/off/missing-dir/mixed flag+env).Local zero-shot decision head — tier 2 of the System One decide ladder (#1057, PR #1072).
/v1/systemoneand/_decidenow answer through an ordered ladder per question: the history vote wins wherever it has labelled support (unchanged); a local ModernBERT-class candle classifier — cargo featuredecide-local, armed byXERJ_DECIDE_MODE=local+XERJ_DECIDE_MODEL_DIR— answers every no-support class (no[decisions]index, a missing/unsearchable one, no labelled neighbour), so the same request that 503'd/422'd answers 200 and tiers mix per-question in one response; a hosted key stays reserved. The head scores NLI (premise, hypothesis) pairs per candidate label and renormalises, loads from a local model directory only (no hf-hub, no new outbound client — the egress inventory is unchanged), echoes its own idxerj-decide-local-1(never a Jev name), and fails loudly (503 naming the directory) rather than fabricating a probability. Default builds keep the exact previous 503/422 contract. Loader validated end to end against a deterministic test fixture; no accuracy/ECE/latency number is claimed until the trainedxerj-decideartifact (#1064) is measured at release.Token-budgeted MCP responses:
max_tokenson every search tool (#1058). The five MCP search tools (xerj_search,xerj_hybrid_search,xerj_semantic_search,xerj_vector_search,xerj_code_search) returned whatever the engine sent, so an agent that asked for 50 hits got all 50 verbatim — context cost unknown and unbounded. All five now takemax_tokensand are held to it: the response text NEVER exceeds the budget, enforced by construction (a note reserve is subtracted before any content is admitted) and pinned by tests across a range of budgets on both response surfaces. Tokens are counted with one named tokenizer — XERJ'sStandardTokenizer(UAX #29 word boundaries, the same word split the engine's analyzer uses for BM25;unicode-segmentationdepended on directly so the count is identical by construction without pulling the FTS stack into the thin proxy) — an honest proxy for model-context tokens that under-counts subword splits, stated in the argument's description. Overlapping file:line passages are deduped before budgeting (higher-ranked shown passage wins); hits that no longer fit are returned as locators only (file:line, index/id — the citation, never the passage) while they fit, and a_token_budgetaccounting block reports kept, locator-only, deduped, dropped, andtokens_used. The engine never sees the argument (request bodies are bit-identical), errors are never truncated, nothing binding returns the engine text byte-identical, and a budget below 32 (not even the envelope fits) is a builder error, never a silent drop. Generalisesxerj_code_search's char-levelfullcap to the whole response. The published schemalanding/docs/agents/schemas/mcp-tools.jsonwas regenerated from the binary;published_schema_driftstays green. (PR #1067.)xerj_mapMCP tool: per-index field map from the autoindex catalog (#1055). The catalog index (autoindex-catalog) holds, per dataset, the fullFieldSpeclist the profiler inferred, but only the CLI could read it — an MCP agent writing query DSL had to guess field names and ate unknown-field 400s on the searches it then issued. The twelfth MCP tool reads the catalog (the samedoc_kind:datasetqueryxerj autoindex mapruns, withsample_queries_jsonprojected out of_source) and renders, per index: record count, time field and range, semantic body field, and every field — name,es_type, date encoding, semantic tag, cardinality estimate, coverage, up to 5 examples, and the sampled date AND numeric min-max. Default response ≤ 4 KB per index (max_bytes, floor 1024); the documented trim ladder cuts examples to one, then examples, then per-field notes, then lowest-coverage fields, withtrimmed/fields_omittedaccounting for every cut. An exactindexname filters engine-side, a*-glob client-side; a missing catalog is an error naming the fix, an empty one a normal empty result.FieldSpecitself gained the numeric range (num_min/num_maxfrom theint_min/int_maxFieldAcchas tracked all along) — populated exactly onlongfields, deliberately absent ondouble(the accumulator tracks only the i64 subset, so a fractional column would carry its integers' range stated as fact), with#[serde(default)]so every pre-existing catalogfields_jsonand state plan still deserialises. Published schema regenerated (12 tools);published_schema_driftstays green. (PR #1074.)_watcheris real:PUT /_watcher/watch/<id>now either evaluates a watch on schedule or refuses it with 501 (#1062). The route used to answer 201"active": trueto every body with no machinery behind it — accepted-and-ignored, the #204 class at route level — and the alert indices.xerj_alert_rules/.xerj_alert_fireswere written by nothing. One watch type is implemented:trigger.schedule.interval(>= 1s),input.searchover exactly one index,condition.xerj_decide(question, positive_label, p_min, k, optional decide index) and anindex_alertaction. A per-watch background task prefilters candidates with the watch's own query (ascending_seq_nokeyset paging, 100 per page — the #1022 shape), runs each rendered question through the real/_decide, and writes a fire record into.xerj_alert_fireswhen the positive label wins non-abstaining at or abovep_min. Anything else (cron schedules, transforms, other action types, multi-index input) is refused with a 501 that names what is not evaluated; malformed supported shapes are 400. Two cursor subtleties are pinned by tests: the empty-index baseline is-1because_seq_nois 0-based on the wire (a 0 baseline strands the first document ever indexed), and the baseline is captured input_watchbefore the evaluator spawns so documents indexed between the 201 and the first poll cannot be absorbed unjudged. Fire records carry the RAW decide confidence, plusp_calwithcalibrated: truewhen the node has a fitted calibration (#1063, PR #1080) — the threshold stays on the raw value. Honest limits: a restart does not resume watches (evaluators are spawned byput_watchonly), and the per-pass decide budget is 1000 with the cursor held below a failure so the backlog is retried, never dropped. Zero watches cost zero tasks; an idle watch's tick measured sub-second (sub-millisecond, pinned by test). (PR #1082.)xerj autoindex <folder> --label <question-set.json>labels each document at ingest throughPOST /_decide(#1062). The question-set file (thenoul/choicevocabulary/v1/systemonealready uses) is validated whole before the folder is walked;{{field}}in a question renders against each record's own fields. Anoulis one binary vote; achoiceis one vote per option with argmax over independent binary votes (an option that lost its vote contributes1 - confidence). Fields written:label/label_pfor a single-question set, pluslabel_<id>/label_<id>_pper question. Abstention is an answer —label: nullwith the raw p carried for later calibration;pis the RAW decide confidence everywhere. A FAILED decide fails the run before the document is bulk-indexed: an index the operator believes is labelled is the #204 defect class. The vote runs at both record sinks — the generated/durable route decides insideprepare_artifactbefore sealing (labels are part of the durable artifact; replays and #971 hardlink reuse carry them without a second vote, and the question-set bytes are part ofprepared_records_identity, so a changed set re-prepares instead of hardlinking differently-labelled NDJSON) and the legacy phase-B sink at the same point.--labelis refused by name on every route that cannot apply it (folder--watch, object-storage watch,map,status,--dry-run). Cost is ingest-time only — one round trip per noul (per option, for a choice) per record; with the flag absent noLabeleris constructed, so idle cost is zero by construction and the measured cost (ms/doc, printed by the end-to-end test against a loopback node) is the floor, not the ceiling. (PR #1082.)
Fixed
- Main's
Build + Testun-reds: #1062's label test stubs are classified in the egress inventory (PR #1083). The merge of #1082 added two unclassified outbound-client files (xerj-autoindex/src/label.rs,label_http_tests.rs) that the author's test scope never exercised — the xerj-rerank guardevery_outbound_client_in_the_engine_is_on_the_published_listruns in a different crate's suite and only CI saw it. Both areRole::TestOnly, verified rather than asserted:label.rs's only marker (TcpStream::connect, the DecideStub drop-wakeup) sits after its#[cfg(test)], and the production Labeler decides through the sharedEsclient already on the published list — no new egress path exists, sodocs/RERANK.md,xerj_rerank::DATA_EGRESSand the air-gapped recipe are unchanged. - PAVA knot-index bug in the calibration benchmark's Python mirror
(PR #1087).
benchmarks/decisions-calibration/fiqa_gate.pyrecorded each pooled block's first-x aslen(blocks)at append time, which stops equaling the pooled-point index after the first backward pool — every knot created after a violation got its x shifted down andapply()mapped probabilities through a compressed curve. Invisible on the bin-level gate data (its even-bin fit points are strictly increasing, so no pooling occurs and buggy and correct code coincide — both reproduce 0.0330, which is why the Rust pinning test stayed green); it corrupted any fit on data with local violations, i.e. the pair-level rerun, where the first pass produced a nonsense isotonic 0.2372. The shipped Rust module (xerj_common::calibration) was audited and is correct — blocks carry theenumerate()pooled index and expand means to every pooled point — so only the mirror diverged and no engine change was needed. The fixed mirror matches the Rust representation exactly, agrees with an independent naive run-pooling PAVA to 8.7e-19 at all 98 pooled points, and still reproduces the bin-level 0.0330 to the digit. Lesson recorded: the pinning test asserted ≤ 0.10, which the wrong number also satisfied — an exact-value pin on a pooling input would have caught the divergence. - Release notes now compare against the previous engine release instead of
the latest corpus pack. GitHub anchors auto-generated notes ("Full
Changelog") on the most recently published release, so since the daily
pack-rust-vulns-*scheduled releases started, every cut's compare base drifted:v1.0.0-rc.78shipped comparingpack-rust-vulns-2026-09-29...v1.0.0-rc.78and had to be hand-patched via REST. The release job now computes the base itself (highestv[0-9]*tag by version sort that is not the tag being cut) and pre-renders the notes with an explicitprevious_tag_name, falling back to auto-generation only when no previous v-tag exists (PR #1069). - Release notes: the pinned compare base now orders a final above its own
rcs, and the derivation is fixture-tested. The PR #1069 pin derived the
base with plain
--sort=-v:refname, which ordersv1.0.0-rc.79abovev1.0.0— the opposite of what that PR's message claims — so the firstv1.0.1cut would have compared againstv1.0.0-rc.79rather thanv1.0.0, and re-dispatching an old tag would have compared against the newest tag instead of that release's own predecessor. The derivation now lives in.github/scripts/prev_tag.sh:versionsort.suffix=-rcputs a prerelease below its final, the base is the version-successor of the tag being cut, and a five-assertion fixture self-test (pack-tag noise,rc.9created afterrc.78, final among rcs, old-tag rebuild, first-ever cut) runs in the release job before the base is trusted. Measured against the live tag set on 2026-09-30: a hypothetical cut selectsv1.0.0-rc.78, while GitHub's auto-detection would anchor onpack-rust-vulns-2026-09-30, published 10:47Z that morning (PR #1081). benchmarks/ask-plan/— the harness behind issue #1056's gate, shipped before the endpoint so the gate is designed in the open: 230 (prompt, gold result set) pairs over three committed public tabular snapshots (USGS earthquakes — public domain; NASA Exoplanet Archivepscomppars— acknowledgment given, statutory basis stated; Gapminder — CC BY 4.0), with gold derived only by filtering raw rows in Python (never by running an engine), deterministic stdlib-only derivation/verification scripts, a loader + fixture self-check, thePOST /_askscoring arm (macro result-set F1, zero-invalid-DSL, 3-pass determinism), and the guarded two-armclaude -pcomparison the issue specifies. Validated end-to-end against a local rc.78 build: all 230query_equivalents reproduce their gold sets; withPOST /_askabsent the harness stops atmeasured: false. The directory contains zero measured numbers by design — every F1/token cell in its README is a gate definition until a real run lands (PR #1071).
Changed
- The shipped release binary is built WITH the
decide-localcargo feature — tier 2 arms on the stock binary (PR #1097). The rc.80 gate runs caught that--decide-mode localon the rc.78 release binary logged a rebuild warning and stayed off: xerj-api'sdecide-localfeature was never enabled in xerj-server's release build, so the tier shipped absent, not merely flagged off.decide-localis now a DEFAULT feature of xerj-server (default-on, deny-by-default at RUNTIME: the tier activates only on the explicit--decide-mode local/XERJ_DECIDE_MODE=local, so every default boot keeps the exact previous behaviour), release.yml builds it into the published assets, and the artifact loader is exercised by the release smoke path. The catch and its queue-ahead status are stated in the blog post; this is the fix it pointed at. xerj autoindexnow creates prose indexes with a stemming analyzer (#1059). When the profiler marks a dataset's prose semantic — the same measured predicate that gates the semantic-body election (word_ratio >= 0.55 && mean_tokens >= 3.0; measuredword_ratio0.00 fortrace_id-shaped identifiers, 0.78–1.00 for prose) — the index CREATE body declares the built-in Snowball Englishstemmeranalyzer assettings.analysis.analyzer.default, the surface #937/#991 made honoured at flush, segment query and merge. Plurals and inflections now match on the lexical arm (databases↔database);stemmerwas chosen overenglishbecause it adds no stop-word removal and so changes no BM25 length norm. Identifier/enum/numeric datasets keep the exactstandardterm space, keyword fields are never stemmed, and existing indexes are untouched by construction (analysis settings are immutable after creation and a CREATE that finds the index already there does not re-send). The declaration rides the plan outside the frozen identity hashes, so no existing state dir aborts on upgrade. The BEIR/NFCorpus zero-hit measurement (7 of 323 NFCorpus queries return no hits on plural-only mismatches) remains the release-time gate. (PR #1070.)
Documentation
- The rc.80 gates post on the blog: "The rc.80 gates: three passed
outright, three failed, one split" (PR
#1096). The blog half of
the ten-item program's packaging rule (#1055–#1064, discussion #1054):
every gate verdict, wins and losses, each figure traced to the gate run
that produced it. PASS:
/_askmacro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms against the 300 ms bar (#1076, #1079);xerj_mapunknown-field 400s 20/30 guessed → 0/30 map-informed, 100% against the ≥80% bar (#1074, #1086); calibration FiQA pair-level ECE 0.3109 → 0.0088 held-out against the ≤0.10 bar with the temperature arm's 0.3163 published as the failing comparison (#1080, #1087). FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045→0.5967, 0.3419→0.2924, 0.2382→0.1650 against the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (#1077, #1090); the flywheel freezes as shipped (12 of 2,000 cached, #1094) and forces 19.72% coverage at 0.9953 accuracy against the 80%/0.97 bars (#1075, #1088); the tier-2 decide head misses SMS 0.9193/0.287 against 0.95/0.05 and AG News 0.2599 against 0.85, with latency 8.4 ms passing, and ships behind the flag with the numbers published (#1072/#1073, #1088). SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 passes while zero-hit 25→15 fails the ≤10 bar — the 15 have no stem anywhere in the corpus (#1070, #1085). The post also tells the gate catch that the stock release binary was built without thedecide-localcargo feature (the flag a no-op,stock-binary-noop.txt) with the fix queued ahead of the cut rather than claimed done, and states the honest gaps: the/_asktoken arm designed not run, the_watcherdetection gate not yet measured, no Hugging Face URL claimed forxerj-decide-v1. Mechanics: hand-writtenlanding/blog/the-rc80-gates.html, registered inscripts/seo/pagedata.py(published/updated 2026-09-30), card prepended to the blog hub with itsdateModifiedbumped, head/footer owned by the seo generators, sitemap regenerated post-commit; all local gates green (build_articles/gen_sitemap/fix_heads/fix_links/mk_og_card--check,seo_lint0 violations,factcheck0 errors, constants guard). - Measured the #1059 stemming gate on the release binary — the BM25 half
passes, the zero-hit half fails at 15 (PR
#1085). Paired arms on one
node (
benchmarks/beir-hybrid, release binary @a179e1ef3): the same create-timesettings.analysis.analyzer.default=stemmersurface #1070 ships, vs the standard-analyzer baseline, which reproduced the rc.74 row exactly (0.6572 SciFact / 0.3016 NFCorpus / the same 25 zero-hit queries). BM25 nDCG@10: SciFact 0.6572 → 0.6732 (+0.0160) and NFCorpus 0.3016 → 0.3195 (+0.0179) — both clear the gate's +0.01, so stemming is a measured BM25 win, not a regression. NFCorpus zero-hit queries went 25 → 15, not the required ≤10: stemming repairs exactly the 10 plural/inflection queries; the remaining 15 are single words (Fosamax, Zoloft, eggnog, Yale…) whose stem also occurs nowhere in the corpus — not bridgeable by any stemmer, the vector arm remains their only repair. Raw per-query outputs inbenchmarks/beir-hybrid/results/2026-09-30-stemming-1059/. Also recorded:GET _settingsdoes not echo the analysis block and_analyzeshows the standard path even when the stemmer is declared and provably applied by search. - Measured the #1060 local-judge gate on the rc.80 release binary: the
lexical judge LOSES, and the number ships (PR
#1090). New BEIR harness
benchmarks/beir-hybrid/judge_gate.py(same qrels/nDCG@10 scoring aseval.py, 3 runs per arm, shuffled order, size-30 pages, judge carrying nomin_p— reorder-only) onxerj v1.0.0-rc.78with the neural embedder, node pinned to 8 cores: hybrid+judge 0.5967 SciFact vs hybrid 0.7045 (−0.1077), 0.2924 NFCorpus vs 0.3419 (−0.0495), bm25+judge 0.1650 FiQA against the 0.30 bar (hosted Jev rerank: 0.3638) — the quality gate FAILS on all three rows and the stage stays opt-in with no quality claim, per its own "loses, does not ship" rule; nothing was tuned or re-rolled. The cost AC passes with room: +0.7 / +0.5 / +0.6 ms added p50 for a judged top-30 page (server-sidejudged.took_msp50 0 ms). Every arm reproduced to the fourth decimal across runs AND across a node restart (post-#940 determinism), and the BM25 arms reproduce the 2026-09-18 baselines bit-for-bit (0.6572 / 0.3016 / 0.2382), validating the harness. Also recorded: hybrid on the 57,638-doc FiQA index costs 13-26 s/query (14.7-15.1 s server-side), and an informationalmin_p: 0.5arm empties 274 of 300 SciFact pages — the lexical scale is not thresholdable at round numbers. Raw logs and sha256s:benchmarks/beir-hybrid/results/2026-09-30-judge-gate/. - Rolled
ROADMAP.md's tracker state after the issue tracker was emptied on 2026-09-29, the day of the rc.78 cut: the four issues still open after the cut — #1038 and #1030 (their shipped halves are the rc.78 record below) and the defects #1031/#1032, closed as deferred, not fixed — were closed, the abandoned pointer #298 was unpinned, "In flight" emptied, the open-defects shortlist became a plain-text deferred record (the release-notes gate checks that section's issue links against the live tracker at every cut), and the corpus-hub and mail-ingest lines state the closed statuses (PR #1053). - Filed the ten-item program to the first official release (discussion #1054) as issues #1055–#1064 with acceptance criteria, measured gates, and code refs verified against the tree (two ref paths corrected; the plan's rc.78 targets slid to rc.79 — rc.78 shipped before the discussion posted); created milestones rc.79/rc.80/v1.1.0 and labels (mcp, agents, autoindex, search, systemone, zero-token, roadmap); stood up the pinned plan tracker #1065; rolled ROADMAP's In flight and GA-program sections to match and pointed the two zero-token items it subsumes at their new trackers (PR #1066).
- llms.txt / llms-full.txt tool inventory now matches the released
thirteen-tool MCP registry (PR
#1078). Both files said
mcp-tools.jsonholds "ten tools" and llms-full's own verify example printedjq '.tools | length'→ 10 while the real registry is thirteen (xerj_mapandxerj_planlanded without the agent-facing docs noticing) — the llms-txt study's §8 lesson repeating verbatim. Hand-written regions only: the mcp line names all seven additions over the HTTP-core six, the anthropic/openai lines state the composition by design,POST /_askenters the endpoint map (it was undocumented in either file), and llms-full §3 is retitled THE CORE AGENT OPERATIONS with the jq example pinned to 13. Checked, not asserted: all 13 names from the publishedmcp-tools.json(pinned totool_specs()bypublished_schema_drift) appear by name across the two files. - Console UX review: the post-indexing surface against the user gate, in the open (PR #1089). Method stated first because the sandbox has no browser: the served SPA fetched from a running node, its API calls curled verbatim under a REAL operator session (first-launch bootstrap completed — magic link from stderr, passkey enrolment with a software P-256 authenticator), and the repo's own pure renderers executed under node against the live payloads, so quoted screens are the SPA's output, not a paraphrase. Corpus: 72 mixed files (mbox, .eml, CSV, NDJSON, wikilinked notes, source) → 8 datasets, 512 records, 290 edges in 2.2 s on rc.78. The three gate questions — does the console show how large the corpus is, what data is there, what XERJ can do — answered per screen with the defects named, and its verdict (the console answered volume but guessed structure) is what the knowledge-surface change in this release (#1095) was cut against. No engine code changed.
- Rolled the corpus-pack status in the agent-facing index files
landing/llms.txtandlanding/llms-full.txt, which still said "planned, no code": the records half shipped in rc.78 —xerj corpus build, per-file checksums, the detached ed25519 signature over SHA256SUMS, the daily datedrust-vulnsrelease — while stating plainly that the pre-indexed half and the hub directory remain unbuilt, and linkingdocs/CORPUS_PACKS.mdas the proving file (PR #1068).
[1.0.0-rc.78] - 2026-09-29
The index-size and corpus-packs release. Two headline changes: stage 1 of the
index-size epic #1038 lands four
measured format changes (ZNV2, ZPS2, ZBS4, ZFM5 — the committed same-day pair
5,398,478 → 3,989,254 B durable on the 100k harness, −26.1 %), and
xerj corpus build ships portable, signed corpus packs with the first showcase
pack, rust-vulns, published from a scheduled workflow. The same window closes
the idle-cost arc (#874 met at ~3× margin, idle 206 → 64 kB per index) and the
by-query truncation class (#1019, #1022).
Fixed
- An idle node no longer spends CPU per index on Prometheus gauges (#874).
The 10 s background loop that fed
xerj_doc_count/xerj_segment_count/xerj_wal_size_bytes/xerj_memory_usage_byteswalked every index's WAL subtree on an async runtime worker on every tick — oneread_dirplus onestat()per WAL shard (16 per index at default sharding) whether anyone read the values or not. The new at-rest fixture (63–69 kB, >3× margin). (PR #1020.)benchmarks/idle-budget/) measured that at ~0.7–0.9 % of one core for 450 idle indices (against #874's < 0.5 % budget), entirely in thexerj-rtworkers; deleting the loop and refreshing the gauges at scrape time in the/v1/metricshandler (the same way the query-cache gauges already reconcile) drops idle CPU to 0.15–0.18 % and wakeups from ~46/s to ~26/s on the same corpus, with the WAL walk moved to the blocking pool. A scrape is the only moment these gauges are observable, so nothing is lost between scrapes. The fixture and its coarse CI gate (idle-budgetjob, calibrated to 256 kB/idx on the runner's host shape) keep the budget honest: CPU < 0.5 % of one core, O(1) wakeups/s, boot-to-green with zero WAL replay on a cleanly-flushed corpus. The 0.2 MB/index product line itself was still at or above the line on 4-vCPU hosts at this point (204.9–209.9 kB measured, N-slope 206.5 kB/idx) — that is #1024, met only after the lazy seen-set work below ( - The flush publication bracket no longer spans the drain (#1015): flush drains now freeze and the bracket wraps only the publish, so the ms-scale bracket constraint the #1014 evenness check enforces finally holds on the drain side too. Reader-fairness part two. (PR #1018.)
- Large-ingest id-position maps are built from the
__idprojection, and the stored reassembly is streamed (#950) — the attribution work closed the issue rather than filing a follow-up: the per-index structures that grew with corpus size during ingest were rebuilt around projections and streaming, withPOST /{index}/_cache/clearadded to release the rebuildable caches on demand (PR #1017; the cache endpoint is PR #1009). _delete_by_querypages the whole match set, ids-only (#1019) — the one-shotsize: 10_000truncation class found there is gone from the delete path. (PR #1021.)_update_by_querypages the match set, with exact sig-text/enrich counts (#1022) — closing the remaining single-shot by-query sites the issue named in the #1019 truncation class; a single request is now a full-corpus traversal with a 10,000,000max_totalbackstop rather than a silent 10,000-hit cap. (PR #1023.)
Added
xerj corpus build— portable, signed corpus packs, and the first one:rust-vulns. The records half of the corpus-hub roadmap item (#1030): a declarative TOML recipe (sources with licences, identity edges, merge precedence, derived fields — the tool is domain-agnostic; what a record is lives in the recipe, never the code) drives harvest → normalize → content-addressed store (file presence is the skip-already-stored dedup — incremental by construction, no sync mode to get wrong) → union-find identity resolution (alias chains close for free) → merge (declared scalar precedence, array union, most-restrictive licence, auditablesources[]) → a deterministic sharded pack whose manifest carries aformat_versionreaders refuse when unknown, per-file SHA-256 verified onxerj corpus add --from <dir|zip>, the recipe verbatim as provenance, and mapping/relation suggestions that are never applied. The showcase packtools/packs/rust-vulns/collapses 4,107 osv.dev + RustSec filings to 1,950 identity-resolved records (1,221 merged from both sources) carrying RustSec's curated vulnerable-function paths, which no export ships; every README number is measured (pack-stats.json,provenance: measured) and pinned by a test. Publishing is a separate step, never part of a build:xerj corpus keygen/signwrite a raw 64-byte ed25519 detached signature over SHA256SUMS (the public key travels beside the recipe, never inside the pack),corpus add --verify-sig <pubkey>checks origin before anything is materialized, and a scheduled workflow rebuilds, signs, and publishes the pack to a dated GitHub Release per build day (immutable releases accept asset uploads only before a release is published — so the workflow uploads to a draft and publishes it last — and make release tags single-use, so there is no one rolling URL; the draft-then-publish mechanics are #1049, the dated per-build-day tags #1050) — verifying its own output against the committed.pubfirst, so a half-rotated key fails the build. The attack the signature exists for is pinned by a test: a self-consistent rebuild (tampered records, honestly rewritten checksums) passes every checksum and fails the signature. No vulnerability database in our prior-art survey ships a consumer-verifiable signature. (PRs #1046 and #1047 for the builder and showcase; the signing/publish half is #1048; the loose<pack>-SHA256SUMS.sigrelease asset that--verify-sigrequires beside a pack zip — without which the documented zip consumer flow failed — is #1051.)- An on-disk size harness for force-merged indexes
(
benchmarks/index-size/) — the measurement half of the index-size effort (epic #1038):run.shclones the DISK_SIZE_2026-07-09 protocol (chat-events cycled to 100k docs enriched with a deterministic textbodyand a high-cardinalitydoc_id, explicit mapping, forcemerge 1, du-stable-30s) and reports a per-extension and per-field byte breakdown of everything durable except the WAL;LEVEL=bestre-runs at the merge-path compression knob. CI prints a 20k-doc pass informationally on every PR via the existing api-checks job.DESIGN.mdrecords where the bytes are today (with file:line anchors), the peer designs adapted (tantivy columnar/sstable/postings, quickwit docstore), and the staged plan: header/delta wins inside existing envelopes → FTS-dictionary reuse → zstd trained dictionaries. No engine format changed yet; numbers from this harness gate the stages that do. POST /{index}/_cache/clearreleases the index's rebuildable caches (#950 follow-up work). (PR #1009.)- An email-labelling benchmark for the System One wire
(discussion #1012):
local
POST /_decidemeasured on 260 originally-authored synthetic emails across the discussion's own four labels — 1.000 accuracy on the templated tier, but 0.625 at 0.902 mean confidence on a hand-authored boundary-crossing tier: wrong and confident, the miscalibration shape the discussion itself flags for the hosted path. Nothing is wired intoautoindex; the README lists what that would still need. (PR #1026.)
Performance
.metaterm records go columnar-varint (ZFM5) (#1038): the per-term{df u32, ttf u64, offset u64, len u32}records under the ZFM4 zstd envelope kept a fixed 24-byte stride, and zstd cannot match across the stride's alternating field widths — on the 100 k-doc size harnessdoc_id.meta(100 k unique terms, every onettf == df, every postings gap zero) still cost 138,727 B on disk for ~200 B of information. ZFM5 re-shapes the section before compression into four columnar varint streams —df,ttf − df(elided entirely when every term in the field hasttf == df, the keyword-field shape),offset gap, andpostings_length— so zstd sees long runs of one-symbol lanes. The reader expands the streams once at open time into the same in-memory flat-record array ZFM3/ZFM4 build, leaving the FST-offset lookup hot path untouched. The encoder fails closed on non-monotone postings offsets (gaps reconstruct by accumulation, so a negative gap is unrepresentable) and the decoder rejects truncated streams, trailing stream bytes, and length mismatches. Peer pattern: tantivy'sTermInfoValueWriter::serialize_block(src/termdict/sstable_termdict/ mod.rs:92-105, MIT) writes block-leading absolute range starts once and then per-term length varints only. Measured on the 100 k-doc harness (LEVEL=balanced, forcemerge 1, same-day control): the.metafamily drops 200,826 → 13,774 B (−93.1 %), total durable −3.5 %, with every other extension byte-identical except 3 B of.jsonlstate noise; the win scales with term cardinality, which is exactly the shape of unique-id fields on large deployments. Measured again on a tree carrying ZBS4 as well (results/result-zbs4-zfm5-balanced.jsonvsresult-zbs4-only-control-balanced.json, same day): 4,176,306 → 3,989,254 B — the two stages together take the pre-stage baseline 5,398,478 → 3,989,254 B (−26.1 %), with the four staged deltas (ZNV2, ZPS2, ZBS4, ZFM5) summing to within 12 B of the directly measured endpoint. ZFM4-and-older segments decode unchanged. (PR #1043.)Stored sections get typed-int columns, duplicate-column references and a merge-path zstd effort chooser (
ZBS4) — stage 1 of the index-size epic #1038, redesigned by measurement: dissecting a harness.segshowed the planned dict-stream work targeted ~5 % of the section while thebodyRAW_JSON fallback alone was 84.2 %,__seq_nowas stored as textual consecutive integers, anddoc_idwas a byte-identical duplicate of__id. Three codecs instead (engine/crates/xerj-storage/src/stored_codec.rs): TYPED_INT stores an all-integer column as a presence bitmap plus 128-value frame-of-reference bit-packed blocks ([width][zigzag-varint min], width 0 = constant block — the ZPS2.postshape) with a zigzag-varint delta residual; blocks whose spread exceeds u32 lanes keep the JSON fallback, floats/strings never qualify. A dict-eligible integer column races the two codecs and keeps the smaller (the Parquet pick-the-smallest rule the.dvZNV2 chooser uses). COPY_OF stores a later column byte-identical to an earlier pass-1 column as a 4-byte reference (digest + full-equality verify before writing), so an id echo in_sourcecosts 4 B instead of a second full payload. The long-effort chooser — merge path only, flush stays pinned to zstd 3 after the ingest regression — additionally compresses each JSON fallback column at level 19 with long-distance matching and an explicit ≤ 16 MB window and keeps whichever payload is smaller. Two measured reasons it is a chooser and not a level bump: the window knee is sharp (body 29.8 MB column: 679 kB at the default window, 330 kB at 16 MB — the sentence pool repeats just past a small window), and high levels REGRESS small-alphabet streams (the id column: 54 kB at zstd-3, 143 kB at 19) — a global 19 would grow the ids while shrinking body. Offline A/B and the container-walk dissection are recorded inbenchmarks/index-size/DESIGN.md(new — it fixes the dangling link the #1039 README and this epic's anchor list pointed at). Compatibility: theZBS4magic is written only when a column actually uses codec 5/6 or a form-1 RAW payload; anything expressible as ZBS2 stays byte-identical ZBS2 (flush parity and downgrade blast radius unchanged,XERJ_FLUSH_PARITY=1verified on a full 100k harness run), ZBS3 present-null wrapping composes, the estimator counts COPY_OF as its source's bound and fails closed on crafted forward references, and the full decoder's fixpoint resolves reference chains while hard-erroring on cycles. Measured on the 100kbenchmarks/index-sizeharness at LEVEL=balanced against a same-day main control (results/result-zbs4*-balanced.json):.seg−73.1 % (1,671,271 → 449,094 B), total durable −22.6 % (5,398,478 → 4,176,300 B); every non-.segextension byte-identical but for a 1 B.jsonmanifest delta (47,366 → 47,365) between the runs. ES-YAML conformance 1380 passed / 0 failed; 10 new storage unit tests cover typed-int round-trips (nulls, negatives, i64-extreme clusters), copy-of decode across full/projection/hydration paths, form-1 decode on all four read paths, and the malformed-payload rejections. (PR #1042.).postblock framing slimmed (ZPS2) (#1038): the packed doc-id-delta and term-freq streams in every 128-doc posting block switch to frame-of-reference coding —[width][vbyte min]with the payload length derived instead of stored, width 0 meaning a constant block that carries no payload at all — and the positions block loses its u32 length prefix (exactly 128 count+delta groups pin its end). A positioned term whosettf == dfomits its ENTIRE freq stream; the reader re-derives that predicate from the.metarecord it already holds, so the decision costs no bytes. An all-equal block now costs 2–3 bytes where the ZPS1 framing paid 21 (1 + 4 header bytes plus a payload of forced-minimum width 1). The frame is subtracted only where it pays: when it narrows the width, or on narrow (≤ 4-bit) blocks whose level-shifted repeats normalise into identical payloads for the outer zstd pass; wide equal-width blocks keep raw lanes, because rewriting them measurably inflated the compressedtop_docfield 12.7 % under an always-FOR encoder while its uncompressed stream shrank. Same shape as tantivy's sorted blocks (vint base + packed residuals) and Lucene 50's per-term freq elision; adapted, not copied. Segments written before the bump stay readable — the envelope magic (ZPS2 vs ZPS1 vs ZPL1 vs raw) selects the inner framing, and old formats keep their decoders. Measured on the 100kbenchmarks/index-sizeharness at LEVEL=balanced against a same-day base control: total −1.5 % (5,517,127 → 5,432,531 B),.post−3.4 % (2,449,405 → 2,364,905 B),body.post−4.3 % (1,828,102 → 1,749,695 B); the widest keyword family (top_doc, 12 terms × df ≈ 8.3 k) moved +1.5 % (PR #1041)..dvnumeric columns get a bit-packed codec (ZNV2), chosen per column by measured size — stage 1 of the index-size epic #1038). The column is mapped through the monotone sign-flip bijection, given a global frame-of-reference (u_min) and a GCD divide, then stored as 128-doc blocks of bit-packedu32residuals — the shape tantivy's columnar uses (min-subtract + GCD-divide + bitpack; citations inbenchmarks/index-size/DESIGN.md). Constant blocks (width 0) carry no payload at all, two-valued columns such as booleans collapse to 1-bit lanes after the GCD divide, block payload length is derived from the width byte (no per-block byte_len to store), and blocks whose post-frame spread exceeds 32 bits (mixed-sign f64 bit patterns) fall back to rawu64lanes. The writer compresses both candidates —ZNV2and the rawZNV1layout — and keeps whichever is smaller after zstd at the actual level (the Parquet pick-the-smallest rule): the first cut shipped bitpack-always and the size harness caught it at 2.0× the.dvsize on the 100k corpus, because that corpus cycles its 4k events and the outer zstd pass matches the repeating raw bytes almost for free, while block-local lanes destroy the byte-aligned repetition. With the chooser, the same 100k run measures.dv−9.0 % (355,156 vs 390,487 B) and total durable −0.9 % (5,483,068 vs 5,532,276 B) on that worst-case-for-bitpacking corpus; monotone strided columns measure ~5× smaller than raw pre-compression in the unit tests, and real timestamp/counter columns have that shape. Null slots are excluded from the frame/GCD and re-zeroed at decode, so the decoded array is exactly whatfrom_iterbuilds and the f64-orderedsortedindex is rebuilt unchanged.ZNV1and the pre-magic legacy layout still decode via magic dispatch, so indexes written by older builds remain readable. (PR #1040.)The request-cache seen-set no longer allocates on the first tracked search (#1024): a lazy seen-set took idle per-index RSS from 206 to 64 kB — ~3× margin under the 0.2 MB/index idle budget line. (PRs #1025 and #1034.)
Documentation
- The README says how XERJ works with Jev, in plain words (PR #1010).
- A field report from a local
/_decideemail-classification prototype session (PR #1027). llms.txtstatus claims caught up withmain: rerank/share/mbox are onmain, #948 is closed by #1002 with the new memory numbers, and the residual per-segment cache retention is re-anchored to #1032 (#1028). (PR #1033.)- The CHANGELOG's rc.19–rc.70 gap is closed (ROADMAP GA gate item). The 52
releases of 2026-08-19 → 2026-08-26 are reconstructed in
CHANGELOG.md from per-release manifests of
git log(merge subjects and commit bodies) plus each tag's GitHub release record — 143 entries carrying 281 issue/PR links, every link checked to sit inside its release window, 52/52 sections verified against their manifest (51 first pass, 1 rewritten). They are honestly drier than ship-time entries — numbers quoted only where a commit body recorded them — under a "Reconstructed 2026-09-26" banner, and nothing was force-fitted: the three entries still unreleased at rc.71 stay under rc.72's "Recorded late". The same PR wrote the post-rc.77 window into Unreleased above and made three audit corrections, notably that the #874 idle gate is calibrated to 256 kB/idx, not the 0.2 MB line (that is #1024). (PR #1035.) ROADMAP.mdrolled forward to a 2026-09-26 review — the stale-evening corrections, plus the rc.78 queue. The 2026-09-21 review of the file was stale within hours: #950, #941 and #1015 closed that same evening, #874 the next day, and the zero-token stage-1 gating trio #937–#939 closed the same evening — every one recorded in the file as open or "under way". This pass re-checks every status claim against live tracker state: the Next release section now records what actually landed since the rc.77 tag (#1009, #1017, #1018, #1020, #1021, #1023, #1025, #1026, #1033, #1034) with PR links — including #874's idle budget met at ~3× margin (206 → 64 kB per idle index) and discussion #1012's email-labelling measurement (1.000 on the templated tier, 0.625 at 0.902 mean confidence on the hard one) — the open-defects shortlist shrinks to #1031 and #1032, the CHANGELOG-gap GA item is marked closed 2026-09-26 (the rc.19–rc.70 backfill, PR #1035), the stage-2 object-storage bullet stops contradicting the file's own Shipping today section (#965 wired in rc.77), and the mail-ingest line is corrected to "shipped in rc.75", carrying the post-#1002 memory numbers (variant-C retention ~677 → ~108 MB) with #1032 named as the residual. The review line itself is scoped honestly as a desk review of post-rc.77mainand the live tracker — not a live re-verification — and the milestones claim is true again (rc.78 milestone created, all four open issues triaged). (PR #1036.)- Every other public surface caught up too — the 2026-09-26 status sweep.
A 70-agent post-merge audit confirmed 58 stale-status findings beyond the
llms.txtset #1033 fixed; this sweep corrects them all:llms-full.txt's NOT-IMPLEMENTED block (object storage removed from it, remaining rows re-stamped against the code), the "no email handler" blanket claims across the answers/compare pages (mbox/.emlshipped rc.75),RERANK.md's falsified "min_scoremeans the same thing on every query" (replaced with the measured ECE 0.10–0.31 calibration and rank-don't-threshold guidance),ZERO_TOKEN_DIRECTION's autoindex-resilience row (fixed in rc.75, #934),XERJ_VS_LUCENE's_update_by_queryfetch cap (post-#1023_id-keyset paging),SCRIPTING.md's code anchors (re-pinned after theindex.rsrefactors), the mail-recipe/takeout memory numbers (reframed as dated before-#1002 state), and the S3-alerting-plugins answer page thatllms.txt:50cites as proof of the rc.77 S3 status while still denying it — rolled to the rc.77 reality with the correction stated in the open. Landing twins, both hubs and the sitemap regenerated viabuild_articles.py --write. Verified per-group adversarially and by a final gate (constants guard,--checkfor articles and sitemap, stale-phrase greps clean). (PR #1037.) - A stage-1 index-size write-up on the blog: "Cutting the index 28%". The
post covers epic
#1038 stage 1 (ZNV2
#1040, ZPS2
#1041, ZBS4
#1042, ZFM5
#1043): the dissect-first
method, the four per-family A/B tables, the raw-vs-index baselines
(63,461,256 B raw → 3,989,254 B durable, with the repetitive-corpus caveat
stated), the final composition, and a losing-rows table (bitpack-always
2.02×, the window knee 679 → 330 kB, ids 54 → 143 kB at level 19,
top_doc+1.5 %, the stride-kept.metaat 138,727 B). The headline is anchored to the committed same-day pair 5,398,478 → 3,989,254 B (−26.1 %), with the −27.7 % reconstruction as the rounding behind "28%", anddoc_id.metaafter ZFM5 is the on-disk 216 B, not the 193 B offline estimate. The pre-publish verification pass also corrected two CHANGELOG numbers in place: ZBS4's total is −22.6 % from its own printed pair, and the byte-identical claim gained its 1 B.jsonexception. (PR #1044.) - The xerj.org corner brand says XERJ, not XERJ.AI — and every blog post now ends on a
conclusion plus a way to try the engine. The top-left
<a class="brand">nav link had been rendering the domain instead of the product name; it now reads "XERJ" on 179 hand-written and generated pages, in the print-header stamp on the 44 docs pages, in the dashboards corner span (landing/playground/src/ux/chrome.jsand the committedlanding/playground/app.bundle.js, edited identically — no node in this sandbox to rebuild), and in thescripts/seo/build_articles.py:676template, so regeneration cannot reintroduce the domain. Footers, tab titles,og:site_name/JSON-LDalternateNameand brand-book specimens deliberately still read XERJ.AI — the scope was the corner, not a de-domain sweep. The blog endings review (user ask 2026-09-27) gave does-xerj-beat-jev a "05 · THE ANSWER" section and jev-vs-a-bm25-vote an "08 · THE TAKEAWAY", each scoped where the posts' evidence is (the ±0.01 tie rule to hosted-judge comparisons, the label-count crossover to Banking77, the calibration superlative to zero-shot arms; the superseded links-dump removed), and added a try-it paragraph to the blog hub (one binary, quickstart, playground, your own corpus as the benchmark that matters);dateModified/UPDATED bylines bumped to 2026-09-27 on the two edited posts with published dates unchanged, and every figure in the new copy verified against the posts' own bodies. Gates build_articles, heads, links, og card, seo_lint (0 violations) and the constants guard all green; sitemap regenerated in the same push to dodge the lastmod trap. (PR #1045.)
[1.0.0-rc.77] - 2026-09-21
The stateless-index and reader-fairness release. Two headline changes:
storage.backend = "s3" starts for real (#965), and readers stop starving
under sustained ingest (#1013) — both with fail-before reproductions.
Added
storage.backend = "s3"works (#965, PR #1008). Before this commit the startup guard refused the setting outright, honestly: a flush uploaded one file per segment while a 25-field segment family is 104 files,snapshot.jsonnever left local disk, and a fresh node pointed at the bucket saw zero segments. Now the whole segment family packs into ONE immutable object — a ZBM1 bundle whose footer is the existing ZCM1 flush-completion manifest extended with absolute offsets (trailer shape adapted from quickwit's split-footer bundle, Apache-2.0, approach only). One object per segment plus the catalogue PUT is ~18% of Cloudflare R2's free Class-A tier at a 30 s flush interval, where 104 PUTs would be ~899%.snapshot.jsonper index is the publication point: merges publish output bundles before retiring inputs, boot fetches the catalogue and adopts backfill, reads hydrate families on demand, per-index meta rides the same bucket. Config grows to 128 settings (storage.s3_bucket); an empty bucket name stays a hard error (XERJ never calls CreateBucket). Fail-before, verified live on the base commit:an_s3_index_round_trips_to_a_fresh_nodepanicked "the flush must publish the snapshot catalogue". Honest limits stay documented indocs/OBJECT_STORAGE.md: single-writer v1, the WAL stays local, in-memory packs with no multipart upload, crash-consistency pinned against the in-process simulation only.
Fixed
/v1/systemonevotes on the payload, never on the instruction prose. A question'sinstructionsused to be concatenated into the retrieval query, so a well-written instruction destroyed accuracy — measured on the SMS history (300 held-out, seed 7): 0.9867 empty vs 0.6700 criteria-rich, a 32-point swing from wording alone (#1000). And an objectstatecontributed only itsqueryfield, so{"message": …}voted on the instruction at the spam base rate: accuracy 0.1733 vs 0.9667 for the same message as a string (#1001). The vote text is now exactly what the question points at: its backtick references (instructions data fields first, then state paths — both client shapes keep working), or, when it references nothing, the state's own string content (a string whole, an object by all string leaves). A reference that resolves to structure rather than text — the rubric objectjev-reranker≥ 0.1.2 ships with its relevance preset — is acknowledged and skipped, not embedded and not fatal. References that resolve to nowhere and payloads with no text are 422s naming the question. Also fixed theno_supporterror body, which interpolatedkand the question ids into each other's slots. Fail-before: five newsystemone_httptests, all failing on the old code ([0.0, 0.0, 0.4545, 0.0]across four instruction wordings of one state); a sixth pins the client's rubric shape, a 422 before the skip rule. After: the 300 held-out messages score 0.9767 identically across all eight wording/shape arms, and the pip-client acceptance gate passes through both client entry points (neutral-query gap 0.8542) and under a deliberately spammy query (gap 0.6087, threshold 0.3) — client 0.1.2, unmodified.Readers no longer starve under sustained ingest (#1013, PR #1014).
CollectionPublicationis a seqlock;search()admitted a reader and validated the token at the end of the search — demanding a writer-free interval spanning the whole scan. The flush path holds its guard across the entire finalize, concurrent flushes are allowed, and every document write takes its own bracket, so under sustained ingest writer-free search-length intervals stop existing: search anddelete_by_queryretried to their deadline and returned HTTP 500 (reproduced twice in the #950 profiling run). Validation now happens at the capture boundary: a generation reference before the first collection-state read, validated with the seqlock evenness check right after the memtable snapshot — readers need a writer-free instant, not a writer-free search. A straddled capture returns a typed retry root (CollectionCaptureCrossedPublication, downcastable throughanyhowcontext wraps) andsearch()retries until its deadline. Fail-before (search_completes_under_publication_churn): a churn thread holds a cancelled 1 ms publication bracket every 3 ms — old code FAILED at its 2 s deadline with exactly the production error; new code passes in 0.37 s, total exactly 5050. Gates: 723/723 engine lib tests, ES-YAML 1376/0 failed. Follow-up #1015 tracks shrinking the flush bracket itself (a latency consideration now, not a starvation hazard).The console tour's pill assertions raced the route hand-off (#1011). The browser security suite's policy tour waited on
aria-busy === 'false', but app.js only ever sets aria-busy to 'false' at the END of a render — nothing flips it 'true' while the next route's query is pending — so the wait was satisfied by the previous route's still-mounted DOM, and the fixed 150 ms sleep then read the old view's pill. Failed only on slow runners (PR #1008's CI): the window scales with runner latency. The tour now waits for the route's own scene, and a latency-injection demonstration failed the old assertion deterministically at 400 ms.CI: the reference-coding toolkit's tests never ran.
cargo test --features x -- xc_ ypasses one filter per invocation, soxc_matched nothing and the step was green on zero tests (3aba5786). And the toolkit job's MCP schema gate looked for a release binary that job never builds (8f8e5444). Both fixed; the job now genuinely executes.docs: roadmap re-reviewed against rc.76 (release-drift guard, 213ce8f8).
[1.0.0-rc.76] - 2026-09-21
The JEV-interface and honest-fixes release. The headline is a measured answer to a specific judge: the node now speaks the Jev reranker's wire protocol locally. Everything else in this cut is a fix with a reproduction behind it.
Added
The node speaks System One —
/v1/systemone+/_decide. A rank-weighted kNN vote over a labelled-history index; nothing leaves the node. Two deliberate, test-pinned breaks from the hosted protocol:modelechoesxerj-history-vote-1(never a Jev name), and zero support is a 422, never a fabricated 0.5. Acceptance gate: the unmodified pip-installedjev-rerankerran against the node and passed its own strict validation (spam 0.9014 vs ham 0.0956)./_decideon the ES-compat port returns the evidence — neighbours, labels, engine scores, weights. (#975)Two public datasets on the record. FiQA (BEIR, 57,638 docs / 648 judged queries; harness validated 0.2382 vs published 0.236): BM25 0.2382 → Jev 0.3638, the judge's biggest lift and its worst calibration case (ECE 0.3109). AG News (120k train history, all 7,600 test items): 0.9182 accuracy, no model, no tokens. Hosted-reranker rows stay labelled "not run by us"; no Jev row exists for AG News and none is claimed.
WAND fast path in xerj-fts — 2.0× on the vote, bit-identical.
wand_should_bool: doc-at-a-time top-cap evaluation for the shape everymatchquery lowers to, strict-below pruning, allocation-free inner loop. AG News vote 68.7 → 34.2 ms/item;/v1/systemonep50 70.6 → 42.7 ms. Bit-identity proven three ways: BEIR figures reproduce to four decimals, a 648-query A/B over a byte-identical index found 0 order and 0 score-bit changes, ES-YAML conformance 0 failed on the final binary.xerj codeandxerj corpus add|index|list— the reference-coding loop is in the binary (#977).xerj code <corpus> "<query>"retrieves the matching definition from a locally indexed corpus of peer projects (-k,--lang,--mode bm25|semantic|hybrid,--full,--no-symbol,--meatl,--json,--stale-ok,--url,--api-key), keeping the wrapper scripts' exit triangle:0hits,1no-match (including--jsonwith empty hits),2usage/transport/30-day-staleness refusal,3corpus-in-ledger-but-not- loaded-on-this-node.xerj corpus addclones (or rebuilds from a pinned manifest via--from, preservingreviewblocks), detects licences with the restrictive-first heuristics and warns on approach-only sources;xerj corpus indexis the #930 build-verify-swap (unverified builds are removed by exact name, an uncountable old index is never presumed empty, state switches by atomic rename only after_count > 0);xerj corpus listshows what is actually loaded on the node. Hybrid retrieval now fuses server-side — one native top-levelhybridquery, RRFk=60, aimed at the semantic-capable indices, with an arms-ran note on every run. Existing~/.xerj-codecorpora, state files and indexes work unchanged: same layout, samecorpus.jsonschema, samexc-<corpus>namespaces. Thexerj_code_searchMCP tool (the 11th) exposes the same pipeline to shell-less agents, withlicence_policy: "strict"stripping passage text from restricted-licence hits. Nopython3/curldependency remains;gitstill is one (forxerj corpus add). Semantics live in the newxerj-common::xccodepure core, shared byte-for-byte by the CLI prose and the MCP tool text.
Fixed
The rerank stage, fixed and measured honestly. As it shipped in rc.75 the
rerankstage scored 0.3822 nDCG@10 where BM25 alone scored 0.7750 (40-query SciFact pilot) — every candidate sat in one sharedstateblob, so the judge scored the pile once. Each candidate now rides in its own noul: 0.8299 through the real stage path. Full runs, 3 shuffled repeats each: SciFact 0.6572 → 0.7410, NFCorpus 0.3016 → 0.3312 (still losing to the local hybrid 0.3448 — kept, not hidden). Two published caveats: provider non-determinism is ±0.01 nDCG@10, and the probabilities order well but are not calibrated (ECE 0.1023 / 0.1332).Ingest memory: a 1 GB mailbox no longer aborts under the default 16 GiB cap, and at-rest settled memory halved (#948, PR #1002). A 1 GB mailbox peaked at 25,356.9 MiB (24.8 GiB) VmHWM under the 16 GiB cap (aborted: 510/2,530 needles missing, 0 edges; 68.5 GiB uncapped). The ingest-memory ledger found four causes:
memtable_shardsretained one parsedArc<Value>per explicit-id write forever (flush drained the FTS memtable, never the storage shards — ~2.4-3.1x source bytes invisible to every ledger category); raw-bytes entries charged a flat 800 B regardless of payload; merge batches were cut by segment count while the executor materialises 5-9x input bytes; and the RSS breaker only 429'd, never drained. Now: published memtable entries are pruned at flush finalise, entry charges are structural, merge batches are cut by input bytes (XERJ_MERGE_BATCH_MAX_INPUT_MB), and an engaged breaker drains (flush + release caches + jemalloc arena purge, rate-limited). Measured, same harness both sides: at-rest settled 972.8 -> 481-488 MiB (variant A, 3 runs) and 883.0 -> 280.8 MiB (variant C, 100k x 100 B; retention ~677 -> ~108 MB). Merge batching follows the quickwit/tantivy approach (streaming_writer.rs:378, log_merge_policy.rs:22 — no code copied). Residual, separately tracked: three per-segment caches (stored_value_cache,dv_cache,id_pos_cache) remain unbounded until a merge retires their segment — #1032.scalar8 keeps its promise: 2.3x faster kNN, and
_scoreno longer depends on the filter (#392, PR #999). scalar8 advertised a ~4x smaller vector working set and an ES-like score and delivered neither: candidates were quantised per query from_sourcef32s (nothing smaller resident), and the SQ8 codebook was fitted per query over the candidate set, so_scorechanged with the filter (measured: up to 2e-05 drift, 19/30 order swaps from an unrelated-only filter). Codes are now written at ingest into a per-field flatSq8CodeStore(slot-addressed, tombstoned on delete, codebook fitted from ingested vectors and widened on out-of-range), re-derived by a background authoritative walk on restart, and served byrun_knn_sq8_codes_scan— which fails safe to the exact scan on any coverage gate miss. Measured, 50k x 128-d cosine: recall@10 0.9700 -> 0.9620, p50 0.55 -> 0.24 ms (2.3x), codes resident exactly 50k*128 bytes, filter drift exactly 0.0. Scoring working set 4x smaller; total memory is not (f32_sourcekept) — docs say both. qdrant's encoded_vectors_u8.rs approach adapted (Apache-2.0, cited in module docs); Lucene approach-only.The Windows build compiles again: the jemalloc purge hook is cfg-gated out on MSVC (PR #999). #948's boot hook called
tikv_jemalloc_ctlunguarded, but the tikv crates are target-gated out on msvc in xerj-server/Cargo.toml — every merge-ref off main failed the Windows FD smoke with E0433 since #948 landed (the serialised runner queue completed the job after the merge decision, which is why main slid red for two merges). Statement-level#[cfg(not(target_env = "msvc"))]on the hook install only; the engine's default no-op purge covers MSVC and the resource sampler still runs everywhere._bulkno longer truncates an action-line_idat an escaped quote: two ids sharing a prefix up to a"collapsed onto one stored id and the second document silently overwrote the first;\\,\/and\uXXXXin ids were stored as literal escape text (#954, PR #983).The docs fact-check rule no longer rejects
xerj search, which exists: the pattern's subcommand list predated the query clients, so every page demonstrating the search CLI shipped with an ERROR-level finding (#963, PR #978).mcp:xerj_searchdefaults its index to theax-*autoindex prefix, so the documented no-argument call works on a fresh node (#962, PR #980).xerj feedback --open-pronly runs in a checkout of xerj-org/xerj — it no longer commits and pushes into whatever repository it is run from (#960, PR #981).Console passkey enrolment works on any port: the WebAuthn RP is derived from the bound address instead of a hard-coded localhost (#935, PR #987).
Filters around a hybrid query are applied instead of ignored: a
bool{must: hybrid, filter}pushes the filter into every leg, andpost_filterbesidehybridis rejected with a 400 (#943, PR #988).filter/filtersaggregations with wildcards or regexps share the query path's matchers instead of counting 0 or everything (#959, PR #992).autoindex --no-graphreplays the index phase through a bounded parallel window — 1.64× the index phase on the repro corpus, deterministic by construction (#933, PR #989).xerj initon a default (auth-on) node writes an MCP entry that actually works: the loopback console falls back to the admin key instead of answering 401 to every tool call (#961, PR #982).A
fieldsclause is no longer silently dropped when_sourceincludes omit the field — both apply, as siblings (#932, PR #991).A declared per-field analyzer keeps applying after
_flush: analysis config persists instead of reverting to the memtable-only default (#937, PR #994).Neural ingest no longer keeps ~3.4 of 32 threads busy behind one window: bounded window concurrency for a single
_bulkstream (settings count now 127, enforced byjourney_zero_config) — first slice of #938 (#938, PR #995).autoindex: a one-file change no longer costs O(corpus) twice — snapshot and catalog updates are incremental (#971, PR #998).Semantic/hybrid over multi-passage documents hydrates only the winners instead of deep-cloning every
_sourceper query — the repro query drops ~410 ms (≈2.3×) (#939, PR #979).Generated articles carry an explicit
updated:date so regenerated pages stop drifting on merge (#974, PR #984).Console graph (Reader panel, Second Brain dashboard) reads go through the signed-in console session instead of keyless routes — a signed-in operator on an auth-on node gets the graph, and a refused listing says so instead of rendering as an empty graph (#936, PR #996).
The Second Brain dashboard and the reader's graph panel are no longer 401 for a signed-in operator on an auth-enabled engine (the default). The console holds a passkey session, not an engine API key, so the SPA's direct
/_graph/{brain}/*,_cat/indices/.xerj-memory-*and reserved-namespace_searchreads were refused — while a share-link guest, who holds a key, could read the graph. The console now serves its own brain-scoped, session-authorized graph read path (GET /_xerj-console/api/v1/graph/brains,/{brain}/ego,/{brain}/overview,POST /{brain}/edges/_search,POST /{brain}/nodes/_search) in the samexerj-second-brain/1contract as the data plane: console rolesownerandadminmay read brains, a role-refused read and a nonexistent brain are byte-identical 404s (no existence oracle), and the nodes/edges searches are pinned server-side to the brain's own meta-doc indices, so the endpoints cannot become a general search proxy and the generic data-sources proxy still refuses the reserved namespace./v1/metrics(the dashboard's searches-per-index tile) stays key-only by decision; the tile shows the refusal honestly. #936
Packaging
- .deb assets ship with every release (amd64/arm64), built by
scripts/build-deb.shand verified byscripts/verify-release.sh; docs/PACKAGING_DEBIAN.md states the Debian archive path honestly (#810).
Removed
tools/xerj-code/scripts/{xc.py,xc-index.sh,xc-corpus.sh}and their Python/shell test suites. Most behaviour is ported and gated by scoped cargo tests (xerj-common::xccode,xerj-autoindex::xc, the MCP schema drift test); the--freshswap contract — the destructive side ofxerj corpus index --fresh— is re-pinned inxerj-autoindex/src/xc.rsas ten contract tests running the REAL flow against a fake node and a fake autoindex runner behind injected seams (#1004): switch-readers- before-retiring, deletes by exact name only, a failed or empty build leaving the old index untouched, a count the node never answered authorising no delete and no switch, sibling corpora never touched, first-build salvage, resume under the recorded prefix and state dir, build-id collisions inside one second, and the legacy no-ledger path. The deliberate drops are the post-query field-report nudge (2026-09-18 llms.txt directive: no obligation language) and per-hit hybrid arm annotations (server-side RRF exposes no per-leg ranks).
[1.0.0-rc.75] - 2026-09-20
Added
xerj share <index|folder>— give one person read-only search over one indexed folder: a link, a passcode, an expiry. The guest opens the link, types the passcode and gets a reading room (search, highlighted snippets, a document view) served by the owner's node; the folder is not uploaded and the guest installs nothing.POST /_share(admin key only) stores only a SHA-256 digest of the share id and an Argon2id hash of the passcode; the unauthenticated, rate-limited, auditedPOST /_share/claimtakes{id, passcode}in its body, so the share id is not in the request line an access log records, and mints a scoped read-only key that expires with the share; a guest-only route allow-list closes_cat,_cluster, every other index, all writes, and scroll / PIT / async contexts. Every response under/_shareand every response to a guest key isno-store; a guest's refused attempts to manage shares or mint a key are audited; theautoindex-catalogindex cannot be shared.--tunnelsupervises your owncloudflaredquick tunnel and revokes the share on Ctrl-C. That traffic passes through Cloudflare, which terminates TLS and can read the passcode, the guest key and the documents; the command and the guest page both say so.--list/--revoke <handle>manage shares; the command refuses a node running with authentication off, where a read-only key would restrict nothing.xerj brainnow prints the share command for the folder it indexed. Found and fixed on the way: the graph API authorized a multi-dataset brain's comma-joinednodes_indexas one literal index name, so every scoped key got403onoverviewfor any folder with more than one dataset (andoverviewreported 0 notes for it); and the Console asset bundle was not rebuilt when a new file appeared underxerj-ux/on a warm target directory. Threat model, guest reach table and the quick-tunnel trade-offs: docs/SHARING.md.xerj autoindexreads mbox mailboxes and Google Takeout exports — an mbox (Takeout, Thunderbird, Apple Mail, mutt) is detected by content, split in a bounded-memory stream with>Fromunquoting, CRLF/LF and a missing final newline handled, and every message goes through the same extractor as a standalone.eml, so message and attachment records (PDF pages included) have one shape; senders and recipients are filterable by bare address (email_from_address,email_to_address,email_cc_address), Subject words are searchable inbody, and Gmail's decimalX-GM-THRIDis stored as the hex id Gmail's web interface uses; messages are parsed on a--workers-wide pool under one process-wide in-flight byte budget and forwarded in order, the progress bar moves inside the file, a Takeout root'sarchive_browser.htmland Keep.htmltwins are skipped by named rules, Keep.jsonnotes are indexed, and an unextracted archive is named under the run's summary with the command to extract it. The newemail-thread@1detector writesreplies_to(In-Reply-To/References) andattachment_ofedges with evidence, and an incremental run keeps a reply across two mailboxes (Inbox → Sent) by loading the untouched mailboxes' messages back from the index. A loading run now waits the server's memory circuit breaker out — a 429, per item or as the HTTP status, is re-offered (only the rejected records) for up to ten minutes instead of aborting the run. Numbers, machine and commands:benchmarks/mbox-ingest/README.md(synthetic mailbox; not yet verified on a real Takeout export). Measured limits, filed on #948: the client peaks under 300 MiB on a 1 GB mailbox, but the node needs 66.9 GiB of RSS to finish it and does not finish under its default 16 GiB cap; at a 16 GiB laptop's 8 GiB cap a 300 MB mailbox completes with the node peaking at 18–20 GiB; and after a restart, a first-time query on the 1 GB index takes either under 50 ms or seconds (up to 16.6 s measured) until the node warms.xerj autoindex <folder> --watchkeeps an index current from filesystem events instead of a re-run. The session indexes once, then places one OS watch per indexed directory (notify: inotify / FSEvents / ReadDirectoryChangesW) and reindexes what changed, debounced (--debounce, default 400 ms) so one editor save is one pass. The watch set comes from the same walk the indexer uses, so an ignoredtarget/costs no watch and produces no events, and a watched run and a re-run agree on what is indexed. A file skips its re-hash only when no event named it or an ancestor AND its(size, mtime, inode)fingerprint is unchanged; the cache is in-memory, so a restart re-hashes in full. Requires--no-graph, refused rather than downgraded: reconciling an ADDED or DELETED file exists only on that route — on the graph path a re-run skips a file added after the resume plan was frozen (exit 3,appeared after the resume plan was frozen) and ABORTS on a deleted one (exit 1, and every re-run after it), so a watcher there would go stale on the first new file and stop reindexing on the first deletion. A file whose CONTENT changed is reconciled on the graph path, measured at 3.07 s; the earlier claim that it is not came from a measurement whose shell append had created a file instead of modifying one. Measured on a 10,000-file / 4,576,300-byte tree, single samples on a shared box: idle costs 0.00 CPU-seconds per minute and 0 bytes read (holding 157 MiB and 295 threads), against 1.7 s wall / 2.1 CPU-s and a full corpus re-read for every poll of a re-run loop. Per change--watchis not faster than re-running the same command (47.6 s / 6.6 CPU-s against 44.3 s / 7.6 CPU-s for one modified file); both beat re-indexing the folder from scratch (293.8 s / 127.9 CPU-s) by an order of magnitude. The pass cost is two O(corpus) terms neither route avoids — a ~13 s snapshot over the whole inventory (sync_executor::create_snapshot_inner) and ~27 s of server CPU rewriting one catalog document per file — the measured next lever, not fixed here. Hittingfs.inotify.max_user_watchesstops the run with the limit, the directory count and thesysctl, because a half-watched tree looks live and silently is not. Docs:docs/LIVE_REINDEXING.md, measurement record indocs/measurements/autoindex-watch-2026-09-19.md.xerj autoindex s3://bucket/prefixindexes an S3-compatible bucket — Amazon S3, Cloudflare R2 (r2://), MinIO, Ceph or anything else that speaks S3, via--endpoint-url(falling back toAWS_ENDPOINT_URL_S3/AWS_ENDPOINT_URL). The bucket is a source, not a second product: the prefix is listed with ListObjectsV2, each object is streamed into a local mirror under--state-dir, and the ordinary discovery pipeline — sniffing, the code and document extractors, the plan, the resume journal, the incremental reconcile — runs over that mirror unchanged. Credentials come only from the environment (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN) — no profile files, no instance metadata, no SSO, and never from the URL (s3://key:secret@…is refused by name).aws-configis deliberately not a dependency: it would re-enable the SDK's default HTTPS client and with it theaws-lc-rsC/assembly crypto backend that this workspace keeps out of its cross-compile matrix, and it would bring a second S3 client with the SDK's own invisible retry layer. Change detection is the ETag plus the size, treated as an opaque token: a multipart-NETag is stored and compared verbatim and never mistaken for an MD5, and a store that returns no ETag falls back to last-modified plus size with the count of such objects reported. Keys that cannot become a safe portable path (.., control characters, Windows reserved names, case collisions), dotfiles and the built-in build-output list are filtered out of the listing, so they never cost a request. Cost is printed by every run, in the two classes that are billed: a scan isceil(N/1000)LIST (class A) plus one GET (class B) per changed object, and the run also prints what the same command would cost daily, hourly and every five minutes against a 1,000,000/month free allowance. Those counts are billed wire attempts: the SDK's own retry layer is disabled and the client charges its counter before each attempt, so a throttled request that succeeded on its third try reports three, and one that exhausts its retries reports what it spent instead of nothing. A run that fails mid-transfer records the objects whose bytes already landed (and checkpoints every ten seconds during a long one), so the re-run pays one GET per object it had not already fetched rather than paying for all of them twice; the LIST cost is paid again. Measured end to end against a live node and MinIO: first index of six keys (including a 12 MiB real multipart object) 1 LIST + 5 GET; unchanged re-run 1 LIST + 0 GET; one changed object 1 LIST + 1 GET; one deleted object (--no-graph, the journal that reconciles deletions) removes exactly that object's documents. A 1 GiB object streams through for 4 MB of RSS growth (22 MB → 26 MB), in 281–318 ms over four loopback runs — the memory figure is the claim; the milliseconds are a same-host transfer and not a throughput benchmark. Nothing is ever written to the bucket, and the index stays on the node's local disk —docs/OBJECT_STORAGE.mdstates both, with the request arithmetic and the measured runs.A real S3-compatible object-storage backend — Cloudflare R2, MinIO and AWS S3 — with per-request cost accounting.
xerj-storage'sS3Backendwas a local-directory simulation that its own doc comment admitted to; it is now anaws-sdk-s3client doing rangedGetObject,PutObject, paginatedListObjectsV2andHeadObject, verified against both MinIO and R2 (including the 1,200-objectListObjectsV2page boundary, listed in exactly two Class A operations). The simulation is kept asSimulatedObjectStore, a test double, because unit tests want a backend with no network and no cost. Because object stores bill per request and Cloudflare R2's free tier allows 1,000,000 Class A operations a month per account — about 23 a minute for everything — the backend counts every billed attempt by class and exposes it throughStorageBackend::ops(), andOpBudgetrefuses to send more past a ceiling. Counted per wire attempt, which is why the AWS SDK's retry layer is disabled in favour of XERJ's own: an SDK-internal retry is invisible to a counter wrapped around the call. A listing is bounded by a seen-token check (which catches a continuation-token cycle of any length, not only an immediately repeated token) and byS3Config::max_list_pages, so a broken or hostile endpoint cannot bill a Class A request per page without end.SegmentCachegained hit/miss/bytes accounting,get_range(whole-object fetch on miss, then slice) andget_range_uncached(fetch the range only) — measured against MinIO on loopback at 5.98 ms cold, 1.67 ms range-only and 0.068 ms warm, and, in a single unrepeated run over a ~1 MB/s link that is not a performance claim about R2, against R2 at 3.72 s, 0.67 s and 0.12 ms.storage.backend = "s3"still refuses to start: nothing routes the index's segment reads and writes through the backend, the flush path that exists uploads 1 of a segment's 104 files, andsnapshot.jsonnever leaves local disk, so a fresh node pointed at a bucket sees zero segments — asserted byobject_store_mode_does_not_yet_make_an_index_stateless, which also shows that a fresh node can fetch and read a segment from the bucket by id. Full arithmetic, measurements and remaining work in docs/OBJECT_STORAGE.md.xerj autoindex s3://bucket/prefix --watchkeeps a bucket-backed index current with a poll whose cost is bounded and visible. An object store has no inotify, so change tracking is polling or event notifications. This lands polling, becauseListObjectsV2is served by S3, R2, MinIO and Ceph RGW alike and because a poll is self-correcting: every cycle re-derives the truth from the bucket, so a dropped notification cannot leave an index permanently wrong. Each cycle lists the prefix, compares ETag, size and last-modified against a per-object journal, reads only what changed, and emits one JSON event per added, changed or deleted object — it produces a change feed and does NOT index, and it is notxerj autoindex s3://bucket/prefix, which indexes a bucket once; wiring the feed into that indexer plugs into the sameChangeSinkand is still to come. The cost model is the feature, not a footnote:ListObjectsV2is a Class A operation at at least one call per 1,000 keys per cycle, charged whether anything changed or not (at least, because when a store stops paging is its own choice: MinIO serves 11 calls for 10,000 keys, not 10, so the projection takes whichever is larger, the arithmetic or what the store just served), against Cloudflare R2's free tier of 1,000,000 Class A operations a month (~23/minute for a whole account). A 5 s poll on an empty bucket costs 518,400/month — half the tier to watch nothing; a 60 s poll on a 100,000-object prefix costs 4,320,000/month, over four times the whole allowance. So the default interval is 300 s, the default budget is 200,000 Class A ops/month (20% of the tier, because the rest of the account spends from it too), and a cycle whose projection exceeds the budget is refused with exit 4 and a decision-request document naming the minimum safe interval — the same contract the folder-indexing gate uses. The budget is a CIRCUIT BREAKER on every cycle, not a greeting on the first: a prefix that grows past it mid-watch stops the watch, and so does a month whose allowance is spent (Class A and Class B are counted in<state-dir>/objwatch-spend.json, which survives restarts, so a supervisor restart loop cannot mint a fresh budget each time — the ledger is written BEFORE the calls it pays for, sixteen at a time, so akill -9in the middle of a long first scan still leaves the run owing what it spent, and a ledger stamped with a future month is a backwards clock step rather than a new allowance). GETs have their own budget,--max-monthly-gets, default 2,000,000 (20% of the 10,000,000 Class B tier).--append-onlyturns a growing key space into one list call per cycle viastart-after(and cannot detect deletes, which is why it is opt-in),--no-fetchis metadata-only at zero Class B operations,--dry-runprices a poll and records nothing at all (no read, no event, no journal) though the listing it makes is real and billed like any other cycle, changed objects are fetched 8 at a time with the events still emitted in listing order, keys come back from the listing byte for byte (whitespace and entity references included), an edit past--max-object-mbis always emitted because a prefix digest cannot prove the bytes are unchanged, and one transient listing failure after the first cycle is retried at the next poll rather than ending the watch (five in a row end it), and every cycle reports list calls, reads, bytes and wall time to stderr plus a status file an operator can read without attaching to the process. Cycles never overlap: the schedule is fixed att0 + k*intervaland deadlines a long cycle passed through are counted and reported, because an overrun means the effective interval is longer than the one the budget was computed from. Event-driven delivery (S3 Event Notifications, R2 event notifications) is not implemented; what it would take, including the reconciling pass that at-least-once delivery makes mandatory, is written down indocs/WATCHING_OBJECT_STORAGE.md.hybrid: trueinPOST /_memory/{ns}/_recallfuses BM25 and server-side semantic recall inside the memory API (#918). Recall used to pick exactly one mode —vector,semantic: true, or plainquery— so an agent that wanted fused recall over a namespace had to bypass/_memoryand issue a rawhybrid_searchagainst the backing index, losing the memory hit shape, therecency_weightblend, the graph coupling and the namespace authorization path.hybrid: truenow runs the BM25 leg and the semantic leg over the samequeryas the two legs of the engine'shybridquery (reciprocal rank fusion,fusion: "linear"opt-in), each leg keeping the filter shape it dispatches with and fetching the same over-fetch width the re-rankers already use. Off by default;vectororsemantic: truebeside it,fusionwithout it, andfusion: "learned"are 400s, so a request never silently degrades to one leg. The MCPxerj_memory_recalltool exposes the same two parameters. Raised by @Vinz2168 from a shared-memory agent integration where neither single mode was enough.rerankstage on_search: hand the top hits to an external relevance judge and reorder by the calibrated probability that comes back."rerank": {}sends the question and the text of the topwindowhits (default 30, max 300) to TypeSafe AI's Jev; the 0–1 probability replaces_score, sorerank.min_scoreis an absolute cut-off, which a BM25 score cannot be. It is the only search-time feature that sends document text off the node ([embedding] default_endpointand the WAL tap also send text off the node when an operator configures them; docs/RERANK.md lists every outbound connection a node can open, and a test fails when the engine source gains one that list does not name): inert until an operator sets[rerank] api_key(orTYPESAFE_API_KEY; config wins over env), opt-in per request, forbidden outright by[rerank] enabled = false, and only fields the response returns are sent —rerank.fieldsis an exhaustive allow-list. Failure policy is degrade on deadline, surface on contract: a slow provider is a 200 with the engine's order and_rerank.applied: false; no key is 503, disabled 403, a rejected key or malformed body 502, all with no hits; a provider that answers for nothing it was sent is a 200 withapplied: false. When applied, every_scoreis a probability ornull(a hit with no verdict, counted in_rerank.unjudged, sorts last) — never an engine score beside probabilities.hits.totalandaggsstay the engine's; paging happens inside the window;sort,search_after,collapse, scroll andsize: 0are 400s; and_msearch, search templates,_async_search,_rank_eval(per request, underfailures), the native/v1search API and gRPC refuse the block instead of dropping it. Every caller-chosen cost knob has a server-side ceiling, the strings included:instructionsis capped at 2,000 characters because the provider's wire format repeats it once per judged document, the question at 4,000 andmodelat 128; a provider response over 2 MiB is a 502, not an allocation.GET /_xerj/rerankreports whether a provider is configured and never the key;/v1/metricsgainsxerj_rerank_requests_total{outcome},xerj_rerank_documents_judged_totalandxerj_rerank_provider_tokens_total{kind}; the MCPxerj_searchandxerj_hybrid_searchtools take an optionalrerankargument. Ranking quality with the real model is not verified — no provider key was available, and every HTTP test runs against an in-process test double. New cratexerj-rerank; reference indocs/RERANK.md; two benchmarks,benchmarks/beir-hybridandbenchmarks/decisions-as-retrieval, both measured with--embed-mode neural, not the default lexical embedder. The one-question-per-document request shape followshev/jev-rerank(Apache-2.0); the failure policy and the retry back-off constants follow Meilisearch's personalization module (MIT; adapted, cited in code).
Fixed
xerj autoindex: one dataset the server refuses no longer aborts the run,--no-graphprogress reports the phase it is in, andxc-index.sh --freshworks on a corpus that was indexed before (#929, #931, #930). On rc.74 a single HTTP 400 on one dataset's mapping ended a 48,533-file run withexit=1 reason=abortedand nothing indexed. A 400 on create-index / put-mapping is now a refusal of that dataset: its files are recorded as junk with the server's reason, every other dataset is indexed, the run exits 3, andxerj-done, the catalog run document andxerj autoindex mapall carrydatasets_refused/files_refused, so a generation that lacks a dataset cannot read as a whole one. 401/403/404/408/429/5xx still abort. On the--no-graphpath the stream used to printphase=scan pct=100.0 eta_quality=stalledfor the whole time documents were landing (48 such lines in the rc.74 capture) — indistinguishable from a real hang; mapping install, sealing, indexing and the read-back barrier are now theprepare,snapshot,index,finalize-catalog,finalize-refreshandfinalize-verifyphases with real denominators.tools/xerj-code/scripts/xc-index.sh --freshno longer forwards the flag to autoindex (which refuses it once a generation has committed): it builds a replacement beside the old index, verifies_count > 0, switches the state file atomically and only then retires the old indices by exact name; a record count the node does not answer is never read as zero, so it cannot get a working index retired or a finished build deleted. A first build that is interrupted after writing records is kept rather than leave no corpus, andxc.pynow reads thesalvaged/autoindex_exitthe script always recorded: it warns on every query that coverage is INCOMPLETE, so a miss against a partial index cannot read as "this code does not exist". Found while verifying the above on the full corpus (#944): a per-item HTTP 429 inside one bulk — the node's memory circuit breaker, which engages and releases within seconds — aborted the 48,533-file run at 60% after 85 minutes. The rejected items are now re-sent, and only those, after a backoff, until the node takes them or the bulk's 600 s of patience runs out — a 429 on the whole bulk request is the same back-pressure, not six transport attempts (that split ended the resumed run); only then is it exit 1, resumable, with an error line that says so. The terminal line carriesbulk_retries=Nwhen it happened, and theraising bulk concurrencyline is printed at most once per 10 s (117 lines for 11 shrinks in the capture). Captures:benchmarks/autoindex-resilience/.xerj autoindex: a corpus whose catalog holds more documents than the server takes in one request no longer fails at the very end (#955). Both indexing paths sent the catalog (one document per file, per dataset, per run) as ONE_bulk. On the 48,533-file reference corpus that was 51,129 actions in 31.9 MB against the engine's defaultlimits.max_actions_per_bulkof 50,000: the--no-graphrun applied all 47,444 operations, then endedexit=1 reason=abortedinfinalize-catalogafter 10,336 s. On the default path the same answer was silent: exit 0 with an empty catalog (reproduced by a test against a stub that returns the engine's literal 413). Every_bulkbody now goes out in windows of at most 10,000 actions, and the catalog also under--bulk-mb. A request the server still refuses as too large (HTTP 413, or one item answered 413 for a body of several actions) is halved and re-sent, and the bound is kept for the rest of the run; the terminal line carriesbulk_splits=Nwhen that happened. Resuming that same generation with the fix committed it (ok=true exit=3 … records=821840, 51,129 catalog documents). A--no-graphrun that runs out of back-pressure patience now endsreason=server-backpressurewithops_applied/ops_remaininginstead ofreason=aborted; it is still exit 1 and resumable.Reference-code passage windows preserve their match score.
xc.pycould select relevant source and then discard its matches while aligning the excerpt to line boundaries, including on long source lines. Line alignment now keeps the selected term score or falls back to the bounded original window. Window scoring also uses original-source offsets when Unicode lowercasing expands characters. This affects the fallback when no matching symbol is available and explicit--no-symboloutput.xc.py --mode hybridkeeps BM25 results when the optional vector arm has a transport failure. Connection failures, read timeouts, and interrupted HTTP responses during semantic mapping discovery or search now take the existing BM25-only fallback instead of aborting and discarding valid hits. Primary BM25 failures still exit with an error. Standalone--mode semanticnow reports mapping/search HTTP and transport failures as errors (exit2) instead of treating failed requests as empty search results (exit1).xc.py --jsonemits JSON for empty search results. Previously, a query with no hits printed the human-readable no-match message before reaching the JSON output branch, breaking callers that parse stdout. Empty results now preserve the JSON response and still exit1; matching results exit0, and a corpus with no live indices retains its distinct exit3diagnostic.
Documentation
llms.txtproposals, revised and fact-checked (docs/research/llms-txt-2026-09/proposals/). A proposedllms.txt, anllms-install.mdand ten paste-ready install prompts, following the study's twelve rules: three complete entry paths (shell, MCP-only, HTTP-only), the lexical-by-default correction first, per-client MCP registration with the key, a verify line after every step, and a feedback ask that is optional and needs no git. Every quote is re-fetched (259 claims, 258 confirmed, the one failure removed) and the XERJ commands the proposals print were run on Linux against v1.0.0-rc.74, except the ones the record's own "Not run" list names — thecurl | shinstaller, macOS and Windows,--embed-mode neural,claude mcp add --scope localand the per-client registration lines, which are quoted from each client's own documentation with a fact-check id. Two product findings came out of it:xerj feedback --open-prbranches, commits and pushes in whatever repository it is run from, andxerj initwrites an MCP entry withoutXERJ_AUTH, so against a default node every tool call returns 401. The livelanding/llms.txtis unchanged; the report's ship checklist says what has to exist first.ROADMAP: the zero-token direction, with every status checked against the tree (#941). A new roadmap section and docs/ZERO_TOKEN_DIRECTION.md lay out judged search, share links and a guest reading room, mail ingest, semantic detections, a real object-storage backend, a block index mode for logs, user-code ingest plugins and a corpus hub of signed packs — and say plainly what exists today:
S3Backendwas a local-directory simulation andstorage.backend = "s3"refuses to start on purpose — the first half of that is no longer true in this same release, see the object-storage backend entry above; the refusal stands, because the index segment path is still local —_watcherstores watches and never evaluates them, alert rules have schemas and no evaluator,xerj-logshas no caller, and there is no wasmtime backend in the tree. The measuring behind it (benchmarks/neural-path-triage/) filed four defects with literal reproductions: #937 a declared analyzer stops applying at flush, #938 neural ingest keeps3.4 of 32 threads busy, #939410 ms p50 on 5,183 documents, forward pass ~14 ms), and #940 tied RRF scores change order across a restart. All four stay open; nothing is fixed by this entry.semanticover multi-passage documents is an exact scan that copies every_source(
Fixed
- A
hybridquery returns the same page on every run and after a restart (#940).fusion: "rrf"andfusion: "linear"drained their accumulator from a hash map into a score-only sort, so documents with an identical fused score came back in hash-iteration order, which is seeded per process. Under RRF such ties are structural (one leg's rank r and the other leg's rank r score the same), and on BEIR SciFact 21 of 40 queries changed their top 10 after a restart on unchanged data. Fused results now use the order every score-ranked page uses since #270: score descending, then arrival (_seq_no), then_id; a NaN fused score sorts last instead of making the sort panic. Measured on the fixed build, lexical embedder: the 300 SciFact test queries (1,509 tied adjacent pairs in their top 100) returned identical result lists across a flush and two restarts. The hybrid recipe's worked example was also re-run and its numbers reconciled with the current engine.
[1.0.0-rc.74] - 2026-09-08
Added
xerj initwires XERJ into the project's coding agents in one command, andxerj gainreports counted usage (#917).initdetects the agent surfaces present and writes the minimal config for each — the MCP server merged into.mcp.json(other servers kept, absolute binary path), a ten-line skill for Claude Code, a.cursor/rulesfile when.cursor/exists, and anAGENTS.mdsection — taking a.bakbackup of any file it edits, previewable with--dry-run, idempotent on re-run, and refusing to touch an invalid.mcp.json. It deliberately does not rewrite prompts or intercept commands: it only makes the tool visible.gainreads the node's hash-chained audit log and prints what actually happened — searches served, hit rate, p50/p95 latency, busiest indices, a recent-search strip — with no estimated-savings arithmetic of any kind; it falls back to the native listener (ES port + 1) where/_audit/_searchis served.
[1.0.0-rc.73] - 2026-09-08
Added
xerj searchnow runs a definition-first query, andxerj defis go-to-definition (#916). The search CLI's singlemulti_matchfound the correct cross-file source file at rank 1 only 38% of the time on an 80-task benchmark; a bool that adds a phrase clause on the ASTdefsfield plus an exact-symboltermonnamemeasures 94% on the same corpus, so that shape is now the default. Results are passage-first — the matching snippet and the symbol declaration, never whole file bodies (~9x fewer tokens across three code hits) — and a zero-hit identifier-shaped query nudges toward the new command.xerj def "<symbol>"answers the sharper question in one call: where is this defined and what is its signature, asfile:line [kind]+ declaration, with--kind/--langfilters; per-symbol documents (#500) first, adefsphrase fallback for indexes that predate them. The MCPxerj_searchtool accepts a plain stringqueryand applies the same shape and passage projection (it previously rejected strings with an HTTP 400, forcing agents to hand-write query DSL); DSL objects pass through unchanged.autoindex's run summary reports throughput asdocs/sec(#904). Split out of #366, which asked for this so an operator could see ingest throughput without timing the command by hand or inferring it from the elapsed line; #897 fixed that issue's neural-embedder batching defect but did not touch the summary output. The rate isrecords_total / wall(0 whenwallis 0.0, e.g. an all-resume no-op, rather than reportinginf), printed on thedone in …line and added to the--jsonrun document asdocs_per_sec.
Performance
Merge copies posting lists instead of re-analysing every document it merges (#876). A merged segment's postings ARE its inputs' postings with the document ids remapped — nothing about them depends on the source text — but the merge rebuilt each side-car by walking back to every surviving document's stored
_source, re-extracting its field values and re-running the whole analyzer chain. Under size-tiered levelling that re-analyses every document once per level, which is why a post-index merge tail could rival the index itself. The merge now streams each input's term dictionary, replays the decoded posting lists with the doc ids remapped and tombstoned documents skipped, and carries the norms and field statistics across.Measured on a 100 MiB text corpus force-merged to one segment (2 000 documents, 32-core box, median of four runs per arm): the merged segment's FTS side-car build drops from 3.15 s to 0.85 s (~3.7x), and peak RSS over the whole force-merge from ~2.8 GiB to ~2.4 GiB. End-to-end force-merge time moves less, ~41 s to ~32 s, and is noisy — because after this change the merge is dominated by the doc-values column build (7-12 s per batch against 0.85 s for FTS), which this change does not touch.
Everything a query can read out of a merged segment is identical to what the old path wrote — every posting list,
doc_freq, norm byte,_scoreandfield_length— and the.fstand.postside-cars are byte-for-byte identical. Two on-disk representations differ without any query being able to tell:- The
.normsarray may be SHORTER. Byte 0 spells both "field absent" and "field length <= 1", and the reader drops byte-0 entries, so a trailing run of such documents cannot be replayed and the dense array stops at the last non-zero document instead of at the last document carrying the field. A single-tokenkeywordfield is entirely byte 0, so its.normsshrinks to its header. Both files load to the same norms table. .meta'stotal_term_frequencyfor a docs-only (keyword) field: a document that repeats the same value merges withtotal_term_frequency = doc_frequency, because that format never stored a per-document frequency and its reader synthesises 1. Nothing outsidexerj-ftsreadstotal_term_frequency;.metais otherwise identical,total_field_lengthincluded, soavgdlstill counts the repeat.
Two further edges exist only when a merge also DROPS documents: a dropped document whose value analysed to zero tokens leaves no trace to reclaim its
total_docsseat, and a dropped docs-only document's length is recovered from the quantised norm byte, exact to length 7.This also changes one behaviour worth stating plainly: because postings are now preserved as written, a merge no longer re-analyses old segments under a changed mapping. Editing an analyzer and running
_forcemergeused to rewrite the old documents' terms as a side effect; it no longer does — that needs a reindex. A change to a field's POSITION setting is still detected and still falls back to the rebuild; an analyzer-only change is not detectable and is not re-applied. This matches Lucene's semantics.A merge falls back to the old rebuild — same output, old cost — when the replay cannot be proved equivalent: an input whose side-cars will not open or enumerate, an input carrying documents but no FTS data, or a field whose stored position setting disagrees with what the merge would write.
Index::merge_reanalysed_document_count()reports how many documents took that fallback,XERJ_PROF=1prints a per-batchmerge-batchphase breakdown, andXERJ_MERGE_FTS_REANALYZE=1forces every merge back onto the old path.- The
--embed-mode neural: a long passage no longer pads a whole batch up to its own length (#366). The built-in Candle encoder tokenized every call withPaddingStrategy:: BatchLongestand ran one rectangular forward pass over it, so a window that held one 512-character chunk beside sixty short lines cost61 × longinstead oflong + 60 × short— the model did the work, the attention tensors were allocated, and the padding contributed nothing. The encoder now sorts a call's passages by token length and cuts batches at 64 rows or a 4096rows × padded_sequence_lengthbudget (xerj-ai/src/microbatch.rs, shared with the experimental ONNX backend), pads each batch to its own longest row, and restores input order. This affects the opt-in neural backend only; the default embedder is unchanged lexical feature-hashing.Measured on this repo's 32-core x86_64 box with MiniLM-L6 (median of three interleaved runs each, box shared with other builds — the spread is wide, so the medians are quoted, not the best runs):
shape before after one 210 KB source file (504 chunks, one call) 46.2–57.8 s, 8.97 GB peak RSS 19.9 s, 0.42 GB peak RSS 45 files of apache/lucene, chunked as ingest chunks them (1214 passages)8.9 passages/s 12.1 passages/s synthetic window of 4 long chunks + 60 short lines 40.2 passages/s 145.7 passages/s uniform short documents, window of 64 171.0 passages/s 175.6 passages/s one passage per call 68.9 passages/s 72.8 passages/s Padding waste (
rows × padded_sequence_length÷ real tokens) on the Lucene corpus fell from 1.50 to 1.13, and on the synthetic mixed window from 2.66 to 1.00.The first row is the other half of #366 — "each of those two files held the run for 30+ seconds" on 200 KB C++ headers.
semantic_embedding_window_endalways admits a whole document even past the scheduling window, so every chunk of a large field reached the model as one forward pass whose attention tensors scaled with the document: 8.97 GB of resident memory for a single 210 KB file, which is an OOM on a laptop rather than a slowdown. The row cap bounds it.What this does not fix, stated plainly: #366 reported ~15 documents/s on short uniform strings and read it as per-document inference. That reading does not hold — HTTP
_bulkalready batches (bulk.rscollects the whole request,index.rswindows 64 passages across documents), and the last two rows above show that shape is unchanged by this work. On uniform short documents ~15 docs/s is the cost of CPU transformer inference, and moving it needs quantization, a GPU provider or a smaller model. What was genuinely broken was mixed-length work, which is what pointingautoindexat a real folder produces. Binary-protocol_bulkembedded one document per inference call when this was written (xerj-api/src/binary_protocol.rs); that half is the entry below.Binary-protocol
_bulkshares one embedding pass across the batch instead of one inference call per document (#903, split out of #366).handle_bulkloopedindex_documentper document, andindex_documentembeds its ownsemantic_textfields — so a 64-document bulk cost 64 inference calls of one passage each. It now collects the batch and hands it toIndex::index_documents_batched, which routes it through the sameapply_semantic_embeddings_batchentry the HTTP_bulkpath uses: passages are windowed atembedding.onnx_scheduling_window(default 64) across documents, publication stays per document, and one document's failure still fails only that document. There is no second batching rule.Measured end to end over a real TCP connection through the handler with the harness added here (
cargo run --release -p xerj-api --features neural --example binary_bulk_throughput -- <documents> <per-bulk>), on this repo's 32-core x86_64 box with--embed-mode neural(built-in Candle backend,sentence-transformers/all-MiniLM-L6-v2). Corpus: 1 000 lines ofapache/lucene'slucene/CHANGES.txt, 40–400 bytes each so every document is exactly one passage. Medians of interleaved runs; the box was shared with other builds, so the spread is wide and the medians are quoted, not the best runs:documents per bulk before after 16 (n=3) 60.7 docs/s 82.6 docs/s 64 (n=5) 64.0 docs/s 102.7 docs/s 256 (n=3) 84.3 docs/s 124.9 docs/s Bounds, stated plainly. The win is a short-document win. A document long enough to fill a scheduling window on its own already got its own inference call:
semantic_embedding_window_endadmits a whole document even past the window (index.rs), so before and after are the same call for it. On the default lexical feature-hashing embedder there is no inference to batch and no change was measured — 872 → 865 docs/s (medians of n=7 interleaved, ranges fully overlapping). And the module this fixes is a library entry point:serve_binary_protocolis not bound by any listener inxerj-server(xerj-engine/src/index_guard.rssays so, and nothing in the tree calls it), so no shipped-server ingest path changes speed with this release.
Fixed
knnbeside aqueryis answered from the inverted index again — and its page no longer drops the documents matched by both halves (#892). This closes the regression rc.72 shipped knowingly: the #825 fix pre-executes the vector leg and pins its top-kinto the query tree asconstant_score{ids}clauses,_idis not in the FTS term dictionary, so the projection declined the whole tree and every segment fell to a stored-document scan.That route was not only slow. The scan admits documents in stored-layout order and stops materialising once it reaches the page cap — no score comparison — so on any index with more lexical matches than that cap, the documents reached by BOTH halves, the ones ES scores
query_score + knn_scoreand ranks first, never entered the page.hits.totalwas right; the page was not. Live-verified on 100 000 documents: for amatch+knnk=10 request whose vector top-10 held exactly two lexical matches, rc.72's top 20 was twenty lexical-only documents tied at 1.693 and neither both-halves document appeared at all. They now rank 1 and 2, at 3.153 and 3.144.The projection now lifts the pinned disjunct to an FTS leaf carrying
(doc_id, score)pairs, resolved per segment through the same_id→ stored-position index thatidsqueries and_mgetalready use. The lexical half keeps the inverted index and exact BM25, the vector half keeps its own scores, and theboolsums them exactly as before.Measured on that corpus (
text+ 8-dimdense_vector, ~10 % lexical selectivity, bulk-loaded once and never updated — which is the only state in which this route engages, see the scope note below — closed-loop, fresh query vector per request so the query cache cannot answer twice, medians of 2 rounds × 7 requests, on a shared build machine — the two control rows bound the noise):request rc.72 with this fix queryalone (control)0.45 ms 0.52 ms knnalone, k=10 (control)1.99 ms 2.12 ms query+knnk=10265.9 ms 6.4 ms query+knnk=100326.9 ms 8.4 ms query+knnk=1000505.0 ms 27.5 ms query+knn+aggsk=10784.1 ms 534.4 ms hits.total, the aggregation buckets and the field-sorted page came back identical across the two arms — one request shape per column, compared as JSON — and so didquery-alone andknn-alone. The agg row keeps most of its cost because atermsagg over this shape still materialises the whole corpus — a different path, untouched here.Two things to know:
_scorevalues change for this shape. The lexical half is now scored by exact BM25 — the same number the identical query returns without aknnbeside it — instead of the stored scan's heuristic. Ordering follows the #825 contract as before; absolute values move, so amin_scoretuned against rc.72's hybrid needs revisiting.- The indexed route applies only to an index that has never taken an
overwrite or a delete — that is the scope of every number above, and it
is a one-way door, not a transient window. The gate is the one the
idspre-filter already applies: the per-segment_idindex keeps one position per id, so a segment physically holding a document twice (an overwrite flushed alongside its predecessor) could hand back the superseded copy. It readsVersionMap::ghost_events, which is monotonic by design — it counts overwrite and delete events for the life of the open index and is never decremented, not even by the merge that purges the superseded copies. OnePUTover an existing_id, or oneDELETE, therefore turns this route off for that index until it is re-opened, and it stays off across that re-open while superseded copies remain on disk. An index that takes updates or deletes keeps rc.72's stored scan forknnbesidequery— still correct, and no slower than it was, but with none of the speedups above. Append-only indexes — bulk load, reindex-and-swap, log and event corpora — get them. Widening the route to ghost-bearing indexes is separate work: it has to prove the #825 union/sum contract with tombstones present, and is not attempted here.
A
date_nanosfield no longer guesses its epoch scale value by value, and an index written before that flag existed recovers its scale instead of guessing (#889, follow-up to #790). #790 made a date field's epoch scale a property of the MAPPING and pinned a declareddateto milliseconds, but deliberately leftdate_nanoson the old per-VALUE guess (">= 4 fractional-second digits means nanoseconds"). The defect #790 describes therefore survived insidedate_nanos: a column holding both2020-06-06T06:06:06.123456Zand2025-01-01T00:00:00Zgave the first a nanosecond key (1.59e18) and the second a millisecond one (1.74e12), six orders of magnitude apart, so 2025 sorted before 2020 and a range bound could not reach both.Pinning
date_nanosto nanoseconds is not safe on its own — it moves adate_nanosfield six orders of magnitude away from adatefield in a cross-index sort — so the prerequisite landed with it:- ES
numeric_typeon a field sort (numeric_type: date/date_nanos). It forces the epoch scale every participating index produces sort values on, which is the only way a sort over a field mappeddatehere anddate_nanosthere compares chronologically. Thesearch_aftercursor is normalised on the same forced scale, so paging works in that one space.long/doubleare parsed and ignored (xerj already compares numeric sort keys asf64), and unlike ES, xerj does not rejectnumeric_typeon a non-numeric field. Withoutnumeric_typethe raw longs are compared, exactly as ES does and astests/es-compat-yaml/yaml/search/240_date_nanos.ymlasserts — that default is unchanged and is pinned by a test. date_nanosis now uniformly nanosecond-scaled, including day-first slash dates (dd/MM/yyyy HH:mm:ss.SSS), which previously had no nanosecond branch at all and would have re-introduced the mixed-scale split by a different route.- A NUMERIC date bound is epoch-MILLISECONDS and is converted at the
bound, not at the value. That is ES's own rule — a
date_nanosfield's default format isstrict_date_optional_time_nanos||epoch_millis, so a bare number is parsed as milliseconds and only then moved to the field's resolution. Moving adate_nanoscolumn to nanosecond keys therefore had to move every numeric bound with it: the segment sort-shadow's bounds (shadow_range_bounds, which serves both the range doc-values pre-filter and the_countshortcut) took a numeric bound verbatim, six orders of magnitude below the keys it bisects. Two shapes reach it — the #788term-on-a-date rewrite, which lowers atermto the degeneraterange [epoch_ms, epoch_ms], and a user-written numeric bound — so on a flusheddate_nanosindex atermand a numeric range both matched nothing while the same query answered correctly from the memtable. Bound and key are now on one scale in both directions, and the flush-invariance of both shapes is pinned by a test that asserts pre- AND post-flush. - One format ladder for both scales.
date_string_to_epochhad a separate nanosecond branch that stopped after RFC3339 and%Y-%m-%dT%H:%M:%S%.f; the%Y-%m-%ddate-only fallback existed only in the millisecond branch. A DATE-ONLY value ("2021-01-01") in adate_nanosfield therefore normalised to nothing at all and was emitted as a raw JSON STRING among i64 epoch keys, where the sort comparator's string arm puts"below every digit and the document sorted FIRST ascending whatever its instant — and the segment's sort/range shadow refused to build on that term. Becausenumeric_typeforces the nanosecond scale on every participating index, an ordinarydatefield holding a date-only value hit the same hole under a cross-index sort. The ladder is now walked once and the accepted instant rendered on the requested scale at the end, so a format either parses on both scales or on neither. Thesearch_aftercursor normaliser, which carried a hand-copied second copy of the same ladder whose fallbacks returned milliseconds even on a nanosecond field, now calls the one function. - Legacy indices are back-filled at open.
date_precisionis recorded when a field is mapped, so an index created by an older build comes back with no precision at all and every date field falls back to the per-value guess.schema.jsoncannot telldatefromdate_nanos— but the raw ES mapping blob beside it (es_mapping.json) still carries the original type string, andIndex::opennow recovers the flag from there. Both directions are covered: a legacydate_nanoscomes back nanosecond-scaled (a blanket default would have silently downgraded it to millisecond resolution) and a legacydatecomes back millisecond-scaled, i.e. it inherits the #790 fix without a reindex. Nothing on disk is rewritten and no data is reindexed: the epoch keys were always derived at read time from the stored value, so correcting the scale is enough. GET _mappingreports adate_nanosfield asdate_nanoswhere the mapping is derived from the schema (it already round-tripped correctly when the raw blob was present). The GET → PUT round trip every reindex helper performs no longer silently downgrades the field.
Not fixed, stated plainly:
- An index whose
schema.jsonpredates the flag AND which has noes_mapping.json(an index that never went through the ES-compat mapping path) still has no record ofdatevsdate_nanos, so its date fields keep the legacy per-value guess. Inventing a default there is the silent downgrade this change exists to avoid. - The back-fill reads
{"properties": …}and{"mappings": {"properties": …}}, which is what the mapping API persists. An ES-6-style type wrapper ({"_doc": {"properties": …}}) is skipped, and that field then keeps the legacy per-value guess — the same safe fallback, not a new failure mode. - Only STRING-shaped date values move onto the field's scale. A numeric
source value (
{"ts": 1591423566123}) in adate_nanosfield stays a raw millisecond number in its doc-values column; ES would convert it. A segment holding both shapes ships no column at all and falls to the correct full scan, but two segments that each hold one shape still mix scales in a sort. numeric_typevalues other thandate/date_nanos/long/doubleare lower-cased and then ignored where ES answers 400, andnumeric_typeon a non-numeric field is accepted where ES rejects it.- Sort-value FORMATTING still infers milliseconds vs nanoseconds from the
magnitude of the epoch number rather than from the resolved scale
(
format_sort_value, threshold2 * 10^15). Unchanged by this work: it reads correctly for every nanosecond instant from 1970-01-24 on (2e15 ns = 2e6 s = 23.1 days) and every millisecond instant below 2e15 ms (~63,000 years past the epoch).
- ES
autoindex: a run no longer deletes a sibling corpus's duplicate-alias catalog documents (#905). Data loss, and the second of the two holes the #890 review found.autoindex-catalogis one index shared by every corpus on the node, but the duplicate-alias sweep issued at the end of a run was{"bool": {"filter": [{"term": {"status": "duplicate"}}, {"terms": {"path": [...]}}]}}with no corpus scope at all — andpathis a logical path inside a corpus, not a node-unique key. Two checkouts on one node that each carryLICENSEandvendor/LICENSEis the ordinary shape of that, and a delete is not recoverable. Unlike #890 this needed no legacy mapping: it was unscoped on a currentkeywordcatalog too.The query still selects the candidates — it is what keeps the candidate set small — but the scope is now applied per hit and the deletes go by
_id, the pattern #890'sdelete_catalog_docs_scopedintroduced (both sweeps share one page walk now, so there is one copy of its paging rules rather than two). A simple scope filter would have traded one defect for another, which is why this was split out of #896 rather than folded into it: the sweep exists because alias ids changed as identity evolved, and its oldest targets (v1.0.0-rc.10..rc.56, before #416 put the corpus prefix into the_id) carry no corpus field and no prefix in their id, so a server-sidecorpus_scope/prefixterm would simply stop matching them. Attribution is therefore layered:corpus_scope(#755), elseprefix(#737) read from raw_source, else the prefix in the_id(#416). Below that the document names no corpus at all, and rather than guess in either direction the sweep deletes such a hit only when its_idis one this corpus itself would have written under the pre-#416 scheme — the reconstruct-the-id move #739 already makes — and otherwise leaves it in place and says so, by path and id, in a new non-fatal note. The sweep stays non-fatal (#345) throughout.Scope, stated rather than implied: this is the legacy (graph) run path only. A
--no-graphrun returns from the generated path long before this sweep and retires stale catalog documents by their prefix-scoped_id(generation_catalog::managed_non_run_ids), which was never unscoped.--embed-mode neural: re-indexing an unchanged corpus no longer fails, and the identity can now tell one model from another (#487, #367). The neural arm of the embedding execution identity hashed the configured model name (neural-unpinned.v1;configured-model=…) and reportedresumable: falseunconditionally. Two consequences, both reproduced on the released rc.71 binary before the change:- every neural node on earth emitted the same
identity_sha256(6e51e5ce3e46…), so the check that exists to catch a swapped model could not see one; and - the five
resumablegates inxerj-autoindexall fired.xerj autoindexagainst a neural node was one-shot: the graph path completed generation 1 (exit=0) and then refused the identical command over an unchanged corpus (exit=1, "server embedding backend cannot safely resume semantic indexing"), and the--no-graphpath refused even the first run at the generation cutover. That is the whole reference-coding loop — index a peer repo, pick up its commits later — so it mattered more than a bookkeeping error looks.
The identity is now the streamed sha256 of
model.safetensors,tokenizer.jsonandconfig.jsonplus the model's ownhidden_size, pooling and truncation limit, resolved by exactly the pathNeuralEmbedder::loadtakes (xerj-ai/src/neural.rs) — the same thing the ONNX arm already did.resumableis then honestlytrue,dimensionsis reported instead of omitted, and a genuinely swapped model is refused where before nothing was. Assets that cannot be resolved still fail closed, but the reason names the file rather than blaming a configuration change that never happened.Measured on this box, 20-file corpus,
--embed-mode neural(MiniLM from the local HuggingFace cache):run rc.71 this change graph path, run 1 exit=0exit=0graph path, run 2 — identical command, unchanged corpus exit=1exit=0graph path, run 3 — one file edited not reachable exit=0--no-graph, run 1exit=1exit=0--no-graph, run 2 — unchanged corpusexit=1exit=0The pre-fix identity was already deterministic — the same digest on every call and every host — so this was never a hashing-stability bug; the flag was simply hardcoded. Batch shape (
MAX_BATCH_ROWS,PADDED_TOKEN_BUDGET), thread count, device and the resolved file paths are deliberately not in the digest: they change how rows are grouped, not what any row computes, so folding them in would have reintroduced the false "identity changed" one layer down.On-disk migration:
embedding_execution.jsonmoves to version 3, so the first reopen of a populated neural index after upgrading is grandfathered and re-baselined instead of refused — the same handling #533 gave the proxy endpoint change.proxyis unchanged and still reportsresumable: false: a remote provider attests nothing about its model's bytes, and probing it with fixed inputs would hash floats that no provider guarantees to be reproducible — which is the bug in this issue, one layer down.- every neural node on earth emitted the same
A frontmatter block's body is judged as YAML by whether it parses, not only by how its lines look (#587). The
---lookahead added for #551 decided "is the text after the closing---YAML?" from line shape alone — a body was YAML if most of its lines read askey:or- item. Two shapes fell through it. A genuine multi-document YAML stream whose later document is dominated by block scalars, comments and indented continuation lines has almost nokey:-shaped lines, so the whole stream was re-classified as a text family and lost its per-document key structure; and the lookahead required a body of at least two non-blank lines, so a document whose body was a single line still went to the YAML extractor, which keeps the frontmatter mapping and drops that line.classify_textnow runs the body throughclassify_structuredfirst (JSON is valid YAML, so this ordering matters), then asks whether the body actually parses as a YAML stream of mappings or sequences — markdown-with-frontmatter does not parse as one, a real stream does — and only falls to the prose families when it does not (xerj-autoindex/src/sniff.rs).Measured over the 36,936 files in this box's reference corpora and local checkout whose first non-blank line is
---(279 distinct by content changed): 69 files moved intoYaml— every one of them a.yml/.yamlfile, including Elasticsearch'syamlRestTestspecs and Cilium manifests that had been demoted to line-records — and 205 moved out ofYamlinto prose plus 5 intoJson, every one of them a.md,.mdxor.snapfile whose one-line body the YAML extractor was discarding. No markdown file moved intoYaml. (Extensions are never consulted by the sniffer, so they are an independent check on the direction of each change.)A legitimate authenticated request whose body arrived in small HTTP/2 DATA frames could be cut off with
GOAWAY ENHANCE_YOUR_CALM(#485). The rc.18h2bump for RUSTSEC-2026-0258 — described at length in that release's notes below — added a per-connection budget for small DATA frames: a frame under 256 bytes consumes256 - payload_len, a frame of 256 or more replenishes, and the refund for a small frame lands only when the application reads it. Inh20.4.16 that budget was a fixed 25,600 bytes however large a window the server had advertised, while hyper advertises a 1 MiB HTTP/2 connection window — so a well-behaved client could have 40× the budget in flight before the handler was ever scheduled. Any client that chunks small (a streaming JSON encoder, a per-document flush, a gRPC stream of small messages) lost its whole connection on a request that would otherwise have returned 200.h2is now floored at 0.4.19 in[workspace.dependencies](engine/Cargo.toml), where the budget is half the configured connection window with 25,600 bytes as the minimum — 512 KiB against hyper's 1 MiB window — and the empty-DATA-frame flood the advisory was actually filed about is capped by its own counter (100 frames per connection) rather than out of the byte budget. The mitigation is not disabled or weakened here, and no XERJ-side window or budget knob was changed; the floor sits in the workspace manifest socargo updatecannot resolve back to a fixed-budget release.Measured on this box, same harness on both sides (
axum::serve, hyper 1.9.0, a handler that reads the whole body, one DATA frame per chunk queued back-to-back with no pause), bisecting for the largest body that completes. The boundary is not a constant — it is a race between arriving charges and the refunds a reading handler returns — so these are the min–max of three bisections each, not thresholds to design against:DATA frame size before (h2 0.4.16) after (h2 0.4.19) 64 B 9.3–16.5 KB (145–258 frames) 262 KB–6.4 MB (4,095–99,793 frames) 100 B 18.9–27.2 KB (189–272 frames) 410 KB–4.4 MB (4,095–43,561 frames) 128 B 29.3–36.6 KB (229–286 frames) 2.1–4.7 MB (16,127–36,628 frames) 200 B 141–419 KB (703–2,094 frames) no cut-off up to 52 MB 256 B no cut-off up to 8 MB no cut-off (never charges) Where the safe line now sits, exactly: with a budget of half the window, a frame of 171 bytes or more cannot exhaust it even if the application never reads a byte — at most
window / sizeframes can be in flight, each charging256 - size, and(256 - size) ≤ size / 2from 171 up. That is why the 200 B row has no ceiling at all. Below 171 bytes there is still a ceiling and it still depends on how fast the handler drains; it is now one to two orders of magnitude further out. A frame of 256 bytes or more never charges, before or after. The rc.18 note advising clients to chunk at 256 bytes or above is no longer needed for ordinary bodies.Deliberately abusive traffic is still refused, and that is asserted rather than assumed: a flood of empty DATA frames still ends in
GOAWAY ENHANCE_YOUR_CALM(upstream's 100-frame cap, unchanged), and a flood of 1-byte frames is still cut off — on this harness the client saw the connection go at roughly frame 4,600 instead of roughly frame 100.What was measured where. The table above comes from a standalone harness pinned to this workspace's own axum/hyper/h2 versions, not from the
xerjbinary. The binary itself is covered byengine/crates/xerj-server/tests/http2_small_data_frames.rs, which spawns the real server with authentication on and drives a 40 KB authenticated_bulkat 100 bytes per DATA frame (401 frames) through the ES-compat listener over prior-knowledge h2c, asserting a 200 with every document indexed. Against the unfixed dependency that same test is answeredGOAWAY ENHANCE_YOUR_CALM (too_many_data_frames)mid-body, which is how it was checked. A second test holds the empty-frame guard in place. The TLS listener and the gRPC listener sit on the sameh2and take the same fix, but neither is exercised by these tests.xerj autoindex --no-graphno longer aborts withAccess denied (os error 5)on Windows (#482). The durable-snapshot path fsynced the directories it had just published withFile::open(dir)?.sync_all()?. That is a Unix-only idiom: on WindowsFile::openon a directory returnsERROR_ACCESS_DENIED(os error 5) on every call, becausestdcannot passFILE_FLAG_BACKUP_SEMANTICS. So the first such call — sealing the source snapshot, immediately after the journal was written and before a single document was indexed — killed the run with a context-free, locale-dependent io error (Acceso denegado. (os error 5)in the report). A second, unconditional open of the same kind in snapshot GC then failed every later run over that state directory, whether or not--no-graphwas passed again.xerj-autoindexnow routes directory durability throughxerj_common::fsio::fsync_dir, which has carried the documented#[cfg(windows)]shim since the same mistake stopped the server creating any index at boot; the surrounding filesystem calls also carry context now, so a future failure names the operation instead of only its errno. This costs Windows no durability it had: the code it replaces flushed nothing there — it returnedErrand ended the run. Windows keeps the engine-wide posture that directory-namespace changes carry no durability claim; file contents inside a snapshot are still fsynced. The windows-latestautoindex-fd-smokeCI job now runs the reported--no-graphflow twice over one state directory, which is what would have caught this: the job existed, but only ever ran the default graph path, which never touchessync-snapshots/.A code symbol's retrievable unit is the whole declaration, not the single physical line the name sits on (#500). Promoting each declaration to its own document (#579) closed the constant-and-field half of that issue. The stored
codewas still literallylines[name_row], so anything that did not fit on that one line came back wrong: a wrapped signature returnedpublic void configure(; a declaration whose modifiers sit above the name (static unsigned long\nhash_bytes(…),#[inline]\npub fn scan(…)) returned the wrong line entirely; and a declaration sharing its line with the body (void m(){ int local = 1; }) returned implementation nobody asked for.codeis now sliced from the declaration's first token — leading attributes included — to where its body begins, so the unit is a complete signature rather than a fragment of one.Measured with the
decl_benchharness added here (XERJ_DECL_BENCH=<repo> cargo test --release -p xerj-autoindex --lib decl_bench -- --ignored --nocapture), which computes the old unit and the new one side by side in one pass over the same parse and the same symbol set:corpus slice ending mid-signature, was → now median bytes, was → now the symbol's whole line..end_linespanLucene — 5 627 Java files, 79 102 declarations 4 137 (5.2 %) → 18 (0.02 %) 49 → 52 (mean 50.5 → 60.7) median 229, mean 1 046 valkey + memcached — 981 C files, 28 347 1 444 (5.1 %) → 54 (0.2 %) 48 → 49 (mean 51.6 → 58.8) median 123, mean 454 tantivy — 434 Rust files, 8 970 851 (9.5 %) → 27 (0.3 %) 41 → 50 (mean 44.2 → 60.9) median 218, mean 721 cilium + dpdk + vpp-agent + xdp-tools — 26 357 Go/C files, 914 561 87 267 (9.5 %) → 1 678 (0.2 %) 49 → 53 (mean 54.6 → 67.4) median 102, mean 387 So the unit costs 7–17 bytes more on average and is a complete declaration instead of a fragment — still 1.9–4.4× smaller than that symbol's own span at the median and 5.7–17× smaller at the mean. 39.6 % of Lucene declarations wrap past one line (the single-line slice could only ever return a fragment of those) and 51.5 % have a name line that does not stop where the declaration does.
Extraction is not free: medians of five interleaved runs over the same 6 106-file Lucene tree, one box,
extract_bench— 1 600 files/s before this change, 1 455 after. About 9 %, for an ancestor climb bounded at 12 levels and a pruned body search bounded at 4 096 nodes.Two honest limits. The cap is unchanged at 400 characters and a declaration that overruns it still falls back to the start line, so the WORST case per symbol is what it always was — but an individual document can grow, since a 45-character name line may now store a 400-character signature. And "stops where the body begins" needs the grammar to say where that is: for a Haskell equation nothing marks it, so the slice is the whole equation. Corpus-wide that is 1.2–5.9 % of symbols whose slice covers their entire span, which
decl_benchnow reports as its own column rather than leaving to assertion.The file document's
symbolssidecar gainsend_line, so a caller that wants the implementation can address exactlyline..end_lineinstead of guessing "up to where the next symbol starts" — the guess that returns a whole class for a class and the rest of the file for the last symbol.Upgrading does not rewrite what is already indexed:
xerj syncreconciles by content digest, so an unchanged file produces no operation and its symbol documents keep the old single-linecodeand carry noend_line. An index upgraded to this release is a mixture until the files change or it is rebuilt from scratch.The exclusion sweep can no longer delete a sibling corpus's catalog documents on a catalog an older build left
text-mapped (#890).autoindex-catalogis shared by every corpus on the node, so when a widened ignore rule excludes a file, the sweep that removes its already-published documents is scoped to the corpus'sprefix. On any catalog a v1.0.0-rc.15..rc.67 build wrote to before the mapping declared that field — which includes catalogs first created by an earlier release — it is dynamically inferredtext, and atermquery against an analyzed field matches the field's tokens:prefix: "ax"also reached documents whose prefix isax-2(tokens[ax, 2]). Conjoined with thepath/file_keya byte-identical file (a LICENSE, a lockfile) shares between corpora, the sweep deleted the sibling corpus's live catalog documents — the cross-corpus over-delete the scope was added in #737 to prevent, and a delete is not recoverable. Present since v1.0.0-rc.68.Both scoped catalog deletes now go through an exact scoped delete: the same query still selects candidates server-side, but every hit's raw scope values are re-checked against
_sourcebefore it is deleted by_id. That is independent of the catalog's mapping, so it holds on a keyword catalog and a legacy text one alike, and it covers thepath/file_keyconjunct as well as the corpus scope. Coverage of the excluding corpus's own documents is unchanged, including alias documents the frozen plan does not name, which only this pass can reach. The candidate walk is bounded at ten pages of 1,000, and it ends on an exacthits.totalas well as on a short page, so the boundary case of exactly 10,000 candidates completes rather than refusing. Past that bound it fails closed — it refuses with a_reindexremedy rather than reporting a removal it only partly made.Not fixed here: on a legacy catalog an analyzed field can also match fewer documents than it should, so a scoped sweep can still miss a document an older build wrote. That is the under-match #755 answers with the keyword
corpus_scopefield, and the run already warns with the_reindexthat retires the legacy mapping.CI's default-parallelism test step no longer races itself for a port (#751, the second half). With the deadlock above fixed, the same step then failed on a real port race that the hang had been masking:
xerj-server's cleartext-bind tests allocated a port by binding127.0.0.1:0, reading the number, releasing it, and handing it to a child process that binds it later. The kernel is free to hand that same ephemeral port to the nextbind(":0"), and several tests in that file keep a real server alive for seconds, so at default parallelism they overlapped —gRPC: bind [::1]:46843 ... Address already in use. Ports now come from a band below every platform's ephemeral range, strided per process and probe-bound on both127.0.0.1and::1(a port free on one family can be taken on the other, which is exactly how it failed). Test-only change.A search that aggregates over a cold segment could deadlock forever (#751). On a multi-thread runtime
Index::searchruns its whole body insideblock_in_place(|| Handle::current().block_on(..)): that hands the worker's core to a tokio blocking-pool thread — which then runs the scheduler loop for the life of the runtime — and parks the calling thread, itself a blocking-pool thread, for the entire search. An aggregation with no columnar fast path assembles the full corpus from inside that park, and the cold segment hydration it needs (stored_values_for_async) queued its decode withspawn_blocking— onto the very pool the waiter was holding a thread of. tokio grows that pool only when it observes zero idle threads at push time, so the core hand-off and the decode submission could both see the same idle thread, both merely notify, and the one thread that woke took the core and never returned. The decode was then queued behind threads that were all permanently running worker cores: the search waited for a task that could never be scheduled. The decode now runs on the engine's own rayon maintenance pool, whose threads are never consumed by tokio, so the wait cannot be circular. This is what made CI's default-parallelism workspace-test step hang on hosted runners and leavemainwith no verdict for hours. Measured on a 4-cpu affinity mask with the reported test binary: 12 hangs in 60 runs before, 0 in 200 after (and 0 in 100 on a 2-cpu mask). One consequence worth knowing: a cold hydration a search is waiting on now runs at the maintenance pool'snice(10)instead of anice(0)blocking thread, which is only observable when every core is already saturated.CI: every workflow job is bounded, so one hang can no longer cost
mainan afternoon of CI (#770). 22 of this repo's 23 GitHub Actions jobs declared notimeout-minutesand so ran against GitHub's 360-minute default. Three workflows serialise on a concurrency group that never auto-cancels (ci-CI-refs/heads/main,pages-deploy,release-metrics), so a single hung job holds that slot for six hours while every later run queues behind it. That is measured, not hypothetical. On 2026-08-25, before #767 cappedbuild-test, the #751 hang ran that job into the 360-minute default and was killed by it (run 32796557309, 01:21:28 → 07:21:44 — every other job in the same run finished within 18 minutes). A second hang the same morning held the slot from 11:28 to 17:03, and the seven main pushes that landed behind it were all discarded with zero jobs run; a normal ~37-minute run would have let at least four of them through, since the gaps between them were 48, 62, 38 and 150 minutes. Every job now carries a cap sized in tiers from the measured duration of every job execution across the last ~245 CI runs (3 873 executions) and 27 release runs: 15 min for jobs that install no toolchain (measured max 2.3 min), 30 min for jobs that install a toolchain but compile no workspace member (max 3.8 min), 60 min for jobs that compile workspace crates (max 48.3 min,Build + Test), and 75 min for the two jobs with the heaviest tails —autoindex-fd-smoke, whosewindows-latestleg runs 18.8 min at the median but 49.6 at the maximum, andrelease.yml's cross-compile matrix (max 38.3 min,aarch64-pc-windows-msvc). Worst-case slot starvation drops from 360 minutes to 75 (CI onmain), 20 (pages-deploy) and 75 (a release tag). A new.github/scripts/workflow-timeout-guard.pykeeps it true for jobs added later. This bounds the blast radius of the next hang; it is not itself a fix for any hang, and #899 remains what fixed #751's.
[1.0.0-rc.72] - 2026-08-31
The idle-cost release. A 2026-08-29 measurement session found XERJ was not
quiet at rest: on a node holding 464 small indices — the shape xerj autoindex
produces from a reference corpus — the process burned 15–27% of a core and 115
wakeups per second with zero requests, and a 154 MB corpus spent 40+ minutes at
~110% CPU merging after ingest "finished". Customers reported it as fans
spinning on an idle laptop. Three of the four mechanisms are fixed here; the
fourth (#876) has a published
design and is not yet implemented. The budget every fix is held to, and the
remaining gap, are tracked in the open meta-issue
#874.
Alongside it, six Elasticsearch-compatibility and autoindex correctness fixes, each shipped with a test proven to fail on the unfixed code.
Performance
An idle index no longer costs a timer, a thread or a wakeup (#871). Every index ran its own
tokiotask that woke every 5 seconds forever and evaluated merge policy whether or not anything had been written — 464 indices meant 93 wakeups per second and two runtime workers each burning ~10% of a core, on a node serving nothing. Merge scheduling is now event-driven, the same shape the WAL fsync scheduler adopted in rc.16 for the identical defect (#334): a segments-changed hook fires on every snapshot swap that alters the segment set (flush publication, merge application, tombstone-only segment, orphan recovery) and a debounced request arms at most one check.XERJ_MERGE_INTERVAL_SECSkeeps its meaning as the debounce delay. Measured at the unit level with an evaluation counter: 768 evaluations across a simulated idle hour become 48.The resource sampler stopped walking every index × every shard lock (#872). The memory breaker's sampler summed memtable bytes every 100 ms by taking a read lock on all 16 shards of every index — 74,240 lock acquisitions per second at 464 indices, in one thread, to compute a number that is almost always zero. The aggregate is now maintained incrementally by the write paths that already compute the byte delta, and the sampler is a single relaxed load. The breaker's starvation-immunity property is unchanged: it still runs on a dedicated OS thread, it just no longer does O(indices × shards) work to do it.
A multi-index search runs its per-index searches concurrently (#875). The fan-out awaited each index before starting the next, so an
ax-*query cost the SUM of its per-index latencies instead of the maximum — the shape every reference-coding retrieval uses. Hit ordering, aggregation merge order and the global search permit that bounds concurrency are all unchanged.Two behaviour changes worth knowing:
queries_by_indexnow increments for every resolvable index even when a later index errors the request (those searches genuinely run now), and per-index deadlines all start at request time rather than sequentially, so a search queued behind the permit pool can returntimed_out: truewith partial results where the serial loop returned complete results slowly.
Fixed
A field holding both numbers and strings in one segment no longer loses the numeric documents from
term/terms. The doc-values builder collects a segment's values into a numeric map (numbers and booleans) and a keyword map (strings), then writes both into one column set — so a type-mixed field had its numeric column silently overwritten by the keyword one, leaving every number/boolean document filed as null in a column the query prefilter treats as exact. Those documents were then dropped from the result with no error. Such a field now ships no doc-values column at all, exactly as a multi-valued field already does, and every consumer falls back to the stored-source scan: correct, at scan cost rather than column speed. Reindex onto a single type to get the column back. Whether it bit you depended on core count — a many-core host scatters a flush across shards and each segment stays single-typed, so this reproduced only where documents of both kinds landed in one segment.A field mapped
integer,long,floatorbooleannow enforces that type on write (#781). The declared type was enforced for nothing:1.9into anintegerfield was stored as1.9,9999999999was kept exact,"abc"was stored as a string,"yes"went into aboolean, and — per the issue's follow-up — an entire nested object{"bad":"x"}was indexed into aninteger. Every one of those answered201 created.The consequence was wrong hits, not just leniency. With
1.9sitting in anintegerfield,range {"i":{"gte":1.5}}matched in XERJ and would not in ES (which indexes the truncated1), whileterm {"i":1}matched in ES and returned nothing here — the same query, over the same document, under the same mapping, giving different answers.Ingest now applies ES 8.x's own rules (
coercedefaults totrue):1.9→1,"5"→5,"1.9"→1,"true"→true; out-of-range for the declared width, an unparseable string, an object, an array element that is none of those, and anything buttrue/falsein abooleanare refused with a 400document_parsing_exception— a per-item 400 under_bulk. An explicit"coerce": falseadditionally refuses a decimal part and a string, and"ignore_malformed": truestill wins: that field's bad values are dropped into_ignoredrather than failing the document.What this changes about stored data, and what it means on upgrade. A coerced value is rewritten in
_source: ES keeps_sourceverbatim and coerces only the indexed value, but XERJ indexes from the stored source, so matching ES's hits costs source fidelity here. A document written{"i": 1.9, "b": "false"}now reads back{"i": 1, "b": false}. An index that spans this upgrade therefore holds both spellings of the same logical value — documents written before hold"false"/"5", documents written after holdfalse/5— and no reindex is performed for you.That mixed index still answers correctly, and closing that gap is part of this change. Because the canonicalisation moves a declared
booleanfrom the string spelling to a real JSON boolean in_source, the query side now runsterm/termsvalues on abooleanfield through the same predicate — ES parses a query value with the sameBooleansits field mapper uses, so"true"andtruename one term. Without it,terms {"b":["true"]}found nothing on a doc-values-only boolean field while the equivalenttermstill matched. The_sourcescan comparator relates the two spellings in both directions (as it already did for a number and its string spelling), for single- and multi-valued fields alike, and atermsaggregation buckets them together — so one query sees the whole index rather than half of it. Reindex if you want a uniform_source; nothing breaks if you do not.Also narrowed deliberately, all documented in
xerj_common::field_coercion: the index-levelindex.mapping.coercesetting is not read (only the field-level parameter),scaled_floatis range-checked but not quantised by itsscaling_factor, an empty string is left alone rather than dropped, and_updateis not re-validated — it merges into a document already checked on write.A top-level
knnbeside aquerynow scores as a sum over the union of both halves, aggregations included (#825). ES 8.x defines this shape as a disjunction: the hit set is the union of the lexical matches and the global top-kneighbours, each document scoresquery_score + knn_score, and aggregations are computed over that union. XERJ folded the request tobool.should[query, knn]but the generic matcher/scorer has noknnarm, so a vector-only document vanished and a lexical match lost its vector score — silently, with_shards.failed: 0. The #458 carve-out covered one narrow shape with RRF rank fusion (itself a scoring divergence from ES) and left every request carryingaggs,sort,collapseorrescoreon the silent drop. The engine now pre-executes theknnleg and pins its top-kinto the tree as per-document constant scores, so one path serves the union with summed scores and every request feature applies to it.Two behaviour changes worth knowing before you upgrade:
{query, knn:{field: <not a vector field>}, aggs}now answers 400 (the field is not answerable by knn), matching ES and the existing #498 rule. It previously answered 200 with lexical-only hits — the wrong answer, quietly.- This shape is slower. A pinned tree contains
idsclauses, which the FTS projection cannot lift, so the request is answered by a stored-document scan instead of the inverted index. Measured on 100 000 documents (text+ 8-dimdense_vector, ~10 % lexical selectivity, closed-loop, fresh query vector per request):query+knnk=10 goes from 2.5 ms (the old RRF route, which returned ES-divergent scores) to ~208 ms; k=1000 to ~385 ms; k=10000 to ~5.1 s. The shapes that previously carried the silent drop — anything withaggs/sort/collapse/rescore— already took the scan and cost about a quarter more (476 ms → 605 ms with a terms agg), but they now return the right documents. Restoring an indexed route is tracked as #892; correctness landed first.
match_phraseslopnow admits transposed terms at Lucene's cost of 2 (#830). The sloppy-phrase walk was in-order only — each next term matched strictly after the previous one — so a reordered pair never matched at ANY slop:{"match_phrase":{"t":{"query":"quick brown","slop":2}}}returned zero hits on a document readingbrown quick, where Lucene/ES match it at distance 2 (SloppyPhraseMatcher's own javadoc example). The evaluator is now the Lucene move-distance semantics — pick one document position per phrase term; the distance is the span of the positions after subtracting each term's query offset — implemented once (xerj_fts::search::phrase_positions_match) and called by both the segment positional clause and the engine's memtable/stored-scan walk (phrase_walk), so slop is evaluated identically on either side of a flush.match_phrase_prefixandmulti_matchphrase/phrase_prefix go through the same evaluator and gain the same behavior. Documents that matched before still match — an in-order pick has strictly increasing positions, so its span telescopes to exactly the old summed-gaps value — and that is checked rather than assumed, by an exhaustive test over every document of length <= 5 and every phrase of length <= 3 from a 3-symbol alphabet at slop 0..3, comparing against both the old walk and a brute-force reference. A repeated phrase term still needs as many DISTINCT document positions as it has slots ("a a"does not match a doc holding onea).A small index reaches a segment instead of living in memory forever (#873).
needs_flushwas threshold-only, so a dataset below the size thresholds never flushed: its documents stayed memtable-resident for the life of the process, pinned their WAL generations on disk, and replayed on every start. Measured on a real corpus: a 100,001-document dataset with zero segments 30+ minutes after ingest. Newstorage.flush_idle_secs(default 300 s,0disables) flushes a non-empty memtable whose contents have not changed for that long, detected by a probe on the existing periodic flusher so the write path pays nothing. This matches Elasticsearch'sindices.memory.shard_inactive_timedefault.match_phraseslopadmits transposed terms at Lucene's cost of 2 (#830). The sloppy-phrase walk was in-order only — each next term had to match strictly after the previous — so"quick brown"~2never matchedbrown quick, diverging from Lucene'sSloppyPhraseMatcher. It is now a minimal-window sweep over adjusted positions, shared by the memtable and segment arms so the answer does not change at flush, and coveringmatch_phrase,match_phrase_prefixandmulti_matchphrase modes.A date field's epoch scale comes from its mapping, not from each value (#790). The scale was guessed per value — four or more fractional-second digits meant nanoseconds, coarser meant milliseconds — so one column could hold both scales six orders of magnitude apart. A sub-millisecond document matched a
rangeortermbefore flush and vanished after it, because the millisecond bound bisected to nothing against a nanosecond sort key. Precision is now a property of the field (dateis milliseconds, a declareddate_nanosis nanoseconds), threaded through every sort and shadow-range site. No on-disk format change.Numeric and boolean fields enforce their declared type (#781). A field mapped
integeraccepted1.9and kept it, accepted values outside the type's range, and accepted non-numeric strings. Writes now follow ES'scoercesemantics:1.9into an integer type stores1,"5"stores5, and out-of-range or unparseable values are refused with a 400 instead of being stored. A field-level"coerce": falseis honoured, andignore_malformedkeeps precedence.Upgrade note: coercion rewrites the value in
_source, so an index that spans this upgrade can hold both spellings of the same logical value.termandtermson a boolean field now normalise their operand the waytermalready did, and the two spellings compare equal, so a query matches documents written on either side of the upgrade.knnbesidequeryscores as a sum over the union (#825). The ES-native hybrid body silently dropped the vector half whenever aggregations, sort, collapse, rescore or highlight were present: the request answered 200 with a purely lexical result set and no warning. The vector leg is now pre-executed and its top-k pinned into the query tree, so the hit set is the union, a document reached by both halves scores the sum, and every request feature applies.Known regression, measured and not hidden: on the hybrid shape that previously took the RRF route, this is substantially slower — the pinned clauses cannot project to the full-text index, so the query falls back to a stored-document scan (~208 ms vs ~2.5 ms on a 100k-document index at k=10). Correct-and-slow was chosen over fast-and-wrong; #892 carries the indexed-route design that removes it.
autoindexnever aborts a run over a legacy global-catalog mapping (#755). Upgrading onto a catalog whoseprefixfield pre-existed astextaborted the entire indexing run. An upgrade must never do that.Removing a file no longer leaves graph edges pointing at it (#736). Deletion removed a file's documents and its outbound edges but left inbound edges dangling, so graph queries returned edges to nodes that no longer existed.
Recorded late
These shipped in rc.70 or earlier and were never written down, because rc.19–rc.70 have no CHANGELOG sections at all. They are recorded here rather than left under an "Unreleased" heading that misdescribes them. (The gap has since been closed — the reconstructed rc.19–rc.70 sections appear below, so these fixes now also appear in their true releases. The entries are kept here because they carry the fuller write-up.)
Wrapping a query in a one-clause
boolno longer changes its_scoreor its ranking (#399).{"bool":{"must":[X]}}and bareXare the same query — Lucene'sBooleanQuery.rewriteerases a one-clause boolean before anyWeightis built (BooleanQuery.java:279-298) — but XERJ steers on the shape of the query tree, andis_doc_scan_querymatches everybool. On an index whose documents were still in the memtable, the wrapped form was therefore scored per-document by the IDF-less brute scorer (1 + ln(1 + tf)) while the identical barematchtook the memtable BM25 arm. Measured on a 120-document fixture with onetextfield, same clause throughout: the wrapped query scoredd004at2.7917595where the bare clause scored it0.0068451734, andsize:1returned a different document for each. The flushed (segment FTS) population already agreed and still does.A one-clause
boolis now erased before any path decision reads the tree — a lonemust, or a loneshouldwithminimum_should_matchabsent,0or1, exactly the cases Lucene rewrites to the clause itself. A lonefilter(BoostQuery(ConstantScoreQuery(q), 0), score 0) and a lonemust_not(pure-negative) are not their child and are left alone. The two page-local score-rewrite gates further down now read the same normalised tree the executor was steered by, rather than the raw request, so the divergence cannot reappear one gate later.Known and unchanged: on an unflushed index a
filter-onlyboolstill scores1.0instead of0.0. That is the separately-tracked filter-context defect in the memtable's brute scorer, not this one, and this change neither fixes nor worsens it.A file excluded by a widened ignore/hidden rule was reported as a user deletion, so the graph resume path refused the rerun and its recovery contract told operators to "restore the removed file(s)" — a file still on disk (#439, first fix). The resume-plan reconciler (
UnsupportedInventoryDelta::between) now splits a plan file that vanished from the walk by whether it is still on disk, probed through the reversible raw-bytespath_idso a hidden non-UTF-8 name (the case #439 was filed for) is matched instead of read as a phantom removal through its lossyrel. A genuine deletion keeps the "restore the removed file(s)" route; a still-on-disk exclusion now says the file is still on disk and names the real routes (narrow the rule to re-admit it, or rebuild isolated to drop it). The refusal itself is unchanged — the documents still stay live — because sweeping an excluded group from the destination (so a widened exclusion actually removes the data) is a larger change tracked in #589.sorton a field XERJ cannot resolve returnednullon every hit instead of an error (#437).compute_sort_valuesspecial-cased_score,_doc,_idand the four meta-fields #420 made resolvable (_seq_no,_version,_primary_term,_index); every other ES meta-field name (_source,_size,_doc_count,_field_names,_meta,_tier,_nested,_nested_path,_feature,_parent,_matched_queries) and any unmapped or misspelled field fell through to a plain_sourcelookup that returnedNullwhen absent. The whole result set tied on[null], HTTP 200, andsearch_afterpaging on that field was stranded at page one with no error anywhere — the same silently-truncated-export failure mode #198 closes for scroll, reached throughsortinstead.A prior attempt, #402, tried a request-level denylist of "known-unresolvable" field names and was closed after independent verification refuted it: XERJ deliberately accepts a metadata-named key inside a document body (
{"_seq_no": 1}, where real Elasticsearch rejects it — see #420's entry above), so whether a name resolves is a property of the corpus being searched, not of the field name, and a static list is guaranteed wrong on some corpus. #402 was also wired into_searchonly, leaving_msearch,_search/templateand_async_searchanswering the identical "unresolvable" field with 200.This fix checks the index's schema instead of the corpus — the same thing real Elasticsearch validates — with a single gate inside the engine's
search_inner, which every search entry point funnels through regardless of endpoint. A sort field that is not one of the seven already-resolvable special cases and is not declared in the schema (via the same recursive dotted-path resolution already used for.keywordmulti-fields and nested/object properties) is now rejected up front with Elasticsearch's real error: a 400search_phase_execution_exceptionwrappingquery_shard_exception, reason"No mapping found for [x] in order to sort on".Known gap, not closed here:
collapse.inner_hits.sortand aggregationtop_hits.sortimplement their own sorting independently ofcompute_sort_valuesand are not covered by this gate — an unresolvable field named in either still sorts silently wrong rather than erroring. Tracked as a follow-up.
[1.0.0-rc.71] - 2026-08-29
An Elasticsearch-compatibility correctness release: query, aggregation, API and autoindex semantics defects, a ~100x incremental re-index, and two metrics counters that had been registered but never incremented.
Note on scope: releases rc.19 through rc.70 shipped without CHANGELOG sections.
This entry covers only rc.70..rc.71 and does not attempt to reconstruct that
gap. The three entries still under [Unreleased] above describe work that
shipped in rc.70 or earlier and are deliberately left there rather than
silently reattributed.
Fixed — query semantics
matchhonorsminimum_should_match(#832). It was ignored outright, so everymatchbehaved as a pure OR.minimum_should_matchhonors negative and combination specs (#846).-2,-25%and"3<90%"now resolve against the should-clause count instead of degrading to OR. TheMinShouldMatchSerializeimpl is now hand-written: an untagged derive emittedNegative(2),Percentage(2)andFixed(2)all as the bare number2, so the query cache — which hashes the serialized request — conflated three semantically different queries into one entry (#860).dis_maxover memtable documents scores through the real per-field BM25 path (#834). Buffered matches took a flat-1.0 catch-all, so a document matching more disjuncts tied with single-disjunct documents and the ranking changed at flush.- A
nestedquery scores parents by the childscore_mode(#836) —avg(the ES default),max,min,sum,none— instead of contributing a flat 1.0. fuzzyandmatchhonorprefix_length(#848), on the doc-scan and segment paths and through nested-path rewriting.- A
must_not-onlyboolno longer projects to a pure-negative FTS bool (#797). - A
termon a date field matches by instant across every path (#788), andtermson date/ip fields normalizes values the same way a singletermdoes (#799, #801). existsrequires a non-null value, not merely key presence (#792).- A keyword
prefix/wildcardinsidebool.must_notstays case-sensitive after flush (#794). - A CIDR
termsurvives flush — rewritten to a range on anipfield (#782) and routed to the source scan when schemaless (#786). function_scoreper-function filters are schema-aware (#802) — the last schemalessdoc_matches_querycaller in production code.sorton a multi-valued field reduces to a min/max representative (#826).
Fixed — aggregations
- Metric aggregations fold ALL values of a multi-valued field (#842), instead of the first.
termsinclude/excluderegexes are fully anchored (#837) — they matched as substrings.termsbucket keys are typed by the field's source shape, not by content (#864). A keyword value that merely looks numeric ("007") stayed a string instead of being coerced to 7.terms_keyordering is numeric on numeric fields (#839).missingcounts empty and only-null arrays as missing (#844) and resolves presence via theexistsresolver (#847).filterssupportsother_bucket/other_bucket_key(#856).
Fixed — API
mgethonors a per-item_source(#855) —false, an includes list, or{"includes":[..],"excludes":[..]}.GET /_doc_source_includes/_source_excludeshonor nested paths (#850)._field_capsfieldsfiltering honors full globs (#853).- A partial
_updatedeep-merges nested objects (#821) — it was shallow and dropped sibling keys.
Fixed — autoindex
- C++ symbols defined inside a function body are no longer indexed
(#852) — a function-local
struct/classand its enumerators leaked as file-scope API symbols. - Top-level constants are indexed as symbols across 7 languages (#500).
Fixed — observability
xerj_bytes_written_totalis wired to the flush and merge segment writes (#804) andxerj_bytes_read_totalto validated segment opens (#819). Both counters were registered but never incremented, so each read a flat zero regardless of load.- The disk flood-stage log reports free space, and the watermark can be
retuned at runtime (#823)
via
PUT _cluster/settings, matching ES's recovery flow without a restart. An override cannot enable the watermark when it is disabled in config.
Performance
Incremental re-index skips the Phase-A parse for byte-identical files (#868). The tree-sitter parse is
72% of extraction cost (57 ns/byte, measured), and an unchanged re-index re-parsed the whole corpus to rebuild sketches. Gating on the committed content digest turns that from O(corpus) into O(changed) — roughly 100x on the edit-and-rerun and CI paths. The digest check inreconcile_planremains the correctness authority: a changed file whose scan was wrongly skipped mismatches there and fails closed rather than carrying stale documents.Known limitation: the fast path is keyed on file content only. It carries no extractor version, so upgrading to a release that extracts more symbols does not re-extract unchanged files until their content changes.
Changed
- Release PRs are gated on their notes matching the tree (#474) — issue states against the live tracker, merge coverage since the previous tag, and freshness against the base.
- The Build+Test job timeout is 60 minutes (was 50) (#770). This is defense in depth, not a fix: #751, the intermittent hang in the default-parallelism workspace test step, is still open and still bounded by that step's own 12-minute cap.
llms.txtno longer hardcodes the current release version (#796) — it drifted behind every release and handed agents the wrong version.
Documentation
- The reference-coding case study's charts are rebuilt as vertical comparison bars, and its colours now resolve in both themes (the page referenced CSS variables that do not exist, so every chart fell back to night-only values).
match_phraseslopis documented as in-order only, not transpositions (#830).
Reconstructed 2026-09-26. Releases rc.19–rc.70 shipped without CHANGELOG sections at cut time (acknowledged in the rc.71 note below). The entries in that range were rebuilt after the fact from
git log(merge subjects and commit bodies) and each tag's GitHub release record; every issue/PR link they carry was checked to exist within its release window. They are drier than ship-time entries — numbers are quoted only where a commit body recorded them — and nothing was force-fitted: the three entries that were still under[Unreleased]when rc.71 shipped were recorded, fuller, under rc.72's "Recorded late" heading above, and stay there.
[1.0.0-rc.70] - 2026-08-26
A CLI and autoindex-ergonomics release: xerj search lands as the one-command
query client, autoindex warnings stop being silencible, and engine invariants
for #421 get binding tests.
Added
xerj search— the binary as a query client (#764). Query from the CLI directly; the reference-coding demo drops its scripts. Agent docs now teach it as the one-command query (#771), and it renders a code-symbol line plus code fields.
Fixed
- XERJ_URL warnings survive
--quiet(#768, via #769). The wrong-node warning is routed throughpr.warninstead of a bareeprintln, so it can no longer be silenced. Companion fix #765 (#766): warn when XERJ_URL is set but--urlwas not passed, extended to map and status. - Truncated files named in the #381 cap note (#759, via #762); the #381 cap is also pinned at its exact boundary.
semantic_texthint suppressed only when the vector dispatches (#394, via #776), and the API docs stop over-claiming thatdispatching_vector_fieldsmirrors peel exactly (#777, #778).- Legacy file-alias catalog doc swept by id (#739, via #754); the #729/#730 review nits documented under #731 (#753).
Changed
- CI Build+Test hangs bounded (#751, via #767): step-6 plus a job-level timeout so main is not stuck for hours.
Changed — tests / docs
- #421 invariants bound: the
_idsort-value type and thebuild_doc_value_columnsunderscore-skip invariant each get tests (#772, #773). - #755 catalog-error enrichment wiring guarded (#760, via #761).
- Six-model first-contact field report (#774):
XERJ works; agents skip the required report. In response, llms.txt surfaces
xerj feedbackas first-run step 4 (#775). - Release chores: landing version stamps bumped to rc.69 (#763).
[1.0.0-rc.69] - 2026-08-25
An autoindex-focused correctness release: two defect fixes in file section-record handling and legacy-catalog errors, plus landing version chores.
Fixed — autoindex
Section records per file are capped and truncation is flagged (#381, #758). The cap is applied on both the graph index path and the
--no-graphgenerated path, and truncation is surfaced rather than silently dropped.Legacy-catalog mapping conflicts return an explanation instead of an opaque 400 (#755, #756).
Changed — chores
- release chores: landing version stamps bumped to the released rc.68, with SEO heads and sitemap regenerated (#757).
[1.0.0-rc.68] - 2026-08-25
An autoindex-correctness release: the exclusion sweep gained the missing invalidations and scoping it needed, plus a batch of --no-graph pipeline fixes; around it, an embedder identity fix, a self-verifying release/installer toolchain, an agentic-SEO content programme, and CI hardening.
Fixed — autoindex
- Exclusion sweeps now purge everything a removed file leaves behind — graph
edges (#694), alias catalog
docs (#693), inbound graph edges
(#736 sweep half), and a legacy
file:catalog doc by id (#739). - Sweeps are prefix-scoped so a sibling corpus is not over-deleted (#737).
--no-graphpipeline fixes: the generated pipeline never consumedas_document(#722); a new document/config is routed to the docs dataset on incremental runs, and acceptance no longer re-counts pruned fields (#729); a--no-graphgenesis bootstrap continued on the graph path is now refused (#585 case 2).- A byte-identical duplicate no longer wedges a completed run (#345).
- Catalog read-back diagnostics name the disagreeing field (#367), with the diff truncated on a char boundary.
- Files skipped after plan freeze surface distinctly, not below the noise
floor (#346); the genesis-guard
recovery message offers
--freshand the dry-run refusal is documented (#718); a frontmatter body is classified by its structured family (#587 item 1).
Fixed — engine
- The proxy endpoint is folded into the embedder identity (#533); the wire identity version stays at 1 with a separate on-disk record version, and the version-delta grandfather still refuses a backend change.
Fixed — release & site
- Correct CLI banner version, /get.sh installer, self-updating RC number (#735), hotfixed for three defects that merge introduced (#741), with a /get.sh fallback when Functions are off (#740).
Added
- Agentic-SEO programme (#550): 76 articles, 16 comparisons, plus site-wide SEO repair.
- grep-vs-retrieval benchmark harness for Lucene (#493) — harness only, no results yet.
Changed — ci / tests / chores
- CI merges resolve before landing, tree-sitter hoisted
(#352); every workflow action
pinned to a commit SHA and the toolchain to 1.97.1
(#456); the default-parallelism
gate widened to the full
--workspace(#384); landing/ deploys to Cloudflare Pages. Test coverage for the Percentage/Field/Script msm unwrap (#643 gap 1). Release chores: version stamps, CLA + artifact-verification reports from a new contributor, stale-comment corrections (#719, #725).
[1.0.0-rc.67] - 2026-08-24
A small correctness-and-docs release: one engine fix to dynamic field-budget admission counting, two test-docstring corrections from review follow-ups, and the rc.66 landing stamp.
Fixed
- Dynamic field-budget counts the whole subtree at admission (#312, #708). The admission-time budget now accounts for the entire field subtree rather than a partial count.
Docs
- Corrected the #643 neutrality-guard docstring (review follow-up) (#643, #704, #706).
- Corrected the #577
residual_gate_boundsmaterialisation_limitcomment (#577, #705, #709).
Release chores: landing version stamp for v1.0.0-rc.66 (#707).
[1.0.0-rc.66] - 2026-08-24
An engine-correctness release: three query-semantics and kNN filtering fixes, plus the rc.65 landing stamp. The metrics chore logged 1928 assets and 1013 binaries downloaded as of cut day.
Fixed
residual_gateeager-trimsall_hitsto the page cap (#577, #705). The gate now trims the hit set to the requested page size instead of carrying the full set.Single-clause
boolis unwrapped throughfunction_scoreanddis_max(#643, #704). Wrappers no longer hide a one-clause bool from scoring semantics.Nested kNN pre-filter evaluates schema-aware (#423, #702). Pre-filtering on nested documents now respects the index schema; the direct commit extends the same treatment to post-filters.
Changed — chores
- Release chores: landing stamp for v1.0.0-rc.65 (#703).
[1.0.0-rc.65] - 2026-08-24
A small correctness release: one engine fix to filtered kNN, plus docs and release-chores merges.
Fixed
- Filtered kNN evaluates its filter schema-aware (#423, #701). The filter used during kNN search is now evaluated with awareness of the schema.
Changed — docs & chores
- Softened
meta_sort_valuedocs claim (#421, #699) — item 2 of #421: the sort-vs-_seq_noagreement claim was toned down. - Release chores: landing stamp for v1.0.0-rc.64 (#700).
[1.0.0-rc.64] - 2026-08-23
A two-fix engine release: a memtable sort-path performance defect and a nested-terms aggregation correctness defect, plus release chores.
Fixed
Meta-aware memtable pre-clone rejection for
_seq_no/_versionsorts (#421, #698). The memtable was pre-cloned on sorts over_seq_no/_version; the clone is now rejected for those meta fields, restoring O(1) behavior.Nested-terms sub-aggregations deferred to surviving top-N buckets (#375, #696). Sub-aggregations under nested
termsnow run only over the buckets that survive top-N truncation rather than over all buckets.
Changed — chores
- Release chores: landing the v1.0.0-rc.63 version stamp (#697).
[1.0.0-rc.63] - 2026-08-23
An autoindex-semantics release: two fixes around excluded groups and pending no-graph generations, one test extraction, and a version stamp.
Fixed — autoindex
- A newly-excluded group's documents are swept, not refused (#589). When a group became excluded, the delta was rejected outright; it is now swept, with exact-dataset indices covered by a mock-ES integration test (#690).
- A pending
--no-graphgeneration resumed on the graph path is refused (#585). "Case 1" of the graph-crosspath defect: a pending sync resumed under a mismatched graph mode is now rejected rather than silently continuing (#692).
Changed — tests
Release chores: landing the v1.0.0-rc.62 version stamp (#691).
[1.0.0-rc.62] - 2026-08-23
A small autoindex correctness release: one fix to correlation catalog id scoping, plus the rc.61 landing stamp.
Fixed
- Prefix-scope for autoindex correlation catalog ids (#673,
#688). Correlation catalog ids
(
corr:/tcorr:) are now scoped by prefix, per the fix in #688.
Release chores: landing the v1.0.0-rc.61 version stamp (#687).
[1.0.0-rc.61] - 2026-08-23
A single-fix correctness release plus release chores.
Fixed
- Single-clause bool unwrap now covers percentage
minimum_should_match(#643, #686). The unwrap of single-clause bool queries handled integer-valuedminimum_should_matchbut not percentage values below 200%.
Release chores: landing the v1.0.0-rc.60 version stamp (#685).
[1.0.0-rc.60] - 2026-08-23
A small query-semantics release: _name on bool and prefix/wildcard clauses,
plus the rc.59 landing stamp.
Fixed — query semantics
_namehonored on a bool clause (#681, #684). Named-query matching previously ignored_nameset on a bool clause._namehonored on the expanded prefix/wildcard form (#681, #682). The internally expanded prefix/wildcard rewrite dropped the clause name; #684 completes this fix for the bool form.
Changed — chores
- release chores: landing version stamp for v1.0.0-rc.59 (#683).
[1.0.0-rc.59] - 2026-08-23
An Elasticsearch-compatibility correctness release: two query-semantics fixes (matched query naming and object sub-field term matching), plus a version stamp. No performance numbers were cited at cut time.
Fixed — query semantics
matched_queries(_name) uses the schema-aware matcher (#669, #680). Named-query reporting now goes through the same schema-aware matching path as the rest of the engine.termon an object sub-field is flush-invariant (#677, #678). A term (and range count) on a dotted object sub-field could diverge depending on flush state; the matcher now falls back to a dotted-path source scan, with a reproducing test (previously ignored) landing alongside.
Changed — chores
- Release chores: landing the v1.0.0-rc.58 version stamp (#679).
[1.0.0-rc.58] - 2026-08-23
A small correctness release: two search-semantics fixes (keyword fuzzy case-sensitivity, object-field prefix/wildcard flush-invariance) plus a landing version stamp.
Fixed
Keyword fuzzy is case-sensitive by default (#423, #406; PR #675). Fuzzy previously ignored
case_insensitiveand always folded case; acase_insensitiveflag is now threaded throughQueryNode::Fuzzy, honored on all matcher paths, and the sub-token fallback on the DV path is gated behind it. Includes a fail-before repro test.Object-field prefix/wildcard is flush-invariant (#413; PR #676). The memtable prefix/wildcard path now mirrors the whole-object blob, so results match across flushes.
Release chores: landing version stamps, changelog housekeeping (rc.56 → rc.57, PR #674).
[1.0.0-rc.57] - 2026-08-23
A small correctness release: two engine/autoindex semantic fixes plus a release stamp.
Fixed
Keyword wildcard matching is case-sensitive by default (#668). Previously the keyword wildcard path folded case; it now honors the Elasticsearch default, with a
case_insensitiveflag threaded throughQueryNode::Wildcardso all matcher paths respect it. Landed with a fail-before repro test.Autoindex catalog doc IDs are scoped by corpus prefix (#416). Catalog documents from different corpora no longer overwrite each other's IDs (no cross-corpus overwrite).
Changed — chores
- release chores: landing version stamp-bump rc.55 → rc.56.
[1.0.0-rc.56] - 2026-08-23
A single-fix correctness release: prefix and wildcard case-folding was flush-invariant, plus the usual release stamp.
Fixed
- Prefix & wildcard case-folding is flush-invariant (#396, #667). A prefix query on a keyword field folded case while the document lived in the memtable but was case-sensitive once flushed to a segment, so the same query returned different results depending on flush state. Landed with fail-before and broadened repros (prefix also folded the query for text fields).
Release chores: landing version stamp rc.54 → rc.55 (#666).
[1.0.0-rc.55] - 2026-08-23
A one-fix release: a count-semantics defect for term queries over keyword
arrays, plus the routine landing stamp.
Fixed
termon a keyword array is flush-invariant for counts (#408, #665). Document counts for atermmatch against a keyword array no longer shift across segment flushes.
Release chores: landing version stamp rc.53 → rc.54 (#664).
[1.0.0-rc.54] - 2026-08-23
A one-fix release: POST /<index>/_close now actually reclaims RAM instead of just flipping a flag, plus the routine version stamp.
Fixed
_closefrees memory;_openreconstructs the index; closed indices stay visible (#463, #663). The engine gainedclose_index/reopen_indexmethods that release caches while keeping the index on disk loadable, the API_close/_openendpoints were wired to them, and_cat/indicesnow lists closed indices with statuscloseinstead of dropping them from view.
Release chores: landing version stamp rc.52 → rc.53 (#662).
[1.0.0-rc.53] - 2026-08-23
A small API-consistency release: one scroll behavior fix plus a release stamp.
Fixed
_search_scrollfirst page now suppresses_sourcelikesearch_impl(#659, #661). The scroll API's first page diverged from the search path's_sourcehandling; it now matches, with accompanying comment refreshes in the API docs.
Changed — chores
- release chores: landing version stamp rc.51 → rc.52 (#660).
[1.0.0-rc.52] - 2026-08-23
A small API/engine correctness release: two _source handling defects —
one in _source filtering inside nested arrays of objects, one in scroll
continuation — plus a release stamp chore.
Fixed
_sourcefilters project inside nested arrays of objects (#653, #656). Field-level_sourceprojection had not been applied to hits whose values were nested arrays of objects.Scroll continuation matches first-page
_sourcesuppression (#637, #658). Continuation pages ignored the_sourcesuppression in effect on the opening request; the fix applies per-hit mapping suppression scoped to the opening route.
Changed — chores
- release chores: landing stamp-bump v1.0.0-rc.50 → rc.51 (#657).
[1.0.0-rc.51] - 2026-08-23
A single-fix correctness release: collapse results under an explicit _source
projection, plus the rc.50 landing stamp.
Fixed
- Collapse sentinels survive an explicit
_sourceprojection (#651, #655). Inner hits undercollapsevanished when the request also carried_source.includes; the engine now preserves the collapse sentinels across that projection. Landed with a reproducing API test and a rustfmt pass.
Release chores: the v1.0.0-rc.49 → rc.50 landing stamp (#654).
[1.0.0-rc.50] - 2026-08-23
A small _source projection correctness release: two fixes around how source
filters apply inside arrays of objects, plus release chores.
Fixed
_sourcefilters are projected inside arrays of objects (#644, #652). Field-level source filtering previously did not descend into arrays of objects; emptied array elements and the empty array key are now pruned from the projection.inner_hitsmembers honour the default_sourceprojection (#646, #650). Collapsed inner_hits entries now respect the default source projection instead of returning the full document.
Changed — chores
- release chores: landing version stamp rc.48 → rc.49 (#649), plus the routine release-downloads metrics snapshot.
[1.0.0-rc.49] - 2026-08-23
A single-fix API release: fields/docvalue_fields now resolve named companion
fields through the default _source, plus the rc.48 landing stamp landing late.
Fixed
fields/docvalue_fieldsresolve a named companion under the default_source(#310, #647). Previously a named companion field was missed when the default_sourcewas in play; the fix pierces_sourceand re-narrows it at the emission sites named in the commits (the top-level_sourceand the scroll snapshot), and an unresolvabledocvalue_fieldis now omitted rather than emitted as[].
Changed — chores
- Release chores: the v1.0.0-rc.47 → rc.48 landing stamp (#645) landed inside this window.
[1.0.0-rc.48] - 2026-08-23
An Elasticsearch-compatibility correctness release: two query/API semantics
fixes (_source glob filtering, one-clause bool scoring) plus regression
guards for collapse and rescore behavior.
Fixed
Faithful ES
_sourceglob automaton for nested paths (#602, #641)._sourceglob filtering now follows ESmakeAnyStringsemantics — globs cross dots, non-accepting scalars are dropped, and dot-less globs route through the same automaton.A one-clause
boolis a no-op (#399, #642). A single-clauseboolno longer changes_scoreand rank order.
Added — tests
- Collapse
inner_hitspage1==page2 byte-identity guard (#623, #640). - Real rescore ranking test + keyword-rescore guard (#627, #639).
Release chores: landing version stamp (#638).
[1.0.0-rc.47] - 2026-08-22
A one-fix release: a scroll API defect plus the routine landing stamp.
Fixed
- Scroll continuation honours
_source: falsefrom the opening request (#624, landed via #636). A scroll continuation was ignoring the_source: falsesetting made on the opening request, returning source documents anyway.
Release chores: landing version stamps for v1.0.0-rc.46 (#635).
[1.0.0-rc.46] - 2026-08-22
A small release: one targeted engine performance fix plus release chores.
Performance
- Residual gate skips Hit materialisation under
count_only(#577, #633). When a count-only query takes the residual-gate path, hits are no longer materialised — the count is produced without building theHitstructures. Landed as part 1 of #577.
Changed — chores
- Release chores: landing the v1.0.0-rc.45 version stamp (#632).
[1.0.0-rc.45] - 2026-08-22
A small scroll-versioning release: one API fix on the _search_scroll first page, one refactor unifying the scroll hit-render paths, and a version stamp.
Fixed
Changed
- Scroll hit-render paths routed through
resolve_hit_versioning(#614, landed via #629) — a refactor unifying the versioning logic ahead of the fix.
Release chores: landing version stamp for rc.44 (#628).
[1.0.0-rc.44] - 2026-08-22
A one-fix correctness release plus the rc.43 stamp: rescore-phase keyword queries now respect the index mapping.
Fixed
- Keyword
match/multi_matchinside a rescore query is mapping-aware (#574, merged via #626). Previously the rescore phase analyzed keyword fields without regard to the mapping, so a keywordmatch/multi_matchthere behaved differently from the same query in the main phase.
Release chores: landing the v1.0.0-rc.43 version stamp (#625).
[1.0.0-rc.43] - 2026-08-22
A small correctness release: one API fix for collapse rendering on scrolled pages, plus the rc.42 landing stamp.
Fixed
- Collapse
inner_hitsrender on scroll continuation pages (#621, #622). The API failed to render collapseinner_hitson pages after the first of a scroll.
Release chores: landing version stamp for v1.0.0-rc.42 (#620).
[1.0.0-rc.42] - 2026-08-22
A small correctness release: one _source filtering fix, plus release chores.
Fixed
_sourceprefix.*include now requires the dot boundary (#602, part of). Aprefix.*include pattern was matching fields whose names merely started withprefix(e.g.prefixfoo); it now matches onlyprefix.something. Landed via #619.
Release chores: landing the v1.0.0-rc.41 version stamp (#618).
[1.0.0-rc.41] - 2026-08-22
A one-fix correctness release plus release chores.
Fixed
- Hybrid total reports
Eqat the exactper_query_topkboundary (#594, #617). The hybrid query's total-hit count reportedEqwhen results landed exactly on the per-query top-k boundary.
Release chores: landing the v1.0.0-rc.40 version stamp (#616).
[1.0.0-rc.40] - 2026-08-22
A small correctness release: one scoring-semantics fix for multi_match queries, plus release chores.
Fixed — query semantics
multi_matchbest_fieldsover keyword+text fields now scores bydis_max(#572, PR #615). A keyword+textbest_fieldsmulti_match was being scored as a sum across fields instead of taking the best-matching field, over-weighting documents that matched both.
Release chores: landing the v1.0.0-rc.39 version stamp (PR #613).
[1.0.0-rc.39] - 2026-08-22
A small correctness release: one API fix for hit versioning metadata, plus release chores.
Fixed
- Unresolved hits no longer fabricate
_seq_no/_version(#603, #612). When a hit could not be resolved, the API had been inventing sequence-number and version values for it; it now omits those fields instead.
Release chores: landing version stamp for v1.0.0-rc.38 (#611).
[1.0.0-rc.38] - 2026-08-22
A small release: one engine test-stability fix plus the rc.37 landing stamp.
Fixed
Release chores: landing version stamp for v1.0.0-rc.37 (#609).
[1.0.0-rc.37] - 2026-08-22
A small correctness release: aggregation memory is now bounded by the configured query-memory limit, plus the usual release chores.
Fixed
- Aggregation owned-source materialisation is bound to
max_query_memory_mb(#464, #608). Both aggregation paths now bound their corpus memory under the setting instead of materialising unbounded owned sources.
Changed — chores
- release chores: landing version stamp-bump v1.0.0-rc.35 → rc.36 (#607).
[1.0.0-rc.36] - 2026-08-22
A one-fix autoindex release plus release chores: Java constant symbols now land in the autoindex, and the landing page version stamp moved rc.34 → rc.35.
Fixed — autoindex
- Java static constants indexed as their own symbols (#500). Static and interface constants were not previously emitted as standalone symbols by the autoindex (#605).
Changed — chores
- release chores: landing version stamp-bump rc.34 → rc.35 (#606).
[1.0.0-rc.35] - 2026-08-22
An Elasticsearch-protocol correctness release: three targeted fixes — keyword
query rewriting inside vector/score-filter contexts, collapse inner_hits
metadata honesty, and _source dotted-filter handling of empty objects.
Fixed
- Keyword
match/multi_matchare mapping-aware inside Knn/SemanticSearch/function_score filters (#574, #604). - Collapse
inner_hitsomit absent snapshot_seq_no/_versioninstead of fabricating values (#566, #601). - An already-empty object is preserved under a dotted
_sourcefilter, with the relaxation gated to an accepting state and the accept test made ES-correct for.*globs (#310, #599).
Changed — chores
- Release chores: landing version stamp-bump rc.33 → rc.34 (#600).
[1.0.0-rc.34] - 2026-08-22
A single-fix engine release: the schema-evolution throttle was keyed too coarsely, plus the routine version stamp for the prior cut.
Fixed
- Schema throttle re-evolves on value-shape changes (#382, #598). The evolution throttle fingerprinted only field names, so a refused key never re-evolved when the leaf value type changed; the leaf value-type is now folded into the hash, and the fingerprint recurses into array elements so array-shape transitions re-evolve too. Landed with an end-to-end guard test and a correction to the accompanying hash-reach comments.
Changed — chores
- Release chores: landing version stamp v1.0.0-rc.32 → rc.33 (#597).
[1.0.0-rc.33] - 2026-08-22
A small correctness release: one API fix for _eql/search event ordering, plus release chores.
Fixed
_eql/searchorders events by@timestampinstant across alias members before truncating (#567, #596). Events from multiple alias members were previously not globally ordered by@timestampbefore the result set was truncated, so events could be cut out of order.
Changed — chores
- release chores: landing version stamp bump v1.0.0-rc.31 → rc.32 (#595).
[1.0.0-rc.32] - 2026-08-22
A small correctness release: one wire-protocol semantics fix for hybrid/fusion totals, one test-flake fix, and release chores.
Fixed
- Hybrid/fusion totals report
relation:gtewhen capped (#569). A capped total was previously reported as a falseeq; it is now surfaced asrelation:gte(#593). - Stale HNSW rebuild test no longer flakes (#536). The test now deterministically awaits the rebuild instead of racing it (#592).
Changed — chores
- Release chores: landing version stamp-bump rc.30 → rc.31 (#591) and metrics-only release-download collection (no engine change).
[1.0.0-rc.31] - 2026-08-22
A one-fix autoindex correctness release, plus a landing-page version stamp.
Fixed
- Autoindex no longer treats an excluded-but-present file as a deletion (#439, #590). Files excluded from indexing while still on disk — including those with non-UTF-8 names — were being misread as deletions; the two cases are now distinguished.
Changed — chores
- Release chores: landing stamp-bump v1.0.0-rc.29 → rc.30 (#588).
[1.0.0-rc.30] - 2026-08-22
An autoindex-correctness release: three fixes to how the autoindexer sniffs files and guards corpus re-runs, plus a landing version stamp.
Fixed — autoindex
Markdown-with-frontmatter is sniffed as its body, not YAML (#551). The sniffer treated the frontmatter as the document, so the indexed content was the metadata block rather than the prose.
A graph-path re-run of a
--no-graphcorpus is refused before it mutates (#490). The cross-path mutation was previously caught only after the damage; the check now precedes it, and the regression test was made load-bearing.No more false "identity changed" for non-resumable backends (#487). A backend that simply cannot resume was misreported as a corpus identity change.
Release chores: landing stamp-bump v1.0.0-rc.28 → rc.29 (#582).
[1.0.0-rc.29] - 2026-08-21
A small autoindex-correctness release: two fixes to the code autoindex/re-index path, plus a version stamp. The code-indexing pipeline had been collapsing declarations together and aborting on a schema edge case.
Fixed
Each code declaration is indexed as its own retrievable document (#500, #579). Previously declarations were not individually retrievable; per-declaration records are also deduped by locator.
Re-index no longer aborts on a field absent-but-unscored in the frozen schema (#580, #581). The re-index run previously treated such fields as fatal.
Release chores: landing version stamp rc.27 → rc.28 (#578).
[1.0.0-rc.28] - 2026-08-21
A one-fix correctness release: a bool-filter scoring defect, plus the rc.27 version stamp landing in the same window.
Fixed
- Non-projectable bool
filter/must_notno longer diverts the scorer (#361). A bool query whose filter or must_not clause was non-projectable was diverting the scorer; it no longer does (#576).
Changed — chores
- Release chores: landing the v1.0.0-rc.26 → rc.27 version stamp (#575).
[1.0.0-rc.27] - 2026-08-21
An ES-compatibility correctness release: hybrid knn + query was not actually
hybrid, keyword match/multi_match ignored the mapping (in the memtable and
inside wrapper queries), and cat_count swallowed per-member errors.
Fixed
knnbesidequerynow folds to a true hybrid only when hybrid-safe (#458). Previously the lexical answer was returned and labelled hybrid; the rework completes thehybrid_safeguard for every fieldrun_hybriddrops, with tests covering the union result and aggs-200.- Keyword
match/multi_matchis mapping-aware in the memtable (#354) and inside wrapper queries (the #354 residual), including boostedmulti_matchafter stripping^boost(#571, #573). cat_countpropagates per-member errors instead of undercounting (#563, #568).
Release chores: site release stamps bumped to v1.0.0-rc.26 (#570).
[1.0.0-rc.26] - 2026-08-21
An alias-semantics and honesty release: filtered-alias reads fixed across the remaining ES-compat endpoints, a collapse inner_hits metadata fix, a semantic-clause dispatch fix, and a benchmark-harness correction alongside site documentation cleanup.
Fixed
- Alias filters applied on
_msearch(#451 class C, #559). The same class-C pass also covers_msearch/template,_cat/count, and a scroll test guarding that scroll honours a filtered alias. _eql/searchfans out to every alias member (#451, the last class-B read site, #561); the alias filter is applied on the fan-out, closing a filtered-alias leak.- Collapse
inner_hitscarry snapshotseq_no/version(#506, #562). - Single-clause
boolwrapping a semantic clause now dispatches (#395 case B, #557).
Changed
- Harbor 3-arm benchmark honesty (#516, #564): partial arm coverage is now flagged and the tuned prompt's pre-asserted claim neutralized.
- Site docs: built-in neural embedder documented, stale tally corrected and dead links fixed;
max_process_memory_mbdropped from the config reference table (#492, #565).
Release chores: version stamps bumped to v1.0.0-rc.25 on the site (#558).
[1.0.0-rc.25] - 2026-08-21
A small ES-wire-compatibility release: one shared alias/index-selector resolver migrating read endpoints off first-index-only behavior, plus a regression test locking filtered-alias by-query delete safety.
Fixed
- Shared index-selector resolver for alias-aware read endpoints
(#451,
#555). Read endpoints resolved
through multi-index aliases by taking only the first member;
_mget,_explain,_cat/segments, and_segmentsnow migrate onto a shared resolver (class B of #451). A direct commit reproduces the original_mgetfailure before the fix.
Added
- Regression coverage for filtered-alias by-query over-delete
(#553,
#556). Tests lock the
filtered-alias by-query slice safety and
_update_by_queryreporting aggregatedeleted:0.
Release chores: site release stamps bumped to v1.0.0-rc.24 (#554).
[1.0.0-rc.24] - 2026-08-21
A small ES-compatibility fix release: _delete_by_query/_update_by_query
through a multi-index alias, plus regression tests locking two kNN invariants.
Fixed — ES compatibility
- By-query APIs fan out to every alias member (#450).
_delete_by_query/_update_by_queryissued through a multi-index alias touched only the first member; they now apply to all members (#552).
Changed — tests
- Locked multi-knn and nested-knn 200 invariants (#547): declared-but-empty knn sections must return 200 (#549).
Release chores: site release stamps bumped to rc.23 (#548).
[1.0.0-rc.23] - 2026-08-20
A small correctness release: one wire-protocol fix for nested kNN clauses, a shipped API regression test, and release chores.
Fixed
- Unanswerable field in a nested
knnclause returns 400 (#542). The nested arm of the kNN validator now rejects such clauses with a client error instead of failing to answer (#546).
Added
- Regression test for semantic query on a non-
semantic_textfield (#530). The negative case is now covered, and the 400 response names the offending field (#545).
Release chores: bumping site release stamps to v1.0.0-rc.22 (#544).
[1.0.0-rc.22] - 2026-08-20
A small kNN-correctness release: two fixes making kNN/semantic queries on unanswerable fields return a 400 instead of a silent empty 200, plus release stamps.
Fixed
- 400 a knn/semantic clause on an unanswerable field (#498, #541). A single top-level knn targeting a field that cannot answer it no longer returns a silent empty 200.
- 400 an unanswerable field in a multi-knn array clause (#542, #543). The same semantics extended to the multi-knn array form.
Release chores: bumping site release stamps to v1.0.0-rc.21 (#540).
[1.0.0-rc.21] - 2026-08-20
A small correctness-and-hygiene release: two columnar-storage fixes, a CI unblock, an end-to-end test, and site housekeeping.
Fixed
- ZBS3 routed through the columnar retained-size estimator (#538, #539). The ZBS3 storage path now goes through the columnar retained-size estimator instead of bypassing it.
- Explicit
_sourcenulls survive the columnar codec (#415, #535). Explicit nulls in_sourceare preserved through the columnar codec rather than dropped. - Clippy 1.98 satisfied across the workspace (#537). Unblocks the Format + Clippy CI jobs.
- sitemap.xml covers the zero-token and semantic-analytics pages (#534).
Added
Changed — chores
- Release chores: landing release stamps bumped, metrics release-download counts captured (1148 assets, 624 binaries).
[1.0.0-rc.20] - 2026-08-19
A small release: one engine fix (alias metadata persistence) and a batch of site/docs corrections ahead of it.
Fixed
- Alias filter metadata survives a restart (#524, #525). Filtered-alias metadata is now persisted instead of being lost across restarts.
llms.txt"Current release" pointed at rc.19 (#515, #520, #527). Corrected, with a guard added so the stamp cannot drift again.
Documentation
- Zero-Token AI Search use-case page (#523). Three real scenarios: code, PDF, Markdown.
- Snapshot/restore scoping corrected (#400,
#528). The
wal_tapdocs now state snapshot/restore seeds a XERJ target only, not a cross-engine one. - Honest "first hit" example: the site query was replaced with one where sniff.rs genuinely ranks first.
Changed — chores
- Release chores: footer release stamps bumped to v1.0.0-rc.19 (#526).
[1.0.0-rc.19] - 2026-08-19
A broad correctness-and-honesty release: ES-compat fixes for aliases, scroll and sort; a stepped default memory cap; an embedding-mode safety guard; plus a wave of README/landing truth-alignment and two field-report harnesses.
Fixed — engine and ES-compat
- Multi-index aliases answer over all members (#449).
A multi-index alias had been answering as an alias over one member on every
read path; alias filters are now read from the names as written and applied
in
_counttoo. - Scroll carries real
_seq_no/_version(#440) on the same read as_source, and the summed scroll snapshot cap is now enforced on the alias route as well (#405). - Sort on an unresolvable field is rejected, honouring
unmapped_type(#437) instead of null-strandingsearch_after. - Embedding-mode guard: the engine refuses to open an index whose vectors
were produced under a different
embedding.mode(#446); the resolved embedder is recorded in a separate sidecar, not the mode string. - Stepped default memory cap (#477):
8/16/32 GiB by RAM instead of a flat 8 GiB, with both memory env knobs
accepting
auto|off|unlimited|<MiB>(#502). %PDF-magic signature qualified in autoindex (#403); first-launch console link prints the bound ES-compat port, not the requested one (#495);feedback --open-prcommits only the field report, not the whole staged index (#484); de-flakedbool_deadline_never_broadens_a_must(#510).
Added
- Reproducible 3-arm Harbor benchmark harness (Claude Code vs XERJ vs tuned, #513) with measured fixes from the first parallel campaign night (#521), and a field report running Frontier-Bench v0.1 as a 3-arm XERJ A/B (#512) plus one on building a token benchmark against Lucene (#494).
Changed — docs and CI
- README reworked in three passes (badges-first plain prose, one value-first opening paragraph, then repositioned around reference coding and token savings); landing pages verify release downloads (#454); the public-sector page drops at-rest-crypto claims the build does not ship (#491).
- CI: the landing-constants-guard is wired in and the site re-stamped (#489); main's CI runs are no longer cancelled, so the badge stops showing cancelled-as-failing (#503).
[1.0.0-rc.18] - 2026-08-18
Security
Patched
h2for RUSTSEC-2026-0258, "unbounded empty DATA frames" (0.4.13 -> 0.4.16).h2is the HTTP/2 implementation beneathhyper, so it sits under every listener this server runs — ES-compat and native REST through axum, gRPC through tonic. An unbounded stream of empty DATA frames was a remote resource-exhaustion vector against any reachable node; the patched version answers withGOAWAY ENHANCE_YOUR_CALMinstead.The fix adds a per-connection budget, and it is a token bucket, not a quota — though, as below, that is not enough to keep it off real traffic.
h2-0.4.16/src/proto/streams/counts.rs:94-101: a DATA frame under 256 bytes consumes256 - payload_len; a frame of 256 bytes or more replenishespayload_len - 256. The refill stops at a 25,600-byte ceiling, which lives elsewhere —Budget::new(DEFAULT_DATA_FRAME_BUDGET)atcounts.rs:89and the.min(self.max)atcounts.rs:26, overDEFAULT_DATA_FRAME_BUDGET = 256 * 100atproto/mod.rs:39. Any ordinary payload frame therefore refills what small frames drain — but only a frame that reaches 256 bytes does. A body chunked below that is made entirely of frames that drain, which is what the paragraphs below are about.Two further details decide whether you can ever meet it. A small frame with a payload is refunded when the application reads it (
streams.rs:1515) — but only once the read happens, which is later than you would think and is the subject of the next paragraph. A frame with an empty payload is not: h2 discards it before the application sees it (recv.rs:757returns early for an empty frame without END_STREAM), so its 256 is never refunded — only offset by a large frame elsewhere on the connection. "Empty" here means zero payload, not zero bytes on the wire: a padding-only frame counts, and so does a small frame on a stream that is reset before the application reads it. An empty frame carrying END_STREAM is delivered, and is refunded. Exhaustion is a connection-levelGOAWAY ENHANCE_YOUR_CALM, and it takes the 101st empty frame with nothing ≥256 bytes in between (checked_subfails onceavailablereaches 0).Reading the body is not by itself protection, and this is the part to take operationally. The charge lands on arrival and the refund only on the read, so once a flow-control window's worth of frames has arrived before the handler is scheduled, the connection is already over budget. Measured on this build, a handler consuming every frame in a tight loop is cut off at exactly the same frame as one that never reads: 101 at 1 byte, 134 at 64, 201 at 128, 356 at
- XERJ advertises
SETTINGS_INITIAL_WINDOW_SIZE = 1048576, far above h2's 64 KiB default, so more frames buffer here than elsewhere before a read lands.
The safe line is frame size, not frame count, and it is sharp: a frame of 256 bytes or more never charges at all. The trip frame for a smaller size is
floor(25600 / (256 - size)) + 1— the first frame whose charge underflows the remaining budget. Below roughly 200 bytes a legitimate, authenticated, fully-read request can take theGOAWAY: a valid 27 KB_bulkchunked into 128-byte DATA frames did so repeatedly across runs, and a 307 KB body at 184 bytes likewise. How often depends on load and on how the client frames and drains — measurements on two machines differed in both directions, so treat the hazard as real and the frequency as unpredictable rather than as a rate. Tracked as #485. If you control the client, chunk at 256 bytes or above. On this release only — see the Unreleased entry for #485 above, where theh2floor moves to 0.4.19 and the budget starts scaling with the connection window.XERJ has such a path:
auth_middlewareis an axum layer, so a request that fails authentication is answered401without the handler ever extracting its body (xerj-api/src/auth.rs). An unauthenticated client that streams many small frames at a node with auth enabled therefore accumulates charges nothing refunds, and the connection is eventually closed withGOAWAY. Authentication is not special here: any rejection that answers without reading the body does the same, so an authenticated client that mistypes a route and streams a small body into the 404 sees the identicalGOAWAY. Worth recognising for what it is rather than reading it as a client bug.Two
unsoundadvisories were cleared in the same pass:anyhow1.0.102 -> 1.0.104 (RUSTSEC-2026-0190,Error::downcast_mut()) andmemmap20.9.10 -> 0.9.11 (RUSTSEC-2026-0186, unchecked pointer offset), plus the yankedspin0.9.8 -> 0.9.9 (#483).- XERJ advertises
Fixed
A node whose ES-compat port was taken printed a success banner and stayed up, and
xerj autoindexthen wrote the user's documents into whichever other node owned that port. The three listeners were bound inside tasks spawned after the banner, where a bind error was logged and explicitly non-fatal, so losing one port was survivable. The process that does own the port answersGET /_cluster/healthwithgreen— the readiness probe the docs prescribe — so nothing downstream could tell the difference. Found by three independent agents in a field study; one of them wrote 905 files, including contracts, invoices and a bank export, into a stranger's data directory and gotok=true exit=0for it. Every listener is now bound before the banner, a refused bind ends startup with the port and the reason on stderr, and the banner is printed from the addresses the kernel actually returned (#465, #466).A
keywordfield holding a JSON array was flattened into one token, and the answer changed at_flush. Two independent causes, which is why the symptom looks inconsistent. In a segment the array was joined into a single string before the FTS layer saw it, and the keyword analyzer emits its whole input as one token — so every clause that projects to a whole-valueFtsQuery::Termmissed the document, the first element included; measured on{"tags":["red","blue"]}, after a flush bothredandbluereturned 0 hits and only the joined artefact"red blue"matched. The type name is load-bearing:termsreads like such a clause but does not compile to one — it routes to the doc-values path, which already bailed on array fields, and it returned the document correctly on both sides of the flush. Before the flush a different cause applied: the memtable's single-valued columns keep only element 0, and the fused columnar walk did not bail out on such a field the way its sibling paths did. That walk servestermandrange, andboolwithmustorfilter, throughconstant_scoreand boosted wrappers — so abool/filterquery was affected too, not only a bareterm. Thereredmatched andbluedid not. The first-element-survives shape people remember is the pre-flush one and does not describe a flushed index. Array elements are now indexed as N independent values separated by a position gap, so a phrase query cannot span two elements, and the columnar path refuses an array-valued field rather than silently reading only its first element (#332, #470).The audit log recorded no writes and could not name the actor, and
/_auditwas readable by any credential. Create, index, update, bulk and delete appended nothing; the one audited data-path op recorded the literal"anonymous"for an authenticated admin call (#329, #471).boolscoring: a scoring-irrelevant clause collapsed_scoreand reordered the page — partially fixed. Afilter, a secondmust, or amust_notchanged relevance scores and hit order, with no error and no warning. Three defects of the same shape, all of them a clause or a pass affecting scoring when it must not:bool.filterwas projected onto the FTSmustslot so its BM25 landed in_score; the IDF rescore countedfilter/must_nottoward its trigger; and that rescore derived IDF fromfinal_hits.len(), which made_scorea function ofsize.Read the scope before you re-baseline. This fix is partial and #361 remains open. What is fixed: a
filterormust_notcarrying aterm-shaped child, on a page served entirely by the segment FTS path. What is not:filter: [{match_all: {}}]andfilter: [{exists: ...}]still return a flat score over unrelated documents, and any page that also carries memtable or stored-scan hits keeps rc.17 behaviour — that fallback is IDF-less (1 + ln(1 + tf)) and was deliberately untouched. On those shapes_scorestill varies withsize; measured on rc.18, one query returned four different top-1 documents atsize2 / 5 / 10 / 50.Where it does apply, this changes
_scorevalues and hit ordering versus rc.17. If you have pinned expected scores, recorded relevance baselines, or tests asserting an exact order, re-baseline them on rc.18. The new behaviour is the correct one — Lucene draws the same line atBooleanClause.isScoring()— but a silent ranking change is exactly the kind of thing that should not arrive unannounced (#361, #387).xc.pycould not tell a corpus that was never loaded from a query that genuinely matched nothing (#476).An alias 404'd on
_refresh,_forcemerge,_cache/clearand_terms_enum.resolve_indices_for_ophad no alias branch, so a valid alias was simply "not an index". It now expands to every member rather than the first — a_refreshreaching one of three members makes the next search non-deterministic with nothing in the response saying so (#459). The remaining read-path half — an alias answered as an alias over one member — is still open (#449).The air-gapped deployment recipe failed open: a bad digest extracted and installed anyway. The verify step did not stop extraction, and because
set -eusat at the top of the block rather than inside it, a failure killed the operator's shell — so the natural recovery, which the page itself named, was to re-paste without it, and that made the digest check non-fatal. Every block that decides something now runs inside its own subshell, the digest is computed locally and asserted against the.sha256withgrep -qxF, and the block that verifies is the block that installs (#441).The install page's checksum step never hashed the archive. It ran
sha256sum -c, which reports success for whatever filenames the.sha256happens to list, skips#comment lines silently, and never checks that any line names the archive being extracted. A.sha256carrying a comment naming the archive plus a valid digest for an unrelated file verified clean at exit 0 while the archive was never hashed. Both the install page andllms.txtnow compute the digest, assert it is 64 hex characters, demand that exact line, and chain extraction and install to the result (#444, #452).A text file whose first two characters were
BM, or whose first four wereGIF8, was junked as an image and never indexed. They were the only two entirely-printable signatures in the magic table accepted without a qualifier — the scoping matters, and the rest of this entry says why — sosniffreturnedFamily::Binaryandscan_fileskipped the file citing "binary content (gif)" / "(bmp)" — a reason naming a format the file did not have. The reported case is a CSV whose first column header isBMW; a health export headedBMIfails the same way, as does prose beginning "BM" or "GIF8". rc.17 shipped this knowingly and said so twice, pinning the wrong answers ingif8_and_bm_are_still_taken_on_faithso the deferral stayed visible. That test is replaced bytext_that_opens_with_gif8_or_bm_is_text, andevery_printable_signature_carries_a_qualifiernow walks every row of the signature table so the defect cannot return through that table. It is not the whole class:%PDF-is matched earlier, is equally printable, and prose opening with it is still handed to the PDF extractor — deliberately pinned and still open as #403.A corpus indexed before this release will not necessarily pick the file up on a re-run. Two things decide it and they are easy to confuse: the indexing path decides whether the file is re-sniffed at all, and the state dir decides whether anything can then accept it. The cases differ enough that the only guidance worth giving is: check the exit code, and read the error if there is one.
On the default (graph) path the frozen plan is reused instead of re-sniffed, so a file recorded in
plan.junk_filesis never reconsidered: the re-run leaves the index unchanged and exits 3 (completed-with-junk), the same code the original run gave — there is no signal that anything is now recoverable.With
--no-graphthe file is re-sniffed and then has to be re-admitted, which is decided by four gates inclassify_new(reconcile_plan.rs:279-323) — the family/group filter,ensure_compatible, a field-overlap threshold, and a refusal when two frozen datasets match equally well — which between them select one of three outcomes. All three are reachable — the file is recovered in place, or the run aborts naming it, or it stays junk — and which one you get depends on what the other files in the corpus froze into the plan. This entry deliberately does not predict it: six attempts to state the rule concisely were each refuted by measurement, and the honest instruction is to run it and read the exit code. Acsvfile rejoining a compatiblecsvdataset is recovered incrementally in seconds; prose never is, because prose datasets recordfamily = docswhile the file sniffstxt-prose, so the first gate can never match.A new
--state-dirwith a new--prefixrecovers the file in every configuration measured.--freshrecovers it in some and is refused in others — the refusal tracks what the state dir contains, not which flag you pass. Both re-extract the whole corpus and re-embed it on a semantic index. The new-prefix route also leaves the old prefix's indices live and serving until you delete them by hand, so budget for two copies of the corpus in the interim. The full matrix, including the cross-path cases where a corpus indexed one way is re-run the other, is #490.A residual class stays
Family::Binaryregardless: the GIF arm junks a file whose prefix fills the sniff buffer or whose bytes cannot be written as a canvas dimension, and the second disjunct has no size floor — the release's own tests pin a 3,000-byte prose file junked as an image under a 1 MiB budget. Describing the cost as "large files only" would name one half of it (#379, #380, #427).A GIF whose comment chain outruns the sniff budget is decided by the chain, not by a canvas heuristic (#427, #442).
Added
xerj feedback— one command drafts the field report the project asks every agent to file, auto-filling version, OS and what was indexed;xerj feedback --open-prstages that report underuser-feedback/16-agent-field-reports/and opens the PR — it stages the report but does not restrict the commit to it, so checkgit statusfirst if you have other work staged. Field-report PRs are CLA-exempt, so they merge without a signature (#473).A build gate against release notes that promise the next release. A file shipping inside the release tag may no longer describe a capability as arriving later — that sentence cannot be checked at review time and is wrong by construction afterwards, whichever way it resolves. Both halves of that shipped during this cycle. The gate is a list of known phrasings rather than a rule about meaning, and says so; #474 tracks what would close the class (#474, #479).
word_delimiter/word_delimiter_graphtoken filters and acodeanalyzer, so an identifier can be matched by its sub-words (getHTTPResponse->get/HTTP/Response) while the whole identifier is preserved for exact matches. Capability only — no performance claim (#468).A reproducible verification harness for the neural embedder's missing batching, measuring lexical 1.3 s against neural 677.5 s on an identical 101-file slice. This is the measurement, not the fix — #366 stays open (#467).
Operational-recipe verification protocol in
docs/CONTRIBUTION_REVIEW.md, including the two clauses that would have caught the round of #441 that went green while still broken: run each negative control with the following block appended, and with the failing block's exit status discarded (#447, thanks @buger).docs/XERJ_VS_LUCENE.md— a side-by-side comparison with Lucene 10.3.1, honest in both directions (#373, #448, thanks @buger).
Changed
XERJ no longer sizes its memory budgets to the whole machine. New default: 8 GiB. Every budget derived from
effective_memory_limit_bytes(), which is min(cgroup limit, total system RAM) — so with no cgroup, the normal laptop and macOS case, a bigger machine bought a hungrier XERJ. Measured on a 121 GiB host, the derivation base was 124,609 MB, giving a 31,152 MB memtable budget and a 24,921 MB hydration cache before anything else allocated. Users reported ~20 GiB resident for two indexed projects.limits.max_process_memory_mbnow defaults to8192and caps the derivation base itself, so every dependent budget shrinks coherently. It only ever lowers: a 4 GiB laptop stays 4 GiB and a smaller cgroup limit still wins. Set0to restore the previous machine-proportional behaviour on a dedicated box (#461).Read this before relying on it as a ceiling. It caps the budgets derived from machine size. It is NOT an RSS limit, and an audit of this release measured a node configured with exactly these budgets reaching 7,924 MB resident while the capped caches held 873 MB — the governed budgets cover under half of peak RSS, and the rest is merge-path allocation, aggregation workspace and allocator retention that consult no budget at all. On a large corpus the cap can therefore surface as an HTTP 429 on ingest rather than as flat memory. If you index a large project and see 429
circuit_breaking_exception, raiselimits.max_process_memory_mb. Reducing the memory that is outside these budgets is follow-up work, not something this release completes.The README's lead was rewritten three times in this release; the third is what ships. #457 moved the lead from reference coding to
autoindex, on the argument that reference coding is one use case standing in for the whole product, and changedllms.txtthe same way. #472 moved it back to reference coding with its measured 2.7x-fewer-output-tokens result at an equal solve rate, and added a community CTA. #478 then replaced the lead again with a paste-to-agent prompt — "One prompt, and your AI agent installs XERJ, indexes your code, and reads the exact implementation" — and removed the top-level## Installsection, demoting installation to a## Install by handsection further down.So the net change versus rc.17 is the #478 shape, not #472's, and rc.18 does not lead install-first. The badge and body corrections from #457 survive all three (#457, #472, #478).
Capability badges corrected. The default embedder is lexical and offline; neural is opt-in and downloads ~90 MB on first use. The previous pair (
semanticbesidebuilt-in, offline) described a configuration that does not exist, and the README body said so correctly eighty lines below. The release badge no longer carriesinclude_prereleases&sort=semver. Recording a correction to this entry rather than restating it: neither parameter is actually broken for this repository's tags — all four combinations, the pre-#455 URL included, renderv1.0.0-rc.17against the live endpoint. The parameters were dropped as unnecessary, not as a fix (#455).autoindex --follow-symlinksno longer follows a link out of the indexed folder unless you ask it to. Pointing autoindex at a folder is not consent to index whatever that folder links to, and the hidden-file rule — the thing keeping.env,.sshand.awsout of a queryable brain — was defeated entirely by a visible link name:notes.txt -> .secretdir/k.txtindexed the secret, andshared -> /etcindexed/etc/shadowundershared/shadow, while the run reported that the hidden directories had been pruned (#438).A followed link is now judged by what it resolves to, not by the name it wears. Targets outside the folder are refused and reported under a new
symlink:outside-rootrule; targets inside a hidden directory are skipped like any dotfile; and one that does both — left the folder AND resolved through a dotted component — is reported assymlink:outside-root+hidden, because it is refused even with the opt-in below and neither of the other two labels tells the operator anything they can act on.This will index less than the previous release for some setups, and the first run after upgrading drops what it previously indexed through such a link. If following links outward is why you turned the flag on — a vendored sibling checkout, a monorepo package link, a mounted volume — pass
--follow-symlinks-outside-rootto restore it (it requires--follow-symlinks, and is refused on its own rather than accepted as a silent no-op). That waives the root boundary and nothing else: the hidden-file rule still applies to the resolved path, judged from the point where the target diverges from the folder you pointed at — sonotes -> ../sibling/pkgis followed andkeys -> ../sibling/.ssh/id_rsais not, while a dotted directory the two paths SHARE (a/tmp/.tmpXXXX, a checkout under~/.local) does not refuse anything, because your own folder is already inside it.Known limit, stated rather than implied: this is a rule about names and about where a link resolves. A hard link has neither, so a visible name hard- linked to a file inside a hidden directory is still indexed, on this release and on every earlier one.
Fixed
A multi-index
scrollreported the wrong_indexon every continuation page, so(_index, _id)stopped being distinct (#414). The scroll context stored bare hits plus one context-levelindex, and every page after the first stamped that single name onto every hit. Page one was correct, because it maps the real per-hit index, so the divergence only appeared once paging crossed into the second index.The two context-creation sites stored different wrong values, which is why the symptom has two faces:
search_impl(/{a,b}/_search?scroll=) storedindex_names.first(), so hits were labelled with the first concrete index (mi_a), whilesearch_with_scroll(/{spec}/_search_scroll) stored the raw path spec, so hits were labelled with the un-split string — the_index: "sa,sb"that #414 reported, 11,900 of 12,000 hits.Ids routinely collide across indices (each numbers its own documents), so a consumer keyed on
(_index, _id)silently kept a fraction of the corpus. Measured on two 300-document indices with identical id ranges: 600 hits collapsed to 300 distinct(_index, _id)pairs, all labelled with the first index.search_afterover the same two indices was unaffected and returned 600 of 600 — the two paging APIs disagreed about which index a document came from. The blast radius is reindex, migration, backup and CDC consumers, which are exactly the readers that use scroll and least survive losing half a corpus.The comment at the context-creation site asserted the opposite of the behaviour — that keeping the raw index spec left per-hit
_index"authoritative when paging" — which is why the defect outlived review.ScrollContext::hitsis nowVec<(String, Hit)>, so a hit carries the index it came from rather than borrowing one from the context. Both context-creation sites previously discarded it with the same.map(|(_, h)| h.clone()).Scope, established by an independent adversarial verification of the fix and stated here rather than discovered later. This covers scrolls addressed by a comma list (
/a,b/_search?scroll=), a wildcard, or_all. It does not cover a scroll addressed through an alias spanning several indices: alias names are not expanded by the index resolver, so every hit still reports the alias and(_index, _id)remains non-distinct there. That matters because reindex and CDC tooling routinely scrolls an alias, which is exactly the audience named above — tracked separately rather than folded into this entry (#433). The type change is also a breaking one for any out-of-tree reader of the publicScrollContext::hitsfield.
Documentation
A three-model reference-coding benchmark, and a number that does not match the headline.
demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.mdmeasures XERJ at ~1.4x fewer output tokens than grep, consistent across all three models, plus a turnkey SWE-bench harness undertools/xerj-code/swebench/. The README,llms.txtand the badge headline 2.7x, from the earlier single-model case study. Both are real measurements of different task sets; neither supersedes the other, and the 1.4x figure is the one measured across models. If you are choosing a number to quote, quote that one and say what it was measured on (#480).Withdrew security and supply-chain claims the build does not back. The security page's air-gap row moved from
GAtoDOCSand itsXERJ_AIRGAP=1 disables all telemetryline was removed — that variable exists nowhere in the tree. TheSBOM · SUPPLY CHAINsection, which promised a CycloneDX SBOM with every release, SLSA provenance, signed git tags and release archives, cosign-compatible signatures and reproducible builds, was replaced by aRELEASE INTEGRITYsection stating what actually ships: a.sha256, which is a checksum and not a signature or an attestation. The public-sector page lost itsSBOM · SLSAcompliance row and one at-rest/BYOK line. Others on that page survive — two of them carrying no status marker at all — so the withdrawal is partial, not complete (#491). A verified air-gapped deployment recipe was added in their place (#430).The 10,000-document scroll snapshot cap is now published, and pinned to the constant that enforces it (#370). XERJ's
_search?scroll=is a bounded up-front snapshot, not a segment-walking cursor, so a query whose exact result set exceedsSCROLL_SNAPSHOT_MAX_HITSis refused with a 400 rather than paged and silently truncated (#198). The refusal is the right behaviour and its message already namessearch_afteras the way past it — but the ceiling appeared in no user-facing document, and scroll is precisely what the ES ecosystem reaches for to read a whole index (elasticsearch-py'shelpers.scan(), reindex internals, export and backup tooling), where the result set is normally larger than the cap. "Scroll is supported, with a 10,000-document snapshot cap; usesearch_afterbeyond that" is a different compatibility claim from "scroll is supported", and only the first one is true.The cap, the 400 it produces and the
search_afteralternative are now stated onlanding/docs/api-es-compat.html(which also gains the three scroll endpoints it never listed), in the "what needs rewriting" list onlanding/docs/migration-from-es.html, and in the honesty/capability-boundary section oflanding/llms-full.txt. Every published figure sits in a<!-- generated:scroll-snapshot-cap -->region checked againstxerj_api::es_compat::SCROLL_SNAPSHOT_MAX_HITSbyxerj-api/tests/scroll_snapshot_cap_is_documented.rs, so changing the constant now fails the build until the pages follow — the mechanism already used for the capability counts indocs_capability_lists.rs. The same test drives the real ES-compat router over a corpus one document past the cap and reads the enforced number back out of the 400 the server produced, so the published number is checked against the server's behaviour rather than against a second hand-written copy of it.The published escape hatch is now executed, not just spell-checked. A pinned number is worth nothing if the recipe past it is wrong, and the first draft of this page's
search_aftertranscript was: it paged withsearch_after: ["999"]against 11,450 numeric-looking_ids._idis a keyword and sorts lexicographically, so"1000" < "999"and page 1 of 1,000 actually ends at_id"10897". Followed exactly as printed, including the page's own stop rule ("until a page comes back empty"), that walk collected 1,010 of 11,450 documents — 10,440 missing, every response a 200 — which is the silently-truncated export that #198 and this very page exist to rule out. The transcript now publishes the cursor the response actually returns, shows the hit it comes from, and states the lexicographic ordering that produced the mistake.published_search_after_recipe_walks_the_whole_corpusparses the transcript out of the page, seeds the corpus the transcript itself names, replays the printed paging request bodies (the two carryingsort) against the real router, and asserts the walk collects every seeded_idover the page count the transcript claims — so a wrong cursor, an early stop, a repeated page or a drifted claim all fail the build.Three further scroll claims elsewhere in the tree were corrected to match:
landing/agent-search/index.htmlsaid scroll was a deep-pagination cursor;landing/llms-full.txt§6 andREADME.mdcarried the bare "scroll is supported" form; and the ES→XERJ migration recipe indocs/recipes/production-deployment.mdtold readers they could point a whole-index scroll copy at a XERJ source with no mention of the ceiling.One boundary is documented rather than fixed here: the cap is applied to the request's summed total on
POST /{index}/_search?scroll=(es_compat.rs:14229) but per index on thePOST /{index}/_search_scrollalias (es_compat.rs:19808), so a multi-index scroll on that route can snapshot the ceiling from each index. That direction is permissive, not lossy. The pages now say so and point at #405, where the engine inconsistency is tracked.Verified by
cargo test -p xerj-api --test scroll_snapshot_cap_is_documented— 3 passed in theci-testprofile (8.67s, overflow-checks off) and 3 passed indev(12.40s, overflow-checks on). It drivesbuild_es_compat_routerover tempdir engines: a 10,001-document index (SCROLL_SNAPSHOT_MAX_HITS + 1) refuses the scroll open with400 illegal_argument_exceptionnaming[10001]and[10000]; a 128-document control scroll returns a_scroll_id; the printed first page returns 1,000 hits ending at_id"10897"withsort: ["10897"]; and the published walk from that cursor returns 11,450 distinct_ids over 12 pages with none missing and none repeated. Reverting only the transcript hunk to the["999"]form fails the same test withleft: Array [String("999")] / right: Array [String("10897")], and with the cursor assertion inverted so the walk still runs, with "collected 1010 of 11450 documents over 2 pages, silently missing 10440". The_search_scrolldivergence above was measured on two 6,000-document indices in one engine:POST /sa,sb/_search?scroll=1m→ 400 naming[12000]/[10000];POST /sa,sb/_search_scroll?scroll=1m→ 200 with a_scroll_idandhits.total {value: 12000, relation: "eq"}.
Fixed
xerj autoindexnow skips hidden names that are not valid UTF-8. The walker already refused.env,.ssh, and.git, but the check usedfile_name().to_str().starts_with('.'). On Unix a name whose first byte is.and that is not valid UTF-8 (to_str()isNone) was walked and indexed. The skip now compares the first encoded byte to.. Typical UTF-8 secrets were already skipped.--dry-run'signored_files_in_pruned_dirscount uses the same predicate, so a hidden non-UTF-8 file (or the contents of a hidden non-UTF-8 directory) is no longer reported as a non-hidden file inside a pruned directory.Upgrade. A corpus indexed by a pre-fix binary that contains a hidden non-UTF-8 name looks like a file removal to the graph-enabled path (the default, and what
xerj brainuses). As of #439's first fix the rerun's refusal names the real cause — the file is still on disk, excluded, not removed — instead of the misleading "restore the removed file(s)". It still exits 1 withunsupported_content_group_removal, writes nothing, and the already-indexed documents still stay live; sweeping them (so a widened exclusion removes the data) is tracked in #589. Recovery that removes the data today: delete the published indices and the state directory, then re-index the folder. An isolated rebuild with a new--state-dir/--prefixalso exits 0, but leaves the old target's documents live until those indices are deleted by hand.Autoindex snapshot and replay one-shot failpoints can no longer be stolen by unrelated parallel tests. (#385). The two test-only failpoints are now owned by the thread that arms them, while retaining their existing one-shot behavior and serialization locks.
Console delete/upsert no longer reports success after a refused write.
DELETE /_xerj-console/api/v1/views/{id}swalloweddelete_documenterrors and returned204while the view stayed on disk (the same class as the passkey revoke fix). Auth and prefs writes used delete-then-create, so a replacement the engine rejected (write block, unparseable date) dropped the only live user, session, token, or prefs row. Deletes now propagate the engine error; upserts use a singleindex_document.sorton an ES meta-field other than_score/_doc/_idresolved tonullon every hit (#401).compute_sort_valuesspecial-cased exactly those three keys and sent everything else toget_field_value(source, field)— a lookup inside_source._seq_no,_version,_primary_termand_indexare engine metadata and are never literal_sourcekeys, so the lookup missed on every document and the whole result set tied on[null]. The damage lands on keyset pagination:search_after: [null]is never strictly greater than the next hit's[null], sosort: [{"_seq_no":"asc"}]+search_after— the consistent full-scan pattern used for migrations — either re-reads page one forever or stops after it, with no error either way. Measured on a 30-document index before the fix: 4 ids collected out of 30, page two empty.These fields now resolve through the same version-map accessors that populate the response meta-fields (
lookup_seq_no/lookup_version), so a hit'ssortvalue and its_seq_no/_versionagree._primary_termis the constant1the hit meta-field already emits and_indexis the index name; both are constant within one index, so they tie like any other constant sort key and a client paginating on them still needs a tie-breaker. Sorting on_seq_nounderindex.disable_sequence_numbersis unchanged — es_compat rejects it at the API edge before the request reaches the engine.Real sort values are only half of it: the memtable's bounded sort-candidate path narrows the heap's input using a doc-values column keyed by the sort field, which for a meta-field is derived from
_sourcerather than from the version map the heap now ranks on. With no such key present it classified every buffered document as "missing the field" and returned an arbitrarymaterialisation_limit-sized prefix. A correctly ranked page over a wrong candidate set is still a silently wrong page, so that path — and the segment-side sort shadow — now decline meta-fields and take the full walk. The same gate fixes an independent instance of the identical defect on_id, which already resolved to a real sort value: over a 2 000-document memtablesort: [{"_id":"desc"}]returned["d1995","d1989","d1981","d1973","d1961"]instead of["d1999","d1998","d1997","d1996","d1995"].There is a third such prefilter, and it is the one the two gates above route meta-sorts onto: the memtable scan's pre-clone rejection (
memtable_primary_key_rejects) skips a document whose primary sort key, read from_source, already loses to the full heap. Once the heap ranks on version-map metadata, a document carrying a_sourcekey of the same name is compared against the wrong frontier and dropped before the heap ever sees it. Measured over ES-compat HTTP on 2 000 documents where onlyd0000carried"_seq_no": 10000000:{"size":5,"sort":[{"_seq_no":"asc"}]}returned[d0001, d0002, d0003, d0004, d0005], omitting the true first document, while"size": 3000returned it — a silent, size-dependent missing document. That path now declines meta-fields too.Behaviour change:
sorton_seq_no/_version/_primary_term/_indexnow ignores a_sourcekey of the same name and always ranks on engine metadata. Previously it ranked on the_sourcevalue where one existed and onnulleverywhere else. This is observable because xerj — unlike Elasticsearch, which rejects a metadata-named key inside a document with adocument_parsing_exception— accepts{"_seq_no": 1}in a document body and echoes it back in_source. Such a key is now inert for sorting; it is still stored, still returned in_source, and still usable for ordinary queries. Rejecting metadata-named_sourcekeys at the API edge is deliberately not part of this change — xerj writes_routinginto_sourceitself, so the edge rule needs its own design.xerj autoindexaborted a whole run when one file declared two or more SQL tables (#360). One file can feed several datasets — a SQL dump is one file and N tables — but the sealed record count was a single whole-file total. Finalisation charged that total to each of the file's datasets in turn and compared it against that one dataset's read-back, so any multi-table file disagreed with itself by construction and exited1withdataset t1 exact read-back count 2 disagrees with sealed count 4. The rows were always indexed correctly; only the reconciliation arithmetic was wrong, soexit=1discarded a complete corpus. Reported against a realunum-cloud/usearchfile —sqlite/README.md, ordinary prose with three```sqlblocks in it, which the sniffer routes to thesqldumpfamily.The sealed ledger is now keyed by the dataset identity each record was written under (
records_by_dataseton the prepared artifact,expected_records_by_dataseton the manifest group), the read-back is scoped byax_datasetas well asax_file, andvalidate_groupsrejects a ledger that names a dataset the group does not feed or that does not sum to the group total. Measured against a local node with the fixed binary: the two-table and three-table repros from the issue exit0, every table's rows are present (_count2 per table), and each dataset's catalog document reportsrecord_count: 2instead of the whole file's total.Existing state directories survive the upgrade: both new fields are omitted from serialization when empty, so a snapshot or manifest sealed by an older version re-serializes to the same bytes and keeps its digest. A pre-upgrade group that fanned out has no recoverable split, so it withdraws from that one equality rather than failing it — the state directory left behind by the aborted repro run resumes and commits on the fixed binary.
xerj autoindexaborted a run because another corpus on the same node had already indexed a byte-identical file — the second of the three aborts #360 reports, and the one that killed the reporter's second corpus. A file's identity is derived from its CONTENT alone (content::full_digest) and the catalog is one global index (autoindex-catalog) that no--prefixnamespaces, so two unrelated checkouts sharing one file — an Apache-2.0LICENSEis the everyday case — share that file's catalog document IDs. A corpus holding the content five times publishes a canonical document plus four alias documents; a corpus holding it once republishes only the canonical, and the other corpus's four alias documents survive under their ownrun_id. Be clear about what that does not mean: the shared file's canonical catalog document (file:<content_id>) belongs to whichever run published it last, so after the second corpus runs, the first corpus's alias documents point at aduplicate_ofwhose canonical entry is now the other corpus's. That is pre-existing — the binary before this change overwrites it too, then aborts — and this change removes the false abort without repairing it. The catalog identity collision itself is tracked in #416. The generation-wide barrier counted every catalog document carrying the group'sfile_key— across every run on the node — against1 + aliases.len()for the group in front of it, so the second run exited1withcatalog canonical/alias count disagrees with desired group axg1-…on a corpus whose data was complete. The count is now scoped to this generation'srun_id, the same way the run-summary read-back already is, so the barrier asks what it exists to ask: did this generation publish one canonical document and its aliases.Measured on the reporter's own corpora, one node, release binary,
aarch64-unknown-linux-gnu.apache/luceneandunum-cloud/usearchshare exactly one indexable content identity — the 11,357-byte Apache-2.0LICENSE, present 5× in lucene and 1× in usearch. Indexing lucene and then usearch on one node, each with its own--prefixand--state-dir: before, usearch exited1after 41.5s withcatalog canonical/alias count disagrees with desired group axg1-f5c714bcb8c945dd3687e83218352c66. After, the same usearch run printsgeneration 1 committed — 4 datasets, 308 records liveand exits3(junk recorded, never fatal —cli.rsEXIT CODES) in 30.2s, and the 43 alias documents lucene published are still in the catalog: the barrier stopped counting them, it does not delete them.This is not what closed #360 — that closed
completedon 2026-08-15 via #407. The third abort the issue reports —catalog read-back for ds:… disagrees with the sealed generation projection, which ended the fullapache/lucenerun after 1,073s before this change and 663s after it — reproduces on this branch and is not an autoindex defect:POST /_bulkdrops explicitnullfields from_sourceonce the request is large enough (measured directly withcurlagainst this binary: 1,500 documents / 129,786 B keeps them, 4,000 documents / 349,786 B loses them, 2,000 documents is where it starts to be mixed), and autoindex's dataset documents carry"time_field": null,"time_min": null,"time_max": nulland"semantic_field": null. The exact read-back is the tripwire that caught the engine rewriting a document, so it is deliberately left armed rather than taught to ignore the difference. Tracked as #415.A lexical query on a
semantic_textfield now says so (#363).match,match_phrase,multi_matchandquery_stringagainst asemantic_textfield score with BM25 over the analysed text and never consult the embedding that field was indexed to produce — the response was byte-identical to the same query against a plaintextfield, same ids and same scores, so a caller had no way to tell which question had been answered._searchresponses now carry an additive_xerj.hintsentry (code: "lexical_on_semantic_text") that names the field, says the embedding was not consulted, and gives a ready-to-pastesemanticquery body. Scoring, hits and every ES-compat response field are unchanged, and the hint stays quiet when the same request already reaches the vector throughsemanticorknn(including the ES 8.x top-levelknnblock).A
quantization: "scalar8"(int8_hnsw) vector field scored updated documents from stale quantized data, silently, forever (#371). There were two write-once caches on this path, and either one alone reproduces the reported symptom.- The per-field SQ8 code store was populated write-once: the first kNN that observed a document computed its codes and nothing ever recomputed them — not an update, not a delete-and-reindex, not a merge.
- The per-field codebook (
Sq8Params, the per-dimension min/scale pair) was fitted from the first <=1000 candidate vectors the field was ever scanned with and then kept for the life of the process. A vector written afterwards that falls outside that fitted range is clamped into it, and when the fitted range is narrow the clamped decode is indistinguishable from the vector the document used to hold.
Both were reproduced at the HTTP boundary before the fix, each against a byte-identical full-precision control index that performs the identical manipulation:
- codes — 8-dimension cosine field, doc
0starts as the query vector and is rewritten with its exact negation. The control drops it as it must; thescalar8index returned it at_score0.99999774, the same value to the last bit as before the rewrite, with the entire 20-document score list unchanged, whileGET /{index}/_doc/0confirmed_sourcehad taken the negation.PUT /{index}/_doc/{id}and_bulkwere both affected;_bulkwas untested in the original report. - codebook — same shape, but over a corpus whose dimension 0 never leaves
+1.0, so it fits the degenerate range[1,1]. Doc0negated to-1.0decodes straight back to+1.0: still first at_score1.000000, the whole score list frozen. On the workload the issue names as what makes it a blocker — re-embedding a corpus after a model change — a codebook fitted to the old embedding space clamped 50 re-embedded documents into a 0.002-wide score band, destroying ranking outright.
The fix removes both caches rather than adding invalidation to them. The brute-force scan already holds each candidate's live f32 vector — it just read it out of
_sourceto apply the filter — so it now fits the codebook over exactly the candidates it is about to score and quantizes/dequantizes each one on the spot, through a newSq8Params::encode_intoandSq8Params::fit_borrowed, into two buffers reused across the scan (no per-document allocation). No SQ8 state of any kind now outlives a query, so there is nothing left to go stale. Fitting over exactly the set being encoded additionally makes clamping impossible on this path rather than merely unlikely: every value passed toencode_intois inside[min,max]by construction.scalar8still scores from 1-byte-per-dimension codes, so its recall profile is unchanged — measured recall@10 0.998 against the exact float32 index on thevector-quantizationrecipe corpus, byte-identical scores to before this change on that (unfiltered, <1000-doc, never-updated) shape.Three defects in the removed objects went with them: unbounded growth (the code map had no eviction and no upper bound — one entry per document ever queried, for the life of the process), an exclusive write lock taken on the query path on every kNN over the field, and a codebook that could be fitted from an empty candidate set — a single query with a wrong-length vector installed an all-zero-scale codec that every later score on that field collapsed onto. That last one is now structurally absent rather than guarded: a codebook that does not outlive its query cannot be observed by another.
Not closed by this change, tracked in #392:
scalar8still reads the full-precision vector from_sourceon every query, so it does not reduce resident memory — it buys the precision profile of int8 and nothing else. Making it a memory win means the ordinal-addressed, ingest-time code array described in #371, written next to the data it describes as Lucene does it.The "~4x smaller vector working set" wording has been corrected in every place this change could find it, listed exhaustively so the claim is checkable rather than taken on trust. Docs and shipped artifacts:
docs/recipes/vector-quantization.md,docs/recipes/README.md,docs/examples/vector-quantization/quant_demo.py,recipes/vector_quantization.py,engine/xerj.default.toml,xerj-common/src/config.rs,xerj-common/src/types.rs,xerj-vector/src/quantizer.rs,demo/playbooks/ES_COMPATIBILITY.md,demo/playbooks/STUB_AUDIT.md,demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md. Published site (landing/docs/*.htmlis the hand-maintained mirror ofdocs/*.md):landing/docs/recipes/vector-quantization.html,landing/docs/recipes/index.html,landing/docs/index.html,landing/docs/vectors.html,landing/docs/config.html(both the reference table and the example TOML),landing/docs/operations.html(the RAM capacity-planning line),landing/docs/playbooks/vector-search.html, plus the twodocs-indexsearch snippets that are duplicated into all 44 docs pages. Marketing copy:landing/solutions/index.html,landing/pricing/index.html,landing/resources/index.html,landing/industries/finserv.html,landing/agent-search/index.html,landing/use-cases/ai-search-retrieval.html,landing/product.html,landing/llms-full.txt.Deliberately NOT corrected here, because replacing them needs numbers this change cannot supply: the interactive capacity model on
landing/industries/retail.html(lines 82/93/177) and the matching row onlanding/industries/healthcare.html:188size a deployment from an SQ8 vector footprint (e.g. "18 GB (SQ8)" vs "92 GB (float32)"), as does the "10 M SKUs ON ONE NODE · SQ8" card atlanding/industries/index.html:123. Those are product claims with a whole page built on them and they are wrong until #392 lands; they need replacement figures and sign-off, not a silent edit. Also left as-is: the dated review recordengine/reports/FEATURE_FAIRNESS_REVIEW_v0.6.0_2026-04-25.md(lines 81/272), which is an archive of what was believed on 2026-04-25, and the synthetic demo corpus article "Scalar quantization: 4x memory savings for free" (demo/data/generate_ai_kb.py:56, echoed as sample search output atdemo/DEMO_RUNBOOK.md:528), which is generic industry text in a sample knowledge base rather than a claim about XERJ.because the codebook is now fitted per query, a
scalar8_scoredepends on the candidate set — and that changes the returned order, not merely the score. Each individual score moves by at most SQ8's own quantization step (1/255 of the fitted per-dimension range), inside the approximation errorscalar8already carries, but documents whose true scores are close swap places, so a caller sees a different ranking. Measured at the HTTP boundary against this branch, on a 60-document 4-dimension cosine field:- adding
filter: {"term": {"grp": "a"}}, which removes only the 30 unrelatedgrp:bdocuments, returned the same 30 survivors with a maximum_scoredifference of 1.976e-05 but a different order at 19 of the 30 positions; - the trigger is the candidate set, not the
filterkeyword — indexing one more unrelated document moved the same 30 by up to 7.100e-06 and reordered their top 10; - on an unfiltered corpus over 1000 documents the scores also differ
from v1.0.0-rc.17, which fitted the codebook from the first ≤1000
candidates and cached it for the life of the process. On 1500 documents
all 40 of the top-40
_scorevalues changed (maximum 4.880e-05) and two adjacent ranks swapped. This is a wire-visible change on anyscalar8field with more than 1000 candidates. Below 1000 documents, unfiltered and never updated, scores are byte-identical to rc.17 — that is the shape the recipe corpus has, and the only shape the "byte-identical scores" note above covers.
None of this could happen on rc.17 once the first query pinned the codebook, so it is behaviour this change introduces; it is the price of the codebook not outliving the query. It is also not the dependency Lucene has: Lucene fits per segment at index time, so a Lucene score is a function of the index state alone, while here it is a function of the query's candidate set as well.
- adding
scalar8continues to disqualify a field from HNSW-served kNN, so opting in still costs ANN.
Cost: measured, but not resolved.
ci-testprofile, 4000 docs x 384 dims, warm, min/p10/p50 of 60 requests, three builds per side. On the scan-bound shape (k=4000) the two sides overlap and the ordering flips with the statistic — best min-of-60 was 22.66 ms on main vs 24.88 ms here, but best p10 was 29.26 ms on main vs 25.09 ms here — on a machine that produced p50 outliers up to 159 ms from background load. No direction can be claimed for that shape from these numbers. The small-kshape (k=10,num_candidates=100) was consistently marginally faster here, 0.15-0.16 ms against 0.17-0.18 ms on main, which is the per-queryRwLockread on the codec store going away. In principle this change adds one min/max pass over the candidate vectors and removes a lock acquisition; both are small against the encode/decode/similarity work the scan already does._countno longer reports documents_searchcannot return (#362).POST /{index}/_countis_searchatsize: 0, so the two are supposed to answer the same question. On atextfield they did not. Atextmapping carries no doc-values, so the term-count shortcut had no column to read and answered from the segment's FTS term dictionary instead — analysed tokens, tokenised and lowercased — while_searchresolves atermon such a field against the whole_sourcevalue. Two oracles, two questions, wrong in both directions on a flushed segment:termquery on atextfieldold _count_search{"term":{"title":"testsegmentreader.java"}}10 hits {"term":{"title":"quick"}}, value"the quick brown fox"10 hits {"term":{"title":"Doc7.java the quick brown fox"}}— the byte-exact_sourcevalue01 hit The shortcut now abandons when a segment has no doc-values column for the field, and the ordinary delete-aware scan — the code that produces the hits — answers. That shortcut is not
_count-only: its result also authorises the bounded stored scan atsize > 0, sohits.totalwas affected too. On a 200,000-document index, with atermon atextfield matching 10,000 documents,_searchatsize: 10returned its 10 hits under"total": {"value": 130}, and_countfor the same query answered0. Both now answer10000.This costs real time, and the bill is not only "the wrong spellings". Scan cost does not depend on how the term is spelled, so every
termcount on atextfield with no doc-values now pays a full scan — the byte-exact spelling included. Its cost changes exactly as much as a misspelling's does, and per the table above its value changes too, from an undercount to the truth. Measured on this branch in theci-testprofile, one segment, 20 distinct query values per shape to defeat the request cache (median of 20, same box, same corpus, onlyindex.rsswapped):_countshape, 200,000 docsbefore after termontext, no doc-values4.2 ms 356 ms same, byte-exact _sourcevalue4.1 ms (and answered 0)356 ms (answers 1)termonkeyword,doc_values: false21 µs 45 µs termonkeywordwith doc-values (control)347 µs 216 µs termonlong,doc_values: false383 ms 353 ms _searchatsize: 10,termontextmatching 10,000 docs9.4 ms (total 130)225 ms (total 10000)At 50,000 documents the first row is 2.3 ms → 81 ms. That is roughly 35× at 50 k and 85× at 200 k, but the box carried load average 70–85 from unrelated work throughout, so treat the multiple as order-of-magnitude; the number to plan around is the absolute one, about a third of a second per
_countat 200,000 documents.The middle rows are there to bound the blast radius, and they are the reassuring ones. A
keywordfield withdoc_values: falsereaches the same abandoned branch, but atermon a declaredkeywordfield still projects into the FTS tree, so the segment scan's ownterm_doc_freqshortcut answers it: its route changed, its cost class did not, and it stays sub-linear rather than becoming a scan. Alongwithdoc_values: falseis unchanged because it was already scanning — a field with no FTS dictionary never reached the removed fallback in the first place. And every count already served from a doc-values column is untouched, which is what the control row shows.One shape's
_searchpage changes, not just its total. With the shortcut gone the stored scan no longer stops after materialisingfrom + size; it walks every match and then picks the page. For an unsortedtermon atextfield with many equally-scoring matches, that changes which documents come back — measured over 2,000 identically-scoring documents atsize: 10, the ten ids differ before and after (and the total goes from80to2000). Every score involved is identical, so neither page is more correct than the other, but the change is observable. Queries with an explicitsortreturn byte-identical pages and identical totals before and after.ES compatibility, stated rather than buried. On the repaired shape this moves
_countaway from the Elasticsearch answer while making it self-consistent: ES answers{"term":{"body":"quick"}}with1against"the quick brown fox", XERJ used to answer1as well — but with zero retrievable hits — and now answers0, which is what its own_searchreturns. A count no page of hits can reproduce is a coincidence, not compatibility. What_searchshould answer for atermon an analysed field is a separate, hit-set-moving decision. It was raised as #397, which was closednot_plannedon 2026-08-15 — not abandoned, but consolidated into #423, which is open and carries #397's repro as acceptance criteria. #423 is the tracker to watch: it treats the cause rather than this symptom —doc_matches_queryevaluates buffered documents against raw_sourcewith no mapping and no analyser, while flushed documents go through the term dictionary. Until it closes, this shape has no sub-linear count: budget for the measured cost above, or avoid_countwith atermon an analysed field in a serving path.Two things this change does not do. It does not touch the other half of #423, which consolidated #362 (closed
not_plannedfor that reason) —case_insensitiveaccepted and silently dropped onprefix/wildcard, the term/prefix case-folding split #362 reported:{"term":{"title":"TestSegmentReader.java"}}still answers1while{"prefix":{"title":"TestSeg"}}answers0. And it is not the narrower fix the issue thread proposed (dropping the lowercase retry the shortcut did on a dictionary miss); that was tried and measured, and it leaves the first two rows of the first table wrong while additionally making the byte-exact{"term":{"title":"TestSegmentReader.java"}}count0where_searchreturns1.
[1.0.0-rc.17] - 2026-08-15
Added
xerj autoindexunderstands Unity projects. Text-serialized scenes, prefabs,.asset/.mat/.animfiles become one record per GameObject/Component (unity_class,unity_class_id,file_id,ref_guids,script_guid, plus the flattened body);.metasidecars become a guid↔asset-path table; MonoBehaviour records carry a denormalizedscript_class/script_pathso "which scenes use this script?" is one query. Detection is by the%YAML+%TAG !u! tag:unity3d.comheader and never by extension, so binary-serialized assets stay junk (enable Force Text serialization). Unity's generated directories (Library/,Temp/,obj/,Logs/,UserSettings/) are pruned and recorded only when a siblingProjectSettings/ProjectVersion.txtproves the tree really is a Unity project.Reland of community PR #274 by @gonchar, brought up to current
mainand corrected — see Fixed below for what changed on the way in.BVH motion capture — one metadata record per clip (
joints,joint_count,frames,frame_time_s,duration_s). The numeric MOTION block, which is most of the file, is never read: extraction stops at theFrame Time:line.--stub <glob>designates files that should be referenceable but not parsed: each match is indexed as one name-card record and its contents are never opened.Push a filtered subset of indices to an external ES-compatible target — the single-node WAL tap (#320). Nothing pushed data out of the engine before this:
_ccr/*answered501, reindex-from-remote was refused up front, and snapshot/restore — scheduled and whole-index — was the only export path. A XERJ node used as a lightweight edge collector had no way to stream a curated slice of its data up to a central cluster.- New
[wal_tap]config section (10 settings, off by default) and a runtime REST surface:GET/PUT/DELETE /_xerj/wal_tapandGET /_xerj/wal_tap/_stats. The allowlist is glob-based and adjustable without a restart. That brings the documented config surface to 115 settings (counted byjourney_zero_config, not by hand). - Every
/_xerj/*route is superuser-only.PUT /_xerj/wal_tapis a whole-node export primitive —target_urlnames any host,indices: ["*"]names every index, and the tap attaches an operator-suppliedAuthorizationheader — so it is data exfiltration and authenticated SSRF in one call. Same reasoningauthz.rsalready applies to snapshot and restore: nothing here is index-scoped, so there is no per-index decision to make. The reads are covered too —GET /_xerj/wal_tapechoestarget_urland_statsenumerates the node's whole index inventory. target_urlmay not carry credentials.https://user:pass@hostis an ordinary URL thatreqwestturns into aBasic Authorizationheader — i.e.target_authby another spelling, in the one field the API echoes back and the boot log prints. It is refused at startup and byPUT, and redacted everywhere it is rendered.- The wire format is
_bulkand nothing else, so the target can be Elasticsearch, OpenSearch, or another XERJ node with nothing installed. - This is not CCR. It is one-directional, single-node, and independent of
the post-GA sharding/replication track.
_ccr/*still answers501. - The tap adds nothing to the write path: it tails WAL files from disk on a
timer and takes no lock the writer wants. A poll costs the bytes appended
since the last one, not the size of the file — the tail reader is a sliding
window over a
File, shaped after Lucene'sBufferedIndexInput.refill(lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317, Apache-2.0). A caught-up poll reads the 16-byte WAL header and stops — measured at exactly 16 bytes over a 4,176,200-byte generation, asserted throughtail_sharditself bywal::tests::a_caught_up_poll_does_not_read_the_whole_generation. (The window's 64 KiB read-ahead is deliberately not applied to that header read: it was, and it made the figure 65,536 bytes per shard per poll rather than 16.) - Delivery is at-least-once, with
version_type: externalcarrying the WALseq_noon every action so a redelivery converges on the same document the source holds. Covered against a stub that implements the actual ES external versioning rule — absent_idis created at any version, present_idis overwritten only by a strictly greater one, everything else comes back409 version_conflict_engine_exception— and asserted on both halves: four documents accepted on the first delivery, four conflicts and zero extradocs_shippedon the redelivery, watermark unchanged. Note that XERJ's own_bulkignores per-actionversion/version_type(only the single-doc path honours them), so a XERJ target degrades to last-write-wins by arrival; against Elasticsearch or OpenSearch the mechanism is live. _statscounts what the target accepted, per item, not what was rendered into the request.lag_seqis derived from the highest acceptedseq_no, and each index carries ahealthyboolean that is false both when sends are failing and when the target answers200while applying none of the actions inside it — for any reason, not only a version conflict. A target rejecting every document withmapper_parsing_exceptionis a200witherrors: true, so no send fails and the cursor advances past every dropped document;last_item_rejectionin_statscarries the target's own word for it, because "fix the mapping" and "look for a second writer" are opposite responses.- Cursor state is flushed once per poll cycle rather than once per index per poll; a deferred flush can only cause a redelivery, which the external versioning above absorbs.
- Deleting an index drops its cursor at the moment of deletion, so a
DELETE+PUTof the same name inside one poll interval — never observed as an absence at a 500 ms default — cannot leave a byte offset pointing into a WAL stream that no longer exists. For a delete that happened while the node was down, the reader treats a cursor offset past the end of its generation as the discontinuity it is (WAL files are append-only) and restarts the stream with a reported gap instead of clamping to EOF. - WAL retention is deliberately not coupled to the tap — a slow remote must
not be able to fill the local disk. A tap that falls far enough behind loses
entries and says so:
gapsin_stats, plus a warning per occurrence. Newwal_tap.min_retained_generations(default0, unchanged behaviour) buys bounded slack: it is a floor, not an Elasticsearch-style retention lease, so it holds at mostn × storage.wal_max_size_mbper WAL shard however far behind any consumer is. The floor is enforced insideIndexStore::wal_maintain_all_verified— the prune pass the engine actually runs — using the same arithmetic and the same "inside the deletion pass, not in its callers" placement as Lucene'sKeepLastNCommitsDeletionPolicy.onCommit(lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58, Apache-2.0). At the default0the loop is unchanged: no extra syscall, no behaviour change. - The retention floor is live, and the API no longer promises otherwise.
min_retained_generationsis the one[wal_tap]setting that does not live in the tap — it lives in every open index'sWalWriter, seeded at open fromEngine.config, anArc<Config>written once at boot and never mutated.PUT /_xerj/wal_taptherefore pushes it straight onto the live writers (Engine::apply_wal_retention_floor, after Lucene'sLiveIndexWriterConfigandIndexFileDeleter.revisitPolicy,lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126andIndexFileDeleter.java:516-543) and answers withretention_floor_applied_to_indices— the number of writers it reached — instead of a warning.DELETE /_xerj/wal_tapreleases it the same way, andEngine::newfolds the tap's effective configuration into the bootConfigso a store opened after a restart is seeded with the persisted floor rather than the file's. Verified over HTTP end to end (xerj-api/tests/wal_tap_retention_floor_is_applied.rs): afterPUT {"min_retained_generations": 3}an index opened before the call reads3on all 16 WAL shards, an index created after it opens with3, and both still read3after a restart whose config file still says0. - A configuration set through
PUT /_xerj/wal_tapis persisted next to the cursors and re-applied over the config file on the next boot, so "runtime config, no restart" does not quietly mean "and gone after one".DELETE /_xerj/wal_tapdrops the overlay and reverts to the file — to the value the tap kept verbatim at construction, not to a re-read ofEngine.config. The state file holdstarget_auth, so it is written0600through the same writer the API-key store uses. - The unapplied-batch health signal counts polls, not
_bulkrequests. How many chunks a poll splits into ismax_batch_bytesarithmetic: atmax_batch_bytes = 1one legitimate at-least-once redelivery of four documents produced four all-conflict_bulkresponses, tripped the three-in-a-row threshold on its own, and emittedhealthy: falsewithlast_error: "… nothing is being replicated"about a poll in which every document was already at the target. The field is nowconsecutive_unapplied_pollsand rises by at most one per poll; three unapplied polls in a row still reports the stall. - Every multiply in the retry-backoff computation saturates.
max_retry_backoff_secs.max(1) * 1000overflowed above ~1.8e16: a panic undercargo test's overflow-checks that killed the spawned tap task silently, and under a release profile a wrapped cap that can be arbitrarily small — the value18_446_744_073_709_552wrapped to a 384 ms ceiling, i.e. a retry storm aimed at an already-failing target. - Every
[wal_tap]numeric knob is range-checked at boot as well as byPUT, through one sharedWalTapConfig::check_limits. The config file was the way around the API's validation:PUT {"min_retained_generations": 100}is a400because the knob costsn × storage.wal_max_size_mbper WAL shard per index, whilexerj.tomltook the same100in silence and the node then held 100 rotated generations per shard forever. Bounds arepoll_interval_ms 50..=60000,max_retry_backoff_secs 1..=86400,min_retained_generations <= 64, andmax_batch_docs/max_batch_bytes/request_timeout_secsat least 1. Same precedent ascompression.block_size_docs(#318): an out-of-range value is a typo the operator should hear about at boot, not as a disk-full page later. - XERJ's own
.xerj*system indices are never shipped, whatever the allowlist says, and a wildcard never expands to a hidden index. - There is no backfill. A new index is shipped whole; an established one ships from the moment it is allowlisted. Seed the target with snapshot/restore first if it needs the existing documents.
- New
Changed
BREAKING —
PUT /{index}now refuses ananalysisblock XERJ cannot honour, instead of accepting it and analysing differently (#204).AnalyzerRegistry::apply_settingsis total by design (it also runs when an existing index is opened, where a hard failure would brick it), so an unresolvable tokenizer silently becamestandardand an unresolvable token filter was dropped — the index was created,GET /_settingsechoed the block back, and the field was analysed in a way nobody asked for. Index creation now validates the block and answers400with the exact constructs it cannot build.PUT /{index}/_settingswith ananalysisblock is likewise refused (it only ever changed whatGET /_settingsechoed; the registry is built once, at create/open, and there is no rebuild path — Elasticsearch also refuses analysis updates on an open index). That refusal covers every spelling this handler accepts, not just the nested one: the flat dotted{"index.analysis.analyzer.x.type": "custom"}and half-dotted{"index": {"analysis.analyzer.x.type": "custom"}}forms that ES clients routinely send used to walk straight past it into the display copy, andGET /_settingsthen echoed back an analyzer that analysed nothing.This rejects settings blocks that XERJ used to accept with a
200, and the ES-YAML conformance suite does not cover them — CI being green is not evidence that your index still creates. What is honoured:honoured (still 200)analysis.filter.*.typesynonym,length,shingle,asciifoldinganalysis.tokenizer.*.typengram,edge_ngram,patternanalysis.analyzer.*.type(non-custom)any name the built-in registry resolves — measured: standard,english,whitespace,keywordanalysis.char_filternothing — the char-filter slot is hard-coded empty analysis.normalizernothing — never built Anything else in those positions is now a
400. There is one important exception to read first: XERJ resolves a declared filter or tokenizer by name, not bytype, so a declaration whose name is a built-in is still honoured whatever its type says. Measured, all still200:{"english_stop": {"type": "stop", "stopwords": "_english_"}},{"lowercase": {"type": "lowercase"}},{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}— the canonical Elasticsearch-docsrebuilt_englishblock among them, which has a regression test pinning it. Naming a built-in analyzer on a field ("analyzer": "english") needs noanalysisblock at all and is unaffected.What now fails is a declaration under a name that does NOT resolve — which is the usual ES habit of naming filters after what they do in your index. Measured blocks that returned
200in rc.14 and return400from this release:- a
keywordfield'snormalizer(any); - a
char_filter(any, includinghtml_stripandmapping), declared either underanalysis.char_filteror as an analyzer'schar_filter; - a custom-named token filter whose
typeislowercase,stop,stemmer,snowball,kstem,porter_stem,ngram,edge_ngram,word_delimiter_graph,trim,unique,elision,decimal_digit,apostrophe,limit,truncate,pattern_replace,keyword_marker,uppercaseorreverse— e.g.{"my_lowercase": {"type": "lowercase"}}, which never lowercased anything, because the analyzer referencingmy_lowercasefound no such filter and dropped it; - a custom-named tokenizer whose
typeisstandard,whitespace,keyword,classic,uax_url_email,path_hierarchy,char_group,letter,lowercaseorsimple_pattern; - an analyzer whose non-custom
typeissimple,stop,pattern,fingerprint, or a language other thanenglish; - a
synonymfilter whosesynonymsis a bare string rather than an array (it built a filter with zero rules).
The gate covers every spelling, at create as well as at update. An
analysisdeclaration written with flat dotted keys ({"index.analysis.filter.my_lower.type": "lowercase"}) or half-dotted ones ({"index": {"analysis.analyzer.a.type": "custom"}}) is the same request as the nested form to an Elasticsearch client, and this handler already parses dotted keys for other settings (index.sort.field). The analyzer registry resolves the nested spellings only, so a dotted declaration was accepted, echoed back byGET /{index}/_settings, and applied to nothing. It is now a400naming the offending key, on bothPUT /{index}andPUT /{index}/_settings. Measured on this release, the samemy_lower/lowercasedeclaration in four spellings — nested,index.- namespaced, fully dotted, half-dotted — answers400,400,400,400; a settings body with no analysis in it (index.number_of_shards,index.sort.field) still answers200. Nothing already on disk changes meaning: the dotted form is refused, not newly honoured, so no existing index's analysis is re-derived on reopen.Upgrading: there is no opt-out flag. Either remove the analysis block and name a built-in analyzer on the field (
"analyzer": "english"), or keep only the constructs in the table. The400names every offending construct in one response, so one round-trip is enough to see the whole list. Two paths are not covered and still accept these blocks silently: an index template'stemplate.settings.analysisis dropped wholesale before the gate ever sees it, and option-level divergence inside a supported type (synonyms_path,length.min,asciifolding.preserve_original) is not checked at all.- a
BREAKING —
PUT /_ingest/pipeline/{id}refuses processor configuration XERJ cannot honour, so a pipeline that used to be acknowledged can now be a400, and a stored one can start refusing writes (#204). This is the same change as the "200 acknowledged… means we will run this" entry under Fixed below, stated from the upgrader's side, because it converts201into400for any pipeline using an ES processor or option this build does not implement. Read that entry for the full list.What is NOT broken, deliberately. A definition persisted by a build with no gate was accepted by whichever build wrote it, and there is no caller left to answer
400at boot. Boot replay therefore compiles those definitions in a compatibility mode that reproduces the previously shipped behaviour for every check this sweep added (unknowngrokpattern name, unknownconverttarget, a bare string where an array is required,set.copy_from, a processor-levelif/on_failure/ignore_failure), so an upgrade alone never turns a running cluster's writes from201into400. Each one is logged at ERROR naming the pipeline and the offending config, and recorded inEngine::degraded_pipelines; a re-PUTof the same definition answers400with the same reason, which is the repair path. A definition that did not compile before the upgrade either (unknown stage type, missing required field) still fails at boot and is recorded unrunnable, as before.Which definitions those are is decided by provenance, not by the calendar. A definition THIS build refused at
PUTtime is recorded in<data_dir>/ingest-pipeline-refusals.jsonand replayed at definition strictness, so the refusal survives a restart. Without that marker the replay could not tell "written by an older, laxer build" from "written by this build and answered400", and a plain restart overturned the answer the operator had already been given: measured,PUTa{"remove": {"field": "secret"}}pipeline, add anifguard to it (refused — every write400s), restart the node and nothing else, and the identical write answered201withsecretdropped from a document the guard EXCLUDES. It covered every check the sweep added; only an unimplemented stage survived a restart, because the raw ES body it stores does not deserialise. This is the same pattern as an index's<index>/analysis-binding.json: the marker's ABSENCE is the load-bearing signal.The marker is provenance, not a tombstone. A marked definition that compiles cleanly on the new build — because that build implements what the refusing one could not — clears its own marker at boot and runs, with an INFO line; a successful re-
PUTor aDELETEclears it too. If the file exists but cannot be read or parsed, no pipeline's provenance is known, so every persisted pipeline is replayed at definition strictness and any that fails is recorded unrunnable with the file named in the reason — fail closed and loudly, rather than silently reopening the hole. In that state a refusal is also not acknowledged: the file is written whole from the in-memory map, which a failed load leaves empty, so the next refusedPUTwould have replaced it with a one-entry map and erased every refusal it still held — after which it parses cleanly, every earlier refusal reads as "no provenance", and the fail-closed behaviour converts itself into the hole it exists to close after exactly one process lifetime.PUT /_ingest/pipeline/{id}now answers500naming the file instead, and the file is left byte-for-byte as found; move it aside and restart to clear the state. An older binary reading a data dir written by this one simply ignores the file; it is a sidecar precisely becausecluster_state.json's format-1 envelope isdeny_unknown_fieldsand widening it would break downgrades.The marker survives concurrent refusals, not just sequential ones. The sidecar is rewritten whole, so two refused
PUTs in flight at once were a lost update rather than a merge: each snapshotted the map, and whichever wrote second with the older snapshot erased the other's marker — while both callers were told{"acknowledged": true}. Measured at a live node against the first cut of this work: 40 concurrent refusedPUTs, 40 acknowledged, 37 markers on disk, and after a plain restart the three that were lost answered201and droppedsecretfrom a document theirifguard excludes. The snapshot and the write now happen under the samecluster_state_writemutex thatflush_cluster_statehas always taken. Measured on this release: 40 concurrent refusedPUTs, 40 acknowledged, 40 markers on disk, plain restart, 40 of 40 still400.The same rule applies one level up. Teaching the analysis resolver the canonical
settings.index.analysisnesting (see the analysis entry above) is a fix at index create and a data bug at index open:settings.jsonkeeps the caller's nesting verbatim, so an index created before this release with the canonical shape has postings that were written withstandard, and honouring the declaration on reopen would tokenise every new write differently from everything already on disk. An index now records the analysis binding it was created with (<index>/analysis-binding.json) and keeps it for life. An index with no marker and analyzers declared only underindex.analysiskeeps the legacy binding if it holds documents — with an ERROR naming the index and saying a reindex is the repair — and adopts the canonical binding if it is empty, recording that so a later restart cannot disagree.
Performance
A field mapped
dense_vector— and the undeclared_chunkscompanion the passage pipeline writes beside it — no longer gets a full-text term dictionary (#328). Both halves are named in that headline because the companion is where most of the bytes are: of the 39,796,380 B removed below, 26,518,005 B (67%) is<seg>.emb_chunks.fst, and in that corpusemb_chunksis not in the mapping at all — it is matched by NAME, not by anyone declaring itdense_vector. The field was fed through the lexical indexing path and given an FST + postings that no query path can read: kNN is served byhnsw/graph.bin,existsfrom_source/doc values, and_field_capsand highlighting never open the field's term dictionary.extract_field_text's array arm joined the components with spaces, so a 128-dim vector became one enormous decimal-string term per document.Measured on a 5,000-doc × 128-dim corpus (text + keyword + long +
dense_vector+ its_chunkscompanion), afterforce_merge(1), summing every file in the data directory except the write-ahead log: 54,068,549 B → 14,251,975 B (−73.6%, 3.79×). Per file,<seg>.emb.fst13,256,659 B → 0,<seg>.emb.post10,834 B → 0,<seg>.emb.norms48 B → 0,<seg>.emb_chunks.fst26,518,005 B → 0,<seg>.emb_chunks.post10,834 B → 0, while the three lexical.fstfiles (body278 B,cat53 B,n209 B) are byte-identical on both sides..walis excluded from that total because its tail is reclaimed asynchronously and it is not reproducible: on the 300-doc fixture the committed regression test uses, runs put the whole directory anywhere between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The test prints both and asserts on the durable one. It reports 3,249,323 B → 857,265–857,769 B for that fixture (before-side measured by revertingindex.rsandmemtable.rsto the merge baseca4d75aon the same fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the committed test on this revision printed 857,265 / 857,529 / 857,533 / 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and still 1.7 MB below the before-side.A
dense_vectornested under an object mapping is covered too — its components were landing in the parent's term dictionary, because the segment builder flattens the whole object into one text field. On the nested fixture<seg>.passages.fstgoes 398,558 B → 45 B, for both mapping shapes that reachFieldType::Vector(a dotted top-levelpassages.vec, and avecsub-mapping under apassagesobject).The exclusion covers the whole family a vector field generates, not just the base name:
<field>_chunks(the per-document multi-vector) and__xerj_passage_meta__<field>are excluded on the same walk that decides which fields get an HNSW graph. The companion is the bigger half — 26,518,005 B againstemb.fst's 13,256,659 B, 49% of the whole pre-change index — so a base-name-only exclusion would leave more behind than it removes.Two rules, and they do not have the same strength. The base field is excluded BY DECLARED TYPE (
dense_vector, unconditionally). The two companions are excluded BY NAME, because they arrive in_sourcewith no mapping entry behind them, so there is no declared type to read — and a name-based rule YIELDS to any declaration it finds. If you have mapped your own field called<vector>_chunksor__xerj_passage_meta__<vector>as anything at all other thandense_vector—text,keyword,long,integer,short,byte,unsigned_long,double,float,half_float,scaled_float,date,date_nanos,boolean,ip,geo_point,binary,object,nested, or a type string XERJ does not recognise — nothing about it changes: it keeps its term dictionary, every query naming it keeps answering exactly what it answered before, and only thedense_vectorbeside it loses its postings. The same holds for a field of that name XERJ mapped for you dynamically, whatever type it inferred, as long as the value is not itself a multi-vector. Measured on{text body, keyword emb_chunks, dense_vector emb}, 50 docs, halftenant-a(before → after):term {emb_chunks:"tenant-a"}25 → 25,terms25 → 25,bool{filter:[term emb_chunks]}25 → 25,bool{must:[match body],filter:[term …]}25 → 25,constant_score{filter:term}25 → 25,bool{must_not:[term emb_chunks]}25 → 25,<seg>.emb_chunks.fst54 B → 54 B,<seg>.emb.fst16,713 B → 0. Same with atextmapping:match/match_phrase/multi_match/simple_query_string/prefixall 25 → 25 and<seg>.emb_chunks.fst71 B → 71 B. Alongmapping keeps a 44 B.fstand adoublemapping a 48 B one, both 25 → 25 onterm,terms,range,bool{filter}andbool{must_not}. A field of that name XERJ mapped for you dynamically is exempt on the same rule — measured 25 → 25 for atextone (54 B.fst) and 25 → 25 for alongone (44 B.fst).What makes "yields to any declaration" affordable is that the companion is stopped from ever acquiring one.
<vector>_chunksis user-supplied_source— an array of float arrays — so dynamic mapping used to register it like any other unmapped key and inferdoublefrom it, producing a mapping entry indistinguishable from one you wrote. Dynamic mapping now declines to register a<vector>_chunkskey when adense_vectornamed<vector>is already mapped and the value is a rectangular, non-empty array of non-empty all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar, a string, a flat array or a ragged one under that name is your field and is mapped as one. Measured on{text body, dense_vector emb}withemb_chunksleft out of the mapping, 50 docs: an[[…],[…]]value gets no mapping entry,<seg>.emb_chunks.fst= 0 B and a lexical clause on it answers 0; a"tenant-a"value is mappedtext, keeps a 54 B.fstand answers 25; a7value is mappedlong, keeps a 44 B.fstand answers 25.Both halves were measured on their own, by disabling each in the tree and re-running the committed tests:
emb_chunks.fst(300-doc)durable index declared longsiblingthis release 0 B 857,265–857,769 B 44 B .fst, answers 25yield restored to long/double-excluded0 B 857,445 B 0 B, answers 0, must_notreturns 50dynamic-mapping refusal disabled 1,592,118 B 2,451,336 B 44 B, answers 25 The middle row is the behaviour rc.16 would have shipped; the bottom row is the 67% of the saving that an unconditional "any declaration wins" gives up. Neither cost is paid here.
A lexical clause that NAMES a
dense_vector(or one of its companions) is now lowered tomatch_noneat plan time rather than falling through to the stored-doc scan. That is the correctness half of the change, not only the speed half: with the postings removed and no lowering,{"multi_match":{"fields":["emb"],"query":"0"}}returns every document instead of none, because the scan renders the float array to text and every component contains a0. Measured on the corpus above, first call for each shape (main→ postings-gone-without-lowering → this release):query mainno lowering this release multi_match {fields:["emb"], query:"0"}0 hits, 0.409 ms 5,000 hits, 123.5 ms 0 hits, 0.012 ms multi_match {fields:["emb_chunks"], query:"0"}0 hits, 2.24 ms 5,000 hits, 138.5 ms 0 hits, 0.014 ms term {emb: "<component>"}0 hits, 89.2 ms 0 hits, 186.8 ms 0 hits, 0.061 ms range {emb: {gte:-2, lte:2}}0 hits, 119.8 ms 0 hits, 134.5 ms 0 hits, 0.009 ms simple_query_string {fields:["emb"]}0 hits, 0.033 ms 0 hits, 114.5 ms 0 hits, 0.015 ms constant_score {filter: term emb}0 hits, 83.0 ms 0 hits, 110.3 ms 0 hits, 0.012 ms exists {field: "emb"}5,000 hits, 83.0 ms 5,000 hits, 108.0 ms 5,000 hits, 88.5 ms match {body: "liquidity"}5,000 hits, 5.18 ms 5,000 hits, 5.96 ms 5,000 hits, 4.95 ms term {cat: "even"}2,500 hits, 2.32 ms 2,500 hits, 2.87 ms 2,500 hits, 2.27 ms exists,knnandsemanticon the field are untouched, a mixedfieldslist keeps its lexical members, and anestedquery is deliberately not rewritten (its field names resolve against the element, not the root).Four behaviour changes to know about:
A
dense_vector(and its undeclared_chunkscompanion) now has no lexical surface at all: every lexical leaf naming one outright answers 0 hits. "Outright" excludes one shape, measured rather than assumed: afieldsentry that is a PATTERN (["emb.*"]) is NOT lowered, because resolving a pattern needs the expansion universe — dynamic fields included — and lowering one that would have expanded onto a real lexical field is the silent-zero failure this release goes out of its way to avoid. So{"multi_match":{"query":"0","fields":["emb.*"]}}still answers 300 of 300 on the 300-doc fixture, exactly as onmain: the unprojectable clause falls to the stored-doc scan, which resolves the pattern against the source's field names and matches the rendered decimals. Unchanged in both columns, so not a regression — but not closed either.["emb"],["emb*"]and["*"]all answer 0. Before this release such a clause fell through to the stored-doc scan, which renders the float array back to decimal text and matches against that, so whether a shape answered 0 or answered most of the corpus depended entirely on whether the rendered decimals happened to contain the probe. This is a CLASS of changed answers, not one row. Measured on the 300-doc × 128-dim regression fixture,main→ this release:query before after wildcard {emb: "0*"}300 (every document) 0 fuzzy {emb: {value:"0", fuzziness:2}}300 (every document) 0 wildcard {emb_chunks: "0*"}300 (every document) 0 fuzzy {emb_chunks: {value:"0", fuzziness:2}}300 0 prefix {emb: "0"}146 0 prefix {emb_chunks: "0"}146 0 match_phrase_prefix {emb: "0"}50 0 term {emb: <exact component>}numeric1 0 terms {emb: [<exact component>]}numeric1 0 Shapes whose probe text never appeared in a rendered float (
termwith a string,match,match_phrase,regexp,range,simple_query_string,query_string) answered 0 on both sides. ES rejects these queries on adense_vectoroutright and Lucene gives a field with no terms an empty scorer, so 0 is the answer this release adopts for all of them. The regression test pins the whole class rather than sampling it.A
boolthat mentions adense_vectorcan change_scoreandmax_scorewithout changing its hit count. Dropping the dead clause makes the surviving bool projectable onto the inverted index, so BM25 scores it where the stored-doc scan used to. Measured on the same fixture:query hits _scoremax_scorebool{should:[term emb, term cat:"even"]}150 → 150 0.008402659 → 0.6931471 1.6931472 → 0.6931471 bool{must:[match body], must_not:[term emb]}300 → 300 0.008402659 (unchanged) 1.6931472 → 0.008402659 Both moves are toward consistency — on
mainthese queries reported amax_scorethat no returned hit carried, and the first now scores exactly like{"term":{"cat":"even"}}on its own, which is what it reduces to. Absolute scores are not part of XERJ's compatibility surface, but anything comparing scores across a version boundary or asserting onmax_scorefor a bool that touches a vector field will see this. Plain lexical queries are unaffected; the blast radius is bools with adense_vectorclause in them.GET /<index>/_mappingno longer lists a<vector>_chunksfield that XERJ mapped for you. The multi-vector companion beside a declareddense_vectorused to be registered by dynamic mapping asdouble(an array of float arrays walks to its first scalar). It is not registered at all now, so it disappears from_mappingand_field_caps— the same treatment__xerj_passage_meta__<vector>already had. Nothing that reads_sourcechanges; the values are stored and returned exactly as before, andknnandsemanticon the vector itself are untouched. A field of that name that you mapped yourself, or that XERJ mapped from a non-multi-vector value, is unaffected and still listed.Aggregating and sorting on that companion still answer
200— measured on the wire against a live node withembmappeddense_vectorandemb_chunksunregistered:termsreturns real buckets,statsreturnscount: 0byte-identical to the previous release, andsortreturns sort values. Nothing that worked before returns an error now.Existing indices keep their bloat until a merge rewrites them, and an index created by rc.16 or earlier only ever reclaims the base-vector half. This is a write-side rule. Segments written by an earlier release keep their
.emb.*/.emb_chunks.*files, and the per-segmentfts_has_fieldgate reads whichever shape it finds.POST /<index>/_forcemerge?max_num_segments=1reclaims.emb.*. It does not reclaim the companion. Measured by building an index on an rc.16 server (embmappeddense_vectorbefore ingest, 500 docs × 32 dim withemb_chunksmulti-vectors), then reopening the same data directory with this release and force-merging:.emb.fst262,904 B → 0, but.emb_chunks.fst525,200 B → 525,355 B, andwildcard {emb_chunks: "0*"}still answers 501 of 501. The reason is not a bug in the merge: rc.16 persistedemb_chunks: {"type": "double"}into the mapping, and the by-name rule correctly yields to a field you (or an earlier release) declared. Since a mapping entry cannot be dropped in place, the remediation for an existing index is to reindex into a fresh index. On the corpus this entry headlines that companion is 67% of the saving, so an in-place upgrade recovers roughly a third of the published number.The companion rule has an ordering limit of the same kind, and it fails in the safe direction: XERJ can only decline to map a companion it can recognise, and recognising one needs the
dense_vectorto be mapped before the documents arrive. Index first and add the mapping afterwards and<vector>_chunksalready carries a dynamicdoubleentry, which the by-name rule then yields to — so that index keeps the companion's term dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of reclaiming it. It costs bytes, not answers.
Known limitation, filed as #382: refusing a
FieldConfigfor the companion leaves the pre-existing schema-hash throttle asserting the key set is unchanged, so if a later document puts a differently-typed value under<vector>_chunksit stays unregistered for up to 100 documents and queries on it answer zero until the throttle expires. Doc-order-dependent and self-healing. The throttle is pre-existing and the fix belongs to it.Not changed: ES rejects
term/match/rangeon adense_vectoroutright, where XERJ answers200. This release removes bytes and adopts Lucene's zero-hit answer; it does not add the rejection.
Fixed
200 acknowledgedon an ingest pipeline now means "we will run this", and every write path enforces it (#204).PUT /_ingest/pipeline/{id}used to store the raw body first and only warn-log a compile failure, so a pipeline that could never run was acknowledged and echoed back byGET. It now compiles first, with three outcomes: compiled → stored and acknowledged; a processor config we refuse →400; a processor or option this build cannot honour → accepted (as Elasticsearch does) but recorded as unrunnable, so the write refuses rather than storing an untransformed document.Defects this surfaced, each of which answered
200/201and did something smaller than asked: every ESrenameprocessor failed to compile (the handler emitted{from, to}, the plugin requires{mappings: {…}}) and renamed nothing; ESappendwas executed asset, replacing the field instead of extending it, and disagreeing with_simulate, which implemented a real append; an ESsetprocessor did not override an existing value where Elasticsearch does (the ES→XERJ translation now writes"override": trueexplicitly when the processor omits it — the nativetype: "set"stage keeps its long-standingoverride: falsedefault, because native definitions are recompiled fromcluster_state.jsonat every boot and flipping the stage default would have changed what an already-running cluster's pipelines do to documents on a binary upgrade alone); an unknownpii_redactiontype built an empty pattern set and redacted nothing; an unknowngrokpattern name silently became the catch-all; an unknownconverttarget left the field unconverted per document; array-typed keys given as a bare string were dropped and the stage behaved as if unconfigured; andDELETE /_ingest/pipeline/{id}removed the definition while leaving the compiled pipeline live under the same name. ESconvertlong/doublenow map to XERJ's equivalentinteger/floatrather than being refused.Two more of the same, at the definition's own edges: a pipeline-level
on_failureblock (the top-level body key, not the processor-level one) was accepted, echoed byGET, and never run — the pipeline is now recorded unrunnable, exactly as its processor-level twin already was; andsetwith Elasticsearch'scopy_fromis reported as an unsupported option (accepted atPUT, refused at the write) rather than asmapper_parsing_exception: missing value, which told the caller their valid ES pipeline was malformed.The refusal reads the value, not the key. A processor-level option is refused only when what the caller wrote asks for behaviour this build does not have; writing the value XERJ already implements is the same as omitting it.
ignore_failure: falseis Elasticsearch's own default and ES clients emit it verbatim — a presence check made that pipeline register and then400every write through it, while the byte-identical processor without the key indexed fine. Accepted:ignore_failure: false, an emptyon_failure: [],ignore_missing: true(which is exactly whatremove/rename/convert/datealready do — they pass the document through unchanged when the field is absent),ignore_empty_value: false,allow_duplicates: true. Refused: the opposite value in each case, and anyifat all. (Modelled on Lucene's consume-then-complain:AbstractAnalysisFactory.getBoolean(args, name, default)atlucene/core/src/java/org/apache/lucene/analysis/AbstractAnalysisFactory.java:213-217removes the key and returns the default only when it was absent, andArabicStemFilterFactoryatlucene/analysis/common/src/java/org/apache/lucene/analysis/ar/ArabicStemFilterFactory.java:44-51then throws on whatever is left. Lucene is Apache-2.0, the same licence as XERJ; no code is reproduced.)removeandrenameno longer walk around the gate. Their ES→XERJ translation built a fresh config object holding only the translated keys, so a processor-levelif/on_failure/ignore_failure/ignore_missingon them was dropped before the gate ran:{"remove": {"field": "secret", "if": "ctx.tenant == 'a'"}}compiled, was acknowledged, and dropped the field on every document — while the identicalifonsetwas refused. Every branch of the translation now edits the caller's config instead of rebuilding it._simulatespeaks Elasticsearch's vocabulary again. It walks the stored pipeline, and once ESrenamestarted compiling the stored form became XERJ's internalstagesshape — whose stage names leaked straight into the response, soprocessor_typereadfield_renamewhere ES reportsrename. The stages→processors conversion now inverts both halves of the translation, name and config.GET /_ingest/pipeline[/{id}]answers in Elasticsearch'sprocessorsvocabulary instead of XERJ's internalstagesshape (#204). Two shapes reach the store: the translatedstagesbody (everything that compiles) and the raw ES body (kept verbatim for a definition XERJ stores but cannot run). Only the second was ever ES-shaped, soGETspoke XERJ's vocabulary for every pipeline that worked — measured before this release, an ESrenamepipeline came back as{"p":{"description":"d","stages":[{"type":"field_rename","config":{"mappings":{"a":"b"}}}]}}. This is a wire-visible change on bothGET /_ingest/pipeline/{id}andGET /_ingest/pipeline. It also mattered beyond cosmetics:GET's output is what an operator edits andPUTs back, and astages-shaped body takes the handler's pass-through branch — which is exactly how the restart hole described under Changed was reached in ordinary use.setis rendered with an explicit"override", always, because it is the one key whose default differs between the two vocabularies (Elasticsearchtrue, XERJ's nativesetstagefalse). Emitting a bare ESsetfor a native stage would claim it overwrites when it preserves. A pipeline created throughPUT /_ingest/pipelinetherefore shows"override": trueeven when the caller omitted it — that is the value XERJ applies, and it round-trips: PUTtingGET's own output back cannot silently flip the stage's behaviour. Relatedly,_simulate'ssetinterpreter honoured nooverrideat all and always overwrote, so it reported a document the ingest path would never produce; it now applies Elasticsearch's rule (defaulttrue;falseleaves an existing non-null value alone).This is also the only read endpoint for a pipeline created through
PUT /v1/pipelines/{name}— the native surface registers noGET— so the round trip has to be lossless in xerj's own vocabulary too, and it was not.PUT /_ingest/pipeline/{id}rebuilt the stored definition as{description, stages}and dropped every other top-level key, so PUTtingGET's own output back silently discarded a natively-defined pipeline'son_errorandtimeout_ms: the error policy reverted frompass(keep the document) to theDropdefault (discard it) under a200 {"acknowledged": true}. The translation now edits the body instead of rebuilding it — the same rule already applied toremove/renameone level down. Measured on this release,{"description":"n3","on_error":"pass", "timeout_ms":250,"stages":[…]}reads back with both keys intact, and PUTting that body back leaves them intact, before and after a restart. Elasticsearch's own pipeline metadata (version,_meta,deprecated) rides along the same way and is now echoed back rather than dropped.PUT /v1/pipelines/{name}given that ES-shaped body used to answer500withinternal error: missing field stages; it now answers400naming/_ingest/pipeline/{name}as the endpoint that accepts it.Four wire-visible refusals that were fixed in this sweep but not written down (#204). All four replace a silent fallback with an error, which is the point of the issue, but each changes a status code:
POST /v1/admin/backupanswers400on a request body that is present but unparseable, where it previously ran a default backup under a success response;GET /{index}/_statsandGET /_all/_statscarry an added non-Elasticsearch keyprimaries.mappings.schema_persist_failuresinside an otherwise ES-shaped response; the authorization layer's response pruning answers500when it cannot parse the body it was asked to filter, instead of falling back to the unfiltered bytes; and the bundled console'sDELETE /_xerj-console/api/v1/auth/passkeys/{id}propagates a failed delete instead of discarding it — it answered204 No Contentunconditionally, so a revoke refused by the engine (index write block, disk flood stage, storage error) reported success while the passkey stayed on disk and kept authenticating. An already-absent credential is still a204; only a genuine failure now surfaces.The refusal now covers every ingest path, not only
PUT /{index}/_doc.POST /_bulk?pipeline=,index.default_pipelineunder_bulk,_reindexdest.pipelineandPOST /{index}/_update_by_query?pipeline=previously read the instruction and dropped it — a redaction pipeline that the single-document endpoint refused was ignored in bulk, and the document was stored verbatim undererrors: false. All four now run the pipeline, and refuse when it cannot run._reindexand_update_by_queryvalidate the name before touching a single document, so a bad pipeline fails the request instead of half of it.Two related honesty fixes on the same paths:
_clone/_shrink/_splitcopied at most 10,000 documents and swallowed every per-document write error before answering{"acknowledged": true}(now full keyset paging, and any failed write — or a failed source flush — fails the request); and a failed_bulkitem no longer reports"result": "deleted"alongside its ownstatus: 400, which claimed a deletion that never happened.ignore_missing: falseis refused on purpose, and it is not the same call asignore_failure: false. Both are Elasticsearch defaults that ES clients emit verbatim, so the difference is worth stating plainly: the rule is accept the value this build actually implements, and the two land on opposite sides of it.ignore_failure: falsemeans "a failing processor is not ignored", which is exactly what XERJ does, so it is accepted.ignore_missing: falsemeans "FAIL the document when the field is absent", which XERJ cannot do at all —ProcessActionhas no error variant, and every field-reading stage passes the document through untouched. Accepting it would be precisely the silent lie this issue exists to remove, so it is refused: aremove/rename/convert/dateprocessor that spells outignore_missing: falseregisters and then400s every write through it. Omitting the key is accepted and behaves astrue. That asymmetry — omission accepted, the ES default written out refused — is deliberate but is a hard400for ES-generated pipelines; strip the key, or set it totrue, to state what XERJ will actually do.ignore_empty_valueis the mirror image: omitting it or writingfalseis honoured exactly,trueis refused.Known gaps, stated rather than closed:
_simulateinterprets the stored ES body directly and covers processors the compiled path does not, so a pipeline recorded as unrunnable can still show a transformation under_simulatethat ingest refuses to perform;GET /_ingest/pipeline/{id}carries no runnability signal — it renders the stored definition in ES vocabulary whether or not this build can run it, so an operator only learns at write time; a write addressed to an alias does not pick up the backing index'sdefault_pipeline;_update_by_queryreports adropprocessor as a per-document failure, because it updates in place and has no delete path;Engine::degraded_pipelinesis in-memory only and has no HTTP surface yet, so a boot-replay degradation is visible in the startup log and nowhere else — unlike a definition-time refusal, which is now durable (see Changed); and a refusal marker is only read back at boot, so a data dir opened by two processes at once is outside what the sidecar can promise (the node lock already prevents that).Unity assets were sampled through a 4 MiB cap, silently junking whole object classes. Unity YAML is a grouped family — each
unity_classis its own dataset and a class's first document can sit anywhere in a scene thatextract/unity.rsitself says can exceed 200 MB. A class first appearing past the cap was never sampled, got no entry in the plan, and phase B then had nowhere to route its records: they becamefile_junkwith nothing said. The cap is now 512 MiB for this family, and any record whose group was never sampled is now reported by name and count instead of silently counted as junk.A
script_guidthat failed to resolve produced no counter, no warning and no report line.build_unity_guid_mapdiscardedextract_meta'sResult, so an unreadable.metawas indistinguishable from a script nothing references — on the feature's own headline query. Unreadable sidecars, sidecars carrying no usable guid, and unresolvablescript_guids are now each reported.script_path/script_classwere mapped only when the phase-A sample happened to contain ascript_guid, while phase B stamps them whenever a guid resolves. A cluster whose sampled window held nom_Scripttherefore got them dynamic-mapped at index time, feeding the field-budget overshoot in #312. They are now registered for every Unity cluster, and the enrichment runs beforecoerce_recordrather than after it, so the fields it stamps are validated like every other field instead of bypassing coercion.build_unity_guid_mapran serially, unmetered, on the critical path of every run — including a resumed no-op incremental, which otherwise has no work to do. A real Unity project has 10k-500k.metasidecars. It now runs oncrate::poolunder the progress meter (the unattributed-stretch pattern of #241).A
--stubfile was named from the content path, which under durable preparation is a content-addressed blob — so every stub's one and only field would have been titled after a blob ordinal (the #294 failure class). It now usesSniffed::logical_name.order::band_from_family_strdisagreed withorder::band. The string form's catch-all rankedbinaryasBulkwhere the enum form ranks itVendored, so a resumed run could order work differently from the run that planned it. Both now agree for every family, pinned by a test that iterates the whole enum.The autoindex use-case README documented three flags and behaviours that do not exist.
--no-default-excludesand--no-gitignoreare spelled--no-default-ignoresand--no-ignore, andcli::parsehard-errors on an unknown argument — so a reader who copied the documented invocation got an error instead of an index. The marker-gated pruning ofnode_modules/andtarget/was described but had been dropped from the branch. A test now parses every--flagnamed in that README and fails if the CLI would reject it.stub_matcher_tests::an_invalid_pattern_fails_loudly_at_startupdid not test its own name — it asserted only that a valid pattern compiles.glob_to_regexescapes every metacharacter, so no glob can produce a syntactically invalid regex; the one reachable failure is the compiled-size limit, which?and**reach at ~10^5 characters. Over-long patterns are now rejected explicitly with a message naming the flag and the limit, and the test exercises that path.
Changed
Two proposed byte-statistics binary heuristics were dropped before they shipped; raw TGA is detected by its header instead. Both were attempts to recognise a raw texture that decodes into printable characters, and both turned out to be tests for "not written in Latin script".
The first, from PR #274, classified any text over 4 KiB with under 5% whitespace as binary.
nonblankis built fromtext.lines(), so newlines are already stripped and only intra-line whitespace counts — which means Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese prose, base64 blobs, FASTA sequences and minified single-line files all score 0%. This repo's ownfailure_resume_http_tests::legacy_key_collision_fails_before_visibility_with_scoped_guidancebuilds a 65,537-byte fixture of one repeated ASCII letter; with that guard reinstated the test fails withleft: 3, right: 0— the run exits 3 instead of 0.The second was introduced by the first attempt to reland #274 and is removed here: "decoded via lossy windows-1252 AND over 30% non-ASCII is pixel soup". windows-1252 is the fallback every legacy 8-bit codepage decodes through, so it is not a test for image data. Measured through
sniff()onca4d75aversus that branch, with identical fixtures:fixture ca4d75awith the guard windows-1251 Russian, 13,000 B txt-prosebinaryKOI8-R Russian, 13,000 B txt-prosebinarywindows-1253 Greek, 12,200 B txt-prosebinarywindows-1255 Hebrew, 11,000 B txt-prosebinarywindows-1256 Arabic, 10,800 B txt-prosebinaryShift-JIS Japanese, 22,401 B (byte pad 1; likewise pad 3) txt-prosebinaryShift-JIS Japanese + ASCII code, 13,360 B txt-linesbinaryShift-JIS Japanese + ASCII code, 66,800 B txt-linesbinaryscan_fileturnsFamily::Binaryintojunk: binary content (unknown), so each of those files stopped being indexed. Alooks_like_legacy_cjkescape hatch shipped with it and could not carry the weight: single-byte codepages never form valid Shift-JIS/GBK/Big5/EUC-KR double-byte pairs, so Cyrillic, Greek, Hebrew and Arabic were never rescued at all; it required a lossless trial decode whilesniff()sees only the first 8192 bytes, so the same Japanese document was text at byte-offset pads 0 and 2 and binary at pads 1 and 3; and a realistic Japanese technical document (prose around ASCII code fences) sits below its 30% ideograph floor while sitting above the guard's 30% non-ASCII ceiling. Every fixture in the table above now classifies exactly as it does onca4d75a.What replaces it is
looks_like_tga_header: TGA has no magic number, but its 18-byte header is constrained enough that bytes 1 and 2 of the file must both be control characters. Headerless raw payloads (.raw,.bytes, uncompressed PCM) still classify as text, exactly as they do onca4d75a. Bounding what a magic-less binary costs is the fix for that and is not attempted here — see #381;for_each_sectionalready streams, so the unbounded quantity is the record count, not resident memory.The new magic-byte signatures now require structural confirmation. Nine signatures (PSD, both TIFF byte orders, RIFF, OGG, FLAC, MP3, FBX and EXR) were added by the first reland attempt as bare
starts_withtests. Six of them are entirely printable ASCII —8BPS,RIFF,OggS,fLaC,ID3and the 18 lettersKaydara FBX Binary— so they matched ordinary text. Measured through the realsniff()onca4d75avs this branch with identical fixtures:fixture ca4d75afirst reland CSV whose first column header is ID3csvbinary/mp3prose opening RIFF is a container format used by WAV files. …txt-prosebinary/riffprose opening OggS pages carry the packets of an Ogg stream …txt-prosebinary/oggprose opening fLaC is the four byte magic of a FLAC audio …txt-prosebinary/flacprose opening 8BPS is the magic of an Adobe Photoshop …txt-prosebinary/psd.mdnote openingKaydara FBX Binary is the 20-byte magic …txt-prosebinary/fbxEach of the six is now qualified by what must follow it: PSD version, RIFF FORM type, Ogg stream-structure version, FLAC block type, ID3v2 major version and synchsafe size, and for FBX the full 23-byte header (
Kaydara FBX Binary, two spaces, NUL,0x1A,0x00). The FBX case was missed by the first pass at this fix and caught in review — at 18 printable characters it is the likeliest of the six to open a real sentence, and likeliest precisely inside the Unity/3D-asset corpus this feature targets. Every fixture in the table above now classifies exactly as it does onca4d75a. The true positives are unchanged and still covered; the FBX true-positive fixture was corrected to the real 23-byte header, which is what any Autodesk tool emits.This is what Lucene's
CodecUtil.checkHeaderdoes withCODEC_MAGIC(lucene/core/src/java/org/apache/lucene/codecs/CodecUtil.java:183, which hands straight tocheckHeaderNoMagicat:202and refuses the file unless the codec name and a version in range follow — the magic alone is never taken as proof). Apache-2.0, same licence as XERJ; adapted, not copied.Not fixed here, deliberately:
GIF8andBMare printable-ASCII signatures with exactly the same defect, but they are pre-existing — they are unchanged fromca4d75a, and prose openingGIF8/GIF89a, a CSV whose first column header isGIF8, and prose openingBMwere each measured to classify asbinary/gifandbinary/bmponca4d75aand on this branch, identically. Fixing them changes behaviour unrelated to Unity, so it is filed as #380 rather than smuggled in here, and today's wrong answers are pinned bygif8_and_bm_are_still_taken_on_faithso #380 has to flip that assertion deliberately.GIF8is listed alongsideBMso the next reader does not concludeBMis the only one.A junk entry could overwrite a successfully indexed file's catalog row. When phase B met a record group phase A never sampled, the worker pushed a whole-file junk entry while leaving
send_errunset — so the file also reachedjournal.file_done, and both passes wrotecatalog::file_docunder the samefile:{file_key}id into the same bulk. The junk document landed second and won: a file that indexed records was reported in the catalog as statusjunkwithrecords: 0,files_junkcounted it, andCodeCoverage::observeran for it twice. Reachable with no Unity involved, via a >64 MiB SQL dump whose first row for some table starts pastSQLDUMP_SAMPLE_LIMIT. The unsampled group is now reported on the progress meter instead, the dropped records stay on that file's own completion where they already were, and the disjointness the code claimed in a comment is now enforced at the one place that holds both sets (shadowed_junk_entries).
Known issues
ref_guids(Unity) andjoints(BVH) are multi-valued keyword arrays, and #332 means array elements are joined into one FTS token, so a post-flushterm/matchon a single element does not hit.script_guid,script_pathandscript_classare single-valued and unaffected, so the documented "which scenes use this script?" query is not impacted. Fixing #332 is an engine-side change to the FTS writer input type.GIF8andBMare still unqualified printable-ASCII magic signatures, so a text file whose first characters areGIF8,GIF89aorBMis classifiedbinaryand junked. This is pre-existing, not introduced here — measured identical onca4d75aand on this branch (binary/gifandbinary/bmpin both) — and is left unchanged because fixing it is a behaviour change with no Unity content. Both names are recorded here on purpose:BMis the obvious one andGIF8is the one a reader would otherwise miss. Tracked as #380;gif8_and_bm_are_still_taken_on_faithpins the current behaviour.A magic-less binary is still sectioned and indexed in full. The two byte-statistics guards this branch removed (they junked non-Latin text) also caught NUL-free binary payloads that decode printable under the windows-1252 fallback. Nothing replaced them except
looks_like_tga_header, which covers TGA only. Measured on this branch: a 4,194,495-byte printable NUL-free blob namedtexture.bytessniffstxt-proseand expands into 2048 indexed records. This is not the peak-RSS bug from #239 —for_each_sectionis still streaming and per-file memory is still bounded; the unbounded quantity is the record COUNT. Removing the guards was still right: they deleted CJK, Cyrillic, Greek, Hebrew and Arabic documents worldwide. Tracked as #381.autoindexaborted permanently on a corpus containing a byte-identical duplicate (#345). A duplicate path is the only thing that makes a run issue the catalog's duplicate-alias_delete_by_queryat all, and that call was fatal. Because it runs after every document is durably indexed and after the per-file journal has committed, a 5xx there turned a fully successful run intoxerj-done ok=false exit=1 reason=aborted— and it stayed that way: the journal was complete, so every rerun indexed zero files and aborted on the same line. The sweep is metadata-only bookkeeping and is now reported instead of fatal: the run finishesok=true exit=3 reason=catalog-alias-sweep-failed, names the paths whose alias document was not swept, and recordscatalog_alias_sweep_failed_paths/catalog_alias_sweep_erroron the run document. Exit3already means "completed, and something is recorded rather than fatal", so no new exit code was introduced — readreasonto tell it apart fromcompleted-with-junk. The catalog bulk that follows the sweep is still fatal, so a catalog index that cannot be written still fails the run.Any 429/5xx from the server was reported as a bare status line. The retry wrapper kept
HTTP 500 Internal Server Errorand discarded the response body, which is the same sentence for a poisoned index, a full disk and a panicking handler — the reason #345 was filed as "not investigated". A bounded prefix of the server's ownerror.type: error.reason(or the raw body, for a proxy's error page) now travels with the status on everywith_retrycall site.The duplicate-alias sweep issued one HTTP request per duplicate path. It now names up to 1024 paths per request with a
termsfilter, so a corpus with K duplicates costsceil(K/1024)round trips instead of K — and has proportionally less surface on which to fail.
[1.0.0-rc.16] - 2026-08-13
Both rc.15 known issues (the progress-stream forgery and the outer-.gitignore
reach into nested checkouts) are fixed in this release — see Fixed below.
Performance
An idle node with many indices burned ~4 CPU cores and made the host machine unusable.
IndexStorespawned one OS thread per index that woke everywal_batch_ms(default 100 ms) and locked every WAL shard of that store to ask whether it was dirty. With the default 16 ingest shards that is ~1.5M lock operations per second on a 9,382-index node, for an answer that is always "no" on an idle index. WAL fsync is now scheduled by one process-wide, event-driven pool ((cores/2).clamp(2,8)threads, spawned lazily): a shard arms itself when a write arrives, and an index with no pending writes costs nothing — no thread, no wakeup.Measured on a real 9,382-index corpus, idle, with zero clients (10 samples × 15s): threads 9,709 → 339; CPU 718-760% → 59.7-68.3%; context switches ~197,000/s → ~4,000/s; interrupts ~142,000/s → ~7,600/s. Thread count no longer scales with index count — 20 indices versus 200 is 335 threads versus 335, where it was 365 versus 541.
Durability is unchanged and was verified with
strace -f -y -ttso WAL fsyncs are distinguishable from snapshot fsyncs:syncfsyncs before ack,batchedissues exactly one fsync inside thewal_batch_mswindow,asyncissues none, andkill -9plus restart recovered every acked document in all three modes. RSS is a smaller win and reported as such: marginal 0.699 → 0.682 MB per index (~12 KB/index of thread stack), since per-index RSS is dominated by index state rather than the thread. (#334)Not fixed here: ~0.65 core remains on that node from node-global sampling and O(indices)-per-tick work on shared timers — a different mechanism, tracked separately.
Security
cargo-auditandcargo-fuzznow actually run in CI — they were only ever documented (#207).user-feedback/09-security/cves-and-vulnerabilities.mdlisted "cargo-auditfor dependency vulnerability scanning" and "fuzz testing (cargo-fuzz) on all input parsing paths" as part of XERJ's answer to Elasticsearch's CVE history. Neither existed: no audit job, no fuzz job, no fuzz target anywhere in the tree. A reader comparing the two engines priced in a fuzzed parser surface that was not there.- New CI job
security-auditrunscargo auditon every push and pull request; a RUSTSEC vulnerability advisory fails the build. - New CI job
fuzzbuilds and runs seven libFuzzer harnesses (engine/fuzz/) over checked-in seeds: the Elasticsearch query DSL, the Lucenequery_stringgrammar, date math and date-format patterns, index-name date math, SQL, Painless scripts, and the aggregation engine's two separate script tokenisers..github/scripts/fuzz-smoke.shis the same entry point locally and in CI. xerj-engine/tests/security_tooling_claims.rsfails if a documented tool stops running, if a fuzz target ships without a seed corpus, or if the "all input parsing paths" overclaim reappears in any prose file.- The claim itself is now specific about what is and is not covered.
- Known gap, stated because the point of this entry is not to overclaim
again: seven harnesses are seven parsers, not the engine's whole input
surface, and one parser outside them is already known to abort.
aggs::parse_time_zone_offsetslices atime_zonestring at byte 2 after a byte-length check, so adate_histogramwith"time_zone": "+中a"in an unauthenticated_searchbody dies on a character boundary. It is pre-existing, is not touched by this change, and is tracked in #272 with a fix and a proposed harness for the aggregation date parsers.
- New CI job
Fixed two high-severity advisories in the document-ingest XML parser. Turning the audit gate on found
quick-xml0.36.2 inxerj-autoindex, which parses.docx,.xlsxand.xmlfiles supplied by whoever owns the folder being indexed: RUSTSEC-2026-0195 (unbounded namespace-declaration allocation → memory-exhaustion DoS, CVSS 7.5) and RUSTSEC-2026-0194 (quadratic run time on duplicate attribute names, CVSS 7.5). Upgraded to 0.41, which also de-duplicates the crate — 0.41 was already in the tree viapdf_oxide.crossbeam-epochmoved 0.9.18 → 0.9.20 for RUSTSEC-2026-0204.
Fixed
Following the documented setup and running
xerj autoindexfailed with a 401.AuthConfig::default()isenabled: true, so copying the shipped config as the docs instruct produced an auth-enabled server, while the pages that showautoindexall pass--insecureand the pages that tell you to write a config never mention a key.ping()also discarded the HTTP status, so the 401 on the first round trip was swallowed and the run printed ~15 lines of healthy-looking progress before dying at the embedding-identity probe with a message about the server's capabilities — leading the reporter to conclude their server lacked a feature rather than a credential.ping()andembedding_execution_identity()now fail fast and actionably on 401/403,autoindexgained theadmin.keyfallbackbrainalready had (loopback only), the server sendsWWW-Authenticate: ApiKey realm="xerj", and the 401 body names the real data dir plusXERJ_API_KEY,--api-keyand--insecure. Reported by a user on macOS. (#333)xerj-donereported an identical success line whether or not any source code was indexed. A corpus in which every code file was junked printed the same terminal record as a healthy one, which is how a silently degraded index went unnoticed. The terminal line and run document now carrycode_files,code_files_indexedandcode_files_junkedon both the generated and legacy paths, and a warning is emitted before the terminal line when code files were detected and none produced an indexed record. Namedcode_files_indexedrather thanastdeliberately: a code file with no captured symbols still indexes correctly, soastwould overstate. (#294, #316)The licence detector was wrong on 2 of the 13 repositories it had classified.
sonicwas recorded asGPLwhen it is MPL-2.0 — MPL-2.0 defines "Secondary License" in terms of the GNU GPL inside its own text, and restrictive-first body matching hit that first.quickwitwas recorded asApache-2.0/UNKNOWNwhen it is plain Apache-2.0 —LICENSE-3rdparty.csvis a dependency inventory, not the project's licence. The classifier now reads the title line before the body and skips third-party inventories. Elasticsearch's triple licence still resolves toAGPLfrom its title, so the safe direction is preserved where it matters most; that is a test.A corpus manifest could write outside the corpus directory and destroy an unrelated checkout.
--fromis documented as "rebuild a corpus someone else defined", so a manifest is untrusted input — but only the corpus name was screened, while the per-reporepofield, from the same file, was used directly as a path. A manifest with"repo": "../../../work/repo"moved that checkout to the manifest's commit, discarded its uncommitted changes and untracked files (checkout --force,clean -fd), and rewrote itsoriginremote.repomust now be a plain directory name, enforced inxc-corpus.shand invalidate_manifest.py, with the destructive path double-guarded.xc-index.shsilently ignored an unrecognised argument.xc-index.sh <corpus> --frshset no flag, ran an incremental index and exited 0. Becauseautoindexkeeps incremental state in~/.xerj/autoindex/, a run that should have been fresh skips every file as "already indexed" and leaves a corpus with 0 documents that retrieves nothing, reporting no error anywhere. Unknown arguments are now rejected by name.xc.pywarned about onlyGPLandLGPL, by exact match, so it was silent on every restricted repository the hub deliberately ships: elasticsearch (AGPL/SSPL/Elastic), meilisearch's BUSL Enterprise Edition parts, and — once the detector above was corrected fromGPLtoMPL-2.0— sonic, which had been warning before. The warning at retrieval time is the one that counts, because that is when an agent is looking at the code and deciding whether to lift it; it now covers the same set asxc-corpus.sh.New CI job
reference-coding: shell and Python syntax, the offline round-trip suite (tools/xerj-code/tests/test_xc_corpus.sh, 28 checks, local git repositories overfile://, never touching the real~/.xerj-code), andvalidate_manifest.py --hubover every shipped manifest.A binary that does not understand
cluster_state.jsonnow fails closed before activating storage. The previous loader accepted future and partially-understood shapes, dropped fields it did not know, and could later rewrite the reduced document over the original. Boot now accepts only the complete shipped format-1 envelope, including duplicate-key and nested-shape checks. A rejected document leaves liveness at 200 and readiness at 503, but opens no user index, Console system index, WAL, segment, or durable audit sink; Console bootstrap and storage background jobs remain disabled until a supported restart. After the existing request-size, authentication, and authorization gates, HTTP storage access returns the stablecluster_state_unavailable503; authenticated gRPC calls returnUNAVAILABLE. Existing 401/403/body-limit precedence is unchanged. Client responses give the category and recovery action without a local path or persisted object names; the server log carries the detailed classification and path for the operator. The original bytes and any legacy staging file stay in place, and a blocked boot creates no salvage copy. Format-1 rewrites retain the existing same-directory temp-file fsync and atomic rename; parent-directory sync is a best-effort attempt whose errors are ignored, so this change does not claim strict namespace durability across power loss. Data-directory creation,node.lockacquisition/PID diagnostics, and the server's earlier credential/TLS preparation retain their existing behavior; the no-open/no- replay claim begins at the cluster-state classification and covers storage stores, not those process-bootstrap files.PUT /{index}/_settings {"index.lifecycle.name": null}actually detaches the index, and the detach survives a restart (#282, ported from #262; regression-tested in #290). The null previously fell through a string-only settings reader, so the operator got200 acknowledgedwhile the index stayed attached — to a policy whose delete phase then destroyed it. A detach now removes the execution cursor, writes a persisted tombstone (ism_managed_indices.jsongrew amanaged/detachedenvelope; the old bare-map file is still read), and scrubs the staleindex.lifecycle.namefrom the stored settings.The lifecycle delete action gained #262's safety rails. It now refuses — visibly, in
explain, never silently — to delete a dot-prefixed internal index (.ds-*backing indices exempt), a data stream's current write index, or an index whose age cannot be established from its execution cursor.An ILM policy naming an action the engine cannot execute is refused at PUT time with the action named (400), instead of being stored with the action silently dropped — the accepted-and-ignored class (#204). Executable ILM actions are
rollover,delete,readonly.autoindexno longer rejects a resumed generation after journal replay when inferred floating-point schema statistics need exact decimal round-tripping. Unchanged reruns and junk-bearing corpus shrink now replay correctly; existing affected journals resume without a rebuild, while the manifest format and integrity checks remain unchanged.A
query_stringcontaining one non-ASCII character aborted the process.{"query_string":{"query":"_\u0660"}}— or the same text in?q=— took the server down. The Lucene tokeniser walks bytes but slices&q[start..i]as a&str, and it broke onis_whitespace(), which is true for Latin-1 NBSP (0xA0). In valid UTF-8 that byte is only ever a continuation byte, so the scan stopped between the two bytes of U+0660 and the slice panicked on a character boundary. Every delimiter test in that scanner is nowis_ascii_*, which makes "the scan can only stop on a character boundary" true by construction. Nothing is lost: the only non-ASCII bytes the old test accepted were 0x85 and 0xA0, neither of which is a standalone character in UTF-8.An index name of
<{}{}>aborted the process. Index-name date math finds the first{and the last}inside the braces; when the}comes first,date_part[2..0]is a panic, not an empty slice. Reachable from any request that names an index. The closing brace is now required to come after the opening one.A Painless script containing one non-ASCII character aborted the process. The second thing the new fuzz harnesses found, inside a second. The tokeniser walks bytes (
bytes[i] as charreinterprets each byte as Latin-1) but slices&src[start..i]as a&str. Every continuation byte of a multi-byte character reads as an alphabetic Latin-1 char, so an identifier scan could start or stop inside one and the slice panicked on a char boundary —end byte index 24 is not a char boundary; it is inside 'ʋ'. Scripts arrive in ordinary search and update bodies and the release profile setspanic = "abort", so this was not a 400, it was the whole server. The identifier scan is now ASCII-only, which keeps both ends on a character boundary by construction; a non-ASCII byte reaches the tokeniser'sunexpected charerror instead. String literals still carry any Unicode.A search body could abort the whole process:
{"range":{"ts":{"gte":"now+33333333333333H"}}}. The first thing the newdate_mathfuzz target found, 20 seconds into its first run.chrono'sDuration::hoursand its siblings panic when the count does not fit aTimeDelta, and they run before thechecked_add_signedthat was supposed to makexerj_query::dates::add_unittotal — so the careful checking around them bought nothing. A range bound goes straight from the request body into this code (parser.rs:931), the release profile setspanic = "abort", and the result is an unauthenticated remote denial of service on a released binary. Fixed with the fallibletry_*constructors. Grepping for the same pattern found three more copies in the index-name date-math resolvers —xerj-engine/src/index.rs, and both branches ofxerj-api/src/es_compat.rs: thenowform (<logs-{now+9999999999999d}>) and the anchored form (<logs-{2026-01-01||+9999999999999d}>), which are handled by two different functions sixty lines apart. All three are fixed, theirn * 30/n * 365multiplications arechecked_mul, the unit is read as one character rather than one byte, and the addition goes throughchecked_add_signed.es_compat's resolver is the one that runs on the wire — the index path parameter of every create/index call and the_searchindex list — socrates/xerj-api/tests/index_name_date_math_is_total.rsnow drives it at the crate boundary; the same-file fix that missed the anchored branch passed because nothing exercised the public entry point.A
_searchaggregation script containing one non-ASCII character aborted the process.painless::tokenizewas not the only Painless-subset scanner in the tree:aggs.rsholds two more with the identical byte-walk /&str-slice shape —lex_script(scripted_metric) andtokenize_script(bucket_script/bucket_selector) — and both took&src[i..i + 2]to test for a two-character operator at a byte index that can sit inside a multi-byte character, because every arm above it is ASCII-only.{"m":{"scripted_metric":{"map_script":"中"}}}wasend byte index 2 is not a char boundary. Both also skipped whitespace withis_whitespace()onbytes[i] as char, which is true for Latin-1 NEL and NBSP — bytes that in valid UTF-8 are only ever continuation bytes. The two-character probe now requires both bytes to be ASCII (every such operator is ASCII, so nothing is lost) and whitespace isis_ascii_whitespace. A newagg_scriptfuzz target covers all three entry points, which is how the next one gets found instead of reported.An XML entity nobody declared no longer discards the text around it. Fallout from the
quick-xmlupgrade above, which is a breaking change to how character data arrives: 0.38 stopped folding&into the surroundingEvent::Textand began emitting each reference as its ownEvent::GeneralRef. Entity handling is preserved across that change (a reader that ignored the new event would have silently dropped every&,<,>andéwhile the words around them still arrived), and one case gets better: 0.36'sBytesText::unescape()failed the whole text node on an entity no DTD declared for us, andunwrap_or_default()then discarded it — now only the reference itself is dropped. Character data is also reassembled from its fragments before being emitted, so an entity-bearing field stays one value instead of becoming an array of pieces.constant_scoreon a mixed flushed/unflushed corpus returned the wrong page, and any tied hit set could be re-sorted into_idorder (#270, #300). Two defects. A top-levelconstant_score(orboost) wrapper defeated the match-all shortcut, so the stored scan ran in exact-counting mode and a bounded page came back all-memtable —size:1on 40 flushed + 300 unflushed identical docs returned the memtable doc where the oldest flushed doc was correct; the wrapper is now peeled for the count/bounds decisions only, with scoring unchanged. Separately, five post-sort re-sorts — the bool-text IDF rescore, the near-zero-BM25 TF-IDF fallback, and the threerescoresorts — tied by_idalone, so any of them firing on tied scores reordered the page; the main sort and all five now route through the one page-order key (score DESC, seq_no ASC, _id ASC). Known residual, documented rather than asserted around: bounded pages for multi-clause bools on a mixed corpus are still admitted under first-pass scores (#188's remit).A shrinking file set on a junk-bearing
autoindexgeneration aborted the cutover withdesired manifest digest does not match sync_begin payload(#283, #296). A fresh plan kept duplicate-file aliases whose content was junk or skipped while the incremental projection dropped them, so the two disagreed on byte-identical junk duplicates — two empty files suffice. The fresh plan now projects exactly like the incremental path, so the shrink reconciles cleanly. Where a journal is genuinely inconsistent, the replay and validation paths now say what to do — no remote data changed, re-running will not repair it, rebuild with a new--state-dirand--prefix— with the invariant kept as the error cause.An outer repository's
.gitignoreno longer judges files inside a nested checkout (the second rc.15 known issue below; #287, repairing #279). The rule walk had no repository-boundary stop, so a root*.mdhid the README of every vendored dependency beneath it. The git-sourced rule kinds now stop at a.gitboundary — the set kept under a nested checkout is byte-for-byte whatgit status --untracked-files=allreports there — while.xerjignoreand the built-in defaults deliberately still govern the whole folder you named. Three more defects from the same merge: hidden directories pruned by the dotfile rule were deep-counted intoignored_files_in_pruned_dirs(97,731 phantom files on this repository's own walk); that number reached thexerj-doneline and--progress jsonas a bare total when it is budget-capped — a newignored_files_in_pruned_dirs_exactflag now says whether it is a total or a floor; and--no-ignore/--no-default-ignoreswere accepted and ignored (#204) onautoindex mapandstatus, which never walk a filesystem — both are now refused there with the reason. The shipped docs that contradicted the code are corrected.A finished
autoindexrun could leave its state-directory lock held, so the immediately following run on the same state dir was refused as already in use (#305).flockownership follows the open file description, so a helper subprocess forked while the journal was live (the PDF extraction helper, git invocations) could inherit the descriptor and keep the lock held past the owner's close. The lock guard now records the acquiring PID and unlocks on drop only in that process — a fork-inherited copy closes its descriptor without releasing the live owner's lock. No retry, wait or takeover behaviour was added; crash recovery is unchanged. Covered by a same-OFD unit oracle and a real-fork integration regression on Linux/macOS.autoindexindexed a.mtsor.ctsfile as prose (#284). The extension registry claimed.mjs/.cjsbut not TypeScript's own ESM/CJS counterparts, and an unclaimed extension does not degrade to "code without symbols" — it fell through to the prose extractor, so the file carried nolanguage, nodefs, nosymbols, was split into several body-only records, and was invisible to symbol search.An exported module
constreacheddefsin neither TypeScript nor JavaScript (#285, #304, closes #293). Capture only fired when the value was a function, so a module whose whole public surface is built by factory calls —export const users = pgTable(...), a router, a config object — extracted zero symbols and could not be found by symbol search at all. The new pattern is exported-only, anchored to module scope, and matches theconsttoken, so function-local bindings and mutable exportedletstay out; module-privateconstwas measured and excluded because it added weight without retrieval gain.A PHP 8.1 enum extracted zero symbols, and class constants were never captured (#286). An enum declares no class, interface or trait, so a file holding one could not be found by its own name through symbol search. Enums, enum cases and
constdeclarations — class-level and file-scope — now land indefs, with cases treated as named constants the way Go'sconst_specmembers are;define()calls stay uncaptured.
Added
xerj mcp— the MCP server now ships in the installed binary. XERJ had a complete, tested 10-tool MCP server that no user could obtain: releases built onlyxerj-server, the installer shipped onlyxerj, and MCP appeared in no agent-facing doc. It is now a subcommand, so anyone who rancurl -fsSL https://xerj.org/get | shgets it on their next upgrade, with no installer or release-matrix change and no new third-party dependencies. Tools:xerj_search,xerj_semantic_search,xerj_vector_search,xerj_hybrid_search,xerj_memory_store,xerj_memory_recall,xerj_brain_ego,xerj_brain_link,xerj_brain_unlink,xerj_brain_overview. A XERJ node must already be running;xerj mcpdoes not start one. The published tool schema at/docs/agents/schemas/mcp-tools.jsonhad drifted to six tools against ten served — it is now generated from a realtools/listand gated in CI against drifting again.[profile.quick]and[profile.ci-test]for the build and test halves of the verify loop.[profile.release]islto = "fat"+codegen-units = 1, deliberately the slowest build Rust can produce because it yields the fastest binary — correct for a shipped artifact, wrong for "does this compile?". CI's test job spent 76m05s compiling for 1m52s of tests. Neither profile is ever valid for a released artifact or a published performance number.docs/FAST_BUILDS.mddocuments the remaining levers.The reference-coding toolkit ships in the repo, and a corpus definition can now rebuild a corpus (#319).
xc-corpus.sh,xc-index.sh,xc.pyandSKILL.mdlived only in a gitignored working copy, sogit ls-files | grep xc-returned nothing while the case study,CONTRIBUTING.mdand the landing page all told a reader to run them. They are now intools/xerj-code/, with a!tools/**/*.mdre-include in.gitignore— the blanket*.mdrule was what swallowedSKILL.md, the same failure mode that hid this repo's pull-request template for its entire history.corpus.jsonrecords the full 40-character SHA. The oldrev-parse --shortvalue cannot be fetched from a remote (git fetch --depth 1 origin e449d17→ "couldn't find remote ref"), so a manifest could not pin anything. A manifest carrying a short SHA is now rejected with a SHA-specific error rather than silently rebuilt at the tip.xc-corpus.sh --from <manifest>rebuilds a corpus at its pinned commits — fetch-by-SHA at depth 1 with progressively deeper fallbacks — and an existing clone is moved to the recorded SHA instead of skipped.tools/xerj-code/hub/carries vetted definitions for the four corpora this repo uses. Each entry has areviewblock a human filled in: SPDX expression,adapt-with-attribution/approach-only/mixed, reviewer and date.- Hosting pre-built indexes remains out of scope (on-disk format version,
embedder identity, redistribution of the indexed source), as does the
measurement harness —
CASE_STUDY.mdandSKILL.mdpreviously pointed at aMEASURE.mdthat is not in the repo, and now say plainly what ships.
GET /_ilm/status,POST /_ilm/start,POST /_ilm/stop(#282): the operator can halt lifecycle execution without stopping the node; a stopped engine's tick touches nothing. The flag is in-memory — a restart resumes execution — while the per-index detach tombstone is the durable stop.POST /{index}/_ilm/remove(#282): ES's own detach verb; goes through the same tombstoned detach path as the settings-null route. A literal name that is not an index answers 404 rather than writing a tombstone for a ghost.A dynamically-discovered nested object field is no longer permanently opaque to
GET _mapping/_field_caps(#292, #307). A subfield likemetadata.kindwas already queryable (dotted-path resolution against_sourcedoesn't care about the mapping) but had no way to be discovered by a client that reads the mapping instead of already knowing the field name — indistinguishable from not existing to Kibana/OSD's own field-list UI. Three compounding gaps, all fixed together: dynamic mapping never recursed intoValue::Object({"type": "object"}was a terminal, not real ES/OS default behavior);GET _mapping/_field_capsserved the index-creation-time explicit mapping blob verbatim forever, so any field discovered dynamically after creation — flat or nested — never surfaced, no matter how many documents arrived; and_field_caps's per-field walk only ever registered top-level names, so a nested object's own children were still invisible even once the mapping correctly described them. Measured end-to-end on/_memory's own backing index:GET .xerj-memory-{ns}/_field_caps?fields=metadata.kindreturned{"fields": {}}before, the real entry after. The cluster-wideGET /_field_caps(as opposed to the per-indexGET /{index}/_field_caps) had the same gaps independently and is fixed the same way — the two no longer disagree about which fields exist. Dynamic mapping also now re-inspects nested fields on every document — a later document adding a new nested key merges it into the existing mapping instead of leaving it permanently invisible — andSchema::field_count()counts nested children and multi-fields recursively, closing amax_fields_per_indexbypass where one top-level object could carry unbounded uncounted children. Note the tightening this implies: a dynamic text field with its.keywordmulti-field now costs 2 against the default limit of 500, which matches how ES countsindex.mapping.total_fields.limit. Two bounded residuals from review (a one-shot budget overshoot on a single deeply-keyed insert; array-nested keys deferred to the 100-doc recheck) are filed as #312. Known limitation: this only benefits indices created after this ships. An index (a/_memorynamespace or otherwise) created before it already has its stored mapping written, and that isn't retroactively edited — existing indices need either re-creation or an explicitPUT _mappingto pick up the fix. No automatic migration is included.A crafted filename can no longer forge records on the agent progress stream (the first rc.15 known issue below). Every externally-controlled string — in-flight paths, the paths and error text interpolated into human notes, the terminal line's reason — is stripped of control characters, bidi overrides and zero-width characters and bounded in length before it reaches any progress surface. Measured on a corpus holding crafted names: a run that previously emitted a forged
xerj-done ok=true exit=0 reason=completed2.2 s ahead of its real terminal line now emits one terminal line and no control characters at all. The same sanitisation covers the walker's "skipping unreadable entry" warning, which renders a path to the same stderr.--progress jsonpaces the bar like--progress plain. Thebarfield bypassed the spacing slot entirely: measured at--progress-interval 1, 178 of 178 ticks carried a bar against 18 bars on the plain surface. It is now a string on exactly the ticks that owe a bar andnullin between — 26 of 320 ticks on the same corpus, against 16 on plain.The bar spacing floor is the 15 s the documentation states. A half-tick tolerance made the enforced floor
interval/2shorter — 12.5 s at the shipped defaults. Measured at--progress-interval 10: 11 of 11 same-phase gaps under 15 s (min 10.0 s) before, none after (min 19.3 s)./{index}/_refreshreports segment-publication failures instead of claiming every shard succeeded. The endpoint previously discarded every error returned by the synchronous engine flush and always answered HTTP 200 withsuccessful == total. It now includes the index, status, and underlying reason in_shards.failures, attempts every resolved index, and returns the first failed shard's HTTP status as Elasticsearch 8.13.4 does for refresh.Text field names containing path separators can be flushed and reopened. FTS side-car filenames previously interpolated the raw field name, so a name such as
styles_blocks_core/site-titlebecame a child path and prevented the segment from being published. Unsafe or overlong field names now use one deterministic bounded filename component. Existing portable names retain their byte-identical paths. Each segment has an immutable filename-layout discriminator: absence means the historical raw layout; an explicit marker means the encoded layout. Readers never infer the layout from side-car file existence, preventing partial files and raw names that resemble digests from aliasing another field. Before the discriminator or first encoded side-car is written, the index durably advances its data-directory marker to format 2; older binaries refuse that index instead of silently missing the new layout. A retry after a post-rename marker failure must re-establish durable marker publication before writing the discriminator or encoded FTS paths. Unix confirms the parent-directory fsync; Windows uses a same-directory Win32 write-through replacement rather than claiming its directory-sync no-op is durable. Safe-only indices remain format 1.
Changed
Default
_searchsource payloads omit engine-generated embedding companions (#309). A search with no_sourcenow returns ordinary source fields without<field>_vectorand<field>_vector_chunks— identified from the mapping's embedding config, not by name; measured payload size fell from 211,432 to 21,069 bytes for 5 hits (about 90%). Explicit"_source": true,"_source": ["field"], and include/exclude forms are unchanged, vectors remain stored and used for kNN/semantic/hybrid scoring, and the document-copying paths (_reindex, index clone, enrich execution) request the full source explicitly so copies keep embeddings byte-for-byte. Add"_source": trueto restore the previous complete payload, including vectors. Two residuals disclosed from review: afieldsrequest naming a companion resolves against the already-filtered source and comes back empty under the new default — name it in_sourceinstead (#310) — and the new default arm copies each hit's source even on indices with no embedding fields, a constant-factor read-path cost filed as #311.merge.strategy = "log_structured"is now refused at startup instead of silently running the size-tiered policy (#207). Nothing in the tree ever readmerge.strategy;run_merge_oncealways builds aSizeTieredMergePolicy. An operator who chose a levelled policy for its read amplification got the other one, quietly. This follows thestorage.backendandvector.default_quantizationguards already inConfig::validate.The three dormant
[merge]settings now say so, in the code, in the shipped TOML, on the docs site, and in the log at startup (#207). The issue reported merge I/O rate limiting as "claimed but dormant" and it is: theRateLimiterthat honoursio_rate_mb_per_secis wired only intoxerj-storage's legacyMergeExecutor, which the engine never constructs.min_segmentsandmax_concurrentare unread too — the real per-tier trigger ismin_merge_count, and merge parallelism comes fromXERJ_MERGE_PARALLELISM.MergeConfig::dormant_overridesnames any of the three the operator has moved off its default andxerj-serverlogs each at WARN, so the operator throttling merges to protect query latency finds out instead of guessing. They stay accepted rather than becoming hard errors because the wrong value costs latency, not data, andio_rate_mb_per_sec = 100has shipped inxerj.default.tomlsince v0.1. Wiring a real throttle intorun_merge_onceis the fix that makes this list shorter.engine/xerj.default.tomldisagreed with the defaults it claims to document, in four places.max_segment_mbsaid 5120 against a real 8192,wal_max_size_mb512 against 1024,flush_size_mb256 against 512, anddefault_quantization"scalar8"against"none"— socp xerj.default.toml xerj.toml, the documented first step, silently changed four engine behaviours including turning on 8-bit vector quantization. The file is corrected andshipped_default_config_documents_the_real_defaultsnow diffs every leaf key againstConfig::default(), so it cannot drift again.The docs site had drifted the same way, and now has the same guard.
landing/docs/config.htmlshippedwal_max_size_mb = 512,flush_size_mb = 256anddefault_quantization = "scalar8"in its copy-pasteable[storage]and[vector]blocks while its own DEFAULT table — thirty lines up, on the same page — said 1024, 512 and"none";landing/docs/storage.htmlsaid the WAL rolls at 512 MiB;landing/docs/vectors.htmlcalledscalar8"the default" and listed ascalar4mode that no config or mapping value can reach; andengine/README.mdputflush_size_mbat 256 MiB. All four are corrected, along with the docs search-index blob duplicated across 44 landing pages, which offeredscalar4as a selectable quantization mode. A second guard,the_docs_site_config_page_agrees_with_the_real_defaults, now diffs both halves of that page — every DEFAULT cell in the reference table and every assignment in the example blocks — againstConfig::default(). An example that deliberately differs (the blocks whose point is switching TLS or cluster mode on) has to carry# not a defaulton the line, so the reader is told as well as the test.- Known gap, found while correcting the
scalar4prose and disclosed rather than quietly fixed: the guard readsconfig.htmlagainstConfig::default(), so it covers the config half only. Nothing checks a mapping, and a mapping is not validated either —es_compat.rsmatches"scalar8" | "int8" | "none"on adense_vectorfield'squantizationand falls through_ => {}, so"quantization": "scalar4"(or"binary", or a typo like"sq8") is accepted with a 200, echoed back verbatim byGET /_mapping, and ignored — the field is stored at full-precision f32 while the mapping says it is quantized. Measured through the ES-compat router at this commit. It is pre-existing and outside this diff; it is another instance of the accepted-and-ignored class in #204, is filed with the repro and a proposed 400 as #275, andlanding/docs/vectors.htmlnow says so on the page instead of leaving "not a mode you can select" to imply the value is rejected.
- Known gap, found while correcting the
Four other claims from #207 now match the source.
- Settings count.
config.rssaid 38 in its header and 56 twenty lines down while the count test asserted 60 — and the test asserted it by re-adding a hardcoded sum (5 + 2 + 3 + … - 1 == 38), an identity that could not fail whateverConfigheld. There was a second copy of the same non-test inconfig.rsitself (count_user_facing_settings,61 == 61). Both now serialiseConfig::default()and count leaf keys, section by section. The measured figure is 105 (103 when this work was done; the rc.14server.allow_insecure_network_bindkey made it 104 and #247'slifecycle.tick_interval_secsmakes it 105 — which is the point: it is measured, so it moves with the code), and 38 / 50 / 56 / 60 / 61 / "<50" are corrected inconfig.rs,xerj-common/src/lib.rs,engine/README.md,xerj.default.tomland the feedback responses, along with the stale per-sub-config annotations (limits 3 → 13, storage 5 → 10, embedding 4 → 19, merge 5 → 8, cluster 4 → 5, tls 3 → 4, auth 2 → 3). This closes item 10 ofuser-feedback/ROADMAP-TO-GA.md. 105 versus 3,000+ is still the winning story, told truthfully. - Wire-compatibility test framing.
journey_es_migrationwas described as proving "the same curl commands" work. It calls the Rust engine API directly and never makes an HTTP request. The response-shape assertions are real and stay; the comment now says what the test does and points at the ES-compat YAML conformance suite, which is what actually covers the wire. - OpenAPI spec.
landing/openapi.jsondeclaredversion: 1.0while specifying 7 routes out of the 200-plus the router registers. It is now titled and versioned as a partial spec, and says so.
- Settings count.
autoindexestimates the job on the user's own machine and hands the decision back before it takes their laptop. Phase A already reads and parses every file to sniff and sample it, so it now times that work per format family and turns it into a range with its basis printed next to it —code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s— rather than a constant calibrated on someone else's hardware. A family is priced only from files phase A provably read end to end (whole-file parsers always; streaming parsers only when the file was under the sampling byte cap and stopped short of the record cap; neversqlite, never gzip), and everything else is named underunmeasured_familieswith its bytes instead of being priced at another family's rate. If nothing could be measured there is no number at all and no gate. The two ends are the classical list-scheduling bounds (Graham, 1969) over the phase-B worker count that #240's resource policy chose. The number is a floor, labelled as one on every surface: it covers client-side extraction only, because measuring the server, the network or embedding would mean writing to the index the estimate exists to ask permission for. Measured on a 68 MB source tree the floor was 0.1 s against a real 8.9 s run; on a 793 MB one, 64.2 s against ~350 s. The gate therefore under-asks and never over-asks, and says so where a reader would otherwise mistake silence for a promise.--max-minutes(default 10) stops the run and asks instead of deciding. Past the threshold with no answer, autoindex indexes nothing, writes a JSON decision request to stdout and exits 4 — a code of its own, because exit 1 is already the catch-all for every real failure and an agent must be able to tell "choose something" from "your endpoint is down". The payload carries the estimate and its basis, file/byte counts, the per-band work order, the heaviest directories with real byte counts (flagged when they match the vendored/generated rule), and four options:proceed,fast,narrower,cancel. Answer with--approve <id>(--yes= proceed), which skips the gate;--max-minutes 0disables it. A person at a terminal gets the same facts as prose plus a prompt; a piped or agent-driven run is never blocked on stdin, and an unanswerable prompt is a cancel, never consent.--approve fastreally applies--no-semantic --no-graph, and--approve narroweris refused with instructions rather than accepted and ignored (#204). Thefastoption states no speed-up factor: it reports the datasets and file count it changes and says plainly that this run did not measure the factor. The gate governs the phase-B route. An incremental reconcile of an already committed generation (a--no-graphre-run over a folder that already has one) processes only what changed and publishes from a sealed snapshot, so--max-minutesdoes not apply there — and that route now says so on stderr rather than accepting the flag and quietly not honouring it.autoindexindexes what matters first. Phase B's queue was sorted by size alone — right about scheduling, silent about value, so a user who stopped early got whatever was largest, which on a source tree isnode_modules. Work is now ordered by value band (source and documents → configuration → structured data → logs and line files → vendored, generated and minified paths), with the old biggest-first rule kept inside each band so a large file still runs alongside its band instead of becoming the tail; a single-worker run goes smallest-first, where there is no tail to hide in. One exception keeps the new order from costing wall clock: a file whose own extraction outlasts everything ranked above it (size × workers > the rest) starts first regardless of band. The bands, their file/byte counts and the reason each sits where it does are printed with the plan and included in the decision payload aspriority_order— an unexplained order is indistinguishable from an arbitrary one. Verified live on a 28 MB mixed corpus at two workers: the first second of phase B was spent onsrc/mod_*.rs, and the 1.7 MBnode_modules/**files — the largest in the corpus, and therefore first under the old rule — drained last. Bytes-based progress and its percent are unaffected: reordering does not change the denominator.
[1.0.0-rc.15] - 2026-08-10
Known issues in this release
Both were found by adversarial review after their pull requests merged, so they are present in these binaries. Fixes are in flight for rc.16.
- A crafted filename can forge records on the agent progress stream. The
in-flight path is rendered onto the progress surface without control-character
sanitisation, so a filename containing a newline can inject something that
looks like a genuine
xerj-progressorxerj-donerecord — including a falseok=truecompletion — into the stream this release tells AI agents to parse. Indexing a repository whose filenames you do not control is enough to trigger it. Until the fix lands, do not treat that stream as trustworthy when the corpus is untrusted. - An outer repository's
.gitignorejudges files inside a nested checkout. The ignore lookup walks every layer with no repository-boundary stop, unlike git itself, so a vendored or submoduled repository is filtered by rules that have no authority over it. Files you expect to be indexed may be skipped.--no-ignorebypasses it.
Also unfixed and worth knowing: --progress json does not pace the bar (a JSON
consumer sees roughly ten times as many bar lines as a plain consumer), and the
documented "at most one bar per 15 s" floor is really 12.5 s.
Added
Index lifecycle policies are executed, not just stored (#199, contributed by @Vinz2168). A single internal engine modeled on OpenSearch's ISM state machine — named states, ordered actions, ordered transitions — exposed through two REST surfaces: native ISM at
_plugins/_ism/*and the Elasticsearch-shaped_ilm/*.spawn_lifecycle_manager()runs the tick onlifecycle.tick_interval_secs(default 300s, matching OpenSearch ISM's own job interval), and the managed-index cursor survives restart. Retention now actually deletes and rolls over instead of being acknowledged and forgotten. One documented divergence:min_ageis measured from when an index entered its current state rather than from rollover time, so a policy that rolls over and has a downstream phase meant to be measured from the rollover will advance on a different clock than Elasticsearch uses.xerj autoindexhonours.gitignoreand.xerjignore(#276). Nested ignore files and negation patterns follow git's own precedence, with build-output defaults (node_modules,vendor,target,dist,build,.venv,__pycache__) on top;--no-ignoreand--no-default-ignoresturn them off. Pointing xerj directly at an ignored path still indexes it — an explicit instruction beats a rule the user did not write for this purpose.--dry-runreports what was skipped and by which rule, because a user whose files did not appear needs to know why. Measured on this repository: 274,826 files walked in 1.18s becomes 1,385 in 0.25s.Progress is relayed to an AI agent as a drawn bar. The stream surface carries a rendered bar alongside its machine-readable
key=valuefields, so an agent drivingxerj autoindexon someone's machine can show a real progress line rather than going silent for minutes.--progress jsonkeeps a single parseable stream;--quietstill writes nothing on either.
Fixed
Tied scores have one total order — a bounded page is a prefix of the full page (#191). A
size:5page and asize:1000page disagreed about which tied document came back. The memtable's bounded candidates are now ranked by the same page key as the segment path, so the two agree. The remainingconstant_scoreand post-sort re-sort cases are tracked separately in #270.date_histogramno longer aborts the process on a multi-bytetime_zone(#272, contributed by @Vinz2168).aggs::parse_time_zone_offsetsliced a&stron a byte index that could land mid-character, and withpanic=abortthat took the whole node down — an unauthenticated request was enough.The legacy path-parameter form of scroll continuation works again (contributed by @Vinz2168), and
_cat/*?format=jsonplus_data_stream/**wildcards are supported (also @Vinz2168) — both are shapes real Elasticsearch clients send.
Changed
autoindexparses each PDF once per fresh run instead of twice. Phase A already paid a complete parse for every PDF —extract::extractroutesFamily::Pdfstraight to the isolated worker and drops the sampling limit, so only delivery ever stopped early — and phase B then spawned the worker again for the same bytes. Phase A now retains each validated worker response in an anonymous temporary file under--state-dirand phase B replays it. Reuse is an optional accelerator, never correctness-critical state: it is bounded to 384 MiB of retained-plus-in-flight bytes and to an artifact-handle cap derived from liveRLIMIT_NOFILEand open-descriptor counts, both re-probed at every admission; a 4 GiB-or-half-free filesystem floor is reserved for the journal and phase-B staging first; and every artifact is verified (physical length, content digest, JSON decode, worker-protocol identity) before a single record is published. Anything that cannot be measured conservatively declines and reparses — which is why the optimization is Linux-only today: other platforms have no live descriptor evidence. A restart has a frozen plan but no trusted handle, so it parses again; the spool is deliberately not journal state.--jsongains apdf_extraction_reuseblock so the behaviour is observable rather than inferred from timing. Original work by Leonid Bugaev (@buger).
Changed — autoindex refuses reruns that would strand documents
- Deleting an indexed file and rerunning
xerj autoindexnow fails instead of exiting 0. Nothing in the pipeline removes the documents, aliases, graph edges or catalog entries that the deleted file published, so a rerun that ignored the deletion left them live and searchable with no source file behind them. The rerun is now refused before any remote call other than the endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph or catalog write is attempted, and the journal is not appended to. The error names the removed files and their content keys — the first ten, then an… and N moretail, since a whole unmounted subtree can vanish at once — and gives three recovery routes: restore the files and rerun, rebuild in place by deleting the named indices and the state directory, or rebuild under a new state directory, prefix and brain.--freshis refused for the same case, because it does not delete those documents either.--jsonemits the same facts — every removed entry, uncapped — asxerj.autoindex.unsupported_sync_delta.v1on stdout with exit 1. Deleting a file that was only ever skipped is not refused: a junk file publishes no documents, and the stale junk-catalog sweep removes its one catalog row, so nothing is stranded. - Files added after a plan was frozen are now called out on stderr. They
are still not indexed by a rerun that resumes an existing plan — the plan is
a crash-resume boundary, not a folder-sync generation — but the run now
lists them, records them as skipped (exit 3, completed-with-junk) and points
at
--fresh, which rebuilds the plan in place and picks them up. Adding or changing files and rerunning keeps working; only removals are refused. xerj brainno longer turns an absent or zero node-count probe into an automatic reset. Journal/server disagreement now fails with the journal, URL, prefix, and brain identity plus recovery instructions, including the explicit--freshrerun for a genuinely wiped data directory. Probe transport and malformed-response failures remain errors instead of being reported as an empty destination. A nodes index that was deleted out from under a surviving brain meta doc reads as absence and reaches that recovery text, rather than surfacing as a raw HTTP 404.--freshstill recovers a state directory whose journal cannot be parsed. The rerun gate reads the durable plan before the run starts, and a plan that is malformed, or recorded for a different root/URL/prefix, is fatal to a resume — but not to--fresh, which deletes that journal unread. The preflight is now no more fatal than the open it precedes: under--freshthe unreadable plan is reported on stderr and rebuilt from the current folder. Without--freshthe refusal is unchanged. Because the removal gate has no comparison basis in that case, the warning says so: documents already published for files that are now gone cannot be identified from an unreadable plan and are not deleted.- Refusal and skip listings are capped at ten entries plus an "and N more"
tail. Unmounting a bind mount under an indexed root vanishes every content
group at once, so the uncapped listing was one rendered entry per journalled
file.
--jsonstill carries every entry.
Fixed
An index whose metadata will not parse is refused instead of silently reopened with an empty mapping (#202).
Index::opentreated "this file is not there" and "this file does not parse" as the same thing:load_schema/load_settingsmapped ENOENT, EACCES, EIO and malformed JSON alike to one anonymous error, and the caller answered all of them with a fresh dynamic mapping. Measured on a field deliberately mappedkeyword: afterschema.jsonwas truncated to half its bytes the index still openedOkwithfield_count = 0, the field came backNone, and one further document re-inferred it aslong— a mapping silently replaced by a different one, with nothing in any response saying so. Absent stays absent (indices predating create-time schema persistence legitimately have noschema.json); present-but-unparseable now fails the open with an error naming the file.es_mapping.json— the full-fidelity mapping behindGET /_mapping— was logged-and-ignored on a parse failure and now fails the index too, on the boot scan, on snapshot restore and on a retry.Visible on upgrade: a node whose data dir already holds a corrupt sidecar now boots red with that index unserved, where it previously came up green with an empty mapping. It joins the failed set #206 introduced, so every surface that set feeds already reports it — measured on a node holding two user indices,
victimwith a truncatedschema.json:GET /_cluster/health -> red, unassigned_primary_shards: 1 GET /_cluster/health?level=indices -> victim red, unassigned_info.details = the absolute path of schema.json and the parse error GET /_cat/indices -> "red open victim <uuid> 1 0 0 0 0b 0b 0b" GET /_cluster/indices/failed -> the reason, plus the retry and delete calls PUT /victim, POST /victim/_doc, POST /victim/_search -> 503 no_shard_available_action_exception carrying that same reason GET /health/ready -> 200 "ready (degraded): 1 of 2 indices serving, 1 failed to open; see GET /_cluster/indices/failed" POST /healthy/_search -> 200The readiness counts above are the two-index test router's. A running node also holds its internal
.xerj_*indices, so the same condition on the shipped binary readsready (degraded): 15 of 16 indices serving, 1 failed to open— the 200 and its meaning are unchanged, only the totals move with what else the node holds.Recovery is the three doors #206 added. Two were measured end to end on the shipped binary: repair the file and
POST /_cluster/indices/failed/{index}/_retry(503 while it is still torn,200 {"reopened": true}once it is not, health back to green), andDELETE /{index}(200, directory removed, health back to green). The third — restore the index from a snapshot — is the path this change touches (a successful restore now clears the recorded failure, which it has to or health would stay red after the repair worked); it is covered by an engine-level test rather than measured over HTTP. The node stays in kubelet rotation as long as any index is still serving.Concurrent sidecar writes can no longer manufacture the torn file the reader now refuses.
write_file_atomicstaged every write inpath.with_extension("tmp")— one shared name for all writers of that file — so two concurrent writers of one sidecar (PUT /{index}/_settingsracing another settings update, two API-key mints forapi_keys.json) both opened itO_TRUNC, interleaved their bytes, and each renamed it into place. The rename is atomic; the content being renamed was not one writer's. Measured with the old shared name, 4 threads × two different-length settings bodies × 200 rounds, over four runs: 86–294 of 800 writes failed with ENOENT (the loser renaming a file the winner had already moved) and, with those errors swallowed as the callers did, 1–16 of 200 rounds left an unparseablesettings.json. The counts are race-dependent and vary with machine load; what does not vary is that both are non-zero on every run with the shared name and 0 and 0 on every run with a unique staging name. Each write now stages in its own sibling file (<file>.tmp.<pid>.<seq>) in the target's directory and removes its own debris on failure — which forapi_keys.jsonis key material.update_settingsalso writessettings.jsonthroughwrite_file_atomicrather than a plainfs::write; it was the last non-atomic sidecar writer left.indexon the mapping is now honoured instead of silently ignored (#204)."index": falsewas accepted byPUT /{index}, echoed back verbatim byGET /{index}/_mapping, and then had no effect at all: the field kept a full inverted index andmatch,term,match_phrase,prefixandwildcardagainst it all returned the document, where Elasticsearch answers 400. It is the sibling of thedoc_valuesfix in rc.12 and follows its shape.The line is drawn where ES draws it. Since 8.1,
"index": falseon a keyword / numeric / date / boolean / ip field keeps the doc-values column and stays queryable —MappedFieldType.isSearchable()is "has postings or has doc values" — so those queries keep working here, unchanged. Only a field with neither (atextfield, or anything with an explicit"doc_values": false) is unsearchable, and a query naming one is now rejected with ES's own sentence,Cannot search on field [f] since it is not indexed nor has doc values., as a 400search_phase_execution_exception/query_shard_exception. The check lives in one place —search_inner— and these surfaces were each measured reporting the rejection rather than absorbing it, including the three that were swallowing it:_search,_counton all four selectors (single index,logs-*, a comma list and the cluster-widePOST /_count),_msearch,_msearch/template,_explain,_delete_by_queryand_rank_eval._explainanswers the 400 instead of a confident"matched": falsefor a query it never ran;_msearchand_msearch/templaterender it as a per-response 400 envelope withtypeandroot_cause, leaving the sibling sub-requests in the batch untouched; and_rank_evalrecords the refused request underfailuresinstead of dropping it fromdetailsandfailures. It still publishes ametric_score, and that number is still the mean over the requests that actually ran — which is ES's behaviour too — but the batch no longer shrinks silently underneath it: a two-request batch in which one is refused now answersdetails: {good},failures: {bad},metric_score: 1.0(measured), so a relevance gate reading a clean score can see what did not contribute to it.A multi-index
_countfollows ES's broadcast rule rather than failing outright: the failure status is returned only when no index answered (RestStatus.status, semantics only), so/t/_countand alogs-*whose indices all carry the offending mapping both 400 exactly as_searchdoes, while a mixed selector returns 200 with the partial count and the refusal visible as_shards.failed: 1plus a_shards.failuresentry carrying thequery_shard_exception. What it will not do any more is what it did before: reportcount: 0with"successful": 2, "failed": 0.existsis unaffected and still returns zero hits rather than an error, which is also what ES does.The multi-field query types follow ES's own split rather than a blanket rule. An explicitly named field is rejected like any other named field —
multi_match/simple_query_string/query_stringwith"fields": ["note"], andquery_stringwith"default_field": "note"— while a wildcard spec such as"fields": ["*"]is never an error, because ES expands a pattern over the searchable fields and silently drops the rest (QueryParserHelper.resolveMappingField). Before this, the same user intent got two different answers: thesimple_query_stringform was rejected (the parser lowers it to amatch) while themulti_matchform returned the document through the unsearchable field.Reaching the
query_stringhalf of that meant fixing a second accepted-and-ignored key in the same family:query_string'sfieldsarray was never read by the parser, so{"query":"x","fields":["title"]}searched every field, and — onceindex: falsebecame a real rejection — disagreed with{"default_field":"title"}, which is the same request written the other way.fieldsnow selects the targets for a clause that named no field of its own (adis_maxover one leaf per field when there is more than one,^boosthonoured) — the same jobsimple_query_string'sfieldslist was already doing. A clause that does name its own field keeps it, as in ES.The field is dropped from the full-text index at flush and merge only where the stored-doc fallback is equivalent — a non-indexed
textfield. An exact-typed field that kept its doc values keeps its whole-value postings, because the fallback scan splits on non-alphanumerics and would start matching192.168.0.1against192.168.0.2. So no byte-saving claim is made for this change: for those types the footprint the option implies is knowingly forgone rather than bought with wrong answers.Known gaps, measured and left open rather than claimed shut. Aggregations and sort do not inherit the check: only the
queryclause is walked, so{"size":0,"aggs":{"a":{"terms":{"field":"note"}}}}still answers 200 with a bucket built from_source, and{"sort":[{"note":"asc"}]}still answers 200. ES rejects those too, but for an unrelated reason and with an unrelated sentence —Fielddata is disabled on [f] in [i]…, which it raises for anytextfield whether or notindex: falsewas declared — so reusing this error there would misreport a wider divergence as this one. The field-less arms of the stored-document scan are schema-free, walking every_sourcekey, so a token living only in an unsearchable field can still match when no field is named:query_string {"query":"…"}andmore_like_thiswith nofieldsboth still return the document. (Their named forms are rejected —more_like_thiswithfieldslowers to abool.shouldofmatchat parse time and inherits the check.) A wildcardfieldsspec reaches that same field-less arm:simple_query_string/query_stringwith"fields": ["*"]lowers to the"*"placeholder and is answered by the schema-free scan, so it can still match through an unsearchable field. It is never an error, which is ES's rule and deliberate — but it is not "answered over the searchable fields" either, and that is a pre-existing property of wildcard expansion, identical onmain. (multi_matchwith a pattern is different: it is answered by the FTS projection, which does applyisSearchable(), and finds nothing in an unsearchable field.) The opaquequery_stringfallback still ignores a multi-fieldfieldslist: a query string the parser declines to lower keeps the whole string for the FTS path, where a one-elementfieldsis carried across asdefault_fieldand a longer list has nowhere to go. And_validate/querynever opens the index — it answers from the parse alone, so it reports{"valid": true}for a body_searchrefuses. That too is unchanged frommain, but it now disagrees with_search, so it is written down rather than left to be discovered. Every one of these gaps is pinned by an assertion inindex_false_is_honoured::documented_gaps_are_still_open_and_still_documented, so closing one fails the test that says it is open.
Added — incremental --no-graph corpus reconciliation
xerj autoindex --no-graphnow reconciles a changed folder instead of re-deriving it. A--no-graphrun commits a durable corpus generation: a manifest of content groups (stable group identity, content digest and byte length, canonical path plus aliases, dataset assignment, validated output counts) plus a sealed source snapshot of the prepared records. A later run projects the current inventory onto the committed manifest and publishes only the difference — additions, content changes, deletions, renames, junk transitions and duplicate-alias moves all converge, and a no-op re-run issues no data bulk and appends no new generation. A run interrupted mid-generation replays from its own sealed snapshot, so the result never depends on whether the source tree changed after the crash. Dataset and mapping identity is frozen at generation 1: a file that would need a new dataset or a mapping change is refused before any remote mutation rather than silently widening the schema.Scope, plainly: only
--no-graph. Graph-enabled runs are unchanged, and each changed generation still copies and prepares the full corpus (O(N) staging) — the win is in what gets published and verified, not yet in what gets read.--snapshot-max-gb(default 64) caps the logical staged payload; it is a payload budget, not a disk-space or peak-memory guarantee.Original work by Leonid Bugaev (@buger).
Junk keeps the contract it always had on the generated path: recorded, never fatal. A file that cannot be read or recognised (
plan.junk_files) and a record that no dataset assignment accepts (stats.junk) both used to abort a--no-graphrun outright — the first withplan has no assignment for <file>, the second withdurable preparation of <file> produced N junk records. A single zero-byte file was enough to make a whole folder unindexable. Both are now counted instead: the sealed prepared artifact carries its junk-record count, the manifest group carries it into the generation, and the catalog's per-filejunkand run-leveljunk_records_totalreport it instead of a hardcoded zero. Such a run exits 3 ("completed with junk"), the same signal the legacy path publishes — the generated path previously returned a flat0and lost it.--dry-runis honoured on an already-generated state directory. It was evaluated only after the pending-replay and reconcile branches had already published and committed, so on any state directory past generation 1 the flag was accepted, silently ignored, and the destination mutated. It is now decided before either branch: it prints the plan a real run would act on — the projected reconcile plan, or the sealed plan of a pending generation — and returns without opening the journal for write, without snapshot GC, and without a single bulk request.
Changed — autoindex --fresh is refused on a durable generation
--freshno longer silently destroys a committed corpus generation.--freshdeletes the resume journal, and the snapshot GC that follows every journal open then sees an empty protected set and removes every sealed snapshot — so on a generated state directory--freshwould discard the committed manifest, the pending replay evidence, and the alias/path/ stale-record knowledge, while cleaning nothing at the destination. It is now refused up front, naming the generation that blocks it, and pointing at the plain re-run (which reconciles the change) or at an isolated rebuild with a new--state-dirand--prefix.Unchanged for everyone else. On a legacy (non-generated) journal — including every graph-enabled corpus and anything
xerj brainwrites —--freshbehaves exactly as before: it ignores the journal and restarts, which is whatxerj brain's documented self-heal for a wiped data directory depends on.
Migration — pre-generation --no-graph state directories must be rebuilt
- A
--no-graphstate directory written before this release cannot be adopted as generation zero, and the first--no-graphrun against it will refuse with a followable rebuild command. This is a deliberate, versioned format boundary (sync::GENERATION_FORMAT_VERSION = 1), not a validation failure: a pre-generation resume plan records no stable group identity, no content byte length and no validated per-group output counts, so no amount of evidence in it could reconstruct a manifest group. The refusal prints the exactxerj autoindexargv for an isolated rebuild into a new--state-dirand--prefix; the old target and the sharedautoindex-catalogare left alone and require explicit, validated cleanup once the new target is verified. Nothing is migrated in place and no destination data is touched by the refusal. Graph-enabled state directories are not affected.
[1.0.0-rc.14] - 2026-08-10
Changed
One documented core and memory policy, and the worker knobs are live (#240). XERJ had no single answer to "how much of this machine may I take?", and the scattered answers it did have disagreed with each other — and, on Darwin, with the OS.
engine.flush_workers,engine.merge_workersandengine.search_workerswere documented but inert; they now drive real pool widths, and a value of0refuses startup instead of being silently reinterpreted.--bulk-mbabove 24 and--workers 0are likewise refused rather than clamped, so a request the engine will not honour fails loudly. Only hand-written configs are affected: the shippedxerj.default.tomlhas no[engine]section.One 16 MiB file no longer costs 65 GB of RSS (#239).
split_sectionswas quadratic in both time and memory: a 16 MiB text file with no blank line needed 65.6 GB of peak RSS and 16.3 s to produce 16.8 MB of sections. Now linear — the same input takes 8.3 ms and 35 MB.BREAKING —
server.bind_addressnow defaults to127.0.0.1, and a cleartext node refuses to publish itself to the network (#228). The old default was0.0.0.0while TLS is off by default, so an out-of-the-box node accepted its admin API key over plain HTTP on every interface the host had — verified on a stock boot, wherecurl -H "Authorization: ApiKey …" http://<lan-ip>:9200/…answered200and the same URL overhttpshad no listener to hand shake with. Auth being on did not help: the credential is the thing on the wire.A fresh node is now reachable from its own host and nowhere else. Exposing it is two statements, not zero: set
server.bind_address(or--bind/XERJ_BIND_ADDRESS), and — whiletls.enabled = false— also setserver.allow_insecure_network_bind = true(envXERJ_ALLOW_INSECURE_NETWORK_BIND). Without the second, startup exits non-zero before the data directory is created or a first-run admin key is minted.--insecuredoes not evade it; it clearstls.enabled, which is exactly what the check keys on. Same fail-closed shape as #229, whose TLS-on gRPC check is untouched and cannot be relaxed by this opt-out.Upgrading: a deployment that relied on the old default becomes unreachable from other hosts until it sets both; one that already wrote
bind_address = "0.0.0.0"without TLS now fails to start with a message naming the setting that unblocks it. The shipped Docker image, compose file and Helm chart set both, because a container's network namespace is the boundary and its published port is where TLS belongs.Two documentation claims that were false as written are now true: the CLI and security pages both said
--insecure"refuses to run with a non-loopback bind", and it did not — a--insecurenode on0.0.0.0accepted an unauthenticatedPUT /leak/_doc/1from the LAN (201). The startup banner also prints the bind address on every listener line, so a loopback node and a world-facing one no longer look identical.A
bind_addressthat is not an IP literal is rejected first, ahead of both exposure checks, with the fault it actually has. Host names have never been resolved — the pre-#228 code parsed"{bind}:{port}"as a socket address and failed on them too — but the exposure predicates fail closed on anything they cannot parse, so without that orderingbind_address = "localhost"was refused with a message asserting that localhost "is not loopback" and would "serve plain HTTP on a network-reachable interface", and pointed at an opt-out that fixed nothing: setting it let the boot run on to the bind and fail there instead, after the data directory, the.xerj_*system indices, the master key and a printed first-runadmin.keyalready existed.
Security
API key secrets are no longer stored in the clear (#201).
api_keys.jsonheld the secret verbatim. 0600 plus atomic rename is right for a secret a process owns, but a file has more readers than a process: a backup, a snapshot, a container layer, a support bundle, a decommissioned disk. Secrets are now stored as a salted SHA-256 ($ssha256$<salt>$<digest>) and compared in constant time, with existing plaintext records migrated on load. The fast hash is deliberate: the secret is two v4 UUIDs — 244 bits of CSPRNG output, never human-chosen — so offline guessing is out of reach whatever the hash costs, while this comparison runs on every authenticated request, where an Argon2id would add tens of milliseconds per request and hand out a free CPU-exhaustion DoS. The same release makes_security/_authenticatereport the key's real identity instead of a hardcodedsuperuser, and gives the audit chain durable storage.The node refuses to start when TLS is on but gRPC would answer in cleartext off-loopback (#229).
tls.enabled = trueencrypted two of the three data-plane listeners; the third never participated, because the gRPC server is built without tonic'stlsfeature. Nothing surfaced it — gRPC clients connected and worked identically either way — so an operator who enabled TLS and bound a network interface got no error, no failed handshake and no symptom, while API keys and document bodies crossed the network in the clear. Auth was never the gap; confidentiality was. Boot now fails closed unlesstls.allow_insecure_grpc_h2csays otherwise.
Fixed
multi_matchreturns the same hit set before and after_flush(#218). The memtable and segment paths disagreed on multi-token semantics, so flushing an index could change which documents a query matched — verified live on a one-document index where an OR query returned 0 hits before_flushand 1 after, with ES returning 1 throughout.Dynamic string fields get Elasticsearch's default
.keywordmulti-field (#209). Dynamic mapping inferred every string as baretext, soGET _mappingnever showed a.keywordsub-field and_field_capsnever listed one. Anyone arriving from Elasticsearch — or any tool that reads_field_capsfor field discovery, Kibana included — could not see thatcategory.keywordexisted.min_scoreis honoured atsize:0, and an external scalar N keeps its ghosts (#193). Asize:0 + min_scorebody never materialised a hit, so the threshold had nothing to filter andhits.totalreported the raw match count — measured at 65 where the same query atsize:100counted 5. The tie-break comparator item from that issue is deliberately not included here; it belongs with #191.Index blocks can be removed,
read_only_allow_deletemeans what its name says, and a blocked write answers 403 rather than 500. Three compounding defects:PUT /_settingswrote only the display-side map while enforcement read the per-indexsettings.json, so a block was acknowledged, invisible toGET /_settings, and — with_blockregistered PUT-only — removable only by restarting with a hand-edited file. The one block that fires automatically did the inverse of its name, and clients hitting any of it were told the server had faulted.llms.txtno longer tells agents six things the binary does not do. The "Running an index for a human" section is executed, not just read: an agent follows it literally and then reports the result to its user. Corrected: the job-size line is not the first line printed;--dry-rundoes not write nothing (it creates and locks the state dir and appends a journal record); a dry run reuses a frozen plan instead of re-costing the job, so it is not a floor for the real run; exit1is the catch-all for every error, not "endpoint unreachable";statusneeds the same explicit--state-dirthe run used; and--workers/--pdf-workersbound the client-side extractor, not the server (with--pdf-workerscapped to 1–4).A failed index can now be inspected, deleted and retried without stopping the server (#206). An index directory that refused to open at boot was recorded in a map that only
Engine::healthever read. Live-reproduced with a corruptsnapshot.json: it was absent from_cat/indicesand_cluster/state,DELETE /{index}answered404 index_not_foundand left the bytes on disk, and the only recovery was to stop the server and edit the data directory by hand. A failed index is now a real state: listed asredin_cat/indices, present in_cluster/stateas anUNASSIGNEDprimary withALLOCATION_FAILEDand the verbatim open error, enumerated with its reason byGET /_cluster/indices/failed, reopenable viaPOST /_cluster/indices/failed/{name}/_retryonce the cause is fixed, and deletable through the ordinaryDELETE /{index}. Searches, writes and creates against it return503 no_shard_available_action_exceptioncarrying the reason instead of a404that claimed it did not exist, while the metadata surfaces —GET /{index},HEAD /{index}(theindices.exists()every ES client calls),_mapping,_settings,_cat/indices— report it as an index that exists, the way ES answers those from cluster metadata rather than from a shard. Before this,HEADsaid the name was free and the followingPUTsaid it was unavailable.Two related defects in the same report are fixed with it. Readiness no longer hard-fails on a partly degraded node —
/health/readyreturned503for any red status, so one broken index pulled a pod holding 200 healthy ones out of service permanently; it now reports200 ready (degraded)while the node can still serve something and503only when every index it holds failed to open. And_cat/healthno longer prints a hardcodedgreen— the one health surface an existing ES dashboard points at was the one that could not report a broken node; it and_cluster/healthnow count an unopenable index as an unassigned primary, agreeing with/v1/healthand/v1/cluster/health.Because
redis now reachable on_cluster/healthfor the first time, the two conditions that gate on it are consulted rather than assumed:wait_for_active_shards=allandwait_for_status=green|yellowboth answer408withtimed_out: trueon a red node instead of200 timed_out: false.GET /_cluster/health?wait_for_status=green&timeout=30sis the bootstrap gate every docker healthcheck, CI wait loop and Kibana startup uses; a red node used to sail straight through it. A green or yellow cluster is unaffected.A
DELETE /{index}that could not remove the bytes no longer strands the index. The open-index path pulled the handle out of the engine beforeremove_dir_all, so a removal that failed (read-only mount,EACCES) freed the name while the directory survived: no handle, no failed-index entry, nothing on_cat/indices, and the nextDELETEanswered404— the same dead end as #206, reached from the other side. The handle is now restored on failure, so the index stays in service and addressable and the operator can retry the delete once the cause is fixed.Index templates, ingest pipelines, data streams and ILM policies survive a restart (#203). Index templates, legacy (v1) templates, component templates, ingest pipelines, data streams and ILM policies lived only in memory:
PUT /_index_template/logsanswered{"acknowledged": true}andGETanswered 404 after the next restart, and the next index that should have matched the template was created without it — with no error anywhere. All six are now persisted together in<data_dir>/cluster_state.json, written atomically (tmp → fsync → rename → fsync the parent directory) so the write is committed at the request, not in a shutdown hook, and restored inEngine::newbefore the listeners come up. A write that fails is rolled back in memory and answered with a 500 instead ofacknowledged. Restored pipelines are recompiled, not just re-read — storing the definition alone would have left?pipeline=xaccepted and silently inert after a restart.Four defects found while verifying it, each reproduced first:
- Concurrent management writes corrupted each other (all flushes staged
through one fixed
.tmppath — 32 parallel template PUTs returnedstore_exception). Snapshot-and-write is now serialized. - A rollover interrupted between "backing index created" and "generation
persisted" wedged the data stream forever: every later rollover recomputed
the same name and got
409 resource_already_exists_exception. Boot now adopts the highest generation actually present on disk, warns, and persists the repair once. - A
cluster_state.jsonthat could not be read (EACCES after a uid change on a container volume, a backup tool's chmod, EIO) came up as empty maps, and the next management write renamed a snapshot of those empty maps over a file whose bytes were perfectly good. The load failure now latches: every management mutation is refused with a 500 naming the file, and the file is not touched, until a boot loads it cleanly. The corrupt-parse arm refuses too. The later format-compatibility fence above tightens that diagnostic boot further: it creates nocluster_state.corrupt.jsoncopy. DELETE /_data_stream/<name>recorded the removal before destroying the backing indices, so a crash in that window stranded.ds-<name>-00000Ndirectories that no data-stream API could reach — GET and DELETE answered 404 whilePUT /_data_stream/<name>answered409 resource_already_exists_exceptionpermanently. The order is reversed: the removal is recorded only once every backing index is gone, and a backing index that cannot be deleted now aborts the DELETE with a 500 (it was previously swallowed and answeredacknowledged) so the operator can retry. Read that 500 as "did not finish", not as "nothing happened": a multi-generation stream still loses every backing index that could be destroyed, the stream itself stays addressable, and the retry after the cause is fixed completes the delete.
Known gaps, deliberately not closed here.
.ds-*indices that no data stream claims — left by a data dir written before this change, or by a DELETE interrupted by an older build — are not cleaned up or adopted automatically; boot now names them in a warning and the recovery isDELETE /<backing-index>per index, because an orphan may hold the only copy of real data. Andaliases.jsonstill has both swallow shapes that were fixed here forcluster_state.json: an unreadable aliases file is swallowed at boot and overwritten by the next alias write, and a failed alias write is only logged, so an alias can survive on disk after the stream it named was deleted with a 200. Both are pre-existing behaviour, not introduced here or made worse here, and are tracked separately.- Concurrent management writes corrupted each other (all flushes staged
through one fixed
autoindexno longer leaves immortal catalog entries for skipped files (#238). A file added after the resume plan was frozen is skipped and reported in the catalog — but that report was written from a per-runVecthat nothing durable remembered, so once the file left the corpus no run could ever remove itsfile:{key}document. The catalog is the data map everymap,statusand agent query reads, and it kept advertising files that were gone. Skipped files are now recorded in the durable plan and swept from the catalog when they leave the corpus. The sweep is safe to do completely because a skipped file is never indexed and never enters the graph corpus — that one catalog document is its entire live footprint. Ordering is deliberate: the plan entry is added only after the document is written and dropped only after the delete has landed, so a failed catalog bulk is retried by the next run instead of stranding the document. This does not implement add/change/delete reconciliation for indexed files, which remains open.Repeated
autoindexscans keep agent-facing map metadata durable. Dataset source bytes, parser-junk counts and coercion-drop notes now derive from committed per-file journal records instead of invocation-local worker counters, so an unchanged resume does not overwrite them with zero. Run timestamps now distinguish invocation start (started) from summary generation (summary_generated_at), andjunk_records_totalon the run document is the same durable sum the per-datasetjunk_recordsreports rather than an invocation-local counter that read0after a no-op resume. Existing catalogs whose historicalstartedfield was dynamically mapped as text remain usable: the additive mapping upgrade no longer attempts an incompatible text-to-date type change. Concretely, this is catalogs written by v1.0.0-rc.4 — the one release that hadautoindexbut not yet dynamic ISO-date inference (added in rc.5). Measured against a live engine, addingstartedasdateto such a catalog is refused 400mapper_parsing_exception("field [started] already exists as [text], cannot add [date]"), which aborts the invocation before any document work. Catalogs written by rc.5 or later already inferredstartedasdateand were never affected.Three consequences worth knowing before you upgrade:
bytesis redefined. It used to be the bytes the latest invocation processed, credited to the first dataset a file was assigned to. It is now the complete canonical source size, counted once in every distinct dataset that source is assigned to. Per-dataset values therefore get larger, and summingbytesacross datasets can exceed the corpus size when one source feeds several datasets — exactly as one source already contributed to several dataset-local file counts. It is a dataset-local source footprint, not a partition of physical storage.- Journals written before this release carry no per-dataset coercion
record, because the new
dropped_by_datasetfield defaults to empty when an older journal is replayed. Byte and junk counts reconstruct fine (their journal fields already existed), but a dataset whose files are all unchanged loses itsN field values dropped by coercionnote on the first resume after upgrading, and only regains it once one of its files is reprocessed. Re-run with--freshif the note matters more than the rescan. junk_records_totalis narrower than the number it replaces, and one class of failure has left it. It used to be an invocation counter that also folded in records the backend refused per bulk item; it is now parser junk only, soxerj autoindex map's header counts exactly what the per-datasetjunk_recordsvalues count. No signal is lost in practice: a single per-item rejection already aborts the run withautoindex stopped with bulk/backend failures, before any run document or map is written, so the folded-in number was never observable in a map. That abort message now also states how many records were refused, which the first-five error sample could not convey.
The installer is now fail-closed on checksum verification, not just on checksum download.
landing/getprintedwarning: sha256sum/shasum not found — skipping checksum verificationand installed anyway — which is not what "refusing to install an unverified binary", three lines above it, implies. It now searchessha256sum,shasum,openssl,sha256,busyboxandcksum, and stops if none is present. A machine with no hashing tool at all can still proceed, but only by settingXERJ_INSECURE_SKIP_CHECKSUM=1— an explicit choice, made by the user, not a silent default. The downloaded checksum is also validated as 64 hex characters before use.XERJ_LIBC=gnumakes the glibc Linux builds reachable from the one-line installer again.landing/get:61unconditionally overrode the target withunknown-linux-muslafter thecasearm had setunknown-linux-gnu, so the script could never request a linux-gnu asset — while the code read as though it could. musl remains the deliberate default (static, no glibc-version floor, and xerj links jemalloc so musl's allocator is not in the path), and the comment now says so;XERJ_LIBC=gnuopts into the glibc build.
Added
autoindexreports real progress, an honest percent and an ETA — no run is silent (#241). Between 47% and 64% of a typical run produced no output at all, which is the worst possible behaviour for the two audiences that matter: a person watching a laptop they cannot use, and an AI agent that has gone quiet on that person's behalf. Progress now covers phase, items done and total, rate and ETA, in both modes — a terminal gets a live redrawn line, a pipe gets periodic structuredxerj-progresslines an agent can parse, and every run ends with a singlexerj-donesummary. The percent is bytes-based so a large file cannot make it stall at 99%, the first ETA is withheld for five seconds and labelledroughuntil there is enough evidence for it, and the line names the file currently being waited on.--quietstill produces exactly nothing on either stream, and--progress noneis honoured everywhere — the resource policy's own startup notes were moved onto the same surface so they cannot leak into--progress json's single parseable stream./getand/get.ps1are counted.functions/get.js(a Cloudflare Pages Function) serves the installer straight out oflanding/get, unchanged, and records the request. The install path had never had a counter of any kind —curlruns no JavaScript, so no page beacon could ever observe it, and the number of installs was simply unknown. It records timestamp, country, User-Agent and a coarse OS guess; it records no IP address, no cookie and no identifier of any kind, so two installs by one person cannot be distinguished from two installs by two people. It is fail-open: if the storage binding is missing or the write throws, the installer is still served. No telemetry was added to the xerj binary, and none is planned.metrics/release-downloads.jsonl+ a daily GitHub Action. GitHub reports releasedownload_countas a running total and keeps no history, so the one uninflated adoption number this project has could be read but never trended. One committed line per day turns it into a series.scripts/adoption-snapshot.shprints the funnel on demand, including which repo-level numbers are contaminated and why.
[1.0.0-rc.13] - 2026-08-08
Security
- All 13 open Dependabot advisories against
engine/Cargo.lockclosed (#220). Twelve by lockfile-only version bumps — openssl 0.10.81 / openssl-sys 0.9.117 (8 advisories: GHSA-8c75-8mhr-p7r9, GHSA-ghm9-cr32-g9qj, GHSA-hppc-g8h3-xhp3, GHSA-pqf5-4pqq-29f5, GHSA-xp3w-r5p5-63rr, GHSA-phqj-4mhp-q6mq, GHSA-xv59-967r-8726, GHSA-xmgf-hq76-4vx2), rustls-webpki 0.103.13 (GHSA-82j2-j2ch-gfr8), quinn-proto 0.11.16 (GHSA-4w2j-m93h-cj5j), rand 0.8.7 (GHSA-cq8v-f236-94qc), webauthn-rs 0.5.5 (GHSA-22w3-693w-x895) — and one by removal:protobufis gone from the dependency graph entirely (GHSA-2gh3-rmm4-6rq5;prometheusnow builds without its protobuf feature — XERJ only ever served the Prometheus text exposition format, and two new tests pin that format). A reachability analysis found none of the 13 exploitable in a default deployment; closed anyway. - npm developer-tooling bumps (#194): basic-ftp 5.3.1, ip-address 10.4.0, js-yaml 4.3.1, ws 8.21.3 — transitive from puppeteer, never part of a release artifact.
Changed
- The Helm chart now defaults to secure mode (#213). It shipped
insecure: true— TLS and auth off — while the/getinstaller shipped auth on. The default is now secure, and a newNOTES.txtstates at install time exactly what insecure mode turns off.
Documentation
- Roadmap to 1.0.0 GA from a source review against the user-feedback corpus (#212).
- Install one-liner moved first on README, xerj.org hero and llms.txt, plus a paste-to-your-agent install prompt (#219).
[1.0.0-rc.12] - 2026-08-07
Changed
doc_valueson the mapping is now honoured instead of silently ignored."doc_values": falsewas accepted, echoed back byGET _mapping, and had no effect — aggregating and sorting on such a field succeeded, where Elasticsearch errors. The doc-values sidecar builder was schema-free and filed a column for every_sourcestring it saw. The default now follows ES: no doc-values fortext,annotated_text,match_only_text,search_as_you_type,semantic_text,binary,objectandnested; an explicit"doc_values": trueon a text field still builds the column and still returns whole-value buckets.Measured on a 500k-doc corpus (one analyzed
textfield), force-merged: index 154,413,167 → 99,424,772 bytes (1.553x smaller,index/raw0.433x → 0.279x) and force-merge 139.13 s → 59.69 s (2.33x faster, merge re-encodes the sidecar and there are 55 MB fewer bytes to build). The.dvartifact falls 95.9%; every other artifact is byte-identical. Atermsaggregation on a text field got 1.34x faster (3414 → 2547 ms) after losing its column, because walking a column of whole document bodies was slower than the brute path it falls back to.The saving is corpus-dependent: it removes 90-99% of the doc-values sidecar, and that sidecar is 2-37% of index size depending on how large text bodies are relative to
_source. On source-code corpora where_sourcedominates it is nearer the low end.
Performance
The per-segment FTS reader is cached instead of rebuilt on every query.
FtsIndexReader::opensat inside the per-segment search loop and performed two zstd decompressions per field — the whole.postblob and the whole.metaarray — into owned buffers, then discarded them. Measured on one production field: ~50 ms and ~41 MB per (segment, field), per query.Readers are now cached per (segment, field-set), charged against the segment hydration budget under a new
fts_readercategory (visible in_nodes/stats), and evicted at merge completion. Segments are immutable, so no invalidation is needed; if the budget refuses the charge the reader is returned uncached, so an oversized corpus degrades to the previous behaviour rather than growing without bound.Measured p50 on 500k docs with the query cache disabled:
matchon text 166.29 → 57.85 ms (2.87x), 500-doc page 168.96 → 64.05 ms (2.64x),prefix170.91 → 69.22 ms (2.47x),wildcard180.52 → 74.79 ms (2.41x),fuzzy306.16 → 195.25 ms (1.57x),match_phrase495.58 → 339.90 ms (1.46x). Brute-scan families (function_score,boosting) are unchanged — their cost is scanning and scoring, not opening a reader.
Fixed
xerj autoindexno longer duplicates an index when an extractor improves (#178). Datasets were inferred by comparing each file's whole field-name set, including names the extractor invents rather than reads from the file (defs,symbols,symbol_count,title,page, …). A better parser makes those names appear, which moved the file into a different dataset — and the dataset slug is an ingredient of every document_id, so the file was re-indexed under a new id in a new index while its previous document stayed behind, unreferenced. Measured on a 846-file Rust corpus, re-planning with an improved symbol extractor grew the index from 53,873 to 53,902 documents with 29 stale leftovers and exit code 0; on a 2,328-file C corpus, 7,923 to 8,023. Clustering now uses only the field names that came from the file, so gaining or losing extractor fields can no longer re-home a document: both re-plans hold at 53,873 and 7,923.Nothing is deleted, so the first re-plan after upgrading (
--fresh, a new--state-dir, or a lost journal) re-homes the files whose datasets merge and leaves their previous documents in place — 53,873 → 58,855 on that Rust corpus, a one-time cost paid once instead of once per extractor change. An ordinary re-run reuses the journal's frozen plan and is unaffected. Datasets inferred from real schemas are untouched: that corpus went from 9 datasets to 7 (the two symbol-presence variants of one code dataset merged, likewise two section-presence variants of one prose dataset), and the C corpus from 407 to 405, with every document accounted for in both.
[1.0.0-rc.11] - 2026-08-04
Added
xerj autoindexpins the server's embedding execution identity (#160). Vectors produced by two different embedding backends — or by two different models, tokenizers, or vector widths within one backend — are not comparable, so mixing them in one index silently degrades every similarity result rather than failing. A semantic autoindex run now fetchesGET /v1/embedding/identity, records the returned opaque digest in its journal before the index is created, and requires the same digest on every resume; a mismatch aborts before the next index-create or_bulk. There is no new command or setup step —xerj autoindex ./reportscalls the endpoint itself over the already-configured--url, and plans with no semantic fields skip it entirely.The endpoint is authenticated (cluster-read) and deliberately opaque: it returns a version, the effective backend, a digest, the semantic contract, whether that backend is resumable, and a sanitized reason when it is not. It never returns credentials, provider URLs, model names, or local paths.
dimensionsis reported only for the backends whose width the server actually pins (lexical, andonnx-experimentalat 384); it is omitted forneural, whose width comes from the loaded model'shidden_size, and forproxy, whose width is whatever the remote returns.lexicalandonnx-experimentalare resumable.neuralandproxyare not: neither attests immutable model bytes, so a fresh semantic run is allowed but a resume is refused.A detector that connects documents by what they say, not only by what they cite (#164). Every shipped detector was structural —
wikilink,mdlink,href,pathciteandcrateciteneed an explicit citation,sequenceandsamedirneed a filesystem relationship — so a folder of PDFs or saved pages, where no document links to another, produced a graph whose only edges ran inside single documents and along directory chains.sharedterm@1emits ashared_termedge (weight 0.45, confidence 0.5) between two documents that use the same distinctive words, and carries those words as the edge evidence so a wrong link is inspectable exactly like a wrongmdlink.Measured on 240 arXiv PDFs filed in 24 topic folders (236 extractable), same binary, same corpus: before, 11,854 edges — 11,642
sequenceinside single documents plus 212same_dir, with 212 cross-document edges and none crossing a folder. After, 462shared_termedges (271 further candidates refused by the density cap), 377 of them joining documents in different folders and 296 joining different top-level topics. Two independent runs produced identical counts and identical evidence.Density is capped rather than trusted: a term in more than 10% of the corpus is that corpus's vocabulary and is ignored, a link needs at least two shared word families (so a singular and its plural are not two signals), and no document takes more than 5
shared_termedges. What the cap refused is reported by the run asedges_cappedinstead of vanishing. A corpus of ten documents or fewer therefore produces noshared_termedges at all, which is the 10% rule being honest rather than an exception.The honest limit, which the docs and the console now state: this links documents that share VOCABULARY, not documents that share MEANING. Terms are compared as strings with no stemming and no model — XERJ's built-in embedder is lexical feature hashing and this detector does not use even that. Against the corpus's own topic filing, a random pair of those documents shares a top-level topic 17.0% of the time;
shared_termlinks do so 35.9% of the time. Real signal, and a noisy one.
Changed
BREAKING: an existing semantic
autoindexstate directory cannot be resumed (#160). A journal written before this change carries no pinned identity, so there is no way to prove the server still produces the same vector space; rather than assume it, the resume fails closed with "this semantic autoindex journal predates embedding identity pinning and cannot be resumed safely". The same path fires when a previous--no-semanticrun is followed by a semantic run over the same state directory. Recover by rebuilding with--freshand a new--prefix; before reusing the old prefix, delete and recreate its prior autoindex indices. Non-semantic runs and brand-new state directories are unaffected.EmbeddingProxy::newnow rejects an endpoint that is nothttp(s)at construction time instead of at first request, so an invalidembedding.default_endpointfalls back to lexical at startup and/v1/embedding/identityreportslexical— the backend the server will really use — rather thanproxy.
[1.0.0-rc.10] - 2026-08-03
Security
The Console data-sources proxy is no longer a way around a brain (#149). The Console router is merged onto the engine routers after their layers are applied, so
authz_middlewarenever ran on it and no index-visibility scope was installed; its only filter matched.xerj_(underscore) and therefore missed the reserved.xerj-memory-namespace. Any authenticated Console session of any role could read another tenant's second-brain documents, and the indices/fields listings leaked brain names and mappings. All three handlers now refuse the reserved namespace with the same404a system index gets, and the prefix moved toxerj-commonso there is one definition.Snapshot and restore can no longer reach another tenant (#152).
authorize_expressionwaved every wildcard through before consulting the principal's grants. That is sound where the engine expands a pattern over the caller's visible set, butcreate_snapshotwalks the index map itself andrestore_snapshotexpands against the snapshot manifest and then removes and rewrites the index directory, never passing the visibility funnel. A non-superuserPOST /_snapshot/{repo}/{snap}/_restore {"indices":".xerj-memory-*"}therefore rolled every tenant's brain back to the backup instant. Patterns on those two verbs are now decided up front: one that may reach the reserved namespace requires an explicit grant, exactly as an index template'sindex_patternsalready did. Narrower patterns are untouched. The create handler also readindicesonly in its array spelling, so the string form ES equally accepts fell through to "absent" and captured the whole node; both spellings now parse, and patterns resolve against the visibility-filtered index list. (This also fixes{"indices":["*"]}previously producing an empty snapshot.)Scripted updates are bounded by the same script budget as search (#153).
_updateand_update_by_queryreach Painless throughtransform_document_serialized, which established neither the request deadline nor the fault sink, so every evaluation fell back to its own full slice. With a fresh context per;-separated statement, a script's cost multiplied by its statement count and again by the hit count, andwait_for_completion=falsedetached the whole thing with no ceiling. One deadline now governs the request, faults are captured once rather than per document, and the hit loop stops at the deadline and reportstimed_outhonestly instead of a hardcodedfalse.
Fixed
A client disconnect can no longer brick an index (#150).
index_documentheld the collection-publication guard acrossself.schema.read().await, the only suspension point betweenbegin()andcommit(). The PUT handler runs inline on the connection task, so a disconnect dropped that future; parked behind schema evolution's write lock the drop landed inside the interval and left the guardPending, setting a sticky poison flag. Every later read and write on that index then failed permanently, and_close/_opendo not reconstruct the index, so only a process restart recovered. The schema guard is now taken before the interval, which contains noawaitat all and cannot be cancelled part-way. Storage failures that occur before any visibility mutation now cancel the publication instead of poisoning it, so a transient disk error fails one request.Columnar aggregations no longer drop a segment's documents (#151, #143).
field_needs_brute_fallbackkept the fast path whenever any flushed segment carried a field's doc-values column, but array suppression is decided per segment, and the executors skip a column-less segment. Indexing scalar values, refreshing, then indexing array values and refreshing produced a terms aggregation that silently omitted the second segment's documents; term predicates and numeric metrics undercounted the same way. Coverage must now be total, and the check moved intoseg_field_kindso it also covers sub-metric fields,top_hitssort fields,matrix_statsfields and composite sources.
Thanks
- @Nicolas0315 for four detailed, measured performance reports (#144, #145,
#146, #147) on fuzzy/wildcard term expansion, top-k scoring and the unused
skip table, the
bbq_*/int8_hnswmapping contract, and filtered kNN never reaching the HNSW graph. Each arrived with a standalone harness and honest caveats. They are tracked for the following release rather than rushed into this one.
Changed — runtime-field types (can break existing mappings)
mappings.runtimeis now validated against ES's runtime-field type registry, not against the mapping-property type list. Runtime fields have their own, much smaller registry in ES:boolean,composite,date,double,geo_point,ip,keyword,long,lookup. Anything else is refused at index creation with the samemapper_parsing_exceptionES emits ("The mapper type [x] declared on runtime field [f] does not exist…").This cuts both ways, and both directions were wrong before:
- Newly rejected. Types that are perfectly valid mapping
properties but illegal under
runtime—text,match_only_text,nested,object,geo_shape,shape,point,dense_vector,sparse_vector,binary,percolator,completion,histogram,alias,join,version,constant_keyword,wildcard,date_nanos, the range types, and the narrow numerics (integer,short,byte,float,half_float,scaled_float,unsigned_long, which collapse intolong/doubleat runtime) — used to be accepted with a200. A mapping that declared any of these undermappings.runtimewill now fail to create. Change the entry to the runtime type ES would have required (usuallykeyword,longordouble), or move the field intoproperties. - Newly accepted.
compositeandlookupare runtime-ONLY types and are absent from the property list, so they were previously rejected outright — even thoughlookupruntime fields are implemented on the search path (via search-bodyruntime_mappings) and covered by the conformance suite. Both are now accepted undermappings.runtime.
Scope: this is
PUT /{index}validation only. It does not change evaluation:mappings.runtimeis still stored and never evaluated — runtime fields are only computed when supplied in the search body asruntime_mappings. Search-bodyruntime_mappingsandPUT /{index}/_mappingstill do not validate runtime types at all (they never did) — unchanged here.Prior to this, the rule was half-enforced:
flat_object/flattenedwere special-cased into a400while every other ES-forbidden type passed.- Newly rejected. Types that are perfectly valid mapping
properties but illegal under
Security
x-forwarded-foris no longer trusted for client identity (#76 S5-4). The per-IP auth rate limiter and the audit-log source address used to be read straight out ofx-forwarded-for— a header the caller writes. A rotatingx-forwarded-for: 1.2.3.$RANDOMtherefore reset the quota on every request, so the unauthenticated magic-link redeem and bootstrap-claim endpoints could be brute-forced without limit, while honest clients (who send no such header) stayed throttled. Client identity now comes from the TCP peer:ConnectInfo<SocketAddr>is threaded through both the plain and the TLS listener, and a handler reads it via the newClientIpextractor.Forwarding headers are still honoured behind a real reverse proxy, but only from a proxy the operator has declared: the new
server.trusted_proxiessetting takes IP addresses or CIDR blocks (["10.0.0.0/8", "::1"]) and is empty by default — an unconfigured node believes nobody. When the peer is a declared proxy the forwarded chain is read right-to-left (the left end is caller-authored) and the right-most address that is not itself a listed proxy wins; a malformed element stops the walk rather than being stepped over. Malformed entries in the setting fail startup instead of silently widening or narrowing trust.Operators terminating TLS or load-balancing in front of XERJ must set
server.trusted_proxiesto their proxy's address, otherwise every user behind it shares one rate-limit bucket.Cluster control frames are now authenticated (issue #75). The inter-node transport on
cluster.portaccepted any TCP connection that spoke its wire format: anyone who could reach the port could send RaftRequestVote/AppendEntriesframes and steer consensus. Every frame now carries an HMAC-SHA256 tag over a per-connection random challenge issued by the receiver, the sender's node id, the frame's position in the connection, and the payload; tags are compared in constant time, and the tag is verified before the payload reaches the JSON deserialiser. The challenge makes a captured frame non-replayable, including against a different node.Cluster mode fails closed.
[cluster] enabled = truenow requirescluster.auth_secret(orXERJ_CLUSTER_AUTH_SECRET), minimum 16 characters. With cluster mode on and no secret from either source the node refuses to start instead of running an unauthenticated control port. Single-node mode — the default — is unaffected and needs no secret.
Breaking
- The cluster wire format changed and is not backward compatible. A node running this version cannot talk to a node running rc.9 or earlier in either direction: the receiver now speaks first (magic, version, challenge) where the old protocol expected the sender to open with a length-prefixed JSON frame. Upgrading a cluster requires stopping every node and restarting them on the new version with a shared secret configured — a rolling restart will not interoperate. Only cluster mode is affected; it is off by default, so single-node deployments need no action.
- Scope note: the HMAC authenticates frames, it does not encrypt them. Cluster
traffic is still plaintext and the secret is cluster-wide, so it proves
cluster membership, not per-node identity.
cluster.portstill belongs on a trusted network segment; mTLS remains unimplemented.
Added
scriptquery support (#87). A Painless predicate evaluated per document, wired into the doc-scan path. Missing fields fail closed the way modern Elasticsearch does:doc['missing'].valueerrors rather than coercing to a default, because since ES 7.0 reading.valueoff an emptyScriptDocValuesthrows (6.x returned a type default with a deprecation warning and 7.0 removed that fallback). Coercing would mean a script filter with a typo'd field name matched every document. The guard idiom ES documents still works, because that idiom isdoc['x'].size() == 0 ? <default> : doc['x'].valueand.size(),.lengthand.emptyare total on a missing field.Compiled scripts are now cached, bounded at 128 entries and 512 KiB of source per thread. A script is evaluated once per document, so re-parsing per document made the per-document cost scale with the script's size. The cache key is attacker-supplied source, so the bound is a security property rather than an optimisation detail.
getDayOfWeekEnum().getDisplayName()in Painless (#92).
Fixed
Painless resource-limit trips surface instead of returning a wrong score (#97, #123).
MAX_CALL_DEPTH = 32is a real ceiling on legitimate recursion, and in the scoring paths it failed silently: a script recursing past it did not error, it returned a wrong score.Two failure classes had been collapsed into one and they need opposite treatment. Unsupported or unparseable script syntax still degrades quietly, which is what ES does and what seven of nine call sites rely on. A tripped resource limit now propagates, because a silently wrong score is undetectable.
_search,_msearch, both template variants, scroll, async_search,_rank_eval,_explainand pivot transforms report it;_count,_reindex,_delete_by_queryand_update_by_queryrefuse rather than act on a selection a fail-closed script truncated. All nineeval_painlesscall sites were audited and behaviour changed at none of them._rank_evalrecords the fault per request infailuresrather than failing the batch, and_explain— a diagnostic endpoint whosematchedverdict does not depend on scoring — answers and discloses the fault rather than refusing._delete_by_queryand_update_by_querypreviously returned HTTP 200 with a 400-shaped body, so a client branching on the status saw success.MAX_CALL_DEPTHstays 32: re-measured in release, the parser's worst case is 40,720 bytes per call level, so depth 32 already uses 1.20 MiB of a 2 MiB worker stack./_msearchno longer skips the request-time script guard (#111, #124). It built its sub-requests straight offparse_request, bypassing the validation the single-search path applies. Measured: a 240,531-byte_msearchbody whose items carried scripts that_searchrejects with a 400 came back 200 on every item._search/templateand_msearch/templatehad the same hole.Closing it initially made
_msearchstricter than_search, because the new guard walked the whole body while_searchchecks six specific fields. A singleGuardedFielddefinition now drives every entry point, with theaggs/aggregationspair resolved exactly as the executor resolves it (aggswins when both are present, regardless of document order). A compile-time assertion pins the guarded set at six, because removing an entry was not a type error and silently reopened the bypass.<field>.keywordpredicates resolve against the parent column (#110). Four raw lookups insideresolve_predfailed closed, asserting that a flushed segment held no matching row, while the in-memory arm resolved the same field correctly. Measured: afiltersaggregation onextension.keywordreported 200 documents out of 3,200, and atermquery ongroup.keywordreported 0 hits out of 6,000.fast_aggsfalls back to brute force for unresolvable nested fields (#104). Found live on real Kibana and OpenSearch Dashboards sample dashboards:geo.destterms,machine.ramavg and the doubly-nestedmachine.os.keywordterms all returned empty or null through the fast path while the brute path computed the right answer.The columnar histogram paths honour
config.limits.max_buckets(#121, partial).exec_date_histogramandexec_histogramcarried their own hardcoded65_536. Measured at cap 37 on a 12,000-document index: 38 buckets returned, no error. The brute paths ignored the setting too, so this was live in both executors. Still open:exec_termshas no bucket cap at all, which is the actual OOM vector; see #121.bucket_scriptresolves_countand evaluates parenthesised ternaries (#95, #105). A_doc_count-weighted bucket and thebucket_scriptinside it disagreed about how many documents the bucket held. Verified per aggregation type. Parenthesised ternaries silently resolved to a null bucket value, because the split only looked at paren depth 0. Two equal infinities compared unequal, becauseinf - infis NaN and an epsilon-only==rejected identical values.Named ES date formats are resolved for
ignore_malformedvalidation (#89). Named shorthands were matched as literal text rather than expanded, so a value genuinely valid under the declared format was silently dropped on ingest. Confirmed against a real OpenSearch 2.11.1 node, which accepts the ISO strings and rejects the bare numbers — the opposite of what XERJ did.Two
xerj-enginetests no longer flake under parallel load (#100). One mutated a process-wide bucket cap under a lock that only protected participants; the other depended on wall-clock timing. Measured on a 32-core box under 14 CPU hogs: 12/12 and 10/10 green, where the prior tree was 18% red._field_capsreportsflat_object/flattenedrather than genericobject(#86), andflat_objectis treated as an alias forflattenedfor OpenSearch callers.Keyed (object-shaped) bucket aggregations merge across multi-index search (#103), and
X-OpenSearch-Versionis sent to OpenSearch callers (#101).
Performance
parse_date_msno longer pays an unbounded scan per document (#91, #96). The no-colon zone-offset branch added in #91 sat before the naive fast paths and cost roughly 2x on offsetless ISO values, a hot per-document ingest path. The probe now reads a fixed six-byte window at the end of the value and nothing else — a constant window is sufficient because%zcloses the pattern and chrono requires the whole value to be consumed.Measured across three independently built, CPU-pinned harnesses: offsetless ISO 217.0 → 110.2 ns/op, and the long-value classes that an interim scan-based guard regressed (4 KB prose 1171.4 → 198.3, 64 KB 16278.0 → 1009.2) are back at or below the pre-guard cost. Best-or-tied on every class, with zero mismatches across 72,462 constructed values.
[1.0.0-rc.9] - 2026-08-01
Ninth release candidate: the cross-platform correctness release.
Headline for users, stated plainly because it is the most serious defect
this project has shipped: the Windows binaries published as rc.4
through rc.8 could not start. Every index creation returned
Access is denied. (os error 5), and because the console bootstrap
creates a system index on first boot, the server aborted before it ever
listened. xerj.org/get.ps1 installed those binaries for three weeks.
The cause was one unguarded Unix idiom in the durability path; the fix
is four lines, and a Windows runner now boots the binary, writes a
document, reads it back and indexes 400 datasets on every pull request.
Nothing else caught this because, until this release, no CI job had ever
run the binary anywhere except Ubuntu — release.yml built eight
targets and executed none of them.
The release also closes the self-contained half of the post-audit
security backlog — a snapshot location that could still escape
data_dir, index-name validation missing at the create boundary, a
field limit that dynamic ingest enforced and explicit mapping updates
did not, an unserialised magic-link redemption, and node identity leaking
from an unauthenticated endpoint — and it carries the consequences of
taking the Windows lesson seriously: a test pass over the surfaces that
shipped untested, and the four real defects writing those tests exposed.
A create-time file-descriptor exhaustion reachable from an index-create
request, inline <script> contents indexed as prose, XML record election
that changed shape between runs of the same file, and a core dump on
xerj … | head. The Rust suite grows from 1,420 to 1,486 test
functions; conformance is unchanged at 1360 passed / 0 failed / 3
skipped. The only performance change is a memory regression fix in the
ONNX embedding path; no speed measurement was taken and none is claimed.
The _reindex keyset behaviour is untouched — a test around it was
wrong, not the code, and that distinction is spelled out below because
it is easy to misread as a data-loss bug.
Fixed
The server can start on Windows.
xerj_common::fsio::fsync_dirwasstd::fs::File::open(dir)followed bysync_all(), with no platform gate. On Windows, obtaining a handle to a directory that way always fails withERROR_ACCESS_DENIED (os error 5): the standard library cannot passFILE_FLAG_BACKUP_SEMANTICS, which that call requires.IndexStore::save_snapshotfsyncs the parent directory after publishing the segment manifest, so every index creation returned an I/O error, and the unconditionalxerj-consolebootstrap — which creates.xerj_userson first boot — turned that into a fatalxerj-console bootstraperror at startup. Introduced in297be60(2026-07-12) as part of the power-loss-ordered publish chain, so it affected rc.4, rc.5, rc.6, rc.7 and rc.8. Windows now takes a separatefsync_dirthat returnsOk(()), because Windows exposes no directory-flush primitive at all —FlushFileBuffersis not defined for directory handles. State the consequence honestly: durability on Windows is weaker than on Unix. The file contents are still fsynced by the callers before the rename; what is not forced is the rename itself, which relies on NTFS metadata journalling. That is the best available behaviour on the platform, and it is now a documented property rather than an error returned on every call.xerj … | headno longer aborts with a core dump. The Rust runtime installsSIG_IGNforSIGPIPEbeforemain, so once the reader closed the pipe the next write returnedEPIPE,println!panicked withfailed printing to stdout: Broken pipe, and the release profile'spanic = "abort"turned that into a core dump — which also swallowed the command's real exit status. The binary now restores the default disposition before anything can write to stdout (unix only), so the kernel terminates the process on the signal and shells report the conventional 141. Found becausexerj autoindex map | head -80core-dumped on every run of a use-case harness. Two consequences worth knowing. Client sockets are unaffected: the standard library writes those withMSG_NOSIGNAL(SO_NOSIGPIPEon macOS), so a client disconnecting mid-response still surfaces asEPIPEand cannot signal the server. The server's own stdout does change — previously tracing discarded write errors and a server outlived a vanished stdout; now the next log line terminates it. Supervised deployments redirect stdout to a file or/dev/null(thebrainboot path included), so in practice this reaches onlyxerj | reader-that-leaves.Inline
<script>and<style>contents are no longer indexed as prose. The HTML extractor'sskip_untilstate suppressed the tags inside those elements but the tokenizer's text branch appended their contents to the buffer unconditionally, and the closing tag continued without discarding it — so the buffered CSS and JavaScript were flushed into the document body at the next tag boundary. Minified stylesheets and scripts became indexed prose in every HTML record: BM25 noise, and — the reason this is more than a quality bug — inline configuration blocks carrying API keys, tokens and internal endpoints were stored in_source. For the "point xerj at my project folder" use case that is a real exposure. Script and style elements are now treated as the raw-text elements they are — the scanner jumps to the literal end tag instead of tokenising inside them, which also bounds an unclosed<script>: the tail of the file is discarded rather than buffered and indexed. Two narrower ways the leak survived a first attempt were caught in review and are closed here. A trailing/inside an unquoted attribute value (<script src=/cdn/lib/>) was read as a self-closing tag, so the element was never entered — legal HTML5, sincescriptis never a void element. And an end tag whose name ran on into a non-alphanumeric character (</script-foo>) was accepted as the terminator, ending raw text early and indexing everything after it. The tag scanner now tracks attribute state to decide self-closing, and the end-tag test follows the spec's appropriate-end-tag rule: whitespace,/, or>.XML record election is deterministic.
elect_record_tagchose the repeating element withmax_by_keyover aHashMap, andmax_by_keysettles ties by iteration order — which is seeded randomly per map. When a wrapper element and one of its children were both structured and equally frequent, elections over the same bytes chose different tags: 200 in-process elections over one fixture split 89 / 111. The record count and the locators held, so documents kept their_ids, but each build gave them a completely different field set, and field inference for that dataset never settled across builds. The comparison is now total — most occurrences, then the outermost tag, then the lowest name — with the outermost tag as the principled key: when a wrapper repeats as often as its child, the wrapper is the record and the child is one of its fields. Two limits worth being precise about. This bites across independent builds — a--freshrun, a new state directory, a second machine, a CI reproducibility check — not across ordinary repeated runs, because the resume journal skips files whose content digest has not changed. For the same reason an index built before this fix does not heal itself: it keeps whatever shape it rolled until the XML file's bytes change or the index is rebuilt with--fresh.Three further inference elections are deterministic. Reviewing the fix above found the same defect —
max_by_keyover aHashMap, ties settled by a per-instance random hash seed — at three more decisions, each with a wider blast radius than the XML one. Each now has a tie-break argued from what the decision means rather than a copied rule. The log extractor elects the template an entire file is parsed with, and every line missing that template is demoted to a continuation, so a flip changes the record count; ties now go to the most specific template, in the orderparse_kindalready tries them (app before clf before syslog), so an ambiguous file resolves the way an ambiguous line does. Field inference elects the date encoding written into the mapping, the catalog and the coercion plan; ties go to the lowestDateEncin declaration order, which isparse_date_str's own richest-first priority, so a field mixing…T00:00:13with… 00:00:13names the encoding that concedes the least. The third elects a field's entity type, keyword versus text.The ONNX embedding path no longer materialises every window upfront.
embed_semantic_jobsbuilt and cloned the passage text of every window before embedding any of them, so peak memory scaled with the whole job rather than with what was in flight. Windows now carry(start, end, passages)and their texts are built lazily and moved intoembed_batch, with at most two batches in flight. This closes a memory regression at default settings (#71); no throughput measurement was taken and no speed claim is made.
Security
Five items from the post-audit backlog, each with a regression test where the defect is testable, plus one found while writing this release's tests. The three that remain open are listed under Known limitations rather than quietly omitted.
- Snapshot repository locations could still escape
data_dir(#73, High). The residual half of F-PATH-02: asettings.locationcontaining..that resolved to a nonexistent target slipped past the guard, because canonicalisation fell back to a lexical path and the containment check compared components of a path that had never been resolved...components are now rejected outright, before any resolution. The regression test covers both the direct form and laundering it through an explicit allowlist entry. - Index-name validation is enforced at the create-index boundary
(#80).
IndexName::validateexisted and was applied elsewhere; the ES-compatiblecreate_indexhandler did not call it. Wiring it up is defense in depth against path traversal through index names — the accepted set is unchanged, so no previously valid name is now rejected. - The field limit applies to explicit mapping updates (#76 S5-5).
max_fields_per_indexwas enforced on dynamic ingest but not on the explicitadd_fieldspath, soPUT _mappingand schema evolution could push an index past the ceiling that exists to bound mapping explosion. - Concurrent magic-link redemptions cannot both mint a session (#76 S5-3). The check-then-consume sequence in the redeem handler was not serialised, so two requests carrying the same token could both pass the used-check. Redemption is now serialised behind a gate. Stated precisely, because it is easy to overclaim: in the shipped engine the second session was already being blocked incidentally, by the index's optimistic concurrency rejecting the colliding write — so the observable pre-fix exposure was a 500 with internal detail instead of a clean 401, plus a single-use guarantee that rested on what the store happened to do with racing writes rather than on the handler. The fix moves that guarantee into the handler where it belongs. This is covered by 25 rounds of 6 concurrent redemptions.
- The unauthenticated
cluster/infobody no longer leaks the node identifier or exact build version (#76 AUTHZ-2). It now carries only mode and uptime. - A per-index setting could exhaust file descriptors at index-create
time.
index.xerj_ingest_shards— added in rc.8 to reduce descriptor usage — was accepted with only a>= 1check, and the ES-compatible create-index handler threads the request body'ssettingsthrough verbatim.IndexStore::openeagerly creates a directory and opens a WAL file descriptor per shard, so{"settings":{"index":{"xerj_ingest_shards": 100000}}}was a single-request descriptor and inode exhaustion — precisely the failure the setting exists to prevent. The override is now bounded to1..=256, the same ceilingEngineConfig::validatealready enforced on the globalengine.ingest_shards. Out-of-range values are refused, not clamped, so a typo falls back to the engine default rather than silently taking a different one. Found while writing the tests that rc.8 shipped without.
Added
- CI runs the binary on macOS and Windows. A new
autoindex-fd-smokejob on an[ubuntu, macos, windows]matrix boots the server under a constrained descriptor budget (soft 256 / hard 4096 — between the pre-fix need of roughly 6,400 for 400 datasets and the post-fix 600), asserts a create-index / index-document / search round-trip, autoindexes a 400-dataset corpus and fails on anyEMFILE. macOS runners default toulimit -n 256, the exact condition the rc.8 descriptor bug regressed under, so the job reproduces it rather than approximating it. The explicit round-trip exists so that the next platform-specific break names the operation that failed instead of surfacing as an opaque "server exited during boot". - The use-case harnesses run in CI. A new
usecase-smoke.shgate drives the second-brain transcript (overview counts and detector families, ego evidence andnot_shown, idempotent link, unlink retiring rather than deleting,as_ofreplay, and the normativehops > 2/ self-edge 400s and unknown-brain 404), the MCP tool surface (initialize handshake, the fourxerj_brain_*tools listed with their honesty strings, and a realtools/callround-trip), and autoindex discovery with post-discovery search. These harnesses already existed but were manual, so a regression in the brain API or the MCP tool list was caught by nothing. Wiring them up immediately found two assertions that could never have passed: the transcript pinned detector versions at@1when five of them are at@2, and the autoindex idempotency check compared whole_cat/indicesrows including on-disk size, which legitimately changes across a re-extract. - THE MAP's bounded-graph claims are tested. The knowledge-graph
pipeline shipped in rc.7 on two load-bearing claims — at most 13
groups at any scale, and byte-identical output across runs — with no
automated test in any language behind either; the only exercise was a
manual browser script needing a live server and a real Chrome. The
pipeline is pure dependency-free JavaScript, so 23 offline cases now
run in under a second over four corpus scales (8, 12, 40 and 120
folders, straddling the 12-cluster cap): the group bound with at most
one pooled "everything else" body, every file landing in exactly one
group with the membership lists and the index agreeing both ways, link
conservation against the honesty row the UI computes from,
run-to-run determinism, the pairwise bundle bound, and
as_ofreplay making a retired link live again. - Regression cover for four surfaces that shipped without any. The
DOCX decompression and paragraph caps from rc.7 (both verified by
removing each guard in turn and confirming the matching test fails);
the six autoindex extractors that had no direct tests at all — HTML,
CSV, JSON, JSONL, XML and SQLite, 49 cases; the magic-link redemption
race hardened in rc.9's Phase-2 work, driven with 25 rounds of 6
concurrent redemptions; and the per-index WAL shard override,
including a reopen test that pins a shard count, writes documents that
live only in the memtable and WAL, restarts, and asserts both the
layout and every document survive. An installer lint gate covers
landing/getandlanding/get.ps1, which had no verification of any kind despite being the only user-facing entry point.
Changed
- A test was measuring the CI runner's core count, not the code.
reindex_pages_past_10k_via_keysetfailed on any machine with real core count (left: 0, right: 10050) and passed on CI's two-core runners, socargo test --workspacewas red for every developer and green for the project. Turbo ingest routes documents to memtable shards by worker thread: on two cores they all land in one incidentally-visible shard, on a real box they scatter into unpublished shards that the search the reindex pages over cannot see. The sanity assertion above it passed either way becauselive_doc_countreads the version map rather than the search surface. The test now refreshes the source. The_reindexproduct path was never affected — driven end to end over HTTP, bulk-indexing 10,050 documents and reindexing withsize: 1000reportstotal: 10050, created: 10050, batches: 11and the destination counts 10,050, both with and without an explicit refresh. This was the test reaching under the API, not a reindex defect, and it is recorded here because a bare changelog line about "reindex returning 0 documents" would badly misrepresent it.
Known limitations
- DOCX truncation at the decompression cap is silent. When a
document's XML inflates past the 72 MiB cap the reader simply hits
end-of-file; because a truncated tag reads as a clean
Eofrather than an error, nothing increments the junk counter. A bomb-truncated document is therefore indistinguishable from a legitimately short one — the caller gets a well-formed record that is quietly missing content. The memory bound itself holds and is tested; only the reporting is missing. Making truncation observable would change the statistics autoindex reports, so it is deliberately left for a separate change. index.xerj_ingest_shardsis only read in its nested spelling.{"index": {"xerj_ingest_shards": 1}}is honoured;{"index.xerj_ingest_shards": 1}is not, unlike the neighbouringindex.sort.*settings which accept both. Nothing in the product emits the flat form, so this is an inconsistency rather than a break.- A brain is still not an authorization boundary. Any authenticated
caller can read any brain's
overviewandego; with a shared engine that is a multi-tenant exposure. This needs an access model rather than a patch and is tracked in #79, deliberately not rushed into a release candidate. The concrete part of that issue — the walker indexing.envand other dotfiles — was fixed in rc.7. - Cluster transport control messages are still unauthenticated (#75). Reachable only with cluster mode enabled, which is off by default.
x-forwarded-foris still trusted for client identity (#76 S5-4), which means a caller can spoof the address the per-IP rate limiter keys on. The fix needsConnectInfothreaded through the shared serve and TLS path and is deliberately not bundled with the smaller items above. (Fixed after this release — see Unreleased.)- Windows durability is weaker than Unix durability, as described under Fixed. The platform provides no way to make it otherwise.
[1.0.0-rc.8] - 2026-07-31
Eighth release candidate: an autoindex robustness + code-AST release.
Headline for users: xerj autoindex no longer crashes with Too many open files on large source trees, and source code is now parsed into searchable
symbols instead of plain text.
Added
- AST code extraction. Source files are parsed with tree-sitter — Python,
JavaScript, TypeScript, TSX, Rust, Go, Java, C, C++, Ruby, PHP, C#, Bash —
rather than indexed as prose. Each file carries its
language, a structuredsymbolsarray ({name, kind, line}), and a searchabledefslist, so a query likeclass Modelretrieves the file that defines it. Grammars are version-decoupled viatree-sitter-language, so one tree-sitter 0.25 core serves all of them and adding a language is a grammar dep plus one row.
Fixed
- Autoindex
Too many open files(os error 24) crash on large repos. Each index holds segment mmaps, a WAL and a merge task, so discovering a repo that infers hundreds of datasets (WooCommerce ≈ 413 indices, ~16 descriptors each) exhausted file descriptors on a default macOS soft limit (256), failing near the start. At startup the server now raisesRLIMIT_NOFILEtoward the hard limit using a concreterlim_max: macOS rejectssetrlimit(RLIMIT_NOFILE)whenrlim_maxisRLIM_INFINITY(its launchd default), so passing that back was a silent no-op and the limit stayed at 256 — the raise now sets a concrete ceiling and steps down for thekern.maxfilesperproccap (best-effort). Reproduced and fixed against real django, redis and WooCommerce clones. - jemalloc
background_thread currently supports pthread onlyprinted on every macOS launch — thebackground_threadoption is now Linux-only, where it is supported; the decay policy is unchanged. - Empty console dashboards on a fresh or brain-only engine. The AI, RAG, vector, agent-memory and logs dashboards fell back to mock data when their backing index was empty; each dashboard's nav entry is now gated on its data existing, so a brain-only launch opens on Second Brain and an empty engine shows only System.
[1.0.0-rc.7] - 2026-07-30
Seventh release candidate: the second brain and security-hardening
release. Headline for users: point one binary at a folder with
xerj brain <folder> and get a bi-temporal, evidence-carrying link graph
over your own documents, with a console view — THE MAP — that stays
readable at any corpus size. Alongside it, six network-reachable security
findings from a whitebox self-audit are fixed.
Added
xerj brain <folder>— one command turns a folder into a queryable knowledge graph. Structural detectors (wiki-links, markdown links, hrefs, folder adjacency, reading order) record, for every link, the exact sentence that created it and the byte offset it came from. Measured: 10 notes → 25 links in 0.1s; a 943 MB repo copy → 7,571 files, 2,128,360 notes, 20,930 links in 322.7s.- Graph HTTP API —
POST/DELETE /_graph/{brain}/link,GET /_graph/{brain}/ego,GET /_graph/{brain}/overview. Bi-temporal: links are retired, never deleted, and any past moment is replayable viaas_of. Traversal is capped at 2 hops per call by design — XERJ is a search engine with a graph-shaped index, not a graph database. - THE MAP (console) — a Canvas knowledge-graph view whose default "helicopter" altitude is bounded by construction: any corpus collapses to at most a dozen named groups plus one catch-all. Measured 19,299 links / 1,550 documents → 13 groups in 44 ms; a synthetic 50,000-link budget → 13 groups / 60 bundles in 122 ms, byte-identical across runs; 60 fps pan/zoom/expand. Small corpora skip clustering and draw real notes, so the view is adaptive rather than one-size.
- Cross-file-type links — new
pathcite@1andcratecite@1detectors relate documents to the code and files they cite, so a PDF or a design note can be shown next to the crate it references, with the citing text as evidence. - Agent surface — four MCP tools (
xerj_brain_ego,xerj_brain_link,xerj_brain_unlink,xerj_brain_overview) undercontract: xerj-second-brain/1. limits.snapshot_repo_allowlist— an Elasticsearchpath.repoequivalent bounding where snapshot repositories may live.
Fixed
- Security (unauthenticated):
POST /_sqlwith a deeply nestedWHEREclause overflowed the stack and aborted the whole process. The WHERE parser now bounds recursion depth (MAX_SQL_DEPTH), as the query_string parser already did. - Security: search-template parameters were spliced into the query JSON unescaped and re-parsed, so a parameter could inject query structure and drop the template's own filter. Parameters are now JSON-escaped and can no longer emit structural tokens.
- Security: a snapshot repository's
settings.locationwas an unvalidated absolute path, so snapshots could read and write index data outsidedata_dir. Locations are now bounded bydata_diror the new allowlist. - Security: the console's session
last_seenrefresh blind-wrote a stale copy of the session, which could resurrect a session revoked moments earlier. The refresh now re-reads and skips revoked sessions. - Console: the Second Brain dashboard is no longer always present — it
appears once at least one brain exists, so a user who has never run
xerj brainis not shown an empty dashboard. - Console: several second-brain counts reported the engine's 10,000-hit
floor as an exact total on multi-million-note brains; totals are now
exact, and floors are shown as
≥.
Notes
- The second brain's link detection is structural and lexical — there is no LLM and no semantic model in the loop, and the node-store embedder remains a lexical feature hash. Scale beyond the measured corpus above is not claimed.
[1.0.0-rc.6] - 2026-07-28
Sixth release candidate: the semantic-analytics and bounded-memory
release. Headline for users: a knn/semantic query can now carry
aggs in the same _search request — aggregations run over the
retrieved top-k neighbour set, forced onto the exact path so bucket
counts are exact — and a semantic hit can opt into a _passage field
that returns the winning chunk's text with byte-exact provenance.
Underneath, the seven immutable-segment hydration caches now share one
process-wide, cgroup-aware memory budget, and the storage layer gains a
bounded, cancellable selective-hydration primitive (deliberately not
wired to any engine route yet). Two correctness holes are closed: kNN
could return a superseded vector when a document ID was re-inserted
while owning the HNSW entry point (fix arrived from the probelabs fork),
and an oversized Painless expression could exhaust the native stack
instead of returning a bounded error. One change arrived from outside
the org: transparent gzip request-body decompression, contributed by
Vinz2168, which unblocks Filebeat and the other ES clients that compress
by default. ES-YAML conformance holds at 1360 passed / 0 failed / 3
skipped. Each change states its own limits in place; the aggregation
gaps left open are listed under Known limitations.
Added
- kNN + aggregations in a single request.
POST /{index}/_searchwith aknn/semanticquery plusaggsnow returns aggregations; previously theaggsfield on kNN responses was hardcodednull. No new API surface — the existingaggsrequest field is honored on the kNN path, across all three executors (HNSW, exact brute-force, multi-kNN). Semantics follow ES top-level-knn: aggregations run over the retrieved top-k neighbour set after thenum_candidatesfan-out, independent offrom/sizepaging —"size": 0returns aggs only with empty hits, andhits.total.valuereports the neighbour pool (k), not a match count, so bucket counts scale withk, never with index size. To keep those counts exact, an aggs-bearing kNN request always executes the exact brute-force path even when it would otherwise qualify for HNSW — ANN recall is below 100% and approximate bucket counts would be silently wrong — trading ANN speed for exactness (no performance claims are made for this path). This removes the previous two-step workaround (kNN → collect ids → separate aggregation request). Limitation, stated up front:significant_termsover a kNN slice returns empty — the vector path does not yet supply a background corpus; usetermsfor raw in-slice counts, or the two-step pattern when statistical significance is needed (filed as a follow-up). - Winning-passage provenance (
_passage). Requesting"fields": ["_passage"]on a singlesemanticor kNN query addsfields._passage[0]to each hit: the winning chunk's sourcefield, zero-basedordinal, UTF-8 bytestart_offset/end_offset, thetextslice, andpage(only when the source carries a numericpage; omitted otherwise). The text is reconstructed by slicing the authoritative_sourceat compact ingest-time offset metadata rather than storing the chunk a second time — the committed measurement over 256 varied Unicode documents reports 80.26 B/doc of raw metadata, 3.98 B/doc after ZBS2 compression. All three ESfieldswire shapes (scalar, string array, object form) are accepted and normalized identically, and scroll continuations keep the passage. Multi-kNN and hybrid-fusion queries requesting_passageare a deliberate HTTP 400 with an actionable message — summed or fused contributions do not define one winning passage, and rejecting is more honest than guessing. Scope: provenance covers the exact per-chunk max-sim scoring path; the HNSW-served and SQ8-quantized paths reconstruct a passage only for single-chunk values, and a kNN clause on an arbitrarydense_vectorfield has no passage metadata, so_passagemay be absent rather than erroring. The internal metadata (reserved prefix__xerj_passage_meta__) is engine-owned: supplying it at ingest is rejected per-document, and it is stripped from_source, Painless, field discovery, full-text indexing, and aggregations. The native_searchendpoint accepts_passage(returned as a hit-level key) and rejects any otherfieldsentry. The default embedder remains lexical feature hashing — neural semantics still require opting into neural embedding mode or an embedding endpoint. - Process-wide segment hydration cache budget. The seven
immutable-segment hydration caches (stored slices, doc values, parsed
stored values, sort shadows, id positions, row sequences, decoded
stored bytes) — previously independent or unbounded — now share one
cgroup-aware retained-payload budget with CAS admission and
last-reader refunds. Configure with
[limits] max_segment_hydration_cache_mb(default0= automatic: 20% of the effective cgroup/system memory limit, no floor and no fixed cap; an explicit value is clamped to 50% with a startup warning) or theXERJ_SEGMENT_HYDRATION_CACHE_MBenv override (auto,off, or a MiB value;offrefuses every admission — a diagnostic switch). Refusal is invisible to queries: the request keeps the value it already built and returns the same result, uncached; publish-time warm-only values are simply dropped, and a counter increments.GET /_nodes/statsgainsindices.segment_hydration_cachewith limit/current/peak, refusal and accounting-error counters, the budget source, and per-category breakdowns. Automatic sizing now honors the tightest cgroup-v2memory.maxfrom leaf to root (previously the nearest finite value won, overstating the ceiling when a parent limit was smaller). A release-build accounting underflow that left counters unchanged is also fixed — both counters now clamp atomically and record exactly one accounting error. Honesty note, repeated in the docs: this is a retained-payload/key ceiling computed from deliberately conservative estimates, not an RSS bound — query-owned materialization, decode scratch, mmaps, memtables, vectors, and allocator behavior stay outside it, and a single highly compressed segment can still create a transient decode peak before admission (bounded/streaming decode is a named follow-up). The motivating profiles (~206 MiB of publish-warming RSS on one 4,096-document diagnostic run; 226.35 MiB of cumulative live allocations under a merge warm) are attributed measurements, not a claimed end-to-end memory reduction. - Bounded selective stored-field hydration (storage primitive). The
xerj-storagecrate can now hydrate selected rows and selected top-level_sourcefields from a ZBS2 V2 stored section without materializing the whole canonical JSON array, with cancellable entry points (deterministic checkpoints every 128 rows), a retained-size preflight computed from framing metadata alone (fails closed on historical zstd frames that omit a content size), and malformed-input hardening — checked arithmetic, fallible reserves, and framing/varint/bit-width validation keep hostile headers on the error path instead of aborting in the allocator. The on-disk stored format is unchanged and historical segments remain fully readable. This is deliberately not wired to any engine route or cache in this release — no endpoint, no query change, and no end-to-end speedup is claimed. The committed report measures the trade-off on its deterministic 20,000-row fixture: one-row hydration at roughly 59× lower peak RSS (~335 MB full decode vs ~5.6–5.7 MB selective) but roughly 4× slower wall time, because RAW/zstd JSON columns still stream through all rows — a memory primitive with a CPU cost, measured as a single-host storage microbenchmark, and LZ4 columns still decompress whole before selective materialization. - gzip request-body decompression (external contribution). Both
HTTP routers now transparently decompress
Content-Encoding: gziprequest bodies before parsing; previously the raw gzip bytes reached handlers and failed JSON parsing (the commit reportsPUT /_ilm/policyreturning 400 andPOST /_bulkreturning 500). This unblocks the ES clients that compress by default — the reproduced failure was Filebeat 8.13.4 with default config 400'ing on its first ILM setup call, marking the output connection permanently failed, and retrying forever with zero documents ingested; after the fix the same default run completes and every test event lands and is queryable. Semantics note: the decompression layer sits outside the body-size limit, so the configured request-body limit now bounds the decompressed size, not the on-wire size. Only gzip is handled (deflate/br/zstd remain unsupported), response compression is unchanged, and verification is the PR's own curl + Filebeat runs — no automated test was added in the diff. Contributed by Vincenzo Lombardo (Vinz2168).
Fixed
- kNN under document-ID reuse (HNSW entry point). Re-inserting a vector under the external ID that owned the HNSW graph entry point could return the old, superseded vector as a hit for that ID, and the v2 graph writer could persist a CRC-valid file whose entry header paired the current slot's external ID with the orphaned slot's higher layer. Both are fixed (arrived from the probelabs fork, PR #64, landed via #66): a slot counts as live only when the reverse ID map resolves its external ID back to that exact slot, the entry point is re-selected as the highest-layer live slot, search excludes superseded slots from results (they stay traversable for graph connectivity — excluded, not reclaimed), and the writer derives the entry header from live topology, repairing a stale in-memory header on save. This is a write/search-time fix, not a retroactive file repair: graph files written before the fix still load as-is and are corrected the next time the fixed writer saves them. The on-disk graph format and byte layout are unchanged, and no authoritative documents or latest vectors are discarded.
- Painless: oversized expressions return a bounded 400 instead of
aborting the process. Flat shapes the parser can build without
recursing (
1+1+…binary chains,.memberchains,[index]chains) bypassed the parser's recursion guards, and the evaluator's depth guard counted frames only after entering evaluation — so a 5,001-node chain could exhaust the native stack before the intended error fired (reproduced as a process abort on the default test-thread stack). Expression depth is now computed exactly at parse time — 500 accepted and evaluated, 501 rejected, covered for all three shapes — and flat binary and member/index chains evaluate iteratively. Scripts found underscript/*_scriptkeys in_searchbodies that exceed the limit get an up-front HTTP 400: "script evaluation exceeded maximum depth; split the expression into smaller statements". The supported depth contract itself is unchanged (still 500) — this enforces the existing contract before evaluator recursion instead of during it — and semantics of accepted expressions (precedence, associativity, short-circuiting and error order,doc/params/Mathdispatch) are test-covered as unchanged. Statement nesting no longer consumes the expression-depth budget; it stays bounded by the parser's separate limit.
Changed — internal (no behavior change claimed)
- HNSW read path behind a storage-view seam. Search, construction traversal, diverse-neighbor selection, diagnostics, and graph serialization in the vector index now run through a private read-only storage trait with static dispatch, so a future immutable graph backend can implement the same read contract without rewriting the search path or adding runtime branching. Public API, mutation model, distance behavior (cosine/L2/dot), filtered-search and tombstone semantics, and graph format v2 are all unchanged — a checked-in pre-seam format fixture loads, rewrites, and reopens correctly, and parity tests compare exact result ordering and exact serialized bytes. The commit's controlled A/B (one 10K × 64-d graph, 30K deterministic queries, nine alternating CPU-pinned runs) reports −0.31% median qps with identical checksums, stated to be within run noise: explicitly not a speedup, and this refactor does not by itself reduce ingest memory.
Docs
- calltree.ai case study
(
docs/case-studies/calltree-analytics/): three runnable scripts + README showing onePOST /conversations/_searchcombiningquery.knnwithterms/percentiles/avgaggregations for deep-research questions, over a 130-conversation generated corpus, with a primitives table mapping sub-questions to query shapes. - RAG hybrid retrieval example
(
docs/examples/rag-hybrid-retrieval/): pure-cosine vs BM25 vsquery.hybrid(RRF) over an 18-chunk synthetic corpus with real 768-dim EmbeddingGemma vectors, documenting the pgvector-to-XERJ drop-in migration (same chunks and embeddings, swap the cosineSELECTfor the hybrid_search). The honest result is included: on the identifier query, equal-weight RRF ranks the right chunk #3, not #1, and the README recommends weight tuning or a query router. - daily.dev Postgres CDC case study
(
docs/case-studies/daily-dev-postgres-cdc/): logical-replication sync from a pgvector-equippedposttable into XERJ via_bulk, with both a polledtest_decodingconsumer and a push-streaming consumer with LSN checkpointing (live-verified resuming fromconfirmed_flush_lsnafter a stop/change/restart). A correction is recorded in place: the first draft's "no fused RRF node" limitation was the author's own request-syntax error, not an engine gap —query.hybridwithfusion: {type: rrf}is the real syntax. - Semantic-analytics use case on xerj.org
(
landing/use-cases/semantic-analytics.html, wired into the use-case subnav, hub, solutions index, sitemap, andllms.txt/llms-full.txt): documents the rc.6 kNN+aggregations behavior, including every gap listed under Known limitations below. All corpora in this Docs group are small and synthetic, and nothing in them was timed at any corpus size; daily.dev and calltree.ai are the parties whose questions motivated the case studies, not customers. - Production-deployment recipe gains "Bound retained segment
hydration" (the budget knob, env override, clamping, refusal
semantics, and the stats fields); the passage-retrieval recipe gains
the
fields: ["_passage"]example and response shape.
Known limitations
Known-open items in this release's own features, stated here rather than left implied:
significant_termsover a kNN slice returns empty buckets — the vector path does not yet supply a background corpus. Usetermsfor raw in-slice counts, or the two-step kNN → ids → aggregation pattern when statistical significance is needed. Filed as a follow-up.- Aggregations on a kNN query are bounded by
k: buckets count the retrieved neighbour set only, never the full matching corpus, andhits.total.valueechoes the neighbour-pool size, not a match count — the documented top-level-knn semantics. - An aggs-bearing kNN request always runs the exact brute-force
scan: cost scales with corpus size × dims,
num_candidateshas no effect there, and HNSW never serves it. A hybrid (RRF) query carryingaggsis rejected with a 400, and a kNN clause insidebool.filteris not peeled and returns no buckets — slice aggregation works for pure kNN only. - The hydration budget is not an RSS bound — it caps conservatively estimated retained payload and keys only, and a single highly compressed segment can still create a transient decode peak before admission; bounded/streaming decode is a separate follow-up.
- The selective-hydration primitive is not wired into any engine read path or the shared cache budget yet, and LZ4 columns still require whole-column decompression before selective materialization.
- Request decompression handles gzip only (deflate/br/zstd request bodies remain unsupported); response compression is unchanged.
- HNSW graph files written before the entry-header fix still load with the inconsistent header retained; a file is only corrected the next time the fixed writer saves it.
1.0.0-rc.5 - 2026-07-27
Fifth release candidate: the real-client compatibility release. Much of
it came from pointing real Elasticsearch and OpenSearch tooling at
XERJ — Kibana 8.13, Kibana OSS 7.10.2, OpenSearch Dashboards 2.11.1/3.6,
and their own shipped sample datasets (flights, eCommerce, logs) — and
fixing the wrong answers, 500s, and stalls those clients produced. The rest
is storage, ingest and autoindex work that did not come from client
testing. Headline for users: the query classes that silently matched
zero documents (booleans, .keyword multi-fields, match_phrase on
arrays — and keyword arrays before a flush, see the qualification below)
now return what ES returns; a real Kibana/OSD instance boots,
logs in, and saves objects against XERJ end to end; and a dashboard firing
several panel queries at once no longer stalls the node. ES-YAML
conformance holds at 1360 passed / 0 failed / 3 skipped. Zero-hit defects
found during this cycle that are not fixed here are listed under Known
limitations.
Fixed — query semantics (classes that silently matched zero docs)
- Keyword arrays (partial — memtable half only): a
term/termsquery on a multi-valued keyword field only ever compared element[0]— the memtable stored just the first value and the segment keyword column is one ordinal per doc — so exact lookups returned silent false negatives whiletermsaggregations andmatch(which join all elements) looked correct. The memtable path is fixed: those doc-values lookups now bail for array fields and fall through to the array-aware source scan, so atermon a multi-valued keyword field is correct before a flush. Flushed segments are not fixed — the segment keyword column is still one ordinal per document, so after a flush aterm/termson a non-first array element can still silently miss. The complete fix needs multi-valued segment keyword columns (a storage-format change); the regression test (xerj-enginetest_term_matches_non_first_array_element) is committed but#[ignore]d until that lands. - Booleans:
term/match_phraseon a boolean field undercounted to 0 while atermsaggregation on the same field bucketedtrue/falsecorrectly. The trigger was the count shortcut over memtable-resident data — i.e. any bulk import before an explicit_flush, which is what every real importer (OSD's own sample-data loader included) produces. .keywordmulti-fields: the brute-force scan's field resolver had no multi-field fallback, soterm/match_phraseoncategory.keyword,manufacturer.keywordand friends matched nothing —_sourcenever contains a literal"category.keyword"key. It now strips the trailing segment and retries against the parent when the parent is a leaf value (guarded, so a genuinely absent nested/object field is still absent).match_phrase/match_phrase_prefixon arrays: both arms only handled scalar values, so an array-valued field never matched any phrase. ES semantics restored: the doc matches if any element does.match_phrasewith a non-string query value:{"query": true}was a hard parse error surfaced assearch_phase_execution_exception— the exact shape OSD's filter bar sends for a boolean filter pill, which broke every panel on the dashboard at once. Scalars are now coerced the waymatchalready coerced them.- Empty query strings:
match/match_phrase/match_phrase_prefixwith""returned 400; ES treats an empty analyzed query as zero terms. They now resolve tomatch_none(200, no hits) — this is what Kibana's saved-objects_find?search=*builds, so that endpoint works again. geo_point: aggregations andgeo_distance/geo_bounding_boxrejected string-encoded{"lat": "50.03", "lon": "8.57"}coordinates (the shape the flights sample data ships), andgeo_centroid/geo_boundsused a flat lookup that never found a nested geo field. Both now behave like ES.- Aggregation scripts:
termsrequired a top-levelfieldand returned{"buckets": []}whenever it was absent — any script-based terms aggregation silently produced no buckets.scriptis now a real key source alongsidefield. - Painless date accessors:
doc['t'].valuereturned a plain string, so.getHour(),.getDayOfWeek(),.getYear()and the rest of the common accessor set failed the whole script. They now parse and extract the requested UTC component. - Dynamic date mapping: ISO-8601 strings are inferred as
dateinstead oftext;date_detection: falsewas accepted-but-ignored and is now honored; a non-date value written into an inferred date field returnsmapper_parsing_exceptioninstead of being silently stored and then being invisible to range queries, sorts, and time filters; anddate_detectionsurvives thePUT /_mappingmerge path. - Segment-path parity: projected FTS bool queries dropped
minimum_should_match(somatch_bool_prefixwithmm=3matched every single-term hit after flush) and doc-values wildcards did not case-fold; nested kNN returned zero hits for any parent doc living in a segment because the reassembled{_id, _seq_no, _source}shape was not unwrapped._countnow resolvestermslookups.
Fixed — API surface (what real clients call)
- Aliases: single-index endpoints (
GET /{alias},/_mapping,/_settings, …) never resolved an alias to its backing index, and aliases were never persisted at all — index data survived a restart but the alias pointing at it did not. Aliases now resolve everywhere ES accepts them and are written toaliases.json(atomic temp+rename). Together this unsticks a fresh OSD container looping on "Another OpenSearch Dashboards instance appears to be migrating the index". _field_caps: never listed declared multi-fields (sofields=*omitted every.keywordentry and index-pattern refresh reported "field not found"), and only special-cased the literal*/_allwildcards — any real glob such aswiki-test*returned empty caps. Both fixed; wildcard/comma index specs also now resolve on 9 further endpoints that previously 404'd, 400'd, or — for_refreshand_cat/count— silently reported success / zero documents while doing nothing (_refreshanswered{"_shards":{"successful":1,…}}without refreshing;_cat/countreturned 0, indistinguishable from a genuinely empty index).POST /{index}/_updatenever returned agetblock, so every Kibana saved-object write crashed client-side readingbody.get._source. It is now returned exactly when the caller passes_source/_source_includes/_source_excludes, matching ES (absent otherwise, so_docresponses are byte-identical to before)._sourcevsstored_fields: the implicit_sourcesuppression thatstored_fieldstriggers is now only a default — an explicit top-level_sourcein the same request wins, as in ES. This is what starved almost every column in OSD's Discover._bulkreports each item's real_seq_noand_versioninstead of a wall-clock microsecond timestamp and a hardcoded1._update_by_query/_delete_by_queryhonorwait_for_completion=falseand return the async{"task": …}form;GET /_tasks/{id}reports completion with the final response./_xpackrespects--compat-version(it had its own hardcoded8.13.0, which Kibana OSS 7.10.2 surfaced as an opaque "license not available" refusal to start), and/_xpack+/_xpack/usagereport the real auth state instead ofsecurity.enabled: true— so--insecureno longer makes Kibana render a login screen for auth that is off.- Missing HTTP verbs:
_refresh,_analyze,_msearchaccept GET and_clone/_shrink/_splitaccept PUT, as ES does — the PUT_clone405 was a fatal error during Kibana's saved-object migration. - Login-path endpoints: HTTP Basic auth (Kibana's interactive realm),
POST /_security/profile/_activate,GET /_security/profile/{uid},GET/POST /_security/user/_has_privileges, and a realGET/PUT/DELETE /_security/privilegestore with ES-shaped 404s — each of these was a 500 or a hang on the Kibana login and home pages. GET /_cat/templates/{pattern}(a 404 that crashed OSD) andPOST /_index_template/_simulate(body-only template preview) added.
Changed — index resolution (can break existing callers)
These three align index addressing with ES 8 and turn some previously-succeeding requests into errors. Review any caller that relies on the old behavior.
- A comma-separated index spec now 404s when a concrete name is
missing, implementing ES's default
ignore_unavailable=false. The missing name used to be dropped silently —POST /real,typo/_refreshanswered200with_shards.total: 1. Wildcard/_allspecs that match nothing are still a valid empty result (ES'sallow_no_indicesdefault). - Wildcard and
_allexpansion no longer sweeps in hidden (dot-prefixed) indices, so an operation likePOST /*/_closecan no longer hit.xerj_users-class system indices. A dot-prefixed pattern opts back in, as ES's hidden addressing does. POST /_closerefuses a wildcard or_alltarget outright with ES 8'saction.destructive_requires_nameillegal_argument_exception.
Added
- OpenSearch client auto-sensing. XERJ answers the identity endpoints
(
GET /,GET /_nodes) per request based on the caller's User-Agent, so one running instance serves anopensearch-py/opensearch-js/OSD client and an Elasticsearch client simultaneously, each seeing a version/distribution block its own compatibility gate accepts. Explicit--compat-distribution/--compat-version(and the matching env vars) still pin the identity for every client. Thex-elastic-productheader is no longer sent to a detected OpenSearch caller. - Opt-in ingest memory attribution (developer feature). Setting
XERJ_INGEST_MEMORY_TRACE=summaryemits a boundedxerj.ingest_memory.v1NDJSON ledger — per-owner logical bytes across HTTP bodies, raw and parsed semantic sources, prepared docs and vectors, active memtables and drained flush snapshots — alongside jemalloc allocated/active/resident, RSS, CPU time, and accounting/dropped-event counters, plus a separate/proc-derivedxerj.process_sample.v1stream. Default is fully off. Merge and read-cache owners are reportedunavailable, not zero. This makes ingest retention measurable; it is not itself a memory bound. A deterministic bounded diagnostic suite (demo/usecases/autoindex/scale/bounded/) drives ingest → refresh → flush → force-merge → restart with exact count and sentinel checks. - Private pprof debugging toolkit. A Linux-only
debug-profilingfeature compiles in bounded CPU and jemalloc heap profiling with no network endpoint; artifacts are written mode-0600 to an operator-created directory, withcapture.py/inspect.pywrappers that hash the binary and every artifact. Not in any shipped feature set; no runtime overhead or speedup is claimed. - Experimental ONNX embedding backend, wired through
autoindexbehind theonnx-experimentalfeature. Its lazy session initialization is cancellation-safe: model loading moved to one process-owned thread whose single terminal result is shared, so a cancelled first request can no longer strand every later one. - The ES-YAML conformance runner now exits non-zero when any case fails (it previously reported failures but exited 0).
Changed — performance (concurrent dashboard bursts)
- Full-corpus aggregations no longer deep-clone the memtable under the
shard lock. The
need_full_corpuspath (any request withaggsthat the columnar fast path can't serve) cloned every buffered document's JSON tree while holding the per-shard read guard, serialising concurrent panels behind O(docs) work each. It now Arc-shares out under the lock and clones after releasing it. Measured: 15 concurrentdate_histogramqueries against a real ~14k-doc memtable-resident index, 1.2–2.7 s → ~150–170 ms (commit6a54cf5). - Full-corpus aggregations no longer re-decode segments per query. The
same path did an unconditional open + decompress + full
serde_jsonparse of a segment's entire stored section on every request once data had flushed. Segments are immutable, so it now uses the existing single-flight stored-value cache. Measured: 20 concurrent requests against a real flushed eCommerce segment, 15–26 s → 1–4 ms including the cold first hit (commitf6daf70). A segment read failure during full-corpus assembly is now a hard error rather than a silent skip that undercounted every bucket. - The scan path no longer stalls on cold, concurrent bursts. Two
compounding costs: every scanned doc's
_sourcewas deep-cloned to splice_idin even though only a deeply-nestedidsclause needs it (now computed once per scan from the query shape), and the raw-fallback decode had no single-flight protection, so N concurrent requests against a cold segment each paid the full open+decompress. Measured against real Kibana OSS 7.10.2 and eCommerce sample data: a 24-concurrentquery_string+match_phraseburst immediately after restart (worst-case cold caches) went from ~4.5 s per request to 100–165 ms, with the warm repeat at 1–5 ms (commit7e963ff). - Semantic and vector work is bounded by the request. One absolute
deadline computed at search entry is carried through single-flight
waiting, admission, embedding, hybrid/multi-kNN recursion, and exact
scanning; partials set
timed_outandhits.total.relation: gte, and the ES handler aborts its child task when the request is dropped. Previously a cancelled semantic search could drain for minutes past its timeout. Cold vector segment loads also moved to boundedspawn_blockingproducers so they can no longer pin every async worker.
Fixed — storage, durability & ingest
- Raw ingest validates before it publishes. Caller bytes were appended
to the WAL before being proven to be complete JSON, so malformed input
could reserve sequence numbers and become durable, with the later parse
path silently substituting
{}. Whole-batch UTF-8/JSON/nesting validation now completes before any sequence, WAL, version-map, memtable, or schema mutation, and malformed bulk sources return per-itemdocument_parsing_exception. The parsedValueis now the single authority for turbo ingest (live indexing and WAL replay could previously disagree), andcopy_tois applied before WAL publication so GET, search, flush, replay, and restart all observe the same source. - Per-shard WAL buffers bounded to 64 KiB (was 8 MiB). Every index
eagerly opens one writer per ingest shard, so a 16×16 default reserved
~2 GiB of allocator capacity before a single document was written;
capacity above a frame cannot batch across requests because every
acknowledged append already drains the buffer. Measured on 256 empty
writers: jemalloc allocated 2,147,503,496 → 16,797,064 bytes (−99.2%),
with ingest throughput ratios of 0.976 (single shard) and 1.020
(eight shards) and exact replay preserved (commit
761b915). Cost: records larger than the 64 KiB buffer issue more write syscalls (measured 1,372 → 3,372 on 131 KiB × 1000), with throughput unchanged. - Same-ID writes are serialized. A keyed publication coordinator now
spans the current-state/CAS check through WAL, version map, FTS, and HNSW
publication for single-document paths, closing races that could admit two
creates, lose an update patch, or let a delete overtake a write. Scripted
_updateand_update_by_queryrun through the same boundary, so two concurrentctx._source.n += 1requests can no longer both readn=0. Distinct IDs do not contend. (Turbo batch paths remain follow-up work.) - Semantic vectors stay out of full-text indexes. The dynamic-field
walkers treated pooled embeddings and their
_chunkscompanions as ordinary text and fed every float into FTS as a decimal token. In one failed diagnostic run that produced 2.338 GiB of vector FSTs inside a 3.449 GiB partial index. Embedding outputs are now excluded by schema designation (not a name heuristic) across pre-analysis, flush, and merge. Existing vector FST sidecars are not rewritten on upgrade — reindexing is the safe way to reclaim them. - The admin API key persists across restarts. The server wrote
data_dir/admin.keybut never read it back, minting a fresh key on every start and locking out every client configured against the previous one. An existing well-formed key is now reused; anything missing or malformed still falls through to fresh generation.
Fixed — autoindex
- Content deduplication with crash-safe replacement. Byte-identical
paths were indexed independently (one reproduction journaled 2,442
records for 1,221 live documents). Sources are now hashed with streaming
XXH3-128 and digest peers byte-verified before one canonical path is
chosen; every current name is preserved in
ax_paths. Publication became an explicit durable generation transaction (syncedfile_replace_start→ staged, source-verified extraction → syncedfile_donecommit), and journal replay repairs torn tails and rolls back failed appends. Ambiguous legacy prefix collisions fail closed before any backend mutation instead of guessing. - Follow-on fixes to that rework: resume now guarantees exclusive
plan-key ownership (two files resolving to one key each ran the
replacement transaction and deleted the other's documents);
delete_by_queryrepeats until a pass deletes nothing, because the server implements it as a singlesize:10000pass and >10k-doc generations left permanent ghost documents; and both recoverable error paths now name the offending files or journal offset instead of advising a blanket--freshre-extract. - PDFs are parsed by
pdf_oxidein isolated same-binary workers. The previous byte scanner interpreted font character codes without resolving page resources or ToUnicode maps, producing NUL-separated and shifted text that schema inference (correctly) refused to elect assemantic_text. Workers get process groups, a 1536 MiBRLIMIT_AS, and descendant kill/reap on timeout — documented as resource isolation, not a security sandbox, with no OS memory cap on non-Unix. New--pdf-workersand--pdf-timeout-secsflags. Cost: the no-default release binary grows 34,160,328 → 39,018,184 bytes (+4.6 MiB). - Text files are classified by sentence density, not line length. The
old
avg_len > 60rule split markdown across two datasets with two different field names — adding## headingsto a document made it more likely to be treated as a record stream, which split BM25 statistics across two corpora. Terminal-punctuation density separates prose (0.43–0.57 on the measured corpus) from logs and source (0.00–0.20); CSV rows fall on the record-stream side of the same threshold.
Changed — autoindex (what it produces)
- Line-oriented text is chunked into overlapping windows (40 lines, 10
overlap) instead of one document per line, and prose sections dropped
from 32 KB to 2 KB with overlap, because BM25 scores per document and a
single line rarely contains the caller's own wording. Measured on this
repository (234 files, 170k LOC + docs + 460 commit messages) with 8
"where/why is X" questions: 3/8 → 7/8 answered, 162,883 → 5,508 records,
indexing 234 s → 1.9 s (commit
4510189). Every chunk carriesstart_line/end_line. semantic_textelection is by language, not length. The rule was "largesttextfield withavg_len >= 200", which elected 300-character base64 and concatenated-id columns while skipping a genuinely semantic 150-character summary. A field must now actually look like natural language —word_ratio >= 0.55(tokens matching[A-Za-z]{3,}) andmean_tokens >= 3. Measuredword_ratiois 0.00 for trace_id/user_id/order_id/numeric columns and 0.78–1.00 for prose, log messages and source. This changes which field gets embedded, so it affects both semantic-search quality and ingest cost (the built-in neural backend measures ~2.8 docs/s). The election note records the numbers that drove the decision.
Added — autoindex
--bulk-timeout-secs(default 300, range 1–3600) applies only toPOST /_bulk, so a legitimately slow neural bulk can be accommodated without loosening the deadline on control requests. Retries are bounded at six attempts with the identical body and deterministic document IDs.
Docs
- WordPress core security audit case study
(
docs/case-studies/wordpress-security-audit/): a reproducible record of an AI agent auditing real WordPress core (1,492 PHP files, ~619k lines) with XERJ as the retrieval substrate — sink census, interprocedural taint analysis, authorization graph, POP-gadget hunt — plus a copyable Claude Code skill and a step-by-step playbook. The honest result is that core came back hardened: the documented findings are three Medium-severity, known-class items (an incomplete SSRF deny-list missing169.254.0.0/16, an ImageMagick parse surface, and an unguardedrolesink inuser-new.php), a fourth downgraded to not-reachable-in-core, and verified negatives for IDOR, SQL de-escaping, and sanitizer composition. None of it is claimed as a novel 0-day. The published grep-wins counter-examples are shown alongside the wins, and the XERJ bug the audit surfaced (the keyword-arraytermdefect partially fixed above — the memtable half) is disclosed. Site use case 06 (/use-cases/code-security-audit.html) is built from it. - Token-usage guidebook (
docs/TOKEN_USAGE.md): a measured decomposition of what a XERJ answer costs an agent in tokens — envelope overhead, answer, and materialized intermediate data — instead of hand-waving. - AST + graph + FTS vulnerability research
(
docs/research/ast-graph-vuln-detection.md,docs/examples/ast-vuln-graph/), including a multi-language tree-sitter taint scanner tested at real WordPress scale. - Verified embedding examples: XERJ with Google AI (EmbeddingGemma, Gemini
API, ADK) and with any OpenAI-compatible
/v1/embeddingsendpoint.
Known limitations
Known-open items found while validating this release and not fixed in it. The first three are zero-hit defects — a query returns no documents that ES would return:
multi_matchlooks inverted. On a 6,022-document index a long query returned 0 hits withoperator: "or"(the ES default) and 2 hits withoperator: "and". OR must be a superset of AND.matchagainst asemantic_textfield returns 0 hits for a long natural-language string, while the same query against a plaintextfield returns thousands.term/termson a multi-valued keyword field can still miss after a flush — only the memtable half of that fix landed (see the first bullet of "Fixed — query semantics").autoindexdataset clustering merges same-shape files with different subjects: 213 source files sharing the schema{text}collapse to one index, and embedding centroids separate the subject groups only weakly.- 3 of 1,363 ES-YAML conformance cases are skipped.
1.0.0-rc.4 - 2026-07-22
Fourth release candidate: the production-hardening release. A 9-review release-readiness audit produced 17 blockers and ~60 follow-on items; all were fixed across four hardening waves and verified against the live binary (ES-YAML conformance 1360/1363, full-matrix benchmark 52 WIN / 0 LOSE / 25 TIE vs live ES 8.13.4). Headline for users: acknowledged writes survive crashes, wrong-but-200 responses are gone, the node defends itself under resource pressure, and the bundled Console gains Kibana-quality editable dashboards.
Fixed — durability (acknowledged writes survive failure)
- Acked-write loss closed: verified WAL prune + power-loss-ordered
publish chain;
wal_sync="sync"honored on ALL bulk paths and thewal_batch_msfsync loop actually implemented; torn-frame recovery so a disk-full/crash tear cannot poison a WAL generation; consecutive_bulkdelete actions no longer dropped; acked deletes survive restart (WAL-shard pinning); delete tombstones end WAL pinning segment-durably. - Merge-window reads: GET never 404s during the merge-publish window;
merges can never silently drop docs;
_forcemergeis synchronous and quiescent like ES. - Startup/data safety: exclusive
node.lockon the data dir (second process fails fast); data-dir format marker refuses newer-than-supported or corrupt dirs BEFORE any destructive GC; refuse-on-corrupt snapshot restore; HNSW persistence fsyncs file + dir around rename; periodic background flusher no longer aborted at spawn; sharded-WAL FTS replay restored on reopen.
Fixed — correctness (no silent wrong answers)
- Fail-loud sweep: the silent-wrong-query classes on
_searchare rejected with real 400s (unknown fields, unsupported constructs), as are CCR auto-follow (501), remote reindex,has_child/has_parent, learned fusion, and SQLHAVING— previously all silently returned wrong data. - Doc CRUD wire semantics: real per-doc
_versionand ES seq_no convention;POST /{index}/_doc/{id}route added; malformed bulk docs rejected per-item with ES-shaped 400s instead of stored as empty{}. - Aggregations: real
sum_other_doc_count; composite bucket keys typed from the source field mapping;multi_termsraisestoo_many_bucketsas a real 400 past the cap;top_hitsemits the doc's real_seq_no. - Query semantics: ES-exact date resolution for range bounds (rounding,
format, date math); Painless compares strings as strings (every string
previously compared equal) with depth + source-length guards; highlight
offsets correct on multibyte text;
combined_fieldsOR pooling;query_stringfallback discloses operator handling; kNN threads filter+boost through top-level kNN and honors similarity cutoffs. - Doc-values counting (P0): a
rangefilter on non-numeric values admitted every memtable document insize:0/_count/filter-agg paths (a one-day date window over-counted 3.4×); date/keyword range bounds now compile to the columnar fast path instead of falling to the brute scan. - Multi-valued fields: a field that is multi-valued anywhere in a segment no longer ships a lying doc-values column that silently dropped those docs from count shortcuts — consumers fall back to the exact scan.
Added — resource governance (the node defends itself)
- Parent circuit breaker keyed on ACTUAL RSS, global search pool, disk
flood-stage watermark, per-query memory guard, ANN coverage guard, and a
search timeout that actually preempts term-dictionary walks; scroll and
async-search contexts are TTL-swept and capped. Classic node-killers
(huge
size, deep pagination windows, bucket explosions) return bounded 400/429 instead of taking the process down.
Added — security
- gRPC listener authenticated; health probes exempt from auth; constant-time
compare for the admin API key;
admin.keyand TLS private keys created 0600; CORS configurable and restrictive by default; API keys persist across restart with an honest role surface;/_memorylist paginated with a documented auth model.
Added — Console: Kibana-quality editable dashboards
- Durable backend CRUD for dashboards (create/replace/patch/delete with
ETag optimistic concurrency) — user dashboards survive localStorage
clears AND server restarts; a real panel builder with live preview
(11 viz types, index/query/metric pickers); free-form
{x,y,w,h}panel resize + move; first-launch seeding of 13 built-in dashboards as durable managed rows; edit-mode chrome no longer overlaps titles or the sub-nav.
Added — observability
- ES
_stats/_catsurfaces and the 101-series Prometheus endpoint reflect real load (docs, bytes, search/indexing counters); slow-query log; structured logging minors;_cat/indicesuuid + bytes columns and ES-shaped snapshot responses.
Changed — performance
- kNN flipped: HNSW-served top-level kNN — official benchmark cell 23,325 ms → 1.87 ms at recall@10 1.00 (vs ES 0.80).
- Date-filtered aggregations: 41–49× (one-day window 9.9 s → 241 ms)
via keyword/date columnar range predicates; filtered
extended_stats/percentiles/percentile_ranks/median_absolute_deviationserved columnar with filter-aware gathers (11–264×). - Scored-columnar family at the ES floor: multi_match, query_string,
fuzzy, prefix/wildcard, highlight, match_phrase, deep pagination,
more_like_this,function_score, composite aggs,rare_terms/significant_terms/ percentile families — full-matrix result 52 WIN / 0 LOSE / 25 TIE against live ES 8.13.4. - Mixed read-under-write hardening: one memtable walk per query, flush cap, merge-publish count seeding, open-loop iso-load writer for honest measurement.
Fixed — autoindex & agent search path
xerj autoindexno longer aborts the whole run on ordinary UTF-8 in the SQL-dump sniffer (byte-buffer accumulation; junk files are skipped and recorded, never fatal) and no longer mojibakes non-ASCII SQL values.highlightis applied before_sourcefiltering, so fragment-only responses work (measured: 3.2× fewer tokens into an agent context at equal recall).
Docs
- Honesty ledger: canonical audited scorecard, ROADMAP claims flipped to measured reality, phantom-claim purge across README/site/docs.
- Production recipes: TLS + auth hardening, air-gapped deploy, ES→XERJ migration.
1.0.0-rc.3 - 2026-07-10
Third release candidate. Headline: XERJ gains a built-in neural embedder — real in-process BERT semantics with no Python and no external service — behind a single backend-agnostic embedding handle, plus two new end-to-end-validated retrieval recipes.
Added
- Built-in neural BERT embedder — shipped in the binary. A pure-Rust sentence
encoder via
candle(defaultall-MiniLM-L6-v2, 384-dim) that runs in-process and downloads its weights (~90 MB) automatically on first use (or reads them fromembedding.local_model_dirfor air-gapped deployments). It is compiled into the default release binary — end users just add--embed-mode neuralat runtime, no special build and no separate binary. A progress bar and one-time-download log make the first run legible. The binary is ~36 MB as a result; a--no-default-featuresslim build without the neural backend is ~23 MB. - Unified three-backend embedding handle (
xerj_ai::Embedder).semantic_textingest andsemantic/hybridqueries run through one of three interchangeable backends — lexical (default, zero-dep feature-hash), neural (built-in BERT), or proxy (external OpenAI-compatible/v1/embeddings) — selected withembedding.mode, the--embed-modeflag, orXERJ_EMBED_MODE. Misconfiguration degrades to lexical, never a crash;autopreserves the historical behaviour. - Recipe — All-you-can-eat search. One corpus retrieved five ways from a single
index: full-text (BM25), semantic, vector kNN (more-like-this), hybrid (RRF), and
semantic-scoped-by-keyword-filter. Guide
docs/recipes/all-way-search.md, runnablerecipes/all_way_search.py. - Recipe — Zero-config folder → neural semantic search.
xerj autoindexa mixed-format folder against a--embed-mode neuralserver, then search the discovered prose by meaning while structured files stay exactly filterable. Guidedocs/recipes/autoindex-semantic-search.md, runnablerecipes/autoindex_semantic.sh, sample corpusdemo/data/support-folder/.
Changed
--embed-mode {lexical|neural|proxy|auto}CLI flag andXERJ_EMBED_MODEenv on the server; newembedding.{mode,neural_model,model_cache_dir,local_model_dir}config keys.- Documentation updated for honesty consistency (README, AGENTS.md, ROADMAP.md, llms.txt, recipe guides): the default embedder is lexical; the neural embedder is an opt-in upgrade — output is only described as neural when that mode runs.
1.0.0-rc.1 - 2026-07-06
First public release candidate of XERJ — an Elasticsearch-wire-compatible search, vector, and log-analytics engine written in Rust and licensed under Apache-2.0. This is a release candidate: the wire protocol and on-disk format are considered stable for evaluation, but may still change before the final 1.0.0.
Added
- Elasticsearch-compatible REST API. Drop-in wire compatibility with the ES
8.x HTTP surface, served from
xerj-api(es_compat.rs) on port9200:- Document APIs:
PUT/GET/DELETE /{index}/_doc/{id}andPOST /{index}/_update/{id}. - Search:
POST /{index}/_searchwithquery,from,size,sort,aggs,_source, andhighlight. - Bulk API:
POST /_bulkwithindex,create,update, anddeleteactions. - Scroll API:
POST /{index}/_search?scroll=1mandPOST /_search/scroll. POST /{index}/_delete_by_query, index templates (PUT /_index_template/{name}), and aliases (POST /_aliaseswithadd/remove).
- Document APIs:
- Full-text search (
xerj-fts). BM25 scoring with an analyzer registry and on-disk postings lists. Supported query types includematch_all,match_none,match,match_phrase,match_phrase_prefix,multi_match,term,terms,range,prefix,wildcard,exists,ids,bool,fuzzy,regexp,query_string,simple_query_string,constant_score,boosting,dis_max, andgeo_distance. - Vector search (
xerj-vector). Dense-vector HNSW index for k-NN and semantic search, exposed through theknn,semantic, andhybridquery types. - Aggregations.
terms,stats,avg,sum,min,max,value_count,cardinality,range,histogram,date_histogram,percentiles,filter,missing, andcomposite, with a columnar fast path forsize: 0aggregations. - Sharded ingest and storage (
xerj-storage). Write-ahead log with a single monotonic sequence-number writer, a 16-shard in-memory memtable (shard = xxh3_64(doc_id) & 15), flush to immutable segments, and background segment merging. WAL replay rebuilds both the storage and FTS memtables on restart. - Log analytics (
xerj-logs). Columnar log ingestion with retention. - AI helpers (
xerj-ai). Text chunking, an embedding proxy, and a memory store for semantic workflows. - Clustering (
xerj-cluster). Embedded Raft consensus for cluster metadata with no external dependencies. - Bundled console (
xerj-console-api). Dashboards, auth, preferences, and cluster awareness, compiled into thexerjbinary and mounted under/_xerj-console/api/v1/*. - Transform pipeline (
xerj-wasm). Built-in transform plugins with an optional WASM backend. - Block compression (
xerj-compress). LZ4 and Zstd codecs for segment blocks. - Single static binary.
cargo build --release -p xerj-serverproducesxerj; run with./target/release/xerj --data-dir ./data --insecure. - ES-YAML conformance harness. A workspace test runner (
es-yaml-runner) that executes the ES 8.13 REST-API-spec YAML suites (search, aggregations, vectors, bulk, indices, scroll, cluster) against a live server. XERJ passes 1,326 of 1,329 cases. - Reproducible head-to-head benchmarks. A 91-cell XERJ-vs-Elasticsearch-8.13 matrix (ingest, read, vector, and disk dimensions), published and reproducible at https://xerj.org/benchmarks. The scorecard is honest about both wins and losses.
Changed
_forcemergeis now synchronous and quiescent, matching Elasticsearch semantics, and merge status is exposed through_stats.- Search hit materialization for
size > 0is bounded to the topfrom + sizecandidates, reducing per-query cost from O(N) toward O(from + size). - Bulk ingest avoids redundant JSON round-trips and batches schema evolution to raise throughput under concurrent load.
Fixed
- Consecutive
_bulkdeleteactions that were previously dropped are now applied correctly. hits.totalforsize > 0searches is delete-aware, resolving a conformance regression.- Corrected top-N sort behavior and delete-awareness across the memtable/segment merge path.
Known limitations
- 3 of 1,329 ES-YAML conformance cases do not yet pass.
- This is a release candidate; some Elasticsearch APIs and query/aggregation options
outside the list above are not yet implemented. See
docs/ARCHITECTURE.md and
engine/CLAUDE.mdfor the current supported surface.