AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Batch: land 11 reviewed fixes (rc.100) (#2377)
xerj-orgf6b1d612db4d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9578e82ec..e7187a640 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,148 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added++- `xerj autoindex` names a file family in the run summary when every file of+ that family was junked, or when junked files are a strict majority+ (`junk_files * 2 > files`; stale-plan skips are not counted). The human+ summary and the progress stream print one `warning:` line per family with+ the count and the most common junk reason. `--json` and the catalog run+ document gain `family_junk_warnings` only when that happens; existing+ summary fields are unchanged and exit codes are unchanged. `xerj autoindex+ map` and the MCP `xerj_map` response include the same warnings+ ([#2370](https://github.com/xerj-org/xerj/issues/2370)).++- `xerj autoindex map <folder>` previews a folder that has not been indexed.+ It runs the same discovery as `xerj autoindex <folder> --dry-run` (walk,+ sniff, infer, and print the plan) and writes nothing to an index.+ `xerj autoindex map` with no folder still prints the catalog of what is+ already indexed+ ([#2368](https://github.com/xerj-org/xerj/issues/2368)).++- With `[cluster] enabled = true`, `POST /{index}/_delete_by_query` runs on+ every shard primary, whichever member accepted the request. Each primary+ deletes the documents in its own shard and copies those deletes to the+ in-sync replicas, with the same quorum as `DELETE /{index}/_doc/{id}`.+ `max_docs` is one budget for the index, spent across primaries in+ node-id order, not once per shard. A primary that does not answer+ refuses the request before any document is written. A copy that misses+ quorum is HTTP 503 `replica_not_acknowledged`: the primaries that ran+ have the delete, and the response still reports `deleted` and+ `failures`. `wait_for_completion=false` uses the same path. Native+ `/v1` ingest still writes on the receiving node. A write still needs+ the primary. `[cluster] enabled = false` does not take this path (part of+ [#1170](https://github.com/xerj-org/xerj/issues/1170); the issue stays+ open).++### Fixed++- A graph-path `xerj autoindex` resume whose plan predates files now on disk+ names those files in the summary. The human text under the `ok:` line, the+ lines immediately before `xerj-done`, the run document, and the+ `--progress json` terminal object list the first five paths+ (`skipped_appeared_files`), how many were not listed+ (`skipped_appeared_not_listed`, only when the list was capped), and say to+ re-run with `--fresh`. `skipped_appeared` stays the full count. Exit code 3+ and `reason=stale-resume-plan` are unchanged. The plain `xerj-done` line+ stays `key=value` numbers, because a path may contain spaces+ ([#2372](https://github.com/xerj-org/xerj/issues/2372)).++- `xerj autoindex` indexes `.ts` and `.tsx` sources under Next.js route+ directories such as `app/[locale]/booking/[token]/`. Gitignore reads+ `[locale]` as a character class, so a line written as the directory name+ (`!app/[locale]/**` or `app/[locale]/`) did not name that directory and a+ broader `*` left the sources out of the docs index with no file named.+ Each ignore-file line that contains an unescaped bracket is also applied+ to the literal directory name. A real class such as `*.[ch]` is+ unchanged. A route-like class is explained in the ignore summary, a+ pruned `[…]` directory is named, and a code file that is still skipped+ is named with the rule+ ([#2357](https://github.com/xerj-org/xerj/issues/2357)).++- `xerj autoindex` extracts JavaScript definitions from inline `<script>`+ blocks in HTML pages. `function openDrawer()` inside a `<script>` in a+ `.html` or `.htm` file becomes a code symbol (`defs` / `symbols`, plus a+ per-declaration record), so `xerj def`, `xerj search`, and the MCP def+ and search tools cite that function at its line in the HTML file. A+ `src` script, a non-JavaScript type (`application/ld+json`, a template),+ and `<style>` are not definitions. The page body stays the visible text.+ A page taken as a data table (five or more consistent rows) stays row+ records. An HTML file already sealed in a generation keeps those records+ until the file bytes change or the tree is indexed under a new+ `--state-dir`+ ([#2356](https://github.com/xerj-org/xerj/issues/2356)).++- Proxy embeddings (`--embed-mode proxy`, `[embedding] default_endpoint`)+ accept an OpenAI-style base URL. A path ending in `/v1` or `/v1/`, and a+ bare host, have `/embeddings` appended. A URL that already ends in+ `/embeddings` is POSTed unchanged, and a trailing slash is dropped. Any+ other path refuses to start in `mode = "proxy"` and names the URL that was+ set plus the `{origin}/v1/embeddings` path to use. In `mode = "auto"` that+ same path is an init failure and keeps the existing lexical fallback+ (#1189) instead of being called. A local vLLM base such as+ `http://localhost:8000/v1` used to be POSTed verbatim, so every embedding+ call 404'd. The embedder identity stays the configured string, so a corpus+ indexed with the full path does not become the `/v1` base+ ([#2355](https://github.com/xerj-org/xerj/issues/2355)).++- `xerj autoindex` no longer aborts the whole folder when a file disappears+ between the walk and the hash, or cannot be opened because another process+ holds it locked on Windows (sharing violation 32 / lock violation 33). The+ file is one `skipped` catalog row with a reason, the same way a file that+ changes size mid-run is skipped, and the run finishes (exit 3 when that+ skip is the junk). A locked file that was already indexed keeps the+ previous generation's copy on `--no-graph` and is not treated as an+ exclusion on the graph path. Unix errno 32 stays a real error+ ([#2352](https://github.com/xerj-org/xerj/issues/2352)).++- A failed FTS side-car publish no longer aborts indexing when the partial+ files can be removed, and building a field's FST no longer panics the+ process. The logged error names the source path, the destination path,+ and the OS error (`os error N`). Two side-car names that a case-folding+ volume would store as one file (ASCII case, plus `µ`/`μ` and `ς`/`σ`)+ are refused before the rename, with both paths, instead of failing the+ batch as a not-found error with no file name or silently keeping only+ one field. The segment is still published: queries scan its stored+ section, and a later merge rebuilds full text. A leftover side-car that+ cannot be removed still fails the flush, and that error names the path+ and the OS error. A data-dir format-marker preflight failure still fails+ the flush and leaves the documents uncommitted, so a retry can write the+ marker and the encoded side-cars. This is the failure reported on Windows rc.98, where+ every broad `xerj autoindex` aborted at FTS side-car publish on the+ aggregate `-docs` index, and the fst-build panic from rc.97+ ([#2349](https://github.com/xerj-org/xerj/issues/2349)).++- `xerj search` and MCP `xerj_search` print a hint when the autoindex of a+ folder on this machine is older than the files there. The hint names the+ last index time and `xerj autoindex <folder> --prefix <name>`. Passages+ are printed along with the hint. `xerj code`'s 30-day ledger refusal is+ unchanged. The check reads the catalog run (`summary_generated_at`, or+ `started` when the summary stamp is absent) and stats at most 4096+ entries, newest directory first, stopping after 100ms. Hidden names,+ `.gitignore` / `.xerjignore` / built-in ignores, and a folder that is not+ on this machine produce no hint+ ([#2345](https://github.com/xerj-org/xerj/issues/2345)).++- The node no longer aborts when a search or a flush has to create a tokio+ blocking thread, and `xerj autoindex` no longer aborts when its phase-A+ scan pool or object-fetch pool cannot map a worker's alternate signal+ stack (part of #1122; the issue stays open).+ `std::thread::Builder::spawn` returns success and+ then panics in the child on `ENOMEM`; the release profile is+ `panic = "abort"`. The server runtime caps that blocking pool (twice the+ worker count, plus a slice, at most 1024, or one thread past the worker+ count when that is already above 1024), keeps an idle thread for a day,+ and starts the spare threads at boot so a later `spawn_blocking` or+ `block_in_place` queues. Tokio adds the worker count on top of+ `max_blocking_threads`, so the value passed in is only the non-worker+ slots. The scan and fetch pools start workers with+ `pthread_create` and, if the OS refuses, run the work on the calling+ thread. The object-store I/O runtime is driven on the caller; if it+ cannot be built, that operation fails and the index stays open. A+ global-allocator abort (`memory allocation of N bytes failed`) is+ unchanged.+## [1.0.0-rc.99] - 2026-10-24Window since v1.0.0-rc.98: 116 merged pull requests. x86_64 release binariesdiff --git a/content/answers/autoindex-exit-codes.md b/content/answers/autoindex-exit-codes.mdindex 26261fd3a..b635a7053 100644--- a/content/answers/autoindex-exit-codes.md+++ b/content/answers/autoindex-exit-codes.md@@ -7,7 +7,7 @@ cluster: "Operations: exit codes"question: "What does indexer exit code 3 mean?"intent: "troubleshooting"published: "2026-08-22"-updated: "2026-10-19"+updated: "2026-10-24"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -38,10 +38,10 @@ links_out:- "autoindex-dataset-refused-by-server"- "autoindex-server-back-pressure-429"evidence:- - claim: "EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) - this includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION - the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition."- source: "engine/crates/xerj-autoindex/src/cli.rs:395"+ - claim: "EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) — SUCCESS: the index is searchable, and the summary on stdout opens with a line starting `ok:` that says so (xerj-done carries ok=true). Treat 0 and 3 alike as a finished run. This includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION — the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition."+ source: "engine/crates/xerj-autoindex/src/cli.rs:771"- claim: "Exit 4 is a code of its own; 1 is the catch-all for real failures, and the decision request is answered by re-running the same command with --approve proceed, fast or cancel."- source: "engine/crates/xerj-autoindex/src/cli.rs:395"+ source: "engine/crates/xerj-autoindex/src/cli.rs:771"- claim: "A forced mapping refusal ended xerj-done ok=true exit=3 reason=completed-with-junk wall=0.7s files=1 records=1 generation=1 datasets_refused=1 files_refused=2, with the other dataset indexed."source: "benchmarks/autoindex-resilience/refusal-e2e.run1.stderr.txt"- claim: "Before issue #955 was fixed, a 48,533-file --no-graph run applied every operation and then ended xerj-done ok=false exit=1 reason=aborted wall=10336.0s, because the catalog went out as one bulk request of 51,129 actions and the engine's max_actions_per_bulk is 50,000."@@ -54,7 +54,7 @@ evidence:source: "engine/crates/xerj-autoindex/src/lib.rs:277"faq:- q: "What does indexer exit code 3 mean?"- a: "It means the run completed and recorded junk. `xerj autoindex --help` states it as `3 completed-with-junk (junk recorded, never fatal)`. Treat it as a success with a refusal list to read."+ a: "It means the run completed and recorded junk. `xerj autoindex --help` states it as `3 completed-with-junk (junk recorded, never fatal) — SUCCESS` and says to treat 0 and 3 alike as a finished run. Treat it as a success with a refusal list to read. Do not rerun it as a failure."- q: "The indexer exited 3 (or 4). Did it fail?"a: "3 did not fail. 4 did not fail either, and it did not index anything: it is a decision request waiting for `--approve`. Only 1 and 2 are failures."- q: "Why did folder indexing stop and ask me to approve?"@@ -80,7 +80,7 @@ The list below is quoted from `xerj autoindex --help` on the installed binary. R| Exit | Name in `--help` | What it means for the corpus || --- | --- | --- || 0 | complete | The run finished and refused nothing. A gate answered with `--approve cancel` also exits 0. |-| 3 | completed-with-junk | The run finished. At least one file was refused and recorded. Never fatal. This includes a whole dataset whose mapping the server refused. |+| 3 | completed-with-junk | SUCCESS. The run finished. At least one file was refused and recorded. Treat it the same as exit 0: the index is searchable. This includes a whole dataset whose mapping the server refused. || 4 | NEEDS A DECISION | The estimate exceeded `--max-minutes`. Nothing was indexed. A JSON decision request is on stdout. || 2 | usage | The command line was wrong. Nothing ran. || 1 | endpoint/journal failure | The node or the resume journal failed, a corpus removal was refused, or an unsafe state transition was refused. |@@ -89,7 +89,7 @@ Read the table in that order rather than in numeric order. It is the order the h## Exit 3 is a finished run-`3 completed-with-junk (junk recorded, never fatal)` is the phrase in the help text, and `never fatal` is doing the work in that sentence.+`3 completed-with-junk (junk recorded, never fatal) — SUCCESS` is the phrase in the help text, and it says to treat 0 and 3 alike as a finished run. Exit 3 is success. Do not rerun it as a failure.A folder of real files almost always contains something XERJ will not parse. An unknown binary, an image, a file above `--max-file-gb`. Those files are recorded in the catalog with a reason, and the rest of the folder is indexed and queryable.@@ -111,7 +111,9 @@ So an exit 3 is still a finished run, but it is not always a small gap. Read `daOnly a 400 is a refusal. A 401, 403, 404, 408, 429 or 5xx on the same request says nothing about that dataset, so it still exits 1.-A 429 *on a bulk* is different, whether it is the whole request or some of its items marked `status: 429`. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on; it exits 1 only after 600 seconds of re-sends the node never takes, with an error line that begins `the server kept rejecting`. The terminal line of a run that met back-pressure and finished carries `bulk_retries=N`. The [back-pressure page](/answers/autoindex-server-back-pressure-429) covers it.+A 429 *on a bulk* is different, whether it is the whole request or some of its items marked `status: 429`. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on. It exits 1 only after 600 seconds of re-sends the node never takes.++The error line begins `the server kept rejecting`. The terminal line of a run that met back-pressure and finished carries `bulk_retries=N`. The [back-pressure page](/answers/autoindex-server-back-pressure-429) covers it.## Exit 4 is a question, and nothing was written@@ -153,13 +155,13 @@ xerj-done ok=false exit=1 reason=aborted wall=10336.0serror: prepared bulk contained 1 rejected items: {"type":"engine_exception","reason":"bulk request contains 102258 lines (~51129 actions); exceeds max_actions_per_bulk of 50000","status":413}```-Every `_bulk` body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries `bulk_splits=N` when that happened. Against real nodes with `max_actions_per_bulk = 64` and with `max_body_bytes = 98304`, each run ended `ok=true exit=3 records=1663 bulk_splits=1`, the same records as a control run. Resuming that same generation with the change committed it:+Every `_bulk` body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries `bulk_splits=N` when that happened. Against real nodes with `max_actions_per_bulk = 64` and with `max_body_bytes = 98304`, each run ended `ok=true exit=3 records=1663 bulk_splits=1`, the same documents as a control run. Resuming that same generation with the change committed it:```textxerj-done ok=true exit=3 reason=completed-with-junk wall=415.0s files=47444 records=821840 generation=1 code_files=34324 code_files_indexed=34324 code_files_junked=0```-One record that is larger than the node's `limits.max_body_bytes` cannot be cut, and it still ends the run with exit 1 and an error that names that setting.+One document that is larger than the node's `limits.max_body_bytes` cannot be cut, and it still ends the run with exit 1 and an error that names that setting.## Do not read the code alone@@ -189,4 +191,8 @@ Pair this with the four-number reconciliation on the [completeness page](/answerThe codes above were read from `xerj autoindex --help` on a built binary and cross-checked against the help string in `engine/crates/xerj-autoindex/src/cli.rs`. Both agree.-One code was forced for this page: the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under `benchmarks/autoindex-resilience/`, together with the full-corpus exit 1 in `finalize-catalog`, the exit 3 of the same generation resumed after issue #955 was fixed (`before-955.stderr.txt`, `after-955.full-corpus-resume.stderr.txt`), and a `reason=server-backpressure` exit 1 forced with a 64 MiB memory cap (`limits-real-node.txt`). The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, `xerj autoindex --help` prints them in one screen.+One code was forced for this page. It is the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under `benchmarks/autoindex-resilience/`.++The folder also holds the full-corpus exit 1 in `finalize-catalog`. It holds the exit 3 of the same generation resumed after issue #955 was fixed (`before-955.stderr.txt`, `after-955.full-corpus-resume.stderr.txt`). It holds a `reason=server-backpressure` exit 1 forced with a 64 MiB memory cap (`limits-real-node.txt`).++The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, `xerj autoindex --help` prints them in one screen.diff --git a/content/answers/check-codebase-index-is-complete.md b/content/answers/check-codebase-index-is-complete.mdindex 51636d1ed..343134182 100644--- a/content/answers/check-codebase-index-is-complete.md+++ b/content/answers/check-codebase-index-is-complete.md@@ -7,7 +7,7 @@ cluster: "Operations: completeness"question: "My codebase indexer says indexed but I don't see my code. How do I check it actually finished?"intent: "troubleshooting"published: "2026-08-21"-updated: "2026-10-19"+updated: "2026-10-24"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -39,7 +39,7 @@ faq:- q: "How do I verify a local code index actually contains my files?"a: "Query for a symbol you know exists. The captured query on the defs field for merge_segments returned 2 hits, src/lib.rs in Rust and src/ingest.py in Python, each with its path."- q: "The indexer exited. Did it finish or just stop?"- a: "Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused, and exit 3 with reason=completed-with-junk means the run finished and refused at least one file."+ a: "Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused. Exit 3 with reason=completed-with-junk is success: the run finished and refused at least one file. Do not treat exit 3 as a failure."- q: "Why does the index hold more documents than files?"a: "XERJ writes one document per extracted row, and for source files it adds one document per declaration on top of the whole-file document. The document count therefore follows how many declarations your code holds, not a fixed multiple of the file count, so read it from your own run."- q: "Why does the plan hold fewer files than the folder?"diff --git a/content/answers/read-autoindex-progress.md b/content/answers/read-autoindex-progress.mdindex e7678f112..9596292bc 100644--- a/content/answers/read-autoindex-progress.md+++ b/content/answers/read-autoindex-progress.md@@ -7,7 +7,7 @@ cluster: "Operations: progress"question: "How can I see whether folder indexing is still working?"intent: "how-to"published: "2026-08-21"-updated: "2026-10-19"+updated: "2026-10-24"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -148,15 +148,15 @@ Every run shows `phase=starting` until its first real phase opens. On that resumThe `index` phase applies one file at a time, so it is the slow one on a large corpus. Read `eta_s` once `eta_quality` leaves `unknown`.-## scan at 100% now means scan+## `scan` at 100% now means `scan`-Through v1.0.0-rc.74 the `--no-graph` route reported only `walk`, `hash` and `scan`. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the scan that had already finished:+Through v1.0.0-rc.74 the `--no-graph` route reported only `walk`, `hash` and `scan`. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the `scan` that had already finished:```textxerj-progress phase=scan basis=bytes pct=100.0 items=48533/48533 bytes=520892779/520892779 rate=17050775.5 eta_s=unknown eta_quality=stalled since_progress_s=15.0 phase_elapsed_s=25.7 elapsed_s=30.0```-The rc.74 capture holds 48 such lines, with `since_progress_s` climbing to 250.0. A real hang at the end of the scan prints exactly the same thing. Neither a person nor an agent could tell the two apart.+The rc.74 capture holds 48 such lines, with `since_progress_s` climbing to 250.0. A real hang at the end of the `scan` prints exactly the same thing. Neither a person nor an agent could tell the two apart.On a current build each of those steps is a phase, and the small-repository capture holds 0 lines that read `scan` at `pct=100.0`. If you are on rc.74 or earlier and see that line, check `_count` on the node before you conclude the run is hung.@@ -168,7 +168,9 @@ On a current build each of those steps is a phase, and the small-repository captxerj-done ok=true exit=0 reason=completed wall=22.2s files=1 records=164441 datasets=1 junk_files=0```-`reason` distinguishes `completed`, `dry-run`, `completed-with-junk`, `aborted` and, on the `--no-graph` path, `server-backpressure`, and the exit code follows it. Exit 3 with `completed-with-junk` means the run refused some files, and the catalog holds a reason for each one. `server-backpressure` is exit 1: the node kept rejecting for the whole 600 s a bulk waits, and the line adds `ops_applied` and `ops_remaining` so you know how much the same command still has to do.+`reason` distinguishes `completed`, `dry-run`, `completed-with-junk`, `aborted` and, on the `--no-graph` path, `server-backpressure`, and the exit code follows it. Exit 3 with `completed-with-junk` means the run refused some files, and the catalog holds a reason for each one. `server-backpressure` is exit 1. The node kept rejecting for the whole 600 s a bulk waits. The line adds `ops_applied` and `ops_remaining` so you know how much the same command still has to do.++Exit 3 is success. Treat it the same as exit 0. Do not rerun it as a failure.If the server refused a whole dataset, the line also carries `datasets_refused` and `files_refused`. They appear only when it happened. The [refused-dataset page](/answers/autoindex-dataset-refused-by-server) covers that case.diff --git a/content/answers/why-autoindex-skipped-files.md b/content/answers/why-autoindex-skipped-files.mdindex 2a15442a8..def88214a 100644--- a/content/answers/why-autoindex-skipped-files.md+++ b/content/answers/why-autoindex-skipped-files.md@@ -7,7 +7,7 @@ cluster: "Operations: exclusions"question: "Why would a folder search miss files that I can see on disk?"intent: "troubleshooting"published: "2026-08-21"-updated: "2026-10-19"+updated: "2026-10-24"author: "XERJ documentation team"reviewer: "XERJ engineering team"schema_type: "TechArticle"@@ -43,7 +43,7 @@ faq:- q: "Does autoindex index my .env file?"a: "Not in this capture. Hidden dotfiles were pruned, and a query across _all for the fixture secrets returned 0 hits. Ignore rules are a filter, not an access control."- q: "What does exit code 3 from autoindex mean?"- a: "The run finished but refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals."+ a: "Exit 3 is success. The run finished and refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals. Do not rerun it as a failure."---**TL;DR** — XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, and excluded 23 through ignore rules. The run exited 3 with `reason=completed-with-junk`.@@ -115,6 +115,6 @@ The captured run ended with a terminal line that names both the outcome and thexerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0```-Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` means the catalog holds at least one reason worth reading.+Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` is success: the run finished, the index is searchable, and the catalog holds at least one reason worth reading. Do not rerun exit 3 as a failure.Every number above comes from a capture on 2026-10-05, on a build of `main` after v1.0.0-rc.82 (commit `06185966`). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as `binary content (unknown)`. The binary was a `quick` profile build, so the `wall` figure is not a performance number.diff --git a/docs/RERANK.md b/docs/RERANK.mdindex 9443877e7..184fb92c6 100644--- a/docs/RERANK.md+++ b/docs/RERANK.md@@ -379,7 +379,10 @@ the picture:`http://localhost:9200` unless you say otherwise. On an empty corpus ledger,`xerj code` and `xerj corpus list` also POST one catalog search to that samenode URL (autoindex run prefixes, and the API key when one is configured);- no document text and no user query.+ no document text and no user query. After a successful search, `xerj search`+ and MCP `xerj_search` POST one catalog search to that same node URL (the+ autoindex run for the prefix, and the API key when one is configured) so a+ folder older than its files can be named; no document text and no user query.`xerj batch-search` sends the batch's queries or identifiers to `/_msearch`and optionally reads index mappings at that node, with the configured APIkey when present (`--url` / `--auth` override `XERJ_URL` / `XERJ_AUTH`).diff --git a/docs/examples/external-embeddings/README.md b/docs/examples/external-embeddings/README.mdindex b526aecd3..9f59ca97c 100644--- a/docs/examples/external-embeddings/README.md+++ b/docs/examples/external-embeddings/README.md@@ -37,10 +37,24 @@ the endpoint in your config). The API key is read from the`XERJ_EMBEDDING_API_KEY` environment variable and sent as`Authorization: Bearer <key>`.-The wire contract is the standard one:+`default_endpoint` is not always the request URL. Accepted forms, trailing+slash optional:++| configured | request URL |+|---|---|+| `http://127.0.0.1:8900/v1` or `…/v1/` | `http://127.0.0.1:8900/v1/embeddings` |+| `http://127.0.0.1:8900` or `…/` | `http://127.0.0.1:8900/embeddings` |+| `http://127.0.0.1:8900/v1/embeddings` | unchanged |++Any other path refuses to start when `mode = "proxy"`. The error names the+URL that was set and suggests `{origin}/v1/embeddings`. The string in the+config stays the embedder identity, so switching a corpus from the full path+to the `/v1` base is a different embedder.++The wire contract is the standard one, posted to that request URL:```-POST {default_endpoint}+POST {request URL}{"input": ["text one", "text two"], "model": "<default_model>"}→ {"data": [{"embedding": [ ... ], "index": 0}, { ... }]}```@@ -55,9 +69,9 @@ Any endpoint that speaks the contract above works — just change| provider | `default_endpoint` ||---|---|-| OpenAI | `https://api.openai.com/v1/embeddings` |+| OpenAI | `https://api.openai.com/v1/embeddings` or `https://api.openai.com/v1` || Gemini (OpenAI-compatible mode) | `https://generativelanguage.googleapis.com/v1beta/openai/embeddings` |-| Local (text-embeddings-inference, LM Studio, llama.cpp server, …) | `http://localhost:PORT/v1/embeddings` |+| Local vLLM, Ollama, text-embeddings-inference, LM Studio, llama.cpp server, … | `http://localhost:PORT/v1` or `http://localhost:PORT/v1/embeddings` |> Verified in this repo against the reproducible mock, which implements the> exact request/response shape XERJ sends. Real providers that speak the samediff --git a/docs/examples/external-embeddings/xerj-proxy.toml b/docs/examples/external-embeddings/xerj-proxy.tomlindex 3f2ebd5a4..010ac9fa3 100644--- a/docs/examples/external-embeddings/xerj-proxy.toml+++ b/docs/examples/external-embeddings/xerj-proxy.toml@@ -1,5 +1,8 @@[embedding]mode = "proxy"+# Full embeddings path (used unchanged). `http://127.0.0.1:8900/v1` and+# `http://127.0.0.1:8900/v1/` also work: `/embeddings` is appended. A bare+# host gets `/embeddings`. Any other path refuses to start.default_endpoint = "http://127.0.0.1:8900/v1/embeddings"default_model = "mock-embed-256"batch_size = 64diff --git a/docs/examples/google-ai/README.md b/docs/examples/google-ai/README.mdindex bea7bdad7..32adf2bb1 100644--- a/docs/examples/google-ai/README.md+++ b/docs/examples/google-ai/README.md@@ -33,7 +33,9 @@ xerj --insecure --data-dir ./data --config embeddinggemma.toml````embeddinggemma.toml` points XERJ's proxy at `http://127.0.0.1:11434/v1/embeddings`,-model `embeddinggemma`.+model `embeddinggemma`. A base of `http://127.0.0.1:11434/v1` (trailing slash+optional) is the same request: `/embeddings` is appended. A bare host gets+`/embeddings`. Any other path refuses to start and names the URL to set.**Verified end-to-end in this repo.** Four docs indexed through EmbeddingGemma,then three semantic queries with *no shared words* with their answers:@@ -57,8 +59,9 @@ xerj --insecure --data-dir ./data --config gemini-embeddings.toml`gemini-embeddings.toml` uses the endpoint from Google's OpenAI-compatibilitydocs — `https://generativelanguage.googleapis.com/v1beta/openai/embeddings`,-model `gemini-embedding-001`. XERJ sends the key as `Authorization: Bearer`,-which is exactly what that endpoint expects.+model `gemini-embedding-001`. That path already ends in `/embeddings`, so it+is used unchanged; a `/v1` base would not reach this route. XERJ sends the key+as `Authorization: Bearer`, which is exactly what that endpoint expects.> Verified structurally: the config matches Google's documented contract, and> that contract is proven working by `run-offline-test.sh` (a mock speaking thediff --git a/docs/examples/google-ai/embeddinggemma.toml b/docs/examples/google-ai/embeddinggemma.tomlindex 8a1ff05a6..e0fb449c1 100644--- a/docs/examples/google-ai/embeddinggemma.toml+++ b/docs/examples/google-ai/embeddinggemma.toml@@ -2,6 +2,8 @@# OpenAI-compatible endpoint. Fully local, no API key, no cloud.[embedding]mode = "proxy"+# Full path, used unchanged. `http://127.0.0.1:11434/v1` also works+# (`/embeddings` is appended). A bare host gets `/embeddings`.default_endpoint = "http://127.0.0.1:11434/v1/embeddings"default_model = "embeddinggemma"batch_size = 32diff --git a/docs/examples/google-ai/gemini-embeddings.toml b/docs/examples/google-ai/gemini-embeddings.tomlindex 6aa1ca940..dc9db7f1f 100644--- a/docs/examples/google-ai/gemini-embeddings.toml+++ b/docs/examples/google-ai/gemini-embeddings.toml@@ -3,6 +3,8 @@# then start: xerj --config gemini-embeddings.toml[embedding]mode = "proxy"+# Full embeddings path, used unchanged. A base ending in `/v1` would have+# `/embeddings` appended; this Gemini route does not, so the full path is required.default_endpoint = "https://generativelanguage.googleapis.com/v1beta/openai/embeddings"default_model = "gemini-embedding-001"batch_size = 64diff --git a/docs/recipes/air-gapped-deployment.md b/docs/recipes/air-gapped-deployment.mdindex 8f1c9046e..ef49b6354 100644--- a/docs/recipes/air-gapped-deployment.md+++ b/docs/recipes/air-gapped-deployment.md@@ -18,7 +18,10 @@ an optional, separately staged path.`local_model_dir` set, XERJ reads those files locally and does not contactthe Hub.- `proxy` calls only the configured OpenAI-compatible- `embedding.default_endpoint`. The experimental `onnx-experimental` backend+ `embedding.default_endpoint`. A base ending in `/v1` (trailing slash+ optional) or a bare host has `/embeddings` appended; a path that already+ ends in `/embeddings` is used unchanged; any other path refuses to start+ in `mode = "proxy"` and names the URL to set. The experimental `onnx-experimental` backendneeds an ONNX-enabled build and explicit local assets; neither is the baserelease path.diff --git a/docs/recipes/passage-retrieval.md b/docs/recipes/passage-retrieval.mdindex 68bca24e9..6c4b18c89 100644--- a/docs/recipes/passage-retrieval.md+++ b/docs/recipes/passage-retrieval.md@@ -154,8 +154,9 @@ so it doubles as a CI gate.one vector and scores exactly as it did pre-pipeline; no existing behaviorchanges.- **The built-in embedder is lexical**, not neural — great for offline demos- and CI. Point XERJ at an OpenAI-compatible `/v1/embeddings` endpoint- (`[embedding]` in `xerj.toml`) for production semantics; the pipeline is+ and CI. Point XERJ at an OpenAI-compatible embeddings endpoint+ (`embedding.default_endpoint` in `xerj.toml`: a `/v1` base, a bare host,+ or the full `/v1/embeddings` URL) for production semantics; the pipeline isidentical, just higher-quality per-passage vectors.- **Scoring is exact (brute-force max-sim)** over the passage vectors, which iswhat makes the ranking deterministic on the corpus sizes these recipes use.diff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.mdindex c28c85d25..0170a7446 100644--- a/docs/recipes/semantic-search-rag.md+++ b/docs/recipes/semantic-search-rag.md@@ -149,7 +149,7 @@ query uses it. Choose with `--embed-mode` (or `embedding.mode` in config, or the|---|---|---|---|| `lexical` *(default)* | Built-in feature-hash | Lexical overlap only | None — offline, deterministic || `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | None — ships in the binary; model auto-downloads on first use |-| `proxy` | Any external OpenAI-compatible `/v1/embeddings` | Whatever model you point at | Set `embedding.default_endpoint` |+| `proxy` | Any external OpenAI-compatible embeddings API | Whatever model you point at | Set `embedding.default_endpoint` to a `/v1` base, a bare host, or the full `/v1/embeddings` URL |### The default is lexical — know where it stops@@ -192,8 +192,11 @@ Two honest takeaways that still apply in every mode:the `semantic_text` field too (or add a second `semantic_text` field and combinewith `hybrid`), so the vocabulary the user might use is actually in the vector.- **Bring your own model when you need a specific one.** `--embed-mode proxy` with- `embedding.default_endpoint` calls any OpenAI-compatible `/v1/embeddings`- service (OpenAI, Cohere, a local vLLM, etc.). Embedding happens at ingest time.+ `embedding.default_endpoint` calls any OpenAI-compatible embeddings+ service (OpenAI, a local vLLM, etc.). A base ending in `/v1` or `/v1/`,+ and a bare host, have `/embeddings` appended; a URL that already ends in+ `/embeddings` is used unchanged. A trailing slash is ignored. Any other+ path refuses to start and names the URL to set. Embedding happens at ingest time.Everything in the numbered walk-through above uses the default lexical embedder —no external service was configured for this recipe.diff --git a/docs/recipes/vector-search-knn.md b/docs/recipes/vector-search-knn.mdindex cdb13fb62..c9bb5c140 100644--- a/docs/recipes/vector-search-knn.md+++ b/docs/recipes/vector-search-knn.md@@ -260,9 +260,10 @@ vectors.***Honest caveat:* the built-in embedder is **lexical, not neural** —excellent for demos and keyword-semantic overlap, but forproduction-grade semantic quality point XERJ at an external- OpenAI-compatible `/v1/embeddings` endpoint- (`embedding.default_endpoint`), after which both ingest and query use- that higher-quality model.+ OpenAI-compatible embeddings endpoint+ (`embedding.default_endpoint`: a `/v1` base, a bare host, or the full+ `/v1/embeddings` URL; any other path refuses to start), after which both+ ingest and query use that higher-quality model.Rule of thumb: **already have embeddings → raw `knn`. Have text and wantit embedded for you → `semantic_text`.** Need to blend keyword relevancediff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex dadc6ca17..d6fc941ae 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -57,7 +57,8 @@ next: `xerj autoindex map --url http://localhost:9280` for the data map; search```The exit code is `3` — "completed with junk": the run finished, and the-binary blob was recorded rather than crashing anything. `0` means a fully+binary blob was recorded rather than crashing anything. Exit 3 is success.+Treat it the same as exit 0: do not rerun it as a failure. `0` means a fullyclean run; junk is *never* fatal. Exit 3 also covers a dataset whose mappingthe server refused — see [section 7](#7-one-dataset-the-server-refuses-does-not-cost-you-the-run).@@ -67,6 +68,11 @@ the server refused — see [section 7](#7-one-dataset-the-server-refuses-does-noxerj autoindex map```+`xerj autoindex map` with no folder reads that catalog from the node. To+preview a folder that has not been indexed yet, run+`xerj autoindex map <folder>` (the same walk, sniff and plan as+`xerj autoindex <folder> --dry-run`). That preview writes nothing to an index.+The data map is a briefing rendered from the catalog: datasets, inferredper-field types with real example values, time ranges, ready-to-sendqueries, and the junk report. Excerpts from the real output:@@ -497,11 +503,15 @@ built-in list are XERJ's rather than git's and apply throughout the folder younamed. Hidden files (`.env`, `.git/`) are skipped by a separate, non-optionalrule. Every run prints what was dropped and by which rule (`ignore rules: …` onstderr), and `--dry-run` also counts the non-hidden files inside each pruned-directory, so "where did my file go?" always has an answer. `--no-ignore` turns+directory, so "where did my file go?" always has an answer. A line that+writes `[locale]` is also applied to the literal directory of that name, so a+Next.js route folder is not dropped because git reads the brackets as a+character class; `*.[ch]` still matches `.c` and `.h`. `--no-ignore` turnsthe rules off; `--no-default-ignores` keeps your ignore files and drops only the-built-in list. Both apply to an indexing run only — `autoindex map` and-`autoindex status` never walk a folder and refuse them rather than accepting a-flag that would do nothing. The folder you point at is never excluded: if it is+built-in list. Both apply to an indexing run and to `autoindex map <folder>`+(the read-only preview). A bare `autoindex map` and `autoindex status` never+walk a folder and refuse them rather than accepting a flag that would do+nothing. The folder you point at is never excluded: if it isitself ignored — say you run this on `~/proj/target` — it is indexed anyway andthe run says which rule it would have matched.diff --git a/engine/crates/xerj-ai/src/embed.rs b/engine/crates/xerj-ai/src/embed.rsindex c38f6e434..f0a5cd2db 100644--- a/engine/crates/xerj-ai/src/embed.rs+++ b/engine/crates/xerj-ai/src/embed.rs@@ -24,7 +24,21 @@ pub type Result<T> = std::result::Result<T, XerjError>;/// Configuration for the embedding proxy.#[derive(Debug, Clone, Serialize, Deserialize)]pub struct EmbeddingProxyConfig {- /// API endpoint URL (e.g., `https://api.openai.com/v1/embeddings`).+ /// API endpoint URL.+ ///+ /// Before [`EmbeddingProxy::new`], this is the configured value+ /// (`embedding.default_endpoint`). `new` rewrites it to the URL that is+ /// actually POSTed (#2355). Accepted forms, trailing slash optional:+ ///+ /// * a base whose path ends in `/v1` — `/embeddings` is appended+ /// (`http://localhost:8000/v1` → `http://localhost:8000/v1/embeddings`)+ /// * a bare origin (`http://host:port` or `http://host:port/`) —+ /// `/embeddings` is appended+ /// * a path that already ends in `/embeddings` — used unchanged, aside+ /// from dropping a trailing slash+ ///+ /// Any other path is refused here, before a request is sent. The error+ /// names the configured URL and suggests `{origin}/v1/embeddings`.pub endpoint: String,/// API key (sent as `Authorization: Bearer <key>`).pub api_key: Option<String>,@@ -135,6 +149,69 @@ fn status_is_retryable(code: u16) -> bool {code >= 500 || code == 429 || code == 408}+/// `scheme://host[:port]` for `url`, with brackets around an IPv6 host.+fn endpoint_origin(url: &reqwest::Url) -> String {+ let host = url.host_str().unwrap_or("");+ let host = if host.contains(':') {+ format!("[{host}]")+ } else {+ host.to_string()+ };+ match url.port() {+ Some(port) => format!("{}://{host}:{port}", url.scheme()),+ None => format!("{}://{host}", url.scheme()),+ }+}++/// Request URL for one OpenAI-compatible embeddings call (#2355).+///+/// `default_endpoint` used to be POSTed verbatim, so the usual client base+/// `http://host:port/v1` 404'd. A base ending in `/v1` (trailing slash+/// optional) and a bare origin have `/embeddings` appended. A path that+/// already ends in `/embeddings` is kept, with a trailing slash removed so+/// `/v1/embeddings/` and `/v1/embeddings` hit the same route. Query strings+/// stay. Any other path is refused: appending would guess a route and send+/// document text there. The error names the configured URL and the full+/// `{origin}/v1/embeddings` path to set instead.+fn resolve_embeddings_endpoint(raw: &str) -> Result<String> {+ let configured = raw.trim();+ let mut url = reqwest::Url::parse(configured)+ .map_err(|e| XerjError::embedding(format!("invalid embedding endpoint: {e}")))?;+ if !matches!(url.scheme(), "http" | "https") {+ return Err(XerjError::embedding(+ "embedding endpoint must use http or https",+ ));+ }+ if url.host_str().is_none() {+ let suggestion = format!("{configured}/v1/embeddings");+ return Err(XerjError::embedding(format!(+ "embedding endpoint `{configured}` has no host, so it was not called. \+ Set embedding.default_endpoint to `{suggestion}`."+ )));+ }++ let trimmed = url.path().trim_end_matches('/');+ let path = if trimmed.is_empty() { "/" } else { trimmed };+ let resolved = if path == "/" {+ "/embeddings".to_string()+ } else if path.ends_with("/embeddings") {+ path.to_string()+ } else if path.ends_with("/v1") {+ format!("{path}/embeddings")+ } else {+ let origin = endpoint_origin(&url);+ let suggestion = format!("{origin}/v1/embeddings");+ return Err(XerjError::embedding(format!(+ "embedding endpoint `{configured}` is not an embeddings URL, so it was not called. \+ A base URL ending in `/v1` or `/v1/` has `/embeddings` appended, and so does a bare \+ host (`{origin}` or `{origin}/`). Any other path must already end in `/embeddings`. \+ Set embedding.default_endpoint to `{suggestion}`."+ )));+ };+ url.set_path(&resolved);+ Ok(url.as_str().to_string())+}+// ─────────────────────────────────────────────────────────────────────────────// EmbeddingProxy// ─────────────────────────────────────────────────────────────────────────────@@ -152,14 +229,13 @@ pub struct EmbeddingProxy {impl EmbeddingProxy {/// Create a new proxy with the given config.- pub fn new(config: EmbeddingProxyConfig) -> Result<Self> {- let endpoint = reqwest::Url::parse(&config.endpoint)- .map_err(|e| XerjError::embedding(format!("invalid embedding endpoint: {e}")))?;- if !matches!(endpoint.scheme(), "http" | "https") {- return Err(XerjError::embedding(- "embedding endpoint must use http or https",- ));- }+ ///+ /// Resolves [`EmbeddingProxyConfig::endpoint`] before the client is+ /// built, so a `/v1` base is completed and an unrecognized path fails+ /// here — at startup, when `mode = "proxy"` — rather than as a 404 on+ /// the first document.+ pub fn new(mut config: EmbeddingProxyConfig) -> Result<Self> {+ config.endpoint = resolve_embeddings_endpoint(&config.endpoint)?;let client = reqwest::Client::builder().timeout(Duration::from_millis(config.timeout_ms)).build()@@ -174,6 +250,11 @@ impl EmbeddingProxy {})}+ /// URL this proxy POSTs to, after base-URL completion (#2355).+ pub fn request_url(&self) -> &str {+ &self.config.endpoint+ }+/// Embed a batch of texts using the configured model.////// Returns one embedding vector per input text, in the same order.@@ -358,6 +439,131 @@ mod tests {}}+ /// #2355: an OpenAI-style base is completed; the full embeddings URL is not+ /// rewritten; a trailing slash does not change the request path.+ #[test]+ fn resolves_v1_base_trailing_slash_bare_host_and_full_url() {+ let cases = [+ (+ "http://127.0.0.1:8000/v1",+ "http://127.0.0.1:8000/v1/embeddings",+ ),+ (+ "http://127.0.0.1:8000/v1/",+ "http://127.0.0.1:8000/v1/embeddings",+ ),+ (+ "http://127.0.0.1:8000/v1/embeddings",+ "http://127.0.0.1:8000/v1/embeddings",+ ),+ (+ "http://127.0.0.1:8000/v1/embeddings/",+ "http://127.0.0.1:8000/v1/embeddings",+ ),+ ("http://127.0.0.1:8000", "http://127.0.0.1:8000/embeddings"),+ ("http://127.0.0.1:8000/", "http://127.0.0.1:8000/embeddings"),+ (+ "https://api.openai.com/v1",+ "https://api.openai.com/v1/embeddings",+ ),+ (+ "https://generativelanguage.googleapis.com/v1beta/openai/embeddings",+ "https://generativelanguage.googleapis.com/v1beta/openai/embeddings",+ ),+ (+ "http://127.0.0.1:8000/proxy/v1",+ "http://127.0.0.1:8000/proxy/v1/embeddings",+ ),+ (+ "http://127.0.0.1:8000/v1?api-version=2024",+ "http://127.0.0.1:8000/v1/embeddings?api-version=2024",+ ),+ ("http://[::1]:8000/v1/", "http://[::1]:8000/v1/embeddings"),+ ];+ for (input, expect) in cases {+ let proxy = EmbeddingProxy::new(EmbeddingProxyConfig::new(input, "m"))+ .unwrap_or_else(|e| panic!("{input} was refused: {e}"));+ assert_eq!(proxy.request_url(), expect, "{input}");+ }+ }++ /// #2355: a path that is neither a `/v1` base nor an embeddings URL is+ /// refused before any request, and the error names both the URL that was+ /// set and the full path to use instead.+ #[test]+ fn unrecognized_endpoint_path_fails_at_construction_and_names_the_full_url() {+ let raw = "http://127.0.0.1:8000/openai";+ // `EmbeddingProxy` is not `Debug` (it would print `api_key`), so map+ // the Ok side away before `expect_err`.+ let err = EmbeddingProxy::new(EmbeddingProxyConfig::new(raw, "m"))+ .map(|_| ())+ .expect_err("an unrecognized path must be refused");+ let msg = err.to_string();+ assert!(msg.contains(raw), "{msg}");+ assert!(msg.contains("http://127.0.0.1:8000/v1/embeddings"), "{msg}");+ assert!(msg.contains("was not called"), "{msg}");+ }++ /// #2355: the completed URL is what the HTTP client actually requests.+ /// A `/v1` base, the same base with a trailing slash, and the full+ /// `/v1/embeddings` URL must all POST to `/v1/embeddings`.+ #[tokio::test]+ async fn posts_v1_base_trailing_slash_and_full_url_to_embeddings() {+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();+ let addr = listener.local_addr().unwrap();+ let handle = std::thread::spawn(move || {+ use std::io::{Read, Write};+ let mut paths = Vec::new();+ for _ in 0..3 {+ let Ok((mut stream, _)) = listener.accept() else {+ break;+ };+ let mut buf = [0u8; 4096];+ let n = stream.read(&mut buf).unwrap_or(0);+ let head = String::from_utf8_lossy(&buf[..n]);+ let path = head+ .lines()+ .next()+ .and_then(|line| line.split_whitespace().nth(1))+ .unwrap_or("")+ .to_string();+ paths.push(path);+ let body = r#"{"object":"list","data":[{"object":"embedding","index":0,"embedding":[0.25]}]}"#;+ let resp = format!(+ "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",+ body.len(),+ body+ );+ let _ = stream.write_all(resp.as_bytes());+ }+ paths+ });++ for suffix in ["/v1", "/v1/", "/v1/embeddings"] {+ let mut cfg = EmbeddingProxyConfig::new(format!("http://{addr}{suffix}"), "m");+ cfg.max_retries = 0;+ let proxy = EmbeddingProxy::new(cfg).unwrap();+ assert!(+ proxy.request_url().ends_with("/v1/embeddings"),+ "{} -> {}",+ suffix,+ proxy.request_url()+ );+ let out = proxy.embed(String::from("hello")).await.unwrap();+ assert_eq!(out, vec![0.25], "{suffix}");+ }++ let paths = handle.join().unwrap();+ assert_eq!(+ paths,+ vec![+ "/v1/embeddings".to_string(),+ "/v1/embeddings".to_string(),+ "/v1/embeddings".to_string(),+ ]+ );+ }+#[tokio::test]async fn embed_empty_batch_returns_empty() {let cfg = EmbeddingProxyConfig::new("http://localhost", "model");diff --git a/engine/crates/xerj-api/src/cluster_shard.rs b/engine/crates/xerj-api/src/cluster_shard.rsindex ad4923bff..468daa5b2 100644--- a/engine/crates/xerj-api/src/cluster_shard.rs+++ b/engine/crates/xerj-api/src/cluster_shard.rs@@ -6,6 +6,8 @@//! and [`bulk_on_primaries`] send `_update` and `_bulk` to that same owner.//! [`update_by_query_on_primaries`] runs `_update_by_query` on every primary//! and each primary copies the documents it changed.+//! [`delete_by_query_on_primaries`] runs `_delete_by_query` on every primary+//! and each primary copies the deletes.//! The primary copies an index, delete, or the document an update stored//! onto the in-sync replicas and acknowledges the caller when a quorum of//! the copies has confirmed. A replica that does not@@ -695,6 +697,23 @@ impl xerj_cluster::ShardHandler for ApiShardHandler {).await}+ ShardOp::DeleteByQuery {+ index,+ query_json,+ max_docs,+ scroll_size,+ check_only,+ } => {+ handle_delete_by_query(+ &self.state,+ &index,+ &query_json,+ max_docs,+ scroll_size,+ check_only,+ )+ .await+ }ShardOp::Search {index,body_json,@@ -2699,7 +2718,7 @@ fn has_replica_nack(body: &Value) -> bool {})}-fn stamp_replica_nack(mut body: Value) -> Value {+fn stamp_replica_nack(mut body: Value, reason: &str) -> Value {if !has_replica_nack(&body) {return body;}@@ -2709,7 +2728,7 @@ fn stamp_replica_nack(mut body: Value) -> Value {"error".to_string(),json!({"type": "replica_not_acknowledged",- "reason": "primary applied the update but a replica copy missed quorum. The primary has the update; retry the update so an in-sync replica can take the copy"+ "reason": reason,}),);}@@ -3023,5 +3042,322 @@ pub(crate) async fn update_by_query_on_primaries(if parts_out.is_empty() {return empty_update_by_query();}- stamp_replica_nack(aggregate_by_query_results(parts_out, "updated"))+ stamp_replica_nack(+ aggregate_by_query_results(parts_out, "updated"),+ "primary applied the update but a replica copy missed quorum. The primary has the update; retry the update so an in-sync replica can take the copy",+ )+}++// ── `_delete_by_query` on every primary ─────────────────────────────────────++/// One `_delete_by_query` after the accepting node has validated the query.+///+/// [`delete_by_query_on_primaries`] borrows this. `max_docs` is the caller's+/// cap for the whole index; the fan-out spends it across primaries.+#[derive(Clone)]+pub(crate) struct DeleteByQueryParts {+ pub query: Value,+ pub max_docs: Option<u64>,+ pub scroll_size: usize,+}++/// Box the per-index fan-out so it is not inlined into `delete_by_query`.+///+/// Same split as [`pin_update_by_query_on_primaries`]: a debug worker stack+/// overflows when the handler and the runner are both built on it.+#[inline(never)]+pub(crate) fn pin_delete_by_query_on_primaries(+ state: AppState,+ index: String,+ parts: DeleteByQueryParts,+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Value> + Send>> {+ Box::pin(async move { delete_by_query_on_primaries(&state, &index, &parts).await })+}++fn delete_by_query_op(+ index: &str,+ parts: &DeleteByQueryParts,+ max_docs: Option<u64>,+ check_only: bool,+) -> ShardOp {+ ShardOp::DeleteByQuery {+ index: index.to_string(),+ query_json: parts.query.to_string(),+ max_docs,+ scroll_size: parts.scroll_size as u64,+ check_only,+ }+}++fn delete_by_query_unavailable(index: &str, detail: &str) -> Value {+ json!({+ "error": {+ "type": "shard_not_ready",+ "reason": format!("delete_by_query on [{index}] failed: {detail}")+ },+ "status": 503+ })+}++fn empty_delete_by_query() -> Value {+ json!({+ "took": 0,+ "timed_out": false,+ "total": 0,+ "deleted": 0,+ "batches": 0,+ "version_conflicts": 0,+ "noops": 0,+ "failures": [],+ "throttled_millis": 0,+ "requests_per_second": -1,+ "throttled_until_millis": 0,+ })+}++/// Index open, and this node is a primary when a plan exists.+///+/// `Err` is a body the HTTP layer can return. Nothing has been written.+fn prepare_delete_by_query(state: &AppState, index: &str) -> Result<(), Value> {+ if state.engine.get_index(index).is_err() {+ return Err(serde_json::from_str(&shard_not_ready_body(index))+ .unwrap_or_else(|_| delete_by_query_unavailable(index, "index is not open")));+ }+ let me = state.engine.routing_node_id();+ let targets = state.engine.shard_search_targets(index);+ if !targets.is_empty() && !targets.iter().any(|id| id == &me) {+ let owner = targets.first().map(String::as_str).unwrap_or("unknown");+ let body = not_this_primary(index, owner);+ return Err(value_from_shard(&body));+ }+ Ok(())+}++async fn prepare_delete_by_query_wait(state: &AppState, index: &str) -> Result<(), Value> {+ let deadline = Instant::now() + OWNER_READY_WAIT;+ loop {+ match prepare_delete_by_query(state, index) {+ Err(body) if is_not_ready_value(&body) && Instant::now() < deadline => {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ other => return other,+ }+ }+}++async fn forward_delete_by_query(state: &AppState, index: &str, owner: &str, op: ShardOp) -> Value {+ let deadline = Instant::now() + OWNER_READY_WAIT;+ loop {+ match state.engine.forward_shard(owner, op.clone()).await {+ Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ Ok(result) => return value_from_shard(&result),+ Err(err) => return delete_by_query_unavailable(index, &err.to_string()),+ }+ }+}++async fn copy_deleted_document(state: &AppState, index: &str, id: &str) -> Result<(), String> {+ if state.engine.primary_of(index, id).is_none() {+ return Ok(());+ }+ if !state.engine.owns_shard(index, id) {+ return Err(format!(+ "primary applied the delete on [{index}] but document [{id}] is not owned by this node"+ ));+ }+ let op = ShardOp::DeleteDoc {+ index: index.to_string(),+ id: id.to_string(),+ if_seq_no: None,+ if_primary_term: None,+ replica: false,+ shard: 0,+ };+ replicate_write(state, index, id, op)+ .await+ .map_err(|nack| nack_reason(&nack))+}++async fn apply_delete_by_query_here(+ state: &AppState,+ index: &str,+ parts: &DeleteByQueryParts,+ max_docs: Option<u64>,+) -> Value {+ let idx = match state.engine.get_index(index) {+ Ok(idx) => idx,+ Err(err) => return delete_by_query_unavailable(index, &err.to_string()),+ };+ let recorded = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));+ let mut body = crate::es_compat::run_delete_by_query_recorded_boxed(+ idx,+ parts.query.clone(),+ max_docs,+ parts.scroll_size,+ Some(std::sync::Arc::clone(&recorded)),+ )+ .await;+ let ids = recorded+ .lock()+ .unwrap_or_else(|err| err.into_inner())+ .clone();+ // A failed run may already have deleted some documents. Copy those before+ // returning the error so a replica does not keep them.+ let mut nacks = Vec::new();+ for id in ids {+ if let Err(reason) = copy_deleted_document(state, index, &id).await {+ nacks.push((id, reason));+ }+ }+ if body.get("status").is_some() {+ return body;+ }+ if nacks.is_empty() {+ return body;+ }+ let Some(obj) = body.as_object_mut() else {+ return body;+ };+ let failures = obj+ .entry("failures".to_string())+ .or_insert_with(|| json!([]));+ if let Some(failures) = failures.as_array_mut() {+ for (id, reason) in nacks {+ failures.push(json!({+ "id": id,+ "cause": {+ "type": "replica_not_acknowledged",+ "reason": reason,+ }+ }));+ }+ }+ body+}++async fn handle_delete_by_query(+ state: &AppState,+ index: &str,+ query_json: &str,+ max_docs: Option<u64>,+ scroll_size: u64,+ check_only: bool,+) -> ShardResult {+ if let Err(body) = prepare_delete_by_query_wait(state, index).await {+ let status = body+ .get("status")+ .and_then(Value::as_u64)+ .and_then(|status| u16::try_from(status).ok())+ .unwrap_or(503);+ return ShardResult::http(status, body.to_string());+ }+ if check_only {+ return ShardResult::http(200, json!({ "ok": true }).to_string());+ }+ let query = match serde_json::from_str::<Value>(query_json) {+ Ok(query) => query,+ Err(err) => {+ return ShardResult::http(+ 400,+ json!({+ "error": {+ "type": "parsing_exception",+ "reason": format!("delete_by_query query is not JSON: {err}")+ },+ "status": 400+ })+ .to_string(),+ );+ }+ };+ let parts = DeleteByQueryParts {+ query,+ max_docs,+ scroll_size: scroll_size.max(1) as usize,+ };+ let value = apply_delete_by_query_here(state, index, &parts, max_docs).await;+ let status = value+ .get("status")+ .and_then(Value::as_u64)+ .and_then(|status| u16::try_from(status).ok())+ .unwrap_or(200);+ ShardResult::http(status, value.to_string())+}++/// Run `_delete_by_query` on every primary of `index` and copy the deletes.+///+/// Primaries are visited in node-id order so `max_docs` is one budget for the+/// index. A primary that does not answer stops the call before any document+/// is written. A later primary that fails after an earlier one has already+/// applied is returned as that error; the earlier deletes stay.+/// `[cluster] enabled = false` does not call this.+pub(crate) async fn delete_by_query_on_primaries(+ state: &AppState,+ index: &str,+ parts: &DeleteByQueryParts,+) -> Value {+ let targets = state.engine.shard_search_targets(index);+ if targets.is_empty() {+ // No plan: this node holds the whole index. There is no primary to+ // call, so the run stays here. `replicate_write` does nothing when+ // `primary_of` is `None`.+ return apply_delete_by_query_here(state, index, parts, parts.max_docs).await;+ }+ // Every primary, including this node, is a shard call. The runner builds+ // a search on the caller's stack, and that overflows a debug worker+ // (2 MiB) once the fan-out is above it. `MemoryShardRpc` already runs a+ // handler on an 8 MiB thread, so this node's own shard takes that path+ // too. A primary that is not answering fails here, before any document+ // is written.+ for target in &targets {+ let value = forward_delete_by_query(+ state,+ index,+ target,+ delete_by_query_op(index, parts, parts.max_docs, true),+ )+ .await;+ if value.get("ok").and_then(Value::as_bool) != Some(true) {+ return ensure_error_status(value);+ }+ }+ if parts.max_docs == Some(0) {+ return empty_delete_by_query();+ }++ let mut remaining = parts.max_docs;+ let mut parts_out = Vec::with_capacity(targets.len());+ for target in &targets {+ if remaining == Some(0) {+ break;+ }+ let cap = remaining;+ let value = forward_delete_by_query(+ state,+ index,+ target,+ delete_by_query_op(index, parts, cap, false),+ )+ .await;+ // A shard that failed its own run (flush, script limit, or a primary+ // that stopped answering) has no `deleted` count. Replica nacks keep+ // `deleted` and sit in `failures` until the aggregate below.+ if value.get("deleted").is_none() {+ return ensure_error_status(value);+ }+ if let Some(left) = remaining.as_mut() {+ let used = value.get("total").and_then(Value::as_u64).unwrap_or(0);+ *left = left.saturating_sub(used);+ }+ parts_out.push(value);+ }+ if parts_out.is_empty() {+ return empty_delete_by_query();+ }+ stamp_replica_nack(+ aggregate_by_query_results(parts_out, "deleted"),+ "primary applied the delete but a replica copy missed quorum. The primary has the delete; retry the delete so an in-sync replica can take the copy",+ )}diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex ac3bfe4db..e0a960872 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -31557,6 +31557,80 @@ pub(crate) fn aggregate_by_query_results(mut results: Vec<Value>, count_key: &stValue::Object(out)}+/// Build the cluster fan-out off this handler's state machine.+///+/// `delete_by_query` is already large enough that a debug build's worker+/// stack (2 MiB, the `#[tokio::test]` default) overflows once a second copy+/// of the runner is inlined into the same future. `[cluster] enabled = false`+/// never polls the fan-out, so the future is allocated here and only the+/// pointer is stored on the handler. `inline(never)` keeps that allocation+/// in this frame: inlining it back into `delete_by_query` puts the fan-out+/// on the worker stack again.+#[inline(never)]+fn begin_cluster_delete_by_query(+ state: AppState,+ index_names: Vec<String>,+ parts: crate::cluster_shard::DeleteByQueryParts,+ wait_for_completion: bool,+ handle: crate::state::TaskHandle,+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Response> + Send>> {+ Box::pin(cluster_delete_by_query(+ state,+ index_names,+ parts,+ wait_for_completion,+ handle,+ ))+}++async fn cluster_delete_by_query(+ state: AppState,+ index_names: Vec<String>,+ parts: crate::cluster_shard::DeleteByQueryParts,+ wait_for_completion: bool,+ handle: crate::state::TaskHandle,+) -> Response {+ if !wait_for_completion {+ let task_key = handle.key().to_string();+ let spawned_key = task_key.clone();+ let tasks = state.tasks.clone();+ let names = index_names.clone();+ // The fan-out calls `Engine::get_index` on the detached task, so+ // the request's visibility rule has to come with it. See+ // `xerj_engine::index_guard`.+ let rule = xerj_engine::index_guard::current();+ tokio::spawn(xerj_engine::index_guard::scoped_opt(rule, async move {+ let mut results = Vec::with_capacity(names.len());+ for name in &names {+ results.push(+ crate::cluster_shard::pin_delete_by_query_on_primaries(+ state.clone(),+ name.clone(),+ parts.clone(),+ )+ .await,+ );+ }+ tasks.complete(&spawned_key, aggregate_by_query_results(results, "deleted"));+ drop(handle);+ }));+ return Json(json!({ "task": task_key })).into_response();+ }+ let mut results = Vec::with_capacity(index_names.len());+ for name in &index_names {+ results.push(+ crate::cluster_shard::pin_delete_by_query_on_primaries(+ state.clone(),+ name.clone(),+ parts.clone(),+ )+ .await,+ );+ }+ drop(handle);+ by_query_response(aggregate_by_query_results(results, "deleted"))+}+pub async fn delete_by_query(State(state): State<AppState>,Path(index): Path<String>,@@ -31565,7 +31639,7 @@ pub async fn delete_by_query() -> impl IntoResponse {// Fan the selector out to every member of a multi-index alias (#450), with// any alias filter already ANDed into the query.- let (indices, _index_names, effective_query) =+ let (indices, index_names, effective_query) =match resolve_by_query_targets(&state, &index, body.query.clone()).await {Ok(t) => t,Err(e) => return ApiError::new(e).into_response(),@@ -31596,6 +31670,22 @@ pub async fn delete_by_query(.unwrap_or(true);let handle = state.tasks.register("indices:data/write/delete/byquery");+ if state.engine.has_cluster() {+ let parts = crate::cluster_shard::DeleteByQueryParts {+ query: effective_query,+ max_docs: body.max_docs,+ scroll_size,+ };+ return begin_cluster_delete_by_query(+ state,+ index_names,+ parts,+ wait_for_completion,+ handle,+ )+ .await;+ }+if !wait_for_completion {let task_key = handle.key().to_string();let spawned_key = task_key.clone();@@ -31609,9 +31699,11 @@ pub async fn delete_by_query(// `Engine::get_index` has to follow instead.tokio::spawn(async move {let mut results = Vec::with_capacity(indices.len());- for idx in &indices {- results- .push(run_delete_by_query(idx, &task_query, task_max_docs, scroll_size).await);+ for idx in indices {+ results.push(+ pin_local_delete_by_query(idx, task_query.clone(), task_max_docs, scroll_size)+ .await,+ );}tasks.complete(&spawned_key, aggregate_by_query_results(results, "deleted"));drop(handle);@@ -31620,8 +31712,11 @@ pub async fn delete_by_query(}let mut results = Vec::with_capacity(indices.len());- for idx in &indices {- results.push(run_delete_by_query(idx, &effective_query, body.max_docs, scroll_size).await);+ for idx in indices {+ results.push(+ pin_local_delete_by_query(idx, effective_query.clone(), body.max_docs, scroll_size)+ .await,+ );}drop(handle);by_query_response(aggregate_by_query_results(results, "deleted"))@@ -31727,11 +31822,70 @@ async fn by_query_matches_anything(Ok(results.total.value > 0)}+fn record_deleted(deleted_ids: &Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>, id: &str) {+ let Some(ids) = deleted_ids else {+ return;+ };+ ids.lock()+ .unwrap_or_else(|err| err.into_inner())+ .push(id.to_string());+}+async fn run_delete_by_query(idx: &std::sync::Arc<xerj_engine::Index>,query: &Value,max_docs: Option<u64>,scroll_size: usize,+) -> Value {+ run_delete_by_query_recorded(idx, query, max_docs, scroll_size, None).await+}++/// Box the single-node runner so `delete_by_query`'s future stays a pointer.+///+/// The cluster branch and this branch share one state machine. Leaving the+/// runner inline makes that machine as large as the runner, and allocating+/// it on a 2 MiB debug worker overflows before the cluster branch can+/// forward. `inline(never)` builds the runner in this frame, then the box+/// is what the handler stores. `[cluster] enabled = false` still runs this+/// runner and no other.+#[inline(never)]+fn pin_local_delete_by_query(+ idx: std::sync::Arc<xerj_engine::Index>,+ query: Value,+ max_docs: Option<u64>,+ scroll_size: usize,+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Value> + Send>> {+ Box::pin(async move { run_delete_by_query(&idx, &query, max_docs, scroll_size).await })+}++/// Own the runner's arguments, then box it.+///+/// A primary's `_delete_by_query` is reached from the fan-out. Inlining the+/// runner into that future puts the search state machine on the caller's+/// stack and overflows a debug worker. The box is the same split+/// [`run_update_by_query_recorded_boxed`] uses.+#[inline(never)]+pub(crate) fn run_delete_by_query_recorded_boxed(+ idx: std::sync::Arc<xerj_engine::Index>,+ query: Value,+ max_docs: Option<u64>,+ scroll_size: usize,+ deleted_ids: Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>,+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Value> + Send>> {+ Box::pin(async move {+ run_delete_by_query_recorded(&idx, &query, max_docs, scroll_size, deleted_ids).await+ })+}++/// Same run as `run_delete_by_query`. When `deleted_ids` is set, every id+/// this run removed is appended, including ids removed before a later page+/// refuses the rest. The cluster primary copies that list onto its replicas.+pub(crate) async fn run_delete_by_query_recorded(+ idx: &std::sync::Arc<xerj_engine::Index>,+ query: &Value,+ max_docs: Option<u64>,+ scroll_size: usize,+ deleted_ids: Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>,) -> Value {let started = Instant::now();@@ -31809,7 +31963,10 @@ async fn run_delete_by_query(for id in chunk {processed += 1;match idx.delete_document(id).await {- Ok(_) => deleted += 1,+ Ok(_) => {+ deleted += 1;+ record_deleted(&deleted_ids, id);+ }Err(e) => {failures.push(json!({"id": id,@@ -31914,7 +32071,10 @@ async fn run_delete_by_query(for hit in results.hits {processed += 1;match idx.delete_document(&hit.id).await {- Ok(_) => deleted += 1,+ Ok(_) => {+ deleted += 1;+ record_deleted(&deleted_ids, &hit.id);+ }Err(e) => {failures.push(json!({"id": hit.id,@@ -39039,7 +39199,11 @@ fn spawn_watch_evaluator(if !alive {break;}- cursor = evaluate_watch_pass(&state, &id, &watch, cursor).await;+ // The pass owns a search state machine. Polling it inline on+ // this task shares the caller's stack with the `#[tokio::test]`+ // runtime (2 MiB). The box keeps that machine off the stack;+ // the cursor and the retry rules are unchanged.+ cursor = Box::pin(evaluate_watch_pass(&state, &id, &watch, cursor)).await;}},));diff --git a/engine/crates/xerj-api/tests/autoindex_html_inline_script.rs b/engine/crates/xerj-api/tests/autoindex_html_inline_script.rsnew file mode 100644index 000000000..8b2555240--- /dev/null+++ b/engine/crates/xerj-api/tests/autoindex_html_inline_script.rs@@ -0,0 +1,373 @@+//! Issue #2356: a JavaScript function defined in an inline `<script>` of an+//! HTML file must be findable by `xerj def` and `xerj search` after+//! `xerj autoindex`, at the line it occupies in that file.+//!+//! Python in the same folder is the control: it already indexed. The page's+//! visible text stays searchable, and script source stays out of that body.++use std::path::Path;+use std::time::Duration;++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;+use xerj_autoindex::cli::IndexCfg;++const PAGE_HTML: &str = r#"<!DOCTYPE html>+<html>+<head><title>Drawer</title></head>+<body>+<p>The drawer holds the menu.</p>+<script>+function openDrawer(id) {+ document.getElementById(id).hidden = false;+}+</script>+<script type="module">+export function openPanel() {+ return 1;+}+</script>+<script type="application/ld+json">+function ignoredJson() {}+</script>+<script src="app.js">+function fromSrc() {}+</script>+<p>After the scripts.</p>+</body>+</html>+"#;++const PAGE_HTM: &str = r#"<!DOCTYPE html>+<html>+<body>+<p>Secondary page.</p>+<script>+function closeDrawer() {+ return false;+}+</script>+</body>+</html>+"#;++const NOTES_PY: &str = "\+def load_notes():\n\+ return \"notes\"\n\+";++fn source_line(text: &str, needle: &str) -> u64 {+ text.lines()+ .position(|line| line.contains(needle))+ .map(|i| (i + 1) as u64)+ .unwrap_or_else(|| panic!("{needle} missing from fixture"))+}++async fn app() -> axum::Router {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.auth.enabled = false;+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ // Leak the temp dir for the process lifetime: the router keeps the data+ // dir path, and dropping `TempDir` at the end of this function would+ // delete it out from under the engine.+ std::mem::forget(dir);+ xerj_api::router::build_es_compat_router(state)+}++async fn call(app: &axum::Router, method: &str, path: &str, body: Value) -> (StatusCode, Value) {+ let mut req = Request::builder().method(method).uri(path);+ let body = if body.is_null() {+ Body::empty()+ } else {+ req = req.header("content-type", "application/json");+ Body::from(body.to_string())+ };+ let response = app.clone().oneshot(req.body(body).unwrap()).await.unwrap();+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .unwrap();+ (+ status,+ serde_json::from_slice(&bytes).unwrap_or(Value::Null),+ )+}++fn index_cfg(root: &Path, state_dir: &Path, url: &str, prefix: &str) -> IndexCfg {+ IndexCfg {+ root: root.to_owned(),+ endpoint_url: None,+ stub_globs: Vec::new(),+ url: url.to_owned(),+ api_key: None,+ api_key_file: None,+ workers: 1,+ scan_workers: 1,+ pdf_workers: 1,+ resource_notes: Vec::new(),+ xerj_url_note: None,+ pdf_timeout_secs: 30,+ bulk_mb: 1,+ bulk_timeout_secs: 30,+ snapshot_max_bytes: 64 << 30,+ prefix: prefix.to_owned(),+ replace_prefix: false,+ prefix_note: None,+ state_dir: Some(state_dir.to_owned()),+ fresh: false,+ follow_symlinks: false,+ follow_symlinks_outside_root: false,+ ignore: xerj_autoindex::ignore_rules::IgnoreOptions::default(),+ max_file_bytes: 1 << 30,+ sample: 50,+ no_semantic: false,+ code_analyzer: Default::default(),+ brain: Some(format!("brain-{prefix}")),+ no_graph: true,+ max_minutes: 0,+ approve: None,+ max_index_bytes: None,+ budget_plan: None,+ drop_indexed: false,+ dry_run: false,+ walk_only: false,+ json: false,+ quiet: true,+ progress: xerj_autoindex::progress::ProgressMode::None,+ progress_interval: None,+ watch: false,+ label: None,+ debounce: Duration::from_millis(0),+ exclude_rels: Vec::new(),+ skip_corpus_data: false,+ }+}++async fn run_autoindex(root: &Path, state_dir: &Path, url: &str, prefix: &str) -> i32 {+ let root = root.to_owned();+ let state_dir = state_dir.to_owned();+ let url = url.to_owned();+ let prefix = prefix.to_owned();+ tokio::task::spawn_blocking(move || {+ let cfg = index_cfg(&root, &state_dir, &url, &prefix);+ match xerj_autoindex::run_index_report(cfg) {+ Ok((code, _)) => code,+ Err(error) => panic!("autoindex {prefix} failed: {error:#}"),+ }+ })+ .await+ .expect("autoindex task")+}++async fn prefix_docs(app: &axum::Router, prefix: &str) -> Vec<Value> {+ let (status, body) = call(+ app,+ "POST",+ &format!("/{prefix}-*/_search"),+ json!({+ "size": 200,+ "query": {"match_all": {}},+ "sort": [{"_id": "asc"}]+ }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "search {prefix}-*: {body}");+ body["hits"]["hits"]+ .as_array()+ .unwrap_or_else(|| panic!("no hits array: {body}"))+ .iter()+ .map(|hit| hit["_source"].clone())+ .collect()+}++fn assert_symbol(docs: &[Value], name: &str, file: &str, line: u64) {+ let hit = docs.iter().find(|doc| {+ doc.get("name").and_then(Value::as_str) == Some(name)+ && doc+ .get("ax_path")+ .and_then(Value::as_str)+ .is_some_and(|path| path.ends_with(file))+ });+ let Some(doc) = hit else {+ panic!(+ "{file} has no symbol {name}: {}",+ serde_json::to_string(docs).unwrap()+ );+ };+ assert_eq!(doc["line"], json!(line), "{name} line: {doc}");+ assert_eq!(doc["kind"], json!("function"), "{name} kind: {doc}");+ assert_eq!(+ doc["language"],+ json!("javascript"),+ "{name} language: {doc}"+ );+ let code = doc["code"].as_str().unwrap_or("");+ assert!(code.contains(name), "{name} declaration: {code}");+ assert_eq!(+ doc["ax_locator"],+ json!(format!("code:{line}:{name}")),+ "{name} locator: {doc}"+ );+}++#[test]+fn autoindex_finds_inline_script_functions_by_def_and_search() {+ // Debug autoindex builds a future that does not fit tokio's 2 MiB+ // default worker. The server runtime is 4 MiB (`RT_THREAD_STACK_SIZE`);+ // this test uses the same floor so a debug `cargo test` matches it.+ let rt = tokio::runtime::Builder::new_multi_thread()+ .worker_threads(2)+ .thread_stack_size(4 * 1024 * 1024)+ .enable_all()+ .build()+ .expect("runtime");+ rt.block_on(autoindex_finds_inline_script_functions_by_def_and_search_inner());+}++async fn autoindex_finds_inline_script_functions_by_def_and_search_inner() {+ let app = app().await;+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ let served = app.clone();+ tokio::spawn(async move {+ axum::serve(listener, served).await.unwrap();+ });+ let url = format!("http://{addr}");++ let root = tempfile::tempdir().unwrap();+ let state = tempfile::tempdir().unwrap();+ std::fs::write(root.path().join("page.html"), PAGE_HTML).unwrap();+ std::fs::write(root.path().join("drawer.htm"), PAGE_HTM).unwrap();+ std::fs::write(root.path().join("notes.py"), NOTES_PY).unwrap();++ let prefix = "htmljs";+ let code = run_autoindex(root.path(), state.path(), &url, prefix).await;+ assert_eq!(code, 0, "autoindex exit {code}");++ let docs = prefix_docs(&app, prefix).await;+ let open_line = source_line(PAGE_HTML, "function openDrawer");+ let panel_line = source_line(PAGE_HTML, "export function openPanel");+ let close_line = source_line(PAGE_HTM, "function closeDrawer");+ assert_symbol(&docs, "openDrawer", "page.html", open_line);+ assert_symbol(&docs, "openPanel", "page.html", panel_line);+ assert_symbol(&docs, "closeDrawer", "drawer.htm", close_line);++ let rendered = serde_json::to_string(&docs).unwrap();+ for absent in ["ignoredJson", "fromSrc"] {+ assert!(+ !rendered.contains(absent),+ "{absent} was indexed from a non-JS script: {rendered}"+ );+ }+ let prose = docs.iter().find(|doc| {+ doc.get("ax_path").and_then(Value::as_str) == Some("page.html")+ && doc+ .get("body")+ .and_then(Value::as_str)+ .is_some_and(|body| body.contains("The drawer holds the menu."))+ });+ let Some(prose) = prose else {+ panic!("page prose was not indexed: {rendered}");+ };+ let prose_body = prose["body"].as_str().unwrap();+ assert!(prose_body.contains("After the scripts."), "{prose_body}");+ assert!(+ !prose_body.contains("openDrawer") && !prose_body.contains("getElementById"),+ "script source was copied into the page body: {prose_body}"+ );++ let (status, mapping) = call(&app, "GET", &format!("/{prefix}-*/_mapping"), Value::Null).await;+ assert_eq!(status, StatusCode::OK, "mapping: {mapping}");+ let mapped = xerj_common::code_query::mapped_property_names(&mapping);+ let numeric = mapped+ .as_ref()+ .map(|_| xerj_common::code_query::numeric_property_names(&mapping));++ let def_body =+ xerj_autoindex::def::definition_query("openDrawer", None, None, 5, Some(&mapping));+ let (status, mut def_resp) =+ call(&app, "POST", &format!("/{prefix}-*/_search"), def_body).await;+ assert_eq!(status, StatusCode::OK, "def query: {def_resp}");+ let def_hits =+ xerj_autoindex::def::finalize_definition_response(&mut def_resp, "openDrawer", 5);+ assert!(+ !def_hits.is_empty(),+ "xerj def openDrawer returned nothing: {def_resp}"+ );+ let top = &def_hits[0];+ let top_src = &top["_source"];+ assert_eq!(top_src["name"], json!("openDrawer"), "def top hit: {top}");+ assert_eq!(top_src["ax_path"], json!("page.html"), "def top hit: {top}");+ assert_eq!(top_src["line"], json!(open_line), "def top hit: {top}");+ assert!(+ top_src["code"]+ .as_str()+ .is_some_and(|code| code.contains("openDrawer")),+ "def declaration: {top}"+ );++ let search_body = xerj_common::code_query::search_query_body_for_mapping(+ "openDrawer",+ 5,+ mapped.as_ref(),+ numeric.as_ref(),+ );+ let (status, search_resp) =+ call(&app, "POST", &format!("/{prefix}-*/_search"), search_body).await;+ assert_eq!(status, StatusCode::OK, "search query: {search_resp}");+ let search_hits = search_resp["hits"]["hits"]+ .as_array()+ .unwrap_or_else(|| panic!("no search hits: {search_resp}"));+ // `xerj search` does not request `name` in `_source`. The hit is the+ // declaration: title, kind, code, and the file line.+ assert!(+ search_hits.iter().any(|hit| {+ let src = &hit["_source"];+ src["title"] == json!("openDrawer")+ && src["kind"] == json!("function")+ && src["ax_path"] == json!("page.html")+ && src["line"] == json!(open_line)+ && src["code"]+ .as_str()+ .is_some_and(|code| code.contains("function openDrawer"))+ }),+ "xerj search did not return openDrawer at page.html:{open_line}: {search_resp}"+ );++ let prose_body = xerj_common::code_query::search_query_body_for_mapping(+ "drawer holds the menu",+ 5,+ mapped.as_ref(),+ numeric.as_ref(),+ );+ let (status, prose_resp) =+ call(&app, "POST", &format!("/{prefix}-*/_search"), prose_body).await;+ assert_eq!(status, StatusCode::OK, "prose search: {prose_resp}");+ let prose_hits = prose_resp["hits"]["hits"].as_array().unwrap();+ assert!(+ prose_hits+ .iter()+ .any(|hit| hit["_source"]["ax_path"] == json!("page.html")),+ "page text was not searchable: {prose_resp}"+ );++ let py_body =+ xerj_autoindex::def::definition_query("load_notes", None, None, 5, Some(&mapping));+ let (status, mut py_resp) = call(&app, "POST", &format!("/{prefix}-*/_search"), py_body).await;+ assert_eq!(status, StatusCode::OK, "python def: {py_resp}");+ let py_hits = xerj_autoindex::def::finalize_definition_response(&mut py_resp, "load_notes", 5);+ assert!(+ py_hits.iter().any(|hit| {+ hit["_source"]["name"] == json!("load_notes")+ && hit["_source"]["ax_path"] == json!("notes.py")+ }),+ "python def regressed: {py_resp}"+ );+}diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rsindex c74588a77..052ea4e0d 100644--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs@@ -446,6 +446,88 @@ async fn cluster_disabled_create_stays_on_the_node_that_received_it() {);}+/// `_delete_by_query` with no cluster handle stays on the node that accepted+/// it. The runner does not fit a 2 MiB debug worker once it is under the+/// router, so this test sizes its own stacks. The HTTP result is the+/// single-node runner: one document deleted, the other left in place, and+/// no replica store.+#[test]+fn cluster_disabled_delete_by_query_stays_on_the_node_that_received_it() {+ std::thread::Builder::new()+ .stack_size(8 * 1024 * 1024)+ .spawn(|| {+ tokio::runtime::Builder::new_multi_thread()+ .worker_threads(2)+ .thread_stack_size(8 * 1024 * 1024)+ .enable_all()+ .build()+ .expect("runtime")+ .block_on(+ cluster_disabled_delete_by_query_stays_on_the_node_that_received_it_inner(),+ )+ })+ .expect("spawn")+ .join()+ .expect("test thread panicked");+}++async fn cluster_disabled_delete_by_query_stays_on_the_node_that_received_it_inner() {+ async fn lone() -> (Router, tempfile::TempDir) {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ assert!(!engine.has_cluster());+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ (xerj_api::router::build_es_compat_router(state), dir)+ }++ let (a, _dir_a) = lone().await;+ let (b, _dir_b) = lone().await;+ let (status, body) = put_json(&a, "/local", json!({})).await;+ assert!(status.is_success(), "{status} {body}");+ let (status, body) = put_json(&a, "/local/_doc/1", json!({ "title": "here" })).await;+ assert!(status.is_success(), "{status} {body}");+ let (status, body) = put_json(&a, "/local/_doc/b1", json!({ "title": "bulk" })).await;+ assert!(status.is_success(), "{status} {body}");++ let (status, body) = post_json(+ &a,+ "/local/_delete_by_query",+ json!({ "query": { "ids": { "values": ["b1"] } } }),+ )+ .await;+ assert!(+ status.is_success(),+ "single-node delete_by_query: {status} {body}"+ );+ assert_eq!(+ body.get("deleted").and_then(|v| v.as_u64()),+ Some(1),+ "{body}"+ );+ let (status, _) = get(&a, "/local/_doc/b1").await;+ assert_eq!(status, StatusCode::NOT_FOUND);+ let (status, got) = get(&a, "/local/_doc/1").await;+ assert_eq!(status, StatusCode::OK, "{got}");+ assert_eq!(+ got.pointer("/_source/title").and_then(|v| v.as_str()),+ Some("here"),+ "an ids query on one node still leaves the other document: {got}"+ );+ let (status, _) = get(&b, "/local/_doc/1").await;+ assert_eq!(status, StatusCode::NOT_FOUND);+ let (status, _) = get(&b, "/local/_doc/b1").await;+ assert_eq!(status, StatusCode::NOT_FOUND);+ let (status, _) = get(&a, "/.xerj_replica_local_0/_doc/b1").await;+ assert_eq!(+ status,+ StatusCode::NOT_FOUND,+ "single-node delete_by_query does not open a replica store"+ );+}+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]async fn an_acknowledged_write_is_on_the_replica_and_readable_after_the_primary_stops() {let members = ring(true).await;@@ -2699,3 +2781,346 @@ async fn update_by_query_from_a_non_owner_updates_every_primary_and_replica() {let _ = member.shutdown.send(true);}}++fn restore_shard_handler(member: &Member) {+ member+ .handle+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(+ member.state.clone(),+ )));+}++async fn index_titled(app: &Router, id: &str, title: &str) {+ let (status, body) = put_json(+ app,+ &format!("/orders/_doc/{id}"),+ json!({ "title": title }),+ )+ .await;+ assert!(status.is_success(), "index {id}: {status} {body}");+}++/// `_delete_by_query` accepted by a node that owns neither document deletes+/// on every primary and copies each delete onto the in-sync replica.+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]+async fn delete_by_query_from_a_non_owner_deletes_on_every_primary_and_replica() {+ let members = ring(true).await;+ let _leader = wait_leader(&members, &[false, false, false]).await;++ let (status, body) = put_json(+ &members[0].app,+ "/orders",+ json!({+ "settings": { "index": { "number_of_replicas": 1 } },+ "mappings": { "properties": { "title": { "type": "text" } } }+ }),+ )+ .await;+ assert!(status.is_success(), "create: {status} {body}");+ for member in &members {+ wait_status(&member.app, "/orders", StatusCode::OK).await;+ }++ let mut placed: Vec<(String, usize, usize, u32)> = Vec::new();+ for n in 0..128 {+ let id = format!("d{n}");+ let Some(primary) = members+ .iter()+ .position(|member| member.engine.owns_shard("orders", &id))+ else {+ continue;+ };+ let (shard, _) = members[primary]+ .engine+ .primary_of("orders", &id)+ .expect("placement");+ let replicas = members[primary].engine.replica_nodes("orders", shard);+ if replicas.len() != 1 {+ continue;+ }+ let Some(replica) = members+ .iter()+ .position(|member| member.handle.node_id() == replicas[0])+ else {+ continue;+ };+ if placed.iter().any(|(_, owner, _, _)| *owner == primary) {+ continue;+ }+ placed.push((id, primary, replica, shard));+ if placed.len() == 2 {+ break;+ }+ }+ assert_eq!(placed.len(), 2, "two primaries: {placed:?}");+ let (id_a, primary_a, replica_a, shard_a) = placed[0].clone();+ let (id_b, primary_b, replica_b, shard_b) = placed[1].clone();+ assert_ne!(primary_a, primary_b);+ assert_ne!(replica_a, replica_b);+ let writer = (0..3)+ .find(|n| *n != primary_a && *n != primary_b)+ .expect("a node that owns neither id");++ index_titled(&members[writer].app, &id_a, "alpha").await;+ index_titled(&members[writer].app, &id_b, "beta").await;+ assert!(+ user_source(&members[writer], &id_a).await.is_none(),+ "the receiver stored {id_a}"+ );++ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query",+ json!({ "query": { "match_all": {} } }),+ )+ .await;+ assert!(+ status.is_success(),+ "delete_by_query from non-owner: {status} {body}"+ );+ assert_eq!(+ body.get("deleted").and_then(|v| v.as_u64()),+ Some(2),+ "{body}"+ );+ assert_eq!(+ body.get("failures")+ .and_then(|v| v.as_array())+ .map(|items| items.len()),+ Some(0),+ "{body}"+ );+ assert!(user_source(&members[primary_a], &id_a).await.is_none());+ assert!(user_source(&members[primary_b], &id_b).await.is_none());+ assert!(replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .is_none());+ assert!(replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .is_none());+ assert!(user_source(&members[writer], &id_a).await.is_none());+ assert!(user_source(&members[writer], &id_b).await.is_none());+ let (status, got) = get(&members[writer].app, &format!("/orders/_doc/{id_a}")).await;+ assert_eq!(+ status,+ StatusCode::NOT_FOUND,+ "GET after delete_by_query: {status} {got}"+ );++ members[primary_a].handle.clear_shard_handler();+ let (status, got) = get(&members[writer].app, &format!("/orders/_doc/{id_a}")).await;+ assert_eq!(+ status,+ StatusCode::NOT_FOUND,+ "a primary that is down still does not serve a copy the delete removed: {status} {got}"+ );+ restore_shard_handler(&members[primary_a]);++ index_titled(&members[writer].app, &id_a, "alpha").await;+ index_titled(&members[writer].app, &id_b, "beta").await;+ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query",+ json!({ "query": { "ids": { "values": [id_a] } } }),+ )+ .await;+ assert!(status.is_success(), "ids delete_by_query: {status} {body}");+ assert_eq!(+ body.get("deleted").and_then(|v| v.as_u64()),+ Some(1),+ "{body}"+ );+ assert!(user_source(&members[primary_a], &id_a).await.is_none());+ assert!(replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .is_none());+ assert_eq!(+ user_source(&members[primary_b], &id_b)+ .await+ .expect("primary b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("beta"),+ "the other shard stays"+ );+ assert_eq!(+ replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .expect("replica b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("beta")+ );++ index_titled(&members[writer].app, &id_a, "alpha").await;+ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query",+ json!({ "query": { "match": { "title": "alpha" } } }),+ )+ .await;+ assert!(+ status.is_success(),+ "match delete_by_query: {status} {body}"+ );+ assert_eq!(+ body.get("deleted").and_then(|v| v.as_u64()),+ Some(1),+ "a non-ids query still deletes on the owning primary: {body}"+ );+ assert!(user_source(&members[primary_a], &id_a).await.is_none());+ assert!(replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .is_none());+ assert_eq!(+ user_source(&members[primary_b], &id_b)+ .await+ .expect("primary b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("beta")+ );+ assert_eq!(+ replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .expect("replica b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("beta")+ );++ index_titled(&members[writer].app, &id_a, "alpha").await;+ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query",+ json!({ "query": { "match_all": {} }, "max_docs": 1 }),+ )+ .await;+ assert!(+ status.is_success(),+ "max_docs delete_by_query: {status} {body}"+ );+ assert_eq!(+ body.get("deleted").and_then(|v| v.as_u64()),+ Some(1),+ "max_docs is one budget for the index, not one per shard: {body}"+ );+ let left_a = user_source(&members[primary_a], &id_a).await.is_some();+ let left_b = user_source(&members[primary_b], &id_b).await.is_some();+ assert_eq!(+ left_a as u8 + left_b as u8,+ 1,+ "exactly one primary still has its document"+ );+ assert_eq!(+ replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .is_some(),+ left_a,+ "replica a matches the primary"+ );+ assert_eq!(+ replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .is_some(),+ left_b,+ "replica b matches the primary"+ );++ index_titled(&members[writer].app, &id_a, "async").await;+ index_titled(&members[writer].app, &id_b, "async").await;+ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query?wait_for_completion=false",+ json!({ "query": { "match_all": {} } }),+ )+ .await;+ assert!(+ status.is_success(),+ "async delete_by_query: {status} {body}"+ );+ let task = body+ .get("task")+ .and_then(|v| v.as_str())+ .unwrap_or_else(|| panic!("task id: {body}"))+ .to_string();+ let deadline = Instant::now() + Duration::from_secs(5);+ let completed = loop {+ let (status, got) = get(&members[writer].app, &format!("/_tasks/{task}")).await;+ if status.is_success() && got.get("completed").and_then(|v| v.as_bool()) == Some(true) {+ break got;+ }+ if Instant::now() > deadline {+ panic!("task {task} did not finish: {status} {got}");+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ };+ assert_eq!(+ completed+ .pointer("/response/deleted")+ .and_then(|v| v.as_u64()),+ Some(2),+ "{completed}"+ );+ assert!(user_source(&members[primary_a], &id_a).await.is_none());+ assert!(user_source(&members[primary_b], &id_b).await.is_none());+ assert!(replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .is_none());+ assert!(replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .is_none());++ index_titled(&members[writer].app, &id_a, "stay").await;+ index_titled(&members[writer].app, &id_b, "stay").await;+ members[primary_a].handle.clear_shard_handler();+ let (status, body) = post_json(+ &members[writer].app,+ "/orders/_delete_by_query",+ json!({ "query": { "match_all": {} } }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "a primary that does not answer refuses before writing: {body}"+ );+ assert_eq!(+ user_source(&members[primary_a], &id_a)+ .await+ .expect("primary a")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("stay"),+ "the down primary was not deleted"+ );+ assert_eq!(+ replica_source(&members[replica_a], shard_a, &id_a)+ .await+ .expect("replica a")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("stay")+ );+ assert_eq!(+ user_source(&members[primary_b], &id_b)+ .await+ .expect("primary b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("stay"),+ "a later primary is not deleted once an earlier one cannot be checked"+ );+ assert_eq!(+ replica_source(&members[replica_b], shard_b, &id_b)+ .await+ .expect("replica b")+ .get("title")+ .and_then(|v| v.as_str()),+ Some("stay")+ );++ for member in &members {+ let _ = member.shutdown.send(true);+ }+}diff --git a/engine/crates/xerj-autoindex/src/budget.rs b/engine/crates/xerj-autoindex/src/budget.rsindex 1da039379..e841357b8 100644--- a/engine/crates/xerj-autoindex/src/budget.rs+++ b/engine/crates/xerj-autoindex/src/budget.rs@@ -1827,6 +1827,20 @@ mod tests {#[test]fn kinds_follow_the_extension_before_sniff() {assert_eq!(classify("a.rs").0, Kind::Code);+ // #2357: brackets are directory names. A Windows rel that still+ // contains `\` (the walk rewrites those to `/` before this runs)+ // keeps the `.tsx` extension.+ assert_eq!(+ classify("app/[locale]/booking/[token]/temple-selection/TempleSelectionWrapper.tsx").0,+ Kind::Code+ );+ assert_eq!(+ classify(+ "app\\[locale]\\booking\\[token]\\temple-selection\\TempleSelectionWrapper.tsx"+ )+ .0,+ Kind::Code+ );assert_eq!(classify("a.md").0, Kind::Prose);assert_eq!(classify("a.pdf").0, Kind::Pdf);assert_eq!(classify("a.csv").0, Kind::Data);diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rsindex c4d10e2a3..709a0769e 100644--- a/engine/crates/xerj-autoindex/src/catalog.rs+++ b/engine/crates/xerj-autoindex/src/catalog.rs@@ -927,6 +927,35 @@ pub fn render_map_page(}s.push('\n');}+ // #2370: a family that was entirely or mostly junked is named here, from+ // the run document, so the map does not look like a complete corpus when+ // a file type was dropped. Absent on a run that did not trip the threshold,+ // which keeps this section off the maps those runs already printed.+ if let Some(warnings) = run+ .and_then(|r| r.get("family_junk_warnings"))+ .and_then(Value::as_array)+ .filter(|warnings| !warnings.is_empty())+ {+ s.push_str("## Families mostly or entirely junked — NOT indexed\n\n");+ s.push_str(+ "Every file of a family, or a strict majority of its files, was junked. \+ These types are missing or barely present in the dataset table. \+ The top reason is the most common junk reason for that family.\n\n",+ );+ for warning in warnings {+ let g = |k: &str| warning.get(k).map(pretty_val).unwrap_or_default();+ s.push_str(&format!(+ "- **{}** — {} of {} files ({}); top reason ({} files): {}\n",+ g("family"),+ g("junk_files"),+ g("files"),+ g("extent"),+ g("top_reason_files"),+ g("top_reason"),+ ));+ }+ s.push('\n');+ }s.push_str(&format!("## Datasets ({})\n\n", datasets.len()));// `slug` is the argument to `--dataset`. The index name is// `{prefix}-{slug}`, and a hyphenated prefix makes the slug impossible@@ -1440,6 +1469,49 @@ mod duplicate_tests {rendered_legacy.contains("0 junk records, wall 1.4s\n\n"),"legacy wall_seconds must still render: {rendered_legacy}");+ // A run document without family warnings keeps the map it had.+ assert!(+ !rendered.contains("Families mostly or entirely junked"),+ "{rendered}"+ );+ }++ #[test]+ fn the_map_names_a_family_that_was_entirely_junked() {+ let run = json!({+ "doc_kind": "run",+ "run_id": "gen-1",+ "root": "/corpus",+ "files_total": 6,+ "unique_content_files": 2,+ "duplicate_files": 0,+ "records_total": 2,+ "junk_records_total": 0,+ "family_junk_warnings": [{+ "family": "pdf",+ "files": 4,+ "junk_files": 4,+ "indexed_files": 0,+ "extent": "all",+ "top_reason": "could not start the isolated PDF parser",+ "top_reason_files": 4+ }]+ });+ let rendered = render_map(Some(&run), &[], &[], &[], &[], 0);+ assert!(+ rendered.contains("## Families mostly or entirely junked — NOT indexed"),+ "{rendered}"+ );+ assert!(+ rendered.contains(+ "**pdf** — 4 of 4 files (all); top reason (4 files): could not start the isolated PDF parser"+ ),+ "{rendered}"+ );+ // The section sits above the dataset table.+ let warning_at = rendered.find("Families mostly or entirely junked").unwrap();+ let datasets_at = rendered.find("## Datasets").unwrap();+ assert!(warning_at < datasets_at, "{rendered}");}/// The summary used to say "0 junk records" on a map whose later sectiondiff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 1a70f5628..63daf2306 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -308,7 +308,12 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20xerj autoindex s3://<bucket>/<prefix> [OPTIONS]\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20discover + index objects in a bucket\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20(r2:// too; see OBJECT STORAGE)\n\- \x20\x20\x20\x20xerj autoindex map [OPTIONS] print the discovered data map\n\+ \x20\x20\x20\x20xerj autoindex map [OPTIONS] print the data map of what is already indexed\n\+ \x20\x20\x20\x20xerj autoindex map <folder> [OPTIONS] preview a folder that is not indexed yet.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20Same discovery as `xerj autoindex <folder>\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20--dry-run`: walk, sniff, infer, print the plan.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20Nothing is written to an index. With no\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20folder, `map` reads the catalog on --url\n\\x20\x20\x20\x20xerj autoindex remove --prefix <P> [--root <DIR>] [--yes]\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20list one folder's indices and catalog rows,\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20then delete them. Nothing is deleted without\n\@@ -453,7 +458,8 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20--drop-indexed delete already-indexed files a reduced plan leaves\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20out. Without this flag that re-run exits 4 and lists\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20them. --plan and --yes do not agree to the deletion.\n\- \x20\x20\x20\x20--dry-run walk+sniff+infer, print the plan, index nothing\n\+ \x20\x20\x20\x20--dry-run walk+sniff+infer, print the plan, index nothing.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`autoindex map <folder>` is this same preview\n\\x20\x20\x20\x20--walk-only stop after the walk. Prints each top-level folder's\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20file count and size, what the ignore rules would drop\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20and which rule, and the totals. Stops before hashing:\n\@@ -484,9 +490,9 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20the JSON decision request on stdout and exits 4 instead,\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20exactly like an agent-driven run. See ESTIMATE + DECISION\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20GATE.\n\- \x20\x20\x20\x20--dataset <SLUG> (map) show a single dataset\n\- \x20\x20\x20\x20--offset <N> (map) skip the first N dataset detail sections\n\- \x20\x20\x20\x20--limit <N> (map) how many dataset detail sections to print\n\+ \x20\x20\x20\x20--dataset <SLUG> (map, no folder) show a single catalog dataset\n\+ \x20\x20\x20\x20--offset <N> (map, no folder) skip the first N dataset detail sections\n\+ \x20\x20\x20\x20--limit <N> (map, no folder) how many dataset detail sections to print\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20(default: all of them when there are few, otherwise\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20the first page — the summary table always lists every\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20dataset)\n\@@ -783,8 +789,9 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20two apart, and rerun the same command once the reported server\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20condition clears;\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20or stale-resume-plan — files on disk appeared after the\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20resume plan was frozen. They are not counted as junk;\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`skipped_appeared` names them. Re-run with --fresh to\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20resume plan was frozen. They are not counted as junk.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`skipped_appeared` is the count; the summary names the\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20first few paths and says to re-run with --fresh to\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20rebuild the plan in place and include them. Read `reason`\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20to tell this apart from completed-with-junk;\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20or files-removed — a graph-path rerun deleted documents,\n\@@ -1379,6 +1386,16 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {xerj_common::feedback::DISABLE_FLAG => {}"--help" | "-h" => return Ok(Cmd::Help),"map" if sub.is_none() && folder.is_none() => sub = Some("map".into()),+ // #2368: `autoindex map <folder>` previews that folder. The folder+ // is the same positional the index command takes; it is accepted+ // only after `map` so `xerj autoindex ./map` still indexes a+ // directory of that name.+ other+ if !other.starts_with('-') && sub.as_deref() == Some("map") && folder.is_none() =>+ {+ root_raw = Some(other.to_string());+ folder = Some(PathBuf::from(other));+ }"status" if sub.is_none() && folder.is_none() => sub = Some("status".into()),"remove" if sub.is_none() && folder.is_none() => sub = Some("remove".into()),other if !other.starts_with('-') && folder.is_none() && sub.is_none() => {@@ -1406,6 +1423,19 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {{first} --prefix <name-a>` then `xerj autoindex {other} --prefix <name-b>`"));}+ other+ if !other.starts_with('-') && sub.as_deref() == Some("map") && folder.is_some() =>+ {+ let first = folder+ .as_ref()+ .map(|p| p.display().to_string())+ .unwrap_or_default();+ return Err(format!(+ "`autoindex map` takes at most one folder, got `{first}` and `{other}`. \+ `xerj autoindex map` with no folder prints the catalog of what is already \+ indexed. `xerj autoindex map <folder>` previews one folder and indexes nothing"+ ));+ }other => return Err(format!("unknown argument: {other}")),}}@@ -1445,13 +1475,11 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {// `--no-ignore` already removes the built-in defaults, so there is nothing// left for `--no-default-ignores` to do — say so instead of taking a flag// that changes nothing.- // Only meaningful for a run that walks a folder. On `map`/`status` neither- // flag does anything at all, and the arm below says so — a message about- // one flag subsuming the other would imply the pair is otherwise honoured.- let sub_walks_a_folder = !matches!(- sub.as_deref(),- Some("map") | Some("status") | Some("remove")- );+ // Only meaningful for a run that walks a folder. A bare `map` (no folder),+ // `status`, and `remove` never walk, and the arm below says so — a message+ // about one flag subsuming the other would imply the pair is otherwise+ // honoured. `map <folder>` does walk: it is the `--dry-run` preview.+ let sub_walks_a_folder = folder.is_some();if no_ignore && no_default_ignores && sub_walks_a_folder {return Err("--no-ignore already turns off the built-in default rules that --no-default-ignores \@@ -1507,6 +1535,44 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {if walk_only {dry_run = true;}+ // #2368: `autoindex map <folder>` previews an unindexed folder. It is the+ // same discovery `--dry-run` runs (walk, hash, sniff, infer, print the+ // plan) and it must not index. Forcing `dry_run` here arms every+ // index-nothing guard. A bare `map` still reads the catalog.+ let map_preview = sub.as_deref() == Some("map") && folder.is_some();+ if map_preview {+ if watch {+ return Err(+ "--watch reindexes a folder as it changes. `autoindex map <folder>` previews \+ that folder and indexes nothing. Drop --watch"+ .into(),+ );+ }+ if label.is_some() {+ return Err(+ "--label labels documents at ingest. `autoindex map <folder>` previews that \+ folder and indexes nothing, so there is nowhere for the labels to go. Drop \+ --label"+ .into(),+ );+ }+ if approve == Some(Approval::Cancel) {+ return Err(+ "`autoindex map <folder>` already indexes nothing, so --approve cancel has \+ nothing to cancel. Drop it"+ .into(),+ );+ }+ if dataset.is_some() || detail_offset.is_some() || detail_limit.is_some() {+ return Err(+ "--dataset, --offset and --limit page the catalog that `autoindex map` reads \+ when no folder is given. `autoindex map <folder>` previews a folder and does \+ not read that catalog. Drop them, or run `xerj autoindex map` with no folder"+ .into(),+ );+ }+ dry_run = true;+ }if approve == Some(Approval::Cancel) && dry_run {return Err("--approve cancel and --dry-run contradict each other: a dry run already indexes \@@ -1574,11 +1640,13 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {);}- // `map` reads the catalog off the server and `status` reads the local- // journal; neither walks a filesystem, so an ignore flag on either cannot- // change one byte of the output. Measured before this check existed:- // `xerj autoindex map --no-ignore` was byte-identical to `xerj autoindex- // map`. Accepting it was the #204 accept-and-ignore class (#279).+ // A bare `map` reads the catalog off the server and `status` reads the+ // local journal; neither walks a filesystem, so an ignore flag on either+ // cannot change one byte of the output. Measured before this check+ // existed: `xerj autoindex map --no-ignore` was byte-identical to+ // `xerj autoindex map`. Accepting it was the #204 accept-and-ignore class+ // (#279). `map <folder>` is the exception: it walks, and the flag is+ // honoured on that preview the same way `--dry-run` honours it.let ignore_flags_used: Vec<&str> = [("--no-ignore", no_ignore),("--no-default-ignores", no_default_ignores),@@ -1876,12 +1944,12 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {walk_only,},}),- (Some("map"), _) | (Some("status"), _) if walk_only => Err(format!(+ (Some("map"), None) | (Some("status"), _) if walk_only => Err(format!("--walk-only summarises a folder walk and does not apply to `autoindex {}`: that \subcommand never walks a tree. Drop it",sub.as_deref().unwrap_or_default())),- (Some("map"), _) | (Some("status"), _)+ (Some("map"), None) | (Some("status"), _)if max_minutes_explicit|| approve_explicit|| max_index_explicit@@ -1894,11 +1962,11 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {sub.as_deref().unwrap_or_default()))}- (Some("map"), _) | (Some("status"), _) if progress_explicit => Err(format!(+ (Some("map"), None) | (Some("status"), _) if progress_explicit => Err(format!("--progress/--progress-interval apply only to indexing, not `autoindex {}`",sub.as_deref().unwrap_or_default())),- (Some("map"), _) | (Some("status"), _) if !ignore_flags_used.is_empty() => {+ (Some("map"), None) | (Some("status"), _) if !ignore_flags_used.is_empty() => {let sub = sub.as_deref().unwrap_or_default();let (verb, them) = if ignore_flags_used.len() == 1 {("applies", "it")@@ -1911,7 +1979,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {ignore_flags_used.join(" and "),))}- (Some("map"), _) | (Some("status"), _) if endpoint_explicit => Err(format!(+ (Some("map"), None) | (Some("status"), _) if endpoint_explicit => Err(format!("--endpoint-url applies only to indexing, not `autoindex {}`: that subcommand reads the \XERJ node named by --url and never talks to an object store",sub.as_deref().unwrap_or_default()@@ -1919,12 +1987,12 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {(Some("map"), _) if bulk_timeout_explicit => {Err("--bulk-timeout-secs applies only to indexing, not `autoindex map`".into())}- (Some("map"), _) | (Some("status"), _) if label.is_some() => Err(format!(+ (Some("map"), None) | (Some("status"), _) if label.is_some() => Err(format!("--label labels indexed documents through /_decide; `autoindex {}` reads the node \and never indexes anything. Drop --label",sub.as_deref().unwrap_or_default())),- (Some("map"), _) => Ok(Cmd::Map(MapCfg {+ (Some("map"), None) => Ok(Cmd::Map(MapCfg {url,api_key,prefix: prefix_explicit.then_some(prefix),@@ -1962,7 +2030,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {state_dir,xerj_url_note,})),- (None, Some(root)) => {+ (None, Some(root)) | (Some("map"), Some(root)) => {// A flag that is accepted and does nothing is the shape this repo// refuses on purpose (#204, #279): the operator believes they asked// for out-of-root targets and gets a run that silently did not.@@ -2353,6 +2421,10 @@ mod tests {"an object-storage watch indexes nothing",),(vec!["map", "--label", "q.json"], "never indexes anything"),+ (+ vec!["map", "knowledge", "--label", "q.json"],+ "indexes nothing",+ ),(vec!["status", "--label", "q.json"],"never indexes anything",@@ -3871,4 +3943,123 @@ mod tests {other => panic!("expected a map config, got {other:?}"),}}++ /// #2368: `map` described itself as printing the discovered data map and+ /// then rejected `map <folder>` as an unknown argument. The folder form+ /// is the `--dry-run` preview. A bare `map` still reads the catalog.+ #[test]+ fn map_with_a_folder_is_the_dry_run_preview_and_bare_map_stays_the_catalog() {+ let help = super::help_text();+ assert!(+ help.contains("xerj autoindex map <folder>"),+ "help must show the folder form: {help}"+ );+ assert!(+ help.contains("what is already indexed"),+ "help must say a bare map reads indexed data: {help}"+ );+ assert!(+ help.contains("`autoindex map <folder>` is this same preview"),+ "help must point the preview at --dry-run: {help}"+ );+ assert!(+ !help.contains("print the discovered data map"),+ "the old usage line is what sent agents to pass a folder and get `unknown argument`"+ );++ match parse(["map"].iter().map(|s| s.to_string()).collect()).expect("bare map") {+ Cmd::Map(cfg) => {+ assert_eq!(cfg.prefix, None);+ assert_eq!(cfg.dataset, None);+ }+ other => panic!("a bare map must stay a catalog read, got {other:?}"),+ }++ let preview = index(&["map", "knowledge"]);+ let dry = index(&["knowledge", "--dry-run"]);+ assert_eq!(+ format!("{preview:?}"),+ format!("{dry:?}"),+ "map <folder> must be the same command as <folder> --dry-run"+ );+ assert_eq!(preview.root, PathBuf::from("knowledge"));+ assert!(preview.dry_run);+ assert!(!preview.walk_only);+ assert!(preview.ignore.deep_count);++ let flagged = index(&[+ "map",+ "--prefix",+ "notes",+ "--no-ignore",+ "--no-semantic",+ "--progress",+ "plain",+ "knowledge",+ ]);+ assert_eq!(flagged.root, PathBuf::from("knowledge"));+ assert_eq!(flagged.prefix, "notes");+ assert!(!flagged.ignore.enabled);+ assert!(flagged.no_semantic);+ assert_eq!(flagged.progress, ProgressMode::Plain);+ assert!(flagged.dry_run);++ let explicit = index(&["map", "knowledge", "--dry-run", "--prefix", "notes"]);+ assert!(explicit.dry_run);+ assert_eq!(explicit.prefix, "notes");+ assert_eq!(explicit.root, PathBuf::from("knowledge"));++ // A folder before `map` is still two positionals: `map` is only the+ // subcommand when it comes first, so `xerj autoindex ./map` indexes+ // a directory of that name.+ let reversed = err(&["knowledge", "map"]);+ assert!(reversed.contains("takes one folder per run"), "{reversed}");++ let two = err(&["map", "knowledge", "other"]);+ assert!(+ two.contains("at most one folder")+ && two.contains("knowledge")+ && two.contains("other"),+ "{two}"+ );+ assert!(!two.contains("unknown argument"), "{two}");++ let dataset = err(&["map", "knowledge", "--dataset", "docs"]);+ assert!(+ dataset.contains("--dataset") && dataset.contains("no folder"),+ "{dataset}"+ );+ let paging = err(&["map", "knowledge", "--limit", "2"]);+ assert!(+ paging.contains("--limit") && paging.contains("no folder"),+ "{paging}"+ );++ // Catalog selectors still belong to the no-folder form.+ match parse(+ ["map", "--dataset", "docs"]+ .iter()+ .map(|s| s.to_string())+ .collect(),+ )+ .expect("catalog map")+ {+ Cmd::Map(cfg) => assert_eq!(cfg.dataset.as_deref(), Some("docs")),+ other => panic!("expected a catalog map, got {other:?}"),+ }++ // Flags that change the walk are honoured on the preview and still+ // refused when there is no folder to walk.+ assert!(err(&["map", "--no-ignore"]).contains("never walks a folder"));+ assert!(err(&["map", "--max-minutes", "5"]).contains("apply only to indexing"));+ assert!(err(&["map", "knowledge", "--bulk-timeout-secs", "900"]).contains("autoindex map"));+ assert!(err(&["map", "knowledge", "--watch"]).contains("Drop --watch"));+ assert!(err(&["map", "knowledge", "--approve", "cancel"]).contains("nothing to cancel"));++ let both = err(&["map", "knowledge", "--no-ignore", "--no-default-ignores"]);+ assert!(+ both.contains("already turns off"),+ "the preview walks, so the redundant ignore pair is the indexing error: {both}"+ );+ }}diff --git a/engine/crates/xerj-autoindex/src/content.rs b/engine/crates/xerj-autoindex/src/content.rsindex 2afb0d894..2907ae21d 100644--- a/engine/crates/xerj-autoindex/src/content.rs+++ b/engine/crates/xerj-autoindex/src/content.rs@@ -10,6 +10,29 @@ use std::io::{BufReader, Read};use std::path::Path;use xxhash_rust::xxh3::Xxh3;+/// Why a walked file could not be opened for the hash (#2352).+///+/// `Absent` is cross-platform (`ErrorKind::NotFound`). `Locked` is the Windows+/// sharing violation (`ERROR_SHARING_VIOLATION` 32 and `ERROR_LOCK_VIOLATION`+/// 33). Unix errno 32 is `EPIPE`, so the raw code is only a lock on Windows.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum ReadFailureKind {+ Absent,+ Locked,+}++/// A file the walk listed and the hash could not open (#2352).+///+/// It is not a member of [`Inventory::files`]: there is no content digest, so+/// it must not become a content group. The run records one `skipped` catalog+/// row and continues.+#[derive(Debug, Clone)]+pub struct UnreadableSource {+ pub file: FileEntry,+ pub detail: String,+ pub kind: ReadFailureKind,+}+pub struct Inventory {pub files: Vec<FileEntry>,/// Durable document identity. New plans use the full-content key.@@ -17,6 +40,82 @@ pub struct Inventory {/// Full digest used to detect mutations even when a legacy key is retained.pub digests: Vec<String>,pub duplicates: Vec<DuplicateFile>,+ /// Walked files that disappeared or were locked before the hash could read+ /// them (#2352). Ordered by relative path.+ pub unreadable: Vec<UnreadableSource>,+}++/// Classify an open/read failure that should skip one file instead of aborting.+///+/// Anything else (permission denied without a Windows lock code, `EIO`, a+/// directory) stays fatal. `ErrorKind::PermissionDenied` alone is not a lock:+/// Rust has moved the sharing-violation kind between releases, and the raw+/// Win32 codes are the stable contract.+pub fn io_read_failure_kind(error: &std::io::Error) -> Option<ReadFailureKind> {+ if error.kind() == std::io::ErrorKind::NotFound {+ return Some(ReadFailureKind::Absent);+ }+ #[cfg(windows)]+ {+ match error.raw_os_error() {+ Some(32 | 33) => return Some(ReadFailureKind::Locked),+ _ => {}+ }+ }+ None+}++/// [`io_read_failure_kind`] over an `anyhow` chain (`with_context` hides the+/// `io::Error` one level down).+pub fn read_failure_kind(error: &anyhow::Error) -> Option<ReadFailureKind> {+ for cause in error.chain() {+ if let Some(io) = cause.downcast_ref::<std::io::Error>() {+ if let Some(kind) = io_read_failure_kind(io) {+ return Some(kind);+ }+ }+ }+ None+}++/// Catalog `reason` for one skipped unreadable file. Must not contain the+/// stale-resume marker (`not in the frozen resume plan`).+pub fn unreadable_skip_reason(detail: &str, kind: ReadFailureKind) -> String {+ match kind {+ ReadFailureKind::Absent => format!(+ "no longer exists (disappeared after it was enumerated: {detail}); skipped and \+ nothing from it was indexed. Rerun to index it if it comes back"+ ),+ ReadFailureKind::Locked => format!(+ "could not be opened because another process holds it locked ({detail}); skipped \+ and nothing from it was indexed. Rerun to index it once it can be opened"+ ),+ }+}++/// Plan key for a file that never produced a content digest.+///+/// `axf2-` keys are content hashes. This prefix cannot collide with one, and+/// it is stable for a path so the same skip is not a new catalog row on the+/// next run. It is not in `inventory.keys`, so the stale-junk sweep must be+/// told to keep it for this run.+pub fn unreadable_file_key(rel_id: &str) -> String {+ format!(+ "axabsent-{:032x}",+ xxhash_rust::xxh3::xxh3_128(rel_id.as_bytes())+ )+}++/// Sniff/read failures that are the same skip as the hash (#2352).+///+/// Other sniff errors stay `junk` with the caller's existing reason.+pub fn unreadable_junk_fields(error: &anyhow::Error) -> Option<(&'static str, String)> {+ read_failure_kind(error).map(|kind| {+ (+ "skipped",+ unreadable_skip_reason(&format!("{error:#}"), kind),+ )+ })}/// Test-only witness: which pool the digest phase actually ran on. Phase A is@@ -84,7 +183,18 @@ pub fn resolve_reporting_carried(// `--workers` has to bound the CPU-bound phase to mean anything (#240 §2).// The progress callback therefore fires from scan-pool threads — the ones// actually doing the hashing — rather than from rayon's global pool (#241).- let digests: Vec<Result<String>> = crate::pool::install(|| {+ // A file that vanished or is locked after the walk is one skip, not a+ // reason to abort the corpus (#2352). Any other open/read error still+ // fails the whole resolve, as before.+ enum HashOutcome {+ Ready(String),+ Unreadable {+ detail: String,+ kind: ReadFailureKind,+ },+ Fatal(anyhow::Error),+ }+ let outcomes: Vec<HashOutcome> = crate::pool::install(|| {files.par_iter().map(|file| {@@ -94,20 +204,20 @@ pub fn resolve_reporting_carried(rayon::current_num_threads(),std::sync::atomic::Ordering::Relaxed,);- let on_scan_pool = std::thread::current()- .name()+ // Unix workers set the pthread comm, not the std name (#1122).+ let on_scan_pool = xerj_common::thread_spawn::current_thread_name().is_some_and(|n| n.starts_with("xerj-scan-"));if !on_scan_pool {DIGEST_RAN_OFF_THE_SCAN_POOL.store(true, std::sync::atomic::Ordering::Relaxed);}}- let (digest, fresh) = match carried(file) {+ let (outcome, fresh) = match carried(file) {// Cache hit: no read, and no byte credited to the hash// phase either — the phase's totals were sized to the files// this pass will actually read, so crediting a skipped file// would push the percent past 100 and invent a rate.- Some(digest) => (Ok(digest), false),+ Some(digest) => (HashOutcome::Ready(digest), false),None => {// Held across the read and dropped before the byte// credit, so a blocked `read` is visible and a finished@@ -116,22 +226,62 @@ pub fn resolve_reporting_carried(let _blocking = crate::dataless::watch_blocking_hash(blocking, file);full_digest(&file.path, file.size)};+ // Credit the attempt even when the open fails. The+ // phase total counted this file; leaving it out parks+ // the bar short of 100% for a skip that is success.hashed(file.size);- (digest, true)+ let outcome = match digest {+ Ok(digest) => HashOutcome::Ready(digest),+ Err(error) => match read_failure_kind(&error) {+ Some(kind) => HashOutcome::Unreadable {+ detail: format!("{error:#}"),+ kind,+ },+ None => HashOutcome::Fatal(error),+ },+ };+ (outcome, true)}};- if let Ok(digest) = digest.as_ref() {+ if let HashOutcome::Ready(digest) = &outcome {observed(file, digest, fresh);}- digest+ outcome}).collect()});- let digests: Vec<String> = digests.into_iter().collect::<Result<_>>()?;- resolve_with_digests(files, digests)+ let mut fatal: Option<anyhow::Error> = None;+ let mut kept_files = Vec::new();+ let mut digests = Vec::new();+ let mut unreadable = Vec::new();+ for (file, outcome) in files.into_iter().zip(outcomes) {+ match outcome {+ HashOutcome::Ready(digest) => {+ kept_files.push(file);+ digests.push(digest);+ }+ HashOutcome::Unreadable { detail, kind } => {+ unreadable.push(UnreadableSource { file, detail, kind });+ }+ HashOutcome::Fatal(error) => {+ if fatal.is_none() {+ fatal = Some(error);+ }+ }+ }+ }+ if let Some(error) = fatal {+ return Err(error);+ }+ unreadable.sort_by(|left, right| left.file.rel.cmp(&right.file.rel));+ resolve_with_digests(kept_files, digests, unreadable)}-fn resolve_with_digests(files: Vec<FileEntry>, digests: Vec<String>) -> Result<Inventory> {+fn resolve_with_digests(+ files: Vec<FileEntry>,+ digests: Vec<String>,+ unreadable: Vec<UnreadableSource>,+) -> Result<Inventory> {let mut groups: HashMap<&str, Vec<usize>> = HashMap::new();for (index, digest) in digests.iter().enumerate() {groups.entry(digest).or_default().push(index);@@ -209,6 +359,7 @@ fn resolve_with_digests(files: Vec<FileEntry>, digests: Vec<String>) -> Result<Ikeys: unique_keys,digests: unique_digests,duplicates,+ unreadable,})}@@ -221,6 +372,9 @@ fn resolve_with_digests(files: Vec<FileEntry>, digests: Vec<String>) -> Result<Ipub struct SourceChanged {pub path: std::path::PathBuf,pub detail: String,+ /// The file is still on disk and another process holds it. A no-graph seal+ /// can keep the previous generation's bytes instead of dropping the file.+ pub locked: bool,}impl std::fmt::Display for SourceChanged {@@ -231,10 +385,36 @@ impl std::fmt::Display for SourceChanged {impl std::error::Error for SourceChanged {}+fn disappeared_or_locked(path: &Path, kind: ReadFailureKind, detail: &str) -> SourceChanged {+ let detail = match kind {+ ReadFailureKind::Absent => format!(+ "{} no longer exists ({detail}); retry the run",+ path.display()+ ),+ ReadFailureKind::Locked => format!(+ "{} could not be opened because another process holds it locked ({detail}); retry \+ the run",+ path.display()+ ),+ };+ SourceChanged {+ path: path.to_path_buf(),+ detail,+ locked: kind == ReadFailureKind::Locked,+ }+}+pub fn verify(path: &Path, expected_size: u64, expected_digest: &str) -> Result<()> {- let actual_size = std::fs::metadata(path)- .with_context(|| format!("stat {}", path.display()))?- .len();+ let metadata = match std::fs::metadata(path) {+ Ok(metadata) => metadata,+ Err(error) => {+ if let Some(kind) = io_read_failure_kind(&error) {+ return Err(disappeared_or_locked(path, kind, &format!("{error}")).into());+ }+ return Err(error).with_context(|| format!("stat {}", path.display()));+ }+ };+ let actual_size = metadata.len();if actual_size != expected_size {return Err(SourceChanged {path: path.to_path_buf(),@@ -244,10 +424,19 @@ pub fn verify(path: &Path, expected_size: u64, expected_digest: &str) -> Result<expected_size,actual_size),+ locked: false,}.into());}- let actual = full_digest(path, actual_size)?;+ let actual = match full_digest(path, actual_size) {+ Ok(digest) => digest,+ Err(error) => {+ if let Some(kind) = read_failure_kind(&error) {+ return Err(disappeared_or_locked(path, kind, &format!("{error:#}")).into());+ }+ return Err(error);+ }+ };if actual != expected_digest {return Err(SourceChanged {path: path.to_path_buf(),@@ -255,6 +444,7 @@ pub fn verify(path: &Path, expected_size: u64, expected_digest: &str) -> Result<"{} changed content during autoindex; retry the run",path.display()),+ locked: false,}.into());}@@ -422,7 +612,7 @@ mod tests {fs::write(dir.path().join("b.txt"), b"unequal-b").unwrap();let files = crate::walk::walk(dir.path(), false).unwrap();let forced = vec!["axf2-forced-9".to_string(); 2];- let inv = resolve_with_digests(files, forced).unwrap();+ let inv = resolve_with_digests(files, forced, Vec::new()).unwrap();assert_eq!(inv.files.len(), 2);assert!(inv.duplicates.is_empty());assert_ne!(inv.keys[0], inv.keys[1]);@@ -479,4 +669,100 @@ mod tests {assert_eq!(inv.duplicates[0].rel, "a-link");assert_eq!(inv.duplicates[0].is_symlink, Some(true));}++ #[test]+ fn a_file_deleted_after_the_walk_is_skipped_and_the_rest_are_hashed() {+ let dir = tempfile::tempdir().unwrap();+ fs::write(dir.path().join("keep.txt"), b"keep\n").unwrap();+ fs::write(dir.path().join("gone.txt"), b"gone\n").unwrap();+ let files = crate::walk::walk(dir.path(), false).unwrap();+ let gone = files+ .iter()+ .find(|file| file.rel.ends_with("gone.txt"))+ .unwrap()+ .path+ .clone();+ fs::remove_file(&gone).unwrap();+ let reported = std::sync::atomic::AtomicU64::new(0);+ let inv = resolve_reporting(files, &|bytes| {+ reported.fetch_add(bytes, std::sync::atomic::Ordering::Relaxed);+ })+ .unwrap();+ assert_eq!(inv.files.len(), 1);+ assert!(inv.files[0].rel.ends_with("keep.txt"));+ assert_eq!(inv.unreadable.len(), 1);+ assert_eq!(inv.unreadable[0].kind, ReadFailureKind::Absent);+ assert!(inv.unreadable[0].file.rel.ends_with("gone.txt"));+ let reason = unreadable_skip_reason(&inv.unreadable[0].detail, inv.unreadable[0].kind);+ assert!(reason.contains("no longer exists"), "{reason}");+ assert!(reason.contains("disappeared"), "{reason}");+ assert!(+ !reason.contains("not in the frozen resume plan"),+ "{reason}"+ );+ assert!(reported.load(std::sync::atomic::Ordering::Relaxed) > 0);+ }++ #[test]+ fn a_directory_handed_to_the_hash_still_aborts_the_resolve() {+ let dir = tempfile::tempdir().unwrap();+ let nested = dir.path().join("subdir");+ fs::create_dir(&nested).unwrap();+ fs::write(dir.path().join("keep.txt"), b"keep\n").unwrap();+ let mut files = crate::walk::walk(dir.path(), false).unwrap();+ files.push(FileEntry {+ path: nested,+ rel: "subdir".into(),+ rel_id: "subdir".into(),+ is_symlink: false,+ size: 0,+ });+ assert!(resolve(files).is_err());+ }++ #[test]+ fn not_found_is_absent_and_a_sharing_violation_is_locked_only_on_windows() {+ let missing = std::io::Error::from_raw_os_error(2);+ assert_eq!(+ io_read_failure_kind(&missing),+ Some(ReadFailureKind::Absent)+ );+ let sharing = std::io::Error::from_raw_os_error(32);+ let lock = std::io::Error::from_raw_os_error(33);+ #[cfg(windows)]+ {+ assert_eq!(+ io_read_failure_kind(&sharing),+ Some(ReadFailureKind::Locked)+ );+ assert_eq!(io_read_failure_kind(&lock), Some(ReadFailureKind::Locked));+ }+ #[cfg(not(windows))]+ {+ assert_eq!(io_read_failure_kind(&sharing), None);+ assert_eq!(io_read_failure_kind(&lock), None);+ }+ let denied = std::io::Error::new(std::io::ErrorKind::PermissionDenied, "denied");+ assert_eq!(io_read_failure_kind(&denied), None);+ let wrapped =+ anyhow::Error::from(std::io::Error::from_raw_os_error(2)).context("open gone.txt");+ assert_eq!(read_failure_kind(&wrapped), Some(ReadFailureKind::Absent));+ }++ #[test]+ fn verification_names_a_file_that_disappeared() {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("a.txt");+ fs::write(&path, b"content-a").unwrap();+ let inv = resolve(crate::walk::walk(dir.path(), false).unwrap()).unwrap();+ fs::remove_file(&path).unwrap();+ let err = verify(&path, inv.files[0].size, &inv.digests[0]).unwrap_err();+ let changed = err.downcast_ref::<SourceChanged>().expect("source changed");+ assert!(!changed.locked);+ assert!(+ changed.detail.contains("no longer exists"),+ "{}",+ changed.detail+ );+ }}diff --git a/engine/crates/xerj-autoindex/src/extract/code.rs b/engine/crates/xerj-autoindex/src/extract/code.rsindex 9e0a40d9f..414d0dbd2 100644--- a/engine/crates/xerj-autoindex/src/extract/code.rs+++ b/engine/crates/xerj-autoindex/src/extract/code.rs@@ -1355,42 +1355,10 @@ fn emit_code_doc(.to_string();let mut fields = Map::new();- fields.insert("title".into(), Value::String(title));+ fields.insert("title".into(), Value::String(title.clone()));fields.insert("language".into(), Value::String(language.to_string()));- if !symbols.is_empty() {- // Searchable "kind name" list — this is what makes `class User` /- // `def save` retrieve the file, and it dedups repeated names.- let mut seen = std::collections::HashSet::new();- let defs: Vec<String> = symbols- .iter()- .filter(|s| seen.insert((s.kind.clone(), s.name.clone())))- .map(|s| format!("{} {}", s.kind, s.name))- .collect();- fields.insert("defs".into(), Value::String(defs.join("\n")));- let arr: Vec<Value> = symbols- .iter()- .map(|s| {- let mut m = Map::new();- m.insert("name".into(), Value::String(s.name.clone()));- m.insert("kind".into(), Value::String(s.kind.clone()));- m.insert("line".into(), Value::Number((s.line as u64).into()));- m.insert("start_line".into(), Value::Number((s.line as u64).into()));- // #500: a client that wants the IMPLEMENTATION can now slice- // exactly `line`..`end_line` out of `body`. Without it the only- // way to bound a symbol was "up to where the next one starts",- // which returns a whole class for a class and the rest of the- // file for the last symbol — the coarse unit this issue is about.- m.insert("end_line".into(), Value::Number((s.end_line as u64).into()));- Value::Object(m)- })- .collect();- fields.insert("symbols".into(), Value::Array(arr));- fields.insert(- "symbol_count".into(),- Value::Number((symbols.len() as u64).into()),- );- }+ insert_symbol_catalog(&mut fields, symbols);// The full source stays full-text searchable.fields.insert("body".into(), Value::String(text.to_string()));// #2351: the file document has no `line`. The span of the stored text@@ -1414,17 +1382,69 @@ fn emit_code_doc(return false;}- // #500: promote each declaration to its OWN retrievable document. A- // constant/field/`#define`/signature lookup then returns the ~40–80 B- // declaration line with an exact `name` (mapped keyword), instead of only- // surviving folded inside the ~2–3 KB enclosing class/method body (the- // measured 32–48× byte blow-up + the rank-35 recall miss). The file- // document above keeps `body`/`defs` for full-text and cross-file search.- let file_path = path- .file_name()- .and_then(|n| n.to_str())- .unwrap_or("source")- .to_string();+ emit_symbol_records(&title, language, text, symbols, stats, sink)+}++/// `defs` / `symbols` / `symbol_count` for one file. Empty when nothing was+/// captured, so a file with no definitions keeps the same field set it had+/// before a grammar learned to see one (#178).+fn insert_symbol_catalog(fields: &mut Map<String, Value>, symbols: &[Symbol]) {+ if symbols.is_empty() {+ return;+ }+ // Searchable "kind name" list — this is what makes `class User` /+ // `def save` retrieve the file, and it dedups repeated names.+ let mut seen = std::collections::HashSet::new();+ let defs: Vec<String> = symbols+ .iter()+ .filter(|s| seen.insert((s.kind.clone(), s.name.clone())))+ .map(|s| format!("{} {}", s.kind, s.name))+ .collect();+ fields.insert("defs".into(), Value::String(defs.join("\n")));+ let arr: Vec<Value> = symbols+ .iter()+ .map(|s| {+ let mut m = Map::new();+ m.insert("name".into(), Value::String(s.name.clone()));+ m.insert("kind".into(), Value::String(s.kind.clone()));+ m.insert("line".into(), Value::Number((s.line as u64).into()));+ // #2351: start_line matches line so a _source include of+ // start_line/end_line locates the declaration. The file document+ // and each symbol record both carry it.+ m.insert("start_line".into(), Value::Number((s.line as u64).into()));+ // #500: a client that wants the IMPLEMENTATION can now slice+ // exactly `line`..`end_line` out of `body`. Without it the only+ // way to bound a symbol was "up to where the next one starts",+ // which returns a whole class for a class and the rest of the+ // file for the last symbol — the coarse unit this issue is about.+ m.insert("end_line".into(), Value::Number((s.end_line as u64).into()));+ Value::Object(m)+ })+ .collect();+ fields.insert("symbols".into(), Value::Array(arr));+ fields.insert(+ "symbol_count".into(),+ Value::Number((symbols.len() as u64).into()),+ );+}++/// #500: promote each declaration to its OWN retrievable document. A+/// constant/field/`#define`/signature lookup then returns the ~40–80 B+/// declaration line with an exact `name` (mapped keyword), instead of only+/// surviving folded inside the ~2–3 KB enclosing class/method body (the+/// measured 32–48× byte blow-up + the rank-35 recall miss). The file+/// document above keeps `body`/`defs` for full-text and cross-file search.+///+/// `text`'s line `n` is source line `n`. Callers that parsed a slice shift+/// `symbols` onto that coordinate first.+fn emit_symbol_records(+ file_path: &str,+ language: &str,+ text: &str,+ symbols: &[Symbol],+ stats: &mut ExtractStats,+ sink: Sink,+) -> bool {// A single declaration can be captured under two kinds at the same line+name// (e.g. `export const f = () => {}` → @function + @const). Those share the// `code:<line>:<name>` locator → the same `doc_id` → ES stores ONE document.@@ -1451,7 +1471,7 @@ fn emit_code_doc(// `name`/`code` stay searchable (the mapping exposes `name.keyword`// for exact-identifier ranking), `kind`/`line` for filtering + citation.sf.insert("title".into(), Value::String(name.clone()));- sf.insert("path".into(), Value::String(file_path.clone()));+ sf.insert("path".into(), Value::String(file_path.to_string()));sf.insert("language".into(), Value::String(language.to_string()));sf.insert("name".into(), Value::String(name.clone()));sf.insert("kind".into(), Value::String(s.kind.clone()));@@ -1525,6 +1545,81 @@ fn emit_code_doc(true}+/// Definitions inside inline `<script>` blocks of an HTML page (#2356).+///+/// `spans` are `(start, end)` byte offsets of script *content* (inside the+/// tags, not including them) in `html`. Each span is parsed with the+/// JavaScript grammar; line numbers are shifted onto the HTML file so+/// `xerj def` cites `page.html:N`. The page's prose record is someone else's+/// — this only adds the file-level `defs`/`symbols` catalog and the+/// per-declaration records a `.js` file would get. It does not copy the+/// script into a `body` field: the HTML extractor keeps API keys and other+/// non-definition script text out of the page body.+///+/// A span over [`CODE_CAP`] is skipped. One minified block is not a file of+/// definitions, and parsing it is the cost the code extractor already refuses.+pub(crate) fn emit_inline_scripts(+ path: &Path,+ html: &str,+ spans: &[(usize, usize)],+ stats: &mut ExtractStats,+ sink: Sink,+) -> bool {+ if spans.is_empty() {+ return true;+ }+ let Some(def) = registry().iter().find(|d| d.name == "javascript") else {+ return true;+ };+ let mut symbols = Vec::new();+ for &(start, end) in spans {+ if start >= end+ || end > html.len()+ || !html.is_char_boundary(start)+ || !html.is_char_boundary(end)+ {+ continue;+ }+ let content = &html[start..end];+ if content.len() as u64 > CODE_CAP || content.trim().is_empty() {+ continue;+ }+ let mut found = parse_symbols(def, content).unwrap_or_default();+ let offset = html[..start].bytes().filter(|&b| b == b'\n').count();+ for s in &mut found {+ s.line = s.line.saturating_add(offset);+ s.end_line = s.end_line.saturating_add(offset);+ }+ symbols.append(&mut found);+ if symbols.len() >= 5000 {+ symbols.truncate(5000);+ break;+ }+ }+ if symbols.is_empty() {+ return true;+ }+ let title = path+ .file_name()+ .and_then(|n| n.to_str())+ .unwrap_or("source")+ .to_string();+ let mut fields = Map::new();+ fields.insert("title".into(), Value::String(title.clone()));+ fields.insert("language".into(), Value::String("javascript".into()));+ insert_symbol_catalog(&mut fields, &symbols);+ stats.records += 1;+ if !sink(RawRecord {+ fields,+ locator: "code".into(),+ group: None,+ origin: FieldOrigin::Extractor,+ }) {+ return false;+ }+ emit_symbol_records(&title, "javascript", html, &symbols, stats, sink)+}+// ── Per-language capture queries. Capture-name == emitted symbol kind. ─────────const PYTHON_Q: &str = r#"diff --git a/engine/crates/xerj-autoindex/src/extract/html.rs b/engine/crates/xerj-autoindex/src/extract/html.rsindex 64f6622e3..ae21beff5 100644--- a/engine/crates/xerj-autoindex/src/extract/html.rs+++ b/engine/crates/xerj-autoindex/src/extract/html.rs@@ -3,6 +3,14 @@//! a dominant `<table>` (≥5 rows, consistent column count, header row) →//! one record per row with header-derived field names; otherwise one//! document record {title, headings, body}.+//!+//! Inline JavaScript in a `<script>` block of a document page is also+//! extracted as code symbols — the same definitions a standalone `.js` file+//! would yield — with line numbers in this HTML file (#2356). The page body+//! stays the visible text: script and style source is not copied into it.+//! A dominant table stays one record per row. Its scripts are not symbols:+//! those fields would land in the table's data dataset, and a sample that+//! stopped on the rows would leave them out of the frozen mapping.use super::{emit_document, sanitize_field_name, ExtractStats, FieldOrigin, RawRecord, Sink,@@ -25,6 +33,10 @@ pub(crate) struct Doc {pub(crate) anchors: Vec<(String, usize)>,tables: Vec<Vec<Vec<String>>>, // tables → rows → cellsheader_cells: Vec<Vec<bool>>, // per table: was first row <th>?+ /// Inline JavaScript blocks as `(content_start, content_end)` byte+ /// offsets into the source. External scripts (`src`), non-JS types, and+ /// `<style>` are not recorded. The page body still omits this text.+ scripts: Vec<(usize, usize)>,}pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {@@ -34,7 +46,8 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {return Ok(stats);};let (text, _) = crate::sniff::decode_text(&bytes);- let doc = parse(&text);+ let mut doc = parse(&text);+ let scripts = std::mem::take(&mut doc.scripts);// Dominant-table rule.if let Some((rows, first_row_th)) = dominant_table(&doc) {@@ -87,6 +100,12 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {return Ok(stats);}}+ // Scripts stay out of this path. Symbol records share the file's+ // ungrouped sketch, so their extractor fields (`name`, `code`,+ // `defs`, …) would join the table's data dataset. Phase A stops at+ // the sample, often before those records, and phase B would then+ // index fields the frozen mapping does not name.+ drop(scripts);return Ok(stats);}@@ -99,7 +118,7 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {} else {doc.title.trim().to_string()};- emit_document(+ let keep_going = emit_document(&title,&doc.headings,doc.body.trim(),@@ -108,6 +127,9 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {sink,&mut stats,);+ if keep_going {+ super::code::emit_inline_scripts(path, &text, &scripts, &mut stats, sink);+ }Ok(stats)}@@ -255,7 +277,15 @@ pub(crate) fn parse(html: &str) -> Doc {.is_some_and(|&c| c.is_ascii_whitespace() || c == b'/' || c == b'>');if !close && name_ended && matches!(name.as_str(), "script" | "style") && !self_closing{- i = raw_text_end(bytes, tag_end + 1, name.as_bytes());+ let content_start = tag_end + 1;+ let content_end = raw_text_end(bytes, content_start, name.as_bytes());+ if name == "script"+ && content_start < content_end+ && inline_javascript_script(&html[j..tag_end])+ {+ doc.scripts.push((content_start, content_end));+ }+ i = content_end;continue;}@@ -532,6 +562,120 @@ fn decode_entities(s: &str) -> String {.replace(" ", " ")}+/// HTML spec JavaScript MIME type essences (mimesniff). `type=module` is a+/// module script, not one of these, and is handled beside them.+const JAVASCRIPT_MIME_TYPES: &[&str] = &[+ "application/ecmascript",+ "application/javascript",+ "application/x-ecmascript",+ "application/x-javascript",+ "text/ecmascript",+ "text/javascript",+ "text/javascript1.0",+ "text/javascript1.1",+ "text/javascript1.2",+ "text/javascript1.3",+ "text/javascript1.4",+ "text/javascript1.5",+ "text/jscript",+ "text/livescript",+ "text/x-ecmascript",+ "text/x-javascript",+];++/// True when this `<script>` element's inline text is JavaScript the page+/// runs. A `src` script's inline text is not executed. A `type` that is not+/// a JavaScript MIME type (JSON-LD, templates, import maps) is data, not+/// code. With no `type`, the legacy `language` attribute is consulted; with+/// neither, the element is a classic script.+fn inline_javascript_script(attrs: &str) -> bool {+ if attr_value(attrs, "src").is_some() {+ return false;+ }+ match attr_value(attrs, "type") {+ Some(ty) => is_javascript_type(ty),+ None => match attr_value(attrs, "language") {+ Some(lang) => is_javascript_language(lang),+ None => true,+ },+ }+}++fn is_javascript_type(ty: &str) -> bool {+ let essence = ty.split(';').next().unwrap_or("").trim();+ if essence.is_empty() {+ return false;+ }+ if essence.eq_ignore_ascii_case("module") {+ return true;+ }+ JAVASCRIPT_MIME_TYPES+ .iter()+ .any(|mime| essence.eq_ignore_ascii_case(mime))+}++fn is_javascript_language(lang: &str) -> bool {+ matches!(+ lang.trim().to_ascii_lowercase().as_str(),+ "javascript" | "js" | "ecmascript" | "jscript" | "livescript"+ )+}++/// The value of attribute `want`, or `None` when it is absent. A present+/// attribute with no value is `Some("")`. `attrs` is the tag text after the+/// element name and before `>`.+fn attr_value<'a>(attrs: &'a str, want: &str) -> Option<&'a str> {+ let b = attrs.as_bytes();+ let mut i = 0;+ while i < b.len() {+ while i < b.len() && (b[i].is_ascii_whitespace() || b[i] == b'/') {+ i += 1;+ }+ if i >= b.len() || b[i] == b'>' {+ break;+ }+ let key_start = i;+ while i < b.len() && !b[i].is_ascii_whitespace() && !matches!(b[i], b'=' | b'/' | b'>') {+ i += 1;+ }+ let key = &attrs[key_start..i];+ while i < b.len() && b[i].is_ascii_whitespace() {+ i += 1;+ }+ let value = if i < b.len() && b[i] == b'=' {+ i += 1;+ while i < b.len() && b[i].is_ascii_whitespace() {+ i += 1;+ }+ if i < b.len() && matches!(b[i], b'"' | b'\'') {+ let q = b[i];+ let v = i + 1;+ i = v;+ while i < b.len() && b[i] != q {+ i += 1;+ }+ let value = &attrs[v..i];+ if i < b.len() {+ i += 1;+ }+ value+ } else {+ let v = i;+ while i < b.len() && !b[i].is_ascii_whitespace() && b[i] != b'>' {+ i += 1;+ }+ &attrs[v..i]+ }+ } else {+ ""+ };+ if key.eq_ignore_ascii_case(want) {+ return Some(value);+ }+ }+ None+}+#[cfg(test)]mod tests {use super::*;@@ -787,6 +931,24 @@ mod tests {assert!(recs.iter().all(|r| r.group.is_none()));}+ /// A dominant table and an inline function share one ungrouped sketch.+ /// Symbol fields must not join the row dataset: phase A often samples+ /// only the rows, and the frozen mapping would then lack `name`/`code`.+ #[test]+ fn a_dominant_table_does_not_emit_inline_script_symbols() {+ let mut html = table_html(Some(["Product Name", "Qty"]), 5);+ html.push_str("<script>function openDrawer(id) { return id; }</script>");+ let (stats, recs) = run("t.html", &html);+ assert_eq!(stats.records, 5);+ assert!(recs.iter().all(|r| r.locator.starts_with("row")));+ assert!(recs.iter().all(|r| r.fields.get("name").is_none()));+ assert!(recs.iter().all(|r| r.fields.get("defs").is_none()));+ assert!(recs.iter().all(|r| !r+ .fields+ .values()+ .any(|v| v.as_str().is_some_and(|s| s.contains("openDrawer")))));+ }+#[test]fn a_headerless_table_gets_positional_column_names_and_keeps_its_first_row() {let (stats, recs) = run("t.html", &table_html(None, 6));@@ -931,4 +1093,169 @@ mod tests {assert_eq!(stats.records, 1);assert_eq!(body_of(&recs[0]), "");}++ /// #2356: a function inside an inline `<script>` is a code symbol at the+ /// line it occupies in the HTML file. The page's prose record stays the+ /// visible text — the script is not copied into `body`. `.html` and+ /// `.htm` take the same extractor. A `src` script, a JSON-LD block, a+ /// template, a style block, and a comment are not definitions.+ #[test]+ fn inline_script_functions_become_symbols_at_their_html_lines() {+ let page = "\+<!DOCTYPE html>+<html>+<head><title>Drawer</title></head>+<body>+<p>The drawer holds the menu.</p>+<script>+function openDrawer(id) {+ document.getElementById(id).hidden = false;+}+</script>+<script type=\"text/javascript\">+const toggleDrawer = () => {+ return true;+};+</script>+<script type=\"module\">+export function openPanel() {+ return 1;+}+</script>+<script type=\"application/ld+json\">+function ignoredJson() {}+</script>+<script src=\"app.js\">+function fromSrc() {}+</script>+<script type=\"text/template\">+function notATemplateFn() {}+</script>+<style>+function notCss() {}+</style>+<!-- <script>function inComment() {}</script> -->+<p>After the scripts.</p>+</body>+</html>+";+ let line_of = |needle: &str| {+ page.lines()+ .position(|line| line.contains(needle))+ .map(|i| i + 1)+ .unwrap_or_else(|| panic!("{needle} is not in the fixture"))+ };+ for name in ["page.html", "page.htm"] {+ let (stats, recs) = run(name, page);+ let prose = recs+ .iter()+ .find(|r| r.locator == "s0")+ .expect("prose record");+ let body = body_of(prose);+ assert!(body.contains("The drawer holds the menu."), "{body:?}");+ assert!(body.contains("After the scripts."), "{body:?}");+ assert_eq!(prose.fields["title"], serde_json::json!("Drawer"));+ for leak in [+ "openDrawer",+ "getElementById",+ "toggleDrawer",+ "openPanel",+ "ignoredJson",+ "fromSrc",+ "notATemplateFn",+ "notCss",+ "inComment",+ ] {+ assert!(+ !body.contains(leak),+ "{leak} leaked into the page body: {body:?}"+ );+ }++ let catalog = recs+ .iter()+ .find(|r| r.locator == "code")+ .expect("defs catalog");+ let defs = catalog.fields["defs"].as_str().unwrap();+ for expect in [+ "function openDrawer",+ "function toggleDrawer",+ "function openPanel",+ ] {+ assert!(+ defs.contains(expect),+ "{name} defs missing {expect}: {defs}"+ );+ }+ for absent in [+ "ignoredJson",+ "fromSrc",+ "notATemplateFn",+ "notCss",+ "inComment",+ ] {+ assert!(!defs.contains(absent), "{name} extracted {absent}: {defs}");+ }+ assert!(+ catalog.fields.get("body").is_none(),+ "script text must not become body"+ );+ assert_eq!(catalog.fields["language"], serde_json::json!("javascript"));++ let symbol = |fn_name: &str| {+ recs.iter()+ .find(|r| {+ r.fields.get("name").and_then(|v| v.as_str()) == Some(fn_name)+ && r.locator.starts_with("code:")+ })+ .unwrap_or_else(|| panic!("{name} has no symbol {fn_name}"))+ };+ let open = symbol("openDrawer");+ let open_line = line_of("function openDrawer");+ assert_eq!(open.fields["line"], serde_json::json!(open_line));+ assert_eq!(open.fields["kind"], serde_json::json!("function"));+ assert_eq!(open.fields["language"], serde_json::json!("javascript"));+ assert_eq!(open.locator, format!("code:{open_line}:openDrawer"));+ let code = open.fields["code"].as_str().unwrap();+ assert!(code.contains("function openDrawer"), "{code}");+ let impl_text = open.fields["impl"].as_str().unwrap();+ assert!(impl_text.contains("getElementById"), "{impl_text}");+ assert!(+ page.lines()+ .nth(open_line - 1)+ .unwrap()+ .contains("openDrawer"),+ "line {open_line} is not the definition"+ );++ let toggle = symbol("toggleDrawer");+ assert_eq!(+ toggle.fields["line"],+ serde_json::json!(line_of("const toggleDrawer"))+ );+ assert_eq!(toggle.fields["kind"], serde_json::json!("function"));++ let panel = symbol("openPanel");+ assert_eq!(+ panel.fields["line"],+ serde_json::json!(line_of("export function openPanel"))+ );+ assert_eq!(panel.fields["kind"], serde_json::json!("function"));++ let names: Vec<&str> = recs+ .iter()+ .filter_map(|r| r.fields.get("name").and_then(|v| v.as_str()))+ .collect();+ assert!(+ !names.iter().any(|n| {+ matches!(+ *n,+ "ignoredJson" | "fromSrc" | "notATemplateFn" | "notCss" | "inComment"+ )+ }),+ "{name} symbols: {names:?}"+ );+ assert!(stats.records >= 4, "{name} records {}", stats.records);+ }+ }}diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex e6642c80a..eb68c67dc 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -1489,6 +1489,63 @@ fn a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it() {assert!(indexed.contains("second.csv"), "{indexed:?}");}+/// #2372: the stale-resume-plan summary names the appeared paths and says+/// `--fresh` next to them. `skipped_appeared` stays the count. The existing+/// `skipped` list is unchanged. The plain `xerj-done` line stays numeric;+/// the names are the lines immediately before it.+fn assert_stale_resume_summary(report: &Value, stream: &str, paths: &[&str]) {+ assert_eq!(report["files_junk"], 0, "{report}");+ assert_eq!(report["skipped_appeared"], paths.len(), "{report}");+ assert_eq!(report["skipped_appeared_files"], json!(paths), "{report}");+ assert!(+ report.get("skipped_appeared_not_listed").is_none(),+ "these paths fit in the summary: {report}"+ );+ let action = report["skipped_appeared_action"]+ .as_str()+ .unwrap_or_else(|| panic!("missing skipped_appeared_action: {report}"));+ assert!(action.contains("--fresh"), "{action}");+ let listed: Vec<&str> = report["skipped"]["files"]+ .as_array()+ .unwrap_or_else(|| panic!("skipped list removed: {report}"))+ .iter()+ .filter_map(|file| file["path"].as_str())+ .collect();+ assert_eq!(+ listed, paths,+ "the existing skipped list still names them: {report}"+ );++ let done_at = stream+ .rfind("xerj-done ")+ .unwrap_or_else(|| panic!("{stream}"));+ let summary_at = stream[..done_at]+ .rfind("not indexed —")+ .unwrap_or_else(|| panic!("summary did not name the files before xerj-done:\n{stream}"));+ let summary = &stream[summary_at..done_at];+ for path in paths {+ assert!(summary.contains(path), "summary missing {path}: {summary}");+ }+ assert!(summary.contains("--fresh"), "{summary}");+ assert!(+ !summary.contains("… and"),+ "these paths are not a remainder: {summary}"+ );+ let done = stream[done_at..]+ .lines()+ .next()+ .unwrap_or_else(|| panic!("{stream}"));+ assert!(done.contains("reason=stale-resume-plan"), "{done}");+ assert!(+ done.contains(&format!("skipped_appeared={}", paths.len())),+ "{done}"+ );+ assert!(+ !done.contains(paths[0]),+ "the plain done line stays numeric; paths are the lines above it: {done}"+ );+}+/// #2076: a resume whose journaled plan predates files now on disk used to/// record those files as junk and finish `reason=completed-with-junk`. That is/// the everyday "some input was unusable" success, so an agent treats the@@ -1539,6 +1596,7 @@ fn a_resume_whose_plan_predates_new_files_reports_a_stale_plan_not_junk() {"post-plan files are not junk: {report}");assert_eq!(report["skipped_appeared"], 2, "{report}");+ assert_stale_resume_summary(&report, &stream, &["added-a.md", "added-b.md"]);assert!(stream.contains("skipped because the resumed plan is stale"),"{stream}"@@ -1574,16 +1632,48 @@ fn a_resume_whose_plan_predates_new_files_reports_a_stale_plan_not_junk() {}// A second resume must not quietly reclassify the same files as junk.+ // `--progress json` is the machine terminal object: the paths and the+ // `--fresh` recovery have to be fields on it, not only prose notes.+ config.progress = crate::progress::ProgressMode::Json;let (again, report, stream) = resume(&config);assert_eq!(again, 3);let report = report.unwrap();assert_eq!(report["files_junk"], 0, "{report}");assert_eq!(report["skipped_appeared"], 2, "{report}");+ assert_eq!(+ report["skipped_appeared_files"],+ json!(["added-a.md", "added-b.md"]),+ "{report}"+ );+ assert!(+ report["skipped_appeared_action"]+ .as_str()+ .unwrap_or("")+ .contains("--fresh"),+ "{report}"+ );let done = stream.lines()- .find(|line| line.starts_with("xerj-done "))- .unwrap_or_else(|| panic!("{stream}"));- assert!(done.contains("reason=stale-resume-plan"), "{done}");+ .filter_map(|line| serde_json::from_str::<Value>(line).ok())+ .find(|event| event["event"] == "done")+ .unwrap_or_else(|| panic!("no JSON done event:\n{stream}"));+ assert_eq!(done["exit"], 3, "{done}");+ assert_eq!(done["reason"], "stale-resume-plan", "{done}");+ assert_eq!(done["ok"], true, "{done}");+ assert_eq!(done["skipped_appeared"], 2, "{done}");+ assert_eq!(+ done["skipped_appeared_files"],+ json!(["added-a.md", "added-b.md"]),+ "{done}"+ );+ assert!(done.get("skipped_appeared_not_listed").is_none(), "{done}");+ assert!(+ done["skipped_appeared_action"]+ .as_str()+ .unwrap_or("")+ .contains("--fresh"),+ "{done}"+ );config.fresh = true;config.quiet = true;@@ -1683,6 +1773,81 @@ fn a_file_that_changes_mid_run_is_skipped_and_the_run_finishes() {assert!(indexed.contains("app.lock"), "{indexed:?}");}+/// #2352: a file that disappears between the walk and the hash (a browser+/// profile lock file another process removes) used to abort the whole run+/// with ENOENT. It is one skipped catalog row; every other file is indexed+/// and the run finishes with exit 3. The next run, with the file still gone,+/// sweeps that row and exits 0.+#[test]+fn a_file_deleted_after_enumeration_is_skipped_and_the_run_finishes() {+ let _guard = FAILPOINT_TEST_LOCK+ .lock()+ .unwrap_or_else(|p| p.into_inner());+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK+ .lock()+ .unwrap_or_else(|p| p.into_inner());+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();+ let live = corpus.path().join("app.lock");+ fs::write(&live, "pid 1\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);++ let gone = live.clone();+ arm_before_hash_hook(state_dir.path(), move || {+ fs::remove_file(gone).unwrap();+ });+ let (code, report) =+ run_index_report(config.clone()).expect("a vanished file must not abort the run");+ assert_eq!(+ code, 3,+ "a skipped file is completed-with-junk, not a failure"+ );+ let report = report.unwrap();+ assert_eq!(report["files_junk"], 1, "{report}");+ assert_eq!(+ file_done_count(state_dir.path()),+ 2,+ "every other file is journaled done"+ );+ {+ let locked = endpoint.state.lock().unwrap();+ let mut paths: Vec<&str> = data_rows(&locked)+ .iter()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ paths.sort_unstable();+ paths.dedup();+ assert_eq!(paths, vec!["first.csv", "second.csv"], "{paths:?}");+ let skipped: Vec<&Value> = locked+ .catalog_docs+ .values()+ .filter(|doc| doc["path"].as_str() == Some("app.lock"))+ .collect();+ assert_eq!(+ skipped.len(),+ 1,+ "the vanished file is recorded: {skipped:?}"+ );+ assert_eq!(skipped[0]["status"], "skipped", "{}", skipped[0]);+ let reason = skipped[0]["reason"].as_str().unwrap_or_default();+ assert!(reason.contains("no longer exists"), "{reason}");+ assert!(reason.contains("disappeared"), "{reason}");+ assert!(+ !reason.contains("not in the frozen resume plan"),+ "{reason}"+ );+ }++ let (code, report) = run_index_report(config).expect("the rerun completes");+ assert_eq!(+ code, 0,+ "a file that is gone on the next walk is not still junk: {report:?}"+ );+}+#[test]fn completed_plan_sweeps_a_vanished_content_group() {let _guard = FAILPOINT_TEST_LOCKdiff --git a/engine/crates/xerj-autoindex/src/family_junk.rs b/engine/crates/xerj-autoindex/src/family_junk.rsnew file mode 100644index 000000000..63a66e859--- /dev/null+++ b/engine/crates/xerj-autoindex/src/family_junk.rs@@ -0,0 +1,389 @@+//! Per-family junk warnings for the autoindex run summary (#2370).+//!+//! Exit 3 already says some files were junk. It does not say that a whole+//! file type disappeared: four PDFs that all failed to parse print the same+//! "completed-with-junk" summary as one unreadable side file next to a healthy+//! corpus. The catalog lists each junked path, and that is the only place the+//! loss used to be visible.+//!+//! A family warns when its junked files are a **strict majority** of the files+//! classified as indexed or junk (`junk_files * 2 > files`). That includes+//! every file of the family (`extent: all`). A tie (1 of 2) stays on the+//! ordinary junk count. Stale-plan skips are neither indexed nor junk; they+//! already have their own summary and are not part of this ratio.+//!+//! `top_reason` is the junk reason shared by the most files of that family.+//! Equal counts break by Unicode code-point order of the reason, smaller+//! first — not by hash-map iteration. An empty reason is published as+//! `(no reason recorded)`.+//!+//! The array is omitted when nothing tripped it, so a clean run document keeps+//! the fields it had. Existing summary fields are not renamed. Exit codes are+//! not changed: ordinary junk stays exit 3, and a family-wide PDF worker+//! startup failure stays exit 1.++use serde::{Deserialize, Serialize};+use serde_json::Value;+use std::collections::BTreeMap;++use crate::state::{JunkFile, Plan};+use crate::sync::ManifestGroup;++/// `all` — every indexed-or-junk file of the family was junked.+/// `most` — junked files are a strict majority, and at least one file was indexed.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub(crate) enum FamilyJunkExtent {+ All,+ Most,+}++/// One family the run summary should name. Field names are the `--json` contract.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub(crate) struct FamilyJunkWarning {+ pub family: String,+ pub files: u64,+ pub junk_files: u64,+ pub indexed_files: u64,+ pub extent: FamilyJunkExtent,+ pub top_reason: String,+ pub top_reason_files: u64,+}++/// What an empty junk reason becomes in the summary. The catalog file row can+/// still carry an empty `reason`; the summary does not print a blank one.+pub(crate) const NO_REASON: &str = "(no reason recorded)";++impl FamilyJunkWarning {+ /// One human line. Starts with `warning:` so it is visible in the stdout+ /// summary and on the progress stream. Names the family, the counts, and+ /// the top reason.+ pub(crate) fn line(&self) -> String {+ let verb = if self.junk_files == 1 { "was" } else { "were" };+ let body = match self.extent {+ FamilyJunkExtent::All => format!(+ "{} family: all {} {verb} junked and not indexed",+ self.family,+ crate::estimate::count_phrase(self.files, "file", "files"),+ ),+ FamilyJunkExtent::Most => format!(+ "{} family: {} of {} {verb} junked and not indexed",+ self.family,+ crate::estimate::count_phrase(self.junk_files, "file", "files"),+ crate::estimate::count_phrase(self.files, "file", "files"),+ ),+ };+ format!(+ "warning: {body}; top reason ({}): {}",+ crate::estimate::count_phrase(self.top_reason_files, "file", "files"),+ self.top_reason,+ )+ }+}++/// Family key for a catalog format string. `pdf` and `pdf(gzip)` are one family.+pub(crate) fn family_of_format(format: &str) -> &str {+ let bare = format.strip_suffix("(gzip)").unwrap_or(format);+ if bare.is_empty() {+ "unknown"+ } else {+ bare+ }+}++#[derive(Default)]+struct Acc {+ indexed: u64,+ junk: u64,+ reasons: BTreeMap<String, u64>,+}++/// Warnings for one finished run.+///+/// `indexed_formats` are files that reached the index (catalog status+/// `indexed`), format strings with the optional `(gzip)` suffix.+/// `junk` is the junk/skipped list after shadowed entries have been dropped.+/// Stale-plan skips are ignored here.+pub(crate) fn warnings<'a>(+ indexed_formats: impl IntoIterator<Item = &'a str>,+ junk: impl IntoIterator<Item = &'a JunkFile>,+) -> Vec<FamilyJunkWarning> {+ let mut by_family: BTreeMap<String, Acc> = BTreeMap::new();+ for format in indexed_formats {+ let acc = by_family+ .entry(family_of_format(format).to_string())+ .or_default();+ acc.indexed = acc.indexed.saturating_add(1);+ }+ for junk_file in junk {+ if crate::is_stale_resume_skip(junk_file) {+ continue;+ }+ let acc = by_family+ .entry(family_of_format(&junk_file.format).to_string())+ .or_default();+ acc.junk = acc.junk.saturating_add(1);+ let reason = if junk_file.reason.is_empty() {+ NO_REASON.to_string()+ } else {+ junk_file.reason.clone()+ };+ *acc.reasons.entry(reason).or_default() += 1;+ }++ let mut out = Vec::new();+ for (family, acc) in by_family {+ let Some(extent) = extent_for(acc.indexed, acc.junk) else {+ continue;+ };+ let (top_reason, top_reason_files) = top_reason(&acc.reasons);+ let files = acc.indexed.saturating_add(acc.junk);+ out.push(FamilyJunkWarning {+ family,+ files,+ junk_files: acc.junk,+ indexed_files: acc.indexed,+ extent,+ top_reason,+ top_reason_files,+ });+ }+ out+}++/// The same classification [`warnings`] uses, read off a sealed generation:+/// groups are the indexed files, `plan.junk_files` is the junk list.+pub(crate) fn from_groups(plan: &Plan, groups: &[ManifestGroup]) -> Vec<FamilyJunkWarning> {+ let indexed: Vec<String> = groups+ .iter()+ .filter_map(|group| {+ let assignment = plan.files.get(&group.content_id)?;+ Some(if assignment.gzip {+ format!("{}(gzip)", assignment.family)+ } else {+ assignment.family.clone()+ })+ })+ .collect();+ warnings(indexed.iter().map(String::as_str), plan.junk_files.iter())+}++/// Lines for a run document that already carries `family_junk_warnings`.+/// Malformed entries are skipped; a missing field is no lines.+pub(crate) fn lines_from_summary(summary: &Value) -> Vec<String> {+ summary+ .get("family_junk_warnings")+ .and_then(Value::as_array)+ .map(|entries| {+ entries+ .iter()+ .filter_map(|entry| serde_json::from_value::<FamilyJunkWarning>(entry.clone()).ok())+ .map(|warning| warning.line())+ .collect()+ })+ .unwrap_or_default()+}++/// Strict majority, with the whole family called out separately.+/// `junk * 2 > indexed + junk`, and `junk > 0`.+fn extent_for(indexed: u64, junk: u64) -> Option<FamilyJunkExtent> {+ let files = indexed.saturating_add(junk);+ if junk == 0 || files == 0 {+ return None;+ }+ if junk == files {+ return Some(FamilyJunkExtent::All);+ }+ if junk.saturating_mul(2) > files {+ Some(FamilyJunkExtent::Most)+ } else {+ None+ }+}++/// Highest count wins. Ties break to the code-point-smaller reason.+fn top_reason(counts: &BTreeMap<String, u64>) -> (String, u64) {+ counts+ .iter()+ .max_by(|(reason_a, count_a), (reason_b, count_b)| {+ count_a.cmp(count_b).then_with(|| reason_b.cmp(reason_a))+ })+ .map(|(reason, count)| (reason.clone(), *count))+ .unwrap_or_else(|| (NO_REASON.to_string(), 0))+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::state::JunkFile;++ fn junk(format: &str, reason: &str) -> JunkFile {+ JunkFile {+ file_key: format!("k-{format}-{reason}"),+ rel: format!("f.{format}"),+ format: format.to_string(),+ status: "junk".into(),+ reason: reason.into(),+ bytes: 1,+ }+ }++ fn stale(format: &str) -> JunkFile {+ let mut file = junk(format, "not in the frozen resume plan");+ file.status = "stale-plan".into();+ file+ }++ #[test]+ fn all_junked_family_names_the_count_and_top_reason() {+ let junked = [+ junk("pdf", "could not start the isolated PDF parser"),+ junk("pdf", "could not start the isolated PDF parser"),+ junk("pdf(gzip)", "could not start the isolated PDF parser"),+ junk("pdf", "damaged trailer"),+ ];+ let warnings = warnings(["code", "code", "txt-prose"], junked.iter());+ assert_eq!(warnings.len(), 1, "{warnings:?}");+ let pdf = &warnings[0];+ assert_eq!(pdf.family, "pdf");+ assert_eq!(pdf.extent, FamilyJunkExtent::All);+ assert_eq!(pdf.files, 4);+ assert_eq!(pdf.junk_files, 4);+ assert_eq!(pdf.indexed_files, 0);+ assert_eq!(pdf.top_reason, "could not start the isolated PDF parser");+ assert_eq!(pdf.top_reason_files, 3);+ let line = pdf.line();+ assert!(line.starts_with("warning: pdf family:"), "{line}");+ assert!(line.contains("all 4 files"), "{line}");+ assert!(line.contains("were junked and not indexed"), "{line}");+ assert!(+ line.contains("top reason (3 files): could not start the isolated PDF parser"),+ "{line}"+ );+ // The indexed families are clean and must not appear.+ assert!(!line.contains("code"));+ assert!(!line.contains("txt-prose"));+ }++ #[test]+ fn mostly_junked_family_warns_at_a_strict_majority() {+ // 3 junk + 1 indexed = 3/4, strict majority, not the whole family.+ let junked = [+ junk("docx", "encrypted"),+ junk("docx", "encrypted"),+ junk("docx", "truncated"),+ ];+ let warnings = warnings(["docx", "code", "code"], junked.iter());+ assert_eq!(warnings.len(), 1, "{warnings:?}");+ let docx = &warnings[0];+ assert_eq!(docx.family, "docx");+ assert_eq!(docx.extent, FamilyJunkExtent::Most);+ assert_eq!((docx.files, docx.junk_files, docx.indexed_files), (4, 3, 1));+ assert_eq!(docx.top_reason, "encrypted");+ assert_eq!(docx.top_reason_files, 2);+ let line = docx.line();+ assert!(line.contains("docx family: 3 files of 4 files"), "{line}");+ assert!(line.contains("were junked"), "{line}");+ assert!(line.contains("top reason (2 files): encrypted"), "{line}");+ }++ #[test]+ fn a_clean_family_and_a_minority_of_junk_do_not_warn() {+ // code: 3 indexed, 0 junk. pdf: 1 junk of 3 (not a majority). html: 1 of 2 (tie).+ let junked = [junk("pdf", "damaged"), junk("html", "empty")];+ let warnings = warnings(+ ["code", "code", "code", "pdf", "pdf", "html"],+ junked.iter(),+ );+ assert!(+ warnings.is_empty(),+ "clean, minority, and tie families stay off the warning list: {warnings:?}"+ );+ assert!(lines_from_summary(&serde_json::json!({})).is_empty());+ assert!(lines_from_summary(&serde_json::json!({"files_junk": 2})).is_empty());+ }++ #[test]+ fn equal_top_reasons_break_by_code_point_order() {+ let junked = [+ junk("pdf", "zeta failure"),+ junk("pdf", "alpha failure"),+ junk("pdf", "zeta failure"),+ junk("pdf", "alpha failure"),+ ];+ let warnings = warnings(std::iter::empty(), junked.iter());+ assert_eq!(warnings.len(), 1);+ assert_eq!(warnings[0].extent, FamilyJunkExtent::All);+ assert_eq!(warnings[0].top_reason, "alpha failure");+ assert_eq!(warnings[0].top_reason_files, 2);+ }++ #[test]+ fn an_empty_reason_is_named_and_a_single_file_uses_the_singular() {+ let junked = [junk("png", "")];+ let warnings = warnings(std::iter::empty(), junked.iter());+ assert_eq!(warnings[0].top_reason, NO_REASON);+ assert_eq!(warnings[0].top_reason_files, 1);+ let line = warnings[0].line();+ assert!(line.contains("all 1 file was junked"), "{line}");+ assert!(+ line.contains("top reason (1 file): (no reason recorded)"),+ "{line}"+ );+ }++ #[test]+ fn stale_plan_skips_are_not_part_of_the_family() {+ // Three stale PDFs and no junk: nothing to warn about.+ let only_stale = [stale("pdf"), stale("pdf"), stale("pdf")];+ assert!(warnings(std::iter::empty(), only_stale.iter()).is_empty());++ // One real junk PDF beside stale skips: the stale files are not a+ // denominator, so the one junked PDF is the whole classified family.+ let mixed = [stale("pdf"), junk("pdf", "damaged")];+ let warnings = warnings(std::iter::empty(), mixed.iter());+ assert_eq!(warnings.len(), 1);+ assert_eq!(warnings[0].extent, FamilyJunkExtent::All);+ assert_eq!(warnings[0].files, 1);+ assert_eq!(warnings[0].top_reason, "damaged");+ }++ #[test]+ fn families_are_sorted_and_the_summary_array_round_trips() {+ let junked = [junk("pdf", "worker"), junk("csv", "header")];+ let warnings = warnings(["md"], junked.iter());+ assert_eq!(+ warnings+ .iter()+ .map(|warning| warning.family.as_str())+ .collect::<Vec<_>>(),+ vec!["csv", "pdf"]+ );+ let summary = serde_json::json!({+ "files_junk": 2,+ "family_junk_warnings": warnings,+ });+ // Existing fields stay put.+ assert_eq!(summary["files_junk"], 2);+ let lines = lines_from_summary(&summary);+ assert_eq!(lines.len(), 2);+ assert!(lines[0].contains("csv family"));+ assert!(lines[1].contains("pdf family"));+ // A non-object entry does not drop the well-formed ones.+ let mut dirty = summary;+ dirty["family_junk_warnings"]+ .as_array_mut()+ .unwrap()+ .push(serde_json::json!("not an object"));+ assert_eq!(lines_from_summary(&dirty).len(), 2);+ }++ #[test]+ fn two_of_three_is_most_and_two_of_four_is_not() {+ assert_eq!(extent_for(1, 2), Some(FamilyJunkExtent::Most));+ assert_eq!(extent_for(0, 2), Some(FamilyJunkExtent::All));+ assert_eq!(extent_for(2, 2), None, "a tie is not a majority");+ assert_eq!(extent_for(3, 1), None);+ assert_eq!(extent_for(5, 0), None);+ }+}diff --git a/engine/crates/xerj-autoindex/src/generation_catalog.rs b/engine/crates/xerj-autoindex/src/generation_catalog.rsindex 4b2cbcd77..33dd79f3f 100644--- a/engine/crates/xerj-autoindex/src/generation_catalog.rs+++ b/engine/crates/xerj-autoindex/src/generation_catalog.rs@@ -377,6 +377,13 @@ pub fn project_generation(if let Some(n) = desired.plan.pdf_worker_unavailable_count() {run_doc["pdf_worker_unavailable"] = json!(n);}+ // #2370: omitted when no family is entirely or mostly junked, so a clean+ // generation serializes the run document it always did.+ let family_warnings = crate::family_junk::from_groups(&desired.plan, &desired.groups);+ if !family_warnings.is_empty() {+ run_doc["family_junk_warnings"] = serde_json::to_value(&family_warnings)+ .expect("family junk warnings are strings and counts");+ }// #929: a generation that lacks a dataset says so in the one document every// reader of the corpus starts from (`Plan::refused_run_fields`).for (key, value) in desired.plan.refused_run_fields() {@@ -684,6 +691,8 @@ mod tests {"a corpus that indexed no source code must say so: {run}");assert_eq!(run["code_files_junked"], 1);+ // Indexed with zero records is not a junked family (#2370).+ assert!(run.get("family_junk_warnings").is_none(), "{run}");// The healthy shape of the same corpus.let desired = manifest(@@ -714,6 +723,102 @@ mod tests {(json!(1), json!(1), json!(0)),"{run}");+ // Zero-record indexed code is the #294 case, not a junked family.+ // A clean corpus must not grow a family-junk field (#2370).+ assert!(run.get("family_junk_warnings").is_none(), "{run}");+ }++ /// #2370: the run document names a family whose files were all junked, and+ /// one that was only mostly junked, and stays silent for a clean family.+ #[test]+ fn the_run_document_warns_for_all_junked_and_mostly_junked_families_only() {+ let mut plan = Plan {+ datasets: vec![dataset()],+ ..Plan::default()+ };+ let mut code = assignment("app.py");+ code.family = "code".into();+ let mut html = assignment("index.html");+ html.family = "html".into();+ plan.files.insert("code".into(), code);+ plan.files.insert("html".into(), html);+ plan.junk_files.push(JunkFile {+ file_key: "pdf-1".into(),+ rel: "a.pdf".into(),+ format: "pdf".into(),+ status: "junk".into(),+ reason: "could not start the isolated PDF parser".into(),+ bytes: 4,+ });+ plan.junk_files.push(JunkFile {+ file_key: "pdf-2".into(),+ rel: "b.pdf".into(),+ format: "pdf(gzip)".into(),+ status: "junk".into(),+ reason: "could not start the isolated PDF parser".into(),+ bytes: 4,+ });+ plan.junk_files.push(JunkFile {+ file_key: "html-junk".into(),+ rel: "broken.html".into(),+ format: "html".into(),+ status: "junk".into(),+ reason: "no records extracted".into(),+ bytes: 2,+ });+ plan.junk_files.push(JunkFile {+ file_key: "html-junk-2".into(),+ rel: "broken-2.html".into(),+ format: "html".into(),+ status: "junk".into(),+ reason: "no records extracted".into(),+ bytes: 2,+ });+ let desired = manifest(+ 1,+ "generation-1",+ plan,+ vec![+ group("code", "app.py", vec![]),+ group("html", "index.html", vec![]),+ ],+ );+ let base = committed(manifest(0, "generation-0", Plan::default(), vec![]));+ let projection = project_generation(+ &base,+ &desired,+ &metadata("generation-1"),+ &stats(),+ &BTreeMap::new(),+ &BTreeSet::new(),+ )+ .unwrap();+ let run = &projection.documents["run:generation-1"];+ // Prior fields are still there.+ assert_eq!(run["files_junk"], 4);+ assert_eq!(run["code_files_indexed"], 1);+ let warnings = run["family_junk_warnings"]+ .as_array()+ .expect("all-junked and mostly-junked families are named");+ assert_eq!(warnings.len(), 2, "{warnings:?}");+ assert_eq!(warnings[0]["family"], "html");+ assert_eq!(warnings[0]["extent"], "most");+ assert_eq!(warnings[0]["files"], 3);+ assert_eq!(warnings[0]["junk_files"], 2);+ assert_eq!(warnings[0]["indexed_files"], 1);+ assert_eq!(warnings[0]["top_reason"], "no records extracted");+ assert_eq!(warnings[0]["top_reason_files"], 2);+ assert_eq!(warnings[1]["family"], "pdf");+ assert_eq!(warnings[1]["extent"], "all");+ assert_eq!(warnings[1]["files"], 2);+ assert_eq!(warnings[1]["junk_files"], 2);+ assert_eq!(warnings[1]["indexed_files"], 0);+ assert_eq!(+ warnings[1]["top_reason"],+ "could not start the isolated PDF parser"+ );+ // code was indexed and not junked.+ assert!(warnings.iter().all(|w| w["family"] != "code"));}/// #971: this was `every_current_document_moves_to_latest_catalog_generation`diff --git a/engine/crates/xerj-autoindex/src/ignore_rules.rs b/engine/crates/xerj-autoindex/src/ignore_rules.rsindex d009f4a0e..2b4fd5733 100644--- a/engine/crates/xerj-autoindex/src/ignore_rules.rs+++ b/engine/crates/xerj-autoindex/src/ignore_rules.rs@@ -57,11 +57,26 @@//! surprise, and it is not visible in the folder being indexed. `.gitignore`,//! `.xerjignore` and `.git/info/exclude` all live in the tree the user pointed//! at, so they are.+//!+//! # Bracket directory names+//!+//! Git reads `[locale]` as a character class (one of those letters), so a+//! pattern written as the Next.js directory `app/[locale]/` does not name+//! that directory. After each ignore-file line that contains an unescaped+//! bracket, the same line is added again with those brackets escaped. The+//! original pattern stays, so a real class such as `*.[ch]` still matches+//! `a.c` and `a.h`. The extra pattern is what makes the literal directory+//! follow the line the author wrote (`!app/[locale]/**` re-includes+//! `app/[locale]/…`, and `app/[locale]/` ignores it). A route-like class is+//! named in the ignore summary. A code file that is still skipped is named+//! with the rule, so a miss is not only a count.use ignore::gitignore::{Gitignore, GitignoreBuilder, Glob};use ignore::Match;use std::collections::BTreeMap;use std::ffi::OsStr;+use std::fs::File;+use std::io::{BufRead, BufReader};use std::path::{Path, PathBuf};/// Hidden names start with `.` even when they are not valid UTF-8.@@ -172,6 +187,10 @@ pub const HIDDEN_OUTSIDE_ROOT_RULE: &str = "symlink:outside-root+hidden";pub const NOT_FOLLOWED_RULE: &str = "symlink:not-followed";const MAX_WARNINGS: usize = 20;+/// Skipped code files named in the summary. The rest are a count: a+/// `*.log` tree is not code, and a `*.ts` rule over a huge tree must not+/// print every path.+const CODE_SKIP_SHOWN: usize = 8;/// Directory entries the `--dry-run` deep count may examine in total. A/// pathological `node_modules` is not allowed to turn "tell me the plan" into/// minutes of stat storm on a laptop; past the budget the count is reported as@@ -251,6 +270,10 @@ pub struct IgnoreReport {/// this into the `--walk-only` per-folder summary; it is not a second walk.pub by_top: BTreeMap<String, BTreeMap<String, RuleCount>>,pub files_skipped: u64,+ /// `rel — rule` for skipped code files, capped at [`CODE_SKIP_SHOWN`].+ pub code_skipped: Vec<String>,+ /// Code files skipped past [`CODE_SKIP_SHOWN`].+ pub code_skipped_omitted: u64,pub dirs_pruned: u64,/// Non-hidden files inside the pruned directories — see/// [`IgnoreStack::count_inside`] for exactly what is and is not counted.@@ -283,6 +306,8 @@ impl Default for IgnoreReport {by_rule: BTreeMap::new(),by_top: BTreeMap::new(),files_skipped: 0,+ code_skipped: Vec::new(),+ code_skipped_omitted: 0,dirs_pruned: 0,files_inside_pruned_dirs: 0,deep_count_complete: true,@@ -405,6 +430,16 @@ impl IgnoreReport {plural((ranked.len() - SHOWN) as u64)));}+ for skipped in &self.code_skipped {+ out.push(format!("ignore rules: skipped code file {skipped}"));+ }+ if self.code_skipped_omitted > 0 {+ out.push(format!(+ "ignore rules: … and {} more skipped code file{}",+ self.code_skipped_omitted,+ plural(self.code_skipped_omitted)+ ));+ }out}}@@ -546,7 +581,7 @@ impl IgnoreStack {continue;}let mut builder = GitignoreBuilder::new(&dir);- let _ = builder.add(&path);+ let _ = add_gitignore_file(&mut builder, &path);let Ok(gi) = builder.build() else { continue };if gi.is_empty() {continue;@@ -612,9 +647,13 @@ impl IgnoreStack {return None;}let mut builder = GitignoreBuilder::new(dir);- if let Some(err) = builder.add(&file) {+ let (err, notes) = add_gitignore_file(&mut builder, &file);+ if let Some(err) = err {self.warn(format!("{}: {err}", file.display()));}+ for note in notes {+ self.warn(format!("{}: {note}", file.display()));+ }match builder.build() {Ok(gi) => {self.report.ignore_files_read += 1;@@ -898,9 +937,26 @@ impl IgnoreStack {.entry(label.to_string()).or_default().files += 1;+ self.note_code_skip(path, label);self.note_top(path, false, label, |count| count.files += 1);}+ /// A skipped `.ts`/`.tsx` (and every other code extension) is named. A+ /// directory prune never reaches here: the walk does not open the+ /// children, so the summary can only name the directory.+ fn note_code_skip(&mut self, path: &Path, label: &str) {+ let ext = path.extension().and_then(|ext| ext.to_str()).unwrap_or("");+ if !crate::extract::code::is_code_ext(ext) {+ return;+ }+ let shown = format!("{} — {label}", self.rel(path));+ if self.report.code_skipped.len() < CODE_SKIP_SHOWN {+ self.report.code_skipped.push(shown);+ } else {+ self.report.code_skipped_omitted += 1;+ }+ }+fn record_dir(&mut self, path: &Path, label: &str) {let inside = self.count_inside(path);let deep = self.opts.deep_count;@@ -912,6 +968,12 @@ impl IgnoreStack {}self.report.dirs_pruned += 1;self.report.files_inside_pruned_dirs += inside;+ let rel = self.rel(path);+ if rel.contains('[') || rel.contains(']') {+ self.warn(format!(+ "pruned directory {rel} ({label}); files under it are not indexed"+ ));+ }self.note_top(path, true, label, |count| {count.dirs += 1;count.files_inside_pruned_dirs += inside;@@ -988,6 +1050,159 @@ impl IgnoreStack {}}+/// Read one ignore file the way `GitignoreBuilder::add` does, then add a+/// literal-bracket twin of every line that contains an unescaped `[` or `]`.+///+/// The twin is inserted immediately after its source line, so a later line+/// in the same file still wins. `*.[ch]` keeps matching `a.c`; the twin+/// only additionally matches a name that contains the brackets.+///+/// Returns the first error (other lines are still added) and one note per+/// route-like class, for the ignore summary. The error is a string because+/// `ignore::Error`'s path and line tagging is crate-private; the line number+/// is in the text instead.+fn add_gitignore_file(+ builder: &mut GitignoreBuilder,+ path: &Path,+) -> (Option<String>, Vec<String>) {+ let file = match File::open(path) {+ Ok(file) => file,+ Err(err) => return (Some(err.to_string()), Vec::new()),+ };+ let reader = BufReader::new(file);+ let mut first_err: Option<String> = None;+ let mut notes = Vec::new();+ for (i, line) in reader.lines().enumerate() {+ let lineno = i + 1;+ let line = match line {+ Ok(line) => line,+ Err(err) => {+ if first_err.is_none() {+ first_err = Some(format!("line {lineno}: {err}"));+ }+ break;+ }+ };+ // `GitignoreBuilder::add` strips a leading BOM on line 1 only.+ let line = if i == 0 {+ line.trim_start_matches('\u{feff}').to_string()+ } else {+ line+ };+ if let Err(err) = builder.add_line(Some(path.to_path_buf()), &line) {+ if first_err.is_none() {+ first_err = Some(format!("line {lineno}: {err}"));+ }+ }+ // A `#` comment is not a pattern. Twisting its brackets would turn+ // the comment into a pattern (`\#…` is not a comment).+ if line.starts_with('#') {+ continue;+ }+ let Some(twin) = literal_bracket_twin(&line) else {+ continue;+ };+ if let Some(note) = route_class_note(&line) {+ notes.push(note);+ }+ if let Err(err) = builder.add_line(Some(path.to_path_buf()), &twin) {+ if first_err.is_none() {+ first_err = Some(format!("line {lineno}: {err}"));+ }+ }+ }+ (first_err, notes)+}++/// The same line with every unescaped `[` and `]` prefixed by `\`.+///+/// `None` when the line has no unescaped bracket, including a line that is+/// already escaped (`app/\[locale\]/`).+fn literal_bracket_twin(line: &str) -> Option<String> {+ let mut out = String::with_capacity(line.len() + 4);+ let mut chars = line.chars().peekable();+ let mut changed = false;+ while let Some(c) = chars.next() {+ if c == '\\' {+ out.push('\\');+ if let Some(next) = chars.next() {+ out.push(next);+ }+ continue;+ }+ if c == '[' || c == ']' {+ out.push('\\');+ changed = true;+ }+ out.push(c);+ }+ changed.then_some(out)+}++/// Note for a class that is shaped like a route segment (`[locale]`,+/// `[...slug]`, or `[id]` inside a path). `*.[ch]` is a real class and+/// stays quiet: it has no `/`, and the body is shorter than three+/// characters.+fn route_class_note(line: &str) -> Option<String> {+ let has_slash = line.contains('/');+ let bodies = route_like_class_bodies(line, has_slash);+ if bodies.is_empty() {+ return None;+ }+ let names = bodies+ .iter()+ .map(|body| format!("[{body}]"))+ .collect::<Vec<_>>()+ .join(", ");+ Some(format!(+ "`{line}` reads {names} as a character class, not as a directory name. Also applying the pattern to that literal name, so a route folder such as [locale] is not dropped for this reason"+ ))+}++fn route_like_class_bodies(line: &str, pattern_has_slash: bool) -> Vec<String> {+ let mut bodies = Vec::new();+ let mut chars = line.chars().peekable();+ while let Some(c) = chars.next() {+ if c == '\\' {+ chars.next();+ continue;+ }+ if c != '[' {+ continue;+ }+ let mut body = String::new();+ let mut closed = false;+ while let Some(next) = chars.next() {+ if next == '\\' {+ if let Some(escaped) = chars.next() {+ body.push(escaped);+ }+ continue;+ }+ if next == ']' {+ closed = true;+ break;+ }+ body.push(next);+ }+ if closed && is_route_like_class(&body, pattern_has_slash) {+ bodies.push(body);+ }+ }+ bodies+}++fn is_route_like_class(body: &str, pattern_has_slash: bool) -> bool {+ if body.contains('-') || body.contains('!') || body.contains('^') {+ return false;+ }+ // `[[...slug]]` yields a first class body of `[...slug`.+ let ident = body.strip_prefix('[').unwrap_or(body);+ let ident = ident.strip_prefix("...").unwrap_or(ident);+ let min = if pattern_has_slash { 2 } else { 3 };+ ident.len() >= min && ident.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')+}+#[cfg(test)]mod tests {use super::*;@@ -1576,4 +1791,206 @@ mod tests {report.summary_lines());}++ const ROUTE_TSX: &str =+ "app/[locale]/booking/[token]/temple-selection/TempleSelectionWrapper.tsx";++ /// #2357: a Next.js route directory is the literal name `[locale]`.+ /// Git reads that text as a character class, so a whitelist written the+ /// way the directory is named does not re-include it and `*` keeps the+ /// sources out of the index. The literal twin re-includes them. A plain+ /// `*.[ch]` class is unchanged.+ #[test]+ fn a_bracket_route_whitelist_includes_the_literal_directory() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(+ root,+ ".gitignore",+ "*\n!app/\n!app/[locale]/\n!app/[locale]/**\n!README.md\n!lib/\n!lib/**\n!data.json\n",+ );+ write(+ root,+ ROUTE_TSX,+ "export function TempleSelectionWrapper() {}\n",+ );+ write(root, "README.md", "# notes\n");+ write(root, "lib/items.json", "{}\n");+ write(root, "data.json", "{}\n");+ write(root, "src/plain.ts", "export const n = 1;\n");++ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert!(+ kept.iter().any(|p| p == ROUTE_TSX),+ "bracketed tsx must be indexed, got {kept:?}"+ );+ assert!(kept.iter().any(|p| p == "README.md"), "{kept:?}");+ assert!(kept.iter().any(|p| p == "data.json"), "{kept:?}");+ assert!(kept.iter().any(|p| p == "lib/items.json"), "{kept:?}");+ assert!(+ !kept.iter().any(|p| p == "src/plain.ts"),+ "`*` still ignores a path the whitelist did not name: {kept:?}"+ );+ let lines = report.summary_lines().join("\n");+ assert!(+ lines.contains("character class"),+ "the pattern must be explained: {lines}"+ );+ assert!(+ lines.contains("[locale]"),+ "the directory name must be in the note: {lines}"+ );+ }++ /// No ignore file at all: the brackets are only in the path, and the+ /// walk must keep the source. This is the case a default Next.js+ /// `.gitignore` (which never names `[locale]`) already had.+ #[test]+ fn a_bracket_route_with_no_ignore_file_is_walked() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(root, ROUTE_TSX, "export const getPujaName = () => \"a\";\n");+ write(root, "src/plain.ts", "export const n = 1;\n");+ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert!(kept.iter().any(|p| p == ROUTE_TSX), "{kept:?}");+ assert!(kept.iter().any(|p| p == "src/plain.ts"), "{kept:?}");+ assert!(+ !report+ .summary_lines()+ .iter()+ .any(|l| l.contains("character class")),+ "{:?}",+ report.summary_lines()+ );+ }++ /// An author who already escaped the brackets meant the literal+ /// directory. The twin must not unescape that into a class, and the+ /// directory stays ignored.+ #[test]+ fn an_escaped_bracket_pattern_still_ignores_the_literal_directory() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(root, ".gitignore", "app/\\[locale\\]/\n");+ write(root, ROUTE_TSX, "export const n = 1;\n");+ write(root, "src/plain.ts", "export const n = 1;\n");+ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert!(+ !kept.iter().any(|p| p.contains("[locale]")),+ "escaped pattern must ignore the route dir: {kept:?}"+ );+ assert!(kept.iter().any(|p| p == "src/plain.ts"), "{kept:?}");+ assert!(+ report+ .summary_lines()+ .iter()+ .any(|l| l.contains("pruned directory") && l.contains("[locale]")),+ "{:?}",+ report.summary_lines()+ );+ }++ /// `*.[ch]` is a character class. The literal twin must not stop it+ /// matching `.c` and `.h`, and a `.rs` file stays.+ #[test]+ fn a_character_class_extension_still_ignores_those_extensions() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(root, ".gitignore", "*.[ch]\n");+ write(root, "foo.c", "int c;\n");+ write(root, "foo.h", "int h;\n");+ write(root, "foo.rs", "fn r() {}\n");+ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert_eq!(kept, vec!["foo.rs"], "{kept:?}");+ assert!(+ !report+ .summary_lines()+ .iter()+ .any(|l| l.contains("character class")),+ "a short class with no slash is not a route folder: {:?}",+ report.summary_lines()+ );+ let lines = report.summary_lines().join("\n");+ assert!(lines.contains("skipped code file foo.c"), "{lines}");+ assert!(lines.contains("skipped code file foo.h"), "{lines}");+ assert!(+ !lines.contains("foo.rs"),+ "the file that was kept is not a skip: {lines}"+ );+ }++ /// A code file the rules still skip is named, with the rule. `*.tsx`+ /// has no bracket, so this is the report the walk owes when the file+ /// really is excluded.+ #[test]+ fn a_skipped_tsx_file_is_named_with_its_rule() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(root, ".gitignore", "*.tsx\n");+ write(root, ROUTE_TSX, "export const n = 1;\n");+ write(root, "keep.ts", "export const n = 1;\n");+ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert_eq!(kept, vec!["keep.ts"], "{kept:?}");+ let lines = report.summary_lines().join("\n");+ assert!(+ lines.contains("skipped code file") && lines.contains(ROUTE_TSX),+ "{lines}"+ );+ assert!(+ lines.contains(".gitignore:*.tsx"),+ "the rule must be on the line: {lines}"+ );+ }++ /// The twin is inserted after its source line, not at the end of the+ /// file. A later `*.tsx` still excludes the route source; a `.ts` the+ /// whitelist named stays.+ #[test]+ fn a_later_rule_still_beats_the_literal_bracket_twin() {+ let dir = tempfile::TempDir::new().unwrap();+ let root = dir.path();+ write(+ root,+ ".gitignore",+ "*\n!app/\n!app/[locale]/\n!app/[locale]/**\n*.tsx\n",+ );+ write(root, ROUTE_TSX, "export const n = 1;\n");+ write(root, "app/[locale]/keep.ts", "export const n = 1;\n");+ let (kept, report) = survivors(root, IgnoreOptions::default());+ assert!(kept.iter().any(|p| p == "app/[locale]/keep.ts"), "{kept:?}");+ assert!(+ !kept.iter().any(|p| p == ROUTE_TSX),+ "a later *.tsx must still win: {kept:?}"+ );+ let lines = report.summary_lines().join("\n");+ assert!(+ lines.contains("skipped code file") && lines.contains(ROUTE_TSX),+ "{lines}"+ );+ assert!(lines.contains(".gitignore:*.tsx"), "{lines}");+ }++ #[test]+ fn literal_bracket_twin_escapes_unescaped_brackets_only() {+ assert_eq!(+ literal_bracket_twin("!app/[locale]/**").as_deref(),+ Some("!app/\\[locale\\]/**")+ );+ assert_eq!(literal_bracket_twin("app/\\[locale\\]/"), None);+ assert_eq!(+ literal_bracket_twin("*.[ch]").as_deref(),+ Some("*.\\[ch\\]")+ );+ // Comments are filtered by `add_gitignore_file` before this runs.+ // The helper itself only escapes brackets.+ assert_eq!(+ literal_bracket_twin("# [locale]").as_deref(),+ Some("# \\[locale\\]")+ );+ assert!(route_class_note("*.[ch]").is_none());+ assert!(route_class_note("app/[locale]/**")+ .unwrap()+ .contains("[locale]"));+ assert!(route_class_note("app/[id]/page.tsx").is_some());+ }}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 3eb3cf34c..fba68e4af 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -16,6 +16,7 @@ pub mod detect;pub mod esclient;pub mod estimate;pub mod extract;+mod family_junk;pub mod feedback;pub mod flood;pub mod gain;@@ -50,6 +51,7 @@ pub mod resources;pub mod search;pub mod sniff;pub mod source;+pub mod staleness;pub mod state;mod sync;mod sync_executor;@@ -1487,10 +1489,203 @@ const STALE_RESUME_PLAN_FILE_REASON: &str =published for it. Re-run with --fresh to rebuild the plan in place and include it (ids stay \idempotent), or index it under a new --state-dir and --prefix";-fn is_stale_resume_skip(junk: &JunkFile) -> bool {+pub(crate) fn is_stale_resume_skip(junk: &JunkFile) -> bool {junk.status == STALE_RESUME_PLAN_STATUS || junk.reason.contains("not in the frozen resume plan")}+/// How many appeared-file paths the stale-resume-plan summary names.+///+/// The rest are a count. Five is enough to see what the resume left out+/// without turning the summary into a directory listing (#2372). The mid-run+/// note still lists up to [`REFUSAL_LIST_CAP`].+const STALE_RESUME_SUMMARY_CAP: usize = 5;++/// Recovery sentence shared by the `ok:` headline, the path list, and the JSON field.+///+/// One sentence, so the summary that names the paths also says `--fresh`+/// right there (#2372).+const STALE_RESUME_SUMMARY_ACTION: &str =+ "Re-run with --fresh to rebuild the plan in place and include them (ids stay idempotent)";++/// Relative paths the frozen resume plan did not cover, in summary order.+///+/// Same predicate as the `skipped_appeared` count ([`is_stale_resume_skip`]),+/// so a legacy journal that stored these as status `skipped` is listed too.+/// Sorted so two runs over the same folder name the same paths. The count+/// stays the full length; this vec is not capped.+fn stale_resume_appeared_paths(junk: &[&JunkFile]) -> Vec<String> {+ let mut paths: Vec<&str> = junk+ .iter()+ .filter(|jf| is_stale_resume_skip(jf))+ .map(|jf| jf.rel.as_str())+ .collect();+ paths.sort_unstable();+ paths.into_iter().map(str::to_string).collect()+}++/// Human lines for the stale-resume-plan summary.+///+/// The `ok:` headline stays one line. These sit under it and name the files,+/// because a count alone sent an agent back to `--help` to find out what was+/// skipped (#2372). Empty when nothing appeared, so a clean run's summary is+/// unchanged.+fn stale_resume_summary_lines(paths: &[String]) -> Vec<String> {+ if paths.is_empty() {+ return Vec::new();+ }+ let shown = paths.len().min(STALE_RESUME_SUMMARY_CAP);+ let rest = paths.len() - shown;+ let mut lines = vec![format!(+ "not indexed — {} appeared after the resume plan was frozen. {STALE_RESUME_SUMMARY_ACTION}:",+ estimate::count_phrase(paths.len() as u64, "file", "files")+ )];+ for path in &paths[..shown] {+ lines.push(format!(" {path}"));+ }+ if rest > 0 {+ lines.push(format!(" … and {rest} more"));+ }+ lines+}++/// Additive summary fields for the same list.+///+/// `skipped_appeared` stays the count and is not in this map. Present only+/// when files appeared. `skipped_appeared_not_listed` is present only when+/// the list was capped, so a reader that finds it knows the summary did not+/// name every path.+fn stale_resume_summary_json(paths: &[String]) -> Map<String, Value> {+ if paths.is_empty() {+ return Map::new();+ }+ let shown = paths.len().min(STALE_RESUME_SUMMARY_CAP);+ let mut fields = Map::new();+ fields.insert("skipped_appeared_files".into(), json!(&paths[..shown]));+ let rest = paths.len() - shown;+ if rest > 0 {+ fields.insert("skipped_appeared_not_listed".into(), json!(rest));+ }+ fields.insert(+ "skipped_appeared_action".into(),+ json!(STALE_RESUME_SUMMARY_ACTION),+ );+ fields+}++#[cfg(test)]+mod stale_resume_summary_tests {+ use super::{+ is_stale_resume_skip, stale_resume_appeared_paths, stale_resume_summary_json,+ stale_resume_summary_lines, JunkFile, STALE_RESUME_PLAN_FILE_REASON,+ STALE_RESUME_PLAN_STATUS, STALE_RESUME_SUMMARY_ACTION, STALE_RESUME_SUMMARY_CAP,+ };++ fn junk(rel: &str, status: &str, reason: &str) -> JunkFile {+ JunkFile {+ file_key: rel.into(),+ rel: rel.into(),+ format: "unknown".into(),+ status: status.into(),+ reason: reason.into(),+ bytes: 1,+ }+ }++ fn appeared(rel: &str) -> JunkFile {+ junk(rel, STALE_RESUME_PLAN_STATUS, STALE_RESUME_PLAN_FILE_REASON)+ }++ #[test]+ fn nothing_appeared_adds_no_summary() {+ let ordinary = junk("notes.bin", "junk", "binary content");+ let refs = [&ordinary];+ assert!(stale_resume_appeared_paths(&refs).is_empty());+ assert!(stale_resume_summary_lines(&[]).is_empty());+ assert!(stale_resume_summary_json(&[]).is_empty());+ }++ #[test]+ fn one_file_is_named_and_fresh_is_on_the_same_summary() {+ let file = appeared("notes/new file.md");+ let paths = stale_resume_appeared_paths(&[&file]);+ assert_eq!(paths, vec!["notes/new file.md".to_string()]);+ let lines = stale_resume_summary_lines(&paths);+ assert_eq!(lines.len(), 2, "{lines:?}");+ assert!(lines[0].contains("1 file"), "{lines:?}");+ assert!(lines[0].contains("--fresh"), "{lines:?}");+ assert!(lines[0].contains(STALE_RESUME_SUMMARY_ACTION), "{lines:?}");+ assert_eq!(lines[1], " notes/new file.md");+ assert!(+ !lines.iter().any(|line| line.contains("… and")),+ "{lines:?}"+ );++ let doc = stale_resume_summary_json(&paths);+ assert_eq!(doc["skipped_appeared_files"], serde_json::json!(paths));+ assert!(doc.get("skipped_appeared_not_listed").is_none(), "{doc:?}");+ assert!(+ doc["skipped_appeared_action"]+ .as_str()+ .unwrap()+ .contains("--fresh"),+ "{doc:?}"+ );+ assert!(doc.get("skipped_appeared").is_none(), "{doc:?}");+ }++ /// The summary names the first five paths, sorted, and counts the rest.+ /// A legacy journal row (status `skipped`, the frozen-plan sentence) is+ /// included; ordinary junk is not.+ #[test]+ fn the_summary_names_five_paths_and_counts_the_rest() {+ let mut files: Vec<JunkFile> = ["m.md", "a.md", "q.md", "b.md", "z.md", "c.md", "d.md"]+ .into_iter()+ .map(appeared)+ .collect();+ files.push(junk(+ "legacy.md",+ "skipped",+ "not in the frozen resume plan, so nothing was published",+ ));+ files.push(junk("logo.png", "junk", "binary content (png)"));+ let refs: Vec<&JunkFile> = files.iter().collect();+ assert!(is_stale_resume_skip(&files[7]));+ assert!(!is_stale_resume_skip(files.last().unwrap()));++ let paths = stale_resume_appeared_paths(&refs);+ assert_eq!(+ paths,+ vec![+ "a.md",+ "b.md",+ "c.md",+ "d.md",+ "legacy.md",+ "m.md",+ "q.md",+ "z.md",+ ]+ );+ let lines = stale_resume_summary_lines(&paths);+ assert_eq!(lines.len(), STALE_RESUME_SUMMARY_CAP + 2, "{lines:?}");+ assert!(lines[0].contains("8 files"), "{lines:?}");+ assert!(lines[0].contains("--fresh"), "{lines:?}");+ for (index, path) in paths.iter().take(STALE_RESUME_SUMMARY_CAP).enumerate() {+ assert_eq!(lines[index + 1], format!(" {path}"));+ }+ assert_eq!(lines.last().unwrap(), " … and 3 more");+ assert!(!lines.iter().any(|line| line.contains("q.md")), "{lines:?}");++ let doc = stale_resume_summary_json(&paths);+ assert_eq!(+ doc["skipped_appeared_files"],+ serde_json::json!(paths[..STALE_RESUME_SUMMARY_CAP])+ );+ assert_eq!(doc["skipped_appeared_not_listed"], 3);+ assert_eq!(doc["skipped_appeared_action"], STALE_RESUME_SUMMARY_ACTION);+ }+}+/// What a run says when the duplicate-alias sweep failed but the corpus is/// indexed.///@@ -1602,8 +1797,7 @@ pub(crate) fn outcome_headline(let files = estimate::count_phrase(stale_plan_files, "file", "files");let mut line = format!("ok: indexed and searchable — exit 3 ({STALE_RESUME_PLAN_REASON}): {files} skipped \- because the resumed plan is stale and were not indexed. Re-run with --fresh to \- rebuild the plan in place and include them (ids stay idempotent)"+ because the resumed plan is stale and were not indexed. {STALE_RESUME_SUMMARY_ACTION}");if junk_files > 0 || junk_records > 0 {let mut recorded = Vec::new();@@ -1696,6 +1890,12 @@ fn finish_generated_progress(pr: &Progress, code: i32, summary: &Value, skipped_if let Some(warning) = coverage.warning() {pr.note(&warning);}+ // #2370: a family that was entirely or mostly junked is a warning, not a+ // count the reader has to notice. `warn` still prints under `--quiet`.+ let family_lines = family_junk::lines_from_summary(summary);+ for line in &family_lines {+ pr.warn(line);+ }let mut extra = vec![("files", count("files_indexed")),("records", count("records_total")),@@ -1725,6 +1925,9 @@ fn finish_generated_progress(pr: &Progress, code: i32, summary: &Value, skipped_if skipped_dataless > 0 {extra.push(("skipped_dataless", skipped_dataless));}+ if !family_lines.is_empty() {+ extra.push(("family_junk_warnings", family_lines.len() as u64));+ }extra.extend(coverage.fields());note_worker_address_space_cap(pr);let pdf_down = count("pdf_worker_unavailable");@@ -1992,6 +2195,8 @@ fn begin_non_graph_generation(keys: inventory.keys.clone(),digests: inventory.digests.clone(),duplicates: inventory.duplicates.clone(),+ // Skips are already on `plan`. Sealing must not see them as content.+ unreadable: Vec::new(),};// #971: the prior committed generation's snapshot is the reuse source for// sealing. A file whose content digest and preparation identity match its@@ -2015,9 +2220,21 @@ fn begin_non_graph_generation(.iter().zip(sealed.digests.iter()).filter_map(|(file, digest)| {- content::verify(&file.path, file.size, digest)- .err()- .map(|error| (file.path.clone(), format!("{error:#}")))+ let error = content::verify(&file.path, file.size, digest).err()?;+ // A lock is not a content change. The snapshot reuses the+ // previous generation's blob when the digest and size still+ // match, without opening the file (#2352). Quarantining+ // here would delete documents that are only temporarily+ // unreadable. A lock whose size no longer matches falls+ // through to the copy, which reports SourceChanged and is+ // skipped on the next attempt.+ if error+ .downcast_ref::<content::SourceChanged>()+ .is_some_and(|changed| changed.locked)+ {+ return None;+ }+ Some((file.path.clone(), format!("{error:#}")))}).collect();let mut quarantined = false;@@ -3021,6 +3238,39 @@ fn phase_b_start_hook(state_dir: &Path) {#[inline]fn phase_b_start_hook(_state_dir: &Path) {}+/// A test-only action run once immediately before the hash, after the walk+/// has stored every [`walk::FileEntry`]. That is the window where a lock file+/// disappears between enumeration and the first open (#2352). Keyed on the+/// state directory for the same reason as [`PHASE_B_START_HOOK`].+#[cfg(test)]+type BeforeHashHook = (std::path::PathBuf, Box<dyn FnOnce() + Send>);+#[cfg(test)]+static BEFORE_HASH_HOOK: Mutex<Option<BeforeHashHook>> = Mutex::new(None);++#[cfg(test)]+pub(crate) fn arm_before_hash_hook(state_dir: &Path, action: impl FnOnce() + Send + 'static) {+ *BEFORE_HASH_HOOK.lock().unwrap_or_else(|p| p.into_inner()) =+ Some((state_dir.to_path_buf(), Box::new(action)));+}++#[cfg(test)]+fn before_hash_hook(state_dir: &Path) {+ let action = {+ let mut armed = BEFORE_HASH_HOOK.lock().unwrap_or_else(|p| p.into_inner());+ match armed.as_ref() {+ Some((dir, _)) if dir == state_dir => armed.take().map(|(_, action)| action),+ _ => None,+ }+ };+ if let Some(action) = action {+ action();+ }+}++#[cfg(not(test))]+#[inline]+fn before_hash_hook(_state_dir: &Path) {}+/// Arm [`replacement_failpoint`] for the run that uses `state_dir`, once.#[cfg(test)]pub(crate) fn arm_replacement_failpoint(boundary: u8, state_dir: &Path) {@@ -3813,7 +4063,10 @@ fn scan_file(match sniff::sniff(path) {Ok(s) => s,Err(e) => {- out.junk = Some(("junk".into(), format!("unreadable: {e}")));+ out.junk = Some(match content::unreadable_junk_fields(&e) {+ Some((status, reason)) => (status.into(), reason),+ None => ("junk".into(), format!("unreadable: {e}")),+ });return out;}}@@ -4857,6 +5110,82 @@ mod phase_a_grouping_tests {assert_eq!(plan.datasets[0].slug, "docs");assert_eq!(plan.datasets[0].file_count, 4);}++ /// #2357: `.tsx` under Next.js route directories (`[locale]`, `[token]`)+ /// is code in the per-scope docs dataset. A `.gitignore` that names those+ /// directories the way they appear on disk (`!app/[locale]/**`) must+ /// re-include them; git would read `[locale]` as a character class and+ /// leave the file out.+ #[test]+ fn bracketed_tsx_routes_land_in_the_docs_dataset() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let tsx = root+ .join("app")+ .join("[locale]")+ .join("booking")+ .join("[token]")+ .join("temple-selection");+ std::fs::create_dir_all(&tsx).unwrap();+ std::fs::write(+ tsx.join("TempleSelectionWrapper.tsx"),+ "export function TempleSelectionWrapper() {\n\+ const getPujaName = () => \"pushpanjali\";\n\+ function calculateCrossLanguageScore(a: string, b: string) {\n\+ return a.length - b.length;\n\+ }\n\+ return getPujaName();\n\+ }\n",+ )+ .unwrap();+ std::fs::write(root.join("README.md"), "# Temple booking\n\nNotes.\n").unwrap();+ std::fs::write(+ root.join(".gitignore"),+ "*\n!app/\n!app/[locale]/\n!app/[locale]/**\n!README.md\n",+ )+ .unwrap();++ let plan = plan_for(root);+ let rel = "app/[locale]/booking/[token]/temple-selection/TempleSelectionWrapper.tsx";+ let file = plan+ .files+ .values()+ .find(|f| f.rel == rel)+ .unwrap_or_else(|| {+ panic!(+ "tsx missing from plan: {:?}",+ plan.files+ .values()+ .map(|f| f.rel.as_str())+ .collect::<Vec<_>>()+ )+ });+ assert_eq!(file.family, "code", "{file:#?}");+ assert!(+ file.assignments.iter().any(|(_, slug)| slug == "docs"),+ "code shares the docs dataset: {file:#?}"+ );+ assert!(+ plan.junk_files.iter().all(|junk| junk.rel != rel),+ "tsx junked: {:?}",+ plan.junk_files+ );+ let docs = plan+ .datasets+ .iter()+ .find(|dataset| dataset.slug == "docs")+ .expect("docs dataset");+ assert_eq!(docs.index, "t-docs");+ assert_eq!(docs.family, "docs");+ assert!(+ docs.specs.iter().any(|spec| spec.name == "name"),+ "symbol name was not indexed: {docs:#?}"+ );+ assert!(+ docs.specs.iter().any(|spec| spec.name == "defs"),+ "defs was not indexed: {docs:#?}"+ );+ }}// ─── Phase A plan building (pure: no server contact) ─────────────────────@@ -6653,6 +6982,14 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -let refs: Vec<&JunkFile> = committed.plan.junk_files.iter().collect();skipped_json(&refs)};+ // #2370: owned here, while `committed` still borrows the journal. A no-op+ // re-run's catalog document may predate the field; the summary below is+ // recomputed from the sealed generation either way.+ let family_warnings = family_junk::from_groups(&committed.plan, &committed.groups);+ let family_warning_lines: Vec<String> = family_warnings+ .iter()+ .map(|warning| warning.line())+ .collect();let sync::SourceExecutionPolicy::DurableSnapshot { reference, .. } = &execution.source_policyelse {anyhow::bail!("generated run does not reference a durable snapshot");@@ -6711,6 +7048,12 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -if let Some(n) = committed.plan.pdf_worker_unavailable_count() {summary["pdf_worker_unavailable"] = json!(n);}+ // Absent when no family tripped the threshold, so a clean run document+ // keeps the fields it already had (#2370).+ if !family_warnings.is_empty() {+ summary["family_junk_warnings"] = serde_json::to_value(&family_warnings)+ .expect("family junk warnings are strings and counts");+ }journal.finish(&summary)?;if cfg.json {println!("{summary}");@@ -6733,6 +7076,9 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -0,));+ for line in &family_warning_lines {+ println!("{line}");+ }for line in &skip_lines {println!("{line}");}@@ -8579,6 +8925,8 @@ fn run_index_report_inner(// forever after a same-size rewrite with restored or stale timestamps.// Hashing reads every byte of the corpus. On a large tree it is minutes of// real work, and before #241 it was minutes with no output at all.+ // Tests delete a file in the gap the walk already closed (#2352).+ before_hash_hook(&state_dir);let mut inventory = match watch {// A plain run hashes the whole corpus, every time, on purpose: see// `content::resolve_reporting`.@@ -8646,13 +8994,24 @@ fn run_index_report_inner(inventory}};+ // A locked file that already has a committed digest stays in the inventory+ // on `--no-graph`, so the reconcile keeps the previous generation instead+ // of deleting it. The graph path must not reinstate: phase A would sniff+ // the locked file and junk it under its content key (#2352).+ if cfg.no_graph {+ if let Some(manifest) = preflight.committed_manifest.as_ref() {+ let kept = carry_locked_priors(&mut inventory, &manifest.plan);+ note_kept_locked(&pr, &kept);+ }+ }+ note_unreadable(&pr, &inventory.unreadable);if cfg.no_graph && preflight.committed_manifest.is_some() && !genesis_recovery {if cfg.dry_run {let base = preflight.committed_manifest.as_ref().context("branch guard proved a committed manifest")?;- let plan = project_reconcile_plan(+ let mut plan = project_reconcile_plan(&inventory,&base.plan,&cfg,@@ -8661,6 +9020,7 @@ fn run_index_report_inner(&scan_meter,tally,)?;+ record_unreadable_skips(&mut plan, &inventory.unreadable);let unchanged = serde_json::to_value(&plan)? == serde_json::to_value(&base.plan)?;let sizes = inventory_size_map(&inventory);let changed_bytes = if unchanged {@@ -8732,7 +9092,7 @@ fn run_index_report_inner(.as_ref().context("generated journal lost its committed manifest")?.clone();- let plan = project_reconcile_plan(+ let mut plan = project_reconcile_plan(&inventory,&base.plan,&cfg,@@ -8741,6 +9101,7 @@ fn run_index_report_inner(&scan_meter,tally,)?;+ record_unreadable_skips(&mut plan, &inventory.unreadable);// Say out loud that the decision gate is not on this route. It is an// incremental reconcile of an already-committed generation, so the work// is the *changed* set and is published from a sealed snapshot rather@@ -8900,12 +9261,13 @@ fn run_index_report_inner(// documents already published. Sweep them here, with the frozen plan// still in hand, the same way an exclusion is swept. `--dry-run`// names them and does not mutate.+ let still_present = files_still_present(&dataless_skipped_files, &inventory.unreadable);let delta = inventory_delta(&cfg.root,&inventory.files,&comparison_keys,prior_plan,- &dataless_skipped_files,+ &still_present,);if !delta.deleted_content_groups.is_empty() {deleted_file_keys.extend(@@ -8985,7 +9347,7 @@ fn run_index_report_inner()?;sync_executor::gc_snapshots(&state_dir, &journal)?;let resumed_with_plan = journal.plan.is_some() && !genesis_recovery;- if inventory.files.is_empty() && !resumed_with_plan {+ if inventory.files.is_empty() && inventory.unreadable.is_empty() && !resumed_with_plan {if dataless_skipped > 0 {println!("no indexable files under {root_str} ({dataless_skipped} dataless placeholder{} \@@ -9268,6 +9630,14 @@ fn run_index_report_inner(clusters_rt = Some(clusters);plan};+ let unreadable_keys: HashSet<String> = inventory+ .unreadable+ .iter()+ .map(|item| content::unreadable_file_key(&item.file.rel_id))+ .collect();+ if record_unreadable_skips(&mut plan, &inventory.unreadable) {+ plan_changed = true;+ }// Second gate, after key selection has settled. Deletions the first gate// already swept are gone from the plan. Anything still classified as a@@ -9279,7 +9649,8 @@ fn run_index_report_inner(// `carry_over_unread_mail`).let mut swept_excluded = false;if resumed_with_plan {- let delta = inventory_delta(&cfg.root, &files, &keys, &plan, &dataless_skipped_files);+ let still_present = files_still_present(&dataless_skipped_files, &inventory.unreadable);+ let delta = inventory_delta(&cfg.root, &files, &keys, &plan, &still_present);if !delta.deleted_content_groups.is_empty() {let gone: HashSet<String> = delta.deleted_content_groups@@ -9366,7 +9737,14 @@ fn run_index_report_inner(.junk_files.iter().filter(|junk| {- !live_keys.contains(junk.file_key.as_str()) && !plan.files.contains_key(&junk.file_key)+ !live_keys.contains(junk.file_key.as_str())+ && !plan.files.contains_key(&junk.file_key)+ // A file that vanished or is locked this run has no content+ // key, so it is not in `live_keys`. Dropping the skip in the+ // same run would hide the only record of it (#2352). The next+ // run, once the walk no longer lists it, does not put the key+ // in `unreadable_keys` and the row is swept.+ && !unreadable_keys.contains(&junk.file_key)}).map(|junk| junk.file_key.clone()).collect();@@ -10293,12 +10671,16 @@ fn run_index_report_inner(let sn = match sn {Ok(s) => s,Err(e) => {+ let (status, reason) = match content::unreadable_junk_fields(&e) {+ Some((status, reason)) => (status.into(), reason),+ None => ("junk".into(), format!("unreadable at index time: {e}")),+ };extra_junk.lock().unwrap().push(JunkFile {file_key: key.clone(),rel: f.rel.clone(),format: "unknown".into(),- status: "junk".into(),- reason: format!("unreadable at index time: {e}"),+ status,+ reason,bytes: f.size,});continue;@@ -11509,6 +11891,9 @@ fn run_index_report_inner(// slice, so a resume reports what is live rather than what it re-parsed.let mut code_coverage = CodeCoverage::default();let mut completed_keys: HashSet<String> = HashSet::new();+ // Format strings of files this run actually indexed. The family warning+ // uses the same set the catalog file docs do, including deletions left out.+ let mut indexed_formats: Vec<String> = Vec::new();{let j = journal_mx.lock().unwrap();for fd in j.done.values() {@@ -11535,6 +11920,7 @@ fn run_index_report_inner(}).unwrap_or_else(|| "unknown".into());code_coverage.observe(&fmt, fd.records);+ indexed_formats.push(fmt.clone());let (id, doc) = catalog::file_doc(&cfg.prefix,&fd.file_key,@@ -11590,6 +11976,16 @@ fn run_index_report_inner());all_junk.retain(|jf| !completed_keys.contains(&jf.file_key));}+ // #2370: indexed formats and the junk list are both final here. Stale-plan+ // skips stay in `all_junk` for the catalog and are ignored by the warning.+ let family_warnings = family_junk::warnings(+ indexed_formats.iter().map(String::as_str),+ all_junk.iter().copied(),+ );+ let family_warning_lines: Vec<String> = family_warnings+ .iter()+ .map(|warning| warning.line())+ .collect();// Counted now: every later reader of this number outlives the borrows// `all_junk` holds on `plan` and `new_unplanned`, which the durable// junk-plan update below mutates.@@ -11603,6 +11999,13 @@ fn run_index_report_inner(.filter(|jf| is_stale_resume_skip(jf)).count();let junk_file_count = all_junk.len() - skipped_appeared;+ // Owned here: the resume-plan update below mutates the junk list these+ // borrow. The count stays `skipped_appeared`; the summary names the paths+ // (#2372).+ let appeared_paths = stale_resume_appeared_paths(&all_junk);+ debug_assert_eq!(appeared_paths.len(), skipped_appeared);+ let appeared_lines = stale_resume_summary_lines(&appeared_paths);+ let appeared_json = stale_resume_summary_json(&appeared_paths);let unextracted_archives = unextracted_archive_lines(&all_junk);for jf in &all_junk {code_coverage.observe(&jf.format, 0);@@ -11829,6 +12232,13 @@ fn run_index_report_inner(if skipped_appeared > 0 {run_doc["skipped_appeared"] = json!(skipped_appeared);}+ // #2372: additive. `skipped_appeared` stays the count. These name the+ // first paths, how many were not listed, and the `--fresh` recovery.+ // Absent when nothing appeared, including `skipped_appeared_not_listed`+ // when every path fit in the summary.+ for (key, value) in &appeared_json {+ run_doc[key.as_str()] = value.clone();+ }// #2314: present only when this run deleted documents for files that left// the folder. A later unchanged rerun does not set it, so exit 0 there is// a real no-op and not another removal.@@ -11852,6 +12262,12 @@ fn run_index_report_inner(if let Some(n) = pdf_worker_unavailable {run_doc["pdf_worker_unavailable"] = json!(n);}+ // #2370: omitted when no family tripped the threshold, so a clean run+ // document keeps every field it had. Added, not renamed.+ if !family_warnings.is_empty() {+ run_doc["family_junk_warnings"] = serde_json::to_value(&family_warnings)+ .expect("family junk warnings are strings and counts");+ }// Owned lines: the resume-plan update below mutates `plan.junk_files`,// which `all_junk` borrows. The summary prints these after that update.let skip_lines = skipped_lines(&all_junk);@@ -11973,6 +12389,12 @@ fn run_index_report_inner(files_removed,));+ for line in &family_warning_lines {+ println!("{line}");+ }+ for line in &appeared_lines {+ println!("{line}");+ }for line in &skip_lines {println!("{line}");}@@ -12116,6 +12538,15 @@ fn run_index_report_inner(if skipped_appeared > 0 {done_fields.push(("skipped_appeared", skipped_appeared as u64));}+ // Remainder only. The paths themselves are not `key=value` numbers: a+ // path may contain spaces, and readers split the plain `xerj-done` line+ // on spaces. They are named on the lines above and in the JSON fields.+ if let Some(rest) = appeared_json+ .get("skipped_appeared_not_listed")+ .and_then(Value::as_u64)+ {+ done_fields.push(("skipped_appeared_not_listed", rest));+ }if files_removed > 0 {done_fields.push(("files_removed", files_removed));}@@ -12126,7 +12557,24 @@ fn run_index_report_inner(done_fields.push(("pdf_worker_unavailable", n));pr.note(&pdf_worker_unavailable_note(n));}+ // #2370: same lines as stdout, on the progress stream, including `--quiet`.+ for line in &family_warning_lines {+ pr.warn(line);+ }+ if !family_warnings.is_empty() {+ done_fields.push(("family_junk_warnings", family_warnings.len() as u64));+ }note_worker_address_space_cap(&pr);+ // #2372: immediately before the terminal line, so an agent that reads the+ // summary (and not the mid-run note) sees the paths and `--fresh`.+ // `--progress json` also gets the fields on the terminal object; the+ // plain line stays numeric.+ for line in &appeared_lines {+ pr.note(line);+ }+ if !appeared_json.is_empty() {+ pr.set_done_json(appeared_json.clone());+ }pr.finish(pdf_worker_unavailable.is_none(),code,@@ -12154,6 +12602,139 @@ fn run_index_report_inner(Ok((code, Some(run_doc)))}+/// Dataless placeholders plus files another process holds locked (#2352).+///+/// Both are still on disk. Leaving them out of the inventory makes+/// [`inventory_delta`] call them exclusions and delete documents indexed the+/// last time the bytes could be read. A file that no longer exists is not+/// included: a rerun sweeps it.+fn files_still_present(+ dataless: &[walk::FileEntry],+ unreadable: &[content::UnreadableSource],+) -> Vec<walk::FileEntry> {+ let mut present = dataless.to_vec();+ present.extend(+ unreadable+ .iter()+ .filter(|item| item.kind == content::ReadFailureKind::Locked)+ .map(|item| item.file.clone()),+ );+ present+}++/// Keep a locked file that already has a committed digest in the inventory.+///+/// `--no-graph` emits Delete for a content group that is absent from the+/// inventory. The unchanged fast path carries the assignment without opening+/// the file, and the seal hardlinks the previous blob when the size still+/// matches. Graph runs must not call this.+fn carry_locked_priors(inventory: &mut content::Inventory, prior: &Plan) -> Vec<String> {+ let mut carried = Vec::new();+ let mut remaining = Vec::new();+ for item in std::mem::take(&mut inventory.unreadable) {+ if item.kind != content::ReadFailureKind::Locked {+ remaining.push(item);+ continue;+ }+ let matched = prior.files.iter().find(|(_, assignment)| {+ (!item.file.rel_id.is_empty()+ && !assignment.path_id.is_empty()+ && assignment.path_id == item.file.rel_id)+ || assignment.rel == item.file.rel+ });+ let Some((key, assignment)) = matched else {+ remaining.push(item);+ continue;+ };+ let Some(digest) = assignment.content_digest.clone() else {+ remaining.push(item);+ continue;+ };+ if inventory.keys.iter().any(|existing| existing == key) {+ remaining.push(item);+ continue;+ }+ let key = key.clone();+ carried.push(item.file.rel.clone());+ inventory.files.push(item.file);+ inventory.keys.push(key);+ inventory.digests.push(digest);+ }+ remaining.sort_by(|left, right| left.file.rel.cmp(&right.file.rel));+ inventory.unreadable = remaining;+ carried.sort();+ carried+}++fn note_unreadable(pr: &Progress, unreadable: &[content::UnreadableSource]) {+ if unreadable.is_empty() {+ return;+ }+ let absent = unreadable+ .iter()+ .filter(|item| item.kind == content::ReadFailureKind::Absent)+ .count();+ let locked = unreadable.len() - absent;+ pr.note(&format!(+ "autoindex: skipped {} file(s) that could not be read after they were enumerated \+ ({absent} no longer exist, {locked} held locked by another process)",+ unreadable.len()+ ));+ for item in unreadable.iter().take(REFUSAL_LIST_CAP) {+ pr.note(&format!(" skipped: {}", item.file.rel));+ }+ let remaining = unreadable.len().saturating_sub(REFUSAL_LIST_CAP);+ if remaining > 0 {+ pr.note(&format!(" … and {remaining} more"));+ }+}++fn note_kept_locked(pr: &Progress, rels: &[String]) {+ if rels.is_empty() {+ return;+ }+ pr.note(&format!(+ "autoindex: {} file(s) could not be opened because another process holds them locked; \+ the previous generation's copy was kept",+ rels.len()+ ));+ for rel in rels.iter().take(REFUSAL_LIST_CAP) {+ pr.note(&format!(" kept: {rel}"));+ }+ let remaining = rels.len().saturating_sub(REFUSAL_LIST_CAP);+ if remaining > 0 {+ pr.note(&format!(" … and {remaining} more"));+ }+}++/// One `skipped` row per file the hash could not open (#2352).+///+/// The key is `axabsent-` plus the path identity, not a content digest. A row+/// already present (a resume) is left as it is.+fn record_unreadable_skips(plan: &mut Plan, unreadable: &[content::UnreadableSource]) -> bool {+ let mut added = false;+ for item in unreadable {+ let key = content::unreadable_file_key(&item.file.rel_id);+ if plan.files.contains_key(&key) || plan.junk_files.iter().any(|junk| junk.file_key == key)+ {+ continue;+ }+ plan.junk_files.push(JunkFile {+ file_key: key,+ rel: item.file.rel.clone(),+ format: "unknown".into(),+ status: "skipped".into(),+ reason: content::unreadable_skip_reason(&item.detail, item.kind),+ bytes: item.file.size,+ });+ added = true;+ }+ if added {+ recount_plan_file_counts(plan);+ }+ added+}+/// The catalog entry for a source file that changed between phase A's digest/// and phase B's read (#1911).///@@ -14683,6 +15264,128 @@ mod code_coverage_tests {);}+ /// #2370: the terminal line of a run that dropped a family is not the line+ /// of a clean run. The warning names the family, the count, and the reason,+ /// and it precedes `xerj-done`.+ #[test]+ fn a_junked_family_is_warned_on_the_terminal_line() {+ let (pr, buffer) = progress::Progress::capture(+ progress::Surface::Plain,+ std::time::Duration::from_secs(3600),+ );+ finish_generated_progress(+ &pr,+ 3,+ &json!({+ "files_indexed": 2,+ "records_total": 4,+ "generation": 1,+ "code_files": 2,+ "code_files_indexed": 2,+ "code_files_junked": 0,+ "files_junk": 4,+ "family_junk_warnings": [{+ "family": "pdf",+ "files": 4,+ "junk_files": 4,+ "indexed_files": 0,+ "extent": "all",+ "top_reason": "could not start the isolated PDF parser",+ "top_reason_files": 4+ }]+ }),+ 0,+ );+ let text = captured(&buffer);+ let done = text+ .lines()+ .find(|line| line.starts_with("xerj-done "))+ .unwrap_or_else(|| panic!("{text}"));+ assert!(done.contains("family_junk_warnings=1"), "{done}");+ assert!(done.contains("exit=3"), "{done}");+ assert!(+ text.lines().any(|line| {+ line.starts_with("warning: pdf family:")+ && line.contains("all 4 files")+ && line.contains("could not start the isolated PDF parser")+ }),+ "{text}"+ );+ assert!(+ text.trim_end()+ .lines()+ .next_back()+ .unwrap()+ .starts_with("xerj-done "),+ "the warning precedes the terminal line: {text}"+ );+ }++ /// `--quiet` drops notes and the terminal line. A dropped family is a+ /// safety warning: it still has to reach stderr (#2370).+ #[test]+ fn a_junked_family_warning_is_not_silenced_by_quiet() {+ let (pr, buffer) = progress::Progress::capture(+ progress::Surface::Silent,+ std::time::Duration::from_secs(3600),+ );+ finish_generated_progress(+ &pr,+ 3,+ &json!({+ "files_indexed": 1,+ "records_total": 1,+ "generation": 1,+ "code_files": 1,+ "code_files_indexed": 1,+ "code_files_junked": 0,+ "family_junk_warnings": [{+ "family": "pdf",+ "files": 4,+ "junk_files": 4,+ "indexed_files": 0,+ "extent": "all",+ "top_reason": "worker down",+ "top_reason_files": 4+ }]+ }),+ 0,+ );+ let text = captured(&buffer);+ assert!(+ text.contains("warning: pdf family:") && text.contains("worker down"),+ "{text}"+ );+ assert!(+ !text.contains("xerj-done"),+ "quiet still prints no terminal line: {text}"+ );+ }++ #[test]+ fn a_clean_summary_does_not_grow_a_family_warning_on_the_terminal_line() {+ let (pr, buffer) = progress::Progress::capture(+ progress::Surface::Plain,+ std::time::Duration::from_secs(3600),+ );+ finish_generated_progress(+ &pr,+ 0,+ &json!({+ "files_indexed": 3,+ "records_total": 9,+ "generation": 1,+ "code_files": 3,+ "code_files_indexed": 3,+ "code_files_junked": 0,+ }),+ 0,+ );+ let text = captured(&buffer);+ assert!(!text.contains("family_junk_warnings"), "{text}");+ assert!(!text.contains("warning:"), "{text}");+ }+#[test]fn skipped_dataless_placeholders_are_counted_on_the_terminal_line() {let (pr, buffer) = progress::Progress::capture(@@ -15700,3 +16403,209 @@ mod legacy_rebuild_argv_tests {}}}++/// #2368: `xerj autoindex map <folder>` previews a folder that has not been+/// indexed. The preview is the `--dry-run` discovery path, so this test runs+/// that command against a node that accepts reads and refuses to pretend a+/// write succeeded: any bulk, document, or mapping request fails the test.+#[cfg(test)]+mod map_folder_preview {+ use super::{cli, run_index};+ use std::io::{BufRead, BufReader, Read, Write};+ use std::net::TcpListener;+ use std::sync::atomic::{AtomicBool, Ordering};+ use std::sync::{Arc, Mutex};+ use std::time::Duration;++ struct Node {+ url: String,+ requests: Arc<Mutex<Vec<(String, String)>>>,+ stop: Arc<AtomicBool>,+ join: Option<std::thread::JoinHandle<()>>,+ }++ impl Node {+ fn start() -> Self {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ listener.set_nonblocking(true).unwrap();+ let url = format!("http://{}", listener.local_addr().unwrap());+ let requests = Arc::new(Mutex::new(Vec::new()));+ let stop = Arc::new(AtomicBool::new(false));+ let requests_bg = Arc::clone(&requests);+ let stop_bg = Arc::clone(&stop);+ let join = std::thread::spawn(move || {+ while !stop_bg.load(Ordering::Relaxed) {+ match listener.accept() {+ Ok((mut stream, _)) => {+ stream.set_nonblocking(false).unwrap();+ stream+ .set_read_timeout(Some(Duration::from_secs(5)))+ .unwrap();+ let mut reader = BufReader::new(stream.try_clone().unwrap());+ let mut request_line = String::new();+ if reader.read_line(&mut request_line).unwrap_or(0) == 0 {+ continue;+ }+ let mut content_length = 0usize;+ loop {+ let mut line = String::new();+ if reader.read_line(&mut line).unwrap_or(0) == 0 {+ break;+ }+ if line == "\r\n" || line.is_empty() {+ break;+ }+ if let Some(value) =+ line.to_ascii_lowercase().strip_prefix("content-length:")+ {+ content_length = value.trim().parse().unwrap_or(0);+ }+ }+ if content_length > 0 {+ let mut body = vec![0; content_length];+ let _ = reader.read_exact(&mut body);+ }+ let mut parts = request_line.split_whitespace();+ let method = parts.next().unwrap_or("").to_string();+ let path = parts.next().unwrap_or("").to_string();+ requests_bg+ .lock()+ .unwrap()+ .push((method.clone(), path.clone()));+ let (status, body) = if method == "GET" && path == "/" {+ (200, r#"{"ok":true}"#)+ } else if method == "GET" && path.starts_with("/_cat/allocation") {+ (+ 200,+ r#"[{"disk.percent":"10","disk.avail":"1073741824b","disk.flood_stage":"95","disk.threshold_enabled":"true","disk.flood_blocked":"false"}]"#,+ )+ } else {+ (404, r#"{"error":"absent"}"#)+ };+ let reason = if status == 200 { "OK" } else { "Not Found" };+ let _ = write!(+ stream,+ "HTTP/1.1 {status} {reason}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",+ body.len()+ );+ }+ Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {+ std::thread::sleep(Duration::from_millis(5));+ }+ Err(error) => panic!("accept failed: {error}"),+ }+ }+ });+ Self {+ url,+ requests,+ stop,+ join: Some(join),+ }+ }++ fn requests(&self) -> Vec<(String, String)> {+ self.requests.lock().unwrap().clone()+ }+ }++ impl Drop for Node {+ fn drop(&mut self) {+ self.stop.store(true, Ordering::Relaxed);+ if let Some(join) = self.join.take() {+ let _ = join.join();+ }+ }+ }++ #[test]+ fn map_folder_previews_an_unindexed_folder_and_writes_no_index() {+ let node = Node::start();+ let root = tempfile::tempdir().unwrap();+ let state = tempfile::tempdir().unwrap();+ let marker = "map-preview-marker-2368";+ std::fs::write(+ root.path().join("readme.md"),+ format!("# Notes\n\n{marker}\n"),+ )+ .unwrap();+ std::fs::write(root.path().join("plain.txt"), "a short note\n").unwrap();+ let before = std::fs::read(root.path().join("readme.md")).unwrap();++ let args = vec![+ "map".to_string(),+ root.path().display().to_string(),+ "--url".to_string(),+ node.url.clone(),+ "--prefix".to_string(),+ "mappreview".to_string(),+ "--state-dir".to_string(),+ state.path().display().to_string(),+ "--quiet".to_string(),+ "--max-minutes".to_string(),+ "0".to_string(),+ ];+ let cmd = cli::parse(args).expect("map <folder> must parse");+ let cli::Cmd::Index(cfg) = cmd else {+ panic!("map <folder> must take the dry-run index path, got {cmd:?}");+ };+ assert!(cfg.dry_run, "the preview must not index");+ assert_eq!(cfg.prefix, "mappreview");+ let code = run_index(*cfg).expect("preview of an unindexed folder");+ assert_eq!(code, 0, "a preview that finishes is exit 0");++ assert_eq!(+ std::fs::read(root.path().join("readme.md")).unwrap(),+ before,+ "the preview must not rewrite the folder"+ );+ // `--dry-run` opens the journal and appends the run header before it+ // prints the plan. The preview is that path, so the header is present+ // and nothing past it (no plan, no file completion, no generation) is.+ let journal = std::fs::read_to_string(state.path().join("journal.ndjson"))+ .expect("the preview opens the same resume journal --dry-run opens");+ let mut kinds = Vec::new();+ for line in journal.lines().filter(|line| !line.is_empty()) {+ let record: serde_json::Value = serde_json::from_str(line).expect(line);+ kinds.push(+ record+ .get("kind")+ .and_then(|kind| kind.as_str())+ .unwrap_or("")+ .to_string(),+ );+ }+ assert_eq!(+ kinds,+ vec!["run".to_string()],+ "the preview journal must be only the run header, not indexing work: {journal}"+ );++ let requests = node.requests();+ assert!(+ requests.iter().any(|(m, p)| m == "GET" && p == "/"),+ "the preview contacts the node: {requests:?}"+ );+ assert!(+ requests+ .iter()+ .any(|(m, p)| m == "POST" && p.contains("autoindex-catalog")),+ "the preview checks the catalog before discovering: {requests:?}"+ );+ let writes: Vec<_> = requests+ .iter()+ .filter(|(method, path)| {+ method == "PUT"+ || method == "DELETE"+ || path.contains("_bulk")+ || path.contains("_doc")+ || path.contains("_mapping")+ || path.contains("_update")+ })+ .collect();+ assert!(+ writes.is_empty(),+ "map <folder> must not write an index: {writes:?}"+ );+ }+}diff --git a/engine/crates/xerj-autoindex/src/objsource.rs b/engine/crates/xerj-autoindex/src/objsource.rsindex 656d8affa..154d9c3ec 100644--- a/engine/crates/xerj-autoindex/src/objsource.rs+++ b/engine/crates/xerj-autoindex/src/objsource.rs@@ -813,12 +813,7 @@ fn fetch_objects(if pending.is_empty() {return Ok(Vec::new());}- let build = || {- rayon::ThreadPoolBuilder::new()- .num_threads(concurrency)- .thread_name(|i| format!("xerj-fetch-{i}"))- .build()- };+ let build = || fetch_pool(concurrency);let results: Vec<Result<FetchOutcome>> = map_in_fetch_pool(build, pending, |entry| {let guard = pr.file(&entry.rel, entry.size);let outcome = fetch_one(run, source, entry);@@ -848,13 +843,26 @@ fn fetch_objects(}}+fn fetch_pool(+ concurrency: usize,+) -> std::result::Result<rayon::ThreadPool, rayon::ThreadPoolBuildError> {+ rayon::ThreadPoolBuilder::new()+ .num_threads(concurrency)+ .thread_name(|i| format!("xerj-fetch-{i}"))+ .spawn_handler(crate::pool::spawn_worker)+ .build()+}+/// `items.map(f)`, in order, on a freshly built object-fetch pool, or as a/// plain sequential loop on the calling thread when the pool cannot be/// started (#1122).////// `build()` fails when `pthread_create` returns `EAGAIN` (thread or memory-/// limit). That used to error the whole object run, after the listing had-/// already been paid for, on exactly the machine that was short of threads.+/// limit). The pool's spawn handler is [`crate::pool::spawn_worker`], so a+/// signal-stack `ENOMEM` is that `Err` rather than a panic in the child+/// (`panic = "abort"` would take the process down, #1122). That used to+/// error the whole object run, after the listing had already been paid for,+/// on exactly the machine that was short of threads./// A sequential fetch is slower but downloads the same objects, so the run/// degrades instead of failing. The fallback deliberately does not go through/// rayon at all: it neither starts a thread nor registers the caller with a@@ -1688,21 +1696,22 @@ mod tests {"the fallback must not register the caller as a rayon worker");- // And with a pool that does start, the same items run on its threads.- let ok = || {- rayon::ThreadPoolBuilder::new()- .num_threads(2)- .thread_name(|i| format!("xerj-fetch-test-{i}"))- .build()- };- let named = map_in_fetch_pool(ok, &items, |&i| {- (- i,- std::thread::current()- .name()- .is_some_and(|n| n.starts_with("xerj-fetch-test-")),- )- });+ // And with the production pool, the same items run on its threads.+ // `std::thread` names are unset: the handler does not use+ // `Builder::name` (that is the signal-stack panic). On Linux the OS+ // comm is the name the handler sets.+ let named = map_in_fetch_pool(+ || super::fetch_pool(2),+ &items,+ |&i| {+ (+ i,+ // Unix workers set the pthread comm, not the std name (#1122).+ xerj_common::thread_spawn::current_thread_name()+ .is_some_and(|name| name.starts_with("xerj-fetch")),+ )+ },+ );assert!(named.iter().enumerate()diff --git a/engine/crates/xerj-autoindex/src/pool.rs b/engine/crates/xerj-autoindex/src/pool.rsindex 12d03638c..6fc83a19f 100644--- a/engine/crates/xerj-autoindex/src/pool.rs+++ b/engine/crates/xerj-autoindex/src/pool.rs@@ -14,8 +14,11 @@//! A pool that cannot be started is not fatal (#1122). `build()` fails when//! `pthread_create` returns `EAGAIN` (thread or memory limit), and release//! builds are `panic = "abort"`, so the old `.expect(..)` took the whole run-//! down. Now phase A runs on the calling thread instead, slower but complete,-//! and the next phase tries to start the pool again.+//! down. Workers are started with [`spawn_worker`], not rayon's default+//! `std::thread::Builder`: that builder returns `Ok` and then panics in the+//! child when the alternate signal stack cannot be mapped, so the `Err`+//! fallback never ran. Now phase A runs on the calling thread instead,+//! slower but complete, and the next phase tries to start the pool again.use std::cell::RefCell;use std::sync::atomic::{AtomicUsize, Ordering};@@ -77,10 +80,18 @@ fn scan_pool_with(threads: usize) -> Option<&'static ThreadPool> {ThreadPoolBuilder::new().num_threads(threads.max(1)).thread_name(|i| format!("xerj-scan-{i}"))+ .spawn_handler(spawn_worker).build()})}+/// Rayon spawn handler that returns `Err` instead of aborting the process+/// (#1122). Shared with the object-fetch pool.+pub(crate) fn spawn_worker(thread: rayon::ThreadBuilder) -> std::io::Result<()> {+ let name = thread.name().unwrap_or("xerj-rayon").to_owned();+ xerj_common::thread_spawn::spawn_detached(&name, thread.stack_size(), move || thread.run())+}+/// The pool in `slot`, built with `build` if it is not there yet. A failed/// build leaves `slot` empty, so the next call tries again.fn pool_in(@@ -182,9 +193,11 @@ mod tests {assert!(widths.iter().all(|&w| w == pool_width));let name_is_ours = install(|| {(0..64).into_par_iter().all(|_| {- std::thread::current()- .name()- .is_some_and(|n| n.starts_with("xerj-scan-"))+ // The safe spawner does not go through `Builder::name`, so+ // `std::thread::current().name()` is empty. The OS comm is+ // what the spawner sets. 15 characters, so `xerj-scan-N` fits.+ xerj_common::thread_spawn::current_thread_name()+ .is_some_and(|name| name.starts_with("xerj-scan"))})});assert!(name_is_ours, "phase A must not run on the global pool");diff --git a/engine/crates/xerj-autoindex/src/progress.rs b/engine/crates/xerj-autoindex/src/progress.rsindex 76a116158..ed55ea59a 100644--- a/engine/crates/xerj-autoindex/src/progress.rs+++ b/engine/crates/xerj-autoindex/src/progress.rs@@ -401,6 +401,12 @@ pub struct Progress {stopped: Mutex<bool>,wake: Condvar,sink: Sink,+ /// Structured fields merged into the `--progress json` terminal object.+ ///+ /// The plain `xerj-done` line is `key=value` numbers split on spaces, so+ /// a path does not belong on it (#2372). Callers print the same facts+ /// with [`Progress::note`] for that surface.+ done_json: Mutex<serde_json::Map<String, serde_json::Value>>,}impl Progress {@@ -432,6 +438,7 @@ impl Progress {stopped: Mutex::new(false),wake: Condvar::new(),sink,+ done_json: Mutex::new(serde_json::Map::new()),})}@@ -652,6 +659,16 @@ impl Progress {self.finish_with_flags(ok, exit, reason, extra, &[]);}+ /// Fields merged into the `--progress json` terminal object.+ ///+ /// The plain `xerj-done` line stays numeric: readers split it on spaces,+ /// and a path may contain spaces. Print the same facts with [`Self::note`]+ /// for that surface. A key already set by the numeric extras or by flags+ /// is left as that caller wrote it.+ pub fn set_done_json(&self, fields: serde_json::Map<String, serde_json::Value>) {+ *self.done_json.lock().unwrap() = fields;+ }+/// [`Self::finish`] plus boolean fields.////// A count alone cannot say whether it is a total or a floor. When a@@ -691,6 +708,10 @@ impl Progress {for (key, value) in flags {doc.insert((*key).to_string(), (*value).into());}+ for (key, value) in self.done_json.lock().unwrap().iter() {+ let key = sanitize(key, SAFE_PATH_MAX);+ doc.entry(key).or_insert(value.clone());+ }let line = serde_json::Value::Object(doc);self.sink.write(format!("{line}\n").as_bytes());}@@ -2268,6 +2289,50 @@ mod tests {assert!(text.contains("files=12 records=7"), "{text}");}+ /// #2372: paths ride on the JSON terminal object. The plain line stays+ /// `key=value` numbers, including when a path contains a space.+ #[test]+ fn json_done_event_names_paths_the_plain_line_does_not() {+ let files = vec!["added-a.md".to_string(), "dir/new file.md".to_string()];+ let mut fields = serde_json::Map::new();+ fields.insert("skipped_appeared_files".into(), serde_json::json!(files));+ fields.insert(+ "skipped_appeared_action".into(),+ serde_json::json!("Re-run with --fresh"),+ );++ let (json_progress, json_buffer) =+ Progress::capture(Surface::Json, Duration::from_secs(3600));+ json_progress.set_done_json(fields.clone());+ json_progress.finish(true, 3, "stale-resume-plan", &[("skipped_appeared", 2)]);+ let doc: serde_json::Value =+ serde_json::from_str(captured(&json_buffer).trim()).expect("one JSON object");+ assert_eq!(doc["event"], "done");+ assert_eq!(doc["exit"], 3);+ assert_eq!(doc["reason"], "stale-resume-plan");+ assert_eq!(doc["skipped_appeared"], 2);+ assert_eq!(doc["skipped_appeared_files"], serde_json::json!(files));+ assert_eq!(doc["skipped_appeared_action"], "Re-run with --fresh");++ let (plain, plain_buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ plain.set_done_json(fields);+ plain.finish(true, 3, "stale-resume-plan", &[("skipped_appeared", 2)]);+ let plain_text = captured(&plain_buffer);+ assert!(+ plain_text.starts_with("xerj-done ok=true exit=3 reason=stale-resume-plan "),+ "{plain_text}"+ );+ assert!(plain_text.contains(" skipped_appeared=2"), "{plain_text}");+ assert!(+ !plain_text.contains("new file.md"),+ "a path with a space must not be spliced into the plain line: {plain_text}"+ );+ assert!(+ !plain_text.contains("skipped_appeared_files"),+ "{plain_text}"+ );+ }+#[test]fn a_run_that_dies_still_closes_the_stream() {let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_millis(20));diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsindex 301bb258c..adff73908 100644--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -1091,6 +1091,7 @@ mod tests {keys: vec![key.into()],digests: vec![format!("digest-{key}")],duplicates: vec![],+ unreadable: vec![],}}@@ -1471,6 +1472,7 @@ mod tests {keys: vec!["cfg".into()],digests: vec!["digest-cfg".into()],duplicates: vec![],+ unreadable: vec![],};let plan = reconcile_plan(&inv, &previous, vec![file_scan], 50, REBUILD).unwrap();assert_eq!(plan.files["cfg"].assignments, vec![(None, "docs".into())]);@@ -1827,6 +1829,7 @@ mod tests {keys: vec!["same-key".into(), "same-key".into()],digests: vec!["digest".into(), "digest".into()],duplicates: vec![],+ unreadable: vec![],};let error = reconcile_plan(&duplicate,diff --git a/engine/crates/xerj-autoindex/src/search.rs b/engine/crates/xerj-autoindex/src/search.rsindex 51b9b6ad3..71f0760ca 100644--- a/engine/crates/xerj-autoindex/src/search.rs+++ b/engine/crates/xerj-autoindex/src/search.rs@@ -11,6 +11,7 @@use std::collections::BTreeSet;use crate::esclient::Es;+use crate::staleness;use serde_json::{json, Value};use xerj_common::xccode::demote_catalog_noise;@@ -37,6 +38,12 @@ EXAMPLE:SEE ALSO:xerj def <symbol> exact go-to-definition when you already know the name++When the autoindex catalog records a folder on this machine for --prefix and a+file there is newer than the last index, or an indexed file has been removed,+the command prints the last index time and a reindex command, then prints the+passages. The check stats at most 4096 entries, stops after 100ms, and skips+hidden and ignored paths.";/// Extract the `_passage` snippet from a hit. The engine returns it under@@ -624,6 +631,36 @@ pub fn resolve_prefix(prefix: &str) -> Result<ResolvedPrefix, String> {})}+/// The autoindex prefix a `xerj_search` call can be checked for staleness.+///+/// An explicit `prefix` wins, resolved the same way as `--prefix`. Neither+/// argument is the CLI default `ax`. An `index` of `{prefix}-*` is that+/// prefix. A concrete index name is not: `proj-docs` is a dataset, and+/// guessing the prefix would check the wrong folder. Both arguments is+/// `None` — the tool already rejects that pair.+pub fn prefix_from_scope(prefix: Option<&str>, index: Option<&str>) -> Option<String> {+ let prefix = prefix.map(str::trim).filter(|s| !s.is_empty());+ let index = index.map(str::trim).filter(|s| !s.is_empty());+ match (prefix, index) {+ (Some(_), Some(_)) => None,+ (Some(prefix), None) => resolve_prefix(prefix).ok().map(|r| r.prefix),+ (None, None) => Some(DEFAULT_SEARCH_PREFIX.to_string()),+ (None, Some(index)) => wildcard_prefix(index),+ }+}++/// `{prefix}-*` → `prefix`. Any other pattern is not one autoindex folder.+fn wildcard_prefix(index: &str) -> Option<String> {+ if index.contains([',', ' ', '\t']) {+ return None;+ }+ let stem = index.strip_suffix("-*")?;+ if stem.is_empty() || stem.contains(['*', '?']) {+ return None;+ }+ resolve_prefix(stem).ok().map(|r| r.prefix)+}+/// Stderr note when the stored prefix is not the characters the caller typed./// `None` when they already match, including surrounding whitespace.pub fn prefix_rewrite_note(given: &str, resolved: &ResolvedPrefix) -> Option<String> {@@ -792,7 +829,7 @@ pub fn run_search_cli() -> i32 {}}- let es = match Es::new(&url, api_key) {+ let es = match Es::new(&url, api_key.clone()) {Ok(es) => es,Err(e) => {eprintln!("xerj search: could not build client for {url}: {e}");@@ -864,7 +901,14 @@ pub fn run_search_cli() -> i32 {// header cite the matched line. `xerj code --json` is the raw server// body and does not share this hit post-pass.xerj_common::xccode::passage::annotate_search_match_lines(&mut resp, &query);+ // After the query, so a failed search does not pay for the catalog read.+ // A missing catalog, a remote folder, or a check that hits its cap leaves+ // the response unchanged (#2345).+ let hint = staleness::check(&url, api_key.as_deref(), &prefix);if as_json {+ if let Some(hint) = &hint {+ staleness::attach(&mut resp, hint);+ }println!("{}",serde_json::to_string_pretty(&resp).unwrap_or_default()@@ -874,6 +918,9 @@ pub fn run_search_cli() -> i32 {if let Some(note) = lexical_semantic_note(&resp) {eprintln!("{note}");}+ if let Some(hint) = &hint {+ println!("{}", hint.cli_line());+ }let hits = resp.pointer("/hits/hits")@@ -966,6 +1013,20 @@ mod tests {"ax-*");+ assert_eq!(prefix_from_scope(None, None).as_deref(), Some("ax"));+ assert_eq!(+ prefix_from_scope(Some("My_Proj"), None).as_deref(),+ Some("my-proj")+ );+ assert_eq!(+ prefix_from_scope(None, Some("ok-knowledge-*")).as_deref(),+ Some("ok-knowledge")+ );+ assert_eq!(prefix_from_scope(Some("proj"), Some("proj-*")), None);+ assert_eq!(prefix_from_scope(None, Some("proj-docs")), None);+ assert_eq!(prefix_from_scope(None, Some("a,b-*")), None);+ assert_eq!(prefix_from_scope(Some("___"), None), None);+let empty = resolve_prefix("___").unwrap_err();assert!(empty.contains("[a-z0-9]"), "{empty}");assert!(resolve_prefix(" ").is_err());diff --git a/engine/crates/xerj-autoindex/src/staleness.rs b/engine/crates/xerj-autoindex/src/staleness.rsnew file mode 100644index 000000000..717b7eae2--- /dev/null+++ b/engine/crates/xerj-autoindex/src/staleness.rs@@ -0,0 +1,1002 @@+//! Hint when a searched autoindex is older than the folder it came from.+//!+//! `xerj code` refuses a ledger older than 30 days+//! (`xerj_common::xccode::state::check_fresh`). This path does not: a search+//! still returns its passages, and says when a file under the recorded root+//! is newer than the last run or an indexed path is gone (#2345). An agent+//! that sees zero hits can then tell "not in the corpus" from "not in the+//! index yet".+//!+//! # Cost+//!+//! One catalog search (2s timeout, no retry) loads the newest run for the+//! prefix. The freshness stamp is `summary_generated_at` when that parses,+//! otherwise `started`. `started` is the beginning of a run, so a file saved+//! while a long generation was in flight can look newer than a stamp that+//! has no summary time; the hint names a reindex, which is safe.+//!+//! The walk then stats at most [`STAT_CAP`] entries and stops after+//! [`WALK_WALL`]. Children of a directory are stat'd, then admitted+//! subdirectories are entered newest-mtime first, so a new page is visited+//! before an old sibling tree spends the cap. Hidden names, symlinks,+//! `.gitignore` / `.xerjignore`, the built-in ignore defaults, and+//! marker-gated generated directories are skipped the same way `xerj+//! autoindex` skips them — a `target/` or `.git` rewrite is not a stale+//! corpus. A root that is not a directory on this machine (a remote node,+//! an object-store source) produces no hint.+//!+//! A directory mtime newer than the stamp, with no newer child, is a+//! removal. That case confirms against catalog `indexed` / `duplicate`+//! paths, and only those, so deleting `target/` does not warn. The+//! confirmation shares the same stat cap and wall clock. A check that hits+//! either cap without a proven change stays silent.++use std::fs;+use std::io::ErrorKind;+use std::path::{Component, Path, PathBuf};+use std::time::{Duration, Instant, SystemTime};++use crate::catalog::CORPUS_SCOPE_FIELD;+use crate::ignore_rules::{is_hidden_name, IgnoreOptions, IgnoreStack};+use crate::walk::marker_generated_dir;+use serde_json::{json, Value};++/// Metadata reads for one check: the root, each listed child, and catalog+/// paths confirmed for deletion.+pub const STAT_CAP: u64 = 4096;+/// A slow disk stops here instead of turning search into a tree walk.+pub const WALK_WALL: Duration = Duration::from_millis(100);+/// mtime must be newer than the index stamp by more than this. Covers+/// one-second filesystem resolution around the moment the run finished.+pub const MTIME_GRACE: Duration = Duration::from_secs(2);+/// Catalog read cap. The search has already talked to the node; a stuck+/// catalog must not hold the answer.+const CATALOG_TIMEOUT: Duration = Duration::from_secs(2);+const RUN_PAGE: usize = 20;+const FILE_PAGE: usize = 4096;+const MAX_DEPTH: usize = 64;+const HINT_CODE: &str = "index_stale";++/// What `xerj search` prints, and what MCP appends under `_xerj.hints`.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct StalenessHint {+ pub code: &'static str,+ pub reason: String,+ pub indexed_at: String,+ pub root: String,+ pub run_id: String,+ pub prefix: String,+ pub path: String,+ /// `newer` (added or changed) or `deleted`.+ pub change: &'static str,+}++impl StalenessHint {+ /// The line `xerj search` prints before its passages.+ pub fn cli_line(&self) -> String {+ format!("xerj search: {}", self.reason)+ }+}++/// One indexed path from the catalog, used only to confirm a deletion.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct CatalogFile {+ pub path: String,+ pub status: Option<String>,+}++/// Caps for one walk. Tests shrink them; search uses [`WalkLimits::default`].+#[derive(Debug, Clone, Copy)]+pub struct WalkLimits {+ pub stat_cap: u64,+ pub wall: Duration,+ pub grace: Duration,+}++impl Default for WalkLimits {+ fn default() -> Self {+ Self {+ stat_cap: STAT_CAP,+ wall: WALK_WALL,+ grace: MTIME_GRACE,+ }+ }+}++/// The run this check compares the folder to.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct IndexStamp {+ pub prefix: String,+ pub run_id: String,+ pub root: String,+ pub indexed_at: SystemTime,+ pub indexed_at_raw: String,+}++/// Look up `prefix` in the catalog and, when the recorded root is a local+/// directory, compare it to the files there.+///+/// `None` on every failure: a catalog the node does not have, a transport+/// error, an unreadable folder. The caller prints the search either way.+pub fn check(url: &str, api_key: Option<&str>, prefix: &str) -> Option<StalenessHint> {+ let prefix = prefix.trim();+ let url = url.trim().trim_end_matches('/');+ if prefix.is_empty() || url.is_empty() {+ return None;+ }+ let client = CatalogClient::new(url, api_key)?;+ let runs = fetch_runs(&client, prefix)?;+ let stamp = latest_stamp(&runs, prefix)?;+ hint_for_stamp(&stamp, &WalkLimits::default(), || {+ fetch_files(&client, prefix).unwrap_or_default()+ })+}++/// Decide from an already-loaded stamp. `files` runs only when a directory+/// mtime says something was removed and the walk did not already find a+/// newer admitted file.+pub fn hint_for_stamp(+ stamp: &IndexStamp,+ limits: &WalkLimits,+ files: impl FnOnce() -> Vec<CatalogFile>,+) -> Option<StalenessHint> {+ let root = Path::new(stamp.root.trim());+ if !root.is_dir() {+ return None;+ }+ let root = root.canonicalize().ok()?;+ if !root.is_dir() {+ return None;+ }+ let walked = walk_tree(&root, stamp.indexed_at, limits);+ let (change, rel) = match walked.kind {+ WalkKind::Newer(rel) => ("newer", rel),+ WalkKind::Suspect => {+ if walked.stats >= limits.stat_cap || walked.started.elapsed() >= limits.wall {+ return None;+ }+ let rel = find_deleted(&root, &files(), limits, walked.stats, walked.started)?;+ ("deleted", rel)+ }+ WalkKind::Fresh => return None,+ };+ Some(compose(stamp, &root, &rel, change))+}++/// Append `hint` to `_xerj.hints`, leaving an existing hint list in place.+/// A response that is not a JSON object is left alone.+pub fn attach(resp: &mut Value, hint: &StalenessHint) {+ let Some(obj) = resp.as_object_mut() else {+ return;+ };+ let xerj = obj.entry("_xerj").or_insert_with(|| json!({}));+ let Some(xerj) = xerj.as_object_mut() else {+ return;+ };+ let hints = xerj.entry("hints").or_insert_with(|| json!([]));+ let Some(hints) = hints.as_array_mut() else {+ return;+ };+ if hints+ .iter()+ .any(|h| h.get("code").and_then(Value::as_str) == Some(hint.code))+ {+ return;+ }+ hints.push(json!({+ "code": hint.code,+ "reason": hint.reason,+ "indexed_at": hint.indexed_at,+ "root": hint.root,+ "run_id": hint.run_id,+ "prefix": hint.prefix,+ "path": hint.path,+ "change": hint.change,+ }));+}++/// Merge `hint` into a search JSON body. Non-JSON text is returned unchanged+/// so a truncated budget response is not rewritten into something else.+pub fn merge_response_json(text: &str, hint: &StalenessHint) -> String {+ let Ok(mut value) = serde_json::from_str::<Value>(text) else {+ return text.to_string();+ };+ attach(&mut value, hint);+ serde_json::to_string(&value).unwrap_or_else(|_| text.to_string())+}++/// Newest run for `prefix`, by the freshness stamp.+///+/// Run documents are filtered here as well as in the query: a `prefix`+/// inferred as `text` can make a `match_phrase` hit a neighbor, and `term`+/// on that field misses. `summary_generated_at` is the end of the run.+/// `started` is the fallback the generation catalog writes.+pub fn latest_stamp(runs: &[Value], prefix: &str) -> Option<IndexStamp> {+ let mut best: Option<IndexStamp> = None;+ for run in runs {+ if run.get("doc_kind").and_then(Value::as_str) != Some("run") {+ continue;+ }+ if !scope_matches(run, prefix) {+ continue;+ }+ let root = run.get("root").and_then(Value::as_str)?.trim();+ if root.is_empty() {+ continue;+ }+ let (raw, when) = freshness(run)?;+ let run_id = run+ .get("run_id")+ .and_then(Value::as_str)+ .unwrap_or("")+ .to_string();+ let stamp = IndexStamp {+ prefix: prefix.to_string(),+ run_id,+ root: root.to_string(),+ indexed_at: when,+ indexed_at_raw: raw,+ };+ let replace = match &best {+ None => true,+ Some(prev) => {+ stamp.indexed_at > prev.indexed_at+ || (stamp.indexed_at == prev.indexed_at && stamp.run_id > prev.run_id)+ }+ };+ if replace {+ best = Some(stamp);+ }+ }+ best+}++fn freshness(run: &Value) -> Option<(String, SystemTime)> {+ if let Some(raw) = field_str(run, "summary_generated_at") {+ if let Some(when) = parse_time(raw) {+ return Some((raw.to_string(), when));+ }+ }+ let raw = field_str(run, "started")?;+ Some((raw.to_string(), parse_time(raw)?))+}++fn field_str<'a>(value: &'a Value, key: &str) -> Option<&'a str> {+ value+ .get(key)+ .and_then(Value::as_str)+ .map(str::trim)+ .filter(|s| !s.is_empty())+}++fn parse_time(raw: &str) -> Option<SystemTime> {+ let dt = chrono::DateTime::parse_from_rfc3339(raw).ok()?;+ let secs = dt.timestamp();+ if secs < 0 {+ return None;+ }+ Some(SystemTime::UNIX_EPOCH + Duration::new(secs as u64, dt.timestamp_subsec_nanos()))+}++fn scope_matches(doc: &Value, prefix: &str) -> bool {+ field_str(doc, "prefix") == Some(prefix) || field_str(doc, CORPUS_SCOPE_FIELD) == Some(prefix)+}++fn compose(stamp: &IndexStamp, root: &Path, rel: &str, change: &str) -> StalenessHint {+ let rel_shown = sanitize_display(rel);+ let root_shown = sanitize_display(&root.display().to_string());+ let run = if stamp.run_id.is_empty() {+ String::new()+ } else {+ format!(" (run {})", sanitize_display(&stamp.run_id))+ };+ let what = if change == "deleted" {+ format!(+ "`{rel_shown}` was indexed and is no longer in that folder, so these results can name a page that was removed"+ )+ } else {+ format!(+ "`{rel_shown}` is newer than that index, so a page on disk can be missing from these results"+ )+ };+ let reason = format!(+ "The index for prefix `{}` was last built at {}{run}. {what}. Reindex with: xerj autoindex {} --prefix {}",+ sanitize_display(&stamp.prefix),+ sanitize_display(&stamp.indexed_at_raw),+ shell_quote(&root_shown),+ shell_quote(&stamp.prefix),+ );+ StalenessHint {+ code: HINT_CODE,+ reason,+ indexed_at: stamp.indexed_at_raw.clone(),+ root: root_shown,+ run_id: stamp.run_id.clone(),+ prefix: stamp.prefix.clone(),+ path: rel_shown,+ change: if change == "deleted" {+ "deleted"+ } else {+ "newer"+ },+ }+}++fn sanitize_display(value: &str) -> String {+ value+ .chars()+ .take(240)+ .map(|ch| {+ if ch.is_control() || ch == '`' {+ '?'+ } else {+ ch+ }+ })+ .collect()+}++fn shell_quote(value: &str) -> String {+ if !value.is_empty()+ && value+ .chars()+ .all(|c| c.is_ascii_alphanumeric() || matches!(c, '/' | '.' | '_' | '-' | ':' | '\\'))+ {+ value.to_string()+ } else {+ format!("'{}'", value.replace('\'', "'\\''"))+ }+}++enum WalkKind {+ Newer(String),+ Suspect,+ Fresh,+}++struct Walked {+ kind: WalkKind,+ stats: u64,+ started: Instant,+}++fn walk_tree(root: &Path, indexed_at: SystemTime, limits: &WalkLimits) -> Walked {+ let started = Instant::now();+ let mut walker = Walker {+ root: root.to_path_buf(),+ indexed_at,+ limits: *limits,+ started,+ stats: 0,+ suspect: false,+ stack: IgnoreStack::new(root, IgnoreOptions::default()),+ };+ if started.elapsed() >= limits.wall || !walker.charge() {+ return Walked {+ kind: WalkKind::Fresh,+ stats: walker.stats,+ started,+ };+ }+ let mtime = fs::symlink_metadata(root)+ .and_then(|meta| meta.modified())+ .unwrap_or(SystemTime::UNIX_EPOCH);+ let kind = match walker.visit(root, 0, mtime) {+ WalkControl::Newer(rel) => WalkKind::Newer(rel),+ WalkControl::Stop if walker.suspect => WalkKind::Suspect,+ WalkControl::Stop => WalkKind::Fresh,+ WalkControl::Continue if walker.suspect => WalkKind::Suspect,+ WalkControl::Continue => WalkKind::Fresh,+ };+ Walked {+ kind,+ stats: walker.stats,+ started,+ }+}++enum WalkControl {+ Newer(String),+ Stop,+ Continue,+}++struct Walker {+ root: PathBuf,+ indexed_at: SystemTime,+ limits: WalkLimits,+ started: Instant,+ stats: u64,+ suspect: bool,+ stack: IgnoreStack,+}++impl Walker {+ fn charge(&mut self) -> bool {+ if self.stats >= self.limits.stat_cap || self.started.elapsed() >= self.limits.wall {+ return false;+ }+ self.stats += 1;+ true+ }++ fn newer(&self, mtime: SystemTime) -> bool {+ match mtime.duration_since(self.indexed_at) {+ Ok(age) => age > self.limits.grace,+ Err(_) => false,+ }+ }++ fn visit(&mut self, dir: &Path, depth: usize, dir_mtime: SystemTime) -> WalkControl {+ if depth >= MAX_DEPTH {+ return WalkControl::Continue;+ }+ self.stack.truncate_to(depth);+ self.stack.enter_dir(dir, depth);+ let entries = match fs::read_dir(dir) {+ Ok(entries) => entries,+ Err(_) => return WalkControl::Continue,+ };+ let mut complete = true;+ let mut child_newer = false;+ let mut subdirs: Vec<(SystemTime, PathBuf)> = Vec::new();+ for entry in entries {+ let Ok(entry) = entry else {+ complete = false;+ break;+ };+ if !self.charge() {+ complete = false;+ break;+ }+ let path = entry.path();+ let meta = match fs::symlink_metadata(&path) {+ Ok(meta) => meta,+ Err(_) => continue,+ };+ let mtime = meta.modified().unwrap_or(SystemTime::UNIX_EPOCH);+ if self.newer(mtime) {+ child_newer = true;+ }+ let name = match path.file_name() {+ Some(name) => name,+ None => continue,+ };+ if is_hidden_name(name) {+ continue;+ }+ let file_type = meta.file_type();+ if file_type.is_symlink() {+ continue;+ }+ if file_type.is_dir() {+ if self.stack.skip_dir(&path).is_some() || marker_generated_dir(&path).is_some() {+ continue;+ }+ subdirs.push((mtime, path));+ continue;+ }+ if !file_type.is_file() || self.stack.skip_file(&path).is_some() {+ continue;+ }+ if self.newer(mtime) {+ return WalkControl::Newer(rel_of(&self.root, &path));+ }+ }+ if complete && self.newer(dir_mtime) && !child_newer {+ self.suspect = true;+ }+ if !complete {+ return WalkControl::Stop;+ }+ subdirs.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));+ for (mtime, path) in subdirs {+ match self.visit(&path, depth + 1, mtime) {+ WalkControl::Continue => {}+ other => return other,+ }+ }+ WalkControl::Continue+ }+}++fn rel_of(root: &Path, path: &Path) -> String {+ path.strip_prefix(root)+ .unwrap_or(path)+ .to_string_lossy()+ .replace('\\', "/")+}++fn find_deleted(+ root: &Path,+ files: &[CatalogFile],+ limits: &WalkLimits,+ mut stats: u64,+ started: Instant,+) -> Option<String> {+ for file in files {+ if !counts_as_indexed(file.status.as_deref()) {+ continue;+ }+ let Some(rel) = safe_rel(&file.path) else {+ continue;+ };+ if stats >= limits.stat_cap || started.elapsed() >= limits.wall {+ return None;+ }+ stats += 1;+ let path = root.join(rel);+ match fs::symlink_metadata(&path) {+ Err(err) if err.kind() == ErrorKind::NotFound => return Some(rel.to_string()),+ _ => {}+ }+ }+ None+}++fn counts_as_indexed(status: Option<&str>) -> bool {+ match status {+ None | Some("indexed" | "duplicate") => true,+ Some(_) => false,+ }+}++/// A catalog path that stays inside the root. Absolute paths and `..` are+/// ignored so a catalog row cannot point the stat at another directory.+fn safe_rel(rel: &str) -> Option<&str> {+ if rel.is_empty() || rel.len() > 1024 || rel.starts_with('/') || rel.starts_with('\\') {+ return None;+ }+ let path = Path::new(rel);+ if path.is_absolute() {+ return None;+ }+ if path+ .components()+ .any(|component| !matches!(component, Component::Normal(_)))+ {+ return None;+ }+ Some(rel)+}++struct CatalogClient {+ http: reqwest::blocking::Client,+ base: String,+ api_key: Option<String>,+}++enum CatalogErr {+ BadRequest,+ Failed,+}++impl CatalogClient {+ fn new(url: &str, api_key: Option<&str>) -> Option<Self> {+ let http = reqwest::blocking::Client::builder()+ .timeout(CATALOG_TIMEOUT)+ .connect_timeout(Duration::from_secs(1))+ .danger_accept_invalid_certs(true)+ .build()+ .ok()?;+ let api_key = api_key+ .map(str::trim)+ .filter(|key| !key.is_empty())+ .map(str::to_string);+ Some(Self {+ http,+ base: url.to_string(),+ api_key,+ })+ }++ fn search(&self, body: &Value) -> Result<Value, CatalogErr> {+ let url = format!("{}/autoindex-catalog/_search", self.base);+ let mut req = self.http.post(url).json(body);+ if let Some(key) = &self.api_key {+ req = req.header("Authorization", format!("ApiKey {key}"));+ }+ let resp = req.send().map_err(|_| CatalogErr::Failed)?;+ let status = resp.status();+ if status.as_u16() == 400 {+ return Err(CatalogErr::BadRequest);+ }+ if !status.is_success() {+ return Err(CatalogErr::Failed);+ }+ resp.json().map_err(|_| CatalogErr::Failed)+ }+}++fn scope_clause(prefix: &str) -> Value {+ json!({+ "bool": {+ "should": [+ {"term": {"prefix": prefix}},+ {"match_phrase": {"prefix": prefix}},+ {"term": {CORPUS_SCOPE_FIELD: prefix}}+ ],+ "minimum_should_match": 1+ }+ })+}++fn fetch_runs(client: &CatalogClient, prefix: &str) -> Option<Vec<Value>> {+ let source = json!([+ "prefix",+ CORPUS_SCOPE_FIELD,+ "root",+ "started",+ "summary_generated_at",+ "run_id",+ "doc_kind"+ ]);+ let query = json!({"bool": {"must": [+ {"term": {"doc_kind": "run"}},+ scope_clause(prefix)+ ]}});+ let sorted = json!({+ "size": RUN_PAGE,+ "sort": [{"_id": "desc"}],+ "_source": source,+ "query": query,+ });+ match client.search(&sorted) {+ Ok(body) => Some(hit_sources(&body)),+ Err(CatalogErr::BadRequest) => {+ let unsorted = json!({+ "size": 200,+ "_source": source,+ "query": query,+ });+ client.search(&unsorted).ok().map(|body| hit_sources(&body))+ }+ Err(CatalogErr::Failed) => None,+ }+}++fn fetch_files(client: &CatalogClient, prefix: &str) -> Option<Vec<CatalogFile>> {+ let body = json!({+ "size": FILE_PAGE,+ "_source": ["path", "status", "prefix", CORPUS_SCOPE_FIELD, "doc_kind"],+ "query": {"bool": {"must": [+ {"term": {"doc_kind": "file"}},+ scope_clause(prefix)+ ]}}+ });+ let sources = hit_sources(&client.search(&body).ok()?);+ let mut files = Vec::new();+ for source in sources {+ if source.get("doc_kind").and_then(Value::as_str) != Some("file") {+ continue;+ }+ if !scope_matches(&source, prefix) {+ continue;+ }+ let Some(path) = field_str(&source, "path") else {+ continue;+ };+ files.push(CatalogFile {+ path: path.to_string(),+ status: field_str(&source, "status").map(str::to_string),+ });+ }+ Some(files)+}++fn hit_sources(body: &Value) -> Vec<Value> {+ body.pointer("/hits/hits")+ .and_then(Value::as_array)+ .map(|hits| {+ hits.iter()+ .filter_map(|hit| hit.get("_source").cloned())+ .collect()+ })+ .unwrap_or_default()+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::fs;++ fn limits(stat_cap: u64) -> WalkLimits {+ WalkLimits {+ stat_cap,+ wall: Duration::from_secs(5),+ grace: MTIME_GRACE,+ }+ }++ fn touch(path: &Path, when: SystemTime) {+ let file = fs::OpenOptions::new()+ .read(true)+ .open(path)+ .unwrap_or_else(|err| panic!("open {}: {err}", path.display()));+ file.set_modified(when)+ .unwrap_or_else(|err| panic!("mtime {}: {err}", path.display()));+ }++ fn stamp(root: &Path, indexed_at: SystemTime) -> IndexStamp {+ IndexStamp {+ prefix: "ok-knowledge".into(),+ run_id: "run-20261023T234652Z-2c98-0000".into(),+ root: root.display().to_string(),+ indexed_at,+ indexed_at_raw: "2026-10-23T23:46:52Z".into(),+ }+ }++ fn hint(+ root: &Path,+ indexed_at: SystemTime,+ cap: u64,+ files: &[CatalogFile],+ ) -> Option<StalenessHint> {+ hint_for_stamp(&stamp(root, indexed_at), &limits(cap), || files.to_vec())+ }++ #[test]+ fn a_page_added_after_the_index_is_named_and_the_search_is_not_refused() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(24 * 3600);+ let old = indexed_at - Duration::from_secs(24 * 3600);+ fs::create_dir_all(root.join("docs")).unwrap();+ fs::write(root.join("docs/old.md"), "yesterday\n").unwrap();+ touch(&root.join("docs/old.md"), old);+ touch(&root.join("docs"), old);+ touch(root, old);+ fs::write(root.join("docs/page.md"), "created the next day\n").unwrap();++ let found = hint(root, indexed_at, STAT_CAP, &[]).expect("newer page");+ let line = found.cli_line();+ assert_eq!(found.change, "newer");+ assert_eq!(found.code, "index_stale");+ assert!(line.starts_with("xerj search: "), "{line}");+ assert!(line.contains("docs/page.md"), "{line}");+ assert!(line.contains("2026-10-23T23:46:52Z"), "{line}");+ assert!(line.contains("run-20261023T234652Z-2c98-0000"), "{line}");+ assert!(+ line.contains("xerj autoindex ") && line.contains("--prefix ok-knowledge"),+ "{line}"+ );+ assert!(+ !line.to_ascii_lowercase().contains("refus"),+ "a stale index is a hint, not a refusal: {line}"+ );+ }++ #[test]+ fn newest_directory_is_visited_before_the_cap_is_spent_on_old_siblings() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(24 * 3600);+ let old = indexed_at - Duration::from_secs(24 * 3600);+ fs::create_dir_all(root.join("old")).unwrap();+ for i in 0..30 {+ let path = root.join(format!("old/f{i}.txt"));+ fs::write(&path, "old\n").unwrap();+ touch(&path, old);+ }+ touch(&root.join("old"), old);+ fs::create_dir_all(root.join("fresh")).unwrap();+ fs::write(root.join("fresh/page.md"), "new\n").unwrap();++ let found = hint(root, indexed_at, 4, &[]).expect("cap 4 reaches fresh/page.md");+ assert_eq!(found.path, "fresh/page.md", "{found:?}");+ assert!(+ hint(root, indexed_at, 3, &[]).is_none(),+ "three stats cover the root and its two children and stop before the new page"+ );+ }++ #[test]+ fn an_unchanged_tree_and_ignored_or_hidden_writes_stay_quiet() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(24 * 3600);+ let old = indexed_at - Duration::from_secs(24 * 3600);+ fs::create_dir_all(root.join("src")).unwrap();+ fs::write(root.join("src/lib.rs"), "fn main() {}\n").unwrap();+ fs::write(root.join(".gitignore"), "*.log\n").unwrap();+ touch(&root.join("src/lib.rs"), old);+ touch(&root.join(".gitignore"), old);+ touch(&root.join("src"), old);+ touch(root, old);++ assert!(hint(root, indexed_at, STAT_CAP, &[]).is_none());++ fs::create_dir_all(root.join(".git")).unwrap();+ fs::write(root.join(".git/HEAD"), "ref: refs/heads/main\n").unwrap();+ fs::create_dir_all(root.join("target")).unwrap();+ fs::write(root.join("target/foo.rs"), "build output\n").unwrap();+ fs::write(root.join("debug.log"), "noise\n").unwrap();+ assert!(+ hint(root, indexed_at, STAT_CAP, &[]).is_none(),+ "hidden, default-ignored, and gitignored writes are not the corpus"+ );+ }++ #[test]+ fn a_marker_generated_directory_is_not_a_stale_page() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(24 * 3600);+ let old = indexed_at - Duration::from_secs(24 * 3600);+ fs::create_dir_all(root.join("ProjectSettings")).unwrap();+ fs::write(+ root.join("ProjectSettings/ProjectVersion.txt"),+ "m_EditorVersion: 1\n",+ )+ .unwrap();+ fs::write(root.join("README.md"), "notes\n").unwrap();+ touch(&root.join("ProjectSettings/ProjectVersion.txt"), old);+ touch(&root.join("README.md"), old);+ touch(&root.join("ProjectSettings"), old);+ touch(root, old);+ fs::create_dir_all(root.join("Library")).unwrap();+ fs::write(root.join("Library/cache.bin"), "generated\n").unwrap();+ assert!(hint(root, indexed_at, STAT_CAP, &[]).is_none());+ }++ #[test]+ fn a_removed_indexed_file_is_a_hint_and_a_removed_ignored_tree_is_not() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(24 * 3600);+ let old = indexed_at - Duration::from_secs(24 * 3600);+ let fresh = SystemTime::now();+ fs::create_dir_all(root.join("docs")).unwrap();+ fs::write(root.join("docs/keep.md"), "stay\n").unwrap();+ touch(&root.join("docs/keep.md"), old);+ touch(&root.join("docs"), fresh);+ touch(root, old);+ let files = vec![+ CatalogFile {+ path: "docs/gone.md".into(),+ status: Some("indexed".into()),+ },+ CatalogFile {+ path: "docs/keep.md".into(),+ status: Some("indexed".into()),+ },+ CatalogFile {+ path: "target/foo.rs".into(),+ status: Some("junk".into()),+ },+ CatalogFile {+ path: "../outside.txt".into(),+ status: Some("indexed".into()),+ },+ ];+ let found = hint(root, indexed_at, STAT_CAP, &files).expect("deleted page");+ assert_eq!(found.change, "deleted");+ assert_eq!(found.path, "docs/gone.md");+ let line = found.cli_line();+ assert!(line.contains("no longer in that folder"), "{line}");+ assert!(line.contains("Reindex with:"), "{line}");+ assert!(!line.to_ascii_lowercase().contains("refus"), "{line}");++ // The directory change is explained by a newer ignored child, so a+ // missing junk path is not reported.+ let explained = tempfile::tempdir().unwrap();+ let root = explained.path();+ fs::create_dir_all(root.join("src")).unwrap();+ fs::write(root.join("src/lib.rs"), "fn a() {}\n").unwrap();+ touch(&root.join("src/lib.rs"), old);+ touch(&root.join("src"), old);+ fs::create_dir_all(root.join("target")).unwrap();+ fs::write(root.join("target/out.rlib"), "bin\n").unwrap();+ assert!(hint(root, indexed_at, STAT_CAP, &files).is_none());+ }++ #[test]+ fn a_mtime_inside_the_grace_window_is_quiet() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ let indexed_at = SystemTime::now() - Duration::from_secs(60);+ fs::write(root.join("page.md"), "same second\n").unwrap();+ touch(&root.join("page.md"), indexed_at + Duration::from_secs(1));+ touch(root, indexed_at);+ assert!(hint(root, indexed_at, STAT_CAP, &[]).is_none());+ }++ #[test]+ fn a_missing_root_produces_no_hint() {+ let stamp = IndexStamp {+ prefix: "ok-knowledge".into(),+ run_id: "run".into(),+ root: "/no/such/xerj-staleness-root-2345".into(),+ indexed_at: SystemTime::now() - Duration::from_secs(3600),+ indexed_at_raw: "2026-10-23T23:46:52Z".into(),+ };+ assert!(hint_for_stamp(&stamp, &limits(8), Vec::new).is_none());+ }++ #[test]+ fn latest_stamp_prefers_the_summary_time_and_this_prefix() {+ let runs = vec![+ json!({+ "doc_kind": "run",+ "prefix": "ok-knowledge",+ "root": "/old",+ "started": "2026-10-22T00:00:00Z",+ "run_id": "run-old"+ }),+ json!({+ "doc_kind": "run",+ "prefix": "other",+ "root": "/other",+ "started": "2026-10-25T00:00:00Z",+ "run_id": "run-other"+ }),+ json!({+ "doc_kind": "run",+ "prefix": "ok-knowledge",+ "root": "/corpus",+ "started": "2026-10-23T23:00:00Z",+ "summary_generated_at": "2026-10-23T23:46:52Z",+ "run_id": "run-20261023T234652Z-2c98-0000"+ }),+ json!({+ "doc_kind": "run",+ "prefix": "ok-knowledge",+ "root": "",+ "started": "2026-10-24T00:00:00Z",+ "run_id": "run-blank-root"+ }),+ ];+ let stamp = latest_stamp(&runs, "ok-knowledge").unwrap();+ assert_eq!(stamp.root, "/corpus");+ assert_eq!(stamp.indexed_at_raw, "2026-10-23T23:46:52Z");+ assert_eq!(stamp.run_id, "run-20261023T234652Z-2c98-0000");+ assert!(latest_stamp(&runs, "missing").is_none());+ }++ #[test]+ fn attach_appends_to_existing_hints_and_leaves_hits_in_place() {+ let hint = StalenessHint {+ code: HINT_CODE,+ reason: "reindex please".into(),+ indexed_at: "2026-10-23T23:46:52Z".into(),+ root: "/corpus".into(),+ run_id: "run-1".into(),+ prefix: "ok-knowledge".into(),+ path: "docs/page.md".into(),+ change: "newer",+ };+ let mut resp = json!({+ "_xerj": {"hints": [{"code": "lexical_on_semantic_text", "reason": "lexical"}]},+ "hits": {"hits": [{"_id": "1"}]}+ });+ attach(&mut resp, &hint);+ attach(&mut resp, &hint);+ let hints = resp+ .pointer("/_xerj/hints")+ .and_then(Value::as_array)+ .unwrap();+ assert_eq!(hints.len(), 2);+ assert_eq!(hints[1]["code"], "index_stale");+ assert_eq!(hints[1]["path"], "docs/page.md");+ assert_eq!(+ resp.pointer("/hits/hits/0/_id").and_then(Value::as_str),+ Some("1")+ );++ let merged = merge_response_json(r#"{"hits":{"hits":[]}}"#, &hint);+ let parsed: Value = serde_json::from_str(&merged).unwrap();+ assert_eq!(+ parsed+ .pointer("/_xerj/hints/0/change")+ .and_then(Value::as_str),+ Some("newer")+ );+ assert_eq!(merge_response_json("not-json", &hint), "not-json");+ }++ #[test]+ fn a_closed_catalog_port_is_silence() {+ assert!(check("http://127.0.0.1:1", None, "ok-knowledge").is_none());+ }+}diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rsindex 84f45d31f..52990d528 100644--- a/engine/crates/xerj-autoindex/src/sync_executor.rs+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs@@ -3887,6 +3887,7 @@ fn source_changed(path: &Path, detail: String) -> anyhow::Error {crate::content::SourceChanged {path: path.to_path_buf(),detail,+ locked: false,}.into()}@@ -5896,6 +5897,7 @@ mod tests {keys: vec![],digests: vec![],duplicates: vec![],+ unreadable: vec![],};let empty_snapshot = create_snapshot(state.path(), "tx-delete", &empty).unwrap();let mut empty_plan = journal.committed_manifest.as_ref().unwrap().plan.clone();@@ -5929,6 +5931,7 @@ mod tests {keys: vec![],digests: vec![],duplicates: vec![],+ unreadable: vec![],};let snapshot = create_snapshot(state.path(), "tx-hostile", &inventory).unwrap();let mut journal =diff --git a/engine/crates/xerj-autoindex/src/walk.rs b/engine/crates/xerj-autoindex/src/walk.rsindex 3a3fd9a5f..5ce0f9776 100644--- a/engine/crates/xerj-autoindex/src/walk.rs+++ b/engine/crates/xerj-autoindex/src/walk.rs@@ -64,7 +64,7 @@ const MARKER_GENERATED_DIRS: &[(&str, &str, &str)] = &[];/// The marker-gated verdict for one directory entry.-fn marker_generated_dir(path: &Path) -> Option<&'static str> {+pub(crate) fn marker_generated_dir(path: &Path) -> Option<&'static str> {let name = path.file_name()?;for (dir, marker, label) in MARKER_GENERATED_DIRS {if name == *dir && path.parent().is_some_and(|p| p.join(marker).is_file()) {diff --git a/engine/crates/xerj-autoindex/tests/scan_pool_width.rs b/engine/crates/xerj-autoindex/tests/scan_pool_width.rsindex 0cc77a988..f35d5b7ef 100644--- a/engine/crates/xerj-autoindex/tests/scan_pool_width.rs+++ b/engine/crates/xerj-autoindex/tests/scan_pool_width.rs@@ -43,10 +43,8 @@ fn configure_bounds_phase_a_to_the_requested_width() {(0..256).into_par_iter().map(|_| {- std::thread::current()- .name()- .unwrap_or("unnamed")- .to_string()+ // Unix workers set the pthread comm, not the std name (#1122).+ xerj_common::thread_spawn::current_thread_name().unwrap_or_else(|| "unnamed".into())}).collect()});diff --git a/engine/crates/xerj-cluster/src/shard.rs b/engine/crates/xerj-cluster/src/shard.rsindex b8651cb59..9bfef4c8e 100644--- a/engine/crates/xerj-cluster/src/shard.rs+++ b/engine/crates/xerj-cluster/src/shard.rs@@ -278,6 +278,22 @@ pub enum ShardOp {#[serde(default)]check_only: bool,},+ /// Run `_delete_by_query` on this primary's shard only.+ ///+ /// The coordinating node sends one of these to every primary. This node+ /// deletes documents in its user index and copies each delete onto the+ /// in-sync replicas. `check_only` confirms the index is open and writes+ /// nothing.+ DeleteByQuery {+ index: String,+ /// JSON query object. Already combined with any alias filter.+ query_json: String,+ #[serde(default)]+ max_docs: Option<u64>,+ scroll_size: u64,+ #[serde(default)]+ check_only: bool,+ },/// Search this member's local shard only. The caller merges hits.Search {index: String,diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 7c54f70a5..14a6a4afa 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1581,9 +1581,14 @@ pub struct NeuralIngestProbe {/// same fail-closed rule as explicit `"proxy"` (#1189). The neural/// backend ships in the standard binary; a `--no-default-features` slim/// build omits it and falls back to lexical, and identity reports lexical.-/// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`+/// * `"proxy"` — call an external OpenAI-compatible embeddings/// endpoint ([`default_endpoint`](Self::default_endpoint)). Lets-/// customers plug in ANY embedding model / provider they already run. An explicit `mode = "proxy"` with a+/// customers plug in ANY embedding model / provider they already run.+/// The configured value may be a base URL ending in `/v1` (trailing+/// slash optional) or a bare host — `/embeddings` is appended — or the+/// full path ending in `/embeddings`, which is used unchanged. Any other+/// path is a startup ERROR that names the URL and the+/// `{origin}/v1/embeddings` path to set. An explicit `mode = "proxy"` with a/// missing or unusable endpoint is a startup ERROR (#1189): silently/// running lexical instead would write the wrong vectors into the index/// while every embedding request appears to succeed.@@ -1607,7 +1612,21 @@ pub struct EmbeddingConfig {/// `"proxy"`, or `"onnx-experimental"`. Unknown values are treated as/// `"auto"`.pub mode: String,- /// OpenAI-compatible endpoint URL (default: `""` — disabled).+ /// OpenAI-compatible embeddings URL (default: `""` — disabled).+ ///+ /// Accepted forms, trailing slash optional (#2355):+ /// * a base whose path ends in `/v1` — `/embeddings` is appended+ /// (`http://localhost:8000/v1` becomes+ /// `http://localhost:8000/v1/embeddings`)+ /// * a bare origin (`http://host:port` or with only `/`) — `/embeddings`+ /// is appended+ /// * a path that already ends in `/embeddings` — used as given+ ///+ /// Any other path refuses to start when [`mode`](Self::mode) is+ /// `"proxy"` and names this URL plus `{origin}/v1/embeddings`. The value+ /// stored here is also the proxy half of the embedder identity, so+ /// changing it after a corpus is indexed is a different embedder even+ /// when the resolved request URL would be the same.pub default_endpoint: String,/// Model name to request from the endpoint (default: `""`).pub default_model: String,diff --git a/engine/crates/xerj-common/src/fsio.rs b/engine/crates/xerj-common/src/fsio.rsindex ad023928d..6d8d241e0 100644--- a/engine/crates/xerj-common/src/fsio.rs+++ b/engine/crates/xerj-common/src/fsio.rs@@ -94,12 +94,20 @@ impl Drop for StagedFile {}fn temp_sibling(path: &Path) -> std::path::PathBuf {- let nonce = format!("{:x}-{:?}", std::process::id(), std::thread::current().id());+ // A counter, not the thread id. One thread stages every side-car of one+ // field and then the next field. On a case-folding volume two destinations+ // that differ only by case (`Page.post` and `page.post`) produced the same+ // temp name, the second `File::create` truncated the first, and the second+ // rename failed with `ERROR_FILE_NOT_FOUND` (os error 2) — the rc.98+ // "publishing FTS side-cars" abort. The thread-id nonce did not save it:+ // both creates ran on that one thread.+ static NONCE: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);+ let nonce = NONCE.fetch_add(1, std::sync::atomic::Ordering::Relaxed);let file_name = path.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_else(|| "file".to_string());- path.with_file_name(format!("{file_name}.tmp.{nonce}"))+ path.with_file_name(format!("{file_name}.tmp.{:x}.{nonce}", std::process::id()))}/// Write `bytes` to a temp file beside `path` and sync it, for a later@@ -171,6 +179,13 @@ pub fn publish_staged(dir: &Path, files: Vec<StagedFile>) -> std::io::Result<()>format!("staged file {:?} is not in {:?}", stray.dest, dir),));}+ // Refuse before any rename. NTFS (and the default Windows volume) treats+ // the two names as one file: the second replace either returns+ // ERROR_FILE_NOT_FOUND once the shared temp is gone, or+ // MOVEFILE_REPLACE_EXISTING silently drops the first field. Layout v3+ // hashes the common pairs before they reach here; this is the backstop+ // for a pair the layout still stores as two literals (`µ`/`μ`, `ς`/`σ`).+ reject_case_folded_duplicate_destinations(&files)?;#[cfg(target_vendor = "apple")]sync_file_full(&std::fs::File::open(dir)?)?;let mut published_dests = Vec::with_capacity(files.len());@@ -218,7 +233,7 @@ pub fn fsync_dir(_dir: &Path) -> std::io::Result<()> {}#[cfg(not(windows))]-fn replace_file_for_durable_write(source: &Path, destination: &Path) -> std::io::Result<()> {+fn replace_file_for_durable_write_raw(source: &Path, destination: &Path) -> std::io::Result<()> {std::fs::rename(source, destination)}@@ -244,7 +259,7 @@ fn require_same_directory(source: &Path, destination: &Path) -> std::io::Result</// this remains a same-volume namespace replacement rather than the API's/// optional copy/delete fallback.#[cfg(windows)]-fn replace_file_for_durable_write(source: &Path, destination: &Path) -> std::io::Result<()> {+fn replace_file_for_durable_write_raw(source: &Path, destination: &Path) -> std::io::Result<()> {const MOVEFILE_REPLACE_EXISTING: u32 = 0x0000_0001;const MOVEFILE_WRITE_THROUGH: u32 = 0x0000_0008;@@ -327,6 +342,87 @@ fn replace_file_for_durable_write(source: &Path, destination: &Path) -> std::io:}}+fn replace_file_for_durable_write(source: &Path, destination: &Path) -> std::io::Result<()> {+ replace_file_for_durable_write_raw(source, destination).map_err(|error| {+ // The OS error's Display already ends with `(os error N)`. Naming both+ // paths is what the rc.98 report was missing: the chain stopped at+ // "publishing FTS side-cars" and the agent could not see which file+ // the volume refused.+ std::io::Error::new(+ error.kind(),+ format!(+ "replacing {} with {}: {error}",+ destination.display(),+ source.display()+ ),+ )+ })+}++/// Key under which a case-folding volume (NTFS, and the default Windows+/// drive) stores `file_name`. Unicode lowercase covers ASCII and the letters+/// whose lower form is the folded form. The extra arms are the NTFS upcase+/// equivalences lowercase does not collapse: U+00B5 `µ` and U+03BC `μ` both+/// upcase to U+039C, and U+03C2 `ς` and U+03C3 `σ` both upcase to U+03A3.+fn case_folding_publish_key(file_name: &str) -> String {+ let mut key = String::with_capacity(file_name.len());+ for character in file_name.chars() {+ push_case_folding_publish_char(character, &mut key);+ }+ key+}++fn push_case_folding_publish_char(character: char, out: &mut String) {+ match character {+ 'µ' | 'μ' => out.push('μ'),+ 'ς' | 'σ' => out.push('σ'),+ 'ı' => out.push('i'),+ 'ſ' => out.push('s'),+ 'K' => out.push('k'),+ 'ß' => out.push_str("ss"),+ other => {+ for lower in other.to_lowercase() {+ match lower {+ 'µ' | 'μ' => out.push('μ'),+ 'ς' | 'σ' => out.push('σ'),+ 'ı' => out.push('i'),+ 'ſ' => out.push('s'),+ 'K' => out.push('k'),+ 'ß' => out.push_str("ss"),+ folded => out.push(folded),+ }+ }+ }+ }+}++fn reject_case_folded_duplicate_destinations(files: &[StagedFile]) -> std::io::Result<()> {+ let mut seen: std::collections::HashMap<String, &Path> = std::collections::HashMap::new();+ for file in files {+ let file_name = file+ .dest+ .file_name()+ .map(|name| name.to_string_lossy().into_owned())+ .unwrap_or_default();+ let key = case_folding_publish_key(&file_name);+ if let Some(previous) = seen.get(&key) {+ if *previous != file.dest.as_path() {+ return Err(std::io::Error::new(+ std::io::ErrorKind::AlreadyExists,+ format!(+ "cannot publish {} and {}: a case-folding filesystem treats them as one file",+ previous.display(),+ file.dest.display()+ ),+ ));+ }+ } else {+ seen.insert(key, file.dest.as_path());+ }+ }+ Ok(())+}+/// Replace `destination` with an already-written, already-synced sibling file/// and wait for the namespace replacement to become durable.///@@ -597,4 +693,63 @@ mod tests {}));}}++ #[test]+ fn staging_one_destination_twice_on_one_thread_publishes_both_temps() {+ // Before the per-call nonce, the second stage truncated the first+ // temp and the second rename failed with not-found. That is the+ // Windows shape of two case-folded side-cars written by one thread;+ // the same destination reproduces the temp collision on Linux.+ let dir = tempfile::tempdir().unwrap();+ let destination = dir.path().join("a.post");+ let first = stage_file(&destination, b"one").unwrap();+ let second = stage_file(&destination, b"two").unwrap();+ publish_staged(dir.path(), vec![first, second]).unwrap();+ assert_eq!(std::fs::read(&destination).unwrap(), b"two");+ }++ #[test]+ fn publish_staged_refuses_names_a_case_folding_volume_would_merge() {+ let dir = tempfile::tempdir().unwrap();+ for (left_name, right_name) in [+ ("Page.fst", "page.fst"),+ ("field.µ.post", "field.μ.post"),+ ("sigma.ς.meta", "sigma.σ.meta"),+ ] {+ let left = dir.path().join(left_name);+ let right = dir.path().join(right_name);+ let staged = vec![+ stage_file(&left, b"left").unwrap(),+ stage_file(&right, b"right").unwrap(),+ ];+ let error = publish_staged(dir.path(), staged).unwrap_err();+ let message = error.to_string();+ assert_eq!(error.kind(), std::io::ErrorKind::AlreadyExists, "{message}");+ assert!(message.contains(left_name), "{message}");+ assert!(message.contains(right_name), "{message}");+ assert!(!left.exists(), "{left_name} was renamed");+ assert!(!right.exists(), "{right_name} was renamed");+ }+ assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 0);+ }++ #[test]+ fn a_failed_replace_names_both_paths_and_the_os_error() {+ let dir = tempfile::tempdir().unwrap();+ let source = dir.path().join("body.post.tmp.1");+ let destination = dir.path().join("body.post");+ std::fs::write(&source, b"new").unwrap();+ // A directory sitting on the destination makes the replace fail+ // after the temp exists, which is the publish_staged failure mode.+ std::fs::create_dir(&destination).unwrap();+ let error = replace_file_durable(&source, &destination).unwrap_err();+ let message = error.to_string();+ assert!(message.contains("body.post"), "{message}");+ assert!(message.contains("body.post.tmp.1"), "{message}");+ assert!(+ message.contains("os error"),+ "the OS code must be visible, got {message}"+ );+ assert!(source.exists(), "a failed replace must keep the temp");+ }}diff --git a/engine/crates/xerj-common/src/thread_spawn.rs b/engine/crates/xerj-common/src/thread_spawn.rsindex ae8960e25..66ca4922e 100644--- a/engine/crates/xerj-common/src/thread_spawn.rs+++ b/engine/crates/xerj-common/src/thread_spawn.rs@@ -402,6 +402,43 @@ where}}+/// Start `job` on a detached thread, for a pool worker that outlives the+/// caller (#1122).+///+/// `Ok` means the thread is running and will be detached when the returned+/// handle is dropped inside this function. `Err` means `job` did not run.+/// This does not consult the watermark gate: a pool is built once, and a+/// soft RSS watermark must not refuse its workers the way it refuses a+/// drain-free thread. `stack_bytes` is the reservation the pool asked for;+/// `None` uses the 2 MiB helper stack.+pub fn spawn_detached(+ name: &str,+ stack_bytes: Option<usize>,+ job: impl FnOnce() + Send + 'static,+) -> io::Result<()> {+ if let Some(stack_bytes) = stack_bytes {+ spawn_detached_with(&StackSpawn { stack_bytes }, name, job)+ } else {+ spawn_detached_with(&StdSpawn, name, job)+ }+}++/// [`spawn_detached`] with an explicit spawner. Tests pass one that fails+/// the way an alternate-stack `ENOMEM` does.+pub fn spawn_detached_with<S, F>(spawner: &S, name: &str, job: F) -> io::Result<()>+where+ S: Spawn + ?Sized,+ F: FnOnce() + Send + 'static,+{+ match try_spawn_with(spawner, true, name, job) {+ Ok(_detached) => Ok(()),+ Err(failure) => {+ drop(failure.job);+ Err(failure.error)+ }+ }+}+/// [`try_spawn`] with an explicit spawner and gate verdict (for tests and/// callers that already sampled the gate).pub fn try_spawn_with<S, F>(@@ -834,6 +871,34 @@ fn set_os_name(bytes: &[u8; 16]) {}}+/// The name of the calling thread, as the spawner set it.+///+/// On Linux and macOS this is the pthread comm. Unix spawn does not go+/// through `Builder::name`, so `std::thread::current().name()` is empty+/// there (#1122). The comm is at most 15 bytes (`xerj-scan-0` fits).+/// Elsewhere this is the std thread name, which `Builder::name` did set.+pub fn current_thread_name() -> Option<String> {+ #[cfg(any(target_os = "linux", target_os = "macos"))]+ {+ let mut buf = [0u8; 16];+ // SAFETY: `buf` is writable for `buf.len()` bytes. On success+ // `pthread_getname_np` writes a NUL-terminated name into it and+ // does not retain the pointer.+ let rc = unsafe {+ libc::pthread_getname_np(libc::pthread_self(), buf.as_mut_ptr().cast(), buf.len())+ };+ if rc != 0 {+ return None;+ }+ let end = buf.iter().position(|byte| *byte == 0).unwrap_or(buf.len());+ String::from_utf8(buf[..end].to_vec()).ok()+ }+ #[cfg(not(any(target_os = "linux", target_os = "macos")))]+ {+ std::thread::current().name().map(str::to_owned)+ }+}+/// Same default std uses for a spawned thread (`2 MiB` on Tier-1). A larger/// stack would reserve more address space than the path this replaces,/// which is the opposite of what a memory-pressure spawn should do.@@ -1008,6 +1073,22 @@ mod tests {assert_eq!(ran.load(Ordering::SeqCst), 1);}+ #[test]+ fn a_detached_worker_spawn_returns_enomem_and_does_not_run_the_job() {+ let ran = Arc::new(AtomicUsize::new(0));+ let job_ran = Arc::clone(&ran);+ let error = spawn_detached_with(&SigaltstackEnomem, "xerj-rayon-0", move || {+ job_ran.store(1, Ordering::SeqCst);+ })+ .expect_err("setup failure is an Err, not a panic");+ assert_eq!(error.raw_os_error(), Some(12));+ assert_eq!(+ ran.load(Ordering::SeqCst),+ 0,+ "a refused pool worker must not run, and must not abort"+ );+ }+#[test]fn an_enomem_setup_failure_returns_the_job_instead_of_panicking() {let ran = Arc::new(AtomicUsize::new(0));@@ -1041,7 +1122,7 @@ mod tests {fn a_successful_spawn_runs_the_job_once_on_the_named_thread() {let (tx, rx) = std::sync::mpsc::channel();let handle = try_spawn_with(&StdSpawn, true, "xerj-t1122", move || {- tx.send(helper_thread_name()).unwrap();+ tx.send(current_thread_name()).unwrap();}).expect("spawn");handle.join().unwrap();@@ -1392,25 +1473,4 @@ mod tests {let rc = unsafe { libc::sigaltstack(std::ptr::null(), &mut stack) };(rc, stack.ss_flags)}-- fn helper_thread_name() -> Option<String> {- #[cfg(any(target_os = "linux", target_os = "macos"))]- {- let mut buf = [0u8; 64];- // SAFETY: `buf` is writable for `buf.len()` bytes and is- // NUL-terminated by pthread_getname_np on success.- let rc = unsafe {- libc::pthread_getname_np(libc::pthread_self(), buf.as_mut_ptr().cast(), buf.len())- };- if rc != 0 {- return None;- }- let end = buf.iter().position(|byte| *byte == 0).unwrap_or(buf.len());- String::from_utf8(buf[..end].to_vec()).ok()- }- #[cfg(not(any(target_os = "linux", target_os = "macos")))]- {- std::thread::current().name().map(str::to_owned)- }- }}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 5a76524ab..86b8ee56a 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -14376,50 +14376,61 @@ impl Index {if let Err(e) = store_for_task.ensure_fts_case_distinguished_field_component_format(){- tracing::warn!(- "merge: case-distinguished FTS filename format preflight failed: {e:#}"- );- return Ok(());+ return degrade_fts_publish(+ &segments_dir_for_task,+ merged_meta.id.as_str(),+ &format!("{e:#}"),+ )+ .map_err(|error| error.to_string());}if let Err(e) = fts_writer.publish_case_distinguished_filename_layout(){- tracing::warn!(- "merge: FTS filename-layout publication failed: {e:#}"- );- return Ok(());+ return degrade_fts_publish(+ &segments_dir_for_task,+ merged_meta.id.as_str(),+ &format!("{e:#}"),+ )+ .map_err(|error| error.to_string());}} else if fts_writer.uses_encoded_field_filename_components() {if let Err(e) = store_for_task.ensure_fts_encoded_field_component_format(){- tracing::warn!(- "merge: encoded FTS filename format preflight failed: {e:#}"- );- return Ok(());+ return degrade_fts_publish(+ &segments_dir_for_task,+ merged_meta.id.as_str(),+ &format!("{e:#}"),+ )+ .map_err(|error| error.to_string());}if let Err(e) =fts_writer.publish_encoded_filename_layout(){- tracing::warn!(- "merge: FTS filename-layout publication failed: {e:#}"- );- return Ok(());+ return degrade_fts_publish(+ &segments_dir_for_task,+ merged_meta.id.as_str(),+ &format!("{e:#}"),+ )+ .map_err(|error| error.to_string());}}if let Err(e) = fts_writer.finish() {- tracing::warn!("merge: FTS build failed: {e:#}");+ degrade_fts_publish(+ &segments_dir_for_task,+ merged_meta.id.as_str(),+ &format!("{e:#}"),+ )+ .map_err(|error| error.to_string())?;}Ok(())});- // LOSS FIREWALL: the preflight already refused every- // input it could not reproduce exactly, so a failure- // here means an input's own side-cars contradict- // themselves. Committing a merged segment whose FTS- // data is missing would silently stop its documents- // matching, so abort the batch — the inputs stay- // live and untouched, exactly as for an unreadable- // stored section.+ // A postings-merge failure, or a side-car publish+ // failure whose leftover files could not be removed,+ // aborts the batch: the inputs stay live. A publish+ // failure that left no FTS files does not: the+ // merged segment is searched by stored scan until a+ // later merge rebuilds full text (#2349).if let Err(message) = built {tracing::error!("merge ABORTED: postings merge failed (inputs \@@ -28632,6 +28643,27 @@ fn build_fts_field_configs(out}+/// Log `error` and remove this segment's FTS artifacts.+///+/// `Ok` means nothing FTS-shaped remains, so publishing the stored segment+/// is safe: a query that finds no side-cars scans the stored section, and a+/// later merge rebuilds full text (#2349). `Err` means a leftover file could+/// not be removed; that partial family must not be published. The error+/// string keeps the original publish failure (path and OS error) and the+/// cleanup failure.+fn degrade_fts_publish(segment_dir: &Path, segment_id: &str, error: &str) -> std::io::Result<()> {+ tracing::error!(+ segment_id,+ "FTS side-car publish failed; this segment stays searchable by stored scan until a merge rebuilds full text: {error}"+ );+ if let Err(cleanup) = xerj_fts::index::discard_segment_fts_files(segment_dir, segment_id) {+ return Err(std::io::Error::other(format!(+ "FTS side-car publish failed ({error}); leftover side-cars could not be removed: {cleanup}"+ )));+ }+ Ok(())+}+/// Open a full FTS reader over every input segment of a merge batch, or/// `None` when this batch has to fall back to re-analysing its documents.///@@ -35249,30 +35281,40 @@ async fn do_flush_shard(fts_writer.add_documents_parallel(&drained_fts_for_build);fts_add_us = t_add.elapsed().as_micros();let t_fin = std::time::Instant::now();- if fts_writer.uses_case_distinguished_field_filename_components() {+ // The data-dir format marker is a fence, not a side-car. If+ // that preflight fails, this flush must fail and leave the+ // documents uncommitted: a retry then writes the marker and+ // the encoded side-cars. Publishing a no-FTS segment here+ // would clear the memtable and skip that recovery.+ let case_distinguished =+ fts_writer.uses_case_distinguished_field_filename_components();+ let encoded =+ !case_distinguished && fts_writer.uses_encoded_field_filename_components();+ if case_distinguished {store_for_warm.ensure_fts_case_distinguished_field_component_format()?;- fts_writer- .publish_case_distinguished_filename_layout()- .map_err(|error| {- xerj_storage::StorageError::Io(std::io::Error::other(format!(- "FTS filename-layout publication failed: {error:#}"- )))- })?;- } else if fts_writer.uses_encoded_field_filename_components() {+ } else if encoded {store_for_warm.ensure_fts_encoded_field_component_format()?;- fts_writer- .publish_encoded_filename_layout()- .map_err(|error| {- xerj_storage::StorageError::Io(std::io::Error::other(format!(- "FTS filename-layout publication failed: {error:#}"- )))- })?;}- fts_writer.finish().map_err(|error| {- xerj_storage::StorageError::Io(std::io::Error::other(format!(- "FTS side-car build failed: {error:#}"- )))- })?;+ // #2349 — a side-car publish that fails on a case-folding+ // volume used to fail this flush, and the aggregate `-docs`+ // refresh then aborted the whole autoindex. When the partial+ // files can be removed, keep the stored segment: search scans+ // it, and a later merge rebuilds full text. The error logged+ // by `degrade_fts_publish` names the path and the OS error.+ let segment_id_for_fts = meta.id.clone();+ let publish_result = (move || -> anyhow::Result<()> {+ if case_distinguished {+ fts_writer.publish_case_distinguished_filename_layout()?;+ } else if encoded {+ fts_writer.publish_encoded_filename_layout()?;+ }+ fts_writer.finish()?;+ Ok(())+ })();+ if let Err(error) = publish_result {+ degrade_fts_publish(&segments_dir, &segment_id_for_fts, &format!("{error:#}"))+ .map_err(xerj_storage::StorageError::Io)?;+ }fts_finish_us = t_fin.elapsed().as_micros();}if prof {@@ -39699,6 +39741,12 @@ pub(crate) fn embedding_execution_identity(// `dimensions` instead, and autoindex compares it on resume. A// provider that swaps weights without changing the name, the// endpoint, or the width is not detectable from here.+ // #2355 completes a `/v1` base (and a bare host) to an embeddings+ // URL on the HTTP request only. The digest stays this configured+ // string: folding the resolved URL in would move identity_sha256+ // for every proxy index recorded before that completion and refuse+ // to open it. `/v1` and `/v1/embeddings` therefore remain distinct+ // identities even when they POST to the same path.let material = format!("proxy-unpinned.v1;configured-model={};endpoint={}",cfg.default_model, cfg.default_endpoint@@ -41177,7 +41225,11 @@ fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::}return match xerj_ai::embed::EmbeddingProxy::new(proxy_config(cfg)) {Ok(p) => {- info!(endpoint = %cfg.default_endpoint, "embedding backend: external proxy");+ info!(+ endpoint = %p.request_url(),+ configured = %cfg.default_endpoint,+ "embedding backend: external proxy"+ );Ok(xerj_ai::Embedder::proxy(p))}Err(e) => {diff --git a/engine/crates/xerj-engine/src/index_guard.rs b/engine/crates/xerj-engine/src/index_guard.rsindex ae424c964..ed0486285 100644--- a/engine/crates/xerj-engine/src/index_guard.rs+++ b/engine/crates/xerj-engine/src/index_guard.rs@@ -65,7 +65,7 @@//! |---|---|//! | `xerj-api::es_compat::spawn_datafeed_task` | **carries the rule** — the one detached path that resolves an index name |//! | `es_compat::search_impl`'s search task | owns one already-authorized `Index`; `terms`/`lookup` targets are resolved earlier, on the request's task |-//! | `es_compat::delete_by_query` (`wait_for_completion=false`) | same: one already-authorized `Index` handle, no `Engine` |+//! | `es_compat::delete_by_query` (`wait_for_completion=false`) | cluster path **carries the rule** — it calls `Engine` to reach shard primaries. The single-node path owns already-authorized `Index` handles and does not |//! | `es_compat::update_by_query` (`wait_for_completion=false`) | cluster path **carries the rule** — it calls `Engine` to reach shard primaries. The single-node path owns already-authorized `Index` handles and does not |//! | `Engine::spawn_pit_sweeper`, `spawn_search_context_sweeper` | started by `Engine::new`, capture only the PIT/scroll maps, touch no index |//! | `Index`'s flush/merge/warm tasks (`index.rs`, `write_publication.rs`) | methods **on** an already-resolved `Index`; the name has already been through the funnel |diff --git a/engine/crates/xerj-engine/src/pools.rs b/engine/crates/xerj-engine/src/pools.rsindex 37345d7f4..4d4ac0602 100644--- a/engine/crates/xerj-engine/src/pools.rs+++ b/engine/crates/xerj-engine/src/pools.rs@@ -304,13 +304,9 @@ fn log_pool_degraded_once(message: impl FnOnce() -> String) {/// fails the build and lets the caller run inline.pub(crate) fn safe_rayon_spawn(thread: rayon::ThreadBuilder) -> std::io::Result<()> {let name = thread.name().unwrap_or("xerj-rayon").to_owned();- match xerj_common::thread_spawn::try_spawn_unguarded(&name, move || thread.run()) {- Ok(_detached) => Ok(()),- Err(failure) => {- drop(failure.job);- Err(failure.error)- }- }+ xerj_common::thread_spawn::spawn_detached(name.as_str(), thread.stack_size(), move || {+ thread.run()+ })}/// Run `op`, and every `par_iter` inside it, on the calling thread.diff --git a/engine/crates/xerj-fts/src/index.rs b/engine/crates/xerj-fts/src/index.rsindex 43b8ca36c..d6bc389b3 100644--- a/engine/crates/xerj-fts/src/index.rs+++ b/engine/crates/xerj-fts/src/index.rs@@ -313,6 +313,107 @@ fn unique_temp_sibling(path: &Path) -> PathBuf {path.with_file_name(format!("{file_name}.tmp.{}.{nonce}", std::process::id()))}+/// Removes `path` on drop unless [`DeleteOnDrop::disarm`] ran. The FST temp+/// is not a [`xerj_common::fsio::StagedFile`] until `stage_written_file`+/// returns; a panic inside `MapBuilder` would otherwise leave the temp behind.+struct DeleteOnDrop {+ path: PathBuf,+ armed: bool,+}++impl DeleteOnDrop {+ fn arm(path: PathBuf) -> Self {+ Self { path, armed: true }+ }++ fn disarm(&mut self) {+ self.armed = false;+ }+}++impl Drop for DeleteOnDrop {+ fn drop(&mut self) {+ if self.armed {+ let _ = fs::remove_file(&self.path);+ }+ }+}++fn panic_payload_message(payload: &Box<dyn std::any::Any + Send>) -> String {+ // Take `&Box<_>`, not `&(dyn Any + Send)`. A function argument of the+ // trait-object type unsizes the box itself, so `downcast_ref` looks for+ // `&str` inside a payload whose concrete type is the box and misses the+ // string `panic!` actually boxed.+ if let Some(message) = payload.downcast_ref::<&str>() {+ (*message).to_owned()+ } else if let Some(message) = payload.downcast_ref::<String>() {+ message.clone()+ } else {+ "non-string panic payload".to_owned()+ }+}++/// `true` when `file_name` is an FTS side-car, layout marker, analyzer marker,+/// or unpublished temp belonging to `segment_id`. Stored-segment files+/// (`.seg`, `.sidx`, `.ids`, `.dv`) are not.+fn is_segment_fts_artifact(segment_id: &str, file_name: &str) -> bool {+ let Some(rest) = file_name+ .strip_prefix(segment_id)+ .and_then(|suffix| suffix.strip_prefix('.'))+ else {+ return false;+ };+ if rest.is_empty() {+ return false;+ }+ if rest.contains(".tmp.") {+ return true;+ }+ matches!(rest, "fts-layout-v2" | "fts-layout-v3" | "ftsan")+ || matches!(+ rest.rsplit_once('.').map(|(_, extension)| extension),+ Some("fst" | "post" | "meta" | "norms")+ )+}++/// Remove this segment's FTS side-cars, filename-layout markers, analyzer+/// marker, and unpublished temps. `.seg`, `.sidx`, `.ids`, and `.dv` stay.+///+/// `Ok(())` means none of those FTS names remain, so the stored segment can+/// be published and searched by a stored-document scan. `Err` names every+/// path that could not be removed, with that path's OS error — a partial+/// family must not be published.+pub fn discard_segment_fts_files(segment_dir: &Path, segment_id: &str) -> std::io::Result<()> {+ let entries = match fs::read_dir(segment_dir) {+ Ok(entries) => entries,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),+ Err(error) => return Err(error),+ };+ let mut failures: Vec<String> = Vec::new();+ for entry in entries {+ let entry = entry?;+ let file_name = entry.file_name();+ let file_name = file_name.to_string_lossy();+ if !is_segment_fts_artifact(segment_id, &file_name) {+ continue;+ }+ let path = entry.path();+ if let Err(error) = fs::remove_file(&path) {+ if error.kind() != std::io::ErrorKind::NotFound {+ failures.push(format!("{}: {error}", path.display()));+ }+ }+ }+ if failures.is_empty() {+ Ok(())+ } else {+ Err(std::io::Error::other(format!(+ "could not remove FTS side-cars for segment {segment_id}: {}",+ failures.join("; ")+ )))+ }+}+/// Derive one FTS side-car path from controlled components only.fn field_sidecar_path(segment_dir: &Path,@@ -667,15 +768,25 @@ fn encode_field_meta_v5(// per-term "needs an entry" decision made in scan order would leave// the terms before the first ttf != df term without their zero// entries, and the reader splits sections by fixed counts.- let ttf_stream_needed = sorted_terms.iter().any(|term| {+ let mut ttf_stream_needed = false;+ for term in sorted_terms {let tp = term_postings.get(term)- .expect("sorted_terms must match term_postings keys");+ .ok_or_else(|| anyhow::anyhow!("field meta is missing postings for term '{term}'"))?;if tp.total_term_frequency < tp.doc_frequency as u64 {- panic!("total_term_frequency below doc_frequency for '{term}'");+ // A panic here aborted the flush task (`resume_unwind`) during+ // indexing. rc.97 reports described that as a panic in the fst+ // merge: this encode runs while the merged field's FST is built.+ bail!(+ "total_term_frequency {} is below doc_frequency {} for term '{term}'",+ tp.total_term_frequency,+ tp.doc_frequency+ );+ }+ if tp.total_term_frequency != tp.doc_frequency as u64 {+ ttf_stream_needed = true;}- tp.total_term_frequency != tp.doc_frequency as u64- });+ }let flags = if ttf_stream_needed {ZFM5_FLAG_TTF_STREAM} else {@@ -689,7 +800,7 @@ fn encode_field_meta_v5(for term in sorted_terms {let tp = term_postings.get(term)- .expect("sorted_terms must match term_postings keys");+ .ok_or_else(|| anyhow::anyhow!("field meta is missing postings for term '{term}'"))?;uvarint_encode(tp.doc_frequency as u64, &mut dfs);if ttf_stream_needed {uvarint_encode(@@ -1142,6 +1253,11 @@ pub struct FtsIndexWriter {/// Zstd effort for the `.meta` (ZFM4) and `.post` (ZPS1) envelopes./// [`ZSTD_DURABLE_LEVEL`] unless [`Self::with_zstd_level`] raised it.zstd_level: i32,+ /// Test-only. When `finish` writes this field, it panics inside the+ /// `catch_unwind` that turns an `fst` builder panic into `Err`. Per+ /// writer, so parallel tests do not share a process-global flag.+ #[cfg(test)]+ panic_on_field: Option<String>,}struct FieldData {@@ -1174,9 +1290,19 @@ impl FtsIndexWriter {fields: HashMap::new(),published_filename_layout: None,zstd_level: ZSTD_DURABLE_LEVEL,+ #[cfg(test)]+ panic_on_field: None,}}+ /// The next [`Self::finish`] panics while writing `field`. The panic is+ /// caught and returned as that field's error, which is what an `fst`+ /// `MapBuilder` assert does on this path.+ #[cfg(test)]+ fn panic_while_writing_field(&mut self, field: impl Into<String>) {+ self.panic_on_field = Some(field.into());+ }+/// The configuration an UNCONFIGURED field is indexed with.////// Pre-#937 this was always [`FieldIndexConfig::default`]@@ -1610,6 +1736,8 @@ impl FtsIndexWriter {let segment_dir = self.segment_dir.clone();let segment_id = self.segment_id.clone();let zstd_level = self.zstd_level;+ #[cfg(test)]+ let panic_on_field = self.panic_on_field.clone();// Drain fields into a Vec so we can parallelise the iterator.// Cloning `stats` before consuming `field_data` — `stats` goes into@@ -1661,16 +1789,40 @@ impl FtsIndexWriter {let results: Vec<Result<(String, FieldStats, Vec<xerj_common::fsio::StagedFile>)>> = fields.into_par_iter().map(|(field_name, stats, field_data)| {- let files = Self::write_field_static(- &segment_dir,- &segment_id,- &field_name,- field_data,- zstd_level,- layout,- )- .with_context(|| format!("writing field '{}'", field_name))?;- Ok((field_name, stats, files))+ // A panic inside `fst::MapBuilder` (its `assert!(out.is_zero())`+ // on a duplicate key the checker missed) or in the meta+ // encoder used to escape the rayon worker and `resume_unwind`+ // the flush task. Indexing then died with no path and no OS+ // error. The same failure is now this field's `Err`.+ #[cfg(test)]+ let panic_this_field = panic_on_field.as_deref() == Some(field_name.as_str());+ let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {+ #[cfg(test)]+ if panic_this_field {+ panic!("simulated fst merge panic");+ }+ Self::write_field_static(+ &segment_dir,+ &segment_id,+ &field_name,+ field_data,+ zstd_level,+ layout,+ )+ }));+ match outcome {+ Ok(result) => {+ let files =+ result.with_context(|| format!("writing field '{field_name}'"))?;+ Ok((field_name, stats, files))+ }+ Err(payload) => {+ bail!(+ "writing field '{field_name}' panicked: {}",+ panic_payload_message(&payload)+ )+ }+ }}).collect();@@ -1807,6 +1959,10 @@ impl FtsIndexWriter {// platform-specific durable replacement publishes it (RC4 W1 #10 —// see the postings note above).let fst_tmp = unique_temp_sibling(&fst_path);+ // Declared before the builder so unwind drops the file (closes it)+ // and only then deletes the temp. Armed until `stage_written_file`+ // takes ownership.+ let mut fst_tmp_cleanup = DeleteOnDrop::arm(fst_tmp.clone());let fst_file = BufWriter::new(File::create(&fst_tmp).with_context(|| format!("creating FST file {:?}", fst_tmp))?,);@@ -1830,6 +1986,7 @@ impl FtsIndexWriter {xerj_common::fsio::stage_written_file(&fst_tmp, &fst_path, fst_out.get_ref()).with_context(|| "fsyncing FST")?,);+ fst_tmp_cleanup.disarm();drop(fst_out);// 4. Write meta in the ZFM5 format — columnar varint streams@@ -4202,4 +4359,106 @@ mod tests {assert!(segment_filename_layout_v3_marker_path(dir.path(), &segment).is_file());}}++ #[test]+ fn total_term_frequency_below_doc_frequency_is_an_error() {+ let stats = FieldStats {+ total_docs: 2,+ total_field_length: 2,+ };+ let mut map = HashMap::new();+ map.insert(+ "alpha".to_owned(),+ TermPostings {+ doc_frequency: 2,+ total_term_frequency: 1,+ postings_offset: 0,+ postings_length: 4,+ },+ );+ let error = encode_field_meta_v5(&stats, true, &["alpha".to_owned()], &map, 3).unwrap_err();+ let message = format!("{error:#}");+ assert!(message.contains("alpha"), "{message}");+ assert!(message.contains("total_term_frequency"), "{message}");+ assert!(message.contains("doc_frequency"), "{message}");+ }++ #[test]+ fn a_panic_while_writing_a_field_is_an_error_naming_the_field() {+ let dir = TempDir::new().unwrap();+ let mut writer = FtsIndexWriter::new(dir.path(), "seg-panic", make_registry());+ writer.panic_while_writing_field("body");+ writer.add_document(+ 0,+ &[("body".to_owned(), FieldValues::from("needle"))]+ .into_iter()+ .collect(),+ );+ let error = writer.finish().unwrap_err();+ let message = format!("{error:#}");+ assert!(message.contains("simulated fst merge panic"), "{message}");+ assert!(message.contains("body"), "{message}");+ assert!(+ !dir.path().join("seg-panic.body.fst").exists(),+ "a panicked field write must not leave a published side-car"+ );+ }++ #[test]+ fn a_blocked_side_car_publish_names_the_path_and_the_os_error() {+ let dir = TempDir::new().unwrap();+ let mut writer = FtsIndexWriter::new(dir.path(), "seg-block", make_registry());+ writer.add_document(+ 0,+ &[("body".to_owned(), FieldValues::from("needle"))]+ .into_iter()+ .collect(),+ );+ // The publish renames onto these names. A directory at every+ // destination makes whichever file is published first fail, the+ // same way a locked or unreplaceable side-car fails on Windows.+ for extension in ["fst", "post", "meta", "norms"] {+ fs::create_dir(dir.path().join(format!("seg-block.body.{extension}"))).unwrap();+ }+ let error = writer.finish().unwrap_err();+ let message = format!("{error:#}");+ assert!(+ message.contains("seg-block.body."),+ "the failing side-car path must be in the error, got {message}"+ );+ assert!(+ message.contains("os error"),+ "the OS code must be in the error, got {message}"+ );+ }++ #[test]+ fn discard_segment_fts_files_keeps_the_stored_segment_and_names_a_stuck_file() {+ let dir = TempDir::new().unwrap();+ fs::write(dir.path().join("seg.body.fst"), b"fst").unwrap();+ fs::write(dir.path().join("seg.fts-layout-v3"), b"marker").unwrap();+ fs::write(dir.path().join("seg.seg"), b"stored").unwrap();+ fs::write(dir.path().join("seg.ids"), b"ids").unwrap();+ fs::write(dir.path().join("seg.dv"), b"dv").unwrap();+ fs::write(dir.path().join("other.body.fst"), b"other").unwrap();+ discard_segment_fts_files(dir.path(), "seg").unwrap();+ assert!(!dir.path().join("seg.body.fst").exists());+ assert!(!dir.path().join("seg.fts-layout-v3").exists());+ assert_eq!(fs::read(dir.path().join("seg.seg")).unwrap(), b"stored");+ assert_eq!(fs::read(dir.path().join("seg.ids")).unwrap(), b"ids");+ assert_eq!(fs::read(dir.path().join("seg.dv")).unwrap(), b"dv");+ assert_eq!(+ fs::read(dir.path().join("other.body.fst")).unwrap(),+ b"other"+ );++ let stuck = dir.path().join("seg.body.post");+ fs::create_dir(&stuck).unwrap();+ fs::write(stuck.join("held"), b"x").unwrap();+ let error = discard_segment_fts_files(dir.path(), "seg").unwrap_err();+ let message = error.to_string();+ assert!(message.contains("seg.body.post"), "{message}");+ assert!(message.contains("os error"), "{message}");+ assert!(stuck.join("held").exists());+ }}diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rsindex 3ae947fab..f7c2c979c 100644--- a/engine/crates/xerj-mcp/src/lib.rs+++ b/engine/crates/xerj-mcp/src/lib.rs@@ -788,7 +788,13 @@ pub fn tool_specs_with(include_hub: bool) -> Value {located or the passage start line is unknown. `start_line` remains \the passage start. \Pass `max_tokens` to cap the response at a token budget (never exceeded; \- see that argument for the tokenizer and the locator-only rule).",+ see that argument for the tokenizer and the locator-only rule). \+ If this prefix was autoindexed from a folder on this machine and a file \+ there is newer than the last run, or an indexed file has been removed, \+ the response adds `_xerj.hints` code `index_stale` with the last index \+ time and a reindex command. Hits are returned with that hint. The check \+ stats at most 4096 entries, stops after 100ms, and skips hidden and \+ ignored paths.","inputSchema": {"type": "object","properties": {@@ -1965,7 +1971,7 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {match result {Ok((method, path, body)) => {let match_query = search_match_query(name, &args);- engine_request(+ let result = engine_request(ctx,method,&path,@@ -1974,7 +1980,15 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {demote_page,match_query.as_deref(),)- .await+ .await;+ // After the search, so a failed call does not pay for the catalog+ // read and the check cannot take a connection the search needs.+ let hint = if result.get("isError") == Some(&Value::Bool(true)) {+ None+ } else {+ search_staleness(ctx, name, &args).await+ };+ apply_staleness(result, hint)}Err(msg) => tool_text(msg, true),}@@ -3178,6 +3192,53 @@ fn search_match_query(name: &str, args: &Value) -> Option<String> {}}+/// Catalog + mtime check for `xerj_search`, after the search has returned.+///+/// The walk uses a blocking HTTP client, so it runs on a blocking thread.+/// It starts only once the search result is in hand: a second connection+/// during the search would be a different request, and a failed search has+/// nothing to annotate. Other tools do not share this path: `xerj code`'s+/// 30-day refusal stays on `xerj_code_search`.+async fn search_staleness(+ ctx: &Ctx,+ name: &str,+ args: &Value,+) -> Option<xerj_autoindex::staleness::StalenessHint> {+ if name != "xerj_search" {+ return None;+ }+ let prefix = xerj_autoindex::search::prefix_from_scope(+ args.get("prefix").and_then(Value::as_str),+ args.get("index").and_then(Value::as_str),+ )?;+ let url = ctx.base_url.clone();+ let key = api_key_from_header(ctx.auth.clone());+ tokio::task::spawn_blocking(move || {+ xerj_autoindex::staleness::check(&url, key.as_deref(), &prefix)+ })+ .await+ .ok()+ .flatten()+}++/// Attach a staleness hint after the token budget so a tight budget cannot+/// drop the warning. An error result is returned unchanged.+fn apply_staleness(result: Value, hint: Option<xerj_autoindex::staleness::StalenessHint>) -> Value {+ let Some(hint) = hint else {+ return result;+ };+ if result.get("isError") == Some(&Value::Bool(true)) {+ return result;+ }+ let Some(text) = result.pointer("/content/0/text").and_then(Value::as_str) else {+ return result;+ };+ tool_text(+ xerj_autoindex::staleness::merge_response_json(text, &hint),+ false,+ )+}+async fn engine_request(ctx: &Ctx,method: Method,@@ -5441,6 +5502,10 @@ mod tests {desc.contains("omitted when no term is located"),"a missing match_line is not a line number: {desc}");+ assert!(+ desc.contains("index_stale") && desc.contains("4096"),+ "xerj_search must describe the staleness hint: {desc}"+ );assert!(at("xerj_field_report") < at("xerj_corpus_feedback"));assert!(at("xerj_corpus_feedback") < at("xerj_hub_find"));let field = specs@@ -5493,6 +5558,32 @@ mod tests {assert_eq!(autoindex_tools, vec!["xerj_autoindex_remove"]);}+ #[test]+ fn a_staleness_hint_is_added_to_a_search_result_and_an_error_is_left_alone() {+ let hint = xerj_autoindex::staleness::StalenessHint {+ code: "index_stale",+ reason: "The index for prefix `ok-knowledge` was last built at 2026-10-23T23:46:52Z. `docs/page.md` is newer than that index, so a page on disk can be missing from these results. Reindex with: xerj autoindex /corpus --prefix ok-knowledge".into(),+ indexed_at: "2026-10-23T23:46:52Z".into(),+ root: "/corpus".into(),+ run_id: "run-20261023T234652Z-2c98-0000".into(),+ prefix: "ok-knowledge".into(),+ path: "docs/page.md".into(),+ change: "newer",+ };+ let ok = tool_text(r#"{"hits":{"hits":[]}}"#, false);+ let merged = apply_staleness(ok, Some(hint.clone()));+ assert_eq!(merged["isError"], false);+ let text = merged["content"][0]["text"].as_str().unwrap();+ let parsed: Value = serde_json::from_str(text).unwrap();+ assert_eq!(parsed["_xerj"]["hints"][0]["code"], "index_stale");+ assert_eq!(parsed["_xerj"]["hints"][0]["path"], "docs/page.md");+ assert!(parsed["hits"]["hits"].as_array().unwrap().is_empty());++ let err = tool_text("XERJ returned HTTP 400: no", true);+ let kept = apply_staleness(err.clone(), Some(hint));+ assert_eq!(kept, err);+ }+/// A process with no hub signing key must not advertise tools that can/// only answer with the no-key message. The count `--help` prints is that/// same list, so the two cannot disagree.diff --git a/engine/crates/xerj-mcp/src/map.rs b/engine/crates/xerj-mcp/src/map.rsindex 042b766ea..e574817c2 100644--- a/engine/crates/xerj-mcp/src/map.rs+++ b/engine/crates/xerj-mcp/src/map.rs@@ -30,6 +30,8 @@//! `fields_omitted`. A single field that alone exceeds the cap is dropped//! whole, which is what guarantees the ladder terminates.+use std::collections::BTreeMap;+use serde_json::{json, Map, Value};use crate::{tool_text, Ctx};@@ -629,6 +631,130 @@ fn field_count_means(mapping_read: bool) -> &'static str {}}+/// Note attached to a map that has family warnings. The array itself is the+/// same objects the autoindex run summary stores on the catalog run document.+const FAMILY_JUNK_NOTE: &str = "Each entry is a file family where every file, or a strict \+ majority of its files, was junked during autoindex and not indexed. `top_reason` is the \+ most common junk reason (ties break by Unicode code-point order). Clean families and \+ minority junk are omitted. Exit codes are unchanged.";++/// Latest run per `prefix`, then that run's `family_junk_warnings`, if any.+///+/// An older run's warnings are not reported once a newer run of the same+/// prefix exists: the latest run is the summary. `started` is compared as a+/// string (RFC3339 sorts chronologically). Equal `started` values break by+/// `run_id`, code-point order, greater first — not by search hit order.+pub(crate) fn family_warnings_from_runs(runs: &[Value]) -> Vec<Value> {+ let mut latest: BTreeMap<&str, &Value> = BTreeMap::new();+ for run in runs {+ let prefix = run.get("prefix").and_then(Value::as_str).unwrap_or("");+ match latest.get(prefix) {+ Some(prev) if !run_is_newer(run, prev) => {}+ _ => {+ latest.insert(prefix, run);+ }+ }+ }+ let mut out = Vec::new();+ for (prefix, run) in latest {+ let Some(warnings) = run.get("family_junk_warnings").and_then(Value::as_array) else {+ continue;+ };+ for warning in warnings {+ let mut copy = warning.clone();+ if let Some(obj) = copy.as_object_mut() {+ if !prefix.is_empty() && !obj.contains_key("prefix") {+ obj.insert("prefix".into(), json!(prefix));+ }+ if let Some(id) = run.get("run_id").filter(|_| !obj.contains_key("run_id")) {+ obj.insert("run_id".into(), id.clone());+ }+ }+ out.push(copy);+ }+ }+ out.sort_by(|left, right| {+ let key = |value: &Value, field: &str| {+ value+ .get(field)+ .and_then(Value::as_str)+ .unwrap_or("")+ .to_string()+ };+ key(left, "prefix")+ .cmp(&key(right, "prefix"))+ .then(key(left, "family").cmp(&key(right, "family")))+ });+ out+}++fn run_field<'a>(value: &'a Value, field: &str) -> &'a str {+ value.get(field).and_then(Value::as_str).unwrap_or("")+}++fn run_is_newer(candidate: &Value, current: &Value) -> bool {+ match run_field(candidate, "started").cmp(run_field(current, "started")) {+ std::cmp::Ordering::Greater => true,+ std::cmp::Ordering::Less => false,+ std::cmp::Ordering::Equal => run_field(candidate, "run_id") > run_field(current, "run_id"),+ }+}++/// Add `family_junk_warnings` to a rendered map. A map with nothing to warn+/// about is returned unchanged, so a clean catalog keeps the response it had.+pub(crate) fn attach_family_junk_warnings(rendered: &str, runs: &[Value]) -> String {+ let warnings = family_warnings_from_runs(runs);+ if warnings.is_empty() {+ return rendered.to_string();+ }+ let Ok(mut value) = serde_json::from_str::<Value>(rendered) else {+ return rendered.to_string();+ };+ let Some(obj) = value.as_object_mut() else {+ return rendered.to_string();+ };+ obj.insert("family_junk_warnings".into(), Value::Array(warnings));+ obj.insert("family_junk_warnings_note".into(), json!(FAMILY_JUNK_NOTE));+ value.to_string()+}++/// Run documents from the catalog. Failure is an empty list: the field map+/// is still returned. No server-side sort — `started` is not a date on every+/// catalog this binary still reads.+async fn fetch_run_sources(ctx: &Ctx) -> Vec<Value> {+ let body = json!({+ "size": 500,+ "query": {"term": {"doc_kind": "run"}},+ "_source": ["prefix", "run_id", "started", "family_junk_warnings"]+ });+ let url = format!("{}/{CATALOG_INDEX}/_search", ctx.base_url);+ let mut req = ctx.client.post(&url).json(&body);+ if let Some(auth) = &ctx.auth {+ req = req.header("Authorization", auth);+ }+ let Ok(resp) = req.send().await else {+ return Vec::new();+ };+ if !resp.status().is_success() {+ return Vec::new();+ }+ let Ok(text) = resp.text().await else {+ return Vec::new();+ };+ let Ok(value) = serde_json::from_str::<Value>(&text) else {+ return Vec::new();+ };+ value+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .map(|hits| {+ hits.iter()+ .filter_map(|hit| hit.get("_source").cloned())+ .collect()+ })+ .unwrap_or_default()+}+fn annotate_field_count(rendered: &str, mapping_read: bool) -> String {let Ok(mut value) = serde_json::from_str::<Value>(rendered) else {return rendered.to_string();@@ -709,10 +835,16 @@ pub(crate) async fn run(ctx: &Ctx, args: &Value) -> Value {let mapping = fetch_mappings(ctx, &index_names(&v)).await;let mapping_read = mapping.is_some();merge_mapped_fields(&mut v, mapping.as_ref());- tool_text(- annotate_field_count(&render_response(&v, &parsed), mapping_read),- false,- )+ let rendered =+ annotate_field_count(&render_response(&v, &parsed), mapping_read);+ // #2370: the field map lists datasets that were indexed.+ // A family that was entirely junked never becomes one,+ // so the map used to look complete. The run document+ // carries `family_junk_warnings` when that happened.+ // A failed read leaves the map as it was — the field+ // list is still the answer to "what can I query".+ let runs = fetch_run_sources(ctx).await;+ tool_text(attach_family_junk_warnings(&rendered, &runs), false)}Err(e) => tool_text(format!("xerj_map: catalog reply was not JSON ({e}): {text}"),@@ -1281,4 +1413,142 @@ mod tests {assert!(text.contains("xerj mcp does not start a node"), "{text}");assert!(!text.contains("\"indexes\""), "{text}");}++ fn warning(family: &str, files: u64, junk: u64, reason: &str) -> Value {+ json!({+ "family": family,+ "files": files,+ "junk_files": junk,+ "indexed_files": files - junk,+ "extent": if junk == files { "all" } else { "most" },+ "top_reason": reason,+ "top_reason_files": junk+ })+ }++ fn run_doc(prefix: &str, run_id: &str, started: &str, warnings: Vec<Value>) -> Value {+ json!({+ "prefix": prefix,+ "run_id": run_id,+ "started": started,+ "family_junk_warnings": warnings+ })+ }++ #[test]+ fn a_clean_map_response_is_unchanged() {+ let rendered = r#"{"matched":1,"indexes":[]}"#;+ assert_eq!(+ attach_family_junk_warnings(rendered, &[]),+ rendered,+ "no runs"+ );+ let clean = run_doc("ax", "run-2", "2026-10-24T00:00:02Z", vec![]);+ assert_eq!(+ attach_family_junk_warnings(rendered, &[clean]),+ rendered,+ "a run with an empty warning list adds nothing"+ );+ }++ #[test]+ fn the_map_names_an_all_junked_family_from_the_latest_run() {+ let older = run_doc(+ "ax",+ "run-1",+ "2026-10-24T00:00:01Z",+ vec![warning("html", 2, 2, "old reason")],+ );+ let latest = run_doc(+ "ax",+ "run-2",+ "2026-10-24T00:00:02Z",+ vec![warning(+ "pdf",+ 4,+ 4,+ "could not start the isolated PDF parser",+ )],+ );+ // Hit order is not the decision: the older run is listed last.+ let rendered =+ attach_family_junk_warnings(r#"{"matched":0,"indexes":[]}"#, &[latest, older]);+ let out: Value = serde_json::from_str(&rendered).unwrap();+ let warnings = out["family_junk_warnings"].as_array().unwrap();+ assert_eq!(warnings.len(), 1, "{out}");+ assert_eq!(warnings[0]["family"], "pdf");+ assert_eq!(warnings[0]["files"], 4);+ assert_eq!(warnings[0]["junk_files"], 4);+ assert_eq!(warnings[0]["extent"], "all");+ assert_eq!(+ warnings[0]["top_reason"],+ "could not start the isolated PDF parser"+ );+ assert_eq!(warnings[0]["prefix"], "ax");+ assert_eq!(warnings[0]["run_id"], "run-2");+ assert!(out["family_junk_warnings_note"]+ .as_str()+ .unwrap()+ .contains("majority"));+ // The field map itself is still there.+ assert_eq!(out["matched"], 0);+ assert!(out["indexes"].as_array().unwrap().is_empty());+ }++ #[test]+ fn a_mostly_junked_family_is_reported_and_a_clean_family_is_not() {+ let run = run_doc(+ "papers",+ "run-9",+ "2026-10-24T12:00:00Z",+ vec![warning("docx", 4, 3, "encrypted")],+ );+ let rendered = attach_family_junk_warnings(+ r#"{"matched":1,"indexes":[{"index":"papers-notes"}]}"#,+ &[run],+ );+ let out: Value = serde_json::from_str(&rendered).unwrap();+ let warnings = out["family_junk_warnings"].as_array().unwrap();+ assert_eq!(warnings.len(), 1);+ assert_eq!(warnings[0]["family"], "docx");+ assert_eq!(warnings[0]["extent"], "most");+ assert_eq!(warnings[0]["junk_files"], 3);+ assert_eq!(warnings[0]["files"], 4);+ assert_eq!(warnings[0]["indexed_files"], 1);+ assert!(+ warnings.iter().all(|w| w["family"] != "code"),+ "a clean family is not added by the map layer: {warnings:?}"+ );+ }++ #[test]+ fn equal_start_times_break_by_run_id_and_prefixes_stay_apart() {+ let low = run_doc(+ "ax",+ "run-a",+ "2026-10-24T00:00:00Z",+ vec![warning("pdf", 1, 1, "low")],+ );+ let high = run_doc(+ "ax",+ "run-b",+ "2026-10-24T00:00:00Z",+ vec![warning("pdf", 2, 2, "high")],+ );+ let other = run_doc(+ "bench",+ "run-1",+ "2026-10-23T00:00:00Z",+ vec![warning("csv", 3, 2, "header")],+ );+ let warnings = family_warnings_from_runs(&[low, other, high]);+ assert_eq!(warnings.len(), 2, "{warnings:?}");+ // Sorted by prefix, then family: ax before bench.+ assert_eq!(warnings[0]["prefix"], "ax");+ assert_eq!(warnings[0]["run_id"], "run-b");+ assert_eq!(warnings[0]["top_reason"], "high");+ assert_eq!(warnings[1]["prefix"], "bench");+ assert_eq!(warnings[1]["family"], "csv");+ assert_eq!(warnings[1]["extent"], "most");+ }}diff --git a/engine/crates/xerj-mcp/tests/token_budget_identity.rs b/engine/crates/xerj-mcp/tests/token_budget_identity.rsindex 786b46ec4..60941f0f0 100644--- a/engine/crates/xerj-mcp/tests/token_budget_identity.rs+++ b/engine/crates/xerj-mcp/tests/token_budget_identity.rs@@ -22,6 +22,12 @@ async fn json_budget_identity_survives_http_dispatch_and_source_projection() {.collect();let requests = Arc::new(Mutex::new(Vec::<Value>::new()));let seen = requests.clone();+ // `xerj_search` also POSTs `/autoindex-catalog/_search` after the+ // query (#2345). That probe is not the engine request the budget+ // must leave alone, and recording it in the same list pairs a search+ // body with a catalog body.+ let catalog = Arc::new(Mutex::new(Vec::<Value>::new()));+ let seen_catalog = catalog.clone();let app = Router::new().route("/:index/_mapping",@@ -31,23 +37,34 @@ async fn json_budget_identity_survives_http_dispatch_and_source_projection() {).route("/:index/_search",- axum::routing::post(move |Json(body): Json<Value>| {- let mut hits = hits.clone();- seen.lock().unwrap().push(body.clone());- async move {- // Model the engine's projection: the plain-query default- // must actually request identity fields, not just consume- // fixtures that always return them.- if let Some(fields) = body.get("_source").and_then(Value::as_array) {- for hit in &mut hits {- hit["_source"].as_object_mut().unwrap().retain(|key, _| {- fields.iter().any(|v| v.as_str() == Some(key.as_str()))- });+ axum::routing::post(+ move |axum::extract::Path(index): axum::extract::Path<String>,+ Json(body): Json<Value>| {+ let mut hits = hits.clone();+ let catalog_probe = index == "autoindex-catalog";+ if catalog_probe {+ seen_catalog.lock().unwrap().push(body.clone());+ } else {+ seen.lock().unwrap().push(body.clone());+ }+ async move {+ if catalog_probe {+ return Json(json!({"hits":{"hits":[]}}));+ }+ // Model the engine's projection: the plain-query default+ // must actually request identity fields, not just consume+ // fixtures that always return them.+ if let Some(fields) = body.get("_source").and_then(Value::as_array) {+ for hit in &mut hits {+ hit["_source"].as_object_mut().unwrap().retain(|key, _| {+ fields.iter().any(|v| v.as_str() == Some(key.as_str()))+ });+ }}+ Json(json!({"hits":{"total":{"value":2,"relation":"eq"},"hits":hits}}))}- Json(json!({"hits":{"total":{"value":2,"relation":"eq"},"hits":hits}}))- }- }),+ },+ ),);let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();let url = format!("http://{}", listener.local_addr().unwrap());@@ -107,6 +124,11 @@ async fn json_budget_identity_survives_http_dispatch_and_source_projection() {}{let requests = requests.lock().unwrap();+ assert!(+ requests.len() >= 2 && requests.len().is_multiple_of(2),+ "each call is made with and without a budget: {}",+ requests.len()+ );for pair in requests.chunks_exact(2) {assert_eq!(pair[0], pair[1],@@ -114,6 +136,18 @@ async fn json_budget_identity_survives_http_dispatch_and_source_projection() {);}}+ {+ // Two `xerj_search` calls, each with and without a budget.+ let catalog = catalog.lock().unwrap();+ assert_eq!(catalog.len(), 4, "{catalog:?}");+ for body in catalog.iter() {+ assert_eq!(body["query"]["bool"]["must"][0]["term"]["doc_kind"], "run");+ assert_eq!(+ body["query"]["bool"]["must"][1]["bool"]["should"][0]["term"]["prefix"],+ "ax"+ );+ }+ }server.abort();let _ = server.await;}diff --git a/engine/crates/xerj-rerank/tests/egress_inventory.rs b/engine/crates/xerj-rerank/tests/egress_inventory.rsindex 74eb0e1d8..7a7f3a2b2 100644--- a/engine/crates/xerj-rerank/tests/egress_inventory.rs+++ b/engine/crates/xerj-rerank/tests/egress_inventory.rs@@ -99,6 +99,18 @@ const KNOWN: &[(&str, Role)] = &[// the operator's own node, so DATA_EGRESS and the air-gapped rows do// not change.("xerj-autoindex/src/xc.rs", Role::Client("`xerj code`")),+ // #2345: after a successful `xerj search` or MCP `xerj_search`,+ // staleness.rs POSTs one catalog search to the node URL that search+ // already used (`--url`, `XERJ_URL`, else localhost:9200). The body is a+ // `doc_kind: run` prefix query, and a `doc_kind: file` query only when a+ // directory mtime says something was removed. The API key goes only when+ // one is configured. No document text and no user query leave, and the+ // peer is the operator's own node, so DATA_EGRESS and the air-gapped+ // rows do not change.+ (+ "xerj-autoindex/src/staleness.rs",+ Role::Client("`xerj search`"),+ ),("xerj-autoindex/src/feedback.rs",Role::Client("`xerj feedback --open-pr`"),diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 99ad847f8..86c49621b 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -330,7 +330,7 @@ fn help_text(feedback: bool) -> String {\x20\x20\x20\x20xerj autoindex <folder> [OPTIONS] zero-config discovery + indexing — the\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20flagship path for most agents (see xerj\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20autoindex --help)\n\- \x20\x20\x20\x20xerj autoindex map print the discovered data map\n\+ \x20\x20\x20\x20xerj autoindex map [folder] data map of indexed data, or preview a folder\n\\x20\x20\x20\x20xerj autoindex remove --prefix <P> drop one autoindexed folder, after a plan\n\\x20\x20\x20\x20xerj code <corpus> \"<query>\" reference-coding retrieval over a corpus\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20of peer projects (see xerj code --help)\n\@@ -388,7 +388,10 @@ fn help_text(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20the vectors written when they were indexed. Changing\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20--embed-mode does not re-embed an existing index; re-index.\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20Lexical is the fast default.\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20proxy external OpenAI-compatible /v1/embeddings endpoint\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20proxy external OpenAI-compatible embeddings endpoint.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20embedding.default_endpoint may be a /v1 base, a bare host,\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20or the full /v1/embeddings URL (a trailing slash is ignored).\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20Any other path refuses to start and names the URL to set.\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20auto proxy if embedding.default_endpoint is set, else lexical\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20onnx-experimental local all-MiniLM-L6-v2-compatible FP32\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20ONNX prototype; width 384, named BERT inputs and\n\@@ -450,7 +453,7 @@ fn help_text(feedback: bool) -> String {\x20\x20\x20\x20xerj batch-search --input FILE ordered batch retrieval, JSON in/out (see xerj batch-search --help)\n\\x20\x20\x20\x20xerj index <opts> direct NDJSON → engine ingest (see xerj index --help)\n\\x20\x20\x20\x20xerj autoindex <folder> [opts] zero-config folder discovery + indexing (see xerj autoindex --help)\n\- \x20\x20\x20\x20xerj autoindex map print the discovered data map\n\+ \x20\x20\x20\x20xerj autoindex map [folder] data map of indexed data, or a read-only folder preview\n\\x20\x20\x20\x20xerj search \"<text>\" retrieve code/passages from a running node — the\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20query client, no wrapper scripts (see xerj search --help)\n\\x20\x20\x20\x20xerj def \"<symbol>\" go-to-definition: where is this symbol defined, and\n\@@ -960,10 +963,18 @@ fn load_config(args: &CliArgs) -> Result<Config> {remove mode=proxy (use auto) if a lexical fallback is wanted");}- if let Err(e) =- xerj_ai::embed::EmbeddingProxy::new(xerj_engine::index::proxy_config(&cfg.embedding))+ match xerj_ai::embed::EmbeddingProxy::new(xerj_engine::index::proxy_config(&cfg.embedding)){- anyhow::bail!("embedding.mode=proxy failed to initialize: {e}");+ Ok(proxy) => {+ info!(+ endpoint = %proxy.request_url(),+ configured = %cfg.embedding.default_endpoint,+ "embedding.mode=proxy request URL"+ );+ }+ Err(e) => {+ anyhow::bail!("embedding.mode=proxy failed to initialize: {e}");+ }}}@@ -3007,17 +3018,120 @@ const _: () = {assert!(RT_THREAD_STACK_SIZE - PAINLESS_WORST_CASE > 1024 * 1024);};+/// How long an idle tokio blocking thread stays up.+///+/// Tokio's default is 10 seconds. After that the thread exits, and the next+/// `spawn_blocking` — including the one `block_in_place` makes so a search+/// can hand its worker core to another thread — calls+/// `std::thread::Builder::spawn`. The child panics if it cannot map an+/// alternate signal stack, and `panic = "abort"` kills the node. That is+/// the #1122 abort on the path the rayon pools no longer take. A day is+/// long enough that a thread started at boot is still there for the+/// indexing run that used to create it again while RSS was already at the+/// cap. `Instant` addition of a day does not overflow.+const BLOCKING_THREAD_KEEP_ALIVE: std::time::Duration =+ std::time::Duration::from_secs(24 * 60 * 60);++/// How many OS threads the blocking pool may hold, workers included.+///+/// Tokio's `max_blocking_threads` counts only the threads *beyond* the+/// workers. The pool's real cap is that value plus the worker count+/// (`runtime/builder.rs`: `max_blocking_threads + worker_threads`), and+/// `Runtime::build` spends one slot per worker. Search then calls+/// `block_in_place`, which queues `run(worker)` on another slot, and flush+/// `spawn_blocking` needs a slot that is not one of those. This is the+/// whole pool: twice the workers, plus a slice, and at most 1024 once the+/// worker count itself is under that. A worker count already past 1024+/// keeps one extra thread, because `max_blocking_threads(0)` panics and a+/// pool with no non-worker thread cannot run `block_in_place`. Past the+/// cap a later call queues instead of calling `pthread_create`.+fn blocking_thread_cap(worker_threads: usize) -> usize {+ let workers = worker_threads.max(1);+ let spare_for_blocking_work = (workers / 8).max(2);+ let wanted = workers+ .saturating_mul(2)+ .saturating_add(spare_for_blocking_work);+ let floor = workers.saturating_add(1);+ wanted.max(floor).min(1024.max(floor))+}++/// The `max_blocking_threads` value for [`blocking_thread_cap`]: every+/// slot that is not already a worker. [`build_runtime`] starts all of+/// them, so a later `spawn_blocking` finds the pool full and queues.+fn additional_blocking_threads(worker_threads: usize) -> usize {+ blocking_thread_cap(worker_threads).saturating_sub(worker_threads.max(1))+}+/// Build the production multi-threaded Tokio runtime with the deliberate worker/// count and [`RT_THREAD_STACK_SIZE`]. Factored out of `main` so the exact/// builder configuration the server boots with is exercised by a test.fn build_runtime(worker_threads: usize) -> Result<tokio::runtime::Runtime> {- tokio::runtime::Builder::new_multi_thread()+ let blocking = additional_blocking_threads(worker_threads);+ let rt = tokio::runtime::Builder::new_multi_thread().enable_all().worker_threads(worker_threads)+ .max_blocking_threads(blocking)+ .thread_keep_alive(BLOCKING_THREAD_KEEP_ALIVE).thread_stack_size(RT_THREAD_STACK_SIZE).thread_name("xerj-rt").build()- .context("build tokio runtime")+ .context("build tokio runtime")?;+ warm_blocking_pool(&rt, blocking);+ Ok(rt)+}++/// Start `spare` blocking threads now, while the process is still near its+/// idle RSS, and then let them go idle.+///+/// `Runtime::build` has already turned `worker_threads` blocking threads+/// into workers. [`additional_blocking_threads`] is the rest of the cap.+/// Those would otherwise be created on the first search or flush, which+/// is when the box is out of memory and the spawn aborts the process. A+/// shortfall is logged. The server still boots: the cap remains, and a+/// later spawn stops at it.+fn warm_blocking_pool(rt: &tokio::runtime::Runtime, spare: usize) {+ if spare == 0 {+ return;+ }+ let started = Arc::new(AtomicUsize::new(0));+ let release = Arc::new(AtomicBool::new(false));+ let pair = Arc::new((std::sync::Mutex::new(()), std::sync::Condvar::new()));+ rt.block_on(async {+ let mut joins = Vec::with_capacity(spare);+ for _ in 0..spare {+ let started = Arc::clone(&started);+ let release = Arc::clone(&release);+ let pair = Arc::clone(&pair);+ joins.push(tokio::task::spawn_blocking(move || {+ started.fetch_add(1, Ordering::SeqCst);+ let (lock, cv) = &*pair;+ let mut guard = lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());+ while !release.load(Ordering::SeqCst) {+ let (next, _) = cv+ .wait_timeout(guard, std::time::Duration::from_secs(2))+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ guard = next;+ }+ }));+ }+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2);+ while started.load(Ordering::SeqCst) < spare && std::time::Instant::now() < deadline {+ tokio::task::yield_now().await;+ std::thread::sleep(std::time::Duration::from_millis(2));+ }+ let up = started.load(Ordering::SeqCst);+ if up < spare {+ warn!(+ "started {up} of {spare} blocking-pool threads at boot; a later \+ spawn_blocking may still create a thread (#1122)"+ );+ }+ release.store(true, Ordering::SeqCst);+ pair.1.notify_all();+ for join in joins {+ let _ = tokio::time::timeout(std::time::Duration::from_secs(2), join).await;+ }+ });}/// #1840: a neural startup whose Hub cannot be reached fails once, here,@@ -3118,6 +3232,25 @@ mod neural_startup_tests {mod help_text_tests {use super::{auth_disabled_line, embed_mode_fixed_at_ingest_warning, help_text};+ /// #2368: top-level help used to call `autoindex map` "the discovered data+ /// map" and not say that a folder argument previews an unindexed folder.+ #[test]+ fn autoindex_map_help_names_the_folder_preview() {+ let help = help_text(false);+ assert!(+ help.contains("map [folder]"),+ "xerj --help must show that map takes an optional folder:\n{help}"+ );+ assert!(+ help.contains("preview"),+ "xerj --help must say the folder form is a preview:\n{help}"+ );+ assert!(+ !help.contains("print the discovered data map"),+ "the old line is what made `map <folder>` look like the catalog command"+ );+ }+/// #1840: `--embed-mode neural` must say the model loads at startup and/// that a failed download refuses to start, with the lexical command.#[test]@@ -3742,7 +3875,7 @@ mod allocator_purge_tests {#[cfg(test)]mod runtime_tests {- use super::{build_runtime, run_to_exit};+ use super::{additional_blocking_threads, blocking_thread_cap, build_runtime, run_to_exit};/// SIGTERM → "Goodbye" → exit must not wait on a blocking job that is/// still running (in production: a background merge compressing a large@@ -3810,6 +3943,109 @@ mod runtime_tests {// to 32640; assert it ran rather than pinning the exact total.assert!(sum > 0, "the stack-heavy worker task actually executed");}++ #[test]+ fn the_blocking_pool_is_wider_than_the_workers_and_stops_at_1024() {+ assert_eq!(blocking_thread_cap(2), 6);+ assert_eq!(additional_blocking_threads(2), 4);+ assert_eq!(blocking_thread_cap(64), 136);+ assert_eq!(additional_blocking_threads(64), 72);+ assert_eq!(blocking_thread_cap(256), 544);+ assert_eq!(blocking_thread_cap(512), 1024);+ assert_eq!(additional_blocking_threads(512), 512);+ assert!(blocking_thread_cap(1) > 1);+ assert!(additional_blocking_threads(1) >= 1);+ // Above the ceiling the pool is the workers plus one thread.+ // `max_blocking_threads(0)` panics, and `block_in_place` needs a+ // thread that is not already a worker.+ assert_eq!(blocking_thread_cap(10_000), 10_001);+ assert_eq!(additional_blocking_threads(10_000), 1);+ }++ /// #1122: a burst of `spawn_blocking` used to call `std::thread::Builder`+ /// once per task until tokio's default cap of 512. The child panics when+ /// the alternate signal stack cannot be mapped, and the release profile+ /// aborts the process. With every non-worker slot busy, a further task+ /// waits. It must not start on a thread of its own.+ #[test]+ fn a_blocking_burst_does_not_grow_a_thread_per_task() {+ let rt = build_runtime(2).expect("prod runtime builds");+ let slots = additional_blocking_threads(2);+ let early = rt.block_on(async move {+ let running = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));+ let release = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));+ let pair = std::sync::Arc::new((std::sync::Mutex::new(()), std::sync::Condvar::new()));+ // An assertion failure must still wake the holders. They sit in+ // `wait_timeout` and dropping the runtime would otherwise join+ // them with no end.+ let _wake = ReleaseOnDrop {+ release: std::sync::Arc::clone(&release),+ pair: std::sync::Arc::clone(&pair),+ };+ let mut holders = Vec::with_capacity(slots);+ for _ in 0..slots {+ let running = std::sync::Arc::clone(&running);+ let release = std::sync::Arc::clone(&release);+ let pair = std::sync::Arc::clone(&pair);+ holders.push(tokio::task::spawn_blocking(move || {+ running.fetch_add(1, std::sync::atomic::Ordering::SeqCst);+ let (lock, cv) = &*pair;+ let mut guard = lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());+ while !release.load(std::sync::atomic::Ordering::SeqCst) {+ let (next, _) = cv+ .wait_timeout(guard, std::time::Duration::from_secs(2))+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ guard = next;+ }+ }));+ }+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(2);+ while running.load(std::sync::atomic::Ordering::SeqCst) < slots+ && std::time::Instant::now() < deadline+ {+ tokio::task::yield_now().await;+ std::thread::sleep(std::time::Duration::from_millis(2));+ }+ assert_eq!(+ running.load(std::sync::atomic::Ordering::SeqCst),+ slots,+ "the non-worker slots must be occupied before the extra tasks"+ );+ let started_early = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));+ let mut extras = Vec::new();+ for _ in 0..slots {+ let started_early = std::sync::Arc::clone(&started_early);+ extras.push(tokio::task::spawn_blocking(move || {+ started_early.fetch_add(1, std::sync::atomic::Ordering::SeqCst);+ }));+ }+ std::thread::sleep(std::time::Duration::from_millis(300));+ let grew = started_early.load(std::sync::atomic::Ordering::SeqCst);+ release.store(true, std::sync::atomic::Ordering::SeqCst);+ pair.1.notify_all();+ for join in holders.into_iter().chain(extras) {+ join.await.expect("blocking task");+ }+ grew+ });+ assert_eq!(+ early, 0,+ "a task past the blocking cap must wait for a slot, not spawn a thread"+ );+ }++ struct ReleaseOnDrop {+ release: std::sync::Arc<std::sync::atomic::AtomicBool>,+ pair: std::sync::Arc<(std::sync::Mutex<()>, std::sync::Condvar)>,+ }++ impl Drop for ReleaseOnDrop {+ fn drop(&mut self) {+ self.release+ .store(true, std::sync::atomic::Ordering::SeqCst);+ self.pair.1.notify_all();+ }+ }}async fn async_main() -> Result<()> {diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex 0a59e8bbc..416bd3c42 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -774,9 +774,13 @@ fn segment_bundle_key(segment_id: &str) -> String {/// this./// - OFF the runtime entirely: the engine runs flush finalizers on plain std/// threads (`xerj-bg-*`), where `Handle::current()` has no answer. There-/// the future is driven on a process-global dedicated runtime, created on-/// first use and shared for the lifetime of the store — one idle worker-/// thread in Local mode (never created), one per process in object mode.+/// the future is driven on a process-global current-thread runtime, created+/// on first use and shared for the lifetime of the process. It spawns no+/// worker thread: a multi-thread runtime's worker is `std::thread::Builder`,+/// which panics in the child when the alternate signal stack cannot be+/// mapped and aborts the node (#1122). Callers take turns on a mutex.+/// Local mode never creates it. A build failure is an error from this+/// call, and the next call tries again.////// The futures driven here are plain backend I/O and never re-enter/// `block_on_backend`, so the dedicated runtime cannot nest `block_on`.@@ -785,7 +789,13 @@ fn block_on_backend<T>() -> crate::Result<T> {match tokio::runtime::Handle::try_current() {Ok(handle) => tokio::task::block_in_place(|| handle.block_on(fut)),- Err(_) => object_io_runtime().block_on(fut),+ Err(_) => {+ let runtime = object_io_runtime()?;+ let guard = runtime+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ guard.block_on(fut)+ }}}@@ -803,16 +813,31 @@ fn segment_file_missing(path: &Path) -> bool {}}-fn object_io_runtime() -> &'static tokio::runtime::Runtime {- static RUNTIME: std::sync::OnceLock<tokio::runtime::Runtime> = std::sync::OnceLock::new();- RUNTIME.get_or_init(|| {- tokio::runtime::Builder::new_multi_thread()- .worker_threads(1)- .thread_name("xerj-obj-io")- .enable_all()- .build()- .expect("object-store I/O runtime")- })+fn object_io_runtime() -> crate::Result<&'static std::sync::Mutex<tokio::runtime::Runtime>> {+ static RUNTIME: std::sync::OnceLock<std::sync::Mutex<tokio::runtime::Runtime>> =+ std::sync::OnceLock::new();+ if let Some(runtime) = RUNTIME.get() {+ return Ok(runtime);+ }+ // Current-thread: `block_on` drives the future on the caller. A+ // multi-thread build would spawn a worker with `std::thread::Builder`+ // and abort the process if that child's signal stack could not be+ // mapped (#1122). `.expect` did the same for a `build()` error.+ // Blocking threads, if a backend future asks for one, stop at 4 and+ // are not reaped after 10 seconds (that reap is a later spawn).+ let runtime = tokio::runtime::Builder::new_current_thread()+ .enable_all()+ .max_blocking_threads(4)+ .thread_keep_alive(std::time::Duration::from_secs(24 * 60 * 60))+ .thread_name("xerj-obj-io")+ .build()+ .map_err(|error| {+ crate::StorageError::Backend(format!(+ "could not start the object-store I/O runtime ({error}); \+ the index stays open and this object operation fails (#1122)"+ ))+ })?;+ Ok(RUNTIME.get_or_init(|| std::sync::Mutex::new(runtime)))}const WAL_MAINTENANCE_INTERVAL_MS: u64 = 1_000;@@ -8405,3 +8430,23 @@ mod tests {assert_eq!(resident, 2, "the live write and the tombstone stay");}}++#[cfg(test)]+mod object_io_runtime_tests {+ /// #1122: the off-runtime object driver used to `.expect` a multi-thread+ /// runtime into existence. `build()` failing, or the worker's signal-stack+ /// mmap failing, aborted the node. The driver now runs on the caller.+ #[test]+ fn an_off_runtime_object_call_finishes_and_a_second_caller_does_too() {+ let first = super::block_on_backend(async { Ok::<u32, crate::StorageError>(7) }).unwrap();+ assert_eq!(first, 7);+ let left = std::thread::spawn(|| {+ super::block_on_backend(async { Ok::<u32, crate::StorageError>(1) })+ });+ let right = std::thread::spawn(|| {+ super::block_on_backend(async { Ok::<u32, crate::StorageError>(2) })+ });+ assert_eq!(left.join().unwrap().unwrap(), 1);+ assert_eq!(right.join().unwrap().unwrap(), 2);+ }+}diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex ed274f047..ec1bea006 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -441,10 +441,19 @@ time_partition = "1h"[embedding]-# OpenAI-compatible endpoint URL for auto-embedding on ingest.-# Leave empty to disable automatic embedding (vectors must be provided by client).-# Example: "https://api.openai.com/v1/embeddings"-# Example: "http://localhost:11434/v1/embeddings" (local Ollama)+# OpenAI-compatible embeddings URL. Leave empty to disable automatic+# embedding (vectors must be provided by the client). Accepted forms,+# trailing slash optional (#2355):+# - a base ending in /v1 — /embeddings is appended+# "http://localhost:8000/v1" → http://localhost:8000/v1/embeddings+# - a bare host — /embeddings is appended+# "http://localhost:8000" → http://localhost:8000/embeddings+# - the full embeddings path — used unchanged+# "https://api.openai.com/v1/embeddings"+# "http://localhost:11434/v1/embeddings" (local Ollama)+# Any other path refuses to start when mode = "proxy" and names this URL+# plus the {origin}/v1/embeddings path to set. The string stored here is+# also the proxy embedder identity, so keep it stable once a corpus is indexed.default_endpoint = ""# Model name to request from the embedding endpoint.diff --git a/landing/answers/autoindex-exit-codes.html b/landing/answers/autoindex-exit-codes.htmlindex 68e7a6690..7967387c3 100644--- a/landing/answers/autoindex-exit-codes.html+++ b/landing/answers/autoindex-exit-codes.html@@ -79,7 +79,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-22",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-24","author": {"@type": "Organization","name": "XERJ documentation team",@@ -116,7 +116,7 @@"name": "What does indexer exit code 3 mean?","acceptedAnswer": {"@type": "Answer",- "text": "It means the run completed and recorded junk. xerj autoindex --help states it as 3 completed-with-junk (junk recorded, never fatal). Treat it as a success with a refusal list to read."+ "text": "It means the run completed and recorded junk. xerj autoindex --help states it as 3 completed-with-junk (junk recorded, never fatal) — SUCCESS and says to treat 0 and 3 alike as a finished run. Treat it as a success with a refusal list to read. Do not rerun it as a failure."}},{@@ -236,9 +236,9 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>THE INDEXER EXITED 3 (OR 4). DID IT FAIL?</div><div class="kicker"><span>ANSWERS · OPERATIONS: EXIT CODES</span></div><h1>The indexer exited 3 (or 4). Did it fail?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-22 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-22 · UPDATED · 2026-10-24</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-24</span><p>What does indexer exit code 3 mean for xerj autoindex? It means success: the run completed and recorded some junk files. Exit 0 is a clean completion, 4 means the estimate exceeded --max-minutes and nothing was indexed, 2 is a usage error, and 1 is an endpoint or journal failure. Only exit 4 asks you a question.</p><p class="key-fact"><strong>Key number:</strong> The decision gate that produces exit 4 defaults to 10 minutes of estimated work (<code class="inline">DEFAULT_MAX_MINUTES</code>). <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/gate.rs">engine/crates/xerj-autoindex/src/gate.rs</a>)</span></p></div>@@ -284,7 +284,7 @@<thead><tr><th>Exit</th><th>Name in <code class="inline">--help</code></th><th>What it means for the corpus</th></tr></thead><tbody><tr><td>0</td><td>complete</td><td>The run finished and refused nothing. A gate answered with <code class="inline">--approve cancel</code> also exits 0.</td></tr>- <tr><td>3</td><td>completed-with-junk</td><td>The run finished. At least one file was refused and recorded. Never fatal. This includes a whole dataset whose mapping the server refused.</td></tr>+ <tr><td>3</td><td>completed-with-junk</td><td>SUCCESS. The run finished. At least one file was refused and recorded. Treat it the same as exit 0: the index is searchable. This includes a whole dataset whose mapping the server refused.</td></tr><tr><td>4</td><td>NEEDS A DECISION</td><td>The estimate exceeded <code class="inline">--max-minutes</code>. Nothing was indexed. A JSON decision request is on stdout.</td></tr><tr><td>2</td><td>usage</td><td>The command line was wrong. Nothing ran.</td></tr><tr><td>1</td><td>endpoint/journal failure</td><td>The node or the resume journal failed, a corpus removal was refused, or an unsafe state transition was refused.</td></tr>@@ -292,7 +292,7 @@</table><p>Read the table in that order rather than in numeric order. It is the order the help text uses, and it is the order of decreasing goodness.</p><h2>Exit 3 is a finished run</h2>- <p><code class="inline">3 completed-with-junk (junk recorded, never fatal)</code> is the phrase in the help text, and <code class="inline">never fatal</code> is doing the work in that sentence.</p>+ <p><code class="inline">3 completed-with-junk (junk recorded, never fatal) — SUCCESS</code> is the phrase in the help text, and it says to treat 0 and 3 alike as a finished run. Exit 3 is success. Do not rerun it as a failure.</p><p>A folder of real files almost always contains something XERJ will not parse. An unknown binary, an image, a file above <code class="inline">--max-file-gb</code>. Those files are recorded in the catalog with a reason, and the rest of the folder is indexed and queryable.</p><p>An agent that treats a non-zero exit as a failure will therefore throw away a perfectly good index. Branch on the specific value.</p><p>The companion check is the refusal list, not the exit code: read <code class="inline">autoindex-catalog</code> and match each refused path to its reason. The <a href="/answers/why-autoindex-skipped-files">skipped-files page</a> covers that query.</p>@@ -303,7 +303,8 @@</code></pre><p>So an exit 3 is still a finished run, but it is not always a small gap. Read <code class="inline">datasets_refused</code> before you report a corpus as searchable. The <a href="/answers/autoindex-dataset-refused-by-server">refused-dataset page</a> covers what is recorded and how to recover.</p><p>Only a 400 is a refusal. A 401, 403, 404, 408, 429 or 5xx on the same request says nothing about that dataset, so it still exits 1.</p>- <p>A 429 <em>on a bulk</em> is different, whether it is the whole request or some of its items marked <code class="inline">status: 429</code>. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on; it exits 1 only after 600 seconds of re-sends the node never takes, with an error line that begins <code class="inline">the server kept rejecting</code>. The terminal line of a run that met back-pressure and finished carries <code class="inline">bulk_retries=N</code>. The <a href="/answers/autoindex-server-back-pressure-429">back-pressure page</a> covers it.</p>+ <p>A 429 <em>on a bulk</em> is different, whether it is the whole request or some of its items marked <code class="inline">status: 429</code>. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on. It exits 1 only after 600 seconds of re-sends the node never takes.</p>+ <p>The error line begins <code class="inline">the server kept rejecting</code>. The terminal line of a run that met back-pressure and finished carries <code class="inline">bulk_retries=N</code>. The <a href="/answers/autoindex-server-back-pressure-429">back-pressure page</a> covers it.</p><h2>Exit 4 is a question, and nothing was written</h2><p>Phase A reads and parses every file to sniff and sample it, so it measures throughput per format family on the machine you are on. It turns that into a range for the indexing phase, and the gate compares the upper end of the range against <code class="inline">--max-minutes</code> (default 10).</p><p>If the upper end is longer and no <code class="inline">--approve</code> or <code class="inline">--yes</code> was given, <strong>nothing is indexed</strong>. A JSON decision request goes to stdout and the process exits 4.</p>@@ -325,10 +326,10 @@ xerj-done ok=false exit=1 reason=server-backpressure wall=… ops_applied=N ops_<pre class="code"><code class="language-text">xerj-done ok=false exit=1 reason=aborted wall=10336.0serror: prepared bulk contained 1 rejected items: {"type":"engine_exception","reason":"bulk request contains 102258 lines (~51129 actions); exceeds max_actions_per_bulk of 50000","status":413}</code></pre>- <p>Every <code class="inline">_bulk</code> body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries <code class="inline">bulk_splits=N</code> when that happened. Against real nodes with <code class="inline">max_actions_per_bulk = 64</code> and with <code class="inline">max_body_bytes = 98304</code>, each run ended <code class="inline">ok=true exit=3 records=1663 bulk_splits=1</code>, the same records as a control run. Resuming that same generation with the change committed it:</p>+ <p>Every <code class="inline">_bulk</code> body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries <code class="inline">bulk_splits=N</code> when that happened. Against real nodes with <code class="inline">max_actions_per_bulk = 64</code> and with <code class="inline">max_body_bytes = 98304</code>, each run ended <code class="inline">ok=true exit=3 records=1663 bulk_splits=1</code>, the same documents as a control run. Resuming that same generation with the change committed it:</p><pre class="code"><code class="language-text">xerj-done ok=true exit=3 reason=completed-with-junk wall=415.0s files=47444 records=821840 generation=1 code_files=34324 code_files_indexed=34324 code_files_junked=0</code></pre>- <p>One record that is larger than the node's <code class="inline">limits.max_body_bytes</code> cannot be cut, and it still ends the run with exit 1 and an error that names that setting.</p>+ <p>One document that is larger than the node's <code class="inline">limits.max_body_bytes</code> cannot be cut, and it still ends the run with exit 1 and an error that names that setting.</p><h2>Do not read the code alone</h2><p>Every run that reaches an exit, success or error, ends with one terminal line in every progress mode except <code class="inline">none</code>.</p><pre class="code"><code class="language-text">xerj-done ok=true exit=3 reason=completed-with-junk wall=0.6s files=4 records=1 generation=1@@ -349,10 +350,12 @@ error: prepared bulk contained 1 rejected items: {"type":"engine_exception","rea<p>Pair this with the four-number reconciliation on the <a href="/answers/check-codebase-index-is-complete">completeness page</a>. The exit code tells you how the process ended; only the counts tell you what is in the index.</p><h2>How this page was checked</h2><p>The codes above were read from <code class="inline">xerj autoindex --help</code> on a built binary and cross-checked against the help string in <code class="inline">engine/crates/xerj-autoindex/src/cli.rs</code>. Both agree.</p>- <p>One code was forced for this page: the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under <code class="inline">benchmarks/autoindex-resilience/</code>, together with the full-corpus exit 1 in <code class="inline">finalize-catalog</code>, the exit 3 of the same generation resumed after issue #955 was fixed (<code class="inline">before-955.stderr.txt</code>, <code class="inline">after-955.full-corpus-resume.stderr.txt</code>), and a <code class="inline">reason=server-backpressure</code> exit 1 forced with a 64 MiB memory cap (<code class="inline">limits-real-node.txt</code>). The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, <code class="inline">xerj autoindex --help</code> prints them in one screen.</p>+ <p>One code was forced for this page. It is the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under <code class="inline">benchmarks/autoindex-resilience/</code>.</p>+ <p>The folder also holds the full-corpus exit 1 in <code class="inline">finalize-catalog</code>. It holds the exit 3 of the same generation resumed after issue #955 was fixed (<code class="inline">before-955.stderr.txt</code>, <code class="inline">after-955.full-corpus-resume.stderr.txt</code>). It holds a <code class="inline">reason=server-backpressure</code> exit 1 forced with a 64 MiB memory cap (<code class="inline">limits-real-node.txt</code>).</p>+ <p>The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, <code class="inline">xerj autoindex --help</code> prints them in one screen.</p><h2>FAQ</h2><h3>What does indexer exit code 3 mean?</h3>- <p>It means the run completed and recorded junk. <code class="inline">xerj autoindex --help</code> states it as <code class="inline">3 completed-with-junk (junk recorded, never fatal)</code>. Treat it as a success with a refusal list to read.</p>+ <p>It means the run completed and recorded junk. <code class="inline">xerj autoindex --help</code> states it as <code class="inline">3 completed-with-junk (junk recorded, never fatal) — SUCCESS</code> and says to treat 0 and 3 alike as a finished run. Treat it as a success with a refusal list to read. Do not rerun it as a failure.</p><h3>The indexer exited 3 (or 4). Did it fail?</h3><p>3 did not fail. 4 did not fail either, and it did not index anything: it is a decision request waiting for <code class="inline">--approve</code>. Only 1 and 2 are failures.</p><h3>Why did folder indexing stop and ask me to approve?</h3>@@ -369,8 +372,8 @@ error: prepared bulk contained 1 rejected items: {"type":"engine_exception","rea<p>Retry nothing on 0 or 3. Re-run with <code class="inline">--approve</code> on 4. Fix the command line on 2. Investigate the endpoint and the journal on 1, because a blind retry repeats the same refusal. The one exit 1 to rerun unchanged is <code class="inline">reason=server-backpressure</code>, once the node accepts writes again: it resumes with the operations that remain.</p><h2>Evidence</h2><ul>- <li>EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) - this includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION - the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition. — <code>engine/crates/xerj-autoindex/src/cli.rs:395</code></li>- <li>Exit 4 is a code of its own; 1 is the catch-all for real failures, and the decision request is answered by re-running the same command with --approve proceed, fast or cancel. — <code>engine/crates/xerj-autoindex/src/cli.rs:395</code></li>+ <li>EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) — SUCCESS: the index is searchable, and the summary on stdout opens with a line starting <code class="inline">ok:</code> that says so (xerj-done carries ok=true). Treat 0 and 3 alike as a finished run. This includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION — the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition. — <code>engine/crates/xerj-autoindex/src/cli.rs:771</code></li>+ <li>Exit 4 is a code of its own; 1 is the catch-all for real failures, and the decision request is answered by re-running the same command with --approve proceed, fast or cancel. — <code>engine/crates/xerj-autoindex/src/cli.rs:771</code></li><li>A forced mapping refusal ended xerj-done ok=true exit=3 reason=completed-with-junk wall=0.7s files=1 records=1 generation=1 datasets_refused=1 files_refused=2, with the other dataset indexed. — <code>benchmarks/autoindex-resilience/refusal-e2e.run1.stderr.txt</code></li><li>Before issue #955 was fixed, a 48,533-file --no-graph run applied every operation and then ended xerj-done ok=false exit=1 reason=aborted wall=10336.0s, because the catalog went out as one bulk request of 51,129 actions and the engine's max_actions_per_bulk is 50,000. — <code>benchmarks/autoindex-resilience/before-955.stderr.txt</code></li><li>Resuming that same generation with the fix ended xerj-done ok=true exit=3 reason=completed-with-junk wall=415.0s files=47444 records=821840 generation=1. — <code>benchmarks/autoindex-resilience/after-955.full-corpus-resume.stderr.txt</code></li>diff --git a/landing/answers/autoindex-exit-codes.md b/landing/answers/autoindex-exit-codes.mdindex 6eeaeb0de..53d54f8f9 100644--- a/landing/answers/autoindex-exit-codes.md+++ b/landing/answers/autoindex-exit-codes.md@@ -2,13 +2,13 @@---title: "What do the xerj autoindex exit codes mean?"canonical: "https://xerj.org/answers/autoindex-exit-codes"-updated: "2026-10-19"+updated: "2026-10-24"source: "content/answers/autoindex-exit-codes.md"---# The indexer exited 3 (or 4). Did it fail?-**Short answer** (updated 2026-10-19): What does indexer exit code 3 mean for xerj autoindex? It means success: the run completed and recorded some junk files. Exit 0 is a clean completion, 4 means the estimate exceeded --max-minutes and nothing was indexed, 2 is a usage error, and 1 is an endpoint or journal failure. Only exit 4 asks you a question.+**Short answer** (updated 2026-10-24): What does indexer exit code 3 mean for xerj autoindex? It means success: the run completed and recorded some junk files. Exit 0 is a clean completion, 4 means the estimate exceeded --max-minutes and nothing was indexed, 2 is a usage error, and 1 is an endpoint or journal failure. Only exit 4 asks you a question.**Key number:** The decision gate that produces exit 4 defaults to 10 minutes of estimated work (`DEFAULT_MAX_MINUTES`). (as of 2026-10-19; source: [engine/crates/xerj-autoindex/src/gate.rs](https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/gate.rs))@@ -69,7 +69,7 @@ The list below is quoted from `xerj autoindex --help` on the installed binary. R| Exit | Name in `--help` | What it means for the corpus || --- | --- | --- || 0 | complete | The run finished and refused nothing. A gate answered with `--approve cancel` also exits 0. |-| 3 | completed-with-junk | The run finished. At least one file was refused and recorded. Never fatal. This includes a whole dataset whose mapping the server refused. |+| 3 | completed-with-junk | SUCCESS. The run finished. At least one file was refused and recorded. Treat it the same as exit 0: the index is searchable. This includes a whole dataset whose mapping the server refused. || 4 | NEEDS A DECISION | The estimate exceeded `--max-minutes`. Nothing was indexed. A JSON decision request is on stdout. || 2 | usage | The command line was wrong. Nothing ran. || 1 | endpoint/journal failure | The node or the resume journal failed, a corpus removal was refused, or an unsafe state transition was refused. |@@ -78,7 +78,7 @@ Read the table in that order rather than in numeric order. It is the order the h## Exit 3 is a finished run-`3 completed-with-junk (junk recorded, never fatal)` is the phrase in the help text, and `never fatal` is doing the work in that sentence.+`3 completed-with-junk (junk recorded, never fatal) — SUCCESS` is the phrase in the help text, and it says to treat 0 and 3 alike as a finished run. Exit 3 is success. Do not rerun it as a failure.A folder of real files almost always contains something XERJ will not parse. An unknown binary, an image, a file above `--max-file-gb`. Those files are recorded in the catalog with a reason, and the rest of the folder is indexed and queryable.@@ -100,7 +100,9 @@ So an exit 3 is still a finished run, but it is not always a small gap. Read `daOnly a 400 is a refusal. A 401, 403, 404, 408, 429 or 5xx on the same request says nothing about that dataset, so it still exits 1.-A 429 *on a bulk* is different, whether it is the whole request or some of its items marked `status: 429`. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on; it exits 1 only after 600 seconds of re-sends the node never takes, with an error line that begins `the server kept rejecting`. The terminal line of a run that met back-pressure and finished carries `bulk_retries=N`. The [back-pressure page](/answers/autoindex-server-back-pressure-429) covers it.+A 429 *on a bulk* is different, whether it is the whole request or some of its items marked `status: 429`. When the node's memory circuit breaker answers that way, the run re-sends the rejected items after a backoff and carries on. It exits 1 only after 600 seconds of re-sends the node never takes.++The error line begins `the server kept rejecting`. The terminal line of a run that met back-pressure and finished carries `bulk_retries=N`. The [back-pressure page](/answers/autoindex-server-back-pressure-429) covers it.## Exit 4 is a question, and nothing was written@@ -142,13 +144,13 @@ xerj-done ok=false exit=1 reason=aborted wall=10336.0serror: prepared bulk contained 1 rejected items: {"type":"engine_exception","reason":"bulk request contains 102258 lines (~51129 actions); exceeds max_actions_per_bulk of 50000","status":413}```-Every `_bulk` body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries `bulk_splits=N` when that happened. Against real nodes with `max_actions_per_bulk = 64` and with `max_body_bytes = 98304`, each run ended `ok=true exit=3 records=1663 bulk_splits=1`, the same records as a control run. Resuming that same generation with the change committed it:+Every `_bulk` body now goes out in windows of at most 10,000 actions. A request the node still refuses for its size is cut in two and sent again, and the terminal line carries `bulk_splits=N` when that happened. Against real nodes with `max_actions_per_bulk = 64` and with `max_body_bytes = 98304`, each run ended `ok=true exit=3 records=1663 bulk_splits=1`, the same documents as a control run. Resuming that same generation with the change committed it:```textxerj-done ok=true exit=3 reason=completed-with-junk wall=415.0s files=47444 records=821840 generation=1 code_files=34324 code_files_indexed=34324 code_files_junked=0```-One record that is larger than the node's `limits.max_body_bytes` cannot be cut, and it still ends the run with exit 1 and an error that names that setting.+One document that is larger than the node's `limits.max_body_bytes` cannot be cut, and it still ends the run with exit 1 and an error that names that setting.## Do not read the code alone@@ -178,13 +180,17 @@ Pair this with the four-number reconciliation on the [completeness page](/answerThe codes above were read from `xerj autoindex --help` on a built binary and cross-checked against the help string in `engine/crates/xerj-autoindex/src/cli.rs`. Both agree.-One code was forced for this page: the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under `benchmarks/autoindex-resilience/`, together with the full-corpus exit 1 in `finalize-catalog`, the exit 3 of the same generation resumed after issue #955 was fixed (`before-955.stderr.txt`, `after-955.full-corpus-resume.stderr.txt`), and a `reason=server-backpressure` exit 1 forced with a 64 MiB memory cap (`limits-real-node.txt`). The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, `xerj autoindex --help` prints them in one screen.+One code was forced for this page. It is the exit 3 of a refused dataset, on a throwaway node, by pre-creating the dataset's index with a conflicting field type. That capture is committed under `benchmarks/autoindex-resilience/`.++The folder also holds the full-corpus exit 1 in `finalize-catalog`. It holds the exit 3 of the same generation resumed after issue #955 was fixed (`before-955.stderr.txt`, `after-955.full-corpus-resume.stderr.txt`). It holds a `reason=server-backpressure` exit 1 forced with a 64 MiB memory cap (`limits-real-node.txt`).++The other codes were not forced, so there is no capture for them here and no timing. If you want the codes on your own build, `xerj autoindex --help` prints them in one screen.## FAQ### What does indexer exit code 3 mean?-It means the run completed and recorded junk. `xerj autoindex --help` states it as `3 completed-with-junk (junk recorded, never fatal)`. Treat it as a success with a refusal list to read.+It means the run completed and recorded junk. `xerj autoindex --help` states it as `3 completed-with-junk (junk recorded, never fatal) — SUCCESS` and says to treat 0 and 3 alike as a finished run. Treat it as a success with a refusal list to read. Do not rerun it as a failure.### The indexer exited 3 (or 4). Did it fail?@@ -216,8 +222,8 @@ Retry nothing on 0 or 3. Re-run with `--approve` on 4. Fix the command line on 2## Evidence-- EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) - this includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION - the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition. — `engine/crates/xerj-autoindex/src/cli.rs:395`-- Exit 4 is a code of its own; 1 is the catch-all for real failures, and the decision request is answered by re-running the same command with --approve proceed, fast or cancel. — `engine/crates/xerj-autoindex/src/cli.rs:395`+- EXIT CODES: 0 complete (also: gate answered with --approve cancel); 3 completed-with-junk (junk recorded, never fatal) — SUCCESS: the index is searchable, and the summary on stdout opens with a line starting `ok:` that says so (xerj-done carries ok=true). Treat 0 and 3 alike as a finished run. This includes a dataset whose mapping the server REFUSED: its files are recorded as junk with the server's reason, the xerj-done line carries datasets_refused / files_refused, and every other dataset is indexed; 4 NEEDS A DECISION — the estimate exceeded --max-minutes and nothing was indexed, a JSON decision request is on stdout; 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a refused unsafe state transition. — `engine/crates/xerj-autoindex/src/cli.rs:771`+- Exit 4 is a code of its own; 1 is the catch-all for real failures, and the decision request is answered by re-running the same command with --approve proceed, fast or cancel. — `engine/crates/xerj-autoindex/src/cli.rs:771`- A forced mapping refusal ended xerj-done ok=true exit=3 reason=completed-with-junk wall=0.7s files=1 records=1 generation=1 datasets_refused=1 files_refused=2, with the other dataset indexed. — `benchmarks/autoindex-resilience/refusal-e2e.run1.stderr.txt`- Before issue #955 was fixed, a 48,533-file --no-graph run applied every operation and then ended xerj-done ok=false exit=1 reason=aborted wall=10336.0s, because the catalog went out as one bulk request of 51,129 actions and the engine's max_actions_per_bulk is 50,000. — `benchmarks/autoindex-resilience/before-955.stderr.txt`- Resuming that same generation with the fix ended xerj-done ok=true exit=3 reason=completed-with-junk wall=415.0s files=47444 records=821840 generation=1. — `benchmarks/autoindex-resilience/after-955.full-corpus-resume.stderr.txt`diff --git a/landing/answers/check-codebase-index-is-complete.html b/landing/answers/check-codebase-index-is-complete.htmlindex 6d3268a6d..9126e706a 100644--- a/landing/answers/check-codebase-index-is-complete.html+++ b/landing/answers/check-codebase-index-is-complete.html@@ -79,7 +79,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-24","author": {"@type": "Organization","name": "XERJ documentation team",@@ -140,7 +140,7 @@"name": "The indexer exited. Did it finish or just stop?","acceptedAnswer": {"@type": "Answer",- "text": "Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused, and exit 3 with reason=completed-with-junk means the run finished and refused at least one file."+ "text": "Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused. Exit 3 with reason=completed-with-junk is success: the run finished and refused at least one file. Do not treat exit 3 as a failure."}},{@@ -228,9 +228,9 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>MY CODEBASE INDEXER SAYS INDEXED BUT I DON'T SEE MY CODE. HOW DO I CHECK IT ACTUALLY FINISHED?</div><div class="kicker"><span>ANSWERS · OPERATIONS: COMPLETENESS</span></div><h1 class="h1-long">My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-24</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-24</span><p>My codebase indexer says indexed but I don't see my code: how do I check it actually finished? Reconcile four numbers: files on disk, the --dry-run plan, xerj autoindex status, and the document count in _cat/indices. A captured run turned 64 non-hidden files into a 3-file plan, naming an ignore rule for every gap.</p><p class="key-fact"><strong>Key number:</strong> Source files above 2 MiB are not parsed for symbols (<code class="inline">CODE_CAP</code>), one of the documented reasons a file can be missing from the code fields. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs">engine/crates/xerj-autoindex/src/extract/code.rs</a>)</span></p></div>@@ -312,7 +312,7 @@ autoindex: ignore rules: .gitignore:coverage/ — 1 directory pruned (5 non-hi<h3>How do I verify a local code index actually contains my files?</h3><p>Query for a symbol you know exists. The captured query on the defs field for merge_segments returned 2 hits, src/lib.rs in Rust and src/ingest.py in Python, each with its path.</p><h3>The indexer exited. Did it finish or just stop?</h3>- <p>Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused, and exit 3 with reason=completed-with-junk means the run finished and refused at least one file.</p>+ <p>Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused. Exit 3 with reason=completed-with-junk is success: the run finished and refused at least one file. Do not treat exit 3 as a failure.</p><h3>Why does the index hold more documents than files?</h3><p>XERJ writes one document per extracted row, and for source files it adds one document per declaration on top of the whole-file document. The document count therefore follows how many declarations your code holds, not a fixed multiple of the file count, so read it from your own run.</p><h3>Why does the plan hold fewer files than the folder?</h3>diff --git a/landing/answers/check-codebase-index-is-complete.md b/landing/answers/check-codebase-index-is-complete.mdindex 173050f0c..950dd5f85 100644--- a/landing/answers/check-codebase-index-is-complete.md+++ b/landing/answers/check-codebase-index-is-complete.md@@ -2,13 +2,13 @@---title: "How do I check my code index actually finished?"canonical: "https://xerj.org/answers/check-codebase-index-is-complete"-updated: "2026-10-19"+updated: "2026-10-24"source: "content/answers/check-codebase-index-is-complete.md"---# My codebase indexer says indexed but I don't see my code. How do I check it actually finished?-**Short answer** (updated 2026-10-19): My codebase indexer says indexed but I don't see my code: how do I check it actually finished? Reconcile four numbers: files on disk, the --dry-run plan, xerj autoindex status, and the document count in _cat/indices. A captured run turned 64 non-hidden files into a 3-file plan, naming an ignore rule for every gap.+**Short answer** (updated 2026-10-24): My codebase indexer says indexed but I don't see my code: how do I check it actually finished? Reconcile four numbers: files on disk, the --dry-run plan, xerj autoindex status, and the document count in _cat/indices. A captured run turned 64 non-hidden files into a 3-file plan, naming an ignore rule for every gap.**Key number:** Source files above 2 MiB are not parsed for symbols (`CODE_CAP`), one of the documented reasons a file can be missing from the code fields. (as of 2026-10-19; source: [engine/crates/xerj-autoindex/src/extract/code.rs](https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-autoindex/src/extract/code.rs))@@ -139,7 +139,7 @@ Query for a symbol you know exists. The captured query on the defs field for mer### The indexer exited. Did it finish or just stop?-Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused, and exit 3 with reason=completed-with-junk means the run finished and refused at least one file.+Read the terminal line and then the journal. Exit 0 with reason=completed means nothing was refused. Exit 3 with reason=completed-with-junk is success: the run finished and refused at least one file. Do not treat exit 3 as a failure.### Why does the index hold more documents than files?diff --git a/landing/answers/index.json b/landing/answers/index.jsonindex 04ffe25ca..c1f61d58d 100644--- a/landing/answers/index.json+++ b/landing/answers/index.json@@ -1016,7 +1016,7 @@"canonical_url": "https://xerj.org/answers/check-codebase-index-is-complete","markdown_url": "https://xerj.org/answers/check-codebase-index-is-complete.md","cluster": "Operations: completeness",- "updated": "2026-10-19",+ "updated": "2026-10-24","summary": "Compare 4 numbers: files on disk, the dry-run plan, autoindex status, and the index count. A captured run cut 64 files to a 3-file plan, naming a rule per gap.","intent_group": "code search for agents","answer": "My codebase indexer says indexed but I don't see my code: how do I check it actually finished? Reconcile four numbers: files on disk, the --dry-run plan, xerj autoindex status, and the document count in _cat/indices. A captured run turned 64 non-hidden files into a 3-file plan, naming an ignore rule for every gap.",@@ -1041,7 +1041,7 @@"canonical_url": "https://xerj.org/answers/why-autoindex-skipped-files","markdown_url": "https://xerj.org/answers/why-autoindex-skipped-files.md","cluster": "Operations: exclusions",- "updated": "2026-10-19",+ "updated": "2026-10-24","summary": "XERJ prints one line per ignore rule and stores a refusal reason per file. A capture reconciles 26 files on disk to 1 indexed, 2 refused and 23 excluded.","intent_group": "operating autoindex","answer": "Why would a folder search miss files that I can see on disk? XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, excluded 23 through ignore rules, and exited 3."@@ -1052,7 +1052,7 @@"canonical_url": "https://xerj.org/answers/autoindex-exit-codes","markdown_url": "https://xerj.org/answers/autoindex-exit-codes.md","cluster": "Operations: exit codes",- "updated": "2026-10-19",+ "updated": "2026-10-24","summary": "xerj autoindex exits 0 complete, 3 completed-with-junk, 4 needs-a-decision, 2 usage and 1 on an endpoint or journal failure. Only 4 is asking you a question.","intent_group": "operating autoindex","answer": "What does indexer exit code 3 mean for xerj autoindex? It means success: the run completed and recorded some junk files. Exit 0 is a clean completion, 4 means the estimate exceeded --max-minutes and nothing was indexed, 2 is a usage error, and 1 is an endpoint or journal failure. Only exit 4 asks you a question."@@ -1141,7 +1141,7 @@"canonical_url": "https://xerj.org/answers/read-autoindex-progress","markdown_url": "https://xerj.org/answers/read-autoindex-progress.md","cluster": "Operations: progress",- "updated": "2026-10-19",+ "updated": "2026-10-24","summary": "XERJ prints xerj-bar, xerj-progress and a final xerj-done line. On a single large file the percentage stays at 0.0%, so read since_progress_s and waiting_on.","intent_group": "operating autoindex","answer": "How can I see whether folder indexing is still working? Pass --progress plain and read the xerj-progress lines on stderr. A captured 22.2 s run emitted 12 phases in order and ended with xerj-done ok=true. On a single 16 MB file the percentage stays at 0.0%, so read since_progress_s and waiting_on instead."diff --git a/landing/answers/read-autoindex-progress.html b/landing/answers/read-autoindex-progress.htmlindex 9cf4ad721..40d2fd77a 100644--- a/landing/answers/read-autoindex-progress.html+++ b/landing/answers/read-autoindex-progress.html@@ -79,7 +79,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-24","author": {"@type": "Organization","name": "XERJ documentation team",@@ -236,9 +236,9 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>HOW DO I READ AUTOINDEX PROGRESS?</div><div class="kicker"><span>ANSWERS · OPERATIONS: PROGRESS</span></div><h1>How do I read autoindex progress?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-24</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-24</span><p>How can I see whether folder indexing is still working? Pass --progress plain and read the xerj-progress lines on stderr. A captured 22.2 s run emitted 12 phases in order and ended with xerj-done ok=true. On a single 16 MB file the percentage stays at 0.0%, so read since_progress_s and waiting_on instead.</p><p class="key-fact"><strong>Key number:</strong> With --no-graph, a 231-file repository reported 9 phases in order, and its index phase read pct=81.7 at 169/231 items. <span class="mute">(as of 2026-10-19 · source: <a href="https://github.com/xerj-org/xerj/blob/main/benchmarks/autoindex-resilience/after-fix.small-repo.stderr.txt">benchmarks/autoindex-resilience/after-fix.small-repo.stderr.txt</a>)</span></p></div>@@ -320,17 +320,18 @@<p>A resumed run skips <code class="inline">walk</code>, <code class="inline">hash</code>, <code class="inline">scan</code> and <code class="inline">snapshot</code>. It reports <code class="inline">replay</code>, then <code class="inline">index</code>. Its <code class="inline">index</code> phase counts only the operations still to apply, so it starts at 0% of what remains. It does not credit this run with an earlier run's writes. In a capture of a full-corpus run interrupted with 12,890 of 47,444 operations committed, the resumed <code class="inline">index</code> phase opened at <code class="inline">items=0/34554</code>.</p><p>Every run shows <code class="inline">phase=starting</code> until its first real phase opens. On that resumed run, with a large journal to load, <code class="inline">starting</code> lasted about 13 seconds.</p><p>The <code class="inline">index</code> phase applies one file at a time, so it is the slow one on a large corpus. Read <code class="inline">eta_s</code> once <code class="inline">eta_quality</code> leaves <code class="inline">unknown</code>.</p>- <h2>scan at 100% now means scan</h2>- <p>Through v1.0.0-rc.74 the <code class="inline">--no-graph</code> route reported only <code class="inline">walk</code>, <code class="inline">hash</code> and <code class="inline">scan</code>. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the scan that had already finished:</p>+ <h2><code class="inline">scan</code> at 100% now means <code class="inline">scan</code></h2>+ <p>Through v1.0.0-rc.74 the <code class="inline">--no-graph</code> route reported only <code class="inline">walk</code>, <code class="inline">hash</code> and <code class="inline">scan</code>. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the <code class="inline">scan</code> that had already finished:</p><pre class="code"><code class="language-text">xerj-progress phase=scan basis=bytes pct=100.0 items=48533/48533 bytes=520892779/520892779 rate=17050775.5 eta_s=unknown eta_quality=stalled since_progress_s=15.0 phase_elapsed_s=25.7 elapsed_s=30.0</code></pre>- <p>The rc.74 capture holds 48 such lines, with <code class="inline">since_progress_s</code> climbing to 250.0. A real hang at the end of the scan prints exactly the same thing. Neither a person nor an agent could tell the two apart.</p>+ <p>The rc.74 capture holds 48 such lines, with <code class="inline">since_progress_s</code> climbing to 250.0. A real hang at the end of the <code class="inline">scan</code> prints exactly the same thing. Neither a person nor an agent could tell the two apart.</p><p>On a current build each of those steps is a phase, and the small-repository capture holds 0 lines that read <code class="inline">scan</code> at <code class="inline">pct=100.0</code>. If you are on rc.74 or earlier and see that line, check <code class="inline">_count</code> on the node before you conclude the run is hung.</p><h2>The line that ends the run</h2><p><code class="inline">xerj-done</code> is the only line a script must parse to decide success. The captured terminal line reads in full:</p><pre class="code"><code class="language-text">xerj-done ok=true exit=0 reason=completed wall=22.2s files=1 records=164441 datasets=1 junk_files=0</code></pre>- <p><code class="inline">reason</code> distinguishes <code class="inline">completed</code>, <code class="inline">dry-run</code>, <code class="inline">completed-with-junk</code>, <code class="inline">aborted</code> and, on the <code class="inline">--no-graph</code> path, <code class="inline">server-backpressure</code>, and the exit code follows it. Exit 3 with <code class="inline">completed-with-junk</code> means the run refused some files, and the catalog holds a reason for each one. <code class="inline">server-backpressure</code> is exit 1: the node kept rejecting for the whole 600 s a bulk waits, and the line adds <code class="inline">ops_applied</code> and <code class="inline">ops_remaining</code> so you know how much the same command still has to do.</p>+ <p><code class="inline">reason</code> distinguishes <code class="inline">completed</code>, <code class="inline">dry-run</code>, <code class="inline">completed-with-junk</code>, <code class="inline">aborted</code> and, on the <code class="inline">--no-graph</code> path, <code class="inline">server-backpressure</code>, and the exit code follows it. Exit 3 with <code class="inline">completed-with-junk</code> means the run refused some files, and the catalog holds a reason for each one. <code class="inline">server-backpressure</code> is exit 1. The node kept rejecting for the whole 600 s a bulk waits. The line adds <code class="inline">ops_applied</code> and <code class="inline">ops_remaining</code> so you know how much the same command still has to do.</p>+ <p>Exit 3 is success. Treat it the same as exit 0. Do not rerun it as a failure.</p><p>If the server refused a whole dataset, the line also carries <code class="inline">datasets_refused</code> and <code class="inline">files_refused</code>. They appear only when it happened. The <a href="/answers/autoindex-dataset-refused-by-server">refused-dataset page</a> covers that case.</p><p>If the node pushed back with HTTP 429 during the run, the line carries <code class="inline">bulk_retries</code>, the number of bulks the run re-sent. It also appears only when it happened. If the node refused a request for its size (HTTP 413) and the run cut it in two, the line carries <code class="inline">bulk_splits</code>, also only when it happened.</p><h2>When the node pushes back</h2>diff --git a/landing/answers/read-autoindex-progress.md b/landing/answers/read-autoindex-progress.mdindex 42637711b..2473a0341 100644--- a/landing/answers/read-autoindex-progress.md+++ b/landing/answers/read-autoindex-progress.md@@ -2,13 +2,13 @@---title: "How do I read autoindex progress?"canonical: "https://xerj.org/answers/read-autoindex-progress"-updated: "2026-10-19"+updated: "2026-10-24"source: "content/answers/read-autoindex-progress.md"---# How do I read autoindex progress?-**Short answer** (updated 2026-10-19): How can I see whether folder indexing is still working? Pass --progress plain and read the xerj-progress lines on stderr. A captured 22.2 s run emitted 12 phases in order and ended with xerj-done ok=true. On a single 16 MB file the percentage stays at 0.0%, so read since_progress_s and waiting_on instead.+**Short answer** (updated 2026-10-24): How can I see whether folder indexing is still working? Pass --progress plain and read the xerj-progress lines on stderr. A captured 22.2 s run emitted 12 phases in order and ended with xerj-done ok=true. On a single 16 MB file the percentage stays at 0.0%, so read since_progress_s and waiting_on instead.**Key number:** With --no-graph, a 231-file repository reported 9 phases in order, and its index phase read pct=81.7 at 169/231 items. (as of 2026-10-19; source: [benchmarks/autoindex-resilience/after-fix.small-repo.stderr.txt](https://github.com/xerj-org/xerj/blob/main/benchmarks/autoindex-resilience/after-fix.small-repo.stderr.txt))@@ -123,15 +123,15 @@ Every run shows `phase=starting` until its first real phase opens. On that resumThe `index` phase applies one file at a time, so it is the slow one on a large corpus. Read `eta_s` once `eta_quality` leaves `unknown`.-## scan at 100% now means scan+## `scan` at 100% now means `scan`-Through v1.0.0-rc.74 the `--no-graph` route reported only `walk`, `hash` and `scan`. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the scan that had already finished:+Through v1.0.0-rc.74 the `--no-graph` route reported only `walk`, `hash` and `scan`. Mapping install, sealing, indexing and the read-back all ran with no phase of their own, so the stream kept describing the `scan` that had already finished:```textxerj-progress phase=scan basis=bytes pct=100.0 items=48533/48533 bytes=520892779/520892779 rate=17050775.5 eta_s=unknown eta_quality=stalled since_progress_s=15.0 phase_elapsed_s=25.7 elapsed_s=30.0```-The rc.74 capture holds 48 such lines, with `since_progress_s` climbing to 250.0. A real hang at the end of the scan prints exactly the same thing. Neither a person nor an agent could tell the two apart.+The rc.74 capture holds 48 such lines, with `since_progress_s` climbing to 250.0. A real hang at the end of the `scan` prints exactly the same thing. Neither a person nor an agent could tell the two apart.On a current build each of those steps is a phase, and the small-repository capture holds 0 lines that read `scan` at `pct=100.0`. If you are on rc.74 or earlier and see that line, check `_count` on the node before you conclude the run is hung.@@ -143,7 +143,9 @@ On a current build each of those steps is a phase, and the small-repository captxerj-done ok=true exit=0 reason=completed wall=22.2s files=1 records=164441 datasets=1 junk_files=0```-`reason` distinguishes `completed`, `dry-run`, `completed-with-junk`, `aborted` and, on the `--no-graph` path, `server-backpressure`, and the exit code follows it. Exit 3 with `completed-with-junk` means the run refused some files, and the catalog holds a reason for each one. `server-backpressure` is exit 1: the node kept rejecting for the whole 600 s a bulk waits, and the line adds `ops_applied` and `ops_remaining` so you know how much the same command still has to do.+`reason` distinguishes `completed`, `dry-run`, `completed-with-junk`, `aborted` and, on the `--no-graph` path, `server-backpressure`, and the exit code follows it. Exit 3 with `completed-with-junk` means the run refused some files, and the catalog holds a reason for each one. `server-backpressure` is exit 1. The node kept rejecting for the whole 600 s a bulk waits. The line adds `ops_applied` and `ops_remaining` so you know how much the same command still has to do.++Exit 3 is success. Treat it the same as exit 0. Do not rerun it as a failure.If the server refused a whole dataset, the line also carries `datasets_refused` and `files_refused`. They appear only when it happened. The [refused-dataset page](/answers/autoindex-dataset-refused-by-server) covers that case.diff --git a/landing/answers/why-autoindex-skipped-files.html b/landing/answers/why-autoindex-skipped-files.htmlindex e8a399c3e..129fd6871 100644--- a/landing/answers/why-autoindex-skipped-files.html+++ b/landing/answers/why-autoindex-skipped-files.html@@ -79,7 +79,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-08-21",- "dateModified": "2026-10-19",+ "dateModified": "2026-10-24","author": {"@type": "Organization","name": "XERJ documentation team",@@ -164,7 +164,7 @@"name": "What does exit code 3 from autoindex mean?","acceptedAnswer": {"@type": "Answer",- "text": "The run finished but refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals."+ "text": "Exit 3 is success. The run finished and refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals. Do not rerun it as a failure."}}]@@ -228,9 +228,9 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>WHY WOULD A FOLDER SEARCH MISS FILES THAT I CAN SEE ON DISK?</div><div class="kicker"><span>ANSWERS · OPERATIONS: EXCLUSIONS</span></div><h1>Why would a folder search miss files that I can see on disk?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-19</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-08-21 · UPDATED · 2026-10-24</div><div class="prompt-card answer-first" id="short-answer">- <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-19</span>+ <span class="prompt-label">SHORT ANSWER · UPDATED 2026-10-24</span><p>Why would a folder search miss files that I can see on disk? XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, excluded 23 through ignore rules, and exited 3.</p><p class="key-fact"><strong>Key number:</strong> The captured run reconciled 26 files on disk as 1 indexed, 2 refused and 23 excluded. <span class="mute">(as of 2026-10-05 · source: the captured run documented on this page)</span></p></div>@@ -303,7 +303,7 @@ autoindex: ignore rules: .gitignore:*.tmp — 1 file<p>The captured run ended with a terminal line that names both the outcome and the refusal count.</p><pre class="code"><code class="language-text">xerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0</code></pre>- <p>Exit 0 with <code class="inline">reason=completed</code> means nothing was refused. Exit 3 with <code class="inline">reason=completed-with-junk</code> means the catalog holds at least one reason worth reading.</p>+ <p>Exit 0 with <code class="inline">reason=completed</code> means nothing was refused. Exit 3 with <code class="inline">reason=completed-with-junk</code> is success: the run finished, the index is searchable, and the catalog holds at least one reason worth reading. Do not rerun exit 3 as a failure.</p><p>Every number above comes from a capture on 2026-10-05, on a build of <code class="inline">main</code> after v1.0.0-rc.82 (commit <code class="inline">06185966</code>). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as <code class="inline">binary content (unknown)</code>. The binary was a <code class="inline">quick</code> profile build, so the <code class="inline">wall</code> figure is not a performance number.</p><h2>FAQ</h2><h3>Why would a folder search miss files that I can see on disk?</h3>@@ -319,7 +319,7 @@ autoindex: ignore rules: .gitignore:*.tmp — 1 file<h3>Does autoindex index my .env file?</h3><p>Not in this capture. Hidden dotfiles were pruned, and a query across _all for the fixture secrets returned 0 hits. Ignore rules are a filter, not an access control.</p><h3>What does exit code 3 from autoindex mean?</h3>- <p>The run finished but refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals.</p>+ <p>Exit 3 is success. The run finished and refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals. Do not rerun it as a failure.</p><h2>Related</h2><ul><li><a href="/answers/search-file-contents-in-a-folder">I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once?</a></li>diff --git a/landing/answers/why-autoindex-skipped-files.md b/landing/answers/why-autoindex-skipped-files.mdindex 257c9977c..d8cb62612 100644--- a/landing/answers/why-autoindex-skipped-files.md+++ b/landing/answers/why-autoindex-skipped-files.md@@ -2,13 +2,13 @@---title: "Why does a folder search miss files on disk?"canonical: "https://xerj.org/answers/why-autoindex-skipped-files"-updated: "2026-10-19"+updated: "2026-10-24"source: "content/answers/why-autoindex-skipped-files.md"---# Why would a folder search miss files that I can see on disk?-**Short answer** (updated 2026-10-19): Why would a folder search miss files that I can see on disk? XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, excluded 23 through ignore rules, and exited 3.+**Short answer** (updated 2026-10-24): Why would a folder search miss files that I can see on disk? XERJ prints one line per ignore rule with an exact file count, and stores a refusal reason for every file it opened. A capture over 26 files on disk indexed 1, refused 2 with named reasons, excluded 23 through ignore rules, and exited 3.**Key number:** The captured run reconciled 26 files on disk as 1 indexed, 2 refused and 23 excluded. (as of 2026-10-05; source: the captured run documented on this page)@@ -113,7 +113,7 @@ The captured run ended with a terminal line that names both the outcome and thexerj-done ok=true exit=3 reason=completed-with-junk wall=3.1s files=1 records=2 datasets=1 junk_files=2 code_files=0 code_files_indexed=0 code_files_junked=0```-Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` means the catalog holds at least one reason worth reading.+Exit 0 with `reason=completed` means nothing was refused. Exit 3 with `reason=completed-with-junk` is success: the run finished, the index is searchable, and the catalog holds at least one reason worth reading. Do not rerun exit 3 as a failure.Every number above comes from a capture on 2026-10-05, on a build of `main` after v1.0.0-rc.82 (commit `06185966`). The Parquet hint is not in the v1.0.0-rc.82 release itself: that release reports such a file as `binary content (unknown)`. The binary was a `quick` profile build, so the `wall` figure is not a performance number.@@ -145,7 +145,7 @@ Not in this capture. Hidden dotfiles were pruned, and a query across _all for th### What does exit code 3 from autoindex mean?-The run finished but refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals.+Exit 3 is success. The run finished and refused at least one file. The terminal line reads reason=completed-with-junk, and junk_files counts the refusals. Do not rerun it as a failure.## Relateddiff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.jsonindex e0f0a8355..7162480bd 100644--- a/landing/docs/agents/schemas/mcp-tools.json+++ b/landing/docs/agents/schemas/mcp-tools.json@@ -131,7 +131,7 @@},{"name": "xerj_search",- "description": "Full-text / keyword / structured search over a XERJ index. Proxies POST /{index}/_search. Pass `prefix` (a `xerj search --prefix` value, resolved by that command) or `index` (a name or pattern) — not both; with neither, searches `ax-*`. Pass `query` as a PLAIN STRING for the definition-first code search (ranks the file that defines a symbol above files that merely mention it — the right default for code), or as an ES query-DSL object (e.g. {\"term\":{\"status\":\"open\"}}, or a bool clause) for structured queries. Omit `query` for match_all. Optional `rerank` adds a second stage that re-judges the top hits with an external relevance model — read that argument's description before using it: it needs a provider the operator configured, and it sends document text off the machine. Optional `judge` re-judges the returned hits LOCALLY (zero tokens, nothing leaves the node) and adds a `_p_relevant` probability per hit — see that argument's description for its honest ceiling. Each hit carries `match_line` when a query term sits in the passage: the absolute 1-based source line of the best-scoring match (a tie takes the earliest line), and `match_lines` for every matched line (capped). Cite `path:match_line`. Both are omitted when no term is located or the passage start line is unknown. `start_line` remains the passage start. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule).",+ "description": "Full-text / keyword / structured search over a XERJ index. Proxies POST /{index}/_search. Pass `prefix` (a `xerj search --prefix` value, resolved by that command) or `index` (a name or pattern) — not both; with neither, searches `ax-*`. Pass `query` as a PLAIN STRING for the definition-first code search (ranks the file that defines a symbol above files that merely mention it — the right default for code), or as an ES query-DSL object (e.g. {\"term\":{\"status\":\"open\"}}, or a bool clause) for structured queries. Omit `query` for match_all. Optional `rerank` adds a second stage that re-judges the top hits with an external relevance model — read that argument's description before using it: it needs a provider the operator configured, and it sends document text off the machine. Optional `judge` re-judges the returned hits LOCALLY (zero tokens, nothing leaves the node) and adds a `_p_relevant` probability per hit — see that argument's description for its honest ceiling. Each hit carries `match_line` when a query term sits in the passage: the absolute 1-based source line of the best-scoring match (a tie takes the earliest line), and `match_lines` for every matched line (capped). Cite `path:match_line`. Both are omitted when no term is located or the passage start line is unknown. `start_line` remains the passage start. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). If this prefix was autoindexed from a folder on this machine and a file there is newer than the last run, or an indexed file has been removed, the response adds `_xerj.hints` code `index_stale` with the last index time and a reindex command. Hits are returned with that hint. The check stats at most 4096 entries, stops after 100ms, and skips hidden and ignored paths.","inputSchema": {"type": "object","properties": {diff --git a/landing/docs/clustering.html b/landing/docs/clustering.htmlindex 81f22bf8f..fc43e202a 100644--- a/landing/docs/clustering.html+++ b/landing/docs/clustering.html@@ -236,12 +236,12 @@<h1>Clustering</h1>- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, or <code class="inline">POST /{index}/_update/{id}</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">POST /{index}/_update_by_query</code> runs on every primary of that index and each primary copies the documents it updated. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, or <code class="inline">POST /{index}/_update/{id}</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">POST /{index}/_update_by_query</code> runs on every primary of that index and each primary copies the documents it updated. <code class="inline">POST /{index}/_delete_by_query</code> runs on every primary and each primary copies the deletes. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p><h2>What cluster mode does not give you yet</h2><div class="enum-list"><div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it until it has copied the primary's live documents and that enlarged set has committed. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>- <div class="item">Native ingest and delete-by-query stay on the receiver<span class="dsc">The native <code class="inline">/v1</code> ingest paths and <code class="inline">_delete_by_query</code> still write on the node that received them. They do not consult the shard plan. <code class="inline">POST /{index}/_update_by_query</code> runs on every primary. Each primary updates only its own shard and copies those documents to the in-sync replicas. <code class="inline">max_docs</code> counts across those primaries, in node-id order. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the update. <code class="inline">?pipeline=</code> runs on the primary. Pipelines are not replicated, so a pipeline that is not on that primary is <code class="inline">400</code> and writes nothing.</span></div>+ <div class="item">Native ingest stays on the receiver<span class="dsc">The native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan. <code class="inline">POST /{index}/_delete_by_query</code> and <code class="inline">POST /{index}/_update_by_query</code> run on every primary. Each primary deletes or updates only the documents in its own shard and copies that change to the in-sync replicas. <code class="inline">max_docs</code> counts across those primaries, in node-id order. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the delete or the update. <code class="inline">?pipeline=</code> on an update-by-query runs on the primary. Pipelines are not replicated, so a pipeline that is not on that primary is <code class="inline">400</code> and writes nothing.</span></div><div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div><div class="item">No replica promotion<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the snapshot plus the log suffix before it serves, so the shard plan and the in-sync set come back. After 64 applied metadata entries the prefix is snapshotted and a member whose log does not overlap receives InstallSnapshot. The snapshot is the metadata commands, not the documents. A placed replica that is missing those documents copies them from the primary and is published back into the in-sync set; a write still needs the primary, and there is no promotion. A metadata history bigger than 8 MiB stays in the log instead of being snapshotted. Changing membership does not move copies.</span></div><div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div>@@ -255,7 +255,7 @@<li><strong>Dead-peer isolation</strong> — a hung or dead peer costs one <code class="inline">WARN</code> on the transition and a retry ladder backing off 100 ms → 6.4 s; it never delays messages to live peers.</li><li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li><li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li>- <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, <code class="inline">POST /{index}/_update/{id}</code>, <code class="inline">POST /{index}/_update_by_query</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. A bulk body is split by primary; the receiver does not keep another member's actions. An update-by-query is sent to every primary, and each primary updates only the documents in its shard. The primary copies an index, a delete, or the document an update or update-by-query stored to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>+ <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, <code class="inline">POST /{index}/_update/{id}</code>, <code class="inline">POST /{index}/_update_by_query</code>, <code class="inline">POST /{index}/_delete_by_query</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. A bulk body is split by primary; the receiver does not keep another member's actions. An update-by-query or a delete-by-query is sent to every primary, and each primary changes only the documents in its shard. The primary copies an index, a delete (one document or a delete-by-query), or the document an update or update-by-query stored to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li><li><strong>On-disk Raft log</strong> — cluster mode opens <code class="inline">{data_dir}/_raft</code> (<code class="inline">raft.log</code>, <code class="inline">snapshot.meta</code> once the prefix has been compacted, plus the commit index, term, and vote). The directory name is not a valid index name, so deleting an index cannot delete the log. On startup the node replays the snapshot and then the committed suffix into the shard plan and the in-sync set before it serves. Indices and documents already in that data directory stay; the plan is what tells a later write where they live. A member whose log does not overlap the leader's copies the log, or installs the snapshot when the leader has compacted the prefix, and applies it.</li></ul>@@ -323,10 +323,10 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre><h2>What replicates, and what stays on one node</h2><p><code class="inline">PUT /orders</code> on any member — leader or follower — creates <code class="inline">orders</code> on every member, including its mapping, the settings and aliases in that request, and a primary on each member. <code class="inline">GET /orders</code> and <code class="inline">GET /orders/_mapping</code> on another member succeed after the call returns. <code class="inline">DELETE /orders</code> removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns <code class="inline">503</code>; it does not create the index locally and report success.</p>- <p><code class="inline">PUT /orders/_doc/1</code>, <code class="inline">POST /orders/_update/1</code>, and each action in <code class="inline">POST /orders/_bulk</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. <code class="inline">POST /orders/_update_by_query</code> does the same for every document it changes. An action whose primary is another member is not written into the accepting node's index. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>+ <p><code class="inline">PUT /orders/_doc/1</code>, <code class="inline">POST /orders/_update/1</code>, and each action in <code class="inline">POST /orders/_bulk</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. <code class="inline">POST /orders/_update_by_query</code> does the same for every document it changes. <code class="inline">POST /orders/_delete_by_query</code> deletes on every primary and copies each delete to the in-sync replicas. An action whose primary is another member is not written into the accepting node's index. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p><p>Several things around that path stay local, on purpose for this step:</p><ul>- <li><strong><code class="inline">_delete_by_query</code> and the native <code class="inline">/v1</code> ingest API</strong> do not route or replicate. <code class="inline">_update_by_query</code> does: it runs on every primary and the primary copies the documents it changed. For a copy of the paths that still stay on the receiving node, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>+ <li><strong>The native <code class="inline">/v1</code> ingest API</strong> does not route or replicate. <code class="inline">_delete_by_query</code> and <code class="inline">_update_by_query</code> do: each runs on every primary and the primary copies the documents it deleted or changed. For a copy of the paths that still stay on the receiving node, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li><li><strong>Index templates</strong> are not replicated. The accepting node applies its templates once and the effective schema is what the log carries.</li><li><strong>ILM/ISM attach</strong> from <code class="inline">index.lifecycle.name</code> runs on the accepting node only.</li><li><strong>Console system indices</strong> created at boot, before the ring is attached, have no shard plan and stay local.</li>@@ -341,7 +341,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre></div><h2>When not to cluster</h2>- <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a document write (<code class="inline">PUT</code>/<code class="inline">POST</code> <code class="inline">_doc</code>, <code class="inline">_bulk</code>, <code class="inline">_update</code>, or <code class="inline">_update_by_query</code>) once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>+ <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a document write (<code class="inline">PUT</code>/<code class="inline">POST</code> <code class="inline">_doc</code>, <code class="inline">_bulk</code>, <code class="inline">_update</code>, <code class="inline">_update_by_query</code>, or <code class="inline">_delete_by_query</code>) once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p><p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 52ffcff9e..6674154db 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -471,6 +471,20 @@ after any --no-graph/--approve fast run over code,POST /<prefix>-*/_count {"query":{"exists":{"field":"language"}}}must be > 0 for a tree that contains source. If it is 0, re-run without --no-graph.+A run that junked every file of one family, or a strict majority of that family's+files (`junk_files * 2 > files`; stale-plan skips are not in the ratio), names+that family in the human summary and on the progress stream. `--json` adds+`family_junk_warnings` — omitted when nothing tripped it, so existing summary+fields are unchanged. Each entry has `family`, `files`, `junk_files`,+`indexed_files`, `extent` (`all` or `most`), `top_reason` (the most common junk+reason; equal counts break by Unicode code-point order, smaller first) and+`top_reason_files`. The same array is stored on the catalog run document:+`xerj autoindex map` prints "Families mostly or entirely junked — NOT indexed"+above the dataset table, and the MCP `xerj_map` result includes+`family_junk_warnings` taken from the latest run of each prefix. Exit codes+are unchanged. Ordinary junk is still exit 3. A family-wide PDF worker startup+failure is still exit 1 (`pdf_worker_unavailable`).+--fresh ignores the resume journal and restarts (ids stay idempotent forunchanged bytes); on a graph-enabled or pre-generation state directory that ishow you pick up files added since the last run. A file that is still present@@ -512,7 +526,12 @@ Both are refused on `autoindex map` and `autoindex status`, which never walk afolder.Your global gitignore (core.excludesFile) is deliberately not read. The folderyou point at is never excluded: if it is itself ignored, it is indexed anyway-and the run says which rule it would have matched.+and the run says which rule it would have matched. A line that writes+`[locale]` is also applied to the literal directory of that name, so a+Next.js route folder is not dropped because git reads the brackets as a+character class. The class reading stays: `*.[ch]` still matches `.c` and+`.h`. The ignore summary names a route-like class, a pruned `[…]` directory,+and each skipped code file with its rule.Exit codes: 0 complete (also: the gate was answered `--approve cancel`) ·3 completed-with-junk (junk recorded, never fatal — this is SUCCESS, do not report@@ -1545,6 +1564,10 @@ apply; if it does not, YOU are the gate — estimate, ask, and only then run.holding a floor, not a total. `--no-ignore` indexes everything the rules would havedropped; `--no-default-ignores` keeps the ignore files and drops only the built-in list;both are refused on `autoindex map` and `autoindex status`, which never walk a folder.+ A line that writes `[locale]` is also applied to the literal directory of that name+ (a Next.js route folder); the character-class reading stays, so `*.[ch]` still+ matches `.c` and `.h`. The summary names a route-like class, a pruned `[…]`+ directory, and each skipped code file with its rule.Your GLOBAL gitignore is deliberately not read.- READ THE PROGRESS STREAM — IT IS DESIGNED FOR YOU. `autoindex` writes its RESULT todiff --git a/landing/sitemap.xml b/landing/sitemap.xmlindex d6dc796e5..68ddebd45 100644--- a/landing/sitemap.xml+++ b/landing/sitemap.xml@@ -38,7 +38,7 @@</url><url><loc>https://xerj.org/answers/autoindex-exit-codes</loc>- <lastmod>2026-10-19</lastmod>+ <lastmod>2026-10-24</lastmod></url><url><loc>https://xerj.org/answers/autoindex-says-extract-the-archive-first</loc>@@ -66,7 +66,7 @@</url><url><loc>https://xerj.org/answers/check-codebase-index-is-complete</loc>- <lastmod>2026-10-19</lastmod>+ <lastmod>2026-10-24</lastmod></url><url><loc>https://xerj.org/answers/classify-text-from-labelled-history-without-llm</loc>@@ -206,7 +206,7 @@</url><url><loc>https://xerj.org/answers/read-autoindex-progress</loc>- <lastmod>2026-10-19</lastmod>+ <lastmod>2026-10-24</lastmod></url><url><loc>https://xerj.org/answers/read-indexed-email-and-pdf-folder</loc>@@ -370,7 +370,7 @@</url><url><loc>https://xerj.org/answers/why-autoindex-skipped-files</loc>- <lastmod>2026-10-19</lastmod>+ <lastmod>2026-10-24</lastmod></url><url><loc>https://xerj.org/architecture/</loc>diff --git a/recipes/README.md b/recipes/README.mdindex bd6626410..a3da17460 100644--- a/recipes/README.md+++ b/recipes/README.md@@ -83,7 +83,8 @@ Every recipe honors `XERJ_URL` (default `http://localhost:9200`).any recipe — start the server with a different `--embed-mode`: `neural`(built-in in-process BERT, all-MiniLM-L6-v2, shipped in the binary; themodel auto-downloads on first use) or `proxy` (any OpenAI-compatible- embeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml`).+ embeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml` to+ a `/v1` base, a bare host, or the full `/v1/embeddings` URL).- `memory_agent.py` recalls with **text (BM25) relevance** scores(unbounded, higher = better) by default. Add `"semantic": true` to the`_recall` body to have XERJ embed the query server-side and recall by