Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Keep Boolean filters non-scoring for buffered text matches

xerj-org29f2ae6cf4a7
diff --cc CHANGELOG.md
index d93c6a1e7,22cf1ee90..8bf9a8670
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,102 -7,20 +7,106 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Added
+
+- Corpus Hub manifest `paths` entries may name regular files as well as
+ directories (#2486). A file selection checks out that file without its
+ siblings; symlinks and submodules are refused.
+
+- MCP `xerj_status` reports node reachability, cluster health and visible
+ indices before a search. It distinguishes a missing autoindex catalog from
+ other indexed data and preserves failed or malformed probes as errors
+ rather than reporting a successful empty node (#2515).
+
+- Empty autoindex maps now explain that managed corpus slugs and autoindex
+ dataset/index names belong to separate discovery surfaces. CLI diagnostics,
+ MCP empty responses and the `xerj_map` description point to corpus discovery
+ without claiming a corpus exists. Missing catalogs remain errors; auth
+ refusals and successful dataset maps retain their behavior (#2412).
+
+- Public Agent Skills Discovery v0.2.0 index and a documentation-routing skill
+ at `/.well-known/agent-skills/`. The index is generated from the published
+ skill's metadata and exact-byte SHA-256, with a CI drift check.
+
### Fixed
-- A `filter` or `must_not` clause no longer changes `_score` before a flush
- (#2403). `bool{must:[match], filter:[term]}` scored buffered documents with
- an IDF-less `1 + ln(1 + tf)`, so on the issue's repro the top hit scored
- 2.609438 and three hits tied at 1.6931472, while the bare `match` scored
- 0.6979745, 0.60089254, 0.5022658 and 0.2530558; the ranking changed at
- flush. The `must` match now takes the same BM25 path as the bare `match`
- and the non-scoring clauses only remove hits. On an index with flushed and
- buffered documents, the shared BM25 statistics also count the buffered
- documents for this shape, so flushed hits score as they do under the bare
- `match`. Covers one `must` match (without its own `analyzer`) plus any
- `filter` or `must_not` clauses with no `should`; other `bool` shapes keep
- their current route.
++- Boolean text queries with one `must` match now keep `filter` and `must_not`
++ clauses non-scoring before a flush and on mixed memory/segment indices.
++ Their retained hits use the same BM25 scores as the bare match (#2403).
++
+- Reindex preflights every source alias member before creating or writing the
+ destination. Aliases cannot hide a source/destination overlap; a missing
+ source, invalid query or source flush failure leaves destination data intact.
+
+- Corpus cloning validates existing checkouts and serializes writers (#2453).
+ New checkouts are staged before publication. Ambiguous existing directories
+ are preserved for recovery rather than deleted or reset automatically.
+
+- MCP `xerj_search` advertises plain text in `query` and structured clauses
+ in `query_dsl`. Legacy object-valued `query` and omitted-query browsing
+ remain supported; conflicting fields fail before HTTP requests (#2549).
+- Search payloads and batch retrieval default to 8,000 UAX #29 words when
+ `max_tokens` is absent or null (#2491). This is not a byte or model-token
+ cap. Explicit larger budgets remain available; errors and mandatory
+ search staleness hints stay outside the payload budget.
+- GitHub field-report submission uses the supported explicit-repository
+ `gh repo fork --clone=false` invocation and adds the fork remote itself
+ (#2596). The rollback test now verifies that the report was committed
+ before a failed push, rather than passing after an earlier fork failure.
+- Local admin-key discovery reports the source of the selected key and warns
+ when its directory is outside the working tree (#2578). Packaged installs
+ have their own provenance, and MCP guidance names `XERJ_AUTH`. Discovery
+ priority and explicit credential overrides are unchanged.
+
+- `xerj autoindex` fails a run when one dataset index holds no documents
+ although the journal says records were written to it, even when the other
+ datasets are live (#2407, defect 2). The check summed live documents over
+ every dataset, so a populated `-json` index let a re-run print `ok: indexed
+ and searchable` over an empty `-docs` index. The error names each empty
+ index and its contributing files; record totals are reported only when
+ those files target that dataset alone. Unknown sibling counts do not hide
+ independently confirmed missing data.
+- SQL source extraction no longer slices statement heads inside a UTF-8
+ character, so unrecognized CJK statement/comment heads do not panic (#2547).
+- Short Markdown files with YAML frontmatter retain their prose body even
+ when bullets or `Key: value` lines also parse as YAML (#2447). Existing
+ frontmatter-title behavior and structured CSV/JSON body detection remain.
+- Empty files keep separate inventory identities instead of being reported
+ as content duplicates of unrelated paths (#2449). Non-empty byte-identical
+ files still deduplicate.
+- A search with both `query` and a top-level `knn` returns `size` hits, not
+ `k` (#2561). The page was widened to `k`, so `size: 5, k: 20` returned
+ 20 hits. `k` still sets how many nearest neighbours join the candidate
+ set, and `from`/`size` page through the union, as in Elasticsearch.
+- Index deletion no longer waits for unrelated index statistics while
+ resolving its selectors. Index listings and health checks also release
+ their index-map locks before awaiting statistics, so a stalled schema
+ update does not hold those locks against DELETE (part of #2405; extracted
+ from #2489). Merge/flush cancellation and the process memory cap remain
+ separate follow-ups.
+- `xerj autoindex` compresses per-file phase-B stages with zstd level 1 and
+ uses temporary files cleaned up by the operating system when the client
+ exits (#2493, part of #2427). It finishes buffered compression before
+ deleting previous records, so a stage finalization error preserves the old
+ records and an ordinary resume can complete the replacement. Compression
+ savings depend on the data; this does not impose a disk or server-memory cap.
+- `autoindex` bulk back-pressure patience is now measured since the last
+ progress, not since the first round. A window the server drains a few
+ records at a time (per-item 429 `resource_exhausted` on an embedding
+ deadline or admission refusal) used to abort with `bulk backend failed`
+ once the rounds together passed the 600 s `THROTTLE_PATIENCE`, even though
+ every round settled records. Each round that settles at least one item now
+ restarts the patience clock and the backoff; rounds that settle nothing
+ still count, so a server that answers 429 forever fails at the same limits
+ as before.
+- `xerj autoindex` titles a prose file that opens with YAML frontmatter from
+ its `title:` key, or from the first line after the closing `---` when there
+ is none (#2556). Every such file (Jekyll, Hugo, Obsidian) used to be titled
+ `---`, its opening fence. A leading `---` block that is not parseable
+ frontmatter (unclosed, not a YAML mapping, over 64 KiB) takes the first line
+ that is not a bare `---`; the body, the frontmatter text in it and
+ `start_line` are unchanged. An unchanged file is not re-extracted, so re-index
+ with `xerj autoindex --fresh` to retitle documents already in an index.
## [1.0.0-rc.100] - 2026-10-26
diff --cc engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rs
index 000000000,31caa521d..9c03b259e
mode 000000,100644..100644
--- a/engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rs
+++ b/engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rs
@@@ -1,0 -1,239 +1,292 @@@
+ //! Issue #2403: `bool{must:[match], filter:[...]}` scored buffered documents
+ //! with the IDF-less brute scorer (`score_query_against_doc`:
+ //! `1 + ln(1 + tf)`), while the identical `must` clause on its own took the
+ //! memtable BM25 arm. Before `_flush` the filtered query ranked a one-word
+ //! document level with an eight-word one, and every score moved at flush.
+ //!
+ //! A `filter` or `must_not` clause is non-scoring (`BooleanClause.isScoring()`,
+ //! BooleanClause.java:84-87), so the contract is the one #361 pinned for the
+ //! segment path, now held in every phase: each hit scores exactly what the
+ //! `must` clause scores alone, in the same phase, and the hit set is the
+ //! `must` hits the filter keeps. `bool_filter_is_non_scoring.rs` covers the
+ //! flushed shapes; this file covers the memtable and the mixed index.
+ //!
+ //! Elasticsearch is referenced for wire semantics only; no ES code is here.
+
+ use std::collections::{BTreeSet, HashMap};
+
+ use serde_json::{json, Value};
+ use tempfile::TempDir;
+ use xerj_common::config::Config;
+ use xerj_common::types::{FieldConfig, FieldType, Schema};
+ use xerj_engine::{Engine, Index};
+ use xerj_query::executor::SearchResult;
+ use xerj_query::parse_request;
+
+ fn make_engine(dir: &TempDir) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ Engine::new(config).expect("engine::new")
+ }
+
+ fn req(q: &Value) -> xerj_query::ast::SearchRequest {
+ parse_request(&json!({ "query": q, "size": 100 })).expect("parse_request")
+ }
+
+ async fn search(idx: &Index, q: &Value) -> SearchResult {
+ idx.search(&req(q)).await.unwrap()
+ }
+
+ /// `alpha` appears in most bodies with varying frequency and body length, so
+ /// BM25 and the brute scorer disagree on order; `tag` splits the corpus.
+ fn doc(i: usize) -> Value {
+ let body = match i % 4 {
+ 0 => "alpha alpha alpha alpha".to_string(),
+ 1 => "alpha".to_string(),
+ 2 => "alpha beta gamma delta epsilon zeta eta theta".to_string(),
+ _ => format!("beta gamma {i}"),
+ };
+ json!({ "body": body, "tag": if i.is_multiple_of(3) { "y" } else { "x" } })
+ }
+
+ async fn seed(name: &str, dir: &TempDir) -> std::sync::Arc<Index> {
+ let engine = make_engine(dir);
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ schema
+ .fields
+ .push(FieldConfig::new("tag", FieldType::Keyword));
+ engine.create_index(name, schema).unwrap();
+ engine.get_index(name).unwrap()
+ }
+
+ async fn index_range(idx: &Index, range: std::ops::Range<usize>) {
+ for i in range {
+ idx.index_document(Some(format!("d{i:02}")), doc(i))
+ .await
+ .unwrap();
+ }
+ }
+
+ /// Every hit of `filtered` scores what `bare` scores for the same `_id`,
+ /// the order is `bare`'s order restricted to the kept set, the kept set is
+ /// `want`, and `hits.total` counts exactly the kept set.
+ async fn assert_scores_like_bare(
+ idx: &Index,
+ bare: &Value,
+ filtered: &Value,
+ want: &BTreeSet<String>,
+ phase: &str,
+ ) {
+ let reference = search(idx, bare).await;
+ let actual = search(idx, filtered).await;
+ let got: BTreeSet<String> = actual.hits.iter().map(|h| h.id.clone()).collect();
+ assert_eq!(&got, want, "{phase}: hit set of {filtered}");
+ assert_eq!(
+ actual.total.value as usize,
+ want.len(),
+ "{phase}: hits.total of {filtered}"
+ );
+ let reference_scores: HashMap<&str, f32> = reference
+ .hits
+ .iter()
+ .map(|h| (h.id.as_str(), h.score))
+ .collect();
+ for hit in &actual.hits {
+ let expected = reference_scores[hit.id.as_str()];
+ assert!(
+ (hit.score - expected).abs() < 1e-4,
+ "{phase}: {} scores {} under {filtered} but {expected} under the bare \
+ must clause; a non-scoring clause changed _score (#2403)",
+ hit.id,
+ hit.score
+ );
+ }
+ let order: Vec<&str> = reference
+ .hits
+ .iter()
+ .map(|h| h.id.as_str())
+ .filter(|id| got.contains(*id))
+ .collect();
+ let actual_order: Vec<&str> = actual.hits.iter().map(|h| h.id.as_str()).collect();
+ assert_eq!(actual_order, order, "{phase}: order of {filtered} (#2403)");
+ }
+
+ /// The docs among `0..n` whose body holds `alpha` and that `keep` admits.
+ fn alpha_docs(n: usize, keep: impl Fn(usize) -> bool) -> BTreeSet<String> {
+ (0..n)
+ .filter(|i| i % 4 != 3 && keep(*i))
+ .map(|i| format!("d{i:02}"))
+ .collect()
+ }
+
+ async fn assert_phase(idx: &Index, n: usize, phase: &str) {
+ let bare = json!({ "match": { "body": "alpha" } });
+ let tag_x = |i: usize| !i.is_multiple_of(3);
+
+ let filter = json!({ "bool": {
+ "must": [ { "match": { "body": "alpha" } } ],
+ "filter": [ { "term": { "tag": "x" } } ],
+ }});
+ assert_scores_like_bare(idx, &bare, &filter, &alpha_docs(n, tag_x), phase).await;
+
+ let must_not = json!({ "bool": {
+ "must": [ { "match": { "body": "alpha" } } ],
+ "must_not": [ { "term": { "tag": "y" } } ],
+ }});
+ assert_scores_like_bare(idx, &bare, &must_not, &alpha_docs(n, tag_x), phase).await;
+
+ // A filter that reads `_id`: the stored-doc matcher needs it injected.
+ let ids_filter = json!({ "bool": {
+ "must": [ { "match": { "body": "alpha" } } ],
+ "filter": [ { "ids": { "values": ["d00", "d01", "d03"] } } ],
+ }});
+ let want: BTreeSet<String> = ["d00", "d01"].iter().map(|s| s.to_string()).collect();
+ assert_scores_like_bare(idx, &bare, &ids_filter, &want, phase).await;
+
+ // A boost on the must clause reaches the score the same way it does bare.
+ let boosted_bare = json!({ "match": { "body": { "query": "alpha", "boost": 3.0 } } });
+ let boosted = json!({ "bool": {
+ "must": [ { "match": { "body": { "query": "alpha", "boost": 3.0 } } } ],
+ "filter": [ { "term": { "tag": "x" } } ],
+ }});
+ assert_scores_like_bare(idx, &boosted_bare, &boosted, &alpha_docs(n, tag_x), phase).await;
+ }
+
+ #[tokio::test]
+ async fn filtered_match_scores_like_the_bare_match_before_and_after_flush() {
+ let dir = TempDir::new().unwrap();
+ let idx = seed("bfmem", &dir).await;
+ index_range(&idx, 0..24).await;
+
+ assert_phase(&idx, 24, "pre-flush").await;
+ idx.flush().await.unwrap();
+ assert_phase(&idx, 24, "post-flush").await;
+ }
+
+ #[tokio::test]
+ async fn filtered_match_scores_like_the_bare_match_on_a_mixed_index() {
+ let dir = TempDir::new().unwrap();
+ let idx = seed("bfmix", &dir).await;
+ index_range(&idx, 0..12).await;
+ idx.flush().await.unwrap();
+ index_range(&idx, 12..24).await;
+
+ assert_phase(&idx, 24, "mixed").await;
+ }
+
+ /// Shapes the memtable BM25 path cannot serve (`operator: and` needs the
+ /// per-token doc-scan) keep their existing route; the hit set is asserted
+ /// in both phases.
+ #[tokio::test]
+ async fn operator_and_must_keeps_its_hit_set() {
+ let dir = TempDir::new().unwrap();
+ let idx = seed("bfand", &dir).await;
+ index_range(&idx, 0..24).await;
+
+ let q = json!({ "bool": {
+ "must": [ { "match": { "body": { "query": "alpha beta", "operator": "and" } } } ],
+ "filter": [ { "term": { "tag": "x" } } ],
+ }});
+ let want: BTreeSet<String> = (0..24)
+ .filter(|i| i % 4 == 2 && i % 3 != 0)
+ .map(|i| format!("d{i:02}"))
+ .collect();
+ for phase in ["pre-flush", "post-flush"] {
+ let got: BTreeSet<String> = search(&idx, &q)
+ .await
+ .hits
+ .iter()
+ .map(|h| h.id.clone())
+ .collect();
+ assert_eq!(got, want, "{phase}: operator:and hit set");
+ if phase == "pre-flush" {
+ idx.flush().await.unwrap();
+ }
+ }
+ }
+
+ /// A per-query `analyzer` keeps the existing route too: the memtable BM25
+ /// path tokenizes with the index default and would lowercase `ALPHA`, while
+ /// `whitespace` keeps its case, so the hit set would change at flush.
+ #[tokio::test]
+ async fn match_with_an_analyzer_keeps_its_hit_set() {
+ let dir = TempDir::new().unwrap();
+ let idx = seed("bfanalyzer", &dir).await;
+ index_range(&idx, 0..24).await;
+
+ let q = json!({ "bool": {
+ "must": [ { "match": { "body": { "query": "ALPHA", "analyzer": "whitespace" } } } ],
+ "filter": [ { "term": { "tag": "x" } } ],
+ }});
+ for phase in ["pre-flush", "post-flush"] {
+ let got: BTreeSet<String> = search(&idx, &q)
+ .await
+ .hits
+ .iter()
+ .map(|h| h.id.clone())
+ .collect();
+ assert!(
+ got.is_empty(),
+ "{phase}: whitespace analyzer kept case, got {got:?}"
+ );
+ if phase == "pre-flush" {
+ idx.flush().await.unwrap();
+ }
+ }
+ }
++
++#[tokio::test]
++async fn selective_filter_keeps_low_ranked_live_hits_across_overwrites_and_flushes() {
++ let dir = TempDir::new().unwrap();
++ let idx = seed("bfselective", &dir).await;
++ for i in 0..128 {
++ idx.index_document(
++ Some(format!("strong-{i}")),
++ json!({"body":"alpha", "tag":"drop"}),
++ )
++ .await
++ .unwrap();
++ }
++ let weak = format!("alpha {}", "unrelated ".repeat(128));
++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"keep"}))
++ .await
++ .unwrap();
++ let query = json!({"bool": {
++ "must":[{"match":{"body":"alpha"}}],
++ "filter":[{"term":{"tag":"keep"}}]
++ }});
++ let request = parse_request(&json!({"query":query, "size":1})).unwrap();
++ for phase in ["memory", "mixed", "flushed"] {
++ let result = idx.search(&request).await.unwrap();
++ assert_eq!(
++ result.total.value, 1,
++ "{phase}: the filter must run before top-k truncation"
++ );
++ assert_eq!(result.hits[0].id, "selected", "{phase}");
++ idx.flush().await.unwrap();
++ if phase == "memory" {
++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"keep"}))
++ .await
++ .unwrap();
++ idx.index_document(
++ Some("deleted".into()),
++ json!({"body":"alpha", "tag":"keep"}),
++ )
++ .await
++ .unwrap();
++ assert!(idx.delete_document("deleted").await.unwrap());
++ }
++ }
++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"drop"}))
++ .await
++ .unwrap();
++ let result = idx.search(&request).await.unwrap();
++ assert_eq!(
++ result.total.value, 0,
++ "the prior matching segment version is a ghost"
++ );
++ assert!(result.hits.is_empty());
++}