AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Keep Boolean filters non-scoring for buffered text matches
xerj-org29f2ae6cf4a7
diff --cc CHANGELOG.mdindex d93c6a1e7,22cf1ee90..8bf9a8670--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,102 -7,20 +7,106 @@@ and this project adheres to [Semantic V## [Unreleased]+### Added++- Corpus Hub manifest `paths` entries may name regular files as well as+ directories (#2486). A file selection checks out that file without its+ siblings; symlinks and submodules are refused.++- MCP `xerj_status` reports node reachability, cluster health and visible+ indices before a search. It distinguishes a missing autoindex catalog from+ other indexed data and preserves failed or malformed probes as errors+ rather than reporting a successful empty node (#2515).++- Empty autoindex maps now explain that managed corpus slugs and autoindex+ dataset/index names belong to separate discovery surfaces. CLI diagnostics,+ MCP empty responses and the `xerj_map` description point to corpus discovery+ without claiming a corpus exists. Missing catalogs remain errors; auth+ refusals and successful dataset maps retain their behavior (#2412).++- Public Agent Skills Discovery v0.2.0 index and a documentation-routing skill+ at `/.well-known/agent-skills/`. The index is generated from the published+ skill's metadata and exact-byte SHA-256, with a CI drift check.+### Fixed-- A `filter` or `must_not` clause no longer changes `_score` before a flush- (#2403). `bool{must:[match], filter:[term]}` scored buffered documents with- an IDF-less `1 + ln(1 + tf)`, so on the issue's repro the top hit scored- 2.609438 and three hits tied at 1.6931472, while the bare `match` scored- 0.6979745, 0.60089254, 0.5022658 and 0.2530558; the ranking changed at- flush. The `must` match now takes the same BM25 path as the bare `match`- and the non-scoring clauses only remove hits. On an index with flushed and- buffered documents, the shared BM25 statistics also count the buffered- documents for this shape, so flushed hits score as they do under the bare- `match`. Covers one `must` match (without its own `analyzer`) plus any- `filter` or `must_not` clauses with no `should`; other `bool` shapes keep- their current route.++- Boolean text queries with one `must` match now keep `filter` and `must_not`++ clauses non-scoring before a flush and on mixed memory/segment indices.++ Their retained hits use the same BM25 scores as the bare match (#2403).+++- Reindex preflights every source alias member before creating or writing the+ destination. Aliases cannot hide a source/destination overlap; a missing+ source, invalid query or source flush failure leaves destination data intact.++- Corpus cloning validates existing checkouts and serializes writers (#2453).+ New checkouts are staged before publication. Ambiguous existing directories+ are preserved for recovery rather than deleted or reset automatically.++- MCP `xerj_search` advertises plain text in `query` and structured clauses+ in `query_dsl`. Legacy object-valued `query` and omitted-query browsing+ remain supported; conflicting fields fail before HTTP requests (#2549).+- Search payloads and batch retrieval default to 8,000 UAX #29 words when+ `max_tokens` is absent or null (#2491). This is not a byte or model-token+ cap. Explicit larger budgets remain available; errors and mandatory+ search staleness hints stay outside the payload budget.+- GitHub field-report submission uses the supported explicit-repository+ `gh repo fork --clone=false` invocation and adds the fork remote itself+ (#2596). The rollback test now verifies that the report was committed+ before a failed push, rather than passing after an earlier fork failure.+- Local admin-key discovery reports the source of the selected key and warns+ when its directory is outside the working tree (#2578). Packaged installs+ have their own provenance, and MCP guidance names `XERJ_AUTH`. Discovery+ priority and explicit credential overrides are unchanged.++- `xerj autoindex` fails a run when one dataset index holds no documents+ although the journal says records were written to it, even when the other+ datasets are live (#2407, defect 2). The check summed live documents over+ every dataset, so a populated `-json` index let a re-run print `ok: indexed+ and searchable` over an empty `-docs` index. The error names each empty+ index and its contributing files; record totals are reported only when+ those files target that dataset alone. Unknown sibling counts do not hide+ independently confirmed missing data.+- SQL source extraction no longer slices statement heads inside a UTF-8+ character, so unrecognized CJK statement/comment heads do not panic (#2547).+- Short Markdown files with YAML frontmatter retain their prose body even+ when bullets or `Key: value` lines also parse as YAML (#2447). Existing+ frontmatter-title behavior and structured CSV/JSON body detection remain.+- Empty files keep separate inventory identities instead of being reported+ as content duplicates of unrelated paths (#2449). Non-empty byte-identical+ files still deduplicate.+- A search with both `query` and a top-level `knn` returns `size` hits, not+ `k` (#2561). The page was widened to `k`, so `size: 5, k: 20` returned+ 20 hits. `k` still sets how many nearest neighbours join the candidate+ set, and `from`/`size` page through the union, as in Elasticsearch.+- Index deletion no longer waits for unrelated index statistics while+ resolving its selectors. Index listings and health checks also release+ their index-map locks before awaiting statistics, so a stalled schema+ update does not hold those locks against DELETE (part of #2405; extracted+ from #2489). Merge/flush cancellation and the process memory cap remain+ separate follow-ups.+- `xerj autoindex` compresses per-file phase-B stages with zstd level 1 and+ uses temporary files cleaned up by the operating system when the client+ exits (#2493, part of #2427). It finishes buffered compression before+ deleting previous records, so a stage finalization error preserves the old+ records and an ordinary resume can complete the replacement. Compression+ savings depend on the data; this does not impose a disk or server-memory cap.+- `autoindex` bulk back-pressure patience is now measured since the last+ progress, not since the first round. A window the server drains a few+ records at a time (per-item 429 `resource_exhausted` on an embedding+ deadline or admission refusal) used to abort with `bulk backend failed`+ once the rounds together passed the 600 s `THROTTLE_PATIENCE`, even though+ every round settled records. Each round that settles at least one item now+ restarts the patience clock and the backoff; rounds that settle nothing+ still count, so a server that answers 429 forever fails at the same limits+ as before.+- `xerj autoindex` titles a prose file that opens with YAML frontmatter from+ its `title:` key, or from the first line after the closing `---` when there+ is none (#2556). Every such file (Jekyll, Hugo, Obsidian) used to be titled+ `---`, its opening fence. A leading `---` block that is not parseable+ frontmatter (unclosed, not a YAML mapping, over 64 KiB) takes the first line+ that is not a bare `---`; the body, the frontmatter text in it and+ `start_line` are unchanged. An unchanged file is not re-extracted, so re-index+ with `xerj autoindex --fresh` to retitle documents already in an index.## [1.0.0-rc.100] - 2026-10-26diff --cc engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rsindex 000000000,31caa521d..9c03b259emode 000000,100644..100644--- a/engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rs+++ b/engine/crates/xerj-engine/tests/bool_filter_memtable_scoring.rs@@@ -1,0 -1,239 +1,292 @@@+ //! Issue #2403: `bool{must:[match], filter:[...]}` scored buffered documents+ //! with the IDF-less brute scorer (`score_query_against_doc`:+ //! `1 + ln(1 + tf)`), while the identical `must` clause on its own took the+ //! memtable BM25 arm. Before `_flush` the filtered query ranked a one-word+ //! document level with an eight-word one, and every score moved at flush.+ //!+ //! A `filter` or `must_not` clause is non-scoring (`BooleanClause.isScoring()`,+ //! BooleanClause.java:84-87), so the contract is the one #361 pinned for the+ //! segment path, now held in every phase: each hit scores exactly what the+ //! `must` clause scores alone, in the same phase, and the hit set is the+ //! `must` hits the filter keeps. `bool_filter_is_non_scoring.rs` covers the+ //! flushed shapes; this file covers the memtable and the mixed index.+ //!+ //! Elasticsearch is referenced for wire semantics only; no ES code is here.++ use std::collections::{BTreeSet, HashMap};++ use serde_json::{json, Value};+ use tempfile::TempDir;+ use xerj_common::config::Config;+ use xerj_common::types::{FieldConfig, FieldType, Schema};+ use xerj_engine::{Engine, Index};+ use xerj_query::executor::SearchResult;+ use xerj_query::parse_request;++ fn make_engine(dir: &TempDir) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ Engine::new(config).expect("engine::new")+ }++ fn req(q: &Value) -> xerj_query::ast::SearchRequest {+ parse_request(&json!({ "query": q, "size": 100 })).expect("parse_request")+ }++ async fn search(idx: &Index, q: &Value) -> SearchResult {+ idx.search(&req(q)).await.unwrap()+ }++ /// `alpha` appears in most bodies with varying frequency and body length, so+ /// BM25 and the brute scorer disagree on order; `tag` splits the corpus.+ fn doc(i: usize) -> Value {+ let body = match i % 4 {+ 0 => "alpha alpha alpha alpha".to_string(),+ 1 => "alpha".to_string(),+ 2 => "alpha beta gamma delta epsilon zeta eta theta".to_string(),+ _ => format!("beta gamma {i}"),+ };+ json!({ "body": body, "tag": if i.is_multiple_of(3) { "y" } else { "x" } })+ }++ async fn seed(name: &str, dir: &TempDir) -> std::sync::Arc<Index> {+ let engine = make_engine(dir);+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ schema+ .fields+ .push(FieldConfig::new("tag", FieldType::Keyword));+ engine.create_index(name, schema).unwrap();+ engine.get_index(name).unwrap()+ }++ async fn index_range(idx: &Index, range: std::ops::Range<usize>) {+ for i in range {+ idx.index_document(Some(format!("d{i:02}")), doc(i))+ .await+ .unwrap();+ }+ }++ /// Every hit of `filtered` scores what `bare` scores for the same `_id`,+ /// the order is `bare`'s order restricted to the kept set, the kept set is+ /// `want`, and `hits.total` counts exactly the kept set.+ async fn assert_scores_like_bare(+ idx: &Index,+ bare: &Value,+ filtered: &Value,+ want: &BTreeSet<String>,+ phase: &str,+ ) {+ let reference = search(idx, bare).await;+ let actual = search(idx, filtered).await;+ let got: BTreeSet<String> = actual.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(&got, want, "{phase}: hit set of {filtered}");+ assert_eq!(+ actual.total.value as usize,+ want.len(),+ "{phase}: hits.total of {filtered}"+ );+ let reference_scores: HashMap<&str, f32> = reference+ .hits+ .iter()+ .map(|h| (h.id.as_str(), h.score))+ .collect();+ for hit in &actual.hits {+ let expected = reference_scores[hit.id.as_str()];+ assert!(+ (hit.score - expected).abs() < 1e-4,+ "{phase}: {} scores {} under {filtered} but {expected} under the bare \+ must clause; a non-scoring clause changed _score (#2403)",+ hit.id,+ hit.score+ );+ }+ let order: Vec<&str> = reference+ .hits+ .iter()+ .map(|h| h.id.as_str())+ .filter(|id| got.contains(*id))+ .collect();+ let actual_order: Vec<&str> = actual.hits.iter().map(|h| h.id.as_str()).collect();+ assert_eq!(actual_order, order, "{phase}: order of {filtered} (#2403)");+ }++ /// The docs among `0..n` whose body holds `alpha` and that `keep` admits.+ fn alpha_docs(n: usize, keep: impl Fn(usize) -> bool) -> BTreeSet<String> {+ (0..n)+ .filter(|i| i % 4 != 3 && keep(*i))+ .map(|i| format!("d{i:02}"))+ .collect()+ }++ async fn assert_phase(idx: &Index, n: usize, phase: &str) {+ let bare = json!({ "match": { "body": "alpha" } });+ let tag_x = |i: usize| !i.is_multiple_of(3);++ let filter = json!({ "bool": {+ "must": [ { "match": { "body": "alpha" } } ],+ "filter": [ { "term": { "tag": "x" } } ],+ }});+ assert_scores_like_bare(idx, &bare, &filter, &alpha_docs(n, tag_x), phase).await;++ let must_not = json!({ "bool": {+ "must": [ { "match": { "body": "alpha" } } ],+ "must_not": [ { "term": { "tag": "y" } } ],+ }});+ assert_scores_like_bare(idx, &bare, &must_not, &alpha_docs(n, tag_x), phase).await;++ // A filter that reads `_id`: the stored-doc matcher needs it injected.+ let ids_filter = json!({ "bool": {+ "must": [ { "match": { "body": "alpha" } } ],+ "filter": [ { "ids": { "values": ["d00", "d01", "d03"] } } ],+ }});+ let want: BTreeSet<String> = ["d00", "d01"].iter().map(|s| s.to_string()).collect();+ assert_scores_like_bare(idx, &bare, &ids_filter, &want, phase).await;++ // A boost on the must clause reaches the score the same way it does bare.+ let boosted_bare = json!({ "match": { "body": { "query": "alpha", "boost": 3.0 } } });+ let boosted = json!({ "bool": {+ "must": [ { "match": { "body": { "query": "alpha", "boost": 3.0 } } } ],+ "filter": [ { "term": { "tag": "x" } } ],+ }});+ assert_scores_like_bare(idx, &boosted_bare, &boosted, &alpha_docs(n, tag_x), phase).await;+ }++ #[tokio::test]+ async fn filtered_match_scores_like_the_bare_match_before_and_after_flush() {+ let dir = TempDir::new().unwrap();+ let idx = seed("bfmem", &dir).await;+ index_range(&idx, 0..24).await;++ assert_phase(&idx, 24, "pre-flush").await;+ idx.flush().await.unwrap();+ assert_phase(&idx, 24, "post-flush").await;+ }++ #[tokio::test]+ async fn filtered_match_scores_like_the_bare_match_on_a_mixed_index() {+ let dir = TempDir::new().unwrap();+ let idx = seed("bfmix", &dir).await;+ index_range(&idx, 0..12).await;+ idx.flush().await.unwrap();+ index_range(&idx, 12..24).await;++ assert_phase(&idx, 24, "mixed").await;+ }++ /// Shapes the memtable BM25 path cannot serve (`operator: and` needs the+ /// per-token doc-scan) keep their existing route; the hit set is asserted+ /// in both phases.+ #[tokio::test]+ async fn operator_and_must_keeps_its_hit_set() {+ let dir = TempDir::new().unwrap();+ let idx = seed("bfand", &dir).await;+ index_range(&idx, 0..24).await;++ let q = json!({ "bool": {+ "must": [ { "match": { "body": { "query": "alpha beta", "operator": "and" } } } ],+ "filter": [ { "term": { "tag": "x" } } ],+ }});+ let want: BTreeSet<String> = (0..24)+ .filter(|i| i % 4 == 2 && i % 3 != 0)+ .map(|i| format!("d{i:02}"))+ .collect();+ for phase in ["pre-flush", "post-flush"] {+ let got: BTreeSet<String> = search(&idx, &q)+ .await+ .hits+ .iter()+ .map(|h| h.id.clone())+ .collect();+ assert_eq!(got, want, "{phase}: operator:and hit set");+ if phase == "pre-flush" {+ idx.flush().await.unwrap();+ }+ }+ }++ /// A per-query `analyzer` keeps the existing route too: the memtable BM25+ /// path tokenizes with the index default and would lowercase `ALPHA`, while+ /// `whitespace` keeps its case, so the hit set would change at flush.+ #[tokio::test]+ async fn match_with_an_analyzer_keeps_its_hit_set() {+ let dir = TempDir::new().unwrap();+ let idx = seed("bfanalyzer", &dir).await;+ index_range(&idx, 0..24).await;++ let q = json!({ "bool": {+ "must": [ { "match": { "body": { "query": "ALPHA", "analyzer": "whitespace" } } } ],+ "filter": [ { "term": { "tag": "x" } } ],+ }});+ for phase in ["pre-flush", "post-flush"] {+ let got: BTreeSet<String> = search(&idx, &q)+ .await+ .hits+ .iter()+ .map(|h| h.id.clone())+ .collect();+ assert!(+ got.is_empty(),+ "{phase}: whitespace analyzer kept case, got {got:?}"+ );+ if phase == "pre-flush" {+ idx.flush().await.unwrap();+ }+ }+ }++++#[tokio::test]++async fn selective_filter_keeps_low_ranked_live_hits_across_overwrites_and_flushes() {++ let dir = TempDir::new().unwrap();++ let idx = seed("bfselective", &dir).await;++ for i in 0..128 {++ idx.index_document(++ Some(format!("strong-{i}")),++ json!({"body":"alpha", "tag":"drop"}),++ )++ .await++ .unwrap();++ }++ let weak = format!("alpha {}", "unrelated ".repeat(128));++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"keep"}))++ .await++ .unwrap();++ let query = json!({"bool": {++ "must":[{"match":{"body":"alpha"}}],++ "filter":[{"term":{"tag":"keep"}}]++ }});++ let request = parse_request(&json!({"query":query, "size":1})).unwrap();++ for phase in ["memory", "mixed", "flushed"] {++ let result = idx.search(&request).await.unwrap();++ assert_eq!(++ result.total.value, 1,++ "{phase}: the filter must run before top-k truncation"++ );++ assert_eq!(result.hits[0].id, "selected", "{phase}");++ idx.flush().await.unwrap();++ if phase == "memory" {++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"keep"}))++ .await++ .unwrap();++ idx.index_document(++ Some("deleted".into()),++ json!({"body":"alpha", "tag":"keep"}),++ )++ .await++ .unwrap();++ assert!(idx.delete_document("deleted").await.unwrap());++ }++ }++ idx.index_document(Some("selected".into()), json!({"body":weak, "tag":"drop"}))++ .await++ .unwrap();++ let result = idx.search(&request).await.unwrap();++ assert_eq!(++ result.total.value, 0,++ "the prior matching segment version is a ghost"++ );++ assert!(result.hits.is_empty());++}