Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): `_count` stops answering a `term` on a text field from the analysed term dictionary (#362)

xerj-orgeaeab4f8eb56
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c2591c4ef..513e693f8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -204,6 +204,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the candidate vectors and removes a lock acquisition; both are small against
the encode/decode/similarity work the scan already does.
+- **`_count` no longer reports documents `_search` cannot return**
+ ([#362](https://github.com/xerj-org/xerj/issues/362)). `POST /{index}/_count`
+ is `_search` at `size: 0`, so the two are supposed to answer the same
+ question. On a `text` field they did not. A `text` mapping carries no
+ doc-values, so the term-count shortcut had no column to read and answered
+ from the segment's FTS term dictionary instead — analysed tokens, tokenised
+ and lowercased — while `_search` resolves a `term` on such a field against
+ the whole `_source` value. Two oracles, two questions, wrong in **both**
+ directions on a flushed segment:
+
+ | `term` query on a `text` field | old `_count` | `_search` |
+ |---|---|---|
+ | `{"term":{"title":"testsegmentreader.java"}}` | `1` | 0 hits |
+ | `{"term":{"title":"quick"}}`, value `"the quick brown fox"` | `1` | 0 hits |
+ | `{"term":{"title":"Doc7.java the quick brown fox"}}` — the byte-exact `_source` value | `0` | **1 hit** |
+
+ The shortcut now abandons when a segment has no doc-values column for the
+ field, and the ordinary delete-aware scan — the code that produces the hits —
+ answers. That shortcut is not `_count`-only: its result also authorises the
+ bounded stored scan at `size > 0`, so `hits.total` was affected too. On a
+ 200,000-document index, with a `term` on a `text` field matching 10,000
+ documents, `_search` at `size: 10` returned its 10 hits under
+ `"total": {"value": 130}`, and `_count` for the same query answered `0`. Both
+ now answer `10000`.
+
+ **This costs real time, and the bill is not only "the wrong spellings".**
+ Scan cost does not depend on how the term is spelled, so every `term` count on
+ a `text` field with no doc-values now pays a full scan — the byte-exact
+ spelling included. Its cost changes exactly as much as a misspelling's does,
+ and per the table above its *value* changes too, from an undercount to the
+ truth. Measured on this branch in the `ci-test` profile, one segment, 20
+ distinct query values per shape to defeat the request cache (median of 20,
+ same box, same corpus, only `index.rs` swapped):
+
+ | `_count` shape, 200,000 docs | before | after |
+ |---|---|---|
+ | `term` on `text`, no doc-values | 4.2 ms | 356 ms |
+ | same, byte-exact `_source` value | 4.1 ms (and answered `0`) | 356 ms (answers `1`) |
+ | `term` on `keyword`, `doc_values: false` | 21 µs | 45 µs |
+ | `term` on `keyword` with doc-values (control) | 347 µs | 216 µs |
+ | `term` on `long`, `doc_values: false` | 383 ms | 353 ms |
+ | `_search` at `size: 10`, `term` on `text` matching 10,000 docs | 9.4 ms (total `130`) | 225 ms (total `10000`) |
+
+ At 50,000 documents the first row is 2.3 ms → 81 ms. That is roughly 35× at
+ 50 k and 85× at 200 k, but the box carried load average 70–85 from unrelated
+ work throughout, so treat the multiple as order-of-magnitude; the number to
+ plan around is the absolute one, about a third of a second per `_count` at
+ 200,000 documents.
+
+ The middle rows are there to bound the blast radius, and they are the
+ reassuring ones. A `keyword` field with `doc_values: false` reaches the same
+ abandoned branch, but a `term` on a declared `keyword` field still projects
+ into the FTS tree, so the segment scan's own `term_doc_freq` shortcut answers
+ it: its **route** changed, its cost class did not, and it stays sub-linear
+ rather than becoming a scan. A `long` with `doc_values: false` is unchanged
+ because it was already scanning — a field with no FTS dictionary never reached
+ the removed fallback in the first place. And every count already served from a
+ doc-values column is untouched, which is what the control row shows.
+
+ **One shape's `_search` page changes, not just its total.** With the shortcut
+ gone the stored scan no longer stops after materialising `from + size`; it
+ walks every match and then picks the page. For an *unsorted* `term` on a
+ `text` field with many equally-scoring matches, that changes which documents
+ come back — measured over 2,000 identically-scoring documents at `size: 10`,
+ the ten ids differ before and after (and the total goes from `80` to `2000`).
+ Every score involved is identical, so neither page is more correct than the
+ other, but the change is observable. Queries with an explicit `sort` return
+ byte-identical pages and identical totals before and after.
+
+ **ES compatibility, stated rather than buried.** On the repaired shape this
+ moves `_count` *away* from the Elasticsearch answer while making it
+ self-consistent: ES answers `{"term":{"body":"quick"}}` with `1` against
+ `"the quick brown fox"`, XERJ used to answer `1` as well — but with zero
+ retrievable hits — and now answers `0`, which is what its own `_search`
+ returns. A count no page of hits can reproduce is a coincidence, not
+ compatibility. What `_search` *should* answer for a `term` on an analysed
+ field is the separate, hit-set-moving decision tracked in
+ [#397](https://github.com/xerj-org/xerj/issues/397); closing that is what
+ makes both properties true at once, and is also the only route back to a
+ sub-linear count for this shape.
+
+ Two things this change does **not** do. It does not touch the other half of
+ #362's thread — `case_insensitive` accepted and silently dropped on
+ `prefix`/`wildcard` — so the issue's headline symptom survives:
+ `{"term":{"title":"TestSegmentReader.java"}}` still answers `1` while
+ `{"prefix":{"title":"TestSeg"}}` answers `0`. And it is not the narrower fix
+ the issue thread proposed (dropping the lowercase retry the shortcut did on a
+ dictionary miss); that was tried and measured, and it leaves the first two
+ rows of the first table wrong while additionally making the byte-exact
+ `{"term":{"title":"TestSegmentReader.java"}}` count `0` where `_search`
+ returns `1`.
+
## [1.0.0-rc.17] - 2026-08-15
### Added
diff --git a/engine/crates/xerj-api/tests/count_agrees_with_search.rs b/engine/crates/xerj-api/tests/count_agrees_with_search.rs
new file mode 100644
index 000000000..22730cc99
--- /dev/null
+++ b/engine/crates/xerj-api/tests/count_agrees_with_search.rs
@@ -0,0 +1,198 @@
+//! Issue #362, from the outside: `_count` must never report documents that
+//! `_search` cannot return.
+//!
+//! `POST /{index}/_count` is implemented as `search({query, size: 0}).total`
+//! (`es_compat::count_docs`), so the two endpoints are supposed to answer the
+//! same question. On a `text` field they did not. A `text` mapping carries
+//! `doc_values: false`, so the engine's term-count shortcut had no column to
+//! read and fell back to the segment's FTS term dictionary — which holds
+//! ANALYZED tokens (tokenised and lowercased by the standard analyzer), while
+//! `_search` resolves a `term` on that field against the whole `_source`
+//! value. Both spellings below counted a document that no `_search`, at any
+//! `size`, could produce:
+//!
+//! ```text
+//! {"term":{"title":"testsegmentreader.java"}} count 1, hits 0
+//! {"term":{"title":"quick"}} count 1, hits 0
+//! ```
+//!
+//! The second row has no case component at all, which is why it is here: it is
+//! the case that proves the disagreement is not about casing.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+async fn app() -> (axum::Router, tempfile::TempDir) {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ (xerj_api::router::build_es_compat_router(state), dir)
+}
+
+async fn json_req(
+ app: &axum::Router,
+ method: &str,
+ path: &str,
+ body: Value,
+) -> (StatusCode, Value) {
+ let response = app
+ .clone()
+ .oneshot(
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+ .expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ (
+ status,
+ serde_json::from_slice(&bytes).unwrap_or(Value::Null),
+ )
+}
+
+/// Two documents, one `text` field, flushed to a segment — the shape the
+/// issue was filed against (a source-file corpus with CamelCase names).
+async fn seeded() -> (axum::Router, tempfile::TempDir) {
+ let (app, dir) = app().await;
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ "/files",
+ json!({
+ "mappings": { "properties": {
+ "title": { "type": "text" },
+ "path": { "type": "keyword" }
+ }}
+ }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "create index failed: {body}");
+
+ for (id, doc) in [
+ (
+ "java",
+ json!({"title": "TestSegmentReader.java", "path": "lucene/TestSegmentReader.java"}),
+ ),
+ (
+ "fox",
+ json!({"title": "the quick brown fox", "path": "docs/fox.txt"}),
+ ),
+ ] {
+ let (status, body) = json_req(&app, "PUT", &format!("/files/_doc/{id}"), doc).await;
+ assert!(status.is_success(), "index {id} failed: {status} {body}");
+ }
+ // The count shortcut under test only runs over flushed segments.
+ let (status, body) = json_req(&app, "POST", "/files/_flush", json!({})).await;
+ assert!(status.is_success(), "flush failed: {status} {body}");
+ (app, dir)
+}
+
+/// `_count` and `_search` must agree on both the total AND the number of
+/// documents actually retrievable.
+async fn assert_agrees(app: &axum::Router, query: Value, label: &str) {
+ let (status, count_body) =
+ json_req(app, "POST", "/files/_count", json!({ "query": query })).await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "{label}: _count failed {count_body}"
+ );
+ let count = count_body["count"].as_u64().expect("count is a number");
+
+ let (status, search_body) = json_req(
+ app,
+ "POST",
+ "/files/_search",
+ json!({ "query": query, "size": 50 }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "{label}: _search failed {search_body}"
+ );
+ let total = search_body["hits"]["total"]["value"]
+ .as_u64()
+ .expect("total is a number");
+ let hits = search_body["hits"]["hits"]
+ .as_array()
+ .expect("hits is an array")
+ .len() as u64;
+
+ assert_eq!(
+ count, total,
+ "{label}: _count said {count}, _search total said {total} for {query}"
+ );
+ assert_eq!(
+ count, hits,
+ "{label}: _count said {count} but _search returned {hits} document(s) for {query}"
+ );
+}
+
+#[tokio::test]
+async fn count_does_not_overreport_a_lowercased_term_on_a_text_field() {
+ let (app, _dir) = seeded().await;
+ assert_agrees(
+ &app,
+ json!({"term": {"title": "testsegmentreader.java"}}),
+ "lowercased spelling",
+ )
+ .await;
+}
+
+#[tokio::test]
+async fn count_does_not_overreport_an_analyzed_token_on_a_text_field() {
+ let (app, _dir) = seeded().await;
+ assert_agrees(&app, json!({"term": {"title": "quick"}}), "analyzed token").await;
+}
+
+/// The spelling that does match keeps working: no undercount in exchange.
+#[tokio::test]
+async fn count_still_agrees_on_the_exact_text_term() {
+ let (app, _dir) = seeded().await;
+ let query = json!({"term": {"title": "TestSegmentReader.java"}});
+ assert_agrees(&app, query.clone(), "exact spelling").await;
+ let (_, body) = json_req(&app, "POST", "/files/_count", json!({ "query": query })).await;
+ assert_eq!(
+ body["count"], 1,
+ "exact-spelling term must still count its document: {body}"
+ );
+}
+
+/// A `keyword` field has doc-values and never took the FTS fallback; it must
+/// keep its exact, case-sensitive count.
+#[tokio::test]
+async fn keyword_term_count_is_unchanged() {
+ let (app, _dir) = seeded().await;
+ let query = json!({"term": {"path": "lucene/TestSegmentReader.java"}});
+ assert_agrees(&app, query.clone(), "keyword field").await;
+ let (_, body) = json_req(&app, "POST", "/files/_count", json!({ "query": query })).await;
+ assert_eq!(body["count"], 1, "keyword term must count 1: {body}");
+
+ let wrong_case = json!({"term": {"path": "LUCENE/TestSegmentReader.java"}});
+ assert_agrees(&app, wrong_case.clone(), "keyword field, wrong case").await;
+ let (_, body) = json_req(
+ &app,
+ "POST",
+ "/files/_count",
+ json!({ "query": wrong_case }),
+ )
+ .await;
+ assert_eq!(
+ body["count"], 0,
+ "keyword term must stay case-sensitive: {body}"
+ );
+}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index fcc9bf55b..80311f9b4 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -19399,14 +19399,15 @@ impl Index {
///
/// Supported shapes:
/// * `MatchAll` — sum of segment `doc_count` + memtable doc count.
- /// * `Term { field, value }` — for a keyword field:
+ /// * `Term { field, value }` — for a field with doc-values:
/// - Counts matches in the memtable via `doc_values_keyword_column`.
- /// - For each on-disk segment, calls `FtsIndexReader::term_doc_freq`
- /// on the raw value; if the segment analyzer lowercased terms,
- /// falls back to the lowercased form.
- /// - If the segment has no FTS data for the field at all, the
- /// shortcut is abandoned (return `None`) because the segment
- /// would otherwise undercount.
+ /// - For each on-disk segment, counts through that segment's
+ /// doc-values column — the same values `_search` compares against.
+ /// - If a segment has no column for the field, the shortcut is
+ /// abandoned (return `None`) and the scan answers. It used to
+ /// consult the FTS term dictionary instead, whose ANALYZED terms
+ /// are not what `_search` matches a `term` against, so the count
+ /// could exceed the retrievable hit set (#362).
///
/// Exact match count for a `bool` that carries `should` and/or `must_not`
/// clauses — the shapes the fused must/filter-intersection arm bails on.
@@ -20265,15 +20266,14 @@ impl Index {
// 2. Keyword column → `doc_freq(term)` is an O(n) ord scan.
// 3. Numeric column → `range_count(v, v, true, true)` bisects
// the sorted index in O(log n).
- // 4. Fall back to FTS `term_doc_freq` if both columns are
- // missing for this segment.
- // 5. If FTS is missing too, abandon — we can't answer safely.
+ // 4. Neither column present for this segment → abandon (#362).
+ // The FTS term dictionary used to answer here and it is the
+ // wrong oracle; see the comment on that branch below.
let segments_dir = self.data_dir.join("segments");
let raw = match value {
Value::String(s) => s.as_str().to_string(),
other => other.to_string(),
};
- let lowered = raw.to_ascii_lowercase();
// Same bool coercion as the memtable side above — `value.as_f64()`
// alone returns `None` for a JSON boolean, which used to make
// `served_by_dv` false for every segment whose on-disk column for
@@ -20316,25 +20316,63 @@ impl Index {
continue;
}
- // Fallback: FTS term dictionary. This is the slow path that
- // parses meta.json; we only take it when the field isn't in
- // the segment's doc-values (e.g. text fields with positions).
- let reader = match FtsIndexReader::open(&segments_dir, &meta.id, &[field]) {
- Ok(r) => r,
- Err(_) => return None, // FST missing — abandon
- };
- reader.field_stats(field)?;
- let df = reader
- .term_doc_freq(field, &raw)
- .or_else(|| {
- if raw == lowered {
- None
- } else {
- reader.term_doc_freq(field, &lowered)
- }
- })
- .unwrap_or(0);
- seg_matches = seg_matches.saturating_add(df as u64);
+ // No doc-values column for this field in this segment: ABANDON
+ // (#362). This used to fall back to the segment's FTS term
+ // dictionary, which answers a DIFFERENT question than the one
+ // `_search` answers, so `_count` published totals no page of hits
+ // could reproduce:
+ //
+ // term title=testsegmentreader.java _count 1, _search total 0
+ // term title=quick, "the quick brown fox"
+ // _count 1, _search total 0
+ //
+ // A field only reaches here because it has no column, which on
+ // this engine means an analyzed `text` field (`es_compat`'s
+ // `doc_values_default`) or an explicit `doc_values: false`. Its
+ // term dictionary holds ANALYZED tokens — `build_fts_field_configs`
+ // gives `Text` the standard analyzer, so the entries are
+ // tokenized and lowercased — while `query_node_to_fts_with_
+ // keyword_fields` declines the FTS projection for a bare `Term`
+ // and `_search` resolves it through `doc_matches_query`, comparing
+ // the WHOLE `_source` value. A dictionary hit therefore says
+ // nothing about whether the scan will produce a document.
+ //
+ // The lowercase re-spelling this replaces (`term_doc_freq(raw)`
+ // `.or_else(term_doc_freq(lowered))`) was scaffolding over that:
+ // it retried with a term the caller never wrote, which is what
+ // made `term` look case-normalising next to `prefix`/`wildcard`
+ // and is what #362 was filed as. Deleting only the retry does
+ // not fix this — the `quick` row above carries no casing at all,
+ // its dictionary hit is genuine, and the byte-exact spelling
+ // would start counting 0 where `_search` still answers 1. The
+ // oracle is wrong, not its spelling.
+ //
+ // Lucene's counts cannot drift this way: the sub-linear answer is
+ // a method on the same `Weight` that builds the scorer, and its
+ // default is "I cannot answer" — `Weight::count`
+ // (lucene/core/src/java/org/apache/lucene/search/Weight.java:198)
+ // returns -1, documented as the count not being computable in
+ // sub-linear time, so `IndexSearcher::count`
+ // (lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:495)
+ // collects for real. The specialisation that does answer,
+ // `TermQuery::TermWeight::count`
+ // (lucene/core/src/java/org/apache/lucene/search/TermQuery.java:260),
+ // reads `termsEnum.docFreq()` off the query's OWN term state and
+ // hands straight back to `super.count` once deletions make that
+ // stop being the live answer. Returning `None` here is that
+ // hand-back: the caller's delete-aware scan — the code that
+ // materialises the hits — answers instead.
+ //
+ // Cost: a `term` count on a text field is a scan again. That is
+ // the price of the number being true, and it is paid only by the
+ // shape that was wrong.
+ //
+ // (What `_search` SHOULD do for a `term` on an analyzed field is
+ // a separate question — ES matches the analyzed dictionary, so
+ // `quick` would find "the quick brown fox" there. Changing that
+ // moves the hit set and needs its own issue; this change only
+ // makes `_count` agree with whatever `_search` does today.)
+ return None;
}
Some(mem_matches + seg_matches)
diff --git a/engine/crates/xerj-engine/tests/count_matches_search_on_text_term.rs b/engine/crates/xerj-engine/tests/count_matches_search_on_text_term.rs
new file mode 100644
index 000000000..259989ac6
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/count_matches_search_on_text_term.rs
@@ -0,0 +1,176 @@
+//! Regression tests for issue #362: `_count` must answer the same question
+//! `_search` answers.
+//!
+//! `POST /{index}/_count` is `search({query, size: 0}).total`, so a `size: 0`
+//! total that disagrees with the `size > 0` hit set is the HTTP-visible bug:
+//! a count describing documents nobody can retrieve.
+//!
+//! Pre-fix, `try_shortcut_count`'s Term arm fell back to the segment FTS term
+//! dictionary whenever the field had no doc-values column — which is exactly
+//! what an analyzed `text` field is. The dictionary holds ANALYZED tokens
+//! (standard analyzer: tokenised, lowercased), while `_search` resolves a
+//! `term` on such a field against the whole `_source` value. Two documents,
+//! two spellings, both counted and neither retrievable.
+
+use serde_json::{json, Value};
+use tempfile::TempDir;
+use xerj_common::config::Config;
+use xerj_common::types::{FieldConfig, FieldType, Schema};
+use xerj_engine::{Engine, Index};
+use xerj_query::parse_request;
+
+fn make_engine(dir: &TempDir) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ Engine::new(config).expect("engine::new")
+}
+
+/// What `POST /_count` runs: `size: 0`, total only (`es_compat::count_docs`).
+async fn count_total(idx: &Index, q: &Value) -> u64 {
+ let req = parse_request(&json!({ "query": q, "size": 0, "from": 0 })).expect("parse_request");
+ idx.search(&req).await.expect("count search").total.value
+}
+
+/// What `POST /_search` runs: a real page of hits.
+async fn search_hits(idx: &Index, q: &Value) -> (u64, usize) {
+ let req = parse_request(&json!({ "query": q, "size": 50 })).expect("parse_request");
+ let r = idx.search(&req).await.expect("search");
+ (r.total.value, r.hits.len())
+}
+
+async fn seed() -> (TempDir, std::sync::Arc<Index>) {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let mut schema = Schema::empty();
+ // `"type": "text"` from an ES mapping lands here with `doc_values: false`
+ // (`es_compat::es_properties_to_fields`), which is precisely what sends a
+ // term count for this field into the FTS term-dictionary fallback.
+ let mut title = FieldConfig::new("title", FieldType::Text);
+ title.options.doc_values = false;
+ schema.fields.push(title);
+ schema
+ .fields
+ .push(FieldConfig::new("path", FieldType::Keyword));
+ // A keyword field with `doc_values: false` reaches the same
+ // no-column branch as `title`, so the abandon must not cost it its
+ // exact count — it is the shape that would turn an overcount into an
+ // undercount if the branch were narrowed the other way.
+ let mut code = FieldConfig::new("code", FieldType::Keyword);
+ code.options.doc_values = false;
+ schema.fields.push(code);
+ engine.create_index("count-parity", schema).unwrap();
+ let idx = engine.get_index("count-parity").unwrap();
+
+ idx.index_document(
+ Some("java".into()),
+ json!({
+ "title": "TestSegmentReader.java",
+ "path": "lucene/TestSegmentReader.java",
+ "code": "AB-1234"
+ }),
+ )
+ .await
+ .unwrap();
+ idx.index_document(
+ Some("fox".into()),
+ json!({"title": "the quick brown fox", "path": "docs/fox.txt", "code": "ab-1234"}),
+ )
+ .await
+ .unwrap();
+ // The count shortcut only runs against flushed segments; the memtable
+ // arm is a different code path and already agrees with the scan.
+ idx.flush().await.unwrap();
+ (dir, idx)
+}
+
+/// The `_count` / `_search` disagreement, one query per row.
+async fn assert_count_agrees_with_search(idx: &Index, q: Value, label: &str) {
+ let count = count_total(idx, &q).await;
+ let (total, hits) = search_hits(idx, &q).await;
+ assert_eq!(
+ count, total,
+ "{label}: _count total {count} != _search total {total} for {q}"
+ );
+ assert_eq!(
+ count, hits as u64,
+ "{label}: _count says {count} but _search can materialise only {hits} document(s) for {q}"
+ );
+}
+
+/// Issue #362's own repro: a term whose spelling differs from `_source`
+/// only in case. The dictionary retry made `_count` find it; `_search`
+/// compares against the raw `_source` value and cannot.
+#[tokio::test]
+async fn count_does_not_overreport_lowercased_term_on_text_field() {
+ let (_dir, idx) = seed().await;
+ assert_count_agrees_with_search(
+ &idx,
+ json!({"term": {"title": "testsegmentreader.java"}}),
+ "lowercased spelling",
+ )
+ .await;
+}
+
+/// The half with no case component at all: an ANALYZED token of a
+/// multi-word text value. The dictionary hit here is genuine, which is why
+/// deleting the lowercase retry alone would not have fixed it.
+#[tokio::test]
+async fn count_does_not_overreport_analyzed_token_on_text_field() {
+ let (_dir, idx) = seed().await;
+ assert_count_agrees_with_search(
+ &idx,
+ json!({"term": {"title": "quick"}}),
+ "analyzed token of a multi-word value",
+ )
+ .await;
+}
+
+/// The exact spelling still counts, and still returns its document — the
+/// fix must not trade the overcount for an undercount.
+#[tokio::test]
+async fn count_still_finds_the_exact_text_term() {
+ let (_dir, idx) = seed().await;
+ let q = json!({"term": {"title": "TestSegmentReader.java"}});
+ assert_count_agrees_with_search(&idx, q.clone(), "exact spelling").await;
+ assert_eq!(
+ count_total(&idx, &q).await,
+ 1,
+ "exact-spelling term must still count its document"
+ );
+}
+
+/// A `keyword` field keeps its exact, byte-comparing count: that field has
+/// a doc-values column and never took the FTS fallback.
+#[tokio::test]
+async fn keyword_term_count_is_unchanged() {
+ let (_dir, idx) = seed().await;
+ let q = json!({"term": {"path": "lucene/TestSegmentReader.java"}});
+ assert_count_agrees_with_search(&idx, q.clone(), "keyword field").await;
+ assert_eq!(count_total(&idx, &q).await, 1, "keyword term must count 1");
+ assert_eq!(
+ count_total(
+ &idx,
+ &json!({"term": {"path": "LUCENE/TestSegmentReader.java"}})
+ )
+ .await,
+ 0,
+ "keyword term must stay case-sensitive"
+ );
+}
+
+/// The abandon is not a licence to undercount: a `keyword` field with
+/// `doc_values: false` takes the very same no-column branch and must still
+/// count exactly one document per spelling, case-sensitively.
+#[tokio::test]
+async fn keyword_without_doc_values_still_counts_exactly() {
+ let (_dir, idx) = seed().await;
+ for (spelling, expected) in [("AB-1234", 1), ("ab-1234", 1), ("zz-9", 0)] {
+ let q = json!({"term": {"code": spelling}});
+ assert_count_agrees_with_search(&idx, q.clone(), "keyword, doc_values:false").await;
+ assert_eq!(
+ count_total(&idx, &q).await,
+ expected,
+ "keyword without doc-values: wrong count for {spelling}"
+ );
+ }
+}