AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(es-compat): report analysis as in force in _settings, run real pipelines in _analyze (#1092)
Vincenzo Lombardo211ea90071a0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d0db6c75e..19dbac936 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **An index's analysis configuration could not be verified over the wire**+ (issue [#1092](https://github.com/xerj-org/xerj/issues/1092)) — search+ honoured a declared `analysis.analyzer.default`, but nothing an agent can+ read said so. `GET /{index}/_settings` and `GET /{index}` replayed an+ in-memory copy of the create body: after a restart it was empty, so the+ analysis block (and every other create-time setting) vanished, and a+ top-level `analysis` beside an `index` block was never echoed at all. Both+ now report `index.analysis` as the block the index's analyzer registry was+ actually built from — none for a pre-#204 index whose declared block is+ not honoured — and fall back to the persisted `settings.json` when the+ in-memory copy is gone. `POST /{index}/_analyze` was an inline lowercase+ splitter that ignored the index; it now runs the index's real pipelines:+ no `analyzer` → the index's default, `field` → the analyzer that field is+ indexed with, a named analyzer from the index's own registry. Ad-hoc+ `tokenizer` / `filter` / `char_filter` chains (names or inline+ definitions) are built by the same registry code index creation uses, with+ `keyword` as the tokenizer when only filters are given. Unknown analyzers,+ tokenizers and filters, `field` on the global endpoint and `normalizer`+ are 400 `illegal_argument_exception`. A missing index is 404. Checked+ side by side against OpenSearch 3.9.0: status codes, error types, tokens,+ offsets and positions match on all 24 probed requests. Only the wording of+ the reason differs on 3 of the 400s.+## [1.0.0-rc.84] - 2026-10-06The tell-the-truth window. Every signal an operator reads — clusterdiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 39601125f..4bc537dce 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -2042,15 +2042,10 @@ async fn get_index_inner(};// Settings: replay what was written (normalized to strings), merged- // with engine defaults. Creation timestamps are synthesized because- // we don't persist them yet (TODO).- let stored_settings = state- .engine- .index_settings- .get(name)- .map(|v| v.clone())- .unwrap_or(Value::Null);- let settings_inner = merge_settings_defaults(&stored_settings, name, human);+ // with engine defaults; `analysis` is the block in force (#1092).+ // Creation timestamps are synthesized because we don't persist them+ // yet (TODO).+ let settings_inner = index_settings_response(&state, name, idx.as_ref(), human).await;let mut index_obj = serde_json::Map::new();if want_aliases {@@ -2068,6 +2063,52 @@ async fn get_index_inner(Json(Value::Object(body)).into_response()}+/// The `settings` object `GET /{index}` and `GET /{index}/_settings` serve.+///+/// Two fixes over replaying the display map alone (#1092):+///+/// * The display map (`engine.index_settings`) lives in memory only, so after a+/// restart it is empty and every create-time setting vanished from the+/// response. A live index with no display entry falls back to its persisted+/// `settings.json` copy — what the index was created with.+/// * `analysis` is reported from the engine, not echoed: whatever spelling the+/// caller used (`settings.analysis` beside an `index` block was dropped+/// outright), the response carries `index.analysis` = the block the index's+/// analyzer registry was actually built from, and nothing for a pre-#204+/// index whose declared block is not honoured. An agent verifying "is my+/// stemmer on?" over the wire gets the answer search would give.+///+/// A failed index (`idx == None`) keeps the display-map-only behaviour.+async fn index_settings_response(+ state: &AppState,+ name: &str,+ idx: Option<&std::sync::Arc<xerj_engine::Index>>,+ human: bool,+) -> Value {+ let mut stored = state+ .engine+ .index_settings+ .get(name)+ .map(|v| v.clone())+ .unwrap_or(Value::Null);+ if stored.is_null() {+ if let Some(i) = idx {+ stored = i.get_settings().await;+ }+ }+ let mut settings = merge_settings_defaults(&stored, name, human);+ if let Some(i) = idx {+ let in_force = i.analysis_in_force().await;+ if let Some(inner) = settings.get_mut("index").and_then(Value::as_object_mut) {+ inner.retain(|k, _| k != "analysis" && !k.starts_with("analysis."));+ if let Some(a) = in_force {+ inner.insert("analysis".to_string(), a);+ }+ }+ }+ settings+}+/// Normalize a raw user-provided settings blob to the ES response shape.////// ES always wraps user settings under `{ "index": { ... } }` and coerces@@ -2767,13 +2808,8 @@ pub async fn get_settings(}let mut out = serde_json::Map::new();for name in &targets {- let stored = state- .engine- .index_settings- .get(name)- .map(|v| v.clone())- .unwrap_or(Value::Null);- let settings = merge_settings_defaults(&stored, name, false);+ let idx = state.engine.get_index(name).ok();+ let settings = index_settings_response(&state, name, idx.as_ref(), false).await;out.insert(name.clone(), json!({ "settings": settings }));}Json(Value::Object(out)).into_response()@@ -21120,8 +21156,26 @@ pub async fn cat_nodes(#[derive(Debug, Deserialize)]pub struct EsAnalyzeBody {pub text: Option<Value>,- #[serde(default = "default_analyzer")]- pub analyzer: String,+ /// Absent = the index's own text analyzer on `/{index}/_analyze` (its+ /// declared `default`, #1092), `standard` on the global endpoint.+ #[serde(default)]+ pub analyzer: Option<String>,+ /// Analyse as this field of the index is indexed (#1092).+ #[serde(default)]+ pub field: Option<String>,+ /// Ad-hoc chain parts (`tokenizer`, `filter`, `char_filter`): each a+ /// name or an inline definition, built into a one-off `custom` analyzer+ /// by the same registry code index creation uses (#1092). `normalizer`+ /// is refused with a 400 rather than silently answered with another+ /// analyzer's tokens.+ #[serde(default)]+ pub tokenizer: Option<Value>,+ #[serde(default)]+ pub filter: Option<Value>,+ #[serde(default)]+ pub char_filter: Option<Value>,+ #[serde(default)]+ pub normalizer: Option<Value>,/// When true, returns step-by-step output of each pipeline stage.////// Shows: tokenizer output → lowercase → stopwords → stemmer.@@ -21130,8 +21184,120 @@ pub struct EsAnalyzeBody {pub explain: bool,}-fn default_analyzer() -> String {- "standard".to_string()+fn analyze_illegal_argument(reason: String) -> Response {+ (+ StatusCode::BAD_REQUEST,+ Json(json!({+ "error": {+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],+ "type": "illegal_argument_exception",+ "reason": reason,+ },+ "status": 400,+ })),+ )+ .into_response()+}++/// Name of the one-off analyzer an ad-hoc `_analyze` chain is built as.+const ADHOC_ANALYZER: &str = "_analyze_adhoc";++/// Turn an ad-hoc `_analyze` chain into an `analysis` settings block declaring+/// one `custom` analyzer, so it is built — and validated — by exactly the+/// registry code an index's own analysis goes through (#1092).+///+/// Names refer to built-ins, or on an index-scoped call to the components the+/// index declares (`index_analysis`, the block in force); inline definitions+/// get reserved `_analyze_*` names. With filters and no tokenizer the chain+/// uses `keyword`, as Elasticsearch and OpenSearch do.+fn adhoc_analysis_settings(body: &EsAnalyzeBody, index_analysis: Option<&Value>) -> Value {+ let mut analysis = serde_json::Map::new();+ for kind in ["tokenizer", "filter", "char_filter"] {+ let defs = index_analysis+ .and_then(|a| a.get(kind))+ .cloned()+ .unwrap_or_else(|| json!({}));+ analysis.insert(kind.to_string(), defs);+ }+ let mut define = |kind: &str, name: String, def: &Value| -> String {+ if let Some(m) = analysis.get_mut(kind).and_then(Value::as_object_mut) {+ m.insert(name.clone(), def.clone());+ }+ name+ };+ let tokenizer = match &body.tokenizer {+ Some(Value::String(s)) => s.clone(),+ Some(def) => define("tokenizer", "_analyze_tokenizer".to_string(), def),+ None => "keyword".to_string(),+ };+ let mut chain = |kind: &str, parts: &Option<Value>| -> Vec<String> {+ let items = match parts {+ Some(Value::Array(a)) => a.clone(),+ Some(single) => vec![single.clone()],+ None => Vec::new(),+ };+ items+ .iter()+ .enumerate()+ .map(|(i, part)| match part {+ Value::String(s) => s.clone(),+ def => define(kind, format!("_analyze_{kind}_{i}"), def),+ })+ .collect()+ };+ let filter = chain("filter", &body.filter);+ let char_filter = chain("char_filter", &body.char_filter);+ analysis.insert(+ "analyzer".to_string(),+ json!({ ADHOC_ANALYZER: {+ "type": "custom",+ "tokenizer": tokenizer,+ "filter": filter,+ "char_filter": char_filter,+ }}),+ );+ json!({ "analysis": Value::Object(analysis) })+}++/// Build and run an ad-hoc chain. `Err` = the registry cannot honour it, with+/// the same reasons index creation would give.+fn run_adhoc_analyze(+ body: &EsAnalyzeBody,+ index_analysis: Option<&Value>,+ input: &str,+) -> Result<Vec<xerj_engine::Token>, String> {+ let settings = adhoc_analysis_settings(body, index_analysis);+ let problems = xerj_engine::AnalyzerRegistry::unsupported_analysis(&settings);+ if !problems.is_empty() {+ return Err(format!(+ "xerj cannot honour this analysis chain — {}",+ problems.join("; ")+ ));+ }+ let mut registry = xerj_engine::AnalyzerRegistry::with_defaults();+ registry.apply_settings(&settings);+ let pipeline = registry+ .get_analyzer(ADHOC_ANALYZER)+ .ok_or_else(|| "xerj could not build this analysis chain".to_string())?;+ Ok(pipeline.analyze(input))+}++/// ES `_analyze` token objects from real pipeline output. `type` stays the+/// `word` this endpoint has always reported; offsets and positions are the+/// pipeline's own.+fn analyze_tokens_json(tokens: &[xerj_engine::Token]) -> Vec<Value> {+ tokens+ .iter()+ .map(|t| {+ json!({+ "token": t.text,+ "start_offset": t.start_offset,+ "end_offset": t.end_offset,+ "type": "word",+ "position": t.position,+ })+ })+ .collect()}/// Global `POST /_analyze` (no index prefix).@@ -21143,8 +21309,8 @@ pub async fn analyze_text_global(}pub async fn analyze_text(- State(_state): State<AppState>,- Path(_index): Path<String>,+ State(state): State<AppState>,+ Path(index): Path<String>,Json(body): Json<EsAnalyzeBody>,) -> impl IntoResponse {let input = match &body.text {@@ -21157,19 +21323,97 @@ pub async fn analyze_text(_ => String::new(),};- if body.explain {- // Step-by-step pipeline explanation.- //- // Shows each filter stage in order so developers can debug why a term- // was dropped, modified, or expanded. Supported for the "standard"- // analyzer (the most common case). Other analyzers fall back to showing- // tokenizer + final output only.- let explanation = analyze_explain(&input, &body.analyzer);- return Json(explanation).into_response();+ if body.normalizer.is_some() {+ return analyze_illegal_argument(+ "_analyze does not support [normalizer] on xerj (it would otherwise be answered \+ with another analyzer's tokens)"+ .to_string(),+ );+ }+ let adhoc = body.tokenizer.is_some() || body.filter.is_some() || body.char_filter.is_some();+ if adhoc {+ if body.analyzer.is_some() || body.field.is_some() {+ return analyze_illegal_argument(+ "cannot define an ad-hoc chain (tokenizer/filter/char_filter) together with \+ [analyzer] or [field]"+ .to_string(),+ );+ }+ if body.explain {+ return analyze_illegal_argument(+ "_analyze explain is not supported for an ad-hoc chain on xerj".to_string(),+ );+ }+ // Index-scoped: the chain may name the index's own components.+ let index_analysis = if index == "_none" {+ None+ } else {+ let targets = resolve_index_selector(&state, &index).await;+ let Some(idx) = targets+ .first()+ .and_then(|name| state.engine.get_index(name).ok())+ else {+ return ApiError::new(xerj_common::XerjError::index_not_found(&index))+ .into_response();+ };+ idx.analysis_in_force().await+ };+ return match run_adhoc_analyze(&body, index_analysis.as_ref(), &input) {+ Ok(tokens) => Json(json!({ "tokens": analyze_tokens_json(&tokens) })).into_response(),+ Err(reason) => analyze_illegal_argument(reason),+ };+ }++ if index == "_none" {+ // Global endpoint: built-in analyzers only, no index to resolve a+ // field or a custom analyzer against.+ if let Some(f) = &body.field {+ return analyze_illegal_argument(format!(+ "analysing field [{f}] requires an index: use POST /{{index}}/_analyze"+ ));+ }+ let name = body.analyzer.as_deref().unwrap_or("standard");+ if body.explain {+ // Step-by-step pipeline explanation (unchanged): stage-by-stage+ // for "standard"/"english", final tokens otherwise.+ return Json(analyze_explain(&input, name)).into_response();+ }+ let Some(pipeline) = xerj_engine::analyzer_registry().get_analyzer(name) else {+ return analyze_illegal_argument(format!("failed to find global analyzer [{name}]"));+ };+ let tokens = analyze_tokens_json(&pipeline.analyze(&input));+ return Json(json!({ "tokens": tokens })).into_response();}- let tokens = tokenize_for_analyze(&input, &body.analyzer);- Json(json!({ "tokens": tokens })).into_response()+ // Index-scoped (#1092): the index's OWN registry and field resolution —+ // the pipeline a search on this index actually runs, so a declared+ // `analysis.analyzer.default` stemmer shows up here.+ let targets = resolve_index_selector(&state, &index).await;+ let Some(idx) = targets+ .first()+ .and_then(|name| state.engine.get_index(name).ok())+ else {+ return ApiError::new(xerj_common::XerjError::index_not_found(&index)).into_response();+ };+ match idx+ .analyze(body.analyzer.as_deref(), body.field.as_deref(), &input)+ .await+ {+ Ok((name, tokens)) => {+ let tokens = analyze_tokens_json(&tokens);+ if body.explain {+ return Json(json!({+ "detail": {+ "custom_analyzer": false,+ "analyzer": { "name": name, "tokens": tokens },+ }+ }))+ .into_response();+ }+ Json(json!({ "tokens": tokens })).into_response()+ }+ Err(reason) => analyze_illegal_argument(reason),+ }}/// Produce step-by-step analysis explanation.diff --git a/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rs b/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rsnew file mode 100644index 000000000..724f92a49--- /dev/null+++ b/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rs@@ -0,0 +1,286 @@+//! Issue #1092: an index's analysis configuration must be observable over the+//! wire exactly as search applies it.+//!+//! Before the fix:+//! * `GET /{index}/_settings` replayed an in-memory display copy of the create+//! body. After a restart that copy is empty, so a declared+//! `analysis.analyzer.default` stemmer vanished from the response while+//! search kept stemming — and a top-level `analysis` beside an `index` block+//! was never echoed at all.+//! * `POST /{index}/_analyze` ignored the index entirely: an inline lowercase+//! splitter answered every request, so a stemmer index showed `bagels`+//! where search indexes `bagel`.+//!+//! An agent verifying "is my stemmer on?" through the ES wire concluded it was+//! off. Elasticsearch and OpenSearch are referenced for wire semantics only+//! (public API docs: `_analyze` without `analyzer`/`field` uses the index's+//! default analyzer; `field` uses that field's analyzer); no code is reproduced.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++fn app_over(dir: &std::path::Path) -> axum::Router {+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ xerj_api::router::build_es_compat_router(state)+}++async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {+ let response = app.clone().oneshot(req).await.expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ let body = if bytes.is_empty() {+ Value::Null+ } else {+ serde_json::from_slice(&bytes).unwrap_or(Value::Null)+ };+ (status, body)+}++fn with_body(method: &str, path: &str, body: Value) -> Request<Body> {+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request")+}++fn get(path: &str) -> Request<Body> {+ Request::get(path).body(Body::empty()).expect("request")+}++async fn analyze(app: &axum::Router, path: &str, body: Value) -> (StatusCode, Vec<String>, Value) {+ let (st, resp) = send(app, with_body("POST", path, body)).await;+ let tokens = resp["tokens"]+ .as_array()+ .map(|a| {+ a.iter()+ .map(|t| t["token"].as_str().unwrap_or_default().to_string())+ .collect()+ })+ .unwrap_or_default();+ (st, tokens, resp)+}++const STEM: &str = "/stem";++async fn create_stem_index(app: &axum::Router) {+ let (st, body) = send(+ app,+ with_body(+ "PUT",+ STEM,+ json!({+ "settings": { "analysis": { "analyzer": { "default": { "type": "stemmer" } } } },+ "mappings": { "properties": {+ "t": { "type": "text" }, "tag": { "type": "keyword" }+ }}+ }),+ ),+ )+ .await;+ assert_eq!(st, StatusCode::OK, "create: {body}");+}++#[tokio::test]+async fn settings_report_the_analysis_in_force_across_a_restart() {+ let dir = tempfile::tempdir().expect("tempdir");+ let want = json!({ "analyzer": { "default": { "type": "stemmer" } } });+ {+ let app = app_over(dir.path());+ create_stem_index(&app).await;+ let (st, body) = send(&app, get("/stem/_settings")).await;+ assert_eq!(st, StatusCode::OK);+ assert_eq!(+ body.pointer("/stem/settings/index/analysis"),+ Some(&want),+ "{body}"+ );+ }+ // Restart: only what reached disk survives.+ let app = app_over(dir.path());+ let (_, body) = send(&app, get("/stem/_settings")).await;+ assert_eq!(+ body.pointer("/stem/settings/index/analysis"),+ Some(&want),+ "the declared stemmer must still be reported after a restart: {body}"+ );+ let (_, body) = send(&app, get("/stem")).await;+ assert_eq!(+ body.pointer("/stem/settings/index/analysis"),+ Some(&want),+ "GET /{{index}} reports the same block: {body}"+ );+ // Search still stems after the restart — the thing being reported.+ let (_, tokens, _) = analyze(&app, "/stem/_analyze", json!({ "text": "bagels" })).await;+ assert_eq!(tokens, ["bagel"]);+}++#[tokio::test]+async fn top_level_analysis_beside_an_index_block_is_reported() {+ let dir = tempfile::tempdir().expect("tempdir");+ let app = app_over(dir.path());+ let (st, body) = send(+ &app,+ with_body(+ "PUT",+ "/both",+ json!({ "settings": {+ "index": { "number_of_replicas": 0 },+ "analysis": { "analyzer": { "default": { "type": "stemmer" } } }+ }}),+ ),+ )+ .await;+ assert_eq!(st, StatusCode::OK, "create: {body}");+ let (_, body) = send(&app, get("/both/_settings")).await;+ assert_eq!(+ body.pointer("/both/settings/index/analysis/analyzer/default/type"),+ Some(&json!("stemmer")),+ "{body}"+ );+ assert_eq!(+ body.pointer("/both/settings/index/number_of_replicas"),+ Some(&json!("0"))+ );+}++#[tokio::test]+async fn analyze_uses_the_index_default_and_the_field_mapping() {+ let dir = tempfile::tempdir().expect("tempdir");+ let app = app_over(dir.path());+ create_stem_index(&app).await;++ // No analyzer, no field → the index's default analyzer.+ let (st, tokens, body) =+ analyze(&app, "/stem/_analyze", json!({ "text": "Bagels running" })).await;+ assert_eq!(st, StatusCode::OK, "{body}");+ assert_eq!(tokens, ["bagel", "run"], "{body}");++ // A text field → the analyzer that field is indexed with.+ let (_, tokens, _) = analyze(+ &app,+ "/stem/_analyze",+ json!({ "field": "t", "text": "Bagels running" }),+ )+ .await;+ assert_eq!(tokens, ["bagel", "run"]);++ // A keyword field → one exact token.+ let (_, tokens, _) = analyze(+ &app,+ "/stem/_analyze",+ json!({ "field": "tag", "text": "Bagels running" }),+ )+ .await;+ assert_eq!(tokens, ["Bagels running"]);++ // An explicitly named built-in still wins over the default.+ let (_, tokens, _) = analyze(+ &app,+ "/stem/_analyze",+ json!({ "analyzer": "standard", "text": "Bagels running" }),+ )+ .await;+ assert_eq!(tokens, ["bagels", "running"]);++ // An index WITHOUT an analysis block analyses as `standard`.+ let (st, _) = send(&app, with_body("PUT", "/plain", json!({}))).await;+ assert_eq!(st, StatusCode::OK);+ let (_, tokens, _) =+ analyze(&app, "/plain/_analyze", json!({ "text": "Bagels running" })).await;+ assert_eq!(tokens, ["bagels", "running"]);+ let (_, body) = send(&app, get("/plain/_settings")).await;+ assert!(+ body.pointer("/plain/settings/index/analysis").is_none(),+ "no analysis is in force, so none is reported: {body}"+ );+}++#[tokio::test]+async fn analyze_refuses_what_it_cannot_honour() {+ let dir = tempfile::tempdir().expect("tempdir");+ let app = app_over(dir.path());+ create_stem_index(&app).await;++ for (path, body) in [+ ("/stem/_analyze", json!({ "analyzer": "nope", "text": "x" })),+ ("/_analyze", json!({ "analyzer": "nope", "text": "x" })),+ ("/_analyze", json!({ "field": "t", "text": "x" })),+ (+ "/_analyze",+ json!({ "tokenizer": "standard", "filter": ["no_such_filter"], "text": "x" }),+ ),+ (+ "/_analyze",+ json!({ "tokenizer": "no_such_tokenizer", "text": "x" }),+ ),+ (+ "/_analyze",+ json!({ "normalizer": "lowercase", "text": "x" }),+ ),+ ] {+ let (st, _, resp) = analyze(&app, path, body.clone()).await;+ assert_eq!(st, StatusCode::BAD_REQUEST, "{path} {body}: {resp}");+ assert_eq!(+ resp.pointer("/error/type"),+ Some(&json!("illegal_argument_exception")),+ "{resp}"+ );+ }++ // Unknown index → 404, as ES and OpenSearch answer.+ let (st, _, _) = analyze(&app, "/missing/_analyze", json!({ "text": "x" })).await;+ assert_eq!(st, StatusCode::NOT_FOUND);++ // The global endpoint runs the real built-in pipelines.+ let (st, tokens, _) = analyze(+ &app,+ "/_analyze",+ json!({ "analyzer": "english", "text": "the running bagels" }),+ )+ .await;+ assert_eq!(st, StatusCode::OK);+ assert_eq!(tokens, ["run", "bagel"]);+ let (_, tokens, _) = analyze(&app, "/_analyze", json!({ "text": "Foo BAR" })).await;+ assert_eq!(tokens, ["foo", "bar"]);+}++/// An ad-hoc chain is built by the same registry code index creation uses,+/// with the Elasticsearch/OpenSearch defaults (filters alone → `keyword`+/// tokenizer) — it used to be answered by the `standard` splitter whatever+/// it named.+#[tokio::test]+async fn analyze_builds_an_adhoc_chain() {+ let dir = tempfile::tempdir().expect("tempdir");+ let app = app_over(dir.path());+ let (st, tokens, body) = analyze(+ &app,+ "/_analyze",+ json!({ "tokenizer": "whitespace", "filter": ["lowercase"], "text": "Foo-Bar BAZ" }),+ )+ .await;+ assert_eq!(st, StatusCode::OK, "{body}");+ assert_eq!(tokens, ["foo-bar", "baz"]);++ let (_, tokens, _) = analyze(+ &app,+ "/_analyze",+ json!({ "filter": ["lowercase"], "text": "Foo Bar" }),+ )+ .await;+ assert_eq!(+ tokens,+ ["foo bar"],+ "filters alone run on the keyword tokenizer"+ );+}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 5102a3e4c..bf9062be6 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -8262,6 +8262,12 @@ pub struct Index {/// (pre-existing, split) behaviour rather than silently changing how/// its unflushed documents match.segment_default_analyzer_honored: bool,+ /// #1092 — which `analysis` spelling built [`Self::registry`]+ /// (`Canonical` for every index this build creates; possibly+ /// `LegacyShorthandOnly` for a pre-#204 index reopened with documents).+ /// Kept so the API can report the analysis block actually in force+ /// rather than the one merely declared — see [`Self::analysis_in_force`].+ analysis_binding: AnalysisBinding,data_dir: PathBuf,/// Doc count threshold for auto-flush (default: 10,000).flush_doc_threshold: usize,@@ -9090,6 +9096,7 @@ impl Index {query_cache_misses: Arc::new(AtomicU64::new(0)),registry,segment_default_analyzer_honored: true,+ analysis_binding: AnalysisBinding::Canonical,data_dir: index_dir,flush_doc_threshold,flush_byte_threshold,@@ -9563,6 +9570,7 @@ impl Index {query_cache_misses: Arc::new(AtomicU64::new(0)),registry,segment_default_analyzer_honored,+ analysis_binding,data_dir: index_dir,flush_doc_threshold,flush_byte_threshold,@@ -24506,6 +24514,62 @@ impl Index {self.settings.read().await.clone()}+ /// The `analysis` block this index's analyzer registry was built from —+ /// resolved with the binding the index was opened under, so a pre-#204+ /// index whose canonical `index.analysis` block is declared but NOT+ /// honoured reports `None` rather than an analyzer that analyses nothing+ /// (#1092). Read from the persisted `settings.json` copy, so it survives a+ /// restart (the API's display map does not).+ pub async fn analysis_in_force(&self) -> Option<Value> {+ let settings = self.settings.read().await;+ let root = settings.pointer("/settings").unwrap_or(&settings);+ AnalyzerRegistry::analysis_block_with_binding(root, self.analysis_binding).cloned()+ }++ /// `_analyze` against this index (#1092): run `text` through the pipeline+ /// a search on this index would actually use, and return its name with+ /// the tokens.+ ///+ /// * `analyzer` named → that analyzer from the index's own registry+ /// (custom analyzers from settings, then the built-ins).+ /// * else `field` named → the analyzer the FTS writer indexes that field+ /// with ([`build_fts_field_configs`]): the text analyzer for a `text`+ /// field, `keyword` for every other mapped type, and the registry's+ /// `default` (else `standard`) for an unmapped field, as the writer's+ /// unconfigured-field fallback does.+ /// * neither → the index's text analyzer: the declared `default` when this+ /// index's segments honour it, else `standard`+ /// ([`Self::segment_text_analyzer`], #937).+ ///+ /// `Err` carries the reason for an unknown analyzer name.+ pub async fn analyze(+ &self,+ analyzer: Option<&str>,+ field: Option<&str>,+ text: &str,+ ) -> std::result::Result<(String, Vec<xerj_fts::analyzer::Token>), String> {+ let name: String = match (analyzer, field) {+ (Some(a), _) => a.to_string(),+ (None, Some(f)) => {+ let schema = self.schema.read().await;+ match schema.schema.field(f).map(|fc| fc.field_type) {+ Some(FieldType::Text) => self.segment_text_analyzer().to_string(),+ Some(_) => "keyword".to_string(),+ None if self.registry.get_analyzer("default").is_some() => {+ "default".to_string()+ }+ None => "standard".to_string(),+ }+ }+ (None, None) => self.segment_text_analyzer().to_string(),+ };+ let pipeline = self+ .registry+ .get_analyzer(&name)+ .ok_or_else(|| format!("failed to find analyzer [{name}]"))?;+ Ok((name, pipeline.analyze(text)))+ }+/// Update the index-level settings.pub async fn put_settings(&self, new_settings: Value) -> Result<()> {// Warn if >1 shard requested.diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex 0f9187158..b269b1576 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -243,7 +243,7 @@ use std::sync::Arc;// The create-time and settings-time gates drifted apart once already — one// refused four spellings, the other two — and one shared symbol is what stops// that (issue #204).-pub use xerj_fts::analyzer::AnalyzerRegistry;+pub use xerj_fts::analyzer::{AnalyzerRegistry, Token};use xerj_fts::analyzer::{LowercaseFilter, StandardTokenizer, StemmerFilter, StopwordsFilter, TokenFilter, Tokenizer,};diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rsindex c43616c0b..e031622a4 100644--- a/engine/crates/xerj-engine/tests/integration.rs+++ b/engine/crates/xerj-engine/tests/integration.rs@@ -9104,6 +9104,14 @@ async fn reopening_a_pre_sweep_index_does_not_silently_change_how_it_analyses()let engine = make_engine(&dir);let idx = engine.get_index("legacy_idx").unwrap();+ // #1092: the declared-but-not-honoured block is not reported as in force,+ // so `GET /{index}/_settings` does not advertise an analyzer that+ // analyses nothing.+ assert_eq!(+ idx.analysis_in_force().await,+ None,+ "a canonical block a pre-sweep index never honoured is not in force"+ );// The document already on disk is still findable…let result = idx.search(&make_search(json!({"match": {"name": "basketball"}})))@@ -9218,6 +9226,18 @@ async fn a_freshly_created_index_honours_the_canonical_analysis_nesting() {result.total.value, 1,"a NEW index must honour `index.analysis` — that half of the fix stands");+ // #1092: and reports that block as the one in force, and `_analyze`+ // without an analyzer runs it.+ assert_eq!(+ idx.analysis_in_force().await.as_ref(),+ namespaced_ngram_settings().pointer("/index/analysis")+ );+ let (name, tokens) = idx.analyze(None, None, "ket").await.unwrap();+ assert_eq!(name, "default");+ assert_eq!(+ tokens.iter().map(|t| t.text.as_str()).collect::<Vec<_>>(),+ ["ket"]+ );assert!(dir.path().join("new_idx")diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rsindex 0cc6fca43..6c77d44a4 100644--- a/engine/crates/xerj-fts/src/analyzer.rs+++ b/engine/crates/xerj-fts/src/analyzer.rs@@ -1898,7 +1898,7 @@ impl AnalyzerRegistry {Self::analysis_block_with_binding(settings, AnalysisBinding::Canonical)}- fn analysis_block_with_binding(+ pub fn analysis_block_with_binding(settings: &serde_json::Value,binding: AnalysisBinding,) -> Option<&serde_json::Value> {diff --git a/engine/tests/es-compat-yaml/src/main.rs b/engine/tests/es-compat-yaml/src/main.rsindex f2c6c62a8..5ad38813a 100644--- a/engine/tests/es-compat-yaml/src/main.rs+++ b/engine/tests/es-compat-yaml/src/main.rs@@ -861,6 +861,14 @@ fn resolve_action(action: &str, params: &serde_yaml::Mapping) -> (String, String}("DELETE".into(), path, None)}+ "indices.analyze" => {+ let path = if index.is_empty() {+ "/_analyze".to_string()+ } else {+ format!("/{}/_analyze", index)+ };+ ("POST".into(), path, body)+ }"indices.refresh" => ("POST".into(),format!("/{}/_refresh", if index.is_empty() { "*" } else { &index }),diff --git a/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.yml b/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.ymlnew file mode 100644index 000000000..f6fdd8834--- /dev/null+++ b/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.yml@@ -0,0 +1,100 @@+---+# Issue #1092: an index's analysis configuration is observable over the wire+# exactly as search applies it. Expected values were checked against+# OpenSearch 3.9.0, which answers every request below identically.+setup:+ - do:+ indices.create:+ index: test_analysis+ body:+ settings:+ analysis:+ analyzer:+ default:+ type: english+ mappings:+ properties:+ t:+ type: text+ tag:+ type: keyword++---+"get_settings reports the declared analysis block":+ - do:+ indices.get_settings:+ index: test_analysis++ - match: { test_analysis.settings.index.analysis.analyzer.default.type: english }++---+"analyze without an analyzer uses the index default":+ - do:+ indices.analyze:+ index: test_analysis+ body:+ text: The running bagels++ - length: { tokens: 2 }+ - match: { tokens.0.token: run }+ - match: { tokens.0.start_offset: 4 }+ - match: { tokens.0.position: 1 }+ - match: { tokens.1.token: bagel }++---+"analyze a field uses the analyzer it is indexed with":+ - do:+ indices.analyze:+ index: test_analysis+ body:+ field: t+ text: The running bagels++ - match: { tokens.0.token: run }++ - do:+ indices.analyze:+ index: test_analysis+ body:+ field: tag+ text: The running bagels++ - length: { tokens: 1 }+ - match: { tokens.0.token: The running bagels }++---+"a named analyzer wins over the index default":+ - do:+ indices.analyze:+ index: test_analysis+ body:+ analyzer: standard+ text: The running bagels++ - length: { tokens: 3 }+ - match: { tokens.2.token: bagels }++---+"an ad-hoc chain with filters only runs on the keyword tokenizer":+ - do:+ indices.analyze:+ body:+ filter: [lowercase]+ text: Foo Bar++ - length: { tokens: 1 }+ - match: { tokens.0.token: foo bar }++---+"an unknown analyzer is a 400":+ - do:+ catch: bad_request+ indices.analyze:+ index: test_analysis+ body:+ analyzer: no_such_analyzer+ text: x++ # The runner records but does not enforce `catch`; assert the error itself.+ - match: { error.type: illegal_argument_exception }+ - match: { status: 400 }