Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(es-compat): report analysis as in force in _settings, run real pipelines in _analyze (#1092)

Vincenzo Lombardo211ea90071a0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d0db6c75e..19dbac936 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Fixed
+
+- **An index's analysis configuration could not be verified over the wire**
+ (issue [#1092](https://github.com/xerj-org/xerj/issues/1092)) — search
+ honoured a declared `analysis.analyzer.default`, but nothing an agent can
+ read said so. `GET /{index}/_settings` and `GET /{index}` replayed an
+ in-memory copy of the create body: after a restart it was empty, so the
+ analysis block (and every other create-time setting) vanished, and a
+ top-level `analysis` beside an `index` block was never echoed at all. Both
+ now report `index.analysis` as the block the index's analyzer registry was
+ actually built from — none for a pre-#204 index whose declared block is
+ not honoured — and fall back to the persisted `settings.json` when the
+ in-memory copy is gone. `POST /{index}/_analyze` was an inline lowercase
+ splitter that ignored the index; it now runs the index's real pipelines:
+ no `analyzer` → the index's default, `field` → the analyzer that field is
+ indexed with, a named analyzer from the index's own registry. Ad-hoc
+ `tokenizer` / `filter` / `char_filter` chains (names or inline
+ definitions) are built by the same registry code index creation uses, with
+ `keyword` as the tokenizer when only filters are given. Unknown analyzers,
+ tokenizers and filters, `field` on the global endpoint and `normalizer`
+ are 400 `illegal_argument_exception`. A missing index is 404. Checked
+ side by side against OpenSearch 3.9.0: status codes, error types, tokens,
+ offsets and positions match on all 24 probed requests. Only the wording of
+ the reason differs on 3 of the 400s.
+
## [1.0.0-rc.84] - 2026-10-06
The tell-the-truth window. Every signal an operator reads — cluster
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 39601125f..4bc537dce 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -2042,15 +2042,10 @@ async fn get_index_inner(
};
// Settings: replay what was written (normalized to strings), merged
- // with engine defaults. Creation timestamps are synthesized because
- // we don't persist them yet (TODO).
- let stored_settings = state
- .engine
- .index_settings
- .get(name)
- .map(|v| v.clone())
- .unwrap_or(Value::Null);
- let settings_inner = merge_settings_defaults(&stored_settings, name, human);
+ // with engine defaults; `analysis` is the block in force (#1092).
+ // Creation timestamps are synthesized because we don't persist them
+ // yet (TODO).
+ let settings_inner = index_settings_response(&state, name, idx.as_ref(), human).await;
let mut index_obj = serde_json::Map::new();
if want_aliases {
@@ -2068,6 +2063,52 @@ async fn get_index_inner(
Json(Value::Object(body)).into_response()
}
+/// The `settings` object `GET /{index}` and `GET /{index}/_settings` serve.
+///
+/// Two fixes over replaying the display map alone (#1092):
+///
+/// * The display map (`engine.index_settings`) lives in memory only, so after a
+/// restart it is empty and every create-time setting vanished from the
+/// response. A live index with no display entry falls back to its persisted
+/// `settings.json` copy — what the index was created with.
+/// * `analysis` is reported from the engine, not echoed: whatever spelling the
+/// caller used (`settings.analysis` beside an `index` block was dropped
+/// outright), the response carries `index.analysis` = the block the index's
+/// analyzer registry was actually built from, and nothing for a pre-#204
+/// index whose declared block is not honoured. An agent verifying "is my
+/// stemmer on?" over the wire gets the answer search would give.
+///
+/// A failed index (`idx == None`) keeps the display-map-only behaviour.
+async fn index_settings_response(
+ state: &AppState,
+ name: &str,
+ idx: Option<&std::sync::Arc<xerj_engine::Index>>,
+ human: bool,
+) -> Value {
+ let mut stored = state
+ .engine
+ .index_settings
+ .get(name)
+ .map(|v| v.clone())
+ .unwrap_or(Value::Null);
+ if stored.is_null() {
+ if let Some(i) = idx {
+ stored = i.get_settings().await;
+ }
+ }
+ let mut settings = merge_settings_defaults(&stored, name, human);
+ if let Some(i) = idx {
+ let in_force = i.analysis_in_force().await;
+ if let Some(inner) = settings.get_mut("index").and_then(Value::as_object_mut) {
+ inner.retain(|k, _| k != "analysis" && !k.starts_with("analysis."));
+ if let Some(a) = in_force {
+ inner.insert("analysis".to_string(), a);
+ }
+ }
+ }
+ settings
+}
+
/// Normalize a raw user-provided settings blob to the ES response shape.
///
/// ES always wraps user settings under `{ "index": { ... } }` and coerces
@@ -2767,13 +2808,8 @@ pub async fn get_settings(
}
let mut out = serde_json::Map::new();
for name in &targets {
- let stored = state
- .engine
- .index_settings
- .get(name)
- .map(|v| v.clone())
- .unwrap_or(Value::Null);
- let settings = merge_settings_defaults(&stored, name, false);
+ let idx = state.engine.get_index(name).ok();
+ let settings = index_settings_response(&state, name, idx.as_ref(), false).await;
out.insert(name.clone(), json!({ "settings": settings }));
}
Json(Value::Object(out)).into_response()
@@ -21120,8 +21156,26 @@ pub async fn cat_nodes(
#[derive(Debug, Deserialize)]
pub struct EsAnalyzeBody {
pub text: Option<Value>,
- #[serde(default = "default_analyzer")]
- pub analyzer: String,
+ /// Absent = the index's own text analyzer on `/{index}/_analyze` (its
+ /// declared `default`, #1092), `standard` on the global endpoint.
+ #[serde(default)]
+ pub analyzer: Option<String>,
+ /// Analyse as this field of the index is indexed (#1092).
+ #[serde(default)]
+ pub field: Option<String>,
+ /// Ad-hoc chain parts (`tokenizer`, `filter`, `char_filter`): each a
+ /// name or an inline definition, built into a one-off `custom` analyzer
+ /// by the same registry code index creation uses (#1092). `normalizer`
+ /// is refused with a 400 rather than silently answered with another
+ /// analyzer's tokens.
+ #[serde(default)]
+ pub tokenizer: Option<Value>,
+ #[serde(default)]
+ pub filter: Option<Value>,
+ #[serde(default)]
+ pub char_filter: Option<Value>,
+ #[serde(default)]
+ pub normalizer: Option<Value>,
/// When true, returns step-by-step output of each pipeline stage.
///
/// Shows: tokenizer output → lowercase → stopwords → stemmer.
@@ -21130,8 +21184,120 @@ pub struct EsAnalyzeBody {
pub explain: bool,
}
-fn default_analyzer() -> String {
- "standard".to_string()
+fn analyze_illegal_argument(reason: String) -> Response {
+ (
+ StatusCode::BAD_REQUEST,
+ Json(json!({
+ "error": {
+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],
+ "type": "illegal_argument_exception",
+ "reason": reason,
+ },
+ "status": 400,
+ })),
+ )
+ .into_response()
+}
+
+/// Name of the one-off analyzer an ad-hoc `_analyze` chain is built as.
+const ADHOC_ANALYZER: &str = "_analyze_adhoc";
+
+/// Turn an ad-hoc `_analyze` chain into an `analysis` settings block declaring
+/// one `custom` analyzer, so it is built — and validated — by exactly the
+/// registry code an index's own analysis goes through (#1092).
+///
+/// Names refer to built-ins, or on an index-scoped call to the components the
+/// index declares (`index_analysis`, the block in force); inline definitions
+/// get reserved `_analyze_*` names. With filters and no tokenizer the chain
+/// uses `keyword`, as Elasticsearch and OpenSearch do.
+fn adhoc_analysis_settings(body: &EsAnalyzeBody, index_analysis: Option<&Value>) -> Value {
+ let mut analysis = serde_json::Map::new();
+ for kind in ["tokenizer", "filter", "char_filter"] {
+ let defs = index_analysis
+ .and_then(|a| a.get(kind))
+ .cloned()
+ .unwrap_or_else(|| json!({}));
+ analysis.insert(kind.to_string(), defs);
+ }
+ let mut define = |kind: &str, name: String, def: &Value| -> String {
+ if let Some(m) = analysis.get_mut(kind).and_then(Value::as_object_mut) {
+ m.insert(name.clone(), def.clone());
+ }
+ name
+ };
+ let tokenizer = match &body.tokenizer {
+ Some(Value::String(s)) => s.clone(),
+ Some(def) => define("tokenizer", "_analyze_tokenizer".to_string(), def),
+ None => "keyword".to_string(),
+ };
+ let mut chain = |kind: &str, parts: &Option<Value>| -> Vec<String> {
+ let items = match parts {
+ Some(Value::Array(a)) => a.clone(),
+ Some(single) => vec![single.clone()],
+ None => Vec::new(),
+ };
+ items
+ .iter()
+ .enumerate()
+ .map(|(i, part)| match part {
+ Value::String(s) => s.clone(),
+ def => define(kind, format!("_analyze_{kind}_{i}"), def),
+ })
+ .collect()
+ };
+ let filter = chain("filter", &body.filter);
+ let char_filter = chain("char_filter", &body.char_filter);
+ analysis.insert(
+ "analyzer".to_string(),
+ json!({ ADHOC_ANALYZER: {
+ "type": "custom",
+ "tokenizer": tokenizer,
+ "filter": filter,
+ "char_filter": char_filter,
+ }}),
+ );
+ json!({ "analysis": Value::Object(analysis) })
+}
+
+/// Build and run an ad-hoc chain. `Err` = the registry cannot honour it, with
+/// the same reasons index creation would give.
+fn run_adhoc_analyze(
+ body: &EsAnalyzeBody,
+ index_analysis: Option<&Value>,
+ input: &str,
+) -> Result<Vec<xerj_engine::Token>, String> {
+ let settings = adhoc_analysis_settings(body, index_analysis);
+ let problems = xerj_engine::AnalyzerRegistry::unsupported_analysis(&settings);
+ if !problems.is_empty() {
+ return Err(format!(
+ "xerj cannot honour this analysis chain — {}",
+ problems.join("; ")
+ ));
+ }
+ let mut registry = xerj_engine::AnalyzerRegistry::with_defaults();
+ registry.apply_settings(&settings);
+ let pipeline = registry
+ .get_analyzer(ADHOC_ANALYZER)
+ .ok_or_else(|| "xerj could not build this analysis chain".to_string())?;
+ Ok(pipeline.analyze(input))
+}
+
+/// ES `_analyze` token objects from real pipeline output. `type` stays the
+/// `word` this endpoint has always reported; offsets and positions are the
+/// pipeline's own.
+fn analyze_tokens_json(tokens: &[xerj_engine::Token]) -> Vec<Value> {
+ tokens
+ .iter()
+ .map(|t| {
+ json!({
+ "token": t.text,
+ "start_offset": t.start_offset,
+ "end_offset": t.end_offset,
+ "type": "word",
+ "position": t.position,
+ })
+ })
+ .collect()
}
/// Global `POST /_analyze` (no index prefix).
@@ -21143,8 +21309,8 @@ pub async fn analyze_text_global(
}
pub async fn analyze_text(
- State(_state): State<AppState>,
- Path(_index): Path<String>,
+ State(state): State<AppState>,
+ Path(index): Path<String>,
Json(body): Json<EsAnalyzeBody>,
) -> impl IntoResponse {
let input = match &body.text {
@@ -21157,19 +21323,97 @@ pub async fn analyze_text(
_ => String::new(),
};
- if body.explain {
- // Step-by-step pipeline explanation.
- //
- // Shows each filter stage in order so developers can debug why a term
- // was dropped, modified, or expanded. Supported for the "standard"
- // analyzer (the most common case). Other analyzers fall back to showing
- // tokenizer + final output only.
- let explanation = analyze_explain(&input, &body.analyzer);
- return Json(explanation).into_response();
+ if body.normalizer.is_some() {
+ return analyze_illegal_argument(
+ "_analyze does not support [normalizer] on xerj (it would otherwise be answered \
+ with another analyzer's tokens)"
+ .to_string(),
+ );
+ }
+ let adhoc = body.tokenizer.is_some() || body.filter.is_some() || body.char_filter.is_some();
+ if adhoc {
+ if body.analyzer.is_some() || body.field.is_some() {
+ return analyze_illegal_argument(
+ "cannot define an ad-hoc chain (tokenizer/filter/char_filter) together with \
+ [analyzer] or [field]"
+ .to_string(),
+ );
+ }
+ if body.explain {
+ return analyze_illegal_argument(
+ "_analyze explain is not supported for an ad-hoc chain on xerj".to_string(),
+ );
+ }
+ // Index-scoped: the chain may name the index's own components.
+ let index_analysis = if index == "_none" {
+ None
+ } else {
+ let targets = resolve_index_selector(&state, &index).await;
+ let Some(idx) = targets
+ .first()
+ .and_then(|name| state.engine.get_index(name).ok())
+ else {
+ return ApiError::new(xerj_common::XerjError::index_not_found(&index))
+ .into_response();
+ };
+ idx.analysis_in_force().await
+ };
+ return match run_adhoc_analyze(&body, index_analysis.as_ref(), &input) {
+ Ok(tokens) => Json(json!({ "tokens": analyze_tokens_json(&tokens) })).into_response(),
+ Err(reason) => analyze_illegal_argument(reason),
+ };
+ }
+
+ if index == "_none" {
+ // Global endpoint: built-in analyzers only, no index to resolve a
+ // field or a custom analyzer against.
+ if let Some(f) = &body.field {
+ return analyze_illegal_argument(format!(
+ "analysing field [{f}] requires an index: use POST /{{index}}/_analyze"
+ ));
+ }
+ let name = body.analyzer.as_deref().unwrap_or("standard");
+ if body.explain {
+ // Step-by-step pipeline explanation (unchanged): stage-by-stage
+ // for "standard"/"english", final tokens otherwise.
+ return Json(analyze_explain(&input, name)).into_response();
+ }
+ let Some(pipeline) = xerj_engine::analyzer_registry().get_analyzer(name) else {
+ return analyze_illegal_argument(format!("failed to find global analyzer [{name}]"));
+ };
+ let tokens = analyze_tokens_json(&pipeline.analyze(&input));
+ return Json(json!({ "tokens": tokens })).into_response();
}
- let tokens = tokenize_for_analyze(&input, &body.analyzer);
- Json(json!({ "tokens": tokens })).into_response()
+ // Index-scoped (#1092): the index's OWN registry and field resolution —
+ // the pipeline a search on this index actually runs, so a declared
+ // `analysis.analyzer.default` stemmer shows up here.
+ let targets = resolve_index_selector(&state, &index).await;
+ let Some(idx) = targets
+ .first()
+ .and_then(|name| state.engine.get_index(name).ok())
+ else {
+ return ApiError::new(xerj_common::XerjError::index_not_found(&index)).into_response();
+ };
+ match idx
+ .analyze(body.analyzer.as_deref(), body.field.as_deref(), &input)
+ .await
+ {
+ Ok((name, tokens)) => {
+ let tokens = analyze_tokens_json(&tokens);
+ if body.explain {
+ return Json(json!({
+ "detail": {
+ "custom_analyzer": false,
+ "analyzer": { "name": name, "tokens": tokens },
+ }
+ }))
+ .into_response();
+ }
+ Json(json!({ "tokens": tokens })).into_response()
+ }
+ Err(reason) => analyze_illegal_argument(reason),
+ }
}
/// Produce step-by-step analysis explanation.
diff --git a/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rs b/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rs
new file mode 100644
index 000000000..724f92a49
--- /dev/null
+++ b/engine/crates/xerj-api/tests/analysis_is_reported_as_in_force.rs
@@ -0,0 +1,286 @@
+//! Issue #1092: an index's analysis configuration must be observable over the
+//! wire exactly as search applies it.
+//!
+//! Before the fix:
+//! * `GET /{index}/_settings` replayed an in-memory display copy of the create
+//! body. After a restart that copy is empty, so a declared
+//! `analysis.analyzer.default` stemmer vanished from the response while
+//! search kept stemming — and a top-level `analysis` beside an `index` block
+//! was never echoed at all.
+//! * `POST /{index}/_analyze` ignored the index entirely: an inline lowercase
+//! splitter answered every request, so a stemmer index showed `bagels`
+//! where search indexes `bagel`.
+//!
+//! An agent verifying "is my stemmer on?" through the ES wire concluded it was
+//! off. Elasticsearch and OpenSearch are referenced for wire semantics only
+//! (public API docs: `_analyze` without `analyzer`/`field` uses the index's
+//! default analyzer; `field` uses that field's analyzer); no code is reproduced.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+fn app_over(dir: &std::path::Path) -> axum::Router {
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ xerj_api::router::build_es_compat_router(state)
+}
+
+async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {
+ let response = app.clone().oneshot(req).await.expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ let body = if bytes.is_empty() {
+ Value::Null
+ } else {
+ serde_json::from_slice(&bytes).unwrap_or(Value::Null)
+ };
+ (status, body)
+}
+
+fn with_body(method: &str, path: &str, body: Value) -> Request<Body> {
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request")
+}
+
+fn get(path: &str) -> Request<Body> {
+ Request::get(path).body(Body::empty()).expect("request")
+}
+
+async fn analyze(app: &axum::Router, path: &str, body: Value) -> (StatusCode, Vec<String>, Value) {
+ let (st, resp) = send(app, with_body("POST", path, body)).await;
+ let tokens = resp["tokens"]
+ .as_array()
+ .map(|a| {
+ a.iter()
+ .map(|t| t["token"].as_str().unwrap_or_default().to_string())
+ .collect()
+ })
+ .unwrap_or_default();
+ (st, tokens, resp)
+}
+
+const STEM: &str = "/stem";
+
+async fn create_stem_index(app: &axum::Router) {
+ let (st, body) = send(
+ app,
+ with_body(
+ "PUT",
+ STEM,
+ json!({
+ "settings": { "analysis": { "analyzer": { "default": { "type": "stemmer" } } } },
+ "mappings": { "properties": {
+ "t": { "type": "text" }, "tag": { "type": "keyword" }
+ }}
+ }),
+ ),
+ )
+ .await;
+ assert_eq!(st, StatusCode::OK, "create: {body}");
+}
+
+#[tokio::test]
+async fn settings_report_the_analysis_in_force_across_a_restart() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let want = json!({ "analyzer": { "default": { "type": "stemmer" } } });
+ {
+ let app = app_over(dir.path());
+ create_stem_index(&app).await;
+ let (st, body) = send(&app, get("/stem/_settings")).await;
+ assert_eq!(st, StatusCode::OK);
+ assert_eq!(
+ body.pointer("/stem/settings/index/analysis"),
+ Some(&want),
+ "{body}"
+ );
+ }
+ // Restart: only what reached disk survives.
+ let app = app_over(dir.path());
+ let (_, body) = send(&app, get("/stem/_settings")).await;
+ assert_eq!(
+ body.pointer("/stem/settings/index/analysis"),
+ Some(&want),
+ "the declared stemmer must still be reported after a restart: {body}"
+ );
+ let (_, body) = send(&app, get("/stem")).await;
+ assert_eq!(
+ body.pointer("/stem/settings/index/analysis"),
+ Some(&want),
+ "GET /{{index}} reports the same block: {body}"
+ );
+ // Search still stems after the restart — the thing being reported.
+ let (_, tokens, _) = analyze(&app, "/stem/_analyze", json!({ "text": "bagels" })).await;
+ assert_eq!(tokens, ["bagel"]);
+}
+
+#[tokio::test]
+async fn top_level_analysis_beside_an_index_block_is_reported() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let app = app_over(dir.path());
+ let (st, body) = send(
+ &app,
+ with_body(
+ "PUT",
+ "/both",
+ json!({ "settings": {
+ "index": { "number_of_replicas": 0 },
+ "analysis": { "analyzer": { "default": { "type": "stemmer" } } }
+ }}),
+ ),
+ )
+ .await;
+ assert_eq!(st, StatusCode::OK, "create: {body}");
+ let (_, body) = send(&app, get("/both/_settings")).await;
+ assert_eq!(
+ body.pointer("/both/settings/index/analysis/analyzer/default/type"),
+ Some(&json!("stemmer")),
+ "{body}"
+ );
+ assert_eq!(
+ body.pointer("/both/settings/index/number_of_replicas"),
+ Some(&json!("0"))
+ );
+}
+
+#[tokio::test]
+async fn analyze_uses_the_index_default_and_the_field_mapping() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let app = app_over(dir.path());
+ create_stem_index(&app).await;
+
+ // No analyzer, no field → the index's default analyzer.
+ let (st, tokens, body) =
+ analyze(&app, "/stem/_analyze", json!({ "text": "Bagels running" })).await;
+ assert_eq!(st, StatusCode::OK, "{body}");
+ assert_eq!(tokens, ["bagel", "run"], "{body}");
+
+ // A text field → the analyzer that field is indexed with.
+ let (_, tokens, _) = analyze(
+ &app,
+ "/stem/_analyze",
+ json!({ "field": "t", "text": "Bagels running" }),
+ )
+ .await;
+ assert_eq!(tokens, ["bagel", "run"]);
+
+ // A keyword field → one exact token.
+ let (_, tokens, _) = analyze(
+ &app,
+ "/stem/_analyze",
+ json!({ "field": "tag", "text": "Bagels running" }),
+ )
+ .await;
+ assert_eq!(tokens, ["Bagels running"]);
+
+ // An explicitly named built-in still wins over the default.
+ let (_, tokens, _) = analyze(
+ &app,
+ "/stem/_analyze",
+ json!({ "analyzer": "standard", "text": "Bagels running" }),
+ )
+ .await;
+ assert_eq!(tokens, ["bagels", "running"]);
+
+ // An index WITHOUT an analysis block analyses as `standard`.
+ let (st, _) = send(&app, with_body("PUT", "/plain", json!({}))).await;
+ assert_eq!(st, StatusCode::OK);
+ let (_, tokens, _) =
+ analyze(&app, "/plain/_analyze", json!({ "text": "Bagels running" })).await;
+ assert_eq!(tokens, ["bagels", "running"]);
+ let (_, body) = send(&app, get("/plain/_settings")).await;
+ assert!(
+ body.pointer("/plain/settings/index/analysis").is_none(),
+ "no analysis is in force, so none is reported: {body}"
+ );
+}
+
+#[tokio::test]
+async fn analyze_refuses_what_it_cannot_honour() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let app = app_over(dir.path());
+ create_stem_index(&app).await;
+
+ for (path, body) in [
+ ("/stem/_analyze", json!({ "analyzer": "nope", "text": "x" })),
+ ("/_analyze", json!({ "analyzer": "nope", "text": "x" })),
+ ("/_analyze", json!({ "field": "t", "text": "x" })),
+ (
+ "/_analyze",
+ json!({ "tokenizer": "standard", "filter": ["no_such_filter"], "text": "x" }),
+ ),
+ (
+ "/_analyze",
+ json!({ "tokenizer": "no_such_tokenizer", "text": "x" }),
+ ),
+ (
+ "/_analyze",
+ json!({ "normalizer": "lowercase", "text": "x" }),
+ ),
+ ] {
+ let (st, _, resp) = analyze(&app, path, body.clone()).await;
+ assert_eq!(st, StatusCode::BAD_REQUEST, "{path} {body}: {resp}");
+ assert_eq!(
+ resp.pointer("/error/type"),
+ Some(&json!("illegal_argument_exception")),
+ "{resp}"
+ );
+ }
+
+ // Unknown index → 404, as ES and OpenSearch answer.
+ let (st, _, _) = analyze(&app, "/missing/_analyze", json!({ "text": "x" })).await;
+ assert_eq!(st, StatusCode::NOT_FOUND);
+
+ // The global endpoint runs the real built-in pipelines.
+ let (st, tokens, _) = analyze(
+ &app,
+ "/_analyze",
+ json!({ "analyzer": "english", "text": "the running bagels" }),
+ )
+ .await;
+ assert_eq!(st, StatusCode::OK);
+ assert_eq!(tokens, ["run", "bagel"]);
+ let (_, tokens, _) = analyze(&app, "/_analyze", json!({ "text": "Foo BAR" })).await;
+ assert_eq!(tokens, ["foo", "bar"]);
+}
+
+/// An ad-hoc chain is built by the same registry code index creation uses,
+/// with the Elasticsearch/OpenSearch defaults (filters alone → `keyword`
+/// tokenizer) — it used to be answered by the `standard` splitter whatever
+/// it named.
+#[tokio::test]
+async fn analyze_builds_an_adhoc_chain() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let app = app_over(dir.path());
+ let (st, tokens, body) = analyze(
+ &app,
+ "/_analyze",
+ json!({ "tokenizer": "whitespace", "filter": ["lowercase"], "text": "Foo-Bar BAZ" }),
+ )
+ .await;
+ assert_eq!(st, StatusCode::OK, "{body}");
+ assert_eq!(tokens, ["foo-bar", "baz"]);
+
+ let (_, tokens, _) = analyze(
+ &app,
+ "/_analyze",
+ json!({ "filter": ["lowercase"], "text": "Foo Bar" }),
+ )
+ .await;
+ assert_eq!(
+ tokens,
+ ["foo bar"],
+ "filters alone run on the keyword tokenizer"
+ );
+}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 5102a3e4c..bf9062be6 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -8262,6 +8262,12 @@ pub struct Index {
/// (pre-existing, split) behaviour rather than silently changing how
/// its unflushed documents match.
segment_default_analyzer_honored: bool,
+ /// #1092 — which `analysis` spelling built [`Self::registry`]
+ /// (`Canonical` for every index this build creates; possibly
+ /// `LegacyShorthandOnly` for a pre-#204 index reopened with documents).
+ /// Kept so the API can report the analysis block actually in force
+ /// rather than the one merely declared — see [`Self::analysis_in_force`].
+ analysis_binding: AnalysisBinding,
data_dir: PathBuf,
/// Doc count threshold for auto-flush (default: 10,000).
flush_doc_threshold: usize,
@@ -9090,6 +9096,7 @@ impl Index {
query_cache_misses: Arc::new(AtomicU64::new(0)),
registry,
segment_default_analyzer_honored: true,
+ analysis_binding: AnalysisBinding::Canonical,
data_dir: index_dir,
flush_doc_threshold,
flush_byte_threshold,
@@ -9563,6 +9570,7 @@ impl Index {
query_cache_misses: Arc::new(AtomicU64::new(0)),
registry,
segment_default_analyzer_honored,
+ analysis_binding,
data_dir: index_dir,
flush_doc_threshold,
flush_byte_threshold,
@@ -24506,6 +24514,62 @@ impl Index {
self.settings.read().await.clone()
}
+ /// The `analysis` block this index's analyzer registry was built from —
+ /// resolved with the binding the index was opened under, so a pre-#204
+ /// index whose canonical `index.analysis` block is declared but NOT
+ /// honoured reports `None` rather than an analyzer that analyses nothing
+ /// (#1092). Read from the persisted `settings.json` copy, so it survives a
+ /// restart (the API's display map does not).
+ pub async fn analysis_in_force(&self) -> Option<Value> {
+ let settings = self.settings.read().await;
+ let root = settings.pointer("/settings").unwrap_or(&settings);
+ AnalyzerRegistry::analysis_block_with_binding(root, self.analysis_binding).cloned()
+ }
+
+ /// `_analyze` against this index (#1092): run `text` through the pipeline
+ /// a search on this index would actually use, and return its name with
+ /// the tokens.
+ ///
+ /// * `analyzer` named → that analyzer from the index's own registry
+ /// (custom analyzers from settings, then the built-ins).
+ /// * else `field` named → the analyzer the FTS writer indexes that field
+ /// with ([`build_fts_field_configs`]): the text analyzer for a `text`
+ /// field, `keyword` for every other mapped type, and the registry's
+ /// `default` (else `standard`) for an unmapped field, as the writer's
+ /// unconfigured-field fallback does.
+ /// * neither → the index's text analyzer: the declared `default` when this
+ /// index's segments honour it, else `standard`
+ /// ([`Self::segment_text_analyzer`], #937).
+ ///
+ /// `Err` carries the reason for an unknown analyzer name.
+ pub async fn analyze(
+ &self,
+ analyzer: Option<&str>,
+ field: Option<&str>,
+ text: &str,
+ ) -> std::result::Result<(String, Vec<xerj_fts::analyzer::Token>), String> {
+ let name: String = match (analyzer, field) {
+ (Some(a), _) => a.to_string(),
+ (None, Some(f)) => {
+ let schema = self.schema.read().await;
+ match schema.schema.field(f).map(|fc| fc.field_type) {
+ Some(FieldType::Text) => self.segment_text_analyzer().to_string(),
+ Some(_) => "keyword".to_string(),
+ None if self.registry.get_analyzer("default").is_some() => {
+ "default".to_string()
+ }
+ None => "standard".to_string(),
+ }
+ }
+ (None, None) => self.segment_text_analyzer().to_string(),
+ };
+ let pipeline = self
+ .registry
+ .get_analyzer(&name)
+ .ok_or_else(|| format!("failed to find analyzer [{name}]"))?;
+ Ok((name, pipeline.analyze(text)))
+ }
+
/// Update the index-level settings.
pub async fn put_settings(&self, new_settings: Value) -> Result<()> {
// Warn if >1 shard requested.
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index 0f9187158..b269b1576 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -243,7 +243,7 @@ use std::sync::Arc;
// The create-time and settings-time gates drifted apart once already — one
// refused four spellings, the other two — and one shared symbol is what stops
// that (issue #204).
-pub use xerj_fts::analyzer::AnalyzerRegistry;
+pub use xerj_fts::analyzer::{AnalyzerRegistry, Token};
use xerj_fts::analyzer::{
LowercaseFilter, StandardTokenizer, StemmerFilter, StopwordsFilter, TokenFilter, Tokenizer,
};
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index c43616c0b..e031622a4 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -9104,6 +9104,14 @@ async fn reopening_a_pre_sweep_index_does_not_silently_change_how_it_analyses()
let engine = make_engine(&dir);
let idx = engine.get_index("legacy_idx").unwrap();
+ // #1092: the declared-but-not-honoured block is not reported as in force,
+ // so `GET /{index}/_settings` does not advertise an analyzer that
+ // analyses nothing.
+ assert_eq!(
+ idx.analysis_in_force().await,
+ None,
+ "a canonical block a pre-sweep index never honoured is not in force"
+ );
// The document already on disk is still findable…
let result = idx
.search(&make_search(json!({"match": {"name": "basketball"}})))
@@ -9218,6 +9226,18 @@ async fn a_freshly_created_index_honours_the_canonical_analysis_nesting() {
result.total.value, 1,
"a NEW index must honour `index.analysis` — that half of the fix stands"
);
+ // #1092: and reports that block as the one in force, and `_analyze`
+ // without an analyzer runs it.
+ assert_eq!(
+ idx.analysis_in_force().await.as_ref(),
+ namespaced_ngram_settings().pointer("/index/analysis")
+ );
+ let (name, tokens) = idx.analyze(None, None, "ket").await.unwrap();
+ assert_eq!(name, "default");
+ assert_eq!(
+ tokens.iter().map(|t| t.text.as_str()).collect::<Vec<_>>(),
+ ["ket"]
+ );
assert!(
dir.path()
.join("new_idx")
diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rs
index 0cc6fca43..6c77d44a4 100644
--- a/engine/crates/xerj-fts/src/analyzer.rs
+++ b/engine/crates/xerj-fts/src/analyzer.rs
@@ -1898,7 +1898,7 @@ impl AnalyzerRegistry {
Self::analysis_block_with_binding(settings, AnalysisBinding::Canonical)
}
- fn analysis_block_with_binding(
+ pub fn analysis_block_with_binding(
settings: &serde_json::Value,
binding: AnalysisBinding,
) -> Option<&serde_json::Value> {
diff --git a/engine/tests/es-compat-yaml/src/main.rs b/engine/tests/es-compat-yaml/src/main.rs
index f2c6c62a8..5ad38813a 100644
--- a/engine/tests/es-compat-yaml/src/main.rs
+++ b/engine/tests/es-compat-yaml/src/main.rs
@@ -861,6 +861,14 @@ fn resolve_action(action: &str, params: &serde_yaml::Mapping) -> (String, String
}
("DELETE".into(), path, None)
}
+ "indices.analyze" => {
+ let path = if index.is_empty() {
+ "/_analyze".to_string()
+ } else {
+ format!("/{}/_analyze", index)
+ };
+ ("POST".into(), path, body)
+ }
"indices.refresh" => (
"POST".into(),
format!("/{}/_refresh", if index.is_empty() { "*" } else { &index }),
diff --git a/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.yml b/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.yml
new file mode 100644
index 000000000..f6fdd8834
--- /dev/null
+++ b/engine/tests/es-compat-yaml/yaml/indices/indices.analyze/10_analysis_in_force.yml
@@ -0,0 +1,100 @@
+---
+# Issue #1092: an index's analysis configuration is observable over the wire
+# exactly as search applies it. Expected values were checked against
+# OpenSearch 3.9.0, which answers every request below identically.
+setup:
+ - do:
+ indices.create:
+ index: test_analysis
+ body:
+ settings:
+ analysis:
+ analyzer:
+ default:
+ type: english
+ mappings:
+ properties:
+ t:
+ type: text
+ tag:
+ type: keyword
+
+---
+"get_settings reports the declared analysis block":
+ - do:
+ indices.get_settings:
+ index: test_analysis
+
+ - match: { test_analysis.settings.index.analysis.analyzer.default.type: english }
+
+---
+"analyze without an analyzer uses the index default":
+ - do:
+ indices.analyze:
+ index: test_analysis
+ body:
+ text: The running bagels
+
+ - length: { tokens: 2 }
+ - match: { tokens.0.token: run }
+ - match: { tokens.0.start_offset: 4 }
+ - match: { tokens.0.position: 1 }
+ - match: { tokens.1.token: bagel }
+
+---
+"analyze a field uses the analyzer it is indexed with":
+ - do:
+ indices.analyze:
+ index: test_analysis
+ body:
+ field: t
+ text: The running bagels
+
+ - match: { tokens.0.token: run }
+
+ - do:
+ indices.analyze:
+ index: test_analysis
+ body:
+ field: tag
+ text: The running bagels
+
+ - length: { tokens: 1 }
+ - match: { tokens.0.token: The running bagels }
+
+---
+"a named analyzer wins over the index default":
+ - do:
+ indices.analyze:
+ index: test_analysis
+ body:
+ analyzer: standard
+ text: The running bagels
+
+ - length: { tokens: 3 }
+ - match: { tokens.2.token: bagels }
+
+---
+"an ad-hoc chain with filters only runs on the keyword tokenizer":
+ - do:
+ indices.analyze:
+ body:
+ filter: [lowercase]
+ text: Foo Bar
+
+ - length: { tokens: 1 }
+ - match: { tokens.0.token: foo bar }
+
+---
+"an unknown analyzer is a 400":
+ - do:
+ catch: bad_request
+ indices.analyze:
+ index: test_analysis
+ body:
+ analyzer: no_such_analyzer
+ text: x
+
+ # The runner records but does not enforce `catch`; assert the error itself.
+ - match: { error.type: illegal_argument_exception }
+ - match: { status: 400 }