Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(api): address #292 review findings 2-5 and the smaller items

Vincenzo Lombardoffd0cb511cf2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9a9824dee..4a5f42b65 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -165,6 +165,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
that is not an index answers 404 rather than writing a tombstone for a
ghost.
+- **A dynamically-discovered nested object field is no longer permanently
+ opaque to `GET _mapping`/`_field_caps`.** A subfield like `metadata.kind`
+ was already queryable (dotted-path resolution against `_source` doesn't
+ care about the mapping) but had no way to be *discovered* by a client that
+ reads the mapping instead of already knowing the field name — indistinguishable
+ from not existing to Kibana/OSD's own field-list UI. Three compounding gaps,
+ all fixed together: dynamic mapping never recursed into `Value::Object`
+ (`{"type": "object"}` was a terminal, not real ES/OS default behavior);
+ `GET _mapping`/`_field_caps` served the index-creation-time explicit mapping
+ blob verbatim forever, so *any* field discovered dynamically after creation
+ — flat or nested — never surfaced, no matter how many documents arrived; and
+ `_field_caps`'s per-field walk only ever registered top-level names, so a
+ nested object's own children were still invisible even once the mapping
+ correctly described them. Measured end-to-end on `/_memory`'s own backing
+ index: `GET .xerj-memory-{ns}/_field_caps?fields=metadata.kind` returned
+ `{"fields": {}}` before, the real entry after. The cluster-wide
+ `GET /_field_caps` (as opposed to the per-index `GET /{index}/_field_caps`)
+ had the same gaps independently and is fixed the same way — the two no
+ longer disagree about which fields exist.
+ **Known limitation**: this only benefits indices created after this
+ ships. An index (a `/_memory` namespace or otherwise) created before it
+ already has its stored mapping written, and that isn't retroactively
+ edited — existing indices need either re-creation or an explicit
+ `PUT _mapping` to pick up the fix. No automatic migration is included.
+
- **A crafted filename can no longer forge records on the agent progress
stream** (the first rc.15 known issue below). Every externally-controlled
string — in-flight paths, the paths and error text interpolated into human
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 7da2f1e96..88578af4d 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -19032,7 +19032,6 @@ pub struct FieldCapsParams {
/// multi-fields under it like `metadata.kind.keyword`) were never their
/// own `_field_caps` entry, even once the mapping/properties correctly
/// described them.
-#[allow(clippy::too_many_arguments)]
fn register_field_cap_entry(
field: &FieldConfig,
dotted_prefix: &str,
@@ -19087,6 +19086,168 @@ fn register_field_cap_entry(
}
}
+#[cfg(test)]
+mod mapping_merge_tests {
+ //! Direct, non-HTTP coverage for `merge_mapping_properties`,
+ //! `semantic_companion_field_names`, and `register_field_cap_entry` --
+ //! the highest-risk code in the nested-object-mapping fix (recursive,
+ //! and where the one regression found before this landed actually
+ //! lived: semantic_text's own companion field leaking into the
+ //! merge). Complements the HTTP-level end-to-end coverage elsewhere
+ //! (`dynamic_string_mapping_advertises_keyword_multi_field`,
+ //! `api_semantic_bulk_builds_complete_hnsw_and_survives_restart`) with
+ //! fast, precise checks of the merge/registration logic in isolation.
+ use super::*;
+
+ #[test]
+ fn stored_declaration_wins_over_live_for_a_field_both_know_about() {
+ let stored = json!({"price": {"type": "keyword", "ignore_above": 64}})
+ .as_object()
+ .unwrap()
+ .clone();
+ // The live schema inferred a plainer shape for the same field --
+ // the stored declaration's own settings must survive the merge.
+ let live = json!({"price": {"type": "text"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ let merged = merge_mapping_properties(&stored, &live);
+ assert_eq!(merged["price"]["type"], "keyword");
+ assert_eq!(merged["price"]["ignore_above"], 64);
+ }
+
+ #[test]
+ fn live_only_field_is_included_not_dropped() {
+ let stored = json!({"a": {"type": "text"}}).as_object().unwrap().clone();
+ let live = json!({"a": {"type": "text"}, "b": {"type": "long"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ let merged = merge_mapping_properties(&stored, &live);
+ assert_eq!(
+ merged["b"]["type"], "long",
+ "a field only the live schema knows about must still surface: {merged:?}"
+ );
+ }
+
+ #[test]
+ fn bare_object_declaration_picks_up_live_nested_children() {
+ // Exactly /_memory's own historical shape before the memory_api.rs
+ // fix: a field explicitly declared as a bare {"type":"object"}
+ // with no properties of its own.
+ let stored = json!({"metadata": {"type": "object"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ let live = json!({
+ "metadata": {
+ "type": "object",
+ "properties": {
+ "kind": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 256}}}
+ }
+ }
+ })
+ .as_object()
+ .unwrap()
+ .clone();
+ let merged = merge_mapping_properties(&stored, &live);
+ assert_eq!(merged["metadata"]["properties"]["kind"]["type"], "text");
+ assert_eq!(
+ merged["metadata"]["properties"]["kind"]["fields"]["keyword"]["type"],
+ "keyword"
+ );
+ }
+
+ #[test]
+ fn nested_merge_is_recursive_not_just_one_level() {
+ let stored = json!({"a": {"type": "object", "properties": {"b": {"type": "object"}}}})
+ .as_object()
+ .unwrap()
+ .clone();
+ let live = json!({
+ "a": {"type": "object", "properties": {
+ "b": {"type": "object", "properties": {"c": {"type": "long"}}}
+ }}
+ })
+ .as_object()
+ .unwrap()
+ .clone();
+ let merged = merge_mapping_properties(&stored, &live);
+ assert_eq!(
+ merged["a"]["properties"]["b"]["properties"]["c"]["type"],
+ "long"
+ );
+ }
+
+ #[test]
+ fn semantic_companion_field_names_finds_default_and_explicit_target() {
+ let stored = json!({
+ "body": {"type": "semantic_text", "dimensions": 32},
+ "summary": {"type": "semantic_text", "target_field": "summary_emb"},
+ "title": {"type": "text"}
+ })
+ .as_object()
+ .unwrap()
+ .clone();
+ let companions = semantic_companion_field_names(&stored);
+ assert!(companions.contains("body_vector"), "{companions:?}");
+ assert!(companions.contains("summary_emb"), "{companions:?}");
+ assert_eq!(
+ companions.len(),
+ 2,
+ "a non-semantic_text field must not contribute a companion name: {companions:?}"
+ );
+ }
+
+ #[test]
+ fn register_field_cap_entry_registers_nested_object_children_directly() {
+ let mut metadata = FieldConfig::new("metadata", FieldType::Object);
+ metadata
+ .fields
+ .push(FieldConfig::new("kind", FieldType::Text));
+
+ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();
+ register_field_cap_entry(&metadata, "", &json!({}), "*", "idx", &mut fields_map);
+
+ assert!(fields_map.contains_key("metadata"), "{fields_map:?}");
+ assert!(
+ fields_map.contains_key("metadata.kind"),
+ "the nested child itself must get its own dotted entry, not just its own further multi-fields: {fields_map:?}"
+ );
+ }
+
+ #[test]
+ fn register_field_cap_entry_respects_the_fields_filter() {
+ let mut metadata = FieldConfig::new("metadata", FieldType::Object);
+ metadata
+ .fields
+ .push(FieldConfig::new("kind", FieldType::Text));
+ metadata
+ .fields
+ .push(FieldConfig::new("topic", FieldType::Text));
+
+ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();
+ register_field_cap_entry(
+ &metadata,
+ "",
+ &json!({}),
+ "metadata.kind",
+ "idx",
+ &mut fields_map,
+ );
+
+ assert!(fields_map.contains_key("metadata.kind"), "{fields_map:?}");
+ assert!(
+ !fields_map.contains_key("metadata.topic"),
+ "a filter naming one nested child must not also register its sibling: {fields_map:?}"
+ );
+ assert!(
+ !fields_map.contains_key("metadata"),
+ "a filter naming only the nested child must not also register the parent: {fields_map:?}"
+ );
+ }
+}
+
pub async fn field_caps(
State(state): State<AppState>,
Path(index): Path<String>,
@@ -19583,6 +19744,152 @@ mod flat_object_field_caps_tests {
);
}
+ /// The cluster-wide `GET /_field_caps` (routed to `global_field_caps`)
+ /// used to be a near-duplicate of the per-index `field_caps` body from
+ /// *before* the nested-object-mapping fix: same stale cached-mapping
+ /// read, same top-level-only field loop, and it never called
+ /// `collect_multi_fields` at all -- so `GET /{index}/_field_caps` and
+ /// `GET /_field_caps` could (and did) disagree about which fields
+ /// exist. Proves both now agree, on the exact nested-object shape the
+ /// whole fix is about.
+ #[tokio::test]
+ async fn global_field_caps_agrees_with_per_index_field_caps_on_nested_objects() {
+ let router = crate::router::build_es_compat_router(test_state());
+
+ let write = router
+ .clone()
+ .oneshot(
+ Request::put("/global-caps-e2e/_doc/1")
+ .header("content-type", "application/json")
+ .body(Body::from(
+ json!({"metadata": {"kind": "preference", "topic": "language"}})
+ .to_string(),
+ ))
+ .expect("request"),
+ )
+ .await
+ .expect("write response");
+ assert!(write.status().is_success());
+
+ let per_index_resp = router
+ .clone()
+ .oneshot(
+ Request::get("/global-caps-e2e/_field_caps?fields=*")
+ .body(Body::empty())
+ .expect("request"),
+ )
+ .await
+ .expect("per-index field_caps response");
+ assert!(per_index_resp.status().is_success());
+ let per_index_bytes = to_bytes(per_index_resp.into_body(), usize::MAX)
+ .await
+ .expect("response bytes");
+ let per_index: Value = serde_json::from_slice(&per_index_bytes).expect("JSON response");
+
+ let global_resp = router
+ .oneshot(
+ Request::get("/_field_caps?fields=*")
+ .body(Body::empty())
+ .expect("request"),
+ )
+ .await
+ .expect("global field_caps response");
+ assert!(global_resp.status().is_success());
+ let global_bytes = to_bytes(global_resp.into_body(), usize::MAX)
+ .await
+ .expect("response bytes");
+ let global: Value = serde_json::from_slice(&global_bytes).expect("JSON response");
+
+ for field in [
+ "metadata",
+ "metadata.kind",
+ "metadata.kind.keyword",
+ "metadata.topic",
+ ] {
+ assert!(
+ per_index["fields"].get(field).is_some(),
+ "sanity: per-index _field_caps must itself have {field}, got: {per_index}"
+ );
+ assert_eq!(
+ global["fields"][field], per_index["fields"][field],
+ "global and per-index _field_caps disagree on '{field}' -- global: {global}, per-index: {per_index}"
+ );
+ }
+ }
+
+ /// The exact case a reviewer asked for on the original nested-object-
+ /// mapping PR: a SECOND document introducing a NEW nested key under a
+ /// top-level field the schema already knows about. Before this test's
+ /// fix, dynamic mapping gated purely on whether `metadata` (the
+ /// top-level key) already existed -- once the first document
+ /// registered it (from `{"kind": ...}`), `dynamic_field_config` was
+ /// never called for it again, so a second document's `metadata.project`
+ /// was permanently invisible to `_mapping`/`_field_caps`, no matter how
+ /// many more documents arrived, despite staying perfectly queryable via
+ /// `_source` dotted-path resolution the whole time -- the exact
+ /// "queryable but invisible" symptom the whole fix is about, one level
+ /// down. `/_memory`'s own `metadata` field is exactly this shape (its
+ /// keys are caller-defined and legitimately vary memory to memory), so
+ /// this reproduces it through the real `/_memory` API rather than a
+ /// synthetic index.
+ #[tokio::test]
+ async fn a_later_document_introducing_a_new_nested_key_is_not_permanently_invisible() {
+ let router = crate::router::build_es_compat_router(test_state());
+
+ let store_one = router
+ .clone()
+ .oneshot(
+ Request::post("/_memory/repro")
+ .header("content-type", "application/json")
+ .body(Body::from(
+ json!({"text": "first", "metadata": {"kind": "preference"}}).to_string(),
+ ))
+ .expect("request"),
+ )
+ .await
+ .expect("first store response");
+ assert!(store_one.status().is_success());
+
+ let store_two = router
+ .clone()
+ .oneshot(
+ Request::post("/_memory/repro")
+ .header("content-type", "application/json")
+ .body(Body::from(
+ json!({"text": "second", "metadata": {"project": "xerj"}}).to_string(),
+ ))
+ .expect("request"),
+ )
+ .await
+ .expect("second store response");
+ assert!(store_two.status().is_success());
+
+ let caps_resp = router
+ .oneshot(
+ Request::get("/.xerj-memory-repro/_field_caps?fields=*")
+ .body(Body::empty())
+ .expect("request"),
+ )
+ .await
+ .expect("field_caps response");
+ assert!(caps_resp.status().is_success());
+ let bytes = to_bytes(caps_resp.into_body(), usize::MAX)
+ .await
+ .expect("response bytes");
+ let caps: Value = serde_json::from_slice(&bytes).expect("JSON response");
+
+ assert!(
+ caps["fields"].get("metadata.kind").is_some(),
+ "the FIRST document's nested key must still be visible: {caps}"
+ );
+ assert!(
+ caps["fields"].get("metadata.project").is_some(),
+ "the SECOND document's nested key, introduced after 'metadata' was already \
+ a known top-level field, must also become visible -- not permanently invisible \
+ just because it lost the race to be first: {caps}"
+ );
+ }
+
/// Regression test for a real bug found live: `GET _mapping` correctly
/// round-trips `float`/`integer` (it returns the stored mapping JSON
/// verbatim), but `_field_caps` derived its reported type from the
@@ -29528,7 +29835,7 @@ pub async fn global_field_caps(
let index_names: Vec<String> = indices.iter().map(|i| i.name.clone()).collect();
let fields_filter = params.fields.as_deref().unwrap_or("*");
- let mut fields_map: HashMap<String, HashMap<String, Value>> = HashMap::new();
+ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();
for index_info in &indices {
let idx = match state.engine.get_index(&index_info.name) {
@@ -29547,60 +29854,81 @@ pub async fn global_field_caps(
.get(index_info.name.as_str())
.map(|v| v.clone())
.unwrap_or(Value::Null);
+ let mut live_properties = schema_to_es_properties(&schema);
let properties = if stored_mapping.is_null() {
- Value::Object(schema_to_es_properties(&schema))
+ Value::Object(live_properties)
} else {
- stored_mapping
+ // Same merge as `field_caps` (per-index) -- without it, a field
+ // discovered dynamically since this index was created (flat or
+ // nested, at any depth) is invisible here too, and this
+ // cluster-wide endpoint would silently disagree with the
+ // per-index one about which fields exist.
+ let stored_properties = stored_mapping
.get("properties")
+ .and_then(|p| p.as_object())
.cloned()
- .unwrap_or_else(|| json!({}))
+ .unwrap_or_default();
+ let companions = semantic_companion_field_names(&stored_properties);
+ live_properties.retain(|k, _| !companions.contains(k));
+ Value::Object(merge_mapping_properties(
+ &stored_properties,
+ &live_properties,
+ ))
};
for field in &schema.fields {
+ register_field_cap_entry(
+ field,
+ "",
+ &properties,
+ fields_filter,
+ &index_info.name,
+ &mut fields_map,
+ );
+ }
+
+ // Multi-fields (`"fields": {"keyword": {...}}`) -- same #209
+ // mechanism as `field_caps` (per-index); this cluster-wide form
+ // never had it at all.
+ let mut multi_fields = Vec::new();
+ collect_multi_fields(&properties, "", &mut multi_fields);
+ for (name, es_type) in multi_fields {
if fields_filter != "*" {
- // Support comma-separated field list and simple wildcard suffix.
let matches = fields_filter
.split(',')
- .any(|f| field_name_matches(&field.name, f.trim()));
+ .any(|f| source_field_matches(&name, f.trim()));
if !matches {
continue;
}
}
-
- let native_es_type = native_type_to_es_str(&field.field_type);
- let es_type = declared_flattened_type(&properties, &field.name)
- .or_else(|| declared_numeric_type(&properties, &field.name))
- .unwrap_or(native_es_type);
- let searchable = field.is_searchable();
- let aggregatable = field.is_aggregatable();
-
- let type_entry = fields_map
- .entry(field.name.clone())
- .or_default()
- .entry(es_type.to_string())
- .or_insert_with(|| {
- json!({
- "type": es_type,
- "searchable": searchable,
- "aggregatable": aggregatable,
- "indices": []
- })
- });
-
- // Append this index to the indices list for this type entry.
+ let (searchable, aggregatable) = match es_type.as_str() {
+ "text" => (true, false),
+ _ => (true, true),
+ };
+ let type_map = fields_map.entry(name).or_default();
+ let type_entry = type_map.entry(es_type.clone()).or_insert_with(|| {
+ json!({
+ "type": es_type,
+ "searchable": searchable,
+ "aggregatable": aggregatable,
+ "indices": []
+ })
+ });
if let Some(arr) = type_entry["indices"].as_array_mut() {
- arr.push(Value::String(index_info.name.clone()));
+ if !arr
+ .iter()
+ .any(|v| v.as_str() == Some(index_info.name.as_str()))
+ {
+ arr.push(Value::String(index_info.name.clone()));
+ }
}
}
}
- // Convert HashMap<String, HashMap<String, Value>> to the ES format.
let fields_val: Value = Value::Object(
fields_map
.into_iter()
- .map(|(field_name, type_map)| {
- (field_name, Value::Object(type_map.into_iter().collect()))
- })
+ .map(|(field_name, type_map)| (field_name, Value::Object(type_map)))
.collect(),
);
diff --git a/engine/crates/xerj-api/src/memory_api.rs b/engine/crates/xerj-api/src/memory_api.rs
index 09eebe3d8..5020d4bdb 100644
--- a/engine/crates/xerj-api/src/memory_api.rs
+++ b/engine/crates/xerj-api/src/memory_api.rs
@@ -165,7 +165,8 @@ fn index_exists(state: &AppState, index: &str) -> bool {
/// Ensure the backing index exists with an appropriate mapping. Created lazily
/// on first store. When the first stored memory carries a vector, the index is
/// created with a `dense_vector` field sized to that vector so kNN recall works
-/// out of the box; text and metadata are always mapped.
+/// out of the box. `text` is always explicitly mapped; `metadata` deliberately
+/// is not -- see the comment where `properties` is built below.
async fn ensure_backing_index(
state: &AppState,
index: &str,
@@ -196,6 +197,16 @@ async fn ensure_backing_index(
// into nested objects (see xerj-engine's `dynamic_field_config`), so
// `metadata.kind`/`metadata.topic`/etc. become real, `_mapping`-visible
// fields instead of being permanently opaque.
+ //
+ // KNOWN LIMITATION, deliberately not addressed here: this only helps
+ // namespaces whose backing index is created AFTER this change ships. A
+ // `.xerj-memory-*` index created before it already has `metadata`
+ // declared as a bare `{"type": "object"}` in ITS OWN stored mapping --
+ // that declaration was written once, at that index's creation time,
+ // and is not retroactively edited by a code change. No migration for
+ // already-existing namespaces is included in this change; one would
+ // need to either re-create the index or explicitly `PUT _mapping` a
+ // corrected `metadata.properties` shape onto it by hand.
let mut properties = json!({
"text": { "type": "semantic_text", "dimensions": 384 },
"stored_at": { "type": "long" }
diff --git a/engine/crates/xerj-common/src/types.rs b/engine/crates/xerj-common/src/types.rs
index 5c9b6ff1d..f4fec2323 100644
--- a/engine/crates/xerj-common/src/types.rs
+++ b/engine/crates/xerj-common/src/types.rs
@@ -420,6 +420,25 @@ impl FieldConfig {
pub fn is_aggregatable(&self) -> bool {
self.options.doc_values && !matches!(self.field_type, FieldType::Text | FieldType::Binary)
}
+
+ /// This field plus every nested child, recursively -- both real
+ /// `properties` children of an `Object`/`Nested` field and leaf
+ /// multi-fields under `fields` live in the same `fields` vec (see the
+ /// dual meaning documented on that field above), so one count covers
+ /// both. `Schema::field_count()` sums this over every top-level field
+ /// instead of just counting `self.fields.len()`, since a single
+ /// top-level `Object` field can carry unboundedly many nested
+ /// children once dynamic mapping recurses into them (see
+ /// `xerj-engine`'s `dynamic_field_config_at_depth`) -- an
+ /// un-recursive count would let nested fields bypass every
+ /// `max_fields_per_index` mapping-explosion guard entirely.
+ pub fn total_field_count(&self) -> usize {
+ 1 + self
+ .fields
+ .iter()
+ .map(FieldConfig::total_field_count)
+ .sum::<usize>()
+ }
}
// ═════════════════════════════════════════════════════════════════════════════
@@ -455,6 +474,14 @@ impl Schema {
self.fields.iter().find(|f| f.name == name)
}
+ /// Mutable counterpart to [`Self::field`] -- lets a caller merge new
+ /// dynamically-discovered nested children into an already-registered
+ /// top-level field's own `fields` in place, instead of only ever being
+ /// able to add brand-new top-level fields via [`Self::add_field`].
+ pub fn field_mut(&mut self, name: &str) -> Option<&mut FieldConfig> {
+ self.fields.iter_mut().find(|f| f.name == name)
+ }
+
/// Return `true` if a field with this name already exists.
pub fn has_field(&self, name: &str) -> bool {
self.field(name).is_some()
@@ -476,9 +503,17 @@ impl Schema {
Ok(())
}
- /// Total number of fields (counting only top-level fields).
+ /// Total number of fields, counting every nested child at every depth
+ /// -- not just top-level fields. Every `max_fields_per_index`
+ /// mapping-explosion guard in the engine (both the dynamic-ingest path
+ /// and the explicit `PUT _mapping`/`POST .../evolve` path, #76 S5-5)
+ /// calls this to bound real schema growth; counting only
+ /// `self.fields.len()` would let a single top-level `Object` field
+ /// carry unboundedly many nested children uncounted, once dynamic
+ /// mapping recurses into them (see `xerj-engine`'s
+ /// `dynamic_field_config_at_depth`).
pub fn field_count(&self) -> usize {
- self.fields.len()
+ self.fields.iter().map(FieldConfig::total_field_count).sum()
}
}