AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(api): address #292 review findings 2-5 and the smaller items
Vincenzo Lombardoffd0cb511cf2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9a9824dee..4a5f42b65 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -165,6 +165,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0that is not an index answers 404 rather than writing a tombstone for aghost.+- **A dynamically-discovered nested object field is no longer permanently+ opaque to `GET _mapping`/`_field_caps`.** A subfield like `metadata.kind`+ was already queryable (dotted-path resolution against `_source` doesn't+ care about the mapping) but had no way to be *discovered* by a client that+ reads the mapping instead of already knowing the field name — indistinguishable+ from not existing to Kibana/OSD's own field-list UI. Three compounding gaps,+ all fixed together: dynamic mapping never recursed into `Value::Object`+ (`{"type": "object"}` was a terminal, not real ES/OS default behavior);+ `GET _mapping`/`_field_caps` served the index-creation-time explicit mapping+ blob verbatim forever, so *any* field discovered dynamically after creation+ — flat or nested — never surfaced, no matter how many documents arrived; and+ `_field_caps`'s per-field walk only ever registered top-level names, so a+ nested object's own children were still invisible even once the mapping+ correctly described them. Measured end-to-end on `/_memory`'s own backing+ index: `GET .xerj-memory-{ns}/_field_caps?fields=metadata.kind` returned+ `{"fields": {}}` before, the real entry after. The cluster-wide+ `GET /_field_caps` (as opposed to the per-index `GET /{index}/_field_caps`)+ had the same gaps independently and is fixed the same way — the two no+ longer disagree about which fields exist.+ **Known limitation**: this only benefits indices created after this+ ships. An index (a `/_memory` namespace or otherwise) created before it+ already has its stored mapping written, and that isn't retroactively+ edited — existing indices need either re-creation or an explicit+ `PUT _mapping` to pick up the fix. No automatic migration is included.+- **A crafted filename can no longer forge records on the agent progressstream** (the first rc.15 known issue below). Every externally-controlledstring — in-flight paths, the paths and error text interpolated into humandiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 7da2f1e96..88578af4d 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -19032,7 +19032,6 @@ pub struct FieldCapsParams {/// multi-fields under it like `metadata.kind.keyword`) were never their/// own `_field_caps` entry, even once the mapping/properties correctly/// described them.-#[allow(clippy::too_many_arguments)]fn register_field_cap_entry(field: &FieldConfig,dotted_prefix: &str,@@ -19087,6 +19086,168 @@ fn register_field_cap_entry(}}+#[cfg(test)]+mod mapping_merge_tests {+ //! Direct, non-HTTP coverage for `merge_mapping_properties`,+ //! `semantic_companion_field_names`, and `register_field_cap_entry` --+ //! the highest-risk code in the nested-object-mapping fix (recursive,+ //! and where the one regression found before this landed actually+ //! lived: semantic_text's own companion field leaking into the+ //! merge). Complements the HTTP-level end-to-end coverage elsewhere+ //! (`dynamic_string_mapping_advertises_keyword_multi_field`,+ //! `api_semantic_bulk_builds_complete_hnsw_and_survives_restart`) with+ //! fast, precise checks of the merge/registration logic in isolation.+ use super::*;++ #[test]+ fn stored_declaration_wins_over_live_for_a_field_both_know_about() {+ let stored = json!({"price": {"type": "keyword", "ignore_above": 64}})+ .as_object()+ .unwrap()+ .clone();+ // The live schema inferred a plainer shape for the same field --+ // the stored declaration's own settings must survive the merge.+ let live = json!({"price": {"type": "text"}})+ .as_object()+ .unwrap()+ .clone();+ let merged = merge_mapping_properties(&stored, &live);+ assert_eq!(merged["price"]["type"], "keyword");+ assert_eq!(merged["price"]["ignore_above"], 64);+ }++ #[test]+ fn live_only_field_is_included_not_dropped() {+ let stored = json!({"a": {"type": "text"}}).as_object().unwrap().clone();+ let live = json!({"a": {"type": "text"}, "b": {"type": "long"}})+ .as_object()+ .unwrap()+ .clone();+ let merged = merge_mapping_properties(&stored, &live);+ assert_eq!(+ merged["b"]["type"], "long",+ "a field only the live schema knows about must still surface: {merged:?}"+ );+ }++ #[test]+ fn bare_object_declaration_picks_up_live_nested_children() {+ // Exactly /_memory's own historical shape before the memory_api.rs+ // fix: a field explicitly declared as a bare {"type":"object"}+ // with no properties of its own.+ let stored = json!({"metadata": {"type": "object"}})+ .as_object()+ .unwrap()+ .clone();+ let live = json!({+ "metadata": {+ "type": "object",+ "properties": {+ "kind": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 256}}}+ }+ }+ })+ .as_object()+ .unwrap()+ .clone();+ let merged = merge_mapping_properties(&stored, &live);+ assert_eq!(merged["metadata"]["properties"]["kind"]["type"], "text");+ assert_eq!(+ merged["metadata"]["properties"]["kind"]["fields"]["keyword"]["type"],+ "keyword"+ );+ }++ #[test]+ fn nested_merge_is_recursive_not_just_one_level() {+ let stored = json!({"a": {"type": "object", "properties": {"b": {"type": "object"}}}})+ .as_object()+ .unwrap()+ .clone();+ let live = json!({+ "a": {"type": "object", "properties": {+ "b": {"type": "object", "properties": {"c": {"type": "long"}}}+ }}+ })+ .as_object()+ .unwrap()+ .clone();+ let merged = merge_mapping_properties(&stored, &live);+ assert_eq!(+ merged["a"]["properties"]["b"]["properties"]["c"]["type"],+ "long"+ );+ }++ #[test]+ fn semantic_companion_field_names_finds_default_and_explicit_target() {+ let stored = json!({+ "body": {"type": "semantic_text", "dimensions": 32},+ "summary": {"type": "semantic_text", "target_field": "summary_emb"},+ "title": {"type": "text"}+ })+ .as_object()+ .unwrap()+ .clone();+ let companions = semantic_companion_field_names(&stored);+ assert!(companions.contains("body_vector"), "{companions:?}");+ assert!(companions.contains("summary_emb"), "{companions:?}");+ assert_eq!(+ companions.len(),+ 2,+ "a non-semantic_text field must not contribute a companion name: {companions:?}"+ );+ }++ #[test]+ fn register_field_cap_entry_registers_nested_object_children_directly() {+ let mut metadata = FieldConfig::new("metadata", FieldType::Object);+ metadata+ .fields+ .push(FieldConfig::new("kind", FieldType::Text));++ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();+ register_field_cap_entry(&metadata, "", &json!({}), "*", "idx", &mut fields_map);++ assert!(fields_map.contains_key("metadata"), "{fields_map:?}");+ assert!(+ fields_map.contains_key("metadata.kind"),+ "the nested child itself must get its own dotted entry, not just its own further multi-fields: {fields_map:?}"+ );+ }++ #[test]+ fn register_field_cap_entry_respects_the_fields_filter() {+ let mut metadata = FieldConfig::new("metadata", FieldType::Object);+ metadata+ .fields+ .push(FieldConfig::new("kind", FieldType::Text));+ metadata+ .fields+ .push(FieldConfig::new("topic", FieldType::Text));++ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();+ register_field_cap_entry(+ &metadata,+ "",+ &json!({}),+ "metadata.kind",+ "idx",+ &mut fields_map,+ );++ assert!(fields_map.contains_key("metadata.kind"), "{fields_map:?}");+ assert!(+ !fields_map.contains_key("metadata.topic"),+ "a filter naming one nested child must not also register its sibling: {fields_map:?}"+ );+ assert!(+ !fields_map.contains_key("metadata"),+ "a filter naming only the nested child must not also register the parent: {fields_map:?}"+ );+ }+}+pub async fn field_caps(State(state): State<AppState>,Path(index): Path<String>,@@ -19583,6 +19744,152 @@ mod flat_object_field_caps_tests {);}+ /// The cluster-wide `GET /_field_caps` (routed to `global_field_caps`)+ /// used to be a near-duplicate of the per-index `field_caps` body from+ /// *before* the nested-object-mapping fix: same stale cached-mapping+ /// read, same top-level-only field loop, and it never called+ /// `collect_multi_fields` at all -- so `GET /{index}/_field_caps` and+ /// `GET /_field_caps` could (and did) disagree about which fields+ /// exist. Proves both now agree, on the exact nested-object shape the+ /// whole fix is about.+ #[tokio::test]+ async fn global_field_caps_agrees_with_per_index_field_caps_on_nested_objects() {+ let router = crate::router::build_es_compat_router(test_state());++ let write = router+ .clone()+ .oneshot(+ Request::put("/global-caps-e2e/_doc/1")+ .header("content-type", "application/json")+ .body(Body::from(+ json!({"metadata": {"kind": "preference", "topic": "language"}})+ .to_string(),+ ))+ .expect("request"),+ )+ .await+ .expect("write response");+ assert!(write.status().is_success());++ let per_index_resp = router+ .clone()+ .oneshot(+ Request::get("/global-caps-e2e/_field_caps?fields=*")+ .body(Body::empty())+ .expect("request"),+ )+ .await+ .expect("per-index field_caps response");+ assert!(per_index_resp.status().is_success());+ let per_index_bytes = to_bytes(per_index_resp.into_body(), usize::MAX)+ .await+ .expect("response bytes");+ let per_index: Value = serde_json::from_slice(&per_index_bytes).expect("JSON response");++ let global_resp = router+ .oneshot(+ Request::get("/_field_caps?fields=*")+ .body(Body::empty())+ .expect("request"),+ )+ .await+ .expect("global field_caps response");+ assert!(global_resp.status().is_success());+ let global_bytes = to_bytes(global_resp.into_body(), usize::MAX)+ .await+ .expect("response bytes");+ let global: Value = serde_json::from_slice(&global_bytes).expect("JSON response");++ for field in [+ "metadata",+ "metadata.kind",+ "metadata.kind.keyword",+ "metadata.topic",+ ] {+ assert!(+ per_index["fields"].get(field).is_some(),+ "sanity: per-index _field_caps must itself have {field}, got: {per_index}"+ );+ assert_eq!(+ global["fields"][field], per_index["fields"][field],+ "global and per-index _field_caps disagree on '{field}' -- global: {global}, per-index: {per_index}"+ );+ }+ }++ /// The exact case a reviewer asked for on the original nested-object-+ /// mapping PR: a SECOND document introducing a NEW nested key under a+ /// top-level field the schema already knows about. Before this test's+ /// fix, dynamic mapping gated purely on whether `metadata` (the+ /// top-level key) already existed -- once the first document+ /// registered it (from `{"kind": ...}`), `dynamic_field_config` was+ /// never called for it again, so a second document's `metadata.project`+ /// was permanently invisible to `_mapping`/`_field_caps`, no matter how+ /// many more documents arrived, despite staying perfectly queryable via+ /// `_source` dotted-path resolution the whole time -- the exact+ /// "queryable but invisible" symptom the whole fix is about, one level+ /// down. `/_memory`'s own `metadata` field is exactly this shape (its+ /// keys are caller-defined and legitimately vary memory to memory), so+ /// this reproduces it through the real `/_memory` API rather than a+ /// synthetic index.+ #[tokio::test]+ async fn a_later_document_introducing_a_new_nested_key_is_not_permanently_invisible() {+ let router = crate::router::build_es_compat_router(test_state());++ let store_one = router+ .clone()+ .oneshot(+ Request::post("/_memory/repro")+ .header("content-type", "application/json")+ .body(Body::from(+ json!({"text": "first", "metadata": {"kind": "preference"}}).to_string(),+ ))+ .expect("request"),+ )+ .await+ .expect("first store response");+ assert!(store_one.status().is_success());++ let store_two = router+ .clone()+ .oneshot(+ Request::post("/_memory/repro")+ .header("content-type", "application/json")+ .body(Body::from(+ json!({"text": "second", "metadata": {"project": "xerj"}}).to_string(),+ ))+ .expect("request"),+ )+ .await+ .expect("second store response");+ assert!(store_two.status().is_success());++ let caps_resp = router+ .oneshot(+ Request::get("/.xerj-memory-repro/_field_caps?fields=*")+ .body(Body::empty())+ .expect("request"),+ )+ .await+ .expect("field_caps response");+ assert!(caps_resp.status().is_success());+ let bytes = to_bytes(caps_resp.into_body(), usize::MAX)+ .await+ .expect("response bytes");+ let caps: Value = serde_json::from_slice(&bytes).expect("JSON response");++ assert!(+ caps["fields"].get("metadata.kind").is_some(),+ "the FIRST document's nested key must still be visible: {caps}"+ );+ assert!(+ caps["fields"].get("metadata.project").is_some(),+ "the SECOND document's nested key, introduced after 'metadata' was already \+ a known top-level field, must also become visible -- not permanently invisible \+ just because it lost the race to be first: {caps}"+ );+ }+/// Regression test for a real bug found live: `GET _mapping` correctly/// round-trips `float`/`integer` (it returns the stored mapping JSON/// verbatim), but `_field_caps` derived its reported type from the@@ -29528,7 +29835,7 @@ pub async fn global_field_caps(let index_names: Vec<String> = indices.iter().map(|i| i.name.clone()).collect();let fields_filter = params.fields.as_deref().unwrap_or("*");- let mut fields_map: HashMap<String, HashMap<String, Value>> = HashMap::new();+ let mut fields_map: HashMap<String, serde_json::Map<String, Value>> = HashMap::new();for index_info in &indices {let idx = match state.engine.get_index(&index_info.name) {@@ -29547,60 +29854,81 @@ pub async fn global_field_caps(.get(index_info.name.as_str()).map(|v| v.clone()).unwrap_or(Value::Null);+ let mut live_properties = schema_to_es_properties(&schema);let properties = if stored_mapping.is_null() {- Value::Object(schema_to_es_properties(&schema))+ Value::Object(live_properties)} else {- stored_mapping+ // Same merge as `field_caps` (per-index) -- without it, a field+ // discovered dynamically since this index was created (flat or+ // nested, at any depth) is invisible here too, and this+ // cluster-wide endpoint would silently disagree with the+ // per-index one about which fields exist.+ let stored_properties = stored_mapping.get("properties")+ .and_then(|p| p.as_object()).cloned()- .unwrap_or_else(|| json!({}))+ .unwrap_or_default();+ let companions = semantic_companion_field_names(&stored_properties);+ live_properties.retain(|k, _| !companions.contains(k));+ Value::Object(merge_mapping_properties(+ &stored_properties,+ &live_properties,+ ))};for field in &schema.fields {+ register_field_cap_entry(+ field,+ "",+ &properties,+ fields_filter,+ &index_info.name,+ &mut fields_map,+ );+ }++ // Multi-fields (`"fields": {"keyword": {...}}`) -- same #209+ // mechanism as `field_caps` (per-index); this cluster-wide form+ // never had it at all.+ let mut multi_fields = Vec::new();+ collect_multi_fields(&properties, "", &mut multi_fields);+ for (name, es_type) in multi_fields {if fields_filter != "*" {- // Support comma-separated field list and simple wildcard suffix.let matches = fields_filter.split(',')- .any(|f| field_name_matches(&field.name, f.trim()));+ .any(|f| source_field_matches(&name, f.trim()));if !matches {continue;}}-- let native_es_type = native_type_to_es_str(&field.field_type);- let es_type = declared_flattened_type(&properties, &field.name)- .or_else(|| declared_numeric_type(&properties, &field.name))- .unwrap_or(native_es_type);- let searchable = field.is_searchable();- let aggregatable = field.is_aggregatable();-- let type_entry = fields_map- .entry(field.name.clone())- .or_default()- .entry(es_type.to_string())- .or_insert_with(|| {- json!({- "type": es_type,- "searchable": searchable,- "aggregatable": aggregatable,- "indices": []- })- });-- // Append this index to the indices list for this type entry.+ let (searchable, aggregatable) = match es_type.as_str() {+ "text" => (true, false),+ _ => (true, true),+ };+ let type_map = fields_map.entry(name).or_default();+ let type_entry = type_map.entry(es_type.clone()).or_insert_with(|| {+ json!({+ "type": es_type,+ "searchable": searchable,+ "aggregatable": aggregatable,+ "indices": []+ })+ });if let Some(arr) = type_entry["indices"].as_array_mut() {- arr.push(Value::String(index_info.name.clone()));+ if !arr+ .iter()+ .any(|v| v.as_str() == Some(index_info.name.as_str()))+ {+ arr.push(Value::String(index_info.name.clone()));+ }}}}- // Convert HashMap<String, HashMap<String, Value>> to the ES format.let fields_val: Value = Value::Object(fields_map.into_iter()- .map(|(field_name, type_map)| {- (field_name, Value::Object(type_map.into_iter().collect()))- })+ .map(|(field_name, type_map)| (field_name, Value::Object(type_map))).collect(),);diff --git a/engine/crates/xerj-api/src/memory_api.rs b/engine/crates/xerj-api/src/memory_api.rsindex 09eebe3d8..5020d4bdb 100644--- a/engine/crates/xerj-api/src/memory_api.rs+++ b/engine/crates/xerj-api/src/memory_api.rs@@ -165,7 +165,8 @@ fn index_exists(state: &AppState, index: &str) -> bool {/// Ensure the backing index exists with an appropriate mapping. Created lazily/// on first store. When the first stored memory carries a vector, the index is/// created with a `dense_vector` field sized to that vector so kNN recall works-/// out of the box; text and metadata are always mapped.+/// out of the box. `text` is always explicitly mapped; `metadata` deliberately+/// is not -- see the comment where `properties` is built below.async fn ensure_backing_index(state: &AppState,index: &str,@@ -196,6 +197,16 @@ async fn ensure_backing_index(// into nested objects (see xerj-engine's `dynamic_field_config`), so// `metadata.kind`/`metadata.topic`/etc. become real, `_mapping`-visible// fields instead of being permanently opaque.+ //+ // KNOWN LIMITATION, deliberately not addressed here: this only helps+ // namespaces whose backing index is created AFTER this change ships. A+ // `.xerj-memory-*` index created before it already has `metadata`+ // declared as a bare `{"type": "object"}` in ITS OWN stored mapping --+ // that declaration was written once, at that index's creation time,+ // and is not retroactively edited by a code change. No migration for+ // already-existing namespaces is included in this change; one would+ // need to either re-create the index or explicitly `PUT _mapping` a+ // corrected `metadata.properties` shape onto it by hand.let mut properties = json!({"text": { "type": "semantic_text", "dimensions": 384 },"stored_at": { "type": "long" }diff --git a/engine/crates/xerj-common/src/types.rs b/engine/crates/xerj-common/src/types.rsindex 5c9b6ff1d..f4fec2323 100644--- a/engine/crates/xerj-common/src/types.rs+++ b/engine/crates/xerj-common/src/types.rs@@ -420,6 +420,25 @@ impl FieldConfig {pub fn is_aggregatable(&self) -> bool {self.options.doc_values && !matches!(self.field_type, FieldType::Text | FieldType::Binary)}++ /// This field plus every nested child, recursively -- both real+ /// `properties` children of an `Object`/`Nested` field and leaf+ /// multi-fields under `fields` live in the same `fields` vec (see the+ /// dual meaning documented on that field above), so one count covers+ /// both. `Schema::field_count()` sums this over every top-level field+ /// instead of just counting `self.fields.len()`, since a single+ /// top-level `Object` field can carry unboundedly many nested+ /// children once dynamic mapping recurses into them (see+ /// `xerj-engine`'s `dynamic_field_config_at_depth`) -- an+ /// un-recursive count would let nested fields bypass every+ /// `max_fields_per_index` mapping-explosion guard entirely.+ pub fn total_field_count(&self) -> usize {+ 1 + self+ .fields+ .iter()+ .map(FieldConfig::total_field_count)+ .sum::<usize>()+ }}// ═════════════════════════════════════════════════════════════════════════════@@ -455,6 +474,14 @@ impl Schema {self.fields.iter().find(|f| f.name == name)}+ /// Mutable counterpart to [`Self::field`] -- lets a caller merge new+ /// dynamically-discovered nested children into an already-registered+ /// top-level field's own `fields` in place, instead of only ever being+ /// able to add brand-new top-level fields via [`Self::add_field`].+ pub fn field_mut(&mut self, name: &str) -> Option<&mut FieldConfig> {+ self.fields.iter_mut().find(|f| f.name == name)+ }+/// Return `true` if a field with this name already exists.pub fn has_field(&self, name: &str) -> bool {self.field(name).is_some()@@ -476,9 +503,17 @@ impl Schema {Ok(())}- /// Total number of fields (counting only top-level fields).+ /// Total number of fields, counting every nested child at every depth+ /// -- not just top-level fields. Every `max_fields_per_index`+ /// mapping-explosion guard in the engine (both the dynamic-ingest path+ /// and the explicit `PUT _mapping`/`POST .../evolve` path, #76 S5-5)+ /// calls this to bound real schema growth; counting only+ /// `self.fields.len()` would let a single top-level `Object` field+ /// carry unboundedly many nested children uncounted, once dynamic+ /// mapping recurses into them (see `xerj-engine`'s+ /// `dynamic_field_config_at_depth`).pub fn field_count(&self) -> usize {- self.fields.len()+ self.fields.iter().map(FieldConfig::total_field_count).sum()}}