AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): stop JSON array files from wedging ingest (#2131)
xerj-org533066b94ce4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2ee6fa4ed..3ea8459a4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- `xerj autoindex` no longer wedges or runs out of memory on a JSON object+ made of several top-level arrays of records (#1856, #1978). When the+ largest array is longer than a config document (32 records), each top-level+ array of objects becomes its own dataset instead of being copied onto every+ row of the largest one. A shared value over 4 KiB is stored once. A data+ column that would be `semantic_text` falls back to plain text — still+ searchable — when the real record count exceeds 100,000 or the elected text+ exceeds 8 MiB, and the plan says so, including `--no-semantic`. The+ pre-index budget counts records in `.json` files from 256 KiB up to 64 MiB+ and can refuse on a structural memory ceiling. That ceiling is not a+ measured store ratio, and no new disk ratio was invented for `.json`.+- `POST /{index}/_refresh` and autoindex `finalize-refresh` no longer waitforever when a shard flush cannot get a finalize-gate or flush-semaphorepermit. Admission gives up after `engine.flush_admit_timeout_secs`diff --git a/engine/crates/xerj-autoindex/src/budget.rs b/engine/crates/xerj-autoindex/src/budget.rsindex 3db31f8ef..1da039379 100644--- a/engine/crates/xerj-autoindex/src/budget.rs+++ b/engine/crates/xerj-autoindex/src/budget.rs@@ -5,7 +5,13 @@//! with the room left under the node's flood-stage watermark, and — when the//! default plan does not fit — names a smaller plan and waits for an explicit//! yes. It never invents a ratio: a kind with no measured calibration is-//! reported as unmeasured, and an unmeasured estimate does not refuse a run.+//! reported as unmeasured, and an unmeasured disk estimate does not refuse a+//! run.+//!+//! JSON record arrays are a separate check. A `.json` file between 256 KiB+//! and 64 MiB is parsed for its record count, and that count is priced with+//! a structural memory ceiling (vector width, not a measured RSS). The+//! ceiling can refuse a run. It cannot make an unmeasured disk plan fit.//!//! Free space and the watermark come from [`crate::flood`], the one//! `GET /_cat/allocation?format=json` already made before the walk. A second@@ -44,7 +50,27 @@ const CAP_1MB: u64 = 1 << 20;/// What the calibration measured, and where. An empty table is not a zero/// estimate. The numbers in [`MEASURED`] are dataset-index store sizes from/// one release binary on one host; they are not a promise about this machine.-const CALIBRATION_NOTE: &str = "per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 on one 4-core Linux x86_64 host. Code and prose were measured on real repository sources and docs; data, pdf and unindexed kinds on synthetic files. Embeddings were the default lexical feature-hash (the neural embedder was not enabled). Catalog store size is not included. time_s is a rough projection from that host, not this machine. A kind with no row is unmeasured and does not refuse the run";+const CALIBRATION_NOTE: &str = "per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 on one 4-core Linux x86_64 host. Code and prose were measured on real repository sources and docs; data, pdf and unindexed kinds on synthetic files. Embeddings were the default lexical feature-hash (the neural embedder was not enabled). Catalog store size is not included. time_s is a rough projection from that host, not this machine. A kind with no row is unmeasured and does not refuse the run. JSON record counts use a separate structural memory ceiling (memory_basis); that ceiling is not a measured store ratio";++/// Bytes charged per counted JSON record when the plan embeds.+///+/// 384-d f32 is 1,536 bytes. 2,048 leaves about 512 bytes for the source+/// document beside that one vector. Not a measured RSS. Not multiplied by+/// chunk count: the semantic cap already refuses a data column whose text+/// would chunk into tens of thousands of vectors.+const SEMANTIC_RECORD_BYTES: u64 = 2048;++/// Bytes charged per counted JSON record when the plan does not embed.+const PLAIN_RECORD_BYTES: u64 = 512;++/// `.json` files smaller than this are not parsed during the pre-flight.+const JSON_PROBE_MIN: u64 = 256 << 10;++/// `.json` files this size or larger are not parsed during the pre-flight.+/// The same bound is the whole-file JSON read cap.+const JSON_PROBE_MAX: u64 = 64 << 20;++const MEMORY_BASIS: &str = "structural ceiling from the vector width, not a measured RSS and not a store-size ratio: 2048 bytes per counted JSON record when semantic vectors are on (one 384-d f32 vector plus about 512 bytes of source) or 512 bytes per record when they are off, plus the .json file size. Only .json files from 256 KiB up to 64 MiB are counted. A file that does not parse contributes nothing and does not refuse the run";/// One measured index of one kind.#[derive(Debug, Clone, Copy)]@@ -327,6 +353,8 @@ struct Fact {kind: Kind,model: Model,vendored: bool,+ /// Rows in a probed `.json` file. `None` when the file was not probed.+ records: Option<u64>,}/// What the caller already decided, plus the walked files and the volume from@@ -340,6 +368,10 @@ pub(crate) struct Request<'a> {pub plan: Option<&'a str>,pub approve: Option<Approval>,pub duplicate_prefixes: &'a [String],+ /// Resident-memory ceiling for counted JSON records. `None` skips the+ /// memory comparison. The live path passes+ /// [`xerj_common::resource::memory_safe_zone_bytes`].+ pub memory_budget: Option<u64>,}/// A line the operator typed at the budget prompt.@@ -368,6 +400,9 @@ struct PlanEst {no_semantic: bool,keep: Option<Vec<String>>,fits: Option<bool>,+ /// Structural memory ceiling for the JSON records this plan keeps.+ /// `None` when the plan kept no probed `.json` file.+ memory_bytes: Option<u64>,}pub struct Assessment {@@ -481,7 +516,15 @@ pub(crate) fn assess(req: &Request<'_>) -> Assessment {let (budget_bytes, budget_basis, disk_json) = budget_view(req.volume, req.max_index_bytes);let plans = PLAN_NAMES.iter()- .filter_map(|name| estimate_plan(name, &facts, semantic_default, budget_bytes))+ .filter_map(|name| {+ estimate_plan(+ name,+ &facts,+ semantic_default,+ budget_bytes,+ req.memory_budget,+ )+ }).collect::<Vec<_>>();let plan_names: Vec<String> = plans.iter().map(|p| p.name.to_string()).collect();let recommended = recommended_plan(&plans).map(|p| p.name);@@ -561,6 +604,9 @@ pub(crate) fn assess(req: &Request<'_>) -> Assessment {"decision_required": needs_choice || matches!(preselect, Preselect::Refuse(_)),"plans": plans.iter().map(plan_json).collect::<Vec<_>>(),"recommended": recommended,+ "json_records": json_record_total(&facts),+ "memory_safe_zone_bytes": req.memory_budget,+ "memory_basis": MEMORY_BASIS,"duplicate_prefixes": req.duplicate_prefixes,"accept": "re-run with --plan <name>, or --yes to take the recommended plan that fits. A named plan runs even when the estimate says it does not fit (the estimate errs high). --approve cancel indexes nothing","calibration": CALIBRATION_NOTE,@@ -879,6 +925,7 @@ pub fn assess_path(plan: None,approve: None,duplicate_prefixes: &duplicates,+ memory_budget: xerj_common::resource::memory_safe_zone_bytes(),});Ok(assessment.body)}@@ -989,6 +1036,7 @@ fn estimate_plan(facts: &[Fact],semantic_default: bool,budget_bytes: Option<u64>,+ memory_budget: Option<u64>,) -> Option<PlanEst> {let force_no_semantic = matches!(name, "no-vectors" | "code-and-docs-no-vectors");let no_semantic = force_no_semantic || !semantic_default;@@ -1031,10 +1079,18 @@ fn estimate_plan(drops.push("vectors (semantic_text)".to_string());}let (index_bytes, time_s) = project_kept(&kept, !no_semantic);- let fits = match (index_bytes, budget_bytes) {+ let memory_bytes = project_memory(&kept, !no_semantic);+ let disk_fits = match (index_bytes, budget_bytes) {(Some(index), Some(budget)) => Some(index <= budget),_ => None,};+ let memory_fits = match (memory_bytes, memory_budget) {+ (Some(used), Some(budget)) => Some(used <= budget),+ _ => None,+ };+ // Memory can refuse. It cannot turn an unmeasured disk plan into a fit:+ // `disk == None && memory == Some(true)` stays `None`.+ let fits = combine_fits(disk_fits, memory_fits);let source_bytes = kept.iter().map(|f| f.size).sum();let keep_rels = if kept.len() == facts.len() && !force_no_semantic {None@@ -1055,9 +1111,42 @@ fn estimate_plan(no_semantic,keep: keep_rels,fits,+ memory_bytes,})}+fn combine_fits(disk: Option<bool>, memory: Option<bool>) -> Option<bool> {+ if disk == Some(false) || memory == Some(false) {+ Some(false)+ } else if disk == Some(true) {+ Some(true)+ } else {+ None+ }+}++/// Structural memory for probed JSON records this plan keeps.+/// `None` when none of the kept files were probed, so a tree of source+/// code is not given a made-up memory bill.+fn project_memory(kept: &[&Fact], semantic: bool) -> Option<u64> {+ let per = if semantic {+ SEMANTIC_RECORD_BYTES+ } else {+ PLAIN_RECORD_BYTES+ };+ let mut any = false;+ let mut bytes = 0u64;+ let mut records = 0u64;+ for fact in kept {+ if let Some(n) = fact.records {+ any = true;+ records = records.saturating_add(n);+ bytes = bytes.saturating_add(fact.size);+ }+ }+ any.then(|| bytes.saturating_add(per.saturating_mul(records)))+}+fn keep(name: &str, fact: &Fact) -> bool {match name {"skip-vendored" => !fact.vendored,@@ -1242,13 +1331,14 @@ fn plan_json(plan: &PlanEst) -> Value {"files": plan.files,"drops": plan.drops,"no_semantic": plan.no_semantic,+ "memory_bytes": plan.memory_bytes,"time_s": plan.time_s,"time_basis": plan.time_s.map(|_| "rough projection from the calibration host, not this machine"),})}fn plan_size_phrase(plan: &PlanEst) -> String {- match plan.index_bytes {+ let disk = match plan.index_bytes {Some(bytes) => format!("estimated index {} for {} ({} source)",human_bytes(bytes),@@ -1260,6 +1350,10 @@ fn plan_size_phrase(plan: &PlanEst) -> String {count_phrase(plan.files, "file", "files"),human_bytes(plan.source_bytes)),+ };+ match plan.memory_bytes {+ Some(bytes) => format!("{disk}; structural memory about {}", human_bytes(bytes)),+ None => disk,}}@@ -1305,6 +1399,12 @@ fn summary_lines(human_bytes(bytes),kind_phrase(facts))];+ if let Some(n) = json_record_total(facts) {+ lines.push(format!(+ "budget: {} in .json (structural memory ceiling, not a measured store ratio)",+ count_phrase(n, "record", "records")+ ));+ }if let Some(default) = plans.iter().find(|p| p.name == "default") {let fit = match default.fits {Some(true) => "fits",@@ -1391,9 +1491,65 @@ fn fact_of(file: &FileEntry) -> Fact {kind,model,vendored: order::vendored_reason(&file.rel).is_some(),+ records: probe_json_records(file),+ }+}++/// Record count for one `.json` file, matching the extractor's split:+/// a top-level array counts every element; an object sums every top-level+/// array of objects of length ≥ 2; anything else is one record. Parse+/// failure, gzip, and sizes outside [`JSON_PROBE_MIN`]..[`JSON_PROBE_MAX`]+/// return `None` — the file is unmeasured and does not refuse the run.+fn probe_json_records(file: &FileEntry) -> Option<u64> {+ if extension(&file.rel) != "json" {+ return None;+ }+ if file.size < JSON_PROBE_MIN || file.size >= JSON_PROBE_MAX {+ return None;+ }+ let bytes = std::fs::read(&file.path).ok()?;+ if bytes.len() >= 2 && bytes[0] == 0x1f && bytes[1] == 0x8b {+ return None;+ }+ let value: Value = serde_json::from_slice(&bytes).ok()?;+ Some(json_value_records(&value))+}++fn json_value_records(value: &Value) -> u64 {+ match value {+ Value::Array(arr) => arr.len() as u64,+ Value::Object(obj) => {+ let mut sum = 0u64;+ let mut any = false;+ for child in obj.values() {+ let Value::Array(arr) = child else {+ continue;+ };+ if arr.len() >= 2 && arr.iter().all(|el| el.is_object()) {+ sum = sum.saturating_add(arr.len() as u64);+ any = true;+ }+ }+ if any {+ sum+ } else {+ 1+ }+ }+ _ => 1,}}+fn json_record_total(facts: &[Fact]) -> Option<u64> {+ facts+ .iter()+ .fold(None, |acc, fact| match (acc, fact.records) {+ (None, None) => None,+ (Some(n), None) | (None, Some(n)) => Some(n),+ (Some(a), Some(b)) => Some(a.saturating_add(b)),+ })+}+/// Extension only. Sniff has not run; the census says so.fn classify(rel: &str) -> (Kind, Model) {let ext = extension(rel);@@ -1628,6 +1784,7 @@ mod tests {plan: None,approve: None,duplicate_prefixes: &[],+ memory_budget: None,});assert!(assessment.needs_choice, "{}", assessment.body);assert_eq!(@@ -1655,6 +1812,7 @@ mod tests {plan: Some("default"),approve: None,duplicate_prefixes: &[],+ memory_budget: None,});assert!(!assessment.needs_choice, "{}", assessment.body);match resolve(&assessment, Choice::Absent) {@@ -1837,4 +1995,147 @@ mod tests {.expect("one file");assert_eq!(body["would_remove_total"], 1);}++ #[test]+ fn json_record_count_sums_top_level_arrays_of_objects() {+ let value = serde_json::json!({+ "functions": [{"n": 1}, {"n": 2}, {"n": 3}],+ "runs": [{"a": 1}, {"a": 2}],+ "note": "x"+ });+ assert_eq!(json_value_records(&value), 5);+ assert_eq!(json_value_records(&serde_json::json!([1, 2, 3, 4])), 4);+ assert_eq!(json_value_records(&serde_json::json!({"a": 1})), 1);+ }++ /// #2102 priced disk only. A wide JSON array has no measured store ratio+ /// (`.json` is `other`), so disk never refuses it. The record count does,+ /// against the memory ceiling, and `no-vectors` is the cheaper plan.+ /// A memory ceiling the file fits under must not flip an unmeasured disk+ /// plan to `fits: true`.+ #[test]+ fn counted_json_records_can_refuse_on_memory_without_a_disk_ratio() {+ let dir = tempfile::tempdir().unwrap();+ let records = 3_000u64;+ let pad = "p".repeat(80);+ let path = dir.path().join("rows.json");+ let mut body = String::new();+ body.push('[');+ for i in 0..records {+ if i > 0 {+ body.push(',');+ }+ body.push_str(&format!(r#"{{"id":{i},"pad":"{pad}"}}"#));+ }+ body.push(']');+ let size = body.len() as u64;+ assert!(size >= JSON_PROBE_MIN, "{size}");+ assert!(+ size <= CAP_1MB,+ "cap-1mb must still see this file, got {size}"+ );+ std::fs::write(&path, &body).unwrap();+ let files = vec![FileEntry {+ path,+ rel: "rows.json".into(),+ rel_id: "rows.json".into(),+ is_symlink: false,+ size,+ }];+ let plain = size + PLAIN_RECORD_BYTES * records;+ let semantic = size + SEMANTIC_RECORD_BYTES * records;+ assert!(plain < semantic);++ // Disk budget is large enough that the measured `other` ratio fits.+ // Memory is what separates default from no-vectors.+ let request = |memory_budget, max_index_bytes| Request {+ root: dir.path().to_str().unwrap(),+ files: &files,+ volume: None,+ max_index_bytes,+ no_semantic: false,+ plan: None,+ approve: None,+ duplicate_prefixes: &[],+ memory_budget,+ };+ let tight = assess(&request(Some(plain + 1), Some(u64::MAX)));+ assert_eq!(tight.body["json_records"], serde_json::json!(records));+ let plan = |body: &Value, name: &str| -> Value {+ body["plans"]+ .as_array()+ .unwrap()+ .iter()+ .find(|p| p["name"] == name)+ .unwrap()+ .clone()+ };+ let default = plan(&tight.body, "default");+ assert_eq!(default["fits"], serde_json::json!(false), "{default}");+ assert!(tight.needs_choice, "{}", tight.body);+ let no_vectors = plan(&tight.body, "no-vectors");+ assert_eq!(no_vectors["fits"], serde_json::json!(true), "{no_vectors}");+ assert_eq!(tight.body["recommended"], serde_json::json!("no-vectors"));+ assert!(+ tight+ .lines+ .iter()+ .any(|line| line.contains("3000") && line.contains("record")),+ "{:?}",+ tight.lines+ );++ let open = assess(&request(Some(u64::MAX), None));+ let default = plan(&open.body, "default");+ assert!(+ default["fits"].is_null(),+ "memory fitting must not accept an unmeasured disk plan: {default}"+ );+ assert!(default["memory_bytes"].as_u64().unwrap() >= semantic);+ assert!(!open.needs_choice);++ let missing = sample_files();+ let untouched = assess(&Request {+ root: "/data",+ files: &missing,+ volume: None,+ max_index_bytes: None,+ no_semantic: false,+ plan: None,+ approve: None,+ duplicate_prefixes: &[],+ memory_budget: Some(1),+ });+ assert!(untouched.body["json_records"].is_null());+ assert!(untouched.body["plans"][0]["fits"].is_null());+ assert!(!untouched.needs_choice);+ }++ #[test]+ fn a_json_file_that_does_not_parse_does_not_refuse() {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("broken.json");+ let body = "{".repeat(JSON_PROBE_MIN as usize + 8);+ std::fs::write(&path, &body).unwrap();+ let files = vec![FileEntry {+ path,+ rel: "broken.json".into(),+ rel_id: "broken.json".into(),+ is_symlink: false,+ size: body.len() as u64,+ }];+ let assessment = assess(&Request {+ root: dir.path().to_str().unwrap(),+ files: &files,+ volume: None,+ max_index_bytes: None,+ no_semantic: false,+ plan: None,+ approve: None,+ duplicate_prefixes: &[],+ memory_budget: Some(1),+ });+ assert!(assessment.body["json_records"].is_null());+ assert!(!assessment.needs_choice, "{}", assessment.body);+ }}diff --git a/engine/crates/xerj-autoindex/src/dataset.rs b/engine/crates/xerj-autoindex/src/dataset.rsindex 427eee7e1..11c80d4a6 100644--- a/engine/crates/xerj-autoindex/src/dataset.rs+++ b/engine/crates/xerj-autoindex/src/dataset.rs@@ -71,7 +71,10 @@ pub(crate) fn small_structured_config(c: &Cluster) -> bool {return false;}let ceiling = (c.members.len() as u64).saturating_mul(DOC_DEMOTE_MAX_RECORDS_PER_FILE);- c.records <= ceiling+ // `population` is the real row count when the extractor knows it (a JSON+ // array sampled at 500 can still hold 100_000 rows). `records` is only+ // how many rows the sampler kept.+ c.population <= ceiling}#[derive(Debug)]@@ -84,6 +87,9 @@ pub struct Sketch {/// `fields`, is the schema-clustering key; empty means "document".pub key_fields: HashSet<String>,pub records: u64,+ /// Rows this group actually holds. Equals `records` when the sample saw+ /// every row; larger when the extractor declared a full array length.+ pub population: u64,}#[derive(Debug)]@@ -95,6 +101,9 @@ pub struct Cluster {/// Union of the members' `key_fields` — what new sketches are compared to.pub key_fields: HashSet<String>,pub records: u64,+ /// Sum of the members' [`Sketch::population`]. The per-scope cap and the+ /// small-config fold consult this, not the sample count in `records`.+ pub population: u64,pub slug: String,/// Repository scope this cluster lives in ("" = the autoindex root).pub scope: String,@@ -138,6 +147,7 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Vecfields: HashMap::new(),key_fields: HashSet::new(),records: 0,+ population: 0,slug: String::new(),scope: scope.to_string(),is_docs: true,@@ -152,6 +162,7 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Veclet c = docs_cluster(&mut docs, scope);c.members.push(sk.file_idx);c.records += sk.records;+ c.population += sk.population;for (k, acc) in sk.fields {match c.fields.get_mut(&k) {Some(existing) => existing.merge(&acc),@@ -181,6 +192,7 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Veclet c = &mut data[ci];c.members.push(sk.file_idx);c.records += sk.records;+ c.population += sk.population;c.key_fields.extend(sk.key_fields);for (k, acc) in sk.fields {match c.fields.get_mut(&k) {@@ -198,6 +210,7 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Vecfields: sk.fields,key_fields: sk.key_fields,records: sk.records,+ population: sk.population,slug: String::new(),scope: scope.to_string(),is_docs: false,@@ -210,10 +223,20 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Vec// YAML/JSON/XML file — to their scope's document dataset. Their sampled// fields are dropped — the caller re-samples the demoted files through// the document renderer, which is also what indexes them.+ //+ // A file that also produced a grouped dataset (a second top-level JSON+ // array) must not be folded: folding marks the whole file `as_document`+ // and phase B then drops the grouped rows.+ let grouped_files: HashSet<usize> = data+ .iter()+ .filter(|c| c.group.is_some())+ .flat_map(|c| c.members.iter().copied())+ .collect();let mut kept: Vec<Cluster> = Vec::with_capacity(data.len());let mut capped: BTreeMap<String, Vec<Cluster>> = BTreeMap::new();for c in data {- if small_structured_config(&c) {+ let shares_grouped_sibling = c.members.iter().any(|m| grouped_files.contains(m));+ if small_structured_config(&c) && !shares_grouped_sibling {let d = docs_cluster(&mut docs, &c.scope);d.members.extend(c.members.iter().copied());d.demoted.extend(c.members);@@ -223,7 +246,8 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Vec// group, and not a large collection. Everything else stays.if demotable_family(c.family)&& c.group.is_none()- && c.records < DATASET_CAP_PROTECT_RECORDS+ && c.population < DATASET_CAP_PROTECT_RECORDS+ && !shares_grouped_sibling{capped.entry(c.scope.clone()).or_default().push(c);continue;@@ -232,10 +256,12 @@ pub fn cluster(sketches: Vec<Sketch>, rels: &[String], scopes: &[String]) -> Vec}for (scope, mut group) in capped {// Largest collections stay. Ties break toward the earlier file so- // the fold does not depend on hash order.+ // the fold does not depend on hash order. Size is the real row+ // count, so a file sampled at 500 does not tie with a 100-row file+ // that happened to fit in the sample.group.sort_by(|a, b| {- b.records- .cmp(&a.records)+ b.population+ .cmp(&a.population).then(b.members.len().cmp(&a.members.len())).then_with(|| {let am = a.members.iter().copied().min().unwrap_or(usize::MAX);@@ -575,6 +601,7 @@ mod tests {fields,key_fields: HashSet::new(),records: 1,+ population: 1,}}@@ -590,6 +617,7 @@ mod tests {fields,key_fields: HashSet::new(),records: 1,+ population: 1,}}@@ -612,6 +640,7 @@ mod tests {fields,key_fields,records,+ population: records,}}@@ -943,6 +972,75 @@ mod tests {assert!(kept_medium.iter().all(|r| *r >= 40 + 8), "{kept_medium:?}");}+ /// Phase A samples 500 rows. A JSON array of 1024+ rows must still be+ /// protected when the sampler's count is below the threshold.+ #[test]+ fn the_cap_protects_a_sampled_array_by_its_real_length() {+ let mut sketches = vec![prose_sketch(0)];+ let mut rels = vec!["README.md".into()];+ for i in 0..MAX_STRUCTURED_DATASETS_PER_SCOPE + 1 {+ let idx = i + 1;+ rels.push(format!("f{idx}.json"));+ sketches.push(data_sketch_n(idx, Family::Json, &[&format!("k{idx}")], 40));+ }+ let big = sketches.len();+ rels.push("huge.json".into());+ let mut huge = data_sketch_n(big, Family::Json, &["id"], 500);+ huge.population = DATASET_CAP_PROTECT_RECORDS;+ sketches.push(huge);+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");+ assert!(+ !docs.demoted.contains(&big),+ "real length {}, sample 500: {docs:#?}",+ DATASET_CAP_PROTECT_RECORDS+ );+ assert!(clusters+ .iter()+ .any(|c| !c.is_docs && c.population == DATASET_CAP_PROTECT_RECORDS));+ }++ /// Folding the primary array of a multi-array file would mark the whole+ /// file as a document and drop the sibling array's rows.+ #[test]+ fn a_grouped_sibling_keeps_its_file_out_of_the_structured_cap() {+ let mut sketches = vec![prose_sketch(0)];+ let mut rels = vec!["README.md".into()];+ let n_medium = MAX_STRUCTURED_DATASETS_PER_SCOPE + 1;+ for i in 0..n_medium {+ let idx = i + 1;+ rels.push(format!("fixtures/f{idx}.json"));+ sketches.push(data_sketch_n(idx, Family::Json, &[&format!("k{idx}")], 40));+ }+ let primary_idx = n_medium + 1;+ rels.push("config/subsystem_map.json".into());+ sketches.push(data_sketch_n(+ primary_idx,+ Family::Json,+ &["name", "addr"],+ 40,+ ));+ let mut sibling = data_sketch_n(primary_idx, Family::Json, &["label", "start"], 30);+ sibling.group = Some("runs".into());+ sketches.push(sibling);++ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");+ assert!(+ !docs.demoted.contains(&primary_idx),+ "the multi-array file must stay extracted as records"+ );+ assert!(+ clusters+ .iter()+ .any(|c| { c.group.as_deref() == Some("runs") && c.members == vec![primary_idx] }),+ "the sibling array stays its own dataset: {clusters:#?}"+ );+ assert!(clusters+ .iter()+ .any(|c| { c.group.is_none() && !c.is_docs && c.members.contains(&primary_idx) }));+ }+/// Scope is a hard boundary for data schemas too: the same CSV schema in/// two repositories is two datasets (per-repo corpora stay self-contained).#[test]diff --git a/engine/crates/xerj-autoindex/src/extract/json.rs b/engine/crates/xerj-autoindex/src/extract/json.rsindex 7bb1f4a8f..971e294c4 100644--- a/engine/crates/xerj-autoindex/src/extract/json.rs+++ b/engine/crates/xerj-autoindex/src/extract/json.rs@@ -1,7 +1,13 @@//! Whole-value JSON files (object or array), capped at 64MB.//! - array of objects → one record per element-//! - object with a dominant top-level array of objects → one record per+//! - object with one dominant top-level array of objects → one record per//! element, remaining top-level scalars merged in as shared fields+//! - object with two or more top-level arrays of objects, the largest longer+//! than a config document → one dataset per array. The largest keeps the+//! ungrouped locators (`{key}:e{i}`); each other array is its own group.+//! Sibling arrays are not copied onto each other's rows (#1856, #1978)+//! - a shared value bigger than [`SHARED_REPLICATE_MAX`] is emitted once,+//! not cloned onto every row//! - a Google Keep note (Takeout `Keep/*.json`) → a DOCUMENT: `title`/`body`//! plus labels and dates — see [`keep_note`]//! - anything else → a single record@@ -40,13 +46,16 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {};match v {Value::Array(arr) => {+ if !arr.is_empty() {+ stats.populations.push((None, arr.len() as u64));+ }for (i, el) in arr.into_iter().enumerate() {if !emit(el, &format!("e{i}"), sink, &mut stats) {break;}}}- Value::Object(mut obj) => {+ Value::Object(obj) => {// A Google Keep note is a document, not a row. Tested before the// array rule below, which would otherwise explode a checklist// note's `listContent` into one record per bullet.@@ -61,47 +70,7 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {);return Ok(stats);}- // find the largest top-level array-of-objects- let mut best: Option<(String, usize)> = None;- for (k, vv) in obj.iter() {- if let Value::Array(a) = vv {- if a.len() >= 2- && a.iter().all(|e| e.is_object())- && best.as_ref().map(|(_, n)| a.len() > *n).unwrap_or(true)- {- best = Some((k.clone(), a.len()));- }- }- }- match best {- Some((key, _)) => {- let Value::Array(arr) = obj.remove(&key).unwrap() else {- unreachable!()- };- let shared = flatten_object(obj);- for (i, el) in arr.into_iter().enumerate() {- if let Value::Object(m) = el {- let mut fields = shared.clone();- for (k, v) in flatten_object(m) {- fields.insert(k, v); // element wins collisions- }- ensure_text_passage(&mut fields);- stats.records += 1;- if !sink(RawRecord {- fields,- locator: format!("{key}:e{i}"),- group: None,- origin: FieldOrigin::Data,- }) {- break;- }- }- }- }- None => {- emit(Value::Object(obj), "doc", sink, &mut stats);- }- }+ extract_object(obj, sink, &mut stats);}other => {let mut m = Map::new();@@ -257,6 +226,171 @@ pub(crate) fn is_keep_note_file(path: &Path) -> bool {}}+/// A shared value longer than this is not cloned onto every row.+///+/// Cloning the compact JSON of one sibling array onto every element of+/// another is what wedged ingest: `subsystem_map.json` (#1856) copied a+/// 25 KB `runs` array onto each of 8,589 function rows and then elected+/// that string as `semantic_text`. A 42 MB file of the same shape (#1978)+/// multiplied several sibling arrays by ~123,000 rows until the process+/// ran out of memory. Past this size the value is emitted once, under its+/// own group.+const SHARED_REPLICATE_MAX: usize = 4096;++/// Shortest largest-array that is split into one dataset per array.+///+/// [`crate::dataset::DOC_DEMOTE_MAX_RECORDS_PER_FILE`] folds a JSON file of+/// at most 32 records into the document corpus. Splitting a file that+/// small would turn one config document into several indices. An array of+/// this length or more is a collection.+const MULTI_ARRAY_SPLIT_MIN: usize = 33;++/// Top-level arrays of objects (len ≥ 2, every element an object), longest+/// first. Equal lengths break toward the lexicographically smaller key so+/// the primary array does not depend on map iteration order.+fn record_arrays(obj: &Map<String, Value>) -> Vec<(String, usize)> {+ let mut arrays = Vec::new();+ for (k, vv) in obj {+ if let Value::Array(a) = vv {+ if a.len() >= 2 && a.iter().all(|e| e.is_object()) {+ arrays.push((k.clone(), a.len()));+ }+ }+ }+ arrays.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));+ arrays+}++fn extract_object(obj: Map<String, Value>, sink: Sink, stats: &mut ExtractStats) {+ let arrays = record_arrays(&obj);+ if arrays.len() >= 2 && arrays[0].1 >= MULTI_ARRAY_SPLIT_MIN {+ extract_split(obj, arrays, sink, stats);+ } else if let Some((key, _)) = arrays.into_iter().next() {+ extract_one_array(obj, key, sink, stats);+ } else {+ emit(Value::Object(obj), "doc", sink, stats);+ }+}++/// One record array. Other arrays stay shared fields, unless one of them+/// is too big to copy onto every row.+fn extract_one_array(+ mut obj: Map<String, Value>,+ key: String,+ sink: Sink,+ stats: &mut ExtractStats,+) {+ let Value::Array(arr) = obj.remove(&key).unwrap() else {+ unreachable!("record_arrays only yields array keys")+ };+ stats.populations.push((None, arr.len() as u64));+ let (shared, sidecars) = partition_shared(obj);+ for (i, el) in arr.into_iter().enumerate() {+ let Value::Object(m) = el else { continue };+ let fields = merge_shared(&shared, m);+ if !emit_fields(fields, format!("{key}:e{i}"), None, sink, stats) {+ break;+ }+ }+ emit_sidecars(sidecars, sink, stats);+}++/// Two or more record arrays. The longest keeps group `None` and the+/// `{key}:e{i}` locator a single-array file already used, so an existing+/// document id for that array does not move. Every other array is a group+/// of its own and is not serialized onto the primary rows.+fn extract_split(+ mut obj: Map<String, Value>,+ arrays: Vec<(String, usize)>,+ sink: Sink,+ stats: &mut ExtractStats,+) {+ let mut pulled = Vec::with_capacity(arrays.len());+ for (key, _) in &arrays {+ let Value::Array(arr) = obj.remove(key).unwrap() else {+ unreachable!("record_arrays only yields array keys")+ };+ pulled.push((key.clone(), arr));+ }+ let (shared, sidecars) = partition_shared(obj);+ for (idx, (key, arr)) in pulled.into_iter().enumerate() {+ let group = if idx == 0 { None } else { Some(key.clone()) };+ stats.populations.push((group.clone(), arr.len() as u64));+ for (i, el) in arr.into_iter().enumerate() {+ let Value::Object(m) = el else { continue };+ let fields = merge_shared(&shared, m);+ // A `false` sink means this group's sample is full. The next+ // array is a different group and still needs its own sample.+ if !emit_fields(fields, format!("{key}:e{i}"), group.clone(), sink, stats) {+ break;+ }+ }+ }+ emit_sidecars(sidecars, sink, stats);+}++fn merge_shared(shared: &Map<String, Value>, element: Map<String, Value>) -> Map<String, Value> {+ let mut fields = shared.clone();+ for (k, v) in flatten_object(element) {+ fields.insert(k, v); // element wins collisions+ }+ fields+}++/// Flatten the wrapper left after record arrays are removed, then hold back+/// any value too large to clone onto every row.+fn partition_shared(obj: Map<String, Value>) -> (Map<String, Value>, Vec<(String, Value)>) {+ let flat = flatten_object(obj);+ let mut small = Map::new();+ let mut sidecars = Vec::new();+ for (k, v) in flat {+ if shared_value_too_big(&v) {+ sidecars.push((k, v));+ } else {+ small.insert(k, v);+ }+ }+ (small, sidecars)+}++fn shared_value_too_big(v: &Value) -> bool {+ match v {+ Value::String(s) => s.len() > SHARED_REPLICATE_MAX,+ Value::Array(a) => serde_json::to_string(a)+ .map(|s| s.len() > SHARED_REPLICATE_MAX)+ .unwrap_or(true),+ _ => false,+ }+}++fn emit_sidecars(sidecars: Vec<(String, Value)>, sink: Sink, stats: &mut ExtractStats) {+ for (key, value) in sidecars {+ stats.populations.push((Some(key.clone()), 1));+ let mut fields = Map::new();+ fields.insert(key.clone(), value);+ // One record. A full sample on an earlier group must not drop it:+ // the sink's `false` is per group, and this group has not been seen.+ let _ = emit_fields(fields, format!("{key}:shared"), Some(key), sink, stats);+ }+}++fn emit_fields(+ mut fields: Map<String, Value>,+ locator: String,+ group: Option<String>,+ sink: Sink,+ stats: &mut ExtractStats,+) -> bool {+ ensure_text_passage(&mut fields);+ stats.records += 1;+ sink(RawRecord {+ fields,+ locator,+ group,+ origin: FieldOrigin::Data,+ })+}+/// Whole-file parse failed — re-parse the same in-memory bytes line by/// line under the JSONL contract (see the fallback note in [`extract`])./// Mirrors `jsonl::extract`'s per-line semantics: blank lines skip, a line@@ -709,4 +843,120 @@ mod tests {.unwrap().contains("count: 7"));}++ /// #1856 / #1978: two top-level arrays of objects are two datasets.+ /// The smaller array used to be serialized and cloned onto every row+ /// of the larger one, and that JSON string then won `semantic_text`.+ #[test]+ fn two_top_level_arrays_stay_separate_datasets() {+ let mut functions = Vec::new();+ for i in 0..MULTI_ARRAY_SPLIT_MIN {+ functions.push(format!(r#"{{"name":"f{i}","addr":{i}}}"#));+ }+ let text = format!(+ r#"{{"generated":"2026-01-01","functions":[{}],"runs":[{{"label":"boot","start":1}},{{"label":"loop","start":9}}]}}"#,+ functions.join(",")+ );+ let (stats, recs) = run(&text);+ let primary: Vec<_> = recs.iter().filter(|r| r.group.is_none()).collect();+ let runs: Vec<_> = recs+ .iter()+ .filter(|r| r.group.as_deref() == Some("runs"))+ .collect();+ assert_eq!(primary.len(), MULTI_ARRAY_SPLIT_MIN);+ assert_eq!(primary[0].locator, "functions:e0");+ assert!(+ primary[0].fields.get("runs").is_none(),+ "the sibling array must not be copied onto the primary row: {:?}",+ primary[0].fields+ );+ assert_eq!(+ primary[0].fields["generated"],+ serde_json::json!("2026-01-01"),+ "a small shared scalar is still merged"+ );+ assert_eq!(primary[0].fields["name"], serde_json::json!("f0"));+ assert_eq!(runs.len(), 2);+ assert_eq!(runs[0].locator, "runs:e0");+ assert_eq!(runs[0].fields["label"], serde_json::json!("boot"));+ assert!(runs[0].fields.get("name").is_none());+ assert_eq!(runs[0].fields["generated"], serde_json::json!("2026-01-01"));+ assert!(stats+ .populations+ .contains(&(None, MULTI_ARRAY_SPLIT_MIN as u64)));+ assert!(stats.populations.contains(&(Some("runs".into()), 2)));+ }++ /// Equal lengths must not depend on object-key order.+ #[test]+ fn equal_length_arrays_pick_the_lexicographically_smaller_key_as_primary() {+ let row = r#"{"n":1},{"n":2},{"n":3},{"n":4},{"n":5},{"n":6},{"n":7},{"n":8},{"n":9},{"n":10},{"n":11},{"n":12},{"n":13},{"n":14},{"n":15},{"n":16},{"n":17},{"n":18},{"n":19},{"n":20},{"n":21},{"n":22},{"n":23},{"n":24},{"n":25},{"n":26},{"n":27},{"n":28},{"n":29},{"n":30},{"n":31},{"n":32},{"n":33}"#;+ let text = format!(r#"{{"zeta":[{row}],"alpha":[{row}]}}"#);+ let (_, recs) = run(&text);+ let primary: Vec<_> = recs.iter().filter(|r| r.group.is_none()).collect();+ assert_eq!(primary.len(), MULTI_ARRAY_SPLIT_MIN);+ assert_eq!(primary[0].locator, "alpha:e0");+ assert!(recs.iter().any(|r| r.group.as_deref() == Some("zeta")));+ assert!(recs.iter().all(|r| r.group.as_deref() != Some("alpha")));+ }++ /// A wrapper value too big to clone is one sidecar record, not a field+ /// on every row. A value under the cap stays shared, which is what the+ /// single-element `errors` array in the largest-array test pins.+ #[test]+ fn an_oversized_shared_value_is_emitted_once() {+ let blob = "x".repeat(SHARED_REPLICATE_MAX + 32);+ let text = format!(r#"{{"blob":"{blob}","items":[{{"id":1}},{{"id":2}},{{"id":3}}]}}"#);+ let (stats, recs) = run(&text);+ let rows: Vec<_> = recs.iter().filter(|r| r.group.is_none()).collect();+ assert_eq!(rows.len(), 3);+ assert!(+ rows.iter().all(|r| r.fields.get("blob").is_none()),+ "the blob must not be cloned onto every row"+ );+ let side: Vec<_> = recs.iter().filter(|r| r.group.is_some()).collect();+ assert_eq!(side.len(), 1);+ assert_eq!(side[0].group.as_deref(), Some("blob"));+ assert_eq!(side[0].locator, "blob:shared");+ assert!(side[0].fields["blob"].as_str().unwrap().len() > SHARED_REPLICATE_MAX);+ assert_eq!(+ stats+ .populations+ .iter()+ .find(|(g, _)| g.is_none())+ .map(|(_, n)| *n),+ Some(3)+ );+ assert!(stats.populations.contains(&(Some("blob".into()), 1)));+ }++ /// The sink stopping one group (the phase-A sample) must not drop the+ /// declared length or the next array.+ #[test]+ fn stopping_one_array_keeps_its_length_and_the_sibling() {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("t.json");+ let mut functions = Vec::new();+ for i in 0..MULTI_ARRAY_SPLIT_MIN {+ functions.push(format!(r#"{{"name":"f{i}"}}"#));+ }+ let text = format!(+ r#"{{"functions":[{}],"runs":[{{"label":"a"}},{{"label":"b"}},{{"label":"c"}}]}}"#,+ functions.join(",")+ );+ std::fs::write(&path, text).unwrap();+ let mut seen = std::collections::HashMap::<Option<String>, usize>::new();+ let stats = extract(&path, false, &mut |r| {+ let n = seen.entry(r.group.clone()).or_insert(0);+ *n += 1;+ *n < 2+ })+ .unwrap();+ assert_eq!(seen.get(&None).copied(), Some(2));+ assert_eq!(seen.get(&Some("runs".into())).copied(), Some(2));+ assert!(stats+ .populations+ .contains(&(None, MULTI_ARRAY_SPLIT_MIN as u64)));+ assert!(stats.populations.contains(&(Some("runs".into()), 3)));+ }}diff --git a/engine/crates/xerj-autoindex/src/extract/mod.rs b/engine/crates/xerj-autoindex/src/extract/mod.rsindex 66b32f283..d7e76fd3e 100644--- a/engine/crates/xerj-autoindex/src/extract/mod.rs+++ b/engine/crates/xerj-autoindex/src/extract/mod.rs@@ -75,7 +75,7 @@ pub struct RawRecord {pub origin: FieldOrigin,}-#[derive(Debug, Default, Clone, Copy)]+#[derive(Debug, Default, Clone)]pub struct ExtractStats {pub records: u64,pub junk: u64,@@ -83,6 +83,11 @@ pub struct ExtractStats {/// split before the whole body was emitted (#381). Carried out so the/// truncation is reported, never silent.pub truncated: bool,+ /// Full length of each record array, keyed by the group the records were+ /// emitted under. `None` is the ungrouped array. Recorded before the+ /// sampler stops, so a 100_000-row array sampled at 500 still reports+ /// 100_000. Empty for families that do not know a length.+ pub populations: Vec<(Option<String>, u64)>,}/// Sink returns false to stop extraction early (sampling).diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rsindex bdb4692ba..e7e7241a8 100644--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs@@ -6450,6 +6450,7 @@ mod delete_gate {plan: None,approve: None,duplicate_prefixes: &[],+ memory_budget: None,});let plan_bytes = |name: &str| -> u64 {let plan = priced.body["plans"]diff --git a/engine/crates/xerj-autoindex/src/infer/mod.rs b/engine/crates/xerj-autoindex/src/infer/mod.rsindex 8aa8d57d0..7317ba00f 100644--- a/engine/crates/xerj-autoindex/src/infer/mod.rs+++ b/engine/crates/xerj-autoindex/src/infer/mod.rs@@ -676,6 +676,93 @@ pub fn infer_fields_with_policy(specs}+/// Automatic `semantic_text` on a data dataset stops past this many rows.+///+/// The built-in lexical embedder is cheap per call and still writes one+/// vector per row (and one per chunk when the value is long). A 123_000-row+/// extract with a prose column is a multi-gigabyte embedding job the+/// dry-run cannot see (#1978). Document datasets are not capped: their+/// body carriers are the product.+pub const SEMANTIC_DATA_MAX_RECORDS: u64 = 100_000;++/// Automatic `semantic_text` on a data dataset stops past this many+/// characters of elected text (`avg_len × population`).+///+/// 8 MiB is enough for a real prose column and small enough that the+/// server-side chunker (512 characters, overlap 64) cannot turn one field+/// into tens of thousands of vectors. Equality stays semantic; one+/// character over falls back.+pub const SEMANTIC_DATA_MAX_TEXT_BYTES: u64 = 8 << 20;++/// Why a data dataset's elected body was stored as plain `text`.+pub struct SemanticCap {+ pub field: String,+ pub records: u64,+ pub text_bytes: u64,+}++/// Downgrade every `semantic_text` field on a data dataset that is over+/// [`SEMANTIC_DATA_MAX_RECORDS`] or [`SEMANTIC_DATA_MAX_TEXT_BYTES`].+///+/// The column stays `text`, so it is still searchable. `None` means the+/// election stands. Document datasets must not be passed here: the caller+/// skips `is_docs`.+pub fn cap_data_semantic(specs: &mut [FieldSpec], population: u64) -> Option<SemanticCap> {+ let elected: Vec<usize> = specs+ .iter()+ .enumerate()+ .filter(|(_, s)| s.es_type == "semantic_text")+ .map(|(i, _)| i)+ .collect();+ if elected.is_empty() {+ return None;+ }+ let field_idx = elected.iter().copied().max_by(|&a, &b| {+ specs[a]+ .avg_len+ .partial_cmp(&specs[b].avg_len)+ .unwrap_or(std::cmp::Ordering::Equal)+ })?;+ let avg = specs[field_idx].avg_len;+ // `avg_len` is characters. ASCII JSON is one byte per character; a+ // multibyte field is larger, so this under-counts. The cap is a trigger,+ // not a measurement of resident memory.+ let text_bytes = (avg as u64).saturating_mul(population);+ if population <= SEMANTIC_DATA_MAX_RECORDS && text_bytes <= SEMANTIC_DATA_MAX_TEXT_BYTES {+ return None;+ }+ let field = specs[field_idx].name.clone();+ let note = semantic_fallback_note(&field, population, text_bytes);+ for i in elected {+ specs[i].es_type = "text".into();+ specs[i]+ .notes+ .retain(|n| !n.starts_with("hybrid lexical+vector body"));+ specs[i].notes.push(note.clone());+ }+ Some(SemanticCap {+ field,+ records: population,+ text_bytes,+ })+}++pub fn semantic_fallback_warning(slug: &str, cap: &SemanticCap) -> String {+ format!(+ "dataset '{slug}': {}",+ semantic_fallback_note(&cap.field, cap.records, cap.text_bytes)+ )+}++fn semantic_fallback_note(field: &str, records: u64, text_bytes: u64) -> String {+ format!(+ "semantic_text on '{field}' fell back to plain text: {records} records and about \+ {text_bytes} characters of text exceed the automatic cap of \+ {SEMANTIC_DATA_MAX_RECORDS} records or {SEMANTIC_DATA_MAX_TEXT_BYTES} bytes. \+ The column stays searchable as text. Pass --no-semantic to index without vectors."+ )+}+/// Elect the dataset time field: date-typed field with the highest coverage.pub fn elect_time_field(specs: &[FieldSpec]) -> Option<String> {specs@@ -940,6 +1027,94 @@ mod tests {}}+ fn prose_note(sentence: &str) -> HashMap<String, FieldAcc> {+ let mut fields = HashMap::new();+ let mut acc = FieldAcc::default();+ for _ in 0..20 {+ acc.add(&Value::String(sentence.to_string()));+ }+ fields.insert("note".into(), acc);+ fields+ }++ /// #1978: a data column that looks like prose is embedded only while the+ /// real row count and the text it would embed stay inside the cap.+ /// Equality stays `semantic_text`. One record or one character over+ /// falls back to plain text and says so, including `--no-semantic`.+ #[test]+ fn semantic_text_falls_back_past_the_record_and_text_caps() {+ // Short enough that 100_000 copies stay under the byte cap, long+ // enough to elect. The record cap is then the only thing that moves.+ let short =+ "Each worker retries every failed handshake and reports why that one attempt failed.";+ let fields = prose_note(short);+ let mut under = infer_fields(&fields, 20, false);+ let avg = under.iter().find(|s| s.name == "note").unwrap().avg_len;+ assert!(+ avg >= 80.0+ && (avg as u64).saturating_mul(SEMANTIC_DATA_MAX_RECORDS)+ <= SEMANTIC_DATA_MAX_TEXT_BYTES,+ "fixture avg_len {avg} must elect and stay inside the byte cap at the record limit"+ );+ assert_eq!(+ under.iter().find(|s| s.name == "note").unwrap().es_type,+ "semantic_text",+ "the fixture itself must elect, or the cap is untested: {under:#?}"+ );+ assert!(cap_data_semantic(&mut under, SEMANTIC_DATA_MAX_RECORDS).is_none());+ assert_eq!(+ under.iter().find(|s| s.name == "note").unwrap().es_type,+ "semantic_text"+ );++ let mut over = infer_fields(&fields, 20, false);+ let hit = cap_data_semantic(&mut over, SEMANTIC_DATA_MAX_RECORDS + 1).unwrap();+ assert_eq!(hit.field, "note");+ assert_eq!(hit.records, SEMANTIC_DATA_MAX_RECORDS + 1);+ let spec = over.iter().find(|s| s.name == "note").unwrap();+ assert_eq!(spec.es_type, "text");+ let note = spec.notes.join(" ");+ assert!(note.contains("--no-semantic"), "{note}");+ assert!(note.contains("note"), "{note}");+ assert!(+ note.contains(&(SEMANTIC_DATA_MAX_RECORDS + 1).to_string()),+ "{note}"+ );+ assert!(+ !note.contains("hybrid lexical+vector body"),+ "the election note must not still claim the field is embedded: {note}"+ );++ let long = "The connection pool retries every failed handshake with backoff. \+ Each worker owns one socket and never shares it across threads.";+ let long_fields = prose_note(long);+ let long_specs = infer_fields(&long_fields, 20, false);+ let avg = long_specs+ .iter()+ .find(|s| s.name == "note")+ .unwrap()+ .avg_len as u64;+ assert!(avg >= 80, "{avg}");+ let exact = SEMANTIC_DATA_MAX_TEXT_BYTES / avg;+ assert!(exact > 0 && exact <= SEMANTIC_DATA_MAX_RECORDS, "{exact}");+ let mut at_bytes = infer_fields(&long_fields, 20, false);+ assert!(+ cap_data_semantic(&mut at_bytes, exact).is_none(),+ "avg {avg} × {exact} characters is inside the byte cap"+ );+ let mut over_bytes = infer_fields(&long_fields, 20, false);+ let hit = cap_data_semantic(&mut over_bytes, exact + 1).unwrap();+ assert!(hit.text_bytes > SEMANTIC_DATA_MAX_TEXT_BYTES);+ assert_eq!(+ over_bytes+ .iter()+ .find(|s| s.name == "note")+ .unwrap()+ .es_type,+ "text"+ );+ }+/// #173: a document dataset merges extractor vocabularies whose records/// each populate exactly one body carrier (`body` for code/prose, `text`/// for line chunks). The docs policy must elect EVERY qualifying carrierdiff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex ad605fe64..2a45a763e 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -3086,6 +3086,8 @@ struct GroupSketch {fields: HashMap<String, infer::FieldAcc>,key_fields: std::collections::HashSet<String>,records: u64,+ /// Rows in this group. `records` stops at the sample; this does not.+ population: u64,}fn take_pdf_spool_if_indexable<T>(@@ -3338,8 +3340,10 @@ fn scan_file(extraction_elapsed = extraction_started.elapsed();parsed};+ let mut populations: Vec<(Option<String>, u64)> = Vec::new();match extraction {Ok(stats) => {+ populations = stats.populations;if groups.is_empty() {out.junk = Some(("junk".into(),@@ -3374,11 +3378,20 @@ fn scan_file(out.sketches = groups.into_iter()- .map(|(group, (fields, records, key_fields))| GroupSketch {- group,- fields,- key_fields,- records,+ .map(|(group, (fields, records, key_fields))| {+ let declared = populations+ .iter()+ .filter(|(g, _)| g == &group)+ .map(|(_, n)| *n)+ .max()+ .unwrap_or(records);+ GroupSketch {+ group,+ fields,+ key_fields,+ records,+ population: declared.max(records),+ }}).collect();out.sketches.sort_by(|a, b| a.group.cmp(&b.group));@@ -3457,6 +3470,7 @@ mod clustering_key_tests {fields: s.sketches[0].fields.clone(),key_fields: s.sketches[0].key_fields.clone(),records: s.sketches[0].records,+ population: s.sketches[0].population,}).collect();let scopes = vec![String::new(); rels.len()];@@ -3480,6 +3494,56 @@ mod clustering_key_tests {names.sort();assert_eq!(names, ["email", "id"]);}++ /// #1856: phase A stops at the sample, but the sketch keeps the real+ /// length of each array, and the sibling array is not a field of the+ /// primary rows.+ #[test]+ fn a_multi_array_json_file_keeps_each_arrays_population() {+ let dir = tempfile::tempdir().unwrap();+ let mut functions = Vec::new();+ for i in 0..40 {+ functions.push(format!(r#"{{"name":"f{i}","addr":{i}}}"#));+ }+ let body = format!(+ r#"{{"functions":[{}],"runs":[{{"label":"boot","start":1}},{{"label":"loop","start":2}},{{"label":"done","start":3}}]}}"#,+ functions.join(",")+ );+ let path = dir.path().join("subsystem_map.json");+ std::fs::write(&path, body).unwrap();+ let size = std::fs::metadata(&path).unwrap().len();+ let budget = extract::pdf::ExtractionSpoolBudget::new(0, 0);+ let progress = Progress::silent();+ let meter = estimate::Meter::new();+ let ctx = PhaseAContext {+ state_dir: dir.path(),+ budget: &budget,+ capacity_warning: None,+ progress: &progress,+ meter: &meter,+ tally: &ScanTally::default(),+ };+ let scanned = scan_file(&path, size, "d0", &ctx, 2, 2 << 30, false);+ let primary = scanned+ .sketches+ .iter()+ .find(|s| s.group.is_none())+ .expect("primary array");+ assert_eq!(primary.records, 2, "the sample stops at 2");+ assert_eq!(primary.population, 40, "the array length is the population");+ assert!(+ !primary.fields.contains_key("runs"),+ "sibling array leaked into the primary fields: {:?}",+ primary.fields.keys().collect::<Vec<_>>()+ );+ let runs = scanned+ .sketches+ .iter()+ .find(|s| s.group.as_deref() == Some("runs"))+ .expect("runs group");+ assert_eq!(runs.records, 2);+ assert_eq!(runs.population, 3);+ }}#[cfg(test)]@@ -3567,6 +3631,116 @@ mod phase_a_grouping_tests {build_phase_a(root, &files, &keys, &digests, Vec::new(), &ctx, cfg).plan}+ /// #1978: the semantic cap sees the array's real length, not the 500-row+ /// sample. 100_001 short prose rows elect `semantic_text` on the sample+ /// and must fall back, with the warning on the field.+ #[test]+ fn a_long_prose_column_past_the_record_cap_falls_back_to_text() {+ let dir = tempfile::tempdir().unwrap();+ let note = "The connection pool retries every failed handshake with backoff. \+ Each worker owns one socket and never shares it across threads.";+ let n = infer::SEMANTIC_DATA_MAX_RECORDS + 1;+ let mut body = String::with_capacity((n as usize) * (note.len() + 24));+ body.push('[');+ for i in 0..n {+ if i > 0 {+ body.push(',');+ }+ body.push_str(r#"{"id":"#);+ body.push_str(&i.to_string());+ body.push_str(r#","note":""#);+ body.push_str(note);+ body.push_str(r#""}"#);+ }+ body.push(']');+ std::fs::write(dir.path().join("rows.json"), body).unwrap();++ let plan = plan_for(dir.path());+ let dataset = plan+ .datasets+ .iter()+ .find(|d| d.specs.iter().any(|s| s.name == "note"))+ .expect("rows dataset");+ assert!(dataset.semantic_field.is_none(), "{dataset:#?}");+ let spec = dataset.specs.iter().find(|s| s.name == "note").unwrap();+ assert_eq!(spec.es_type, "text");+ let notes = spec.notes.join(" ");+ assert!(notes.contains("--no-semantic"), "{notes}");+ assert!(notes.contains(&n.to_string()), "{notes}");+ }++ /// A prose column under both caps still elects semantic_text. The cap+ /// must not turn every JSON text field into plain text.+ #[test]+ fn a_short_prose_column_stays_semantic_text() {+ let dir = tempfile::tempdir().unwrap();+ let note = "The connection pool retries every failed handshake with backoff. \+ Each worker owns one socket and never shares it across threads.";+ let mut body = String::from("[");+ // More than a config document, so the file stays a data dataset.+ for i in 0..40 {+ if i > 0 {+ body.push(',');+ }+ body.push_str(&format!(r#"{{"id":{i},"note":"{note}"}}"#));+ }+ body.push(']');+ std::fs::write(dir.path().join("rows.json"), body).unwrap();+ let plan = plan_for(dir.path());+ let dataset = plan+ .datasets+ .iter()+ .find(|d| d.specs.iter().any(|s| s.name == "note"))+ .expect("rows dataset");+ assert_eq!(+ dataset.semantic_field.as_deref(),+ Some("note"),+ "{dataset:#?}"+ );+ }++ /// #1856: a file shaped like `subsystem_map.json` (one long array of small+ /// objects beside a second array) plans two data datasets. The sibling+ /// array is not a field, and the short rows do not elect semantic_text.+ #[test]+ fn a_multi_array_json_file_plans_one_dataset_per_array() {+ let dir = tempfile::tempdir().unwrap();+ let mut functions = Vec::new();+ for i in 0..40 {+ functions.push(format!(r#"{{"name":"fn{i}","addr":{i}}}"#));+ }+ let body = format!(+ r#"{{"generated":"2026-01-01","functions":[{}],"runs":[{{"label":"boot","start":1}},{{"label":"loop","start":2}},{{"label":"done","start":3}}]}}"#,+ functions.join(",")+ );+ std::fs::write(dir.path().join("subsystem_map.json"), body).unwrap();+ let plan = plan_for(dir.path());+ let data: Vec<_> = plan+ .datasets+ .iter()+ .filter(|d| d.family != "docs")+ .collect();+ assert!(+ data.iter().any(|d| d.group.is_none()),+ "primary array: {:?}",+ plan.datasets+ .iter()+ .map(|d| (&d.slug, &d.group, d.family.as_str()))+ .collect::<Vec<_>>()+ );+ assert!(+ data.iter().any(|d| d.group.as_deref() == Some("runs")),+ "runs array: {:?}",+ plan.datasets+ .iter()+ .map(|d| (&d.slug, &d.group))+ .collect::<Vec<_>>()+ );+ let primary = data.iter().find(|d| d.group.is_none()).unwrap();+ assert!(primary.semantic_field.is_none(), "{primary:#?}");+ assert!(primary.specs.iter().all(|s| s.name != "runs"));+ }+/// #1198 end to end through the real planner: `--code-analyzer code` puts/// `code` on the document dataset that holds source (README included —/// code and prose share the per-scope dataset), while the default leaves@@ -4102,6 +4276,7 @@ fn build_phase_a(fields: gs.fields,key_fields: gs.key_fields,records: gs.records,+ population: gs.population,});}}@@ -4136,6 +4311,7 @@ fn build_phase_a(sampled_total += sampled;}c.records += sampled_total;+ c.population += sampled_total;}// per-file assignments@@ -4167,6 +4343,17 @@ fn build_phase_a(for c in &clusters {let mut specs =infer::infer_fields_with_policy(&c.fields, c.records, cfg.no_semantic, c.is_docs);+ // Data datasets only. A document dataset elects every body carrier;+ // capping that would drop code and prose from the vector arm. The+ // count is the real row count, not the sample: phase A stops at+ // `cfg.sample` rows, and a 123_000-row array sampled at 500 looks+ // cheap if the cap only sees 500 (#1978).+ if !c.is_docs && !cfg.no_semantic {+ let population = c.population.max(c.records);+ if let Some(hit) = infer::cap_data_semantic(&mut specs, population) {+ pr.note(&infer::semantic_fallback_warning(&c.slug, &hit));+ }+ }// Unity script-link enrichment fields are stamped by the phase-B// pipeline (not the extractor), so inference never sees them —// register them here or they would be dynamic-mapped coarsely.@@ -7044,6 +7231,7 @@ fn run_index_report_inner(plan: cfg.budget_plan.as_deref(),approve: cfg.approve,duplicate_prefixes: &duplicate_prefixes,+ memory_budget: xerj_common::resource::memory_safe_zone_bytes(),});for line in &assessment.lines {if assessment.needs_choice {diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsindex 07f50b6a6..689f2f1d4 100644--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -767,6 +767,7 @@ mod tests {fields,key_fields: std::collections::HashSet::new(),records: 1,+ population: 1,}],junk: None,// Reconciliation re-samples from the source tree and publishes