AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
autoindex: dataset caps — fold tiny structured datasets, cap data datasets per run, true row counts (#2422) (#2430)
xerj-org6adf61275eee
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d04e93123..339e53386 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -174,6 +174,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0queries on stray digits. Re-index to drop them from existing indices.(#2423)+- `xerj autoindex` no longer makes one index for every small table or JSON+ file (#2422). On a 5 GB benchmark tree the old rules made 454 indices,+ two thirds of them under 16 rows. On a 300 MB test tree the same run now+ makes 34 indices instead of 228. Now:+ - A JSON, YAML, XML or HTML dataset with fewer than 16 rows, in at most+ 8 files, goes into the folder's document index (`<prefix>-docs`, or+ `<prefix>-<repo>-docs` inside a repository) and is searchable there as+ text. Every record still carries its file path. An HTML page indexed+ this way keeps its title and headings, and its table rows are in the+ body.+ - JSON files with several arrays (for example `grammar.json` or OpenAPI+ specs) now count toward the limit of 12 small datasets per folder or+ repository. Before, each array of such a file was its own index with no+ limit. A file's arrays stay together: all of them stay datasets, or the+ whole file goes to the document index.+ - One run makes at most 32 datasets. Past that, the smallest JSON, YAML,+ XML and HTML datasets go to the document index, and small CSV, JSONL and+ log datasets of the same folder or repository share one index per+ format, so their rows stay rows. Datasets with 1024 rows or more, SQL+ tables and spreadsheet sheets always keep their own index, so a run can+ still go over 32 with those. Rows of large CSV, JSONL and log files are+ counted in full for this, not only the 500 rows read as a sample.+ - A file is moved whole or not at all, so a dataset that stays never+ loses rows.+- **Indexing is fast again under a memory cap** (issue[#2438](https://github.com/xerj-org/xerj/issues/2438), part of[#2423](https://github.com/xerj-org/xerj/issues/2423)). Since rc.78 every@@ -390,6 +415,13 @@ samuelmbabhazi, elewashy, WYSTAF, kamwoh, mgtf, mariomarcolongo and Jun516.### Upgrade notes+- #2422: an existing autoindex keeps the datasets it already has. Later+ runs on the same folder use the new rules only for new files. To get the+ smaller set of indices, index the folder again with a new `--prefix` and a+ new `--state-dir`, check it, then delete the old indices with+ `xerj autoindex remove --prefix <old>`. `--fresh` does not do this for a+ `--no-graph` run that has already committed: it refuses, as before.+- **No downgrade after a case-distinguished field write** (PR[#2261](https://github.com/xerj-org/xerj/pull/2261)). Opening a datadirectory does not by itself change its format. The marker moves fromdiff --git a/engine/crates/xerj-autoindex/src/dataset.rs b/engine/crates/xerj-autoindex/src/dataset.rsindex 40eac1053..844b67be8 100644--- a/engine/crates/xerj-autoindex/src/dataset.rs+++ b/engine/crates/xerj-autoindex/src/dataset.rs@@ -21,9 +21,17 @@//! *demoted* to the scope's document dataset and indexed as a document//! (title/body), because its one-off key set is configuration, not a//! collection (#173: valkey's per-command JSON files made 382 such-//! "datasets"). A scope that still has more medium structured clusters than-//! [`MAX_STRUCTURED_DATASETS_PER_SCOPE`] folds the smallest into documents;-//! a cluster of [`DATASET_CAP_PROTECT_RECORDS`] or more stays.+//! "datasets"). Any JSON/YAML/XML/HTML cluster with fewer than+//! [`MIN_DATASET_RECORDS`] rows folds the same way, grouped or not: a page's+//! 5-row table is not an index. A scope that still has+//! more medium structured clusters than [`MAX_STRUCTURED_DATASETS_PER_SCOPE`]+//! folds the smallest into documents. A run keeps at most+//! [`MAX_DATASETS_PER_RUN`] data datasets in all: past it the smallest+//! self-describing clusters fold into documents, then the smallest CSV/JSONL/+//! log clusters merge into one shared dataset per scope and family (their+//! rows stay rows, and every row keeps its `path`). A cluster of+//! [`DATASET_CAP_PROTECT_RECORDS`] or more never moves. A file folds whole or+//! not at all, so no kept dataset loses a row.//!//! Only DATA-derived names take part in schema clustering. A name the//! extractor invented — `defs`, `symbols`, `title`, `page` — is not evidence@@ -51,16 +59,37 @@ pub const DOC_DEMOTE_MAX_FILES: usize = 8;pub const DOC_DEMOTE_MAX_RECORDS_PER_FILE: u64 = 32;/// After the small-config fold, one repository scope keeps at most this many-/// `json`/`yaml`/`xml` data datasets below [`DATASET_CAP_PROTECT_RECORDS`].-/// The smallest are folded into the document dataset until the scope is at-/// the cap. CSV, JSONL, logs and SQL are collections by nature and are never-/// folded by this cap.+/// `json`/`yaml`/`xml`/`html` data datasets below+/// [`DATASET_CAP_PROTECT_RECORDS`]. The smallest are folded into the document+/// dataset until the scope is at the cap. CSV, JSONL and logs are collections+/// by nature and are not counted here; SQL tables never fold.pub const MAX_STRUCTURED_DATASETS_PER_SCOPE: usize = 12;-/// A `json`/`yaml`/`xml` cluster with at least this many records is a real-/// collection. The per-scope cap never folds it.+/// A cluster with at least this many records is a real collection. Neither+/// cap folds it.pub const DATASET_CAP_PROTECT_RECORDS: u64 = 1024;+/// A data cluster of a [`foldable_family`] with fewer rows than this, across+/// all its files, is folded into the scope's document dataset. A separate+/// index for a 5-row page table costs every query over the prefix a shard+/// visit and holds nothing a document search does not find. On a 5 GB+/// benchmark tree, 278 of 357 JSON/HTML/XML/YAML datasets were this small.+pub const MIN_DATASET_RECORDS: u64 = 16;++/// One run aims for at most this many data datasets (document datasets are+/// not counted), over all scopes (#2422: one 5 GB folder made 454 indices,+/// and a search over the prefix visits every one). Past the cap the smallest+/// clusters below [`DATASET_CAP_PROTECT_RECORDS`] rows go first:+///+/// 1. a [`foldable_family`] cluster folds into its scope's documents;+/// 2. a [`mergeable_family`] cluster (CSV, JSONL, logs) merges into one+/// shared dataset per scope and family, so its rows stay rows.+///+/// A protected collection, a workbook sheet, an SQL table and a grouped+/// (multi-array) file never move, so a tree with more of those than the+/// cap keeps them all and ends above it.+pub const MAX_DATASETS_PER_RUN: usize = 32;+/// A small `json`/`yaml`/`xml` cluster (no sub-file group) is configuration,/// not a dataset: few files, and not many records in each file.pub(crate) fn small_structured_config(c: &Cluster) -> bool {@@ -132,6 +161,35 @@ pub(crate) fn demotable_family(f: Family) -> bool {matches!(f, Family::Json | Family::Yaml | Family::Xml)}+/// May a data cluster of this family fold into the document dataset by size+/// or by a cap? Self-describing formats only — the shapes one tree repeats+/// per file: a page's table, a config's keys, one small JSON array. Each+/// renders as useful text through `extract_as_document` (an HTML page+/// through the HTML renderer). CSV, JSONL and logs are collections by+/// nature, and workbooks, SQLite and SQL carry real table schemas: they+/// stay datasets.+pub(crate) fn foldable_family(f: Family) -> bool {+ matches!(f, Family::Json | Family::Yaml | Family::Xml | Family::Html)+}++/// May a data cluster of this family merge, past [`MAX_DATASETS_PER_RUN`],+/// with others of its scope and family into one shared dataset? Line-record+/// formats only: every record is a row of its own and the merged mapping is+/// the union of the members' fields. Workbooks and SQL keep their tables.+pub(crate) fn mergeable_family(f: Family) -> bool {+ matches!(f, Family::Csv | Family::Jsonl | Family::Logs)+}++/// A foldable data cluster too small to be a dataset of its own. A schema+/// that recurs across more than [`DOC_DEMOTE_MAX_FILES`] files is a real+/// collection however few rows each file holds, and stays.+pub(crate) fn below_min_dataset(c: &Cluster) -> bool {+ foldable_family(c.family)+ && !c.members.is_empty()+ && c.members.len() <= DOC_DEMOTE_MAX_FILES+ && c.population < MIN_DATASET_RECORDS+}+/// Cluster sketches into datasets. `rels` and `scopes` are indexed by/// `Sketch::file_idx`: `rels` are root-relative paths (slug naming), `scopes`/// the repository scope of each file ("" = root) — see `module` docs.@@ -251,64 +309,251 @@ impl Clusterer {/// Demote, cap and name the clusters. `rels` is indexed by file index.pub fn finish(self, rels: &[String]) -> Vec<Cluster> {- let Clusterer { mut docs, data } = self;- // Demote small config clusters — one record each, or a short multi-doc- // YAML/JSON/XML file — to their scope's document dataset. Their sampled- // fields are dropped — the caller re-samples the demoted files through- // the document renderer, which is also what indexes them.+ let Clusterer { mut docs, mut data } = self;+ // Fold what is not a dataset into the scope's document dataset. Three+ // rules choose the clusters; a fourth keeps files whole://- // A file that also produced a grouped dataset (a second top-level JSON- // array) must not be folded: folding marks the whole file `as_document`- // and phase B then drops the grouped rows.- let grouped_files: HashSet<usize> = data- .iter()- .filter(|c| c.group.is_some())- .flat_map(|c| c.members.iter().copied())+ // 1. A small config cluster — one record each, or a short multi-doc+ // YAML/JSON/XML file — is configuration (#173).+ // 2. A JSON/YAML/XML/HTML cluster below `MIN_DATASET_RECORDS` rows is+ // not a collection anyone queries as one: a page's 5-row table, one+ // tiny grouped JSON array. Each used to become its own index (one+ // 5 GB tree → 454 indices, two thirds of them under 16 rows).+ // 3. Caps on the same families below `DATASET_CAP_PROTECT_RECORDS`+ // rows: at most `MAX_STRUCTURED_DATASETS_PER_SCOPE` per scope, and+ // the run as a whole at `MAX_DATASETS_PER_RUN`. The smallest fold+ // first; past the run cap, small CSV/JSONL/log clusters then merge+ // per scope and family (below).+ // 4. Folding marks the whole file `as_document`, so phase B indexes the+ // file as one document and none of its rows. A file folds only when+ // EVERY cluster it is a member of folds; otherwise all of its+ // clusters stay, and no row of a kept dataset is dropped.+ //+ // Folded members' sampled fields are dropped — the caller re-samples+ // them through the document renderer, which is also what indexes them.+ let n = data.len();+ let mut fold = vec![false; n];+ for (i, c) in data.iter().enumerate() {+ fold[i] = small_structured_config(c) || below_min_dataset(c);+ }+ // Clusters of each file. A file that also belongs to an unfoldable+ // cluster (a large collection, an SQL table, a workbook sheet) never+ // folds: its unit below is not a cap candidate.+ let mut file_clusters: HashMap<usize, Vec<usize>> = HashMap::new();+ for (i, c) in data.iter().enumerate() {+ for &m in &c.members {+ file_clusters.entry(m).or_default().push(i);+ }+ }+ let foldable_at_all =+ |c: &Cluster| foldable_family(c.family) && c.population < DATASET_CAP_PROTECT_RECORDS;+ // The caps move FILES, so they work on units of clusters joined by a+ // shared file: a multi-array JSON file (grammar.json, an OpenAPI spec)+ // and every grouped cluster of it. On main the caps skipped grouped+ // clusters outright (`c.group.is_none()`), and those were most of the+ // 454 indices in #2422. A unit folds whole, and only when every cluster+ // in it may fold and their rows together are below the protect size.+ let mut parent: Vec<usize> = (0..n).collect();+ fn root(p: &mut [usize], mut i: usize) -> usize {+ while p[i] != i {+ p[i] = p[p[i]];+ i = p[i];+ }+ i+ }+ for cs in file_clusters.values() {+ for w in cs.windows(2) {+ let (x, y) = (root(&mut parent, w[0]), root(&mut parent, w[1]));+ if x != y {+ parent[x.max(y)] = x.min(y);+ }+ }+ }+ let mut units: BTreeMap<usize, Vec<usize>> = BTreeMap::new();+ for i in 0..n {+ let r = root(&mut parent, i);+ units.entry(r).or_default().push(i);+ }+ // A unit with any cluster left unfolded keeps ALL its clusters (rule 4+ // unfolds the rest of its files), so a kept unit counts u.len().+ let mut total_live = units+ .values()+ .filter(|u| u.iter().any(|&j| !fold[j]))+ .map(Vec::len)+ .sum::<usize>();+ // (clusters, rows, first file, members) per foldable unit.+ let mut cand: Vec<(usize, u64, usize, Vec<usize>)> = units+ .into_values()+ .filter_map(|u| {+ if u.iter().all(|&j| fold[j]) || !u.iter().all(|&j| foldable_at_all(&data[j])) {+ return None;+ }+ let rows: u64 = u.iter().map(|&j| data[j].population).sum();+ if rows >= DATASET_CAP_PROTECT_RECORDS {+ return None;+ }+ let first = u+ .iter()+ .flat_map(|&j| data[j].members.iter().copied())+ .min()+ .unwrap_or(usize::MAX);+ Some((u.len(), rows, first, u))+ }).collect();- let mut kept: Vec<Cluster> = Vec::with_capacity(data.len());- let mut capped: BTreeMap<String, Vec<Cluster>> = BTreeMap::new();- for c in data {- let shares_grouped_sibling = c.members.iter().any(|m| grouped_files.contains(m));- if small_structured_config(&c) && !shares_grouped_sibling {- let d = docs_cluster(&mut docs, &c.scope);- d.members.extend(c.members.iter().copied());- d.demoted.extend(c.members);- continue;+ // Largest collections stay. Ties break toward the earlier file so the+ // fold does not depend on hash order. Size is the real row count, so a+ // file sampled at 500 does not tie with a 100-row file that happened to+ // fit in the sample.+ cand.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.cmp(&a.0)).then(a.2.cmp(&b.2)));+ let order = |a: &usize, b: &usize| {+ let (a, b) = (&data[*a], &data[*b]);+ b.population+ .cmp(&a.population)+ .then(b.members.len().cmp(&a.members.len()))+ .then_with(|| {+ let am = a.members.iter().copied().min().unwrap_or(usize::MAX);+ let bm = b.members.iter().copied().min().unwrap_or(usize::MAX);+ am.cmp(&bm)+ })+ };+ // Per-scope cap on small self-describing datasets.+ let mut kept_in_scope: HashMap<String, usize> = HashMap::new();+ let mut folded_unit = vec![false; cand.len()];+ for (k, (live, _, _, u)) in cand.iter().enumerate() {+ let scope = data[u[0]].scope.clone();+ let kept = kept_in_scope.entry(scope).or_default();+ if *kept + live > MAX_STRUCTURED_DATASETS_PER_SCOPE {+ for &j in u {+ fold[j] = true;+ }+ folded_unit[k] = true;+ total_live -= live;+ } else {+ *kept += live;+ }+ }+ // Per-run cap, all scopes together (#2422). The fold half: the smallest+ // self-describing units go to documents until the run is at the cap.+ // The merge half runs after rule 4, on what is left.+ let mut live = total_live;+ for (k, (n_live, _, _, u)) in cand.iter().enumerate().rev() {+ if live <= MAX_DATASETS_PER_RUN {+ break;}- // Candidates for the per-scope cap: structured formats, no sub-file- // group, and not a large collection. Everything else stays.- if demotable_family(c.family)- && c.group.is_none()- && c.population < DATASET_CAP_PROTECT_RECORDS- && !shares_grouped_sibling- {- capped.entry(c.scope.clone()).or_default().push(c);+ if folded_unit[k] {continue;}- kept.push(c);+ for &j in u {+ fold[j] = true;+ }+ live -= n_live;}- for (scope, mut group) in capped {- // Largest collections stay. Ties break toward the earlier file so- // the fold does not depend on hash order. Size is the real row- // count, so a file sampled at 500 does not tie with a 100-row file- // that happened to fit in the sample.- group.sort_by(|a, b| {- b.population- .cmp(&a.population)- .then(b.members.len().cmp(&a.members.len()))- .then_with(|| {- let am = a.members.iter().copied().min().unwrap_or(usize::MAX);- let bm = b.members.iter().copied().min().unwrap_or(usize::MAX);- am.cmp(&bm)- })- });- let keep_n = MAX_STRUCTURED_DATASETS_PER_SCOPE.min(group.len());- for c in group.drain(keep_n..) {- let d = docs_cluster(&mut docs, &scope);- d.members.extend(c.members.iter().copied());- d.demoted.extend(c.members);+ // Rule 4, to a fixed point: unfolding only ever clears flags.+ loop {+ let mut changed = false;+ for i in 0..n {+ if fold[i]+ && data[i]+ .members+ .iter()+ .any(|m| file_clusters[m].iter().any(|&j| !fold[j]))+ {+ fold[i] = false;+ changed = true;+ }+ }+ if !changed {+ break;+ }+ }+ // The merge half of the per-run cap. Still over it, the smallest+ // CSV/JSONL/log clusters (no sub-file group, below the protect size,+ // whose files hold no other cluster) merge into ONE dataset per scope+ // and family. The first candidate of a bucket becomes the shared+ // dataset; each later one joins it and saves an index.+ let mut merge_into: Vec<Option<usize>> = vec![None; n];+ let mut live = fold.iter().filter(|f| !**f).count();+ if live > MAX_DATASETS_PER_RUN {+ let mut cands: Vec<usize> = (0..n)+ .filter(|&i| {+ let c = &data[i];+ !fold[i]+ && mergeable_family(c.family)+ && c.group.is_none()+ && c.population < DATASET_CAP_PROTECT_RECORDS+ && c.members.iter().all(|m| file_clusters[m].len() == 1)+ })+ .collect();+ cands.sort_by(order);+ let mut shared: BTreeMap<(String, &'static str), usize> = BTreeMap::new();+ for &i in cands.iter().rev() {+ if live <= MAX_DATASETS_PER_RUN {+ break;+ }+ let key = (data[i].scope.clone(), data[i].family.as_str());+ match shared.get(&key) {+ Some(&t) => {+ merge_into[i] = Some(t);+ live -= 1;+ }+ None => {+ shared.insert(key, i);+ }+ }+ }+ }+ for i in 0..n {+ let Some(t) = merge_into[i] else { continue };+ let src = std::mem::replace(+ &mut data[i],+ Cluster {+ family: Family::Code,+ group: None,+ members: Vec::new(),+ fields: HashMap::new(),+ key_fields: HashSet::new(),+ records: 0,+ population: 0,+ slug: String::new(),+ scope: String::new(),+ is_docs: false,+ demoted: Vec::new(),+ },+ );+ let dst = &mut data[t];+ dst.members.extend(src.members);+ dst.records += src.records;+ dst.population += src.population;+ dst.key_fields.extend(src.key_fields);+ for (k, acc) in src.fields {+ match dst.fields.get_mut(&k) {+ Some(existing) => existing.merge(&acc),+ None => {+ dst.fields.insert(k, acc);+ }+ }+ }+ }+ let mut kept: Vec<Cluster> = Vec::with_capacity(n);+ let mut folded_files: HashSet<usize> = HashSet::new();+ for (i, c) in data.into_iter().enumerate() {+ if merge_into[i].is_some() {+ continue;+ }+ if !fold[i] {+ kept.push(c);+ continue;+ }+ let d = docs_cluster(&mut docs, &c.scope);+ for m in c.members {+ // A file with several folded clusters is one document.+ if folded_files.insert(m) {+ if !d.members.contains(&m) {+ d.members.push(m);+ }+ d.demoted.push(m);+ }}- kept.extend(group);}// Docs slugs are scope-derived — independent of which other files exist,@@ -1139,7 +1384,9 @@ mod tests {}/// Folding the primary array of a multi-array file would mark the whole- /// file as a document and drop the sibling array's rows.+ /// file as a document and drop the sibling array's rows. Since #2422 the+ /// file's two arrays count toward the scope cap as one unit; this unit is+ /// the largest, so it stays whole and the single-array files fold.#[test]fn a_grouped_sibling_keeps_its_file_out_of_the_structured_cap() {let mut sketches = vec![prose_sketch(0)];@@ -1179,6 +1426,56 @@ mod tests {.any(|c| { c.group.is_none() && !c.is_docs && c.members.contains(&primary_idx) }));}+ /// #2422 root cause: the per-scope cap skipped every grouped cluster, so+ /// a tree of multi-array JSON files (tree-sitter `grammar.json`, OpenAPI+ /// specs) kept one index per array. Grouped clusters now count toward+ /// the cap; a file's arrays fold together, so a kept file keeps all of+ /// its arrays and a folded one is one document.+ #[test]+ fn grouped_arrays_count_toward_the_scope_cap_and_fold_per_file() {+ let mut sketches = vec![prose_sketch(0)];+ let mut rels = vec!["README.md".to_string()];+ let files = 20;+ for f in 1..=files {+ rels.push(format!("grammars/g{f}/grammar.json"));+ sketches.push(data_sketch_n(+ f,+ Family::Json,+ &[&format!("name{f}"), "word"],+ 20,+ ));+ let mut g = data_sketch_n(+ f,+ Family::Json,+ &[&format!("rule{f}"), "kind"],+ 20 + f as u64,+ );+ g.group = Some(format!("rules{f}"));+ sketches.push(g);+ }+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ assert_eq!(data.len(), MAX_STRUCTURED_DATASETS_PER_SCOPE, "{data:#?}");+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");+ let kept_files = MAX_STRUCTURED_DATASETS_PER_SCOPE / 2;+ assert_eq!(+ docs.demoted.len(),+ files - kept_files,+ "one document per folded file"+ );+ for c in &data {+ let f = c.members[0];+ assert!(!docs.demoted.contains(&f));+ assert_eq!(+ data.iter().filter(|d| d.members == vec![f]).count(),+ 2,+ "both arrays kept"+ );+ }+ // The largest files (by rows) are the ones kept.+ assert!(data.iter().all(|c| c.members[0] > files - kept_files));+ }+/// Scope is a hard boundary for data schemas too: the same CSV schema in/// two repositories is two datasets (per-repo corpora stay self-contained).#[test]@@ -1256,4 +1553,171 @@ mod tests {let after: Vec<String> = untouched.iter().map(|c| c.slug.clone()).collect();assert_eq!(before, after, "the default prefix must not rename slugs");}++ /// One page per table used to be one index per page: a tree of 200 HTML+ /// reports with 5-row tables became 200 indices of a few documents each.+ /// A page table below `MIN_DATASET_RECORDS` rows folds into the scope's+ /// documents; a real table stays a dataset.+ #[test]+ fn small_html_tables_fold_into_documents_and_a_real_table_stays() {+ let mut sketches = vec![prose_sketch(0)];+ let mut rels = vec!["README.md".to_string()];+ for i in 1..=40 {+ rels.push(format!("reports/r{i}.html"));+ sketches.push(data_sketch_n(+ i,+ Family::Html,+ &[&format!("measure{i}"), "value"],+ 5,+ ));+ }+ rels.push("reports/table.html".into());+ sketches.push(data_sketch_n(+ 41,+ Family::Html,+ &["crate", "downloads"],+ 400,+ ));+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");+ assert_eq!(docs.demoted.len(), 40, "{clusters:#?}");+ assert_eq!(docs.members.len(), 41, "README + 40 folded pages");+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ assert_eq!(data.len(), 1, "{data:#?}");+ assert_eq!(data[0].members, vec![41]);+ }++ /// A multi-array JSON file folds only whole. When every array is tiny the+ /// file is one document (listed once, not once per array); when one+ /// array is a real collection, every array of the file stays a dataset,+ /// so no kept dataset loses rows.+ #[test]+ fn a_multi_array_file_folds_whole_or_not_at_all() {+ let mut sketches = vec![prose_sketch(0)];+ let rels: Vec<String> = vec!["README.md".into(), "tiny.json".into(), "mixed.json".into()];+ sketches.push(data_sketch_n(1, Family::Json, &["a", "b"], 3));+ let mut g = data_sketch_n(1, Family::Json, &["c", "d"], 2);+ g.group = Some("rules".into());+ sketches.push(g);+ sketches.push(data_sketch_n(2, Family::Json, &["x", "y"], 600));+ let mut g = data_sketch_n(2, Family::Json, &["p", "q"], 2);+ g.group = Some("notes".into());+ sketches.push(g);+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");+ assert_eq!(docs.demoted, vec![1], "{clusters:#?}");+ assert_eq!(docs.members.iter().filter(|&&m| m == 1).count(), 1);+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ assert_eq!(data.len(), 2, "both arrays of mixed.json stay: {data:#?}");+ assert!(data.iter().all(|c| c.members == vec![2]));+ }++ /// The per-scope cap alone let a tree of many repositories keep 12 small+ /// datasets in each. One run keeps at most `MAX_DATASETS_PER_RUN` data+ /// datasets: the protected collection and the CSV count toward it but+ /// never fold, and the largest small JSON datasets fill the rest.+ #[test]+ fn a_run_keeps_at_most_the_run_cap_of_small_structured_datasets() {+ let mut sketches = Vec::new();+ let mut rels = Vec::new();+ let mut scopes = Vec::new();+ let repos = 10;+ let per_repo = MAX_STRUCTURED_DATASETS_PER_SCOPE;+ for r in 0..repos {+ for k in 0..per_repo {+ let idx = rels.len();+ rels.push(format!("repo{r}/f{k}.json"));+ scopes.push(format!("repo{r}"));+ sketches.push(data_sketch_n(+ idx,+ Family::Json,+ &[&format!("k{r}_{k}")],+ 40 + idx as u64,+ ));+ }+ }+ let big = rels.len();+ rels.push("repo0/events.json".into());+ scopes.push("repo0".into());+ sketches.push(data_sketch_n(+ big,+ Family::Json,+ &["evt"],+ DATASET_CAP_PROTECT_RECORDS,+ ));+ let csv = rels.len();+ rels.push("repo0/rows.csv".into());+ scopes.push("repo0".into());+ sketches.push(data_sketch_n(csv, Family::Csv, &["a", "b"], 40));+ let clusters = cluster(sketches, &rels, &scopes);+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ assert_eq!(data.len(), MAX_DATASETS_PER_RUN);+ let small: Vec<&&Cluster> = data+ .iter()+ .filter(|c| c.family == Family::Json && c.population < DATASET_CAP_PROTECT_RECORDS)+ .collect();+ let keep_small = MAX_DATASETS_PER_RUN - 2;+ assert_eq!(small.len(), keep_small);+ let smallest_kept = small.iter().map(|c| c.population).min().unwrap();+ assert_eq!(+ smallest_kept,+ 40 + (repos * per_repo - keep_small) as u64,+ "the largest are the ones kept"+ );+ assert!(clusters.iter().any(|c| c.members == vec![big]));+ assert!(clusters+ .iter()+ .any(|c| c.members == vec![csv] && !c.is_docs));+ let demoted: usize = clusters.iter().map(|c| c.demoted.len()).sum();+ assert_eq!(demoted, repos * per_repo - keep_small);+ }++ /// Past the run cap, small CSV/JSONL datasets are not turned into text:+ /// they merge into one dataset per scope and family, every row still a+ /// row. A protected CSV stays as it is.+ #[test]+ fn past_the_run_cap_small_record_datasets_merge_per_scope_and_family() {+ let mut sketches = Vec::new();+ let mut rels = Vec::new();+ let mut scopes = Vec::new();+ let n_csv = MAX_DATASETS_PER_RUN + 10;+ for k in 0..n_csv {+ let idx = rels.len();+ rels.push(format!("bench/run{k}.csv"));+ scopes.push(String::new());+ sketches.push(data_sketch_n(+ idx,+ Family::Csv,+ &[&format!("col{k}"), &format!("val{k}")],+ 100 + idx as u64,+ ));+ }+ let big = rels.len();+ rels.push("bench/all.csv".into());+ scopes.push(String::new());+ sketches.push(data_sketch_n(+ big,+ Family::Csv,+ &["q", "ndcg"],+ DATASET_CAP_PROTECT_RECORDS * 4,+ ));+ let clusters = cluster(sketches, &rels, &scopes);+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ assert_eq!(data.len(), MAX_DATASETS_PER_RUN, "{data:#?}");+ assert!(+ clusters.iter().all(|c| c.demoted.is_empty()),+ "no CSV became text"+ );+ let total: usize = data.iter().map(|c| c.members.len()).sum();+ assert_eq!(total, n_csv + 1, "every file still lands in one dataset");+ assert!(data.iter().any(|c| c.members == vec![big]));+ let shared = data.iter().max_by_key(|c| c.members.len()).unwrap();+ assert_eq!(shared.members.len(), n_csv + 1 - (MAX_DATASETS_PER_RUN - 1));+ assert!(+ shared.fields.contains_key("col0"),+ "the union of the members' columns"+ );+ // The merge takes the smallest: file 0 (100 rows) is in the shared one.+ assert!(shared.members.contains(&0));+ }}diff --git a/engine/crates/xerj-autoindex/src/extract/html.rs b/engine/crates/xerj-autoindex/src/extract/html.rsindex ae21beff5..3d4d5ec07 100644--- a/engine/crates/xerj-autoindex/src/extract/html.rs+++ b/engine/crates/xerj-autoindex/src/extract/html.rs@@ -133,6 +133,49 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {Ok(stats)}+/// Render a page as ONE document — `{title, headings, body}` — even when it+/// has a dominant table. This is how a page whose table was too small to be+/// a dataset of its own is indexed after the fold into the scope's document+/// dataset (`dataset::MIN_DATASET_RECORDS`). The table's rows are already in+/// `body` as ` | `-joined lines, so no cell text is lost. The title comes+/// from `logical_path`, as in `extract_as_document_with_name`.+pub(crate) fn extract_document(+ content_path: &Path,+ logical_path: &Path,+ gzip: bool,+ sink: Sink,+) -> Result<ExtractStats> {+ let mut stats = ExtractStats::default();+ let Some(bytes) = super::read_whole(content_path, gzip, MAX_WHOLE_FILE)? else {+ stats.junk += 1;+ return Ok(stats);+ };+ let (text, _) = crate::sniff::decode_text(&bytes);+ let doc = parse(&text);+ if doc.title.trim().is_empty() && doc.headings.is_empty() && doc.body.trim().is_empty() {+ stats.junk += 1;+ return Ok(stats);+ }+ let title = if doc.title.trim().is_empty() {+ doc.headings+ .first()+ .cloned()+ .unwrap_or_else(|| stem_of(logical_path))+ } else {+ doc.title.trim().to_string()+ };+ emit_document(+ &title,+ &doc.headings,+ doc.body.trim(),+ None,+ MAX_RECORDS_PER_FILE,+ sink,+ &mut stats,+ );+ Ok(stats)+}+fn stem_of(path: &Path) -> String {path.file_stem().map(|s| s.to_string_lossy().to_string())@@ -1258,4 +1301,28 @@ function notCss() {}assert!(stats.records >= 4, "{name} records {}", stats.records);}}++ /// A page whose small table folded into the document dataset is indexed+ /// as the page: its title, and a body holding the table's rows as text,+ /// with no markup and no per-row records.+ #[test]+ fn a_folded_table_page_renders_as_one_tag_free_document() {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("report.html");+ let page = table_html(Some(["Measure", "Value"]), 5)+ .replace("<html>", "<html><head><title>Port audit</title></head>");+ std::fs::write(&path, page).unwrap();+ let mut recs = Vec::new();+ let stats = crate::extract::extract_as_document(&path, false, &mut |r| {+ recs.push(r);+ true+ })+ .unwrap();+ assert_eq!(stats.records, 1, "{recs:?}");+ assert_eq!(recs[0].fields["title"], "Port audit");+ let body = body_of(&recs[0]);+ assert!(body.contains("item3 | 3"), "{body}");+ assert!(!body.contains('<'), "{body}");+ assert!(!recs[0].fields.contains_key("Measure"));+ }}diff --git a/engine/crates/xerj-autoindex/src/extract/mod.rs b/engine/crates/xerj-autoindex/src/extract/mod.rsindex f6c703960..81ffbbcd3 100644--- a/engine/crates/xerj-autoindex/src/extract/mod.rs+++ b/engine/crates/xerj-autoindex/src/extract/mod.rs@@ -199,6 +199,12 @@ pub fn extract_as_document_with_name(if skip_phrase_index(logical_path, content_path, gzip) {return emit_phrase_skip(logical_path, sink);}+ // A folded HTML page is rendered as the page, not as its markup: the+ // same title/headings/body an HTML document gets, with its table rows in+ // the body. Raw text would index tags and scripts.+ if is_html_name(logical_path) {+ return html::extract_document(content_path, logical_path, gzip, sink);+ }let mut stats = ExtractStats::default();let Some(bytes) = read_whole(content_path, gzip, MAX_WHOLE_FILE)? else {stats.junk += 1;@@ -225,6 +231,17 @@ pub fn extract_as_document_with_name(Ok(stats)}+/// `.html`/`.htm`/`.xhtml`, optionally gzipped, by the logical name.+fn is_html_name(logical: &Path) -> bool {+ let name = logical+ .file_name()+ .and_then(|s| s.to_str())+ .unwrap_or("")+ .to_ascii_lowercase();+ let name = name.strip_suffix(".gz").unwrap_or(&name);+ name.ends_with(".html") || name.ends_with(".htm") || name.ends_with(".xhtml")+}+/// Bytes of a JS/CSS prefix read to decide "this is a bundle, not source".const MINIFIED_PREFIX: u64 = 8192;/// Below this, a one-line file is a small module, not the huge bundle thediff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 3e572dea1..8fbbb8f76 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -4258,6 +4258,36 @@ fn scan_file(}}).collect();+ // CSV, JSONL and logs declare no length, so `records` is the sample: it+ // stops at `sample` rows (500 by default) or at the sample byte cap. A+ // 100_000-row CSV then looked like a 500-row one, and the dataset caps+ // (`dataset::DATASET_CAP_PROTECT_RECORDS`) merged it into a shared index+ // as if it were small. When the sample may have stopped early, count on+ // with the same extractor up to the protect threshold. That is enough to+ // decide the caps and stays bounded however large the file is.+ if matches!(sn.family, Family::Csv | Family::Jsonl | Family::Logs) {+ let threshold = dataset::DATASET_CAP_PROTECT_RECORDS;+ let sample_cut = |s: &GroupSketch| {+ s.population < threshold+ && (s.records >= sample as u64 || limit.is_some_and(|l| sn.gzip || size > l))+ };+ if out.sketches.iter().any(sample_cut) {+ let groups_seen = out.sketches.len();+ let mut counts: HashMap<Option<String>, u64> = HashMap::new();+ let mut count = |rec: extract::RawRecord| -> bool {+ *counts.entry(rec.group).or_default() += 1;+ counts.len() < groups_seen || counts.values().any(|n| *n < threshold)+ };+ // A read error part-way still leaves a lower bound: every+ // record counted was really there.+ let _ = extract::extract(path, &sn, None, &mut count);+ for s in &mut out.sketches {+ if let Some(&n) = counts.get(&s.group) {+ s.population = s.population.max(n);+ }+ }+ }+ }out.sketches.sort_by(|a, b| a.group.cmp(&b.group));out.sniffed = Some(sn);out@@ -4445,6 +4475,108 @@ mod clustering_key_tests {assert_eq!(runs.records, 2);assert_eq!(runs.population, 3);}++ fn csv_body(tag: &str, rows: usize) -> String {+ let mut b = format!("{tag}_id,{tag}_name,{tag}_val\n");+ for i in 0..rows {+ b.push_str(&format!("{i},name{i},{}\n", i * 3));+ }+ b+ }++ fn jsonl_body(tag: &str, rows: usize) -> String {+ let mut b = String::new();+ for i in 0..rows {+ b.push_str(&format!(+ "{{\"{tag}_id\":{i},\"{tag}_name\":\"name{i}\",\"{tag}_val\":{}}}\n",+ i * 3+ ));+ }+ b+ }++ /// Line-record files declare no length, so the sample (500 rows) used to+ /// be the population and a 2000-row CSV/JSONL counted as small (#2422).+ /// Phase A now counts on to the protect threshold; a 10-row file still+ /// reports exactly its 10 rows.+ #[test]+ fn a_sampled_line_record_file_reports_rows_past_the_sample() {+ let dir = tempfile::tempdir().unwrap();+ for (name, body) in [+ ("big.csv", csv_body("b", 2000)),+ ("big.jsonl", jsonl_body("j", 2000)),+ ] {+ let s = scan(dir.path(), name, &body);+ assert_eq!(s.sketches.len(), 1, "{name}");+ let sk = &s.sketches[0];+ assert_eq!(sk.records, 500, "{name}: the sample still stops at 500");+ assert!(+ sk.population >= dataset::DATASET_CAP_PROTECT_RECORDS,+ "{name}: population {} is the sample, not the file",+ sk.population+ );+ }+ let small = scan(dir.path(), "small.csv", &csv_body("s", 10));+ assert_eq!(small.sketches[0].records, 10);+ assert_eq!(small.sketches[0].population, 10);+ }++ /// End to end through `cluster()`: past the run cap, 2000-row CSV and+ /// JSONL files keep their own datasets, and the 10-row CSVs still merge+ /// into one shared CSV dataset.+ #[test]+ fn past_the_run_cap_large_sampled_csv_and_jsonl_keep_their_own_index() {+ let dir = tempfile::tempdir().unwrap();+ let mut files: Vec<(String, String)> = Vec::new();+ let n_big = dataset::MAX_DATASETS_PER_RUN + 2;+ for i in 0..n_big {+ files.push((format!("big{i}.csv"), csv_body(&format!("b{i}"), 2000)));+ }+ files.push(("big.jsonl".into(), jsonl_body("j", 2000)));+ let n_small = 3;+ for i in 0..n_small {+ files.push((format!("small{i}.csv"), csv_body(&format!("s{i}"), 10)));+ }+ let mut rels = Vec::new();+ let mut sketches = Vec::new();+ for (idx, (name, body)) in files.iter().enumerate() {+ let s = scan(dir.path(), name, body);+ assert_eq!(s.sketches.len(), 1, "{name}");+ let sk = &s.sketches[0];+ sketches.push(dataset::Sketch {+ file_idx: idx,+ group: sk.group.clone(),+ family: s.sniffed.as_ref().unwrap().family,+ fields: sk.fields.clone(),+ key_fields: sk.key_fields.clone(),+ records: sk.records,+ population: sk.population,+ });+ rels.push(name.clone());+ }+ let scopes = vec![String::new(); rels.len()];+ let clusters = dataset::cluster(sketches, &rels, &scopes);+ let data: Vec<&dataset::Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();+ let home = |idx: usize| {+ data.iter()+ .find(|c| c.members.contains(&idx))+ .unwrap_or_else(|| panic!("{} has no data dataset: {clusters:#?}", rels[idx]))+ };+ // Every large file — the CSVs and the JSONL — is alone in its dataset.+ for (idx, rel) in rels.iter().enumerate().take(n_big + 1) {+ assert_eq!(home(idx).members, vec![idx], "{rel} was merged");+ }+ // The 10-row CSVs share one dataset, rows intact.+ let shared = home(n_big + 1);+ let mut members = shared.members.clone();+ members.sort();+ assert_eq!(+ members,+ (n_big + 1..n_big + 1 + n_small).collect::<Vec<_>>()+ );+ assert_eq!(shared.records, 10 * n_small as u64);+ assert_eq!(data.len(), n_big + 2);+ }}#[cfg(test)]diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsindex adff73908..49719464a 100644--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -298,10 +298,17 @@ pub(crate) fn reconcile_plan(.iter().any(|d| d.family == sniffed.family.as_str() && d.group.is_none());let new_document = docs_target && all_empty_key_fields;+ // A file too small to be a dataset (`dataset::MIN_DATASET_RECORDS`)+ // folds the same way a fresh plan folds it: a 3-row HTML table or a+ // 2-key JSON file joins docs rather than failing the run for want of a+ // frozen dataset of its own.let new_demoted_config = docs_target&& !all_empty_key_fields- && crate::dataset::demotable_family(sniffed.family)- && scan.sketches.iter().map(|s| s.records).sum::<u64>() == 1;+ && ((crate::dataset::demotable_family(sniffed.family)+ && scan.sketches.iter().map(|s| s.records).sum::<u64>() == 1)+ || (crate::dataset::foldable_family(sniffed.family)+ && scan.sketches.iter().map(|s| s.population).sum::<u64>()+ < crate::dataset::MIN_DATASET_RECORDS));let route_to_docs = new_document || new_demoted_config;// Only a demoted CONFIG re-samples through the document renderer (its