Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

autoindex: dataset caps — fold tiny structured datasets, cap data datasets per run, true row counts (#2422) (#2430)

xerj-org6adf61275eee
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d04e93123..339e53386 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -174,6 +174,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
queries on stray digits. Re-index to drop them from existing indices.
(#2423)
+- `xerj autoindex` no longer makes one index for every small table or JSON
+ file (#2422). On a 5 GB benchmark tree the old rules made 454 indices,
+ two thirds of them under 16 rows. On a 300 MB test tree the same run now
+ makes 34 indices instead of 228. Now:
+ - A JSON, YAML, XML or HTML dataset with fewer than 16 rows, in at most
+ 8 files, goes into the folder's document index (`<prefix>-docs`, or
+ `<prefix>-<repo>-docs` inside a repository) and is searchable there as
+ text. Every record still carries its file path. An HTML page indexed
+ this way keeps its title and headings, and its table rows are in the
+ body.
+ - JSON files with several arrays (for example `grammar.json` or OpenAPI
+ specs) now count toward the limit of 12 small datasets per folder or
+ repository. Before, each array of such a file was its own index with no
+ limit. A file's arrays stay together: all of them stay datasets, or the
+ whole file goes to the document index.
+ - One run makes at most 32 datasets. Past that, the smallest JSON, YAML,
+ XML and HTML datasets go to the document index, and small CSV, JSONL and
+ log datasets of the same folder or repository share one index per
+ format, so their rows stay rows. Datasets with 1024 rows or more, SQL
+ tables and spreadsheet sheets always keep their own index, so a run can
+ still go over 32 with those. Rows of large CSV, JSONL and log files are
+ counted in full for this, not only the 500 rows read as a sample.
+ - A file is moved whole or not at all, so a dataset that stays never
+ loses rows.
+
- **Indexing is fast again under a memory cap** (issue
[#2438](https://github.com/xerj-org/xerj/issues/2438), part of
[#2423](https://github.com/xerj-org/xerj/issues/2423)). Since rc.78 every
@@ -390,6 +415,13 @@ samuelmbabhazi, elewashy, WYSTAF, kamwoh, mgtf, mariomarcolongo and Jun516.
### Upgrade notes
+- #2422: an existing autoindex keeps the datasets it already has. Later
+ runs on the same folder use the new rules only for new files. To get the
+ smaller set of indices, index the folder again with a new `--prefix` and a
+ new `--state-dir`, check it, then delete the old indices with
+ `xerj autoindex remove --prefix <old>`. `--fresh` does not do this for a
+ `--no-graph` run that has already committed: it refuses, as before.
+
- **No downgrade after a case-distinguished field write** (PR
[#2261](https://github.com/xerj-org/xerj/pull/2261)). Opening a data
directory does not by itself change its format. The marker moves from
diff --git a/engine/crates/xerj-autoindex/src/dataset.rs b/engine/crates/xerj-autoindex/src/dataset.rs
index 40eac1053..844b67be8 100644
--- a/engine/crates/xerj-autoindex/src/dataset.rs
+++ b/engine/crates/xerj-autoindex/src/dataset.rs
@@ -21,9 +21,17 @@
//! *demoted* to the scope's document dataset and indexed as a document
//! (title/body), because its one-off key set is configuration, not a
//! collection (#173: valkey's per-command JSON files made 382 such
-//! "datasets"). A scope that still has more medium structured clusters than
-//! [`MAX_STRUCTURED_DATASETS_PER_SCOPE`] folds the smallest into documents;
-//! a cluster of [`DATASET_CAP_PROTECT_RECORDS`] or more stays.
+//! "datasets"). Any JSON/YAML/XML/HTML cluster with fewer than
+//! [`MIN_DATASET_RECORDS`] rows folds the same way, grouped or not: a page's
+//! 5-row table is not an index. A scope that still has
+//! more medium structured clusters than [`MAX_STRUCTURED_DATASETS_PER_SCOPE`]
+//! folds the smallest into documents. A run keeps at most
+//! [`MAX_DATASETS_PER_RUN`] data datasets in all: past it the smallest
+//! self-describing clusters fold into documents, then the smallest CSV/JSONL/
+//! log clusters merge into one shared dataset per scope and family (their
+//! rows stay rows, and every row keeps its `path`). A cluster of
+//! [`DATASET_CAP_PROTECT_RECORDS`] or more never moves. A file folds whole or
+//! not at all, so no kept dataset loses a row.
//!
//! Only DATA-derived names take part in schema clustering. A name the
//! extractor invented — `defs`, `symbols`, `title`, `page` — is not evidence
@@ -51,16 +59,37 @@ pub const DOC_DEMOTE_MAX_FILES: usize = 8;
pub const DOC_DEMOTE_MAX_RECORDS_PER_FILE: u64 = 32;
/// After the small-config fold, one repository scope keeps at most this many
-/// `json`/`yaml`/`xml` data datasets below [`DATASET_CAP_PROTECT_RECORDS`].
-/// The smallest are folded into the document dataset until the scope is at
-/// the cap. CSV, JSONL, logs and SQL are collections by nature and are never
-/// folded by this cap.
+/// `json`/`yaml`/`xml`/`html` data datasets below
+/// [`DATASET_CAP_PROTECT_RECORDS`]. The smallest are folded into the document
+/// dataset until the scope is at the cap. CSV, JSONL and logs are collections
+/// by nature and are not counted here; SQL tables never fold.
pub const MAX_STRUCTURED_DATASETS_PER_SCOPE: usize = 12;
-/// A `json`/`yaml`/`xml` cluster with at least this many records is a real
-/// collection. The per-scope cap never folds it.
+/// A cluster with at least this many records is a real collection. Neither
+/// cap folds it.
pub const DATASET_CAP_PROTECT_RECORDS: u64 = 1024;
+/// A data cluster of a [`foldable_family`] with fewer rows than this, across
+/// all its files, is folded into the scope's document dataset. A separate
+/// index for a 5-row page table costs every query over the prefix a shard
+/// visit and holds nothing a document search does not find. On a 5 GB
+/// benchmark tree, 278 of 357 JSON/HTML/XML/YAML datasets were this small.
+pub const MIN_DATASET_RECORDS: u64 = 16;
+
+/// One run aims for at most this many data datasets (document datasets are
+/// not counted), over all scopes (#2422: one 5 GB folder made 454 indices,
+/// and a search over the prefix visits every one). Past the cap the smallest
+/// clusters below [`DATASET_CAP_PROTECT_RECORDS`] rows go first:
+///
+/// 1. a [`foldable_family`] cluster folds into its scope's documents;
+/// 2. a [`mergeable_family`] cluster (CSV, JSONL, logs) merges into one
+/// shared dataset per scope and family, so its rows stay rows.
+///
+/// A protected collection, a workbook sheet, an SQL table and a grouped
+/// (multi-array) file never move, so a tree with more of those than the
+/// cap keeps them all and ends above it.
+pub const MAX_DATASETS_PER_RUN: usize = 32;
+
/// A small `json`/`yaml`/`xml` cluster (no sub-file group) is configuration,
/// not a dataset: few files, and not many records in each file.
pub(crate) fn small_structured_config(c: &Cluster) -> bool {
@@ -132,6 +161,35 @@ pub(crate) fn demotable_family(f: Family) -> bool {
matches!(f, Family::Json | Family::Yaml | Family::Xml)
}
+/// May a data cluster of this family fold into the document dataset by size
+/// or by a cap? Self-describing formats only — the shapes one tree repeats
+/// per file: a page's table, a config's keys, one small JSON array. Each
+/// renders as useful text through `extract_as_document` (an HTML page
+/// through the HTML renderer). CSV, JSONL and logs are collections by
+/// nature, and workbooks, SQLite and SQL carry real table schemas: they
+/// stay datasets.
+pub(crate) fn foldable_family(f: Family) -> bool {
+ matches!(f, Family::Json | Family::Yaml | Family::Xml | Family::Html)
+}
+
+/// May a data cluster of this family merge, past [`MAX_DATASETS_PER_RUN`],
+/// with others of its scope and family into one shared dataset? Line-record
+/// formats only: every record is a row of its own and the merged mapping is
+/// the union of the members' fields. Workbooks and SQL keep their tables.
+pub(crate) fn mergeable_family(f: Family) -> bool {
+ matches!(f, Family::Csv | Family::Jsonl | Family::Logs)
+}
+
+/// A foldable data cluster too small to be a dataset of its own. A schema
+/// that recurs across more than [`DOC_DEMOTE_MAX_FILES`] files is a real
+/// collection however few rows each file holds, and stays.
+pub(crate) fn below_min_dataset(c: &Cluster) -> bool {
+ foldable_family(c.family)
+ && !c.members.is_empty()
+ && c.members.len() <= DOC_DEMOTE_MAX_FILES
+ && c.population < MIN_DATASET_RECORDS
+}
+
/// Cluster sketches into datasets. `rels` and `scopes` are indexed by
/// `Sketch::file_idx`: `rels` are root-relative paths (slug naming), `scopes`
/// the repository scope of each file ("" = root) — see `module` docs.
@@ -251,64 +309,251 @@ impl Clusterer {
/// Demote, cap and name the clusters. `rels` is indexed by file index.
pub fn finish(self, rels: &[String]) -> Vec<Cluster> {
- let Clusterer { mut docs, data } = self;
- // Demote small config clusters — one record each, or a short multi-doc
- // YAML/JSON/XML file — to their scope's document dataset. Their sampled
- // fields are dropped — the caller re-samples the demoted files through
- // the document renderer, which is also what indexes them.
+ let Clusterer { mut docs, mut data } = self;
+ // Fold what is not a dataset into the scope's document dataset. Three
+ // rules choose the clusters; a fourth keeps files whole:
//
- // A file that also produced a grouped dataset (a second top-level JSON
- // array) must not be folded: folding marks the whole file `as_document`
- // and phase B then drops the grouped rows.
- let grouped_files: HashSet<usize> = data
- .iter()
- .filter(|c| c.group.is_some())
- .flat_map(|c| c.members.iter().copied())
+ // 1. A small config cluster — one record each, or a short multi-doc
+ // YAML/JSON/XML file — is configuration (#173).
+ // 2. A JSON/YAML/XML/HTML cluster below `MIN_DATASET_RECORDS` rows is
+ // not a collection anyone queries as one: a page's 5-row table, one
+ // tiny grouped JSON array. Each used to become its own index (one
+ // 5 GB tree → 454 indices, two thirds of them under 16 rows).
+ // 3. Caps on the same families below `DATASET_CAP_PROTECT_RECORDS`
+ // rows: at most `MAX_STRUCTURED_DATASETS_PER_SCOPE` per scope, and
+ // the run as a whole at `MAX_DATASETS_PER_RUN`. The smallest fold
+ // first; past the run cap, small CSV/JSONL/log clusters then merge
+ // per scope and family (below).
+ // 4. Folding marks the whole file `as_document`, so phase B indexes the
+ // file as one document and none of its rows. A file folds only when
+ // EVERY cluster it is a member of folds; otherwise all of its
+ // clusters stay, and no row of a kept dataset is dropped.
+ //
+ // Folded members' sampled fields are dropped — the caller re-samples
+ // them through the document renderer, which is also what indexes them.
+ let n = data.len();
+ let mut fold = vec![false; n];
+ for (i, c) in data.iter().enumerate() {
+ fold[i] = small_structured_config(c) || below_min_dataset(c);
+ }
+ // Clusters of each file. A file that also belongs to an unfoldable
+ // cluster (a large collection, an SQL table, a workbook sheet) never
+ // folds: its unit below is not a cap candidate.
+ let mut file_clusters: HashMap<usize, Vec<usize>> = HashMap::new();
+ for (i, c) in data.iter().enumerate() {
+ for &m in &c.members {
+ file_clusters.entry(m).or_default().push(i);
+ }
+ }
+ let foldable_at_all =
+ |c: &Cluster| foldable_family(c.family) && c.population < DATASET_CAP_PROTECT_RECORDS;
+ // The caps move FILES, so they work on units of clusters joined by a
+ // shared file: a multi-array JSON file (grammar.json, an OpenAPI spec)
+ // and every grouped cluster of it. On main the caps skipped grouped
+ // clusters outright (`c.group.is_none()`), and those were most of the
+ // 454 indices in #2422. A unit folds whole, and only when every cluster
+ // in it may fold and their rows together are below the protect size.
+ let mut parent: Vec<usize> = (0..n).collect();
+ fn root(p: &mut [usize], mut i: usize) -> usize {
+ while p[i] != i {
+ p[i] = p[p[i]];
+ i = p[i];
+ }
+ i
+ }
+ for cs in file_clusters.values() {
+ for w in cs.windows(2) {
+ let (x, y) = (root(&mut parent, w[0]), root(&mut parent, w[1]));
+ if x != y {
+ parent[x.max(y)] = x.min(y);
+ }
+ }
+ }
+ let mut units: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
+ for i in 0..n {
+ let r = root(&mut parent, i);
+ units.entry(r).or_default().push(i);
+ }
+ // A unit with any cluster left unfolded keeps ALL its clusters (rule 4
+ // unfolds the rest of its files), so a kept unit counts u.len().
+ let mut total_live = units
+ .values()
+ .filter(|u| u.iter().any(|&j| !fold[j]))
+ .map(Vec::len)
+ .sum::<usize>();
+ // (clusters, rows, first file, members) per foldable unit.
+ let mut cand: Vec<(usize, u64, usize, Vec<usize>)> = units
+ .into_values()
+ .filter_map(|u| {
+ if u.iter().all(|&j| fold[j]) || !u.iter().all(|&j| foldable_at_all(&data[j])) {
+ return None;
+ }
+ let rows: u64 = u.iter().map(|&j| data[j].population).sum();
+ if rows >= DATASET_CAP_PROTECT_RECORDS {
+ return None;
+ }
+ let first = u
+ .iter()
+ .flat_map(|&j| data[j].members.iter().copied())
+ .min()
+ .unwrap_or(usize::MAX);
+ Some((u.len(), rows, first, u))
+ })
.collect();
- let mut kept: Vec<Cluster> = Vec::with_capacity(data.len());
- let mut capped: BTreeMap<String, Vec<Cluster>> = BTreeMap::new();
- for c in data {
- let shares_grouped_sibling = c.members.iter().any(|m| grouped_files.contains(m));
- if small_structured_config(&c) && !shares_grouped_sibling {
- let d = docs_cluster(&mut docs, &c.scope);
- d.members.extend(c.members.iter().copied());
- d.demoted.extend(c.members);
- continue;
+ // Largest collections stay. Ties break toward the earlier file so the
+ // fold does not depend on hash order. Size is the real row count, so a
+ // file sampled at 500 does not tie with a 100-row file that happened to
+ // fit in the sample.
+ cand.sort_by(|a, b| b.1.cmp(&a.1).then(b.0.cmp(&a.0)).then(a.2.cmp(&b.2)));
+ let order = |a: &usize, b: &usize| {
+ let (a, b) = (&data[*a], &data[*b]);
+ b.population
+ .cmp(&a.population)
+ .then(b.members.len().cmp(&a.members.len()))
+ .then_with(|| {
+ let am = a.members.iter().copied().min().unwrap_or(usize::MAX);
+ let bm = b.members.iter().copied().min().unwrap_or(usize::MAX);
+ am.cmp(&bm)
+ })
+ };
+ // Per-scope cap on small self-describing datasets.
+ let mut kept_in_scope: HashMap<String, usize> = HashMap::new();
+ let mut folded_unit = vec![false; cand.len()];
+ for (k, (live, _, _, u)) in cand.iter().enumerate() {
+ let scope = data[u[0]].scope.clone();
+ let kept = kept_in_scope.entry(scope).or_default();
+ if *kept + live > MAX_STRUCTURED_DATASETS_PER_SCOPE {
+ for &j in u {
+ fold[j] = true;
+ }
+ folded_unit[k] = true;
+ total_live -= live;
+ } else {
+ *kept += live;
+ }
+ }
+ // Per-run cap, all scopes together (#2422). The fold half: the smallest
+ // self-describing units go to documents until the run is at the cap.
+ // The merge half runs after rule 4, on what is left.
+ let mut live = total_live;
+ for (k, (n_live, _, _, u)) in cand.iter().enumerate().rev() {
+ if live <= MAX_DATASETS_PER_RUN {
+ break;
}
- // Candidates for the per-scope cap: structured formats, no sub-file
- // group, and not a large collection. Everything else stays.
- if demotable_family(c.family)
- && c.group.is_none()
- && c.population < DATASET_CAP_PROTECT_RECORDS
- && !shares_grouped_sibling
- {
- capped.entry(c.scope.clone()).or_default().push(c);
+ if folded_unit[k] {
continue;
}
- kept.push(c);
+ for &j in u {
+ fold[j] = true;
+ }
+ live -= n_live;
}
- for (scope, mut group) in capped {
- // Largest collections stay. Ties break toward the earlier file so
- // the fold does not depend on hash order. Size is the real row
- // count, so a file sampled at 500 does not tie with a 100-row file
- // that happened to fit in the sample.
- group.sort_by(|a, b| {
- b.population
- .cmp(&a.population)
- .then(b.members.len().cmp(&a.members.len()))
- .then_with(|| {
- let am = a.members.iter().copied().min().unwrap_or(usize::MAX);
- let bm = b.members.iter().copied().min().unwrap_or(usize::MAX);
- am.cmp(&bm)
- })
- });
- let keep_n = MAX_STRUCTURED_DATASETS_PER_SCOPE.min(group.len());
- for c in group.drain(keep_n..) {
- let d = docs_cluster(&mut docs, &scope);
- d.members.extend(c.members.iter().copied());
- d.demoted.extend(c.members);
+ // Rule 4, to a fixed point: unfolding only ever clears flags.
+ loop {
+ let mut changed = false;
+ for i in 0..n {
+ if fold[i]
+ && data[i]
+ .members
+ .iter()
+ .any(|m| file_clusters[m].iter().any(|&j| !fold[j]))
+ {
+ fold[i] = false;
+ changed = true;
+ }
+ }
+ if !changed {
+ break;
+ }
+ }
+ // The merge half of the per-run cap. Still over it, the smallest
+ // CSV/JSONL/log clusters (no sub-file group, below the protect size,
+ // whose files hold no other cluster) merge into ONE dataset per scope
+ // and family. The first candidate of a bucket becomes the shared
+ // dataset; each later one joins it and saves an index.
+ let mut merge_into: Vec<Option<usize>> = vec![None; n];
+ let mut live = fold.iter().filter(|f| !**f).count();
+ if live > MAX_DATASETS_PER_RUN {
+ let mut cands: Vec<usize> = (0..n)
+ .filter(|&i| {
+ let c = &data[i];
+ !fold[i]
+ && mergeable_family(c.family)
+ && c.group.is_none()
+ && c.population < DATASET_CAP_PROTECT_RECORDS
+ && c.members.iter().all(|m| file_clusters[m].len() == 1)
+ })
+ .collect();
+ cands.sort_by(order);
+ let mut shared: BTreeMap<(String, &'static str), usize> = BTreeMap::new();
+ for &i in cands.iter().rev() {
+ if live <= MAX_DATASETS_PER_RUN {
+ break;
+ }
+ let key = (data[i].scope.clone(), data[i].family.as_str());
+ match shared.get(&key) {
+ Some(&t) => {
+ merge_into[i] = Some(t);
+ live -= 1;
+ }
+ None => {
+ shared.insert(key, i);
+ }
+ }
+ }
+ }
+ for i in 0..n {
+ let Some(t) = merge_into[i] else { continue };
+ let src = std::mem::replace(
+ &mut data[i],
+ Cluster {
+ family: Family::Code,
+ group: None,
+ members: Vec::new(),
+ fields: HashMap::new(),
+ key_fields: HashSet::new(),
+ records: 0,
+ population: 0,
+ slug: String::new(),
+ scope: String::new(),
+ is_docs: false,
+ demoted: Vec::new(),
+ },
+ );
+ let dst = &mut data[t];
+ dst.members.extend(src.members);
+ dst.records += src.records;
+ dst.population += src.population;
+ dst.key_fields.extend(src.key_fields);
+ for (k, acc) in src.fields {
+ match dst.fields.get_mut(&k) {
+ Some(existing) => existing.merge(&acc),
+ None => {
+ dst.fields.insert(k, acc);
+ }
+ }
+ }
+ }
+ let mut kept: Vec<Cluster> = Vec::with_capacity(n);
+ let mut folded_files: HashSet<usize> = HashSet::new();
+ for (i, c) in data.into_iter().enumerate() {
+ if merge_into[i].is_some() {
+ continue;
+ }
+ if !fold[i] {
+ kept.push(c);
+ continue;
+ }
+ let d = docs_cluster(&mut docs, &c.scope);
+ for m in c.members {
+ // A file with several folded clusters is one document.
+ if folded_files.insert(m) {
+ if !d.members.contains(&m) {
+ d.members.push(m);
+ }
+ d.demoted.push(m);
+ }
}
- kept.extend(group);
}
// Docs slugs are scope-derived — independent of which other files exist,
@@ -1139,7 +1384,9 @@ mod tests {
}
/// Folding the primary array of a multi-array file would mark the whole
- /// file as a document and drop the sibling array's rows.
+ /// file as a document and drop the sibling array's rows. Since #2422 the
+ /// file's two arrays count toward the scope cap as one unit; this unit is
+ /// the largest, so it stays whole and the single-array files fold.
#[test]
fn a_grouped_sibling_keeps_its_file_out_of_the_structured_cap() {
let mut sketches = vec![prose_sketch(0)];
@@ -1179,6 +1426,56 @@ mod tests {
.any(|c| { c.group.is_none() && !c.is_docs && c.members.contains(&primary_idx) }));
}
+ /// #2422 root cause: the per-scope cap skipped every grouped cluster, so
+ /// a tree of multi-array JSON files (tree-sitter `grammar.json`, OpenAPI
+ /// specs) kept one index per array. Grouped clusters now count toward
+ /// the cap; a file's arrays fold together, so a kept file keeps all of
+ /// its arrays and a folded one is one document.
+ #[test]
+ fn grouped_arrays_count_toward_the_scope_cap_and_fold_per_file() {
+ let mut sketches = vec![prose_sketch(0)];
+ let mut rels = vec!["README.md".to_string()];
+ let files = 20;
+ for f in 1..=files {
+ rels.push(format!("grammars/g{f}/grammar.json"));
+ sketches.push(data_sketch_n(
+ f,
+ Family::Json,
+ &[&format!("name{f}"), "word"],
+ 20,
+ ));
+ let mut g = data_sketch_n(
+ f,
+ Family::Json,
+ &[&format!("rule{f}"), "kind"],
+ 20 + f as u64,
+ );
+ g.group = Some(format!("rules{f}"));
+ sketches.push(g);
+ }
+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));
+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ assert_eq!(data.len(), MAX_STRUCTURED_DATASETS_PER_SCOPE, "{data:#?}");
+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");
+ let kept_files = MAX_STRUCTURED_DATASETS_PER_SCOPE / 2;
+ assert_eq!(
+ docs.demoted.len(),
+ files - kept_files,
+ "one document per folded file"
+ );
+ for c in &data {
+ let f = c.members[0];
+ assert!(!docs.demoted.contains(&f));
+ assert_eq!(
+ data.iter().filter(|d| d.members == vec![f]).count(),
+ 2,
+ "both arrays kept"
+ );
+ }
+ // The largest files (by rows) are the ones kept.
+ assert!(data.iter().all(|c| c.members[0] > files - kept_files));
+ }
+
/// Scope is a hard boundary for data schemas too: the same CSV schema in
/// two repositories is two datasets (per-repo corpora stay self-contained).
#[test]
@@ -1256,4 +1553,171 @@ mod tests {
let after: Vec<String> = untouched.iter().map(|c| c.slug.clone()).collect();
assert_eq!(before, after, "the default prefix must not rename slugs");
}
+
+ /// One page per table used to be one index per page: a tree of 200 HTML
+ /// reports with 5-row tables became 200 indices of a few documents each.
+ /// A page table below `MIN_DATASET_RECORDS` rows folds into the scope's
+ /// documents; a real table stays a dataset.
+ #[test]
+ fn small_html_tables_fold_into_documents_and_a_real_table_stays() {
+ let mut sketches = vec![prose_sketch(0)];
+ let mut rels = vec!["README.md".to_string()];
+ for i in 1..=40 {
+ rels.push(format!("reports/r{i}.html"));
+ sketches.push(data_sketch_n(
+ i,
+ Family::Html,
+ &[&format!("measure{i}"), "value"],
+ 5,
+ ));
+ }
+ rels.push("reports/table.html".into());
+ sketches.push(data_sketch_n(
+ 41,
+ Family::Html,
+ &["crate", "downloads"],
+ 400,
+ ));
+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));
+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");
+ assert_eq!(docs.demoted.len(), 40, "{clusters:#?}");
+ assert_eq!(docs.members.len(), 41, "README + 40 folded pages");
+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ assert_eq!(data.len(), 1, "{data:#?}");
+ assert_eq!(data[0].members, vec![41]);
+ }
+
+ /// A multi-array JSON file folds only whole. When every array is tiny the
+ /// file is one document (listed once, not once per array); when one
+ /// array is a real collection, every array of the file stays a dataset,
+ /// so no kept dataset loses rows.
+ #[test]
+ fn a_multi_array_file_folds_whole_or_not_at_all() {
+ let mut sketches = vec![prose_sketch(0)];
+ let rels: Vec<String> = vec!["README.md".into(), "tiny.json".into(), "mixed.json".into()];
+ sketches.push(data_sketch_n(1, Family::Json, &["a", "b"], 3));
+ let mut g = data_sketch_n(1, Family::Json, &["c", "d"], 2);
+ g.group = Some("rules".into());
+ sketches.push(g);
+ sketches.push(data_sketch_n(2, Family::Json, &["x", "y"], 600));
+ let mut g = data_sketch_n(2, Family::Json, &["p", "q"], 2);
+ g.group = Some("notes".into());
+ sketches.push(g);
+ let clusters = cluster(sketches, &rels, &root_scopes(rels.len()));
+ let docs = clusters.iter().find(|c| c.is_docs).expect("docs");
+ assert_eq!(docs.demoted, vec![1], "{clusters:#?}");
+ assert_eq!(docs.members.iter().filter(|&&m| m == 1).count(), 1);
+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ assert_eq!(data.len(), 2, "both arrays of mixed.json stay: {data:#?}");
+ assert!(data.iter().all(|c| c.members == vec![2]));
+ }
+
+ /// The per-scope cap alone let a tree of many repositories keep 12 small
+ /// datasets in each. One run keeps at most `MAX_DATASETS_PER_RUN` data
+ /// datasets: the protected collection and the CSV count toward it but
+ /// never fold, and the largest small JSON datasets fill the rest.
+ #[test]
+ fn a_run_keeps_at_most_the_run_cap_of_small_structured_datasets() {
+ let mut sketches = Vec::new();
+ let mut rels = Vec::new();
+ let mut scopes = Vec::new();
+ let repos = 10;
+ let per_repo = MAX_STRUCTURED_DATASETS_PER_SCOPE;
+ for r in 0..repos {
+ for k in 0..per_repo {
+ let idx = rels.len();
+ rels.push(format!("repo{r}/f{k}.json"));
+ scopes.push(format!("repo{r}"));
+ sketches.push(data_sketch_n(
+ idx,
+ Family::Json,
+ &[&format!("k{r}_{k}")],
+ 40 + idx as u64,
+ ));
+ }
+ }
+ let big = rels.len();
+ rels.push("repo0/events.json".into());
+ scopes.push("repo0".into());
+ sketches.push(data_sketch_n(
+ big,
+ Family::Json,
+ &["evt"],
+ DATASET_CAP_PROTECT_RECORDS,
+ ));
+ let csv = rels.len();
+ rels.push("repo0/rows.csv".into());
+ scopes.push("repo0".into());
+ sketches.push(data_sketch_n(csv, Family::Csv, &["a", "b"], 40));
+ let clusters = cluster(sketches, &rels, &scopes);
+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ assert_eq!(data.len(), MAX_DATASETS_PER_RUN);
+ let small: Vec<&&Cluster> = data
+ .iter()
+ .filter(|c| c.family == Family::Json && c.population < DATASET_CAP_PROTECT_RECORDS)
+ .collect();
+ let keep_small = MAX_DATASETS_PER_RUN - 2;
+ assert_eq!(small.len(), keep_small);
+ let smallest_kept = small.iter().map(|c| c.population).min().unwrap();
+ assert_eq!(
+ smallest_kept,
+ 40 + (repos * per_repo - keep_small) as u64,
+ "the largest are the ones kept"
+ );
+ assert!(clusters.iter().any(|c| c.members == vec![big]));
+ assert!(clusters
+ .iter()
+ .any(|c| c.members == vec![csv] && !c.is_docs));
+ let demoted: usize = clusters.iter().map(|c| c.demoted.len()).sum();
+ assert_eq!(demoted, repos * per_repo - keep_small);
+ }
+
+ /// Past the run cap, small CSV/JSONL datasets are not turned into text:
+ /// they merge into one dataset per scope and family, every row still a
+ /// row. A protected CSV stays as it is.
+ #[test]
+ fn past_the_run_cap_small_record_datasets_merge_per_scope_and_family() {
+ let mut sketches = Vec::new();
+ let mut rels = Vec::new();
+ let mut scopes = Vec::new();
+ let n_csv = MAX_DATASETS_PER_RUN + 10;
+ for k in 0..n_csv {
+ let idx = rels.len();
+ rels.push(format!("bench/run{k}.csv"));
+ scopes.push(String::new());
+ sketches.push(data_sketch_n(
+ idx,
+ Family::Csv,
+ &[&format!("col{k}"), &format!("val{k}")],
+ 100 + idx as u64,
+ ));
+ }
+ let big = rels.len();
+ rels.push("bench/all.csv".into());
+ scopes.push(String::new());
+ sketches.push(data_sketch_n(
+ big,
+ Family::Csv,
+ &["q", "ndcg"],
+ DATASET_CAP_PROTECT_RECORDS * 4,
+ ));
+ let clusters = cluster(sketches, &rels, &scopes);
+ let data: Vec<&Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ assert_eq!(data.len(), MAX_DATASETS_PER_RUN, "{data:#?}");
+ assert!(
+ clusters.iter().all(|c| c.demoted.is_empty()),
+ "no CSV became text"
+ );
+ let total: usize = data.iter().map(|c| c.members.len()).sum();
+ assert_eq!(total, n_csv + 1, "every file still lands in one dataset");
+ assert!(data.iter().any(|c| c.members == vec![big]));
+ let shared = data.iter().max_by_key(|c| c.members.len()).unwrap();
+ assert_eq!(shared.members.len(), n_csv + 1 - (MAX_DATASETS_PER_RUN - 1));
+ assert!(
+ shared.fields.contains_key("col0"),
+ "the union of the members' columns"
+ );
+ // The merge takes the smallest: file 0 (100 rows) is in the shared one.
+ assert!(shared.members.contains(&0));
+ }
}
diff --git a/engine/crates/xerj-autoindex/src/extract/html.rs b/engine/crates/xerj-autoindex/src/extract/html.rs
index ae21beff5..3d4d5ec07 100644
--- a/engine/crates/xerj-autoindex/src/extract/html.rs
+++ b/engine/crates/xerj-autoindex/src/extract/html.rs
@@ -133,6 +133,49 @@ pub fn extract(path: &Path, gzip: bool, sink: Sink) -> Result<ExtractStats> {
Ok(stats)
}
+/// Render a page as ONE document — `{title, headings, body}` — even when it
+/// has a dominant table. This is how a page whose table was too small to be
+/// a dataset of its own is indexed after the fold into the scope's document
+/// dataset (`dataset::MIN_DATASET_RECORDS`). The table's rows are already in
+/// `body` as ` | `-joined lines, so no cell text is lost. The title comes
+/// from `logical_path`, as in `extract_as_document_with_name`.
+pub(crate) fn extract_document(
+ content_path: &Path,
+ logical_path: &Path,
+ gzip: bool,
+ sink: Sink,
+) -> Result<ExtractStats> {
+ let mut stats = ExtractStats::default();
+ let Some(bytes) = super::read_whole(content_path, gzip, MAX_WHOLE_FILE)? else {
+ stats.junk += 1;
+ return Ok(stats);
+ };
+ let (text, _) = crate::sniff::decode_text(&bytes);
+ let doc = parse(&text);
+ if doc.title.trim().is_empty() && doc.headings.is_empty() && doc.body.trim().is_empty() {
+ stats.junk += 1;
+ return Ok(stats);
+ }
+ let title = if doc.title.trim().is_empty() {
+ doc.headings
+ .first()
+ .cloned()
+ .unwrap_or_else(|| stem_of(logical_path))
+ } else {
+ doc.title.trim().to_string()
+ };
+ emit_document(
+ &title,
+ &doc.headings,
+ doc.body.trim(),
+ None,
+ MAX_RECORDS_PER_FILE,
+ sink,
+ &mut stats,
+ );
+ Ok(stats)
+}
+
fn stem_of(path: &Path) -> String {
path.file_stem()
.map(|s| s.to_string_lossy().to_string())
@@ -1258,4 +1301,28 @@ function notCss() {}
assert!(stats.records >= 4, "{name} records {}", stats.records);
}
}
+
+ /// A page whose small table folded into the document dataset is indexed
+ /// as the page: its title, and a body holding the table's rows as text,
+ /// with no markup and no per-row records.
+ #[test]
+ fn a_folded_table_page_renders_as_one_tag_free_document() {
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("report.html");
+ let page = table_html(Some(["Measure", "Value"]), 5)
+ .replace("<html>", "<html><head><title>Port audit</title></head>");
+ std::fs::write(&path, page).unwrap();
+ let mut recs = Vec::new();
+ let stats = crate::extract::extract_as_document(&path, false, &mut |r| {
+ recs.push(r);
+ true
+ })
+ .unwrap();
+ assert_eq!(stats.records, 1, "{recs:?}");
+ assert_eq!(recs[0].fields["title"], "Port audit");
+ let body = body_of(&recs[0]);
+ assert!(body.contains("item3 | 3"), "{body}");
+ assert!(!body.contains('<'), "{body}");
+ assert!(!recs[0].fields.contains_key("Measure"));
+ }
}
diff --git a/engine/crates/xerj-autoindex/src/extract/mod.rs b/engine/crates/xerj-autoindex/src/extract/mod.rs
index f6c703960..81ffbbcd3 100644
--- a/engine/crates/xerj-autoindex/src/extract/mod.rs
+++ b/engine/crates/xerj-autoindex/src/extract/mod.rs
@@ -199,6 +199,12 @@ pub fn extract_as_document_with_name(
if skip_phrase_index(logical_path, content_path, gzip) {
return emit_phrase_skip(logical_path, sink);
}
+ // A folded HTML page is rendered as the page, not as its markup: the
+ // same title/headings/body an HTML document gets, with its table rows in
+ // the body. Raw text would index tags and scripts.
+ if is_html_name(logical_path) {
+ return html::extract_document(content_path, logical_path, gzip, sink);
+ }
let mut stats = ExtractStats::default();
let Some(bytes) = read_whole(content_path, gzip, MAX_WHOLE_FILE)? else {
stats.junk += 1;
@@ -225,6 +231,17 @@ pub fn extract_as_document_with_name(
Ok(stats)
}
+/// `.html`/`.htm`/`.xhtml`, optionally gzipped, by the logical name.
+fn is_html_name(logical: &Path) -> bool {
+ let name = logical
+ .file_name()
+ .and_then(|s| s.to_str())
+ .unwrap_or("")
+ .to_ascii_lowercase();
+ let name = name.strip_suffix(".gz").unwrap_or(&name);
+ name.ends_with(".html") || name.ends_with(".htm") || name.ends_with(".xhtml")
+}
+
/// Bytes of a JS/CSS prefix read to decide "this is a bundle, not source".
const MINIFIED_PREFIX: u64 = 8192;
/// Below this, a one-line file is a small module, not the huge bundle the
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 3e572dea1..8fbbb8f76 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -4258,6 +4258,36 @@ fn scan_file(
}
})
.collect();
+ // CSV, JSONL and logs declare no length, so `records` is the sample: it
+ // stops at `sample` rows (500 by default) or at the sample byte cap. A
+ // 100_000-row CSV then looked like a 500-row one, and the dataset caps
+ // (`dataset::DATASET_CAP_PROTECT_RECORDS`) merged it into a shared index
+ // as if it were small. When the sample may have stopped early, count on
+ // with the same extractor up to the protect threshold. That is enough to
+ // decide the caps and stays bounded however large the file is.
+ if matches!(sn.family, Family::Csv | Family::Jsonl | Family::Logs) {
+ let threshold = dataset::DATASET_CAP_PROTECT_RECORDS;
+ let sample_cut = |s: &GroupSketch| {
+ s.population < threshold
+ && (s.records >= sample as u64 || limit.is_some_and(|l| sn.gzip || size > l))
+ };
+ if out.sketches.iter().any(sample_cut) {
+ let groups_seen = out.sketches.len();
+ let mut counts: HashMap<Option<String>, u64> = HashMap::new();
+ let mut count = |rec: extract::RawRecord| -> bool {
+ *counts.entry(rec.group).or_default() += 1;
+ counts.len() < groups_seen || counts.values().any(|n| *n < threshold)
+ };
+ // A read error part-way still leaves a lower bound: every
+ // record counted was really there.
+ let _ = extract::extract(path, &sn, None, &mut count);
+ for s in &mut out.sketches {
+ if let Some(&n) = counts.get(&s.group) {
+ s.population = s.population.max(n);
+ }
+ }
+ }
+ }
out.sketches.sort_by(|a, b| a.group.cmp(&b.group));
out.sniffed = Some(sn);
out
@@ -4445,6 +4475,108 @@ mod clustering_key_tests {
assert_eq!(runs.records, 2);
assert_eq!(runs.population, 3);
}
+
+ fn csv_body(tag: &str, rows: usize) -> String {
+ let mut b = format!("{tag}_id,{tag}_name,{tag}_val\n");
+ for i in 0..rows {
+ b.push_str(&format!("{i},name{i},{}\n", i * 3));
+ }
+ b
+ }
+
+ fn jsonl_body(tag: &str, rows: usize) -> String {
+ let mut b = String::new();
+ for i in 0..rows {
+ b.push_str(&format!(
+ "{{\"{tag}_id\":{i},\"{tag}_name\":\"name{i}\",\"{tag}_val\":{}}}\n",
+ i * 3
+ ));
+ }
+ b
+ }
+
+ /// Line-record files declare no length, so the sample (500 rows) used to
+ /// be the population and a 2000-row CSV/JSONL counted as small (#2422).
+ /// Phase A now counts on to the protect threshold; a 10-row file still
+ /// reports exactly its 10 rows.
+ #[test]
+ fn a_sampled_line_record_file_reports_rows_past_the_sample() {
+ let dir = tempfile::tempdir().unwrap();
+ for (name, body) in [
+ ("big.csv", csv_body("b", 2000)),
+ ("big.jsonl", jsonl_body("j", 2000)),
+ ] {
+ let s = scan(dir.path(), name, &body);
+ assert_eq!(s.sketches.len(), 1, "{name}");
+ let sk = &s.sketches[0];
+ assert_eq!(sk.records, 500, "{name}: the sample still stops at 500");
+ assert!(
+ sk.population >= dataset::DATASET_CAP_PROTECT_RECORDS,
+ "{name}: population {} is the sample, not the file",
+ sk.population
+ );
+ }
+ let small = scan(dir.path(), "small.csv", &csv_body("s", 10));
+ assert_eq!(small.sketches[0].records, 10);
+ assert_eq!(small.sketches[0].population, 10);
+ }
+
+ /// End to end through `cluster()`: past the run cap, 2000-row CSV and
+ /// JSONL files keep their own datasets, and the 10-row CSVs still merge
+ /// into one shared CSV dataset.
+ #[test]
+ fn past_the_run_cap_large_sampled_csv_and_jsonl_keep_their_own_index() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut files: Vec<(String, String)> = Vec::new();
+ let n_big = dataset::MAX_DATASETS_PER_RUN + 2;
+ for i in 0..n_big {
+ files.push((format!("big{i}.csv"), csv_body(&format!("b{i}"), 2000)));
+ }
+ files.push(("big.jsonl".into(), jsonl_body("j", 2000)));
+ let n_small = 3;
+ for i in 0..n_small {
+ files.push((format!("small{i}.csv"), csv_body(&format!("s{i}"), 10)));
+ }
+ let mut rels = Vec::new();
+ let mut sketches = Vec::new();
+ for (idx, (name, body)) in files.iter().enumerate() {
+ let s = scan(dir.path(), name, body);
+ assert_eq!(s.sketches.len(), 1, "{name}");
+ let sk = &s.sketches[0];
+ sketches.push(dataset::Sketch {
+ file_idx: idx,
+ group: sk.group.clone(),
+ family: s.sniffed.as_ref().unwrap().family,
+ fields: sk.fields.clone(),
+ key_fields: sk.key_fields.clone(),
+ records: sk.records,
+ population: sk.population,
+ });
+ rels.push(name.clone());
+ }
+ let scopes = vec![String::new(); rels.len()];
+ let clusters = dataset::cluster(sketches, &rels, &scopes);
+ let data: Vec<&dataset::Cluster> = clusters.iter().filter(|c| !c.is_docs).collect();
+ let home = |idx: usize| {
+ data.iter()
+ .find(|c| c.members.contains(&idx))
+ .unwrap_or_else(|| panic!("{} has no data dataset: {clusters:#?}", rels[idx]))
+ };
+ // Every large file — the CSVs and the JSONL — is alone in its dataset.
+ for (idx, rel) in rels.iter().enumerate().take(n_big + 1) {
+ assert_eq!(home(idx).members, vec![idx], "{rel} was merged");
+ }
+ // The 10-row CSVs share one dataset, rows intact.
+ let shared = home(n_big + 1);
+ let mut members = shared.members.clone();
+ members.sort();
+ assert_eq!(
+ members,
+ (n_big + 1..n_big + 1 + n_small).collect::<Vec<_>>()
+ );
+ assert_eq!(shared.records, 10 * n_small as u64);
+ assert_eq!(data.len(), n_big + 2);
+ }
}
#[cfg(test)]
diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
index adff73908..49719464a 100644
--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs
+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
@@ -298,10 +298,17 @@ pub(crate) fn reconcile_plan(
.iter()
.any(|d| d.family == sniffed.family.as_str() && d.group.is_none());
let new_document = docs_target && all_empty_key_fields;
+ // A file too small to be a dataset (`dataset::MIN_DATASET_RECORDS`)
+ // folds the same way a fresh plan folds it: a 3-row HTML table or a
+ // 2-key JSON file joins docs rather than failing the run for want of a
+ // frozen dataset of its own.
let new_demoted_config = docs_target
&& !all_empty_key_fields
- && crate::dataset::demotable_family(sniffed.family)
- && scan.sketches.iter().map(|s| s.records).sum::<u64>() == 1;
+ && ((crate::dataset::demotable_family(sniffed.family)
+ && scan.sketches.iter().map(|s| s.records).sum::<u64>() == 1)
+ || (crate::dataset::foldable_family(sniffed.family)
+ && scan.sketches.iter().map(|s| s.population).sum::<u64>()
+ < crate::dataset::MIN_DATASET_RECORDS));
let route_to_docs = new_document || new_demoted_config;
// Only a demoted CONFIG re-samples through the document renderer (its