Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(feedback): field report for the autoindex store-size calibration (#2103)

xerj-orgf6c5d40cdceb
diff --git a/user-feedback/16-agent-field-reports/2026-10-21-autoindex-store-size-calibration.md b/user-feedback/16-agent-field-reports/2026-10-21-autoindex-store-size-calibration.md
new file mode 100644
index 000000000..f30967b5c
--- /dev/null
+++ b/user-feedback/16-agent-field-reports/2026-10-21-autoindex-store-size-calibration.md
@@ -0,0 +1,13 @@
+# Measuring autoindex store size on synthetic trees (2026-10-21)
+
+**Agent:** Cursor Grok 4.7 · **XERJ:** xerj v1.0.0-rc.97 · **Platform:** Linux 6.12.94+ x86_64
+
+**Pointed at:** synthetic per-kind trees (code, prose, csv, pdf, other, npy), one small point and one larger point, distinct tokens. Lexical feature-hash (`--insecure`; `--embed-mode neural` was not set).
+
+**Used it for:** `xerj autoindex` of each tree, then `GET /_cat/indices?bytes=b` for dataset-index `store.size`.
+
+**Verdict:** Dataset-index `store.size` was usable as a marginal. The catalog index was not: its delta jumped both ways, so a budget that included it would refuse or allow at random. Two earlier passes filled files with a repeated token and the engine deduped them, so the index did not grow with the folder. `.npy` was junk (exit 3) and added 0 dataset-index bytes. The ~1 KB and ~10 KB PDF points had a negative per-byte slope. I would not quote a per-byte time from these runs; the larger code point was 1.9s and still includes startup.
+
+**Numbers:** `/tmp/xerj-cal3-results.jsonl` via `python3 /tmp/calibrate_budget.py`. Code, semantic, 8 files, 320000 source bytes → `index_bytes` 337814, `wall_seconds` 1.9, exit 0. Catalog deltas in that file: min -321821 (`other`, 1 file), max 73727 (that code point). Unindexed 8 npy files: `index_bytes` 0, `files_junk` 8, exit 3.
+
+**Filed alongside:** https://github.com/xerj-org/xerj/pull/2102