Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(storage): publish reproducible projection evidence

bugerc753333ee0f5
diff --git a/engine/reports/2026-07-27_bounded_stored_projection.md b/engine/reports/2026-07-27_bounded_stored_projection.md
new file mode 100644
index 000000000..ae788f3c7
--- /dev/null
+++ b/engine/reports/2026-07-27_bounded_stored_projection.md
@@ -0,0 +1,118 @@
+# Bounded stored projection probe — 2026-07-27
+
+## Scope and source
+
+This report measures the storage primitive only. It does not claim an
+end-to-end engine speedup or cache-memory reduction because no engine consumer
+is included in this branch.
+
+- Source commit: `b3776019cf9d100c66c9dfde6ee005651822ebee`
+- Upstream base: `c2020229e70c2041c6d0b199c79a3c76ffb5e04f`
+- Probe source SHA-256:
+ `86326cab5a4e2f7f2fb141d081855ac8ba790a3403f9103a973732a19c109953`
+- Probe: `crates/xerj-storage/examples/stored_projection_probe.rs`
+- Rust: `rustc 1.97.1 (8bab26f4f 2026-07-14)`, LLVM 22.1.6
+- Host: x86-64 Linux 6.12.95, 62.8 GiB RAM, no swap
+- Build: release profile; no workspace-wide build and no `cargo clean`
+
+Exact commands, from `engine/`:
+
+```sh
+cargo build --release -j 32 -p xerj-storage --example stored_projection_probe
+probe=target/release/examples/stored_projection_probe
+fixture=/workspace/.tmp/stored-decode-probe/finance-stream-final.zbs2
+"$probe" generate "$fixture" 20000
+sha256sum "$fixture"
+for mode in full row projected; do
+ for run in 1 2 3; do
+ "$probe" "$mode" "$fixture" "/workspace/.tmp/stored-decode-probe/${mode}-stream-final-${run}.json"
+ done
+ sha256sum /workspace/.tmp/stored-decode-probe/${mode}-stream-final-*.json
+done
+for run in 1 2 3; do
+ "$probe" zstd-compare 20
+done
+```
+
+The deterministic fixture contains 20,000 finance-like rows and 86,834,777
+bytes of input JSON. Its encoded size is **137,876 bytes** and its SHA-256 is
+`71aef08c71077cec61a2482a3d8089571007f5167e7fa6a0b1d89b6b6f25e76d`.
+This supersedes the earlier 139,062/139,064-byte scratch fixtures; neither is
+evidence for this source commit.
+
+## Decode results
+
+All nine canonical output files have the same 4,225-byte content and SHA-256:
+`d505b2317c26eec30cf7ff40604d37a5b54986b652bb7348730c5d7ce5d2b35d`.
+This is the equality check between full decode, selective row hydration, and
+selective row-plus-field projection.
+
+| Mode | Wall µs, run 1 | Run 2 | Run 3 | Median | Peak RSS range |
+|---|---:|---:|---:|---:|---:|
+| Full | 352,653 | 327,372 | 336,715 | 336,715 | 334,647,296–337,702,912 |
+| One row | 1,350,147 | 1,454,028 | 1,392,531 | 1,392,531 | 5,636,096–5,742,592 |
+| One row, `body` only | 1,410,533 | 1,360,279 | 1,356,389 | 1,360,279 | 5,500,928–5,644,288 |
+
+Raw probe output:
+
+```text
+mode=full rows=20000 decoded_bytes=86832275 output_bytes=4225 wall_us=352653 cpu_ticks=35 peak_rss_bytes=336138240
+mode=full rows=20000 decoded_bytes=86832275 output_bytes=4225 wall_us=327372 cpu_ticks=32 peak_rss_bytes=334647296
+mode=full rows=20000 decoded_bytes=86832275 output_bytes=4225 wall_us=336715 cpu_ticks=33 peak_rss_bytes=337702912
+mode=row rows=1 output_bytes=4225 visited=80000 selected_values=8 decompressed_bytes=47264 wall_us=1350147 cpu_ticks=135 peak_rss_bytes=5636096
+mode=row rows=1 output_bytes=4225 visited=80000 selected_values=8 decompressed_bytes=47264 wall_us=1454028 cpu_ticks=144 peak_rss_bytes=5689344
+mode=row rows=1 output_bytes=4225 visited=80000 selected_values=8 decompressed_bytes=47264 wall_us=1392531 cpu_ticks=139 peak_rss_bytes=5742592
+mode=projected rows=1 output_bytes=4225 visited=60000 selected_values=3 decompressed_bytes=0 wall_us=1410533 cpu_ticks=140 peak_rss_bytes=5517312
+mode=projected rows=1 output_bytes=4225 visited=60000 selected_values=3 decompressed_bytes=0 wall_us=1360279 cpu_ticks=136 peak_rss_bytes=5500928
+mode=projected rows=1 output_bytes=4225 visited=60000 selected_values=3 decompressed_bytes=0 wall_us=1356389 cpu_ticks=135 peak_rss_bytes=5644288
+```
+
+Raw logical-work counters:
+
+- One row: 80,000 encoded rows visited, 8 selected JSON values, 47,264
+ decompressed buffer bytes.
+- One row plus field projection: 60,000 encoded rows visited, 3 selected JSON
+ values, zero whole-column decompressed-buffer bytes.
+
+Interpretation: selective hydration cuts peak RSS by roughly 59x, but is about
+4x slower in this RAW-column fixture because streaming JSON still visits O(N)
+rows. That is a useful bounded-memory primitive, not a latency win. LZ4 columns
+still require whole-column decompression before row selection. A single giant
+selected cell is not globally budgeted by this API.
+
+`VmHWM` comes from `/proc/self/status`; CPU is reported by the probe as Linux
+process ticks. Fresh processes are used for each measurement.
+
+## Stream `encode_all` versus bulk zstd
+
+Each cell below is one process timing 20 compression/decompression iterations.
+The three raw wall-time runs are retained, rather than presenting only a
+favorable aggregate.
+
+| Shape | Input bytes | Stream bytes | Bulk bytes | Stream wall µs (3 runs) | Bulk wall µs (3 runs) |
+|---|---:|---:|---:|---|---|
+| Repeated text | 4,160,000 | 413 | 417 | 53,707 / 55,046 / 55,614 | 86,177 / 82,769 / 91,054 |
+| JSON numbers | 945,611 | 8,507 | 8,510 | 6,671 / 6,920 / 6,876 | 5,216 / 6,251 / 5,290 |
+| Packed IDs | 2,000,000 | 1,335,160 | 1,335,163 | 63,980 / 73,980 / 69,865 | 60,947 / 65,366 / 66,119 |
+
+Bulk frames are 3–4 bytes larger. They are modestly faster for two shapes but
+materially slower for repeated text. Therefore this branch does **not** change
+the production encoder from `zstd::encode_all` to `zstd::bulk::compress`.
+Current stream frames commonly omit a content-size field, so
+`stored_slices_retained_upper_bound` deliberately returns `Ok(None)` and cache
+admission must skip warming. Emitting content-size frames should be evaluated
+with the future cache-resident integration, where its end-to-end benefit can be
+measured.
+
+## Verification
+
+```text
+cargo test -p xerj-storage --lib
+121 passed; 0 failed; 0 ignored
+
+cargo clippy -p xerj-storage --all-targets -- -D warnings
+passed
+
+cargo fmt --all --check
+passed
+```