Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

test(ingest): add bounded shipped-path diagnostics

bugereaaa176fce2e
diff --git a/demo/usecases/autoindex/scale/bounded/.gitignore b/demo/usecases/autoindex/scale/bounded/.gitignore
new file mode 100644
index 000000000..c18dd8d83
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/.gitignore
@@ -0,0 +1 @@
+__pycache__/
diff --git a/demo/usecases/autoindex/scale/bounded/README.md b/demo/usecases/autoindex/scale/bounded/README.md
new file mode 100644
index 000000000..973ad0dc4
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/README.md
@@ -0,0 +1,114 @@
+# Bounded shipped-path ingest diagnostics
+
+This suite replaces expensive full-corpus debugging loops with a small,
+deterministic experiment through XERJ's real ES-compatible HTTP path. It proves
+bulk ingest, refresh, explicit flush, force-merge, graceful shutdown, restart
+recovery, exact counts, and three sentinel lookups while collecting two
+independent telemetry streams. The `body` mapping is `semantic_text`, so the
+default `--embed-mode lexical` exercises XERJ's deterministic feature-hash
+embedding lifecycle; it does not claim neural semantics.
+
+- `xerj.ingest_memory.v1`: XERJ's opt-in logical ownership, jemalloc, RSS, and
+ process CPU sampler (`XERJ_INGEST_MEMORY_TRACE=summary`);
+- `xerj.process_sample.v1`: external Linux `/proc` status, `smaps_rollup`, I/O,
+ thread, and file-descriptor observations.
+
+This is a debugging instrument, not the FinanceBench acceptance benchmark. It
+should expose memory-shape defects in seconds or minutes; a candidate still
+needs the real correctness and performance benches before publication.
+
+## Run it
+
+Build only the server crate. Fat LTO is useful for release artifacts, but is a
+poor default for edit/profile loops because it makes linking slow and obscures
+rapid iteration. Use the repository's development profile or a dedicated
+profiling profile for experiments; re-run accepted measurements with the exact
+release profile used by the compared baseline.
+
+```bash
+cd engine
+CARGO_PROFILE_RELEASE_LTO=off cargo build --release -j 32 -p xerj-server
+
+cd ..
+SUITE=demo/usecases/autoindex/scale/bounded
+python3 "$SUITE/generate_corpus.py" \
+ --output /tmp/xerj-bounded-corpus \
+ --documents 4096
+python3 "$SUITE/run_suite.py" \
+ --binary engine/target/release/xerj \
+ --corpus /tmp/xerj-bounded-corpus \
+ --output /tmp/xerj-bounded-run \
+ --profile nightly
+python3 "$SUITE/analyze.py" /tmp/xerj-bounded-run
+```
+
+`run_suite.py` refuses to overwrite an output directory. Every cell uses a
+fresh data directory and server process. The restart phase reuses only that
+cell's durable data. The manifest records the binary SHA-256 and size, Git
+commit and dirty state, corpus hash, host, configuration, correctness results,
+and hashes of every raw artifact.
+
+The analyzer fails on:
+
+- malformed or wrong-schema telemetry;
+- a missing start/stop event or non-contiguous trace sequence;
+- dropped events or retain/release accounting errors;
+- nonzero measured ownership at process stop;
+- artifact hash/size drift;
+- count or sentinel-search corruption after refresh, merge, or restart.
+
+It also reports incremental RSS slopes and peak RSS per document. Those
+relations deliberately have no arbitrary absolute threshold. Small cells have
+fixed startup cost and host noise, so an accelerating slope is an investigation
+signal rather than a release failure.
+
+Once repeated pinned-host runs establish an envelope, write a versioned budget
+and check later runs against it:
+
+```bash
+python3 "$SUITE/analyze.py" /tmp/xerj-bounded-run \
+ --write-baseline-budget demo/usecases/autoindex/scale/bounded/budget.json \
+ --budget-headroom 1.05
+python3 "$SUITE/analyze.py" /tmp/xerj-candidate-run \
+ --budget demo/usecases/autoindex/scale/bounded/budget.json
+```
+
+The multiplier is explicit because the repository must choose it from repeated
+host variance; the suite does not invent one. A budget can cap per-cell ingest
+and restart RSS plus RSS-per-added-document slopes. Do not commit a budget
+generated from a single run.
+
+## CI and nightly profiles
+
+- **CI smoke (`--profile ci`):** generate 256 documents and run `64,256`. This validates the
+ harness and shipped path; it is not a performance assertion.
+- **Nightly diagnostics (`--profile nightly`):** run `256,1024,4096` at least three times on a pinned,
+ quiet host. Keep all raw runs and compare medians; do not average away a
+ correctness or telemetry-integrity failure.
+- **CPU/heap profiles:** use the opt-in profiling build and capture profiles
+ alongside this suite. Record their hashes in the run directory. Profiling
+ and non-profiling timings are separate populations; never compare one as the
+ performance baseline of the other.
+
+## Prompts for an AI reviewer
+
+The artifacts are intentionally JSON/NDJSON plus a short Markdown report, so an
+agent can inspect them without a proprietary UI.
+
+> Read `run_manifest.json`, `analysis.json`, and every trace named by the
+> manifest. Verify hashes, correctness after refresh/merge/restart, sequence
+> continuity, dropped/accounting errors, and stop-time ownership balance.
+> Identify the first phase where RSS separates from jemalloc allocated and from
+> logical attribution. Do not infer unavailable categories are zero.
+
+> Compare two bounded-run directories produced from the same corpus and cell
+> sizes. Confirm binary hashes and build/config differences first. For each
+> cell, compare peak and final RSS, allocator allocated/active/resident, logical
+> category peaks, write bytes, wall time, and restart RSS. Flag faster-but-wrong
+> or incomplete telemetry as a failed comparison.
+
+> Inspect the `/proc` and ingest-memory time series for one cell. Find the
+> timestamp and runner phase of peak RSS. Explain whether growth is live
+> attributed memory, allocator fragmentation/retention, page cache/file-backed
+> RSS, or unattributed memory. State uncertainty and name the next CPU or heap
+> profile needed to resolve it.
diff --git a/demo/usecases/autoindex/scale/bounded/analyze.py b/demo/usecases/autoindex/scale/bounded/analyze.py
new file mode 100644
index 000000000..b8ab2e4cd
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/analyze.py
@@ -0,0 +1,455 @@
+#!/usr/bin/env python3
+"""Validate a bounded run and report relational memory slopes."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+RUN_SCHEMA = "xerj.bounded_run.v1"
+TRACE_SCHEMA = "xerj.ingest_memory.v1"
+PROC_SCHEMA = "xerj.process_sample.v1"
+BUDGET_SCHEMA = "xerj.bounded_budget.v1"
+CHECK_PHASES = ["after_refresh", "after_flush", "after_merge", "after_restart"]
+CATEGORIES = [
+ "http_body",
+ "http_rewrite_buffer",
+ "raw_source",
+ "parsed_source",
+ "parsed_vector",
+ "prepared_document",
+ "prepared_vector",
+ "memtable_active",
+ "flush_drained",
+ "merge_decoded",
+ "merge_survivor",
+ "merge_json_buffer",
+ "merge_parsed",
+ "merge_encoded",
+ "cache_stored_slices",
+ "cache_doc_values",
+ "cache_stored_values",
+ "cache_sort_shadow",
+ "cache_id_positions",
+ "cache_row_sequences",
+ "cache_decoded_stored",
+]
+MEASUREMENTS = {
+ "http_body": "serialized_size",
+ "http_rewrite_buffer": "exact_capacity",
+ "raw_source": "serialized_size",
+ "parsed_source": "estimated",
+ "parsed_vector": "exact_capacity",
+ "prepared_document": "estimated",
+ "prepared_vector": "exact_capacity",
+ "memtable_active": "estimated",
+ "flush_drained": "estimated",
+ "merge_decoded": "unavailable",
+ "merge_survivor": "unavailable",
+ "merge_json_buffer": "unavailable",
+ "merge_parsed": "unavailable",
+ "merge_encoded": "unavailable",
+ "cache_stored_slices": "unavailable",
+ "cache_doc_values": "unavailable",
+ "cache_stored_values": "unavailable",
+ "cache_sort_shadow": "unavailable",
+ "cache_id_positions": "unavailable",
+ "cache_row_sequences": "unavailable",
+ "cache_decoded_stored": "unavailable",
+}
+
+
+def hash_file(path: Path) -> str:
+ digest = hashlib.sha256(path.read_bytes()).hexdigest()
+ return digest
+
+
+def write_private(path: Path, content: str) -> None:
+ path.write_text(content, encoding="utf-8")
+ path.chmod(0o600)
+
+
+def read_ndjson(path: Path) -> list[dict]:
+ rows = []
+ for number, line in enumerate(path.read_text().splitlines(), 1):
+ try:
+ rows.append(json.loads(line))
+ except json.JSONDecodeError as error:
+ raise ValueError(f"{path}:{number}: invalid JSON: {error}") from error
+ if not rows:
+ raise ValueError(f"{path}: no records")
+ return rows
+
+
+def validate_trace(path: Path) -> dict:
+ rows = read_ndjson(path)
+ problems = []
+ if any(row.get("schema") != TRACE_SCHEMA for row in rows):
+ problems.append("schema mismatch")
+ seq = [row.get("seq") for row in rows]
+ if seq != list(range(1, len(rows) + 1)):
+ problems.append("sequence is not contiguous from 1")
+ if rows[0].get("event") != "trace_start":
+ problems.append("first event is not trace_start")
+ if rows[-1].get("event") != "trace_stop":
+ problems.append("last event is not trace_stop")
+ expected_events = ["trace_start"] + ["trace_snapshot"] * (len(rows) - 2) + ["trace_stop"]
+ if [row.get("event") for row in rows] != expected_events:
+ problems.append("summary trace contains an unexpected event or event order")
+ if max(int(row.get("accounting_errors", 0)) for row in rows) != 0:
+ problems.append("accounting_errors is nonzero")
+ if max(int(row.get("dropped_events", 0)) for row in rows) != 0:
+ problems.append("dropped_events is nonzero")
+ for position, row in enumerate(rows, 1):
+ gauges = row.get("logical")
+ if not isinstance(gauges, list) or [gauge.get("category") for gauge in gauges] != CATEGORIES:
+ problems.append(f"record {position}: logical category vocabulary/order mismatch")
+ continue
+ if any(
+ gauge.get("measurement") != MEASUREMENTS[gauge["category"]] for gauge in gauges
+ ):
+ problems.append(f"record {position}: logical measurement mismatch")
+ if any(
+ not isinstance(gauge.get("current"), int)
+ or not isinstance(gauge.get("peak"), int)
+ or gauge["current"] < 0
+ or gauge["peak"] < gauge["current"]
+ for gauge in gauges
+ ):
+ problems.append(f"record {position}: invalid current/peak gauge")
+ if row.get("logical_total_current") != sum(gauge["current"] for gauge in gauges):
+ problems.append(f"record {position}: logical_total_current is not the gauge sum")
+ allocator = row.get("allocator", {})
+ process = row.get("process", {})
+ flags = row.get("flags", {})
+ allocator_values = [allocator.get(key) for key in ("allocated", "active", "resident")]
+ allocator_ok = allocator.get("epoch_ok") is True
+ if allocator_ok != all(type(value) is int and value >= 0 for value in allocator_values):
+ problems.append(f"record {position}: allocator availability/value mismatch")
+ if flags.get("allocator_supported") is not allocator_ok:
+ problems.append(f"record {position}: allocator_supported disagrees with epoch_ok")
+ rss_ok = process.get("rss_ok") is True
+ if rss_ok != (type(process.get("rss")) is int and process["rss"] >= 0):
+ problems.append(f"record {position}: RSS availability/value mismatch")
+ cpu_ok = process.get("cpu_time_ok") is True
+ if cpu_ok != (
+ type(process.get("cpu_time_ns")) is int and process["cpu_time_ns"] >= 0
+ ):
+ problems.append(f"record {position}: CPU-time availability/value mismatch")
+ expected_sampled = row.get("event") == "trace_snapshot"
+ if flags.get("sampled") is not expected_sampled or flags.get("logical_exact") is not False:
+ problems.append(f"record {position}: event flags mismatch")
+ stop_nonzero = {
+ gauge["category"]: gauge["current"]
+ for gauge in rows[-1].get("logical", [])
+ if gauge.get("measurement") != "unavailable" and gauge.get("current") != 0
+ }
+ if stop_nonzero:
+ problems.append(f"retained logical bytes at trace_stop: {stop_nonzero}")
+ return {
+ "records": len(rows),
+ "peak_rss_bytes": max(
+ (int(row["process"]["rss"]) for row in rows if row["process"].get("rss") is not None),
+ default=None,
+ ),
+ "peak_allocated_bytes": max(
+ (
+ int(row["allocator"]["allocated"])
+ for row in rows
+ if row["allocator"].get("allocated") is not None
+ ),
+ default=None,
+ ),
+ "peak_logical_bytes": max(int(row.get("logical_total_current", 0)) for row in rows),
+ "peak_logical_bytes_by_category": {
+ category: max(
+ int(
+ next(
+ gauge["peak"]
+ for gauge in row["logical"]
+ if gauge["category"] == category
+ )
+ )
+ for row in rows
+ )
+ for category in CATEGORIES
+ }
+ if not any("logical category vocabulary/order mismatch" in problem for problem in problems)
+ else None,
+ "problems": problems,
+ }
+
+
+def validate_proc(path: Path) -> dict:
+ rows = read_ndjson(path)
+ problems = []
+ if any(row.get("schema") != PROC_SCHEMA for row in rows):
+ problems.append("schema mismatch")
+ timestamps = [int(row["ts_unix_ns"]) for row in rows]
+ if timestamps != sorted(timestamps) or len(set(timestamps)) != len(timestamps):
+ problems.append("timestamps are not strictly increasing")
+ rss = [int(row["vmrss_bytes"]) for row in rows if "vmrss_bytes" in row]
+ if not rss:
+ problems.append("no VmRSS samples")
+ return {
+ "records": len(rows),
+ "peak_rss_bytes": max(rss, default=None),
+ "last_rss_bytes": rss[-1] if rss else None,
+ "peak_threads": max((int(row.get("threads", 0)) for row in rows), default=None),
+ "peak_fds": max((int(row.get("fds", 0)) for row in rows), default=None),
+ "write_bytes": max((int(row.get("io_write_bytes", 0)) for row in rows), default=None),
+ "problems": problems,
+ }
+
+
+def validate_checks(documents: int, checks) -> list[str]:
+ problems = []
+ expected_ids = [
+ "doc-00000000",
+ f"doc-{documents // 2:08d}",
+ f"doc-{documents - 1:08d}",
+ ]
+ expected_sentinels = {doc_id: [doc_id] for doc_id in expected_ids}
+ if not isinstance(checks, list) or not all(isinstance(check, dict) for check in checks):
+ return ["correctness checks are not an object list"]
+ if [check.get("phase") for check in checks] != CHECK_PHASES:
+ problems.append("correctness phases/order mismatch")
+ return problems
+ for check in checks:
+ if check.get("count") != documents:
+ problems.append(f"{check['phase']}: correctness count mismatch")
+ if check.get("sentinels") != expected_sentinels:
+ problems.append(f"{check['phase']}: sentinel result mismatch")
+ return problems
+
+
+def analyze(root: Path) -> dict:
+ manifest = json.loads((root / "run_manifest.json").read_text())
+ problems = []
+ if manifest.get("schema") != RUN_SCHEMA:
+ problems.append("run manifest schema mismatch")
+ results = []
+ for cell in manifest["cells"]:
+ directory = root / f"docs-{cell['documents']}"
+ artifact_problems = []
+ for name, expected in cell["artifacts"].items():
+ path = directory / name
+ if not path.exists() or path.stat().st_size != expected["bytes"]:
+ artifact_problems.append(f"{name}: size mismatch or missing")
+ elif hash_file(path) != expected["sha256"]:
+ artifact_problems.append(f"{name}: sha256 mismatch")
+ ingest_trace = validate_trace(directory / "ingest-memory-ingest.ndjson")
+ restart_trace = validate_trace(directory / "ingest-memory-restart.ndjson")
+ ingest_proc = validate_proc(directory / "proc-ingest.ndjson")
+ restart_proc = validate_proc(directory / "proc-restart.ndjson")
+ cell_problems = (
+ artifact_problems
+ + ingest_trace["problems"]
+ + restart_trace["problems"]
+ + ingest_proc["problems"]
+ + restart_proc["problems"]
+ )
+ category_peaks = ingest_trace["peak_logical_bytes_by_category"]
+ if category_peaks is not None and category_peaks["prepared_vector"] == 0:
+ cell_problems.append("semantic embedding path produced no prepared vectors")
+ cell_problems.extend(validate_checks(cell["documents"], cell.get("checks")))
+ results.append(
+ {
+ "documents": cell["documents"],
+ "ingest_trace": ingest_trace,
+ "restart_trace": restart_trace,
+ "ingest_process": ingest_proc,
+ "restart_process": restart_proc,
+ "problems": cell_problems,
+ }
+ )
+ problems.extend(f"docs-{cell['documents']}: {item}" for item in cell_problems)
+
+ points = [
+ (result["documents"], result["ingest_process"]["peak_rss_bytes"])
+ for result in results
+ if result["ingest_process"]["peak_rss_bytes"] is not None
+ ]
+ slopes = []
+ for (docs_a, rss_a), (docs_b, rss_b) in zip(points, points[1:]):
+ slopes.append(
+ {
+ "from_documents": docs_a,
+ "to_documents": docs_b,
+ "rss_bytes_per_added_document": (rss_b - rss_a) / (docs_b - docs_a),
+ }
+ )
+ per_doc = [
+ {"documents": docs, "peak_rss_bytes_per_document": rss / docs}
+ for docs, rss in points
+ ]
+ relational = {
+ "incremental_slopes": slopes,
+ "peak_rss_per_document": per_doc,
+ "non_accelerating_incremental_slope": all(
+ later["rss_bytes_per_added_document"] <= earlier["rss_bytes_per_added_document"]
+ for earlier, later in zip(slopes, slopes[1:])
+ )
+ if len(slopes) > 1
+ else None,
+ "declining_peak_rss_per_document": all(
+ later["peak_rss_bytes_per_document"] <= earlier["peak_rss_bytes_per_document"]
+ for earlier, later in zip(per_doc, per_doc[1:])
+ )
+ if per_doc
+ else None,
+ "interpretation": (
+ "These are relational diagnostics, not an absolute memory budget. "
+ "A false verdict identifies acceleration worth investigating; it is not by itself "
+ "a release failure because fixed startup cost and host noise affect small cells."
+ ),
+ }
+ return {
+ "schema": "xerj.bounded_analysis.v1",
+ "status": "PASS" if not problems else "FAIL",
+ "problems": problems,
+ "cells": results,
+ "relational_memory": relational,
+ }
+
+
+def apply_budget(result: dict, budget: dict) -> list[str]:
+ problems = []
+ if budget.get("schema") != BUDGET_SCHEMA:
+ return ["budget schema mismatch"]
+ limits = budget.get("limits", {})
+ by_docs = {str(cell["documents"]): cell for cell in result["cells"]}
+ for metric, result_key in (
+ ("max_peak_ingest_rss_bytes_by_documents", ("ingest_process", "peak_rss_bytes")),
+ ("max_peak_restart_rss_bytes_by_documents", ("restart_process", "peak_rss_bytes")),
+ ):
+ for documents, ceiling in limits.get(metric, {}).items():
+ cell = by_docs.get(documents)
+ if cell is None:
+ problems.append(f"budget requires missing {documents}-document cell")
+ continue
+ observed = cell[result_key[0]][result_key[1]]
+ if observed is None or observed > ceiling:
+ problems.append(f"{metric}[{documents}] observed {observed}, ceiling {ceiling}")
+ slope_by_range = {
+ f"{slope['from_documents']}:{slope['to_documents']}": slope[
+ "rss_bytes_per_added_document"
+ ]
+ for slope in result["relational_memory"]["incremental_slopes"]
+ }
+ for span, ceiling in limits.get("max_rss_bytes_per_added_document_by_range", {}).items():
+ observed = slope_by_range.get(span)
+ if observed is None or observed > ceiling:
+ problems.append(
+ f"max_rss_bytes_per_added_document_by_range[{span}] "
+ f"observed {observed}, ceiling {ceiling}"
+ )
+ return problems
+
+
+def make_budget(result: dict, source_manifest: dict, headroom: float) -> dict:
+ def ceilings(process: str):
+ return {
+ str(cell["documents"]): int(cell[process]["peak_rss_bytes"] * headroom)
+ for cell in result["cells"]
+ if cell[process]["peak_rss_bytes"] is not None
+ }
+
+ return {
+ "schema": BUDGET_SCHEMA,
+ "source": {
+ "binary_sha256": source_manifest["binary"]["sha256"],
+ "corpus_sha256": source_manifest["corpus"]["sha256"],
+ "documents": source_manifest["configuration"]["documents"],
+ "headroom_multiplier": headroom,
+ "note": "Use a pinned-host repeated-run envelope; one noisy run is not a stable budget.",
+ },
+ "limits": {
+ "max_peak_ingest_rss_bytes_by_documents": ceilings("ingest_process"),
+ "max_peak_restart_rss_bytes_by_documents": ceilings("restart_process"),
+ "max_rss_bytes_per_added_document_by_range": {
+ f"{slope['from_documents']}:{slope['to_documents']}": (
+ max(0.0, slope["rss_bytes_per_added_document"]) * headroom
+ )
+ for slope in result["relational_memory"]["incremental_slopes"]
+ },
+ },
+ }
+
+
+def render(result: dict) -> str:
+ lines = [
+ "# Bounded ingest diagnostic result",
+ "",
+ f"Integrity and correctness: **{result['status']}**",
+ "",
+ "| Documents | Peak ingest RSS | Peak restart RSS | Trace records |",
+ "|---:|---:|---:|---:|",
+ ]
+ for cell in result["cells"]:
+ lines.append(
+ f"| {cell['documents']} | {cell['ingest_process']['peak_rss_bytes']} | "
+ f"{cell['restart_process']['peak_rss_bytes']} | {cell['ingest_trace']['records']} |"
+ )
+ lines += ["", "## Relational memory diagnostics", ""]
+ for slope in result["relational_memory"]["incremental_slopes"]:
+ lines.append(
+ f"- {slope['from_documents']} → {slope['to_documents']}: "
+ f"{slope['rss_bytes_per_added_document']:.2f} RSS bytes per added document"
+ )
+ lines += [
+ "",
+ f"- Non-accelerating incremental slope: "
+ f"`{result['relational_memory']['non_accelerating_incremental_slope']}`",
+ f"- Declining peak RSS/document: "
+ f"`{result['relational_memory']['declining_peak_rss_per_document']}`",
+ "",
+ result["relational_memory"]["interpretation"],
+ "",
+ ]
+ if result["problems"]:
+ lines += ["## Problems", ""] + [f"- {problem}" for problem in result["problems"]] + [""]
+ return "\n".join(lines)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("run", type=Path)
+ parser.add_argument("--budget", type=Path, help="apply versioned absolute/relational ceilings")
+ parser.add_argument("--write-baseline-budget", type=Path)
+ parser.add_argument(
+ "--budget-headroom",
+ type=float,
+ default=1.0,
+ help="explicit multiplier when writing a budget (default: exact observed envelope)",
+ )
+ args = parser.parse_args()
+ result = analyze(args.run)
+ manifest = json.loads((args.run / "run_manifest.json").read_text())
+ if args.budget:
+ budget_problems = apply_budget(result, json.loads(args.budget.read_text()))
+ result["budget"] = {"path": str(args.budget), "problems": budget_problems}
+ result["problems"].extend(f"budget: {problem}" for problem in budget_problems)
+ if budget_problems:
+ result["status"] = "FAIL"
+ if args.write_baseline_budget:
+ if args.budget_headroom <= 0:
+ parser.error("--budget-headroom must be positive")
+ budget = make_budget(result, manifest, args.budget_headroom)
+ args.write_baseline_budget.write_text(
+ json.dumps(budget, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ args.write_baseline_budget.chmod(0o600)
+ write_private(
+ args.run / "analysis.json",
+ json.dumps(result, indent=2, sort_keys=True) + "\n",
+ )
+ write_private(args.run / "report.md", render(result))
+ print(render(result))
+ return 0 if result["status"] == "PASS" else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/demo/usecases/autoindex/scale/bounded/generate_corpus.py b/demo/usecases/autoindex/scale/bounded/generate_corpus.py
new file mode 100644
index 000000000..ffe26be15
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/generate_corpus.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""Generate a deterministic, compact corpus for shipped-path memory diagnostics."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+SCHEMA = "xerj.bounded_corpus.v1"
+WORDS = (
+ "ledger invoice quarter revenue margin audit payable receivable "
+ "forecast covenant depreciation treasury subsidiary"
+).split()
+
+
+def document(number: int) -> dict:
+ """Return a deterministic document whose heap shape is stable across runs."""
+ words = [WORDS[(number * 7 + offset * 3) % len(WORDS)] for offset in range(48)]
+ return {
+ "doc_key": f"doc-{number:08d}",
+ "company": f"company-{number % 31:02d}",
+ "year": 2018 + number % 9,
+ "quarter": f"Q{1 + number % 4}",
+ "amount": round(((number * 104729) % 10_000_000) / 100.0, 2),
+ "body": " ".join(words),
+ "labels": [f"bucket-{number % 17:02d}", f"region-{number % 7:02d}"],
+ }
+
+
+def sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as source:
+ for chunk in iter(lambda: source.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def generate(output: Path, count: int) -> dict:
+ if output.exists() and any(output.iterdir()):
+ raise ValueError(f"refusing nonempty output directory: {output}")
+ output.mkdir(parents=True, exist_ok=True, mode=0o700)
+ output.chmod(0o700)
+ docs_path = output / "documents.ndjson"
+ with docs_path.open("w", encoding="utf-8", newline="\n") as target:
+ for number in range(count):
+ target.write(json.dumps(document(number), sort_keys=True, separators=(",", ":")))
+ target.write("\n")
+ manifest = {
+ "schema": SCHEMA,
+ "generator": Path(__file__).name,
+ "documents": count,
+ "file": docs_path.name,
+ "bytes": docs_path.stat().st_size,
+ "sha256": sha256(docs_path),
+ "first_id": "doc-00000000",
+ "middle_id": f"doc-{count // 2:08d}",
+ "last_id": f"doc-{count - 1:08d}",
+ }
+ (output / "corpus_manifest.json").write_text(
+ json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ docs_path.chmod(0o600)
+ (output / "corpus_manifest.json").chmod(0o600)
+ return manifest
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--documents", type=int, default=4096)
+ args = parser.parse_args()
+ if args.documents < 1:
+ parser.error("--documents must be positive")
+ manifest = generate(args.output, args.documents)
+ print(json.dumps(manifest, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/demo/usecases/autoindex/scale/bounded/run_suite.py b/demo/usecases/autoindex/scale/bounded/run_suite.py
new file mode 100644
index 000000000..4663e7677
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/run_suite.py
@@ -0,0 +1,481 @@
+#!/usr/bin/env python3
+"""Run bounded ingest diagnostics through XERJ's real ES-compatible HTTP path."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import platform
+import signal
+import socket
+import subprocess
+import threading
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+SCHEMA = "xerj.bounded_run.v1"
+PROC_SCHEMA = "xerj.process_sample.v1"
+HERE = Path(__file__).resolve().parent
+
+
+def hash_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as source:
+ for chunk in iter(lambda: source.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def toml_string(value: str) -> str:
+ """Encode a TOML basic string without interpolating path syntax."""
+ return json.dumps(value)
+
+
+def free_port() -> int:
+ with socket.socket() as listener:
+ listener.bind(("127.0.0.1", 0))
+ return int(listener.getsockname()[1])
+
+
+def request(url: str, method: str = "GET", value=None, raw: bytes | None = None) -> dict:
+ if raw is None and value is not None:
+ raw = json.dumps(value, separators=(",", ":")).encode()
+ req = urllib.request.Request(
+ url, data=raw, method=method, headers={"content-type": "application/json"}
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=300) as response:
+ payload = response.read()
+ except urllib.error.HTTPError as error:
+ body = error.read().decode(errors="replace")
+ raise RuntimeError(f"{method} {url}: HTTP {error.code}: {body[:2000]}") from error
+ return json.loads(payload or b"{}")
+
+
+def read_proc(pid: int) -> dict:
+ root = Path("/proc") / str(pid)
+ result: dict[str, object] = {}
+ try:
+ for line in (root / "status").read_text().splitlines():
+ key, _, value = line.partition(":")
+ if key in {"VmRSS", "VmHWM", "VmSize", "RssAnon", "RssFile", "RssShmem"}:
+ result[f"{key.lower()}_bytes"] = int(value.split()[0]) * 1024
+ for line in (root / "smaps_rollup").read_text().splitlines():
+ key, _, value = line.partition(":")
+ if key in {"Rss", "Pss", "Private_Clean", "Private_Dirty", "Swap"}:
+ result[f"smaps_{key.lower()}_bytes"] = int(value.split()[0]) * 1024
+ for line in (root / "io").read_text().splitlines():
+ key, _, value = line.partition(":")
+ if key in {"read_bytes", "write_bytes", "rchar", "wchar"}:
+ result[f"io_{key}"] = int(value.strip())
+ stat = (root / "stat").read_text().split()
+ result["cpu_user_ticks"] = int(stat[13])
+ result["cpu_system_ticks"] = int(stat[14])
+ result["threads"] = len(list((root / "task").iterdir()))
+ result["fds"] = len(list((root / "fd").iterdir()))
+ except (FileNotFoundError, ProcessLookupError, PermissionError, ValueError):
+ result["sample_error"] = True
+ return result
+
+
+class ProcSampler:
+ def __init__(self, pid: int, path: Path, phase, interval: float):
+ self.pid = pid
+ self.path = path
+ self.phase = phase
+ self.interval = interval
+ self.stop = threading.Event()
+ self.thread = threading.Thread(target=self._run, name="xerj-proc-sampler", daemon=True)
+
+ def _run(self):
+ with self.path.open("w", encoding="utf-8", buffering=1) as target:
+ os.fchmod(target.fileno(), 0o600)
+ while not self.stop.is_set():
+ process = read_proc(self.pid)
+ if process.get("sample_error"):
+ break
+ sample = {
+ "schema": PROC_SCHEMA,
+ "ts_unix_ns": time.time_ns(),
+ "pid": self.pid,
+ "phase": self.phase(),
+ **process,
+ }
+ target.write(json.dumps(sample, sort_keys=True) + "\n")
+ self.stop.wait(self.interval)
+
+ def __enter__(self):
+ self.thread.start()
+ return self
+
+ def __exit__(self, *_):
+ self.stop.set()
+ self.thread.join(timeout=max(2.0, self.interval * 2))
+
+
+def start_server(
+ binary: Path, cell: Path, data: Path, suffix: str, sample_ms: int, embed_mode: str
+):
+ es_port, rest_port, grpc_port = free_port(), free_port(), free_port()
+ config = cell / f"server-{suffix}.toml"
+ trace = cell / f"ingest-memory-{suffix}.ndjson"
+ log = cell / f"server-{suffix}.log"
+ config.write_text(
+ "[server]\n"
+ f'rest_port = {rest_port}\n'
+ f'grpc_port = {grpc_port}\n'
+ f'es_compat_port = {es_port}\n'
+ 'bind_address = "127.0.0.1"\n'
+ f"data_dir = {toml_string(str(data))}\n\n"
+ "[auth]\nenabled = false\n",
+ encoding="utf-8",
+ )
+ config.chmod(0o600)
+ env = os.environ.copy()
+ env.update(
+ {
+ "XERJ_INGEST_MEMORY_TRACE": "summary",
+ "XERJ_INGEST_MEMORY_SAMPLE_MS": str(sample_ms),
+ "XERJ_INGEST_MEMORY_OUTPUT": str(trace),
+ "XERJ_EMBED_MODE": embed_mode,
+ "XERJ_LOG": "info",
+ }
+ )
+ stream = log.open("wb")
+ os.fchmod(stream.fileno(), 0o600)
+ try:
+ process = subprocess.Popen(
+ [str(binary), "--insecure", "--config", str(config), "--data-dir", str(data)],
+ stdout=stream,
+ stderr=subprocess.STDOUT,
+ env=env,
+ )
+ except BaseException:
+ stream.close()
+ raise
+ url = f"http://127.0.0.1:{es_port}"
+ for _ in range(240):
+ if process.poll() is not None:
+ stream.close()
+ raise RuntimeError(f"server exited {process.returncode}; inspect {log}")
+ try:
+ request(url)
+ return process, stream, url, trace, config
+ except Exception:
+ time.sleep(0.05)
+ process.kill()
+ process.wait()
+ stream.close()
+ raise RuntimeError(f"server did not become ready; inspect {log}")
+
+
+def stop_server(process: subprocess.Popen, stream) -> int:
+ process.send_signal(signal.SIGINT)
+ try:
+ code = process.wait(timeout=60)
+ except subprocess.TimeoutExpired:
+ process.kill()
+ code = process.wait()
+ raise RuntimeError("server did not stop gracefully within 60 seconds")
+ finally:
+ stream.close()
+ if code != 0:
+ raise RuntimeError(f"server exited with status {code}")
+ return code
+
+
+def cleanup_server(process: subprocess.Popen, stream) -> None:
+ """Best-effort no-throw cleanup for an interrupted or failed cell.
+
+ This is deliberately safe to call after ``stop_server``. It never replaces
+ the request/verification exception that caused a cell to abort.
+ """
+ try:
+ if process.poll() is None:
+ try:
+ process.send_signal(signal.SIGINT)
+ process.wait(timeout=10)
+ except (ProcessLookupError, subprocess.TimeoutExpired):
+ if process.poll() is None:
+ process.kill()
+ process.wait()
+ except BaseException:
+ # Cleanup is a last-resort guard. The original cell failure is more
+ # actionable and must retain its traceback.
+ pass
+ finally:
+ try:
+ stream.close()
+ except BaseException:
+ pass
+
+
+def verify(url: str, index: str, count: int, ids: list[str], phase: str) -> dict:
+ counted = request(f"{url}/{index}/_count")
+ actual = int(counted.get("count", -1))
+ if actual != count:
+ raise RuntimeError(f"{phase}: expected {count} documents, got {actual}")
+ observed = {}
+ for doc_id in ids:
+ result = request(
+ f"{url}/{index}/_search",
+ "POST",
+ {"size": 2, "query": {"term": {"doc_key": doc_id}}, "track_total_hits": True},
+ )
+ total = int(result["hits"]["total"]["value"])
+ hits = [hit["_id"] for hit in result["hits"]["hits"]]
+ if total != 1 or hits != [doc_id]:
+ raise RuntimeError(f"{phase}: sentinel {doc_id} returned total={total}, ids={hits}")
+ observed[doc_id] = hits
+ return {"phase": phase, "count": actual, "sentinels": observed}
+
+
+def bulk_load(url: str, index: str, docs: Path, count: int, batch_size: int, set_phase):
+ batch: list[bytes] = []
+ sent = 0
+ with docs.open("rb") as source:
+ for number, raw in enumerate(source):
+ if number >= count:
+ break
+ action = json.dumps(
+ {"index": {"_index": index, "_id": f"doc-{number:08d}"}},
+ separators=(",", ":"),
+ ).encode()
+ batch.extend((action, raw.rstrip(b"\n")))
+ if number % batch_size == batch_size - 1 or number + 1 == count:
+ set_phase(f"bulk:{sent}-{number + 1}")
+ response = request(f"{url}/_bulk", "POST", raw=b"\n".join(batch) + b"\n")
+ if response.get("errors"):
+ errors = [
+ item for item in response.get("items", []) if next(iter(item.values())).get("error")
+ ]
+ raise RuntimeError(f"bulk errors: {json.dumps(errors[:3])}")
+ sent = number + 1
+ batch.clear()
+ if sent != count:
+ raise RuntimeError(f"corpus ended at {sent}, expected {count}")
+
+
+def run_cell(
+ binary: Path,
+ corpus: Path,
+ root: Path,
+ count: int,
+ batch: int,
+ sample_ms: int,
+ embed_mode: str,
+):
+ cell = root / f"docs-{count}"
+ cell.mkdir(mode=0o700)
+ data = cell / "data"
+ data.mkdir(mode=0o700)
+ index = "bounded-diagnostics"
+ phase_lock = threading.Lock()
+ phase_name = "boot"
+
+ def phase(value=None):
+ nonlocal phase_name
+ with phase_lock:
+ if value is not None:
+ phase_name = value
+ return phase_name
+
+ timeline = []
+ process, stream, url, trace, config = start_server(
+ binary, cell, data, "ingest", sample_ms, embed_mode
+ )
+ try:
+ with ProcSampler(process.pid, cell / "proc-ingest.ndjson", phase, sample_ms / 1000):
+ phase("create")
+ request(
+ f"{url}/{index}",
+ "PUT",
+ {
+ "mappings": {
+ "properties": {
+ "doc_key": {"type": "keyword"},
+ "company": {"type": "keyword"},
+ "year": {"type": "long"},
+ "quarter": {"type": "keyword"},
+ "amount": {"type": "double"},
+ "body": {"type": "semantic_text"},
+ "labels": {"type": "keyword"},
+ }
+ }
+ },
+ )
+ started = time.monotonic_ns()
+ bulk_load(url, index, corpus / "documents.ndjson", count, batch, phase)
+ timeline.append({"phase": "bulk", "elapsed_ns": time.monotonic_ns() - started})
+ ids = ["doc-00000000", f"doc-{count // 2:08d}", f"doc-{count - 1:08d}"]
+ phase("refresh")
+ started = time.monotonic_ns()
+ request(f"{url}/{index}/_refresh", "POST")
+ checks = [verify(url, index, count, ids, "after_refresh")]
+ timeline.append(
+ {"phase": "refresh_and_verify", "elapsed_ns": time.monotonic_ns() - started}
+ )
+ phase("flush")
+ started = time.monotonic_ns()
+ flush = request(f"{url}/{index}/_flush", "POST")
+ checks.append(verify(url, index, count, ids, "after_flush"))
+ timeline.append(
+ {"phase": "flush_and_verify", "elapsed_ns": time.monotonic_ns() - started}
+ )
+ phase("merge")
+ started = time.monotonic_ns()
+ merge = request(f"{url}/{index}/_forcemerge?max_num_segments=1", "POST")
+ checks.append(verify(url, index, count, ids, "after_merge"))
+ timeline.append(
+ {"phase": "merge_and_verify", "elapsed_ns": time.monotonic_ns() - started}
+ )
+ phase("shutdown")
+ started = time.monotonic_ns()
+ stop_server(process, stream)
+ timeline.append({"phase": "shutdown", "elapsed_ns": time.monotonic_ns() - started})
+ finally:
+ cleanup_server(process, stream)
+
+ phase("restart")
+ started = time.monotonic_ns()
+ process2, stream2, url2, trace2, config2 = start_server(
+ binary, cell, data, "restart", sample_ms, embed_mode
+ )
+ try:
+ with ProcSampler(process2.pid, cell / "proc-restart.ndjson", phase, sample_ms / 1000):
+ checks.append(verify(url2, index, count, ids, "after_restart"))
+ timeline.append(
+ {"phase": "restart_and_verify", "elapsed_ns": time.monotonic_ns() - started}
+ )
+ phase("restart_shutdown")
+ started = time.monotonic_ns()
+ stop_server(process2, stream2)
+ timeline.append(
+ {"phase": "restart_shutdown", "elapsed_ns": time.monotonic_ns() - started}
+ )
+ finally:
+ cleanup_server(process2, stream2)
+ for path in cell.iterdir():
+ if path.is_file():
+ path.chmod(0o600)
+ return {
+ "documents": count,
+ "batch_size": batch,
+ "checks": checks,
+ "flush_response": flush,
+ "merge_response": merge,
+ "timeline": timeline,
+ "artifacts": {
+ path.name: {"bytes": path.stat().st_size, "sha256": hash_file(path)}
+ for path in sorted(cell.iterdir())
+ if path.is_file()
+ },
+ }
+
+
+def git_identity(binary: Path) -> dict:
+ try:
+ root = subprocess.check_output(
+ ["git", "-C", str(HERE), "rev-parse", "--show-toplevel"], text=True
+ ).strip()
+ return {
+ "root": root,
+ "commit": subprocess.check_output(
+ ["git", "-C", root, "rev-parse", "HEAD"], text=True
+ ).strip(),
+ "dirty": bool(
+ subprocess.check_output(["git", "-C", root, "status", "--porcelain"], text=True)
+ ),
+ }
+ except (subprocess.CalledProcessError, FileNotFoundError):
+ return {"root": None, "commit": None, "dirty": None}
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--binary", type=Path, required=True)
+ parser.add_argument("--corpus", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--profile", choices=("ci", "nightly"), default="nightly")
+ parser.add_argument("--documents", help="override profile with increasing comma-separated cells")
+ parser.add_argument("--batch-size", type=int, default=128)
+ parser.add_argument("--sample-ms", type=int, default=100)
+ parser.add_argument("--embed-mode", default="lexical")
+ args = parser.parse_args()
+ profile_counts = {"ci": "64,256", "nightly": "256,1024,4096"}
+ counts = [int(value) for value in (args.documents or profile_counts[args.profile]).split(",")]
+ if sorted(set(counts)) != counts or len(counts) < 2 or counts[0] < 1:
+ parser.error("--documents must contain at least two strictly increasing positive counts")
+ if args.batch_size < 1:
+ parser.error("--batch-size must be positive")
+ if args.sample_ms < 1:
+ parser.error("--sample-ms must be positive")
+ args.binary = args.binary.resolve()
+ args.corpus = args.corpus.resolve()
+ if not os.access(args.binary, os.X_OK):
+ parser.error(f"not executable: {args.binary}")
+ corpus_manifest = json.loads((args.corpus / "corpus_manifest.json").read_text())
+ if corpus_manifest["documents"] < counts[-1]:
+ parser.error("corpus contains fewer documents than the largest cell")
+ if hash_file(args.corpus / corpus_manifest["file"]) != corpus_manifest["sha256"]:
+ parser.error("corpus hash does not match corpus_manifest.json")
+ if args.output.exists():
+ parser.error(f"output already exists: {args.output}")
+ args.output.mkdir(parents=True, mode=0o700)
+ args.output.chmod(0o700)
+ started = time.time_ns()
+ cells = [
+ run_cell(
+ args.binary,
+ args.corpus,
+ args.output,
+ count,
+ args.batch_size,
+ args.sample_ms,
+ args.embed_mode,
+ )
+ for count in counts
+ ]
+ manifest = {
+ "schema": SCHEMA,
+ "started_unix_ns": started,
+ "finished_unix_ns": time.time_ns(),
+ "binary": {
+ "path": str(args.binary),
+ "bytes": args.binary.stat().st_size,
+ "sha256": hash_file(args.binary),
+ "version": subprocess.check_output([str(args.binary), "--version"], text=True).strip(),
+ },
+ "suite_git": git_identity(args.binary),
+ "host": {
+ "platform": platform.platform(),
+ "machine": platform.machine(),
+ "python": platform.python_version(),
+ "cpu_count": os.cpu_count(),
+ "page_size": os.sysconf("SC_PAGE_SIZE"),
+ },
+ "configuration": {
+ "documents": counts,
+ "profile": args.profile,
+ "batch_size": args.batch_size,
+ "sample_ms": args.sample_ms,
+ "embedding_mode": args.embed_mode,
+ "instrumentation": "summary",
+ },
+ "corpus": corpus_manifest,
+ "cells": cells,
+ }
+ (args.output / "run_manifest.json").write_text(
+ json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+ )
+ (args.output / "run_manifest.json").chmod(0o600)
+ print(args.output / "run_manifest.json")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/demo/usecases/autoindex/scale/bounded/tests/test_suite.py b/demo/usecases/autoindex/scale/bounded/tests/test_suite.py
new file mode 100644
index 000000000..a1018b450
--- /dev/null
+++ b/demo/usecases/autoindex/scale/bounded/tests/test_suite.py
@@ -0,0 +1,253 @@
+import importlib.util
+import json
+import os
+import tempfile
+import tomllib
+import unittest
+from io import BytesIO
+from pathlib import Path
+from unittest import mock
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def load(name):
+ spec = importlib.util.spec_from_file_location(name, ROOT / f"{name}.py")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+generate = load("generate_corpus")
+analyze = load("analyze")
+run_suite = load("run_suite")
+
+
+class BoundedSuiteTests(unittest.TestCase):
+ def test_corpus_is_reproducible(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ left, right = Path(tmp) / "left", Path(tmp) / "right"
+ a = generate.generate(left, 17)
+ b = generate.generate(right, 17)
+ self.assertEqual(a["sha256"], b["sha256"])
+ self.assertEqual(a["bytes"], b["bytes"])
+ self.assertEqual(
+ (left / "documents.ndjson").read_bytes(),
+ (right / "documents.ndjson").read_bytes(),
+ )
+
+ def test_toml_path_escaping_handles_quotes_and_backslashes(self):
+ value = 'directory with "quotes" and a \\\\ separator'
+ parsed = tomllib.loads(f"data_dir = {run_suite.toml_string(value)}\n")
+ self.assertEqual(parsed["data_dir"], value)
+
+ def test_trace_validator_rejects_gap_and_accounting_error(self):
+ gauges = [
+ {
+ "category": category,
+ "current": 0,
+ "peak": 1,
+ "measurement": analyze.MEASUREMENTS[category],
+ }
+ for category in analyze.CATEGORIES
+ ]
+ event = {
+ "schema": analyze.TRACE_SCHEMA,
+ "seq": 1,
+ "event": "trace_start",
+ "accounting_errors": 0,
+ "dropped_events": 0,
+ "logical": gauges,
+ "logical_total_current": 0,
+ "allocator": {"allocated": 1, "active": 2, "resident": 3, "epoch_ok": True},
+ "process": {
+ "rss": 2,
+ "rss_ok": True,
+ "cpu_time_ns": 3,
+ "cpu_time_ok": True,
+ },
+ "flags": {
+ "sampled": False,
+ "logical_exact": False,
+ "allocator_supported": True,
+ },
+ }
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "trace.ndjson"
+ stop = {**event, "seq": 3, "event": "trace_stop", "accounting_errors": 1}
+ path.write_text(json.dumps(event) + "\n" + json.dumps(stop) + "\n")
+ result = analyze.validate_trace(path)
+ self.assertIn("sequence is not contiguous from 1", result["problems"])
+ self.assertIn("accounting_errors is nonzero", result["problems"])
+
+ def test_trace_validator_rejects_category_and_total_corruption(self):
+ gauges = [
+ {
+ "category": category,
+ "current": 0,
+ "peak": 0,
+ "measurement": analyze.MEASUREMENTS[category],
+ }
+ for category in analyze.CATEGORIES
+ ]
+ base = {
+ "schema": analyze.TRACE_SCHEMA,
+ "accounting_errors": 0,
+ "dropped_events": 0,
+ "logical": gauges,
+ "logical_total_current": 9,
+ "allocator": {
+ "allocated": None,
+ "active": None,
+ "resident": None,
+ "epoch_ok": False,
+ },
+ "process": {
+ "rss": None,
+ "rss_ok": False,
+ "cpu_time_ns": None,
+ "cpu_time_ok": False,
+ },
+ "flags": {
+ "sampled": False,
+ "logical_exact": False,
+ "allocator_supported": False,
+ },
+ }
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "trace.ndjson"
+ start = {**base, "seq": 1, "event": "trace_start"}
+ stop = {**base, "seq": 2, "event": "trace_stop"}
+ stop["logical"] = list(reversed(gauges))
+ path.write_text(json.dumps(start) + "\n" + json.dumps(stop) + "\n")
+ result = analyze.validate_trace(path)
+ self.assertTrue(
+ any("logical_total_current" in problem for problem in result["problems"])
+ )
+ self.assertTrue(
+ any("category vocabulary/order" in problem for problem in result["problems"])
+ )
+
+ def test_budget_enforces_absolute_and_relational_ceilings(self):
+ result = {
+ "cells": [
+ {
+ "documents": 10,
+ "ingest_process": {"peak_rss_bytes": 100},
+ "restart_process": {"peak_rss_bytes": 80},
+ },
+ {
+ "documents": 20,
+ "ingest_process": {"peak_rss_bytes": 300},
+ "restart_process": {"peak_rss_bytes": 90},
+ },
+ ],
+ "relational_memory": {
+ "incremental_slopes": [
+ {
+ "from_documents": 10,
+ "to_documents": 20,
+ "rss_bytes_per_added_document": 20.0,
+ }
+ ]
+ },
+ }
+ budget = {
+ "schema": analyze.BUDGET_SCHEMA,
+ "limits": {
+ "max_peak_ingest_rss_bytes_by_documents": {"20": 299},
+ "max_rss_bytes_per_added_document_by_range": {"10:20": 19.9},
+ },
+ }
+ self.assertEqual(len(analyze.apply_budget(result, budget)), 2)
+
+ def test_result_validator_rejects_missing_phase_and_sentinel_tampering(self):
+ checks = [
+ {
+ "phase": phase,
+ "count": 64,
+ "sentinels": {
+ "doc-00000000": ["doc-00000000"],
+ "doc-00000032": ["doc-00000032"],
+ "doc-00000063": ["doc-00000063"],
+ },
+ }
+ for phase in analyze.CHECK_PHASES
+ ]
+ checks.pop(1)
+ self.assertEqual(
+ analyze.validate_checks(64, checks),
+ ["correctness phases/order mismatch"],
+ )
+ checks.insert(
+ 1,
+ {
+ "phase": "after_flush",
+ "count": 64,
+ "sentinels": {"doc-00000000": ["wrong-id"]},
+ },
+ )
+ self.assertEqual(
+ analyze.validate_checks(64, checks),
+ ["after_flush: sentinel result mismatch"],
+ )
+
+ def test_failed_cell_cleans_up_server_without_masking_request_error(self):
+ class FakeProcess:
+ pid = -1
+
+ def __init__(self):
+ self.running = True
+ self.signals = []
+
+ def poll(self):
+ return None if self.running else 0
+
+ def send_signal(self, value):
+ self.signals.append(value)
+ self.running = False
+
+ def wait(self, timeout=None):
+ self.running = False
+ return 0
+
+ def kill(self):
+ self.running = False
+
+ process = FakeProcess()
+ stream = BytesIO()
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ corpus = root / "corpus"
+ corpus.mkdir()
+ (corpus / "documents.ndjson").write_text("{}\n")
+
+ def fake_start(_binary, cell, _data, suffix, _sample_ms, _embed_mode):
+ return (
+ process,
+ stream,
+ "http://127.0.0.1:1",
+ cell / f"ingest-memory-{suffix}.ndjson",
+ cell / f"server-{suffix}.toml",
+ )
+
+ with mock.patch.object(run_suite, "start_server", side_effect=fake_start), mock.patch.object(
+ run_suite, "request", side_effect=RuntimeError("original request failure")
+ ):
+ with self.assertRaisesRegex(RuntimeError, "original request failure"):
+ run_suite.run_cell(Path("/unused"), corpus, root, 1, 1, 100, "lexical")
+ self.assertTrue(stream.closed)
+ self.assertEqual(process.signals, [run_suite.signal.SIGINT])
+
+ def test_derived_artifact_writer_forces_private_mode(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "analysis.json"
+ path.write_text("old")
+ path.chmod(0o644)
+ analyze.write_private(path, '{"status":"PASS"}\n')
+ self.assertEqual(path.read_text(), '{"status":"PASS"}\n')
+ self.assertEqual(os.stat(path).st_mode & 0o777, 0o600)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs
new file mode 100644
index 000000000..17e56ad11
--- /dev/null
+++ b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs
@@ -0,0 +1,239 @@
+//! Fast resource-accounting regressions for the real engine ingest path.
+//!
+//! These tests deliberately assert logical ownership and collection-capacity
+//! formulas. They do not sample RSS, allocator timing, CPU time, or background
+//! thread scheduling, so they remain deterministic in the normal parallel
+//! suite and require no model download or network access.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use serde_json::{json, Value};
+use tempfile::TempDir;
+use xerj_common::{
+ config::Config,
+ types::{EmbeddingConfig, FieldConfig, FieldType, Schema},
+};
+use xerj_query::ast::SearchRequest;
+
+use crate::{
+ bulk::{
+ process_bulk, with_semantic_lifecycle_hook, SemanticLifecycleFault, SemanticLifecycleHook,
+ SemanticLifecyclePoint,
+ },
+ ingest_memory::{self, Category, Ledger, Measurement},
+ Engine,
+};
+
+const SEMANTIC_CATEGORIES: [Category; 5] = [
+ Category::RawSource,
+ Category::ParsedSource,
+ Category::ParsedVector,
+ Category::PreparedDocument,
+ Category::PreparedVector,
+];
+
+fn engine_with_index(name: &str, semantic: bool) -> (TempDir, Engine) {
+ let dir = TempDir::new().expect("temp data directory");
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ let engine = Engine::new(config).expect("engine");
+ let mut schema = Schema::empty();
+ if semantic {
+ let mut content = FieldConfig::new("content", FieldType::Text);
+ content.options.dimensions = Some(16);
+ content.embedding = Some(EmbeddingConfig {
+ endpoint: None,
+ model: None,
+ target_field: Some("content_vector".into()),
+ });
+ schema.fields.push(content);
+ }
+ engine.create_index(name, schema).expect("create index");
+ if semantic {
+ engine.put_index_mapping(
+ name,
+ json!({"properties": {"content": {"type": "semantic_text"}}}),
+ );
+ }
+ (dir, engine)
+}
+
+fn current(ledger: &Ledger, category: Category) -> u64 {
+ ledger.gauge(category, Measurement::Estimated).current
+}
+
+fn semantic_current(ledger: &Ledger) -> u64 {
+ SEMANTIC_CATEGORIES
+ .into_iter()
+ .map(|category| current(ledger, category))
+ .sum()
+}
+
+#[test]
+fn json_and_caller_vector_capacity_formulas_are_exact() {
+ let value = json!({
+ "content": "owned text",
+ "nested": {"label": "inner"},
+ "content_vector": [0.1, 0.2, [0.3, 0.4]],
+ });
+ let fields = HashSet::from(["content_vector".to_string()]);
+ let vector = value["content_vector"].as_array().unwrap();
+ let nested = vector[2].as_array().unwrap();
+ let expected_vector = vector.capacity() * std::mem::size_of::<Value>()
+ + nested.capacity() * std::mem::size_of::<Value>();
+ assert_eq!(
+ ingest_memory::vector_value_capacity(&value, &fields),
+ expected_vector
+ );
+
+ let expected_document = ingest_memory::estimated_json_heap_excluding(&value, &fields);
+ assert!(expected_document >= "owned text".len() + "inner".len());
+ assert_eq!(
+ expected_document + expected_vector,
+ ingest_memory::estimated_json_heap(&value)
+ );
+}
+
+#[tokio::test]
+async fn unique_docs_vectors_and_parse_failure_release_every_owner() {
+ let name = "bounded-semantic";
+ let (_dir, engine) = engine_with_index(name, true);
+ let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));
+ let valid_derived = r#"{"content":"derive a deterministic lexical embedding"}"#;
+ let valid_supplied = r#"{"content":"keep caller vector","content_vector":[0.1,0.2,0.3,0.4]}"#;
+ let malformed = r#"{"content":"unterminated""#;
+ let body = format!(
+ "{{\"index\":{{\"_index\":\"{name}\",\"_id\":\"derived\"}}}}\n{valid_derived}\n\
+ {{\"index\":{{\"_index\":\"{name}\",\"_id\":\"supplied\"}}}}\n{valid_supplied}\n\
+ {{\"index\":{{\"_index\":\"{name}\",\"_id\":\"bad\"}}}}\n{malformed}\n"
+ );
+
+ let result = ingest_memory::with_test_ledger(ledger, process_bulk(&engine, None, &body)).await;
+ assert!(result.errors);
+ assert_eq!(
+ result
+ .items
+ .iter()
+ .map(|item| item.status)
+ .collect::<Vec<_>>(),
+ [201, 201, 400]
+ );
+
+ // Malformed lines are rejected before an Arc-backed semantic batch owner
+ // exists. Thus the exact owner formula is admitted source-line bytes.
+ let expected_raw = valid_derived.len() + valid_supplied.len();
+ assert_eq!(
+ ledger
+ .gauge(Category::RawSource, Measurement::SerializedSize)
+ .peak,
+ expected_raw as u64
+ );
+ let supplied: Value = serde_json::from_str(valid_supplied).unwrap();
+ let vector_fields = HashSet::from(["content_vector".to_string()]);
+ let supplied_vector = ingest_memory::vector_value_capacity(&supplied, &vector_fields) as u64;
+ assert!(
+ ledger
+ .gauge(Category::ParsedVector, Measurement::ExactCapacity)
+ .peak
+ >= supplied_vector
+ );
+ assert!(
+ ledger
+ .gauge(Category::PreparedVector, Measurement::ExactCapacity)
+ .peak
+ >= supplied_vector
+ );
+ assert_eq!(semantic_current(ledger), 0);
+ assert_eq!(ledger.accounting_errors(), 0);
+ assert_eq!(engine.get_index(name).unwrap().live_doc_count(), 2);
+}
+
+#[tokio::test]
+async fn injected_embedding_and_publication_failures_release_exact_owners() {
+ for (suffix, injected, expected_status) in [
+ ("embed", SemanticLifecycleFault::Embedding, 400),
+ ("publish", SemanticLifecycleFault::Publication, 400),
+ ] {
+ let name = format!("bounded-failure-{suffix}");
+ let (_dir, engine) = engine_with_index(&name, true);
+ let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));
+ let hook: SemanticLifecycleHook = Arc::new(move |_, point| match (injected, point) {
+ (SemanticLifecycleFault::Embedding, SemanticLifecyclePoint::EmbeddingInside) => {
+ injected
+ }
+ (SemanticLifecycleFault::Publication, SemanticLifecyclePoint::PublicationInside) => {
+ injected
+ }
+ _ => SemanticLifecycleFault::None,
+ });
+ let source = r#"{"content":"fault cleanup","content_vector":[0.1,0.2]}"#;
+ let body = format!("{{\"index\":{{\"_index\":\"{name}\",\"_id\":\"one\"}}}}\n{source}\n");
+ let result = ingest_memory::with_test_ledger(
+ ledger,
+ with_semantic_lifecycle_hook(&name, hook, process_bulk(&engine, None, &body)),
+ )
+ .await;
+
+ assert!(result.errors);
+ assert_eq!(result.items[0].status, expected_status);
+ assert_eq!(semantic_current(ledger), 0, "{suffix} leaked an owner");
+ assert_eq!(ledger.accounting_errors(), 0);
+ assert_eq!(engine.get_index(&name).unwrap().live_doc_count(), 0);
+ }
+}
+
+#[tokio::test]
+async fn overwrite_flush_and_merge_churn_returns_to_a_bounded_baseline() {
+ let name = "bounded-overwrite";
+ let (_dir, engine) = engine_with_index(name, false);
+ let index = engine.get_index(name).unwrap();
+ let mut cycle_high_water = Vec::new();
+
+ for cycle in 0..3 {
+ for revision in 0..12 {
+ index
+ .index_document(
+ Some("same-id".into()),
+ json!({
+ "content": format!("fixed-shape revision {revision:02}"),
+ "cycle": cycle,
+ "rank": revision,
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ assert_eq!(index.live_doc_count(), 1);
+ let before_flush = index.memtable_bytes();
+ assert!(before_flush > 0);
+ cycle_high_water.push(before_flush);
+
+ index.flush().await.unwrap();
+ assert_eq!(index.memtable_bytes(), 0);
+ assert_eq!(index.live_doc_count(), 1);
+
+ // Exercise segment hydration and query-cache replacement after every
+ // publication. Correctness is checked on each generation; retained
+ // cache bytes are not asserted here because those logical categories
+ // are deliberately marked unavailable until their owners are wired.
+ let result = index.search(&SearchRequest::default()).await.unwrap();
+ assert_eq!(result.total.value, 1);
+ let cached = index.search(&SearchRequest::default()).await.unwrap();
+ assert_eq!(cached.total.value, 1);
+ }
+
+ // Same number and shape of writes. Sequence metadata may change, but
+ // retained memtable growth must not accumulate across flush boundaries.
+ let first = cycle_high_water[0];
+ assert!(
+ cycle_high_water
+ .iter()
+ .all(|bytes| bytes.abs_diff(first) <= 128),
+ "same-shape overwrite cycles grew across flushes: {cycle_high_water:?}"
+ );
+
+ index.force_merge(1).await.unwrap();
+ assert_eq!(index.memtable_bytes(), 0);
+ assert_eq!(index.live_doc_count(), 1);
+}
diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rs
index d3d1ca908..e08ef6aed 100644
--- a/engine/crates/xerj-engine/src/bulk.rs
+++ b/engine/crates/xerj-engine/src/bulk.rs
@@ -37,7 +37,7 @@ pub(crate) enum SemanticLifecycleFault {
}
#[cfg(test)]
-type SemanticLifecycleHook = std::sync::Arc<
+pub(crate) type SemanticLifecycleHook = std::sync::Arc<
dyn Fn(&str, SemanticLifecyclePoint) -> SemanticLifecycleFault + Send + Sync + 'static,
>;
@@ -102,7 +102,7 @@ pub(crate) fn semantic_lifecycle_hook(
}
#[cfg(test)]
-async fn with_semantic_lifecycle_hook<F: std::future::Future>(
+pub(crate) async fn with_semantic_lifecycle_hook<F: std::future::Future>(
target: impl Into<String>,
hook: SemanticLifecycleHook,
future: F,
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index d53fd7fb0..d692ee240 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -31,6 +31,9 @@ pub mod sql;
pub mod turbo_ingest;
mod write_publication;
+#[cfg(test)]
+mod bounded_ingest_resources_tests;
+
// ── Re-exports ────────────────────────────────────────────────────────────────
pub use engine::{Engine, HealthStatus, IndexInfo};