AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
test(ingest): add bounded shipped-path diagnostics
bugereaaa176fce2e
diff --git a/demo/usecases/autoindex/scale/bounded/.gitignore b/demo/usecases/autoindex/scale/bounded/.gitignorenew file mode 100644index 000000000..c18dd8d83--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/.gitignore@@ -0,0 +1 @@+__pycache__/diff --git a/demo/usecases/autoindex/scale/bounded/README.md b/demo/usecases/autoindex/scale/bounded/README.mdnew file mode 100644index 000000000..973ad0dc4--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/README.md@@ -0,0 +1,114 @@+# Bounded shipped-path ingest diagnostics++This suite replaces expensive full-corpus debugging loops with a small,+deterministic experiment through XERJ's real ES-compatible HTTP path. It proves+bulk ingest, refresh, explicit flush, force-merge, graceful shutdown, restart+recovery, exact counts, and three sentinel lookups while collecting two+independent telemetry streams. The `body` mapping is `semantic_text`, so the+default `--embed-mode lexical` exercises XERJ's deterministic feature-hash+embedding lifecycle; it does not claim neural semantics.++- `xerj.ingest_memory.v1`: XERJ's opt-in logical ownership, jemalloc, RSS, and+ process CPU sampler (`XERJ_INGEST_MEMORY_TRACE=summary`);+- `xerj.process_sample.v1`: external Linux `/proc` status, `smaps_rollup`, I/O,+ thread, and file-descriptor observations.++This is a debugging instrument, not the FinanceBench acceptance benchmark. It+should expose memory-shape defects in seconds or minutes; a candidate still+needs the real correctness and performance benches before publication.++## Run it++Build only the server crate. Fat LTO is useful for release artifacts, but is a+poor default for edit/profile loops because it makes linking slow and obscures+rapid iteration. Use the repository's development profile or a dedicated+profiling profile for experiments; re-run accepted measurements with the exact+release profile used by the compared baseline.++```bash+cd engine+CARGO_PROFILE_RELEASE_LTO=off cargo build --release -j 32 -p xerj-server++cd ..+SUITE=demo/usecases/autoindex/scale/bounded+python3 "$SUITE/generate_corpus.py" \+ --output /tmp/xerj-bounded-corpus \+ --documents 4096+python3 "$SUITE/run_suite.py" \+ --binary engine/target/release/xerj \+ --corpus /tmp/xerj-bounded-corpus \+ --output /tmp/xerj-bounded-run \+ --profile nightly+python3 "$SUITE/analyze.py" /tmp/xerj-bounded-run+```++`run_suite.py` refuses to overwrite an output directory. Every cell uses a+fresh data directory and server process. The restart phase reuses only that+cell's durable data. The manifest records the binary SHA-256 and size, Git+commit and dirty state, corpus hash, host, configuration, correctness results,+and hashes of every raw artifact.++The analyzer fails on:++- malformed or wrong-schema telemetry;+- a missing start/stop event or non-contiguous trace sequence;+- dropped events or retain/release accounting errors;+- nonzero measured ownership at process stop;+- artifact hash/size drift;+- count or sentinel-search corruption after refresh, merge, or restart.++It also reports incremental RSS slopes and peak RSS per document. Those+relations deliberately have no arbitrary absolute threshold. Small cells have+fixed startup cost and host noise, so an accelerating slope is an investigation+signal rather than a release failure.++Once repeated pinned-host runs establish an envelope, write a versioned budget+and check later runs against it:++```bash+python3 "$SUITE/analyze.py" /tmp/xerj-bounded-run \+ --write-baseline-budget demo/usecases/autoindex/scale/bounded/budget.json \+ --budget-headroom 1.05+python3 "$SUITE/analyze.py" /tmp/xerj-candidate-run \+ --budget demo/usecases/autoindex/scale/bounded/budget.json+```++The multiplier is explicit because the repository must choose it from repeated+host variance; the suite does not invent one. A budget can cap per-cell ingest+and restart RSS plus RSS-per-added-document slopes. Do not commit a budget+generated from a single run.++## CI and nightly profiles++- **CI smoke (`--profile ci`):** generate 256 documents and run `64,256`. This validates the+ harness and shipped path; it is not a performance assertion.+- **Nightly diagnostics (`--profile nightly`):** run `256,1024,4096` at least three times on a pinned,+ quiet host. Keep all raw runs and compare medians; do not average away a+ correctness or telemetry-integrity failure.+- **CPU/heap profiles:** use the opt-in profiling build and capture profiles+ alongside this suite. Record their hashes in the run directory. Profiling+ and non-profiling timings are separate populations; never compare one as the+ performance baseline of the other.++## Prompts for an AI reviewer++The artifacts are intentionally JSON/NDJSON plus a short Markdown report, so an+agent can inspect them without a proprietary UI.++> Read `run_manifest.json`, `analysis.json`, and every trace named by the+> manifest. Verify hashes, correctness after refresh/merge/restart, sequence+> continuity, dropped/accounting errors, and stop-time ownership balance.+> Identify the first phase where RSS separates from jemalloc allocated and from+> logical attribution. Do not infer unavailable categories are zero.++> Compare two bounded-run directories produced from the same corpus and cell+> sizes. Confirm binary hashes and build/config differences first. For each+> cell, compare peak and final RSS, allocator allocated/active/resident, logical+> category peaks, write bytes, wall time, and restart RSS. Flag faster-but-wrong+> or incomplete telemetry as a failed comparison.++> Inspect the `/proc` and ingest-memory time series for one cell. Find the+> timestamp and runner phase of peak RSS. Explain whether growth is live+> attributed memory, allocator fragmentation/retention, page cache/file-backed+> RSS, or unattributed memory. State uncertainty and name the next CPU or heap+> profile needed to resolve it.diff --git a/demo/usecases/autoindex/scale/bounded/analyze.py b/demo/usecases/autoindex/scale/bounded/analyze.pynew file mode 100644index 000000000..b8ab2e4cd--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/analyze.py@@ -0,0 +1,455 @@+#!/usr/bin/env python3+"""Validate a bounded run and report relational memory slopes."""++from __future__ import annotations++import argparse+import hashlib+import json+from pathlib import Path++RUN_SCHEMA = "xerj.bounded_run.v1"+TRACE_SCHEMA = "xerj.ingest_memory.v1"+PROC_SCHEMA = "xerj.process_sample.v1"+BUDGET_SCHEMA = "xerj.bounded_budget.v1"+CHECK_PHASES = ["after_refresh", "after_flush", "after_merge", "after_restart"]+CATEGORIES = [+ "http_body",+ "http_rewrite_buffer",+ "raw_source",+ "parsed_source",+ "parsed_vector",+ "prepared_document",+ "prepared_vector",+ "memtable_active",+ "flush_drained",+ "merge_decoded",+ "merge_survivor",+ "merge_json_buffer",+ "merge_parsed",+ "merge_encoded",+ "cache_stored_slices",+ "cache_doc_values",+ "cache_stored_values",+ "cache_sort_shadow",+ "cache_id_positions",+ "cache_row_sequences",+ "cache_decoded_stored",+]+MEASUREMENTS = {+ "http_body": "serialized_size",+ "http_rewrite_buffer": "exact_capacity",+ "raw_source": "serialized_size",+ "parsed_source": "estimated",+ "parsed_vector": "exact_capacity",+ "prepared_document": "estimated",+ "prepared_vector": "exact_capacity",+ "memtable_active": "estimated",+ "flush_drained": "estimated",+ "merge_decoded": "unavailable",+ "merge_survivor": "unavailable",+ "merge_json_buffer": "unavailable",+ "merge_parsed": "unavailable",+ "merge_encoded": "unavailable",+ "cache_stored_slices": "unavailable",+ "cache_doc_values": "unavailable",+ "cache_stored_values": "unavailable",+ "cache_sort_shadow": "unavailable",+ "cache_id_positions": "unavailable",+ "cache_row_sequences": "unavailable",+ "cache_decoded_stored": "unavailable",+}+++def hash_file(path: Path) -> str:+ digest = hashlib.sha256(path.read_bytes()).hexdigest()+ return digest+++def write_private(path: Path, content: str) -> None:+ path.write_text(content, encoding="utf-8")+ path.chmod(0o600)+++def read_ndjson(path: Path) -> list[dict]:+ rows = []+ for number, line in enumerate(path.read_text().splitlines(), 1):+ try:+ rows.append(json.loads(line))+ except json.JSONDecodeError as error:+ raise ValueError(f"{path}:{number}: invalid JSON: {error}") from error+ if not rows:+ raise ValueError(f"{path}: no records")+ return rows+++def validate_trace(path: Path) -> dict:+ rows = read_ndjson(path)+ problems = []+ if any(row.get("schema") != TRACE_SCHEMA for row in rows):+ problems.append("schema mismatch")+ seq = [row.get("seq") for row in rows]+ if seq != list(range(1, len(rows) + 1)):+ problems.append("sequence is not contiguous from 1")+ if rows[0].get("event") != "trace_start":+ problems.append("first event is not trace_start")+ if rows[-1].get("event") != "trace_stop":+ problems.append("last event is not trace_stop")+ expected_events = ["trace_start"] + ["trace_snapshot"] * (len(rows) - 2) + ["trace_stop"]+ if [row.get("event") for row in rows] != expected_events:+ problems.append("summary trace contains an unexpected event or event order")+ if max(int(row.get("accounting_errors", 0)) for row in rows) != 0:+ problems.append("accounting_errors is nonzero")+ if max(int(row.get("dropped_events", 0)) for row in rows) != 0:+ problems.append("dropped_events is nonzero")+ for position, row in enumerate(rows, 1):+ gauges = row.get("logical")+ if not isinstance(gauges, list) or [gauge.get("category") for gauge in gauges] != CATEGORIES:+ problems.append(f"record {position}: logical category vocabulary/order mismatch")+ continue+ if any(+ gauge.get("measurement") != MEASUREMENTS[gauge["category"]] for gauge in gauges+ ):+ problems.append(f"record {position}: logical measurement mismatch")+ if any(+ not isinstance(gauge.get("current"), int)+ or not isinstance(gauge.get("peak"), int)+ or gauge["current"] < 0+ or gauge["peak"] < gauge["current"]+ for gauge in gauges+ ):+ problems.append(f"record {position}: invalid current/peak gauge")+ if row.get("logical_total_current") != sum(gauge["current"] for gauge in gauges):+ problems.append(f"record {position}: logical_total_current is not the gauge sum")+ allocator = row.get("allocator", {})+ process = row.get("process", {})+ flags = row.get("flags", {})+ allocator_values = [allocator.get(key) for key in ("allocated", "active", "resident")]+ allocator_ok = allocator.get("epoch_ok") is True+ if allocator_ok != all(type(value) is int and value >= 0 for value in allocator_values):+ problems.append(f"record {position}: allocator availability/value mismatch")+ if flags.get("allocator_supported") is not allocator_ok:+ problems.append(f"record {position}: allocator_supported disagrees with epoch_ok")+ rss_ok = process.get("rss_ok") is True+ if rss_ok != (type(process.get("rss")) is int and process["rss"] >= 0):+ problems.append(f"record {position}: RSS availability/value mismatch")+ cpu_ok = process.get("cpu_time_ok") is True+ if cpu_ok != (+ type(process.get("cpu_time_ns")) is int and process["cpu_time_ns"] >= 0+ ):+ problems.append(f"record {position}: CPU-time availability/value mismatch")+ expected_sampled = row.get("event") == "trace_snapshot"+ if flags.get("sampled") is not expected_sampled or flags.get("logical_exact") is not False:+ problems.append(f"record {position}: event flags mismatch")+ stop_nonzero = {+ gauge["category"]: gauge["current"]+ for gauge in rows[-1].get("logical", [])+ if gauge.get("measurement") != "unavailable" and gauge.get("current") != 0+ }+ if stop_nonzero:+ problems.append(f"retained logical bytes at trace_stop: {stop_nonzero}")+ return {+ "records": len(rows),+ "peak_rss_bytes": max(+ (int(row["process"]["rss"]) for row in rows if row["process"].get("rss") is not None),+ default=None,+ ),+ "peak_allocated_bytes": max(+ (+ int(row["allocator"]["allocated"])+ for row in rows+ if row["allocator"].get("allocated") is not None+ ),+ default=None,+ ),+ "peak_logical_bytes": max(int(row.get("logical_total_current", 0)) for row in rows),+ "peak_logical_bytes_by_category": {+ category: max(+ int(+ next(+ gauge["peak"]+ for gauge in row["logical"]+ if gauge["category"] == category+ )+ )+ for row in rows+ )+ for category in CATEGORIES+ }+ if not any("logical category vocabulary/order mismatch" in problem for problem in problems)+ else None,+ "problems": problems,+ }+++def validate_proc(path: Path) -> dict:+ rows = read_ndjson(path)+ problems = []+ if any(row.get("schema") != PROC_SCHEMA for row in rows):+ problems.append("schema mismatch")+ timestamps = [int(row["ts_unix_ns"]) for row in rows]+ if timestamps != sorted(timestamps) or len(set(timestamps)) != len(timestamps):+ problems.append("timestamps are not strictly increasing")+ rss = [int(row["vmrss_bytes"]) for row in rows if "vmrss_bytes" in row]+ if not rss:+ problems.append("no VmRSS samples")+ return {+ "records": len(rows),+ "peak_rss_bytes": max(rss, default=None),+ "last_rss_bytes": rss[-1] if rss else None,+ "peak_threads": max((int(row.get("threads", 0)) for row in rows), default=None),+ "peak_fds": max((int(row.get("fds", 0)) for row in rows), default=None),+ "write_bytes": max((int(row.get("io_write_bytes", 0)) for row in rows), default=None),+ "problems": problems,+ }+++def validate_checks(documents: int, checks) -> list[str]:+ problems = []+ expected_ids = [+ "doc-00000000",+ f"doc-{documents // 2:08d}",+ f"doc-{documents - 1:08d}",+ ]+ expected_sentinels = {doc_id: [doc_id] for doc_id in expected_ids}+ if not isinstance(checks, list) or not all(isinstance(check, dict) for check in checks):+ return ["correctness checks are not an object list"]+ if [check.get("phase") for check in checks] != CHECK_PHASES:+ problems.append("correctness phases/order mismatch")+ return problems+ for check in checks:+ if check.get("count") != documents:+ problems.append(f"{check['phase']}: correctness count mismatch")+ if check.get("sentinels") != expected_sentinels:+ problems.append(f"{check['phase']}: sentinel result mismatch")+ return problems+++def analyze(root: Path) -> dict:+ manifest = json.loads((root / "run_manifest.json").read_text())+ problems = []+ if manifest.get("schema") != RUN_SCHEMA:+ problems.append("run manifest schema mismatch")+ results = []+ for cell in manifest["cells"]:+ directory = root / f"docs-{cell['documents']}"+ artifact_problems = []+ for name, expected in cell["artifacts"].items():+ path = directory / name+ if not path.exists() or path.stat().st_size != expected["bytes"]:+ artifact_problems.append(f"{name}: size mismatch or missing")+ elif hash_file(path) != expected["sha256"]:+ artifact_problems.append(f"{name}: sha256 mismatch")+ ingest_trace = validate_trace(directory / "ingest-memory-ingest.ndjson")+ restart_trace = validate_trace(directory / "ingest-memory-restart.ndjson")+ ingest_proc = validate_proc(directory / "proc-ingest.ndjson")+ restart_proc = validate_proc(directory / "proc-restart.ndjson")+ cell_problems = (+ artifact_problems+ + ingest_trace["problems"]+ + restart_trace["problems"]+ + ingest_proc["problems"]+ + restart_proc["problems"]+ )+ category_peaks = ingest_trace["peak_logical_bytes_by_category"]+ if category_peaks is not None and category_peaks["prepared_vector"] == 0:+ cell_problems.append("semantic embedding path produced no prepared vectors")+ cell_problems.extend(validate_checks(cell["documents"], cell.get("checks")))+ results.append(+ {+ "documents": cell["documents"],+ "ingest_trace": ingest_trace,+ "restart_trace": restart_trace,+ "ingest_process": ingest_proc,+ "restart_process": restart_proc,+ "problems": cell_problems,+ }+ )+ problems.extend(f"docs-{cell['documents']}: {item}" for item in cell_problems)++ points = [+ (result["documents"], result["ingest_process"]["peak_rss_bytes"])+ for result in results+ if result["ingest_process"]["peak_rss_bytes"] is not None+ ]+ slopes = []+ for (docs_a, rss_a), (docs_b, rss_b) in zip(points, points[1:]):+ slopes.append(+ {+ "from_documents": docs_a,+ "to_documents": docs_b,+ "rss_bytes_per_added_document": (rss_b - rss_a) / (docs_b - docs_a),+ }+ )+ per_doc = [+ {"documents": docs, "peak_rss_bytes_per_document": rss / docs}+ for docs, rss in points+ ]+ relational = {+ "incremental_slopes": slopes,+ "peak_rss_per_document": per_doc,+ "non_accelerating_incremental_slope": all(+ later["rss_bytes_per_added_document"] <= earlier["rss_bytes_per_added_document"]+ for earlier, later in zip(slopes, slopes[1:])+ )+ if len(slopes) > 1+ else None,+ "declining_peak_rss_per_document": all(+ later["peak_rss_bytes_per_document"] <= earlier["peak_rss_bytes_per_document"]+ for earlier, later in zip(per_doc, per_doc[1:])+ )+ if per_doc+ else None,+ "interpretation": (+ "These are relational diagnostics, not an absolute memory budget. "+ "A false verdict identifies acceleration worth investigating; it is not by itself "+ "a release failure because fixed startup cost and host noise affect small cells."+ ),+ }+ return {+ "schema": "xerj.bounded_analysis.v1",+ "status": "PASS" if not problems else "FAIL",+ "problems": problems,+ "cells": results,+ "relational_memory": relational,+ }+++def apply_budget(result: dict, budget: dict) -> list[str]:+ problems = []+ if budget.get("schema") != BUDGET_SCHEMA:+ return ["budget schema mismatch"]+ limits = budget.get("limits", {})+ by_docs = {str(cell["documents"]): cell for cell in result["cells"]}+ for metric, result_key in (+ ("max_peak_ingest_rss_bytes_by_documents", ("ingest_process", "peak_rss_bytes")),+ ("max_peak_restart_rss_bytes_by_documents", ("restart_process", "peak_rss_bytes")),+ ):+ for documents, ceiling in limits.get(metric, {}).items():+ cell = by_docs.get(documents)+ if cell is None:+ problems.append(f"budget requires missing {documents}-document cell")+ continue+ observed = cell[result_key[0]][result_key[1]]+ if observed is None or observed > ceiling:+ problems.append(f"{metric}[{documents}] observed {observed}, ceiling {ceiling}")+ slope_by_range = {+ f"{slope['from_documents']}:{slope['to_documents']}": slope[+ "rss_bytes_per_added_document"+ ]+ for slope in result["relational_memory"]["incremental_slopes"]+ }+ for span, ceiling in limits.get("max_rss_bytes_per_added_document_by_range", {}).items():+ observed = slope_by_range.get(span)+ if observed is None or observed > ceiling:+ problems.append(+ f"max_rss_bytes_per_added_document_by_range[{span}] "+ f"observed {observed}, ceiling {ceiling}"+ )+ return problems+++def make_budget(result: dict, source_manifest: dict, headroom: float) -> dict:+ def ceilings(process: str):+ return {+ str(cell["documents"]): int(cell[process]["peak_rss_bytes"] * headroom)+ for cell in result["cells"]+ if cell[process]["peak_rss_bytes"] is not None+ }++ return {+ "schema": BUDGET_SCHEMA,+ "source": {+ "binary_sha256": source_manifest["binary"]["sha256"],+ "corpus_sha256": source_manifest["corpus"]["sha256"],+ "documents": source_manifest["configuration"]["documents"],+ "headroom_multiplier": headroom,+ "note": "Use a pinned-host repeated-run envelope; one noisy run is not a stable budget.",+ },+ "limits": {+ "max_peak_ingest_rss_bytes_by_documents": ceilings("ingest_process"),+ "max_peak_restart_rss_bytes_by_documents": ceilings("restart_process"),+ "max_rss_bytes_per_added_document_by_range": {+ f"{slope['from_documents']}:{slope['to_documents']}": (+ max(0.0, slope["rss_bytes_per_added_document"]) * headroom+ )+ for slope in result["relational_memory"]["incremental_slopes"]+ },+ },+ }+++def render(result: dict) -> str:+ lines = [+ "# Bounded ingest diagnostic result",+ "",+ f"Integrity and correctness: **{result['status']}**",+ "",+ "| Documents | Peak ingest RSS | Peak restart RSS | Trace records |",+ "|---:|---:|---:|---:|",+ ]+ for cell in result["cells"]:+ lines.append(+ f"| {cell['documents']} | {cell['ingest_process']['peak_rss_bytes']} | "+ f"{cell['restart_process']['peak_rss_bytes']} | {cell['ingest_trace']['records']} |"+ )+ lines += ["", "## Relational memory diagnostics", ""]+ for slope in result["relational_memory"]["incremental_slopes"]:+ lines.append(+ f"- {slope['from_documents']} → {slope['to_documents']}: "+ f"{slope['rss_bytes_per_added_document']:.2f} RSS bytes per added document"+ )+ lines += [+ "",+ f"- Non-accelerating incremental slope: "+ f"`{result['relational_memory']['non_accelerating_incremental_slope']}`",+ f"- Declining peak RSS/document: "+ f"`{result['relational_memory']['declining_peak_rss_per_document']}`",+ "",+ result["relational_memory"]["interpretation"],+ "",+ ]+ if result["problems"]:+ lines += ["## Problems", ""] + [f"- {problem}" for problem in result["problems"]] + [""]+ return "\n".join(lines)+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("run", type=Path)+ parser.add_argument("--budget", type=Path, help="apply versioned absolute/relational ceilings")+ parser.add_argument("--write-baseline-budget", type=Path)+ parser.add_argument(+ "--budget-headroom",+ type=float,+ default=1.0,+ help="explicit multiplier when writing a budget (default: exact observed envelope)",+ )+ args = parser.parse_args()+ result = analyze(args.run)+ manifest = json.loads((args.run / "run_manifest.json").read_text())+ if args.budget:+ budget_problems = apply_budget(result, json.loads(args.budget.read_text()))+ result["budget"] = {"path": str(args.budget), "problems": budget_problems}+ result["problems"].extend(f"budget: {problem}" for problem in budget_problems)+ if budget_problems:+ result["status"] = "FAIL"+ if args.write_baseline_budget:+ if args.budget_headroom <= 0:+ parser.error("--budget-headroom must be positive")+ budget = make_budget(result, manifest, args.budget_headroom)+ args.write_baseline_budget.write_text(+ json.dumps(budget, indent=2, sort_keys=True) + "\n", encoding="utf-8"+ )+ args.write_baseline_budget.chmod(0o600)+ write_private(+ args.run / "analysis.json",+ json.dumps(result, indent=2, sort_keys=True) + "\n",+ )+ write_private(args.run / "report.md", render(result))+ print(render(result))+ return 0 if result["status"] == "PASS" else 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/demo/usecases/autoindex/scale/bounded/generate_corpus.py b/demo/usecases/autoindex/scale/bounded/generate_corpus.pynew file mode 100644index 000000000..ffe26be15--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/generate_corpus.py@@ -0,0 +1,82 @@+#!/usr/bin/env python3+"""Generate a deterministic, compact corpus for shipped-path memory diagnostics."""++from __future__ import annotations++import argparse+import hashlib+import json+from pathlib import Path++SCHEMA = "xerj.bounded_corpus.v1"+WORDS = (+ "ledger invoice quarter revenue margin audit payable receivable "+ "forecast covenant depreciation treasury subsidiary"+).split()+++def document(number: int) -> dict:+ """Return a deterministic document whose heap shape is stable across runs."""+ words = [WORDS[(number * 7 + offset * 3) % len(WORDS)] for offset in range(48)]+ return {+ "doc_key": f"doc-{number:08d}",+ "company": f"company-{number % 31:02d}",+ "year": 2018 + number % 9,+ "quarter": f"Q{1 + number % 4}",+ "amount": round(((number * 104729) % 10_000_000) / 100.0, 2),+ "body": " ".join(words),+ "labels": [f"bucket-{number % 17:02d}", f"region-{number % 7:02d}"],+ }+++def sha256(path: Path) -> str:+ digest = hashlib.sha256()+ with path.open("rb") as source:+ for chunk in iter(lambda: source.read(1024 * 1024), b""):+ digest.update(chunk)+ return digest.hexdigest()+++def generate(output: Path, count: int) -> dict:+ if output.exists() and any(output.iterdir()):+ raise ValueError(f"refusing nonempty output directory: {output}")+ output.mkdir(parents=True, exist_ok=True, mode=0o700)+ output.chmod(0o700)+ docs_path = output / "documents.ndjson"+ with docs_path.open("w", encoding="utf-8", newline="\n") as target:+ for number in range(count):+ target.write(json.dumps(document(number), sort_keys=True, separators=(",", ":")))+ target.write("\n")+ manifest = {+ "schema": SCHEMA,+ "generator": Path(__file__).name,+ "documents": count,+ "file": docs_path.name,+ "bytes": docs_path.stat().st_size,+ "sha256": sha256(docs_path),+ "first_id": "doc-00000000",+ "middle_id": f"doc-{count // 2:08d}",+ "last_id": f"doc-{count - 1:08d}",+ }+ (output / "corpus_manifest.json").write_text(+ json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"+ )+ docs_path.chmod(0o600)+ (output / "corpus_manifest.json").chmod(0o600)+ return manifest+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--output", type=Path, required=True)+ parser.add_argument("--documents", type=int, default=4096)+ args = parser.parse_args()+ if args.documents < 1:+ parser.error("--documents must be positive")+ manifest = generate(args.output, args.documents)+ print(json.dumps(manifest, sort_keys=True))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/demo/usecases/autoindex/scale/bounded/run_suite.py b/demo/usecases/autoindex/scale/bounded/run_suite.pynew file mode 100644index 000000000..4663e7677--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/run_suite.py@@ -0,0 +1,481 @@+#!/usr/bin/env python3+"""Run bounded ingest diagnostics through XERJ's real ES-compatible HTTP path."""++from __future__ import annotations++import argparse+import hashlib+import json+import os+import platform+import signal+import socket+import subprocess+import threading+import time+import urllib.error+import urllib.request+from pathlib import Path++SCHEMA = "xerj.bounded_run.v1"+PROC_SCHEMA = "xerj.process_sample.v1"+HERE = Path(__file__).resolve().parent+++def hash_file(path: Path) -> str:+ digest = hashlib.sha256()+ with path.open("rb") as source:+ for chunk in iter(lambda: source.read(1024 * 1024), b""):+ digest.update(chunk)+ return digest.hexdigest()+++def toml_string(value: str) -> str:+ """Encode a TOML basic string without interpolating path syntax."""+ return json.dumps(value)+++def free_port() -> int:+ with socket.socket() as listener:+ listener.bind(("127.0.0.1", 0))+ return int(listener.getsockname()[1])+++def request(url: str, method: str = "GET", value=None, raw: bytes | None = None) -> dict:+ if raw is None and value is not None:+ raw = json.dumps(value, separators=(",", ":")).encode()+ req = urllib.request.Request(+ url, data=raw, method=method, headers={"content-type": "application/json"}+ )+ try:+ with urllib.request.urlopen(req, timeout=300) as response:+ payload = response.read()+ except urllib.error.HTTPError as error:+ body = error.read().decode(errors="replace")+ raise RuntimeError(f"{method} {url}: HTTP {error.code}: {body[:2000]}") from error+ return json.loads(payload or b"{}")+++def read_proc(pid: int) -> dict:+ root = Path("/proc") / str(pid)+ result: dict[str, object] = {}+ try:+ for line in (root / "status").read_text().splitlines():+ key, _, value = line.partition(":")+ if key in {"VmRSS", "VmHWM", "VmSize", "RssAnon", "RssFile", "RssShmem"}:+ result[f"{key.lower()}_bytes"] = int(value.split()[0]) * 1024+ for line in (root / "smaps_rollup").read_text().splitlines():+ key, _, value = line.partition(":")+ if key in {"Rss", "Pss", "Private_Clean", "Private_Dirty", "Swap"}:+ result[f"smaps_{key.lower()}_bytes"] = int(value.split()[0]) * 1024+ for line in (root / "io").read_text().splitlines():+ key, _, value = line.partition(":")+ if key in {"read_bytes", "write_bytes", "rchar", "wchar"}:+ result[f"io_{key}"] = int(value.strip())+ stat = (root / "stat").read_text().split()+ result["cpu_user_ticks"] = int(stat[13])+ result["cpu_system_ticks"] = int(stat[14])+ result["threads"] = len(list((root / "task").iterdir()))+ result["fds"] = len(list((root / "fd").iterdir()))+ except (FileNotFoundError, ProcessLookupError, PermissionError, ValueError):+ result["sample_error"] = True+ return result+++class ProcSampler:+ def __init__(self, pid: int, path: Path, phase, interval: float):+ self.pid = pid+ self.path = path+ self.phase = phase+ self.interval = interval+ self.stop = threading.Event()+ self.thread = threading.Thread(target=self._run, name="xerj-proc-sampler", daemon=True)++ def _run(self):+ with self.path.open("w", encoding="utf-8", buffering=1) as target:+ os.fchmod(target.fileno(), 0o600)+ while not self.stop.is_set():+ process = read_proc(self.pid)+ if process.get("sample_error"):+ break+ sample = {+ "schema": PROC_SCHEMA,+ "ts_unix_ns": time.time_ns(),+ "pid": self.pid,+ "phase": self.phase(),+ **process,+ }+ target.write(json.dumps(sample, sort_keys=True) + "\n")+ self.stop.wait(self.interval)++ def __enter__(self):+ self.thread.start()+ return self++ def __exit__(self, *_):+ self.stop.set()+ self.thread.join(timeout=max(2.0, self.interval * 2))+++def start_server(+ binary: Path, cell: Path, data: Path, suffix: str, sample_ms: int, embed_mode: str+):+ es_port, rest_port, grpc_port = free_port(), free_port(), free_port()+ config = cell / f"server-{suffix}.toml"+ trace = cell / f"ingest-memory-{suffix}.ndjson"+ log = cell / f"server-{suffix}.log"+ config.write_text(+ "[server]\n"+ f'rest_port = {rest_port}\n'+ f'grpc_port = {grpc_port}\n'+ f'es_compat_port = {es_port}\n'+ 'bind_address = "127.0.0.1"\n'+ f"data_dir = {toml_string(str(data))}\n\n"+ "[auth]\nenabled = false\n",+ encoding="utf-8",+ )+ config.chmod(0o600)+ env = os.environ.copy()+ env.update(+ {+ "XERJ_INGEST_MEMORY_TRACE": "summary",+ "XERJ_INGEST_MEMORY_SAMPLE_MS": str(sample_ms),+ "XERJ_INGEST_MEMORY_OUTPUT": str(trace),+ "XERJ_EMBED_MODE": embed_mode,+ "XERJ_LOG": "info",+ }+ )+ stream = log.open("wb")+ os.fchmod(stream.fileno(), 0o600)+ try:+ process = subprocess.Popen(+ [str(binary), "--insecure", "--config", str(config), "--data-dir", str(data)],+ stdout=stream,+ stderr=subprocess.STDOUT,+ env=env,+ )+ except BaseException:+ stream.close()+ raise+ url = f"http://127.0.0.1:{es_port}"+ for _ in range(240):+ if process.poll() is not None:+ stream.close()+ raise RuntimeError(f"server exited {process.returncode}; inspect {log}")+ try:+ request(url)+ return process, stream, url, trace, config+ except Exception:+ time.sleep(0.05)+ process.kill()+ process.wait()+ stream.close()+ raise RuntimeError(f"server did not become ready; inspect {log}")+++def stop_server(process: subprocess.Popen, stream) -> int:+ process.send_signal(signal.SIGINT)+ try:+ code = process.wait(timeout=60)+ except subprocess.TimeoutExpired:+ process.kill()+ code = process.wait()+ raise RuntimeError("server did not stop gracefully within 60 seconds")+ finally:+ stream.close()+ if code != 0:+ raise RuntimeError(f"server exited with status {code}")+ return code+++def cleanup_server(process: subprocess.Popen, stream) -> None:+ """Best-effort no-throw cleanup for an interrupted or failed cell.++ This is deliberately safe to call after ``stop_server``. It never replaces+ the request/verification exception that caused a cell to abort.+ """+ try:+ if process.poll() is None:+ try:+ process.send_signal(signal.SIGINT)+ process.wait(timeout=10)+ except (ProcessLookupError, subprocess.TimeoutExpired):+ if process.poll() is None:+ process.kill()+ process.wait()+ except BaseException:+ # Cleanup is a last-resort guard. The original cell failure is more+ # actionable and must retain its traceback.+ pass+ finally:+ try:+ stream.close()+ except BaseException:+ pass+++def verify(url: str, index: str, count: int, ids: list[str], phase: str) -> dict:+ counted = request(f"{url}/{index}/_count")+ actual = int(counted.get("count", -1))+ if actual != count:+ raise RuntimeError(f"{phase}: expected {count} documents, got {actual}")+ observed = {}+ for doc_id in ids:+ result = request(+ f"{url}/{index}/_search",+ "POST",+ {"size": 2, "query": {"term": {"doc_key": doc_id}}, "track_total_hits": True},+ )+ total = int(result["hits"]["total"]["value"])+ hits = [hit["_id"] for hit in result["hits"]["hits"]]+ if total != 1 or hits != [doc_id]:+ raise RuntimeError(f"{phase}: sentinel {doc_id} returned total={total}, ids={hits}")+ observed[doc_id] = hits+ return {"phase": phase, "count": actual, "sentinels": observed}+++def bulk_load(url: str, index: str, docs: Path, count: int, batch_size: int, set_phase):+ batch: list[bytes] = []+ sent = 0+ with docs.open("rb") as source:+ for number, raw in enumerate(source):+ if number >= count:+ break+ action = json.dumps(+ {"index": {"_index": index, "_id": f"doc-{number:08d}"}},+ separators=(",", ":"),+ ).encode()+ batch.extend((action, raw.rstrip(b"\n")))+ if number % batch_size == batch_size - 1 or number + 1 == count:+ set_phase(f"bulk:{sent}-{number + 1}")+ response = request(f"{url}/_bulk", "POST", raw=b"\n".join(batch) + b"\n")+ if response.get("errors"):+ errors = [+ item for item in response.get("items", []) if next(iter(item.values())).get("error")+ ]+ raise RuntimeError(f"bulk errors: {json.dumps(errors[:3])}")+ sent = number + 1+ batch.clear()+ if sent != count:+ raise RuntimeError(f"corpus ended at {sent}, expected {count}")+++def run_cell(+ binary: Path,+ corpus: Path,+ root: Path,+ count: int,+ batch: int,+ sample_ms: int,+ embed_mode: str,+):+ cell = root / f"docs-{count}"+ cell.mkdir(mode=0o700)+ data = cell / "data"+ data.mkdir(mode=0o700)+ index = "bounded-diagnostics"+ phase_lock = threading.Lock()+ phase_name = "boot"++ def phase(value=None):+ nonlocal phase_name+ with phase_lock:+ if value is not None:+ phase_name = value+ return phase_name++ timeline = []+ process, stream, url, trace, config = start_server(+ binary, cell, data, "ingest", sample_ms, embed_mode+ )+ try:+ with ProcSampler(process.pid, cell / "proc-ingest.ndjson", phase, sample_ms / 1000):+ phase("create")+ request(+ f"{url}/{index}",+ "PUT",+ {+ "mappings": {+ "properties": {+ "doc_key": {"type": "keyword"},+ "company": {"type": "keyword"},+ "year": {"type": "long"},+ "quarter": {"type": "keyword"},+ "amount": {"type": "double"},+ "body": {"type": "semantic_text"},+ "labels": {"type": "keyword"},+ }+ }+ },+ )+ started = time.monotonic_ns()+ bulk_load(url, index, corpus / "documents.ndjson", count, batch, phase)+ timeline.append({"phase": "bulk", "elapsed_ns": time.monotonic_ns() - started})+ ids = ["doc-00000000", f"doc-{count // 2:08d}", f"doc-{count - 1:08d}"]+ phase("refresh")+ started = time.monotonic_ns()+ request(f"{url}/{index}/_refresh", "POST")+ checks = [verify(url, index, count, ids, "after_refresh")]+ timeline.append(+ {"phase": "refresh_and_verify", "elapsed_ns": time.monotonic_ns() - started}+ )+ phase("flush")+ started = time.monotonic_ns()+ flush = request(f"{url}/{index}/_flush", "POST")+ checks.append(verify(url, index, count, ids, "after_flush"))+ timeline.append(+ {"phase": "flush_and_verify", "elapsed_ns": time.monotonic_ns() - started}+ )+ phase("merge")+ started = time.monotonic_ns()+ merge = request(f"{url}/{index}/_forcemerge?max_num_segments=1", "POST")+ checks.append(verify(url, index, count, ids, "after_merge"))+ timeline.append(+ {"phase": "merge_and_verify", "elapsed_ns": time.monotonic_ns() - started}+ )+ phase("shutdown")+ started = time.monotonic_ns()+ stop_server(process, stream)+ timeline.append({"phase": "shutdown", "elapsed_ns": time.monotonic_ns() - started})+ finally:+ cleanup_server(process, stream)++ phase("restart")+ started = time.monotonic_ns()+ process2, stream2, url2, trace2, config2 = start_server(+ binary, cell, data, "restart", sample_ms, embed_mode+ )+ try:+ with ProcSampler(process2.pid, cell / "proc-restart.ndjson", phase, sample_ms / 1000):+ checks.append(verify(url2, index, count, ids, "after_restart"))+ timeline.append(+ {"phase": "restart_and_verify", "elapsed_ns": time.monotonic_ns() - started}+ )+ phase("restart_shutdown")+ started = time.monotonic_ns()+ stop_server(process2, stream2)+ timeline.append(+ {"phase": "restart_shutdown", "elapsed_ns": time.monotonic_ns() - started}+ )+ finally:+ cleanup_server(process2, stream2)+ for path in cell.iterdir():+ if path.is_file():+ path.chmod(0o600)+ return {+ "documents": count,+ "batch_size": batch,+ "checks": checks,+ "flush_response": flush,+ "merge_response": merge,+ "timeline": timeline,+ "artifacts": {+ path.name: {"bytes": path.stat().st_size, "sha256": hash_file(path)}+ for path in sorted(cell.iterdir())+ if path.is_file()+ },+ }+++def git_identity(binary: Path) -> dict:+ try:+ root = subprocess.check_output(+ ["git", "-C", str(HERE), "rev-parse", "--show-toplevel"], text=True+ ).strip()+ return {+ "root": root,+ "commit": subprocess.check_output(+ ["git", "-C", root, "rev-parse", "HEAD"], text=True+ ).strip(),+ "dirty": bool(+ subprocess.check_output(["git", "-C", root, "status", "--porcelain"], text=True)+ ),+ }+ except (subprocess.CalledProcessError, FileNotFoundError):+ return {"root": None, "commit": None, "dirty": None}+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--binary", type=Path, required=True)+ parser.add_argument("--corpus", type=Path, required=True)+ parser.add_argument("--output", type=Path, required=True)+ parser.add_argument("--profile", choices=("ci", "nightly"), default="nightly")+ parser.add_argument("--documents", help="override profile with increasing comma-separated cells")+ parser.add_argument("--batch-size", type=int, default=128)+ parser.add_argument("--sample-ms", type=int, default=100)+ parser.add_argument("--embed-mode", default="lexical")+ args = parser.parse_args()+ profile_counts = {"ci": "64,256", "nightly": "256,1024,4096"}+ counts = [int(value) for value in (args.documents or profile_counts[args.profile]).split(",")]+ if sorted(set(counts)) != counts or len(counts) < 2 or counts[0] < 1:+ parser.error("--documents must contain at least two strictly increasing positive counts")+ if args.batch_size < 1:+ parser.error("--batch-size must be positive")+ if args.sample_ms < 1:+ parser.error("--sample-ms must be positive")+ args.binary = args.binary.resolve()+ args.corpus = args.corpus.resolve()+ if not os.access(args.binary, os.X_OK):+ parser.error(f"not executable: {args.binary}")+ corpus_manifest = json.loads((args.corpus / "corpus_manifest.json").read_text())+ if corpus_manifest["documents"] < counts[-1]:+ parser.error("corpus contains fewer documents than the largest cell")+ if hash_file(args.corpus / corpus_manifest["file"]) != corpus_manifest["sha256"]:+ parser.error("corpus hash does not match corpus_manifest.json")+ if args.output.exists():+ parser.error(f"output already exists: {args.output}")+ args.output.mkdir(parents=True, mode=0o700)+ args.output.chmod(0o700)+ started = time.time_ns()+ cells = [+ run_cell(+ args.binary,+ args.corpus,+ args.output,+ count,+ args.batch_size,+ args.sample_ms,+ args.embed_mode,+ )+ for count in counts+ ]+ manifest = {+ "schema": SCHEMA,+ "started_unix_ns": started,+ "finished_unix_ns": time.time_ns(),+ "binary": {+ "path": str(args.binary),+ "bytes": args.binary.stat().st_size,+ "sha256": hash_file(args.binary),+ "version": subprocess.check_output([str(args.binary), "--version"], text=True).strip(),+ },+ "suite_git": git_identity(args.binary),+ "host": {+ "platform": platform.platform(),+ "machine": platform.machine(),+ "python": platform.python_version(),+ "cpu_count": os.cpu_count(),+ "page_size": os.sysconf("SC_PAGE_SIZE"),+ },+ "configuration": {+ "documents": counts,+ "profile": args.profile,+ "batch_size": args.batch_size,+ "sample_ms": args.sample_ms,+ "embedding_mode": args.embed_mode,+ "instrumentation": "summary",+ },+ "corpus": corpus_manifest,+ "cells": cells,+ }+ (args.output / "run_manifest.json").write_text(+ json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8"+ )+ (args.output / "run_manifest.json").chmod(0o600)+ print(args.output / "run_manifest.json")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/demo/usecases/autoindex/scale/bounded/tests/test_suite.py b/demo/usecases/autoindex/scale/bounded/tests/test_suite.pynew file mode 100644index 000000000..a1018b450--- /dev/null+++ b/demo/usecases/autoindex/scale/bounded/tests/test_suite.py@@ -0,0 +1,253 @@+import importlib.util+import json+import os+import tempfile+import tomllib+import unittest+from io import BytesIO+from pathlib import Path+from unittest import mock++ROOT = Path(__file__).resolve().parents[1]+++def load(name):+ spec = importlib.util.spec_from_file_location(name, ROOT / f"{name}.py")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++generate = load("generate_corpus")+analyze = load("analyze")+run_suite = load("run_suite")+++class BoundedSuiteTests(unittest.TestCase):+ def test_corpus_is_reproducible(self):+ with tempfile.TemporaryDirectory() as tmp:+ left, right = Path(tmp) / "left", Path(tmp) / "right"+ a = generate.generate(left, 17)+ b = generate.generate(right, 17)+ self.assertEqual(a["sha256"], b["sha256"])+ self.assertEqual(a["bytes"], b["bytes"])+ self.assertEqual(+ (left / "documents.ndjson").read_bytes(),+ (right / "documents.ndjson").read_bytes(),+ )++ def test_toml_path_escaping_handles_quotes_and_backslashes(self):+ value = 'directory with "quotes" and a \\\\ separator'+ parsed = tomllib.loads(f"data_dir = {run_suite.toml_string(value)}\n")+ self.assertEqual(parsed["data_dir"], value)++ def test_trace_validator_rejects_gap_and_accounting_error(self):+ gauges = [+ {+ "category": category,+ "current": 0,+ "peak": 1,+ "measurement": analyze.MEASUREMENTS[category],+ }+ for category in analyze.CATEGORIES+ ]+ event = {+ "schema": analyze.TRACE_SCHEMA,+ "seq": 1,+ "event": "trace_start",+ "accounting_errors": 0,+ "dropped_events": 0,+ "logical": gauges,+ "logical_total_current": 0,+ "allocator": {"allocated": 1, "active": 2, "resident": 3, "epoch_ok": True},+ "process": {+ "rss": 2,+ "rss_ok": True,+ "cpu_time_ns": 3,+ "cpu_time_ok": True,+ },+ "flags": {+ "sampled": False,+ "logical_exact": False,+ "allocator_supported": True,+ },+ }+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "trace.ndjson"+ stop = {**event, "seq": 3, "event": "trace_stop", "accounting_errors": 1}+ path.write_text(json.dumps(event) + "\n" + json.dumps(stop) + "\n")+ result = analyze.validate_trace(path)+ self.assertIn("sequence is not contiguous from 1", result["problems"])+ self.assertIn("accounting_errors is nonzero", result["problems"])++ def test_trace_validator_rejects_category_and_total_corruption(self):+ gauges = [+ {+ "category": category,+ "current": 0,+ "peak": 0,+ "measurement": analyze.MEASUREMENTS[category],+ }+ for category in analyze.CATEGORIES+ ]+ base = {+ "schema": analyze.TRACE_SCHEMA,+ "accounting_errors": 0,+ "dropped_events": 0,+ "logical": gauges,+ "logical_total_current": 9,+ "allocator": {+ "allocated": None,+ "active": None,+ "resident": None,+ "epoch_ok": False,+ },+ "process": {+ "rss": None,+ "rss_ok": False,+ "cpu_time_ns": None,+ "cpu_time_ok": False,+ },+ "flags": {+ "sampled": False,+ "logical_exact": False,+ "allocator_supported": False,+ },+ }+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "trace.ndjson"+ start = {**base, "seq": 1, "event": "trace_start"}+ stop = {**base, "seq": 2, "event": "trace_stop"}+ stop["logical"] = list(reversed(gauges))+ path.write_text(json.dumps(start) + "\n" + json.dumps(stop) + "\n")+ result = analyze.validate_trace(path)+ self.assertTrue(+ any("logical_total_current" in problem for problem in result["problems"])+ )+ self.assertTrue(+ any("category vocabulary/order" in problem for problem in result["problems"])+ )++ def test_budget_enforces_absolute_and_relational_ceilings(self):+ result = {+ "cells": [+ {+ "documents": 10,+ "ingest_process": {"peak_rss_bytes": 100},+ "restart_process": {"peak_rss_bytes": 80},+ },+ {+ "documents": 20,+ "ingest_process": {"peak_rss_bytes": 300},+ "restart_process": {"peak_rss_bytes": 90},+ },+ ],+ "relational_memory": {+ "incremental_slopes": [+ {+ "from_documents": 10,+ "to_documents": 20,+ "rss_bytes_per_added_document": 20.0,+ }+ ]+ },+ }+ budget = {+ "schema": analyze.BUDGET_SCHEMA,+ "limits": {+ "max_peak_ingest_rss_bytes_by_documents": {"20": 299},+ "max_rss_bytes_per_added_document_by_range": {"10:20": 19.9},+ },+ }+ self.assertEqual(len(analyze.apply_budget(result, budget)), 2)++ def test_result_validator_rejects_missing_phase_and_sentinel_tampering(self):+ checks = [+ {+ "phase": phase,+ "count": 64,+ "sentinels": {+ "doc-00000000": ["doc-00000000"],+ "doc-00000032": ["doc-00000032"],+ "doc-00000063": ["doc-00000063"],+ },+ }+ for phase in analyze.CHECK_PHASES+ ]+ checks.pop(1)+ self.assertEqual(+ analyze.validate_checks(64, checks),+ ["correctness phases/order mismatch"],+ )+ checks.insert(+ 1,+ {+ "phase": "after_flush",+ "count": 64,+ "sentinels": {"doc-00000000": ["wrong-id"]},+ },+ )+ self.assertEqual(+ analyze.validate_checks(64, checks),+ ["after_flush: sentinel result mismatch"],+ )++ def test_failed_cell_cleans_up_server_without_masking_request_error(self):+ class FakeProcess:+ pid = -1++ def __init__(self):+ self.running = True+ self.signals = []++ def poll(self):+ return None if self.running else 0++ def send_signal(self, value):+ self.signals.append(value)+ self.running = False++ def wait(self, timeout=None):+ self.running = False+ return 0++ def kill(self):+ self.running = False++ process = FakeProcess()+ stream = BytesIO()+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ corpus = root / "corpus"+ corpus.mkdir()+ (corpus / "documents.ndjson").write_text("{}\n")++ def fake_start(_binary, cell, _data, suffix, _sample_ms, _embed_mode):+ return (+ process,+ stream,+ "http://127.0.0.1:1",+ cell / f"ingest-memory-{suffix}.ndjson",+ cell / f"server-{suffix}.toml",+ )++ with mock.patch.object(run_suite, "start_server", side_effect=fake_start), mock.patch.object(+ run_suite, "request", side_effect=RuntimeError("original request failure")+ ):+ with self.assertRaisesRegex(RuntimeError, "original request failure"):+ run_suite.run_cell(Path("/unused"), corpus, root, 1, 1, 100, "lexical")+ self.assertTrue(stream.closed)+ self.assertEqual(process.signals, [run_suite.signal.SIGINT])++ def test_derived_artifact_writer_forces_private_mode(self):+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "analysis.json"+ path.write_text("old")+ path.chmod(0o644)+ analyze.write_private(path, '{"status":"PASS"}\n')+ self.assertEqual(path.read_text(), '{"status":"PASS"}\n')+ self.assertEqual(os.stat(path).st_mode & 0o777, 0o600)+++if __name__ == "__main__":+ unittest.main()diff --git a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rsnew file mode 100644index 000000000..17e56ad11--- /dev/null+++ b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs@@ -0,0 +1,239 @@+//! Fast resource-accounting regressions for the real engine ingest path.+//!+//! These tests deliberately assert logical ownership and collection-capacity+//! formulas. They do not sample RSS, allocator timing, CPU time, or background+//! thread scheduling, so they remain deterministic in the normal parallel+//! suite and require no model download or network access.++use std::collections::HashSet;+use std::sync::Arc;++use serde_json::{json, Value};+use tempfile::TempDir;+use xerj_common::{+ config::Config,+ types::{EmbeddingConfig, FieldConfig, FieldType, Schema},+};+use xerj_query::ast::SearchRequest;++use crate::{+ bulk::{+ process_bulk, with_semantic_lifecycle_hook, SemanticLifecycleFault, SemanticLifecycleHook,+ SemanticLifecyclePoint,+ },+ ingest_memory::{self, Category, Ledger, Measurement},+ Engine,+};++const SEMANTIC_CATEGORIES: [Category; 5] = [+ Category::RawSource,+ Category::ParsedSource,+ Category::ParsedVector,+ Category::PreparedDocument,+ Category::PreparedVector,+];++fn engine_with_index(name: &str, semantic: bool) -> (TempDir, Engine) {+ let dir = TempDir::new().expect("temp data directory");+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ let engine = Engine::new(config).expect("engine");+ let mut schema = Schema::empty();+ if semantic {+ let mut content = FieldConfig::new("content", FieldType::Text);+ content.options.dimensions = Some(16);+ content.embedding = Some(EmbeddingConfig {+ endpoint: None,+ model: None,+ target_field: Some("content_vector".into()),+ });+ schema.fields.push(content);+ }+ engine.create_index(name, schema).expect("create index");+ if semantic {+ engine.put_index_mapping(+ name,+ json!({"properties": {"content": {"type": "semantic_text"}}}),+ );+ }+ (dir, engine)+}++fn current(ledger: &Ledger, category: Category) -> u64 {+ ledger.gauge(category, Measurement::Estimated).current+}++fn semantic_current(ledger: &Ledger) -> u64 {+ SEMANTIC_CATEGORIES+ .into_iter()+ .map(|category| current(ledger, category))+ .sum()+}++#[test]+fn json_and_caller_vector_capacity_formulas_are_exact() {+ let value = json!({+ "content": "owned text",+ "nested": {"label": "inner"},+ "content_vector": [0.1, 0.2, [0.3, 0.4]],+ });+ let fields = HashSet::from(["content_vector".to_string()]);+ let vector = value["content_vector"].as_array().unwrap();+ let nested = vector[2].as_array().unwrap();+ let expected_vector = vector.capacity() * std::mem::size_of::<Value>()+ + nested.capacity() * std::mem::size_of::<Value>();+ assert_eq!(+ ingest_memory::vector_value_capacity(&value, &fields),+ expected_vector+ );++ let expected_document = ingest_memory::estimated_json_heap_excluding(&value, &fields);+ assert!(expected_document >= "owned text".len() + "inner".len());+ assert_eq!(+ expected_document + expected_vector,+ ingest_memory::estimated_json_heap(&value)+ );+}++#[tokio::test]+async fn unique_docs_vectors_and_parse_failure_release_every_owner() {+ let name = "bounded-semantic";+ let (_dir, engine) = engine_with_index(name, true);+ let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));+ let valid_derived = r#"{"content":"derive a deterministic lexical embedding"}"#;+ let valid_supplied = r#"{"content":"keep caller vector","content_vector":[0.1,0.2,0.3,0.4]}"#;+ let malformed = r#"{"content":"unterminated""#;+ let body = format!(+ "{{\"index\":{{\"_index\":\"{name}\",\"_id\":\"derived\"}}}}\n{valid_derived}\n\+ {{\"index\":{{\"_index\":\"{name}\",\"_id\":\"supplied\"}}}}\n{valid_supplied}\n\+ {{\"index\":{{\"_index\":\"{name}\",\"_id\":\"bad\"}}}}\n{malformed}\n"+ );++ let result = ingest_memory::with_test_ledger(ledger, process_bulk(&engine, None, &body)).await;+ assert!(result.errors);+ assert_eq!(+ result+ .items+ .iter()+ .map(|item| item.status)+ .collect::<Vec<_>>(),+ [201, 201, 400]+ );++ // Malformed lines are rejected before an Arc-backed semantic batch owner+ // exists. Thus the exact owner formula is admitted source-line bytes.+ let expected_raw = valid_derived.len() + valid_supplied.len();+ assert_eq!(+ ledger+ .gauge(Category::RawSource, Measurement::SerializedSize)+ .peak,+ expected_raw as u64+ );+ let supplied: Value = serde_json::from_str(valid_supplied).unwrap();+ let vector_fields = HashSet::from(["content_vector".to_string()]);+ let supplied_vector = ingest_memory::vector_value_capacity(&supplied, &vector_fields) as u64;+ assert!(+ ledger+ .gauge(Category::ParsedVector, Measurement::ExactCapacity)+ .peak+ >= supplied_vector+ );+ assert!(+ ledger+ .gauge(Category::PreparedVector, Measurement::ExactCapacity)+ .peak+ >= supplied_vector+ );+ assert_eq!(semantic_current(ledger), 0);+ assert_eq!(ledger.accounting_errors(), 0);+ assert_eq!(engine.get_index(name).unwrap().live_doc_count(), 2);+}++#[tokio::test]+async fn injected_embedding_and_publication_failures_release_exact_owners() {+ for (suffix, injected, expected_status) in [+ ("embed", SemanticLifecycleFault::Embedding, 400),+ ("publish", SemanticLifecycleFault::Publication, 400),+ ] {+ let name = format!("bounded-failure-{suffix}");+ let (_dir, engine) = engine_with_index(&name, true);+ let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));+ let hook: SemanticLifecycleHook = Arc::new(move |_, point| match (injected, point) {+ (SemanticLifecycleFault::Embedding, SemanticLifecyclePoint::EmbeddingInside) => {+ injected+ }+ (SemanticLifecycleFault::Publication, SemanticLifecyclePoint::PublicationInside) => {+ injected+ }+ _ => SemanticLifecycleFault::None,+ });+ let source = r#"{"content":"fault cleanup","content_vector":[0.1,0.2]}"#;+ let body = format!("{{\"index\":{{\"_index\":\"{name}\",\"_id\":\"one\"}}}}\n{source}\n");+ let result = ingest_memory::with_test_ledger(+ ledger,+ with_semantic_lifecycle_hook(&name, hook, process_bulk(&engine, None, &body)),+ )+ .await;++ assert!(result.errors);+ assert_eq!(result.items[0].status, expected_status);+ assert_eq!(semantic_current(ledger), 0, "{suffix} leaked an owner");+ assert_eq!(ledger.accounting_errors(), 0);+ assert_eq!(engine.get_index(&name).unwrap().live_doc_count(), 0);+ }+}++#[tokio::test]+async fn overwrite_flush_and_merge_churn_returns_to_a_bounded_baseline() {+ let name = "bounded-overwrite";+ let (_dir, engine) = engine_with_index(name, false);+ let index = engine.get_index(name).unwrap();+ let mut cycle_high_water = Vec::new();++ for cycle in 0..3 {+ for revision in 0..12 {+ index+ .index_document(+ Some("same-id".into()),+ json!({+ "content": format!("fixed-shape revision {revision:02}"),+ "cycle": cycle,+ "rank": revision,+ }),+ )+ .await+ .unwrap();+ }+ assert_eq!(index.live_doc_count(), 1);+ let before_flush = index.memtable_bytes();+ assert!(before_flush > 0);+ cycle_high_water.push(before_flush);++ index.flush().await.unwrap();+ assert_eq!(index.memtable_bytes(), 0);+ assert_eq!(index.live_doc_count(), 1);++ // Exercise segment hydration and query-cache replacement after every+ // publication. Correctness is checked on each generation; retained+ // cache bytes are not asserted here because those logical categories+ // are deliberately marked unavailable until their owners are wired.+ let result = index.search(&SearchRequest::default()).await.unwrap();+ assert_eq!(result.total.value, 1);+ let cached = index.search(&SearchRequest::default()).await.unwrap();+ assert_eq!(cached.total.value, 1);+ }++ // Same number and shape of writes. Sequence metadata may change, but+ // retained memtable growth must not accumulate across flush boundaries.+ let first = cycle_high_water[0];+ assert!(+ cycle_high_water+ .iter()+ .all(|bytes| bytes.abs_diff(first) <= 128),+ "same-shape overwrite cycles grew across flushes: {cycle_high_water:?}"+ );++ index.force_merge(1).await.unwrap();+ assert_eq!(index.memtable_bytes(), 0);+ assert_eq!(index.live_doc_count(), 1);+}diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rsindex d3d1ca908..e08ef6aed 100644--- a/engine/crates/xerj-engine/src/bulk.rs+++ b/engine/crates/xerj-engine/src/bulk.rs@@ -37,7 +37,7 @@ pub(crate) enum SemanticLifecycleFault {}#[cfg(test)]-type SemanticLifecycleHook = std::sync::Arc<+pub(crate) type SemanticLifecycleHook = std::sync::Arc<dyn Fn(&str, SemanticLifecyclePoint) -> SemanticLifecycleFault + Send + Sync + 'static,>;@@ -102,7 +102,7 @@ pub(crate) fn semantic_lifecycle_hook(}#[cfg(test)]-async fn with_semantic_lifecycle_hook<F: std::future::Future>(+pub(crate) async fn with_semantic_lifecycle_hook<F: std::future::Future>(target: impl Into<String>,hook: SemanticLifecycleHook,future: F,diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex d53fd7fb0..d692ee240 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -31,6 +31,9 @@ pub mod sql;pub mod turbo_ingest;mod write_publication;+#[cfg(test)]+mod bounded_ingest_resources_tests;+// ── Re-exports ────────────────────────────────────────────────────────────────pub use engine::{Engine, HealthStatus, IndexInfo};