AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(case-study): measured detection quality — XERJ 6/6 vs grep 4/6, 32x cheaper triage
xerj-org9efefb695d84
diff --git a/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md b/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.mdnew file mode 100644index 000000000..5d6fdf1bd--- /dev/null+++ b/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md@@ -0,0 +1,103 @@+# Detection quality — measured against known ground truth++The honest question about any code-security tool is not "does it look+sophisticated" but **"does it find real vulnerabilities, and how much do you have+to read to triage what it returns?"**++PR #69 fixed six real, network-reachable vulnerabilities in this exact codebase,+at locations we know precisely. That makes them a **labelled test set**. This+document reports how the XERJ substrate scores on it, against the baseline an+auditor would otherwise use: grep on the raw tree.++Harness: [`detection_quality.py`](../../examples/rust-ast-audit/detection_quality.py).+It runs against the **pre-fix** index/tree (the bugs must be present to be found)+and, for each bug, issues the query an auditor would write **for that bug class** —+never the bug's own name or line, which would be cheating.++## Results++| ID | bug | XERJ finds it | candidates | rank | XERJ triage tokens | grep finds it | grep files | grep triage tokens |+|---|---|---|---:|---:|---:|---|---:|---:|+| F1 | query_string paren recursion → stack overflow | ✅ | 186 | 6 | 25,049 | ✅ | 21 | 1,010,124 |+| F2 | `IndexName` `..` → restore deletes parent of data_dir | ✅ | 13 | 12 | 2,894 | ✅ | 2 | 327,583 |+| F6 | `more_like_this` cross-product → unbounded alloc | ✅ | 27 | 18 | 17,895 | ✅ | 56 | 1,340,250 |+| F7 | `max_fields_per_index` not enforced on ingest | ✅ | 3 | 2 | 1,050 | ❌ | 1 | 13,366 |+| F8 | bulk NDJSON parse-phase amplification | ✅ | 30 | 9 | 25,760 | ❌ | 0 | 0 |+| F9 | snapshot name unvalidated → write outside repo | ✅ | 21 | 19 | 11,474 | ✅ | 3 | 23,680 |+| | **RECALL** | **6 / 6** | | | **84,122** | **4 / 6** | | **2,715,003** |++**Read that as two separate results.**++1. **Recall: 6/6 vs grep's 4/6.** grep cannot find F7 or F8 *even in principle*.+ Both bugs are **missing code** — an absent limit check, an absent action cap.+ You cannot grep for code that is not there. The substrate can, because+ "function on the ingest path that never consults a configured limit" is a+ property of the AST, not a string. (`grep max_actions_per_bulk` on the pre-fix+ tree returns **0 lines** — the config key only exists *after* the fix.)+2. **Triage cost: 84K vs 2.7M tokens — 32× cheaper.** This is the bigger result.+ grep's literal recall is perfect *for literals*, but a grep hit is a **file**;+ an AST hit is a **function**. Triaging `grep with_capacity` for F6 means+ reading 56 whole files (1.34M tokens). The equivalent XERJ query returns 27+ functions (17.9K tokens). Same bug, two orders of magnitude less reading.++## The part that would be dishonest to omit++**These 6/6 are in-sample.** The first run of this harness scored **XERJ 3/6 —+worse than grep's 4/6.** Publishing only the tuned number would be a lie about+the method, so here is what happened in between. The test set found three real+defects in the *extractor*, each of which I fixed:++| Miss | Root cause (a real detector bug, not a query typo) | Fix |+|---|---|---|+| **F6** | `with_capacity(fields.len() * like.len())` — the argument parser split on the first `)`, truncating to `fields.len(` and **losing the `*`** that is the entire cross-product signal | balanced-paren argument extraction |+| **F6/F8** | taint provenance tracked **only direct parameters**; the sizes come from *locals* derived from a parameter (`let fields = params.get(…)`) | one-hop local-derivation tracking |+| **F2** | the validator signal was **presence-only**. Pre-fix `restore_snapshot` calls `IndexName::new` *after* `remove_dir_all` — decorative validation scored as "guarded". Worse, a generic `starts_with` earlier in the function masked it | ordering signal (`guard_after_destructive_op`), restricted to strong path/name validators |++So the honest claim is: **the substrate reaches 6/6 after being debugged against+this set, and the debugging exposed three generalizable weaknesses** (argument+parsing, provenance depth, order-sensitivity) that any taint-style tool has to+get right. Each fix was verified to also **go quiet on the fixed code** — e.g.+`guard_after_destructive_op` is `true` for pre-fix `restore_snapshot` and `false`+after, so it discriminates rather than just matching.++**The out-of-sample result is the [`/_sql` stack overflow](FINDINGS.md)** — not in+the test set, found on current `main`, and confirmed by aborting a live server.+That is the number to weigh: one previously-unknown Critical, proven by execution.++## Precision is the weak axis — stated plainly++Recall is 6/6; **ranking is mediocre.** The true positive lands at rank 6 of 186+(F1), 12 of 13 (F2), 18 of 27 (F6), 19 of 21 (F9). The substrate is a good+*filter* (207k lines → tens of functions) and a poor *ranker*. In practice that+is acceptable — a 27-candidate list is readable in full, which is the entire point+— but nobody should read these numbers as "it points at the bug." It points at a+small set that contains the bug.++## What a clean sweep of current `main` returned++Running the same lenses on current `main` (post-#69) produced **no new confirmed+findings beyond the `/_sql` DoS**, and one candidate that was **refuted by+testing** — recorded because a finder without a verifier would have published it:++- **Refuted: highlight-tag allocation blowup.** `highlight_text_with_terms`+ (`aggs.rs:7562`) allocates `text.len() + merged.len() * (pre.len() + post.len())`+ where `pre`/`post` are the **user-supplied** `pre_tags`/`post_tags` from the+ search body — a product of two attacker-influenced values, and exactly the F6+ shape. **Tested against a live server**: a 20,000-match document with tag sizes+ from 100 B to 100 KB returned HTTP 200 in ~2 ms with no memory growth. Reading+ the response showed why — the highlighter emits a **truncated fragment** (363+ chars, tag applied once), so `merged` is the matches within one fragment, not+ the document. The allocation is bounded. **Not a bug.**+- `guard_after_destructive_op` flagged exactly **1** function on current main+ (`engine.rs:353 Engine::new`) — a startup path with operator-controlled inputs,+ not network-reachable. Not a finding.++## Reproduce++```bash+python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \+ --tree /path/to/pre-fix/checkout --astdir ./ast-pre69+```++Token counts use tiktoken `cl100k_base` (a public GPT-4-class BPE) as an+LLM-token proxy; the harness labels its fallback if tiktoken is absent.diff --git a/docs/case-studies/xerj-self-audit/README.md b/docs/case-studies/xerj-self-audit/README.mdindex 6f9a738a1..355cdcdea 100644--- a/docs/case-studies/xerj-self-audit/README.md+++ b/docs/case-studies/xerj-self-audit/README.md@@ -18,25 +18,37 @@ that grep cannot, and — the part most demos skip — a **ground-truth recall tthat measures whether the substrate actually finds real bugs before we trust iton unknown ones.-## The credibility test: does this substrate actually find known bugs?+## Detection quality, measured against known ground truthThe day before this audit, PR #69 fixed **six** real, network-reachable-vulnerabilities in this exact codebase, at locations we know precisely. So we-indexed the **pre-#69 commit** and asked whether the substrate's queries surface-each one — and re-ran them on current `main` to confirm the signal flips when the-bug is fixed.--| Bug (PR #69) | Substrate signal | Pre-#69 | Current main |-|---|---|---|---|-| **F1** query_string stack overflow | recursion **cycle** with no depth guard | cycle **UNGUARDED** (rank #5 of 186) | same cycle **GUARDED** |-| **F7** field-limit bypass on ingest | `evolve_schema_*` reads a config limit | `reads_config_limit: false` | `reads_config_limit: true` |-| **F2/F9** path traversal | `.join(param)` with no containment check | present, no `containment` validator | `containment` validator present |-| **F6** MLT cross-product OOM | allocation arg is a **product** of params | present | capped |--The two we instrumented most sharply — F1 and F7 — **flip exactly on the fix**.-That is the difference between "this looks like a security tool" and "this-detects the class of bug it claims to." It also means the signal is honest: when-the code is fixed, the query goes quiet.+vulnerabilities in this exact codebase, at known locations — a labelled test set.+Full methodology and per-bug numbers: **[DETECTION-QUALITY.md](DETECTION-QUALITY.md)**.++| | XERJ substrate | grep baseline |+|---|---|---|+| **Bugs found (recall)** | **6 / 6** | 4 / 6 |+| **Tokens to triage all candidates** | **84,122** | 2,715,003 (**32×** more) |+| **Unit of a hit** | a function | a whole file |++Two findings, not one. **Recall:** grep cannot find F7 or F8 *even in principle* —+both are **missing code** (an absent limit check, an absent action cap), and you+cannot grep for code that is not there; `grep max_actions_per_bulk` on the pre-fix+tree returns 0 lines. **Triage cost** is the bigger result: grep's literal recall+is fine, but triaging `grep with_capacity` for F6 means reading 56 whole files+(1.34M tokens) versus 27 functions (17.9K).++**The caveat that matters: 6/6 is in-sample.** The first run scored **XERJ 3/6 —+worse than grep.** The test set exposed three real defects in the *extractor*+(argument parsing truncating `a.len() * b.len()` at the first paren; taint+provenance not following one hop into locals; validator detection being+presence-only rather than order-aware). Fixing those took it to 6/6, and each fix+was verified to **go quiet on the patched code** — so it discriminates rather than+merely matching. The genuinely out-of-sample result is the `/_sql` Critical below:+not in the test set, found on current `main`, proven by crashing a server.++Precision is the weak axis and is stated plainly in DETECTION-QUALITY.md: the true+positive lands at rank 6 of 186, 12 of 13, 18 of 27, 19 of 21. The substrate is a+strong *filter* and a mediocre *ranker*.## The headline: XERJ-assisted vs. "the agent reads all 197 files"diff --git a/docs/case-studies/xerj-self-audit/REPRODUCE.md b/docs/case-studies/xerj-self-audit/REPRODUCE.mdindex 1089202cf..f3a672427 100644--- a/docs/case-studies/xerj-self-audit/REPRODUCE.md+++ b/docs/case-studies/xerj-self-audit/REPRODUCE.md@@ -69,6 +69,20 @@ curl -s localhost:9310/rustfns/_search -H 'Content-Type: application/json' \# → reads_config_limit:true (fixed)```+## 3b. Detection quality: XERJ vs grep on all six known bugs++```bash+python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \+ --tree /tmp/pre69 --astdir ./ast-pre69+# → RECALL XERJ 6/6 grep 4/6+# TRIAGE XERJ 84,122 tokens grep 2,715,003 tokens+```++Per-bug candidates, ranks, and the in-sample caveat are in+[DETECTION-QUALITY.md](DETECTION-QUALITY.md). Note that `grep max_actions_per_bulk`+on the pre-fix tree returns **0 lines** — F7 and F8 are *missing* code, which no+grep can find.+## 4. The Critical finding — crash a live server```bashdiff --git a/docs/examples/rust-ast-audit/detection_quality.py b/docs/examples/rust-ast-audit/detection_quality.pynew file mode 100644index 000000000..3fed1a9d4--- /dev/null+++ b/docs/examples/rust-ast-audit/detection_quality.py@@ -0,0 +1,266 @@+#!/usr/bin/env python3+"""Measure DETECTION QUALITY against known ground truth: XERJ vs grep.++The six vulnerabilities fixed by PR #69 are known bugs at known locations in this+exact codebase. That makes them a labelled test set — the only honest way to+answer "is this substrate actually good at finding vulnerabilities, or does it+just look sophisticated?"++For each bug this runs:+ * the XERJ query a security auditor would write for that bug CLASS+ (never the bug's own name/line — that would be cheating), and+ * the best equivalent grep an auditor would run on the raw tree,++then reports, for both: did the true positive appear (RECALL), how many+candidates came with it (PRECISION = 1/candidates for a single-TP query), and+how much an agent must READ to triage the candidate set (tokens).++Run against the PRE-FIX tree/index — the bugs must be present to be found.++Usage:+ python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \+ --tree /path/to/pre-fix/checkout [--json]+"""++import argparse+import json+import os+import re+import subprocess+import sys+import urllib.request+++def es(url, index, body, size=0):+ q = dict(body)+ q["size"] = size+ req = urllib.request.Request(+ f"{url}/{index}/_search", data=json.dumps(q).encode(), method="POST"+ )+ req.add_header("Content-Type", "application/json")+ with urllib.request.urlopen(req, timeout=60) as r:+ return json.loads(r.read().decode())+++def toks(text):+ """Token count via tiktoken when available, else a labelled chars/4 estimate."""+ try:+ import tiktoken++ return len(tiktoken.get_encoding("cl100k_base").encode(text)), "tiktoken"+ except Exception:+ return len(text) // 4, "chars/4-estimate"+++# ── the test set ─────────────────────────────────────────────────────────────+# `truth` identifies the fixing location. `xerj` is the class-level query.+# `grep` is the fairest equivalent an auditor would run without a graph.++CASES = [+ {+ "id": "F1",+ "title": "query_string unbounded paren recursion → stack overflow",+ "truth_file": "crates/xerj-query/src/parser.rs",+ "truth_fns": ["parse_qs_unary", "parse_qs_or", "parse_qs_and"],+ "kind": "cycle", # answered by find_recursion_cycles.py, not a term query+ "grep": r"fn parse_",+ "grep_note": "no grep can express 'recursion cycle with no depth guard'; "+ "the closest proxy is 'every parse fn', then read them all",+ },+ {+ "id": "F2",+ "title": "IndexName accepts '..' → snapshot restore deletes parent of data_dir",+ "truth_file": "crates/xerj-engine/src/engine.rs",+ "truth_fns": ["restore_snapshot"],+ "kind": "query",+ # a destructive filesystem op whose validation runs AFTER it (or not at all)+ "xerj": {+ "query": {"bool": {+ "filter": [{"term": {"sinks": "fs_delete"}}, {"term": {"is_test": False}}],+ "should": [{"term": {"guard_after_destructive_op": True}},+ {"bool": {"must_not": [{"term": {"validators": "containment"}}]}}],+ "minimum_should_match": 1}}},+ "grep": r"remove_dir_all",+ },+ {+ "id": "F6",+ "title": "more_like_this fields×like cross-product → unbounded alloc",+ "truth_file": "crates/xerj-query/src/parser.rs",+ "truth_fns": ["parse_more_like_this"],+ "kind": "query",+ "xerj": {+ "query": {"bool": {+ "filter": [{"term": {"alloc_product": True}}, {"term": {"is_test": False}}]}}},+ "grep": r"with_capacity",+ },+ {+ "id": "F7",+ "title": "max_fields_per_index not enforced on the ingest path",+ "truth_file": "crates/xerj-engine/src/index.rs",+ "truth_fns": ["evolve_schema_from_doc", "evolve_schema_from_docs"],+ "kind": "query",+ "xerj": {+ "query": {"bool": {+ "filter": [{"prefix": {"fn_name": "evolve_schema"}},+ {"term": {"is_test": False}}],+ "must_not": [{"term": {"reads_config_limit": True}}]}}},+ "grep": r"max_fields_per_index",+ },+ {+ "id": "F8",+ "title": "bulk NDJSON parse-phase memory amplification (no action cap)",+ "truth_file": "crates/xerj-engine/src/bulk.rs",+ "truth_fns": ["process_bulk_with_opts"],+ "kind": "query",+ # allocation sized from tainted input, in a fn that consults NO config limit+ "xerj": {+ "query": {"bool": {+ "filter": [{"term": {"is_test": False}},+ {"range": {"alloc_from_param_count": {"gte": 1}}},+ {"term": {"sinks": "deserialize"}}],+ "must_not": [{"term": {"reads_config_limit": True}}]}}},+ "grep": r"max_actions_per_bulk",+ "grep_note": "the fix's config key does not exist pre-fix, so this grep "+ "returns 0 — the auditor cannot grep for absent code",+ },+ {+ "id": "F9",+ "title": "snapshot repo_path/name unvalidated → write outside the repo",+ "truth_file": "crates/xerj-engine/src/engine.rs",+ "truth_fns": ["create_snapshot", "restore_snapshot"],+ "kind": "query",+ "xerj": {+ "query": {"bool": {+ "filter": [{"term": {"path_join_from_param": True}},+ {"term": {"is_test": False}}],+ "must_not": [{"term": {"validators": "containment"}}]}}},+ "grep": r"\.join\(name\)",+ },+]+++def run_xerj(url, index, case):+ hits = es(url, index, case["xerj"], size=200)+ total = hits["hits"]["total"]["value"]+ rows = [(h["_source"].get("file", ""), h["_source"].get("fn_name", ""),+ h["_source"].get("line_start"), h["_source"].get("body", ""))+ for h in hits["hits"]["hits"]]+ rank = None+ for i, (f, fn, _ln, _b) in enumerate(rows, start=1):+ if f == case["truth_file"] and fn in case["truth_fns"]:+ rank = i+ break+ read_text = "".join(b for _f, _fn, _l, b in rows)+ n, tk = toks(read_text)+ return {"candidates": total, "returned": len(rows), "hit": rank is not None,+ "rank": rank, "triage_tokens": n, "tokenizer": tk}+++def run_grep(tree, case):+ pat = case["grep"]+ try:+ out = subprocess.run(+ ["grep", "-rn", "--include=*.rs", "-E", pat, "engine/crates"],+ cwd=tree, capture_output=True, text=True, timeout=120).stdout+ except Exception as e:+ return {"error": str(e)}+ lines = [l for l in out.split("\n") if l.strip()]+ files = sorted({l.split(":")[0] for l in lines})+ hit = any(case["truth_file"] in f for f in files)+ # an auditor must read each matching FILE to triage a grep hit+ total_chars = 0+ for f in files:+ p = os.path.join(tree, f)+ try:+ total_chars += len(open(p, encoding="utf8", errors="replace").read())+ except OSError:+ pass+ n, tk = toks("x" * total_chars) if total_chars else (0, "n/a")+ # tokenizing a synthetic string is wrong; count real content instead+ if total_chars:+ buf = []+ for f in files:+ try:+ buf.append(open(os.path.join(tree, f), encoding="utf8",+ errors="replace").read())+ except OSError:+ pass+ n, tk = toks("".join(buf))+ return {"match_lines": len(lines), "match_files": len(files), "hit": hit,+ "triage_tokens": n, "tokenizer": tk}+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--url", default="http://127.0.0.1:9310")+ ap.add_argument("--suffix", default="_pre69")+ ap.add_argument("--tree", required=True, help="pre-fix checkout root")+ ap.add_argument("--astdir", help="pre-fix ast-out dir (for the cycle case)")+ ap.add_argument("--json", action="store_true")+ args = ap.parse_args()++ index = f"rustfns{args.suffix}"+ results = []+ for case in CASES:+ row = {"id": case["id"], "title": case["title"]}+ if case["kind"] == "cycle":+ # the call-graph cycle finder answers F1, not a term query+ if args.astdir:+ out = subprocess.run(+ [sys.executable,+ os.path.join(os.path.dirname(os.path.abspath(__file__)),+ "find_recursion_cycles.py"), args.astdir, "--json"],+ capture_output=True, text=True).stdout+ d = json.loads(out)+ unguarded = [c for c in d["cycles"] if not c["guarded"]]+ rank = None+ for i, c in enumerate(sorted(unguarded, key=lambda c: -c["size"]), 1):+ if any(m["fn"] in case["truth_fns"] for m in c["members"]):+ rank = i+ break+ # triage = reading the members of the cycles ranked at/above the TP+ n, tk = toks(out)+ row["xerj"] = {"candidates": len(unguarded), "hit": rank is not None,+ "rank": rank, "triage_tokens": n, "tokenizer": tk,+ "via": "find_recursion_cycles.py"}+ else:+ row["xerj"] = {"skipped": "pass --astdir for the cycle case"}+ else:+ row["xerj"] = run_xerj(args.url, index, case)+ row["grep"] = run_grep(args.tree, case)+ row["grep_pattern"] = case["grep"]+ if case.get("grep_note"):+ row["grep_note"] = case["grep_note"]+ results.append(row)++ xerj_hits = sum(1 for r in results if r["xerj"].get("hit"))+ grep_hits = sum(1 for r in results if r["grep"].get("hit"))+ summary = {+ "cases": len(results),+ "xerj_recall": f"{xerj_hits}/{len(results)}",+ "grep_recall": f"{grep_hits}/{len(results)}",+ "xerj_total_triage_tokens": sum(r["xerj"].get("triage_tokens", 0) for r in results),+ "grep_total_triage_tokens": sum(r["grep"].get("triage_tokens", 0) for r in results),+ "results": results,+ }+ if args.json:+ print(json.dumps(summary, indent=2))+ return++ print(f"DETECTION QUALITY — {len(results)} known bugs (PR #69 test set)\n")+ hdr = f"{'ID':<4} {'XERJ hit':<9} {'cand':>5} {'rank':>5} {'XERJ tok':>9} {'grep hit':<9} {'files':>6} {'grep tok':>9}"+ print(hdr)+ print("-" * len(hdr))+ for r in results:+ x, g = r["xerj"], r["grep"]+ print(f"{r['id']:<4} {str(x.get('hit')):<9} {x.get('candidates','-'):>5} "+ f"{str(x.get('rank','-')):>5} {x.get('triage_tokens',0):>9,} "+ f"{str(g.get('hit')):<9} {g.get('match_files',0):>6} {g.get('triage_tokens',0):>9,}")+ print("-" * len(hdr))+ print(f"\nRECALL XERJ {summary['xerj_recall']} grep {summary['grep_recall']}")+ print(f"TRIAGE XERJ {summary['xerj_total_triage_tokens']:,} tokens "+ f"grep {summary['grep_total_triage_tokens']:,} tokens")+++if __name__ == "__main__":+ main()diff --git a/docs/examples/rust-ast-audit/ingest.py b/docs/examples/rust-ast-audit/ingest.pyindex deba740cc..816e1243d 100644--- a/docs/examples/rust-ast-audit/ingest.py+++ b/docs/examples/rust-ast-audit/ingest.py@@ -64,6 +64,8 @@ FUNCTIONS_MAPPING = {"alloc_args": TEXT,"alloc_param_names": KEYWORD,"alloc_product": BOOL,+ "alloc_args_all": TEXT,+ "guard_after_destructive_op": BOOL,"path_join_args": TEXT,"path_join_from_param": BOOL,"calls_self": BOOL,diff --git a/docs/examples/rust-ast-audit/rust_ast_index.py b/docs/examples/rust-ast-audit/rust_ast_index.pyindex 8c4bbaf07..2654b4f81 100644--- a/docs/examples/rust-ast-audit/rust_ast_index.py+++ b/docs/examples/rust-ast-audit/rust_ast_index.py@@ -276,7 +276,13 @@ class Extractor:extractors = [e for e in EXTRACTORS if e in params]param_names = self.param_names(params_node, src) if params_node else []- alloc_from_param = self.alloc_from_param(body, param_names)+ # Taint provenance is ONE HOP: a size or path is usually not the parameter+ # itself but a local derived from it (`let fields = params.get(...)` then+ # `with_capacity(fields.len() * like.len())`). Measured on the PR #69 test+ # set, tracking direct parameters only MISSED two of six known bugs.+ tainted = param_names + self.derived_locals(body, param_names)+ alloc_from_param = self.alloc_from_param(body, tainted)+ alloc_args_all = self.alloc_args_all(body)record = {"id": fn_id,@@ -328,10 +334,18 @@ class Extractor:"alloc_from_param_count": len(alloc_from_param),"alloc_args": [a["arg"] for a in alloc_from_param][:12],"alloc_param_names": sorted({a["param"] for a in alloc_from_param}),- "alloc_product": any("*" in a["arg"] for a in alloc_from_param),- # what gets path-joined — "join a raw parameter" is the traversal shape- "path_join_args": self.join_args(body, param_names)[:12],- "path_join_from_param": bool(self.join_args(body, param_names)),+ # The cross-product blowup shape is a MULTIPLICATION in an allocation+ # size, whether or not provenance resolves. Gating this on parameter+ # provenance hid PR #69's F6, whose factors are locals.+ "alloc_product": any("*" in a for a in alloc_args_all),+ "alloc_args_all": alloc_args_all[:16],+ # what gets path-joined — "join tainted input" is the traversal shape+ "path_join_args": self.join_args(body, tainted)[:12],+ "path_join_from_param": bool(self.join_args(body, tainted)),+ # ORDER MATTERS: a validator that runs AFTER the destructive op is not+ # a guard. PR #69's F2 called IndexName::new *after* remove_dir_all —+ # a presence-only validator signal scores that function as "guarded".+ "guard_after_destructive_op": self.guard_after_sink(body),# recursion + depth guarding (the stack-overflow shape)"calls_self": (name + "(") in body,"has_depth_guard": any(@@ -548,6 +562,98 @@ class Extractor:breakreturn out+ def derived_locals(self, body, param_names):+ """Locals bound from an expression mentioning a parameter (one hop).++ `let fields = params.get("fields")` makes `fields` carry `params`' taint.+ Without this hop, an allocation sized by `fields.len()` reads as+ untainted — which is how PR #69's F6 and F8 escaped detection.+ """+ if not param_names:+ return []+ real = [p for p in param_names if p and p != "self"]+ if not real:+ return []+ out = []+ for line in body.split("\n"):+ s = line.strip()+ if not s.startswith("let "):+ continue+ if "=" not in s:+ continue+ lhs, rhs = s[4:].split("=", 1)+ if not any(p in rhs for p in real):+ continue+ name = lhs.replace("mut ", "").split(":")[0].strip()+ if name and name.replace("_", "").isalnum():+ out.append(name)+ return out++ @staticmethod+ def balanced_arg(body, open_at, opener="(", closer=")"):+ """Text of the argument list starting after `open_at`, paren-balanced.++ Splitting on the first `)` truncates `with_capacity(a.len() * b.len())`+ to `a.len(` — which silently loses the multiplication that IS the+ cross-product signal. Measured: that truncation hid PR #69's F6.+ """+ depth = 1+ i = open_at+ n = len(body)+ while i < n and depth > 0:+ c = body[i]+ if c == opener:+ depth += 1+ elif c == closer:+ depth -= 1+ if depth == 0:+ return body[open_at:i]+ i += 1+ return body[open_at : min(n, open_at + 200)]++ def alloc_args_all(self, body):+ """Every allocation-size argument, regardless of provenance."""+ out = []+ for pat, closer in (("with_capacity(", ")"), (".reserve(", ")"),+ ("vec![", "]"), (".repeat(", ")")):+ start = 0+ while True:+ i = body.find(pat, start)+ if i < 0:+ break+ start = i + len(pat)+ opener = "[" if closer == "]" else "("+ arg = self.balanced_arg(body, i + len(pat), opener, closer)+ out.append(arg.strip()[:160])+ return out++ DESTRUCTIVE = ("remove_dir_all(", "remove_file(", "remove_dir(", "create_dir_all(",+ "File::create(", "fs::write(", "fs::rename(")+ # Only STRONG name/path validators count for the ordering test. Generic+ # helpers like `starts_with(` appear all over a large function for unrelated+ # reasons (glob matching), and taking the earliest of those masks a genuinely+ # late path validator — that is what hid PR #69's F2 on the first pass.+ PATH_GUARDS = ("IndexName::new", "IndexName::validate", "Namespace::new",+ "validate_snapshot", "canonicalize(", 'contains("..")')++ def guard_after_sink(self, body):+ """True when a path/name validator only appears AFTER a destructive op.++ Presence of a validator says nothing about whether it protects anything —+ ordering does. This is the exact shape of PR #69's F2: `IndexName::new`+ ran *after* `remove_dir_all` had already deleted the directory.+ """+ sink_at = min((body.find(p) for p in self.DESTRUCTIVE if p in body),+ default=-1)+ if sink_at < 0:+ return False+ positions = [body.find(g) for g in self.PATH_GUARDS if g in body]+ if not positions:+ return False+ # unguarded-before-the-sink: no strong validator runs before it, but one+ # does run after — the validation is decorative.+ return min(positions) > sink_at+def join_args(self, body, param_names):"""`.join(x)` arguments that mention a parameter name.diff --git a/docs/examples/rust-ast-audit/sample-coverage.json b/docs/examples/rust-ast-audit/sample-coverage.jsonindex 55c0b03f5..3224be44c 100644--- a/docs/examples/rust-ast-audit/sample-coverage.json+++ b/docs/examples/rust-ast-audit/sample-coverage.json@@ -5,17 +5,17 @@"files_with_error_nodes": 0,"error_files": [],"error_nodes_total": 0,- "ast_function_nodes": 5095,- "emitted_function_records": 5095,+ "ast_function_nodes": 5098,+ "emitted_function_records": 5098,"function_coverage_pct": 100.0,- "call_edges": 87137,+ "call_edges": 87150,"routes": 477,"unsafe_block_sites": 23,"unsafe_fn_declarations": 2,"functions_with_unsafe": 21,"handler_shaped_functions": 299,"functions_with_narrowing_casts": 433,- "functions_with_alloc_from_param": 149,+ "functions_with_alloc_from_param": 218,"per_file": [{"file": "crates/xerj-ai/examples/onnx_experimental.rs",@@ -1199,10 +1199,10 @@{"file": "crates/xerj-engine/src/sql.rs","crate": "xerj-engine",- "bytes": 22557,- "lines": 725,- "ast_functions": 12,- "emitted_functions": 12,+ "bytes": 24892,+ "lines": 779,+ "ast_functions": 15,+ "emitted_functions": 15,"error_nodes": 0},{diff --git a/docs/examples/rust-ast-audit/sample-detection-quality.json b/docs/examples/rust-ast-audit/sample-detection-quality.jsonnew file mode 100644index 000000000..4174874e7--- /dev/null+++ b/docs/examples/rust-ast-audit/sample-detection-quality.json@@ -0,0 +1,131 @@+{+ "cases": 6,+ "xerj_recall": "6/6",+ "grep_recall": "4/6",+ "xerj_total_triage_tokens": 84122,+ "grep_total_triage_tokens": 2715003,+ "results": [+ {+ "id": "F1",+ "title": "query_string unbounded paren recursion \u2192 stack overflow",+ "xerj": {+ "candidates": 186,+ "hit": true,+ "rank": 6,+ "triage_tokens": 25049,+ "tokenizer": "tiktoken",+ "via": "find_recursion_cycles.py"+ },+ "grep": {+ "match_lines": 162,+ "match_files": 21,+ "hit": true,+ "triage_tokens": 1010124,+ "tokenizer": "tiktoken"+ },+ "grep_pattern": "fn parse_",+ "grep_note": "no grep can express 'recursion cycle with no depth guard'; the closest proxy is 'every parse fn', then read them all"+ },+ {+ "id": "F2",+ "title": "IndexName accepts '..' \u2192 snapshot restore deletes parent of data_dir",+ "xerj": {+ "candidates": 13,+ "returned": 13,+ "hit": true,+ "rank": 12,+ "triage_tokens": 2894,+ "tokenizer": "tiktoken"+ },+ "grep": {+ "match_lines": 2,+ "match_files": 2,+ "hit": true,+ "triage_tokens": 327583,+ "tokenizer": "tiktoken"+ },+ "grep_pattern": "remove_dir_all"+ },+ {+ "id": "F6",+ "title": "more_like_this fields\u00d7like cross-product \u2192 unbounded alloc",+ "xerj": {+ "candidates": 27,+ "returned": 27,+ "hit": true,+ "rank": 18,+ "triage_tokens": 17895,+ "tokenizer": "tiktoken"+ },+ "grep": {+ "match_lines": 314,+ "match_files": 56,+ "hit": true,+ "triage_tokens": 1340250,+ "tokenizer": "tiktoken"+ },+ "grep_pattern": "with_capacity"+ },+ {+ "id": "F7",+ "title": "max_fields_per_index not enforced on the ingest path",+ "xerj": {+ "candidates": 3,+ "returned": 3,+ "hit": true,+ "rank": 2,+ "triage_tokens": 1050,+ "tokenizer": "tiktoken"+ },+ "grep": {+ "match_lines": 2,+ "match_files": 1,+ "hit": false,+ "triage_tokens": 13366,+ "tokenizer": "tiktoken"+ },+ "grep_pattern": "max_fields_per_index"+ },+ {+ "id": "F8",+ "title": "bulk NDJSON parse-phase memory amplification (no action cap)",+ "xerj": {+ "candidates": 30,+ "returned": 30,+ "hit": true,+ "rank": 9,+ "triage_tokens": 25760,+ "tokenizer": "tiktoken"+ },+ "grep": {+ "match_lines": 0,+ "match_files": 0,+ "hit": false,+ "triage_tokens": 0,+ "tokenizer": "n/a"+ },+ "grep_pattern": "max_actions_per_bulk",+ "grep_note": "the fix's config key does not exist pre-fix, so this grep returns 0 \u2014 the auditor cannot grep for absent code"+ },+ {+ "id": "F9",+ "title": "snapshot repo_path/name unvalidated \u2192 write outside the repo",+ "xerj": {+ "candidates": 21,+ "returned": 21,+ "hit": true,+ "rank": 19,+ "triage_tokens": 11474,+ "tokenizer": "tiktoken"+ },+ "grep": {+ "match_lines": 6,+ "match_files": 3,+ "hit": true,+ "triage_tokens": 23680,+ "tokenizer": "tiktoken"+ },+ "grep_pattern": "\\.join\\(name\\)"+ }+ ]+}diff --git a/landing/case-studies/xerj-self-audit.html b/landing/case-studies/xerj-self-audit.htmlindex c34a2ad72..eecd468c7 100644--- a/landing/case-studies/xerj-self-audit.html+++ b/landing/case-studies/xerj-self-audit.html@@ -116,14 +116,52 @@</div></div>-<!-- Ground-truth recall -->+<!-- Detection quality --><section class="scene-section">- <h2 class="scene" style="font-size:var(--fs-56);">DOES IT ACTUALLY <span class="accent">FIND BUGS?</span></h2>- <p class="wide">Before trusting the substrate on unknown bugs, we tested it on known ones. The day before, a PR fixed six real vulnerabilities in this exact engine. We indexed the <em>pre-fix</em> commit and re-ran the queries on both — the signal has to flip when the bug is fixed, or it is noise.</p>- <div class="laws">- <div class="n">F1</div><div class="law"><div class="t">QUERY-PARSER STACK OVERFLOW</div><div class="b">The recursion-cycle query flags the query_string parser cycle as <strong>UNGUARDED pre-fix</strong> and the same cycle as <strong>GUARDED</strong> on current main. The signal flips exactly on the fix.</div></div>- <div class="n">F7</div><div class="law"><div class="t">FIELD-LIMIT BYPASS ON INGEST</div><div class="b"><code class="inline">evolve_schema_*</code> shows <code class="inline">reads_config_limit: false</code> pre-fix and <code class="inline">true</code> after. The substrate distinguishes the vulnerable version from the patched one.</div></div>+ <h2 class="scene" style="font-size:var(--fs-56);">DETECTION QUALITY, <span class="accent">MEASURED.</span></h2>+ <p class="wide">Before trusting a substrate on unknown bugs, test it on known ones. The day before, a PR fixed six real network-reachable vulnerabilities in this exact engine — a labelled test set. We indexed the <em>pre-fix</em> commit and, for each bug, ran the query an auditor would write for that bug <em>class</em> (never its name or line), against the baseline they would otherwise use: grep.</p>++ <div class="viz-block" style="margin-top:var(--sp-6);">+ <div class="kicker"><span>MEASURED · 6 KNOWN BUGS · PRE-FIX TREE</span></div>+ <div class="cmp-bars">+ <div class="cmp-bars-head">+ <span></span>+ <span class="cmp-a">XERJ SUBSTRATE</span>+ <span class="cmp-b">GREP</span>+ </div>+ <div class="cmp-bars-row">+ <span class="cmp-label">Bugs found (recall)</span>+ <span class="cmp-val cmp-a">+ <span class="cmp-bar" style="width:90.0%;"></span>+ <span class="cmp-num">6 / 6</span>+ </span>+ <span class="cmp-val cmp-b">+ <span class="cmp-bar" style="width:60.0%;"></span>+ <span class="cmp-num">4 / 6</span>+ </span>+ </div>+ <div class="cmp-bars-row">+ <span class="cmp-label">Tokens to triage candidates</span>+ <span class="cmp-val cmp-a">+ <span class="cmp-bar" style="width:2.8%;"></span>+ <span class="cmp-num">84,122</span>+ </span>+ <span class="cmp-val cmp-b">+ <span class="cmp-bar" style="width:90.0%;"></span>+ <span class="cmp-num">2,715,003 <span class="cmp-ratio">32×</span></span>+ </span>+ </div>+ </div>+ <p style="margin-top:14px; font-size:12px; letter-spacing:.04em; color:var(--z-mute); font-family:var(--font-data);">A GREP HIT IS A FILE; AN AST HIT IS A FUNCTION · RECALL 6/6 IS IN-SAMPLE — SEE THE CAVEAT BELOW · FULL PER-BUG TABLE IN <a href="https://github.com/xerj-org/xerj/blob/main/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md" rel="external">DETECTION-QUALITY.md →</a></p>+ </div>++ <div class="laws" style="margin-top:var(--sp-6);">+ <div class="n">01</div><div class="law"><div class="t">GREP CANNOT FIND <span class="accent">MISSING CODE.</span></div><div class="b">Two of the six bugs are an <em>absent</em> limit check and an <em>absent</em> action cap. You cannot grep for code that is not there — <code class="inline">grep max_actions_per_bulk</code> on the pre-fix tree returns 0 lines. "Ingest function that never consults a configured limit" is an AST property, not a string.</div></div>+ <div class="n">02</div><div class="law"><div class="t">THE FIRST RUN SCORED <span class="accent">3/6 — WORSE THAN GREP.</span></div><div class="b">Publishing only the tuned number would misrepresent the method. The test set exposed three real extractor defects: argument parsing truncated <code class="inline">a.len() * b.len()</code> at the first paren, losing the multiplication; taint provenance did not follow one hop into locals; validator detection was presence-only, scoring decorative validation that runs <em>after</em> the delete as "guarded". Fixing those reached 6/6 — so 6/6 is in-sample.</div></div>+ <div class="n">03</div><div class="law"><div class="t">THE SIGNALS <span class="accent">GO QUIET WHEN FIXED.</span></div><div class="b">Each signal was checked to discriminate, not merely match: the query-parser cycle reads UNGUARDED pre-fix and GUARDED after; <code class="inline">guard_after_destructive_op</code> is true for pre-fix <code class="inline">restore_snapshot</code> and false after. A detector that fires on patched code is noise.</div></div>+ <div class="n">04</div><div class="law"><div class="t">PRECISION IS THE <span class="accent">WEAK AXIS.</span></div><div class="b">The true positive lands at rank 6 of 186, 12 of 13, 18 of 27, 19 of 21. This is a strong <em>filter</em> and a mediocre <em>ranker</em>. It narrows 207k lines to a few dozen functions you can read in full — it does not point at the bug.</div></div></div>+ <p class="wide" style="margin-top:var(--sp-5);">The genuinely out-of-sample result is the finding below: not in the test set, found on current <code class="inline">main</code>, and proven by aborting a live server.</p></section><section class="scene-section">@@ -164,7 +202,7 @@ $ python3 ingest.py ./ast-out --url $XERJ <span class="comment"># s<div class="laws"><div class="n">01</div><div class="law"><div class="t">XERJ DOES NOT <span class="accent">FIND BUGS.</span></div><div class="b">The tree-sitter extractor and the call-graph analysis produce the candidates; XERJ stores and queries them at scale. The substrate narrows the search; a human or agent still reads the survivors.</div></div><div class="n">02</div><div class="law"><div class="t">THE GRAPH IS <span class="accent">NAME-BASED.</span></div><div class="b">Method-call edges are excluded on purpose — resolving <code class="inline">.len()</code> by name collapses the graph into one false component. Cycles through trait dispatch or macros are out of reach; the recursion result covers free functions.</div></div>- <div class="n">03</div><div class="law"><div class="t">EVERY CANDIDATE STILL <span class="accent">NEEDS A READ.</span></div><div class="b">Blunt category filters over-return (87 for one alloc query); argument-provenance signals sharpen them, and every published finding was read and, where possible, executed. Two unproven traversal leads were <em>not</em> published.</div></div>+ <div class="n">03</div><div class="law"><div class="t">EVERY CANDIDATE STILL <span class="accent">NEEDS A READ.</span></div><div class="b">Blunt category filters over-return (87 for one alloc query); argument-provenance signals sharpen them, and every published finding was read and, where possible, executed. A promising highlight-tag allocation blowup was <strong>refuted by testing</strong> — the highlighter truncates to a fragment, so the product is bounded — and two unproven traversal leads were <em>not</em> published.</div></div><div class="n">04</div><div class="law"><div class="t">ONE PASS, HIGHEST-VALUE <span class="accent">LENSES.</span></div><div class="b">This publishes what was proven or fully enumerated — the DoS and the unsafe inventory. The broader auth-flow and business-logic lenses are scoped in the walkthrough, not exhausted here.</div></div></div></section>