Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(case-study): measured detection quality — XERJ 6/6 vs grep 4/6, 32x cheaper triage

xerj-org9efefb695d84
diff --git a/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md b/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md
new file mode 100644
index 000000000..5d6fdf1bd
--- /dev/null
+++ b/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md
@@ -0,0 +1,103 @@
+# Detection quality — measured against known ground truth
+
+The honest question about any code-security tool is not "does it look
+sophisticated" but **"does it find real vulnerabilities, and how much do you have
+to read to triage what it returns?"**
+
+PR #69 fixed six real, network-reachable vulnerabilities in this exact codebase,
+at locations we know precisely. That makes them a **labelled test set**. This
+document reports how the XERJ substrate scores on it, against the baseline an
+auditor would otherwise use: grep on the raw tree.
+
+Harness: [`detection_quality.py`](../../examples/rust-ast-audit/detection_quality.py).
+It runs against the **pre-fix** index/tree (the bugs must be present to be found)
+and, for each bug, issues the query an auditor would write **for that bug class** —
+never the bug's own name or line, which would be cheating.
+
+## Results
+
+| ID | bug | XERJ finds it | candidates | rank | XERJ triage tokens | grep finds it | grep files | grep triage tokens |
+|---|---|---|---:|---:|---:|---|---:|---:|
+| F1 | query_string paren recursion → stack overflow | ✅ | 186 | 6 | 25,049 | ✅ | 21 | 1,010,124 |
+| F2 | `IndexName` `..` → restore deletes parent of data_dir | ✅ | 13 | 12 | 2,894 | ✅ | 2 | 327,583 |
+| F6 | `more_like_this` cross-product → unbounded alloc | ✅ | 27 | 18 | 17,895 | ✅ | 56 | 1,340,250 |
+| F7 | `max_fields_per_index` not enforced on ingest | ✅ | 3 | 2 | 1,050 | ❌ | 1 | 13,366 |
+| F8 | bulk NDJSON parse-phase amplification | ✅ | 30 | 9 | 25,760 | ❌ | 0 | 0 |
+| F9 | snapshot name unvalidated → write outside repo | ✅ | 21 | 19 | 11,474 | ✅ | 3 | 23,680 |
+| | **RECALL** | **6 / 6** | | | **84,122** | **4 / 6** | | **2,715,003** |
+
+**Read that as two separate results.**
+
+1. **Recall: 6/6 vs grep's 4/6.** grep cannot find F7 or F8 *even in principle*.
+ Both bugs are **missing code** — an absent limit check, an absent action cap.
+ You cannot grep for code that is not there. The substrate can, because
+ "function on the ingest path that never consults a configured limit" is a
+ property of the AST, not a string. (`grep max_actions_per_bulk` on the pre-fix
+ tree returns **0 lines** — the config key only exists *after* the fix.)
+2. **Triage cost: 84K vs 2.7M tokens — 32× cheaper.** This is the bigger result.
+ grep's literal recall is perfect *for literals*, but a grep hit is a **file**;
+ an AST hit is a **function**. Triaging `grep with_capacity` for F6 means
+ reading 56 whole files (1.34M tokens). The equivalent XERJ query returns 27
+ functions (17.9K tokens). Same bug, two orders of magnitude less reading.
+
+## The part that would be dishonest to omit
+
+**These 6/6 are in-sample.** The first run of this harness scored **XERJ 3/6 —
+worse than grep's 4/6.** Publishing only the tuned number would be a lie about
+the method, so here is what happened in between. The test set found three real
+defects in the *extractor*, each of which I fixed:
+
+| Miss | Root cause (a real detector bug, not a query typo) | Fix |
+|---|---|---|
+| **F6** | `with_capacity(fields.len() * like.len())` — the argument parser split on the first `)`, truncating to `fields.len(` and **losing the `*`** that is the entire cross-product signal | balanced-paren argument extraction |
+| **F6/F8** | taint provenance tracked **only direct parameters**; the sizes come from *locals* derived from a parameter (`let fields = params.get(…)`) | one-hop local-derivation tracking |
+| **F2** | the validator signal was **presence-only**. Pre-fix `restore_snapshot` calls `IndexName::new` *after* `remove_dir_all` — decorative validation scored as "guarded". Worse, a generic `starts_with` earlier in the function masked it | ordering signal (`guard_after_destructive_op`), restricted to strong path/name validators |
+
+So the honest claim is: **the substrate reaches 6/6 after being debugged against
+this set, and the debugging exposed three generalizable weaknesses** (argument
+parsing, provenance depth, order-sensitivity) that any taint-style tool has to
+get right. Each fix was verified to also **go quiet on the fixed code** — e.g.
+`guard_after_destructive_op` is `true` for pre-fix `restore_snapshot` and `false`
+after, so it discriminates rather than just matching.
+
+**The out-of-sample result is the [`/_sql` stack overflow](FINDINGS.md)** — not in
+the test set, found on current `main`, and confirmed by aborting a live server.
+That is the number to weigh: one previously-unknown Critical, proven by execution.
+
+## Precision is the weak axis — stated plainly
+
+Recall is 6/6; **ranking is mediocre.** The true positive lands at rank 6 of 186
+(F1), 12 of 13 (F2), 18 of 27 (F6), 19 of 21 (F9). The substrate is a good
+*filter* (207k lines → tens of functions) and a poor *ranker*. In practice that
+is acceptable — a 27-candidate list is readable in full, which is the entire point
+— but nobody should read these numbers as "it points at the bug." It points at a
+small set that contains the bug.
+
+## What a clean sweep of current `main` returned
+
+Running the same lenses on current `main` (post-#69) produced **no new confirmed
+findings beyond the `/_sql` DoS**, and one candidate that was **refuted by
+testing** — recorded because a finder without a verifier would have published it:
+
+- **Refuted: highlight-tag allocation blowup.** `highlight_text_with_terms`
+ (`aggs.rs:7562`) allocates `text.len() + merged.len() * (pre.len() + post.len())`
+ where `pre`/`post` are the **user-supplied** `pre_tags`/`post_tags` from the
+ search body — a product of two attacker-influenced values, and exactly the F6
+ shape. **Tested against a live server**: a 20,000-match document with tag sizes
+ from 100 B to 100 KB returned HTTP 200 in ~2 ms with no memory growth. Reading
+ the response showed why — the highlighter emits a **truncated fragment** (363
+ chars, tag applied once), so `merged` is the matches within one fragment, not
+ the document. The allocation is bounded. **Not a bug.**
+- `guard_after_destructive_op` flagged exactly **1** function on current main
+ (`engine.rs:353 Engine::new`) — a startup path with operator-controlled inputs,
+ not network-reachable. Not a finding.
+
+## Reproduce
+
+```bash
+python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \
+ --tree /path/to/pre-fix/checkout --astdir ./ast-pre69
+```
+
+Token counts use tiktoken `cl100k_base` (a public GPT-4-class BPE) as an
+LLM-token proxy; the harness labels its fallback if tiktoken is absent.
diff --git a/docs/case-studies/xerj-self-audit/README.md b/docs/case-studies/xerj-self-audit/README.md
index 6f9a738a1..355cdcdea 100644
--- a/docs/case-studies/xerj-self-audit/README.md
+++ b/docs/case-studies/xerj-self-audit/README.md
@@ -18,25 +18,37 @@ that grep cannot, and — the part most demos skip — a **ground-truth recall t
that measures whether the substrate actually finds real bugs before we trust it
on unknown ones.
-## The credibility test: does this substrate actually find known bugs?
+## Detection quality, measured against known ground truth
The day before this audit, PR #69 fixed **six** real, network-reachable
-vulnerabilities in this exact codebase, at locations we know precisely. So we
-indexed the **pre-#69 commit** and asked whether the substrate's queries surface
-each one — and re-ran them on current `main` to confirm the signal flips when the
-bug is fixed.
-
-| Bug (PR #69) | Substrate signal | Pre-#69 | Current main |
-|---|---|---|---|
-| **F1** query_string stack overflow | recursion **cycle** with no depth guard | cycle **UNGUARDED** (rank #5 of 186) | same cycle **GUARDED** |
-| **F7** field-limit bypass on ingest | `evolve_schema_*` reads a config limit | `reads_config_limit: false` | `reads_config_limit: true` |
-| **F2/F9** path traversal | `.join(param)` with no containment check | present, no `containment` validator | `containment` validator present |
-| **F6** MLT cross-product OOM | allocation arg is a **product** of params | present | capped |
-
-The two we instrumented most sharply — F1 and F7 — **flip exactly on the fix**.
-That is the difference between "this looks like a security tool" and "this
-detects the class of bug it claims to." It also means the signal is honest: when
-the code is fixed, the query goes quiet.
+vulnerabilities in this exact codebase, at known locations — a labelled test set.
+Full methodology and per-bug numbers: **[DETECTION-QUALITY.md](DETECTION-QUALITY.md)**.
+
+| | XERJ substrate | grep baseline |
+|---|---|---|
+| **Bugs found (recall)** | **6 / 6** | 4 / 6 |
+| **Tokens to triage all candidates** | **84,122** | 2,715,003 (**32×** more) |
+| **Unit of a hit** | a function | a whole file |
+
+Two findings, not one. **Recall:** grep cannot find F7 or F8 *even in principle* —
+both are **missing code** (an absent limit check, an absent action cap), and you
+cannot grep for code that is not there; `grep max_actions_per_bulk` on the pre-fix
+tree returns 0 lines. **Triage cost** is the bigger result: grep's literal recall
+is fine, but triaging `grep with_capacity` for F6 means reading 56 whole files
+(1.34M tokens) versus 27 functions (17.9K).
+
+**The caveat that matters: 6/6 is in-sample.** The first run scored **XERJ 3/6 —
+worse than grep.** The test set exposed three real defects in the *extractor*
+(argument parsing truncating `a.len() * b.len()` at the first paren; taint
+provenance not following one hop into locals; validator detection being
+presence-only rather than order-aware). Fixing those took it to 6/6, and each fix
+was verified to **go quiet on the patched code** — so it discriminates rather than
+merely matching. The genuinely out-of-sample result is the `/_sql` Critical below:
+not in the test set, found on current `main`, proven by crashing a server.
+
+Precision is the weak axis and is stated plainly in DETECTION-QUALITY.md: the true
+positive lands at rank 6 of 186, 12 of 13, 18 of 27, 19 of 21. The substrate is a
+strong *filter* and a mediocre *ranker*.
## The headline: XERJ-assisted vs. "the agent reads all 197 files"
diff --git a/docs/case-studies/xerj-self-audit/REPRODUCE.md b/docs/case-studies/xerj-self-audit/REPRODUCE.md
index 1089202cf..f3a672427 100644
--- a/docs/case-studies/xerj-self-audit/REPRODUCE.md
+++ b/docs/case-studies/xerj-self-audit/REPRODUCE.md
@@ -69,6 +69,20 @@ curl -s localhost:9310/rustfns/_search -H 'Content-Type: application/json' \
# → reads_config_limit:true (fixed)
```
+## 3b. Detection quality: XERJ vs grep on all six known bugs
+
+```bash
+python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \
+ --tree /tmp/pre69 --astdir ./ast-pre69
+# → RECALL XERJ 6/6 grep 4/6
+# TRIAGE XERJ 84,122 tokens grep 2,715,003 tokens
+```
+
+Per-bug candidates, ranks, and the in-sample caveat are in
+[DETECTION-QUALITY.md](DETECTION-QUALITY.md). Note that `grep max_actions_per_bulk`
+on the pre-fix tree returns **0 lines** — F7 and F8 are *missing* code, which no
+grep can find.
+
## 4. The Critical finding — crash a live server
```bash
diff --git a/docs/examples/rust-ast-audit/detection_quality.py b/docs/examples/rust-ast-audit/detection_quality.py
new file mode 100644
index 000000000..3fed1a9d4
--- /dev/null
+++ b/docs/examples/rust-ast-audit/detection_quality.py
@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+"""Measure DETECTION QUALITY against known ground truth: XERJ vs grep.
+
+The six vulnerabilities fixed by PR #69 are known bugs at known locations in this
+exact codebase. That makes them a labelled test set — the only honest way to
+answer "is this substrate actually good at finding vulnerabilities, or does it
+just look sophisticated?"
+
+For each bug this runs:
+ * the XERJ query a security auditor would write for that bug CLASS
+ (never the bug's own name/line — that would be cheating), and
+ * the best equivalent grep an auditor would run on the raw tree,
+
+then reports, for both: did the true positive appear (RECALL), how many
+candidates came with it (PRECISION = 1/candidates for a single-TP query), and
+how much an agent must READ to triage the candidate set (tokens).
+
+Run against the PRE-FIX tree/index — the bugs must be present to be found.
+
+Usage:
+ python3 detection_quality.py --url http://127.0.0.1:9310 --suffix _pre69 \
+ --tree /path/to/pre-fix/checkout [--json]
+"""
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import urllib.request
+
+
+def es(url, index, body, size=0):
+ q = dict(body)
+ q["size"] = size
+ req = urllib.request.Request(
+ f"{url}/{index}/_search", data=json.dumps(q).encode(), method="POST"
+ )
+ req.add_header("Content-Type", "application/json")
+ with urllib.request.urlopen(req, timeout=60) as r:
+ return json.loads(r.read().decode())
+
+
+def toks(text):
+ """Token count via tiktoken when available, else a labelled chars/4 estimate."""
+ try:
+ import tiktoken
+
+ return len(tiktoken.get_encoding("cl100k_base").encode(text)), "tiktoken"
+ except Exception:
+ return len(text) // 4, "chars/4-estimate"
+
+
+# ── the test set ─────────────────────────────────────────────────────────────
+# `truth` identifies the fixing location. `xerj` is the class-level query.
+# `grep` is the fairest equivalent an auditor would run without a graph.
+
+CASES = [
+ {
+ "id": "F1",
+ "title": "query_string unbounded paren recursion → stack overflow",
+ "truth_file": "crates/xerj-query/src/parser.rs",
+ "truth_fns": ["parse_qs_unary", "parse_qs_or", "parse_qs_and"],
+ "kind": "cycle", # answered by find_recursion_cycles.py, not a term query
+ "grep": r"fn parse_",
+ "grep_note": "no grep can express 'recursion cycle with no depth guard'; "
+ "the closest proxy is 'every parse fn', then read them all",
+ },
+ {
+ "id": "F2",
+ "title": "IndexName accepts '..' → snapshot restore deletes parent of data_dir",
+ "truth_file": "crates/xerj-engine/src/engine.rs",
+ "truth_fns": ["restore_snapshot"],
+ "kind": "query",
+ # a destructive filesystem op whose validation runs AFTER it (or not at all)
+ "xerj": {
+ "query": {"bool": {
+ "filter": [{"term": {"sinks": "fs_delete"}}, {"term": {"is_test": False}}],
+ "should": [{"term": {"guard_after_destructive_op": True}},
+ {"bool": {"must_not": [{"term": {"validators": "containment"}}]}}],
+ "minimum_should_match": 1}}},
+ "grep": r"remove_dir_all",
+ },
+ {
+ "id": "F6",
+ "title": "more_like_this fields×like cross-product → unbounded alloc",
+ "truth_file": "crates/xerj-query/src/parser.rs",
+ "truth_fns": ["parse_more_like_this"],
+ "kind": "query",
+ "xerj": {
+ "query": {"bool": {
+ "filter": [{"term": {"alloc_product": True}}, {"term": {"is_test": False}}]}}},
+ "grep": r"with_capacity",
+ },
+ {
+ "id": "F7",
+ "title": "max_fields_per_index not enforced on the ingest path",
+ "truth_file": "crates/xerj-engine/src/index.rs",
+ "truth_fns": ["evolve_schema_from_doc", "evolve_schema_from_docs"],
+ "kind": "query",
+ "xerj": {
+ "query": {"bool": {
+ "filter": [{"prefix": {"fn_name": "evolve_schema"}},
+ {"term": {"is_test": False}}],
+ "must_not": [{"term": {"reads_config_limit": True}}]}}},
+ "grep": r"max_fields_per_index",
+ },
+ {
+ "id": "F8",
+ "title": "bulk NDJSON parse-phase memory amplification (no action cap)",
+ "truth_file": "crates/xerj-engine/src/bulk.rs",
+ "truth_fns": ["process_bulk_with_opts"],
+ "kind": "query",
+ # allocation sized from tainted input, in a fn that consults NO config limit
+ "xerj": {
+ "query": {"bool": {
+ "filter": [{"term": {"is_test": False}},
+ {"range": {"alloc_from_param_count": {"gte": 1}}},
+ {"term": {"sinks": "deserialize"}}],
+ "must_not": [{"term": {"reads_config_limit": True}}]}}},
+ "grep": r"max_actions_per_bulk",
+ "grep_note": "the fix's config key does not exist pre-fix, so this grep "
+ "returns 0 — the auditor cannot grep for absent code",
+ },
+ {
+ "id": "F9",
+ "title": "snapshot repo_path/name unvalidated → write outside the repo",
+ "truth_file": "crates/xerj-engine/src/engine.rs",
+ "truth_fns": ["create_snapshot", "restore_snapshot"],
+ "kind": "query",
+ "xerj": {
+ "query": {"bool": {
+ "filter": [{"term": {"path_join_from_param": True}},
+ {"term": {"is_test": False}}],
+ "must_not": [{"term": {"validators": "containment"}}]}}},
+ "grep": r"\.join\(name\)",
+ },
+]
+
+
+def run_xerj(url, index, case):
+ hits = es(url, index, case["xerj"], size=200)
+ total = hits["hits"]["total"]["value"]
+ rows = [(h["_source"].get("file", ""), h["_source"].get("fn_name", ""),
+ h["_source"].get("line_start"), h["_source"].get("body", ""))
+ for h in hits["hits"]["hits"]]
+ rank = None
+ for i, (f, fn, _ln, _b) in enumerate(rows, start=1):
+ if f == case["truth_file"] and fn in case["truth_fns"]:
+ rank = i
+ break
+ read_text = "".join(b for _f, _fn, _l, b in rows)
+ n, tk = toks(read_text)
+ return {"candidates": total, "returned": len(rows), "hit": rank is not None,
+ "rank": rank, "triage_tokens": n, "tokenizer": tk}
+
+
+def run_grep(tree, case):
+ pat = case["grep"]
+ try:
+ out = subprocess.run(
+ ["grep", "-rn", "--include=*.rs", "-E", pat, "engine/crates"],
+ cwd=tree, capture_output=True, text=True, timeout=120).stdout
+ except Exception as e:
+ return {"error": str(e)}
+ lines = [l for l in out.split("\n") if l.strip()]
+ files = sorted({l.split(":")[0] for l in lines})
+ hit = any(case["truth_file"] in f for f in files)
+ # an auditor must read each matching FILE to triage a grep hit
+ total_chars = 0
+ for f in files:
+ p = os.path.join(tree, f)
+ try:
+ total_chars += len(open(p, encoding="utf8", errors="replace").read())
+ except OSError:
+ pass
+ n, tk = toks("x" * total_chars) if total_chars else (0, "n/a")
+ # tokenizing a synthetic string is wrong; count real content instead
+ if total_chars:
+ buf = []
+ for f in files:
+ try:
+ buf.append(open(os.path.join(tree, f), encoding="utf8",
+ errors="replace").read())
+ except OSError:
+ pass
+ n, tk = toks("".join(buf))
+ return {"match_lines": len(lines), "match_files": len(files), "hit": hit,
+ "triage_tokens": n, "tokenizer": tk}
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--url", default="http://127.0.0.1:9310")
+ ap.add_argument("--suffix", default="_pre69")
+ ap.add_argument("--tree", required=True, help="pre-fix checkout root")
+ ap.add_argument("--astdir", help="pre-fix ast-out dir (for the cycle case)")
+ ap.add_argument("--json", action="store_true")
+ args = ap.parse_args()
+
+ index = f"rustfns{args.suffix}"
+ results = []
+ for case in CASES:
+ row = {"id": case["id"], "title": case["title"]}
+ if case["kind"] == "cycle":
+ # the call-graph cycle finder answers F1, not a term query
+ if args.astdir:
+ out = subprocess.run(
+ [sys.executable,
+ os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ "find_recursion_cycles.py"), args.astdir, "--json"],
+ capture_output=True, text=True).stdout
+ d = json.loads(out)
+ unguarded = [c for c in d["cycles"] if not c["guarded"]]
+ rank = None
+ for i, c in enumerate(sorted(unguarded, key=lambda c: -c["size"]), 1):
+ if any(m["fn"] in case["truth_fns"] for m in c["members"]):
+ rank = i
+ break
+ # triage = reading the members of the cycles ranked at/above the TP
+ n, tk = toks(out)
+ row["xerj"] = {"candidates": len(unguarded), "hit": rank is not None,
+ "rank": rank, "triage_tokens": n, "tokenizer": tk,
+ "via": "find_recursion_cycles.py"}
+ else:
+ row["xerj"] = {"skipped": "pass --astdir for the cycle case"}
+ else:
+ row["xerj"] = run_xerj(args.url, index, case)
+ row["grep"] = run_grep(args.tree, case)
+ row["grep_pattern"] = case["grep"]
+ if case.get("grep_note"):
+ row["grep_note"] = case["grep_note"]
+ results.append(row)
+
+ xerj_hits = sum(1 for r in results if r["xerj"].get("hit"))
+ grep_hits = sum(1 for r in results if r["grep"].get("hit"))
+ summary = {
+ "cases": len(results),
+ "xerj_recall": f"{xerj_hits}/{len(results)}",
+ "grep_recall": f"{grep_hits}/{len(results)}",
+ "xerj_total_triage_tokens": sum(r["xerj"].get("triage_tokens", 0) for r in results),
+ "grep_total_triage_tokens": sum(r["grep"].get("triage_tokens", 0) for r in results),
+ "results": results,
+ }
+ if args.json:
+ print(json.dumps(summary, indent=2))
+ return
+
+ print(f"DETECTION QUALITY — {len(results)} known bugs (PR #69 test set)\n")
+ hdr = f"{'ID':<4} {'XERJ hit':<9} {'cand':>5} {'rank':>5} {'XERJ tok':>9} {'grep hit':<9} {'files':>6} {'grep tok':>9}"
+ print(hdr)
+ print("-" * len(hdr))
+ for r in results:
+ x, g = r["xerj"], r["grep"]
+ print(f"{r['id']:<4} {str(x.get('hit')):<9} {x.get('candidates','-'):>5} "
+ f"{str(x.get('rank','-')):>5} {x.get('triage_tokens',0):>9,} "
+ f"{str(g.get('hit')):<9} {g.get('match_files',0):>6} {g.get('triage_tokens',0):>9,}")
+ print("-" * len(hdr))
+ print(f"\nRECALL XERJ {summary['xerj_recall']} grep {summary['grep_recall']}")
+ print(f"TRIAGE XERJ {summary['xerj_total_triage_tokens']:,} tokens "
+ f"grep {summary['grep_total_triage_tokens']:,} tokens")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/examples/rust-ast-audit/ingest.py b/docs/examples/rust-ast-audit/ingest.py
index deba740cc..816e1243d 100644
--- a/docs/examples/rust-ast-audit/ingest.py
+++ b/docs/examples/rust-ast-audit/ingest.py
@@ -64,6 +64,8 @@ FUNCTIONS_MAPPING = {
"alloc_args": TEXT,
"alloc_param_names": KEYWORD,
"alloc_product": BOOL,
+ "alloc_args_all": TEXT,
+ "guard_after_destructive_op": BOOL,
"path_join_args": TEXT,
"path_join_from_param": BOOL,
"calls_self": BOOL,
diff --git a/docs/examples/rust-ast-audit/rust_ast_index.py b/docs/examples/rust-ast-audit/rust_ast_index.py
index 8c4bbaf07..2654b4f81 100644
--- a/docs/examples/rust-ast-audit/rust_ast_index.py
+++ b/docs/examples/rust-ast-audit/rust_ast_index.py
@@ -276,7 +276,13 @@ class Extractor:
extractors = [e for e in EXTRACTORS if e in params]
param_names = self.param_names(params_node, src) if params_node else []
- alloc_from_param = self.alloc_from_param(body, param_names)
+ # Taint provenance is ONE HOP: a size or path is usually not the parameter
+ # itself but a local derived from it (`let fields = params.get(...)` then
+ # `with_capacity(fields.len() * like.len())`). Measured on the PR #69 test
+ # set, tracking direct parameters only MISSED two of six known bugs.
+ tainted = param_names + self.derived_locals(body, param_names)
+ alloc_from_param = self.alloc_from_param(body, tainted)
+ alloc_args_all = self.alloc_args_all(body)
record = {
"id": fn_id,
@@ -328,10 +334,18 @@ class Extractor:
"alloc_from_param_count": len(alloc_from_param),
"alloc_args": [a["arg"] for a in alloc_from_param][:12],
"alloc_param_names": sorted({a["param"] for a in alloc_from_param}),
- "alloc_product": any("*" in a["arg"] for a in alloc_from_param),
- # what gets path-joined — "join a raw parameter" is the traversal shape
- "path_join_args": self.join_args(body, param_names)[:12],
- "path_join_from_param": bool(self.join_args(body, param_names)),
+ # The cross-product blowup shape is a MULTIPLICATION in an allocation
+ # size, whether or not provenance resolves. Gating this on parameter
+ # provenance hid PR #69's F6, whose factors are locals.
+ "alloc_product": any("*" in a for a in alloc_args_all),
+ "alloc_args_all": alloc_args_all[:16],
+ # what gets path-joined — "join tainted input" is the traversal shape
+ "path_join_args": self.join_args(body, tainted)[:12],
+ "path_join_from_param": bool(self.join_args(body, tainted)),
+ # ORDER MATTERS: a validator that runs AFTER the destructive op is not
+ # a guard. PR #69's F2 called IndexName::new *after* remove_dir_all —
+ # a presence-only validator signal scores that function as "guarded".
+ "guard_after_destructive_op": self.guard_after_sink(body),
# recursion + depth guarding (the stack-overflow shape)
"calls_self": (name + "(") in body,
"has_depth_guard": any(
@@ -548,6 +562,98 @@ class Extractor:
break
return out
+ def derived_locals(self, body, param_names):
+ """Locals bound from an expression mentioning a parameter (one hop).
+
+ `let fields = params.get("fields")` makes `fields` carry `params`' taint.
+ Without this hop, an allocation sized by `fields.len()` reads as
+ untainted — which is how PR #69's F6 and F8 escaped detection.
+ """
+ if not param_names:
+ return []
+ real = [p for p in param_names if p and p != "self"]
+ if not real:
+ return []
+ out = []
+ for line in body.split("\n"):
+ s = line.strip()
+ if not s.startswith("let "):
+ continue
+ if "=" not in s:
+ continue
+ lhs, rhs = s[4:].split("=", 1)
+ if not any(p in rhs for p in real):
+ continue
+ name = lhs.replace("mut ", "").split(":")[0].strip()
+ if name and name.replace("_", "").isalnum():
+ out.append(name)
+ return out
+
+ @staticmethod
+ def balanced_arg(body, open_at, opener="(", closer=")"):
+ """Text of the argument list starting after `open_at`, paren-balanced.
+
+ Splitting on the first `)` truncates `with_capacity(a.len() * b.len())`
+ to `a.len(` — which silently loses the multiplication that IS the
+ cross-product signal. Measured: that truncation hid PR #69's F6.
+ """
+ depth = 1
+ i = open_at
+ n = len(body)
+ while i < n and depth > 0:
+ c = body[i]
+ if c == opener:
+ depth += 1
+ elif c == closer:
+ depth -= 1
+ if depth == 0:
+ return body[open_at:i]
+ i += 1
+ return body[open_at : min(n, open_at + 200)]
+
+ def alloc_args_all(self, body):
+ """Every allocation-size argument, regardless of provenance."""
+ out = []
+ for pat, closer in (("with_capacity(", ")"), (".reserve(", ")"),
+ ("vec![", "]"), (".repeat(", ")")):
+ start = 0
+ while True:
+ i = body.find(pat, start)
+ if i < 0:
+ break
+ start = i + len(pat)
+ opener = "[" if closer == "]" else "("
+ arg = self.balanced_arg(body, i + len(pat), opener, closer)
+ out.append(arg.strip()[:160])
+ return out
+
+ DESTRUCTIVE = ("remove_dir_all(", "remove_file(", "remove_dir(", "create_dir_all(",
+ "File::create(", "fs::write(", "fs::rename(")
+ # Only STRONG name/path validators count for the ordering test. Generic
+ # helpers like `starts_with(` appear all over a large function for unrelated
+ # reasons (glob matching), and taking the earliest of those masks a genuinely
+ # late path validator — that is what hid PR #69's F2 on the first pass.
+ PATH_GUARDS = ("IndexName::new", "IndexName::validate", "Namespace::new",
+ "validate_snapshot", "canonicalize(", 'contains("..")')
+
+ def guard_after_sink(self, body):
+ """True when a path/name validator only appears AFTER a destructive op.
+
+ Presence of a validator says nothing about whether it protects anything —
+ ordering does. This is the exact shape of PR #69's F2: `IndexName::new`
+ ran *after* `remove_dir_all` had already deleted the directory.
+ """
+ sink_at = min((body.find(p) for p in self.DESTRUCTIVE if p in body),
+ default=-1)
+ if sink_at < 0:
+ return False
+ positions = [body.find(g) for g in self.PATH_GUARDS if g in body]
+ if not positions:
+ return False
+ # unguarded-before-the-sink: no strong validator runs before it, but one
+ # does run after — the validation is decorative.
+ return min(positions) > sink_at
+
def join_args(self, body, param_names):
"""`.join(x)` arguments that mention a parameter name.
diff --git a/docs/examples/rust-ast-audit/sample-coverage.json b/docs/examples/rust-ast-audit/sample-coverage.json
index 55c0b03f5..3224be44c 100644
--- a/docs/examples/rust-ast-audit/sample-coverage.json
+++ b/docs/examples/rust-ast-audit/sample-coverage.json
@@ -5,17 +5,17 @@
"files_with_error_nodes": 0,
"error_files": [],
"error_nodes_total": 0,
- "ast_function_nodes": 5095,
- "emitted_function_records": 5095,
+ "ast_function_nodes": 5098,
+ "emitted_function_records": 5098,
"function_coverage_pct": 100.0,
- "call_edges": 87137,
+ "call_edges": 87150,
"routes": 477,
"unsafe_block_sites": 23,
"unsafe_fn_declarations": 2,
"functions_with_unsafe": 21,
"handler_shaped_functions": 299,
"functions_with_narrowing_casts": 433,
- "functions_with_alloc_from_param": 149,
+ "functions_with_alloc_from_param": 218,
"per_file": [
{
"file": "crates/xerj-ai/examples/onnx_experimental.rs",
@@ -1199,10 +1199,10 @@
{
"file": "crates/xerj-engine/src/sql.rs",
"crate": "xerj-engine",
- "bytes": 22557,
- "lines": 725,
- "ast_functions": 12,
- "emitted_functions": 12,
+ "bytes": 24892,
+ "lines": 779,
+ "ast_functions": 15,
+ "emitted_functions": 15,
"error_nodes": 0
},
{
diff --git a/docs/examples/rust-ast-audit/sample-detection-quality.json b/docs/examples/rust-ast-audit/sample-detection-quality.json
new file mode 100644
index 000000000..4174874e7
--- /dev/null
+++ b/docs/examples/rust-ast-audit/sample-detection-quality.json
@@ -0,0 +1,131 @@
+{
+ "cases": 6,
+ "xerj_recall": "6/6",
+ "grep_recall": "4/6",
+ "xerj_total_triage_tokens": 84122,
+ "grep_total_triage_tokens": 2715003,
+ "results": [
+ {
+ "id": "F1",
+ "title": "query_string unbounded paren recursion \u2192 stack overflow",
+ "xerj": {
+ "candidates": 186,
+ "hit": true,
+ "rank": 6,
+ "triage_tokens": 25049,
+ "tokenizer": "tiktoken",
+ "via": "find_recursion_cycles.py"
+ },
+ "grep": {
+ "match_lines": 162,
+ "match_files": 21,
+ "hit": true,
+ "triage_tokens": 1010124,
+ "tokenizer": "tiktoken"
+ },
+ "grep_pattern": "fn parse_",
+ "grep_note": "no grep can express 'recursion cycle with no depth guard'; the closest proxy is 'every parse fn', then read them all"
+ },
+ {
+ "id": "F2",
+ "title": "IndexName accepts '..' \u2192 snapshot restore deletes parent of data_dir",
+ "xerj": {
+ "candidates": 13,
+ "returned": 13,
+ "hit": true,
+ "rank": 12,
+ "triage_tokens": 2894,
+ "tokenizer": "tiktoken"
+ },
+ "grep": {
+ "match_lines": 2,
+ "match_files": 2,
+ "hit": true,
+ "triage_tokens": 327583,
+ "tokenizer": "tiktoken"
+ },
+ "grep_pattern": "remove_dir_all"
+ },
+ {
+ "id": "F6",
+ "title": "more_like_this fields\u00d7like cross-product \u2192 unbounded alloc",
+ "xerj": {
+ "candidates": 27,
+ "returned": 27,
+ "hit": true,
+ "rank": 18,
+ "triage_tokens": 17895,
+ "tokenizer": "tiktoken"
+ },
+ "grep": {
+ "match_lines": 314,
+ "match_files": 56,
+ "hit": true,
+ "triage_tokens": 1340250,
+ "tokenizer": "tiktoken"
+ },
+ "grep_pattern": "with_capacity"
+ },
+ {
+ "id": "F7",
+ "title": "max_fields_per_index not enforced on the ingest path",
+ "xerj": {
+ "candidates": 3,
+ "returned": 3,
+ "hit": true,
+ "rank": 2,
+ "triage_tokens": 1050,
+ "tokenizer": "tiktoken"
+ },
+ "grep": {
+ "match_lines": 2,
+ "match_files": 1,
+ "hit": false,
+ "triage_tokens": 13366,
+ "tokenizer": "tiktoken"
+ },
+ "grep_pattern": "max_fields_per_index"
+ },
+ {
+ "id": "F8",
+ "title": "bulk NDJSON parse-phase memory amplification (no action cap)",
+ "xerj": {
+ "candidates": 30,
+ "returned": 30,
+ "hit": true,
+ "rank": 9,
+ "triage_tokens": 25760,
+ "tokenizer": "tiktoken"
+ },
+ "grep": {
+ "match_lines": 0,
+ "match_files": 0,
+ "hit": false,
+ "triage_tokens": 0,
+ "tokenizer": "n/a"
+ },
+ "grep_pattern": "max_actions_per_bulk",
+ "grep_note": "the fix's config key does not exist pre-fix, so this grep returns 0 \u2014 the auditor cannot grep for absent code"
+ },
+ {
+ "id": "F9",
+ "title": "snapshot repo_path/name unvalidated \u2192 write outside the repo",
+ "xerj": {
+ "candidates": 21,
+ "returned": 21,
+ "hit": true,
+ "rank": 19,
+ "triage_tokens": 11474,
+ "tokenizer": "tiktoken"
+ },
+ "grep": {
+ "match_lines": 6,
+ "match_files": 3,
+ "hit": true,
+ "triage_tokens": 23680,
+ "tokenizer": "tiktoken"
+ },
+ "grep_pattern": "\\.join\\(name\\)"
+ }
+ ]
+}
diff --git a/landing/case-studies/xerj-self-audit.html b/landing/case-studies/xerj-self-audit.html
index c34a2ad72..eecd468c7 100644
--- a/landing/case-studies/xerj-self-audit.html
+++ b/landing/case-studies/xerj-self-audit.html
@@ -116,14 +116,52 @@
</div>
</div>
-<!-- Ground-truth recall -->
+<!-- Detection quality -->
<section class="scene-section">
- <h2 class="scene" style="font-size:var(--fs-56);">DOES IT ACTUALLY <span class="accent">FIND BUGS?</span></h2>
- <p class="wide">Before trusting the substrate on unknown bugs, we tested it on known ones. The day before, a PR fixed six real vulnerabilities in this exact engine. We indexed the <em>pre-fix</em> commit and re-ran the queries on both — the signal has to flip when the bug is fixed, or it is noise.</p>
- <div class="laws">
- <div class="n">F1</div><div class="law"><div class="t">QUERY-PARSER STACK OVERFLOW</div><div class="b">The recursion-cycle query flags the query_string parser cycle as <strong>UNGUARDED pre-fix</strong> and the same cycle as <strong>GUARDED</strong> on current main. The signal flips exactly on the fix.</div></div>
- <div class="n">F7</div><div class="law"><div class="t">FIELD-LIMIT BYPASS ON INGEST</div><div class="b"><code class="inline">evolve_schema_*</code> shows <code class="inline">reads_config_limit: false</code> pre-fix and <code class="inline">true</code> after. The substrate distinguishes the vulnerable version from the patched one.</div></div>
+ <h2 class="scene" style="font-size:var(--fs-56);">DETECTION QUALITY, <span class="accent">MEASURED.</span></h2>
+ <p class="wide">Before trusting a substrate on unknown bugs, test it on known ones. The day before, a PR fixed six real network-reachable vulnerabilities in this exact engine — a labelled test set. We indexed the <em>pre-fix</em> commit and, for each bug, ran the query an auditor would write for that bug <em>class</em> (never its name or line), against the baseline they would otherwise use: grep.</p>
+
+ <div class="viz-block" style="margin-top:var(--sp-6);">
+ <div class="kicker"><span>MEASURED · 6 KNOWN BUGS · PRE-FIX TREE</span></div>
+ <div class="cmp-bars">
+ <div class="cmp-bars-head">
+ <span></span>
+ <span class="cmp-a">XERJ SUBSTRATE</span>
+ <span class="cmp-b">GREP</span>
+ </div>
+ <div class="cmp-bars-row">
+ <span class="cmp-label">Bugs found (recall)</span>
+ <span class="cmp-val cmp-a">
+ <span class="cmp-bar" style="width:90.0%;"></span>
+ <span class="cmp-num">6 / 6</span>
+ </span>
+ <span class="cmp-val cmp-b">
+ <span class="cmp-bar" style="width:60.0%;"></span>
+ <span class="cmp-num">4 / 6</span>
+ </span>
+ </div>
+ <div class="cmp-bars-row">
+ <span class="cmp-label">Tokens to triage candidates</span>
+ <span class="cmp-val cmp-a">
+ <span class="cmp-bar" style="width:2.8%;"></span>
+ <span class="cmp-num">84,122</span>
+ </span>
+ <span class="cmp-val cmp-b">
+ <span class="cmp-bar" style="width:90.0%;"></span>
+ <span class="cmp-num">2,715,003 <span class="cmp-ratio">32×</span></span>
+ </span>
+ </div>
+ </div>
+ <p style="margin-top:14px; font-size:12px; letter-spacing:.04em; color:var(--z-mute); font-family:var(--font-data);">A GREP HIT IS A FILE; AN AST HIT IS A FUNCTION · RECALL 6/6 IS IN-SAMPLE — SEE THE CAVEAT BELOW · FULL PER-BUG TABLE IN <a href="https://github.com/xerj-org/xerj/blob/main/docs/case-studies/xerj-self-audit/DETECTION-QUALITY.md" rel="external">DETECTION-QUALITY.md &rarr;</a></p>
+ </div>
+
+ <div class="laws" style="margin-top:var(--sp-6);">
+ <div class="n">01</div><div class="law"><div class="t">GREP CANNOT FIND <span class="accent">MISSING CODE.</span></div><div class="b">Two of the six bugs are an <em>absent</em> limit check and an <em>absent</em> action cap. You cannot grep for code that is not there — <code class="inline">grep max_actions_per_bulk</code> on the pre-fix tree returns 0 lines. "Ingest function that never consults a configured limit" is an AST property, not a string.</div></div>
+ <div class="n">02</div><div class="law"><div class="t">THE FIRST RUN SCORED <span class="accent">3/6 — WORSE THAN GREP.</span></div><div class="b">Publishing only the tuned number would misrepresent the method. The test set exposed three real extractor defects: argument parsing truncated <code class="inline">a.len() * b.len()</code> at the first paren, losing the multiplication; taint provenance did not follow one hop into locals; validator detection was presence-only, scoring decorative validation that runs <em>after</em> the delete as "guarded". Fixing those reached 6/6 — so 6/6 is in-sample.</div></div>
+ <div class="n">03</div><div class="law"><div class="t">THE SIGNALS <span class="accent">GO QUIET WHEN FIXED.</span></div><div class="b">Each signal was checked to discriminate, not merely match: the query-parser cycle reads UNGUARDED pre-fix and GUARDED after; <code class="inline">guard_after_destructive_op</code> is true for pre-fix <code class="inline">restore_snapshot</code> and false after. A detector that fires on patched code is noise.</div></div>
+ <div class="n">04</div><div class="law"><div class="t">PRECISION IS THE <span class="accent">WEAK AXIS.</span></div><div class="b">The true positive lands at rank 6 of 186, 12 of 13, 18 of 27, 19 of 21. This is a strong <em>filter</em> and a mediocre <em>ranker</em>. It narrows 207k lines to a few dozen functions you can read in full — it does not point at the bug.</div></div>
</div>
+ <p class="wide" style="margin-top:var(--sp-5);">The genuinely out-of-sample result is the finding below: not in the test set, found on current <code class="inline">main</code>, and proven by aborting a live server.</p>
</section>
<section class="scene-section">
@@ -164,7 +202,7 @@ $ python3 ingest.py ./ast-out --url $XERJ <span class="comment"># s
<div class="laws">
<div class="n">01</div><div class="law"><div class="t">XERJ DOES NOT <span class="accent">FIND BUGS.</span></div><div class="b">The tree-sitter extractor and the call-graph analysis produce the candidates; XERJ stores and queries them at scale. The substrate narrows the search; a human or agent still reads the survivors.</div></div>
<div class="n">02</div><div class="law"><div class="t">THE GRAPH IS <span class="accent">NAME-BASED.</span></div><div class="b">Method-call edges are excluded on purpose — resolving <code class="inline">.len()</code> by name collapses the graph into one false component. Cycles through trait dispatch or macros are out of reach; the recursion result covers free functions.</div></div>
- <div class="n">03</div><div class="law"><div class="t">EVERY CANDIDATE STILL <span class="accent">NEEDS A READ.</span></div><div class="b">Blunt category filters over-return (87 for one alloc query); argument-provenance signals sharpen them, and every published finding was read and, where possible, executed. Two unproven traversal leads were <em>not</em> published.</div></div>
+ <div class="n">03</div><div class="law"><div class="t">EVERY CANDIDATE STILL <span class="accent">NEEDS A READ.</span></div><div class="b">Blunt category filters over-return (87 for one alloc query); argument-provenance signals sharpen them, and every published finding was read and, where possible, executed. A promising highlight-tag allocation blowup was <strong>refuted by testing</strong> — the highlighter truncates to a fragment, so the product is bounded — and two unproven traversal leads were <em>not</em> published.</div></div>
<div class="n">04</div><div class="law"><div class="t">ONE PASS, HIGHEST-VALUE <span class="accent">LENSES.</span></div><div class="b">This publishes what was proven or fully enumerated — the DoS and the unsafe inventory. The broader auth-flow and business-logic lenses are scoped in the walkthrough, not exhausted here.</div></div>
</div>
</section>