Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

bench(corpus-ab): record corpus-x100 design + measured S1 sample validation

xerj-org2727fac4973b
diff --git a/benchmarks/corpus-ab-2026-10/CORPUS-X100.md b/benchmarks/corpus-ab-2026-10/CORPUS-X100.md
new file mode 100644
index 000000000..cf5c59ed7
--- /dev/null
+++ b/benchmarks/corpus-ab-2026-10/CORPUS-X100.md
@@ -0,0 +1,188 @@
+# CORPUS-X100 — scaling rust-vulns ×100, and what we expect it to change
+
+Status: design recorded 2026-10-01, immediately after PR #1116 (the #1111 A/B tie).
+This file is (a) the engineering plan for the scaled corpus, (b) the **pre-registered
+protocol + predictions for the re-test** (same discipline as #1111: written before the
+run, tie/loss published as such), and (c) the source notes for the blogpost. Sample
+artifacts that back every claim below: [corpus-x100/](corpus-x100/) (miner script, seeds,
+raw output).
+
+---
+
+## 1. Why: the measured autopsy of the #1111 tie
+
+From [RESULTS.md](RESULTS.md): arm X (corpus+XERJ) filed higher-precision findings but
+**zero confirmed findings trace to corpus-assisted discovery** (x1: one unused sweep
+query; x2: zero invocations). Baseline measured off the gen-1 pack (`rust-vulns-pack.zip`,
+gen 1, 2026-10-01, 16 shards, 10.5 MB):
+
+| fact | number |
+|---|---|
+| records | 1,963 (advisory prose, identity-resolved RustSec+OSV; 4,128 envelopes → 1,963) |
+| records containing **any code** | 277 (**14%**) — advisory bodies, mostly prose |
+| `affected_functions` populated | 271 (13%), names only, no source |
+| published ≥2025 (**un-memorized stratum**) | **1,001 (50%)** — 716 from 2026 alone |
+| seeds with a GitHub fix-commit ref + semver range | 995 (50%) |
+
+Three causes, in order of weight:
+
+1. **Surface — zero invocations.** The agent never queried the corpus. Nothing in the
+ task, tool description, or workflow made retrieval the natural move at the moment of
+ "is this pattern a known bug class?". A corpus that is never queried cannot help,
+ regardless of content.
+2. **Shape — prose cannot match code.** A reviewer's mental query is *code-shaped*
+ ("does `&fmt[i..]` walking bytes have precedent?"). Advisory prose about
+ `hubuum_client` does not retrieve against that, semantically or lexically. 14% code
+ coverage ≈ no code corpus.
+3. **Size/memorization — the cause we expected, half-wrong.** We hypothesized the model
+ had memorized the content; in fact **half the pack is post-cutoff** and was still
+ unused — because causes 1 and 2 dominated. Size alone would not have fixed the tie.
+ But scale still matters for the *tail*: bug classes outside the famous ones live in
+ niche crates' histories, which is where retrieval beats memorization (our own
+ reference-coding measurement: 1.65× fewer output tokens on un-memorized code).
+
+**Consequence for design:** ×100 must fix shape first, surface second, size third.
+A ×100 of more prose would tie again.
+
+## 2. What ×100 honestly requires (the arithmetic is recorded, not guessed)
+
+Measured on a 4-seed live sample (zeptoclaw, fuel-vm, ml-dsa/RustCrypto monorepo,
+hubuum_client — `corpus-x100/s1-sample-output.jsonl`): **9.8 vulnerable functions and
+3.8 trigger tests per advisory seed**, ~35 published crate versions ≤ fixed per seed
+(version-pinning metadata). One seed walked end-to-end (GHSA-5wp8-q9mx-8jx8, zeptoclaw
+shell-allowlist bypass): advisory → fix commit → `validate_command()` source at `fix^`
+(3,047 B) → fix diff → the before/after regression-test pair (the before-test even
+asserts the *wrong* behaviour and the after-test cites the GHSA id — the trigger pair
+is a self-documenting exploitability record).
+
+The strata:
+
+| stratum | content | projected unique records | provenance/confidence | status |
+|---|---|---|---|---|
+| S0 (existing) | advisory spine, identity-resolved | 1,963 | advisory-backed | done (gen 1) |
+| S1 | version-pinned **vulnerable function source** extracted at `fix^` per advisory | **~10,000** (995 × 9.8, measured; tree-sitter pass will trim over-attribution to ~5–15k) | advisory-backed | sample validated |
+| S2 | **trigger pairs**: before/after regression tests from the fix commit | **~4,000** (995 × 3.8, measured) | advisory-backed | sample validated |
+| S3 | **mined hardening commits**: security-relevant fixes (panic/overflow/traversal/escape/injection keywords + `.rs` diff) from the git histories of the top ~2,000 crates — bugs that never got an advisory | **~150,000–200,000** (2,000 crates × ~50–100 commits × ~2 fns; measured per-repo rates TBD at stage-3 dry-run) | **provisional** (keyword+LLM-classified, never advisory-backed) | designed, not run |
+| **total** | | **~196,000 ≈ ×100** | ~8% advisory-backed, ~92% provisional | |
+
+Two things this arithmetic forces us to say plainly (blogpost §3):
+
+- **Advisory-backed scaling caps at ~×8.** The world has ~2k Rust advisories; mining them
+ perfectly yields ~16k code records. Anyone promising a big advisory-only corpus is
+ padding with duplicates.
+- **The ×100 is real only via S3**, whose records are provisional by construction. That
+ is acceptable *for retrieval evidence* (a reviewer wants precedent, and precedent with
+ a confidence tag), and every record carries `provenance` so downstream claims can
+ stratify. Uniqueness key: `(repo, path, function, introducing commit)`; the same
+ function fixed twice for two different bugs is **two** records (legitimately — review
+ value). Cross-version near-duplicates collapse into one record + version list — the
+ ×100 counts unique records, not shards.
+
+## 3. Record schema (additive; S0 unchanged)
+
+One new discriminated field, `record_type`:
+
+- `advisory` — S0, current shape untouched.
+- `vuln_function` — `{record_type, advisory_ids[], package, versions[], first_seen_version,
+ fixed_version, repo, commit, path, function, lang_line_span, source, fix_diff,
+ bug_class, licence, provenance: "advisory-backed"|"mined-provisional",
+ extraction: "tree-sitter-rust"|"regex", mined_at}`.
+- `trigger_pair` — `{...same header..., test_before, test_after, asserts}` — the PoC shape.
+
+`bug_class` is a normalized taxonomy (~40 canonical Rust security classes, seeded from
+the #1111 validation gate's own 33 confirmed classes — panic-under-abort, slice
+char-boundary, unbounded recursion on request-controlled input, path join traversal,
+CRC-not-covering-footer, silent-merge data loss, early-exit auth, unbounded map growth…
+our own audit output becomes taxonomy seeds). Queries against the corpus then match
+*class + code shape*, which is the thing a reviewer actually holds in their head.
+
+**Licence discipline (non-negotiable, same rules as corpus.json):** per-record `licence`
+field — RustSec/OSV metadata CC0; crate and repo source under each project's own licence
+(MIT/Apache dominant; GPL/BUSL crates are fine **for retrieval evidence** but tagged, so
+any future "copy this pattern into code" decision can filter; copying rules unchanged
+from CLAUDE.md). S3 records carry the repo's licence and are never copied, only cited.
+
+**Scale budget:** ~200k records × ~5 KB ≈ 1.0–1.5 GB JSONL → ~0.6–0.9 GB zipped pack
+(GitHub asset cap 2 GB/file — one asset, no split). Build compute: 995 partial clones +
+`git show fix^` ≈ ~1 h parallel-16; S3 = 2,000 partial clones + `git log --grep` scans
+≈ 6–11 h single-threaded, <1 h at ×16. Indexing cost measured at build time (expect
+minutes; goes in the blogpost as a XERJ scale datapoint).
+
+## 4. Re-test protocol (pre-registered — committed before any re-run)
+
+Changes vs #1111, each addressing a diagnosed cause:
+
+1. **Surface fix:** arm X's task file names the corpus the way a tool would:
+ "a Rust vulnerability-pattern corpus is queryable via `xerj code rust-vulns-xl '<code
+ shape or bug class>'`; consulting precedent before filing is part of the method."
+ Arm P gets the symmetric sentence about its own available knowledge ("you have no
+ external knowledge base; use your own judgment"), so instruction effort is matched.
+2. **Shape fix:** corpus is the xl build (S0+S1+S2+S3) with `bug_class` taxonomy.
+3. **Instrumentation (no more inferred provenance):** every `xerj code` call is logged;
+ a finding may cite corpus record ids; the gate scores `cited_records` per finding
+ instead of replaying transcripts.
+4. **Two targets, because they should behave differently:**
+ a. **Memorized-idiom target** — XERJ's own engine code (same as #1111).
+ b. **Un-memorized target** — a niche dependency-heavy Rust project (selected after the
+ corpus freeze; ≥6 direct deps, <5k GitHub stars, not in the model's comfort zone),
+ with ground truth from its post-cutoff advisories. Held-out discipline: the
+ project's advisories are *in* the corpus — that is not leakage, it is the corpus
+ doing its job; the gate scores whether the agent finds what the corpus could have
+ told it.
+5. **Stratified scoring:** every scored finding is tagged pre-cutoff/post-cutoff
+ memorization stratum and advisory-backed/mined provenance.
+
+### Pre-registered predictions (falsifiable; published now, before the run)
+
+- **P1 (surface):** arm X issues ≥1 corpus query per run. If zero again, the re-test is
+ void and the surface design failed — say so and stop.
+- **P2 (shape):** corpus-cited findings cite S1/S3 `vuln_function`/`trigger_pair`
+ records, not S0 prose. Cites confined to prose = shape fix failed.
+- **P3 (where an advantage is even possible):** on the un-memorized target, X's confirmed
+ findings on post-cutoff ground truth exceed P's. On the XERJ-own-code target we
+ **expect a tie** — that is the prediction, not a hedge.
+- **P4 (precision):** X's false-positive rate stays ≤ #1111's (0). If the provisional
+ S3 stratum degrades precision below P's, S3 gets down-weighted in the pack's suggested
+ ranking and that is recorded as a cost of scale.
+- **P5 (the honest end):** if P1–P3 hold and end-to-end advantage still does not, the
+ publishable conclusion is "retrieval corpora of this shape do not measurably help
+ LLM security review at this scale" — a real negative result, and the blogpost says it.
+
+## 5. Blogpost skeleton (for the next post; numbers trace to this repo)
+
+1. **The bet and the honest tie** — #1111 by the table (RESULTS.md): 13 vs 16
+ confirmed-exploitable, 1 FP vs 0, $5.06 vs $4.41 per confirmed finding — and zero
+ corpus-assisted discoveries. Published as a tie because that is what happened.
+2. **Autopsy of a tie** — the three causes with the baseline numbers (§1 above). The
+ genuinely uncomfortable finding: half the corpus was already un-memorized and it
+ still never got queried. The bottleneck was not model knowledge; it was that a prose
+ corpus has no answer to a code-shaped question.
+3. **What ×100 really means** — the arithmetic nobody publishes: advisories cap at ~2k
+ worldwide; honest advisory-backed scaling is ×8 (measured 9.8 fns + 3.8 triggers per
+ seed); the ×100 lives in the ecosystem's un-advised fix history, at the cost of
+ provisional provenance. Hero demo: the zeptoclaw walkthrough (advisory →
+ `validate_command()` at `fix^` → the regression test that documents the exploit).
+4. **The discipline** — pre-registration, provenance tags, licence tags, stratified
+ scoring; P1–P5 published before the run.
+5. **The re-run** — whatever it says. Either the corpus earns a qualified claim
+ (un-memorized stratum, code-shaped corpus), or the negative result stands.
+6. **Portable lessons** (true regardless of P1–P5): dynamic re-verification beats any
+ corpus for precision; two independent reviewers find ≈1.5× one (33 classes, 2 shared);
+ retrieval's lane is precedent the model cannot have memorized.
+
+## 6. Execution order (tracked as #61 / #1110 follow-on)
+
+1. **S1+S2 production harvester** — extend `xerj-autoindex` harvest with a
+ `git-fix-mining` source (partial clone, `git show fix^:<path>`, tree-sitter-rust
+ function extraction — the sample miner's regex pass is the prototype and its
+ known miss is recorded: fuel-vm's fix touched no `fn`-attributable code, whole-file
+ fallback required). Engine-side change; reference-code mandate note: mining design
+ adapts the published CVEfixes/Vul4J method (commit-mining, not copying) — cited in
+ the harvester's docs.
+2. **Taxonomy pass** — `bug_class` classifier over S0–S2 (LLM-assisted, human-audited
+ sample ≥200), seeded from #1111's 33 confirmed classes.
+3. **S3 dry-run** — 20 crates, measure commits/repo, then full 2,000-crate run.
+4. **Pack gen 2** — `rust-vulns-xl`, signed, published; freshness badge reports both
+ total and advisory-backed counts (a 200k-record pack whose badge hides that 92% is
+ provisional would be a dishonest pack).
+5. **Re-test** per §4, then RESULTS-2.md and the blogpost.
diff --git a/benchmarks/corpus-ab-2026-10/corpus-x100/s1-sample-output.jsonl b/benchmarks/corpus-ab-2026-10/corpus-x100/s1-sample-output.jsonl
new file mode 100644
index 000000000..3f5d00495
--- /dev/null
+++ b/benchmarks/corpus-ab-2026-10/corpus-x100/s1-sample-output.jsonl
@@ -0,0 +1,5 @@
+# seeds=4 vuln_functions_total=39
+{"advisory": "GHSA-2625-rw7m-5q5x", "crate": "hubuum_client", "repo": "hubuum/hubuum-client-rust", "changed_rs": 15, "vuln_functions": [{"file": "src/client/async.rs", "fn": "send_with_retry", "line": 423, "bytes": 1459}, {"file": "src/client/async.rs", "fn": "auth_providers", "line": 553, "bytes": 756}, {"file": "src/client/async.rs", "fn": "login", "line": 567, "bytes": 1043}, {"file": "src/client/async.rs", "fn": "login_with_token", "line": 592, "bytes": 808}, {"file": "src/client/async.rs", "fn": "healthz", "line": 628, "bytes": 703}, {"file": "src/client/async.rs", "fn": "readyz", "line": 644, "bytes": 704}, {"file": "src/client/async.rs", "fn": "request_stream_with_endpoint", "line": 888, "bytes": 851}, {"file": "src/client/async.rs", "fn": "send", "line": 3260, "bytes": 660}, {"file": "src/client/async.rs", "fn": "output_stream", "line": 3393, "bytes": 1028}, {"file": "src/client/async.rs", "fn": "send", "line": 3519, "bytes": 857}, {"file": "src/client/async.rs", "fn": "send", "line": 3649, "bytes": 658}, {"file": "src/client/async.rs", "fn": "send", "line": 3789, "bytes": 834}, {"file": "src/client/async.rs", "fn": "stream", "line": 4118, "bytes": 853}, {"file": "src/client/shared.rs", "fn": "redacted_url_for_log", "line": 103, "bytes": 494}, {"file": "src/client/shared.rs", "fn": "process_transport_response", "line": 262, "bytes": 1189}, {"file": "src/client/sync.rs", "fn": "send_with_retry", "line": 389, "bytes": 1438}, {"file": "src/client/sync.rs", "fn": "auth_providers", "line": 518, "bytes": 695}, {"file": "src/client/sync.rs", "fn": "login", "line": 529, "bytes": 983}, {"file": "src/client/sync.rs", "fn": "login_with_token", "line": 551, "bytes": 751}, {"file": "src/client/sync.rs", "fn": "healthz", "line": 584, "bytes": 642}, {"file": "src/client/sync.rs", "fn": "readyz", "line": 597, "bytes": 643}, {"file": "src/client/sync.rs", "fn": "request_stream_with_endpoint", "line": 824, "bytes": 794}, {"file": "src/client/sync.rs", "fn": "send", "line": 3068, "bytes": 623}, {"file": "src/client/sync.rs", "fn": "send", "line": 3306, "bytes": 820}, {"file": "src/client/sync.rs", "fn": "send", "line": 3432, "bytes": 621}, {"file": "src/client/sync.rs", "fn": "send", "line": 3567, "bytes": 797}, {"file": "src/client/sync.rs", "fn": "new", "line": 3808, "bytes": 259}, {"file": "src/client/tests.rs", "fn": "sync_export_run_failed_errors", "line": 429, "bytes": 1162}, {"file": "src/client/tests.rs", "fn": "async_export_run_failed_errors", "line": 519, "bytes": 1200}, {"file": "src/errors.rs", "fn": "fmt", "line": 117, "bytes": 4403}, {"file": "src/types/remote.rs", "fn": "fmt", "line": 399, "bytes": 996}, {"file": "tests/foundations.rs", "fn": "response_and_error_debug_output_redacts_sensitive_values", "line": 473, "bytes": 1198}], "trigger_tests_touched": 0, "trigger_tests": [], "patch_bytes": 110135, "published_versions": 23, "versions_le_fixed": 15}
+{"advisory": "GHSA-2pgj-5cv2-6xxw", "crate": "fuel-vm", "repo": "FuelLabs/fuel-vm", "changed_rs": 3, "vuln_functions": [], "trigger_tests_touched": 1, "trigger_tests": ["test_store_word"], "patch_bytes": 6340, "published_versions": 115, "versions_le_fixed": 98}
+{"advisory": "GHSA-5wp8-q9mx-8jx8", "crate": "zeptoclaw", "repo": "qhkm/zeptoclaw", "changed_rs": 2, "vuln_functions": [{"file": "src/security/shell.rs", "fn": "validate_command", "line": 271, "bytes": 3047}, {"file": "src/security/shell.rs", "fn": "test_allowlist_strict_empty_blocks_nothing", "line": 638, "bytes": 607}, {"file": "src/tools/android/actions.rs", "fn": "device_shell", "line": 408, "bytes": 1117}, {"file": "src/tools/android/actions.rs", "fn": "test_blocked_shell_commands", "line": 550, "bytes": 594}, {"file": "src/tools/android/actions.rs", "fn": "test_device_shell_blocks_normalized_whitespace", "line": 746, "bytes": 355}], "trigger_tests_touched": 14, "trigger_tests": ["test_allowlist_blocks_command_injection_via_ampersand", "test_allowlist_blocks_command_injection_via_and_and", "test_allowlist_blocks_command_injection_via_pipe", "test_allowlist_blocks_command_injection_via_semicolon", "test_allowlist_blocks_command_injection_via_subshell", "test_allowlist_strict_empty_blocks_everything", "test_allowlist_strict_empty_blocks_nothing", "test_allowlist_strips_path_prefix", "test_device_shell_blocks_rm_flag_permutations", "test_is_rm_recursive_force_variants", "test_literal_blocks_glob_wildcard_bypass", "test_literal_glob_does_not_block_bare_star"], "patch_bytes": 21926, "published_versions": 26, "versions_le_fixed": 12}
+{"advisory": "GHSA-5x2r-hc65-25f9", "crate": "ml-dsa", "repo": "RustCrypto/signatures", "changed_rs": 2, "vuln_functions": [{"file": "ml-dsa/src/hint.rs", "fn": "monotonic", "line": 117, "bytes": 114}, {"file": "ml-dsa/src/hint.rs", "fn": "bit_unpack", "line": 121, "bytes": 949}], "trigger_tests_touched": 0, "trigger_tests": [], "patch_bytes": 3243, "published_versions": 21, "versions_le_fixed": 14}
diff --git a/benchmarks/corpus-ab-2026-10/corpus-x100/sample-s1-miner.py b/benchmarks/corpus-ab-2026-10/corpus-x100/sample-s1-miner.py
new file mode 100644
index 000000000..5454fe165
--- /dev/null
+++ b/benchmarks/corpus-ab-2026-10/corpus-x100/sample-s1-miner.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""S1 sample miner — validates the code-archaeology stratum of corpus-x100.
+
+For each advisory seed (from the gen-1 rust-vulns pack) with a fix-commit
+reference and a semver range:
+ 1. partial-clone the repo (blobs on demand),
+ 2. diff fix^..fix for changed *.rs files,
+ 3. extract the vulnerable file content at fix^ (always tree-aligned),
+ 4. split out the functions whose hunks touch them (regex-level split;
+ the production harvester uses tree-sitter-rust),
+ 5. capture before/after regression tests (the trigger pair),
+ 6. count published crate versions in the affected range (version pinning).
+
+Emits one JSON summary line per seed. This is the feasibility probe for
+CORPUS-X100.md, not the production harvester.
+"""
+import json, re, subprocess, sys, tempfile, urllib.request, os, shutil
+from collections import namedtuple
+
+UA = {'User-Agent': 'xerj-corpus-x100-sample (contact: [email protected])'}
+
+Seed = namedtuple('Seed', 'adv_id crate repo sha fixed')
+
+
+def http_json(url):
+ req = urllib.request.Request(url, headers=UA)
+ with urllib.request.urlopen(req, timeout=30) as r:
+ return json.loads(r.read().decode())
+
+
+def run(cmd, cwd):
+ p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
+ if p.returncode != 0:
+ raise RuntimeError(f"{' '.join(cmd[:4])}… -> {p.returncode}: {p.stderr[:200]}")
+ return p.stdout
+
+
+def parse_url(u):
+ # https://github.com/OWNER/REPO/commit/SHA -> (owner/repo, sha)
+ m = re.match(r'https://github\.com/([^/]+)/([^/]+)/commit/([0-9a-f]{7,40})', u)
+ return (f'{m.group(1)}/{m.group(2)}', m.group(3)) if m else None
+
+
+FN_RE = re.compile(r'^(\s*)(pub(?:\([^)]*\))?\s+)?(async\s+)?fn\s+(\w+)', re.M)
+
+
+def functions_touching(file_src, hunk_lines):
+ """Functions whose declared line range contains any hunk line number."""
+ out = []
+ for m in FN_RE.finditer(file_src):
+ start = file_src[:m.start()].count('\n') + 1
+ body = file_src[m.start():]
+ # brace-match from the fn's opening brace
+ i, depth, opened = body.find('{'), 0, False
+ end = start
+ for j in range(i, len(body)):
+ if body[j] == '{':
+ depth, opened = depth + 1, True
+ elif body[j] == '}':
+ depth -= 1
+ if opened and depth == 0:
+ end = start + body[:j].count('\n')
+ break
+ if any(start <= ln <= end for ln in hunk_lines):
+ out.append({'fn': m.group(4), 'line': start, 'source': body[:j + 1]})
+ return out
+
+
+def mine(seed, workdir):
+ rec = {'advisory': seed.adv_id, 'crate': seed.crate, 'repo': seed.repo}
+ d = os.path.join(workdir, re.sub(r'\W', '_', seed.repo))
+ if not os.path.isdir(d):
+ run(['git', 'clone', '--quiet', '--filter=blob:none', '--no-checkout',
+ f'https://github.com/{seed.repo}', d], workdir)
+ # resolve short shas
+ sha = run(['git', 'rev-parse', seed.sha + '^{commit}'], d).strip()
+ files = [f for f in run(['git', 'diff', '--name-only', sha + '^', sha, '--', '*.rs'],
+ d).splitlines() if f]
+ rec['changed_rs'] = len(files)
+ vuln_fns, tests = [], []
+ for f in files:
+ try:
+ src = run(['git', 'show', f'{sha}^:{f}'], d)
+ except RuntimeError:
+ continue # file added by the fix itself — no vulnerable parent copy
+ hunks = run(['git', 'diff', '-U0', sha + '^', sha, '--', f], d)
+ lines = []
+ for h in re.finditer(r'^@@+\s+-(\d+)(?:,(\d+))?', hunks, re.M):
+ a, n = int(h.group(1)), int(h.group(2) or 1)
+ lines += list(range(a, a + n))
+ for fn in functions_touching(src, lines):
+ vuln_fns.append({'file': f, **{k: fn[k] for k in ('fn', 'line')},
+ 'bytes': len(fn['source'])})
+ if '/test' in f or f.startswith('tests/') or 'mod tests' in src:
+ tests.append(f)
+ # in-file test modules count as trigger sources too
+ patch = run(['git', 'diff', sha + '^', sha], d)
+ test_fns = sorted(set(re.findall(r'fn (test_\w+)', patch)))
+ rec['vuln_functions'] = vuln_fns
+ rec['trigger_tests_touched'] = len(test_fns)
+ rec['trigger_tests'] = test_fns[:12]
+ rec['patch_bytes'] = len(patch)
+ # version pinning: published versions in range (approx: all <= fixed)
+ try:
+ vs = http_json(f'https://crates.io/api/v1/crates/{seed.crate}/versions')['versions']
+ nums = [v['num'] for v in vs]
+ in_range = [n for n in nums if n < (seed.fixed or '∞')] if seed.fixed else nums
+ rec['published_versions'] = len(nums)
+ rec['versions_le_fixed'] = len(in_range)
+ except Exception as e:
+ rec['version_error'] = str(e)[:80]
+ return rec
+
+
+def main():
+ seeds = [Seed(*s) for s in json.load(open(sys.argv[1]))]
+ total_fns = 0
+ with tempfile.TemporaryDirectory(prefix='s1mine-') as wd:
+ for s in seeds:
+ try:
+ rec = mine(s, wd)
+ except Exception as e:
+ rec = {'advisory': s.adv_id, 'error': str(e)[:200]}
+ total_fns += len(rec.get('vuln_functions') or [])
+ print(json.dumps(rec))
+ print(f"# seeds={len(seeds)} vuln_functions_total={total_fns}", file=sys.stderr)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/benchmarks/corpus-ab-2026-10/corpus-x100/seeds-sample.json b/benchmarks/corpus-ab-2026-10/corpus-x100/seeds-sample.json
new file mode 100644
index 000000000..092a89db9
--- /dev/null
+++ b/benchmarks/corpus-ab-2026-10/corpus-x100/seeds-sample.json
@@ -0,0 +1 @@
+[["GHSA-2625-rw7m-5q5x", "hubuum_client", "hubuum/hubuum-client-rust", "5a5c275ffa45f342459b7d3e977926da643bde50", "0.6.1"], ["GHSA-2pgj-5cv2-6xxw", "fuel-vm", "FuelLabs/fuel-vm", "9c97c2bf782626b35ba48e154f210f91c847a513", "0.60.1"], ["GHSA-5wp8-q9mx-8jx8", "zeptoclaw", "qhkm/zeptoclaw", "68916c3e4f3af107f11940b27854fc7ef517058b", "0.6.2"], ["GHSA-5x2r-hc65-25f9", "ml-dsa", "RustCrypto/signatures", "400961412be2e2ab787942cf30e0a9b66b37a54a", "0.1.0-rc.4"]]
\ No newline at end of file