Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): rust-vulns showcase pack — rustsec-md adapter, measured build, claims-pinned docs (M5)

xerj-org1962f792b37f
diff --git a/ROADMAP.md b/ROADMAP.md
index 0cc74ed6e..4e93c7202 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -139,7 +139,7 @@ The 1.0 bar: **every public claim verified against the release binary, and every
- **A real object-storage backend.** *Done in rc.77:* [#965](https://github.com/xerj-org/xerj/issues/965) wired the segment path to object storage — `storage.backend = "s3"` packs each segment family into one immutable ZBM1 bundle object with `snapshot.json` as the publication point, merges publish before retiring inputs, and a fresh node adopts the bucket (see *Shipping today* for the full statement). The client (`s3.rs`, real S3-compatible: Cloudflare R2, MinIO, AWS S3), the read-through segment cache, per-request cost accounting by billing class and the stop-don't-warn budget had landed in rc.75. Separately, `xerj autoindex s3://bucket/prefix` reads documents OUT of a bucket, which is a source, not a home.
- **A block index mode for logs.** An index mode for log-shaped data that stores rows in time-partitioned columnar blocks and skips whole blocks by their min/max metadata, instead of building a per-term inverted index over every field. *Today:* the `xerj-logs` crate implements that design (columnar encoding, log-template extraction, time-range queries with block skipping, retention), is compiled into `xerj-engine` and `xerj-server` as a dependency, and is **called from no non-test code**. Log-shaped analytics run through the general columnar segment format and the aggregation suite. Wire it behind an explicit index setting and measure it against the general path on the same data, or remove it — the *Log-analytics data path* theme below is this same item.
- **User-code ingest plugins.** *Today:* the ingest pipeline's transforms are built-in native Rust plugins — rename, drop, add, JSON parse, timestamp parse, PII redaction, grok, route — and they do run on `_bulk`. The crate is named `xerj-wasm`, but **the wasmtime backend is not in the tree**: there is no `wasmtime` dependency and no `wasm` feature, only a note that one could be added behind the same trait. Planned: a sandboxed runtime for user-supplied transforms with fuel and memory limits and no ambient filesystem or network access. Until then the refusal rule holds: a pipeline naming a processor this build does not implement is stored as unrunnable and every ingest through it is refused, never quietly run as a shorter pipeline.
-- **A corpus hub of signed, pre-indexed packs.** Download a reference corpus — a standard library, a specification set — already indexed, and mount it, instead of every user spending the same CPU-hours indexing the same public text. *Today:* nothing; no pack code exists — though the substrate now has a tracker: [#1030](https://github.com/xerj-org/xerj/issues/1030) covers the portable one-file bundle (indexed data that moves between machines), which is the format this item would distribute. The pack format is the design work: a **format version** readers refuse when they do not know it (the rule segment headers already follow), **per-file checksums**, a **detached signature** over the manifest, and **licence and provenance on every record** so a hit can say where its text came from and under what terms. Three risks decide whether this ships at all: **redistribution licence** (an index is a derived copy of its source text — a pack may carry only what its licence lets us redistribute, which excludes much of what people most want indexed); **pack safety** (a pack is untrusted input to the segment readers, so it needs the fuzzing the on-disk formats get, and a signature proves who built a pack, not that it is safe to open); and **format stability** (a published pack outlives the release that built it, so the pack format needs a compatibility promise the internal segment format has never had to make).
+- **A corpus hub of signed, pre-indexed packs.** Download a reference corpus — a standard library, a specification set — already indexed, and mount it, instead of every user spending the same CPU-hours indexing the same public text. *Today:* the **records half is built** — `xerj corpus build` (PR [#1046](https://github.com/xerj-org/xerj/pull/1046)) turns a declarative recipe (sources with licences, identity edges, merge precedence, derived fields) into a checksummed portable pack of *records* with a `format_version` readers refuse when unknown, per-file checksums verified on `corpus add --from <pack>`, and licence + provenance on every record; the first real pack is `rust-vulns` (`tools/packs/rust-vulns/`, identity-resolved across osv.dev and RustSec, with a test pinning its README numbers to a measured build). Still unbuilt: the **detached signature** (checksums ship today; signing is the next milestone), the hub itself, and the **pre-indexed** half — [#1030](https://github.com/xerj-org/xerj/issues/1030) remains the tracker for the indexed-segment bundle that is the format this item would distribute, since indexing a records pack still costs each consumer the same CPU-hours. Three risks decide whether the rest ships: **redistribution licence** (an index is a derived copy of its source text — a pack may carry only what its licence lets us redistribute, which excludes much of what people most want indexed; the builder carries licence as record data and the pack author owns the choice); **pack safety** (a pack is untrusted input to the segment readers, so it needs the fuzzing the on-disk formats get, and a signature proves who built a pack, not that it is safe to open); and **format stability** (a published pack outlives the release that built it, so the pack format needs a compatibility promise the internal segment format has never had to make).
## Beyond 1.0 — themes
diff --git a/engine/crates/xerj-autoindex/src/harvest/mod.rs b/engine/crates/xerj-autoindex/src/harvest/mod.rs
index cf5634c7d..5886e6d91 100644
--- a/engine/crates/xerj-autoindex/src/harvest/mod.rs
+++ b/engine/crates/xerj-autoindex/src/harvest/mod.rs
@@ -336,10 +336,13 @@ fn glob_regex(glob: &str) -> regex::Regex {
regex::Regex::new(&re).expect("glob translation always yields a valid regex")
}
-/// Parse one harvested file: `.json`/`.jsonl`/`.ndjson`. A `.json` array
-/// yields its elements; `.jsonl` yields one per non-empty line. Other
-/// extensions are skipped by the glob, not here — reaching this with a
-/// non-JSON file is a hard error, not a silent drop.
+/// Parse one harvested file: `.json`/`.jsonl`/`.ndjson`/`.md`. A `.json`
+/// array yields its elements; `.jsonl` yields one per non-empty line; a
+/// `.md` yields its raw text as ONE string value — the format adapter (e.g.
+/// `rustsec-md`) owns interpreting it, so an `.md` globbed by a json-format
+/// source is visible junk, not a silent parse error. Other extensions are
+/// skipped by the glob, not here — reaching this with an unlisted type is a
+/// hard error, not a silent drop.
fn parse_file(path: &Path) -> Result<Vec<Value>> {
let raw =
std::fs::read_to_string(path).with_context(|| format!("cannot read {}", path.display()))?;
@@ -348,6 +351,9 @@ fn parse_file(path: &Path) -> Result<Vec<Value>> {
.and_then(|n| n.to_str())
.unwrap_or_default()
.to_lowercase();
+ if name.ends_with(".md") || name.ends_with(".markdown") {
+ return Ok(vec![Value::String(raw)]);
+ }
let jsonl = name.ends_with(".jsonl") || name.ends_with(".ndjson");
if jsonl {
let mut out = Vec::new();
@@ -420,6 +426,11 @@ mod tests {
let c = tmp.path().join("c.json");
std::fs::write(&c, r#"{"id":"one"}"#).unwrap();
assert_eq!(parse_file(&c).unwrap().len(), 1);
+ let d = tmp.path().join("d.md");
+ std::fs::write(&d, "```toml\n[advisory]\nid = \"R\"\n```\n\n# T\n").unwrap();
+ let parsed = parse_file(&d).unwrap();
+ assert_eq!(parsed.len(), 1, "one raw string value, not json-parsed");
+ assert!(parsed[0].as_str().unwrap().starts_with("```toml"));
}
// ── end to end: recipe → harvest → identity → pack, twice ─────────────
diff --git a/engine/crates/xerj-autoindex/src/harvest/normalize.rs b/engine/crates/xerj-autoindex/src/harvest/normalize.rs
index ae8b10b76..7d9ba6042 100644
--- a/engine/crates/xerj-autoindex/src/harvest/normalize.rs
+++ b/engine/crates/xerj-autoindex/src/harvest/normalize.rs
@@ -1,12 +1,14 @@
//! Input-format adapters: one function per known record FORMAT, each turning
-//! a source JSON value into flat key paths.
+//! a source value into flat key paths.
//!
//! An adapter knows the INPUT format, never our domain: `osv` understands
-//! the ossf/osv-schema shape (shared by osv.dev dumps, the RustSec `osv`
-//! branch and github-reviewed GHSA files); `flat` does generic flattening
-//! for everything else. Which keys of the result feed the query envelope is
-//! the recipe's job, not the adapter's — the same `osv` adapter would serve
-//! a Go vulnerability corpus unchanged.
+//! the ossf/osv-schema shape (shared by osv.dev dumps and github-reviewed
+//! GHSA files); `rustsec-md` understands RustSec's native advisory file
+//! (TOML frontmatter + markdown — the mechanism is generic frontmatter
+//! extraction, the curated table names are upstream's); `flat` does generic
+//! flattening for everything else. Which keys of the result feed the query
+//! envelope is the recipe's job, not the adapter's — the same `osv` adapter
+//! would serve a Go vulnerability corpus unchanged.
//!
//! Flattening mirrors `extract::flatten_object` semantics (two levels of
//! nesting become `a_b` keys; deeper structure and arrays-of-objects are
@@ -29,6 +31,10 @@ pub fn normalize(format: Format, v: &Value) -> Map<String, Value> {
match format {
Format::Flat => flatten(v),
Format::Osv => normalize_osv(v),
+ Format::RustsecMd => match v {
+ Value::String(s) => normalize_rustsec_md(s),
+ other => flatten(other),
+ },
}
}
@@ -255,6 +261,173 @@ fn normalize_osv(v: &Value) -> Map<String, Value> {
out
}
+// ── RustSec native advisory ─────────────────────────────────────────────────
+
+/// Split a RustSec advisory into (frontmatter TOML text, markdown body). The
+/// file is one ```toml fenced block followed by prose; a file without the
+/// fence yields no frontmatter and the whole text as body.
+fn split_frontmatter(text: &str) -> (Option<&str>, &str) {
+ let t = text.trim_start_matches('\u{feff}');
+ let Some(after) = t.strip_prefix("```toml\n") else {
+ return (None, t);
+ };
+ match after.find("\n```\n") {
+ Some(i) => (Some(&after[..i]), &after[i + 5..]),
+ // a fence that never closes: treat everything after the opener as
+ // frontmatter-shaped text we cannot trust, keep it as body instead
+ None => (None, t),
+ }
+}
+
+/// RustSec's native `crates/<crate>/RUSTSEC-*.md`: a ```toml frontmatter
+/// (`[advisory]`/`[versions]`/`[affected]` tables) plus the advisory's
+/// markdown prose. The `osv`-branch conversion drops `affected.functions`
+/// and nulls `ecosystem_specific` (measured 2026-09-27: 0 of 1,252 records
+/// carry functions there), so the native files are the ONLY source of the
+/// curated vulnerable-function paths.
+///
+/// Facts both adapters carry land in the SAME key names (`id`, `aliases`,
+/// `summary`, `severity`, `affected_functions`, `withdrawn`, …) so a recipe
+/// can merge this source with an `osv` one without a bridge; RustSec-only
+/// facts (`patched_versions`, `categories`, `keywords`, `date`) keep their
+/// own names. Uncurated frontmatter keys flow through the generic
+/// flattener prefixed by their table (`advisory_foo`), mirroring the osv
+/// adapter's passthrough.
+fn normalize_rustsec_md(text: &str) -> Map<String, Value> {
+ let (fm, body) = split_frontmatter(text);
+ let mut out = Map::new();
+
+ let parsed = fm
+ .map(|f| f.parse::<toml::Value>())
+ .transpose()
+ .ok()
+ .flatten();
+ let tables: Option<&toml::value::Table> = match parsed.as_ref() {
+ Some(toml::Value::Table(t)) => Some(t),
+ _ => None,
+ };
+
+ // Track what curated extraction consumed so the passthrough loop below
+ // does not duplicate it under a prefixed name.
+ let mut curated: Vec<(&str, &str)> = Vec::new();
+ let sub = |name: &str| tables.and_then(|t| t.get(name)).and_then(|v| v.as_table());
+ let advisory = sub("advisory");
+ let versions = sub("versions");
+ let affected = sub("affected");
+ let put_str = |out: &mut Map<String, Value>, key: &str, s: String| {
+ if !s.is_empty() {
+ out.insert(key.to_string(), Value::String(s));
+ }
+ };
+
+ if let Some(advisory) = advisory {
+ for k in ["id", "date", "url", "title", "withdrawn", "informational"] {
+ if let Some(s) = advisory.get(k).and_then(|v| v.as_str()) {
+ curated.push(("advisory", k));
+ put_str(&mut out, k, s.to_string());
+ }
+ }
+ // `cvss` is a bare vector string here; the osv adapter keeps the
+ // first score in `severity` and all in `severity_scores` — same.
+ if let Some(cvss) = advisory.get("cvss").and_then(|v| v.as_str()) {
+ curated.push(("advisory", "cvss"));
+ put_str(&mut out, "severity", cvss.to_string());
+ out.insert(
+ "severity_scores".to_string(),
+ Value::Array(vec![Value::String(cvss.to_string())]),
+ );
+ }
+ if let Some(pkg) = advisory.get("package").and_then(|v| v.as_str()) {
+ curated.push(("advisory", "package"));
+ out.insert(
+ "packages".to_string(),
+ Value::Array(vec![Value::String(pkg.to_string())]),
+ );
+ }
+ for k in ["aliases", "related", "categories", "keywords"] {
+ if let Some(arr) = advisory.get(k).and_then(|v| v.as_array()) {
+ let vals: Vec<Value> = arr
+ .iter()
+ .filter_map(|v| v.as_str())
+ .map(|s| Value::String(s.to_string()))
+ .collect();
+ if !vals.is_empty() {
+ curated.push(("advisory", k));
+ out.insert(k.to_string(), Value::Array(vals));
+ }
+ }
+ }
+ }
+ if let Some(versions) = versions {
+ for (src, dst) in [
+ ("patched", "patched_versions"),
+ ("unaffected", "unaffected_versions"),
+ ] {
+ if let Some(arr) = versions.get(src).and_then(|v| v.as_array()) {
+ let vals: Vec<Value> = arr
+ .iter()
+ .filter_map(|v| v.as_str())
+ .map(|s| Value::String(s.to_string()))
+ .collect();
+ if !vals.is_empty() {
+ curated.push(("versions", src));
+ out.insert(dst.to_string(), Value::Array(vals));
+ }
+ }
+ }
+ }
+ if let Some(affected) = affected {
+ // `functions` is an inline table of function path → version ranges;
+ // the PATHS are the joinable fact (same key the osv adapter's
+ // ecosystem_specific extraction produces). Sorted: the file's key
+ // order must not reach the pack bytes.
+ if let Some(fns) = affected.get("functions").and_then(|v| v.as_table()) {
+ let mut paths: Vec<String> = fns.keys().cloned().collect();
+ paths.sort();
+ if !paths.is_empty() {
+ curated.push(("affected", "functions"));
+ out.insert(
+ "affected_functions".to_string(),
+ Value::Array(paths.into_iter().map(Value::String).collect()),
+ );
+ }
+ }
+ }
+
+ // Everything not curated flows through the generic flattener prefixed by
+ // its table, so unknown-but-real frontmatter survives into the pack.
+ if let Some(t) = tables {
+ for (tname, table_val) in t {
+ let Some(sub) = table_val.as_table() else {
+ continue;
+ };
+ for (k, v) in sub {
+ if curated.iter().any(|(tn, ck)| *tn == tname && ck == k) {
+ continue;
+ }
+ let toml_json =
+ |tv: &toml::Value| -> Value { serde_json::to_value(tv).unwrap_or(Value::Null) };
+ flatten_into(&format!("{tname}_{k}"), toml_json(v), 0, &mut out);
+ }
+ }
+ }
+
+ // The prose: first `# ` heading is the advisory's title when the
+ // frontmatter did not carry one; the whole body is the `details`.
+ let heading = body
+ .lines()
+ .find(|l| l.starts_with("# "))
+ .map(|l| l[2..].trim().to_string());
+ if let Some(h) = heading {
+ put_str(&mut out, "summary", h);
+ }
+ let trimmed = body.trim();
+ if !trimmed.is_empty() {
+ put_str(&mut out, "details", trimmed.to_string());
+ }
+ out
+}
+
fn string_array(v: Option<&Value>) -> Option<Vec<Value>> {
match v {
Some(Value::Array(a)) if a.iter().all(|e| e.is_string()) => Some(a.to_vec()),
@@ -378,4 +551,123 @@ mod tests {
assert_eq!(m["schema_version"], json!("1.9.0"));
assert_eq!(m["upstream_of"], json!("y"));
}
+
+ // ── rustsec-md ─────────────────────────────────────────────────────────
+
+ /// The real shape of crates/smallvec/RUSTSEC-2021-0003.md, trimmed.
+ const RUSTSEC_MD: &str = r#"```toml
+[advisory]
+id = "RUSTSEC-2021-0003"
+package = "smallvec"
+aliases = ["CVE-2021-25900", "GHSA-43w2-9j62-hq99"]
+cvss = "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"
+date = "2021-01-08"
+url = "https://github.com/servo/rust-smallvec/issues/252"
+categories = ["memory-corruption"]
+keywords = ["buffer-overflow", "heap-overflow", "unsound"]
+
+[versions]
+patched = [">= 0.6.14, < 1.0.0", ">= 1.6.1"]
+unaffected = ["< 0.6.3"]
+
+[affected]
+functions = { "smallvec::SmallVec::insert_many" = [">= 0.6.3, < 0.6.14"] }
+```
+
+# Buffer overflow in SmallVec::insert_many
+
+A bug in the `SmallVec::insert_many` method caused it to allocate a buffer
+that was smaller than needed.
+"#;
+
+ #[test]
+ fn rustsec_md_extracts_the_curated_tables() {
+ let m = normalize(Format::RustsecMd, &json!(RUSTSEC_MD));
+ assert_eq!(m["id"], json!("RUSTSEC-2021-0003"));
+ assert_eq!(
+ m["aliases"],
+ json!(["CVE-2021-25900", "GHSA-43w2-9j62-hq99"])
+ );
+ assert_eq!(m["packages"], json!(["smallvec"]));
+ assert_eq!(
+ m["summary"],
+ json!("Buffer overflow in SmallVec::insert_many")
+ );
+ assert_eq!(
+ m["severity"],
+ json!("CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H")
+ );
+ assert_eq!(
+ m["affected_functions"],
+ json!(["smallvec::SmallVec::insert_many"])
+ );
+ assert_eq!(
+ m["patched_versions"],
+ json!([">= 0.6.14, < 1.0.0", ">= 1.6.1"])
+ );
+ assert_eq!(m["unaffected_versions"], json!(["< 0.6.3"]));
+ assert_eq!(m["categories"], json!(["memory-corruption"]));
+ assert_eq!(
+ m["keywords"],
+ json!(["buffer-overflow", "heap-overflow", "unsound"])
+ );
+ assert_eq!(m["date"], json!("2021-01-08"));
+ assert_eq!(
+ m["url"],
+ json!("https://github.com/servo/rust-smallvec/issues/252")
+ );
+ assert!(
+ m["details"]
+ .as_str()
+ .unwrap()
+ .starts_with("# Buffer overflow"),
+ "the markdown body is the details"
+ );
+ assert!(
+ !m.contains_key("advisory_id"),
+ "curated keys are not duplicated by the passthrough"
+ );
+ }
+
+ #[test]
+ fn rustsec_md_key_names_line_up_with_osv_for_merge() {
+ // The whole point of the shared names: a recipe merges a rustsec-md
+ // source with an osv one over `id` without a field_precedence bridge.
+ let md = normalize(Format::RustsecMd, &json!(RUSTSEC_MD));
+ let osv = normalize(
+ Format::Osv,
+ &json!({
+ "id": "RUSTSEC-2021-0003",
+ "aliases": ["CVE-2021-25900", "GHSA-43w2-9j62-hq99"],
+ "summary": "Out of bounds write in SmallVec::insert_many",
+ "modified": "2023-06-20T00:00:00Z"
+ }),
+ );
+ for key in ["id", "aliases"] {
+ assert_eq!(md[key], osv[key], "{key} must carry identical meaning");
+ }
+ }
+
+ #[test]
+ fn rustsec_md_without_a_fence_degrades_to_body() {
+ let m = normalize(
+ Format::RustsecMd,
+ &json!("# just prose\n\nno frontmatter here"),
+ );
+ assert!(!m.contains_key("id"));
+ assert_eq!(m["summary"], json!("just prose"));
+ assert!(m["details"].as_str().unwrap().contains("no frontmatter"));
+ }
+
+ #[test]
+ fn rustsec_md_unknown_frontmatter_survives_prefixed() {
+ const ADVICE: &str = "```toml\n[advisory]\nid = \"RUSTSEC-2000-0001\"\ncwe = \"CWE-787\"\n\n[versions]\npatched = [\">= 1.0\"]\n```\n\n# Title\n";
+ let m = normalize(Format::RustsecMd, &json!(ADVICE));
+ assert_eq!(
+ m["advisory_cwe"],
+ json!("CWE-787"),
+ "uncurated key, prefixed by table"
+ );
+ assert_eq!(m["id"], json!("RUSTSEC-2000-0001"));
+ }
}
diff --git a/engine/crates/xerj-autoindex/src/harvest/recipe.rs b/engine/crates/xerj-autoindex/src/harvest/recipe.rs
index 87c5170db..ca99e23ff 100644
--- a/engine/crates/xerj-autoindex/src/harvest/recipe.rs
+++ b/engine/crates/xerj-autoindex/src/harvest/recipe.rs
@@ -90,7 +90,8 @@ pub struct SourceSpec {
/// `http-zip`/`git` sources: where the bytes come from. A git url may
/// also be a local path (git itself accepts both — tests rely on it).
pub url: Option<String>,
- /// `git` sources: a pinned commit sha, or absent to track remote HEAD.
+ /// `git` sources: a full commit sha (pinned), a branch or tag name, or
+ /// absent to track the remote's default branch (HEAD).
pub rev: Option<String>,
pub glob: String,
pub format: Format,
@@ -108,10 +109,15 @@ pub enum SourceKind {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Format {
- /// OSV v1.x (the `ossf/osv-schema` shape shared by osv.dev dumps, the
- /// RustSec `osv` branch and github-reviewed GHSA files). Curated
- /// extraction of the standard fields plus generic flattening of the rest.
+ /// OSV v1.x (the `ossf/osv-schema` shape shared by osv.dev dumps and
+ /// github-reviewed GHSA files). Curated extraction of the standard
+ /// fields plus generic flattening of the rest.
Osv,
+ /// RustSec's native advisory: a `crates/<crate>/RUSTSEC-*.md` file —
+ /// ```toml frontmatter tables plus markdown prose. The only source of
+ /// the curated `affected.functions` paths (the osv-branch conversion
+ /// drops them).
+ RustsecMd,
/// Already-flat JSON records: generic two-level flattening only. This is
/// the "we have not written a format adapter for this yet" path.
Flat,
@@ -420,9 +426,10 @@ fn compile(path: &Path, file: RawRecipe) -> Result<Recipe> {
};
let format = match s.format.as_str() {
"osv" => Format::Osv,
+ "rustsec-md" => Format::RustsecMd,
"flat" => Format::Flat,
other => bail!(at(&format!(
- "source '{}': unknown format '{other}' (supported: osv, flat)",
+ "source '{}': unknown format '{other}' (supported: osv, rustsec-md, flat)",
s.slug
))),
};
diff --git a/engine/crates/xerj-autoindex/src/harvest/source.rs b/engine/crates/xerj-autoindex/src/harvest/source.rs
index dbda8910c..ad2a7a09c 100644
--- a/engine/crates/xerj-autoindex/src/harvest/source.rs
+++ b/engine/crates/xerj-autoindex/src/harvest/source.rs
@@ -30,6 +30,7 @@ use super::recipe::SourceSpec;
/// 3 MiB file can claim to expand to terabytes.
const MAX_EXTRACT_BYTES: u64 = 2 << 30;
+#[derive(Debug)]
pub struct Fetched {
/// The directory the glob walk starts from.
pub root: PathBuf,
@@ -93,9 +94,10 @@ pub fn save_watermark(cache_dir: &Path, slug: &str, watermark: &str) -> Result<(
.with_context(|| format!("save watermark for '{slug}'"))
}
-/// Shallow promisor checkout of `rev` (a sha or `HEAD`) into `target`,
-/// reusing the corpus-clone machinery: init + blob:none filter so a pin at
-/// an old sha costs one small fetch, never a full history. Returns HEAD.
+/// Shallow promisor checkout of `rev` into `target`, reusing the
+/// corpus-clone machinery: init + blob:none filter so a pin at an old sha
+/// costs one small fetch, never a full history. `rev` is a full sha, a
+/// branch/tag name, or `HEAD` (the remote's default). Returns HEAD.
fn git_checkout(target: &Path, url: &str, rev: &str, slug: &str) -> Result<String> {
if !target.join(".git").exists() {
std::fs::create_dir_all(target)
@@ -109,25 +111,29 @@ fn git_checkout(target: &Path, url: &str, rev: &str, slug: &str) -> Result<Strin
)
.ok();
}
- // A pinned rev (full sha) can reuse checkout_at_sha; tracking HEAD goes
- // through FETCH_HEAD, which checkout_at_sha cannot name.
+ // A pinned rev (full sha) can reuse checkout_at_sha; a named ref (branch
+ // or tag, e.g. RustSec's `osv` branch) and plain HEAD go through
+ // FETCH_HEAD, which checkout_at_sha cannot name. Anything else used to
+ // silently fetch HEAD — a typo'd branch name now fails loudly instead.
let ok = |r: Result<(i32, String)>| r.map(|(rc, _)| rc == 0).unwrap_or(false);
let looks_like_sha = rev.len() == 40 && rev.chars().all(|c| c.is_ascii_hexdigit());
if looks_like_sha {
crate::xc::checkout_at_sha(target, url, rev)
.with_context(|| format!("source '{slug}': checkout {rev}"))?;
} else {
+ // `rev` works directly as the refspec: a branch/tag name fetches that
+ // ref, and the literal "HEAD" fetches the remote's default HEAD.
if !ok(crate::xc::git(
Some(target),
- &["fetch", "--depth", "1", "origin", "HEAD"],
+ &["fetch", "--depth", "1", "origin", rev],
)) {
- bail!("source '{slug}': git fetch HEAD from {url} failed");
+ bail!("source '{slug}': git fetch {rev} from {url} failed");
}
if !ok(crate::xc::git(
Some(target),
&["checkout", "--quiet", "--force", "--detach", "FETCH_HEAD"],
)) {
- bail!("source '{slug}': git checkout FETCH_HEAD failed");
+ bail!("source '{slug}': git checkout FETCH_HEAD ({rev}) failed");
}
crate::xc::git(Some(target), &["clean", "-qfd"]).ok();
}
@@ -358,4 +364,67 @@ mod tests {
"checkout moved to the new tip"
);
}
+
+ #[test]
+ fn git_source_checks_out_a_named_branch() {
+ // RustSec's useful branch (`osv`) is not the default: `rev` must be
+ // able to name it. Before this was supported, a non-sha rev silently
+ // fetched the default branch's HEAD — the wrong tree, quietly.
+ let tmp = tempfile::tempdir().unwrap();
+ let origin = tmp.path().join("origin");
+ std::fs::create_dir_all(&origin).unwrap();
+ assert!(crate::xc::git(Some(&origin), &["init", "--quiet", "-b", "main"]).is_ok());
+ std::fs::write(origin.join("main.json"), r#"{"id":"M"}"#).unwrap();
+ let commit = |m: &str| {
+ crate::xc::git(Some(&origin), &["add", "."]).ok();
+ crate::xc::git(
+ Some(&origin),
+ &[
+ "-c",
+ "user.email=t@t",
+ "-c",
+ "user.name=t",
+ "commit",
+ "--quiet",
+ "-m",
+ m,
+ ],
+ )
+ .ok();
+ };
+ commit("main only");
+ // branch `export` carries a different tree
+ assert!(crate::xc::git(Some(&origin), &["checkout", "--quiet", "-b", "export"]).is_ok());
+ std::fs::write(origin.join("export.json"), r#"{"id":"E"}"#).unwrap();
+ std::fs::remove_file(origin.join("main.json")).unwrap();
+ commit("export tree");
+ assert!(crate::xc::git(Some(&origin), &["checkout", "--quiet", "main"]).is_ok());
+
+ let mut spec = super::super::recipe::SourceSpec {
+ slug: "repo".into(),
+ kind: super::super::recipe::SourceKind::Git,
+ path: None,
+ url: Some(origin.to_string_lossy().to_string()),
+ rev: Some("export".into()),
+ glob: "**/*.json".into(),
+ format: super::super::recipe::Format::Flat,
+ licence: "CC0-1.0".into(),
+ };
+ let f = fetch(&spec, tmp.path(), &tmp.path().join("cache")).unwrap();
+ assert!(
+ f.root.join("export.json").exists(),
+ "the named branch's tree"
+ );
+ assert!(
+ !f.root.join("main.json").exists(),
+ "not the default branch's"
+ );
+
+ // a rev that names nothing fails loudly, not as a silent HEAD fetch
+ spec.rev = Some("no-such-branch".into());
+ let err = fetch(&spec, tmp.path(), &tmp.path().join("cache"))
+ .unwrap_err()
+ .to_string();
+ assert!(err.contains("no-such-branch"), "{err}");
+ }
}
diff --git a/engine/crates/xerj-autoindex/tests/pack_claims.rs b/engine/crates/xerj-autoindex/tests/pack_claims.rs
new file mode 100644
index 000000000..945289525
--- /dev/null
+++ b/engine/crates/xerj-autoindex/tests/pack_claims.rs
@@ -0,0 +1,165 @@
+//! The rust-vulns pack's public numbers must be numbers a real build produced.
+//!
+//! Same discipline as `security_tooling_claims.rs` (issue #207's lesson): a
+//! README that advertises record counts nobody measured is a worse failure
+//! than admitting the gap. `tools/packs/rust-vulns/pack-stats.json` is the
+//! committed record of a measured build (`provenance: "measured"`); these
+//! tests fail when the README, the stats and the recipe disagree — so a
+//! rebuild with different numbers forces the docs to move with them, and a
+//! recipe edit without a rebuild fails the sha pin.
+
+use std::path::{Path, PathBuf};
+
+/// Walk up from this crate to the repository root. Panics rather than
+/// skipping: a claims check that turns into a no-op when it cannot find its
+/// inputs is the tautological pass this file exists to prevent.
+fn repo_root() -> PathBuf {
+ let mut dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+ loop {
+ if dir.join(".github/workflows/ci.yml").is_file() {
+ return dir;
+ }
+ assert!(
+ dir.pop(),
+ "no .github/workflows/ci.yml above {} — this test must run from a \
+ repository checkout, and must not pass by failing to look",
+ env!("CARGO_MANIFEST_DIR"),
+ );
+ }
+}
+
+fn pack_dir(root: &Path) -> PathBuf {
+ root.join("tools/packs/rust-vulns")
+}
+
+fn read(path: &Path) -> String {
+ std::fs::read_to_string(path).unwrap_or_else(|e| panic!("read {}: {e}", path.display()))
+}
+
+fn stats(root: &Path) -> serde_json::Value {
+ let dir = pack_dir(root);
+ let text = read(&dir.join("pack-stats.json"));
+ serde_json::from_str(&text).unwrap_or_else(|e| panic!("pack-stats.json is not valid JSON: {e}"))
+}
+
+/// The README's table row for `label` must carry exactly the number in
+/// pack-stats.json under `key`.
+fn readme_row_pins(root: &Path, label: &str, key: &str) {
+ let readme = read(&pack_dir(root).join("README.md"));
+ let want = stats(root)
+ .get(key)
+ .and_then(|v| v.as_u64())
+ .unwrap_or_else(|| panic!("pack-stats.json has no numeric '{key}'"));
+ // rows look like `| … label … | 1950 |` or `| … label … | 270 (707 …) |`,
+ // so pin on maximal digit runs, not whole cells
+ let needle = label.to_string();
+ let row = readme
+ .lines()
+ .find(|l| l.contains(&needle))
+ .unwrap_or_else(|| panic!("README has no row containing '{needle}'"));
+ let numbers: Vec<String> = row
+ .split(|c: char| !c.is_ascii_digit())
+ .filter(|s| !s.is_empty())
+ .map(str::to_string)
+ .collect();
+ assert!(
+ numbers.contains(&want.to_string()),
+ "README row '{needle}' does not carry {want} (pack-stats.{key}); rebuild \
+ the pack and move the docs with the numbers",
+ );
+}
+
+#[test]
+fn pack_stats_are_measured_not_asserted() {
+ let root = repo_root();
+ let s = stats(&root);
+ assert_eq!(
+ s["provenance"], "measured",
+ "pack-stats.json must record provenance 'measured' — 'estimated' numbers \
+ are exactly what this file exists to keep out of the README"
+ );
+ for key in [
+ "envelopes",
+ "records",
+ "merged_from_both_sources",
+ "records_with_functions",
+ "function_paths_distinct",
+ "function_entries_total",
+ "records_withdrawn",
+ "packages",
+ ] {
+ assert!(
+ s.get(key).and_then(|v| v.as_u64()).is_some_and(|v| v > 0),
+ "pack-stats.json lacks a positive numeric '{key}'"
+ );
+ }
+ // internal consistency: the parts cannot exceed the whole
+ let records = s["records"].as_u64().unwrap();
+ let merged = s["merged_from_both_sources"].as_u64().unwrap();
+ let with_fn = s["records_with_functions"].as_u64().unwrap();
+ assert!(merged <= records, "more merged records than records");
+ assert!(with_fn <= records, "more function records than records");
+}
+
+#[test]
+fn readme_counts_match_the_measured_stats() {
+ let root = repo_root();
+ readme_row_pins(&root, "Source records harvested", "envelopes");
+ readme_row_pins(&root, "Records after identity resolution", "records");
+ readme_row_pins(
+ &root,
+ "…merged from both sources",
+ "merged_from_both_sources",
+ );
+ readme_row_pins(
+ &root,
+ "Records with affected-function paths",
+ "records_with_functions",
+ );
+ readme_row_pins(
+ &root,
+ "Records with affected-function paths",
+ "function_paths_distinct",
+ );
+ readme_row_pins(&root, "Records withdrawn", "records_withdrawn");
+ readme_row_pins(&root, "Packages covered", "packages");
+}
+
+#[test]
+fn the_recipe_is_the_one_the_stats_measured() {
+ let root = repo_root();
+ let dir = pack_dir(&root);
+ let recipe = read(&dir.join("recipe.toml"));
+ let mut hasher = sha2::Sha256::new();
+ use sha2::Digest;
+ hasher.update(recipe.as_bytes());
+ let got = format!("{:x}", hasher.finalize());
+ let want = stats(&root)["recipe_sha256"]
+ .as_str()
+ .expect("pack-stats.json pins recipe_sha256")
+ .to_string();
+ assert_eq!(
+ got, want,
+ "recipe.toml changed since pack-stats.json was measured — rebuild the \
+ pack and refresh pack-stats.json (and the README numbers with it)"
+ );
+
+ // and the stats' source table names exactly the recipe's sources
+ for slug in ["rustsec", "osv"] {
+ assert!(
+ recipe.contains(&format!("slug = \"{slug}\"")),
+ "recipe no longer declares source '{slug}'"
+ );
+ assert!(
+ stats(&root)["sources"][slug].is_object(),
+ "pack-stats.json no longer records source '{slug}'"
+ );
+ }
+ // every licence in the recipe is CC0; the README's licence claims must
+ // not silently grow a restricted term
+ assert!(
+ !recipe.contains("CC-BY") && !recipe.contains("BUSL") && !recipe.contains("GPL"),
+ "the pack README says all record data is CC0; a new source with a \
+ different licence must move that claim too"
+ );
+}
diff --git a/tools/packs/rust-vulns/README.md b/tools/packs/rust-vulns/README.md
new file mode 100644
index 000000000..d99ec6aa4
--- /dev/null
+++ b/tools/packs/rust-vulns/README.md
@@ -0,0 +1,93 @@
+# rust-vulns — the showcase corpus pack
+
+Rust vulnerability advisories, **identity-resolved across sources**: the same
+vulnerability that arrives as `RUSTSEC-2021-0003`, `GHSA-43w2-9j62-hq99` and
+`CVE-2021-25900` is one record, carrying every field any source knew — and
+the curated vulnerable-function paths no export carries.
+
+Built with `xerj corpus build` from this directory's
+[recipe.toml](./recipe.toml). Every number below is measured from the actual
+build ([pack-stats.json](./pack-stats.json), `provenance: measured`) and a
+test pins README against stats so the two cannot drift apart.
+
+| What | Count |
+|---|---|
+| Source records harvested (envelopes) | 4107 |
+| Records after identity resolution | 1950 |
+| …merged from both sources | 1221 |
+| …canonical id RUSTSEC / GHSA / MAL | 1221 / 711 / 18 |
+| Records with affected-function paths | 270 (663 distinct paths) |
+| Records withdrawn (kept, flagged `is_withdrawn`) | 124 |
+| Records with a CVE alias | 992 |
+| Packages covered | 1813 |
+| Records JSONL bytes | 10140789 |
+| Cold build (network fetch + harvest + pack) | ~34 s |
+
+## Why this pack exists
+
+Each source alone is incomplete:
+
+- **osv.dev's crates.io export** (CC0, 2,856 records) never merges aliases —
+ the same vulnerability sits there as separate GHSA and RUSTSEC records
+ sharing a CVE — and carries **zero** affected-functions data.
+- **RustSec's advisory-db** (CC0, 1,251 native advisories) is the only
+ source of the hand-curated `affected.functions` paths, CVSS vectors,
+ categories, keywords and full advisory prose — but has no GHSA-only
+ records.
+
+The recipe's union-find over `id` + every `aliases` element closes the gap:
+4,107 envelopes collapse to 1,950 records. Sibling filings union their
+function lists — e.g. one 2026 CVE filed against three `libcrux-*` crates
+becomes one record carrying all three crates' function paths.
+
+GHSA is deliberately **not** a direct source: its crates.io records are
+already inside the osv.dev export under CC0, and the only fields a direct
+`github/advisory-database` clone would add live in `database_specific`,
+which the tool never reads (upstream marks it internal-use-only). HackerOne
+content is unredistributable under its ToS and is out of scope — the pack
+author owns what goes in a pack.
+
+## Build, consume, query
+
+```sh
+# build (or incrementally refresh) — fetches both sources, dedups, packs
+xerj corpus build rust-vulns --recipe tools/packs/rust-vulns/recipe.toml
+# pack lands at $XERJ_CODE_HOME/builds/rust-vulns/pack/rust-vulns/
+
+# consume: install as a reference-coding corpus
+xerj corpus add rust-vulns --from ~/.xerj-code/builds/rust-vulns/pack/rust-vulns
+
+# query it — the envelope targets both ES-compat search and `xerj code`
+xerj code rust-vulns "smallvec insert_many buffer overflow"
+```
+
+The pack ships `suggestions.md` / `mapping.suggested.json` /
+`relations.jsonl` — field-type verdicts and join candidates computed from a
+sample. Suggestions, not decisions: the index infers the real mapping.
+
+## Fields worth knowing
+
+| Field | What it carries |
+|---|---|
+| `id` / `aliases` | canonical advisory id and every alias (CVE/GHSA/RUSTSEC) unioned across sources |
+| `cve_ids` | derived: CVE references extracted from id + aliases |
+| `affected_functions` | RustSec's curated vulnerable-function paths — **unique to this pack** |
+| `defs` | affected_functions joined — feeds `xerj code`'s symbol search |
+| `patched_versions` / `unaffected_versions` | RustSec version requirements |
+| `introduced_versions` / `fixed_versions` | osv.dev range events, unioned |
+| `severity` / `severity_scores` | CVSS vectors |
+| `categories` / `keywords` | RustSec facets |
+| `is_withdrawn` / `informational` | lifecycle flags — withdrawn records stay, flagged |
+| `licence` / `sources` / `origin` | provenance per record |
+
+## Honest limits
+
+- **Freshness is whatever the last build fetched.** Rebuild to refresh;
+ a scheduled build with a freshness badge is the next milestone.
+- **Not a security product.** This is a searchable corpus of advisories for
+ agents and humans; it is not a scanner and makes no completeness claim
+ beyond "what these two sources carried at build time".
+- Withdrawn (124) and informational advisories are kept and flagged, not
+ dropped — filtering is the consumer's decision.
+- v1 ships checksums (SHA256SUMS verifies on `corpus add`); signing lands
+ with the publishing milestone.
diff --git a/tools/packs/rust-vulns/pack-stats.json b/tools/packs/rust-vulns/pack-stats.json
new file mode 100644
index 000000000..c9ba4ddfa
--- /dev/null
+++ b/tools/packs/rust-vulns/pack-stats.json
@@ -0,0 +1,26 @@
+{
+ "provenance": "measured",
+ "measured_at": "2026-09-27",
+ "pack": "rust-vulns",
+ "recipe_sha256": "fe25ecac5ae1921e945fdf9d495f3a88ea81cfa3e613688b6a128af252290349",
+ "sources": {
+ "rustsec": { "kind": "git", "envelopes": 1251, "licence": "CC0-1.0" },
+ "osv": { "kind": "http-zip", "envelopes": 2856, "licence": "CC0-1.0" }
+ },
+ "envelopes": 4107,
+ "records": 1950,
+ "canonical_ids": { "RUSTSEC": 1221, "GHSA": 711, "MAL": 18 },
+ "merged_from_both_sources": 1221,
+ "records_with_functions": 270,
+ "function_paths_distinct": 663,
+ "function_entries_total": 707,
+ "records_withdrawn": 124,
+ "records_with_cve": 992,
+ "packages": 1813,
+ "records_bytes": 10140789,
+ "build_seconds": 34,
+ "notes": {
+ "function_files_vs_records": "275 advisory files carry affected.functions; their paths land in 270 records — 5 advisories are sibling filings of one CVE/GHSA (libcrux x3, hpke-rs x3, wasmtime pairs) and their function lists UNION into the shared cluster, nothing is dropped. 663 paths are distinct across the pack; the same path legitimately appears in 707 record entries when separate advisories hit the same function",
+ "dedup": "4107 envelopes to 1950 records (52.5% collapsed): every RUSTSEC advisory merged with its osv.dev mirror via id, and GHSA/MAL records merged into RUSTSEC clusters via shared CVE aliases"
+ }
+}
diff --git a/tools/packs/rust-vulns/recipe.toml b/tools/packs/rust-vulns/recipe.toml
new file mode 100644
index 000000000..4c11c3459
--- /dev/null
+++ b/tools/packs/rust-vulns/recipe.toml
@@ -0,0 +1,81 @@
+# rust-vulns — the showcase corpus: Rust vulnerability advisories,
+# identity-resolved across sources so the same vulnerability arriving as
+# RUSTSEC-2021-0003, GHSA-43w2-9j62-hq99 and CVE-2021-25900 is ONE record.
+#
+# Sources, and why exactly these two (measured 2026-09-27):
+#
+# - osv.dev's crates.io export (CC0): 2,856 records (1,584 GHSA + 1,252
+# RUSTSEC + 20 MAL) with CVSS scores — but ZERO affected-functions data
+# and a null ecosystem_specific.
+# - RustSec's native advisory-db (CC0): 1,251 markdown advisories — the
+# ONLY source of the curated affected-functions paths (275 files), the
+# cvss vector, categories, keywords, and the full advisory prose.
+#
+# GHSA is deliberately NOT a direct source: its crates.io records are
+# already inside the osv.dev export under CC0, and the only fields a direct
+# github/advisory-database clone would add live in `database_specific`,
+# which the `osv` adapter never reads by policy (upstream marks it
+# internal-use-only, subject to change without notice).
+#
+# HackerOne-derived data is out of scope for this pack: HackerOne report
+# content is unredistributable (its ToS), and the pack author is
+# responsible for what goes in a pack — so this pack ships only
+# redistributable-by-licence sources.
+
+[recipe]
+format = 1
+name = "rust-vulns"
+description = "Rust vulnerability advisories, identity-resolved across osv.dev and RustSec"
+
+[[sources]]
+slug = "rustsec"
+kind = "git"
+url = "https://github.com/rustsec/advisory-db"
+glob = "crates/**/RUSTSEC-*.md"
+format = "rustsec-md"
+licence = "CC0-1.0"
+
+[[sources]]
+slug = "osv"
+kind = "http-zip"
+url = "https://osv-vulnerabilities.storage.googleapis.com/crates.io/all.zip"
+format = "osv"
+licence = "CC0-1.0"
+
+[envelope]
+id_from = ["id"]
+title_from = ["summary", "id"]
+body_join = ["details", "summary"]
+defs_from = ["affected_functions"]
+
+[identity]
+# id: same advisory id across sources. aliases, each: a shared CVE/GHSA
+# alias IS the same vulnerability — union-find closes the chain for free.
+edges = [{ field = "id" }, { field = "aliases", each = true }]
+canonical_source_order = ["rustsec", "osv"]
+
+[merge]
+# rustsec wins where both carry the fact: it owns the curated fields
+# (functions, categories, keywords, prose). osv fills modified/published
+# and the GHSA-sourced records wholesale.
+precedence = ["rustsec", "osv"]
+
+[[derived]]
+name = "cve_ids"
+op = "regex_extract"
+from = ["id", "aliases"]
+pattern = '(?i)\bCVE-\d{4}-\d{4,}\b'
+unique = true
+
+[[derived]]
+name = "is_withdrawn"
+op = "present"
+from = ["withdrawn"]
+
+[[derived]]
+name = "has_functions"
+op = "present"
+from = ["affected_functions"]
+
+[emit]
+shards = 16
diff --git a/tools/xerj-code/SKILL.md b/tools/xerj-code/SKILL.md
index d72f6daf4..e4c315ac4 100644
--- a/tools/xerj-code/SKILL.md
+++ b/tools/xerj-code/SKILL.md
@@ -136,6 +136,7 @@ a fixed setup cost; a task you would finish in two tool calls will not repay it.
```
xerj corpus add <name> <git-url>... # clone reference repos (once per domain)
xerj corpus add --from <manifest> # ...or rebuild a corpus someone else defined
+xerj corpus build <name> --recipe <r> # ...or harvest RECORDS into a portable pack
xerj corpus index <name> # index them with xerj autoindex (once per corpus)
xerj code <name> "<what you need>" # retrieve before writing (every task)
```
@@ -180,6 +181,21 @@ filesystem — a name with `/`, `*`, a leading `.`, or a reserved word is
rejected at parse time, because the pinned-clone checkout runs `git checkout
--force` inside that path.
+Not every corpus is source code. **Record packs** — advisories, datasets, any
+structured data — are built by a recipe instead of cloned:
+
+```sh
+xerj corpus build rust-vulns --recipe tools/packs/rust-vulns/recipe.toml
+xerj corpus add rust-vulns --from ~/.xerj-code/builds/rust-vulns/pack/rust-vulns
+```
+
+The recipe declares sources, identity edges (the same vulnerability arriving as
+RUSTSEC/GHSA/CVE becomes ONE record), merge precedence and derived fields; the
+pack it emits is checksummed and installs with the same `corpus add --from` as
+a corpus definition. The in-repo example is
+[`tools/packs/rust-vulns/`](../packs/rust-vulns/) — see its README for what a
+measured pack looks like.
+
### 2. Index it
```sh