Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

ci(pack-publish): dated daily releases — immutable-release tags are single-use

xerj-org52cdc9ef41e0
diff --git a/.github/workflows/pack-publish.yml b/.github/workflows/pack-publish.yml
index c35b12da6..c5cbe21c4 100644
--- a/.github/workflows/pack-publish.yml
+++ b/.github/workflows/pack-publish.yml
@@ -122,42 +122,46 @@ jobs:
cp SHA256SUMS "$RUNNER_TEMP/rust-vulns-SHA256SUMS"
cp SHA256SUMS.sig "$RUNNER_TEMP/rust-vulns-SHA256SUMS.sig"
- # This repository has immutable releases: assets may only be uploaded
- # BEFORE a release is published. Run 36355121167 died uploading the
- # signature — "Cannot upload asset … to an immutable release" — because
- # the action created the release already published. Deleting a release
- # (and its tag) is still allowed — verified by hand against the
- # half-published first attempt. So a rolling release under one URL is
- # three moves:
- # 1. drop the previous release and tag (first-ever run: nothing to
- # delete, `|| true`),
- # 2. recreate it as a DRAFT and upload the assets to the draft,
+ # This repository has immutable releases, which make release tags
+ # SINGLE-USE: assets may only be uploaded before a release is published,
+ # and a tag that has named an immutable release can never name another —
+ # run 36358399114 got "Cannot create ref … / tag_name was used by an
+ # immutable release" trying to reuse the deleted first attempt's tag,
+ # and 36355121167 got "Cannot upload asset … to an immutable release"
+ # trying to upload after publish. So each build is its own dated
+ # release, published the way GitHub's error message prescribes:
+ # 1. compute pack-rust-vulns-YYYY-MM-DD (a tag never used before;
+ # a same-day re-run deletes and recreates today's release first),
+ # 2. create it as a DRAFT and upload the assets to the draft,
# 3. publish the draft (`release.published` fires).
- # Consumers keep the one stable set of URLs already printed in the pack
- # README, docs/CORPUS_PACKS.md and the blog post; the seconds between
- # (1) and (3) are the only window they 404. Yesterday's build stays
- # exactly what was downloaded from it — the signature pins the bytes.
- - name: Drop the previous rolling release
+ # Yesterday's release is never touched: under immutable releases it is
+ # exactly the bytes that shipped, and the signature over SHA256SUMS
+ # pins them. A best-effort retention step keeps only the newest 7.
+ - name: Compute the daily tag
+ run: echo "PACK_TAG=pack-rust-vulns-$(date -u +%F)" >> "$GITHUB_ENV"
+
+ - name: Drop today's release if this is a same-day re-run
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: >
- gh release delete pack-rust-vulns --repo "$GITHUB_REPOSITORY"
+ gh release delete "$PACK_TAG" --repo "$GITHUB_REPOSITORY"
--cleanup-tag --yes || true
- name: Upload the assets to a draft release
uses: softprops/action-gh-release@v2
with:
- tag_name: pack-rust-vulns
- name: "rust-vulns corpus pack (rolling)"
+ tag_name: ${{ env.PACK_TAG }}
+ name: "rust-vulns corpus pack (${{ env.PACK_TAG }})"
draft: true
prerelease: true
body: |
The `rust-vulns` corpus pack, built and signed by the scheduled
`pack publish` workflow from `tools/packs/rust-vulns/recipe.toml`.
- Each scheduled run deletes and recreates this release, so these
- URLs always carry the newest signed build; build history lives in
- the workflow runs.
+ One release per build day; the newest is always at
+ https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns
+ Build history lives in the releases themselves and in the
+ workflow runs.
Install (public key: `tools/packs/keys/rust-vulns.pub` in the repo):
@@ -166,7 +170,8 @@ jobs:
xerj corpus index rust-vulns
`rust-vulns-freshness.json` carries the build date and record
- count; the badge in `tools/packs/rust-vulns/README.md` reads it.
+ count; the same 30-day staleness refusal that governs every
+ reference corpus governs this one.
files: |
${{ runner.temp }}/rust-vulns-pack.zip
${{ runner.temp }}/rust-vulns-SHA256SUMS
@@ -179,5 +184,23 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: >
- gh release edit pack-rust-vulns --repo "$GITHUB_REPOSITORY"
+ gh release edit "$PACK_TAG" --repo "$GITHUB_REPOSITORY"
--draft=false
+
+ # Immutable releases allow deletion (verified by hand); tags sort
+ # lexicographically = chronologically. Best effort only — a retention
+ # failure must never cost the daily publish.
+ - name: Retire pack releases beyond the newest 7
+ continue-on-error: true
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ mapfile -t old < <(gh release list --repo "$GITHUB_REPOSITORY" \
+ --limit 100 --json tagName,isDraft \
+ --jq '[.[] | select(.tagName | startswith("pack-rust-vulns-"))
+ | select(.isDraft | not) | .tagName]
+ | sort | reverse | .[7:][]')
+ for tag in "${old[@]:-}"; do
+ [ -n "$tag" ] && gh release delete "$tag" \
+ --repo "$GITHUB_REPOSITORY" --cleanup-tag --yes
+ done
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6761fb69..101fed558 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -80,7 +80,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
over SHA256SUMS (the public key travels beside the recipe, never inside the
pack), `corpus add --verify-sig <pubkey>` checks origin *before* anything is
materialized, and a scheduled workflow rebuilds, signs, and publishes the
- pack to a rolling GitHub Release — verifying its own output against the
+ pack to a dated GitHub Release per build day (immutable releases make
+ release tags single-use, so there is no one rolling URL) — verifying its
+ own output against the
committed `.pub` first, so a half-rotated key fails the build. The attack
the signature exists for is pinned by a test: a self-consistent rebuild
(tampered records, honestly rewritten checksums) passes every checksum and
diff --git a/ROADMAP.md b/ROADMAP.md
index 227735bab..a0270b283 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -139,7 +139,7 @@ The 1.0 bar: **every public claim verified against the release binary, and every
- **A real object-storage backend.** *Done in rc.77:* [#965](https://github.com/xerj-org/xerj/issues/965) wired the segment path to object storage — `storage.backend = "s3"` packs each segment family into one immutable ZBM1 bundle object with `snapshot.json` as the publication point, merges publish before retiring inputs, and a fresh node adopts the bucket (see *Shipping today* for the full statement). The client (`s3.rs`, real S3-compatible: Cloudflare R2, MinIO, AWS S3), the read-through segment cache, per-request cost accounting by billing class and the stop-don't-warn budget had landed in rc.75. Separately, `xerj autoindex s3://bucket/prefix` reads documents OUT of a bucket, which is a source, not a home.
- **A block index mode for logs.** An index mode for log-shaped data that stores rows in time-partitioned columnar blocks and skips whole blocks by their min/max metadata, instead of building a per-term inverted index over every field. *Today:* the `xerj-logs` crate implements that design (columnar encoding, log-template extraction, time-range queries with block skipping, retention), is compiled into `xerj-engine` and `xerj-server` as a dependency, and is **called from no non-test code**. Log-shaped analytics run through the general columnar segment format and the aggregation suite. Wire it behind an explicit index setting and measure it against the general path on the same data, or remove it — the *Log-analytics data path* theme below is this same item.
- **User-code ingest plugins.** *Today:* the ingest pipeline's transforms are built-in native Rust plugins — rename, drop, add, JSON parse, timestamp parse, PII redaction, grok, route — and they do run on `_bulk`. The crate is named `xerj-wasm`, but **the wasmtime backend is not in the tree**: there is no `wasmtime` dependency and no `wasm` feature, only a note that one could be added behind the same trait. Planned: a sandboxed runtime for user-supplied transforms with fuel and memory limits and no ambient filesystem or network access. Until then the refusal rule holds: a pipeline naming a processor this build does not implement is stored as unrunnable and every ingest through it is refused, never quietly run as a shorter pipeline.
-- **A corpus hub of signed, pre-indexed packs.** Download a reference corpus — a standard library, a specification set — already indexed, and mount it, instead of every user spending the same CPU-hours indexing the same public text. *Today:* the **records half is built and signed** — `xerj corpus build` (PR [#1046](https://github.com/xerj-org/xerj/pull/1046)) turns a declarative recipe (sources with licences, identity edges, merge precedence, derived fields) into a checksummed portable pack of *records* with a `format_version` readers refuse when unknown, per-file checksums verified on `corpus add --from <pack>`, and licence + provenance on every record; the first real pack is `rust-vulns` (`tools/packs/rust-vulns/`, identity-resolved across osv.dev and RustSec, with a test pinning its README numbers to a measured build), published by a scheduled workflow as a rolling GitHub Release with a **detached ed25519 signature** over SHA256SUMS (`xerj corpus keygen`/`sign`; `corpus add --verify-sig <pubkey>` checks origin before anything is indexed — the public key travels beside the recipe, never inside the pack; nobody in the prior-art survey signs their database). Still unbuilt: the **hub itself** (a directory of packs beyond this first one) and the **pre-indexed** half — [#1030](https://github.com/xerj-org/xerj/issues/1030) remains the tracker for the indexed-segment bundle that is the format this item would distribute, since indexing a records pack still costs each consumer the same CPU-hours. Three risks decide whether the rest ships: **redistribution licence** (an index is a derived copy of its source text — a pack may carry only what its licence lets us redistribute, which excludes much of what people most want indexed; the builder carries licence as record data and the pack author owns the choice); **pack safety** (a pack is untrusted input to the segment readers, so it needs the fuzzing the on-disk formats get, and a signature proves who built a pack, not that it is safe to open); and **format stability** (a published pack outlives the release that built it, so the pack format needs a compatibility promise the internal segment format has never had to make).
+- **A corpus hub of signed, pre-indexed packs.** Download a reference corpus — a standard library, a specification set — already indexed, and mount it, instead of every user spending the same CPU-hours indexing the same public text. *Today:* the **records half is built and signed** — `xerj corpus build` (PR [#1046](https://github.com/xerj-org/xerj/pull/1046)) turns a declarative recipe (sources with licences, identity edges, merge precedence, derived fields) into a checksummed portable pack of *records* with a `format_version` readers refuse when unknown, per-file checksums verified on `corpus add --from <pack>`, and licence + provenance on every record; the first real pack is `rust-vulns` (`tools/packs/rust-vulns/`, identity-resolved across osv.dev and RustSec, with a test pinning its README numbers to a measured build), published by a scheduled workflow as a daily dated GitHub Release (immutable releases make tags single-use) with a **detached ed25519 signature** over SHA256SUMS (`xerj corpus keygen`/`sign`; `corpus add --verify-sig <pubkey>` checks origin before anything is indexed — the public key travels beside the recipe, never inside the pack; nobody in the prior-art survey signs their database). Still unbuilt: the **hub itself** (a directory of packs beyond this first one) and the **pre-indexed** half — [#1030](https://github.com/xerj-org/xerj/issues/1030) remains the tracker for the indexed-segment bundle that is the format this item would distribute, since indexing a records pack still costs each consumer the same CPU-hours. Three risks decide whether the rest ships: **redistribution licence** (an index is a derived copy of its source text — a pack may carry only what its licence lets us redistribute, which excludes much of what people most want indexed; the builder carries licence as record data and the pack author owns the choice); **pack safety** (a pack is untrusted input to the segment readers, so it needs the fuzzing the on-disk formats get, and a signature proves who built a pack, not that it is safe to open); and **format stability** (a published pack outlives the release that built it, so the pack format needs a compatibility promise the internal segment format has never had to make).
## Beyond 1.0 — themes
diff --git a/docs/CORPUS_PACKS.md b/docs/CORPUS_PACKS.md
index 8a6638311..9dbf9c598 100644
--- a/docs/CORPUS_PACKS.md
+++ b/docs/CORPUS_PACKS.md
@@ -153,8 +153,13 @@ procedure is in
The [pack publish workflow](../.github/workflows/pack-publish.yml) rebuilds
and signs `rust-vulns` daily and attaches the zip, the loose checksums, the
-signature, and a freshness badge file to a rolling GitHub Release
-([pack-rust-vulns](https://github.com/xerj-org/xerj/releases/tag/pack-rust-vulns)).
+signature, and a freshness file to a GitHub Release per build day — dated
+tags `pack-rust-vulns-YYYY-MM-DD`, newest first at
+[releases?q=pack-rust-vulns](https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns).
+This repository has immutable releases (release tags are single-use), which
+is why the pack ships one release per day instead of one rolling URL —
+each day's release is exactly the bytes that shipped, pinned by its
+signature, and the workflow keeps only the newest 7.
Freshness is an operational promise the schedule keeps, not a README claim:
the pack's own 30-day staleness refusal applies to our published corpus
exactly as it does to a user's reference corpora.
diff --git a/landing/blog/index.html b/landing/blog/index.html
index 8eecb0dcb..def572e40 100644
--- a/landing/blog/index.html
+++ b/landing/blog/index.html
@@ -155,7 +155,7 @@
<span class="blp-num">ed25519 over SHA256SUMS <span class="blp-chip">prior art: 0 of 4</span></span><span class="blp-bar a" style="width:100%"></span>
</div>
</div>
- <p class="blp-foot">daily rebuild · rolling release pack-rust-vulns · cold build ~34 s</p>
+ <p class="blp-foot">daily rebuild · signed releases pack-rust-vulns-* · cold build ~34 s</p>
</div>
<div class="bl-meta">
<p class="bl-date"><time datetime="2026-09-27">2026-09-27</time> · CORPUS PACKS · SECURITY DATA</p>
diff --git a/landing/blog/one-vulnerability-three-names.html b/landing/blog/one-vulnerability-three-names.html
index 9b940782f..2a7373b7c 100644
--- a/landing/blog/one-vulnerability-three-names.html
+++ b/landing/blog/one-vulnerability-three-names.html
@@ -309,10 +309,10 @@ canonical_source_order = ["rustsec", "osv"]</pre>
not a scanner and will not tell you what is running in your production.</p>
<p><strong>Freshness is a cron job, not a promise.</strong> The publish workflow rebuilds daily;
- between a source updating and the next run, the pack lags by up to a day. The README badge reads
- the build date off the release itself, and the same 30-day staleness refusal that governs every
- reference corpus governs ours — a workflow that stops running shows up as a stale pack, not a
- silently aging artifact.</p>
+ between a source updating and the next run, the pack lags by up to a day. Each release carries
+ its build date and record count as a freshness file, and the same 30-day staleness refusal that
+ governs every reference corpus governs ours — a workflow that stops running shows up as a stale
+ pack, not a silently aging artifact.</p>
<p><strong>Records, not an index.</strong> Installing the pack still costs every consumer the
same CPU-minutes to index — the pre-indexed half of the corpus-hub idea is a separate, harder
@@ -340,10 +340,13 @@ canonical_source_order = ["rustsec", "osv"]</pre>
documented end to end in
<a href="https://github.com/xerj-org/xerj/blob/main/docs/CORPUS_PACKS.md" rel="external">docs/CORPUS_PACKS.md</a>.</p>
- <p>The pack itself is a rolling GitHub Release —
- <a href="https://github.com/xerj-org/xerj/releases/tag/pack-rust-vulns" rel="external">pack-rust-vulns</a>
+ <p>The pack itself ships as a GitHub Release per build day — dated
+ <code class="inline">pack-rust-vulns-YYYY-MM-DD</code> tags, newest first at
+ <a href="https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns" rel="external">releases?q=pack-rust-vulns</a>
— rebuilt and signed on a schedule, with the zip, the loose checksums, the signature, and a
- freshness file as assets. Build it yourself instead, from the recipe, if you would rather trust
+ freshness file as assets. (Our repository has immutable releases — release tags are single-use —
+ so there is no single rolling URL; each day's release is exactly the bytes that shipped, pinned
+ by its signature.) Build it yourself instead, from the recipe, if you would rather trust
your own machine than our key; that is a supported path, not a workaround. If your agent works
with Rust dependencies, point it at the pack and ask it the question a raw export cannot
answer: <em>which function is the vulnerable one?</em></p>
diff --git a/tools/packs/rust-vulns/README.md b/tools/packs/rust-vulns/README.md
index 428f798fd..a5856288c 100644
--- a/tools/packs/rust-vulns/README.md
+++ b/tools/packs/rust-vulns/README.md
@@ -1,7 +1,7 @@
# rust-vulns — the showcase corpus pack
[![pack publish](https://github.com/xerj-org/xerj/actions/workflows/pack-publish.yml/badge.svg)](https://github.com/xerj-org/xerj/actions/workflows/pack-publish.yml)
-[![rust-vulns freshness](https://img.shields.io/endpoint?url=https://github.com/xerj-org/xerj/releases/download/pack-rust-vulns/rust-vulns-freshness.json)](https://github.com/xerj-org/xerj/releases/tag/pack-rust-vulns)
+[![rust-vulns pack: signed · daily builds](https://img.shields.io/badge/rust--vulns%20pack-signed%20%C2%B7%20daily%20builds-blue)](https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns)
Rust vulnerability advisories, **identity-resolved across sources**: the same
vulnerability that arrives as `RUSTSEC-2021-0003`, `GHSA-43w2-9j62-hq99` and
@@ -52,18 +52,25 @@ author owns what goes in a pack.
## Build, consume, query
-The published pack is a rolling GitHub Release
-([pack-rust-vulns](https://github.com/xerj-org/xerj/releases/tag/pack-rust-vulns)),
-rebuilt and signed daily by the
+The published pack is a GitHub Release per build day (dated tags
+`pack-rust-vulns-YYYY-MM-DD`, newest first:
+[releases?q=pack-rust-vulns](https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns)),
+built and signed daily by the
[pack publish](https://github.com/xerj-org/xerj/actions/workflows/pack-publish.yml)
-workflow.
+workflow. This repository has immutable releases — release tags are
+single-use and yesterday's release is never modified — so there is no
+one-rolling-URL; each day's release is exactly the bytes that shipped,
+pinned by its signature.
```sh
# consume the published, signed pack:
-# 1. fetch the zip and the public key (the key travels OUT of band —
-# next to the recipe in this repo, never inside the pack)
-curl -LO https://github.com/xerj-org/xerj/releases/download/pack-rust-vulns/rust-vulns-pack.zip
-curl -LO https://github.com/xerj-org/xerj/releases/download/pack-rust-vulns/rust-vulns-SHA256SUMS.sig
+# 1. fetch the zip and signature from the newest pack release, and the
+# public key (the key travels OUT of band — next to the recipe in
+# this repo, never inside the pack)
+gh release download -R xerj-org/xerj \
+ -p 'rust-vulns-pack.zip' -p 'rust-vulns-SHA256SUMS.sig' \
+ "$(gh release list -R xerj-org/xerj -L 100 --json tagName \
+ --jq '[.[].tagName | select(startswith("pack-rust-vulns-"))] | .[0]')"
curl -LO https://raw.githubusercontent.com/xerj-org/xerj/main/tools/packs/keys/rust-vulns.pub
# 2. install it with signature verification — refuses the pack whole if
@@ -131,9 +138,10 @@ the new one.
## Honest limits
- **Freshness is whatever the last build fetched.** The scheduled workflow
- rebuilds daily; the badge above shows the last build's date and record
- count. Between the sources updating and the next cron tick, the pack lags
- by up to a day — a rebuild you run yourself is always current.
+ rebuilds daily; the newest release's `rust-vulns-freshness.json` carries
+ its build date and record count. Between the sources updating and the next
+ cron tick, the pack lags by up to a day — a rebuild you run yourself is
+ always current.
- **Not a security product.** This is a searchable corpus of advisories for
agents and humans; it is not a scanner and makes no completeness claim
beyond "what these two sources carried at build time".
diff --git a/tools/xerj-code/SKILL.md b/tools/xerj-code/SKILL.md
index e5db9b4bc..820f075cc 100644
--- a/tools/xerj-code/SKILL.md
+++ b/tools/xerj-code/SKILL.md
@@ -192,8 +192,8 @@ xerj corpus add rust-vulns --from ~/.xerj-code/builds/rust-vulns/pack/rust-vulns
The recipe declares sources, identity edges (the same vulnerability arriving as
RUSTSEC/GHSA/CVE becomes ONE record), merge precedence and derived fields; the
pack it emits is checksummed and installs with the same `corpus add --from` as
-a corpus definition. A pack from someone else (e.g. the rolling
-[rust-vulns release](https://github.com/xerj-org/xerj/releases/tag/pack-rust-vulns))
+a corpus definition. A pack from someone else (e.g. the daily
+[rust-vulns releases](https://github.com/xerj-org/xerj/releases?q=pack-rust-vulns))
carries an ed25519 signature over its SHA256SUMS — add it with
`--verify-sig <pubkey-file>` to check origin before anything is indexed
(the key ships beside the recipe, never inside the pack). The in-repo example