AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
packaging: ship amd64/arm64 .deb release assets; measure the Debian archive gap honestly
xerj-orga16774d8040b
diff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex f66b75042..6f6cc56cf 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -210,6 +210,31 @@ jobs:elseshasum -a 256 "${STAGE}.tar.gz" > "${STAGE}.tar.gz.sha256"fi+ # .deb for the linux-gnu targets (issue #810), packed from the SAME+ # staged binary the tar.gz above just packed — same file, no rebuild,+ # so the two assets cannot drift. musl targets stay tar.gz-only.+ # These legs all run on ubuntu-latest, where dpkg-deb is preinstalled.+ # The control-file version must be a real Debian version string: on a+ # tag that is the tag; on a main-push canary the ref name ("main") is+ # not, so the engine workspace version ships instead while the ASSET+ # keeps the stage name (xerj-main-<target>) to match its tar.gz+ # sibling — build-deb.sh refuses a non-numeric version, which is the+ # guard that keeps that honest.+ case "${{ matrix.target }}" in+ x86_64-unknown-linux-gnu|aarch64-unknown-linux-gnu)+ command -v dpkg-deb >/dev/null 2>&1 \+ || { echo "dpkg-deb not on PATH on ${RUNNER_OS} — cannot build the .deb asset" >&2; exit 1; }+ case "${GITHUB_REF_NAME}" in+ v[0-9]*) DEB_VERSION="${GITHUB_REF_NAME#v}" ;;+ *) DEB_VERSION=$(sed -n 's/^version = "\(.*\)"$/\1/p' ../engine/Cargo.toml | head -1) ;;+ esac+ ../scripts/build-deb.sh \+ --stage-dir "../dist/${STAGE}" \+ --target "${{ matrix.target }}" \+ --version "${DEB_VERSION}" \+ --out-dir ../dist+ ;;+ esac- name: Stage artifacts (Windows)if: matrix.windows == true@@ -235,6 +260,7 @@ jobs:path: |dist/xerj-*.tar.gzdist/xerj-*.zip+ dist/xerj-*.debdist/xerj-*.sha256if-no-files-found: errorretention-days: 7@@ -266,4 +292,5 @@ jobs:files: |dist/*.tar.gzdist/*.zip+ dist/*.debdist/*.sha256diff --git a/AGENTS.md b/AGENTS.mdindex fed49c320..50f3addf0 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -228,6 +228,7 @@ time — is [scripts/seo/README.md](./scripts/seo/README.md).| Pre-submission and review protocol | [docs/CONTRIBUTION_REVIEW.md](./docs/CONTRIBUTION_REVIEW.md) || How to send a bug or a fix back upstream (written for agents) | [.github/AI_CONTRIBUTIONS.md](./.github/AI_CONTRIBUTIONS.md) || Checking a published release is actually good | `scripts/verify-release.sh [tag]` — downloads every asset, asserts every target we ship is present, verifies checksums, asserts each binary reports its tag's version, then boots the host-native binary and runs a real search (Linux and macOS hosts; on a host where it cannot run one it says so and exits non-zero rather than passing) |+| The `.deb` release asset and the Debian-archive question | [docs/PACKAGING_DEBIAN.md](./docs/PACKAGING_DEBIAN.md) · regenerate the dependency-coverage measurement with `scripts/debian-rust-coverage.sh --verbose` |## How to evaluate this project fairlydiff --git a/docs/PACKAGING_DEBIAN.md b/docs/PACKAGING_DEBIAN.mdnew file mode 100644index 000000000..39061369c--- /dev/null+++ b/docs/PACKAGING_DEBIAN.md@@ -0,0 +1,105 @@+# Debian packaging: the release asset, and the archive question++This answers [issue #810](https://github.com/xerj-org/xerj/issues/810). "Packaging+for Debian" is two different problems, and Christian Kastner (Debian Developer)+split them cleanly when we asked:++1. **A `.deb` shipped as a GitHub release asset** — "fairly simple". Done; see below.+2. **Inclusion in the official Debian archive** — requires that every build+ dependency is itself packaged in Debian and rebuildable there. Measured+ below, and on the strength of that measurement: **not claimed, and not being+ pursued**.++## The `.deb` release asset++From the first release cut after this document landed, every GitHub release+carries two extra assets alongside the tar.gz/zip set:++```+xerj-<version>-x86_64-unknown-linux-gnu.deb (Architecture: amd64)+xerj-<version>-aarch64-unknown-linux-gnu.deb (Architecture: arm64)+```++Install with `sudo dpkg -i xerj-<version>-x86_64-unknown-linux-gnu.deb`; the+binary lands in `/usr/bin/xerj` with docs under `/usr/share/doc/xerj`+(README, LICENSE, machine-readable `copyright`). There is no systemd unit and+no `postinst`: xerj is one self-contained binary with no default data+directory, so the package installs the binary and nothing pretends to know+where your data lives. Run it under whatever supervisor you already use.++Properties, all enforced at build time by `scripts/build-deb.sh`:++- **Same bits as the tar.gz.** The `.deb` is packed from the exact staged+ binary the tar.gz for the same target packs — no rebuild, no strip — so the+ two assets cannot diverge. (`cargo-deb` was rejected for exactly this+ reason: it rebuilds.)+- **Dependencies measured, not assumed.** TLS is rustls + ring in every+ release target, so the binary is expected to need nothing beyond glibc. The+ build script reads the ELF `NEEDED` entries, **fails the release build** if+ anything outside the glibc set appears (a `libssl.so.3` here would mean a+ feature regression, not a packaging tweak), and derives the `libc6` floor+ from the newest `GLIBC_*` symbol version the binary actually references.+ Consequence worth knowing before you install on anything old: the floor is+ the CI runner's glibc (at the time of writing the official linux-gnu+ binaries reference symbols up to `GLIBC_2.39`, measured by running+ `scripts/verify-release.sh` on a glibc-2.35 host and reading its failure),+ so the `.deb` will not install-and-run on Debian 12 (bookworm, glibc 2.36)+ or older — this is a property of the binaries we already ship, not of the+ `.deb` format.+- The musl targets stay tar.gz-only (a static-musl `.deb` is unusual and+ nobody asked); macOS and Windows are unaffected.++`scripts/verify-release.sh` checks the `.deb`s like every other asset from the+first tag that ships them: presence for both gnu targets, `.sha256` companion,+`dpkg-deb -x` extraction (`ar` + `tar` on hosts without `dpkg-deb`), binary ++LICENSE + README inside, and the same tag-vs-banner version-drift check that+was added after v1.0.0-rc.10.++## The Debian archive: measured, not claimed++Archive inclusion requires the whole dependency tree to be packaged in Debian.+We measured it rather than guess. Method: unique third-party crate names in+`engine/Cargo.lock`, mapped to Debian source packages (`serde_derive` →+`rust-serde-derive`) and looked up in Debian **unstable**'s `Sources` index.+Presence only — the packaged *versions* were not compared against what+`Cargo.lock` pins, so even the "present" set is a lower bound on the remaining+work.++Measured 2026-09-20, `Cargo.lock` at main (`9c64a7c6`), against+`dists/unstable/main/source/Sources.gz` of the same day:++| | |+|---|---|+| `[[package]]` entries in `Cargo.lock` | 737 |+| workspace `xerj-*` crates (built from this source) | 17 |+| third-party entries | 720 |+| **third-party crate names (unique)** | **654** |+| already packaged in Debian unstable | **506 (77.4%)** |+| **not in Debian unstable** | **148** |++The 720→654 difference is crates pinned at two versions in the tree (53 names).++What the 148 contains: the whole AWS SDK family the object-storage support+pulls in (17 `aws-*`/`aws-smithy-*` crates), `axum-core`/`axum-extra`/+`axum-macros` (Debian carries `rust-axum` itself), Windows/Android shims that+only exist for cross-platform builds, and a long tail of small crates. The+full list prints with:++```sh+scripts/debian-rust-coverage.sh --verbose+```++which re-runs the measurement against the live unstable index (or+`--index <saved Sources.gz>` offline). The join is a single `awk` pass on+purpose: an earlier shell-loop implementation gave run-to-run varying counts+on an idle machine, and a measurement that changes between runs is worse than+none.++**Conclusion.** 148 missing crates — each of which would need its own Debian+packaging and maintenance, several of which (notably `ring`) are famously+nontrivial — plus an unaudited version-sufficiency question on the 506 that+are present. That is a multi-year Debian Rust team effort, not an XERJ+release task. Debian archive inclusion is therefore **not claimed, not+implied, and not pursued**; if the number above changes materially (either+Debian catches up or our dependency tree shrinks), re-run the script and+update this page before saying anything new.diff --git a/scripts/build-deb.sh b/scripts/build-deb.shnew file mode 100755index 000000000..658dd49f6--- /dev/null+++ b/scripts/build-deb.sh@@ -0,0 +1,200 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# build-deb.sh — pack the ALREADY-BUILT release binary as a .deb.+#+# Issue #810: ship a .deb alongside the tar.gz on GitHub releases. This is the+# "fairly simple" half of Christian Kastner's split — a release asset a Debian+# user can `dpkg -i`. It is NOT Debian archive packaging: nothing here makes+# any claim about xerj entering the Debian archive (that audit lives in+# docs/PACKAGING_DEBIAN.md).+#+# Design constraint, on purpose: the .deb packs the SAME file the tar.gz packs+# — the binary from the staging directory release.yml already built — so the+# two assets ship a bit-identical executable. That is why this is a plain+# `dpkg-deb --build` over a hand-written pkg root and NOT cargo-deb: cargo-deb+# would rebuild the crate on its own and the .deb could drift from the tar.gz+# the checksums and verify-release.sh vouch for.+#+# The runtime dependency claim is measured, not assumed. TLS is rustls+ring in+# every release target (engine workspace Cargo.toml), so the binary should need+# nothing beyond glibc — but "should" is not a Depends field. The script reads+# the ELF NEEDED entries, refuses to build if anything outside the glibc set+# appears (libssl.so.3 sneaking in via a feature change would otherwise ship a+# .deb that dies on first boot with a missing library), and derives the libc6+# floor from the newest GLIBC_* symbol version the binary actually references.+#+# Only the *-unknown-linux-gnu targets are packaged (→ amd64, arm64). The musl+# targets stay tar.gz-only: a static-musl .deb is unusual and nobody asked.+#+# Usage (exactly what release.yml runs, runnable locally the same way):+# scripts/build-deb.sh \+# --stage-dir dist/xerj-1.0.0-rc.76-x86_64-unknown-linux-gnu \+# --target x86_64-unknown-linux-gnu \+# --version 1.0.0-rc.76 \+# --out-dir dist+#+# --stage-dir is the directory the "Stage artifacts (Unix)" step already+# created for the tar.gz: it must hold `xerj`, README.md and LICENSE. The .deb+# is named after the stage dir, so on a tag build (stage xerj-<ver>-<target>)+# the asset is xerj-<ver>-<target>.deb. On a main-push canary build the stage+# dir is named xerj-main-<target> while --version must still be a real+# workspace version ("main" is not a valid Debian version string), so the two+# are passed separately.+#+# Outputs: <out-dir>/<stage-name>.deb and <stage-name>.deb.sha256, in the same+# sha256sum format every other asset uses. Prints `dpkg-deb --info` for the+# build log, so a CI run records exactly what it shipped.+#+# Requires: dpkg-deb, binutils (readelf, objdump), sha256sum.+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++# Modes of script-created FILES must not depend on the caller's umask: under a+# 077 umask `copyright` would ship 0600 and be unreadable for non-root. (The+# directories are unaffected either way — dpkg-deb records every directory in+# data.tar as 0700 no matter what is on disk, and dpkg creates the real ones+# 0755 at install time; verified both halves against dpkg-deb 1.21.)+umask 022++STAGE_DIR=""+TARGET=""+VERSION=""+OUT_DIR="."++die() { echo "build-deb.sh: $*" >&2; exit 1; }++while [ $# -gt 0 ]; do+ case "$1" in+ --stage-dir) STAGE_DIR="$2"; shift ;;+ --target) TARGET="$2"; shift ;;+ --version) VERSION="$2"; shift ;;+ --out-dir) OUT_DIR="$2"; shift ;;+ -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "$0"; exit 0 ;;+ *) die "unknown argument: $1" ;;+ esac+ shift+done++[ -n "$STAGE_DIR" ] || die "--stage-dir is required"+[ -n "$TARGET" ] || die "--target is required"+[ -n "$VERSION" ] || die "--version is required"+[ -x "$STAGE_DIR/xerj" ] || die "$STAGE_DIR/xerj is not an executable — stage the binary first"+[ -f "$STAGE_DIR/README.md" ] || die "$STAGE_DIR/README.md missing — the tar.gz stage step provides it"+[ -f "$STAGE_DIR/LICENSE" ] || die "$STAGE_DIR/LICENSE missing — the tar.gz stage step provides it"+command -v dpkg-deb >/dev/null 2>&1 || die "dpkg-deb not on PATH (run on the release runner, or install dpkg)"+for t in readelf objdump sha256sum; do+ command -v "$t" >/dev/null 2>&1 || die "$t not on PATH"+done++# Debian Architecture, mapped from the Rust triple. Deliberately exhaustive+# refuse-list: packaging a new target is a decision, not a default.+case "$TARGET" in+ x86_64-unknown-linux-gnu) DEB_ARCH=amd64 ;;+ aarch64-unknown-linux-gnu) DEB_ARCH=arm64 ;;+ *) die "no Debian architecture mapping for target '$TARGET' (only the *-unknown-linux-gnu targets are packaged)" ;;+esac++# A Debian version string must start with a digit. Tag builds pass+# ${GITHUB_REF_NAME#v}; canary builds must pass the workspace version instead+# of the ref name (which is "main").+case "$VERSION" in+ [0-9]*) : ;;+ *) die "version '$VERSION' is not a valid Debian version (must start with a digit; canary builds pass the workspace version, not the ref name)" ;;+esac+dpkg --compare-versions "$VERSION" eq "$VERSION" 2>/dev/null \+ || die "version '$VERSION' rejected by dpkg --compare-versions"++# ── runtime dependencies, measured off the ELF we are about to pack ──────────+# readelf/objdump parse ELF of any architecture, so this works unchanged on+# the cross-built aarch64 binary.+NEEDED=$(readelf -d "$STAGE_DIR/xerj" | sed -n 's/.*Shared library: \[\(.*\)\]/\1/p' | sort -u || true)+GLIBC_FLOOR=$(objdump -T "$STAGE_DIR/xerj" 2>/dev/null | grep -o 'GLIBC_[0-9][0-9.]*' | sort -Vu | tail -1 || true)+GLIBC_FLOOR="${GLIBC_FLOOR#GLIBC_}"++DEB_DEPENDS="libc6"+if [ -n "$GLIBC_FLOOR" ]; then+ DEB_DEPENDS="libc6 (>= $GLIBC_FLOOR)"+fi++# The allowlist is glibc itself, nothing else. Anything else in NEEDED means+# the binary grew a shared-library dependency the .deb's Depends does not+# express — the honest move is to fail the build, not to guess a mapping.+while IFS= read -r lib; do+ [ -z "$lib" ] && continue+ case "$lib" in+ libc.so.6|libm.so.6|libpthread.so.0|libdl.so.2|librt.so.1|ld-linux-*.so.*)+ # folded into libc6 on any glibc this floor can run on+ ;;+ libgcc_s.so.1)+ DEB_DEPENDS="$DEB_DEPENDS, libgcc-s1" ;;+ *)+ die "binary needs '$lib', which this .deb's Depends does not cover — add it or change the build (expected: glibc only; TLS is rustls, so libssl here would be a regression)" ;;+ esac+done <<< "$NEEDED"++echo "build-deb.sh: NEEDED libraries: ${NEEDED:-<none>}"+echo "build-deb.sh: newest GLIBC symbol version: ${GLIBC_FLOOR:-<none found>}"+echo "build-deb.sh: Depends: $DEB_DEPENDS"++# ── package root ─────────────────────────────────────────────────────────────+STAGE_NAME=$(basename "$STAGE_DIR")+PKG=$(mktemp -d "${TMPDIR:-/tmp}/xerj-deb-XXXXXX")+trap 'rm -rf "$PKG"' EXIT++mkdir -p "$PKG/DEBIAN" "$PKG/usr/bin" "$PKG/usr/share/doc/xerj"++# The exact file the tar.gz ships — same inode content, no strip, no rebuild.+cp "$STAGE_DIR/xerj" "$PKG/usr/bin/xerj"+cp "$STAGE_DIR/README.md" "$PKG/usr/share/doc/xerj/README.md"+cp "$STAGE_DIR/LICENSE" "$PKG/usr/share/doc/xerj/LICENSE"++# Debian machine-readable copyright, pointing at the system copy of the+# Apache-2.0 text instead of pasting it a third time into /usr/share/doc.+cat > "$PKG/usr/share/doc/xerj/copyright" <<EOF+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/+Upstream-Name: xerj+Source: https://github.com/xerj-org/xerj++Files: *+Copyright: xerj Contributors+License: Apache-2.0+ On Debian systems, the complete text of the Apache License 2.0 can be+ found at /usr/share/common-licenses/Apache-2.0.+EOF++INSTALLED_SIZE=$(du -sk "$PKG/usr" | cut -f1)++# Description continuation lines must start with a single space; the package+# description is the one place a Debian user meets before the binary, so it+# claims only what the binary does — search engine, one file, ES-compatible+# REST surface included.+cat > "$PKG/DEBIAN/control" <<EOF+Package: xerj+Version: $VERSION+Architecture: $DEB_ARCH+Maintainer: xerj Contributors <https://github.com/xerj-org/xerj>+Installed-Size: $INSTALLED_SIZE+Depends: $DEB_DEPENDS+Section: net+Priority: optional+Homepage: https://xerj.org+Built-Using: rustc ($TARGET)+Description: single-binary search engine for AI workloads+ XERJ is a search engine written from scratch in Rust that ships as one+ executable: full-text, vector and hybrid search over the same indices,+ built-in agent memory, and log analytics. It also speaks the Elasticsearch+ 8.x REST protocol as a compatibility bridge, so existing ES clients work+ unmodified. This package contains the release binary built from+ https://github.com/xerj-org/xerj at version $VERSION.+EOF++mkdir -p "$OUT_DIR"+DEB="$OUT_DIR/${STAGE_NAME}.deb"++# --root-owner-group: files land root:root without this script being root.+dpkg-deb --root-owner-group --build "$PKG" "$DEB" >/dev/null++( cd "$OUT_DIR" && sha256sum "$(basename "$DEB")" > "$(basename "$DEB").sha256" )++echo "build-deb.sh: built $DEB ($(du -h "$DEB" | cut -f1))"+dpkg-deb --info "$DEB"diff --git a/scripts/debian-rust-coverage.sh b/scripts/debian-rust-coverage.shnew file mode 100755index 000000000..768eb27ab--- /dev/null+++ b/scripts/debian-rust-coverage.sh@@ -0,0 +1,115 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# debian-rust-coverage.sh — how much of xerj's Rust dependency tree is already+# packaged in Debian?+#+# Issue #810's second half. A package enters the Debian ARCHIVE only if every+# build dependency is itself in Debian (Christian Kastner's split: the .deb+# release asset is "fairly simple"; archive inclusion is a function of how+# much of the tree Debian already carries). This script measures that number+# instead of guessing it, and docs/PACKAGING_DEBIAN.md records the result.+#+# Method:+# * third-party crates read from engine/Cargo.lock (workspace xerj-* crates+# excluded — they would be built from this source package);+# * compared against the SOURCE packages of Debian unstable+# (dists/unstable/main/source/Sources.gz), where the Debian Rust team+# packages each crate as rust-<crate> with '_' mapped to '-'+# (serde_derive → rust-serde-derive, per Debian policy on '_' in package+# names);+# * a crate counts as PRESENT if the source package exists. PRESENCE ONLY:+# the packaged version may be older than the one Cargo.lock pins, and this+# script does not compare versions — that deeper audit is what "archive+# inclusion not pursued yet" means in the doc.+#+# Usage:+# scripts/debian-rust-coverage.sh # fetch index, print report+# scripts/debian-rust-coverage.sh --verbose # also list missing crates+# scripts/debian-rust-coverage.sh --index F # reuse a saved Sources.gz+# scripts/debian-rust-coverage.sh --lock F # another Cargo.lock+#+# Requires: curl, gzip. Network unless --index is given.+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)+LOCK="${REPO_ROOT}/engine/Cargo.lock"+INDEX=""+VERBOSE=0+SOURCES_URL="http://deb.debian.org/debian/dists/unstable/main/source/Sources.gz"++while [ $# -gt 0 ]; do+ case "$1" in+ --lock) LOCK="$2"; shift ;;+ --index) INDEX="$2"; shift ;;+ --verbose) VERBOSE=1 ;;+ -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "$0"; exit 0 ;;+ *) echo "unknown argument: $1" >&2; exit 2 ;;+ esac+ shift+done++[ -f "$LOCK" ] || { echo "no such Cargo.lock: $LOCK" >&2; exit 2; }++# Third-party crate names: every [[package]] name that is not a workspace+# xerj-* crate. awk state machine, not grep -A, because the entry body is+# multi-line and the name must be tied to its [[package]] stanza.+CRATES=$(awk '+ /^\[\[package\]\]$/ { in_pkg = 1; next }+ in_pkg && /^name = "/ { gsub(/"/, "", $3); print $3; in_pkg = 0; next }+ /^$/ { in_pkg = 0 }+' "$LOCK" | grep -v '^xerj-' | sort -u)+TOTAL=$(printf '%s\n' "$CRATES" | wc -l | tr -d ' ')++if [ -n "$INDEX" ]; then+ [ -f "$INDEX" ] || { echo "no such index file: $INDEX" >&2; exit 2; }+ DEB_PACKAGES=$(gzip -dc "$INDEX" | sed -n 's/^Package: //p' | sort -u)+else+ command -v curl >/dev/null 2>&1 || { echo "curl is required to fetch $SOURCES_URL" >&2; exit 2; }+ TMPIDX=$(mktemp "${TMPDIR:-/tmp}/xerj-sources-XXXXXX.gz")+ trap 'rm -f "$TMPIDX" "$JOIN" "$JOIN.pkgs" "$JOIN.crates" 2>/dev/null' EXIT+ echo "fetching $SOURCES_URL" >&2+ curl -sSf -m 300 -o "$TMPIDX" "$SOURCES_URL"+ DEB_PACKAGES=$(gzip -dc "$TMPIDX" | sed -n 's/^Package: //p' | sort -u)+fi++[ -n "$DEB_PACKAGES" ] || { echo "no Package: entries parsed from the index — refusing to report an empty denominator" >&2; exit 2; }++# Membership via ONE awk pass over both lists, not a shell loop of per-crate+# greps. That is not a style choice: the loop form was measured giving+# run-to-run varying counts on an idle machine (sporadic short reads through+# 654 sequential `printf | grep` pipelines), and a measurement tool that+# reports a different number each run is worse than no tool. The awk join is+# deterministic and order-independent.+JOIN=$(mktemp "${TMPDIR:-/tmp}/xerj-coverage-XXXXXX")+trap 'rm -f "${TMPIDX:-}" "$JOIN" "$JOIN.pkgs" "$JOIN.crates" 2>/dev/null' EXIT+printf '%s\n' "$DEB_PACKAGES" > "$JOIN.pkgs"+printf '%s\n' "$CRATES" > "$JOIN.crates"+RESULT=$(awk '+ NR == FNR { pkgs[$0] = 1; next }+ {+ name = $0+ gsub(/_/, "-", name)+ if (pkgs["rust-" name]) print "P\t" $0; else print "M\t" $0+ }+' "$JOIN.pkgs" "$JOIN.crates")+PRESENT=$(printf '%s\n' "$RESULT" | grep -c '^P' | tr -d ' ')+MISSING=$(printf '%s\n' "$RESULT" | sed -n 's/^M\t//p')++MISSING_COUNT=$((TOTAL - PRESENT))+PCT=$(awk -v p="$PRESENT" -v t="$TOTAL" 'BEGIN { if (t == 0) print "n/a"; else printf "%.1f", 100 * p / t }')++if [ -n "$INDEX" ]; then+ INDEX_LINE="index: $INDEX (local copy)"+else+ INDEX_LINE="index: $SOURCES_URL"+fi++echo "third-party crates in $LOCK: $TOTAL unique names"+echo "packaged in Debian unstable (rust-*): $PRESENT ($PCT%)"+echo "NOT in Debian unstable: $MISSING_COUNT"+echo "$INDEX_LINE — $(date -u +%Y-%m-%d)"+echo "presence only; packaged versions were NOT compared against Cargo.lock pins"+if [ "$VERBOSE" = 1 ]; then+ printf 'missing: %s\n' "$MISSING" | tr ' ' '\n' | sed '/^$/d' | sed 's/^/ /'+fidiff --git a/scripts/verify-release.sh b/scripts/verify-release.shindex 59b1c4342..8221f88de 100755--- a/scripts/verify-release.sh+++ b/scripts/verify-release.sh@@ -19,11 +19,16 @@# Checks, in order:# 1. every target we ship is present — the set is DECLARED below, not read off# the release page, so a release that lost a matrix leg fails instead of-# quietly verifying the targets that did make it+# quietly verifying the targets that did make it. Since the first tag that+# ships them (FIRST_DEB_TAG below), that includes a .deb for each of the+# two linux-gnu targets.# 2. every archive has a .sha256 companion, and matches it-# 3. every archive contains the binary + LICENSE + README. An archive that-# matched its published checksum and still will not extract is a defect in-# the release, so it FAILS rather than being skipped+# 3. every archive contains the binary + LICENSE + README — .tar.gz/.zip via+# tar/unzip, .deb via dpkg-deb (ar+tar as the fallback a macOS host uses).+# An archive that matched its published checksum and still will not+# extract is a defect in the release, so it FAILS rather than being+# skipped. A .deb with neither dpkg-deb nor ar on the host is a counted+# SKIP, never a silent drop.# 4. every binary — including the ones this host cannot execute — carries the# tag's version in its startup banner, and carries no other version# 5. the host-native binary: --version, boot on a clean data dir, health@@ -45,8 +50,8 @@# scripts/verify-release.sh --keep # keep the download dir# scripts/verify-release.sh --no-smoke # checksums + versions only#-# Requires: gh (authenticated), curl, tar, sha256sum|shasum, and unzip for the-# Windows archives.+# Requires: gh (authenticated), curl, tar, sha256sum|shasum, unzip for the+# Windows archives, and dpkg-deb (or ar) for the .deb assets.# ─────────────────────────────────────────────────────────────────────────────set -euo pipefail@@ -90,6 +95,47 @@ x86_64-unknown-linux-gnux86_64-unknown-linux-musl"TOTAL_TARGETS=$(printf '%s\n' "$EXPECTED_TARGETS" | wc -l | tr -d ' ')+# The two linux-gnu targets additionally ship a .deb (issue #810): the release+# workflow packs one per gnu target from the same staged binary as the tar.gz.+# musl deliberately ships no .deb, and neither do macOS or Windows.+DEB_TARGETS="aarch64-unknown-linux-gnu+x86_64-unknown-linux-gnu"++# The first tag whose release was built by a workflow that ships the .debs.+# Presence of the .debs is only DEMANDED from this tag on: a release cut+# before it legitimately has none, and failing the gate against history would+# make the verifier useless for the tags that predate the asset (the rc.10+# drift check above still runs against any tag). If a release older than this+# constant is re-dispatched with the current workflow it gains .debs, so ANY+# .deb on the release turns the gate on regardless of the tag (see step 2).+# Keep in step with the release.yml change that introduced the .deb assets.+FIRST_DEB_TAG="v1.0.0-rc.76"++# vtag_ge A B — is tag A at or after tag B? Tags here are vMAJOR.MINOR.PATCH+# with an optional -rc.NN suffix; a pre-release sorts before the release it+# precedes (1.0.0-rc.76 < 1.0.0). Pure bash on purpose: this runs on macOS+# hosts too, where sort has no -V and dpkg is absent.+vtag_ge() {+ local a="${1#v}" b="${2#v}"+ [ "$a" = "$b" ] && return 0+ local a_pre="" b_pre=""+ case "$a" in *-*) a_pre="${a#*-}"; a="${a%%-*}" ;; esac+ case "$b" in *-*) b_pre="${b#*-}"; b="${b%%-*}" ;; esac+ local i a_parts b_parts n x y+ IFS='.' read -ra a_parts <<<"$a"; IFS='.' read -ra b_parts <<<"$b"+ n=$(( ${#a_parts[@]} > ${#b_parts[@]} ? ${#a_parts[@]} : ${#b_parts[@]} ))+ for ((i = 0; i < n; i++)); do+ x=$((10#${a_parts[i]:-0})); y=$((10#${b_parts[i]:-0}))+ if [ "$x" -ne "$y" ]; then [ "$x" -gt "$y" ] && return 0 || return 1; fi+ done+ # same mainline: the release outranks its own pre-releases, and a later+ # pre-release outranks an earlier one (rc.2 > rc.1; the suffix is "rc.N" so+ # its trailing number is the discriminator).+ if [ -z "$a_pre" ] && [ -n "$b_pre" ]; then return 0; fi+ if [ -n "$a_pre" ] && [ -z "$b_pre" ]; then return 1; fi+ [ "$((10#${a_pre##*.}))" -ge "$((10#${b_pre##*.}))" ]+}+FAILURES=0# Targets that were present but could not be checked (archive not unpackable on# this host). Counted separately from failures: not a defect in the release, but@@ -158,8 +204,8 @@ cd "$WORKDIR"step "1. download every published asset"gh release download "$TAG" --repo "$REPO" --dir "$WORKDIR" --clobber >/dev/null-ARCHIVES=$(find . -maxdepth 1 -type f \( -name '*.tar.gz' -o -name '*.zip' \) | sed 's|^\./||' | sort)-[ -n "$ARCHIVES" ] || { fail "no .tar.gz or .zip assets on $TAG"; exit 1; }+ARCHIVES=$(find . -maxdepth 1 -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' \) | sed 's|^\./||' | sort)+[ -n "$ARCHIVES" ] || { fail "no .tar.gz, .zip or .deb assets on $TAG"; exit 1; }printf ' %s%s archives%s\n' "$DIM" "$(printf '%s\n' "$ARCHIVES" | wc -l | tr -d ' ')" "$RST"step "2. every target we ship is present"@@ -172,6 +218,22 @@ for t in $EXPECTED_TARGETS; dofail "$t — NO archive published for this target"fidone+# The .deb assets get the same declared-not-discovered treatment, with a time+# gate: they exist from FIRST_DEB_TAG on (and on any release that has one at+# all — a re-dispatched older tag rebuilt by the new workflow). Before that,+# demanding them would fail against every historical tag.+DEB_GATE=0+if vtag_ge "$TAG" "$FIRST_DEB_TAG"; then DEB_GATE=1; fi+if find . -maxdepth 1 -type f -name '*.deb' | grep -q .; then DEB_GATE=1; fi+for t in $DEB_TARGETS; do+ if [ "$DEB_GATE" = 1 ]; then+ if printf '%s\n' "$ARCHIVES" | grep -qx "xerj-$EXPECTED-$t.deb"; then+ pass "$t (.deb)"+ else+ fail "$t — NO .deb published (shipped for every release from $FIRST_DEB_TAG on)"+ fi+ fi+donestep "3. checksums"for a in $ARCHIVES; do@@ -190,7 +252,7 @@ donestep "4. archive contents"for a in $ARCHIVES; do- d="unpack/${a%.tar.gz}"; d="${d%.zip}"+ d="unpack/${a%.tar.gz}"; d="${d%.zip}"; d="${d%.deb}"mkdir -p "$d"# An archive that matched its published checksum and still will not extract is# a defect in the RELEASE, not a gap in this host's tooling — so it is a FAIL,@@ -205,6 +267,30 @@ for a in $ARCHIVES; doUNPACKABLE="$UNPACKABLE $a"continuefi ;;+ *.deb)+ # dpkg-deb on a Debian-ish host; ar + tar as the fallback (macOS has no+ # dpkg-deb but ships ar and a tar that reads the data member directly).+ # Neither tool is a release defect, so a .deb that cannot be opened for+ # want of them is a counted SKIP, not a FAIL.+ if command -v dpkg-deb >/dev/null 2>&1; then+ if ! dpkg-deb -x "$a" "$d"; then+ fail "$a — matched its published checksum but did not extract"+ UNPACKABLE="$UNPACKABLE $a"+ continue+ fi+ elif command -v ar >/dev/null 2>&1; then+ member=$(ar t "$a" 2>/dev/null | grep '^data\.tar\.' | head -1 || true)+ zflag=j; case "$member" in *.xz) zflag=J ;; *.gz) zflag=z ;; *.zst) zflag=--zstd ;; esac+ if [ -z "$member" ] || ! ar p "$a" "$member" | tar -x"$zflag" -C "$d"; then+ fail "$a — matched its published checksum but did not extract"+ UNPACKABLE="$UNPACKABLE $a"+ continue+ fi+ else+ warn "$a — no dpkg-deb or ar on PATH, contents NOT checked"+ SKIPPED_TARGETS=$((SKIPPED_TARGETS + 1))+ continue+ fi ;;*.zip)if ! command -v unzip >/dev/null 2>&1; thenwarn "$a — unzip not on PATH, contents NOT checked"; continue@@ -229,8 +315,8 @@ step "5. version string matches the tag, on every target"# The rc.10 check. A binary whose banner disagrees with its own filename is the# defect; a binary carrying two different versions is a stale-artifact defect.for a in $ARCHIVES; do- d="unpack/${a%.tar.gz}"; d="${d%.zip}"- target=$(printf '%s' "$a" | sed "s/^xerj-$EXPECTED-//; s/\.tar\.gz$//; s/\.zip$//")+ d="unpack/${a%.tar.gz}"; d="${d%.zip}"; d="${d%.deb}"+ target=$(printf '%s' "$a" | sed "s/^xerj-$EXPECTED-//; s/\.tar\.gz$//; s/\.zip$//; s/\.deb$//")bin=$(find "$d" -type f \( -name xerj -o -name xerj.exe \) 2>/dev/null | head -1 || true)if [ -z "$bin" ]; thencase " $UNPACKABLE " in