Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

packaging: ship amd64/arm64 .deb release assets; measure the Debian archive gap honestly

xerj-orga16774d8040b
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index f66b75042..6f6cc56cf 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -210,6 +210,31 @@ jobs:
else
shasum -a 256 "${STAGE}.tar.gz" > "${STAGE}.tar.gz.sha256"
fi
+ # .deb for the linux-gnu targets (issue #810), packed from the SAME
+ # staged binary the tar.gz above just packed — same file, no rebuild,
+ # so the two assets cannot drift. musl targets stay tar.gz-only.
+ # These legs all run on ubuntu-latest, where dpkg-deb is preinstalled.
+ # The control-file version must be a real Debian version string: on a
+ # tag that is the tag; on a main-push canary the ref name ("main") is
+ # not, so the engine workspace version ships instead while the ASSET
+ # keeps the stage name (xerj-main-<target>) to match its tar.gz
+ # sibling — build-deb.sh refuses a non-numeric version, which is the
+ # guard that keeps that honest.
+ case "${{ matrix.target }}" in
+ x86_64-unknown-linux-gnu|aarch64-unknown-linux-gnu)
+ command -v dpkg-deb >/dev/null 2>&1 \
+ || { echo "dpkg-deb not on PATH on ${RUNNER_OS} — cannot build the .deb asset" >&2; exit 1; }
+ case "${GITHUB_REF_NAME}" in
+ v[0-9]*) DEB_VERSION="${GITHUB_REF_NAME#v}" ;;
+ *) DEB_VERSION=$(sed -n 's/^version = "\(.*\)"$/\1/p' ../engine/Cargo.toml | head -1) ;;
+ esac
+ ../scripts/build-deb.sh \
+ --stage-dir "../dist/${STAGE}" \
+ --target "${{ matrix.target }}" \
+ --version "${DEB_VERSION}" \
+ --out-dir ../dist
+ ;;
+ esac
- name: Stage artifacts (Windows)
if: matrix.windows == true
@@ -235,6 +260,7 @@ jobs:
path: |
dist/xerj-*.tar.gz
dist/xerj-*.zip
+ dist/xerj-*.deb
dist/xerj-*.sha256
if-no-files-found: error
retention-days: 7
@@ -266,4 +292,5 @@ jobs:
files: |
dist/*.tar.gz
dist/*.zip
+ dist/*.deb
dist/*.sha256
diff --git a/AGENTS.md b/AGENTS.md
index fed49c320..50f3addf0 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -228,6 +228,7 @@ time — is [scripts/seo/README.md](./scripts/seo/README.md).
| Pre-submission and review protocol | [docs/CONTRIBUTION_REVIEW.md](./docs/CONTRIBUTION_REVIEW.md) |
| How to send a bug or a fix back upstream (written for agents) | [.github/AI_CONTRIBUTIONS.md](./.github/AI_CONTRIBUTIONS.md) |
| Checking a published release is actually good | `scripts/verify-release.sh [tag]` — downloads every asset, asserts every target we ship is present, verifies checksums, asserts each binary reports its tag's version, then boots the host-native binary and runs a real search (Linux and macOS hosts; on a host where it cannot run one it says so and exits non-zero rather than passing) |
+| The `.deb` release asset and the Debian-archive question | [docs/PACKAGING_DEBIAN.md](./docs/PACKAGING_DEBIAN.md) · regenerate the dependency-coverage measurement with `scripts/debian-rust-coverage.sh --verbose` |
## How to evaluate this project fairly
diff --git a/docs/PACKAGING_DEBIAN.md b/docs/PACKAGING_DEBIAN.md
new file mode 100644
index 000000000..39061369c
--- /dev/null
+++ b/docs/PACKAGING_DEBIAN.md
@@ -0,0 +1,105 @@
+# Debian packaging: the release asset, and the archive question
+
+This answers [issue #810](https://github.com/xerj-org/xerj/issues/810). "Packaging
+for Debian" is two different problems, and Christian Kastner (Debian Developer)
+split them cleanly when we asked:
+
+1. **A `.deb` shipped as a GitHub release asset** — "fairly simple". Done; see below.
+2. **Inclusion in the official Debian archive** — requires that every build
+ dependency is itself packaged in Debian and rebuildable there. Measured
+ below, and on the strength of that measurement: **not claimed, and not being
+ pursued**.
+
+## The `.deb` release asset
+
+From the first release cut after this document landed, every GitHub release
+carries two extra assets alongside the tar.gz/zip set:
+
+```
+xerj-<version>-x86_64-unknown-linux-gnu.deb (Architecture: amd64)
+xerj-<version>-aarch64-unknown-linux-gnu.deb (Architecture: arm64)
+```
+
+Install with `sudo dpkg -i xerj-<version>-x86_64-unknown-linux-gnu.deb`; the
+binary lands in `/usr/bin/xerj` with docs under `/usr/share/doc/xerj`
+(README, LICENSE, machine-readable `copyright`). There is no systemd unit and
+no `postinst`: xerj is one self-contained binary with no default data
+directory, so the package installs the binary and nothing pretends to know
+where your data lives. Run it under whatever supervisor you already use.
+
+Properties, all enforced at build time by `scripts/build-deb.sh`:
+
+- **Same bits as the tar.gz.** The `.deb` is packed from the exact staged
+ binary the tar.gz for the same target packs — no rebuild, no strip — so the
+ two assets cannot diverge. (`cargo-deb` was rejected for exactly this
+ reason: it rebuilds.)
+- **Dependencies measured, not assumed.** TLS is rustls + ring in every
+ release target, so the binary is expected to need nothing beyond glibc. The
+ build script reads the ELF `NEEDED` entries, **fails the release build** if
+ anything outside the glibc set appears (a `libssl.so.3` here would mean a
+ feature regression, not a packaging tweak), and derives the `libc6` floor
+ from the newest `GLIBC_*` symbol version the binary actually references.
+ Consequence worth knowing before you install on anything old: the floor is
+ the CI runner's glibc (at the time of writing the official linux-gnu
+ binaries reference symbols up to `GLIBC_2.39`, measured by running
+ `scripts/verify-release.sh` on a glibc-2.35 host and reading its failure),
+ so the `.deb` will not install-and-run on Debian 12 (bookworm, glibc 2.36)
+ or older — this is a property of the binaries we already ship, not of the
+ `.deb` format.
+- The musl targets stay tar.gz-only (a static-musl `.deb` is unusual and
+ nobody asked); macOS and Windows are unaffected.
+
+`scripts/verify-release.sh` checks the `.deb`s like every other asset from the
+first tag that ships them: presence for both gnu targets, `.sha256` companion,
+`dpkg-deb -x` extraction (`ar` + `tar` on hosts without `dpkg-deb`), binary +
+LICENSE + README inside, and the same tag-vs-banner version-drift check that
+was added after v1.0.0-rc.10.
+
+## The Debian archive: measured, not claimed
+
+Archive inclusion requires the whole dependency tree to be packaged in Debian.
+We measured it rather than guess. Method: unique third-party crate names in
+`engine/Cargo.lock`, mapped to Debian source packages (`serde_derive` →
+`rust-serde-derive`) and looked up in Debian **unstable**'s `Sources` index.
+Presence only — the packaged *versions* were not compared against what
+`Cargo.lock` pins, so even the "present" set is a lower bound on the remaining
+work.
+
+Measured 2026-09-20, `Cargo.lock` at main (`9c64a7c6`), against
+`dists/unstable/main/source/Sources.gz` of the same day:
+
+| | |
+|---|---|
+| `[[package]]` entries in `Cargo.lock` | 737 |
+| workspace `xerj-*` crates (built from this source) | 17 |
+| third-party entries | 720 |
+| **third-party crate names (unique)** | **654** |
+| already packaged in Debian unstable | **506 (77.4%)** |
+| **not in Debian unstable** | **148** |
+
+The 720→654 difference is crates pinned at two versions in the tree (53 names).
+
+What the 148 contains: the whole AWS SDK family the object-storage support
+pulls in (17 `aws-*`/`aws-smithy-*` crates), `axum-core`/`axum-extra`/
+`axum-macros` (Debian carries `rust-axum` itself), Windows/Android shims that
+only exist for cross-platform builds, and a long tail of small crates. The
+full list prints with:
+
+```sh
+scripts/debian-rust-coverage.sh --verbose
+```
+
+which re-runs the measurement against the live unstable index (or
+`--index <saved Sources.gz>` offline). The join is a single `awk` pass on
+purpose: an earlier shell-loop implementation gave run-to-run varying counts
+on an idle machine, and a measurement that changes between runs is worse than
+none.
+
+**Conclusion.** 148 missing crates — each of which would need its own Debian
+packaging and maintenance, several of which (notably `ring`) are famously
+nontrivial — plus an unaudited version-sufficiency question on the 506 that
+are present. That is a multi-year Debian Rust team effort, not an XERJ
+release task. Debian archive inclusion is therefore **not claimed, not
+implied, and not pursued**; if the number above changes materially (either
+Debian catches up or our dependency tree shrinks), re-run the script and
+update this page before saying anything new.
diff --git a/scripts/build-deb.sh b/scripts/build-deb.sh
new file mode 100755
index 000000000..658dd49f6
--- /dev/null
+++ b/scripts/build-deb.sh
@@ -0,0 +1,200 @@
+#!/usr/bin/env bash
+# ─────────────────────────────────────────────────────────────────────────────
+# build-deb.sh — pack the ALREADY-BUILT release binary as a .deb.
+#
+# Issue #810: ship a .deb alongside the tar.gz on GitHub releases. This is the
+# "fairly simple" half of Christian Kastner's split — a release asset a Debian
+# user can `dpkg -i`. It is NOT Debian archive packaging: nothing here makes
+# any claim about xerj entering the Debian archive (that audit lives in
+# docs/PACKAGING_DEBIAN.md).
+#
+# Design constraint, on purpose: the .deb packs the SAME file the tar.gz packs
+# — the binary from the staging directory release.yml already built — so the
+# two assets ship a bit-identical executable. That is why this is a plain
+# `dpkg-deb --build` over a hand-written pkg root and NOT cargo-deb: cargo-deb
+# would rebuild the crate on its own and the .deb could drift from the tar.gz
+# the checksums and verify-release.sh vouch for.
+#
+# The runtime dependency claim is measured, not assumed. TLS is rustls+ring in
+# every release target (engine workspace Cargo.toml), so the binary should need
+# nothing beyond glibc — but "should" is not a Depends field. The script reads
+# the ELF NEEDED entries, refuses to build if anything outside the glibc set
+# appears (libssl.so.3 sneaking in via a feature change would otherwise ship a
+# .deb that dies on first boot with a missing library), and derives the libc6
+# floor from the newest GLIBC_* symbol version the binary actually references.
+#
+# Only the *-unknown-linux-gnu targets are packaged (→ amd64, arm64). The musl
+# targets stay tar.gz-only: a static-musl .deb is unusual and nobody asked.
+#
+# Usage (exactly what release.yml runs, runnable locally the same way):
+# scripts/build-deb.sh \
+# --stage-dir dist/xerj-1.0.0-rc.76-x86_64-unknown-linux-gnu \
+# --target x86_64-unknown-linux-gnu \
+# --version 1.0.0-rc.76 \
+# --out-dir dist
+#
+# --stage-dir is the directory the "Stage artifacts (Unix)" step already
+# created for the tar.gz: it must hold `xerj`, README.md and LICENSE. The .deb
+# is named after the stage dir, so on a tag build (stage xerj-<ver>-<target>)
+# the asset is xerj-<ver>-<target>.deb. On a main-push canary build the stage
+# dir is named xerj-main-<target> while --version must still be a real
+# workspace version ("main" is not a valid Debian version string), so the two
+# are passed separately.
+#
+# Outputs: <out-dir>/<stage-name>.deb and <stage-name>.deb.sha256, in the same
+# sha256sum format every other asset uses. Prints `dpkg-deb --info` for the
+# build log, so a CI run records exactly what it shipped.
+#
+# Requires: dpkg-deb, binutils (readelf, objdump), sha256sum.
+# ─────────────────────────────────────────────────────────────────────────────
+set -euo pipefail
+
+# Modes of script-created FILES must not depend on the caller's umask: under a
+# 077 umask `copyright` would ship 0600 and be unreadable for non-root. (The
+# directories are unaffected either way — dpkg-deb records every directory in
+# data.tar as 0700 no matter what is on disk, and dpkg creates the real ones
+# 0755 at install time; verified both halves against dpkg-deb 1.21.)
+umask 022
+
+STAGE_DIR=""
+TARGET=""
+VERSION=""
+OUT_DIR="."
+
+die() { echo "build-deb.sh: $*" >&2; exit 1; }
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --stage-dir) STAGE_DIR="$2"; shift ;;
+ --target) TARGET="$2"; shift ;;
+ --version) VERSION="$2"; shift ;;
+ --out-dir) OUT_DIR="$2"; shift ;;
+ -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "$0"; exit 0 ;;
+ *) die "unknown argument: $1" ;;
+ esac
+ shift
+done
+
+[ -n "$STAGE_DIR" ] || die "--stage-dir is required"
+[ -n "$TARGET" ] || die "--target is required"
+[ -n "$VERSION" ] || die "--version is required"
+[ -x "$STAGE_DIR/xerj" ] || die "$STAGE_DIR/xerj is not an executable — stage the binary first"
+[ -f "$STAGE_DIR/README.md" ] || die "$STAGE_DIR/README.md missing — the tar.gz stage step provides it"
+[ -f "$STAGE_DIR/LICENSE" ] || die "$STAGE_DIR/LICENSE missing — the tar.gz stage step provides it"
+command -v dpkg-deb >/dev/null 2>&1 || die "dpkg-deb not on PATH (run on the release runner, or install dpkg)"
+for t in readelf objdump sha256sum; do
+ command -v "$t" >/dev/null 2>&1 || die "$t not on PATH"
+done
+
+# Debian Architecture, mapped from the Rust triple. Deliberately exhaustive
+# refuse-list: packaging a new target is a decision, not a default.
+case "$TARGET" in
+ x86_64-unknown-linux-gnu) DEB_ARCH=amd64 ;;
+ aarch64-unknown-linux-gnu) DEB_ARCH=arm64 ;;
+ *) die "no Debian architecture mapping for target '$TARGET' (only the *-unknown-linux-gnu targets are packaged)" ;;
+esac
+
+# A Debian version string must start with a digit. Tag builds pass
+# ${GITHUB_REF_NAME#v}; canary builds must pass the workspace version instead
+# of the ref name (which is "main").
+case "$VERSION" in
+ [0-9]*) : ;;
+ *) die "version '$VERSION' is not a valid Debian version (must start with a digit; canary builds pass the workspace version, not the ref name)" ;;
+esac
+dpkg --compare-versions "$VERSION" eq "$VERSION" 2>/dev/null \
+ || die "version '$VERSION' rejected by dpkg --compare-versions"
+
+# ── runtime dependencies, measured off the ELF we are about to pack ──────────
+# readelf/objdump parse ELF of any architecture, so this works unchanged on
+# the cross-built aarch64 binary.
+NEEDED=$(readelf -d "$STAGE_DIR/xerj" | sed -n 's/.*Shared library: \[\(.*\)\]/\1/p' | sort -u || true)
+GLIBC_FLOOR=$(objdump -T "$STAGE_DIR/xerj" 2>/dev/null | grep -o 'GLIBC_[0-9][0-9.]*' | sort -Vu | tail -1 || true)
+GLIBC_FLOOR="${GLIBC_FLOOR#GLIBC_}"
+
+DEB_DEPENDS="libc6"
+if [ -n "$GLIBC_FLOOR" ]; then
+ DEB_DEPENDS="libc6 (>= $GLIBC_FLOOR)"
+fi
+
+# The allowlist is glibc itself, nothing else. Anything else in NEEDED means
+# the binary grew a shared-library dependency the .deb's Depends does not
+# express — the honest move is to fail the build, not to guess a mapping.
+while IFS= read -r lib; do
+ [ -z "$lib" ] && continue
+ case "$lib" in
+ libc.so.6|libm.so.6|libpthread.so.0|libdl.so.2|librt.so.1|ld-linux-*.so.*)
+ # folded into libc6 on any glibc this floor can run on
+ ;;
+ libgcc_s.so.1)
+ DEB_DEPENDS="$DEB_DEPENDS, libgcc-s1" ;;
+ *)
+ die "binary needs '$lib', which this .deb's Depends does not cover — add it or change the build (expected: glibc only; TLS is rustls, so libssl here would be a regression)" ;;
+ esac
+done <<< "$NEEDED"
+
+echo "build-deb.sh: NEEDED libraries: ${NEEDED:-<none>}"
+echo "build-deb.sh: newest GLIBC symbol version: ${GLIBC_FLOOR:-<none found>}"
+echo "build-deb.sh: Depends: $DEB_DEPENDS"
+
+# ── package root ─────────────────────────────────────────────────────────────
+STAGE_NAME=$(basename "$STAGE_DIR")
+PKG=$(mktemp -d "${TMPDIR:-/tmp}/xerj-deb-XXXXXX")
+trap 'rm -rf "$PKG"' EXIT
+
+mkdir -p "$PKG/DEBIAN" "$PKG/usr/bin" "$PKG/usr/share/doc/xerj"
+
+# The exact file the tar.gz ships — same inode content, no strip, no rebuild.
+cp "$STAGE_DIR/xerj" "$PKG/usr/bin/xerj"
+cp "$STAGE_DIR/README.md" "$PKG/usr/share/doc/xerj/README.md"
+cp "$STAGE_DIR/LICENSE" "$PKG/usr/share/doc/xerj/LICENSE"
+
+# Debian machine-readable copyright, pointing at the system copy of the
+# Apache-2.0 text instead of pasting it a third time into /usr/share/doc.
+cat > "$PKG/usr/share/doc/xerj/copyright" <<EOF
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: xerj
+Source: https://github.com/xerj-org/xerj
+
+Files: *
+Copyright: xerj Contributors
+License: Apache-2.0
+ On Debian systems, the complete text of the Apache License 2.0 can be
+ found at /usr/share/common-licenses/Apache-2.0.
+EOF
+
+INSTALLED_SIZE=$(du -sk "$PKG/usr" | cut -f1)
+
+# Description continuation lines must start with a single space; the package
+# description is the one place a Debian user meets before the binary, so it
+# claims only what the binary does — search engine, one file, ES-compatible
+# REST surface included.
+cat > "$PKG/DEBIAN/control" <<EOF
+Package: xerj
+Version: $VERSION
+Architecture: $DEB_ARCH
+Maintainer: xerj Contributors <https://github.com/xerj-org/xerj>
+Installed-Size: $INSTALLED_SIZE
+Depends: $DEB_DEPENDS
+Section: net
+Priority: optional
+Homepage: https://xerj.org
+Built-Using: rustc ($TARGET)
+Description: single-binary search engine for AI workloads
+ XERJ is a search engine written from scratch in Rust that ships as one
+ executable: full-text, vector and hybrid search over the same indices,
+ built-in agent memory, and log analytics. It also speaks the Elasticsearch
+ 8.x REST protocol as a compatibility bridge, so existing ES clients work
+ unmodified. This package contains the release binary built from
+ https://github.com/xerj-org/xerj at version $VERSION.
+EOF
+
+mkdir -p "$OUT_DIR"
+DEB="$OUT_DIR/${STAGE_NAME}.deb"
+
+# --root-owner-group: files land root:root without this script being root.
+dpkg-deb --root-owner-group --build "$PKG" "$DEB" >/dev/null
+
+( cd "$OUT_DIR" && sha256sum "$(basename "$DEB")" > "$(basename "$DEB").sha256" )
+
+echo "build-deb.sh: built $DEB ($(du -h "$DEB" | cut -f1))"
+dpkg-deb --info "$DEB"
diff --git a/scripts/debian-rust-coverage.sh b/scripts/debian-rust-coverage.sh
new file mode 100755
index 000000000..768eb27ab
--- /dev/null
+++ b/scripts/debian-rust-coverage.sh
@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+# ─────────────────────────────────────────────────────────────────────────────
+# debian-rust-coverage.sh — how much of xerj's Rust dependency tree is already
+# packaged in Debian?
+#
+# Issue #810's second half. A package enters the Debian ARCHIVE only if every
+# build dependency is itself in Debian (Christian Kastner's split: the .deb
+# release asset is "fairly simple"; archive inclusion is a function of how
+# much of the tree Debian already carries). This script measures that number
+# instead of guessing it, and docs/PACKAGING_DEBIAN.md records the result.
+#
+# Method:
+# * third-party crates read from engine/Cargo.lock (workspace xerj-* crates
+# excluded — they would be built from this source package);
+# * compared against the SOURCE packages of Debian unstable
+# (dists/unstable/main/source/Sources.gz), where the Debian Rust team
+# packages each crate as rust-<crate> with '_' mapped to '-'
+# (serde_derive → rust-serde-derive, per Debian policy on '_' in package
+# names);
+# * a crate counts as PRESENT if the source package exists. PRESENCE ONLY:
+# the packaged version may be older than the one Cargo.lock pins, and this
+# script does not compare versions — that deeper audit is what "archive
+# inclusion not pursued yet" means in the doc.
+#
+# Usage:
+# scripts/debian-rust-coverage.sh # fetch index, print report
+# scripts/debian-rust-coverage.sh --verbose # also list missing crates
+# scripts/debian-rust-coverage.sh --index F # reuse a saved Sources.gz
+# scripts/debian-rust-coverage.sh --lock F # another Cargo.lock
+#
+# Requires: curl, gzip. Network unless --index is given.
+# ─────────────────────────────────────────────────────────────────────────────
+set -euo pipefail
+
+REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)
+LOCK="${REPO_ROOT}/engine/Cargo.lock"
+INDEX=""
+VERBOSE=0
+SOURCES_URL="http://deb.debian.org/debian/dists/unstable/main/source/Sources.gz"
+
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --lock) LOCK="$2"; shift ;;
+ --index) INDEX="$2"; shift ;;
+ --verbose) VERBOSE=1 ;;
+ -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "$0"; exit 0 ;;
+ *) echo "unknown argument: $1" >&2; exit 2 ;;
+ esac
+ shift
+done
+
+[ -f "$LOCK" ] || { echo "no such Cargo.lock: $LOCK" >&2; exit 2; }
+
+# Third-party crate names: every [[package]] name that is not a workspace
+# xerj-* crate. awk state machine, not grep -A, because the entry body is
+# multi-line and the name must be tied to its [[package]] stanza.
+CRATES=$(awk '
+ /^\[\[package\]\]$/ { in_pkg = 1; next }
+ in_pkg && /^name = "/ { gsub(/"/, "", $3); print $3; in_pkg = 0; next }
+ /^$/ { in_pkg = 0 }
+' "$LOCK" | grep -v '^xerj-' | sort -u)
+TOTAL=$(printf '%s\n' "$CRATES" | wc -l | tr -d ' ')
+
+if [ -n "$INDEX" ]; then
+ [ -f "$INDEX" ] || { echo "no such index file: $INDEX" >&2; exit 2; }
+ DEB_PACKAGES=$(gzip -dc "$INDEX" | sed -n 's/^Package: //p' | sort -u)
+else
+ command -v curl >/dev/null 2>&1 || { echo "curl is required to fetch $SOURCES_URL" >&2; exit 2; }
+ TMPIDX=$(mktemp "${TMPDIR:-/tmp}/xerj-sources-XXXXXX.gz")
+ trap 'rm -f "$TMPIDX" "$JOIN" "$JOIN.pkgs" "$JOIN.crates" 2>/dev/null' EXIT
+ echo "fetching $SOURCES_URL" >&2
+ curl -sSf -m 300 -o "$TMPIDX" "$SOURCES_URL"
+ DEB_PACKAGES=$(gzip -dc "$TMPIDX" | sed -n 's/^Package: //p' | sort -u)
+fi
+
+[ -n "$DEB_PACKAGES" ] || { echo "no Package: entries parsed from the index — refusing to report an empty denominator" >&2; exit 2; }
+
+# Membership via ONE awk pass over both lists, not a shell loop of per-crate
+# greps. That is not a style choice: the loop form was measured giving
+# run-to-run varying counts on an idle machine (sporadic short reads through
+# 654 sequential `printf | grep` pipelines), and a measurement tool that
+# reports a different number each run is worse than no tool. The awk join is
+# deterministic and order-independent.
+JOIN=$(mktemp "${TMPDIR:-/tmp}/xerj-coverage-XXXXXX")
+trap 'rm -f "${TMPIDX:-}" "$JOIN" "$JOIN.pkgs" "$JOIN.crates" 2>/dev/null' EXIT
+printf '%s\n' "$DEB_PACKAGES" > "$JOIN.pkgs"
+printf '%s\n' "$CRATES" > "$JOIN.crates"
+RESULT=$(awk '
+ NR == FNR { pkgs[$0] = 1; next }
+ {
+ name = $0
+ gsub(/_/, "-", name)
+ if (pkgs["rust-" name]) print "P\t" $0; else print "M\t" $0
+ }
+' "$JOIN.pkgs" "$JOIN.crates")
+PRESENT=$(printf '%s\n' "$RESULT" | grep -c '^P' | tr -d ' ')
+MISSING=$(printf '%s\n' "$RESULT" | sed -n 's/^M\t//p')
+
+MISSING_COUNT=$((TOTAL - PRESENT))
+PCT=$(awk -v p="$PRESENT" -v t="$TOTAL" 'BEGIN { if (t == 0) print "n/a"; else printf "%.1f", 100 * p / t }')
+
+if [ -n "$INDEX" ]; then
+ INDEX_LINE="index: $INDEX (local copy)"
+else
+ INDEX_LINE="index: $SOURCES_URL"
+fi
+
+echo "third-party crates in $LOCK: $TOTAL unique names"
+echo "packaged in Debian unstable (rust-*): $PRESENT ($PCT%)"
+echo "NOT in Debian unstable: $MISSING_COUNT"
+echo "$INDEX_LINE — $(date -u +%Y-%m-%d)"
+echo "presence only; packaged versions were NOT compared against Cargo.lock pins"
+if [ "$VERBOSE" = 1 ]; then
+ printf 'missing: %s\n' "$MISSING" | tr ' ' '\n' | sed '/^$/d' | sed 's/^/ /'
+fi
diff --git a/scripts/verify-release.sh b/scripts/verify-release.sh
index 59b1c4342..8221f88de 100755
--- a/scripts/verify-release.sh
+++ b/scripts/verify-release.sh
@@ -19,11 +19,16 @@
# Checks, in order:
# 1. every target we ship is present — the set is DECLARED below, not read off
# the release page, so a release that lost a matrix leg fails instead of
-# quietly verifying the targets that did make it
+# quietly verifying the targets that did make it. Since the first tag that
+# ships them (FIRST_DEB_TAG below), that includes a .deb for each of the
+# two linux-gnu targets.
# 2. every archive has a .sha256 companion, and matches it
-# 3. every archive contains the binary + LICENSE + README. An archive that
-# matched its published checksum and still will not extract is a defect in
-# the release, so it FAILS rather than being skipped
+# 3. every archive contains the binary + LICENSE + README — .tar.gz/.zip via
+# tar/unzip, .deb via dpkg-deb (ar+tar as the fallback a macOS host uses).
+# An archive that matched its published checksum and still will not
+# extract is a defect in the release, so it FAILS rather than being
+# skipped. A .deb with neither dpkg-deb nor ar on the host is a counted
+# SKIP, never a silent drop.
# 4. every binary — including the ones this host cannot execute — carries the
# tag's version in its startup banner, and carries no other version
# 5. the host-native binary: --version, boot on a clean data dir, health
@@ -45,8 +50,8 @@
# scripts/verify-release.sh --keep # keep the download dir
# scripts/verify-release.sh --no-smoke # checksums + versions only
#
-# Requires: gh (authenticated), curl, tar, sha256sum|shasum, and unzip for the
-# Windows archives.
+# Requires: gh (authenticated), curl, tar, sha256sum|shasum, unzip for the
+# Windows archives, and dpkg-deb (or ar) for the .deb assets.
# ─────────────────────────────────────────────────────────────────────────────
set -euo pipefail
@@ -90,6 +95,47 @@ x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl"
TOTAL_TARGETS=$(printf '%s\n' "$EXPECTED_TARGETS" | wc -l | tr -d ' ')
+# The two linux-gnu targets additionally ship a .deb (issue #810): the release
+# workflow packs one per gnu target from the same staged binary as the tar.gz.
+# musl deliberately ships no .deb, and neither do macOS or Windows.
+DEB_TARGETS="aarch64-unknown-linux-gnu
+x86_64-unknown-linux-gnu"
+
+# The first tag whose release was built by a workflow that ships the .debs.
+# Presence of the .debs is only DEMANDED from this tag on: a release cut
+# before it legitimately has none, and failing the gate against history would
+# make the verifier useless for the tags that predate the asset (the rc.10
+# drift check above still runs against any tag). If a release older than this
+# constant is re-dispatched with the current workflow it gains .debs, so ANY
+# .deb on the release turns the gate on regardless of the tag (see step 2).
+# Keep in step with the release.yml change that introduced the .deb assets.
+FIRST_DEB_TAG="v1.0.0-rc.76"
+
+# vtag_ge A B — is tag A at or after tag B? Tags here are vMAJOR.MINOR.PATCH
+# with an optional -rc.NN suffix; a pre-release sorts before the release it
+# precedes (1.0.0-rc.76 < 1.0.0). Pure bash on purpose: this runs on macOS
+# hosts too, where sort has no -V and dpkg is absent.
+vtag_ge() {
+ local a="${1#v}" b="${2#v}"
+ [ "$a" = "$b" ] && return 0
+ local a_pre="" b_pre=""
+ case "$a" in *-*) a_pre="${a#*-}"; a="${a%%-*}" ;; esac
+ case "$b" in *-*) b_pre="${b#*-}"; b="${b%%-*}" ;; esac
+ local i a_parts b_parts n x y
+ IFS='.' read -ra a_parts <<<"$a"; IFS='.' read -ra b_parts <<<"$b"
+ n=$(( ${#a_parts[@]} > ${#b_parts[@]} ? ${#a_parts[@]} : ${#b_parts[@]} ))
+ for ((i = 0; i < n; i++)); do
+ x=$((10#${a_parts[i]:-0})); y=$((10#${b_parts[i]:-0}))
+ if [ "$x" -ne "$y" ]; then [ "$x" -gt "$y" ] && return 0 || return 1; fi
+ done
+ # same mainline: the release outranks its own pre-releases, and a later
+ # pre-release outranks an earlier one (rc.2 > rc.1; the suffix is "rc.N" so
+ # its trailing number is the discriminator).
+ if [ -z "$a_pre" ] && [ -n "$b_pre" ]; then return 0; fi
+ if [ -n "$a_pre" ] && [ -z "$b_pre" ]; then return 1; fi
+ [ "$((10#${a_pre##*.}))" -ge "$((10#${b_pre##*.}))" ]
+}
+
FAILURES=0
# Targets that were present but could not be checked (archive not unpackable on
# this host). Counted separately from failures: not a defect in the release, but
@@ -158,8 +204,8 @@ cd "$WORKDIR"
step "1. download every published asset"
gh release download "$TAG" --repo "$REPO" --dir "$WORKDIR" --clobber >/dev/null
-ARCHIVES=$(find . -maxdepth 1 -type f \( -name '*.tar.gz' -o -name '*.zip' \) | sed 's|^\./||' | sort)
-[ -n "$ARCHIVES" ] || { fail "no .tar.gz or .zip assets on $TAG"; exit 1; }
+ARCHIVES=$(find . -maxdepth 1 -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' \) | sed 's|^\./||' | sort)
+[ -n "$ARCHIVES" ] || { fail "no .tar.gz, .zip or .deb assets on $TAG"; exit 1; }
printf ' %s%s archives%s\n' "$DIM" "$(printf '%s\n' "$ARCHIVES" | wc -l | tr -d ' ')" "$RST"
step "2. every target we ship is present"
@@ -172,6 +218,22 @@ for t in $EXPECTED_TARGETS; do
fail "$t — NO archive published for this target"
fi
done
+# The .deb assets get the same declared-not-discovered treatment, with a time
+# gate: they exist from FIRST_DEB_TAG on (and on any release that has one at
+# all — a re-dispatched older tag rebuilt by the new workflow). Before that,
+# demanding them would fail against every historical tag.
+DEB_GATE=0
+if vtag_ge "$TAG" "$FIRST_DEB_TAG"; then DEB_GATE=1; fi
+if find . -maxdepth 1 -type f -name '*.deb' | grep -q .; then DEB_GATE=1; fi
+for t in $DEB_TARGETS; do
+ if [ "$DEB_GATE" = 1 ]; then
+ if printf '%s\n' "$ARCHIVES" | grep -qx "xerj-$EXPECTED-$t.deb"; then
+ pass "$t (.deb)"
+ else
+ fail "$t — NO .deb published (shipped for every release from $FIRST_DEB_TAG on)"
+ fi
+ fi
+done
step "3. checksums"
for a in $ARCHIVES; do
@@ -190,7 +252,7 @@ done
step "4. archive contents"
for a in $ARCHIVES; do
- d="unpack/${a%.tar.gz}"; d="${d%.zip}"
+ d="unpack/${a%.tar.gz}"; d="${d%.zip}"; d="${d%.deb}"
mkdir -p "$d"
# An archive that matched its published checksum and still will not extract is
# a defect in the RELEASE, not a gap in this host's tooling — so it is a FAIL,
@@ -205,6 +267,30 @@ for a in $ARCHIVES; do
UNPACKABLE="$UNPACKABLE $a"
continue
fi ;;
+ *.deb)
+ # dpkg-deb on a Debian-ish host; ar + tar as the fallback (macOS has no
+ # dpkg-deb but ships ar and a tar that reads the data member directly).
+ # Neither tool is a release defect, so a .deb that cannot be opened for
+ # want of them is a counted SKIP, not a FAIL.
+ if command -v dpkg-deb >/dev/null 2>&1; then
+ if ! dpkg-deb -x "$a" "$d"; then
+ fail "$a — matched its published checksum but did not extract"
+ UNPACKABLE="$UNPACKABLE $a"
+ continue
+ fi
+ elif command -v ar >/dev/null 2>&1; then
+ member=$(ar t "$a" 2>/dev/null | grep '^data\.tar\.' | head -1 || true)
+ zflag=j; case "$member" in *.xz) zflag=J ;; *.gz) zflag=z ;; *.zst) zflag=--zstd ;; esac
+ if [ -z "$member" ] || ! ar p "$a" "$member" | tar -x"$zflag" -C "$d"; then
+ fail "$a — matched its published checksum but did not extract"
+ UNPACKABLE="$UNPACKABLE $a"
+ continue
+ fi
+ else
+ warn "$a — no dpkg-deb or ar on PATH, contents NOT checked"
+ SKIPPED_TARGETS=$((SKIPPED_TARGETS + 1))
+ continue
+ fi ;;
*.zip)
if ! command -v unzip >/dev/null 2>&1; then
warn "$a — unzip not on PATH, contents NOT checked"; continue
@@ -229,8 +315,8 @@ step "5. version string matches the tag, on every target"
# The rc.10 check. A binary whose banner disagrees with its own filename is the
# defect; a binary carrying two different versions is a stale-artifact defect.
for a in $ARCHIVES; do
- d="unpack/${a%.tar.gz}"; d="${d%.zip}"
- target=$(printf '%s' "$a" | sed "s/^xerj-$EXPECTED-//; s/\.tar\.gz$//; s/\.zip$//")
+ d="unpack/${a%.tar.gz}"; d="${d%.zip}"; d="${d%.deb}"
+ target=$(printf '%s' "$a" | sed "s/^xerj-$EXPECTED-//; s/\.tar\.gz$//; s/\.zip$//; s/\.deb$//")
bin=$(find "$d" -type f \( -name xerj -o -name xerj.exe \) 2>/dev/null | head -1 || true)
if [ -z "$bin" ]; then
case " $UNPACKABLE " in