Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(air-gapped): copy the archive once — the double read was the defect all along

xerj-org4fa1b02ed63d
diff --git a/docs/recipes/air-gapped-deployment.md b/docs/recipes/air-gapped-deployment.md
index 99ba9422e..a9cea952c 100644
--- a/docs/recipes/air-gapped-deployment.md
+++ b/docs/recipes/air-gapped-deployment.md
@@ -108,9 +108,14 @@ VERSION="${TAG#v}"
STAGE="xerj-${VERSION}-x86_64-unknown-linux-musl"
ASSET="${STAGE}.tar.gz"
+work="$(mktemp -d)"
+trap 'rm -rf "$work"' EXIT
+cp "/opt/xerj-staging/release/$ASSET" "$work/$ASSET"
+cp "/opt/xerj-staging/release/$ASSET.sha256" "$work/$ASSET.sha256"
+
(
set -eu
- cd /opt/xerj-staging/release
+ cd "$work"
want="$(sha256sum "$ASSET" | cut -d ' ' -f 1)"
if [[ ! "$want" =~ ^[0-9a-f]{64}$ ]]; then
@@ -120,10 +125,8 @@ ASSET="${STAGE}.tar.gz"
LC_ALL=C grep -qxF -e "$want $ASSET" -e "$want *$ASSET" \
< <(tr -d '\r' < "$ASSET.sha256")
- work="$(mktemp -d)"
- trap 'rm -rf "$work"' EXIT
- tar -xzf "$ASSET" -C "$work"
- test -x "$work/$STAGE/xerj"
+ tar -xzf "$ASSET"
+ test -x "$STAGE/xerj"
if ! getent passwd xerj >/dev/null; then
sudo useradd --system --user-group --home-dir /var/lib/xerj \
@@ -131,7 +134,7 @@ ASSET="${STAGE}.tar.gz"
fi
sudo install -d -m 0755 /opt/xerj/bin /etc/xerj
sudo install -d -o xerj -g xerj -m 0750 /var/lib/xerj
- sudo install -m 0755 "$work/$STAGE/xerj" /opt/xerj/bin/xerj
+ sudo install -m 0755 "$STAGE/xerj" /opt/xerj/bin/xerj
echo "installed $ASSET ($want)"
)
```
@@ -159,21 +162,35 @@ operator's login shell on a bad digest — actively harmful over a serial consol
where the natural recovery is to re-paste without it, which is the fail-open
form. Inside, a failure ends the subshell and the operator keeps their session.
-**The block that verifies is the block that installs.** Six earlier versions of
-this recipe handed the verified tree to a later section through a path on disk,
-and every one was broken through that handoff rather than through the digest:
-a directory planted at the agreed name, a `mv` that nested onto it, an attacker
-who computed the digest-named path themselves and swapped the tree before the
-install ran. There is no handoff now — the binary is installed out of the
-private work directory it was extracted into, inside the same subshell — so
-there is nothing for a later step to trust and no exit status that has to be
+**The archive is copied once, then hashed and extracted from that copy.** This
+is the defect eight earlier versions of this recipe shared without noticing.
+Whatever the digest check looked like, the archive at the staging path was
+opened TWICE — once by `sha256sum`, once by `tar` — and anyone who can write
+that directory swaps the file between the two opens. The genuine, untouched
+`.sha256` then verifies, the block prints the genuine digest, exits `0`, and
+installs a backdoor. Reproduced with plain regular files, no FIFOs, on a
+correctly configured system. Copying first collapses that to a single read: the
+bytes that are hashed are the same bytes that are extracted, because they are
+the same file, and it is a file in a directory the attacker cannot reach.
+
+**The block that verifies is the block that installs.** Six versions before that
+handed the verified tree to a later section through a path on disk, and every
+one was broken through the handoff rather than through the digest. There is no
+handoff now, so nothing later has to be trusted and no exit status has to be
checked.
-`mktemp -d` with no argument puts that directory under `TMPDIR`, deliberately
-NOT under the staging directory: renaming or removing a directory entry needs
-write permission on its PARENT, so a work directory inside an attacker-writable
-staging area can be moved aside and replaced however restrictive its own mode
-is.
+**`mktemp -d` puts the work directory under `TMPDIR`, and that is an
+assumption.** It is safe when `TMPDIR` is sticky and not attacker-writable —
+the default `/tmp` on any normal system. It is NOT safe if `/tmp` has been made
+world-writable without the sticky bit, or if `TMPDIR` points into a directory
+the attacker controls: renaming a directory entry needs write on the parent, so
+the work directory can then be swapped however restrictive its own mode is. If
+your enclave is unusual here, set `TMPDIR` to a root-owned directory before
+running these blocks.
+
+An attacker who rewrites the archive AND recomputes its `.sha256` still passes,
+and always will: that is the one case a checksum cannot answer, and the prose
+above already says this is an integrity check and not a signature.
## 2. Configure the base lexical node
@@ -238,9 +255,15 @@ installing anything — the same fail-closed rule as section 1, and the reason
the check is a command here rather than a sentence:
```bash
+mwork="$(mktemp -d)"
+trap 'rm -rf "$mwork"' EXIT
+cp /opt/xerj-staging/model/all-MiniLM-L6-v2/{config.json,tokenizer.json,model.safetensors} \
+ "$mwork/"
+cp /opt/xerj-staging/model/model.sha256 "$mwork/model.sha256"
+
(
set -eu
- cd /opt/xerj-staging/model/all-MiniLM-L6-v2
+ cd "$mwork"
for f in config.json tokenizer.json model.safetensors; do
want="$(sha256sum "$f" | cut -d ' ' -f 1)"
@@ -249,7 +272,7 @@ the check is a command here rather than a sentence:
exit 1
fi
LC_ALL=C grep -qxF -e "$want $f" -e "$want *$f" \
- < <(tr -d '\r' < ../model.sha256)
+ < <(tr -d '\r' < model.sha256)
done
sudo install -d -o xerj -g xerj -m 0750 /opt/xerj/models/all-MiniLM-L6-v2
diff --git a/landing/docs/recipes/air-gapped-deployment.html b/landing/docs/recipes/air-gapped-deployment.html
index eb0da76ef..dfb50a246 100644
--- a/landing/docs/recipes/air-gapped-deployment.html
+++ b/landing/docs/recipes/air-gapped-deployment.html
@@ -112,9 +112,14 @@ VERSION="${TAG#v}"
STAGE="xerj-${VERSION}-x86_64-unknown-linux-musl"
ASSET="${STAGE}.tar.gz"
+work="$(mktemp -d)"
+trap 'rm -rf "$work"' EXIT
+cp "/opt/xerj-staging/release/$ASSET" "$work/$ASSET"
+cp "/opt/xerj-staging/release/$ASSET.sha256" "$work/$ASSET.sha256"
+
(
set -eu
- cd /opt/xerj-staging/release
+ cd "$work"
want="$(sha256sum "$ASSET" | cut -d ' ' -f 1)"
if [[ ! "$want" =~ ^[0-9a-f]{64}$ ]]; then
@@ -124,10 +129,8 @@ ASSET="${STAGE}.tar.gz"
LC_ALL=C grep -qxF -e "$want $ASSET" -e "$want *$ASSET" \
&lt; &lt;(tr -d '\r' &lt; "$ASSET.sha256")
- work="$(mktemp -d)"
- trap 'rm -rf "$work"' EXIT
- tar -xzf "$ASSET" -C "$work"
- test -x "$work/$STAGE/xerj"
+ tar -xzf "$ASSET"
+ test -x "$STAGE/xerj"
if ! getent passwd xerj &gt;/dev/null; then
sudo useradd --system --user-group --home-dir /var/lib/xerj \
@@ -135,10 +138,10 @@ ASSET="${STAGE}.tar.gz"
fi
sudo install -d -m 0755 /opt/xerj/bin /etc/xerj
sudo install -d -o xerj -g xerj -m 0750 /var/lib/xerj
- sudo install -m 0755 "$work/$STAGE/xerj" /opt/xerj/bin/xerj
+ sudo install -m 0755 "$STAGE/xerj" /opt/xerj/bin/xerj
echo "installed $ASSET ($want)"
)</pre>
- <p>Three details are load-bearing, and each replaces something that looked right and was not. <strong>The digest is computed here and the file is searched for it, not the other way round</strong> &mdash; <code class="inline">sha256sum -c</code> reads the filename out of the <code class="inline">.sha256</code>, a file that crossed the airgap on the same medium as the archive, skips <code class="inline">#</code> comments silently, and exits <code class="inline">0</code> if any one line verified, so a decoy digest plus a comment naming the archive passes while the archive is never hashed. Filtering that file first does not fix it: <code class="inline">awk '$2 == asset'</code> compares one token while <code class="inline">sha256sum</code> takes the whole rest of the line as the filename, so a line naming <code class="inline">$ASSET decoy</code> satisfies the filter and hashes a decoy. Building the expected line in the shell and demanding it with <code class="inline">grep -qxF</code> removes the disagreement. The two patterns accept GNU text mode and <code class="inline">-b</code> binary mode; <code class="inline">tr -d '\r'</code> accepts a checksum file written on Windows. <strong><code class="inline">set -eu</code> is inside the subshell</strong> &mdash; at the top of the block it kills the operator&rsquo;s login shell on a bad digest, and the natural recovery is to re-paste without it, which is the fail-open form. <strong>Section 2 installs from <code class="inline">$STAGE.verified</code></strong>, which <code class="inline">mv</code> creates only after the digest, the extraction and <code class="inline">test -x</code> have all succeeded &mdash; so the block is safe even when its exit status is ignored, because on any failure there is nothing at the install path. That last part is only true because <code class="inline">rm -rf</code> runs <strong>first</strong>: placed after the digest check it is unreachable on exactly the failure that matters, and a <code class="inline">$STAGE.verified</code> left by an earlier run &mdash; or planted by anyone who can write to the staging directory &mdash; would survive and be installed.</p>
+ <p><strong>The archive is copied once, then hashed and extracted from that copy.</strong> Eight earlier versions of this recipe opened the archive twice at the staging path &mdash; once by <code class="inline">sha256sum</code>, once by <code class="inline">tar</code> &mdash; and anyone who can write that directory swaps the file between the two opens: the genuine <code class="inline">.sha256</code> verifies, the block prints the genuine digest and exits 0, and a backdoor is installed. Copying first collapses that to a single read, because the bytes hashed and the bytes extracted are the same file, in a directory the attacker cannot reach. <strong>The digest is computed here and the file is searched for it</strong>, not the other way round: <code class="inline">sha256sum -c</code> reads filenames out of the <code class="inline">.sha256</code> and exits 0 if any one line verified. <strong>The block that verifies is the block that installs</strong>, so nothing later has to be trusted and no exit status has to be checked. <strong><code class="inline">mktemp -d</code> puts the work directory under <code class="inline">TMPDIR</code></strong>, which is safe when that is sticky and not attacker-writable (the default <code class="inline">/tmp</code>); if yours is not, point <code class="inline">TMPDIR</code> at a root-owned directory first.</p>
<h2>2. Configure the base lexical node</h2>
<p>Create the service account first, then install only the staged binary. No model directory is needed for lexical mode:</p>
@@ -174,9 +177,15 @@ done
sha256sum config.json tokenizer.json model.safetensors &gt; ../model.sha256
)</pre>
<p>Transfer <code class="inline">OUT/model</code> with the release. Verify it inside the enclave before installing anything &mdash; the same fail-closed rule as section 1, and the reason the check is a command here rather than a sentence:</p>
-<pre class="code">(
+<pre class="code">mwork="$(mktemp -d)"
+trap 'rm -rf "$mwork"' EXIT
+cp /opt/xerj-staging/model/all-MiniLM-L6-v2/{config.json,tokenizer.json,model.safetensors} \
+ "$mwork/"
+cp /opt/xerj-staging/model/model.sha256 "$mwork/model.sha256"
+
+(
set -eu
- cd /opt/xerj-staging/model/all-MiniLM-L6-v2
+ cd "$mwork"
for f in config.json tokenizer.json model.safetensors; do
want="$(sha256sum "$f" | cut -d ' ' -f 1)"
@@ -185,7 +194,7 @@ done
exit 1
fi
LC_ALL=C grep -qxF -e "$want $f" -e "$want *$f" \
- &lt; &lt;(tr -d '\r' &lt; ../model.sha256)
+ &lt; &lt;(tr -d '\r' &lt; model.sha256)
done
sudo install -d -o xerj -g xerj -m 0750 /opt/xerj/models/all-MiniLM-L6-v2