Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

installer: refuse x86_64 builds on CPUs without AVX2/BMI2/FMA, document the floor (#2178)

samuel mbabhazic46d394cd533
diff --git a/README.md b/README.md
index 1d4a2725d..e5ab2dfe9 100644
--- a/README.md
+++ b/README.md
@@ -197,8 +197,11 @@ irm https://xerj.org/get.ps1 | iex
```
One static binary, no JVM, no dependencies. Prebuilt for Linux, macOS and Windows on x86-64
-and arm64. You can also [build from source](#build-from-source). It speaks the Elasticsearch
-API, so existing clients, dashboards and tooling work against it unchanged.
+and arm64. The x86-64 builds need a CPU with AVX2, BMI2 and FMA (x86-64-v3: Intel Haswell
+2013 and later, AMD Excavator/Zen and later); on an older CPU they stop at launch with an
+illegal-instruction error, so the installer checks before downloading and tells you to
+[build from source](#build-from-source) instead. It speaks the Elasticsearch API, so existing
+clients, dashboards and tooling work against it unchanged.
First commands after install (the installer prints where `xerj` landed; add it to your PATH
if needed): `xerj --insecure --data-dir ~/xerj-data &`, wait until `http://localhost:9200`
@@ -577,6 +580,15 @@ cargo build --release -p xerj-server
./target/release/xerj --insecure --data-dir ~/xerj-data
```
+The x86-64 release builds target `x86-64-v3` (AVX2, BMI2, FMA; set per target in
+`engine/.cargo/config.toml`), which is why the prebuilt binaries crash at launch on CPUs older
+than Intel Haswell (2013) or AMD Excavator/Zen. `RUSTFLAGS` overrides that file, so on such a
+CPU build with a lower baseline instead of installing the prebuilt binary:
+
+```sh
+RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release -p xerj-server
+```
+
To run the conformance suite against a running server:
```sh
diff --git a/landing/docs/install.html b/landing/docs/install.html
index c6eda9482..f650a803c 100644
--- a/landing/docs/install.html
+++ b/landing/docs/install.html
@@ -241,7 +241,7 @@
<li><strong>OS:</strong> Linux (any distro — the Linux build is fully static musl, Alpine included), macOS (Apple Silicon arm64 / Intel x86_64), or Windows 10+ (x64 / ARM64).</li>
<li><strong>RAM:</strong> 400 MB idle. 2 GB working set per active index. Scales linearly with segment resident set.</li>
<li><strong>Disk:</strong> Any fs that supports <code class="inline">fsync</code> and mmap. Prefer SSD.</li>
- <li><strong>CPU:</strong> x86_64 with AVX2 or arm64 with NEON. The release build uses SIMD for hot paths.</li>
+ <li><strong>CPU:</strong> x86_64 with AVX2, BMI2 and FMA (x86-64-v3: Intel Haswell 2013 and later, AMD Excavator/Zen and later), or arm64 with NEON. The x86_64 release builds are compiled for that baseline and stop at launch with an illegal-instruction error on older CPUs; both installers check the CPU before downloading and refuse with the reason. On such a CPU, <a href="https://github.com/xerj-org/xerj#build-from-source">build from source</a> with <code class="inline">RUSTFLAGS="-C target-cpu=x86-64-v2"</code>.</li>
<li><strong>Open files:</strong> 65536 ulimit is sufficient for most deployments.</li>
</ul>
@@ -254,7 +254,7 @@
<p>Both installers verify <strong>fail-closed</strong>: each computes the archive's own SHA-256 and compares it against the published digest, and if the checksum cannot be downloaded, or no SHA-256 tool (<code class="inline">sha256sum</code>, <code class="inline">shasum</code>, <code class="inline">openssl</code>, <code class="inline">busybox</code>…) can be found, the installer stops rather than installing an unverified binary.</p>
<p>On Linux the one-liner installs the <strong>static musl</strong> build by default — it runs on any distro including Alpine and has no glibc-version floor. Ask for the glibc build explicitly if you need it:</p>
<pre class="code">$ curl -fsSL https://xerj.org/get | XERJ_LIBC=gnu sh</pre>
- <p>Other environment overrides: <code class="inline">XERJ_VERSION</code> (install a specific tag), <code class="inline">XERJ_INSTALL_DIR</code> (default <code class="inline">~/.local/bin</code> on Linux and macOS, <code class="inline">%LOCALAPPDATA%\Programs\xerj</code> on Windows), <code class="inline">XERJ_REPO</code>.</p>
+ <p>Other environment overrides: <code class="inline">XERJ_VERSION</code> (install a specific tag), <code class="inline">XERJ_INSTALL_DIR</code> (default <code class="inline">~/.local/bin</code> on Linux and macOS, <code class="inline">%LOCALAPPDATA%\Programs\xerj</code> on Windows), <code class="inline">XERJ_REPO</code>, <code class="inline">XERJ_SKIP_CPU_CHECK=1</code> (install an x86_64 build even when the CPU does not advertise AVX2; it will crash at launch if the hardware really lacks it).</p>
<h2>Manual download</h2>
<p>Every release publishes eight targets, each with a <code class="inline">.sha256</code> alongside: <code class="inline">x86_64</code>/<code class="inline">aarch64</code> × <code class="inline">unknown-linux-musl</code> (static, any distro), <code class="inline">unknown-linux-gnu</code>, <code class="inline">apple-darwin</code>, and <code class="inline">pc-windows-msvc</code> (.zip).</p>
diff --git a/landing/get b/landing/get
index 93f778abb..8533db951 100644
--- a/landing/get
+++ b/landing/get
@@ -15,6 +15,10 @@
# install without verifying SHA-256. Only useful on a
# box with no hashing tool at all; you are choosing to
# run an unverified binary.
+# XERJ_SKIP_CPU_CHECK=1 install an x86_64 build even when this CPU does
+# not advertise AVX2/BMI2/FMA (see "CPU feature
+# floor" below). The binary will crash at launch if
+# the hardware really lacks them.
# =============================================================================
set -eu
@@ -74,6 +78,55 @@ case "$arch" in
aarch64|arm64) cpu="aarch64" ;;
*) die "unsupported CPU: $arch (xerj ships x86_64/amd64 and aarch64/arm64 builds for Linux, macOS and Windows)" ;;
esac
+# An x86_64 shell on Apple Silicon is Rosetta 2 (`arch -x86_64`, an Intel
+# terminal app, …): uname lies about the hardware. Install the native arm64
+# build instead of an x86_64 one that would run translated (and, before
+# macOS 15, without AVX2 at all). rustup-init makes the same correction.
+if [ "$os" = "Darwin" ] && [ "$cpu" = "x86_64" ] \
+ && [ "$(sysctl -n sysctl.proc_translated 2>/dev/null)" = "1" ]; then
+ say "${YEL}note:${RST} this shell runs under Rosetta 2 on Apple Silicon — installing the native arm64 build"
+ cpu="aarch64"
+fi
+
+# --- CPU feature floor (x86_64 only) -----------------------------------------
+# Every x86_64 release binary is built with `-C target-cpu=x86-64-v3`
+# (engine/.cargo/config.toml), so it executes AVX2/BMI2/FMA instructions
+# unconditionally. On a CPU without them — Intel before Haswell (2013), AMD
+# before Excavator/Zen, some Atom/Celeron/Pentium parts, VMs that hide the
+# flags — the process dies at launch with "Illegal instruction" and prints
+# nothing (#2161, field report #2158). Refuse here, before the download, and
+# say why. The arm64 builds have no such floor (ARMv8-A baseline).
+#
+# Only positive evidence fails the check: a readable feature list that lacks
+# a feature. When the list cannot be read at all (no /proc/cpuinfo, no
+# sysctl) the check is skipped, not failed. XERJ_SKIP_CPU_CHECK=1 bypasses it.
+if [ "$cpu" = "x86_64" ] && [ "${XERJ_SKIP_CPU_CHECK:-}" != "1" ]; then
+ missing=''
+ if [ "$os" = "Darwin" ]; then
+ # hw.optional.* answers 1/0 per feature, nothing when the key is unknown.
+ for f in avx1_0 avx2_0 bmi1 bmi2 f16c fma; do
+ [ "$(sysctl -n "hw.optional.$f" 2>/dev/null)" = "0" ] && missing="$missing $f"
+ done
+ elif [ -r /proc/cpuinfo ]; then
+ # Linux, and the /proc emulation of Git Bash / MSYS / Cygwin on Windows.
+ # The flag names are the kernel's: LZCNT is advertised as `abm`.
+ flags=$(sed -n 's/^flags[[:space:]]*:[[:space:]]*//p' /proc/cpuinfo | head -n1)
+ if [ -n "$flags" ]; then
+ for f in avx avx2 bmi1 bmi2 f16c fma abm movbe xsave; do
+ case " $flags " in *" $f "*) ;; *) missing="$missing $f" ;; esac
+ done
+ fi
+ fi
+ if [ -n "$missing" ]; then
+ die "this CPU lacks${missing} — the x86_64 release binaries are built for x86-64-v3
+ (AVX2/BMI2/FMA: Intel Haswell 2013+, AMD Excavator/Zen+) and crash at launch with
+ 'Illegal instruction' without them (https://github.com/${REPO}/issues/2161).
+ Build for this CPU instead: RUSTFLAGS='-C target-cpu=x86-64-v2' cargo build --release -p xerj-server
+ (https://github.com/${REPO}#build-from-source)
+ To install the prebuilt binary anyway (e.g. a sandbox hides flags the hardware has):
+ XERJ_SKIP_CPU_CHECK=1"
+ fi
+fi
if [ "$os" = "Linux" ]; then
case "${XERJ_LIBC:-musl}" in
musl) ;; # the default set above
diff --git a/landing/get.ps1 b/landing/get.ps1
index 7890d19e5..9c9844289 100644
--- a/landing/get.ps1
+++ b/landing/get.ps1
@@ -11,6 +11,9 @@
# XERJ_INSTALL_DIR=path install location (default: %LOCALAPPDATA%\Programs\xerj)
# XERJ_REPO=owner/name source repo (default: xerj-org/xerj)
# XERJ_NO_PATH=1 do not add the install dir to the user PATH
+# XERJ_SKIP_CPU_CHECK=1 install the x64 build even when this CPU does not
+# advertise AVX2 (see "CPU feature floor" below). It
+# will crash at launch if the hardware really lacks it.
#
# Works on Windows PowerShell 5.1 and PowerShell 7+.
#
@@ -53,6 +56,54 @@
}
$target = "$cpu-pc-windows-msvc"
+ # --- CPU feature floor (x64 only) -------------------------------------------
+ # Every x64 release binary is built with `-C target-cpu=x86-64-v3`
+ # (engine/.cargo/config.toml), so it executes AVX2/BMI2/FMA instructions
+ # unconditionally. On a CPU without them - Intel before Haswell (2013), AMD
+ # before Excavator/Zen, some Atom/Celeron/Pentium parts, VMs that hide the
+ # flags - xerj.exe dies at launch with STATUS_ILLEGAL_INSTRUCTION
+ # (0xc000001d) and prints nothing (#2161, field report #2158). Refuse here,
+ # before the download, and say why. The ARM64 build has no such floor.
+ #
+ # Only positive evidence fails the check; when the features cannot be read
+ # it is skipped, not failed. $env:XERJ_SKIP_CPU_CHECK=1 bypasses it.
+ if ($cpu -eq 'x86_64' -and $env:XERJ_SKIP_CPU_CHECK -ne '1') {
+ $missing = @()
+ try {
+ if ($PSVersionTable.PSVersion.Major -ge 7) {
+ # PowerShell 7+ (.NET Core 3.0+): CPUID-backed, independent of the OS build.
+ foreach ($f in 'Avx', 'Avx2', 'Bmi1', 'Bmi2', 'Fma', 'Lzcnt') {
+ $t = "System.Runtime.Intrinsics.X86.$f" -as [type]
+ if ($t -and -not $t::IsSupported) { $missing += $f.ToLower() }
+ }
+ } else {
+ # Windows PowerShell 5.1 (.NET Framework has no intrinsics API):
+ # kernel32!IsProcessorFeaturePresent(PF_AVX2_INSTRUCTIONS_AVAILABLE = 40).
+ # That constant exists since Windows 10 version 2004 (build 19041);
+ # older builds answer 0 whatever the hardware, so skip them.
+ $build = [int](Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion').CurrentBuildNumber
+ if ($build -ge 19041) {
+ if (-not ('XerjInstaller.Cpu' -as [type])) {
+ Add-Type -Namespace 'XerjInstaller' -Name 'Cpu' -MemberDefinition @'
+[DllImport("kernel32.dll")] public static extern bool IsProcessorFeaturePresent(int feature);
+'@
+ }
+ if (-not [XerjInstaller.Cpu]::IsProcessorFeaturePresent(40)) { $missing += 'avx2' }
+ }
+ }
+ } catch {
+ $missing = @() # could not read the features: do not guess
+ }
+ if ($missing.Count -gt 0) {
+ Fail ("this CPU lacks $($missing -join ', ') - the x64 release binaries are built for x86-64-v3 " +
+ "(AVX2/BMI2/FMA: Intel Haswell 2013+, AMD Excavator/Zen+) and crash at launch with " +
+ "STATUS_ILLEGAL_INSTRUCTION without them (https://github.com/$Repo/issues/2161). " +
+ "Build for this CPU instead: `$env:RUSTFLAGS='-C target-cpu=x86-64-v2'; cargo build --release -p xerj-server " +
+ "(https://github.com/$Repo#build-from-source). To install the prebuilt binary anyway " +
+ "(e.g. a sandbox hides a feature the hardware has): `$env:XERJ_SKIP_CPU_CHECK='1'")
+ }
+ }
+
# --- resolve version -------------------------------------------------------
if ($env:XERJ_VERSION) {
$tag = $env:XERJ_VERSION
diff --git a/landing/llms.txt b/landing/llms.txt
index 23fb8ed23..d3f88c124 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -13,7 +13,7 @@ This file asks two things of you. **Reference-code before you write** — retrie
- Linux / macOS: `curl -fsSL https://xerj.org/get | sh`
- Windows PowerShell: `irm https://xerj.org/get.ps1 | iex`
-The installer detects OS/CPU (Linux x86_64/aarch64 static musl, macOS x86_64/arm64, Windows x64/ARM64), downloads the latest GitHub release, verifies its SHA-256 (fail-closed: a missing or mismatched checksum aborts, and the `sh` installer tries `sha256sum`, `shasum`, `openssl`, `sha256`, `busybox` and `cksum` in turn and **refuses to install** if none exists — installing unverified is opt-in only, with `XERJ_INSECURE_SKIP_CHECKSUM=1`), and installs one `xerj` binary — default `~/.local/bin` on Unix (it prints the path and a PATH hint; it does not edit your shell profile), `%LOCALAPPDATA%\Programs\xerj` on Windows (added to user PATH; opt out with `XERJ_NO_PATH=1`). Pin with `XERJ_VERSION=vX.Y.Z`; both honour `XERJ_INSTALL_DIR`.
+The installer detects OS/CPU (Linux x86_64/aarch64 static musl, macOS x86_64/arm64, Windows x64/ARM64; the x86_64 builds need AVX2/BMI2/FMA, i.e. x86-64-v3, and the installer refuses before downloading on an older CPU — build from source with `RUSTFLAGS="-C target-cpu=x86-64-v2"` there), downloads the latest GitHub release, verifies its SHA-256 (fail-closed: a missing or mismatched checksum aborts, and the `sh` installer tries `sha256sum`, `shasum`, `openssl`, `sha256`, `busybox` and `cksum` in turn and **refuses to install** if none exists — installing unverified is opt-in only, with `XERJ_INSECURE_SKIP_CHECKSUM=1`), and installs one `xerj` binary — default `~/.local/bin` on Unix (it prints the path and a PATH hint; it does not edit your shell profile), `%LOCALAPPDATA%\Programs\xerj` on Windows (added to user PATH; opt out with `XERJ_NO_PATH=1`). Pin with `XERJ_VERSION=vX.Y.Z`; both honour `XERJ_INSTALL_DIR`.
**PATH consequence — read this before the next command.** Because the installer does not edit your shell profile, in a fresh shell `~/.local/bin` is often not on `PATH` and a bare `xerj …` fails with `command not found`. Either call it by full path (`$HOME/.local/bin/xerj …`) or `export PATH="$HOME/.local/bin:$PATH"` first. Every `xerj` below assumes you did one of those.