AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
installer: refuse x86_64 builds on CPUs without AVX2/BMI2/FMA, document the floor (#2178)
samuel mbabhazic46d394cd533
diff --git a/README.md b/README.mdindex 1d4a2725d..e5ab2dfe9 100644--- a/README.md+++ b/README.md@@ -197,8 +197,11 @@ irm https://xerj.org/get.ps1 | iex```One static binary, no JVM, no dependencies. Prebuilt for Linux, macOS and Windows on x86-64-and arm64. You can also [build from source](#build-from-source). It speaks the Elasticsearch-API, so existing clients, dashboards and tooling work against it unchanged.+and arm64. The x86-64 builds need a CPU with AVX2, BMI2 and FMA (x86-64-v3: Intel Haswell+2013 and later, AMD Excavator/Zen and later); on an older CPU they stop at launch with an+illegal-instruction error, so the installer checks before downloading and tells you to+[build from source](#build-from-source) instead. It speaks the Elasticsearch API, so existing+clients, dashboards and tooling work against it unchanged.First commands after install (the installer prints where `xerj` landed; add it to your PATHif needed): `xerj --insecure --data-dir ~/xerj-data &`, wait until `http://localhost:9200`@@ -577,6 +580,15 @@ cargo build --release -p xerj-server./target/release/xerj --insecure --data-dir ~/xerj-data```+The x86-64 release builds target `x86-64-v3` (AVX2, BMI2, FMA; set per target in+`engine/.cargo/config.toml`), which is why the prebuilt binaries crash at launch on CPUs older+than Intel Haswell (2013) or AMD Excavator/Zen. `RUSTFLAGS` overrides that file, so on such a+CPU build with a lower baseline instead of installing the prebuilt binary:++```sh+RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release -p xerj-server+```+To run the conformance suite against a running server:```shdiff --git a/landing/docs/install.html b/landing/docs/install.htmlindex c6eda9482..f650a803c 100644--- a/landing/docs/install.html+++ b/landing/docs/install.html@@ -241,7 +241,7 @@<li><strong>OS:</strong> Linux (any distro — the Linux build is fully static musl, Alpine included), macOS (Apple Silicon arm64 / Intel x86_64), or Windows 10+ (x64 / ARM64).</li><li><strong>RAM:</strong> 400 MB idle. 2 GB working set per active index. Scales linearly with segment resident set.</li><li><strong>Disk:</strong> Any fs that supports <code class="inline">fsync</code> and mmap. Prefer SSD.</li>- <li><strong>CPU:</strong> x86_64 with AVX2 or arm64 with NEON. The release build uses SIMD for hot paths.</li>+ <li><strong>CPU:</strong> x86_64 with AVX2, BMI2 and FMA (x86-64-v3: Intel Haswell 2013 and later, AMD Excavator/Zen and later), or arm64 with NEON. The x86_64 release builds are compiled for that baseline and stop at launch with an illegal-instruction error on older CPUs; both installers check the CPU before downloading and refuse with the reason. On such a CPU, <a href="https://github.com/xerj-org/xerj#build-from-source">build from source</a> with <code class="inline">RUSTFLAGS="-C target-cpu=x86-64-v2"</code>.</li><li><strong>Open files:</strong> 65536 ulimit is sufficient for most deployments.</li></ul>@@ -254,7 +254,7 @@<p>Both installers verify <strong>fail-closed</strong>: each computes the archive's own SHA-256 and compares it against the published digest, and if the checksum cannot be downloaded, or no SHA-256 tool (<code class="inline">sha256sum</code>, <code class="inline">shasum</code>, <code class="inline">openssl</code>, <code class="inline">busybox</code>…) can be found, the installer stops rather than installing an unverified binary.</p><p>On Linux the one-liner installs the <strong>static musl</strong> build by default — it runs on any distro including Alpine and has no glibc-version floor. Ask for the glibc build explicitly if you need it:</p><pre class="code">$ curl -fsSL https://xerj.org/get | XERJ_LIBC=gnu sh</pre>- <p>Other environment overrides: <code class="inline">XERJ_VERSION</code> (install a specific tag), <code class="inline">XERJ_INSTALL_DIR</code> (default <code class="inline">~/.local/bin</code> on Linux and macOS, <code class="inline">%LOCALAPPDATA%\Programs\xerj</code> on Windows), <code class="inline">XERJ_REPO</code>.</p>+ <p>Other environment overrides: <code class="inline">XERJ_VERSION</code> (install a specific tag), <code class="inline">XERJ_INSTALL_DIR</code> (default <code class="inline">~/.local/bin</code> on Linux and macOS, <code class="inline">%LOCALAPPDATA%\Programs\xerj</code> on Windows), <code class="inline">XERJ_REPO</code>, <code class="inline">XERJ_SKIP_CPU_CHECK=1</code> (install an x86_64 build even when the CPU does not advertise AVX2; it will crash at launch if the hardware really lacks it).</p><h2>Manual download</h2><p>Every release publishes eight targets, each with a <code class="inline">.sha256</code> alongside: <code class="inline">x86_64</code>/<code class="inline">aarch64</code> × <code class="inline">unknown-linux-musl</code> (static, any distro), <code class="inline">unknown-linux-gnu</code>, <code class="inline">apple-darwin</code>, and <code class="inline">pc-windows-msvc</code> (.zip).</p>diff --git a/landing/get b/landing/getindex 93f778abb..8533db951 100644--- a/landing/get+++ b/landing/get@@ -15,6 +15,10 @@# install without verifying SHA-256. Only useful on a# box with no hashing tool at all; you are choosing to# run an unverified binary.+# XERJ_SKIP_CPU_CHECK=1 install an x86_64 build even when this CPU does+# not advertise AVX2/BMI2/FMA (see "CPU feature+# floor" below). The binary will crash at launch if+# the hardware really lacks them.# =============================================================================set -eu@@ -74,6 +78,55 @@ case "$arch" inaarch64|arm64) cpu="aarch64" ;;*) die "unsupported CPU: $arch (xerj ships x86_64/amd64 and aarch64/arm64 builds for Linux, macOS and Windows)" ;;esac+# An x86_64 shell on Apple Silicon is Rosetta 2 (`arch -x86_64`, an Intel+# terminal app, …): uname lies about the hardware. Install the native arm64+# build instead of an x86_64 one that would run translated (and, before+# macOS 15, without AVX2 at all). rustup-init makes the same correction.+if [ "$os" = "Darwin" ] && [ "$cpu" = "x86_64" ] \+ && [ "$(sysctl -n sysctl.proc_translated 2>/dev/null)" = "1" ]; then+ say "${YEL}note:${RST} this shell runs under Rosetta 2 on Apple Silicon — installing the native arm64 build"+ cpu="aarch64"+fi++# --- CPU feature floor (x86_64 only) -----------------------------------------+# Every x86_64 release binary is built with `-C target-cpu=x86-64-v3`+# (engine/.cargo/config.toml), so it executes AVX2/BMI2/FMA instructions+# unconditionally. On a CPU without them — Intel before Haswell (2013), AMD+# before Excavator/Zen, some Atom/Celeron/Pentium parts, VMs that hide the+# flags — the process dies at launch with "Illegal instruction" and prints+# nothing (#2161, field report #2158). Refuse here, before the download, and+# say why. The arm64 builds have no such floor (ARMv8-A baseline).+#+# Only positive evidence fails the check: a readable feature list that lacks+# a feature. When the list cannot be read at all (no /proc/cpuinfo, no+# sysctl) the check is skipped, not failed. XERJ_SKIP_CPU_CHECK=1 bypasses it.+if [ "$cpu" = "x86_64" ] && [ "${XERJ_SKIP_CPU_CHECK:-}" != "1" ]; then+ missing=''+ if [ "$os" = "Darwin" ]; then+ # hw.optional.* answers 1/0 per feature, nothing when the key is unknown.+ for f in avx1_0 avx2_0 bmi1 bmi2 f16c fma; do+ [ "$(sysctl -n "hw.optional.$f" 2>/dev/null)" = "0" ] && missing="$missing $f"+ done+ elif [ -r /proc/cpuinfo ]; then+ # Linux, and the /proc emulation of Git Bash / MSYS / Cygwin on Windows.+ # The flag names are the kernel's: LZCNT is advertised as `abm`.+ flags=$(sed -n 's/^flags[[:space:]]*:[[:space:]]*//p' /proc/cpuinfo | head -n1)+ if [ -n "$flags" ]; then+ for f in avx avx2 bmi1 bmi2 f16c fma abm movbe xsave; do+ case " $flags " in *" $f "*) ;; *) missing="$missing $f" ;; esac+ done+ fi+ fi+ if [ -n "$missing" ]; then+ die "this CPU lacks${missing} — the x86_64 release binaries are built for x86-64-v3+ (AVX2/BMI2/FMA: Intel Haswell 2013+, AMD Excavator/Zen+) and crash at launch with+ 'Illegal instruction' without them (https://github.com/${REPO}/issues/2161).+ Build for this CPU instead: RUSTFLAGS='-C target-cpu=x86-64-v2' cargo build --release -p xerj-server+ (https://github.com/${REPO}#build-from-source)+ To install the prebuilt binary anyway (e.g. a sandbox hides flags the hardware has):+ XERJ_SKIP_CPU_CHECK=1"+ fi+fiif [ "$os" = "Linux" ]; thencase "${XERJ_LIBC:-musl}" inmusl) ;; # the default set abovediff --git a/landing/get.ps1 b/landing/get.ps1index 7890d19e5..9c9844289 100644--- a/landing/get.ps1+++ b/landing/get.ps1@@ -11,6 +11,9 @@# XERJ_INSTALL_DIR=path install location (default: %LOCALAPPDATA%\Programs\xerj)# XERJ_REPO=owner/name source repo (default: xerj-org/xerj)# XERJ_NO_PATH=1 do not add the install dir to the user PATH+# XERJ_SKIP_CPU_CHECK=1 install the x64 build even when this CPU does not+# advertise AVX2 (see "CPU feature floor" below). It+# will crash at launch if the hardware really lacks it.## Works on Windows PowerShell 5.1 and PowerShell 7+.#@@ -53,6 +56,54 @@}$target = "$cpu-pc-windows-msvc"+ # --- CPU feature floor (x64 only) -------------------------------------------+ # Every x64 release binary is built with `-C target-cpu=x86-64-v3`+ # (engine/.cargo/config.toml), so it executes AVX2/BMI2/FMA instructions+ # unconditionally. On a CPU without them - Intel before Haswell (2013), AMD+ # before Excavator/Zen, some Atom/Celeron/Pentium parts, VMs that hide the+ # flags - xerj.exe dies at launch with STATUS_ILLEGAL_INSTRUCTION+ # (0xc000001d) and prints nothing (#2161, field report #2158). Refuse here,+ # before the download, and say why. The ARM64 build has no such floor.+ #+ # Only positive evidence fails the check; when the features cannot be read+ # it is skipped, not failed. $env:XERJ_SKIP_CPU_CHECK=1 bypasses it.+ if ($cpu -eq 'x86_64' -and $env:XERJ_SKIP_CPU_CHECK -ne '1') {+ $missing = @()+ try {+ if ($PSVersionTable.PSVersion.Major -ge 7) {+ # PowerShell 7+ (.NET Core 3.0+): CPUID-backed, independent of the OS build.+ foreach ($f in 'Avx', 'Avx2', 'Bmi1', 'Bmi2', 'Fma', 'Lzcnt') {+ $t = "System.Runtime.Intrinsics.X86.$f" -as [type]+ if ($t -and -not $t::IsSupported) { $missing += $f.ToLower() }+ }+ } else {+ # Windows PowerShell 5.1 (.NET Framework has no intrinsics API):+ # kernel32!IsProcessorFeaturePresent(PF_AVX2_INSTRUCTIONS_AVAILABLE = 40).+ # That constant exists since Windows 10 version 2004 (build 19041);+ # older builds answer 0 whatever the hardware, so skip them.+ $build = [int](Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion').CurrentBuildNumber+ if ($build -ge 19041) {+ if (-not ('XerjInstaller.Cpu' -as [type])) {+ Add-Type -Namespace 'XerjInstaller' -Name 'Cpu' -MemberDefinition @'+[DllImport("kernel32.dll")] public static extern bool IsProcessorFeaturePresent(int feature);+'@+ }+ if (-not [XerjInstaller.Cpu]::IsProcessorFeaturePresent(40)) { $missing += 'avx2' }+ }+ }+ } catch {+ $missing = @() # could not read the features: do not guess+ }+ if ($missing.Count -gt 0) {+ Fail ("this CPU lacks $($missing -join ', ') - the x64 release binaries are built for x86-64-v3 " ++ "(AVX2/BMI2/FMA: Intel Haswell 2013+, AMD Excavator/Zen+) and crash at launch with " ++ "STATUS_ILLEGAL_INSTRUCTION without them (https://github.com/$Repo/issues/2161). " ++ "Build for this CPU instead: `$env:RUSTFLAGS='-C target-cpu=x86-64-v2'; cargo build --release -p xerj-server " ++ "(https://github.com/$Repo#build-from-source). To install the prebuilt binary anyway " ++ "(e.g. a sandbox hides a feature the hardware has): `$env:XERJ_SKIP_CPU_CHECK='1'")+ }+ }+# --- resolve version -------------------------------------------------------if ($env:XERJ_VERSION) {$tag = $env:XERJ_VERSIONdiff --git a/landing/llms.txt b/landing/llms.txtindex 23fb8ed23..d3f88c124 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -13,7 +13,7 @@ This file asks two things of you. **Reference-code before you write** — retrie- Linux / macOS: `curl -fsSL https://xerj.org/get | sh`- Windows PowerShell: `irm https://xerj.org/get.ps1 | iex`-The installer detects OS/CPU (Linux x86_64/aarch64 static musl, macOS x86_64/arm64, Windows x64/ARM64), downloads the latest GitHub release, verifies its SHA-256 (fail-closed: a missing or mismatched checksum aborts, and the `sh` installer tries `sha256sum`, `shasum`, `openssl`, `sha256`, `busybox` and `cksum` in turn and **refuses to install** if none exists — installing unverified is opt-in only, with `XERJ_INSECURE_SKIP_CHECKSUM=1`), and installs one `xerj` binary — default `~/.local/bin` on Unix (it prints the path and a PATH hint; it does not edit your shell profile), `%LOCALAPPDATA%\Programs\xerj` on Windows (added to user PATH; opt out with `XERJ_NO_PATH=1`). Pin with `XERJ_VERSION=vX.Y.Z`; both honour `XERJ_INSTALL_DIR`.+The installer detects OS/CPU (Linux x86_64/aarch64 static musl, macOS x86_64/arm64, Windows x64/ARM64; the x86_64 builds need AVX2/BMI2/FMA, i.e. x86-64-v3, and the installer refuses before downloading on an older CPU — build from source with `RUSTFLAGS="-C target-cpu=x86-64-v2"` there), downloads the latest GitHub release, verifies its SHA-256 (fail-closed: a missing or mismatched checksum aborts, and the `sh` installer tries `sha256sum`, `shasum`, `openssl`, `sha256`, `busybox` and `cksum` in turn and **refuses to install** if none exists — installing unverified is opt-in only, with `XERJ_INSECURE_SKIP_CHECKSUM=1`), and installs one `xerj` binary — default `~/.local/bin` on Unix (it prints the path and a PATH hint; it does not edit your shell profile), `%LOCALAPPDATA%\Programs\xerj` on Windows (added to user PATH; opt out with `XERJ_NO_PATH=1`). Pin with `XERJ_VERSION=vX.Y.Z`; both honour `XERJ_INSTALL_DIR`.**PATH consequence — read this before the next command.** Because the installer does not edit your shell profile, in a fresh shell `~/.local/bin` is often not on `PATH` and a bare `xerj …` fails with `command not found`. Either call it by full path (`$HOME/.local/bin/xerj …`) or `export PATH="$HOME/.local/bin:$PATH"` first. Every `xerj` below assumes you did one of those.