ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.
Python ai-agentsamdcomfyuidockerllama-cppllmlocal-ain8nnvidiaopen-webuiragself-hostedspeech-to-textstrix-halotext-to-speechworkflow-automation
fix(embeddings): run TEI natively on Linux arm64 (#8262)
Mike Bradley0174d1eaae3d
diff --git a/ods/config/dependency-lock.json b/ods/config/dependency-lock.jsonindex 63b5b9e92..3de2e0e1a 100644--- a/ods/config/dependency-lock.json+++ b/ods/config/dependency-lock.json@@ -205,6 +205,12 @@"path": "extensions/services/embeddings/compose.yaml","value": "ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.1@sha256:b7772cdd9dcbced147b16a7dff17d4aed1ab36333f8d3e686c50d2175e1d2126"},+ {+ "id": "embeddings.tei-arm64",+ "type": "image",+ "path": "extensions/services/embeddings/compose.arm64.yaml",+ "value": "ghcr.io/huggingface/text-embeddings-inference:cpu-arm64-sha-e80ef22@sha256:798cd7615e060a72ccbdf19aa0ada52289c6a0147490195bafaece6ce04d70f1"+ },{"id": "hermes.agent","type": "image",diff --git a/ods/docs/EXTENSIONS.md b/ods/docs/EXTENSIONS.mdindex de0d6bb51..8f2f276ef 100644--- a/ods/docs/EXTENSIONS.md+++ b/ods/docs/EXTENSIONS.md@@ -94,6 +94,7 @@ extensions/services/compose.yaml # Docker Compose fragment (for extension services)compose.amd.yaml # GPU overlay for AMD (optional)compose.nvidia.yaml # GPU overlay for NVIDIA (optional)+ compose.arm64.yaml # arm64 image pin, when upstream tags are amd64 only (optional, bundled services)```**Core services** (llama-server, open-webui, dashboard, dashboard-api) have only a `manifest.yaml` — their compose definitions live in `docker-compose.base.yml`.@@ -581,7 +582,7 @@ If the host agent is unreachable, file-level operations (install, enable, disabl1. Selects the base compose files based on GPU backend and tier (e.g., `docker-compose.base.yml` + `docker-compose.nvidia.yml`)2. Walks `extensions/services/*/` and includes each extension's `compose.yaml` if it exists (not `.disabled`), skipping extensions incompatible with the current GPU backend-3. Picks up GPU-specific overlays (`compose.<backend>.yaml`) and mode-specific overlays (`compose.local.yaml`) per extension+3. Picks up GPU-specific overlays (`compose.<backend>.yaml`) and mode-specific overlays (`compose.local.yaml`) per extension, and on arm64 hosts other than macOS a bundled extension's `compose.arm64.yaml` (an arm64 pin for an image whose release tags are amd64 only; Docker Desktop on macOS runs those under Rosetta 2)4. Walks `data/user-extensions/*/` and includes compose files from user-installed extensions5. Appends `docker-compose.override.yml` if it exists (user customizations)diff --git a/ods/extensions/services/embeddings/README.md b/ods/extensions/services/embeddings/README.mdindex 3d720d611..6a5a0f755 100644--- a/ods/extensions/services/embeddings/README.md+++ b/ods/extensions/services/embeddings/README.md@@ -12,7 +12,7 @@ The embeddings service runs Hugging Face's Text Embeddings Inference (TEI) serve- **OpenAI-compatible API**: Drop-in replacement for OpenAI's embeddings endpoint- **Configurable model**: Switch embedding models via a single environment variable- **Persistent model cache**: Downloaded models are stored locally and survive restarts-- **CPU-packaged runtime**: The shipped Compose service uses the pinned TEI CPU image across supported hosts+- **CPU-packaged runtime**: The shipped Compose service uses the pinned TEI CPU image across supported hosts. TEI's release tags are amd64 only, so Linux arm64 hosts (DGX Station, DGX Spark, Ampere) run TEI's own arm64 build of the same release line from `compose.arm64.yaml`, and Apple Silicon runs the amd64 image under Rosetta 2## Configurationdiff --git a/ods/extensions/services/embeddings/compose.arm64.yaml b/ods/extensions/services/embeddings/compose.arm64.yamlnew file mode 100644index 000000000..5c090e5f0--- /dev/null+++ b/ods/extensions/services/embeddings/compose.arm64.yaml@@ -0,0 +1,13 @@+# Linux arm64 hosts: DGX Station, DGX Spark and other NVIDIA Grace systems,+# Ampere servers, Windows on Arm (WSL2). TEI publishes no multi-architecture+# index: its release tags are amd64 only, and Docker on Linux has no amd64+# emulation by default ("exec format error"). TEI's own CI builds every commit+# for arm64; this pins the build of the v1.9.4 release commit (e80ef22). It+# also runs on 64 KB-page kernels (Workstation06, 2026-10-27).+#+# scripts/resolve-compose-stack.sh adds this file on arm64 hosts other than+# macOS, where Docker Desktop runs the base amd64 image under Rosetta 2.+services:+ embeddings:+ image: ghcr.io/huggingface/text-embeddings-inference:cpu-arm64-sha-e80ef22@sha256:798cd7615e060a72ccbdf19aa0ada52289c6a0147490195bafaece6ce04d70f1+ platform: linux/arm64diff --git a/ods/extensions/services/embeddings/compose.yaml b/ods/extensions/services/embeddings/compose.yamlindex 9e8c5d602..6bef75d44 100644--- a/ods/extensions/services/embeddings/compose.yaml+++ b/ods/extensions/services/embeddings/compose.yaml@@ -1,7 +1,7 @@services:embeddings:image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.1@sha256:b7772cdd9dcbced147b16a7dff17d4aed1ab36333f8d3e686c50d2175e1d2126- platform: linux/amd64 # TEI has no arm64 image -- Rosetta 2 on Apple Silicon+ platform: linux/amd64 # Release tags are amd64 only: Rosetta 2 on Apple Silicon, compose.arm64.yaml on Linux arm64container_name: ods-embeddingsrestart: unless-stoppedsecurity_opt:diff --git a/ods/installers/phases/03-features.sh b/ods/installers/phases/03-features.shindex 5bc47f363..ebb2c40e8 100755--- a/ods/installers/phases/03-features.sh+++ b/ods/installers/phases/03-features.sh@@ -493,11 +493,8 @@ if ! $DRY_RUN; thenENABLE_QDRANT=falsefifi-- if [[ "${ENABLE_EMBEDDINGS:-${ENABLE_RAG:-false}}" == "true" ]]; then- ai_warn "Embeddings (TEI): upstream image is amd64-only - disabled on aarch64."- ENABLE_EMBEDDINGS=false- fi+ # Embeddings (TEI) runs here from its arm64 pin in+ # extensions/services/embeddings/compose.arm64.yaml.fiunset _host_arch _host_page_sizediff --git a/ods/installers/phases/08-images.sh b/ods/installers/phases/08-images.shindex 51f83560c..3c88b34e1 100755--- a/ods/installers/phases/08-images.sh+++ b/ods/installers/phases/08-images.sh@@ -117,7 +117,15 @@ if [[ "$ENABLE_HERMES" == "true" ]]; thenPULL_LIST+=("${HERMES_AGENT_IMAGE:-nousresearch/hermes-agent:v2026.9.24@sha256:fca358f12efd65bfaaca05884166f15c0e2788375ca30d77061ac1ebc96452b7}|HERMES — default agent (Nous Research)")PULL_LIST+=("caddy:2.11.3-alpine@sha256:86deaf5e3d3408a6ccec08fbb79989783dd26e206ae10bcf78a801dc8c9ab794|HERMES PROXY — magic-link auth gate (Caddy)")fi-[[ "${ENABLE_EMBEDDINGS:-${ENABLE_RAG:-false}}" == "true" ]] && PULL_LIST+=("ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.1@sha256:b7772cdd9dcbced147b16a7dff17d4aed1ab36333f8d3e686c50d2175e1d2126|TEI — embedding engine")+if [[ "${ENABLE_EMBEDDINGS:-${ENABLE_RAG:-false}}" == "true" ]]; then+ # TEI's release tags are amd64 only; arm64 hosts run the pin in+ # extensions/services/embeddings/compose.arm64.yaml.+ if [[ "${HOST_ARCH:-$(uname -m)}" == "arm64" || "${HOST_ARCH:-$(uname -m)}" == "aarch64" ]]; then+ PULL_LIST+=("ghcr.io/huggingface/text-embeddings-inference:cpu-arm64-sha-e80ef22@sha256:798cd7615e060a72ccbdf19aa0ada52289c6a0147490195bafaece6ce04d70f1|TEI — embedding engine")+ else+ PULL_LIST+=("ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.1@sha256:b7772cdd9dcbced147b16a7dff17d4aed1ab36333f8d3e686c50d2175e1d2126|TEI — embedding engine")+ fi+fiif command -v ods_compose_external_images >/dev/null 2>&1 && [[ -n "${COMPOSE_FLAGS:-}" ]]; thenread -ra _phase08_compose_flags_arr <<< "$COMPOSE_FLAGS"diff --git a/ods/scripts/resolve-compose-stack.sh b/ods/scripts/resolve-compose-stack.shindex a58544d3f..1a34ea70a 100755--- a/ods/scripts/resolve-compose-stack.sh+++ b/ods/scripts/resolve-compose-stack.sh@@ -178,6 +178,12 @@ enable_open_webui = os.environ.get("ENABLE_OPEN_WEBUI", "true").lower() == "trueIS_DARWIN = platform.system() == "Darwin"APPLE_OVERLAY = "installers/macos/docker-compose.macos.yml" if IS_DARWIN else "docker-compose.apple.yml"+# Docker runs images natively except on macOS, where Docker Desktop runs amd64+# images under Rosetta 2. A bundled service whose upstream publishes no arm64+# image under its pinned tag ships an arm64 pin in compose.arm64.yaml (TEI).+# HOST_ARCH is the installer's detect_host_arch value (amd64/arm64).+ARM64_HOST = not IS_DARWIN and (+ os.environ.get("HOST_ARCH") or platform.machine()).strip().lower() in {"arm64", "aarch64"}macos_cloud_auth = script_dir / "data" / "generated" / "docker-compose.macos-cloud-auth.yml"def existing(overlays):@@ -1391,6 +1397,9 @@ if selected_stack is None and ext_dir.exists():gpu_overlay = service_dir / f"compose.{gpu_backend}.yaml"if service_dir.name.lower() not in skip_gpu_overlays and gpu_overlay.exists():resolved.append(str(gpu_overlay.relative_to(script_dir)))+ arch_overlay = service_dir / "compose.arm64.yaml"+ if ARM64_HOST and arch_overlay.exists():+ resolved.append(str(arch_overlay.relative_to(script_dir)))# Mode-specific overlay — depends_on for local/hybrid mode only.# Skip on Apple Silicon: macOS runs llama-server natively on the hostdiff --git a/ods/tests/ci-suite.txt b/ods/tests/ci-suite.txtindex 546649009..009025ddc 100644--- a/ods/tests/ci-suite.txt+++ b/ods/tests/ci-suite.txt@@ -213,6 +213,7 @@ tests/test-preview-inspection.mjstests/test-resolve-compose-nondict-service.shtests/test-resolve-compose-symlinked-install.shtests/test-resolver-assume-enabled.py+tests/test-resolver-arm64-overlay.pytests/test-restore-data-transaction-contract.shtests/test-restore-empty-config.shtests/test-restore-safety-ux.shdiff --git a/ods/tests/contracts/test-installer-contracts.sh b/ods/tests/contracts/test-installer-contracts.shindex abd5d9649..0e3282255 100755--- a/ods/tests/contracts/test-installer-contracts.sh+++ b/ods/tests/contracts/test-installer-contracts.sh@@ -566,10 +566,13 @@ echo "$guard_4k" | grep -q '^ENABLE_QDRANT=true$' \|| { echo "[FAIL] 4K-page aarch64 must keep ENABLE_QDRANT enabled"; echo "$guard_4k"; exit 1; }echo "$guard_4k" | grep -q '^QDRANT_COMPOSE=enabled$' \|| { echo "[FAIL] 4K-page aarch64 must keep Qdrant compose enabled"; echo "$guard_4k"; exit 1; }-echo "$guard_4k" | grep -q '^ENABLE_EMBEDDINGS=false$' \- || { echo "[FAIL] aarch64 must still disable amd64-only embeddings"; echo "$guard_4k"; exit 1; }-echo "$guard_4k" | grep -q '^EMBEDDINGS_COMPOSE=disabled$' \- || { echo "[FAIL] aarch64 must still disable embeddings compose"; echo "$guard_4k"; exit 1; }+# TEI runs on arm64 from compose.arm64.yaml, with 4K and 64K pages alike.+for guard in "$guard_4k" "$guard_64k"; do+ echo "$guard" | grep -q '^ENABLE_EMBEDDINGS=true$' \+ || { echo "[FAIL] aarch64 must keep embeddings enabled (arm64 TEI pin)"; echo "$guard"; exit 1; }+ echo "$guard" | grep -q '^EMBEDDINGS_COMPOSE=enabled$' \+ || { echo "[FAIL] aarch64 must keep embeddings compose enabled"; echo "$guard"; exit 1; }+doneecho "[contract] non-interactive reinstall reuses valid GPU assignment"grep -q '_load_existing_gpu_assignment_json' "$features_phase" \diff --git a/ods/tests/test-phase08-selected-images.sh b/ods/tests/test-phase08-selected-images.shindex 73b6bd9b5..a7fc2bb11 100755--- a/ods/tests/test-phase08-selected-images.sh+++ b/ods/tests/test-phase08-selected-images.sh@@ -72,4 +72,28 @@ assert_image "$plan" 'COMFYUI' absent 'host-native CPU stack'plan="$(image_plan amd local '' '' false true)"assert_image "$plan" 'COMFYUI' present 'separately selected AMD ComfyUI'+# TEI's release tags are amd64 only. Each architecture pre-pulls the image its+# Compose files run: compose.yaml, or compose.arm64.yaml on arm64.+tei_plan() (+ export SCRIPT_DIR="$ROOT_DIR" LOG_FILE=/dev/null DRY_RUN=true COMPOSE_FLAGS=''+ export GPU_BACKEND=cpu ODS_MODE=local EXTERNAL_LLM_URL='' NATIVE_LLM_BASE_URL=''+ export ENABLE_PERPLEXICA=false ENABLE_COMFYUI=false ENABLE_OPEN_WEBUI=false+ export ENABLE_VOICE=false ENABLE_WORKFLOWS=false ENABLE_RAG=false+ export ENABLE_QDRANT=false ENABLE_EMBEDDINGS=true ENABLE_HERMES=false HOST_ARCH="$1"+ ods_progress() { :; }; show_phase() { :; }; ai() { :; }+ ai_ok() { :; }; ai_warn() { :; }; bootline() { :; }; signal() { :; }+ source "$ROOT_DIR/installers/phases/08-images.sh"+ for entry in "${PULL_LIST[@]}"; do+ if [[ "$entry" == *"|TEI — "* ]]; then printf '%s\n' "$entry"; fi+ done+)+for arch_file in "amd64 compose.yaml" "x86_64 compose.yaml" "arm64 compose.arm64.yaml" "aarch64 compose.arm64.yaml"; do+ read -r arch file <<<"$arch_file"+ expected="$(sed -nE 's/^[[:space:]]*image:[[:space:]]*([^[:space:]]+).*/\1/p' "$ROOT_DIR/extensions/services/embeddings/$file")"+ [[ -n "$expected" ]] || { echo "FAIL: no image in embeddings/$file" >&2; exit 1; }+ tei="$(tei_plan "$arch")"+ [[ "$tei" == "$expected|TEI — embedding engine" ]] \+ || { echo "FAIL: $arch pre-pulls '$tei', but embeddings/$file runs $expected" >&2; exit 1; }+done+echo 'PASS: Phase 08 image plans follow the selected route and services'diff --git a/ods/tests/test-resolver-arm64-overlay.py b/ods/tests/test-resolver-arm64-overlay.pynew file mode 100644index 000000000..486a9c1ec--- /dev/null+++ b/ods/tests/test-resolver-arm64-overlay.py@@ -0,0 +1,74 @@+"""A bundled service's compose.arm64.yaml joins the stack on arm64 hosts.++TEI publishes no arm64 image under its release tags, so arm64 hosts run the+arm64 pin in extensions/services/embeddings/compose.arm64.yaml; without it TEI+crash-looped on "exec format error" (Workstation06, 2026-10-27). Docker+Desktop on macOS keeps the base image under Rosetta 2.+"""++import os+import platform+from pathlib import Path+import subprocess++import pytest++ROOT = Path(__file__).resolve().parents[1]+RESOLVER = ROOT / "scripts" / "resolve-compose-stack.sh"+BASE = "extensions/services/demo/compose.yaml"+ARM64 = "extensions/services/demo/compose.arm64.yaml"+MANIFEST = """schema_version: ods.services.v1+service:+ id: demo+ compose_file: compose.yaml+ gpu_backends: [all]+"""+++def install(tmp_path):+ (tmp_path / "docker-compose.base.yml").write_text(+ "services:\n dashboard-api:\n image: example/dashboard-api:1\n", encoding="utf-8")+ bundled = tmp_path / "extensions" / "services" / "demo"+ bundled.mkdir(parents=True)+ (bundled / "manifest.yaml").write_text(MANIFEST, encoding="utf-8")+ (bundled / "compose.yaml").write_text(+ "services:\n demo:\n image: example/demo:1\n", encoding="utf-8")+ (bundled / "compose.arm64.yaml").write_text(+ "services:\n demo:\n image: example/demo:1-arm64\n platform: linux/arm64\n",+ encoding="utf-8")+ return tmp_path+++def resolve(root, host_arch):+ env = {**os.environ, "ODS_MODE": "local", "HOST_ARCH": host_arch}+ for selector in ("ODS_RESOLVE_ASSUME_ENABLED", "ODS_SKIP_GPU_OVERLAYS", "ENABLE_OPEN_WEBUI"):+ env.pop(selector, None)+ result = subprocess.run(+ ["bash", str(RESOLVER), "--script-dir", str(root), "--tier", "1", "--gpu-backend", "cpu",+ "--gpu-count", "1", "--ods-mode", "local"],+ capture_output=True, text=True, env=env, timeout=60,+ )+ assert result.returncode == 0, result.stderr+ return [flag.replace("\\", "/") for flag in result.stdout.split()]++[email protected](platform.system() == "Darwin",+ reason="Docker Desktop on macOS runs the base image under Rosetta 2")[email protected]("host_arch", ["arm64", "aarch64"])+def test_an_arm64_host_layers_the_arm64_pin_over_the_base(install, host_arch):+ flags = resolve(install, host_arch)+ assert BASE in flags and ARM64 in flags+ assert flags.index(ARM64) > flags.index(BASE)++[email protected]("host_arch", ["amd64", "x86_64"])+def test_an_amd64_host_keeps_the_base_image(install, host_arch):+ flags = resolve(install, host_arch)+ assert BASE in flags+ assert ARM64 not in flags+++def test_a_disabled_service_brings_no_arm64_pin(install):+ (install / BASE).rename(install / f"{BASE}.disabled")+ assert not any("extensions/services/demo" in flag for flag in resolve(install, "arm64"))