Skip to content

armgit / ODS

Public Imported from GitHub
0

ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.

10.3 KB
Raw Blame

Intel Arc GPU Guide

Last updated: 2026-10-04

ODS supports Intel Arc discrete GPUs via the llama.cpp SYCL backend (docker-compose.arc.yml). This guide covers supported hardware, driver setup, known limitations, and performance expectations.


Supported Hardware

Tier: ARC (≥ 12 GB VRAM)

GPU VRAM Estimated tok/s Concurrent users Model
Arc A770 16 GB ~35 3–5 Qwen3.5 9B Q4_K_M
Arc B580 (untested) 12 GB ~30 2–4 Qwen3.5 9B Q4_K_M

Tier: ARC_LITE (< 12 GB VRAM)

GPU VRAM Estimated tok/s Concurrent users Model
Arc A750 8 GB ~20 1–2 Qwen3.5 4B Q4_K_M
Arc A380 6 GB ~15 1 Qwen3.5 4B Q4_K_M
Arc A310 4 GB ~10 1 Qwen3.5 4B Q4_K_M (tight)

A310 note: 4 GB VRAM is borderline for Qwen3.5 4B Q4_K_M (~3.3 GB). The model will load but leaves little headroom for KV cache. Consider --ctx-size 4096 (set CTX_SIZE=4096 in .env) to reduce pressure.

Future / untested

Intel Arc B-series (Battlemage) cards are untested. Hardware detection does not map any Arc card to these tiers yet (see Installation), so pass --tier ARC for cards with 12 GB or more and --tier ARC_LITE below that.


Host Driver Setup (Ubuntu / Debian)

1 — Add Intel GPU repository

# Import Intel GPG key and repo
wget -qO - https://repositories.intel.com/gpu/intel-graphics.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/intel-graphics.gpg

echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] \
  https://repositories.intel.com/gpu/ubuntu jammy unified" \
  | sudo tee /etc/apt/sources.list.d/intel-gpu-jammy.list

sudo apt update

2 — Install kernel and user-mode drivers

# Kernel module + firmware (i915 / xe)
sudo apt install -y linux-headers-$(uname -r) \
    intel-i915-dkms intel-fw-gpu

# Level Zero runtime (required for SYCL)
sudo apt install -y intel-level-zero-gpu level-zero

# OpenCL runtime (required for llama.cpp OpenCL fallback)
sudo apt install -y intel-opencl-icd

# Monitoring tools (optional but recommended)
sudo apt install -y intel-gpu-tools clinfo

3 — Add user to GPU groups

sudo usermod -aG video,render $USER
# Re-login (or newgrp render) for the change to take effect

4 — Verify

# Should list Intel Arc as an OpenCL device
clinfo | grep -A3 "Device Name"

# Should show Level Zero GPU
ze_info 2>/dev/null | grep -i "device name" || \
    ldconfig -p | grep libze_loader

# Should show render node
ls -la /dev/dri/renderD*

# Live GPU usage (Ctrl+C to exit)
sudo intel_gpu_top

Installation

Hardware detection does not identify Intel Arc yet, so pass the tier yourself:

./install.sh --tier ARC        # Arc cards with 12 GB or more (A770)
./install.sh --tier ARC_LITE   # Arc cards under 12 GB (A750, A380)

What the installer does for Intel Arc

  1. Phase 01 (preflight) — checks disk space (≥ 20 GB for model download)
  2. Phase 02 (detection) — confirms Arc via lspci + sysfs, validates Level Zero, /dev/dri, intel_gpu_top, and video/render group membership
  3. Phase 05 (docker) — validates docker-compose.arc.yml syntax
  4. Phase 06 (directories) — writes .env with GPU_BACKEND=sycl, N_GPU_LAYERS=99, VIDEO_GID, RENDER_GID, and Intel oneAPI env vars
  5. Phase 07 (devtools) — installs OpenCode and CLI tooling
  6. Phase 08 (launch) — runs docker compose -f docker-compose.base.yml -f docker-compose.arc.yml up -d

Docker Compose Overlay

ODS provides two Intel Arc overlays:

File Image When to use
docker-compose.arc.yml Built locally from intel/oneapi-basekit Default. Requires docker compose up --build on first run (~10–20 min).
docker-compose.intel.yml Pre-built ghcr.io/ggml-org/llama.cpp:server-intel-* Quick start — no build time. Set LLAMA_ARC_IMAGE=<tag> in .env.

Manual compose start

# Build and start (first time ~10–20 min build)
docker compose -f docker-compose.base.yml -f docker-compose.arc.yml up -d --build

# Subsequent starts (no rebuild)
docker compose -f docker-compose.base.yml -f docker-compose.arc.yml up -d

# Skip local build — use a pre-built image
LLAMA_ARC_IMAGE=ghcr.io/ggml-org/llama.cpp:server-intel-b9014@sha256:9c7bbaad3663523a3deb8927d3cfbf58d33f00a7634c69843e9eeeda01568c1b \
  docker compose -f docker-compose.base.yml -f docker-compose.arc.yml up -d

Key .env variables for Arc

GPU_BACKEND=sycl
N_GPU_LAYERS=99
VIDEO_GID=44          # auto-set by installer
RENDER_GID=992        # auto-set by installer
ONEAPI_DEVICE_SELECTOR=level_zero:gpu   # level_zero:0 on hosts with more than one Intel GPU
ZES_ENABLE_SYSMAN=1
CTX_SIZE=32768        # ARC tier default

Known Limitations vs NVIDIA / AMD

Feature NVIDIA (CUDA) AMD (Vulkan; ROCm optional) Intel Arc (SYCL)
Installer maturity Tier B Tier A Tier C (experimental)
llama.cpp backend CUDA (native) Vulkan (default) or HIP/ROCm SYCL (via oneAPI)
SYCL kernel cache — — JIT compile on every container start (~30 s). ODS does not set SYCL_CACHE_PERSISTENT: with the oneAPI 2025.3 runtime in the b9014 image it crashes llama-server.
Multi-GPU ✅ (native) ✅ (layer split) ❌ Not supported. SYCL backend targets a single Arc GPU.
ComfyUI (image gen) ✅ CUDA overlay ✅ ROCm overlay ⚠️ No dedicated overlay. ComfyUI will use CPU fallback.
Whisper STT ✅ CUDA overlay ✅ ROCm overlay ⚠️ Runs on CPU (no Arc-accelerated Whisper image).
Flash attention ✅ ✅ ❌ llama.cpp SYCL does not yet implement Flash Attention.
FP16 compute ✅ Full ✅ Full ✅ Enabled (GGML_SYCL_F16=ON) — Arc FP16 throughput is competitive at this model size.
Docker image size ~6 GB ~0.2 GB compressed (Vulkan); ~7 GB (ROCm) ~15 GB (oneAPI Base Toolkit is large).
First-run build time Pull only Pull only ~10–20 min (compiles llama.cpp from source).
Windows support ✅ WSL2 ✅ WSL2 ⚠️ Experimental. Arc drivers for WSL2 are less mature than NVIDIA's.

Performance Expectations

Performance figures below are measured with Qwen3 models at Q4_K_M quantisation, --n-gpu-layers 99 (all layers on GPU), --ctx-size 16384.

GPU Model Prompt tok/s Generate tok/s Notes
Arc A770 (16 GB) Qwen3.5 9B Q4_K_M ~120 ~35 Comfortable fit; KV cache well within VRAM
Arc A750 (8 GB) Qwen3.5 4B Q4_K_M ~90 ~20 Model fits; limit CTX_SIZE to ≤ 16384
Arc A380 (6 GB) Qwen3.5 4B Q4_K_M ~70 ~15 Tight. Set CTX_SIZE=8192 for safety

Comparison to equivalent NVIDIA tiers

Intel Arc Comparable NVIDIA VRAM Generate tok/s delta
A770 (ARC) RTX 3060 12 GB (T1) 16 vs 12 GB Arc ~+5 tok/s on 8B (more VRAM headroom)
A750 (ARC_LITE) RTX 3060 12 GB (T1) 8 vs 12 GB Arc ~-10 tok/s (less VRAM, smaller model)

Intel Arc SYCL throughput is broadly similar to an equivalent NVIDIA card at the same VRAM tier. Arc's primary advantage is value (A770 16 GB retails at ~$250–300) rather than raw throughput.


Troubleshooting

llama-server exits immediately with SYCL error

SYCL error: code 6, ZE_RESULT_ERROR_DEVICE_LOST

Cause: Level Zero cannot enumerate a GPU device. Fix:

# Verify host driver
clinfo | grep "Device Name"
# If empty:
sudo apt install intel-level-zero-gpu level-zero
# Then restart the container
docker compose restart llama-server

Slow first inference after container start

Cause: SYCL kernel JIT compilation on first call (~20–60 s). Expected: every container start pays this. Do not set SYCL_CACHE_PERSISTENT=1 to avoid it: with the oneAPI 2025.3 runtime in the llama.cpp b9014 image, the persistent kernel cache crashes llama-server (ggml-org/llama.cpp#21474, #22095; intel/llvm#22853). ODS no longer passes it to the container, and earlier installs that have it in .env are unaffected.


llama-server crashes at start on a host with two or more Intel GPUs

Cause: the oneAPI runtime in the b9014 image can crash when ONEAPI_DEVICE_SELECTOR=level_zero:gpu exposes more than one Level Zero GPU (ggml-org/llama.cpp#21747; fixed upstream after b9014 by #22968). Fix: set ONEAPI_DEVICE_SELECTOR=level_zero:0 in .env (the index of the GPU to use) and recreate llama-server. The installer keeps this value on reruns.


/dev/dri not found inside container

Error opening /dev/dri/renderD128: Permission denied

Cause: User not in render group, or Docker socket not passed through. Fix:

sudo usermod -aG render $USER
# Re-login, then:
docker compose -f docker-compose.base.yml -f docker-compose.arc.yml up -d

Container fails to start on WSL2

Intel Arc drivers on WSL2 are less mature than NVIDIA's. If the Arc GPU is not visible inside WSL2:

  1. Update Windows to the latest version (22H2+).
  2. Install the latest Intel Graphics driver from intel.com/arc-drivers.
  3. Verify the GPU is visible: wsl -- ls /dev/dri
  4. If still missing, fall back to CPU mode: ./install.sh --tier 1 (runs inference on CPU, no GPU passthrough).

intel_gpu_top shows 0% GPU engine utilisation during inference

This is a known display quirk when the compute engine is used heavily — intel_gpu_top sometimes under-reports Arc engine utilisation in older versions of intel-gpu-tools. Verify the model is actually running on GPU by checking VRAM:

# Should show non-zero VRAM used
sudo intel_gpu_top -l 1 | grep -i mem

Related Docs