Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
docs: Mac, AMD, Intel, older GPUs and multi-GPU for 0.2.0
Niko1221b27c46cbd654
diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.mdindex d282ad603..626376d25 100644--- a/docs/AMD_HIP.md+++ b/docs/AMD_HIP.md@@ -264,7 +264,7 @@ sizes itself automatically and leaves 1 GiB of VRAM headroom.GPU memory to system RAM (GTT), the OOM killer then ends KWin/plasmashell or `systemd-oomd` ends apps, or thecompositor fails with "Failed to pin framebuffer with error -12".-The installer supports this backend (see "Install with setup" above). Pictures are read on the card (`strata-vision-hip`) when the engine has it, else on the CPU (`strata-vision`).+The installer supports this backend (see "Install with setup" above). Pictures are on by default. They are read on the CPU (`strata-vision`) under 24 GB of VRAM and on the card (`strata-vision-hip`) from 24 GB, the same rule as for every make of card; if the engine has no card reader, or it cannot start, the CPU reader is used.Setup installs one AMD card, or several with `--gpus` (the engine's layer split; see RDNA4 below).**Opt-in switches for RDNA2 prompts and hangs (0.1.40, off by default):** `STRATA_DENSE_MMQ=1` (#820, HIP) runs the@@ -495,7 +495,12 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 394K / 20K residual dumps identical on both packs.- **Known:** rarely (about 1 start in 10) a HIP run's greedy output differs from another start's at some token, onone card or two and on engine 0.1.29 as well; not yet explained.-- **Not validated:** images, long contexts beyond 16K, answer-quality benchmarks.+- **Pictures:** the card's reader (`strata-vision-hip`) is measured on an R9700 (gfx1201): 57 ms for a 260-token picture and 0.40 s for a+ 972-token one, against 3.13 s and 14.1 s on the CPU reader (EPYC, 32 threads); the embeddings agree with the CPU reader's (cosine mean 0.99985). It keeps+ about 1.4 GiB of VRAM (+1,470 MiB on the R9700). A test picture (red circle, blue square, green triangle, the text "STRATA 42") was answered correctly with both readers, and the+ fallback to the CPU reader was tested with a reader built for another card. The same reader built for gfx1103 ran on a Radeon 780M (0.40 s for 260 tokens+ against 2.42 s on its CPU). It has not run on an RDNA3 card (gfx1100-gfx1102) or on Windows with an AMD card.+- **Not validated:** long contexts beyond 16K, answer-quality benchmarks.## Community-validated cardsdiff --git a/docs/INTEL.md b/docs/INTEL.mdindex 77fa10ef0..8b89f9e4f 100644--- a/docs/INTEL.md+++ b/docs/INTEL.md@@ -759,7 +759,7 @@ Measured on an Arc A750 (8 GB, `i915`, PCIe 4.0) with the Flash-Next IQ3_XXS in8192 --prefill-repeat 5` for the time of both kernels. Not taken: the same kernel with its state in SLM (no registerspills, the same bits, ~4x slower: 13.7 vs 3.1-3.3 ms at T=1000).-### Prompt speed on Alchemist: the grouped XMX expert launch is the default (0.1.48, `STRATA_PF_XMX`; owner approved 2026-10-18)+### Prompt speed on Alchemist: the grouped XMX expert launch is the default (engine 0.1.45.3, `STRATA_PF_XMX`)The A750's prompt path decodes and multiplies a group of experts in one fused launch on the XMX units (`STRATA_PF_XMX=2`, `sycl/src/prefill/xmx_moe.dp.cpp`)instead of dequantizing each expert to FP16 and calling oneMKL. It is the default on Alchemist cards; `STRATA_PF_XMX=0` turns it off and `=1` is the@@ -770,6 +770,23 @@ teacher-forced KL against the default over 6,006 forced tokens 0.0002 (the defauis 0.012), first-token KL 0.0017 (floor 0.0005), top-1 agreement 99.7%. HumanEval (every 2nd, 82) 73, 69 and 74 for the mode and two default runs, reasoning 21 / 23 / 22 of30, a 35-question Chinese/Japanese/Korean set 35/35 in all, needles at 2K-6K 9/9, an 8K-context tool-call agent 6/6.+### A750: a prompt that never finished is fixed (engine 0.1.45.3, commit `fca3c5958`)++On an Arc A750 some prompts never finished: with default flags a 2,744-token documentation excerpt hung in 6 of 6 runs (the+thread spun in `sched_yield` inside `libze_intel_gpu` with every engine idle), and the same hang was seen on a 5.3K-token needle prompt. The+cause: SYCL has no `cudaHostRegister`, so the expert arena is ordinary pageable memory even where the engine calls it+pinned. Copying a 2.3 MB expert straight out of it went through the Level Zero driver's staging path on the calling thread,+and on some prompts that call never returned. Now only experts in USM host memory (the `--stream-experts` pinned mirror)+are copied directly (the answer is remembered per 2 MiB); every other expert goes through the stager's pinned buffers, like+an unpinned one. **`STRATA_PF_DMA_PAGEABLE=1` restores the old direct copy.**++Checked: the 2,744-token repro finished 3 of 3 times after the fix (0 of 6 before); the first-token logits and the tokens are+identical old against new on the A750 (1.4K, 2.5K and 4K prompts) and on the Arc Pro B70 (stock config, and without+`--stream-experts`); speed is unchanged where it did not hang (A750 4K prompt 79.4 against 79.3 tok/s, B70 stock 1,246+against 1,240), and on the B70 without `--stream-experts` the prompt is faster (680-740 against 77-456 tok/s). The+regression test is `sycl/tools/test_prompt_deadlock.sh` (the 2,744-token prompt; it times out on the old build and passes on+the fix).+## Not done- Images, on both Intel engines. Strata's vision path encodes with `strata-vision` into embeddings the CUDAdiff --git a/docs/INTEL_ARC.md b/docs/INTEL_ARC.mdindex cc434b6e1..d9d1d60a1 100644--- a/docs/INTEL_ARC.md+++ b/docs/INTEL_ARC.md@@ -5,11 +5,15 @@ maxfridbe wrote it in [#423](https://github.com/Niko1221/Strata/pull/423), withThe code is in `sycl/`, and the port's own notes, measurements and maintenance procedure are in[docs/INTEL.md](INTEL.md). This page covers what you need, how to build it, and what has been tested.-**Experimental means:** the Strata maintainers have no Intel GPU. We compile the port and run its kernel tests, but-we have not run it on an Arc. Every result on Arc hardware below comes from the community. The NVIDIA and AMD-engines are unchanged: the Intel build is a separate CMake target, off by default.+**Experimental means:** the Intel engine is still young and Linux only. When the port arrived (0.1.39) the maintainers had no+Intel GPU and every result on Arc hardware came from the community. Since 0.1.40.2 the maintainers also test on an Arc Pro B70+(32 GB, `xe`) and an Arc A750 (8 GB, `i915`); the results from those two cards are marked in the table below. Other cards+are still community reports. The NVIDIA and AMD engines are unchanged: the Intel build is a separate CMake target, off by default.-There is **no ready-made Intel engine** in the release zips. You build it from source on Linux.+**Where the engine comes from.** There is **no Intel engine in the release zips** (the ones `START-HERE.bat` and `setup.sh`+use): you build it from source on Linux, as described below. A ready-made Intel engine does exist as a Docker image,+`ghcr.io/niko1221/strata:0.2.0-sycl`, built for the 0.2.0 release (the image needs prepared model files; see+[Docker: Intel Arc](DOCKER.md#intel-arc)). The Mac and Windows builds have no Intel engine.## What has been run, and by whom@@ -48,9 +52,10 @@ Full flags, per-request timings and engine logs are in the report linked in the- **Unchanged:** the CUDA engine's greedy output, checked byte-identical against the gated 0.1.39 build (Q2_0 and Coder).The HIP build is not affected (the option is off by default).-**Not tested by anyone yet:** Windows (no native build path; see below), Arc on WSL2 for the 0.1.39 port, the Alchemist-A-series, integrated Arc GPUs (the B390 / Panther Lake in #515; the shared-memory planning does not exist yet), and-images (not wired on Intel).+**Not tested by anyone yet:** Windows (no native build path; see below), Arc on WSL2 for the 0.1.39 port, integrated Arc+GPUs (the B390 / Panther Lake in #515; the shared-memory planning does not exist yet), and images (not wired on Intel: there+is no picture reader for the Arc engine). The Alchemist A-series was tested later, on an A750 (rows above and+[INTEL.md](INTEL.md), "Arc A750 and the other Alchemist cards").## What you needdiff --git a/docs/MAC.md b/docs/MAC.mdindex 88e14d983..59ff725df 100644--- a/docs/MAC.md+++ b/docs/MAC.md@@ -1,9 +1,18 @@# Strata on a Mac (Apple Silicon, Metal)Strata runs on Apple Silicon Macs through its own engine on Metal. This is the same engine as on a PC. The NVIDIA-kernels are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. `./setup.sh` sets it up the way-`START-HERE.bat` does on Windows. There is no ready-made Mac engine: setup builds it from source on your Mac-(about 1-15 minutes). The NVIDIA, AMD and Intel engines are unchanged.+kernels are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. There are two ways in:++- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it+ inside, and the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line+ Tools ([below](#the-desktop-app-and-strata-launch-on-a-mac)). macOS 15 or newer, as for the source build. If macOS says the+ app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal); how the builds are signed:+ [packaging/mac/README.md](../packaging/mac/README.md).+- **From source: `./setup.sh`.** It sets things up the way `START-HERE.bat` does on Windows and builds the engine on your Mac+ (about 1-15 minutes).++**Pictures:** the app and the Terminal have no picture reader on the Mac yet. Only the source build can read pictures+([Limits](#limits-this-version)). The NVIDIA, AMD and Intel engines are unchanged.**Status:** new. It is measured on three Macs: an M5 Max with 128 GB, where the model stays in memory; and an M4 Mac miniwith 32 GB and an M1 Max with 34 GB, where the experts stream from the SSD (the M1 Max: 7.9 tok/s decode, 186 tok/s on a@@ -179,13 +188,17 @@ The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medi## Limits (this version)-- **Images (vision):** `--vision yes` in setup. Checked on one test picture with Q2_0 and UD-Q4_K_XL: the answers equal the HIP engine's, and text output is unchanged with vision on. No other tests yet.+- **Images (vision): the source build only.** `./setup.sh` asks "Should the model also read images?" (`--vision yes|no`) and+ builds the picture reader (`strata-vision`) itself. Checked on one test picture with Q2_0 and UD-Q4_K_XL: the answers equal+ the HIP engine's, and text output is unchanged with vision on. No other tests yet. **The app and the Terminal have no+ picture reader on the Mac yet**: they show models as text only and do not turn pictures on.- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can be set, but it is not measured.- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).- **Streaming:** measured at 32 GB only. The model's file must be on the internal SSD; an external disk is untested and likely much slower.- **Families:** the original Qwen3.8-Flash-Next and Unsloth's UD-Q4/Q5/Q6_K_XL are measured (above). Setup repacks Unsloth's shards into one file for the GPU. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine.-- **Engine only from source.** There is no ready-made Mac engine in the release zips.+- **The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips.** The zips+ (for `START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.## Switches (for testing; all off unless set, the defaults are above)diff --git a/docs/MODELS.md b/docs/MODELS.mdindex d1b075a7c..5b595117b 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -100,7 +100,7 @@ picture, nothing taken from the card) and on the graphics card from 24 GB (aboutpicture). The rule is the same for every make of card. `--vision off` (or "No" in the app and in `strata launch`) turnspictures off; `--vision cpu` or `--vision gpu` picks the reader. On AMD, `strata-vision-hip` is the card's reader; if itcannot start, Strata uses the CPU reader by itself ([AMD_HIP.md](AMD_HIP.md)). **The Mac app and the Mac Terminal have no-picture reader yet** ([MAC.md](MAC.md)). More: [Images](DETAILS.md#images-vision).+picture reader yet** ([MAC.md](MAC.md)), and neither does the Intel Arc engine ([INTEL.md](INTEL.md#not-done)). More: [Images](DETAILS.md#images-vision).### Qwen3.8-Flash-Nextdiff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.mdindex 3f7b09e4d..ef67fe8d9 100644--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@ -11,6 +11,18 @@ verify window (a few hundred KB through pinned RAM), not twice per layer. No NVLneeded; cards on x4 or x1 slots work, and the PCIe share of each card is probed on its own link. A `--pcie-frac` you giveis every card's share and skips those probes; there is no per-card setting yet.+**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is two+models, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card+list then says which card a loaded model uses ("Already used by ..."), preselects a free card for the next model, and+explains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 on+one card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats+(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only;+the same app code drives NVIDIA cards, but we have not run two models on two NVIDIA cards.++**One make of card per engine.** An engine drives one make of card (NVIDIA, AMD or Intel). The app, `strata launch` and setup+offer only the cards the engine can drive, never split a model across two makes, say in one plain line when the other make has+the bigger card, and say that two models at once on two makes is not supported.+## Using it**Nothing to type.** `START-HERE.bat` (Linux: `./setup.sh`) lists your NVIDIA cards and says for each one whetherdiff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.mdindex 87415ba5a..0bb73ccca 100644--- a/docs/OLDER_GPUS.md+++ b/docs/OLDER_GPUS.md@@ -2,8 +2,9 @@Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cardsbelow run through **opt-in, experimental** paths that community members wrote and measured on their own machines. The-maintainers have none of these cards: each path is compile-checked and unit-tested here, and the ready-made engines-and their output stay exactly as they were. Numbers are the reporters' own, on one machine each.+maintainers have one of these cards, a Tesla P100, and test on it; they have none of the others: each path is+compile-checked and unit-tested here, and the ready-made engines and their output stay exactly as they were. Numbers are+the reporters' own, on one machine each, unless the text says the maintainers measured them.## Support matrix@@ -95,6 +96,17 @@ sums round differently): #540 measured a mean KL of 8.4e-3 on the next-token dis(the same top-1 in 23), the size of other summation-order changes; #655 a worst relative difference of 3.5e-5 perproduct on an RTX 2080 Ti.+### Tesla P100 (sm_60): seven decode switches on by default (engine 0.1.45.3)++On a Tesla P100 the engine now turns on a set of seven switches that change only how the work is scheduled, so the answers+stay bit for bit the same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`,+`STRATA_B6_MMVQ_ROWS=1`, `STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a+switch you set yourself, to any value, is kept. The set is used when every visible card has compute capability 6.0.+Measured by the maintainers on a Tesla P100 16 GB, IQ3_XXS, default flags, 8 interleaved pairs of fresh engines: decode 36.56+-> 41.76 tokens/s (+14.2%, faster in 8 of 8 pairs; a run against itself gave 0.975), with the same number of expert-cache+slots; the identity gate (10 prompts and 4K/20K prompt dumps) is identical with the set on. The full list, the RTX 30 set+and the per-switch results: [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default).+### Pascal GP100: a Q8_0 decode kernel (opt-in)GP100 (P100, PH402) is the one Pascal chip without `__dp4a`, and the decode kernels are written for the cards that havediff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex fa814f144..9e413d4e5 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -183,7 +183,7 @@ reader runs on the CPU under 24 GB of graphics memory and on the card from 24 GBseveral seconds (about 3 s at 300 image tokens on 8 cores, more with more tokens) are read by the encoder on the CPU;`--vision gpu` (NVIDIA and AMD, ~1.4 GB of VRAM) makes it 0.1-0.5 s. On AMD the card's reader is `strata-vision-hip`; if itcannot start, the server prints "the GPU picture reader did not start (...); pictures are read on the CPU" and goes on with-the CPU reader. The Mac app and the Mac Terminal have no picture reader yet.+the CPU reader. The Mac app, the Mac Terminal and the Intel Arc engine have no picture reader yet.## AMD cards