Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

APR packs: drop the STRATA_APR opt-in; warn only on unvalidated setups

Laurent Zuijdwijk294c95ebbce0
diff --git a/docs/GYRO.md b/docs/GYRO.md
index 8e34e6332..5b73b0a52 100644
--- a/docs/GYRO.md
+++ b/docs/GYRO.md
@@ -11,7 +11,7 @@ The targets are the published files of `agentionai/Qwen3.8-Flash-Next-Gyro-GGUF`
| `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` (92.0 GB) | TQ2_T, block-128 Hadamard | 29.9 GiB | validated end to end (RTX 5090, RTX 3090); the experts do not all fit a 32 GB card, `auto` streams the rest |
Both files carry their own experts in the trellis types TQ2_T/TQK6/TQK7 and record the activation rotation in
-`prism.hadamard.*` metadata. Strata reads both natively (set `STRATA_APR=1`). About half of each file is the
+`prism.hadamard.*` metadata. Strata reads both natively. About half of each file is the
n-gram (PLE) table, which stays on disk and is read row by row; it is not loaded into RAM or VRAM.
The draft head is Strata's own MTP runtime, prepared from the original model's MTP weights as for the other
@@ -97,7 +97,7 @@ Save as `strata-gyro-s.json` at the repository root:
```
```sh
-STRATA_APR=1 .venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080
+.venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080
```
The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1` (OpenAI) or
@@ -181,3 +181,5 @@ Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, J
- **AMD:** the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated
correctly (~28-31 tokens/s); no speed tuning or RDNA4 run yet.
- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest.
+- **Not validated end to end yet** (Strata runs them and logs a warning): an AMD GPU other than gfx1151, and
+ prompt processing on a peer GPU over P2P.
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 4ea168e88..881d1c88e 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -2028,6 +2028,11 @@ bool Prefill::set_peer(core::PeerExperts* peer, int64_t cap_rows, std::string& e
pp->compact ? (pp->ps_frac > 0.0 ? (", compact group buffers, streams " + std::to_string((int) (pp->ps_frac * 100 + 0.5)) +
"% of the primary's streamed experts through a " + std::to_string(pp->RP) + "-slot ring").c_str()
: ", compact group buffers") : "", fb >> 20);
+ // APR: this path rotates on the peer too (peer_had_signs), but no P2P pair has run it end to end with a
+ // Hadamard-folded pack yet (the validated two-GPU runs had no P2P, so their prompts stayed on the first GPU)
+ if (strata::kernels::cpu::hadamard_spec().any())
+ std::fprintf(stderr, "strata prefill: warning: prompt processing on a P2P peer GPU is not validated end to end "
+ "with Hadamard-folded experts yet\n");
m.pp = std::move(pp);
return true;
}
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index fb72da92e..e5a08ddff 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -3017,18 +3017,26 @@ int main(int argc, char** argv) {
#endif
}
// APR (prism.hadamard) packs: every expert path rotates its activations (CPU pool: native_quant_act / _h;
- // GPU verify, remote GPUs, prompt FP16 path: hadamard_rows), but the GPU side has been checked
- // by native_expert_parity only, not end to end on a GPU: opt-in until it has (STRATA_APR=1).
+ // GPU verify, remote GPUs, prompt FP16 path: hadamard_rows). Validated end to end on CUDA (sm_86, sm_89 with
+ // a two-GPU layer split and peer decode, sm_120) and on HIP gfx1151. Other HIP architectures run the same
+ // kernels but have not been run end to end with such a pack: said here, not refused (the P2P prompt peer
+ // says the same in Prefill::set_peer). The former opt-in STRATA_APR is no longer read.
if (strata::kernels::cpu::hadamard_spec().any()) {
- const char* apr = std::getenv("STRATA_APR");
- if (apr == nullptr || apr[0] != '1') {
- std::fprintf(stderr, "strata generate: this pack's experts are Hadamard-folded (hadamard.txt, an APR "
- "model), whose GPU path is not validated end to end yet; set STRATA_APR=1 to "
- "run it anyway\n");
- return 1;
- }
- std::fprintf(stderr, "strata generate: APR pack: Hadamard-folded experts (block %d), experimental\n",
+ std::fprintf(stderr, "strata generate: APR pack: Hadamard-folded experts (block %d)\n",
strata::kernels::cpu::hadamard_spec().block);
+#if defined(STRATA_USE_HIP) || defined(STRATA_HIP_GFX906)
+ int dev = 0;
+ cudaDeviceProp p{};
+ if (cudaGetDevice(&dev) == cudaSuccess && cudaGetDeviceProperties(&p, dev) == cudaSuccess) {
+ std::string arch(p.gcnArchName);
+ if (const size_t colon = arch.find(':'); colon != std::string::npos) arch.resize(colon);
+ if (arch != "gfx1151")
+ std::fprintf(stderr, "strata generate: warning: Hadamard-folded experts are validated end to end "
+ "on gfx1151 among AMD GPUs, not yet on %s\n", arch.c_str());
+ } else {
+ (void) cudaGetLastError();
+ }
+#endif
}
}
const bool native_pack = strata::kernels::cpu::expert_layout().native;
diff --git a/tools/gyro_setup.sh b/tools/gyro_setup.sh
index 73dc5cf96..de1e71132 100755
--- a/tools/gyro_setup.sh
+++ b/tools/gyro_setup.sh
@@ -34,7 +34,7 @@ if [ "$BACKEND" = cuda ]; then
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES="$CUDA_ARCH" -DSTRATA_GGML_DIR="$LLAMA"
elif [ "$BACKEND" = hip ]; then
- echo "note: AMD is experimental for Gyro (kernel parity validated on gfx1151, no end-to-end run yet)"
+ echo "note: AMD is experimental for Gyro (kernel parity and an end-to-end run validated on gfx1151 only)"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \
-DCMAKE_HIP_ARCHITECTURES="$HIP_ARCH" -DSTRATA_GGML_DIR="$LLAMA" \
-DCMAKE_PREFIX_PATH="${ROCM_PATH:-/opt/rocm}"
@@ -81,5 +81,5 @@ cat > "strata-$P.json" <<JSON
JSON
step "done. Start the server with:"
-echo " STRATA_APR=1 .venv/bin/python -m serve.server --engine strata --config strata-$P.json --port $PORT"
+echo " .venv/bin/python -m serve.server --engine strata --config strata-$P.json --port $PORT"
echo " web: http://127.0.0.1:$PORT API: http://127.0.0.1:$PORT/v1"