Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
SYCL Gyro: trellis packs take the link share from a 12 GB/s reference (A750: 0.29 -> 0.48, decode 13.7 -> 15.8 tok/s, 5/5 pairs); Gyro memory warnings on an Arc read the real RAM
Niko12218bf77c075528
diff --git a/sycl/setup_intel.py b/sycl/setup_intel.pyindex cd6818246..c4395e424 100644--- a/sycl/setup_intel.py+++ b/sycl/setup_intel.py@@ -373,6 +373,7 @@ def install(argv) -> None:where = ("the experts the card does not hold are computed on the CPU" if alchelse "the experts are streamed into VRAM")msg = msg.replace(f"RAM: {fake_ram:.0f} GB", f"RAM: {real_ram:.0f} GB ({where})")+ msg = msg.replace(f"({fake_ram:.0f} GB RAM +", f"({real_ram:.0f} GB RAM +").replace(f"you have {fake_ram:.0f} GB RAM", f"you have {real_ram:.0f} GB RAM")if re.match(r"\s+\S+\s+needs ~\d+ GB RAM:", msg): # --check's CUDA verdicts: replaced by the Intel onem = msg.split()[0]d = S.MODELS.get(m, {})@@ -402,6 +403,11 @@ def install(argv) -> None:S.build_engine_hip = build_engine_hipS.hipblaslt_table = lambda *a, **k: NoneS.ram_gb = lambda: fake_ram # the experts are in VRAM: setup's RAM rule does not apply+ # Gyro (mem_total) counts VRAM + RAM together and is the one family whose experts are NOT streamed from the file into VRAM: on+ # an Arc they sit in the RAM arena and the CPU computes what the card does not hold. Its memory warning ("needs 27.6 GB of VRAM ++ # RAM together, you have ...", never a refusal) therefore reads the real RAM, not the 1024 GB above.+ mem_have = S.mem_have+ S.mem_have = lambda model, ram, vram=0.0: mem_have(model, real_ram if ram >= fake_ram else ram, vram)bench_tips = S.bench_tipsdiff --git a/sycl/src/program/generate.cpp b/sycl/src/program/generate.cppindex ca4b84142..99ff0f59a 100644--- a/sycl/src/program/generate.cpp+++ b/sycl/src/program/generate.cpp@@ -1627,8 +1627,8 @@ catch (sycl::exception const &exc) {// what the x16 share costs. Continuous (#485): before, 19.9 GB/s gave 0.42 and 20.0 the full 0.55 (and 4.0 GB/s// gave 0.08, 3.9 none), so a reading near either edge moved the share by a quarter of its range. From 20 GB/s up// (an x16 PCIe 4/5 link: ~26-28 GB/s) the share is unchanged.-double pcie_frac_for_gbps(double gbps, double base) {- return gbps <= 0.0 ? base : base * std::min(1.0, gbps / 20.0);+double pcie_frac_for_gbps(double gbps, double base, double ref_gbps = 20.0) {+ return gbps <= 0.0 ? base : base * std::min(1.0, gbps / ref_gbps);}// 0.1.42 (CUDA) / 0.1.48 (SYCL, opt-in STRATA_PCIE_FRAC_DEFAULT=auto): the share refined from what the CPU pool measures in@@ -2647,7 +2647,16 @@ int main(int argc, char **argv) try {if (!native_pack) {o.pcie_frac = base;} else if (bw > 0.0) {- o.pcie_frac = pcie_frac_for_gbps(bw, base);+ // Gyro's trellis experts: the CPU pool decodes them at ~8 GB/s (48 cycles a block on AVX2, compute bound, a third of the+ // i-quants' rate), so the link carries a bigger share of the misses than the 20 GB/s reference says. Measured on the+ // Arc A750 (10.5 GB/s link): 0.29 -> 12.8 tok/s, 0.45 -> 15.5, 0.60 -> 15.2 (5 interleaved pairs each). The reference+ // for these packs is 12 GB/s: 10.5 GB/s gives 0.48; from 12 GB/s up the measured 0.55 stays.+ bool trellis = false;+ {+ const auto& lay = strata::kernels::cpu::expert_layout();+ for (size_t l = 0; lay.native && l < lay.fmt.size(); ++l) trellis = trellis || lay.fmt[l].gu_type >= 144;+ }+ o.pcie_frac = pcie_frac_for_gbps(bw, base, trellis ? 12.0 : 20.0);std::fprintf(stderr, "strata generate: PCIe probe: %.1f GB/s host->device (best of %s) -> pcie_frac %.2f ""(default %.2f)\n", bw, bursts.c_str(), o.pcie_frac, base);{ // PcieAuto on SYCL: opt-in (STRATA_PCIE_FRAC_DEFAULT=auto), one card, not the xe NO_HOST path