Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

SYCL Gyro: trellis packs take the link share from a 12 GB/s reference (A750: 0.29 -> 0.48, decode 13.7 -> 15.8 tok/s, 5/5 pairs); Gyro memory warnings on an Arc read the real RAM

Niko12218bf77c075528
diff --git a/sycl/setup_intel.py b/sycl/setup_intel.py
index cd6818246..c4395e424 100644
--- a/sycl/setup_intel.py
+++ b/sycl/setup_intel.py
@@ -373,6 +373,7 @@ def install(argv) -> None:
where = ("the experts the card does not hold are computed on the CPU" if alch
else "the experts are streamed into VRAM")
msg = msg.replace(f"RAM: {fake_ram:.0f} GB", f"RAM: {real_ram:.0f} GB ({where})")
+ msg = msg.replace(f"({fake_ram:.0f} GB RAM +", f"({real_ram:.0f} GB RAM +").replace(f"you have {fake_ram:.0f} GB RAM", f"you have {real_ram:.0f} GB RAM")
if re.match(r"\s+\S+\s+needs ~\d+ GB RAM:", msg): # --check's CUDA verdicts: replaced by the Intel one
m = msg.split()[0]
d = S.MODELS.get(m, {})
@@ -402,6 +403,11 @@ def install(argv) -> None:
S.build_engine_hip = build_engine_hip
S.hipblaslt_table = lambda *a, **k: None
S.ram_gb = lambda: fake_ram # the experts are in VRAM: setup's RAM rule does not apply
+ # Gyro (mem_total) counts VRAM + RAM together and is the one family whose experts are NOT streamed from the file into VRAM: on
+ # an Arc they sit in the RAM arena and the CPU computes what the card does not hold. Its memory warning ("needs 27.6 GB of VRAM +
+ # RAM together, you have ...", never a refusal) therefore reads the real RAM, not the 1024 GB above.
+ mem_have = S.mem_have
+ S.mem_have = lambda model, ram, vram=0.0: mem_have(model, real_ram if ram >= fake_ram else ram, vram)
bench_tips = S.bench_tips
diff --git a/sycl/src/program/generate.cpp b/sycl/src/program/generate.cpp
index ca4b84142..99ff0f59a 100644
--- a/sycl/src/program/generate.cpp
+++ b/sycl/src/program/generate.cpp
@@ -1627,8 +1627,8 @@ catch (sycl::exception const &exc) {
// what the x16 share costs. Continuous (#485): before, 19.9 GB/s gave 0.42 and 20.0 the full 0.55 (and 4.0 GB/s
// gave 0.08, 3.9 none), so a reading near either edge moved the share by a quarter of its range. From 20 GB/s up
// (an x16 PCIe 4/5 link: ~26-28 GB/s) the share is unchanged.
-double pcie_frac_for_gbps(double gbps, double base) {
- return gbps <= 0.0 ? base : base * std::min(1.0, gbps / 20.0);
+double pcie_frac_for_gbps(double gbps, double base, double ref_gbps = 20.0) {
+ return gbps <= 0.0 ? base : base * std::min(1.0, gbps / ref_gbps);
}
// 0.1.42 (CUDA) / 0.1.48 (SYCL, opt-in STRATA_PCIE_FRAC_DEFAULT=auto): the share refined from what the CPU pool measures in
@@ -2647,7 +2647,16 @@ int main(int argc, char **argv) try {
if (!native_pack) {
o.pcie_frac = base;
} else if (bw > 0.0) {
- o.pcie_frac = pcie_frac_for_gbps(bw, base);
+ // Gyro's trellis experts: the CPU pool decodes them at ~8 GB/s (48 cycles a block on AVX2, compute bound, a third of the
+ // i-quants' rate), so the link carries a bigger share of the misses than the 20 GB/s reference says. Measured on the
+ // Arc A750 (10.5 GB/s link): 0.29 -> 12.8 tok/s, 0.45 -> 15.5, 0.60 -> 15.2 (5 interleaved pairs each). The reference
+ // for these packs is 12 GB/s: 10.5 GB/s gives 0.48; from 12 GB/s up the measured 0.55 stays.
+ bool trellis = false;
+ {
+ const auto& lay = strata::kernels::cpu::expert_layout();
+ for (size_t l = 0; lay.native && l < lay.fmt.size(); ++l) trellis = trellis || lay.fmt[l].gu_type >= 144;
+ }
+ o.pcie_frac = pcie_frac_for_gbps(bw, base, trellis ? 12.0 : 20.0);
std::fprintf(stderr, "strata generate: PCIe probe: %.1f GB/s host->device (best of %s) -> pcie_frac %.2f "
"(default %.2f)\n", bw, bursts.c_str(), o.pcie_frac, base);
{ // PcieAuto on SYCL: opt-in (STRATA_PCIE_FRAC_DEFAULT=auto), one card, not the xe NO_HOST path