Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Fix #3425: the memory tiers count RAM + VRAM less 5 GB that Windows and the system keep (setup.py and tools/strata_mcp.py, one constant each; the app and strata launch read the engine's pick; the Mac keeps its own macOS share)
Niko122176f90e58436d
diff --git a/README.md b/README.mdindex eca102aa1..138657921 100644--- a/README.md+++ b/README.md@@ -164,6 +164,8 @@ Gyro runs on NVIDIA and AMD cards (AMD is experimental) on Intel Arc cards on Li### The other sizes+The tiers count your RAM plus your VRAM, less 5 GB that Windows and the system keep for themselves. Every other size stays a choice.+| Your memory (RAM + VRAM) | Take | Notes || --- | --- | --- || **32 GB** | **Gyro-S**, or the **[Coder](docs/MODELS.md#coder)** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code |diff --git a/docs/MODELS.md b/docs/MODELS.mdindex f3b03ba3c..f60f94c89 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -14,7 +14,7 @@ makers' compressions). It also runs a smaller model, [Qwen3.6-35B-A3B](#qwen36-3The tiers count RAM and VRAM (the graphics card's memory) together, the same as the [README](../README.md#the-other-sizes).`strata launch`, setup and the install tool of the MCP server pre-select by them; the desktop app does from 0.2.1. Every-other size stays a choice. A PC with a card of 12 GB and 32 GB of RAM is in the 40 GB tier (32 + 12 = 44).+other size stays a choice. Strata leaves 5 GB for Windows and the system, so the sum is RAM + VRAM minus 5 GB: a PC with a card of 12 GB and 32 GB of RAM is in the 40 GB tier (32 + 12 - 5 = 39), and a PC with a 16 GB card and 32 GB of RAM is too, not in the 48 GB tier (#3425).| Your memory (RAM + VRAM) | Pre-selected | Notes || --- | --- | --- |diff --git a/docs/release-notes-0.2.2.md b/docs/release-notes-0.2.2.mdindex ebb290782..9486736b4 100644--- a/docs/release-notes-0.2.2.md+++ b/docs/release-notes-0.2.2.md@@ -26,6 +26,8 @@- **Problems with the computer's own setup are repaired or explained, not refused.** If the Windows runtime folder lists another account, Strata locks it down itself instead of stopping. A Documents folder that cannot be used (Controlled folder access, an offline OneDrive) falls back to the Strata home folder and Settings > Storage says so. A port that is taken or reserved gets one sentence with the next step. Local requests ignore a system proxy. File-in-use errors and a full disk say what happened. `strata launch` asks plain questions in a terminal that reports a tty late. `strata update` copes with a slow `--version`, a busy graphics card is not held against an engine, pre-release Control builds hand over correctly, and a wrong PC clock is named as the reason a feed or certificate looks bad. install.sh and install.ps1 give clearer messages (ARM, TLS 1.2, proxy credentials, no `tar.exe`). The Terminal parts are already in 0.2.1.3; the Documents fallback and the port handling are app parts.- **A stale update notice goes away.** After a version change the app and the Terminal no longer keep showing an offer for a version you already have.- **Idle GPU time was measured, with no default change.** On a two-card test PC decode leaves 47 to 49% of the card time idle (batch 51%, prompts 24 to 42%), on a one-card PC 5%. Overlap gave +0.3, +2.1 and -0.3% against 2.8% noise; pipeline windows 2 gained 5.4% on code only, cost 5 to 13% with the draft layer on the first card, and `--batch-groups 2` cost 15.3% with 4 chats, so nothing is switched on. Raw numbers are in `bench/results/2026-10-20-0.2.1-idle-gpu/`.+- **The recommended size leaves 5 GB for Windows.** The memory tiers now count RAM + VRAM less 5 GB, so a PC with 32 GB of RAM and a 16 GB card gets the 40 GB tier instead of the 48 GB one, which had it reading part of the model from the SSD (#3425). Every other size is still a choice.+## Faster, with what we measureddiff --git a/setup.py b/setup.pyindex 0d37d7d74..8358380b9 100644--- a/setup.py+++ b/setup.py@@ -3441,6 +3441,10 @@ def mem_have(model: str, ram: float, vram: float = 0.0) -> float:# (35 GB) as the other choice. Below 40 GB the older rules stand (Gyro-S, or the Coder for code). The Mac has its own# rules (strata_control/mac_fit.py). tools/strata_mcp.py's recommend() holds the same floors (the app and `strata launch`).MEMORY_TIERS = ((62.0, "qwen", "IQ3_S"), (46.0, "qwen", "IQ3_XXS"), (38.0, "swift", "IQ2_XS"))+# #3425 (owner's rule): the tiers count RAM + VRAM less this much, which Windows and the system keep for themselves. A PC+# with a 16 GB card and 32 GB of RAM used to land in the 48 GB tier and then read ~11 GB of it from the SSD on every answer.+# tools/strata_mcp.py has the same constant; the Mac (strata_control/mac_fit.py) keeps its own macOS share and does not use it.+TIER_SYSTEM_RESERVE_GB = 5.0def tier_pick(ram: float, vram: float = 0.0):@@ -3448,7 +3452,7 @@ def tier_pick(ram: float, vram: float = 0.0):Gyro-S, or the Coder). With less RAM than the size's own figure the low-RAM mode starts by itself and the model isslower. A recommendation: every size stays a choice."""for floor, fam, m in MEMORY_TIERS:- if ram + vram >= floor:+ if ram + vram - TIER_SYSTEM_RESERVE_GB >= floor:return fam, mreturn Nonediff --git a/tools/strata_mcp.py b/tools/strata_mcp.pyindex 916bbf037..defaed0aa 100644--- a/tools/strata_mcp.py+++ b/tools/strata_mcp.py@@ -118,6 +118,8 @@ FALLBACK_FAMILIES = {}OWN_PATH_ENGINE = (0, 1, 45) # the first engine that runs Qwen3.6-35B-A3B with pictures and parallel requests (setup.py MIN_ENGINE note)MEMORY_TIERS = ((62.0, "qwen", "IQ3_S"), (46.0, "qwen", "IQ3_XXS"), (38.0, "swift", "IQ2_XS")) # as setup.py's: RAM + VRAM floors of the owner's 64 / 48 / 40 GB tiers+# #3425 (owner's rule, same as setup.py's TIER_SYSTEM_RESERVE_GB): the floors count RAM + VRAM less the 5 GB Windows and the system keep+TIER_SYSTEM_RESERVE_GB = 5.0GYRO_ROOM_GB = 4.0 # recommend() suggests a Gyro size only when VRAM + RAM exceed its memory figure by this muchFALLBACK_CONTEXTS = [8192, 32768, 65536, 131072, 204800, 262144, 393216, 524288]BENCH_PROMPT = ("Write a short story (about 300 words) about a lighthouse keeper who finds a message in a bottle. "@@ -1115,10 +1117,10 @@ class Strata:# 0.2.0, the owner's memory tiers (README "The other sizes"; setup.py MEMORY_TIERS holds the same floors): RAM + VRAM# together. An APU's shared memory is the RAM itself, so only its BIOS carve-out counts as VRAM.tvram = float(best.get("dedicated_gb", 0.0)) if best.get("uma") else float(vram)- tier = next(((f, n) for lo, f, n in MEMORY_TIERS if ram + tvram >= lo), None)+ tier = next(((f, n) for lo, f, n in MEMORY_TIERS if ram + tvram - TIER_SYSTEM_RESERVE_GB >= lo), None)if tier and tier[1] in models and tier[0] in families:fam, model = tier- total = f"{ram:.0f} GB of RAM + {tvram:.0f} GB of VRAM = {ram + tvram:.0f} GB"+ total = f"{ram:.0f} GB of RAM + {tvram:.0f} GB of VRAM = {ram + tvram:.0f} GB, {TIER_SYSTEM_RESERVE_GB:.0f} GB of it kept for Windows and the system"if model == "IQ2_XS":why = (f"{total}, the 40 GB tier: Swift 1.5's IQ2_XS (thinks for a shorter time)"+ ("; Gyro-M (35 GB) is the other choice there, close to IQ3_XXS in quality but 16-21% slower at "diff --git a/tools/test_setup_golden.json b/tools/test_setup_golden.jsonindex 430fc2992..e4e654c2e 100644--- a/tools/test_setup_golden.json+++ b/tools/test_setup_golden.json@@ -739,11 +739,11 @@"config": {"args": ["--pack",- "<T>/data/packs/iq3_s",+ "<T>/data/packs/iq3_xxs","--native",- "<T>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",+ "<T>/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf","--ple-gguf",- "<T>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",+ "<T>/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf","--expert-profile","<T>/data/expert-profile.bin","--expert-cache",@@ -779,16 +779,16 @@"lib_dirs": ["<cuda>"],- "log": "<T>/strata-iq3_s.log",- "model_name": "qwen3.8-flash-next-iq3_s",+ "log": "<T>/strata-iq3_xxs.log",+ "model_name": "qwen3.8-flash-next-iq3_xxs","port": 8080,- "tokenizer": "<T>/data/packs/iq3_s/tokenizer",+ "tokenizer": "<T>/data/packs/iq3_xxs/tokenizer","vision": {"exe": "<T>/engine/strata-vision.exe","gpu": false,"max_tokens": 300,"mmproj": "<T>/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",- "model": "<T>/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",+ "model": "<T>/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf","threads": 6}}diff --git a/tools/test_setup_gyro.py b/tools/test_setup_gyro.pyindex 7b041b3f2..ae8b40cb7 100644--- a/tools/test_setup_gyro.py+++ b/tools/test_setup_gyro.py@@ -62,10 +62,10 @@ class GyroCatalog(unittest.TestCase):self.assertEqual(s.recommend(hardware(ram, vram))["family"], family, (ram, vram))got = s.recommend(hardware(20.0, 12.0)) # 32 GB in total, the Coder's 32 GB of RAM missingself.assertEqual((got["family"], got["model"]), ("gyro", "Gyro-S"))- got = s.recommend(hardware(24.0, 16.0)) # 40 GB in total: the owner's 40 GB tier, Swift 1.5's IQ2_XS+ got = s.recommend(hardware(28.0, 16.0)) # 44 GB in total, 39 after the 5 GB for the system (#3425): the owner's 40 GB tier, Swift 1.5's IQ2_XSself.assertEqual((got["family"], got["model"]), ("swift", "IQ2_XS"))self.assertIn("Gyro-M", got["why"]) # named as the other choice (NVIDIA, AMD and Intel Arc)- intel = hardware(24.0, 16.0)+ intel = hardware(28.0, 16.0)intel["gpus"] = [{**g, "vendor": "intel"} for g in intel["gpus"]]self.assertIn("Gyro-M", s.recommend(intel)["why"]) # the SYCL engine has the trellis kernels (0.2.2)self.assertIsNone(s.recommend(hardware(12.0, 8.0))["family"])diff --git a/tools/test_setup_menu.py b/tools/test_setup_menu.pyindex d59e9d19c..d6b0e9398 100644--- a/tools/test_setup_menu.py+++ b/tools/test_setup_menu.py@@ -103,20 +103,35 @@ class RecommendedSizes(unittest.TestCase):for fam in ("qwen", "swift"):n = self.names(fam)self.assertIn("IQ3_S", n)- self.assertEqual(setup.recommended_size(fam, n, 63.7), "IQ3_S")+ self.assertEqual(setup.recommended_size(fam, n, 67.7), "IQ3_S")self.assertEqual(setup.recommended_size(fam, n, 191.0), "IQ3_S") # not the largest: IQ3_S stays the pickself.assertEqual([m for m in n if setup.size_tag(fam, m)], ["IQ3_S"]) # the others are choices, unmarkeddef test_without_the_ram_the_largest_that_fits_and_iq3_s_stays_marked(self):n = self.names("qwen")self.assertEqual(setup.recommended_size("qwen", n, 61.0), "IQ3_XXS") # IQ3_S needs 62 GB- self.assertEqual(setup.recommended_size("qwen", n, 50.0), "IQ3_XXS") # 48 GB tier (RAM + VRAM)- self.assertEqual(setup.recommended_size("qwen", n, 40.0, 8.0), "IQ3_XXS") # 40 + 8 = 48 GB together- self.assertEqual(setup.recommended_size("qwen", n, 32.0, 12.0), "IQ2_XS") # 44 GB: the 40 GB tier+ self.assertEqual(setup.recommended_size("qwen", n, 52.0), "IQ3_XXS") # 48 GB tier (RAM + VRAM - 5 GB)+ self.assertEqual(setup.recommended_size("qwen", n, 40.0, 11.0), "IQ3_XXS") # 40 + 11 - 5 = 46 GB together+ self.assertEqual(setup.recommended_size("qwen", n, 32.0, 12.0), "IQ2_XS") # 44 - 5 = 39 GB: the 40 GB tierself.assertEqual(setup.recommended_size("swift", self.names("swift"), 32.0, 12.0), "IQ2_XS")- self.assertEqual(setup.recommended_size("qwen", n, 32.0, 32.0), "IQ3_S") # 64 GB together+ self.assertEqual(setup.recommended_size("qwen", n, 32.0, 35.0), "IQ3_S") # 67 - 5 = 62 GB togetherself.assertEqual(setup.size_tag("qwen", "IQ3_S"), " (recommended)")+ def test_3425_the_tiers_leave_5_gb_for_windows(self):+ # the reporter: RTX 5080 (16 GB) + 32 GB of RAM summed to 48 GB and got IQ3_XXS, which read ~11 GB from the SSD+ n = self.names("qwen")+ self.assertEqual(setup.TIER_SYSTEM_RESERVE_GB, 5.0)+ self.assertEqual(setup.recommended_size("qwen", n, 31.9, 15.9), "IQ2_XS")+ self.assertEqual(setup.tier_pick(31.9, 15.9), ("swift", "IQ2_XS"))+ # boundaries: floors 38 / 46 / 62 plus the 5 GB+ self.assertEqual(setup.tier_pick(43.0), ("swift", "IQ2_XS"))+ self.assertIsNone(setup.tier_pick(42.9))+ self.assertEqual(setup.tier_pick(51.0), ("qwen", "IQ3_XXS"))+ self.assertEqual(setup.tier_pick(50.9), ("swift", "IQ2_XS"))+ self.assertEqual(setup.tier_pick(67.0), ("qwen", "IQ3_S"))+ self.assertEqual(setup.tier_pick(66.9), ("qwen", "IQ3_XXS"))+ self.assertEqual(setup.tier_pick(40.0, 3.0), ("swift", "IQ2_XS")) # VRAM counts the same as RAM+def test_coder_has_nothing_marked(self):n = self.names("coder")self.assertEqual([setup.size_tag("coder", m) for m in n], [""] * len(n))diff --git a/tools/test_strata_mcp.py b/tools/test_strata_mcp.pyindex 62230b8ef..7716051ca 100644--- a/tools/test_strata_mcp.py+++ b/tools/test_strata_mcp.py@@ -664,15 +664,34 @@ class Helpers(unittest.TestCase):self.assertEqual(r["context"], 32768)def test_a_pick_that_reads_from_the_ssd_says_so(self):- # #3425: 32 GB RAM + 16 GB VRAM is the 48 GB tier; about 11 GB of IQ3_XXS would come from the SSD+ # #3425: 36 GB RAM + 16 GB VRAM - 5 GB is the 48 GB tier; about 8 GB of IQ3_XXS would come from the SSDs = M.Strata(HERE.parent)gpu = [{"index": 0, "name": "x", "vram_gb": 15.9, "usable": True, "vendor": "nvidia"}]- r = s.recommend({"ram_gb": 31.9, "gpus": gpu, "cpu": {"avx2": True}})+ r = s.recommend({"ram_gb": 36.0, "gpus": gpu, "cpu": {"avx2": True}})self.assertEqual((r["family"], r["model"]), ("qwen", "IQ3_XXS"))self.assertTrue(any("read from the SSD" in n for n in r["notes"]), r["notes"])roomy = s.recommend({"ram_gb": 64, "gpus": gpu, "cpu": {"avx2": True}})self.assertFalse(any("read from the SSD" in n for n in roomy["notes"]), roomy["notes"])+ def test_3425_the_tiers_leave_5_gb_for_windows(self):+ # the reporter: RTX 5080 (16 GB) + 32 GB of RAM used to sum to 48 GB and get IQ3_XXS; now the 40 GB tier+ s = M.Strata(HERE.parent)+ gpu = [{"index": 0, "name": "x", "vram_gb": 15.9, "usable": True, "vendor": "nvidia"}]+ self.assertEqual(M.TIER_SYSTEM_RESERVE_GB, 5.0)+ r = s.recommend({"ram_gb": 31.9, "gpus": gpu, "cpu": {"avx2": True}})+ self.assertEqual((r["family"], r["model"]), ("swift", "IQ2_XS"))+ # boundary: 43 GB of RAM + VRAM is the lowest 40 GB tier PC, 51 GB the lowest 48 GB tier PC+ def pick(ram, vram):+ g = [{**gpu[0], "vram_gb": vram}]+ r = s.recommend({"ram_gb": ram, "gpus": g, "cpu": {"avx2": True}})+ return r["family"], r["model"]+ self.assertEqual(pick(30.0, 13.0), ("swift", "IQ2_XS"))+ self.assertNotEqual(pick(30.0, 12.9), ("swift", "IQ2_XS"))+ self.assertEqual(pick(38.0, 13.0), ("qwen", "IQ3_XXS"))+ self.assertEqual(pick(38.0, 12.9), ("swift", "IQ2_XS"))+ self.assertEqual(pick(54.0, 13.0), ("qwen", "IQ3_S"))+ self.assertEqual(pick(54.0, 12.9), ("qwen", "IQ3_XXS"))+class PhysicalCores(unittest.TestCase):"""#3417: the hardware report's cpu.cores is the physical cores; the Work profile takes 80% of it for --pool-workers."""