Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Split serving: docs pointers, check tool puts session files in /dev/shm
Niko122160e6a06086bc
diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.mdindex 36b35a81b..549c9796d 100644--- a/docs/MULTI_GPU.md+++ b/docs/MULTI_GPU.md@@ -90,6 +90,7 @@ estimated 2 GB of its own buffers.- Intel GPUs in a split, and a mix of NVIDIA and AMD cards. An engine drives one make of card. The app, `strata launch` and setupoffer only the cards the engine can drive, and say so when the other make has the bigger card. AMD cards share a model amongthemselves: `./setup.sh --backend hip --gpus 1,0` ([AMD_HIP.md](AMD_HIP.md)).+ A prompt read on one card and an answer made on another, with the session handed over in RAM, is a different opt-in: [SPLIT_SERVE.md](SPLIT_SERVE.md).## How the split worksdiff --git a/docs/README.md b/docs/README.mdindex ed4ebe063..c991c57f0 100644--- a/docs/README.md+++ b/docs/README.md@@ -26,6 +26,7 @@ Use the guide for your GPU and operating system. Model and feature support can d| Intel Arc on Linux | [Intel Arc](INTEL_ARC.md) || Apple Silicon | [Mac / Metal](MAC.md) || More than one GPU | [Multiple GPUs](MULTI_GPU.md) |+| Two cards: one reads the prompt, one makes the answer (opt-in) | [Split serving](SPLIT_SERVE.md) || Older GPUs | [Experimental hardware](OLDER_GPUS.md) |## Developers and contributorsdiff --git a/tools/split_serve_check.py b/tools/split_serve_check.pyindex 810ed036f..af89aceb2 100644--- a/tools/split_serve_check.py+++ b/tools/split_serve_check.py@@ -43,6 +43,18 @@ def read_ids(path):return [int(x) for x in open(path).read().replace("\n", ",").split(",") if x.strip()]+def session_dir(cfg):+ """Where the session files go: cfg["session_dir"], else /dev/shm when it has room (as the server chooses), else cfg["dir"]."""+ d = cfg.get("session_dir")+ if not d and os.path.isdir("/dev/shm"):+ import shutil+ if shutil.disk_usage("/dev/shm").free > (6 << 30):+ d = "/dev/shm/strata-split-check"+ d = d or cfg["dir"]+ os.makedirs(d, exist_ok=True)+ return d++def mem_available_gib():for line in open("/proc/meminfo"):if line.startswith("MemAvailable"):@@ -144,7 +156,7 @@ def cmd_exact(a, cfg):for _ in pe.generate(ids, 0, {}, cancel, prefill_only=True):passhp = state_hashes(log_tail(pe, pos))- path = os.path.join(cfg["dir"], "exact.session")+ path = os.path.join(session_dir(cfg), "exact.session")saved = pe.session_file("save", path)print(f"prefill engine: held {saved['tokens']} tokens, saved {saved['bytes'] / 2**20:.0f} MiB in {saved['ms']:.0f} ms",flush=True)@@ -227,7 +239,7 @@ def cmd_bench(a, cfg):"min_prompt_tokens": cfg.get("min_prompt_tokens", 2048)}})pe = make_engine(cfg, P, "prefill", extra_args=cfg.get("prefill_args", []), arena=arena)de = make_engine(cfg, Dg, "decode", extra_args=cfg.get("decode_args", []), arena=arena)- eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=cfg["dir"], say=print)+ eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=session_dir(cfg), say=print)closers = [pe, de]else:raise SystemExit("--arm layer|single|split")