Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Split serving: docs pointers, check tool puts session files in /dev/shm

Niko122160e6a06086bc
diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.md
index 36b35a81b..549c9796d 100644
--- a/docs/MULTI_GPU.md
+++ b/docs/MULTI_GPU.md
@@ -90,6 +90,7 @@ estimated 2 GB of its own buffers.
- Intel GPUs in a split, and a mix of NVIDIA and AMD cards. An engine drives one make of card. The app, `strata launch` and setup
offer only the cards the engine can drive, and say so when the other make has the bigger card. AMD cards share a model among
themselves: `./setup.sh --backend hip --gpus 1,0` ([AMD_HIP.md](AMD_HIP.md)).
+ A prompt read on one card and an answer made on another, with the session handed over in RAM, is a different opt-in: [SPLIT_SERVE.md](SPLIT_SERVE.md).
## How the split works
diff --git a/docs/README.md b/docs/README.md
index ed4ebe063..c991c57f0 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -26,6 +26,7 @@ Use the guide for your GPU and operating system. Model and feature support can d
| Intel Arc on Linux | [Intel Arc](INTEL_ARC.md) |
| Apple Silicon | [Mac / Metal](MAC.md) |
| More than one GPU | [Multiple GPUs](MULTI_GPU.md) |
+| Two cards: one reads the prompt, one makes the answer (opt-in) | [Split serving](SPLIT_SERVE.md) |
| Older GPUs | [Experimental hardware](OLDER_GPUS.md) |
## Developers and contributors
diff --git a/tools/split_serve_check.py b/tools/split_serve_check.py
index 810ed036f..af89aceb2 100644
--- a/tools/split_serve_check.py
+++ b/tools/split_serve_check.py
@@ -43,6 +43,18 @@ def read_ids(path):
return [int(x) for x in open(path).read().replace("\n", ",").split(",") if x.strip()]
+def session_dir(cfg):
+ """Where the session files go: cfg["session_dir"], else /dev/shm when it has room (as the server chooses), else cfg["dir"]."""
+ d = cfg.get("session_dir")
+ if not d and os.path.isdir("/dev/shm"):
+ import shutil
+ if shutil.disk_usage("/dev/shm").free > (6 << 30):
+ d = "/dev/shm/strata-split-check"
+ d = d or cfg["dir"]
+ os.makedirs(d, exist_ok=True)
+ return d
+
+
def mem_available_gib():
for line in open("/proc/meminfo"):
if line.startswith("MemAvailable"):
@@ -144,7 +156,7 @@ def cmd_exact(a, cfg):
for _ in pe.generate(ids, 0, {}, cancel, prefill_only=True):
pass
hp = state_hashes(log_tail(pe, pos))
- path = os.path.join(cfg["dir"], "exact.session")
+ path = os.path.join(session_dir(cfg), "exact.session")
saved = pe.session_file("save", path)
print(f"prefill engine: held {saved['tokens']} tokens, saved {saved['bytes'] / 2**20:.0f} MiB in {saved['ms']:.0f} ms",
flush=True)
@@ -227,7 +239,7 @@ def cmd_bench(a, cfg):
"min_prompt_tokens": cfg.get("min_prompt_tokens", 2048)}})
pe = make_engine(cfg, P, "prefill", extra_args=cfg.get("prefill_args", []), arena=arena)
de = make_engine(cfg, Dg, "decode", extra_args=cfg.get("decode_args", []), arena=arena)
- eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=cfg["dir"], say=print)
+ eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=session_dir(cfg), say=print)
closers = [pe, de]
else:
raise SystemExit("--arm layer|single|split")