Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Measurement records
Raw gate records for Soup's layer-streaming feature and its release gate, published as written.
These are not a report assembled after the fact. They are the working records kept while each item was built and verified, so they contain the failures, the assumptions that turned out wrong, and the numbers that were measured and then discarded — in the order those things happened.
They are the evidence behind the preprint:
Makazhan, A. (2026). Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU. Zenodo. 10.5281/zenodo.21771064
| File | What it gates | Headline |
|---|---|---|
gate-674-quest-mixed-route.md |
Evaluation-only confirmation of the mixed W4/A4+A16 route retained after strict W4A4 failed (#674); not training-quality or efficiency evidence | Gate passed: the predeclared seven-A16 rescue measured 0.086344 nat [0.080241, 0.093190] above fixed FP over 704 examples / 25,017 targets and improved native W4A4 by -0.015452 [-0.017552, -0.013528]. Three training seeds, one model, fake quantization, zero backwards/updates; FINAL256 remains untouched. |
gate-674-quest-w4a4-sft.md |
Experimental QuEST W4A4 SFT feasibility (#674); not an integration | Gate failed: 0.114 nat [0.099, 0.130] above the strongest retained FP on the same panel. One training seed, one model, fake quantization; no parity or efficiency claim. Retains failed directions, costs, source/result hashes and spent-panel history; FINAL256 remains untouched. |
quest-674-postfit-development-results.md |
Best retained post-fit QuEST checkpoint against the strongest retained FP on the reused DEV704 panel (#674); descriptive after selection | Gate failed: Q/FP response-NLL ratio 1.038367, gap +0.078519 nat/target with paired 95% interval [+0.071493, +0.086026] over 704 examples / 25,017 targets. Preserves the checkpoint recipe and nine negative or inconclusive follow-ups; FINAL256 remains untouched. |
quest-674-development-followup.md |
W4A16 reconstruction and native QuEST follow-ups for #674 | W4A16 DEV2048 ratios 1.016327 / 1.016500; zero-margin parity still fails. Keeps all replicas, 642 audited aggregate contrasts, native strict/mixed negative results and the prospective acceptance question. No production or efficiency claim. |
gate-655-pinned-store-accounting.md |
Pinned host-memory accounting and the /dev/shm preflight decision (#655) |
Two pinned 4 GiB allocations on an L4/Linux/CUDA stack each add exactly 4 GiB to RssShmem without using /dev/shm; the pageable control adds 4 GiB to RssAnon. Preserves the original host-counter drift and probe. |
gate-v0.72.0-layer-streaming.md |
The streaming path itself | Bit-exactness vs a resident reference; 3B bf16 trained on a 4 GB card |
gate-v0.72.2-nf4.md |
NF4 quantised streaming | Llama-3.1-8B at 119.6 tok/s at 952 MHz in a 3.32 GB peak (pre-repair); post-#331 row measured 2026-09-19 (#361): 208.6 tok/s in a 2.40 GB peak at 1935 MHz — read with its clock, not directly comparable |
gate-v0.72.3-breadth.md |
Nine architectures, batching, accumulation, resume, disk tier | Peak-VRAM predictor at 0.85% worst-case error; accumulation is per-token I/O-neutral |
gate-v0.72.4-preference-losses.md |
DPO / ORPO / SimPO / KTO over the streaming engine | DPO's reference model costs no extra weights — 0.914x the SFT peak, against +730.44 MB for a real second instance |
probe-v0.73.0-what-bounds-streaming.md |
What the streamed step is actually bound by, and Cut Cross-Entropy on top of it | Not transfer-bound: 71.3% of the card's same-session GEMM ceiling, and deleting every host-to-device byte buys 1.4%. CCE triples the usable microbatch for +9.6% |
probe-rtx5070-l2l-step0.md |
Layer-major micro-batching (L2L) on the shipped streamed runtime, step 0 of the giant-MoE track: one load per layer per step for k micro-batches, measured on the RTX 5070 box against gradient accumulation on the cold 70B-shaped NF4 disk tier | G1 and G2 pass, G3 and G4 no verdict. L2L's LoRA gradients and losses are bit-exact to GA on Mistral-7B (k = 4) and the 70B shape (k = 2) under deterministic algorithms (GA is not reproducible against itself without them); the read hides at k = 16 (73.3 tok/s against 0.8 x 73.4); G3 and G4 have no verdict because the k = 1 L2L arm's two rounds differ by 14.1% (one 25.0 s step). Three informational arms were voided by a trigger-restarted SearchIndexer.exe |
run-t4-colab-free-tier.md |
Not a gate — one completed run on hardware the maintainer does not own | An 8B NF4 streamed run finishes on a free-tier Colab T4 (sm_75, Turing) inside a 4.00 GB process cap, peak 2.91 GB against a predicted 3.02 (the estimator over-predicts by 3.8%, the safe direction). A second run on the same card (#844) measured 1.83 GB peak instead — the embeddings and head moved out of the resident allocation into a streamed large-layer slot between the two runs, which explains both figures; both are recorded. No throughput is quoted — a capped card is not a benchmark — and gradient exactness on Turing is not shown |
gate-v0.73.1-measured-vram-fit.md |
Measuring the streaming VRAM fit instead of predicting it | The peak-VRAM formula under-predicts at long sequence — 0.934x the real peak at seq 5120 and 0.787x at 6144, measured through the real soup train, a direction the v0.72.3 grid could not see because all ten of its rows sit at seq 256 or 512. Carries three readings that were withdrawn during the work, including two that looked like the headline result |
gate-395-second-stack-vram.md |
The VRAM fit on a second GPU/stack — the verification gate-v0.73.1 could not do on one card |
The under-prediction does not reproduce: flat 1.16x over-prediction across seq 2048-6144 on two models, against 0.934x / 0.787x on the RTX 3050. The gap is a single stack-specific term, not drift with sequence. The SDPA math-path hypothesis is published as a negative result with a positive control, and explicitly not extended to the 3050. Carries a discarded first sweep whose numbers were an artifact of max_length truncating rather than padding |
gate-v0.73.2-leg2-scoring.md |
soup ship's leg-2 scorers — the release gate itself, not streaming |
A suite scored 0.000 for a stub answering every item correctly, twice over: \boxed{C} was unknown to the MCQ extractor, and a tool call one closing brace short fell through to the inner object. Two models with byte-identical scores on all seven suites, one refusing every benign request, were indistinguishable to the gate. The measured noise floor on this box is 0.0000 — CPU greedy decode is deterministic, and the H100's 0.015/0.020 is explicitly not re-claimed here. Carries a withdrawn order-dependence scare, a control that varied nothing, and a review finding that was checked against three implementations and partly rejected |
gate-v0.76.0-cuda-graph-decode.md |
Opt-in CUDA graph decoding for soup infer / soup bench infer (--cuda-graphs), gated UNPINNED against the same-day tree without the flag, A-B-B-A-B-A in fresh processes, on the RTX 5070 box |
All four pre-declared criteria passed: Qwen2.5-1.5B FP16 decode, slowest graph run 2.90x the fastest baseline run (60.10 vs 20.71 tok/s), 2.76x under an unmerged LoRA (49.19 vs 17.80), +42 / +40 MiB reserved, identical token IDs on all 84 harness requests; soup infer end to end over 24 varied prompts, load and compile included, 282 / 131 s -> 102 / 96 s with identical responses. Keeps the pinned bridge to the experiment's 32.16 -> 62.57 (31.29 -> 65.64 here), the finding that the UNPINNED baseline is the unstable arm (13.5-20.7 tok/s run to run against 4% for graphs), and a 1024-row re-record probe showing no eager fallback and flat memory. Launched from the agent session in the background, not the owner's foreground window; the record says so |
gate-v0.76.0-tool-call-discrimination.md |
mini_tool_call discrimination after #405 |
Qwen2.5-7B-Instruct moves from the shipped 1.000 ceiling to 0.850 (34/40) while SmolLM2-135M-Instruct scores 0.050 (2/40); the record keeps the rejected 0.969 candidate and the provenance-scale change. |
gate-v0.76.0-arithmetic-discrimination.md |
mini_arithmetic discrimination for #1192 |
Qwen2.5-1.5B-Instruct scores 0.800 (32/40) and SmolLM2-135M-Instruct 0.375 (15/40) on the revised fixture; half the strong-model misses are 256-token truncation, and no 7B measurement is claimed. |
gate-v0.76.0-common-sense-discrimination.md |
mini_common_sense discrimination for #1192 |
Qwen2.5-1.5B-Instruct scores 0.850 (34/40) and SmolLM2-135M-Instruct 0.275 (11/40) on the revised fixture, from a 24-row candidate pool; the answer key no longer lets an always-"B" model score 0.583. No 7B measurement is claimed. |
gate-v0.76.0-mmlu-discrimination.md |
mini_mmlu discrimination for #1192 |
Qwen2.5-1.5B-Instruct moves from 1.000 (26/26) to 0.850 (34/40) and SmolLM2-135M-Instruct scores 0.350 (14/40), at chance; 14 rows selected from two 24-row candidate pools, and the answer key no longer lets an always-"B" model score 0.462. No 7B measurement is claimed. |
run-m1-8gb-mlx-sft.md |
Not a gate — the MLX backend's training loop on hardware CI does not have | MLXSFTTrainerWrapper runs end to end on an 8 GB M1, and the shipped llama3.1-8b-sft-mlx recipe fits: peak 5.154 GB, 48 iterations in 71 s. A model too large does not fail here, it pages — free memory sat at 5-6% while the machine stayed responsive, so an allocation-failure pre-flight would not fire (the #649 trap from the other platform). Carries a failed fixture — the model id #23 itself recommends ships legacy weights.NN.safetensors and cannot be loaded — and two corrections I made mid-measurement |
gate-qwen4-ple-m4-max.md |
Qwen4-Exp/oQ decoder mapping and read-only PLE on Apple Silicon | Partial pass only: 1,167/1,167 cached tensors and tiny CPU/MPS parity passed; the production optimizer step was stopped and is explicitly not validated |
gate-h100-validation.md |
The method on someone else's hardware: bit-exactness at real sizes, convergence quality, DeepSpeed, variance | Forward bit-exact to 72B; backward bit-exact to 14B NF4 pre-repair, re-gated after the STEP 14 fix at 32B (256/256) and at 72B (320/320, the size where the defect was worst); 2.93x DeepSpeed ZeRO-3 offload in 9.7x less VRAM; and the silent wrong-gradient defect that fix repairs. Carries three dated 2026-08-13 corrections: it explains the H100 replication as host-to-device transfer, which the probe record above later measured and refuted. The original lines are left standing with the correction beside them |
probe-rtx5070-what-bounds-streaming.md |
The FIRST baseline on the new dev box (RTX 5070 Laptop 8 GB, PCIe 5.0 x8, 32 GB RAM, 2x NVMe) — the same "what bounds the streamed step" question as probe-v0.73.0, plus the shipped DISK tier warm and cold, through a re-runnable harness |
RAM tier is still compute-bound (Mistral-7B NF4, batch 1 x 512: 486 tok/s, compute-stream stall 0.34%), but the copy stream is now 42% busy and the NF4 dequantisation is 23% of the step (was 9.8%) — #842 is worth a quarter of the step here. The shipped disk tier is 2.3x slower than RAM with the whole store in the page cache (the Python thread blocks in get_tensor; GPU stall ~0), and cold on a 36 GB 70B-shaped NF4 store it delivers 4.1 tok/s — 124 s per step, 0.57 GB/s through mmap page faults on an NVMe that reads at 3.5+ GB/s. Also finds that the shipped sharder dies on Windows past ~100 GB of live safetensors mappings (a real 70B is 141 GB), reproduced without Soup (#926); sharded through a documented harness-side workaround §21 (2026-09-15, on the post-#974 direct-I/O reader): the cold 70B step is read-bound outright — 17.0–17.2 s at seq 512 AND at seq 256 while the compute floor halves (6.4 → 3.8 s) — so the per-visit NF4 dequantisation (#842: 8–9 ms/visit, 1.3–1.5 s/step, 23% of the compute floor) and the pure per-visit fixed cost (#841: 0.65 ms/visit, ~0.1 s/step) are both hidden inside the read; #841 folds into #842, which is the RAM-tier lever and 0% on the cold disk tier. The step moves 70.4 GB at 4.1 GB/s averaged, 73% of the drive's 5.65 GB/s unbuffered ceiling |
probe-rtx5070-batch-control.md |
R2 of the disk-tier read-rate track, and the control arm of the layer-major (L2L) experiment: on the cold 70B-shaped NF4 step at seq 512, does tok/s scale with batch 1 -> 2 -> 3 as the read-bound model of probe-rtx5070-what-bounds-streaming.md §21 predicts (the step is the larger of the read and a compute floor that grows with the tokens), on one drive and on two striped drives, and what peak VRAM does each batch need on the 8 GB card? Two rounds, the second reversed, one process per arm, with the read-free floor measured in session; an out-of-memory error, an arm past its cap or a WDDM spill (the card as full as §18's spill and the step slower than read plus compute in series) is a fit result, not a slow step. Judged per batch against the model's predicted class under a rule committed before the run |
Rule committed 2026-10-01; results pending. No arm has run. The model predicts one drive near-free to batch 3 (x2.00, x3.00) and two drives compute-bound from batch 2 (x1.72, x1.78), so that the second drive, worth 1.75x at batch 1, would be worth about 4% at batch 3 |
gate-971-async-nvme-source.md |
The asynchronous disk-tier source (#971) — a background reader with its own header parsing (no mmap) and pinned host staging, measured cold and warm against a SAME-DAY control of the source it replaces | Cold, the step is 2.1x-3.1x faster: a 70B-shaped NF4 step at batch 1 x 512 goes 100.35 -> 48.09 s early and 92.25 -> 30.28 s late, each pair position-matched (5.1-5.6 -> 10.6-16.9 tok/s; 0.70-0.76 -> 1.46-2.32 GB/s at the source; 2.59x against the other record's published 124.5 s). Warm it is a REGRESSION — 1.03-1.20x slower than the source it replaces with the whole store in the page cache, the two figures being the two run orders, where §8's published before would have made the same number look like a 1.33x improvement. Carries a depth result that turned out to be an ordering result (the same config is 48.09 s run first and 30.28 s run last, larger than the whole spread across depths 1/2/4), a localisation that was wrong and is corrected in place, an 11.35x instrumented outlier that did not reproduce, and a 2026-09-14 revision pass that re-measured §8 from the committed harness, re-ran the warm trio in reverse order, and dated six corrections where the text claimed more than the JSON supported |
gate-974-disk-tier-direct-io.md |
The disk tier's reads, warm and cold (#974) — the reported warm regression re-measured position-matched, the cold read primitive measured buffered against unbuffered on a store larger than RAM, then the direct-I/O range reader gated in both run orders against the shipped synchronous source AND the reader it replaces (imported from origin/main, the tree stamped in every JSON) |
The warm regression does not exist position-matched (0.91–0.97x of the synchronous control before the change, 0.76–0.94x of the reader it replaces after it), and cold the 70B-shaped step is 16.2–17.3 s at BOTH run positions — 6.4x the synchronous path in the same slot (110.5 s), 2.0–2.8x the reader it replaces (34.8–45.6 s), read share 84.7% -> 62–64% at 6.0–7.0 GB/s, the two positions within 1.07x where the old reader's differed 1.31x; peak VRAM unchanged at 4.38 GB. The reason: cold, every BUFFERED read primitive tops out at 1.2–2.9 GB/s on this NVMe whatever the shape, unbuffered reads of the same layers reach 3.5–5.65. Keeps a first cold series killed by a peer session, an order-B control that starved itself on its own 42 GB commit charge (reported, not built on), the handoff's page-cache hypothesis refuted in 30 s, and a 2.4x-slow block that would not come back until the re-check found the ordering that brings it |
gate-two-drive-striping.md |
Two-drive striping of the disk-tier cache (R4): decoder layer i written to drive i mod 2 (C: on CPU lanes, D: behind the chipset), one read in flight per drive, read-ahead 3. Measured on the cold 70B-shaped NF4 step against the single-root cache, under a rule committed before the striped cache existed | SHIP: the cold 70B-shaped step at 9.971 s, 1.75x faster. Sequence 4, the first to complete (three rounds, six valid arms), measured STRIPED at 10.050, 9.971 and 9.929 s, under the 12.0 s line in every round. SINGLE read 17.552, 17.431 and 17.438 s in the same rounds, for speed-ups of 1.746-1.756x. STRIPED's implied rate, 7.00-7.09 GB/s, is above anything one drive reached. direct_io and pinned were true throughout, the two caches are byte-identical tensor by tensor (2,405 tensors, 36.388 GB), and the losses differ between arms no more than between runs of one arm. Three earlier sequences stopped with no verdict and are kept, not counted: two on losses of AC power (one confirmed as the laptop left unplugged), and one on the rule's validity row after an unexplained slowdown of both drives (SINGLE 22.107 s, with STRIPED at 18.052 s in the same stretch). Keeps two void arms, the driver's incomplete not_run list, and a first-step loss pattern the later arms broke |
probe-rtx5070-two-drive-read.md |
R1 of the disk-tier read-rate track: does a second NVMe add read bandwidth? The #974 unbuffered read primitive on C: (CPU lanes) and D: (behind the chipset), alone and both at once, under a rule committed before the run; plus three diagnostics written the same way (D1 the CPU package's idle state, D2 PCIe ASPM, R3' the reader's per-layer barrier) | No formal verdict. The rule's validity row fired in all three runs (drive A, C:, read alone below the band the rule requires of the instrument), and D1, D2 and R3' are ambiguous by their own rules. Measured without a verdict: the two drives read 7.65-9.15 GB/s together in nine of nine both-at-once arms, each at its alone rate or faster, which no shared bottleneck produces. Striping was built on that evidence by the owner's decision and gated on its own, on the training step (gate-two-drive-striping.md) |
probe-rtx5070-two-drive-sustained.md |
Does the two-drive gate's 1.75x hold under sustained reading? The same cold 70B-shaped step, each arm one process reading for about six minutes (SINGLE 21, STRIPED 36 timed steps), three interleaved rounds of about 40 minutes in all, judged on the late-window speed-up under a rule committed before the run | NO VERDICT, twice. Sequence 2 (2026-10-02): rounds 0 and 1 and round 2's SINGLE arm completed, but round 2's STRIPED arm was void twice: Windows Search's indexer read 1.27 GB, then 1.94 GB, over the 1 GB foreign-reader bar. Rounds 0 and 1 alone read u_late 1.279 and 1.412, which is not a verdict. The 2.5 GB/s per-drive plateau now held to the end of every STRIPED arm that reached it (round 1's from its third step), and C: alone dipped about once a minute in the later SINGLE arms. Attempt 1 (2026-10-01): the laptop lost AC and hibernated inside round 1, leaving only round 0 (u_late 2.065, not a verdict) |
probe-rtx5070-drive-throttle.md |
Does either drive slow down by itself under sustained reading? The sustained striping probe saw recurring stretches with both drives at about 2.5 GB/s instead of about 4, and cannot say which drive slowed. Six arms of 300 s, one process each (C: alone, D: alone, both at once, then the reverse order), reading the 70B caches' own files with the R1 unbuffered primitive and logging every span; judged per drive (THROTTLES, ONLY WHEN BOTH, STEADY, INCONSISTENT) under a rule committed before the run | NO VERDICT (2026-10-02). The first position, C: alone in round 0, was void twice: Windows Search's indexer (SearchIndexer.exe) read 2.37 GB and then 1.82 GB during the arm, over the 1 GB foreign-reader bar. The sequence stopped there, and no D: or BOTH arm ran. Descriptive only, from the void attempts: C: read from rest for 300 s had no dip. Read again straight after (about 7.5 minutes into continuous reading), it fell to about 2.6 GB/s for 11-12 s three times, about once a minute, shorter than the rule's 15-s episode |
gate-901-14b-on-8gb.md |
The #901 report — Qwen2.5-14B streamed as NF4 "OOMs inside SFTTrainer.__init__" on an 8 GB card with 32 GB of RAM (#901) — reproduced on a synthetic 14B shape, reduced to pure torch, and fixed |
Memory was never the problem. A failed page-lock of the 9.93 GB store leaves a stale CUDA error that the run's first kernel launch reports as CUDA error: out of memory with 7.3 GB free (cudaMalloc, synchronize and memcpy all pass; the launch check reads it), and the page-lock failed because torch's caching host allocator rounds every pinned request up to a power of two — the store cost 1.73x its bytes per tensor (a 100 000-byte request shows as 131 072 in host_memory_stats). After draining the error and pinning the store in power-of-two arenas the same config trains end to end here: measured peak 2.944 GB allocated / 3.244 GB reserved against the pre-flight's 3.39 GB, the store page-locked as exactly 10 GiB (1.08x) where it fell back to pageable before. Carries a memory hypothesis refuted by a 3 KB allocation failing, and no throughput or quality claim: the fixture is a shape with random weights |
probe-moe-expert-coverage.md |
MoE step 0 — is expert-granularity streaming worth building? Streaming one expert at a time instead of one decoder layer only saves READS to the extent that a step touches fewer experts than the layer holds, so this measures union coverage, routing skew and prefetch predictability per layer on two trained MoE models over three corpora. The decision rule was committed before any model was downloaded (4908d52b) and the numbers arrived afterwards | A training step touches essentially every expert: C = 0.967-1.000 over 18 corpus/shape combinations, worst single layer 0.909, against a chance baseline of 1.000 — so the read volume of an expert-streamed step equals that of a layer-streamed one and the feature is a capacity tier with no throughput gain, which is the branch the pre-committed rule assigns to C >= 0.85. Concentration is real but orthogonal and corpus-dependent (Gini 0.271 on prose to 0.558 on code; the busiest quartile takes 41-67% of traffic), and the rule's hybrid-tier threshold fires only on code at 64 experts. Carries an arithmetic correction to the rule itself: at coverage ~1 a resident hot set saves the RESIDENT FRACTION of bytes, not the traffic share those experts carry. NF4 against a bf16 CPU control moves nothing (coverage delta 0.0008, worst layer 0.0068), and the prefetch numbers are reported as uninterpretable at this coverage — which the record said they would be, before the run |
gpu-test-runs.md |
Not a gate — every run of the CUDA-dependent tests (pytest -m gpu) on a real card, since CI has none (#833) |
Two runs recorded, both before the marker existed, and both found defects CI could not see: 8x H100 (9 failed, 419 passed on the four streaming suites, three defects) and an RTX 5070 Laptop (sm_120, full suite 20709 passed, 2 failed, #776) |
gate-725-lorafa-operating-point.md |
PEFT LoRA-FA optimizer operating point (#725) | Measured 50.0% fewer trainable adapter parameters and 50.0% fewer AdamW state elements; analytic $r / d_{in}$ activation retention for adapted projections. Micro-benchmark only; explicitly not total LLM peak VRAM, throughput, or quality claim. |
gate-836-bench-train-contract.md |
What soup bench train reports on a real card (#836, shipped in #1146) — the published reference run that acceptance item 7 asked for, on an RTX 5070 Laptop on battery at a 50 W cap |
The contract holds; the number does not repeat. Over thirteen runs of a byte-identical config the config hash, token counts, max_memory_allocated, max_memory_reserved and all three checks are identical to the byte, while reported throughput spans 376.4–915.3 supervised tok/s, 2.43x (CV 35.3%). The noise is additive, not multiplicative: across four shapes whose fast-mode step spans 26x (0.197–5.26 s) the fastest and slowest repeat differ by a near-constant 0.17–0.28 s per step — 4.6% on a 5.3 s step, 1.9x on a 0.2 s one — and a longer window does not average it away (160 counted steps still spread 1.89x). Two explanations were tested and both refuted by their own covariate (peer processes; CPU frequency, where the highest-frequency run is in the slow half). Also: one shape completed while allocating 1.51x the card's VRAM (WDDM spill — memory is the allocator's truth, not a fit), provenance.sm_clock_mhz_after_run is an idle reading that cannot discriminate a 1.83x difference (since replaced by the busy-window sm_clock_mhz_busy in #1177; see the record's addendum), and config_hash changes with --steps because the tool's own save_steps override is inside it. Four refusals exercised verbatim; one contaminated run kept in |
gate-1227-ab-burn-in.md |
The burn-in of soup ab's two-sided mSPRT (#1227): how many rows per arm before any verdict, so that re-running the test after every new pair keeps its false-positive rate at --alpha |
30 rows at alpha >= 0.05, 40 below, calibrated over runs of up to 1000 rows per arm: worst Type-I 0.0512 at alpha 0.05 (bound 0.0521). At alpha 0.01 the sweep's 0.01088 is inside its bound (0.01094) only in this seed set: two of three other seed sets reached 0.0113 and 0.0116, and pooled over seeds the worst case is near 0.0108, about 8% over nominal, a floor set by the estimated variance that more rows do not lower (#1265). A 200-row horizon had picked 20 rows, which reach 0.0534 by 1000 rows. Not calibrated below alpha 0.01 (0.0058 at 0.005). With the true variance the rate stays below alpha everywhere |
gate-1265-ab-nig.md |
soup ab's variance-robust statistic (#1265): the normal-inverse-gamma Bayes factor that replaced the #1227 burn-in, with the first rows per arm held out to set the prior scale |
No burn-in; PRIOR_SCALE_ROWS = 5. Worst Type-I under a peek after every pair, runs to 1000 rows: 0.0328 at alpha 0.05, 0.0066 at alpha 0.01, 0.0034 at 0.005 (burn-in: 0.0511 / 0.0110 / 0.0058). Power >= 0.996 from effect_size / sigma 0.3 up (burn-in accept_h0 under H0 ( |
gate-1418-ab-cs-accept.md |
soup ab's accept rule (#1418): accept_h0 once the always-valid confidence sequence for the difference, the #1265 mixture inverted at level alpha, lies inside +-effect_size, instead of the Bayes factor falling to log(beta/(1-alpha)); --beta retired |
Accept 2.0-2.4x sooner with no difference at alpha 0.05: 307 pairs at effect_size / sigma 0.3 (was 621), 117 at 0.5 (was 281), runs to 1000 rows. reject_h0 rule unchanged; Type-I only falls (worst 0.0328 to 0.0289 at alpha 0.05). Wrong accept_h0 at a true difference of exactly effect_size at most 0.0083 at alpha 0.05 and 0.0019 at 0.01 (bound alpha; was 0.0042 / 0.0032), power there within 0.005 of before at alpha 0.05 and no lower from 0.01 down. Level beta measured too and not taken (power 0.965 at beta 0.20) |
gate-1702-arena-packing.md |
Largest-first packing of the pinned host-memory arenas on the layer-streaming RAM tier (#1702, shipped in #1713); Qwen3-8B NF4 on the RTX 5070 Laptop 8 GB | Memory saving confirmed, no speed claim. The RAM tier page-locks 6,442,450,944 bytes where the in-order plan page-locks 8,589,934,592, and three deterministic optimizer steps are byte-identical between the two plans (870 of 870 digests). The 40-run speed series measured +1.96% on the RAM tier, inside the +/-15.13% band its own control set, so by the rule written beforehand no speed-up and no slow-down is claimed; the band is wide and cannot exclude a smaller change |
gate-1892-nf4-scale-cache.md |
End-to-end A/B of the opt-in training.stream_nf4_scale_cache (#1892) on a streamed Mistral-7B NF4 run |
Confirmed on one laptop: median paired reduction 3.387 % [2.389, 6.936] over five AB/BA pairs against a rule of >= 2 % with lower bound > 0; +0.446 GB VRAM. The companion --train-threads 1 test showed no gain (median -0.555 %) and is dropped. Exactness is covered by the PR tests, not by this record. |
gate-stream-recompute.md |
training.stream_recompute: streamed layers that keep their activations instead of the checkpoint recompute, on a streamed Mistral-7B NF4 run (RTX 5070 Laptop 8 GB, a new baseline) |
Measured on one laptop: -22 % / -17 % / -8.5 % step time on the RAM tier at 512 / 1024 / 2048 tokens per step (32 / 24 / 12 of 32 layers fit), loss and 256 of 256 LoRA gradients bit-identical under deterministic algorithms; no gain on a read-bound disk tier. soup train with auto chose 32/32 layers at max_length 512 and 10/32 at 2048, train_loss equal to on. |
gate-fp8-shard.md |
Sharding a DeepSeek-V3 FP8 block-quantised checkpoint by dequantising it at shard time (CPU only): dequant exactness, streamed-from-FP8 vs resident parity on a tiny DeepseekV3ForCausalLM, and Moonlight-16B-A3B converted to the format and sharded |
G1, G2, G3 pass. Bit-exact against an independent per-block reference (and transformers' own dequant where its block rule applies); logits, loss and LoRA gradients bit-exact in both expert layouts, caches tensor-for-tensor equal incl. NF4; the 27-layer NF4 cache of the FP8 copy came within 0.0002% (0.000115%) of the pre-flight estimate, and every checked FP8-derived tensor equals the reference. No GPU run |
probe-rtx5070-moonlight-fp8-source.md |
The GPU loss check gate-fp8-shard.md leaves to the maintainer: does a soup train LoRA run on the FP8-block copy of Moonlight-16B-A3B train like the same run on its bf16 source? Two arms, the same code and config apart from base and output, fused NF4 caches; limits on the step-1, per-step, epoch-3 and evaluation loss differences, GPU peak and seconds per step, set from a matched-size reference pair measured before the run |
FAIL (T2_max_step, T6_step_time), 2026-10-20. Both arms complete on the 8 GB card at the same 3.1 GiB GPU peak. Step-1, final-evaluation and epoch-3-mean losses agree to 0.081, 0.072 and 0.101, within their limits, but the 5-token row diverges by 3.27 nats at step 7 (limit 1.6), and B's median step took 1.91x A's (limit 1.10). A diagnosis after the verdict finds the cache layouts identical and the reader on direct I/O in both arms, with B's files reading at A's rate or faster on a rested drive; the most likely cause of T6 is the D: drive's sustained-read plateau (section 6), with B's timed steps starting 17 s after its own shard I/O. The rule's cool-down covered the GPU, not the drive. Speed parity and per-step parity are not shown |
probe-rtx5070-giant-moe-step.md |
Does one LoRA training step of a synthetic Kimi-K2-shaped MoE (1,026,408,232,448 parameters, random weights, 533 GB NF4 per-expert store on the disk tier) complete on the RTX 5070 Laptop 8 GB? Measurement harness benchmarks/harness/moe_stream_step.py, not soup train; seq 512, batch 1, two steps |
COMPLETES. With routing forced uniform (every expert read), step 0 took 867.95 s (0.590 tok/s) and step 1 1,108.77 s (0.462 tok/s), reading 69,120 experts and 1,570.34 GB per step at a 4,342 / 4,723 MiB peak allocation; loss and all 610 gradients finite. With the model's own routing only about 9.7 of 384 experts per layer were read, so that run is not a K2 step time. Shape only: no quality, convergence or real checkpoint |
probe-rtx5070-giant-moe-seq-scaling.md |
Does a longer sequence raise the tokens/s of that synthetic 1T step, since the per-step expert reads do not grow with the tokens? ABBA series at seq 512 and 2048 (loss over the last 512 positions), one step per process, under a rule committed before the run; random weights, shape only, measurement harness, not soup train |
NO VERDICT (2026-10-23). Gate V5 (at most 2 GiB read from C: inside the step) failed in three of four runs, so the rule's claim is not made. Measured without standing: 1,036.4 s at seq 512 (0.494 tok/s) against about 1,119 s at seq 2048 (1.83 tok/s), ratios 3.70-3.71 against a 3.0 SHIP floor; the step is fetch-bound, not disk-bound (1.42-1.53 GB/s from D:) |
Harnesses
harness/ holds the measurement scripts that can be run against a
released Soup, so a claim in a record above can be re-measured rather than taken
on trust. It starts small on purpose — most of this session's ~20 harnesses live
only in a scratchpad on a machine that is gone, which is
#379.
| Script | STEP / record | Question or claim | Cost / requirements |
|---|---|---|---|
quest_mixed_route.py |
#674 / gate-674-quest-mixed-route.md / gate-674-quest-w4a4-sft.md |
Does the experimental integration reconstruct exactly 168 W4 weights / 161 A4 + 7 block-23 A16 activations, and what dense fake-quant overhead does one A4/A16 layer add on this host? The fixture is synthetic and the JSON explicitly refuses quality, packed-INT4, throughput and memory-saving claims | CPU or CUDA; no downloads; seconds at the default width 128 |
quest_parity_evidence.py |
#674 / quest-674-postfit-development-results.md |
Recomputes the retained Q/strongest-FP development ratio, paired gap interval, and failed zero-margin gate from the published evidence; optional local artifact checks need the preserved checkpoints | CPU for published evidence; local checkpoint files for --artifact-root; no downloads |
issue331_qlora_scope.py |
#331 | Does the wrong-gradient defect reach ordinary QLoRA? Three arms in one process, with a positive control. Answer: no — 0.0 against a control that diverges by 3.77e-01 | ~15 s, 4 GB card, no downloads |
bnb_repro.py |
Upstream bitsandbytes #2034 / NF4 defect | Minimal standalone reproducer for the NF4 stale-gradient mechanism: recycled buffers expose MatMul4Bit's ctx lifetime issue; private buffers are the reference and bf16 is the control |
CUDA GPU required; no downloads; historical environment: torch 2.13.0+cu130, bitsandbytes 0.50.0; ~1 min |
mechanism_cost.py |
STEP 9 / #331 | Measure the recorded NF4 de-aliasing cost on a local Qwen2.5-32B-Instruct checkpoint: control, clone_fwd_quant, and clone, with correctness asserted in-process before timing; --self-test exercises the negative empty-intersection acceptance case |
CUDA GPU + local checkpoint required; about 20 GiB GPU memory for the clone arms; checkpoint must contain real .safetensors files (symlinked HF-cache snapshots are refused); no downloads; historical protocol: seq 256, batch 1, 5 repeats, 4 warm-up steps, 15 timed steps, 2 stream buffers |
mechanism.py |
STEP 9 / #331 | Is the NF4 streamed-gradient defect caused by pooled-buffer aliasing? Five arms: control, sync, clone, clone_quantstate, and clone_packed; every arm is compared with an independent resident NF4 reference while the single-sided clones test which captured state matters |
CUDA GPU required; native runs are GPU-memory intensive; no downloads; historical environment: torch 2.13.0+cu130, bitsandbytes 0.50.0, H100 sm90 |
fixture_window_cpu.py |
STEP 13 / historical fixture_window_cpu.py and cpu_mode_probe.py questions in gate-h100-validation.md |
Per-layer reconstruction, not a reproduction of the recorded full-model logits. Replays the 64x64 / 256x64 CPU M=8..256 window and packed-inference mechanism. Every row publishes an emulated-bf16 reference; the JSON reports the worst relative bf16 error and how many times the 2% packed-effect envelope exceeds it. The training control must be exact on every row; default exit codes: 0 reproduced, 2 capability absent, 3 control failed, 4 packed effect absent, 5 packed effect out of range. --survey waives only exit 2. |
pip install -e ".[train]"; CPU + bitsandbytes. AVX512-BF16 is required to demonstrate packing. The JSON records whether CPU gemv_4bit is registered and a best-effort native/kernels-community attribution; importing bitsandbytes may fetch optional kernels-community code. |
bitexact.py |
v0.72.0 / STEP 1; gate-h100-validation.md Reproducing | Shard -> stream -> compare logits, LoRA gradients, and loss curve against a resident reference using matching numerics; NF4 uses a resident NF4 reference | CUDA GPU required; model/checkpoint weights required; historical run: Qwen2.5-0.5B-Instruct NF4, seq 64; CUDA validation: torch 2.5.1+cu121, transformers 4.57.6, trl 0.19.1, peft 0.18.1 |
variant2_gate.py |
v0.72.5 / STEP 14 | Re-run the streamed-NF4 repair gate with the defect-present control and repaired arm in one process against one resident NF4 reference. Canonical parameter intersection and shared LoRA gradients must both be non-empty; the control must break and the repair must be exact | CUDA GPU with room for one streamed model plus its resident reference. Historical 32B/72B gates: H100 80 GB, torch 2.13.0+cu130, bitsandbytes 0.50.0, transformers 4.57.6, trl 0.26.2, peft 0.20.0; batch 1, seq 128, 2 pinned buffers, 5 repetitions. The maintainer's small-model smoke on an RTX 5070 Laptop (sm_120, 8 GB; torch 2.14.0+cu130, bitsandbytes 0.50.2, transformers 5.17.0, peft 0.20.0) failed closed, as expected: below roughly 164–172 MiB per NF4 layer the control cannot break, and sm_120 can distinguish fused bitsandbytes from explicit dequantisation. Such RESULT: failed outcomes are not evidence that the repair regressed. The new harness still has not been validated on an H100-class card. |
pincost.py |
STEP 8 / gate-h100-validation.md §8 | Does turning host-memory pinning off impose the historical throughput cost? Uses Qwen2.5-32B-Instruct NF4, seq 256, batch 1, 5 warm-up steps + 20 timed forward/backward steps, n=3 per arm, with CUDA synchronization around each timed step. Both arms require canonical overlap, finite gradients and bit-exact finite losses; pageable gradients must be exact on every one of three backwards. The pinned arm may be timed while WRONG, with per-repetition counts and timing status reported. Historical record: 425.07 tok/s pinned vs 64.79 tok/s pageable, 6.56x slower without pinning; the pinned arm was WRONG 8/256, and absolute tok/s is not a standalone claim. The historical control is installed by default; pinning cost scales with bytes per layer | CUDA GPU with sufficient memory for Qwen2.5-32B-Instruct NF4; 26,713 MiB recorded requirement with both models resident; local checkpoint required and resolve_model_weights may download a non-directory; historical stack: H100 80 GB, torch 2.13.0+cu130, bitsandbytes 0.50.0 |
graddiff.py |
STEP 2b / gate-h100-validation.md | Historical reconstruction of the GRADDIFF investigation: compare all LoRA gradients after one backward, then run each model's own 5-step loss curve twice. Historical Qwen2.5-32B-Instruct NF4 on H100 had 256/256 gradients exact on the first backward; the record also includes a 20/256 run, so this harness reports the measured count and applies its conservative exactness gate without claiming first-pass stability. Measurement 1 recorded non-identical streamed and resident self-curves (C resident self-identical: False); this harness deliberately restores the adapter and creates a fresh AdamW optimizer for each curve, requiring resident self-identity as a control. The record does not explain its C=False result. The historical control is installed by default; the old repaired-path #968 loss mismatch is not the default controlled measurement. The harness uses canonical parameter overlap and a negative empty-intersection self-test; it reports that the historical pattern was not reproduced instead of treating that as a pass |
VRAM floor: about 26,713 MiB recorded with resident NF4 32B + streamed model live; local Qwen2.5-32B-Instruct bf16 safetensors checkpoint as real files (the harness quantizes it to NF4; symlinked HF snapshots are refused); resident NF4 reference required; no downloads; historical validation stack: H100 80GB, torch 2.13.0+cu130, bitsandbytes 0.50.0; default seq 128, batch 1, two stream buffers, 5 curve steps |
fsdp_sharding_probe.py |
STEP 20 / #373 | Is the base actually sharded, or was the flag merely accepted? Per-rank local == total / world_size asserted, with the single-GPU control expecting the full count so a probe that always divides fails instead of passing everything. Repairs the STEP 20 probe, which raised TypeError('FullyShardedDataParallel does not support len()') on all eight ranks at 70B — a torch.compile wrapper, not FSDP |
Accounting is pure-Python and CPU-tested; a real reading needs the multi-GPU box under test |
mlx_sft_smoke.py |
#23 / run-m1-8gb-mlx-sft.md |
Does MLXSFTTrainerWrapper.train() actually run against a real MLX runtime, and what does it peak at? Asserts MLX dispatch before measuring, so a transformers-path number cannot be published as an MLX one (#363). Generates its own fixture; prints host memory and swap either side |
Apple Silicon + pip install -e ".[mlx]"; downloads the chosen model; 20 s at 0.5B, ~6 min at 8B including the Hub fetch |
issue395_second_stack_vram.py |
#395 / gate-395-second-stack-vram.md |
Does the long-sequence under-prediction reproduce on a second stack, and is the gap the logits multiplier? Answer: no and no — the loss term measures 12.000000 (spread 0), and 14 x E alone exceeds the whole real peak on every row, so the +2 retained copy is absent. Three modes: sweep, --measure, --sdpa |
24 GB card (largest arm peaks 11.85 GB); downloads SmolLM2-135M and Qwen2.5-0.5B; needs jinja2>=3.1.0; a few minutes |
stream_loss_chunk_vram.py |
#1693 | Peak VRAM of one streamed sft step at 512 / 1024 / 2048 tokens with and without training.stream_loss_chunk, beside the pre-flight's own prediction for the same run and the free VRAM it compared against; a row that ran out of memory is a row, and one whose prediction is below its real peak (or that ran out of memory inside the predicted fit) under-predicts and exits 1. --self-test runs the same protocol on a tiny local Llama without reading a peak. Not yet run on a GPU: there is no record for it |
CUDA GPU required; downloads the model given as --base; --self-test needs no GPU and no download, a few seconds |
issue361_nf4_throughput.py |
#361 / gate-v0.72.2-nf4.md |
Re-run the v0.72.2 8B NF4 protocol on repaired code (batch 1, S=512, 50 steps after 10 warm-up, PagedAdamW8bit, same-session GEMM ceiling)? Answer: recorded as the post-#331 row in gate-v0.72.2-nf4.md: 208.6 tok/s at 1935 MHz, 59% of the same-session ceiling against 68% pre-repair; read it with its clock. |
RTX 3050 Laptop 4 GB; Llama-3.1-8B weights; CPU-tested protocol guards only |
stream_probe.py |
probe-rtx5070-what-bounds-streaming.md, gate-971-async-nvme-source.md |
What bounds the streamed step on THIS card? Builds the model through the shipped path, replicates the two shipped copy paths with CUDA events added (a source guard refuses to run if the shipped body drifts), and runs the ceiling / step / sweep / interleaved-ablation modes of the v0.73.0 probe, writing JSON after every point. --tier disk measures the ASYNC source (#971) at the depth --read-ahead N sets, and --control-sync-source swaps in the shipped synchronous DiskSource for a same-session control; every run prints and records the source class it actually built and refuses when that disagrees with the flag, so a control cannot be published as a measurement. --no-pin selects pageable host memory on EITHER tier. --lazy-shard-handles was the sharding workaround for the Windows mapping limit and is no longer needed: #934 (merged 2026-09-14) bounds the shipped sharder's live mappings, and its gate re-sharded the 138 GB 70B-shaped fixture with no workaround at all — 80/80 layers in 221.4 s against 216.9 s with the flag, and 83 output files / 36.39 GB byte-identical to the workaround's cache, index.json included. The flag is kept so the 2026-09-12 record can be re-run as it was measured. Since #974 every JSON's meta also records soup_cli_file, the source tree the run actually imported — an arm that claims to be "the old code" is only evidence if the JSON says where it came from — and --skip-events runs only the uninstrumented step_plain point (halves a cold 70B block; not used by the #974 record, whose blocks run both points as gate-971 did). Since probe-rtx5070-batch-control.md every timed step records its wall-clock start, an out-of-memory error inside a block is written as an oom point and the run exits 3 instead of losing the arm, and every step point carries a SHA-256 per row of its token ids and the allocator's counters |
CUDA GPU; model weights; a 7B NF4 run is ~10 min all modes; the 70B-shaped cold-disk run is 10-60 min |
batch_control_probe.py |
probe-rtx5070-batch-control.md §1-§4 |
The batch control's driver: one stream_probe.py process per arm (SINGLE and STRIPED at batch 1, 2 and 3, two rounds, the second reversed, after a build arm and a 300 s settle), importing the two-drive gate's stamps, pre-arm checks, sampler and void rule unchanged. Runs the read-free floor (--ablate --arms B_nocopy) after every SINGLE arm's step block, requires an empty card before each arm, voids an arm another process read 1 GB during, and treats an out-of-memory error, an arm past a 600 s cap or a spill as a result that skips the larger batches. --summarize applies the rule from the committed files. No torch import |
Windows only; a CUDA box holding both caches; about 25 min, 29 with a re-shard |
batch_control_rule.py |
probe-rtx5070-batch-control.md §2 |
The batch control's rule from the committed files alone: per-arm facts (the step, peak VRAM, the out-of-memory point, the read-free floor, nvidia-smi inside the timed steps), the six rows, the model's predicted class per batch and the crossover estimates |
No GPU; seconds |
batch_control_sampler.py |
probe-rtx5070-batch-control.md §4 |
The batch control's GPU samplers, on top of the gate's: nvidia-smi every 2 s (memory in use, SM clock, temperature, power, utilisation, throttle reasons) and the Windows GPU memory counters every 1 s (dedicated and shared usage, per adapter and for the arm's own process, the query reopened until that process appears). Windows spills an over-allocation into shared host memory without an error, and these are where it shows |
Windows only; no torch |
layer0_wait.py |
gate-971-async-nvme-source.md §8 |
Which layer pays for the vocabulary-sized read planned at the head of every step? stream_probe.py brackets every load with CUDA events but SUMS them per step; this drives the probe's own instruments from outside and labels each bracket with its layer. Answer: layer 0's forward load is the fastest of the step (16-17 ms against a 213-252 ms mean), because the reader stages it during the build and embedding phase — so the behaviour is not material and needs no fix. The largest single compute-stream stall is the vocabulary fetch itself, 32.6-32.9 ms in every step against layer 0's 0.17/14.99/14.35 ms and a 0.002 ms mean over the other 79 decoder layers: the whole step-head stall is 33/48/48 ms, 0.07-0.12% of the step, so it does not change the answer either. Instruments from step 0 with no warm-up, so its absolute step times are deliberately NOT throughput evidence |
CUDA GPU; model weights; minutes |
synth_checkpoint.py |
probe-rtx5070-what-bounds-streaming.md §3; gate-901-14b-on-8gb.md §1 |
Writes a Llama- or Qwen2-SHAPED synthetic checkpoint (random N(0, 0.02) weights, real shapes) so a store larger than RAM can be built without a Hub token or a 140 GB download, and so an out-of-memory report can be reproduced from shapes alone (--shape qwen2.5-14b --arch qwen2 is #901's). Timing and memory only — never a correctness or quality claim |
~155 s and 138 GB for the 70B shape, 29.5 GB for the 14B one; needs a GPU for the random draw to be fast |
issue901_stream_step_probe.py |
gate-901-14b-on-8gb.md §2, §4 |
Does this shape fit, and where does the memory go? Builds the streamed model through the shipped path (stream_probe.build), runs ONE forward+backward at the configured shape, and marks memory_allocated / peak at the embedding, at chosen decoder layers (forward and the start of their backward), at lm_head and at the loss, with the allocator history recorded so an OOM is attributed to live blocks rather than guessed. Answer for #901: the first CUDA op after the build failed at 3 KB with 5.4 GB free — not memory — and after the fix the 14B NF4 step peaks at 2.944 GB |
CUDA GPU; model weights (the synthetic 14B shape serves); ~1 min on a cached shard set, +1 min to shard |
mmap_limit_repro.py |
#926 / probe-rtx5070-what-bounds-streaming.md §13–§14 |
Does the sharder's all-handles-open pattern die on its own, without Soup, CUDA or bitsandbytes? Two modes over one directory of *.safetensors: all holds every handle open, perfile opens, reads and closes each in turn. On the Windows box all dies once ~96–106 GB of mappings are live, at 14 handles or at 62; perfile passes. all was the shipped pass-2 pattern when this was written; since #934 (merged 2026-09-14, closing #926) the sharder keeps at most two source handles alive and every read owns its memory, so all now reproduces the historical defect rather than the current code |
Any box; a directory of large safetensors totalling more than ~100 GB to see the crash (the synthetic 70B shape serves); seconds — it maps, it does not read unless --touch |
moe_expert_coverage.py |
probe-moe-expert-coverage.md |
Does one training step touch every expert of a MoE layer? Expert-granularity streaming only saves READS to the extent that it does not. Hooks every top-k router and reports, per layer: union coverage, routing skew (Gini plus the share taken by the busiest 10/25/50%), hot-set stability, and the share of the next layer's traffic that prefetching either the previous layer's used set or a corpus-wide heat quartile would have caught. Every coverage figure is printed beside the uniform-random baseline 1 - (1 - k/E)^T for the same shape, because coverage alone is not interpretable: at 64 experts / top-8 over 512 tokens chance alone predicts 1 - 10^-29.7, so a high coverage is the default rather than a finding. Router discovery is by SHAPE, not an architecture table — granitemoe returns the indices FIRST where olmoe, qwen3_moe and mixtral return them last, and both are validated end to end. Tokens are packed, never padded, since a padded batch routes its PAD positions too. No soup_cli import |
Any box for the validation (it builds its own 6.6M-param models); a real reading needs the model weights and, for a 7B-class MoE, either a CUDA card with --load-4bit or ~14 GB of host RAM |
issue974_warm_stages.py |
gate-974-disk-tier-direct-io.md §1, §2, §6 |
Where does a WARM disk-tier step spend its time, stage by stage? stream_probe.build plus host wall-clock timers around each stage: consumer_wait (the compute thread blocked in AsyncDiskSource.get), reader_read (one bracket per layer read — _read_layer — so the K parallel ranges are one number, not K overlapping ones; since R4 _read_layer only dispatches the ranges, so the bracket waits on them, and the harness refuses a striped source), reader_drain (the pool event the reader waits on before refilling a slot), consumer_read and consumer_load_async for the synchronous control. One arm per process, so run order is explicit. Answer: position-matched, the async source was 0.91-0.97x the control before #974 — the 1.03-1.20x was run order — and the drain is 1-8 ms/step |
CUDA GPU; the Mistral-7B NF4 shard cache; ~40 s per arm |
issue974_cache_populate.py |
gate-974-disk-tier-direct-io.md §2 |
Does a buffered readinto repopulate the Windows page cache, or only mmap? Runs the two primitives over the same shards in a chosen order with an eviction where the sequence says, and reports GB/s per pass. Answer: every readinto flavour (plain / O_RANDOM / O_SEQUENTIAL / buffering=0) warms the cache on its second pass (2.60 -> 3.96 GB/s); the handoff's hypothesis was wrong |
CUDA GPU (pinned staging); a shard cache; ~1 min |
issue974_evict.py |
gate-974-disk-tier-direct-io.md §2, §4 |
Evict a store from the page cache without admin rights: touch ~17 GB of pageable RAM (available physical falls to ~2.5 GB), then read 8 GB of OTHER files through buffered readinto to displace the survivors — pressure alone leaves the most recently used ~2.5 GB of the standby list resident, i.e. exactly the store under test. Prints the host baseline around both stages | Any Windows box; --displace-dir must name large files unrelated to the store; ~15 s |
issue974_cold_ranges.py |
gate-974-disk-tier-direct-io.md §4 |
How fast can the disk tier read a layer that is NOT in the page cache, through BUFFERED reads? Every (mode, threads) configuration reads its own fresh layers of a store larger than RAM, in two configuration orders. Answer: 1.25-2.4 GB/s whatever the shape — per-tensor readinto, one readinto per range, 1-8 threads — the cache manager is the ceiling | CUDA GPU (pinned staging); the 70B-shaped NF4 cache (34 GB > RAM); run issue974_evict.py first; ~1 min |
issue974_unbuffered.py |
gate-974-disk-tier-direct-io.md §4 |
Does UNBUFFERED I/O (FILE_FLAG_NO_BUFFERING, one handle per thread, into a 4 KiB-aligned pinned arena over the aligned superset of the data section) reach the drive? Same fresh-layers-per-configuration policy, buffered arm as the control, byte check on the last layer. Answer: 3.5-5.65 GB/s cold, best at 2-4 ranges, against 1.2-2.9 buffered in the same positions — the number the #974 reader is built on. Carries the alignment rules the reader inherits (an unaligned LENGTH is EINVAL; the last request of every layer runs past EOF and returns short) |
Windows (ctypes CreateFileW); CUDA GPU; the 70B-shaped NF4 cache; run issue974_evict.py first; ~1 min |
issue974_read_paths.py |
gate-974-disk-tier-direct-io.md §3 |
The WARM read-primitive micro-benchmark that motivated the first fix plan: per-tensor readinto vs mmap+copy vs one readinto per layer of the whole data section into an arena, split across K threads, store in the page cache. Answer: 3.9 -> 9.3 GB/s at 8 threads for the arena read — a page-cache -> pinned memcpy number, which §4 shows says nothing about cold. Kept because a record that changed its plan should show the number it changed it from | CUDA GPU; the Mistral-7B NF4 cache; ~30 s |
two_drive_read.py |
probe-rtx5070-two-drive-read.md §2, §5 (R1) |
Do two NVMe drives read more together than apart? Each drive reads fixed-size, sector-aligned spans (one ~70B NF4 layer each) as K parallel ranges through the #974 unbuffered primitive (imported from issue974_unbuffered.py, not re-typed): A alone, B alone, both at once, in a rotating order. Reports r_sum and r_alt over the window in which both drives were reading. Every read is cold: unbuffered reads never touch the page cache |
Windows only; CUDA (pinned arena); two drives with room for the spans; minutes |
two_drive_host_probe.py |
probe-rtx5070-two-drive-read.md §5 (D1) |
Is one drive's read held back by the CPU package going idle? Reads one drive alone with and without busy-loop spinner processes, same primitive (imported from two_drive_read.py). Answer: ambiguous by its rule; the spinners moved the median by +3% and reversed in one round |
Windows only; CUDA; minutes |
two_drive_aspm_probe.py |
probe-rtx5070-two-drive-read.md §5 (D2, R1 run 3) |
Does PCIe link power management slow one drive? Reads drive A alone at the power scheme's AC ASPM index and at Off, interleaved, then optionally R1 again with ASPM Off. Changes a system power setting for its duration (restored in a finally and read back; exits non-zero if the read-back disagrees), so it runs only with the owner's consent. Answer: D2 ambiguous by its rule; R1 run 3 no verdict |
Windows only; CUDA; refuses on battery; minutes |
two_drive_depth_probe.py |
probe-rtx5070-two-drive-read.md §5 (R3') |
Does the reader's per-layer barrier cost one drive throughput? The same K workers and ranges read one drive at depth 1 (the reader before R4: dispatch a span, wait for all of it) and depth 2, interleaved, through one function. Answer: ambiguous by its rule | Windows only; CUDA; refuses on battery; minutes |
two_drive_stripe_gate.py |
gate-two-drive-striping.md §1-§5 |
The striping gate's driver: one stream_probe.py process per arm, in the rule's order (SINGLE and STRIPED interleaved over three rounds), a box stamp around every arm, AC/commit sampled every 2 s and per-volume counters every 1 s during it. An arm on battery, one the box was suspended in, or one with no step_plain point is void and re-run once in place. Applies only the rule rows a single arm can settle; decides nothing about shipping. No torch import |
Windows only; a CUDA box holding both caches; ~2.5 min per arm |
compare_shard_caches.py |
gate-two-drive-striping.md §5.5 |
Do the striped and the single-root caches hold the same bytes? Pairs every safetensors file through the striped index's layer_roots and compares every tensor by name, dtype, shape and a SHA-256 of its data. Parses the headers itself and reads in 64 MiB chunks: no torch, no mapped file. Answer: identical, tensor by tensor (2,405 tensors) |
No GPU; reads both caches in full, so it runs after the timed arms, never between them |
two_drive_sustained.py |
probe-rtx5070-two-drive-sustained.md §1-§4 |
The sustained probe's driver: one long stream_probe.py process per arm in the gate's round order, importing the gate driver's stamps, pre-arm checks, sampler and void rule unchanged. Adds a 10-s GPU sampler, a foreign-reader void (another process reading 1 GB or more during an arm), the per-arm early/late medians and per-drive window rates the rule reads, and --summarize, which recomputes the rule from the committed files. No torch import |
Windows only; a CUDA box holding both caches; about 6.5 min per arm, about 51 min for the whole probe |
two_drive_throttle.py |
probe-rtx5070-drive-throttle.md §1-§4 |
The throttle diagnostic's driver: one reader process per arm in the plan's order (C: alone, D: alone, both, then the reverse), importing the gate's stamps, pre-arm checks, sampler and void rule and the sustained driver's GPU sampler and foreign-reader check unchanged. Adds a rest between arms, a metadata-only listing of the two cache folders (--check-files), --summarize, which recomputes the verdict from the committed files, and --selftest. No torch import |
Windows only; a CUDA box holding both caches; about 37 min for the whole probe |
two_drive_throttle_read.py |
probe-rtx5070-drive-throttle.md §1 |
One arm of the throttle diagnostic: the R1 unbuffered read primitive (imported from two_drive_read.py) turned into a fixed-time cyclic read of C:, D: or both, K = 4 ranges per 400 MiB span, one span in flight per drive, every span's start and duration logged. In a both-drives arm the drives share only a start barrier, so a slow drive slows only itself |
Windows only; CUDA (pinned arenas, no compute); 300 s per arm |
two_drive_throttle_rule.py |
probe-rtx5070-drive-throttle.md §2 |
The throttle diagnostic's decision rule as pure code: 5-s bins from the reader's spans and the 1-s PDH samples, dip bins, episodes, the validity checks and the per-drive rows. --selftest runs it on synthetic arms (mutation-checked); --calibrate runs the detector on the sustained probe's committed PDH samples |
Any OS; no torch, no disk beyond the result files |
| issue725_lorafa_operating_point.py | #725 / gate-725-lorafa-operating-point.md | Measures trainable parameter counts, AdamW state tensor allocations, step latency, and multi-step loss finiteness between LoRA and LoRA-FA | ~20 s, CPU or GPU, no downloads |
ab_burn_in_sweep.py |
#1227 / gate-1227-ab-burn-in.md |
What false-positive rate does soup ab's burn-in leave when the test is re-run after every new pair? A vectorised H0 simulation of the shipped statistic, checked against _verdict_from_summary first, by effect_size / sigma, burn-in, alpha and horizon; picks BURN_IN_ROWS_BY_ALPHA |
CPU + numpy; ~55 s per ratio at 100,000 runs over 1000 rows; run writes JSON parts, report merges them |
head_prefetch_ab.py |
#1174 / #1258 | Interleaved A/B, in one process on one model, of where the output head's load into the shared large-layer slot is issued: StreamPrefetcher.head_prefetch_layer at None (the last decoder layer, the original timing) against a chosen layer (0, right after the embedding lookup). Judges each arm by the step's TOTAL wait from the committed stream_probe.Instruments brackets, plus per-group medians (embedding, head, layer 0, other decoder layers), since a decoder entry behind the head on the copy stream can get worse while the head's own entry improves. Covers both tiers, --read-ahead, and a --step-shape preference (a no-grad forward first) for the disk-tier case where the reader's queue gets replanned. Refuses to run on a tree without head_prefetch_layer instead of comparing an arm with itself; the JSON records soup_cli_file and git_sha, the tree it measured |
CUDA GPU; model weights; a few minutes per arm pair |
determinism.py |
STEP 2b / Measurements 1-2 | Historical reconstruction that measures forward, backward repetitions 1 and 3, and trained loss-curve reproducibility in-process against a resident NF4 reference; optional JSON replay remains separate from measurement | CUDA measurement requires the recorded H100-class stack and checkpoint; CPU tests cover the real measurement and invalid-measurement guards; empty overlap, non-finite values, missing measurements, or absent asymmetry fail |
repeat_backward.py |
STEP 2b / Measurement 3 | In-process repeated streamed backwards against one persistent resident NF4 reference; reports the first-pass count and requires later corruption while every loss remains finite and equal to the resident loss | CUDA GPU + real checkpoint files required for live NF4 measurement; seq 128, batch 1, two buffers, five repeats; historical stack: H100 80 GB, torch 2.13.0+cu130, bitsandbytes 0.50.0; CPU tests cover healthy, later-corruption, changed-loss, and invalid-measurement controls |
depth_vs_bytes.py |
STEP 6 / NF4 bytes-per-layer bracket | Historical reconstruction of the fixed 48-layer synthetic-Llama sweep: all three NF4 reads at or below 163.8 MiB/layer must be exact, later reads at or above 171.5 MiB/layer must be wrong, and the bf16 control above the bracket must remain exact | --measure creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index |
layercount.py |
STEP 6 / rejected layer-count hypothesis | Historical reconstruction: small NF4 and bf16 layers remain exact across the 48-to-64 depth boundary, while a separate 48-layer NF4 control above 171.5 MiB/layer must turn wrong; each row carries three backwards and exactly four LoRA gradients per layer | --measure creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index |
ab_nig_sweep.py |
#1265 / gate-1265-ab-nig.md |
Type-I and power of soup ab's NIG mixture under a peek after every pair, side by side with the #1227 burn-in design, by effect_size / sigma, held-out rows, alpha and horizon; checked against _nig_log_bayes_factor and msprt_step first; picks PRIOR_SCALE_ROWS |
CPU + numpy; ~350 s per ratio at 100,000 H0 runs over 1000 rows; run --mode h0/h1 writes JSON parts, report merges them |
ab_cs_accept_sweep.py |
#1418 / gate-1418-ab-cs-accept.md |
Time to accept_h0 under H0 and wrong accept_h0 under H1 for soup ab's confidence-sequence accept rule against #1265's, at level beta or alpha and with or without the accept hold; replays #1265's seeded runs and checks its counts against that record; checked against _nig_confidence_half_width and msprt_step first |
CPU + numpy; ~430-600 s per ratio at 100,000 H0 runs over 1000 rows; run --mode h0/h1 writes JSON parts, report merges them, check re-runs only the self-check |
fp8_block_quantize.py |
gate-fp8-shard.md G1-G3 |
Converts a bf16 DeepSeek-V3-family checkpoint to the FP8 block format it is published in (float8_e4m3fn + weight_scale_inv per 128 x 128 block, ragged edges as DeepSeek has them), one tensor at a time, and reports each tensor's dequant error against its bf16 original. Also the quantiser the FP8 tests use |
CPU; Moonlight-16B-A3B: 76 s, 2.5 GB peak RSS, 16.6 GB written |
fp8_shard_gate.py |
gate-fp8-shard.md G3 |
Shards an FP8 block copy with Soup's sharder into a new cache and holds it against estimate_shard_cache_bytes (payload exact per layer) and against an independent per-block dequant of the source (bf16 tensors equal, NF4 sidecars equal bitsandbytes' CPU quantisation); a RAM watchdog stops it below a floor |
CPU; Moonlight 27 layers NF4: 93 s, 4.6 GB peak RSS, 30.6 GB written |
fp8_loss_check_run.ps1 |
probe-rtx5070-moonlight-fp8-source.md |
One arm of the FP8-source loss check: pre-checks (src at the commit, the import path, RAM, AC, an idle GPU, the cache state the rule expects, B's cool-down and disk room), opens the RAM watchdog, runs soup train with its console copied to train.log and the tracker in the arm's folder, then re-checks the cache |
Windows, CUDA, PowerShell 7; A ~7.5 min, B ~9 min |
fp8_loss_check_watchdog.py |
probe-rtx5070-moonlight-fp8-source.md |
Follows a training process found by its command line: every 5 s, available RAM, the tree's RSS, peak working set, nvidia-smi, AC and disk free to JSONL; stops the tree below a RAM floor (exit 3); records sleep gaps, off-AC samples and foreign GPU processes. No torch import |
psutil, nvidia-smi |
fp8_loss_check_cache.py |
probe-rtx5070-moonlight-fp8-source.md §3.2 |
Will soup train reuse a layer cache or re-shard? Resolves the root and slug as soup train does (refusing when the variable would silently fall back to ~/.soup), asks inspect_shard_cache with its arguments, and sizes a re-shard with estimate_shard_cache_bytes. Read-only |
CPU; seconds |
fp8_loss_check_weights.py |
probe-rtx5070-moonlight-fp8-source.md §3.3 |
How far apart are the weights two caches stream? Dequantises NF4 on the CPU and reports per-class relative distances to the bf16 source and between caches, byte-equality of everything not derived from FP8, and a per-token squared-distance budget for two cache pairs. Read-only | CPU; seconds for 3 layers |
fp8_loss_check_compare.py |
probe-rtx5070-moonlight-fp8-source.md §2 |
The verdict from the two arm folders: COMPLETES per arm, T1-T6 with the committed limits, the INCONCLUSIVE conditions, per-step and per-epoch tables | CPU; seconds |
fp8_loss_check_layout.py |
probe-rtx5070-moonlight-fp8-source.md §6 |
Do two shard caches differ in anything a disk reader sees? Per safetensors file: header length, data start, size, metadata, tensor order, every tensor's dtype, shape, length and offset (mod 4096 and 512 for the large ones), then index.json field by field. Headers only, read-only |
CPU; seconds |
fp8_loss_check_readbench.py |
probe-rtx5070-moonlight-fp8-source.md §6 |
GB/s of one layer file read the way the disk tier reads it (open_direct, 4 aligned ranges, 4 threads), alternating two caches; refuses below 10 GB free. No torch import |
Windows NVMe; ~1 GB per read, 9 GB as committed |
stream_recompute_ab.py |
gate-stream-recompute.md section 4 |
Interleaved ON vs OFF-for-the-last-K blocks over several batch x seq shapes in one process, K from one layer's retained bytes after the forward under an allocated-peak cap | CUDA GPU required; Mistral-7B NF4: minutes per shape |
stream_recompute_exactness.py |
gate-stream-recompute.md section 3 |
Loss and every LoRA gradient, recompute ON vs OFF, bit for bit under deterministic algorithms and MATH sdpa, with an ON-vs-ON control and a one-token negative control; exits 1 on a failed gate | CUDA GPU required; Mistral-7B NF4 at 1 x 512: about a minute after sharding |
Hardware
Every number in the four gate-v0.72.* records was measured on one machine:
- GPU — RTX 3050 Laptop, 4 GB (4.29 GB usable)
- Host — 16.9 GB RAM, NVMe
- OS — Windows 11
gate-h100-validation.md is the exception and the reason it exists: 8x H100
80 GB, 503 GB RAM, Ubuntu 24.04, on a much newer torch/bitsandbytes/trl/peft
stack. It is the first record from hardware other than the laptop, and the first
able to hold a resident reference for an 8B–72B model — which is what turns
"bit-exact on a 3-layer toy" into "bit-exact on real models".
gate-395-second-stack-vram.md is the third box: an NVIDIA A10G 23 GB
(AWS g5.xlarge), Ubuntu 22.04, torch 2.13.0+cu130 / transformers 5.16.1 /
trl 0.29.1 / peft 0.20.0. It exists to answer a question one card cannot —
whether the long-sequence under-prediction is a property of the formula or of
the stack it was fitted on. Note the direction of the dependency floor:
pyproject.toml [train] now requires transformers>=5.16.1, trl>=0.29.0,
peft>=0.20.0, so this stack is that floor while the RTX 3050 stack below
is beneath it.
run-t4-colab-free-tier.md is the second exception and a much weaker one: a free
Colab Tesla T4 (sm_75, Turing), one run, no repeats, no captured
correctness comparison, on a session that cannot be returned to. It is filed here
because it is the only evidence that the streaming path executes at all on a
pre-Ampere card, not because it gates anything.
probe-rtx5070-what-bounds-streaming.md is the new dev box as of 2026-09-10: an
RTX 5070 Laptop GPU (Blackwell sm_120, 8 GB) on a PCIe 5.0 x8 link, i9-14900HX,
31.7 GB DDR5-5600, two Samsung PM9B1 NVMe, Windows 11, torch 2.14.0+cu130 /
transformers 5.17.0 / peft 0.20.0 / bitsandbytes 0.50.2. Twice the VRAM and ~4.2x
the GEMM ceiling of the RTX 3050, so nothing in it is comparable to the four
gate-v0.72.* records — it is a new baseline, and the file says so in its title.
gate-836-bench-train-contract.md is the same box a fortnight later and adds one
line to its description that turned out to matter: it ran on battery, with the
GPU's enforced power limit at 50.00 W against a 115.00 W maximum. Nothing in
that record is a statement about what this card does on mains, and a re-run on
mains is a new baseline rather than a comparison.
Windows/WDDM matters for reading these: it spills into shared host memory rather
than raising CUDA out of memory, so a run completing is not evidence that its
configuration fits. That is why peak VRAM is reported alongside every throughput
figure, and why the fit decision refuses rather than warns.
Reading the numbers
- Throughput is quoted with the SM clock it was taken at. This card's boost clock varies about 13% between sessions, so a fraction-of-ceiling stated without its clock is not meaningful. Where a GEMM ceiling is compared against, it was measured in the same session.
- The correctness reference always matches the numerics under test — a streamed NF4 run is compared against a resident NF4 run, never against resident bf16, which would hide a real defect inside quantisation error.
- "Bit-exact" is always two claims, never one. The forward (logits,
torch.equal) and the backward (every LoRA gradient tensor) are measured independently and do not always agree: ingate-h100-validation.mdthe forward is exact at every size up to 72B while the backward, pre-repair, was wrong above ~165 MiB per NF4 layer. So "bit-exact at 72B" on its own is not a statement this record makes — check which half, at which quantisation, at which MiB/layer. That file opens with a per-model ledger giving all four for every row, and marks anything unmeasured "not tested" rather than leaving it blank. - Derived figures are labelled as arithmetic. Where a line says "1M tokens = 2.3 h", that is division, not a measured wall-clock run.
Reproducing
The implementation ships in Soup under Apache-2.0. Reproduction commands are in Appendix A of the paper; the correctness protocol runs as part of the project's test suite, so a regression in bit-exactness fails CI rather than reaching a user.
pip install "soup-cli[train]"