Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Gate record — #971: the async NVMe source, measured cold and warm
Status: MEASURED, 2026-09-14, with one negative result and one confound that changed the headline. The cold disk tier got 2.1x–3.1x faster depending on which position-matched pair you take. Warm, with the whole store in the page cache, the async source is SLOWER than the one it replaces — 1.20x in the first run order and 1.03x with that order reversed — measured against a same-day control, not against a published number from another session. What looked at first like a read-ahead depth effect is consistent with the page cache warming across blocks, and is published as the ordering result it is.
Revised 2026-09-14 15:30–16:01, after a whole-branch review. Six things
changed and every one of them is a claim this record made that its own JSON did
not support: a GPU-clock cell printed as —, a missing spread on the row that
anchors the best ratio, a "roughly half the step" that contradicted two other
sections, an undisclosed run order under the warm regression, a §8 JSON produced
by a scratchpad copy of the harness rather than the committed one, and no table
naming its source file. The corrections are dated and inline; nothing was
deleted. §8 was re-measured and the warm trio re-run in reverse order.
Unit convention: decimal GB, matching the other records.
0. The question, and what it is measured against
Layer streaming's disk tier read each layer synchronously, on the compute
thread, through a memory map. probe-rtx5070-what-bounds-streaming.md measured
what that costs: §17 put a cold 70B-shaped NF4 step at 124.5 s (4.1 tok/s,
0.57 GB/s) on an NVMe that reads 3.5+ GB/s, §17a found it unstable by an order
of magnitude between blocks, and §18a put the read-free floor at ~14.25 s at
seq 256. That record's verdict named the fix: "an asynchronous source with
sequential unbuffered reads and pinned staging, off the compute thread".
Tasks 1–5 of #971 built the first half of that — a background reader with its
own header parsing (no mmap), pinned host staging, and a training.stream_read_ahead
depth. This record asks whether the cold step actually got faster, and by how
much. The honest ceiling for this project alone is the ~14 s floor, not the RAM
tier's numbers.
Two comparisons are available and they are not the same thing.
- The published before: §17/§17a/§8, measured 2026-09-12, in a different session, on a card whose boost clock varies ~13% between sessions.
- A same-day control: the shipped synchronous
DiskSource, re-measured in this session on this fixture. Section 1(d) says how that was made without touchingsrc/.
Both are reported. Where they disagree, the same-day control wins, and §2 shows exactly why that ruling mattered.
1. The harness had to change first
stream_probe.py is the instrument, and as it stood it could not have measured
this source honestly. Four changes, all in the harness, none in src/.
(a) --read-ahead N, default 2, passed to build_streamed_model beside
buffers. The source allocates min(N, members) staging slots per distinct
layer shape, so the run's own store line is independent evidence of the depth
actually allocated rather than of the flag that was typed. On the 70B fixture
that arithmetic checks out exactly: 2 × 441.43 MB decoder + 2 × 536.87 MB vocab
= 1.9566 GB, which is the reported store at depth 2.
(b) pin= no longer forces pageable staging on the disk tier. The shipped
line was pin=(args.tier == "ram" and not args.no_pin), correct while the disk
tier allocated a fresh tensor per call and had nothing to page-lock. The async
source reads ahead into reusable host staging, so that expression would have
measured the pageable fallback rather than the shipped path. --no-pin now
selects the pageable arm on either tier.
(c) The Instruments replicas now call _release_source, which the shipped
LayerBufferPool.load_async and LargeLayerBufferPool.load_async do and the
replicas did not. Under pinned staging the source refuses to lend a second layer
while the first is still on loan, so the replica as it stood raised at the second
layer — that refusal working as designed, not a probe bug. The four release
lines were added to both drift-guard needle sets at the same time. Checked both
ways: the needles are present in the shipped bodies, and a deliberately drifted
needle (self.events[slot + 1]) makes the guard raise.
(d) --control-sync-source, which is how the same-day control was made
without changing src/. The runtime no longer constructs DiskSource for
tier='disk', so the switch replaces the one name _build_source imports
lazily with a subclass of the shipped DiskSource that accepts and ignores the
two keyword arguments the async source added. The buffer pool, the prefetcher
and the layer wrapper stay the shipped ones — exactly as before Task 5 — and
_release_source is duck-typed, so it is a no-op against a source with no
release. The control therefore isolates the read path, not a different
scheduler. Every run now prints and records the source class it actually built
(source_class, control_sync_source, read_ahead in the JSON) and refuses
when that disagrees with the flag, so a control block cannot be mistaken for a
measurement; --tier ram is refused up front.
Verification before any cold block
Warm 7B, --steps 2 --warmup 1. These three are FUNCTIONAL verification, not
throughput evidence: another session's full pytest suite was running throughout
(baseline commit charge 28.46 GB of 39.44 GB, free physical 15.04 GB).
| arm | the run's own store line |
step |
|---|---|---|
| disk, pinned (default) | 0.762 GB pinned on tier disk (disk 4.138 GB) |
3.484 s |
disk, --no-pin |
0.762 GB pageable on tier disk (disk 4.138 GB) |
3.627 s |
| ram (control) | 4.138 GB pinned on tier ram (disk 0.000 GB) |
0.819 s |
The first row could not have been produced before change (b): the disk tier reports pinned staging.
These three figures and every store line quoted in this record were read off
the run logs; no JSON was retained for them (--out was written and then
overwritten by the next verification arm). They are labelled functional
verification and are never cited as throughput, so this is a note rather than a
defect — but every other table here names a committed file and these cannot.
(Added 2026-09-14 15:30.)
2. The cold headline
Synthetic Llama-70B shape (random weights — timing only, never a quality
claim), NF4, 36.39 GB on disk against 20–25 GB of free physical RAM, so no page
cache can hold it. Batch 1 × seq 512, 6 timed steps after 2 warm-up,
uninstrumented (§4 says why that qualifier is load-bearing). Sharding was a
cache hit in every block (shard 0.0 s), so every block read the same store.
| block | order | source | step | tok/s | source rate | GPU clock | JSON |
|---|---|---|---|---|---|---|---|
control, sync DiskSource |
2nd | pageable, no reader | 100.35 s (89.6–105.7) | 5.10 | 0.701 GB/s | 180 → 180 MHz | control_sync_cold_synth70b_nf4.json |
async, read_ahead 2 |
1st | 1.957 GB pinned staging | 48.09 s (43.3–52.5) | 10.65 | 1.464 GB/s | 1792 → 2355 MHz | async_cold_synth70b_nf4.json |
control, sync DiskSource |
6th | pageable, no reader | 92.25 s (85.4–101.8) | 5.55 | 0.763 GB/s | 1417 → 210 MHz | control_sync_cold_synth70b_nf4_last.json |
async, read_ahead 2 (repeat) |
5th | 1.957 GB pinned staging | 30.28 s (29.1–34.0) | 16.91 | 2.324 GB/s | 180 → 705 MHz | async_cold_synth70b_nf4_ra2_repeat.json |
Correction, 2026-09-14 15:30. The third row printed — for its GPU clock
and carried no spread, and it is the row the 3.05x rests on. Both were in its
JSON the whole time: 1417 → 210 MHz, and 85.42–101.75 s. They are filled in
above. The spread is wide — wider than the async repeat's 29.1–34.0 — which
strengthens the finding rather than weakening it, and a reader could not see
that. Every table in this record now names the JSON it came from, for the same
reason: nine files, two near-identically named, and the mapping was previously
reconstructible only from started timestamps.
Bytes moved is 70.38 GB per step in every row (157 decoder loads + 2 large), and peak VRAM is 4.378 GB allocated / 4.80 reserved in every row — streaming still bounds the weights exactly as before; only the read path changed.
Finding 2 — the cold step is 2.09x faster at the early position and 3.05x faster at the late one. Both pairs are same-session, same fixture, same shape, adjacent in run order. Against §17's published 124.5 s — the other first-in- session block — it is 2.59x. The spread between those ratios is not noise, it is §3.
Finding 2a — the async source converts a warming page cache into throughput and the synchronous one largely cannot. Across the same warming, early to late in this session, the control moved 100.35 → 92.25 s (1.09x) while the async source moved 48.09 → 30.28 s (1.59x). That is consistent with the shipped source being bound by per-page synchronous faults on the compute thread, where even a cache hit costs a fault and the GPU still waits.
Finding 2b — the GPU is at idle clocks in one control block and not in any
async block, on two samples per block. The harness reads clocks.sm — the
INSTANTANEOUS clock — exactly twice, once before the loop and once after, and
that is all the evidence there is. All six cold samples:
| block | start → end |
|---|---|
| control (early) | 180 → 180 MHz |
| control (late) | 1417 → 210 MHz |
async read_ahead 2 (early) |
1792 → 2355 MHz |
async read_ahead 1 |
1417 → 1095 MHz |
async read_ahead 4 |
202 → 435 MHz |
async read_ahead 2 repeat |
180 → 705 MHz |
Correction, 2026-09-14 15:30. This finding first read "the control's SM
clock is 180 → 180 MHz: it never leaves idle across a 10-minute block …
the async blocks reach 1417–2355 MHz". Two samples ten minutes apart cannot
support "never", and sm_clock_mhz's own docstring warns that this box's boost
clock moved 442–952 MHz inside one measurement run. The second half was
wrong in a second way: 1417–2355 MHz describes two of the four async blocks,
and the headline block itself STARTS at 180 MHz. What the six samples support is
the weaker claim above — one control block sampled at idle on both ends, no
async block did — and that is all this instrument can say. The claim, as
originally written, propagated to benchmarks/README.md and to the changelog
fragment; both are corrected with it.
Against the floor. §18a put the read-free step at 14.25 s at seq 256;
the numbers above are at seq 512, so that is not a like-for-like ratio and
is quoted as an order of magnitude, not a factor. For what the read path costs
in this block, the instrumented copy brackets are the measurement to use, not
that comparison: 84.7% of the headline step (25.168 s of copy per 29.708 s
step, async_cold_synth70b_nf4_ra2_repeat.json), agreeing with §8's independent
per-layer accounting of 82.6–84.9%. Still read-bound, and by about the same
fraction as before.
Correction, 2026-09-14 15:30. The sentence here previously read "the best cold step here is 30.28 s, so the read path is still roughly half the step" — which is a factor, computed from the pair the sentence before it had just declared not like-for-like, and it contradicted §8 and §10's 90–93%. It is replaced by the one figure the block's own instrumentation gives. See also §11: there is no valid read-free floor at seq 512 to compare against.
3. The depth series is an ORDERING result, not a depth result
This is the part that changed the headline, and it is left in the order it happened.
| order | depth | step (uninstrumented) | source rate | free phys at its baseline | JSON |
|---|---|---|---|---|---|
| 1st | read_ahead 2 |
48.09 s | 1.464 GB/s | 19.95 GB | async_cold_synth70b_nf4.json |
| 2nd | (sync control) | 100.35 s | 0.701 GB/s | 19.50 GB | control_sync_cold_synth70b_nf4.json |
| 3rd | read_ahead 1 |
39.23 s | 1.794 GB/s | 24.99 GB | async_cold_synth70b_nf4_ra1.json |
| 4th | read_ahead 4 |
33.59 s | 2.095 GB/s | ~25 GB | async_cold_synth70b_nf4_ra4.json |
| 5th | read_ahead 2 repeat |
30.28 s | 2.324 GB/s | ~24 GB | async_cold_synth70b_nf4_ra2_repeat.json |
Read the first four rows alone and depth 4 is the winner and depth 2 the worst. Read the fifth and that collapses: the same configuration is 48.09 s run first and 30.28 s run last, 1.59x apart with nothing changed but position — larger than the entire spread across depths (33.59–48.09 s).
Finding 3 — depths 1, 2 and 4 are not distinguishable on this evidence, and
what looked like a depth effect is consistent with the page cache warming across
blocks. (That verb was "was" until 2026-09-14 15:30. The ordering result is
proved by the repeat; the mechanism is correlational — free physical memory
grew 19.95 → ~25 GB alongside 48.09 → 30.28 s — and session-scale clock and
thermal effects are not excluded, which the sm_clock_mhz docstring cited in
Finding 2b warns about directly.) The
brief anticipated "if depth changes nothing, say so"; the honest version is
stronger, because the confound would have produced a confident and wrong
recommendation. Free physical RAM grew across the sequence (Windows enlarged the
pagefile during the first control, §6), and with a 36.39 GB store against 20–25
GB of free RAM the cached fraction grows with it.
read_ahead 8 was not run. The brief asks for it only if 4 still moves the
number, and after the repeat there is no evidence that any depth moves it.
Staging cost, which is a clean function of depth and is worth stating because it is what a user pays: 1.515 GB at depth 1, 1.957 GB at depth 2, 2.839 GB at depth 4 (441.43 MB per decoder slot; the two vocabulary groups hold one member each and so take one slot apiece however deep the decoder runs).
4. The instrumentation is not always free here, and one outlier did not reproduce
--step runs each block twice, once plain and once with the CUDA-event
instrumentation on. On the RAM tier those agree to 0.06% (§3 of the probe
record). Here:
| block | plain | instrumented | ratio | JSON |
|---|---|---|---|---|
| sync control (cold) | 100.35 s | 104.95 s | 1.046x | control_sync_cold_synth70b_nf4.json |
async read_ahead 1 |
39.23 s | 38.63 s | 0.98x | async_cold_synth70b_nf4_ra1.json |
async read_ahead 2 |
48.09 s | 545.66 s | 11.35x | async_cold_synth70b_nf4.json |
async read_ahead 4 |
33.59 s | 33.38 s | 0.99x | async_cold_synth70b_nf4_ra4.json |
| sync control (cold, repeat) | 92.25 s | 93.48 s | 1.013x | control_sync_cold_synth70b_nf4_last.json |
async read_ahead 2 repeat |
30.28 s | 29.71 s | 0.98x | async_cold_synth70b_nf4_ra2_repeat.json |
The 545.66 s is a block MEAN over one pathological step, not a block. Its
six instrumented steps are 49.39, 47.71, 46.42, 47.07, 48.32 and 3035.06 s.
Five of the six sit inside the plain block's own 43.3–52.5 s range, so the
instrumentation was free in that block too; the artifact is a single step, and
3024 s of it is inside the copy brackets. (Added 2026-09-14 15:30. The per-step
shape is a sharper statement than the mean, and it would have refuted the 13:15
localisation below immediately, without needing the read_ahead 4 block.)
Correction, 2026-09-14 13:15 → 13:35. On the strength of the first three
rows I wrote that the artifact was "specific to the async source" and localised
it to the read-ahead/release path — _plan_queue's own docstring says the plan
is empty at depth 1, so depth 1 having no lookahead looked like the discriminator.
read_ahead 4 refutes that: it has lookahead and release, and its
instrumentation is free. The localisation was wrong and is left standing with
this correction beside it.
Finding 4 — the 545.66 s block is an outlier that did not reproduce. The same configuration instrumented, run last, is 29.71 s. §17a recorded the same shape of thing on the shipped source (124 s plain against a 746 s instrumented mean, with one step at 3129 s) and attributed it to the page cache. A targeted pytest run belonging to another session appears in that block's AFTER stamp and not in its BEFORE stamp, so contention is a candidate and is not established. It is published as measured. Every headline in this record is an uninstrumented block on both sides, which is like-for-like and unaffected either way.
5. Warm, the async source is SLOWER — a negative result
Mistral-7B NF4, 4.138 GB store entirely in the page cache, batch 1 × seq 512, 8 timed after 3 warm-up, uninstrumented. Run in the order listed; the store is 4.1 GB against ~24 GB of free RAM, so all three were fully cached.
| order | arm | step | tok/s | source rate | JSON |
|---|---|---|---|---|---|
| 1st | async disk, read_ahead 2, pinned |
1.843 s | 277.8 | 4.016 GB/s | async_warm_mistral7b_nf4.json |
| 2nd | control, shipped sync DiskSource |
1.535 s | 333.5 | 4.822 GB/s | control_sync_warm_mistral7b_nf4.json |
| 3rd | RAM tier, same session | 0.816 s | 627.8 | 9.075 GB/s | ram_warm_mistral7b_nf4_today.json |
The async arm ran FIRST, which is the confound §3 exists to catch, and it was
not disclosed here until 2026-09-14 15:30. The three started stamps are
14:11:34, 14:14:21 and 14:15:15, and the hour before them was cold 70B blocks
whose reads would have pressured the 4.1 GB Mistral store out of the page cache.
Three warm-up steps pull it back in, so the timed steps are cached either way —
but §3 had just established that position alone moved an identical configuration
by 1.59x on this box, and the bias here runs against the async arm. That is an
argument, not a measurement, and this record's standard is the difference.
So the trio was re-run in the reverse order, 2026-09-14 15:57–16:01, same
fixture, same flags, same harness, baseline commit charge 27.25 GB of 51.32 GB, free phys 17.50 GB before and 27.39 GB / 17.30 GB after:
| order | arm | step | tok/s | source rate | JSON |
|---|---|---|---|---|---|
| 1st | RAM tier | 1.101 s (1.090–1.121) | 465.1 | 6.723 GB/s | ram_warm_mistral7b_nf4_reverse.json |
| 2nd | control, shipped sync DiskSource |
2.022 s (2.003–2.057) | 253.2 | 3.660 GB/s | control_sync_warm_mistral7b_nf4_reverse.json |
| 3rd | async disk, read_ahead 2, pinned |
2.089 s (2.034–2.141) | 245.1 | 3.544 GB/s | async_warm_mistral7b_nf4_reverse.json |
Finding 5 — with the whole store in the page cache the async source is slower than the source it replaces in BOTH orders, by 1.20x when it runs first and 1.03x when it runs last. The direction survives the confound; the magnitude does not. Neither order gives a position-matched pair — the async arm and the control never occupy the same slot — so 1.03–1.20x is the honest range and neither end is the answer.
Two things make that reading defensible rather than a shrug. The arms that are NOT the async source agree across the two sessions: control/RAM is 1.88x in the first block and 1.84x in the second. And every absolute is ~1.3x slower in the second block (RAM 0.816 → 1.101 s, control 1.535 → 2.022 s), so cross-session absolutes are not comparable at all — only the within-block ratios are, which is exactly why the same-day control exists.
The earlier claim here was "the async source costs 1.19x" (1.84/1.54 rounded from already-rounded inputs; from the stored values it is 1.20x), stated without the order. Against §8's published warm before (2.46 s, 177–208 tok/s) the same number would read as a 1.33x improvement, and that framing would be wrong: today's box is simply faster than that session. This is the single clearest argument for the same-day control, and the reason §8 is not used as the warm baseline here.
The reading is unsurprising once stated: with the store cached, a synchronous
mmap read is close to a memcpy, so there is nothing for a background reader
to hide, and the handoff, the staging copy and the release/drain synchronisation
are pure overhead. The async source exists for a store that does not fit
RAM. When it does fit, the RAM tier is the right answer anyway — 1.88x faster
than the synchronous disk arm and 2.26x faster than the async one in the first
block, 1.84x and 1.90x in the reverse one. (Those two figures read "1.9x" and
"2.2x" until 2026-09-14 15:30; both were rounded from already-rounded inputs,
and the second is 1.84315/0.81554 = 2.2601.)
Ablation on the warm async source (2 interleaved rounds, uninstrumented):
JSON: async_warm_mistral7b_nf4.json (the same file as the first warm block —
the ablation arms are extra records in it).
| arm | round 0 | round 1 |
|---|---|---|
| A baseline | 1.710 s | 1.752 s |
| B no source read, no copy | 0.740 s | 0.738 s |
| C no NF4 dequantisation | 1.789 s | 1.808 s |
| D neither | 0.601 s | 0.589 s |
Removing the read path buys 57% (§10 measured 65% on the shipped source in its own session), and arm B at 0.74 s sits just under the RAM tier's 0.82 s, as it should — B removes the device copy too.
Arm C is SLOWER than arm A in both rounds and that is not explained here. Removing NF4 dequantisation "costs" 4.6% and 3.2%; the sibling record's §10 saw the same sign on the same fixture (−8.7% / −2.4%). The two rounds of each arm differ by 2.5% (A) and 1.1% (C), so the effect is around the size of the round-to-round spread and this record does not claim a mechanism for it. (Added 2026-09-14 15:30 — publishing a negative saving with no comment was the one place this section fell short of the record's own standard.)
6. Two incidental observations
The commit limit moved, and only under the control. Block 1's AFTER stamp
reads commit charge 24.87 GB of 39.44 GB; the first control's reads
21.48 GB of 51.36 GB. Windows grew the pagefile by ~12 GB during the
control and not during any async block. DiskSource memory-maps all 80 layer
shards and holds them for the run (36.39 GB), and a private mapping charges
commit for the file — #926 measured 48.99 GB of mappings raising the charge
46.17 GB. The async source never maps: it parses headers and readintos a
pre-allocated buffer. That is the "no mmap" half of the spec showing up as a
side effect rather than as a designed measurement.
The 30 s no-progress guard never fired. No log from any block contains
"no progress", including the 3035.06 s step (the block whose MEAN is 545.66 s;
see §4). Reading the code, that is expected rather than lucky: the raise in
get is guarded by self._in_flight != idx and idx not in self._queue, so a
read that is merely slow keeps looping and only a reader that has stopped making
progress trips it. The guess was not tested by these runs — nothing here
establishes whether 30 s is the right number for a reader that genuinely stalls.
Follow-up, 2026-09-14 15:30. The whole-branch review then showed the
conjunction could not fire at all on the single-consumer path: the demand push
puts the layer in _queue before the wait, and the reader moves it
queue → _in_flight → _slot_of under the lock, so it is always in exactly one
of the three. It has been replaced by two checks that CAN fire — a reader thread
that exited without recording an error, and one read in flight past a limit
derived from §8 below. The 3035.06 s step would not have tripped the new limit
either, and should not have: its brackets are a consumer waiting, not one read.
7. read_ahead staging, for the operator
| depth | staging on the 70B fixture |
|---|---|
| 1 | 1.515 GB |
| 2 (default) | 1.957 GB |
| 4 | 2.839 GB |
All three measured, read off each run's own store line. The 7B fixture was
only ever run at depth 2, where it reports 0.762 GB; the other depths are
not tabulated for it because they were not measured.
Each level costs one more decoder layer of pinned host memory. Since no depth was distinguishable on throughput here (§3), the default of 2 is not challenged by this record, and 1 is the cheaper choice if host RAM is tight.
8. The step-head stall is the embed fetch, not layer 0 — and neither is material
_plan_queue walks one index out of a single-member group, so a
vocabulary-sized read is planned ahead of decoder layer 0 at the moment the
compute thread blocks on layer 0. benchmarks/harness/layer0_wait.py drives the
probe's own instruments and labels each load bracket with the layer it belongs
to. Cold 70B, depth 2, three steps, no warm-up, instrumented from step 0 — so
its absolute step times (49.2, 39.4, 39.3 s) are not throughput evidence;
the ratio is what it measures. JSON:
layer0_wait_cold_synth70b_nf4.json.
Correction, 2026-10-12. For the embedding entry, #975 supersedes this
section's opening sentence: embed_tokens and lm_head share one device slot,
so each step's _prime() re-copies the whole matrix host to device with nothing
to overlap it (a copy, not a read; fixed by #1178). The record's other findings,
including the 0.07–0.12% materiality ruling, stand.
| step | layer 0, forward / backward | other layers, mean / median / max | vocabulary loads | brackets |
|---|---|---|---|---|
| 0 | 17 / 271 ms | 252 / 283 / 420 ms | 662, 386 ms | 40.6 s of 49.2 s = 82.6% |
| 1 | 17 / 259 ms | 214 / 266 / 533 ms | 33, 22 ms | 33.5 s of 39.4 s = 84.9% |
| 2 | 16 / 268 ms | 213 / 270 / 327 ms | 33, 22 ms | 33.3 s of 39.3 s = 84.8% |
RE-MEASURED 2026-09-14 15:53, and this table is the re-run. The JSON
published first was produced by a copy of the driver in a session scratchpad: it
recorded "driver": "C:\\…\\scratchpad\\layer0_wait.py" and carried no
source_class key at all, while the committed benchmarks/harness/ layer0_wait.py hardcodes its own path and always writes source_class. The
block was identifiable as the async source by inference (pinned: true,
read_ahead: 2, store_gb: 1.9566 — exactly depth-2 async staging, and
impossible for the control, whose store_gb is 0.0), but not verifiable, and
#379 — published numbers whose harness left with the machine — is the whole
reason that file is committed. Re-run from the committed path, baseline commit
charge 27.65 GB of 51.32 GB, free phys 16.96 GB before and 27.25 GB / 17.24 GB after, with another session's Python resident throughout (§9's
standing condition). The new JSON records "driver": "benchmarks/harness/layer0_wait.py" and "source_class": "AsyncDiskSource".
Every figure moved and one finding did not hold. The steps are faster
(49.2/39.4/39.3 against 65.0/55.0/52.9 — a warmer page cache, §3's effect
again), the absolute brackets are smaller, and the first-touch vocabulary read
is 662 ms rather than 874 ms. The previous table's numbers are left above in
this paragraph rather than deleted. Every first-run figure quoted here
(65.0/55.0/52.9 s, 874 ms, layer 0's 23.21 and 19.49 ms, the 90–93% bracket
share) is SUPERSEDED, and because the re-run wrote the same filename the
working tree no longer holds the JSON they came from — it survives in git
history, at 0fd0f42f, and it is the file that records a driver path under a
session scratchpad and no source_class key at all: the copy of the driver
that predates the committed harness. They are quoted as history, never as
evidence, and the commit is named so that "history" stays checkable. (Added
2026-09-14 16:45, mirroring §1's note.)
Finding 8 — layer 0's forward load is the FASTEST load of the step, 16–17 ms against a 213–252 ms mean, because the reader has the whole build and embedding phase to stage it before the compute thread asks for it. Its backward visit is ordinary. The vocabulary reads cost 662 and 386 ms exactly once, on the first touch of the first step, then 33 and 22 ms. The addendum offered a follow-up (refuse to plan out of a single-member group); on this evidence it is not needed, and that is scoped to one fixture, one depth, one shape.
Finding 8a — the largest single compute-stream stall is the VOCABULARY fetch,
not layer 0 — and that IS the step-head item the addendum asked this section to
check. _plan_queue walks one index out of a single-member group, so the
embedding is the step's first read with nothing staged ahead of it. Per step,
from the stalls array of the JSON named above:
| step | total stall | large:model.embed_tokens.weight |
layer000 forward |
mean, other 79 decoder layers |
|---|---|---|---|---|
| 0 | 33.17 ms | 32.66 ms | 0.17 ms | 0.0021 ms |
| 1 | 47.92 ms | 32.61 ms | 14.99 ms | 0.0020 ms |
| 2 | 47.58 ms | 32.91 ms | 14.35 ms | 0.0020 ms |
The embedding out-stalls layer 0 in every one of the three steps, layer 0 is second in every one of them, and the other 79 decoder layers (158 stall entries) sit at 0.002 ms — four orders of magnitude below the embedding in all three steps (15,700x / 16,300x / 16,200x), and three to four below layer 0 in steps 1 and 2 (7,500x / 7,100x; in step 0 layer 0 barely stalls, so that gap is only 84x). In steps 1 and 2 the embedding's stall is nearly its entire load — 32.61 of 33.12 ms, 32.91 of 33.46 ms — i.e. the compute thread waits out the step's first read almost exactly, which is the predicted single-member-group behaviour and not an inference about it. Step 0 is the exception in the useful direction: the reader has the build phase to start that 662 ms first touch, so only 32.66 ms of it reaches the compute thread.
Correction, 2026-10-12. For the embedding entry, the cause in this finding's first sentence and the "single-member-group behaviour" above are superseded by the shared-slot copy in the note under this section's first paragraph (#975). The stall figures and the ruling below stand.
It is still not material, so Finding 8's ruling stands. The step's entire
stall — the embedding, lm_head, layer 0 and all 79 other decoder layers
together, every one of the 162 entries — is 33/48/48 ms, which is
0.07–0.12% of the step (0.067% / 0.122% / 0.121%), against per-load read
brackets that are 82.6–84.9% of the same steps. Refusing to plan out of a
single-member group would buy back at most that.
Correction, 2026-09-14 16:45. This finding first read "layer 0 IS the largest
single compute-stream stall … against a 0.206 ms mean over every other layer",
and benchmarks/README.md carried the same sentence. Both halves are wrong
against the file this section names, and they fail TOGETHER. The 0.206 ms mean
is only reachable by counting the 32.6 ms embedding entry among "every other
layer" — the mean over all 160 non-layer000 entries is 0.2062 ms, of which the
embedding alone supplies 0.2041 (32.657/160), so the comparison figure is
carried almost entirely by the one entry that outranks layer 0. Exclude the
embedding and the mean is 0.002 ms; include it and layer 0 is not the largest.
And it was not true of the FIRST run either, which is worth stating because
the obvious story — a conclusion refreshed with new numbers and left standing —
is not what happened here. Recovered from git history (0fd0f42f, the JSON this
re-run overwrote), that run stalls on large:model.embed_tokens.weight for
39.06 / 39.06 / 32.97 ms against layer 0's 0.002 / 23.21 / 19.49 ms: the
vocabulary fetch out-stalled layer 0 there too, in all three steps. What layer 0
did have in that run was a three-to-four-order gap over the other decoder layers
in the two steps where it stalled at all (23.21 ms against a 0.018 ms mean,
19.49 against 0.002). So the defect was never a stale figure. It was comparing
layer 0 against the decoder layers, and then quoting a mean that silently
included the one entry that comparison had left out.
The same table carries the step's own accounting: the per-load brackets are 82.6–84.9% of the step, agreeing with the headline block's own instrumented share of 84.7% (§2). This step is the read path. (That figure read "90–93%" before the re-run; §2 and §10 are corrected to the re-measured one.)
The read limit in get is derived from this table. The slowest single load
bracket anywhere in the three steps is 662 ms; _MAX_READ_SECONDS = 300 is
three orders of magnitude above it, and ~15x the ~20 s a 441 MB decoder layer
would take at this project's own worst measured source rate of 22 MB/s. A wedge
detector that fired on a merely slow read would be deleted by the first operator
to meet it.
9. Contention, block by block
The box is shared with a contributor-PR review session that runs test suites and cannot be stopped. Every block stamps the commit charge, the free physical memory and the other Python processes immediately before and after itself.
| block | baseline commit charge | free phys | other work |
|---|---|---|---|
| §1 verification (3 arms) | 28.46 GB of 39.44 GB | 15.04 GB | a full pytest suite throughout |
async read_ahead 2 |
21.66 GB of 39.44 GB | 19.95 GB | none at start; a targeted pytest in the AFTER stamp |
| control (early) | 21.82 GB of 39.44 GB | 19.50 GB | none at start; 1 pytest by 12:55, 3 by 12:59 |
async read_ahead 1 |
21.54 GB of 51.36 GB | 24.99 GB | none |
async read_ahead 4 |
21.38 GB of 51.36 GB | 24.88 GB | none |
async read_ahead 2 repeat |
~21 GB of 51.36 GB | ~24 GB | none |
| control (late) | ~21 GB of 51.36 GB | ~24 GB | none |
| warm 7B, three arms | ~21 GB of 51.36 GB | ~24 GB | none |
| layer-0 probe (first run) | 25.46 GB of 51.33 GB | 20.40 GB | none |
| layer-0 probe (re-run, §8) | 27.65 GB of 51.32 GB | 16.96 GB | another session's Python resident (1.3 GB) |
| warm 7B, reverse order (§5) | 27.25 GB of 51.32 GB | 17.50 GB | another session's Python resident (1.3 GB) |
Three of those stamps are approximations (~21 GB / ~24 GB), including both
halves of the late pair the verdict rests on, where the protocol asks for the
measured one-liner immediately before every timed block. The load-bearing half —
whether other work was running — is stated definitely, so this is presentation
rather than a gap in the protocol, but it is a gap in the presentation and is
named rather than tidied away. The two 15:53–16:01 blocks carry real stamps.
The early control ran with pytest present and the early async block did not, which biases that pair in favour of the reported speedup. The late pair (§2, rows 3–4) is clean on both sides, which is why it is the one quoted as the position-matched result.
The warm trio's ORDER is its own confound and §5 now says so, with a reverse-order re-run beside it.
10. Verdict
The cold disk tier is no longer bound by synchronous page faults on the compute thread. Position-matched and same-session, a cold 70B-shaped NF4 step went from 92.25 s to 30.28 s (3.05x) at the late position and from 100.35 s to 48.09 s (2.09x) at the early one, with the source rate rising from 0.70–0.76 GB/s to 1.46–2.32 GB/s. Against §17's published 124.5 s it is 2.59x. The GPU clock samples are deliberately NOT part of this verdict: two per block support only the weaker statement §2b makes, and an earlier draft of this paragraph claimed "the GPU leaving idle clocks for the first time on this tier" — the claim §2b retracts (corrected 2026-09-14 16:45). What bounds it now is still the read: the per-load brackets are 84.7% of the headline step, and §8's independent per-layer accounting puts them at 82.6–84.9%. There is no valid read-free floor at this sequence to compare against — see §11 — so no ratio is quoted for the remaining headroom. The remaining levers are the ones §19 of the probe record already named and this project did not touch: unbuffered sequential reads, both NVMe drives, layer-major micro-batching, and #842.
Two things temper it. Warm, this is a regression (§5): with the store in the page cache the async source costs 1.03–1.20x against the one it replaces depending on run order, and the right answer there is the RAM tier. And no read-ahead depth was distinguishable (§3) once the ordering confound was controlled, so the depth knob is, on this evidence, a staging-cost knob rather than a throughput one.
11. What was NOT measured
- A real 70B. The cold fixture is a synthetic Llama-70B shape with random weights. Nothing here is a correctness claim; the bit-exactness gates are Task 3's and were not re-run.
- Correctness of any kind. This record is timing only.
- The NVMe itself. No block-level read test; 3.5+ GB/s sequential is the drive's published figure, not measured here.
- A second drive, striping, unbuffered/O_DIRECT reads, or #842. All four are named in §19 of the probe record as the remaining levers and none is in #971.
read_ahead 8, and any depth on the warm fixture (§3 says why 8 was dropped).- A read-free floor at seq 512, so there is none to compare the cold step
against. §18a's ~14.25 s is at seq 256 and extrapolated from 128/256; the
sibling record's only seq-512 read-free arm (
probe-rtx5070-what-bounds- streaming.md§18, 39.8 s) is higher than the best step measured here and is invalidated there by WDDM spill. §2 and §10 therefore quote the measured bracket share and no headroom ratio at all. - A position-matched warm pair. §5 now carries both run orders, but the async arm and the control never occupy the same slot in either, so the warm regression is bracketed (1.03–1.20x) rather than measured to a figure.
- Any shape other than batch 1 × seq 512, and any sequence sweep. The read-free floor this is compared against was measured at seq 256/128.
- Whether 30 s is the right no-progress timeout. It never fired (§6); a reader that genuinely stalls was not constructed. The guard has since been replaced (§6's follow-up) and the new limit is derived from §8's slowest measured bracket — which is a bound taken from data, still not a stall constructed and observed on this hardware.
- Any architecture beyond llama (the 70B shape) and mistral, any OS other
than Windows 11, any card other than this one, and any tier interaction with
stream_source: auto's own RAM-vs-disk decision. - A pageable-staging cold arm.
--no-pinwas verified to work on the disk tier (§1) but no cold block was run with it, so the value of pinning on this tier is unquantified.