Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
GPU test runs
CI has no GPU, so the CUDA-dependent tests (pytest -m gpu, see
CONTRIBUTING.md) only run when someone runs them on
a card. This file records every such run, so a regression on real hardware has a
last-known-good row to compare against. One row per card and date. Figures are
copied from the record each row links, not re-derived.
Both recorded runs happened before the gpu marker existed (#833), so neither
is a pytest -m gpu run. The scope column says what was actually run.
| Date | Card (compute capability) | Driver / torch / bitsandbytes | Commit | Scope | Result | Found |
|---|---|---|---|---|---|---|
| 2026-08-06 (the session's setup date; STEP 1 itself is undated) | 8x NVIDIA H100 80GB HBM3 | 590.48.01 / 2.13.0+cu130 / 0.50.0 | not recorded | tests/test_v07200.py, test_v07202.py, test_v07203.py, test_v07204.py, --no-cov |
9 failed, 419 passed; with one card visible (CUDA_VISIBLE_DEVICES=0): 6 failed, 422 passed |
three defects: streaming vs nn.DataParallel on a multi-GPU box, the #328 meta leak, a laptop-calibrated GEMM-ceiling bound (gate-h100-validation.md, STEP 1, FINDINGS 1–3) |
| 2026-09-10 | NVIDIA GeForce RTX 5070 Laptop GPU (sm_120), 8151 MiB | 616.92 (CUDA 13.4) / 2.14.0+cu130 / 0.50.2 | bb0ae6b6 |
full suite | 20709 passed, 2 failed, 186 skipped |
streamed NF4 not bit-exact vs resident NF4 on Blackwell: test_issue385_stream_dtype.py [nf4] in float16 and bfloat16 (#776) |
| 2026-09-18 | NVIDIA GeForce RTX 5070 Laptop GPU (sm_120), 8123 MiB | 616.92 (CUDA 13.4) / 2.14.0+cu130 / 0.50.2 | 4ae040fb |
pytest tests/ -m gpu — 88 selected, 23091 deselected; the first run of the marker itself |
1 failed, 86 passed, 2 skipped in 1:46. Reproduced twice (106.11 s and 98.90 s), same single failure. The 2 skips are both correct: test_rewind_mlx.py (no mlx on this box) and test_stream_multi_gpu_guard.py (needs >1 real CUDA device, SOUP_TEST_MULTI_GPU=1) |
one order-dependent failure, test_issue974_direct_range_reads.py::TestOnRealHardware::test_every_region_is_a_pinned_sector_aligned_view_of_a_power_of_two_arena (data_ptr % 4096). It passes alone (1 passed in 2.17 s) and fails only after the test_issue971_async_disk_source.py tests have run in the same process — pinned-arena pressure, in the maintainer's own streaming code. Not a regression: the same failure is recorded on bare main and predates the marker |
| 2026-09-24 | NVIDIA GeForce RTX 5070 Laptop GPU (sm_120), 8151 MiB | 616.92 (CUDA 13.4) / 2.14.0+cu130 / 0.50.2 | fe657585 (branch feat/cuda-graph-decode, off 8736def5) |
pytest tests/ -m gpu — 93 selected, 24897 deselected; includes the two new --cuda-graphs tests in test_cuda_graphs.py |
1 failed, 90 passed, 2 skipped in 2:14. The failure is the same order-dependent test as the row above (test_issue974_direct_range_reads.py::...power_of_two_arena), this time on assert source.pinned is True. It passes alone (1 passed in 0.78 s), and the branch changes no streaming code. Both --cuda-graphs tests pass: bitwise-equal generated IDs against eager for a tiny Llama, and for a tiny Llama carrying an unmerged fp32 LoRA over an fp16 base, with at least one compile through the backend counted (the test counts cudagraphify_impl calls, not recordings) |
nothing new. On the same card, soup infer --cuda-graphs on Qwen2.5-1.5B-Instruct (full, and with a synthetic LoRA) returned responses identical to the run without the flag, greedy, 3 prompts each |
What a green run here does and does not mean
2026-09-18, the first -m gpu run. Three things that are only visible once the marker
exists, recorded so the next reader does not over-read a green row:
- The selection is 88 tests across 18 files, and its composition is not obvious from the
names. The largest contributors are
test_v07204.py(14),test_issue971_async_disk_source.py(11),test_v07203.py(10) andtest_issue968_bnb_nf4_path_divergence.py(9) — i.e. layer streaming and the bitsandbytes 4-bit paths are most of what a card actually exercises here. - A green
-m gpurun does NOT mean #776 is resolved. The opposite: the nine tests intest_issue968_bnb_nf4_path_divergence.pyPASS by asserting the divergence between the two bitsandbytes 4-bit compute paths is still non-zero and still within an order of magnitude of the recorded value. They are written to fail if the paths ever agree, which would be the signal to re-measure #776 — so "86 passed" includes nine tests whose passing is evidence that an open defect is still present. -m gpuruns in under two minutes, against ~30 minutes for the full suite. That is the argument for running it on every card that touches this project: it is cheap, and CI can never run it — GitHub's runners have no GPU, so all 88 skip in all nine cells.
Adding a run
Run pytest tests/ -m gpu --no-cov -v on a clean checkout, then send the details
listed in CONTRIBUTING.md. A failure is as useful
as a pass: file it, and link the issue in the Found column.