Skip to content

SolarRenderer: Few-Step Generative Reference Rendering

Python

docs: measured speedups of the optional kernels for steps 3-5

henry-y19d1239daf66
diff --git a/docs/training.md b/docs/training.md
index a5c1ad7..60e5cca 100644
--- a/docs/training.md
+++ b/docs/training.md
@@ -59,7 +59,8 @@ kept under `TORCH_EXTENSIONS_DIR`, else the temporary directory) against the CUT
container; elsewhere point `SOLAR_CUTLASS_INCLUDE` at a CUTLASS `include/` directory. `SOLAR_ATTN_CPU_CACHE=24` keeps
every block's attention output for the backward pass instead of recomputing it, 24 blocks of them in pinned host memory
(about 25 GB per GPU); raise it if GPU memory runs short. `SOLAR_SP_PACK_B1=1` packs the sequence-parallel exchange
-directly by destination. These switches were measured and checked on this step only.
+directly by destination. This step is the only one checked in full; the other steps below were timed in short runs and
+compared with the original step by step.
**3. Post-train the four-step LoRA with structure guidance** (800 steps at global batch 4). Convert the public DMAD LoRA first,
then point `train.dmd.init_lora` at it and `train.dmd.control` at the control run's checkpoint directory from step 2 (the
@@ -76,6 +77,10 @@ The structure guidance is `train.dmd.structure_guidance` (1 = plain distribution
1.5; take the live adapter at the last step. To render with your own runs, export each run's LoRA with
`python scripts/export_lora.py --checkpoint <checkpoint dir> --out <file>.safetensors`.
+The same switches except `SOLAR_ATTN_CPU_CACHE` (the three model copies leave too little GPU memory for it) make this
+step about 1.75x faster (254.5 s to 145.4 s per step on 8 H200s); the losses stayed within about 1% of the original
+step by step, the student loss within 4.5%.
+
## Causal version
The causal LoRAs follow the same two stages on top of the bidirectional ones. As above, the released causal LoRAs
@@ -89,6 +94,9 @@ control run's checkpoint directory from step 2.
torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/causal_control_lora.yaml
```
+With the same switches (without `SOLAR_ATTN_CPU_CACHE`) a step takes 55.4 s instead of 60.5 s (1.09x): most of this
+step's attention is block-causal and keeps the original kernels.
+
**5. Post-train the causal four-step LoRA with structure guidance** (1,600 steps at global batch 1). The causal control
LoRA from step 4 (`train.dmd.student_init`) is frozen in the student's base; the student is a separate four-step LoRA
that starts as the bidirectional one (`train.dmd.accel_init`) and is distilled on its own block-by-block streams, with
@@ -98,5 +106,7 @@ the bidirectional control LoRA from step 2 (`train.dmd.control`) as the teacher:
torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/causal_structure_guided_4step.yaml
```
+With the same switches (without `SOLAR_ATTN_CPU_CACHE`) a step takes 53.7 s instead of 74.6 s (1.39x).
+
Export both runs with `scripts/export_lora.py` and render with `scripts/render.py --causal`.