Skip to content

SolarRenderer: Few-Step Generative Reference Rendering

Python
6.6 KB
Raw Blame

Training

Training runs in three steps on one 8-GPU node with FSDP and sequence parallelism; the configurations in configs/ are set for it. The released weights were trained with the same recipe and the same number of samples at a larger global batch. Training needs about 140 GB of memory per GPU. On one such node a control step takes about 84 s and a post-training step 3 to 8 minutes, so the full schedules take weeks and days respectively. Replace the /path/to/... entries before launching. Data and checkpoints can be local paths or remote object-storage prefixes (<scheme>://...); remote ones are read and written through fsspec, so install the fsspec backend for your storage.

1. Encode the training samples. Each sample is one line of a JSONL file: the target clip, the prompt, and the observations, each laid over the target frame by frame ("all"), on given latent frames ([0] for a first frame), or not aligned (null):

{"id": "clip_0001_full", "target": "clips/clip_0001.mp4", "text": "subject_definitions:\n<Video 1> is ...",
 "observations": [{"media": "structure/clip_0001.mp4", "frames": "all"},
                  {"media": "first/clip_0001.png", "frames": [0]}]}

To train on the released data, download the shards and let tools/hf_samples.py write the clips and the sample list (each clip with the inputs and prompt training gave it; --per-source N keeps N clips per source for a quick test):

hf download Enderfga/SolarRenderer-Data --repo-type dataset --include "train/*" --local-dir ./SolarRenderer-Data
python tools/hf_samples.py --data ./SolarRenderer-Data --out ./samples     # writes ./samples/samples.jsonl

Each clip is one sample. Its inputs (structure and first frame, structure only, first frame only, or text only) are fixed by a hash of its id, and the prompt carries the matching fixed sentences. Encoding runs one worker per GPU and writes samples/ and one index per worker:

H3_REF_ROOT=./MiniMax-H3 torchrun --nproc-per-node 8 tools/omni_preencode.py --samples ./samples/samples.jsonl --out ./encoded
cat ./encoded/index.*.jsonl | sort -u > ./encoded/train.jsonl   # workers help each other, so rows can repeat

The data branch holds the data processing, cleaning and captioning pipeline that produced our training samples; the data itself is on Hugging Face.

2. Train the control LoRA (flow matching, 38,400 steps at global batch 4):

torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/control_lora.yaml

On Hopper GPUs, optional kernels make this step about 1.35x faster (80.8 s to 59.7 s per step on 8 H200s). They are off by default and keep the original precision (bf16 data, fp32 norm statistics, softmax and accumulation):

export SOLAR_HOPPER_ATTENTION=1 SOLAR_HOPPER_FWD192=1 SOLAR_FUSED_ADALN=1 SOLAR_FUSED_QK_ROPE=1 SOLAR_ATTN_CPU_CACHE=24 \
       SOLAR_SP_PACK_B1=1
torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/control_lora.yaml

The attention kernels are FlashAttention's Hopper kernels, compiled on the first launch (about 3 minutes; the build is kept under TORCH_EXTENSIONS_DIR, else the temporary directory) against the CUTLASS headers of the NGC PyTorch container; elsewhere point SOLAR_CUTLASS_INCLUDE at a CUTLASS include/ directory. SOLAR_ATTN_CPU_CACHE=24 keeps every block's attention output for the backward pass instead of recomputing it, 24 blocks of them in pinned host memory (about 25 GB per GPU); raise it if GPU memory runs short. SOLAR_SP_PACK_B1=1 packs the sequence-parallel exchange directly by destination. This step is the only one checked in full; the other steps below were timed in short runs and compared with the original step by step.

3. Post-train the four-step LoRA with structure guidance (800 steps at global batch 4). Convert the public DMAD LoRA first, then point train.dmd.init_lora at it and train.dmd.control at the control run's checkpoint directory from step 2 (the one holding adapter.pt):

hf download ZhengmingYu/DMAD minimax_h3/dmad_minimax_h3_4step_lora_critic.safetensors --local-dir ./dmad
python scripts/convert_dmad.py --h3 ./MiniMax-H3 \
    --src ./dmad/minimax_h3/dmad_minimax_h3_4step_lora_critic.safetensors --out ./weights/dmad_4step_lora.safetensors
torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/structure_guided_4step.yaml

The structure guidance is train.dmd.structure_guidance (1 = plain distribution matching); the released weights use 1.5; take the live adapter at the last step. To render with your own runs, export each run's LoRA with python scripts/export_lora.py --checkpoint <checkpoint dir> --out <file>.safetensors.

The same switches except SOLAR_ATTN_CPU_CACHE (the three model copies leave too little GPU memory for it) make this step about 1.75x faster (254.5 s to 145.4 s per step on 8 H200s); the losses stayed within about 1% of the original step by step, the student loss within 4.5%.

Causal version

The causal LoRAs follow the same two stages on top of the bidirectional ones. As above, the released causal LoRAs saw the same number of samples as the configurations below, at a larger batch.

4. Train the causal control LoRA (flow matching with block-causal attention and teacher forcing, 19,200 steps at global batch 2). It starts from the bidirectional control LoRA: point checkpoint.initialization.student.path at the control run's checkpoint directory from step 2.

torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/causal_control_lora.yaml

With the same switches (without SOLAR_ATTN_CPU_CACHE) a step takes 55.4 s instead of 60.5 s (1.09x): most of this step's attention is block-causal and keeps the original kernels.

5. Post-train the causal four-step LoRA with structure guidance (1,600 steps at global batch 1). The causal control LoRA from step 4 (train.dmd.student_init) is frozen in the student's base; the student is a separate four-step LoRA that starts as the bidirectional one (train.dmd.accel_init) and is distilled on its own block-by-block streams, with the bidirectional control LoRA from step 2 (train.dmd.control) as the teacher:

torchrun --standalone --nproc-per-node 8 -m solarwm train --config configs/causal_structure_guided_4step.yaml

With the same switches (without SOLAR_ATTN_CPU_CACHE) a step takes 53.7 s instead of 74.6 s (1.39x).

Export both runs with scripts/export_lora.py and render with scripts/render.py --causal.