Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

119 KB
Raw Blame

Training Tasks & Methods

← Back to the Soup README

SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, and the loop-hardening detectors.

Training a model bigger than your GPU? training.stream_layers: true streams the frozen base from CPU RAM (with NVMe disk overflow) one decoder layer at a time, so peak VRAM is bounded by one layer instead of the whole model. Add quantization: 4bit and an 8B base fits a 4 GB card at batch 1 and 512 tokens per step; on a 128k-vocabulary model every further 512 tokens costs about 1 GB more (see Sizing a streaming run). Works for sft and, from v0.72.4, for dpo / orpo / simpo / kto — DPO's reference model is the same streamed base with its adapters switched off, so it needs no second copy of the model (on an untied checkpoint dpo and kto still hold one copy of the output head per step) — see Layer Streaming.

Contents:


Multimodal SFT (task: sft with modality: vision or audio) requires backend: transformers rather than backend: unsloth. The vision and audio setup paths do not use Unsloth; requesting it is refused at config load instead of silently falling back to transformers. Text SFT continues to support Unsloth.

Validation split & evaluation (training.eval_steps)

data.val_split (default 0.1) holds that share of the rows out of training, and the run evaluates them:

data:
  val_split: 0.1    # the default
training:
  eval_steps: 50    # optional; unset = evaluate at the end of every epoch
  • Default: every epoch. At the end of each epoch the trainer runs one pass over the validation split and records eval_loss. It shows up as the Val loss row on the live panel, is stored as val_loss in the run's metrics in the experiment tracker, and is streamed as TrainEvent.val_loss to the Web UI. The pass is a forward over the held-out rows, a few percent of the run at the default split.
  • training.eval_steps: N evaluates every N optimizer steps instead, counted after gradient accumulation like save_steps. The last step is evaluated too when it is not a multiple of N, so a value larger than the run still evaluates once.
  • Batch size. The evaluation runs at the resolved per-device train batch (after batch_size: auto), not Hugging Face's default of 8, so a card sized to train fits the evaluation too. Layer-streamed runs (stream_layers: true) evaluate the same way, streaming the frozen layers as a training step does.
  • val_split: 0 trains on every row and evaluates nothing, unless the data brings its own validation split: an HF-hub dataset's validation split, or a pre-tokenized cache's val / validation split.

Generation-based tasks (grpo, ppo, online_dpo). Evaluating them means generating completions, which costs about as much per row as training on it. So they do not evaluate by default, and they do not withhold rows either: data.val_split is ignored, every row trains, and the run prints a one-line note saying so. On grpo, set training.eval_steps to hold the split out and evaluate it; TRL generates completions for the held-out prompts and logs eval_loss with the evaluation rewards. On grpo the run's validation number is the held-out reward, eval_reward (the mean reward over the held-out completions, higher is better): the live panel shows it as Val reward, the tracker stores it in the val_reward column of the metrics table, and the training event stream carries it as val_reward (#1389). TRL's eval_loss on grpo is its policy objective at importance ratio 1, not a likelihood: advantages are normalised within each group, so it stays near zero, can be negative, and does not measure held-out quality. It is not recorded or shown as a loss: a grpo run's val_loss series is empty, and Val loss does not appear. The per-task choice lives in VALIDATION_METRICS in utils/eval_schedule.py; every other task records eval_loss as val_loss. TRL needs whole groups of num_generations completions in an evaluation batch, so grpo evaluates at the largest multiple of num_generations that fits in the train batch, and says so when that differs from the train batch. The evaluation's rewards never reach the reward-hack detector, the mitigation controller or the echo-trap detector: they read training generations only. ppo and online_dpo refuse training.eval_steps: TRL's PPO loop never calls evaluate(), and online DPO's evaluate() crashes on prompt-only rows. A generation-based evaluation for those two is a separate feature.

On backend: mlx (#1990) task: sft honours training.eval_steps in the same unit: completed optimizer updates, after gradient accumulation. Soup runs those evaluations itself, after updates N, 2N, ... and once after the last update, over every validation row (a tail smaller than one batch included), weighted by supervised tokens. mlx-lm's own pass would evaluate before the update it is scheduled at and read at most 25 whole batches. An evaluation takes no step and leaves the training order and the model's mode as they were. Unset, MLX keeps mlx-lm's cadence (a quarter of the run). The MLX preference tasks (dpo, orpo) evaluate once per epoch and refuse the field.

Refused at config load. training.eval_steps on the MLX preference tasks (see above), on task: unlearn (it trains on forget_set / retain_set and has no validation split), and with val_split: 0 on local files or remote URIs, where there would be nothing to evaluate.

Evaluated but not tracked: prm and moe_lora_routing. They attach no live training callback (#802), so their eval_loss reaches the trainer's log history and not the experiment tracker.

Until #1223 no trainer on the transformers backend scheduled an evaluation: the default split was held out of training and never used, and val_loss stayed empty.


Seeds & reproducibility (training.seed)

Two knobs, both unset by default:

training:
  seed: 1234        # weight init of new params, data order, dropout
  data_seed: 99     # optional — data order ONLY, so init stays fixed

seed reaches TrainingArguments.seed (which Trainer.__init__ hands to transformers.set_seed, covering random, numpy and torch) and the multipack FFD sampler. data_seed reaches TrainingArguments.data_seed; set it alongside seed to vary only the order rows are seen in while holding initialisation fixed.

Why you want this. Without a seed knob every run of a config took the same default, so "run it again with a different seed" was impossible: replicates of one arm differed only by row permutation and GPU nondeterminism. That understates run-to-run spread, and spread is the yardstick a real between-arm difference has to beat. Three replicates at three seeds is the cheapest honest error bar you can put on a training change. Create one config per replicate so the seed and its output directory stay together in the recorded recipe:

# soup-seed-1.yaml (repeat for seeds 2 and 3)
training:
  seed: 1
output: runs/seed-1
for s in 1 2 3; do
  soup train --config "soup-seed-$s.yaml" && echo "seed $s done"
done

"Unset" does not mean the same thing for both fields. An unset seed resolves to 42, HuggingFace's own TrainingArguments default. An unset data_seed stays None, which HF reads as "follow seed" rather than as a seed of its own, so leaving it out is not the same kind of default as leaving seed out. The multipack sampler still gets 0, the value it has had since v0.37.0. The fields are Optional[int] rather than defaulting to 42 precisely so the trainer can tell "unset" from "explicitly 42" and keep those different historical defaults intact.

What changes for a run that sets neither field. The values it trains at are the same as before: seed 42, data_seed at None. What is new is when the seed arrives. Before #353 nothing called set_seed ahead of get_peft_model, so lora_A and any freshly initialised classification head were drawn from torch's default generator, which is seeded from entropy once per process. An unseeded run's initialisation therefore varied from one process to the next, and it is now deterministic at 42. If you were getting replicate spread out of runs that set no seed, that is where it was coming from: those runs are identical to each other now, and varying a replicate means setting training.seed on purpose.

Bounds: [0, 2**32 - 1] (set_seed feeds numpy.random.seed, which rejects anything outside that range), 0 is a legitimate seed, and a YAML true is rejected rather than silently becoming seed 1. data_seed is forwarded to Accelerate's dataloader configuration and needs accelerate >= 1.1.0; below that, transformers warns and ignores it (seed is unaffected).

Scope. Every task wrapper threads both fields into the TrainingArguments subclass it builds, and applies seed before it loads the model, so the LoRA adapter and any freshly initialised head are drawn at the configured seed rather than at whatever the process happened to be sitting on (#353). unlearn builds no Trainer at all, and its RMU control vector follows training.seed too, staying at 0 when the seed is unset. The pretrain and layer-streaming paths pick seed up for their samplers as well.

Through v0.73.0 this reached the SFT trainer only, so training.seed: 7 on a DPO or GRPO run was accepted and silently trained at 42.

On the MLX backend (backend: mlx), SFT seeds MLX's own RNG (mx.random, which draws LoRA A and dropout) with training.seed before the adapter is built, and numpy's global RNG, which mlx-lm's batch shuffle draws from, with training.data_seed (or training.seed when that is unset) right before training starts (#1850). Before #1850 it seeded neither and warned MLX backend ignores: training.seed, so every MLX run trained on a different batch order.

Not a determinism guarantee. A fixed seed makes the software RNG reproducible. It does not make CUDA kernels bit-reproducible — non-deterministic atomics, autotuned algorithms and a different GPU or library version can still move the last digits. For bit-exact reruns you also need torch.use_deterministic_algorithms(True), which Soup does not set for you.


Rewind — which row spiked the loss (soup rewind)

While an SFT run trains, Soup writes <output>/rewind.jsonl: one line per micro-batch with the dataset rows in it, each row's mean supervised-token loss, and its supervised-token count. When the loss jumps, soup rewind reads that file and names the rows that carried the step. It needs no model or GPU, and runs in seconds.

training:
  rewind_log: true   # default; set false to write nothing
soup rewind                      # most recent run: list spikes, detail the worst
soup rewind <run_id> --step 340  # rank the rows of one step
soup rewind <run_id> --top 25    # how many rows to list (default 10)
soup rewind <run_id> --json r.json --no-preview

A spike is a step whose loss is non-finite, or more than 2x the median of the previous 20 measured steps. Rows are ranked by their share of the step's summed token loss, so one long, badly-formed row stands out even when its per-token mean is ordinary. Example output (illustrative numbers):

                Step 340 rows
 Row    Loss   Tokens   Share  Preview
  91  6.8120    3,900   94.1%  iVBORw0KGgoAAAANSUhEUgAAAyAAAAJYCAYAAAC…
  12  1.1034      212    0.8%  Sure — here is a summary of the article…
row 91: 94% of step 340's loss, 3,900 tokens
Inspect or drop this row, then retrain.

Scope: task sft on the transformers and MLX backends, single process, on the plain text path. The recorder turns itself off, with one warning naming the reason, for resumed runs, packing / padding_free, multipack, vision, audio, a pre-tokenised dataset, and anything else it cannot attribute row by row. Previews are shown only when the dataset on disk still matches the one the run trained on.

What it costs. Writing one line per micro-batch measured under 0.1 ms per micro-batch, which is negligible against any real step, and a run's peak memory was unchanged with the recorder on and off. The file grows without a cap: roughly 14 MB per 50,000 micro-batches at batch 8, or about 100 MB for a million rows over three epochs. Delete it, or set rewind_log: false, if that matters to you.

What it contains. Integer row indices, one loss and one token count per row — never any text from your dataset. Previews in soup rewind are rebuilt at read time from your own data file, and only while its fingerprint still matches. The indices and token counts are still weak metadata about a private dataset, so treat the file as you would the output directory it sits in.


Full fine-tuning (lora.r: 0)

Train every parameter, no adapter:

base: HuggingFaceTB/SmolLM2-135M
task: sft
training:
  quantization: none    # full-FT trains float weights
  lora:
    r: 0                # <- no adapter; the base itself trains

lora.r: 0 is the supported spelling for plain full fine-tuning. It is the one soup's classifier trainer has read as "no adapter" since v0.71.12, and the one soup card already resolves to a dense model rather than an adapter — so the model card, the registry entry and the trainer all agree without extra flags. On a rank of 0 the SFT trainer skips get_peft_model entirely: soup train prints Full fine-tuning: N parameter tensor(s) trainable (lora.r=0, no adapter) instead of LoRA applied, and the output directory holds a complete model, not an adapter to merge.

Requirements, each rejected at config load with the reason named: task: sft, backend: transformers, modality: text, and quantization: none (quantized weights cannot be trained directly — use LoRA on top of them, i.e. QLoRA). It is mutually exclusive with every LoRA feature (use_dora / use_vera / use_olora / use_rslora / rank_pattern / alpha_pattern / init_strategy / moe_lora / use_longlora / relora_steps / loraplus_lr_ratio) — a LoRA knob next to r: 0 is a contradiction rather than something to silently ignore — and with the other two "LoRA off" modes, Spectrum (unfrozen_parameters) and LISA (lisa_enabled), since each independently decides what trains. It cannot be combined with layer streaming: streaming keeps the decoder on the meta device and trains only the adapter, so there would be nothing to full fine-tune.

task: pretrain refuses lora.r: 0 at config load (#1813): its trainer always attaches a LoRA adapter, so a rank of 0 would only fail inside peft when training starts. To train the base weights there, use LISA (lisa_enabled: true), which trains a rotating set of whole layers without an adapter and so still accepts lora.r: 0, or switch to task: sft.

freeze_layers / freeze_ratio do stay legal with r: 0 — "train everything above layer N" is a real technique — and the trainer respects whatever they froze instead of silently unfreezing it. If they leave nothing trainable the run is refused rather than burning GPU-hours on a no-op.

Pick between the three "LoRA off" modes by how much you want to train:

Spelling Trains Use when
lora.r: 0 everything you have the VRAM and want the strongest baseline
unfrozen_parameters (Spectrum) a hand-picked / SNR-ranked set you want full-FT quality on the layers that matter
lisa_enabled (LISA) a rotating random subset you want full-FT quality at LoRA-like memory

Full fine-tuning needs far more memory than LoRA: optimizer state alone is ~8 bytes per parameter for AdamW. batch_size: "auto" estimates memory from a LoRA-shaped model, so on a full-FT run it errs optimistic — set batch_size explicitly, or start low and raise it. (This is not new to r: 0; the same is true of the Spectrum and LISA full-FT paths.)

Load dtype (#339, #471, #492): a frozen base — LoRA, or QLoRA — loads at the checkpoint's own dtype (torch_dtype="auto") instead of always upcasting to fp32, since the base never receives an optimizer step. Measured on an H100, Llama-3.1-8B, LoRA: 48,241 MiB peak -> 18,658 MiB, a 28.9 GB / 2.59x saving. On a pre-Ampere CUDA card (T4 / P100 / V100 / GTX 16xx / RTX 20xx), "auto" would give bf16 storage while training compute correctly stays fp16 — the same card question _resolve_mixed_precision already asks — so the frozen base explicitly loads torch.float16 there instead. All three "LoRA off" modes above (lora.r: 0, unfrozen_parameters, lisa_enabled) are the trainable-base case and explicitly load torch.float32 master weights instead — a deliberate precision choice for the parameters an optimizer actually steps, not an accidental upcast, and unaffected by the card check above.


Which tasks apply training.quantization

quantization defaults to 4bit, but not every trainer reads it. These eight load the base (and, for distill, the teacher) unquantised whatever the field says; prm loads it as fp32 master weights (see Process Reward Model):

task quantization notes
distill none only student and frozen teacher both load unquantised
classifier, reranker, cross_encoder none only full fine-tune unless classifier_lora: true
prm none only always a full fine-tune; a lora block is refused (lora.r: 0 is allowed)
moe_lora_routing none only base frozen, only the router trains
unlearn none only policy and reference copy both load unquantised
asr none only full fine-tune unless asr_lora: true

For these tasks an unset quantization resolves to none, so the stored config and the VRAM pre-flight describe the run that actually happens (#795).

An explicit 4bit or 8bit (or load_in_8bit: true) loads with a warning and resolves to none, because every config Soup dumped while 4bit was the default carries it literally. The warning names the task and the release that will refuse it; set quantization: none to silence it. A Quant Menu value (gptq, awq, ...) or a 4-bit-only setting such as bnb_4bit_quant_storage is refused at config load, naming the task. Every other task applies the field as documented in Performance & Quantization.


Continual-learning rehearsal (--replay)

Fine-tuning on a new task can erase the old one. Rehearsal is the standard defence: mix a slice of the old data back in.

soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1

or in soup.yaml:

data:
  train: new_task.jsonl
  replay: old_task.jsonl
  replay_ratio: 0.1      # fraction of the FINAL mixed set
  replay_seed: 0

The ratio is a share of the final set, not of the new data: n_replay = round(r/(1-r) · n_new). At 0.1 over 1000 new rows that is 111 replay rows → 1111 total → exactly 10%. The console reports what it did:

Replay: +26 old rows interleaved (30.2% of 86)

Three guarantees worth knowing:

  • Interleaved, never appended. A trailing block of old rows would mean the model sees them all in the final steps — a second mini-finetune, which is the failure rehearsal exists to prevent.
  • train only; validation stays pure new-task, so your eval still measures the task you are learning. Measure old-task retention separately with soup eval custom / soup ship.
  • An undersized pool reports a shortfall rather than repeating rows — a row seen twice per epoch is a different experiment.

The replay file gets its own format detection, so the old set may be alpaca while the new one is sharegpt.

Scope (v1): sft and pretrain only, and incompatible with packing/multipack — those concatenate rows into fixed blocks, so the ratio stops being meaningful at block boundaries. Both are rejected with a clear error rather than silently mis-mixing.

Honest result. Validated at proof-of-mechanism scale (SmolLM2-135M + LoRA): training task B from a model that knew task A, replay retained A 7% better than a no-replay control (loss 0.565 → 0.526) at a ~5% cost to task B — the expected trade. But forgetting without replay was only +4%, i.e. mild: LoRA on a 135M model barely drifts. The direction and the mechanism are proven; the effect size at full fine-tuning or 7B+ is unproven on a 4 GB box.

Loop Hardening

Six surfaces protect the training loop from the failure modes that cost a real GPU-hour. The schema + math kernels shipped in v0.70.0; the live trainer-callback wiring shipped in v0.71.11, validated end-to-end on SmolLM2-135M.

# Reward-hacking detector — auto-halt when the policy starts gaming the RM
# (InfoRM cluster-separation index, Wang et al. 2024 arXiv:2402.09345)
soup train --config soup.yaml \
    --reward-hack-detector info_rm --reward-hack-halt   # halt on HACK verdict

# Closed-loop reward-hacking MITIGATION (v0.71.26) — detect AND self-correct
soup train --config grpo.yaml --reward-hack-mitigation kl_control   # raise KL, recover
soup train --config grpo.yaml --reward-hack-mitigation log_only     # observe only, no action

# Wasserstein-distance distillation between two models that SHARE a
# tokenizer, e.g. two sizes in the same family (#681: wasserstein / topk_align
# forward the student's token ids to the teacher unchanged, so the pair must
# tokenize identically; for a genuinely different tokenizer see
# wasserstein_aligned below).
# (Universal Logit Distillation, Boizard et al. 2024 arXiv:2402.12030)
#   training:
#     uld_strategy: wasserstein
soup train --config soup.yaml

# Cross-tokenizer distillation for DIFFERENT tokenizers, e.g. Llama -> Mistral,
# no shared vocab needed (v0.71.18). Aligns student/teacher token sequences
# over decoded character spans, so you can distill a GPT-2 BPE student from a
# Llama SentencePiece teacher. A student token with no teacher counterpart
# (a byte-fallback piece, an empty piece, a student-only special token) has
# no target and is left out of the loss.
#   training:
#     uld_strategy: wasserstein_aligned

# MiniLLM reverse-KL distillation (Gu et al. 2024 arXiv:2306.08543).
# Config-only: there is no --minillm-* flag beyond --minillm-on-policy below
# (#979). Offline blend: mix ratio is the teacher weight in the reverse-KL
# target and must be > 0 (ratio 0 is KL(student || stopgrad(student)) and is
# rejected). On-policy: mix 0 is legal — student-only sampling, loss still
# KL(student || teacher).
#   training:
#     minillm_enabled: true
#     minillm_teacher_mix_ratio: 0.3
#     minillm_pretrain_anchor_weight: 0.1
#     minillm_pretrain_anchor_path: ./pretrain.jsonl
soup train --config soup.yaml

# MiniLLM TRUE on-policy rollout (v0.71.18, Gu et al. §3.1) — sample a fresh
# autoregressive rollout from the per-token teacher/student mixture each step,
# then length-normalised reverse-KL. training.minillm_rollout_length tunes the
# rollout (auto min(max_length, 32)). Mix 0 here means student-only sampling.
# --minillm-on-policy is the one real flag here; minillm_enabled: true still
# has to be set in the config (#979).
soup train --config soup.yaml --minillm-on-policy

# Mid-epoch checkpoint for PPO/GRPO — TorchTune punts this; Soup ships it
#   training:
#     rl_checkpoint_save_every_steps: 500
#     rl_checkpoint_keep_last: 3
#     rl_checkpoint_include_optimizer: true
soup train --config grpo.yaml

# Iterative DPO loop driver — sample -> RM-score -> re-pair -> retrain
# (drop --plan-only to run the loop; --plan-only just renders the per-round plan)
soup iterative-dpo \
    --base-model meta-llama/Llama-3.1-8B \
    --reward-model ./output_rm \
    --prompts ./prompts.jsonl \
    --output-dir ./iterative_dpo_out \
    --rounds 5 \
    --pairs-per-round 1000

# RAGEN echo-trap detector — auto-halt when trajectories collapse to self-repetition
# (Zhu et al. 2025 arXiv:2504.14437)
#   training:
#     echo_trap_enabled: true
#     echo_trap_threshold: 0.6
#     echo_trap_halt: true
#     echo_trap_tokenizer_aware: true
soup train --config grpo.yaml

training.echo_trap_tokenizer_aware (also available as the real --echo-trap-tokenizer-aware option) switches echo-trap n-grams from whitespace tokens to the active tokenizer's integer ids. This catches subword repetition that punctuation-heavy decoded text can hide, but the score becomes tokenizer-specific rather than vocabulary-agnostic.

The info_rm detector stays silent on any step whose median split leaves either half constant — a two-valued reward (e.g. reward_fn: accuracy's 0/1) always does, and so does a continuous reward where half the batch lands on the same value (e.g. rewards clipped at 0). The separation index there would measure the success rate rather than reward-model health, so the step casts no vote: no baseline is recorded and nothing halts or mutates β (one warning is logged per run). The single exception: once a baseline exists, a step where every reward is identical still votes 0.0 — a 0/1 run never records a baseline and stays silent, while a genuinely collapsed continuous reward reads HACK as before. For two-valued rewards use reward_hack_detector: rm_ensemble or the length/repetition signals instead. Changed: before this version a constant-half split could record an inflated separation (~3×10⁴ at a 50% first step) as the baseline and classify later, healthier steps as HACK — detector logs and mitigation_log.jsonl files from runs before this version are not comparable.

Closed-loop reward-hacking auto-mitigation (v0.71.26)

The detectors above halt; training.reward_hack_mitigation (or the --reward-hack-mitigation flag) makes the trainer self-correct mid-run. It requires reward_hack_detector on a grpo transformers run, and has four modes:

  • log_only — observe only. Appends a per-step mitigation_log.jsonl under the run's output dir (the InfoRM/ensemble drop, the OK/WARN/HACK verdict, reward mean/std, completion-length trend, repetition) and provably never mutates β. Run this first to see hacking before you let a controller act on it.
  • kl_control — a reversible bang-bang + hysteresis controller. When a multi-signal vote (reward_hack_signals: the detector drop + length_trend + repetition) stays above reward_hack_trip_band for reward_hack_dwell_steps, it multiplies β by reward_hack_kl_gain (clamped to [reward_hack_beta_floor > 0, reward_hack_beta_ceil], never crossing 0); after reward_hack_release_patience below-band steps it relaxes β back toward the floor. Dwell + release-patience stop it flapping. β is written to both trainer.beta and trainer.args.beta so it takes effect on stock GRPO and Soup's GRPO variants (and trainer.args.kl_coef on PPO).
  • pid_lagrangian — a PID-Lagrangian controller (Stooke et al. 2020) that holds the hacking signal at reward_hack_signal_target (Kp/Ki/Kd with integral anti-windup via reward_hack_integral_clamp), plus an escalation ladder: raise β → after reward_hack_rollback_patience persistent-HACK steps roll back to the last-good RL checkpoint (needs rl_checkpoint_save_every_steps) → after reward_hack_max_recovery_attempts rollbacks, early-stop with a plain-English give-up explanation.
  • Anti-gaming hardening (any control mode): reward_hack_signal_smoothing (ema/median over reward_hack_smoothing_window), reward_hack_conservative_on_disagreement (when detectors disagree, keep KL high + guard against a bimodal reward-distribution collapse), and reward_hack_reward_shaping (subtract a bounded reward_hack_shaping_strength penalty on the gamed proxy — length/repetition/sentinel — over the reward-fn seam).

Scope: proof-of-mechanism only. Validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 (all four modes live, including a real mid-run rollback). The on-GPU proof is GRPO-only. On task: ppo the reward-hacking and echo-trap flags are refused (reward_hack_detector, reward_hack_mitigation, echo_trap_enabled): the trl PPO trainer this build supports takes no reward functions, so the reward/completion signal those callbacks read is never captured, and a run carrying them would be announced and then inert. Whether the loop suppresses hacking without collapsing true reward on 7B+ with a real reward model is an open, community-validatable question. reward_hack_mitigation ∈ {kl_control, pid_lagrangian} is mutually exclusive with ref_model_ema_alpha (both drive the KL/ref dynamics).

Every detector composes with v0.34 soup why (anomaly explainer), v0.32 spike recovery, and the v0.53.11 #127 GRPOStabilityCallback so a single GRPO run can have InfoRM + echo-trap + spike-recovery + in-place ref-model EMA all active simultaneously without duplicating trajectory / state collection. The reward-hack and echo-trap callbacks read the per-step rewards through a shared, thread-safe capture buffer (Soup wraps your reward functions so it never has to monkeypatch TRL); rm_ensemble needs ≥2 reward functions to compute a divergence. The MiniLLM teacher-mix is an offline distribution-blend analog of the paper's on-policy teacher-mixed sampling, and ULD compares the distributions after clamping teacher ids to the teacher vocab (correct for same-family / extended-vocab pairs; a genuinely different tokenization needs a sequence-alignment step). The reference-model EMA (--ref-model-ema-alpha) updates in place — no full state_dict round-trip — so it is cheap at 70B+ scale.

Unlearning (task='unlearn', NPO / SimNPO / RMU)

GDPR right-to-be-forgotten + CSAM/PII leak response, productized. Three method backends:

  • NPO — Negative Preference Optimization (DPO-shaped negative-only loss; needs a reference model).
  • SimNPO — length-normalised NPO without a ref model (faster, more stable on long sequences).
  • RMU — Representation Misdirection Unlearning (residual-stream noise on forget inputs).
# unlearn.yaml
base: HuggingFaceTB/SmolLM2-135M
task: unlearn
data:
  train: traces.jsonl
  forget_set: gdpr_deletion_set.jsonl   # rows to unlearn (messages / prompt+completion / text)
  retain_set: capability_anchors.jsonl  # optional — anchors general capability
training:
  unlearn_method: npo           # or simnpo / rmu
  unlearn_alpha: 0.5            # retain-set weighting [0.0, 10.0]
# Run the unlearn loop (validated on SmolLM2-135M — NPO/SimNPO drive forget loss down).
soup train --config unlearn.yaml --yes

# Score the run on TOFU / MUSE / WMDP (OK / MINOR / MAJOR verdict).
soup eval unlearning <run-id> --benchmark tofu --evidence evidence.json --output report.json

task: unlearn is live (v0.71.9): it loads a LoRA-wrapped policy, a frozen reference copy (NPO / RMU), and the forget / retain JSONL sets, then optimises the per-method loss — NPO's (2/β)·mean(-logσ(-β·(π_logp − ref_logp))) drives the policy's forget-set log-prob below the reference (= forgetting), while the retain set anchors capability. Run NPO/SimNPO with a retain_set — without one the policy has no utility anchor and Soup warns loudly.

Unlearning honors training.optimizer, scheduler, warmup_ratio, weight_decay, max_grad_norm, batch_size, gradient_accumulation_steps and gradient_checkpointing (any enabled tier checkpoints the whole policy model). The default batch_size: auto resolves to 1 for unlearning rather than to the memory estimate the other trainers use, so a config makes the same number of updates on any card. This memory-constrained loop processes one example at a time and accumulates gradients for batch_size * gradient_accumulation_steps examples per optimizer update; a final partial group is averaged by its actual size. initial_loss and final_loss remain unscaled per-example losses, and total_steps and the 2,000-step budget continue to count examples rather than optimizer updates.

data.max_length now controls forget and retain tokenization instead of the old 256-token cap. Its default is 2,048, so existing configurations can use more memory and take longer. Set data.max_length: 256 to retain the old sequence-length limit, or explicitly choose a value that fits the model and device.

Three orthogonal axes: Forget Quality (pre/post forget-loss delta), Model Utility (retain-accuracy preserved), PrivLeak (membership-inference AUC distance from 0.5). Bundled mini-fixtures for all three benchmarks ship in the box (v0.71.1 added MUSE + WMDP alongside the existing TOFU set), so --benchmark muse|wmdp runs without supplying evidence. The WMDP forget-set probes ship redacted (placeholder prompts + REFUSED responses) — Soup never bundles verbatim hazardous-knowledge content.

Continued Pre-training

Continue training a model on raw text for domain adaptation:

base: meta-llama/Llama-3.1-8B
task: pretrain

data:
  train: ./data/corpus.jsonl   # {"text": "..."} or plain .txt files
  format: plaintext
  max_length: 4096

training:
  epochs: 1
  lr: 1e-5
  quantization: 4bit
soup init --template pretrain
soup train

Knowledge Distillation

Train a small student model to match a larger teacher's output distribution. Every train/val row must keep at least one causal-loss target after truncation at data.max_length, and a row without one is refused at setup by split and row number, as SFT does. Both token-level and sequence-level distillation require modality: text; vision and audio inputs are refused at config load because this trainer is text-only.

base: HuggingFaceTB/SmolLM2-135M
task: distill
modality: text
backend: transformers

data:
  train: ./data/chat.jsonl
  max_length: 2048
  chat_template: chatml

training:
  teacher_model: meta-llama/Llama-3.1-8B
  distill_divergence: forward_kl   # kl | forward_kl | reverse_kl | js
  distill_temperature: 2.0
  distill_chunk_size: 256          # token chunk size for divergence evaluation
  distill_checkpoint: true         # non-reentrant activation checkpointing
  epochs: 3
  lr: 5e-5

Loss = student CE + (T**2) × KL(teacher_logits / T || student_logits / T). Teacher is loaded once, frozen via requires_grad_(False) + .eval(), and its inputs / logits are auto-bridged across CPU / CUDA devices.

Distillation runs on backend: transformers only; backend: mlx and backend: unsloth are refused at config load. Gradient accumulation uses the number of shifted, non-masked training targets across the complete optimizer window. Splitting the same rows into unequal-length microbatches therefore preserves the full-batch token mean instead of weighting every microbatch equally.

The token-divergence kernel evaluates all three divergences in FP32 and deliberately returns an FP32 scalar, including for FP16/BF16 logits. Forward-KL values can therefore also differ slightly from the previous low-precision calculation. The log-space reverse-KL and Jensen-Shannon formulas prevent finite losses with non-finite gradients; the upcast also preserves small losses that would underflow to zero. Non-finite logits propagate into the loss/gradients, allowing AMP's GradScaler to skip overflowed steps rather than aborting training with a validation exception.

FP32 intermediates cost time and memory. The maintainer's PR #736 measurement on an RTX 5070 (BF16, B=1/S=512/V=32000, forward plus backward) found the originally submitted kernel took 1.71 times as long and 1.24 times the peak memory of main. That measurement included finite-value guards removed here: without them it took 9.40 ms versus 6.39 ms, with the same 376.5 MiB versus 303.6 MiB peak. These are hardware-specific measurements, not a benchmark of this revised implementation, which also avoids unused probability tensors. Two FP32 logits copies alone occupy about 7.8 GiB at B=4/S=2048/V=128256; budget for additional intermediates as well.

Chunking and Activation Checkpointing

Distillation with large vocabularies (e.g. 150k for Qwen 2.5) and long sequences can exhaust GPU memory due to massive intermediate logit and probability tensors. Soup provides two controls to bound activation memory:

  • distill_chunk_size: Evaluates divergence in chunks of active response tokens (labels != -100). Pre-filtering immediately sheds unmasked prompt and padding tokens from retained autograd memory, while chunking bounds transient peak activation tensors during divergence evaluation. Moderate chunk sizes (e.g. 64 to 256 tokens) balance peak memory reduction with kernel launch overhead.
  • distill_checkpoint: Wraps chunk evaluation in non-reentrant activation checkpointing (torch.utils.checkpoint.checkpoint(..., use_reentrant=False)). Discards intermediate log_softmax and probability tensors during the forward pass and recomputes them during backward, substantially reducing retained autograd tensor bytes at the cost of recomputation time in the backward pass. Note that enabling distill_checkpoint: true without setting distill_chunk_size processes all active tokens in a single chunk, which reduces retained bytes via checkpointing but leaves transient peak activations unbounded.

Set distill_mode: sequence (default token) to train on the teacher's generated continuations instead of per-token logit matching - a hard-label, cross-tokenizer-friendly KD that works when student and teacher do not share a vocabulary. sequence mode is mutually exclusive with the cross-tokenizer uld_strategy logit path (they are different objectives over the same task; the trainer rejects the combination at setup). Rows with no prompt turn are skipped, the count and first positions are printed at setup, a split left with no usable row is refused before training, and a refusal names the row's position in your data. (v0.71.12)

Sequence Classification

Train a classifier head on top of any base model — supports single-label, multi-label, and cross-encoder reranking.

base: BAAI/bge-base-en-v1.5
task: classifier              # or `reranker`, `cross_encoder`
modality: text
backend: transformers

data:
  train: ./data/labelled.jsonl   # rows: {"text": "...", "label": "spam"} or {"text": "...", "label": [0, 2]}
  max_length: 256

training:
  num_labels: 3
  classifier_kind: single_label   # or `multi_label`
  label_names: [ham, spam, promo] # required when labels are strings
  epochs: 5
  lr: 2e-5
  batch_size: 32

Row shapes by task:

  • task: classifier: Single-input sequences via {"text": "sample", "label": 1} or {"text": "sample", "label": "spam"} (also accepts ChatML rows carrying label). Multi-label datasets pass lists of active label indices or names: {"text": "sample", "label": [0, 2]} or {"text": "sample", "label": ["ham", "promo"]}.
  • task: reranker: Query and document text via {"text": "query doc", "label": "relevant"} (or an integer class index).
  • task: cross_encoder: Paired inputs via {"text_a": "query", "text_b": "doc", "label": 1} or {"question": "query", "answer": "doc", "label": 1}.

Routes classifier / reranker / cross_encoder through AutoModelForSequenceClassification. Multi-label heads cap at 1024 entries per row, dedup via set conversion, and reject null bytes in label strings.

Add a lora: section to train a frozen encoder + LoRA adapter classifier instead of the full model — the small adapter plus the (freshly-initialised) classification head train, the encoder backbone stays frozen:

training:
  num_labels: 3
  lora:
    r: 16
    alpha: 32

(v0.71.12)

Reasoning Effort + EOT Control

gpt-oss-style reasoning-effort control for instruction tuning.

training:
  reasoning_effort: high      # low | medium | high
  train_on_eot: true          # do NOT mask the EOT/EOS token in the loss

reasoning_effort injects <|reasoning_effort|>high<|/reasoning_effort|> into the system turn (creating one if absent). train_on_eot=True makes the model learn when to stop generating by training on the trailing EOS token instead of masking it out. Both load only on task: sft, tts and distill.

EBFT / GDPO Loss Variants

training.gdpo_variant is refused at config load (#1309). This breaks configs that set standard, length_normalized, or margin: previously they loaded but silently trained the default loss, because supported TRL versions (0.29 and later) do not expose the DPOTrainer.dpo_loss hook. Every non-null value is refused; unset and null still load. No shipped recipe, template or example sets the field.

Remove gdpo_variant to train plain task: dpo for standard. For length_normalized, task: simpo is the nearest objective, not an identical replacement. margin has no equivalent. Plain DPO, without a GDPO setting:

task: dpo
training:
  dpo_beta: 0.1

EBFT (ebft_variant: structured | strided) is refused at config load (#1230): it is not yet a distinct objective. The term it added scored each position's logits against that position's own input token, with no causal shift, so it rewarded copying the input over predicting the next token; shifted onto the next token it is the model's own cross-entropy, so the loss would count cross-entropy twice. A config that set it never trained correctly. Remove ebft_variant and ebft_temperature; the refusal stays until the intended EBFT objective is implemented from its reference.

GRPO Objective Variants

Soup ships live math kernels for 6 GRPO objective variants in addition to the default. Set grpo_variant in training and the trainer automatically subclasses trl.GRPOTrainer to route compute_loss through the matching kernel:

task: grpo
training:
  reward_fn: accuracy
  num_generations: 4
  grpo_variant: gspo         # sequence-level length-normalized ratio
  # or: dapo / dr_grpo / bnpo / rft / two_sided
  # grpo_delta: 0.2          # required when grpo_variant=two_sided; optional for gspo

Variants:

  • standard — DeepSeek-R1-style baseline (delegates to TRL's compute_loss).
  • gspo — Group Sequence Policy Optimization (sequence-level length-normalized ratio with clipping).
  • dapo — decoupled asymmetric clipping (eps_lo=0.2, eps_hi=0.28).
  • dr_grpo — token-sum without per-sample length normalisation.
  • bnpo — length-normalised PPO surrogate.
  • two_sided — symmetric clipping with operator-supplied grpo_delta.
  • rft — rejection-sampling fine-tuning (only positive-advantage tokens contribute).

Every variant applies grpo_beta (default 0.1) as a KL penalty against the reference policy, where trl's own GRPO loss puts it: trl's per-token estimator exp(ref - logp) - (ref - logp) - 1, weighted by grpo_beta and reduced the same way as that variant's policy term. rft applies it only to the accepted completions it trains on. The β that the reward-hack controller sets at runtime (kl_control / pid_lagrangian) reaches every variant the same way. Set grpo_beta: 0 for a KL-free run (no KL penalty against the reference policy, skipping the reference forward pass entirely), as used by the published DAPO and Dr. GRPO recipes (#1247).

With a non-zero grpo_beta, a variant run logs the same kl metric as trl's stock loss: the batch mean of that per-token estimate over the completion tokens (over the accepted tokens only, for rft), as kl in training and eval_kl in evaluation.

The stability callback (EMA ref-model update, replay buffer, TIS alert counter) attaches automatically when any of ref_model_ema_alpha / replay_buffer_size / tis_threshold / etc. is set.

Process Reward Model (PRM)

Train a scalar reward head over stepwise-supervised reasoning chains. Data format is the v0.42.0 prm shape — one row per {prompt, completions: [step1, step2, ...], labels: [r1, r2, ...]}:

task: prm
data:
  format: prm
  train: ./prm_train.jsonl
  max_length: 2048
training:
  epochs: 1
  lr: 1.0e-5

The trainer loads AutoModelForCausalLM, attaches an nn.Linear(hidden, 1) reward head, and computes MSE between predicted scalars at step-boundary tokens and the per-step labels. The reward head is saved inside the model checkpoint (reward_head.* in model.safetensors) and the tokenizer is saved alongside it, so the resulting directory is loadable standalone.

PRM trains every base parameter along with the head, so all of them load as fp32 master weights on every device. On CUDA, autocast runs the forward pass in bf16 (fp16 on pre-Ampere cards); on MPS it uses bf16 where the runtime supports it; CPU trains in fp32. With the default optimizer (AdamW), budget about 16 bytes per parameter before activations (fp32 weights, fp32 gradients and two fp32 AdamW moments), which is what the VRAM pre-flight predicts for task: prm when batch_size is an integer (with the default batch_size: auto the pre-flight does not run). The saved checkpoint is fp32, and without DeepSpeed loading the fp32 base needs about twice the host RAM of a bf16 load. Under DeepSpeed, each rank loads the base in fp32 on the host before the engine exists (4 bytes per parameter of host RAM per rank); the engine then casts it to its bf16/fp16 dtype, keeps its own fp32 master copy and saves a 16-bit checkpoint. A bf16 base without master weights would round most updates away at these learning rates (#1235).

PRM-guided GRPO (process-supervised RL)

Use a trained PRM as the per-step reward inside GRPO — the o1-era process-supervision signal. Set training.prm_reward to a PRM directory (a task=prm checkpoint) or a Hub id, org/name or org/name@revision; a Hub id is downloaded (weights, tokenizer and custom-code files) when the reward is built, before training, and a repo that is not a Soup-trained PRM (no reward head) is refused there, naming the configured value (#1466). The PRM splits each generated completion into reasoning steps (newline heuristic), scores every step with its reward head, and folds the per-step scores into one scalar reward that GRPO optimises. It replaces reward_fn and rides the existing reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller still observes it (TRL logs it as rewards/prm_reward).

task: grpo
backend: transformers        # required (the PRM reward runs a transformers forward)
modality: text               # required
data:
  format: chatml
  train: ./grpo_prompts.jsonl
training:
  prm_reward: ./my-prm       # a `soup train task=prm` checkpoint dir (or HF id)
  prm_aggregate: min         # weakest-link (default) | prod | last
  num_generations: 4
  grpo_beta: 0.04

prm_aggregate='min' (weakest-link, the standard PRM aggregation) is the safe default; prod assumes calibrated [0,1] step scores (Soup's PRM head is trained with unconstrained MSE, so prod can blow up on uncalibrated labels).

Bundled rollout environments. Three deterministic pure-Python toy environments seed the openenv rollout path out-of-the-box — pair any of them with rollout_backend=openenv:

training:
  rollout_backend: openenv
  rollout_func: soup_cli.envs.calculator:rollout   # or retrieval_qa / guess_number
  reward_fn: verifiable
  verifiable_domain: math

Ready-made recipes: grpo-env-calculator, grpo-env-retrieval-qa, grpo-env-guess-number. The environments are deterministic single-shot prompt/answer seeders (the live openenv contract passes only the seed prompts, not the model) — not interactive multi-turn episodes.

Scope: proof-of-mechanism only — validated on SmolLM2-135M with a tiny synthetic PRM (the PRM reward scores good completions above bad and drives GRPO's advantages). Not a production reward-model claim; scale validation is help-wanted (#286).

Online DPO (task='online_dpo') — judge in the loop (v0.71.31)

Unlike offline DPO (static prompt/chosen/rejected rows), Online DPO generates two completions per prompt on-policy each step and asks a judge — or a reward model — which is better; the winner becomes chosen, the loser rejected. The judge closes the loop. Wraps TRL OnlineDPOTrainer; data is prompt-only (like GRPO). Transformers + text only.

base: HuggingFaceTB/SmolLM2-135M-Instruct
task: online_dpo
data:
  train: ./data/prompts.jsonl      # prompt-only (or any format — prompts are extracted)
training:
  online_dpo_judge: "ollama://llama3.1"   # a pairwise judge (ollama://|https://|http://localhost)
  # OR: reward_model: ./my-reward-model   # exactly one of judge / reward_model
  online_dpo_loss_type: sigmoid           # sigmoid | ipo
  online_dpo_max_new_tokens: 64
  dpo_beta: 0.1
  lora: { r: 8, alpha: 16, target_modules: auto }

The judge is Soup's own OpenAI-compatible JudgeEvaluator adapted to TRL's BasePairwiseJudge (swap-debiased: a winner is only recorded when both A,B and B,A orders agree). Recipe: online-dpo-smollm2-135m. Proof-of-mechanism was validated on SmolLM2-135M with a synthetic judge (not a production RLHF claim; #286). An https:// judge URL uses OPENAI_API_KEY only when its host is api.openai.com; other hosts are called as an OpenAI-compatible server without that key. An online_dpo_judge whose host is a private, link-local or reserved IP literal is refused when soup.yaml loads (loopback stays allowed); address an internal judge by its hostname. A host written only as numbers that is not a valid IPv4 address (10.0.0.256, 4294967296) is refused as well.

A pair the judge cannot rank is left out of the loss: a tie, a failed or unreadable judge call, or a verdict that changes when the two completions are swapped. Such a pair adds no gradient, and the loss is the mean over the ranked pairs of each batch. The share of unranked pairs is logged as judge/invalid_rate, with a WARNING the first time it happens and on every step in which nothing was ranked. The logged loss and train_loss average only the batches that ranked a pair, and a logging window in which nothing was ranked logs no loss at all. A step in which nothing was ranked is not skipped: the optimizer still steps, so AdamW momentum and weight decay keep moving the weights and the learning-rate schedule advances. That drift is bounded: if the judge ranks none of 32 pairs in a row (for example because its server is down), training stops with an error that names the judge, and a shorter run in which it ranked no pair at all fails the same way instead of saving an adapter. Before this, TRL trained every such pair as if the second completion had won (#1225).

Weighted Multi-Objective Preference Loss

Mix DPO / SimPO / ORPO / IPO terms in one training run by setting preference_loss_weights (must sum to 1.0):

task: preference
training:
  preference_loss_weights:
    simpo: 0.6
    orpo: 0.4

The combine wrapper computes a weighted sum via the in-tree compute_dpo_term / compute_simpo_term / compute_orpo_term / compute_ipo_term kernels. The loss with the largest weight is the primary: its trl trainer is the one that is built, and on trl 0.29 the per-sequence log-probs the kernels need are read out of that trainer's own forward pass. Where they come from depends on which trainer is built:

Blend Result on trl 0.29
simpo + orpo Trains. The CPO (SimPO) and ORPO trainers both return their per-sequence log-probs (concatenated_forward).
dpo or ipo has the largest weight, with any of dpo, ipo, simpo, orpo beside it Trains. Both build trl's DPOTrainer: the policy log-probs are taken from the logits of its own forward pass, the reference log-probs from its compute_ref_log_probs.
simpo or orpo has the largest weight, and the blend names dpo or ipo Trains. The CPO / ORPO trainers run no reference, so the blend runs one: the same forward once more, without gradients and with the LoRA adapter switched off. Refused with init_strategy: pissa / olora / loftq or relora_steps, where that is not the starting model ("Which reference" below).
bco mixed with anything Rejected at config load (data format incompatible).

What a term is. In a blend, simpo and orpo are their preference terms only — the margin / odds-ratio objective — without the likelihood (NLL) term the standalone preference_loss: simpo / preference_loss: orpo runs carry. A {simpo: 1.0} blend is therefore not the same number as a preference_loss: simpo run, and training.cpo_alpha does not affect it. Use training.preference_loss when you want the full standalone loss. Each term does honour its own config field: training.simpo_gamma for SimPO, training.orpo_beta for ORPO, whichever loss has the larger weight or not.

The dpo and ipo terms are the losses preference_loss: dpo and preference_loss: ipo train: DPO's sigmoid loss with training.dpo_beta, and IPO's squared loss with training.ipo_tau, where each side's log-ratio is divided by its response length, as trl does. They read those two fields whichever of them is the primary, so {dpo: 0.5, ipo: 0.5} and {ipo: 0.5, dpo: 0.5} are the same loss. dpo_beta_schedule is not available with a blend.

Which reference. Under a dpo / ipo primary it is trl's: the frozen ref adapter, a copy of the adapter as it started. Under a simpo / orpo primary it is the base model with the adapter switched off. For a new LoRA adapter, whose update starts at zero, the two compute the same thing.

That holds while the base weights are the ones that were loaded, which covers the default init (init_strategy: random), DoRA and rsLoRA. init_strategy: pissa, olora and loftq (and use_olora) move part of every targeted weight into the adapter and leave a residual in the base, and relora_steps merges the adapter into the base at every restart; with the adapter off the model is then no longer the one the policy started from. A dpo / ipo term under a simpo / orpo primary is refused with either of them, before the model is loaded. With such an init, give dpo or ipo the largest weight: trl's copy of the adapter carries the residual's other half, so its reference is still the starting model. relora_steps has no such way out: trl's ref adapter sits on the same base weights, so a blend that names dpo or ipo refuses it whatever the largest weight is (ReLoRA).

Which tokens. The terms are computed on the primary trainer's batch, and the two trainer families do not render a row the same way: the CPO / ORPO batches start with the tokenizer's BOS token, and the DPOTrainer's need not (with a plain tokenizer and no chat template they do not). An even split between a reference loss and a reference-free one therefore depends on which key comes first, because the first key is the primary.

Cost. The policy is run once per step, as in a single-loss run. A dpo / ipo primary runs the reference twice: trl computes its own reference log-probs inline and keeps only their means, so the blend asks the trainer for them again (compute_ref_log_probs, without gradients). A simpo / orpo primary runs the reference once, and only when the blend names dpo or ipo; with val_split it runs on evaluation batches too.

With 16-bit logits (bf16 or fp16 autocast) a dpo / ipo primary also keeps one more tensor the size of the logits for the backward pass, next to the one trl's own loss keeps: 2 bytes × rows × tokens × vocabulary, where a batch of 4 pairs is 8 rows. That is about 250 MiB for 4 pairs of 512 tokens and a 32,000-token vocabulary. With float32 logits it keeps none.

Evaluation. With val_split set and a simpo / orpo primary, trl's prediction_step calls get_batch_loss_metrics directly, so the eval loss is the primary loss's own trl loss — not the blend's value. Training and evaluation therefore report different numbers for the same batch. With a dpo / ipo primary, trl's prediction_step goes through compute_loss, so the eval loss is the blend's value.

MoE Model Support

Fine-tune Mixture of Experts models (Mixtral, Qwen3-30B-A3B, DeepSeek V3) with ScatterMoE LoRA — applies LoRA to both attention layers and expert FFN layers:

base: Qwen/Qwen3-30B-A3B
task: sft

training:
  moe_lora: true              # target expert + attention layers
  moe_aux_loss_coeff: 0.01    # router load-balancing loss
  quantization: 4bit

Soup auto-detects MoE architectures. Not every task reads moe_lora: the per-task table in Performance and quantization lists the tasks that apply it and the ones that refuse it at config load.

soup init --template moe
soup train

Vision / Multimodal Fine-tuning

Fine-tune vision-language models (LLaMA-3.2-Vision, Qwen2-VL, Pixtral) on image+text data:

# Install vision support
pip install "soup-cli[vision]"

# Create a vision config
soup init --template vision

# Train
soup train --config soup.yaml
base: meta-llama/Llama-3.2-11B-Vision-Instruct
task: sft
modality: vision

data:
  train: ./data/vision_train.jsonl
  format: llava
  image_dir: ./data/images
  val_split: 0.1

training:
  epochs: 3
  lr: 1e-5
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Supported vision data formats:

LLaVA:

{"image": "photo.jpg", "conversations": [{"from": "human", "value": "<image>\nDescribe this image."}, {"from": "gpt", "value": "A cat on a mat."}]}

ShareGPT4V:

{"image": "chart.png", "conversations": [{"from": "human", "value": "<image>\nWhat does this show?"}, {"from": "gpt", "value": "Quarterly revenue."}]}

A relative image path resolves against data.image_dir, or against the data file's directory when it is unset. A Hub dataset or a remote URI has no data file directory, so set data.image_dir when its rows hold image paths; see Data Pipeline Pro. Decoded in-memory PIL images (such as from a Hugging Face Hub Image() column) and Parquet image structs ({"bytes": ..., "path": ...}) written by datasets are also accepted; a struct's path follows the same directory resolution and path containment rules as a plain path string.

soup data inspect resolves image paths the same way, or uses --image-dir when provided, and automatically shows image statistics (count, formats, missing files) for vision datasets.

Audio / Speech Fine-tuning

Fine-tune audio-language models (Qwen2-Audio, Whisper) on audio+text data:

# Install audio support
pip install "soup-cli[audio]"

# Create an audio config
soup init --template audio

# Train
soup train --config soup.yaml
base: Qwen/Qwen2-Audio-7B-Instruct
task: sft
modality: audio

data:
  train: ./data/audio_train.jsonl
  format: audio
  audio_dir: ./data/audio
  val_split: 0.1

training:
  epochs: 3
  lr: 1e-5
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Audio data format:

{"audio": "recording.wav", "messages": [{"role": "user", "content": "Transcribe this audio."}, {"role": "assistant", "content": "Hello world."}]}

ASR fine-tuning (task='asr', Whisper) — v0.71.32

Fine-tune Whisper on your accent or domain. whisper-tiny (39M) and base (74M) train on a 4 GB GPU. Rows are {"audio": <path>, "text": <transcript>} under data.format='asr'; audio decodes to 16 kHz mono through the hardened loader.

base: openai/whisper-tiny
task: asr
data:
  train: ./data/train.jsonl
  format: asr                 # rows: {"audio": "clip.wav", "text": "hello world"}
  audio_dir: ./data/audio     # audio paths resolve here (containment-checked)
training:
  epochs: 3
  lr: 1e-4
  batch_size: 2
  asr_language: en            # optional; sets + persists the decoder prefix
  asr_task: transcribe        # transcribe | translate
  asr_lora: true              # optional LoRA on q/v; default = full fine-tune
  quantization: none
output: ./out
{"audio": "clip0.wav", "text": "hello world"}

Transcribe + score after training:

soup infer --task asr --model ./out --input eval.jsonl --output preds.jsonl --audio-dir ./data/audio
# -> preds carry {"transcription", "wer", "cer"} per row + a corpus WER summary

Notes: task='asr' requires backend='transformers' and a Whisper base (a non-Whisper base is rejected before download). asr_language/asr_task persist to an asr_generation.json sidecar so soup infer --task asr restores them (override with --asr-language/--asr-task). WER/CER use a light normalizer — good for before/after deltas, not leaderboard-comparable absolutes. whisper-large-v3-asr ships parse-only (needs a larger GPU).

GRPO Plus — Objective Variants, Long-Context RL, Multi-Turn Agents

Soup ships seven GRPO objective variants, between-rollouts vLLM standby, four agent-rollout backends, seven stability/efficiency knobs, plus Process Reward Models.

# soup.yaml — DAPO with replay buffer and TIS truncation masking
base: meta-llama/Llama-3.1-8B-Instruct
task: grpo
data:
  train: ./prompts.jsonl
  format: chatml
training:
  reward_fn: accuracy
  num_generations: 8
  # New: GRPO objective variants
  grpo_variant: dapo                  # one of: gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard
  # grpo_delta: 0.2                   # required when grpo_variant: two_sided (optional for gspo)
  grpo_fp16: true                     # FP16 RL (unsloth parity)
  # Long-context + memory-efficient RL
  # long_context_grpo: true           # staged for future Tiled MLP; refused as of v0.77 — #808
  vllm_sleep_mode: true               # between-rollouts vLLM standby — LIVE (vLLM >= 0.7)
  # Multi-turn agent rollout — openenv is LIVE: your function's rows replace the prompt dataset
  rollout_backend: openenv            # one of: art / ruler / nemo_gym / openenv
  rollout_func: my_module:my_rollout  # module:function resolver (openenv; trusted operator code)
  # Stability / efficiency knobs
  ref_model_ema_alpha: 0.99           # EMA sync policy → reference
  replay_buffer_size: 2048
  async_grpo_prefetch: true           # overlap rollout + train
  tis_threshold: 2.0                  # truncated importance sampling
  mask_truncated_completions: true    # paired with tis_threshold
  defer_rerolling: true
  skip_zero_advantage: true
  off_policy_mask_threshold: 0.5

Process Reward Models (stepwise-supervised):

# soup.yaml
base: meta-llama/Llama-3.1-8B
task: prm                              # New: Process Reward Model
data:
  train: ./prm_dataset.jsonl
  format: prm                          # stepwise-supervised data shape
training:
  epochs: 3
  lr: 1e-5

Vision RL on Qwen2-VL / Pixtral / InternVL (Staged):

# soup.yaml
base: Qwen/Qwen2-VL-7B-Instruct
task: grpo
modality: vision
data:
  train: ./vlm_prompts.jsonl
  format: llava
training:
  reward_fn: accuracy
  # vision_grpo: true                  # staged for VLM-RL; refused as of v0.77 — #808

All flags shipped as schema gates in v0.50.0. vllm_sleep_mode, openenv rollout, and PRM training (task: prm) are live. Other rollout backends (art, ruler, nemo_gym) raise "not yet validated", while unconsumed staged flags (e.g. long_context_grpo, vision_grpo) warn in v0.76 and are refused as of v0.77 (#808).

DPO Training

Train with preference data using Direct Preference Optimization:

base: meta-llama/Llama-3.1-8B-Instruct
task: dpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  epochs: 3
  dpo_beta: 0.1
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

A base published as a vision-language wrapper (MiniMax-M3, model_type: minimax_m3_vl) is not registered under AutoModelForCausalLM, so a plain DPO config cannot load it. Set modality: vision and DPO loads it the way SFT's vision path does, through AutoModelForImageTextToText with the model's processor, and trains the language tower; the LoRA targets resolve to that tower as before. Preference rows carry no images, and soup recipes verify builds the same class the trainer loads. backend: unsloth does not read modality for DPO and is refused at config load for it (#1393).

Preference Variety — BCO + Unified Dispatcher + KL Variants

Five preference losses live behind one config knob. Pick a loss without renaming your task, anneal β over training, and periodically refresh the frozen reference.

BCO (Binary Classifier Optimization)

Same input format as DPO; rows are split internally to TRL's BCO unpaired schema ({prompt, completion, label}).

task: bco
data:
  train: ./data/preferences.jsonl
  format: dpo
training:
  bco_beta: 0.1

Unified preference dispatcher

Use task: preference + training.preference_loss to swap losses without touching task. Hyperparameter sweeps over the loss type itself become trivial.

task: preference
data:
  train: ./data/preferences.jsonl
  format: dpo
training:
  preference_loss: dpo   # or simpo, orpo, ipo, bco

Legacy task: dpo / task: simpo / etc. remain first-class — the unified surface is additive.

KL-controlled DPO variants

Anneal β over training with dpo_beta_schedule:

task: dpo   # or task: preference + preference_loss: dpo, or task: ipo
training:
  dpo_beta: 0.1
  dpo_beta_schedule: linear   # linear | cosine | exponential
  dpo_beta_end: 0.01

Gated to DPO-family tasks (dpo, ipo, or preference with preference_loss in {dpo, ipo}); transformers backend only.

Periodically sync the frozen reference model with the active student using dpo_ref_regen_epochs:

training:
  epochs: 4
  dpo_beta: 0.1
  dpo_ref_regen_epochs: 2  # copies .default. adapter weights into .ref. every 2 epochs

With LoRA, TRL creates a frozen .ref. adapter on the model. On every Nth epoch (1-indexed, skipping epoch 0), parameters from the active .default. adapter are copied into the frozen .ref. adapter under torch.no_grad(). Requires LoRA (lora.r >= 1); full fine-tuning (lora.r: 0) is refused at config load (#1345).

Multi-objective preference loss

task: preference
training:
  preference_loss_weights: {simpo: 0.7, orpo: 0.3}

Schema validates 2–5 entries summing to 1, and rejects bco mixed with a paired loss at config load. On trl 0.29 every blend of dpo, ipo, simpo and orpo trains; which reference and which tokens the terms use depends on the loss with the largest weight (see Weighted Multi-Objective Preference Loss). Set training.preference_loss for a single loss.

GRPO Training (Reasoning)

Train reasoning models with Group Relative Policy Optimization (DeepSeek-R1 style):

base: meta-llama/Llama-3.1-8B-Instruct
task: grpo

data:
  train: ./data/reasoning_train.jsonl
  format: sharegpt
  max_length: 4096

training:
  epochs: 3
  lr: 1e-5
  grpo_beta: 0.1  # KL penalty; 0 = KL-free (DAPO / Dr. GRPO)
  num_generations: 4
  reward_fn: accuracy   # or 'format', or path to custom .py
  lora:
    r: 64
    alpha: 16
  quantization: 4bit
# Create a reasoning config
soup init --template reasoning

# Train
soup train --config soup.yaml

Gradient watchdog (#342). If non-finite gradients (NaN or Inf) appear during a GRPO run, the optimizer step is skipped: weights and optimizer state are unchanged, and the step still counts toward the step total and the LR schedule. The skipped-step count is logged at the end of the run (as a console warning when the fraction exceeds 5%) and persisted in trainer_state.json so soup adapters audit can see it.

Built-in reward functions:

  • accuracy — 1.0 when the completion's final answer matches the gold's, else 0.0 (no partial credit)
  • format — checks for structured <think>...</think> reasoning blocks
  • judge_rank — an LLM judge scores each prompt's completions against each other, from 0 to 1; see LLM-judge ranking reward below

accuracy and verifiable math read the completion and the gold with the same parser. The final answer is, in this order of precedence:

  1. the rest of the line after the last #### (a #### Final Answer heading means the next line);
  2. the content of the last \boxed{} (a space before the brace and nested braces such as \boxed{\frac{1}{2}} are fine);
  3. what follows the last The answer is or Answer: (also **Answer**:, and the answer may be on the next line), up to the end of its clause: a . , , or ; outside brackets, or at the connective because or since (case-insensitive, requiring whitespace on both sides; not as). A comma or semicolon that a number follows continues a list instead (41, 42 or 43), and a parenthetical aside belongs to the clause. So The answer is Washington, D.C. reads Washington, The answer is 42 (six times seven). reads 42, and The answer is (3, 4). reads (3, 4).

A box outranks a phrase, and a \boxed{} or phrase that comes after a #### line outranks it (the line was a markdown heading, or an answer the text went on to correct). That includes chatter: #### Paris followed by I hope this answer is helpful! reads helpful, so end a completion at its #### line. A completion with none of these is read by its last line and its last number, so Six times seven is 42. and even Not 42. read as 42; only the final number counts, so listing candidates earns nothing.

An answer phrase's number is read from its own clause: The answer is 41 apples, not 42. reads 41, because the clause ends at the comma. A clause that names more than one distinct value is a hedge and states no answer: The answer is either 41 or 42., Answer: 41 or 42, The answer is 42 (or 43). and the answer is 41, 42 or 43 score 0.0 against every gold, and a gold written that way is refused. A because / since inline or after punctuation ends the clause (The answer is 42 because 6*7=42. and The answer is 42, because 6*7=42. both read 42), while parenthetical justifications belong to the clause and hedge it (The answer is 42 (6*7=42).). The same value twice is not a hedge (42 (i.e. 42.0)), and the digits of one bracketed or LaTeX answer ((3, 4), \begin{pmatrix} 3 \\ 4 \end{pmatrix}, \frac{14}{3}, 2^{10}) or of a time or a ratio (3:45, 1:1,000) are not separate values; such an answer is compared as text. \boxed{} and #### answers are compared whole, so a list there is one answer, and it can only match a gold that is the same list.

A numeric gold is compared by value, so #### 1,000, \boxed{1000} and The answer is $1000. all match a gold of 1000. Any other gold (\frac{14}{3}, p - q, Paris) is compared as text, ignoring case and whitespace, $, \(...\) and \[...\], \left / \right, and \dfrac / \tfrac versus \frac; so \boxed{\dfrac{14}{3}} matches a gold of \frac{14}{3}. Both sides also drop the trailing punctuation . , ; : !, LaTeX thousands separators such as 1{,}000, the LaTeX spacing commands \, \! \; \: and \ , and a Unicode minus sign. A \\ row break is kept whole, so a matrix matches however its rows are spaced. One \text{} / \textbf{} / \mathrm{} / \mbox{} wrapper is unwrapped to its contents, ^\circ / ^{\circ} / ° are dropped, a compact \frac argument is braced to match whether it is a single bare character or an already-braced group, with or without a space before it (\frac12, \frac1{2}, \frac{1}2, \frac 34 and \frac9{19} all read \frac{N}{D}), and a one-letter variable prefix reads its right-hand side (x = 7 reads 7, on either side), though when both sides name a variable and the names differ (x = 3 against y = 3), the pair scores 0.0, since a directrix or an asymptote's variable is part of its answer. Units are still not stripped (42 apples against 42), and nothing is evaluated (\frac{1}{2} does not equal 0.5).

For GRPO, Soup preserves source dataset columns and TRL passes them to reward functions as keyword arguments. An Alpaca output or the final assistant turn in ShareGPT/ChatML is also exposed as answer; an explicit answer column takes precedence. Gold-dependent built-ins validate their inputs before generation:

Reward Required source metadata
accuracy or verifiable math answer, or an assistant reference response, that states a final answer
verifiable code expected or answer
verifiable json_schema schema

A gold states its final answer with ####, \boxed{}, The answer is or Answer:, or by being the bare answer on one line (42, Paris, \frac{14}{3}). A row whose gold states none (for example a multi-line reference solution with no marked answer, or one under a #### Solution heading: a #### line that is not a number and has more text after it may be a markdown heading, so a gold cannot rely on it), or whose answer phrase hedges between values (The answer is either 41 or 42.), is refused before generation, with its split, row number and field and a count of the other rows with the same problem, because such a gold would score every completion 0.0 and give GRPO no signal. A dataset that mixes numeric and LaTeX golds, such as MATH-500, loads under math too; validation prints how many golds are non-numeric and therefore compared as normalised text.

Custom reward functions — point to a Python file:

# my_reward.py
def reward_fn(completions, **kwargs):
    """Score each completion. Return list of floats."""
    return [1.0 if "correct" in c[-1]["content"] else 0.0 for c in completions]
training:
  reward_fn: ./my_reward.py

Custom rewards can read any preserved source column through kwargs. They must return exactly one finite numeric score per completion; Soup checks this contract and reports the reward name and count before TRL attempts to build a reward tensor.

Reward ensembles — list several rewards, comma-separated, and they combine (GRPO only). This also unlocks the rm_ensemble reward-hack detector, which needs ≥ 2 rewards:

training:
  reward_fn: "accuracy,format"   # both are scored every step

LLM-judge ranking reward (reward_fn: judge_rank)

For tasks without a checkable answer (writing, support replies, agent turns), judge_rank asks an LLM judge to compare the completions of each prompt with each other. GRPO normalises rewards within a prompt's group, so only the order inside a group matters: one judge call sees all num_generations completions of a prompt together and scores each from 0 to 1 (relative scoring, as in OpenPipe's RULER). Where an answer can be checked, a rule-based reward (accuracy, verifiable, a .py file) is cheaper and cannot be talked into a score. judge_rank also runs in an ensemble, e.g. reward_fn: "format,judge_rank".

task: grpo
training:
  reward_fn: judge_rank
  grpo_judge: "ollama://llama3.1"        # or https://host/model, http://localhost:8000/model
  grpo_judge_max_calls: 20000            # required: the run is refused if it could make more
  # grpo_judge_rubric: ./rubric.yaml     # optional; only each criterion's name + description
  # grpo_judge_max_tokens: 1024          # judge reply budget (default 1024)
  # grpo_judge_max_failure_rate: 0.2     # stop past this share of failed groups (default 0.2)

grpo_judge takes the same URL form as online_dpo_judge and is checked at load: an unsupported scheme, plain http:// to a host other than localhost, or a private IP literal is refused before anything downloads. A rubric (the soup eval judge format) contributes the name and description of each criterion; a ranking scores from 0 to 1, so weight and scale are not used. Without a rubric the judge uses Soup's default criteria (helpfulness, accuracy, safety).

What it costs. One judge call per group, i.e. per training prompt per epoch, plus one per held-out prompt at each evaluation (training.eval_steps). A reply that does not parse is asked for once more, so a group takes at most 2 calls. Inside each counted call, a request the judge answers with 429 or a 5xx, or that times out or loses its connection, is retried up to twice, so the judge host can see up to three requests per counted call. The calls are made one after another while the step waits, so a step takes at least one judge round trip per prompt in its generation batch. After the dataset loads and before the model does, Soup counts (train rows x epochs + held-out rows x evaluations) x 2 and refuses the run when that exceeds grpo_judge_max_calls, stating the numbers and the value to raise it to. The number of evaluations is counted at the smallest batch the run can train at, so under batch_size: auto it is an upper bound. rollout_backend is refused with judge_rank, because it replaces the prompts after they were counted.

What the judge host receives. The whole conversation of each prompt (system prompt, earlier turns and the last user turn) and every completion of the group. Only point grpo_judge at a host you would send your training data to. The conversation and each completion are enclosed between BEGIN/END lines that carry a code drawn for that call, which none of them contains, and the judge is told to ignore instructions inside them. The completions are shown in a fresh order on every call (seeded from training.seed), so a judge that favours a position does not keep favouring the same completion.

The judge's answer must be one JSON object with exactly one key per response number, "1" to "N", each a number from 0 to 1 (one ```json fence around it is accepted). A duplicate, missing or extra key, NaN or Infinity, a boolean, a string or a number outside [0, 1] makes the reply unusable, and it is asked for once more. Soup never shortens a completion: a prompt that does not fit the judge's context fails its group when the server refuses it (OpenAI-compatible servers answer with a 4xx). Some servers, Ollama among them, cut an over-long prompt instead of refusing it, so give the judge model a context that holds a whole group.

Failures are visible. A group with no usable answer scores None: TRL leaves it out of rewards/judge_rank/mean (its summed reward counts it as 0), and the group gets no advantage. The share of such training groups is logged as judge/failed_group_rate, and the share of answered groups whose completions all got the same score (no advantage either) as judge/equal_score_group_rate. Once at least 20 training groups have been judged, training stops with an error naming the judge when more than grpo_judge_max_failure_rate of them failed; evaluation groups are counted separately and never stop the run. A shorter run in which the judge scored no training group at all fails at its end instead of saving an adapter. A judge that cannot be reached at all (3 requests in a row with their retries spent) ends the run at once. The counters are per process, so judge_rank refuses a run with more than one process (WORLD_SIZE above 1).

Reward hacking. The policy is trained to please the judge, not to be right, and it can learn what a particular judge rewards: length, a confident tone, flattery, or text addressed to the judge. Read samples while it trains, keep a held-out check with a different judge or a person, and consider the reward-hacking detectors under Loop Hardening above.

Synthesize a verifier from your data (soup reward synth)

Don't hand-write a verifier — generate one from reference (gold) outputs. Soup infers a deterministic verifier (numeric / JSON-schema / regex / tool-call), writes a readable, editable .py, and refuses to emit one that can't tell your references from auto-generated bad answers (the mandatory calibration report). The emitted file is a normal reward_fn: reward.py.

# infer + calibrate + emit (exit 0 kept, 2 refused, 1 error)
soup reward synth references.jsonl -o reward.py --output-report calib.json

# preview the induced spec without writing anything
soup reward synth references.jsonl --plan-only

# force a family instead of auto-detecting
soup reward synth answers.jsonl -o reward.py --kind numeric --tolerance 1e-6

References are a JSONL where each row's gold answer is in an answer field (override with --field) or the last assistant turn of a messages list. --min-discrimination sets how strongly the verifier must separate references from perturbed negatives before it's emitted. v1 is deterministic families only — a \boxed{}/#### marker helps the numeric verifier, and completions are prompted to mark their answer (standard RLVR practice). The numeric verifier reads both the gold and the completion with the parser the built-in math / accuracy rewards use (soup_cli.utils.final_answer), so #### 1,000, \boxed {42} and a 6*7=42\n#### 42 gold score the same under both; references written 1,000 count as numeric. The calibration report names every reference the emitted verifier rejects (rejected_references in --output-report).

Changed: a reward.py synthesized before this version pulled the last number with a local regex; ones synthesized now read both sides with the shared answer parser, so 1,000-style golds and The answer is … completions score where they scored 0 before — regenerate old verifiers before comparing runs. Regeneration also stops paying some completions the old regex paid: a #### or \boxed{} answer carrying units, % or a non-dollar currency symbol or code (#### 42 apples, \boxed{42\%}, #### €42, #### 42 USD), a hedged or justified answer phrase (The answer is either 41 or 42, The answer is 42 because 6*7=42), and a completion whose last \boxed{} is wrong after a right first one now score 0 — matching the built-ins. A bare 42% or The answer is 42%. still pays, as under the built-ins; only marker-delimited answers refuse suffixes. The emitted file imports soup_cli, so it runs where Soup is installed rather than being fully self-contained.

Stress-test a verifier for gameability (soup reward stress)

A verifier that passes calibration still might pay out for junk. soup reward stress feeds the verifier deterministic degenerate completions — empty, length-padded, repeated, and sentinel-spam — scored against your real gold answers, and flags any it accepts. It's the adversarial counterpart to synth: calibration proves the verifier tells references from friendly bad answers; stress asks whether a reward-hacking model could game it.

# probe a synthesized verifier (or any reward .py) — exit 0 robust, 2 gameable, 1 error
soup reward stress reward.py --references golds.jsonl --output-report stress.json

# probe a builtin verifier instead of a .py file
soup reward stress verifiable --verifiable-domain math --references golds.jsonl

# JSON-schema references may be stored as objects in a `schema` field
soup reward stress verifiable --verifiable-domain json_schema \
    --references schemas.jsonl --field schema

# tune the attack set / accept threshold / gameability tolerance
soup reward stress reward.py --references golds.jsonl \
    --attacks empty,length,repetition,sentinel,wrapped_junk,answer_spray \
    --sentinel GOLD --threshold 0.5 --max-gameable 0.0

The report shows a per-attack accept-rate and an overall verdict. A gold-requiring verifier probed with no --references is a hard error (it can't be measured), never a false "robust". Because each attack family evaluates multiple distinct variants across sampled references (up to 23 batched verifier invocations, or 4,600 scored completions at the 200-gold cap), slow or model-based verifiers will take proportionally longer than simple string checks. Probing a .py executes its module code, like any custom reward — only stress files you trust. For the builtin json_schema domain, references are forwarded as schema= metadata; JSON objects selected by --field are decoded before the verifier scores them.

Current limitation: the built-in attack families emit plain text that is not valid JSON, so json_schema rejects them during parsing before schema-specific constraints are evaluated. The result therefore does not yet distinguish a strict schema from a permissive one. Its reference_accept value is also not a meaningful self-acceptance control for this domain because it scores the schema document as though it were an instance of itself (and is normally 0%).

Verifiable Rewards (RLVR)

Use reward_fn: verifiable with a verifiable_domain for deterministic, math-checkable rewards — no judge model, no heuristics. Great for GRPO on math, code, or structured-output tasks.

training:
  reward_fn: verifiable
  verifiable_domain: math          # or: code, json_schema
  num_generations: 4

Three built-in domains:

Domain What it checks
math Reads the final answer of the completion and of the gold with the shared parser described under "Built-in reward functions" (####, \boxed{}, The answer is / Answer:, else the completion's last number). A numeric gold scores 1.0 within 1e-4 and 0.6 within 1e-2 (compared as exact decimals); any other gold scores 1.0 on a normalised text match. Nothing is evaluated: no eval() on user output
code Executes generated Python with a 5s timeout, 512 MB RLIMIT on POSIX, python -I -S, socket patch, ephemeral cwd. Output capped at 10KB. Warning panel on first use
json_schema Validates output against a JSON Schema provided per-example in the dataset

Note: code domain runs untrusted generations. Soup sandboxes aggressively but never trust it for production-grade isolation — run in a VM or container for public data.

Tool-Calling Fine-Tuning

Train models to emit structured function calls (OpenAI-style tool_calls with JSON arguments).

base: meta-llama/Llama-3.1-8B-Instruct
task: sft

data:
  train: ./data/tool_calls.jsonl
  format: tool-calling

training:
  epochs: 3
  lr: 2e-5
  quantization: 4bit

Tool-calling data format:

{"messages": [
  {"role": "user", "content": "What's the weather in Paris?"},
  {"role": "assistant", "tool_calls": [
    {"id": "c1", "type": "function",
     "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}
  ]}
]}

Add a top-level tools list (OpenAI function schemas) to put the schemas in front of the model as a system turn; with format: auto, a row with messages and tools is detected as tool-calling. Multi-turn trajectories keep their order: each assistant turn keeps its tool_calls and each tool turn its tool_call_id. The older shape, with the calls in a top-level tool_calls list, still loads: those calls become one assistant turn after messages.

Arguments are parsed as JSON only — never eval(). soup eval custom can score tool-call accuracy (function name + argument JSON equality).

soup init --template tool-calling

PPO / Full RLHF Pipeline

Train models with the full RLHF pipeline: SFT warmup → Reward Model → PPO alignment.

# Create an RLHF config
soup init --template rlhf

Step 1: SFT warmup — fine-tune a base model on your data:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
  train: ./data/train.jsonl
  format: alpaca
output: ./output_sft

Step 2: Train reward model — learn preferences from human feedback:

base: meta-llama/Llama-3.1-8B-Instruct
task: reward_model
data:
  train: ./data/preferences.jsonl
  format: dpo
output: ./output_rm

Step 3: PPO alignment — optimize the policy using the reward model:

base: meta-llama/Llama-3.1-8B-Instruct
task: ppo
data:
  train: ./data/prompts.jsonl
  format: chatml
training:
  reward_model: ./output_rm
  epochs: 1
  ppo_epochs: 4
  ppo_clip_ratio: 0.2
  ppo_kl_penalty: 0.05
  lora:
    r: 64
    alpha: 16
  quantization: 4bit
output: ./output_ppo

epochs controls complete passes over the training dataset. ppo_epochs controls optimization passes within each PPO update. Soup forwards both values, plus ppo_kl_penalty, to the active TRL PPOConfig names and prints the effective schedule during setup. data.max_length sets PPO's response length (moving the default from TRL's 53 to data.max_length), --deepspeed is forwarded, and save_total_limit=3 keeps up to 3 checkpoints.

One PPO rollout batch is batch_size x gradient_accumulation_steps prompts on every process, and TRL drops a partial batch, so the train set needs at least batch_size x gradient_accumulation_steps x the number of processes rows. A smaller one would never reach a step, so soup train refuses it before loading any model and names the row count and both settings.

PPO supports two reward sources:

  • Reward model (reward_model): pre-trained reward model (from step 2)
  • Reward function (reward_fn): callable function (same as GRPO — accuracy, format, or custom .py)

KTO Training (Unpaired Preferences)

Train with unpaired preference data — no need for chosen+rejected pairs:

base: meta-llama/Llama-3.1-8B-Instruct
task: kto

data:
  train: ./data/kto_train.jsonl
  format: kto

training:
  epochs: 3
  kto_beta: 0.1
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

Batch size. KTO needs a per-device batch_size of at least 2: TRL's KL term is degenerate at batch 1, so batch_size: 1 is refused when the config is loaded. batch_size: auto never resolves below 2, and soup local-rl train --train-method kto writes batch_size: 2. Raise gradient_accumulation_steps for a larger effective batch.

KTO data format:

{"prompt": "What is 2+2?", "completion": "4", "label": true}
{"prompt": "What is 2+2?", "completion": "Fish", "label": false}

On Apple Silicon, backend: mlx runs KTO natively with the same objective; see KTO on MLX.

ORPO Training (No Reference Model)

ORPO combines SFT and alignment in one step — no reference model needed:

base: meta-llama/Llama-3.1-8B-Instruct
task: orpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  epochs: 3
  orpo_beta: 0.1
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

On Apple Silicon, backend: mlx runs ORPO natively with the same objective; see ORPO on MLX.

SimPO Training (Simple Preference)

SimPO uses length-normalized log probabilities as implicit rewards — reference-free:

base: meta-llama/Llama-3.1-8B-Instruct
task: simpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  epochs: 3
  simpo_gamma: 0.5
  cpo_alpha: 1.0
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

Long prompts are capped as in DPO and ORPO. trl's CPO trainer cuts only the answers, so after it tokenizes, soup train cuts the prompt to data.max_length // 2 tokens, keeping its end, and the answers to the room left, for the train and validation splits. Every sequence is then at most data.max_length. The cut goes by the prompt alone: a prompt longer than data.max_length // 2 tokens loses its start even when prompt and answers would fit in data.max_length. A row whose prompt is at most data.max_length // 2 tokens and which fits is unchanged.

Rows whose completion trl truncates away are refused. trl's CPO trainer truncates each answer to data.max_length minus the longer answer's length, so a long chosen beside a short rejected leaves the short side with zero trainable tokens — SimPO's length-normalised log-probability is then 0/0, every adapter tensor trains to NaN, and transformers' nan-inf filter reports the loss as 0.0. soup train stops before the first step, naming how many rows and which, with the data.max_length involved. Raise data.max_length, balance the pair, or drop the row.

On Apple Silicon, backend: mlx runs SimPO natively with the same objective; see SimPO on MLX.

IPO Training (Regularized Preference)

IPO is a theoretically grounded DPO variant with stronger regularization:

base: meta-llama/Llama-3.1-8B-Instruct
task: ipo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  epochs: 3
  ipo_tau: 0.1
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

RAFT — Retrieval-Augmented Fine-Tuning

When you need a model to cite the document it's reading instead of hallucinating, RAFT (Stanford 2024) is the canonical recipe. Each training row carries a query, a golden document, a list of distractor documents, and the answer — the model learns to attend to the relevant doc while ignoring the noise.

# soup.yaml
data:
  train: ./data/raft.jsonl
  format: raft

training:
  citation_faithful: true        # enable citation precision/recall scoring
  citation_style: bracket        # cite as [doc-1] inline

training.citation_recall_threshold is validated but nothing gates a save or a run on it. Setting it warns in v0.76 and is refused as of v0.77 (#761).

# RAFT JSONL row shape
{"query": "When was Python released?", "golden_doc": "Python was released in 1991 by Guido van Rossum.", "distractor_docs": ["Ruby was released in 1995.", "Java was released in 1995."], "answer": "1991 [doc-1]"}
# Ready-made 8B Llama recipe
soup recipes show raft-llama3-8b
soup recipes use raft-llama3-8b

Citation scoring is exposed as a pure kernel for the eval gate:

from soup_cli.utils.citation_faithful import score_citations

score = score_citations(
    predicted="The answer is 1991 [doc-1].",
    expected_ids=("doc-1",),
)
# CitationScore(precision=1.0, recall=1.0, f1=1.0, predicted_count=1, expected_count=1)

Citation-faithful FT requires task: sft + data.format: raft — misconfigured runs fail at config load with a named-field message. The pretrain trainer does not apply citation span weights and refuses training.citation_faithful: true.

Under the hood, a format: raft run trains answer-only: each row is composed into a prompt (golden + distractor docs, shuffled deterministically by data.raft_shuffle_seed, each labelled [doc-N]) followed by the answer; the prompt span is masked out of the loss and — when citation_faithful: true — the bracketed [doc-id] spans in the answer get a boosted per-token loss weight. Rows whose prompt fills max_length (answer fully truncated) are dropped with a warning rather than silently shrinking the dataset.

By default the document order is fixed for the whole run. Set data.raft_epoch_shuffle: true to re-permute the golden + distractor documents each epoch (a per-epoch salt folded into the shuffle seed) so the model can't latch onto a fixed citation slot — useful for multi-epoch runs. epoch=0 reproduces the legacy single-permutation order exactly, so enabling it never changes the first epoch. (v0.71.17)

Score a trained model's citations from the CLI:

# {predicted, expected_ids} rows, OR RAFT rows scored against their own golden [doc-N]
soup eval citation preds.jsonl --style bracket
# RAFT rows: pass the train-time shuffle seed so the golden id lines up
soup eval citation raft.jsonl --shuffle-seed 0 --output citation.json

soup diagnose also gains a citation failure mode that flags a model that stopped citing the supporting document.

RA-DIT — Retrieval-Augmented Dual Instruction Tuning

RA-DIT (Meta 2023) is the two-stage version of RAFT: first train a sentence-transformer retriever (contrastive), then fine-tune the generator on the RAFT-style rows. Two recipes ship paired:

# Stage 1 — train the retriever (uses Soup's v0.16 embedding trainer)
soup recipes use ra-dit-retriever
soup train

# Stage 2 — train the generator on RAFT data, pointing at the retriever
soup recipes use ra-dit-llama3-8b
soup train

The schema enforces stage-task pairing — ra_dit_stage: retriever requires task: embedding; ra_dit_stage: generator requires task: sft. A misconfigured recipe fails at config load with a named-field message.

Run both stages in one command with soup ra-dit:

soup ra-dit --retriever-config retriever.yaml --generator-config generator.yaml
# preview the plan + the resolved retriever link without training:
soup ra-dit -r retriever.yaml -g generator.yaml --plan-only

It trains the retriever, then records that trained retriever as the generator's paired retriever (writing its output dir into the generator's training.ra_dit_retriever_model) and trains the generator RAFT-style. The recorded retriever is the one used at deploy/serve time — stage-2 does not fuse the retriever weights. A plain soup train of a generator-stage config with no retriever model set auto-links the most-recent RA-DIT retriever run from the Registry; pass --retriever-model <m> to override.

Both configs must write to different, non-nested output directories: soup ra-dit refuses a pair whose effective outputs (output plus experiment_name, exactly as the trainers build them) are equal or nested, so give each stage its own output, or give both stages different experiment_name values. The two shipped recipes both declare output: ./output, so setting an experiment_name on only one stage is not enough.

Curriculum-Aware Training (BETA)

Layer dynamic re-weighting on top of the static curriculum bucketer. Every N steps the trainer aggregates per-sample loss + grad-norm into a per-bucket uncertainty signal, runs it through a softmax (temperature-controlled) with floor (water-filling so no bucket drops below curriculum_dynamic_floor), and re-weights the sampler. Empty buckets fall back to the median of populated buckets; degenerate inputs return uniform.

training:
  curriculum: true                          # static bucketer (v0.23.0)
  curriculum_buckets: 4
  curriculum_metric: perplexity             # length (default) | loss | perplexity
  curriculum_dynamic: true                  # NEW — dynamic re-weighting
  curriculum_dynamic_recompute_steps: 50    # refresh every 50 global steps
  curriculum_dynamic_floor: 0.05            # min weight per bucket
  curriculum_dynamic_temperature: 1.0       # softmax temp on uncertainty

Bucketing by difficulty percentile (v0.71.5). When curriculum_metric is loss or perplexity, the dynamic callback assigns each step's sample to a bucket by its rank within a rolling 512-step window of the difficulty signal (perplexity = exp(min(loss, 50))), instead of the round-robin fallback used for length. This keeps the buckets calibrated to the live loss distribution rather than a static length sort. length (the default) keeps the round-robin assignment.

Visualise the recorded bucket-weight evolution with soup runs curriculum-curve <run_id>.

DDP / grad-accum safety: multi-rank launches must wire an all_reduce hook on per-bucket stats (a cross-validator rejects un-coordinated multi-rank runs upfront). Multi-trainer expansion beyond sft / pretrain is tracked for v0.48.1.

TTS Fine-Tuning (task='tts', BETA, live in v0.71.20)

Live as of v0.71.20 (lifted from the v0.52.0 schema stub). The codec-string families (orpheus, llasa, spark, oute) train with next-token cross-entropy over interleaved [text][audio-codec-token] chat sequences, so TTSTrainerWrapper reuses the SFT model/tokenizer/LoRA/CE machinery and adds per-family emotion templating plus codec-token registration. sesame_csm is different: current CSM training uses text plus 32 Mimi codebooks as parallel multimodal frames. Soup routes it to a dedicated native trainer (see below), and continues to refuse pre-encoded CSM codec-string chat.

There are two workflows:

Pre-encoded chat (live for codec-string families). Run the family's audio codec offline so the assistant turn already contains the discrete codec-token string, then train with data.format: chatml. This is plain cross-entropy and runs on any GPU (validated end-to-end on SmolLM2-135M-Instruct). This workflow does not turn Sesame CSM's parallel Mimi codebooks into a valid CSM training example.

base: HuggingFaceTB/SmolLM2-135M-Instruct   # or canopylabs/orpheus-3b-0.1-ft
task: tts
modality: audio_out
data:
  train: ./data/tts_pre_encoded.jsonl   # assistant turns carry codec tokens
  format: chatml
  new_special_tokens: ["<|codec_0|>", "<|codec_1|>"]   # your codec vocab
training:
  tts_family: orpheus
  tts_emotion: neutral   # Orpheus + Oute only
  lora:
    r: 16
    alpha: 32

Operator-supplied data.new_special_tokens are registered (deduplicated, only tokens not already in the vocab) and the embedding matrix is resized through the (possibly PEFT-wrapped) model so the codec-token ids have rows. That happens on backend: transformers only: the unsloth setup has no step that adds tokens, so backend: unsloth with new_special_tokens is refused at load (#1358). Orpheus + Oute support emotion conditioning via training.tts_emotion from a per-family allowlist (Orpheus: neutral / happy / sad / angry / excited / calm / whisper / laugh; Oute: neutral / happy / sad / angry / calm / excited) — the wrapper prepends the family's emotion control string to the first user turn.

Live-codec (hardware/dependency-gated). Setting data.format: audio asks the trainer to encode raw audio at train time. Orpheus and Llasa are live on the codec-string path: Orpheus uses pip install snac at 24 kHz; Llasa uses Soup's [audio] extra (torchaudio + soundfile). Torchaudio must match the installed Torch release — recent torchaudio metadata may not make pip enforce that pairing — then Soup resamples to 16 kHz and calls the Transformers-native HKUSTAudio/xcodec2-hf codec, and renders the resulting ids as <|s_ID|> between Llasa's speech-generation boundary tokens. Audio remains duration/byte-capped and is read through an O_NOFOLLOW fd. Spark and Oute raw-audio live encoding now fails closed: Spark-TTS has no installable sparktts package and its official environment pins Torch/Transformers below Soup's supported stack; current outetts pins Transformers 4.52.3 and Oute preparation also needs transcript/word alignment. For those two families, pre-encode in the upstream environment and train the resulting codec-token chat with data.format: chatml. Sesame CSM uses the native multimodal path below, rather than this codec-string adapter.

Three ready-made codec-string recipes ship: orpheus-tts-sft, llasa-tts, oute-tts — copy with soup recipes use <name>. Cross-validators reject the mlx backend, modality != audio_out, and emotion tags outside the per-family allowlist.

Native Sesame CSM

training.tts_family: sesame_csm selects CsmForConditionalGeneration and CsmProcessor from Transformers, with no separate Moshi installation. The processor builds text conditioning and audio-frame labels; the model encodes audio with its frozen Mimi codec and optimizes the sum of the backbone loss (first codebook) and depth-decoder loss (the other 31 codebooks). Text and padding are excluded from the audio targets. Mimi stays in evaluation mode during training. This follows the Transformers CSM training contract.

Install pip install 'soup-cli[train,audio]', then prepare one local clip and its exact transcript per JSONL row. role is the native speaker id "0" or "1":

{"audio": "hello.wav", "messages": [{"role": "0", "content": "Hello, how are you?"}]}
base: sesame/csm-1b
backend: transformers
task: tts
modality: audio_out
output: ./output/csm
data:
  train: ./data/speech.jsonl
  format: audio
  audio_dir: ./data/audio
  max_length: 2048
training:
  tts_family: sesame_csm
  quantization: none
  epochs: 3
  batch_size: 1
  gradient_accumulation_steps: 4
  lr: 0.00002
  lora:
    r: 16
    alpha: 32

Run soup train -c csm.yaml. The checkpoint may require Hugging Face access; obtain that access and authenticate before running. Local Transformers-format CSM checkpoints are also accepted. Audio is loaded through Soup's local-file safety checks and resampled to mono 24 kHz. Sequences longer than max_length are refused rather than truncating audio markers and breaking codec alignment.

The initial native path supports one device, single-clip rows, standard LoRA on the backbone and depth decoder (excluding Mimi), or full fine-tuning with lora.r: 0 (Mimi remains frozen). batch_size: auto conservatively selects 1. Validation loss, boolean gradient checkpointing, seeds, checkpoint resume and processor saving are supported. Multi-turn rows, quantization, packing, distributed training and additional SFT optimization knobs are refused explicitly. The MLX backend remains unsupported for TTS. On Apple Silicon, contributor @Amix29 ran test_native_mps_training_step with PyTorch/MPS, torch 2.13.0 and transformers 5.16.1: one LoRA training step on a tiny randomly initialized CSM/Mimi model, with finite loss and a frozen codec. This is the evidence for the MPS bf16 opt-in. Full fine-tuning, gradient checkpointing, evaluation, resume and the pretrained sesame/csm-1b checkpoint were not run on MPS.

Offline tests construct a tiny real CSM model with all 32 Mimi codebooks and exercise forward/backward, training, evaluation, save/reload and resume. These tests validate the integration; they do not establish speech quality or a full training run on the pretrained sesame/csm-1b checkpoint.

Classifier / Reranker / Cross-Encoder Training (BETA, v0.52.0)

Three new task types build on the existing embedding trainer: task: classifier (single-label or multi-label sequence classification), task: reranker (pointwise retrieval scoring), task: cross_encoder (paired-input scoring). Schema-only; live trainer wrapper in v0.52.1.

base: BAAI/bge-base-en-v1.5
task: classifier
data:
  train: ./data/classification.jsonl
training:
  num_labels: 3
  classifier_kind: single_label
  label_names: [negative, neutral, positive]

num_labels is bounded [1, 1024] with explicit bool-before-int rejection; label_names (optional) must be unique, ≤128 chars each, and match num_labels in length when set.

Knowledge Distillation (BETA, v0.52.0)

New task: distill with training.teacher_model (HF id or local path), training.distill_divergence (kl / forward_kl / reverse_kl / js — kl canonicalises to forward_kl), and training.distill_temperature (bounded [0.05, 100.0], finite-only). Schema-only; live loop in v0.52.1.

base: meta-llama/Llama-3.2-1B
task: distill
data:
  train: ./data/distill.jsonl
training:
  teacher_model: meta-llama/Llama-3.1-8B
  distill_divergence: forward_kl
  distill_temperature: 2.0

The cross-validator rejects task='distill' without teacher_model, and rejects teacher_model / distill_* fields when task is anything other than distill. It also refuses task='distill' on backend: mlx and backend: unsloth at config load; distillation runs on backend: transformers only. Both token-level and sequence-level distillation require modality: text; vision and audio are refused at config load because the distill trainer has no multimodal input path.

EBFT + GDPO (BETA, v0.52.0)

Both fields are refused at config load. training.gdpo_variant silently did nothing on supported TRL versions because the DPO loss hook no longer exists (#1309); remove it and use plain DPO for standard, or SimPO as the nearest (not identical) objective for length_normalized. margin has no equivalent. Energy-Based Fine-Tuning (training.ebft_variant + training.ebft_temperature) remains refused (#1230): its term had no causal shift, so it rewarded copying the input, and shifted it would duplicate the cross-entropy. See EBFT / GDPO Loss Variants.

gpt-oss reasoning_effort + train_on_eot (v0.52.0)

training.reasoning_effort: low | medium | high injects a system-prefix token at training time for gpt-oss models; training.train_on_eot: true includes explicit EOT/EOS control tokens in the SFT loss (axolotl train_on_eot). Both are applied by the SFT formatter, which runs on sft, tts and distill, so those are the tasks they load on. Setting either on any other task (pretrain, DPO, GRPO, PPO, classifier, reranker, cross_encoder) fails at config load. Live formatter wiring in v0.52.1.

MoLE — Per-Token Adapter Routing (task='moe_lora_routing')

Train a small gating network that routes each token to a weighted blend of N frozen task LoRAs (Mixture of LoRA Experts, Wu et al. 2024). The base model and every task adapter stay frozen — only the router learns which adapter(s) each token should use.

base: HuggingFaceTB/SmolLM2-135M
task: moe_lora_routing
modality: text
backend: transformers

data:
  train: ./data/chat.jsonl
  max_length: 512

training:
  mole_task_adapters:        # 2-64 LoRA adapter paths (HF ids or local dirs)
    - ./adapters/math
    - ./adapters/code
    - ./adapters/chat
  mole_top_k: 2              # 1 <= top_k <= len(mole_task_adapters)
  mole_temperature: 1.0      # [1e-6, 100.0]
  epochs: 1

The gate is the only trainable parameter; it is saved as mole_gate.pt alongside the run, saved into every checkpoint-N, and restored by --resume. It trains as an fp32 master weight on every device, so its gradient and AdamW moments are fp32 too, even where the frozen base loads in bf16 (on CUDA), and mole_gate.pt is saved in fp32: a Linear(hidden, N) of 4 x hidden x N bytes, about 7 KB for the example above and 2 MiB at hidden 8192 with 64 adapters. A bf16 gate with no fp32 copy would round most AdamW steps away at the default lr (#1266). compute_loss runs N+1 forwards per step (base + each adapter under torch.no_grad(), blended by the per-token gate weights) so step time scales with the number of task adapters. To serve the trained router, see soup serve --mole in Serving and Export. (v0.71.12)

Architecture Knobs — Mixture-of-Depths, LLaMA Pro, LongLoRA

Two architecture transforms that were schema-only are now live on Llama / Qwen / Mistral: Mixture-of-Depths on SFT and pretrain, LLaMA Pro on SFT only (#1813). Both apply at trainer setup.

training.use_mod: true installs a per-layer top-k token router (mod_capacity_factor, default 0.125). It is applied only on the routes below. Every other active request is refused at config load, naming training.use_mod, the task, the settings that leave the applying path, and what to change or use_mod: false (#1752).

Config use_mod: true
task: sft, backend: transformers, modality: text, no layer streaming, lora.r >= 1, LISA off, no unfrozen_parameters applied
task: tts, modality: audio_out, backend: transformers, lora.r >= 1 applied (tts trains through that SFT LoRA arm)
task: pretrain, backend: transformers, LoRA or LISA applied
task: sft with modality: vision or audio, stream_layers: true, LISA, nonempty unfrozen_parameters, or lora.r: 0 refused
task: sft or tts with backend: unsloth, or task: sft with backend: mlx refused
task: tts with lora.r: 0 refused
task: pretrain with backend: unsloth refused
any other task refused

Pretrain follows its own dispatch. Modality and streaming are not MoD refusals there, and a pretrain LISA run still installs the router. A config that is already invalid — backend: mlx on tts or pretrain, stream_layers outside its task list, or lora.r: 0 on pretrain without LISA — keeps that earlier error instead of this one.

training:
  # Mixture-of-Depths (arXiv 2404.02258): route only the top-k tokens through each
  # block. capacity_factor is the fraction of tokens that get the residual update.
  use_mod: true
  mod_capacity_factor: 0.125
training:
  # LLaMA Pro: append zero-initialised identity decoder blocks and train only the new
  # ones (freeze_trainable_layers must equal expand_layers and freezes the
  # originals). Needs quantization: none and lora.r: 0 (task: sft).
  expand_layers: 4
  freeze_trainable_layers: 4
  quantization: none
  lora:
    r: 0

use_mod attaches AFTER get_peft_model on the SFT/TTS LoRA arm, and after the LISA/LoRA split on pretrain, so the new routers are trainable. expand_layers runs BEFORE it, and get_peft_model would freeze the appended blocks again, so a LoRA rank is refused with expand_layers and the blocks train with lora.r: 0 (#1326). Unsupported architectures warn + skip. Pick one of MoD / LLaMA Pro per run. (v0.71.12; use_mod route gate #1752)

LongLoRA S² (use_longlora: true) is refused at config load (#1240). Its override rolled the query/key projections of half the heads with wrap-around under full causal attention, so earlier positions saw the last tokens of the sequence, and it applied no grouped attention. To extend the context, use rope_scaling_type with plain LoRA; see Long Context.

Spectrum — Targeted Training on Layer SNR (soup spectrum scan, v0.71.23)

Spectrum (arXiv:2406.06623) fine-tunes only the layers with the most signal. soup spectrum scan streams a model's .safetensors shards one tensor at a time — there is no model load, so it runs on a CPU box even for very large models — and computes a singular-value signal-to-noise ratio per weight matrix with a Marchenko-Pastur noise threshold. It ranks the layers within each module-type group and prints the top --top-percent as a ready-to-paste config block:

soup spectrum scan --model HuggingFaceTB/SmolLM2-135M --top-percent 25 --modules mlp,attn -o patch.yaml
# patch.yaml — paste into your soup.yaml
training:
  unfrozen_parameters:
  - model.layers.0.mlp.down_proj
  - model.layers.29.self_attn.v_proj
  # ...

Then train with the patch — the SFT trainer freezes every parameter and unfreezes only the matched set (full fine-tuning, LoRA off):

base: HuggingFaceTB/SmolLM2-135M
task: sft
training:
  quantization: none        # Spectrum trains float weights — quantization off
  unfrozen_parameters:
  - model.layers.0.mlp.down_proj
  - model.layers.29.self_attn.v_proj

unfrozen_parameters entries are regex patterns matched against parameter names. It requires task: sft, backend: transformers, modality: text, and quantization: none, and is mutually exclusive with LoRA features (use_dora / use_vera / moe_lora / relora_steps / …) and the other freezing knobs (freeze_layers / freeze_ratio / train_router_only / expand_layers) — a conflicting combo is rejected loudly at config load. Scans cache under ~/.soup/spectrum/ (override with SOUP_SPECTRUM_CACHE_DIR); --no-cache skips it. --modules mlp,attn (vs the all default) is recommended for very large models — it skips the giant embedding/lm_head matrices. The SNR kernel is pure-numpy and transpose-invariant, so GPT-2 Conv1D weights score the same as Linear weights. (v0.71.23)

Training source identity

soup train captures training_data_sha256 before loading or transforming a supported single local raw text-training file. The digest is SHA-256 of the complete file bytes, including any BOM and original newline encoding. It records source bytes at startup; it does not prove which transformed examples, external assets or later streaming reads trained the model. The source is not frozen.

Capture uses 1 MiB reads, a 256 MiB byte limit and a 5-second time budget checked between filesystem operations. An operation already blocked on I/O cannot be interrupted by that budget. Larger, unreadable, changing or incomplete files are recorded as unknown (NULL), never as the hash of a prefix. Descriptor and path metadata detect accidental replacement/modification during capture; this is not an adversarial filesystem snapshot.

Mixed/interleaved/replay, remote, streaming, tokenized and directory inputs remain unknown. The initial scope is text input through data.train; TTS, unlearning and non-text modalities remain unknown. No unknown component is silently omitted to manufacture an identity for a composite input. Dry runs do not capture a digest.

The recorded value is retained in soup_run.json beside the final artifact and copied by soup registry push --run-id without reopening today's source. Explicit --data still describes the file registered now: its hash takes precedence for data_hash and the entry fingerprint, while training_data_sha256 keeps the historical startup digest. Copied artifacts retain that digest without the original dataset. Existing runs stay unknown; no historical identity is inferred from today's files. Sweep and other training entry points do not gain capture in this change.