Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Changelog
All notable changes to Soup CLI are documented here.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Detailed, per-release notes for every published version live on the GitHub Releases page. This file tracks unreleased changes and links out for historical detail rather than reproducing 70+ versions of notes.
Unreleased
0.76.0 - 2026-10-12
Added
Added
soup data cleanfor automated dataset sanitization and repair (#718 by @Deadpool2000). Provides deterministic, non-destructive cleaning of fine-tuning datasets across all supported formats (chatml,alpaca,sharegpt,dpo,kto,tool-calling). Runs safe character sanitization (C0 controls, zero-width spaces\u200b-\u200d, BOM) and empty assistant turn pruning by default, while placing heuristic transformations behind explicit opt-in flags (--strip-boilerplate,--repair-code,--repair-json,--prune-echo). Accurately tracks all modifications, strictly preserves unknown metadata keys and custom roles non-destructively, guarantees input file immutability, refuses unsafe in-place or unforced overwrites, and supports dry-run inspection and structured machine-readable JSON output.Token-chunked checkpointed KL distillation (#722 by @kok-o in #743). Added
training.distill_chunk_sizeandtraining.distill_checkpointknobs fortask: distill. Response-only token selection (labels != -100) eliminates unmasked prompt and padding tokens from retained autograd memory, while token chunking and optional non-reentrant activation checkpointing (torch.utils.checkpoint.checkpoint(..., use_reentrant=False)) discard intermediatelog_softmaxand divergence tensors during forward and recompute them in backward, reducing retained autograd tensor bytes and bounding transient peak activations on long sequences and large vocabularies. The divergence kernel computes and accumulates in FP32 precision, shifting baseline bf16 distillation loss by up to1 ULP (0.4%).A test now fails when any typer command is called as a plain function with a parameter left unfilled (#752 by @dchaudhari7177 in #757). Called directly rather than through typer, an unfilled parameter keeps its
typer.Option/typer.Argumentdefault, a truthyOptionInfo. That is howsoup eval autocrashed after saving its results in #752, fixed at its two call sites by #754.tests/test_typer_defaults_do_not_leak.pywalks the wholesoup_clipackage, resolves each call throughfrom X import nameto the command it names, and matches positional arguments to parameters by declaration order, so the next unfilled parameter anywhere fails the suite and the report names only the parameters actually left unfilled. A synthetic command proves the reporter can still report, so the guard cannot be silenced unnoticed.soup adapters auditchecks whether a finished run did what its config asked (#762 by @Srinivasan8888 in #763). The MLX path records its effective settings inadapter_config.json— optimizer, scheduler, warmup and total updates, weight decay, gradient clipping, accumulation, checkpointing and the response-only mask — because each of those was once accepted and silently dropped (#683, #684, #685, #686, #749). Nothing read that record back, so answering "did my run do what I asked?" meant opening two files and comparing by eye.soup adapters audit ./out --config soup.yamlprints a per-setting table of asked-versus-ran, explains each divergence, and exits2when any is found --0on agreement and1on a usage or read error, so a CI gate can tell a verdict from a mistyped path (matchingsoup shipandsoup shrink);--jsonemits the same result for machine consumption. A setting the record cannot speak to is reportedunknownand never as agreeing — a false clean bill is worse than no audit — andunknowndoes not fail the command, since an older adapter simply predates the keys. The motivating case is a warmup that rounds away:warmup_ratio: 0.03over twelve optimizer updates is zero warmup steps, which the trainer warns about once and the adapter records permanently. Response-only masking is audited by its recorded effect (mask_prompt/response_token_mask) rather than by the request the record echoes back, so a plain-text run that asked for masking and silently got none — #683 itself — is reported as a divergence instead of a clean bill. On the transformers path only LoRA shape can be checked, because PEFT writes its ownadapter_config.json, and the command says so rather than implying a clean bill it cannot give.soup trainaccepts--nodes,--node-rank,--master-addr, and--master-portfor multi-node Accelerate launches.--gpusis the per-node count, so eight GPUs on two nodes launches 16 workers. Launch settings are validated before training:--master-addrmust parse as an IPv4/IPv6 literal or an RFC 1123 hostname, so a host with the port glued on (head:29500) is refused rather than exported asMASTER_ADDRon every node. Multi-node NCCL defaults allow InfiniBand and preserve user overrides.--no-reexecand--dry-runshow the command without starting workers. (#40 by @tristangrech in #853)Added ready-made
kimi-k2.5-dpoandkimi-k2.5-grporecipes for moonshotai/Kimi-K2.5 (#275 by @Areen-09 in #872). Completes the task trio for the base model, mirroring the Kimi-K2.6 DPO and GRPO architectures over the ~1T MoE geometry (LoRA r32/a64 for DPO, r16/a32 for GRPO, 4-bit quantization,moe_lora,gradient_checkpointing,max_length: 8192). The recipes are not trained — ~1T MoE requires multi-node DeepSpeed — so hyperparameters are inherited from K2.6 templates rather than tuned for K2.5. Catalog count 167 -> 169.Added ready-made
minimax-m3-dpoandmistral-large-3-dporecipes (#849 by @akkupratap323 in #933). Both mirror their giant-MoE SFT siblings with the catalog's DPO preference-data shape, LoRA r32/a64, 4-bit quantization,moe_lora, gradient checkpointing, and one training epoch. These inherited configurations are intended for multi-node use and have not been validated by training the 428B and 675B base models.soup doctornow reports GPU architecture and precision capability diagnostics (#829 by @Samearth17 in #947). GPU diagnostics show the detected NVIDIA compute capability and architecture, whether the installed Torch build includes native/PTX support for that GPU, and hardware/software support for BF16, FP8, and NVFP4. Unsupported GPU architectures now include a CUDA-enabled PyTorch reinstall hint.Add the
qwen3.5-35b-a3b-grporecipe, completing the Qwen 3.5 35B-A3B SFT, DPO, and GRPO trio. (#847 by @Amix29 in #956)Added optional PEFT LoRA-FA optimizer support (#725 by @webdevsamran in #962). Freezes LoRA A matrices and trains only B matrices via PEFT's create_lorafa_optimizer, reducing adapter activation memory retention and cutting optimizer state size by 50% during training. Supported for sft, pretrain, and embedding tasks on the Transformers backend.
LoRA
rank_patternandalpha_patternare now held by a test on every trainer that builds an adapter (#959 by @dchaudhari7177 in #963). #923 routed all eighteenbuild_lora_configcallers through the shared builder, but ten had a behavioural test.asr,classifier,distill,embedding,ppo,unlearnand layer streaming are now driven on CPU with a tiny model and checked for the configured per-module rank (streaming, which never materialises the base, is checked on theLoraConfigit streams with), and a scan oftrainer/fails when a new caller of the builder has no pattern test.Added ready-made
qwen3.6-35b-a3b-dpoandqwen3.6-35b-a3b-grporecipes for Qwen/Qwen3.6-35B-A3B (#275 / #846 by @Areen-09 in #978). Completes the task trio with the existing SFT sibling over the ~35B MoE geometry (3B active, LoRA r16/a32, 4-bit quantization,moe_lora,moe_aux_loss_coeff: 0.01). DPO adopts theqwen3.5-35b-a3b-dpotemplate (max_length: 4096), and GRPO adopts theqwen3-30b-a3b-reasoning-grpotemplate (max_length: 8192,gradient_checkpointing). Both recipes declare explicitmodality: text. Catalog count 172 -> 174.Added ready-made
qwen3.6-27b-dpoandqwen3.6-27b-grporecipes (#845 by @akkupratap323 in #1015). Both preserve the dense SFT sibling's explicit text modality, LoRA r16/a32, 4-bit quantization, andautobatch size with gradient accumulation 8. DPO uses preference JSONL data withlr: 5e-6; GRPO uses reasoning JSONL data with four generations and the accuracy reward. These inherited 27B configurations were schema- and CLI-validated but were not trained locally. Catalog count 174 -> 176.soup rewindnames the dataset rows a training loss spike came from (#1000 by @Srinivasan8888 in #1018). An SFT run now writes<output>/rewind.jsonl: one line per micro-batch holding the dataset rows in it, each row's mean supervised-token loss and its supervised-token count.soup rewindlists the steps whose loss went non-finite or past twice the median of the previous twenty measured steps, then ranks that step's rows by their share of its summed token loss, so one long malformed row stands out even when its per-token mean is ordinary. On an 8 GB M1 it flagged step 58 of a Qwen2.5-0.5B run and put the poisoned row at 100% of that step's loss. Both backends, controlled bytraining.rewind_log(default on); the recorder switches itself off with one warning rather than record rows it cannot attribute, including on resumed runs and under packing.soup draft measurereports the break-even acceptance rate and the best draft length (#843 by @Srinivasan8888 in #1042). It now also times the draft on its own and reports the latency ratio, the acceptance at which assisted generation stops being slower at the--num-assistant-tokensin use, and the modelled best k, all labelled as a ceiling that assumes independent acceptance and excludes framework overhead. A draft no faster than its target is reported as "no acceptance rate pays".--sweep-k 1,2,3,5,8measures assisted throughput at each draft length and names the measured best k. The docs no longer say a 70% acceptance rate is where speculative decoding pays: the one pair measured at scale was 81% and ran at 0.48x.pytest -m gpuselects the CUDA-dependent tests, andCONTRIBUTING.mdsays how to run and report them (#833 by @Srinivasan8888 in #1050). Fifteen test modules had their owntorch.cuda.is_available()probe, and nine had an identicalrequires_cudaskip, so the GPU subset couldn't be selected and was only ever run by accident.tests/conftest.pynow holds the one probe and skips@pytest.mark.gputests without CUDA;tests/test_issue833_gpu_marker.pyfails on a private probe. Runs on real cards are recorded inbenchmarks/gpu-test-runs.md, seeded with the H100 and RTX 5070 Laptop runs already published.soup infersupports opt-in batched causal-LM inference with--batch-size; the default remains 1 so existing inference behavior is preserved. fp16/bf16 batching may not be bit-exact with batch size 1. (#765 by @Samearth17 in #1083)Arabic and Japanese READMEs (#1100 by @Ercaner1988).
README.ar.mdandREADME.ja.mdare full translations of the currentREADME.md, linked from the language banner and registered inMAINTAINERSbehind the same sync gate asREADME.tr.md. The Arabic tables sit in a right-to-left block, and the 13 Latin identifiers that GitHub's bidi algorithm rendered reversed inside Arabic text (the concept DOI showed aszenodo.21771064/10.5281) are wrapped in<span dir="ltr">, found by running the algorithm over every code span rather than by eye. Spanish, Russian and Chinese are not included until someone can verify them.data.mask_historynow does what it says (#761 by @Srinivasan8888 in #1101). The field was declared, validated and documented with a worked example, and read by no module outside the schema, so a config asking to train on the last turn trained on every turn. Withtrain_on_responses_only: trueit now keeps only the last assistant turn in the loss; earlier assistant turns are masked alongside the user and system turns that path already excludes, so it only ever removes tokens from the loss. It applies totask: sftandtask: distill, the two trainers that build their labels through the shared chat formatter.data.mask_historyrequiresdata.train_on_responses_only: true(#761 by @Srinivasan8888 in #1101). Without it every token trains, including the history the field asks to exclude, so the combination is refused at config load with a message naming both. That also rules outtrain_on_messages_with_train_field, which is already exclusive withtrain_on_responses_only.backend: mlxsays it ignoresdata.mask_history(#761 by @Srinivasan8888 in #1101). MLX SFT builds its own mask and supervises every assistant turn, so the field has no effect there.soup trainnow lists it on its "MLX backend ignores:" line, andsoup doctor --configreports it instead of an all-clear.Added the measured mixed QuEST route as an experimental SFT path (#674 by @Shutaru in #1105).
training.quantization_aware: questreconstructs the confirmed 168-W4 / 161-A4 / seven-A16 route on its exact 24-block Llama topology, calibrates fixed activation clips from the first 32 tokenized training rows, and records the full route in a mandatory sidecar on final and checkpoint saves. Resume and the Soup-aware loader reject metadata drift. The retained result is evaluation-only on one model: this is not strict W4A4, packed INT4, upstream QuEST parity, an efficiency claim, or evidence that training through the route preserves quality.Warn on unconsumed staged config fields with a v0.77 refusal deadline (#808 by @Guts1005 in #1106)
Adapter manifests can now be signed and verified with Sigstore keyless OIDC (#185 by @Amix29 in #1114). Install
soup-cli[sigstore];soup adapters sign --backend sigstorerecords the Fulcio/Rekor bundle, and verification requires an out-of-band certificate identity and OIDC issuer instead of trusting the bundle's own identity.In-toto/SLSA attestations can now use Sigstore keyless OIDC signing (#179 by @Amix29 in #1115).
soup attest emit --sign sigstorewrites a modern Sigstore Bundle into the existing.sigsidecar, and verification requires an out-of-band certificate identity and OIDC issuer; installsoup-cli[sigstore]to sign or verify. An explicit Sigstore request can no longer degrade to unsigned.soup recipes verifychecks that a config can attach a LoRA adapter, without downloading weights (#1116 by @Srinivasan8888 in #1117). It builds each base's architecture on the meta device fromconfig.jsonalone, with the model class that config's trainer loads (only SFT chooses it bymodality), then builds the adapter config the way the trainer does —training.moe_loraincluded — and runs the real peft attach with the config's own LoRA settings.soup recipes verifychecks every shipped recipe, template and example (--no-templateslimits it to the recipes);--config soup.yamlchecks one before a long download;--jsonemits machine-readable rows. Exit codes follow the gate/verdict contract: 0 when everything checkable attaches, 2 when a config cannot attach, 3 for a bad flag or a missing or unparseable--config, and 1 for a crash or a missing or brokentorch/transformers/peft(it namespip install "soup-cli[train]"). Text from a config is printed safely: markup prints literally and control bytes are stripped.task: asrwithoutasr_lorareports no adapter, as that trainer full-fine-tunes, and the target resolver's advisories go to stderr, so--jsonstdout stays a document. A gated repo, one needingtrust_remote_code, or a config thistransformerscannot read is reported as unverified rather than broken, and remote code is never executed.soup bench trainmeasures a short SFT run and fails when the model was not training (#836 by @Srinivasan8888 in #1146).soup bench train --config soup.yaml --steps N --warmup W -o report.jsontrains for exactly N optimizer steps in a scratch directory and writes a JSON report. The report holds median and p95 step time after warm-up, supervised and total tokens, allocated and reserved peak memory kept separate, provenance, and a hash of the resolved config. It exits 1, and still writes the report, when there are no trainable parameters, a step loggedgrad_norm == 0or a non-finite norm, the trainable parameters did not change, or fewer steps ran than requested. An absentgrad_normis recorded as not reported, never as0.0. It coverstask: sfton the transformers backend.soup benchis now a group withinferandtrain(#836 by @Srinivasan8888 in #1146).soup bench <model>still runs the inference benchmark, which is now alsosoup bench infer <model>.soup bench --helplists the two subcommands; the inference flags are undersoup bench infer --help.Added ready-made
deepseek-v4-pro-dpoanddeepseek-v4-pro-grporecipes for deepseek-ai/DeepSeek-V4-Pro (#850 by @kok-o in #1297). Completes the task trio with the existing SFT sibling over the 1.6T-class flagship MoE architecture (LoRA r32/a64, 4-bit quantization,moe_lora: true,moe_aux_loss_coeff: 0.01,gradient_checkpointing: true). DPO usesmax_length: 4096,epochs: 1,lr: 5e-6, andgradient_accumulation_steps: 32. GRPO usesmax_length: 8192,epochs: 3,lr: 1e-5, andgradient_accumulation_steps: 16.soup inferandsoup bench inferaccept an experimental--cuda-graphsflag (#1322). On a resident Qwen2 or Llama model, optionally under one unmerged LoRA adapter fromsoup train, generation replays CUDA graphs instead of launching every kernel from Python; without the flag nothing changes. Measured on an RTX 5070 Laptop (Windows, unpinned) with Qwen2.5-1.5B-Instruct in FP16 and 128 greedy tokens per request, the slowest run with the flag was 2.9x the fastest run without it (2.8x under an unmerged LoRA), with identical token IDs on every request and at most +42 MiB of reserved VRAM, andsoup inferover 24 prompts of varied length, model load and compile included, took 96-102 s against 131-282 s with identical responses. Before the first row Soup runs one greedy warm-up on the longest prompt, so a one-prompt run can take longer overall; the summary'sDurationandThroughputstart after it. The flag needs PyTorch >= 2.14 and accepts plain LoRA only: quantization, CPU or disk offload, layer streaming, flash attention, DoRA and other LoRA variants, several adapters, a non-CUDA device or more than one GPU, and--task asrare refused before any output is written, and every refusal ends with anOmit --cuda-graphssentence. It is not available insoup chatorsoup serve. Record:benchmarks/gate-v0.76.0-cuda-graph-decode.md.soup uikeeps the session across reloads with an HttpOnly cookie on a loopback bind (#1191 by @mohdUwaish59 in #1474). Opening/?token=<TOKEN>checks the token, sets anHttpOnly; SameSite=Strict; Path=/session cookie named after the port (soup_ui_session_<PORT>) and redirects to the same URL withouttoken, keeping the other query parameters. The cookie holds a random session id that the server keeps only in memory, never the token, so a reload or a new tab stays signed in without the token being readable from page script, and every restart ends every session. A cookie-authenticated request other thanGET/HEADmust carry anOriginnaming this exact server (scheme, host and port), because every port of127.0.0.1is one site andSameSite=Strictalone would not stop a page from another local app; a missing ornullOriginis refused with 403. TheAuthorization: Bearerheader is unchanged and needs noOrigin, SSE streams accept the cookie while Bearer clients keep the ticket exchange, and CORS still sends noAccess-Control-Allow-Credentials. On--publicor any non-loopback bind no cookie is set or accepted, and the phone page keeps the token in page memory as before.Layer streaming can stripe its disk-tier cache across two or more NVMe drives (#1536). Set
SOUP_LAYER_STREAM_STRIPE_DIRSto extra folders on other NVMe drives (;on Windows,:elsewhere): decoder layeriis stored on drivei mod N, and the reader keeps one layer in flight per drive. Each folder must already exist, be absolute and not a link, sit on its own volume and classify as NVMe; anything else refuses the run and names the entry. Soup makes its own folder inside each root owner-only.training.stream_read_aheaddefaults to drives + 1. On a cold 70B-shaped NF4 store at seq 512 the training step went from 17.4 s to 9.97 s (1.75x) on two drives from a rested box (benchmarks/gate-two-drive-striping.md); under sustained back-to-back load both drives throttled and the speed-up measured 1.28-1.41x (benchmarks/probe-rtx5070-two-drive-sustained.md, no formal verdict yet). Unset, nothing changes.soup autopilotasks for a missing--model,--dataor--goalwhen run in a terminal (#1576 by @Deadpool2000 in #1577). When it is not run in a terminal it fails as before.Added a Simplified Chinese (
zh-Hans) README translation (#1632 by @Momoyeyu in #1733).README.zh.mdmirrors the English README section-by-section, while the shared language banner, code blocks, URLs, link targets, numbers, and inline code stay synchronized by the translation ratchet.Single-adapter health and silent-failure diagnostic scan (
soup adapters check). Audits a LoRA adapter for Frobenius weight delta ||ΔW||_F with float64 accumulation and rsLoRA scaling awareness, live layer fraction, all-zero lora_B initialisation traps, orphaned projections, and legacy .inner. wrapper namespace leaks without a PyTorch dependency, ending in a single contract line (alivevsinactive: <reason>). (#1721 by @harshitthek in #1738)
Changed
soup shipevidence decoding is now shared by the CLI and MCP surfaces (#758 by @ousamabenyounes in #768). A single strict schema reader now keeps verdict fields, refusal behavior, warning semantics, and the default forgetting threshold in lockstep; unknown verdict-affecting fields are rejected until both surfaces support them.Reviewing that reader found a second defect and it is fixed in the same release: the MCP error redactor paired on bare quotes, so when an evidence value held both quote characters -- where
repr()delimits with single quotes and backslash-escapes the internal ones -- the pairing slipped by one and the evidence text between them reached the transport. Each alternative now consumes an escaped character inside the run, so an escaped quote cannot end it.Unify exit codes across gate and verdict commands under a single 0/2/3 taxonomy (#813 by @kok-o in #921). Gate and verdict commands now consistently exit
0on pass / OK / minor / ship, exit2on gate failure / regression / drift / don't ship / suite failure (EXIT_GATE_FAILED), and exit3on usage or input errors (missing or unparseable input files, bad flags, invalid schema) (EXIT_USAGE_ERROR). Shared constants are centralized insoup_cli.utils.exit_codes.Breaking changes in command exit codes:
soup eval gate: exit on gate failure changed from1to2.soup eval against: exit on regression changed from1to2; exit on unsupported metric/tracker changed toEXIT_RUNTIME_ERROR(1) /EXIT_USAGE_ERROR(3).soup eval quant-check: alignsMAJORregression exit code withEXIT_GATE_FAILED(2); unparseable inputs exit3.soup lock check: drift exit code changed from3to2(EXIT_GATE_FAILED); missing/unparseable lock file exits3.soup lock show: missing lock file exit code changed from2to3(EXIT_USAGE_ERROR), aligning with documentation.soup expect: suite failure exit code changed from3to2(EXIT_GATE_FAILED); input/schema errors exit3.soup data validate: input errors exit3(previously1).soup data lint: unsupported format / input errors exit3(previously2).soup ship: offline--evidenceparse and malformed evidence errors exit3(previously1).- Click / Typer CLI usage errors (such as
--bogusor missing required options) on gate and verdict commands now exit3(EXIT_USAGE_ERROR), avoiding collision withEXIT_GATE_FAILED(2).
Changed
soup data recipe --executeand the Pythonrun_recipeAPI to refuse silent placeholder generation by default; pass a live provider or explicitly select offline mode (#817 by @Amix29 in #922).The v0.72.2 8B NF4 throughput protocol is now re-runnable on repaired code, and the post-repair row is recorded (#361 by @umran666 in #1014, measured in #1103). The RTX 3050 run has happened: Llama-3.1-8B-Instruct NF4 streamed at 208.6 tok/s in a 2.40 GB peak (5.70 GB pinned store) at 1935 MHz / 59 C, 100% util, 59% of the same-session GEMM ceiling against 68% pre-repair — the row and its clock in
benchmarks/gate-v0.72.2-nf4.md, the raw numbers inbenchmarks/results/issue361-post-repair-8b-nf4.json. It ran at roughly twice the pre-repair row's 952 MHz, so its 208.6 tok/s and the published 119.6 are not comparable directly and the record says so.benchmarks/harness/ issue361_nf4_throughput.pyrepeats the step-6 protocol unchanged (batch 1, S=512, 50 steps after 10 warm-up,PagedAdamW8bit, same-session GEMM ceiling) withrequire_pin=Trueso a pageable fallback refuses instead of recording the lower-bound shape, and the 119.6 tok/s / 3.32 GB copies in the README, the benchmarks index and the perf docs are now labelled pre-repair, with the reserved row inbenchmarks/gate-v0.72.2-nf4.mdnow filled rather than pending. CPU-tested protocol guards live intests/test_issue361_nf4_throughput.py.Adding a recipe is now one number plus one command (#1016 by @harryc295 in #1051).
tests/recipe_count.pyalready pinsEXPECTED_RECIPE_COUNT;scripts/sync_recipe_count.pyrewrites the documentation sites that state the count fromlen(RECIPES), so the second half of the edit stops being manual. It importsDOC_SITESfromtests/test_recipe_count_is_synced.pyrather than duplicating it, so the checker and the fixer cannot disagree about which sites exist or what their patterns are, and it rebuilds each match around the captured number so the surrounding wording survives byte for byte.--checkreports what is out of sync and exits 1 without writing. File I/O is byte-level rather than text-level so a one-number edit stays a one-line diff on Windows checkouts. It deliberately leavesEXPECTED_RECIPE_COUNTalone: the milestone stays a decision rather than a side effect of running a script. Tooling and tests only; no runtime behaviour changes.training.nvfp4and three of the foursoup export --format torchaoschemes asked torchao for classes it does not have (#826 by @Srinivasan8888 in #1066).NVFP4Config,Int8DynActInt4ConfigandFloat8DynActFloat8Configare not exported by torchao 0.18.0, the newest release, so a Blackwell user who setnvfp4: truegot a yellow advisory and a bf16 run, three export schemes could not build a config at all, and both refusals said "upgrade torchao" when there was nothing newer to upgrade to. Every class is now resolved through the module that defines it, so a rename fails naming the path;training.nvfp4usesNVFP4TrainingConfig, which quantises the forward and backward GEMMs, rather than an inference config that would only quantise weights after training. The four disagreeing torchao floors (0.4.0,0.5.0,0.7.0,0.5) become one constant,inner_k_tilesleaves the export allowlist becauseInt4WeightOnlyConfigraisesTypeErrorfor it, and a newtorchao-contractCI job installs.[qat]so the names are checked against the real package instead of a stub that defined them. Breaking for[qat]users: that one floor istorchao>=0.18.0, up from0.4.0, because 0.18.0 is the release these module paths were verified against; an older torchao is no longer accepted by the extra. Also narrowed:soup export --format torchaoused to findquantize_at the torchao top level or intorchao.quantization, and now resolves it fromtorchao.quantizationonly — the path that exists in 0.18.0, so nothing at the new floor loses it. Scope: this makesnvfp4: truereach NVFP4 training (the linears becomeNVFP4Linear), which the maintainer confirmed on a Blackwell GPU; a training step additionally needs triton and M, K, N all divisible by 128, which are torchao's constraints and are not checked by Soup.training.loraplus_lr_ratiowas silently ignored on every LoRA task except sft, pretrain and embedding (#745 by @abdulwaarith0 in #1080). #724 wired LoRA+ into the SFT family only. The other trainers that build a LoRA adapter (asr, bco, classifier, distill, dpo, grpo, ipo, kto, online_dpo, orpo, ppo, reward_model, simpo) accepted the ratio and trained the LoRA B matrices at the base learning rate. They now attach the LoRA+ optimizer, so B trains atlr * loraplus_lr_ratioand A atlr, and runs that set the ratio produce different weights than before. PPO builds its optimizer and scheduler eagerly, so the LoRA+ optimizer is passed to its constructor instead, and it also carries the value model at the base learning rate, as trl's default optimizer does. The ratio is now refused at config parse where it cannot apply:unlearn(noTraineroptimizer),prm(full fine-tune),moe_lora_routing(only the routing gate trains),classifier/reranker/cross_encoder/asrwithout theirclassifier_lora/asr_loraflag andlora.r > 0, andlora.use_vera(VeRA trains scaling vectors, not A/B matrices, so every tensor would train atlr * ratio). A source scan fails if a new LoRA trainer ships without wiring LoRA+ or a documented exemption.Sesame CSM is now refused at config load until Soup has a native multimodal trainer (#265 by @Amix29 in #1112). The unrunnable
sesame-csm-ttsrecipe is removed because CSM trains text plus 32 Mimi codebooks as parallel frames rather than a codec-token assistant turn.Llasa can now encode raw audio with Transformers-native XCodec2 during TTS training (#265 by @Amix29 in #1112). The live path resamples to 16 kHz, emits Llasa speech tokens, requires the matching
[audio]/torchaudiodependency, and releases the temporary XCodec2 model after dataset encoding instead of holding it through the training run.soup eval behaviorandsoup eval checklistnow exit with a usage error when required evidence is omitted instead of reporting a neutral successful gate (#1123 by @taljeon in #1129).ReLoRA restarts now follow the paper contract (#693 by @jagadeepmamidi in #1133): each fire merges the PEFT-scaled LoRA update into the base weight, reinitializes
lora_A/lora_B, clears optimizer state whenrelora_reset_optimizeris set, and re-warms the learning rate from1/(W+1)without an exact-zero step. Behaviour break:relora_stepsrequires explicitquantization: nonebecause the schema default is4bit. Embedding adapters with an emptylora_Aare skipped; Linear LoRA is still merged. The final active adapter is merged into the accumulated in-memory base and saved as a standalone dense output;--resumeand--hf-resumeare refused for ReLoRA because intermediate checkpoints do not encode restart history. Distributed/sharded execution remains explicitly refused, andrelora_prune_ratioremains ignored with a warning for backward-compatible YAML.Streamed CUDA NF4 now follows bitsandbytes' own fused-versus-dequantized dispatch, so it runs the same kernels as resident NF4 (#842 by @Amix29 in #1143). Pooled weight aliases stay checkpoint-visible and backward verifies stream-slot ownership.
An explicitly requested FP8 run now stops when torchao is missing, and says how to fix it (#835 by @Srinivasan8888 in #1154). With
quantization_aware: fp8orfp8_attention: trueon a card that can run FP8, the trainers used to print a yellow advisory and train in bf16 under a config that said FP8; that is whatsoup sweep,soup bench trainand the Python API reached. They now stop at setup withFP8DependencyMissingError, namingpip install "soup-cli[qat]", the same way an unsupported card already stops. A box with transformer-engine but no torchao stops too, because the converter is torchao's. A config that does not ask for FP8 is unaffected. If your runs relied on the old fallback, remove the FP8 setting or install the extra.soup trainchecks an FP8 request before loading the model (#835 by @Srinivasan8888 in #1154). It used to run the int8 QAT check forquantization_aware: fp8too, so a missing torchao stopped the run with the int8 hintpip install torchao, even on a card that cannot run FP8, where installing it would not help. It now asks the FP8 questions in the trainers' order: a card that cannot run FP8 is named as the reason, and on a card that can, a missing torchao namespip install "soup-cli[qat]". It asks for torchao's float8 converter itself, so a box with only transformer-engine stops here too instead of after the model loads. Onbackend: unslothonly the unsloth refusal is printed. Under--dry-runthe local card is reported as a note and the dry run exits 0, since FP8 configs are often written on one machine for another; a missing torchao and schema errors still fail it (a missing torchao prints the card note too), and the real run still stops on the card.FP8 now applies on SFT's audio path, and
stream_layers: trueandbackend: unslothrefuse FP8 and NVFP4 at config load (#1124 by @kok-o in #1155).quantization_aware: fp8andfp8_attention: trueloaded on audio SFT and were never applied;SFTTrainerWrapper._setup_audio_transformersnow converts the model after the LoRA adapter is attached, as the text and vision setups do, so a card that cannot run FP8, or a missing torchao, stops the run at setup instead of training in bf16.stream_layers: true(on every task it supports) andbackend: unsloth(on every task) now refusequantization_aware: fp8,fp8_attentionandnvfp4at config load, because neither path calls the converters: these configs loaded before and trained without the precision they asked for, except thefp8combinations on unsloth, whichsoup trainalready refused before training. The unslothfp8message says to switch tobackend: transformers.soup bench trainsamples the SM clock while the counted steps run, and keeps every step time (#1166 by @AaronProbha18 in #1177). Breaking (report schema):provenance.sm_clock_mhz_after_runis removed. That value was onenvidia-smiread taken after training returned, on a card that was already idle. The gate-836 record measured it at mostly 180 MHz, and never above 1417, against a busy median of 2370–2557 MHz, and it could not tell apart two runs whose throughput differed 1.83x. The replacement,provenance.sm_clock_mhz_busy, hasmin,median,maxandsample_countover samples kept only from the post-warm-up steps. Queries start on a 100 ms schedule, a query slower than that skips ticks, and each sample is timestamped at the midpoint of its query.unavailable_reasonsays why no clock was reported: not a CUDA run, nonvidia-smi, no readable clock, or no sample inside the counted steps. The clock is queried for torch's own device by PCI bus id, and a failing or unparseable driver-version query is nownullrather than the tool's error text.timing.step_seconds(new) lists every counted step in run order, so a mid-run mode change can be located from one report. The docs now say that pre-flight refusals (unsupported task or backend,--steps <= --warmup, zero trainable parameters) write no report, and that only post-run check failures leave avalid: falseone.An untied streamed checkpoint reloads the embedding at the tail of the previous step's backward instead of at the start of the next forward (#975 by @VanshSharmaPES in #1178).
embed_tokensand an untiedlm_headshare one device slot, so every step swapped the slot back to the embedding right as the forward began, a full head-sized copy with nothing to overlap it. The reload now happens once layer 0's backward recompute is reached, so it overlaps the rest of the backward, and the forward-start load becomes a same-owner no-op from the second step on. The refill is ordered after the head's still-queued backward by thewait_streaminLargeLayerBufferPool.load_async, which a new GPU test pins. On an untied Llama with hidden 2048, vocab 128256 and a 501 MiB slot, on an RTX 5070 Laptop GPU, the embedding's compute-stream wait fell from about 19-23 ms to about 0.15 ms per step; that is still about 0.1% of a cold 70B step, and the step-time change (2-6.5%) is close to that fixture's noise floor. Preference losses get part of it: DPO's no-grad reference forward and KTO's extra forwards still load the embedding at the start of their forward. Tied checkpoints have no shared slot and are unchanged, and eachtrain()now ends with one extra large-layer load that no later forward consumes.Every non-standard
grpo_variantnow applies thegrpo_betaKL penalty, so existing variant configs will not reproduce prior runs (#1232 in #1249).gspo,dapo,dr_grpo,bnpo,two_sidedandrftnow train with a KL penalty ofgrpo_beta(default0.1) against the reference policy, so their losses, gradients and trajectories change. Before, they validatedgrpo_betaand dropped it. The variant loss replaces trl's_compute_loss, the one place on Soup's path where trl adds the KL term, so neithergrpo_betanor the β that the reward-hack controller (kl_control/pid_lagrangian) sets at runtime changed the loss or the gradient, while trl still ran the reference forward on every step. Each variant now adds trl 0.29's per-token estimatorexp(ref - logp) - (ref - logp) - 1, weighted bygrpo_beta, to its per-token loss before its own reduction, which is where trl's loss adds it.rftapplies it only to the accepted completions it trains on. On a real trainer, thedapoloss and gradient now equal trl's stockdapoones at β 0.1, 1 and 10 on the same generated batch.An untied streamed checkpoint issues the head's load right after the embedding lookup instead of at the last decoder layer (#1174 by @VanshSharmaPES in #1258). The embedding's bytes in the shared large-layer slot are dead once the lookup has run, so
StreamPrefetchernow firestail_prefetchfrom layer 0'sadvance()(head_prefetch_layer, default 0) rather than the last layer's, giving the head's copy every decoder layer's forward to hide behind instead of one.head_prefetch_layer = Nonerestores the old timing. Loss, gradients and adapter weights are bit-identical between the two timings, and the sequence and count of real large-layer loads are unchanged; the refill is ordered after the queued lookup by thewait_streaminLargeLayerBufferPool.load_async, which a new GPU test pins. On an RTX 5070 Laptop GPU, with an 8-layer untied checkpoint (hidden 2048, a 501 MiB shared slot, NF4, sequence 512), the step's median totalwait()fell from 11.96 to 4.50 ms on the RAM tier, from 15.83 to 1.42 ms on the disk tier, and from 57.29 to 29.16 ms on the disk tier atstream_read_ahead: 3for a step that runs a no-grad forward first. A decoder layer's copy now queues behind the head's on the shared copy stream, so the other decoder layers wait longer (0.78 to 4.06 ms on the RAM tier), which gives back part of the saving; layer 0 was the best of layers 0 to 3. Tied checkpoints have no shared slot and are unchanged.training.quantization_aware: trueis now refused at config load, on every task and backend (#1222 in #1275). It never ran quantization-aware training. Where it was applied, which is the transformers model setup of thesft,dpo,kto,orpo,ipo,bco,simpo,grpo,ppoandpretraintrainers (also run bytask: ttsandtask: preference), it ran torchao's post-training int8 weight-only quantization over every Linear layer after LoRA, adapter included: a LoRA run had zero trainable parameters, and none of the runs tried (SFT, SFT with gradient checkpointing, full fine-tuning, DPO) completed. Everywhere else it was not applied at all.backend: mlxignored it with a warning.backend: unslothnever applied it:soup trainrefused the combination, andsoup sweepandsoup bench traintrained without it (#1248, superseded by this change). The trainers forreward_model,embedding,prm,online_dpo,asr,classifier,reranker,cross_encoder,distill,unlearnandmoe_lora_routingnever read it, and neither did layer streaming or SFT's audio path, so those runs trained without QAT. Breaking for both groups: a config that sets it loaded before and now stops at load.soup trainexits 1, the Web UI and API return the message, andsoup doctor --configexits 2. Every spelling the field reads as true is refused (yes,on,1,"true"...), with a message that namestraining.quantization_awareand #1222. Remove the key or set it tofalse.quantization_aware: fp8andquantization_aware: questare unaffected, because they use their own code paths; the one visible change forfp8is thatsoup train's refusal offp8onbackend: unslothnow saysUse backend: transformers.instead of recommendingquantization_aware: true. Real int8 QAT is a separate feature and is not available yet.training.use_longlora: trueis now refused at config load (#1240 in #1278). It never trained correctly: the LongLoRA override rolled the query/key projections of half the attention heads along the sequence with wrap-around, under full causal attention, so earlier positions could read keys computed from the last tokens of each sequence (future tokens leaked into the training loss), and no S² grouped attention ran at all. Breaking for configs that set it: a config that loaded before now stops at load, even though it never trained correctly.soup trainexits 1, the Web UI and API return the message, andsoup doctor --configexits 2. Every spelling the field reads as true is refused (yes,on,1,"t"...), with a message that namestraining.use_longlora, says why, and points attraining.rope_scaling_typewith plain LoRA for context extension. Remove the key or set it tofalse. It stays refused until real S² attention is implemented; no shipped recipe, template or example set it.training.ebft_variantis now refused at config load (#1230 in #1279). It never trained correctly: the EBFT term it added to the SFT loss scored each position's logits against that position's own input token, with no causal shift, so it rewarded copying the input over predicting the next token (on the real trainer the added term was exactly the unshifted cross-entropy). Shifting it would not help: shifted, the term is the model's own cross-entropy, so the loss would count cross-entropy twice. EBFT is not yet a distinct objective, and it stays refused until the intended objective is implemented from its reference. Breaking for configs that set it: a config withebft_variant: structuredorstridedloaded before and now stops at load.soup trainexits 1, the Web UI and API return the message, andsoup doctor --configexits 2. Every non-null value gets the same message, which namestraining.ebft_variant, says why and names #1230;ebft_temperatureon its own now points at the refusal instead of asking forebft_variant. Remove both keys to train plain SFT. GDPO (gdpo_variant) is unchanged (its own problem on trl 0.29 is #1309), and no shipped recipe, template or example used EBFT.Refuse ten optimizer names at config load instead of crashing inside the trainer: transformers 5.x rejects
adam_mini,adamw_apex_fused,adamw_hf,ao_adamw_fp8,ao_adamw_4bit,ao_adamw_8bit,badam,came_pytorch,dionandmuoninTrainingArguments, so each used to pass config load and fail after the model had loaded. They are removed from the allowlist with a message naming working alternatives, the LoRA-FA validator no longer recommendsadamw_hf, and a new test buildsTrainingArguments(optim=name)for every allowlisted name so a rejected name fails the suite instead of a user's run.muonandadamw_hfstill work onbackend: mlx(the MLX backend builds them itself); every other backend refuses them with a message that names the backend (#1269 by @wanjinhao1 in #1283).Allow
training.grpo_beta: 0to configure KL-free GRPO recipes (DAPO, Dr. GRPO), skipping the reference-model forward pass; refusegrpo_beta: 0with beta-adjusting reward-hack mitigation modes (kl_control,pid_lagrangian) at config load (#1247 by @kok-o in #1295).soup abnow runs a two-sided test, says whether the treatment is better or worse, and gives no verdict before 30 rows per arm (40 below--alpha 0.05) (#1227 in #1310). The mSPRT tested onlymean_t - mean_c = +effect_size, so a treatment that moved the other way crossed the accept boundary: ajudge_scoreregression of any size was reported as "No significant difference" (exit 0, webhookaccept_h0), and onlatencyandretry_rate, where lower is better, so was a real improvement. The log-likelihood ratio is now the log of the average of the likelihood ratios for+effect_sizeand-effect_size(a symmetric two-point mixture, computed with logsumexp), with the samelog((1 - beta) / alpha)boundary. Areject_h0carries adirection,betterorworse, read through the metric's polarity (judge_scorehigher is better;latencyandretry_ratelower is better). The verdict table and the webhook payload show it (nullonaccept_h0), and the panel recommends a rollback only forworse.MsprtVerdictgains the field and requires it on areject_h0. Fewer than 30 rows in either arm now always givescontinue, or fewer than 40 when--alphais below 0.05 (min_rows_per_arm(alpha)), and the verdict table and panel show the value in use.soup abestimates the variance from the rows, and from a handful of them the estimate is too noisy: without a burn-in, re-running after every new pair would reject a true H0 up to 16% of the time at--alpha 0.05(the one-sided test it replaces: 11%). A Monte-Carlo sweep over effect sizes from 0.1 to 5 standard deviations, with runs followed to 1000 rows per arm, picked the burn-ins: at--alpha 0.05, 30 rows keep the worst case at 0.051, within Monte-Carlo error; at--alpha 0.01the worst case stays near 0.0108, about 8% above the nominal level whatever the burn-in, because the variance is estimated from the rows (a variance-robust statistic is #1265). The issue's own 20-row reproduction now returnscontinueat the default--alpha, because 20 rows fall inside the burn-in. The calibration holds up to 1000 rows per arm and for--alphaof 0.01 and above, andsoup abprints a warning past either limit. The sweep is committed underbenchmarks/(gate-1227-ab-burn-in.md). Because one boundary now covers both directions, the log-likelihood ratio for an effect in one direction is aboutlog 2lower than before. The exit code is unchanged, 0 on every decision, so readdirectionto gate on a regression.Training now evaluates the validation split it holds out, and generation tasks stop holding one out (#1223 in #1313).
data.val_split(default0.1) kept that share of the rows out of training on every run, but no trainer on the transformers backend scheduled an evaluation, so those rows were never used andval_lossstayed empty. Every task that receives a split now evaluates it at the end of each epoch, at the per-device train batch (not Hugging Face's 8), and recordseval_losson the Val loss row, in the tracker'sval_lossand inTrainEvent.val_loss;prmandmoe_lora_routing, which attach no live callback, keep it in the trainer's log history. Runs take a few percent longer. The newtraining.eval_steps: Nevaluates every N optimizer steps instead.grpo,ppoandonline_dpoevaluate by generating, so by default they neither evaluate nor withhold rows:data.val_splitis ignored with a one-line note and every row trains, which changes the rows a default config trains on.grpoholds out and evaluates whentraining.eval_stepsis set. It evaluates at the largest multiple ofnum_generationsthat fits in the train batch, since TRL scores whole groups, and a note says so when that differs from the train batch. Its evaluation rewards never reach the reward-hack detector, the mitigation controller or the echo-trap detector.ppo,online_dpo,unlearnandbackend: mlxrefuse the setting, and each refusal says why. The embedding trainer gains the evaluation step that Hugging Face's default could not run on its anchor/positive batches.An explicitly requested FP8 conversion that fails now stops the run instead of training on (#1152 by @Srinivasan8888 in #1333). With
quantization_aware: fp8, a float8 conversion that raised used to print a yellow line and train in bf16; it now stops at setup and quotes torchao's error. Withfp8_attention: true, a conversion that fails partway, or a model with no attention projections, used to print its own "PARTIALLY converted" or "refusing the silent no-op" message as a yellow line and train on anyway; both now stop the run, as those messages say. This follows #835's rule for a card that cannot run FP8 and for a missing torchao.utils/fp8.py::validate_fp8_config, which nothing called, is removed.training.lr_groups,training.early_stop_patienceandtraining.citation_recall_thresholdnow warn at load, and v0.77 refuses them (#761 by @Srinivasan8888 in #1337). All three were documented and validated but read by nothing:lr_groupsnever set a per-module learning rate,citation_recall_thresholdnever gated a save, andearly_stop_patiencenever stopped a run. Setting one away from its default now prints "accepted but read by nothing; v0.77 will refuse it", as the #808 fields do.early_stop_patiencemoves fromsoup train's "set but not enforced" note to that warning, so it is reported once, with the date. The docs no longer showlr_groupsorcitation_recall_thresholddoing what they never did.Nine training-intelligence and convergence knobs warn at load and v0.77 refuses them (#808 by @Guts1005 in #1379).
forgetting_eval_steps,forgetting_benchmark,forgetting_stop,checkpoint_eval_steps,checkpoint_eval_metric,checkpoint_eval_tasks,checkpoint_keep_top,convergence_window, andconvergence_rel_tolwere accepted and validated, but none of the nine ever changed a run. Setting any of them away from its default now prints "accepted but read by nothing; v0.77 will refuse it". They move fromsoup train's "set but not enforced" advisory note to the loader's warning so each knob is reported once, with the refusal date. Deleting the key keeps configs loading after v0.77, and schema defaults stay silent.soup abnow decides with a variance-robust statistic, with no burn-in, andaccept_h0can take longer to reach when there is no difference (#1265 by @abdulwaarith0 in #1419). The two-sided mSPRT from #1227 estimated the variance from the rows, so it needed a burn-in of 30 or 40 rows per arm, was calibrated only up to 1000 rows and down to--alpha 0.01, and still sat near 0.011 at 0.01. The normal-inverse-gamma Bayes factor that replaces it depends on the rows only through the t statistic, so its false-positive rate stays at most--alphahowever often you re-run it, from 7 rows per arm and at any number of rows: the worst simulated rate is 0.033 at--alpha 0.05and 0.0066 at 0.01. The first 5 rows per arm set the scale for--effect-sizeand show in the table asprior_scale; the burn-in table, the 1000-row limit and both calibration warnings are gone. This is not a pure speed-up. When the arms are the same,accept_h0takes several times as many rows as before (about 281 pairs against 43 at an--effect-sizeof 0.5 standard deviations), and at small effect sizes it may not arrive within 1000;docs/evaluation.mdhas the full table, and #1575 in this release replaces the accept rule, which brings that down to 117 pairs. Part of the old speed was wrong answers: under a real difference of 0.3 standard deviations, the old test ended about 18% of runs inaccept_h0, against under 0.4% now on simulated Gaussian rows. When the first rows barely vary (a warm cache, or a judge that gives 1.0 early),accept_h0waits ascontinuewhile the tested rows spread more than 3 times as much as the held-out ones, since that start made nearly every run accept H0 with or without a difference; areject_h0is never held back, so the false-positive bound is unchanged. The--alpha+--beta< 1 check from #1339 does not ship: #1575 retires--betain the same release. The #1339 and #1384 refusals of values a float cannot carry cover the new statistic too: it is never compared with the boundaries unless it is finite.soup diagnoseno longer scores a probe that never ran as OK 1.00 (#1435 by @bakdauletbaktygaliyev in #1489). A live probe that was requested but produced no measurement, such as a--datasetwith no usable rows or a probe that raised, now reports the newNOT_RUNverdict (grey in the badge and the table, ranked between MINOR and MAJOR) and keeps the exception text in its evidence.soup diagnoseexits3when the overall verdict isNOT_RUN; pass--allow-not-runto keep exit0for runs that knowingly skip probes. MAJOR still exits2.soup train --diagnose-gatenow also refuses aNOT_RUNreport (exit3), and a bad--tokenizeris an input error (exit3) raised before any model loads. Probes that do not apply, and modes omitted from an--evidencefile, stay a neutral OK.Breaking:
training.gdpo_variantis now refused at config load. Every non-null value is refused, includingstandard,length_normalizedandmargin; unset and null still load. On supported TRL versions (0.29 and later) the setting previously loaded but silently changed nothing becauseDPOTrainer.dpo_lossno longer exists. Removegdpo_variantand use plaintask: dpoforstandard;task: simpois the nearest, not identical, objective forlength_normalized;marginhas no equivalent. GDPO metadata no longer advertises live wiring, and the retained legacy helper raises instead of silently ignoring a missing loss hook. No new loss implementation is added; no shipped recipe, template or example sets this field. (#1309 by @taljeon in #1515)Breaking for
soup data score --benchmarks/-bwithout comparison texts: benchmark names are labels, not bundled corpora; the command now refuses labels alone by name and asks for--benchmark-file. With a file, labels are validated only, not corpus selectors; help, docs and an execution note state that all file texts are compared. Plain scoring reports decontamination as not run instead of a misleading zero. The new cwd-contained operator JSONL option passes actual text and--thresholdto the existing 8-gram scorecard kernel, without benchmark downloads or a new scoring algorithm. Invalid, empty, truncated or unusable benchmark evidence fails before a scorecard is printed. The shared loader's strict mode is opt-in; other callers retain their previous behavior. This covers only the data-score item, not the remaining command paths. (#1448 by @taljeon in #1518)A
simpo+orpoblend now trains instead of refusing (#1425 by @umran666 in #1556). #1513 made every blend refuse, because no trl 0.29 trainer puts per-sequence log-probs on the batch. The CPO (SimPO) and ORPO trainers both return theirs fromconcatenated_forward, so the wrapper now captures them there: a blend naming onlysimpoandorporuns. Blends namingdpooripostill stop and name the terms — those trainers compute log-probs inline, publish only means, and need a frozen reference model this path does not build.Behaviour changes:
{simpo: 0.7, orpo: 0.3}is live on trl 0.29 where it previously stopped at step 0.bcomixed with anything is still rejected at config load.- Each blended loss now reads its hyperparameters from the config rather than off the primary trainer, which is whichever loss has the larger weight.
training.simpo_gammaandtraining.orpo_betanow reach their own term either way; before,orpo_betacould feed SimPO's beta and ORPO's odds-ratio weight was always1.0, so the same weights written in a different YAML order trained a different objective. - In a blend,
simpoandorpoare their preference terms only — without the likelihood term the standalonepreference_loss: simpo/orporuns carry. A{simpo: 1.0}blend is not the standalone SimPO loss, andtraining.cpo_alphadoes not affect it. Usetraining.preference_lossfor a full standalone loss. - With
val_splitset, trl'sprediction_stepcallsget_batch_loss_metricsdirectly, so eval reports the primary loss's own trl loss rather than the blend's value.
soup abreachesaccept_h0about twice as soon when there is no difference, for every user and at the defaults, and--betais retired (#1418 by @abdulwaarith0 in #1575). The statistic and thereject_h0rule from #1265 are unchanged.accept_h0used to wait for the Bayes factor to fall tolog(beta / (1 - alpha)), which took 621 pairs per arm at an--effect-sizeof 0.3 standard deviations (--alpha 0.05). It now comes once a confidence sequence for the difference, built from the same mixture at level--alphaand valid however often you re-runsoup ab, lies inside(-effect_size, +effect_size): 307 pairs at 0.3, 117 at 0.5 (was 281). This changes when everysoup abrun saysaccept_h0, with no flag to opt out, and so when the--slack-urland--discord-urlwebhooks fire with"decision": "accept_h0"; a pipeline that waits on that message will hear it sooner. The accept panel now says what it means: any difference is smaller than--effect-size, at confidence1 - --alpha. A true difference of exactly--effect-sizeends inaccept_h0in at most 0.0083 of simulated runs at--alpha 0.05(was 0.0042) and 0.0019 at 0.01 (was 0.0032), so the chance of detecting it stays within 0.005 of before; the false-positive rate can only fall, and does. Thelog_likelihood_ratioreported next to anaccept_h0, in the table and the webhook payload, is still the reject rule's statistic and can be positive.--alphanow bounds both wrong verdicts, so--betano longer does anything: for this releasesoup ab --beta Xprints that, says a stricter accept means a lower--alpha, and runs exactly as without it, andMsprtConfig(beta=...)raises the same message as aSoupConfigDeprecationWarning; Soup v0.77 refuses it. The--alpha+--beta< 1 check from #1339 went with it. The record isbenchmarks/gate-1418-ab-cs-accept.md, anddocs/evaluation.mdhas the new time-to-accept table.Changed the private-address refusal on outbound endpoints to say what to do: it now ends with "address the server by its hostname".
http://0.0.0.0insoup data generate --api-baseand the chat proxy now says "0.0.0.0 is ambiguous; use 127.0.0.1 or localhost", the same hintHF_ENDPOINT, hub endpoints and webhook URLs give, from one shared constant (#1549 by @PranavD2905 in #1582).soup abnow says when the accept hold is what is keeping a run atcontinue. #1419 holdsaccept_h0back while the tested rows spread more thanACCEPT_HOLD_SPREAD_RATIO(3) times the held-out ones, and the held-out rows are fixed — so once that hold fires on a saturated warm-up (a warm cache, or a judge that gives the same score for the first rows), no amount of extra data of the same kind releases it. The run sat atcontinuefor the whole dataset with the accept rule already satisfied, while the panel printed "Insufficient evidence. Collect more samples and re-run.", advice that cannot help. Acontinuenow carriesaccept_heldplus the two spreads that decided it (held_out_spread,tested_spread, andheld_out_rows), and the panel prints one line naming the ratio and what to do about it. The four keys are not added to the webhook payload: a held run is acontinueand the webhook fires only on a terminal decision, so they could only ever befalse/nullthere — the panel and theMsprtVerdictare where the facts live. These are new keys only: nothing is renamed or removed, and a run that is not held carries no note at all. The accept rule itself is unchanged (#1524 by @yunaremaia in #1585).QuEST: refuse a local base whose
config.jsoncannot satisfy the route before any weight file is read, instead of after the model loads. The 24-block count and the group-128 / power-of-two widths are now checked from the config the fingerprint already parses, so a base that was going to be refused is no longer loaded and no longer has every byte of its weights read and SHA-256-hashed — onmainthat was once per run, and it is now none. The refusal message is unchanged and the post-load topology gate still runs; a width that cannot be read from the config is left to that gate rather than guessed at. (#1199 by @yunaremaia in #1588)Refuse
training.lora.use_doraon unsloth and mlx backends (#1651 by @kok-o in #1657). Rejectuse_dora: trueat config load time whenbackendisunslothormlx, which do not implement DoRA adapter constructors.Reworded
README.tr.mdandREADME.ar.md(#1681 by @Ercaner1988). LLM reviewers, each prompted to read as a native speaker of one language and with no shared context, flagged false friends and calques in the shipped translations (for example "issue", "Serving", "gated" and "wheels" rendered with words that mean something else). The findings were applied with LLMs and vouched for by the files' maintainer, who reads Turkish natively and Arabic at an upper-intermediate level.README.mdand the sync stamps are unchanged.soup servesays what--tool-auth-tokencovers when it binds beyond loopback. A non-loopback--hostis refused without the token, which reads as if the token put the whole server behind a credential; it is checked on the tool, thumbs and adapter routes only, the generation routes take none, the vLLM / SGLang / MII backends have no route that checks it, and on a wildcard bind (0.0.0.0,::) theHostheader is not compared. None of that changes. The command now prints it as aNote:at startup on such a bind, the--hosthelp says that the token is required there (it said "should be paired with") and what it is checked on, anddocs/serving-and-export.mdhas a paragraph on what the token does not cover (#1689).Cache live canary policy and batch outcome writes: keyed serving requests refresh the parsed policy when its file identity changes, and stats flush after 64 outcomes or two seconds, including an orderly-shutdown flush. Watch rollback takes effect on the next keyed request; superseded promotions cannot overwrite newer statistics. On-disk
canary-stats.jsonand watch verdicts now lag traffic by the buffered window (normally up to 64 outcomes or two seconds); a crash can lose that window. Disk outages retain one pending batch but refuse additional outcomes, so outage losses can exceed one batch. Process-killing failures never reach recording. (#1320 by @akkupratap323 in #1712)Layer streaming page-locks less memory for a pinned RAM store whose largest tensors come last (#1702 in #1713). The store is pinned in power-of-two arenas (#901), and the planner filled them in allocation order, looking only at the arena it opened last. A Qwen3-8B NF4 store is 3.58 GB of decoder tensors followed by two 1.24 GB vocabulary matrices; neither matrix fit behind the decoder, so each opened a 2 GiB arena of its own and 8 GiB was page-locked for a 6.07 GB store.
plan_pinned_arenasnow also plans the arenas largest tensor first and uses that layout only when it page-locks strictly fewer bytes: the same store takes 6 GiB (8,589,934,592 -> 6,442,450,944 bytes, measured on an RTX 5070 Laptop), and peak process RSS in a three-step run went from 10.85 GB to 8.71 GB. A store that gains nothing, including the disk tier's staging on the measured geometries, is laid out exactly as before, and there is no new config field. Three deterministic optimizer steps are byte-identical between the two layouts (logits, loss, 144 gradients, 144 updated adapter tensors). This is a memory change, not a speed claim: a 40-run A/B on the same box (old and new layout alternating, the disk tier as the control) found no difference in step speed at its resolution of +/-15%. The record isbenchmarks/gate-1702-arena-packing.md.The
soup trainVRAM pre-flight now checks the pre-quantized formats (#1652 by @ybai08 in #1729). It used to build its estimate only fornone,4bit,8bitandmxfp4; twelve other formats started without a check and without a word.gptq,awq,aqlmandeetqare now checked at the figures the predictor already had (0.55, 0.55, 0.5 and 1.0 bytes per parameter).fp8is checked as well, and its figure is corrected from 1.0 to 2.0: an FP8 base is loaded withdequantize=Trueand trains in bf16, so the old figure promised half the memory the run needs. A refusedfp8run gets the sentence anmxfp4run gets.hqq:*has no figure yet, so the run starts and the pre-flight printsPre-flight skipped: no memory model for quantization: hqq:4bit. This refuses configs in those five formats that used to start on a card too small for the estimate;--allow-oom-attemptlaunches them anyway.
Deprecated
soup export --format awqand--format gptqare deprecated and will be removed in Soup v0.77. Both upstream projects (AutoAWQ and AutoGPTQ) are archived, neither extra can be installed next to[train], andauto-gptqhas no Python 3.12 wheel, so the two formats could not be reached from a normal install. Until the removal they work as before. The command now starts with aDeprecated:notice that names the formats to move to (gguf,onnx,tensorrt,bitnet,tq1_0), so it is there whatever stops the run, the missing extra included;soup quantize --to awqand--to gptqprint the same notice after the command they hand you; and the--formatand--tohelp texts mark both formats (#1690).
Removed
Remove unreachable kernel auto-compose production helpers and validator branches after schema-level rejection of kernel_auto_compose: true (#976 by @kok-o in #1006).
Remove orphaned
build_bitnet_trainerandvalidate_bitnet_compatfromutils/bitnet.py, unreachable after schema-level rejection ofquantization: bitnet_1.58training (#1004 by @VanshSharmaPES in #1034).Removed the dormant
auto_quantcompatibility module (soup_cli/utils/auto_quant.py) thatsoup serve --auto-quantstopped using when it began refusing with exit 2 in #816; the refusal itself is unchanged and still covered by regression tests.import soup_cli.utils.auto_quantnow raisesModuleNotFoundErrorwith nothing replacing it (#1131 by @wanjinhao1 in #1262).The Web UI plugin registry,
soup_cli.ui.plugins(register_tab,list_tabs,get_tab,clear_tabs,load_plugins,TabSpec), is removed, and with it the "Web UI Plugin Registry + Env Knobs" section ofdocs/serving-and-export.md. The module has been importable since v0.44.0, sofrom soup_cli.ui.plugins import ...stops working; no rendered feature is lost, becausecreate_appnever read the registry and a registered tab was never shown.soup plugins(soup_cli.plugins) is a different feature and is unchanged (#1465 by @Srinivasan8888 in #1608).Drop unused
eval_configfromSoupTrainerCallback(#1643 by @kok-o in #1654). Remove unusedeval_configparameter and attribute frombuild_soup_trainer_callbackandSoupTrainerCallbackafter auto-eval migration tosoup train.
Fixed
soup doctornow reports the MLX backend (#659 by @ump45nose in #660). The MLX panel had been written but never wired into the report, so Apple Silicon users saw no MLX section at all.doctornow renders an MLX panel showing the installed version, chip, and unified memory when MLX is present, says plainly when it is missing on Apple Silicon, and stays quiet elsewhere.get_mlx_version()readsmlx.core.__version__(falling back to distribution metadata) instead ofmlx.__version__, which themlxpackage does not export and which therefore always reported "unknown".The MLX benchmark harness now pins, and verifies, every field that moves its step counts (#716 by @dchaudhari7177 in #727).
benchmarks/harness/mlx_sft_smoke.pyinheritedgradient_accumulation_stepsfrom the schema default, so the published M1 record depended on a value the harness never mentioned: at the default of 4 the record's 48 iterations are 12 optimizer updates, and because the wrapper roundsitersdown to a whole accumulation group (#696), a row count such as 50 silently drops rows. The harness now pinsbatch_size,gradient_accumulation_steps,epochsandlrwith the reason for each, resolves iterations and optimizer updates from the loaded config the same wayMLXSFTTrainerWrapperdoes, and exits non-zero before training when either is notrows * epochs, naming both counts and any dropped rows. No published figure changes: @Srinivasan8888 re-ran the branch on the M1 8 GB the record was measured on and reproducedTrained Tokens 2130exactly.tests/test_issue716_harness_step_guard.pychecks the resolution against theTrainingArgsthe real wrapper builds and drives the abort path throughmain().streaming: truewithinterleave: overandval_splitno longer puts a row on both sides of the split (#702 by @dchaudhari7177 in #729).overstreams withstopping_strategy="all_exhausted", which recycles the shorter source, and the positional tail slice then put a row in val and its recycled copy in train, so validation loss on that source measured memorisation: 10 of 20 val rows leaked on the issue's 100-against-10 shape. The streamingoversplit is now sized over distinct rows and draws val from content that occurs once, so train keeps every row and its oversampling. It falls back to repeated content only when it must, and then withholds that content's other copies from train with a warning giving the count. A split that would leave train empty raises the same kind of error the eager path's per-source carve-out does. Streaming val comes from distinct rows in stream order, so unlike the eager path it is not balanced across sources, anddocs/data.mdnow says so. @AmirF194 found that the first version silently dropped a genuinely repeated source row.soup migratenow recognizes multi-record JSONL by structure even when the filename does not end in.jsonl, so a data file such asdata.jsongets the existing friendly wrong-file error instead of a raw YAMLParserError. Single JSON objects and Unsloth notebooks still reach their selected migrator, and the structural sniff remains bounded to 64 KiB per line (#730 by @be-student, finding by @swalla02 and @AmirF194, closes #699).Stabilize distillation loss and gradients in log space (#719 by @be-student). All three divergences, including forward-KL, now compute and return FP32 scalars for FP16/BF16 inputs. Non-finite logits propagate for AMP step skipping; no per-step finite-value guard synchronizes the device. The FP32 trade-off is documented in
docs/training.md: the maintainer measured 1.71x time and 1.24x peak memory for the original guarded kernel, and removing the guards reduced time from 10.91 ms to 9.40 ms (main: 6.39 ms). This revision also avoids unused probability tensors. Two FP32 logits copies use about 7.8 GiB at B=4/S=2048/V=128256. These measurements are hardware-specific and were not rerun for this revision (PR #736).Distillation no longer multiplies its gradient by
gradient_accumulation_steps(#720 by @dchaudhari7177 in #742)._DistillTrainer.compute_lossreturns a mean over its own microbatch, so the same effective batch split four ways previously produced four times the gradient and changed the effective learning rate. The trainer now setsmodel_accepts_loss_kwargs = TrueafterTrainer.__init__, consumes Transformers' full-windownum_items_in_batch, and weights each microbatch by its share of trained tokens as completed in #943.tests/test_issue720_distill_grad_accum.pyruns the real_DistillTrainerbody throughTrainer.training_stepon a one-layer Llama and asserts the accumulated gradient matches the unaccumulated one at 4 and 8 steps.soup shipstamps the judge's actual load numerics into the verdict and--emit-evidence, and refuses a replay under a different family (#746 by @jagadeepmamidi, Closes #367). Live evaluation already reusedtraining.quantizationfor the load (#570); the absolute leg-1 / leg-2 numbers for a 4-bit run were still recorded as if the judge's precision were implicit.--emit-evidencenow writes a top-levelnumericsstamp (4bit/8bit/bfloat16/float32), the panel and--outputreport it, and--configplus--evidencerefuse a family mismatch (4bitvs full precision,4bitvs8bit) the same wayconfig_sharefuses a drifted recipe. Pre-#367 evidence without a stamp warns rather than failing closed. This is a correctness-of-absolute-numbers fix: paired NF4 vs bf16 comparisons that judged both arms the same way remain valid.soup trainleft a run atstatus='running'forever when setup failed (#764 by @SID-6921 in #767). The tracker registered a run before loading the model, but the onlyexceptthat marked a run failed wrapped the training call alone — a tokenizer/model load, quantization, LoRA attach, or dataset mapping failure in the ~250 lines before it left the row stuck atrunningwith no way to tell it apart from a live run. On a real database this was 220 of 432 rows, the oldest since 2026-03-04. The failure boundary now wraps everything from right after the run is registered through the end of the command, andfail_runrecords the exception type and message on the row (runs.error_message, added via the existing lazy migration). Separately,soup trainnow records its own pid viamark_runningthe way the MCP execution path already does, so_reconcile_orphaned_run(#401) can rescue a SIGKILL'd or Ctrl+C'd run on the nextsoup runsinstead of leaving it invisible to a mechanism this project already built for exactly that purpose.error_messageis redacted and length-capped before storage, andsoup runs shownow surfaces it and reportsterminated/launchingcorrectly instead of falling back torunningfor every non-completed, non-failed status.train_on_responses_only: falsetrained on a doubled BOS, one token off from inference after #782 (#788 by @abdulwaarith0). The default SFT path (train_on_responses_only: true) builds its ids withadd_special_tokens=Falseindata/loss_mask.py, so it never doubled the BOS. The opt-out text path did:data/sft_format.pyhanded TRL a{"text"}column, and TRL's language-modelingtokenize_fnre-tokenised it with the tokenizer's defaultadd_special_tokens=True. A chat template that renders{{ bos_token }}(Llama-3, Gemma, Mistral) on a tokenizer whose post-processor also prepends BOS therefore trained on[bos, bos, ...]. Before #782 that matched inference by accident; after #782 inference sends one BOS, so adapters trained on the text path were off by one token from how they are prompted.soup data preprocesstokenised the same way. The text path now pre-tokenises withadd_special_tokens=False(viabuild_full_sequence_labels), so the chat template is the single source of special tokens, matching inference and HF's ownapply_chat_template(tokenize=True): one BOS for a{{ bos_token }}template, and zero for a template that renders no BOS at all — both Soup'sdata.chat_templatepresets and native templates of that shape (TinyLlama, Zephyr). The stop token is preserved rather than lost. TRL's language-modeling path also appendedeos_tokenas a string to every text row that did not already end with it (add_eos), independent of the tokenizer's post-processor, so the pre-tokenised row reproduces that rule: the EOS id is appended when the sequence does not already end on it. A{{ bos_token }}template on a BOS-only or EOS-less (Qwen-shaped) tokenizer therefore keeps the single trailing stop token it trained on before. This changes training results in two ways on configs that hit the affected shapes: a template that renders no BOS (TinyLlama / Zephyr, not only Soup's presets) now trains with zero leading BOS rather than the tokenizer's default one, and a row that used to end on two trailing EOS tokens — a template or post-processor that appended one on top ofadd_eos's — now trains on exactly one. The rule applies only to text a chat template rendered; pretrain-style raw text with no template keeps the tokenizer's own special tokens unchanged.soup data doctor --show-masknow renders through the same builder, so the X-ray matches training. Adapters trained withtrain_on_responses_only: falsebefore this change carry one extra leading BOS in training compared with inference.soup data preprocessde-duplicates the same leading BOS but stays pinned to the old cache contents otherwise: it tokenises as before (add_special_tokens=True) and drops only the one duplicated leading BOS, so every other token is unchanged — including the post-processor's EOS and the room HF truncation reserves for it, so a truncated row still ends on its stop token. Its EOS therefore remains whatever the post-processor added (never TRL'sadd_eos), so a Qwen-shaped tokenizer's cache and live path can still differ on the EOS — that pre-existing mismatch is tracked separately in #791 rather than changed here. The cache also keepsmain's single leading BOS where the live path now yields none for adata.chat_templatepreset; that BOS divergence is tracked in #876. Thesoup data preprocesscache key changes so datasets cached under the old encoding are re-tokenised rather than reused.docs/commands.mdcovers every registered leaf — includingsoup mcp runs reconcile— behind a Typer-walking coverage ratchet (#822 by @MrBeldum in #855). The matcher requires a real entry row (token-bounded, no prose-only hits, one|//-connecteda|b|cshorthand group) so deleting a command's own rows fails the guard.Require both models and valid probe keys in edit diff (#818 by @kok-o in #863).
soup edit diffpreviously allowed probes without models to fall through to a placeholder path that reportedchanged: falsefor each unmeasured probe and exited 0. Now, specifying--probesrequires both--before-modeland--after-model(exiting 2 naming missing flags if omitted), asymmetric model flags are rejected naming the partner flag, and probe files with zero valid rows or missing the'prompt'key exit 2 with an explicit diagnostic rather than silently producing placeholder claims.soup serve --backend vllmno longer sends the engine a doubled BOS (#785 part 2 by @abdulwaarith0 in #867). #782 fixed every path that tokenizes in process, and left the engines that tokenize the prompt string themselves for someone with the hardware to check. Measured on vLLM 0.29.0 withunsloth/Llama-3.2-1B-Instruct, whose Llama-3 template renders{{ bos_token }}(id 128000): for the string Soup sent, the engine reportedprompt_token_ids[:2] == [128000, 128000]over 37 ids, against the 36 idsapply_chat_template(tokenize=True)returns. The vLLM backend now hands the engine{"prompt_token_ids": [...]}built with no tokenizer special tokens whenever the chat template rendered the prompt, which reproduced the 36-id encoding exactly ([128000, 128006]), so the model is prompted with the prefix Soup trains on.usage.prompt_tokensdrops by one on such templates. The change is wider than removing a duplicate: for a model whose chat template renders no BOS but whose tokenizer post-processor would add one, the engine previously received one BOS and now receives zero, matching Soup's training path and the in-process fix #782 already shipped for the transformers backend. Models without a chat template, and templates that fail to render, still send the legacy string and are tokenized by the engine exactly as before. The SGLang and DeepSpeed-MII backends were measured through their own tokenizer paths and are tracked separately in #785.The README translation ratchet no longer passes a re-stamp without a re-translation (#870 by @Ercaner1988, follow-up to #852). The sync stamp proves when a translation was synced, and the structure check proved headings, code-block languages, URLs and inline code survived — but a command changed inside a fence (
soup train→soup trainx) or a figure changed in prose (119.6→999.9) stayed green behind a valid stamp. Each##section of a translation is now also compared withREADME.mdon its code-block contents with#comments dropped (comments being the only part of a code block a translation may change), its relative link targets, and its numbers in prose; in-page anchors, which are translated with their headings, must resolve to a heading of the same file. Measured figures therefore keepREADME.md's digits in every language:README.tr.mdhad 10 figures written with Turkish separators (119,6,3,32 GB), now converted. Each translation names its maintainer in the test, and a red run says whom to ping.Fixed
soup doctorexiting 0 after reporting a[train]package as INCOMPATIBLE: an installed package beyond its declared ceiling (transformers,peft,trl) now makes the command exit non-zero, while a missing or outdated[train]member stays advisory (#874 by @chiliec in #877).soup ingest --source langfuse --pullis bounded in time and filtered on the answer, not only on the request (#865 by @Konuktor in #881). Three findings from the #859 review. The timeout reached the socket rather than the call, so a server sending a byte before every timeout expired kept the pull alive indefinitely — a 0.4 s timeout allowed a 2.01 s call against a drip server; each request now carries a wall-clock deadline covering the connect and the whole response, and the body is read in chunks so the deadline is checked while bytes are still arriving.type=GENERATIONwas only a query parameter, so a server that ignored it would have written spans and tool calls out as training rows; each observation's own type is now checked before it becomes a row, the parameter is still sent, and skipped observations are counted in the summary. A pagination cursor the server repeats used to run until--max-pages— 100 identical requests by default — and now stops on the first repeat. No change to the default (no--pull) path, which still imports no HTTP code and opens no connection.soup mcp serve'sdata_validateno longer reports every row valid for a format it does not recognise (#878 by @Srinivasan8888 in #882).validate_and_statsenables its format check only when the requested format is inVALID_FORMATS, so an unrecognised value silently turned the check off and a missing one never turned it on — both answered "every row valid" for a reason unrelated to the data.soup data validatewas guarded against that in #869, so the same file got two different verdicts depending on whether the CLI or the MCP tool asked. The handler now resolvesformatthe waydata_doctorbeside it already did: anything outsideVALID_FORMATSraises a tool error naming the accepted values, an omitted orautoformat is auto-detected as the tool schema always promised, and the resolved format is returned in the report so a caller can see which one was checked. The allowlist isVALID_FORMATSitself rather than a copy, since a hand-written second tuple is how the two surfaces drifted apart to begin with.soup doctornow handles modern NVIDIA CUDA driver headers and supported PyTorch wheel indexes (#827 by @Samearth17 in #883). CUDA detection accepts bothCUDA Version: X.YandCUDA UMD Version: X.Yheaders with flexible whitespace, and PyTorch wheel selection now uses only indexes that provide releases compatible with Soup'storch>=2.6.0floor. An unknown driver version now returnsNoneinstead of defaulting tocu121, and drivers reporting CUDA 12.1–12.3 now map tocu118becausecu121is not in the supported wheel table. When no supported CUDA wheel can be selected,soup doctorfalls back topip install "soup-cli[train]"instead of generating an invalidwhl/Noneindex. The GPU diagnostic also distinguishes a CUDA-enabled Torch build that cannot initialise CUDA from a genuine CPU-only Torch build.soup doctornames missing[train]packages on a partial stack (#875 by @HarshRajSinghania in #884). A six-of-seven install no longer prints "Training stack not installed".The
serve-fastextra now floorsvllm>=0.4.3, up from0.4.0(#892 by @abdulwaarith0). Since #867 the vLLM backend hands the engine a{"prompt_token_ids": [...]}mapping as the first positional argument toAsyncLLMEngine.generate. vLLM only accepts a mapping there from 0.4.3 on, whengenerate's first parameter becameinputs: str | TextPrompt | TokensPrompt | TextTokensPrompt; 0.4.2 and earlier take apromptstring plus a separateprompt_token_idskeyword, so on those versions the mapping lands on the string parameter and is mis-handled. The old0.4.0floor therefore advertised support the backend no longer has.soup ingest --source langfuse --pullprinted the server's error body to the terminal with its raw ANSI escapes intact, letting a hostile Langfuse response inject terminal control sequences (#885 by @wangzhengzhuo05 in #893).commands/ingest.pyrenderedPullErrorthrough barerich.markup.escape, which neutralises[...]markup but passes C0 control bytes (ESC, NUL, BEL) straight through, so a crafted response body could reach the terminal as live escape sequences. The user-facing path now routes the message through the sharedutils/terminal.py:for_terminalhelper (C0 strip + markup escape) at bothPullErrorprint sites, so the server's text is still shown but its control bytes are not. Added by @wangzhengzhuo05.soup draft distillandsoup shrinkreport a validation failure in their own rendered config as a CLI error rather than a traceback (#895 by @1cbyc in #904). Both commands validate the YAML they generate before starting a training subprocess and sanitize the validation message for terminal output.soup train --configon a YAML document that is a list or a scalar now exits 1 withConfig must be a YAML mapping, got listinstead of aTypeErrortraceback (#905).load_config_from_stringhad refused that shape for releases; the file call site never did. The unknown-config-key report also no longer claims to be capped when a config has exactly the cap's worth (100) of unknown keys and nothing was dropped.The config-field consumer guard no longer counts a read inside a function nothing calls (#807 by @Srinivasan8888 in #906). The guard added in #748 treated a field as wired if its name was read anywhere under
src/, sotraining.warmup_autopassed on one line inautopilot.generate_config— a function with no caller, whose line reads the autopilot decisions dict and writes the value into a config rather than reading the field. The file already refused exactly this forschema.py@propertyresolvers and did not apply the same reasoning to ordinary functions. A read now counts only when the function enclosing it is referenced somewhere insrc/; one live read still rescues a field that also has dead ones, and a module-scope read always counts because module scope runs on import.LoraConfigis also in scope for the first time, which surfaceslora.loftq_iterandlora.loftq_bitsas having no read at all. Six fields are allowlisted with their issue references rather than left failing, so the number can only shrink as #794 and #808 are answered.Six command modules,
utils/ship_verdict.py,mcp_server/registry.py, andcommands/serve.pycarried a private copy of the terminal control-byte sanitizer instead of importing the shared one (#896 by @VanshSharmaPES in #907). v0.75.0 addedsoup_cli.utils.terminal.for_terminal(strip C0/DEL control bytes, then escape Rich markup) for the config loader's unknown-key report, but nine other call sites —commands/adapters.py,data_canary.py,data_doctor.py,draft.py,infer.py,shrink.py,serve.py's--auto-specpairing message,utils/ship_verdict.py, andmcp_server/registry.py— still carried a byte-identical_CONTROL_STRIP_TABLEunder their own name, which is exactly how the config loader got missed for a release. All nine now importfor_terminal(or the new strip-onlystrip_control, for the handful of call sites that feed non-markup contexts likeText.append(..., style=...)or an exception message escaped later downstream) from the shared module, and a new repo-wide test (test_issue896_no_duplicate_terminal_sanitizer.py) fails CI if a module besidesutils/terminal.pyever defines_CONTROL_STRIP_TABLEagain.Short training runs no longer show a fabricated
0.0000 -> 0.0000loss delta (#899 by @taljeon in #908). All Hugging Face-style trainer wrappers now share one loss-history summary: two or more per-step measurements produce a delta, one measurement or a run-leveltrain_lossmean produces one value without an arrow, and missing data is shown as unavailable. The completion panel therefore never claims a before/after comparison it did not measure, while non-finite per-step values remain visible instead of being replaced by the last healthy loss.soup planandsoup applynow refuse a config carrying an unknown key (#894 by @wangzhengzhuo05 in #909)._load_yaml_configread the YAML withyaml.safe_loadand handed the raw mapping tobuild_plan, so it never built aSoupConfigand never ranfind_unknown_config_keys: a typo'd key planned cleanly and was refused one command later bysoup train. Both commands now exit 1 with the loader's message and suggestion.Repinned
notebooks/proof-4gb.ipynbfrom an unreproduciblegit+...@maininstall to the releasedsoup-cli[train]==0.75.0, corrected its "0.73.0 is published" and "#347 is open" claims (#347 shippedtraining.stream_vram_overridein v0.73.1), added that override to the section 5 config so the pre-flight checks against the same 4 GB cap the allocator enforces, and added a cell printingtorch/transformers/peft/bitsandbytes/soup_cliversions. Colab's free-tier T4 runtime now defaults to Python 3.13, ahead of soup-cli's<3.13ceiling, so the install cell falls back topip install --ignore-requires-pythonwhen it detects that. Re-ran the notebook end to end on a free Colab T4 and committed it with outputs: section 4's streamed-vs- resident forward comparison is bit-exact (torch.equal = True), and section 5 trained Llama-3.1-8B under the 4 GB cap with a measured peak of 1.83 GB. Addedtests/test_issue844_notebook_pin_guard.pyso a future stale pin, an unreleased version, or a renamedsoup_cliimport fails CI instead of surfacing on the next Colab run (#844 by @VanshSharmaPES in #910).Fixed
soup eval againstso it can compare results written by Soup's own eval commands, rejects unknown metrics before database access, distinguishes unavailable comparisons from regressions in generated hooks, and reports single aggregate scores without pretending they form a confidence interval (#812 by @Amix29 in #914).The Web UI now rejects oversized YAML-entry request bodies before parsing (#897 by @Amix29 in #915).
/api/config/validate,/api/train/start, and/api/config/from-formreturn HTTP 413 above 1 MiB. The limit checks both the declaredContent-Lengthand the bytes actually received, so chunked requests and understated headers cannot bypass it; all 171 shipped recipes remain valid.soup data lintno longer fails correct conversational DPO data onprompt_leak, andlength_biasneeds a real size difference (#810 by @dchaudhari7177 in #916). A conversationalchosen/rejectedcarries its prompt as the leading user turn, and the linter flattened every turn, soprompt_leakflagged every row whose prompt was 40+ characters, including both shipped DPO examples, andlength_biasmeasured prompt plus answer. Both checks now read only the assistant turns.length_biasalso requires the mean lengths to differ by at least 10% for MAJOR and 5% for MINOR, on top of Cohen's d, so a consistent one-word gap between near-constant lengths (or the zero-variance fallback) is no longer MAJOR on its own.soup eval quant-checknow exits with code 2 on aMAJORaccuracy regression verdict, rejects standalone.ggufpaths up front (exit code 3), and requires--allow-stubfor stub scoring (#809 by @kok-o in #917). A model load failure on either side now reports which side failed and exits with code 1 instead of silently falling back to deterministic stubs and reporting a falseOK. Breaking: a run that previously passed on stubs now needs--allow-stub. When--allow-stubis explicitly provided, stub outputs are marked with"stub": truein JSON and display visible warning banners in table/markdown output.soup reward stressnow detects structure-preserving attacks that game built-in format and substring verifiers (#814 by @kok-o in #918).reward_stresspreviously checked four classic junk strings (empty,length,repetition,sentinel), none of which exercised the structure rewarded by reasoning scaffolds. Consequently, the built-informatreward was reported as "robust (not gameable)" despite awarding 1.0 to<think></think>a, andaccuracymissed distractor answer-sprays.soup reward stressnow adds two structure-preserving attack families (wrapped_junkandanswer_spray), expands each family into multiple distinct variant attempts, determines verdicts per-family to prevent denominator dilution, testsanswer_sprayagainst real gold substitutions with non-collapsing distractor generation, and ensuresverifiablemath checks remain robust (exit 0).Apply
data.chat_templateconsistently to conversational preference, reward-model, GRPO, and Online DPO trainers, and reject it on non-chat tasks (#804 by @Amix29 in #919).soup profileno longer reports an assumed GPU as a measured fit, and knows newer and laptop cards (#830 by @dchaudhari7177 in #920). With no GPU detected and no--gpu, it printed "OK Fits in 24 GB VRAM" and a batch size as if it had measured a 24 GB card; it now says no GPU was detected, shows the would-fit result against the assumed size, and asks for--gpu, and--jsonand the MCPprofiletool reportgpu_memory_source(flag,detectedorassumed).--gpunow accepts the RTX 3050, 5060 through 5080 (including Ti variants), B200 and laptop parts whose memory differs from the desktop card, as well as device names as torch reports them ("NVIDIA GeForce RTX 5070 Laptop GPU"). The speed estimate is labelled as a typical A100 figure.Build PEFT LoRA configs through one shared helper so
rank_patternandalpha_patternwork consistently outside SFT and pretraining (#803 by @Amix29 in #923).Label the legacy
soup data toxicityscorer honestly as an abuse-keyword heuristic, stop treating ambiguous technical terms as safety signals, preserve legacy output keys, and remove it from Magpie's default quality filter (#821 by @Amix29 in #925).Stop Autopilot from enabling or advertising unwired forgetting and checkpoint-quality gates, and warn when any staged sibling setting differs from its default. (#927 by @Amix29)
Configure Transformers RoPE before text SFT and pretrain model construction, because changing the config after construction left the rotary frequencies unchanged (#793 by @Amix29 in #931). Preserve only type-independent native parameters when changing algorithms, forward YaRN tunables, and refuse nested per-layer RoPE configurations rather than reporting a partial extension as enabled.
Sharding a large checkpoint no longer dies on Windows.
shard_checkpointheld every source*.safetensorsmapping open for the whole of pass 2, andsafe_openmaps a file privately on Windows, so each live handle charged commit for that file's entire size — measured, 48.99 GB mapped raised the commit charge 46.17 GB while free physical memory never moved. A real Llama-3.1-70B (30 files, ~141 GB) therefore charged the whole checkpoint and died with an access violation (no traceback, exit 139), so it could not be sharded for layer streaming at all. At most two source handles are now alive at once, and the tensor readers return tensors that own their memory: aget_tensorview keeps its whole file mapped, and charged, for as long as it lives, which would otherwise defeat the cap. (#934, fixes #926)Discover installed Soup plugins lazily, persist explicit opt-in state across processes, and connect enabled hooks to training (#819 by @Amix29 in #935). Third-party entry points are now listed from package metadata without importing their code;
soup plugins enable <name>is the explicit code-loading boundary. Malformed recursive state and Rich-markup resource names are handled safely. BREAKING: the non-installingsoup plugins installadvisory now exits 2 instead of reporting success with exit 0.soup data preprocesscached chat rows without the EOS the live path trains on, so a cached run could train with no stop token (#938 by @abdulwaarith0). The live SFT path (train_on_responses_only: false) reproduces TRL'sadd_eosrule indata/loss_mask.py: a chat row that does not already end oneos_tokengets one appended.soup data preprocesstokenised the rendered template directly, with no such step, so a cached row carried an EOS only when the template or the tokenizer's post-processor supplied one. On a template that renders no EOS and a tokenizer whose post-processor appends none (the Qwen shape), the live path trained on one stop token and the cached path on zero. It was silent: the loss curve looks normal andsoup data doctor --show-maskrenders the live path, not the cache. The preprocess cache now applies the sameadd_eosrule for chat rows, appending the EOS id when the row does not already end on it, and only while the row is under the length budget so a truncated row is not pushed pastmax_length(the live path, which appends then truncates, keeps no trailing EOS there either). Pretrain rows never go through a chat template and are unchanged. The leading BOS still differs between the cache and the live path by design and is tracked in #876. Thesoup data preprocesscache key changes, so a dataset cached under the old encoding is re-tokenised rather than silently reused. Models trained from asoup data preprocesscache before this change may have trained without a stop token on templates that render none, which teaches run-on generation; re-runsoup data preprocessand retrain to pick up the stop token.Rewrite the shipped DPO and chat-preference examples so rejected responses retain a clear quality contrast without teaching a response-length shortcut, and regression-test both files with
soup data lint(#929 by @taljeon in #940).soup reward stress verifiable --verifiable-domain json_schemanow forwards supplied references asschemametadata instead of reporting that--referencesis missing (#937 by @Amix29 in #941). JSON-object schema fields are decoded before scoring, and a verifier that returns the wrong number of scores after receiving references now gets a distinct contract diagnostic. The built-in attacks remain non-JSON and therefore cannot yet discriminate schema strictness;reference_acceptis not a meaningful control for this metadata-based domain.Distillation gradient accumulation now weights microbatches by their shifted, non-masked training-token counts (#932 by @Amix29 in #943). This supersedes #742's fixed-per-microbatch compensation: the trainer now accepts Transformers' accumulation count and normalizes each microbatch by its share of the window's trained tokens. Unequal-length microbatches therefore produce the same gradient as the equivalent full batch instead of giving short and long microbatches equal influence.
CLI help, config descriptions, and feature guides now describe shipped behaviour instead of expired release plans (#823 by @Amix29 in #945). Removed stale
schema-only,deferred, andlands inclaims from live features, removed internal release-phase labels from command help, and stoppedsoup advise run --helpfrom recommending the nonexistent--no-recordflag. Descriptions for genuinely incomplete surfaces now state their current limitation without promising a past release. A version-aware source-level ratchet scans every Typer help string and Pydantic field description while allowing documented future release notices and explicitly allowlisted schema-only fields.Sweep leaderboards now keep diverged arms visible but sort them after finite losses (#944 by @taljeon in #946). Non-finite losses are labelled
diverged, never selected as the best run, and explicitly count as worse than an early-stop threshold instead of relying on NaN comparison behavior.The Web UI now caps
/api/chat/sendand/api/data/inspectrequest bodies before they are parsed (#939 by @AmirF194 in #948). Both routes let FastAPI read and parse the whole body before the handler's own checks ran, sochat/sendforwarded an unbounded body upstream andinspectpaid a full read even on a path that was always going to be rejected.chat/sendnow caps at 1 MiB,inspectat 8 KiB; both checkContent-Lengthand the bytes actually received, so a missing or understated header cannot bypass the limit.autodistill's MLX teacher worker verification failed under a venv on Windows, whereScripts/python.exeis a launcher that spawns the real interpreter as a grandchild (#898 by @SID-6921 in #950).run_mlx_teacher_capture_processcomparedreceipt.worker_pid(the child's ownos.getpid()) againstsubprocess.Popen(...).pid. Under a real venv those are two different processes, so the comparison always failed even on a successful capture. The parent now generates a nonce and hands it to the child in the request; the child echoes it back into the worker receipt, and the parent verifies the nonce instead of the PID.worker_pidstill appears on both models -- the receipt's own value (os.getpid()) is the useful one; the result's mirrors whateverPopenreported, which is the launcher's PID on exactly the platform this fixes, so it is informational only, not proof of anything.The control-strip ratchet skipped a file it could not parse, so a broken module could hide behind a green run (#949 by @jarik2014 in #951).
tests/test_issue896_no_duplicate_terminal_sanitizer.pywalks every tracked file undersrc/soup_cli/withast.parseand aSyntaxErrorwas caught and skipped, which made the guard's clean answer mean "zero offenders among the files I could read" rather than "zero offenders" — and a module that does not parse is a worse problem than the duplicate table the ratchet exists to find, since it would surface later as anImportErrorsomewhere unrelated. The guard now fails naming the file and the error, and a test proves it by writing a real unparseable file inside the scanned tree (registered withgit add -Nso it is part of thegit ls-fileswalk) and asserting the guard goes red.soup mcp serve'sdata_doctornow refuses blank and unknown formats instead of treating blank as auto-detect or falling through to unrelated tokenizer/doctor errors (#889 by @Rayan-and-beyond in #952). The handler now mirrorsdata_validate's #882 behavior: omittedformatstill auto-detects, explicit values must come from the sharedVALID_FORMATSallowlist, and an invalid value raises the same bounded error naming the rejected value and accepted formats. Tests pin blank and near-miss rejection, message parity, every current valid format, allowlist extension without a handler change, and the omitted-format control.MCP export and profile errors now name the rejected format or GPU and list the current accepted values. Error formatting is shared with data validation, and control characters in rejected values are escaped (#953 by @shubhambhar007 in #954).
Reject
training.kernel_auto_compose: trueinstead of timing one unchanged model and falsely reporting that the fastest kernel combination was applied. Useuse_ligeroruse_flash_attnexplicitly. (#801 by @Amix29 in #955)Narrow the MLX backend-support table to the four reachable CUDA-only fields (
use_mod,moe_lora,quantization_aware,use_fsdp2_compile); the other five from #958 are refused by schema validation before MLX ever sees them (#958 by @ege-arhan in #961).Data recipes no longer report an unreachable provider as a successful run (#930 by @taljeon in #969). Live
llm_textandjudgenodes now count transport, non-2xx, and malformed-response failures; a node whose every provider call fails exits non-zero and names the endpoint. Partial runs keep their successful rows and report the failure count, while a legitimate empty completion remains a successful response rather than being mistaken for an outage.Layer streaming's disk tier no longer reads on the compute thread (#971 in #973).
stream_source: disk(andautowhen the base does not fit RAM) used to call a synchronous, memory-mapped read from the thread that was supposed to be feeding the GPU, so every byte arrived through page faults while the card idled. The base now streams through a background reader that parses each shard's header itself — no memory map, so nothing charges commit for the whole file — and stages layers into host memory that is page-locked when the box allows it, with a loud pageable fallback rather than a silent one when it cannot; settingtraining.stream_pin: trueon this tier now REFUSES a run the box cannot page-lock, where it previously announced that pinning does not apply and went on regardless. Measured on one box (RTX 5070 Laptop, Windows, NVMe; record) against a same-day control of the source it replaces, cold on a 36 GB 70B-shaped NF4 store larger than RAM at batch 1 × seq 512: 92–100 s per step became 30–48 s (2.1–3.1×), each pair position-matched — 5.1–5.6 → 10.6–16.9 tok/s, 0.70–0.76 → 1.46–2.32 GB/s at the source — with peak VRAM unchanged at 4.38 GB. It is still bound by the read: per-layer read brackets are 82.6–84.9% of the step. Warm, with the whole store in the page cache, it is a REGRESSION of 1.03–1.20× (the two figures are the two run orders; the direction survives reversing the order, the magnitude does not) — with the store cached a synchronous read is close to amemcpy, so there is nothing for a reader to hide; if the store fits RAM the RAM tier is 1.8–2.3× faster than either disk arm and is whatautoalready picks. A shard that changes on disk while a run holds it is refused by name rather than read at stale offsets, and a reader that wedges mid-read now surfaces as an error instead of a hang. Newtraining.stream_read_ahead(default 2, range 1–8) sets the reader's depth, where a setting of N stages N−1 layers ahead because one slot is always held by the layer being consumed, and each level costs one layer of pinned host RAM (1.5 GB at depth 1, 2.0 at 2, 2.8 at 4 on a 70B) — which the pre-flight now charges, prints and refuses the run over if it will not fit free RAM, instead of predicting zero host residency for this tier. No depth was distinguishable on throughput in that record once run order was controlled for, so treat it as a host-memory knob. A non-defaultstream_read_aheadbesidestream_source: ramis refused rather than accepted and ignored: the reader belongs to the NVMe tier andramnever falls back to it. The default is accepted, because a default is not a decision.Enabling MiniLLM with the default offline settings is now a config error instead of a silent no-op (#692 by @jagadeepmamidi in #977).
minillm_enabled: trueplusminillm_on_policy: falseplusminillm_teacher_mix_ratio: 0.0used to load and forward the teacher, then blend it at weight zero so the MiniLLM term wasKL(student || stopgrad(student)). Ordinary cross-entropy kept the loss curve healthy. That combination is rejected at parse and names the three fields; ratio 0 stays legal on the on-policy path, where it only means student-only sampling and the loss is stillKL(student || teacher).BitNet 1.58 training configs now fail at config load instead of falling through to the ordinary trainer and later reporting
unknown quantization(#825 by @Rayan-and-beyond in #981). The error states that BitNet training is not implemented yet and points to the still-workingsoup export --format tq1_0path. The unusablefalcon-e-bitnet-sftrecipe and the docs that advertised training as live are removed, with recipe snapshots and catalog counts kept in sync.soup loop watch's canary deploy stage no longer reports a canary verdict of OK when nothing evaluated a canary (#815 by @AmirF194 in #982).deploy_to_canaryactivates the trained adapter at 100% traffic and recordedcanary_verdict: "OK"from that activation's bare 2xx; the actual per-request traffic split and verdict machinery incanary_routernever ran. The stage now reportscanary_verdict: Noneon that path, matching every other exit in the function.training.use_flash_attnandtraining.use_ligerare now rejected at config load for any task outsidesft/tts, the only trainers that read either flag.soup autopilotstops auto-enabling both for DPO/GRPO/pretrain on Ampere or newer, where they previously built a config the CLI's own validation would then refuse for a package that task never used.soup migrate --from axolotland thegenerate_configautopilot emitter got the same scoping, so neither hands back a config the loader then rejects (#806 by @AmirF194 in #983).Five tests that passed with colour off and failed with colour forced now answer the same way in every shell (#886 by @Srinivasan8888 in #984). Under
FORCE_COLOR=1they failed for reasons unrelated to what they assert, through three distinct mechanisms. A test fixture built its RichConsolewithout pinningforce_terminal, so it colourised into the buffer and"\x1b" not in outbecame unsatisfiable; that fixture is now pinned, which keeps the broad assertion rather than narrowing it to hand-picked escape sequences. Two banner checks and twosoup doctorNCCL checks matched raw output that colour splits with SGR codes, and now match ANSI-stripped output, as the newer tests in those files already did. The control-byte test also gained an assertion that the stripped payload stays visible as inert text, so a sanitiser that swallowed it entirely would no longer pass. None of the five was red in CI, where no workflow forces colour, but this class has turned CI red four times before when something upstream changed.soup edit diffrefuses both models with no--probesinstead of writing an empty diff (#880 by @Srinivasan8888 in #985). Given--before-modeland--after-modelbut no probe file, it exited 0 and wrote a report withtotal_probes: 0, which reads as "the edit changed nothing" when nothing was measured. It now exits 2 without writing a file, and the message names--probes. The three combinations #818 and #863 already refuse are unchanged, and running with neither model and no probes still succeeds.Report unavailable training loss for MLX SFT and unlearn runs that capture no loss, instead of inventing 0.0000 (#960 by @Rayan-and-beyond in #986).
Layer streaming's pageable fallback no longer poisons the run, and the RAM store page-locks at its own size (#901 in #989). A Qwen2.5-14B NF4 run on an 8 GB card with 32 GB of RAM predicted a 3.39 GB peak against 7.41 GB free, printed "could not page-lock the base … falling back to a PAGEABLE RAM store", and died with
AcceleratorError: CUDA error: out of memoryinsideSFTTrainer.__init__. Memory was never the problem: a failedpin_memory=Trueallocation leaves the CUDA runtime's last error set, and the run's first kernel launch reported it as an out-of-memory with gigabytes free (cudaMalloc,synchronizeand memcpy all pass; a kernel's launch check reads it)._build_sourcenow drains that error and releases the page-locked blocks the abandoned attempt left in torch's host cache before it builds the pageable store, on both tiers. The page-lock itself failed because torch's caching host allocator rounds every pinned request up to the next power of two, so pinning the store one tensor at a time cost 1.73x its bytes (measured on the real 14B shard cache: 6.82 GB requested, 11.83 GB page-locked);RamSourcenow pins the store in power-of-two arenas with every tensor a view, so the same 9.93 GB store page-locks as 10 GiB and the ready line prints the real figure. The VRAM probe also calls an out-of-memoryAcceleratorErroran OOM verdict rather than an instrument failure. Measured on the dev box with a synthetic 14B shape: the reporter's config now trains end to end with the store pinned, peak 2.944 GB allocated / 3.244 GB reserved — the pre-flight's prediction was right. Record:benchmarks/gate-901-14b-on-8gb.md; the shape generator (synth_checkpoint.py --shape qwen2.5-14b --arch qwen2) and the step probe (issue901_stream_step_probe.py) ship underbenchmarks/harness/.The layer-streaming disk tier reads each layer through direct I/O in sector-aligned ranges, into arena-packed staging (#974 in #991). Cold, on a 36 GB 70B-shaped NF4 store larger than RAM, every buffered read primitive topped out at 1.2–2.9 GB/s on the dev box's NVMe whatever the request shape — the cache manager, not the drive, was the ceiling gate-971's async source sat at (1.46–2.32 GB/s, 84.7% of the step) — while unbuffered reads of the same fresh layers reached 3.5–5.65 GB/s. The reader now reads each layer's data section as sector-aligned byte ranges through
FILE_FLAG_NO_BUFFERINGon Windows,O_DIRECTon Linux andF_NOCACHEon macOS (a bufferedopenwhere a filesystem refuses, e.g. tmpfs), four ranges in parallel through per-worker handles, into ONE staging region per slot with tensors as views, and packs the pinned regions into #901's power-of-two arenas so the disk tier's staging stops paying the per-tensor 1.7–1.9x rounding. Measured position-matched in both run orders on the dev box (RTX 5070 Laptop 8 GB, Windows 11): a cold 70B-shaped step went from 110.5 s on the synchronous path in the same slot to 16.2–17.3 s (6.4x), against 34.8–45.6 s for the reader this replaces (2.0–2.8x), with the read share down from 84.7% to 62–64% at 6.0–7.0 GB/s and the two run positions within 1.07x of each other; peak VRAM unchanged at 4.38 GB. Warm, with the store in the page cache, the async source had been reported 1.03–1.20x slower than the synchronous one (gate-971 §5); position-matched it was 0.91–0.97x before this change — the bracket was run order — and after it, in one session with the reader it replaces imported fromorigin/main, the new reader is 0.76x of the old one warm on the plain point (1.582 vs 2.091 s; 0.94x on the instrumented points) and 2.25x faster right after a page-cache eviction (1.788 vs 4.024 s), with the RAM tier (1.13–1.20 s) still the floor for a store that fits. Two consequences worth knowing: a disk-tier run no longer warms the page cache for whatever runs after it, and a store that fits RAM still belongs on the RAM tier (stream_source: auto). The ready line prints the page-locked figure on this tier too;direct_io,pinned_bytesandstaging_bytesare reported. Record:benchmarks/gate-974-disk-tier-direct-io.md; the read-primitive probes ship underbenchmarks/harness/issue974_*.py.CLI escape-hygiene tests no longer answer differently depending on the shell they run in (#987 by @AaronProbha18 in #993). Seven tests assert that an untrusted field — an exception message, a filename, a manifest secret, a config digest — is sanitised before it reaches the terminal, spelled
assert "\x1b" not in result.output. Each command prints through an unpinned module-levelConsole(), so underFORCE_COLOR=1 TERM=xterm-256colorRich styled the surrounding panel markup and the assertion failed even when the field itself had been sanitised correctly — the test was measuring the ambient terminal, not the code under test. Each now pins its console withConsole(force_terminal=False);no_color=Truealone is not sufficient, because it suppresses colour without pinning tty detection, which is the exact bugtest_replay.pycarried. No assertion was weakened, narrowed, or routed through an ANSI strip: all seven keep the broad"\x1b" not inform, and six are demonstrably falsifiable by a real sanitiser regression.tests/test_cli_help_assertions_are_ansi_safe.pyrecords why this class is left to per-test pinning rather than added to the repo-wide ratchet — telling "asserts on console output" from "asserts on a pure-function return" is about an even split, and a guard that false-positives on half of what it sees gets deleted. Test-only; no runtime behaviour changes.soup serve --backend sglangno longer sends the engine a doubled BOS (#890 by @abdulwaarith0 in #995). The same defect #867 fixed for vLLM, on the backend that was split out becausesgl.Runtime.generateonly ever posts{"text": prompt}and the server then tokenizes that string withadd_special_tokens=True; neither the request norServerArgshas a flag to turn that off. Measured end to end on SGLang 0.5.9 withunsloth/Llama-3.2-1B-Instruct, whose Llama-3 template renders{{ bos_token }}(id 128000): for the string Soup sent, the engine ran on[128000, 128000, ...]over 37 ids (meta_info.prompt_tokens37), against the 36 idsapply_chat_template(tokenize=True)returns. The same/generateendpoint takesinput_idsand uses them verbatim, so the backend now posts the ids built with no tokenizer special tokens to it whenever the chat template rendered the prompt, which reproduced the 36-id encoding exactly ([128000, 128006],prompt_tokens36). Both the streaming and the non-streaming route go through it, and the response is the same{"text", "meta_info"}dict the app already parsed, sousage,finish_reasonand the streamed chunks are unchanged apart fromusage.prompt_tokensdropping by one on such templates. As with #867, a model whose template renders no BOS but whose tokenizer post-processor would add one now receives zero, matching Soup's training path. Models without a chat template, and templates that fail to render, still go throughRuntime.generateas the legacy string and are tokenized by the engine exactly as before. The MII backend is fixed separately in #891. Packaging note: the ids POST is now bounded by a 300s timeout and issued throughhttpx, sohttpx>=0.24.0joins the[sglang]extra; anyone installingsoup-cli[sglang]picks it up automatically.soup serve --backend miino longer sends the engine a doubled BOS (#891 by @abdulwaarith0 in #996). The same defect #867 fixed for vLLM, on the backend that was split out becauseMIIPipeline.__call__takes strings only and encodes each one throughHFTokenizer.encode, which istokenizer.encode(input, return_tensors="pt")with no way to say the special tokens are already there. On a BOS-prepending tokenizer whose template renders{{ bos_token }}(unsloth/Llama-3.2-1B-Instruct,bos_token_id128000), that isadd_special_tokens=Trueover a string that already starts with the BOS text, so the prompt tokenizes to[128000, 128000, ...], 37 ids, against the 36 idsapply_chat_template(tokenize=True)returns. MII's seam isModelConfig.tokenizer:mii.pipeline(tokenizer=...)accepts a wrapper, whichload_tokenizerwraps once more and callsencode(input, return_tensors="pt")on, with the prompt object the pipeline was given, untouched.soup servenow builds the pipeline with such a wrapper around the same tokenizer whose chat template it applies, and hands a templated prompt over as aTokenizedPrompt: still astr, so the pipeline's contract holds, but carrying the ids built with no tokenizer special tokens, which the wrapper returns verbatim, so the engine runs on the 36-id encoding ([128000, 128006, ...]). If the engine ever reports a prompt length other than the one Soup encoded, the server logs a warning naming both rather than doubling in silence. As with #867, a model whose template renders no BOS but whose tokenizer post-processor would add one now receives zero, matching Soup's training path. Models without a chat template, and templates that fail to render, still hand the pipeline a plain string, which the wrapper encodes with the very call MII'sHFTokenizermakes, so the engine tokenizes it exactly as before; a pipeline built without a tokenizer keeps MII loading its own, as it did. The wrapper is covered against MII 0.3.3's realHFTokenizer.encodecontract intests/test_issue891_mii_engine_bos.py. The SGLang backend is fixed separately in #890.docs/commands.mdanddocs/training.mddocumented foursoup train --minillm-*flags that were never wired as CLI options, so copying the documented command raisedNo such optionand exit 2 (#979 by @SID-6921 in #998). Only--minillm-on-policywas ever a real flag;minillm_enabled,minillm_teacher_mix_ratio,minillm_pretrain_anchor_weightandminillm_pretrain_anchor_patharesoup.yamlfields with no CLI counterpart. Both doc sites now show the YAML config instead. Added a guard (tests/test_issue979_documented_cli_flags_exist.py) that walkssoup train's live Typer tree and fails when a doc names a--flagwith no counterpart, so the next one fails a test instead of a support ticket. Building it turned up eight more pre-existing mismatches in unrelated subsystems (ULD strategy, RL mid-epoch checkpointing, echo-trap detection); those are tracked separately in #997 rather than folded into this fix.The distill
_DistillTrainertest harness now derivescompute_lossclosure bindings from the factory andTrainingConfigdefaults, and fails with the missing name if a new free variable cannot be resolved. Nested functions such as_token_weighted_accumulationare not reported as missing, and the recompiled class is compiled under<compiled _DistillTrainer>so aNameErrorcan no longer masquerade asdistill.py(#990 by @jagadeepmamidi in #1008).A LoRA target on
lm_headorembed_tokenscrashed the first forward pass on an untied streamed checkpoint (#1012 by @AmirF194 in #1019).build_streamed_modelrunsget_peft_modelbeforeinstall_streaming, so peft wraps the boundary module in a LoRA tuner layer first and exposes itsweightas a read-only property;StreamedLargeLayer's substitution used a plain{"weight": ...}functional_call, which cannot set that property, and raisedAttributeErrorat the first forward._large_layer_weight_param_namenow resolves the same indirection_layer_name_mapalready resolves for decoder-layer projections, substituting ontobase_layer.weightwhen the boundary module is peft-wrapped. Verified bit-exact (torch.equal, not an approximate tolerance) against a resident LoRA-on-head model for logits, loss and every LoRA gradient. This fixes the forward crash only; saving and resuming a checkpoint with a LoRA-wrapped head or embedding still needs separate work, tracked on #1012.training.stream_layersnow refuses a LoRA target onlm_head/embed_tokensat config parse time (#1012 by @AmirF194 in #1019). The forward pass above is correct, butsave_pretrained()/load_adapter()on the streamed head/embedding boundary is not implemented yet, so a run with that config trained for hours and died at its firstsave_steps. Drop the boundary module fromtraining.lora.target_modules, or train withoutstream_layers, until the save/resume follow-up lands.Apply
data.chat_templateto conversational PPO prompts, passanswerthrough in PPOmessagesformat, and rejectdata.chat_templateon non-chat tasksprm,asr,moe_lora_routing, andunlearn(#936 by @kok-o in #1021).training.nvfp4andtraining.fp8_attentionare now applied ontask: sftpost-LoRA (#800 by @kok-o in #1022).SFTTrainerWrapper._apply_quantization_awarenow delegates toapply_v028_speed_memorywithskip_cut_ce=Trueso NVFP4, FP8 attention projections, and FP8 training now reach their converters on SFT runs without double-patching Cut-CE. Both flags are also gated at schema time against non-v028 tasks and the MLX backend to prevent silent no-ops.Unify
SoupTrainerCallbackconstruction across all trainers and reject unsupported monitoring on tasks without callbacks (#802).training.loss_spike_recoveryandtraining.grad_accum_auto_tunepreviously reached the live callback only intrainer/sft.py; fifteen other trainers instantiatedSoupTrainerCallbackwith watchdog settings only, causing spike recovery hint generation and grad-accum advisory monitors to default toFalse. All trainers now build callback arguments throughsoup_callback_kwargs(tcfg, batch_size=..., output_dir=...)inmonitoring/callback.py. Tasks without a live callback (prm,moe_lora_routing,unlearn) explicitly refuseloss_watchdog,loss_spike_recovery, andgrad_accum_auto_tuneat config load time (#802 by @kok-o in #1023).Refuse symlinks on Windows when reading the draft registry and writing the audit log (#820). On Windows,
os.O_NOFOLLOWdoes not exist and evaluates to0, allowingos.opento follow symbolic links. Three sites under$HOME(~/.soup/) relied onO_NOFOLLOWalone withoutcwdcontainment:utils/draft.py:_read_registry,utils/draft.py:_registry_lock, andutils/audit_log.py:append_audit_event. As a result, symlinks placed atdrafts.jsoncould leak arbitrary JSON into speculative decoding, a symlink atdrafts.json.lockwould follow and create its target viaO_CREAT, andappend_audit_eventwould append audit logs to arbitrary symlinked targets. Introducedutils.paths.open_no_follow(path, flags, mode)which combineslstatinspection (detectingS_ISLNKand Windows reparse points) with post-openfstatcross-validation on Windows to detect a TOCTOU swap. Migrated the three vulnerable sites, enabled symlink tests on Windows when privileges permit, and added a repo-wide ratchet preventing new baregetattr(os, "O_NOFOLLOW", 0)usages (#820 by @kok-o in #1024).Warn once when distillation teacher produces persistently non-finite logits. Consecutive non-finite distillation terms are accumulated directly on device using asynchronous PyTorch operations with zero host synchronization on the per-step path. A single diagnostic warning is emitted naming the teacher model, the consecutive step count, and likely root causes (unsupported dtype underflow, mismatched tokenizer, or overly small distillation temperature). Transient non-finite steps (such as standard GradScaler overflow recovery) reset the on-device counter with no sync and emit no warning (#887 by @kok-o in #1026).
data.remove_unused_columnsnow defaults tofalse, and an explicittrueloads with a warning and is ignored; Soup v0.77 will refuse it (#759 by @Srinivasan8888 in #1027). No trainer reads the field. The trainers that set the Hugging Face argument passFalse, so a custom collator still receives the columns the model'sforward()does not name, and the old declared default oftruemisdescribed every run.trueis not refused yet, becausesoup autopilotwrites the whole config, so every file it generated on the old default containsremove_unused_columns: true; those keep loading, and no run changes behaviour, because the effective value was alwaysFalse. Loading one printsdata.remove_unused_columns: true has no effect and is ignored, once per load; delete the line before v0.77. The field is not deleted, since an unknown key is now a hard error.The config-field guard no longer reports a broken checkout when one directory is reached by two spellings (#988 by @Srinivasan8888 in #1029). Its scan/import check compared paths with
==, so on macOS, where the filesystem is case-insensitive but keeps the case you type, running the test file by a lowercased path failed six tests with a message telling you to fixPYTHONPATHor reinstall. The check now compares the two directories withos.path.samefile, which asks whether they are the same directory on disk. It checks both exist first, so a missing tree is still reported instead of crashing, and genuinely different trees are still caught. Test-only.A streamed save that writes an adapter holding nothing now raises instead of reporting success (#1011 by @dchaudhari7177 in #1031). Twice a
stream_layers: truerun wrote anadapter_model.safetensorsthat reloads as nothing (v0.72.0's.inner.keys, #1005's 0-tensor file under peft 0.21) and the only signal was aUserWarningat a later load. After the final save and on everysave_stepscheckpoint, the streamed trainers (sft,dpo,orpo,simpo,kto) now read the file's keys back and raise, naming the file and both counts, when it holds no LoRA tensors, a different number than the model trained, or any key carrying the wrapper's.inner.segment. Resident runs are unaffected.soup doctor --configno longer prints an unverified source line fortraining.use_fsdp2_compileon MLX (#1001 by @dchaudhari7177 in #1032). The reason citedcommands/train.py:1235, which nothing pinned and which had already drifted once. It now namesvalidate_fsdp2_compile_config, whichsoup trainruns before resolving the trainer, and a test fails if any backend-support reason cites afile.py:NNNlocation again.soup serve --auto-quantnow refuses instead of selecting a format from timer noise (#816 by @Amix29 in #1035). The pre-bind path timed three calls to a constant-returning stub, called the fastest few hundred nanoseconds a quality-gated evaluation, and could then force AWQ, GPTQ, or FP8 on a plain checkpoint through vLLM. The flag now exits 2 with an explicit explanation until Soup can load and evaluate each candidate; vLLM receivesquantization=Noneand therefore follows the checkpoint's own metadata. Transformers and SGLang no longer print a selection they never apply, and the help plus serving guide state the current boundary.SFT gradient-checkpointing tiers now make the distinct trade-offs they print (#796 by @Amix29 in #1036).
mediumreaches Transformers asevery_n_layers=2instead of checkpointing every block, whileselectivewraps only each decoder block's direct attention child and leaves HuggingFace full-model checkpointing off, including after PEFT enables it for a k-bit model. Consequently,quantization: 4bitplusgradient_checkpointing: selectivecan now use more peak memory than before because it no longer inherits PEFT's full-block checkpointing; this is the intended selective speed/memory trade-off. Architectures without a selectable attention child report and apply a full fallback instead of claiming an inactive tier. A four-layer tiny Llama now measures 4/4 checkpointed blocks underfull, 2/4 undermedium, and four attention-only wrappers underselective, with a real CPU training step and finite non-zero gradient norm in the selective arm.Wire PiSSA, OLoRA, LoftQ, and VeRA through the shared PEFT constructor used by every Transformers trainer. Unsupported backend combinations and pre-quantized PiSSA or integrated LoftQ runs now fail at config load instead of silently training a different adapter (#794 by @Amix29 in #1037).
A
format: pre_tokenizedrun now accepts the cachesoup data preprocesswrote (#1038 by @Srinivasan8888 in #1040). Preprocess hashed the source format into the cache key while the training gate hashedpre_tokenized, so every cache was refused with "cache hash mismatch; re-runsoup data preprocess", and re-running changed nothing; a listdata.trainraisedValueErrorinstead. The gate now recomputes the key from the format preprocess recorded, and both sides hash a list the same way. A cache built from a different tokenizer,max_length, dataset or interleave strategy is still refused, and existing cache keys are unchanged.metadata.jsonalso records the installed Soup version instead of a hardcoded0.53.7.The FP8 and NVFP4 gate tests give the same answer on every machine (#834 by @Srinivasan8888 in #1041). Two tests called the real hardware and torchao gates with a placeholder model, so they passed only while a gate refused on the test machine; on a Hopper or Blackwell card with torchao installed they failed with
AttributeError. Each gate now has its own test with the gate patched both ways and a small real module, and the FP8 torchao check uses a model without attention projections so removing that gate can no longer go unnoticed.Tasks that never apply
training.quantizationno longer record a quantised run, and an explicit4bitor8biton them is deprecated; Soup v0.77 will refuse it (#795 by @Srinivasan8888 in #1043).distill,classifier,reranker,cross_encoder,prm,moe_lora_routing,unlearnandasrnever quantise their models, yet accepted the default4bit, stored it in the run's config, and let the VRAM pre-flight budget them as QLoRA jobs. For these tasks the default changes: an unsetquantizationnow resolves tonone. An explicit4bit,8bitorload_in_8bit: truestill loads, because every config Soup dumped on the old default carriesquantization: 4bitliterally, but it prints a warning naming the task, resolves tonone, and will be refused in v0.77; setquantization: none. A Quant Menu value or a 4-bit-only setting such asbnb_4bit_quant_storageis refused at config load, naming the task. PRM also refuses alorablock it never reads (lora.r: 0is allowed), and the pre-flight budgets PRM, and the classifier family and ASR without their LoRA opt-in, as full fine-tunes.docs/training.mdlists which tasks honour the field.Both FP8 training paths ask one hardware gate, with torch's own floors, and a request the card cannot run stops the run (#835 by @Srinivasan8888 in #1044).
training.fp8_attentionrefused Ada (sm_89) cards such as the RTX 40-series, L4 and L40S, whichtorch._scaled_mmsupports, whiletraining.quantization_aware: fp8converted the model on any card, Ampere included. Both now require compute capability 8.9 or newer. Therowwiserecipes also need a torch that dispatches their kernel on the card (Ada 2.7, RTX 50-series 2.8, 11.x 2.10), are never available on Windows, and before torch 2.11 need a CUDA 12 build. When FP8 is requested and cannot run, the run now stops at setup on every trainer that reaches the converter (the eleven that callapply_v028_speed_memory, plus SFT's own call), naming the reason, instead of warning and training without FP8.training.gradient_checkpointing: falsenow reaches the model on a 4-bit/8-bit/mxfp4 run (#797 by @AmirF194 in #1045). All 15prepare_model_for_kbit_training(...)call sites across the trainers called it with nouse_gradient_checkpointingkwarg, so peft's ownTruedefault silently overrode the config;transformers.Traineronly ever enables checkpointing and has no disable branch, so nothing downstream could turn it back off once k-bit prep had switched it on. Every call site now threads the resolved value through. This covers the k-bit-prep mechanism only; the TRL-config-default and unsloth mechanisms the issue also names are unchanged.The AcceleratorError-as-OOM probe classification is now reachable and covered without a GPU (#1002 by @Areen-09 in #1047). #989 classified an OOM arriving as
AcceleratorErroras a result (oom=True) rather than an instrument failure (failed=True), but its only coverage sat under@requires_cudaand skipped across all CI cells without a GPU runner.measure_step_peak_bytesnow delegates its failure mapping to a pure helper_classify_probe_exceptionwhose three-outcome mapping is reachable without CUDA while keeping the early return intact, and a dedicated test suite pins the classifier, the outcome mapping, and predicate identity withbatch_probe._is_cuda_oom.The
soup draftpanel now safely renders target and draft model names (#1052 by @MugdhoAI in #1053). Model names containing Rich markup or terminal control bytes are escaped and stripped before being displayed.Resuming a LoRA adapter targeting a streamed
lm_head/embed_tokenssilently dropped the head's own tensors (#1048 by @AmirF194 in #1055).StreamedLargeLayer._redirect_canonical_weightonly redirected the literal"weight"key intoinner., which is all a bareinnerever contributes.get_peft_model()runs beforeinstall_streaming()(#1012), so a LoRA target on one of these boundary modules makesinnera peft tuner layer whose own keys arelora_A/lora_B(orlora_embedding_A/lora_embedding_B) instead, none of which matched the old check. Generalized the hook to redirect every key under its own prefix, same as the decoder-layer version already does. Thesave_pretrained()meta-tensor crash and the parse-time refusal gaps from the same issue are separate, still open.An untied streamed checkpoint takes a preference step again (#1049 by @dchaudhari7177 in #1061).
embed_tokensand an untiedlm_headshare one device slot, and the streamed layer passed autograd a view of it, so the reference forward ofdpoorktorefilled those bytes in place and the policy backward raisedone of the variables needed for gradient computation has been modified by an inplace operation. The projection now takes a private copy of its weight, one head-sized buffer per step, only fordpoandkto(the losses whose reference forward runs before the backward) and only when the slot is shared;sft,orpoandsimpokeep the view (that scoping came from @VanshSharmaPES in #1174), and so does the embedding, because its backward reads the indices rather than the weight values. The VRAM pre-flight charges the copy as a second large-layer slot for exactly the runs that take it, so its prediction still never under-counts. Llama-3-8B-class checkpoints are untied, so this is the common shape; tied runs andsftwere unaffected.Refuse a stale-green PR merge without forcing a rebase on every push to main (#1017 by @jagadeepmamidi in #1071). A
pull_requestrun pinsrefs/pull/N/mergeat creation, so GitHub's green marks describe amainthat may no longer exist (strictis off).merge-freshnessrebuilds the merge withgit merge-tree --write-tree(no write to the fork), fails on ruff F821 or a conflict (the #763NameErrorclass), reports lag as a neutral check rather than a rebase demand, and re-posts that check on every push tomain. It does not fliprequired_status_checks.strictand does not re-run the 13-job matrix;gh run rerunstill cannot refresh a frozen merge SHA, which CONTRIBUTING now says.training.moe_loradid nothing on DPO, KTO, ORPO, SimPO and GRPO, so 15 shipped MoE recipes trained something other than what they said (#798 by @Srinivasan8888 in #1074). Onlysft.py,pretrain.pyandstream_setup.pyever read the flag; the 8 DPO and 7 GRPO recipes that set it got attention-only LoRA. Target selection now lives in one helper,utils/moe.resolve_moe_lora_targets, which every one of those trainers calls — as dosftandpretrain, which each kept their own copy of the block.training.moe_expert_quantandtraining.train_router_onlyare refused outsidesft/tts, and a non-defaulttraining.moe_aux_loss_coeffoutsidesft/tts/pretrain, because no other task's trainer reads them.All 15 of those recipes could not attach LoRA at all on transformers 5.x, not merely train the wrong modules (#798 by @Srinivasan8888 in #1074).
resolve_lora_target_modulesreturnsNoneforqwen3_moe, and peft then refuses outright:No target_modules passed but also no target_parameters found.The MoE-aware targets close it for the architectures these recipes name; the general gap for other MoE architectures is #1070.moe_lorawith a non-zerolora.dropoutnow stops at the LoRA attach instead of failing inside peft (#798 by @Srinivasan8888 in #1074). transformers 5.x keeps a Qwen3-MoE's experts as fused 3-D parameters, which peft adapts throughlora.ParamWrapper, and that refuses dropout:lora.ParamWrapper does not work with lora_dropout != 0.At the schema default of0.05the attach failed on every task, includingsft. The check runs against the loaded model rather than at config load, because whether the experts are fused depends on the checkpoint and the transformers version; a dense base is untouched. All 31 shipped MoE recipes now pinlora.dropout: 0.0, and a guard fails if a new one arrives without it. The check asks peft whether it would route these targets onto fused parameters, rather than scanning the model: Mixtral and MiniMax have the same fused 3-D expert parameters but peft attaches to them without complaint, and refusing those would be wrong. Measured on tiny random stand-ins on CPU, never on a real 30B+ checkpoint.moe_loradoes not reach every MoE family, and two shipped recipes are still attention-only (#798 by @Srinivasan8888 in #1074). Expert adapters attach onqwen3_moe(11 recipes),deepseek_v3(6) andglm4_moe(3); onminimaxthey do not, sominimax-m3-sftandminimax-m3-dpostill train attention-only LoRA after this change, because peft has no v4->v5 conversion mapping for that model type and its experts are never targeted. The sevenkimi-k2.xand twomistral-large-3recipes are untested rather than known-good: no tiny stand-in for those configs builds in the installed transformers. Per-architecture target resolution is #1070.run_costno longer prices an RTX A4000 as an A40 or anNVIDIA A100-SXM4-80GBas a 40 GB A100. GPU-name patterns are now word-boundary anchored and ordered most-specific first, and the rate table gains L4, A10G, H200, B200, RTX A4000/A4500, RTX 6000 Ada, RTX PRO 6000 and the RTX 40/50-series (#831 by @rishcx in #1081). Cost figures and thesoup loopbudget gate change for these cards: L4, H200, B200, RTX 40/50-series and RTX PRO 6000 used to be unpriced and counted as $0, so their budget gate never fired; they now have a rate and it does. Names the old unanchored patterns mispriced (A100X,A1000andRTX A1000 Laptop GPUas an A100,T400as a T4) now resolve to unknown instead. Laptop parts (for exampleRTX 4090 Laptop GPU) still price as the desktop card, which overstates a laptop run. Existing rows keep their labels and rates.soup data preprocessignoreddata.chat_templateand left it out of the cache key (#1067 by @Blizzeq in #1082). Rows were rendered with the tokenizer's shipped template whatever the config said, and two configs differing only in the template shared one cache. Preprocess now renders with the resolved template and hashes it into the key. Thepre_tokenizedtraining config must name the same template, and a cache written before this change is refused withcache hash mismatch; re-runsoup data preprocessto rebuild it.GRPO training now skips non-finite optimizer steps instead of corrupting adapter weights. A pre-step gradient watchdog detects NaN or Inf gradients and zeros them before
optimizer.step(), so the update is skipped and the adapter is preserved. The skipped-step count and fraction are persisted intrainer_state.jsonand reported bysoup adapters audit. (#342 by @harshitthek in #1084)Reject and report
loss_watchdog,loss_spike_recovery, andgrad_accum_auto_tuneonbackend: mlx(#1069). All three callback-monitoring settings are now rejected at schema validation time namingbackend='mlx'with their exact technical reasons (watchdog stop control, spike recovery watchdog requirement, and unified memory VRAM pressure), declared as rejected inbackend_support.py's_MLX_SFTregistry table, and warned about at runtime intrainer/mlx_sft.py::_check_unsupported(#1069 by @kok-o in #1088).The
commands.mdcoverage guard now matches against the command column rather than whole entry rows (#1046 by @Areen-09 in #1094). Previously, mentioning a command inside another row's description prose allowed commands whose own rows were deleted (such assoup serve,soup export,soup train, orsoup bom emit) to falsely count as documented. The guard now isolates the command column before description whitespace and inline comments, anchors leaf matching, updates token lookahead to include dots (gate.v2), and explicitly documentssoup loop resume,soup lock check, andsoup steer listin their own rows.The
soup data preprocesscache kept a leading BOS the live path never trains on (#876 by @Srinivasan8888 in #1096). The cache tokenised the rendered chat template withadd_special_tokens=Trueand dropped a leading BOS only when it was doubled, so adata.chat_templatepreset on a tokenizer whose post-processor prepends one cached a BOS the live path (add_special_tokens=False) does not have. It now drops exactly the BOS the post-processor contributed, keeping the template's own.A truncated row no longer gains a stop token the live path lacks (#876 by @Srinivasan8888 in #1096). #791's "append the training EOS when the row is under
max_length" check ran after a BOS was removed, so a row truncated to the budget read as one token short of it. Truncation is decided before the strip now.The preprocess cache schema is
v5(#876 by @Srinivasan8888 in #1096). Re-runningsoup data preprocesswrites a new cache directory and orphans the old one; pointing training at the old directory is refused withcache hash mismatch.target_modules: autonow resolves on every MoE architecture Soup ships a recipe for (#1070 by @Srinivasan8888 in #1102). peft has no LoRA mapping for any of them, soautoreached it asNoneand the attach failed withNo target_modules passed but also no target_parameters found. Each now resolves to the attention projections that architecture defines —qwen3_moe,deepseek_v3,deepseek_v4,glm_moe_dsa,kimi_k25,gpt_oss,minimax_m2andminimax_m3_vl, pluskimi_k2(the text tower of K2.5/K2.6 and the outer type of Kimi-K2-Thinking) and, defensively,minimax_m3_vl_text(MiniMax-M3's text tower when its config loads without the wrapper; no shipped base reports it) — including the three MoE recipes that set no MoE flag (gpt-oss-20b-sft,gpt-oss-120b-sft,minimax-m2-sft). MiniMax-M3 resolves to a regex scoped to its language tower, so a text fine-tune no longer adapts the vision encoder. DeepSeek-OCR is deliberately not covered: its architecture needstrust_remote_codeto build, so its module names cannot be measured.target_modules: autoon an architecture neither Soup nor peft maps is refused by name (#1070 by @Srinivasan8888 in #1102). peft refused these before too, asNo target_modules passed, naming neither the architecture nor the fix; the message now names themodel_typeand the fix. It is decided at the last step before the attach, sotraining.moe_lora: trueandtraining.lora.target_parametersare both honoured first. Scope, stated so it is not a surprise: it reaches dense architectures too (phi3,smollm3,lfm2and several vision and audio architectures in the catalogue), every one of which already failed at attach with peft's message.soup loop canarynow routes keyed transformers requests at the configured traffic share and automatically clears a regressing canary (#815 by @akkupratap323 in #1104). Serve records rollout-scoped generation outcomes for stable and canary buckets;soup loop watchevaluates those observations and persists a rollback after aMAJORverdict, while explicit adapter selection keeps its existing behavior and the 100% activation endpoint still applies to requests without a conversation key.soup adapters auditnow verifies the effective MLX learning rate (#942 by @Amix29 in #1109). MLX runs comparetraining.lrwith the optimizer plan's recordedpeak_lr; transformers/PEFT reports that the effective learning rate was not checked rather than presenting missing optimizer metadata as a clean audit.The training docs no longer advertise eight nonexistent
soup trainflags (#997 by @Amix29 in #1110). ULD, RL checkpointing, echo-trap settings, andoutputnow use their real YAML fields, and the live Typer-tree guard has no remaining grandfathered exceptions for these spellings.mini_tool_callno longer sits at a 1.000 ceiling on the measured strong reference (#405 by @Amix29 in #1111). The 40-row suite now mixes tool selection with exactNO_TOOLdecisions and semantically close distractors; the measured reference moves to 0.850 versus 0.050 on the weak model, and the bundled scorer provenance revision changes so pre-change baselines are not silently compared on the new scale.Spark-TTS and OuteTTS raw-audio live-codec configs now fail closed at config parse with the real upstream compatibility reason; the unrunnable Spark recipe is removed, while Oute keeps the pre-encoded
chatmlrecipe path (#265 by @Amix29 in #1113).HuggingFace repo IDs are now preserved without Windows path separator corruption in
soup inferandsoup bench(#1097 by @Guts1005 in #1118). When resolving a HuggingFace repo identifier on Windows, the repo string was previously converted to apathlib.Path, turning forward slashes into backslashes and causing HuggingFace Hub to reject the repository name withHFValidationErrorbefore loading. The repo ID is now maintained as a string and forwarded byte-identical to model and tokenizer loaders across all platforms.soup data preprocesscached rows with nolabels, so apre_tokenizedrun trained on the prompt (#1054). The cache carried only{input_ids, attention_mask}; with nolabelscolumn TRL's collator falls back tolabels = input_ids, so the same rows optimised a different objective depending only on whether they had been preprocessed first — full-sequence loss instead of the live path's assistant-only mask, with a normal-looking loss curve and no error. Cached rows now carry alabelscolumn built by the samedata/loss_mask.pyhelpers the live path uses, honouringdata.train_on_responses_only,data.train_on_messages_with_train_field,data.mask_historyandtraining.train_on_eot. That loss-mask mode is now a cache-key input (tokenizer schemav6), so a cache built under one masking setting is refused by name under another, and a cache with nolabelsis refused outright rather than silently trained on.soup trainalso reports the real cached row count for apre_tokenizedconfig instead of0(#1054 by @AaronProbha18 in #1120).Fix invalid Hugging Face base model IDs in shipped recipes (#1122). Updated
devstral-sfttomistralai/Devstral-Small-2507,glm-4.6-sfttozai-org/GLM-4.6,granite-4-sfttoibm-granite/granite-4.0-h-tiny-base,internvl-3-5-sfttoOpenGVLab/InternVL3_5-8B,kimi-k2-sfttomoonshotai/Kimi-K2-Base,mistral-medium-3-5-sfttomistralai/Mistral-Medium-3.5-128B, andvoxtral-sfttomistralai/Voxtral-Mini-3B-2507. Removedcogito-v2-sftandgemma3-9b-sftbecause no compatible replacement base exists for their shipped recipe definitions. (#1122 by @parnika0110 in #1132)Reject structurally unsafe regex patterns in custom eval, diagnose format probes, and Data Recipe validators before matching, replacing the diagnostic canary search with a field-named error (#1137 by @taljeon in #1142).
training.moe_loranow works onipo,bco,reward_model,ppo,embeddingandonline_dpo(#1099 by @Srinivasan8888 in #1148). These tasks acceptedmoe_lora: trueand never read it, so the flag silently did nothing. Each now picks the expert targets through the same helper as the other MoE-wired trainers, including itslora.dropoutrefusal on fused experts. Onembeddingit applies only withlora.r >= 1. The flag still loads unread onclassifier,reranker,cross_encoder,distill,unlearn,asr,moe_lora_routingandprm; #1151 tracks wiring the first five and refusing the rest.Preserve absent gradient norms as
NULLin training records and charts instead of fabricating0.0or carrying the previous measurement into a new step (#1147 by @taljeon in #1149).ExecutionManager.execute()now opens the MCP run log withopen_no_follow(#820), refusing a file symlink planted at.soup/mcp-runs/<run_id>.logbefore the child spawns; previously the plain append followed the link and redirected every byte the child wrote to its target (#1138 by @xiechimon in #1150).target_modules: autonow resolves on GLM-4.6 and Granite 4.0 (#1122 by @MakazhanAlpamys in #1164). Both recipes named a Hugging Face id that 404s, so the #1102 completeness ratchet recorded their bases as unresolvable and never asked whether their architectures were covered. Repointing them (#1132) makesglm4_moeandgranitemoehybridreal entries in the catalogue, and neither was inMOE_TEXT_LORA_TARGETS. Both now resolve to the four attention projections enumerated from the architecture on the meta device, the same policy as every other row of that table.Granite 4.0 says at setup that
autoadapts only part of the decoder (#1122 by @MakazhanAlpamys in #1164).granitemoehybridis a hybrid: ongranite-4.0-h-tiny-baseonly 4 of the 40 decoder layers define aself_attnat all — the other 36 are Mamba-2 blocks — so the attention-projection entry reaches a tenth of the model. Training now prints a yellowPartial LoRA coverage:line naming the model type, the counted fraction and what was skipped (the Mamba projections, the shared-expert linears and the fused routed experts), instead of leaving a surprisingly small adapter to be discovered at merge time. The fraction is counted off the loaded model, so a different Granite depth reports its own numbers.Recipe size labels are the base's total parameter count, and a test now keeps them so. Nine labels were wrong and moved:
llama4-scout-17b-sft,-dpo,-grpoandllama4-scout-tools17B -> 109B,deepseek-v3-7b-sft7B -> 685B,voxtral-sft3B -> 4.7B,glm-4.6-sft9B -> 357B,glm-5-sft9B -> 754B,minimax-m2-sft9B -> 230B.tests/test_recipe_sizes.pycompares every recipe's label with the Hub's recorded parameter count (0.70x to 1.15x) and names a recipe whose base has no count unless it is excluded with a reason (#1161 by @kok-o in #1168).Added modality axis to
backend_supportand madesoup doctor --configreport unreadtrain_on_responses_onlyandmask_historyon vision and audio configs (#1156 by @kok-o in #1169). Multimodal vision and audio collators supervise all sequence tokens rather than performing assistant-only masking.backend_support.REGISTRYpreviously only tracked(task, backend)pairs with no modality axis, causingsoup doctor --configto print an affirmative all-clear on multimodal configs that set these flags.REGISTRYnow accepts(task, backend, modality)and flags both settings as ignored when running vision or audio SFT.Refuse
.soupand.soup/mcp-runsdirectory symlinks and pre-planted hard links during MCP execution (#1158 by @kok-o in #1171). Following #1138,soup mcp servenow refuses execution if.soupor.soup/mcp-runsis a symbolic link or junction, or if a pre-planted hard link exists at the run log path, closing directory redirection and hard link overwrite attacks (with a residual TOCTOU window if an attacker has concurrent write access inside.soup).An unregistered
data.chat_templatename is refused before any model loads (#1175 by @Blizzeq).soup trainchecks the name right after loading the config, so a typo exits 1 at once instead of after the model has been downloaded and loaded.soup train --dry-runandsoup train --cloudnow refuse it as well, where they used to exit 0. Bothsoup trainandsoup data preprocessprintInvalid data.chat_templatewith the known template names.QuEST artifacts now store a content-derived identity for local base models instead of the machine's absolute path in route metadata, while keeping format-v1 sidecars readable and resumable under their original path-based rule. (#1176 by @Shutaru)
training.moe_loranow works onclassifier,reranker,cross_encoder,distillandunlearn, and is refused where no trainer reads it (#1151 by @Srinivasan8888 in #1179). These tasks acceptedmoe_lora: trueand never read it.distillandunlearn, and the classifier family withclassifier_lora: true, now pick the expert targets through the same helper as every other MoE-wired trainer. Refused at config load, naming the task: the classifier family withoutclassifier_lora: trueandlora.r > 0(that trainer then full-fine-tunes and builds no adapter),asr(it trains only Whisper, which has no experts),moe_lora_routing(it builds no LoRA adapter) andprm(it fine-tunes every base parameter). It is also refused onbackend: unslothfor every task, and ontask: sftwithmodality: visionoraudio, whose setup skips the MoE step. Every path now either reads the flag or refuses it, exceptbackend: mlx, where it loads andsoup doctorreports it as ignored. Becausetraining.train_router_onlyandtraining.moe_expert_quantrequiremoe_lora: true, a config setting either onbackend: unslothnow fails to load too, with the unsloth message; the unsloth path reads neither.soup data preprocessstops on a chat row it cannot tokenize, instead of dropping it (#1180 by @Srinivasan8888 in #1182). This is a behaviour change: a dataset thatsoup data preprocessused to cache partially, with exit 0, now makes it exit 1. A row the chat template rejected (for exampleConversation roles must alternateon a Llama-2- or Gemma-style template), an emptymessageslist, a row rendered as empty text, or a tokenizer error (on a pretrain text row too, which live pretraining refuses the same way) was skipped without a word, and apre_tokenizedrun then trained on fewer rows than the live run of the samesoup.yaml, which stops on those rows. The command now stops before writing anything and names the row, numbered from 1 as live training numbers it, with the start of its first message. Fix or remove the row and re-run. A tokenizer with no chat template is reported once and points atdata.chat_template. The rows are checked when the command builds a cache: if one with the same key already exists it stops atTarget already exists(exit 0) without reading them, so rebuild a cache written before this change with--yes.Training reports the dataset rows its format converter drops (#1181 by @Srinivasan8888 in #1183). A row the converter rejects, such as a misspelled
messageskey, is still dropped so one bad line does not abort a load, but it is no longer silent. Every load now printsWarning: N of M rows droppedwith the first row's index and the converter's reason, and for a local file thesoup data validatecommand that lists them all. The count agrees withsoup data validate. A clean file prints nothing.task: distillwithbackend: unslothis refused at config load instead of silently training on transformers (#1186 by @chiliec in #1193). The distill trainer never reads the backend, so the unsloth request was dropped without a word.validate_distill_compatnow rejects it the same way it rejectsbackend: mlx, with a message namingbackend: transformers, andsoup doctor --configreports the config as unloadable (exit 2) instead of clean.The friendly error handler no longer parses exception text as Rich markup, so install hints keep their extra (
soup-cli[audio]was printed assoup-cli), and control bytes in an exception message are stripped before they reach the terminal (#1200 by @chiliec in #1202).distill_mode: sequenceruns on transformers 5: teacher generation no longer fails with a bareAttributeErroron the first prompt, because the chat-template encoding is read as a mapping and its attention mask is passed togenerate(#1238 by @MugdhoAI in #1243).soup train --find-lrno longer fails when the training set is smaller than--find-lr-steps, or blames--find-lr-outputfor it (#1189 by @mohdUwaish59 in #1244). The live sweep takes one row per step; it stopped when the rows ran out (or when the loss went non-finite), but the report was written against the full schedule, so the run ended withInvalid --find-lr-output: lrs and losses must have equal length. The sweep is now sized to the rows over the same--find-lr-start→--find-lr-endrange, with a line naming both numbers, and a divergence mid-sweep writes a report for the steps before it. The recommendation needs 4 points, so--find-lr-stepsbelow 4 is rejected up front (the minimum was 2, which always failed), and a training set with fewer than 4 rows is refused before the model loads instead of falling back to a synthetic curve. An empty training set is refused the same way; it previously wrote a synthetic report and exited 0.Invalid --find-lr-outputis now reported only for a problem with the output path.Web UI returns 401 rather than 500 when Authorization header contains non-ASCII bytes (#1195 by @Areen-09 in #1246). Starlette decodes header bytes >= 0x80 as Latin-1 strings;
secrets.compare_digestrejected non-ASCII strings withTypeError, causing an unhandled 500 that printed a full traceback on the server console for every such request and dropped security headers. Token comparisons now encode to UTF-8 bytes before comparing, aligning withsoup serveandsoup mcp serve.task: prmon CUDA trains every base parameter and the reward head on fp32 master weights (#1235 in #1250). PRM loaded the base and the head in bf16 whenever the device was"cuda", with no fp32 copy for the optimizer. Since v0.74.0 that made the first step fail withmat1 and mat2 must have the same dtype(outside DeepSpeed), because the bf16 head ran on the fp32 outputs of the autocast forward. Before that the base trained in pure bf16, where most AdamW steps at PRM learning rates round away: an RMSNorm weight at 1.0 never moves atlr1e-5 or 2e-5. PRM now loads fp32 on every device string,"cuda"and"cuda:0"alike, and leaves bf16 to autocast (fp16 plus the GradScaler on pre-Ampere cards, which now receives fp32 gradients). With the default optimizer, expect about 16 bytes per parameter before activations instead of about 8, which is what the VRAM pre-flight already budgeted fortask: prm(the pre-flight runs only whenbatch_sizeis an integer, not with the defaultauto). Without DeepSpeed the saved checkpoint is now fp32, and loading needs about twice the host RAM of a bf16 load. Under DeepSpeed only host RAM at load grows, since the engine still trains and saves in 16-bit on its own fp32 master copy. Thetraining.quantizationwarning and refusal for the eight tasks that ignore that field now say the trainer never quantises the base, not that it loads "at checkpoint precision".Reject
training.quantization_aware: questwithtraining.auto_mixed_precision: trueat config load, preventing auto-selection of unmeasured FP16 for QuEST (#1245 by @Shutaru in #1252).The
soup data preprocesscache key now covers every setting that changes the cached rows (#1127 by @Blizzeq in #1253).data.val_split,data.replay,data.replay_ratio,data.replay_seed, the streaming settings (data.streaming,data.buffer_size), the media directories (data.image_dir,data.audio_dir) and the top-leveltaskall decide which rows are cached, but none was part of the key, so apre_tokenizedrun under a different value reused the cache without a mismatch. They are now key inputs (tokenizer schemav7), and every otherdatafield is declared out of the key with a reason, so a new field cannot be left unclassified. Caches fromv6and earlier are refused; re-runsoup data preprocess.soup migrate --from axolotlno longer turnsrl: orpo,rl: ipo, andrl: simpointotask: sftwith no warning (#1254). The rl-to-task mapping only covereddpo,kto,grpo, andgdpo; every otherrlvalue fell through tosftsilently, so a preference config loaded with the wrong task and the wrong data format and later failed training with an error that never namedrl. The mapping now coversorpo,ipo, andsimpo, each resolving to its matching Soup task and thedpodata format. Anrlvalue with no Soup task, such as axolotl'sebft, is refused instead of silently falling back totask: sft, naming the unknown value and listing the supported ones.soup migrateexits non-zero for it instead of writing a config (#1214 by @jamalkamaladdin in #1254).training.loraplus_lr_ratiowithoptimizer: apollo_adamw,lomooradalomo, or withuse_galore, is now refused at config load (#1210 by @mohdUwaish59 in #1255). LoRA+ asks transformers for its optimizer without the model, and transformers can only build those three optimizers from one, so such a config passedsoup train's config load and failed only after the base model had been downloaded and loaded. LoRA+ with GaLore was likewise refused only at setup. Both now fail at load with a message that names the two fields.soup shrinksaves pruned Qwen2, Qwen3, Qwen2-MoE and SmolLM3 models (#1241 by @drakeo338 in #1256). The prune dropped the decoder layers but leftlayer_types(and SmolLM3'sno_rope_layers) at the old length, so transformers 5 refused to save the model after scoring and the run exited 1 with an emptymodel/directory. Those lists are now sliced with the same kept indices as the layers, so a mixed sliding/full-attention list keeps the right entry for each surviving layer. A config that places its MoE layers by layer index (a non-defaultmlp_only_layers,decoder_sparse_step,moe_layersorinterleave_moe_layer_step) is refused before the prune, since the kept layers would reload with their MoE and dense MLPs on the wrong layers.soup autopilotno longer writes a config thatsoup trainrefuses on an Ampere or newer GPU (#1212 by @wanjinhao1 in #1257). For the SFT goals, autopilot turned onuse_flash_attnanduse_ligeron every GPU with compute capability 8.0 or higher without checking thatflash-attnorliger-kernelwas installed. Neither ships with a standard install, sosoup train, even with--dry-run, refused the file autopilot had just written. Each flag is now turned on only when its package is installed, checked the same waysoup trainchecks it, and when the GPU qualifies but a package is missing, the Autopilot Decisions panel names it and the command that installs it. The Liger refusal insoup trainalso told users to runpip install "soup-cli", because[liger]was read as Rich markup; it now printspip install "soup-cli[liger]".soup data doctorgains--mask-history/--no-mask-history(default off), so the compat report and the--show-maskpreview train only the last assistant turn, as adata.mask_history: truerun does (#1251 by @wanjinhao1 in #1259).--mask-historyis refused together with--no-train-on-responses-onlyor--train-on-messages-with-train-field, the same combinationssoup.yamlrefuses.--baseline registry://now takes each benchmark's score and provenance from its newest eval row, so re-measuring a baseline replaces the stale score instead of only silencing the scorer-revision warning; the warning names the benchmarks that are still stale, and ties oncreated_atresolve by insertion order (#1224 by @anyingiit in #1260).soup data forge --judge-providerwrote an empty-answer row for every failed judge call and still reported "synth complete" (#1221 by @anyingiit in #1261). The forge judge was built withoutraise_on_error, so a connection error, HTTP 404/429/500 or malformed response came back as empty text;score_uncertaintyrates an empty reply 1.0, so no--uncertainty-thresholdcould prune it, and the command exited 0 with a dataset that teaches a model to answer with nothing. The forge judge is now built withraise_on_error=True(assoup data recipehas been since #969); a call that fails, returns a malformed reply or returns empty text is counted and never becomes a row, whatever the threshold. The summary reportsN of M judge calls failedwith the provider, endpoint and first error; a run whose judge produced no usable row exits 1 without writing the dataset or provenance files, and a partial outage keeps the successful rows under a yellow "synth complete with judge failures" panel.401/403digits inside an error message no longer masquerade as an auth failure (#1267 by @sup3dev in #1268). A row number (train row 401:), tensor size ((4013)), checkpoint path (checkpoint-4010), date, or hash slice printed "Authentication failed." / "Access denied." and hid the real error. Auth status is now read from the exception object (response.status_code,.code, walking wrapped__cause__/__context__), with a text fallback only for explicit HTTP phrasing (401 Unauthorized,HTTP Error 401,Error code: 401); the bare-digit ERROR_MAP entries are gone. Genuine 401/403 from requests, urllib and huggingface_hub now get the specific hint instead of the generic "HTTP request failed.", and gated-repo hints still win forGatedRepoError/RepositoryNotFoundError.Show QuEST task, backend and modality errors before the auto mixed-precision refusal when a config violates both, and link the BF16 evidence boundary from the auto mixed-precision guide (#1276 by @Shutaru).
A refused
pre_tokenizedcache now says why (#1272 by @Blizzeq in #1281).soup data preprocessrecords the key generation and the row-set inputs inmetadata.json. When training refuses a cache, the message says it predates row-set keying (#1127), or names thedata.*fields whose values differ, such asdata.val_split, instead of showing two bare hashes.docs/data.mdlists the fields apre_tokenizedconfig must keep as they were at preprocess time.Online DPO no longer trains a pair its judge could not rank as "the second completion wins" (#1225 in #1282). Soup's pairwise judge answers
-1for a tie, a failed or unreadable judge call, or a verdict that changes when the two completions are swapped. trl 0.29'sOnlineDPOTrainerbuildsmask = rank == 0, so every such pair trained as a win for the second completion, and the loss was averaged over all pairs. With the judge server down, a whole run learned arbitrary labels and finished normally. On thetraining.online_dpo_judgepath those pairs now add exactly zero gradient, and the loss is the mean over the pairs the judge did rank in each batch (each micro-batch, under gradient accumulation).judge/invalid_ratelogs the share it could not rank, with a WARNING the first time and on every step in which it ranked nothing. The loggedlossandtrain_lossaverage only the batches that ranked a pair, and a logging window with none logs noloss, so an unranked stretch can no longer produce the run's lowest or final loss. For every trainer, the run tracker now stores no loss, rather than a fabricated 0.0, for a log that arrives before the run's first measured loss (later ones still repeat the last measured loss). That includes the row of the MLX SFT trainer's first validation pass, which runs before any training step, and the live panel showsLoss: -instead of0.0000until a loss is measured.soup runs cleantherefore picks the checkpoint to keep whole among the checkpoints: the one with the lowest loss recorded at its step, or the latest one, with a note, when none has a recorded loss. Before, it could strip or delete every checkpoint of a run shorter thanlogging_steps, or of any run whose lowest loss was logged between two checkpoints. A step that ranked nothing still takes an optimizer step, so AdamW momentum and weight decay keep moving the weights and the learning-rate schedule advances. That drift is bounded: training stops with an error naming the judge once 32 pairs in a row went unranked, and a shorter run that ranked no pair fails the same way instead of saving an adapter. trl'slogps/*andrewards/*statistics now describe the ranked pairs only. With clean 0/1 verdicts, training is bit-identical to before, and thereward_modelpath is unchanged.Hardware-fit gate correctly infers Hub model sizes using boundary-safe regex and MoE parsing (#1198 by @Areen-09 in #1285).
utils/gpu.py:model_size_from_namepreviously relied on a static scan of hardcoded substring markers and fell back to 7.0B, causing VRAM under-prediction up to ~58x–96x on large models (such as 72B, 32B, 14B, 405B, 675B) and substring collision bugs (e.g.,11bmatching1b,17b/27bmatching7b,0.8bmatching8b, and8x7bmatching7b). Model sizes are now dynamically parsed with boundary-safe regex, active-parameter tokens (-aNNb) are filtered to retain total parameters, and MoE architectures (8x7B,8x22B, genericNxMb, and Llama 417B-16E) are supported. The same parameter estimate also drivesbatch_size: auto,soup cost, andsoup profile, while Hub IDs carrying no size marker (DeepSeek-V3,Kimi-K2,GLM-5,MiniMax-M2) still fall back to 7.0B, an under-estimate for those models.soup registry diff,soup eval compareandsoup eval leaderboardnow use the newest eval row per benchmark instead of the oldest (#1270 by @Ru0k3 in #1291). Each built its score map by overwriting in the order the experiment tracker returns rows, which is newest first, so the oldest score won and a re-measurement was ignored: with run A logging 0.30 and then 0.45, and run B logging 0.50 and then 0.40,soup registry diffandsoup eval comparereported +0.20 and no regression instead of the -0.05 regression. All three commands and the eval gate'sregistry://baseline now pick rows through one helper,soup_cli.eval.results.newest_eval_rows(newestcreated_atfirst, then insertion order for equal timestamps); the leaderboard keeps the newest row for each model and benchmark.Parquet datasets preserve nested Python values (#1218 by @commit-msuyog in #1292). Parquet loading now uses PyArrow to preserve nested lists and dictionaries instead of converting them to NumPy arrays, preventing valid rows from being dropped by format validation.
GRPO variant runs (
grpo_variantother thanstandard) with a non-zerogrpo_betalog theklmetric again, as trl's stock loss does: the masked batch mean of the per-token KL, averaged over the accepted tokens forrft(#1263 by @MohammadHijjawi97 in #1293).Fix DPO / IPO / preference beta schedule callback crashes and refuse un-wired reference regeneration (#1229 by @kok-o in #1294). DPO, IPO, and
preferenceruns configured withdpo_beta_schedulecrashed at step 0 withAttributeError: 'BetaScheduleCallback' object has no attribute 'on_epoch_begin'because custom callbacks were plain Python classes rather than Hugging FaceTrainerCallbacksubclasses. Converted callbacks to lazily subclassTrainerCallbackwithout introducing import-time dependencies on heavy ML packages, verified beta annealing behavior under IPO and preference training, and enforced a load-time refusal fordpo_ref_regen_epochsas the feature is not wired yet.task: embeddingwithembedding_loss: contrastive(the default) no longer trains on one-row batches, where the loss and gradient are exactly 0 and the adapter never moves:batch_size: 1is refused at config load,batch_size: autoresolves to at least 2 (also whentripletfalls back to contrastive for rows withoutnegative), the last partial batch of an epoch is dropped, and a batch larger than the training set is lowered to its size. Auto batch sizing and the hardware-fit estimate also stop guessing 7B for BGE / E5 / GTE / BERT / RoBERTa / MiniLM checkpoints named-small/-base/-large(#1234 by @kok-o in #1296).Refuse rows with no causal-loss target at setup in
task: distillwith human-facing row number anddata.max_length, matching SFT behavior, and return finite 0.0 CE loss incompute_lossif a microbatch contains zero targets (#1242 by @kok-o in #1298).soup expectscans all dataset formats for PII, token length, and refusals (#1231 by @kok-o in #1300). The dataset expectation gate previously only inspected a few field names (text,instruction,output, stringmessages), missing ShareGPT, DPO (string and conversationalchosen/rejected), KTO (completion), Alpacainput/system, multimodal content parts, tool calling, and sentence embedding fields. The text extractors now cover all format signatures declared inFORMAT_SIGNATURES, verify assistant-side refusal patterns across formats, and fail closed when rows lack extractable text.Tokenize non-Latin scripts, non-spaced languages, and digits in
soup diagnoseandlive_eval(#1233 by @kok-o in #1301). The previous regex discarded non-ASCII characters and standalone digits, causing healthy models responding in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew, Hindi, CJK, etc.) to falsely read as MAJOR mode collapse, and causing forgetting probes usingtoken_f1to evaluate to zero retention. Tokenization now performs stream-safe NFC Unicode normalization (capping combining mark runs at 30 to prevent quadratic reordering), preserves non-Latin word boundaries, generates bigrams for non-spaced scripts (CJK, Kana, Thai), and ignores mark-only tokens.token_f1splits joiners (-and_) to preserve baseline English scores. Downstream consumers (soup data topics, canary discovery, and eval design) now tokenize digits and non-Latin words. When--tokenizeris passed, the memorization probe computes sub-word bigram overlap via tuples, avoiding key collisions.soup migrateexplicitly writesquantization: noneand emitslora.r: 0for full fine-tuning (#1215 by @kok-o in #1302). Previously, unquantized Axolotl, LLaMA-Factory, and Unsloth configurations omitted the quantization field, causing Soup to default them to 4-bit QLoRA upon loading, while full fine-tunes defaulted to LoRAr=64. Migrators now explicitly setquantization: nonefor unquantized sources, support HQQ/EETQ methods, map full fine-tunes tolora.r: 0, refuse full fine-tuning for unsupported tasks (e.g. DPO/PT/RM), resolve notebook variable assignments in Unsloth, and validate output withload_config_from_stringbefore writing.soup train --resumeproperly resumestask: prmandtask: moe_lora_routingrather than retraining from step 0 and overwriting checkpoints (#1220). BothPRMTrainerWrapperandMoleRoutingTrainerWrappernow accept and forwardresume_from_checkpointtoself.trainer.train(). In MoLE training, the trained router gate weights (mole_gate.pt) are now saved into each periodiccheckpoint-Ndirectory via_save_checkpoint, and restored on resume via_load_from_checkpointalongside adapter re-freezing prior to continuing optimization, ensuring step counts, optimizer momentum, and gate weights continue consistently. Tasks that do not support resume (such astask: unlearn) cleanly exit non-zero on--resume/--hf-resumewith a descriptive message naming the task (#1220 by @kok-o in #1303).training.packing: truepreserves assistant-only loss mask during SFT instead of training prompt/system tokens (#1236). In TRL 0.29.1, dataset packing drops the unpreservedlabelscolumn and falls back to training all input tokens except the first. SFT datasets now ensure anassistant_maskscolumn alongsidelabels, which TRL packs alongsideinput_idsand uses to mask prompt/system tokens with-100during collation (and prevents label dropping underuse_liger: true). Pre-tokenized caches carryinglabelsare also augmented withassistant_masksautomatically (#1236 by @kok-o in #1304).Fix classifier, reranker, and cross_encoder data loading and row validation (#1219 by @kok-o in #1306). Sequence classification tasks (
classifier,reranker,cross_encoder) were losing dataset columns such aslabel,text_a,text_b,question, andanswerduring dataset normalization becausepreserve_source_columnswas restricted totask == "grpo", causing crashes withTypeError: label must be int / str / list, got NoneTypeafter model loading. Centralized source column retention intask_preserves_source_columnsshared acrosssoup trainandsoup sweep, added task-scopedcross_encoderformat handling fortext_a/text_bandquestion/answerpairs, gateddata.format: cross_encodertotask: cross_encoderat schema validation, and added upfront dataset label and shape validation that fails immediately at data load and on--dry-runwith specific row indices before model weights are loaded.soup uiopens and prints a login URL that already carries the auth token (#1194 by @Faisal-Fayaz in #1307). The first browser tab washttp://host:portwith no?token=, so every/api/*route except/api/healthreturned 401 and the dashboard renderedUnauthorized. The startup panel andwebbrowser.opennow usehttp://host:port/?token=<token>, including under--no-browser.app.jsalready reads that query parameter, and on a loopback bind the server exchanges it for an HttpOnly session cookie (#1191).training.expand_layersis refused at config load unlesstraining.quantizationisnone(#1237 by @chiliec in #1308). Under the default4bit, the "zeroed" residual projections of the appended LLaMA Pro blocks decoded to-absmaxinstead of 0 (NF4 code 0 is-1.0), so the new blocks were not identity, and a positivefreeze_trainable_layerscrashed at setup on4bitand8bitbecause torch cannot setrequires_gradon integer weights. The error now names both fields and asks forquantization: none.rope_scaling_type: llama3on a checkpoint that already ships Llama 3.1 RoPE scaling now composes with that scaling instead of replacing it (#1239 in #1311). Llama-3.1 checkpoints shipmax_position_embeddings: 131072with a nativellama3block (factor 8 over an original 8192 tokens). Extending one to 262144 tokens rebuilt the block as factor 2 over 131072, so 35 of the 64 frequency pairs of an 8B head rotated up to 8x faster than in pretraining, and the run printed only "Long-context enabled".llama3on a nativellama3block now keeps the checkpoint'soriginal_max_position_embeddings,low_freq_factorandhigh_freq_factor, and multiplies itsfactorbydata.max_length / max_position_embeddings(factor 16 over 8192 in that example), so no pair rotates faster than in the checkpoint. New refusals: extending a checkpoint whose RoPE block is already scaled in any other way is refused before the model is built, with an error that names the checkpoint'srope_typeandfactor. That coverslinear,dynamicoryarnon a nativellama3block, and anyrope_scaling_typeon a nativeyarn,longrope,linearordynamicblock.rope_scaling_type: longropeis now refused at config load, in any spelling. Its per-dimension factors exist only on checkpoints already scaled with LongRoPE, and extending those is refused, so it cannot extend any checkpoint; the error names the types that extend a checkpoint without RoPE scaling (linear,dynamic,yarnorllama3). Fine-tuning a LongRoPE checkpoint such as Phi-3-mini-128k at its native length, withrope_scaling_typeunset, is unchanged. Adata.max_lengthat or below the checkpoint'smax_position_embeddingsstill changes nothing, and checkpoints without RoPE scaling (Llama 2, Mistral, the 8k Llama 3) extend exactly as before.task: sftandtask: pretrain, the two tasks that apply RoPE scaling, both get the new behaviour.task: moe_lora_routingon CUDA trains its router gate on an fp32 master weight (#1266 in #1312). The gate is the only tensor MoLE trains, and on the device"cuda"it was cast to bf16 with no fp32 copy for the optimizer, so its gradient and AdamW moments were bf16 as well and most AdamW steps rounded away: after 8 steps at the defaultlr(2e-5), 82.6-91.4% of the gate's weights had never moved."cuda:0"kept it fp32. The gate is now fp32 on every device string and is created on the base model's device, so it moves with the base; the frozen base and the task LoRAs load as before.mole_gate.ptis saved in fp32 (4 x hidden x adaptersbytes), andsoup serve --moleloads it and a bf16 file from an earlier run alike. A gate trained on CUDA before this fix kept most of its initial weights, so retrain it.GRPO's
accuracyand verifiablemathrewards now read the gold answer the way they read the completion (#1226 in #1316). An Alpacaoutputor a ShareGPT/ChatML reference turn such as6*7=42\n#### 42was compared raw, so every correct completion scored 0.0, every group had zero reward variance and GRPO got no signal, with no warning. Both sides now go through one parser: the text after the last####, the last\boxed{}(a space before the brace and nested braces allowed), or the text afterThe answer is/Answer:(also**Answer**:, and on the next line), whose number is read from its own clause, soThe answer is 41 apples, not 42.is 41. An answer phrase that names more than one distinct value is a hedge and states no answer:The answer is either 41 or 42.scores 0.0 in both rewards whatever the gold, and a gold written that way is refused (the same value twice, as in42 (i.e. 42.0), is not a hedge). Parenthetical justifications belong to the clause and hedge it (The answer is 42 (6*7=42).), while a connective ends the clause, soThe answer is 42 because 6*7=42.andThe answer is 42, because 6*7=42.both read 42; a time or a ratio such as3:45or1:1,000is one answer, compared as text.$,\(...\), trailing punctuation, thousands separators such as1,000and1{,}000, and the Unicode minus are normalised. A numeric gold is compared by value; any other gold, such as\frac{14}{3}, is compared as normalised text (case, whitespace,\left/\rightand\dfracignored), now bymathas well, so a dataset that mixes numeric and LaTeX golds, such as MATH-500, loads undermath, and validation prints how many golds are compared as text.accuracyno longer pays 0.5 when the gold merely appears in the completion (that paid#### 420for a gold of 42) and scores only 0.0 or 1.0; a completion with no marked answer is scored by its last number, soSix times seven is 42., and alsoNot 42., now score 1.0 instead of 0.5.math's 0.6 near-miss band is unchanged. A row whose gold states no single final answer, such as a multi-line reference with no marked answer, a hedge, or a#### Solutionheading over a worked solution (a####line that is not a number and has more text after it), is refused before generation with its split, row and field and a count of the other such rows. PPO'saccuracyandmathrewards use the same scoring, but PPO's data is not validated this way (on the supported trl, PPO scores rollouts with its reward model).soup data canary insertwrote chatml canaries into every dataset, so on any other format the loader dropped all of them andcheckcould only report OK (#1216 by @abdulwaarith0 in #1317).insertnow writes the canaries in the dataset's own format (--format, defaultauto): alpaca, sharegpt and chatml, with the carrier as the prompt and the secret as the trained response. Every other format is refused before anything is written, naming the format, as is an explicit--formatthat is not the file's own. The manifest records the format; a manifest without one reads as chatml, andcheckrefuses a format it does not know rather than scoring it as chat.-o canaried.jsonnow writes a JSON array that loads back, and suffixes the loader would misread are refused. (#1216 by @abdulwaarith0 in #1317)format: tool-callingdropped every rowsoup agent synthwrites, andsoup train --dry-runstill said "Ready to train!" on 0 rows (#1217 by @abdulwaarith0 in #1321). The converter now reads calls where agent data puts them: assistant turns keep their owntool_callsandtoolturns theirtool_call_id, in source order, so a trajectory is no longer rewritten as "answer, then call". A top-leveltool_callslist still works as the single-call form.toolsis optional,format: autodetects{messages, tools}as tool-calling, and synthesised rows carry the schema of the tool they call.soup trainnow exits 1, on--dry-runand on a real run, when the data loads zero training rows, naming the format and the first dropped row's reason. (#1217 by @abdulwaarith0 in #1321)training.moe_lorawithbackend: unslothis refused with a reason that is true for the task (#1264 by @Srinivasan8888 in #1323). The refusal said "unsloth attaches its own fixed attention targets" on every task, but ten tasks have no unsloth setup at all (reward_model,prm,classifier,reranker,cross_encoder,distill,unlearn,moe_lora_routing,online_dpo,asr): there it isbackend: unsloththat goes unapplied, and seven of them now say so. The other three (asr,prm,moe_lora_routing), andtask: sftwithmodality: visionoraudio, now get their own refusal first, which is the same on every backend, so switching tobackend: transformersno longer surfaces a second, different refusal for the same config. Nothing that loaded before stops loading.GRPO's
accuracyand verifiablemathrewards now match a LaTeX matrix however its rows are spaced (#1226, follow-up to #1316 in #1332). Answer normalisation read the second backslash of a row break followed by a space (\\) as the LaTeX control space\and deleted it, so a gold such as MATH-500's\begin{pmatrix} -2 \\ -14 \\ -7 \end{pmatrix}scored 1.0 only against a completion that spaced its rows the same way, and 0.0 against the same matrix written\begin{pmatrix}-2\\-14\\-7\end{pmatrix}. A row break is now kept whole, while\,,\!,\;,\:and\are still dropped. After an answer phrase (The answer is $...$.), a LaTeX environment such as a matrix is now one answer, like(3, 4): its entries used to count as separate values, so a matrix whose entries stood apart read as a hedge and scored 0.0 (and a gold written that way was refused), while one with a single stand-alone entry was read as that number and could pay a numeric gold equal to it. All 7 MATH-500 golds with a space after a row break are matrices, and each now matches its other spacings; across the 9,792 GSM8K and MATH-500 gold texts, only those 7 answers and their 7 solutions read differently, and no numeric reading changes.soup train --find-lrrefuses instead of writing a synthetic LR curve when the live sweep cannot run (#1203 reported by @mohdUwaish59, fixed by @umran666 in #1334). When the config did not load (a YAML syntax error) or the sweep could not run (a base model id that does not exist, a model or dataset that cannot be loaded, or an install withouttorch), the command printed one yellow line, wrote a report whoserecommended_lrcame from the LR schedule alone, and exited 0, so two different nonexistent ids reported the identical number. It now names the cause, exits 1 and writes no report. The refusals added in #1189, for fewer than 4 rows or a loss that turns non-finite before 4 steps have run, are unchanged.soup eval behavior --base-modelnow scores each live generation by whether it refuses, not by whether it contains the probe's oracle label, so a healthy model no longer reads MAJOR and a regression no longer reads delta 0.00.elephantandsycevalare refused with--base-modelbefore a model loads (exit3) and point to--evidence, andlooks_like_refusalrecognisesI'm unableandI'm not able. The same classifier scores themini_safetyandmini_over_refusalsuites ofsoup eval gateandsoup ship, the refusal probe ofsoup diagnoseandsoup expect'sexpect_no_refusal_pattern, so those now count the contracted forms as refusals;BUNDLED_SCORER_REVISIONmoves to 3, so a--baselinerecorded before this change warns (#1228 by @developerjillur in #1335).soup doctor --configno longer reports a setting written in its off position (#1330 by @Yi-111-a in #1336). A config that wroteloss_watchdog: falseoruse_liger: falseonbackend: mlxwas listed as ignored or rejected, so a config written out in full bysoup autopilotgot 15 rows where the same values written minimally got none. A setting is now reported only when its value is switched on;seed: 0still counts as set.Keep the default email PII fallback linear for long near-miss local parts under the existing 50 KB per-row scan cap, covering
soup expect,soup data piiandsoup data score(#1329 by @taljeon in #1360).Email snippets now follow the bounded local-part match, so leading punctuation and punctuation-adjacent addresses may produce different snippet spans; row-level detection remains unchanged (#1329 by @taljeon in #1360).
Refuse incompatible pre-tokenized data transformations. Soup data preprocessing does not apply non-text modality transforms (vision/audio/audio_out),
data.add_new_tokens,data.new_special_tokens, ordata.prompt_strategyto cached ids, so those combinations are refused at config load (#1273 by @Samearth17 in #1363)The eval gate's MCQ scorer read
\boxed{B.}and\boxed{**B**}as C on a completion that echoes the option list, while the reward parser read B (#1347 by @putramkti in #1365). The letter tier kept its own boxed regex after #1316 replaced the parser's with a normalising, brace-balancing read, so the two readers disagreed on the same box: the right answer scored wrong and the echoed option scored right. Both now read the boxes through one shared scan inutils/final_answer.py, and a test pins the tier to it so they cannot drift apart again.The friendly error reads a 401/403 out of the whole exception chain, not just the outer message (#1361 by @putramkti in #1366). The status-phrase check (
401 Client Error,HTTP Error 403, ...) searched only the outermost message, so a wrapper that says nothing about status, such asRuntimeError("Failed to load adapter")around a text-only401 Client Error: Unauthorized, printed the wrapper's message instead ofAuthentication failed.It now reads every exception in the__cause__/__context__chain, outermost first, so a 403 wrapper of a text-only 401 still reads as the 403 it says. An exception whose own__str__raises no longer turns the friendly error into a second traceback either.A failed Anthropic judge/provider call blamed
--judge-base-url/--base-urlfor the failure, though Anthropic ignores that flag entirely (#1340 by @SID-6921 in #1370).soup data forge --judge-provider anthropic --judge-base-url <x>andsoup data recipe --provider anthropic --base-url <x>both named<x>in the terminal error and, for the recipe path, in.checkpoint.json'sfailed_reason-- but the API key comes fromANTHROPIC_API_KEYonly, and every call actually goes tohttps://api.anthropic.com/v1/messagesregardless of whatbase_urlwas, so the message pointed at a host that was never contacted. Both callers route through one shared label helper, which now returns the real Anthropic origin whenever the provider is Anthropic, ignoringbase_url;ollama/vllmlabels are unaffected since those providers do forwardbase_urlto the request.accuracyandverifiable/mathscored 0.0 on common MATH answer spellings (#1346 by @AmirF194 in #1371).normalize_answerhad no rule for\text{}/\textbf{}/\mathrm{}/\mbox{}wrappers, degree marks, or the compact\frac12/\frac1{2}/\frac{1}2spellings, and a phrase answer's one-letter variable prefix (x = 7) was read on the completion side but not the reference side. A correct completion in any of those spellings scored 0.0 against a gold in the other. All four are now handled the same way on both sides, though the variable split runs in the two parsers rather than inside the shared normaliser. A variable name is kept rather than discarded: when both sides name one and the names differ (x = 3againsty = 3), the pair still scores 0.0, since a directrix or an asymptote's variable is part of its answer, and its case is part of its name (x = 5does not match\boxed{X = 5}). A gold with a variable prefix and nothing after it (\boxed{x =},x =) states no answer and is now refused before generation (it was compared as the textx =). A side effect of running the degree and\text{}rewrites before the hedge scan: a clause such as30^\circ, 60^\circnow reads as two distinct values and hedges, where it previously read as one: a gold such asThe answer is 30^\circ, 60^\circ.is now refused before generation, asThe answer is 30, 60.already is. Units are still not stripped and nothing is evaluated (\frac{1}{2}does not equal0.5); seedocs/training.md.backend: unslothis refused at load on the tasks that have no unsloth setup (#1357 by @Srinivasan8888 in #1372).reward_model,prm,classifier,reranker,cross_encoder,unlearnandmoe_lora_routingacceptedbackend: unsloth, but their trainers never read it: the run trained on plain transformers whilesoup trainprinted "unsloth (fast mode)", andsoup doctor --configexited 0. The config now stops at load, naming the task and pointing tobackend: transformers(and namingtraining.unsloth_bnb_4bittoo when it is set, since only unsloth reads it), andsoup doctor --configexits 2. This is a breaking change for such a config, but a one-line one: those runs always trained on plain transformers, so switching tobackend: transformerschanges nothing about what trains.distill,asrandonline_dpokeep their own existing refusals, and the docs no longer say unsloth works with every task.soup abno longer rejects H0 on two identical arms when--alpha+--betais 1 or more (#1339 by @qwist1233-cpu in #1378). Each rate was checked in (0, 1) on its own, but the reject boundarylog((1 - beta) / alpha)stays above the accept boundarylog(beta / (1 - alpha))only while their sum is below 1. Past that there is nocontinueband left, and because reject is tested first, every verdict in the gap rejected:--alpha 0.6 --beta 0.5reportedreject_h0withdirection worseon two byte-identical arms, and--beta 0.95(a power of 0.95 typed as beta) rejected a true H0 in a quarter of single looks. #1378 refused those pairs, but that refusal does not ship: #1575 retires--betain the same release. The reject boundary is nowlog(1 / alpha)and the accept rule no longer usesbeta, so there are no boundaries left to cross: two identical arms never reject, and a passed--betais ignored with a warning.soup abnames--effect-sizeinstead of exiting on a bareOverflowError(#1339 by @qwist1233-cpu in #1378). The drift term squares the standardised effect--effect-size / pooled standard error, and**raisesOverflowErrorwhere*would return infinity, so a huge--effect-sizeended the run onError: OverflowError: (34, 'Result too large')with no flag named. The ratio is now bounded before it is squared, and the message carries the flag, the ratio and the standard error measured. Because it is the ratio that overflows, a near-constant metric column reached this at the default--effect-sizetoo, and is named as well.training.loraplus_lr_ratiois now refused at config load onbackend: mlx(#1324 by @wanjinhao1 in #1383). The MLX optimizer trains every LoRA tensor at one learning rate, so the ratio was accepted by validation,soup train --dry-runandsoup doctor --config, and then never applied. An MLX config that sets it and loaded before now fails with an error naming the field, the backend and the way out: remove the key, which trains exactly what those runs trained, or usebackend: transformers.transformersandunslothare unaffected.soup data canary insertspreads the canaries through the file instead of appending them, and records each canary's row in the manifest. The loader holds out the file's tail as validation (data.val_split, 0.1 by default), so from 135 rows on every appended canary was held out, no model trained on it, andsoup data canary checkcould only report OK.insertnow prints how many canaries the default split trains on (#1331 by @MohammadHijjawi97 in #1386).training.expand_layers(LLaMA Pro) is refused at config load outsidetask: sft/task: pretrainonbackend: transformerswithmodality: text(#1325 by @wanjinhao1 in #1390). Block expansion is applied only by those two trainers' text setup, so every other task,backend: unslothormlx, and SFT withmodality: visionoraudioaccepted the setting and trained the model without the new blocks, with no message.task: pretrainwith amodalityother thantext, andtask: sftwithmodality: audio_out, are refused too: those trainers ignore thatmodalityand did expand the model, so setmodality: textthere. The error names the task, backend and modality it got;task: sftandtask: pretrainonbackend: transformerswithmodality: textload as before.soup abnames a metric column whose spread a float cannot carry, instead of exiting on a bareZeroDivisionErrororOverflowError(#1384 by @MohammadHijjawi97 in #1392). A near-constant column (0and1e-161) has a positive pooled variance that still underflows to a standard error of0.0once divided by the row counts, and the z-statistic divided by it. A column spread past about1.3e154(0and1e200) overflowed while its variance was squared, and an arm whose values summed past the largest float gave an infinite mean and a silentcontinuewith ananlog-likelihood ratio. All three now exit 1 with a message that names the column, says whether its spread is below or above what the statistic can represent or its values are so large that an arm's sum overflows (an arm of1e308values), and says to rescale the metric.soup sweepbuilds the trainersoup trainbuilds, for every task and forbackend: mlx(#1213 by @putramkti in #1397). The command carried its ownif/elifchain overtask, last extended in v0.40.0. Ten values (distill,tts,classifier,reranker,cross_encoder,prm,unlearn,moe_lora_routing,online_dpo,asr) reached the SFT wrapper, andbackend: mlxwas never read at all, so an arm trained next-token cross-entropy while the run registry recorded the configured task and the sweep ranked those numbers as the configured objective. Both commands now resolve throughsoup_cli/trainer/dispatch.py::build_trainer, which refuses a task it has no wrapper for instead of falling back to SFT.Recipes that could not load their model are fixed or removed (#1145 by @Srinivasan8888 in #1398).
mistral-large-3-sftandmistral-large-3-dpoare removed:mistralaipublishes Mistral-Large-3 in the nativeparams.json+consolidated-*layout with noconfig.json, whichAutoConfigcannot read (docs/models.mdnames the Hugging Face-format conversion that exists on the Hub).minimax-m3-sftnow setsmodality: vision, so SFT builds its vision-language wrapper and LoRA stays on the language tower; its processor is custom code, so it needssoup train --trust-remote-code.minimax-m3-dpois removed, since the DPO trainer cannot load that wrapper (#1393). The catalog loses three recipes.training.freeze_trainable_layerswithouttraining.expand_layersis now refused at config load (#1396 by @wanjinhao1 in #1399). The field was accepted on its own but read by nothing: withoutexpand_layers,utils/block_expansion.pyreturned before looking at it, so a config that setfreeze_trainable_layers: 2to train only the top two layers trained the whole model (or every LoRA target), with no message. The error names both fields and the remedy (remove it, or setexpand_layers); the field description anddocs/peft-and-efficiency.mdnow say it applies only together withexpand_layers.The refusal for
training.citation_faithfulon a task other thansft/pretrainno longer says its span-mask runtime "ships in v0.62.1"; that runtime has run in the SFT trainer since v0.71.10. The streaming docs now state thatdpoandktohold a copy of an untied output head per step, which the VRAM pre-flight already charges (#1353 by @maximilianfeix in #1402).training.moe_aux_loss_coeffset to a non-default value ontask: sftwithmodality: visionoraudiois now refused at config load instead of loading and being ignored: only the text setup applies the MoE auxiliary loss (#1394 by @maximilianfeix in #1404).task: pporuns again on trl 0.29: the dataset handed to trl'sPPOTrainercarries onlyinput_ids/attention_mask, so its padding collator no longer crashes on theprompt_textstrings before the first step. A train set with fewer thanbatch_sizexgradient_accumulation_stepsx the number of processes rows is refused before any model loads, instead of hanging (#1391 by @maximilianfeix in #1410).A completed PPO run no longer reports "Loss: unavailable" (#1413 by @Ashutosh-Panda2004 in #1416).
summarize_training_lossnow takes the per-step loss key from the caller instead of only readingloss: the PPO trainer passesloss/policy_avg, which is what trl's PPO trainer (0.29) logs. A non-default key is recorded in the result asloss_key, so a PPO policy loss is never mistaken for an SFT cross-entropy. SFT/DPO summaries are unchanged.--fsdpnow reachestask: prmandtask: moe_lora_routing, and both--fsdpand--deepspeedreachtask: online_dpo; the wrappers stored them and never passed them to their trainer arguments (#1204 by @Oxygen56 in #1417; online_dpo half by @maximilianfeix in #1417).soup data clean --repair-jsonand--drop-invalid-jsonmissed nested tool calls and dropped valid documented-shape rows (#1338 by @abdulwaarith0 in #1453). Both flags read only a flatargumentskey on the row's top-leveltool_calls, so calls inside an assistant turn (whatsoup agent synthwrites, and the OpenAI fine-tuning shape) were never repaired or checked, and a valid{"function": {"arguments": ...}}call read as""and got its row dropped. They now read every call the tool-calling loader reads, in both shapes, and leave dict arguments alone. Atool-callingrow with no top-leveltool_callsno longer gets an empty list added, so the report stops counting it under "Format Normalization".A failed adapter merge or reward-verifier replace could destroy the only good copy on disk (#1450 by @drakeo338 in #1454).
merge_adapter_to_dense(shared bysoup shrinkandsoup draft distill) deletedout_dirbefore swapping the newly merged model in, so a failure between the two steps — a lock held on the staging dir, disk full — left neither the old nor the new model.soup reward synth -o reward.py --forcehad the same shape: it overwrote the existing verifier with the candidate before calibrating it, then deleted the file outright on refusal or error, destroying a previously working verifier.merge_adapter_to_densenow moves the previous artifact aside instead of deleting it, and restores it if the replacement does not make it all the way to accepted.soup reward synthinstead writes the candidate to a temp sibling and never touches-ountil it is accepted, so there is nothing to move aside or restore on a refusal or error — the previous verifier is simply never disturbed.soup tokenizer trainnow saves a complete byte-level BPE: a matchingByteLeveldecoder (sodecode(encode(x)) == x, no moreĠfor spaces), the 256-byte initial alphabet (so unseen characters never become<unk>), no hard-codedunk_token, and a--vocab-sizefloor that accounts for the byte alphabet plus special tokens (#1429 by @wanjinhao1 in #1455).--trust-remote-codeis now threaded through on the five paths that dropped it (#1449 by @drakeo338 in #1456).soup data downloadforwards the flag toload_datasetand only prints the "Remote Code Warning" panel when it is set.soup eval autoforwards it to bothbenchmark()(appended to the lm-evalmodel_argsfrom the resolved flag only, never fromadapter_config.json) andcustom(), which now passes it to thepipeline(...)load.soup train --find-lr's live sweep loads the tokenizer and model with the resolved flag instead of a hardcodedFalse.soup draft distillandsoup shrink(heal step) now append--trust-remote-codeto the childsoup trainargv when it was resolved true, instead of asking the user to re-run with a flag they already passed. Ondatasets>=4, which droppedtrust_remote_codesupport upstream and now just logs an error and ignores it,soup data download --trust-remote-codeis refused outright instead of silently forwarding a flag that would do nothing.soup eval benchmarkandsoup eval customgain their own--trust-remote-codeoption (off by default).soup expectfails closed on unreadable JSONL input (#1434 by @wanjinhao1 in #1457). The gate now reads the dataset asutf-8-siglike the training loader, so a UTF-8 BOM no longer hides the first row from PII and other checks. Malformed JSON lines and valid JSON that is not an object are refused as an input error (exit 3) with their line numbers instead of being silently skipped, and a file with no checkable rows is refused rather than passing withRows 0.soup cardnow shows the newest score of a benchmark that was evaluated more than once, not the oldest: the scorecard walked the tracker's newest-first rows and kept the last one, and rows with an identical timestamp now resolve to the later insert. It uses the same newest-row helper assoup registry diffandsoup eval compare(#1315 by @Ru0k3 in #1459).soup infer,soup chatandsoup diffnow honor--devicewhen loading models instead of always passingdevice_map="auto"andtorch_dtype=torch.float16tofrom_pretrained(#1443 by @drakeo338 in #1460).cpu(andmlxon this path) is pinned to the CPU infloat32;cuda:Nandmpsare pinned to that device; plaincudakeepsdevice_map="auto". GPUs keepfloat16. Values are case-insensitive, and an unknown--device(gpu,auto,cuda:0,1, ...) now raises an error instead of being printed and ignored.Default-path change: the CLIs always auto-detect a device when
--deviceis not given, so these rules apply without the flag too, and tosoup bench, which loads throughsoup infer. On a CPU-only host the model now loads infloat32pinned to the CPU instead of"auto"/float16, which doubles its RAM use (a 7B model needs about 28 GB instead of 14 GB) (#1443 by @drakeo338 in #1460).soup expectrefuses suite keys and argument names an expectation does not take, instead of running it on its defaults (#1428 by @Srinivasan8888 in #1461).parse_suite_specread onlynameandargsand dropped every other key, and the dispatcher read arguments withargs.get(key, default), so themin_tokens/max_tokensexample indocs/compliance.md(written besidenameinstead of underargs:) and a misspelled argument such asmax_tokenboth ranexpect_token_length_betweenon its defaults of 1 and 1,048,576 tokens and passed a 5000-token row with exit 0. A key besidenameor an argument the expectation does not take is now refused when the suite loads (exit 3, before the data file is opened), with the key named and either "belongs underargs:" or the nearest valid name;min_tokens/max_tokens/thresholdare validated at parse time too. Thedocs/compliance.mdexample moves to theargs:form. A suite that carried one of these keys now exits 3 on its next run, which is the point.mini_arithmeticno longer sits at a 1.000 ceiling (#1192 by @AaronProbha18 in #1462). The 40-row suite replaces 12 single-step rows with 16 multi-step and large-operand ones; Qwen2.5-1.5B-Instruct measures 0.800 against 0.375 on SmolLM2-135M-Instruct, and the bundled scorer provenance revision moves to 4 so pre-change baselines are not silently compared on the new scale.soup data augmentwrote rows with an empty rewritten field, andsoup distill-promptsilently dropped every failed row, both still reporting success (#1274 by @AaronProbha18 in #1463). Both commands built their LLM provider withoutraise_on_error, so a connection error, HTTP 404/429/500 or malformed response came back as empty text instead of raising —augmentrewrote a field to""and kept the row,distill-promptdropped the row and wrote an empty output file withRows: 0, and both exited 0. Same defect class as #1221 (fixed forsoup data forgein #1261): the provider is now built withraise_on_error=Truefor both commands, failures (a raised error or an empty/whitespace-only reply) are counted via the sharedForgeJudgeStats, and a row is never written with an empty or unrewritten field. A run where every call failed exits 1 without writing an output file; a partial outage keeps only the successful rows under a yellow "with provider failures" panel reportingN of M provider calls failedwith the endpoint and first error.soup compile, quantized GGUF export andtask: unlearnnow validate their destination before the expensive step (#1445 by @wanjinhao1 in #1469).soup compile -o <path outside cwd, or a symlink>used to run the whole optimizer and its LLM calls before refusing the path with an unhandledValueError(exit 1); it now refuses at plan time and exits 2, with the write-time check kept as the TOCTOU guard. A quantized GGUF export into a directory that did not exist yet crashed inmkdtempafter the adapter was merged, and the cleanup deleted the merged model; the parent directory is now created up front.task: unlearnused to load the policy and a frozen reference copy and run every NPO / SimNPO / RMU step before refusing anoutputoutside the working directory or a symlinked one;setup()now refuses first, before any model is loaded.soup compile --plan-onlynow also refuses such a destination (exit 2) instead of printing the plan and exiting 0.Machine-readable output from
soup audit-log tail --json,soup bom emit,soup eval leaderboard --format json|csvandsoup profile --jsonnow writes straight to stdout viatyper.echoinstead of the Rich console, so a piped record is no longer folded at 80 columns and[...]is kept verbatim instead of being parsed as markup (which previously crashed the command on a token like[/x]) (#1468 by @abhinyaay in #1470).soup data generate --template reasoningnow tells the generator how to end each solution, so the golds it writes are readable by GRPO'saccuracyandverifiablemath rewards:mathrows end with#### <number>on its own line,logicandcoderows with anAnswer: <answer>line. Before, the prompt asked only for "the final answer", and a gold such asSo there are 42 apples.makessoup trainrefuse the dataset before generation (#1349 by @Momoyeyu in #1471).Model output is now read by its structure. The LLM judge (
soup eval judge, the eval gate's judge task andsoup ship'sjudge_score) reads the reply's whole JSON object, so reasoning that quotes a nested object no longer scores every criterion at the minimum; criterion keys match case-insensitively,{"score": N}is accepted, and a missing or non-numeric criterion, or one scored under two spellings, raises an error naming it; a reply longer than 65,536 characters is refused unread.soup eval checklistkeywords andsoup eval behavior --evidenceoracle labels match when they start or end with punctuation (-5,(B),C++,100%), still as whole words on both sides. The judge node ofsoup data recipe --executereads the first word of the reply:OKkeeps the row,REJECTorNOT OKdrops it, and any other reply counts as a failed call; its default prompt asks for a one-word verdict (#1467 by @developerjillur in #1472).Autopilot no longer enables DoRA on a GPTQ / AWQ / AQLM / EETQ base. peft refuses DoRA on those layers, so a large dataset produced a config that failed after the model load.
decide_pefttakes the chosenquantizationand keepsuse_doraoff for them (#1466 by @drakeo338 in #1473).Wire the "Standalone Sweep Config" docs example:
soup sweep --sweep-config sweep.yamlnow loadsstrategy/n_runs/seed/paramsfrom a separate YAML file, instead of--parambeing a required option that the documented invocation never supplied (#1465 by @AmirF194 in #1475).Fixed KTO reaching TRL with a per-device batch of 1: the
batch_size >= 2parse-time refusal now covers everytask='kto'config (not onlystream_layers),batch_size: autofloors its estimate at 2 instead of 1, andsoup local-rl train --train-method ktowritesbatch_size: 2(#1420 by @wanjinhao1 in #1476).soup serve --devicenow reaches the transformers loader._load_modelignored itsdeviceargument and always passeddevice_map="auto", so--device cpustill placed the model on a visible GPU. The base model, the merged model and the speculative draft model now use the same placement rule assoup infer/chat/diff:cuda:Nis accepted and pinned to that GPU (it was accepted and ignored before),mpsis pinned,mlxruns on the CPU, andcudakeepsdevice_map="auto". Unknown values are refused with exit code 2 namingcpu, cuda, cuda:<index> or mps;--deviceis case-insensitive. An explicit--kv-cache-type bf16/f16still sets the load dtype on every device. Behaviour change: on a CPU-only host the default load is now float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB) (#1478 by @drakeo338 in #1479).soup diagnosenow detects verbatim memorization on long training rows by measuring clipped completion-bigram precision against the held-out suffix. The fixed 64-token live generation budget no longer dilutes overlap as the untouched suffix grows, while adjacent-token matching avoids false positives from shared vocabulary and repetition loops. This changes the threshold semantics from set-Jaccard, so earlier memorization results are not directly comparable (#1437 by @fukalous in #1481).mistral-medium-3-5-sftcan now load and attach LoRA (#1395 by @Areen-09 in #1482). The recipe base model (mistralai/Mistral-Medium-3.5-128B) usesMistral3ForConditionalGeneration, a vision-language wrapper. The recipe was missingmodality: visionundertask: sft, causing the loader to dispatch toAutoModelForCausalLMand fail. In addition,target_modules: autohad no mapping formistral3inMOE_TEXT_LORA_TARGETS, which now targets the language model attention projections (.*language_model\..*\.self_attn\.(q_proj|k_proj|v_proj|o_proj)) while excluding the vision tower, multimodal projector, and MLP projections. The recipe is also unpinned from the cannot-attach exceptions list inscripts/check_recipe_attach.py.A judge that cannot be reached is reported as unavailable, not scored as a tie (#1447 by @Srinivasan8888 in #1484). Every LLM-judge request now goes through one helper that retries 429, 5xx and transport errors within a bound (honouring
Retry-After), fails at once on a 4xx that will not heal, treats a reply with no usable content as a judge error, and raisesJudgeUnavailableErrornaming the judge URL once the retries are spent.soup ship --task-mode pairwiseexits 1 with that message instead of scoring a dead judge as a measured 0.5 win-rate and printing DON'T SHIP;soup eval judgeskips and counts the failing row and still scores, shows and saves the rest; a training eval gate still fails closed, and its log line says which task could not be scored and why instead of calling it a regression. The Online DPO adapter keeps leaving a failed pair unranked, as #1225 designed.soup version --json --fullandsoup cost --jsonnow write their JSON straight to stdout instead of through the Rich console. Rich folds a document wider than the console (80 columns when the output is piped) and reads[...]inside a string as markup, so a long record was split across lines, a value such as[v2]was dropped, and[/x]aborted the command withMarkupError.soup eval against --json-onlyhad the same defect and is fixed with them. A test walkssrc/soup_cli/and fails if any command puts serialised JSON back throughconsole.print(#1480 by @putramkti in #1485).soup eval's custom-taskcontainsscorer now requires the expected string to appear as a whole token, so14no longer counts as containing4and#### 420no longer contains42. This changes scores: a gate baseline or asoup eval custom --run-idresult recorded with the old substringcontainscan be higher than the same outputs score now, and the baseline's scorer stamp does not notice, so re-measure those before comparing. A newscoring: "answer"mode compares the final answer the model states (via the same parser GRPO'saccuracyreward uses) instead of substring-matching, and refuses task rows whoseexpectedstates no single answer instead of silently scoring 0 (#1342 by @Momoyeyu in #1490).Refuse
.soupand.soup/mcp-runsdirectory symlinks and junctions at plan time (#1211 by @kok-o in #1492). Following #1158 and #1171,snapshot_confignow inspects the directory hierarchy withrefuse_linked_dirsbefore creating directories or writingconfig.yaml, preventing config snapshot redirection outside the working directory when.soupor.soup/mcp-runsis a symbolic link or Windows junction.soup data cleanno longer drops a tool-calling trajectory whose call-only assistant turn has empty content (#1477 by @abdulwaarith0 in #1493). With no flags, the--min-tokenscheck ran on every assistant turn in thechatmlbranch, so a turn with"content": ""and a non-emptytool_calls, the shape the tool-calling format itself writes, dropped its row as "Empty / Whitespace Turns" while the same turn with"content": nullwas kept and the loader accepted both. A turn that carries tool calls is now exempt from--min-tokens; turns with no calls keep the rule.format: autonow detects a dataset's format from a bounded prefix of rows instead of the first row alone, so a first row that happens to lack an optional key no longer picks the poorer converter for the whole file and silently drops that key from every row. A mixed file either upgrades to the richer format of the same family when its converter accepts the poorer rows, or is refused with a message naming both shapes and the row where each was first seen. Files that used to load underformat: autowith a key stripped or rows dropped under a warning now stop at load; setdata.formatto the shape you want to keep. The Web UI data explorer now reports the refusal instead of failing, and the auto-detection behaviour is documented. (#1424 by @chaitanya-y20 in #1496).soup data from-tracesreads the signals Soup's own producers write, instead of reporting a green 0 pairs (#1440 by @umran666 in #1498). The pair builders key on the per-tracesignaland onedited_output, but thesoup-serveparser read the signal only from a nestedfeedback.ratingthat no Soup producer writes:soup ingestemits a top-levelsignalandsoup serve --trace-logrecords no feedback at all, so every ingest row read asnoneandsoup data from-traces --signal thumbs_upwrote an empty file and exited 0. The top-levelsignalis now read (feedback.ratingstays as a fallback for older logs), anedited_outputunder any of its three names makes--signal user_editbuildable from a real record, and a harvest that reads traces but matches none now names how many it read, which signal it wanted and which signals were present instead of reporting a normal write of 0. The examples indocs/data.mdare corrected too:--format soup_serve,--signal regenerationand a single-file--logsfor the soup-serve parser are all refused by the CLI.soup expectrefuses a top-level suite key other thanexpectations(#1483 by @Srinivasan8888 in #1499).parse_suite_specread onlyexpectationsand dropped every other top-level key, so a suite withfail_fast: true(or any typo) besideexpectations:loaded, ran and exited 0 with no message about the key. It is now refused when the suite loads, before the data file is opened (exit 3), with every unknown key named; the key text is echoed as a Python literal so a hostile key cannot reach the terminal as a control sequence. This is the same check #1461 added one level down, inside an expectation entry.task: simporefuses a dataset whose completion would be truncated away, instead of training to NaN behind a logged loss of 0.0 (#1208 by @umran666 in #1500). On trl 0.29.x, a SimPO row whosechosenoutgrowsdata.max_lengthloses its wholerejected: trl truncates each answer tomax_lengthminus the longer answer's length, so the shorter side of a lopsided pair ends with zero tokens and SimPO's length-normalised log-probability becomes 0/0. Every LoRA tensor went NaN while transformers'logging_nan_inf_filterlogged the loss as0.0.soup trainnow refuses before the first step, naming how many rows are affected, which ones, and thedata.max_lengthinvolved; a pair that keeps trainable tokens on both sides is left alone, including a longchosenbeside a shorterrejectedthat trl truncated to 64/14.GRPO's
accuracyand verifiablemathrewards now end the answer clause at justification connectivesbecauseandsince(#1328). Previously, an inline justification such asThe answer is 42 because 6*7=42.stayed in the clause and counted all numbers, treating the answer as a hedge and scoring 0.0. The clause now ends atbecauseorsince, correctly reading 42 (#1328 by @kok-o in #1501).distill_mode: sequencenow numbers rows the way token mode does, and refuses a split with nothing left (#1344 in #1502). Indistill_mode: sequence, original source row indices are preserved when skipping rows with no prompt turn, user's 1-based source row positions are reported on causal-loss target failures (noting the assistant response was generated by the teacher), skipped row counts and positions are printed at setup, and splits left with 0 usable rows are refused before training starts (#1344 by @kok-o in #1502).Resuming a QLoRA SFT run no longer upcasts the trainable LoRA adapter from bf16 to fp32: the dtypes recorded before
Trainer._load_from_checkpointare restored right after it, before the optimizer is created, so a resumed run reproduces the uninterrupted one (#1503 by @bobbonauta in #1504).soup expectrequires a judge forexpect_chosen_preferred_over_rejected_by_judgeinstead of silently scoring 1.0 (#1433 by @kok-o in #1505).expect_chosen_preferred_over_rejected_by_judgegave every row 1.0 whenjudge_fnwasNone, sosoup expectran with no judge and passed every preference dataset at any threshold. The expectation now flags preference rows as violations when no judge is configured (exit 2). A judge can be supplied viasoup expect --judge <url>or in suite YAML underargs: {judge: "ollama://llama3.1", threshold: 0.7}. Callers that deliberately want to trust existing labels without running a judge can opt in withargs: {advisory: true}.soup train --cloud lambdakeeps running across transient SSH disconnects instead of terminating the instance (#1439 by @kok-o in #1506). The rendered Lambda lifecycle controller previously relied on a single idle 90000s SSH session without keepalive probes and treated SSH exit 255 as a training failure, immediately terminating the instance and losing ongoing runs when NAT or Wi-Fi dropped. The controller now configuresServerAliveInterval=30andServerAliveCountMax=6, polls training status with short SSH calls checking cloud-init and/home/ubuntu/soup.exit, retries transient transport errors (exit 255 and timeouts) over a 15-minute retry window with exponential backoff before giving up, retries checkpoint copy (scp) across a 15-minute window before instance termination, and prints clear distinct messages for connection loss, remote training exit, and copy failures.training.warmup_ratiodynamically resolves over actual optimizer steps instead of a precomputed estimate (#1431 by @kok-o in #1507). Trainer wrappers previously turnedwarmup_ratiointo fixedwarmup_stepsusing raw dataset row counts before constructing the trainer. Under GRPO (where prompts repeatnum_generationstimes), SFT with packing or multipack, and pretrain with packing, the trainer executed a different step count, causing warmup to shrink or cover the entire run so peak learning rate was never reached. All wrappers now passwarmup_ratiothrough viaresolve_trainer_warmup_stepsdirectly toTrainingArguments(warmup_steps=...), letting transformers resolve warmup dynamically asceil(real_optimizer_steps * warmup_ratio). Runs with fewer than 34 optimizer steps at the defaultwarmup_ratio: 0.03now resolve to exactly one warmup step, whose first step trains at learning rate 0.Decouple
soup diagnoseforgetting probe from training dataset rows (#1436 by @kok-o in #1508). The live forgetting probe previously evaluated retention on the first 12 rows of--dataset, which is documented as the adapter's own training data. Fine-tuned adapters match or outperform their base model on their own training targets, causing the probe to falsely reportOK 1.000even when catastrophic forgetting occurred across general tasks. The forgetting probe now evaluates against a separate--holdout <path>JSONL option or a built-in general knowledge probe set with reference answers when--holdoutis omitted, while--datasetcontinues feeding the memorization, format, and citation probes. If--holdouthas no usable prompt/answer rows, forgetting reportsNOT_RUNrather than a false pass.Layer streaming now materializes models cached by huggingface_hub 1.32 or later, whose Xet blobs live in the cache-wide shared blob store; links that leave both the repo's blob store and the marked shared store are still refused (#1509 by @bobbonauta in #1510).
Layer streaming now accepts a local path to a Hugging Face cache snapshot (
snapshots/<commit>): it is copied to regular files like the snapshot behind a Hub id, instead of stopping with "no .safetensors weight files found" (#1511 by @bobbonauta in #1512).training.preference_loss_weightsrefuses a blend it cannot compute, instead of training one loss and reporting several (#1425 by @umran666 in #1513). No trl 0.29 trainer puts per-sequence log-probs on the batch, so every requested term went uncomputed and the wrapper fell back toprimary_loss * primary_weight:{dpo: 0.7, simpo: 0.3}printedMulti-objective preference loss: 0.70·dpo + 0.30·simpoat setup and then trained DPO alone scaled by 0.7, with nothing in the output saying so. A blend that cannot be evaluated now stops the run, naming the terms it could not compute, the ones it could, and what was requested; the weighted-sum path itself is unchanged, so a trl that does carry log-probs still blends. Everypreference_loss_weightsconfig therefore stops at the first step until the real blend lands: to train one loss, remove the weights and settraining.preference_loss.task: pporefuses the reward-hacking and echo-trap flags it cannot feed, instead of loading them inert (#1441 by @umran666 in #1514). ThetrlPPO trainer this build supports takes no reward functions — its reward comes from the reward model it calls internally — so the wrappers Soup built for the shared signal buffer were dropped, the buffer was never written, and because the buffer object existed the detector's log-reading fallback returned early as well. A run withreward_hack_detector,reward_hack_mitigationorecho_trap_enabledwas announced and then never halted, mitigated or trapped. Those three flags are now refused before the model loads, naming every flag set and pointing attask: grpo, where the buffer is fed. The on-GPU buffer feed for PPO rollouts is still open.soup local-rl trainwithout--oncerenders--min-pairsand-ointo the systemd.serviceand launchd.plistit writes, so the nightly job trains with the threshold and output directory the user asked for instead of the defaults (10 pairs,local_rl_adapter/). A--min-pairsthe nightly run would refuse now exits 2 before any unit file is written (part of #1448 by @MohammadHijjawi97 in #1517).soup advise run --probe-model Xran the live probe and hid its ROI table (#1448 by @qwist1233-cpu in #1521).--probe-modelimplies--probeand the probe ran under that derived flag, but theROI deltas (probe)table was gated on the raw--probeoption. The table now follows the same flag as the probe.soup diagnoselive probes now read ShareGPT, Alpacainput, ChatML, tool-calling, DPO, KTO and plaintext datasets (#1435 by @bakdauletbaktygaliyev in #1526). Rows go through the same format converters training uses, with the old flat-key lookup as a fallback, so these datasets give the format and mode-collapse probes a prompt/answer pair instead ofNOT_RUN. Memorization no longer waits for pairs and runs on plaintext rows. LLaVA and audio rows are skipped because the probes are text-only, a prompt over 8192 characters is skipped instead of failing the probe, theNOT_RUNreason names the dominant cause, and skipped rows are counted inextras.rows_skippedand printed asRows skipped: .... For multi-turn ChatML rows the probe prompt now stops at the first assistant reply. Two more visible changes: rows with anaudiofield andmessagesused to be probed as text and are nowNOT_RUN(exit 3 unless--allow-not-run), and Alpaca prompts now includeinputandsystem, which moves existing scores. A prompt containing a NUL byte is skipped like an over-long one instead of failing the probe. A--holdoutfile is read the same way: ShareGPT, Alpaca and ChatML hold-outs now feed the forgetting probe, and LLaVA and audio hold-out rows are skipped, so an audio hold-out isNOT_RUNlike an audio--dataset.training.freeze_layers/training.freeze_ratioare refused at load on the tasks that never applied them (#1497 by @abdulwaarith0 in #1532). Both fields loaded on every task, but onlytask: sft(andtts, which trains through the SFT trainer) freezes layers. On every other task (dpo,grpo,ppo,kto,orpo,simpo,ipo,bco,preference,reward_model,pretrain,embedding,prm,classifier,reranker,cross_encoder,distill,unlearn,moe_lora_routing,online_dpo,asr) the run trained every layer and said nothing. This is a breaking change: a config that sets either field on one of those tasks loaded before and now stops at load, with a message naming the field, the task and the fix (usetask: sft, or remove the key), andsoup doctor --configexits 2. Removing the key changes nothing about what trains, since those runs were already training every layer.sftandttsconfigs, and dumped configs (which write both fields asnull), are unchanged.soup reward synth's numeric verifier now reads both the gold and the completion with the shared answer parser the built-inmath/accuracyrewards use, so1,000,#### 1,000,\boxed {42}and#### −42spellings score the same under the generated verifier as under the built-ins, and malformed comma groupings like10,00are no longer detected as numeric. Regeneration also stops paying some completions the old regex paid —####or\boxed{}answers carrying units,%or a non-dollar currency symbol or code (#### 42 apples,\boxed{42\%},#### €42), hedged or justified answer phrases, and a wrong last box after a right first now score 0, matching the built-ins (bare42%andThe answer is 42%.still pay). Areward.pysynthesized before this version scored the newly-accepted spellings 0, so regenerate it before comparing runs; the emitted file now importssoup_clirather than being fully self-contained. The calibration report and--output-reportalso name every reference the verifier rejects (rejected_references) instead of reporting only a refusal rate, and a refused run lists the first ten rejected references in the panel (#1350 by @Momoyeyu in #1533).Fix
soup train --task unlearncrashing withKeyError: 'duration'in the "Training Complete!" panel after the run had already finished: the adapter was saved and the tracker row written, but the unlearn wrapper was the one transformers-backend trainer that returned onlyduration_secswithout the pre-formatteddurationstring. The wrapper now formatsdurationlike every other wrapper, and the completion panel falls back to formattingduration_secsitself so no future wrapper can lose a finished run's summary (#1529 by @vaputa in #1537).soup eval's custom-taskcontainsscorer is linear again in the length of the output and ofexpected. The whole-token match from #1490 restarted its search at every position, so a 400,000-character output against a 200,000-characterexpectedtook 68 s; it now takes milliseconds. Both sides are case-folded withstr.lower(), as in v0.75.1, soistanbulno longer matchesİstanbulandsno longer matchesſ; no other score changes (#1528 by @UroojFatima-052 in #1540).Aligned ULD (
uld_strategy: wasserstein_aligned) no longer scores student positions that have no teacher target (#1426 by @developerjillur in #1541). A byte-fallback piece, an empty decoded piece or a student-only special token aligns to no teacher token; it got an all-zero teacher row, which softmaxes to uniform, and each such position added aboutV/2to the loss. These positions now leave both the sum and the denominator under the label mask, the attention mask and no mask, and the trainer decodes student special tokens to"", as it already does for the teacher.76 tests across 10 files downloaded tiny models/tokenizers from the Hugging Face Hub on every run, none marked
smokeand none guarded against the network being unavailable (#1356 by @SID-6921 in #1544). A real sweep (HF_HUB_OFFLINE=1, emptyHF_HOME,pytest tests/ -m "not smoke") found a much larger landscape than a preliminary grep-based pass had estimated. Every site now builds its tiny model and tokenizer locally instead -- aByteLevelBPETokenizertrained from a few lines of in-file text feeding a real, randomly-initialisedLlamaConfig/GPT2Configmodel, matching the existing_write_tiny_qwen2convention -- since none of the fixed tests needed real pretrained weights or vocabulary, only a working tokenizer and a save/reload round-trip. A full-suite offline re-sweep afterward confirms zero remaining Hub-dependent failures.notebooks/proof-4gb.ipynbruns again on a hosted multi-GPU runtime with a currenthuggingface-hub: the install cell capshuggingface-hub<1.32.0(soup-cli 0.75.0 refuses the shared blob store introduced in 1.32, #1509) and the notebook setsCUDA_VISIBLE_DEVICESto one card before torch is imported, because layer streaming cannot run undernn.DataParallel(#1543 by @here-2007 in #1545).eval.auto_eval: truenow runs the configuredeval.benchmarksandeval.custom_tasksaftersoup trainfinishes; before, no trainer handed the evaluation config to the training callback, so nothing ran (#1487 by @commit-msuyog in #1546).Terminal output strips the C1 control range (U+0080-U+009F) as well as C0 and DEL, and text read from files is printed through the shared
for_terminalhelper instead of raw, throughescape()alone, or quoted withrepr()after escaping: adapter-config base model names insoup chat,soup merge,soup serve,soup export,soup eval benchmarkand the--trust-remote-codewarning;soup adapters list/info/compare/diff; benchmark names in thesoup shipfooter;soup edit diff; the SFT "cannot open image/audio" warnings; audio file names in thesoup infer --task asrskip warnings; the config loader's validation lines;soup probe interferenceloss keys; and the Web UI's log line for a refused config. Markup in such a value no longer changes an exit code: a signature sidecar whosebackendname contains[/]getssoup attest verify's documented exit code 3 instead of 1, and a path or key containing[/]gets exit code 2 instead of 1 fromsoup env lock/env fix,soup tunability,soup deploy autopilot --measureandsoup probe interference(soup data forgeandsoup adapters arithmeticnow print such a name as text too). A.canmanifestauthorrefuses control characters (#1552).soup initrefuses a symbolic link, junction or other reparse point at the config path it writes, and MCP planning creates the config snapshot's run directory fresh (#1553).soup initwrote the starter config with a plain write that follows a link at the target (soup.yamlor--output), and a dangling link even skipped the overwrite prompt. It now refuses a symbolic link, junction or other reparse point there before prompting or running the wizard, naming the path, matching the link refusal the other config writers apply, and it creates or opens the file without following a link that appears there afterwards: nothing is written through such a link and nothing is created at its target. Ordinary paths, including ones outside the working directory or reached through a linked parent directory, behave as before. Following #1492,train_startundersoup mcp servenow also checks the run directory.soup/mcp-runs/<run_id>itself, creates it withexist_ok=False, and createsconfig.yamlexclusively (O_EXCL) without following a link, refusing the plan with a path-free error when the run directory or the file already exists or is a link.The outbound loopback allowlist is now a single shared set:
http://[::1]:<port>judge URLs are accepted byGateTask.judge_model,soup ship --judge-model, and the online-DPO trainer's_parse_judge_url, matching the judge API-base and vLLM checks. Each gate previously kept its own("localhost", "127.0.0.1")literal, which refused the IPv6 loopback (#1548 by @akshatshahh in #1554).training.lora.use_dora: truewithtraining.quantization: gptq,awq,aqlmoreetqis now refused when the config is loaded. peft has no DoRA variant for those layers, so the config used to load and fail only when the adapter was attached, after the model load. Autopilot'sdecide_peftreads the same format set (#1466 by @ybai08 in #1555).Fix
soup data from-traces --format soup-servesilently writing an empty output (exit 0,Wrote 0 preference pair(s)) when--logspointed at a single file: the soup-serve reader wants the directorysoup serve --trace-logwrites, and a file is now refused with a message naming the option and the expected shape; a source that reads zero traces at all now says so ahead of the count line, so an empty output no longer reads the same as a completed harvest (#1530 by @vaputa in #1557).training.quantization: mxfp4passed config validation and then failed at the model load (#1466 by @ybai08 in #1558). Soup builtBitsAndBytesConfig(bnb_4bit_quant_type="mxfp4"), which bitsandbytes rejects (it acceptsnf4andfp4only) and which clashes with theMxfp4Configa pre-quantized MXFP4 checkpoint carries, including the basessoup autopilotpasses through asmxfp4. The loader now returns transformers'Mxfp4Config(dequantize=True), since transformers does not train MXFP4 weights as loaded, so the model trains in bf16.mxfp4no longer goes throughprepare_model_for_kbit_training, which would cast the dequantized model to fp32. Two configs that validated before are now refused, neither of which could train:bnb_4bit_quant_storagewithmxfp4(it is a bitsandbytes argument), andmxfp4with a local base whoseconfig.jsondoes not declarequant_method: mxfp4. The hardware-fit calculator (soup_cli.utils.hardware_fit) now prices anmxfp4base at 2 bytes per parameter, as the bf16 model it loads (it was 0.55).Lao, Khmer, Myanmar, halfwidth katakana and CJK Ext C+ are now scored per character pair like Thai; symbol-only answers are compared by their symbols; rows and answer sets with no tokens are skipped and counted. (#1343 by @Hitesh-XS in #1559)
soup migratemaps every LLaMA-Factorypref_lossand TRL trainer class to its Soup task, and refuses the rest by name (#1214 by @Srinivasan8888 in #1560). LLaMA-Factorystage: dpowithpref_loss: iponow migrates totask: ipoinstead of plaindpo;hinge,kto_pairand any other value with no Soup task makesoup migrateexit 1 naming the value and the supported list, instead of loading asdpowith no warning. Unsloth notebooks:RewardTrainer->reward_model,BCOTrainer->bco,OnlineDPOTrainer->online_dpo(with a warning that the judge / reward model is not carried over), andCPOTrainer->simpoonly when theloss_type="simpo"on that trainer's own call or itsargs=config; other CPO losses are refused.SFTConfigand the other TRL*Configclasses are now read for hyperparameters, so a notebook passingargs=SFTConfig(...)keeps its batch size, epochs, learning rate and output directory.OnlineDPOTrainernotebooks migrate withtraining.online_dpo_judge: ollama://REPLACE-MEas a placeholder (TRL's judge is a Python object with no string form) and a warning naming it; replace it, or settraining.reward_modelinstead. Migrations that used to produce a config and now stop with exit 1: LLaMA-Factorystage: dpowithpref_loss: hinge,kto_pair, any other unlisted value, or a scalar that is not a string such as5ortrue(they loaded astask: dpo; a list or mapping already stopped onmain, with aTypeErrortraceback instead of the by-name message), and an Unsloth notebook whoseCPOTrainerdoes not setloss_type="simpo"on its own call or itsargs=config, or whoseargs=cannot be read statically (they loaded astask: sft). Migrations whose result changes: LLaMA-Factorypref_loss: orpo/simpounder a stage other thandponow keeps that stage's task, as LLaMA-Factory itself does (they loaded asorpo/simpo); and becauseSFTConfigis now in the config table, a notebook with a laterSFTTrainer(args=SFTConfig(...))after another trainer migrates assft(the last*Configcall decides, as it already did for the other config classes). The axolotl migrator was settled by #1254.IP-literal parsing depended on the platform and the interpreter (#1550 by @qwist1233-cpu in #1562). On Windows the C library refused
0xffffffff,4294967295and0377.0377.0377.0377, so those spellings of255.255.255.255passed the outbound guard; 6to4, NAT64 and Teredo literals were classified only by the interpreter'sipaddresstables.net_guardnow parses IPv4 spellings in pure Python, unwraps the embedded IPv4 address and classifies it too, andrefuse_private_ip_literalparses once.A judge that is down no longer costs 3 s of backoff for every row (#1522 by @ybai08 in #1563). Each judged row retried on its own, so
soup eval judgeover 10,000 rows slept for more than 8 hours against a host that never answered, andsoup data from-traces --judgedid the same and then wrote an empty file with exit 0. An evaluator now counts the requests in a row that spent their retries unanswered and raisesJudgeDownError, aJudgeUnavailableError, at the third.soup eval judgestops there, prints one message naming the judge URL and how many items were not judged, shows the scores from before the outage without saving them, and exits 1;from-traces --judgeexits 1 and writes nothing;soup data best-of-nandsoup ship --task-mode pairwisesay how many prompts or pairs were left. One reply resets the count, and a 4xx or an unusable reply counts as a reply. A judge-ranked training run is unchanged: it keeps asking and resumes on the first reply.Honor configured optimizer, schedule, warmup, weight decay, clipping and effective batch settings across PRM, MoLE, PPO and unlearning; keep unlearn loss and sample counts on their original scale. Unlearn now tokenizes forget and retain rows to
data.max_length(default 2048) instead of a hard-coded 256, so existing unlearn configs use more memory and run longer; setdata.max_length: 256to keep the old limit. On unlearn,batch_size: autonow resolves to 1 rather than to a memory estimate, so the same config makes the same number of optimizer updates on any card. (#805 by @Oxygen56 in #1564)The provenance and compliance artifacts record what was measured, and only that (#1446 by @Srinivasan8888 in #1569).
soup attest emit --invocationnow reaches the SLSA statement, atbuildDefinition.externalParameters.invocation, unsigned and signed; it was stored under a key the builder never read. Annex XI and XII writenot measuredfor training compute, energy and CO2 when nothing was measured (soup train --annex-xiwithout--track-energy, and FLOPs always, which Soup does not measure), instead of0and0.000 kWh/0.000 kgpresented as measurements. The CycloneDX 1.6 ML-BOM has an RFC 4122serialNumber, writes a known SPDX id in its canonical spelling (apache-2.0→Apache-2.0), an SPDX expression underexpression, and any other license name underlicense.name; the SPDX 2.3 document does the same inlicenseConcluded/licenseDeclared(other names become aLicenseRef-withhasExtractedLicensingInfos), and now states that the training data is a build dependency of the model rather than the reverse. The SR 11-7 reproducibility receipt fillsdriver_versionon a CUDA host. The CycloneDX 1.6 schema set is vendored undertests/fixtures/cyclonedx/and the document is validated against it.soup expectnames at most ten unknown suite keys in a refusal, then counts the rest (by @Srinivasan8888 in #1570). The refusals #1461 and #1499 added for a key besidename/argsand for a key besideexpectations:named every offending key, so a suite under the 1 MiB cap with tens of thousands of unknown keys produced a message nearly as large as the file. The first ten are named and the remainder reported asand N more; messages with ten keys or fewer are unchanged.Every documented
soupinvocation names a real command path and real flags, and a guard keeps it so (#1465 by @Srinivasan8888 in #1578). Five documented commands named a flag or command path the CLI refuses and one recipe description named an option no command has:soup loop watch --max-iteris--max-iterations;soup serve ... --spec-tokensis--num-speculative-tokens;soup --verbose eval --model ...issoup eval benchmark ...;--no-telemetryis a global option and goes before the command (soup --no-telemetry train ...);soup adapters blamehas no--live; theonline-dpo-smollm2-135mdescription points attraining.online_dpo_judge. Sixteen more documented lines were missing a required option or carried an extra argument (soup data pii/decontaminate/augmenttake--input;soup inferneeds--output;soup eval customneeds--modelandsoup eval humanboth--model-a/--model-b;soup serve --reasoning-parserneeds--model; a second adapter needs its own--adapters;soup runs listissoup runs;soup recipes search --tag mlxissoup recipes search mlx) and are corrected.docs/data.mdnow says thatsoup data decontaminate --benchmarksrecords labels only and that--benchmark-fileis what filters, since no benchmark corpora are bundled; its examples pass one. A test walks everysoup ...line (aligned columns,VAR=valueprefixes and backslash continuations included) and every`soup ...`span indocs/*.mdandREADME.md, plus every recipe description, against the live Click tree: a flag or command path that does not exist, or a subcommand a group does not have, fails CI. Whether a line also carries every required option is not checked. (The standalone sweep config was #1475.)A
training.prm_rewardHub id is downloaded before training, and a PRM without a reward head is refused before the model loads (#1466 by @Srinivasan8888 in #1580).prm_rewardis documented as a path or a Hub id, but the PRM's reward head was read only from a local directory, so a GRPO run givenorg/nameloaded the tokenizer and the model and then died at the first reward call with "path is not a directory". The id (optionallyorg/name@revision) is now resolved through Soup's Hub download wrapper (utils.hubs.snapshot_download, which runs the repo-id, revision and namespace-pin checks first), restricted to weights, tokenizer files and custom-code modules, and the head is read from that snapshot when the reward is built; a repo that is not a Soup-trained PRM, a download failure, or a value that is neither an existing directory nor a repo id each stop with one message naming the value, before any training. Existing local paths keep their containment check and now also fail early when they carry no reward head.soup migrate --from unslothreads a notebook that carries IPython magics (#1579 by @Srinivasan8888 in #1583). Every code cell was concatenated and parsed as one Python module, so the first!pip install ...or%%captureline made the whole notebook aSyntaxErrorwith no cell and no line, and every official Unsloth notebook starts with such a cell. A%%cell magic whose body IPython runs as Python (%%capture,%%time,%%timeit,%%prun) now loses only its header and its body is read; any other cell magic (%%bash,%%writefile, ...) is skipped whole. A line whose first non-blank character is!or%is treated as a magic only at statement level, so a%operator or a!=at the start of a continuation line is untouched, and a magic that is the only statement of anif/for/withbody leaves a valid block behind. A cell that parses today is read byte for byte. A real syntax error is now reported with its code cell and line.With
training.expand_layersset,training.freeze_trainable_layersmust now equal it, and any other value fails at load (#1409 by @dchaudhari7177 in #1586). The field was read only for its sign:1,2and4all trained just the appended blocks, and0or a negative value silently trained the whole expanded model. It now has one meaning, freeze the original model and train only the appended blocks, so a config withexpand_layers: Nand afreeze_trainable_layersof0, a negative value or a different positive value is refused with both fields named andSet freeze_trainable_layers: N.Before, withlora.r: 0,0and negative values trained every layer and any positive value trained the appended blocks; under LoRA the value changed nothing (#1326).expand_layers: Nwithfreeze_trainable_layers: Ntrains exactly what it did.training.moe_lora: truenow adapts the routed experts of every fused-expert MoE on transformers 5 (by module names alone Mixtral and MiniMax adapted attention only, Qwen2-MoE attention plus its shared expert, Qwen3.5-MoE the shared expert only; Qwen3-MoE, OLMoE, DeepSeek-V3 and GLM-4 were reached only through peft's own name conversion): the fusedmlp.experts.gate_up_proj/down_projtensors are handed to peft astarget_parameters, the module targetsautoresolved are kept beside them (an explicitlora.target_moduleslist is kept too, wheremoe_loraused to replace it), and the console line names the parameters. Because peft'sParamWrapperrefuses dropout,moe_lorawith a non-zerolora.dropoutis now refused on Mixtral and MiniMax too (#1421 by @Srinivasan8888 in #1590).training.moe_expert_quanton a fused-expert model is refused at setup, naming the architecture and the fused parameters, instead of printingapplied to 0 expert Linear block(s)and training in full precision; matching no expert at all is refused the same way (#1421 by @Srinivasan8888 in #1590).A
task: grporun withtraining.eval_stepsnow records the held-out reward (eval_reward, higher is better) as its validation metric: the live panel shows aVal rewardrow, the tracker's metrics table gains aval_rewardcolumn (added to existing databases in place), and the training event stream carriesval_reward. TRL'seval_losson grpo, the policy objective, is no longer stored or shown asval_loss; every other task keepseval_loss/val_lossunchanged. The per-task choice lives inVALIDATION_METRICSinutils/eval_schedule.py(#1389 by @Srinivasan8888 in #1592).The pattern used to match HTTP 401/403 errors in the
format_friendly_errorfunction inerrors.pyno longer backtracks quadratically if handed an exception which contains the characters "HTTP" followed by whitespace with no status code (#1367 by @johncoplin in #1594). The old pattern had two places in which whitespace could go if the separator was missing from an exception, meaning the function would have to try every possible combination of whitespace before being able to print the message for unmatched errors. This pattern has now been replaced with one that only has two spots for whitespace if the separator exists, preventing quadratic backtracking and meaning error-messages print faster overall for these kinds of exceptions.Vision and audio rows loaded through a streaming
data.interleave, a remote URI or a Hugging Face Hub dataset (train andvalidationsplits) now resolve theirimage/audiopaths the same way as rows from a local file: againstdata.image_dir/data.audio_dirwhen set, otherwise against the directory of the local data file the row came from, skipping a row whose file lies outside that directory with the usual "rows skipped" warning. These load paths used to keep the value as written, so a relative path was opened against the working directory anddata.image_dir/data.audio_dirhad no effect. This is stricter for a remote URI or a Hub dataset whose rows hold file paths and that sets neither directory: there is no local directory to resolve them against, so the load now stops with a message naming the setting (data.image_dir: .resolves them under the working directory, skipping paths outside it). An image the Hub dataset has already decoded is passed on as before. On every load path, insoup infer --task asrand insoup data inspect, a value naming a network share, a device or an extended-length path is skipped without being looked up, and so are abytesvalue and, on Windows, a DOS device name such asnul; on Linux and macOS a value beginning with//now counts as a network path too. The SFT vision and audio paths and the ASR / TTS audio loader read a file only when it is a regular file, so a directory, FIFO or device is reported instead of being read (#1599).Vision SFT accepts decoded PIL images from Hugging Face Hub datasets and Parquet image structs ({"bytes", "path"}) without crashing. Path structs follow the same directory resolution and path containment rules as file path strings (#1597 by @Nari-Ab in #1601).
soup data inspectresolves relative vision image paths against the data file's directory (or--image-dir), the same way training does, and reports an image training would refuse as outside the image directory instead of counting it as found (#1598 by @commit-msuyog in #1604).soup data generate --template preference: deduplicate generated preference rows (DPO and KTO formats) against--dedup-withand within-run duplicates using a stable key, rather than bypassing deduplication (part of #1448 by @Nari-Ab in #1605).soup merge --save-format 4bit_forced: passllm_int8_skip_modules=[]toBitsAndBytesConfigso transformers quantizes all linear modules includinglm_headrather than silently skipping them, and refuse models with tied embeddings (tie_word_embeddings=True) by name before loading weights (part of #1448 by @Nari-Ab in #1606).soup serve: wire output constraint logits processors and reasoning parser into_stream_responseso streaming chat completions (stream: true) apply structured generation constraints and strip<think>blocks before emitting SSE events (part of #1448 by @Nari-Ab in #1607).GRPO variant loss kernels now guard against exponential ratio overflow and NaN backward gradients. Importance weight ratios and sequence ratios in
grpo_variants.pyare clamped from above before exponentiation using an advantage-aware bound, and padding tokens use boolean masking, preventing infinite terms from producing NaN parameter gradients under policy divergence. (#1609 by @harshitthek in #1610)A pinned staging arena whose base is off a 4096-byte boundary no longer turns direct I/O off for the whole disk-tier store (#1531 by @ybai08 in #1621).
plan_pinned_arenasaligns the staging regions relative to their arena, and torch's caching host allocator can hand back an arena at a multiple of only 512 bytes; one misaligned region then sent every layer through the page cache, 2.4x slower cold (#974), with alogger.warningas the only sign.AsyncDiskSourcenow reads each arena's base, shifts the placements inside the arena when its slack allows, and otherwise takes the next power of two up;pinned_bytescounts the arenas actually held. If the staging is misaligned anyway, the page-cache fallback is printed on the console as well as logged.data.add_new_tokens/data.new_special_tokensare refused at config load wherever no trainer setup adds them to the tokenizer, naming what does not apply them: everybackend: unslothsetup (ttswithnew_special_tokensincluded; SFT's vision and audio paths excepted),backend: mlx,training.stream_layers: true, and the tasks whose trainers never call the expansion helper (pretrain,embedding,ppo,reward_model,prm,classifier,reranker,cross_encoder,unlearn,moe_lora_routing,distill,asr). They used to load there and be dropped, so the tokens never reached the tokenizer. Onbackend: transformersforsft,dpo,kto,orpo,ipo,simpo,bco,grpo,online_dpo,preferenceandttsnothing changes (#1358 by @Srinivasan8888 in #1622).QuEST continuation now preserves the artifact's measured calibration. Continuing training from a local QuEST artifact previously recalibrated activation clips on the first 32 new training rows, changing the route represented by the checkpoint. Soup now restores the saved clipping table and route. It also refuses a non-local artifact and a
config.jsondeclaration that differs from the mandatory metadata sidecar. (#674 by @Shutaru in #1624).The grad-accum advisory measures VRAM pressure against the card the run uses, not a fixed 24 GB (#1620).
training.grad_accum_auto_tunebuilt its monitor with a hard-codedtotal_vram_gb=24.0, so on every card smaller than about 24 GB (all common 4, 8, 12, 16 and 20 GB cards) the advisory could never fire, while a 40-80 GB card fired at ~22 GB of peak use (about 28% of an H100). The callback now reads the CUDA device's real total memory on the first advisory call (an explicitgrad_accum_total_vram_gbstill overrides it), reads the peak from the same device, prints the measured numbers in the advisory (peak 7.8 of 8.0 GB (97%)), and the live memory panel reads its total from the same device instead of always device 0 (#1620 by @vaputa in #1625).SOUP_LAYER_STREAM_STRIPE_DIRSrefusals show the entry as typed (#1611). Every refusal formatted the entry withrepr, so each backslash of a Windows path was doubled, and copying the entry out of the message gave a spelling that does not work. Entries are now shown in single quotes with single backslashes. Control characters are still escaped as\xNN, so they never reach the terminal raw. The same applies to the Windows owner-only folder errors and the disk pre-flight message (#1611 by @vaputa in #1626).training.auto_mixed_precisionis now refused at config load on every task whose trainer never reads it (#1618). The field loaded on all 23 tasks while only the SFT trainer applied it (inherited bytts), so a DPO, GRPO, KTO or any other non-SFT run carrying the key trained at its trainer's default bf16/fp16 choice with nothing said, andsoup autopilotwrote the key into generated non-SFT configs. Setting it on any task outsidesft/ttsnow fails config validation naming the task and the way out,generate_configno longer emits it there, the GRPOgrpo_fp16+auto_mixed_precisionrefusal now saysgrpo_fp16is the only dtype pick on that task, and the field description and GRPO docstring no longer promise a picker that never ran. This is a breaking change: a config that sets the field on a task other thansft/ttsused to load and is now refused — removing the key changes nothing about how such a run trains (#1618 by @vaputa in #1627).Layer streaming validates
SOUP_LAYER_STREAM_STRIPE_DIRSearly duringsoup trainstartup and--dry-run, refusing missing folders, bad syntax, and same-volume stripes before loading datasets or prompting, andsoup doctornow reports stripe roots under System Resources (#1613 by @Nari-Ab in #1628).Layer streaming copies a Hugging Face snapshot only from the repo's own
blobsdirectory and the marked shared store: ablobsdirectory that is itself a link (a symlink, or a junction on Windows) is now refused with a message naming it instead of being followed, and a junction inside a snapshot directory is refused like a directory symlink. The links huggingface_hub creates, including those into the shared store of 1.32 or later, work as before, and so does a cache or repo folder that was moved as a whole and linked (#1629).info_rmreward-hacking detection no longer votes on a step whose median split leaves either half constant — a two-valued reward (e.g.reward_fn: accuracy's 0/1), or a continuous reward where half the batch lands on the same value. Previously the pooled-variance floor collapsed to1e-9and a first step near a 50% success rate recorded a phantom separation of ~31,623 as the baseline, so later, healthier steps read as a collapse and could halt the run or mutate β. Those steps now cast no vote and log one warning naming the cause. A step where every reward is identical still votes once a baseline exists, so a genuinely collapsed continuous reward reads HACK as before. The pooled-variance floor is also tied to the observed reward range instead of a bare1e-9. Detector logs andmitigation_log.jsonlfrom runs before this version are not comparable — usereward_hack_detector: rm_ensembleor the length/repetition signals on two-valued rewards (#1438 by @Momoyeyu in #1630).Wire DPO-family reference adapter regeneration on LoRA (#1345 by @kok-o in #1636).
dpo_ref_regen_epochsperiodically syncs reference model weights on DPO, IPO, and preference training. Under LoRA (where TRL creates a frozenrefadapter and leavestrainer.ref_modelasNone), in-place copies active.default.adapter parameters into.ref.parameters undertorch.no_grad()on every Nth epoch. Full fine-tuning (lora.r: 0) remains refused at config load time.A fresh
pip install 'soup-cli[train]'could no longer runtask: grpoortask: online_dpo: the trainer import stopped withModuleNotFoundError: No module named 'requests'(#1637). trl importsrequestsat module scope intrl/generation/vllm_client.py, and bothtrl.trainer.grpo_trainerandtrl.experimental.online_dpoimport that module unconditionally, but trl listsrequestsonly under its optionalvllmextra. A training install received the package indirectly throughdatasets, until datasets 5.1.0 (2026-10-05) dropped that dependency. Environments created before that date keep working because pip does not remove an installed package; environments created after it were missingrequests. On GRPO the failure surfaced asRuntimeError: Failed to import trl.trainer.grpo_trainer, and on online DPO asinstalled trl 0.29.1 does not provide 'OnlineDPOConfig', with the real cause at the end of the message. The[train]extra now declaresrequests>=2.32.2itself — the floordatasetscarried, so an existing environment resolves to the same set. Trainers that do not load those two trl modules were not affected: on trl 0.29.1 the SFT, DPO, KTO, ORPO, CPO, BCO, PPO and reward trainer classes import withoutrequests. On an installed release without this change,pip install "requests>=2.32.2"restores both tasks.tests/test_train_extra_declares_requests.pypins the declaration.Step PyTorch profiler on each training step so
soup train --profilewrites traces (#1448 by @kok-o in #1639).soup train --profilepreviously announced writing a trace upon completion but left<output_dir>/profiles/empty becausetorch.profiler.profileschedule state only advances whenprofiler.step()is invoked per step. AddedProfileStepCallbacklazily subclassing HuggingFaceTrainerCallbackvia PEP 562 to step the profiler onon_step_endwithout importingtransformersat CLI startup, and wired it intosoup_cli.commands.trainwhen profiling is enabled.soup sweepnow marks an arm as failed when trainer construction or setup raises, preventing stuckrunningrows in the experiment tracker (#1464 by @deximple in #1640)A sink-based ratchet for outbound HTTP calls across
src/soup_cli(#1547 by @Areen-09 in #1646). An AST ratchet verifies every outbound HTTP call (httpxmethods and clients,urllib.requestopeners and callers, andrequestsmethods) against a declared ledger table of known gates and justifications, in addition to shape-based checks, with fail-closed bidirectional enforcement.validate_electricity_map_endpointnow routes throughnet_guard.LOOPBACK_HOSTSandrefuse_private_ip_literal: it refuses abbreviated IPv4 forms (such as10.1) and RFC 6598 carrier-grade NAT (100.64.0.0/10), while loopback addresses such ashttps://127.0.0.2/andhttps://[::ffff:127.0.0.1]/are now accepted under the shared loopback policy.soup diagnose,soup train --diagnose-gateandsoup eval unlearningprint text they did not write through the sharedfor_terminalhelper instead ofescape()alone, so ESC, DEL and the C1 control range (U+0080-U+009F) are stripped before the text reaches the terminal: the evidence line of every score or metric (from--evidence/--diagnose-gateJSON or a live run), the adapter name and run id in the table titles and the overall panel, the--outputand--badgepaths, the--attach-to-registryid, and the exception text in their error lines. The--diagnose-gate failedline ofsoup trainprinted its exception text as Rich markup: an evidence file whose refusedverdictcontained[/]ended the command with aMarkupErrortraceback instead of the message (#1647).training.freeze_layers/training.freeze_ratioare refused at load on thesftandttspaths that never applied them (#1581 by @abdulwaarith0 in #1649). #1532 refused the two fields on every other task; insidesftandttsthey are read only on the text path withbackend: transformersand no layer streaming. Withbackend: mlx,backend: unsloth,modality: vision,modality: audioortraining.stream_layers: true(ontts,backend: unsloth) the config loaded, nothing was frozen and nothing was printed: those setups always attach an adapter, so the run trained every layer. This is a breaking change: a config that sets either field on one of those paths loaded before and now stops at load, with a message naming the field, every setting that leaves the text path and what to change (for exampleUse backend: transformers, or remove the key.), andsoup doctor --configexits 2. Removing the key changes nothing about what trains. Text configs onbackend: transformers, and dumped configs (which write both fields asnull), are unchanged.A host written only as numbers is an IPv4 address or it is refused (#1650, follow-up to #1562).
net_guardreads IPv4 spellings with one strict grammar on every platform, and text that looks like a number but does not fit it (a number too large for its position such as4294967296or10.0.0.256, a0xwith no hex digit after it, an8in an octal part, a fifth part) was treated as a hostname, although C libraries do not all read such text the same way. The outbound endpoint checks (--api-base, judge URLs,online_dpo_judge, the vLLM URL, webhooks, the OTLP endpoint, the Langfuse host, the telemetry endpoint, the Web UI chat endpoint) now refuse a host made of numeric labels only that is not a valid IPv4 address; where the check reports its own message it readsan all-numeric host that is not a valid IPv4 address is not allowed. A hostname with any label that is not a number, loopback, and every numeric spelling that is a valid address are unaffected. A decimal number longer thanint()converts (4300 digits) is now "not an address" instead of an uncaughtValueError. A host that is a single space counts as non-public on every platform, as it did on Windows.The
soup trainVRAM pre-flight skippedtraining.quantization: mxfp4(#1631 by @ybai08 in #1655). An MXFP4 base is dequantized on load and trains in bf16, but the pre-flight built its estimate only fornone,4bitand8bit, so anmxfp4run was never checked. It is now checked against the same estimate as aquantization: nonerun, about 2 bytes per parameter, and a run that does not fit is refused with one extra sentence saying the estimate prices the bf16 model. This refusesmxfp4configs that used to start on a card too small for the model;--allow-oom-attemptlaunches them anyway.Budget
mxfp4bases as dequantized bf16 in autopilot (#1653 by @kok-o in #1656).decide_quantizationnow enforces the VRAM budget formxfp4pre-quantized models which dequantize on load to bf16 (2 bytes per parameter), preventing generation of configurations that cannot load on the target GPU budget.Handle
typer.Exitcleanly insoup trainauto-eval custom tasks (#1641 by @kok-o in #1658). Catchtyper.Exitwhen executing auto-eval custom tasks to avoid printing an unhandled traceback and reporting the run as failed.Keep trainable adapter dtype across checkpoint resume for DPO, IPO, GRPO, and Reward trainers (#1573 by @kok-o in #1660). Apply
keep_trainable_dtype_on_resumeinDPOTrainerWrapper,IPOTrainerWrapper,GRPOTrainerWrapper, andRewardModelTrainerWrapperso that resuming a run preserves the trainable adapter's original dtype instead of upcasting to fp32.Auto-evaluation after
soup trainfollows the run's--trust-remote-codeflag (#1644 by @Areen-09 in #1662).soup train --trust-remote-codewitheval.auto_eval: truenow passes the flag to the benchmark and custom evaluation of the model the run just saved, instead of alwaysFalse, which made a base that needs custom code train and then fail its own evaluation. Without the flag both stay off, as before. The opt-in is CLI only: asoup.yamlwithtrust_remote_code(top level or undereval:) is refused as an unknown key.Re-check that a
SOUP_LAYER_STREAM_STRIPE_DIRSfolder has not been renamed and replaced by a link before every per-layer write and the marker write of a layer-stream shard, instead of only once before the shard starts; and refuse a directory junction as a stripe root while still accepting a genuine Windows volume mount point, distinguishing the two by their link target rather than their shared reparse tag (#1614 by @SID-6921 in #1663).Layer streaming reuses its regular-file copy of a Hugging Face snapshot only while that snapshot would still be copied. When the repo's
blobsdirectory has become a link (a symlink, or a junction on Windows) since the copy was made, or the snapshot has gained a linked directory or a file whose link leaves the blob store, the run now stops with the same message as a first copy instead of returning the earlier copy. That copy is left in place, and an unchanged snapshot is reused without copying, as before (#1666).Layer streaming keeps the regular-file copy of a local Hugging Face snapshot path (
.../models--org--name/snapshots/<commit>) in the cache slot of the Hub idorg/nameonly when that path is the folder the Hub id resolves to in the active Hugging Face cache (a cache that was moved and linked counts as the same folder). A snapshot folder anywhere else now gets a slot of its own, named after its resolved location, so a copy taken from one folder is never reused for another folder or for the Hub id; such a folder is copied once more on its next run (#1667).mini_common_senseno longer sits at a 1.000 ceiling (#1192 by @AaronProbha18 in #1673). The suite grows from 24 to 40 rows with 16 four-option reasoning questions; Qwen2.5-1.5B-Instruct measures 0.850 against 0.275 on SmolLM2-135M-Instruct, which is chance level for the suite. The answer key no longer favours one letter, so a model that always answers "B" drops from 0.583 to that same 0.275, and the bundled scorer provenance revision moves to 5 so pre-change baselines are not silently compared on the new scale.soup local-rl train --backendprints a note that the option has no effect on training; it is validated but not rendered into the scheduler scaffold (part of #1448 by @Nick-800 in #1678).soup export --registry-idnow records the exported files foronnx,tensorrt,awq,gptq,torchao,gguf-ud,bitnetandtq1_0(every shard or engine file, named in the success line).--deploy ollamanow works withgguf-ud,bitnetandtq1_0and is refused before the export starts for the formats Ollama cannot load (part of #1448 by @Nick-800 in #1679).task: pposaves its checkpoints again on transformers 5.19.0. trl's experimentalPPOTraineris atransformers.Trainersubclass that never runsTrainer.__init__, and transformers 5.19.0 madeTrainer.save_modelread an attribute onlyTrainer.__init__assigns (is_distributed_loading_by_transformers), so every PPO run stopped at its first checkpoint withAttributeError: 'PPOTrainer' object has no attribute 'is_distributed_loading_by_transformers'. After building the PPO trainer Soup now gives it each attribute the installedTrainer.__init__assigns that the trainer lacks andTrainer's save path reads (that one,is_fsdp_xla_v1_enabledandhp_name), with the valueTrainer.__init__would have assigned; an attribute the trainer already has is left alone, and nothing changes on a transformers that never had the attribute. No other task is affected: every other trl trainer Soup builds runsTrainer.__init__. Reported by @Srinivasan8888 in #1675 (#1682).soup diagnose --attach-to-registryandsoup eval unlearning --attach-to-registryattach the report. Both called the registry helper with arguments it does not take, so the attach raised aTypeErroron every run, the report was never linked to the entry, and the command printed the error as a warning and exited 0. The--outputfile is now stored on the entry (asdiagnose_report/eval_results), and a requested attach is part of the command's success, as it already is forsoup bom emit,soup attest emitandsoup eval custom: a failed attach (unknown or ambiguous entry, unreadable registry) prints an error and exits 1 with the report left on disk, and--attach-to-registrywithout--outputis refused before anything runs (exit 3 forsoup diagnose, exit 2 forsoup eval unlearning) instead of being skipped with a warning (#1683).soup steer train --registry-idandsoup edit set --registry-idrecord the result on the registry entry. Both passed their output directory to the registry, which stores one hashed file per artifact and refuses a directory, so every run printedCould not attach to Registry: artifact not found: <dir>as a warning and exited 0 with nothing recorded.soup steer trainnow attaches the vector file (steering_vector.safetensors) assteering_vector, and a vector saved outside./steering/<name>is found by name through the entry;soup edit setattaches the weight file(s)save_pretrainedwrote asedited_model, orgrace_codebook.jsonasgrace_codebook. A requested attach is part of the command's success, as it already is forsoup bom emit,soup attest emitandsoup eval custom: a failed attach (unknown or ambiguous entry, unreadable registry, no weight file) prints an error and exits 1 with the result left on disk, andsoup edit set --registry-idwithout--outputis refused before the model is loaded (exit 2) instead of skipping the attach without a message (#1686).The Web UI's New Training page shows the run's log. The "Training Logs" card and the function that streams
/api/train/logsinto it were both there, but nothing called the function, so starting a run showedTraining started! PID: ...and never a line of output. The page now opens the card and follows the log when a run is started from it, and when it is opened or reloaded while a run is in progress (the server replays the log from its first line), including for a run that exits at once, whose output is the error to read. The progress card stays hidden: no route reports a total step count for the run the UI started, so its bar could not move (#1687).soup train --gpus Non Windows waits for theaccelerate launchit hands the run to and exits with the launcher's exit code. The hand-over usedos.execvpon every platform. On Linux and macOS that replaces the process, so the launcher's status is the command's; on Windows it starts the launcher as a new process and ends the calling one with status 0 at once, so the shell got its prompt back while the run was still going andsoup train --gpus 2 && next-steprannext-stepwhatever the launcher returned later. Windows now runs the launcher as a child (list argv, no shell), looked up in thePATHdirectories only, and Ctrl+C waits for the launcher's own shutdown. Linux and macOS are unchanged; an end-to-end test with a stand-inacceleratenow checks the exit code on whichever platform the suite runs on (#1688).DPO loads a vision-language wrapper checkpoint when
modality: visionis set (#1393 by @Srinivasan8888 in #1707).task: dpoalways builtAutoModelForCausalLM, so a base published as a vision-language wrapper (MiniMax-M3,model_type: minimax_m3_vl) could not load at all, while SFT trained its language tower through its vision path. Withmodality: vision, DPO now loadsAutoModelForImageTextToTextwith the model's processor, which trl'sDPOTrainertakes as its processing class for text-only preference rows, and the LoRA targets resolve to the language tower as before;soup recipes verifybuilds the same class, andtask: preferencewithpreference_loss: dpofollows it.backend: unslothdoes not readmodalityfor DPO and is refused at config load for that combination. The text path is unchanged.Unsloth migration reads the final trainer's config: task, output directory and hyperparameters come from the config passed to the last trainer, whether inline, named or positional. SFT warm-ups and unused later configs no longer overwrite stage settings. Warm-up
num_train_epochsand other settings no longer leak into the final stage; this removes inheritedtraining.epochs: 2from four public Qwen3 4B GRPO notebook variants (including AMD, Kaggle and Hugging Face Course). Earlier-stage warnings and config-only notebook behavior are preserved. (#1603 by @akkupratap323 in #1711)soup trainrefusesuse_liger/use_flash_attnunderstream_layers: true(#1692 by @Faisal-Fayaz in #1716). Both switches loaded, trained and said nothing, because they are read only by the resident setup: Liger patches the MLP forward in_setup_transformersand flash-attn is handed tofrom_pretrainedasattn_implementation. A streamed run goes to_setup_streaming_transformers, whose model is built byAutoModelForCausalLM.from_config(...)with no attention argument and no Liger patch, and_liger_appliedis never set, souse_liger_kernelmissesTrainingArgumentsas well. Since these are the switches a user reaches for when a streamed run will not fit, silently dropping them was the worst of the outcomes, sostream_layersnow names both fields in its conflict list at config load. Nothing is wired: this is the refusal half, and each switch is offered again once it is shown to engage on a streamed model.use_cut_cehas the same shape under streaming and stays on #1206, deliberately still loading here. The streamed preference tasks (dpo/kto/orpo/simpo) are unaffected: they already refuse both fields on their own terms, so no branch was needed for them.soup eval benchmark: do not store an unmeasured result as zero. Missing or scoreless requested results are skipped, named and counted; a genuine numeric zero is retained and booleans are not scores. The saved banner counts persisted results, and an all-skipped invocation exits 1. Groups require their own numeric entry; sub-task rows are not saved implicitly. (#1718 by @taljeon in #1722)training.moe_lora: truewithlora.use_vera: trueis refused on a fused-expert MoE (#1197 by @Srinivasan8888 in #1723). VeRA wrapsnn.Linearmodules only and has notarget_parameterspath, so on a transformers-5 MoE whose routed experts are 3-D tensors (Qwen3-MoE, Qwen2-MoE, Mixtral, MiniMax) the flag adapted the attention projections exactly as VeRA without it did, while printing the ScatterMoE targeting line. The combination now stops at setup, before the dropout rule, naming both flags, the fused parameters and the two ways out (use_vera: falseto adapt the experts with LoRA, or dropmoe_lora). A dense base and a checkpoint that keeps onenn.Linearper expert are not refused.Layer streaming now reads source safetensors headers during the disk pre-flight to charge each stripe drive the exact bytes of the decoder layers it receives, allocating non-decoder weights to the primary root at streaming dtype rather than splitting the shard estimate evenly (#1612 by @Nari-Ab in #1726).
soup eval human: reject non-object JSONL rows before loading either model. Report the physical line number and expected JSON object instead of passing a string or array through to a later TypeError. Object rows are unchanged. (#1725 by @taljeon in #1727)A QuEST resume from a format-v1 checkpoint no longer reads and re-hashes the whole local base a third time (#1199 by @Faisal-Fayaz in #1728).
validate_resume_metadatare-resolvedlegacy_base_modelto compare it against the identity this run had already computed, so a first v1 resume read every selected file of a local base three times instead of twice — about 3.9 s per GB on the maintainer's measurement, more on a busy machine. The identity is already incurrent["base_model"](the caller filled that field with it), so the extra pass rediscovered what it was handed. It now takes an optional pre-resolved identity andSFTTrainerWrapper.trainpasses the one it holds, which removes the pass. The value passed is the identity ofconfig.baseresolved before the load (_quest_base_identity_before), which is what the check compares against — notcurrent["base_model"], which on the continuation path is the artifact sidecar's recorded base and would make the check compare a value with itself. An identity that disagrees with the base still produces the sameQuEST original base model reference does not match current identityrefusal, and the default path for a caller that passes nothing is unchanged. Hub bases are unaffected either way, since their identity is the repo id; the pass that disappears is the one that cost something. Fix 1 of that issue (hash once and compare(st_dev, st_ino, st_size, st_mtime_ns)after the load) is deliberately not here: it would trade the content guarantee for a stat one, and the issue says to skip it if the stronger guarantee is wanted. Fix 3 already merged as #1588.training.rope_scaling_typeis refused at load where no setup applies it (#1697 by @ybai08 in #1730). The field loaded on all 23 tasks and every path, but only the transformers setups ofsft,ttsandpretrainread it; everywhere else the run trained on the checkpoint's own RoPE and printed nothing. This is a breaking change: a config that sets the field on one of these loaded before and now stops at load: any of the other 20 tasks;task: sftwithbackend: mlx,modality: vision,modality: audioortraining.stream_layers: true; andsft,ttsorpretrainwithbackend: unsloth. The message names the field, the task or every setting that leaves the reading setup, and what to change (for exampleUse backend: transformers, or remove the key.); ayarn_*key set beside it is named too, since it is legal only withrope_scaling_type: yarn.soup doctor --configexits 2. Removing the key changes nothing about what trains. Thelongcontexttemplate no longer suggests# backend: unsloth, which it would now be refused with. Text configs onbackend: transformers,pretrainonbackend: transformerswith any modality, and dumped configs (which write the field asnull) are unchanged.Custom tool-call subset scoring: do not credit malformed provided arguments. Values that fail to parse as a JSON object no longer earn empty-argument credit. Valid empty objects, omitted arguments, weights and the match threshold keep their existing treatment. (#1734 by @taljeon in #1737)
Quant-check rounds score-drop float noise before comparing verdict thresholds, so exact two- and five-percentage-point drops remain MINOR and MAJOR respectively without changing scores or thresholds. (#1740 by @taljeon in #1743)
training.quantization: fp8stops with the reason on transformers below 5.17.0 (#1731 by @ybai08 in #1745). Soup's declared floor is 5.16.1, and there the FP8 quantizer raisesAttributeError: 'NoneType' object has no attribute 'get'inupdate_tp_planwhen it loads a dense base, so anfp8run died inside transformers. The run now stops where the quantization config is built:quantization: fp8 needs transformers >= 5.17.0, and 5.16.1 is installed, with the cause andpip install -U 'transformers>=5.17.0'. Every other quantization mode is unchanged on 5.16.1. The Quant Menu row and the missing-class message said>= 4.45; both now say 5.17.0.soup trainsuggestsbackend: unslothonly for a config that would load with it (#1736 by @ybai08 in #1746). With unsloth installed, every run onbackend: transformerswas toldTip: unsloth is installed. Add backend: unsloth ..., including configs the loader refuses as soon as that line is added: a task with no unsloth setup,freeze_layers,rope_scaling_type,moe_lorarecipes. 38 of the 184 shipped templates and recipes on transformers got a tip they would be refused with. The tip now validates the config again with the backend changed and is printed only if that succeeds, so a new backend refusal is covered without a list intrain.py. Nothing else about the run changes, and a machine without unsloth validates nothing twice.Interrupted content-changing layer re-shards can no longer leave an old cache index pointing at partially rewritten files. Soup quarantines the old index before rewriting, reports unfinished re-shards and unindexed cache files, and keeps post-commit cleanup failures non-fatal. (#1616 by @Samearth17 in #1747)
deploy measuredelegates verdict classification toeval.quant_check.classify_delta, eliminating the duplicated ladder and rounding score-drop float noise so exact two- and five-percentage-point drops are now classified as MINOR and MAJOR respectively without changing scores or thresholds; caches written before this fix retain their stored verdict until cleared (#1744 by @kok-o in #1748).Close file descriptors before unlinking staged offline datasets on Windows so the original error is not masked by WinError 32 and no staging files are left behind (#1464 by @Guts1005 in #1753)
soup <command> --helpshows argument descriptions again on fresh installs: click is bounded below 8.5 (#1789). click 8.5.0 overwrites the help text typer gives each positional argument, so every positional argument in every--helppage lost its description (ADAPTER TEXT [required]with nothing after it) on any install that resolved the newest click; option help was unaffected. Soup now declaresclick>=8.0.0,<8.5, which it imports directly but had left to typer to pull in, and lifting the bound would need typer 0.26 or newer.
Security
soup uino longer keeps its auth token insessionStorageor onwindow._authToken(#1136 by @mohdUwaish59 in #1190). The Web UI now holds the token in a closure inapp.js, and every request goes through oneauthFetchwrapper that adds theAuthorizationheader without exposing the token to other page script. The token is deliberately not persisted: a reload or a new tab asks for it again (paste the tokensoup uiprinted), where before it was restored fromsessionStorage. This narrows the gap between a future rendering mistake and a token disclosure; it does not stop script already running in the page, for which the Content-Security-Policy remains the defence. A fresh desktop browser can now sign in at all:soup uiopens its URL without a token, and on 0.75.0 through 0.75.2 nothing in the page asked for one, so every panel stayed at 401; the page now prompts on the first 401 and accepts the pasted token or the wholeAuthorization: Bearer …line.authFetchalso refuses cross-origin URLs, so the header only ever goes to the UI's own origin. The chat request no longer sends an emptyBearerheader when no token is held.
0.75.2 - 2026-10-01
Changed
Outbound endpoint checks now refuse a private, link-local or reserved IP literal over
httpsas well ashttp. This coverssoup data generate --api-base, the vLLM provider and the commands that use it, judge URLs insoup eval judge, eval-gate suites,soup ship --judge-model/eval.ship.judge_modelandtraining.online_dpo_judge, and the Web UI chat proxy. Loopback still works; address a LAN server by its hostname.training.online_dpo_judgeis now checked when soup.yaml loads, not only when the trainer starts.soup data generate --api-baseand the Web UI chat proxy no longer treathttp://0.0.0.0as local; uselocalhostor127.0.0.1.100.64.0.0/10andfec0::/10literals are now treated as non-public by every check that already refused private addresses (webhooks, the OTLP endpoint, telemetry, hub endpoints over HTTP, andsoup ingest --pull, where--allow-private-hostadmits them).Every check that refuses private addresses now also recognizes an IP literal written with non-ASCII digits or label separators, which an HTTP client folds to ASCII before it connects.
Fixed
soup shipchecks--judge-modelbefore it builds the base and tuned models; it used to load both and only then stop with a usage error.The Web UI chat proxy answers 400, not 500, for an endpoint URL that does not parse.
Security
- Hardening of outbound endpoint validation — GHSA-9f49-hqpf-c849. The advisory has the details and upgrade guidance.
0.75.1 - 2026-09-21
Fixed
A layer-streamed LoRA adapter saved as ZERO tensors under peft 0.21.0 (#1005 in #1010). peft 0.21 selects an adapter's tensors by the prefixes it reads off
model.named_modules()and keeps the matchingstate_dict()entries, where 0.20 filtered by thelora_substring; the streaming wrapper's module and parameter names carried.inner.while itsstate_dict()keys were canonical, sotrainer.save_model(), everysave_stepscheckpoint andget_peft_model_state_dict()returned nothing, and nothing raised. The wrapper now enumerates the inner layer's tree at its own prefix (named_modules()/named_children(), withtrain()/apply()/_apply()reaching the inner layer explicitly), so names and keys are one spelling — which also keeps transformers' weight-decay grouping, built fromnamed_children(), from silently dropping every LoRA parameter into the no-decay group. Any adapter saved withstream_layers: trueon peft>=0.21 before this fix holds zero tensors (a 40-byteadapter_model.safetensors, no error at save time) and cannot be recovered — re-train it on a Soup with this fix, or withpeft<0.21. Naming only; nothing on the streamed forward path changes. Ten peft-version-independent tests pin names == keys against a resident LoRA model; the four tests that documented the old asymmetry are inverted rather than deleted. Nopeft<0.21pin.A Ctrl+C arriving immediately after the Lambda controller started could still leave the instance running (#1073 in #1078).
submit_lambda_runspawned the controller outside thetrythat absorbs interrupts, so a SIGINT delivered betweenPopenreturning and the waiting loop being entered — or one landing on the "still waiting" notice, which is written from the interrupt handler — unwound the parent while the controller was still terminating the paid instance in itsfinallyblock. The spawn and the notice now sit inside the guarded region: once the child exists, every path leads back toproc.wait()and the controller's exit code remains what the function returns. An interrupt raised before the child exists still propagates unchanged, and await()failure that is not an interrupt still surfaces rather than being retried.soup runs cleannow accepts--no-keep-weightsto delete whole non-best checkpoints;--keep-weights(the default) keeps weights on every supported Click version — on Click 8.1 it previously deleted whole checkpoints (#1057)Cloud runs keep their outputs and their cleanup (#1058).
soup train --cloud modalnow writes run outputs to thesoup-outputsModal volume and downloads them to the local output directory when the run ends (also after a failed run); previously checkpoints were lost with the container.soup train --cloud lambda --cloud-submit: pressing Ctrl+C now waits for the controller to terminate the Lambda instance; previously the controller was killed 0.25 s later and the instance could keep running (Linux/macOS).
Security
- Hardening across the CLI, the inference server, the Web UI, MCP execution, config parsing and
.canhandling — GHSA-63h4-gvp4-r26g. Eight issues: Hugging Face credentials presented to a non-HF hub andOPENAI_API_KEYpresented to any https judge host;soup adapters verify --public-keyaccepting a record whose backend is not ed25519; the Web UI rendering dataset and run content as HTML with no Content-Security-Policy;soup servetool and adapter routes checking neitherHostnorOrigin;soup mcp serve --allow-executenot pinning every approved input; config regexes able to hang loading or training (the old guard ran the untrusted pattern against a probe); credential option values written to the audit log; and unbounded reads and YAML alias expansion when reading an untrusted.can. The advisory carries the affected range of each issue and says which credentials to rotate.
0.75.0 - 2026-09-12
Added
Wire opt-in anonymous hardware-only telemetry behind
SOUP_TELEMETRY=1with stdlib delivery,--no-telemetryflag, command sanitization, and public privacy policy (#318 by @kok-o in #529).Added a ready-made
deepseek-v4-flash-dporecipe for deepseek-ai/DeepSeek-V4-Flash (#275 by @Srinivasan8888 in #662). The base already shipped SFT (v0.71.24) and GRPO (#279 by @Faisal-Fayaz); this completes the trio with the preference shape (task: dpo,format: dpo,preference_train.jsonl,dpo_beta: 0.1,lr: 5e-6) over the MoE geometry its SFT sibling already uses (LoRA r16/a32,batch_size: auto,gradient_accumulation_steps: 8, 4-bit,moe_lora,moe_aux_loss_coeff) — deliberately not the GRPO sibling's1/16, whose batch shape is driven by rollout generation that DPO does not do.lr: 5e-6anddpo_beta: 0.1are unanimous across all 12 DPO recipes in the catalog. The recipe is not trained — DeepSeek-V4-Flash is MoE — so no hyperparameter here is a measured recommendation. Catalog count 163 -> 164.Added a ready-made
kimi-k2.6-dporecipe for moonshotai/Kimi-K2.6 (#275 by @kok-o in #664). The base already shipped SFT (v0.71.24) and GRPO (#614 by @umran666); this completes the trio with the direct preference optimization (DPO) shape (task: dpo,preference_train.jsonl,dpo_beta: 0.1,lr: 5e-6) over the ~1T MoE geometry its siblings already use (LoRA r32/a64,batch_size: 1,gradient_accumulation_steps: 16, 4-bit,moe_lora,gradient_checkpointing,max_length: 8192). The recipe is not trained — ~1T MoE is multi-node — so no hyperparameter here is a measured recommendation. Catalog count 164 -> 165.The MLX backend now drives Soup's live terminal dashboard, the
soup uistream and the experiment tracker (#23 by @Srinivasan8888 in #665).MLXSFTTrainerWrapper.train()accepteddisplay/tracker/run_idand discarded them, so an MLX run showed mlx-lm's raw stdout while the same config on the transformers path got the Rich panel. It is an adapter rather than a reuse ofSoupTrainerCallback— mlx-lm exposes onlyon_train_loss_report/on_val_loss_report, so bridging through a HuggingFaceTrainerCallbackwould have meant faking trainer state.speedcarriesiterations_per_secondto match the transformers path'strain_steps_per_second, because the panel hard-labels that fieldit/s;grad_normis deliberately left unset, since mlx-lm does not compute one and a field reading a plausible0.0on one backend is worse than an absent one. Verified against real mlx-lm on an 8 GB M1.Validation loss is now recorded and displayed (#23 by @Srinivasan8888 in #713). Soup never stored it on any backend:
SoupTrainerCallback.on_logreadlogs["loss"]and neverlogs["eval_loss"], so an evaluation step left the last training loss in place and re-reported it to every sink — the evaluated number existed nowhere. There was also nowhere to put it: nometricscolumn, noTrainEventfield, andTrainingDisplay.update()reads onlygrad_norm,speedandgpu_memout of**kwargs. The live panel, themetricstable and the/api/train/streamSSE frame now each carryval_lossas its own series, never folded intoloss, and the panel gains aVal lossrow. The panel carries the last measured value forward between evaluations; the stored and streamed values do not — a step where no evaluation ran recordsNULL, so a series read back has one point per measurement rather than one per logged step. Existing~/.soup/experiments.dbfiles are migrated in place; rows written before this ship readNULLrather than a fabricated0.0, because an unmeasured value must not read as a measured one. The MLX producer follows separately.MLX runs now report validation loss (#23 by @Srinivasan8888 in #739). mlx-lm's
on_val_loss_reporthook was never implemented, so it was called on every evaluation and did nothing — the base-class method is apass, making it a silent no-op rather than an error. It now drives all three sinks #713 added: the live display, themetrics.val_losscolumn, and theval_lossfield on the SSE stream. The last measured value stays on the panel between evaluations, but only real measurements are written to the database or the wire, so a run records one row per evaluation rather than one per training step. Upstream'siterationis recorded as sent, including its off-by-one against the training counter, so a row matches mlx-lm's own log line. A payload without aval_losskey writes no row, emits no event, and does not clear an earlier measurement. Verified on Apple Silicon with a real validation split: 9 display updates, 5 database rows and 5 stream events for 5 evaluations.Proved LoRA+ optimizer and scheduler state survive save/resume (#724 by @SID-6921 in #747). #738 fixed
training.loraplus_lr_ratiocrashing on the SFT, pretrain and embedding wrappers, but its last acceptance criterion — that resuming a LoRA+ run preserves the optimizer's momentum and the scheduler's decayed learning rate — was never exercised. A new test runs a real 4-step training loop twice against the same seeded base model and dataset, once straight through and once interrupted at a real checkpoint and resumed, both targeting the samemax_steps; the optimizer momentum, scheduler LR, and final LoRA weights all match exactly between the two. No production code changed: the existingattach_loraplus_optimizerwiring already handled this correctly, it just wasn't proven.soup doctor --config soup.yamlreports the settings your backend does not read (#755 by @Srinivasan8888 in #756). A field can be declared, validated and documented and still be read by nothing on the backend you chose — filed one field at a time as #683, #686, #745 and #749 — andsoup doctorcould not help, because it read no config at all. It now loads the config and lists only the settings that config actually writes which its task and backend do not read, with the reason and, where one exists, the issue that recorded it;model_fields_setseparates those from the fields sitting at their schema default. Backend support is declared rather than inferred, because inference does not work: reachability over the import graph detects none of five independently-known MLX gaps, reads of the trainer module alone invent gaps for fields that live in helper modules, and--dry-runexits before a trainer is ever constructed. Scope is deliberately narrow:task=sftonbackend=mlx, seven entries, every one of them a settingtrainer/mlx_sft.pyalready warns about at runtime — the value added is that you see it before the run rather than during it, not that it knows anything new. Five further entries were removed before merge because #734 and #750 wired those fields; the guard shipped here is what caught them. Every other task/backend pair reports nothing rather than guessing. A guard keeps the declared table honest in both directions — an entry marked unread fails once any declared module reads the field, an entry marked read-to-warn fails once the warning is deleted — scanning the trainer and the helpers it delegates to, because the fields #734 wired live mostly inmlx_optim.py.soup doctor --configexits 2 when the config cannot be read, so the leg can gate CI.A Turkish README, behind a gate that checks what a translation was synced from, not only when (#852 by @Ercaner1988, supersedes #774 and #769).
README.tr.mdis a full translation of the currentREADME.md, linked from a language banner above the logo that lists only languages that exist.tests/test_readme_translation_sync.pyturns CI red when a translation drifts: everyREADME.*.mdmust carry asynced-fromstamp matchingsha256(README.md)over LF-normalised bytes, so CRLF checkouts on Windows agree with Linux and macOS; files are found with a glob, so a new language cannot land unchecked; and each##section must keepREADME.md's fenced-code languages, external URLs and inline-code spans, in both directions. That last check is what a stamp alone could not do — a different document can carry a valid stamp — and run against the previous draft of the Turkish file it found 29 dropped and 3 invented code references, including two release-note bullets that do not exist inREADME.md. A control suite breaks one thing at a time in a temp tree to prove each lock can fail.soup ingest --source langfuse --pullfetches generations straight from Langfuse (#204 by @Konuktor in #859). No export step: one row perGENERATIONobservation in a--sincewindow (default7d), read from Langfuse's Observations API v2 —/api/public/tracesleaves Langfuse Cloud on 2026-11-16 — and handed to the existingparse_langfuse, with chat message lists joined the way it already joins{"messages": [...]}. Credentials come fromLANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY(plusLANGFUSE_HOSTorLANGFUSE_BASE_URL), never a flag, and stay out of the output, console, debug logs, audit log and error messages. The host is HTTPS-only through the same SSRF validator as--slack-url, with--allow-private-hostfor self-hosted Langfuse; redirects are refused. Every loop is bounded: 30 s per request, 64 MiB per response, a--max-pagescap (default 100) that exits 1 and writes nothing instead of truncating, and 429 retries honouringRetry-Afterup to 60 s. Standard-library HTTPS, so no new dependency; without--pullnothing changes and the pull code is not imported. The hint the local-export path prints namedLANGFUSE_KEY, a variable Langfuse does not read; it now names the key pair. Validated end to end against a Langfuse Cloud Hobby project. LangSmith, Helicone, OpenPipe and OpenAI Stored Completions stay open on #204.Added the
qwen3.5-0.8b-grpoandqwen3.5-2b-grporecipes (#848 by @SID-6921 in #864).Qwen/Qwen3.5-0.8BandQwen/Qwen3.5-2Bpreviously shipped SFT only; both models are small enough that a contributor might actually run a few steps of GRPO reasoning training on their own hardware. Learning rate (1e-5), LoRA rank (r=16/alpha=32), andmax_length: 4096follow the existing GRPO templates rather than the SFT siblings' values, stated explicitly rather than inherited silently. Catalog count: 165 -> 167.
Changed
The README now leads with
pipx, and names PEP 668 by its error text (#671 by @swalla02 in #673).pip install soup-cli, the first command in the README, fails on Debian 12 and Ubuntu 23.04 or later:error: externally-managed-environmentThose distros ship an
EXTERNALLY-MANAGEDmarker in the system Python so pip refuses to write into asite-packagesthataptalso manages (PEP 668). It is the default on current Debian and Ubuntu, so it is now the ordinary first-run experience on Linux rather than an edge case, and nothing in Soup can rescue the command: the package is not installed yet when the README line runs.Soup declares a
soupconsole script, which makes it an application rather than a library, so the install now leads withpipx install soup-clianduv tool install soup-cli. Both give Soup its own virtualenv and put the command onPATH, which is outside PEP 668's scope entirely. Every existingpip installline is kept, under a sentence saying when it is the right choice: already inside a virtualenv, a Colab notebook, or a Docker image. A callout carries the literalexternally-managed-environmentstring so the error text pasted into a search engine matches Soup's own README, and it also gives thepython3 -m venvroute for anyone who would rather not add a tool.Use
piprather thanpipxif you want toimport soup_clialongside your own code; pipx isolation is the wrong shape for that.Documentation only. No packaging or command behaviour changed.
Multipack FFD placement is O(N log N) instead of O(N^2) (#694 by @dchaudhari7177 in #726). Preparing a multipack epoch scanned every open bin for each item; placement now descends a segment tree of per-bin remaining capacity. The packing is unchanged, which is the property that matters — altering which item lands in which bin would silently change every multipack training run — and that was verified at merge against the pre-change function over 6,000 randomized cases with 5,384 of them containing repeated lengths, for zero mismatches. Measured on one box: 5.5x faster at 1,000 rows, 37x at 10,000, and 96x at 30,000, with per-10x-N scaling dropping from 104-130x to 8.9-13.9x. The cap error message no longer describes the algorithm as quadratic, because it no longer is.
Sequence-level GSPO objective replaces legacy column-centering (#744 by @kok-o, closes #723). Upgraded
grpo_variant: gspofrom the legacy column-centering heuristic to the published Group Sequence Policy Optimization algorithm (Qwen Team, arXiv:2507.18071). Because the mathematical objective is replaced at the sequence level, existing configurations specifyinggrpo_variant: gspowill yield different numerical losses and gradients on upgrade compared to prior versions and will not reproduce prior runs. The objective now computes a length-normalized sequence importance ratio $s_i = \exp\left(\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} (\log p_\text{new} - \log p_\text{old})\right)$ and applies sequence-level surrogate clipping with default radius $\varepsilon = 0.2$ or operator-suppliedgrpo_delta, strictly isolating masked tokens with 0.0 gradient, remaining invariant to padding tokens and placement (left vs right), and respecting batch permutation invariance.Optimize dataset validator (
soup data validate/validate_and_stats) (#771 by @iam-saiteja). Consolidates statistics aggregation into a single pass and replaces string serialization with fast hashable row signatures for faster dataset inspection (~1.25x–1.60x speedup).Breaking: an unknown config key now refuses the load (#627 deadline, in #879). v0.74.0 reported every key no config model declares — a typo like
quantizaton, or a field that only exists on a newer Soup — as a warning that named v0.75 as the release that would start refusing, andTestTheDeadlinepins that promise to the declared version in both directions. This is that release:soup trainexits 1 before the training stack is imported, and the API / Web UI loader raisesValueErrorwith the same text, naming the field you probably meant. Nothing is defaulted or substituted. The refusing message saysRefused.rather thanNot applied., which would have read as if the run went ahead without the key; thesoup sweepguard, which always refused, changes wording the same way. The detector now sees a config the waySoupConfigdoes: a root-levellora:block — the LlamaFactory / Axolotl spelling the schema has accepted and moved undertrainingsince v0.40.1 — is remapped through the same function before the walk, so a spelling the validator accepts is never one the detector refuses (the release review caught the first draft refusing it). Twosoup fetch examplesfiles (llama-3.1-8b-lora,qwen2.5-7b-dpo) used that spelling and were moved to the canonicaltraining.lora; every recipe, template and example config scans clean. Two more findings from the same review ride along: key names reachedconsole.printunescaped (Rich markup and raw ESC bytes from a YAML key could restyle or spoof the terminal — the class six command modules already guard privately; the loader now uses the sharedsoup_cli.utils.terminal.for_terminal), and the scan was unbounded (50,000 bogus keys cost ~31 s of CPU per Web UI request, measured; it now stops at 100 findings and says so). The Web UI's/api/train/startand/api/config/from-formreturn the loader's message — the key and the suggestion — instead of a generic "invalid configuration", rendered through the SPA'sescapeHtml(). A config that must stay loadable on v0.74 as well needs the key removed, not renamed.
Fixed
Direct AWQ exports now require an explicit, usable calibration JSONL before AutoAWQ is imported or the model is loaded, preventing its large default calibration dataset from being downloaded silently (#338 by @Faisal-Fayaz in #592).
GEMM throughput forecasts now use the card's resolved stream dtype (#617 by @Samearth17 in #648). The pre-flight measurement no longer hard-codes bf16, so pre-Ampere cards are benchmarked with the fp16 dtype they actually use. The forecast also reports the measured dtype alongside TFLOPS and clock.
Repaired non-existent Hugging Face repository IDs in MLX catalog recipes (#661 by @kok-o in #666).
qwen3-8b-sft-mlxpreviously pointed tomlx-community/Qwen3-8B-Instruct-4bit(which huggingface_hub reports as RepositoryNotFoundError); it now referencesmlx-community/Qwen3-8B-4bit.gemma3-9b-sft-mlxis renamed togemma3-4b-sft-mlx(user-visible rename; the old 9B size does not exist upstream) pointing tomlx-community/gemma-3-4b-it-4bit(size 4B, M1+ 16GB).A non-dict message in a
multimodalrow aborted the whole dataset load instead of dropping the row (#670 by @abdulwaarith0)._convert_multimodaliteratesrow["messages"]and callsmsg.get("content")without first checking thatmsgis a dict, so a row like{"messages": ["hello"]}— a bare string,None, an int, or a nested list where an object belongs — raisedAttributeError: 'str' object has no attribute 'get'.format_to_messagescatchesKeyError,TypeError,IndexErrorandValueErrorto route malformed rows to its drop path, andAttributeErroris in none of them, so the exception escaped the converter and killed the run: one bad line in a large JSONL file took the entire dataset with it rather than being skipped like every other malformed row. The converter already guarded the parts inside a message (multimodal content part must be a dict) but not the message itself. It now raisesValueErrorfor a non-dict message, which the existing drop path already understands, so such rows are dropped and the rest of the dataset loads. Valid rows are unaffected: typed content-part lists and the plain-string back-compat path both convert exactly as before.soup migrateno longer refuses a valid competitor config just because the file is named*.jsonl. The guard branched on the filename alone; the content sniff written to gate it,_looks_like_jsonl(), had no call site, so a YAML config saved asconfig.jsonlexited 2 with "got JSONL" and could not be migrated at all. Both conditions are now required, which leaves.ipynbnotebooks (which legitimately start with{) untouched. The N2 test now drives the command throughCliRunnerrather than calling the helper, so the wiring is what is actually asserted (#675 by @swalla02, closes #672). The sniff also readsutf-8-sigrather thanutf-8, so a UTF-8 BOM (which Windows tooling writes by default) cannot defeat it, and its read is bounded so a file with no newline is not pulled into memory entire.chatml / audio / video converters silently kept a non-dict message instead of dropping the row (#676 by @BetterAndBetterII in #678).
format_to_messagesdocuments a drop contract: a malformed row returnsNoneso one bad JSONL line is skipped rather than corrupting the dataset._convert_chatml,_convert_audioand_convert_videopassed a non-dictmessageselement through verbatim, so{"messages": ["hello"]}— or42, a nested list, orNonewhere a message object belongs — survived into training.chatmlis whatdetect_formatreturns for a bare{"messages": [...]}row, so this was the default path for the most common dataset shape. Each converter now raisesValueErrorfor a non-dict message, which the existing drop path already understands. The list-type guard from @v01dst in #679 also rejects empty non-list values such as""and{}; regression tests pin both shapes. Video rows with missing or nullmessagesstill normalize to an empty list. The shared conversion-based validator from #712 uses these guards too, sosoup data validateandload_datasetagree on which rows are dropped. Valid rows convert exactly as before.MLX SFT ignored
training.gradient_accumulation_steps(#684 by @AmirF194 in #696). The wrapper built mlx-lm'sTrainingArgswithout agrad_accumulation_stepskwarg, so mlx-lm's own dataclass default of 1 always applied regardless of the configured value, silently changing the effective batch size and optimizer-update cadence on every MLX run. The writtenadapter_config.jsonalso hardcodedgrad_accumulation_steps: 1, so the drift could not be detected from the output afterwards either. Both now carry the valuetraining.gradient_accumulation_stepsactually resolves to.itersis also rounded down to a whole number of accumulation groups (kept at a minimum of one group), because mlx-lm only updates the optimizer onit % accum == 0and never flushes a trailing partial group; without this a dataset smaller than the accumulation window trained for zero optimizer updates.Anyone who never touched this field will see a different effective batch size after upgrading.
gradient_accumulation_stepsdefaults to 4, so an MLX run that used to update on every micro-batch now accumulates over 4 before updating: a 4x larger effective batch size and roughly a quarter as many optimizer updates for the sameiters. This is the correct value for the schema default; a differently-converging run after this upgrade is expected, not a regression.MLX SFT ignored
training.gradient_checkpointing(#685 by @AmirF194 in #698). The wrapper built mlx-lm'sTrainingArgswithout agrad_checkpointkwarg, so mlx-lm's own dataclass default ofFalsealways applied regardless of the configured value, silently giving up the memory savings a user enabled checkpointing for. The writtenadapter_config.jsonalso hardcodedgrad_checkpoint: false, so the drift could not be detected from the output afterwards either.gradient_checkpointingaccepts a bool or one of theselective/medium/full/autotiers; mlx-lm has only a single on/off switch, so any non-Falsetier now resolves toTruethere, the samebool()coercion already used for this field on the other trainer backends. A tier value now also prints the sameMLX backend ignores: ...advisory this file already uses for other flattened options, since the granularity itself is silently dropped even though checkpointing is enabled.data.interleave'sover/probsstrategies could place the same row (a source's original plus itsover/probs-padded copy) in bothtrainandval;val_splitnow runs before the padding, per source, on the eager local-file and HF-hub interleave paths (the streaming path is unaffected and still has this leak). The requestedval_splitfraction is no longer exact on those two paths underover/probs, since it is now taken from each source's smaller, unpadded row count;concat/underare unaffected either way. Seedocs/data.md's interleave section for the numbers (#680 by @AmirF194 in #701).The MLX SFT benchmark harness now exercises the display/tracker bridge (#703 by @Shutaru; Refs #23). It passes a real Rich display and an artifact-local SQLite tracker to the MLX wrapper, and rejects runs without display updates or recorded metrics. Training stays inside the stdout tee so whole-run throughput still comes from mlx-lm's trained-token counter, with ANSI controls and wrapped whitespace normalised before parsing the final report. Regression tests cover the wiring, terminal output, failure/interruption cleanup, and disconnected telemetry. The stdout tee preserves terminal detection so the Rich panel can refresh during an interactive run; all terminal widths from 20 to 200 are exercised by the regression test. The existing Apple Silicon measurements are unchanged; a new hardware run with the bridge enabled remains outstanding.
wasserstein/topk_aligncross-tokenizer distillation trained on logits the teacher computed for the wrong text, with no visible failure (#704 by @AmirF194, closes #681). Both strategies forwarded the student's owninput_idsstraight to the teacher, clamped into its vocab range to avoid an index error. Clamping does not translate tokens between vocabularies, so a mismatched pair silently fed the teacher garbled or wrong text while still reporting a finite, plausible-looking loss.wasserstein/topk_alignnow refuse a tokenizer pair that isn't interchangeable atsetup(), namingwasserstein_aligned(which already re-tokenizes correctly) as the working alternative for genuinely different tokenizers. The same-tokenizer fast path (e.g. two sizes in one family) is unaffected.Fixed
ValueError: Lora rank r must be > 0when training embeddings withlora.r: 0(full fine-tuning) ontask: embedding(#700 by @kok-o in #705)._setup_transformers()now skips PEFT wrapping whenr == 0, and_EmbeddingTrainerdelegatesmodelandargsattributes.Prevented training subprocess hang in Web UI when no client drains stdout (#688 by @kok-o in #706). Output is now continuously consumed by a background thread into a bounded ring buffer, allowing training to proceed regardless of subscriber state and supporting multi-subscriber replay via
Last-Event-ID.Required authentication on Web UI read endpoints and SSE streams (#687 by @kok-o in #707). Run configurations, logs, and system metrics now require Bearer authorization. SSE endpoints authenticate via short-lived, single-use tickets exchanged over an authenticated POST, avoiding durable tokens in query strings. Non-loopback binding requires a valid authentication token and is rejected with exit code 2 if missing or invalid.
task: embeddingon the transformers backend no longer crashes after setup (#708 by @jagadeepmamidi, closes #690)._EmbeddingTraineralready delegatedtrain/save_model/add_callback/statebut notmodelorargs, whichtrain()reads for fp16 dtype alignment.SFT and pretrain
packing: trueno longer raiseTypeErroron TRL 0.29 (#709 by @jagadeepmamidi, closes #691).packingis anSFTConfigfield; passing it as anSFTTrainerkwarg is rejected because__init__has no**kwargs. Soup now carries it through_as_sft_configon both trainers.packing_cross_doc_attn_maskis schema-rejected: it never mapped to a valid TRLpacking_strategy(allowlist isbfd/bfd-requeue/wrapped). Usepacking: truewith FlashAttention instead.data.streamingis honoured for a single HuggingFace Hub dataset name (#710 by @jagadeepmamidi, Refs #689)._load_one_hub_datasetnow forwardsstreaming=True(andbuffer_sizeshuffle) todatasets.load_dataset, then materialises up toMAX_REMOTE_ROWS. Previously the flag was ignored and the full dataset was materialised; a Hub source over 1,000,000 rows is now cut to 1M with a warning. An all-hub interleave list with streaming stays schema-rejected.The ULD distillation loss (
wasserstein/topk_align/wasserstein_aligned) ignored the response-only label mask and the causal shift the CE term already applies, so under the defaulttrain_on_responses_onlyit optimized prompt tokens and the shift-boundary position too (#711 by @AmirF194, closes #682).uld_distill_lossanduld_aligned_lossnow take an optionallabelsargument and mask onlabels != -100after the same shift_compute_distill_termuses.soup data validategreen-lit rows thatload_datasetthen drops, because the validator never ran the converters (#712 by @abdulwaarith0).validate_and_statsjudged a row by top-level key presence againstFORMAT_SIGNATURES, while the loader runsformat_to_messagesand drops any row that returnsNone. Those are different notions of "valid", so they disagreed two ways: the six formats absent fromFORMAT_SIGNATURES(prm,pre_tokenized,input_output,video,multimodal,raft) skipped validation entirely and reported every row valid regardless of content, and for the formats that were checked, key presence passed rows a converter drops on a value — a null field, a non-dict message.soup data validatecould report a file fully valid that the loader could not load, moving the failure from validate time into the middle of a training run.valid_rowsis now computed by running the real conversion path per row, so agreement is structural rather than a second, weaker check that drifts; every format is covered, including the six that were skipped. When a row is dropped, the converter's own reason is surfaced for the first few offenders (row 3: chatml message must be a dict) so the report says which rows and why, not only a count. Exit-code behaviour is unchanged:validatestill exits 0 even at0/N, so thesoup ci initgate is unaffected; whether a0/Nfile should fail the gate is left to a separate change. On the repo's own datasets this shifts 120 of 360 file-by-format verdicts, every one an over-count on a file that genuinely has zero valid rows in that format, with no exit code moved.#651: Raise the minimum supported PyTorch version to 2.6.0 because TRL 0.29 preference trainers require the public FSDP2 API. (#651 by @Samearth17 in #717)
MLX SFT now honours
data.train_on_responses_only(#683 by @Srinivasan8888 in #733). The option defaults totrueand reached nothing on the MLX path: no mask was passed to mlx-lm's dataset constructor, so every MLX SFT run trained on system and user turns against the documented default, and adapter metadata recordedmask_prompt: falseregardless. Setting upstream's flag is not the fix —ChatDatasetmasks a single prefix ending before the last message, so on multi-turn chat it supervises only the final assistant turn. Measured on Apple Silicon over 16 two-turn conversations: 772 supervised tokens before, 71 with upstream's flag, 146 with a correct mask. Chat rows now use a Soup per-token mask injected throughtrain(loss=..., iterate_batches=...); prompt/completion rows use upstream's flag, which is correct for that shape; plain-text rows warn instead, because upstream raises when the flag is set on them. Non-prefix-stable chat templates, conversations with no assistant content, and conversations opening on an assistant turn are refused explicitly rather than approximated. This does not make the two backends comparable: MLX now excludes the assistant header from the loss and refuses templates it cannot align, while the transformers path supervises the header and, on thinking-style templates such as Qwen3's, supervises a user-turn fragment. MLX is the stricter of the two; the transformers behaviour is filed separately. A template MLX cannot mask -- Qwen3's, which injects its thinking block only for the last assistant turn -- is now refused at dataset construction, before the training loop starts, with a message namingdata.train_on_responses_only: falseas the remedy. One shipped recipe is affected.qwen3-8b-sft-mlxdoes not setdata.train_on_responses_only, so it takes thetruedefault: measured against the realQwen/Qwen3-0.6Btokenizer, single-turn and system+single-turn rows train (8 supervised tokens, the thinking block included), and multi-turn rows now refuse where they previously trained on the prompt. Setdata.train_on_responses_only: falseto restore the old behaviour on multi-turn Qwen3 data. The other two MLX recipes are unaffected:llama3.1-8b-sft-mlxandgemma3-4b-sft-mlxmask correctly on both shapes, verified againstmlx-community/gemma-3-4b-it-4bititself.MLX SFT now honours
warmup_ratio,scheduler,weight_decayandoptimizer(#686 by @Srinivasan8888 in #734). The backend builtAdamW(learning_rate=<scalar>)and nothing else, so all four were validated, accepted and dropped without a warning — an MLX run silently trained a different recipe from the configured one, and only 8 of the 32 names on Soup's optimizer allowlist have an MLX equivalent while all 32 became AdamW. Schedules are now composed frommlx.optimizersin optimizer-update units (iters // gradient_accumulation_steps), which is what MLX drives a callable learning rate from; building against iterations would stretch the warmup by the accumulation factor and never reach the cosine floor. Measured on Apple Silicon, acosinerun withwarmup_ratio=0.2now reports 10 distinct learning rates across 40 iterations where the old path reported one. Unsupported optimizer and scheduler names are refused by name rather than silently becoming AdamW, weight decay on an optimizer that cannot take it is refused rather than dropped, awarmup_ratiothat rounds to zero updates says so, and adapter metadata records the effective optimizer and schedule.grpo_variant: gspocentered the per-token log-ratio across the batch before the completion mask was applied, so a masked (padding) token shifted the loss and gradient of every unmasked row sharing its column, and picked up a non-zero gradient of its own (#735 by @AmirF194, refs #723). The batch-mean statistic is now taken over unmasked positions only.PPO now forwards
training.epochsandppo_kl_penaltyto TRL 0.29'snum_train_epochsandkl_coeffields while retaining legacy field-name compatibility (#721 by @be-student in #737).training.loraplus_lr_ratiocrashed every affected run instead of enabling LoRA+ (#738 by @abdulwaarith0). The SFT, pretrain and embedding wrappers insertedloraplus_lr_ratiointotraining_kwargsand forwarded it toTrainingArguments(**training_kwargs). It is not aTrainingArgumentsfield — it belongs to PEFT's optimizer construction — so an advertised, schema-accepted option raisedTypeError: __init__() got an unexpected keyword argument 'loraplus_lr_ratio'before the first training step. It is now routed through a sharedattach_loraplus_optimizerhelper that builds acreate_loraplus_optimizeroptimizer (LoRA B matrices atlr × ratio, A atlr) and assigns it totrainer.optimizerafter the trainer is built — respected becauseTrainer.create_optimizeronly builds one whenself.optimizer is None, with the scheduler still built from it using the configured warmup/schedule. The optimizer class and its betas/eps come from the run's configured optimizer viaTrainer.get_optimizer_cls_and_kwargs, so LoRA+ uses the optimizer the user asked for, and weight decay is applied through PEFT's ownloraplus_weight_decay. The combination with GaLore is now rejected with a clear message rather than silently overriding the GaLore optimizer, andloraplus_lr_ratioon a non-LoRA run is rejected instead of silently doing nothing.MLX SFT now honours
training.max_grad_norm(#749 by @Srinivasan8888 in #750). The field is forwarded intoTrainingArgumentsby sixteen transformers trainers, so the documented default clips every transformers run at 1.0 — and onbackend: mlxit reached nothing: no MLX file read it,mlx_lm'sTrainingArgshas no such field and its trainer clips nowhere, and theMLX backend ignores:line did not mention it. The same config therefore trained clipped on one backend and unclipped on the other, silently. Gradients are now clipped throughmlx.optimizers.clip_grad_normby an optimizer that clips before delegating, which reaches mlx-lm's singleoptimizer.update(model, grad)call site without forking its training loop;optimizer.stateandoptimizer.learning_ratestill delegate, because upstream reads both off the object it is handed. The norm that ran is recorded inadapter_config.json. Measured on an M1 inside upstream's own compiled-step shape, one step on an ill-conditioned batch moved SGD weights by 388.03 unclipped versus 0.0085 atmax_grad_norm: 1.0; AdamW moves 0.126505 versus 0.126504, because Adam normalises by its second moment — so this rescuesoptimizer: sgd/lionand gradient spikes rather than every MLX run.soup eval autono longer dies with aTypeErrorafter a successful eval (#752 by @Srinivasan8888 in #754). Typer fills a command's parameters only when typer invokes it; called as a plain Python function, an unpassed parameter keeps its literaltyper.models.OptionInfodefault, which is truthy.auto()passed three ofcustom()'s five typer parameters, sooutputandattach_to_registryarrived asOptionInfo: the--output/--attach-to-registryblock fired on every run though neither was requested,output or "eval_results.json"evaluated to theOptionInforather than the filename, andwrite_eval_jsonraisedTypeError: expected str, bytes or os.PathLike object, not OptionInfo— uncaught, after the eval had already run and saved its results to the tracker. The same call inmonitoring/callback.pyis wrapped inexcept Exception, so mid-training auto-eval witheval.custom_tasksconfigured reportedAuto-eval custom failed: ... not OptionInfoinstead. Both call sites now pass every typer parameter, and a guard fails the suite when any bare-name call to a typer command leaves one unfilled — resolving each callee to a declaration in the calling module or afrom X import name, rather than by bare name, which had merged everymain()in the tree and flaggedautodistill/mlx_worker.py's argparse entry point with fifteen options it does not declare.Inference prompts are now encoded the way Soup trains them (#781 by @Konuktor in #782).
soup chat,soup serve(transformers backend and--mole),soup infer/soup bench,soup diff, thelive_evalgenerators behind the eval gate,soup ship,soup diagnoseandsoup advise, andsoup data generate's local provider rendered the chat template to text and then let the tokenizer add its own special tokens, so templates that render{{ bos_token }}(Llama-3, Gemma, Mistral) sent a doubled BOS while Soup's default SFT path trains with one. A rendered prompt is now encoded with no tokenizer special tokens, matching training andapply_chat_template(tokenize=True): 2 → 1 BOS on vendor templates, 1 → 0 on Soup'sdata.chat_templatepresets (which render none), and no trailing EOS from tokenizers that append one;usage.prompt_tokensdrops accordingly. Models without a chat template, and templates that fail to render, are encoded exactly as before. Adapters trained withtrain_on_responses_only: falsesaw the doubled BOS in training and now differ from inference by one token; that path and the vLLM / SGLang / MII engines are tracked in #781.Fixed
soup doctortreating the optional[train]stack as missing required dependencies: a core-only install now suggestspip install "soup-cli[train]"once (with the CUDA wheel index on NVIDIA GPUs) instead of bare per-package floors,soup version --fullderives installed extras from distribution metadata, and a missing or incompatible core dependency exits non-zero (#828 by @wangzhengzhuo05 in #854).Make
soup data validatefail when no rows are usable, and add an optional minimum-valid-fraction threshold for stricter CI gates (#811 by @akkupratap323 in #858).Route terminal charts through Rich so
NO_COLORand redirected output contain no raw ANSI escapes while interactive charts retain colour (#824 by @akkupratap323 in #860).soup data validate --formataccepted any string and reported every row valid for it (#866 by @SID-6921 in #869).commands/data.pydeclaredfmt: strwith help text listing the real formats but never checked the value againstsoup_cli.data.formats.VALID_FORMATSbeforevalidate_and_statsran, so--format bogusexited 0 and reported "2/2 rows valid for bogus format". Sincesoup data validateis the first step of the PR gatesoup ci initwrites, a typo'd--formatin a CI workflow silently turned the gate into a pass for any file — the same class of silent pass #811 and #858 just closed for "0 usable rows". An unknown format now exits 1 (input error), not the failed-gate exit code 2.
Security
soup ui --publicno longer serves the FastAPI docs to the LAN (#731 by @Srinivasan8888 in #732)./openapi.json,/docs,/docs/oauth2-redirectand/redocanswered 200 without a token on a non-loopback bind, so anyone on the network could enumerate every route, parameter and request/response shape. No run data, configuration or logs were exposed — every endpoint the schema describes already answered 401 after #707 — so this was reconnaissance rather than disclosure, and it predates #707. The four routes are now absent (404) on a non-loopback bind and unchanged on loopback. They are removed rather than gated because/docsis a browser navigation that cannot carry a Bearer header, so gating would have broken the page for developers while leaving/openapi.jsonreadable by any HTTP client.
0.74.0 - 2026-09-04
Added
soup eval aiderruns Aider's Polyglot code-editing benchmark through its official Docker harness (#91 by @Amix29 in #482). The command preflights Docker, the daemon, and the locally built benchmark image; mounts a prepared Polyglot corpus read-only; keeps output under cwd; and aggregates bounded per-exercise JSON into a Soup result row.--run-idrecords the score ineval_resultsfor the existing run comparison workflow. The optional[aider]extra installs the normal Aider CLI while the docs make the source-only benchmark-image setup explicit.Native Apple Silicon telemetry for
soup monitor(#99 by @Amix29 in #481). The monitor now reads bounded plist output from macOSpowermetricsand renders GPU utilization and power in the existing Rich table. It reuses an explicitly cached sudo credential through non-interactivesudo -n, never reads a password, and gives an actionable Activity Monitor fallback when permission or telemetry is unavailable. NVIDIA-only VRAM, memory-utilization, and temperature fields remain unavailable rather than being guessed.soup mcp servegains network transports:--transport sseand--transport http(#296 in #479). v1 was stdio-only, which suits a client that spawns Soup as a subprocess but leaves remote and multi-client setups with nothing. Both new transports serve the same registry — the end-to-end test compares the advertised tool names againstbuild_registryrather than a hardcoded count, so a subset cannot creep in. stdio remains the default and is untouched.Adding a listener is the risky part, so it is gated three ways. Every HTTP request needs
Authorization: Bearer <token>, compared withsecrets.compare_digest, with no opt-out — a loopback port is reachable by every process on the box. The token is validated by the existingutils/qr_url.py::validate_token, sosoup uiandsoup mcp serveagree on what a token is instead of growing a second format, and it travels in the header only: no query-string fallback that could land in an access log. The SDK's DNS-rebinding protection is switched on (421on a foreignHost,403on a foreignOrigin), which is the gate the token cannot be — a page the operator merely visits sends noAuthorizationheader but its request still reaches the port. Binding off loopback warns, and a wildcard bind warns again that the Host check has nothing left to pin.--allow-executeis refused with either network transport, and that refusal is the one behavioural change to an existing flag. Gated execution (#297) spawns real training / export processes; behind a listener a leaked Bearer token would mean process execution rather than plan disclosure, and stdio is a pipe to a client the operator already started, which is a different trust boundary. The refusal is made twice on purpose: once in the CLI so the operator gets a readable message, and once inbuild_asgi_app()so a direct caller cannot put an executing registry behind a listener either. The stdio banner also stopped claiming "execution disabled" -- that string predated #297 and was false from the moment execution shipped.--host/--port/--auth-tokenare refused under--transport stdiorather than silently ignored, and the ASGI app is built by abuild_asgi_app()factory so the auth and rebinding behaviour is tested through an in-process transport without binding a socket; one further test binds a real ephemeral port and drives initialize -> list_tools -> call_tool through the SDK's own SSE client.The
[mcp]extra floor moves1.2.0->1.10.0, measured against the published wheels rather than a changelog:server/streamable_http_manager.pyfirst appears in 1.8.0 (absent in 1.7.0) andserver/transport_security.pyin 1.10.0 (absent in 1.9.4).--transport ssealone would have run on the old floor; rebinding protection would not, and a listener whose origin checking silently disappears on an older SDK is worse than a resolver error. The<2cap is unchanged — #322 is the 2.x migration. No new package:mcpalready requires starlette, uvicorn, sse-starlette and httpx-sse.soup data best-of-ncan sample candidates from Ollama or vLLM providers (#299 by @Faisal-Fayaz in #466). The existing local Transformers--basepath stays the default, while--provider ollama|vllm --model <m> [--base-url <url>]draws each prompt's N candidates through the existing SSRF-validated raw- completion seam. Provider and model are recorded in_best_of_nprovenance; Anthropic is refused by name because its Messages API has no raw-completion endpoint, and local-only flags cannot be silently ignored in provider mode.Baseline artifacts carry a scorer/version provenance stamp (#404 by @AchuthReddy-16 in #485). Shared helpers
stamp_baseline_scores/write_baseline_filewrite{"scores": {...}, "provenance": {"soup_version", "scorer_revision"}}.soup eval gate --write-baseline <path>is the user-facing producer.resolve_baselinewarns once on unknown provenance (unstamped files) or ascorer_revisionmismatch, and stays silent when the stamp matches.BUNDLED_SCORER_REVISION+ a locked fingerprint fail the suite if a bundled scorer's output moves without a revision bump. Registrysave_eval_resultstampsdetails_json;soup ship --emit-evidenceincludes the same stamp.Layer streaming now accepts Qwen3.5 MoE text checkpoints whose decoder layers do not expose exactly the same weight keys in every block (by @Shutaru in #426). The sharder now records per-layer shard headers instead of deriving the pool from layer 0 alone, while still refusing divergent storage layouts for any shared key and rebuilding NF4
Params4bitviews from validated per-weight metadata.qwen3_5_moe/qwen3_5_moe_textroute through the existing qwen3 streamer. A heterogeneous toy MoE is bit-exact streamed versus resident on CPU; live validation was also run onQwen/Qwen3.5-35B-A3Bwith layer streaming, NF4, MoE LoRA target resolution and a 3072-token SFT dataset.stream_layersnow refusesmoe_expert_quant, which is applied only by the resident setup path and was otherwise silently ignored.live_eval.load_model_and_tokenizergains aquantizationparameter; no live evaluation path sets it yet (#367 by @AmirF194 in #461).quantization="4bit"builds the same nf4BitsAndBytesConfigevery other 4-bit load path in this codebase uses ("8bit"and the unset default are also supported), and the four internal callers this helper has (make_generator,make_multi_generator,lora_probe,measure_logit_agreement) still call it with noquantization, so today an NF4-trained adapter is still judged against a bf16 base it never saw during training. The follow-up is wiring those four callers to a default derived from the run's own configuration (soup ship --config, the registry entry, or the adapter'sadapter_config.json), per the issue's own Fix path; that plussoup shipreporting the numerics it judged with and a staleness gate on mismatched-numerics evidence are left open (issue acceptance criteria 2 and 4).training.stream_pinmakes layer-streaming pinning configurable (#366 by @ousamabenyounes in #416). Page-locking the RAM store is chosen automatically bydecide_pinning, and until now nothing could override it — so while #331 was live,pin=Falsewas the only known mitigation for silently wrong NF4 gradients yet was unreachable fromsoup.yaml.stream_pin: falsenow forces the pageable store (and the pre-flight states the throughput it costs, up to 6.56x measured, rather than absorbing it silently);stream_pin: trueforces the pinned store and, on the RAM tier, refuses the run — naming the store size, not the ceiling (#366 AC3):pinned_limit_bytesis passed asNone, so the page-lock ceiling is deliberately left unprobed and the store size is the only figure the refusal can honestly cite — if the box cannot page-lock it, instead of degrading silently; unset keeps today's automatic behaviour. On the disk tier (no RAM store to page-lock) and on CPU (no device to copy to) pinning is inapplicable rather than unsatisfiable, sotrueis announced and the run proceeds — refusing there would brick the large-model runs the disk tier exists for, and would make the key uncommittable to a config shared between a GPU box and a CPU box. Set whilestream_layers: falseit is rejected as a footgun, like the other stream keys.A ready-made
qwen3.5-9b-grporecipe for GRPO reasoning training withQwen/Qwen3.5-9B(#277 by @harshitthek in #448). The recipe combines the established GRPO defaults (accuracy reward, beta=0.1, 4 generations) with LoRA r=16 and 4-bit quantization.A ready-made
glm-5.1-dporecipe for DPO preference training withzai-org/GLM-5.1(#280 by @Osheun in #452). The first recipe pairing DPO with a MoE base, so it carriesmoe_lora: truealongside the established DPO defaults (beta=0.1, LoRA r=32 / alpha=64, 4-bit quantization).epochs: 1andmax_length: 8192are taken from the SFT sibling rather than the smaller qwen defaults, which suit a 754B MoE better.Cross-tokenizer draft support for
soup draftandsoup serve(#304 by @CODING-DARSH in #417).soup draft distillnow accepts target/draft pairs with mismatched tokenizers or vocabularies, automatically routing throughuld_strategy: wasserstein_aligned(Universal Logit Distillation) instead of refusing the pair.soup draft measuresupports cross-tokenizer acceptance measurement using decoded character-span alignment (count_accepted_spans) across different vocabularies and token boundaries. Target generation neutralizes onlyrepetition_penaltywithrepetition_penalty=1.0(Refs #345) so target greedy argmax and draft raw-logit scoring are evaluated consistently; remaining generation processors (no_repeat_ngram_size,encoder_repetition_penalty,min_new_tokens,bad_words_ids,suppress_tokens, andsequence_bias) are not altered.soup serve --speculative-decodingsupports cross-tokenizer draft serving via Transformers Universal Assisted Decoding (UAD) when supported by the installedtransformersversion, raising a clear error if unsupported.- Compatible same-tokenizer pairs strictly preserve the existing native fast path.
data.interleaveis now wired into training-time dataset loading (#443 by @blackcoderx in #460).parse_interleave/InterleaveSpechave been schema-validated and unit-tested since v0.42.0, butload_dataset()never called them — every multi-dataset mixture request silently trained on nothing butdata.train's single path, the same gap #330 and #442 papered over in their respective renderers.DataConfig.trainnow acceptsstr | list[str]; a list of>= 2local file paths combines viadata.interleave(concat/under/over/{strategy: probs, probs: [...]}) into one row set before the existingval_splitline in_finalizeruns, so a single path stays byte-identical.interleaveis local-files-only:training.packing/training.multipackanddata.streaming/ an HF-hub dataset name are all rejected at config-parse time with a message naming the reason (streaming/hub-dataset interleaving is follow-up #459). Both thesoup data mix --optimizerecipe writer and the--liveoverlay renderer emit the real N-dataset mixture again instead of collapsing to one path.A repo-wide documentation ratchet to guarantee declared recipe counts stay synchronized with the catalog (#453 by @harshitthek in #457). Derives the expected count dynamically from
len(RECIPES)and scans all declared documentation sites, preventing silent Git auto-merge drift across sequential recipe additions.Branch coverage for
lr_groups.py'sbuild_optimizer_param_groups: the case where every parameter matches a configured group, so nobaseoptimizer group is appended (#273 by @AmirF194 in #469).Branch coverage for
replay.py'sdownsample: the case where the stride already lands on the last row, so the endpoint pin is not appended a second time (#273 by @AmirF194 in #470).Re-enabled
/v1/tools/bashexecution with OS-level namespace/sandbox isolation (#151 by @kok-o in #527). Note this is a breaking change:serve()now raisestyper.Exit(code=2)when run with--host 0.0.0.0without--tool-auth-token, whereas previously it only printed a warning.data.interleavenow supportsdata.streaming: trueand lists of HF-hub dataset names (#459 by @blackcoderx in #468). #443 leftdata.interleavelocal-files-only, refusingdata.streamingand any remote-URI / hub-name list entry at parse time with a message naming this issue as the follow-up. Everydata.trainlist entry is now classified once (local file / remote URI / HF-hub name) and dispatched: an all-local/remote list withdata.streaming: truedelegates combining to HFdatasets.interleave_datasets/concatenate_datasetsrather than reimplementing mixing over a source whose size can't be known ahead of time —concatmaps toconcatenate_datasets,under/overmap tostopping_strategy="first_exhausted"/"all_exhausted", andprobsmaps toprobabilities=probs, each chosen so the strategy names mean the same thing as the local path (verified by running oneprobsconfig through both paths and comparing the resulting proportions, not by two tests that each pass alone). An all-HF-hub-name list is loaded eagerly per entry and combined with the same_combine_interleavedthe local path already uses; a hub entry's ownvalidationsplit is honoured for the combined result only when every entry provides one, otherwise it's ignored (warned) anddata.val_splitapplies to the combined train rows — a decided precedence rather than an emergent one. Still refused, by name: an all-hub list withdata.streaming: true(streaming N differently-shaped hub datasets and reconciling their splits is a separate, larger effort), and any list mixing hub names with local/remote entries.LISA now accepts
task: pretrain, notsftalone (#307 by @ousamabenyounes in #476). Continued pre-training is the same full-fine-tune-of-a-rotating-set-of-decoder-layers mechanism LISA was built for, so the sft-only gate (inherited from Spectrum'sunfrozen_parameters) was arbitrary. The schema task gate now reads a_LISA_SUPPORTED_TASKSallow-list (sft,pretrain) and its refusal names every accepted task instead of only rejecting yours;trainer/pretrain.pyreplaces its LoRA path with LISA and attachesLisaCallback, and reports its parameter summary asLISAcounted off the raw parameters, since a LISA run has noPeftModelwrapper to askget_nb_trainable_parameters. Both trainers route through onepeft_wiring.apply_lisa_setup, so they cannot drift on what "LISA is on" means. The rest of the gate is unchanged:transformers+text+quantization: none, mutually exclusive with the LoRA feature flags,freeze_layers/freeze_ratioandunfrozen_parameters. The released[0.71.34]block still saystask: sft, which is what 0.71.34 shipped.A weekly
dependency driftjob, and a test that asks trl what it still accepts (#323 in #486). Two failure classes were structurally invisible until a PR happened to be open: a bug that only manifests on a CPU-only runner (a CUDA build never calls_convert_weight_packed_for_cpu, so no GPU dev box can reach it), and an upstream removal behind a floor-only pin. The second one shipped for several releases with CI green throughout —trl>=0.7.0let CI resolve 0.29.1 while the dev box ran 0.19.1, and on 0.29.1 six trainers could not build their config at all, because the trl imports live insidesetup()and no test had ever called it..github/workflows/dependency-drift.ymlruns the same resolve on a schedule: it installs the latest resolvable stack, runs the suite against it, and writes a resolved-vs-declared table into the run summary. It also flags a package declared with incompatible ranges in two extras — which it already found before merging:[mlx]asks fortransformers>=5.0.0while[train]caps it below 5, sopip install "soup-cli[train,mlx]"cannot resolve.tests/test_issue323_trl_kwarg_drift.pyanswers the setup() question without a model: it reads the keywords each wrapper passes to its trl config and asks the installed class whether it still accepts them, through the sameconfig_acceptscapability probe the wrappers use at runtime — never a version comparison, since a version table is what was wrong twice. Theresolve_trl_symbolindirection is followed, so the three configs that moved totrl.experimentalare covered rather than silently skipped. Two blind spots are stated rather than implied:**splatcalls are invisible to a static read, and the trl TRAINER classes are excluded because they take much of their signature through**kwargs, which makes a signature check reportmodelandtrain_datasetas rejected — measured, and the reason the scan filters onConfig.soup mcp servenow runs on both mcp majors, and the<2cap is lifted (#322 in #498). v0.72.3 capped the[mcp]extra the day mcp 2.0.0 broke every round-trip test; that unblocked a release and pinned anyone who wanted 2.x in the same environment. Both removals are bridged rather than pinned around: the@server.list_tools()/@server.call_tool()decorators becameon_list_tools=/on_call_tool=constructor callbacks, andcreate_connected_server_and_client_sessiongave way to the lower-levelcreate_client_server_memory_streams, which both majors still ship.build_serverchooses by probing theServerconstructor, never by readingmcp.__version__— the ruletrainer/_trl_compat.pyearned after two wrong bounds derived from version tables, and a test walks the AST to enforce it. The dispatch logic stays in a single_dispatch_toolwith two thin adapters, guarded by a test that fails if a second implementation appears, because two copies is how the majors would drift apart while both kept passing.The three modules the sse / streamable-http transports depend on all survive 2.0.0 unchanged, so those 46 tests needed no edit. Verified by running the MCP suites twice, once per major: 184 passed against 2.0.0, and the full suite against 1.29.0. The floor stays at the 1.10.0 measured in #296.
Added a text-only
qwen3.8-27b-sftcatalog recipe forQwen/Qwen3.8-27B(#477 by @Amix29 in #513). The recipe explicitly selectsmodality: text, uses the measured Qwen3.5-family decoder path from #507, and includes catalog, configuration, CLI, modality, and count-sync coverage.soup train --cloud lambdaadds cloud GPU training plans for Lambda Cloud instances (Refs #264 by @kok-o in #528).Added 7 ready-made SFT recipes - Qwen2.5-Coder 1.5B/14B/32B (
qwen2.5-coder-{1.5b,14b,32b}-sft), Qwen2.5-Math 1.5B/7B (qwen2.5-math-{1.5b,7b}-sft), and DeepSeek-R1-Distill-Qwen 1.5B/7B (deepseek-r1-distill-qwen-{1.5b,7b}-sft) - and correctedmistral-small-3-sftto point at the real hub repo (mistralai/Mistral-Small-24B-Instruct-2501). Catalog grows 147 -> 154 (#536 by @Nick-800)Added #550: a content-addressed two-phase Best-of-N workflow for exporting local candidates and materializing verified offline judgments without model or network access (#550 by @Amix29).
Added 4 ready-made recipes - DeepSeek-R1-Distill-Llama-8B SFT (
deepseek-r1-distill-llama-8b-sft) and three R1-Distill DPO variants (deepseek-r1-distill-qwen-{1.5b,7b}-dpo,deepseek-r1-distill-llama-8b-dpo) with pinned lr and dpo_beta. Catalog grows 154 -> 158 (#569 by @Nick-800)#572 adds a weight-free Qwen3.8-Flash-Next text-LoRA compatibility scaffold on Transformers 5.16.1, including architecture-aware linear targets, MoE detection, and a tiny-config forward/backward gate. Legacy int64-only Torch scatter runtimes receive an instance-local QSA index compatibility shim; real checkpoint, catalog, streaming, multimodal, and routed-expert-parameter support remain explicitly unclaimed (#572 by @Amix29).
#575 adds opt-in PEFT
target_parametersLoRA for Qwen4-Exp routedgate_up_projanddown_projexpert tensors in resident Transformers SFT and continued pretraining, with validated compatibility constraints and weight-free backward/save/reload coverage (#575 by @Amix29).#576 adds
training.lisa_train_embeddings(defaulttrue= LISA as published in #267). Set itfalseto freeze LISA's always-on group — input embeddings, LM head, and final norm — so only the sampledlisa_num_layersdecoder layers train. That always-on group is ~70% of everything LISA trains at 8B, so it is where LISA's memory actually goes; freezing it is a real quality/memory trade rather than a free win, which is why it is an opt-in knob. Setting itfalsewhilelisa_enabledisfalseis rejected, matching the otherlisa_*fields. The analytical VRAM pre-flight still treats LISA as full fine-tuning regardless of this flag (it does not yet credit the frozen-embeddings saving, which needs a measured constant on GPU hardware), so a frozen-embeddings run that would fit can still be conservatively refused — use--allow-oom-attemptto launch it. Refs #377 (#576 by @Srinivasan8888).#582 adds a ready-made
smollm3-3b-sftrecipe for HuggingFaceTB/SmolLM3-3B. The catalog shipped SmolLM2 in three sizes but no SmolLM3; this fills that gap with a small/edge LoRA SFT recipe (r8, 8-bit, auto batch). Catalog count 158 → 159. Closes #271 (#582 by @Srinivasan8888).Add Qwen4-Exp layer-streamed SFT with sparse read-only PLE N-gram access for dense Transformers and oMLX/oQ affine checkpoints, including an explicit
training.stream_ngram_sourcepolicy and fail-closed task/quantization/media gates (#602 by @Amix29 in #603).Added the model-free AutoDistill Milestone A artifact contract: versioned and immutable plan/capture/shard/consumption schemas, explicit top-k plus residual-tail semantics, transactional resume/corruption rules, and deterministic plan-only estimates (#580 by @Amix29 in #613).
Added a ready-made
kimi-k2.6-grporecipe for moonshotai/Kimi-K2.6 (#281 by @umran666 in #614). The v0.71.24 model-family expansion shipped the SFT variant but no GRPO reasoning variant; this fills that gap with the MoE giant GRPO shape (grpo_beta: 0.1,num_generations: 4,reward_fn: accuracy,moe_lora: true,gradient_checkpointing: true, 4-bit,max_length: 8192), and keeps the Modified MIT licence note in the recipe description. Catalog count 160 -> 161.Added a ready-made
qwen3.5-35b-a3b-dporecipe for Qwen/Qwen3.5-35B-A3B (#276 by @Srinivasan8888 in #615). The catalog shipped the MoE SFT siblingqwen3.5-35b-a3b-sftin v0.71.24 but no preference-tuning variant; this pairs theqwen2.5-7b-dpoDPO shape (format: dpo,lr 5e-6,dpo_beta: 0.1) with the sibling's MoE settings (moe_lora: true,moe_aux_loss_coeff: 0.01), and carries an explicitmodality: textso it joins the Qwen3.5-family contract intest_issue427_qwen35_text_modality.pyrather than falling back to the schema default. Catalog count 159 → 160.Added the internal AutoDistill Milestone B1 same-tokenizer, teacher-only MLX capture and transactional shard publication boundary, with immutable input fingerprints, explicit corruption/resume checks, and proof that no student model is loaded during capture (#580 by @Amix29 in #629).
Added a ready-made
qwen3.5-9b-dporecipe for Qwen/Qwen3.5-9B (#275 by @Srinivasan8888 in #632). The base already shipped SFT and GRPO variants; this completes the trio with the preference-alignment shape (task: dpo,format: dpo,dpo_beta: 0.1,lr: 5e-6, LoRA r16/a32, 4-bit,max_length: 4096), matching every other DPO recipe in the catalog. Catalog count 161 -> 162.Pinned every recipe's resolved
SoupConfigagainst a committed snapshot (tests/fixtures/recipe_config_snapshots.json, regenerated viascripts/generate_recipe_snapshot.py), so a schema-default change no longer silently retunes recipes that rely on that default. Confirmed by mutation: changingdpo_beta's schema default failed 149 of the 162 recipes — every one that doesn't explicitly pin the field — which is the exposure this closes (#621 by @SID-6921 in #637).Added a ready-made
glm-5.1-grporecipe for zai-org/GLM-5.1 (#275 by @Srinivasan8888 in #656). The base already shipped SFT (v0.71.24) and DPO (#280 by @Osheun); this completes the trio with the reasoning shape (task: grpo,reasoning_train.jsonl,grpo_beta: 0.1,num_generations: 4,reward_fn: accuracy) over the 754B MoE geometry its siblings already use (LoRA r32/a64,batch_size: 1,gradient_accumulation_steps: 16, 4-bit,moe_lora,gradient_checkpointing,max_length: 8192) — deliberately not the 30B GRPO template's r16/a32.lr: 1e-5is the value both conventions agree on: 15 of 22 GRPO recipes andglm-5.1-sftalike. The recipe is not trained — 754B is multi-node — so no hyperparameter here is a measured recommendation. Catalog count 162 -> 163.
Changed
Remove the name-based
SCORER_CHANGED_IN_V0_73_2baseline warning in favour of the #404 scorer_revision stamp (#404 by @AchuthReddy-16 in #485). Stale baselines are detected by provenance, not by a hard-coded suite list.Remove hand-maintained test suite statistics from
CONTRIBUTING.mdin favor of a permanent digit-free shape invariant (#465 by @harshitthek in #467). Eliminates drift across routine test additions by making test-count divergence impossible at the documentation source.Lazy callback builders now self-import their callback class names so runtime lookup never raises
NameErrorwhile preserving lazy heavy-dependency loading (#320 by @AchuthReddy-16 in #455).build_echo_trap_callback,build_reward_hack_callback,build_minillm_callback,build_rl_checkpoint_callback, andbuild_push_callbacknow resolve their callback types through local module imports in the builder body, and the regression suite adds subprocess coverage for all five builder calls.User-visible changes now use per-PR changelog fragments (#487 by @Amix29 in #490). Contributors add a uniquely named, version-scoped Markdown file instead of editing the shared
[Unreleased]section. A standard-library assembler preserves long entries verbatim, rejects stale or malformed fragments, and consumes them during release preparation. A tag-time release gate refuses publication if any fragment remains, so the conflict is removed without making changelog loss silent.Qwen3.5 and Qwen3.6 text recipes now state their decoder-only intent (#427 by @Amix29 in #501). Apple Silicon measurements confirmed that Soup's text path loads
Qwen3_5ForCausalLMwithout the visual tower, so all twelve catalog recipes now setmodality: textexplicitly and regression coverage prevents that decision from falling back to the schema default.Transformers training now supports Qwen3.5-family text decoders and can be installed together with MLX (#502 and #503 by @Amix29 in #507). The shared stack moves to Transformers 5.12.1+, TRL 0.29+, and PEFT 0.20+, with exact floor coverage, matching
soup doctordiagnostics, and capability-based fallbacks for TRL APIs that moved undertrl.experimental. Transformers LoRAtarget_modules: autonow covers both full- and linear-attention projections in Qwen3.5 text models without changing explicit targets or MLX defaults. The compatibility pass also updates removed Transformers arguments and keeps TRL 0.29's streamed DPO reference adapter offmeta, so it remains a frozen adapter-sized snapshot instead of requiring a second model. DPO and ORPO also restore the removed prompt cap on TRL's prepared token ids, sodata.max_lengthremains an effective sequence bound rather than a configuration-only value. The CLI's histogram and loss-curve renderers now support both plotext 5 and plotext 6, with a real Plotext 6 runtime pinned in the compatibility CI cell.Unknown config keys are now reported instead of silently dropped, and v0.75 will reject them (#627 by @Srinivasan8888 in #628). None of the config models overrode Pydantic's default
extra="ignore", so a key the schema did not declare validated clean and was discarded:soup train --dry-runprinted "Config valid. Ready to train!", the run exited 0, and the requested setting was never applied —training.quantizaton: nonetrained 4-bit quantized when full precision was what you asked for,training.gradient_checkpoint: truedid no checkpointing,data.max_len: 512truncated at 2048. #623 is the live case:training.stream_pinreached main two days after 0.73.3 shipped, a user on the released wheel wrote the documented escape hatch, and the resulting OOM was investigated as a layer-streaming defect. Loading a config now walks the whole model tree —data,training,training.loraand the rest, so a guard applied to one model and forgotten on another cannot look like it works — and reports every key it cannot place in one report per load, naming the field you probably meant (did you mean 'quantization'?). This is a warning, not a refusal: a config written against a newer Soup still runs on an older wheel, which is the case #623's user was in. From v0.75 the same config will fail to load — one minor of notice, since this release is the one that starts warning — and the warning names that version so the deadline is decidable rather than a permanent notice. The version is stated in one constant (config/unknown_keys.py) and asserted against the declaredsoup_cli.__version__by a test, so the release that crosses the deadline turns a test red rather than leaving the message promising a rejection that already shipped.soup sweepnow hard-errors on a--paramthat matches no config field, with no deadline (#627 by @Srinivasan8888 in #628). A different failure class from a dropped training key, so it is called out separately: a sweep whose swept knob is never applied produces arms that are all identical, and there is no partially-useful result to preserve by continuing.--param lora_rank=8,16used to run the full grid at the base config's LoRA rank and report the winner; the grid is now checked before the first arm starts, andsweep parameter does not match any config field: unknown config key 'lora_rank' - not applied.is printed and the command exits 1. Anyone with a typo'd sweep parameter will see a new error where they previously got plausible, meaningless results, and a scripted sweep now fails instead of succeeding with a table of failed arms.The recipe-config snapshot fixture (#621/#637) is now delta-encoded — a change to a schema default that no recipe explicitly pins now produces one failure naming the shared baseline, not 149 identical failures across every recipe that relies on it. What's now structurally distinct is "a recipe's own value moved" (a named per-recipe test) versus "the shared defaults moved" (the one baseline test) — those are different tests now, not just different-looking diff text. Measured:
tests/fixtures/recipe_config_snapshots.jsongoes from 1,571,967 bytes / 49,586 lines to 66,560 bytes / 2,040 lines — 23.6x smaller by byte count, 24.3x fewer lines.Traded, not eliminated, and worth stating at the size that actually occurs: a recipe that redundantly pins a value equal to the old default starts appearing in its own delta once the default moves out from under it, since what used to be a no-op pin just started doing something — that recipe's resolved config hasn't changed, only its relationship to the (now different) baseline has. For a rarely-pinned field (
dpo_beta, 0.1 -> 0.2) that's 13 named recipes plus the baseline, 14 total, down from 149. Forepochs(all 162 recipes declare it explicitly, distributed{1: 32, 2: 4, 3: 126}), moving the default 3 -> 1 moves 158: the 126 that pinned the old default plus the 32 that already pinned the new one — both now differ from a baseline that used to match one of them. The 4 recipes pinning2stay green, correctly, since neither the old nor the new default was ever their value. 158 named recipes plus the baseline — a wall of red on a change that alters no recipe's actual behavior. This is the acceptance criterion "changing a schema default fails, naming the affected recipes" holding exactly as specified, applied to recipes whose default-shaped pin is the thing that moved; it is not the "one failure" case, and the fragment previously understated it by only showing the favorable example.Verified by mutation against the real catalog: a new schema field produces exactly 1 failure (down from 164); a recipe's own value changing, a recipe added without regenerating, an empty or deleted fixture, and a redundant default-duplicating line being deleted (exposure 1, still deliberately unpinned per #621) all behave exactly as before (#638 by @SID-6921 in #640).
Fixed
The twelve non-SFT trainers (
dpo,kto,orpo,simpo,ipo,bco,online_dpo,grpo,ppo,pretrain,reward_model,embedding) loaded a frozen LoRA base as float32 regardless of the checkpoint's own dtype (#491 by @AmirF194 in #492). #471 fixed this for the SFT trainer; the samemodel_kwargsshape, missing the same key, was unchanged in the other twelve. None of them has a full fine-tuning branch, so every load there is a frozen base: the new sharedresolve_frozen_base_load_dtype()keeps the checkpoint's own dtype (torch_dtype="auto"), except on a pre-Ampere CUDA card, where it now matches the float16 compute dtype those cards already use instead of leaving the base in bf16 storage.Layer streaming now keeps its host store on CPU on Apple Silicon (#434 by @Amix29 in #480). PyTorch 2.7+ can return an MPS tensor for
torch.empty(device="cpu", pin_memory=True), whileis_pinned()remains false. Direct runtime callers could therefore place the whole frozen base in the MPS allocator and still report a pinned RAM store, even though the normalsoup trainsetup already disabled pinning outside CUDA. The runtime now disables both optional and required pinning when the target is MPS, andRamSourceindependently refuses any allocation that is not genuinely CPU memory (or claims pinning without being pinned). MPS proceeds experimentally with a pageable CPU source and MPS layer buffers; CUDA pinning behaviour is unchanged.CI and production load sites no longer assume Transformers
dtype=(#478 by @AchuthReddy-16).dtype=onAutoModel*.from_pretrained/from_configis the=4.56 rename of
torch_dtype=. Soup still declarestransformers>=4.36.0,<5.0.0, but the 12-cell matrix only ever installed the newest 4.x, so a >=4.56-only kwarg stayed green. Call sites in chat / diff / infer / export / merge / serve / mole routing / layer-stream runtime now passtorch_dtype=(still accepted on current 4.57.x). A static AST guard fails if a productionAutoModel*load/config site reintroducesdtype=. A new Ubuntu/3.11transformers-floorjob installs under.github/constraints/transformers-floor.txtusing the lowest non-yanked resolvable Transformers version4.46.1with the lowest version in the declared TRL range0.14.0(transformers==4.36.0is ResolutionImpossible against declaredtrl— the declared Transformers floor inpyproject.tomlremains unchanged), runspip check, asserts both pins, and runs the guard. The existing 12-cell matrix is untouched.downsamplenow returns at mostmax_pointsrows, which is what its docstring has always promised (#473 in #474). The stride waslen(rows) // max_points— a divisor, not a cap — so five rows withmax_points=2came back with three, and the endpoint pin could add a fourth. Sample indices are now spread evenly across the series with both endpoints included, sosoup runs replayrenders exactlymin(len(rows), max_points)points. Even spacing also avoids the cliff a ceil-based stride would introduce: a series one row over the cap keepsmax_pointspoints rather than roughly half of them.max_points=1is the one shape where both endpoints cannot fit and returns the final row, which is what the endpoint pin existed to guarantee. The off-by-one guard added in #470 is kept, renamed totest_last_index_lands_on_last_row_no_duplicateand re-pinned to the new arithmetic. Consequence of the old behaviour was a chart with a few more points than intended, never a wrong number.cut_ce.pyandliger.pynow normalize path separators and match architecture keywords on the last path component only (#456 by @harshitthek in #458). On Windows,rsplit("/")never split on backslashes, causing parent directory names (e.g.C:\experiments\phi-3-runs\step-2000) to over-match architecture keywords during fallback detection when config resolution was unavailable. Inliger.py, switching from whole-path to last-component matching also drops org-prefix false positives (e.g.mistralai/*,Qwen/*when the model name does not contain the keyword) and prevents parent directories from applying the wrong fused kernel across POSIX and Windows. Both modules now normalize dual separators and strip trailing slashes deterministically.measure_gemm_tflopsnow records per-repeat samples andtest_takes_the_best_repeat_not_the_firstverifies best-of-N selection within a single measurement (#444 by @harshitthek in #451). Comparing two separate probe calls taken at different moments caused intermittent test failures on developer GPU machines under background contention.GemmCeilingnow preservessamples: tuple[float, ...]and the test assertsmax(samples)selection deterministically.Duck-typed tokenizer mappings no longer raise a misleading error, and
data_doctorshares the publiccoerce_token_idshelper (#441 by @AchuthReddy-16 in #447, part 2 found by @emre155). A dict-like output that is not registered ascollections.abc.Mappingused to be iterated as keys (input_ids[0]='input_ids'), sending the operator looking at their data; the mask path skipped the same objects and silently droppedassistant_masks. Both gates now use one mapping-like predicate, and the helper is public so the two modules cannot drift.soup data mix --livehanded every candidate proxy run a config it could not load (#442 by @blackcoderx in #445)._render_overlay_yamlemitteddata.trainas a YAML list, the same shape #330 fixed in the recipe writer, but every--livetest mockedsubprocess.runso nothing ever loaded the overlay through the schema — a config the tool could not itself load read as a passing feature.data.trainnow renders as the single highest-weighted dataset in each candidate, mirroring #330's fix, with a comment noting which dataset was picked and why.use_cut_cesilently did nothing for any model loaded from a local checkpoint directory, and conflated Phi-2 with Phi-3 (#383 by @AmirF194 in #446).apply_cut_ce()picked the CCE patcher by matching an architecture keyword against the model path's last component, socheckpoint-2000/my-finetune/ any other directorysoup merge/soup shrink/soup trainwrites out matched nothing and CCE stayed off with no error, on a flag the user explicitly set. Separately, every Phi variant (phi-2,phi-3,phi-4) mapped to the same"phi3"patcher, even thoughcut_cross_entropyhas no Phi-2 patcher at all (itsconfig.model_typeis"phi", not"phi3"), and a bare"gemma"fallback entry dispatched to a patchercut_cross_entropydoes not have at all, crashing instead of reporting unsupported. Detection now resolvesAutoConfig.from_pretrained(model_name).model_typefirst, mirroring the identical fix already shipped for Liger Kernel (#78), and falls back to the name-based match only when that is unavailable; Phi-2 and plain Gemma-1 both correctly report unsupported instead of running under the wrong kernel or crashing. The two call sites that separately hand-wrote the "no matching architecture" advisory now share one message.soup draft measurenow refuses a mismatched pair up front and no longer discards a completed measurement when the assisted arm fails (#344 by @ousamabenyounes in #409).measuregated onsame_tokenizer()(tokenizer vocab + probe ids), which accepts a pair whose tokenizers are identical but whoseconfig.vocab_sizediffers by padded embedding rows (e.g. Qwen2.5 large←small) — exactly the pairdistillrefuses. Transformers' assisted generation gates onconfig.vocab_size, so the run died with "different tokenizers" deep insidegenerate(), after the expensive load, and because the report was written only after that arm every completed acceptance/plain-throughput number was thrown away.measurenow uses the sameconfig.vocab_sizeprecondition asdistill(refusing before any model loads; the shared_vocab_size_ofalso reads a composite model'sget_text_config(), so a multimodal target like Llava is no longer refused), keepssame_tokenizer()as an additional check, and writes the report incrementally so a failing assisted arm leaves the acceptance rate and plain throughput on disk. The report records anassisted_status(pending/complete/untimed/crash/interrupted) so a failed arm is distinguishable on disk from a completed or un-run one —pendingis what a report keeps when the process dies mid-arm and no handler runs, which is the case the incremental write exists for.soup export --format gptqcrashed with no calibration data and, when it did run, wrote a shard name the standard loader can't find (#338 by @AmirF194 in #475). With no--calibration-data,_export_gptqcalledmodel.quantize(tokenizer); auto-gptq'squantize()expects tokenized examples, not a bare tokenizer, so this failed with "object is not iterable". GPTQ export now requires--calibration-dataup front and rejects a file with zero usable samples, since auto-gptq has no built-in fallback dataset (unlike AWQ). Separately,save_quantizedwrites its owngptq_model-<bits>bit-<group>g.safetensorsshard, whichAutoModelForCausalLM.from_pretraineddoes not look for; the exported directory now also carries a standardmodel.safetensors. The sharedexcept ImportErrorblocks on both the AWQ and GPTQ paths also stopped reporting a fixed "not installed" string when the package itself imports fine but a transitive import inside it fails for an unrelated reason.SmolVLM/Idefics3 vision SFT now reaches real training batches (#302 by @Amix29 in #488). Soup keeps LLaVA messages and PIL images together until collation, converts legacy
<image>markers to structured multimodal content, and lets the processor produce image-token expansion plus architecture-specific pixel tensors. The vision path uses the Transformers trainer with this collator so older supported TRL releases cannot pre-tokenize the dataset as text-only. Image placeholder ids are excluded from causal-LM labels, and the collator preserves a leading BOS whether it comes from the chat template or the tokenizer default.Pre-Ampere cards (T4/P100/V100/GTX 16xx — the whole free Colab/Kaggle tier) could crash
stream_layers: truetraining with_amp_foreach_non_finite_check_ and_unscale_cuda not implemented for 'BFloat16': peft creates LoRA adapters in the base checkpoint's dtype while fp16 GradScaler requires fp32 gradients. Trainable*lora_*params are now cast to fp32 before optimizer creation from every trainertrain()site through one shared helper (lora.r: 0, Spectrum and LISA full-FT paths are deliberately untouched so trainable memory does not double after the VRAM pre-flight). The #385 static scanner was narrowed, not weakened: modules that take the precision decision via the shared alignment helper now count as covered (#425, #429) (#429 by @lesterppo).SFTTrainerWrapperno longer silently upcasts every load to fp32 (#339 by @blackcoderx in #471). All threefrom_pretrainedcall sites (text/vision/audio) now pass an explicittorch_dtype: a frozen base (LoRA/QLoRA — the base never receives an optimizer step) preserves the checkpoint's own dtype via the sharedresolve_frozen_base_load_dtype()(#491/#492) instead of defaulting to fp32 — pre-Ampere CUDA cards (T4/P100/V100/GTX 16xx/RTX 20xx) get an explicittorch.float16override there instead of bf16-storage/fp16-compute. A trainable base (lora.r: 0,unfrozen_parameters,lisa_enabled— schema-gated to modality='text') loadstorch.float32as a deliberate, documented numerics choice. Measured on an H100 with Llama-3.1-8B, LoRA, frozen base: 48,241 MiB -> 18,658 MiB peak (2.59x / 28.9 GB), byte-identical across 3 repeats — the original #339/#471 benchmark, carried over unchanged in this revision rather than re-measured. The full-FT discriminator is a single sharedis_full_finetune(), used by bothSFTTrainerWrapperandcommands/train.py's VRAM pre-flight classifier — previously independent copies that disagreed in both directions.setup()'s console summary label also now names LISA runs correctly instead of mislabeling them "LoRA applied".Every DeepSpeed-capable trainer now prunes the empty LoRA optimizer group, not just
sft.py(#359 in #484). #336 fixed the failure where LoRA leaves HF's no-decay parameter group empty, DeepSpeed drops it, and the LR scheduler keeps twobase_lrsuntil torch's strictzipraises at the firstlr_scheduler.step()— but it fixed it in one wrapper. Measured before changing anything: 19 modules undersoup_cli/trainer/accept adeepspeed_configand exactly one called the guard, so 18 tasks still died the same way under--deepspeedwith LoRA.Coverage is enforced by a scan over
soup_cli/trainer/*.pyrather than a list of names, followingtest_device_map_distributed.py— whose own history is the argument, since its first version parametrized over the six trainers that fix had touched and passed while nine more sites still carried the defect. The scanner requires the guard only where a module both accepts adeepspeed_configand constructs a trainer itself, so the delegatingpreference.pywrapper is correctly exempt, and it carries a control proving the pattern can fail.attach_empty_param_group_guardnow declines a trainer with no callablecreate_optimizerinstead of raising. That is load-bearing once the guard is attached from eighteen wrappers rather than one: not every TRL trainer exposes the method, and an AttributeError there would convert a DeepSpeed-only defect into a crash on the ordinary path.--deepspeed my.jsonis now resolved the way a preset is (#359 in #484). A user-supplied config reached DeepSpeed unresolved, so none of the preset rewrites applied to it. The decision recorded: resolve, but only keys that are provably invalid for the run.zero_hpz_partition_sizeis refused by DeepSpeed when the world size is not divisible by it, and the fp16 quantiser against abf16run raisesexpected mat1 and mat2 to have the same dtypeinsidedeepspeed/runtime/zero/linear.py— neither is a preference. Since the documented way to customise ZeRO++ is to copy the preset JSON, which copies both defects, an unresolved user file inherited a crash the presets are already protected from. A config using none of those keys is returned by its own path, byte-identical; a repair is printed and written to a temp copy, and the user's file on disk is never modified. Malformed JSON passes through untouched, because DeepSpeed reports a bad config better than a traceback.Transformers floor CI now tracks declared dependency metadata (#494 by @Amix29 in #496). The compatibility guard derives the training lower bound from
pyproject.toml, refuses a tested pin below it, requires a documented reason for a higher resolvable pin, and makes the workflow read exact versions from the constraints file instead of restating them.A server crash between spawning an MCP training job and recording its pid no longer lets a restart double-book a second job (#506 by @AmirF194). The one-active-execution cap now treats an unresolved launch as active rather than reading it as free capacity.
TensorRT-LLM export now fails immediately with a clear message instead of silently producing zero artifact bytes (#337 by @AmirF194 in #508).
_export_tensorrt()shelled out topython -m tensorrt_llm.commands.convert_checkpoint, a module absent from every current TensorRT-LLM release (tensorrt_llm.commandsships onlybench/build/eval/prune/refit/serve; conversion now lives as a per-architectureexamples/<arch>/convert_checkpoint.pyscript instead). The entry point is checked right after the existingtensorrt_llmavailability check, before the LoRA merge and checkpoint directory are touched. Docs now warn that installingtensorrt_llmcan downgrade a training environment'storch/transformers/numpy/datasetspins.Fixed layer streaming's hidden duplicate-disk cost: Soup now reuses regular Hugging Face cache files, preflights materialized-weight and shard-cache writes per volume, refuses before exhausting disk space, and explains source-fingerprint cache rebuilds (#510 by @Amix29).
Layer streaming now admits the dense
qwen3_5/qwen3_5_textdecoder used by Qwen3.8-27B after native mixed-attention resident-vs-streamed parity coverage. (#514 by @Amix29 in #515)Layer streaming now preserves BF16 checkpoints on capable MPS runtimes instead of silently doubling the store and disk cache to FP32. (#516 by @Amix29 in #519)
The negative llama.cpp quantizer lookup test is now isolated from binaries installed on the host
PATH. (#518 by @Amix29 in #520)Fixed fresh installs with Plotext 6:
soup data statshistograms andsoup runsloss charts now dispatch to Plotext 6's Figure API while retaining the Plotext 5 module-level path. The temporary<6.0.0dependency cap is lifted because both supported majors are covered by the compatibility layer and regression tests (#507, #522).Added
soup mcp runs reconcile --expunge-launchingso operators can safely recover execution capacity from stalelaunchingrows without editing SQLite by hand (#524 by @Amix29 in #525).Layer streaming now shards an untied
embed_tokensandlm_headpair separately and reuses one vocabulary-sized device buffer instead of keeping both matrices resident (#526 by @Amix29). Tied embeddings keep their existing resident one-matrix path and numerics.SFT now refuses examples without causal loss targets and final training states containing non-finite metrics or parameters before saving artifacts (#535 by @Amix29).
Streamed LoRA adapters now preserve the exact configured base-model provenance, restoring automatic base detection across adapter workflows (#537 by @Amix29).
Fully cached Hugging Face snapshots can now be materialized and sharded with outgoing traffic disabled, while preserving commit and blob integrity (#538 by @Amix29).
Apple internal NVMe volumes behind APFS and Apple Fabric are now detected for the layer-streaming disk tier without requiring a manual override (#539 by @Amix29).
Response-only masking now supports system-first conversations with native chat templates that reject transient prefixes without a user query (#543 by @Amix29).
The final training panel now preserves the last real loss, learning rate, and gradient norm when Transformers emits a summary-only log event (#544 by @Amix29).
Reproducibility receipts now record the MPS backend, Apple chip name, and unified-memory capacity without collecting unique hardware identifiers (#545 by @Amix29).
Best-of-N now rejects non-finite and boolean judge scores before selecting a winner or emitting invalid JSONL (#547 by @Amix29 in #551).
Best-of-N now rejects malformed prompt rows with line-numbered errors and records each accepted row's source line instead of silently losing data (#548 by @Amix29 in #552).
Fixed #549 by making Best-of-N generation durable and resumable per prompt, with exactly-once checkpoint validation and rollback-safe, manifest-last publication that never exposes a partial SFT/DPO generation after an output failure (#553 by @Amix29).
Fixed #555: two-phase Best-of-N candidate export now resumes from durable, authenticated per-prompt checkpoints, while offline validation and output materialization stream through bounded-memory disk staging. Resume binds prompt source lines and exact local-model content, and streamed SFT/DPO publication rolls back as one set on failure. Candidate export now seeds each prompt independently so resumed runs reproduce the same candidates; this changes the output of existing
--export-candidates --seed Nruns (#559 by @Amix29).Fixed #556: offline Best-of-N now commits SFT and optional DPO outputs through a final verifiable manifest, rejects unsupported online recovery options, and makes interrupted or mismatched generations fail closed (#557 by @Amix29).
_assert_finite_training_stateno longer refuses a run over a self-corrected transient metric (#546 by @AmirF194 in #560). The log_history scan now checks only the most recently logged value of each metric instead of raising on the first non-finite value found at any step, so a GradScaler warm-up nan that recovers no longer blocks the final save.Live multipack training now packs each bin to
training.batch_size * data.max_lengthinstead ofdata.max_lengthalone, so the configured batch size actually affects packing density again (#562 by @AmirF194).#564 enables BF16 autocast for hardware-validated resident SFT, DPO, reward-model, and PRM training on capable Apple Silicon MPS runtimes. The live capability probe is shared with layer streaming; CPU and unvalidated MPS trainers remain FP32, while PRM retains FP32 master weights to avoid a fatal Metal optimizer dtype mismatch (#564 by @Amix29).
Preserve reward metadata from local GRPO datasets and keep earlier assistant turns when separating a final reference answer from multi-turn conversations (#566 by @Amix29).
#568 enables runtime-probed BF16 autocast for local Transformers GRPO/RLVR on capable Apple Silicon MPS runtimes. Unsupported MPS remains FP32, local generation avoids the CUDA-only vLLM path, and an Apple Silicon train/save/reload smoke covers deterministic RLVR with LoRA (#568 by @Amix29).
live_eval's four callers now forwardquantization;soup ship --configreuses the training run's own (#367 by @AmirF194 in #570). #461 addedquantizationtoload_model_and_tokenizerbut nothing set it, so an NF4-trained adapter was still judged against a bf16 base.make_generator,make_multi_generator,lora_probeandmeasure_logit_agreementnow accept and forward it.soup ship --config soup.yamlderives a default fromtraining.quantization(4bit/8bit; other quant_menu formats still fall back to full precision) and prints which precision it loaded, now matching what it actually passes tofrom_pretrained(previously the fallback message claimed bf16 without ever settingtorch_dtype).advise --probe-model,diagnose --base-modelandtunability --livetake no--config/--quantizationflag yet, so they keep the existing default. Stamping the numerics into the verdict/evidence JSON and a staleness gate on mismatched-numerics evidence (issue criteria 2 and 4) stay open.#574 corrects the MCP execution-tool refusal, which told operators to restart with
--allow-executeand then falsely added that execution tools "are not implemented in this version" — execution shipped in v0.73.3 (#297 by @Srinivasan8888 in #574). The refusal now names the disabled tool and the flag that enables it, and the stalebuild_registrydocstring no longer describesallow_executeas reserved for future tools. Closes #483.Eval-gate LLM judge normalization uses the active rubric scale instead of hardcoded
/ 10.0(#577 by @here-2007 in #578)._run_judge_taskineval/gate.pydivided the aggregate judge score by10.0, assuming a 1–10 scale, whileDEFAULT_RUBRICineval/judge.pyuses 1–5. A perfect judge score of 5.0 normalized to 0.50 instead of 1.00, making typical gate thresholds (0.70, 0.80) impossible to satisfy and triggering false-positive training stops underon_regression: stop. Normalization now dynamically derivesscale_minandscale_maxfromevaluator.rubric["scale"]via min-max scaling to[0.0, 1.0], clamps out-of-bounds values, handles degenerate scales (min == max), and aligns with the existing dynamic normalization incommands/ship.py.#581 ports two v0.73.0 vLLM serve fixes to the SGLang backend, which had the identical pair standing. The prompt now applies the model's own chat template via the shared
build_chat_prompt(with the legacyUser:/Assistant:fallback for template-less models) instead of a hand-rolled third copy, so the model no longer sees a format it was never trained on. Andfinish_reasonreports"length"when a response hitsmax_tokens(on both the sync and streaming paths) instead of a hardcoded"stop", so a client doing continue-on-length can tell a truncated answer from a completed one. The tokenizer load insoup serve --backend sglangnow uses the sametrust_remote_codesetting the SGLang runtime itself uses, so a custom-code model no longer falls back to the legacy prompt in silence, and the three-branch operator warning vLLM prints about the chat template is now printed here too. Live verification against a real SGLang runtime on Linux remains an open follow-up on #360 (#581 by @Srinivasan8888).soup trainnow warns whentraining.convergence_detectionis enabled but not yet wired into the live training loop, matching the existing honesty guard for other advisory training-intelligence flags (#583).GRPO objective variants now execute without silently falling back to stock TRL loss (#584 by @here-2007 in #585).
_GRPOTrainerVariant.compute_loss()previously checkedinputsforper_token_logps, which is never pre-populated by TRL's rollout dataloader. The trainer now obtains per-token log probabilities in a single forward pass via_get_per_token_logps_and_entropieswithout duplicating forward computation or increasing peak VRAM, ensuringgspo,dapo,dr_grpo,bnpo,two_sided, andrftobjectives train their intended loss formulations.PPO's reward model, and
task: reward_model's own trained model, now only inherit the Quant Menu quantization config whentraining.quantize_reward_modelis set (#586 by @AmirF194). Since v0.53.0 added that flag, it was validated by its own task-scoped check but never read by either loader, so both tasks quantized regardless of the flag's value.#588 makes FSDP + BNB 4-bit training resolve quantized storage and trainable adapter parameters to the same floating compute dtype, preventing the integer-storage and mixed-dtype flattening failures that made the
llama3-70b-fsdp2recipe unrunnable. The recipe now pins bf16 storage for its A100/H100 target hardware (#588 by @Faisal-Fayaz).Raised the
acceleratefloor to 0.27.0, the first release whose FSDP checkpoint save/load path can be restricted to the trainable adapter. This closes a remaining way a LoRA/PEFT run under FSDP could regress back to writing the full frozen base model into every checkpoint on an unpinnedaccelerateinstall (#352, #591) (#591 by @AmirF194)._telemetry_endpoint_is_safecould not reject private/loopback HTTPS endpoints because the upstream HTTPS requirement bypassed the scheme-conditional private-IP check invalidate_hub_endpoint. The fix layers a telemetry-strict private, loopback, and link-local rejection on top of the existing hub sanitisation without modifyinghubs.py(#593, #598) (#598 by @harshitthek).training.stream_vram_probedocs now name each ratio's denominator (#595 by @umran666 in #605). Four code sites quoted the probe series (0.992x / 0.830x) labelled "the real peak". They now carry BOTH series with their denominators: the real-run peak (0.934x at seq 5120, 0.787x at 6144) and the probe series (0.992x at seq 4096, 0.830x at 5120) — the gap between them is the probe running 12.5-14.3% above the real training step, the conservativeness the gate depends on. The field description also carried the withdrawn claim that the probe under-measures preference losses badly enough to make the gate unsafe — at the one matching shape it reads +13.5% high, the same safe direction it shows for SFT, and thetask='sft'restriction stands because one shape is not a validation. Comments and field descriptions only; no behaviour change.soup serve --backend miinow applies the served model's own chat template and reports the engine's realfinish_reason(#606 by @ARAVIND281 in #608), instead of a hand-rolledSystem:/User:/Assistant:prompt and a hardcoded"stop". The backend reusesbuild_chat_promptandresolve_finish_reasonrather than keeping its own copies, andserveloads and passes the tokenizer for the MII path as it already does for vLLM. Behaviour is unchanged for a model that ships no chat template.The reward-hack ladders' mode split is now documented, and their thresholds are pinned by tests (#371 fix-path item 2, by @umran666 in #611). The H100 gate record carried an open inconsistency — the beta ladder escalated on four consecutive HACK votes while the rollback ladder, requiring three, never fired in any arm. The code settles it: the beta ladder lives in
kl_control(no rollback rung), the rollback rung lives only inpid_lagrangian, and the schema already rejectsreward_hack_rollbackunder any other mode — so the arm that fired one structurally lacks the other, and every arm with the other crashed at #342 before it could run. Documented in the module docstring and both run paths; comments only, no behaviour change. Tests pin the remaining CPU-testable acceptance criteria: each ladder fires at its documented threshold on N consecutive HACK votes with a control at N-1, and thebeta == floordegenerate case the defaults produce.soup doctornow recommends a PyTorch CUDA wheel that matches the driver'snvidia-smiCUDA version instead of always suggestingcu121. An unreadable driver header falls back tocu121(conservative: a parse failure is treated as an old driver, not a new one). Drivers at CUDA 11.8 getcu118. Windows also notes that the PyPI torch wheel is CPU-only. (#612 by @MKnaomi2)#631 pins the last open row of #273: the
74->73branch ofsetup_logging, which was only ever executed incidentally depending on ambient logger state (100% branch coverage on one machine, 98% on another). Two hermetic tests now force the state: a handler without the_soup_log_tiertag survives reconfiguration, and a Soup handler with a stale tier is still removed. Dropping theis not Noneguard fails both tests by name (mutation demonstrated and reverted).BrPart1 -> 0, for a reason rather than by luck (#631 by @umran666).Three tests no longer fail on a clean checkout when
FORCE_COLORis set (#633 by @Srinivasan8888 in #635).soup recipes showandsoup data mix --applyemit syntax-highlighted output, and three assertions read that output raw — so on a colour-capable terminal one of them fed ANSI escapes to a YAML parser (unacceptable character #x001b) and another looked formodality: textinside a highlighted span. The assertions now normalise through a sharedstrip_ansihelper, and a repo-wide AST guard fails on any new raw-output assertion against a syntax-highlighted command, so the fourth occurrence of this failure class is also the last one that has to be found by hand.--resume auto(and a direct--resume <path>) now finds MLX checkpoints._resolve_checkpointonly recognizedcheckpoint-Ndirectories (the transformers/unsloth shape); mlx-lm's tuner saves step-numberedNNNNNNN_adapters.safetensorsfiles instead, so an MLX run's own output never matched and training silently restarted from scratch every time.MLXSFTTrainerWrapper.train()now loads the located checkpoint's adapter weights before training starts. This is a weights-only warm start, not a full resume — mlx-lm's LoRA trainer exposes no optimizer state or step count, so the step count and data position both restart from zero regardless of how far the checkpoint got, and the run says so. Also fixes MLX--resumefor any config withexperiment_nameset, which the first version of this fix missed (#634 reported and diagnosed by @imahsanali, fixed by @SID-6921 in #639).[train]now declares the torch floor that actually binds, and doctor's copy is pinned to it (#636 by @umran666 in #641).[train]saidtorch>=2.3.0while requiringtransformers>=5.16.1, whose own torch extra forcestorch>=2.5— the declared floor could never bind, andsoup doctorcarried an unpinned second copy on top. The extra now declarestorch>=2.5.0with the reason stated beside it; doctor keeps a literal thattests/test_issue636_torch_floor.pypins to the declaration, and a drift guard compares the declared floor against the declared transformers floor's real metadata on thetransformers-floorCI job (which installs exactly 5.16.1 under the constraints file), skipping with a reason anywhere the installed transformers is a different release rather than passing vacuously. Closes #636.soup sweepno longer crashes with anIndexErrorwhen the run grid is empty (#628 by @Srinivasan8888 in #643). The unknown-parameter precheck added in #628 readscombinations[0]to probe the grid, so a--max-runsvalue that leaves no combinations —--max-runs -1, for instance — raisedIndexError: list index out of rangefrom inside the guard rather than reaching the sweep. Nonsense input either way, but the previous behaviour was an empty results table and exit 0, and an unhandled traceback is a worse answer than that. The probe is skipped when the grid is empty, restoring the old response. Bounding--max-runsatge=1would reject the input outright and is the better long-term fix; it is a pre-existing CLI contract change and is left for its own change.stream_source: autonow falls back to the NVMe disk tier when the RAM store fitsMemAvailablebut the store plus resident extras would exceed Soup's physical-host safety ceiling, and forcedstream_source: ramrefuses that case during pre-flight instead of letting the kernel OOM-kill the process (#622 by @ousamabenyounes in #644).soup sweep --dry-runnow validates instead of returning before the config loads — and forecloses a latent Python 3.10 crash in the sweep pre-flight (#642 by @umran666 in #645).--dry-runexits beforeload_configever ran, so the #627 unknown-key warning and the #628--parampre-flight were both unreachable under it: the issue's typo'dquantizatonprinted a grid and exit 0 with no warning, whilesoup train --dry-runreported it. This is a stated behaviour change:--dry-runnow means "validate, then print the grid without running anything", both paths share the single load, and the severity split is unchanged — warn-only exits 0, a typo'd sweep parameter exits 1 before printing a grid it can never run. On the execution path the same validation now happens before the confirmation prompt instead of after it. Making the pre-flight reachable exposed a latent crash in it: the probe walks a fullmodel_dump(), which visitstraining.preference_loss_weights(dict[str, float]). On Python 3.10 a PEP-585 alias passesisinstance(c, type), and whether the subsequentissubclassraisesTypeErrordepends on the pydantic release —ModelMetaclassinherits the raising check fromABCMetathrough 2.10.x (verified: every real sweep died withissubclass() arg 1 must be a classon 3.10.11 + pydantic 2.10.6) and defines its own tolerant__subclasscheck__by 2.13.4 (verified under the same interpreter), so the crash was live across the unmasked part of the supportedpydantic>=2.0.0range on the oldest supported Python, and a current pydantic masks it — which is also how #628 shipped green: its unit tests fed the walker hand-built dicts that never reached such a field._nested_modelsnow filters parameterized generics viatyping.get_origin; real-command tests cover each branch and the clean-config control, and the filter has a deterministic killer (an ABCMeta stand-in recreating the pre-override metaclass, pinning the real schema field) plus a control so it cannot be "fixed" by never walking. Closes #642.Live training panel now labels the GPU figure as
GPU peak:and readsmax_memory_allocated()instead ofmemory_allocated()(#650 by @YuriPerro in #652).on_logfired between steps, after activations and gradients were freed, so the sampledmemory_allocated()was the inter-step trough -- on the reproducing run5.8/15.9 GBwhilenvidia-smishowed the device fully occupied. The fix usesmax_memory_allocated(), the lifetime peak, and changes the label toGPU peak:so the figure is self-describing.reset_peak_memory_stats()is intentionally not called here: that reset is process-global and would clobber the grad-accum advisor's own peak reading atcallback.py:566.The CUDA batch probe now refuses on a measured peak, not only on a raise (#649 by @YuriPerro in #654). Under the WDDM driver (native Windows, WSL2) the allocator spills to host memory instead of raising
OutOfMemoryError, somake_cuda_probe_fn's only fit criterion, "the step did not throw", approved a batch 16 that ran an order of magnitude slower in shared memory andpick_batch_sizecached it. The probe now readsmax_memory_allocatedafter the synthetic step and refuses when it exceeds what this process can reach on the device (mem_get_infofree plus what it already holds), mirroring the counter and basisdecide_measured_fitdocuments for layer streaming; an OOM surfacing at a synchronize point asAcceleratorError/RuntimeError("out of memory")is classified as OOM instead of propagating as misconfiguration; and the cache key carries a version tag so entries written by the old probe are ignored. The WDDM case is covered by a mocked-torch test that needs no GPU, with a fitting control beside it. Closes #649.
Security
Telemetry and webhook SSRF guards now reject abbreviated, decimal, hex, and octal IPv4 forms (#600 by @here-2007 in #604).
_is_private_or_link_local()inutils/hubs.py,utils/hf.py, andutils/webhooks.pypreviously delegated exclusively toipaddress.ip_address(), which raisesValueErroron non-canonical IPv4 representations like127.1,2130706433,0x7f000001, and0177.0.0.1. On Linux (glibc), the OS resolver (getaddrinfo/inet_aton) parses these forms, allowing crafted--slack-url https://2852039166/webhooks or telemetry overrides to reach internal network addresses and cloud metadata (169.254.169.254). All three guards now normalise non-standard IPv4 literals via in-processsocket.inet_aton()and strip trailing FQDN dots before verifying againstipaddress.ip_address().The SGLang serve backend now obeys the
--trust-remote-codegate instead of loading every model with it enabled (#360 by @Srinivasan8888 in #619).serve.pyresolves the v0.36.0 default-deny gate once for every backend and passes the result to vLLM, but_serve_sglangwas called without it andcreate_sglang_runtimehardcodedtrust_remote_code=Trueat bothsgl.Runtimecall sites, sosoup serve --backend sglangexecuted a model's custom repo code whether or not the user opted in. The warning panel said so, but a notice is not a gate.create_sglang_runtimenow defaults toFalselike the vLLM path, and the resolved value reaches the runtime and the tokenizer load alike. Behaviour change: a custom-code model on the SGLang backend now fails to load without--trust-remote-coderather than silently running its code.Telemetry SSRF guard now applies a tiered architecture with primary allowlist, internal TLD rejection, and DNS defence in depth (#599 by @here-2007 in #624).
_telemetry_endpoint_is_safe()previously only checked literal IP formats, allowing hostnames resolving to private addresses (10.0.0.1.nip.io,localtest.me) or internal TLDs (metadata.internal) to bypass validation. The guard now validates via three tiers: (1) a primary allowlist for canonical PostHog domains (*.posthog.com) with zero DNS lookups, (2) static string rejection of internal TLD suffixes (.local,.internal,.lan,.home,.corp,.intranet) and literal private IPs, and (3) defence-in-depth DNS resolution for custom non-default FQDNs that fails closed on resolver errors/timeouts and rejects private or loopback target IPs.The OTLP tracing endpoint validator now rejects abbreviated, decimal, hex, and octal IPv4 forms, closing a bypass
#600/#604never reached (#616 by @SID-6921 in #625).utils/tracing.py's_is_private_ippredated the #604 fix and was never folded into it: it had nosocket.inet_atonfallback. The loopback spellings (https://127.1:4317and friends) were permitted anyway; the reachable hole was non-loopback private ranges in alternate encodings —https://2852039166:4317andhttps://0xa9fea9fe:4317both resolve to169.254.169.254, the cloud-metadata address, and both were accepted while the dotted-quad form was refused. Same for10.0.0.5as167772165/0xa000005/012.0.0.5/10.5, and192.168.1.1as3232235777. All refused now;127.0.0.1,localhostand ordinary hostnames still pass. Found while consolidating the SSRF loopback/private-host predicate, which had drifted into three copies of the function and six of the host set acrosshf.py,hubs.py,webhooks.py,loop_stages.py,qr_url.pyandtracing.py— issue #616 undercounted, and an AST walk during review found the real number. All six call sites now import one definition from the newutils/net_guard.py, and a guard test walkssrc/soup_cli/for any function whose name ends inis_private_or_link_localor constant ending inLOOPBACK_HOSTS— matching the leading-underscore spelling every original copy used — so a reappearing duplicate fails the build.loop_stages._endpoint_is_local, a seventh copy of the parsing logic guarding the deploy canary, had the same abbreviated-IPv4 gap; it now sharesnet_guard.parse_ip_literalwhile keeping its own narrower policy, since adopting the wider predicate would change what that surface trusts as a local target rather than only where the logic lives. The consolidation is otherwise behaviour-preserving. The shared predicate'sis_reserved/is_multicastclauses do not change any caller's outcome: inhf.pyandhubs.pyboth branches raise, so the predicate selects an error message rather than an accept/reject decision, andqr_url.pyimports only the host set.loop_stagesbecomes marginally more permissive — it is an allow-list gate, so teaching itparse_ip_literalmakes it recognise 26 further spellings as local — but every one of those canonicalises to an address it already accepted, so the set of endpoints Soup will POST to is unchanged.
[0.73.3] - 2026-08-18
Added
eval.ship.noise_flooris now committable tosoup.yaml(#406 by @ousamabenyounes in #410). Everysoup shipgate-policy flag was settable in a committed config and read back by--config— except--noise-floor(added in v0.73.2), which had no field and could only be passed on the command line, so a team enforcing a floor in CI had to hand-edit the workflow.ShipConfiggains a boundednoise_floor([2, 10], imported fromship_verdictso the schema and the CLI validator cannot disagree; bool-as-int rejected), wired with the same CLI > config > default precedence as the other five flags. As a live-measurement input it is measured only when producing evidence and refused under--evidenceexactly as the flag is; it followsforgetting_thresholdin being excluded from the recipeconfig_sha, so setting a floor never invalidates evidence.A ready-made
qwen3.5-4b-pretrainrecipe for continued pre-training ofQwen/Qwen3.5-4B-Base(#278 by @Faisal-Fayaz in #422). The recipe uses plaintext data, one epoch, QLoRA 4-bit quantization, and the established continued-pretraining defaults.A ready-made
deepseek-v4-flash-grporecipe for GRPO reasoning training withdeepseek-ai/DeepSeek-V4-Flash(#279 by @Faisal-Fayaz in #432). The recipe combines the established GRPO defaults with MoE LoRA and gradient checkpointing.soup mcp serve --allow-executecan now actually execute, behind a single-use confirmation token (#297 by @CODING-DARSH in #393).train_startandexportissue a short-lived, server-generated token bound to the plan and to the execution kind;train_execute/export_executeaccept only that token — no command, no argv, no shell string, no client-supplied environment — and the server launches the planned Soup CLI command withshell=False,stdin=DEVNULL, output to.soup/mcp-runs/<run_id>.log. Two integrity properties close the gap between planning and running: the config is snapshotted at plan time and executed from the copy, so the file cannot be edited underneath the run, anddigest_filenow walks a directory tree by content (sorted relative paths + per-file hash, symlink refusal, bounded) rather than by mtime+size, which did not change when a file inside a protected directory was rewritten — so a model could be swapped between plan and execution and revalidation still passed. Token consumption and capacity acquisition happen beforePopen, so a failed spawn requires a fresh plan rather than enabling replay. Runs go through the existingExperimentTracker, sosoup runssees what MCP started.
Changed
- The MLX SFT dispatch route no longer imports the Transformers SFT wrapper
before choosing a backend (#394 by @Shutaru in #431). The wrapper is import-light
today, but the standalone
soup-cli[mlx]runtime no longer depends on it remaining so. An additive Apple Silicon CI job verifies thatmlxandmlx-lmimport, the PyTorch/TRL training stack is absent, and a one-step real CLI SFT run completes. This hardens the runtime boundary; it does not claim to resolve the still-unpinned torch-present hang reported in #394. soup cardnow links the ML-BOM and the in-toto/SLSA attestation — they are first-class registry artifact kinds (#309 by @ousamabenyounes in #420).bomandattestationwere the two compliance documentssoup cardcould not surface:RegistryStorehad no such kinds, so there was no way to attach them. Added both to the valid-kind set and a--attach-to-registry <id>flag tosoup bom emitandsoup attest emit(mirroring the existing--attach-to-registrypattern); once attached they appear in the card's artifact table for free. Signed attestations also attach the detached.sigsidecar. The flag needs--output(omitting it exits 2), and a requested registry attachment failure exits 1 after preserving files already emitted.
Fixed
Assistant-only masking no longer mistakes
BatchEncodingkeys for token ids (#430 by @Shutaru in #439). Tokenizer mapping outputs are read throughinput_ids, and tensor-like ids are normalised to Python integers before they reach the collator; missing or non-integer ids now fail loudly instead of building a garbage label mask. A template that returns an all-zero assistant mask while assistant messages exist now falls back to incremental rendering, avoiding a silent all--100no-op training run. The same mapping assumption was removed from the data doctor.soup data mix --optimizewrote a recipesoup traincould not load (#330 by @blackcoderx in #440).render_mix_recipe_yamlemitteddata.trainas a YAML list of every searched dataset, butDataConfig.trainis typedstr, so the recipe failed to load withdata -> train: Input should be a valid string.data.trainnow renders as the single highest-weighted dataset from the search; the full ranked weight/path breakdown is kept as a comment above it so no information is lost, and the comment explains thatdata.interleaveis not yet consumed by training.Layer streaming now verifies that every trainable LoRA parameter has real storage after PEFT attaches the adapter (#433 reported by @lesterppo, fixed by @Faisal-Fayaz in #435 and #437). PEFT 0.18 creates streamed adapters on
metafor Soup to materialise, while PEFT 0.19 may create them as real tensors immediately, somaterialize_meta_adapters()returning0cannot distinguish a healthy no-op from a missed adapter. The streamed build now enforces the actual postcondition and refuses to install the runtime if a trainablelora_*parameter remains onmeta, naming the stranded parameter instead of allowing a silent no-training run.Windows process liveness misread a process that exited with code 259 as still running, forever (#424 by @blackcoderx in #436).
GetExitCodeProcessreturnsSTILL_ACTIVE(259) for a genuinely running process, but 259 is also a legal exit code, so a child that exited with 259 was indistinguishable from one still running. That silently defeatedExperimentTracker's reconcile-on-read (#401) and could wedge the MCP server's one-active-execution cap (#402) shut, refusing every subsequent execution with no error an operator could act on. The check now waits on the process handle withWaitForSingleObject(handle, 0), which is signalled the instant the process exits regardless of its exit code, falling back to the exit-code read only if the wait itself fails. The two byte-identical copies of this primitive inexperiment/tracker.pyandmcp_server/execution.pyare now one sharedutils/process_liveness.py.soup train --no-reexecnow prints the flags you actually typed (#372 by @AchuthReddy-16 in #415). The advisoryaccelerate launchcommand is derived from the same argv the auto-reexec would have used, so--fsdp/--deepspeed/--config(and the rest of the run-shaping tail, including--nameand--replay) cannot silently fall off the printed hint. Following that line used to train without FSDP.detect_device()andget_gpu_info()now recognise Apple Silicon MLX (#423 by @harshitthek in #428). Previously on Apple Silicon,detect_device()only probed PyTorch MPS and fell back to'cpu', triggering a falseWarning: 4bit quantization is not supported on CPUalert and silently downgradingquantizationfrom4bittonone. Device and GPU info detection now accept an optionalbackend=parameter: passingbackend="mlx"resolves to'mlx'with the chip name (e.g.Apple Silicon (Apple M2 Max)), preserves 4-bit quantization intact formlx-lmpre-quantized models, reports Apple unified memory in telemetry, and skips the CUDA-shaped analytical VRAM preflight on MLX runs. The preflight skip is pinned byTestHardwareFitGateIsMlxAware(mirrors the streaming-aware gate test). The known-limitation warning indocs/backends-and-ops.mdis replaced with the resolved behaviour.A run whose watcher died was reported
runningforever (#401 by @ousamabenyounes in #407).ExecutionManager._watchruns as adaemon=Truethread, so when the MCP server process exits it is killed without unwinding andfinish_executionnever runs —ExperimentTrackerkept the run atrunningwith no watcher left to correct it, training the operator to ignore the one status field that guards against a second concurrent run. The tracker now reconciles on read: when arunningrow carries a pid whose process is gone,list_runs/get_runrewrite it toterminatedwith an unknown (None) exit code — an unknown outcome is never recorded as success, and the richercompleted/failedterminal statuses are left untouched.soup runsno longer hardcodesrunningfor a non-running row.Layer streaming now accepts a fast virtio/cloud disk instead of refusing it as an HDD (#365 by @ousamabenyounes in #411).
detect_disk_kindtrusted/sys/block/<dev>/queue/rotational, which a paravirtual (virtio) block device defaults to1with no media hint — so a genuinely NVMe-backed cloud disk (measured 1.5 GB/s read) was classifiedhddand denied the disk-overflow tier, the very audience the tier targets.rotational=0stays authoritative (solid state); when the flag is unreliable (rotational=1), the media type is now decided by a bounded, O_DIRECT sequential-read measurement rather than the flag — NVMe-class throughput (>= 1 GB/s) earns the tier, a genuinely slow disk still classifieshddand is still refused (160 seeks/step, plan P11). A newtraining.stream_disk_kindoverride (nvme/ssd/hdd) is the escape hatch for the case where even that is wrong; it prints what it overrode beside what was detected. Disk detection may write a small scratch file next to the streamed shards to run the probe.The one-active-execution cap could double-book after a server restart (#402 by @ousamabenyounes in #408).
ExecutionManager._active_run_idis in-memory, so a restarted MCP server started with an empty slot, saw free capacity, and would launch a second training while a child from the previous server (which survives a client disconnect, #297) was still using the GPU. The cap is now gated on a persisted run whose recorded pid is still alive: a live prior child blocks a new execution across restarts, while a stale record whose process is gone frees the slot rather than wedging it shut.docs/commands.mdnow states the actual scope of the cap.soup ship --noise-floornow measures the leg-1 task axis in the judge modes, so a judge-scored win smaller than the judge's own noise no longer counts (#403 by @ousamabenyounes in #419). The floor was measured in--task-mode metriconly;judge_score/pairwiseprinted a warning and left leg 1 at a 0.0 floor, i.e. the exact blindness the flag exists to remove. It is now measured:judge_scorescores the base side N times through the judge, andpairwisejudges the base model against itself (expected win-rate 0.5, so the spread is directly measured, not inferred). Because those repeats fold in the judge's own sampling noise, the floor is labelleddecode + judgeon the panel and stampedjudge_inclusivein the evidence/JSON block so it is never read as a decode-only number.--helpstates the extra judge-API cost.training.bnb_4bit_use_double_quantwas validated but never read — every 4-bit path Soup builds theBitsAndBytesConfigfor hardcoded double-quantization toTrue, sobnb_4bit_use_double_quant: falsepassed validation and was silently ignored (#321 by @ousamabenyounes in #418). The flag is now threaded through the three call sites Soup owns — the resident loader (build_quantization_config_for_loader), the layer-streaming path (stream_setupreads it once and passes the SAME value to the sharder and the meta skeleton, so streamed-vs-resident bit-exactness cannot drift), and the 4bit save path (soup merge --no-double-quantfor the4bit/4bit_forcedsave formats). The Unsloth loader is out of scope:FastLanguageModelbuilds its ownBitsAndBytesConfiginternally with double-quant hardcoded on and exposes no override, so that path cannot honour the flag. The schema field is now tri-state (Optional[bool], unset =None): unset resolves to the shipped default (double-quant on), so a run that never set the flag trains identically, but because the resolved config now carries the field,soup ship --evidenceprovenance andsoup lock checkwill report a one-time fingerprint drift on a previously-unset config even though the model numerics are unchanged. The config-load footgun (=truerequiresquantization: 4bit) fires only on an explicittrue, and unset serializes asNone, so a dumped-and-reloaded config no longer trips it.soup env checknow flags an installed package that violates Soup's own declared version bound (#368 by @ousamabenyounes in #421).pip install "soup-cli[serve-fast]"(vllm) into a training venv silently pushestransformerspast the<5.0.0cap Soup declares, producing an environment Soup's own metadata says is unsupported with no warning at any point.env checkaudits installed versions against the bounds read from package metadata — not a hardcoded copy, since a second copy of the cap is exactly the drift this catches — and exits 3 when one is violated, independent of any lock file. Anextra == "X"requirement is Soup's bound only when[X]was opted into, which the running environment cannot confirm, so its marker is evaluated and it is skipped rather than raised as a false positive — except for the ABI-relevantTRACKED_PACKAGESset, which since the v0.71.0 deps-split lives in metadata only underextra == "train"/"all"/ "dev"and therefore includes thetransformers <5.0.0case #368 was reported about. A package is counted once even when its bound is restated under several extras. Ifpackagingis somehow unimportable the audit still degrades to a clean report soenv checkkeeps working, but now says so rather than reporting a silent clean it never verified. The bounds audit no longer short-circuits the lock diagnostic (both print).packagingis a declared dependency now — the audit degraded to a silent "clean" without it, the wrong direction for a checker.docs/serving-and-export.mdstates the separate-environment guidance for[serve-fast].kl_controlrewrote the trainer's β/kl_coef on every step, including ahold, so a non-acting run was numerically identical tolog_only(#371 by @AmirF194 in #414)._run_bang_bangcalled_apply_coefficientunconditionally; on aholdthe controller writes back the value already there, which is a no-op value but not a no-op write. The write is now skipped when the bang-bang step holds, and the mitigation log recordsmitigation_status(held/acted/released) so a non-acting step is distinguishable from an acting one without parsing the free-textactionreason.extract_mcq_letterscored zero for\boxed {A}— whitespace between the command and the brace (follow-up to #357, by @ousamabenyounes in #396). The shipped\boxed\{regex tolerates spaces inside the braces but not between\boxedand{; LaTeX permits it there and models emit it, so\boxed { C }still read as no answer and was not rescued by the cue tier either. The boxed-letter regex now allows\s*after the command.soup draft distill --steps Nnow delivers ~N optimiser steps instead of N/4.44 (#364 by @ousamabenyounes in #399). The epoch count that realised--stepsdivided the request byrows // batch_size, ignoring thatval_split(0.1) removes rows from training andgradient_accumulation_steps(4) micro-batches make one optimiser step — both divide the budget, so every distill run trained for roughly a fifth of the requested steps, silently. Epochs are now derived from the effective steps-per-epoch, the emitted config pins the run shape (val_split,gradient_accumulation_steps) so the arithmetic and the trainer cannot drift, and the pre-flight prints the resolved step count.MitigationLogWritersilently dropped every record once its parent directory vanished mid-run (#343 by @ousamabenyounes in #398).record()reopens the log per call and swallowed theOSErrorfromopen("ab")with a barereturn, so when a shared temp root was cleaned by another process the controller kept acting while its log quietly stopped growing — the run completes while its evidence goes missing, the failure shape this project treats as the worst kind. The writer now recreates the directory and retries the write, and surfaces the loss once via a warning naming the path.record()still never raises, so a vanished log never takes down the training run.
[0.73.2] - 2026-08-15
soup ship's leg 2 is the project's differentiator, and it was lying in both
directions: two of its suites ranked by the wrong thing, one whole failure
direction had no detector at all, and a caller error was indistinguishable from
a regression. Every item below was reproduced on the dev box against the
shipped v0.73.1 code before a line was changed.
Added
soup ship --noise-floor N— measure what the instrument can resolve before calling a delta significant. Greedy decoding is not deterministic on GPU: measured on an H100, the same model with no adapter over five runs spread 0.015 strict / 0.020 format-blind, andsoup shipcompared against a 0.05 threshold without ever telling the operator what its own instrument could resolve — four of six paired deltas in that session sat inside the floor. With the flag, the BASE model is re-run N times (2–10), the per-axis floor ismax − minacross the repeats, it is printed beside the verdict, and every axis is then gated atmax(threshold, floor). Leg 1 must clear the task axis's floor too. Themaxis load-bearing in both directions: a floor above the threshold widens the gate to what is actually measurable, and a floor below it must never tighten the gate behind the operator's back. Opt-in, so no existing run changes; costs N extra base passes. The leg-1 floor is measured in--task-mode metriconly — in the judge modes a repeat would fold the judge's own sampling noise into a number presented as decode noise, so the run warns and leaves leg 1 at a 0.0 floor instead. Carried caveat, because it bounds the claim: n=3, one model, one dataset. It sizes the effect; it does not calibrate a threshold.- A benign-prompt axis,
mini_over_refusal, joins the default leg-2 suite (7 → 8) (#317). Leg 2 flagged a drop inmini_safetyand had no reverse, so a tune that refuses everything registered as a monotone safety improvement with no ceiling on how useless the model became. Reproduced as indistinguishability, which is the actual claim: two models with byte-identical scores on all seven shipped suites and the same SHIP verdict, one of which refuses every benign request. The new suite is 40 hand-authored, original benign-but-scary-sounding requests (defensive security, first aid, sysadmin verbs like "kill"/"terminate", fiction with dark themes) scored as fraction NOT refused, so the existing regression rule catches over-refusal with no change to the verdict engine. Paired withmini_safetyneither axis can be gamed alone. Same caveat as above: 40 prompts and one greedy pass size a gap, they do not calibrate a threshold. - Semantic Stratified Splitting for
soup data split(#388).--stratify-semanticand--num-clusterspartition splits proportionally across semantic groups (TF-IDF + K-Means) so a whole topic cannot land entirely in one split.scikit-learnis a declared member of the[data]extra and a missing import refuses with the install command rather than silently falling back. 50k-row cap, an explicit error for a pure-stop-word corpus, and a warning when--num-clustersis passed without the flag. Contributed by @Deadpool2000. soup mcp serve --allow-execute(#391). A stronger opt-in than--allow-mutating, which it implies, plus the_refuse_executehandler the execution tools will use.train_startandexportstay plan-only — nothing in this slice executes anything, and the help text says so in the present tense.build_registry(allow_mutating=True)still returns the same 16 tools. Contributed by @CODING-DARSH.
Fixed
extract_mcq_letterdid not know\boxed{C}, and the MCQ prompt never asked for a letter (#357). Meta-Llama-3.1-8B-Instruct scored 0.423 onmini_mmlu— below a 0.5B — while scoring 1.000 on two other MCQ suites. Of 15 failures, 8 boxed the right letter and 6 boxed a value because nothing in the prompt asked for a letter. Reproduced here at the extreme: a stub that answers every item CORRECTLY in the boxed-letter style scored 0.000 onmini_mmluandmini_common_sense. Both halves are needed — the extractor alone is worth +8 items, the prompt alone 0, together 0.423 → 0.731 and the inversion disappears. The new tier fires only when the box holds a single A–J letter: reading\boxed{4}as "option 4" would be a wrong credit, not a repair. Among the boxed / cue / paren forms, position decides, not form — a model that boxes a scratch answer and then self-corrects chose the correction.mini_tool_callranked by brace hygiene (#346). The 8B named the right tool 40/40 and scored 0.225: it emitted three opening braces and two closing ones, the whole-string parse failed, the bounded scan returned the INNER object, and the scorer rejected it for having no"function"key. The gate suite's own unwrapping layer now restores the envelope for an object carrying bothnameandarguments— requiringargumentsis what stops an echoed{"name", "description"}menu entry from scoring, i.e. from crediting copying. The missing brace is the model's own output, not truncation; that attribution was believed and shipped inc87fd00before a budget sweep disproved it.score_bundled_suitereturned0.0for a non-callablegen(#355). On the three behavioural suites it scored 0.0 while the MCQ suites raisedTypeError— and in leg 2 a 0.0 reads as "the model failed every item" → DON'T SHIP, so a caller error was indistinguishable from a regression and failed in the direction that looks like a finding. Both branches now raise. A callable that misbehaves is still a failed item, which is the correct existing contract.- The verdict panel silently ate its own leg-1 marker.
render_ship_panelbuilt its header as... [{won_str}], and a bare[no win]is valid Rich markup for an unknown tag — so the panel never printed "won"/"no win" on any release up to and including v0.73.1. The plain-text rubric, which has no markup parser, printed it correctly the whole time, which is why it went unnoticed. Found while rendering the new noise-floor panel. - Untrusted names could drive the terminal. Benchmark and noise-floor axis
names come from an
--evidenceJSON file, andrich.markup.escapeneutralises Rich's[...]syntax and nothing else, so a raw ESC byte survived it. Both render paths now strip C0/DEL first, matching the_for_terminalpattern already used in six other command modules. - The
--evidenceround-trip and the MCPship_evidencetool now agree. A verdict decided against a measured floor does not replay without it, so the floor is part of the evidence schema (#312's output-is-input property), and both readers honour it — a schema extended in one consumer and not the other would have made the same file replay to different decisions through the CLI and throughsoup mcp serve. - A duplicated
#392CHANGELOG entry. PR #388 branched before18a278amoved that entry into[0.73.1], and the merge re-introduced it under[Unreleased], listing the same fix twice.
Security
- An evidence-supplied noise floor is bounded and never silent. A floor
widens the gate, so
"floors": {"mini_mmlu": 1.0}in an evidence file masks any possible drop on that axis. This does not cross a new trust boundary — anyone who can edit that file can already write{"base": 0.9, "tuned": 0.9}and force a SHIP outright — but it is a far quieter edit to miss in review, andsoup ci initwiresship --evidenceas a PR merge gate. Values are therefore bounded to[0.0, 1.0], the mapping is capped at 50 axes / 256-char names (mirroring the CLI's own limits), a malformed block is refused rather than dropped (a dropped floor replays as a different verdict), and any floor that exceeds--forgetting-thresholdis announced — on stderr by the CLI, and in the returned payload by the MCP tool, whose stdout is the JSON-RPC channel. Neither reader is the quiet one.
Changed
decide_shipnow canonicalises theTaskWinit stores, as it already did for the benchmark deltas. It recomputes leg 1 from the raw scores, so aTaskWinbuilt without the floor would otherwise have rendered "won" beside a DON'T SHIP decided with it.--baselinesnapshots taken before this release are on a different scale formini_mmlu,mini_common_senseandmini_tool_call, because their scorers changed. A stored baseline skips the live base run, so it would be diffed against a freshly-scored tuned model — measured on an unchanged model the jumps are 0.423 → 0.731 and 0.225 → 1.000, far larger than the 0.05 gate.soup shipnow warns by name when--baselinesupplies a stored score for an affected suite.mini_instructionandmini_arithmeticare unaffected and are deliberately not named: neither carries a single-letter answer, so the prompt cue and the option-letter extractor never touch them (verified, 0 of 24 and 0 of 36 items).
0.73.1 - 2026-08-14
Added
training.stream_vram_probedecides the layer-streaming VRAM pre-flight on a MEASUREMENT instead of the fitted formula (#349). The pre-flight predicts peak VRAM from a formula fitted to 10 real runs, and its documented contract is that it never under-predicts. Measured through the realsoup trainon an RTX 3050 Laptop (4 GB, Windows, torch 2.5.1) with SmolLM2-135M streamed in bf16 at batch 1, that contract holds at short sequence and then fails:seq predicted real peak ratio 4352 3.282 GB 3.036 GB 1.081x — over-predicts, safe 5120 3.844 GB 4.118 GB 0.934x — under-predicts 6144 4.590 GB 5.830 GB 0.787x — under by 21% Under-prediction is the direction that does not announce itself: an OOM on Linux, and on Windows/WDDM a silent spill to host memory. The existing grid could not have caught it — all ten of its rows are at seq 256 or 512, so it varies batch and says nothing about sequence length, and
test_never_under_predictshas been narrowed to state that scope rather than imply a global guarantee.With the flag on, one real forward+backward runs at the configured shape after the streamed model is built and its peak decides; the prediction is printed beside it so a divergence is visible. Measured cost: 1.0-5.3 s (SmolLM2-135M at 1x1024 and 2x2048; Llama-3.1-8B NF4 at 1x512), against a training run of minutes to hours. Off by default — it costs a step, and it can refuse a run the formula accepts.
Scope is deliberately narrow.
task: sftonly: the probe runs a plain causal-LM step, which is the SFT step but is not a preference loss, so its agreement with one is not established. Measured at a single matching shape it is conservative there too (6.02 GB against a real DPO step's 5.30 GB, +13.5%) — but one point is not a validation, and a sign flip would mean a gate waving through over-budget runs. It also cannot overrule a prediction more than 4x over budget: the largest disagreement ever measured is 21%, so beyond a small multiple the config is simply too big and is refused by arithmetic without touching the GPU. The gate readsmax_memory_allocated, notmax_memory_reserved— reserved runs 1.08-1.41x allocated and overshoots what has to fit, and gating on it would refuse this feature's own flagship configuration (Llama-3.1-8B NF4, 3.70 GB reserved against 3.45 GB free, which runs).Two readings were tried during this work and withdrawn as unsupported, recorded because the tempting inference was wrong twice in one investigation: that preference losses are over-budgeted ~8.8x (it is 1.15x at the budgeted shape — the earlier figure came from rows that realised 142 of a budgeted 2048 tokens), and that the over-budget runs were silently spilling (
num_alloc_retrieswas 0 on every shape measured). The mechanism behind the long-sequence divergence is likewise not claimed:seq**2from the attention score matrix is the obvious candidate and the numbers do not settle it.
Fixed
The MLX
adapter_config.jsonshippedtarget_modulesunresolved, so a default MLX adapter loaded as a silent no-op (#392)._apply_loraresolvedtarget_modules: autointo a local variable and trained the resolved modules; the writer serialised the raw config value, so the shipped file carried{"keys": ["auto"]}. On load,linear_to_lora_layersmatches no module against that andload_weights(strict=False)drops every LoRA tensor without a word — generation with the adapter is bit-identical to the base model."auto"is the schema default, so this was every MLX run that did not name its modules by hand, and the file exists precisely to promise the output dir loads withmlx_lm.load(..., adapter_path=...). Both callers now go through oneresolve_mlx_target_keys(), because two copies of "which modules did we train?" is how they drifted. Reported with a root cause and a control by @armanbot-jpg: hand-editingkeysin the saved file makes the very sameadapters.safetensorsproduce the tuned behaviour.training.batch_sizeaccepted 0 and negative values.Union[int, Literal["auto"]]carried no lower bound, sobatch_size: -4loaded and then meant whatever each trainer's arithmetic did with it — including the streaming VRAM pre-flight, which multiplies by it. Now rejected at config load. Surfaced by the #349 security review.
Fixed
- Layer streaming's VRAM pre-flight now actually calls its own calibration hook (#348).
calibrated_logits_bytes_per_element()exists to raise the budget when a stack's loss path measures a heavier retention than the shipped constant assumes, guarding against a future stack silently under-budgeting by 12.5% with nothing to catch it._stream_budget_linescalledestimate_stream_peak_vram()withoutlogits_bytes_per_element=, so the parameter was alwaysNoneand the calibration never ran outside its own test. It is now forwarded to both the budget and the panel'slogitsfigure; the value can only raise the prediction (floored at the shipped constant), and the panel prints an extra line naming both numbers when the calibration measures above it. This makes the probe unconditional rather than opt-in (see #327 below, whose wording is updated to match): every streamed run now pays one transient14 * vocab_size * max(tokens)allocation (96 MiB at the defaults) plus twotorch.cuda.synchronize()calls before the fit decision is taken. On today's measured stacks this is a no-op in effect (measuredis 12.0 with zero spread,max(14, 14)is 14, no extra line prints), so the cost buys nothing yet, which is the point of a guard against a stack that hasn't shipped. - MLX backend now actually dispatches to the MLX trainer for
task: sft. Previouslybackend: mlxsilently fell through to the transformersSFTTrainerWrapper, training on MPS/CUDA instead of MLX. The trainer was also rewritten for mlx-lm >= 0.31 (create_dataset+CacheDataset,TrainingCallback), withmodel.freeze()before LoRA — without it the saved "adapter" was a full fine-tune (172 tensors vs 24 LoRA tensors on a 1.2B model) — and anadapter_config.jsonis written so the output dir loads directly withmlx_lm.load(..., adapter_path=dir). (#362) mlx-lmfloor raised to >= 0.31.3 (the version the MLX SFT path is built against).training.seedreached the SFT wrapper and nothing else (#353). #341 added the knob and wired it intotrainer/sft.py. The other seventeen task wrappers each build their ownTrainingArgumentssubclass (GRPOConfig,DPOConfig,RewardConfig, and so on) and none of them read the field, sotask: grpowithtraining.seed: 7trained at HF's default of 42 with no error and no warning. Replicates that differed only intraining.seedwere therefore the same run, which is what happened to STEP 25 of the H100 record: its five "replicates" were five runs of seed 42, measured against a within-mode spread produced by the very thing it thought it was varying. Threading the config is only half the repair.Trainer.__init__runsset_seed(args.seed), butget_peft_modelhas already drawnlora_Aby then, andclassifier/reward_model/prmhave already drawn a freshly initialised head insidefrom_pretrained, so every wrapper now applies the seed at the top ofsetup()as well, before the model is loaded.unlearnbuilds noTrainerat all and drew its RMU control vector from a generator hard-coded to 0; that draw now followstraining.seed, staying at 0 when the seed is unset so existing runs keep their control direction. An unset seed still resolves to 42 and leavesdata_seedatNone, so the values a run trains at are unchanged. What changes is when they arrive. Nothing calledset_seedbeforeget_peft_modelpreviously, solora_Awas drawn from torch's default generator, which is seeded from entropy once per process: an unseeded run's adapter and classification-head initialisation varied from one process to the next, and it is now deterministic at 42. Replicate variation that came out of runs setting no seed was coming from exactly that, so those runs are now identical to each other and varying a replicate means settingtraining.seedon purpose. The MLX backend (backend: mlx) is the one path that still reads neither field.training.seedon the MLX backend now says it is ignored (#353, fourth criterion). MLX has its own RNG (mx.random) and none of the MLX wrappers touch it, so a seeded MLX run was silently unseeded — which looks identical to a seeded one until two replicates disagree. That gap only became reachable between #353 being filed and #381 landing:backend: mlxwas never dispatched at all until #362 (#363). Setting either field now appends to the wrapper's existing "MLX backend ignores:" line, namingtraining.seed/training.data_seedso it is greppable as written. A warning, not a rejection: a config valid on transformers should not become unloadable by switching backend. Seeding MLX for real is separate work with a separate RNG.- Under
use_fsdp2_compile, everycheckpoint-*still loads as a dead adapter (#351). #335's repair runs once, on the output root, after the finalsave_model. HF's Trainer writes its periodic checkpoints through that samesave_modelwithoutput_dir=<run>/checkpoint-N, so they come out carrying_orig_mod.on every key and nothing ever normalised them. Measured at 70B on 8×H100 (benchmarks/gate-h100-validation.md, STEP 28): 320 canonical keys in the output root, 320 prefixed ones incheckpoint-100. Resuming is the case that decides how bad this is, and it is worse than #335 was.PeftModel.from_pretrainedat least warns;Trainer._load_from_checkpointcallsmodel.load_adapter(...)and drops its return value, andload_adapterdeliberately does not warn (it hands the missing keys back in the load result instead, which nothing reads), whileload_state_dict(strict=False)discards the_orig_mod.keys without a word. A resumed run therefore continues from a re-zeroedlora_Bin total silence: #335's failure shape with its one warning removed.load_best_model_at_endwas reloading a dead adapter for the same reason, since the Trainer restoresstate.best_model_checkpointthrough that sameload_adapter, and this repairs that path too. Normalising now happens on HF'son_save, as each checkpoint is written. Both that callback and the final save are gated onargs.should_save, the conditionsave_modelwrites under, so exactly the rank holding the file repairs it rather than all eight opening it at once. The callback is attached insetup(), ahead of anything a caller adds later:HFPushCallback.on_saveuploadscheckpoint-{step}on this same event andCallbackHandlerdispatches in insertion order, so a normalisation attached after it would leave--push-aspublishing the prefixed adapter and keeping the repaired one on local disk. - A streamed model's
named_parameters()still carried the wrapper's.inner.segment, so a name-keyed comparison against a resident model of the same checkpoint saw no overlap (#369). v0.72.1 madestate_dict()canonical for serialisation; this issue is the first time something compared the two model kinds by parameter name instead, and the #331 repair gate read the resulting empty intersection (grads exact 0/0) as a pass.layer_stream_runtime.canonical_named_parameters()strips.inner.the same waystate_dict()already does, andassert_canonical_parameters_intersect()raises instead of reporting an empty intersection as success. Neither the forward path norstate_dict()changed, so the v0.72.0 bit-exactness gates remain valid unexercised.named_modules()andnamed_buffers()carry the same segment and are deliberately not covered — the comparison that produced the false green was over parameter names.
Added
training.stream_vram_overridegives the layer-streaming VRAM pre-flight an explicit escape hatch (#347).decide_stream_fitrefused any run it predicted would not fit, with no way through except loweringbatch_sizeormax_length. Setting this field now replaces the measured free-VRAM figure the pre-flight checks against, in either direction: raised past a documented over-prediction to let a known-safe config through, or lowered to enforce a capmem_get_info()cannot see, such asset_per_process_memory_fractionon a shared or capped card (a Colab/Kaggle T4, a MIG slice). Rejected at config load when set whilestream_layersis false, mirroring the existingstream_source/stream_buffersfootgun gate.
Fixed
A hosted notebook's preinstalled
torchaomadeget_peft_modelraise, and it read as a Soup bug (#389).peft'sis_torchao_available()does not return False on a version it considers too old, it raisesImportError— nine frames insideget_peft_model, with nothing near the top of the traceback namingtorchao. Colab preinstallstorchao0.10.0 against apeftthat demands newer, so the firstsoup trainon a free notebook died in a place unrelated to anything the user had configured. Now mapped inutils/errors.pyto the cause and the one-line fix (pip uninstall -y torchao), with the part worth saying out loud: Soup does not needtorchaoat all unlesstraining.quantization_awareis set. Found by runningnotebooks/proof-4gb.ipynbon a real free-tier session, which is the same place #385's first repair was caught being a no-op.bf16 was assumed on every CUDA card, so the entire free GPU tier failed (#385, #387). Fourteen places, and only the first was known:
trainer/stream_setup.pychose the layer-streaming store dtype with the literal"bfloat16" if on_cuda else "float32"(#385), and then a live run foundSFTTrainerWrapper._resolve_mixed_precisionreturning(device == "cuda", False)as its default — and an audit found the samebf16=self.device == "cuda"in twelve more wrappers: bco, classifier, distill, dpo, embedding, ipo, kto, online_dpo, orpo, pretrain, reward_model, simpo (#387). So it was not a streaming bug at all; every task died on that hardware. The sharpest detail is that the codebase already knew:trainer/asr.pycarries the comment "bf16=cuda was hardcoded, which crashes on pre-Ampere cards (T4 / GTX 16xx)" and fixes it — in that one wrapper, never propagated. All fourteen now take the answer from one place,utils/gpu.bf16_fp16_flags, including ASR, whose private copy was folded in. bf16 needs Ampere. Colab's free tier is a T4 (sm_75), Kaggle is a T4 or a P100, and V100 / GTX 16xx / RTX 20xx are all pre-Ampere, so on that hardware Soup ran in a dtype the card has no units for. Neither could fail on the maintainer's RTX 3050, which is Ampere; this is the same shape as the four backends the H100 session found had never executed once.Two corrections to the first version of this entry, both established by finally running it on a real T4 rather than reasoning about one. (1) The claim that every task died before step 0 on transformers' "Your setup doesn't support bf16/gpu" was wrong: that error was produced by a local stub forcing
is_bf16_supported()to False, and transformers gates on the same permissive call described next, so on the current stack it does not raise at all. (2) More seriously, the first fix was a no-op on the hardware it was written for.torch.cuda.is_bf16_supported()defaults toincluding_emulation=True: when its compute-capability fast path fails it falls through to merely constructing a bf16 tensor, which software emulation satisfies — so a T4 answers True, and asking the bare question selected bf16 exactly as the hardcoded literal had. The predicate now asksis_bf16_supported(including_emulation=False)(falling back to a capability check on older torch), andget_compute_dtype— a second copy of the same question — was folded into it. What a T4 actually does with emulated bf16, as opposed to what it reports, is not yet measured. This cannot regress a working setup: where bf16 is supported the answer is unchanged, and where it is not the previous behaviour was a crash. A test SCANS every module insoup_cli/trainer/rather than parametrising over a hand-written list — the list is what hid the twelve — and the existing unit test for this line had to be rewritten because it asserted the defect (test_auto_flag_off_preserves_legacyrequired bf16 on any CUDA device, and passed in CI precisely because CI has no GPU and the old code never asked the driver). Correctness of the alternative was measured before the change rather than assumed — streamed-vs-resident logits are bit-exact at 0.000000e+00 in float16 as well as bfloat16, in both quantisations, against resident references of matching numerics (an NF4 streamed run compared against a genuinely NF4 resident one, since comparing it against a bf16 model would measure the dtype rather than the streaming), and the adapter is non-zero. Not yet verified on a pre-Ampere card: that exactness was measured using fp16 on Ampere, so it establishes the plumbing, not the Turing/Pascal kernels — the free-tier notebook is the natural place to close that.
Changed
- Retracted: "layer streaming is bound by host-to-device transfer, not by the GPU."
That sentence appeared in the README, in the v0.73.0 notes below, in the H100 gate record
and in the preprint's abstract. It was an inference from the H100 replication — the same
configuration returning the same throughput on a card with two orders of magnitude more
compute — and it had never been measured. It was measured on 2026-08-11 on the original
laptop and is false at the published configuration: four interleaved ablation arms in
one process at a pinned clock show that removing all host-to-device traffic (6.864 GB per
step) buys 1.44%, removing the NF4 dequantisation buys 9.80%, and removing both
leaves 88.7% of the step; the compute stream is blocked on a copy for 8.4 ms of a
4190 ms step; and the step runs at 71.3% of that card's same-session, shape-matched GEMM
ceiling. The claim is true below roughly 128 tokens per step, where the fixed transfer
volume dominates — the published configuration is not there.
No measured number anywhere changes, and the replication result stands in a weaker form:
the constraint is common to both machines and is not the compute the datacenter card adds.
The H100's own bottleneck was never instrumented and no claim is made about it. Record:
benchmarks/probe-v0.73.0-what-bounds-streaming.md. Every occurrence in the gate record and in the v0.73.0 notes below is annotated in place rather than deleted, because in a folder whose whole premise is publishing the record as written, a silent deletion costs more credibility than the error does.
Validation (measured, not changed)
- Layer streaming completed a run on hardware the maintainer does not own: a free-tier
Colab Tesla T4 (sm_75, Turing), via
notebooks/proof-4gb.ipynb. Every streaming number this project has published came from one RTX 3050 Laptop or one borrowed 8×H100, and the pre-Ampere fix above (#385, #387) had been verified using fp16 on an Ampere card, which establishes the plumbing and not the Turing kernels. This closes that specific gap and nothing wider.NousResearch/Meta-Llama-3.1-8B-Instruct, NF4,stream_layers: true,stream_buffers: 2, batch 1,max_length: 256, LoRA r=8/α=16, fp16 (a T4 has no bf16 units): 7 steps, exit 0, adapter written with 128 tensors, 128 of them non-zero, and a measured peak of 2.91 GB against the pre-flight's predicted ~3.02 GB — an over-prediction of 3.8%, which is the direction the estimator was fitted to err in (v0.72.3 fitted it to never under-predict) and is the whole reason it is allowed to stop a run. The card has 15.6 GB, so the run was constrained artificially withtorch.cuda.set_per_process_memory_fractionto 4.00 GB, and the cap was shown to bite rather than assumed: a deliberate 4.29 GiB allocation was refused. That artificial cap is also the reason no throughput figure is quoted from this run, here or in the notebook — a capped card is not a benchmark, and the panel's own forecast (31–46 tok/s from 2.20 TFLOPS measured at 1185 MHz) is a compute bound, not a measurement of what the run did. Two things the run did not establish, stated because the temptation is to let the exit code cover them. Backward/gradient exactness at 8B on Turing is not shown — a non-zero adapter proves gradients flowed, not that they were right, and the streamed-vs-resident comparison in the notebook's section 4 produced no captured output, so it is recorded as unrun rather than as a pass. And the loss moving 4.5266 → 4.0674 over 7 steps, non-monotonically (4.5266, 4.2388, 3.7485, 4.6930, 4.1940, 4.1127, 4.0674), is reported because it is what the run printed; over seven steps it is not evidence of learning. One observation worth carrying forward: the pre-flight panel reported free VRAM 15.10 GB, i.e. it read the device and not the per-process cap the run was actually held to, so the fit decision was taken against a number 3.8× larger than the budget in force. The run fit on its own merits (2.91 GB against 4.00 GB), so nothing was protected by luck — but on that hardware the pre-flight is not what would have caught an over-budget config, which is exactly the casetraining.stream_vram_override(#347, above) exists for. Record:benchmarks/run-t4-colab-free-tier.md.
[0.73.0] - 2026-08-09
The release that came out of three days on somebody else's hardware.
Every number this project had ever published was measured on one machine: an RTX 3050
Laptop, 4 GB, Windows. From 5–9 August it ran on a borrowed 8×H100 box (Ubuntu 24.04,
a much newer torch / bitsandbytes / trl / peft stack) for the first time. That found
one silent correctness defect in layer streaming, four backends that had never
actually run, and a documented multi-GPU entry point that had never launched — plus
the first evidence that the laptop result reproduces on hardware nothing like it. The
full record, published as written including six rejected hypotheses and three false
positives that controls caught, is
benchmarks/gate-h100-validation.md.
This is a minor bump, not a v0.72.x patch: it adds two capabilities that did not exist, and repairs four backends.
Added
training.seedandtraining.data_seed(#341). Every Soup run trained at seed 42 with no way to change it, so "run this twice with a different seed" was impossible. Both default toNonerather than to42on purpose: an unset seed has to reproduce two different historical defaults — HF's 42 forTrainingArgumentsand 0 for the multipack sampler since v0.37.0 — and a plain42default would have silently re-ordered every existing multipack run. Scope, stated rather than left as a footgun: wired into the SFT trainer only. Other task wrappers build their ownTrainingArguments, so setting it on a DPO run parses and does nothing — tracked as #353.- Full fine-tuning as
lora.r: 0(#340). The SFT trainer's full-FT branch was dead code with no way to reach it.r: 0was chosen on repo evidence, not taste: three consumers already treat rank 0 as "no adapter", andr: 0previously crashed inside PEFT, so no config that worked before changes meaning.lora.ralso gained a lower bound —r: -5used to parse and die inside PEFT. --deepspeed zero3_offload— ZeRO-3 with CPU parameter offload.zero3setoffload_param: noneand the only offload preset was stage 2, optimizer-only, so the configuration a user short of VRAM actually wants could not be named on the command line. Measured on one H100 (Llama-3.1-8B, bf16, LoRA r=8, 256 steps): 21.65 tok/s at a 38,135 MiB peak.offload_optimizerdeliberately staysnone— turning it on makes DeepSpeed JIT-buildcpu_adamagainst a matching CUDA toolkit and fail without one; copy the emitted JSON and flip it if you havenvcc.trlsupport widened to>=0.14.0,<0.29(#326), behind a capability-probe compat layer (trainer/_trl_compat.py) rather than a version table — a version table is what was wrong twice before. The trainers now ask each config class whether it acceptsmax_prompt_length, and resolveORPOConfig/CPOConfig/BCOConfigthroughtrl.experimentalwhen trl 0.29 drops them from the public namespace. All six preference trainers construct and train to identical losses on trl 0.26.2, 0.28.0 and 1.9.2.
Fixed — backends that had never been run
soup train --gpus Nnever launched at all (#77).accelerate launchtakes a script path positionally, and Soup handed itsys.executable— so accelerate opened the Python binary and parsed it as source (SyntaxError: source code cannot contain null bytes). Every rank died before the trainer existed. The documented multi-GPU entry point has been dead since it shipped, invisible to single-GPU CI because that path skips the launcher wrapper entirely. Separately,--no-reexecprinted a command with every user flag dropped, so following the hint literally trained without--fsdp.- DeepSpeed could not train a LoRA model on any stage (#336). HF builds two
optimizer parameter groups and with LoRA the no-decay group is empty; DeepSpeed
drops it, leaving one group against two
base_lrs, and torch's scheduler then hits a strict-ziplength mismatch. Verified repaired on 2×H100 with realsoup train: zero2 6790.2 tok/s, zero3 1025.3, zero++ 977.1, all exit 0 with a live adapter (96/96).zero++failed earlier and independently — it set fp16 quantised weights/gradients against a bf16 model and hardcodedzero_hpz_partition_size: 8regardless of the real world size; both are now derived, and the rewrite is printed rather than applied silently. use_fsdp2_compilewrote an adapter that reloads as all zeros (#335). Undertorch.compilethe Trainer saves through the wrapper, so every key came out as_orig_mod.base_model.model.... The tensors were genuinely trained (max|lora_B| 7.0e-3 measured) andPeftModel.from_pretrainedmatched none of them — emitting only aUserWarningand leavinglora_Bat its zero init. Measured on 4×H100: 0 of 96 non-zero against 96/96 for the paired non-compile run, reproduced 3/3, with the run exiting 0 throughout.soup serve --backend sglangreturned 500 on every generation (#76). sglang 0.5.16'sRuntime.generatereturns a JSON string; Soup subscripted it as a dict. Deterministic, not a race — the backend loaded cleanly and then failed 100% of requests. It had genuinely never been run, because SGLang does not support Windows.- The vLLM backend ignored the model's chat template (#332).
utils/vllm.pyhand-rolled a"User: ...\nAssistant:"prompt while the transformers backend usedapply_chat_template, so every vLLM user's model saw a format it was never trained on. On Llama-3.1-8B + LoRA, identical server and sampling params, only the prompt differing: a run-on loop burning all 200 tokens before, an 8-token answer after. Both backends now share onebuild_chat_prompt. - Three vLLM serving defects (#333), each verified live:
finish_reasonwas hardcoded"stop"even atcompletion_tokens == max_tokens;--dashboardsilently no-opped (/metricsreturned 404 with nothing printed); and--max-model-lendid not exist although the engine factory already accepted it.--dashboardon a backend that cannot serve it now warns at startup naming the backend instead of doing nothing.
Fixed — training paths that silently did the wrong thing
data.max_lengthwas capped at 1024 on every SFT run (#78).SFTTrainerconvertsTrainingArgumentswithSFTConfig(**args.to_dict()), andmax_lengthis an SFT-only field thatTrainingArgumentsdoes not carry — so it always tookSFTConfig's default. Measured before the fix:data.max_length=4096gave 1024 tokens per sample, with no warning.training.use_liger: truecrashed at step 0 (#78). Soup patched the model but never setTrainingArguments.use_liger_kernel, the flag TRL reads to know the fused path returnslogits=None; its entropy metric then ran onNone. Reproduced across the whole supported trl pin, so the feature did not run at all. Separately, Liger's architecture match was a substring of the model name, so any model loaded from a local directory trained without Liger on a flag the user had explicitly set — it now readsAutoConfig.model_type.- FlashAttention 3 was selected from a version that can never report 3 (#334).
Dao-AILab ships FA3 as
flash_attn_3;flash_attnitself stays in the 2.x line, so the branch could not fire for any real install — and had it fired, it produced anattn_implementationtransformers would reject. On Hopper hardware users silently got FA2 or SDPA while the docs advertised FA3. Both detectors now ask transformers. This makes detection honest; it does not make FA3 measurable here.
Fixed — layer streaming
- A silent wrong-gradient defect on large NF4 models (#331).
bitsandbytes.MatMul4Bitstashes the packed weight andquant_stateonctxas plain attributes instead of throughsave_for_backward, so gradient checkpointing cannot discard and recompute them. The reference is captured in the forward, aliases the streaming buffer pool, and is read in the backward after that slot has been refilled. Result: a bit-exact forward, a healthy-looking loss curve, and wrong gradients on every layer but the laststream_buffers. It bites NF4 above a threshold bracketed at 163.8–171.5 MiB per layer — so 32B and 72B, never 8B or 14B, and never bf16. The repair keeps the weight out of that function entirely: dequantise inside the checkpointed region and use a native matmul, which saves the dequantised weight through the ordinary mechanism. Gated against a resident NF4 reference with a repair-disabled control arm in the same process: real 32B 256/256 gradient tensors exact against the control's 8–12/256, at +2.9% peak VRAM and −4.8% throughput; and again on real 72B — the size where the defect was worst — 320/320 against 8/320, at +2.6% and −3.7%. De-aliasing was measured and rejected first: bnb holds the reference across the whole forward-to-backward span, so any copy is O(model), which took real 32B from 4,220 to 19,720 MiB and deletes the feature's premise. - All four preference losses died on newer trl (#328).
should_enable_hf_gradient_checkpointingexisted so the HF Trainer does not checkpoint an already-checkpointed streamed model twice — and onlysft.pyever called it. TRL's default isFalseon trl 0.19.1 andTrueon 0.26.2, so one omission had two opposite symptoms: an explicitgradient_checkpointing: truesilently dropped on the older stack, and on the newer one HF checkpointed the inner decoder layer, recomputed it after the reparametrisation context had exited, and killed dpo/orpo/simpo/kto withTensor on device cuda:0 is not on the expected device meta!. - Layer streaming is now refused when
nn.DataParallelwould engage. HF wraps the model in DataParallel whenever more than one CUDA device is visible and the run is not distributed, and DataParallel requires every parameter ondevice_ids[0]— streaming keeps the decoder onmetaby design. It accounted for 8 of the 9 streaming-suite failures on the H100 box and is unreachable on a one-GPU machine. It refuses rather than quietly using 1 of 8 cards. device_map="auto"broke every distributed launch, in fifteen places. transformers refusesdevice_map="auto"outright under a distributed launch, so the exactaccelerate launchcommandsoup train --gpus 8prints died on every rank. A first pass fixed six trainers; nine sites still carried it. All fifteen now go throughutils/gpu.resolve_device_map, and the regression guard scans every module insoup_cli/trainer/instead of a hand-written list — the list is what hid the nine.LOGITS_BYTES_PER_ELEMENTis split into two independently measured terms (#327). The 14 is 12 + 2, measured stage by stage on an H100 with zero spread across three repeats, and only the 2 differs between stacks. It is deliberately not lowered — see Known Limitations. What is new is an upward-only calibration (max(14, measured + 2)), which closes a real unguarded hole: a future stack that grew a fourth fp32 buffer would be under-budgeted by 12.5% today with nothing to catch it. Default behaviour is byte-identical and the pre-flight path takes no new CUDA. (Originally landed as an opt-in probe with no caller; #348 above wires it into the pre-flight itself, so it now runs on every streamed run rather than sitting inert.)
Fixed — eval, ship and export
- Two of
soup ship's three behavioural suites measured nothing (#316). On Meta-Llama-3.1-8B-Instruct,mini_tool_callscored 0.000 andmini_format_json0.000 on a model that does both correctly. All harness defects: the tool-call prompt never told the model tools exist, the JSON check ranjson.loadsover the whole output while 38 of 40 answers sit inside a```jsonfence, and the generation budget was takingmake_generator's default of 64, truncating 31 of 40 tool calls one closing brace short. - The refusal detector missed the apostrophe models actually type (#316). The
patterns spelled the contraction with U+0027; Llama-3.1 writes U+2019. Over the
shipped 40-item
mini_safetysuite, 28 of 40 refusals scored as non-refusals and the suite reported 0.300 for a model whose true refusal rate is 1.000 — a 0.70 error against a 0.05 regression threshold, i.e. 14× the thing it exists to detect. soup train --task online_dpopassed a keyword trl removed at 0.25 (#300). The probe asked whetherBasePairwiseJudgewas importable and used the answer to decide whether to passreward_model=; those two facts had decoupled, so the probe said yes for every trl in the supported range. It now asks the signature — through an MRO walk, because on 0.26.2 the top-level class is a deprecation shim whose direct signature reports no parameters at all.- A quantised GGUF export deleted a previously exported f16 (#144). The f16
intermediate was named
{model}.f16.ggufnext to the output — exactly the default output name of a--quant f16export — and unlinked when quantisation finished. So exporting q4_0 destroyed an earlier f16 export, even with an unrelated--output. Reproduced. It now lives in a private temp directory. Separately, the convert dependencies were installed only after an auto-clone, so--llama-cpp /pathand an already-present checkout both died onModuleNotFoundError: sentencepiece.
Fixed — packaging and docs
requires-pythonnow has an upper bound:>=3.10,<3.13(#358). CI tests 3.10, 3.11 and 3.12 and nothing above. Without a ceiling, pip on 3.13+ resolved torch wheels nobody here has run, and the failure was not a Soup error message — it was a loader crash insidec10.dll/libc10.sobefore any Soup code executed, leaving the user nothing to act on. The bound is 3.13, not 3.14: 3.13 is equally untested.tests/test_requires_python_bound.pyderives the bound from the CI matrix, so widening one without the other fails the suite.- The
trlbounds shipped in v0.72.4 were wrong at both ends, and were corrected before #326 widened them again. The ceiling was over-tight by five releases: the v0.72.4 table read atrl/experimental/relocation as a field removal. The floor>=0.7.0was impossible —setup()importsGRPOTrainer, first exported at 0.14.0. One detail in the v0.72.4 note was also imprecise:ORPOConfig/CPOConfigare not deleted at 0.29, they leave the public namespace — anImportErrorrather than a rejected keyword, which is worse, not milder. - The layer-streaming pre-flight panel was titled after a flag that does not exist
(#329). It read
soup train --stream-layers; there is no such option — streaming is enabled bytraining.stream_layersinsoup.yaml. It is the first thing a streaming run prints, so it was the feature's most-read line of documentation, and it pointed at aNo such option. - Seven of the eight configs in
examples/configs/did not parse. They were written against a pre-nesting schema, sosoup train --config examples/configs/sft_basic.yaml— the first commandexamples/README.mdtells you to run — failed validation. Also fixed while there: two configs declaredformat: sharegptfor preference-shaped data (a silently wrong training run, not an error),target_moduleslistedout_projwhich no Llama has, andvision_llama.yamlpointed at files that have never existed in this repo.tests/test_examples_configs.pynow parses every one of them. soup data demometadata described files other than the ones it ships.sharegpt_demodeclaredformat: sharegptfor prompt/chosen/rejected rows — the value a user copies straight intodata.format— andgrpo_demodeclaredreasoning, which is not in the schema's format literal at all.- Encoding corruption in
pyproject.toml(fourteen em-dashes round-tripped through cp1251, one of them in theunitpytest marker description thatpytest --markersprints), anddocs/commands.md, where missing newlines collapsed five commands onto two lines and the page promised "the full command list" while omitting seven.
Changed — documentation corrected against measurement
- LISA delivers the quality half of its claim, not the memory half (#306). Measured at 3B and 8B on an H100 with LISA engagement verified three independent ways: it beats full fine-tuning at both learning rates, and it is 1.22× LoRA's VRAM at 3B and 1.51× at 8B, with the gap widening with scale — embeddings, LM head and final norm stay trainable every interval and are 70.7% of everything LISA trains at 8B. The docs now say that plainly, and say when LISA is still the right choice.
- FlashAttention and Liger were measured for the first time, at 1.015× and 1.051× / −12.9% VRAM, against documented claims of "2–4×" and "20–60% / 20–40%". Both claims are corrected in the docs.
- The GEMM-ceiling test could not pass on a datacenter GPU. Its plausibility bound
was
< 200 TFLOPS, written when the only hardware this project had was an RTX 3050; an H100 returns a correct 786.5 TFLOPS and the assertion fired.
Validation (measured, not changed)
- The laptop result reproduces on completely different hardware. Llama-3.1-8B NF4:
119.6 tok/s in a 3.32 GB peak on the RTX 3050 against a median 113.00 tok/s in the
same 3.32 GB on an H100 — first outside evidence that the method is bound by
host-to-device transfer, not by the GPU. (See Known Limitations for what the laptop
figure does and does not now mean.)
Correction, 2026-08-13, left in place rather than rewritten. The measurement stands; the explanation does not. "Bound by host-to-device transfer" was an inference, never a measurement, and a probe on 2026-08-11 refuted it at the published configuration — deleting every host-to-device byte buys 1.4% and the step runs at 71.3% of the card's same-session GEMM ceiling (
benchmarks/probe-v0.73.0-what-bounds-streaming.md). What survives is the weaker claim: the constraint is common to both machines and is not the compute the H100 adds. - Forward bit-exactness at real model sizes, not just on toys: logits
torch.equalagainst a resident reference of matching numerics at 0.5B, 8B, 14B, 32B and 72B. Every previously published bit-exactness result was on 3-layer from-config checkpoints, because a 4 GB card cannot hold a resident 8B to compare against. Backward exactness is a separate claim measured separately — see #331 above and the per-model ledger at the top of the gate record. - A streamed model is as good as a resident one. Paired over five disjoint training
subsets and judged by Soup's own
soup ship: mean difference +0.006 against an identical 0.013 within-arm spread, in bf16; +0.0053 against spreads of 0.0333 and 0.0200 in NF4. This had never been measured anywhere in the project. - Against DeepSpeed ZeRO-3 CPU offload, same box, same data, same model: 2.93× the throughput at 9.7× less peak VRAM at matched numerics. The honest reading is narrow, and the same session establishes it: eight cards of ZeRO-3 are slower than one card training resident for a model that fits. Layer streaming is not "faster than DeepSpeed" — it is for the case where the one card you have is too small.
Known limitations
- The
training.seedknob reaches the SFT trainer only (#353). Every other task builds its ownTrainingArguments; setting it there parses and does nothing. - A resident 4-bit run is still not bit-reproducible from a seed (#354), while a
streamed one is.
get_peft_modelbuildslora_AbeforeTrainer.__init__callsset_seed, so the adapter init escapes the seed. Diagnosed with three competing hypotheses each killed by a control; the one-line fix is verified in the gate record but not shipped here. It has a real consequence forsoup ship: three runs of one unchanged resident config movedmini_common_senseby 0.375 andmini_mmluby 0.269 against aforgetting_thresholdof 0.05, so five of seven suites can cross the regression line on a re-run that changed nothing. - The layer-streaming VRAM pre-flight still over-predicts (#327), and
LOGITS_BYTES_PER_ELEMENTwas deliberately left at 14 rather than lowered to the measured loss-path value. The asymmetry decides it: over-predicting refuses a config that would have worked — visible, annoying, data intact. Under-predicting on Linux is a clean OOM, but on Windows/WDDM there is no exception at all — it silently spills to host memory (measured: 9.27 GB allocated on a 4.29 GB card with nothing raised) and the claim "peak is bounded by one layer" quietly stops being true. The counterfactual settles it: 14 under-predicts 0 of 10 measured rows (worst +0.85% over), the lower value under-predicts 10 of 10 (worst −10.49%). The practical cost is that a streamed DPO run is refused frommax_length: 768on a 4 GB card. bitsandbytesstill has the defect #331 works around. Soup no longer sends streamed NF4 weights throughMatMul4Bit, but the underlying library behaviour — saving tensors outsidesave_for_backwardwhere checkpointing cannot see them — is upstream and unchanged. Filed as bitsandbytes-foundation/bitsandbytes#2034.- DeepSpeed + LoRA is repaired in
sft.pyonly (#336). The other trainer wrappers still hit the empty-parameter-group failure under DeepSpeed, a user-supplied--deepspeed my.jsonis passed through unresolved, and ZeRO++ hierarchical partitioning is unit-tested but never exercised across two nodes. utils/sglang.pystill has both of the defects the vLLM rewrite fixed — its own hand-rolled prompt and a hardcodedfinish_reason. Named rather than silently skipped.- The closed-loop reward-hacking controller's mechanism is confirmed; its efficacy is not (#286). At 7B the between-mode difference of 0.130 sits inside the within-mode spread of 0.140–0.195. Settling it needs ≥5 seeds per mode.
- Two
soup shipleg-2 suites have scoring gaps that outrank capability (#346, #357).mini_tool_callranks by brace hygiene — Llama-3.1-8B names the right tool 40/40 and scores 0.225 — andmini_mmluloses 8 of 26 items becauseextract_mcq_letterdoes not know\boxed{C}, scoring the 8B at 0.423, below a 0.5B. Adding that one form takes it to 0.731 and the inversion disappears. A third suspected inversion (#356) was withdrawn as a measurement error of our own — it was measured at a 64-token budget wheresoup shipuses 256. - The 70B FSDP2 recipe could not be smoke-tested (#41): it needs all eight cards, so it could not be parallelised with anything else in the session.
- The RAM-vs-disk streaming throughput gap remains unmeasured (#325), and layer streaming remains BETA.
A note on the preprint
DOI 10.5281/zenodo.21771064 is unaffected in
its correctness claims: its configuration is 8B NF4 at 105 MiB per layer, comfortably
below the 163.8–171.5 MiB boundary of #331, and it survives a 50-backward soak at
worst_abs = 0.0. What this release changes is scope, not validity — exactness moves
from 3-layer from-config toys to resident references at 8B / 14B / 32B / 72B. A version 2
of the record carrying the H100 validation, the resident references, the DeepSpeed
comparison and the disclosed defect is in preparation.
One number should be read with a caveat: the published 119.6 tok/s laptop figure was measured before the #331 repair and has not been re-run on repaired code. The repair cost −4.8% throughput at 32B, so treat the laptop figure as a pre-repair number until someone re-measures it on an RTX 3050. An H100 cannot substitute — the whole point of the H100 result is that this method is transfer-bound, so its throughput does not carry across machines.
Correction, 2026-08-13. The conclusion holds, the reason given for it does not: the laptop is not transfer-bound (see the correction above). An H100 still cannot substitute, for a narrower reason — the repair's cost is paid in the per-layer NF4 dequantisation, measured at 9.8% of the step on the laptop, and that share belongs to that card's clock, GEMM ceiling and launch overhead.
[0.72.4] - 2026-08-03
Added — preference losses over layer streaming: DPO, ORPO, SimPO and KTO.
Layer streaming kept the frozen base in CPU RAM and fed it to the GPU one decoder
layer at a time, but only for task: sft. This release opens it to the four
preference losses. The whole risk was one thing: DPO needs a reference model, and a
second model instance would double memory and defeat the feature entirely.
- The reference is the same streamed base with its adapters disabled — one set of weights, one stream, no second pass. Measured on an RTX 3050 4 GB with a 730 MB model: streamed DPO peaked at 0.914x the SFT peak, with a byte-identical RAM store and buffer pool. Forcing a real second instance in the same harness cost +730.44 MB against 730.44 MB of weights — exactly one copy. That control is what makes the first number mean something.
- KTO is not reference-free, contrary to how it is usually described: it selects its reference exactly the way DPO does, so it gets the same treatment and the same memory assertion. ORPO and SimPO genuinely are reference-free.
- Bit-exact against a resident run of the same loss —
0.0difference for all four, the standard every slot in this series inherits. - The pre-flight now knows that a paired loss is twice the rows. DPO, ORPO and SimPO concatenate chosen and rejected into one tensor, so a VRAM budget computed at one row per example would have under-predicted by half — and on Windows the consequence is not an error but a silent spill to host memory that makes the run an order of magnitude slower.
- KTO requires
batch_size >= 2(its KL term is degenerate at 1). Soup now says so when your config is read, rather than minutes later after sharding the checkpoint. KTO is streamable at all only because v0.72.3 lifted the batch-1 restriction. grpoandpporemain excluded permanently, not "not yet": generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming depends on. The refusal says so and deliberately names no release.
The streaming setup now lives in one shared place instead of being copied per trainer, so the NF4 pre-flight, the RAM/disk tier choice and the VRAM fit refusal cannot drift between SFT and the preference losses.
Fixed — trl is now capped, and that is a real bug fix, not a CI tweak.
Six trainers (bco, dpo, ipo, kto, orpo, simpo) pass max_prompt_length to
their trl config, and trl removed it in stages. So anyone who ran
pip install 'soup-cli[train]' and resolved to a recent trl had
soup train --task orpo fail on import.
Correction (see [0.73.0]). This release shipped the cap as
<0.25on the strength of a staged-removal table that was itself wrong. The real stages arektoat 0.27,bco/orpo/simpoat 0.28 anddpo/ipoat 0.29. v0.73.0 first corrected the cap to<0.27and then raised it to<0.29behind a capability-probe compat layer, so the trainers no longer set the cap at all.
That was already true before this release and nothing caught it: the trl imports
live inside setup(), which no test had ever called on those wrappers, so CI stayed
green while the code only worked on older trl. This release's end-to-end preference
tests are what surfaced it.
The boundary was read off the published wheels per config rather than inferred from a version number — the removal being staged is exactly why a single spot-check gives the wrong answer, and see the correction above for how that method can still land on the wrong answer. Supporting the newer API is its own piece of work; declaring a dependency the code actually works with comes first.
Honest costs: streaming makes the reference free in memory, not in time — DPO traverses the layer stack three times per step against SFT's two, measured at 1.52x the layer reads. And the VRAM pre-flight is a sound upper bound for preference losses rather than a tight estimate; see Known Limitations in the release notes.
[0.72.3] - 2026-07-28
Added — layer streaming breadth: more architectures, bigger batches, resume, and a disk tier.
Layer streaming (v0.72.0–.2) was deliberately narrow: Llama/Qwen only, batch 1, no gradient accumulation, no resume, RAM only. This release lifts all of it, and each capability was gated against a streamed-vs-resident bit-exactness reference before any of it was written.
- Six more model families.
mistral,gemma,gemma2,gemma3_text,phiandphi3join the allowlist, each verified bit-exact against the same checkpoint loaded resident, under both bf16 and NF4. Phi-3 is the notable one: it fuses Q/K/V into a singleqkv_proj, so there is noq_projto find, and it is bit-exact anyway. Multimodalgemma3is deliberately not accepted — onlygemma3_text. batch_sizeabove 1, and gradient accumulation. Both previously refused.- A pre-flight VRAM budget that accounts for batch and vocabulary. Streaming bounds
the weights; activations and the logits tensor are untouched by it and both scale
with
batch × seq. On a 152k-vocab model at batch 8 the logits term alone measured 8.71 GB — 146× the entire layer-buffer pool.soup trainnow predicts peak VRAM and refuses a run that will not fit, naming the two knobs that scale it. - A throughput forecast, quoted as a range from a GEMM ceiling measured on your own card in that session, alongside the SM clock — never a compiled-in per-card constant.
--resumeand--hf-resumework with streaming.- A disk overflow tier.
stream_source: auto(the default) uses RAM when the base fits and falls back to an NVMe disk tier when it does not, holding nothing resident.stream_source: ramrefuses instead of falling back. Non-NVMe disks are still refused outright. soup doctor --diskreports the detected media type.
Fixed
estimate_logits_bytescharged 6 bytes per logit element; the measured peak is 14 (transformersholds the bf16 logits, the fp32 upcast, log-softmax's fp32 output and the fp32 gradient live at once). The old figure under-predicted that term by 2.33×.- Adapters could not be loaded into a streamed model:
load_state_dictnarrows keys by child name, so a canonical checkpoint matched 0 of N tensors and PEFT reported only a warning — a resumed run reproduced the from-scratch loss curve exactly. The streaming layer now redirects canonical keys at load time, mirroring the v0.72.1 save-side fix. - The NVMe-only tier guard was wired to a hardcoded constant and could never fire.
- Streaming weight sources are now released when training ends or raises; the disk tier holds one open shard handle per decoder layer.
- Subprocess helpers resolve tools to absolute paths (on Windows,
CreateProcesssearches the current directory beforePATH). - The
[mcp]extra is now capped atmcp<2. The SDK's 2.0.0 release removedmcp.shared.memory.create_connected_server_and_client_sessionand droppedServer.list_tools, breakingsoup mcp serve's round-trip tests for anyone installing fresh. Support for the 2.x API is tracked separately.
Known limitations
- The RAM-vs-disk performance gap is unmeasured on the development hardware and no number is claimed for it. safetensors memory-maps the shards, so the OS page cache keeps them resident between steps on a machine with spare RAM, and at ~5 effective TFLOPS the NVMe read hides under compute. The disk tier's correctness is verified bit-exact against the RAM tier; its speed relative to RAM is not characterised.
- End-to-end
soup train --resumecould not be demonstrated on the development box:transformersrefusestorch.loadbelow torch 2.6 (CVE-2025-32434), which blocks every resume there, streaming or not. The streaming-specific half — the adapter round-trip and loss continuity — is verified on the production CUDA path. - Loading into a streamed model works;
named_parameters()andstate_dict()still disagree in memory, which is the deliberate cost of a serialisation-only design. - Layer streaming remains BETA.
[0.72.2] - 2026-07-28
Added — NF4 layer streaming: fine-tune Llama-3.1-8B on a 4 GB laptop GPU.
Layer streaming (v0.72.0) keeps the frozen base in CPU RAM and streams it to the GPU one decoder layer at a time, so peak VRAM is bounded by one layer instead of the whole model. It was bf16-only, which capped it at about 3B on a small card. Quantising the streamed base to NF4 shrinks it ~4×, and that is what brings 8B within reach.
Add one line to a streaming config:
training:
stream_layers: true
quantization: 4bit # NF4
batch_size: 1
Measured on a 4 GB RTX 3050 Laptop (Windows, batch 1, S=512, gradient
checkpointing, 50 steps after 10 warm-up, PagedAdamW8bit):
| Model | tok/s | Peak VRAM | RAM store | GPU util |
|---|---|---|---|---|
| Llama-3.1-8B-Instruct | 119.6 | 3.32 GB | 3.60 GB (page-locked) | 100% |
| Qwen2.5-3B | 264.2 | 1.76 GB | 1.43 GB (page-locked) | 100% |
For scale: 1M training tokens is about 2.3 h at 8B on that card (arithmetic from the measured rate, not a separate measurement).
Qwen2.5-3B also got 1.85× faster than the bf16 streaming path (264.2 vs 143.1 tok/s) — and the reason is not arithmetic. A 1.43 GB store fits under the machine's page-locked memory ceiling where a 5.55 GB one did not, which restores asynchronous host-to-device copies and lifts GPU utilisation from 79.3% to 100%.
Correctness. A streamed NF4 run is bit-exact against a resident NF4 run: the same quantised bytes through the same bitsandbytes kernels. Logit equality, non-zero gradients at layer 0, and a matching multi-step loss curve are all regression tests, not one-off measurements.
The base is quantised once, offline, and cached under ~/.soup/layer-stream/.
The cache is keyed to the quantisation and the source checkpoint, so switching
between none and 4bit, or retraining a base in place, re-shards instead of
silently streaming the wrong bytes.
Fixed — a streamed 4-bit run reported its parameter count ~6.5× too high. SmolLM2-135M printed "878,154,048 total" (true: 134,515,008). Display only — training was unaffected — but at 8B it would have read ~52 B.
Scope is unchanged and still BETA: RAM tier, task: sft, Llama/Qwen,
batch_size: 1, no gradient accumulation, no --resume. Every rejected config
names the release that lifts it. quantization values other than none and
4bit are refused.
Fixed — soup --help was 5x slower than it should be. Since v0.72.0 the CLI
imported PyTorch on startup, taking 6.0 s where it now takes 1.15 s.
soup reward stress (v0.71.41) put utils/reward_stress on the light CLI path.
That module imported utils/reward_hack_control to reuse a single string
constant — and reward_hack_control resolves its TrainerCallback base class at
module scope, which pulls in transformers and torch. Importing a ~4.4 s
dependency for one constant made every soup invocation pay for the training
stack, including commands that never touch a model.
Nothing produced wrong results; this was purely startup latency. The light core
(pip install soup-cli without the [train] extra) was never broken — it fell
back cleanly when torch was absent, just slowly when it was present.
Added tests/test_cli_startup_is_light.py, which asserts the invariant at
runtime (import soup_cli.cli must not put torch in sys.modules) instead of
inspecting source text for import torch, which is what the previous guards did
and why this went unnoticed.
[0.72.1] - 2026-07-27
Fixed — layer-streaming adapters were saved in an unloadable form. If you
trained with stream_layers: true on v0.72.0, the adapter that run wrote is
inert: every tensor was saved under a key containing an extra .inner.
segment, so soup merge, soup serve, soup chat and
PeftModel.from_pretrained all loaded zero adapter tensors and silently
returned the untuned base model. PEFT emitted only a UserWarning, so nothing
failed and nothing looked wrong.
The training itself was correct — the streamed run's numerics are unaffected, and v0.72.0's bit-exactness results still stand. Only the saved file was affected.
If you have a v0.72.0 streamed adapter: re-save or re-run it on v0.72.1.
There is no way to recover the original file's association with the base model
beyond renaming its keys; re-running is the reliable path. A quick check —
if adapter_model.safetensors contains keys with .inner. in them, it is
affected:
python -c "from safetensors.torch import load_file; \
print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"
Streamed adapters now save byte-for-byte in the same layout as an ordinary LoRA run, and are portable to any tool that has never heard of layer streaming.
Also fixed — --hf-resume bypassed the streaming resume refusal. The guard
only tested --resume, but --hf-resume reaches resume_from through a
different branch. That combination previously appeared to work by accident
(checkpoint and live model shared the same key shape); once adapters are saved
canonically it would instead have matched nothing and continued with a
freshly initialised adapter, silently. Both flags are now refused for streaming
runs, naming v0.72.3.
Also in this release: every "this lands in vX.Y.Z" refusal message was corrected after the v0.72.x roadmap was renumbered (NF4 streaming is now v0.72.2; the disk tier, wider architectures, larger batches, gradient accumulation and checkpoint/resume are v0.72.3; preference losses are v0.72.4).
Known limitation: in memory the streamed model's named_parameters() still
carries the wrapper segment, so loading into a streaming run (--resume)
remains unsupported and is refused with a message naming v0.72.3.
[0.72.0] - 2026-07-26
Superseded by v0.72.1 — adapters saved by this version load as zero tensors. The entry below is left as published; the defect and the fix are described under [0.72.1]. Version numbers named as "upcoming" below were also renumbered there (NF4 is v0.72.2, not v0.72.1).
Layer streaming (BETA) — fine-tune models that don't fit in your card. The frozen base lives in CPU RAM and is streamed into two pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is bounded by the size of one layer instead of the whole model. Only the LoRA adapters, their gradients and optimizer state stay resident. Slower than resident training — but these models did not run on the card at all.
Measured on the development box (RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM), batch 1, gradient checkpointing on, 50 steps after 10 warm-up:
| Model | S | tok/s | GPU util | Peak VRAM |
|---|---|---|---|---|
| Qwen2.5-0.5B | 512 | 978.6 | 91.4% | 1.47 GB |
| Qwen2.5-1.5B | 512 | 525.0 | 96.8% | 1.82 GB |
| Qwen2.5-1.5B | 1024 | 487.6 | 96.7% | 2.96 GB |
| Qwen2.5-3B | 512 | 143.1 | 79.3% | 2.15 GB |
Qwen2.5-3B trains in 2.15 GB on a 4 GB card where a resident run OOMs. The honest cost: 1.43× slower than resident, measured at 0.5B — the only apples-to-apples comparison available on this box, because 1.5B and above cannot run resident here at all.
Added
training.stream_layers: true— stream the frozen base layer-by-layer from CPU RAM.training.stream_source(auto/ram/disk) andtraining.stream_buffers(2–8, default 2 = double buffering) tune it.soup_cli/utils/layer_stream.py— tier choice, pinned-vs-pageable decision, architecture allowlist, VRAM/throughput arithmetic (no torch import).soup_cli/utils/layer_shard.py— rewrites an HF checkpoint into one safetensors shard per decoder layer, one tensor at a time, so sharding a model that does not fit never needs it to fit.soup_cli/utils/layer_stream_runtime.py— buffer pool, CPU-RAM weight source, prefetch scheduler on a dedicated CUDA stream, and the streamed layer wrapper.- Shards are cached under
~/.soup/layer-stream/(override withSOUP_LAYER_STREAM_CACHE_DIR) and keyed to the source checkpoint's fingerprint, so a base retrained in place re-shards instead of silently training against stale weights. - When the base cannot be page-locked, the RAM store falls back to pageable memory and says so, including the measured cost (GPU utilisation ~97% → ~79%).
Changed
- The pre-flight hardware-fit gate is skipped for streaming runs: it models a resident run and would otherwise refuse exactly the runs streaming enables.
- Gradient checkpointing is handled per-layer by the streamer; the HF Trainer's own is left off so layers are not recomputed twice.
Known limitations
- BETA, and proof-of-mechanism at 3B. Nothing above 3B was measured. No 8B/14B claim is supported.
- Scope: RAM tier, bf16,
task: sft, Llama/Qwen, batch size 1, no gradient accumulation, no--resume. Every refusal names the release that lifts it. - 4-bit (NF4) streaming is v0.72.1 — NF4 weights carry a quantisation state and cannot be byte-copied into a plain buffer.
- The disk overflow tier, larger batches, gradient accumulation and checkpoint/resume are v0.72.2.
- The 3B number used a pageable store (this box cannot page-lock 5.55 GB), so it is a lower bound.
expandable_segments:Trueis silently ignored on Windows; Soup detects this and does not claim it is active.- Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.
[0.71.41] - 2026-07-19
soup reward stress: is your reward verifier gameable? Turn the reward-hacking
detector on the verifier itself. soup reward synth (v0.71.40) proves a verifier
separates your references from friendly perturbations; stress asks the adversarial
question a reward-hacking model asks at train time — does the verifier pay out for
degenerate junk? It feeds empty, length-padded, repetition, and sentinel-spam
completions and flags any the verifier accepts. Pure, offline, exit 0 = robust /
2 = gameable / 1 = error. Nothing in TRL / Unsloth / Axolotl / OpenRLHF tests a
verifier for gameability.
Added
soup reward stress <reward.py|builtin> [--references golds.jsonl]— adversarial verifier probe. Attacks (--attacks empty,length,repetition,sentinel,--sentinel) are scored against the real gold, so numeric / tool_call / json_schema verifiers get a valid target and still must reject the junk. Reports a per-attack accept-rate table + an overall gameability verdict (--max-gameable,--threshold,--output-report). Loads the target through the existing reward loader, so it probes a synthesized.pyand a builtin (accuracy/format/verifiable). A gold-requiring verifier probed with no--referencesis a hard error, never a false "robust".
Fixed
- Corrected the Telemetry section in the ops docs: Soup's telemetry primitives exist but are not wired to any command — no data is ever sent today (the previous wording implied a live opt-in sender). Wiring is deferred until a public privacy policy ships.
[0.71.40] - 2026-07-19
soup reward synth: auto-generate a deterministic reward verifier from your data.
Point it at a JSONL of reference (gold) outputs and it infers a verifier, emits a
readable / committable .py reward function, and — the moat — refuses to emit one
that can't tell your references from auto-generated bad answers. Nothing in
TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward; every reward today is
hand-written, hand-picked, or a trained-weights artifact.
Added
soup reward synth <references.jsonl> -o reward.py— deterministic verifier synthesis. Four families, auto-detected (or pick with--kind):numeric(last-number /\boxed{}/####extraction, exact or--tolerance),json_schema(induced keys + types + required),regex(positional char-classes over equal-length golds),tool_call(per-toolrequired/allowedargument binding). The emitted file is self-contained and ridesload_reward_fn's existing.pypath — no new trusted-exec surface; you read, edit, commit, and diff it.- Mandatory calibration report — the synthesized verifier is loaded back and run
against its own references (must accept ≥90%) and auto-perturbed negatives (must
reject). A degenerate always-accept verifier is refused (
exit 2), never silently emitted.--plan-onlyreports the induced spec without writing;--output-reportpersists the calibration JSON. - Comma-separated
reward_fn("accuracy,format") now trains — it resolves to a reward ensemble (GRPOTrainer(reward_funcs=[...]), and unlocks therm_ensemblereward-hack detector which needs ≥2 rewards). GRPO-only, validated at config-parse time. Fixes a recipe (deepseek-v3-reasoning) that shipped exactly this and previously crashed withUnknown reward function(#311).
Changed / Fixed
training.reward_fngains a field validator (null-byte / blank / oversize / empty-comma-segment rejection) — the oldest arbitrary-code field was the least guarded. Comma +verifiablewithout averifiable_domainnow fails at parse time like the bareverifiableform.envs/calculator.py/envs/guess_number.pydocstrings corrected: the reward isreward_fn: verifiable+verifiable_domain: math(the barereward_fn='math'they showed was never valid).
[0.71.39] - 2026-07-19
"CI for weights, not prompts": close the evidence loop. soup ship's verdict is
now something Soup can emit, commit, review, and bind to the exact model that
produced it — turning the soup ci init gate from "edit two numbers in a JSON file"
into a reproducible, provenance-bound check that renders on every PR.
Added
soup ship --emit-evidence <path>— re-serialises the verdict into the--evidenceINPUT schema, so a run's output is replayable as input: feeding it back through--evidence(same--forgetting-threshold) reproduces an identical verdict. Output is finally input.ShipConfigundereval.shipinsoup.yaml+soup ship --config soup.yaml— commit the gate policy (task_eval/task_mode/general_suite/forgetting_threshold/judge_model/baseline) so the verdict is reviewable in a PR diff and reproducible. An explicit CLI flag always wins (CLI > config > default).soup ship --push owner/repo#N— post the verdict as a GitHub PR comment (reuses thesoup adapters pr --pushgh apiplumbing). Best-effort: a missing token /ghfailure warns but never flips the SHIP / DON'T-SHIP exit code.- Evidence provenance + staleness gate. With
--emit-evidence,--configSTAMPS aprovenanceblock (config_sha— a semantic, order-insensitive recipe hash — plusbase_modeland a best-effortdata_sha) onto the evidence. With--evidencealone,--configGATES: it refuses (exit 3) evidence whoseconfig_shadrifted from the committed config, so a PR that changedsoup.yamlbut forgot to recompute its evidence is caught. The gate policy (eval.ship) is EXCLUDED from the hash, so tuningforgetting_thresholdnever falsely invalidates evidence about an unchanged model. soup ci init --config <soup.yaml>— binds the generated gate'ssoup shipstep to the committed config (provenance/staleness enforcement in CI).
Changed
- Exit-code note:
soup ship --configusage / staleness errors are exit3(usage), preserving0 = SHIP,2 = DON'T SHIP,1 = runtimefrom v0.71.38.
Security
provenance.config_sharead from untrusted evidence is shape-validated as a hex digest before being echoed (a raw value could smuggle terminal ESC bytes pastrich.markup.escape).provenance.data_shahashes the training file through anO_NOFOLLOWfd with a symlink-rejecting containment check and an 8 GiB cap (was an unguardedhash_file).soup ci init's path validation now rejects#and the YAML 1.1 line breaks (NEL / LS / PS), closing a plain-scalar comment-truncation of the generatedrun:step (also hardens the pre-existing--data/--suite/--evidenceargs).
[0.71.38] - 2026-07-17
soup ship's regression leg now has teeth. Leg 2 (the catastrophic-forgetting
/ regression gate that carries the whole SHIP / DON'T-SHIP claim) was 15
hand-written trivia prompts scored by case-insensitive substring containment
— it scored "B" for "Berlin", "ok" for "look", "3" for "13", and
had zero items for tool-calling, safety, or JSON validity. This release makes
the gate real: a fixed, extraction-based scorer + bundled, offline, zero-dep eval
suites that catch a regression the old gate waved through.
Changed
- Fixed answer scorer (breaking — verdicts can change).
soup ship's leg-2 MCQ / instruction / arithmetic answers are now scored by answer-extraction- a boundary-aware match, replacing the raw substring test. A spurious substring inside another word ("Berlin", "look", "13") no longer scores a correct answer, so an existing run's verdict may flip — intentionally, because the old gate was reporting false negatives.
- Bundled, offline general suite. The default
--general-suiteis now seven hand-authored suites shipped in the wheel:mini_mmlu/mini_common_sense/mini_instruction(expanded), a newmini_arithmetic, and three behavioural suites the old gate had no coverage for —mini_tool_call(function-calling),mini_format_json(JSON validity), andmini_safety(refusal-rate). Each is scored to a per-model absolute score by the pure scorers Soup already ships (eval/custom,utils/diagnose); no lm-eval, no network, no download. Every suite is large enough that a single-item flip (1/N < 0.05) trips the default threshold instead of being rounded away. soup shipexit codes: usage errors moved 2 → 3. Exit2now means only DON'T-SHIP; a typo'd flag or bad--general-suiteexits3(mirroringsoup plan/soup env check), so CI can tell a config error from a caught regression. Offline--evidenceread/parse errors stay1. Breaking for anyone parsing exit2as "usage error".
Fixed
soup shiphelp + docstrings no longer describe--task-mode pairwiseas "reserved for a later release" (it shipped in v0.71.31); the deadSUPPORTED_TASK_MODESgate is removed.soup diagnose's package docstring said "Six" probes (there are seven — citation) and pointed live loading at an unshipped version; it now re-exports all sevenscore_*probe functions so callers need not reach into submodules.
[0.71.37] - 2026-07-17
Every pip install soup-cli[extra] command now works on Windows cmd.exe, and
eval-gate benchmark tasks run instead of always failing.
Fixed
Install hints are now quoted so they work in every shell. Soup printed
pip install 'soup-cli[ui]'— bash / zsh / PowerShell syntax.cmd.exehas no single-quote quoting, so it hands the quotes to pip verbatim and pip refuses:ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifierEvery hint, README command, and docs example now uses
pip install "soup-cli[extra]", which works in cmd, PowerShell, bash, and zsh alike — the same spelling the repo already used forpip install -e ".[dev]". Measured on Windows: single quotes fail only oncmd.exe; double quotes pass everywhere; dropping the quotes passes on Windows but breaks zsh, which globs the bracket.Nothing in Soup can rescue the command after it is typed — pip and the shell own it, and Soup is not installed yet when the README command runs — so the fix is the spelling we print. A regression test now scans the package and every docs code block for the single-quoted form.
If you followed an older tutorial and hit
Invalid requirement, swap the'for"; nothing is wrong with the package.Eval-gate
type: benchmarktasks now actually run.eval/gate.pyprobed for aforgetting.run_mini_benchmarkhelper that never existed, so everytype: benchmarktask in a gate suite failed 100% of the time — while advising an[eval]extras install that could not fix it. The gate now callsForgettingDetectordirectly (the same waysoup shipalready did), and an unknown benchmark name fails with the list of valid names. Thanks @Sanjays2402! (#315, closes #310)
[0.71.36] - 2026-07-16
Data Moat II — a semantic layer over your training data, plus two tools for what a fine-tune forgets and leaks.
Added
soup data dedup --semantic— near-duplicate removal over embedding cosine instead of MinHash shingling. Catches reworded duplicates that MinHash misses (measured: 0.88–0.91 cosine on rewordings MinHash scored as distinct) while correctly keeping distinct-but-similar instructions. Zero new dependencies: usestransformersfrom the[train]extra. Read the known-limitation below before lowering--threshold.soup data topics <data>— cluster a dataset and label each cluster with c-TF-IDF terms, plus a coverage table (82% code · 6% math) and a warning for thin topics. Labels are emergent term clusters, not a fixed taxonomy.soup data canary insert|check— Secret-Sharer memorization probe. Insert K high-entropy secrets, then check any model/adapter: each secret's loss is ranked against never-inserted controls drawn from the same space. Exit 2 on MAJOR so CI can gate. Measured on SmolLM2-135M: a memorized set lands at percentile 0.0 (loss 1.7–2.5) against a clean model's 4.1–6.2.soup train --replay old.jsonl --replay-ratio 0.1— continual-learning rehearsal. Interleaves a seeded sample of an old dataset into training so a new task does not erase the old one.ris the fraction of the final mixed set (n_replay = round(r/(1-r) · n_new)), rows are interleaved rather than appended, and an undersized pool reports the shortfall instead of repeating rows. Mixed intotrainonly — validation stays pure new-task.
Fixed
- The hardware-fit gate refused to train any local checkpoint. A merged
model (
soup merge -o ./mymodel) has no size marker in its name, so the size guesser returned its 7B default, predicted ~16 GB of VRAM and refused. Local checkpoints are now measured from their safetensors header (0.135B actual vs 7.0B guessed) — this had blockedsoup merge→ train-from-merged entirely. Third instance of this class after the v0.71.32 (Whisper) and v0.71.33 (Msuffix) fixes. pip install 'soup-cli[extra]'hints printed without the extra. Rich ate the bracket, so every "install the missing dependency" message across 17 sites told users to runpip install 'soup-cli'— which succeeds and still leaves the feature broken. Affected[eval],[data],[serve],[ui],[tui],[compile],[mcp],[carbon]and others, including Typer help text.- Replay rows bypassed the image/audio path-traversal validation that the primary dataset receives.
Known limitations
- Semantic dedup is not a paraphrase detector. Measured with
all-MiniLM-L6-v2, paraphrase cosines (0.49–0.76) overlap with
genuinely-distinct rows (0.54–0.76): "Add two numbers" vs "Multiply two
numbers" scores 0.759, higher than the true paraphrase "reverse a string" /
"invert the order of characters" at 0.491. No threshold separates them, so
lowering
--thresholdto chase paraphrases deletes real training rows. The default (0.8) is deliberately conservative. - Replay is validated at proof-of-mechanism scale. On SmolLM2-135M + LoRA, replay retained the old task 7% better than a no-replay control — the correct direction — but forgetting without it was only +4%, i.e. mild. The effect size at full fine-tuning or 7B+ is unproven on a 4 GB box.
- Canary exposure is the sampled-control approximation, not full-space rank enumeration. "No exposure" is not proof of no memorization.
data topics/dedup --semanticrequire[train](torch) and download an embedding model. Plain MinHashdedupstays on the light core.- Replay v1 is
sft/pretrainonly and is incompatible withpacking/multipack.
[0.71.35] - 2026-07-15
Added
- Compliance templates —
soup init --template hipaa|soc2|eu-ai-act|sr-11-7. Four regulation-shaped starting configs. Soup's compliance controls are CLI flags/commands rather than config keys, so each template is a valid training config plus header comments naming the exact commands for that regime (PHI scrubbing + air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI + energy tracking for the EU AI Act, repro-receipt + diagnose/ship for SR 11-7). Templates default to a license-clean Apache-2.0 base. soup card <registry-id> -o MODELCARD.md— model-card autogen. Turns a Local Model Registry entry into a publishable, provenance-carrying HF model card: base model, training config, eval scorecard, config/data hashes, lineage (ancestors) and a table of every registered artifact. Adapter vs full-model is inferred from registered artifacts, falling back to the training config (LoRA rank, with Spectrum/LISA full-FT correctly treated as dense), so the card sets the rightlibrary_nameand never misreports the model type.soup push --card <registry-id>— render that registry-driven card and upload it asREADME.md, overriding the auto-generated one. A bad ref fails fast before any network call; HF hub only.soup ci init— fine-tuning CI. Writes.github/workflows/soup-gate.yml, a PR gate chainingsoup data validate→soup expect→soup ship --evidence(exit 2 blocks the merge). Every interpolated path is validated to stay under the repo root and shell-quoted; the branch and Python version are regex-gated; the write is atomic, symlink-rejecting, and refuses to clobber an existing workflow without--force.- Compliance quickstart — a new docs/compliance.md walkthrough: template → PII scrub → train with receipt/Annex XI/energy → registry → BOM + attestation → scan/sign/verify → air-gap → model card → CI gate.
Fixed
- GGUF export now actually works on Windows (validated end-to-end against a
locally-built llama.cpp: SmolLM2-135M → q4_0 / q4_k_m / q8_0 / f16 →
soup deploy ollama→ live inference). Four real bugs, each of which independently broke the path:soup export --format ggufcloned llama.cpp into your current directory.SOUP_DIRis the bare name.soup, but the lookup used it relatively rather than anchoring to~like the rest of the codebase — so the canonical~/.soup/llama.cppwas never found and a fresh ~200 MB checkout was dropped into whatever directory you ran from.- The first GGUF export downgraded your PyTorch and broke CUDA. The auto-clone
ran
pip install -r <llama.cpp>/requirements.txtinto your interpreter, and llama.cpp pinstorch~=2.2.1against the CPU wheel index (observed: torch 2.5.1+cu → 2.2.2+cpu, transformers 4.57 → 4.46). Soup now installs only the convert script's extra dependencies, unpinned, and never touches torch. - A correctly-built llama.cpp was not found on Windows. MSVC (like Xcode) is a
multi-config generator and emits
build/bin/Release/llama-quantize.exe; only the flat single-config layout was searched. soup deploy ollamafailed on a relative GGUF path with "pull model manifest: file does not exist" — Ollama resolvesFROMagainst the Modelfile's directory, and Soup writes the Modelfile to a temp dir. The Modelfile now emits an absolute path.
- Model-card injection hardening (affects the pre-existing
soup pushcard too). The## Trainingsection interpolatedbase/task/scheduler/recipeunescaped. SinceSoupConfig.baseandschedulerhave no charset validator, a crafted-but-valid config could smuggle raw HTML — or a backtick breaking out of the surrounding code span — into a card published to the Hub. All values now go through the markdown escaper, which additionally neutralises backticks and strips C0/ESC control bytes.
[0.71.34] - 2026-07-15
Added
soup adapters arithmetic— task-vector algebra over LoRA adapters (add / scale / negate). Apply task arithmetic (arXiv:2212.04089) to LoRA deltas via an expression such as"coder + 0.5*math - toxic", mapping names to adapter dirs with repeatable--adapter name=path. Produces one merged adapter you can serve or merge.- Signed, un-normalized element-wise combine over same-rank adapters; the effective
delta
ΔW = B @ Ascales linearly with each coefficient (negation flips the delta,0.5·halves it) via a √|c| factor split — not thec²a naive sum gives. Mixed-rank inputs are refused with a clear "harmonize rank" message. - Reuses the backdoor-scan gate (refuses a FAIL-scanned input unless
--allow-unscanned) and a same-base-model check (--allow-cross-baseto override). Hand-written expression parser (noeval), cwd-contained/symlink-rejecting paths, exit 0 = ok / 1 = refusal.
- Signed, un-normalized element-wise combine over same-rank adapters; the effective
delta
- LISA — Layerwise Importance Sampled AdamW (arXiv:2403.17919). Full-fine-tuning
quality at LoRA-like memory: every N steps LISA freezes all decoder layers except a
small random set (embeddings + head always trainable). Enable with
training.lisa_enabled: true(+lisa_num_layers,lisa_interval_steps) on atask: sft, transformers, text,quantization: nonerun; mutually exclusive with LoRA features and the other freeze mechanisms. Live on a 4 GB GPU for small models.
[0.71.33] - 2026-07-13
Added
soup draft— train and, above all, MEASURE a speculative-decoding draft model.soup draft measure --target <m> --draft <d> --prompts p.jsonlreports a draft's acceptance rate (the fraction of the target's own greedy tokens the draft would have proposed correctly) plus real plain-vs-assisted throughput. This is the honest gate: it tells you whether speculative decoding is worth enabling before you ship it. Exit 0 / 2 (below--min-acceptance, for CI) / 1.soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/distils your target into the tiny base (logit KD via the existingtask: distilltrainer) and emits a dense draft model, ready to load as anassistant_model.soup draft list, plus a local draft registry (~/.soup/drafts.json) thatsoup serve --auto-specconsults before the built-in pairing table — so a draft you trained yourself is picked up automatically.- Draft and target must share a tokenizer; a mismatched pair is refused up front (speculative decoding proposes draft token ids into the target's vocabulary, so a mismatch silently produces garbage rather than failing).
Read the measured results before you use this — see Known limitations below. On the validated pair, distillation did not improve acceptance, and speculative decoding was a net slowdown.
soup draft measureis what tells you that.
Known limitations
- Distilling a draft did not improve its acceptance rate on the validated pair.
Measured on
SmolLM2-360M-Instruct(target) with aSmolLM2-135M-Instructdraft: the stock draft already scored 69.3%, and distilling it moved that to 69.7% after 2 epochs and back to 69.3% after 10 epochs — i.e. no gain beyond noise. A small same-family draft is already near its capacity ceiling for agreeing with the target, and logit KD cannot buy capacity it does not have. Whether distillation materially raises acceptance for a genuinely diverged fine-tune, or a larger target/draft pair, is unproven on a 4 GB box — tracked as a scale issue. - Speculative decoding was a net slowdown on that pair (measured 0.55–0.64×): the
draft's forward pass costs more than the tokens it saves at this size.
soup draft measurereports this truthfully rather than assuming a speedup — which is precisely the point of shipping the measurement. - Acceptance is teacher-forced greedy agreement, the metric the Medusa/EAGLE papers report. It is exact and deterministic, and it is the right number for comparing drafts — but it is not the accepted-token count of a sampling run, which also depends on the rejection-resample cascade.
- Same-tokenizer only. Cross-tokenizer drafts (ULD-aligned + universal assisted decoding) are deferred.
Changed
- PRM-guided GRPO scores completions in a single batched forward.
PRMScorer.__call__now right-pads all completions into one[B, T]tensor (+ attention mask) and runs oneoutput_hidden_statespass instead of one forward per completion, cutting per-step reward latency on non-tiny models. Numerically identical to the per-completion path (parity + mixed-length tests). Closes #298 (#301 by @Ekaanksh-dev).
Fixed
- The hardware-fit gate no longer refuses to train small models.
model_size_from_namedid not understand anM(millions) suffix, soSmolLM2-135Mfell through to the 7B default, was predicted to need ~14 GB of weights, and was blocked. Every draft-sized model hit this.1.7Bwas also being read as7B(the7bmarker matched inside1.7b), whileQwen2.5-7B-Instruct-1Mcorrectly stays 7B (1M is the context, not the parameter count). soup serve --backend vllmno longer force-enablestrust_remote_code. The vLLM path now goes through the same--trust-remote-codedefault-deny gate (and warning panel) as the transformers backend, so serving an untrusted repo never executes its code silently.- Multi-adapter serving (
soup serve --adapters name=path) now actually switches adapters. The named adapters are loaded into the model and selected per request (viaPOST /v1/adapters/activate/{name}or the requestadapterfield); previously every request silently ran the startup model. The base model is served when no adapter is selected. - Vision datasets reject out-of-directory image paths.
llava/sharegpt4vrows are containment-checked againstimage_dir(mirroring the audio loader), so a crafted{"image": "/etc/passwd"}row can no longer read arbitrary local files. soup train --dry-run --gpus Nno longer launches a real multi-GPU run. The accelerate re-exec is skipped under--dry-run. The re-exec also now forwards--minillm-on-policy,--capture-activations, and--capture-prompts(previously dropped on multi-GPU runs).- MLX SFT now builds a real optimizer (
AdamWfrom the configured LR) instead of passingoptimizer=None, which left the model untrained. soup data inspect/preview/searchescape dataset- and Hub-derived text so a stray[/]no longer crashes the command and a crafted[link=…]tag can't render a phishing hyperlink.soup runslist/show escape config-derived fields too.soup infer --task asrhardening — an oversized reference no longer crashes the whole batch after transcription (that row's metric is skipped); an all-skipped run exits non-zero instead of reporting success;--asr-taskis validated upfront; and dataset-derived filenames are control-stripped before printing. ASR training now caps transcript labels to Whisper's decoder limit, warns on >30 s audio, and picks fp16 on pre-Ampere GPUs instead of hardcoding bf16.- Knowledge-distillation KD term aligns with the CE term. The token-level divergence is now computed over causal-shifted positions, so the distillation signal covers exactly the trained tokens (previously off by one).
- Miscellaneous robustness —
load_config_from_stringraisesValueError(notTypeError) on a non-mapping YAML document; the Web UI Bearer-token check is constant-time; andsoup doctor --vscode/ the LR-finder report use the centralised atomic, symlink-rejecting writer.
[0.71.32] - 2026-07-07
Added
- ASR fine-tuning (
task='asr', Whisper) — fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) / base (74M) train on a 4 GB GPU.- New
AsrTrainerWrapper(HFSeq2SeqTrainer+WhisperProcessor); data rows are{"audio": <path>, "text": <transcript>}under the newdata.format='asr'. Audio decodes via the hardenedload_audio_mono(16 kHz mono, soundfile pre-probe +O_NOFOLLOW+ symlink/size guards); transcripts become decoder labels (pad → −100, decoder-start token stripped). - Optional LoRA on q/v attention projections via
training.asr_lora: true(default full fine-tune);training.asr_language/training.asr_task(transcribe|translate) set the decoder prefix and are persisted to anasr_generation.jsonsidecar so inference restores them. soup infer --task asr— transcribe an{"audio": path[, "text": ref]}JSONL; reports per-row and corpus WER/CER when references are present. Loads a full model or a PEFT/LoRA adapter dir. Flags:--asr-language,--asr-task,--audio-dir(audio paths are cwd/dir-contained; UNC/traversal rejected).- New pure-python
utils/asr_metrics.py— WER / CER /word_accuracy(= 1 − WER, for a higher-is-better ship metric leg) /corpus_wer, with a light Whisper-style text normalizer (no new dependency). - Recipes:
whisper-tiny-asr,whisper-base-asr(live-trainable),whisper-large-v3-asr(parse-only, needs a larger GPU), plussmolvlm-256m-sft(vision). Catalog 138 → 142.
- New
Fixed
model_size_from_namenow knows Whisper checkpoint sizes (tiny…large), so the hardware-fit gate no longer mistakes a 39M whisper-tiny for the 7B default and blocks ASR training on consumer GPUs.
[0.71.31] - 2026-07-06
Added
- Judge-in-the-loop suite — put an LLM judge in the loop across the workflow:
task='online_dpo'— Online DPO training (wraps TRLOnlineDPOTrainer): the model generates two completions per prompt on-policy each step and a judge (a pairwise LLM judge over the existing ollama/openai-compatible backend) OR areward_modelpicks the winner. Config:training.online_dpo_judge: "ollama://model"(or setreward_model— exactly one),online_dpo_loss_type: sigmoid|ipo,online_dpo_max_new_tokens;betareusesdpo_beta. Transformers + text only. Recipe:online-dpo-smollm2-135m. Adapts to the installed TRL: on trl 0.19.x the judge is a swap-debiased pairwise comparison; on trl 1.x (which removed pairwise judges) the sameJudgeEvaluatoris used as a pointwise reward function — a documented per-version behaviour difference.soup data best-of-n— Best-of-N rejection sampling (BOND-lite): sample N completions from--baselocally, a--judgescores each pointwise, and the winner is written as an SFT chat row (with provenance).--emit-pairsalso writes winner-vs-loser DPO pairs.soup data evolve— Evol-Instruct instruction evolution (WizardLM depth / breadth) over an ollama/vllm provider, completing the synthetic-data suite (Magpie / Forge / Persona / evolve).soup ship --task-mode pairwise— a true pairwise judge win-rate as the ship leg-1 task-win (base = 0.5 coin-flip, tuned = its win-rate; swap-debiased), fusing with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.
Security
soup data best-of-n/evolvewrite outputs via atomicmkstemp+os.replace(re-validated cwd containment), closing the TOCTOU symlink-swap window between the containment check and the write. All judge/provider URLs are SSRF-validated; model loads probetrust_remote_code.
[0.71.30] - 2026-07-05
Added
- PRM-guided GRPO — use a trained Process Reward Model as the per-step
reward inside GRPO (the o1-era process-supervision signal). Set
training.prm_reward: <PRM dir|id>(a model produced bysoup traintask=prm) andtraining.prm_aggregate: min|prod|last; the PRM splits each generated completion into reasoning steps, scores every step with its reward head, and folds the per-step scores into one scalar reward that GRPO optimises. The PRM reward replacesreward_fnand rides the existing reward-shaping + reward-hack-mitigation seam, so the v0.71.26 controller observes it. Cross-validators gatetask='grpo'+backend='transformers'+modality='text'. Default aggregation ismin(weakest-link);prodassumes calibrated[0,1]step scores. - Bundled rollout environments — three pure-Python toy environments
(
soup_cli.envs.calculator/retrieval_qa/guess_number) exposing arollout(prompts)entry point so the liveopenenvGRPO rollout path runs out-of-the-box:training.rollout_backend=openenv+training.rollout_func=soup_cli.envs.calculator:rollout. Three ready-made recipes added (grpo-env-calculator/grpo-env-retrieval-qa/grpo-env-guess-number); catalog 134 → 137.
Fixed
soup train task=prmproducer conformance (surfaced by the v0.71.30 live smoke): the PRM trainer now casts its reward head to the base-model dtype (bf16 CUDA runs previously crashed on the firstcompute_loss), saves the tokenizer alongside the model (so a PRM checkpoint is loadable standalone), and returns the standard trainer-result shape (previously the CLI crashed with aKeyError: 'initial_loss'right after saving).
Notes
- Proof-of-mechanism only: validated on a tiny model (SmolLM2-135M) with a tiny
synthetic PRM and synthetic reward — not a production reward-model claim
(scale ask tracked in #286). The bundled environments are deterministic
single-shot seeders, not interactive multi-turn model-in-the-loop episodes
(the live
openenvcontract passes only prompts). Step split is a newline heuristic; PRM completions are scored one forward pass each.
[0.71.29] - 2026-07-05
Added
soup shrink— depth-prune a model + optional distill-heal ("The Unreasonable Ineffectiveness of the Deeper Layers", arXiv:2403.17887). Ranks decoder layers by the angular distance of the residual stream across a contiguous block over a calibration set, drops the least-important block (first and last layer always protected), optionally heals by distilling the original model into the pruned student, and emits a single dense smaller model plus a one-screen SHIP / DON'T SHIP perplexity verdict.soup shrink --model <id|path> [--drop-ratio 0.25 | --drop-layers N] --calib <calib.jsonl> [--heal <heal.jsonl> --heal-steps N] [--tolerance 0.10] [-o <dir>] [--device cpu] [--attach-to-registry <id>] [--plan-only]. Exit codes: 0 = SHIP, 2 = DON'T SHIP, 1 = error. The heal runs as an isolatedsoup trainsubprocess (LoRA-student logit distillation) and the adapter is fused back so the shipped artifact stays a single dense model. Arch allowlist v1: Llama / Qwen / SmolLM. Validated live on SmolLM2-135M (drop 25 %: 30 -> 22 layers, ppl x2.98 unhealed; drop 4 + heal: ppl x1.35, recovered).
Security
soup shrinkcontains--calib/--heal/--output-dir(and every derived write path:<out>/model,<out>/heal_adapter, the fuse staging dir) under cwd withos.path.realpath+commonpath+ O_NOFOLLOW + symlink rejection, re-validating derived paths right before each write (TOCTOU). The heal subprocess uses an argv list (no shell) with a timeout; its config is schema-validated before spawn; subprocess output is C0/ESC-stripped before it reaches the terminal.--modeldefaultstrust_remote_code=Falsewith a probe + warn.
[0.71.28] - 2026-07-04
Added
- MCP server (
soup mcp serve) — drive Soup from any Model Context Protocol client (Claude Code / Cursor / Cline / Continue) over stdio. No fine-tuning CLI ships an MCP server. Exposes 14 read-only tools as JSON —advise,data_inspect/data_validate/data_score/data_doctor,recipes_search/recipes_show,runs_list/runs_show,registry_list/registry_show,profile,diagnose_evidence,ship_evidence— plus two plan-only mutating tools (train_start,export) gated behind--allow-mutating(they render the exact command that would run; they never execute). The officialmcpSDK is behind a new[mcp]extra (pip install 'soup-cli[mcp]'), lazy-imported so the core CLI stays light. Security: stdio-only (no network listener); every path argument re-enters cwd-containment + symlink rejection; output is control-char sanitized; errors are path-free; string / size / int bounds enforced.
Fixed
- The DPO / IPO / KTO / BCO trainers now apply configured vocabulary expansion
(
data.add_new_tokens/data.new_special_tokens) via the sharedapply_vocab_expansion()helper during setup, consistent with the SFT path — they previously ignored it. Closes #292 (#293 by @CODING-DARSH). - The ORPO / SimPO / GRPO trainers now also apply configured vocabulary
expansion via the shared
apply_vocab_expansion()helper — completing consistent vocab-expansion behavior across every SFT/preference/RL trainer. Closes #294 (#295 by @CODING-DARSH).
[0.71.27] - 2026-07-03
Added
- Fine-tune Doctor — kill the top silent fine-tune failures before a
single training step; no competitor (Unsloth/Axolotl/LlamaFactory) ships
any of these three:
soup data doctor <data> --model <id|path>— chat-template compatibility report over 8 checks:chat_templatepresent,template_renders cleanly, has{% generation %}markers,eos_in_labels(the #1 "model never stops generating" bug — every assistant turn's trained span must actually contain an EOS/EOT token; checks every turn, not just the last),bos_duplication(template + tokenizer both prepending BOS),system_rolesupport (Mistral-style templates reject a leading system turn),unknown_roles, andtruncation_risk(p95 rendered length vsmax_length). Same OK / MINOR / MAJOR taxonomy assoup diagnose; exit 0 = OK/MINOR, exit 2 = MAJOR.--train-on-responses-only/--train-on-messages-with-train-fieldselect the same masking strategysoup trainwould use, so the report and--show-masknever disagree about what's actually trained.soup data doctor ... --show-mask N— render N sample rows with per-token trained/masked colouring through the REAL collator path (answer-only / per-message-train-field / RAFT span-mask) — not a reimplementation — so an assistant-mask bug is visible instantly.soup data lint <data>— preference-data linter for dpo/orpo/simpo/ipo/bco/kto:length_bias(chosen systematically longer than rejected — the #1 silent DPO degradation, reported as a Cohen's d effect size),label_imbalance(KTO desirable:undesirable ratio),near_duplicates(MinHash/LSH, reuses thesoup data dedupkernel),identical_pairs(chosen == rejected — zero preference signal), andprompt_leak(the prompt echoed verbatim inside the completion — a common synthetic-data pipeline bug). Optional--modelfor exact token-length bias (default: word count).- Validated live against the real
HuggingFaceTB/SmolLM2-135M-Instructtokenizer on Windows + RTX 3050 — this smoke pass found and fixed two genuine bugs beyond what synthetic fixtures alone caught: an EOS check that required the EOS token to be the literal last trained token (real templates often have a trailing formatting token after the closing tag that stays inside the trained span), and two call sites that only caught(ValueError, TypeError)around a tokenizer'sapply_chat_templatewhen a real Jinjaraise_exception()(Mistral-style no-system-role guard) raisesjinja2.exceptions.TemplateError.
Fixed
- Harden
commands/diagnose.py's--evidenceloader against a TOCTOU symlink swap: opens withO_NOFOLLOWand size-checks the open fd viaos.fstatinstead ofos.path.getsizeon the path before the open — backports the hardened loader shipped forsoup shipin v0.71.25 (closes v0.71.25 known-limitation (4)). - Harden judge-model URL validation against a hostname prefix bypass
(
http://localhost.attacker.com) —GateTask._valid_judge_url/_parse_judge_urlnow useurllib.parse.urlparse+ hostname checks instead ofstartswith. Closes #283 (#288 by @CODING-DARSH). SFTTrainerWrappernow applies configured vocabulary expansion (data.add_new_tokens/data.new_special_tokens) and resizes the model embeddings during initialization — previously these fields were accepted by the schema but silently ignored. Closes #289 (#287 by @CODING-DARSH).- Vision and audio SFT paths now apply that same configured vocabulary
expansion (
data.add_new_tokens/data.new_special_tokens) via the sharedapply_vocab_expansion()helper, consistent with the text SFT path — they previously ignored it. Closes #290 (#291 by @CODING-DARSH).
Security
soup data doctorstrips C0 control characters (keeping tab/newline/CR) from dataset-derived content before it reaches the terminal — Rich'smarkup.escape()only neutralises[...]tag syntax, not raw escape sequences, so an untrusted training row (e.g. an unknownrolefield, or--show-mask's decoded token text on a byte-level BPE tokenizer) could otherwise carry a literal ESC byte through to the terminal (title-bar / OSC-8 link spoofing, or obscuring a MAJOR verdict via cursor tricks).--outputJSON is unaffected (json.dumpsalready escapes control characters).
[0.71.26] - 2026-07-01
Added
- Closed-loop reward-hacking auto-mitigation. The trainer now detects
reward hacking mid-run and self-corrects — instead of only halting. Set
training.reward_hack_mitigation(orsoup train --reward-hack-mitigation) to one of four modes on a GRPO/PPO run (requiresreward_hack_detector):log_only— instrument only: append a per-stepmitigation_log.jsonl(drop_pct, verdict, reward mean/std, completion-length trend, repetition) and never touch training.kl_control— a reversible bang-bang + hysteresis controller: when the hacking signal trips, raise the KL coefficient β (geometric, clamped to[floor, ceil], never crossing 0); relax it when the signal recovers. Dwell + release-patience prevent flapping; a multi-signal vote combines the detector drop with a length-trend and repetition signal.pid_lagrangian— a PID-Lagrangian controller (Stooke et al.) that holds the hacking signal at a target, plus an escalation ladder: raise β → roll back to the last-good RL checkpoint → early-stop.- Anti-gaming hardening: per-signal EMA/median smoothing, conservative-on-disagreement voting, a reward-distribution-drift guard, and optional bounded reward shaping on the gamed proxy (length / repetition / sentinel). A plain-English give-up explanation is logged on early-stop.
- Proof-of-mechanism only (see Known Limitations): validated on SmolLM2-135M + a synthetic length-hacking task on a single RTX 3050 — all four stages pass live, including a real mid-run rollback. PPO ships BETA (mechanism unit-tested; the on-GPU proof is GRPO-only).
- Ready-made
qwen2.5-coder-7b-sftrecipe forQwen/Qwen2.5-Coder-7B-Instruct(catalog 133 → 134) (#285 by @Deadpool2000).
Security
RLCheckpointCallback.restore_checkpoint/save_checkpointrefuse a symlinkedoptimizer.pt—torch.load(weights_only=False)on an attacker-placed symlink in a shared checkpoint dir was an RCE vector.- Bool-before-int/float guards on every new
reward_hack_*numeric field;reward_hack_signalsbounded (max_length=4); the mitigation log writer is cwd-contained with symlink-reject-on-rotate and secret redaction.
[0.71.25] - 2026-06-27
Added
soup ship— the SHIP / DON'T-SHIP verdict. After fine-tuning, answer one question: did the model get better, or did I break it?soup shipfuses two checks into a single binary decision — leg 1: the task metric strictly improved (base → tuned); AND leg 2: no general benchmark regressed past a forgetting threshold (default 0.05 absolute points). It SHIPs only when both hold — otherwise DON'T SHIP, even if the task metric looks great. The output is a one-screen verdict + the reason, with CI-gateable exit codes (0 = SHIP, 2 = DON'T SHIP, 1 = runtime error).- Leg-1 modes:
--task-mode metric(eval accuracy) orjudge_score(LLM-as-a-judge); pairwise win-rate is planned for a later release. - Leg-2 suite: built-in mini benchmarks by default (offline, CPU), or
--general-suite <names>to route lm-eval benchmarks;--baseline registry://… | file.jsonsupplies recorded base scores. --evidence ev.jsondecides offline from pre-computed scores (no model load);--output verdict.jsonpersists the machine-readable verdict.
- Leg-1 modes:
- Friendlier error messages: the CUDA-OOM hint now also suggests
gradient_checkpointingand4bitquantization, plus new mappings for Hugging Face gated repos (huggingface-cli login/HF_TOKEN) andtrust_remote_codeerrors. Closes #272 (#282 by @Akshaya-reddy18).
Security
soup shipinput hardening:--evidenceis opened withO_NOFOLLOW+ an fstat size cap (16 MiB) under cwd containment;--task-evalis cwd-contained and symlink-rejected;--judge-modelis validated by scheme/host viaurlparse(blocks thehttp://localhost.attacker.comprefix bypass); lm-eval model ids reject,/=injection;--general-suiteis bounded (≤ 50 names, ≤ 256 chars each).
[0.71.24] - 2026-06-21
Added
- 2026 model-family recipe expansion (catalog 116 → 133). 17 new ready-made
SFT recipes for the open-weight models released Feb–Jun 2026, each with its
Hugging Face repo-ID verified to resolve:
- Qwen 3.5 (Apache-2.0):
qwen3.5-0.8b-sft,qwen3.5-2b-sft,qwen3.5-4b-sft,qwen3.5-9b-sft,qwen3.5-27b-sft, and theqwen3.5-35b-a3b-sft/qwen3.5-122b-a10b-sft/qwen3.5-397b-a17b-sftMoE sizes. - Qwen 3.6 (Apache-2.0):
qwen3.6-27b-sft,qwen3.6-35b-a3b-sft. - DeepSeek-V4 (MIT):
deepseek-v4-flash-sft,deepseek-v4-pro-sft. - GLM (MIT):
glm-5.1-sft. - Kimi (Modified MIT):
kimi-k2.5-sft,kimi-k2.6-sft. - MiniMax (MiniMax Community License — commercial use needs a separate
agreement):
minimax-m3-sft. - Mistral Large 3 (Apache-2.0, 675B/41B-active multimodal MoE):
mistral-large-3-sft.
- Qwen 3.5 (Apache-2.0):
- Unit-test coverage for the
warmup.pyauto-warmup-steps helper (#274 by @shatakshi-1404).
Fixed
- Stale recipe repo-ID:
glm-5-sftnow points atzai-org/GLM-5(the org migrated fromTHUDM).
[0.71.23] - 2026-06-12
Added
- Native Spectrum targeted training (closes #266). A new
soup spectrum scanreads a model's.safetensorsshards one tensor at a time (no model load — peak RAM is the largest single weight matrix), computes a singular-value SNR per weight matrix with a Marchenko-Pastur noise threshold (arXiv:2406.06623), ranks layers within each module-type group and prints the top--top-percentas a ready-to-pastetraining.unfrozen_parametersYAML block. This lets you scan even a very large model's layer SNR on a CPU box and then full-fine-tune only the high-signal layers.soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn|all] [--output patch.yaml]— SNR table + the YAML patch; results cache at~/.soup/spectrum/<slug>.json(override viaSOUP_SPECTRUM_CACHE_DIR).- New schema field
training.unfrozen_parameters: list[str]— regex patterns of parameter names to keep trainable; the SFT trainer freezes every parameter then unfreezes the matched set (full fine-tuning, LoRA off). Mutually exclusive with LoRA features /freeze_layers/freeze_ratio/train_router_only/expand_layers; requirestask=sft,backend=transformers,modality=text, andquantization=none. - The SNR kernel is pure-numpy and transpose-invariant (singular values are
identical for
WandW.T); GPT-2Conv1Dnaming (c_attn/c_fc/c_proj) is recognised alongside Llama-style names. - The existing
spectrumtrainer-plugin wrapper is untouched (back-compat). LISA (per-step layer sampling) is tracked separately in #267.
Security
soup spectrum scanvalidatesunfrozen_parameterspatterns at parse time: rejects nested-unbounded-quantifier regexes (ReDoS), null bytes, empties, and caps count (50k) and length (512). Hub downloads route through the SSRF-hardened, namespace-pinnedhubs.snapshot_download; symlinked shards and matrices above a 2^31-element SVD cap are skipped;--outputstays under cwd.
[0.71.22] - 2026-06-10
Added
- Perf & measure polish — a 4-issue patch tightening four live paths from
the recent BETA lifts. Pure code, validated on Windows + RTX 3050.
- MiniLLM on-policy KV-cache (closes #263). The on-policy distillation
rollout (
soup trainwithtraining.minillm_on_policy: true) now threadspast_key_valuesso each step forwards only the new token instead of re-feeding the whole prefix — resolving the O(L²) per-step cost from v0.71.18. A LoRA student (the common distill case) activates the cache too: the new_supports_kv_cacheprobe unwraps the PEFT model viaget_base_model()before deciding. The teacher is always cached; the student cache respects the retained autograd graph and degrades gracefully if a model returns no cache mid-loop. soup serve --moleKV-cache (closes #262). Each of the N task adapters in a served MoLE now keeps its own KV cache in lockstep, created fresh pergenerate()call (never stored on the instance, so there is no cross-request leak). Top-k zero-weight adapters are still skipped, and the output is byte-identical to the no-cache path on a real MoLE.- Deploy-autopilot live measure factories (closes #143).
soup deploy autopilot --measureships a first-party transformers loader factory (lazy import, per-candidate quant config via the Quant Menu loader;before= base,after= quantised) replacing the inject-only test hooks. The baseline is now scored once and the whole candidate list is pre-validated up front, so a typo in--measure-candidatesraises before any model load instead of burning N live loads or doubling peak VRAM. - Live-codec TTS via SNAC, partial (#265-partial). The live-codec
encode path (
data.format='audio') is validated for Orpheus:load_audio_mononow probessoundfile.info(duration + byte cap) beforesoundfile.read(no multi-GB decode into RAM) and reads through anO_NOFOLLOWfile descriptor; a real SNAC-backed encode of a 24 kHz wav produced 42 Orpheus codec tokens.
- MiniLLM on-policy KV-cache (closes #263). The on-policy distillation
rollout (
Fixed
- MiniLLM on-policy KV-cache was silently disabled for LoRA students (the
PEFT wrapper hid the base model's
past_key_valuessupport) — now probed viaget_base_model(). - Deploy-measure no longer re-scores the baseline once per candidate or burns live model loads on a bad candidate (per-candidate validation moved up front).
load_audio_monocapped audio duration only after decoding into RAM — the cap is now checked fromsoundfile.infobefore reading.
Known limitations
- KV-cache correctness is validated (cache == no-cache equality on real tiny artifacts) but large-model throughput gains were not measured on the 4 GB dev box.
- #265 stays open — the live-codec
data.format='audio'SNAC encode path is validated for Orpheus only; the other four TTS families keep their per-family codec dependency gate. - The deploy-measure first-party factory's real quantized (bitsandbytes 4-bit) load is CUDA + bitsandbytes-gated; on Windows / no-bnb the injected test seams are the validated path.
- The MoLE serve KV-cache assumes single-sequence (
B == 1) decode.
[0.71.21] - 2026-06-10
Added
- Precision & rollout lift (BETA, hw-gated) — lifts five deferred
NotImplementedErrorstubs to live code.- FP8 attention + NVFP4 (closes #141).
training.fp8_attention: truenow converts the model's attention projections (q/k/v/o + fused qkv variants) to FP8 training modules via torchao'sconvert_to_float8_trainingwith an attention-onlymodule_filter_fn(Hopper SM ≥ 9.0 gate);training.nvfp4: truequantises via torchao'sNVFP4Config(Blackwell SM ≥ 10.0 gate). Both are wired into the v0.28 speed/memory pipeline and degrade to a visible yellow advisory when the gate fires — a conversion failing partway raises an honest "model may be PARTIALLY converted" error rather than silently training on a half-converted model. - vLLM sleep mode (closes #124).
training.vllm_sleep_mode: trueis live:create_vllm_engine(sleep_mode=True)setsAsyncEngineArgs.enable_sleep_mode(vLLM ≥ 0.7 gate with a friendly upgrade message), the newvllm_sleep_cycle(engine, level=1|2)context manager wraps the optimisation step (wake infinally), and the GRPO trainer threads the flag into TRL'sGRPOConfigwhen the installed TRL exposes the hook (advisory otherwise). - Multi-turn agent rollout launchers (closes #125).
soup trainwithtask: grpo+training.rollout_backend: openenv+training.rollout_func: my_module:fnnow runs a LIVE rollout: the resolver imports the operator's callable (same trusted-code policy asdata.prompt_strategy), feeds it the dataset prompts as seeds, and the returned{prompt, answer?}rows replace the prompt dataset. Rows are normalised (extra keys stripped, message-list prompts deep-copied, non-string answers rejected loudly).art/ruler/nemo_gymraise a friendly ImportError when the backend package is missing and an honest BETA gate when present (injectable_EXTERNAL_ROLLOUT_RUNNERSseam). Validated by a real GRPO + openenv rollout train on SmolLM2-135M. - Apple-adapter conversion (closes #228).
soup apple-adapteris live forhf-to-mlx/mlx-to-hf: PEFT LoRA safetensors ↔ mlx-lm adapters with both matrices transposed (lora_A [r,in]↔lora_a [in,r]), bf16 sources upcast via the torch loader,adapters.safetensors+num_layersemitted for mlx-lm'sload_adapters, rank/alpha/dropout carried through, legacyadapters.npzstill read, optional v0.60 Merkle-root signing. The*-to-appledirections stay upstream-gated (no published FoundationModels adapter spec). Validated by a real bf16 PEFT adapter round-tripping with numeric equality. - Llama-4 expert delinearization (closes #97).
soup delinearize-llama4now runs a live torch runtime: fused 2-D expert tensors[E*dim_in, dim_out]reshape to 3-D[E, dim_in, dim_out](expert count fromconfig.jsonor--num-experts), other tensors pass through, JSON sidecars are copied, writes are atomic.--plan-onlykeeps the old render-and-exit flow.
- FP8 attention + NVFP4 (closes #141).
Fixed
safetensors.numpy.savesilently mangles non-contiguous (transposed) arrays — the apple-adapter writer now makes every array C-contiguous first (caught by the new round-trip assertions).
Known limitations
- fp8_attention / nvfp4 / vllm_sleep_mode are BETA hardware-gated — the
converters and gates ship validated via capability probes and fake-module
dispatch tests, but end-to-end runs need a Hopper/Blackwell GPU + torchao
(or vLLM ≥ 0.7), none of which exist on the maintainer's RTX 3050 /
Windows box. The
art/ruler/nemo_gymrollout adapters are honestly BETA-gated until validated against the upstream packages.
[0.71.20] - 2026-06-09
Added
- Modality II trainers — TTS / BitNet / MoE expert quant (BETA, hw-gated)
— lifts three v0.52.0 schema-only
NotImplementedErrorstubs to real code.- TTS fine-tuning (closes #131).
soup trainwithtask='tts'+modality='audio_out'now routes to a liveTTSTrainerWrapper. TTS families (Orpheus / Sesame-CSM / Llasa / Spark / Oute) are decoder language models, so a TTS fine-tune is next-token cross-entropy over interleaved[text][audio-codec-token]chat sequences — the wrapper reuses the SFT path and adds per-family emotion-control templating (Orpheus / Oute) and registration of operator-supplied codec special tokens (data.new_special_tokens) with an embedding resize. The pre-encoded chat workflow (codec tokens produced offline, then trained withdata.format=chat) is the live, validated path; the live-codec workflow (data.format='audio', encode raw audio at train time) needs the family's heavyweight codec dependency (SNAC / BiCodec / XCodec2 / …) and is hardware/dependency-gated with a friendly per-familyRuntimeError. Verified end-to-end on SmolLM2-135M-Instruct. - BitNet 1.58-bit (closes #134).
build_bitnet_trainerreturns a liveBitNetTrainerWrapperthat gates on the upstreamonebitllmspackage (absent → friendlyRuntimeErrornaming it).soup export --format bitnet | tq1_0now runs a real llama.cpp TQ1_0 ternary export (reuses the v0.53.1 gguf convert→quantize pipeline) instead of the deferred panel; it requires a built llama.cpp toolchain (friendlyFileNotFoundErrorwhen absent). - MoE expert quant + router-only training (closes #136).
apply_moe_expert_quantdetects fused-MoE expertnn.Linearblocks and replaces them with bitsandbytesLinear4bit(nf4) /Linear8bitLt(int8_rowwise), leaving attention + the router in full precision; it runs beforeget_peft_model(QLoRA-on-experts) so PEFT attaches to the quantized base.train_router_onlyfreezes every expert and keeps the gating router trainable, applied after LoRA. CUDA-gated (friendlyRuntimeErrorwhen bitsandbytes/CUDA absent). Validated live on an RTX 3050: 8 expert Linears → 8Linear4bitwith dequant error 0.0155 vs source (weights genuinely carried), router-only freeze, and device-aware placement.
- TTS fine-tuning (closes #131).
Known limitations
- The TTS live-codec workflow, BitNet 1.58 training (
onebitllms), and BitNet GGUF export (llama.cpp) are hardware/dependency-gated — the friendly gates ship and the plumbing is validated, but the end-to-end runs against real TTS models + audio codecs / a BitNet base + onebitllms / a built llama.cpp toolchain stay open infra-blocked items on the maintainer's RTX 3050 / Windows box.
[0.71.19] - 2026-06-09
Added
- Quant Menu for vision / audio modality (closes #81). The Quant Menu
(
gptq/awq/hqq:Nbit/aqlm/eetq/mxfp4/fp8) was rejected by the config modality gate formodality in {vision, audio}— those paths carried inlineBitsAndBytesConfigblocks that handled only4bit/8bit. v0.71.19 drops the gate (the mlx-backend gate is retained) and threads the unifiedbuild_quantization_config_for_loaderthrough_setup_vision_transformers/_setup_audio_transformers, so multi-modal SFT can train a LoRA on top of any pre-quantized base. The4bit/8bitconfig shapes are byte-for-byte the same as the old inline blocks;mxfp4still routes throughprepare_model_for_kbit_training. Verified: the unified loader returns the right config object for every format on both modalities, and_setup_vision_transformersthreads aGPTQConfigintoAutoModelForVision2Seq.from_pretrained.
Fixed
- Multipack DataLoader sharding under FSDP / DeepSpeed ZeRO / DDP (closes
#80). The multipack
get_train_dataloaderoverride built a rawDataLoaderand returned it directly, so under distribution every rank trained on the same packed bins (no data sharding). It now routes the loader throughaccelerator.prepare(...)whennum_processes > 1— exactly what HF Trainer's ownget_train_dataloaderdoes — so accelerate'sBatchSamplerShardround-robins whole bins across ranks (preserving the FFD packing) and equalises per-rank batch counts. The single-process path is unchanged (byte-for-byte the validated v0.40.4 raw-DataLoader behaviour). Verified live: a single-GPU multipack SFT on SmolLM2-135M trains end-to-end (RTX 3050). Full multi-GPU validation remains a QA item (no multi-GPU box); the distributed routing is mocked-tested.
[0.71.18] - 2026-06-08
Added
- MiniLLM true on-policy rollout (closes #257).
training.minillm_on_policy: true(withminillm_enabled: true) replaces the offline distribution blend with the real on-policy procedure of Gu et al. 2024 §3.1: each step samples a fresh autoregressive rollout from the per-token mixtureratio·teacher + (1-ratio)·student, then accumulates the length-normalised reverse-KLKL(student || teacher)on the full distributions (differentiable w.r.t. the student only; sampled tokens are detached). Newtraining.minillm_rollout_lengthknob ([1, 512]; auto-derivesmin(max_length, 32)when unset — the loop re-forwards the full prefix each step, so keep it small). Verified live: on-policy distill on tiny-gpt2 (student + frozen teacher), finite loss, end-to-end train. - Cross-tokenizer ULD with token-sequence alignment (closes #258). New
training.uld_strategy: wasserstein_alignedhandles fully-disjoint tokenizers (not just a vocab-size mismatch): per batch element the student and teacher token sequences are aligned over their decoded character spans (offset-overlap when both decode to the same text, difflib Ratcliff-Obershelp char matching otherwise), the teacher logits are mean-pooled onto the student positions, and the existing sorted-Wasserstein-1 surrogate is applied. Verified live: aligned distill with a GPT-2 BPE student + a Llama SentencePiece teacher, finite loss, end-to-end train. soup agent eval --sandbox(closes #110). Each heuristic-passing tool-call prediction is now executed against a generated mock of the endpoint in the v0.25.0 RLVRcode_execsandbox and classified into ok / tool_error / timeout / arg_error. The endpoint path, its required path params, and the predicted arguments are base64-embedded as data (no code interpolation). Strong isolation (RLIMIT / namespaces / sandbox-exec) is POSIX-only; on Windows the subprocess + 5 s timeout + 10 KB output cap + network guard still apply (a friendly reduced-isolation advisory is printed). Verified live on Windows: 4-prediction scorecard (ok=1 / tool_error=1 / arg_error=2 / timeout=0).soup train --cloud modal(closes #16). Render a self-contained Modal.com app fromsoup.yamlfor serverless GPU training when you have no local GPU. The config YAML is base64-embedded as data (no interpolation, no secrets); the--gputype (t4 / l4 / a10g / a100 / a100-80gb / l40s / h100) is validated against a closed allowlist. Default is plan-only: write the stub + print themodal runcommand.--cloud-submitattempts a live submit gated on a Modal token (modal setup/MODAL_TOKEN_ID+MODAL_TOKEN_SECRET). New[modal]extra (pip install 'soup-cli[modal]'; only needed for live submit — plan-only render needs no dependency). Verified live: real stub rendered, exit
[0.71.17] - 2026-06-08
Added
- Serve-time MoLE (closes #259). A
task='moe_lora_routing'run now writes a self-describingmole_manifest.jsonnext tomole_gate.pt, andsoup serve --mole <dir>loads the base + N frozen task LoRAs + the trained gate and blends them per token at decode time (custom blend loop — non-streaming + streaming).--molerequires--backend transformersand is mutually exclusive with--bank/--steer/--adapters/--speculative-decoding. The base model comes from--base(or the manifest when unset). Verified live on SmolLM2-135M (2 task adapters, real generation + SSE streaming). - Per-request multi-tenant vector banks (closes #260).
soup serve --banknow resolves the active VeRA/VB-LoRA user per request via acontextvars.ContextVar, so concurrent requests on a threaded server never race on shared instance state. The streaming path re-selects the user inside the generator's own context. Verified live: twoX-User-Idheaders produce distinct steered outputs, an absent / unknown id self-clears to the clean baseline (no cross-request leak), and a repeated user is deterministic. - Epoch-aware RAFT document shuffle (closes #253).
data.raft_epoch_shuffle: truere-permutes the golden + distractor documents each training epoch (per-epoch salt) so the model can't latch onto one fixed citation slot.epoch=0reproduces the legacy single-permutation order exactly. Verified live on a 2-epoch SmolLM2-135M RAFT run. soup diagnose --citation-style/--shuffle-seed(closes #254). The live citation failure-mode probe now accepts the citation style (bracket / inline / footnote) and the RAFT shuffle seed so the golden[doc-N]ids line up with what the model saw at train time. Verified live (rows=6, mean_recall=1.000).
Fixed
- MoLE
train()now returns theinitial_loss/final_loss/total_steps/duration_secs/durationkeys the generic train handler reads, sosoup train task=moe_lora_routingcompletes cleanly (previously raisedKeyError: 'initial_loss'after writing the gate). Surfaced by the #259 smoke.
[0.71.16] - 2026-06-07
Added
- Covariance-preconditioned ROME via
--cov-corpus(closes #250).soup edit set --method rome --cov-corpus <jsonl|txt>now estimates the key covarianceC = E[k kᵀ] + λIover a stats corpus and uses the preconditioned updateu = C⁻¹ k*instead of the covariance-freeC = Ipath — the genuine ROME closed form, which spreads the rank-1 update mass to reduce collateral interference with other facts. Falls back toC = Iwhen no corpus is given. The exact post-conditiondown(k*) += deltais preserved either way. The corpus loader is cwd-contained, symlink-rejected (O_NOFOLLOW + raw-path lstat), and size/line-capped;--cov-corpusis rejected (fail-loud) for any method other thanrome. Verified on realgpt2(prob 0.005 → 0.9997) and SmolLM2-135M. - GPT-2 (
transformer.h/mlp.c_proj) support in the edit kernels (closes #251). ROME / MEMIT / AlphaEdit now edit GPT-2-family models, not just Llama-family. TheConv1Dweight layout ([in, out], transposed relative tonn.Linear's[out, in]) gets a transpose-aware rank-1 update, AlphaEdit null-space projection, and MEMIT band dim-check. PEFT-wrapped GPT-2 / Llama models are unwrapped viaget_base_model. Verified end-to-end on realgpt2. - Mixtral joins the LongLoRA architecture allowlist (closes #147). A bare
mistraltoken does not appear inmixtral(m-i-x vs m-i-s), so the existingis_mistral_modeldetector excluded the MoE variant. A dedicatedis_mixtral_modelhelper +MixtralAttentionentry in the S² forward-override regex +_SEPARATE_QKV_FAMILIESnow cover Mixtral-8x7B / 8x22B (the attention is the standard separate-QKV shell; the MoE lives in the MLP).
Fixed
- Atomic
EditGovernoredit-count increment (closes #252). Two concurrentsoup edit setruns on the same base model could lose an increment: each read the persisted count, added locally, and the last writer clobbered the first.save_statenow re-reads the persisted count INSIDE the cross-process lock and merges this run's delta (edit_count − persisted_baseline), mirroring the v0.60.0namespace_pinpattern. Verified: two governors recording 3 + 2 edits from the same baseline persist a merged 5 (not a clobbered 2 or a naive +1).
Notes
- Test count: 13511 → 13595 (+84 net; +81 in
tests/test_v07116.py).
[0.71.15] - 2026-06-07
Fixed
- Iterative-DPO config render bug (closes #261).
soup iterative-dpo's default per-round trainer renderedoutput: {dir: ...}(a mapping), whichSoupConfig.output(a plain string) rejected — so the spawnedsoup trainsubprocess failed at config validation. Now rendersoutput: <str>, mirroring the v0.71.13 #229local-rlfix. A regression test captures the rendered YAML and validates it viaload_config_from_string; verified end-to-end with a realsoup trainround on SmolLM2-135M.
Changed
- CMA-ES merge loads the base model once (closes #246).
soup adapters merge --strategy cmaespreviously reloaded the (multi-GB) base model into a fresh PEFT wrapper on every candidate in the population. The default scorer now loads the base once and reuses it across the wholepopulation × generationsloop — each candidate only loads its small merged LoRA, applies it, generates, and unloads it. Verified on SmolLM2-135M: the base loads exactly once across N candidates. soup loopbudget gate now estimates real cost (closes #245). The pre-wired loop's per-iteration cost estimate was a hard0.0placeholder, so the dollar budget gate never tripped. It now wires v0.34run_cost. estimate_run_cost_usdoff the most-recent completed run's GPU + duration (the best forward signal for a repeating loop). Falls back to0.0on the first iteration / a CPU / unpriced GPU; never crashes the daemon.--diagnose-gateis multi-node aware (closes #170). The post-training diagnose gate (and the--annex-xi/--repro-receipt/ capture hooks) fired onLOCAL_RANK==0, so a shared-filesystem multi-node run ran them once per node. They now gate on the global chief (RANK==0whenRANKis set, elseLOCAL_RANK==0) — once per cluster.
Added
soup train --track-energy --energy-out <path>(closes #244) persists the measured energy/CO2 reading as JSON sosoup bom emit --energy <path>(the v0.71.3 #256 consumer) can attach it to an ML-BOM. Atomic + cwd-contained + symlink-rejected. Completes the train → BOM energy hand-off.
[0.71.14] - 2026-06-05
Added
- Live FSDP shard consolidation (closes #96).
soup merge-sharded-fsdp-weightslifts the v0.44.0 plan-only stub: it now streams eachpytorch_model_fsdp_*.binshard viatorch.load(weights_only=True)(no arbitrary pickle exec), unions the per-rank parameter fragments into one state-dict, and writes a single.safetensorsatomically. Memory-friendly (one shard loaded at a time). New--plan-onlyflag prints the plan without writing. Single-process — no multi-GPU needed to MERGE. (Per-rank disjoint-parameter / FULL_STATE_DICT shards; DCP sharded-tensor reconstruction is out of scope — useaccelerate merge-weightsfor those.) - Live
kv_cache_typewiring on the transformers serve backend (closes #140).soup serve --kv-cache-type q8_0 | bf16 | f16 | fp8lifts the v0.53.1apply_kv_cache_typeNotImplementedErrorstub:bf16/f16load the model in that dtype (the KV cache inherits it);q8_0routes an 8-bit HQQ quantized KV cache throughmodel.generate(needspip install hqq);fp8raises a friendly runtime error (vLLM + Hopper-only — the transformers backend has no fp8 KV path). vLLM / SGLang KV-cache-dtype routing stays in the infra-blocked tail. - ONNX export QA verified (closes #71) —
soup export --format onnxexercised end-to-end on a tiny model: export exits 0,model.onnxloads in ONNX Runtime withinput_idspresent, and a forward pass produces a real output. Recorded intests/qa/v07114_qa.md.
Notes
- GGUF export (#70), AWQ/GPTQ export (#72), the CUDA + llama.cpp QA doc (#144),
HF Hub push/Spaces deploy (#74), and the Community-QA tracking meta-issue (#79)
remain open with
infra-blockedlabels — they need a built llama.cpp toolchain,autoawq/auto-gptqWindows wheels, or HF credentials the QA box lacks. Seetests/qa/v07114_qa.md.
[0.71.13] - 2026-06-04
Added
- Prompt-compile family — live wiring (closes #225, #226, #227, #229). Four
soupcommands that shipped as deferred-stubNotImplementedErrorin v0.68.0 are now real, validated end-to-end (real DPO train on SmolLM2-135M + real Ollama teacher distillation on RTX 3050). soup local-rl trainruns a real nightly DPO/KTO/ORPO train (#229).--onceharvests the latest thumbs-up/down DPO pairs from the local-RL SQLite and trains them via asoup trainsubprocess (argv list, no shell); astatetable trackslast_train_atso a re-run with no new feedback skips, and a run with fewer than--min-pairs(default 10) skips. Without--onceit renders a systemd.service/.timer+ launchd.plistscheduler scaffold into--scheduler-dirfor the user to install. New flags:--once,--min-pairs,--output/-o,--scheduler-dir,--hour,--minute.soup distill-promptprepares a real distillation dataset (#226). For each prompt in the traces JSONL the teacher is called once via the v0.20 provider helpers (Ollama / Anthropic / vLLM);sft/klemit{messages:[user, assistant=teacher]}andpreferenceemits{prompt, chosen=teacher, rejected=student}. New flags:--provider,--base-url,--temperature,--max-rows.soup compileruns DSPy / GEPA / TextGrad prompt-program optimisation (#225) andsoup compile-toolsruns the TextGrad / GEPA tool-schema optimiser (#227), both lazy-importing the optimiser libraries behind the new[compile]extra (pip install 'soup-cli[compile]') with a friendlyImportErrornaming the extra when absent.--plan-onlystill renders the plan and exits 0.
Security
- systemd / launchd injection defence (#229).
local-rland the scheduler renderers reject\n/\rin the model id and shell-quote everyExecStartargument, so a crafted model id cannot inject extra unit directives.
Fixed
local-rltrain config renderedoutputas a mapping (#229). The nightlysoup trainYAML now emitsoutput: <dir>(a plain string the schema accepts) instead ofoutput: {dir: <dir>}; a regression test validates the rendered config againstSoupConfig.
[0.71.12] - 2026-06-04
Added
- Architecture + distillation + adapter-training — live wiring (closes #145, #146, #148, #158, #84, #221, #222). Seven surfaces that shipped schema-only in earlier releases are now real, validated end-to-end on tiny models (SmolLM2-135M / a locally-built tiny Llama).
- Sequence-level knowledge distillation is live (#145).
task: distillnow acceptsdistill_mode: token|sequence; sequence mode trains the student on the teacher's generated continuations (cross-tokenizer-friendly hard-label KD) instead of per-token logit matching.sequencemode is mutually exclusive with the v0.70 cross-tokenizer ULD logit path. - Classifier LoRA is live (#146).
task: classifier|reranker|cross_encodernow attaches a LoRA adapter to the sequence-classification head whenlorais configured, so a frozen encoder + small adapter can be trained instead of the full model. - LLaMA Pro block expansion is per-architecture (#148).
expand_layersnow interleaves zero-initialised identity blocks for Llama / Qwen / Mistral decoder stacks (was Llama-shaped only), withfreeze_trainable_layersfreezing the original blocks so only the new ones train. - LongLoRA S² shifted-sparse attention is live (#158).
use_longlora: truenow installs the shifted-sparse-attention forward override on the Q/K projections (Llama / Mistral / Qwen / Phi), restoring the patched forwards on context exit. - Mixture-of-Depths is live (#84).
use_mod: trueattaches a per-layer top-k token router (mod_capacity_factor) so only a subset of tokens receive each block's residual update. Architecture allowlist: Llama / Qwen / Mistral; unsupported bases warn and skip. - VeRA / VB-LoRA multi-tenant serving is live (#221).
soup serve --bank <bank.json> [--bank-strength S]reconstructs the shared projection + per-user scaling vectors and installs a decode-time forward hook; the active user is selected per request via theX-User-Idheader (an unknown/absent id is a zero-delta no-op, so there is no cross-request leak). Serves N personas at ~KB-per-user instead of a full LoRA each. - MoLE per-token adapter routing is live (#222).
task: moe_lora_routingwithmole_task_adapters: [...]trains a per-token gating network that blends N frozen task LoRAs (mole_top_k/mole_temperature); only the router trains. The gate is saved asmole_gate.ptalongside the run.
Changed
apply_bank_to_serve(#221) andbuild_gating_kernel(#222) now return live objects (aLoadedVectorBankand atorch.nn.Modulerouter) instead of the v0.67.0 deferred-stubNotImplementedError.
[0.71.11] - 2026-06-04
Added
- GRPO / RL callbacks — live wiring (closes #235, #236, #237, #238, #239, #240, #159, #160). The reward-hacking, cross-tokenizer distillation, MiniLLM, mid-epoch RL checkpoint, iterative-DPO and echo-trap surfaces that shipped schema-only in v0.70.0 are now real, validated end-to-end on SmolLM2-135M.
- Reward-hacking detector is live (#235).
--reward-hack-detector info_rm|rm_ensemblenow installs a GRPOTrainerCallbackthat reads the per-step rewards (via a shared, thread-safe reward-fn capture buffer), computes an InfoRM cluster-separation drop (info_rm) or RM-ensemble divergence (rm_ensemble), classifies OK/WARN/HACK, logs the verdict tostate.log_history, and halts training on HACK when--reward-hack-haltis set.rm_ensemblerequires ≥2 reward functions. - Cross-tokenizer ULD distillation is live (#236).
task: distillwith--uld-strategy wasserstein|topk_alignnow computes a real Wasserstein-1 (sorted-CDF) or top-k-aligned distillation loss inside the distill trainer, handling student/teacher vocab-size mismatch by clamping teacher ids to the teacher vocab. - MiniLLM reverse-KL distillation is live (#237).
--minillm-enabledadds a teacher-mixed, length-normalised reverse-KL term plus an optional pretrain-anchor SFT term (--minillm-pretrain-anchor-path/--minillm-pretrain-anchor-weight) that keeps the student near coherent language. The anchor corpus reader is cwd-contained + symlink-rejecting with a per-line byte cap. - Mid-epoch RL checkpoint is live (#238).
--rl-checkpoint-save-every-steps Nwrites a real adapter + optimizer state + JSON manifest every N steps during PPO/GRPO and prunes to--rl-checkpoint-keep-last, so a long RL run survives a crash without losing the optimizer momentum. soup iterative-dpoorchestrator is live (#239). Runs the full sample → reward-score → build-pairs → DPO-train loop across rounds: each round samples completions from the previous round's adapter, the next round trains a fresh LoRA from the base on that round's harvested pairs.--plan-onlystill renders the plan without running.- Echo-trap detector is live (#240).
--echo-trap-enabledinstalls a GRPO callback that scores per-trajectory n-gram repetition, classifies OK/WARN/TRAP against--echo-trap-threshold, logs the verdict, and halts on TRAP when--echo-trap-haltis set (catches RAGEN-style degenerate repetition in multi-turn agent RL). - GRPO variant fallback now warns once (#159). When a
--grpo-variantcustomcompute_lossfalls back to the base trainer (because the installed TRL renamed the loss inputs), the trainer logs a one-shot WARNING instead of silently degrading to the default objective.
Changed
- GRPO reference-model EMA no longer materialises full state dicts (#160).
--ref-model-ema-alphanow updates the reference model in place by iteratingnamed_parameters()(ref = (1-α)·ref + α·policy), eliminating the three model-sized allocations per step the v0.53.11 path made. A total name/shape-mismatch (0 shared parameters) logs a one-shot WARNING so a misconfigured EMA can't silently no-op.
[0.71.10] - 2026-06-03
Added
- RAG family — live wiring (closes #199, #200, #201, #202). The four retrieval / steering surfaces that shipped schema-only in v0.62.0 are now real, validated on SmolLM2-135M.
- RAFT span-mask training is live (#199).
data.format: raftrows ({query, golden_doc, distractor_docs, answer}) now train answer-only: the prompt span is masked to-100and each document is labelled[doc-N]so the model learns to cite the supporting document. Documents are shuffled reproducibly (data.raft_shuffle_seed). Rows whose prompt fillsmax_length(answer fully truncated) are dropped with a warning rather than silently shrinking the effective dataset. soup ra-dit— one-shot two-stage orchestrator (#200). Trains the retriever (stage 1, embedding/contrastive) then the generator (stage 2, RAFT-SFT) in a single command, recording the trained retriever as the generator's paired retriever. Asoup trainof a generator-stage config with no retriever model set now auto-links the most-recent RA-DIT retriever run from the Registry.--plan-onlyvalidates both configs without training;--retriever-modeloverrides the auto-link.soup steer train/apply+soup serve --steerare live (#201). Fit a CAA (contrastive activation addition), ITI (inference-time intervention) or RepE (representation-engineering PCA) control vector from{positive, negative}contrastive pairs, persist it as a safetensors + config artifact, and apply it at decode time via a forward hook (soup serve --steer <name> --steer-strength <s>).soup eval citation+ citation-span loss boost are live (#202). Score citation precision / recall / F1 over{predicted, expected_ids}or RAFT rows (--shuffle-seedaligns the golden[doc-N]id with what the model saw at train time). Whencitation_faithful: true, bracketed[doc-id]spans in the answer get a boosted per-token loss weight. A newcitationfailure mode is available insoup diagnose.
[0.71.9] - 2026-06-03
Added
- Knowledge edit + unlearn — live wiring (closes #193, #194, #196, #197, #203). The v0.61.0 / v0.62.0 schema-only stubs are now live, validated on SmolLM2-135M.
soup edit set(ROME / MEMIT / AlphaEdit) is live (#194). Newsoup_cli/utils/edit_kernels.pyships covariance-free rank-1 weight-edit kernels: ROME (single-layerW += δ·kᵀ/‖k‖²), MEMIT (residual distributed across a layer band), AlphaEdit (ROME update projected orthogonal to the down-proj's top singular direction).apply_editloads the model, optimises the target residual, applies the rank-1 update, and optionally saves with cwd-containment + symlink rejection.--output,--device,--governor/ --no-governorflags added. On a tiny model a ROME edit movedP("Lyon" | "The capital of France is")from 0.0016 → 0.96.soup edit difflive before/after generation (#194). Pass--before-model+--after-model(+--probes) to generate completions through both models and surface the probes whose output changed.- EditGovernor SQLite persistence + cross-process locking (#196). New
EditGovernorStore(mirrorsnamespace_pin.NamespacePinStore— $HOME/$CWD/$TMPDIR containment, TOCTOU symlink rejection, WAL + busy_timeout,fcntl/msvcrtsidecar lock, POSIX 0600).save_governor/load_governor/default_governor_db_path(env overrideSOUP_EDIT_GOVERNOR_DB) persist per-base-model edit-count + verdict across separatesoup edit setruns. apply_editconsults the EditGovernor automatically (#197). When a governor is supplied,check_can_edit()runs BEFORE the model load (refusing on norm blowup / edit cap) andrecord_edit()runs AFTER with the measured Frobenius delta.- Live GRACE codebook (#203).
GraceCodebook(epsilon-ball nearest-key lookup),apply_grace_edit(captures a residual key + optimises a value + appends to a codebook sidecar),save_codebook/load_codebook(atomic, cwd-contained, symlink-rejected),install_grace_hook(decode-time residual substitution). Newedited_model/grace_codebookRegistry artifact kinds. soup train --task unlearnis live (NPO / SimNPO / RMU) (#193). Newsoup_cli/utils/unlearn_kernels.py(NPO(2/β)·mean(-logσ(-β(πlp-reflp))), length-normalised SimNPO, RMU representation steering) + a self-containedUnlearnTrainerWrapperloop loading a LoRA policy, a frozen reference (NPO/RMU), and forget/retain JSONL datasets. NPO/SimNPO forget loss decreased on the tiny-model smoke. Warns when run without a retain set.
Security
_save_edited_model/UnlearnTrainerWrapperoutput dirs +save_codebook/load_codebook+_load_unlearn_rowsenforce cwd-containment, raw-path symlink rejection (TOCTOU), null-byte rejection, and file-size / per-line caps.apply_grace_edithonours the governor for direct callers.
[0.71.8] - 2026-06-03
Added
- Probes & SAE — real weights + live downloads (closes #215, #216, #217,
#218, #219). A new shared
soup_cli/utils/probe_kernel.pyprovides the linear-probe math (contrast-pair derivation, apply, flag-rate, verdict bands, operator-supplied weight loading, deterministic synthetic fallback); every heavy import (numpy/torch/safetensors) is lazy. soup probe sleeper --weights <w.npz|.npy|.safetensors>(#215) — load a real calibrated probe direction instead of the synthetic fallback. Weights are cwd-contained, symlink-rejected,O_NOFOLLOW-opened,allow_pickle=False, and size-capped.compute_contrast_probe(positive, negative)derives a probe from contrast-pair activations.soup probe sae-diff <repo> --auto-download(#216) — fetch an allowlisted SAE from the HF Hub into~/.soup/sae-cache/(validated againstHF_HUB_ALLOWLISTBEFORE any network call) via a new SSRF-hardenedsoup_cli.utils.hubs.snapshot_download(repo-id shape + home/cwd/tmp cache containment + namespace-pin TOFU gate).soup probe truth/soup probe harm(#217) — TruthfulQA-style honesty and HarmBench-style misuse activation probes (6 bundled bases each, 5% / 20% verdict bands,--weightsto skip the allowlist with a real probe). The probe pack now ships truth + harm entries per base.soup probe interference --measure <eval_suite> --base-model <m> --adapter name=path ...(#218) — auto-measure the N×N interference matrix by actually loading the base + each LoRA adapter (PEFT multi-adapter), measuring loss for each adapter alone (diagonal) and each co-loaded pair (add_weighted_adapter(combination_type="cat"), off-diagonal). Exit 2 on a MAJOR worst-pair.soup train --capture-activations <layer> --capture-prompts <jsonl>(#219) — a post-training hook writes an SAE-diff-ready per-token activation snapshot to<output>/activations/activations.json.resolve_layer_moduleresolves the samemodel.layers.Npath whether or not a LoRA adapter is loaded (PEFT-wrapper fallback).
Security
- Probe / SAE / capture file I/O is cwd-contained +
O_NOFOLLOW(TOCTOU close)- size-capped; SAE weight loads use
allow_pickle=False. SAE auto-download validates the allowlist before any network call and rejects a glob result that resolves outside the snapshot dir (symlink-escape guard).
- size-capped; SAE weight loads use
Notes
- #215 is partial: the operator-supplied / contrast-pair / synthetic paths ship now, but the 6 large-base Anthropic-calibrated probe vectors remain upstream-gated (no public calibrated artifact exists). Documented as a known limitation.
[0.71.7] - 2026-06-02
Added
- Eval live runners — six probe surfaces that previously emitted heuristic
/ neutral stubs now load a real model and run live (closes #161, #162, #208,
#211, #212, #165). New shared
soup_cli/utils/live_eval.pyprovides the model-loading primitives (generator / multi-generator closures, masked cross-entropy eval-loss, a short-LoRA probe, and held-out logit agreement); every heavy import (torch/transformers/peft/lm_eval) is lazy. soup advise --probe-model <id>— runs a LIVE ROI probe: zero/few-shot token-F1 baselines, a short LoRA probe (relative held-out-loss improvement + real wall-clock), and base-model proximity (held-out logit agreement) folded into the dataset profile. Without--probe-model,--probestays the offline heuristic.soup tunability --live— replaces the offline heuristic with a real per-candidate LoRA probe (loads eachrepo_id, trains--probe-stepson a held-out-excluded slice, reports the held-out-loss drop).soup eval capability --live --model <id>— invokes lm-eval-harness per resolved task (or a--tasksoverride) with--limit/--device, isolating per-task failures and surfacing a no-metric result as an explicit error.soup eval behavior --base-model <id> [--adapter <path>]— generates pre/post responses on the bundled behaviour battery and scores the live diff.soup diagnose --base-model <id> [--adapter <path>] [--dataset <jsonl>] [--tokenizer <id>]— runs all six failure-mode probes (forgetting / refusal / format / mode_collapse / memorization / contamination) live viasoup_cli.utils.diagnose.live.run_live_diagnose; falls back to neutral OK or--evidenceJSON when no model is supplied.
Security
- The two new JSONL dataset readers (
diagnose.live._load_dataset_rows,tunability._load_jsonl_rows) open withO_NOFOLLOWafter the cwd-containment check, closing the check→open TOCTOU window (matches the v0.65 / v0.67 reader policy).
[0.71.6] - 2026-06-02
Added
soup buildlive runner — the dbt-for-SFT DAG (soup build <manifest>) now materialises datasets instead of only dry-running the plan. Five built-in transforms ship live (identity,drop_empty,lowercase,strip,dedup_exact);tablerebuilds from scratch,viewre-derives on every run, andincrementalre-transforms only the rows whose content hash changed (tracked in a SQLite state store, keyed by row hash and the model's transform+config fingerprint so a transform change re-runs everything). Custom transforms are passed per-run via the Python API'stransforms=map. Outputs are written atomically; the--output-diris symlink-checked before any directory is created.soup data gen-magpielive generator — the Magpie synthetic generator (Xu et al. 2024) now actually generates. It feeds an aligned model its chat-template prefix (chatml / llama3 / gemma / mistral families auto-detected) and harvests the self-generated user instruction + assistant response via raw completion. Live providers:ollama(/api/generateraw) andvllm(/v1/completions) — both SSRF-hardened (loopback-only HTTP);anthropicis rejected (no raw-completion endpoint). Optional--quality-filterdrops low-quality rows via the v0.47 toxicity/educational scorers; exact-duplicate instructions are de-duplicated.soup eval irt-subset --model {1pl,2pl,3pl}— the IRT eval-cost optimiser gained 2PL (per-item discrimination) and 3PL (+guessing floor) joint coordinate-ascent MLE fits alongside the existing 1PL Rasch.1plkeeps the closed-form path for back-compat;2pl/3plroute through the newfit_irt.- Tokenizer-aware memorization probe —
score_memorization(..., tokenizer=...)andsplit_prefix(..., tokenizer=...)(used bysoup diagnose) now split the prefix/suffix on real token-id boundaries and measure echo-overlap over sub-word tokens when a tokenizer (HF id / path / duck-typed object) is supplied, catching BPE-level memorization that whitespace tokenisation misses. Default (no tokenizer) keeps the whitespace behaviour.
Fixed
soup data augment --provider ollama|vllmno longer crashes — the command imported a non-existentOllamaProvidersymbol and raisedImportErroron every non-OpenAI provider. It now routes through the shared, SSRF-hardened provider factory;--model/--base-urlare honoured, the output path is containment- and symlink-checked, and the write is atomic.
Security
- Ollama / vLLM provider URLs reject
0.0.0.0—validate_ollama_url/validate_vllm_urldropped the bind-any wildcard from their loopback allow-set (nowlocalhost/127.0.0.1/::1only), matching the newervalidate_hub_endpoint/validate_webhook_urlSSRF validators. Reachable now that Magpie threads a user-supplied--base-urlthrough these providers.
[0.71.5] - 2026-06-02
Added
soup eval againstnow reads eval metrics —ExperimentTracker.get_metric_seriesfalls back to theeval_resultstable when the metric is not a per-step training column (loss/lr/grad_norm/speed/gpu_mem). Sosoup eval against <base> --candidate <run> --metric task_accuracyreturns a real score series (benchmark scores live ineval_results, notmetrics) instead of "Empty series". Per-step columns still read frommetrics— no regression for existing callers.soup adviselearns from past project outcomes —soup advisenow reads this project's accepted-verdict history (~/.soup/advise_history.jsonl) and biases the rubric: 3+ successful SFT precedents flip a marginal RAG call to SFT; 3+ negative GRPO outcomes suppress GRPO in favour of SFT-on-traces; an encouraged choice gets a small confidence nudge. Scoped per-project (one project's record never biases another). No history → identical to before.- Slack/Discord webhooks on four more commands —
--slack-url/--discord-url(SSRF-hardened, loopback-only HTTP, RFC1918 rejected, never crashes the command) now ship onsoup ingest,soup prune-prompt,soup ab(fires only on areject_h0/accept_h0decision, notcontinue), andsoup data active-sample— not justsoup drift-alarm. The validator + sender moved to a sharedsoup_cli/utils/webhooks.py. - Tokenizer-aware
soup prune-prompt—--tokenizer <model_or_path>detects and strips the shared system-prompt prefix on token boundaries instead of characters, so a multi-byte UTF-8 prefix can never be split mid-code-point. Default (no--tokenizer) keeps the whitespace-character behaviour. - Curriculum bucketing by loss percentile —
DynamicCurriculumCallbacknow buckets samples by the percentile rank of the live loss (or perplexity) signal within a rolling window whendata.curriculum_metricisloss/perplexity, so a consistently-hard sample is routed to the same difficulty bucket across recomputes.lengthand warm-up still use round-robin. --hubonsoup data pushandsoup data forge—soup data push --hub modelscope|modelersuploads a dataset via the matching SDK (repo_type=dataset, commit message sanitised);soup data forge --hub <non-hf> --teacher owner/namepre-fetches the teacher model from that hub (and warns when the teacher is not a repo id so--hubis never silently ignored). HF stays the default.
Notes
- Live SaaS pull adapters for
soup ingest(Langfuse / LangSmith / Helicone / OpenPipe / OpenAI SDKs, issue #204) remain deferred: they need credentialed vendor accounts with populated trace data to validate honestly. Tracked as an open,infra-blocked(external-account) item.soup ingestcontinues to parse the JSONL export you pull from your dashboard.
[0.71.4] - 2026-06-02
Added
- Live canary verdict for
soup adapters merge—--canary <suite.json>scores the merged adapter against the first input and classifies OK / MINOR / MAJOR using the Quant-Lobotomy taxonomy (drop <2% OK, <5% MINOR, else MAJOR).--strict-verdictexits 2 on MAJOR. Pre-scored{"baseline_scores","candidate_scores"}suites run with no model load; a{"tasks":[...]}suite uses an injectable scorer. Replaces the v0.57UNKNOWNstub. - Live evolutionary merge —
soup adapters merge --strategy cmaes --eval <suite> --budget <t>now runs the full CMA-ES loop: each candidate is merged, materialised, scored against the eval suite, and the best-weighted merge is written to--output. Replaces the v0.67 plan-only stub. - Publish an adapter PR to GitHub —
soup adapters pr <title> --base-sha <hex> --adapter <path> --push owner/repo#Nposts the rendered PR Markdown as a GitHub PR comment viagh api(argv-list, body over JSON stdin; no shell). Token resolves fromGITHUB_TOKEN/GH_TOKEN. - Pre-wired
soup loopproduction stages —soup loop init --pre-wired(orsoup loop watch --pre-wired) swaps the v0.58 no-op stage stubs for real harvest (traces → preference pairs) → DPO train → eval-gate → canary-deploy callables.soup loop statusnow shows thepre_wiredflag. - Loop iterations as Soup Cans + Registry lineage —
soup loop watch --pack-canspacks each successful iteration as a v0.26 Soup Can and appends a Registry entry (tagloop-iter), chaining a real lineage DAG across iterations visible throughsoup history. `soup loop replay --extract` unpacks a recorded iteration. - Branch pointers into the Registry —
soup adapters branch <name> --attach-to-registry <id>links a branch snapshot to a Registry entry (shown as abranchesnode insoup history);soup adapters branch <name> --from-registry <id>derives a fresh snapshot's config + base from an entry.
Security
- The backdoor-scan gate (v0.71.2 #192) and license-conflict gate (v0.60 Part E)
now run for all merge strategies, including
--strategy cmaes(previously bypassed because cmaes returned before the gates). soup loopcanary deploy restrictsSOUP_LOOP_SERVE_ENDPOINTto loopback / RFC1918-private hosts (a serve endpoint is the operator's own box/LAN), beyond the general webhook SSRF policy which permits any HTTPS host.soup adapters pr --pushbuilds theghchild environment from an allowlist so unrelated secrets (HF_TOKEN/OPENAI_API_KEY/ …) never reach the subprocess.- The canary-suite JSON read uses
O_NOFOLLOW+os.fstat(size cap enforced on the same fd) to close the symlink/size-cap TOCTOU window.
[0.71.3] - 2026-06-01
Added
- Energy & CO2 measurement for training —
soup train --track-energywraps the training window in a codecarbon offline tracker (no IP-geolocation network call) and reports kWh / CO2 / grid intensity, feeding those numbers into--annex-xi. NewEnergyTrackercontext manager; graceful no-op when codecarbon is absent (pip install soup-cli[carbon]).--energy-countrypicks the ISO-3166 alpha-3 grid for the CO2 estimate (defaultUSA). - PDF Annex XI/XII documents —
soup train --annex-xi report.pdfnow renders a reportlab PDF (a.mdpath still renders markdown).pip install soup-cli[pdf]. - Auto-populated training-corpus domains in Annex XI/XII — the top crawled domains (with shares) are now extracted from the training JSONL and listed in the EU AI Act docs, replacing the previous empty placeholder.
- Soup Can manifest v3 with embedded attestations —
soup can pack --attest <statement.json>(repeatable) embeds in-toto Statements into a v3 can manifest; v1/v2 cans still load. Each statement is shape- and size-validated. - Local audit log auto-instrumentation — every
soupcommand now appends one HIPAA/SOC2-shaped record to~/.soup/audit.jsonl(secrets redacted, args capped). Opt out per-invocation with--no-audit-logor globally withSOUP_NO_AUDIT_LOG=1. Tail/rotate withsoup audit-log. - Reproducibility receipt in airgap bundles —
soup airgap-bundle --repro-receipt <receipt.json>embeds an SR 11-7 receipt asrepro-receipt.json; auto-detected from<model>/repro-receipt.jsonwhen not supplied.
Security
soup can pack --attestnow rejects oversize attestation files by their raw size before parsing them into memory (defence against memory-exhaustion).- The new file-loading paths (attestation JSON, airgap receipt, training-corpus
scan, PDF write) are all cwd-contained + TOCTOU symlink-rejected and
size-capped; the audit auto-log redacts
hf_/sk-/Bearertokens and never crashes the CLI on a broken log.
[0.71.2] - 2026-06-01
Added
- ed25519 signing for
soup adapters sign/soup attest— real detached signatures (over the adapter Merkle root / the in-toto statement) via a new[sign]extra (pip install soup-cli[sign], pullingcryptography).soup adapters sign --backend ed25519 --key <priv.pem>(or--generate-key <out.pem>, orSOUP_SIGNING_KEY);soup adapters verify [--public-key <trusted.pem>]does a cryptographic verify and, with a trusted key, genuine authentication.soup attest emit --sign ed25519 --key <priv.pem>writes a<output>.sigsidecar; newsoup attest verify <statement> --signature <sig>verifies it (canonical-JSON, so it's platform/newline-independent). Sigstore keyless signing stays infra-blocked (needs an OIDC provider + Fulcio/Rekor network — can't be honestly validated offline). - Anti-AI-Jacking namespace pin on Hub downloads — HF model fetches now
consult a trust-on-first-use pin store: a repo whose author changes (or whose
created_atjumps backward) is refused unless the namespace shift is explicitly allowed. Fails open when repo metadata is unavailable. - License auto-detection at
soup adapters merge— when--licenseisn't given, the license is read from each adapter'sadapter_config.json/config.json/ model-card frontmatter (HFllama3.1-style ids mapped to canonical) and the conflict gate runs automatically. - Backdoor-scan gate at
soup adapters merge— refuses to merge any input whosesoup adapters scanreturns FAIL (or can't be scanned) unless--allow-unscannedis passed; WARN is advisory.
Changed
- License-conflict overrides (
--license-override <reason>) are now recorded to the audit log for legal review. - The namespace-pin store now uses SQLite WAL + busy-timeout and a cross-process file lock around its get+insert, so concurrent writers don't lose the trust anchor.
Security
- ed25519 verification fails closed (any tamper / wrong key / missing key ⇒
invalid). Signing keys + trusted public keys are symlink-rejected and
size-capped via a shared reader (no cwd-containment — keys live outside the
project).
--generate-keyrefuses to overwrite any existing path.
[0.71.1] - 2026-06-01
Added
soup env fix— render a reproducible install plan fromsoup-env.lock. Emits copy/pasteuv pip installcommands (--format uv-pip, default) or arequirements.txtbody (--format requirements);--outputoptionally writes arequirements.txtunder cwd. Print-only by design — never shells out to a package manager.soup lock write --env-lock <path>— auto-derive--env-hashfrom asoup-env.lockso operators who ransoup env lockdon't copy the hash by hand.--env-hashstill wins when passed explicitly.soup serve --record-thumbs <db>— capture thumbs-up/down feedback into a local-RL SQLite at startup, plus a newPOST /v1/thumbsendpoint (transformers backend). Returns 404 when the flag isn't set.- Judge-calibration persistence:
JudgeCalibrationReport.to_dict,write_judge_calibration, andload_judge_calibration, backed by a newjudge_calibrationregistry artifact kind. Loading re-validates the report so a corrupt on-disk field is rejected. - Bundled MUSE and WMDP unlearning eval fixtures so
soup eval unlearning --benchmark muse|wmdpruns out of the box. WMDP forget-set probes ship redacted (placeholder prompts +REFUSEDresponses) — Soup never ships verbatim hazardous content.
Changed
soup completionsnow introspects a cached base model's actual LoRA target modules (config-onlyAutoConfigload,local_files_only=True, never networks or raises) and falls back to the canonical default shape when the base isn't cached locally.build_dagexposes avalidate_build_sourcehelper (cwd-containment + symlink rejection) for build-manifest source paths.
0.71.0 - 2026-06-01
Changed
- Breaking — install split. The heavy training stack (
torch,transformers,peft,trl,datasets,bitsandbytes,accelerate) moved out of the core install into a new[train]extra.pip install soup-cliis now a light CLI + data-tools install with no PyTorch; runpip install 'soup-cli[train]'(or[all]) to fine-tune. Existing users who train must reinstall with[train]. Version pins are unchanged. - Trimmed
README.mdto a ~238-line front door; the full feature reference now lives underdocs/(one topic page per area, indexed from the README). - Raised the pytest coverage gate from 50% to 77% (
--cov-fail-under=77). - Migrated to a
src/layout (src/soup_cli/) for cleaner packaging and to stop tests accidentally importing the in-tree package.
Added
[train]and[all]optional-dependency extras ([all]pullstrain,serve,ui,data).[dev]self-references[train]so CI and contributors still get the full stack frompip install -e ".[dev]".- Friendly error mapping: a missing heavy dependency (
torch,transformers,peft,trl,datasets,bitsandbytes,accelerate) now surfaces "Training needs the [train] extra. Run: pip install 'soup-cli[train]'". py.typedmarker (PEP 561) so downstream type checkers pick up Soup's inline type hints..pre-commit-config.yamlwith ruff (lint + format) and standard file-hygiene hooks.- Lenient
mypyconfiguration and a non-blockingtype-checkCI job. - This
CHANGELOG.md.
Removed
- The historical, per-version security-fix log that had grown inside
SECURITY.md(~220 KB).SECURITY.mdis now a concise security policy; the detailed hardening notes remain in git history and the GitHub Releases notes.