Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Contributing to Soup
Thank you for your interest in contributing to Soup! We welcome bug reports, feature requests, and pull requests from the community.
Getting Started
1. Fork & Clone
git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
2. Set Up Development Environment
Requirements: Python 3.11, 3.12 or 3.13 (requires-python = ">=3.11,<3.14"; 3.10 was
dropped in #1805). CI runs exactly those three; tests/test_requires_python_bound.py derives
the declared bound from the CI matrix, so widening one without the other fails the suite.
Work inside a virtualenv, then install the project in editable mode with dev dependencies:
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
The venv step is not optional on Debian 12, Ubuntu 23.04 or later: without it
pip refuses with error: externally-managed-environment, because those distros
stop pip writing into the system Python that apt manages
(PEP 668). pipx is the answer for
installing Soup and the wrong tool here, since an editable checkout needs to be
importable by the test suite rather than isolated from it.
This installs:
pytestfor testingrufffor linting + formattingpytest-covfor coveragehttpxfor HTTP testingmypyfor type checkingpre-commitfor the lint/format git hooks- the full training stack —
[dev]self-references[train], sotorch,transformers,peft,trl,datasets,bitsandbytes,accelerateare pulled in too (since v0.71.0 these are an opt-in extra, not core deps) cryptography—[dev]also pulls it in (it's the[sign]extra) so the ed25519 signing tests (soup adapters sign/soup attest) run in CIreportlab—[dev]also pulls it in (it's the[pdf]extra) so the Annex XI/XII PDF tests (soup train --annex-xi *.pdf) run in CI
3. Verify Setup
Run the test suite to confirm everything works:
pytest tests/ -v --tb=short
Run the linter:
ruff check src/soup_cli/ scripts/ tests/ benchmarks/
If both pass — you're ready to contribute!
Code Style
We use ruff for all code style and linting. Before committing, run:
# Check for issues
ruff check src/soup_cli/ scripts/ tests/ benchmarks/
# Auto-fix issues
ruff check --fix src/soup_cli/ scripts/ tests/
Style Guidelines
- Line length: 100 characters (enforced by ruff)
- Imports: Sorted and organized (ruff I rule)
- Naming: No single-letter variable names (ruff E741) — use
entry,part,lengthinstead ofl,p, etc. - Lazy imports: Heavy dependencies (torch, transformers, peft, trl, etc.) should be imported inside functions, not at module level, to keep the CLI responsive
- No third-party licence headers. This project is Apache-2.0 and ships to PyPI, so an
SPDX-License-Identifieror copyright line naming anyone else is a licensing claim we cannot make. If you generate files from a template — including with automated tooling, which is how this has happened three times — strip its header before you push.tests/test_no_foreign_license_headers.pyfails the build on this rather than relying on you or me to spot it in a diff. - Config validation: Always use Pydantic v2 with
BaseModelandField - Output: Use
rich.console.Consolefor all output — never bareprint() - Type hints: Always include type hints for function parameters and return values
Example:
# WRONG
from torch import cuda
import transformers
def train():
print("Starting training")
model = transformers.AutoModel.from_pretrained("llama-7b")
# CORRECT
def train():
from torch import cuda
import transformers
console = Console()
console.print("Starting training")
model = transformers.AutoModel.from_pretrained("llama-7b")
Project Structure
Key directories:
src/soup_cli/
cli.py - Main entry point, command routing
commands/ - Command implementations (train, chat, eval, deploy, etc.)
config/ - Config schema (schema.py) and loader (loader.py)
data/ - Data loading, format conversion, providers, templates
trainer/ - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding, BCO, Preference, Distill, Classifier, PRM, Unlearn, MoLE Routing)
monitoring/ - Callbacks and live dashboard
experiment/ - SQLite experiment tracking
eval/ - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
bench/ - Benchmark instruments behind `soup bench train` (collector, train_report, train_run) (#836)
migrate/ - Config migration (LLaMA-Factory, Axolotl, Unsloth)
recipes/ - Ready-made configs for popular models (171 recipes)
autopilot/ - Zero-config decision engine (v0.25.0)
registry/ - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
cans/ - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
data/traces/ - Trace-to-Preference harvester (v0.26.0)
data/collators.py - CrossDocCollator for sample packing (v0.33.0)
utils/ - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth, reward_stress, layer_stream, layer_shard, layer_stream_runtime
templates/ - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
ui/ - Web UI (FastAPI + HTML/JS SPA)
tests/ - Test suite covering all CLI commands, trainers, and model architectures
examples/ - Real-world config examples and datasets
Running Tests
All Tests
pytest tests/ -v --tb=short
Single Test File
pytest tests/test_config.py -v
Single Test
pytest tests/test_data.py::test_detect_alpaca_format -v
With Coverage
pytest tests/ --cov=soup_cli --cov-report=html
GPU Tests
Every CI runner is GPU-less, so the tests that need a CUDA device skip there and
only mean something on a real card. They carry the gpu marker (#833):
pytest tests/ -m gpu --no-cov -v
--no-cov matters: the default --cov-fail-under gate is computed over the whole
package and fails a subset run. To gate a new test on CUDA, decorate it with
@pytest.mark.gpu (or @pytest.mark.gpu(reason="...")); do not write your own
torch.cuda.is_available() check. tests/conftest.py holds the one probe, and
tests/test_issue833_gpu_marker.py fails on a private one. Use
tests.conftest.cuda_available() only to pick a device, never to skip.
If you have a card, a run is a real contribution: both recorded runs so far found defects CI could not see. Open an issue (or comment on #833) with:
- the card, its compute capability (
torch.cuda.get_device_capability()), and VRAM - the driver version, and the torch build (
torch.__version__,torch.version.cuda) - the
bitsandbytesversion, if installed - the commit you ran (
git rev-parse HEAD) - the summary line (
N passed, N failed, N skipped), and the name and first error line of every failure
Accepted runs are recorded in benchmarks/gpu-test-runs.md.
Test Files
A representative sample of the suite below. Run
pytest tests/ -vfor the complete list.
Name a new test file after the module or feature it covers (test_<module>.py), not after
an issue number or a version. Existing test files keep their names.
| File | Covers |
|---|---|
| test_config.py | Config loading, validation, defaults |
| test_data.py | Format detection, conversion, validation |
| test_gpu.py | GPU detection, batch size estimation |
| test_cli.py | CLI commands, version --full |
| test_tracker.py | SQLite experiment tracker |
| test_runs.py | soup runs CLI commands |
| test_data_tools.py | Data convert/merge/dedup/stats commands |
| test_eval.py | Eval command |
| test_smoke_train.py | Full pipeline smoke tests (GPU) |
| test_chat.py | Chat command, _detect_base_model |
| test_push.py | Push command, _format_size, _generate_model_card |
| test_init.py | Init command, templates, overwrite logic |
| test_callback.py | SoupTrainerCallback (mock-based) |
| test_display.py | TrainingDisplay rendering |
| test_loader.py | Data loading (JSONL/JSON/CSV, edge cases) |
| test_validator.py | validate_and_stats, extended_stats, _percentile |
| test_formats.py | Reverse conversion, round-trips, edge cases |
| test_merge.py | Merge command, adapter detection, validation |
| test_export.py | Export command, GGUF quant types, validation |
| test_resume.py | Resume checkpoint resolution, W&B flag |
| test_serve.py | Serve command, FastAPI app, endpoints, streaming |
| test_generate.py | Data generate, JSON parsing, validation, prompts |
| test_sweep.py | Sweep params parsing, combinations, nested config |
| test_diff.py | Diff prompts collection, metrics, CLI |
| test_deepspeed.py | DeepSpeed configs, multi-GPU detection, trainer integration |
| test_errors.py | Friendly error messages, --verbose flag, error mapping |
| test_doctor.py | soup doctor command, version checking, system resources, dependency table |
| test_quickstart.py | soup quickstart demo, data/config creation, --dry-run |
| test_grpo.py | GRPO config, rewards, data prep, template, sweep shortcuts |
| test_progress.py | Rich download progress bar, _enable_hf_transfer_progress |
| test_unsloth.py | Unsloth backend config, detection, trainer integration, templates |
| test_vision.py | Vision modality config, LLaVA/ShareGPT4V formats, loader, trainer, templates |
| test_qat.py | QAT config, validation, trainer integration, export compatibility |
| test_ui.py | Web UI command, FastAPI endpoints, auth, static files, config validation |
| test_vllm_serve.py | vLLM backend detection, engine creation, serve --backend flag, FastAPI app |
| test_ppo.py | PPO config, reward model config, data prep, RLHF template, routing, sweep |
| test_kto.py | KTO config, data format, template, routing, sweep, train guard, wizard |
| test_orpo.py | ORPO config, template, routing, sweep, train guard, wizard |
| test_simpo.py | SimPO config, template, routing, sweep, train guard |
| test_ipo.py | IPO config, template, routing, sweep, train guard |
| test_advanced_peft.py | DoRA, LoRA+, GaLore config, validation, sweep shortcuts |
| test_infer.py | Batch inference command, prompt reading, CLI validation |
| test_tensorboard.py | TensorBoard flag, wandb conflict, report_to routing |
| test_pretrain.py | Pretrain task, plaintext format, MoE config, templates, routing |
| test_moe.py | MoE detection, ScatterMoE LoRA targets, MoE info extraction |
| test_bugfixes.py | v0.10.1-v0.14.3 regression fixes |
| test_cli_subprocess.py | Subprocess CLI tests: entry point, encoding, paths, platform regressions |
| test_performance.py | Liger Kernel, FlashAttention, FSDP2, Ring Attention, long-context, RoPE scaling |
| test_embedding.py | Embedding task config, format, template, routing, sweep, pooling |
| test_onnx_tensorrt_export.py | ONNX export, TensorRT-LLM export, format support |
| test_speculative_decoding.py | Speculative decoding CLI, draft model, vLLM integration |
| test_server_generate.py | Server provider for data generate, SSRF validation |
| test_quality_filter.py | Perplexity + coherence scoring, soup data filter |
| test_audio.py | Audio modality config, format, template, routing, loader |
| test_sglang_serve.py | SGLang backend detection, runtime creation, serve --backend |
| test_deploy_ollama.py | Ollama deploy, Modelfile gen, template mapping, security validation |
| test_eval_platform.py | Custom eval, judge, human eval (Elo), leaderboard, compare, auto-eval, security |
| test_synth_data_pro.py | Providers (Ollama, Anthropic, vLLM), templates, quality pipeline, SSRF |
| test_migrate.py | LLaMA-Factory/Axolotl/Unsloth migration, path traversal, round-trip validation |
| test_recipes.py | Recipe catalog, search, CLI (list/show/use), path traversal |
| test_neftune_rslora.py | NEFTune config/validation/sweep, rsLoRA config/validation/sweep |
| test_profile.py | Training profiler: memory estimation, speed, GPU recommendations, CLI |
| test_multi_adapter.py | Multi-adapter serving: validation, parsing, FastAPI endpoints, CLI |
| test_data_sample.py | Data sampling: random/diverse/hard strategies, CLI, edge cases |
| test_adapters.py | Adapter management: list/info/compare, discovery, metadata |
| test_export_removed_formats.py | AWQ/GPTQ removal verification: rejection at CLI, pyproject extras, codebase clean |
| test_packing.py | Sample packing: config, YAML, trainer integration, sweep |
| test_data_split.py | Data split: ratio/absolute/stratified splits, seed, edge cases |
| test_curriculum.py | Curriculum learning: config, length sort, buckets, sweep |
| test_dataset_hub.py | HF dataset search, preview, download, format conversion, security |
| test_freeze_training.py | Freeze training: config, layer freezing, GPT-2 naming, sweep |
| test_loss_watchdog.py | Loss watchdog: config, callback behavior, patience, sweep |
| test_dataset_registry.py | Dataset registry: CRUD, CLI, name validation, error handling |
| test_tool_calling.py | Tool-calling format detection, normalization, eval scoring, recipes (v0.25.0) |
| test_rlvr.py | RLVR verifiable rewards: math_verify, code_exec sandbox, json_schema (v0.25.0) |
| test_peft_methods.py | VeRA + OLoRA LoraConfig, peft_builder, sweep integration (v0.25.0) |
| test_mlx_backend.py | Apple Silicon MLX backend: detection, trainers, routing (v0.25.0) |
| test_data_augment.py | Data augmentation: rephrase/translate/style strategies, CLI, security (v0.25.0) |
| test_training_intelligence.py | Forgetting detection + checkpoint intelligence + SQLite (v0.25.0) |
| test_autopilot.py | Autopilot: analyzers, decision engine, CLI (v0.25.0) |
| test_registry.py | Model Registry: hashing, store CRUD, artifacts, lineage DAG, diff, CLI, history (v0.26.0) |
| test_eval_gate.py | Eval-Gated Training: config, suite loading, baseline, callback, CLI (v0.26.0) |
| test_trace_to_pref.py | Trace-to-Preference: LangChain/OpenAI/Soup-serve parsers, pair builder, CLI (v0.26.0) |
| test_quant_check.py | Quant-Lobotomy: classify_delta, resolve_model_ref, render formats, CLI (v0.26.0) |
| test_cans.py | Soup Cans: manifest schema, pack/unpack, tar traversal, fork security, CLI (v0.26.0) |
| test_multi_gpu.py | Multi-GPU Mastery: topology, --gpus, accelerate launcher, ZeRO++, FSDP2+compile, pipeline (v0.27.0) |
| test_training_speed.py | Training Speed & Memory: CCE, FP8, grad-ckpt tiers, kernel picker, cross-doc attn, activation offload (v0.28.0) |
| test_hf_integration.py | HF Hub Deep Integration: token/endpoint/repo_id, auto-push callback, model card v2, collections, data push, HF Spaces, private-IP SSRF (v0.29.0) |
| test_inference_advanced.py | Inference Excellence: prefix caching, spec-decoding auto-pairing, LoRA hot-swap, structured output, dashboard + /metrics, OpenTelemetry tracing (v0.30.0) |
| test_recipes_v031.py | Model & Recipe Breadth: 34 new recipes (vision/audio/reasoning/edge/domain/multimodal); catalog-wide invariants; CI workflow validation (v0.31.0) |
| test_auto_tuning.py | Training Stability & Auto-Tuning: LR range finder, grad-accum monitor, auto mixed-precision, auto warmup, spike recovery, convergence detector, autopilot wiring (v0.32.0) |
| test_part_f_hardening.py | Live Wire hardening: RLVR OS-level sandbox isolation + prune_checkpoints TOCTOU (v0.33.0) |
| test_part_a_wave1.py | Live Wire: live eval-gate scoring + registry attach (v0.33.0) |
| test_part_a_wave2.py | Live Wire: soup can run / publish + DeployTarget schema (v0.33.0) |
| test_part_e.py | Live Wire: --find-lr live loop + spike recovery hint + auto mixed-precision push + grad-accum advisory (v0.33.0) |
| test_part_d.py | Historical Live Wire primitives: structured-output LogitsProcessor + HF push integration smoke (v0.33.0) |
| test_part_c.py | Live Wire: multi-trainer v0.28.0 features + selective ckpt hooks + CrossDocCollator (v0.33.0) |
| test_part_b.py | Live Wire: auto-reexec under accelerate launch + DeepSpeed-MII live serve (v0.33.0) |
| test_log_level.py | Smart logging tiers --log-level quiet/normal/verbose/debug (v0.34.0) |
| test_run_cost.py | Per-run cost: GPU-rate lookup + estimate + format + tracker integration (v0.34.0) |
| test_why.py | soup why heuristic explainer: NaN / plateau / divergence / grad-norm / LR bounds (v0.34.0) |
| test_crash_reporter.py | .crash bundle: classify, redact secrets, write under cwd, oversize truncation (v0.34.0) |
| test_replay.py | soup runs replay: summarise + downsample + CLI rendering (v0.34.0) |
| test_profiling.py | Auto-profiling: ProfilerSchedule + path containment + torch-less degradation (v0.34.0) |
| test_tui.py | soup tui: CLI bounds + missing-textual error + row builders (v0.34.0) |
| test_trainer_coverage_v035.py | Multi-trainer v0.28.0 wiring smoke matrix: every trainer × every speed/memory feature + benchmark_kernel_combos + schema-gate lift (v0.35.0 — #60, #61, #45) |
| test_v0470_part_a.py | Synthetic Data Forge: ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses + VALID_TASKS allowlist + chunk_document + score_uncertainty + discover_documents (cwd-contained, symlink-rejecting, extension allowlist) + build_forge_plan validators (task / target_rows / teacher / NaN+Inf threshold) + synthesise_forge_rows (judge-exception swallow at DEBUG) + write_forge_dataset + write_provenance (atomic, TOCTOU-safe) + CLI smoke (v0.47.0) |
| test_v0470_part_b.py | Data Quality Moat: BENCHMARKS MappingProxyType + ScoreReport frozen + ngram_set / ngram_overlap_ratio / decontaminate_rows (containment ratio) + detect_pii (ReDoS-hardened regexes + 50 KB pre-cap) + detect_language (6-language stopword heuristic) + score_toxicity (keyword baseline) + score_educational_value + compute_scorecard + load_jsonl_rows / write_jsonl_rows (cwd-contained, symlink-rejecting, atomic) + CLI smoke per subcommand (v0.47.0) |
| test_v0480_part_a.py | Curriculum-Aware Trainer (BETA): DynamicCurriculumPolicy frozen + bounds; compute_bucket_weights softmax + water-fill (floor-strict invariant); validate_distributed_curriculum DDP gate; render_curve + parse_history_jsonl with 100k-row DoS cap; SoupConfig cross-validators (requires-curriculum / mlx-rejected / non-SFT-rejected / floor ≤ 1/buckets); soup runs curriculum-curve CLI (TOCTOU symlink reject + 50 MB cap + corrupt-JSONL exit 2) (v0.48.0) |
| test_v0490.py | v0.49.0 Long Context & Architecture: YaRN math kernels (yarn_find_correction_dim / yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale — bool/NaN/Inf rejection, auto-disambiguated low==high); YaRN schema fields (yarn_factor / yarn_attn_factor / yarn_beta_fast / yarn_beta_slow) + cross-validator rejecting yarn fields without rope_scaling_type=yarn; Dynamic NTK harden + verify; LongLoRA schema gate (Llama-family + sft + transformers + !ring_attn); is_llama_model word-boundary regex (mirrors v0.39.0 is_gemma4_model / v0.44.0 is_llama4_model policy); Llama 3.1 NTK-aware (scale_inv_freq_llama3 smooth-transition / bool-rejected on every param / zero-old_context_len rejected; detect_llama3_rope_in_config with rope_type alias + explicit is None check); public-boundary input validation on get_rope_scaling_config (target_length / original_length / yarn_factor — security review fix) (v0.49.0) |
| test_v0480_part_b.py | Data Mixing Optimizer (BETA): parse_budget (digits + s/m/h suffix [60s, 24h]) + validate_datasets (containment + symlink + dedup + 32-cap) + MixCandidate (simplex + finite + bool-rejected) + BudgetTracker (injectable clock) + run_mix_optimizer (isolated proxy failures + KeyboardInterrupt propagation + NaN skip + partial budget trip) + render_mix_recipe_yaml (YAML injection defence) + write_mix_recipe / load_mix_recipe (atomic + TOCTOU + 256 KB cap) + soup data mix --optimize / --apply CLI (v0.48.0) |
| test_v0500_part_a.py | v0.50.0 — GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard); closed allowlist + frozen GRPOVariantSpec + MappingProxyType immutability; validate_grpo_variant (bool / null-byte / oversize / unknown rejected); validate_grpo_delta (bool / NaN / Inf / out-of-range rejected); apply_variant_loss deferred-live stub; TrainingConfig integration (Literal accept + two_sided requires delta cross-validator + NaN/Inf field_validator) (v0.50.0) |
| test_v0500_part_b.py | v0.50.0 — Long-context GRPO + vLLM sleep; validate_long_context_grpo_compat (task / backend / ring-attention exclusivity + null-byte + bool guards); validate_vllm_sleep_mode_compat (transformers / unsloth allowlist + null-byte); apply_vllm_sleep_mode deferred stub with v0.50.1 marker; SoupConfig integration (vllm_sleep_mode requires task='grpo' — code-review HIGH fix) (v0.50.0) |
| test_v0500_part_c.py | v0.50.0 — Multi-turn agent rollout backend allowlist (art / ruler / nemo_gym / openenv); frozen RolloutBackendSpec + MappingProxyType immutability; validate_rollout_backend (bool rejected); required_rollout_package per-entry mapping; launch_rollout deferred stub; SoupConfig task-gate + mlx rejection (v0.50.0) |
| test_v0500_part_d.py | v0.50.0 — 7 stability/efficiency knobs (ref_model_ema_alpha / replay_buffer_size / async_grpo_prefetch / tis_threshold / mask_truncated_completions / defer_rerolling / skip_zero_advantage / off_policy_mask_threshold); explicit bool-rejection field_validator across all numeric fields (tdd-guide HIGH fix); mask_truncated_completions requires tis_threshold cross-validator; SoupConfig task-gate naming every offending field; grpo_fp16 task-gate (code-review HIGH fix) (v0.50.0) |
| test_v0500_part_e.py | v0.50.0 — task='prm' (Process Reward Model) + vision_grpo flag; validate_prm_compat (data.format / modality / mlx gates); validate_vision_grpo_compat (task ∈ {grpo, ppo} / modality='vision' / non-mlx); build_prm_trainer deferred stub; SoupConfig integration with all rejection paths exercised (v0.50.0) |
| test_v0520.py | v0.52.0 Modality II — TTS / classifier / distill / BitNet / EBFT-GDPO / MoE quant / reasoning_effort: TTS family allowlist + per-family emotion allowlists (Orpheus + Oute) + validate_tts_compat; classifier / reranker / cross_encoder tasks + num_labels (with field_validator bool guard) + label_names dedup + classifier-only field gates; distill divergence (kl alias canonicalised, Literal excludes alias) + teacher_model + distill_temperature bounds; BitNet 1.58 quant + bitnet/tq1_0 export-format stubs + Falcon-E recipe + is_bitnet_model org-prefix detect; EBFT (structured/strided) + GDPO (standard/length_normalized/margin) variant allowlists + task gates; MoE expert quant (nf4/int8_rowwise) + train_router_only requiring moe_lora=true; reasoning_effort + train_on_eot with SFT-family task gate; 6 new recipes (5 TTS + Falcon-E BitNet); review-fix coverage (num_labels bool guard, Oute emotion allowlist, lazy-import in classifier validator, task gates, oversize / NaN / Inf matrices). Test count: 272 (v0.52.0) |
| test_v0531_82.py | v0.53.1 #82 autopilot pre-quantized detection: detect_prequantized_format + decide_quantization(prequantized=...) + detect_prequantized_format_from_path with cwd-containment + config.json symlink rejection + name/config aliases + word-boundary regex (v0.53.1) |
| test_v0531_142.py | v0.53.1 #142 merge_4bit + export_torchao live wiring: BNB-4bit single-stage merge + TorchAO PTQ with per-scheme kwarg allowlist + CLI --save-format + --quant-config + load_quant_config (yaml.safe_load + 256 KB cap + extension allowlist) + path TOCTOU (v0.53.1) |
| test_v0531_139.py | v0.53.1 #139 export_advanced_gguf live: 3-stage llama.cpp pipeline (convert → imatrix → quantize) + UD-prefix strip + subprocess argv shape + _prepare_calibration_text JSONL alias fallback + null-byte strip + 50 MB cap + POSIX O_NOFOLLOW + _safe_stderr Rich escape (v0.53.1) |
| test_v0531_109.py | v0.53.1 #109 deploy autopilot --measure: compute_cache_key + sha_of_file + measure_candidate OK/MINOR/MAJOR bands + pick_best soft-fallback (max-by-delta) + cache round-trip with symlink rejection on load AND save + CLI integration + _MAX_CANDIDATES=32 cap + render_measure_table markup escape regression (v0.53.1) |
| test_v0530.py | v0.53.0 Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats: GGUF (UD ladder 14 entries + IQ 12 + Apple/ARM 10 frozensets + non-overlap invariant + validate_* case-insensitive + rejection matrix + is_advanced_gguf_format union + _LOWER_INDEX O(1) lookup + MappingProxyType immutability + validate_calibration_data_path shape rejection + 4096-boundary + export_advanced_gguf v0.53.1 deferred stub); KV cache (KV_CACHE_TYPES frozenset + validate_kv_cache_type case + bool/null/oversize/non-string rejection + requires_hopper delegates to spec + get_kv_cache_spec frozen + schema fp8-on-mlx rejected with specific message + q8_0-on-mlx allowed); advanced precision (fp8_attention requires quantization_aware='fp8' BEFORE mlx-gate ordering + bool guards on every string param + schema rejects-without-fp8-qat; nvfp4 mlx + vision rejection + bool guards; unsloth_bnb_4bit backend='unsloth' + quantization='4bit' rejection matrix; apply_* deferred); LF parity (bnb_4bit_use_double_quant rejects none/8bit/gptq parametrize; llm_int8 rejects default-none + 4bit; quantize_ref_model happy on dpo/grpo/kto + rejects sft/pretrain; quantize_reward_model happy on ppo/reward_model + rejects dpo; explicit TypeError("v0.53.0 flag must be bool") from _validate_v053_bool_fields; explicit-null surfaces as valid boolean ValidationError); save formats (MERGE_SAVE_FORMATS lowercase normalisation + rejection matrix; TORCHAO_PTQ_SCHEMES CASE-SENSITIVE — int4weightonly rejected; validate_quant_config_path 4096-boundary; MergeSaveSpec + TorchAOPTQSpec frozen + MappingProxyType immutability; merge_4bit + export_torchao deferred); Cross-cutting (full 5-field YAML round-trip + cardinality invariant + tautological-assert replaced with allowlist + idempotent re-validate + get_gguf_spec unknown raises + bool guards on backend/modality/quantization across every advanced-precision validator). Test count: 154 (v0.53.0) |
| test_v0570_part_a.py | v0.57.0 soup adapters diff — effective_rank SVD entropy (identity / concentrated / empty / 1D / zero / bool-eps / non-finite eps); compute_layer_diffs (identical-zero / known-norm-5 / shape-mismatch-skipped / intersection-only / non-mapping / oversize cap); render_report_json + render_report_markdown (roundtrip / non-report TypeError / only-lists section); LayerDiff frozen via FrozenInstanceError; end-to-end compute_adapter_diff (safetensors fixture + outside-cwd reject + bool top_k + 1/201 boundary + missing safetensors + .bin rejection + POSIX symlink at weights file); CLI (table/json/markdown smoke + unknown --format + --output requires non-table + outside-cwd + atomic write); no-top-level-torch source-grep guard. Test count: 36 (v0.57.0) |
| test_v0570_part_b.py | v0.57.0 soup adapters merge — 4 strategies in pure numpy: merge_linear (average / weighted / intersection / shape-mismatch / single-rejected / 17-too-many / bool weight / negative / NaN / inf rejection / zero-sum / length-mismatch); merge_ties (density top-half / majority-sign election / tied-sign-defaults-positive review fix / density bounds / density=1.0 inclusive / bool-density); merge_dare (deterministic-by-seed / different-seeds-diverge / bool-seed / negative-seed / density=1 equals linear); merge_svd (no-rank equals linear / rank reduces / non-2d passthrough / rank clamp / invalid-rank); merge_adapters e2e linear + unknown-strategy + output-outside-cwd + POSIX symlink-at-output-safetensors rejection; SUPPORTED_STRATEGIES frozenset + STRATEGY_ORDER tuple invariants; predict_merged_verdict stub + non-report + canary_suite type; MergeReport FrozenInstanceError; CLI help / linear / unknown-strategy / invalid-weights / single-adapter rejection; no-top-level-torch source-grep. Test count: 43 (v0.57.0) |
| test_v0570_part_c.py | v0.57.0 soup adapters blame — parse_budget (60/60s/5m/2h / below-floor / above-cap / invalid-format / empty / bool / null-byte / non-string); plan_blame (happy / infeasible-budget / shard offsets / uneven split / empty dataset / outside-cwd-adapter / outside-cwd-dataset / invalid-layer / shard bounds / bool=True AND False shards / bool budget_seconds); run_blame stub raises NotImplementedError v0.57.1 + non-plan TypeError; BlamePlan + BlameShardWork FrozenInstanceError; CLI help + plan-only smoke + invalid-budget + live runner advisory exit-0. Test count: 33 (v0.57.0) |
| test_v0580.py | v0.58.0 soup loop CLI-first data flywheel — LoopState frozen + 3 closed allowlist + bool/null/oversize/non-string reject; with_status / bumped (unknown counter + bool delta rejected); state file I/O atomic + cwd containment + POSIX symlink rejection + 1 MiB cap + forward-compat unknown-field-dropping; init_state refuses overwrite without --force; CanaryPolicy (cross-field, NaN/Inf, frozen); deterministic SHA-256 routing (stable-only / 0% / 100% / determinism / 25% split approximate / empty key / NUL key / non-policy); rollback (clears canary / route-after-rollback all-stable / reason / non-policy); BucketStats (record + verdict OK/MAJOR/UNKNOWN + bounds + invalid bucket + snapshot MappingProxyType); parse_budget_string (5 happy + empty/garbage/NUL/negative/overflow); check_budget (within/budget-blocked/daily-cap-blocked/None=unlimited + negative/NaN/bool guards); reset_daily_counter_if_new_day (same-day/new-day/None-prior/negative); IterationRecord frozen + verdict allowlist + path-separator id; iteration write/read roundtrip + outside-cwd + missing + sorted listing + non-record + invalid manifest; new_iteration_id unique × 20; run_once (defaults + budget-skip + custom callbacks + invalid types); WatchConfig (defaults + bad poll_interval/max_iterations/callable); watch finite-runs + on_iteration flips status to stop; maybe_rollback (OK no-op / MAJOR clears / UNKNOWN no-op / non-policy / non-str); CLI smoke (loop --help / init / refuse overwrite / --force / invalid budget / status without init / status / pause+resume cycle / pause-stopped / resume-running / watch --max-iterations / --foreground/--detach mutex / canary / canary invalid traffic / canary same-as-stable / replay list-empty / replay show / replay unknown); source-grep (cli.py registers loop / __init__.py == 0.58.0 / no top-level torch in 5 loop modules / typer.Typer). Review-fix coverage: 1 CRITICAL + 3 HIGH + 3 MEDIUM + 1 LOW. Test count: 160 (188 pass + 1 POSIX-skipped). (v0.58.0) |
| test_v0570_part_d.py | v0.57.0 soup adapters branch/checkout/branches — create_branch happy + with-dataset + invalid-name + null-byte name + bool name + config-outside-cwd + missing-config + empty-base-model + null-byte base + bool base_model rejected + oversize 1MiB config + atomic write; list_branches empty + sorted; load_branch roundtrip + missing + invalid-name + POSIX symlink rejection; delete_branch true-when-present + false-missing + POSIX symlink rejection + traversal rejection; write_checkout writes target + drift detection + outside-cwd reject + non-branch + missing source; Branch FrozenInstanceError; SOUP_BRANCHES_DIR env override (valid-honoured + null-byte falls-back via mocked os.environ.get + CRLF falls-back); CLI smoke create / list / checkout / invalid-name / missing-config / missing-branch / empty-list. Test count: 37 (v0.57.0, 4 POSIX-skipped on Windows) |
| test_issue349_measured_fit.py | v0.73.1 #349 — measured stream VRAM fit: training.stream_vram_probe (bool, default false, sft-only) decides layer-streaming VRAM pre-flight by MEASURING a real forward+backward at the configured shape instead of using the fitted formula; justified on RTX 3050 where the formula under-predicts 0.934x at seq 5120 / 0.787x at seq 6144. Config validation + schema bounds (task gate sft, rejects stream_layers=false). Plus: training.batch_size field_validator now rejects 0 and negative values (was accepting -4 before). Test count: 33 (v0.73.1) |
| test_issue372_no_reexec_hint.py | #372 — --no-reexec hint is derived from the same argv as auto-reexec (build_train_reexec_argv + hint_argv_from_reexec); tokenised printed command is a superset of user flags parametrised over --fsdp / --deepspeed / --config / --trust-remote-code; control with no extra flags still prints a valid minimal command; CLI smoke does not skip. |
| test_issue392_mlx_target_keys.py | v0.73.1 #392 — the MLX adapter_config.json serialised target_modules UNRESOLVED, so a default MLX adapter loaded as a silent no-op. _apply_lora resolved auto into a LOCAL variable and trained the resolved modules; the writer serialised the raw config value, so the file carried {"keys": ["auto"]}, linear_to_lora_layers matched no module and load_weights(strict=False) dropped all 96 LoRA tensors WITHOUT A WORD — generation with the adapter bit-identical to the base. auto is the schema DEFAULT, so this was every MLX run that did not name its modules by hand, and adapter_config.json exists specifically to promise the output dir loads with mlx_lm.load(..., adapter_path=...). Reported by @armanbot-jpg with the control that isolates it: hand-editing keys makes the very same adapters.safetensors produce the tuned behaviour, i.e. the weights were fine and the config was not. Third time in this project a healthy loss curve shipped a dead artifact (v0.72.0's .inner. keys, #362's full-fine-tune adapter, this) — the loss curve structurally cannot see it, only the artifact can. Repaired as ONE shared resolve_mlx_target_keys() rather than teaching the writer to resolve too, since two copies of "which modules did we train?" is how they drifted. 10 red before, 10 green after; two are controls (the resolved value must never contain auto whatever the input, and rank/scale/dropout must still come from the config, so a repair that hardcoded the block would fail rather than pass). (v0.73.1) |
| test_issue441_coerce_token_ids.py | #441 — coerce_token_ids is the public shared helper (data_doctor must import that name, not _coerce_token_ids, and must not define its own copy); duck-typed mapping (has get, not a collections.abc.Mapping) returns [1, 2, 3] rather than raising input_ids[0]='input_ids'; the same object carrying assistant_masks takes the mask path (labels keep only the masked token — fails if only the coerce gate is widened); missing input_ids still raises the existing error; both call sites share _is_mapping_like (isinstance(out, Mapping) lives in exactly one place). |
| test_v07302.py | v0.73.2 soup ship leg-2 suite enhancements + noise floor: --noise-floor N (N in [2,10], metric-mode-only) re-runs base N times, per-axis floor = max-min spread, gates each at max(threshold, floor) — VramFit.noise_floor_block frozen + measure_base_spreads (base-only, N forward passes) + compute_noise_floors aggregation; evidence JSON schema extended with optional noise_floor block (runs + floors dict); 8th bundled suite mini_over_refusal (inverse safety gate, benign-not-refused) + DEFAULT_GENERAL_SUITE count 8; scorer fixes: extract_mcq_letter now understands \boxed{C} single-letter format, tool_call_name_match tolerates missing outer {"function": ...} envelope (carries BOTH name + arguments), score_bundled_suite raises TypeError for non-callable generator (was silently returning 0.0); baseline scale warning for the three affected suites (mini_mmlu / mini_common_sense / mini_tool_call); MCP ship_evidence tool honours noise_floor in evidence schema. (v0.73.2) |
| test_v0560.py | v0.56.0 soup diagnose post-training failure-mode report card — 6 probes (forgetting Δ-accuracy + tolerance band; refusal advbench/xstest delta + _MAX_REFUSAL_SCAN=8192 cap; format JSON/regex/tool_call with ReDoS probe + _VALID_KINDS frozenset; mode_collapse pairwise n-gram Jaccard over K completions; memorization training-prefix echo via split_prefix; contamination v0.47 ngram-overlap reuse + combined-complexity cap N×M>1e9); FailureReport + FailureScore frozen dataclasses with OK/MINOR/MAJOR taxonomy (≥0.85/≥0.60 thresholds) + compose_report / build_report SDK + atomic write_report (realpath containment + symlink reject) + render_badge_svg HTML-escaped 6-cell SVG + CLI smoke (--evidence/--output/--badge/--attach-to-registry); diagnose_report artifact kind + soup train --diagnose-gate MAJOR-rejection helper. Review-fix coverage: atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), 16 MiB evidence size cap, extract_row_text centralisation, tokenize delegates to _eval_text, extras null-byte sanitisation, source-grep regression guards. Test count: 123 (v0.56.0) |
| test_v0540.py | v0.54.0 soup advise pre-flight decision — Verdict engine (TASK_CATEGORIES + CHOICES allowlists; frozen Verdict / DatasetProfile / ROIEstimate; classify_task keyword + tool_calls + reasoning-trace signals + goal-steers; compute_dataset_profile shape + diversity + chosen/rejected + reasoning detection; build_verdict 5-branch rubric with _MIN_ROWS_FOR_GRPO=500; load_advise_dataset cwd-containment + symlink reject + BOM strip + malformed-JSON reject); Probe runner (synth_probe_baselines + synth_probe_lora_delta heuristic stubs with forward-compat model/device/lr/timeout_seconds kwargs; format_verdict_rubric + next_command_for handoff); Cross-project learning (record_verdict + load_history + _append_with_lock cross-process fcntl/msvcrt locking; ~/.soup/advise_history.jsonl + sidecar .lock on Windows; history_path env override containment; per-line 64 KB cap on history reads); CLI smoke (run / explain / compare subcommands + _rewrite_advise_argv scoped to argv[1]); review-fix coverage (atomic scratch write + symlink reject on read; concurrent 8-thread record stress; 49↔50 / 499↔500 / 4096↔4097 boundary). Test count: 136 (v0.54.0) |
| test_v0510.py | v0.51.0 Model Catalog Expansion + Alternative Model Hubs: hubs.py (SUPPORTED_HUBS + validate_hub_name + validate_hub_endpoint SSRF parity / CRLF rejection / IPv6 mapped private rejected / IPv6 loopback ok / control chars; resolve_endpoint env-var override; default_endpoint + endpoint_env_var + required_hub_package + is_hf with bool guards; MappingProxyType immutability); TrainingConfig hub field (default + Literal accept + None reject + case-insensitive normalisation + YAML round-trip) + SoupConfig _validate_hub_supported (mlx + non-hf rejected; mlx + hf accepted; modelers + transformers accepted); MULTIPACK_ARCHITECTURES extension (20 new arches parametrize + legacy preserved + exact count=38 + frozenset immutability); 26 new recipes (parametrize over every name × {get_recipe / RecipeMeta / SoupConfig load / yaml.safe_load / model id no null/whitespace/empty parts / max_length bounds / GRPO required fields}); baichuan-sft uses hub: modelscope; total recipe count >= 105 (v0.51.0) |
(Note: the test-file table above is a representative subset, not exhaustive — every tests/test_*.py file maps to a feature area or release.)
Claiming an Issue
Comment on it saying you are taking it. GitHub only lets us set the formal assignee badge for repo collaborators, so the comment is the claim — it will look unassigned either way, and we treat the thread as the record.
The current board is the pinned issue #1945, regenerated daily.
Two commitments in return:
- If a maintainer ends up doing the work instead, you get told, in the thread, before or when it lands. This has been broken once (#313): a contributor was green-lit, scoped a careful protocol, and then the measurement was run on borrowed hardware and the issue closed without a word to them. That was a process failure on our side, and this paragraph exists because of it.
- A claim that goes quiet is released with a comment, not silently. We check in first, and "life got in the way" needs no explanation — see #280 for the shape.
If an issue needs hardware you do not have, say so in the claim. Several open
issues are labelled infra-blocked for exactly that reason, and knowing early
is more useful than a stalled branch.
How work is queued
Review time and hosted CI are the scarce resources here, so a few rules keep the line moving. They are applied with judgment, not mechanically, and nothing is closed or released as a penalty.
- Reviews go in issue-number order. A pull request that is green and whose review rounds are answered moves up; one that waits on its author does not hold the others back.
- Open pull requests per author. A contributor with fewer than 3 merged pull requests can have 2 open at a time; with 3 to 9 merged, 3; with 10 or more, 5. Extra pull requests are not closed: a comment says what they wait behind, and they may be converted to drafts. Mark one ready for review when another lands.
- Claims. Hold at most 2 claims whose pull request is not open yet. A comment without
a plan is read as a question, not as a claim. A claim that is blocked on another issue
(
infra-blocked, or "lands after #N" in the thread) does not take one of the two slots, and neither does a claim whose pull request is open and waiting only for the merge line. - Regression tests. A bug-fix pull request carries a test that fails on
mainwithout the fix; the review checks that, and checks that undoing the fix makes the test fail. Merged pull requests that meet this get theregression-testlabel, visible in the pull-request list. - A quiet claim. With no pull request and no word from you for 7 days, you get a friendly check-in in the thread. If another 7 days pass without an answer, the claim is released with a comment. Nothing you wrote is lost, and you can take it back.
- A quiet pull request. A pull request waiting on its author for 14 days gets a ping in the thread. If another 7 days pass, we close it with a note that it can be reopened at any time.
- Full CI. The required test cells run when a maintainer adds the
ci:fulllabel to a pull request that is ready to merge, not on every push; other pushes run a quick subset. A draft pull request runs no CI at all until you mark it ready for review.
Issue difficulty
Maintainers rate every triaged issue from 1 to 10 with a difficulty:N label, so you can pick
something that fits your time and experience. How the number is chosen and what each level
means is in docs/issue-difficulty.md. When you open an issue you
can suggest a difficulty in the form; the maintainer sets the final one.
Maintained areas
Contributors with 3 or more merged pull requests can hold an area: a part of the project to
own end to end, from triage to the last follow-up. An area has an area:<name> label and a
tracker issue that lists its open items. Holding an area means:
- First refusal on new issues. When a maintainer triages an issue with the area's label, the holder is pinged, and the issue stays with them for 3 days before it is open to anyone.
- A higher open-pull-request line. Inside the area the limit from How work is queued is two higher than your usual one.
- No new powers. The holder is a contributor, not a maintainer: reviews and merges stay with the maintainers.
An area is released the way a claim is: after 14 quiet days you get a check-in in the tracker issue, and if another 7 days pass without an answer it is released with a comment. You can take it back.
| Holder | Area | Label |
|---|---|---|
| @kok-o | Data pipeline and traces | area:data |
| @Nari-Ab | CLI and help text | area:cli |
| @ybai08 | ReLoRA, preference training and layer streaming | area:training |
| @taljeon | Eval gate | area:eval |
| @akkupratap323 | Ingest and feedback sources | area:ingest |
| @dryn85-max | The MLX backend (maintainer's exception: named ahead of the threshold, held from the first merge) | area:mlx |
Making Changes
1. Create a Branch
git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix
2. Make Your Changes
- Write code following the style guidelines above
- Write tests first (TDD) — then implement to pass them
- Keep commits focused and logical
For each user-visible pull request, add its entry to
changelog.d/<latest-release>/<number>.<category>.md instead of editing CHANGELOG.md;
use the newest released version in CHANGELOG.md, prefer the PR number rather than the
issue it closes, and choose added, changed, deprecated, removed, fixed, or
security.
If you add or remove a CLI command, option, argument or a soup.yaml field, or change its
type, default or bounds, run python scripts/gen_reference_docs.py and commit the pages
it rewrites (docs/cli-reference.md, docs/config-reference.md); the tests fail until
you do and name what differs. Rewording a help string or a description needs nothing:
maintainers regenerate the wording at release. On a merge conflict in either generated
page, keep either side and re-run python scripts/gen_reference_docs.py.
3. Run Tests & Lint
Before pushing, ensure everything passes:
# Lint first
ruff check --fix src/soup_cli/ scripts/ tests/
# Then run tests
pytest tests/ -v --tb=short
4. Commit
Write clear, descriptive commit messages following Conventional Commits:
git add <specific-files>
git commit -m "feat: add support for X"
# or
git commit -m "fix: resolve Y when Z"
Types: feat, fix, refactor, docs, test, chore, perf, ci
5. Push & Open a PR
git push origin feature/your-feature-name
Then open a pull request on GitHub with:
- Clear title describing the change
- Description of what and why
- Reference any related issues (e.g., "Closes #123")
- Test results
Pull Request Checklist
When you open a PR, the GitHub template will show this checklist:
-
ruff check src/soup_cli/ scripts/ tests/ benchmarks/passes -
pytest tests/ -vpasses - Updated relevant docs (
README.mdand the matching page underdocs/) if needed - New tests added for new functionality
- No breaking changes (or documented in PR description)
Contributions are accepted under the Apache License, Version 2.0, the project's own terms (section 5 of the license): by opening a pull request you agree that your contribution is licensed that way and you keep your copyright; there is no CLA and no DCO sign-off for now.
Architecture & Design Decisions
Typer and Click compatibility
The supported Typer range spans releases using standalone Click and releases
vendoring it. Resolve exception and parameter-source identities through
soup_cli.utils.typer_compat; importing those objects from standalone click
can silently stop matching the live command tree. Inspect command and parameter
capabilities when traversing help or reference pages. Keep standalone Click
declared while the minimum supported Typer requires it.
The core and training floor jobs exercise Typer 0.15.4. The separate typer-vendored
job installs a core-only environment pinned to Typer 0.27.3 / Click 8.4.2,
records those versions, asserts the real command/exception/parameter-source
identities use Typer's vendored Click, and runs the unchanged 20-case
tests/test_typer_click_compat.py. This explicit lane runs only on schedule
and workflow_dispatch, keeping it off pull-request and release-push runs;
it does not replace the minimum-version environment.
Typer is not pinned in the ordinary CI stack, so its test matrix resolves the
newest release admitted by pyproject.toml. The compatibility regressions
exercise gate exit codes, noninteractive missing options, configured defaults,
explicit tokens, legacy bench routing and complete command-tree traversal.
Lazy Imports for Speed
Heavy ML imports (torch, transformers, trl) are imported inside command handlers so the CLI stays fast. Users can run soup version or soup --help instantly without waiting for PyTorch to load.
Pydantic for Config Validation
All YAML configs are validated using Pydantic v2 models. These models are the single source of truth for valid fields and defaults. See config/schema.py.
Trainers as Wrappers
trainer/sft.py, trainer/dpo.py, trainer/grpo.py, trainer/ppo.py wrap HuggingFace TRL trainers with:
- Auto quantization (BitsAndBytes, torchao QAT)
- Auto LoRA setup (PEFT)
- Auto batch size estimation
- Progress bar integration
Experiment Tracking is SQLite
No external dependencies required. All runs, metrics, and eval results go to ~/.soup/experiments.db.
Data Format Normalization
Multiple formats (Alpaca, ShareGPT, ChatML, LLaVA, ShareGPT4V) are normalized to a unified {"messages": [...]} structure in data/formats.py.
Adding a New Feature
1. New Training Task Type
If adding a new training algorithm:
- Create
trainer/your_trainer.pywrapping the appropriate TRL trainer - Add config fields to
config/schema.py(Pydantic v2) - Add a template YAML under
src/soup_cli/templates/+ amanifest.jsonentry (see the existing 21 templates) - Update
commands/train.pyto route to your trainer - Add 30+ tests in
tests/test_your_trainer.py - Update
README.md+ the relevant page underdocs/
2. New Data Format
- Add detection and conversion logic to
data/formats.py - Add tests in
tests/test_formats.py - Update
data/loader.pyif needed - Document in the relevant
docs/*.md
3. New Command
- Create
commands/your_command.pywith a handler function - Register in
src/soup_cli/cli.pywith@app.command() - Add tests in
tests/test_your_command.py - Update help text and README
4. New Recipe
- Add a
RecipeMetaentry inrecipes/catalog.py - Add tests in
tests/test_recipes.py - Update
README.mdrecipes section - Regenerate
tests/fixtures/recipe_config_snapshots.jsonwithpython scripts/generate_recipe_snapshot.pyand review the diff — the new recipe's resolved config must be in the committed snapshot, ortest_fixture_and_catalog_have_the_same_recipe_namesfails (#621)
Good First Issues
Look for issues labeled good first issue on GitHub. These are beginner-friendly tasks that help you get familiar with the codebase.
Great areas for first contributions:
- New recipes — add a ready-made config for a popular model (see
recipes/catalog.py) - Documentation — improve docstrings, README examples, or example configs
- Tests — increase coverage for existing commands
- Bug fixes — check open issues labeled
bug
CI/CD
GitHub Actions runs on every push and PR:
- ruff linting on Python 3.11 (must pass)
- pytest on Python 3.11, 3.12, 3.13 across Ubuntu, Windows, macOS (must pass)
- transformers-floor on Ubuntu / Python 3.11 (#478): installs
.[dev]under.github/constraints/transformers-floor.txt, asserts the pinnedtransformersversion, and runs the staticdtype=/torch_dtype=load-site guard. The declared[train]floor remainstransformers>=4.36.0,<5.0.0inpyproject.toml;transformers==4.36.0cannot resolve against the declaredtrlrange, so this job does not rewrite that pin. - benchmark (
.github/workflows/codspeed.yml, #1065): a CodSpeed regression benchmark, its own workflow and not part of the matrix above. It is off: the workflow starts only by manual dispatch, so it adds no check to your pull request. Turning it on takes the CodSpeed GitHub App, the repository variableCODSPEED_ENABLEDset totrue, and thepush/pull_requesttriggers restored in the workflow file (its header is the checklist). Even then it never runs on a fork PR — GitHub does not grant apull_requestrun from a fork an OIDC token, so the upload could not authenticate. It is not a required check and blocks nothing.
Pull requests run a quick subset first. A push to a pull request runs lint
and the tests on Ubuntu / Python 3.12 only. The required cells (Ubuntu on Python
3.11 and 3.12, Windows and macOS on 3.12, plus the smoke jobs) run when a
maintainer adds the ci:full label at approval, and on every later push while
the label stays. Until then the other test (...) checks show as Expected and
the smoke jobs as skipped; neither is a failure on your side. A PR is merged
only when those checks are green on its current head. After the merge, main
runs the same quick set on every commit and the full matrix (every Python from
3.11 to 3.13, with 3.11 on Ubuntu only) once a night; a push to a release/**
branch runs every cell. Python 3.13 runs in the nightly and release runs only, not
on a pull request, so a failure that exists only there shows on main in the
next nightly run.
Coverage is measured on main, not on a pull request. Every test (...)
cell of a pull-request run, test (ubuntu-latest, 3.11) at approval included,
runs the suite with --no-cov: a pull request gets no Codecov report and the
77% floor is not checked on it. The nightly run on main (and a manual dispatch
or a push to a release/** branch) measures coverage in test (ubuntu-latest, 3.11), checks the floor and uploads the report. So a change that takes coverage
below 77% merges green and turns the next nightly run red. A plain
pytest tests/ measures it and enforces the floor locally (it is in the
addopts of pyproject.toml), so run that before you push a change that
removes tests or adds untested code.
A draft pull request runs no CI. Nothing starts while a pull request is a draft, whatever is pushed to it. Mark it ready for review and the quick set starts on its current head.
Pull-request CI installs a pinned upstream stack. On a pull request and on a
push to main, every job that installs torch or transformers (the test cells,
the smoke jobs, Recipe Validation and recipe attach) takes them and the
packages they resolve with at the exact versions in
.github/constraints/pinned-upstream.txt, so a release upstream cannot turn
every open pull request red at once. The one exception is transformers-floor,
which installs its own exact floor, and packages outside that file (numpy,
pydantic, mlx, torchao, ...) still resolve the newest on every run. The nightly
run on main installs with no constraints, and that is where a new release of
a pinned package shows first. If your change moves a
bound in pyproject.toml past one of those pins, move the pin in the same pull
request: tests/test_ci_pinned_upstream.py names the pin, and the header of the
constraints file says how to pick the new version.
CI reuses pip downloads selectively. Only the Ubuntu / Python 3.12 test cell
caches pip's downloads and wheels via setup-python, keyed by pyproject.toml
and .github/constraints/*.txt. Other test cells and the smoke/contract jobs
leave pip caching disabled to limit large dependency archives competing with
the HF model caches. Every job still runs its installation command on a cache
hit, so the nightly run continues to resolve the newest allowed packages and
the floor job keeps its constraints. Retained entries can still vary by
dependency inputs and Git refs; virtual environments and HF caches are separate.
CI runs the tests in parallel. Every test (...) cell runs the suite with
pytest-xdist, one test file per worker process (-n logical --dist loadfile).
The tests of one file still run in order in one process, but two files can run
at the same moment and in any order. So a test must not depend on another file
having run first, on the working directory another test left behind, or on the
terminal width of the process: give the console an explicit width in the test
instead of setting COLUMNS. To reproduce a failure that shows only in CI,
install the plugin (pip install "pytest-xdist[psutil]") and run
pytest tests/ -n logical --dist loadfile.
A docs-only pull request runs the suite once. When every file a pull request
changes is under docs/ or changelog.d/, or is CONTRIBUTORS.md or
CONTRIBUTING.md (a renamed file counts under both names; README*.md,
benchmarks/ and everything else do not count), the plan job marks the run
docs-only: test (ubuntu-latest, 3.12) runs the whole suite as always, and the
other test cells and the three smoke jobs still report, as success, without
running a step. If the file list cannot be read, or is not the whole list, the
full set runs. A push, the nightly run and a release branch are never docs-only.
A hung test is cut off. The test steps run with --timeout=600 --timeout-method=thread --max-worker-restart=2 (pytest-timeout, part of the
dev extra), a 45-minute step limit and a 60-minute job limit. A test that
runs for ten minutes fails and the worker running it is killed and replaced;
xdist runs that test again on the new worker, and gives up after two
replacements. After five minutes (-o faulthandler_timeout=300) the stacks of
all threads are written to the log, so a hang says where it is. A healthy test
takes seconds. A test that really needs longer marks itself
@pytest.mark.timeout(1200). A local pytest run has no cap unless you pass
the flags.
See .github/workflows/ci.yml.
Stale CI marks (#1017)
A pull_request workflow run pins refs/pull/N/merge at run creation.
actions/checkout fetches that SHA; later movement of main is not re-resolved.
gh run rerun replays the frozen merge and cannot produce a different answer.
That is why a green tick can describe a main that no longer exists (#763's
NameError on _for_terminal) and why a red X can describe a test that main
has already fixed. Neither is the contributor's fault, and neither clears by
waiting.
If your PR is red for failures you do not touch: merge or rebase onto current
origin/main and push. That is the only way to rebuild the merge SHA for the
test matrix. Maintainers will not gh pr update-branch onto a fork without
asking.
If your PR is green and main has moved: the merge-freshness check is
re-posted on every push to main. It rebuilds the merge with
git merge-tree --write-tree (no write to your branch) and fails on ruff F821
or a conflict. Lag behind main is reported as a neutral check, not a
failure — a 10-commit cap at this repo's merge rate is strict: true with extra
steps. It does not re-run the 13-job test matrix.
required_status_checks.strict stays off: flipping it would force a rebase on
every merge. A docs-only pre-merge procedure is what maintainers already do by
hand and is not a control — GitHub will still merge a CLEAN PR on stale marks.
Ask a maintainer to add merge-freshness to the required checks on main so
the GitHub merge button honours it. The gate job on a pull_request event runs
the PR's own copy of scripts/merge_freshness.py, so it is self-graded. Only the
merge-freshness check posted by the reeval job from main's copy is
trustworthy.
Before merging anything, gh pr checks returning all-green is necessary and not
sufficient. Compare the newest run's created_at against main's commits since,
or look at merge-freshness.
Releases
The project follows semantic versioning: MAJOR.MINOR.PATCH
Version Bump Process
- Refresh the dev stack, or read the last
dependency driftrun (#323). A green local suite on a stale environment is a weaker signal than it looks: during v0.72.x the dev box ran torch 2.5.1 / trl 0.19.1 while CI resolved torch 2.13.0 / trl 0.29.1, and six trainers could not build their config at all on what a freshpip installactually gives a user. Eitherpip install -U -e ".[dev]"before testing, or check the latestdependency driftrun — it installs the current stack weekly, runs the suite against it, and writes the resolved-vs-declared table into its run summary. - Run
python scripts/assemble_changelog.pyand review the assembledCHANGELOG.md - Update version in
pyproject.tomlandsrc/soup_cli/__init__.py - Run full test suite and linting — this includes
tests/test_issue621_recipe_snapshot.py, which fails loudly if a schema default changed since the last release without regeneratingtests/fixtures/recipe_config_snapshots.json(python scripts/generate_recipe_snapshot.py); a red run here means the fixture is stale, not that the release is unsafe to cut once regenerated and reviewed - Update
README.md, the relevant page underdocs/, andSECURITY.md(if security-related) - Commit with message:
Release v0.X.0 - Tag:
git tag v0.X.0 && git push --tags - GitHub Actions verifies that no fragment was left behind, then publishes to PyPI
on the
v*tag (Trusted Publisher / OIDC)
Recognition
We credit every contributor — recognition is free and you earned it.
- CONTRIBUTORS.md: every merged (or in-tree-adopted) PR adds you here, with your work and PR number.
- CHANGELOG.md: from the release that ships your change, the entry is tagged
(#NNN by @you). - Commit trailers: PRs land with a
Co-Authored-By:line for you, so the GitHub contributors graph reflects your work even after a squash merge.
If you contributed and aren't credited somewhere, that's a bug — open a PR or an issue and we'll fix it.
Community
- Issues: Report bugs and request features on GitHub Issues
- Discussions: Ask questions on GitHub Discussions
- Discord: Chat with maintainers and other users on Discord — good for quick questions and pairing on a PR; anything worth finding later still belongs in Issues or Discussions
- Code of Conduct: Please read CODE_OF_CONDUCT.md — it applies on Discord too
- Security: Report security issues via SECURITY.md — privately, never in Discord
Questions?
- Check the README for quick start and features
- Check the docs for the full feature reference and architecture
- Open a GitHub Discussion for questions
Thank you for contributing!