Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

63.6 KB
Raw Blame

Contributing to Soup

Thank you for your interest in contributing to Soup! We welcome bug reports, feature requests, and pull requests from the community.

Getting Started

1. Fork & Clone

git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup

2. Set Up Development Environment

Requirements: Python 3.11, 3.12 or 3.13 (requires-python = ">=3.11,<3.14"; 3.10 was dropped in #1805). CI runs exactly those three; tests/test_requires_python_bound.py derives the declared bound from the CI matrix, so widening one without the other fails the suite.

Work inside a virtualenv, then install the project in editable mode with dev dependencies:

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

The venv step is not optional on Debian 12, Ubuntu 23.04 or later: without it pip refuses with error: externally-managed-environment, because those distros stop pip writing into the system Python that apt manages (PEP 668). pipx is the answer for installing Soup and the wrong tool here, since an editable checkout needs to be importable by the test suite rather than isolated from it.

This installs:

  • pytest for testing
  • ruff for linting + formatting
  • pytest-cov for coverage
  • httpx for HTTP testing
  • mypy for type checking
  • pre-commit for the lint/format git hooks
  • the full training stack — [dev] self-references [train], so torch, transformers, peft, trl, datasets, bitsandbytes, accelerate are pulled in too (since v0.71.0 these are an opt-in extra, not core deps)
  • cryptography — [dev] also pulls it in (it's the [sign] extra) so the ed25519 signing tests (soup adapters sign / soup attest) run in CI
  • reportlab — [dev] also pulls it in (it's the [pdf] extra) so the Annex XI/XII PDF tests (soup train --annex-xi *.pdf) run in CI

3. Verify Setup

Run the test suite to confirm everything works:

pytest tests/ -v --tb=short

Run the linter:

ruff check src/soup_cli/ scripts/ tests/ benchmarks/

If both pass — you're ready to contribute!

Code Style

We use ruff for all code style and linting. Before committing, run:

# Check for issues
ruff check src/soup_cli/ scripts/ tests/ benchmarks/

# Auto-fix issues
ruff check --fix src/soup_cli/ scripts/ tests/

Style Guidelines

  • Line length: 100 characters (enforced by ruff)
  • Imports: Sorted and organized (ruff I rule)
  • Naming: No single-letter variable names (ruff E741) — use entry, part, length instead of l, p, etc.
  • Lazy imports: Heavy dependencies (torch, transformers, peft, trl, etc.) should be imported inside functions, not at module level, to keep the CLI responsive
  • No third-party licence headers. This project is Apache-2.0 and ships to PyPI, so an SPDX-License-Identifier or copyright line naming anyone else is a licensing claim we cannot make. If you generate files from a template — including with automated tooling, which is how this has happened three times — strip its header before you push. tests/test_no_foreign_license_headers.py fails the build on this rather than relying on you or me to spot it in a diff.
  • Config validation: Always use Pydantic v2 with BaseModel and Field
  • Output: Use rich.console.Console for all output — never bare print()
  • Type hints: Always include type hints for function parameters and return values

Example:

# WRONG
from torch import cuda
import transformers

def train():
    print("Starting training")
    model = transformers.AutoModel.from_pretrained("llama-7b")

# CORRECT
def train():
    from torch import cuda
    import transformers

    console = Console()
    console.print("Starting training")
    model = transformers.AutoModel.from_pretrained("llama-7b")

Project Structure

Key directories:

src/soup_cli/
  cli.py              - Main entry point, command routing
  commands/           - Command implementations (train, chat, eval, deploy, etc.)
  config/             - Config schema (schema.py) and loader (loader.py)
  data/               - Data loading, format conversion, providers, templates
  trainer/            - Training wrappers (SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, Pretrain, Reward Model, Embedding, BCO, Preference, Distill, Classifier, PRM, Unlearn, MoLE Routing)
  monitoring/         - Callbacks and live dashboard
  experiment/         - SQLite experiment tracking
  eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
  bench/              - Benchmark instruments behind `soup bench train` (collector, train_report, train_run) (#836)
  migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
  recipes/            - Ready-made configs for popular models (171 recipes)
  autopilot/          - Zero-config decision engine (v0.25.0)
  registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
  cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
  data/traces/        - Trace-to-Preference harvester (v0.26.0)
  data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve, reward_synth, reward_stress, layer_stream, layer_shard, layer_stream_runtime
  templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
  ui/                 - Web UI (FastAPI + HTML/JS SPA)

tests/                - Test suite covering all CLI commands, trainers, and model architectures
examples/             - Real-world config examples and datasets

Running Tests

All Tests

pytest tests/ -v --tb=short

Single Test File

pytest tests/test_config.py -v

Single Test

pytest tests/test_data.py::test_detect_alpaca_format -v

With Coverage

pytest tests/ --cov=soup_cli --cov-report=html

GPU Tests

Every CI runner is GPU-less, so the tests that need a CUDA device skip there and only mean something on a real card. They carry the gpu marker (#833):

pytest tests/ -m gpu --no-cov -v

--no-cov matters: the default --cov-fail-under gate is computed over the whole package and fails a subset run. To gate a new test on CUDA, decorate it with @pytest.mark.gpu (or @pytest.mark.gpu(reason="...")); do not write your own torch.cuda.is_available() check. tests/conftest.py holds the one probe, and tests/test_issue833_gpu_marker.py fails on a private one. Use tests.conftest.cuda_available() only to pick a device, never to skip.

If you have a card, a run is a real contribution: both recorded runs so far found defects CI could not see. Open an issue (or comment on #833) with:

  • the card, its compute capability (torch.cuda.get_device_capability()), and VRAM
  • the driver version, and the torch build (torch.__version__, torch.version.cuda)
  • the bitsandbytes version, if installed
  • the commit you ran (git rev-parse HEAD)
  • the summary line (N passed, N failed, N skipped), and the name and first error line of every failure

Accepted runs are recorded in benchmarks/gpu-test-runs.md.

Test Files

A representative sample of the suite below. Run pytest tests/ -v for the complete list.

Name a new test file after the module or feature it covers (test_<module>.py), not after an issue number or a version. Existing test files keep their names.

File Covers
test_config.py Config loading, validation, defaults
test_data.py Format detection, conversion, validation
test_gpu.py GPU detection, batch size estimation
test_cli.py CLI commands, version --full
test_tracker.py SQLite experiment tracker
test_runs.py soup runs CLI commands
test_data_tools.py Data convert/merge/dedup/stats commands
test_eval.py Eval command
test_smoke_train.py Full pipeline smoke tests (GPU)
test_chat.py Chat command, _detect_base_model
test_push.py Push command, _format_size, _generate_model_card
test_init.py Init command, templates, overwrite logic
test_callback.py SoupTrainerCallback (mock-based)
test_display.py TrainingDisplay rendering
test_loader.py Data loading (JSONL/JSON/CSV, edge cases)
test_validator.py validate_and_stats, extended_stats, _percentile
test_formats.py Reverse conversion, round-trips, edge cases
test_merge.py Merge command, adapter detection, validation
test_export.py Export command, GGUF quant types, validation
test_resume.py Resume checkpoint resolution, W&B flag
test_serve.py Serve command, FastAPI app, endpoints, streaming
test_generate.py Data generate, JSON parsing, validation, prompts
test_sweep.py Sweep params parsing, combinations, nested config
test_diff.py Diff prompts collection, metrics, CLI
test_deepspeed.py DeepSpeed configs, multi-GPU detection, trainer integration
test_errors.py Friendly error messages, --verbose flag, error mapping
test_doctor.py soup doctor command, version checking, system resources, dependency table
test_quickstart.py soup quickstart demo, data/config creation, --dry-run
test_grpo.py GRPO config, rewards, data prep, template, sweep shortcuts
test_progress.py Rich download progress bar, _enable_hf_transfer_progress
test_unsloth.py Unsloth backend config, detection, trainer integration, templates
test_vision.py Vision modality config, LLaVA/ShareGPT4V formats, loader, trainer, templates
test_qat.py QAT config, validation, trainer integration, export compatibility
test_ui.py Web UI command, FastAPI endpoints, auth, static files, config validation
test_vllm_serve.py vLLM backend detection, engine creation, serve --backend flag, FastAPI app
test_ppo.py PPO config, reward model config, data prep, RLHF template, routing, sweep
test_kto.py KTO config, data format, template, routing, sweep, train guard, wizard
test_orpo.py ORPO config, template, routing, sweep, train guard, wizard
test_simpo.py SimPO config, template, routing, sweep, train guard
test_ipo.py IPO config, template, routing, sweep, train guard
test_advanced_peft.py DoRA, LoRA+, GaLore config, validation, sweep shortcuts
test_infer.py Batch inference command, prompt reading, CLI validation
test_tensorboard.py TensorBoard flag, wandb conflict, report_to routing
test_pretrain.py Pretrain task, plaintext format, MoE config, templates, routing
test_moe.py MoE detection, ScatterMoE LoRA targets, MoE info extraction
test_bugfixes.py v0.10.1-v0.14.3 regression fixes
test_cli_subprocess.py Subprocess CLI tests: entry point, encoding, paths, platform regressions
test_performance.py Liger Kernel, FlashAttention, FSDP2, Ring Attention, long-context, RoPE scaling
test_embedding.py Embedding task config, format, template, routing, sweep, pooling
test_onnx_tensorrt_export.py ONNX export, TensorRT-LLM export, format support
test_speculative_decoding.py Speculative decoding CLI, draft model, vLLM integration
test_server_generate.py Server provider for data generate, SSRF validation
test_quality_filter.py Perplexity + coherence scoring, soup data filter
test_audio.py Audio modality config, format, template, routing, loader
test_sglang_serve.py SGLang backend detection, runtime creation, serve --backend
test_deploy_ollama.py Ollama deploy, Modelfile gen, template mapping, security validation
test_eval_platform.py Custom eval, judge, human eval (Elo), leaderboard, compare, auto-eval, security
test_synth_data_pro.py Providers (Ollama, Anthropic, vLLM), templates, quality pipeline, SSRF
test_migrate.py LLaMA-Factory/Axolotl/Unsloth migration, path traversal, round-trip validation
test_recipes.py Recipe catalog, search, CLI (list/show/use), path traversal
test_neftune_rslora.py NEFTune config/validation/sweep, rsLoRA config/validation/sweep
test_profile.py Training profiler: memory estimation, speed, GPU recommendations, CLI
test_multi_adapter.py Multi-adapter serving: validation, parsing, FastAPI endpoints, CLI
test_data_sample.py Data sampling: random/diverse/hard strategies, CLI, edge cases
test_adapters.py Adapter management: list/info/compare, discovery, metadata
test_export_removed_formats.py AWQ/GPTQ removal verification: rejection at CLI, pyproject extras, codebase clean
test_packing.py Sample packing: config, YAML, trainer integration, sweep
test_data_split.py Data split: ratio/absolute/stratified splits, seed, edge cases
test_curriculum.py Curriculum learning: config, length sort, buckets, sweep
test_dataset_hub.py HF dataset search, preview, download, format conversion, security
test_freeze_training.py Freeze training: config, layer freezing, GPT-2 naming, sweep
test_loss_watchdog.py Loss watchdog: config, callback behavior, patience, sweep
test_dataset_registry.py Dataset registry: CRUD, CLI, name validation, error handling
test_tool_calling.py Tool-calling format detection, normalization, eval scoring, recipes (v0.25.0)
test_rlvr.py RLVR verifiable rewards: math_verify, code_exec sandbox, json_schema (v0.25.0)
test_peft_methods.py VeRA + OLoRA LoraConfig, peft_builder, sweep integration (v0.25.0)
test_mlx_backend.py Apple Silicon MLX backend: detection, trainers, routing (v0.25.0)
test_data_augment.py Data augmentation: rephrase/translate/style strategies, CLI, security (v0.25.0)
test_training_intelligence.py Forgetting detection + checkpoint intelligence + SQLite (v0.25.0)
test_autopilot.py Autopilot: analyzers, decision engine, CLI (v0.25.0)
test_registry.py Model Registry: hashing, store CRUD, artifacts, lineage DAG, diff, CLI, history (v0.26.0)
test_eval_gate.py Eval-Gated Training: config, suite loading, baseline, callback, CLI (v0.26.0)
test_trace_to_pref.py Trace-to-Preference: LangChain/OpenAI/Soup-serve parsers, pair builder, CLI (v0.26.0)
test_quant_check.py Quant-Lobotomy: classify_delta, resolve_model_ref, render formats, CLI (v0.26.0)
test_cans.py Soup Cans: manifest schema, pack/unpack, tar traversal, fork security, CLI (v0.26.0)
test_multi_gpu.py Multi-GPU Mastery: topology, --gpus, accelerate launcher, ZeRO++, FSDP2+compile, pipeline (v0.27.0)
test_training_speed.py Training Speed & Memory: CCE, FP8, grad-ckpt tiers, kernel picker, cross-doc attn, activation offload (v0.28.0)
test_hf_integration.py HF Hub Deep Integration: token/endpoint/repo_id, auto-push callback, model card v2, collections, data push, HF Spaces, private-IP SSRF (v0.29.0)
test_inference_advanced.py Inference Excellence: prefix caching, spec-decoding auto-pairing, LoRA hot-swap, structured output, dashboard + /metrics, OpenTelemetry tracing (v0.30.0)
test_recipes_v031.py Model & Recipe Breadth: 34 new recipes (vision/audio/reasoning/edge/domain/multimodal); catalog-wide invariants; CI workflow validation (v0.31.0)
test_auto_tuning.py Training Stability & Auto-Tuning: LR range finder, grad-accum monitor, auto mixed-precision, auto warmup, spike recovery, convergence detector, autopilot wiring (v0.32.0)
test_part_f_hardening.py Live Wire hardening: RLVR OS-level sandbox isolation + prune_checkpoints TOCTOU (v0.33.0)
test_part_a_wave1.py Live Wire: live eval-gate scoring + registry attach (v0.33.0)
test_part_a_wave2.py Live Wire: soup can run / publish + DeployTarget schema (v0.33.0)
test_part_e.py Live Wire: --find-lr live loop + spike recovery hint + auto mixed-precision push + grad-accum advisory (v0.33.0)
test_part_d.py Historical Live Wire primitives: structured-output LogitsProcessor + HF push integration smoke (v0.33.0)
test_part_c.py Live Wire: multi-trainer v0.28.0 features + selective ckpt hooks + CrossDocCollator (v0.33.0)
test_part_b.py Live Wire: auto-reexec under accelerate launch + DeepSpeed-MII live serve (v0.33.0)
test_log_level.py Smart logging tiers --log-level quiet/normal/verbose/debug (v0.34.0)
test_run_cost.py Per-run cost: GPU-rate lookup + estimate + format + tracker integration (v0.34.0)
test_why.py soup why heuristic explainer: NaN / plateau / divergence / grad-norm / LR bounds (v0.34.0)
test_crash_reporter.py .crash bundle: classify, redact secrets, write under cwd, oversize truncation (v0.34.0)
test_replay.py soup runs replay: summarise + downsample + CLI rendering (v0.34.0)
test_profiling.py Auto-profiling: ProfilerSchedule + path containment + torch-less degradation (v0.34.0)
test_tui.py soup tui: CLI bounds + missing-textual error + row builders (v0.34.0)
test_trainer_coverage_v035.py Multi-trainer v0.28.0 wiring smoke matrix: every trainer × every speed/memory feature + benchmark_kernel_combos + schema-gate lift (v0.35.0 — #60, #61, #45)
test_v0470_part_a.py Synthetic Data Forge: ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses + VALID_TASKS allowlist + chunk_document + score_uncertainty + discover_documents (cwd-contained, symlink-rejecting, extension allowlist) + build_forge_plan validators (task / target_rows / teacher / NaN+Inf threshold) + synthesise_forge_rows (judge-exception swallow at DEBUG) + write_forge_dataset + write_provenance (atomic, TOCTOU-safe) + CLI smoke (v0.47.0)
test_v0470_part_b.py Data Quality Moat: BENCHMARKS MappingProxyType + ScoreReport frozen + ngram_set / ngram_overlap_ratio / decontaminate_rows (containment ratio) + detect_pii (ReDoS-hardened regexes + 50 KB pre-cap) + detect_language (6-language stopword heuristic) + score_toxicity (keyword baseline) + score_educational_value + compute_scorecard + load_jsonl_rows / write_jsonl_rows (cwd-contained, symlink-rejecting, atomic) + CLI smoke per subcommand (v0.47.0)
test_v0480_part_a.py Curriculum-Aware Trainer (BETA): DynamicCurriculumPolicy frozen + bounds; compute_bucket_weights softmax + water-fill (floor-strict invariant); validate_distributed_curriculum DDP gate; render_curve + parse_history_jsonl with 100k-row DoS cap; SoupConfig cross-validators (requires-curriculum / mlx-rejected / non-SFT-rejected / floor ≤ 1/buckets); soup runs curriculum-curve CLI (TOCTOU symlink reject + 50 MB cap + corrupt-JSONL exit 2) (v0.48.0)
test_v0490.py v0.49.0 Long Context & Architecture: YaRN math kernels (yarn_find_correction_dim / yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale — bool/NaN/Inf rejection, auto-disambiguated low==high); YaRN schema fields (yarn_factor / yarn_attn_factor / yarn_beta_fast / yarn_beta_slow) + cross-validator rejecting yarn fields without rope_scaling_type=yarn; Dynamic NTK harden + verify; LongLoRA schema gate (Llama-family + sft + transformers + !ring_attn); is_llama_model word-boundary regex (mirrors v0.39.0 is_gemma4_model / v0.44.0 is_llama4_model policy); Llama 3.1 NTK-aware (scale_inv_freq_llama3 smooth-transition / bool-rejected on every param / zero-old_context_len rejected; detect_llama3_rope_in_config with rope_type alias + explicit is None check); public-boundary input validation on get_rope_scaling_config (target_length / original_length / yarn_factor — security review fix) (v0.49.0)
test_v0480_part_b.py Data Mixing Optimizer (BETA): parse_budget (digits + s/m/h suffix [60s, 24h]) + validate_datasets (containment + symlink + dedup + 32-cap) + MixCandidate (simplex + finite + bool-rejected) + BudgetTracker (injectable clock) + run_mix_optimizer (isolated proxy failures + KeyboardInterrupt propagation + NaN skip + partial budget trip) + render_mix_recipe_yaml (YAML injection defence) + write_mix_recipe / load_mix_recipe (atomic + TOCTOU + 256 KB cap) + soup data mix --optimize / --apply CLI (v0.48.0)

| test_v0500_part_a.py | v0.50.0 — GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard); closed allowlist + frozen GRPOVariantSpec + MappingProxyType immutability; validate_grpo_variant (bool / null-byte / oversize / unknown rejected); validate_grpo_delta (bool / NaN / Inf / out-of-range rejected); apply_variant_loss deferred-live stub; TrainingConfig integration (Literal accept + two_sided requires delta cross-validator + NaN/Inf field_validator) (v0.50.0) | | test_v0500_part_b.py | v0.50.0 — Long-context GRPO + vLLM sleep; validate_long_context_grpo_compat (task / backend / ring-attention exclusivity + null-byte + bool guards); validate_vllm_sleep_mode_compat (transformers / unsloth allowlist + null-byte); apply_vllm_sleep_mode deferred stub with v0.50.1 marker; SoupConfig integration (vllm_sleep_mode requires task='grpo' — code-review HIGH fix) (v0.50.0) | | test_v0500_part_c.py | v0.50.0 — Multi-turn agent rollout backend allowlist (art / ruler / nemo_gym / openenv); frozen RolloutBackendSpec + MappingProxyType immutability; validate_rollout_backend (bool rejected); required_rollout_package per-entry mapping; launch_rollout deferred stub; SoupConfig task-gate + mlx rejection (v0.50.0) | | test_v0500_part_d.py | v0.50.0 — 7 stability/efficiency knobs (ref_model_ema_alpha / replay_buffer_size / async_grpo_prefetch / tis_threshold / mask_truncated_completions / defer_rerolling / skip_zero_advantage / off_policy_mask_threshold); explicit bool-rejection field_validator across all numeric fields (tdd-guide HIGH fix); mask_truncated_completions requires tis_threshold cross-validator; SoupConfig task-gate naming every offending field; grpo_fp16 task-gate (code-review HIGH fix) (v0.50.0) | | test_v0500_part_e.py | v0.50.0 — task='prm' (Process Reward Model) + vision_grpo flag; validate_prm_compat (data.format / modality / mlx gates); validate_vision_grpo_compat (task ∈ {grpo, ppo} / modality='vision' / non-mlx); build_prm_trainer deferred stub; SoupConfig integration with all rejection paths exercised (v0.50.0) | | test_v0520.py | v0.52.0 Modality II — TTS / classifier / distill / BitNet / EBFT-GDPO / MoE quant / reasoning_effort: TTS family allowlist + per-family emotion allowlists (Orpheus + Oute) + validate_tts_compat; classifier / reranker / cross_encoder tasks + num_labels (with field_validator bool guard) + label_names dedup + classifier-only field gates; distill divergence (kl alias canonicalised, Literal excludes alias) + teacher_model + distill_temperature bounds; BitNet 1.58 quant + bitnet/tq1_0 export-format stubs + Falcon-E recipe + is_bitnet_model org-prefix detect; EBFT (structured/strided) + GDPO (standard/length_normalized/margin) variant allowlists + task gates; MoE expert quant (nf4/int8_rowwise) + train_router_only requiring moe_lora=true; reasoning_effort + train_on_eot with SFT-family task gate; 6 new recipes (5 TTS + Falcon-E BitNet); review-fix coverage (num_labels bool guard, Oute emotion allowlist, lazy-import in classifier validator, task gates, oversize / NaN / Inf matrices). Test count: 272 (v0.52.0) | | test_v0531_82.py | v0.53.1 #82 autopilot pre-quantized detection: detect_prequantized_format + decide_quantization(prequantized=...) + detect_prequantized_format_from_path with cwd-containment + config.json symlink rejection + name/config aliases + word-boundary regex (v0.53.1) | | test_v0531_142.py | v0.53.1 #142 merge_4bit + export_torchao live wiring: BNB-4bit single-stage merge + TorchAO PTQ with per-scheme kwarg allowlist + CLI --save-format + --quant-config + load_quant_config (yaml.safe_load + 256 KB cap + extension allowlist) + path TOCTOU (v0.53.1) | | test_v0531_139.py | v0.53.1 #139 export_advanced_gguf live: 3-stage llama.cpp pipeline (convert → imatrix → quantize) + UD-prefix strip + subprocess argv shape + _prepare_calibration_text JSONL alias fallback + null-byte strip + 50 MB cap + POSIX O_NOFOLLOW + _safe_stderr Rich escape (v0.53.1) | | test_v0531_109.py | v0.53.1 #109 deploy autopilot --measure: compute_cache_key + sha_of_file + measure_candidate OK/MINOR/MAJOR bands + pick_best soft-fallback (max-by-delta) + cache round-trip with symlink rejection on load AND save + CLI integration + _MAX_CANDIDATES=32 cap + render_measure_table markup escape regression (v0.53.1) | | test_v0530.py | v0.53.0 Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats: GGUF (UD ladder 14 entries + IQ 12 + Apple/ARM 10 frozensets + non-overlap invariant + validate_* case-insensitive + rejection matrix + is_advanced_gguf_format union + _LOWER_INDEX O(1) lookup + MappingProxyType immutability + validate_calibration_data_path shape rejection + 4096-boundary + export_advanced_gguf v0.53.1 deferred stub); KV cache (KV_CACHE_TYPES frozenset + validate_kv_cache_type case + bool/null/oversize/non-string rejection + requires_hopper delegates to spec + get_kv_cache_spec frozen + schema fp8-on-mlx rejected with specific message + q8_0-on-mlx allowed); advanced precision (fp8_attention requires quantization_aware='fp8' BEFORE mlx-gate ordering + bool guards on every string param + schema rejects-without-fp8-qat; nvfp4 mlx + vision rejection + bool guards; unsloth_bnb_4bit backend='unsloth' + quantization='4bit' rejection matrix; apply_* deferred); LF parity (bnb_4bit_use_double_quant rejects none/8bit/gptq parametrize; llm_int8 rejects default-none + 4bit; quantize_ref_model happy on dpo/grpo/kto + rejects sft/pretrain; quantize_reward_model happy on ppo/reward_model + rejects dpo; explicit TypeError("v0.53.0 flag must be bool") from _validate_v053_bool_fields; explicit-null surfaces as valid boolean ValidationError); save formats (MERGE_SAVE_FORMATS lowercase normalisation + rejection matrix; TORCHAO_PTQ_SCHEMES CASE-SENSITIVE — int4weightonly rejected; validate_quant_config_path 4096-boundary; MergeSaveSpec + TorchAOPTQSpec frozen + MappingProxyType immutability; merge_4bit + export_torchao deferred); Cross-cutting (full 5-field YAML round-trip + cardinality invariant + tautological-assert replaced with allowlist + idempotent re-validate + get_gguf_spec unknown raises + bool guards on backend/modality/quantization across every advanced-precision validator). Test count: 154 (v0.53.0) | | test_v0570_part_a.py | v0.57.0 soup adapters diff — effective_rank SVD entropy (identity / concentrated / empty / 1D / zero / bool-eps / non-finite eps); compute_layer_diffs (identical-zero / known-norm-5 / shape-mismatch-skipped / intersection-only / non-mapping / oversize cap); render_report_json + render_report_markdown (roundtrip / non-report TypeError / only-lists section); LayerDiff frozen via FrozenInstanceError; end-to-end compute_adapter_diff (safetensors fixture + outside-cwd reject + bool top_k + 1/201 boundary + missing safetensors + .bin rejection + POSIX symlink at weights file); CLI (table/json/markdown smoke + unknown --format + --output requires non-table + outside-cwd + atomic write); no-top-level-torch source-grep guard. Test count: 36 (v0.57.0) | | test_v0570_part_b.py | v0.57.0 soup adapters merge — 4 strategies in pure numpy: merge_linear (average / weighted / intersection / shape-mismatch / single-rejected / 17-too-many / bool weight / negative / NaN / inf rejection / zero-sum / length-mismatch); merge_ties (density top-half / majority-sign election / tied-sign-defaults-positive review fix / density bounds / density=1.0 inclusive / bool-density); merge_dare (deterministic-by-seed / different-seeds-diverge / bool-seed / negative-seed / density=1 equals linear); merge_svd (no-rank equals linear / rank reduces / non-2d passthrough / rank clamp / invalid-rank); merge_adapters e2e linear + unknown-strategy + output-outside-cwd + POSIX symlink-at-output-safetensors rejection; SUPPORTED_STRATEGIES frozenset + STRATEGY_ORDER tuple invariants; predict_merged_verdict stub + non-report + canary_suite type; MergeReport FrozenInstanceError; CLI help / linear / unknown-strategy / invalid-weights / single-adapter rejection; no-top-level-torch source-grep. Test count: 43 (v0.57.0) | | test_v0570_part_c.py | v0.57.0 soup adapters blame — parse_budget (60/60s/5m/2h / below-floor / above-cap / invalid-format / empty / bool / null-byte / non-string); plan_blame (happy / infeasible-budget / shard offsets / uneven split / empty dataset / outside-cwd-adapter / outside-cwd-dataset / invalid-layer / shard bounds / bool=True AND False shards / bool budget_seconds); run_blame stub raises NotImplementedError v0.57.1 + non-plan TypeError; BlamePlan + BlameShardWork FrozenInstanceError; CLI help + plan-only smoke + invalid-budget + live runner advisory exit-0. Test count: 33 (v0.57.0) | | test_v0580.py | v0.58.0 soup loop CLI-first data flywheel — LoopState frozen + 3 closed allowlist + bool/null/oversize/non-string reject; with_status / bumped (unknown counter + bool delta rejected); state file I/O atomic + cwd containment + POSIX symlink rejection + 1 MiB cap + forward-compat unknown-field-dropping; init_state refuses overwrite without --force; CanaryPolicy (cross-field, NaN/Inf, frozen); deterministic SHA-256 routing (stable-only / 0% / 100% / determinism / 25% split approximate / empty key / NUL key / non-policy); rollback (clears canary / route-after-rollback all-stable / reason / non-policy); BucketStats (record + verdict OK/MAJOR/UNKNOWN + bounds + invalid bucket + snapshot MappingProxyType); parse_budget_string (5 happy + empty/garbage/NUL/negative/overflow); check_budget (within/budget-blocked/daily-cap-blocked/None=unlimited + negative/NaN/bool guards); reset_daily_counter_if_new_day (same-day/new-day/None-prior/negative); IterationRecord frozen + verdict allowlist + path-separator id; iteration write/read roundtrip + outside-cwd + missing + sorted listing + non-record + invalid manifest; new_iteration_id unique × 20; run_once (defaults + budget-skip + custom callbacks + invalid types); WatchConfig (defaults + bad poll_interval/max_iterations/callable); watch finite-runs + on_iteration flips status to stop; maybe_rollback (OK no-op / MAJOR clears / UNKNOWN no-op / non-policy / non-str); CLI smoke (loop --help / init / refuse overwrite / --force / invalid budget / status without init / status / pause+resume cycle / pause-stopped / resume-running / watch --max-iterations / --foreground/--detach mutex / canary / canary invalid traffic / canary same-as-stable / replay list-empty / replay show / replay unknown); source-grep (cli.py registers loop / __init__.py == 0.58.0 / no top-level torch in 5 loop modules / typer.Typer). Review-fix coverage: 1 CRITICAL + 3 HIGH + 3 MEDIUM + 1 LOW. Test count: 160 (188 pass + 1 POSIX-skipped). (v0.58.0) | | test_v0570_part_d.py | v0.57.0 soup adapters branch/checkout/branches — create_branch happy + with-dataset + invalid-name + null-byte name + bool name + config-outside-cwd + missing-config + empty-base-model + null-byte base + bool base_model rejected + oversize 1MiB config + atomic write; list_branches empty + sorted; load_branch roundtrip + missing + invalid-name + POSIX symlink rejection; delete_branch true-when-present + false-missing + POSIX symlink rejection + traversal rejection; write_checkout writes target + drift detection + outside-cwd reject + non-branch + missing source; Branch FrozenInstanceError; SOUP_BRANCHES_DIR env override (valid-honoured + null-byte falls-back via mocked os.environ.get + CRLF falls-back); CLI smoke create / list / checkout / invalid-name / missing-config / missing-branch / empty-list. Test count: 37 (v0.57.0, 4 POSIX-skipped on Windows) | | test_issue349_measured_fit.py | v0.73.1 #349 — measured stream VRAM fit: training.stream_vram_probe (bool, default false, sft-only) decides layer-streaming VRAM pre-flight by MEASURING a real forward+backward at the configured shape instead of using the fitted formula; justified on RTX 3050 where the formula under-predicts 0.934x at seq 5120 / 0.787x at seq 6144. Config validation + schema bounds (task gate sft, rejects stream_layers=false). Plus: training.batch_size field_validator now rejects 0 and negative values (was accepting -4 before). Test count: 33 (v0.73.1) | | test_issue372_no_reexec_hint.py | #372 — --no-reexec hint is derived from the same argv as auto-reexec (build_train_reexec_argv + hint_argv_from_reexec); tokenised printed command is a superset of user flags parametrised over --fsdp / --deepspeed / --config / --trust-remote-code; control with no extra flags still prints a valid minimal command; CLI smoke does not skip. | | test_issue392_mlx_target_keys.py | v0.73.1 #392 — the MLX adapter_config.json serialised target_modules UNRESOLVED, so a default MLX adapter loaded as a silent no-op. _apply_lora resolved auto into a LOCAL variable and trained the resolved modules; the writer serialised the raw config value, so the file carried {"keys": ["auto"]}, linear_to_lora_layers matched no module and load_weights(strict=False) dropped all 96 LoRA tensors WITHOUT A WORD — generation with the adapter bit-identical to the base. auto is the schema DEFAULT, so this was every MLX run that did not name its modules by hand, and adapter_config.json exists specifically to promise the output dir loads with mlx_lm.load(..., adapter_path=...). Reported by @armanbot-jpg with the control that isolates it: hand-editing keys makes the very same adapters.safetensors produce the tuned behaviour, i.e. the weights were fine and the config was not. Third time in this project a healthy loss curve shipped a dead artifact (v0.72.0's .inner. keys, #362's full-fine-tune adapter, this) — the loss curve structurally cannot see it, only the artifact can. Repaired as ONE shared resolve_mlx_target_keys() rather than teaching the writer to resolve too, since two copies of "which modules did we train?" is how they drifted. 10 red before, 10 green after; two are controls (the resolved value must never contain auto whatever the input, and rank/scale/dropout must still come from the config, so a repair that hardcoded the block would fail rather than pass). (v0.73.1) | | test_issue441_coerce_token_ids.py | #441 — coerce_token_ids is the public shared helper (data_doctor must import that name, not _coerce_token_ids, and must not define its own copy); duck-typed mapping (has get, not a collections.abc.Mapping) returns [1, 2, 3] rather than raising input_ids[0]='input_ids'; the same object carrying assistant_masks takes the mask path (labels keep only the masked token — fails if only the coerce gate is widened); missing input_ids still raises the existing error; both call sites share _is_mapping_like (isinstance(out, Mapping) lives in exactly one place). | | test_v07302.py | v0.73.2 soup ship leg-2 suite enhancements + noise floor: --noise-floor N (N in [2,10], metric-mode-only) re-runs base N times, per-axis floor = max-min spread, gates each at max(threshold, floor) — VramFit.noise_floor_block frozen + measure_base_spreads (base-only, N forward passes) + compute_noise_floors aggregation; evidence JSON schema extended with optional noise_floor block (runs + floors dict); 8th bundled suite mini_over_refusal (inverse safety gate, benign-not-refused) + DEFAULT_GENERAL_SUITE count 8; scorer fixes: extract_mcq_letter now understands \boxed{C} single-letter format, tool_call_name_match tolerates missing outer {"function": ...} envelope (carries BOTH name + arguments), score_bundled_suite raises TypeError for non-callable generator (was silently returning 0.0); baseline scale warning for the three affected suites (mini_mmlu / mini_common_sense / mini_tool_call); MCP ship_evidence tool honours noise_floor in evidence schema. (v0.73.2) | | test_v0560.py | v0.56.0 soup diagnose post-training failure-mode report card — 6 probes (forgetting Δ-accuracy + tolerance band; refusal advbench/xstest delta + _MAX_REFUSAL_SCAN=8192 cap; format JSON/regex/tool_call with ReDoS probe + _VALID_KINDS frozenset; mode_collapse pairwise n-gram Jaccard over K completions; memorization training-prefix echo via split_prefix; contamination v0.47 ngram-overlap reuse + combined-complexity cap N×M>1e9); FailureReport + FailureScore frozen dataclasses with OK/MINOR/MAJOR taxonomy (≥0.85/≥0.60 thresholds) + compose_report / build_report SDK + atomic write_report (realpath containment + symlink reject) + render_badge_svg HTML-escaped 6-cell SVG + CLI smoke (--evidence/--output/--badge/--attach-to-registry); diagnose_report artifact kind + soup train --diagnose-gate MAJOR-rejection helper. Review-fix coverage: atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), 16 MiB evidence size cap, extract_row_text centralisation, tokenize delegates to _eval_text, extras null-byte sanitisation, source-grep regression guards. Test count: 123 (v0.56.0) | | test_v0540.py | v0.54.0 soup advise pre-flight decision — Verdict engine (TASK_CATEGORIES + CHOICES allowlists; frozen Verdict / DatasetProfile / ROIEstimate; classify_task keyword + tool_calls + reasoning-trace signals + goal-steers; compute_dataset_profile shape + diversity + chosen/rejected + reasoning detection; build_verdict 5-branch rubric with _MIN_ROWS_FOR_GRPO=500; load_advise_dataset cwd-containment + symlink reject + BOM strip + malformed-JSON reject); Probe runner (synth_probe_baselines + synth_probe_lora_delta heuristic stubs with forward-compat model/device/lr/timeout_seconds kwargs; format_verdict_rubric + next_command_for handoff); Cross-project learning (record_verdict + load_history + _append_with_lock cross-process fcntl/msvcrt locking; ~/.soup/advise_history.jsonl + sidecar .lock on Windows; history_path env override containment; per-line 64 KB cap on history reads); CLI smoke (run / explain / compare subcommands + _rewrite_advise_argv scoped to argv[1]); review-fix coverage (atomic scratch write + symlink reject on read; concurrent 8-thread record stress; 49↔50 / 499↔500 / 4096↔4097 boundary). Test count: 136 (v0.54.0) | | test_v0510.py | v0.51.0 Model Catalog Expansion + Alternative Model Hubs: hubs.py (SUPPORTED_HUBS + validate_hub_name + validate_hub_endpoint SSRF parity / CRLF rejection / IPv6 mapped private rejected / IPv6 loopback ok / control chars; resolve_endpoint env-var override; default_endpoint + endpoint_env_var + required_hub_package + is_hf with bool guards; MappingProxyType immutability); TrainingConfig hub field (default + Literal accept + None reject + case-insensitive normalisation + YAML round-trip) + SoupConfig _validate_hub_supported (mlx + non-hf rejected; mlx + hf accepted; modelers + transformers accepted); MULTIPACK_ARCHITECTURES extension (20 new arches parametrize + legacy preserved + exact count=38 + frozenset immutability); 26 new recipes (parametrize over every name × {get_recipe / RecipeMeta / SoupConfig load / yaml.safe_load / model id no null/whitespace/empty parts / max_length bounds / GRPO required fields}); baichuan-sft uses hub: modelscope; total recipe count >= 105 (v0.51.0) |

(Note: the test-file table above is a representative subset, not exhaustive — every tests/test_*.py file maps to a feature area or release.)

Claiming an Issue

Comment on it saying you are taking it. GitHub only lets us set the formal assignee badge for repo collaborators, so the comment is the claim — it will look unassigned either way, and we treat the thread as the record.

The current board is the pinned issue #1945, regenerated daily.

Two commitments in return:

  • If a maintainer ends up doing the work instead, you get told, in the thread, before or when it lands. This has been broken once (#313): a contributor was green-lit, scoped a careful protocol, and then the measurement was run on borrowed hardware and the issue closed without a word to them. That was a process failure on our side, and this paragraph exists because of it.
  • A claim that goes quiet is released with a comment, not silently. We check in first, and "life got in the way" needs no explanation — see #280 for the shape.

If an issue needs hardware you do not have, say so in the claim. Several open issues are labelled infra-blocked for exactly that reason, and knowing early is more useful than a stalled branch.

How work is queued

Review time and hosted CI are the scarce resources here, so a few rules keep the line moving. They are applied with judgment, not mechanically, and nothing is closed or released as a penalty.

  • Reviews go in issue-number order. A pull request that is green and whose review rounds are answered moves up; one that waits on its author does not hold the others back.
  • Open pull requests per author. A contributor with fewer than 3 merged pull requests can have 2 open at a time; with 3 to 9 merged, 3; with 10 or more, 5. Extra pull requests are not closed: a comment says what they wait behind, and they may be converted to drafts. Mark one ready for review when another lands.
  • Claims. Hold at most 2 claims whose pull request is not open yet. A comment without a plan is read as a question, not as a claim. A claim that is blocked on another issue (infra-blocked, or "lands after #N" in the thread) does not take one of the two slots, and neither does a claim whose pull request is open and waiting only for the merge line.
  • Regression tests. A bug-fix pull request carries a test that fails on main without the fix; the review checks that, and checks that undoing the fix makes the test fail. Merged pull requests that meet this get the regression-test label, visible in the pull-request list.
  • A quiet claim. With no pull request and no word from you for 7 days, you get a friendly check-in in the thread. If another 7 days pass without an answer, the claim is released with a comment. Nothing you wrote is lost, and you can take it back.
  • A quiet pull request. A pull request waiting on its author for 14 days gets a ping in the thread. If another 7 days pass, we close it with a note that it can be reopened at any time.
  • Full CI. The required test cells run when a maintainer adds the ci:full label to a pull request that is ready to merge, not on every push; other pushes run a quick subset. A draft pull request runs no CI at all until you mark it ready for review.

Issue difficulty

Maintainers rate every triaged issue from 1 to 10 with a difficulty:N label, so you can pick something that fits your time and experience. How the number is chosen and what each level means is in docs/issue-difficulty.md. When you open an issue you can suggest a difficulty in the form; the maintainer sets the final one.

Maintained areas

Contributors with 3 or more merged pull requests can hold an area: a part of the project to own end to end, from triage to the last follow-up. An area has an area:<name> label and a tracker issue that lists its open items. Holding an area means:

  • First refusal on new issues. When a maintainer triages an issue with the area's label, the holder is pinged, and the issue stays with them for 3 days before it is open to anyone.
  • A higher open-pull-request line. Inside the area the limit from How work is queued is two higher than your usual one.
  • No new powers. The holder is a contributor, not a maintainer: reviews and merges stay with the maintainers.

An area is released the way a claim is: after 14 quiet days you get a check-in in the tracker issue, and if another 7 days pass without an answer it is released with a comment. You can take it back.

Holder Area Label
@kok-o Data pipeline and traces area:data
@Nari-Ab CLI and help text area:cli
@ybai08 ReLoRA, preference training and layer streaming area:training
@taljeon Eval gate area:eval
@akkupratap323 Ingest and feedback sources area:ingest
@dryn85-max The MLX backend (maintainer's exception: named ahead of the threshold, held from the first merge) area:mlx

Making Changes

1. Create a Branch

git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix

2. Make Your Changes

  • Write code following the style guidelines above
  • Write tests first (TDD) — then implement to pass them
  • Keep commits focused and logical

For each user-visible pull request, add its entry to changelog.d/<latest-release>/<number>.<category>.md instead of editing CHANGELOG.md; use the newest released version in CHANGELOG.md, prefer the PR number rather than the issue it closes, and choose added, changed, deprecated, removed, fixed, or security.

If you add or remove a CLI command, option, argument or a soup.yaml field, or change its type, default or bounds, run python scripts/gen_reference_docs.py and commit the pages it rewrites (docs/cli-reference.md, docs/config-reference.md); the tests fail until you do and name what differs. Rewording a help string or a description needs nothing: maintainers regenerate the wording at release. On a merge conflict in either generated page, keep either side and re-run python scripts/gen_reference_docs.py.

3. Run Tests & Lint

Before pushing, ensure everything passes:

# Lint first
ruff check --fix src/soup_cli/ scripts/ tests/

# Then run tests
pytest tests/ -v --tb=short

4. Commit

Write clear, descriptive commit messages following Conventional Commits:

git add <specific-files>
git commit -m "feat: add support for X"
# or
git commit -m "fix: resolve Y when Z"

Types: feat, fix, refactor, docs, test, chore, perf, ci

5. Push & Open a PR

git push origin feature/your-feature-name

Then open a pull request on GitHub with:

  • Clear title describing the change
  • Description of what and why
  • Reference any related issues (e.g., "Closes #123")
  • Test results

Pull Request Checklist

When you open a PR, the GitHub template will show this checklist:

  • ruff check src/soup_cli/ scripts/ tests/ benchmarks/ passes
  • pytest tests/ -v passes
  • Updated relevant docs (README.md and the matching page under docs/) if needed
  • New tests added for new functionality
  • No breaking changes (or documented in PR description)

Contributions are accepted under the Apache License, Version 2.0, the project's own terms (section 5 of the license): by opening a pull request you agree that your contribution is licensed that way and you keep your copyright; there is no CLA and no DCO sign-off for now.

Architecture & Design Decisions

Typer and Click compatibility

The supported Typer range spans releases using standalone Click and releases vendoring it. Resolve exception and parameter-source identities through soup_cli.utils.typer_compat; importing those objects from standalone click can silently stop matching the live command tree. Inspect command and parameter capabilities when traversing help or reference pages. Keep standalone Click declared while the minimum supported Typer requires it.

The core and training floor jobs exercise Typer 0.15.4. The separate typer-vendored job installs a core-only environment pinned to Typer 0.27.3 / Click 8.4.2, records those versions, asserts the real command/exception/parameter-source identities use Typer's vendored Click, and runs the unchanged 20-case tests/test_typer_click_compat.py. This explicit lane runs only on schedule and workflow_dispatch, keeping it off pull-request and release-push runs; it does not replace the minimum-version environment. Typer is not pinned in the ordinary CI stack, so its test matrix resolves the newest release admitted by pyproject.toml. The compatibility regressions exercise gate exit codes, noninteractive missing options, configured defaults, explicit tokens, legacy bench routing and complete command-tree traversal.

Lazy Imports for Speed

Heavy ML imports (torch, transformers, trl) are imported inside command handlers so the CLI stays fast. Users can run soup version or soup --help instantly without waiting for PyTorch to load.

Pydantic for Config Validation

All YAML configs are validated using Pydantic v2 models. These models are the single source of truth for valid fields and defaults. See config/schema.py.

Trainers as Wrappers

trainer/sft.py, trainer/dpo.py, trainer/grpo.py, trainer/ppo.py wrap HuggingFace TRL trainers with:

  • Auto quantization (BitsAndBytes, torchao QAT)
  • Auto LoRA setup (PEFT)
  • Auto batch size estimation
  • Progress bar integration

Experiment Tracking is SQLite

No external dependencies required. All runs, metrics, and eval results go to ~/.soup/experiments.db.

Data Format Normalization

Multiple formats (Alpaca, ShareGPT, ChatML, LLaVA, ShareGPT4V) are normalized to a unified {"messages": [...]} structure in data/formats.py.

Adding a New Feature

1. New Training Task Type

If adding a new training algorithm:

  1. Create trainer/your_trainer.py wrapping the appropriate TRL trainer
  2. Add config fields to config/schema.py (Pydantic v2)
  3. Add a template YAML under src/soup_cli/templates/ + a manifest.json entry (see the existing 21 templates)
  4. Update commands/train.py to route to your trainer
  5. Add 30+ tests in tests/test_your_trainer.py
  6. Update README.md + the relevant page under docs/

2. New Data Format

  1. Add detection and conversion logic to data/formats.py
  2. Add tests in tests/test_formats.py
  3. Update data/loader.py if needed
  4. Document in the relevant docs/*.md

3. New Command

  1. Create commands/your_command.py with a handler function
  2. Register in src/soup_cli/cli.py with @app.command()
  3. Add tests in tests/test_your_command.py
  4. Update help text and README

4. New Recipe

  1. Add a RecipeMeta entry in recipes/catalog.py
  2. Add tests in tests/test_recipes.py
  3. Update README.md recipes section
  4. Regenerate tests/fixtures/recipe_config_snapshots.json with python scripts/generate_recipe_snapshot.py and review the diff — the new recipe's resolved config must be in the committed snapshot, or test_fixture_and_catalog_have_the_same_recipe_names fails (#621)

Good First Issues

Look for issues labeled good first issue on GitHub. These are beginner-friendly tasks that help you get familiar with the codebase.

Great areas for first contributions:

  • New recipes — add a ready-made config for a popular model (see recipes/catalog.py)
  • Documentation — improve docstrings, README examples, or example configs
  • Tests — increase coverage for existing commands
  • Bug fixes — check open issues labeled bug

CI/CD

GitHub Actions runs on every push and PR:

  • ruff linting on Python 3.11 (must pass)
  • pytest on Python 3.11, 3.12, 3.13 across Ubuntu, Windows, macOS (must pass)
  • transformers-floor on Ubuntu / Python 3.11 (#478): installs .[dev] under .github/constraints/transformers-floor.txt, asserts the pinned transformers version, and runs the static dtype= / torch_dtype= load-site guard. The declared [train] floor remains transformers>=4.36.0,<5.0.0 in pyproject.toml; transformers==4.36.0 cannot resolve against the declared trl range, so this job does not rewrite that pin.
  • benchmark (.github/workflows/codspeed.yml, #1065): a CodSpeed regression benchmark, its own workflow and not part of the matrix above. It is off: the workflow starts only by manual dispatch, so it adds no check to your pull request. Turning it on takes the CodSpeed GitHub App, the repository variable CODSPEED_ENABLED set to true, and the push / pull_request triggers restored in the workflow file (its header is the checklist). Even then it never runs on a fork PR — GitHub does not grant a pull_request run from a fork an OIDC token, so the upload could not authenticate. It is not a required check and blocks nothing.

Pull requests run a quick subset first. A push to a pull request runs lint and the tests on Ubuntu / Python 3.12 only. The required cells (Ubuntu on Python 3.11 and 3.12, Windows and macOS on 3.12, plus the smoke jobs) run when a maintainer adds the ci:full label at approval, and on every later push while the label stays. Until then the other test (...) checks show as Expected and the smoke jobs as skipped; neither is a failure on your side. A PR is merged only when those checks are green on its current head. After the merge, main runs the same quick set on every commit and the full matrix (every Python from 3.11 to 3.13, with 3.11 on Ubuntu only) once a night; a push to a release/** branch runs every cell. Python 3.13 runs in the nightly and release runs only, not on a pull request, so a failure that exists only there shows on main in the next nightly run.

Coverage is measured on main, not on a pull request. Every test (...) cell of a pull-request run, test (ubuntu-latest, 3.11) at approval included, runs the suite with --no-cov: a pull request gets no Codecov report and the 77% floor is not checked on it. The nightly run on main (and a manual dispatch or a push to a release/** branch) measures coverage in test (ubuntu-latest, 3.11), checks the floor and uploads the report. So a change that takes coverage below 77% merges green and turns the next nightly run red. A plain pytest tests/ measures it and enforces the floor locally (it is in the addopts of pyproject.toml), so run that before you push a change that removes tests or adds untested code.

A draft pull request runs no CI. Nothing starts while a pull request is a draft, whatever is pushed to it. Mark it ready for review and the quick set starts on its current head.

Pull-request CI installs a pinned upstream stack. On a pull request and on a push to main, every job that installs torch or transformers (the test cells, the smoke jobs, Recipe Validation and recipe attach) takes them and the packages they resolve with at the exact versions in .github/constraints/pinned-upstream.txt, so a release upstream cannot turn every open pull request red at once. The one exception is transformers-floor, which installs its own exact floor, and packages outside that file (numpy, pydantic, mlx, torchao, ...) still resolve the newest on every run. The nightly run on main installs with no constraints, and that is where a new release of a pinned package shows first. If your change moves a bound in pyproject.toml past one of those pins, move the pin in the same pull request: tests/test_ci_pinned_upstream.py names the pin, and the header of the constraints file says how to pick the new version.

CI reuses pip downloads selectively. Only the Ubuntu / Python 3.12 test cell caches pip's downloads and wheels via setup-python, keyed by pyproject.toml and .github/constraints/*.txt. Other test cells and the smoke/contract jobs leave pip caching disabled to limit large dependency archives competing with the HF model caches. Every job still runs its installation command on a cache hit, so the nightly run continues to resolve the newest allowed packages and the floor job keeps its constraints. Retained entries can still vary by dependency inputs and Git refs; virtual environments and HF caches are separate.

CI runs the tests in parallel. Every test (...) cell runs the suite with pytest-xdist, one test file per worker process (-n logical --dist loadfile). The tests of one file still run in order in one process, but two files can run at the same moment and in any order. So a test must not depend on another file having run first, on the working directory another test left behind, or on the terminal width of the process: give the console an explicit width in the test instead of setting COLUMNS. To reproduce a failure that shows only in CI, install the plugin (pip install "pytest-xdist[psutil]") and run pytest tests/ -n logical --dist loadfile.

A docs-only pull request runs the suite once. When every file a pull request changes is under docs/ or changelog.d/, or is CONTRIBUTORS.md or CONTRIBUTING.md (a renamed file counts under both names; README*.md, benchmarks/ and everything else do not count), the plan job marks the run docs-only: test (ubuntu-latest, 3.12) runs the whole suite as always, and the other test cells and the three smoke jobs still report, as success, without running a step. If the file list cannot be read, or is not the whole list, the full set runs. A push, the nightly run and a release branch are never docs-only.

A hung test is cut off. The test steps run with --timeout=600 --timeout-method=thread --max-worker-restart=2 (pytest-timeout, part of the dev extra), a 45-minute step limit and a 60-minute job limit. A test that runs for ten minutes fails and the worker running it is killed and replaced; xdist runs that test again on the new worker, and gives up after two replacements. After five minutes (-o faulthandler_timeout=300) the stacks of all threads are written to the log, so a hang says where it is. A healthy test takes seconds. A test that really needs longer marks itself @pytest.mark.timeout(1200). A local pytest run has no cap unless you pass the flags.

See .github/workflows/ci.yml.

Stale CI marks (#1017)

A pull_request workflow run pins refs/pull/N/merge at run creation. actions/checkout fetches that SHA; later movement of main is not re-resolved. gh run rerun replays the frozen merge and cannot produce a different answer. That is why a green tick can describe a main that no longer exists (#763's NameError on _for_terminal) and why a red X can describe a test that main has already fixed. Neither is the contributor's fault, and neither clears by waiting.

If your PR is red for failures you do not touch: merge or rebase onto current origin/main and push. That is the only way to rebuild the merge SHA for the test matrix. Maintainers will not gh pr update-branch onto a fork without asking.

If your PR is green and main has moved: the merge-freshness check is re-posted on every push to main. It rebuilds the merge with git merge-tree --write-tree (no write to your branch) and fails on ruff F821 or a conflict. Lag behind main is reported as a neutral check, not a failure — a 10-commit cap at this repo's merge rate is strict: true with extra steps. It does not re-run the 13-job test matrix.

required_status_checks.strict stays off: flipping it would force a rebase on every merge. A docs-only pre-merge procedure is what maintainers already do by hand and is not a control — GitHub will still merge a CLEAN PR on stale marks. Ask a maintainer to add merge-freshness to the required checks on main so the GitHub merge button honours it. The gate job on a pull_request event runs the PR's own copy of scripts/merge_freshness.py, so it is self-graded. Only the merge-freshness check posted by the reeval job from main's copy is trustworthy.

Before merging anything, gh pr checks returning all-green is necessary and not sufficient. Compare the newest run's created_at against main's commits since, or look at merge-freshness.

Releases

The project follows semantic versioning: MAJOR.MINOR.PATCH

Version Bump Process

  1. Refresh the dev stack, or read the last dependency drift run (#323). A green local suite on a stale environment is a weaker signal than it looks: during v0.72.x the dev box ran torch 2.5.1 / trl 0.19.1 while CI resolved torch 2.13.0 / trl 0.29.1, and six trainers could not build their config at all on what a fresh pip install actually gives a user. Either pip install -U -e ".[dev]" before testing, or check the latest dependency drift run — it installs the current stack weekly, runs the suite against it, and writes the resolved-vs-declared table into its run summary.
  2. Run python scripts/assemble_changelog.py and review the assembled CHANGELOG.md
  3. Update version in pyproject.toml and src/soup_cli/__init__.py
  4. Run full test suite and linting — this includes tests/test_issue621_recipe_snapshot.py, which fails loudly if a schema default changed since the last release without regenerating tests/fixtures/recipe_config_snapshots.json (python scripts/generate_recipe_snapshot.py); a red run here means the fixture is stale, not that the release is unsafe to cut once regenerated and reviewed
  5. Update README.md, the relevant page under docs/, and SECURITY.md (if security-related)
  6. Commit with message: Release v0.X.0
  7. Tag: git tag v0.X.0 && git push --tags
  8. GitHub Actions verifies that no fragment was left behind, then publishes to PyPI on the v* tag (Trusted Publisher / OIDC)

Recognition

We credit every contributor — recognition is free and you earned it.

  • CONTRIBUTORS.md: every merged (or in-tree-adopted) PR adds you here, with your work and PR number.
  • CHANGELOG.md: from the release that ships your change, the entry is tagged (#NNN by @you).
  • Commit trailers: PRs land with a Co-Authored-By: line for you, so the GitHub contributors graph reflects your work even after a squash merge.

If you contributed and aren't credited somewhere, that's a bug — open a PR or an issue and we'll fix it.

Community

  • Issues: Report bugs and request features on GitHub Issues
  • Discussions: Ask questions on GitHub Discussions
  • Discord: Chat with maintainers and other users on Discord — good for quick questions and pairing on a PR; anything worth finding later still belongs in Issues or Discussions
  • Code of Conduct: Please read CODE_OF_CONDUCT.md — it applies on Discord too
  • Security: Report security issues via SECURITY.md — privately, never in Discord

Questions?

  • Check the README for quick start and features
  • Check the docs for the full feature reference and architecture
  • Open a GitHub Discussion for questions

Thank you for contributing!