Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

56.8 KB
Raw Blame

Command Reference

← Back to the Soup README

The full soup command list.

All Commands

soup init [--template chat|code|...|audio]       Create config
soup init --template hipaa|soc2|eu-ai-act|sr-11-7  Compliance-shaped starting config + the commands for that regime (v0.71.35)
soup autopilot [--model <id>] [--data d.jsonl] [--goal <g>]  Zero-config: pick task/quant/LR/epochs (prompts when omitted)
soup advise <data> --goal "..."               Pre-flight decision: PROMPT_ENG / RAG / SFT / DPO / GRPO — run BEFORE spending GPU hours
soup advise compare                           Show prior verdicts from advise history
soup advise explain                           Rubric + evidence trail of the last verdict
soup fetch <name>                             Fetch a ready-to-edit example config from the bundled catalog
soup train --config soup.yaml                 Start training
soup train --config soup.yaml --resume ./output/checkpoint-100  Resume from local checkpoint (not with training.relora_steps)
soup train --config soup.yaml --tensorboard   Train with TensorBoard logging
soup train --config soup.yaml --replay old.jsonl --replay-ratio 0.1  Continual-learning rehearsal: interleave old data so the new task doesn't erase it (sft/pretrain)
soup train --config soup.yaml --fsdp full_shard  Train with FSDP2
soup train --config soup.yaml --deepspeed zero++  DeepSpeed ZeRO++ (quantized comms)
soup train --config soup.yaml --gpus auto|N      Launch on local GPUs through Accelerate
soup train --config soup.yaml --gpus 8 --nodes 2 --node-rank 0 --master-addr 10.0.0.10 --master-port 29500  Multi-node launch; run on each node with its own rank
soup train --config soup.yaml --gpus 4 --no-reexec  Print the launch command without running it
soup train --config soup.yaml --gate evals/gate.yaml  Eval-gated training
soup train --config soup.yaml --push-as user/repo  Auto-push each checkpoint to HF as branch
soup train --config soup.yaml --push-as user/repo --hf-resume  Resume from latest HF checkpoint branch (not with training.relora_steps)
soup train --config soup.yaml --find-lr        LR range finder: write recommended LR JSON
soup train --config soup.yaml --cloud modal|lambda|runpod --gpu a100  Render a cloud GPU controller (plan-only; --cloud-submit submits live)
soup infer --model ./output --input p.jsonl --output out.jsonl [--batch-size N]   Batch inference; default batch size is 1; fp16/bf16 batching may not be bit-exact with batch size 1
soup infer --model ./output --input p.jsonl --output out.jsonl --cuda-graphs   Experimental CUDA graph decode (Qwen2/Llama, one GPU, PyTorch >= 2.14; batch size 1 only)
soup infer --task asr --model <whisper|adapter> --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate]  Whisper transcription + WER/CER
soup chat --model ./output                    Interactive chat
soup infer|chat|diff ... --device cpu|cuda|cuda:N|mps  Pick where the model loads (see below)
soup push --model ./output --repo user/name   Upload to HuggingFace
soup push --model ./output --repo user/name --collection user/coll-abc123  Add to HF Collection
soup merge --adapter ./output                 Merge LoRA with base model
soup merge --adapter ./output --low-memory    Merge tensor by tensor without loading the whole model into RAM
soup merge --adapter ./output --save-format 4bit --no-double-quant  Save a BNB-4bit merge with double-quantization disabled (default on; #321)
soup merge-sharded-fsdp-weights ./shards -o merged.safetensors  Consolidate FSDP shards into one safetensors (v0.71.14; --plan-only previews)
soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only]  Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21)
soup spectrum scan --model <id|path> --top-percent 50 [--modules mlp,attn] [-o patch.yaml]  Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23)
soup train --config sft.yaml  # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps lisa_train_embeddings]  LISA layerwise importance sampling — full-FT quality at LoRA-like memory; lisa_train_embeddings: false freezes embeddings+head+norm for the memory saving (sft or pretrain/transformers/text/quantization=none) (v0.71.34, pretrain #307, #377)
soup train --config sft.yaml  # training.stream_layers: true [stream_source stream_ngram_source stream_buffers stream_read_ahead]  BETA layer streaming — the frozen base streams from CPU RAM/NVMe one decoder layer at a time; Qwen4-Exp PLE rows can stream read-only from original safetensors; embed_tokens + untied lm_head reuse one large-layer device slot; quantization: 4bit streams validated decoder families as NF4, ~4x smaller (sft/dpo/orpo/simpo/kto on transformers+text, 10 archs; grpo/ppo permanently excluded) (v0.72.0; NF4 v0.72.2; disk+batch+accum v0.72.3; preference losses v0.72.4; Qwen4 PLE #602)
soup export --model ./output --format gguf    Export to GGUF (Ollama)
soup export --model ./output --deploy ollama  Export GGUF + auto-deploy to Ollama
soup export --model ./output --format onnx    Export to ONNX
soup export --model ./output --format tensorrt  Export to TensorRT-LLM
soup export --model ./output --format gguf --verbose  Show export phases and stream converter/quantizer logs while they run (#1760)
soup export --model ./output --format gguf --no-install-deps  Never invoke pip; refuse missing conversion packages before merge/toolchain work (#1760)
soup deploy ollama --model m.gguf --name x    Deploy GGUF to Ollama
soup deploy ollama --list                     List Soup-deployed models
soup deploy ollama --remove <name>            Remove model from Ollama
soup deploy hf-space --model user/m --space user/s --template gradio-chat|streamlit-chat  Create HF Space
soup deploy autopilot --target mac-m3|rtx-4090-24gb|...  Pick PEFT+quant+spec-decoding for a hardware target
soup deploy autopilot --list                  List all 10 deploy profiles
soup agent synth --spec api.yaml -o ds.jsonl  Parse OpenAPI/MCP/GraphQL spec into a tool-calling SFT dataset
soup agent train --spec api.yaml --base model  One-shot synth + planned soup train invocation
soup agent eval --spec api.yaml --predictions p.jsonl  Score predicted tool-calls vs spec catalog
soup agent eval --spec api.yaml --predictions p.jsonl --sandbox  Execute each tool-call in the RLVR sandbox: ok/tool_error/timeout/arg_error
soup eval benchmark --model ./output          Evaluate on standard benchmarks
soup eval aider --model openai/gpt-4.1 --output ./aider-results --exercises-dir ./polyglot-benchmark  Run Aider Polyglot in Docker
soup eval custom --model ./output --tasks eval.jsonl   Custom eval tasks from JSONL
soup eval judge --target resp.jsonl           LLM-as-a-judge evaluation
soup eval auto --config soup.yaml             Auto-eval from config
soup eval compare <run1> <run2>               Compare eval results
soup eval leaderboard                         Local model leaderboard
soup eval human --model-a ./a --model-b ./b --input p.jsonl   Human A/B evaluation
soup eval gate --suite gate.yaml              Run eval-gate suite standalone
soup eval quant-check --before X --after Y --tasks t.jsonl  Before/after quantization eval (OK/MINOR/MAJOR verdict)
soup eval design <data> --goal "..."          Draft an eval suite from training data + goal
soup eval discover <data>                       Discover a held-out canary set
soup eval lock <design>                         Freeze a design as a checksummed suite
soup eval coverage <design> --task <task>       Coverage / gap analysis for an eval suite
soup eval against <baseline_run_id> --candidate <run_id>  Run-vs-run regression check (paired bootstrap)
soup eval gate-install --baseline <run_id>      Install a pre-push regression gate
soup eval behavior <run_id>                     Behaviour battery pre/post diff
soup eval capability <run_id>                   Capability profile (MMLU-Pro / GPQA / AIME ...)
soup eval checklist <spec>                      CheckList MFT / INV / DIR tests
soup eval irt-subset <responses>                Minimum-cost eval subset via IRT
soup diagnose <run-id>                        Post-training report card: forgetting / refusal / format / mode collapse / memorization / contamination
soup serve --model ./output --port 8000       OpenAI-compatible API server
soup serve --model ./output --backend vllm    vLLM backend (2-4x throughput)
soup serve --model ./output --backend sglang  SGLang backend
soup serve --model ./output --backend mii     DeepSpeed-MII backend (live)
soup serve --model ./output --speculative-decoding draft-model  Speculative decoding
soup serve --model <m> --auto-spec            Auto-pair draft model for speculative decoding
soup serve --model <m> --auto-quant           Refuses until candidates can be loaded and measured; quantize explicitly instead
soup serve --model <m> --backend vllm --prefix-cache  vLLM prefix caching (RAG/agent)
soup serve --model <m> --structured-output json --json-schema s.json  Constrained output
soup serve --model <m> --structured-output regex --regex-pattern '...'  Regex-constrained output
soup serve --model <m> --dashboard            Live dashboard + /metrics endpoint (transformers + vllm only; warns on sglang/mii)
soup serve --model <m> --backend vllm --max-model-len 8192  Cap the vLLM sequence length (lower it when the KV cache does not fit)
soup serve --model <m> --trace --trace-endpoint http://localhost:4317  OpenTelemetry tracing
soup serve --model <m> --trace-log ./serve.jsonl  Per-request JSONL log + rotation + secret redaction
soup serve --model <m> --record-thumbs ./rl.db  Capture 👍/👎 feedback into local-RL SQLite + POST /v1/thumbs (transformers)
soup serve --model <m> --kv-cache-type bf16|f16|q8_0|fp8  KV-cache type (transformers; q8_0 needs hqq; fp8 = vLLM+Hopper only) (v0.71.14)
POST /v1/adapters/activate/<name>             Hot-swap active LoRA adapter
soup sweep --config soup.yaml --param lr=...  Hyperparameter search
soup sweep --config soup.yaml --sweep-config sweep.yaml  Sweep from a standalone strategy/n_runs/seed/params file
soup diff --model-a ./a --model-b ./b         Compare two models
soup data inspect <path> [--image-dir <dir>]  View dataset stats; optionally override vision image directory
soup data validate <path>                     Check format (auto-detect)
soup data doctor <path> --model <id>          Chat-template compat report: 8 checks, OK/MINOR/MAJOR
soup data doctor <path> --model <id> --show-mask N  Per-token trained/masked colouring via the real collator
soup data lint <path>                         Preference-data linter: length bias, near-dups, chosen==rejected
soup data clean <path> [-o out]               Sanitize dataset: control chars, zero-width spaces, empty turns
soup data convert <path> --to chatml          Convert between formats
soup data merge data1.jsonl data2.jsonl       Combine datasets
soup data dedup <path> --threshold 0.8        Remove duplicates (MinHash)
soup data dedup <path> --semantic             Dedup by embedding cosine — catches rewordings MinHash misses ([train])
soup data topics <path> [--clusters N|auto]   Cluster + c-TF-IDF labels + coverage table + thin-topic warnings ([train])
soup data canary insert <path> -o <out> --manifest <m>  Insert K secrets to later prove memorization (manifest = SECRET)
soup data canary check --manifest <m> --base <model>    Rank each secret's loss vs never-inserted controls; exit 2 = leak
soup data stats <path>                        Extended statistics
soup data generate --prompt "..." --count 100  Generate synthetic data
soup data generate ... --provider ollama      Use local Ollama instance
soup data generate ... --provider anthropic   Use Claude API
soup data generate ... --provider vllm        Use local vLLM server
soup data generate ... --template code        Domain templates (code/conversation/qa/preference/reasoning)
soup data generate ... --quality-pipeline     Auto validate + filter + dedup
soup data augment <path> --strategy rephrase|translate|style [--provider ollama|vllm --model <m> --base-url <url>]  LLM-driven augmentation
soup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
soup data review prefs.jsonl --sample 10      Preview preference pairs
soup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
soup data sample <path> --n 1000             Random sample subset
soup data sample <path> --n 1000 --strategy diverse  Cluster-based diverse sampling
soup data sample <path> --n 1000 --strategy hard     Sample hardest examples
soup data sample <path> --pct 10             Sample by percentage
soup data split <path> --val 10 --test 10    Split into train/val/test
soup data split <path> --val 500 --absolute  Split with absolute counts
soup data split <path> --val 10 --stratify category  Stratified by field
soup data split <path> --val 10 --stratify-semantic --num-clusters 5  Semantic stratified split
soup data search "code instructions"         Search HuggingFace Hub for datasets
soup data search --sort likes --limit 10     Sort and paginate search results
soup data preview teknium/OpenHermes-2.5     Preview remote dataset metadata
soup data download user/dataset -o data.jsonl  Download HF dataset as JSONL
soup data download user/ds --samples 1000    Stream first 1000 samples
soup data register --name my-ds --path d.jsonl --format alpaca  Register dataset
soup data unregister --name my-ds            Remove from registry
soup data push --input d.jsonl --hf-dataset user/name  Upload local JSONL as HF dataset
soup data push --input d.jsonl --hf-dataset u/n --hub modelscope|modelers  Upload to an alternative hub
soup data registry                           List all registered datasets
soup data demo                                List bundled demo JSONL fixtures
soup data demo alpaca_demo --output ./d.jsonl  Copy a bundled demo JSONL fixture
soup data ingest <file>                         PDF/DOCX/MD/TXT -> JSONL (PDF/DOCX: pip install "soup-cli[documents]")
soup data preprocess <config>                   AOT-tokenize and cache for reuse across runs
soup data recipe <path>                         Validate / execute a Data Recipe DAG
soup data mix                                 BETA mixture-weight optimiser (proxy runs)
soup data forge --docs ./docs --task sft --target-rows 1000  Synthetic data pipeline + provenance
soup data forge --docs ./docs --hub modelscope --teacher owner/name  Pre-fetch the teacher from an alternative hub
soup data score --input rows.jsonl            Composite quality scorecard (PII + keyword triage + lang + edu)
soup data score --input rows.jsonl --benchmark-file benchmark.jsonl --threshold 0.8  Score with real n-gram comparison texts
soup data score ... -b gsm8k --benchmark-file benchmark.jsonl  Labels are validated only, not corpus selectors; all file texts are compared
soup data decontaminate --input rows.jsonl --benchmark-file benchmarks.jsonl  Drop rows overlapping the benchmark texts in that file
soup data toxicity --input rows.jsonl -o tox.jsonl  Flag abuse-keyword matches (heuristic)
soup data langdetect --input rows.jsonl -o tagged.jsonl  Tag each row with language code
soup data pii --input rows.jsonl -o pii.jsonl  Flag rows containing email/phone/SSN/credit-card
soup data educational --input rows.jsonl -o scored.jsonl  Score educational value per row
soup train --config soup.yaml --tracker mlflow  MLflow / SwanLab / Trackio integration
soup profile --config soup.yaml              Estimate memory/speed before training
soup profile --config soup.yaml --gpu a100   Estimate for specific GPU
soup profile --config soup.yaml --json       Machine-readable output
soup cost --config soup.yaml                 Estimate training cost in USD across providers
soup cost --config soup.yaml --gpu H100      Estimate training cost for specific GPU
soup adapters list ./output/                 Scan for LoRA adapters
soup adapters info ./output/checkpoint-500/  Show adapter metadata
soup adapters compare adapter1/ adapter2/    Compare two adapters
soup adapters audit <adapter> --config soup.yaml [--json]   Compare a finished run's adapter_config.json against the config that started it (#762); a setting the record cannot speak to is 'unknown', never 'ok'. Exit codes 0=agreement, 2=DIVERGED, 1=usage error, so CI can tell a verdict from a mistyped path.
soup adapters branches                        List snapshotted branches
soup adapters checkout <name>                   Restore a snapshotted branch's config
soup adapters diff <a> <b>                      Per-layer ΔW Frobenius diff + effective-rank drift
soup loop init <model> --eval <s> --baseline <b> [--pre-wired] [--thumbs-db <db>]  Create .soup/loop.yaml (data flywheel; --pre-wired = real stages; --thumbs-db = harvest `serve --record-thumbs` thumbs)
soup loop status                              Counters + status + pre_wired flag
soup loop watch [--detach] [--max-iterations N] [--pre-wired] [--pack-cans] [--tool-auth-token T]  Harvest → train → gate → deploy daemon (pre-wired stages + Soup Can packing)
soup loop pause                              Atomic status flip: pause watch daemon at next iteration boundary
soup loop resume                             Atomic status flip: resume a paused loop
soup loop canary <adapter> --traffic 5%      Route keyed serve traffic to a loaded adapter; watch auto-rolls back on MAJOR
soup loop replay [<iter-id>] [--extract <dir>]  Replay / unpack a recorded iteration manifest
soup serve --model m --adapters chat=./c --adapters code=./d  Multi-adapter serving
soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
soup migrate --from axolotl config.yml        Import config from Axolotl
soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
soup recipes list                             List all 171 ready-made recipes
soup recipes show llama3.1-8b-sft            Print recipe YAML
soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
soup recipes search "reasoning"              Search by keyword/task/size
soup recipes verify                           Attach every shipped config on the meta device; exit 2 if one cannot (#1116)
soup recipes verify --config soup.yaml        Same check for one config, before downloading weights
soup recipes verify --no-templates            Recipes only; skip src/soup_cli/templates and examples/
soup recipes verify --json                    One JSON row per config on stdout; advisories go to stderr
soup registry push --run-id <id> --name n --tag v1  Register run
soup registry push ... --data train.jsonl     Also record that file's SHA-256 as data_hash
soup registry push ... --idempotency-key k    Retry-safe: same key + same registration -> existing entry
soup registry list [--name n] [--tag v1]     List registry entries
soup registry show <ref>                      Entry details + artifacts + ancestors
soup registry diff <a> <b>                    Side-by-side config + eval delta
soup registry search "medical"                Search name/base/task/notes
soup registry promote <ref> --tag prod        Tag an entry (e.g. promote to prod)
soup registry delete <ref> --yes              Remove entry (cascades)
soup history <name>                           Lineage DAG tree for a name
soup can pack --entry-id <id> --out r.can     Pack registry entry as .can
soup can inspect r.can                        Preview manifest without extracting
soup can verify r.can                         Verify schema + config parseability
soup can fork r.can --out fork.can --modify training.lr=5e-5  Fork + re-pack
soup can run r.can --yes [--deploy] [--env-capture env.txt]  Run a .can end-to-end
soup can publish r.can --hf-hub user/name    Publish .can to HF Hub as dataset
soup runs                                     List training runs
soup runs show <run_id>                       Run details + loss graph + cost (shows an Error: line for failed runs, and distinguishes terminated/launching from running; #767)
soup runs compare <run_1> <run_2>             Compare two runs
soup runs replay <run_id>                     Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results)
soup runs clean [<run_id>] [--all]              Clean redundant checkpoint files
soup runs curriculum-curve <run_id>             BETA curriculum bucket-weight plot
soup runs delete <run_id>                       Delete a run and its metrics
soup why [run_id]                             Explain training anomalies (heuristic)
soup rewind [run_id] [--step N]               Name the dataset rows behind a loss spike
soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no regression on the bundled suite (exit 0=SHIP / 2=DON'T / 3=usage / 1=runtime) (v0.71.25; leg-2 real + usage-off-2 v0.71.38)
soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
soup ship ...  # leg-2 default = 8 bundled offline suites (MCQ/arithmetic/over_refusal + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38; +mini_over_refusal v0.73.2)
soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 override + recorded base scores
soup ship ... --emit-evidence ev.json  Re-serialise the scores as replayable --evidence input (output-is-input, #312) (v0.71.39)
soup ship ... --config soup.yaml  Read eval.ship gate defaults; --evidence GATES on provenance.config_sha and numerics family, --emit-evidence STAMPS both (v0.71.39 / #367); live-loads base/tuned at training.quantization when it is 4bit/8bit, else full precision (#367)
soup ship ... --push owner/repo#N  Post the verdict as a GitHub PR comment (best-effort; never flips the exit code) (v0.71.39)
soup ship ... --noise-floor N  Re-run base model N times; per-axis floor = max-min spread; gate at max(threshold, floor); leg-1 measured in every --task-mode (judge modes cost N judge passes, floor labelled decode+judge) (v0.73.2)
soup ship ... --chat-template-kwargs '{"enable_thinking":false}'  Pass template variables equally to base/tuned live generation; JSON object, live-only (#1758)
soup card <registry-id> -o MODELCARD.md       HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
soup push --model ./out --repo you/m --card <registry-id>  Upload that registry-driven card as the README (HF only) (v0.71.35)
soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--config soup.yaml] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35); --config binds the gate to a committed config so it refuses stale evidence (v0.71.39)
soup mcp serve                                MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
soup mcp serve --allow-mutating               Also expose plan-only train_start / export tools (never execute) (v0.71.28)
soup mcp serve --allow-execute                Implies --allow-mutating; enables train_execute / export_execute via server confirmation tokens
soup mcp serve --transport sse [--host H --port N]  Serve the same registry over HTTP+SSE instead of stdio; binds 127.0.0.1 and requires a Bearer token (#296)
soup mcp serve --transport http [--auth-token T]    Same over the streamable-HTTP transport (/mcp); --auth-token pins the token instead of generating one (#296)
soup mcp serve --transport sse|http --allow-execute   REFUSED - gated execution spawns real processes and is stdio-only (#296)
soup mcp runs reconcile [--expunge-launching] [--older-than-seconds N]  Clear stale launching rows left by a crashed server; refuses if any candidate PID is still alive (#402)
soup shrink --model <id|path> --drop-ratio 0.25 --calib c.jsonl -o shrunk  Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29)
soup shrink ... --drop-layers N --heal h.jsonl --heal-steps 200 --device cpu  Drop N layers + distill-heal (fuse LoRA back to one dense model)
soup shrink ... --tolerance 0.10 --plan-only [--attach-to-registry <id>]  Ppl-regression tolerance / print importance table only / registry attach
soup draft measure --target <m> --draft <d> --prompts p.jsonl  Draft acceptance rate + real plain-vs-assisted tok/s (exit 0 measured — a best-effort assisted-arm failure stays 0 and is recorded as `assisted_status`; 2 below `--min-acceptance`; 1 error before results) (v0.71.33)
soup draft measure ... --min-acceptance 0.6 -o report.json  Exit 2 below the floor (CI gate) / write the JSON report (fields incl. `assisted_status`: pending/complete/untimed/crash/interrupted)
soup draft measure ... --sweep-k 1,2,3,5,8  Time assisted generation at each draft length (at most 8, each 1..64). The JSON adds `tok_s_draft`, `draft_status`, `latency_ratio`, `breakeven_acceptance`, `modelled_best_k`, `modelled_speedup_best_k`, `k_sweep` and `measured_best_k` (#843)
soup draft distill --target <tuned> --draft-base <tiny> --data d.jsonl -o draft/  Distil a DENSE speculative-decoding draft + register it (v0.71.33)
soup draft distill ... --steps N --device cpu --force --plan-only  Training budget / device / overwrite -o / render the config only
soup draft list                               List local drafts that `soup serve --auto-spec` will pick up (v0.71.33)
soup reward synth refs.jsonl -o reward.py     Synthesize a deterministic reward verifier from gold outputs (v0.71.40)
soup reward synth ... --kind numeric|json_schema|regex|tool_call  Force a verifier family (default: auto-detect)
soup reward synth ... --plan-only             Report the induced spec + calibration plan; write nothing
soup reward synth ... --output-report r.json --min-discrimination 0.5  Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error)
soup reward stress reward.py --references golds.jsonl  Adversarially probe a verifier for gameability — classic and structure-preserving attacks (v0.71.41)
soup reward stress verifiable --verifiable-domain math --references golds.jsonl  Probe a builtin verifier instead of a .py file
soup reward stress ... --attacks empty,length,repetition,sentinel,wrapped_junk,answer_spray --sentinel GOLD --threshold 0.5 --max-gameable 0.0  Tune the attack set / accept threshold / tolerance
soup reward stress ... --output-report r.json  Save the per-attack report JSON (exit 0 robust / 2 gameable / 1 error)
soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
soup --log-level quiet|normal|verbose|debug   Global logging tier (Rich-formatted)
soup ui [--port 7860]                         Web UI (experiments, training, data); loopback Host must name the bound host or localhost / 127.0.0.1 / [::1] with the bound port (omitted = HTTP 80), otherwise 403
soup ui --public [--auth-token T]             Phone-scannable Web UI (v0.53.9); /docs + /openapi.json are loopback-only
soup tokenizer train --input c.jsonl --vocab-size N  Train BPE tokenizer (v0.53.9)
soup bench <model>                            Inference speed + memory (same as `soup bench infer <model>`)
soup bench infer <model> --p50 --p95          Bench with tail-latency percentiles (v0.53.9)
soup bench infer <model> --cuda-graphs         Bench with experimental CUDA graph decode
soup bench train --config soup.yaml --steps 20 --warmup 3 -o bench-train.json  Timed SFT steps; exits 1 when the model was not training (#836)
soup bench <model> --backend auto             Auto-detect transformers/mlx backend (v0.53.9)
soup serve --model ./output --reasoning-parser deepseek-r1   Strip <think> blocks from responses (v0.53.9)
soup doctor [--nccl] [--disk] [--config F]    Check environment (optionally check NCCL bandwidth, media type; --disk ~9s cold / ~2.4s warm).
                                              --config also reports which settings that config switches on (a `false` or unset value is not reported; `seed: 0` is) that its task/backend does not read (#755, #1330); exits 2 if it cannot be read, and exits 1 when a required core dependency is missing, or when any installed package is beyond its declared ceiling — core or [train] (#828, #874).
soup monitor                                  NVIDIA / Apple Silicon GPU monitor: util / temp / VRAM / power
soup quickstart [--dry-run]                   Full demo
soup plugins list|install|enable|disable      Manage Soup plugins
soup llama cli|mtmd-cli|gguf-split|server|quantize ...  Proxy to the llama.cpp binaries
soup quantize <model> --to <fmt>              Quantize a model — ergonomic alias for `soup export --format <fmt>`
soup bom emit --name <n> --base-sha <hex> --config-sha <hex> --format cyclonedx|spdx|both  CycloneDX ML-BOM / SPDX AI bill of materials
soup adapters scan <adapter>                  Spectral backdoor scan (rank-1 dominance + outlier detection)
soup adapters sign <adapter> [--backend unsigned|ed25519|sigstore] [--key <pem>|--generate-key <pem>] [--interactive-oidc]  Merkle manifest + ed25519/Sigstore sign
soup adapters verify <adapter> [--strict] [--public-key <pem>] [--cert-identity <san> --cert-oidc-issuer <url>]  Verify manifest + ed25519/Sigstore signature
soup adapters check <adapter> [--json]        Is the adapter trained? Total/per-layer ||dW||_F, live fraction, all-zero lora_B, leaked .inner. keys; ends in alive or inactive: <reason> (exit 0 / 2; 1 = usage error)
soup adapters check-safetensors <adapter> [--strict]  Refuse pickle / PyTorch-classic weights
soup adapters merge ... [--license <id>] [--license-override <reason>] [--allow-unscanned]  License + backdoor-scan gates (auto-detect license; scan FAIL refused)
soup adapters arithmetic "coder + 0.5*math - toxic" --adapter coder=<p> --adapter math=<p> --adapter toxic=<p> -o <out> [--allow-unscanned --allow-cross-base]  Task-vector algebra over LoRA adapters (add/scale/negate; same-rank; scan + same-base gated) (v0.71.34)
soup attest emit ... [--sign unsigned|ed25519|sigstore] [--key <pem>] [--interactive-oidc] [-o att.json]  in-toto/SLSA-3 attestation; Sigstore requires -o
soup attest verify <statement> --signature <sig> [--public-key <pem>] [--cert-identity <san> --cert-oidc-issuer <url>]  Verify ed25519/Sigstore attestation signature
soup airgap-bundle --model <m> --output <out.tar> [--repro-receipt <r.json>]  Signed tarball for data-diode transfer (embeds repro-receipt)
soup train --config soup.yaml --annex-xi <out.md|out.pdf>  EU AI Act Annex XI/XII doc (markdown or PDF; top_domains auto-filled)
soup train --config soup.yaml --track-energy [--energy-country USA]  codecarbon offline kWh/CO2 → annex-xi (pip install soup-cli[carbon])
soup train --config soup.yaml --track-energy --energy-out <energy.json>  persist measurement for `soup bom emit --energy <energy.json>`
soup train --config soup.yaml --repro-receipt <out.json>  SR 11-7 reproducibility receipt
soup can pack --entry-id <id> --out r.can --attest <statement.json>  Embed in-toto Statements into a v3 can manifest
soup audit-log tail / rotate  Tail / rotate the per-command HIPAA/SOC2 audit log (~/.soup/audit.jsonl)
soup --no-audit-log <cmd> / SOUP_NO_AUDIT_LOG=1  Opt out of the per-command audit line
soup eval unlearning <run-id> --benchmark tofu|muse|wmdp  Forget Quality + Model Utility + PrivLeak verdict
soup edit set --base <m> --method rome|memit|alphaedit|grace --subject "..." --target "..." [--output <dir>] [--device cpu] [--governor/--no-governor] [--registry-id <id>] [--cov-corpus <jsonl|txt>]  Live surgical knowledge edit (GPT-2 Conv1D + Llama; --cov-corpus = covariance-preconditioned ROME, rome-only; --plan-only available)
soup edit diff <before-run> <after-run> --probes p.jsonl --before-model <m> --after-model <m>  Knowledge-injection diff (live before/after generation across probe prompts)
soup train  # task: unlearn  NPO/SimNPO/RMU unlearning from data.forget_set (+ optional data.retain_set)
soup train  # data.format='raft'  Answer-only span-mask RAFT training (golden+distractor docs, [doc-N] citations); generator-stage configs auto-link the latest RA-DIT retriever
soup ra-dit --retriever-config <r.yaml> --generator-config <g.yaml> [--retriever-model <m>] [--plan-only]  One-shot two-stage RA-DIT: train retriever → record pairing → train generator
soup eval citation <data> [--style bracket|inline|footnote] [--shuffle-seed N] [--output o.json]  Citation precision/recall/F1 over predictions or RAFT rows
soup steer train --base <m> --method caa|iti|repe --name <id> --pairs <jsonl>  Fit a CAA/ITI/RepE activation-steering vector from {positive, negative} pairs
soup steer apply --name <id> --strength <s>  Preview a stored steering vector
soup steer list                              List locally-stored steering vectors
soup serve --steer <name> [--steer-strength <s>]  Apply a steering vector at decode time via a forward hook (transformers backend)
soup serve --bank <bank.json> [--bank-strength <s>]  Multi-tenant VeRA/VB-LoRA serving; active user per request via X-User-Id header, ContextVar-isolated (v0.71.12 / v0.71.17)
soup serve --mole <dir>                              Serve a trained MoLE: base + N frozen task LoRAs + mole_gate.pt, blended per-token at decode (transformers-only) (v0.71.17)
soup ingest --source langfuse|langsmith|helicone|openpipe|otel|openai-stored --logs <jsonl>  Universal trace importer (6 SaaS adapters → normalised JSONL)
soup ingest --source langfuse --pull [--since 7d --max-pages 100 --allow-private-host]  Live pull of Langfuse generations (Observations API v2; LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST) (#204)
soup prune-prompt --input <jsonl> --output <jsonl> --min-frequency 0.95  Detect + strip shared system-prompt prefix
soup prune-prompt ... --tokenizer <id-or-path>  Tokenizer-aware prefix detection (decodes remaining ids, boundary-safe)
soup data active-sample --input <jsonl> --output <jsonl> --budget N  Top-N uncertain prod traces for human review
soup ab --input <jsonl> --metric latency|judge_score|retry_rate  Two-sided mSPRT sequential A/B (decision: continue / reject_h0 / accept_h0; a reject_h0 reports direction better / worse; valid when re-run after every new row; the first 5 rows per arm set the --effect-size scale)
soup ingest|prune-prompt|ab|data active-sample ... --slack-url <https> | --discord-url <https>  Shared SSRF-validated webhook on completion
soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2  Rolling-KL drift alarm (exit 3 on drift)
soup drift-alarm ... --slack-url <https> | --discord-url <https>  Optional SSRF-validated webhook on drift detected
soup tunability --list                                   List built-in candidate-base catalogue
soup tunability --dataset <jsonl> [--candidates a,b,c]   Probe candidate bases + Pareto frontier report
soup tunability --dataset <jsonl> --live [--device cpu]  LIVE per-candidate LoRA probe (loads each repo)
soup plan --config soup.yaml                             Pre-flight summary + write soup.tfstate
soup apply --config soup.yaml [--dry-run]                Lock-and-execute; refuses on drift (exit 3)
soup env lock | status | check                           Hermetic env lockfile + ABI drift + declared-bound violation detection (exit 3)
soup env fix [--format uv-pip|requirements] [--output req.txt]  Render a reproducible install plan from soup-env.lock (print-only)
soup completions bash | zsh | fish                       Shell completion script (sourceable via eval)
soup license-advisor --target b2c|defense|embedded       Recommend license-clean base for deploy target
soup license-advisor ... --license <id> --mau N          Per-license downstream-risk check (exit 3 on block)
soup probe sae-diff <sae> <pre.json> <post.json> [--top-k N]  SAE feature diff between pre/post-FT activations (v0.66.0)
soup probe sae-diff <repo> <pre.json> <post.json> --auto-download  Fetch an allowlisted SAE into ~/.soup/sae-cache (v0.71.8)
soup probe sleeper <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  Sleeper-agent defection probe; --weights = real calibrated probe (v0.66.0; v0.71.8)
soup probe truth <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  TruthfulQA-style honesty probe (v0.71.8)
soup probe harm <base> [--evidence ev.json] [--weights w.npz] [--output o.json]  HarmBench-style misuse probe (v0.71.8)
soup probe interference <losses.json> [--output o.json]  Pairwise N×N adapter interference matrix (exit 2 on MAJOR; v0.66.0)
soup probe interference --measure <eval.jsonl> --base-model <m> --adapter name=path ... [--device cpu]  Auto-measure live interference (v0.71.8)
soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0; +truth/harm v0.71.8)
soup probe pack --list                        List bundled probe-pack bases (v0.66.0)
soup train --capture-activations <layer> --capture-prompts <jsonl>  Post-train SAE-diff-ready per-token activation snapshot (v0.71.8)
soup adapters blame ... --top-k 50            Live DataInf-style influence runner (v0.66.0, closes #171)
soup adapters merge ... --strategy cmaes --eval <s> --budget 1h  CMA-ES evolutionary merge — live loop (v0.67.0 schema / v0.71.4 live)
soup adapters merge ... --canary <suite.json> [--strict-verdict]  Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4)
soup adapters pr <title> --base-sha <hex> --adapter <path>  GitHub-shaped adapter PR Markdown / JSON (v0.67.0)
soup adapters pr <title> ... --push owner/repo#N  Post the PR as a GitHub comment via gh api (v0.71.4)
soup adapters branch <name> --from-registry <id> | --attach-to-registry <id>  Branch ↔ Registry lineage (v0.71.4)
soup adapters bisect <ckpt>... --eval-command "..."  Binary search over training history (v0.67.0)
soup lock write --base-sha <h> --dataset-sha <h> --env-hash <h>  Write soup.lock (v0.67.0)
soup lock write --base-sha <h> --dataset-sha <h> --env-lock soup-env.lock  Auto-derive --env-hash from soup-env.lock (v0.71.1)
soup lock show [PATH]                        Print tracked lock file
soup lock check [PATH] --base-sha <h> --dataset-sha <h> --env-hash <h> --base-model <m>  Refuse with exit 2 on drift, 3 on usage/missing lock
soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only]  DSPy / GEPA / TextGrad prompt-program compiler — live (v0.71.13; pip install "soup-cli[compile]")
soup distill-prompt --traces <jsonl> --teacher <m> --student <m> --strategy sft|preference|kl [--provider ollama|anthropic|vllm] [--base-url <url>] [--temperature F] [--max-rows N]  Distill prompt-heavy traces via a live teacher (v0.71.13); a teacher/student call that fails or returns nothing drops that trace, and the command exits 1 without writing the file when nothing is left
soup compile-tools <spec.json|yaml> --eval <jsonl> [--optimizer textgrad|gepa] [--task-lm <m>] [--reflection-lm <m>] [--engine <name>] [--plan-only]  TextGrad / GEPA tool-schema optimiser — live (v0.71.13; pip install "soup-cli[compile]")
soup apple-adapter <source-dir> --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output <dir> [--sign] [--plan-only]  PEFT LoRA <-> mlx-lm adapter conversion — live (v0.71.21; *-to-apple upstream-gated exit 3)
soup local-rl init --db <path>                Create personal-LLM flywheel SQLite schema (v0.68.0)
soup local-rl status --db <path>              Print interactions / thumbs-up / thumbs-down counters
soup local-rl record --db <path> --prompt <q> --response <r> --thumb up|down  Append thumbs record
soup local-rl harvest --db <path> -o <pairs.jsonl>  Harvest DPO pairs from thumbs into JSONL
soup local-rl train --db <path> --model <id> --once [--train-method dpo|kto|orpo] [--min-pairs N] [-o <dir>]  Ad-hoc DPO/KTO/ORPO train from harvested thumbs — live (v0.71.13)
soup local-rl train --db <path> --model <id> [--scheduler-dir <dir>] [--hour H] [--minute M] [--train-method dpo|kto|orpo] [--min-pairs N] [-o <dir>]  Render a systemd/launchd nightly-train scaffold (no --once) that runs `--once` with the same flags (v0.71.13)
soup build <manifest.yaml> [--dry-run] [--output-dir <dir>]  dbt-for-SFT DAG: validate + plan + live materialise (v0.69.0; live v0.71.6)
soup expect <data.jsonl> <suite.yaml> [--judge URL]  Expectations suite: PII / token-length / refusal / judge (v0.69.0)
soup data gen-magpie --base <m> --provider ollama|vllm --target N --output <jsonl> [--base-url <url>] [--quality-filter]  Magpie synthetic generator — live (v0.69.0; live v0.71.6)
soup data best-of-n (--base <m> | --provider ollama|vllm --model <m> [--base-url <url>]) --prompts <jsonl> --n 8 --judge <url> -o <sft.jsonl> [--emit-pairs <dpo.jsonl>] [--resume] [--checkpoint <journal.jsonl>] [--manifest <manifest.json>]  Best-of-N rejection sampling with durable per-prompt recovery and manifest-last publication
soup data best-of-n (--base <m> [--revision <rev>] | --provider ollama|vllm --model <m>) --prompts <jsonl> --n 8 --export-candidates <jsonl> [--checkpoint <jsonl>] [--resume]  Resumable sampling-only phase; no judge is constructed
soup data best-of-n --candidate-artifact <jsonl> --judgments <jsonl> -o <sft.jsonl> [--emit-pairs <dpo.jsonl>] [--manifest <json>]  Validate offline judgments and materialize byte-stable training rows; a final manifest binds the requested output set
soup data evolve --input <seeds.jsonl> --provider ollama|vllm --model <m> --strategy depth|breadth --rounds N -o <jsonl>  Evol-Instruct (WizardLM) instruction evolution (v0.71.31)
soup data persona-mix --prompts <jsonl> --n N --output <jsonl>  Persona-Hub diversity sampler (v0.69.0)
soup data brain-rot <data.jsonl> [--strict]   Brain-rot detector — arXiv 2510.13928 (v0.69.0)
soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only]  Iterative DPO loop driver — LIVE sample→score→pair→train (v0.70.0; live v0.71.11)
soup train --reward-hack-detector info_rm|rm_ensemble [--reward-hack-halt]  Reward-hacking detector for GRPO — LIVE callback (v0.70.0; live v0.71.11)
soup train --reward-hack-mitigation off|log_only|kl_control|pid_lagrangian  Closed-loop reward-hacking auto-mitigation (detect → raise KL/β → rollback → early-stop); GRPO only, requires --reward-hack-detector; refused on task: ppo (v0.71.26)
soup train --config soup.yaml  # training.uld_strategy: wasserstein_aligned  Cross-tokenizer ULD on task='distill' (different tokenizers) — LIVE (v0.71.18)
soup train --config soup.yaml  # training.minillm_enabled: true [minillm_teacher_mix_ratio 0.3]  MiniLLM reverse-KL distillation, config-only — LIVE; offline mix 0 rejected (v0.70.0; live v0.71.11; #692, #979)
soup train --config grpo.yaml  # training.rl_checkpoint_save_every_steps: N [rl_checkpoint_keep_last: N; rl_checkpoint_include_optimizer: true]  Mid-epoch checkpoint for GRPO/PPO — LIVE (v0.70.0; live v0.71.11)
soup train --config grpo.yaml  # training.echo_trap_enabled: true [echo_trap_threshold: 0.6; echo_trap_halt: true]  RAGEN echo-trap detector for GRPO — LIVE callback (v0.70.0; live v0.71.11)
soup train  # task='moe_lora_routing' + mole_task_adapters  MoLE per-token gate over N frozen task LoRAs (gate-only train) — LIVE (v0.71.12)
soup train  # task='distill' + distill_mode=token|sequence  Token logit-KL or sequence-level teacher-continuation KD — LIVE (v0.71.12)
soup train  # task=classifier|reranker|cross_encoder + lora  LoRA-adapter classifier (frozen encoder) — LIVE (v0.71.12)
soup train  # use_mod | expand_layers  Mixture-of-Depths / LLaMA Pro (Llama/Qwen/Mistral) — LIVE (v0.71.12). use_mod is applied only on the resident transformers LoRA route of task sft or tts (text, no layer streaming, lora.r >= 1, no LISA or Spectrum) and on transformers pretrain (LoRA or LISA); every other route is refused at config load (#1752)
soup train  # use_longlora  LongLoRA S² is refused at config load: the override leaked future tokens and applied no S² grouping; use rope_scaling_type with plain LoRA (#1240)
soup train  # task='tts' + tts_family + modality='audio_out'  Codec-string SFT + emotion templating — LIVE (v0.71.20); native Sesame CSM with raw audio + 32 Mimi codebooks (#265); Spark/Oute raw-audio refused
soup train  # task in {sft,tts} + moe_expert_quant=nf4|int8_rowwise [+moe_lora]  bnb per-expert quant of fused-MoE experts (CUDA); refused on other tasks, which never applied it (#798) — LIVE (v0.71.20)
soup train  # task in {sft,tts} + train_router_only=true [+moe_lora]  Freeze MoE experts, train only the gating router; refused on other tasks (#798). moe_lora needs lora.dropout: 0.0 on fused-expert MoEs — LIVE (v0.71.20)
soup train  # quantization='bitnet_1.58'  BitNet 1.58 training is not implemented; config load refuses it
soup export --model ./output --format bitnet|tq1_0  BitNet 1.58 TQ1_0 ternary GGUF via llama.cpp — LIVE (v0.71.20)
soup version [--full] [--json]                Show version (--full: system info, --json: JSON output)
soup --version                                Show version (prints the same text as soup version)
soup --verbose <command>                      Full traceback on errors

Choosing the device for soup infer, soup chat and soup diff

--device decides where these commands load the model. Without it, the device is auto-detected and the same rules apply to what was detected (this also covers soup bench, which loads through the soup infer path).

--device Placement Dtype
cpu pinned to the CPU float32 (twice the RAM of float16: a 7B model needs about 28 GB)
cuda device_map="auto": accelerate may use every visible GPU and spill to the CPU if the model does not fit float16
cuda:N pinned to GPU N float16
mps pinned to the Apple GPU float16
mlx runs on the CPU on this path float32

Values are case-insensitive. Any other value (gpu, auto, cuda:0,1, ...) is refused with an error instead of being ignored.

Interactive Autopilot (soup autopilot)

When standard input is an interactive terminal and any of --model, --data, or --goal are omitted, soup autopilot prompts for the missing values:

  • Model: Base model Hugging Face repository ID (or local path). Blank entries are refused.
  • Data: Local training dataset path (JSONL). Must exist under current working directory.
  • Goal: One of chat, reasoning, code, classification, tool-calling, alignment, or domain-adapt.

When run with all three options supplied, or when standard input is not a terminal (e.g. CI, scripts, or pipelines), soup autopilot never prompts: complete flags execute directly, and omitted required flags exit with code 2 and the standard Missing option error.

Best-of-N recovery and publication

soup data best-of-n appends and synchronizes one checkpoint record after each fully sampled and judged prompt. The default journal is <output>.checkpoint.jsonl; override it with --checkpoint. If a sampler or judge backend fails, Soup reports the completed prompt count and journal path. Rerun the same command with --resume to reuse that exact prefix without sampling it again. The prompt sequence and generation options are bound into the journal, so changing them makes resume fail closed.

Final SFT and optional DPO JSONL files are published only after every prompt is complete. The manifest (default <output>.manifest.json, or --manifest) is written last and records their row counts and SHA-256 digests. Consumers should treat only files listed by that final manifest as committed output. Invalid local data such as a non-finite judge score remains a validation failure rather than being presented as a recoverable backend outage.

For offline Best-of-N, the candidate and judgment artifacts are the recovery checkpoint. Offline materialization rejects --resume and --checkpoint; rerun the exact materialization command after a late failure. The final manifest is written last and is the only commit marker; consumers must verify it and use only the SFT/DPO files it lists.

Fine-tune from your coding agent (MCP)

soup mcp serve runs a Model Context Protocol server over stdio (default) or a local SSE / HTTP listener, so any MCP client — Claude Code, Cursor, Cline, Continue — can drive Soup conversationally. Install the extra first:

pip install "soup-cli[mcp]"

Register it with your client. For Claude Code (.mcp.json in the repo) or the Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "soup": { "command": "soup", "args": ["mcp", "serve"] }
  }
}

Remote or multi-client: the SSE and HTTP transports

stdio suits a client that spawns Soup as a subprocess. For a client on another machine, or several clients sharing one server, run a listener instead:

soup mcp serve --transport sse                 # GET /sse + POST /messages/
soup mcp serve --transport http                # streamable HTTP on /mcp
soup mcp serve --transport sse --host 127.0.0.1 --port 8765 --auth-token "$TOKEN"

Both bind 127.0.0.1:8765 by default and require a Bearer token on every request. With --auth-token omitted, a fresh one is generated at startup and printed to stderr; it is 16-128 urlsafe-base64 characters, the same shape soup ui uses. Point a client at it with a header:

{
  "mcpServers": {
    "soup": {
      "url": "http://127.0.0.1:8765/sse",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}

The registry is identical across transports — same tools, same schemas, same --allow-mutating / --allow-execute gating. Only the wire changes.

Security:

  • The token is required and there is no opt-out. A loopback listener is reachable by every process on the machine, not just by you.
  • The token travels in the header, never the URL. There is deliberately no query-string fallback, so it cannot be captured in proxy or access logs.
  • DNS-rebinding protection is on. A request whose Host is not the address the server bound to is refused with 421, a foreign Origin with 403. This is the gate the Bearer token cannot be: a web page the operator merely visits attaches no Authorization header, but its request still arrives at the port.
  • A non-loopback --host prints a warning, and a wildcard bind (0.0.0.0) prints a second one — with no single advertised name there is nothing to pin, so the Host check degrades to accepting any Host.
  • --host / --port / --auth-token are refused under --transport stdio rather than silently ignored: stdio has no listener and nothing to authorize.

The network transports need mcp >= 1.10.0 — streamable HTTP landed in 1.8.0 and rebinding protection in 1.10.0 — which is the floor soup-cli[mcp] pins.

The server exposes 14 read-only tools — advise, data_inspect, data_validate, data_score, data_doctor, recipes_search, recipes_show, runs_list, runs_show, registry_list, registry_show, profile, diagnose_evidence, ship_evidence — each returning JSON. Two plan-only mutating tools (train_start, export) are gated behind --allow-mutating ("args": ["mcp", "serve", "--allow-mutating"]); when --allow-mutating alone is active, they only render the exact command that would run — they never execute training or export.

--allow-execute implies --allow-mutating and enables full background subprocess execution via two execution tools (train_execute and export_execute). Execution requires a server-issued one-time confirmation token returned during the planning phase (train_start or export). The token state is kept in-memory with a 5-minute TTL and is consumed before subprocess invocation.

Execution Security & Boundaries:

  • Flag Safety: --allow-execute is default-off and dangerous. --allow-mutating alone can NEVER trigger subprocess execution.
  • One-Time Confirmation Tokens: Authorization requires a server-issued random token. Client confirmation is UX-only; security relies entirely on the server-side token state.
  • Subprocess Isolation: Execution runs the Soup CLI as an isolated subprocess (shell=False, stdin=DEVNULL, cwd pinned to server startup directory). Child stdout/stderr is redirected to .soup/mcp-runs/<run_id>.log to avoid corrupting the MCP JSON-RPC stdio stream. Planning (train_start config snapshot) and execution are refused if .soup or .soup/mcp-runs is a symbolic link or junction, or if a hard link exists at the run log path. Planning also refuses a link or anything already present at the run directory .soup/mcp-runs/<run_id>: the snapshot is written only into a run directory the plan has just created, with config.yaml created exclusively and without following a link.
  • Concurrency & Disconnects: Enforces 1 active execution at a time, gated on a persisted run whose process is still alive — so a restarted server does not launch a second training while a child from a previous server is still running, and a stale record whose process is gone never blocks execution. A launching record (committed before the child process exists, pid not yet written) blocks restart capacity too: it is indistinguishable from "about to spawn", so treating it as live is the safe direction after a crash in that window. If a server crash leaves that row behind, soup mcp runs reconcile --expunge-launching removes rows older than five minutes and prints every removed run_id; use --older-than-seconds N to choose another positive threshold. The command refuses the whole operation if any candidate records a PID that is still alive. Launches run in background (fire-and-forget). Disconnecting the MCP client does not terminate an already-running subprocess.
  • Config Snapshotting & Input Revalidation: At plan time (train_start), the validated config is snapshotted to .soup/mcp-runs/<run_id>/config.yaml, and execution uses this snapshot rather than the original mutable config path. External protected inputs (such as datasets and model/checkpoint directories or files) are not frozen in the snapshot; instead, their content digests (computed via SHA-256 for regular files, or deterministic recursive content hashing over sorted relative file paths for directory trees, bounded by file-count and total-byte safety limits) are recorded at plan time and revalidated immediately before spawn. Modifying an external protected input between plan and execute invalidates the token. Modifying the original config path after planning has no effect because execution strictly uses the snapshotted config. Snapshotting freezes only the configuration itself, not external filesystem assets.
  • Every Local Input Is Pinned: A train plan pins every local file or directory the run reads (model, datasets, reward/teacher/PRM models, reward .py files, unlearning sets); an input outside the working directory is refused at plan time, and an input that did not exist at plan time must still not exist at execution.

Gate and Verdict Exit Codes

Soup gate and verdict commands follow a unified, CI-friendly exit-code contract:

Exit Code Constant Meaning Examples
0 EXIT_OK PASS / OK / SHIP Model passes gate; no regression; lock matches closure; valid dataset
2 EXIT_GATE_FAILED GATE FAILED / REGRESSION / DRIFT / DON'T SHIP Metric regressed; behavior/checklist MAJOR; lock closure drifted; dataset unusable
3 EXIT_USAGE_ERROR USAGE / INPUT / CONFIG ERROR Bad or invalid CLI flag; missing or unparseable input file; empty series
1 EXIT_RUNTIME_ERROR RUNTIME ERROR Unexpected crash, environment incompatibility, or live inference error

The taxonomy applies consistently across soup ship, soup eval gate, soup eval against, soup eval checklist, soup eval behavior, soup eval quant-check, soup lock check, soup expect, soup data validate, soup data lint, and soup recipes verify.

soup eval checklist requires --evidence. soup eval behavior also requires --evidence unless --base-model selects the live path; the elephant and syceval batteries cannot be scored live and exit 3 with --base-model. Omitting the required evidence exits 3 and names the JSON input to provide; it never reports a neutral pass for a gate that measured nothing.

Gate Verdict Drift vs. Operational Drift

soup lock check operates as a CI gate command where exit 2 (EXIT_GATE_FAILED) indicates that the lock closure has drifted from the configuration (requiring regeneration via soup lock write), and exit 3 (EXIT_USAGE_ERROR) indicates an unparseable or missing lock file. In contrast, operational and environment commands (soup env check, soup apply, and soup drift-alarm) belong to the operational drift family where runtime divergence or ABI mismatch signals exit 3 (with exit 2 reserved for invalid arguments or path errors in those commands). Operational inspection tools such as soup adapters audit deliberately reside outside the gate taxonomy, exiting 1 on usage errors so CI can distinguish an agreement (0) or diverged (2) verdict from invocation errors.