Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
CLI reference
Generated file — do not edit by hand. It is rendered from the live Typer command tree (
soup_cli.cli:app). After adding or removing a command, an option or an argument, or changing its type, its default or whether it is required, runpython scripts/gen_reference_docs.pyand commit the result; the test suite fails until the page matches. Help text is refreshed when a release is cut, so between releases the wording here can trail the code.
Every soup command with every option and argument. For a one-line-per-task listing
see the Command reference; for the keys of soup.yaml see the
Config reference.
How to read a table:
- Name is an option with its short form, or an argument in capitals (
PATH;PATH...takes any number of values).--flag/--no-flagis one switch with both spellings. - Type: a
flagtakes no value;text,integer,floatandpathtake one; arepeatableoption may be given more than once. - Default is the value used when the option is left out (
on/offfor a flag); an empty cell means there is none. (set at runtime) marks a default computed on the machine, which this page does not print. - Required says
yeswhen the command refuses to run without it.
--help works on every command and is not listed. Global options go before the
command name: soup --verbose train --config soup.yaml.
Global options
Usage: soup [OPTIONS] <command> [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--version |
flag | off | Show Soup CLI version. | |
--verbose, -V |
flag | off | Show full traceback on errors | |
--log-level |
text | normal |
Logging tier: quiet | normal | verbose | debug | |
--no-audit-log |
flag | off | Disable the local HIPAA/SOC2 audit log for this invocation (also via SOUP_NO_AUDIT_LOG=1). Default: a one-line record per command under ~/.soup/audit.jsonl. v0.71.3. | |
--no-telemetry |
flag | off | Opt-out of anonymous hardware-only telemetry for this run. | |
--install-completion |
flag | Install shell completion for the current shell. | ||
--show-completion |
flag | Print the shell completion script, to copy or customise it. |
Commands
| Command | Summary |
|---|---|
soup ab |
Two-sided mSPRT sequential A/B harness on latency / judge_score / retry_rate: reports whether the treatment is better or worse, valid under re-running after every new row (v0.63.0). |
soup adapters |
Adapter management: list, info, compare LoRA adapters. |
soup advise |
Pre-flight decision: PROMPT_ENG / RAG / SFT / DPO / GRPO. Run BEFORE you spend 8 hours on a GPU (v0.54.0). |
soup agent |
Agent Forge: spec -> tool-calling dataset / train / eval (v0.46.0). |
soup airgap-bundle |
Build a signed tarball with model + datasets + wheels + kernels (v0.60.0). |
soup apple-adapter |
Convert / sign HF / MLX / Apple FoundationModels adapters (v0.68.0). |
soup apply |
Execute the planned training run, refusing on drift between soup.yaml and soup.tfstate (v0.64.0). |
soup attest |
In-toto + SLSA-3 attestations per Soup Can stage (v0.59.0). |
soup audit-log |
HIPAA/SOC2-shaped JSONL audit log: tail + rotate (v0.59.0). |
soup autopilot |
Autopilot: give model+data+goal, Soup picks optimal hyperparameters. |
soup bench |
Benchmark a model: inference speed (infer, the default) or training (train). |
soup bom |
CycloneDX ML-BOM + SPDX AI bill-of-materials emitter (v0.59.0). |
soup build |
dbt-for-SFT DAG: validate, plan, and materialise dataset transforms. |
soup can |
Soup Cans: pack/inspect/verify/fork shareable .can artifacts. |
soup card |
Generate a HuggingFace model card from a registry entry. |
soup chat |
Chat with a fine-tuned model in the terminal. |
soup ci |
Fine-tuning CI: init a PR gate workflow. |
soup compile |
Compile a DSPy / GEPA prompt program against an eval suite (v0.68.0). |
soup compile-tools |
Compile / optimize tool schemas + descriptions (v0.68.0). |
soup completions |
Emit a bash / zsh / fish completion script. Use with eval "$(soup completions bash)" (v0.64.0). |
soup cost |
Estimate training cost in USD across cloud providers. |
soup data |
Dataset tools: inspect, convert, merge, dedup, validate, stats. |
soup delinearize-llama4 |
Delinearize Llama 4 fused expert weights to 3-D form. |
soup deploy |
Deploy models: Ollama integration (deploy, list, remove). |
soup diagnose |
Post-training report card: forgetting / refusal / format / mode_collapse / memorization / contamination (v0.56.0). |
soup diff |
Compare outputs of two models side-by-side on the same prompts. |
soup distill-prompt |
Distill prompt-heavy traces into a small FT plan (v0.68.0). |
soup doctor |
Check system dependencies, GPU, and compatibility. |
soup draft |
Train + measure a speculative-decoding draft model: distil your tuned model into a tiny draft, then serve it with --auto-spec (v0.71.33). |
soup drift-alarm |
Rolling-KL drift alarm on output-token distribution with optional Slack/Discord webhook (v0.63.0). |
soup edit |
Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts without re-training (v0.61.0). |
soup env |
Hermetic env lockfile + ABI drift detection (v0.64.0). |
soup eval |
Evaluate models: benchmarks, custom evals, LLM judge, leaderboard. |
soup expect |
Run an expectations suite against a JSONL dataset. |
soup export |
Export a model to GGUF, ONNX, TensorRT-LLM, BitNet, or TorchAO format. |
soup fetch |
Fetch a ready-to-edit example config from the bundled catalog. |
soup history |
Show the lineage tree for all entries with a given name. |
soup infer |
Run batch inference on a JSONL file of prompts. |
soup ingest |
Universal trace importer: Langfuse / LangSmith / Helicone / OpenPipe / OTel / OpenAI Stored Completions (v0.63.0). |
soup init |
Create a new soup.yaml config interactively or from a template. |
soup iterative-dpo |
Iterative DPO sample, score, pair, and train loop. |
soup license-advisor |
Recommend a license-clean base for a deploy target (b2c / defense / embedded) + flag downstream risk (v0.64.0). |
soup llama |
Proxy to llama.cpp binaries (cli / mtmd-cli / gguf-split / server). |
soup local-rl |
Personal-LLM flywheel daemon (init / status / record / harvest / train) (v0.68.0). |
soup lock |
Shared run lockfile (write / show / check). |
soup loop |
Data flywheel: traces -> preference pairs -> DPO -> gate -> canary deploy -> rollback, all from the CLI (v0.58.0). |
soup mcp |
Model Context Protocol server - drive Soup from any MCP client (v0.71.28). |
soup merge |
Merge a LoRA adapter with its base model into a full model. |
soup merge-sharded-fsdp-weights |
Consolidate FSDP shard files into a single safetensors file. |
soup migrate |
Import a config from LLaMA-Factory, Axolotl, or Unsloth notebook. |
soup monitor |
Live GPU monitor: Util / Temp / VRAM / Power per GPU. |
soup plan |
Render a pre-flight training plan (cost / ETA / SHA / VRAM) and write soup.tfstate for soup apply to consult (v0.64.0). |
soup plugins |
Discover, list, enable, and disable Soup plugins. |
soup probe |
Activation probes: sleeper-agent defection / honesty / misuse / pairwise interference / SAE feature diff / probe pack (v0.66.0, truth+harm v0.71.8). |
soup profile |
Estimate memory, speed, and GPU requirements BEFORE training. |
soup prune-prompt |
Detect + strip a shared system-prompt prefix across training data so the FT model internalises it (v0.63.0). |
soup push |
Push a trained model to HuggingFace Hub (or alternate hub). |
soup quantize |
Quantize a model - ergonomic alias for soup export --format <to>. |
soup quickstart |
Run a complete demo: create sample data, config, and train. |
soup ra-dit |
RA-DIT two-stage orchestrator: retriever -> generator (v0.71.10). |
soup recipes |
Ready-made configs: list, show, use, search recipes for popular models. |
soup registry |
Model Registry: push, list, show, diff, search, promote, delete. |
soup reward |
Synthesize a deterministic reward verifier from reference outputs, with a calibration report that refuses degenerate verifiers (v0.71.40). |
soup rewind |
Name the dataset rows behind a loss spike, from the training flight recorder. |
soup runs |
Experiment tracking: list, show, compare runs. |
soup serve |
Start a local inference server with OpenAI-compatible API. |
soup ship |
SHIP / DON'T SHIP verdict after fine-tuning: task win AND no catastrophic forgetting, fused into one decision (v0.71.25). |
soup shrink |
Depth-prune a model (drop the least-important contiguous layer block by residual angular distance) + optional distill-heal (v0.71.29). |
soup spectrum |
Spectrum SNR scan for targeted training (v0.71.23). |
soup steer |
Activation steering (CAA / ITI / RepE) - inference-time intervention without retraining (v0.62.0). |
soup sweep |
Run hyperparameter sweep: grid or random search over training parameters. |
soup tokenizer |
Tokenizer tools: train a BPE tokenizer from JSONL (v0.53.9). |
soup train |
Start training from a soup.yaml config. |
soup tui |
Full-screen terminal dashboard: list runs, inspect metrics, tail logs. |
soup tunability |
Probe-train 6-10 small bases on a held-out slice + report Pareto frontier of (eval delta, train cost, license) (v0.64.0). |
soup ui |
Launch the Soup Web UI. |
soup version |
Show Soup CLI version. |
soup why |
Explain training anomalies for a run in plain English. |
soup ab
Two-sided mSPRT sequential A/B harness on latency / judge_score / retry_rate: reports whether the treatment is better or worse, valid under re-running after every new row (v0.63.0).
Usage: soup ab [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | JSONL with {arm, <metric>} per row. | |
--metric |
text | yes | Metric: latency | judge_score | retry_rate. | |
--alpha |
float | 0.05 |
Error rate (0, 1) of both verdicts, kept for a test re-run after every new row, at any number of rows: with no difference the test ends in reject_h0, and with a true difference of --effect-size or more in accept_h0, each in at most this share of runs. | |
--beta |
float | Retired (#1418): ignored, with a warning. --alpha now bounds wrong accept_h0 verdicts too. | ||
--effect-size |
float | 0.1 |
Minimum detectable difference in means. | |
--slack-url |
text | Optional Slack webhook URL — POSTed on a reject_h0 / accept_h0 decision (not on continue). SSRF-validated. | ||
--discord-url |
text | Optional Discord webhook URL — POSTed on a reject_h0 / accept_h0 decision (not on continue). SSRF-validated. |
soup adapters
Adapter management: list, info, compare LoRA adapters.
Usage: soup adapters <command> [ARGS]...
Subcommands:
arithmeticauditbisectblamebranchbranchescheckcheck-safetensorscheckoutcomparediffinfolistmergeprscansignverify
soup adapters arithmetic
Task-vector arithmetic over LoRA adapters (add / scale / negate) (v0.71.34).
Usage: soup adapters arithmetic [OPTIONS] EXPRESSION
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
EXPRESSION |
text | yes | Task-vector expression, e.g. "coder + 0.5*math - toxic". | |
--adapter |
text, repeatable | yes | name=path mapping for each adapter referenced in the expression (repeatable). Names use [A-Za-z0-9_.-]. | |
--output, -o |
text | yes | Output directory for the merged adapter | |
--rank |
integer, at least 1 | Truncated-SVD rank cap. Passing it routes the merge through the exact concat path and caps the output rank at N via SVD (works for same-rank AND mixed-rank inputs). Without it, mixed-rank inputs keep the full concatenated rank (Σ rᵢ) and same-rank inputs use the fast element-wise path. | ||
--allow-unscanned |
flag | off | Skip the backdoor-scan gate (v0.71.2 #192). By default refuses an input whose adapters scan returns FAIL or cannot be scanned. |
|
--allow-cross-base |
flag | off | Allow combining adapters trained on different base models. By default a base-model mismatch is refused (task vectors from different bases are not comparable). |
soup adapters audit
Check whether a finished run did what its config asked for (#762).
Usage: soup adapters audit [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory | |
--config, -c |
text | yes | soup.yaml the run was started from | |
--json |
flag | off | Emit machine-readable JSON |
soup adapters bisect
Binary-search a training history to find the first failing checkpoint.
Usage: soup adapters bisect [OPTIONS] HISTORY...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
HISTORY... |
text, any number | yes | Ordered checkpoint identifiers (oldest first). At least 2. Example: soup adapters bisect ckpt-100 ckpt-200 ckpt-300 ckpt-400 |
|
--eval-command |
text | yes | Shell command template; '{ckpt}' is replaced with each checkpoint id. Exit code 0 means PASS, non-zero means FAIL. Example: 'soup eval custom --model {ckpt} --tasks tasks.jsonl' | |
--plan-only |
flag | off | Print the plan and exit without running the bisect | |
--output, -o |
text | Write JSON result to path |
soup adapters blame
Attribute weight movement to dataset rows via DataInf influence (v0.66.0).
Usage: soup adapters blame [OPTIONS] ADAPTER_DIR
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER_DIR |
text | yes | Path to trained adapter | |
--dataset |
text | yes | Training JSONL the adapter was built on | |
--layer |
text | yes | Layer to attribute (e.g. q_proj.7) | |
--budget |
text | 4h |
Wall-clock budget (e.g. 4h, 30m) | |
--shards |
integer, 2 to 100 | 10 |
Number of dataset shards for leave-one-out | |
--top-k |
integer, 1 to 10000 | 50 |
Number of top influencer rows to report (v0.66.0) | |
--plan-only |
flag | off | Print plan and exit without running | |
--output, -o |
text | Write JSON blame result to path (v0.66.0) |
soup adapters branch
Snapshot a training environment as a comparable branch (v0.57.0).
Usage: soup adapters branch [OPTIONS] NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Branch name (alphanumeric + ._-) | |
--config, -c |
text | Path to soup.yaml (required unless --from-registry) | ||
--base |
text | Base model id (required unless --from-registry) | ||
--dataset |
text | Training dataset path (optional) | ||
--attach-to-registry |
text | Registry entry id/ref to attach this branch pointer to as a branch_ref artifact + link via registry_entry_id (v0.71.4). | ||
--from-registry |
text | Derive config + base_model + dataset hash from a Registry entry instead of --config/--base (v0.71.4). |
soup adapters branches
List all snapshotted branches (v0.57.0).
Usage: soup adapters branches
Takes no options or arguments.
soup adapters check
Audit whether an adapter is trained: Frobenius delta, zero B, inner keys (#1721).
Usage: soup adapters check [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory | |
--json |
flag | off | Emit machine-readable JSON document |
soup adapters check-safetensors
Refuse pickle / PyTorch-classic weights at the boundary (v0.60.0).
Usage: soup adapters check-safetensors [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter / model directory | |
--strict |
flag | off | Exit 3 on any unsafe (pickle / PyTorch-classic) weight file |
soup adapters checkout
Restore a snapshotted branch's config into cwd (v0.57.0).
Usage: soup adapters checkout [OPTIONS] NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Branch name to check out | |
--output, -o |
text | soup.yaml |
Where to write the restored config |
soup adapters compare
Compare two LoRA adapters side-by-side.
Usage: soup adapters compare ADAPTER1 ADAPTER2
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER1 |
text | yes | Path to first adapter | |
ADAPTER2 |
text | yes | Path to second adapter |
soup adapters diff
Per-layer ΔW Frobenius diff + effective-rank drift (v0.57.0).
Usage: soup adapters diff [OPTIONS] ADAPTER_A ADAPTER_B
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER_A |
text | yes | Path to first adapter | |
ADAPTER_B |
text | yes | Path to second adapter | |
--top-k |
integer, 1 to 200 | 10 |
Number of top changed projections to report | |
--format |
text | table |
Output format: table | json | markdown | |
--output, -o |
text | Write report to file (json/markdown only) |
soup adapters info
Show detailed metadata for a LoRA adapter.
Usage: soup adapters info ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory |
soup adapters list
Scan a directory for LoRA adapters and list them.
Usage: soup adapters list [DIRECTORY]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DIRECTORY |
text | . |
Directory to scan for adapters (recursive) |
soup adapters merge
Merge LoRA adapters via linear / ties / dare / svd (v0.57.0, v0.60.0 license gate).
Usage: soup adapters merge [OPTIONS] ADAPTERS...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTERS... |
text, any number | yes | Two or more adapter paths to merge | |
--output, -o |
text | yes | Output directory for merged adapter | |
--strategy |
text | linear |
linear | ties | dare | svd | cmaes | |
--weights |
text | Comma-separated weights (default: equal) | ||
--density |
float | 0.2 |
Trim density for ties/dare in (0, 1] | |
--seed |
integer | 0 |
Random seed for dare / cmaes | |
--rank |
integer | SVD rank (svd strategy only) | ||
--eval |
text | Path to eval suite (required for --strategy cmaes). Used by the evolutionary loop to score candidate merges. | ||
--budget |
text | 1h |
Wall-clock budget for cmaes (60s..24h, e.g. 1h, 30m) | |
--population |
integer, 2 to 256 | 8 |
cmaes population size per generation | |
--max-generations |
integer, 1 to 10000 | 20 |
cmaes generation cap | |
--license |
text, repeatable | SPDX license id per adapter (repeatable; same order as inputs). v0.60.0 refuses merges with conflicting licenses unless --license-override <reason> is passed. | ||
--license-override |
text | Free-text justification (>=8 chars) to merge across a license conflict. Logged to the audit log for legal review (v0.71.2). | ||
--allow-unscanned |
flag | off | Skip the v0.71.2 backdoor-scan gate. By default adapters merge refuses to merge an input whose adapters scan returns FAIL (or that cannot be scanned). |
|
--canary |
text | Canary-suite JSON to compute a live OK/MINOR/MAJOR verdict for the merged adapter vs the first input (v0.71.4 #172). Shape: {"baseline_scores": [...], "candidate_scores": [...]}. | ||
--strict-verdict |
flag | off | Exit 2 when the canary verdict is MAJOR (CI gate). |
soup adapters pr
Render a GitHub-style PR for an adapter.
Usage: soup adapters pr [OPTIONS] TITLE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
TITLE |
text | yes | Short PR title (e.g. 'add-support-tone') | |
--base-sha |
text | yes | 64-hex SHA of the base model | |
--adapter |
text | yes | Path to candidate adapter | |
--eval |
text | Path to JSON eval-deltas: list of {"metric": ..., "baseline": ..., "candidate": ...} | ||
--samples |
text | Path to JSON sample diffs: list of {"prompt": ..., "baseline_output": ..., "candidate_output": ...} | ||
--dataset-diff |
text | Path to a text file containing the dataset diff | ||
--output, -o |
text | Write rendered Markdown to path (default: stdout) | ||
--format, -f |
text | markdown |
markdown | json | |
--push |
text | Post the rendered Markdown as a comment on a GitHub PR: owner/repo#N (e.g. MakazhanAlpamys/Soup#42). Auth via GITHUB_TOKEN / GH_TOKEN (v0.71.4). |
soup adapters scan
Spectral backdoor scan over LoRA adapter weights (v0.60.0).
Usage: soup adapters scan [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory | |
--format |
text | text |
Output format: text | json |
soup adapters sign
Compute manifest + write .soup-signature.json (v0.60.0, ed25519 v0.71.2).
Usage: soup adapters sign [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory | |
--backend |
text | unsigned |
Signing backend: unsigned | ed25519 | sigstore (keyless OIDC) | |
--key |
text | ed25519 private-key PEM path (or set SOUP_SIGNING_KEY). | ||
--generate-key |
text | Generate a fresh ed25519 keypair, persist the private key here (PEM, 0600), and sign with it. | ||
--interactive-oidc |
flag | off | Allow Sigstore browser OIDC when no ambient credential exists. Off by default so headless runners fail instead of hanging. |
soup adapters verify
Verify .soup-signature.json against current files (v0.60.0, ed25519 v0.71.2).
Usage: soup adapters verify [OPTIONS] ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ADAPTER |
text | yes | Path to adapter directory | |
--strict |
flag | off | Exit 3 on any verification failure (CI-friendly) | |
--public-key |
text | Trusted ed25519 public-key PEM. When set, the embedded signing key must match it and requires an ed25519 signature (genuine authentication, not just consistency). | ||
--cert-identity |
text | Trusted Sigstore certificate identity (SAN). Requires --cert-oidc-issuer and is required for sigstore-signed adapters. | ||
--cert-oidc-issuer |
text | Trusted OIDC issuer. Required together with --cert-identity. |
soup advise
Pre-flight decision: PROMPT_ENG / RAG / SFT / DPO / GRPO. Run BEFORE you spend 8 hours on a GPU (v0.54.0).
Usage: soup advise <command> [ARGS]...
Subcommands:
soup advise compare
Show prior verdicts from advise_history.jsonl.
Usage: soup advise compare [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--limit |
integer, 1 to 1000 | 20 |
How many recent verdicts to show. |
soup advise explain
Print the rubric, weights, and evidence trail of the last verdict.
Usage: soup advise explain
Takes no options or arguments.
soup advise run
Render a Verdict for the supplied dataset.
Usage: soup advise run [OPTIONS] DATA
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | Path to JSONL dataset to advise on (must stay under cwd). | |
--goal, -g |
text | One-line goal statement: e.g. "make our chatbot more concise". Sharpens task classification. | ||
--probe |
flag | off | Also run an ROI probe (zero/few-shot + RAG baseline + N-step LoRA). Heuristic by default; pass --probe-model <id> for a LIVE probe that loads the model + LoRA-trains on a tiny held-out slice. | |
--probe-model |
text | Base model id for a LIVE probe (e.g. HuggingFaceTB/SmolLM2-135M). When set, the probe loads this model, scores zero/few-shot, LoRA-trains, and measures base-model proximity. Implies --probe. | ||
--probe-device |
text | Device for the live probe (cuda / cpu). Auto-detected when omitted. | ||
--record |
flag | off | Append this verdict to advise_history.jsonl with accepted=True. Omit --record to skip. Stored under ~/.soup/. | |
--notes |
text | "" |
Optional notes attached to the recorded verdict (<=4096 chars). |
soup agent
Agent Forge: spec -> tool-calling dataset / train / eval (v0.46.0).
Usage: soup agent <command> [ARGS]...
Subcommands:
soup agent eval
Score predicted tool-calls against the spec's tool catalog.
Usage: soup agent eval [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--spec, -s |
text | yes | API spec (under cwd). | |
--predictions, -p |
text | yes | JSONL of model outputs, one per line, with at minimum {tool: <name>, arguments: {...}}. | |
--sandbox |
flag | off | Execute each heuristic-passing prediction against a sandboxed mock of the endpoint (v0.71.18 #110): classify ok / tool_error / timeout / arg_error. Strong isolation is POSIX-only (reduced isolation on Windows; a friendly advisory is printed). |
soup agent synth
Parse an API spec and synthesise a tool-calling SFT dataset.
Usage: soup agent synth [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--spec, -s |
text | yes | Path to OpenAPI / MCP / GraphQL spec (YAML or JSON, under cwd). | |
--output, -o |
text | agent_dataset.jsonl |
Where to write the synthesised JSONL dataset (under cwd). | |
--kind, -k |
text | Spec kind override: openapi | mcp | graphql. Auto-detected if omitted. | ||
--examples-per-endpoint, -n |
integer, 1 to 32 | 1 |
Number of synthetic rows to emit per endpoint. |
soup agent train
One-shot wrapper: synth + planned soup train invocation (printed).
Usage: soup agent train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--spec, -s |
text | yes | API spec (under cwd). | |
--base, -b |
text | yes | Base model HF repo id to fine-tune. | |
--dataset-out |
text | agent_dataset.jsonl |
Where the synth step writes its dataset (under cwd). | |
--output-dir |
text | ./agent_train_output |
Where the planned soup train run will store checkpoints. | |
--examples-per-endpoint, -n |
integer, 1 to 32 | 4 |
Number of synthetic examples to create per API endpoint. |
soup airgap-bundle
Build a signed tarball with model + datasets + wheels + kernels (v0.60.0).
Usage: soup airgap-bundle [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--output, -o |
text | yes | Output tarball path (cwd-contained) | |
--model |
text | yes | Path to model directory | |
--dataset |
text, repeatable | Dataset directory (repeatable) | ||
--wheel |
text, repeatable | Wheel directory (repeatable) | ||
--kernel |
text, repeatable | CUDA / kernel directory (repeatable) | ||
--bundle-size-cap |
float | 100.0 |
Cap in GiB (default 100). Build aborts when exceeded. | |
--repro-receipt |
text | Embed a reproducibility receipt JSON (from soup train --repro-receipt) into the bundle. Auto-detected from <model>/repro-receipt.json when not given. v0.71.3 #188. |
soup apple-adapter
Convert / sign HF / MLX / Apple FoundationModels adapters (v0.68.0).
Usage: soup apple-adapter [OPTIONS] SOURCE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SOURCE |
text | yes | Source adapter directory | |
--direction |
text | hf-to-mlx |
Allowed: hf-to-apple, hf-to-mlx, mlx-to-apple, mlx-to-hf | |
--output, -o |
text | yes | Output directory | |
--sign / --no-sign |
flag | off | Sign converted adapter via v0.60 Merkle-root signing. | |
--plan-only |
flag | off | Render plan + exit 0 (no live conversion). |
soup apply
Execute the planned training run, refusing on drift between soup.yaml and soup.tfstate (v0.64.0).
Usage: soup apply [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml. | |
--state |
text | soup.tfstate |
Path to the state file written by soup plan. |
|
--dry-run |
flag | off | Verify drift and exit without invoking training. |
soup attest
In-toto + SLSA-3 attestations per Soup Can stage (v0.59.0).
Usage: soup attest <command> [ARGS]...
Subcommands:
soup attest emit
Emit a per-stage in-toto/SLSA-3 attestation.
Usage: soup attest emit [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--stage |
text | yes | Stage: extract / train / eval / export / publish. | |
--subject |
text | yes | Artefact name. | |
--sha |
text | yes | 64-hex SHA-256 of the artefact. | |
--builder |
text | soup-cli |
Builder identity (default: soup-cli). | |
--invocation |
text | Free-form invocation marker (e.g. command line). | ||
--sign |
text | unsigned |
Signature backend: unsigned (default) | ed25519 | sigstore (keyless OIDC + Fulcio/Rekor). | |
--key |
text | ed25519 private-key PEM path (or set SOUP_SIGNING_KEY). | ||
--output, -o |
text | Output file path (cwd-contained). | ||
--attach-to-registry |
text | Attach the emitted attestation statement to a registry entry id (needs --output). | ||
--interactive-oidc |
flag | off | Allow Sigstore browser OIDC when no ambient credential exists. Off by default so headless runners fail instead of hanging. |
soup attest verify
Verify an ed25519 or Sigstore-signed attestation.
Usage: soup attest verify [OPTIONS] STATEMENT
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
STATEMENT |
text | yes | Path to the in-toto Statement JSON. | |
--signature, -s |
text | yes | Path to the .sig JSON sidecar written by attest emit --sign. |
|
--public-key |
text | Trusted ed25519 public-key PEM. When set, the embedded key must match it (genuine authentication). | ||
--cert-identity |
text | Trusted Sigstore certificate identity (SAN). Requires --cert-oidc-issuer and is required for sigstore-signed attestations. | ||
--cert-oidc-issuer |
text | Trusted OIDC issuer. Required together with --cert-identity. |
soup audit-log
HIPAA/SOC2-shaped JSONL audit log: tail + rotate (v0.59.0).
Usage: soup audit-log <command> [ARGS]...
Subcommands:
soup audit-log rotate
Force a rotation pass at the current cap.
Usage: soup audit-log rotate [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--path |
text | Override audit log path. | ||
--cap-mb |
integer, 1 to 10000 | 100 |
Rotation cap in MiB. |
soup audit-log tail
Show the most recent audit records.
Usage: soup audit-log tail [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--limit |
integer | 50 |
Max records to show (1-100000). | |
--path |
text | Override audit log path (default: $SOUP_AUDIT_LOG_PATH or ~/.soup/audit.jsonl). | ||
--json |
flag | off | Emit raw JSONL instead of a table. |
soup autopilot
Autopilot: give model+data+goal, Soup picks optimal hyperparameters.
Usage: soup autopilot [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | Base model (HF model id) | ||
--data, -d |
text | Dataset path (JSONL) | ||
--goal, -g |
text | Goal: chat | reasoning | code | classification | tool-calling | alignment | domain-adapt | ||
--gpu-budget |
text | "" |
VRAM budget, e.g. 24GB (default: auto-detect) | |
--output, -o |
text | soup.yaml |
Output config path | |
--dry-run |
flag | off | Show decisions but don't write the config | |
--yes, -y |
flag | off | Skip confirmation prompts |
soup bench
Benchmark a model: inference speed (infer, the default) or training (train).
Usage: soup bench <command> [ARGS]...
Subcommands:
soup bench infer
Run an inference benchmark (speed and memory) on a loaded model.
Usage: soup bench infer [OPTIONS] MODEL
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
MODEL |
text | yes | Path to model (LoRA adapter or full model) to benchmark | |
--base, -b |
text | Base model for LoRA adapter (auto-detected if not set) | ||
--max-tokens |
integer | 128 |
Maximum tokens to generate per prompt | |
--num-prompts, -n |
integer | 3 |
Number of prompts to run for averaging (ignored when --prompts-file is set) | |
--prompts-file |
text | Path to custom prompts file (.txt or .jsonl) | ||
--p50 |
flag | off | Print p50 (median) per-prompt latency. v0.53.9 #26. | |
--p95 |
flag | off | Print p95 per-prompt latency. v0.53.9 #26. | |
--backend |
text | auto |
Inference backend hint: auto (default) | transformers | mlx. v0.53.9 #28. | |
--cuda-graphs |
flag | off | Experimental CUDA graph decode for resident, unquantized Qwen2/Llama models. |
soup bench train
Train for a fixed number of steps and report timing, memory and tokens -- failing when the model was not actually training.
Usage: soup bench train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Config to train | |
--steps |
integer, at least 1 | 20 |
Optimizer steps to run | |
--warmup |
integer, at least 0 | 3 |
Leading steps measured but left out of the timing | |
--output, -o |
text | bench-train.json |
Where to write the JSON report |
soup bom
CycloneDX ML-BOM + SPDX AI bill-of-materials emitter (v0.59.0).
Usage: soup bom <command> [ARGS]...
Subcommands:
soup bom emit
Emit a CycloneDX + SPDX BOM from CLI-supplied SHAs.
Usage: soup bom emit [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--name |
text | yes | Model / adapter name. | |
--version |
text | 0.1.0 |
Model version string. | |
--base-model |
text | yes | HF repo id of the base model. | |
--base-sha |
text | yes | SHA-256 of the base model. | |
--config-sha |
text | yes | SHA-256 of the resolved soup.yaml config. | |
--data-sha |
text | SHA-256 of the training dataset. | ||
--task |
text | sft |
Training task (sft / dpo / grpo / ...). | |
--license |
text | License: an SPDX id (e.g. Apache-2.0, MIT; case-insensitive), an SPDX expression (Apache-2.0 OR MIT), or any other name. | ||
--format, -f |
text | cyclonedx |
Output BOM format: cyclonedx | spdx | both. | |
--output, -o |
text | Output file path (cwd-contained). When --format=both, this is the prefix and Soup writes <prefix>.cdx.json + <prefix>.spdx.json. | ||
--energy, -e |
text | Path to energy measurement JSON. | ||
--attach-to-registry |
text | Attach the emitted BOM file(s) to a registry entry id (needs --output). |
soup build
dbt-for-SFT DAG: validate, plan, and materialise dataset transforms.
Usage: soup build [OPTIONS] CONFIG
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
CONFIG |
text | yes | Path to build manifest (YAML) | |
--output-dir, -o |
text | build_out |
Directory (under cwd) for materialised model JSONL + state store. | |
--dry-run |
flag | off | Validate the manifest + print the topological plan; do not execute. |
soup can
Soup Cans: pack/inspect/verify/fork shareable .can artifacts.
Usage: soup can <command> [ARGS]...
Subcommands:
soup can fork
Fork a can with config modifications and re-pack.
Usage: soup can fork [OPTIONS] SOURCE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SOURCE |
text | yes | Source .can path | |
--out, -o |
text | yes | Output forked .can path | |
--modify |
text, repeatable | Modification like 'training.lr=5e-5' (repeatable) | ||
--author |
text | unknown |
Author name recorded in the forked manifest. |
soup can inspect
Preview a .can file's manifest without extracting.
Usage: soup can inspect PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to .can file |
soup can pack
Pack a registry entry into a shareable .can file.
Usage: soup can pack [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--entry-id |
text | yes | Registry entry id / prefix / name:tag | |
--out, -o |
text | yes | Output .can path | |
--author |
text | unknown |
Author handle | |
--description |
text | "" |
Free-form description | |
--attest |
text, repeatable | Embed an in-toto Statement JSON into the can manifest (v3, repeatable). Each file is shape-validated. v0.71.3. |
soup can publish
Publish a .can file to HF Hub as a dataset.
Usage: soup can publish [OPTIONS] CAN_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
CAN_PATH |
text | yes | Path to .can file | |
--hf-hub |
text | yes | HF Hub dataset repo (e.g. 'me/my-recipe-can') | |
--private |
flag | off | Create a private HF Hub dataset repository. | |
--message, -m |
text | Commit message (first line only, capped at 200 chars) |
soup can run
Run a can end-to-end: extract -> train (-> optional deploy).
Usage: soup can run [OPTIONS] CAN_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
CAN_PATH |
text | yes | Path to .can file | |
--yes, -y |
flag | off | Skip the security confirmation panel (auto-trains, auto-fetches data) | |
--deploy |
flag | off | Run the can's deploy_targets after a successful train | |
--extract-dir |
text | Where to extract (default: fresh tmp dir) | ||
--env-capture |
text | Write env summary (pip freeze + GPU info) to this path |
soup can verify
Verify a .can file's schema and config parseability.
Usage: soup can verify PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to .can file |
soup card
Generate a HuggingFace model card from a registry entry.
Usage: soup card [OPTIONS] REF
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REF |
text | yes | Registry entry id / prefix / name:tag | |
-o, --output |
text | MODELCARD.md |
Output markdown path (must stay under the current directory) |
soup chat
Chat with a fine-tuned model in the terminal.
Usage: soup chat [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to LoRA adapter directory or full model | |
--base, -b |
text | Base model ID. Auto-detected from adapter_config.json if not set. | ||
--device |
text | Device: cuda, mps, cpu. Auto-detected if not set. | ||
--max-tokens |
integer | 512 |
Maximum number of tokens to generate | |
--temperature, -t |
float | 0.7 |
Sampling temperature (0.0 = greedy, 1.0 = creative) | |
--system, -s |
text | System prompt for the conversation | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--hub |
text | hf |
Source hub for the base model: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK; the base model is snapshotted to a cwd-contained cache before chat starts (v0.53.10 #152). |
soup ci
Fine-tuning CI: init a PR gate workflow.
Usage: soup ci <command> [ARGS]...
Subcommands:
soup ci init
Write a GitHub Actions fine-tuning gate workflow.
Usage: soup ci init [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--data |
text | ./data/train.jsonl |
Training data path (repo-relative) | |
--suite |
text | expectations.yaml |
Expectations suite YAML (repo-relative) | |
--evidence |
text | ship_evidence.json |
Ship evidence JSON (repo-relative) | |
--python |
text | 3.11 |
Python version for the CI runner (e.g. 3.11) | |
--branch |
text | main |
Branch the workflow triggers on | |
--config |
text | Bind the ship gate to a committed soup.yaml (refuses stale evidence) | ||
-o, --output |
text | .github/workflows/soup-gate.yml |
Workflow output path | |
--force, -f |
flag | off | Overwrite an existing workflow |
soup compile
Compile a DSPy / GEPA prompt program against an eval suite (v0.68.0).
Usage: soup compile [OPTIONS] PROGRAM
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PROGRAM |
text | yes | Path to DSPy / GEPA program (.py) | |
--eval |
text | yes | Path to eval-suite JSON / JSONL file | |
--optimizer |
text | mipro |
Optimizer name. Allowed: bootstrap_fewshot, copro, gepa, mipro, textgrad | |
--max-iters |
integer | 10 |
Maximum optimizer iterations | |
--output, -o |
text | compiled_program.py |
Output program path | |
--plan-only |
flag | off | Render the resolved plan + exit 0 (no live compile). |
soup compile-tools
Compile / optimize tool schemas + descriptions (v0.68.0).
Usage: soup compile-tools [OPTIONS] SPEC
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SPEC |
text | yes | OpenAPI / MCP / GraphQL spec path | |
--eval |
text | yes | Path to tool-call eval JSONL | |
--optimizer |
text | textgrad |
Allowed: gepa, textgrad | |
--output, -o |
text | compiled_tools.json |
Output schema path | |
--task-lm |
text | Task language model for GEPA optimizer | ||
--reflection-lm |
text | Reflection language model for GEPA optimizer | ||
--engine |
text | Backward engine for TextGrad optimizer (e.g. gpt-4o) | ||
--plan-only |
flag | off | Render plan + exit 0 (no live optimise). |
soup completions
Emit a bash / zsh / fish completion script. Use with eval "$(soup completions bash)" (v0.64.0).
Usage: soup completions SHELL
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SHELL |
text | yes | One of: bash, zsh, fish. |
soup cost
Estimate training cost in USD across cloud providers.
Usage: soup cost [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml config file | |
--gpu, -g |
text | Filter by specific GPU (e.g., A100, H100, RTX 4090) | ||
--json |
flag | off | Output as JSON for scripting |
soup data
Dataset tools: inspect, convert, merge, dedup, validate, stats.
Usage: soup data <command> [ARGS]...
Subcommands:
active-sampleaugmentbest-of-nbrain-rotcanarycleanconvertdecontaminatededupdemodoctordownloadeducationalevolvefilterforgefrom-tracesgen-magpiegenerateingestinspectlangdetectlintmergemixpersona-mixpiipreprocesspreviewpushreciperegisterregistryreviewsamplescoresearchsplitstatstopicstoxicityunregistervalidate
soup data active-sample
Surface the most uncertain prod traces for human review.
Usage: soup data active-sample [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | JSONL of trace rows with rm_score or rm_scores fields. | |
--output, -o |
text | active_samples.jsonl |
Output JSONL of the top-uncertainty rows (default: active_samples.jsonl). | |
--budget |
integer | 100 |
Max rows to surface for human review (1 - 100_000). | |
--slack-url |
text | Optional Slack webhook URL — POSTed on completion. SSRF-validated. | ||
--discord-url |
text | Optional Discord webhook URL — POSTed on completion. SSRF-validated. |
soup data augment
Augment a dataset via LLM (rephrase / translate / style).
Usage: soup data augment [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Source JSONL file | |
--output, -o |
text | augmented.jsonl |
Output JSONL path | |
--strategy, -s |
text | rephrase |
Augmentation strategy: rephrase, translate, style | |
--provider, -p |
text | ollama |
LLM provider: ollama, anthropic, vllm | |
--model |
text | "" |
Provider model id (default per provider) | |
--base-url |
text | "" |
Provider base URL (ollama/vllm; defaults to loopback) | |
--count, -c |
integer, 1 to 10 | 2 |
Augmentation multiplier (1-10) | |
--lang |
text | "" |
Comma-separated target languages for translate | |
--styles |
text | "" |
Comma-separated styles for style strategy | |
--requests-per-minute |
integer, 1 to 600 | 60 |
Rate limit for provider requests | |
--dedup |
flag | off | Deduplicate augmented + original data |
soup data best-of-n
Best-of-N rejection sampling: sample N per prompt, a judge picks the winner.
Usage: soup data best-of-n [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base |
text | "" |
Local base model id/path to sample from | |
--prompts |
text | "" |
JSONL of {prompt|messages} rows | |
--judge |
text | "" |
Judge URL (ollama://|https://|http://localhost) | |
--provider |
text | "" |
Provider sampler: ollama | vllm | |
--model |
text | "" |
Provider sampler model id | |
--base-url |
text | "" |
Provider sampler base URL | |
--n |
integer | 8 |
Candidates per prompt [2, 64] | |
--output, -o |
text | "" |
Output SFT JSONL (required unless --plan-only) | |
--emit-pairs |
text | "" |
Also write winner/loser DPO pairs to this JSONL | |
--checkpoint |
text | "" |
Recovery journal (default: <output-or-artifact>.checkpoint.jsonl) | |
--manifest |
text | "" |
Commit manifest (default: <output>.manifest.json) | |
--resume |
flag | off | Resume a matching checkpoint | |
--export-candidates |
text | "" |
Sampling-only candidate artifact JSONL | |
--candidate-artifact |
text | "" |
Offline candidate artifact to materialize | |
--judgments |
text | "" |
Offline verified judgments JSONL | |
--temperature |
float | 1.0 |
Sampling temp [0, 2] | |
--max-new-tokens |
integer | 256 |
Max new tokens per candidate [1, 4096] | |
--device |
text | "" |
Local sampler device: cuda | cpu | |
--seed |
integer | 0 |
Local sampling seed | |
--revision |
text | "" |
Pinned local model revision | |
--trust-remote-code |
flag | off | Allow custom model code on local --base | |
--plan-only |
flag | off | Validate + print plan |
soup data brain-rot
Score a dataset for brain-rot per arXiv 2510.13928.
Usage: soup data brain-rot [OPTIONS] DATA
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | Path to JSONL dataset | |
--strict |
flag | off | Exit 3 on MAJOR verdict (CI gate mode) | |
--max-major-fraction |
float | 0.25 |
Strict-mode MAJOR-row fraction cap [0, 1] | |
--lang |
text | en |
Per-language heuristic bundle: en | es | fr | de | ru | auto. Closes issue #234. |
soup data canary
Dataset canaries (memorization probe).
Usage: soup data canary <command> [ARGS]...
Subcommands:
soup data canary check
Check a model for memorization of the manifest's canaries.
Usage: soup data canary check [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--manifest |
text | yes | Canary manifest | |
--base |
text | yes | Base model id or path | |
--adapter |
text | LoRA adapter to check | ||
--controls |
integer | 128 |
Never-inserted controls to rank against | |
--device |
text | auto |
auto/cpu/cuda | |
--output, -o |
text | Write the report as JSON |
soup data canary insert
Insert K unique canaries into a dataset and record them.
Usage: soup data canary insert [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--output, -o |
text | yes | Output file: .jsonl, or .json for a JSON array | |
--manifest |
text | yes | Where to write the canary manifest (CONTAINS THE SECRETS) | |
--count, -k |
integer | 16 |
Number of canaries | |
--seed |
integer | 0 |
Seed for the canaries and the rows they go to | |
--format |
text | auto |
Dataset format to write the canaries in: auto (detect), alpaca, sharegpt, chatml |
soup data clean
Clean and repair a fine-tuning dataset: control characters, whitespace, format sanity.
Usage: soup data clean [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to input dataset file (JSONL) | |
--output, -o |
text | Path to write cleaned dataset (default: <input>_cleaned.jsonl) | ||
--force, -f |
flag | off | Overwrite output file if it already exists | |
--dry-run |
flag | off | Preview modifications without writing output file | |
--min-tokens |
integer | 1 |
Minimum character/token length for assistant turns (shorter turns are dropped) | |
--prune-echo / --no-prune-echo |
flag | off | Prune assistant turns that merely echo the user prompt (opt-in) | |
--strip-boilerplate / --no-strip-boilerplate |
flag | off | Strip canned AI disclaimers ('As an AI...', 'Certainly!...') | |
--repair-code / --no-repair-code |
flag | off | Auto-close unclosed markdown ``` code fences in assistant completions | |
--repair-json / --no-repair-json |
flag | off | Repair trailing commas and markdown fences in tool-call arguments | |
--drop-invalid-json |
flag | off | Drop tool-calling rows with unrepairable JSON syntax errors | |
--json |
flag | off | Output machine-readable JSON summary |
soup data convert
Convert a dataset between formats (alpaca, sharegpt, chatml).
Usage: soup data convert [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Input dataset file | |
--to, -t |
text | yes | Target format: alpaca, sharegpt, chatml | |
--output, -o |
text | Output file path (default: <input>_<format>.jsonl) |
soup data decontaminate
Drop rows that overlap public benchmarks (n-gram heuristic).
Usage: soup data decontaminate [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL file to decontaminate. | |
--benchmarks, -b |
text | Comma-separated benchmark names (label only; corpora not bundled). | ||
--benchmark-file |
text | v0.53.7: operator-supplied benchmark JSONL (under cwd). Each row's text, prompt, Alpaca, preference or conversation fields contribute to the n-gram contamination set. Compatible with --benchmarks (extends). | ||
--output, -o |
text | clean.jsonl |
Output JSONL of rows that survived decontamination. | |
--threshold |
float, 0.0 to 1.0 | 0.8 |
Minimum n-gram overlap ratio for removing a row. | |
--n |
integer, 1 to 32 | 8 |
N-gram size used for overlap detection. |
soup data dedup
Remove near-duplicate rows: MinHash (default) or embeddings (--semantic).
Usage: soup data dedup [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--output, -o |
text | Output file path (default: <input>_deduped.jsonl) | ||
--threshold |
float | 0.8 |
Similarity threshold (0.0-1.0): MinHash Jaccard by default, embedding cosine under --semantic. | |
--field, -f |
text | Field to hash/embed (default: all text fields concatenated) | ||
--semantic |
flag | off | Use embedding cosine (SemDeDup) instead of MinHash. Catches paraphrases MinHash misses. Requires soup-cli[train]. | |
--embed-model |
text | sentence-transformers/all-MiniLM-L6-v2 |
Embedding model used by --semantic. | |
--device |
text | auto |
Device for --semantic (auto/cpu/cuda). |
soup data demo
List or copy a bundled demo dataset (v0.43.0).
Usage: soup data demo [OPTIONS] [NAME]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | Bundle name (alpaca_demo / sharegpt_demo / dpo_demo / grpo_demo). Omit to list available bundles. | ||
--output, -o |
text | Destination JSONL path (defaults to ./<name>.jsonl). |
soup data doctor
Chat-template compatibility report + loss-mask X-ray.
Usage: soup data doctor [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file (jsonl/json/csv/parquet). | |
--model, -m |
text | yes | Tokenizer id or local path to check chat-template compatibility against. | |
--format, -f |
text | auto |
Dataset format (auto-detects by default). | |
--max-length |
integer | 2048 |
Training max_length — used for the truncation-risk check. | |
--sample |
integer | 200 |
Number of rows to sample for the checks. | |
--show-mask |
integer | Render N sample rows with per-token trained/masked colouring (the real collator path). | ||
--train-on-responses-only / --no-train-on-responses-only |
flag | on | Mask non-assistant tokens (mirrors data.train_on_responses_only). | |
--train-on-messages-with-train-field |
flag | off | Per-message train:bool field masking (mirrors the same soup.yaml flag). | |
--mask-history / --no-mask-history |
flag | off | Train only the LAST assistant turn (mirrors data.mask_history). | |
--train-on-eot |
flag | off | Extend the trained span through the trailing EOS/EOT token. | |
--trust-remote-code |
flag | off | Allow loading a tokenizer that ships custom code. | |
--output, -o |
text | Write the report JSON to this path. |
soup data download
Download a HuggingFace dataset and save as JSONL.
Usage: soup data download [OPTIONS] DATASET_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATASET_ID |
text | yes | HuggingFace dataset ID (e.g. teknium/OpenHermes-2.5) | |
--output, -o |
text | Output file path (default: <dataset-name>.jsonl) | ||
--split |
text | train |
Dataset split to download (e.g. train, test, train[:1000]) | |
--samples, -n |
integer | Max number of samples to download (streams, no full download) | ||
--format, -f |
text | Convert to Soup format after download: alpaca, sharegpt, chatml | ||
--trust-remote-code |
flag | off | Allow datasets that ship custom Python loaders. Default deny (v0.36.0). Only enable if you trust the source. | |
--hub |
text | hf |
Source hub: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK (v0.53.8 #130). |
soup data educational
Tag each row with an educational-value score [0, 1].
Usage: soup data educational [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL file to score. | |
--output, -o |
text | educational.jsonl |
Output JSONL file with educational-value scores. |
soup data evolve
Evol-Instruct: deepen (depth) or diversify (breadth) seed instructions.
Usage: soup data evolve [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input |
text | yes | Seed JSONL ({prompt|instruction|messages} per row) | |
--provider |
text | yes | ollama | vllm | |
--model |
text | yes | Provider model id | |
--strategy |
text | depth |
depth | breadth | |
--rounds |
integer | 1 |
Evolution rounds [1, 5] | |
--output, -o |
text | "" |
Output JSONL (required unless --plan-only) | |
--base-url |
text | "" |
Provider base URL | |
--temperature |
float | 1.0 |
Sampling temp [0, 2] | |
--max-tokens |
integer | 512 |
Max tokens [1, 16384] | |
--plan-only |
flag | off | Validate + print plan |
soup data filter
Filter dataset by quality: perplexity and/or coherence scoring.
Usage: soup data filter [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--output, -o |
text | Output file path (default: <input>_filtered.jsonl) | ||
--perplexity, --ppl |
float | Max perplexity threshold (rows above this are removed) | ||
--coherence, --min-coherence |
float | Min coherence threshold 0.0-1.0 (rows below this are removed) | ||
--ppl-model |
text | gpt2 |
Model for perplexity scoring (default: gpt2) | |
--field, -f |
text | Field to score (default: all text fields concatenated) | ||
--score-only |
flag | off | Add scores to data without filtering (writes _scored.jsonl) |
soup data forge
Run the multi-stage synthetic data pipeline with provenance.
Usage: soup data forge [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--docs, -d |
text | yes | Directory of documents (txt/md/json/jsonl, under cwd). | |
--task, -t |
text | sft |
Forge task: sft | preference | tool. | |
--target-rows, -r |
integer, 1 to 1000000 | 100 |
Maximum number of synthesised rows. | |
--teacher |
text | local-judge |
Label recorded in provenance for the judge backend. | |
--output, -o |
text | forge_dataset.jsonl |
JSONL output path (under cwd). | |
--provenance |
text | forge_provenance.json |
Provenance manifest output path (under cwd). | |
--uncertainty-threshold |
float, 0.0 to 1.0 | 0.0 |
Minimum Jaccard-distance score required to keep a synthesised row. | |
--max-chunk-chars |
integer, 1 to 64000 | 1000 |
Maximum chars per document chunk before judge call. | |
--judge-provider |
text | v0.53.7 #111: live judge backend. One of ollama / anthropic / vllm. Omit for the deterministic offline stub. | ||
--judge-model |
text | llama3.1 |
Model name for the live judge provider (default llama3.1). | |
--judge-base-url |
text | Override base URL for Ollama (localhost-only) / vLLM (scheme allowlist + loopback). Ignored for Anthropic. | ||
--hub |
text | hf |
Teacher hub: hf (default) / modelscope / modelers. When non-HF and --teacher is a repo id (owner/name), the teacher is pre-fetched from that hub (v0.71.5 #157). |
soup data from-traces
Harvest preference pairs from production traces.
Usage: soup data from-traces [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--logs |
text | yes | Trace source: a JSONL file for langchain/openai, or the directory soup serve --trace-log writes for soup-serve (a single file is refused) |
|
--format |
text | yes | Trace format: langchain | openai | soup-serve | |
--signal |
text | thumbs_up |
Signal to extract pairs from: thumbs_up | regenerations | user_edit | |
--output, -o |
text | prefs.jsonl |
Output path for preference pairs (JSONL) | |
--judge |
flag | off | Filter pairs via LLM-as-a-judge confidence (v0.40.3 #33). | |
--judge-provider |
text | openai |
Judge backend: openai | server | ollama. Used with --judge. | |
--judge-model |
text | gpt-4o-mini |
Judge model id (e.g. 'gpt-4o-mini', 'llama3', 'qwen2.5'). Used with --judge. | |
--judge-api-base |
text | Judge API base URL. SSRF-protected. Used with --judge. | ||
--min-confidence |
float | 0.7 |
Drop pairs with judge-confidence below this threshold (0.0 - 1.0). |
soup data gen-magpie
Generate synthetic data from a chat-template prefix.
Usage: soup data gen-magpie [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base |
text | yes | Aligned chat-tuned base model id | |
--provider |
text | yes | ollama | anthropic | vllm | |
--target |
integer | yes | Target row count [1, 1_000_000] | |
--output, -o |
text | "" |
Output JSONL path (required unless --plan-only) | |
--base-url |
text | "" |
Provider base URL (ollama/vllm; defaults to loopback) | |
--max-tokens |
integer | 512 |
Max tokens per generation [1, 16384] | |
--quality-filter / --no-quality-filter |
flag | on | Apply non-empty + educational heuristics (not a toxicity classifier) | |
--plan-only |
flag | off | Validate + print plan; do not generate. |
soup data generate
Generate synthetic training data using an LLM.
Usage: soup data generate [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--prompt, -p |
text | yes | System prompt describing what kind of data to generate | |
--count, -n |
integer | 100 |
Number of examples to generate | |
--output, -o |
text | generated.jsonl |
Output file path | |
--format, -f |
text | alpaca |
Output format: alpaca, sharegpt, chatml | |
--provider |
text | openai |
LLM provider: openai, local, server, ollama, anthropic, vllm | |
--model, -m |
text | gpt-4o-mini |
Model name (OpenAI model ID, Ollama model, or local model path) | |
--api-key |
text | [deprecated] Use OPENAI_API_KEY env var instead. Env: OPENAI_API_KEY. |
||
--api-base |
text | Custom API base URL (must use HTTPS for remote APIs) | ||
--batch-size |
integer | 5 |
Number of examples per API call | |
--temperature, -t |
float | 0.8 |
Sampling temperature for generation | |
--dedup-with |
text | Path to existing dataset to deduplicate against | ||
--seed |
text | Path to seed examples file (JSONL) to guide generation | ||
--template |
text | Domain template: code, conversation, qa, preference, reasoning | ||
--language |
text | Python |
Language for code template (Python, JavaScript, Go, Rust, Java) | |
--task-type |
text | function |
Task type for code template (function, debug, explain, refactor, test) | |
--turns |
integer | 4 |
Number of turns for conversation template (2-10) | |
--topic |
text | general knowledge |
Topic for conversation template | |
--context |
text | Path to context document for QA template | ||
--pref-task |
text | dpo |
Preference task format: dpo, kto, orpo | |
--domain |
text | math |
Domain for reasoning template (math, logic, code) | |
--ollama-model |
text | Ollama model name (e.g. llama3.1). Shorthand for --provider ollama --model X | ||
--validate |
flag | off | Auto-validate generated data after generation | |
--filter |
flag | off | Auto-filter generated data by quality (perplexity + coherence) | |
--dedup |
flag | off | Auto-dedup generated data (MinHash) | |
--quality-pipeline |
flag | off | Run full quality pipeline: validate + filter + dedup | |
--requests-per-minute, --rpm |
integer | 60 |
Rate limit for API requests (default: 60) | |
--trust-remote-code |
flag | off | Allow loading local-provider models that ship custom Python via auto_map. Default deny (v0.36.0). |
soup data ingest
Ingest a document into JSONL with one row per page / heading.
Usage: soup data ingest [OPTIONS] FILE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
FILE |
text | yes | PDF / DOCX / MD / TXT file | |
--output, -o |
text | ./ingested.jsonl |
Output JSONL path |
soup data inspect
Inspect a dataset: show stats and sample rows.
Usage: soup data inspect [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file (jsonl, csv, parquet) | |
--rows, -r |
integer | 5 |
Number of sample rows to show | |
--image-dir |
text | Directory containing referenced images |
soup data langdetect
Tag each row with a 2-letter language code (heuristic).
Usage: soup data langdetect [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL file to tag. | |
--output, -o |
text | langdetect.jsonl |
Output JSONL file with language tags. |
soup data lint
Preference-data linter (dpo/orpo/simpo/ipo/bco/kto).
Usage: soup data lint [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to preference dataset file (dpo/kto). | |
--format, -f |
text | auto |
Dataset format: auto, dpo, kto. | |
--model, -m |
text | Optional tokenizer for exact token-length bias (default: word count). | ||
--sample |
integer | 2000 |
Number of rows to sample for the checks. | |
--trust-remote-code |
flag | off | Allow loading a tokenizer that ships custom code. | |
--output, -o |
text | Write the report JSON to this path. |
soup data merge
Merge multiple datasets into a single file.
Usage: soup data merge [OPTIONS] FILES...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
FILES... |
text, any number | yes | Paths to dataset files to merge | |
--output, -o |
text | merged.jsonl |
Output file path | |
--shuffle |
flag | off | Shuffle after merging |
soup data mix
BETA: optimise per-dataset mixture weights against a proxy run.
Usage: soup data mix [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--optimize |
flag | off | Run Bayesian search over dataset mixture weights. | |
--apply |
text | Re-print a previously written mix-recipe (path under cwd). | ||
--datasets |
text | Comma-separated list of dataset JSONL paths (>= 2, all under cwd). | ||
--budget |
text | 1h |
Wall-clock cap: digits + optional s/m/h suffix (e.g. 1h, 30m). | |
--num-probes, -n |
integer, 1 to 256 | 8 |
Maximum number of proxy runs. | |
--seed |
integer, 0 to 2147483647 | 42 |
RNG seed for the optimiser. | |
--output, -o |
text | mix_recipe.yaml |
YAML recipe output path (under cwd). | |
--overwrite |
flag | off | Overwrite the output path when it exists. | |
--live |
flag | off | v0.53.5 #116 — run live short soup train proxy runs per candidate (requires --base-yaml). Defaults to the offline synthetic proxy. |
|
--base-yaml |
text | Path to a base soup.yaml (under cwd) supplying base/task/training/output. Required with --live. |
soup data persona-mix
Sample a prompt × persona × style matrix.
Usage: soup data persona-mix [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--prompts |
text | yes | JSONL file with one prompt per row | |
--n |
integer | yes | Number of rows to emit [1, 1_000_000] | |
--output |
text | yes | Output JSONL path | |
--personas |
text | "" |
Optional JSONL of personas (defaults to bundled set) | |
--styles |
text | "" |
Optional JSONL of styles (defaults to bundled set) | |
--seed |
integer | 0 |
Deterministic seed |
soup data pii
Flag rows containing email / phone / SSN / credit-card patterns.
Usage: soup data pii [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL file to scan. | |
--output, -o |
text | pii.jsonl |
Output JSONL file containing rows with detected PII. |
soup data preprocess
AOT-tokenize a dataset and cache to disk for reuse across runs.
Usage: soup data preprocess [OPTIONS] CONFIG_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
CONFIG_PATH |
text | yes | Path to soup.yaml — uses data.train + tokenizer + max_length | |
--output, -o |
text | ./.soup-tokenized |
Cache directory under cwd | |
--yes |
flag | off | Skip confirmation prompt |
soup data preview
Preview a remote HuggingFace dataset: metadata, splits, features.
Usage: soup data preview DATASET_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATASET_ID |
text | yes | HuggingFace dataset ID (e.g. teknium/OpenHermes-2.5) |
soup data push
Upload a local JSONL dataset to a model hub as a dataset repo.
Usage: soup data push [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Path to local JSONL dataset file | |
--hf-dataset |
text | yes | HuggingFace dataset repo id (e.g. user/my-dataset) | |
--private |
flag | off | Make the HF dataset repo private | |
--message |
text | Upload dataset with Soup CLI |
Commit message for the dataset upload | |
--hub |
text | hf |
Target hub: hf (default) / modelscope / modelers. Non-HF hubs upload via the matching SDK (v0.71.5 #157) — repo_type=dataset, commit message sanitised to first line + 200 chars. |
soup data recipe
Validate a Data Recipe DAG and optionally execute every node.
Usage: soup data recipe [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to recipe.yaml under cwd | |
--execute |
flag | off | Run the validated DAG end-to-end (v0.53.7 #106 — live per-node execution: seed / llm_text / code / judge / validator / sampler). | |
--output, -o |
text | Output dir for sampler node (required with --execute). | ||
--provider |
text | LLM provider for llm_text/judge nodes: ollama, anthropic, or vllm. | ||
--model |
text | Provider model name (defaults to llama3.1). | ||
--base-url |
text | Provider base URL (Ollama/vLLM; defaults to loopback). | ||
--offline |
flag | off | Explicitly allow placeholder generation and accept-all judging. |
soup data register
Register a local dataset by name for use in soup.yaml.
Usage: soup data register [OPTIONS] [NAME_ARG] [PATH_ARG]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME_ARG |
text | Dataset name (positional alternative to --name) | ||
PATH_ARG |
text | Path to dataset file (positional alternative to --path) | ||
--name, -n |
text | Dataset name | ||
--path, -p |
text | Path to dataset file | ||
--format, -f |
text | auto |
Dataset format: alpaca, sharegpt, chatml, dpo, kto, auto |
soup data registry
List all registered datasets.
Usage: soup data registry
Takes no options or arguments.
soup data review
Preview preference pairs for manual review.
Usage: soup data review [OPTIONS] INPUT
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
INPUT |
text | yes | Path to preference JSONL (chosen/rejected) | |
--sample, -s |
integer | 10 |
How many pairs to preview (1-100) |
soup data sample
Sample a subset of a dataset using various strategies.
Usage: soup data sample [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--output, -o |
text | Output file path (default: <input>_sampled.jsonl) | ||
--n, -n |
integer | Number of samples to select | ||
--pct |
float | Percentage of dataset to sample (0-100) | ||
--strategy, -s |
text | random |
Sampling strategy: random, diverse (TF-IDF + clusters), hard (by length) | |
--seed |
integer | Random seed for reproducibility |
soup data score
Composite data-quality scorecard (PII + keyword triage + lang + edu + dec).
Usage: soup data score [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | JSONL input under cwd. | |
--benchmarks, -b |
text | Benchmark labels (e.g. mmlu,gsm8k), validated only; they do not select or filter --benchmark-file texts. Requires --benchmark-file; corpora are not bundled. | ||
--benchmark-file |
text | Operator-supplied benchmark JSONL under cwd, with usable 8-gram text in every row. | ||
--threshold |
float, 0.0 to 1.0 | 0.8 |
Decontamination overlap threshold. |
soup data search
Search HuggingFace Hub for datasets.
Usage: soup data search [OPTIONS] QUERY
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
QUERY |
text | yes | Search query for HuggingFace datasets | |
--limit, -l |
integer | 20 |
Maximum results to show | |
--sort, -s |
text | downloads |
Sort by: downloads, likes, lastModified, trending, createdAt |
soup data split
Split dataset into train/val/test files.
Usage: soup data split [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--val |
integer | Validation split: percentage (default) or absolute count (with --absolute) | ||
--test |
integer | Test split: percentage (default) or absolute count (with --absolute) | ||
--train |
integer | Train split (informational; the train remainder is implied by --val + --test). Accepted for command parity. | ||
--absolute |
flag | off | Treat --val/--test as absolute sample counts instead of percentages | |
--seed |
integer | Random seed for reproducible splits | ||
--stratify |
text | Field name for stratified splitting (preserves category distribution) | ||
--stratify-semantic |
flag | off | Use semantic clustering (TF-IDF + K-Means) to perform stratified splitting without requiring a category field | |
--num-clusters |
integer | Number of semantic clusters to use for semantic stratified splitting (default: 5) |
soup data stats
Extended dataset statistics: length distribution, token counts, languages.
Usage: soup data stats PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file |
soup data topics
Cluster a dataset into topics and show a coverage table.
Usage: soup data topics [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file (jsonl/json/csv) | |
--clusters, -k |
text | auto |
Number of topic clusters, or 'auto' (sqrt heuristic). | |
--embed-model |
text | sentence-transformers/all-MiniLM-L6-v2 |
Embedding model. | |
--device |
text | auto |
auto/cpu/cuda | |
--seed |
integer | 0 |
Clustering seed. | |
--output, -o |
text | Write the report as JSON. |
soup data toxicity
Run an abuse-keyword heuristic (not a toxicity classifier).
Usage: soup data toxicity [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL file to score. | |
--output, -o |
text | toxicity.jsonl |
Output JSONL file for rows meeting the threshold. | |
--threshold |
float, 0.0 to 1.0 | 0.05 |
Rows meeting the abuse-keyword threshold are kept. |
soup data unregister
Remove a dataset from the local registry.
Usage: soup data unregister [OPTIONS] [NAME_ARG]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME_ARG |
text | Dataset name (positional alternative to --name) | ||
--name, -n |
text | Dataset name to remove |
soup data validate
Validate a dataset, returning exit 3 for input errors and 2 for unusable data.
Usage: soup data validate [OPTIONS] PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | yes | Path to dataset file | |
--format, -f |
text | auto |
Expected format: auto, alpaca, sharegpt, chatml, dpo, kto, plaintext | |
--min-valid-fraction |
float, 0.0 to 1.0 | 0.0 |
Exit 2 when the fraction of valid rows is below this value |
soup delinearize-llama4
Delinearize Llama 4 fused expert weights to 3-D form.
Usage: soup delinearize-llama4 [OPTIONS] SOURCE_DIR
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SOURCE_DIR |
text | yes | Llama 4 checkpoint directory (containing *.safetensors). | |
--target, -o |
text | yes | Destination directory for the delinearized weights (under cwd). | |
--model-id |
text | Optional model id; warn if it doesn't look like a Llama 4 model. | ||
--num-experts |
integer | Expert count for the reshape. Defaults to config.json's num_local_experts. | ||
--plan-only |
flag | off | Render the plan and exit without reshaping (v0.44.0 flow). |
soup deploy
Deploy models: Ollama integration (deploy, list, remove).
Usage: soup deploy <command> [ARGS]...
Subcommands:
soup deploy autopilot
Pick PEFT + quant + spec-decoding combo for a hardware target.
Usage: soup deploy autopilot [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--target, -t |
text | Profile name (e.g. mac-m3, rtx-4090-24gb, ollama-local). | ||
--base, -b |
text | meta-llama/Llama-3.2-1B |
Base model HF repo id or local path to embed in the recipe. | |
--recipe-out |
text | deploy_autopilot.yaml |
Where to write the rendered soup.yaml recipe (under cwd). | |
--script-out |
text | deploy_autopilot.sh |
Where to write the planned deploy shell script (under cwd). | |
--output-dir |
text | ./output |
Recipe's training output directory. | |
--list, -l |
flag | off | List all known deploy profiles. | |
--measure |
flag | off | Run Quant-Lobotomy measurement on candidate quants. Requires --tasks. Results cached at ~/.soup/deploy_autopilot_cache.json (v0.53.1 #109). | |
--tasks |
text | JSONL eval tasks for --measure (one prompt + expected per line). | ||
--measure-candidates |
text | Comma-separated quant candidates to measure (default: profile's primary quant). Example: 4bit,gptq,awq |
soup deploy hf-space
Create a HuggingFace Space wrapping a fine-tuned model.
Usage: soup deploy hf-space [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | HuggingFace model repo id to wrap in the Space (e.g. user/my-model) | |
--space, -s |
text | yes | HuggingFace Space repo id to create (e.g. user/my-space) | |
--template, -t |
text | gradio-chat |
Space template: gradio-chat, streamlit-chat | |
--template-dir |
text | Custom template directory (overrides --template). Must contain app.py + README.md, optionally requirements.txt. Use {MODEL_REPO} placeholder for substitution. | ||
--private |
flag | off | Create the Space as private | |
--yes, -y |
flag | off | Skip confirmation |
soup deploy ollama
Deploy a GGUF model to local Ollama instance.
Usage: soup deploy ollama [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | Path to GGUF model file | ||
--name, -n |
text | Ollama model name (e.g. soup-my-model) | ||
--system, -s |
text | System prompt for the model | ||
--template, -t |
text | auto |
Chat template: auto, chatml, llama, mistral, vicuna, zephyr | |
--parameter, -p |
text, repeatable | Ollama parameter (repeatable): temperature=0.7, top_p=0.9, etc. | ||
--list, -l |
flag | off | List Soup-deployed models in Ollama | |
--remove, -r |
text | Remove a model from Ollama by name | ||
--yes, -y |
flag | off | Skip confirmation prompt |
soup diagnose
Post-training report card: forgetting / refusal / format / mode_collapse / memorization / contamination (v0.56.0).
Usage: soup diagnose [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Registry run id (or any opaque tag). | |
--base |
text | "" |
Base model name (informational). | |
--adapter |
text | "" |
Adapter name (informational). | |
--evidence |
text | JSON file with pre-computed probe scores (see README). | ||
--output, -o |
text | Write the report JSON to this path. | ||
--badge |
text | Write an SVG badge to this path. | ||
--attach-to-registry |
text | Attach the --output report to a registry entry id (needs --output). Exits 1 when the attach fails. | ||
--base-model |
text | Base model id to LOAD for a live diagnose run (e.g. HuggingFaceTB/SmolLM2-135M). When set, the 6 probes run live against the model instead of emitting neutral scores. | ||
--dataset |
text | Training JSONL for live format / memorization / citation probes (must stay under cwd). | ||
--holdout |
text | Held-out evaluation JSONL for the live forgetting probe (must stay under cwd; falls back to built-in general prompts if unset). | ||
--tokenizer |
text | Tokenizer id/path for a sub-word memorization probe (live). | ||
--device |
text | Device for the live run (cuda / cpu). | ||
--citation-style |
text | bracket |
Citation style for the live RAFT citation probe: bracket ([doc-id]) / inline ((doc-id)) / footnote ([^id]). Must match the style the model was trained to emit. (v0.71.17 #254) | |
--allow-not-run |
flag | off | Exit 0 even when a requested probe did not run (verdict NOT_RUN). Without it such a run exits 3. | |
--shuffle-seed |
integer | RAFT document-shuffle seed for the live citation probe (composes the [doc-N] ids the model is scored against). (v0.71.17 #254) |
soup diff
Compare outputs of two models side-by-side on the same prompts.
Usage: soup diff [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model-a, -a |
text | yes | Path to first model (LoRA adapter or full model) | |
--model-b, -b |
text | yes | Path to second model (LoRA adapter or full model) | |
--prompts, -p |
text | Path to prompts file (JSONL with 'prompt' field, or one prompt per line) | ||
--prompt |
text, repeatable | Single prompt to compare (can be repeated) | ||
--base-a |
text | Base model for model A (auto-detected for LoRA adapters) | ||
--base-b |
text | Base model for model B (auto-detected for LoRA adapters) | ||
--max-tokens |
integer | 256 |
Maximum tokens to generate per response | |
--temperature, -t |
float | 0.7 |
Sampling temperature | |
--device |
text | Device: cuda, mps, cpu. Auto-detected if not set. | ||
--output, -o |
text | Save results to JSONL file | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. |
soup distill-prompt
Distill prompt-heavy traces into a small FT plan (v0.68.0).
Usage: soup distill-prompt [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--traces |
text | yes | Path to traces JSONL | |
--teacher |
text | yes | Teacher model id | |
--student |
text | yes | Student model id | |
--strategy |
text | sft |
Distill strategy. Allowed: kl, preference, sft | |
--provider |
text | ollama |
Teacher/student provider: ollama / anthropic / vllm. | |
--base-url |
text | Provider base URL (ollama / vllm). | ||
--temperature |
float | 0.0 |
Sampling temperature for teacher/student. | |
--max-rows |
integer | Cap the number of distilled rows. | ||
--output, -o |
text | distilled.jsonl |
Output JSONL path | |
--plan-only |
flag | off | Render plan + exit 0 (no live preparation). |
soup doctor
Check system dependencies, GPU, and compatibility.
Usage: soup doctor [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--nccl |
flag | off | Measure NCCL bandwidth and check against reference table. | |
--disk |
flag | off | Detect the disk media type (NVMe / SSD / HDD). Layer streaming's disk overflow tier needs NVMe. Off by default: the probe costs ~9 s on Windows, and on Linux it WRITES a ~64 MiB scratch file (.soup-diskprobe-*, git-ignored) into the current directory to measure sequential read throughput. | |
--config, -c |
text | Also check a soup.yaml: report which of the settings it actually sets are not read on its task and backend (#755). |
soup draft
Train + measure a speculative-decoding draft model: distil your tuned model into a tiny draft, then serve it with --auto-spec (v0.71.33).
Usage: soup draft <command> [ARGS]...
Subcommands:
soup draft distill
Distil a target model into a tiny dense speculative-decoding draft.
Usage: soup draft distill [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--target |
text | yes | The tuned model to speed up (the teacher). | |
--draft-base |
text | yes | Tiny model to distil into (the student). Must share the target's tokenizer. A soup shrink output qualifies by construction. |
|
--data |
text | yes | JSONL distillation set (chat / instruction rows). | |
-o, --output |
text | draft |
Directory for the dense draft model. | |
--steps |
integer, 1 to 1000000 | 500 |
Approximate training steps. | |
--device |
text | cpu | cuda (default: auto-detect). | ||
--no-register |
flag | off | Do not record the draft in ~/.soup/drafts.json (it then won't be picked up by soup serve --auto-spec). |
|
--trust-remote-code |
flag | off | Allow custom modelling code from the target / draft-base (required for architectures that ship an auto_map). | |
--force |
flag | off | Overwrite -o even if it already contains a non-draft directory. | |
--plan-only |
flag | off | Print the distill config and exit; write nothing. |
soup draft list
List locally-trained drafts that soup serve --auto-spec can use.
Usage: soup draft list
Takes no options or arguments.
soup draft measure
Report a draft's acceptance rate + throughput against its target.
Usage: soup draft measure [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--target |
text | yes | The model being served. | |
--draft |
text | yes | The draft model (a soup draft distill output). |
|
--prompts |
text | yes | JSONL prompts representative of production traffic. | |
--max-new-tokens |
integer, 1 to 4096 | 64 |
Tokens to generate per prompt. | |
--num-assistant-tokens |
integer, 1 to 64 | 5 |
Tokens the draft proposes per step (the assisted-generation knob). | |
--device |
text | cpu | cuda. | ||
--min-acceptance |
float, 0.0 to 1.0 | Exit 2 if the acceptance rate falls below this (for CI gating). | ||
--trust-remote-code |
flag | off | Allow custom modelling code from the target / draft (required for architectures that ship an auto_map). | |
-o, --output |
text | Write the report as JSON. | ||
--sweep-k |
text | Also time assisted generation at each of these draft lengths, e.g. 1,2,3,5,8 (at most 8, each in 1..64). |
soup drift-alarm
Rolling-KL drift alarm on output-token distribution with optional Slack/Discord webhook (v0.63.0).
Usage: soup drift-alarm [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--reference |
text | yes | JSONL of FT-time reference outputs. | |
--live |
text | yes | JSONL of live production outputs. | |
--threshold |
float | 0.2 |
Drift threshold (KL divergence). Default 0.2. | |
--slack-url |
text | Optional Slack webhook URL — POSTed on drift. SSRF-validated. | ||
--discord-url |
text | Optional Discord webhook URL — POSTed on drift. SSRF-validated. |
soup edit
Knowledge editing (ROME / MEMIT / AlphaEdit) - patch facts without re-training (v0.61.0).
Usage: soup edit <command> [ARGS]...
Subcommands:
soup edit diff
Knowledge-injection diff: facts changed between before / after.
Usage: soup edit diff [OPTIONS] BEFORE AFTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BEFORE |
text | yes | Label or run id of the model BEFORE the edit (metadata). | |
AFTER |
text | yes | Label or run id of the model AFTER the edit (metadata). | |
--probes |
text | JSONL file with probe prompts; required when both --before-model and --after-model are given. Each row should have a 'prompt' field. Capped at 1000 rows. | ||
--output, -o |
text | Where to write the rendered diff JSON. | ||
--top-k, -k |
integer | 10 |
Number of changed facts to surface (1-100). | |
--before-model |
text | Model path / HF id of the BEFORE model (live generation with --after-model and --probes). | ||
--after-model |
text | Model path / HF id of the AFTER model (live generation with --before-model and --probes). | ||
--device |
text | torch device (cpu / cuda). Defaults to CUDA when available. |
soup edit set
Apply a single surgical knowledge edit.
Usage: soup edit set [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base, -b |
text | yes | Base model HF id or local path. | |
--method, -m |
text | rome |
Edit method: rome / memit / alphaedit. | |
--subject, -s |
text | yes | Prefix sentence (e.g. "Paris is the capital of France"). | |
--target, -t |
text | yes | New completion target (e.g. "Lyon"). | |
--layer, -l |
integer | MLP layer index to edit (defaults to method-specific recommended layer). | ||
--output, -o |
text | Directory to save the edited model / GRACE codebook (cwd-contained). | ||
--device |
text | torch device (cpu / cuda). Defaults to CUDA when available. | ||
--governor / --no-governor |
flag | on | Consult the sequential-edit governor (refuse on norm blowup). | |
--cov-corpus |
text | ROME only: JSONL/text corpus to estimate the key covariance C for a C^{-1}-preconditioned update (reduces collateral damage). | ||
--plan-only |
flag | off | Print the resolved EditPlan and exit without applying. | |
--registry-id |
text | Registry entry id to attach the saved result to (needs --output): the weight file(s) as edited_model, or the GRACE codebook as grace_codebook. Exits 1 when the attach fails. |
soup env
Hermetic env lockfile + ABI drift detection (v0.64.0).
Usage: soup env <command> [ARGS]...
Subcommands:
soup env check
Compare the current env against the lock and report drift.
Usage: soup env check [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--lock |
text | soup-env.lock |
Path to the lock file to compare against. |
soup env fix
Render a reproducible install plan from soup-env.lock.
Usage: soup env fix [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--lock |
text | soup-env.lock |
Path to the lock file to render an install plan from. | |
--format |
text | uv-pip |
Install-plan format: uv-pip (copy/paste uv commands) | requirements. | |
--output, -o |
text | Optionally also write a requirements.txt to this path (under cwd). |
soup env lock
Snapshot the current env into a lock file.
Usage: soup env lock [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--output, -o |
text | soup-env.lock |
Path to write the lock file (default: ./soup-env.lock). |
soup env status
Print currently-locked env summary.
Usage: soup env status [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--lock |
text | soup-env.lock |
Path to the lock file (default: ./soup-env.lock). |
soup eval
Evaluate models: benchmarks, custom evals, LLM judge, leaderboard.
Usage: soup eval <command> [ARGS]...
Subcommands:
againstaiderautobehaviorbenchmarkcapabilitychecklistcitationcomparecoveragecustomdesigndiscovergategate-installhumanirt-subsetjudgeleaderboardlockquant-checkunlearning
soup eval against
Run-vs-run regression check (paired-bootstrap CI).
Usage: soup eval against [OPTIONS] BASELINE_RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BASELINE_RUN_ID |
text | yes | Baseline run id (from soup runs). |
|
--candidate |
text | yes | Candidate run id whose metrics are compared to the baseline. | |
--metric |
text | task_accuracy |
Metric to check: task_accuracy | refusal_rate | format_validity | p95_latency_ms | custom | aider_polyglot | judge:<model> | benchmark:<lm-eval-task>. | |
--n-samples |
integer | 1000 |
Paired-bootstrap samples (100-100000). | |
--seed |
integer | 0 |
Deterministic seed. | |
--json-only |
flag | off | Suppress Rich output; emit a single JSON verdict line. | |
--suite |
text | Locked eval suite (from soup eval lock) to validate as a gate precondition — a missing / unparseable suite BLOCKS the check (exit 3). cwd-contained. The generated pre-push hook passes $GATE_SUITE here so the locked suite is actually enforced. |
soup eval aider
Run Aider's Polyglot coding benchmark in its official Docker image.
Usage: soup eval aider [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Aider model identifier (for example, openai/gpt-4.1) | |
--output, -o |
text | yes | Contained directory for upstream and Soup result files | |
--exercises-dir |
text | tmp.benchmarks/polyglot-benchmark |
Prepared Aider-AI/polyglot-benchmark checkout | |
--image |
text | aider-benchmark |
Locally built upstream Aider benchmark image | |
--threads, -t |
integer, 1 to 64 | 1 |
Number of benchmark exercises to run concurrently | |
--num-tests, -n |
integer | -1 |
Number of exercises to run (-1 runs all exercises) | |
--timeout |
integer, 60 to 86400 | 86400 |
Maximum benchmark runtime in seconds | |
--run-id |
text | Save the aggregate score under an existing Soup run ID | ||
--allow-host-services |
flag | off | Let the benchmark container reach services on the host |
soup eval auto
Run automatic evaluation using config from soup.yaml.
Usage: soup eval auto [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml config file | |
--benchmarks, -b |
text | Comma-separated benchmarks (overrides config) | ||
--tasks, -t |
text | Path to custom eval JSONL (overrides config) | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. |
soup eval behavior
Score a run on a bundled behaviour battery (pre/post diff).
Usage: soup eval behavior [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run identifier. | |
--battery, -b |
text | xstest |
Battery: xstest / harmbench / jailbreakbench / elephant / syceval. | |
--evidence, -e |
text | Path to a JSON file with {pre_responses, post_responses, oracle} arrays. Here each oracle entry is a word the matching response must contain; --base-model scores refusal behaviour instead. | ||
--output, -o |
text | Where to write the rendered BehaviorDiffReport JSON. | ||
--base-model |
text | Base model id for a LIVE diff (generates pre/post responses). | ||
--adapter |
text | LoRA adapter path for the 'post' model in a live diff. | ||
--device |
text | Device for the live diff (cuda / cpu). |
soup eval benchmark
Evaluate on standard benchmarks (wraps lm-evaluation-harness).
Usage: soup eval benchmark [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to model or LoRA adapter directory | |
--benchmarks, -b |
text | mmlu |
Comma-separated benchmark names (mmlu, gsm8k, hellaswag, etc.) | |
--fewshot, -f |
integer | Number of few-shot examples (benchmark default if not set) | ||
--batch-size |
integer | 8 |
Batch size for evaluation | |
--run-id |
text | Link results to an existing training run | ||
--device |
text | Device: cuda, mps, cpu. Auto-detected if not set. | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. |
soup eval capability
Run a bundled capability profile (MMLU-Pro / GPQA / AIME / ...).
Usage: soup eval capability [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run identifier. | |
--suite, -s |
text | fast |
Profile: full / fast / math / code. | |
--output, -o |
text | Where to write the rendered CapabilityReport JSON. | ||
--live |
flag | off | Run the suite LIVE via lm-eval-harness against --model. | |
--model, -m |
text | HF model id for a live run (required with --live). | ||
--tasks |
text | Comma-separated lm-eval task override (live; bypasses --suite). | ||
--limit |
integer | Cap eval examples per task (live; use 1-5 for a smoke). | ||
--device |
text | Device for the live run (cuda / cpu). |
soup eval checklist
Run CheckList MFT / INV / DIR behavioural tests.
Usage: soup eval checklist [OPTIONS] SPEC_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SPEC_PATH |
text | yes | Path to CheckList DSL YAML. | |
--evidence, -e |
text | Required JSON with operator-supplied per-test responses. | ||
--output, -o |
text | Where to write the rendered CheckListReport JSON. |
soup eval citation
Score citation precision / recall / F1 over a JSONL (#202).
Usage: soup eval citation [OPTIONS] DATA
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | JSONL of {predicted, expected_ids} rows OR RAFT rows {query, golden_doc, distractor_docs, answer}. | |
--style |
text | bracket |
Citation style: bracket / inline / footnote. | |
--shuffle-seed |
integer | For RAFT-shaped rows: the data.raft_shuffle_seed used at train time, so the golden [doc-N] id matches what the model saw. Ignored for {predicted, expected_ids} rows. | ||
--output, -o |
text | Write the per-row + aggregate CitationScore JSON here. |
soup eval compare
Compare eval results between two training runs.
Usage: soup eval compare RUN1 RUN2
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN1 |
text | yes | First run ID | |
RUN2 |
text | yes | Second run ID |
soup eval coverage
Heuristic coverage / gap analysis for a locked or drafted suite.
Usage: soup eval coverage [OPTIONS] DESIGN_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DESIGN_PATH |
text | yes | Path to an EvalDesign JSON | |
--task |
text | yes | Task category from v0.54.0 taxonomy: factual_lookup | style_shaping | format_conversion | reasoning | tool_use | summarization | classification. |
soup eval custom
Run custom evaluation tasks from a JSONL file.
Usage: soup eval custom [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--tasks, -t |
text | yes | Path to eval tasks JSONL file | |
--model, -m |
text | yes | Path to model directory | |
--run-id |
text | Link results to an existing training run | ||
--attach-to-registry |
text | Attach the eval JSON to a registry entry as kind=eval_results | ||
--output, -o |
text | Path to write the eval JSON output. Honored independently of --attach-to-registry (v0.40.1 / G10). | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. |
soup eval design
Draft an evaluation suite from training data + a one-line goal.
Usage: soup eval design [OPTIONS] DATA
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | Training-data JSONL path | |
--goal, -g |
text | yes | One-line goal description (e.g. 'better at SQL'). | |
--num-dimensions, -n |
integer | 5 |
Number of eval dimensions to draft (1-20). | |
--output, -o |
text | evals/design.json |
Where to write the rendered EvalDesign JSON. |
soup eval discover
Discover a held-out canary set for regression detection.
Usage: soup eval discover [OPTIONS] DATA
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | Training-data JSONL path | |
--base |
text | Base model id (recorded; consumed by soup diagnose). |
||
--num-clusters |
integer | 5 |
Number of behavioural clusters to discover (1-64). | |
--per-cluster |
integer | 3 |
Held-out canaries to draw per cluster (1-64). | |
--seed |
integer | 0 |
Deterministic seed. | |
--output, -o |
text | evals/canaries.json |
Where to write the rendered CanarySet JSON. |
soup eval gate
Run an eval-gate suite standalone (post-hoc verdict).
Usage: soup eval gate [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--suite, -s |
text | yes | Path to eval suite YAML (see evals/gate.yaml example) | |
--baseline, -b |
text | Baseline: registry://<id> or path to stamped/flat score JSON | ||
--regression-threshold |
float | 0.05 |
Max absolute drop vs baseline before regression fires (0.0-1.0) | |
--model, -m |
text | Model path or HF id to evaluate (required for live scoring) | ||
--write-baseline |
text | Write this run's task scores as a stamped baseline JSON ({'scores', 'provenance'}) consumable by --baseline (#404). |
soup eval gate-install
Install a pre-push regression gate for the selected baseline.
Usage: soup eval gate-install [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--baseline |
text | yes | Baseline run id the pre-push hook compares against. | |
--suite |
text | evals/locked.json |
Path to the locked eval suite (cwd-contained). | |
--hook-path |
text | .git/hooks/pre-push |
Hook target — usually .git/hooks/pre-push. | |
--force |
flag | off | Overwrite an existing hook. |
soup eval human
Run human A/B evaluation between two models.
Usage: soup eval human [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Path to JSONL file with evaluation prompts | |
--model-a, -a |
text | yes | Path to first model | |
--model-b, -b |
text | yes | Path to second model | |
--output, -o |
text | human_eval_results.json |
Output file for results |
soup eval irt-subset
Pick a minimum-cost eval subset that preserves ranking power.
Usage: soup eval irt-subset [OPTIONS] RESPONSES_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RESPONSES_PATH |
text | yes | Path to JSONL with rows {item_id, correct(bool), respondent_id? (required for 2pl/3pl)}. | |
--size, -z |
text | small |
Subset profile: full / small / tiny. | |
--model, -m |
text | 1pl |
IRT model: 1pl (Rasch) / 2pl (discrimination) / 3pl (+ guessing). | |
--output, -o |
text | Where to write the rendered IrtSubsetPlan JSON. |
soup eval judge
Evaluate model outputs using LLM-as-a-judge.
Usage: soup eval judge [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--target |
text | yes | Path to JSONL with prompt+response pairs to evaluate | |
--model, -m |
text | gpt-4o-mini |
Judge model name | |
--provider |
text | openai |
Judge provider: openai, server, ollama | |
--rubric, -r |
text | Path to rubric YAML file | ||
--api-base |
text | API base URL for judge model | ||
--run-id |
text | Link results to an existing training run |
soup eval leaderboard
Show local leaderboard across all evaluated models.
Usage: soup eval leaderboard [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--sort-by, -s |
text | Sort by specific benchmark (default: average) | ||
--format, -f |
text | table |
Output format: table, json, csv |
soup eval lock
Freeze an EvalDesign as a checksummed eval_suite artifact.
Usage: soup eval lock [OPTIONS] DESIGN_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DESIGN_PATH |
text | yes | Path to an EvalDesign JSON (from soup eval design) |
|
--output, -o |
text | evals/locked.json |
Where to write the canonicalised locked suite. | |
--attach-to-registry |
text | Registry entry id or name to attach the locked suite to. |
soup eval quant-check
Compare accuracy before vs after quantization on the same eval suite.
Usage: soup eval quant-check [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--before |
text | yes | Before-quantization model path or registry://<id> | |
--after |
text | yes | After-quantization model path or registry://<id> | |
--tasks |
text | yes | JSONL eval tasks file (see 'soup eval custom') | |
--format |
text | table |
Output format: table | json | markdown | |
--allow-stub |
flag | off | Allow deterministic stub generators if live model loading fails. |
soup eval unlearning
Score an unlearn run on TOFU / MUSE / WMDP.
Usage: soup eval unlearning [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run identifier (e.g. registry id of the unlearn run). | |
--benchmark, -b |
text | tofu |
Benchmark: tofu / muse / wmdp. | |
--evidence, -e |
text | Path to a JSON file with pre-computed evidence. Schema: {forget_quality: {pre_loss, post_loss}, model_utility: {pre_acc, post_acc}, priv_leak: {mia_auc}}. Missing keys fall through to a neutral OK score. | ||
--output, -o |
text | Where to write the rendered UnlearnReport JSON. | ||
--attach-to-registry |
text | Registry entry id to attach the --output report to as an eval_results artifact (needs --output). Exits 1 when the attach fails. |
soup expect
Run an expectations suite against a JSONL dataset.
Usage: soup expect [OPTIONS] DATA SUITE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
DATA |
text | yes | Path to JSONL dataset | |
SUITE |
text | yes | Path to expectations suite YAML | |
--judge |
text | Judge model URL (e.g. ollama://llama3.1, https://api.openai.com/gpt-4o-mini) |
soup export
Export a model to GGUF, ONNX, TensorRT-LLM, BitNet, or TorchAO format.
Usage: soup export [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to model directory (full model or LoRA adapter) | |
--format, -f |
text | gguf |
Export format: gguf, onnx, tensorrt, bitnet, tq1_0, torchao, gguf-ud (torchao needs --quant-config) | |
--quant, -q |
text | q4_k_m |
Quantization type: q4_0, q4_k_m, q5_k_m, q8_0, f16, f32 | |
--output, -o |
text | Output file path. Default: <model-name>.<quant>.gguf | ||
--base, -b |
text | Base model ID (for LoRA adapters). Auto-detected if not set. | ||
--llama-cpp |
text | Path to llama.cpp directory. Auto-detected or cloned to ~/.soup/llama.cpp | ||
--onnx-task |
text | text-generation |
ONNX export task: text-generation (causal LM) or feature-extraction (embedding) | |
--deploy |
text | Auto-deploy after export. Currently supported: ollama | ||
--deploy-name |
text | Model name for deployment (used with --deploy) | ||
--calibration-data |
text | Path to calibration JSONL for imatrix GGUF export. | ||
--registry-id |
text | Attach exported artifact to this registry entry (default: auto-match by source --model output dir) | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--quant-config |
text | Path to YAML for torchao PTQ export (v0.53.1 #142). Required when --format=torchao. | ||
--gguf-flavour |
text | Advanced GGUF format flag — UD-Q*_K_XL / IQ*_M / Q4_0_4_4 / etc. Required when --format=gguf-ud (v0.53.1 #139). | ||
--hub |
text | hf |
Source hub for the base model when --model is a LoRA adapter: hf (default) / modelscope / modelers (v0.53.10 #152). | |
--verbose |
flag | off | Stream GGUF converter and quantizer logs while they run. | |
--no-install-deps |
flag | off | GGUF only: never invoke pip; refuse missing conversion dependencies before work. |
soup fetch
Fetch a ready-to-edit example config from the bundled catalog.
Usage: soup fetch [OPTIONS] NAMESPACE [NAME]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAMESPACE |
text | yes | One of: examples, configs, deepspeed_configs. | |
NAME |
text | Catalog entry name (omit to list). | ||
--output, -o |
text | Destination path (default: ./<filename> in cwd). | ||
--force |
flag | off | Overwrite if --output already exists. |
soup history
Show the lineage tree for all entries with a given name.
Usage: soup history NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Registry entry name to trace (e.g. 'medical-chat') |
soup infer
Run batch inference on a JSONL file of prompts.
Usage: soup infer [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to model (LoRA adapter or full model) | |
--input, -i |
text | yes | Path to input JSONL file (each line: {"prompt": "..."}) | |
--output, -o |
text | yes | Path to output JSONL file for results | |
--base, -b |
text | Base model for LoRA adapter (auto-detected if not set) | ||
--max-tokens |
integer, 1 to 16384 | 256 |
Maximum tokens to generate per response (1-16384) | |
--temperature, -t |
float | 0.7 |
Sampling temperature (0 = greedy) | |
--batch-size |
integer, at least 1 | 1 |
Number of causal-LM prompts to generate together (default: 1; batching is opt-in; fp16/bf16 batching may not be bit-exact with batch size 1) | |
--device |
text | Device: cuda, mps, cpu. Auto-detected if not set. | ||
--task |
text | text |
Inference task: 'text' (default, chat generation) or 'asr' (Whisper transcription; input rows are {"audio": path[, "text": reference]}). | |
--asr-language |
text | ASR decode language (overrides the training sidecar). --task asr only. | ||
--asr-task |
text | ASR decode objective: transcribe | translate. --task asr only. | ||
--audio-dir |
text | Base directory audio paths in --input must stay under (--task asr; defaults to the --input file's directory). Traversal / UNC paths are rejected. | ||
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--hub |
text | hf |
Source hub for the base model: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK (v0.53.10 #152). | |
--cuda-graphs |
flag | off | Experimental CUDA graph decode for resident, unquantized Qwen2/Llama models. |
soup ingest
Universal trace importer: Langfuse / LangSmith / Helicone / OpenPipe / OTel / OpenAI Stored Completions (v0.63.0).
Usage: soup ingest [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--source |
text | yes | Trace source: langfuse | langsmith | helicone | openpipe | otel | openai-stored | |
--logs |
text | Path to JSONL trace export (one event per line). Required unless --pull. | ||
--output, -o |
text | Output JSONL (default: traces.jsonl in cwd). | ||
--pull |
flag | off | Fetch generations live from the Langfuse API instead of reading --logs (--source langfuse only). Reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY, plus LANGFUSE_HOST for another region or a self-hosted instance. | |
--since |
text | With --pull: how far back to fetch, e.g. 30m, 24h, 7d (default 7d, max 365d). | ||
--max-pages |
integer, 1 to 10000 | With --pull: stop with an error, writing nothing, if results are still pending after this many pages of 100 generations (default 100). | ||
--allow-private-host |
flag | off | With --pull: allow LANGFUSE_HOST to be a private or loopback address (self-hosted Langfuse). HTTPS is still required. | |
--slack-url |
text | Optional Slack webhook URL — POSTed on completion. SSRF-validated. | ||
--discord-url |
text | Optional Discord webhook URL — POSTed on completion. SSRF-validated. |
soup init
Create a new soup.yaml config interactively or from a template.
Usage: soup init [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--template, -t |
text | Template: audio, bco, chat, code, embedding, eu-ai-act, hipaa, ipo, kto, longcontext, medical, moe, orpo, pretrain, reasoning, rlhf, simpo, soc2, sr-11-7, tool-calling, vision | ||
--output, -o |
text | soup.yaml |
Output config file path | |
--force, -f |
flag | off | Overwrite existing config without prompting (v0.40.1 / M2). |
soup iterative-dpo
Iterative DPO sample, score, pair, and train loop.
Usage: soup iterative-dpo [OPTIONS] [<command>] [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base-model |
text | yes | HF id / local path | |
--reward-model |
text | yes | RM HF id / path | |
--prompts |
text | yes | Source prompts JSONL | |
--output-dir |
text | yes | Output dir | |
--rounds |
integer | 3 |
Number of iterative-DPO rounds | |
--pairs-per-round |
integer | 500 |
Number of (chosen, rejected) pairs per round | |
--plan-only |
flag | off | Print the resolved plan and exit (no training). |
soup license-advisor
Recommend a license-clean base for a deploy target (b2c / defense / embedded) + flag downstream risk (v0.64.0).
Usage: soup license-advisor [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--target, -t |
text | yes | Deploy target: b2c | defense | embedded. | |
--license |
text | Optional: per-license risk check (e.g. apache-2.0, llama-3). | ||
--monthly-active-users, --mau |
integer | 0 |
Expected MAU for the per-license risk check (default 0). |
soup llama
Proxy to llama.cpp binaries (cli / mtmd-cli / gguf-split / server).
Usage: soup llama [<command>] [ARGS]...
Subcommands:
soup llama cli
Forward args to llama.cpp binary llama-cli.
Usage: soup llama cli [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ARGS... |
text, any number | Args forwarded to llama-cli. |
soup llama gguf-split
Forward args to llama.cpp binary llama-gguf-split.
Usage: soup llama gguf-split [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ARGS... |
text, any number | Args forwarded to llama-gguf-split. |
soup llama mtmd-cli
Forward args to llama.cpp binary llama-mtmd-cli.
Usage: soup llama mtmd-cli [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ARGS... |
text, any number | Args forwarded to llama-mtmd-cli. |
soup llama quantize
Forward args to llama.cpp binary llama-quantize.
Usage: soup llama quantize [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ARGS... |
text, any number | Args forwarded to llama-quantize. |
soup llama server
Forward args to llama.cpp binary llama-server.
Usage: soup llama server [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ARGS... |
text, any number | Args forwarded to llama-server. |
soup local-rl
Personal-LLM flywheel daemon (init / status / record / harvest / train) (v0.68.0).
Usage: soup local-rl <command> [ARGS]...
Subcommands:
soup local-rl harvest
Harvest DPO pairs from thumbs into a JSONL file.
Usage: soup local-rl harvest [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--db |
text | local_rl.db |
Path to local-RL SQLite database | |
--output, -o |
text | dpo_pairs.jsonl |
Output JSONL path |
soup local-rl init
Create the local-RL SQLite schema.
Usage: soup local-rl init [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--db |
text | local_rl.db |
Path to local-RL SQLite database |
soup local-rl record
Append a thumbs-up / thumbs-down record.
Usage: soup local-rl record [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--db |
text | local_rl.db |
Path to local-RL SQLite database | |
--prompt |
text | yes | Prompt text | |
--response |
text | yes | Model response | |
--thumb |
text | yes | up / down |
soup local-rl status
Print counters from the local-RL database.
Usage: soup local-rl status [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--db |
text | local_rl.db |
Path to local-RL SQLite database |
soup local-rl train
Run (--once) or schedule the nightly DPO/KTO/ORPO train.
Usage: soup local-rl train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--db |
text | local_rl.db |
Path to local-RL SQLite database | |
--backend |
text | Accepted for compatibility; has no effect on training. Allowed: mlx, ollama | ||
--model |
text | yes | Training base — HF repo id or local path (NOT the Ollama tag) | |
--train-method |
text | dpo |
Allowed: dpo, kto, orpo | |
--once |
flag | off | Run the train now (ad-hoc); skip scheduler install. | |
--min-pairs |
integer | 10 |
Skip training when fewer pairs are harvested. | |
--output, -o |
text | local_rl_adapter |
Adapter output directory | |
--scheduler-dir |
text | local-rl-scheduler |
Directory to render the systemd / launchd scaffold into. | |
--hour |
integer | 3 |
Daily train hour (local time). | |
--minute |
integer | 0 |
Daily train minute. |
soup lock
Shared run lockfile (write / show / check).
Usage: soup lock <command> [ARGS]...
Subcommands:
soup lock check
Refuse with exit 2 if the lock has drifted from current state.
Usage: soup lock check [OPTIONS] [PATH]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | soup.lock |
Path to tracked soup.lock | |
--base-sha |
text | yes | 64-hex current base-model SHA | |
--dataset-sha |
text | yes | 64-hex current dataset SHA | |
--env-hash |
text | yes | 64-hex current env hash | |
--base-model |
text | yes | Current base model id |
soup lock show
Print a tracked lock file.
Usage: soup lock show [PATH]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
PATH |
text | soup.lock |
Path to soup.lock |
soup lock write
Render a soup.lock from the base/dataset/env hashes.
Usage: soup lock write [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base-model |
text | yes | HF model id / path | |
--base-sha |
text | yes | 64-hex base-model SHA | |
--dataset-sha |
text | yes | 64-hex dataset SHA | |
--env-hash |
text | 64-hex env hash. If omitted, auto-derived from --env-lock (soup-env.lock) via soup env lock. |
||
--env-lock |
text | soup-env.lock |
Path to a soup-env.lock to auto-derive --env-hash from (used only when --env-hash is omitted). | |
--output, -o |
text | soup.lock |
Output path |
soup loop
Data flywheel: traces -> preference pairs -> DPO -> gate -> canary deploy -> rollback, all from the CLI (v0.58.0).
Usage: soup loop <command> [ARGS]...
Subcommands:
soup loop canary
Promote new_adapter as the canary with a traffic split.
Usage: soup loop canary [OPTIONS] NEW_ADAPTER
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NEW_ADAPTER |
text | yes | Adapter id/path to canary. | |
--traffic |
text | 5% |
Traffic share, e.g. "5%" or "5". | |
--autoroll-on-regress / --no-autoroll-on-regress |
flag | on | Roll back automatically on MAJOR verdict. |
soup loop init
Create the .soup/loop.yaml control file (one-time setup).
Usage: soup loop init [OPTIONS] SERVED_MODEL
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SERVED_MODEL |
text | yes | Served model id (e.g. registry://abc12). | |
--eval |
text | yes | Eval suite path or registry ref. | |
--baseline |
text | yes | Baseline registry id or file. | |
--monthly-budget |
text | Monthly USD cap (e.g. 50usd, 100). | ||
--max-runs-per-day |
integer | Cap on iteration starts per UTC day. | ||
--pre-wired |
flag | off | Use the pre-wired harvest/train/gate/deploy stages (utils/loop_stages.py) on watch instead of no-op stubs (v0.71.4). | |
--thumbs-db |
text | Local-RL SQLite written by soup serve --record-thumbs; the pre-wired harvest turns its thumbs into preference pairs. Must stay under cwd. SOUP_LOOP_THUMBS_DB overrides it. |
||
--force |
flag | off | Overwrite existing loop.yaml. |
soup loop pause
Pause the watch daemon at the next iteration boundary.
Usage: soup loop pause
Takes no options or arguments.
soup loop replay
Replay a recorded loop iteration manifest.
Usage: soup loop replay [OPTIONS] [ITERATION_ID]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
ITERATION_ID |
text | Iteration id (omit to list all). | ||
--extract |
text | Extract the iteration's .can to this directory for a what-if re-run (v0.71.4). |
soup loop resume
Resume a paused loop (next watch iteration picks up automatically).
Usage: soup loop resume
Takes no options or arguments.
soup loop status
Show counters and current status.
Usage: soup loop status
Takes no options or arguments.
soup loop watch
Run the harvest → train → gate → deploy daemon.
Usage: soup loop watch [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--foreground |
flag | off | Run in foreground (default). | |
--detach |
flag | off | Spawn a background subprocess running --foreground. | |
--max-iterations |
integer | Stop after N iterations (test/demo use). | ||
--poll-interval |
float | 60.0 |
Seconds between iterations [1, 3600]. | |
--pre-wired |
flag | off | Force the pre-wired production stages even if loop.yaml didn't set it. | |
--pack-cans |
flag | off | Pack each iteration as a v0.26 Soup Can + Registry entry (v0.71.4). | |
--tool-auth-token |
text | The soup serve --tool-auth-token value, sent as a Bearer token when the deploy stage activates an adapter. Defaults to $SOUP_TOOL_AUTH_TOKEN, which keeps it out of the shell history. Env: SOUP_TOOL_AUTH_TOKEN. |
soup mcp
Model Context Protocol server - drive Soup from any MCP client (v0.71.28).
Usage: soup mcp <command> [ARGS]...
Subcommands:
soup mcp runs
Recover persisted MCP execution runs.
Usage: soup mcp runs <command> [ARGS]...
Subcommands:
soup mcp runs reconcile
Recover execution capacity left blocked by a crashed MCP server.
Usage: soup mcp runs reconcile [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--expunge-launching |
flag | off | Delete stale launching rows after checking that no recorded PID is alive. | |
--older-than-seconds |
integer, at least 1 | 300 |
Only consider launching rows at least this many seconds old. |
soup mcp serve
Start the MCP server (read-only tools by default).
Usage: soup mcp serve [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--allow-mutating |
flag | off | Enable the plan-only mutating tools (train_start / export). Even when enabled they only render the command that would run - v1 never executes training or export. Off by default: those tools refuse. | |
--allow-execute |
flag | off | Reserve the execution gate for future MCP tools. Implies --allow-mutating, but train_start and export remain plan-only and never execute in this version. | |
--transport |
text | stdio |
stdio (default), sse, or http. stdio speaks JSON-RPC over the process pipes and is what a client that spawns Soup wants. sse and http open a local listener instead, for remote or multi-client setups, and require a Bearer token. | |
--host |
text | Interface to bind for --transport sse/http. Defaults to 127.0.0.1. Binding anything else exposes the server to the network, where the Bearer token is the only gate. | ||
--port |
integer | Port to bind for --transport sse/http. Defaults to 8765. | ||
--auth-token |
text | Override the auto-generated Bearer token (16-128 urlsafe base64 chars) for --transport sse/http. When omitted, a fresh token is generated each startup and printed. |
soup merge
Merge a LoRA adapter with its base model into a full model.
Usage: soup merge [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--adapter, -a |
text | yes | Path to the LoRA adapter directory | |
--base, -b |
text | Base model ID. Auto-detected from adapter_config.json if not set. | ||
--output, -o |
text | ./merged |
Output directory for the merged model | |
--dtype |
text | float16 |
Data type for the merged model: float16, bfloat16, float32 | |
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--save-format |
text | fp16 |
Merged-checkpoint save format. fp16 (default) writes a standard fp16 merge. 4bit / 4bit_forced write a single BNB-4bit-quantized merge without the dequant-merge-requant cycle (v0.53.1 #142). | |
--double-quant / --no-double-quant |
flag | on | BNB double-quantization for the 4bit / 4bit_forced save formats (#321). Default on, matching the training loaders; pass --no-double-quant to disable. Ignored for the fp16 save format. | |
--low-memory |
flag | off | Merge tensor by tensor straight from the base safetensors files instead of loading the whole model into RAM. Needs the fp16 save format and a plain LoRA adapter. | |
--hub |
text | hf |
Source hub for the base model: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK (v0.53.10 #152). |
soup merge-sharded-fsdp-weights
Consolidate FSDP shard files into a single safetensors file.
Usage: soup merge-sharded-fsdp-weights [OPTIONS] SHARD_DIR
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SHARD_DIR |
text | yes | Directory containing pytorch_model_fsdp_*.bin shard files. | |
--output, -o |
text | yes | Destination .safetensors file path (under cwd). | |
--plan-only |
flag | off | Print the consolidation plan and exit without writing anything. |
soup migrate
Import a config from LLaMA-Factory, Axolotl, or Unsloth notebook.
Usage: soup migrate [OPTIONS] CONFIG_FILE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--from |
text | yes | Source tool: llamafactory, axolotl, or unsloth | |
CONFIG_FILE |
text | yes | Path to the source config file (.yaml or .ipynb) | |
--output, -o |
text | soup.yaml |
Output path for generated soup.yaml | |
--dry-run |
flag | off | Print generated config without writing to file | |
--yes, -y |
flag | off | Skip confirmation prompts |
soup monitor
Live GPU monitor: Util / Temp / VRAM / Power per GPU.
Usage: soup monitor [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--refresh, -r |
float | 2.0 |
Refresh interval in seconds (0.25 to 30). | |
--once |
flag | off | Print one snapshot and exit (skip the live panel). |
soup plan
Render a pre-flight training plan (cost / ETA / SHA / VRAM) and write soup.tfstate for soup apply to consult (v0.64.0).
Usage: soup plan [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml. | |
--state |
text | soup.tfstate |
Path to write the state file (default: ./soup.tfstate). |
soup plugins
Discover, list, enable, and disable Soup plugins.
Usage: soup plugins [<command>] [ARGS]...
Subcommands:
soup plugins disable
Disable a registered Soup plugin.
Usage: soup plugins disable NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Plugin name |
soup plugins enable
Enable a registered Soup plugin.
Usage: soup plugins enable NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Plugin name |
soup plugins install
Explain how to install plugins without pretending an install occurred.
Usage: soup plugins install NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Plugin name |
soup plugins list
List all registered plugins.
Usage: soup plugins list
Takes no options or arguments.
soup probe
Activation probes: sleeper-agent defection / honesty / misuse / pairwise interference / SAE feature diff / probe pack (v0.66.0, truth+harm v0.71.8).
Usage: soup probe <command> [ARGS]...
Subcommands:
soup probe harm
Apply the HarmBench-style misuse probe to activations (v0.71.8 #217).
Usage: soup probe harm [OPTIONS] BASE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BASE |
text | yes | Base model id (must have a bundled probe) | |
--evidence |
text | JSON with 'activations': [[...], ...] (2D float matrix) | ||
--weights |
text | Operator-supplied calibrated probe weights (.npz / .npy / .safetensors) | ||
--output, -o |
text | Write probe result JSON to path |
soup probe interference
Pairwise N×N adapter interference matrix (v0.66.0; --measure v0.71.8 #218).
Usage: soup probe interference [OPTIONS] [LOSSES]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
LOSSES |
text | JSON: {adapters: [...], losses: {'a|b': loss_value, ...}} | ||
--measure |
text | Auto-measure losses against this eval-suite JSONL instead of supplying a losses JSON. Requires --base-model + ≥2 --adapter (v0.71.8 #218). | ||
--base-model |
text | Base model id for --measure | ||
--adapter |
text, repeatable | name=path adapter spec for --measure (repeatable) | ||
--device |
text | Device for --measure (cpu / cuda) | ||
--output, -o |
text | Write matrix JSON to path |
soup probe pack
Assemble or list calibrated probe packs for a base (v0.66.0).
Usage: soup probe pack [OPTIONS] [BASE]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BASE |
text | Base model id (e.g. meta-llama/Llama-3-8B) | ||
--list |
flag | off | List all bases that ship a probe pack | |
--output, -o |
text | Write pack manifest JSON to path |
soup probe sae-diff
SAE feature diff between pre- and post-FT activations (v0.66.0).
Usage: soup probe sae-diff [OPTIONS] SAE PRE_ACTS POST_ACTS
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
SAE |
text | yes | Path to SAE safetensors checkpoint (or repo id with --auto-download) | |
PRE_ACTS |
text | yes | JSON: {'activations': [[...]]} | |
POST_ACTS |
text | yes | JSON: {'activations': [[...]]} | |
--top-k |
integer, 1 to 10000 | 20 |
Top-K changed features to report | |
--auto-download |
flag | off | Treat <sae> as an HF Hub repo id and download it into ~/.soup/sae-cache/ first (must be in HF_HUB_ALLOWLIST; v0.71.8 #216). | |
--output, -o |
text | Write diff report JSON to path |
soup probe sleeper
Apply the sleeper-agent defection probe to activations (v0.66.0).
Usage: soup probe sleeper [OPTIONS] BASE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BASE |
text | yes | Base model id (must have a bundled probe) | |
--evidence |
text | JSON with 'activations': [[...], ...] (2D float matrix) | ||
--weights |
text | Operator-supplied calibrated probe weights (.npz / .npy / .safetensors). When set, replaces the synthetic seed fallback and accepts any base id (v0.71.8 #215). | ||
--output, -o |
text | Write probe result JSON to path |
soup probe truth
Apply the TruthfulQA-style honesty probe to activations (v0.71.8 #217).
Usage: soup probe truth [OPTIONS] BASE
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
BASE |
text | yes | Base model id (must have a bundled probe) | |
--evidence |
text | JSON with 'activations': [[...], ...] (2D float matrix) | ||
--weights |
text | Operator-supplied calibrated probe weights (.npz / .npy / .safetensors) | ||
--output, -o |
text | Write probe result JSON to path |
soup profile
Estimate memory, speed, and GPU requirements BEFORE training.
Usage: soup profile [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml config file | |
--gpu, -g |
text | Target GPU for recommendations (e.g., rtx3090, rtx4090, a100, h100). Auto-detects if not set. | ||
--json |
flag | off | Output as JSON for scripting |
soup prune-prompt
Detect + strip a shared system-prompt prefix across training data so the FT model internalises it (v0.63.0).
Usage: soup prune-prompt [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Input JSONL with {prompt, output} rows (e.g. from soup ingest). |
|
--output, -o |
text | yes | Output JSONL with the shared prefix stripped from prompt. |
|
--min-frequency |
float | 0.95 |
Prefix must appear in >= this fraction of rows to be stripped (0.0 - 1.0). | |
--tokenizer |
text | Optional HF tokenizer (model id or local path) for token-aware prefix detection. Default: whitespace-character level. | ||
--slack-url |
text | Optional Slack webhook URL — POSTed on completion. SSRF-validated. | ||
--discord-url |
text | Optional Discord webhook URL — POSTed on completion. SSRF-validated. |
soup push
Push a trained model to HuggingFace Hub (or alternate hub).
Usage: soup push [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to the trained model / LoRA adapter directory | |
--repo, -r |
text | yes | HuggingFace repo ID, e.g. username/my-model | |
--private |
flag | off | Make the HuggingFace repo private | |
--token, -t |
text | Upload token. For --hub hf: [deprecated] use HF_TOKEN or cached login. For other hubs: that hub's token (or MODELSCOPE_API_TOKEN / MODELERS_TOKEN). Env: HF_TOKEN. |
||
--message |
text | Upload model trained with Soup CLI |
Commit message for the upload | |
--collection |
text | Add the pushed repo to an existing HF Collection (slug: 'owner/title-hash') | ||
--hub |
text | hf |
Destination hub: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK and skip the HF-specific Collections / model-card auto-render path (v0.53.10 #152). | |
--card |
text | Registry entry id / prefix / name:tag — render its provenance-rich model card (via soup card) and upload it as README.md, overriding the auto-generated card (v0.71.35). HF hub only. |
soup quantize
Quantize a model - ergonomic alias for soup export --format <to>.
Usage: soup quantize [OPTIONS] MODEL_PATH
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
MODEL_PATH |
text | yes | Source checkpoint (safetensors directory or single .safetensors). | |
--to |
text | gguf |
Target format: gguf | onnx | tensorrt. | |
--bits |
integer | 4 |
Quantization bits (1-16; respected by gguf). | |
--output, -o |
text | Destination directory (default: <model_path>-<to>). |
soup quickstart
Run a complete demo: create sample data, config, and train.
Usage: soup quickstart [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--yes, -y |
flag | off | Skip confirmation prompt | |
--dry-run |
flag | off | Create data and config only, do not train | |
--output, -o |
text | Output directory for data, config, and run artifacts (default: current directory). |
soup ra-dit
RA-DIT two-stage orchestrator: retriever -> generator (v0.71.10).
Usage: soup ra-dit [OPTIONS] [<command>] [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--retriever-config, -r |
text | yes | Stage-1 embedding/contrastive recipe (cwd-contained). | |
--generator-config, -g |
text | yes | Stage-2 RAFT-SFT recipe (cwd-contained). | |
--retriever-model |
text | Manual retriever-model override; skips Registry auto-link. | ||
--timeout |
integer | 21600 |
Per-stage hard timeout in seconds (60..21600). | |
--plan-only |
flag | off | Validate config paths + render the plan; skip training. |
soup recipes
Ready-made configs: list, show, use, search recipes for popular models.
Usage: soup recipes <command> [ARGS]...
Subcommands:
soup recipes list
List all available recipes.
Usage: soup recipes list
Takes no options or arguments.
soup recipes search
Search recipes by keyword, task, or model size.
Usage: soup recipes search [OPTIONS] [QUERY]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
QUERY |
text | Search keyword | ||
--task |
text | Filter by task | ||
--size |
text | Filter by model size |
soup recipes show
Show a recipe config (print YAML to stdout).
Usage: soup recipes show NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Recipe name |
soup recipes use
Copy a recipe to soup.yaml (or custom path).
Usage: soup recipes use [OPTIONS] NAME
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
NAME |
text | yes | Recipe name | |
--output, -o |
text | soup.yaml |
Output path for config file | |
--yes, -y |
flag | off | Skip confirmation prompts |
soup recipes verify
Check that a config can attach a LoRA adapter, without downloading weights.
Usage: soup recipes verify [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | Verify one config instead of the whole catalogue. | ||
--json |
flag | off | Machine-readable rows on stdout. | |
--templates / --no-templates |
flag | on | Also verify src/soup_cli/templates and examples/ (catalogue mode). |
soup registry
Model Registry: push, list, show, diff, search, promote, delete.
Usage: soup registry <command> [ARGS]...
Subcommands:
soup registry delete
Delete a registry entry and its lineage / artifact rows.
Usage: soup registry delete [OPTIONS] REF
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REF |
text | yes | Entry ref to delete | |
--yes, -y |
flag | off | Skip confirmation |
soup registry diff
Side-by-side config + eval delta between two entries.
Usage: soup registry diff LEFT RIGHT
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
LEFT |
text | yes | Left entry ref | |
RIGHT |
text | yes | Right entry ref |
soup registry list
List registry entries.
Usage: soup registry list [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--name, -n |
text | Filter by registry entry name. | ||
--tag, -t |
text | Filter by tag. | ||
--base, -b |
text | Filter by base model. | ||
--task |
text | Filter by training task. | ||
--limit, -l |
integer | 100 |
Maximum number of entries to show. |
soup registry promote
Add a tag to an existing entry (e.g. promote to 'prod').
Usage: soup registry promote [OPTIONS] REF
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REF |
text | yes | Entry ref to promote | |
--tag, -t |
text | yes | Tag to assign |
soup registry push
Register a completed training run as a registry entry.
Usage: soup registry push [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--run-id |
text | yes | Training run id to register | |
--name, -n |
text | yes | Registry entry name | |
--tag, -t |
text | v1 |
Tag (e.g. v1, prod-2024) | |
--notes |
text | "" |
Free-form notes | |
--data |
text | Training data file to record: its SHA-256 is stored as data_hash. This is the file registered as the training input, not proof of what the run trained on. Must be a file under the current directory. | ||
--idempotency-key |
text | Make a retry safe: a later push with the same key and the same run, name, tag and data returns the existing entry instead of a new one; anything different is refused. 1-128 characters from letters, digits, '.', '_', ':', '-'. |
soup registry search
Search registry entries by name/base/task/notes.
Usage: soup registry search [OPTIONS] QUERY
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
QUERY |
text | yes | Keyword to search for | |
--limit, -l |
integer | 50 |
Maximum number of matching entries to show. |
soup registry show
Show full details of a registry entry.
Usage: soup registry show REF
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REF |
text | yes | Entry id / prefix / name:tag |
soup reward
Synthesize a deterministic reward verifier from reference outputs, with a calibration report that refuses degenerate verifiers (v0.71.40).
Usage: soup reward <command> [ARGS]...
Subcommands:
soup reward stress
Adversarially probe a reward verifier for gameability (exit 0 robust / 2 gameable).
Usage: soup reward stress [OPTIONS] REWARD_TARGET
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REWARD_TARGET |
text | yes | Verifier to probe: a .py path, a builtin name, or 'verifiable'. | |
--references |
text | JSONL of gold outputs (enables gold-aware probing). | ||
--field |
text | answer |
Gold field (default: answer). | |
--verifiable-domain |
text | Domain for a 'verifiable' target. | ||
--sentinel |
text | GOLD |
Sentinel-spam token. | |
--threshold |
float | 0.5 |
Reward >= this = accept. | |
--max-gameable |
float | 0.0 |
Max junk accept-rate allowed before the verdict flips to gameable. | |
--attacks |
text | empty,length,repetition,sentinel,wrapped_junk,answer_spray |
Comma list: empty,length,repetition,sentinel,wrapped_junk,answer_spray. | |
--output-report |
text | Also write the stress report as JSON. |
soup reward synth
Synthesize a deterministic reward verifier from reference outputs.
Usage: soup reward synth [OPTIONS] REFERENCES
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
REFERENCES |
text | yes | JSONL of reference (gold) outputs — a gold field or chat messages. | |
-o, --output |
text | Where to write the verifier .py. | ||
--kind |
text | auto |
Verifier family: auto | numeric | json_schema | regex | tool_call. | |
--field |
text | answer |
Gold-output field (default: answer). | |
--tolerance |
float | Numeric match tolerance (numeric kind only). | ||
--min-discrimination |
float | 0.5 |
Refuse to emit unless (accept_rate refs - accept_rate negatives) >= this. | |
--force |
flag | off | Overwrite an existing output. | |
--plan-only |
flag | off | Detect + report the induced spec; write nothing. | |
--output-report |
text | Also write the calibration report as JSON. |
soup rewind
Name the dataset rows behind a loss spike, from the training flight recorder.
Usage: soup rewind [OPTIONS] [RUN_ID]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | Run ID (or prefix). Defaults to the most recent run. | ||
--step |
integer | Detail this step instead of the worst spike. | ||
--top |
integer, at least 1 | 10 |
How many rows to list for the detailed step (at least 1). | |
--json |
text | Write the spikes and the ranked rows to this path as JSON (must stay under the current directory). | ||
--no-preview |
flag | off | Skip the row previews (does not load the dataset). |
soup runs
Experiment tracking: list, show, compare runs.
Usage: soup runs [OPTIONS] [<command>] [ARGS]...
Subcommands:
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--limit, -l |
integer | 20 |
Max runs to show | |
--cwd-only |
flag | off | Only show runs whose output_dir is under the current working directory (default: show all in ~/.soup/experiments.db). |
soup runs clean
Intelligently clean up redundant checkpoint files to reclaim disk space.
Usage: soup runs clean [OPTIONS] [RUN_ID]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | Run ID (or prefix) to clean. Omit if --all is used. | ||
--all |
flag | off | Cleanup all historical runs. | |
--dry-run |
flag | off | Estimate space savings without deleting. | |
--force, -f |
flag | off | Skip confirmation | |
--keep-weights / --no-keep-weights |
flag | on | Keep intermediate model weights and delete only optimizer and scheduler states (default). --no-keep-weights deletes whole non-best checkpoints. |
soup runs compare
Compare two training runs side by side.
Usage: soup runs compare RUN_1 RUN_2
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_1 |
text | yes | First run ID (or prefix) | |
RUN_2 |
text | yes | Second run ID (or prefix) |
soup runs curriculum-curve
v0.48.0 (BETA) — visualise dynamic curriculum bucket weights.
Usage: soup runs curriculum-curve [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run ID (or prefix). | |
--history |
text | Override path to curriculum_history.jsonl (default: under run output_dir). | ||
--width |
integer, 4 to 200 | 10 |
Per-bucket column width. |
soup runs delete
Delete a training run and its metrics.
Usage: soup runs delete [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run ID (or prefix) to delete | |
--force, -f |
flag | off | Skip confirmation |
soup runs replay
Replay a completed run's metrics — re-renders summary + loss curve.
Usage: soup runs replay [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run ID (or prefix) to replay. | |
--plot / --no-plot |
flag | on | Render loss curve |
soup runs show
Show detailed info about a specific run, including loss curve.
Usage: soup runs show [OPTIONS] RUN_ID
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | yes | Run ID (or prefix) to show | |
--plot / --no-plot |
flag | on | Show loss curve |
soup serve
Start a local inference server with OpenAI-compatible API.
Usage: soup serve [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | Path to LoRA adapter directory or full model | |
--base, -b |
text | Base model ID. Auto-detected from adapter_config.json if not set. | ||
--port, -p |
integer | 8000 |
Port to serve on | |
--host |
text | 127.0.0.1 |
Host to bind to. Defaults to loopback (127.0.0.1). Any other host needs --tool-auth-token, which is checked on the tool, thumbs and adapter routes only: the generation routes take no token. | |
--device |
text | Device: cuda, cuda:<index>, mps or cpu. Auto-detected if not set. With the transformers backend, cpu loads on the CPU in float32 (unless --kv-cache-type bf16/f16 picks the dtype); cuda:<index> and mps are pinned to that device; cuda uses device_map=auto, all in float16. | ||
--max-tokens |
integer | 512 |
Default max tokens for generation | |
--backend |
text | transformers |
Inference backend: transformers (default), vllm, sglang, or mii | |
--tensor-parallel, --tp |
integer | 1 |
Number of GPUs for tensor parallelism (vLLM only) | |
--gpu-memory |
float | 0.9 |
Fraction of GPU memory to use (vLLM only, 0.0-1.0) | |
--max-model-len |
integer | Maximum sequence length for the vLLM engine (vLLM only). Lower this when the engine refuses to start because the KV cache does not fit. Default: the model's own maximum. | ||
--speculative-decoding |
text | Draft model for speculative decoding (smaller/faster model ID or path) | ||
--num-speculative-tokens |
integer | 5 |
Number of tokens the draft model generates per step (speculative decoding) | |
--adapters |
text, repeatable | LoRA adapters as name=path pairs (repeatable). E.g. chat=./chat-adapter | ||
--prefix-cache |
flag | off | Enable vLLM prefix caching for shared system prompts (RAG/agent workloads). | |
--auto-spec |
flag | off | Auto-pair draft model for speculative decoding based on target model. | |
--structured-output |
text | off |
Constrain generation: off (default) | json | regex. | |
--json-schema |
text | Path to JSON schema file (used with --structured-output json). | ||
--regex-pattern |
text | Regex pattern (used with --structured-output regex). | ||
--dashboard |
flag | off | Enable live continuous-batching dashboard + /metrics endpoint. | |
--trace |
flag | off | Enable OpenTelemetry request tracing (requires opentelemetry-sdk). | |
--trace-endpoint |
text | OTLP endpoint URL (default: http://localhost:4317). | ||
--auto-quant |
flag | off | Unavailable until Soup can measure each loaded candidate; the command refuses instead of guessing a format. | |
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--trace-log |
text | Append per-request {prompt, response, latency_ms, tokens, ts} to JSONL at this path. Path must stay under cwd. Rotates at 100 MB (one backup retained). Added in v0.40.3 (#33). | ||
--trace-log-cap-mb |
integer | 100 |
Rotation cap in MB for --trace-log (1 - 10000). Default 100. | |
--record-thumbs |
text | Auto-capture thumbs-up/down feedback into a local-rl SQLite at this path via POST /v1/thumbs. Path must stay under cwd. Transformers backend only. v0.71.1 (#230). | ||
--reasoning-parser |
text | Strip reasoning-trace blocks from responses. One of: deepseek-r1 | qwen3 | phi4 | openthinker. v0.53.9 #98. | ||
--steer |
text | Apply a stored activation-steering vector at decode time (CAA / ITI / RepE). Pass the name registered via soup steer train; the transformers backend installs the decode hook after loading the model. |
||
--steer-strength |
float | 1.0 |
Steering strength multiplier (|s| <= 10.0). Ignored when --steer is unset. | |
--hub |
text | hf |
Source hub for the base model: hf (default) / modelscope / modelers. Non-HF hubs require the matching SDK (v0.53.10 #152). | |
--bank |
text | Path to a VeRA / VB-LoRA vector bank (JSON). Multi-tenant LoRA serving at MB-per-user: the per-token delta is v_u ⊙ Px, routed by the X-User-Id request header. Requires --backend transformers. (v0.71.12 #221) | ||
--bank-strength |
float | 1.0 |
Vector-bank delta strength multiplier. Ignored when --bank is unset. | |
--mole |
text | Path to a MoLE training output dir (mole_gate.pt + mole_manifest.json). Serves the base + N frozen task LoRAs with per-token gate blending at decode time. Requires --backend transformers; not combinable with --bank / --steer / --adapters. (v0.71.17 #259) | ||
--kv-cache-type |
text | KV-cache type for decoding: q8_0 (8-bit quantized, needs hqq) / bf16 / f16 (cache dtype) / fp8 (vLLM+Hopper only). Transformers backend only — vLLM / SGLang routing is in the blocked tail. v0.71.14 (#140). | ||
--tool-auth-token |
text | Require 'Authorization: Bearer <token>' on the tool endpoints (/v1/tools/python, /v1/tools/bash, /v1/tools/web_search). Without it, /v1/tools/python and /v1/tools/bash answer 401 on every bind and a non-loopback --host is refused. Env: SOUP_TOOL_AUTH_TOKEN. |
soup ship
SHIP / DON'T SHIP verdict after fine-tuning: task win AND no catastrophic forgetting, fused into one decision (v0.71.25).
Usage: soup ship [OPTIONS] [<command>] [ARGS]...
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base |
text | Base model id/path (the 'before'). | ||
--tuned |
text | Tuned model id/path (a separate 'after' model). | ||
--adapter |
text | LoRA adapter path (tuned = base + this adapter). | ||
--task-eval |
text | JSONL of leg-1 task-win eval tasks. | ||
--task-mode |
text | metric |
Leg-1 mode: metric | judge_score | pairwise (judge win-rate). | |
--noise-floor |
integer | Re-run the BASE model N times (2-10) to measure what this instrument can resolve, print it beside the verdict, and refuse to call any delta smaller than the measured floor significant. Costs N extra base passes. The leg-1 task floor is measured in every --task-mode; in judge_score / pairwise each repeat is N extra JUDGE passes (N x the judge API calls), and the floor is labelled decode + judge. | ||
--judge-model |
text | Judge model URL for --task-mode judge_score or pairwise. | ||
--general-suite |
text | Comma list of leg-2 benchmarks (default: the bundled offline suite — MCQ/arithmetic + tool-call/JSON/safety; non-bundled names route through lm-eval). | ||
--baseline |
text | registry://<id> or stamped JSON of base leg-2 scores (skips base run). Unstamped or scorer_revision-mismatched baselines warn once — re-measure so both sides share one scorer scale (#404). | ||
--forgetting-threshold |
float | 0.05 |
Max allowed leg-2 drop in absolute points before DON'T SHIP. | |
--evidence |
text | JSON of pre-computed scores (offline, no model load). | ||
--output, -o |
text | Write the verdict JSON to this path. | ||
--emit-evidence |
text | Write the scores in the --evidence INPUT schema so this run can be replayed offline (output-is-input). With --config it also STAMPS the config's provenance onto the emitted evidence. | ||
--config |
text | soup.yaml whose eval.ship block supplies defaults (CLI flags win). With --evidence alone it GATES (refuses evidence whose config_sha drifted from this config, or whose numerics family does not match); with --emit-evidence it STAMPS this config's provenance. | ||
--require-provenance |
flag | off | Strict offline drift check: requires --evidence, --config, --adapter and task eval. Compare artifact/task/source bytes, suite version and effective verdict policy. | |
--data-sha256 |
text | Caller-asserted training-data SHA-256 for emitted/strict evidence; not proof of which bytes trained the model. | ||
--push |
text | Post the verdict as a GitHub PR comment (owner/repo#N). Auth via GITHUB_TOKEN / GH_TOKEN; needs the gh CLI. |
||
--device |
text | Device for the live run (cuda / cpu). | ||
--chat-template-kwargs |
text | Live chat-template variables as a JSON object, e.g. {"enable_thinking":false}. |
soup shrink
Depth-prune a model (drop the least-important contiguous layer block by residual angular distance) + optional distill-heal (v0.71.29).
Usage: soup shrink [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model |
text | yes | Model id or local path to shrink. | |
--drop-ratio |
float | Fraction of layers to drop (0-1); e.g. 0.25. | ||
--drop-layers |
integer | Explicit number of contiguous layers to drop. | ||
--calib |
text | yes | Calibration JSONL (prompts) - must stay under cwd. | |
--heal |
text | Heal JSONL (chat rows) - distill the original into the pruned model. | ||
--heal-steps |
integer | 200 |
Approx. optimiser steps for the distill heal. | |
--tolerance |
float | 0.1 |
Perplexity-regression tolerance for the verdict. | |
--output-dir, -o |
text | ./shrunk |
Directory for the shrunk model + report. | |
--device |
text | Device for the importance/ppl passes (cuda / cpu). | ||
--trust-remote-code |
flag | off | Allow custom modeling code (auto_map). | |
--attach-to-registry |
text | Attach the shrink report to a registry entry id. | ||
--plan-only |
flag | off | Print the importance table + chosen block and exit. |
soup spectrum
Spectrum SNR scan for targeted training (v0.71.23).
Usage: soup spectrum <command> [ARGS]...
Subcommands:
soup spectrum scan
Scan a model's layer SNR and emit a training.unfrozen_parameters patch.
Usage: soup spectrum scan [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--model, -m |
text | yes | HF Hub id or local model directory. | |
--top-percent |
float | 50.0 |
Keep the top N% of layers (by SNR) within each module type. | |
--modules |
text | all |
Comma list of module types to scan: e.g. 'mlp,attn' — or 'all'. | |
--output, -o |
text | Write the YAML patch to this path (must stay under cwd). | ||
--no-cache |
flag | off | Skip the ~/.soup/spectrum scan cache. |
soup steer
Activation steering (CAA / ITI / RepE) - inference-time intervention without retraining (v0.62.0).
Usage: soup steer <command> [ARGS]...
Subcommands:
soup steer apply
Resolve + load a stored steering vector and print its metadata.
Usage: soup steer apply [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--name, -n |
text | yes | Identifier of a stored steering vector. | |
--strength, -s |
float | 1.0 |
Steering strength multiplier (|s| <= 10.0). |
soup steer list
List locally-stored steering vectors (Registry artifact kind steering_vector).
Usage: soup steer list
Takes no options or arguments.
soup steer train
Train a steering vector from contrastive prompt pairs.
Usage: soup steer train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--base, -b |
text | yes | Base model HF id or local path. | |
--method, -m |
text | caa |
Steering method: caa / iti / repe. | |
--name, -n |
text | yes | Identifier for the trained vector (e.g. 'safety-v1'). | |
--pairs, -p |
text | yes | Path to JSONL of contrastive (positive, negative) prompt pairs. | |
--layer, -l |
integer | Decoder layer index to extract the residual-stream vector from (default: the middle layer). | ||
--output, -o |
text | Directory to write the steering vector to (default: ./steering/<name>). Cwd-contained. | ||
--device |
text | torch device (cpu / cuda). Defaults to CUDA when available. | ||
--top-k, -k |
integer | 8 |
ITI only: number of attention heads to intervene on (1-256). | |
--plan-only |
flag | off | Validate inputs + print the resolved plan; skip the live fit. | |
--registry-id |
text | Registry entry id to attach the trained vector file to, as a steering_vector artifact. Exits 1 when the attach fails. |
soup sweep
Run hyperparameter sweep: grid or random search over training parameters.
Usage: soup sweep [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to base soup.yaml config file | |
--param, -p |
text, repeatable | Parameter to sweep: key=val1,val2,val3 (e.g., lr=1e-5,2e-5,5e-5) | ||
--sweep-config |
text | Path to a standalone sweep YAML (strategy/n_runs/seed/params) instead of --param; its strategy, n_runs and seed take precedence over --strategy/--max-runs | ||
--strategy, -s |
text | grid |
Search strategy: grid, random | |
--max-runs |
integer | Max number of runs (useful for random strategy) | ||
--name, -n |
text | Sweep experiment name prefix | ||
--dry-run |
flag | off | Show planned runs without executing | |
--early-stop |
float | Stop early if run's loss exceeds best loss by this factor (e.g. 1.5 = 50% worse) | ||
--yes, -y |
flag | off | Skip confirmation prompt |
soup tokenizer
Tokenizer tools: train a BPE tokenizer from JSONL (v0.53.9).
Usage: soup tokenizer <command> [ARGS]...
Subcommands:
soup tokenizer train
Train a BPE tokenizer from a local corpus.
Usage: soup tokenizer train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--input, -i |
text | yes | Path to JSONL or .txt corpus (must stay under cwd). | |
--vocab-size, -v |
integer | 32000 |
Target vocabulary size (256-200000). | |
--output, -o |
text | tokenizer_out |
Output directory for tokenizer.json + vocab.json (under cwd). | |
--min-frequency |
integer | 2 |
Minimum pair frequency (>=1). | |
--special-token |
text, repeatable | Add special token (repeatable). Defaults: <pad>, <s>, </s>, <unk>. |
soup train
Start training from a soup.yaml config.
Usage: soup train [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--config, -c |
text | soup.yaml |
Path to soup.yaml config file | |
--name, -n |
text | Experiment name (auto-generated if not set) | ||
--dry-run |
flag | off | Validate config and data without training | |
--resume, -r |
text | Resume from checkpoint: path to checkpoint dir ('auto' for latest); on the MLX backend, a path to a .safetensors adapter file instead. Not supported when training.relora_steps is configured | ||
--wandb |
flag | off | Enable Weights & Biases logging | |
--tensorboard |
flag | off | Enable TensorBoard logging (logs to output_dir/runs/) | |
--tracker |
text | Experiment tracker: mlflow / swanlab / trackio (v0.43.0). Mutually exclusive with --wandb / --tensorboard. | ||
--deepspeed |
text | Enable DeepSpeed: zero2, zero3, zero2_offload, zero3_offload (stage 3 + CPU parameter offload), zero++ (ZeRO++), or path to config JSON | ||
--fsdp |
text | Enable FSDP2: full_shard, shard_grad, or full_offload | ||
--gpus |
text | GPUs per node for distributed training ('auto' or integer) | ||
--nodes |
integer | 1 |
Number of machines; each must use the same --gpus count | |
--node-rank |
integer | This machine's rank, from 0 to nodes minus 1 (default 0) | ||
--master-addr |
text | Rank 0 hostname or IP reachable by all nodes; required with --nodes > 1 | ||
--master-port |
integer | Shared coordinator port for multiple nodes (default 29500) | ||
--no-reexec |
flag | off | Print the distributed launch command instead of launching it | |
--gate |
text | Enable eval-gated training with a suite file (shortcut for training.eval_gate.enabled=true + suite=<path>) | ||
--push-as |
text | Auto-push each save_steps checkpoint to HF Hub as 'checkpoint-<step>' branch of the given repo (e.g. user/my-model) | ||
--hf-resume |
flag | off | Download the latest checkpoint branch from the --push-as repo and resume from it. Requires --push-as; not supported when training.relora_steps is configured. | |
--find-lr |
flag | off | LR range finder (v0.32.0): run a short geometric LR sweep, write a JSON report with the recommended LR, then exit without training. | |
--find-lr-start |
float | 1e-07 |
LR range finder: starting LR (default 1e-7) | |
--find-lr-end |
float | 0.1 |
LR range finder: ending LR (default 1e-1) | |
--find-lr-steps |
integer | 100 |
LR range finder: number of sweep steps (default 100) | |
--find-lr-output |
text | lr_finder.json |
LR range finder: JSON report path (default ./lr_finder.json) | |
--yes, -y |
flag | off | Skip the start-training confirmation. Required when stdin is not a TTY (scripts, CI, pipes). | |
--trust-remote-code |
flag | off | Allow loading models that ship custom Python via auto_map. Default deny (v0.36.0). Only enable if you trust the source. | |
--echo-trap-tokenizer-aware |
flag | off | Use tokenizer-id n-grams for echo-trap scoring. Requires training.echo_trap_enabled=true on grpo/ppo. | |
--reward-hack-detector |
text | Reward-hacking detector for GRPO/PPO: info_rm | rm_ensemble. Overrides training.reward_hack_detector. (v0.71.26) | ||
--reward-hack-halt |
flag | off | Auto-halt training on a HACK verdict. Requires --reward-hack-detector (or training.reward_hack_detector). (v0.71.26) | |
--replay |
text | Old dataset to interleave as continual-learning rehearsal, so training on the new task does not erase the previous one. sft/pretrain only; incompatible with packing/multipack. Overrides data.replay. (v0.71.36) | ||
--replay-ratio |
float | Fraction of the FINAL mixed train set that is replay rows (default 0.1). Overrides data.replay_ratio. (v0.71.36) | ||
--replay-seed |
integer | Seed for the replay sample + interleave. Overrides data.replay_seed. (v0.71.36) | ||
--reward-hack-mitigation |
text | Closed-loop reward-hacking mitigation mode: off | log_only | kl_control | pid_lagrangian. Requires training.reward_hack_detector on grpo/ppo. Overrides training.reward_hack_mitigation. (v0.71.26) | ||
--minillm-on-policy |
flag | off | Use the TRUE on-policy MiniLLM teacher-mixed rollout (v0.71.18 #257) instead of the offline distribution blend. Requires training.minillm_enabled=true on task='distill'. | |
--profile |
flag | off | Record a torch.profiler trace (Chrome trace JSON) during early training steps. Output: <output>/profiles/<run_id>.trace.json | |
--allow-oom-attempt |
flag | off | Bypass the analytical hardware-fit VRAM gate and launch even when the run is predicted to run out of GPU memory (opt-out). | |
--diagnose-gate |
text | After training, run soup diagnose against the supplied evidence JSON (or scratch evidence). Refuses to mark the run successful if any failure mode returns MAJOR (exit 2) or NOT_RUN (exit 3). |
||
--annex-xi |
text | After training, render an EU AI Act Annex XI/XII auto-doc to the given output path (cwd-contained). Markdown body now; PDF in v0.59.1. | ||
--repro-receipt |
text | After training, write an SR 11-7-style reproducibility receipt (seeds + kernel versions + GPU + OS) to the given path. v0.59.0. | ||
--capture-activations |
text | After training, capture residual-stream activations at the named decoder layer (e.g. model.layers.5) on --capture-prompts and write them to <output>/activations/activations.json for soup probe sae-diff / sleeper. v0.71.8 #219. | ||
--capture-prompts |
text | JSONL (or .txt) of prompts to run for --capture-activations (one prompt per line; 'prompt'/'text' field or raw text). | ||
--track-energy |
flag | off | Measure the training window's energy + CO2 via codecarbon (offline; requires pip install soup-cli[carbon]). Feeds the kWh / CO2 into --annex-xi. v0.71.3. |
|
--energy-country |
text | USA |
ISO 3166-1 alpha-3 country code for the CO2 grid-intensity estimate used by --track-energy (default USA). | |
--energy-out |
text | Write the --track-energy measurement to this JSON file (cwd-contained) so soup bom emit --energy <path> can consume it. v0.71.15. |
||
--cloud |
text | Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: modal, lambda, runpod. Renders a cloud app stub from the config (plan-only); use --cloud-submit to submit live. | ||
--gpu |
text | a100 |
Cloud GPU type for --cloud (t4 / l4 / a10 / a10g / a100 / a100-80gb / l40s / h100 / a6000). Default a100. Provider-specific allowlists apply. | |
--cloud-submit |
flag | off | With --cloud, submit the rendered run live via the cloud's SDK or API (gated on respective provider token/API key). Default is plan-only (render + print the command). Lambda requires a registered SSH key. |
soup tui
Full-screen terminal dashboard: list runs, inspect metrics, tail logs.
Usage: soup tui [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--refresh |
float | 1.0 |
Refresh interval in seconds (0.25 - 10.0). | |
--limit |
integer | 50 |
Maximum runs to load into the dashboard. |
soup tunability
Probe-train 6-10 small bases on a held-out slice + report Pareto frontier of (eval delta, train cost, license) (v0.64.0).
Usage: soup tunability [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--dataset, -d |
text | Path to held-out JSONL slice used for the probe. | ||
--candidates |
text | Comma-separated subset of default candidates by name. | ||
--probe-steps |
integer | 100 |
Steps per candidate probe (10-10000). | |
--holdout-size |
integer | 64 |
Number of holdout rows used for delta scoring (10-100000). | |
--output, -o |
text | Optional path for the JSON report. | ||
--plan-only |
flag | off | Print planned candidates + cost estimate, exit without probing. | |
--list |
flag | off | List built-in candidate catalogue + exit. | |
--live |
flag | off | Run a LIVE LoRA probe per candidate (loads each repo + trains --probe-steps on a tiny held-out slice). Without it, a deterministic offline heuristic is used. | |
--device |
text | Device for the live probe (cuda / cpu). Auto-detected when omitted. |
soup ui
Launch the Soup Web UI.
Usage: soup ui [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--port, -p |
integer | 7860 |
Port to serve on | |
--host |
text | 127.0.0.1 |
Host to bind to | |
--no-browser |
flag | off | Don't open browser automatically | |
--show-token |
flag | off | Print the auth token and exit (for scripting) | |
--public |
flag | off | Bind to 0.0.0.0 so the UI is reachable from a phone on the same LAN. Prints a phone-scannable URL + ASCII QR at startup. Loopback HTTP is auto-upgraded to require a Bearer token in the query string. v0.53.9 #95. | |
--auth-token |
text | Override the auto-generated Bearer token (16-128 urlsafe base64 chars). Useful for stable phone bookmarks. When omitted, a fresh token is generated each startup. |
soup version
Show Soup CLI version.
Usage: soup version [OPTIONS]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
--full, -f |
flag | off | Show system info and extras | |
--json |
flag | off | Output in JSON format |
soup why
Explain training anomalies for a run in plain English.
Usage: soup why [RUN_ID]
| Name | Type | Default | Required | Help |
|---|---|---|---|---|
RUN_ID |
text | Run ID (or prefix). Defaults to the most recent run. |