Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

50.7 KB
Raw Blame

Adapters, Registry & Governance

← Back to the Soup README

Adapter lifecycle/management, the model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, activation steering, and the supply-chain / governance surfaces (scan, sign, BOM, attest, audit, airgap).

Contents:


Adapter Lifecycle (soup adapters {merge,pr,bisect}, soup lock)

v0.57 shipped adapters diff / merge / blame / branch. v0.67 finishes the lifecycle: evolutionary merge driven by your eval, GitHub-shaped PRs for adapter review, a shared soup.lock for team reproducibility, and binary-search bisect over training history.

# 1. Evolutionary merge: search the simplex of merge weights via CMA-ES.
#    v0.71.4 makes this a LIVE loop — each candidate is merged, scored against
#    --eval, and the best blend is written to --output.
soup adapters merge \
  adapter-finance/ adapter-medical/ adapter-legal/ \
  --strategy cmaes \
  --eval evals/domain_mix.yaml \
  --budget 1h \
  --population 8 \
  --max-generations 20 \
  --output merged/

# 1b. One-shot merge with a live canary verdict (v0.71.4). Scores the merged
#     adapter vs the first input → OK / MINOR / MAJOR; --strict-verdict exits 2
#     on a MAJOR regression. A pre-scored canary suite needs no model load.
soup adapters merge adapter-a/ adapter-b/ -o merged/ \
  --canary evals/canary.json --strict-verdict

# 2. Render the merge as a GitHub PR for review (eval deltas + sample diffs).
soup adapters pr "merge: 3-domain blend" \
  --base-sha $(git rev-parse HEAD) \
  --adapter merged/ \
  --eval evals/deltas.json \
  --samples evals/samples.json \
  --dataset-diff data/diff.txt \
  --format markdown -o pr.md

# 2b. Post it straight to a GitHub PR comment (v0.71.4). Auth via GITHUB_TOKEN
#     / GH_TOKEN; uses `gh api` under the hood (no secret leaks to the child).
soup adapters pr "merge: 3-domain blend" \
  --base-sha $(git rev-parse HEAD) --adapter merged/ \
  --eval evals/deltas.json --push your-org/your-repo#42

# 3. Lock a reproducible run state. Closure = sha(base + dataset + env).
soup env lock                                     # v0.64 — capture env hash
soup lock write \
  --base-model meta-llama/Llama-3.1-8B \
  --base-sha $BASE_SHA \
  --dataset-sha $DATA_SHA \
  --env-hash $(jq -r .closure soup-env.lock) \
  -o soup.lock
# Teammates re-check the lock; exit 2 on drift.
soup lock check soup.lock \
  --base-model meta-llama/Llama-3.1-8B \
  --base-sha $BASE_SHA --dataset-sha $DATA_SHA --env-hash $ENV_HASH

# 4. Bisect a training history to find the step that broke an eval.
soup adapters bisect \
  ckpt-step-100 ckpt-step-200 ckpt-step-400 ckpt-step-800 \
  --eval-command "soup eval custom --model {ckpt} --tasks eval.jsonl" \
  -o bisect.json
# Exits 3 on BROKEN_AT — pipe into `soup adapters blame` for attribution.

CMA-ES is pure-Python (no cma dependency); the eval is operator-supplied via a closure so any scoring code works. PR rendering escapes Markdown table cells, so crafted metric names cannot inject table rows or links. The lockfile composes with v0.64 soup env lock — drift in any of {base_model, base_model_sha, dataset_sha, env_hash, closure_sha} exits 3 (soup_version and created_at are advisory-only). Bisect uses shlex.split + shlex.quote(ckpt) in argv-list mode (no shell=True), so checkpoint ids cannot inject shell metacharacters.

VeRA / VB-LoRA bank storage (soup_cli.utils.vector_bank) supports live multi-tenant serving, and task='moe_lora_routing' trains the live sparse per-token MoLE gating kernel.

Data Flywheel (soup loop)

The full production traces → preference pairs → Eval-Gated DPO → canary deploy → rollback loop, driven from a single CLI. Connects v0.26 Trace-to-Preference + Eval-Gated Training + Registry lineage + Quant-Lobotomy verdicts + Soup Cans + v0.25 Autopilot + v0.54 Advise + v0.55 Eval Design + v0.56 Diagnose.

# One-time setup
soup loop init registry://abc12 --eval evals/lock.json --baseline registry://prod \
    --monthly-budget 50usd --max-runs-per-day 3

# Inspect counters + status
soup loop status

# Run the daemon (foreground)
soup loop watch --poll-interval 300

# Pre-wired production stages (v0.71.4): real traces → DPO → eval-gate → canary,
# instead of the v0.58 no-op stubs. Opt in once via `loop init --pre-wired` or
# per-run via --pre-wired. Pack each iteration as a shareable Soup Can with
# Registry lineage via --pack-cans.
soup loop init registry://abc12 --eval evals/lock.json --baseline registry://prod --pre-wired
soup loop watch --pre-wired --pack-cans --poll-interval 300

# Learn from Soup's own serving (#1440): the pre-wired harvest also reads the
# thumbs `soup serve --record-thumbs` captures through POST /v1/thumbs, pairing
# the latest thumbs-up and thumbs-down answer per prompt. The database must
# stay under cwd and exist as a thumbs database: start the serve first, or
# `soup loop watch` refuses, naming the field.
soup serve --model ./base --record-thumbs ./rl.db
soup loop init registry://abc12 --eval evals/lock.json --baseline registry://prod \
    --pre-wired --thumbs-db ./rl.db
# or per run, overriding loop.yaml:
SOUP_LOOP_THUMBS_DB=./rl.db soup loop watch --pre-wired

# Unpack a recorded iteration's Soup Can for what-if analysis (v0.71.4)
soup loop replay iter-20260515T120000-abcdef01 --extract ./iter-dump

# Background subprocess (writes PID, no shell)
soup loop watch --detach

# The deploy stage activates the adapter on SOUP_LOOP_SERVE_ENDPOINT. A server
# started with --tool-auth-token requires that token on the adapter routes, so
# give the loop the same one: --tool-auth-token, or SOUP_TOOL_AUTH_TOKEN to keep
# it out of the shell history. A --detach child gets it through its environment
# (so does the train stage's `soup train` child, which ignores it). soup serve has
# no TLS and the loop refuses plain http off-loopback, so run the loop on the
# serving box against loopback, or put a TLS-terminating proxy in front.
soup serve --model base-model --adapters chat=./chat --host 0.0.0.0 --tool-auth-token "$TOKEN" &
SOUP_LOOP_SERVE_ENDPOINT=http://127.0.0.1:8000 SOUP_TOOL_AUTH_TOKEN="$TOKEN" soup loop watch --detach

# Start serve with the candidate loaded, then promote it at 5% traffic.
soup serve --model ./base --adapters candidate=./candidate
soup loop canary candidate --traffic 5% --autoroll-on-regress

# Pause/resume the daemon between iterations (atomic state flip)
soup loop pause
soup loop resume

# Replay any recorded iteration
soup loop replay iter-20260515T120000-abcdef01

State lives in .soup/loop.yaml (atomic write, cwd-contained, symlink-rejected). Per-iteration manifests under .soup-loops/<iter-id>/iteration.json are laid out so a v0.26 Soup Can can wrap them directly.

Canary routing is available on the transformers backend when the candidate name was loaded with soup serve --adapters. Send a stable key as the conversation_id field or X-Conversation-Id header on POST /v1/chat/completions; requests without a key keep the normal active-adapter behavior. The router hashes the key, so every request in one conversation stays in the same stable or canary bucket. An explicit adapter field still takes precedence.

Completed generations and generation errors are accumulated separately for the two buckets in .soup/canary-stats.json. soup loop watch evaluates those observations after at least 30 canary samples. If the canary success rate trails the stable rate by more than five percentage points, the verdict is MAJOR; with --autoroll-on-regress, the watcher atomically clears the canary and its traffic share from loop.yaml. /v1/adapters/activate/<name> remains an explicit 100% cutover for requests without a conversation key and does not change the canary policy. Keyed requests still follow the canary split; their stable bucket uses the served base model, even when another adapter was activated manually. Start soup serve from the directory containing .soup/, because the loop state and canary statistics paths are resolved from the working directory. Clients choose their own stable key, so a client can deliberately select a key that hashes into the canary bucket.

The serving process caches the parsed policy using the state file's (st_mtime_ns, st_size, st_ino) identity. Atomic replacement invalidates it, so a rollback written by soup loop watch affects the next keyed request. State-path failures are warned about once per failure episode rather than on every request; a successful policy read allows a later failure to warn again.

Outcome counters are buffered per serving app and persisted after 64 completed outcomes or two seconds from the first pending outcome, whichever comes first. Orderly shutdown flushes the remaining counters. With a healthy writable filesystem, an abrupt crash can lose the unflushed window (at most one 64-outcome batch, including a flush in progress, normally at most two seconds); scheduling delays or disk errors can delay persistence. A failed write retains the pending counters for retry. While that batch is full, additional outcomes are refused until persistence recovers; a disk outage can therefore lose more outcomes than the normal one-batch crash window. The on-disk canary-stats.json and watch verdicts lag traffic by the buffered window (normally up to 64 outcomes or two seconds). Rollout IDs prevent promotions from sharing samples. Pending samples and late completions from superseded promotions are discarded rather than replacing the new rollout's file. The existing stats lock protects threads in one process, not multiple server processes writing the same file.

A canary that kills the serving process never reaches outcome recording: recording happens only after generation returns or raises. Such crashes remain invisible to the canary verdict unless external restart monitoring detects them.

Knowledge Editing (soup edit set, ROME / MEMIT / AlphaEdit)

Surgical factual patches WITHOUT a full fine-tuning loop. Hospital data team correcting a misattributed drug interaction, lab fixing a wrong historical date, security team responding to a hallucinated CVE — all one CLI invocation.

# Live edit: optimises a residual + applies a rank-1 weight update, then saves.
soup edit set \
  --base HuggingFaceTB/SmolLM2-135M \
  --method rome \
  --subject "The capital of France is" \
  --target "Lyon" \
  --output ./edited --device cuda

# Covariance-preconditioned ROME (genuine C^-1 k* update) from a stats corpus.
soup edit set --base ./model --method rome \
  --subject "..." --target "..." \
  --cov-corpus ./stats.jsonl --output ./edited

# Plan-only mode validates the request + prints the resolved EditPlan + exits 0.
soup edit set --base ./model --method rome --subject "..." --target "..." --plan-only

# Diff what the model says before vs after the edit (live generation).
soup edit diff before after \
  --before-model ./base --after-model ./edited \
  --probes probes.jsonl --output diff.json

The kernels run a real rank-1 weight update at an MLP down-projection. They support both the Llama-family (model.model.layers[L].mlp.down_proj, an nn.Linear with [out, in] weights) and the GPT-2-family (model.transformer.h[L].mlp.c_proj, a Conv1D with the transposed [in, out] layout) — the update is transpose-aware so the post-condition down(k*) += delta holds exactly for either layout (validated on real gpt2: P(target) 0.005 → 0.9996, and on SmolLM2-135M). PEFT-wrapped models are unwrapped automatically. ROME edits a single layer; MEMIT distributes the residual across a layer band; AlphaEdit projects the update orthogonal to the down-proj's top singular direction.

By default ROME uses the covariance-free C = I form. Pass --cov-corpus <jsonl|txt> to estimate the key covariance C = E[k kᵀ] + λI over a stats corpus and apply the genuine ROME closed form u = C⁻¹ k* — this spreads the rank-1 update mass per the closed form and reduces collateral interference with other facts (the exact post-condition is preserved either way). The corpus loader is cwd-contained, symlink-rejected, and size/line-capped; --cov-corpus is rejected for any method other than rome.

The sequential edit governor is persisted (SQLite, cross-process-locked, SOUP_EDIT_GOVERNOR_DB override) so the per-base-model edit count + norm-blowup verdict survive across separate soup edit set runs. The count increment is atomic — two concurrent soup edit set runs on the same base are merged under the cross-process lock (no lost increment). soup edit set consults the governor automatically: it refuses BEFORE the model load past the per-base cap or after a BLOWUP verdict, and records the measured ||ΔW||_F after each edit. Pass --no-governor to opt out. --registry-id <id> (needs --output) records the saved result on that Registry entry: the weight file(s) save_pretrained wrote (model.safetensors, its numbered shards, or pytorch_model*.bin) as edited_model artifacts, or grace_codebook.json as a grace_codebook artifact. The Registry stores one hashed file per artifact, never a directory. A requested attach is part of the command's success: soup edit set exits 1 when it fails (unknown or ambiguous entry, unreadable registry, no weight file), with the edited model left on disk, and exits 2 before the model is loaded when the flag is given without --output.

Activation Steering (soup steer)

Sometimes you don't want to retrain — you want to push the model along a learned direction at decode time. Soup ships three control-vector backends:

  • CAA (Contrastive Activation Addition) — add a contrastive vector to the residual stream.
  • ITI (Inference-Time Intervention) — shift specific attention heads along a learned direction.
  • RepE (Representation Engineering) — PCA-based direction in the residual stream.
# Train a steering vector from contrastive (positive, negative) prompt pairs
soup steer train --base meta-llama/Llama-3.1-8B-Instruct \
                 --method caa --name safety-v1 \
                 --pairs ./data/pairs.jsonl

# Apply at decode time via soup serve
soup serve --model ./adapter --steer safety-v1 --steer-strength 1.5

# List locally-stored steering vectors
soup steer list

Steering names are validated against a strict regex (^[A-Za-z0-9][A-Za-z0-9._\-]{0,127}$ — no path separators, no shell metacharacters); strength is bounded |s| <= 10.0. With --registry-id <id> the trained vector file (steering_vector.safetensors) is recorded on that Registry entry as a steering_vector artifact, so lineage is preserved and soup steer list shows it; <id> is an entry id, a unique id prefix, name:tag or registry://<id>. A vector saved outside ./steering/<name> with --output is then found by name through an entry called <name>. A requested attach is part of the command's success: soup steer train exits 1 when it fails, with the vector left on disk. Without the flag nothing is written to the Registry.

As of v0.71.10 the fit and the decode hook are live (validated on SmolLM2-135M): soup steer train captures residual-stream activations (CAA / RepE) or per-head o_proj-input activations (ITI) on the contrastive pairs, computes the control vector, and persists steering_vector.safetensors + steering_config.json. soup serve --steer <name> installs a forward hook on the loaded model that adds strength × vector at decode time (transformers backend; --steer is rejected with a clear error on vLLM/SGLang). RepE / ITI need at least two contrastive pairs; CAA works from one.

GRACE Codebook — Lifelong Knowledge Edits

Vanilla ROME / MEMIT degrade after dozens of sequential edits — the model's norms blow up. GRACE (Hartvigsen et al., 2023) stores each edit in a discrete latent codebook so thousands of sequential patches survive:

soup edit set --base ./model --method grace \
              --subject "The CEO of Acme is" --target "Jane Doe"
# Planned for soup.yaml when training with GRACE-aware lookups (staged; refused as of v0.77 — #808):
# training:
#   grace_codebook: true
#   grace_codebook_size: 1024    # codebook entries (max 100k)
#   grace_codebook_dim: 768      # residual-stream width

grace joins the existing rome / memit / alphaedit allowlist on soup edit set; the sequential edit governor still gates the call when the per-base-model edit count or norm-blowup verdict trips. GRACE is live for editing: soup edit set --method grace --output ./ckpt captures the residual key at the subject's last token, optimises a replacement value, and appends a (key, value) triple to a grace_codebook.json sidecar (atomic, cwd-contained). At inference the codebook is applied via a forward hook that substitutes the residual whenever it falls within an epsilon ball of a stored key — so the base weights are never modified and thousands of edits survive without norm blowup. Training-time lookup integration remains staged.

Model Registry & Lineage

Every fine-tune you ship should be reproducible. Soup's local registry (~/.soup/registry.db) tracks each entry by a content hash of its config + data + base model, plus lineage pointers to parent entries.

# Register a completed run
soup registry push --run-id run_202611_abc123 --name llama31-chat --tag v1

# Also record the training data file's SHA-256 as data_hash
soup registry push --run-id run_202611_abc123 --name llama31-chat --tag v1 --data data/train.jsonl

# Safe to retry: the same key and the same registration return the existing entry
soup registry push --run-id run_202611_abc123 --name llama31-chat --tag v1 --idempotency-key job-42

# List entries (filter by name, tag, base model, task)
soup registry list
soup registry list --name llama31-chat --tag prod

# Show full details: config, eval baseline, artifacts, ancestors
soup registry show llama31-chat-v1

# Side-by-side config diff + eval delta between two entries
soup registry diff llama31-chat-v1 llama31-chat-v2

# Full-text search across name / base model / task / notes
soup registry search "medical reasoning"

# Promote an entry (add a tag, e.g. "prod")
soup registry promote llama31-chat-v1 --tag prod

# Delete (cascades to artifacts + lineage links)
soup registry delete llama31-chat-v1 --yes

Training data: push --data <file> stores the file's SHA-256 as the entry's data_hash, shown by soup registry show. It records the file registered as the training input, not proof of what the run trained on. The file must be under the current directory. Without --data, register_from_run copies a stored historical training_data_sha256 without reopening today's path; legacy runs remain unknown. When both are present, the explicit file takes precedence for data_hash, while the separate historical training_data_sha256 is retained. entry_hash binds the chosen data digest and any recorded base revision/source. Training-time hash capture itself remains #1764 part 2; reserving nullable storage does not claim the transformed examples consumed by training.

Base identity: Registration copies the run's nullable base revision and its source; soup registry show exposes both. hub_commit identifies the commit passed to supported base loaders, while cache_path only describes the local directory name and does not verify local contents. The original human-readable base ID and config hash retain their existing meaning. Registrations with no known revision/hash retain their legacy entry fingerprint. Retries also compare the recorded revision/source and historical data hash.

Retries: push --idempotency-key <key> (or RegistryStore.push(idempotency_key=...)) makes a registration safe to repeat after a crash or a timeout. A later push with the same key writes nothing and returns the existing entry, provided it repeats that entry's name, base model, task, run, config and data hash and its tag is one of the entry's tags; anything else under the same key is refused with an error naming the entry. Notes are not compared. Two writers pushing the same key at once end up with one entry. A key is 1 to 128 characters from letters, digits, ., _, : and -; pushes without one behave as before. RegistryStore.get_by_idempotency_key(key) finds the entry. Deleting the entry frees its key.

Lineage DAG — every entry can point to a parent (its ancestor run). Walk the DAG for any name with:

soup history llama31-chat

Refs resolve flexibly — you can use a registry ID, a unique ID prefix, or name:tag. A bare name is not a ref; soup history <name> lists every entry with that name. Pushing the same name:tag twice keeps both entries. A name:tag that matches more than one entry raises an error naming the candidates, and an ambiguous prefix raises one too, rather than silently picking an entry; use the full ID then. Registry files are stored with 600 perms on POSIX; override the path with SOUP_REGISTRY_DB_PATH.

Adapter Management (git for LoRA)

soup adapters is the git-for-LoRA surface: weight-aware diff, four merge strategies, leave-one-out blame, and SHA-256 branch snapshots. All commands operate on adapter_model.safetensors directories (peft-compatible).

# Per-layer ΔW Frobenius diff + effective-rank drift + top-K changed projections
soup adapters diff ./run-v17 ./run-v18

# Machine-readable JSON for CI
soup adapters diff ./run-v17 ./run-v18 --format json --output diff.json

# Weighted merge with linear / ties / dare / svd strategies
soup adapters merge ./run-v17 ./run-v18 ./run-v19 -o ./merged --strategy ties \
  --weights 0.5,0.3,0.2 --density 0.2

# DARE merge (deterministic via --seed)
soup adapters merge ./run-v17 ./run-v18 -o ./merged --strategy dare \
  --density 0.5 --seed 42

# Task-vector arithmetic (v0.71.34) — add / scale / NEGATE trained behaviours
# into one adapter (arXiv:2212.04089). Names map to dirs via --adapter name=path.
soup adapters arithmetic "coder + 0.5*math - toxic" \
  --adapter coder=./coder-lora --adapter math=./math-lora \
  --adapter toxic=./toxic-lora -o ./blended

# Leave-one-out ablation plan against a 4-hour wall-clock budget
soup adapters blame ./run-v18 --dataset train.jsonl --layer q_proj.7 \
  --budget 4h --shards 10 --plan-only

# Snapshot a training environment as a comparable branch
soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
  --dataset train.jsonl

# Link a branch into the Registry lineage DAG (v0.71.4) — shows as a
# `branches` node under the entry in `soup history`.
soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
  --attach-to-registry reg_20260601_abc123

# Or derive a fresh snapshot's config + base straight from a Registry entry (v0.71.4)
soup adapters branch v18-from-reg --from-registry reg_20260601_abc123

# Restore the snapshot's config (refuses if source SHA drifted)
soup adapters checkout v18 --output soup.yaml

# List all snapshotted branches
soup adapters branches

Four merge strategies (pure numpy, no torch import at module level):

Strategy Math Use case
linear Weighted average per layer Baseline; tasks share a basis
ties Trim by density → elect majority sign → disjoint average Conflicting task adapters (Yadav et al. 2023)
dare Random drop with density + rescale 1/density, then average Sparse-merge; reduces parameter interference (Yu et al. 2024)
svd Linear-merge → low-rank reconstruction via SVD (--rank) Constrain effective rank of the merged delta

Defaults & safety:

  • Output paths are containment-checked under cwd and reject pre-placed symlinks (TOCTOU defence).
  • Safetensors writes are atomic via tempfile.mkstemp + os.replace — a crash mid-write never leaves a partial adapter at the target path.
  • .bin (PyTorch pickle) adapter format is rejected with an explicit "re-save as safetensors" message.
  • Branch pointers live under ~/.soup/branches/ (override via SOUP_BRANCHES_DIR, constrained to $HOME / $CWD / $TMPDIR).
  • soup adapters checkout SHA-checks the source config — refuses to restore when the source has drifted from the snapshot, so reproducibility never silently lies.

v0.66.0: soup adapters blame is now LIVE — the v0.57 NotImplementedError stub (#171) is lifted via a DataInf-style influence-function approximation. Pass --top-k 50 to control the reported top-influencer count; pass a real probe_fn (Python API) to feed real gradients, or use the default deterministic synthetic probe for offline planning.

v0.71.4: the merge verdict is now LIVE — soup adapters merge … --canary suite.json lifts the MergeReport.verdict UNKNOWN stub. A pre-scored {"baseline_scores","candidate_scores"} suite classifies the blend OK / MINOR / MAJOR with no model load; a {"tasks":[...]} suite uses an injectable scorer. --strict-verdict exits 2 on MAJOR. The backdoor-scan and license-conflict gates now run for every strategy, including cmaes. soup adapters branch <name> --from-registry <id> / --attach-to-registry <id> link training-env snapshots into the Registry lineage DAG (shown as a branches node in soup history).

v0.71.34 — task-vector arithmetic. soup adapters arithmetic "<expr>" applies task arithmetic (arXiv:2212.04089) to LoRA deltas: add (blend two skills), scale (2*coder), and — the differentiator — negate (- toxic removes a behaviour). Names in the expression map to adapter dirs via repeatable --adapter name=path. The math is done at the effective-delta level: a LoRA contributes ΔW = B·A, so the coefficient is applied so that ΔW scales linearly (subtracting an adapter actually negates its delta — a naïve element-wise sum would scale by c², making negation a no-op). Adapters must share the base model (--allow-cross-base to override) and rank (mixed ranks are refused, not silently approximated). Each input passes the v0.71.2 backdoor-scan gate (--allow-unscanned to skip). Output is a single loadable adapter; exit 0 = ok, 1 = refusal.

Soup Cans (Shareable Recipes)

Share a reproducible recipe as a single .can file — a tarball of the manifest, full config, and a reference to the training data (URL or HF dataset). Not the weights, not the dataset bytes: just enough for someone else to re-run the same training.

# Pack a registry entry into a .can
soup can pack --entry-id llama31-chat-v1 --out ./llama31-chat.can

# Embed in-toto attestations into the manifest (v3 cans; repeatable)
soup can pack --entry-id llama31-chat-v1 --out ./signed.can --attest att.json

# Preview the manifest without extracting
soup can inspect ./llama31-chat.can

# Verify schema + config parseability
soup can verify ./llama31-chat.can

# Fork with modifications (dotted-path overrides) and re-pack
soup can fork ./llama31-chat.can --out ./llama31-chat-hot.can \
  --modify training.lr=5e-5 --modify training.epochs=5

# Run a .can end-to-end: extract → train (→ optional deploy)
soup can run ./llama31-chat.can --yes
soup can run ./llama31-chat.can --yes --deploy --env-capture ./env.txt

# Publish a .can to HF Hub as a dataset
soup can publish ./llama31-chat.can --hf-hub me/llama31-chat-recipe

Security — tar extraction uses filter="data" on Python 3.12+ with symlink/hardlink rejection fallback for older runtimes. Size cap: 100 MB. DataRef.url must be HTTPS. Fork overrides reject dunder keys (__class__, __init__) and null bytes. Manifest format version supports 1 and 2 (additive bump in v0.33.0 added deploy_targets). soup can run requires --yes (mandatory consent — auto-downloads data + auto-trains). soup can publish validates repo_id and resolves the HF token via env / cache files; commit messages are first-line + 200-char capped.

Bill of Materials (soup bom emit)

Emit a CycloneDX 1.6 ML-BOM or SPDX 2.3 + AI profile bill of materials from any training run. Procurement teams and compliance auditors can ingest the BOM directly into their existing tooling — no custom parser required.

soup bom emit \
  --name adapter-v1 --version 0.1.0 \
  --base-model meta-llama/Llama-3.1-8B \
  --base-sha aaaa...64hex \
  --config-sha bbbb...64hex \
  --task sft --license Apache-2.0 \
  --format both --output bom
# writes bom.cdx.json + bom.spdx.json

Root component is type=machine-learning-model (per CycloneDX ML-BOM extension). Base model + parent adapters + per-artifact files appear as components with SHA-256 hashes. --license is written by what it is: a listed SPDX id (any case, apache-2.0 becomes Apache-2.0) goes in CycloneDX license.id and in SPDX licenseConcluded / licenseDeclared; a valid SPDX expression (Apache-2.0 OR MIT, upper-case operators, every operand a listed id or a LicenseRef-) goes in CycloneDX expression and in those SPDX fields with its ids canonicalised, each LicenseRef- operand defined in hasExtractedLicensingInfos (a DocumentRef- qualified operand makes the value a name, since the document carries no external references); anything else, including a name that merely contains "and" or "with" (Gemma Terms of Use and Prohibited Use Policy), is CycloneDX license.name and an SPDX LicenseRef- with its text in hasExtractedLicensingInfos. The CycloneDX serialNumber is an RFC 4122 urn:uuid: (#1446).

Attaching energy + CO₂ (--energy)

Pass a measurement JSON to record energy and carbon in the BOM:

# 1. Train and persist the measured energy (needs `pip install soup-cli[carbon]`):
soup train --config soup.yaml --track-energy --energy-out energy.json

# 2. Attach it to the BOM:
soup bom emit ... --format both --output bom --energy energy.json

soup train --track-energy --energy-out <path> writes the measured reading as JSON in exactly the energy schema (energy_kwh / co2_kg / pue / grid_intensity_g_per_kwh / source); soup bom emit --energy <path> then consumes it. (You can also hand-write the JSON.) The path is validated (kept under cwd, symlinks rejected) on both the write and the read. Energy lands in both outputs: under metadata.properties (soup:energy_kwh, soup:co2_kg, …) in CycloneDX, and as OTHER annotations on the model package in SPDX.

Provenance Attestations (soup attest emit)

Emit an in-toto v1 Statement wrapping a SLSA-3 provenance v1 predicate for each Soup Can lifecycle stage:

soup attest emit \
  --stage train \
  --subject adapter-v1 \
  --sha aaaa...64hex \
  --builder [email protected] \
  --output att.json

Stages are a closed allowlist: extract / train / eval / export / publish. Subject SHA must be 64-hex (sha256). --invocation "<command line>" is recorded at predicate.buildDefinition.externalParameters.invocation (up to 4096 characters; a longer one is refused, not cut), beside the stage; the statement printed without --output is the same JSON, verbatim (#1446). The default --sign unsigned backend remains offline-only tamper metadata. The ed25519 backend is live with pip install soup-cli[sign]; Sigstore is live with pip install soup-cli[sigstore]:

# offline key signing
soup attest emit --stage train --subject adapter-v1 --sha aaaa...64hex \
  --sign ed25519 --key signing.pem --output att.json
soup attest verify att.json --signature att.json.sig --public-key trusted.pub

# keyless OIDC / Fulcio / Rekor
soup attest emit --stage train --subject adapter-v1 --sha aaaa...64hex \
  --sign sigstore --output att.json  # add --interactive-oidc only for browser auth
soup attest verify att.json --signature att.json.sig \
  --cert-identity 'https://github.com/acme/repo/.github/workflows/release.yml@refs/heads/main' \
  --cert-oidc-issuer 'https://token.actions.githubusercontent.com'

Both signing backends use the same <output>.sig JSON sidecar. For Sigstore it contains the complete modern Bundle (certificate, signature, and transparency proof). verify re-canonicalises the statement JSON and requires both the Sigstore certificate identity and OIDC issuer out of band; either value without the other is refused. Sigstore emission requires --output so the public Rekor entry can never outlive a discarded local bundle; browser OIDC is explicit opt-in. Verification exits 3 on tamper or policy mismatch, and exits 1 when Sigstore verification itself is unavailable (for example TUF/network failure) so an operational outage is not reported as tampering. Usage/input errors exit 2. The statement and signature sidecar accept UTF-8 with or without a leading BOM. Verification still uses the same canonical JSON and preserves U+FEFF inside field values; a leading BOM does not bypass tamper or trusted-key checks. An explicit Sigstore signing request fails closed if OIDC/Fulcio/Rekor is unavailable; it is never silently downgraded to unsigned. A valid signature proves the signer asserted the statement; it does not re-verify the subject digest against an artifact.

EU AI Act Annex XI/XII Auto-Doc (soup train --annex-xi)

Render an EU AI Act Annex XI (technical documentation, Sections 1+2) or Annex XII (Article 53(1)(d) public training summary) directly from a training run:

soup train --config soup.yaml --annex-xi annex.md           # markdown
soup train --config soup.yaml --annex-xi annex.pdf           # PDF (pip install soup-cli[pdf])
soup train --config soup.yaml --track-energy --annex-xi annex.md  # + measured kWh/CO2

Top-10 domains by share, modality breakdown, training compute / kWh / CO₂, model description, base model, run id. A .pdf output path renders a reportlab PDF (a .md path renders markdown). The top crawled domains are auto-extracted from the training JSONL (cfg.data.train). With --track-energy, the measured energy is recorded in the doc; without it, the energy and CO₂ rows read "not measured" rather than 0, and the training-compute (FLOPs) row always reads "not measured", because nothing estimates it yet (#1446). Operator-controlled fields are escape-neutralised (|[](){}!<> + newline / CR / tab) so a malicious model name can't inject a forged heading into downstream renderers.

Energy & CO₂ measurement (--track-energy)

soup train --track-energy wraps the training window in a codecarbon offline tracker (no IP-geolocation network call). It reports kWh / CO₂ / grid intensity and feeds them into --annex-xi. --energy-country <ISO3> picks the grid for the CO₂ estimate (default USA; the kWh figure itself is country-independent). Add --energy-out <path> to persist the measurement as JSON for soup bom emit --energy <path> (the hand-off into an ML-BOM — see Attaching energy + CO₂). Requires pip install soup-cli[carbon]; without it, --track-energy is a graceful no-op.

Audit Log (soup audit-log)

Every soup command now appends one HIPAA/SOC2-shaped record to the JSONL audit log at ~/.soup/audit.jsonl (override via SOUP_AUDIT_LOG_PATH, containment-checked to $HOME / $CWD / $TMPDIR). Opt out per-invocation with soup --no-audit-log <cmd> or globally with SOUP_NO_AUDIT_LOG=1. Tail or rotate:

soup audit-log tail --limit 50          # Rich table view
soup audit-log tail --json              # raw JSONL for SIEM ingestion
soup audit-log rotate --cap-mb 100      # force a rotation pass
soup --no-audit-log version             # skip the audit line for this command

PII redaction across every string field (hf_* / sk-* / Bearer … → <redacted>) via the v0.40.3 _SECRET_RE policy. POSIX O_NOFOLLOW + 0o600 perms, atomic-append, rotation at 100 MiB with symlink rejection at the backup path. The auto-instrumentation is best-effort — a broken audit log never crashes the CLI. Lines written by older versions get the same webhook URL masking as new ones, applied once the next time the log is written (audit.jsonl.masked-v1 marks it done). A file it cannot replace is left as it is; the attempt is recorded in audit.jsonl.masked-v1.tried and the next one waits a day, so a log another process keeps open is not re-read by every command. Temp files an interrupted pass left beside the log are removed by the next pass once they are an hour old.

Reproducibility Receipt (soup train --repro-receipt)

SR 11-7-style reproducibility receipt captures seeds (torch + numpy + python), kernel versions (CUDA + cuDNN + NCCL), accelerator backend, GPU model + driver, OS + arch. On Apple Silicon it records the privacy-safe chip name and unified-memory capacity, but never a serial number, hardware UUID, or user-specific path:

soup train --config soup.yaml --repro-receipt repro.json

Bank model-risk teams and regulated-org auditors get a single JSON file that fingerprints the exact environment the run executed in. Atomic write, cwd-contained.

Run-vs-Config Audit (soup adapters audit)

Does the finished adapter match the config that asked for it? Five merged fixes taught the MLX path to record what it actually did into adapter_config.json — #683 (masking), #684 (accumulation), #685 (gradient checkpointing), #686 (optimizer and schedule), #749 (gradient clipping). Each existed because a setting was accepted and silently dropped. Nothing read that record back until this command.

soup adapters audit ./output --config soup.yaml
soup adapters audit ./output --config soup.yaml --json   # machine-readable
Audit: output
│ optimizer                    │ adamw_torch │ AdamW  │ ok      │
│ learning_rate                │ 0.0001      │ 0.0001 │ ok      │
│ warmup_ratio                 │ 0.03        │ —      │ unknown │
│ weight_decay                 │ 0.01        │ 0.01   │ ok      │
│ lora.r                       │ 8           │ 8      │ ok      │

unknown is never ok. A setting the record cannot speak to is reported unknown, never agreement — a false clean bill turns "I do not know" into "I checked", which is exactly the substitution this command exists to undo. It is not a failure either: an older adapter predates the keys, which is not the user's fault.

The config is read through the schema, so a setting you omitted is audited against the default Soup would actually have used, not against a second copy of the defaults kept by the audit.

The MLX learning-rate comparand is peak_lr. training.lr is the target peak/base value passed to the resolved optimizer schedule. Warmup starts below it and controls when the schedule reaches it; cosine or linear decay controls what follows. Comparing training.lr with an early schedule sample would therefore flag a correct warmup run. The audit instead compares it with the optimizer plan's recorded peak_lr. It deliberately does not use the record's learning_rate, because that field is only the config value echoed when the file is written, not evidence of what reached the optimizer.

PEFT's adapter_config.json on the transformers backend does not record the effective optimizer learning rate. That row is therefore unknown, and the output says explicitly that the effective learning rate was not checked; it never presents a transformers adapter with a false clean bill.

Masking is audited by effect, not by request. The record's train_on_responses_only is the config's own request echoed back; the effect is mask_prompt (upstream's single masked prefix) and response_token_mask (Soup's per-token mask). Plain-text rows carry no role boundaries, so a run that asked for response-only masking on them warns once and trains on the full sequence — leaving a record that says train_on_responses_only: true beside two false effect keys. The audit compares against the effect, so that run is reported DIVERGED.

Evidence is weighed the same way everywhere else in this command: one truthy key proves masking happened, whatever the other says, but proving it did not happen needs both keys. A record carrying neither (written before #683) — or carrying one false key with the other absent — is unknown, not a divergence, because a missing key read as False would claim a check the record cannot support.

--json emits checked_count alongside diverged_count and unknown_count, so a CI job can tell "everything agreed" from "nothing was checkable" — both of which exit 0 — plus unknown_reason, the same explanation the table prints. Config warnings (an unknown key, for instance) go to stderr, so stdout under --json is the payload and nothing else.

Exit codes: 0 = agreement, 2 = DIVERGED, 1 = usage or read error. The verdict is kept off 1 so a CI gate can tell "the run did not do what the config asked" from "the path was wrong" -- a missing adapter_config.json, a missing --config and a path outside the working directory all exit 1. This follows soup ship / soup shrink (0 pass / 2 failed gate / 1 error) rather than the older adapters scan, which predates that convention. unknown rows exit 0. For GRPO runs, any non-zero nan_skip_fraction reads as DIVERGED (exit 2); even a single skipped step out of thousands is surfaced so the user can decide whether it matters.

Strings in adapter_config.json are untrusted -- an adapter can be downloaded -- so record-derived text is stripped of ANSI/OSC control bytes and escaped against Rich markup before it is printed. --json is not sanitised: a machine consumer gets the bytes the record actually holds.

Adapter Backdoor Scanner (soup adapters scan)

Spectral analysis of LoRA adapter weights pre-load. Flags rank-1 dominance (the canonical weight-space trojan pattern), top-1 singular-vector energy concentration, NaN/Inf in weights, and Frobenius-norm outliers via robust median + MAD bucketing. Pure numpy, no torch. Exit codes 0=OK / 1=WARN / 3=FAIL so CI can grep specifically for security failures. Reuses the v0.57.0 adapter_diff loader so the on-disk surface stays single-source.

Adapter Sign + Verify (soup adapters sign / verify)

Deterministic Merkle-root manifest over every file in the adapter dir (including nested tokenizer/ / processor/ subdirs). Tamper any file and verify fails. The unsigned backend gives offline tamper detection (the Merkle-root hash is the trust anchor). The ed25519 backend is live (pip install soup-cli[sign]): it adds a real detached signature over the Merkle root for authentication — proof the signer holds the private key.

# generate a keypair + sign in one shot
soup adapters sign ./adapter --backend ed25519 --generate-key signing.pem
# or sign with an existing key (or set SOUP_SIGNING_KEY)
soup adapters sign ./adapter --backend ed25519 --key signing.pem
# self-consistency verify (embedded public key)
soup adapters verify ./adapter
# genuine authentication: require the signer's key match a trusted one
soup adapters verify ./adapter --public-key trusted.pub

verify fails closed — any tamper, wrong/missing/unreadable key, or a signature from an untrusted key marks the adapter invalid. With --public-key, an adapter whose signature record is not ed25519 is also invalid. Signing keys and trusted public keys are symlink-rejected and size-capped but not cwd-contained (keys are secrets that live outside the project). Signature persists as .soup-signature.json (atomic write).

The sigstore backend is also live with pip install soup-cli[sigstore]. It uses an ambient OIDC identity when one is available (for example GitHub Actions), requests a Fulcio certificate, submits to Rekor, and stores the complete Sigstore bundle with the adapter signature record. Browser OIDC is not opened implicitly on headless/default runs; opt in explicitly with --interactive-oidc. Verification is deliberately fail-closed and requires both certificate identity and OIDC issuer supplied out of band:

soup adapters sign ./adapter --backend sigstore
soup adapters verify ./adapter \
  --cert-identity 'https://github.com/acme/repo/.github/workflows/release.yml@refs/heads/main' \
  --cert-oidc-issuer 'https://token.actions.githubusercontent.com'

The identity embedded in the bundle is not trusted automatically; doing so would reduce authentication to self-consistency. Identity and issuer are one certificate policy: either value without the other is refused. --strict mode exits 3 on any verify failure (CI gate code distinct from generic errors).

Strict Safetensors Mode (soup adapters check-safetensors)

Refuses pickle / PyTorch-classic weights at the boundary — closed 8-entry unsafe-extension allowlist (.bin / .pt / .pth / .ckpt / .pkl / .pickle / .joblib / .msgpack). Picklemod gives every loader the right to execute arbitrary code on load; refusing the file at the boundary is the only sound mitigation. Friendly advisory names the offending file and the canonical from safetensors.torch import save_file recipe. Exit code 3 under --strict for CI gating.

Pre-Flight Adapter Health Audit (soup adapters check)

Pre-flight scan auditing single LoRA adapters for silent training failures (#1721):

  • Total and per-layer ||dW||_F: Frobenius norm of LoRA parameter updates (dW = scaling * (B @ A), with alpha / r for standard LoRA and alpha / sqrt(r) for rsLoRA) with float64 accumulation via matrix trace contraction.
  • Live fraction: Proportion of adapter layers with active, non-zero updates.
  • All-zero lora_B layers: Flags un-trained or zero-initialized layers that failed to move during training.
  • Incomplete LoRA pairs: Flags orphaned or partially-saved projections missing either A or B.
  • Leaked .inner. keys: Flags legacy wrapper namespaces for adapters saved prior to the #1011 save-time guard.
soup adapters check path/to/adapter/
# Machine-readable output
soup adapters check path/to/adapter/ --json

Outputs human diagnostics ending in a single verdict line: alive (exit code 0) or inactive: <reason> (exit code 2 for inactive/silent failure, distinct from usage error 1). Possible inactivity reasons include:

  • all lora_B layers are zero (n/n)
  • lora_B layers are zero (k/n)
  • total ||ΔW||_F is 0.0
  • leaked .inner. keys detected (<n> tensors)
  • incomplete LoRA pairs detected (<n> orphans)
  • shape mismatch in LoRA projections (<n> layers)
  • non-finite weights detected (NaN/Inf)
  • no lora weights found

Namespace Pinning (Anti-AI-Jacking)

Trust-on-first-use SQLite cache: records (repo_id, author, created_at) the first time Soup sees a HuggingFace repo. Subsequent loads compare the current Hub fingerprint to the recorded pin — refuses author change or backward created_at jump unless the operator passes --allow-namespace-shift <new-author> (case-insensitive author match; bool rejected so callers can't smuggle a free-for-all bypass). Threat model: an attacker watches a popular repo, waits for the original owner to delete or expire it, then re-creates the same owner/name with malicious weights. Without this control, anyone with Soup pinned to that namespace silently pulls poison on the next run. The gate is wired into Hub model downloads — it consults the pin before fetching and refuses a mismatch (raising before any weights land). It fails open when repo metadata can't be fetched (offline / rate-limited) so a transient hiccup never blocks a legitimate download. The pin store uses SQLite WAL + a cross-process file lock so concurrent fetches don't lose the trust anchor.

License-Conflict Matrix at Merge

Closed compatibility table over 33 SPDX-ish licenses spanning Apache / MIT / BSD / LGPL / MPL / GPL / AGPL / CC-BY / CC-BY-NC / Llama-2/3.x / Gemma / Qwen-research / Mistral-research / OpenRAIL / OpenAI-ToS / Anthropic-AUP. soup adapters merge --license apache-2.0 --license cc-by-nc-4.0 -o merged refuses (non-commercial cannot combine with permissive). When you don't pass --license, the license is auto-detected from each adapter's adapter_config.json / config.json / model-card frontmatter (HF llama3.1-style ids map to canonical) and the gate runs automatically when every adapter resolves to a known license. To proceed past a flagged conflict, pass --license-override "legal-cleared 2026-05-19 by alice" (8-char min, 4096-char max reason) — the decision is recorded to the audit log for later legal review.

Merge is additionally gated by the backdoor scanner: any input that soup adapters scan flags FAIL (or that can't be scanned) refuses the merge with exit 3 unless you pass --allow-unscanned.

Airgap Bundle (soup airgap-bundle)

Single signed tarball with model + datasets + wheels + CUDA kernels + embedded manifest.json listing SHA-256 per file. Sized for one-way physical-media transfer through a data diode. A reproducibility receipt is embedded as a top-level repro-receipt.json when you pass --repro-receipt <receipt.json> (or auto-detected from <model>/repro-receipt.json):

soup airgap-bundle --model ./out --output bundle.tar --repro-receipt repro.json

Default 100 GiB cap; refuses oversize. Deterministic dataset labeling by sorted basename (NOT argv order) so the same inputs in different argv order produce identical manifests. TOCTOU defence: os.lstat + S_ISLNK re-check on parent + final output path before mkstemp; atomic os.replace from a sibling tempfile. tarfile.data_filter set on Python 3.12+ so any future caller adding tar.extractall automatically gets the safe-mode extraction filter. soup airgap-bundle is intentionally a top-level command (not soup deploy airgap-bundle) — it's an export operation, not a deploy target.