Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Compliance & governance quickstart
Soup ships an end-to-end compliance workflow: start from a regulation-shaped config, train with provenance capture, sign + attest the artifact, keep an audit trail, and move it across an air gap — then publish a documented model card and gate future changes in CI.
- Pre-wired configs:
soup init --template hipaa|soc2|eu-ai-act|sr-11-7 - Provenance:
soup train --repro-receipt·soup bom emit·soup attest emit - Integrity:
soup adapters sign/verify·soup adapters scan - Audit trail: the HIPAA/SOC2 audit log is on by default (
soup audit-log tail) - Air gap:
soup airgap-bundle - Publish:
soup card <registry-id>→MODELCARD.md(orsoup push --card) - CI gate:
soup ci init→ a PR workflow that runs validate → expect → ship
1. Start from a compliance template
Each template is a normal training config plus header comments listing the exact compliance commands to run around it. Pick the regime you operate under:
soup init --template hipaa # Protected Health Information
soup init --template soc2 # SOC 2 Trust Services Criteria
soup init --template eu-ai-act # EU AI Act Annex XI/XII
soup init --template sr-11-7 # SR 11-7 Model Risk Management
The compliance controls are Soup CLI flags/commands, not config keys — the template header documents which ones apply. The steps below are the common path.
2. Clean the data before training
soup data pii --input ./data/train.jsonl # flag emails / phones / SSNs / MRNs
soup data decontaminate --input ./data/train.jsonl --benchmark-file benchmarks.jsonl # drop benchmark overlap
3. Train with a reproducibility receipt (+ Annex XI / energy for the EU)
# SR 11-7 / SOC 2 / HIPAA: capture seeds, kernels, GPU, OS
soup train --config soup.yaml --repro-receipt receipt.json
# EU AI Act: auto-generate the Annex XI/XII documentation + measure energy
soup train --config soup.yaml \
--annex-xi annex_xi.md \
--track-energy --energy-country DEU --energy-out energy.json
The audit log records every command automatically:
soup audit-log tail # review the trail
soup audit-log rotate # force a rotation pass
4. Register the run, then emit BOM + attestation
soup registry push --run-id <run-id> --name my-model --tag v1
soup bom emit --name my-model --base-model <model-id> \
--base-sha <hex> --config-sha <hex> \
--energy energy.json --format both -o my-model.bom \
--attach-to-registry my-model:v1 # CycloneDX + SPDX, linked to the entry
soup attest emit --stage train --subject my-model --sha <hex> \
--sign ed25519 --key key.pem -o my-model.attest.json \
--attach-to-registry my-model:v1 # in-toto + SLSA-3, linked to the entry
--attach-to-registry registers the emitted BOM / attestation as bom /
attestation artifacts on the entry. Signed attestations include their .sig
sidecar, so the model card (step 7) links every file needed for verification.
The option needs --output; omitting it is a usage error. A registry lookup or
attachment failure exits non-zero while leaving the emitted files on disk.
5. Sign, scan, and verify the artifact
soup adapters scan ./output # weight-space backdoor scan
soup adapters sign ./output --backend ed25519 --generate-key key.pem
soup adapters verify ./output --strict --public-key key.pub.pem
6. Air-gap transfer (optional)
soup airgap-bundle --model ./output --output my-model.tar --repro-receipt receipt.json
7. Generate a documented model card
Turn the registry entry into a provenance-rich MODELCARD.md — base model,
training config, eval scorecard, config/data hashes, lineage, and every
registered artifact — including the BOM and attestation attached in step 4:
soup card my-model:v1 -o MODELCARD.md
# or, when uploading to the Hub, override the auto-generated card:
soup push --model ./output --repo you/my-model --card my-model:v1
8. Gate future changes in CI
Write a GitHub Actions workflow that blocks a PR unless the data validates, the expectations suite passes, and the SHIP verdict is green:
soup ci init --data data/train.jsonl --suite expectations.yaml --evidence ship_evidence.json
# writes .github/workflows/soup-gate.yml
The generated job runs, in order:
soup data validate <data> --min-valid-fraction 1.0 # every row must convert
soup expect <data> <suite> # PII / token-length / refusal / judge expectations
soup ship --evidence <ev.json> # SHIP / DON'T-SHIP (exit 2 blocks the merge)
The generated data gate refuses any dropped row, including skipped JSONL records.
The standalone validator keeps its permissive default. To tolerate partial data,
edit --min-valid-fraction in your workflow. Existing workflow files are unchanged
until you regenerate them with soup ci init --force.
A minimal expectations.yaml for the second step:
expectations:
- name: expect_no_pii
- name: expect_token_length_between
args:
min_tokens: 1
max_tokens: 512
Supported names: expect_no_pii, expect_token_length_between,
expect_no_refusal_pattern, expect_chosen_preferred_over_rejected_by_judge.
Arguments go under args:. An entry key other than name / args, or an
argument the expectation does not take (max_token), is refused when the suite
loads (exit 3), so a mistyped bound cannot silently run on the defaults. A
top-level key other than expectations (a fail_fast: or a name: label
beside it) is refused the same way, naming the key.
expect_chosen_preferred_over_rejected_by_judge requires a judge model configured
under args: {judge: "ollama://llama3.1", threshold: 0.7} or passed via CLI with
soup expect --judge <url>. When both are set, --judge takes precedence over
the suite's args.judge. Without a judge configured, the gate fails (or pass
args: {advisory: true} to explicitly trust existing labels in advisory mode).
Scanned fields per format:
- ChatML / tool-calling / audio: every
messages[].content(string or text parts), tool-call arguments (per message and top-leveltool_calls), andtools[].function.description. - ShareGPT / LLaVA / ShareGPT4V: every
conversations[].value. - Alpaca:
instruction,input,output,system,response. - DPO:
prompt,chosen,rejected(string or message list). KTO:prompt,completion. - Embedding:
anchor,positive,negative(string or list). - Plaintext / ASR:
text. RAFT:query,golden_doc,distractor_docs,answer. PRM:prompt,completions. input_output:segments[].text.
expect_no_refusal_pattern reads only the assistant side: assistant/gpt turns, DPO chosen, KTO completion, Alpaca output/response, RAFT answer, PRM completions, segments with label: true, and text for plaintext rows. A row with text but no assistant side passes it.
Empty-text safeguard: rows containing no extractable text fail closed on all expectation gates to prevent unparseable or blank rows from passing CI. Malformed JSON lines and valid JSON that is not an object are refused as an input error (exit 3) with their line numbers, a UTF-8 BOM is stripped the way the training loader strips it, and a file that yields no checkable rows is refused rather than passed.
Every path is shell-quoted and validated to stay under the repo root, so the rendered workflow is injection-safe. Edit the paths to match your repo.
See also: Adapters, registry & governance for the
full supply-chain command set, and Evaluation & probes for the
soup ship verdict engine.