Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

7.6 KB
Raw Blame

Compliance & governance quickstart

← Back to the docs index

Soup ships an end-to-end compliance workflow: start from a regulation-shaped config, train with provenance capture, sign + attest the artifact, keep an audit trail, and move it across an air gap — then publish a documented model card and gate future changes in CI.

  • Pre-wired configs: soup init --template hipaa|soc2|eu-ai-act|sr-11-7
  • Provenance: soup train --repro-receipt · soup bom emit · soup attest emit
  • Integrity: soup adapters sign / verify · soup adapters scan
  • Audit trail: the HIPAA/SOC2 audit log is on by default (soup audit-log tail)
  • Air gap: soup airgap-bundle
  • Publish: soup card <registry-id> → MODELCARD.md (or soup push --card)
  • CI gate: soup ci init → a PR workflow that runs validate → expect → ship

1. Start from a compliance template

Each template is a normal training config plus header comments listing the exact compliance commands to run around it. Pick the regime you operate under:

soup init --template hipaa      # Protected Health Information
soup init --template soc2       # SOC 2 Trust Services Criteria
soup init --template eu-ai-act  # EU AI Act Annex XI/XII
soup init --template sr-11-7    # SR 11-7 Model Risk Management

The compliance controls are Soup CLI flags/commands, not config keys — the template header documents which ones apply. The steps below are the common path.

2. Clean the data before training

soup data pii --input ./data/train.jsonl    # flag emails / phones / SSNs / MRNs
soup data decontaminate --input ./data/train.jsonl --benchmark-file benchmarks.jsonl  # drop benchmark overlap

3. Train with a reproducibility receipt (+ Annex XI / energy for the EU)

# SR 11-7 / SOC 2 / HIPAA: capture seeds, kernels, GPU, OS
soup train --config soup.yaml --repro-receipt receipt.json

# EU AI Act: auto-generate the Annex XI/XII documentation + measure energy
soup train --config soup.yaml \
    --annex-xi annex_xi.md \
    --track-energy --energy-country DEU --energy-out energy.json

The audit log records every command automatically:

soup audit-log tail          # review the trail
soup audit-log rotate        # force a rotation pass

4. Register the run, then emit BOM + attestation

soup registry push --run-id <run-id> --name my-model --tag v1

soup bom emit --name my-model --base-model <model-id> \
    --base-sha <hex> --config-sha <hex> \
    --energy energy.json --format both -o my-model.bom \
    --attach-to-registry my-model:v1         # CycloneDX + SPDX, linked to the entry
soup attest emit --stage train --subject my-model --sha <hex> \
    --sign ed25519 --key key.pem -o my-model.attest.json \
    --attach-to-registry my-model:v1         # in-toto + SLSA-3, linked to the entry

--attach-to-registry registers the emitted BOM / attestation as bom / attestation artifacts on the entry. Signed attestations include their .sig sidecar, so the model card (step 7) links every file needed for verification. The option needs --output; omitting it is a usage error. A registry lookup or attachment failure exits non-zero while leaving the emitted files on disk.

5. Sign, scan, and verify the artifact

soup adapters scan ./output                              # weight-space backdoor scan
soup adapters sign ./output --backend ed25519 --generate-key key.pem
soup adapters verify ./output --strict --public-key key.pub.pem

6. Air-gap transfer (optional)

soup airgap-bundle --model ./output --output my-model.tar --repro-receipt receipt.json

7. Generate a documented model card

Turn the registry entry into a provenance-rich MODELCARD.md — base model, training config, eval scorecard, config/data hashes, lineage, and every registered artifact — including the BOM and attestation attached in step 4:

soup card my-model:v1 -o MODELCARD.md
# or, when uploading to the Hub, override the auto-generated card:
soup push --model ./output --repo you/my-model --card my-model:v1

8. Gate future changes in CI

Write a GitHub Actions workflow that blocks a PR unless the data validates, the expectations suite passes, and the SHIP verdict is green:

soup ci init --data data/train.jsonl --suite expectations.yaml --evidence ship_evidence.json
# writes .github/workflows/soup-gate.yml

The generated job runs, in order:

soup data validate <data> --min-valid-fraction 1.0  # every row must convert
soup expect <data> <suite>      # PII / token-length / refusal / judge expectations
soup ship --evidence <ev.json>  # SHIP / DON'T-SHIP (exit 2 blocks the merge)

The generated data gate refuses any dropped row, including skipped JSONL records. The standalone validator keeps its permissive default. To tolerate partial data, edit --min-valid-fraction in your workflow. Existing workflow files are unchanged until you regenerate them with soup ci init --force.

A minimal expectations.yaml for the second step:

expectations:
  - name: expect_no_pii
  - name: expect_token_length_between
    args:
      min_tokens: 1
      max_tokens: 512

Supported names: expect_no_pii, expect_token_length_between, expect_no_refusal_pattern, expect_chosen_preferred_over_rejected_by_judge. Arguments go under args:. An entry key other than name / args, or an argument the expectation does not take (max_token), is refused when the suite loads (exit 3), so a mistyped bound cannot silently run on the defaults. A top-level key other than expectations (a fail_fast: or a name: label beside it) is refused the same way, naming the key. expect_chosen_preferred_over_rejected_by_judge requires a judge model configured under args: {judge: "ollama://llama3.1", threshold: 0.7} or passed via CLI with soup expect --judge <url>. When both are set, --judge takes precedence over the suite's args.judge. Without a judge configured, the gate fails (or pass args: {advisory: true} to explicitly trust existing labels in advisory mode).

Scanned fields per format:

  • ChatML / tool-calling / audio: every messages[].content (string or text parts), tool-call arguments (per message and top-level tool_calls), and tools[].function.description.
  • ShareGPT / LLaVA / ShareGPT4V: every conversations[].value.
  • Alpaca: instruction, input, output, system, response.
  • DPO: prompt, chosen, rejected (string or message list). KTO: prompt, completion.
  • Embedding: anchor, positive, negative (string or list).
  • Plaintext / ASR: text. RAFT: query, golden_doc, distractor_docs, answer. PRM: prompt, completions. input_output: segments[].text.

expect_no_refusal_pattern reads only the assistant side: assistant/gpt turns, DPO chosen, KTO completion, Alpaca output/response, RAFT answer, PRM completions, segments with label: true, and text for plaintext rows. A row with text but no assistant side passes it.

Empty-text safeguard: rows containing no extractable text fail closed on all expectation gates to prevent unparseable or blank rows from passing CI. Malformed JSON lines and valid JSON that is not an object are refused as an input error (exit 3) with their line numbers, a UTF-8 BOM is stripped the way the training loader strips it, and a file that yields no checkable rows is refused rather than passed.

Every path is shell-quoted and validated to stay under the repo root, so the rendered workflow is injection-safe. Edit the paths to match your repo.


See also: Adapters, registry & governance for the full supply-chain command set, and Evaluation & probes for the soup ship verdict engine.