Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
feat(cloud): add RunPod controller with verified cleanup (#2081)
Nurkhan Esenbekfc47970a456b
diff --git a/changelog.d/0.76.0/2081.added.md b/changelog.d/0.76.0/2081.added.mdnew file mode 100644index 00000000..deeed6a0--- /dev/null+++ b/changelog.d/0.76.0/2081.added.md@@ -0,0 +1 @@+- Add a RunPod cloud-training controller with plan-only rendering, bounded YAML validation, isolated credential-gated submission, SSH identity isolation, per-run host files and confirmed Pod deletion with transient-error retries. Preserve proxy, temporary-directory, Windows and SSH-agent settings while filtering other parent variables, and replace unencodable log characters under isolated Python. Report safe failure reasons and distinguish rejected creates from ambiguous launches; not yet live-validated. (#2081 by @kok-o)diff --git a/docs/backends-and-ops.md b/docs/backends-and-ops.mdindex d22ea04f..09b94f9e 100644--- a/docs/backends-and-ops.md+++ b/docs/backends-and-ops.md@@ -372,7 +372,7 @@ training:## Cloud GPU Training-No local GPU? `soup train --cloud modal|lambda` renders a provider-specific controller+No local GPU? `soup train --cloud modal|lambda|runpod` renders a provider-specific controllerfrom your `soup.yaml`. The config YAML is base64-embedded as **data**; credentials are read fromthe environment only when a live submission starts.@@ -397,11 +397,64 @@ including after a failed run, the app downloads every file in that directory intoutput directory and prints a `modal volume get soup-outputs /<run name> <output dir> --force`command that retries the download; that command places the files under `<output dir>/<run name>`.-### RunPod (Planned)+### RunPod-RunPod support is currently in development and descoped from live CLI dispatch pending automated-lifecycle and termination safeguards. Running `soup train --cloud runpod` informs the operator that-RunPod is not yet live and points to active cloud backends (`--cloud modal` and `--cloud lambda`).+`soup train --cloud runpod --gpu a100` writes `soup_runpod_app.py` without contacting+RunPod. GPUs: `a100` (80 GB SXM), `h100`, `a40`, `rtx3090`, `rtx4090`. The controller+uses Python's standard library and local OpenSSH `ssh`/`scp`; no RunPod SDK extra+is needed. For live submission, set `RUNPOD_API_KEY` and `RUNPOD_SSH_PRIVATE_KEY`+to your private key path, with its matching OpenSSH public key at `<path>.pub`.+Only ed25519 and RSA public keys are accepted:++```bash+soup train --config soup.yaml --cloud runpod --gpu rtx4090 --cloud-submit+```++The local controller creates one Secure Cloud GPU Pod with TCP SSH access, installs+the running Soup version, runs training over SSH and copies the configured relative+output directory back before deleting the Pod. Choose a fresh local output path;+existing outputs, symlink/junction paths and path components starting with `-` are+refused before allocation and rechecked before copying. SCP separates options from+its operands with `--`. Submission starts Python in isolated mode (`-I`) and passes+only the RunPod token/key path, `PATH`, `HOME`, `USERPROFILE`, proxy settings+(`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY` and their lowercase forms),+temporary-directory settings (`TEMP`, `TMP`, `TMPDIR`), Windows settings+(`SYSTEMROOT`, `SYSTEMDRIVE`, `WINDIR`) and `SSH_AUTH_SOCK` when+present. Environment names are matched case-insensitively on Windows. The controller+reconfigures stdout/stderr to replace characters the pipe encoding cannot represent,+because isolated Python ignores `PYTHONIOENCODING` and `PYTHONUTF8`.+SSH/SCP use `IdentitiesOnly=yes` with the selected key and a temporary per-run known-hosts+file, removed even after failure; the user's known-hosts file is not modified.+The config is base64-embedded data; planning applies the+shared YAML size, nesting and alias-expansion limits, and refuses inline credential+fields. The API key and private SSH key remain local;+only the public key enters the Pod environment. Models and datasets must be publicly+accessible remotely: local input files and local credentials are not uploaded.++Deletion runs in `finally` on training/copy failure, timeout and Ctrl+C, and polls+until the API returns 404. The parent waits through interrupts so it does not kill+the cleanup controller. Transient transport failures, including truncated HTTP+responses during DELETE or confirmation GET, are retried within the five-minute+cleanup window. A lost or ambiguous create response (including HTTP 5xx and 408)+is reconciled by the unique run name, without repeating the create request or+deleting unrelated Pods. A definite HTTP 4xx rejection other than 408 reports the+status without claiming that cleanup of an allocated Pod failed. Fixed controller+failure reasons are shown; arbitrary transport/subprocess exception text is hidden.+Keep the controller running; force-killing it or losing API access can leave cleanup+unconfirmed. An unconfirmed cleanup exits nonzero and names the run to check in+the RunPod console. A single interrupted SSH training session is failed rather+than retried, to avoid duplicate training. Keepalive probes run every 30 seconds+with six missed probes allowed (roughly three minutes); the training command has+a 25-hour timeout. Artifacts are copied only after a successful training command,+with one SCP attempt capped at 15 minutes. A failed copy still terminates the Pod,+so an incomplete download cannot be recovered from that Pod afterward. Remote log+tail retrieval and a copy retry window are not implemented in this controller.++**Not yet live-validated.** Tests use local mocked API/SSH transports. The code+PR does not include a paid run; provider execution, GPU behavior, retrieval and+billing termination remain a separate live-validation task, per #264's ruling.+API contract: [RunPod REST v1 OpenAPI](https://rest.runpod.io/v1/openapi.json),+[SSH setup](https://docs.runpod.io/pods/configuration/use-ssh).### Lambda Cloud@@ -435,7 +488,7 @@ soup train --config soup.yaml --cloud lambda --gpu a100 --cloud-submitcontroller can copy the artifact back safely. The API key stays on the caller and is never embeddedin cloud-init or instance logs.-The Lambda submission path still requires the paid live-validation checklist in #264 before+The Lambda submission path still requires the paid live-validation checklist in #1280 beforeit can be described as provider-validated. Plan-only rendering and the lifecycle boundaries arecovered by offline tests.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex bcff680a..402f0c0b 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -3667,7 +3667,7 @@ Usage: `soup train [OPTIONS]`| `--track-energy` | flag | off | | Measure the training window's energy + CO2 via codecarbon (offline; requires `pip install soup-cli[carbon]`). Feeds the kWh / CO2 into --annex-xi. v0.71.3. || `--energy-country` | text | `USA` | | ISO 3166-1 alpha-3 country code for the CO2 grid-intensity estimate used by --track-energy (default USA). || `--energy-out` | text | | | Write the --track-energy measurement to this JSON file (cwd-contained) so `soup bom emit --energy <path>` can consume it. v0.71.15. |-| `--cloud` | text | | | Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: modal, lambda (runpod is planned). Renders a cloud app stub from the config (plan-only); use --cloud-submit to submit live. |+| `--cloud` | text | | | Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: modal, lambda, runpod. Renders a cloud app stub from the config (plan-only); use --cloud-submit to submit live. || `--gpu` | text | `a100` | | Cloud GPU type for --cloud (t4 / l4 / a10 / a10g / a100 / a100-80gb / l40s / h100 / a6000). Default a100. Provider-specific allowlists apply. || `--cloud-submit` | flag | off | | With --cloud, submit the rendered run live via the cloud's SDK or API (gated on respective provider token/API key). Default is plan-only (render + print the command). Lambda requires a registered SSH key. |diff --git a/docs/commands.md b/docs/commands.mdindex a2d4057a..4fba8f86 100644--- a/docs/commands.md+++ b/docs/commands.md@@ -27,7 +27,7 @@ soup train --config soup.yaml --gate evals/gate.yaml Eval-gated trainingsoup train --config soup.yaml --push-as user/repo Auto-push each checkpoint to HF as branchsoup train --config soup.yaml --push-as user/repo --hf-resume Resume from latest HF checkpoint branch (not with training.relora_steps)soup train --config soup.yaml --find-lr LR range finder: write recommended LR JSON-soup train --config soup.yaml --cloud modal|lambda --gpu a100 Render a cloud GPU controller (plan-only; --cloud-submit submits live)+soup train --config soup.yaml --cloud modal|lambda|runpod --gpu a100 Render a cloud GPU controller (plan-only; --cloud-submit submits live)soup infer --model ./output --input p.jsonl --output out.jsonl [--batch-size N] Batch inference; default batch size is 1; fp16/bf16 batching may not be bit-exact with batch size 1soup infer --model ./output --input p.jsonl --output out.jsonl --cuda-graphs Experimental CUDA graph decode (Qwen2/Llama, one GPU, PyTorch >= 2.14; batch size 1 only)soup infer --task asr --model <whisper|adapter> --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate] Whisper transcription + WER/CERdiff --git a/src/soup_cli/cloud/__init__.py b/src/soup_cli/cloud/__init__.pyindex 39aee73d..d56b03de 100644--- a/src/soup_cli/cloud/__init__.py+++ b/src/soup_cli/cloud/__init__.py@@ -1,8 +1,8 @@"""Cloud GPU backends for ``soup train --cloud`` (v0.71.18 #16, v0.71.22 #264).-Ships Modal and Lambda Cloud backends, with RunPod planned.+Ships Modal, Lambda Cloud and RunPod backends."""from __future__ import annotations-__all__ = ["lambda_labs", "modal"]+__all__ = ["lambda_labs", "modal", "runpod"]diff --git a/src/soup_cli/cloud/_runpod_controller.py b/src/soup_cli/cloud/_runpod_controller.pynew file mode 100644index 00000000..88a3f781--- /dev/null+++ b/src/soup_cli/cloud/_runpod_controller.py@@ -0,0 +1,330 @@+"""Standalone controller template appended after validated plan constants.++REST contract: https://rest.runpod.io/v1/openapi.json+SSH contract: https://docs.runpod.io/pods/configuration/use-ssh+Only main() performs network operations. This module is also copied into plans.+"""++import http.client+import ipaddress+import json+import os+import pathlib+import re+import shutil+import signal+import stat+import subprocess+import sys+import tempfile+import time+import urllib.error+import urllib.request+import uuid++# Isolated Python ignores PYTHONIOENCODING/PYTHONUTF8. A redirected Windows+# stream may not encode a log character; print a replacement instead of failing.+for _stream in (sys.stdout, sys.stderr):+ if hasattr(_stream, "reconfigure"):+ _stream.reconfigure(errors="replace")++_GPU_ID = globals().get("_GPU_ID", "")+_CONFIG_B64 = globals().get("_CONFIG_B64", "")+_OUTPUT = globals().get("_OUTPUT", "")+_SOUP_VERSION = globals().get("_SOUP_VERSION", "")++_API_BASE = "https://rest.runpod.io/v1"+_POD_ID = re.compile(r"[a-zA-Z0-9_-]{1,64}")+_MAX_RESPONSE = 1024 * 1024+_IMAGE = "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404"+_BOOT = (+ "set -eu; if [ ! -x /usr/sbin/sshd ]; then apt-get update; "+ "apt-get install -y openssh-server; fi; "+ "mkdir -p /root/.ssh /run/sshd; chmod 700 /root/.ssh; "+ "printf '%s\\n' \"$PUBLIC_KEY\" > /root/.ssh/authorized_keys; "+ "chmod 600 /root/.ssh/authorized_keys; ssh-keygen -A; exec /usr/sbin/sshd -D -e"+)+++def _notice(message):+ sys.stderr.write(message + "\n")+ sys.stderr.flush()+++class _ControllerError(RuntimeError):+ """Only fixed, controller-owned diagnostics are safe to show to the user."""+++class _APIError(_ControllerError):+ def __init__(self, method: str, status: int) -> None:+ super().__init__(f"RunPod API {method} failed: HTTP {status}")+ # A server/request timeout can still have allocated a Pod. Reconcile it.+ self.create_rejected = method == "POST" and 400 <= status < 500 and status != 408+++class _NoRedirect(urllib.request.HTTPRedirectHandler):+ def redirect_request(self, req, fp, code, msg, headers, newurl):+ return None+++def _request_json(api_key, path, *, method="GET", payload=None):+ data = None if payload is None else json.dumps(payload).encode("utf-8")+ request = urllib.request.Request(_API_BASE + path, data=data, method=method)+ request.add_header("Authorization", "Bearer " + api_key)+ request.add_header("Content-Type", "application/json")+ try:+ with urllib.request.build_opener(_NoRedirect()).open(request, timeout=60) as response:+ raw = response.read(_MAX_RESPONSE + 1)+ if len(raw) > _MAX_RESPONSE:+ raise _ControllerError("RunPod API response exceeds 1 MB")+ if not raw and method == "DELETE":+ return None+ parsed = json.loads(raw)+ if not isinstance(parsed, (dict, list)):+ raise _ControllerError("RunPod API returned an invalid JSON shape")+ return parsed+ except urllib.error.HTTPError as exc:+ if exc.code == 404 and method in {"GET", "DELETE"} and path.startswith("/pods/"):+ return None+ raise _APIError(method, exc.code) from None+ except (OSError, ValueError, http.client.HTTPException):+ raise _ControllerError("RunPod API transport or JSON response failed") from None+++def _launch_payload(public_key, name):+ return {+ "name": name,+ "imageName": _IMAGE,+ "cloudType": "SECURE",+ "computeType": "GPU",+ "gpuTypeIds": [_GPU_ID],+ "gpuCount": 1,+ "containerDiskInGb": 80,+ "supportPublicIp": True,+ "ports": ["22/tcp"],+ "env": {"PUBLIC_KEY": public_key},+ "dockerEntrypoint": ["/bin/bash", "-lc"],+ "dockerStartCmd": [_BOOT],+ }+++def _valid_id(value):+ if not isinstance(value, str) or not _POD_ID.fullmatch(value):+ raise _ControllerError("RunPod returned an invalid Pod ID; check the console")+ return value+++def _wait_for_ssh(api_key, pod_id):+ path = "/pods/" + _valid_id(pod_id)+ deadline = time.monotonic() + 900+ while time.monotonic() < deadline:+ record = _request_json(api_key, path)+ if not isinstance(record, dict) or record.get("desiredStatus") in {"EXITED", "TERMINATED"}:+ raise _ControllerError("RunPod is unavailable before SSH startup")+ ip = record.get("publicIp")+ port = (record.get("portMappings") or {}).get("22")+ if ip and isinstance(port, int) and not isinstance(port, bool) and 1 <= port <= 65535:+ address = ipaddress.ip_address(ip)+ # The API chooses the destination; refuse local/metadata addresses.+ if address.version != 4 or not address.is_global:+ raise _ControllerError("RunPod did not return a public SSH address")+ return str(address), port+ time.sleep(5)+ raise _ControllerError("RunPod did not expose SSH within 15 minutes")+++def _train_and_copy(ip, port, private_key):+ with tempfile.TemporaryDirectory(prefix="soup-runpod-ssh-") as directory:+ known_hosts = pathlib.Path(directory) / "known_hosts"+ _run_ssh_session(ip, port, private_key, known_hosts)+++def _run_ssh_session(ip: str, port: int, private_key: str, known_hosts: pathlib.Path) -> None:+ common = [+ "-i",+ private_key,+ "-o",+ "BatchMode=yes",+ "-o",+ "IdentitiesOnly=yes",+ "-o",+ f'UserKnownHostsFile="{known_hosts.as_posix()}"',+ "-o",+ "StrictHostKeyChecking=accept-new",+ "-o",+ "ConnectTimeout=15",+ "-o",+ "ServerAliveInterval=30",+ "-o",+ "ServerAliveCountMax=6",+ ]+ script = (+ "import base64, pathlib, subprocess, sys\n"+ "root = pathlib.Path('/workspace/soup'); root.mkdir(parents=True, exist_ok=True)\n"+ "import os; os.chdir(root)\n"+ f"pathlib.Path('soup.yaml').write_bytes(base64.b64decode({_CONFIG_B64!r}))\n"+ "subprocess.run([sys.executable, '-m', 'pip', 'install', "+ f"'soup-cli[train]=={_SOUP_VERSION}'], "+ "check=True, stdout=sys.stderr)\n"+ "with open('train.log', 'w') as log:\n"+ " result = subprocess.run(['soup', 'train', '--config', 'soup.yaml', '--yes'], "+ "stdout=log, stderr=subprocess.STDOUT)\n"+ "sys.exit(result.returncode)\n"+ )+ # A readiness mapping can precede sshd. Retry a harmless readiness command,+ # never the training command: reconnecting must not start training twice.+ deadline = time.monotonic() + 300+ while True:+ ready = subprocess.run(+ ["ssh", *common, "-p", str(port), f"root@{ip}", "true"], capture_output=True, timeout=30+ )+ if ready.returncode == 0:+ break+ if time.monotonic() >= deadline:+ raise _ControllerError("RunPod SSH did not become ready within five minutes")+ time.sleep(5)+ result = subprocess.run(+ ["ssh", *common, "-p", str(port), f"root@{ip}", "python3 -"],+ input=script,+ text=True,+ capture_output=True,+ timeout=25 * 3600,+ )+ if result.returncode != 0:+ raise _ControllerError("RunPod remote training failed; no successful artifact copy claimed")+ destination = _local_destination()+ destination.mkdir(parents=True, exist_ok=True)+ copied = subprocess.run(+ [+ "scp",+ *common,+ "-P",+ str(port),+ "-r",+ "--",+ f"root@{ip}:/workspace/soup/{_OUTPUT}",+ str(destination),+ ],+ capture_output=True,+ timeout=900,+ )+ if copied.returncode != 0:+ raise _ControllerError("RunPod artifact copy failed")+++def _local_destination():+ """Recheck the download path before allocation and immediately before SCP."""+ root = os.path.realpath(os.getcwd())+ target = pathlib.Path(_OUTPUT)+ if (target.is_absolute() or ".." in target.parts or str(target) == "."+ or any(part.startswith("-") for part in target.parts)):+ raise _ControllerError("RunPod output must stay under cwd without option-like components")+ candidate = pathlib.Path.cwd()+ for part in target.parts:+ candidate = candidate / part+ if os.path.lexists(candidate):+ info = os.lstat(candidate)+ if stat.S_ISLNK(info.st_mode) or getattr(info, "st_file_attributes", 0) & 0x400:+ raise _ControllerError("RunPod output must not traverse symlinks or junctions")+ if os.path.commonpath([root, os.path.realpath(target)]) != root:+ raise _ControllerError("RunPod output must stay under cwd")+ if os.path.lexists(target):+ raise _ControllerError("RunPod output already exists; choose a fresh output directory")+ return target.parent+++def _terminate(api_key, pod_id):+ path = "/pods/" + _valid_id(pod_id)+ deadline = time.monotonic() + 300+ while time.monotonic() < deadline:+ try:+ _request_json(api_key, path, method="DELETE")+ # A successful DELETE alone is not evidence of termination.+ if _request_json(api_key, path) is None:+ return+ except RuntimeError:+ pass+ time.sleep(5)+ raise _ControllerError("RunPod deletion was not confirmed within five minutes")+++def _recover_created(api_key, name):+ """A lost POST response must not cause another POST or delete unrelated Pods."""+ deadline = time.monotonic() + 60+ while time.monotonic() < deadline:+ records = _request_json(api_key, "/pods")+ if not isinstance(records, list):+ raise _ControllerError("RunPod could not reconcile the launch; check the console")+ found = [+ item["id"] for item in records if isinstance(item, dict) and item.get("name") == name+ ]+ if found:+ return [_valid_id(item) for item in found]+ time.sleep(5)+ raise _ControllerError("RunPod launch result is unknown; check the console")+++def main():+ if not all((_GPU_ID, _CONFIG_B64, _OUTPUT, _SOUP_VERSION)):+ _notice("Generate a RunPod plan with soup train --cloud runpod first.")+ return 2+ if not all(shutil.which(command) for command in ("ssh", "scp")):+ _notice("OpenSSH ssh and scp must be installed before RunPod submission.")+ return 2+ try:+ _local_destination()+ except (OSError, RuntimeError):+ _notice("RunPod needs a fresh local output path under cwd, without symlinks.")+ return 2+ api_key = os.environ.get("RUNPOD_API_KEY")+ private_key = os.path.expanduser(os.environ.get("RUNPOD_SSH_PRIVATE_KEY", ""))+ if not api_key or not private_key or not pathlib.Path(private_key).is_file():+ _notice("Set RUNPOD_API_KEY and RUNPOD_SSH_PRIVATE_KEY to a readable key file.")+ return 2+ try:+ public_key = pathlib.Path(private_key + ".pub").read_text(encoding="utf-8").strip()+ if not re.fullmatch(r"ssh-(?:ed25519|rsa) [A-Za-z0-9+/=]+(?: [^\r\n]+)?", public_key):+ raise ValueError("invalid public key")+ except (OSError, ValueError):+ _notice("A matching .pub file with one OpenSSH public key is required.")+ return 2+ name = "soup-" + uuid.uuid4().hex+ pod_id = None+ launched = False+ rc = 1+ try:+ launched = True+ response = _request_json(+ api_key, "/pods", method="POST", payload=_launch_payload(public_key, name)+ )+ pod_id = _valid_id(response.get("id") if isinstance(response, dict) else None)+ _notice("RunPod Pod launched: " + pod_id)+ ip, port = _wait_for_ssh(api_key, pod_id)+ _train_and_copy(ip, port, private_key)+ rc = 0+ except (Exception, KeyboardInterrupt, SystemExit) as exc:+ if isinstance(exc, _APIError) and exc.create_rejected:+ launched = False+ _notice(str(exc) if isinstance(exc, _ControllerError) else (+ "RunPod training/controller failed; cleaning up the Pod."+ ))+ finally:+ # Repeated Ctrl+C must not abandon cleanup of a billable resource.+ previous = signal.signal(signal.SIGINT, signal.SIG_IGN)+ try:+ ids = [pod_id] if pod_id else (_recover_created(api_key, name) if launched else [])+ for owned_id in ids:+ _terminate(api_key, owned_id)+ _notice("RunPod termination verified." if ids else (+ "RunPod create request rejected; no Pod to clean up."+ ))+ except Exception:+ _notice("CRITICAL: RunPod cleanup unconfirmed; check the console for " + name)+ rc = 1+ finally:+ signal.signal(signal.SIGINT, previous)+ return rc+++if __name__ == "__main__":+ sys.exit(main())diff --git a/src/soup_cli/cloud/runpod.py b/src/soup_cli/cloud/runpod.pynew file mode 100644index 00000000..e0035bd8--- /dev/null+++ b/src/soup_cli/cloud/runpod.py@@ -0,0 +1,220 @@+"""RunPod plans and credential-gated submission; no provider SDK is imported."""++from __future__ import annotations++import base64+import os+import re+import shlex+import sys+import types+from collections.abc import Callable, Mapping+from pathlib import Path, PurePosixPath++from soup_cli.cloud._common import (+ _MAX_CONFIG_BYTES,+ _MAX_VERSION_LEN,+ _VERSION_RE,+ CloudPlan,+ validate_choice,+ validate_path_shape,+ write_cloud_stub,+)++SUPPORTED_CLOUDS = frozenset({"runpod"})+# REST v1 gpuTypeIds, verified against the provider's published OpenAPI schema.+_GPU_IDS = types.MappingProxyType(+ {+ "a100": "NVIDIA A100-SXM4-80GB",+ "h100": "NVIDIA H100 80GB HBM3",+ "a40": "NVIDIA A40",+ "rtx3090": "NVIDIA GeForce RTX 3090",+ "rtx4090": "NVIDIA GeForce RTX 4090",+ }+)+SUPPORTED_GPUS = frozenset(_GPU_IDS)+_RUNPOD_SUBMIT_OVERRIDE: Callable[[CloudPlan], int] | None = None+_CREDENTIAL_FIELDS = frozenset(+ {+ "api_key",+ "token",+ "access_token",+ "auth_token",+ "hf_token",+ "password",+ "private_key",+ "secret",+ "wandb_api_key",+ "runpod_api_key",+ "aws_secret_access_key",+ }+)+++def validate_cloud(name: object) -> str:+ return validate_choice(name, "cloud", SUPPORTED_CLOUDS)+++def validate_gpu(gpu: object) -> str:+ return validate_choice(gpu, "gpu", SUPPORTED_GPUS)+++def _validate_config(config_yaml: str) -> None:+ import yaml++ from soup_cli.utils.yaml_limits import load_yaml_limited++ if not isinstance(config_yaml, str):+ raise ValueError("config must be a string")+ key = os.environ.get("RUNPOD_API_KEY")+ if key and key in config_yaml:+ raise ValueError("config must not embed provider credentials")+ try:+ content = load_yaml_limited(config_yaml, "RunPod config", max_bytes=_MAX_CONFIG_BYTES)+ except yaml.YAMLError:+ raise ValueError("config must be valid YAML; credentials must stay local") from None++ def walk(value: object, active: set[int]) -> None:+ if not isinstance(value, (dict, list)):+ return+ if id(value) in active:+ raise ValueError("recursive config aliases are not supported")+ active.add(id(value))+ try:+ if isinstance(value, dict):+ for name, item in value.items():+ if str(name).lower() in _CREDENTIAL_FIELDS and item not in (None, ""):+ raise ValueError("inline config credentials are not supported")+ walk(item, active)+ else:+ for item in value:+ walk(item, active)+ finally:+ active.remove(id(value))++ walk(content, set())+++def render_runpod_stub(+ config_yaml: str,+ *,+ gpu: str,+ output_dir: str,+ soup_version: str,+) -> str:+ """Render a self-contained local controller; config is embedded only as data."""+ gpu = validate_gpu(gpu)+ validate_path_shape(output_dir, "output_dir")+ output = PurePosixPath(output_dir)+ if (+ output.is_absolute()+ or ".." in output.parts+ or any(part.startswith("-") for part in output.parts)+ or str(output) == "."+ or not re.fullmatch(r"[A-Za-z0-9._/-]+", output_dir)+ ):+ raise ValueError("output_dir must be a simple relative path under cwd")+ if (+ not isinstance(soup_version, str)+ or len(soup_version) > _MAX_VERSION_LEN+ or not _VERSION_RE.fullmatch(soup_version)+ ):+ raise ValueError("soup_version must be a safe package version")+ _validate_config(config_yaml)+ encoded = base64.b64encode(config_yaml.encode("utf-8")).decode("ascii")+ constants = (+ f"_GPU_ID = {_GPU_IDS[gpu]!r}\n_CONFIG_B64 = {encoded!r}\n"+ f"_OUTPUT = {str(output)!r}\n_SOUP_VERSION = {soup_version!r}\n"+ )+ controller = Path(__file__).with_name("_runpod_controller.py").read_text(encoding="utf-8")+ return constants + controller+++def plan_runpod_run(+ config_path: str,+ *,+ gpu: str,+ output_dir: str,+ soup_version: str,+ stub_path: str = "soup_runpod_app.py",+) -> CloudPlan:+ from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink++ for value, field in (+ (config_path, "--config"),+ (output_dir, "output_dir"),+ (stub_path, "stub_path"),+ ):+ validate_path_shape(value, field)+ enforce_under_cwd_and_no_symlink(value, field)+ with open(config_path, encoding="utf-8") as source:+ config_yaml = source.read(_MAX_CONFIG_BYTES + 1)+ if os.path.lexists(output_dir):+ raise ValueError("output_dir already exists; choose a fresh directory for retrieval")+ stub = render_runpod_stub(+ config_yaml,+ gpu=gpu,+ output_dir=output_dir,+ soup_version=soup_version,+ )+ return CloudPlan(+ "runpod", validate_gpu(gpu), output_dir, stub_path, stub,+ f"python -I {shlex.quote(stub_path)}",+ )+++def write_stub(plan: CloudPlan) -> str:+ return write_cloud_stub(plan)+++_CONTROLLER_ENV_NAMES: frozenset[str] = frozenset({+ "RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY", "PATH", "HOME", "USERPROFILE", "SYSTEMROOT",+ "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY",+ "http_proxy", "https_proxy", "all_proxy", "no_proxy",+ "TEMP", "TMP", "TMPDIR", "SYSTEMDRIVE", "WINDIR", "SSH_AUTH_SOCK",+})+++def _controller_environment(+ environ: Mapping[str, str], *, case_insensitive: bool | None = None,+) -> dict[str, str]:+ """Keep the controller's credentials and OS/network settings, with OS name rules."""+ if case_insensitive is None:+ case_insensitive = os.name == "nt"+ allowed = ({name.upper() for name in _CONTROLLER_ENV_NAMES}+ if case_insensitive else _CONTROLLER_ENV_NAMES)+ return {+ name: value for name, value in environ.items()+ if (name.upper() if case_insensitive else name) in allowed+ }+++def submit_runpod_run(plan: CloudPlan, *, env: Mapping | None = None) -> int:+ """Keep waiting through interrupts so the controller can finish its cleanup."""+ if not isinstance(plan, CloudPlan) or plan.cloud != "runpod":+ raise TypeError("plan must be a RunPod CloudPlan")+ if _RUNPOD_SUBMIT_OVERRIDE is not None:+ return _RUNPOD_SUBMIT_OVERRIDE(plan)+ environ = dict(env if env is not None else os.environ)+ missing = [+ name for name in ("RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY") if not environ.get(name)+ ]+ if missing:+ raise RuntimeError("RunPod not configured. Set " + ", ".join(missing))+ environ = _controller_environment(environ)+ import subprocess++ proc = None+ while True:+ try:+ if proc is None:+ proc = subprocess.Popen([sys.executable, "-I", plan.stub_path], env=environ)+ return proc.wait()+ except BaseException as exc:+ if proc is None or not isinstance(exc, (KeyboardInterrupt, SystemExit)):+ raise+ try:+ sys.stderr.write("Interrupted: waiting for RunPod cleanup to finish.\n")+ sys.stderr.flush()+ except BaseException:+ passdiff --git a/src/soup_cli/commands/train.py b/src/soup_cli/commands/train.pyindex ebb94d8e..4f3eb3ac 100644--- a/src/soup_cli/commands/train.py+++ b/src/soup_cli/commands/train.py@@ -843,7 +843,7 @@ def train("--cloud",help=("Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: "- "modal, lambda (runpod is planned). Renders a cloud app stub from the config "+ "modal, lambda, runpod. Renders a cloud app stub from the config ""(plan-only); use --cloud-submit to submit live."),),@@ -1064,10 +1064,7 @@ def train(cloud = cloud.lower()if cloud == "runpod":- console.print(- "[yellow]RunPod cloud training is not yet live; use --cloud modal or lambda.[/]"- )- raise typer.Exit(2)+ from soup_cli.cloud import runpod as cloud_modelif cloud == "modal":from soup_cli.cloud import modal as cloud_modelif cloud == "lambda":@@ -1075,7 +1072,7 @@ def train(else:console.print(f"[red]Invalid --cloud:[/] {markup_escape(cloud)}. "- "Supported: modal, lambda (runpod is planned)."+ "Supported: modal, lambda, runpod.")raise typer.Exit(2)diff --git a/tests/test_issue1547_outbound_http_sinks.py b/tests/test_issue1547_outbound_http_sinks.pyindex 9697e1ff..5c8edc46 100644--- a/tests/test_issue1547_outbound_http_sinks.py+++ b/tests/test_issue1547_outbound_http_sinks.py@@ -159,6 +159,14 @@ class DeclaredSink:# - gate: (caller_func, gate_func) where caller_func is AST-verified to call gate_func# - reason: explicit justification for exemptions (fixed host, loopback only, operator env)DECLARED_SINKS: dict[tuple[str, str], DeclaredSink] = {+ ("cloud/_runpod_controller.py", "_request_json"): DeclaredSink(+ calls=("urllib.request.build_opener",),+ gate=None,+ reason=(+ "fixed host: _API_BASE is hardcoded to https://rest.runpod.io/v1; "+ "the standalone controller refuses redirects and caps response reads"+ ),+ ),("commands/generate.py", "_generate_openai"): DeclaredSink(calls=("soup_cli.utils.http_response.post_capped",),gate=("_generate_openai", "refuse_private_ip_literal"),diff --git a/tests/test_issue264_cloud.py b/tests/test_issue264_cloud.pyindex 4caf15de..40738721 100644--- a/tests/test_issue264_cloud.py+++ b/tests/test_issue264_cloud.py@@ -194,7 +194,7 @@ class TestSubmitLambdaLabsRun:class TestTrainCloudCliNew:- def test_cloud_runpod_refuses_not_yet_live(self, tmp_path, monkeypatch):+ def test_cloud_runpod_refuses_unknown_gpu(self, tmp_path, monkeypatch):monkeypatch.chdir(tmp_path)(tmp_path / "soup.yaml").write_text(_SOUP_YAML, encoding="utf-8")from typer.testing import CliRunner@@ -202,13 +202,13 @@ class TestTrainCloudCliNew:from soup_cli.cli import appresult = CliRunner().invoke(- app, ["train", "--config", "soup.yaml", "--cloud", "runpod", "--gpu", "rtx-4090"]+ app, ["train", "--config", "soup.yaml", "--cloud", "runpod", "--gpu", "tpu"])assert result.exit_code == 2, (result.output, repr(result.exception))assert not (tmp_path / "soup_runpod_app.py").exists()txt = _strip_ansi(result.output)- assert "not yet live" in txt- assert "--cloud modal or lambda" in txt+ assert "Invalid --cloud / --gpu" in txt+ assert "rtx4090" in txtdef test_cloud_lambda_plan_only(self, tmp_path, monkeypatch):monkeypatch.chdir(tmp_path)@@ -239,13 +239,12 @@ class TestTrainCloudCliNew:assert result.exit_code == 0, (result.output, repr(result.exception))assert (tmp_path / "soup_lambda_app.py").exists()- # And verify uppercase RUNPOD is also refused+ # RunPod uses the same case-insensitive provider dispatch.res_runpod = CliRunner().invoke(app, ["train", "--config", "soup.yaml", "--cloud", "RUNPOD", "--gpu", "a100"])- assert res_runpod.exit_code == 2- assert not (tmp_path / "soup_runpod_app.py").exists()- assert "not yet live" in _strip_ansi(res_runpod.output)+ assert res_runpod.exit_code == 0, (res_runpod.output, repr(res_runpod.exception))+ assert (tmp_path / "soup_runpod_app.py").exists()class TestCloudNoSecrets:diff --git a/tests/test_runpod_cloud.py b/tests/test_runpod_cloud.pynew file mode 100644index 00000000..8ebdc1ff--- /dev/null+++ b/tests/test_runpod_cloud.py@@ -0,0 +1,292 @@+"""RunPod planning and generated controller behavior, without provider traffic."""++import base64+import io+import json+import subprocess+import types+import urllib.error+from pathlib import Path++import pytest+from typer.testing import CliRunner++CONFIG = "base: public/model\ndata: {train: public/dataset}\noutput: out\n"+++def _backend():+ from soup_cli.cloud import runpod++ return runpod+++def _controller():+ text = _backend().render_runpod_stub(+ CONFIG, gpu="a100", output_dir="out", soup_version="0.76.0"+ )+ namespace = {"__name__": "test_controller"}+ exec(compile(text, "soup_runpod_app.py", "exec"), namespace)+ namespace["shutil"] = types.SimpleNamespace(which=lambda command: command)+ return namespace+++def credentials(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ private = tmp_path / "id_test"+ private.write_text("not-a-real-private-key", encoding="utf-8")+ Path(str(private) + ".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(private))+ return private++[email protected]("value", [True, 1, "", "tpu", "a100\x00", "x" * 100])+def test_closed_gpu_allowlist(value):+ with pytest.raises(ValueError, match="gpu"):+ _backend().validate_gpu(value)+++def test_render_is_data_only_and_uses_documented_rest_payload(credentials):+ ns = _controller()+ assert base64.b64decode(ns["_CONFIG_B64"]).decode() == CONFIG+ payload = ns["_launch_payload"]("ssh-ed25519 AAAATEST fixture", "soup-test")+ assert payload["gpuTypeIds"] == ["NVIDIA A100-SXM4-80GB"]+ assert payload["gpuCount"] == 1+ assert payload["ports"] == ["22/tcp"]+ assert payload["env"] == {"PUBLIC_KEY": "ssh-ed25519 AAAATEST fixture"}+ assert "test-provider-secret" not in json.dumps(payload)+ assert "not-a-real-private-key" not in json.dumps(payload)+ assert payload["dockerEntrypoint"] == ["/bin/bash", "-lc"]+ assert "sshd" in payload["dockerStartCmd"][0]+++ "output", ["/tmp/out", "../out", "a/../../out", ".", "out space", "a;exit"]+)+def test_unsafe_remote_outputs_are_refused(output):+ with pytest.raises(ValueError, match="output_dir"):+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir=output, soup_version="0.76.0")+++ "config", ["hf_token: secret\n", "x: {api_key: secret}\n", "# test-provider-secret\n"]+)+def test_inline_secrets_are_refused(config, credentials):+ with pytest.raises(ValueError, match="credential"):+ _backend().render_runpod_stub(config, gpu="a100", output_dir="out", soup_version="0.76.0")+++def test_malformed_config_is_refused_without_echoing_its_content():+ with pytest.raises(ValueError, match="valid YAML") as error:+ _backend().render_runpod_stub(+ "private-value: [", gpu="a100", output_dir="out", soup_version="0.76.0",+ )+ assert "private-value" not in str(error.value)+++def test_modest_alias_reuse_is_preserved_as_config_data():+ config = CONFIG + "settings: &settings {epochs: 1}\ncopy: *settings\n"+ stub = _backend().render_runpod_stub(+ config, gpu="a100", output_dir="out", soup_version="0.76.0",+ )+ namespace = {"__name__": "test_controller"}+ exec(compile(stub, "soup_runpod_app.py", "exec"), namespace)+ assert base64.b64decode(namespace["_CONFIG_B64"]).decode() == config++[email protected]("version", ["x; echo bad", "", "x\n", True])+def test_bad_package_version_is_refused(version):+ with pytest.raises(ValueError, match="soup_version"):+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir="out", soup_version=version)+++def test_plan_rejects_path_escape_and_writes_atomically(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ backend = _backend()+ for output in ("out", "../outside"):+ if output == "out":+ plan = backend.plan_runpod_run(+ "soup.yaml", gpu="a100", output_dir=output, soup_version="0.76.0"+ )+ backend.write_stub(plan)+ assert (tmp_path / plan.stub_path).read_text(encoding="utf-8") == plan.stub_text+ else:+ with pytest.raises(ValueError, match="output_dir"):+ backend.plan_runpod_run(+ "soup.yaml", gpu="a100", output_dir=output, soup_version="0.76.0"+ )++[email protected]_symlink+def test_config_and_output_symlinks_are_refused(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ target = tmp_path / "real.yaml"+ target.write_text(CONFIG, encoding="utf-8")+ (tmp_path / "soup.yaml").symlink_to(target)+ with pytest.raises(ValueError):+ _backend().plan_runpod_run("soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0")+ (tmp_path / "soup.yaml").unlink()+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ (tmp_path / "actual").mkdir()+ (tmp_path / "out").symlink_to(tmp_path / "actual", target_is_directory=True)+ with pytest.raises(ValueError):+ _backend().plan_runpod_run("soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0")+++ "failure",+ [+ None,+ RuntimeError("secret from training"),+ KeyboardInterrupt(),+ subprocess.TimeoutExpired("ssh", 1),+ SystemExit(2),+ ],+)+def test_controller_terminates_and_confirms_even_after_failure(credentials, failure):+ ns = _controller()+ calls = []++ def request(key, path, *, method="GET", payload=None):+ calls.append((method, path))+ if method == "POST":+ return {"id": "pod-1"}+ if method == "GET":+ return None++ def training(*args):+ if failure is not None:+ raise failure++ ns["_request_json"] = request+ ns["_wait_for_ssh"] = lambda *args: ("192.0.2.1", 2222)+ ns["_train_and_copy"] = training+ ns["_notice"] = lambda message: calls.append(("notice", message))+ assert ns["main"]() == (0 if failure is None else 1)+ assert ("DELETE", "/pods/pod-1") in calls+ assert ("GET", "/pods/pod-1") in calls+ assert "secret from training" not in str(calls)+++def test_cleanup_failure_cannot_report_success(credentials):+ ns = _controller()+ ns["_request_json"] = lambda *a, **kw: {"id": "pod-1"}+ ns["_wait_for_ssh"] = lambda *args: ("192.0.2.1", 2222)+ ns["_train_and_copy"] = lambda *args: None+ ns["_terminate"] = lambda *args: (_ for _ in ()).throw(RuntimeError("secret"))+ assert ns["main"]() == 1+++def test_uncertain_launch_is_reconciled_by_unique_name(credentials):+ ns = _controller()+ calls = []+ name = []++ def request(key, path, *, method="GET", payload=None):+ calls.append((method, path))+ if method == "POST":+ name.append(payload["name"])+ raise RuntimeError("lost create response")+ if path == "/pods":+ return [{"id": "pod-1", "name": name[0]}, {"id": "other", "name": "unrelated"}]+ return None++ ns["_request_json"] = request+ assert ns["main"]() == 1+ assert ("DELETE", "/pods/pod-1") in calls+ assert ("DELETE", "/pods/other") not in calls+++def test_rest_auth_uses_header_and_refuses_redirects(monkeypatch):+ ns = _controller()+ seen = []++ class Response(io.BytesIO):+ status = 200++ class Opener:+ def open(self, request, timeout):+ seen.append((request, timeout))+ return Response(b'{"id":"pod-1"}')++ def build_opener(*handlers):+ assert len(handlers) == 1+ assert isinstance(handlers[0], ns["_NoRedirect"])+ assert handlers[0].redirect_request(+ None, None, None, None, None, "https://evil.invalid",+ ) is None+ return Opener()++ monkeypatch.setattr(ns["urllib"].request, "build_opener", build_opener)+ assert ns["_request_json"]("provider-secret", "/pods/pod-1") == {"id": "pod-1"}+ request, timeout = seen[0]+ assert request.full_url == "https://rest.runpod.io/v1/pods/pod-1"+ assert request.get_header("Authorization") == "Bearer provider-secret"+ assert "provider-secret" not in request.full_url+ assert timeout == 60+ assert (+ ns["_NoRedirect"]().redirect_request(None, None, None, None, None, "https://evil.invalid")+ is None+ )++[email protected]("status", [401, 500, 302])+def test_rest_error_details_never_expose_credentials(monkeypatch, status):+ ns = _controller()++ class Opener:+ def open(self, request, timeout):+ raise urllib.error.HTTPError(+ request.full_url, status, "secret", {}, io.BytesIO(b"secret")+ )++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ with pytest.raises(RuntimeError, match=f"HTTP {status}") as error:+ ns["_request_json"]("secret", "/pods/pod-1")+ assert "secret" not in str(error.value)+++def test_training_failure_or_copy_failure_is_not_success(credentials, monkeypatch, tmp_path):+ ns = _controller()+ monkeypatch.chdir(tmp_path)+ calls = []++ def run(argv, **kwargs):+ calls.append((argv, kwargs))+ return subprocess.CompletedProcess(argv, 0 if argv[0] == "ssh" else 1)++ monkeypatch.setattr(ns["subprocess"], "run", run)+ with pytest.raises(RuntimeError, match="copy"):+ ns["_train_and_copy"]("192.0.2.1", 2222, str(credentials))+ assert [item[0][0] for item in calls] == ["ssh", "ssh", "scp"]+ assert "-P" in calls[2][0] and "2222" in calls[2][0]+ script = calls[1][1]["input"]+ assert "test-provider-secret" not in script+ assert "soup-cli[train]==0.76.0" in script++[email protected]("submit,expected", [(False, 0), (True, 7)])+def test_real_cli_plan_and_submit_seam(tmp_path, monkeypatch, submit, expected):+ from soup_cli.cli import app++ monkeypatch.chdir(tmp_path)+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ plans = []+ monkeypatch.setattr(_backend(), "_RUNPOD_SUBMIT_OVERRIDE", lambda plan: plans.append(plan) or 7)+ args = ["train", "--cloud", "runpod", "--gpu", "a100", "--config", "soup.yaml"]+ result = CliRunner().invoke(app, args + (["--cloud-submit"] if submit else []))+ assert result.exit_code == expected, (result.output, repr(result.exception))+ assert bool(plans) == submit+ assert (tmp_path / "soup_runpod_app.py").is_file()+++def test_submit_requires_credentials_before_spawning(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ plan = _backend().plan_runpod_run(+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0"+ )+ with pytest.raises(RuntimeError, match="RUNPOD_API_KEY"):+ _backend().submit_runpod_run(plan, env={})diff --git a/tests/test_runpod_controller.py b/tests/test_runpod_controller.pynew file mode 100644index 00000000..e6f91133--- /dev/null+++ b/tests/test_runpod_controller.py@@ -0,0 +1,176 @@+"""Lifecycle boundaries exercised by running rendered controller functions."""++import io+import subprocess+import types+import urllib.error+from pathlib import Path++import pytest++from tests.test_runpod_cloud import CONFIG, _backend, _controller+++def credentials(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ key = tmp_path / "id_test"+ key.write_text("not-a-real-private-key", encoding="utf-8")+ key.with_suffix(".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(key))+ return key+++def _clock(ns, step=1):+ ticks = iter(range(0, 100000, step))+ sleeps = []+ ns["time"] = types.SimpleNamespace(monotonic=lambda: next(ticks), sleep=sleeps.append)+ return sleeps+++def test_delete_polls_past_stopped_state_until_actual_absence():+ ns = _controller()+ reads = iter([{"desiredStatus": "EXITED"}, None])+ calls = []+ sleeps = _clock(ns)++ def request(*args, method="GET", **kwargs):+ calls.append(method)+ return None if method == "DELETE" else next(reads)++ ns["_request_json"] = request+ ns["_terminate"]("secret", "pod-1")+ assert calls == ["DELETE", "GET", "DELETE", "GET"]+ assert sleeps == [5]+++def test_deletion_timeout_is_a_failure():+ ns = _controller()+ _clock(ns, 100)+ ns["_request_json"] = lambda *args, **kwargs: {"desiredStatus": "EXITED"}+ with pytest.raises(RuntimeError, match="not confirmed"):+ ns["_terminate"]("secret", "pod-1")+++ "body",+ [b"", b"null", b"bad", b"x" * (1024 * 1024 + 1)],+ ids=["empty", "null", "malformed", "oversized"],+)+def test_only_http_404_can_establish_absence(monkeypatch, body):+ ns = _controller()++ class Opener:+ def open(self, request, timeout):+ return io.BytesIO(body)++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ with pytest.raises(RuntimeError):+ ns["_request_json"]("secret", "/pods/pod-1")+++def test_rest_404_get_and_empty_delete_are_accepted(monkeypatch):+ ns = _controller()++ class Opener:+ def open(self, request, timeout):+ if request.method == "DELETE":+ return io.BytesIO(b"")+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ assert ns["_request_json"]("secret", "/pods/pod-1") is None+ assert ns["_request_json"]("secret", "/pods/pod-1", method="DELETE") is None+ with pytest.raises(RuntimeError, match="HTTP 404"):+ ns["_request_json"]("secret", "/pods", method="POST", payload={})+++ "record",+ [+ None,+ {"desiredStatus": "EXITED"},+ {"publicIp": "127.0.0.1", "portMappings": {"22": 1234}},+ {"publicIp": "169.254.169.254", "portMappings": {"22": 1234}},+ ],+)+def test_ssh_refuses_missing_stopped_or_private_addresses(record):+ ns = _controller()+ ns["_request_json"] = lambda *args: record+ with pytest.raises(RuntimeError):+ ns["_wait_for_ssh"]("secret", "pod-1")+++def test_ssh_waits_for_real_port_and_has_a_deadline():+ ns = _controller()+ reads = iter([{}, {"publicIp": "8.8.8.8", "portMappings": {"22": 2222}}])+ ns["_request_json"] = lambda *args: next(reads)+ sleeps = _clock(ns)+ assert ns["_wait_for_ssh"]("secret", "pod-1") == ("8.8.8.8", 2222)+ assert sleeps == [5]+ _clock(ns, 1000)+ with pytest.raises(RuntimeError, match="15 minutes"):+ ns["_wait_for_ssh"]("secret", "pod-1")+++def test_training_nonzero_is_not_copied_or_retried(monkeypatch, credentials):+ ns = _controller()+ calls = []++ def run(argv, **kwargs):+ calls.append(argv)+ return subprocess.CompletedProcess(argv, 1 if "input" in kwargs else 0)++ monkeypatch.setattr(ns["subprocess"], "run", run)+ with pytest.raises(RuntimeError, match="training failed"):+ ns["_train_and_copy"]("192.0.2.1", 2222, str(credentials))+ assert [argv[0] for argv in calls] == ["ssh", "ssh"]+++def test_submit_keeps_one_controller_through_repeated_interrupts(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ plan = _backend().plan_runpod_run(+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0"+ )+ waits = [KeyboardInterrupt(), SystemExit(2), 0]+ spawns = []++ class Process:+ def wait(self):+ value = waits.pop(0)+ if isinstance(value, BaseException):+ raise value+ return value++ monkeypatch.setattr(subprocess, "Popen", lambda *a, **kw: spawns.append((a, kw)) or Process())+ assert (+ _backend().submit_runpod_run(+ plan, env={"RUNPOD_API_KEY": "test", "RUNPOD_SSH_PRIVATE_KEY": "key"}+ )+ == 0+ )+ assert len(spawns) == 1+ assert waits == []+++def test_missing_ssh_or_existing_output_never_launches(credentials):+ ns = _controller()+ calls = []+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))+ ns["shutil"].which = lambda command: None+ assert ns["main"]() == 2+ ns["shutil"].which = lambda command: command+ Path("out").mkdir()+ assert ns["main"]() == 2+ assert not calls++[email protected]_symlink+def test_local_copy_destination_refuses_symlinks(credentials, tmp_path):+ ns = _controller()+ (tmp_path / "real").mkdir()+ (tmp_path / "out").symlink_to(tmp_path / "real", target_is_directory=True)+ with pytest.raises(RuntimeError, match="symlinks"):+ ns["_local_destination"]()diff --git a/tests/test_runpod_hardening.py b/tests/test_runpod_hardening.pynew file mode 100644index 00000000..7614dfe7--- /dev/null+++ b/tests/test_runpod_hardening.py@@ -0,0 +1,489 @@+"""Run rendered RunPod controllers through failure and process boundaries."""++import http.client+import io+import json+import os+import subprocess+import sys+import tempfile+import types+import urllib.error+from pathlib import Path++import pytest++from tests.test_runpod_cloud import CONFIG, _backend, _controller+from tests.test_runpod_controller import _clock+++def credentials(tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ key = tmp_path / "id_test"+ key.write_text("not-a-real-private-key", encoding="utf-8")+ Path(str(key) + ".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(key))+ return key++[email protected]("method", ["DELETE", "GET"])[email protected]("error", [+ http.client.IncompleteRead(b"private-response"),+ http.client.BadStatusLine("private-response"),+ http.client.LineTooLong("private-response"),+ ConnectionResetError("private-response"),+])+def test_cleanup_retries_transient_http_failures(monkeypatch, method, error):+ ns = _controller()+ sleeps = _clock(ns)+ calls = []+ failed = False++ class Opener:+ def open(self, request, timeout):+ nonlocal failed+ calls.append(request.method)+ if request.method == method and not failed:+ failed = True+ raise error+ if request.method == "GET":+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)+ return io.BytesIO(b"")++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ ns["_terminate"]("provider-secret", "pod-1")+ expected = ["DELETE", "DELETE", "GET"] if method == "DELETE" else [+ "DELETE", "GET", "DELETE", "GET",+ ]+ assert calls == expected+ assert sleeps == [5]++[email protected]("status", [400, 401, 403, 422, 429])+def test_definite_create_rejection_has_reason_and_no_false_cleanup(+ credentials, monkeypatch, status,+):+ ns = _controller()+ _clock(ns, 100)+ calls, notices = [], []++ class Opener:+ def open(self, request, timeout):+ calls.append(request.method)+ raise urllib.error.HTTPError(+ request.full_url, status, "private-response", {}, io.BytesIO(b"provider-secret"),+ )++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ ns["_notice"] = notices.append+ assert ns["main"]() == 1+ assert calls == ["POST"]+ assert any(f"RunPod API POST failed: HTTP {status}" in line for line in notices)+ assert "CRITICAL" not in str(notices)+ assert "termination verified" not in str(notices)+ assert "private-response" not in str(notices)+ assert "provider-secret" not in str(notices)++[email protected]("failure", [500, 503, 408, "timeout", "incomplete", "json"])+def test_ambiguous_create_still_recovers_only_owned_pod(credentials, monkeypatch, failure):+ ns = _controller()+ _clock(ns)+ calls, notices = [], []+ run_name = None++ class Opener:+ def open(self, request, timeout):+ nonlocal run_name+ path = request.full_url.removeprefix(ns["_API_BASE"])+ calls.append((request.method, path))+ if request.method == "POST":+ run_name = json.loads(request.data)["name"]+ if isinstance(failure, int):+ raise urllib.error.HTTPError(request.full_url, failure, "secret", {}, None)+ if failure == "timeout":+ raise TimeoutError("provider-secret")+ if failure == "incomplete":+ raise http.client.IncompleteRead(b"provider-secret")+ return io.BytesIO(b"invalid json: provider-secret")+ if path == "/pods":+ return io.BytesIO(json.dumps([+ {"id": "owned", "name": run_name}, {"id": "other", "name": "other"},+ ]).encode())+ if request.method == "DELETE":+ return io.BytesIO(b"")+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)++ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())+ ns["_notice"] = notices.append+ assert ns["main"]() == 1+ assert calls == [("POST", "/pods"), ("GET", "/pods"),+ ("DELETE", "/pods/owned"), ("GET", "/pods/owned")]+ assert "RunPod termination verified." in notices+ assert "provider-secret" not in str(notices)++[email protected]("stage,reason", [+ ("ssh", "did not expose SSH within 15 minutes"),+ ("training", "remote training failed"),+ ("copy", "artifact copy failed"),+])+def test_main_reports_safe_controller_failure_reason(credentials, monkeypatch, stage, reason):+ ns = _controller()+ _clock(ns)+ notices, deletes = [], []++ def request(*args, method="GET", **kwargs):+ if method == "POST":+ return {"id": "pod-1"}+ if method == "DELETE":+ deletes.append(args[1])+ return {} if stage == "ssh" and not deletes else None++ def run(argv, **kwargs):+ failed = (stage == "training" and "input" in kwargs) or (+ stage == "copy" and argv[0] == "scp"+ )+ return subprocess.CompletedProcess(argv, int(failed))++ ns["_request_json"] = request+ ns["_notice"] = notices.append+ if stage != "ssh":+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)+ monkeypatch.setattr(ns["subprocess"], "run", run)+ assert ns["main"]() == 1+ assert any(reason in line for line in notices)+ assert deletes == ["/pods/pod-1"]+ assert "RunPod termination verified." in notices++[email protected]("error", [RuntimeError("private-error"), ValueError("private-error")])+def test_main_does_not_print_untrusted_exception_text(credentials, error):+ ns = _controller()+ notices = []+ ns["_request_json"] = lambda *a, method="GET", **kw: {"id": "pod-1"} if (+ method == "POST"+ ) else None+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)++ def train(*args):+ raise error++ ns["_train_and_copy"] = train+ ns["_notice"] = notices.append+ assert ns["main"]() == 1+ assert "private-error" not in str(notices)+ assert "RunPod termination verified." in notices+++def test_output_appearing_during_training_prevents_scp(credentials, monkeypatch):+ ns = _controller()+ calls = []++ def run(argv, **kwargs):+ calls.append(argv)+ if "input" in kwargs:+ Path("out").mkdir()+ Path("out/keep").write_text("original", encoding="utf-8")+ return subprocess.CompletedProcess(argv, 0)++ monkeypatch.setattr(ns["subprocess"], "run", run)+ with pytest.raises(RuntimeError, match="already exists"):+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))+ assert [argv[0] for argv in calls] == ["ssh", "ssh"]+ assert Path("out/keep").read_text(encoding="utf-8") == "original"++[email protected]("cleanup_fails", [False, True])+def test_cleanup_masks_ctrl_c_and_restores_previous_handler(credentials, cleanup_fails):+ ns = _controller()+ previous = object()+ handlers = [previous]+ calls = []++ def install(sig, handler):+ assert sig == "INT"+ old = handlers[-1]+ handlers.append(handler)+ return old++ def terminate(*args):+ assert handlers[-1] == "IGNORE"+ calls.append(args)+ if cleanup_fails:+ raise RuntimeError("private-error")++ def interrupted(*args):+ raise KeyboardInterrupt++ ns["signal"] = types.SimpleNamespace(SIGINT="INT", SIG_IGN="IGNORE", signal=install)+ ns["_request_json"] = lambda *a, **kw: {"id": "pod-1"}+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)+ ns["_train_and_copy"] = interrupted+ ns["_terminate"] = terminate+ assert ns["main"]() == 1+ assert calls == [("test-provider-secret", "pod-1")]+ assert handlers == [previous, "IGNORE", previous]+++def test_readiness_retries_before_training_once(credentials, monkeypatch):+ ns = _controller()+ sleeps = _clock(ns)+ calls = []++ def run(argv, **kwargs):+ calls.append((argv, kwargs))+ return subprocess.CompletedProcess(argv, 255 if len(calls) == 1 else 0)++ monkeypatch.setattr(ns["subprocess"], "run", run)+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))+ assert [argv[0] for argv, _ in calls] == ["ssh", "ssh", "ssh", "scp"]+ assert calls[0][0][-1] == calls[1][0][-1] == "true"+ assert sum("input" in kwargs for _, kwargs in calls) == 1+ assert sleeps == [5]++[email protected]("pod_id", ["../x", "x" * 65, 17, None, "", True])+def test_invalid_pod_id_never_reaches_delete(pod_id):+ ns = _controller()+ calls = []+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))+ with pytest.raises(RuntimeError, match="invalid Pod ID"):+ ns["_terminate"]("secret", pod_id)+ assert calls == []++[email protected]("key", [+ "ecdsa-sha2-nistp256 AAAATEST", "ssh-ed25519 AAAA\nssh-rsa BBBB", "garbage",+])+def test_invalid_public_key_never_allocates(credentials, key):+ Path(str(credentials) + ".pub").write_text(key, encoding="utf-8")+ ns = _controller()+ calls = []+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))+ assert ns["main"]() == 2+ assert calls == []++[email protected]("kind", ["ed25519", "rsa"])+def test_supported_public_keys_launch_secure_cloud(credentials, kind):+ Path(str(credentials) + ".pub").write_text(f"ssh-{kind} AAAATEST fixture", encoding="utf-8")+ ns = _controller()+ payloads = []++ def request(*args, method="GET", payload=None):+ if method == "POST":+ payloads.append(payload)+ return {"id": "pod-1"}++ ns["_request_json"] = request+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)+ ns["_train_and_copy"] = lambda *args: None+ assert ns["main"]() == 0+ assert len(payloads) == 1+ assert payloads[0]["cloudType"] == "SECURE"+ assert payloads[0]["env"] == {"PUBLIC_KEY": f"ssh-{kind} AAAATEST fixture"}+++def test_realpath_escape_is_refused_before_allocation(credentials, monkeypatch, tmp_path):+ ns = _controller()+ realpath = os.path.realpath+ monkeypatch.setattr(ns["os"].path, "realpath", lambda path, **kw: (+ str(tmp_path.parent / "escaped") if str(path) == "out" else realpath(path, **kw)+ ))+ with pytest.raises(RuntimeError, match="stay under cwd"):+ ns["_local_destination"]()++[email protected]("output", ["-out", "a/-out", "a/-b/out"])+def test_option_like_output_component_refused_when_rendering(output):+ with pytest.raises(ValueError, match="output_dir"):+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir=output, soup_version="0.76.0")++[email protected]("output", ["-out", "a/-out", "a/-b/out"])+def test_option_like_output_component_refused_by_controller(credentials, output):+ ns = _controller()+ ns["_OUTPUT"] = output+ with pytest.raises(RuntimeError, match="output"):+ ns["_local_destination"]()+++def _plan(tmp_path):+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")+ return _backend().plan_runpod_run(+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0",+ )+++def test_submit_isolates_python_and_filters_environment(credentials, monkeypatch, tmp_path):+ plan = _plan(tmp_path)+ environment = {name: f"value-{name}" for name in (+ "RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY", "PATH", "HOME", "USERPROFILE", "SYSTEMROOT",+ )}+ untrusted = {name: "private-value" for name in (+ "PYTHONPATH", "PYTHONHOME", "HF_TOKEN", "WANDB_API_KEY", "LD_PRELOAD",+ )}+ spawns = []+ monkeypatch.setattr(subprocess, "Popen", lambda argv, **kw: (+ spawns.append((argv, kw)) or types.SimpleNamespace(wait=lambda: 7)+ ))+ assert _backend().submit_runpod_run(plan, env={**environment, **untrusted}) == 7+ assert spawns == [([sys.executable, "-I", plan.stub_path], {"env": environment})]+ assert plan.run_command == "python -I soup_runpod_app.py"++[email protected]("name", [+ "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY",+ "http_proxy", "https_proxy", "all_proxy", "no_proxy",+ "TEMP", "TMP", "TMPDIR", "SYSTEMDRIVE", "WINDIR", "SSH_AUTH_SOCK",+])+def test_controller_forwards_network_temp_and_agent_settings(+ credentials, monkeypatch, tmp_path, name,+):+ plan = _plan(tmp_path)+ supplied = {+ "RUNPOD_API_KEY": "fixture-key", "RUNPOD_SSH_PRIVATE_KEY": str(credentials),+ name: "setting-value", "HF_TOKEN": "not-for-this-controller",+ }+ expected = {key: value for key, value in supplied.items() if key != "HF_TOKEN"}+ # Check exact POSIX spellings too: on Windows a differently cased proxy name+ # can otherwise conceal a missing allow-list entry after normalisation.+ assert _backend()._controller_environment(supplied, case_insensitive=False) == expected+ launches = []+ monkeypatch.setattr(subprocess, "Popen", lambda argv, **kw: (+ launches.append((argv, kw)) or types.SimpleNamespace(wait=lambda: 0)+ ))+ assert _backend().submit_runpod_run(plan, env=supplied) == 0+ assert launches == [([sys.executable, "-I", plan.stub_path], {"env": expected})]++[email protected]("case_insensitive", [False, True, None])+def test_controller_environment_matches_windows_names_only_on_windows(case_insensitive):+ supplied = {"Path": "bin", "SystemRoot": "windows", "Https_Proxy": "proxy", "OTHER": "no"}+ expected = {key: value for key, value in supplied.items() if key != "OTHER"}+ windows = os.name == "nt" if case_insensitive is None else case_insensitive+ assert _backend()._controller_environment(+ supplied, case_insensitive=case_insensitive,+ ) == (expected if windows else {})++[email protected]("stream", ["stdout", "stderr"])[email protected]("mode", ["ascii", "native"])+def test_isolated_controller_replaces_unencodable_output(tmp_path, monkeypatch, stream, mode):+ monkeypatch.chdir(tmp_path)+ plan = _plan(tmp_path)+ _backend().write_stub(plan)+ # Run the rendered controller under -I, not a replica of its bootstrap.+ # The narrow mode models Windows redirected output on every platform.+ harness = tmp_path / "encoding_harness.py"+ harness.write_text(+ "import sys\n"+ "stream, mode, stub = sys.argv[1:]\n"+ "if mode == 'ascii':\n"+ " for handle in (sys.stdout, sys.stderr):\n"+ " handle.reconfigure(encoding='ascii', errors='strict')\n"+ "namespace = {'__name__': 'controller_under_test'}\n"+ "with open(stub, encoding='utf-8') as source:\n"+ " exec(compile(source.read(), stub, 'exec'), namespace)\n"+ "getattr(sys, stream).write('controller line: \\u2588 \\u00e9 \\u65e5\\n')\n"+ "getattr(sys, stream).flush()\n",+ encoding="utf-8",+ )+ environment = {name: value for name, value in os.environ.items() if name.upper() in {+ "PATH", "SYSTEMROOT", "WINDIR", "SYSTEMDRIVE", "TEMP", "TMP", "TMPDIR",+ }}+ result = subprocess.run(+ [sys.executable, "-I", str(harness), stream, mode, plan.stub_path],+ env=environment, cwd=tmp_path, capture_output=True, timeout=30,+ )+ assert result.returncode == 0, result.stderr+ output = getattr(result, stream).decode("ascii" if mode == "ascii" else "utf-8", "replace")+ assert "controller line:" in output+ if mode == "ascii":+ assert "controller line: ? ? ?" in output+ assert "UnicodeEncodeError" not in result.stderr.decode("utf-8", "replace")+++def test_actual_child_cannot_import_parent_pythonpath_or_read_unrelated_secret(+ credentials, tmp_path,+):+ plan = _plan(tmp_path)+ _backend().write_stub(plan)+ # Execute an inert probe in the same process-launch path; no provider traffic.+ Path(plan.stub_path).write_text(+ "import importlib.util, json, os, sys\n"+ "from pathlib import Path\n"+ "Path('child.json').write_text(json.dumps({"+ "'isolated': sys.flags.isolated, 'secret': os.environ.get('HF_TOKEN'), "+ "'injected': importlib.util.find_spec('parent_injection') is not None, "+ "'key': os.environ.get('RUNPOD_API_KEY')}))\n", encoding="utf-8",+ )+ (tmp_path / "parent_injection.py").write_text(+ "raise AssertionError('injected')", encoding="utf-8",+ )+ environment = {name: os.environ[name] for name in ("PATH", "SYSTEMROOT") if name in os.environ}+ environment.update(RUNPOD_API_KEY="fixture-key", RUNPOD_SSH_PRIVATE_KEY=str(credentials),+ PYTHONPATH=str(tmp_path), HF_TOKEN="private-parent-secret")+ assert _backend().submit_runpod_run(plan, env=environment) == 0+ assert json.loads(Path("child.json").read_text(encoding="utf-8")) == {+ "isolated": 1, "secret": None, "injected": False, "key": "fixture-key",+ }++[email protected]("failure", [None, "training", "copy", "interrupt"])+def test_ssh_uses_per_run_hosts_and_only_selected_identity(+ credentials, monkeypatch, tmp_path, failure,+):+ ns = _controller()+ scratch = tmp_path / "ssh scratch"+ scratch.mkdir()+ user_hosts = tmp_path / ".ssh/known_hosts"+ user_hosts.parent.mkdir()+ user_hosts.write_text("keep user hosts", encoding="utf-8")+ monkeypatch.setenv("HOME", str(tmp_path))+ monkeypatch.setenv("USERPROFILE", str(tmp_path))+ ns["tempfile"] = types.SimpleNamespace(TemporaryDirectory=lambda **kw: (+ tempfile.TemporaryDirectory(dir=scratch, **kw)+ ), mkdtemp=lambda **kw: tempfile.mkdtemp(dir=scratch, **kw))+ all_hosts = []+ for _ in range(2):+ hosts, calls = [], []++ def run(argv, **kwargs):+ calls.append(argv)+ assert "IdentitiesOnly=yes" in argv+ options = [arg for arg in argv if arg.startswith("UserKnownHostsFile=")]+ assert len(options) == 1+ host_file = Path(options[0].split("=", 1)[1].strip('"'))+ assert host_file.parent.is_dir()+ assert host_file != user_hosts+ hosts.append(host_file)+ host_file.touch()+ if "input" in kwargs and failure == "interrupt":+ raise KeyboardInterrupt+ failed = ("input" in kwargs and failure == "training") or (+ argv[0] == "scp" and failure == "copy"+ )+ if argv[0] == "scp":+ assert "--" in argv+ delimiter = argv.index("--")+ assert argv[delimiter + 1:] == ["[email protected]:/workspace/soup/out", "."]+ return subprocess.CompletedProcess(argv, int(failed))++ monkeypatch.setattr(ns["subprocess"], "run", run)+ if failure:+ with pytest.raises(KeyboardInterrupt if failure == "interrupt" else RuntimeError):+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))+ else:+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))+ assert len(set(hosts)) == 1+ assert not hosts[0].parent.exists()+ assert [argv[0] for argv in calls] == (+ ["ssh", "ssh"] if failure in {"training", "interrupt"} else ["ssh", "ssh", "scp"]+ )+ all_hosts.append(hosts[0])+ assert len(set(all_hosts)) == 2+ assert user_hosts.read_text(encoding="utf-8") == "keep user hosts"diff --git a/tests/test_yaml_reads_are_limited.py b/tests/test_yaml_reads_are_limited.pyindex f09cd9d6..4f6fa5a3 100644--- a/tests/test_yaml_reads_are_limited.py+++ b/tests/test_yaml_reads_are_limited.py@@ -41,6 +41,7 @@ _NOT_PARSING = frozenset({"safe_dump", "dump", "YAMLError"})YAML_USERS = {"autopilot/generate_config.py": ("safe_dump",),"cans/pack.py": ("safe_dump",),+ "cloud/runpod.py": ("YAMLError",),"commands/adapters.py": ("YAMLError",),"commands/deploy.py": ("YAMLError",),"commands/push.py": ("YAMLError",),@@ -378,6 +379,12 @@ def _quant_config(text):return load_quant_config(_write("quant.yaml", text))+def _runpod(text):+ from soup_cli.cloud.runpod import render_runpod_stub++ return render_runpod_stub(text, gpu="a100", output_dir="out", soup_version="0.76.0")++def _sweep(text):from soup_cli.utils.sweep_config import parse_sweep_yaml@@ -386,6 +393,7 @@ def _sweep(text):#: reader id -> (driver, how the refusal names the document)_RAISING_READERS = {+ "cloud/runpod.py": (_runpod, "RunPod config"),"commands/plan.py": (_plan, "config 'soup.yaml'"),"eval/gate.py": (_gate_suite, "eval-gate suite 'gate.yaml'"),"eval/judge.py": (_rubric, "Rubric file rubric.yaml"),@@ -586,6 +594,10 @@ def _cap(module: str, name: str) -> int:#: reader id -> (module, name of its cap, how it words a file one byte over the cap)_READER_CAPS = {+ "cloud/runpod.py": (+ "cloud.runpod", "_MAX_CONFIG_BYTES",+ "RunPod config is larger than {cap} bytes; refusing",+ ),"commands/plan.py": ("utils.yaml_limits", "MAX_CONFIG_BYTES","config 'soup.yaml' is larger than {cap} bytes; refusing",@@ -708,6 +720,7 @@ def test_license_frontmatter_falls_back_one_byte_over_its_cap(tmp_path, monkeypa#: Every call of the limited loader names its cap: a module constant, never a number._CAP_NAMES = {"cans/unpack.py": {"MAX_CONFIG_BYTES", "MAX_MANIFEST_BYTES"},+ "cloud/runpod.py": {"_MAX_CONFIG_BYTES"},"commands/adapters.py": {"MAX_CONFIG_BYTES"},"commands/deploy.py": {"MAX_CONFIG_BYTES"},"commands/plan.py": {"MAX_CONFIG_BYTES"},