Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

feat(cloud): add RunPod controller with verified cleanup (#2081)

Nurkhan Esenbekfc47970a456b
diff --git a/changelog.d/0.76.0/2081.added.md b/changelog.d/0.76.0/2081.added.md
new file mode 100644
index 00000000..deeed6a0
--- /dev/null
+++ b/changelog.d/0.76.0/2081.added.md
@@ -0,0 +1 @@
+- Add a RunPod cloud-training controller with plan-only rendering, bounded YAML validation, isolated credential-gated submission, SSH identity isolation, per-run host files and confirmed Pod deletion with transient-error retries. Preserve proxy, temporary-directory, Windows and SSH-agent settings while filtering other parent variables, and replace unencodable log characters under isolated Python. Report safe failure reasons and distinguish rejected creates from ambiguous launches; not yet live-validated. (#2081 by @kok-o)
diff --git a/docs/backends-and-ops.md b/docs/backends-and-ops.md
index d22ea04f..09b94f9e 100644
--- a/docs/backends-and-ops.md
+++ b/docs/backends-and-ops.md
@@ -372,7 +372,7 @@ training:
## Cloud GPU Training
-No local GPU? `soup train --cloud modal|lambda` renders a provider-specific controller
+No local GPU? `soup train --cloud modal|lambda|runpod` renders a provider-specific controller
from your `soup.yaml`. The config YAML is base64-embedded as **data**; credentials are read from
the environment only when a live submission starts.
@@ -397,11 +397,64 @@ including after a failed run, the app downloads every file in that directory int
output directory and prints a `modal volume get soup-outputs /<run name> <output dir> --force`
command that retries the download; that command places the files under `<output dir>/<run name>`.
-### RunPod (Planned)
+### RunPod
-RunPod support is currently in development and descoped from live CLI dispatch pending automated
-lifecycle and termination safeguards. Running `soup train --cloud runpod` informs the operator that
-RunPod is not yet live and points to active cloud backends (`--cloud modal` and `--cloud lambda`).
+`soup train --cloud runpod --gpu a100` writes `soup_runpod_app.py` without contacting
+RunPod. GPUs: `a100` (80 GB SXM), `h100`, `a40`, `rtx3090`, `rtx4090`. The controller
+uses Python's standard library and local OpenSSH `ssh`/`scp`; no RunPod SDK extra
+is needed. For live submission, set `RUNPOD_API_KEY` and `RUNPOD_SSH_PRIVATE_KEY`
+to your private key path, with its matching OpenSSH public key at `<path>.pub`.
+Only ed25519 and RSA public keys are accepted:
+
+```bash
+soup train --config soup.yaml --cloud runpod --gpu rtx4090 --cloud-submit
+```
+
+The local controller creates one Secure Cloud GPU Pod with TCP SSH access, installs
+the running Soup version, runs training over SSH and copies the configured relative
+output directory back before deleting the Pod. Choose a fresh local output path;
+existing outputs, symlink/junction paths and path components starting with `-` are
+refused before allocation and rechecked before copying. SCP separates options from
+its operands with `--`. Submission starts Python in isolated mode (`-I`) and passes
+only the RunPod token/key path, `PATH`, `HOME`, `USERPROFILE`, proxy settings
+(`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY` and their lowercase forms),
+temporary-directory settings (`TEMP`, `TMP`, `TMPDIR`), Windows settings
+(`SYSTEMROOT`, `SYSTEMDRIVE`, `WINDIR`) and `SSH_AUTH_SOCK` when
+present. Environment names are matched case-insensitively on Windows. The controller
+reconfigures stdout/stderr to replace characters the pipe encoding cannot represent,
+because isolated Python ignores `PYTHONIOENCODING` and `PYTHONUTF8`.
+SSH/SCP use `IdentitiesOnly=yes` with the selected key and a temporary per-run known-hosts
+file, removed even after failure; the user's known-hosts file is not modified.
+The config is base64-embedded data; planning applies the
+shared YAML size, nesting and alias-expansion limits, and refuses inline credential
+fields. The API key and private SSH key remain local;
+only the public key enters the Pod environment. Models and datasets must be publicly
+accessible remotely: local input files and local credentials are not uploaded.
+
+Deletion runs in `finally` on training/copy failure, timeout and Ctrl+C, and polls
+until the API returns 404. The parent waits through interrupts so it does not kill
+the cleanup controller. Transient transport failures, including truncated HTTP
+responses during DELETE or confirmation GET, are retried within the five-minute
+cleanup window. A lost or ambiguous create response (including HTTP 5xx and 408)
+is reconciled by the unique run name, without repeating the create request or
+deleting unrelated Pods. A definite HTTP 4xx rejection other than 408 reports the
+status without claiming that cleanup of an allocated Pod failed. Fixed controller
+failure reasons are shown; arbitrary transport/subprocess exception text is hidden.
+Keep the controller running; force-killing it or losing API access can leave cleanup
+unconfirmed. An unconfirmed cleanup exits nonzero and names the run to check in
+the RunPod console. A single interrupted SSH training session is failed rather
+than retried, to avoid duplicate training. Keepalive probes run every 30 seconds
+with six missed probes allowed (roughly three minutes); the training command has
+a 25-hour timeout. Artifacts are copied only after a successful training command,
+with one SCP attempt capped at 15 minutes. A failed copy still terminates the Pod,
+so an incomplete download cannot be recovered from that Pod afterward. Remote log
+tail retrieval and a copy retry window are not implemented in this controller.
+
+**Not yet live-validated.** Tests use local mocked API/SSH transports. The code
+PR does not include a paid run; provider execution, GPU behavior, retrieval and
+billing termination remain a separate live-validation task, per #264's ruling.
+API contract: [RunPod REST v1 OpenAPI](https://rest.runpod.io/v1/openapi.json),
+[SSH setup](https://docs.runpod.io/pods/configuration/use-ssh).
### Lambda Cloud
@@ -435,7 +488,7 @@ soup train --config soup.yaml --cloud lambda --gpu a100 --cloud-submit
controller can copy the artifact back safely. The API key stays on the caller and is never embedded
in cloud-init or instance logs.
-The Lambda submission path still requires the paid live-validation checklist in #264 before
+The Lambda submission path still requires the paid live-validation checklist in #1280 before
it can be described as provider-validated. Plan-only rendering and the lifecycle boundaries are
covered by offline tests.
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index bcff680a..402f0c0b 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -3667,7 +3667,7 @@ Usage: `soup train [OPTIONS]`
| `--track-energy` | flag | off | | Measure the training window's energy + CO2 via codecarbon (offline; requires `pip install soup-cli[carbon]`). Feeds the kWh / CO2 into --annex-xi. v0.71.3. |
| `--energy-country` | text | `USA` | | ISO 3166-1 alpha-3 country code for the CO2 grid-intensity estimate used by --track-energy (default USA). |
| `--energy-out` | text | | | Write the --track-energy measurement to this JSON file (cwd-contained) so `soup bom emit --energy <path>` can consume it. v0.71.15. |
-| `--cloud` | text | | | Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: modal, lambda (runpod is planned). Renders a cloud app stub from the config (plan-only); use --cloud-submit to submit live. |
+| `--cloud` | text | | | Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: modal, lambda, runpod. Renders a cloud app stub from the config (plan-only); use --cloud-submit to submit live. |
| `--gpu` | text | `a100` | | Cloud GPU type for --cloud (t4 / l4 / a10 / a10g / a100 / a100-80gb / l40s / h100 / a6000). Default a100. Provider-specific allowlists apply. |
| `--cloud-submit` | flag | off | | With --cloud, submit the rendered run live via the cloud's SDK or API (gated on respective provider token/API key). Default is plan-only (render + print the command). Lambda requires a registered SSH key. |
diff --git a/docs/commands.md b/docs/commands.md
index a2d4057a..4fba8f86 100644
--- a/docs/commands.md
+++ b/docs/commands.md
@@ -27,7 +27,7 @@ soup train --config soup.yaml --gate evals/gate.yaml Eval-gated training
soup train --config soup.yaml --push-as user/repo Auto-push each checkpoint to HF as branch
soup train --config soup.yaml --push-as user/repo --hf-resume Resume from latest HF checkpoint branch (not with training.relora_steps)
soup train --config soup.yaml --find-lr LR range finder: write recommended LR JSON
-soup train --config soup.yaml --cloud modal|lambda --gpu a100 Render a cloud GPU controller (plan-only; --cloud-submit submits live)
+soup train --config soup.yaml --cloud modal|lambda|runpod --gpu a100 Render a cloud GPU controller (plan-only; --cloud-submit submits live)
soup infer --model ./output --input p.jsonl --output out.jsonl [--batch-size N] Batch inference; default batch size is 1; fp16/bf16 batching may not be bit-exact with batch size 1
soup infer --model ./output --input p.jsonl --output out.jsonl --cuda-graphs Experimental CUDA graph decode (Qwen2/Llama, one GPU, PyTorch >= 2.14; batch size 1 only)
soup infer --task asr --model <whisper|adapter> --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate] Whisper transcription + WER/CER
diff --git a/src/soup_cli/cloud/__init__.py b/src/soup_cli/cloud/__init__.py
index 39aee73d..d56b03de 100644
--- a/src/soup_cli/cloud/__init__.py
+++ b/src/soup_cli/cloud/__init__.py
@@ -1,8 +1,8 @@
"""Cloud GPU backends for ``soup train --cloud`` (v0.71.18 #16, v0.71.22 #264).
-Ships Modal and Lambda Cloud backends, with RunPod planned.
+Ships Modal, Lambda Cloud and RunPod backends.
"""
from __future__ import annotations
-__all__ = ["lambda_labs", "modal"]
+__all__ = ["lambda_labs", "modal", "runpod"]
diff --git a/src/soup_cli/cloud/_runpod_controller.py b/src/soup_cli/cloud/_runpod_controller.py
new file mode 100644
index 00000000..88a3f781
--- /dev/null
+++ b/src/soup_cli/cloud/_runpod_controller.py
@@ -0,0 +1,330 @@
+"""Standalone controller template appended after validated plan constants.
+
+REST contract: https://rest.runpod.io/v1/openapi.json
+SSH contract: https://docs.runpod.io/pods/configuration/use-ssh
+Only main() performs network operations. This module is also copied into plans.
+"""
+
+import http.client
+import ipaddress
+import json
+import os
+import pathlib
+import re
+import shutil
+import signal
+import stat
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.request
+import uuid
+
+# Isolated Python ignores PYTHONIOENCODING/PYTHONUTF8. A redirected Windows
+# stream may not encode a log character; print a replacement instead of failing.
+for _stream in (sys.stdout, sys.stderr):
+ if hasattr(_stream, "reconfigure"):
+ _stream.reconfigure(errors="replace")
+
+_GPU_ID = globals().get("_GPU_ID", "")
+_CONFIG_B64 = globals().get("_CONFIG_B64", "")
+_OUTPUT = globals().get("_OUTPUT", "")
+_SOUP_VERSION = globals().get("_SOUP_VERSION", "")
+
+_API_BASE = "https://rest.runpod.io/v1"
+_POD_ID = re.compile(r"[a-zA-Z0-9_-]{1,64}")
+_MAX_RESPONSE = 1024 * 1024
+_IMAGE = "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404"
+_BOOT = (
+ "set -eu; if [ ! -x /usr/sbin/sshd ]; then apt-get update; "
+ "apt-get install -y openssh-server; fi; "
+ "mkdir -p /root/.ssh /run/sshd; chmod 700 /root/.ssh; "
+ "printf '%s\\n' \"$PUBLIC_KEY\" > /root/.ssh/authorized_keys; "
+ "chmod 600 /root/.ssh/authorized_keys; ssh-keygen -A; exec /usr/sbin/sshd -D -e"
+)
+
+
+def _notice(message):
+ sys.stderr.write(message + "\n")
+ sys.stderr.flush()
+
+
+class _ControllerError(RuntimeError):
+ """Only fixed, controller-owned diagnostics are safe to show to the user."""
+
+
+class _APIError(_ControllerError):
+ def __init__(self, method: str, status: int) -> None:
+ super().__init__(f"RunPod API {method} failed: HTTP {status}")
+ # A server/request timeout can still have allocated a Pod. Reconcile it.
+ self.create_rejected = method == "POST" and 400 <= status < 500 and status != 408
+
+
+class _NoRedirect(urllib.request.HTTPRedirectHandler):
+ def redirect_request(self, req, fp, code, msg, headers, newurl):
+ return None
+
+
+def _request_json(api_key, path, *, method="GET", payload=None):
+ data = None if payload is None else json.dumps(payload).encode("utf-8")
+ request = urllib.request.Request(_API_BASE + path, data=data, method=method)
+ request.add_header("Authorization", "Bearer " + api_key)
+ request.add_header("Content-Type", "application/json")
+ try:
+ with urllib.request.build_opener(_NoRedirect()).open(request, timeout=60) as response:
+ raw = response.read(_MAX_RESPONSE + 1)
+ if len(raw) > _MAX_RESPONSE:
+ raise _ControllerError("RunPod API response exceeds 1 MB")
+ if not raw and method == "DELETE":
+ return None
+ parsed = json.loads(raw)
+ if not isinstance(parsed, (dict, list)):
+ raise _ControllerError("RunPod API returned an invalid JSON shape")
+ return parsed
+ except urllib.error.HTTPError as exc:
+ if exc.code == 404 and method in {"GET", "DELETE"} and path.startswith("/pods/"):
+ return None
+ raise _APIError(method, exc.code) from None
+ except (OSError, ValueError, http.client.HTTPException):
+ raise _ControllerError("RunPod API transport or JSON response failed") from None
+
+
+def _launch_payload(public_key, name):
+ return {
+ "name": name,
+ "imageName": _IMAGE,
+ "cloudType": "SECURE",
+ "computeType": "GPU",
+ "gpuTypeIds": [_GPU_ID],
+ "gpuCount": 1,
+ "containerDiskInGb": 80,
+ "supportPublicIp": True,
+ "ports": ["22/tcp"],
+ "env": {"PUBLIC_KEY": public_key},
+ "dockerEntrypoint": ["/bin/bash", "-lc"],
+ "dockerStartCmd": [_BOOT],
+ }
+
+
+def _valid_id(value):
+ if not isinstance(value, str) or not _POD_ID.fullmatch(value):
+ raise _ControllerError("RunPod returned an invalid Pod ID; check the console")
+ return value
+
+
+def _wait_for_ssh(api_key, pod_id):
+ path = "/pods/" + _valid_id(pod_id)
+ deadline = time.monotonic() + 900
+ while time.monotonic() < deadline:
+ record = _request_json(api_key, path)
+ if not isinstance(record, dict) or record.get("desiredStatus") in {"EXITED", "TERMINATED"}:
+ raise _ControllerError("RunPod is unavailable before SSH startup")
+ ip = record.get("publicIp")
+ port = (record.get("portMappings") or {}).get("22")
+ if ip and isinstance(port, int) and not isinstance(port, bool) and 1 <= port <= 65535:
+ address = ipaddress.ip_address(ip)
+ # The API chooses the destination; refuse local/metadata addresses.
+ if address.version != 4 or not address.is_global:
+ raise _ControllerError("RunPod did not return a public SSH address")
+ return str(address), port
+ time.sleep(5)
+ raise _ControllerError("RunPod did not expose SSH within 15 minutes")
+
+
+def _train_and_copy(ip, port, private_key):
+ with tempfile.TemporaryDirectory(prefix="soup-runpod-ssh-") as directory:
+ known_hosts = pathlib.Path(directory) / "known_hosts"
+ _run_ssh_session(ip, port, private_key, known_hosts)
+
+
+def _run_ssh_session(ip: str, port: int, private_key: str, known_hosts: pathlib.Path) -> None:
+ common = [
+ "-i",
+ private_key,
+ "-o",
+ "BatchMode=yes",
+ "-o",
+ "IdentitiesOnly=yes",
+ "-o",
+ f'UserKnownHostsFile="{known_hosts.as_posix()}"',
+ "-o",
+ "StrictHostKeyChecking=accept-new",
+ "-o",
+ "ConnectTimeout=15",
+ "-o",
+ "ServerAliveInterval=30",
+ "-o",
+ "ServerAliveCountMax=6",
+ ]
+ script = (
+ "import base64, pathlib, subprocess, sys\n"
+ "root = pathlib.Path('/workspace/soup'); root.mkdir(parents=True, exist_ok=True)\n"
+ "import os; os.chdir(root)\n"
+ f"pathlib.Path('soup.yaml').write_bytes(base64.b64decode({_CONFIG_B64!r}))\n"
+ "subprocess.run([sys.executable, '-m', 'pip', 'install', "
+ f"'soup-cli[train]=={_SOUP_VERSION}'], "
+ "check=True, stdout=sys.stderr)\n"
+ "with open('train.log', 'w') as log:\n"
+ " result = subprocess.run(['soup', 'train', '--config', 'soup.yaml', '--yes'], "
+ "stdout=log, stderr=subprocess.STDOUT)\n"
+ "sys.exit(result.returncode)\n"
+ )
+ # A readiness mapping can precede sshd. Retry a harmless readiness command,
+ # never the training command: reconnecting must not start training twice.
+ deadline = time.monotonic() + 300
+ while True:
+ ready = subprocess.run(
+ ["ssh", *common, "-p", str(port), f"root@{ip}", "true"], capture_output=True, timeout=30
+ )
+ if ready.returncode == 0:
+ break
+ if time.monotonic() >= deadline:
+ raise _ControllerError("RunPod SSH did not become ready within five minutes")
+ time.sleep(5)
+ result = subprocess.run(
+ ["ssh", *common, "-p", str(port), f"root@{ip}", "python3 -"],
+ input=script,
+ text=True,
+ capture_output=True,
+ timeout=25 * 3600,
+ )
+ if result.returncode != 0:
+ raise _ControllerError("RunPod remote training failed; no successful artifact copy claimed")
+ destination = _local_destination()
+ destination.mkdir(parents=True, exist_ok=True)
+ copied = subprocess.run(
+ [
+ "scp",
+ *common,
+ "-P",
+ str(port),
+ "-r",
+ "--",
+ f"root@{ip}:/workspace/soup/{_OUTPUT}",
+ str(destination),
+ ],
+ capture_output=True,
+ timeout=900,
+ )
+ if copied.returncode != 0:
+ raise _ControllerError("RunPod artifact copy failed")
+
+
+def _local_destination():
+ """Recheck the download path before allocation and immediately before SCP."""
+ root = os.path.realpath(os.getcwd())
+ target = pathlib.Path(_OUTPUT)
+ if (target.is_absolute() or ".." in target.parts or str(target) == "."
+ or any(part.startswith("-") for part in target.parts)):
+ raise _ControllerError("RunPod output must stay under cwd without option-like components")
+ candidate = pathlib.Path.cwd()
+ for part in target.parts:
+ candidate = candidate / part
+ if os.path.lexists(candidate):
+ info = os.lstat(candidate)
+ if stat.S_ISLNK(info.st_mode) or getattr(info, "st_file_attributes", 0) & 0x400:
+ raise _ControllerError("RunPod output must not traverse symlinks or junctions")
+ if os.path.commonpath([root, os.path.realpath(target)]) != root:
+ raise _ControllerError("RunPod output must stay under cwd")
+ if os.path.lexists(target):
+ raise _ControllerError("RunPod output already exists; choose a fresh output directory")
+ return target.parent
+
+
+def _terminate(api_key, pod_id):
+ path = "/pods/" + _valid_id(pod_id)
+ deadline = time.monotonic() + 300
+ while time.monotonic() < deadline:
+ try:
+ _request_json(api_key, path, method="DELETE")
+ # A successful DELETE alone is not evidence of termination.
+ if _request_json(api_key, path) is None:
+ return
+ except RuntimeError:
+ pass
+ time.sleep(5)
+ raise _ControllerError("RunPod deletion was not confirmed within five minutes")
+
+
+def _recover_created(api_key, name):
+ """A lost POST response must not cause another POST or delete unrelated Pods."""
+ deadline = time.monotonic() + 60
+ while time.monotonic() < deadline:
+ records = _request_json(api_key, "/pods")
+ if not isinstance(records, list):
+ raise _ControllerError("RunPod could not reconcile the launch; check the console")
+ found = [
+ item["id"] for item in records if isinstance(item, dict) and item.get("name") == name
+ ]
+ if found:
+ return [_valid_id(item) for item in found]
+ time.sleep(5)
+ raise _ControllerError("RunPod launch result is unknown; check the console")
+
+
+def main():
+ if not all((_GPU_ID, _CONFIG_B64, _OUTPUT, _SOUP_VERSION)):
+ _notice("Generate a RunPod plan with soup train --cloud runpod first.")
+ return 2
+ if not all(shutil.which(command) for command in ("ssh", "scp")):
+ _notice("OpenSSH ssh and scp must be installed before RunPod submission.")
+ return 2
+ try:
+ _local_destination()
+ except (OSError, RuntimeError):
+ _notice("RunPod needs a fresh local output path under cwd, without symlinks.")
+ return 2
+ api_key = os.environ.get("RUNPOD_API_KEY")
+ private_key = os.path.expanduser(os.environ.get("RUNPOD_SSH_PRIVATE_KEY", ""))
+ if not api_key or not private_key or not pathlib.Path(private_key).is_file():
+ _notice("Set RUNPOD_API_KEY and RUNPOD_SSH_PRIVATE_KEY to a readable key file.")
+ return 2
+ try:
+ public_key = pathlib.Path(private_key + ".pub").read_text(encoding="utf-8").strip()
+ if not re.fullmatch(r"ssh-(?:ed25519|rsa) [A-Za-z0-9+/=]+(?: [^\r\n]+)?", public_key):
+ raise ValueError("invalid public key")
+ except (OSError, ValueError):
+ _notice("A matching .pub file with one OpenSSH public key is required.")
+ return 2
+ name = "soup-" + uuid.uuid4().hex
+ pod_id = None
+ launched = False
+ rc = 1
+ try:
+ launched = True
+ response = _request_json(
+ api_key, "/pods", method="POST", payload=_launch_payload(public_key, name)
+ )
+ pod_id = _valid_id(response.get("id") if isinstance(response, dict) else None)
+ _notice("RunPod Pod launched: " + pod_id)
+ ip, port = _wait_for_ssh(api_key, pod_id)
+ _train_and_copy(ip, port, private_key)
+ rc = 0
+ except (Exception, KeyboardInterrupt, SystemExit) as exc:
+ if isinstance(exc, _APIError) and exc.create_rejected:
+ launched = False
+ _notice(str(exc) if isinstance(exc, _ControllerError) else (
+ "RunPod training/controller failed; cleaning up the Pod."
+ ))
+ finally:
+ # Repeated Ctrl+C must not abandon cleanup of a billable resource.
+ previous = signal.signal(signal.SIGINT, signal.SIG_IGN)
+ try:
+ ids = [pod_id] if pod_id else (_recover_created(api_key, name) if launched else [])
+ for owned_id in ids:
+ _terminate(api_key, owned_id)
+ _notice("RunPod termination verified." if ids else (
+ "RunPod create request rejected; no Pod to clean up."
+ ))
+ except Exception:
+ _notice("CRITICAL: RunPod cleanup unconfirmed; check the console for " + name)
+ rc = 1
+ finally:
+ signal.signal(signal.SIGINT, previous)
+ return rc
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/src/soup_cli/cloud/runpod.py b/src/soup_cli/cloud/runpod.py
new file mode 100644
index 00000000..e0035bd8
--- /dev/null
+++ b/src/soup_cli/cloud/runpod.py
@@ -0,0 +1,220 @@
+"""RunPod plans and credential-gated submission; no provider SDK is imported."""
+
+from __future__ import annotations
+
+import base64
+import os
+import re
+import shlex
+import sys
+import types
+from collections.abc import Callable, Mapping
+from pathlib import Path, PurePosixPath
+
+from soup_cli.cloud._common import (
+ _MAX_CONFIG_BYTES,
+ _MAX_VERSION_LEN,
+ _VERSION_RE,
+ CloudPlan,
+ validate_choice,
+ validate_path_shape,
+ write_cloud_stub,
+)
+
+SUPPORTED_CLOUDS = frozenset({"runpod"})
+# REST v1 gpuTypeIds, verified against the provider's published OpenAPI schema.
+_GPU_IDS = types.MappingProxyType(
+ {
+ "a100": "NVIDIA A100-SXM4-80GB",
+ "h100": "NVIDIA H100 80GB HBM3",
+ "a40": "NVIDIA A40",
+ "rtx3090": "NVIDIA GeForce RTX 3090",
+ "rtx4090": "NVIDIA GeForce RTX 4090",
+ }
+)
+SUPPORTED_GPUS = frozenset(_GPU_IDS)
+_RUNPOD_SUBMIT_OVERRIDE: Callable[[CloudPlan], int] | None = None
+_CREDENTIAL_FIELDS = frozenset(
+ {
+ "api_key",
+ "token",
+ "access_token",
+ "auth_token",
+ "hf_token",
+ "password",
+ "private_key",
+ "secret",
+ "wandb_api_key",
+ "runpod_api_key",
+ "aws_secret_access_key",
+ }
+)
+
+
+def validate_cloud(name: object) -> str:
+ return validate_choice(name, "cloud", SUPPORTED_CLOUDS)
+
+
+def validate_gpu(gpu: object) -> str:
+ return validate_choice(gpu, "gpu", SUPPORTED_GPUS)
+
+
+def _validate_config(config_yaml: str) -> None:
+ import yaml
+
+ from soup_cli.utils.yaml_limits import load_yaml_limited
+
+ if not isinstance(config_yaml, str):
+ raise ValueError("config must be a string")
+ key = os.environ.get("RUNPOD_API_KEY")
+ if key and key in config_yaml:
+ raise ValueError("config must not embed provider credentials")
+ try:
+ content = load_yaml_limited(config_yaml, "RunPod config", max_bytes=_MAX_CONFIG_BYTES)
+ except yaml.YAMLError:
+ raise ValueError("config must be valid YAML; credentials must stay local") from None
+
+ def walk(value: object, active: set[int]) -> None:
+ if not isinstance(value, (dict, list)):
+ return
+ if id(value) in active:
+ raise ValueError("recursive config aliases are not supported")
+ active.add(id(value))
+ try:
+ if isinstance(value, dict):
+ for name, item in value.items():
+ if str(name).lower() in _CREDENTIAL_FIELDS and item not in (None, ""):
+ raise ValueError("inline config credentials are not supported")
+ walk(item, active)
+ else:
+ for item in value:
+ walk(item, active)
+ finally:
+ active.remove(id(value))
+
+ walk(content, set())
+
+
+def render_runpod_stub(
+ config_yaml: str,
+ *,
+ gpu: str,
+ output_dir: str,
+ soup_version: str,
+) -> str:
+ """Render a self-contained local controller; config is embedded only as data."""
+ gpu = validate_gpu(gpu)
+ validate_path_shape(output_dir, "output_dir")
+ output = PurePosixPath(output_dir)
+ if (
+ output.is_absolute()
+ or ".." in output.parts
+ or any(part.startswith("-") for part in output.parts)
+ or str(output) == "."
+ or not re.fullmatch(r"[A-Za-z0-9._/-]+", output_dir)
+ ):
+ raise ValueError("output_dir must be a simple relative path under cwd")
+ if (
+ not isinstance(soup_version, str)
+ or len(soup_version) > _MAX_VERSION_LEN
+ or not _VERSION_RE.fullmatch(soup_version)
+ ):
+ raise ValueError("soup_version must be a safe package version")
+ _validate_config(config_yaml)
+ encoded = base64.b64encode(config_yaml.encode("utf-8")).decode("ascii")
+ constants = (
+ f"_GPU_ID = {_GPU_IDS[gpu]!r}\n_CONFIG_B64 = {encoded!r}\n"
+ f"_OUTPUT = {str(output)!r}\n_SOUP_VERSION = {soup_version!r}\n"
+ )
+ controller = Path(__file__).with_name("_runpod_controller.py").read_text(encoding="utf-8")
+ return constants + controller
+
+
+def plan_runpod_run(
+ config_path: str,
+ *,
+ gpu: str,
+ output_dir: str,
+ soup_version: str,
+ stub_path: str = "soup_runpod_app.py",
+) -> CloudPlan:
+ from soup_cli.utils.paths import enforce_under_cwd_and_no_symlink
+
+ for value, field in (
+ (config_path, "--config"),
+ (output_dir, "output_dir"),
+ (stub_path, "stub_path"),
+ ):
+ validate_path_shape(value, field)
+ enforce_under_cwd_and_no_symlink(value, field)
+ with open(config_path, encoding="utf-8") as source:
+ config_yaml = source.read(_MAX_CONFIG_BYTES + 1)
+ if os.path.lexists(output_dir):
+ raise ValueError("output_dir already exists; choose a fresh directory for retrieval")
+ stub = render_runpod_stub(
+ config_yaml,
+ gpu=gpu,
+ output_dir=output_dir,
+ soup_version=soup_version,
+ )
+ return CloudPlan(
+ "runpod", validate_gpu(gpu), output_dir, stub_path, stub,
+ f"python -I {shlex.quote(stub_path)}",
+ )
+
+
+def write_stub(plan: CloudPlan) -> str:
+ return write_cloud_stub(plan)
+
+
+_CONTROLLER_ENV_NAMES: frozenset[str] = frozenset({
+ "RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY", "PATH", "HOME", "USERPROFILE", "SYSTEMROOT",
+ "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY",
+ "http_proxy", "https_proxy", "all_proxy", "no_proxy",
+ "TEMP", "TMP", "TMPDIR", "SYSTEMDRIVE", "WINDIR", "SSH_AUTH_SOCK",
+})
+
+
+def _controller_environment(
+ environ: Mapping[str, str], *, case_insensitive: bool | None = None,
+) -> dict[str, str]:
+ """Keep the controller's credentials and OS/network settings, with OS name rules."""
+ if case_insensitive is None:
+ case_insensitive = os.name == "nt"
+ allowed = ({name.upper() for name in _CONTROLLER_ENV_NAMES}
+ if case_insensitive else _CONTROLLER_ENV_NAMES)
+ return {
+ name: value for name, value in environ.items()
+ if (name.upper() if case_insensitive else name) in allowed
+ }
+
+
+def submit_runpod_run(plan: CloudPlan, *, env: Mapping | None = None) -> int:
+ """Keep waiting through interrupts so the controller can finish its cleanup."""
+ if not isinstance(plan, CloudPlan) or plan.cloud != "runpod":
+ raise TypeError("plan must be a RunPod CloudPlan")
+ if _RUNPOD_SUBMIT_OVERRIDE is not None:
+ return _RUNPOD_SUBMIT_OVERRIDE(plan)
+ environ = dict(env if env is not None else os.environ)
+ missing = [
+ name for name in ("RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY") if not environ.get(name)
+ ]
+ if missing:
+ raise RuntimeError("RunPod not configured. Set " + ", ".join(missing))
+ environ = _controller_environment(environ)
+ import subprocess
+
+ proc = None
+ while True:
+ try:
+ if proc is None:
+ proc = subprocess.Popen([sys.executable, "-I", plan.stub_path], env=environ)
+ return proc.wait()
+ except BaseException as exc:
+ if proc is None or not isinstance(exc, (KeyboardInterrupt, SystemExit)):
+ raise
+ try:
+ sys.stderr.write("Interrupted: waiting for RunPod cleanup to finish.\n")
+ sys.stderr.flush()
+ except BaseException:
+ pass
diff --git a/src/soup_cli/commands/train.py b/src/soup_cli/commands/train.py
index ebb94d8e..4f3eb3ac 100644
--- a/src/soup_cli/commands/train.py
+++ b/src/soup_cli/commands/train.py
@@ -843,7 +843,7 @@ def train(
"--cloud",
help=(
"Train on a cloud GPU instead of locally (v0.71.18 #16). Supported: "
- "modal, lambda (runpod is planned). Renders a cloud app stub from the config "
+ "modal, lambda, runpod. Renders a cloud app stub from the config "
"(plan-only); use --cloud-submit to submit live."
),
),
@@ -1064,10 +1064,7 @@ def train(
cloud = cloud.lower()
if cloud == "runpod":
- console.print(
- "[yellow]RunPod cloud training is not yet live; use --cloud modal or lambda.[/]"
- )
- raise typer.Exit(2)
+ from soup_cli.cloud import runpod as cloud_mod
elif cloud == "modal":
from soup_cli.cloud import modal as cloud_mod
elif cloud == "lambda":
@@ -1075,7 +1072,7 @@ def train(
else:
console.print(
f"[red]Invalid --cloud:[/] {markup_escape(cloud)}. "
- "Supported: modal, lambda (runpod is planned)."
+ "Supported: modal, lambda, runpod."
)
raise typer.Exit(2)
diff --git a/tests/test_issue1547_outbound_http_sinks.py b/tests/test_issue1547_outbound_http_sinks.py
index 9697e1ff..5c8edc46 100644
--- a/tests/test_issue1547_outbound_http_sinks.py
+++ b/tests/test_issue1547_outbound_http_sinks.py
@@ -159,6 +159,14 @@ class DeclaredSink:
# - gate: (caller_func, gate_func) where caller_func is AST-verified to call gate_func
# - reason: explicit justification for exemptions (fixed host, loopback only, operator env)
DECLARED_SINKS: dict[tuple[str, str], DeclaredSink] = {
+ ("cloud/_runpod_controller.py", "_request_json"): DeclaredSink(
+ calls=("urllib.request.build_opener",),
+ gate=None,
+ reason=(
+ "fixed host: _API_BASE is hardcoded to https://rest.runpod.io/v1; "
+ "the standalone controller refuses redirects and caps response reads"
+ ),
+ ),
("commands/generate.py", "_generate_openai"): DeclaredSink(
calls=("soup_cli.utils.http_response.post_capped",),
gate=("_generate_openai", "refuse_private_ip_literal"),
diff --git a/tests/test_issue264_cloud.py b/tests/test_issue264_cloud.py
index 4caf15de..40738721 100644
--- a/tests/test_issue264_cloud.py
+++ b/tests/test_issue264_cloud.py
@@ -194,7 +194,7 @@ class TestSubmitLambdaLabsRun:
class TestTrainCloudCliNew:
- def test_cloud_runpod_refuses_not_yet_live(self, tmp_path, monkeypatch):
+ def test_cloud_runpod_refuses_unknown_gpu(self, tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
(tmp_path / "soup.yaml").write_text(_SOUP_YAML, encoding="utf-8")
from typer.testing import CliRunner
@@ -202,13 +202,13 @@ class TestTrainCloudCliNew:
from soup_cli.cli import app
result = CliRunner().invoke(
- app, ["train", "--config", "soup.yaml", "--cloud", "runpod", "--gpu", "rtx-4090"]
+ app, ["train", "--config", "soup.yaml", "--cloud", "runpod", "--gpu", "tpu"]
)
assert result.exit_code == 2, (result.output, repr(result.exception))
assert not (tmp_path / "soup_runpod_app.py").exists()
txt = _strip_ansi(result.output)
- assert "not yet live" in txt
- assert "--cloud modal or lambda" in txt
+ assert "Invalid --cloud / --gpu" in txt
+ assert "rtx4090" in txt
def test_cloud_lambda_plan_only(self, tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
@@ -239,13 +239,12 @@ class TestTrainCloudCliNew:
assert result.exit_code == 0, (result.output, repr(result.exception))
assert (tmp_path / "soup_lambda_app.py").exists()
- # And verify uppercase RUNPOD is also refused
+ # RunPod uses the same case-insensitive provider dispatch.
res_runpod = CliRunner().invoke(
app, ["train", "--config", "soup.yaml", "--cloud", "RUNPOD", "--gpu", "a100"]
)
- assert res_runpod.exit_code == 2
- assert not (tmp_path / "soup_runpod_app.py").exists()
- assert "not yet live" in _strip_ansi(res_runpod.output)
+ assert res_runpod.exit_code == 0, (res_runpod.output, repr(res_runpod.exception))
+ assert (tmp_path / "soup_runpod_app.py").exists()
class TestCloudNoSecrets:
diff --git a/tests/test_runpod_cloud.py b/tests/test_runpod_cloud.py
new file mode 100644
index 00000000..8ebdc1ff
--- /dev/null
+++ b/tests/test_runpod_cloud.py
@@ -0,0 +1,292 @@
+"""RunPod planning and generated controller behavior, without provider traffic."""
+
+import base64
+import io
+import json
+import subprocess
+import types
+import urllib.error
+from pathlib import Path
+
+import pytest
+from typer.testing import CliRunner
+
+CONFIG = "base: public/model\ndata: {train: public/dataset}\noutput: out\n"
+
+
+def _backend():
+ from soup_cli.cloud import runpod
+
+ return runpod
+
+
+def _controller():
+ text = _backend().render_runpod_stub(
+ CONFIG, gpu="a100", output_dir="out", soup_version="0.76.0"
+ )
+ namespace = {"__name__": "test_controller"}
+ exec(compile(text, "soup_runpod_app.py", "exec"), namespace)
+ namespace["shutil"] = types.SimpleNamespace(which=lambda command: command)
+ return namespace
+
+
+def credentials(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ private = tmp_path / "id_test"
+ private.write_text("not-a-real-private-key", encoding="utf-8")
+ Path(str(private) + ".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")
+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")
+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(private))
+ return private
+
+
[email protected]("value", [True, 1, "", "tpu", "a100\x00", "x" * 100])
+def test_closed_gpu_allowlist(value):
+ with pytest.raises(ValueError, match="gpu"):
+ _backend().validate_gpu(value)
+
+
+def test_render_is_data_only_and_uses_documented_rest_payload(credentials):
+ ns = _controller()
+ assert base64.b64decode(ns["_CONFIG_B64"]).decode() == CONFIG
+ payload = ns["_launch_payload"]("ssh-ed25519 AAAATEST fixture", "soup-test")
+ assert payload["gpuTypeIds"] == ["NVIDIA A100-SXM4-80GB"]
+ assert payload["gpuCount"] == 1
+ assert payload["ports"] == ["22/tcp"]
+ assert payload["env"] == {"PUBLIC_KEY": "ssh-ed25519 AAAATEST fixture"}
+ assert "test-provider-secret" not in json.dumps(payload)
+ assert "not-a-real-private-key" not in json.dumps(payload)
+ assert payload["dockerEntrypoint"] == ["/bin/bash", "-lc"]
+ assert "sshd" in payload["dockerStartCmd"][0]
+
+
+ "output", ["/tmp/out", "../out", "a/../../out", ".", "out space", "a;exit"]
+)
+def test_unsafe_remote_outputs_are_refused(output):
+ with pytest.raises(ValueError, match="output_dir"):
+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir=output, soup_version="0.76.0")
+
+
+ "config", ["hf_token: secret\n", "x: {api_key: secret}\n", "# test-provider-secret\n"]
+)
+def test_inline_secrets_are_refused(config, credentials):
+ with pytest.raises(ValueError, match="credential"):
+ _backend().render_runpod_stub(config, gpu="a100", output_dir="out", soup_version="0.76.0")
+
+
+def test_malformed_config_is_refused_without_echoing_its_content():
+ with pytest.raises(ValueError, match="valid YAML") as error:
+ _backend().render_runpod_stub(
+ "private-value: [", gpu="a100", output_dir="out", soup_version="0.76.0",
+ )
+ assert "private-value" not in str(error.value)
+
+
+def test_modest_alias_reuse_is_preserved_as_config_data():
+ config = CONFIG + "settings: &settings {epochs: 1}\ncopy: *settings\n"
+ stub = _backend().render_runpod_stub(
+ config, gpu="a100", output_dir="out", soup_version="0.76.0",
+ )
+ namespace = {"__name__": "test_controller"}
+ exec(compile(stub, "soup_runpod_app.py", "exec"), namespace)
+ assert base64.b64decode(namespace["_CONFIG_B64"]).decode() == config
+
+
[email protected]("version", ["x; echo bad", "", "x\n", True])
+def test_bad_package_version_is_refused(version):
+ with pytest.raises(ValueError, match="soup_version"):
+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir="out", soup_version=version)
+
+
+def test_plan_rejects_path_escape_and_writes_atomically(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ backend = _backend()
+ for output in ("out", "../outside"):
+ if output == "out":
+ plan = backend.plan_runpod_run(
+ "soup.yaml", gpu="a100", output_dir=output, soup_version="0.76.0"
+ )
+ backend.write_stub(plan)
+ assert (tmp_path / plan.stub_path).read_text(encoding="utf-8") == plan.stub_text
+ else:
+ with pytest.raises(ValueError, match="output_dir"):
+ backend.plan_runpod_run(
+ "soup.yaml", gpu="a100", output_dir=output, soup_version="0.76.0"
+ )
+
+
+def test_config_and_output_symlinks_are_refused(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ target = tmp_path / "real.yaml"
+ target.write_text(CONFIG, encoding="utf-8")
+ (tmp_path / "soup.yaml").symlink_to(target)
+ with pytest.raises(ValueError):
+ _backend().plan_runpod_run("soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0")
+ (tmp_path / "soup.yaml").unlink()
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ (tmp_path / "actual").mkdir()
+ (tmp_path / "out").symlink_to(tmp_path / "actual", target_is_directory=True)
+ with pytest.raises(ValueError):
+ _backend().plan_runpod_run("soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0")
+
+
+ "failure",
+ [
+ None,
+ RuntimeError("secret from training"),
+ KeyboardInterrupt(),
+ subprocess.TimeoutExpired("ssh", 1),
+ SystemExit(2),
+ ],
+)
+def test_controller_terminates_and_confirms_even_after_failure(credentials, failure):
+ ns = _controller()
+ calls = []
+
+ def request(key, path, *, method="GET", payload=None):
+ calls.append((method, path))
+ if method == "POST":
+ return {"id": "pod-1"}
+ if method == "GET":
+ return None
+
+ def training(*args):
+ if failure is not None:
+ raise failure
+
+ ns["_request_json"] = request
+ ns["_wait_for_ssh"] = lambda *args: ("192.0.2.1", 2222)
+ ns["_train_and_copy"] = training
+ ns["_notice"] = lambda message: calls.append(("notice", message))
+ assert ns["main"]() == (0 if failure is None else 1)
+ assert ("DELETE", "/pods/pod-1") in calls
+ assert ("GET", "/pods/pod-1") in calls
+ assert "secret from training" not in str(calls)
+
+
+def test_cleanup_failure_cannot_report_success(credentials):
+ ns = _controller()
+ ns["_request_json"] = lambda *a, **kw: {"id": "pod-1"}
+ ns["_wait_for_ssh"] = lambda *args: ("192.0.2.1", 2222)
+ ns["_train_and_copy"] = lambda *args: None
+ ns["_terminate"] = lambda *args: (_ for _ in ()).throw(RuntimeError("secret"))
+ assert ns["main"]() == 1
+
+
+def test_uncertain_launch_is_reconciled_by_unique_name(credentials):
+ ns = _controller()
+ calls = []
+ name = []
+
+ def request(key, path, *, method="GET", payload=None):
+ calls.append((method, path))
+ if method == "POST":
+ name.append(payload["name"])
+ raise RuntimeError("lost create response")
+ if path == "/pods":
+ return [{"id": "pod-1", "name": name[0]}, {"id": "other", "name": "unrelated"}]
+ return None
+
+ ns["_request_json"] = request
+ assert ns["main"]() == 1
+ assert ("DELETE", "/pods/pod-1") in calls
+ assert ("DELETE", "/pods/other") not in calls
+
+
+def test_rest_auth_uses_header_and_refuses_redirects(monkeypatch):
+ ns = _controller()
+ seen = []
+
+ class Response(io.BytesIO):
+ status = 200
+
+ class Opener:
+ def open(self, request, timeout):
+ seen.append((request, timeout))
+ return Response(b'{"id":"pod-1"}')
+
+ def build_opener(*handlers):
+ assert len(handlers) == 1
+ assert isinstance(handlers[0], ns["_NoRedirect"])
+ assert handlers[0].redirect_request(
+ None, None, None, None, None, "https://evil.invalid",
+ ) is None
+ return Opener()
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", build_opener)
+ assert ns["_request_json"]("provider-secret", "/pods/pod-1") == {"id": "pod-1"}
+ request, timeout = seen[0]
+ assert request.full_url == "https://rest.runpod.io/v1/pods/pod-1"
+ assert request.get_header("Authorization") == "Bearer provider-secret"
+ assert "provider-secret" not in request.full_url
+ assert timeout == 60
+ assert (
+ ns["_NoRedirect"]().redirect_request(None, None, None, None, None, "https://evil.invalid")
+ is None
+ )
+
+
[email protected]("status", [401, 500, 302])
+def test_rest_error_details_never_expose_credentials(monkeypatch, status):
+ ns = _controller()
+
+ class Opener:
+ def open(self, request, timeout):
+ raise urllib.error.HTTPError(
+ request.full_url, status, "secret", {}, io.BytesIO(b"secret")
+ )
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ with pytest.raises(RuntimeError, match=f"HTTP {status}") as error:
+ ns["_request_json"]("secret", "/pods/pod-1")
+ assert "secret" not in str(error.value)
+
+
+def test_training_failure_or_copy_failure_is_not_success(credentials, monkeypatch, tmp_path):
+ ns = _controller()
+ monkeypatch.chdir(tmp_path)
+ calls = []
+
+ def run(argv, **kwargs):
+ calls.append((argv, kwargs))
+ return subprocess.CompletedProcess(argv, 0 if argv[0] == "ssh" else 1)
+
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ with pytest.raises(RuntimeError, match="copy"):
+ ns["_train_and_copy"]("192.0.2.1", 2222, str(credentials))
+ assert [item[0][0] for item in calls] == ["ssh", "ssh", "scp"]
+ assert "-P" in calls[2][0] and "2222" in calls[2][0]
+ script = calls[1][1]["input"]
+ assert "test-provider-secret" not in script
+ assert "soup-cli[train]==0.76.0" in script
+
+
[email protected]("submit,expected", [(False, 0), (True, 7)])
+def test_real_cli_plan_and_submit_seam(tmp_path, monkeypatch, submit, expected):
+ from soup_cli.cli import app
+
+ monkeypatch.chdir(tmp_path)
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ plans = []
+ monkeypatch.setattr(_backend(), "_RUNPOD_SUBMIT_OVERRIDE", lambda plan: plans.append(plan) or 7)
+ args = ["train", "--cloud", "runpod", "--gpu", "a100", "--config", "soup.yaml"]
+ result = CliRunner().invoke(app, args + (["--cloud-submit"] if submit else []))
+ assert result.exit_code == expected, (result.output, repr(result.exception))
+ assert bool(plans) == submit
+ assert (tmp_path / "soup_runpod_app.py").is_file()
+
+
+def test_submit_requires_credentials_before_spawning(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ plan = _backend().plan_runpod_run(
+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0"
+ )
+ with pytest.raises(RuntimeError, match="RUNPOD_API_KEY"):
+ _backend().submit_runpod_run(plan, env={})
diff --git a/tests/test_runpod_controller.py b/tests/test_runpod_controller.py
new file mode 100644
index 00000000..e6f91133
--- /dev/null
+++ b/tests/test_runpod_controller.py
@@ -0,0 +1,176 @@
+"""Lifecycle boundaries exercised by running rendered controller functions."""
+
+import io
+import subprocess
+import types
+import urllib.error
+from pathlib import Path
+
+import pytest
+
+from tests.test_runpod_cloud import CONFIG, _backend, _controller
+
+
+def credentials(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ key = tmp_path / "id_test"
+ key.write_text("not-a-real-private-key", encoding="utf-8")
+ key.with_suffix(".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")
+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")
+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(key))
+ return key
+
+
+def _clock(ns, step=1):
+ ticks = iter(range(0, 100000, step))
+ sleeps = []
+ ns["time"] = types.SimpleNamespace(monotonic=lambda: next(ticks), sleep=sleeps.append)
+ return sleeps
+
+
+def test_delete_polls_past_stopped_state_until_actual_absence():
+ ns = _controller()
+ reads = iter([{"desiredStatus": "EXITED"}, None])
+ calls = []
+ sleeps = _clock(ns)
+
+ def request(*args, method="GET", **kwargs):
+ calls.append(method)
+ return None if method == "DELETE" else next(reads)
+
+ ns["_request_json"] = request
+ ns["_terminate"]("secret", "pod-1")
+ assert calls == ["DELETE", "GET", "DELETE", "GET"]
+ assert sleeps == [5]
+
+
+def test_deletion_timeout_is_a_failure():
+ ns = _controller()
+ _clock(ns, 100)
+ ns["_request_json"] = lambda *args, **kwargs: {"desiredStatus": "EXITED"}
+ with pytest.raises(RuntimeError, match="not confirmed"):
+ ns["_terminate"]("secret", "pod-1")
+
+
+ "body",
+ [b"", b"null", b"bad", b"x" * (1024 * 1024 + 1)],
+ ids=["empty", "null", "malformed", "oversized"],
+)
+def test_only_http_404_can_establish_absence(monkeypatch, body):
+ ns = _controller()
+
+ class Opener:
+ def open(self, request, timeout):
+ return io.BytesIO(body)
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ with pytest.raises(RuntimeError):
+ ns["_request_json"]("secret", "/pods/pod-1")
+
+
+def test_rest_404_get_and_empty_delete_are_accepted(monkeypatch):
+ ns = _controller()
+
+ class Opener:
+ def open(self, request, timeout):
+ if request.method == "DELETE":
+ return io.BytesIO(b"")
+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ assert ns["_request_json"]("secret", "/pods/pod-1") is None
+ assert ns["_request_json"]("secret", "/pods/pod-1", method="DELETE") is None
+ with pytest.raises(RuntimeError, match="HTTP 404"):
+ ns["_request_json"]("secret", "/pods", method="POST", payload={})
+
+
+ "record",
+ [
+ None,
+ {"desiredStatus": "EXITED"},
+ {"publicIp": "127.0.0.1", "portMappings": {"22": 1234}},
+ {"publicIp": "169.254.169.254", "portMappings": {"22": 1234}},
+ ],
+)
+def test_ssh_refuses_missing_stopped_or_private_addresses(record):
+ ns = _controller()
+ ns["_request_json"] = lambda *args: record
+ with pytest.raises(RuntimeError):
+ ns["_wait_for_ssh"]("secret", "pod-1")
+
+
+def test_ssh_waits_for_real_port_and_has_a_deadline():
+ ns = _controller()
+ reads = iter([{}, {"publicIp": "8.8.8.8", "portMappings": {"22": 2222}}])
+ ns["_request_json"] = lambda *args: next(reads)
+ sleeps = _clock(ns)
+ assert ns["_wait_for_ssh"]("secret", "pod-1") == ("8.8.8.8", 2222)
+ assert sleeps == [5]
+ _clock(ns, 1000)
+ with pytest.raises(RuntimeError, match="15 minutes"):
+ ns["_wait_for_ssh"]("secret", "pod-1")
+
+
+def test_training_nonzero_is_not_copied_or_retried(monkeypatch, credentials):
+ ns = _controller()
+ calls = []
+
+ def run(argv, **kwargs):
+ calls.append(argv)
+ return subprocess.CompletedProcess(argv, 1 if "input" in kwargs else 0)
+
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ with pytest.raises(RuntimeError, match="training failed"):
+ ns["_train_and_copy"]("192.0.2.1", 2222, str(credentials))
+ assert [argv[0] for argv in calls] == ["ssh", "ssh"]
+
+
+def test_submit_keeps_one_controller_through_repeated_interrupts(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ plan = _backend().plan_runpod_run(
+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0"
+ )
+ waits = [KeyboardInterrupt(), SystemExit(2), 0]
+ spawns = []
+
+ class Process:
+ def wait(self):
+ value = waits.pop(0)
+ if isinstance(value, BaseException):
+ raise value
+ return value
+
+ monkeypatch.setattr(subprocess, "Popen", lambda *a, **kw: spawns.append((a, kw)) or Process())
+ assert (
+ _backend().submit_runpod_run(
+ plan, env={"RUNPOD_API_KEY": "test", "RUNPOD_SSH_PRIVATE_KEY": "key"}
+ )
+ == 0
+ )
+ assert len(spawns) == 1
+ assert waits == []
+
+
+def test_missing_ssh_or_existing_output_never_launches(credentials):
+ ns = _controller()
+ calls = []
+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))
+ ns["shutil"].which = lambda command: None
+ assert ns["main"]() == 2
+ ns["shutil"].which = lambda command: command
+ Path("out").mkdir()
+ assert ns["main"]() == 2
+ assert not calls
+
+
+def test_local_copy_destination_refuses_symlinks(credentials, tmp_path):
+ ns = _controller()
+ (tmp_path / "real").mkdir()
+ (tmp_path / "out").symlink_to(tmp_path / "real", target_is_directory=True)
+ with pytest.raises(RuntimeError, match="symlinks"):
+ ns["_local_destination"]()
diff --git a/tests/test_runpod_hardening.py b/tests/test_runpod_hardening.py
new file mode 100644
index 00000000..7614dfe7
--- /dev/null
+++ b/tests/test_runpod_hardening.py
@@ -0,0 +1,489 @@
+"""Run rendered RunPod controllers through failure and process boundaries."""
+
+import http.client
+import io
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import types
+import urllib.error
+from pathlib import Path
+
+import pytest
+
+from tests.test_runpod_cloud import CONFIG, _backend, _controller
+from tests.test_runpod_controller import _clock
+
+
+def credentials(tmp_path, monkeypatch):
+ monkeypatch.chdir(tmp_path)
+ key = tmp_path / "id_test"
+ key.write_text("not-a-real-private-key", encoding="utf-8")
+ Path(str(key) + ".pub").write_text("ssh-ed25519 AAAATEST fixture", encoding="utf-8")
+ monkeypatch.setenv("RUNPOD_API_KEY", "test-provider-secret")
+ monkeypatch.setenv("RUNPOD_SSH_PRIVATE_KEY", str(key))
+ return key
+
+
[email protected]("method", ["DELETE", "GET"])
+ http.client.IncompleteRead(b"private-response"),
+ http.client.BadStatusLine("private-response"),
+ http.client.LineTooLong("private-response"),
+ ConnectionResetError("private-response"),
+])
+def test_cleanup_retries_transient_http_failures(monkeypatch, method, error):
+ ns = _controller()
+ sleeps = _clock(ns)
+ calls = []
+ failed = False
+
+ class Opener:
+ def open(self, request, timeout):
+ nonlocal failed
+ calls.append(request.method)
+ if request.method == method and not failed:
+ failed = True
+ raise error
+ if request.method == "GET":
+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)
+ return io.BytesIO(b"")
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ ns["_terminate"]("provider-secret", "pod-1")
+ expected = ["DELETE", "DELETE", "GET"] if method == "DELETE" else [
+ "DELETE", "GET", "DELETE", "GET",
+ ]
+ assert calls == expected
+ assert sleeps == [5]
+
+
[email protected]("status", [400, 401, 403, 422, 429])
+def test_definite_create_rejection_has_reason_and_no_false_cleanup(
+ credentials, monkeypatch, status,
+):
+ ns = _controller()
+ _clock(ns, 100)
+ calls, notices = [], []
+
+ class Opener:
+ def open(self, request, timeout):
+ calls.append(request.method)
+ raise urllib.error.HTTPError(
+ request.full_url, status, "private-response", {}, io.BytesIO(b"provider-secret"),
+ )
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ ns["_notice"] = notices.append
+ assert ns["main"]() == 1
+ assert calls == ["POST"]
+ assert any(f"RunPod API POST failed: HTTP {status}" in line for line in notices)
+ assert "CRITICAL" not in str(notices)
+ assert "termination verified" not in str(notices)
+ assert "private-response" not in str(notices)
+ assert "provider-secret" not in str(notices)
+
+
[email protected]("failure", [500, 503, 408, "timeout", "incomplete", "json"])
+def test_ambiguous_create_still_recovers_only_owned_pod(credentials, monkeypatch, failure):
+ ns = _controller()
+ _clock(ns)
+ calls, notices = [], []
+ run_name = None
+
+ class Opener:
+ def open(self, request, timeout):
+ nonlocal run_name
+ path = request.full_url.removeprefix(ns["_API_BASE"])
+ calls.append((request.method, path))
+ if request.method == "POST":
+ run_name = json.loads(request.data)["name"]
+ if isinstance(failure, int):
+ raise urllib.error.HTTPError(request.full_url, failure, "secret", {}, None)
+ if failure == "timeout":
+ raise TimeoutError("provider-secret")
+ if failure == "incomplete":
+ raise http.client.IncompleteRead(b"provider-secret")
+ return io.BytesIO(b"invalid json: provider-secret")
+ if path == "/pods":
+ return io.BytesIO(json.dumps([
+ {"id": "owned", "name": run_name}, {"id": "other", "name": "other"},
+ ]).encode())
+ if request.method == "DELETE":
+ return io.BytesIO(b"")
+ raise urllib.error.HTTPError(request.full_url, 404, "absent", {}, None)
+
+ monkeypatch.setattr(ns["urllib"].request, "build_opener", lambda *handlers: Opener())
+ ns["_notice"] = notices.append
+ assert ns["main"]() == 1
+ assert calls == [("POST", "/pods"), ("GET", "/pods"),
+ ("DELETE", "/pods/owned"), ("GET", "/pods/owned")]
+ assert "RunPod termination verified." in notices
+ assert "provider-secret" not in str(notices)
+
+
[email protected]("stage,reason", [
+ ("ssh", "did not expose SSH within 15 minutes"),
+ ("training", "remote training failed"),
+ ("copy", "artifact copy failed"),
+])
+def test_main_reports_safe_controller_failure_reason(credentials, monkeypatch, stage, reason):
+ ns = _controller()
+ _clock(ns)
+ notices, deletes = [], []
+
+ def request(*args, method="GET", **kwargs):
+ if method == "POST":
+ return {"id": "pod-1"}
+ if method == "DELETE":
+ deletes.append(args[1])
+ return {} if stage == "ssh" and not deletes else None
+
+ def run(argv, **kwargs):
+ failed = (stage == "training" and "input" in kwargs) or (
+ stage == "copy" and argv[0] == "scp"
+ )
+ return subprocess.CompletedProcess(argv, int(failed))
+
+ ns["_request_json"] = request
+ ns["_notice"] = notices.append
+ if stage != "ssh":
+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ assert ns["main"]() == 1
+ assert any(reason in line for line in notices)
+ assert deletes == ["/pods/pod-1"]
+ assert "RunPod termination verified." in notices
+
+
[email protected]("error", [RuntimeError("private-error"), ValueError("private-error")])
+def test_main_does_not_print_untrusted_exception_text(credentials, error):
+ ns = _controller()
+ notices = []
+ ns["_request_json"] = lambda *a, method="GET", **kw: {"id": "pod-1"} if (
+ method == "POST"
+ ) else None
+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)
+
+ def train(*args):
+ raise error
+
+ ns["_train_and_copy"] = train
+ ns["_notice"] = notices.append
+ assert ns["main"]() == 1
+ assert "private-error" not in str(notices)
+ assert "RunPod termination verified." in notices
+
+
+def test_output_appearing_during_training_prevents_scp(credentials, monkeypatch):
+ ns = _controller()
+ calls = []
+
+ def run(argv, **kwargs):
+ calls.append(argv)
+ if "input" in kwargs:
+ Path("out").mkdir()
+ Path("out/keep").write_text("original", encoding="utf-8")
+ return subprocess.CompletedProcess(argv, 0)
+
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ with pytest.raises(RuntimeError, match="already exists"):
+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))
+ assert [argv[0] for argv in calls] == ["ssh", "ssh"]
+ assert Path("out/keep").read_text(encoding="utf-8") == "original"
+
+
[email protected]("cleanup_fails", [False, True])
+def test_cleanup_masks_ctrl_c_and_restores_previous_handler(credentials, cleanup_fails):
+ ns = _controller()
+ previous = object()
+ handlers = [previous]
+ calls = []
+
+ def install(sig, handler):
+ assert sig == "INT"
+ old = handlers[-1]
+ handlers.append(handler)
+ return old
+
+ def terminate(*args):
+ assert handlers[-1] == "IGNORE"
+ calls.append(args)
+ if cleanup_fails:
+ raise RuntimeError("private-error")
+
+ def interrupted(*args):
+ raise KeyboardInterrupt
+
+ ns["signal"] = types.SimpleNamespace(SIGINT="INT", SIG_IGN="IGNORE", signal=install)
+ ns["_request_json"] = lambda *a, **kw: {"id": "pod-1"}
+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)
+ ns["_train_and_copy"] = interrupted
+ ns["_terminate"] = terminate
+ assert ns["main"]() == 1
+ assert calls == [("test-provider-secret", "pod-1")]
+ assert handlers == [previous, "IGNORE", previous]
+
+
+def test_readiness_retries_before_training_once(credentials, monkeypatch):
+ ns = _controller()
+ sleeps = _clock(ns)
+ calls = []
+
+ def run(argv, **kwargs):
+ calls.append((argv, kwargs))
+ return subprocess.CompletedProcess(argv, 255 if len(calls) == 1 else 0)
+
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))
+ assert [argv[0] for argv, _ in calls] == ["ssh", "ssh", "ssh", "scp"]
+ assert calls[0][0][-1] == calls[1][0][-1] == "true"
+ assert sum("input" in kwargs for _, kwargs in calls) == 1
+ assert sleeps == [5]
+
+
[email protected]("pod_id", ["../x", "x" * 65, 17, None, "", True])
+def test_invalid_pod_id_never_reaches_delete(pod_id):
+ ns = _controller()
+ calls = []
+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))
+ with pytest.raises(RuntimeError, match="invalid Pod ID"):
+ ns["_terminate"]("secret", pod_id)
+ assert calls == []
+
+
+ "ecdsa-sha2-nistp256 AAAATEST", "ssh-ed25519 AAAA\nssh-rsa BBBB", "garbage",
+])
+def test_invalid_public_key_never_allocates(credentials, key):
+ Path(str(credentials) + ".pub").write_text(key, encoding="utf-8")
+ ns = _controller()
+ calls = []
+ ns["_request_json"] = lambda *a, **kw: calls.append((a, kw))
+ assert ns["main"]() == 2
+ assert calls == []
+
+
[email protected]("kind", ["ed25519", "rsa"])
+def test_supported_public_keys_launch_secure_cloud(credentials, kind):
+ Path(str(credentials) + ".pub").write_text(f"ssh-{kind} AAAATEST fixture", encoding="utf-8")
+ ns = _controller()
+ payloads = []
+
+ def request(*args, method="GET", payload=None):
+ if method == "POST":
+ payloads.append(payload)
+ return {"id": "pod-1"}
+
+ ns["_request_json"] = request
+ ns["_wait_for_ssh"] = lambda *args: ("8.8.8.8", 2222)
+ ns["_train_and_copy"] = lambda *args: None
+ assert ns["main"]() == 0
+ assert len(payloads) == 1
+ assert payloads[0]["cloudType"] == "SECURE"
+ assert payloads[0]["env"] == {"PUBLIC_KEY": f"ssh-{kind} AAAATEST fixture"}
+
+
+def test_realpath_escape_is_refused_before_allocation(credentials, monkeypatch, tmp_path):
+ ns = _controller()
+ realpath = os.path.realpath
+ monkeypatch.setattr(ns["os"].path, "realpath", lambda path, **kw: (
+ str(tmp_path.parent / "escaped") if str(path) == "out" else realpath(path, **kw)
+ ))
+ with pytest.raises(RuntimeError, match="stay under cwd"):
+ ns["_local_destination"]()
+
+
[email protected]("output", ["-out", "a/-out", "a/-b/out"])
+def test_option_like_output_component_refused_when_rendering(output):
+ with pytest.raises(ValueError, match="output_dir"):
+ _backend().render_runpod_stub(CONFIG, gpu="a100", output_dir=output, soup_version="0.76.0")
+
+
[email protected]("output", ["-out", "a/-out", "a/-b/out"])
+def test_option_like_output_component_refused_by_controller(credentials, output):
+ ns = _controller()
+ ns["_OUTPUT"] = output
+ with pytest.raises(RuntimeError, match="output"):
+ ns["_local_destination"]()
+
+
+def _plan(tmp_path):
+ (tmp_path / "soup.yaml").write_text(CONFIG, encoding="utf-8")
+ return _backend().plan_runpod_run(
+ "soup.yaml", gpu="a100", output_dir="out", soup_version="0.76.0",
+ )
+
+
+def test_submit_isolates_python_and_filters_environment(credentials, monkeypatch, tmp_path):
+ plan = _plan(tmp_path)
+ environment = {name: f"value-{name}" for name in (
+ "RUNPOD_API_KEY", "RUNPOD_SSH_PRIVATE_KEY", "PATH", "HOME", "USERPROFILE", "SYSTEMROOT",
+ )}
+ untrusted = {name: "private-value" for name in (
+ "PYTHONPATH", "PYTHONHOME", "HF_TOKEN", "WANDB_API_KEY", "LD_PRELOAD",
+ )}
+ spawns = []
+ monkeypatch.setattr(subprocess, "Popen", lambda argv, **kw: (
+ spawns.append((argv, kw)) or types.SimpleNamespace(wait=lambda: 7)
+ ))
+ assert _backend().submit_runpod_run(plan, env={**environment, **untrusted}) == 7
+ assert spawns == [([sys.executable, "-I", plan.stub_path], {"env": environment})]
+ assert plan.run_command == "python -I soup_runpod_app.py"
+
+
+ "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY",
+ "http_proxy", "https_proxy", "all_proxy", "no_proxy",
+ "TEMP", "TMP", "TMPDIR", "SYSTEMDRIVE", "WINDIR", "SSH_AUTH_SOCK",
+])
+def test_controller_forwards_network_temp_and_agent_settings(
+ credentials, monkeypatch, tmp_path, name,
+):
+ plan = _plan(tmp_path)
+ supplied = {
+ "RUNPOD_API_KEY": "fixture-key", "RUNPOD_SSH_PRIVATE_KEY": str(credentials),
+ name: "setting-value", "HF_TOKEN": "not-for-this-controller",
+ }
+ expected = {key: value for key, value in supplied.items() if key != "HF_TOKEN"}
+ # Check exact POSIX spellings too: on Windows a differently cased proxy name
+ # can otherwise conceal a missing allow-list entry after normalisation.
+ assert _backend()._controller_environment(supplied, case_insensitive=False) == expected
+ launches = []
+ monkeypatch.setattr(subprocess, "Popen", lambda argv, **kw: (
+ launches.append((argv, kw)) or types.SimpleNamespace(wait=lambda: 0)
+ ))
+ assert _backend().submit_runpod_run(plan, env=supplied) == 0
+ assert launches == [([sys.executable, "-I", plan.stub_path], {"env": expected})]
+
+
[email protected]("case_insensitive", [False, True, None])
+def test_controller_environment_matches_windows_names_only_on_windows(case_insensitive):
+ supplied = {"Path": "bin", "SystemRoot": "windows", "Https_Proxy": "proxy", "OTHER": "no"}
+ expected = {key: value for key, value in supplied.items() if key != "OTHER"}
+ windows = os.name == "nt" if case_insensitive is None else case_insensitive
+ assert _backend()._controller_environment(
+ supplied, case_insensitive=case_insensitive,
+ ) == (expected if windows else {})
+
+
[email protected]("stream", ["stdout", "stderr"])
[email protected]("mode", ["ascii", "native"])
+def test_isolated_controller_replaces_unencodable_output(tmp_path, monkeypatch, stream, mode):
+ monkeypatch.chdir(tmp_path)
+ plan = _plan(tmp_path)
+ _backend().write_stub(plan)
+ # Run the rendered controller under -I, not a replica of its bootstrap.
+ # The narrow mode models Windows redirected output on every platform.
+ harness = tmp_path / "encoding_harness.py"
+ harness.write_text(
+ "import sys\n"
+ "stream, mode, stub = sys.argv[1:]\n"
+ "if mode == 'ascii':\n"
+ " for handle in (sys.stdout, sys.stderr):\n"
+ " handle.reconfigure(encoding='ascii', errors='strict')\n"
+ "namespace = {'__name__': 'controller_under_test'}\n"
+ "with open(stub, encoding='utf-8') as source:\n"
+ " exec(compile(source.read(), stub, 'exec'), namespace)\n"
+ "getattr(sys, stream).write('controller line: \\u2588 \\u00e9 \\u65e5\\n')\n"
+ "getattr(sys, stream).flush()\n",
+ encoding="utf-8",
+ )
+ environment = {name: value for name, value in os.environ.items() if name.upper() in {
+ "PATH", "SYSTEMROOT", "WINDIR", "SYSTEMDRIVE", "TEMP", "TMP", "TMPDIR",
+ }}
+ result = subprocess.run(
+ [sys.executable, "-I", str(harness), stream, mode, plan.stub_path],
+ env=environment, cwd=tmp_path, capture_output=True, timeout=30,
+ )
+ assert result.returncode == 0, result.stderr
+ output = getattr(result, stream).decode("ascii" if mode == "ascii" else "utf-8", "replace")
+ assert "controller line:" in output
+ if mode == "ascii":
+ assert "controller line: ? ? ?" in output
+ assert "UnicodeEncodeError" not in result.stderr.decode("utf-8", "replace")
+
+
+def test_actual_child_cannot_import_parent_pythonpath_or_read_unrelated_secret(
+ credentials, tmp_path,
+):
+ plan = _plan(tmp_path)
+ _backend().write_stub(plan)
+ # Execute an inert probe in the same process-launch path; no provider traffic.
+ Path(plan.stub_path).write_text(
+ "import importlib.util, json, os, sys\n"
+ "from pathlib import Path\n"
+ "Path('child.json').write_text(json.dumps({"
+ "'isolated': sys.flags.isolated, 'secret': os.environ.get('HF_TOKEN'), "
+ "'injected': importlib.util.find_spec('parent_injection') is not None, "
+ "'key': os.environ.get('RUNPOD_API_KEY')}))\n", encoding="utf-8",
+ )
+ (tmp_path / "parent_injection.py").write_text(
+ "raise AssertionError('injected')", encoding="utf-8",
+ )
+ environment = {name: os.environ[name] for name in ("PATH", "SYSTEMROOT") if name in os.environ}
+ environment.update(RUNPOD_API_KEY="fixture-key", RUNPOD_SSH_PRIVATE_KEY=str(credentials),
+ PYTHONPATH=str(tmp_path), HF_TOKEN="private-parent-secret")
+ assert _backend().submit_runpod_run(plan, env=environment) == 0
+ assert json.loads(Path("child.json").read_text(encoding="utf-8")) == {
+ "isolated": 1, "secret": None, "injected": False, "key": "fixture-key",
+ }
+
+
[email protected]("failure", [None, "training", "copy", "interrupt"])
+def test_ssh_uses_per_run_hosts_and_only_selected_identity(
+ credentials, monkeypatch, tmp_path, failure,
+):
+ ns = _controller()
+ scratch = tmp_path / "ssh scratch"
+ scratch.mkdir()
+ user_hosts = tmp_path / ".ssh/known_hosts"
+ user_hosts.parent.mkdir()
+ user_hosts.write_text("keep user hosts", encoding="utf-8")
+ monkeypatch.setenv("HOME", str(tmp_path))
+ monkeypatch.setenv("USERPROFILE", str(tmp_path))
+ ns["tempfile"] = types.SimpleNamespace(TemporaryDirectory=lambda **kw: (
+ tempfile.TemporaryDirectory(dir=scratch, **kw)
+ ), mkdtemp=lambda **kw: tempfile.mkdtemp(dir=scratch, **kw))
+ all_hosts = []
+ for _ in range(2):
+ hosts, calls = [], []
+
+ def run(argv, **kwargs):
+ calls.append(argv)
+ assert "IdentitiesOnly=yes" in argv
+ options = [arg for arg in argv if arg.startswith("UserKnownHostsFile=")]
+ assert len(options) == 1
+ host_file = Path(options[0].split("=", 1)[1].strip('"'))
+ assert host_file.parent.is_dir()
+ assert host_file != user_hosts
+ hosts.append(host_file)
+ host_file.touch()
+ if "input" in kwargs and failure == "interrupt":
+ raise KeyboardInterrupt
+ failed = ("input" in kwargs and failure == "training") or (
+ argv[0] == "scp" and failure == "copy"
+ )
+ if argv[0] == "scp":
+ assert "--" in argv
+ delimiter = argv.index("--")
+ assert argv[delimiter + 1:] == ["[email protected]:/workspace/soup/out", "."]
+ return subprocess.CompletedProcess(argv, int(failed))
+
+ monkeypatch.setattr(ns["subprocess"], "run", run)
+ if failure:
+ with pytest.raises(KeyboardInterrupt if failure == "interrupt" else RuntimeError):
+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))
+ else:
+ ns["_train_and_copy"]("8.8.8.8", 2222, str(credentials))
+ assert len(set(hosts)) == 1
+ assert not hosts[0].parent.exists()
+ assert [argv[0] for argv in calls] == (
+ ["ssh", "ssh"] if failure in {"training", "interrupt"} else ["ssh", "ssh", "scp"]
+ )
+ all_hosts.append(hosts[0])
+ assert len(set(all_hosts)) == 2
+ assert user_hosts.read_text(encoding="utf-8") == "keep user hosts"
diff --git a/tests/test_yaml_reads_are_limited.py b/tests/test_yaml_reads_are_limited.py
index f09cd9d6..4f6fa5a3 100644
--- a/tests/test_yaml_reads_are_limited.py
+++ b/tests/test_yaml_reads_are_limited.py
@@ -41,6 +41,7 @@ _NOT_PARSING = frozenset({"safe_dump", "dump", "YAMLError"})
YAML_USERS = {
"autopilot/generate_config.py": ("safe_dump",),
"cans/pack.py": ("safe_dump",),
+ "cloud/runpod.py": ("YAMLError",),
"commands/adapters.py": ("YAMLError",),
"commands/deploy.py": ("YAMLError",),
"commands/push.py": ("YAMLError",),
@@ -378,6 +379,12 @@ def _quant_config(text):
return load_quant_config(_write("quant.yaml", text))
+def _runpod(text):
+ from soup_cli.cloud.runpod import render_runpod_stub
+
+ return render_runpod_stub(text, gpu="a100", output_dir="out", soup_version="0.76.0")
+
+
def _sweep(text):
from soup_cli.utils.sweep_config import parse_sweep_yaml
@@ -386,6 +393,7 @@ def _sweep(text):
#: reader id -> (driver, how the refusal names the document)
_RAISING_READERS = {
+ "cloud/runpod.py": (_runpod, "RunPod config"),
"commands/plan.py": (_plan, "config 'soup.yaml'"),
"eval/gate.py": (_gate_suite, "eval-gate suite 'gate.yaml'"),
"eval/judge.py": (_rubric, "Rubric file rubric.yaml"),
@@ -586,6 +594,10 @@ def _cap(module: str, name: str) -> int:
#: reader id -> (module, name of its cap, how it words a file one byte over the cap)
_READER_CAPS = {
+ "cloud/runpod.py": (
+ "cloud.runpod", "_MAX_CONFIG_BYTES",
+ "RunPod config is larger than {cap} bytes; refusing",
+ ),
"commands/plan.py": (
"utils.yaml_limits", "MAX_CONFIG_BYTES",
"config 'soup.yaml' is larger than {cap} bytes; refusing",
@@ -708,6 +720,7 @@ def test_license_frontmatter_falls_back_one_byte_over_its_cap(tmp_path, monkeypa
#: Every call of the limited loader names its cap: a module constant, never a number.
_CAP_NAMES = {
"cans/unpack.py": {"MAX_CONFIG_BYTES", "MAX_MANIFEST_BYTES"},
+ "cloud/runpod.py": {"_MAX_CONFIG_BYTES"},
"commands/adapters.py": {"MAX_CONFIG_BYTES"},
"commands/deploy.py": {"MAX_CONFIG_BYTES"},
"commands/plan.py": {"MAX_CONFIG_BYTES"},