Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

fix(serve): require --tool-auth-token for the python and bash tool routes on every bind (#2174)

Alpamys Makazhan38aded14c0d6
diff --git a/changelog.d/0.76.0/2174.changed.md b/changelog.d/0.76.0/2174.changed.md
new file mode 100644
index 00000000..945b2d40
--- /dev/null
+++ b/changelog.d/0.76.0/2174.changed.md
@@ -0,0 +1 @@
+- **`soup serve`: `/v1/tools/python` and `/v1/tools/bash` require `--tool-auth-token` on every bind, as they already did beyond loopback.** A server started without a token now answers both routes with 401 on a loopback bind too, before the request's content is used; the reply names the fix (start `soup serve` with `--tool-auth-token <secret>` or set `SOUP_TOOL_AUTH_TOKEN`, and send `Authorization: Bearer <secret>`). `--tool-auth-token` now also reads `SOUP_TOOL_AUTH_TOKEN`, which keeps the token off the command line. `/v1/tools/web_search`, `/v1/thumbs`, the adapter routes and the generation routes are unchanged, and `soup serve` still starts without a token on loopback. To keep calling the two routes after upgrading, pass the token to the server and send it with each request (#2174)
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index b3ee895a..c7f8075f 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -3448,7 +3448,7 @@ Usage: `soup serve [OPTIONS]`
| `--bank-strength` | float | `1.0` | | Vector-bank delta strength multiplier. Ignored when --bank is unset. |
| `--mole` | text | | | Path to a MoLE training output dir (mole_gate.pt + mole_manifest.json). Serves the base + N frozen task LoRAs with per-token gate blending at decode time. Requires --backend transformers; not combinable with --bank / --steer / --adapters. (v0.71.17 #259) |
| `--kv-cache-type` | text | | | KV-cache type for decoding: q8_0 (8-bit quantized, needs hqq) / bf16 / f16 (cache dtype) / fp8 (vLLM+Hopper only). Transformers backend only — vLLM / SGLang routing is in the blocked tail. v0.71.14 (#140). |
-| `--tool-auth-token` | text | | | Require 'Authorization: Bearer &lt;token>' on the code-exec tool endpoints (/v1/tools/python, /v1/tools/bash, /v1/tools/web_search). Strongly recommended whenever --host is not loopback, since those endpoints run caller-supplied Python in a best-effort sandbox. |
+| `--tool-auth-token` | text | | | Require 'Authorization: Bearer &lt;token>' on the tool endpoints (/v1/tools/python, /v1/tools/bash, /v1/tools/web_search). Without it, /v1/tools/python and /v1/tools/bash answer 401 on every bind and a non-loopback --host is refused. Env: `SOUP_TOOL_AUTH_TOKEN`. |
## `soup ship`
diff --git a/docs/serving-and-export.md b/docs/serving-and-export.md
index 0c216050..a6693437 100644
--- a/docs/serving-and-export.md
+++ b/docs/serving-and-export.md
@@ -996,7 +996,7 @@ Three POST routes are now available on `soup serve`:
- **`POST /v1/tools/bash`** — runs a bash command with strict OS-level network
isolation (`unshare` on supported Linux runtimes, `sandbox-exec` on macOS).
- Enforces mandatory Bearer token authentication when bound to a non-loopback host,
+ Enforces the `--tool-auth-token` Bearer token on every bind,
a 5-second wall-clock timeout, a minimal secret-scrubbed environment, and a combined
10KB stdout/stderr streaming kill limit. Returns the real stdout, stderr, exit code,
and timeout state. This is not a filesystem sandbox: the child retains the server
@@ -1004,6 +1004,27 @@ Three POST routes are now available on `soup serve`:
HTTP 501 when strict OS isolation is unavailable (including on Windows or restricted
Linux containers).
+**`/v1/tools/python` and `/v1/tools/bash` need `--tool-auth-token` on every bind.** A
+server started without it answers both routes with 401 on loopback too, as it already did
+on any other bind, and the reply names the fix. The token is checked before the OS-level
+isolation requirement above, so without it the reply is 401 even on a host where the route
+would answer 501; with it, the isolation rule decides. Start the server with a token, from
+the option or from `SOUP_TOOL_AUTH_TOKEN` (which keeps it off the command line and out of
+the shell history), and send it as a Bearer header:
+
+```bash
+export SOUP_TOOL_AUTH_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
+soup serve --model ./output
+
+curl -X POST http://127.0.0.1:8000/v1/tools/python \
+ -H "Authorization: Bearer $SOUP_TOOL_AUTH_TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{"code": "print(1)"}'
+```
+
+`/v1/tools/web_search`, `/v1/thumbs` and the adapter routes keep their rule: on a loopback
+bind they take a token only when the server has one.
+
Tool routes, `/v1/thumbs` and adapter activate/deactivate accept requests only when the
`Host` header names the bound address (any loopback name for a loopback bind) and any
`Origin` header names the same; otherwise they answer 421 or 403. That check runs first.
diff --git a/src/soup_cli/commands/serve.py b/src/soup_cli/commands/serve.py
index 53575eb7..bafdd47e 100644
--- a/src/soup_cli/commands/serve.py
+++ b/src/soup_cli/commands/serve.py
@@ -309,11 +309,12 @@ def serve(
tool_auth_token: Optional[str] = typer.Option(
None,
"--tool-auth-token",
+ envvar="SOUP_TOOL_AUTH_TOKEN",
help=(
- "Require 'Authorization: Bearer <token>' on the code-exec tool "
- "endpoints (/v1/tools/python, /v1/tools/bash, /v1/tools/web_search). "
- "Strongly recommended whenever --host is not loopback, since those "
- "endpoints run caller-supplied Python in a best-effort sandbox."
+ "Require 'Authorization: Bearer <token>' on the tool endpoints "
+ "(/v1/tools/python, /v1/tools/bash, /v1/tools/web_search). Without it, "
+ "/v1/tools/python and /v1/tools/bash answer 401 on every bind and a "
+ "non-loopback --host is refused."
),
),
):
@@ -1245,6 +1246,13 @@ _METRICS_BACKENDS = frozenset({"transformers", "vllm"})
# The binds `soup serve` accepts without --tool-auth-token (the same three
# spellings the refusal in serve() and _check_tool_auth compare against).
_LOOPBACK_BINDS = frozenset({"127.0.0.1", "localhost", "::1"})
+# What /v1/tools/python and /v1/tools/bash answer, on any bind, when the server
+# was started without --tool-auth-token: the 401 detail names the fix.
+_CODE_TOOL_TOKEN_REQUIRED = (
+ "Authentication required: /v1/tools/python and /v1/tools/bash need a token on "
+ "every bind. Start soup serve with --tool-auth-token <secret> (or set "
+ "SOUP_TOOL_AUTH_TOKEN) and send 'Authorization: Bearer <secret>'."
+)
def _non_loopback_notices(host: str, backend: str) -> list[str]:
@@ -1725,12 +1733,13 @@ def _create_app(
"""Create the FastAPI application with OpenAI-compatible endpoints.
Args:
- auth_token: optional Bearer-token gate for the v0.53.7 tool
- endpoints (``/v1/tools/python``, ``/v1/tools/bash`` and
- ``/v1/tools/web_search``).
- When ``None`` (default), endpoints inherit the server's
- loopback-only CORS trust boundary. When set, callers must
- supply ``Authorization: Bearer <token>``.
+ auth_token: Bearer token for the v0.53.7 tool endpoints
+ (``/v1/tools/python``, ``/v1/tools/bash`` and
+ ``/v1/tools/web_search``), ``/v1/thumbs`` and the adapter routes.
+ When set, callers must supply ``Authorization: Bearer <token>``.
+ When ``None`` (default), ``/v1/tools/python`` and ``/v1/tools/bash``
+ answer 401 on every bind; the other routes take no token on a
+ loopback bind and answer 401 on any other.
"""
import secrets as _secrets
import threading as _threading
@@ -1789,6 +1798,12 @@ def _create_app(
status_code=401, detail="Invalid or missing bearer token"
)
+ def _check_code_tool_auth(authorization: Optional[str]) -> None:
+ """The python and bash tool routes take a token on every bind, loopback included."""
+ if not auth_token:
+ raise HTTPException(status_code=401, detail=_CODE_TOOL_TOKEN_REQUIRED)
+ _check_tool_auth(authorization)
+
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
@@ -1824,7 +1839,8 @@ def _create_app(
# None of the three covers a non-browser client, which sends no Origin at
# all and can name the bind address in Host — for those, on a non-loopback
# bind, the bearer token checked by `_check_tool_auth` is the only thing
- # protecting these routes.
+ # protecting these routes. The python and bash tool routes take that token
+ # on every bind, loopback included (`_check_code_tool_auth`).
app.add_middleware(
CORSMiddleware,
allow_origin_regex=r"^https?://(localhost|127\.0\.0\.1)(:\d+)?$",
@@ -2308,7 +2324,7 @@ def _create_app(
payload: dict,
authorization: Optional[str] = Header(default=None),
) -> dict:
- _check_tool_auth(authorization)
+ _check_code_tool_auth(authorization)
if not isinstance(payload, dict):
raise HTTPException(status_code=400, detail="Invalid request")
code = payload.get("code")
@@ -2347,7 +2363,7 @@ def _create_app(
payload: dict,
authorization: Optional[str] = Header(default=None),
) -> dict:
- _check_tool_auth(authorization)
+ _check_code_tool_auth(authorization)
if not isinstance(payload, dict):
raise HTTPException(status_code=400, detail="Invalid request")
command = payload.get("command")
diff --git a/tests/conftest.py b/tests/conftest.py
index f0dea438..b4cb2920 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -171,6 +171,17 @@ def _isolate_experiments_db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) ->
monkeypatch.setenv("SOUP_DB_PATH", str(tmp_path / "experiments.db"))
[email protected](autouse=True)
+def _no_ambient_tool_auth_token(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Start every test without ``SOUP_TOOL_AUTH_TOKEN`` from the developer's shell.
+
+ ``soup serve --tool-auth-token`` reads it, so an exported token would turn a test
+ of a refused bind, or of a route without a token, into one that passes the check.
+ A test that needs it sets it itself.
+ """
+ monkeypatch.delenv("SOUP_TOOL_AUTH_TOKEN", raising=False)
+
+
@pytest.fixture(autouse=True)
def _restore_working_directory():
"""Put the working directory back after every test.
diff --git a/tests/test_code_review_critical.py b/tests/test_code_review_critical.py
index 6849fb07..6f3531be 100644
--- a/tests/test_code_review_critical.py
+++ b/tests/test_code_review_critical.py
@@ -146,14 +146,26 @@ def test_tool_python_endpoint_requires_bearer_when_token_set():
)
-def test_tool_python_gate_is_noop_without_token():
+def test_tool_python_refuses_without_a_configured_token():
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
client = TestClient(_build_tool_app(None), base_url="http://127.0.0.1")
- # No token configured -> auth passes -> empty body fails the code check
- # with 400 (NOT 401). Proves the gate is opt-in, and reachable.
- assert client.post("/v1/tools/python", json={}).status_code == 400
+ # No token configured -> 401 on loopback too, before the body is read, and
+ # the detail names the option that fixes it.
+ resp = client.post("/v1/tools/python", json={})
+ assert resp.status_code == 401
+ assert "--tool-auth-token" in resp.json()["detail"]
+
+
+def test_tool_python_gate_is_reachable_with_the_right_token():
+ pytest.importorskip("fastapi")
+ from fastapi.testclient import TestClient
+
+ client = TestClient(_build_tool_app("secret"), base_url="http://127.0.0.1")
+ # Right token -> auth passes -> empty body fails the code check with 400.
+ resp = client.post("/v1/tools/python", json={}, headers={"Authorization": "Bearer secret"})
+ assert resp.status_code == 400
# ─────────────────────────── CRITICAL 3: eval benchmark injection ──────────────
diff --git a/tests/test_serve_code_tool_token.py b/tests/test_serve_code_tool_token.py
new file mode 100644
index 00000000..778ee697
--- /dev/null
+++ b/tests/test_serve_code_tool_token.py
@@ -0,0 +1,306 @@
+"""``soup serve``: the code-execution tool routes take ``--tool-auth-token`` on every bind.
+
+``/v1/tools/python`` and ``/v1/tools/bash`` answer 401 unless the server was started
+with ``--tool-auth-token`` (or ``SOUP_TOOL_AUTH_TOKEN``) and the request sends
+``Authorization: Bearer <token>``. They already did so beyond loopback; they now do so
+on a loopback bind too. ``/v1/tools/web_search``, ``/v1/thumbs`` and the adapter routes
+keep their rule: a token only when one is configured.
+
+The token check comes after the Host/Origin check and before the request's content is
+used or a sandbox starts, so a refused request runs nothing. An empty token, from the
+option, the environment or ``_create_app``, counts as no token.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pytest
+
+pytest.importorskip("fastapi")
+pytest.importorskip("httpx")
+
+from fastapi.testclient import TestClient # noqa: E402
+from typer.testing import CliRunner # noqa: E402
+
+from soup_cli.cli import app as cli_app # noqa: E402
+from tests.conftest import strip_ansi # noqa: E402
+
+runner = CliRunner()
+
+TOKEN = "s3cret-tool-token"
+AUTH = {"Authorization": f"Bearer {TOKEN}"}
+
+# The exact refusal a client reads when the server has no token configured.
+TOKEN_REQUIRED = (
+ "Authentication required: /v1/tools/python and /v1/tools/bash need a token on "
+ "every bind. Start soup serve with --tool-auth-token <secret> (or set "
+ "SOUP_TOOL_AUTH_TOKEN) and send 'Authorization: Bearer <secret>'."
+)
+BAD_BEARER = "Invalid or missing bearer token"
+
+CODE_ROUTES = [
+ ("/v1/tools/python", {"code": "print(1)"}),
+ ("/v1/tools/bash", {"command": "echo 1"}),
+]
+CODE_ROUTE_IDS = ["python", "bash"]
+LOOPBACK_BINDS = ["127.0.0.1", "localhost", "::1"]
+
+
+def _plain(text: str) -> str:
+ """ANSI-stripped, whitespace-collapsed CLI output, safe to substring-match."""
+ return " ".join(strip_ansi(text).split())
+
+
+def _app(**kwargs):
+ from soup_cli.commands.serve import _create_app
+
+ base = dict(
+ model_obj=MagicMock(),
+ tokenizer=MagicMock(),
+ device="cpu",
+ model_name="test-model",
+ max_tokens_default=64,
+ )
+ base.update(kwargs)
+ return _create_app(**base)
+
+
+def sandbox_calls(monkeypatch):
+ """Stand in for both sandboxes and record every run they are asked for.
+
+ The isolation strategy is pinned to one that lets both routes run, so what is
+ measured here is the token check alone, on any host.
+ """
+ from soup_cli.trainer import rewards
+
+ calls: list[tuple[str, str]] = []
+
+ def _fake_python(code, **_options):
+ calls.append(("python", code))
+ return "ran\n"
+
+ def _fake_bash(command, **_options):
+ calls.append(("bash", command))
+ return rewards.SandboxProcessResult(returncode=0, stdout="ran\n", stderr="")
+
+ monkeypatch.setattr(rewards, "_get_isolation_strategy", lambda: "namespaces")
+ monkeypatch.setattr(rewards, "_run_code_sandbox", _fake_python)
+ monkeypatch.setattr(rewards, "_run_bash_sandbox", _fake_bash)
+ return calls
+
+
+class TestNoTokenConfigured:
+ @pytest.mark.parametrize("bind", LOOPBACK_BINDS)
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_code_routes_refuse_on_a_loopback_bind(self, bind, path, body, sandbox_calls):
+ client = TestClient(_app(host=bind), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json=body)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_code_routes_refuse_beyond_loopback(self, path, body, sandbox_calls):
+ client = TestClient(_app(host="0.0.0.0"), base_url="http://10.0.0.7:8000")
+
+ resp = client.post(path, json=body)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_a_bearer_header_does_not_stand_in_for_a_server_token(
+ self, path, body, sandbox_calls
+ ):
+ client = TestClient(_app(), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json=body, headers=AUTH)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize(
+ "header",
+ [None, AUTH, {"Authorization": "Bearer"}],
+ ids=["no-header", "bearer", "empty-bearer"],
+ )
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_an_empty_token_is_no_token(self, header, path, body, sandbox_calls):
+ client = TestClient(_app(auth_token=""), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json=body, headers=header)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize(("path", "_body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_the_token_is_checked_before_the_body_content(self, path, _body, sandbox_calls):
+ client = TestClient(_app(), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json={})
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_the_token_is_checked_before_the_isolation_requirement(
+ self, path, body, sandbox_calls, monkeypatch
+ ):
+ from soup_cli.trainer import rewards
+
+ # A host without OS-level isolation answers 501 once a request is let in.
+ monkeypatch.setattr(rewards, "_get_isolation_strategy", lambda: "best-effort")
+ client = TestClient(_app(), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json=body)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == TOKEN_REQUIRED
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_the_host_check_still_answers_first(self, path, body, sandbox_calls):
+ client = TestClient(_app(), base_url="http://evil.example:8000")
+
+ resp = client.post(path, json=body)
+
+ assert resp.status_code == 421, (path, resp.text)
+ assert sandbox_calls == []
+
+ def test_web_search_keeps_its_rule_on_loopback(self):
+ from soup_cli.utils.server_tools import WebSearchConfig
+
+ def fake_backend(query, max_results, allowlist):
+ return [{"url": "https://example.com/a", "snippet": "s"}]
+
+ client = TestClient(
+ _app(
+ web_search_config=WebSearchConfig(domain_allowlist=("example.com",)),
+ web_search_backend=fake_backend,
+ ),
+ base_url="http://127.0.0.1:8000",
+ )
+
+ resp = client.post("/v1/tools/web_search", json={"query": "q"})
+
+ assert resp.status_code == 200, resp.text
+ assert [r["url"] for r in resp.json()["results"]] == ["https://example.com/a"]
+
+ def test_adapter_listing_keeps_its_rule_on_loopback(self):
+ client = TestClient(
+ _app(adapter_map={"chat": "/fake/chat"}), base_url="http://127.0.0.1:8000"
+ )
+
+ resp = client.get("/v1/adapters")
+
+ assert resp.status_code == 200, resp.text
+ assert [a["name"] for a in resp.json()["adapters"]] == ["chat"]
+
+
+class TestTokenConfigured:
+ @pytest.mark.parametrize("bind", ["127.0.0.1", "0.0.0.0"])
+ @pytest.mark.parametrize(
+ "header",
+ [None, "Bearer wrong", TOKEN, f"Bearer {TOKEN}x", f"bearer {TOKEN}"],
+ ids=["missing", "wrong", "no-scheme", "suffix", "lowercase-scheme"],
+ )
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_code_routes_refuse_a_bad_bearer(self, bind, header, path, body, sandbox_calls):
+ client = TestClient(_app(host=bind, auth_token=TOKEN), base_url="http://127.0.0.1:8000")
+ headers = {"Authorization": header} if header is not None else {}
+
+ resp = client.post(path, json=body, headers=headers)
+
+ assert resp.status_code == 401, (path, resp.text)
+ assert resp.json()["detail"] == BAD_BEARER
+ assert sandbox_calls == []
+
+ @pytest.mark.parametrize("bind", LOOPBACK_BINDS + ["0.0.0.0"])
+ @pytest.mark.parametrize(("path", "body"), CODE_ROUTES, ids=CODE_ROUTE_IDS)
+ def test_code_routes_run_with_the_right_bearer(self, bind, path, body, sandbox_calls):
+ client = TestClient(_app(host=bind, auth_token=TOKEN), base_url="http://127.0.0.1:8000")
+
+ resp = client.post(path, json=body, headers=AUTH)
+
+ assert resp.status_code == 200, (path, resp.text)
+ assert resp.json()["stdout"] == "ran\n"
+ assert [route for route, _ in sandbox_calls] == [path.rsplit("/", 1)[1]]
+
+ def test_with_the_right_bearer_the_body_is_checked(self, sandbox_calls):
+ client = TestClient(_app(auth_token=TOKEN), base_url="http://127.0.0.1:8000")
+
+ resp = client.post("/v1/tools/python", json={}, headers=AUTH)
+
+ assert resp.status_code == 400, resp.text
+ assert sandbox_calls == []
+
+
+class TestCommandLine:
+ def test_help_says_the_code_routes_need_the_token_on_every_bind(self, monkeypatch):
+ monkeypatch.setenv("COLUMNS", "200")
+
+ result = runner.invoke(cli_app, ["serve", "--help"])
+
+ assert result.exit_code == 0, (result.output, repr(result.exception))
+ out = _plain(result.output.replace("│", " "))
+ assert (
+ "Without it, /v1/tools/python and /v1/tools/bash answer 401 on every bind "
+ "and a non-loopback --host is refused." in out
+ )
+ assert "SOUP_TOOL_AUTH_TOKEN" in out
+ assert "Strongly recommended whenever --host is not loopback" not in out
+
+ def test_the_env_var_supplies_the_token(self, monkeypatch):
+ monkeypatch.setenv("COLUMNS", "400")
+ monkeypatch.setenv("SOUP_TOOL_AUTH_TOKEN", "env-token-value-123")
+
+ # The model path does not exist, so the command stops before a server
+ # starts; the bind check runs before that.
+ result = runner.invoke(
+ cli_app, ["serve", "--model", "no-such-model-dir", "--host", "0.0.0.0"]
+ )
+
+ out = _plain(result.output)
+ assert "requires --tool-auth-token" not in out, out
+ assert "Note: bound to '0.0.0.0', not loopback" in out, out
+ assert "env-token-value-123" not in out
+
+ @pytest.mark.parametrize(
+ ("env", "option"),
+ [("", []), (None, ["--tool-auth-token", ""])],
+ ids=["empty-env-var", "empty-option"],
+ )
+ def test_an_empty_token_is_no_token_at_start(self, env, option, monkeypatch):
+ monkeypatch.setenv("COLUMNS", "400")
+ if env is not None:
+ monkeypatch.setenv("SOUP_TOOL_AUTH_TOKEN", env)
+
+ result = runner.invoke(
+ cli_app, ["serve", "--model", "no-such-model-dir", "--host", "0.0.0.0", *option]
+ )
+
+ assert result.exit_code == 2, (result.output, repr(result.exception))
+ out = _plain(result.output)
+ assert "binding non-loopback host '0.0.0.0' requires --tool-auth-token <secret>" in out
+ assert "Note: bound to" not in out
+
+ def test_without_the_env_var_a_non_loopback_bind_is_still_refused(self, monkeypatch):
+ monkeypatch.setenv("COLUMNS", "400")
+ monkeypatch.delenv("SOUP_TOOL_AUTH_TOKEN", raising=False)
+
+ result = runner.invoke(
+ cli_app, ["serve", "--model", "no-such-model-dir", "--host", "0.0.0.0"]
+ )
+
+ assert result.exit_code == 2, (result.output, repr(result.exception))
+ assert (
+ "binding non-loopback host '0.0.0.0' requires --tool-auth-token <secret>"
+ in _plain(result.output)
+ )
diff --git a/tests/test_serve_python_tool_isolation.py b/tests/test_serve_python_tool_isolation.py
index 9bcd5768..3cbe9aad 100644
--- a/tests/test_serve_python_tool_isolation.py
+++ b/tests/test_serve_python_tool_isolation.py
@@ -35,6 +35,8 @@ TOKEN = "s3cret-token"
def _client(**kwargs) -> TestClient:
+ # /v1/tools/python and /v1/tools/bash answer 401 without a token on every bind
+ # (tests/test_serve_code_tool_token.py), so the app has one and the client sends it.
from soup_cli.commands.serve import _create_app
base = dict(
@@ -43,9 +45,14 @@ def _client(**kwargs) -> TestClient:
device="cpu",
model_name="test-model",
max_tokens_default=128,
+ auth_token=TOKEN,
)
base.update(kwargs)
- return TestClient(_create_app(**base), base_url="http://127.0.0.1")
+ return TestClient(
+ _create_app(**base),
+ base_url="http://127.0.0.1",
+ headers={"Authorization": f"Bearer {TOKEN}"},
+ )
@pytest.fixture
diff --git a/tests/test_v0536.py b/tests/test_v0536.py
index 35367cb9..140018cb 100644
--- a/tests/test_v0536.py
+++ b/tests/test_v0536.py
@@ -262,6 +262,12 @@ def _fake_model_and_tokenizer():
return model, tokenizer
+# /v1/tools/python and /v1/tools/bash answer 401 without a token on every bind
+# (tests/test_serve_code_tool_token.py), so the app is built with one.
+_TOOL_TOKEN = "tool-test-token"
+_TOOL_AUTH = {"Authorization": f"Bearer {_TOOL_TOKEN}"}
+
+
def _build_app(**overrides):
pytest.importorskip("fastapi")
from soup_cli.commands.serve import _create_app
@@ -273,6 +279,7 @@ def _build_app(**overrides):
device="cpu",
model_name="test-model",
max_tokens_default=64,
+ auth_token=_TOOL_TOKEN,
)
kwargs.update(overrides)
return _create_app(**kwargs)
@@ -508,9 +515,9 @@ def test_tool_endpoint_returns_501(path: str, monkeypatch):
monkeypatch.setattr(rewards, "_get_isolation_strategy", lambda: "namespaces")
monkeypatch.setattr(os, "unshare", lambda *args, **kwargs: None, raising=False)
app = _build_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
response = client.post(path, json={"code": "print(1)"})
- assert response.status_code != 501
+ assert response.status_code not in (401, 501), response.text
# ----------------------------------------------------------------------
diff --git a/tests/test_v0537.py b/tests/test_v0537.py
index eaf3e6d7..3c1c63e6 100644
--- a/tests/test_v0537.py
+++ b/tests/test_v0537.py
@@ -876,7 +876,14 @@ class TestInstantiateTrainerPlugins:
# ----- #103 /v1/tools/{python,bash,web_search} live ---------------------
-def _create_test_app(host: str = "127.0.0.1", auth_token: str | None = None, **kwargs):
+# /v1/tools/python and /v1/tools/bash answer 401 without a token on every bind
+# (tests/test_serve_code_tool_token.py), so the apps below are built with one by
+# default and the live-route tests send it.
+_TOOL_TOKEN = "tool-test-token"
+_TOOL_AUTH = {"Authorization": f"Bearer {_TOOL_TOKEN}"}
+
+
+def _create_test_app(host: str = "127.0.0.1", auth_token: str | None = _TOOL_TOKEN, **kwargs):
"""Build a test FastAPI app via _create_app (lazy fastapi import)."""
try:
import fastapi # noqa: F401
@@ -925,7 +932,7 @@ class TestToolEndpointsLive:
from fastapi.testclient import TestClient
_allow_live_python_tool(monkeypatch)
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post(
"/v1/tools/python",
json={"code": "print('hello')"},
@@ -940,14 +947,14 @@ class TestToolEndpointsLive:
def test_python_tool_rejects_missing_code(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post("/v1/tools/python", json={})
assert resp.status_code == 400
def test_python_tool_rejects_oversize_code(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
oversize = "x" * (64 * 1024 + 1)
resp = client.post("/v1/tools/python", json={"code": oversize})
assert resp.status_code == 400
@@ -955,7 +962,7 @@ class TestToolEndpointsLive:
def test_python_tool_non_dict_rejected(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
# FastAPI body parsing accepts list as dict-typed param? It coerces.
# Use empty code which is rejected as ValueError-equivalent.
resp = client.post("/v1/tools/python", json={"code": ""})
@@ -969,7 +976,7 @@ class TestToolEndpointsLive:
from soup_cli.trainer.rewards import SandboxProcessResult
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
monkeypatch.setattr(
"soup_cli.trainer.rewards._get_isolation_strategy", lambda: "namespaces"
)
@@ -990,14 +997,14 @@ class TestToolEndpointsLive:
def test_bash_tool_rejects_missing_command(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post("/v1/tools/bash", json={})
assert resp.status_code == 400
def test_bash_tool_rejects_oversize_command(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
oversize = "x" * (64 * 1024 + 1)
resp = client.post("/v1/tools/bash", json={"command": oversize})
assert resp.status_code == 400
@@ -1010,7 +1017,7 @@ class TestToolEndpointsLive:
from soup_cli.trainer.rewards import SandboxProcessResult
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
monkeypatch.setattr(
"soup_cli.trainer.rewards._get_isolation_strategy", lambda: "namespaces"
)
@@ -1041,7 +1048,7 @@ class TestToolEndpointsLive:
def test_bash_tool_windows_returns_501(self, monkeypatch):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
monkeypatch.setattr(
"soup_cli.trainer.rewards._get_isolation_strategy", lambda: "best-effort"
)
@@ -1054,7 +1061,7 @@ class TestToolEndpointsLive:
def test_bash_tool_auth_required_on_non_loopback_host(self):
from fastapi.testclient import TestClient
- app = _create_test_app(host="0.0.0.0")
+ app = _create_test_app(host="0.0.0.0", auth_token=None)
client = TestClient(app, base_url="http://127.0.0.1")
resp = client.post(
"/v1/tools/bash",
@@ -1226,7 +1233,7 @@ class TestToolEndpointsLive:
def test_bash_tool_restricted_linux_fails_closed_501(self, monkeypatch):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
monkeypatch.setattr(
"soup_cli.trainer.rewards._get_isolation_strategy", lambda: "namespaces"
)
@@ -1248,7 +1255,7 @@ class TestToolEndpointsLive:
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
monkeypatch.setattr(
"soup_cli.trainer.rewards._get_isolation_strategy", lambda: "namespaces"
)
@@ -1270,7 +1277,7 @@ class TestToolEndpointsLive:
def test_web_search_default_deny_all(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post(
"/v1/tools/web_search",
json={"query": "anything"},
@@ -1281,7 +1288,7 @@ class TestToolEndpointsLive:
def test_web_search_rejects_oversize_query(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post(
"/v1/tools/web_search",
json={"query": "x" * 2000},
@@ -1291,7 +1298,7 @@ class TestToolEndpointsLive:
def test_web_search_rejects_bool_max_results(self):
from fastapi.testclient import TestClient
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
resp = client.post(
"/v1/tools/web_search",
json={"query": "q", "max_results": True},
@@ -1342,7 +1349,7 @@ class TestToolEndpointsLive:
from fastapi.testclient import TestClient
_allow_live_python_tool(monkeypatch)
app = _create_test_app()
- client = TestClient(app, base_url="http://127.0.0.1")
+ client = TestClient(app, base_url="http://127.0.0.1", headers=_TOOL_AUTH)
# Python: should return 200 with sandbox response.
resp = client.post("/v1/tools/python", json={"code": "print(1)"})
assert resp.status_code != 501
@@ -1947,16 +1954,21 @@ class TestReviewFixesAuthToken:
_allow_live_python_tool(monkeypatch)
from soup_cli.commands.serve import _create_app
+ from soup_cli.utils.server_tools import WebSearchConfig
+ # web_search keeps the opt-in gate on loopback; /v1/tools/python and
+ # /v1/tools/bash take a token on every bind (test_serve_code_tool_token.py).
app = _create_app(
model_obj=MagicMock(),
tokenizer=MagicMock(),
device="cpu",
model_name="test-model",
max_tokens_default=128,
+ web_search_config=WebSearchConfig(domain_allowlist=("example.com",)),
+ web_search_backend=lambda query, max_results, allowlist: [],
)
client = TestClient(app, base_url="http://127.0.0.1")
- resp = client.post("/v1/tools/python", json={"code": "print(1)"})
+ resp = client.post("/v1/tools/web_search", json={"query": "q"})
assert resp.status_code == 200