AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
feat(platform): require a shared key on calls between the backend's internal services (#15949)
Nicholas Tindle7fafc428b01e
diff --git a/.claude/skills/pr-test/SKILL.md b/.claude/skills/pr-test/SKILL.mdindex f0abeed781..e279d0ae01 100644--- a/.claude/skills/pr-test/SKILL.md+++ b/.claude/skills/pr-test/SKILL.md@@ -668,10 +668,10 @@ fi| Frontend | 3000 | http://localhost:3000 || Backend REST | 8006 | http://localhost:8006 || Supabase Auth (via Kong) | 8000 | http://localhost:8000 |-| Executor | 8002 | http://localhost:8002 |-| Copilot Executor | 8008 | http://localhost:8008 |+| Executor (metrics only) | 8002 | http://localhost:8002 in native mode; not published by Docker Compose |+| Copilot Executor (metrics only) | 8008 | http://localhost:8008 in native mode; not published by Docker Compose || WebSocket | 8001 | http://localhost:8001 |-| Database Manager | 8005 | http://localhost:8005 |+| Database Manager | 8005 | http://localhost:8005 in native mode; not published by Docker Compose. Every method needs `Authorization: Bearer <internal key>` (`backend/util/service_auth.py`) || Redis | 6379 | localhost:6379 || RabbitMQ | 5672 | localhost:5672 |diff --git a/.github/scripts/platform-single-container-smoke.sh b/.github/scripts/platform-single-container-smoke.shindex ff4a40c2cf..fab64de7e4 100644--- a/.github/scripts/platform-single-container-smoke.sh+++ b/.github/scripts/platform-single-container-smoke.sh@@ -564,6 +564,7 @@ forbidden = {"DIRECT_URL","ENCRYPTION_KEY","GRAPHITI_FALKORDB_PASSWORD",+ "INTERNAL_API_KEY","POSTGRES_PASSWORD","RABBITMQ_DEFAULT_PASS","REDIS_PASSWORD",diff --git a/.github/workflows/platform-backend-ci.yml b/.github/workflows/platform-backend-ci.ymlindex ca26f80729..1a3070ea8d 100644--- a/.github/workflows/platform-backend-ci.yml+++ b/.github/workflows/platform-backend-ci.yml@@ -570,6 +570,7 @@ jobs:run: |echo "ENCRYPTION_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"echo "UNSUBSCRIBE_SECRET_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"+ echo "INTERNAL_API_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"- name: Test shardshell: bashdiff --git a/.github/workflows/platform-fullstack-ci.yml b/.github/workflows/platform-fullstack-ci.ymlindex 060f9abea9..5dfb9983fa 100644--- a/.github/workflows/platform-fullstack-ci.yml+++ b/.github/workflows/platform-fullstack-ci.yml@@ -469,7 +469,9 @@ jobs:exit 1}echo "Waiting for scheduler_server to be ready..."- timeout 180 sh -c 'until curl -fsS http://localhost:8003/health_check >/dev/null; do sleep 2; done' || {+ # Its port is internal (not published to the host), so ask from+ # inside its own container; the image has Python but no curl.+ timeout 180 sh -c 'until docker compose -f ../docker-compose.resolved.yml exec -T scheduler_server python -c "import urllib.request; urllib.request.urlopen(\"http://localhost:8003/health_check\", timeout=5)" >/dev/null 2>&1; do sleep 2; done' || {echo "scheduler_server did not become healthy in 180s"docker compose -f ../docker-compose.resolved.yml logs --tail=100 scheduler_serverexit 1diff --git a/.oss-scanner/Dockerfile b/.oss-scanner/Dockerfileindex d9eccdf426..1e7cbf41a3 100644--- a/.oss-scanner/Dockerfile+++ b/.oss-scanner/Dockerfile@@ -347,6 +347,7 @@ EOFRUN { \echo "export ENCRYPTION_KEY=$(openssl rand -base64 32 | tr '+/' '-_')"; \echo "export UNSUBSCRIBE_SECRET_KEY=$(openssl rand -base64 32 | tr '+/' '-_')"; \+ echo "export INTERNAL_API_KEY=$(openssl rand -base64 32 | tr '+/' '-_')"; \echo "export RABBITMQ_DEFAULT_USER=rabbitmq_user_default"; \echo "export RABBITMQ_DEFAULT_PASS=$(openssl rand -hex 16)"; \} > /etc/autogpt-scanner/secrets.envdiff --git a/autogpt_platform/README.md b/autogpt_platform/README.mdindex de43f207c8..c7ceb6ea63 100644--- a/autogpt_platform/README.md+++ b/autogpt_platform/README.md@@ -29,7 +29,7 @@ To run the AutoGPT Platform, follow these steps:make init-env```- This command copies each `.env.default` file to `.env` (in `autogpt_platform`, `backend` and `frontend`) where no `.env` exists yet, and generates the secrets those files leave blank: `ENCRYPTION_KEY`, `UNSUBSCRIBE_SECRET_KEY` and `BETTER_AUTH_SECRET`. The backend refuses to start without `ENCRYPTION_KEY`. It never overwrites an existing file or value, so it is safe to re-run. You can then modify the `.env` files to add your own environment variables.+ This command copies each `.env.default` file to `.env` (in `autogpt_platform`, `backend` and `frontend`) where no `.env` exists yet, and generates the secrets those files leave blank: `ENCRYPTION_KEY`, `UNSUBSCRIBE_SECRET_KEY`, `INTERNAL_API_KEY` and `BETTER_AUTH_SECRET`. The backend refuses to start without `ENCRYPTION_KEY`. It never overwrites an existing file or value, so it is safe to re-run. You can then modify the `.env` files to add your own environment variables.Without `make`, run `installer/setup-autogpt.sh` (Linux/macOS) or `installer\setup-autogpt.bat` (Windows) instead. They do the same and then start the services.diff --git a/autogpt_platform/backend/.env.default b/autogpt_platform/backend/.env.defaultindex 209655c1e6..c9729c6b7f 100644--- a/autogpt_platform/backend/.env.default+++ b/autogpt_platform/backend/.env.default@@ -55,6 +55,10 @@ JWT_JWKS_URL=http://localhost:3000/api/auth/jwks# To make one by hand: from cryptography.fernet import Fernet;Fernet.generate_key().decode()ENCRYPTION_KEY=UNSUBSCRIBE_SECRET_KEY=+# The key the backend's services require from each other on every internal+# call; every service needs the same one. Left blank, it is derived from+# ENCRYPTION_KEY.+INTERNAL_API_KEY=# Web Push (VAPID) — generate with: poetry run python -c "# from py_vapid import Vapid; import base64diff --git a/autogpt_platform/backend/backend/copilot/swap_service_test.py b/autogpt_platform/backend/backend/copilot/swap_service_test.pyindex 0b4e74d58f..9866be9034 100644--- a/autogpt_platform/backend/backend/copilot/swap_service_test.py+++ b/autogpt_platform/backend/backend/copilot/swap_service_test.py@@ -9,7 +9,9 @@ from starlette.routing import Routefrom backend.copilot.swap_credentials import NoLiveBox, SwapCredentialfrom backend.copilot.swap_service import SwapCredentialServicefrom backend.data.db_manager import DatabaseManager+from backend.util import service_authfrom backend.util.service import EXPOSED_FLAG, UnhealthyServiceError+from backend.util.settings import Settings_SWAP_METHODS = {"get_swap_bindings", "resolve_swap_credential"}@@ -17,8 +19,16 @@ _SWAP_METHODS = {"get_swap_bindings", "resolve_swap_credential"}@pytest.fixture(scope="module")def client():# Not entered as a context manager: the lifespan (database, Redis) is not- # what is under test, the routes are.- return TestClient(SwapCredentialService().build_fastapi_app())+ # what is under test, the routes are. Calls carry the internal key, as the+ # swap proxy's do.+ settings = Settings()+ settings.secrets.internal_api_key = "swap-service-test-key-0123456789abcdef"+ with pytest.MonkeyPatch.context() as patch:+ patch.setattr(service_auth, "Settings", lambda: settings)+ yield TestClient(+ SwapCredentialService().build_fastapi_app(),+ headers=service_auth.internal_auth_headers(),+ )def _database_manager_methods() -> list[str]:diff --git a/autogpt_platform/backend/backend/util/process.py b/autogpt_platform/backend/backend/util/process.pyindex ae45bb0167..63e17748eb 100644--- a/autogpt_platform/backend/backend/util/process.py+++ b/autogpt_platform/backend/backend/util/process.py@@ -13,6 +13,7 @@ from backend.util.logging import configure_loggingfrom backend.util.metrics import sentry_initfrom backend.util.retry import stop_retry_loopsfrom backend.util.secrets_guard import check_secrets+from backend.util.service_auth import internal_api_keyfrom backend.util.settings import (AppEnvironment,Config,@@ -156,6 +157,10 @@ class AppProcess(ABC):# secret. Deliberately here rather than at import time so tooling and# tests that merely import backend modules are unaffected.check_secrets()+ # Raises on a malformed INTERNAL_API_KEY, or an ENCRYPTION_KEY the key+ # cannot be derived from, here rather than inside get_service_client's+ # connection retries, which would retry it for most of an hour.+ internal_api_key()if not background:self.execute_run_command(silent)diff --git a/autogpt_platform/backend/backend/util/service.py b/autogpt_platform/backend/backend/util/service.pyindex c9c622c45c..090f62b9c5 100644--- a/autogpt_platform/backend/backend/util/service.py+++ b/autogpt_platform/backend/backend/util/service.py@@ -43,6 +43,7 @@ from backend.util.json import to_dictfrom backend.util.metrics import sentry_initfrom backend.util.process import AppProcess, freeze_startup_heapfrom backend.util.retry import conn_retry, create_retry_decorator, stop_retry_loops+from backend.util.service_auth import internal_auth_headers, require_internal_keyfrom backend.util.settings import Config, get_service_namelogger = logging.getLogger(__name__)@@ -564,6 +565,9 @@ class AppService(BaseAppService, ABC):"/health_check_async", self.health_check, methods=["POST", "GET"])self._register_exception_handlers(app)+ # Last, so it is the outermost middleware: every request but the health+ # checks and /metrics needs the internal key before anything reads it.+ require_internal_key(app, self.service_name)return app@asynccontextmanager@@ -705,14 +709,19 @@ def get_service_client(host = service_type.get_host()port = service_type.get_port()self.base_url = f"http://{host}:{port}".rstrip("/")+ self._headers = internal_auth_headers()self._connection_failure_count = 0self._last_client_reset = 0self._async_clients = {} # None key for default async clientself._sync_clients = {} # For sync clients (no event loop concept)+ # trust_env=False: an HTTP_PROXY meant for outbound traffic must never+ # carry these plain-HTTP calls, nor the internal key they hold.def _create_sync_client(self) -> httpx.Client:return httpx.Client(base_url=self.base_url,+ headers=self._headers,+ trust_env=False,timeout=call_timeout,limits=httpx.Limits(max_keepalive_connections=200, # 10x default for async concurrent calls@@ -725,6 +734,8 @@ def get_service_client(def _create_async_client(self) -> httpx.AsyncClient:return httpx.AsyncClient(base_url=self.base_url,+ headers=self._headers,+ trust_env=False,timeout=call_timeout,limits=httpx.Limits(max_keepalive_connections=200, # 10x default for async concurrent callsdiff --git a/autogpt_platform/backend/backend/util/service_auth.py b/autogpt_platform/backend/backend/util/service_auth.pynew file mode 100644index 0000000000..e13c451087--- /dev/null+++ b/autogpt_platform/backend/backend/util/service_auth.py@@ -0,0 +1,154 @@+"""The shared key that authenticates calls between internal services.++A method an ``AppService`` exposes runs for anything that can reach the+service's port, and many of them read or change any user's data. So every+request to an RPC service must carry ``Authorization: Bearer <key>``, which+``get_service_client`` adds to every call, and a service refuses any other+request before reading its body. Only the health checks and ``/metrics`` are+open, for probes and the Prometheus scraper; neither returns user data. The+header is ``Authorization`` because Sentry and the event scrubber in+``backend.util.metrics`` already keep its value out of error reports.++The key is ``INTERNAL_API_KEY``, which setup generates along with the other+secrets. An install set up before it existed has none, so there it is derived+from ``ENCRYPTION_KEY``, which every backend service of a deployment already+shares. A derived key changes with ``ENCRYPTION_KEY``, so restart every service+together when rotating that. A service with neither refuses every call.+"""++import base64+import hashlib+import hmac+import logging+import re++from fastapi import FastAPI+from starlette.types import ASGIApp, Receive, Scope, Send++from backend.util.secrets_guard import RETIRED_HINT+from backend.util.settings import Settings++logger = logging.getLogger(__name__)++OPEN_PATHS = frozenset({"/health_check", "/health_check_async", "/metrics"})++_VALID_KEY = re.compile(r"[A-Za-z0-9._~-]{32,}={0,2}")+_DERIVATION_LABEL = b"autogpt internal service key v1"+_REFUSAL = b'{"detail":"Invalid internal service key"}'+++def internal_api_key(settings: Settings | None = None) -> str:+ """The key internal calls carry: ``INTERNAL_API_KEY``, else one derived+ from ``ENCRYPTION_KEY``, else ``""`` when neither is set.++ Raises ``ValueError`` on a malformed ``INTERNAL_API_KEY``, and on an+ ``ENCRYPTION_KEY`` that is not a Fernet key when the key is derived from+ it: a guessable one would make the internal key guessable too. Every+ process checks this at start (``AppProcess.start``).+ """+ secrets = (settings or Settings()).secrets+ if secrets.internal_api_key:+ if not _VALID_KEY.fullmatch(secrets.internal_api_key):+ raise ValueError(+ "INTERNAL_API_KEY must be at least 32 URL-safe characters: "+ "letters, digits, '-', '_', '.', '~' (and '=' padding)."+ )+ return secrets.internal_api_key+ if not secrets.encryption_key:+ return ""+ source = _fernet_key_bytes(secrets.encryption_key)+ if source is None:+ raise ValueError(+ "ENCRYPTION_KEY is not a Fernet key (32 random bytes, URL-safe "+ "base64), and the internal service key is derived from it. "+ + RETIRED_HINT.format(name="ENCRYPTION_KEY")+ )+ digest = hmac.new(source, _DERIVATION_LABEL, hashlib.sha256).digest()+ return base64.urlsafe_b64encode(digest).rstrip(b"=").decode()+++def internal_auth_headers(settings: Settings | None = None) -> dict[str, str]:+ """The headers an RPC client sends with every call."""+ key = internal_api_key(settings)+ return {"Authorization": f"Bearer {key}"} if key else {}+++def require_internal_key(+ app: FastAPI, service_name: str, settings: Settings | None = None+) -> None:+ """Make *app* refuse every request outside ``OPEN_PATHS`` that does not+ carry the key."""+ key = internal_api_key(settings)+ if not key:+ logger.error(+ f"[SECURITY] [{service_name}] No internal service key: set "+ "ENCRYPTION_KEY (every service needs it) or INTERNAL_API_KEY. Until "+ "then every call but the health checks is refused."+ )+ app.add_middleware(InternalKeyMiddleware, service_name=service_name, key=key)+++class InternalKeyMiddleware:+ """Refuses a request outside ``OPEN_PATHS`` without the key, before its+ body is read; the bearer token is compared in constant time. Without a+ key it refuses all of them."""++ def __init__(self, app: ASGIApp, service_name: str, key: str):+ self.app = app+ self.service_name = service_name+ self.expected = key.encode()++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ if scope["type"] not in ("http", "websocket") or scope["path"] in OPEN_PATHS:+ await self.app(scope, receive, send)+ return+ # Without a key nothing matches, not even an empty token.+ if self.expected and hmac.compare_digest(_bearer_token(scope), self.expected):+ await self.app(scope, receive, send)+ return+ await self._refuse(scope, send)++ async def _refuse(self, scope: Scope, send: Send) -> None:+ caller = (scope.get("client") or ("unknown",))[0]+ logger.warning(+ f"[{self.service_name}] Refused {scope['path']!r} from {caller}: "+ "missing or wrong internal service key"+ )+ if scope["type"] == "websocket":+ await send({"type": "websocket.close", "code": 1008})+ return+ await send(+ {+ "type": "http.response.start",+ "status": 401,+ "headers": [+ (b"content-type", b"application/json"),+ (b"content-length", str(len(_REFUSAL)).encode()),+ (b"www-authenticate", b"Bearer"),+ # Close instead of taking in the rest of a refused body.+ (b"connection", b"close"),+ ],+ }+ )+ await send({"type": "http.response.body", "body": _REFUSAL})+++def _bearer_token(scope: Scope) -> bytes:+ """The first ``Authorization`` header's bearer token, or ``b""`` when+ there is no such header or it holds anything but a bearer token."""+ for name, value in scope["headers"]:+ if name == b"authorization":+ scheme, _, token = value.partition(b" ")+ return token if scheme.lower() == b"bearer" else b""+ return b""+++def _fernet_key_bytes(value: str) -> bytes | None:+ """The 32 bytes ``Fernet``, and so ``JSONCryptor``, reads from *value*, or+ ``None`` where it refuses it. Deriving from these rather than the text+ gives every encoding of one key that Fernet accepts the same internal key."""+ try:+ key = base64.urlsafe_b64decode(value.encode())+ except ValueError: # binascii.Error is a ValueError+ return None+ return key if len(key) == 32 else Nonediff --git a/autogpt_platform/backend/backend/util/service_auth_test.py b/autogpt_platform/backend/backend/util/service_auth_test.pynew file mode 100644index 0000000000..b2dc61b6cc--- /dev/null+++ b/autogpt_platform/backend/backend/util/service_auth_test.py@@ -0,0 +1,503 @@+import base64+import hashlib+import hmac+import re+import socket+import threading+import time+from collections.abc import Iterator+from contextlib import contextmanager+from typing import Protocol, cast++import httpx+import pytest+import uvicorn+from cryptography.fernet import Fernet+from fastapi import FastAPI+from fastapi.testclient import TestClient++from backend.util import process, service_auth+from backend.util.process import AppProcess+from backend.util.service import (+ AppService,+ AppServiceClient,+ HTTPClientError,+ endpoint_to_async,+ expose,+ get_service_client,+)+from backend.util.settings import Settings++KEY = "internal-key-for-tests-0123456789abcdef"+ENC_1 = base64.urlsafe_b64encode(b"1" * 32).decode()+ENC_2 = base64.urlsafe_b64encode(b"2" * 32).decode()+_port = 0+++class KeyedService(AppService):+ @classmethod+ def get_port(cls) -> int:+ return _port++ @expose+ def add(self, a: int, b: int) -> int:+ return a + b+++class KeyedServiceClient(AppServiceClient):+ @classmethod+ def get_service_type(cls):+ return KeyedService++ add = KeyedService.add+ add_async = endpoint_to_async(KeyedService.add)+++class MetricsService(AppService):+ """Built only by the /metrics test: metrics register once per service name."""++ @classmethod+ def get_port(cls) -> int:+ return 0++ @expose+ def ping(self) -> str:+ return "pong"+++class _Unstarted(AppProcess):+ reads_feature_flags = False+ ran = False++ def run(self) -> None:+ pass++ def execute_run_command(self, silent: bool) -> None:+ self.ran = True+++class _SupportsClientCache(Protocol):+ _sync_clients: dict[str, httpx.Client]+++def _settings(key: str = KEY, *, encryption_key: str = ""):+ settings = Settings()+ settings.secrets.internal_api_key = key+ settings.secrets.encryption_key = encryption_key+ return settings+++def use_settings(monkeypatch: pytest.MonkeyPatch):+ """Make services and their clients read these settings."""++ def use(settings: Settings) -> None:+ monkeypatch.setattr(service_auth, "Settings", lambda: settings)++ return use+++def _app(use_settings, settings: Settings) -> FastAPI:+ use_settings(settings)+ return KeyedService().build_fastapi_app()+++def _call_add(client: TestClient, **headers: str) -> httpx.Response:+ return client.post("/add", json={"a": 2, "b": 3}, headers=headers)+++def _derived(encryption_key: str) -> str:+ """Pinned: changing the derivation makes every caller still running the+ previous release present a different key."""+ digest = hmac.new(+ base64.urlsafe_b64decode(encryption_key.encode()),+ b"autogpt internal service key v1",+ hashlib.sha256,+ ).digest()+ return base64.urlsafe_b64encode(digest).rstrip(b"=").decode()+++# Which key+++def test_unset_the_key_is_derived_from_the_encryption_key():+ key = service_auth.internal_api_key(_settings("", encryption_key=ENC_1))++ assert key == _derived(ENC_1)+ assert re.fullmatch(r"[A-Za-z0-9_-]{43}", key)+ assert key != service_auth.internal_api_key(_settings("", encryption_key=ENC_2))+++def test_a_set_key_wins_over_the_derived_one():+ settings = _settings(KEY, encryption_key=ENC_1)+ assert service_auth.internal_api_key(settings) == KEY+++ "encryption_key",+ [+ "e",+ "changeme",+ "0" * 64, # hex: 32 bytes written out, not encoded+ base64.urlsafe_b64encode(b"x" * 31).decode(),+ "not base64 at all!",+ ],+)+def test_no_key_is_derived_from_an_encryption_key_fernet_refuses(encryption_key):+ """A guessable ENCRYPTION_KEY would make the derived key guessable too."""+ with pytest.raises(ValueError):+ Fernet(encryption_key.encode())+ with pytest.raises(ValueError, match="ENCRYPTION_KEY is not a Fernet key"):+ service_auth.internal_api_key(_settings("", encryption_key=encryption_key))+ # A set key does not depend on it.+ assert service_auth.internal_api_key(_settings(KEY, encryption_key=encryption_key))+++ "encryption_key",+ [+ Fernet.generate_key().decode(),+ base64.b64encode(b"\xfb\xff" * 16).decode(), # standard alphabet: + and /+ ],+)+def test_any_key_fernet_accepts_derives_one(encryption_key):+ """No stricter than Fernet: an install whose credentials decrypt starts."""+ Fernet(encryption_key.encode())+ key = service_auth.internal_api_key(_settings("", encryption_key=encryption_key))+ assert key == _derived(encryption_key)+++def test_every_encoding_of_one_key_derives_the_same_internal_key():+ """Services that decrypt the same credentials must agree on the key, even+ when one was given the key in another encoding Fernet also reads."""+ raw = b"\xfb\xff" * 16+ encodings = [+ base64.urlsafe_b64encode(raw).decode(),+ base64.b64encode(raw).decode(), # standard alphabet: + and /+ base64.urlsafe_b64encode(raw).decode() + "\n",+ ]+ token = Fernet(encodings[0].encode()).encrypt(b"a credential")+ for encoding in encodings:+ assert Fernet(encoding.encode()).decrypt(token) == b"a credential"+ keys = {+ service_auth.internal_api_key(_settings("", encryption_key=encoding))+ for encoding in encodings+ }+ assert len(keys) == 1+++ "malformed", ["short", "x" * 31, "a key with spaces that is long enough", "é" * 40]+)+def test_a_malformed_key_is_refused(malformed):+ with pytest.raises(ValueError, match="INTERNAL_API_KEY"):+ service_auth.internal_api_key(_settings(malformed))+++ "settings, refused",+ [+ (_settings("short"), "INTERNAL_API_KEY"),+ (_settings("", encryption_key="changeme"), "ENCRYPTION_KEY"),+ ],+)+def test_a_bad_key_stops_every_process_starting(+ use_settings, monkeypatch, settings, refused+):+ monkeypatch.setattr(process, "check_secrets", lambda: None)+ use_settings(settings)+ unstarted = _Unstarted()++ with pytest.raises(ValueError, match=refused):+ unstarted.start()+ assert not unstarted.ran+++def test_client_headers_carry_the_key():+ assert service_auth.internal_auth_headers(_settings()) == {+ "Authorization": f"Bearer {KEY}"+ }+ assert service_auth.internal_auth_headers(_settings("", encryption_key=ENC_1)) == {+ "Authorization": f"Bearer {_derived(ENC_1)}"+ }+ assert service_auth.internal_auth_headers(_settings("")) == {}+++# What a service accepts+++def test_an_exposed_method_needs_the_key(use_settings):+ client = TestClient(_app(use_settings, _settings()))++ assert _call_add(client).status_code == 401+ assert _call_add(client, Authorization="Bearer wrong-key").status_code == 401+ assert _call_add(client, Authorization=f"Basic {KEY}").status_code == 401+ assert _call_add(client, Authorization=KEY).status_code == 401+ assert _call_add(client, Authorization=f"Bearer {KEY}").status_code == 401++ response = _call_add(client, Authorization=f"Bearer {KEY}")+ assert response.status_code == 200+ assert response.json() == 5+++def test_the_derived_key_is_enforced(use_settings):+ client = TestClient(_app(use_settings, _settings("", encryption_key=ENC_1)))++ assert _call_add(client).status_code == 401+ assert _call_add(client, Authorization=f"Bearer {_derived(ENC_1)}").json() == 5+++def test_the_key_is_checked_before_the_body_is_read(use_settings):+ """Unauthenticated bodies are never buffered or parsed: a malformed one is+ refused as unauthenticated, not reported as invalid."""+ client = TestClient(_app(use_settings, _settings()))+ json_type = {"content-type": "application/json"}+ with_key = {**json_type, "Authorization": f"Bearer {KEY}"}++ assert (+ client.post("/add", content=b"{not json", headers=json_type).status_code == 401+ )+ assert (+ client.post("/add", content=b"{not json", headers=with_key).status_code == 422+ )+++async def _through_middleware(+ *,+ key: str = KEY,+ authorization: bytes | None = None,+ scope_type: str = "http",+) -> tuple[list[dict], bool]:+ """One request straight through the middleware: what it sent back, and+ whether the service behind it was reached. Reading the body fails."""+ reached = False++ async def app(scope, receive, send):+ nonlocal reached+ reached = True+ await send({"type": "http.response.start", "status": 200, "headers": []})+ await send({"type": "http.response.body", "body": b""})++ async def receive():+ raise AssertionError("the middleware read a request body")++ sent: list[dict] = []++ async def send(message):+ sent.append(message)++ middleware = service_auth.InternalKeyMiddleware(app, service_name="svc", key=key)+ headers = [] if authorization is None else [(b"authorization", authorization)]+ scope = {"type": scope_type, "path": "/add", "headers": headers}+ await middleware({**scope, "client": ("10.0.0.9", 1)}, receive, send)+ return sent, reached+++async def test_the_middleware_never_reads_a_refused_body():+ sent, reached = await _through_middleware()++ assert not reached+ assert sent[0]["type"] == "http.response.start"+ assert sent[0]["status"] == 401+++async def test_a_websocket_without_the_key_is_closed():+ sent, reached = await _through_middleware(scope_type="websocket")++ assert not reached+ assert sent == [{"type": "websocket.close", "code": 1008}]++[email protected]("authorization", [None, b"Bearer ", b"Bearer", b"Basic abc"])+async def test_without_a_key_not_even_an_empty_token_gets_through(authorization):+ sent, reached = await _through_middleware(key="", authorization=authorization)++ assert not reached+ assert sent[0]["status"] == 401+++def test_a_non_ascii_key_is_refused_not_an_error(use_settings):+ client = TestClient(_app(use_settings, _settings()), raise_server_exceptions=False)+ response = client.post(+ "/add",+ json={"a": 2, "b": 3},+ headers={"Authorization": "Bearer clé".encode("latin-1")},+ )+ assert response.status_code == 401+++def test_a_refusal_cannot_forge_a_log_line(use_settings, caplog):+ client = TestClient(_app(use_settings, _settings()))+ client.post("/add%0A2026-10-21 INFO forged", json={})++ assert "\n2026-10-21 INFO forged" not in caplog.text+ assert "\\n2026-10-21 INFO forged" in caplog.text+++def test_health_checks_stay_open(use_settings):+ client = TestClient(_app(use_settings, _settings()))+ for path in ("/health_check", "/health_check_async"):+ assert client.get(path).status_code == 200+ assert client.post(path).status_code == 200+++def test_metrics_stay_open(use_settings, monkeypatch: pytest.MonkeyPatch):+ monkeypatch.setenv("ENABLE_METRICS", "true")+ use_settings(_settings())+ client = TestClient(MetricsService().build_fastapi_app())++ assert client.get("/metrics").status_code == 200+ assert client.post("/ping").status_code == 401+++def test_the_api_docs_need_the_key(use_settings):+ client = TestClient(_app(use_settings, _settings()))+ for path in ("/openapi.json", "/docs"):+ assert client.get(path).status_code == 401+ assert client.get(path, headers={"Authorization": f"Bearer {KEY}"}).is_success+++def test_without_any_key_a_service_refuses_every_call(use_settings, caplog):+ client = TestClient(_app(use_settings, _settings("")))++ assert _call_add(client).status_code == 401+ assert client.get("/health_check").status_code == 200+ assert "No internal service key" in caplog.text+++# Real clients over a real socket+++def _free_port() -> int:+ with socket.socket() as probe:+ probe.bind(("127.0.0.1", 0))+ return probe.getsockname()[1]+++@contextmanager+def _serving(app: FastAPI) -> Iterator[int]:+ global _port+ _port = _free_port()+ server = uvicorn.Server(+ uvicorn.Config(+ app, host="127.0.0.1", port=_port, lifespan="off", log_config=None+ )+ )+ thread = threading.Thread(target=server.run, daemon=True)+ thread.start()+ deadline = time.monotonic() + 10+ while not server.started:+ if time.monotonic() > deadline or not thread.is_alive():+ raise RuntimeError("test RPC server did not start")+ time.sleep(0.02)+ try:+ yield _port+ finally:+ server.should_exit = True+ thread.join(timeout=10)+++@contextmanager+def _trap_proxy(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[bytes]]:+ """An outbound proxy configured the usual way, recording what reaches it."""+ reached: list[bytes] = []+ listener = socket.socket()+ listener.bind(("127.0.0.1", 0))+ listener.listen(8)+ listener.settimeout(0.2)+ stop = threading.Event()++ def serve() -> None:+ while not stop.is_set():+ try:+ conn, _ = listener.accept()+ except OSError:+ continue+ with conn:+ conn.settimeout(2)+ try:+ reached.append(conn.recv(1024))+ except OSError:+ reached.append(b"")++ thread = threading.Thread(target=serve, daemon=True)+ thread.start()+ proxy = f"http://127.0.0.1:{listener.getsockname()[1]}"+ for name in ("HTTP_PROXY", "http_proxy", "ALL_PROXY", "all_proxy"):+ monkeypatch.setenv(name, proxy)+ for name in ("NO_PROXY", "no_proxy"):+ monkeypatch.delenv(name, raising=False)+ try:+ yield reached+ finally:+ stop.set()+ thread.join(timeout=2)+ listener.close()+++def local_host(monkeypatch: pytest.MonkeyPatch):+ monkeypatch.setenv(f"{KeyedService.__name__.upper()}_HOST", "127.0.0.1")+++async def test_every_rpc_client_sends_the_key(use_settings, local_host):+ with _serving(_app(use_settings, _settings())) as port:+ async with httpx.AsyncClient() as raw:+ unauthenticated = await raw.post(+ f"http://127.0.0.1:{port}/add", json={"a": 2, "b": 3}+ )+ assert unauthenticated.status_code == 401++ client = get_service_client(KeyedServiceClient, request_retry=False)+ assert client.add(2, 3) == 5+ assert await client.add_async(2, 3) == 5++ # A client rebuilt after connection failures still carries it.+ cast(_SupportsClientCache, client)._sync_clients.clear()+ assert client.add(4, 5) == 9+++def test_a_refused_caller_cannot_make_the_service_take_in_its_body(use_settings):+ """The connection closes with the refusal, so a body promised in the+ headers is never waited for or read."""+ with (+ _serving(_app(use_settings, _settings())) as port,+ socket.create_connection(("127.0.0.1", port), timeout=5) as conn,+ ):+ conn.sendall(+ b"POST /add HTTP/1.1\r\nHost: rpc\r\n"+ b"Content-Type: application/json\r\nContent-Length: 100000000\r\n\r\n"+ )+ received = b""+ while chunk := conn.recv(4096): # b"" once the service hangs up+ received += chunk++ assert received.startswith(b"HTTP/1.1 401")+++def test_a_client_with_another_key_is_refused_and_not_retried(use_settings, local_host):+ with _serving(_app(use_settings, _settings())):+ use_settings(_settings("another-key-for-tests-0123456789abcdef"))+ client = get_service_client(KeyedServiceClient, request_retry=True)+ started = time.monotonic()+ with pytest.raises(HTTPClientError) as refused:+ client.add(2, 3)+ assert refused.value.status_code == 401+ assert time.monotonic() - started < 5+++async def test_rpc_calls_never_go_through_an_environment_proxy(+ use_settings, monkeypatch+):+ """The calls are plain HTTP and carry the key: an outbound proxy set in the+ environment must not be where they go."""+ use_settings(_settings())+ # A host that never resolves: going direct fails fast, a proxy would be asked.+ monkeypatch.setenv(f"{KeyedService.__name__.upper()}_HOST", "keyed.invalid")+ with _trap_proxy(monkeypatch) as reached:+ client = get_service_client(KeyedServiceClient, request_retry=False)+ with pytest.raises(httpx.HTTPError):+ client.add(2, 3)+ with pytest.raises(httpx.HTTPError):+ await client.add_async(2, 3)+ assert reached == []diff --git a/autogpt_platform/backend/backend/util/service_test.py b/autogpt_platform/backend/backend/util/service_test.pyindex eb29cf35e8..efcf0ce96d 100644--- a/autogpt_platform/backend/backend/util/service_test.py+++ b/autogpt_platform/backend/backend/util/service_test.py@@ -39,6 +39,7 @@ from backend.util.service import (get_service_client,resolve_type_hints,)+from backend.util.service_auth import internal_auth_headersTEST_SERVICE_PORT = 8765@@ -917,7 +918,7 @@ async def wait_until_service_ready(base_url: str, timeout: float = 10):async def send_slow_request(base_url: str) -> dict:"""Send a slow request and return the result"""- async with httpx.AsyncClient(timeout=30) as client:+ async with httpx.AsyncClient(timeout=30, headers=internal_auth_headers()) as client:response = await client.post(f"{base_url}/slow_endpoint", json={"duration": 5})assert response.status_code == 200return response.json()@@ -943,7 +944,9 @@ async def test_graceful_shutdown(test_service):# Try to send a new request - should be rejected or connection refusedtry:- async with httpx.AsyncClient(timeout=5) as client:+ async with httpx.AsyncClient(+ timeout=5, headers=internal_auth_headers()+ ) as client:response = await client.post(f"{test_service_url}/fast_endpoint", json={})# Should get 503 Service Unavailable during shutdownassert response.status_code == 503diff --git a/autogpt_platform/backend/backend/util/settings.py b/autogpt_platform/backend/backend/util/settings.pyindex 32de8b0ebc..37980fd254 100644--- a/autogpt_platform/backend/backend/util/settings.py+++ b/autogpt_platform/backend/backend/util/settings.py@@ -1218,6 +1218,15 @@ class Secrets(UpdateTrackingModel["Secrets"], BaseSettings):encryption_key: str = Field(default="", description="Encryption key")+ internal_api_key: str = Field(+ default="",+ description="Key every internal service sends on its RPC calls and "+ "requires on the methods it exposes; every backend service of one "+ "deployment must have the same value. Setup generates it along with "+ "the other secrets. Blank derives one from ENCRYPTION_KEY, so an "+ "install set up before it existed keeps working.",+ )+rabbitmq_default_user: str = Field(default="", description="RabbitMQ default user")rabbitmq_default_pass: str = Field(default="", description="RabbitMQ default password"diff --git a/autogpt_platform/docker-compose.platform.yml b/autogpt_platform/docker-compose.platform.ymlindex 58623fbbef..1ce99a9ec6 100644--- a/autogpt_platform/docker-compose.platform.yml+++ b/autogpt_platform/docker-compose.platform.yml@@ -282,8 +282,9 @@ services:<<: *backend-envCODEX_TEMP_ROOT: /run/autogpt-codextmpfs: *codex-tmpfs- ports:- - "8002:8002"+ # Metrics only. Reachable from the other containers, not from the host.+ expose:+ - "8002"networks:- app-networklogging:@@ -321,8 +322,9 @@ services:CODEX_TEMP_ROOT: /run/autogpt-codexPYTHONUNBUFFERED: "1"tmpfs: *codex-tmpfs- ports:- - "8008:8008"+ # Metrics only. Reachable from the other containers, not from the host.+ expose:+ - "8008"volumes:- workspace-data:/app/autogpt_platform/backend/workspacesnetworks:@@ -387,8 +389,9 @@ services:<<: *backend-env-filesenvironment:<<: *backend-env- ports:- - "8005:8005"+ # Internal service. Reachable from the other containers, not from the host.+ expose:+ - "8005"volumes:- store-media-data:/app/autogpt_platform/backend/store-medianetworks:@@ -425,8 +428,9 @@ services:<<: *backend-env-filesenvironment:<<: *backend-env- ports:- - "8003:8003"+ # Internal service. Reachable from the other containers, not from the host.+ expose:+ - "8003"networks:- app-networklogging:@@ -459,8 +463,9 @@ services:<<: *backend-env-filesenvironment:<<: *backend-env- ports:- - "8007:8007"+ # Internal service. Reachable from the other containers, not from the host.+ expose:+ - "8007"networks:- app-networklogging:@@ -493,8 +498,9 @@ services:<<: *backend-env-filesenvironment:<<: *backend-env- ports:- - "8009:8009"+ # Internal service. Reachable from the other containers, not from the host.+ expose:+ - "8009"networks:- app-networklogging:diff --git a/autogpt_platform/frontend/src/playwright/journeys/README.md b/autogpt_platform/frontend/src/playwright/journeys/README.mdindex 8405179a76..aedd8c1387 100644--- a/autogpt_platform/frontend/src/playwright/journeys/README.md+++ b/autogpt_platform/frontend/src/playwright/journeys/README.md@@ -187,7 +187,7 @@ Differences from hosted that are still open:| Copilot | The scripted model: the real engine (Claude Agent SDK), tools, persistence, streaming and cost, with scripted model turns. Memory (Graphiti) calls fail, so turns run without memory | Real models || Flag source | Fixed values from `hosted-flags.json`; JSON flags (model routing, tier limits) at code defaults | PostHog/LaunchDarkly with targeting || `BEHAVE_AS` | `LOCAL` on both sides; the frontend's is build time | `CLOUD` |-| Pods | One `executor`, one `copilot_executor` | Many. Scaling needs the fixed host ports 8002/8008 removed first |+| Pods | One `executor`, one `copilot_executor` | Many || Frontend runtime | Next.js standalone server in Docker | Vercel (body size limits, edge routing) || External services | No Stripe, OAuth providers, Postmark or E2B; ClamAV, Redis, RabbitMQ and FalkorDB are real | Real || LLM keys | None; blocks that need platform credentials fail | Platform keys |diff --git a/autogpt_platform/installer/setup-autogpt.bat b/autogpt_platform/installer/setup-autogpt.batindex aca1739867..395b50ac43 100644--- a/autogpt_platform/installer/setup-autogpt.bat+++ b/autogpt_platform/installer/setup-autogpt.bat@@ -229,12 +229,12 @@ REM SubroutinesREM ============================================================================:init_env-REM The .env.default files leave ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY and-REM BETTER_AUTH_SECRET blank and the backend refuses to start without an-REM ENCRYPTION_KEY, so generate them here exactly as `make init-env` does.-REM Values that are already set are never overwritten. The generator is-REM stdlib-only Python; it runs in a container because Docker is the one-REM prerequisite every Windows install already has.+REM The .env.default files leave ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY,+REM INTERNAL_API_KEY and BETTER_AUTH_SECRET blank and the backend refuses to+REM start without an ENCRYPTION_KEY, so generate them here exactly as+REM `make init-env` does. Values that are already set are never overwritten.+REM The generator is stdlib-only Python; it runs in a container because Docker+REM is the one prerequisite every Windows install already has.cd /d "%REPO_DIR%\autogpt_platform"if errorlevel 1 (echo Failed to navigate to autogpt_platform directory.diff --git a/autogpt_platform/installer/setup-autogpt.sh b/autogpt_platform/installer/setup-autogpt.shindex 10eb77905f..f46fd05651 100755--- a/autogpt_platform/installer/setup-autogpt.sh+++ b/autogpt_platform/installer/setup-autogpt.sh@@ -326,10 +326,10 @@ _bootstrap_ollama_macos() {}init_env() {- # The .env.default files leave ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY and- # BETTER_AUTH_SECRET blank and the backend refuses to start without an- # ENCRYPTION_KEY, so generate them here exactly as `make init-env` does.- # Values that are already set are never overwritten.+ # The .env.default files leave ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY,+ # INTERNAL_API_KEY and BETTER_AUTH_SECRET blank and the backend refuses to+ # start without an ENCRYPTION_KEY, so generate them here exactly as+ # `make init-env` does. Values that are already set are never overwritten.cd "$REPO_DIR/autogpt_platform" || handle_error "Failed to navigate to autogpt_platform"print_color "BLUE" "Generating secrets for this install..."diff --git a/autogpt_platform/single-container/common.sh b/autogpt_platform/single-container/common.shindex 6292d2b26c..73ba2acdb0 100755--- a/autogpt_platform/single-container/common.sh+++ b/autogpt_platform/single-container/common.sh@@ -71,7 +71,7 @@ load_runtime_config() {local value="${line#*=}"[[ "${name}" != "${line}" ]] || fatal "malformed runtime config entry"case "${name}" in- AUTOGPT_RUNTIME_CONFIG_VERSION | POSTGRES_PASSWORD | RABBITMQ_DEFAULT_USER | RABBITMQ_DEFAULT_PASS | REDIS_PASSWORD | BETTER_AUTH_SECRET | ENCRYPTION_KEY | UNSUBSCRIBE_SECRET_KEY | GRAPHITI_FALKORDB_PASSWORD | VAPID_PRIVATE_KEY | VAPID_PUBLIC_KEY)+ AUTOGPT_RUNTIME_CONFIG_VERSION | POSTGRES_PASSWORD | RABBITMQ_DEFAULT_USER | RABBITMQ_DEFAULT_PASS | REDIS_PASSWORD | BETTER_AUTH_SECRET | ENCRYPTION_KEY | UNSUBSCRIBE_SECRET_KEY | GRAPHITI_FALKORDB_PASSWORD | INTERNAL_API_KEY | VAPID_PRIVATE_KEY | VAPID_PUBLIC_KEY)export "${name}=${value}";;*)diff --git a/autogpt_platform/single-container/runtime_config.py b/autogpt_platform/single-container/runtime_config.pyindex a337fb5dfb..d3c8636ad3 100755--- a/autogpt_platform/single-container/runtime_config.py+++ b/autogpt_platform/single-container/runtime_config.py@@ -16,7 +16,7 @@ from collections.abc import Callable, Mappingfrom pathlib import Pathfrom urllib.parse import urlsplit, urlunsplit-CONFIG_VERSION = "1"+CONFIG_VERSION = "2"SAFE_SECRET = re.compile(r"^[A-Za-z0-9._~-]+={0,2}$")SAFE_USERNAME = re.compile(r"^[A-Za-z0-9._-]+$")SAFE_DNS_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")@@ -74,12 +74,22 @@ def main() -> int:def ensure_runtime_config(path: Path, environment: Mapping[str, str]) -> dict[str, str]:- """Return the existing secrets or atomically create them on first boot."""+ """Return the existing secrets or atomically create them on first boot.++ A version 1 config, written before INTERNAL_API_KEY existed, gets one. It+ is saved as the current version once it passes every check, so a boot that+ fails them leaves the file as it was."""if path.is_symlink():raise ValueError(f"refusing symlink at {path}")if path.exists():values = _read_config(path)+ upgrade = values.get("AUTOGPT_RUNTIME_CONFIG_VERSION") == "1"+ if upgrade:+ values = _upgraded_from_version_1(values, environment)+ _validate_values(values)_verify_environment_matches(values, environment)+ if upgrade:+ _write_config(path, values)path.chmod(0o600)return values@@ -208,6 +218,24 @@ def _configured_or_generated(return environment.get(name) or generator()+def _upgraded_from_version_1(+ values: dict[str, str], environment: Mapping[str, str]+) -> dict[str, str]:+ """A version 1 config is a version 2 config without INTERNAL_API_KEY.++ This step produces version 2, whatever the current version is: a later+ version's own step takes it from there."""+ if "INTERNAL_API_KEY" in values:+ raise ValueError("runtime configuration has missing or unknown keys")+ return {+ **values,+ "AUTOGPT_RUNTIME_CONFIG_VERSION": "2",+ "INTERNAL_API_KEY": _configured_or_generated(+ environment, "INTERNAL_API_KEY", SECRET_GENERATORS["INTERNAL_API_KEY"]+ ),+ }++def _configured_or_generated_vapid(environment: Mapping[str, str],) -> tuple[str, str]:@@ -251,6 +279,7 @@ SECRET_GENERATORS: dict[str, Callable[[], str]] = {"ENCRYPTION_KEY": _fernet_key,"UNSUBSCRIBE_SECRET_KEY": lambda: secrets.token_urlsafe(36),"GRAPHITI_FALKORDB_PASSWORD": lambda: secrets.token_urlsafe(36),+ "INTERNAL_API_KEY": lambda: secrets.token_urlsafe(32),}# Secrets already retired from the public .env.default files (SECRT-2611).@@ -258,7 +287,11 @@ SECRET_GENERATORS: dict[str, Callable[[], str]] = {# a freshly generated key pasted back into one of these lines, so the invariant# is asserted directly. Paths are relative to autogpt_platform/.BLANK_IN_ENV_DEFAULT: dict[str, tuple[str, ...]] = {- "backend/.env.default": ("ENCRYPTION_KEY", "UNSUBSCRIBE_SECRET_KEY"),+ "backend/.env.default": (+ "ENCRYPTION_KEY",+ "UNSUBSCRIBE_SECRET_KEY",+ "INTERNAL_API_KEY",+ ),"frontend/.env.default": ("BETTER_AUTH_SECRET",),}@@ -278,6 +311,7 @@ def _validate_values(values: dict[str, str]) -> None:"ENCRYPTION_KEY","UNSUBSCRIBE_SECRET_KEY","GRAPHITI_FALKORDB_PASSWORD",+ "INTERNAL_API_KEY","VAPID_PRIVATE_KEY","VAPID_PUBLIC_KEY",}@@ -333,7 +367,6 @@ def _read_config(path: Path) -> dict[str, str]:if not separator or not name or name in values:raise ValueError(f"invalid runtime configuration line {line_number}")values[name] = value- _validate_values(values)return valuesdiff --git a/autogpt_platform/single-container/tests/test_compose_internal_ports.py b/autogpt_platform/single-container/tests/test_compose_internal_ports.pynew file mode 100644index 0000000000..22c080be3b--- /dev/null+++ b/autogpt_platform/single-container/tests/test_compose_internal_ports.py@@ -0,0 +1,266 @@+"""The Docker Compose stack publishes no internal service port on the host.++Those ports carry calls between the backend's own services; the containers+reach each other over the Compose network, so nothing outside needs them.+Read with a line scan rather than a YAML parser, as this suite runs on the+runner's bare Python. The scan fails closed: a `ports:` entry it cannot read+fails the check instead of passing it.+"""++from __future__ import annotations++import re+import unittest+from pathlib import Path++PLATFORM_DIR = Path(__file__).parents[2]+INTERNAL_PORTS = {+ "executor": "8002",+ "scheduler_server": "8003",+ "database_manager": "8005",+ "notification_server": "8007",+ "copilot_executor": "8008",+ "platform_linking_manager": "8009",+}+# Every internal service port, including those with no Compose service today.+ANY_INTERNAL = {int(port) for port in INTERNAL_PORTS.values()} | {8010, 8011, 8012}+_PORTS_KEY = re.compile(r"^\s*(?:-\s+)?ports:\s*(.*)$")+# Any other spelling of the key where a key can stand: quoted or spaced at+# the start of a line, or inside a flow mapping. A value such as a URL is not+# a key.+_ANY_PORTS_KEY = re.compile(r"""(?:^\s*(?:-\s+)?|[{,]\s*)["']?ports["']?\s*:""")+# A YAML explicit key (`? ports` with its value on the next line).+_EXPLICIT_KEY = re.compile(r"^\s*(?:-\s+)?\?(?:\s|$)")+_TARGET = re.compile(r"""["']?\btarget["']?\s*:\s*["']?(\d+)""")+# The other keys of a long-syntax entry, which carry no container port.+_OTHER_LONG_KEYS = re.compile(+ r"""^\s*(?:-\s+)?["']?(?:published|host_ip|protocol|app_protocol|mode|name)"""+ r"""["']?\s*:"""+)+++def _compose_files() -> list[Path]:+ return sorted(+ {*PLATFORM_DIR.glob("*compose*.y*ml"), *PLATFORM_DIR.glob("*/*compose*.y*ml")}+ )+++def _services(text: str) -> dict[str, list[str]]:+ """The lines under each service of a Compose file's top-level `services:`."""+ blocks: dict[str, list[str]] = {}+ in_services = False+ current: str | None = None+ for line in text.splitlines():+ if line and not line[0].isspace() and not line.startswith("#"):+ in_services = line.rstrip() == "services:"+ current = None+ continue+ if not in_services:+ continue+ header = re.match(r"^ ([\w.-]+):\s*$", line)+ if header:+ current = header.group(1)+ blocks[current] = []+ elif current is not None:+ blocks[current].append(line)+ return blocks+++def _key_items(block: list[str], key: str) -> list[str] | None:+ """The list items under `key:` in a service block, or None if absent."""+ items: list[str] | None = None+ for line in block:+ if re.match(rf"^ {key}:\s*$", line):+ items = []+ elif items is not None:+ item = re.match(r"^ -\s*[\"']?([^\"'#\s]+)", line)+ if item:+ items.append(item.group(1))+ elif line.strip() and not line.lstrip().startswith("#"):+ break+ return items+++def _published_ports(text: str) -> set[int]:+ """The container port of every `ports:` entry in a Compose file: the+ short syntax (`"8005"`, `"18005:8005"`, `"127.0.0.1:18005:8005/tcp"`,+ ranges), the long one (`target: 8005`), and either written inline.++ Raises ValueError on an entry it cannot read (an alias, a port taken from+ a variable, an unknown key) and on a `ports` key written as anything but+ a plain `ports:` line, so a form it does not know fails the check+ instead of slipping past it.+ """+ published: set[int] = set()+ ports_indent: int | None = None+ for raw in text.splitlines():+ line = raw.split("#", 1)[0].rstrip()+ if not line.strip():+ continue+ indent = len(line) - len(line.lstrip())+ is_item = line.lstrip().startswith("-")+ if ports_indent is not None and (+ indent < ports_indent or (indent == ports_indent and not is_item)+ ):+ ports_indent = None+ key = _PORTS_KEY.match(line)+ if _EXPLICIT_KEY.match(line) or (not key and _ANY_PORTS_KEY.search(line)):+ raise ValueError(f"write this as a plain `ports:` line: {line.strip()!r}")+ if key:+ inline = key.group(1).strip()+ if inline:+ published |= _inline_ports(inline)+ else:+ ports_indent = indent+ elif ports_indent is not None:+ published |= _list_line_ports(line)+ return published+++def _list_line_ports(line: str) -> set[int]:+ """One line of a `ports:` list written as a block."""+ target = _TARGET.search(line)+ if target:+ return {int(target.group(1))}+ if _OTHER_LONG_KEYS.match(line):+ return set()+ entry = re.match(r"^\s*-\s+(\S+)$", line)+ ports = _container_ports(entry.group(1)) if entry else set()+ if not ports:+ raise ValueError(f"cannot read this ports entry: {line.strip()!r}")+ return ports+++def _inline_ports(inline: str) -> set[int]:+ """A `ports:` list written on one line: short entries and `{target: ...}`+ mappings, in any mix."""+ listed = re.sub(r"^!\w+\s*", "", inline) # a tag such as Compose's !reset+ if not (listed.startswith("[") and listed.endswith("]")):+ raise ValueError(f"cannot read this ports list: {inline!r}")+ ports: set[int] = set()+ for element in re.findall(r"\{[^}]*\}|[^,{}\[\]]+", listed[1:-1]):+ if not element.strip():+ continue+ target = _TARGET.search(element)+ found = {int(target.group(1))} if target else _container_ports(element)+ if not found:+ raise ValueError(f"cannot read this ports entry: {element.strip()!r}")+ ports |= found+ return ports+++def _container_ports(entry: str) -> set[int]:+ """`[host_ip:][host_port:]container_port[/protocol]`: the container side,+ a port or a range."""+ container = entry.strip().strip("\"'").rsplit(":", 1)[-1].split("/", 1)[0]+ bounds = re.fullmatch(r"(\d+)(?:-(\d+))?", container)+ if not bounds:+ return set()+ first = int(bounds.group(1))+ return set(range(first, int(bounds.group(2) or first) + 1))+++def _ports_of(entries: str) -> set[int]:+ return _published_ports(f"services:\n db:\n ports:\n{entries}\n")+++class ComposeInternalPortsTest(unittest.TestCase):+ def test_internal_services_are_not_published_to_the_host(self) -> None:+ platform = (PLATFORM_DIR / "docker-compose.platform.yml").read_text(+ encoding="utf-8"+ )+ services = _services(platform)+ for name, port in INTERNAL_PORTS.items():+ with self.subTest(service=name):+ self.assertIn(name, services)+ self.assertIsNone(_key_items(services[name], "ports"))+ self.assertEqual(_key_items(services[name], "expose"), [port])++ def test_no_compose_file_maps_an_internal_port_to_the_host(self) -> None:+ files = _compose_files()+ self.assertIn(PLATFORM_DIR / "docker-compose.platform.yml", files)+ for compose_file in files:+ with self.subTest(file=compose_file.name):+ text = compose_file.read_text(encoding="utf-8")+ self.assertEqual(_published_ports(text) & ANY_INTERNAL, set())++ def test_the_scan_sees_a_published_port_in_any_form(self) -> None:+ """Guards the line scan itself."""+ for entries in (+ ' - "8005:8005"',+ " - 8005:8005",+ ' - "18005:8005"',+ ' - "127.0.0.1:18005:8005"',+ ' - "[::1]:18005:8005/tcp"',+ ' - "8005"',+ " - 8000-8010:8000-8010",+ ' - "8005:8005"', # items level with the key are still its items+ " - target: 8005\n published: 18005",+ " - published: 18005\n target: 8005\n protocol: tcp",+ " - {target: 8005, published: 18005}",+ ' - {"target": 8005, "published": 18005}',+ " - {target : 8005, published: 18005}",+ ):+ with self.subTest(entries=entries):+ self.assertIn(8005, _ports_of(entries))+ for inline in (+ '["3000:3000", "8005:8005"]',+ '["3000:3000", {target: 8005, published: "18005"}]',+ '[{"target": 8005, "published": 18005}, "3000:3000"]',+ '!override ["3000:3000", "8005:8005"]',+ ):+ with self.subTest(inline=inline):+ text = f"services:\n db:\n ports: {inline}\n"+ self.assertEqual(_published_ports(text), {3000, 8005})++ def test_the_scan_refuses_an_entry_it_cannot_read(self) -> None:+ for entries in (+ " - *internal_ports",+ ' - "${DB_MANAGER_PORT}"',+ " - {published: 18005, protocol: tcp}",+ " - port_from: 8005",+ ):+ with self.subTest(entries=entries), self.assertRaises(ValueError):+ _ports_of(entries)+ for inline in (+ "8005",+ "*internal_ports",+ "[*internal_ports]",+ '["${PORT}:${PORT}"]',+ ):+ with self.subTest(inline=inline), self.assertRaises(ValueError):+ _published_ports(f"services:\n db:\n ports: {inline}\n")+ for text in (+ 'services:\n db:\n "ports": ["18005:8005"]\n',+ 'services:\n db:\n ports : ["18005:8005"]\n',+ 'services: {db: {image: busybox, ports: ["18005:8005"]}}\n',+ 'services:\n db:\n ? ports\n : ["18005:8005"]\n',+ ):+ with self.subTest(text=text), self.assertRaises(ValueError):+ _published_ports(text)++ def test_the_scan_sees_only_published_ports(self) -> None:+ text = (+ "services:\n"+ " db:\n"+ " expose:\n"+ ' - "8005"\n'+ " ports:\n"+ ' - "8005:18005"\n'+ ' - "127.0.0.1:3000:3000" # the frontend\n'+ " environment:\n"+ " - PORT=8005\n"+ " x-ports: not a key the check reads\n"+ " command: [serve, --port, '8005']\n"+ )+ self.assertEqual(_published_ports(text), {18005, 3000})+ url = "services:\n db:\n environment:\n UPSTREAM: http://ports:8005\n"+ self.assertEqual(_published_ports(url), set())+ self.assertEqual(_published_ports("services:\n db:\n ports: []\n"), set())+ block = [" ports:", ' - "8005:8005"', " networks:", " - x"]+ self.assertEqual(_key_items(block, "ports"), ["8005:8005"])+ self.assertIsNone(_key_items(block, "expose"))+++if __name__ == "__main__":+ unittest.main()diff --git a/autogpt_platform/single-container/tests/test_frontend_environment.py b/autogpt_platform/single-container/tests/test_frontend_environment.pyindex 40ee2dd414..6eb4889816 100644--- a/autogpt_platform/single-container/tests/test_frontend_environment.py+++ b/autogpt_platform/single-container/tests/test_frontend_environment.py@@ -49,6 +49,7 @@ class FrontendEnvironmentTest(unittest.TestCase):"DIRECT_URL": "postgresql://superuser:[email protected]/postgres","ENCRYPTION_KEY": "master-encryption-key","GRAPHITI_FALKORDB_PASSWORD": "falkor-password",+ "INTERNAL_API_KEY": "internal-service-key","JWT_VERIFY_KEY": "legacy-backend-secret","POSTGRES_PASSWORD": "postgres-superuser-password","RABBITMQ_DEFAULT_PASS": "rabbitmq-password",diff --git a/autogpt_platform/single-container/tests/test_runtime_config.py b/autogpt_platform/single-container/tests/test_runtime_config.pyindex 23eec61718..aa274413cb 100644--- a/autogpt_platform/single-container/tests/test_runtime_config.py+++ b/autogpt_platform/single-container/tests/test_runtime_config.py@@ -68,6 +68,7 @@ class RuntimeConfigTest(unittest.TestCase):"ENCRYPTION_KEY","UNSUBSCRIBE_SECRET_KEY","GRAPHITI_FALKORDB_PASSWORD",+ "INTERNAL_API_KEY","VAPID_PRIVATE_KEY","VAPID_PUBLIC_KEY",},@@ -88,6 +89,7 @@ class RuntimeConfigTest(unittest.TestCase):"ENCRYPTION_KEY": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","UNSUBSCRIBE_SECRET_KEY": "u" * 40,"GRAPHITI_FALKORDB_PASSWORD": "f" * 40,+ "INTERNAL_API_KEY": "i" * 40,"VAPID_PRIVATE_KEY": vapid_private,"VAPID_PUBLIC_KEY": vapid_public,}@@ -219,8 +221,8 @@ class RuntimeConfigTest(unittest.TestCase):("unsupported version",lambda content: content.replace(- "AUTOGPT_RUNTIME_CONFIG_VERSION=1","AUTOGPT_RUNTIME_CONFIG_VERSION=2",+ "AUTOGPT_RUNTIME_CONFIG_VERSION=3",),"unsupported runtime configuration version",),@@ -256,6 +258,101 @@ class RuntimeConfigTest(unittest.TestCase):with self.assertRaisesRegex(ValueError, "not a regular file"):runtime_config.ensure_runtime_config(path, {})+ def test_version_1_config_gains_an_internal_key_and_keeps_the_rest(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "runtime.env"+ original = _write_version_1_config(path)+ path.chmod(0o644)++ upgraded = runtime_config.ensure_runtime_config(path, {})++ key = upgraded.pop("INTERNAL_API_KEY")+ self.assertGreaterEqual(len(key), 32)+ self.assertRegex(key, r"^[A-Za-z0-9_-]+$")+ self.assertEqual(upgraded["AUTOGPT_RUNTIME_CONFIG_VERSION"], "2")+ self.assertEqual(+ {**upgraded, "AUTOGPT_RUNTIME_CONFIG_VERSION": "1"}, original+ )+ self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600)+ again = runtime_config.ensure_runtime_config(path, {})+ self.assertEqual(again["INTERNAL_API_KEY"], key)+ self.assertEqual(again["AUTOGPT_RUNTIME_CONFIG_VERSION"], "2")++ def test_version_1_config_takes_a_configured_internal_key(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "runtime.env"+ _write_version_1_config(path)+ configured = {"INTERNAL_API_KEY": "k" * 40}++ upgraded = runtime_config.ensure_runtime_config(path, configured)++ self.assertEqual(upgraded["INTERNAL_API_KEY"], "k" * 40)+ self.assertEqual(+ runtime_config.ensure_runtime_config(path, configured),+ upgraded,+ )++ def test_rejects_version_1_config_that_does_not_match_version_1(self) -> None:+ cases = [+ ("extra key", lambda values: {**values, "SURPRISE": "s" * 40}),+ (+ "missing key",+ lambda values: {+ name: value+ for name, value in values.items()+ if name != "REDIS_PASSWORD"+ },+ ),+ (+ "already has an internal key",+ lambda values: {**values, "INTERNAL_API_KEY": "i" * 40},+ ),+ (+ "invalid secret",+ lambda values: {**values, "REDIS_PASSWORD": "short"},+ ),+ ]+ for name, mutate in cases:+ with self.subTest(name=name), tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "runtime.env"+ values = mutate(_version_1_values())+ _write_config_lines(path, values)+ before = path.read_bytes()++ with self.assertRaises(ValueError):+ runtime_config.ensure_runtime_config(path, {})+ self.assertEqual(path.read_bytes(), before)++ def test_the_version_1_step_produces_version_2_after_later_bumps(self) -> None:+ with mock.patch.object(runtime_config, "CONFIG_VERSION", "3"):+ upgraded = runtime_config._upgraded_from_version_1(_version_1_values(), {})++ self.assertEqual(upgraded["AUTOGPT_RUNTIME_CONFIG_VERSION"], "2")++ def test_a_refused_boot_leaves_a_version_1_config_as_it_was(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "runtime.env"+ _write_version_1_config(path)+ before = path.read_bytes()++ with self.assertRaisesRegex(ValueError, "persisted on first boot"):+ runtime_config.ensure_runtime_config(+ path, {"POSTGRES_PASSWORD": "x" * 40}+ )+ self.assertEqual(path.read_bytes(), before)++ def test_rejects_version_1_upgrade_with_an_unsafe_configured_key(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / "runtime.env"+ _write_version_1_config(path)+ before = path.read_bytes()++ with self.assertRaisesRegex(ValueError, "INTERNAL_API_KEY"):+ runtime_config.ensure_runtime_config(+ path, {"INTERNAL_API_KEY": "too-short"}+ )+ self.assertEqual(path.read_bytes(), before)+class PublicUrlTest(unittest.TestCase):def test_normalizes_origin(self) -> None:@@ -472,7 +569,8 @@ class FillEnvSecretsTest(unittest.TestCase):(root / relative_path).parent.mkdir(parents=True, exist_ok=True)(root / relative_path).write_text("", encoding="utf-8")(root / "backend/.env.default").write_text(- "ENCRYPTION_KEY=not-blank\nENCRYPTION_KEY=\nUNSUBSCRIBE_SECRET_KEY=\n",+ "ENCRYPTION_KEY=not-blank\nENCRYPTION_KEY=\nUNSUBSCRIBE_SECRET_KEY=\n"+ "INTERNAL_API_KEY=\n",encoding="utf-8",)(root / "frontend/.env.default").write_text(@@ -502,6 +600,38 @@ class FillEnvSecretsTest(unittest.TestCase):)+def _version_1_values() -> dict[str, str]:+ """Every key a version 1 image wrote, before INTERNAL_API_KEY existed."""+ return {+ "AUTOGPT_RUNTIME_CONFIG_VERSION": "1",+ "POSTGRES_PASSWORD": "p" * 40,+ "RABBITMQ_DEFAULT_USER": "autogpt",+ "RABBITMQ_DEFAULT_PASS": "r" * 40,+ "REDIS_PASSWORD": "d" * 40,+ "BETTER_AUTH_SECRET": "b" * 40,+ "ENCRYPTION_KEY": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",+ "UNSUBSCRIBE_SECRET_KEY": "u" * 40,+ "GRAPHITI_FALKORDB_PASSWORD": "f" * 40,+ "VAPID_PRIVATE_KEY": base64.urlsafe_b64encode(b"p" * 32).rstrip(b"=").decode(),+ "VAPID_PUBLIC_KEY": (+ base64.urlsafe_b64encode(b"\x04" + b"q" * 64).rstrip(b"=").decode()+ ),+ }+++def _write_version_1_config(path: Path) -> dict[str, str]:+ values = _version_1_values()+ _write_config_lines(path, values)+ return values+++def _write_config_lines(path: Path, values: dict[str, str]) -> None:+ lines = ["# Generated once by the AutoGPT all-in-one image."]+ lines += [f"{name}={value}" for name, value in values.items()]+ path.write_text("\n".join(lines) + "\n", encoding="ascii")+ path.chmod(0o600)++def _replace_config_value(content: str, name: str, value: str) -> str:prefix = f"{name}="lines = [diff --git a/autogpt_platform/swap_proxy/README.md b/autogpt_platform/swap_proxy/README.mdindex 7631d8c075..3e3f6181ef 100644--- a/autogpt_platform/swap_proxy/README.md+++ b/autogpt_platform/swap_proxy/README.md@@ -183,6 +183,7 @@ poetry run swap-proxy| --- | --- | --- || `SWAP_PROXY_LISTEN_HOST` / `SWAP_PROXY_LISTEN_PORT` | `0.0.0.0` / `1080` | SOCKS5 listener || `SWAP_PROXY_BACKEND_URL` | `http://localhost:8012` | the backend's swap service (`SwapCredentialService`), which serves only the two calls the proxy makes |+| `INTERNAL_API_KEY` | none: required | the key the backend's services require on every internal call, so the same value as the backend's own `INTERNAL_API_KEY`. The backend derives its key from `ENCRYPTION_KEY` when that is unset, and the proxy has no `ENCRYPTION_KEY`, so a deployment with the proxy sets one explicitly on both. The proxy does not start without it || `SWAP_PROXY_CONFDIR` | `~/.mitmproxy` | directory the signing CA is mounted into; see below || `SWAP_PROXY_GENERATE_CA` | `false` | local runs only: generate a CA if the directory has none || `SWAP_PROXY_EGRESS_ALLOW` | empty | comma-separated private hosts or CIDRs boxes may reach anyway |diff --git a/autogpt_platform/swap_proxy/swap_proxy/__main__.py b/autogpt_platform/swap_proxy/swap_proxy/__main__.pyindex 565b8c4043..e4ae5bcf48 100644--- a/autogpt_platform/swap_proxy/swap_proxy/__main__.py+++ b/autogpt_platform/swap_proxy/swap_proxy/__main__.py@@ -110,7 +110,7 @@ async def run() -> None:require_ca(settings.confdir, settings.generate_ca)redis = connect_redis(settings)await redis.ping()- source = BackendCredentialSource(settings.backend_url)+ source = BackendCredentialSource.from_settings(settings)# Raises on a setting that would quietly weaken the quota: no start.quota = RequestQuota(redis,diff --git a/autogpt_platform/swap_proxy/swap_proxy/settings.py b/autogpt_platform/swap_proxy/swap_proxy/settings.pyindex d7c2931391..cae3cd6fb7 100644--- a/autogpt_platform/swap_proxy/swap_proxy/settings.py+++ b/autogpt_platform/swap_proxy/swap_proxy/settings.py@@ -31,6 +31,11 @@ class Settings:"SWAP_PROXY_BACKEND_URL", "http://localhost:8012"))+ # The key the backend's services require on internal calls: the same+ # value as the backend's own INTERNAL_API_KEY.+ internal_api_key: str = field(+ default_factory=lambda: os.getenv("INTERNAL_API_KEY", "")+ )# Where the CA the proxy signs with is mounted (``mitmproxy-ca.pem``: key# and certificate). The boxes' image trusts its certificate, so it is one# CA for every replica and every restart: provisioned, never generated.diff --git a/autogpt_platform/swap_proxy/swap_proxy/source.py b/autogpt_platform/swap_proxy/swap_proxy/source.pyindex f43ef6af17..1285dd94fc 100644--- a/autogpt_platform/swap_proxy/swap_proxy/source.py+++ b/autogpt_platform/swap_proxy/swap_proxy/source.py@@ -16,11 +16,13 @@ and the request fails at the provider: loudly, and with nothing leaked."""import logging+import reimport timefrom typing import Optional, Protocolimport httpx+from swap_proxy.settings import Settingsfrom swap_proxy.swap import Credential, host_in_listlogger = logging.getLogger(__name__)@@ -33,6 +35,8 @@ _BINDINGS_RETRY = 30.0_CREDENTIAL_TTL = 15.0_CACHE_MAX = 10_000_TIMEOUT = httpx.Timeout(10.0, connect=3.0)+# The same rule as the backend's (backend/util/service_auth.py).+_VALID_KEY = re.compile(r"[A-Za-z0-9._~-]{32,}={0,2}")class SourceUnavailable(Exception):@@ -48,14 +52,40 @@ class CredentialSource(Protocol):class BackendCredentialSource:- def __init__(self, base_url: str, client: Optional[httpx.AsyncClient] = None):+ def __init__(+ self,+ base_url: str,+ client: Optional[httpx.AsyncClient] = None,+ api_key: str = "",+ ):self._base_url = base_url.rstrip("/")- self._client = client or httpx.AsyncClient(timeout=_TIMEOUT)+ # trust_env=False: an HTTP_PROXY in the environment must never carry+ # these plain-HTTP calls, nor the internal key they hold.+ self._client = client or httpx.AsyncClient(timeout=_TIMEOUT, trust_env=False)+ # The backend's internal service key, which its services require on+ # every call (backend/util/service_auth.py).+ self._headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}self._bindings: Optional[tuple[float, dict[str, tuple[str, ...]]]] = Noneself._credentials: dict[tuple[str, str, str, str], tuple[float, Optional[Credential]]] = {}+ @classmethod+ def from_settings(+ cls, settings: Settings, client: Optional[httpx.AsyncClient] = None+ ) -> "BackendCredentialSource":+ """Raises on a missing or malformed key, so the proxy does not start+ without one: the backend refuses every swap lookup that has none."""+ key = settings.internal_api_key+ if not _VALID_KEY.fullmatch(key):+ raise ValueError(+ "INTERNAL_API_KEY must be the backend's internal service key, "+ "at least 32 URL-safe characters: letters, digits, '-', '_', "+ "'.', '~' (and '=' padding). The backend refuses swap lookups "+ "without it."+ )+ return cls(settings.backend_url, client, key)+async def bindings(self) -> dict[str, tuple[str, ...]]:now = time.monotonic()if self._bindings and self._bindings[0] > now:@@ -112,7 +142,9 @@ class BackendCredentialSource:async def _call(self, method: str, body: dict):try:- response = await self._client.post(f"{self._base_url}/{method}", json=body)+ response = await self._client.post(+ f"{self._base_url}/{method}", json=body, headers=self._headers+ )response.raise_for_status()return response.json()except (httpx.HTTPError, ValueError) as e:diff --git a/autogpt_platform/swap_proxy/swap_proxy/source_test.py b/autogpt_platform/swap_proxy/swap_proxy/source_test.pyindex fc9b605708..17dbab279e 100644--- a/autogpt_platform/swap_proxy/swap_proxy/source_test.py+++ b/autogpt_platform/swap_proxy/swap_proxy/source_test.py@@ -1,12 +1,15 @@+import asyncioimport jsonimport httpximport pytestfrom swap_proxy import source as source_module+from swap_proxy.settings import Settingsfrom swap_proxy.source import BackendCredentialSource, SourceUnavailableTOKEN = "ghp_a-real-looking-token"+KEY = "internal-key-0123456789abcdefghijklmnop"def _backend(handler):@@ -36,6 +39,83 @@ def _ok(path, body):)+def _authorization_seen(api_key: str = ""):+ seen: list[str | None] = []++ def transport(request: httpx.Request) -> httpx.Response:+ seen.append(request.headers.get("authorization"))+ return _ok(request.url.path, json.loads(request.content or b"{}"))++ client = httpx.AsyncClient(transport=httpx.MockTransport(transport))+ return BackendCredentialSource("http://backend:8012/", client, api_key), seen+++async def test_every_call_carries_the_internal_key():+ source, seen = _authorization_seen(KEY)+ await source.bound_names("api.github.com")+ await source.resolve("user-a", "github", "api.github.com", "box-1")+ assert seen == [f"Bearer {KEY}", f"Bearer {KEY}"]+++async def test_without_a_key_no_authorization_is_sent():+ source, seen = _authorization_seen()+ await source.bound_names("api.github.com")+ await source.resolve("user-a", "github", "api.github.com", "box-1")+ assert seen == [None, None]+++async def test_the_key_is_the_backends_own_setting(monkeypatch):+ monkeypatch.setenv("INTERNAL_API_KEY", KEY)+ seen: list[str | None] = []++ def transport(request: httpx.Request) -> httpx.Response:+ seen.append(request.headers.get("authorization"))+ return _ok(request.url.path, {})++ client = httpx.AsyncClient(transport=httpx.MockTransport(transport))+ source = BackendCredentialSource.from_settings(Settings(), client)+ await source.bound_names("api.github.com")+ assert seen == [f"Bearer {KEY}"]++[email protected]("key", ["", "too-short", "a key with spaces " * 3, "é" * 40])+def test_a_missing_or_malformed_key_stops_the_proxy_starting(monkeypatch, key):+ monkeypatch.setenv("INTERNAL_API_KEY", key)+ with pytest.raises(ValueError, match="INTERNAL_API_KEY"):+ BackendCredentialSource.from_settings(Settings())+++def test_an_unset_key_stops_the_proxy_starting(monkeypatch):+ """Every lookup would be refused, while the proxy looked healthy."""+ monkeypatch.delenv("INTERNAL_API_KEY", raising=False)+ with pytest.raises(ValueError, match="INTERNAL_API_KEY"):+ BackendCredentialSource.from_settings(Settings())+++async def test_an_environment_proxy_never_sees_a_call(monkeypatch):+ """The calls are plain HTTP and carry the key: an outbound proxy set in the+ environment must not be where they go."""+ reached: list[bytes] = []++ async def trap(reader: asyncio.StreamReader, writer: asyncio.StreamWriter):+ reached.append(await reader.read(1024))+ writer.close()++ server = await asyncio.start_server(trap, "127.0.0.1", 0)+ proxy = f"http://127.0.0.1:{server.sockets[0].getsockname()[1]}"+ for name in ("HTTP_PROXY", "http_proxy", "ALL_PROXY", "all_proxy"):+ monkeypatch.setenv(name, proxy)+ for name in ("NO_PROXY", "no_proxy"):+ monkeypatch.delenv(name, raising=False)+ # A host that never resolves: going direct fails fast, a proxy would be asked.+ source = BackendCredentialSource("http://swap-service.invalid:8012", api_key=KEY)+ async with server:+ with pytest.raises(SourceUnavailable):+ await source.bindings()+ await source.aclose()+ assert reached == []++async def test_bound_names_come_from_the_bindings_table_asked_once():source, calls = _backend(_ok)assert await source.bound_names("api.github.com") == {"github"}diff --git a/docs/platform/getting-started.md b/docs/platform/getting-started.mdindex 8895a18572..6254ead3f7 100644--- a/docs/platform/getting-started.md+++ b/docs/platform/getting-started.md@@ -170,8 +170,8 @@ To run the platform, follow these steps:This copies each `.env.default` to `.env` (for `autogpt_platform`, `backend`and `frontend`) and fills in the secrets that `.env.default` deliberately- leaves blank — `ENCRYPTION_KEY`, `UNSUBSCRIBE_SECRET_KEY` and- `BETTER_AUTH_SECRET` — with values generated for your machine. Those files+ leaves blank — `ENCRYPTION_KEY`, `UNSUBSCRIBE_SECRET_KEY`, `INTERNAL_API_KEY`+ and `BETTER_AUTH_SECRET` — with values generated for your machine. Those filesare public, so shipping working values in them would mean every install inthe world shared one publicly-readable key. It is safe to re-run: it neveroverwrites an existing `.env` or a value you set yourself. You can then editdiff --git a/docs/platform/installer.md b/docs/platform/installer.mdindex 07cb9a4101..5060183099 100644--- a/docs/platform/installer.md+++ b/docs/platform/installer.md@@ -216,7 +216,8 @@ script uses that checkout; otherwise it clones the repository into a newBoth create the three `.env` files if they are missing and generate thesecrets `.env.default` leaves blank (`ENCRYPTION_KEY`,-`UNSUBSCRIBE_SECRET_KEY`, `BETTER_AUTH_SECRET`), the same as `make init-env`.+`UNSUBSCRIBE_SECRET_KEY`, `INTERNAL_API_KEY`, `BETTER_AUTH_SECRET`), the same+as `make init-env`.Running a script again never changes a value that is already set. If you areupgrading an install that ran without its own `ENCRYPTION_KEY`, read[Upgrading: secrets are generated per install](getting-started.md#upgrading-secrets-are-generated-per-install)diff --git a/docs/platform/single-container.md b/docs/platform/single-container.mdindex 1b53e2a5ae..1ba30dee36 100644--- a/docs/platform/single-container.md+++ b/docs/platform/single-container.md@@ -525,13 +525,15 @@ receive `BETTER_AUTH_SECRET`, configured `AUTH_*` social-login client secrets,and optional OpenAI, transcription, and legacy-JWT values because the Next.jsserver uses them. Those values remain server-side and are not baked into thebrowser bundle. The frontend does not receive the PostgreSQL superuser password,-RabbitMQ or Valkey passwords, the FalkorDB password, or encryption keys.+RabbitMQ or Valkey passwords, the FalkorDB password, the internal service key,+or encryption keys.-Generated database, queue, cache, memory, encryption, authentication, and-signing secrets are created on first boot and stored in+Generated database, queue, cache, memory, encryption, authentication,+signing, and internal service secrets are created on first boot and stored in`/data/config/runtime.env` as `root:root` mode `0600`. Reusing the named volumereuses those secrets. Supplying a different value for a persisted secret on a-later boot fails instead of silently rotating it.+later boot fails instead of silently rotating it. A volume created before the+internal service key existed gets one the first time a newer image starts.### Suspected secret exposure@@ -551,7 +553,8 @@ Treat the host environment file, `/data`, backups, and unredacted diagnosticoutput as secret-bearing material.Only port `3000` should be published. Internal AppService RPC is bound to the-container's loopback interface, and Valkey traffic requires authentication.+container's loopback interface and requires the internal service key, and+Valkey traffic requires authentication.On Linux, Docker-managed forwarding can bypass firewall policy expressed onlythrough tools such as UFW. The loopback address in`--publish 127.0.0.1:3000:3000` is the exposure control; changing it to a