AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
fix(platform): generate local secrets at bootstrap instead of shipping them (#14518)
Krzysztof Czerwinski8da6c931be8b
diff --git a/.github/workflows/copilot-setup-steps.yml b/.github/workflows/copilot-setup-steps.ymlindex 9e43118569..2c069611a5 100644--- a/.github/workflows/copilot-setup-steps.yml+++ b/.github/workflows/copilot-setup-steps.yml@@ -123,11 +123,9 @@ jobs:- name: Copy default environment filesworking-directory: autogpt_platform- run: |- # Copy default environment files for development- cp .env.default .env- cp backend/.env.default backend/.env- cp frontend/.env.default frontend/.env+ # Creates each .env from its .env.default and generates the secrets that+ # .env.default deliberately leaves blank.+ run: make init-env# Phase 1: Cache and load Docker images for faster setup- name: Set up Docker image cachediff --git a/.github/workflows/platform-backend-ci.yml b/.github/workflows/platform-backend-ci.ymlindex e55d19f12c..94c51011d1 100644--- a/.github/workflows/platform-backend-ci.yml+++ b/.github/workflows/platform-backend-ci.yml@@ -548,6 +548,13 @@ jobs:export DIRECT_URL="$DATABASE_URL"poetry run prisma migrate deploy+ - name: Generate ephemeral secrets+ # A fixed value here would be a published value. These live only for+ # the duration of this job.+ run: |+ echo "ENCRYPTION_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"+ echo "UNSUBSCRIBE_SECRET_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"+- name: Test shardshell: bashenv:@@ -559,7 +566,6 @@ jobs:EXECUTION_SCHEDULER_PORT: "18003"NOTIFICATION_SERVICE_PORT: "18007"REDIS_HOST: localhost- ENCRYPTION_KEY: "dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw="TRIAL_TEST_DATABASE: "1"run: |export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/${DATABASE_NAME}"diff --git a/.github/workflows/platform-fullstack-ci.yml b/.github/workflows/platform-fullstack-ci.ymlindex 8698a4f2c0..865585dd9a 100644--- a/.github/workflows/platform-fullstack-ci.yml+++ b/.github/workflows/platform-fullstack-ci.yml@@ -160,13 +160,14 @@ jobs:with:submodules: recursive- - name: Set up Platform - Copy default platform .env- run: |- cp ../.env.default ../.env+ - name: Set up Platform - Create .env files with generated secrets+ # Same bootstrap a contributor runs: .env.default ships its secrets+ # blank, and this fills them with values generated for this run.+ working-directory: autogpt_platform+ run: make init-env- - name: Set up Platform - Copy backend .env and set OpenAI API key+ - name: Set up Platform - Configure backend .envrun: |- cp ../backend/.env.default ../backend/.envecho "OPENAI_INTERNAL_API_KEY=${{ secrets.OPENAI_API_KEY }}" >> ../backend/.envecho "CHAT_TEST_MODE=true" >> ../backend/.envecho "SCHEDULER_STARTUP_EMBEDDING_BACKFILL=false" >> ../backend/.envdiff --git a/.github/workflows/platform-preview-seed-fixture.yml b/.github/workflows/platform-preview-seed-fixture.ymlindex eeba3f4073..c2a4e9a0ad 100644--- a/.github/workflows/platform-preview-seed-fixture.yml+++ b/.github/workflows/platform-preview-seed-fixture.yml@@ -86,9 +86,6 @@ jobs:# so they only need to exist. Nothing in this job ever connects to them.SUPABASE_URL: "http://localhost:54321"SUPABASE_SERVICE_ROLE_KEY: "synthetic-bake-no-real-key" # pragma: allowlist secret- # Non-production test key (identical to the one used in backend CI) so- # backend module imports that read settings.secrets don't error.- ENCRYPTION_KEY: "dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw=" # pragma: allowlist secret # DO NOT USE IN PRODUCTIONsteps:- name: Checkout repository@@ -102,6 +99,11 @@ jobs:# persist credentials into the workspace git config.persist-credentials: false+ - name: Generate ephemeral secrets+ # backend module imports read settings.secrets, so ENCRYPTION_KEY has to+ # exist. Generated per run: a value committed here would be published.+ run: echo "ENCRYPTION_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> "$GITHUB_ENV"+- name: Set up Python 3.12uses: actions/setup-python@v5with:diff --git a/.gitleaks.toml b/.gitleaks.tomlindex 75867a7f50..641c3034e5 100644--- a/.gitleaks.toml+++ b/.gitleaks.toml@@ -6,8 +6,6 @@ useDefault = true[allowlist]description = "Global allowlist"paths = [- # Template/example env files (no real secrets)- '''\.env\.(default|example|template)$''',# Lock files'''pnpm-lock\.yaml$''','''poetry\.lock$''',@@ -28,9 +26,7 @@ paths = ['''google-services\.json$''','''classic/frontend/(lib|web)/''',]-# CI test-only encryption key (marked DO NOT USE IN PRODUCTION)regexes = [- '''dvziYgz0KSK8FENhju0ZYi8''',# LLM model name enum values falsely flagged as API keys'''Llama-\d.*Instruct''',]diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yamlindex d8ae4c8b31..7f7f35d364 100644--- a/.pre-commit-config.yaml+++ b/.pre-commit-config.yaml@@ -27,7 +27,9 @@ repos:files: ^autogpt_platform/# The style eval's baseline is generated and holds sha256 content# hashes of the assembled prompts, which read as high-entropy hex.- exclude: (pnpm-lock\.yaml|\.env\.(default|example|template)|copilot/eval/style/baseline\.json)$+ # .env.default files are deliberately NOT excluded: they are public, so+ # a working value committed to one is a published secret (SECRT-2611).+ exclude: (pnpm-lock\.yaml|copilot/eval/style/baseline\.json)$- repo: https://github.com/gitleaks/gitleaksrev: v8.24.3@@ -35,6 +37,21 @@ repos:- id: gitleaksname: Detect secrets (gitleaks)+ - repo: local+ hooks:+ - id: check-env-defaults-blank+ name: Check .env.default ships no working secrets+ description: >-+ .env.default is public, so a working value committed to one is a+ published secret. Entropy scanners miss a freshly generated key, so+ the retired settings are asserted blank directly (SECRT-2611).+ entry: >-+ python3 autogpt_platform/single-container/runtime_config.py+ check-env-defaults --root autogpt_platform+ language: system+ files: ^autogpt_platform/(backend|frontend)/\.env\.default$+ pass_filenames: false+- repo: local# For proper type checking, all dependencies need to be up-to-date.# It's also a good idea to check that poetry.lock is consistent with pyproject.toml.diff --git a/.secrets.baseline b/.secrets.baselineindex 17dab85c38..27a2d5afca 100644--- a/.secrets.baseline+++ b/.secrets.baseline@@ -130,7 +130,6 @@"pattern": ["\\.env$","pnpm-lock\\.yaml$",- "\\.env\\.(default|example|template)$","__pycache__","_test\\.py$","test_.*\\.py$",@@ -141,6 +140,16 @@}],"results": {+ "autogpt_platform/.env.default": [+ {+ "type": "Secret Keyword",+ "filename": "autogpt_platform/.env.default",+ "hashed_secret": "de6b16a1aa2c97b8963a4f966774ae09fbe47300",+ "is_verified": false,+ "line_number": 15,+ "is_secret": false+ }+ ],"autogpt_platform/backend/backend/api/external/v1/integrations.py": [{"type": "Secret Keyword",@@ -437,6 +446,16 @@"is_secret": false}],+ "autogpt_platform/frontend/.env.default": [+ {+ "type": "Basic Auth Credentials",+ "filename": "autogpt_platform/frontend/.env.default",+ "hashed_secret": "de6b16a1aa2c97b8963a4f966774ae09fbe47300",+ "is_verified": false,+ "line_number": 12,+ "is_secret": false+ }+ ],"autogpt_platform/frontend/src/app/(platform)/build/components/FlowEditor/nodes/helpers.ts": [{"type": "Secret Keyword",diff --git a/autogpt_platform/AGENTS.md b/autogpt_platform/AGENTS.mdindex 3c72d39071..cc1fd2781a 100644--- a/autogpt_platform/AGENTS.md+++ b/autogpt_platform/AGENTS.md@@ -38,6 +38,32 @@ AutoGPT Platform is a monorepo containing:3. Docker Compose `environment:` sections provide service-specific overrides4. Shell environment variables have highest precedence+#### Never commit a working secret value++`.env.default` files are public. Any real key, token, password, or signing+secret committed to one is a **published** secret: scanners pick it up, it gets+filed as a GitHub security advisory against the repo, and self-hosters end up+running on a value anyone can read — even when the hosted platform never uses+it. Deleting it later does not unpublish it.++For any new secret setting:++1. Leave it **blank** in `.env.default` (keep the key and its comment).+2. Add it to `SECRET_GENERATORS` in `single-container/runtime_config.py`.+3. Let `make init-env` generate a per-developer value into `.env`.++If the stack genuinely cannot boot without a local default, it must be+*generated at setup time*, not checked in. This applies equally to+`.env.default` files, CI workflow `env:` blocks, and docker-compose+`environment:` blocks — generate ephemeral values in CI rather than pasting a+literal. Two things enforce this: `.gitleaks.toml` and the `detect-secrets`+pre-commit hook no longer exempt `.env.default` files, and the+`check-env-defaults-blank` pre-commit hook (plus the matching unit test in+`single-container/tests/test_runtime_config.py`, which CI runs) asserts that+the settings listed in `BLANK_IN_ENV_DEFAULT` stay empty. Entropy scanners+alone do not reliably flag a freshly generated key, so add each newly-retired+setting to `BLANK_IN_ENV_DEFAULT` too.+#### Key Points- All services use hardcoded defaults in docker-compose files (no `${VARIABLE}` substitutions)diff --git a/autogpt_platform/Makefile b/autogpt_platform/Makefileindex f69251fe30..3b76efdd0b 100644--- a/autogpt_platform/Makefile+++ b/autogpt_platform/Makefile@@ -25,10 +25,17 @@ format:cd frontend && pnpm formatcd frontend && pnpm lint+# Create any missing .env from its .env.default, then fill the secrets that+# .env.default deliberately leaves blank (see single-container/runtime_config.py+# SECRET_GENERATORS). Safe to re-run: existing files and existing values are+# never overwritten.init-env:cp -n .env.default .env || truecd backend && cp -n .env.default .env || truecd frontend && cp -n .env.default .env || true+ python3 single-container/runtime_config.py fill-env --missing-ok --path .env+ python3 single-container/runtime_config.py fill-env --missing-ok --path backend/.env+ python3 single-container/runtime_config.py fill-env --missing-ok --path frontend/.env# Run migrations for backenddiff --git a/autogpt_platform/backend/.env.default b/autogpt_platform/backend/.env.defaultindex 6037704873..5ceda6fdab 100644--- a/autogpt_platform/backend/.env.default+++ b/autogpt_platform/backend/.env.default@@ -47,9 +47,14 @@ JWT_JWKS_URL=http://localhost:3000/api/auth/jwks# JWKS_ALLOW_INSECURE_TRANSPORT=false## ===== REQUIRED SECURITY KEYS ===== ##-# Generate using: from cryptography.fernet import Fernet;Fernet.generate_key().decode()-ENCRYPTION_KEY=dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw=-UNSUBSCRIBE_SECRET_KEY=HlP8ivStJjmbf6NKi78m_3FnOogut0t5ckzjsIqeaio=+# Left blank on purpose. This file is public, so any value committed here is a+# published value: the keys that used to sit below are readable by anyone and+# must never be used again. `make init-env` fills each blank with a secret+# generated for your machine when it creates backend/.env, and never overwrites+# a value you set yourself. The backend refuses to start without ENCRYPTION_KEY.+# To make one by hand: from cryptography.fernet import Fernet;Fernet.generate_key().decode()+ENCRYPTION_KEY=+UNSUBSCRIBE_SECRET_KEY=# Web Push (VAPID) — generate with: poetry run python -c "# from py_vapid import Vapid; import base64diff --git a/autogpt_platform/backend/backend/util/process.py b/autogpt_platform/backend/backend/util/process.pyindex 664b7f3c28..84ecd13347 100644--- a/autogpt_platform/backend/backend/util/process.py+++ b/autogpt_platform/backend/backend/util/process.py@@ -9,6 +9,7 @@ from typing import Optionalfrom backend.util.logging import configure_loggingfrom backend.util.metrics import sentry_initfrom backend.util.retry import stop_retry_loops+from backend.util.secrets_guard import check_secretsfrom backend.util.settings import set_service_namelogger = logging.getLogger(__name__)@@ -122,6 +123,11 @@ class AppProcess(ABC):Returns:the process id or 0 if the process is not running in the background."""+ # Startup guard: no service comes up on a missing or publicly-known+ # secret. Deliberately here rather than at import time so tooling and+ # tests that merely import backend modules are unaffected.+ check_secrets()+if not background:self.execute_run_command(silent)return 0diff --git a/autogpt_platform/backend/backend/util/secrets_guard.py b/autogpt_platform/backend/backend/util/secrets_guard.pynew file mode 100644index 0000000000..e760b6535b--- /dev/null+++ b/autogpt_platform/backend/backend/util/secrets_guard.py@@ -0,0 +1,81 @@+"""Refuse to boot on secrets that were once shipped in a public file.++`backend/.env.default` carried working values for these settings until+SECRT-2611. The file is public, so those values are public: a deployment still+using one encrypts stored integration credentials with a key anybody can read,+and signs Redis cache entries with a forgeable HMAC key (`util/cache.py`+derives that key from `ENCRYPTION_KEY`).++The retired values are matched by SHA-256 digest, so detecting them does not+require putting the literals back into a tracked file.+"""++import hashlib+import logging++from backend.util.settings import Settings++logger = logging.getLogger(__name__)++# `make init-env` only fills `NAME=` lines that are present and blank, so each+# hint spells out the one edit that makes it apply.+MISSING_HINT = (+ "Add a line `ENCRYPTION_KEY=` to autogpt_platform/backend/.env (create the "+ "file if needed) and run `make init-env` in autogpt_platform/ to generate a "+ "value for it. Values you already set are never overwritten."+)+RETIRED_HINT = (+ "Clear the value in autogpt_platform/backend/.env (leave `{name}=`) and run "+ "`make init-env` in autogpt_platform/ to generate a fresh one."+)++# SHA-256 of each value published in backend/.env.default before SECRT-2611.+# Treat all of them as permanently burned.+_RETIRED_DIGESTS: dict[str, set[str]] = {+ "ENCRYPTION_KEY": {+ "731473d9f10755f30ae97baafe65bf0aba95f83100ed418a7c3bc257a2a1ef7c" # pragma: allowlist secret+ },+ "UNSUBSCRIBE_SECRET_KEY": {+ "0a2f4a5ce1e9e072b5773bccab2839f62d46e61cc81fdf6aba590f7983694517" # pragma: allowlist secret+ },+}+++def check_secrets(settings: Settings | None = None) -> None:+ """Raise if a required secret is missing or is a known-published value."""+ secrets = (settings or Settings()).secrets+ configured = {+ "ENCRYPTION_KEY": secrets.encryption_key,+ "UNSUBSCRIBE_SECRET_KEY": secrets.unsubscribe_secret_key,+ }++ if not configured["ENCRYPTION_KEY"]:+ raise ValueError(+ "ENCRYPTION_KEY is not set. It encrypts stored integration "+ "credentials and signs cached values, so the backend will not "+ f"start without it. {MISSING_HINT}"+ )++ for name, value in configured.items():+ if not value or hashlib.sha256(value.encode()).hexdigest() not in (+ _RETIRED_DIGESTS[name]+ ):+ continue+ rotation_note = (+ " Credentials stored under the old key become unreadable, so those "+ "integrations need reconnecting."+ if name == "ENCRYPTION_KEY"+ else ""+ )+ raise ValueError(+ f"{name} is set to a value that was published in this repository's "+ "public .env.default and must be treated as compromised. "+ f"{RETIRED_HINT.format(name=name)}{rotation_note}"+ )++ if not configured["UNSUBSCRIBE_SECRET_KEY"]:+ logger.warning(+ "[SECURITY] UNSUBSCRIBE_SECRET_KEY is not set: email unsubscribe "+ "links cannot be signed. "+ + MISSING_HINT.replace("ENCRYPTION_KEY", "UNSUBSCRIBE_SECRET_KEY")+ )diff --git a/autogpt_platform/backend/backend/util/secrets_guard_test.py b/autogpt_platform/backend/backend/util/secrets_guard_test.pynew file mode 100644index 0000000000..d411bdec43--- /dev/null+++ b/autogpt_platform/backend/backend/util/secrets_guard_test.py@@ -0,0 +1,72 @@+import base64+import hashlib+import os++import pytest++from backend.util import secrets_guard+from backend.util.secrets_guard import check_secrets+from backend.util.settings import Settings++# The values this guard rejects are deliberately NOT reproduced here: putting+# them back in a tracked file is the exact problem SECRT-2611 fixes. To check a+# registered digest against what was actually published, re-derive it from git+# history instead:+# git show 79168d31f1:autogpt_platform/backend/.env.default \+# | grep -E '^(ENCRYPTION_KEY|UNSUBSCRIBE_SECRET_KEY)=' \+# | cut -d= -f2- | tr -d '\n' | shasum -a 256+++def _settings(encryption_key: str, unsubscribe_key: str = "fresh-unsubscribe-key"):+ settings = Settings()+ settings.secrets.encryption_key = encryption_key+ settings.secrets.unsubscribe_secret_key = unsubscribe_key+ return settings+++def _fresh_key() -> str:+ return base64.urlsafe_b64encode(os.urandom(32)).decode()+++def test_generated_secrets_pass():+ check_secrets(_settings(_fresh_key()))+++def test_missing_encryption_key_is_refused_with_bootstrap_hint():+ with pytest.raises(ValueError, match="make init-env"):+ check_secrets(_settings(""))+++def test_blank_unsubscribe_key_only_warns(caplog):+ check_secrets(_settings(_fresh_key(), ""))+ assert "UNSUBSCRIBE_SECRET_KEY" in caplog.text+++ "name, build_settings",+ [+ ("ENCRYPTION_KEY", lambda value: _settings(value)),+ ("UNSUBSCRIBE_SECRET_KEY", lambda value: _settings(_fresh_key(), value)),+ ],+)+def test_registered_published_value_is_refused(monkeypatch, name, build_settings):+ """A configured value whose digest is registered as published is refused."""+ published = f"pretend-this-was-shipped-in-env-default-{name}"+ monkeypatch.setitem(+ secrets_guard._RETIRED_DIGESTS,+ name,+ {hashlib.sha256(published.encode()).hexdigest()},+ )++ with pytest.raises(ValueError, match=f"{name}.*compromised"):+ check_secrets(build_settings(published))+++def test_guard_covers_every_secret_that_env_default_used_to_ship():+ assert set(secrets_guard._RETIRED_DIGESTS) == {+ "ENCRYPTION_KEY",+ "UNSUBSCRIBE_SECRET_KEY",+ }+ for digests in secrets_guard._RETIRED_DIGESTS.values():+ assert digests, "a name with no digests silently disables the guard"+ assert all(len(digest) == 64 for digest in digests)diff --git a/autogpt_platform/frontend/.env.default b/autogpt_platform/frontend/.env.defaultindex d19bfad573..0d7ca01e1a 100644--- a/autogpt_platform/frontend/.env.default+++ b/autogpt_platform/frontend/.env.default@@ -1,6 +1,9 @@# Better Auth (runs inside this Next.js app at /api/auth/*)-# Dev-only default. MUST be overridden with a real secret in production.-BETTER_AUTH_SECRET=dev-only-better-auth-secret-9f4e8c2a71b6d05e3a8f12c4b7d9e6a1+# Left blank on purpose. This file is public, so any value committed here is a+# published value, and a session cookie signed with a published secret can be+# forged. `make init-env` fills this with a secret generated for your machine+# when it creates frontend/.env, and never overwrites a value you set yourself.+BETTER_AUTH_SECRET=BETTER_AUTH_URL=http://localhost:3000# Postgres connection for Better Auth; tables live in the platform schema# (created by the backend's Prisma migrations). This matches the local `db`diff --git a/autogpt_platform/single-container/runtime_config.py b/autogpt_platform/single-container/runtime_config.pyindex 93683c6d91..a337fb5dfb 100755--- a/autogpt_platform/single-container/runtime_config.py+++ b/autogpt_platform/single-container/runtime_config.py@@ -20,6 +20,8 @@ CONFIG_VERSION = "1"SAFE_SECRET = re.compile(r"^[A-Za-z0-9._~-]+={0,2}$")SAFE_USERNAME = re.compile(r"^[A-Za-z0-9._-]+$")SAFE_DNS_LABEL = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")+# A ``NAME=`` line in a .env file, i.e. a secret left blank for setup to fill.+BLANK_ASSIGNMENT = re.compile(r"^(?P<name>[A-Za-z_][A-Za-z0-9_]*)=[ \t]*\r?\n?$")def main() -> int:@@ -29,6 +31,13 @@ def main() -> int:ensure_parser = subparsers.add_parser("ensure")ensure_parser.add_argument("--path", type=Path, required=True)+ fill_parser = subparsers.add_parser("fill-env")+ fill_parser.add_argument("--path", type=Path, required=True)+ fill_parser.add_argument("--missing-ok", action="store_true")++ check_parser = subparsers.add_parser("check-env-defaults")+ check_parser.add_argument("--root", type=Path, required=True)+url_parser = subparsers.add_parser("validate-public-url")url_parser.add_argument("url")@@ -36,6 +45,26 @@ def main() -> int:try:if args.command == "ensure":ensure_runtime_config(args.path, os.environ)+ elif args.command == "fill-env":+ if args.missing_ok and not args.path.exists():+ return 0+ if args.path.is_symlink():+ print(f"{args.path}: symlink, left untouched", file=sys.stderr)+ return 0+ names = fill_env_blanks(args.path)+ if names:+ print(f"{args.path}: generated {', '.join(names)}")+ elif args.command == "check-env-defaults":+ offenders = check_env_defaults(args.root)+ for location in offenders:+ print(+ f"{location}: ships a working secret value. .env.default is "+ "public, so this value is published. Leave it blank and let "+ "`make init-env` generate one.",+ file=sys.stderr,+ )+ if offenders:+ return 1else:print(validate_public_url(args.url))except (OSError, ValueError) as exc:@@ -60,6 +89,60 @@ def ensure_runtime_config(path: Path, environment: Mapping[str, str]) -> dict[streturn values+def fill_env_blanks(path: Path) -> list[str]:+ """Give every blank secret in a .env file a freshly generated value.++ Idempotent by construction: only ``NAME=`` lines with nothing after the+ ``=`` are touched, so re-running setup never rotates a value an operator+ (or an earlier run) already put there. Returns the names that were filled.+ """+ if path.is_symlink():+ raise ValueError(f"refusing symlink at {path}")+ lines = path.read_text(encoding="utf-8").splitlines(keepends=True)++ filled: list[str] = []+ for index, line in enumerate(lines):+ match = BLANK_ASSIGNMENT.match(line)+ if match is None:+ continue+ generator = SECRET_GENERATORS.get(match.group("name"))+ if generator is None:+ continue+ lines[index] = f"{match.group('name')}={generator()}\n"+ filled.append(match.group("name"))++ if filled:+ _replace_atomically(path, "".join(lines))+ return filled+++def check_env_defaults(root: Path) -> list[str]:+ """Return `path:NAME` for every retired secret not assigned exactly once, blank."""+ offenders: list[str] = []+ for relative_path, names in BLANK_IN_ENV_DEFAULT.items():+ content = (root / relative_path).read_text(encoding="utf-8")+ for name in names:+ values = re.findall(rf"(?m)^{re.escape(name)}=([^\r\n]*)\r?$", content)+ if values != [""]:+ offenders.append(f"{relative_path}:{name}")+ return offenders+++def _replace_atomically(path: Path, content: str) -> None:+ """Rewrite `path` owner-only, with no window where it is truncated or+ half-written. The file now holds generated secrets, so it stops inheriting+ whatever mode the copied `.env.default` had."""+ temporary = path.with_name(f".{path.name}.{secrets.token_hex(8)}.tmp")+ descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)+ try:+ with os.fdopen(descriptor, "w", encoding="utf-8") as stream:+ stream.write(content)+ os.replace(temporary, path)+ except BaseException:+ temporary.unlink(missing_ok=True)+ raise++def validate_public_url(value: str) -> str:"""Validate and normalize the externally reachable origin."""parsed = urlsplit(value)@@ -107,30 +190,11 @@ def _new_values(environment: Mapping[str, str]) -> dict[str, str]:vapid_private, vapid_public = _configured_or_generated_vapid(environment)values = {"AUTOGPT_RUNTIME_CONFIG_VERSION": CONFIG_VERSION,- "POSTGRES_PASSWORD": _configured_or_generated(- environment, "POSTGRES_PASSWORD", lambda: secrets.token_urlsafe(36)- ),"RABBITMQ_DEFAULT_USER": environment.get("RABBITMQ_DEFAULT_USER") or "autogpt",- "RABBITMQ_DEFAULT_PASS": _configured_or_generated(- environment, "RABBITMQ_DEFAULT_PASS", lambda: secrets.token_urlsafe(36)- ),- "REDIS_PASSWORD": _configured_or_generated(- environment, "REDIS_PASSWORD", lambda: secrets.token_urlsafe(36)- ),- "BETTER_AUTH_SECRET": _configured_or_generated(- environment, "BETTER_AUTH_SECRET", lambda: secrets.token_urlsafe(48)- ),- "ENCRYPTION_KEY": _configured_or_generated(- environment, "ENCRYPTION_KEY", _fernet_key- ),- "UNSUBSCRIBE_SECRET_KEY": _configured_or_generated(- environment, "UNSUBSCRIBE_SECRET_KEY", lambda: secrets.token_urlsafe(36)- ),- "GRAPHITI_FALKORDB_PASSWORD": _configured_or_generated(- environment,- "GRAPHITI_FALKORDB_PASSWORD",- lambda: secrets.token_urlsafe(36),- ),+ **{+ name: _configured_or_generated(environment, name, generator)+ for name, generator in SECRET_GENERATORS.items()+ },"VAPID_PRIVATE_KEY": vapid_private,"VAPID_PUBLIC_KEY": vapid_public,}@@ -176,6 +240,29 @@ def _fernet_key() -> str:return base64.urlsafe_b64encode(os.urandom(32)).decode("ascii")+# The single source of truth for "this setting is a secret, and this is how a+# local one is made". `.env.default` ships these blank (a working value in a+# public file is a published value); every bootstrap path fills them from here.+SECRET_GENERATORS: dict[str, Callable[[], str]] = {+ "POSTGRES_PASSWORD": lambda: secrets.token_urlsafe(36),+ "RABBITMQ_DEFAULT_PASS": lambda: secrets.token_urlsafe(36),+ "REDIS_PASSWORD": lambda: secrets.token_urlsafe(36),+ "BETTER_AUTH_SECRET": lambda: secrets.token_urlsafe(48),+ "ENCRYPTION_KEY": _fernet_key,+ "UNSUBSCRIBE_SECRET_KEY": lambda: secrets.token_urlsafe(36),+ "GRAPHITI_FALKORDB_PASSWORD": lambda: secrets.token_urlsafe(36),+}++# Secrets already retired from the public .env.default files (SECRT-2611).+# `check-env-defaults` keeps them blank: entropy scanners do not reliably catch+# a freshly generated key pasted back into one of these lines, so the invariant+# is asserted directly. Paths are relative to autogpt_platform/.+BLANK_IN_ENV_DEFAULT: dict[str, tuple[str, ...]] = {+ "backend/.env.default": ("ENCRYPTION_KEY", "UNSUBSCRIBE_SECRET_KEY"),+ "frontend/.env.default": ("BETTER_AUTH_SECRET",),+}++def _base64url(value: bytes) -> str:return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii")diff --git a/autogpt_platform/single-container/tests/test_runtime_config.py b/autogpt_platform/single-container/tests/test_runtime_config.pyindex 9e24d25cc9..23eec61718 100644--- a/autogpt_platform/single-container/tests/test_runtime_config.py+++ b/autogpt_platform/single-container/tests/test_runtime_config.py@@ -404,6 +404,104 @@ class FatalListenerTest(unittest.TestCase):self.assertEqual(calls, [])+class FillEnvSecretsTest(unittest.TestCase):+ """`make init-env` bootstrap: .env.default ships secrets blank (SECRT-2611)."""++ def test_fills_only_blank_known_secrets(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / ".env"+ path.write_text(+ "# comment\n"+ "ENCRYPTION_KEY=\n"+ "UNSUBSCRIBE_SECRET_KEY=\n"+ "OPENAI_API_KEY=\n"+ "DB_PASS=already-set\n",+ encoding="utf-8",+ )++ filled = runtime_config.fill_env_blanks(path)++ self.assertEqual(filled, ["ENCRYPTION_KEY", "UNSUBSCRIBE_SECRET_KEY"])+ values = dict(+ line.split("=", 1)+ for line in path.read_text(encoding="utf-8").splitlines()+ if "=" in line+ )+ self.assertEqual(values["DB_PASS"], "already-set")+ # Not a secret this bootstrap owns: a blank must stay blank.+ self.assertEqual(values["OPENAI_API_KEY"], "")+ self.assertEqual(+ len(base64.urlsafe_b64decode(values["ENCRYPTION_KEY"])), 32+ )+ self.assertTrue(values["UNSUBSCRIBE_SECRET_KEY"])++ def test_rerunning_setup_does_not_rotate_existing_values(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / ".env"+ path.write_text("ENCRYPTION_KEY=\n", encoding="utf-8")++ runtime_config.fill_env_blanks(path)+ after_first = path.read_text(encoding="utf-8")+ self.assertEqual(runtime_config.fill_env_blanks(path), [])+ self.assertEqual(path.read_text(encoding="utf-8"), after_first)++ def test_generated_values_differ_between_installs(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ keys = []+ for index in range(2):+ path = Path(directory) / f"{index}.env"+ path.write_text("ENCRYPTION_KEY=\n", encoding="utf-8")+ runtime_config.fill_env_blanks(path)+ keys.append(path.read_text(encoding="utf-8"))+ self.assertNotEqual(keys[0], keys[1])++ def test_generated_file_becomes_owner_only(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ path = Path(directory) / ".env"+ path.write_text("ENCRYPTION_KEY=\n", encoding="utf-8")+ path.chmod(0o644)++ runtime_config.fill_env_blanks(path)++ self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600)++ def test_check_env_defaults_rejects_populated_duplicate(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ for relative_path in runtime_config.BLANK_IN_ENV_DEFAULT:+ (root / relative_path).parent.mkdir(parents=True, exist_ok=True)+ (root / relative_path).write_text("", encoding="utf-8")+ (root / "backend/.env.default").write_text(+ "ENCRYPTION_KEY=not-blank\nENCRYPTION_KEY=\nUNSUBSCRIBE_SECRET_KEY=\n",+ encoding="utf-8",+ )+ (root / "frontend/.env.default").write_text(+ "BETTER_AUTH_SECRET=\n", encoding="utf-8"+ )++ self.assertEqual(+ runtime_config.check_env_defaults(root),+ ["backend/.env.default:ENCRYPTION_KEY"],+ )++ def test_env_default_ships_its_secrets_blank_and_generatable(self) -> None:+ """`.env.default` is public, so these must ship blank (SECRT-2611) —+ and every blank one must be fillable, or setup leaves a broken stack."""+ platform_root = Path(__file__).parents[2]+ self.assertEqual(runtime_config.check_env_defaults(platform_root), [])++ with tempfile.TemporaryDirectory() as directory:+ for relative_path, names in runtime_config.BLANK_IN_ENV_DEFAULT.items():+ content = (platform_root / relative_path).read_text(encoding="utf-8")+ copy = Path(directory) / relative_path.replace("/", "_")+ copy.write_text(content, encoding="utf-8")++ # Blank is only safe if setup can actually fill it back in.+ self.assertEqual(+ sorted(runtime_config.fill_env_blanks(copy)), sorted(names)+ )++def _replace_config_value(content: str, name: str, value: str) -> str:prefix = f"{name}="lines = [diff --git a/docs/platform/getting-started.md b/docs/platform/getting-started.mdindex 9d1210d8d6..04c68246e0 100644--- a/docs/platform/getting-started.md+++ b/docs/platform/getting-started.md@@ -80,13 +80,23 @@ To run the platform, follow these steps:cd AutoGPT/autogpt_platform```-- Copy the `.env.default` file to `.env` in `autogpt_platform`:+- Create the `.env` files and generate your local secrets:```- cp .env.default .env+ make init-env```- This command will copy the `.env.default` file to `.env` in the `autogpt_platform` directory. You can modify the `.env` file to add your own environment variables.+ This copies each `.env.default` to `.env` (for `autogpt_platform`, `backend`+ and `frontend`) and fills in the secrets that `.env.default` deliberately+ leaves blank — `ENCRYPTION_KEY`, `UNSUBSCRIBE_SECRET_KEY` and+ `BETTER_AUTH_SECRET` — with values generated for your machine. Those files+ are public, so shipping working values in them would mean every install in+ the world shared one publicly-readable key. It is safe to re-run: it never+ overwrites an existing `.env` or a value you set yourself. You can then edit+ the `.env` files to add your own environment variables.++ The backend **refuses to start** while `ENCRYPTION_KEY` is empty, so run this+ before bringing the stack up.- Run the platform services:```@@ -106,7 +116,7 @@ Inside the `autogpt_platform` directory, you can use:| Command | What it Does ||------------------------|-------------------------------------------------------------------------------|-| `make init-env` | Create missing `.env` files from `.env.default` (`autogpt_platform`, `backend`, and `frontend`) |+| `make init-env` | Create missing `.env` files from `.env.default` (`autogpt_platform`, `backend`, and `frontend`) and generate the secrets they leave blank || `make start-core` | Start just the core services (Postgres, Redis, RabbitMQ) in background || `make stop-core` | Stop the core services || `make logs-core` | Tail the logs for core services |@@ -162,6 +172,15 @@ three things changed:`.env` against its `.env.default` and copy the new keys across yourself.The `SUPABASE_*` URL/key variables are gone; the frontend now uses`BETTER_AUTH_SECRET` and `DATABASE_URL`.++ **Rotate your secrets.** `backend/.env.default` used to ship working values+ for `ENCRYPTION_KEY` and `UNSUBSCRIBE_SECRET_KEY`, and+ `frontend/.env.default` one for `BETTER_AUTH_SECRET`. Those values are+ public. If your `.env` files still carry them, replace each with a freshly+ generated secret — the backend now refuses to start on the published+ `ENCRYPTION_KEY`. Note that rotating `ENCRYPTION_KEY` makes stored+ integration credentials unreadable, so reconnect those integrations+ afterwards.2. **Database location**: the database now lives in a plain Postgrescontainer (`pgvector/pgvector:pg15`) with its data in`autogpt_platform/data/db/data`. Your old data is untouched at@@ -279,9 +298,10 @@ A fresh install (empty database) needs none of this.### Additional Notes-You may want to change your encryption key in the `.env` file in the `autogpt_platform/backend` directory.--To generate a new encryption key, run the following command in python:+`make init-env` already generates a unique `ENCRYPTION_KEY` for your install, so+there is normally nothing to change here. To rotate it — for example if you+carried a key over from an older checkout, back when `.env.default` shipped a+working (and therefore public) one — generate a new key in python:```pythonfrom cryptography.fernet import Fernet;Fernet.generate_key().decode()@@ -293,7 +313,11 @@ Or run the following command in the `autogpt_platform/backend` directory:poetry run cli gen-encrypt-key```-Then, replace the existing key in the `autogpt_platform/backend/.env` file with the new one.+Then replace the value in `autogpt_platform/backend/.env`. **Rotating the key+makes previously stored integration credentials unreadable**, so you will need+to reconnect those integrations afterwards. The backend refuses to start on any+value that was once published in `.env.default`: those must be treated as+compromised.#### Auth transport security (JWKS over untrusted networks)