Multiplayer agent harness for work.
fix: field-test findings — honest core-outage reporting, docs corrections
Josh Francef65fc82f1812
diff --git a/cli/templates/deployment/references/porter.md b/cli/templates/deployment/references/porter.mdindex 1ffa8be17..85da40cb4 100644--- a/cli/templates/deployment/references/porter.md+++ b/cli/templates/deployment/references/porter.md@@ -8,8 +8,11 @@ as the reference for every failure mode named below.## PreflightInstall the [Porter CLI](https://docs.porter.run/cli/installation) and prefer-it over raw API calls wherever a command exists. Require Docker Buildx too —-every image must be built `--platform linux/amd64`:+it over raw API calls wherever a command exists. On Homebrew that means the+`porter-dev` tap — Homebrew core's `porter` is an unrelated tool — and+`docker-credential-porter` is a separate binary the tap does not install+(fetch it from Porter's releases and verify the checksum). Require Docker+Buildx too — every image must be built `--platform linux/amd64`:```bashporter auth login@@ -42,6 +45,17 @@ operator the exact URL and wait:contract (`docs/porter.md` → "Giving published apps stable hostnames").Attaching them after creation is the wedged-cluster path.+## Provision Postgres before first boot++Sign-in hard-requires a durable store: the auth broker's single-use claim+endpoint lives on core's database, so without `DATABASE_URL` the instance+boots and then refuses every sign-in. Porter has no `qm`-CLI database+provisioning — create a datastore in the dashboard's Datastores tab or run+in-cluster Postgres. An API-created datastore can report your cluster in+`connected_cluster_ids` while its security group still refuses connections;+verify with a `psql` from a pod before booting, and prefer in-cluster+Postgres when in doubt.+## Configure the administrator before first bootThe workflow's step 1 collected the administrator's email; on Porter nothing@@ -64,10 +78,12 @@ to the portal's `OIDC_ALLOWED_EMAILS`; on the raw `porter/apps/` path set both.for f in porter/apps/*.yaml; do porter apply -f "$f"; done```-Apply runs once per app file, twice overall — assigned hostnames land on the-second pass. Then hand-wire the six service URLs per the table in-`docs/porter.md` ("Hosting the qm surfaces"), push secrets as Porter env-groups, and verify by signing in as the administrator and opening the Admin-tab. Published-app serving needs nothing further: apps are reachable+Push images only to repositories that already exist — ECR does not create+them on push. In the v2 app YAML, `env` belongs at the app level; nested+under a service it is silently dropped. Apply runs once per app file, twice+overall — assigned hostnames land on the second pass. Then hand-wire the six+service URLs per the table in `docs/porter.md` ("Hosting the qm surfaces"),+push secrets as Porter env groups, and verify by signing in as the+administrator and opening the Admin tab. Published-app serving needs nothing further: apps are reachablesigned-in at `/d/<app>/`; setting `DEPLOY_APPS_DOMAIN` upgrades them toper-app subdomains (`docs/porter.md`, onboarding checklist).diff --git a/docs/porter.md b/docs/porter.mdindex 290081ac2..a1a443305 100644--- a/docs/porter.md+++ b/docs/porter.md@@ -21,7 +21,10 @@ Everything below this section is reference and field notes; this is the order thactually gets a new operator from zero to a working instance (the deployment workflow'sagent-facing version lives at `cli/templates/deployment/references/porter.md`). If you are an agent doingthis deployment: install and prefer the `porter` CLI over raw API calls wherever a-command exists ([install instructions](https://docs.porter.run/cli/installation)); use+command exists ([install instructions](https://docs.porter.run/cli/installation) — on+Homebrew that is the `porter-dev` tap, Homebrew core's `porter` is an unrelated tool, and+`docker-credential-porter` is a separate checksum-verified binary the tap does not+install); usethe operator's browser for the dashboard steps when you have browser access, andotherwise hand the operator the exact URL and wait — several steps below cannot be doneany other way.@@ -46,17 +49,33 @@ any other way.3. **Create the cluster with the sandbox load balancer in the creation contract** —sandbox ingress, `sandboxesEnabled`, and the apps root domain all belong in thecontract the cluster is born with; attaching them later is the wedged-cluster- forensics that "Giving published apps stable hostnames" below exists to debug.-4. **Name the administrator before first boot.** Set `ADMIN_GRANTS=<email>:org_admin`+ forensics that "Giving published apps stable hostnames" below exists to debug. The+ Route53 hosted zone for the root domain is yours to create either way (dedicated+ zone, `NS` delegation in the parent, wildcard record inside — then reconcile; see+ below), and so is the LoadBalancer in front of the egress proxy ("Forcing sandbox+ egress through the proxy" — without it egress runs fail-open).+4. **Provision Postgres and set `DATABASE_URL` before first boot.** Sign-in is not+ optional here: the auth broker's single-use claim endpoint needs core's durable+ store, so an instance without a database accepts no sign-ins at all. The Porter+ path has no `qm`-CLI database provisioning — create a datastore in the dashboard+ (Datastores tab) or run in-cluster Postgres. A datastore created through the API+ can come back with `connected_cluster_ids` naming your cluster while its security+ group still refuses it — verify with a `psql` from a pod before booting, and+ prefer in-cluster Postgres when in doubt.+5. **Name the administrator before first boot.** Set `ADMIN_GRANTS=<email>:org_admin`on core and the sign-in allowlist (`AUTH_ALLOWED_EMAILS=<email>` — the Helm chartbridges it to the portal's `OIDC_ALLOWED_EMAILS`). This is the step the old docsnever mentioned: with Postgres and no `ADMIN_GRANTS`, the instance boots, everyoneallowed can sign in, and the admin console is permanently unreachable — there is noin-product way to grant the first admin afterwards.-5. **Build, push, and apply**: images built `--platform linux/amd64`, then+6. **Build, push, and apply**: images built `--platform linux/amd64` and pushed to+ repositories that already exist (ECR does not create them on push — create or seed+ each repository first), then`for f in porter/apps/*.yaml; do porter apply -f "$f"; done` (twice — the hostname- lands on the second pass), then wire the service URLs per the table below. Verify by- signing in as the admin and opening the Admin tab.+ lands on the second pass), then wire the service URLs per the table below. In the+ v2 app YAML, `env` belongs at the app level: nested under a service it is silently+ dropped, and the app boots without its configuration. Verify by signing in as the+ admin and opening the Admin tab.## Hosting the qm surfaces@@ -252,12 +271,17 @@ Once the revision reconciles, `GETRouting, DNS and TLS all come up on their own, provided the sandbox load balancer was inthe contract the cluster was created with. Porter reads its `dnsProviderConfig` and-`rootDomains` and, through the cluster's Route53 pod identity, **creates a dedicated hosted-zone for the root domain, adds the `NS` delegation for it to the parent zone, and puts the-wildcard alias record inside it** pointing at the sandbox load balancer. cert-manager's-DNS-01 challenge then has a zone it can write to, and the wildcard certificate issues by-itself — a real Let's Encrypt `*.<root domain>` certificate, verified by ordinary clients-with no `-k`. You do not create the zone, the delegation, or the wildcard record by hand.+`rootDomains`, but **the hosted zone itself is yours to create** — confirmed against+Porter's team and a live cluster: even with the sandbox load balancer in the creation+contract, Porter never creates the zone, the delegation, or the wildcard record. Create a+dedicated Route53 hosted zone for the root domain, add its `NS` delegation to the parent+zone, put the wildcard record inside it pointing at the sandbox load balancer, then+reconcile the cluster (the dashboard's infra **Update**, or re-POST the contract) so Porter+creates the certificate issuer against the zone. cert-manager's DNS-01 challenge then has a+zone it can write to, and the wildcard certificate issues — a real Let's Encrypt+`*.<root domain>` certificate, verified by ordinary clients with no `-k`. A wildcard CNAME+sitting in the parent zone is not enough: DNS resolves, but there is no delegated zone for+the challenge and the certificate never issues.Porter also mints the credentials for it: a per-cluster role`porter-cert-manager-route53-<cluster id>` whose inline policy allows@@ -265,20 +289,15 @@ Porter also mints the credentials for it: a per-cluster roleplus `GetChange` and `ListHostedZonesByName`. If TLS is not issuing, check that this roleexists — its absence, not a cert-manager misconfiguration, is the usual cause.-Two things follow. The parent zone must be in the same AWS account the cluster runs in, so-Porter can write the delegation. And **teardown has to remove what Porter created**: the-delegated hosted zone and the `NS` record in the parent zone both outlive the cluster.--This is worth stating plainly because the failure it replaces was so confusing: a cluster-whose sandbox load balancer was attached _after_ creation, or whose root domain was never-delegated, has no usable zone for the DNS-01 challenge, so the certificate never issues and-the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake Certificate_.-The remediation (confirmed with Porter support) is manual: create the dedicated hosted zone-for the root domain yourself, add its `NS` delegation to the parent zone, move the wildcard-record into it, then reconcile the cluster (the dashboard's infra **Update**, or re-POST the-contract) so Porter actually creates the certificate issuer against the new zone.-Every client then fails certificate verification against an app that is otherwise serving-correctly — including the agent's own probe of the app it just published, which is why it+Two things follow. The parent zone must be in the same AWS account the cluster runs in.+And **teardown has to remove what you created**: the delegated hosted zone and the `NS`+record in the parent zone both outlive the cluster.++This is worth stating plainly because the failure is so confusing: a cluster whose root+domain was never delegated has no usable zone for the DNS-01 challenge, so the certificate+never issues and the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake+Certificate_. Every client then fails certificate verification against an app that is+otherwise serving correctly — including the agent's own probe of the app it just published, which is why itreports a gateway error it cannot explain. `kubectl get certificate -n porter-sandbox` tellsyou which situation you are in before you go hunting.diff --git a/plugins/auth/src/server.ts b/plugins/auth/src/server.tsindex ff4d90693..e1e24b086 100644--- a/plugins/auth/src/server.ts+++ b/plugins/auth/src/server.ts@@ -3,7 +3,7 @@ import { readBody, PayloadTooLargeError, serveEmojiFavicon } from "../../chassisimport { errMessage } from "../../chassis/src/errors.ts";import type { AuthConfig } from "./config.ts";import { validEmail } from "./config.ts";-import { claimOnce, withinRateLimit, type ClaimStore } from "../../chassis/src/claims.ts";+import { claimOnce, withinRateLimit, ClaimStoreUnavailableError, type ClaimStore } from "../../chassis/src/claims.ts";import { mintIdToken, pkceMatches, safeEqual, subjectFor, TokenSigner, type AuthRequest } from "./tokens.ts";import { ID_TOKEN_ALG, type SigningKey } from "./keys.ts";import { renderSignInEmail, type Mailer } from "./email.ts";@@ -171,12 +171,20 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: Sconsole.warn(`[auth] sign-in link suppressed: ${email} is not on the permitted list`);return;}- if (!(await within("ip", ip, cfg.sendLimitPerIp))) {- console.warn("[auth] sign-in link suppressed: per-address rate limit reached for the requesting client");- return;- }- if (!(await within("mailbox", email, cfg.sendLimitPerEmail))) {- console.warn("[auth] sign-in link suppressed: per-mailbox rate limit reached");+ try {+ if (!(await within("ip", ip, cfg.sendLimitPerIp))) {+ console.warn("[auth] sign-in link suppressed: per-address rate limit reached for the requesting client");+ return;+ }+ if (!(await within("mailbox", email, cfg.sendLimitPerEmail))) {+ console.warn("[auth] sign-in link suppressed: per-mailbox rate limit reached");+ return;+ }+ } catch (e) {+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;+ console.error(+ "[auth] sign-in link suppressed: core is unreachable, so rate limits cannot be enforced — sign-in fails closed until core is healthy (this is a core outage, not a rate limit)",+ );return;}const sealed = await signer.sealLink({ ...request, email }, cfg.linkTtlS, nowMs);@@ -247,7 +255,19 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S}const opened = await signer.openLink(new URLSearchParams(raw).get("token") ?? "", now());if (!opened) return staleLink(res);- if (!(await claimOnce(claims, `link:${opened.jti}`, opened.expiresAtMs))) return staleLink(res);+ let linkClaimed: boolean;+ try {+ linkClaimed = await claimOnce(claims, `link:${opened.jti}`, opened.expiresAtMs);+ } catch (e) {+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;+ return problem(+ res,+ 503,+ "Sign-in is temporarily unavailable",+ "The sign-in service cannot reach its backend. Try again in a minute — this link stays valid.",+ );+ }+ if (!linkClaimed) return staleLink(res);const { claims: link } = opened;if (!safeEqual(link.clientId, cfg.clientId) || !safeEqual(link.redirectUri, cfg.redirectUri)) {return problem(@@ -305,8 +325,14 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S) {return sendJson(res, 400, { error: "invalid_grant" });}- if (!(await claimOnce(claims, `code:${opened.jti}`, opened.expiresAtMs)))- return sendJson(res, 400, { error: "invalid_grant" });+ let codeClaimed: boolean;+ try {+ codeClaimed = await claimOnce(claims, `code:${opened.jti}`, opened.expiresAtMs);+ } catch (e) {+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;+ return sendJson(res, 503, { error: "temporarily_unavailable" });+ }+ if (!codeClaimed) return sendJson(res, 400, { error: "invalid_grant" });if (!pkceMatches(form.get("code_verifier") ?? "", granted.codeChallenge))return sendJson(res, 400, { error: "invalid_grant" });if (!emailAllowed(cfg, granted.email)) return sendJson(res, 400, { error: "invalid_grant" });@@ -373,6 +399,15 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: Sconst path = url.pathname;if (method === "GET" && path === "/healthz") return sendJson(res, 200, { ok: true });+ if (method === "GET" && path === "/readyz") {+ try {+ const r = await fetch(`${cfg.coreApiUrl}/healthz`, { signal: AbortSignal.timeout(2_000) });+ if (r.ok) return sendJson(res, 200, { ok: true });+ return sendJson(res, 503, { ok: false, core: `HTTP ${r.status}` });+ } catch (e) {+ return sendJson(res, 503, { ok: false, core: errMessage(e) });+ }+ }if (method === "GET" && (path === "/favicon.ico" || path === "/favicon.svg")) {return serveEmojiFavicon(res, "✉️", "max-age=86400");}diff --git a/plugins/auth/test/flow.test.ts b/plugins/auth/test/flow.test.tsindex 5ed36fa80..40e18d427 100644--- a/plugins/auth/test/flow.test.ts+++ b/plugins/auth/test/flow.test.ts@@ -1,6 +1,7 @@import test from "node:test";import assert from "node:assert/strict";import { createHash } from "node:crypto";+import { ClaimStoreUnavailableError } from "../../chassis/src/claims.ts";import { createLocalJWKSet, decodeProtectedHeader, jwtVerify, type JWK } from "jose";import {authorizeQuery,@@ -382,6 +383,32 @@ test("the broker fails closed when core cannot record a single-use claim", asyncassert.notEqual(response.status, 302, "an unrecordable link claim must not mint a code");});+test("a core outage reads as an outage, never as a rate limit or a stale link", async (t) => {+ const unavailable = {+ calls: [] as string[][],+ async claimFirst(): Promise<string | null> {+ throw new ClaimStoreUnavailableError("core claim store unreachable: fetch failed");+ },+ };+ const h = await startHarness({ claims: unavailable });+ t.after(() => h.close());+ await requestLink(h);+ assert.equal(h.mailer.sent.length, 0, "sign-in fails closed while core is down");++ const healthy = await startHarness();+ t.after(() => healthy.close());+ await requestLink(healthy);+ const link = linkFrom(healthy.mailer);+ const downMidVerify = await startHarness({+ claims: unavailable,+ env: { AUTH_SIGNING_JWK: healthy.cfg.signingJwk ? JSON.stringify(healthy.cfg.signingJwk) : undefined },+ });+ t.after(() => downMidVerify.close());+ const response = await openLink(downMidVerify, link);+ assert.equal(response.status, 503, "an outage is a retryable 503, not the dead-end stale-link page");+ assert.match(await response.text(), /temporarily unavailable/i);+});+test("the sign-in link is single-use across broker instances that share the claim store", async (t) => {const claims = memoryClaimStore();const first = await startHarness({ claims });diff --git a/plugins/chassis/src/claims.ts b/plugins/chassis/src/claims.tsindex 667baf62f..22625037b 100644--- a/plugins/chassis/src/claims.ts+++ b/plugins/chassis/src/claims.ts@@ -9,6 +9,8 @@ export interface ClaimStore {const CLAIM_PATH = "/v1/auth/broker/claim";const CLAIM_TIMEOUT_MS = 4_000;+export class ClaimStoreUnavailableError extends Error {}+export function coreClaimStore(coreApiUrl: string, signingSecret: string | undefined, label = "chassis"): ClaimStore {return {async claimFirst(ids, expiresAtMs) {@@ -21,6 +23,9 @@ export function coreClaimStore(coreApiUrl: string, signingSecret: string | undefbody,signal: AbortSignal.timeout(CLAIM_TIMEOUT_MS),});+ if (r.status >= 500) {+ throw new ClaimStoreUnavailableError(`core answered the claim endpoint with HTTP ${r.status}`);+ }if (!r.ok) {console.error(`[${label}] core refused a single-use claim: HTTP ${r.status}`);return null;@@ -28,8 +33,13 @@ export function coreClaimStore(coreApiUrl: string, signingSecret: string | undefconst parsed = (await r.json()) as { claimed?: unknown };return typeof parsed.claimed === "string" ? parsed.claimed : null;} catch (e) {- console.error(`[${label}] core single-use claim failed: ${errMessage(e)}`);- return null;+ if (e instanceof ClaimStoreUnavailableError) {+ console.error(`[${label}] ${e.message} — failing closed, this is a core outage, not a rate limit`);+ throw e;+ }+ const unavailable = new ClaimStoreUnavailableError(`core claim store unreachable: ${errMessage(e)}`);+ console.error(`[${label}] ${unavailable.message} — failing closed, this is a core outage, not a rate limit`);+ throw unavailable;}},};diff --git a/plugins/portal/src/index.ts b/plugins/portal/src/index.tsindex 58a31fa44..66f338b68 100644--- a/plugins/portal/src/index.ts+++ b/plugins/portal/src/index.ts@@ -38,7 +38,7 @@ import {FORWARD_WEBHOOK_HEADERS,} from "./proxy.ts";import { signedHeaders, withSourceAuthNonce } from "../../chassis/src/core-client.ts";-import { coreClaimStore, withinRateLimit } from "../../chassis/src/claims.ts";+import { coreClaimStore, withinRateLimit, ClaimStoreUnavailableError } from "../../chassis/src/claims.ts";import { mintPortalIdentity, PORTAL_IDENTITY_HEADER } from "../../chassis/src/portal-identity.ts";import { errMessage } from "../../chassis/src/errors.ts";import { json, escapeHtml, serveEmojiFavicon } from "../../chassis/src/http.ts";@@ -813,6 +813,9 @@ async function mintPlaygroundSession(req: IncomingMessage, res: ServerResponse):limit: PLAYGROUND_MINTS_PER_IP,windowS: PLAYGROUND_MINT_WINDOW_S,nowMs: Date.now(),+ }).catch((e) => {+ if (e instanceof ClaimStoreUnavailableError) return false;+ throw e;});if (!allowed) return null;const now = Math.floor(Date.now() / 1000);diff --git a/porter/apps/egress-proxy.yaml b/porter/apps/egress-proxy.yamlindex 106be2f5b..08b1031d8 100644--- a/porter/apps/egress-proxy.yaml+++ b/porter/apps/egress-proxy.yaml@@ -4,6 +4,12 @@ build:method: dockercontext: .dockerfile: ./deploy/egress-proxy/Dockerfile+# Deliberately a worker, not a web service: sandboxes cannot reach in-cluster or+# private addresses (Porter's sandbox NetworkPolicy blocks every RFC1918 range),+# and Porter's web ingress cannot carry HTTP CONNECT proxy traffic. Expose port+# 48080 with a LoadBalancer Service in front of this deployment and point+# PORTER_SANDBOX_EGRESS_PROXY_URL at it, or sandbox egress runs fail-open —+# docs/porter.md, "Forcing sandbox egress through the proxy".services:- name: egress-proxytype: worker