Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

fix: field-test findings — honest core-outage reporting, docs corrections

Josh Francef65fc82f1812
diff --git a/cli/templates/deployment/references/porter.md b/cli/templates/deployment/references/porter.md
index 1ffa8be17..85da40cb4 100644
--- a/cli/templates/deployment/references/porter.md
+++ b/cli/templates/deployment/references/porter.md
@@ -8,8 +8,11 @@ as the reference for every failure mode named below.
## Preflight
Install the [Porter CLI](https://docs.porter.run/cli/installation) and prefer
-it over raw API calls wherever a command exists. Require Docker Buildx too —
-every image must be built `--platform linux/amd64`:
+it over raw API calls wherever a command exists. On Homebrew that means the
+`porter-dev` tap — Homebrew core's `porter` is an unrelated tool — and
+`docker-credential-porter` is a separate binary the tap does not install
+(fetch it from Porter's releases and verify the checksum). Require Docker
+Buildx too — every image must be built `--platform linux/amd64`:
```bash
porter auth login
@@ -42,6 +45,17 @@ operator the exact URL and wait:
contract (`docs/porter.md` → "Giving published apps stable hostnames").
Attaching them after creation is the wedged-cluster path.
+## Provision Postgres before first boot
+
+Sign-in hard-requires a durable store: the auth broker's single-use claim
+endpoint lives on core's database, so without `DATABASE_URL` the instance
+boots and then refuses every sign-in. Porter has no `qm`-CLI database
+provisioning — create a datastore in the dashboard's Datastores tab or run
+in-cluster Postgres. An API-created datastore can report your cluster in
+`connected_cluster_ids` while its security group still refuses connections;
+verify with a `psql` from a pod before booting, and prefer in-cluster
+Postgres when in doubt.
+
## Configure the administrator before first boot
The workflow's step 1 collected the administrator's email; on Porter nothing
@@ -64,10 +78,12 @@ to the portal's `OIDC_ALLOWED_EMAILS`; on the raw `porter/apps/` path set both.
for f in porter/apps/*.yaml; do porter apply -f "$f"; done
```
-Apply runs once per app file, twice overall — assigned hostnames land on the
-second pass. Then hand-wire the six service URLs per the table in
-`docs/porter.md` ("Hosting the qm surfaces"), push secrets as Porter env
-groups, and verify by signing in as the administrator and opening the Admin
-tab. Published-app serving needs nothing further: apps are reachable
+Push images only to repositories that already exist — ECR does not create
+them on push. In the v2 app YAML, `env` belongs at the app level; nested
+under a service it is silently dropped. Apply runs once per app file, twice
+overall — assigned hostnames land on the second pass. Then hand-wire the six
+service URLs per the table in `docs/porter.md` ("Hosting the qm surfaces"),
+push secrets as Porter env groups, and verify by signing in as the
+administrator and opening the Admin tab. Published-app serving needs nothing further: apps are reachable
signed-in at `/d/<app>/`; setting `DEPLOY_APPS_DOMAIN` upgrades them to
per-app subdomains (`docs/porter.md`, onboarding checklist).
diff --git a/docs/porter.md b/docs/porter.md
index 290081ac2..a1a443305 100644
--- a/docs/porter.md
+++ b/docs/porter.md
@@ -21,7 +21,10 @@ Everything below this section is reference and field notes; this is the order th
actually gets a new operator from zero to a working instance (the deployment workflow's
agent-facing version lives at `cli/templates/deployment/references/porter.md`). If you are an agent doing
this deployment: install and prefer the `porter` CLI over raw API calls wherever a
-command exists ([install instructions](https://docs.porter.run/cli/installation)); use
+command exists ([install instructions](https://docs.porter.run/cli/installation) — on
+Homebrew that is the `porter-dev` tap, Homebrew core's `porter` is an unrelated tool, and
+`docker-credential-porter` is a separate checksum-verified binary the tap does not
+install); use
the operator's browser for the dashboard steps when you have browser access, and
otherwise hand the operator the exact URL and wait — several steps below cannot be done
any other way.
@@ -46,17 +49,33 @@ any other way.
3. **Create the cluster with the sandbox load balancer in the creation contract** —
sandbox ingress, `sandboxesEnabled`, and the apps root domain all belong in the
contract the cluster is born with; attaching them later is the wedged-cluster
- forensics that "Giving published apps stable hostnames" below exists to debug.
-4. **Name the administrator before first boot.** Set `ADMIN_GRANTS=<email>:org_admin`
+ forensics that "Giving published apps stable hostnames" below exists to debug. The
+ Route53 hosted zone for the root domain is yours to create either way (dedicated
+ zone, `NS` delegation in the parent, wildcard record inside — then reconcile; see
+ below), and so is the LoadBalancer in front of the egress proxy ("Forcing sandbox
+ egress through the proxy" — without it egress runs fail-open).
+4. **Provision Postgres and set `DATABASE_URL` before first boot.** Sign-in is not
+ optional here: the auth broker's single-use claim endpoint needs core's durable
+ store, so an instance without a database accepts no sign-ins at all. The Porter
+ path has no `qm`-CLI database provisioning — create a datastore in the dashboard
+ (Datastores tab) or run in-cluster Postgres. A datastore created through the API
+ can come back with `connected_cluster_ids` naming your cluster while its security
+ group still refuses it — verify with a `psql` from a pod before booting, and
+ prefer in-cluster Postgres when in doubt.
+5. **Name the administrator before first boot.** Set `ADMIN_GRANTS=<email>:org_admin`
on core and the sign-in allowlist (`AUTH_ALLOWED_EMAILS=<email>` — the Helm chart
bridges it to the portal's `OIDC_ALLOWED_EMAILS`). This is the step the old docs
never mentioned: with Postgres and no `ADMIN_GRANTS`, the instance boots, everyone
allowed can sign in, and the admin console is permanently unreachable — there is no
in-product way to grant the first admin afterwards.
-5. **Build, push, and apply**: images built `--platform linux/amd64`, then
+6. **Build, push, and apply**: images built `--platform linux/amd64` and pushed to
+ repositories that already exist (ECR does not create them on push — create or seed
+ each repository first), then
`for f in porter/apps/*.yaml; do porter apply -f "$f"; done` (twice — the hostname
- lands on the second pass), then wire the service URLs per the table below. Verify by
- signing in as the admin and opening the Admin tab.
+ lands on the second pass), then wire the service URLs per the table below. In the
+ v2 app YAML, `env` belongs at the app level: nested under a service it is silently
+ dropped, and the app boots without its configuration. Verify by signing in as the
+ admin and opening the Admin tab.
## Hosting the qm surfaces
@@ -252,12 +271,17 @@ Once the revision reconciles, `GET
Routing, DNS and TLS all come up on their own, provided the sandbox load balancer was in
the contract the cluster was created with. Porter reads its `dnsProviderConfig` and
-`rootDomains` and, through the cluster's Route53 pod identity, **creates a dedicated hosted
-zone for the root domain, adds the `NS` delegation for it to the parent zone, and puts the
-wildcard alias record inside it** pointing at the sandbox load balancer. cert-manager's
-DNS-01 challenge then has a zone it can write to, and the wildcard certificate issues by
-itself — a real Let's Encrypt `*.<root domain>` certificate, verified by ordinary clients
-with no `-k`. You do not create the zone, the delegation, or the wildcard record by hand.
+`rootDomains`, but **the hosted zone itself is yours to create** — confirmed against
+Porter's team and a live cluster: even with the sandbox load balancer in the creation
+contract, Porter never creates the zone, the delegation, or the wildcard record. Create a
+dedicated Route53 hosted zone for the root domain, add its `NS` delegation to the parent
+zone, put the wildcard record inside it pointing at the sandbox load balancer, then
+reconcile the cluster (the dashboard's infra **Update**, or re-POST the contract) so Porter
+creates the certificate issuer against the zone. cert-manager's DNS-01 challenge then has a
+zone it can write to, and the wildcard certificate issues — a real Let's Encrypt
+`*.<root domain>` certificate, verified by ordinary clients with no `-k`. A wildcard CNAME
+sitting in the parent zone is not enough: DNS resolves, but there is no delegated zone for
+the challenge and the certificate never issues.
Porter also mints the credentials for it: a per-cluster role
`porter-cert-manager-route53-<cluster id>` whose inline policy allows
@@ -265,20 +289,15 @@ Porter also mints the credentials for it: a per-cluster role
plus `GetChange` and `ListHostedZonesByName`. If TLS is not issuing, check that this role
exists — its absence, not a cert-manager misconfiguration, is the usual cause.
-Two things follow. The parent zone must be in the same AWS account the cluster runs in, so
-Porter can write the delegation. And **teardown has to remove what Porter created**: the
-delegated hosted zone and the `NS` record in the parent zone both outlive the cluster.
-
-This is worth stating plainly because the failure it replaces was so confusing: a cluster
-whose sandbox load balancer was attached _after_ creation, or whose root domain was never
-delegated, has no usable zone for the DNS-01 challenge, so the certificate never issues and
-the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake Certificate_.
-The remediation (confirmed with Porter support) is manual: create the dedicated hosted zone
-for the root domain yourself, add its `NS` delegation to the parent zone, move the wildcard
-record into it, then reconcile the cluster (the dashboard's infra **Update**, or re-POST the
-contract) so Porter actually creates the certificate issuer against the new zone.
-Every client then fails certificate verification against an app that is otherwise serving
-correctly — including the agent's own probe of the app it just published, which is why it
+Two things follow. The parent zone must be in the same AWS account the cluster runs in.
+And **teardown has to remove what you created**: the delegated hosted zone and the `NS`
+record in the parent zone both outlive the cluster.
+
+This is worth stating plainly because the failure is so confusing: a cluster whose root
+domain was never delegated has no usable zone for the DNS-01 challenge, so the certificate
+never issues and the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake
+Certificate_. Every client then fails certificate verification against an app that is
+otherwise serving correctly — including the agent's own probe of the app it just published, which is why it
reports a gateway error it cannot explain. `kubectl get certificate -n porter-sandbox` tells
you which situation you are in before you go hunting.
diff --git a/plugins/auth/src/server.ts b/plugins/auth/src/server.ts
index ff4d90693..e1e24b086 100644
--- a/plugins/auth/src/server.ts
+++ b/plugins/auth/src/server.ts
@@ -3,7 +3,7 @@ import { readBody, PayloadTooLargeError, serveEmojiFavicon } from "../../chassis
import { errMessage } from "../../chassis/src/errors.ts";
import type { AuthConfig } from "./config.ts";
import { validEmail } from "./config.ts";
-import { claimOnce, withinRateLimit, type ClaimStore } from "../../chassis/src/claims.ts";
+import { claimOnce, withinRateLimit, ClaimStoreUnavailableError, type ClaimStore } from "../../chassis/src/claims.ts";
import { mintIdToken, pkceMatches, safeEqual, subjectFor, TokenSigner, type AuthRequest } from "./tokens.ts";
import { ID_TOKEN_ALG, type SigningKey } from "./keys.ts";
import { renderSignInEmail, type Mailer } from "./email.ts";
@@ -171,12 +171,20 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S
console.warn(`[auth] sign-in link suppressed: ${email} is not on the permitted list`);
return;
}
- if (!(await within("ip", ip, cfg.sendLimitPerIp))) {
- console.warn("[auth] sign-in link suppressed: per-address rate limit reached for the requesting client");
- return;
- }
- if (!(await within("mailbox", email, cfg.sendLimitPerEmail))) {
- console.warn("[auth] sign-in link suppressed: per-mailbox rate limit reached");
+ try {
+ if (!(await within("ip", ip, cfg.sendLimitPerIp))) {
+ console.warn("[auth] sign-in link suppressed: per-address rate limit reached for the requesting client");
+ return;
+ }
+ if (!(await within("mailbox", email, cfg.sendLimitPerEmail))) {
+ console.warn("[auth] sign-in link suppressed: per-mailbox rate limit reached");
+ return;
+ }
+ } catch (e) {
+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;
+ console.error(
+ "[auth] sign-in link suppressed: core is unreachable, so rate limits cannot be enforced — sign-in fails closed until core is healthy (this is a core outage, not a rate limit)",
+ );
return;
}
const sealed = await signer.sealLink({ ...request, email }, cfg.linkTtlS, nowMs);
@@ -247,7 +255,19 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S
}
const opened = await signer.openLink(new URLSearchParams(raw).get("token") ?? "", now());
if (!opened) return staleLink(res);
- if (!(await claimOnce(claims, `link:${opened.jti}`, opened.expiresAtMs))) return staleLink(res);
+ let linkClaimed: boolean;
+ try {
+ linkClaimed = await claimOnce(claims, `link:${opened.jti}`, opened.expiresAtMs);
+ } catch (e) {
+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;
+ return problem(
+ res,
+ 503,
+ "Sign-in is temporarily unavailable",
+ "The sign-in service cannot reach its backend. Try again in a minute — this link stays valid.",
+ );
+ }
+ if (!linkClaimed) return staleLink(res);
const { claims: link } = opened;
if (!safeEqual(link.clientId, cfg.clientId) || !safeEqual(link.redirectUri, cfg.redirectUri)) {
return problem(
@@ -305,8 +325,14 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S
) {
return sendJson(res, 400, { error: "invalid_grant" });
}
- if (!(await claimOnce(claims, `code:${opened.jti}`, opened.expiresAtMs)))
- return sendJson(res, 400, { error: "invalid_grant" });
+ let codeClaimed: boolean;
+ try {
+ codeClaimed = await claimOnce(claims, `code:${opened.jti}`, opened.expiresAtMs);
+ } catch (e) {
+ if (!(e instanceof ClaimStoreUnavailableError)) throw e;
+ return sendJson(res, 503, { error: "temporarily_unavailable" });
+ }
+ if (!codeClaimed) return sendJson(res, 400, { error: "invalid_grant" });
if (!pkceMatches(form.get("code_verifier") ?? "", granted.codeChallenge))
return sendJson(res, 400, { error: "invalid_grant" });
if (!emailAllowed(cfg, granted.email)) return sendJson(res, 400, { error: "invalid_grant" });
@@ -373,6 +399,15 @@ export function createAuthHandler(deps: AuthDeps): (req: IncomingMessage, res: S
const path = url.pathname;
if (method === "GET" && path === "/healthz") return sendJson(res, 200, { ok: true });
+ if (method === "GET" && path === "/readyz") {
+ try {
+ const r = await fetch(`${cfg.coreApiUrl}/healthz`, { signal: AbortSignal.timeout(2_000) });
+ if (r.ok) return sendJson(res, 200, { ok: true });
+ return sendJson(res, 503, { ok: false, core: `HTTP ${r.status}` });
+ } catch (e) {
+ return sendJson(res, 503, { ok: false, core: errMessage(e) });
+ }
+ }
if (method === "GET" && (path === "/favicon.ico" || path === "/favicon.svg")) {
return serveEmojiFavicon(res, "✉️", "max-age=86400");
}
diff --git a/plugins/auth/test/flow.test.ts b/plugins/auth/test/flow.test.ts
index 5ed36fa80..40e18d427 100644
--- a/plugins/auth/test/flow.test.ts
+++ b/plugins/auth/test/flow.test.ts
@@ -1,6 +1,7 @@
import test from "node:test";
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
+import { ClaimStoreUnavailableError } from "../../chassis/src/claims.ts";
import { createLocalJWKSet, decodeProtectedHeader, jwtVerify, type JWK } from "jose";
import {
authorizeQuery,
@@ -382,6 +383,32 @@ test("the broker fails closed when core cannot record a single-use claim", async
assert.notEqual(response.status, 302, "an unrecordable link claim must not mint a code");
});
+test("a core outage reads as an outage, never as a rate limit or a stale link", async (t) => {
+ const unavailable = {
+ calls: [] as string[][],
+ async claimFirst(): Promise<string | null> {
+ throw new ClaimStoreUnavailableError("core claim store unreachable: fetch failed");
+ },
+ };
+ const h = await startHarness({ claims: unavailable });
+ t.after(() => h.close());
+ await requestLink(h);
+ assert.equal(h.mailer.sent.length, 0, "sign-in fails closed while core is down");
+
+ const healthy = await startHarness();
+ t.after(() => healthy.close());
+ await requestLink(healthy);
+ const link = linkFrom(healthy.mailer);
+ const downMidVerify = await startHarness({
+ claims: unavailable,
+ env: { AUTH_SIGNING_JWK: healthy.cfg.signingJwk ? JSON.stringify(healthy.cfg.signingJwk) : undefined },
+ });
+ t.after(() => downMidVerify.close());
+ const response = await openLink(downMidVerify, link);
+ assert.equal(response.status, 503, "an outage is a retryable 503, not the dead-end stale-link page");
+ assert.match(await response.text(), /temporarily unavailable/i);
+});
+
test("the sign-in link is single-use across broker instances that share the claim store", async (t) => {
const claims = memoryClaimStore();
const first = await startHarness({ claims });
diff --git a/plugins/chassis/src/claims.ts b/plugins/chassis/src/claims.ts
index 667baf62f..22625037b 100644
--- a/plugins/chassis/src/claims.ts
+++ b/plugins/chassis/src/claims.ts
@@ -9,6 +9,8 @@ export interface ClaimStore {
const CLAIM_PATH = "/v1/auth/broker/claim";
const CLAIM_TIMEOUT_MS = 4_000;
+export class ClaimStoreUnavailableError extends Error {}
+
export function coreClaimStore(coreApiUrl: string, signingSecret: string | undefined, label = "chassis"): ClaimStore {
return {
async claimFirst(ids, expiresAtMs) {
@@ -21,6 +23,9 @@ export function coreClaimStore(coreApiUrl: string, signingSecret: string | undef
body,
signal: AbortSignal.timeout(CLAIM_TIMEOUT_MS),
});
+ if (r.status >= 500) {
+ throw new ClaimStoreUnavailableError(`core answered the claim endpoint with HTTP ${r.status}`);
+ }
if (!r.ok) {
console.error(`[${label}] core refused a single-use claim: HTTP ${r.status}`);
return null;
@@ -28,8 +33,13 @@ export function coreClaimStore(coreApiUrl: string, signingSecret: string | undef
const parsed = (await r.json()) as { claimed?: unknown };
return typeof parsed.claimed === "string" ? parsed.claimed : null;
} catch (e) {
- console.error(`[${label}] core single-use claim failed: ${errMessage(e)}`);
- return null;
+ if (e instanceof ClaimStoreUnavailableError) {
+ console.error(`[${label}] ${e.message} — failing closed, this is a core outage, not a rate limit`);
+ throw e;
+ }
+ const unavailable = new ClaimStoreUnavailableError(`core claim store unreachable: ${errMessage(e)}`);
+ console.error(`[${label}] ${unavailable.message} — failing closed, this is a core outage, not a rate limit`);
+ throw unavailable;
}
},
};
diff --git a/plugins/portal/src/index.ts b/plugins/portal/src/index.ts
index 58a31fa44..66f338b68 100644
--- a/plugins/portal/src/index.ts
+++ b/plugins/portal/src/index.ts
@@ -38,7 +38,7 @@ import {
FORWARD_WEBHOOK_HEADERS,
} from "./proxy.ts";
import { signedHeaders, withSourceAuthNonce } from "../../chassis/src/core-client.ts";
-import { coreClaimStore, withinRateLimit } from "../../chassis/src/claims.ts";
+import { coreClaimStore, withinRateLimit, ClaimStoreUnavailableError } from "../../chassis/src/claims.ts";
import { mintPortalIdentity, PORTAL_IDENTITY_HEADER } from "../../chassis/src/portal-identity.ts";
import { errMessage } from "../../chassis/src/errors.ts";
import { json, escapeHtml, serveEmojiFavicon } from "../../chassis/src/http.ts";
@@ -813,6 +813,9 @@ async function mintPlaygroundSession(req: IncomingMessage, res: ServerResponse):
limit: PLAYGROUND_MINTS_PER_IP,
windowS: PLAYGROUND_MINT_WINDOW_S,
nowMs: Date.now(),
+ }).catch((e) => {
+ if (e instanceof ClaimStoreUnavailableError) return false;
+ throw e;
});
if (!allowed) return null;
const now = Math.floor(Date.now() / 1000);
diff --git a/porter/apps/egress-proxy.yaml b/porter/apps/egress-proxy.yaml
index 106be2f5b..08b1031d8 100644
--- a/porter/apps/egress-proxy.yaml
+++ b/porter/apps/egress-proxy.yaml
@@ -4,6 +4,12 @@ build:
method: docker
context: .
dockerfile: ./deploy/egress-proxy/Dockerfile
+# Deliberately a worker, not a web service: sandboxes cannot reach in-cluster or
+# private addresses (Porter's sandbox NetworkPolicy blocks every RFC1918 range),
+# and Porter's web ingress cannot carry HTTP CONNECT proxy traffic. Expose port
+# 48080 with a LoadBalancer Service in front of this deployment and point
+# PORTER_SANDBOX_EGRESS_PROXY_URL at it, or sandbox egress runs fail-open —
+# docs/porter.md, "Forcing sandbox egress through the proxy".
services:
- name: egress-proxy
type: worker