Multiplayer agent harness for work.
Let Porter name published apps, so a deploy needs no domain of its own
Josh Francef63a7e110c84
diff --git a/docs/porter.md b/docs/porter.mdindex 5dbbc3f5b..354d38dae 100644--- a/docs/porter.md+++ b/docs/porter.md@@ -98,16 +98,20 @@ principal with `AmazonEKSClusterAdminPolicy`, then `aws eks update-kubeconfig`.## Giving published apps stable hostnames-Apps the agent deploys are reachable only if the cluster gives them an address. With-`PORTER_DEPLOY_APPS_DOMAIN` unset, core warns at startup and every publish is refused up-front, before anything is created.+Published apps get their address from the cluster, not from qm. Declare the sandbox load+balancer with a root domain at cluster creation and **Porter names every published app+itself**: `<app>.<root domain>`, served on a wildcard Let's Encrypt certificate that Porter+also obtains. `PORTER_DEPLOY_APPS_DOMAIN` is then only for choosing a different name — leave+it unset and the cluster-assigned hostname is used as-is. Set it when the cluster serves+more than one apps domain, or when you want a name other than the cluster's own root domain;+it has to resolve to the same sandbox ingress either way.The address comes from Porter's **sandbox ingress**, which is separate from the ingressthat serves Porter apps and is configured per cluster. A cluster without it rejects everypublish before DNS is ever consulted, with `HTTP 400: validation error: visibility: noprivate sandbox ingress is configured on this cluster` (`no public sandbox ingress` when-`PORTER_DEPLOY_VISIBILITY=public`). Turn sandbox ingress on first; the wildcard record-below is what gives the resulting apps stable names.+`PORTER_DEPLOY_VISIBILITY=public`). A cluster that has it but no root domain names no host,+and the deploy fails after the body is created (the body is then retired, so nothing leaks).The dashboard has a toggle for it, but it is also a plain field on the cluster contract,so it can be turned on headlessly. `GET /api/projects/<project>/contracts` returns the@@ -132,6 +136,26 @@ true`. Route53 authenticates through the cluster's own pod identity and needs no/api/v2/projects/<project>/clusters/<cluster>/dns/credentials`. The same route creates acluster in the first place — omit `cluster.clusterId` and Porter provisions a new one.+Note where that array hangs: `additionalLoadBalancers` is a field of `cluster`, a sibling of+`eksKind`, while `sandboxesEnabled` is inside `eksKind`. Putting the load balancer in the+obvious-looking spot next to `sandboxesEnabled` does not fail — Porter parses the contract+with unknown fields discarded, so a misplaced or misspelled key is dropped in silence and+the POST still answers `201` with a revision id. The cluster then provisions with no sandbox+ingress and the mistake only surfaces 40 minutes later, when the first publish is refused.+**Always read the contract back after submitting it** and confirm the fields you set are+actually there:++```bash+curl -s -H "Authorization: Bearer $PORTER_DEPLOY_API_TOKEN" \+ https://dashboard.porter.run/api/projects/<project>/contracts |+ jq -r '.[] | select(.cluster_id==<cluster>) | .base64_contract' | tail -1 | base64 -d | jq .+```++The same silent-drop rule makes the contract useless for guessing at fields the schema may+not have: an invented key and a real one are equally accepted. To test whether a field+exists, send it together with a deliberate control key that certainly is not a field, then+read back and compare which of the two survived.+Put the sandbox load balancer in the contract you create the cluster with, and treat everylater revision as a one-at-a-time operation: **submit a revision only when the clusterreads `READY`, and let it finish.** A revision submitted while another is reconciling@@ -169,29 +193,47 @@ Once the revision reconciles, `GET/api/v2/projects/<project>/clusters/<cluster>/load-balancers` returns the new`owner: "sandbox"` entry whose `address.value` is what the wildcard record points at.-Routing and TLS arrive separately, and only the first is reliable. The load balancer and-the ingress rules come up within a couple of minutes and published apps answer on the-wildcard domain straight away — but the wildcard certificate is a `Certificate` in the-`porter-sandbox` namespace naming a `letsencrypt-sandbox-<visibility>-route53-prod`-ClusterIssuer, and on a cluster whose Porter IAM came from the standard CloudFormation-template that issuer is never created: the template grants no Route53 actions to any-`porter-*` role, and cert-manager gets neither an IAM role annotation nor an EKS pod-identity association, so the DNS-01 challenge has no credentials. Until that is sorted out-the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake Certificate_,-so every client — including the agent's own probe of the app it just published, which is-why it reports a gateway error it cannot explain — fails certificate verification against-an app that is otherwise serving correctly. Check with `kubectl get certificate -n-porter-sandbox` before believing a publish is broken.--Point a wildcard DNS record at the cluster's ingress load balancer and name it:+Routing, DNS and TLS all come up on their own, provided the sandbox load balancer was in+the contract the cluster was created with. Porter reads its `dnsProviderConfig` and+`rootDomains` and, through the cluster's Route53 pod identity, **creates a dedicated hosted+zone for the root domain, adds the `NS` delegation for it to the parent zone, and puts the+wildcard alias record inside it** pointing at the sandbox load balancer. cert-manager's+DNS-01 challenge then has a zone it can write to, and the wildcard certificate issues by+itself — a real Let's Encrypt `*.<root domain>` certificate, verified by ordinary clients+with no `-k`. You do not create the zone, the delegation, or the wildcard record by hand.++Porter also mints the credentials for it: a per-cluster role+`porter-cert-manager-route53-<cluster id>` whose inline policy allows+`route53:ChangeResourceRecordSets` and `ListResourceRecordSets` on that one delegated zone,+plus `GetChange` and `ListHostedZonesByName`. If TLS is not issuing, check that this role+exists — its absence, not a cert-manager misconfiguration, is the usual cause.++Two things follow. The parent zone must be in the same AWS account the cluster runs in, so+Porter can write the delegation. And **teardown has to remove what Porter created**: the+delegated hosted zone and the `NS` record in the parent zone both outlive the cluster.++This is worth stating plainly because the failure it replaces was so confusing: a cluster+whose sandbox load balancer was attached _after_ creation, or whose root domain was never+delegated, has no usable zone for the DNS-01 challenge, so the certificate never issues and+the ingress serves nginx's self-signed _Kubernetes Ingress Controller Fake Certificate_.+Every client then fails certificate verification against an app that is otherwise serving+correctly — including the agent's own probe of the app it just published, which is why it+reports a gateway error it cannot explain. `kubectl get certificate -n porter-sandbox` tells+you which situation you are in before you go hunting.++Naming the domain is optional. Porter assigns every sandbox that exposes a port a hostname+of its own, `<sandbox name>.<cluster root domain>`, whether or not the create request asked+for a domain — sending `networking: [{port: 8080}]` with no `domains` key, or with+`domains: []`, both come back with a populated `host`. So a cluster whose sandbox load+balancer carries a root domain publishes apps at a working HTTPS address with nothing set:```bash-PORTER_DEPLOY_APPS_DOMAIN=apps.example.com # *.apps.example.com -> cluster ingress+PORTER_DEPLOY_APPS_DOMAIN=apps.example.com # optional; overrides the assigned hostname```-Each deployment then publishes at `<name>.apps.example.com`. Deployments are **private**-by default, matching the other providers; `PORTER_DEPLOY_VISIBILITY=public` opts a-deployment's domain into public ingress.+Set it only to choose the name yourself; the wildcard record it relies on is the one Porter+already created. Deployments are **private** by default, matching the other providers;+`PORTER_DEPLOY_VISIBILITY=public` opts a deployment's domain into public ingress.## Forcing sandbox egress through the proxy@@ -208,9 +250,14 @@ available on this cluster` — so the agent ends up with no computer rather thanwith `{"sandbox_config":{"egress_enabled":true}}`, then `POST.../trigger-system-application-reconcile` with `{"dry_run":false,"system_application_name":"sandbox-api"}`. `GET` on the same path reads it back, and- turning it on installs Cilium, which enforces the allowlist. A project API token is- refused here with `PERMISSION_DENIED` even though it may create clusters — this call- needs a Porter account with admin rights on the project.+ turning it on installs Cilium, which enforces the allowlist. There is no contract field+ for any of this — the cluster contract carries no egress key at all, so it cannot be set+ at creation time and this PATCH is the only route. The call is gated on the token's role,+ not on being a human: a Developer-role API token is refused with `PERMISSION_DENIED` even+ though it may create clusters, while an **Admin-role API token** (Settings → API tokens)+ is accepted. Minting an admin token is enough; no dashboard session is required. The same+ distinction governs `DELETE /api/projects/<project>/clusters/<cluster>`, which answers a+ Developer token with `403 insufficient permissions to perform action`.- **The proxy has to be reachable from outside the cluster.** Porter attaches aNetworkPolicy to every sandbox that permits DNS plus `0.0.0.0/0` _except_ `10.0.0.0/8`,`172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16` and `100.64.0.0/10` — every RFC1918diff --git a/src/deploy/porter-deploy-provider.ts b/src/deploy/porter-deploy-provider.tsindex 86fd5d3e9..b2325be40 100644--- a/src/deploy/porter-deploy-provider.ts+++ b/src/deploy/porter-deploy-provider.ts@@ -154,11 +154,6 @@ export function createPorterDeployProvider(opts: PorterDeployProviderOptions): Dapply: (d, version) =>serialized(d, async () => {- if (!appsDomain) {- throw new Error(- `porter deploy ${d.id}: PORTER_DEPLOY_APPS_DOMAIN is not set, so the app would have no address — enable sandbox ingress on the cluster, point a wildcard DNS record at it, and set PORTER_DEPLOY_APPS_DOMAIN (docs/porter.md)`,- );- }resolveCache.delete(d.id);const { id: volumeId } = await ensurePorterVolume(client, volumeName(d));await retireBodies(d, true);@@ -186,7 +181,12 @@ export function createPorterDeployProvider(opts: PorterDeployProviderOptions): D});try {await waitPorterRunning(name, sb);- const host = (await sb.refresh()).host || domainOf(d)!;+ const host = (await sb.refresh()).host || domainOf(d);+ if (!host) {+ throw new Error(+ `porter deploy ${d.id}: the cluster named no host for the app — its sandbox ingress has no root domain, so set PORTER_DEPLOY_APPS_DOMAIN to a domain whose wildcard resolves to that ingress (docs/porter.md)`,+ );+ }await materialize(sb.id, version);await startApp(sb.id, version);await store.put(d.id, { deploymentId: d.id, sandboxId: sb.id, name, host, createdAtMs: Date.now() });diff --git a/test/porter-deploy-provider.test.ts b/test/porter-deploy-provider.test.tsindex d2542b337..a4697e93c 100644--- a/test/porter-deploy-provider.test.ts+++ b/test/porter-deploy-provider.test.ts@@ -249,7 +249,25 @@ test("a body that vanished between list and terminate does not break destroy orassert.match(await fetchText("/"), /^hello from/);});-test("without an apps domain apply refuses up front and creates nothing", async () => {+test("without an apps domain the cluster names the host itself", async () => {+ const bare = createPorterDeployProvider({+ namePrefix: "qmt",+ appPort,+ readyWindowSec: 10,+ client: fake.client,+ store,+ });+ const d = deployment("dep-8");+ const endpoint = await bare.apply(d, version({ "server.js": SERVER }, "node server.js"));+ assert.deepEqual(fake.bodies().find((b) => b.phase === "running")!.networking, [{ port: appPort }]);+ assert.match(endpoint.host, /^qmt-app-dep-8-[a-z0-9]{5}\.fake\.test$/);+ assert.equal(endpoint.publicUrl, `https://${endpoint.host}/`);+ assert.match(await fetchText("/"), /^hello from/);+});++test("a cluster that names no host fails the deploy and leaves no body behind", async () => {+ fake.cleanup();+ fake = installFakePorter({ assignHost: false });const bare = createPorterDeployProvider({namePrefix: "qmt",appPort,@@ -258,11 +276,10 @@ test("without an apps domain apply refuses up front and creates nothing", asyncstore,});await assert.rejects(- bare.apply(deployment("dep-8"), version({ "server.js": SERVER }, "node server.js")),- /PORTER_DEPLOY_APPS_DOMAIN is not set/,+ bare.apply(deployment("dep-8a"), version({ "server.js": SERVER }, "node server.js")),+ /named no host for the app/,);- assert.equal(fake.bodies().length, 0);- assert.equal(fake.volumeNames().length, 0);+ assert.equal(fake.bodies().filter((b) => b.phase === "running").length, 0);});test("a cluster-assigned hostname wins over the derived domain", async () => {diff --git a/test/support/fake-porter.ts b/test/support/fake-porter.tsindex f3cadfd3c..48ee0937c 100644--- a/test/support/fake-porter.ts+++ b/test/support/fake-porter.ts@@ -18,6 +18,7 @@ export interface FakePorterBody {export interface FakePorterOptions {terminateLag?: number;pageSize?: number;+ assignHost?: boolean;}export interface FakePorter {@@ -156,7 +157,8 @@ export function installFakePorter(opts: FakePorterOptions = {}): FakePorter {mkdirSync(home, { recursive: true });mkdirSync(app, { recursive: true });const exposed = spec.networking?.[0];- const host = exposed ? (exposed.domains?.[0]?.domain ?? `${spec.name}.fake.test`) : "";+ const named = opts.assignHost === false ? undefined : `${spec.name}.fake.test`;+ const host = exposed ? (exposed.domains?.[0]?.domain ?? named ?? "") : "";bodies.set(spec.name!, {id: `sb-${++bodySeq}`,phase: "running",