Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Match AWS first-party workload defaults to published image architecture (#1807)

Sina Matian98c2fa46469d
diff --git a/cli/README.md b/cli/README.md
index 8d1b37191..941419b7f 100644
--- a/cli/README.md
+++ b/cli/README.md
@@ -30,7 +30,15 @@ path locally.
The CLI deploys long-running QM services; it is not the runtime. Docker runs
them locally, Fly runs them as Fly apps with Fly Machines for agent computers, and AWS
-runs digest-pinned ARM64 tasks on ECS Fargate with Lambda MicroVM agent computers.
+runs digest-pinned tasks on ECS Fargate (amd64 by default for first-party services) with Lambda MicroVM agent computers.
+
+When upgrading an AWS source deployment that previously relied on the ARM64 default,
+set each existing workload's `architecture: "arm64"` explicitly until rebuilding
+its image for another platform. Secret rotation refuses a task-platform change
+before uploading secrets. New release candidates record their workload architectures;
+legacy source-built first-party candidates without that record require an explicit
+architecture matching the image, or regeneration with the current CLI. Candidate
+deployment and migration reject platform mismatches before accessing AWS.
## Deployment directory
diff --git a/cli/src/backends/aws.ts b/cli/src/backends/aws.ts
index 1f681b6b8..cdee149bc 100644
--- a/cli/src/backends/aws.ts
+++ b/cli/src/backends/aws.ts
@@ -772,10 +772,18 @@ function releaseCandidate(config: QmConfig, path: string): AwsReleaseCandidate {
Array.isArray(candidate.images) ||
!candidate.imageProvenance ||
typeof candidate.imageProvenance !== "object" ||
- Array.isArray(candidate.imageProvenance)
+ Array.isArray(candidate.imageProvenance) ||
+ (candidate.architectures !== undefined &&
+ (!candidate.architectures ||
+ typeof candidate.architectures !== "object" ||
+ Array.isArray(candidate.architectures)))
) {
throw new CliError(`AWS release candidate ${path} has an invalid contract or targets another AWS account/region`);
}
+ for (const [workload, architecture] of Object.entries(candidate.architectures ?? {})) {
+ if (!candidate.images[workload] || (architecture !== "arm64" && architecture !== "amd64"))
+ throw new CliError(`AWS release candidate ${path} has an invalid ${workload} architecture`);
+ }
for (const [workload, image] of Object.entries(candidate.images)) {
const provenance = candidate.imageProvenance[workload];
const validProvenance =
@@ -793,6 +801,20 @@ function releaseCandidate(config: QmConfig, path: string): AwsReleaseCandidate {
) {
throw new CliError(`AWS release candidate ${path} has an invalid ${workload} image or provenance`);
}
+ const architecture = candidate.architectures?.[workload];
+ if (candidate.architectures && architecture !== workloadArchitecture(config, workload))
+ throw new CliError(
+ `AWS release candidate ${path} architecture for ${workload} does not match its configuration; use a matching architecture or rebuild the candidate`,
+ );
+ if (
+ architecture === undefined &&
+ provenance?.kind === "source-build" &&
+ isServiceName(workload) &&
+ !aws.services[workload]!.architecture
+ )
+ throw new CliError(
+ `AWS release candidate ${path} has no architecture for source-built ${workload}; set aws.services.${workload}.architecture to the image platform or rebuild the candidate`,
+ );
}
return candidate as AwsReleaseCandidate;
}
@@ -1245,6 +1267,7 @@ interface AwsReleaseCandidate {
label: string;
images: Record<string, string>;
imageProvenance: Record<string, DeploymentImageProvenance>;
+ architectures?: Record<string, "arm64" | "amd64">;
}
interface DeploymentManifest {
@@ -2322,6 +2345,7 @@ export async function awsUp(config: QmConfig, _configDir: string, opts: AwsUpOpt
label: opts.imageLabel!,
images,
imageProvenance,
+ architectures: Object.fromEntries(services.map((service) => [service, workloadArchitecture(config, service)])),
};
writeFileSync(opts.candidateOut!, `${JSON.stringify(release, null, 2)}\n`, { mode: 0o600 });
ok(`AWS release candidate written to ${opts.candidateOut}`);
@@ -3690,6 +3714,23 @@ export async function awsSecretsPush(config: QmConfig, configDir: string, envFil
const affected = workloads.filter((workload) =>
workloadSecrets(config, workload, uploaded).some((secret) => uploaded[secret.name]),
);
+ if (baseline && before) {
+ for (const workload of affected) {
+ if (aws.backgroundWorkControl && workload === "core") continue;
+ const live = awsJson<{ taskDefinition?: { runtimePlatform?: { cpuArchitecture?: string } } }>(aws, [
+ "ecs",
+ "describe-task-definition",
+ "--task-definition",
+ before.tasks[workload]!,
+ ]).taskDefinition;
+ const architecture = live?.runtimePlatform?.cpuArchitecture ?? "X86_64";
+ const expected = workloadArchitecture(config, workload) === "arm64" ? "ARM64" : "X86_64";
+ if (architecture !== expected)
+ throw new CliError(
+ `cannot rotate secrets while ${workload} uses ${architecture} but its configuration selects ${expected}; set aws.services.${workload}.architecture to match the deployed image or rebuild it with qm up --build-from`,
+ );
+ }
+ }
if (baseline?.backgroundDeploymentId) {
for (const secret of staged.filter((item) =>
["DEPLOYMENT_CONTROL_SECRET", "CORE_SIGNING_SECRET"].includes(item.name),
diff --git a/cli/src/config.ts b/cli/src/config.ts
index 9f31c77ba..fff93fd0d 100644
--- a/cli/src/config.ts
+++ b/cli/src/config.ts
@@ -113,7 +113,7 @@ export function awsWorkloadArchitecture(config: QmConfig, workload: string): "ar
`aws.services.${workload}.architecture is required because ${workload} uses an external prebuilt image`,
);
}
- return service.architecture ?? "arm64";
+ return service.architecture ?? (isServiceName(workload) ? "amd64" : "arm64");
}
export const MODEL_PROVIDERS = ["anthropic", "openai", "openrouter"] as const;
diff --git a/cli/test/aws.test.ts b/cli/test/aws.test.ts
index 8396e7283..2b2411090 100644
--- a/cli/test/aws.test.ts
+++ b/cli/test/aws.test.ts
@@ -1068,6 +1068,7 @@ test("AWS deploy can prepare an exact candidate on the inactive production stack
core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`,
},
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
}),
);
const single = oneServiceConfig();
@@ -1991,7 +1992,7 @@ else console.log("");`,
test("AWS task definitions are digest-pinned and route only computed secrets", () => {
const image = `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`;
const task = renderTaskDefinition(config, "core", image);
- assert.equal(task.runtimePlatform.cpuArchitecture, "ARM64");
+ assert.equal(task.runtimePlatform.cpuArchitecture, "X86_64");
assert.equal(task.executionRoleArn, "arn:aws:iam::123456789012:role/acme-qm-task-execution");
assert.equal(task.taskRoleArn, "arn:aws:iam::123456789012:role/acme-qm-core-task");
const container = task.containerDefinitions[0]!;
@@ -2024,6 +2025,8 @@ test("AWS task architecture allows per-workload overrides", () => {
};
const coreImage = `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`;
assert.equal(renderTaskDefinition(amd64Core, "core", coreImage).runtimePlatform.cpuArchitecture, "X86_64");
+ amd64Core.aws!.services.core!.architecture = "arm64";
+ assert.equal(renderTaskDefinition(amd64Core, "core", coreImage).runtimePlatform.cpuArchitecture, "ARM64");
});
test("AWS task parity ignores only ECS response defaults and catches live-only fields", () => {
@@ -2585,6 +2588,45 @@ test("AWS secret upload rejects active services when no deployment manifest exis
}
});
+test("AWS secret rotation rejects architecture changes before uploading or restarting", async () => {
+ const dir = mkdtempSync(join(tmpdir(), "qm-aws-secret-architecture-"));
+ const secretsConfig: QmConfig = { ...structuredClone(oneServiceConfig()), env: {} };
+ const operator = computedSecrets(secretsConfig).filter(
+ (secret) => secret.managedBy === "operator" && secret.required,
+ );
+ writeFileSync(join(dir, ".env"), operator.map((secret) => `${secret.name}=${TEST_SECRET_VALUE}`).join("\n"));
+ const fake = statefulAws(dir, secretsConfig);
+ const state = JSON.parse(readFileSync(fake.state, "utf8"));
+ const taskArn = state.services["acme-core"].taskDefinition;
+ const image = `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`;
+ state.definitions[taskArn] = renderTaskDefinition(secretsConfig, "core", image);
+ state.definitions[taskArn].runtimePlatform.cpuArchitecture = "ARM64";
+ state.dynamo = manifestItems([{ id: "current", imageLabel: "release", tasks: { core: taskArn } }], "current");
+ writeFileSync(fake.state, JSON.stringify(state));
+ try {
+ for (const architecture of [undefined, "amd64"] as const) {
+ secretsConfig.aws!.services.core!.architecture = architecture;
+ await assert.rejects(
+ () => awsSecretsPush(secretsConfig, dir),
+ /core uses ARM64 but its configuration selects X86_64/,
+ );
+ assert.doesNotMatch(
+ readFileSync(fake.log, "utf8"),
+ /secretsmanager put-secret-value|ecs (?:register-task-definition|update-service)/,
+ );
+ }
+ secretsConfig.aws!.services.core!.architecture = "arm64";
+ await awsSecretsPush(secretsConfig, dir);
+ const rotated = JSON.parse(readFileSync(fake.state, "utf8"));
+ const task = rotated.definitions[rotated.services["acme-core"].taskDefinition];
+ assert.equal(task.runtimePlatform.cpuArchitecture, "ARM64");
+ assert.equal(task.containerDefinitions[0].image, image);
+ } finally {
+ fake.restore();
+ rmSync(dir, { recursive: true, force: true });
+ }
+});
+
test("AWS secret rotation holds the deploy lease across the complete write set", async () => {
const dir = mkdtempSync(join(tmpdir(), "qm-aws-secret-lease-"));
const secretsConfig: QmConfig = { ...oneServiceConfig(), env: {} };
@@ -2789,6 +2831,7 @@ test("AWS migration runs the exact candidate core image inside the service VPC",
label: "candidate-deadbeef",
images: { core: image },
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
}),
);
const single = oneServiceConfig();
@@ -2849,6 +2892,7 @@ test("AWS builds one immutable candidate manifest and deploys its exact digest w
);
assert.equal(candidate.imageProvenance.core.kind, "source-build");
assert.equal(candidate.imageProvenance.core.source, "checkout");
+ assert.deepEqual(candidate.architectures, { core: "amd64" });
assert.match(readFileSync(dockerLog, "utf8"), /buildx build .*candidate-deadbeef/);
assert.doesNotMatch(readFileSync(fake.log, "utf8"), /dynamodb|ecs |secretsmanager/);
@@ -3066,6 +3110,7 @@ test("AWS candidate deploy fails closed on account, repository, or missing-workl
core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`,
},
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
};
const fake = fakeAws(dir, "");
try {
@@ -3085,7 +3130,7 @@ test("AWS candidate deploy fails closed on account, repository, or missing-workl
() => awsUp(oneServiceConfig(), dir, { dryRun: true, candidate: candidatePath }),
/invalid core image/,
);
- writeFileSync(candidatePath, JSON.stringify({ ...base, images: {}, imageProvenance: {} }));
+ writeFileSync(candidatePath, JSON.stringify({ ...base, images: {}, imageProvenance: {}, architectures: {} }));
await assert.rejects(
() => awsUp(oneServiceConfig(), dir, { dryRun: true, candidate: candidatePath }),
/does not contain core/,
@@ -3096,6 +3141,65 @@ test("AWS candidate deploy fails closed on account, repository, or missing-workl
}
});
+test("AWS candidate consumption rejects unknown or mismatched source image architectures before AWS access", async () => {
+ const dir = mkdtempSync(join(tmpdir(), "qm-aws-candidate-architecture-"));
+ const candidatePath = join(dir, "candidate.json");
+ const single = structuredClone(oneServiceConfig());
+ const candidate = {
+ contract: 1,
+ accountId: "123456789012",
+ region: "us-west-2",
+ label: "legacy-arm64",
+ images: { core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}` },
+ imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ };
+ const fake = statefulAws(dir, single);
+ try {
+ for (const architectures of [
+ undefined,
+ null,
+ [],
+ "amd64",
+ {},
+ { core: "arm64" },
+ { core: "invalid" },
+ { core: "amd64", unknown: "amd64" },
+ ]) {
+ writeFileSync(candidatePath, JSON.stringify({ ...candidate, architectures }));
+ await assert.rejects(
+ () => awsUp(single, dir, { yes: true, candidate: candidatePath }),
+ /architecture|invalid contract/,
+ );
+ await assert.rejects(() => awsMigrateCandidate(single, dir, candidatePath), /architecture|invalid contract/);
+ assert.equal(readFileSync(fake.log, "utf8"), "");
+ }
+ single.aws!.services.core!.architecture = "arm64";
+ writeFileSync(candidatePath, JSON.stringify(candidate));
+ await awsMigrateCandidate(single, dir, candidatePath);
+ const state = JSON.parse(readFileSync(fake.state, "utf8"));
+ const registered = Object.values(state.definitions) as Array<{
+ runtimePlatform?: { cpuArchitecture?: string };
+ containerDefinitions: Array<{ image: string }>;
+ }>;
+ assert.equal(registered.at(-1)?.runtimePlatform?.cpuArchitecture, "ARM64");
+ assert.equal(registered.at(-1)?.containerDefinitions[0]?.image, candidate.images.core);
+ delete single.aws!.services.core!.architecture;
+ writeFileSync(
+ candidatePath,
+ JSON.stringify({ ...candidate, imageProvenance: { core: { kind: "configured", source: manifestRef("core") } } }),
+ );
+ await awsMigrateCandidate(single, dir, candidatePath);
+ const configured = JSON.parse(readFileSync(fake.state, "utf8"));
+ const configuredTasks = Object.values(configured.definitions) as Array<{
+ runtimePlatform?: { cpuArchitecture?: string };
+ }>;
+ assert.equal(configuredTasks.at(-1)?.runtimePlatform?.cpuArchitecture, "X86_64");
+ } finally {
+ fake.restore();
+ rmSync(dir, { recursive: true, force: true });
+ }
+});
+
test("AWS migration failure deregisters the candidate task and releases the deploy lease", async () => {
const dir = mkdtempSync(join(tmpdir(), "qm-aws-candidate-migrate-failure-"));
const candidatePath = join(dir, "candidate.json");
@@ -3110,6 +3214,7 @@ test("AWS migration failure deregisters the candidate task and releases the depl
core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`,
},
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
}),
);
const fake = statefulAws(dir, oneServiceConfig(), {}, { migrationExitCode: 1 });
@@ -3138,6 +3243,7 @@ test("AWS candidate deploy migration failure preserves the runtime and releases
core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`,
},
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
}),
);
const fake = statefulAws(dir, oneServiceConfig(), {}, { migrationExitCode: 1 });
@@ -3908,6 +4014,17 @@ test("AWS live check uses the package-pinned source image without consulting mut
process.env.PATH = `${dir}:${priorPath}`;
try {
await assert.doesNotReject(() => awsCheckLive(single, { report: false }));
+ await assert.rejects(
+ () =>
+ awsCheckLive(
+ {
+ ...single,
+ aws: { ...single.aws!, services: { core: { ...single.aws!.services.core!, architecture: "arm64" } } },
+ },
+ { report: false },
+ ),
+ /task-definition drift.*runtimePlatform.cpuArchitecture/,
+ );
process.env.AWS_FAKE_SECRET_VALUE = "short";
await assert.rejects(() => awsCheckLive(single, { report: false }), /secret CORE_SIGNING_SECRET/);
if (priorSecretValue === undefined) delete process.env.AWS_FAKE_SECRET_VALUE;
@@ -3917,7 +4034,7 @@ test("AWS live check uses the package-pinned source image without consulting mut
const overridden: QmConfig = {
...single,
imageOverrides: { core: `ghcr.io/acme/core@sha256:${"b".repeat(64)}` },
- aws: { ...single.aws!, services: { core: { ...single.aws!.services.core!, architecture: "arm64" } } },
+ aws: { ...single.aws!, services: { core: { ...single.aws!.services.core!, architecture: "amd64" } } },
};
await assert.rejects(
() => awsCheckLive(overridden, { report: false }),
@@ -5391,6 +5508,7 @@ test("shared deployment state isolates company manifests and leases without muta
core: `123456789012.dkr.ecr.us-west-2.amazonaws.com/qm-core@sha256:${"a".repeat(64)}`,
},
imageProvenance: { core: { kind: "source-build", source: "checkout" } },
+ architectures: { core: "amd64" },
}),
);
const single = oneServiceConfig();
@@ -5557,6 +5675,7 @@ for (const mode of ["success", "migration-failure", "update-failure", "write-fai
imageProvenance: Object.fromEntries(
single.services.map((name) => [name, { kind: "source-build", source: "checkout" }]),
),
+ architectures: Object.fromEntries(single.services.map((name) => [name, "amd64"])),
}),
);
const fake = statefulAws(
diff --git a/cli/test/config.test.ts b/cli/test/config.test.ts
index ed3b98814..63cfb91a2 100644
--- a/cli/test/config.test.ts
+++ b/cli/test/config.test.ts
@@ -7,6 +7,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import {
CONFIG_FILENAME,
+ awsWorkloadArchitecture,
loadConfigAt,
loadConfigInDir,
localSandboxActive,
@@ -1505,3 +1506,19 @@ test("proxy deployment rendering names the mode and classifier", () => {
},
);
});
+
+test("AWS built-in architecture defaults match published images and retain explicit choices", () => {
+ const workloads = ["core", "web-ui", "admin", "portal", "auth", "linear"];
+ const config = {
+ plugins: [{ name: "linear" }],
+ imageOverrides: {},
+ aws: { services: Object.fromEntries(workloads.map((name) => [name, {}])) },
+ } as unknown as import("../src/config.ts").QmConfig;
+ for (const workload of workloads) {
+ assert.equal(awsWorkloadArchitecture(config, workload), workload === "linear" ? "arm64" : "amd64");
+ for (const architecture of ["arm64", "amd64"] as const) {
+ config.aws!.services[workload]!.architecture = architecture;
+ assert.equal(awsWorkloadArchitecture(config, workload), architecture);
+ }
+ }
+});
diff --git a/cli/test/terraform.test.ts b/cli/test/terraform.test.ts
index 08ba5d47f..c65423831 100644
--- a/cli/test/terraform.test.ts
+++ b/cli/test/terraform.test.ts
@@ -433,6 +433,11 @@ test("MicroVM build and runtime roles can write only their stack-owned log group
}
});
+test("terraform defaults built-in workloads to the published image architecture", () => {
+ const rendered = terraformVars(config, "", declared);
+ assert.match(rendered, /"core": \{[\s\S]*?"architecture": "amd64"/);
+});
+
test("terraform propagates workload architecture to bootstrap task definitions", () => {
const rendered = terraformVars(
{
diff --git a/docs/deploy-directory.md b/docs/deploy-directory.md
index 56017d395..f7b806497 100644
--- a/docs/deploy-directory.md
+++ b/docs/deploy-directory.md
@@ -25,7 +25,7 @@ The root object requires `contract: 1`, `orgId`, `publicUrl`, `target`, and `ser
Common optional fields select the model, plugins, extra skill directories, per-service non-secret environment values, image overrides, sandbox settings, and an external security screen. `botName` (at most 31 characters, so the generated "`<botName>` SSO" app name fits Slack's 35-character cap) names the bot everywhere users see it — the generated Slack app manifests, the prompt identity, and sign-in pages — and `orgName` (at most 40) is how the bot refers to the organization; both default to neutral values and can be changed live from the Admin page's Branding card, which then takes precedence over the deployed values. `sandbox.backend` selects the aws-target sandbox substrate (see above); `sandbox.baseImage` records the digest-pinned build input for local layer builds; `sandbox.env` is non-secret runtime environment; `sandbox.secretEnv` lists org-wide secret names whose values are forwarded to every sandbox. `securityScreen` sets `mode` (`off`, `observe`, or `enforce`) and `classifier` (`model`, the default, or `proxy`). A proxy classifier adds a lowercase `provider` label and an HTTPS `endpoint`, and requires `secretEnv.core.SECURITY_SCREEN_PROXY_TOKEN`. Absence disables screening. See "Content screening" below.
-Fly requires `region` and `flyOrg`. AWS requires a 12-digit account, region, deployment label, ECS cluster, deploy-role ARN, Secrets Manager prefix, DNS-valid Cloud Map namespace, and an entry for every enabled first-party service and discovered plugin containing a unique valid ECR repository, a unique valid ECS service, and a valid Fargate CPU/memory combination. The cluster is constrained so every IAM, RDS, ALB, and related name derived by the reference module is valid. `imageLabel` identifies the complete deployment manifest used by rollback and live drift checks; the matching OCI/ECR tag is a convenience pointer. Workloads may also set `arm64`/`amd64` architecture, non-secret build arguments, or role ARNs. External prebuilt images must declare their architecture; source-built and built-in workloads use their platform default. Cloud Map names are the private workload addresses. The reference AWS module exposes CloudFront over HTTPS and restricts its HTTP ALB origin to CloudFront's managed origin prefix. With portal enabled, it is the ALB's sole target; access to private core, web, and admin surfaces requires signed portal identity. Without portal, only core is an ALB target. A real harness requires an HTTPS `publicUrl`.
+Fly requires `region` and `flyOrg`. AWS requires a 12-digit account, region, deployment label, ECS cluster, deploy-role ARN, Secrets Manager prefix, DNS-valid Cloud Map namespace, and an entry for every enabled first-party service and discovered plugin containing a unique valid ECR repository, a unique valid ECS service, and a valid Fargate CPU/memory combination. The cluster is constrained so every IAM, RDS, ALB, and related name derived by the reference module is valid. `imageLabel` identifies the complete deployment manifest used by rollback and live drift checks; the matching OCI/ECR tag is a convenience pointer. Workloads may also set `arm64`/`amd64` architecture, non-secret build arguments, or role ARNs. External prebuilt images must declare their architecture; built-in workloads default to amd64 to match published images, while source-built plugins default to arm64. Explicit architecture choices are honored for source builds. Cloud Map names are the private workload addresses. The reference AWS module exposes CloudFront over HTTPS and restricts its HTTP ALB origin to CloudFront's managed origin prefix. With portal enabled, it is the ALB's sole target; access to private core, web, and admin surfaces requires signed portal identity. Without portal, only core is an ALB target. A real harness requires an HTTPS `publicUrl`.
`publicUrl` is the one public coordinate. The CLI derives the core, Slack, web, admin, and portal URL environment from it. Config `env` is for non-secret values only; secret-shaped keys are rejected.