Multiplayer agent harness for work.
Let qm up replace a relinquished core cohort while its members finish admitted work (#2047)
Regan Bellc8e8515602aa
diff --git a/cli/README.md b/cli/README.mdindex 941419b7f..1fd2489db 100644--- a/cli/README.md+++ b/cli/README.md@@ -110,9 +110,9 @@ while changing durable ownership independently.The exported `awsBackgroundWorkCapacity(config, configDir, candidatePath?)` provesthat an inactive controlled stack is currently reusable. It requires another-owner, fully drained current membership, stable native deployments and exact task-inventories for every workload, resolved deployment preparation, and explicitly-disabled protection on every current core task. It never changes task protection+owner, a current membership that has relinquished, stable native deployments and+exact task inventories for every workload, resolved deployment preparation, and+explicitly disabled protection on every current core task. It never changes task protectionor deployment state. The deploy role needs `ecs:GetTaskProtection` on its tasks.The result binds the manifest and deployment identities, ownership generation,workload task definitions, native deployment IDs, task ARNs, and protection proof.@@ -137,7 +137,9 @@ requests without restarting ECS tasks. Activation waits for prior owners to stopclaiming and every expected task to finish activation. Pausing stops new claims;in-flight turns can continue draining. Replacing or rolling back the active corecohort requires an explicit pause or handover and proof that every member has-drained first. A demotion refuses to pause a different current owner. Unresponsive members are never+relinquished first; members still finishing admitted work keep running under+task protection beside the replacement tasks until they drain. A demotion+refuses to pause a different current owner. Unresponsive members are neverassumed dead: `awsRetireBackgroundWorkMembers` requires exact instance, task ARN,and generation identities plus ECS evidence that each task stopped. This recoveryis also available before bootstrap for stopped legacy members.diff --git a/cli/src/backends/aws.ts b/cli/src/backends/aws.tsindex cdee149bc..6e7a304c5 100644--- a/cli/src/backends/aws.ts+++ b/cli/src/backends/aws.ts@@ -3256,7 +3256,7 @@ export async function awsBackgroundWorkCapacity(const ownership = await readBackgroundWork(transport, cohort.deploymentId);if (!ownership.enabled || !ownership.desiredDeploymentId || ownership.desiredDeploymentId === cohort.deploymentId)throw new CliError("capacity requires enabled background ownership held by another deployment");- assertBackgroundMembersDrained(ownership, cohort.deploymentId);+ assertBackgroundCohortRelinquished(ownership, cohort.deploymentId);if (cohort.taskArns.some((taskArn) =>@@ -3265,11 +3265,11 @@ export async function awsBackgroundWorkCapacity(member.taskArn === taskArn &&member.deploymentId === cohort.deploymentId &&!member.retired &&- member.state === "drained",+ member.state !== "admitted",),))- throw new CliError("capacity requires every current core task to be enrolled and drained");+ throw new CliError("capacity requires every current core task to be enrolled and relinquished");const states = describedServices(config, workloads);assertOwnedServices(config, states, workloads);const snapshot = serviceSnapshotFromStates(states, workloads);@@ -3334,7 +3334,7 @@ export async function awsBackgroundWorkCapacity(finalOwnership.desiredDeploymentId !== proof.desiredDeploymentId)throw new CliError("background ownership changed while checking capacity");- assertBackgroundMembersDrained(finalOwnership, cohort.deploymentId);+ assertBackgroundCohortRelinquished(finalOwnership, cohort.deploymentId);if (cohort.taskArns.some((taskArn) =>@@ -3343,7 +3343,7 @@ export async function awsBackgroundWorkCapacity(member.taskArn === taskArn &&member.deploymentId === cohort.deploymentId &&!member.retired &&- member.state === "drained",+ member.state !== "admitted",),))@@ -3488,18 +3488,18 @@ async function assertBackgroundCohortReplaceable(): Promise<void> {if (!current?.backgroundDeploymentId) return;const status = await readBackgroundWork(awsBackgroundWorkTransport(config), current.backgroundDeploymentId);- if (status.enabled) assertBackgroundMembersDrained(status, current.backgroundDeploymentId);+ if (status.enabled) assertBackgroundCohortRelinquished(status, current.backgroundDeploymentId);}-function assertBackgroundMembersDrained(status: BackgroundWorkStatus, deploymentId: string): void {+function assertBackgroundCohortRelinquished(status: BackgroundWorkStatus, deploymentId: string): void {if (status.desiredDeploymentId === deploymentId ||status.members.some(- (member) => !member.retired && member.deploymentId === deploymentId && member.state !== "drained",+ (member) => !member.retired && member.deploymentId === deploymentId && member.state === "admitted",))throw new CliError(- "pause or hand over background ownership and wait for every member to drain before replacing the current core cohort",+ "pause or hand over background ownership and wait for every member to relinquish before replacing the current core cohort",);}diff --git a/cli/test/aws.test.ts b/cli/test/aws.test.tsindex 2b2411090..4f151e454 100644--- a/cli/test/aws.test.ts+++ b/cli/test/aws.test.ts@@ -5773,7 +5773,7 @@ test("AWS deployment progress rejects invalid options before AWS calls", async (}});-test("inactive capacity proves exact drained unprotected cohorts without mutating deployment state", async () => {+test("inactive capacity proves exact relinquished unprotected cohorts without mutating deployment state", async () => {const dir = mkdtempSync(join(tmpdir(), "qm-capacity-"));const dockerBin = join(dir, "docker");writeFileSync(dockerBin, `#!/usr/bin/env node\nconsole.log("Digest: sha256:${"a".repeat(64)}");\n`);@@ -5857,11 +5857,12 @@ test("inactive capacity proves exact drained unprotected cohorts without mutatin/transact-write|put-item|delete-item|update-item|update-service|register-task-definition|run-task|update-task-protection|stop-task/,);- for (const state of ["admitted", "relinquished"]) {- ownership.members[0]!.state = state;- reset();- await assert.rejects(awsBackgroundWorkCapacity(configured, dir), /every member to drain/);- }+ ownership.members[0]!.state = "admitted";+ reset();+ await assert.rejects(awsBackgroundWorkCapacity(configured, dir), /every member to relinquish/);+ ownership.members[0]!.state = "relinquished";+ reset();+ assert.deepEqual(await awsBackgroundWorkCapacity(configured, dir), proof);ownership.members[0]!.state = "drained";ownership.desiredDeploymentId = manifest.backgroundDeploymentId;reset();@@ -6211,9 +6212,10 @@ test("controlled AWS cohorts bind immutable identities and hand over without ECSassert.equal(retired.generation, 3);assert.equal(retired.members.at(-1)!.retired, true);await awsSetBackgroundWork(single, dir, false);- await assert.rejects(awsUp(single, dir, { yes: true, restart: ["core"] }), /every member to drain/);- await assert.rejects(awsRollback(single, manifest.id), /every member to drain/);- ownership.members[0]!.state = "drained";+ ownership.members[0]!.state = "admitted";+ await assert.rejects(awsUp(single, dir, { yes: true, restart: ["core"] }), /every member to relinquish/);+ await assert.rejects(awsRollback(single, manifest.id), /every member to relinquish/);+ ownership.members[0]!.state = "relinquished";await awsUp(single, dir, { yes: true, restart: ["core"] });const replacement = JSON.parse(readFileSync(fake.state, "utf8"));const replacementManifest = JSON.parse(diff --git a/docs/background-ownership.md b/docs/background-ownership.mdindex c8c08a588..c3fd03fee 100644--- a/docs/background-ownership.md+++ b/docs/background-ownership.md@@ -46,7 +46,7 @@ A process records `admitted` before starting background resources. The desired d`relinquished` acknowledges that the process stopped new claims and closed its Slack ingress. Already admitted turns may still run. Cron polling can resume after rollback while earlier callbacks keep their queue connection and heartbeats. Maintenance callbacks that cannot relinquish safely are joined before acknowledgment.-`drained` means the process's admitted background work has finished. Releasing ownership and making a deployment safe to replace are separate gates. Do not terminate a relinquished but undrained process merely to meet a rollout time target.+`drained` means the process's admitted background work has finished. Releasing ownership and making a deployment safe to replace are separate gates: `qm up` replaces a cohort once every member has relinquished, and members still finishing admitted work keep running beside the new tasks under their leases and task protection until they drain. The ECS service's `maximumPercent` must leave room for those lingering tasks. Do not terminate a relinquished but undrained process merely to meet a rollout time target.Database errors or an expired local validity watchdog fence new local work. They do not establish durable relinquishment or authorize another deployment to bypass an outstanding member.